An AI can agree with itself and still be wrong

Imagine asking a language model whether “a red bicycle” is a kind of “bicycle.” Then ask the reverse question. In this simplified example, the first answer should be yes and the reverse should be no: the second description is broader.

If a system says yes both ways, it has given us something useful to investigate. But detecting a contradiction is only the beginning. It does not tell us which answer to change, whether a second attempt will help, or how many wrong answers remain outside the check.

Agreement has a blind spot

Consider a different question: what is the capital of France? A system that answers “Lyon” twice is consistent and wrong. This is a deliberately simple counterexample, not a report of any model’s behavior. Agreement alone cannot establish that an answer matches the world.

For directional phrase relations, the same distinction matters. Two labels may obey the expected reversal rule while both disagree with the reference relation. A consistency score and a correctness score therefore answer different questions. One measures compatibility between predictions; the other compares predictions with an external reference.

A check is useful when its signal helps us make a better decision—not merely when it makes the outputs look tidier.

The research question is about selection

Suppose a system can afford to recheck only some of its answers. One policy spends those calls on conflicts. Another spends the same number on randomly selected eligible cases. A third changes how the original answer is produced, perhaps predicting both directions together.

These are different strategies. Beating random rechecking would suggest that the selection signal carries useful information in that setting. It would not show that the whole system is better than a simpler approach. The comparison needs both kinds of baseline.

This is the question behind my ongoing selective-verification study: can conflicting forward and reverse judgments identify productive opportunities for another attempt? The study concerns a bounded phrase-relation task. Any conclusion will need to stay within the evaluated data and model settings.

Count the errors the check never sees

A useful evaluation should account for at least four outcomes: an incorrect answer becomes correct; a correct answer becomes incorrect; an error survives the recheck; or an error is never selected at all. Reporting only successful repairs hides the other three.

The last category is especially revealing. A very selective trigger can look impressive on the few cases it handles while missing most of the problem. The fraction of original errors reached by the trigger belongs beside the headline score.

Some labels may not support the reversal rule. They need their own accounting rather than being silently described as consistent. A failure to produce a usable response also needs to stay in the appropriate denominator.

A fair budget needs more than a call count

Matching the number of rechecks is a useful first control. It is not the whole cost comparison. Longer inputs and outputs can change token usage, and retries can change both spending and latency. The cost of collecting every experimental condition is also different from the cost of running one selected policy.

Before drawing a practical conclusion, I want to see the accuracy comparison, the errors reached, the new errors introduced, and the resource cost together. A small improvement may be worth it in one workflow and irrelevant in another. That is a decision the evaluation should help the reader make.

Where this fits

Consistency-based methods have an established history. Mitchell and colleagues’ ConCoRD paper studies consistency correction using natural language inference and joint optimization. My question here is narrower: whether a conflict signal helps allocate rechecks under a bounded budget. The idea that consistency can matter is prior work, not a novelty claim.

The most useful outcome may be a boundary: a clear account of when the signal helps, what it misses, and which simpler alternatives remain competitive. That would be a result worth keeping.