Imagine you want to build a study app. It turns class notes into quizzes and gives extra practice on questions a student misses. You describe it to a chatbot, which calls the idea “highly original.”

Before spending a weekend building it, you would want to know what that judgment rests on. Which existing tools did it examine? What, specifically, would yours do differently?

This is an invented example. It illustrates a practical distinction: an encouraging description and a documented comparison answer different questions.

Testing the judge

In an October 27 announcement, Ai2 described a study testing six AI models as judges of research originality. Its underlying preprint first appeared October 1 and was revised October 26.

One result is striking. For comparisons between human-written and AI-generated research ideas, the authors reported GPT-5.4 accuracy of 93.5% with one prompt and 40.9% with another, measured against their benchmark labels.

What changed matters. The first prompt told the model that conference reviewers had found exactly one idea novel. The second asked which idea was novel without that reviewer framing. As the paper explains, these instructions supply different information. This was not a test of swapping two equivalent words.

The result shows why the judging setup deserves scrutiny. A verdict can depend substantially on the assumptions supplied alongside the idea.

The answer key has limits too

For that comparison, the researchers assumed the generated ideas were less original than selected human ideas endorsed by reviewers. They did not independently verify every such label. The paper acknowledges this uncertainty.

Its findings concern research ideas in AI. They do not establish an error rate for evaluating your app, a business pitch or a novel. The public project includes code and data; rerunning the evaluations requires external model APIs. This article does not reproduce those experiments.

Ask for something you can inspect

For the imagined study app, a useful next request would be:

Find the closest existing tools or published approaches. Link to each source, explain the overlap with my proposal, and identify a specific difference worth investigating. Say what you searched and what you could not check.

This is a suggested working method, not a prompt the study proved reliable. You still need to open the sources and check whether they support the comparison.

Make the proposed difference concrete. “Better learning” is an ambition. “Changes tomorrow’s practice based on which step of today’s explanation was wrong” is a behavior you can compare with other systems. It may already exist; specificity makes that easier to investigate.

Also separate originality from usefulness. An idea could overlap with earlier work and still solve a worthwhile problem for a particular group. Conversely, finding an unusual combination tells you little about whether anyone needs it.

Keep a short record of the nearest examples, the overlap and the unresolved difference. That gives you a better next step than a novelty score: something to read, test or revise before you commit to the idea.