Imagine testing an AI tutor with this made-up exchange. A learner says that one third plus one third is two sixths. The tutor replies: “Think about what the bottom number means.” The learner changes the answer to two thirds.

Did the hint help? If the learner was another AI that already understood fractions, the exchange tells us surprisingly little. It might have reached the answer after almost any nudge. A convincing beginner has to make the right kinds of mistakes, not merely sound unsure.

That problem is the focus of StudentSim, described by Microsoft Research on October 12. It is a research system for developing AI tutors, rather than a classroom product. The idea is to use simulated learners to try out teaching strategies before undertaking studies with people.

Getting things wrong is part of the job

In our imaginary fraction lesson, consider two learners. One adds the top and bottom numbers separately. Another understands the rule but makes an occasional arithmetic slip. The same wrong answer would not necessarily mean they need the same explanation.

StudentSim starts with records from real learners. It first trains on pooled records within a subject, then adapts a model to one individual's records. Microsoft's explanation describes this as a way to learn shared patterns while preserving differences between students. Simply telling a capable model to act like a beginner can leave too much of its underlying knowledge in play.

The September 1 preprint evaluates simulators representing 60 learners across chess, English as a second language, and mathematics. It asks two separate questions:

  • Does the simulator resemble the learner? Its unaided responses are compared with recorded learner behavior, using measures suited to each subject.
  • Can it follow guidance? After receiving help, does it reach the corrected answer that the guidance points toward?

Those questions are useful, but the second has a limit: reaching the intended answer does not establish that the real learner would react the same way to that hint. The paper's discussion of the measure explains what it scores. A model can follow an explanation without accurately predicting a person's learning.

What would count as better teaching?

The researchers also used simulated feedback to train a chess tutor. Human evaluators rated that tutor's responses more highly than the two comparison conditions. These were judgments about the tutor's output, not measurements of improved learning by students. The authors present the experiment as a proof of concept. See the study's tutor evaluation.

The training and evaluation code is public, so other researchers can examine the approach. That makes it inspectable; it does not turn the reported results into an independent replication.

For someone choosing a learning tool, a useful follow-up question is: what can the learner do after the help goes away? In our fraction example, try a different problem later and ask for an explanation. Copying today's corrected answer and understanding tomorrow's question are different outcomes.

Simulated students could make it easier to improve an AI tutor before people spend time with it. The next important evidence would connect those improvements to real learners tackling problems on their own.