Valid JSON is only the beginning

An AI assistant is asked to update a record. It returns well-formed JSON, the request parses, and the tool accepts it. Has the task succeeded? Only if the resulting record is the one the user requested.

That distinction becomes sharper when the tool cannot express the desired change at all. The following example uses an invented record and a deliberately restricted contract: one update request, with an object as the final record.

One request, two meanings

Start with {"nickname":"River","active":true}. The request is: set nickname to JSON null, keep the field, and leave active unchanged.

Under JSON Merge Patch, RFC 7396 §2, a null-valued member in an object patch removes that member. So {"nickname":null} deletes the field. It does not set the field’s value to null.

Under JSON Patch, RFC 6902 §4.3, an operation such as {"op":"replace","path":"/nickname","value":null} replaces the existing value with literal null. The same desired state therefore has a different feasibility answer under these two interfaces.

{"nickname": null}

{"active": true}

The boundary needs to be stated precisely

The limitation concerns introducing this object member with a null value through the declared single-request Merge Patch contract. It is not a claim that Merge Patch cannot contain null anywhere. Existing null-valued fields can survive omission, and replaced arrays can contain null. Root replacement also has different semantics; replacing the entire record with null would violate our object-valued final-record requirement.

Those details are what make an evaluation interpretable. If the rules change halfway through, a supposed model failure may actually be a disagreement about the allowed tools.

Score the decision and the result

For this example, I would ask an evaluator to keep three questions separate. Is the response structurally valid? Can the tool execute it? Does the final state exactly match the requested state without unintended changes?

A fourth question comes first when the interface is restrictive: should the assistant attempt an update at all? A system that always refuses would avoid some bad edits but be useless on feasible ones. A system that always acts would fail when the contract cannot express the request.

A useful paired diagnostic gives the same starting record and desired state to both interfaces. Success requires the appropriate decision for each: explain the limitation where the target is unrepresentable, and produce the correct change where it is representable. That is a proposed evaluation design, not evidence that any model succeeds or fails it.

From a toy example to an honest study

My structured-editing prototype explores this kind of boundary. Its current cases are synthetic development fixtures. They can reveal mistakes in an evaluator or clarify a question, but they do not establish how often the same problem occurs in real applications.

The next useful evidence would be a fixed, independently checked task set, explicit model and tool settings, and results that retain invalid responses and unnecessary refusals. If the chosen models solve the diagnostic easily, that is information too. The purpose is to find a defensible boundary, not to manufacture a failure.