The laboratory
Perfect JSON, completely wrong answer
Three gates: syntax, schema and fidelity to the document. One is not enough.
Constructed experiment · no model running
The situation · Extract the right information
A team is arranging a customer meeting. The message mentions headquarters in Paris and a meeting in Lyon. The assistant must fill the meeting-city field; a mistake would send the team to the wrong place.
What you will learn
Understand why usable formatting and correct information are separate requirements.
Your mission
Inspect and edit the proposed answer, identifying exactly which check fails. Explain why Paris is plausible but incorrect.
By the end of this activity. You will be able to define an extraction contract and separate syntax, structure and meaning errors.
Your turn
The headquarters is in Paris. The meeting will take place in Lyon.
Extract only the meeting city into meeting_city (string or null). No extra fields. Use null only when the city is absent.
Paris appears in the document but has the wrong role. Constrained output can guarantee a form; it does not prove that the right information was extracted.
Why we chose this document
The text is deliberately short so you can determine the answer without a model. Its cities are not interchangeable. Paris is associated with the headquarters, while Lyon is associated with the meeting. Correct extraction must recognize this relationship. Simply detecting a city name is insufficient.
The initial output contains Paris. Look at the three checks before correcting it. Passing the first two does not contradict failing the third: they concern different properties. The JSON can be parsed and the value has the right type, but its meaning does not match the request.
Understand what each gate checks
The syntax check acts like a strict reader requiring JSON quotation marks, braces and separators. The schema check then examines the object’s organization. It rejects a list of cities or a number, for example, even though both can be valid JSON. Finally, the fidelity check compares the value with our reference.
A downstream check is not evaluated when the preceding check does not provide a usable structure. This distinction supports diagnosis: an unreadable output should not be presented as though its meaning had already been checked.
In a real project, the reference must come from the document and an explicit annotation rule. Here it is known in advance to make the reasoning visible. The capsule does not claim to automatically verify the truth of arbitrary text.
Three gates, three questions
The first check asks whether the text can be parsed as JSON. The second requires an object with exactly one meeting_city field, containing a string or null. The third compares its value with the meeting city in our document: Lyon.
Sabotage your answer
Remove a quotation mark to break syntax. Replace the city with 42 to break the schema. Write Paris to pass the first two checks and fail the last. Then try null: its format is acceptable, but it is wrong here because the city is given.
The capsule’s limit
The comparison uses a known reference and exact equality. A real system must handle spelling variants, ambiguous documents and missing information. It must distinguish absence, uncertainty and values outside a catalogue. No model generates the answers in this experiment.