Exam questions are not asked in the abstract. They are grouped under a described production system — a support agent, a research pipeline, a build integration — and the description persists across all the questions in that group. Six such systems exist in the blueprint, and any given exam paper draws four of the six. You cannot choose which four.
That constraint is useful, because each system has a fixed repertoire of things that go wrong, and the same failure produces the same intended answer every time. Learn the repertoire and the questions stop feeling novel.
| Column | What it contains |
|---|---|
| What the question describes | The observable problem, written the way a question would present it — logs, metrics, a complaint, a symptom in production. |
| What is actually wrong | The underlying cause. This is usually one layer upstream of the symptom, and identifying it is most of the work. |
| The intended answer, and why | The option the exam wants, plus the reason it beats the tempting alternative. Terms are defined in the . |
| Task | The syllabus code being tested. Clickable — it opens the full treatment. Every code is decoded in the . |
Where the guide's own worked examples sit. The blueprint prints twelve fully explained sample questions, grouped three each under scenarios 1, 2, 3 and 5. Scenarios 4 and 6 get none, so for those two the practice items on this site are the only worked examples you have. All twelve originals are reproduced in the , tagged Official item.
Coverage note. Domain 1 appears in scenarios 1, 3 and 4; Domain 3 in scenarios 2, 4 and 5. Whichever four you draw, your two priority domains will be on the paper.
You are building a customer support resolution agent using the Claude Agent SDK. The agent handles high-ambiguity requests like returns, billing disputes, and account issues. It has access to your backend systems through custom Model Context Protocol (MCP) tools (get_customer, lookup_order, process_refund, escalate_to_human). Your target is 80%+ first-contact resolution while knowing when to escalate.
Primary domains: Agentic Architecture & Orchestration · Tool Design & MCP Integration · Context Management & Reliability
An autonomous customer support agent built on the Agent SDK. It handles high-ambiguity requests — returns, billing disputes, account problems — where the right sequence of steps is not known before the conversation starts. That ambiguity is why it is built as a loop that decides its own next action rather than as a fixed script.
Its tools, all custom MCP tools: get_customer (identify and verify who you are talking to), lookup_order (fetch order details), process_refund (move money), escalate_to_human (hand off to a person). Its target: resolve 80% or more of cases on first contact, while escalating the right ones.
Why those four tools matter. Two of them are dangerous — process_refund moves money, escalate_to_human ends the automated path — and two of them look similar to a model choosing between them. Nearly every question in this scenario grows out of one of those two facts.
| What the question describes | What is actually wrong | The intended answer, and why | Task |
|---|---|---|---|
The agent skips get_customer in 12% of cases and refunds the wrong account | Nothing enforces the ordering; the requirement lives only in the prompt | Block lookup_order and process_refund in code until get_customer has returned a verified customer ID. Prompt emphasis and worked examples both raise compliance without guaranteeing it, and a 12% chance of refunding a stranger is not a rate you can improve your way out of. | |
The agent calls get_customer when the user asks about an order. Both tools say only "retrieves customer information" and "retrieves order details" | The descriptions are too thin to distinguish the tools | Expand both descriptions: what each accepts, example queries, edge cases, and explicitly when to use it rather than the other tool. The model selects from the description alone, so this is the cause. Consolidating the two tools would work but is far more than a "first step" warrants. | |
| First-contact resolution is 55% against a target of 80%. Logs show easy cases escalated and policy-exception cases attempted | The escalation decision boundary was never defined | Write explicit escalation criteria into the system prompt with two to four examples showing escalate-versus-resolve reasoning. Rejected alternatives: a self-reported confidence score, because the agent is already confidently wrong on hard cases; a classifier trained on ticket history, because prompt work has not been tried; sentiment analysis, because how upset someone is does not track how complex their case is. | |
| Refunds above the $500 policy threshold occasionally get processed anyway | A hard business threshold is being enforced by persuasion | Intercept the outgoing process_refund call, block it above the threshold, and route the case into human escalation. Note the second half: blocking without redirecting leaves the customer with an unresolved issue and no path forward. | |
| Three tools return timestamps differently — epoch seconds, ISO 8601 strings, numeric status codes — and the agent miscalculates return windows | The model is being asked to do format conversion on every turn | A hook on tool results normalises the payloads into one shape before the model ever sees them. Deterministic transformation belongs in code; asking the model to juggle three formats is a recurring cost and a recurring risk. | |
| After summarisation, the agent starts quoting vague amounts and dates — "a refund for a delayed order" instead of the figure it promised | Summarisation is lossy exactly where precision matters | Extract the transactional facts — amount, order number, date, status — into a persistent block included in every prompt that sits outside the summarised history. Instructing the summariser to preserve numbers still routes them through the lossy step. | |
| Each order lookup returns 40-plus fields; five are relevant. After ten calls the context window is exhausted | Verbose tool output is accumulating unfiltered | Trim the tool's output at the source so only the fields the next step needs enter the conversation. Summarising afterwards is lossy on the very numbers you must keep verbatim, and compaction is remediation rather than prevention. | |
get_customer returns three possible matches | The agent is about to guess which human being it is acting for | Ask the customer for an additional identifier — an order number, a registered email. Never select by heuristic such as "the account with the most orders", because the cost of guessing wrong is acting on a stranger's account. | |
| The customer's opening message is "I don't want a bot, transfer me to a person" | An explicit request is being treated as a negotiable preference | Escalate immediately, without investigating first. "Let me just check your account" reads as ignoring the request and costs trust. Contrast with a merely frustrated customer whose problem you can actually solve: there, acknowledge and offer the fix, and escalate only if they repeat the request. | |
| The human agent receiving an escalation gets only a link they cannot open, and re-interviews the customer | The handoff carries no compiled context | Send a structured summary: customer ID, root cause, amount, and recommended action. A transcript dump is volume rather than context — it makes the human redo the analysis the agent already did. | |
| One message contains three separate complaints — a damaged item, a double charge, an ignored email | The workflow assumes one issue per conversation | Split it into three items, investigate them in parallel over the shared customer context, and answer with one unified resolution. Handling the first and asking them to write back about the rest destroys the first-contact metric and pushes your decomposition work onto the customer. | |
| Every tool failure returns the text "Operation failed". The agent retries policy rejections eight times and tells customers "something went wrong" | All failures look identical, so no recovery decision is possible | Return a category (transient, validation, business, permission), a retryable flag, and a human-readable explanation. Then a timeout gets retried, a policy rejection gets explained to the customer in plain language, and neither is confused for the other. |
19 practice items are tagged to this scenario — including the 3 worked items printed in the guide. The quiz explains every wrong option and names the task statement it tests.
You are using Claude Code to accelerate software development. Your team uses it for code generation, refactoring, debugging, and documentation. You need to integrate it into your development workflow with custom slash commands, CLAUDE.md configurations, and understand when to use plan mode vs direct execution.
Primary domains: Claude Code Configuration & Workflows · Context Management & Reliability
A development team using Claude Code — the command-line coding tool — for generating code, refactoring, debugging and documentation. The questions here are about configuration: which file an instruction goes in, which scope shares it with the team, and whether to plan before acting.
The vocabulary you need. CLAUDE.md is a plain markdown file of standing project instructions, loaded automatically at the start of every session. .claude/ is a directory inside the repository holding project configuration, so anything in it travels to teammates through version control. A path beginning ~/ is the user's home directory, so anything there is personal and never shared. A skill or slash command is a saved prompt you invoke by typing /name. Plan mode is a read-only mode where Claude drafts an approach and cannot modify files until you approve it.
| What the question describes | What is actually wrong | The intended answer, and why | Task |
|---|---|---|---|
The team wants a /review command running their checklist, available to every developer who clones the repository | Nothing yet — this is a placement question | .claude/commands/ in the project, because it is version-controlled and therefore arrives with a clone or pull. The home-directory equivalent is personal and would never reach anyone else. A .claude/config.json with a commands list does not exist and appears as a distractor. | |
| A new team member does not get the standards, although they work perfectly for the person who wrote them | The standards are in personal configuration | They are in ~/.claude/CLAUDE.md, which is user-level and never enters version control. Move them to project level so they are shared and reviewable. The instinct being trained: when a rule works for one person and not another, check the file path before anything else. | |
| Different areas need different conventions, and test files sit beside the components they test, all over the tree | The convention is being stored somewhere location-bound | Rule files in .claude/rules/ with a filename-pattern filter in their frontmatter, so the rule activates whenever a matching file is touched regardless of folder. An instruction file placed in a directory only covers that directory, which cannot follow scattered files. | |
| Restructuring a monolithic application into services — dozens of files, plus decisions about service boundaries | Nothing — this is a mode-selection question | Plan mode, because the requirements already state multi-file scope and architectural decisions. "Start executing and switch to planning if complexity appears" is wrong precisely because the complexity is stated up front rather than latent. | |
| A single-file bug with a clear stack trace | Nothing — the inverse question | Direct execution. Planning a two-line fix is the same error in the other direction, and the exam tests both. Scope, number of valid approaches, and architectural weight are the discriminators. | |
| The instruction file has grown to 1,100 lines and adherence has got worse, not better | Bloat measurably reduces how reliably instructions are followed | Split topics into separate rule files, and reference shared standards with an import rather than duplicating them. Also cut what the tool can work out for itself — folder layouts, dependency lists — and keep the pitfalls and the reasoning. Target is roughly 200 lines. | |
| Behaviour is inconsistent between sessions in the same repository | You do not know which configuration files are actually being loaded | Inspect the load rather than guessing. The exam keys the /memory command; in current versions /context is what reports the loaded memory files — both are covered in . Either way the lesson holds: diagnose, do not shuffle files hopefully. | |
| A codebase-analysis skill produces 4,000 lines of dependency output and wrecks the rest of the session | Verbose output is entering the main conversation | Set context: fork in the skill's frontmatter so it runs in an isolated sub-context and returns only a summary. Restricting its tools would cripple the analysis; the problem is where the output lands, not what the skill may do. | |
| A described data transformation is interpreted three different ways across three attempts | Prose is carrying a specification it cannot carry | Give two or three concrete input/output pairs. More precise prose is the thing that has already failed three times, and examples pin down format and edge handling simultaneously. | |
| Working in an unfamiliar domain where you do not yet know what the failure modes are | You cannot specify what you have not thought of | Ask Claude to interview you before implementing. The questions surface decisions you had not pinned down — caching behaviour, error handling, edge cases — and turn a vague request into a specification. | |
| A long exploration session degrades in accuracy | Findings exist only in a context window that is filling up | Write key findings to a scratchpad file the agent re-reads, push verbose investigation into a separate agent that returns a summary, and reclaim context between phases rather than after quality has dropped. |
25 practice items are tagged to this scenario — including the 3 worked items printed in the guide. The quiz explains every wrong option and names the task statement it tests.
You are building a multi-agent research system using the Claude Agent SDK. A coordinator agent delegates to specialized subagents: one searches the web, one analyzes documents, one synthesizes findings, and one generates reports. The system researches topics and produces comprehensive, cited reports.
Primary domains: Agentic Architecture & Orchestration · Tool Design & MCP Integration · Context Management & Reliability
A research pipeline built on the Agent SDK. One coordinator agent plans the work and delegates to four subagents: web search, document analysis, synthesis, and report generation. The output is a comprehensive report with citations.
The vocabulary you need. A coordinator is the planner: it breaks the task into subtasks, hands each to a specialist, combines what comes back, and decides which specialists are needed at all. A subagent is a specialist with its own separate context window and its own restricted tool set. They communicate only with the coordinator, never with each other — a shape called hub-and-spoke, which exists so there is one place to observe the workflow, handle failures consistently, and control what each specialist sees.
The one fact that generates the most questions: a subagent inherits nothing. Not the coordinator's conversation, not another subagent's output, and nothing from a previous invocation of itself. Anything it needs must be written into its prompt.
| What the question describes | What is actually wrong | The intended answer, and why | Task |
|---|---|---|---|
| Asked about "AI in creative industries", the report covers only visual arts. The coordinator's log shows three subtasks: digital art, graphic design, photography | The coordinator's decomposition was too narrow | Every subagent executed its assigned scope correctly. Music, writing and film are absent because nobody was asked to research them, and no downstream agent can recover work that was never commissioned. Options blaming the search agent's queries, the analysis agent's relevance filter, or the synthesis agent's gap detection all point at agents that behaved correctly. | |
| The coordinator never delegates. Its configuration lists tools for reading, searching and web search | The delegation tool itself is missing from its permitted tools | The list must include the delegation tool — called Task in the blueprint, renamed Agent in current versions. Without it the coordinator physically cannot create a subagent, so it does the work itself with whatever it holds. No amount of prompt instruction grants a capability the configuration withholds. | |
| The synthesis agent writes generic statements and ignores what the search agent found, even though the coordinator has all the findings | The findings were never transmitted | Put the search and analysis results directly into the synthesis agent's prompt. This is not a capacity problem, so a larger context window changes nothing — there is nothing in that window to begin with. And synthesis cannot fetch the results itself, because subagents do not talk to each other. | |
| Research takes 90 seconds because search runs, then analysis, then synthesis — although the first two do not depend on each other | Independent work is being serialised | Have the coordinator emit both delegation calls in a single response. Issuing them in consecutive turns is still sequential, because the loop appends each result before the next turn begins. Merging the two specialists into one agent would fake parallelism at the cost of scoping and quality. | |
| The search subagent times out | Nothing yet — the question is how to report it | Return the failure type, the query attempted, any partial results already gathered, and possible alternative approaches. That is what lets the coordinator choose between retrying with a narrower query, substituting a source, and proceeding with partial coverage that it then annotates as a gap. | |
| The synthesis agent constantly bounces back for fact-checks. 85% are simple lookups — dates, names, figures — and each round trip adds 40% latency | A high-frequency simple need is routed through an expensive path | Give synthesis one narrow verification tool for the simple case, and keep the complex 15% flowing through the coordinator. Giving it every search tool over-provisions it, and agents holding tools outside their role misuse them. | |
| The synthesis agent, given 18 tools, starts performing its own web searches | Its toolbox is far wider than its role | Scope each agent's tools to its job. Selection reliability degrades as the toolbox grows — the guide's own contrast is 18 tools where four or five would do — and an agent will use what it is given. | |
| Citations disappear during synthesis, so the final report cannot attribute its claims | Attribution was carried as prose and got compressed away | Pass structured findings where each item keeps its claim, an evidence excerpt, the source name or URL, and a date — and require synthesis to preserve and merge those mappings. "Remember to cite sources" does not survive summarisation. | |
| Two credible sources give different figures for the same market size | An agent is arbitrating a question it should be reporting | Keep both values with their attribution and dates, mark the disagreement, and let the coordinator decide reconciliation. Choosing silently hides information; averaging manufactures a figure no source published. | |
| A 2021 figure is treated as contradicting a 2024 figure | Facts are travelling without their dates | Require publication or collection dates in the structured output. Both numbers can be correct; without dates the pipeline has no way to know that, and will either flag a false contradiction or silently drop one. | |
| In a 40,000-token aggregate of findings, items from the opening and closing sections appear in the report and the middle vanishes | A positional effect, not a capacity one | Lead with a key-findings summary and organise the detail under explicit section headers. Long inputs are processed most reliably at the beginning and end; instructing the model to "read carefully" does not move a positional effect. | |
| The report has gaps because one source was unavailable, and the reader cannot tell which conclusions are thin | Coverage is being presented as uniform when it is not | Annotate the synthesis: which findings are well supported, and which topic areas have gaps due to unavailable sources. Honesty about coverage is a deliverable, not a caveat. | |
| A downstream agent's context budget is exhausted by verbose prose from the agent before it | Reasoning narrative is being passed where facts were needed | Have the upstream agent return structured key facts, citations and relevance scores rather than its content and reasoning chains. Downstream agents need receipts, not narratives. |
19 practice items are tagged to this scenario — including the 3 worked items printed in the guide. The quiz explains every wrong option and names the task statement it tests.
You are building developer productivity tools using the Claude Agent SDK. The agent helps engineers explore unfamiliar codebases, understand legacy systems, generate boilerplate code, and automate repetitive tasks. It uses the built-in tools (Read, Write, Bash, Grep, Glob) and integrates with Model Context Protocol (MCP) servers.
Primary domains: Tool Design & MCP Integration · Claude Code Configuration & Workflows · Agentic Architecture & Orchestration
An engineering organisation using Claude to explore codebases, understand legacy systems, generate boilerplate and automate routine work. This scenario is where the built-in tools live, and where MCP configuration gets tested.
The vocabulary you need. The six built-in tools: Read opens a file, Write creates or overwrites one, Edit changes part of one by matching exact text, Bash runs shell commands, Grep searches inside files for content, Glob matches file paths by pattern. MCP — Model Context Protocol — is a standard for connecting Claude to external systems; an MCP server exposes tools (actions Claude can call) and resources (read-only data it can pull in).
| What the question describes | What is actually wrong | The intended answer, and why | Task |
|---|---|---|---|
| Find every caller of a function | Nothing — a tool-selection question | Grep, because you are searching file contents for a name. Then Read the hits to follow imports and trace the flow. The distinction the exam tests repeatedly: contents means Grep, filenames means Glob. | |
List every file matching **/*.test.tsx | The inverse question | Glob, because you are matching paths rather than looking inside files. | |
An Edit fails because its anchor text appears six times in the file | The edit cannot identify a unique target | Widen the surrounding text to make the match unique, or replace all occurrences deliberately. Only if neither works does reading the whole file and writing it back become the fallback — a last resort, since a targeted edit produces a clean diff and a whole-file write does not. | |
The agent keeps using built-in Grep instead of a far better semantic search tool you added. That tool's description reads "searches the codebase" | The better tool describes itself worse | Enrich the description with what it can do and what it returns — finds functions, classes, related symbols and usage; better for large codebases. Removing Grep to force the choice breaks the cases where Grep is genuinely right. | |
| The whole team needs the same external integration, each authenticating with their own token, and no secrets in the repository | Nothing — a configuration question | Project-scoped .mcp.json, committed, with the credential referenced as an environment variable rather than written in. That is exactly what variable expansion exists for, and committing a token is the mistake it prevents. | |
| One developer has a personal, experimental server | The inverse question | Their own home-directory configuration, so it stays private. Shared team tooling goes in the committed project file; personal and experimental servers do not. | |
| The agent spends four or five calls paginating a list just to discover what data exists before doing anything useful | Discovery is being done by trial rather than by catalogue | Expose a catalogue as an MCP resource — an issue summary, a schema, a documentation hierarchy — so the agent sees what exists and then makes one precise call. Tools perform actions; resources show what is available. | |
| The team needs a Jira integration | Nothing — a build-or-adopt question | Use the existing community MCP server. Reserve custom servers for genuinely team-specific workflows; rebuilding a standard integration spends weeks on solved problems. | |
| Verbose discovery is exhausting the context window in the middle of a multi-phase task | Exploration output is landing in the wrong place | Delegate the discovery to a read-only exploration agent that works in its own context and returns a summary. The main session keeps the plan; the noise stays elsewhere. | |
| Resuming yesterday's investigation, after three of the analysed files were refactored overnight | A resumed session still trusts the file contents it read yesterday | Resume the named session and tell it which files changed, so it re-analyses those specifically. Resuming silently means reasoning about code that no longer exists; re-exploring everything throws away a mostly valid analysis. | |
| You want to compare two refactoring strategies from one completed analysis, without either contaminating the other | Nothing — a session-operation question | Fork the session into two independent branches from the shared baseline. One session means one history and mutual contamination; two fresh sessions mean doing the analysis twice. | |
| A multi-agent pipeline crashes four hours in and everything is lost | No durable state existed between phases | Each agent exports structured state to a known location, and the coordinator loads a manifest on resume and injects it into the agents' prompts. Raw logs are not resumable state, and forking is for exploring alternatives rather than for durability. |
21 practice items are tagged to this scenario — no worked items in the guide for this scenario. The quiz explains every wrong option and names the task statement it tests.
You are integrating Claude Code into your Continuous Integration/Continuous Deployment (CI/CD) pipeline. The system runs automated code reviews, generates test cases, and provides feedback on pull requests. You need to design prompts that provide actionable feedback and minimize false positives.
Primary domains: Claude Code Configuration & Workflows · Prompt Engineering & Structured Output
Claude Code running unattended inside a build pipeline: reviewing every pull request, generating tests, posting feedback. Nobody is watching, so output must be machine-readable and false alarms are expensive.
The vocabulary you need. CI/CD is the automated system that builds and checks code on every change. A pull request is a proposed set of changes under review. Headless means running with no interactive terminal and nobody to answer a prompt. A false positive is a finding the reviewer reports that is not actually a problem — the central concern here, because a noisy reviewer gets ignored entirely, including when it is right.
| What the question describes | What is actually wrong | The intended answer, and why | Task |
|---|---|---|---|
| The pipeline job hangs indefinitely; logs show it waiting for interactive input | An interactive tool was launched where no human exists | The non-interactive print flag, -p. It runs the prompt to completion, prints to standard output, and exits with a status code the pipeline can branch on. Redirecting input from /dev/null is a shell workaround rather than the intended mechanism, and both --batch and a headless environment variable are invented distractors. | |
| Findings must be posted as comments on the exact changed lines, parsed by a script. They currently come back as prose | The output shape is unconstrained | Request JSON output and enforce its shape with a schema. Prose plus a regular-expression parser is a brittle machine you need not build; a prompt instruction to reply in JSON works most of the time, which is not the same as structurally. | |
| Every new commit re-posts the same twelve comments the author already fixed or declined | Each run has no knowledge of the last one | Include the prior findings in context and instruct it to report only new or still-unaddressed issues. Deduplicating in the posting script hides the noise instead of fixing the review, and reviewing only the newest commit loses cross-commit context. | |
| Generated tests duplicate coverage that already exists | The generator cannot see what is already tested | Put the existing test files in context. It is not a prompting failure; it is missing information. | |
| Generated tests are technically valid but low value — asserting that getters return what was set, inventing fixture helpers that already exist | The pipeline has no project context | Document testing standards, what makes a test valuable, and the available fixtures in the project instruction file, which loads in the pipeline as well as at a developer's desk. Moving the work to a cheaper API makes the waste cheaper rather than eliminating it. | |
| A comment-accuracy check produces 70% false positives — flagging any comment whose wording differs from the code, even when the behaviour matches | The criterion is far broader than the intent | Rewrite it as a testable rule: flag a comment only when the claimed behaviour contradicts the actual behaviour. "Be conservative" and "only report high-confidence findings" are not criteria and do not move precision. | |
| Because of that noise, developers now ignore every finding, including accurate security ones | One unreliable category has destroyed trust in the reliable ones | Temporarily disable the noisy category while you fix its prompt. Four checks people act on beat five where one is ignored — and a reviewer nobody reads has zero value regardless of its accuracy elsewhere. | |
| The same class of issue is labelled critical in one run and minor in the next | Severity is an adjective with nothing anchoring it | Define each level with a concrete code example: critical blocks the release, major needs fixing soon, minor is nice to fix. Fewer undefined buckets are still undefined, and averaging three runs spends money smoothing over an ambiguity you could have defined away. | |
| Findings are formatted differently on every run | The output structure was described rather than demonstrated | Show two to four examples with the exact shape: location, issue, severity, suggested fix. Examples pin format down where more description does not. | |
| A manager proposes moving both the blocking pre-merge check and the overnight technical-debt report onto the cheaper batch API | One of the two workflows has someone waiting on it | Batch the overnight report; keep the pre-merge check synchronous. The batch API is half price with no latency guarantee — ideal when nobody is waiting, unacceptable when a developer is blocked. "Batches are usually faster than the worst case" is not a basis for a blocking workflow. | |
| A 14-file review is detailed on some files, superficial on others, and flags a pattern in one file while approving identical code in another | One pass over too much material dilutes attention | Review each file individually for local issues, then run one separate pass for cross-file data flow. A larger context window does not improve attention quality; requiring developers to split their pull requests moves your cost onto them; and reporting only issues appearing in two of three runs filters out real bugs caught intermittently. | |
| The same session that generated code reviews it, and misses its own bugs | The reviewer retains the author's reasoning | Review from an independent instance with no generation context. A model that just made a decision is biased toward defending it, which is why "review your work carefully" and extended reasoning both underperform a fresh reviewer. | |
| You want to learn which code shapes are producing the dismissed findings | The dismissal data is unanalysable | Add a field to every finding naming the construct that triggered it. Then the dismissals aggregate into a pattern you can fix, instead of a volume of noise you can only count. |
17 practice items are tagged to this scenario — including the 3 worked items printed in the guide. The quiz explains every wrong option and names the task statement it tests.
You are building a structured data extraction system using Claude. The system extracts information from unstructured documents, validates the output using JavaScript Object Notation (JSON) schemas, and maintains high accuracy. It must handle edge cases gracefully and integrate with downstream systems.
Primary domains: Prompt Engineering & Structured Output · Context Management & Reliability
A pipeline that reads unstructured documents — invoices, contracts, forms — and produces validated structured data for downstream systems. It needs high accuracy, graceful handling of unusual documents, and output a database can accept.
The vocabulary you need. A JSON schema is a machine-readable declaration of the fields you expect and their types. A field marked required must be present; nullable means it may legitimately be empty. An enum restricts a field to a fixed list of allowed values. A syntax error means the output is not parseable; a semantic error means it parses perfectly and the values are wrong — and the distinction between those two carries several questions on its own.
| What the question describes | What is actually wrong | The intended answer, and why | Task |
|---|---|---|---|
| About 4% of responses fail to parse — trailing commas, unescaped quotes, an occasional sentence of preamble | The output shape is being requested rather than constrained | Define the schema as a tool's input and read the structured result, which eliminates this class of error rather than reducing it. Stripping markdown fences and repairing malformed output builds tolerance for a defect you can prevent. | |
| The model sometimes replies conversationally instead of returning data, and the document type varies so you have several possible schemas | Nothing forces a structured response | Require that some tool is called, letting the model pick which schema fits. Forcing one specific tool would be wrong here precisely because you do not know the document type in advance. | |
| A metadata extraction must always run before enrichment, and sometimes gets skipped | Step ordering is left to the model's discretion | Force that specific tool on the first turn, then handle the later steps in follow-up turns. Listing it first in the tools array is not a sequencing mechanism, and a prompt instruction is not enforcement. | |
| Some invoices have no purchase-order number, no delivery date and no tax ID, and the model invents plausible values | The schema declares those fields mandatory | Make them optional or nullable, so returning nothing is a legal answer. "Never guess" fights the schema and loses, because a required field is itself an instruction to produce a value. | |
| Real documents include types outside your fixed list, and some whose type genuinely cannot be determined | The enum has no exit | Add an "other" value paired with a free-text detail field, and an "unclear" value for genuine ambiguity. Otherwise the model jams unusual inputs into the nearest wrong category — worse than an honest "other", because it looks like a real classification. | |
| Validation passes, but on 6% of invoices the line items do not add up to the stated total | A schema validates shape, never meaning | Have the model extract a calculated total alongside the stated total so the discrepancy is detectable, plus a flag for internally inconsistent source data. No schema can compute a derived value, and running the extraction twice only detects instability — a consistently wrong sum passes both times. | |
| An extraction fails validation and you want a corrected version | Nothing — a retry-design question | Send back three things: the original document, the failed extraction, and the specific validation error. "Try again" produces the same mistake, because the model does not know what was wrong. Cap the retries at two or three, then flag for a human. | |
| Retries never succeed on one cluster of documents, where a required identifier exists only in a master agreement you never send | The information is not in the input | Stop retrying. Make the field nullable or supply the missing document. Format and structure errors are fixable by retry; absence is not, and more attempts only buy fabrication. | |
| Required fields come back empty on documents with unusual layouts — inline citations rather than a bibliography, narrative prose rather than tables | The prompt has only ever demonstrated one document shape | Add worked examples spanning the varied structures, so the model generalises across layouts rather than recognising one. Include at least one example where the honest answer is "not specified". | |
| 100 documents a day, a 30-hour turnaround promise, and cost pressure | Nothing — an API-selection and cadence question | Use the batch API, submitting every four hours. With a 24-hour worst case, a four-hour submission cycle keeps the total inside 30 hours with headroom. A nightly batch cannot meet a tight promise at all, because a document arriving after the run waits for the next one. | |
| Some documents in a batch failed | Nothing — a recovery question | Resubmit only the failures, identified by the ID you attached to each request, with the appropriate fix — splitting documents that exceeded the context limit, reformatting malformed inputs. Resubmitting all of them doubles the cost and the same subset fails again. | |
| Accuracy is 97% overall and you want to stop having humans check the output | An average is concealing at least one weak slice | Break the accuracy down by document type and by field before removing the safety net. 97% overall is arithmetically consistent with invoices at 99% and contracts at 62%, and it is the contracts that will hurt you. Raising a confidence threshold, or auto-approving and watching complaint volume, both act on an aggregate you have not yet examined. | |
| Reviewer capacity is limited and you must decide what they see | Nothing — a routing question | Score confidence per field, calibrate those scores against a hand-labelled validation set so the numbers mean what they claim, and route the low end plus anything with ambiguous or contradictory sources to humans. Raw self-reported confidence used without calibration is a number, not a measurement. | |
| You now auto-approve high-confidence extractions and want ongoing assurance | Nothing — a monitoring question | Randomly sample within the auto-approved tier and check those by hand. High confidence does not guarantee accuracy, and sampling the tier you stopped watching is how you catch drift and novel error patterns before customers do. |
16 practice items are tagged to this scenario — no worked items in the guide for this scenario. The quiz explains every wrong option and names the task statement it tests.