Build Log: Teaching an Agent to Show Its Receipts

The constraint that shaped everything

The DEV Sanity Challenge's Path 1 asks for an agent whose answers come from structured content, with an explicit bar: keyword search is not enough. That bar ruled out the obvious architecture on day one. A retrieval bot that embeds doc pages and quotes whichever chunk scores highest cannot answer "at least 128K context, under $2/M input, still available after date X" - the answer requires conjoining facts, not finding a passage. And it definitely cannot answer "which primary sources disagree?" because disagreement is a relationship between claims, not a property of any single chunk.

So the desk inverts the usual pipeline. Instead of: documents → chunks → embeddings → hopefully-relevant text, it is: primary sources → atomic claims → structured filters → an answer assembled from exactly the claims that satisfy the question, each with its quote, source URL, and observation date. The model's job is to translate a question into constraints and to narrate the result. The truth work is done by the schema.

The claim atom

Everything lives in one Sanity document type, evidenceClaim. Here is the schema, condensed from sanity/schemaTypes/evidenceClaim.ts:

evidenceClaim { provider: string (required) subject: string (required) // e.g. "Claude Sonnet 4.5" predicate: string (required) // e.g. "context_window" value: string (required) // as written at the source normalizedValue: number // comparable form unit: string // "tokens", "usd_per_mtok", ... effectiveFrom: datetime (required) effectiveUntil: datetime // null = still in force observedAt: datetime (required) // when WE read the source source: { title, url, publisher, publishedAt, sourceType: official_docs | model_card | system_card | pricing | deprecation | benchmark_methodology } quote: text (required) confidence: high | medium | low }

Every field earned its place. A tour of the non-obvious ones:

value vs normalizedValue + unit

Sources write "1M", "1,048,576", "200k", and "$0.30 / 1M tokens". String-matching those against a constraint like "at least 128K" is hopeless. So every claim carries the value as written (for the citation) and a normalized number plus unit (for the comparison). "1,048,576 tokens" and "1M tokens" both normalize to 1048576-ish numbers in unit "tokens" - and the display still shows the provider's own wording. The quote is required, not optional: a claim you cannot quote from the source does not enter the base. That single required field is the difference between an evidence system and a notes app.

effectiveFrom / effectiveUntil: time as data

Facts about models have lifespans. Claude Sonnet 4.5 had a 1M context window (as a beta) from its launch until April 30, 2026, and a 200K window after. Neither claim is wrong; they have non-overlapping effective intervals. Modeling validity as an interval - rather than overwriting a "current value" field - is the decision that makes both history questions and contradiction detection possible. It also mirrors how the best primary sources behave: Anthropic's release notes preserve the beta announcement, the deprecation, and the retirement as three dated records. My schema just makes that structure queryable.

observedAt vs publishedAt

Two timestamps, two different honesty jobs. publishedAt is when the source says it spoke. observedAt is when the desk actually read the page. They diverge constantly - a pricing page "updated in June" is re-read today - and the freshness math runs on observedAt, because "we saw this with our own eyes on date X" is a much stronger statement than "the page claims a date." The README says it flatly: observed-at is retrieval time, not proof the provider has not changed the page since.

The interval logic: twenty lines that do the heavy lifting

The whole "which claims are in force?" question is one function in src/lib/analysis.ts:

export function activeOn(c: Claim, at = new Date()): boolean { const start = new Date(c.effectiveFrom) const end = c.effectiveUntil ? new Date(c.effectiveUntil) : null return start <= at && (!end || end > at) }

Three lines, and they carry the entire epistemology. A claim is active at time at if its interval covers that time. Half-open intervals (end > at) so a retirement and its successor can share a boundary without both being true. Default now so the desk always answers in the present unless asked for a time-travel view.

The contradiction detector builds directly on it:

export function findContradictions(claims: Claim[], at = new Date()): Contradiction[] { // group by provider | subject | predicate // keep only claims ACTIVE at `at` // count distinct normalized values among the active claims // > 1 distinct value => contradiction card, newest observation first }

The crucial design choice is what is not there: expired claims never enter the comparison. When the desk is asked about Sonnet 4.5's context window, the retired 1M beta claim and the current 200K claim sit in the same provider|subject|predicate group with different values - but they are not both active, so there is no contradiction. The history view says so explicitly, in words I wrote into the summary template because I wanted the behavior to be legible to a skeptic: "Expired claims are kept as history, not reported as current conflicts." Today's live demo of the history mode returns Sonnet 4.5 with 2 current claims, 1 historical claim, and 0 active conflicts - the retirement record preserved, the false alarm not raised.

Why this is the whole ballgame. A naive "find disagreeing sources" system flags every spec change in history as a scandal. A naive "keep only the latest" system loses the audit trail. Intervals give you both: a present that is conflict-checked and a past that is intact.

Answering: constraints in, citations out

The constraint path is equally mechanical. A question like "128K+ context, under $2/M input" becomes a list of typed constraints - {field: "context_window", operator: "gte", value: 128000} and {field: "input_price", operator: "lte", value: 2}. The matcher is deliberately boring:

satisfies(claim, constraint): lte / gte -> numeric compare on normalizedValue eq -> case-insensitive string equality default -> substring match

Then buildAnswer: filter to active claims, group by provider+subject, and - the strict part - a model is eligible only if the group contains satisfying evidence for every stated constraint. A model with a documented context window but no documented price does not pass "under $2/M" on vibes; it fails, and the answer says how many claims support each survivor. The summary sentence is generated from the sets, not from the model's confidence: "1 model satisfies every stated constraint, supported by 4 primary-source claims." Every returned claim carries its quote and URL, so the answer screen is a receipt, not an assertion.

Today's live run of that exact question returns Claude Haiku 4.5 (200K context, $1/M input, both cited to official Anthropic pages) and, since the KB grew to 10 claims, Gemini 2.5 Flash (1,048,576 context, $0.30/M) - two real candidates, four supporting claims, zero filler.

Freshness is a number, not a vibe

Every answer also computes a freshness block: the newest and oldest observedAt among the supporting claims, and a stale count - any claim not observed within 30 days. The point is not that 30 days is magic; it is that the desk must show its decay. An answer built on 90-day-old observations is presented differently than one built on claims read this morning, and the user never has to take "current" on faith.

The query plan is part of the answer

Each response includes the actual plan it executed - "Inspect schema → Filter active claims → Group by provider and model → Require evidence for each explicit constraint → Compare active values → Return citations." This sounds cosmetic. It is not. An agent that can show its plan can be audited step by step; an agent that only shows conclusions asks for trust it has not earned. I hold myself to the same standard on this portfolio, incidentally - every audit post ends with a dated source list for the same reason.

The Sanity Context layer

Sanity's hosted Context MCP endpoint is what turns the claim store into agent tools. The desk uses three of them, and the README names exactly which and why:

initial_context - inspect the live schema and document counts before querying. An agent that queries a schema it has not inspected is guessing at field names.
groq_query - the workhorse: structured filters, effective-date conditions, and semantic ranking combined in one GROQ query over the claim set.
schema_explorer - reserved for schema debugging in the demo checklist.

The endpoint URL and read token stay server-side in environment variables on Netlify; the token never enters the repo (.env.example documents the shape, the README drills "never commit the Sanity token," and the deployed functions read it at runtime). The app itself is a small Next.js service: /api/ask translates the question, queries claims through MCP, runs the analysis layer, and renders the answer with its evidence cards. Quality gates are the boring, necessary kind: npm test, typecheck, lint, build, plus a unit-tested analysis module - the interval and contradiction logic is too load-bearing to leave untested.

Original Wraith doodle: the desk's pipeline - claims carry their own dates, and the gate only lets the present through.

What the desk refuses to do

The README carries an "honest limits" section, and I want to defend keeping it prominent, because limits are a feature of an evidence system, not an apology:

A recommendation is only as complete as the curated primary-source set. Ten strong claims beat a thousand scraped pages, and the desk says "no model has evidence satisfying every constraint" rather than inventing a candidate.
Observed-at is retrieval time. The desk never claims a provider has not changed a page since it looked.
Conflicts are deterministic at the atomic-claim level. Semantic disputes - "is this benchmark methodology comparable to that one" - still need human interpretation, and the schema has a benchmark_methodology source type precisely so that question can be modeled instead of elided.
No benchmark score is treated as comparable unless its methodology is represented. A number without its method is marketing.

What I would build next

Ordered by expected value, not by flash:

Source watchers. A scheduled re-observation loop for every source URL, bumping observedAt when the page re-verifies and opening a review task when the quote no longer appears. That turns freshness from a report into an alarm.
Time-travel answers. The interval schema already supports "answer as of March 1, 2026" - the UI just does not expose it yet.
Cross-provider constraint bundles. Saved constraint sets ("frontier coding agent under budget X") re-evaluated as claims change, so a price cut or retirement proactively updates the answer.
Claim provenance graphs. When a claim's source is itself a summary of another document, model the chain so the desk can prefer primary over secondary automatically.

The announcement post for this project is here, the demo is live, and every file discussed above is readable in the public repo. If you find a claim in the KB that lacks its quote, that is a bug - tell me.

Sources and reading trail

Model Truth Desk repository - all code referenced above.
evidenceClaim schema - the claim atom, field by field.
src/lib/analysis.ts - activeOn, findContradictions, satisfies, buildAnswer, buildHistoryAnswer.
Live demo - constraint and history modes against the real KB.
Sanity Studio - the KB behind the demo (project lj9x9set, dataset production).
Anthropic release notes - the Sonnet 4.5 1M-beta retirement trail used as the history-mode case.

Build note: development began 19 September 2026 for the DEV Sanity Challenge, Path 1. Older projects (Odyssius, Sprynn, Wraith) were not reused as application code. Code excerpts above are condensed for layout; the repo is the source of truth.