Validate LLM Output with Structured Contracts

Large language models are useful when software needs to turn ambiguous text into a structured decision, extraction, or plan. The dangerous shortcut is to treat a model response as if it were already trusted application data.

Even when a provider can constrain output to JSON or a schema, the result can still be semantically wrong: a date can be impossible, an identifier can refer to a nonexistent record, or a supposedly positive amount can be negative. Reliable integrations therefore need a contract boundary between model output and the rest of the system.

Treat model output as untrusted input

An LLM response has more in common with an HTTP request body than with a return value from a normal function. It crosses a boundary where assumptions can fail.

A useful pipeline is:

1. request structured output;
2. parse the response;
3. validate its structural contract;
4. validate domain rules;
5. resolve references against authoritative data;
6. either accept, repair, retry, or reject the result.

These stages should remain separate because they answer different questions.

Syntax is not semantics

Consider an extraction result:

{
  "invoice_number": "INV-1042",
  "currency": "USD",
  "total": -125.50
}

This is valid JSON. It may also satisfy a schema that says total is a number. It is still invalid if the application requires invoice totals to be non-negative.

Schema validation catches shape errors. Domain validation catches business errors. Neither replaces the other.

Design the smallest useful contract

Model-facing schemas work best when they describe only what the next application step actually needs.

Prefer:

status: one of "approved", "needs_review", "rejected"
reason: short string
evidence_ids: list of known evidence identifiers

over a large object containing speculative fields that downstream code never uses.

A smaller contract has several advantages:

• fewer fields for the model to misunderstand;
• fewer optional combinations to test;
• simpler validation and observability;
• easier compatibility when the prompt evolves.

Use enums for finite choices, explicit nullability for missing values, and bounded collections where practical. Avoid encoding important state in free-form prose when the application will immediately parse that prose again.

Validate in layers

Layer 1: parse and structural validation

The first layer should reject malformed data and incorrect types. If your model provider supports schema-constrained output, use it, but keep local validation as the application boundary.

Provider guarantees can change by model or feature, and local validation also protects stored responses that are replayed later.

Layer 2: domain invariants

Domain validation checks rules such as:

• end_date must not precede start_date;
• percentages must be between 0 and 100;
• a requested quantity must not exceed a known limit;
• mutually exclusive fields must not be set together.

These rules belong in normal application code, where they are deterministic and testable.

Layer 3: reference validation

Models should not be allowed to invent authoritative identifiers.

If a result contains customer_id, document_id, or product_code, verify that the identifier exists and that the caller is authorized to use it. When possible, give the model a constrained list of opaque identifiers rather than asking it to reproduce names and later map them back.

Choose a failure policy deliberately

Not every invalid response deserves the same treatment.

A retry is appropriate when failure is likely transient or formatting-related. A repair pass can be useful when the invalid value can be described precisely without adding ambiguity. A hard rejection is safer when the action has significant consequences or when repeated attempts could create inconsistent behavior.

Keep retries bounded. A model that misunderstands the task often repeats the same mistake, so an unbounded retry loop increases cost without increasing correctness.

A practical policy might allow one corrective retry that includes machine-readable validation errors, then route the request to a deterministic fallback or human review.

Preserve the original response

When a validation failure occurs, store enough diagnostic information to understand the failure without leaking sensitive prompt data into logs.

Useful telemetry includes:

• model and prompt version;
• validation stage that failed;
• validation error code;
• retry count;
• latency and token usage;
• final outcome.

Avoid logging full prompts or responses by default when they can contain personal, confidential, or regulated data. Structured error codes are usually more useful for dashboards anyway.

Keep prompts and contracts versioned

A schema change is an API change. If a field is renamed or its meaning changes, old stored outputs may no longer deserialize correctly.

Version the prompt and output contract together. For workflows that persist model output, record the contract version next to the data. This makes migrations and replay behavior explicit instead of relying on the current prompt to interpret historical records.

Backward-compatible changes, such as adding an optional field, are easier to deploy. Required-field changes should be coordinated with consumers just like changes to any other service contract.

Test the boundary without calling a model

Most validation tests do not need an LLM request. Build a corpus of representative payloads and test the deterministic boundary directly.

Include cases such as:

• missing required fields;
• unknown enum values;
• extra fields when they should be rejected;
• invalid dates and numeric ranges;
• nonexistent references;
• very long strings;
• valid but adversarial text inside string fields.

Then keep a smaller integration suite for real model calls. Separating deterministic contract tests from probabilistic model tests makes failures easier to diagnose and keeps the test suite cheaper.

Common pitfalls

Trusting schema-constrained generation completely

Constrained generation improves structural reliability, but it cannot prove that a value is factually or semantically correct.

Asking the model to enforce authorization

Authorization must be checked against trusted application state. A model can help classify intent, but it should not decide whether a caller owns a resource.

Hiding validation inside prompt instructions

“Always return a positive total” is useful guidance, not enforcement. The application still needs to check the value.

Retrying every failure

Some failures are systematic. Record the reason, cap retries, and provide a deterministic fallback path.

A reliable mental model

The most important design choice is simple: the model proposes data; the application accepts or rejects it.

Structured generation reduces parsing failures, but production reliability comes from ordinary software engineering around the model: narrow contracts, deterministic validation, authoritative lookups, bounded failure handling, versioning, and observability. Once that boundary is explicit, an LLM becomes easier to integrate without giving probabilistic output more trust than it deserves.