Structured Output

The Core Problem:

LLMs generate free-form text by default. But your application needs structured data - JSON objects, typed fields, validated formats. Without structured output, you end up writing fragile regex parsers to extract data from LLM responses, and they break every time the LLM changes its response format.

Think of it like this: you order from Zomato and expect the order details in a structured format (item name, quantity, price, address). If Zomato sent you a paragraph of text describing your order in prose, your payment system would not know what to charge. Structured output solves this for AI systems.

Real-World Impact:

• Without Structured Output: Parse errors, broken pipelines, inconsistent responses, hours debugging regex patterns
• With Structured Output: Guaranteed JSON schema compliance, type-safe integration, zero parse errors, clean data pipelines

Three Approaches to Structured Output:

How JSON Mode Works:

JSON Mode is a parameter you set when calling the API. When enabled, the LLM constrains its token generation to only produce valid JSON. It will never output text that is not valid JSON. This happens at the model level, not through post-processing.

Important distinction: JSON Mode guarantees valid JSON syntax, but does NOT guarantee the JSON matches a specific schema. You might get valid JSON with unexpected keys or types.

JSON Mode Across Providers:

JSON Mode Limitations:

• No Schema Enforcement: You get valid JSON but the structure might differ from what you expect
• Extra Fields: LLM might add fields you did not ask for
• Wrong Types: A number field might come as a string
• Nested Issues: Deeply nested structures are less reliable

How Function Calling Works:

You define a function schema (name, description, parameters with JSON Schema types) and pass it to the API. The LLM generates a structured response that matches the schema exactly. The output is guaranteed to be valid against the schema - correct types, required fields present, enum values respected.

Think of it like a Google Form. The form defines the fields, types, and validation rules. The respondent can only submit data that matches the form structure. Function calling is like giving the LLM a Google Form to fill out.

Function Calling is NOT Just for Calling Functions:

Despite the name, function calling is used for two purposes:

• Tool Use: Agent needs to call an external API or function (the original purpose)
• Structured Data Extraction: You just want structured output without actually calling any function. You define a "fake" function schema that describes the output structure you want.

OpenAI Structured Outputs (strict mode):

OpenAI introduced strict: true in function calling, which guarantees 100% schema compliance. The model uses constrained decoding to only generate tokens that are valid according to the schema. This is the most reliable approach available.

• Supports all JSON Schema types: string, number, boolean, array, object
• Supports enums, required fields, nested objects
• First request is slower (schema compilation), subsequent requests are fast

Comparison Table:

Pattern 1: Data Extraction from Unstructured Text

Extract structured data from resumes, invoices, emails, or documents.

Schema: {
  "name": string,
  "email": string,
  "phone": string,
  "skills": string[],
  "experience_years": number,
  "education": { "degree": string, "institution": string }
}

Input: A resume in PDF text format
Output: Clean, structured JSON matching the schema exactly

Pattern 2: Classification with Confidence

Classify input into predefined categories with confidence scores.

Schema: {
  "category": enum["bug", "feature", "question", "docs"],
  "confidence": number (0-1),
  "reasoning": string,
  "priority": enum["low", "medium", "high", "critical"]
}

Pattern 3: Multi-Step Extraction Pipeline

For complex documents, chain multiple extraction steps:

• Step 1: Extract all entities (names, dates, amounts)
• Step 2: Classify relationships between entities
• Step 3: Validate against business rules
• Step 4: Output final structured document

Error Handling Best Practices:

• Schema Validation: Always validate the output against the schema even with strict mode (defense in depth)
• Retry Logic: If validation fails, retry with an error message appended to the prompt
• Default Values: Define sensible defaults for optional fields
• Partial Extraction: If the LLM cannot fill all fields, return what it can with null for unknowns

Gotcha 1: Schema Complexity Limits

Extremely complex schemas with deep nesting (5+ levels), many optional fields, or recursive types can confuse the LLM. Even in strict mode, complex schemas increase latency significantly because the model has to navigate more constraints during generation.

Fix: Keep schemas flat and simple. Split complex schemas into multiple extraction steps.

Gotcha 2: Hallucinated Values in Correct Format

Structured output guarantees the FORMAT is correct, not that the VALUES are correct. The LLM can still hallucinate a fake email address that is a perfectly formatted string. Schema validation catches format issues, not factual correctness.

Fix: Add semantic validation on top of schema validation. Verify extracted data against known sources when possible.

Gotcha 3: Token Cost of Schema

The function/tool definition itself consumes tokens from your context window. Complex schemas with many fields and descriptions can use 500-1000+ tokens just for the schema definition, leaving less room for actual content.

Fix: Keep field descriptions concise. Remove optional fields you do not need for a specific task.

Gotcha 4: Provider Lock-in

Each provider (OpenAI, Anthropic, Google) has slightly different APIs for structured output. Switching providers means rewriting your integration layer.

Fix: Use an abstraction library (Instructor, LiteLLM) that normalizes structured output across providers.

Q: What is the difference between JSON Mode and Function Calling?

Q: Does structured output prevent hallucination?

Q: How does constrained decoding work in strict mode?

Q: What are the limitations of structured output with complex schemas?

Q: Why is Function Calling used for data extraction even when no function is called?