Scribd
Upload
21 views3 pages

Structuring LLM Output as JSON

Uploaded by

Deepankar Gupta
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
21 views3 pages

Structuring LLM Output as JSON

Uploaded by

Deepankar Gupta
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd

The Core Problem: Making LLMs Talk to Machines

The central objective of this lecture is to solve a fundamental problem: Large Language
Models (LLMs) naturally produce unstructured text, which is great for humans but difficult
for other computer systems (like APIs or databases) to understand. We need to make the
LLM's output structured—specifically, in a well-defined data format like JSON—so that it can
be programmatically parsed and used by other machines. This capability is the key to building
powerful, integrated AI applications.

What is Structured Output?


Structured output is the practice of compelling an LLM to return its response in a predictable,
pre-defined format.
●​ Unstructured Example (The Default):
○​ Question: "What is the capital of India?"
○​ LLM Output: "New Delhi is the capital of India." (A simple string)
●​ Structured Example (The Goal):
○​ Request: "Create an itinerary for a day in Paris."
○​ LLM Output (JSON):​
JSON​
[​
{"time": "9:00 AM", "activity": "Visit the Louvre Museum"},​
{"time": "1:00 PM", "activity": "Lunch near the Eiffel Tower"},​
{"time": "3:00 PM", "activity": "Stroll along the Seine River"}​
]
○​ This JSON is easy for another program to read and process.

Why Do We Need It? Key Use Cases


1.​ Data Extraction:
○​ Analogy: Imagine building a job portal like [Link]. When a user uploads a
resume (unstructured text), you need to extract specific fields—like name, last
company, and marks—to populate a database.
○​ Solution: An LLM can read the resume and output the required information in a
clean JSON format, ready for database insertion.
2.​ Building APIs:
○​ Example: Analyzing Amazon product reviews. A review is often a long, unstructured
block of text.
○​ Solution: An LLM can process the review and extract structured insights, such as the
topic (battery, display), pros, cons, and overall sentiment. This structured data can
then be served via an API built with Flask or FastAPI.
3.​ Building Agents:
○​ Concept: Agents are like "chatbots on steroids" because they can use tools (like a
calculator or an API) to perform actions.
○​ The Problem: Tools require structured input. You can't just send the text "find the
square root of two" to a calculator function.
○​ Solution: The LLM first extracts the necessary information ("square root", "2") into a
structured format, which can then be passed as arguments to the tool.

How to Generate Structured Output:


with_structured_output
For models that natively support it (like OpenAI's GPT series), LangChain provides a simple
and powerful function: with_structured_output.

You define a schema (the desired data structure) and pass it to this function. The LLM will
then ensure its output conforms to your schema.

Defining Your Schema: 3 Methods


You can define the structure of your desired output in three ways.

1. TypedDict
●​ What it is: A standard Python feature (from typing import TypedDict) that lets you define
the keys and value types for a dictionary. It's primarily for type hinting to help your code
editor.
●​ Crucial Point: TypedDict does not perform runtime validation. If you create a Person
dictionary with age: int and accidentally assign a string, Python won't complain.
●​ Enhancing with Annotations:
○​ Annotated: You can add descriptions to fields to give the LLM more context (e.g.,
summary: Annotated[str, "A brief summary of the review"]).
○​ Optional: Mark fields that may not always be present.
○​ Literal: Restrict a field's value to a specific set of strings (e.g., sentiment:
Literal["Positive", "Negative"]).
2. Pydantic (The Go-To Method)
●​ What it is: A robust data validation library for Python. It's far more powerful than
TypedDict.
●​ Core Features:
○​ Runtime Validation: It enforces the data types you define. If you provide a string
where an integer is expected, Pydantic will raise an error.
○​ Type Coercion: It's smart enough to convert types when it makes sense (e.g., it will
convert the string "32" to the integer 32).
○​ Built-in Validations: Has special types for common formats like emails (EmailStr).
○​ The Field Function: Allows you to add detailed validation rules and metadata:
■​ Constraints (e.g., CGPA: Field(gt=0, lt=10) for a value greater than 0 and less
than 10).
■​ Descriptions (just like Annotated).
■​ Default values.
●​ Why it's preferred: Pydantic provides the validation and robustness needed for
real-world applications.
3. JSON Schema
●​ What it is: A language-agnostic, universal standard for defining the structure of a JSON
object.
●​ When to use it: This is the best choice for multi-language projects. If your Python
backend needs to share a data schema with a JavaScript frontend, you define it once
using JSON Schema, and both systems can understand it.
●​ Structure: You define the type (e.g., "object"), properties (the fields and their types), and
which fields are required.

How Does It Work Behind the Scenes?


When you use with_structured_output, LangChain generates a system prompt that instructs
the LLM to act as a data extraction assistant and to return its response only in the specified
JSON format. This prompt is sent to the model along with your user request.

method Parameter: function_calling vs. json_mode


●​ function_calling: The default for OpenAI models. Use this when the structured output is
meant to call a tool or function.
●​ json_mode: Use this for other models like Claude or Gemini that support returning JSON.

Common questions

Powered by AI

Structured data provides significant advantages when building APIs for analyzing product reviews by converting unstructured text into structured insights. Instead of dealing with long blocks of text that can be challenging to interpret programmatically, structured data allows for the extraction of key elements such as topics (e.g., battery, display), pros, cons, and overall sentiment. This structured representation can then be used to deliver insightful analytics via an API, making it easier to perform operations like aggregation, search, and filtering .

The with_structured_output function assists in generating structured responses from LLMs by defining a desired data structure or schema, which the LLM must adhere to in its output. This function uses a system prompt to instruct the LLM to return responses in a specified format, like JSON. By doing so, it ensures that the LLM's output is structured and suitable for further processing by machines, making it powerful for AI applications that interact with various technologies .

JSON Schema is preferred over Python-specific validation libraries such as TypedDict or Pydantic in scenarios where data structures need to be shared between systems using different programming languages. JSON Schema is a language-agnostic, universal standard for defining the structure of a JSON object, which makes it ideal for multi-language projects. For instance, when a Python backend shares data with a JavaScript frontend, defining the schema once using JSON Schema allows both systems to understand and utilize the data format consistently .

LangChain leverages structured output to enhance AI application functionality by providing methods such as with_structured_output that enforce well-defined data schemas on LLM outputs. This assures that responses conform to structured formats like JSON, which are essential for seamlessly interacting and integrating with other software systems. Structured outputs enable various applications, such as automating data extraction, building APIs, and developing intelligent agents, all of which rely on consistency and compatibility with machine-readable formats .

Pydantic enhances data schema definition compared to TypedDict by providing runtime validation, which ensures that the data types you define are enforced. Unlike TypedDict, which only provides type hinting and does not perform runtime checks, Pydantic raises an error if you provide incorrect data types. Additionally, Pydantic supports type coercion, converting compatible data types (like converting a string to an integer) when it makes sense. It also includes built-in validations for common data formats, and allows detailed validation rules and metadata through the Field Function. These features make Pydantic more robust and suitable for real-world applications .

The method function_calling solves the problem of integrating LLMs with computational tools by ensuring that the LLM's output is structured specifically to call functions or tools directly. It allows for generating arguments in a predictable format, such as JSON, which computational tools require to perform specific operations correctly. Without this structured interaction, tools would be unable to process the LLM's output, as unstructured text cannot serve as valid input for computational functions .

Annotations in TypedDict enhance the context given to LLMs for structured data generation by allowing additional descriptive information to be attached to fields. For instance, using Annotated, you can provide explanations like 'A brief summary of the review' for specific fields. This added context clarifies the intended content and format of each field, helping LLMs generate more accurate and semantically meaningful structured outputs aligned with user expectations .

Structured output is needed from Large Language Models (LLMs) because it allows the LLMs' output to be programmatically parsed and utilized by other computer systems, such as APIs or databases. While LLMs naturally produce unstructured text suitable for human understanding, converting this output into structured formats like JSON makes it easy for programs to read and process the data. This capability is essential for applications like data extraction from unstructured text, building APIs that provide structured insights, and enabling agents to perform actions with tools that require structured input .

Data extraction from resumes benefits from structured output formats by enabling the extraction of crucial information fields, such as name, last company, and marks, into clean, structured formats like JSON. This structured format makes it easy to automatically populate databases, simplifying tasks that involve collecting and analyzing potential employee data. Structured outputs improve data consistency and reliability, allowing systems to efficiently process and use extracted data .

Using TypedDict is not sufficient for runtime validation of data structures in Python because it primarily serves as a type hinting feature that assists with code readability and editor support, rather than enforcing data integrity. TypedDict does not check the types at runtime; hence, if you assign a value of an incorrect type (e.g., a string where an integer is expected), Python will not raise an error. For applications requiring strict type validation at runtime, more robust solutions like Pydantic should be used .

You might also like