Editor: Editorial Department【New Wisdom Introduction】Good news for programmers! All of OpenAI’s newly launched model APIs support structured output, with a JSON Schema matching rate of up to 100%, and the cost is reduced by half.
Are you still racking your brains to come up with a bunch of prompt words, and having a headache over the various output results after a series of operations?OpenAI has finally heard the voice of the masses and delivered the long-awaited first major feature to developers.OpenAI announced today the launch of a new feature, ChatGPT API now supports JSON structured output.JSON (JavaScript Object Notation) is the industry standard for file and data interchange formats because it is both easy for humans to read and easy for machines to parse.However, LLM often struggles with JSON, often producing hallucinations and either generating responses that only partially follow instructions, or generating a bunch of "magic" that can't be fully parsed at all.This requires developers to use multiple open source tools, try different prompts or repeat requests to generate the desired output results, which is time-consuming and labor-intensive.The structured output feature released today solves this thorny problem and ensures that the output generated by the model matches the schema specified in the JSON.The structured output function has always been the number one feature most requested by developers. Ultraman also stated in a tweet that this version was released in response to the requests of the majority of users.The new features released by OpenAI really hit the hearts of many developers, and they all agreed that "This is a big deal."Many people left comments to express their admiration, calling it "Excellent!".Some are happy while others are sad. This update of OpenAI has made people worry that it will swallow up start-ups.However, for more ordinary users, the question they are more concerned about is when GPT-5 will be released. As for JSON Schema, "What is that?"After all, without any news about GPT-5, OpenAI's DevDay this fall may be much quieter than last year.Easily ensure schema consistency
With structured output, you only need to define a JSON Schema, and AI will no longer be "willful" and will output data according to the instructions.Moreover, the new function not only makes AI more obedient, but also greatly improves the reliability of the output content.In a trace evaluation on a complex JSON schema, the new model gpt-4o-2024-08-06 with structured outputs achieved a perfect score of 100%. In comparison, gpt-4-0613 scored less than 40%.In fact, the JSON Schema feature was introduced by OpenAI at last year’s DevDay.Now, OpenAI has expanded this functionality in its API, ensuring that the outputs generated by the model exactly match the JSON Schema provided by the developer.Generating structured data from unstructured input is one of the core use cases of artificial intelligence in today’s applications.Developers use the OpenAI API to build powerful assistants capable of ingesting data and answering questions through function calls, extracting structured data for data input, and building multi-step agentic workflows that allow LLMs to take actions.OpenAI took a two-pronged approach to improve the match of model outputs to JSON Schema.Although model performance has improved significantly, reaching 93% accuracy on benchmark tests, inherent uncertainties remain.To ensure the stability of the applications built by developers, OpenAI provides a more accurate method to constrain the output of the model, thereby achieving 100% reliability.Constraint decoding
OpenAI uses a technique called constrained sampling or constrained decoding. By default, the model is completely unconstrained when generating outputs and may choose any token from the vocabulary as the next output.This flexibility can lead to errors, such as arbitrarily inserting invalid characters when generating valid JSON.To avoid such errors, OpenAI uses a dynamic constrained decoding method to ensure that the generated output tokens always conform to the provided schema.To achieve this, OpenAI converts the provided JSON Schema into a context-free grammar (CFG).For each JSON Schema, OpenAI computes a grammar that represents the schema and efficiently accesses the preprocessed components during sampling.This approach not only makes the generated output more accurate, but also reduces unnecessary latency. The first request for a new schema may have additional processing time, but subsequent requests are quickly responded to through the caching mechanism.Options
In addition to the CFG method, other methods usually use finite state machines (FSM) or regular expressions for constraint decoding.However, these methods have limited capabilities in dynamically updating valid tokens. Especially for complex nested or recursive data structures, FSM is usually difficult to handle.OpenAI's CFG approach excels at expressing complex schemas. For example, JSON schema that supports recursive patterns is implemented on the OpenAI API, but cannot be expressed using the FSM approach.Cut input costs in half
Structured output is available for all models that support function calls, including the latest GPT-4o and GPT-4o-mini models, as well as fine-tuned models.This feature is available on the Chat Completions API, Assistants API, and Batch API, and is compatible with visual input.Compared with the gpt-4o-2024-05-13 version, the gpt-4o-2024-08-06 version is also more cost-effective. Developers can save 50% of the cost on the input side ($2.50/1M oken) and 33% of the cost on the output side ($10.00/1M token).How to use structured output
There are two ways to import structured output into the API:Function Call
Structured output through tools can be achieved by setting strict: true in the function definition.This feature is available for all models that support the tool, including all models gpt-4-0613 and gpt-3.5-turbo-0613 and later.When structured output is enabled, the model output will match the provided tool definition.{
"table_name": "orders",
"columns": ["id", "status", "expected_delivery_date", "delivered_at"],
"conditions": [
{
"column": "status",
"operator": "=",
"value": "fulfilled"
},
{
"column": "ordered_at",
"operator": ">=",
"value": "2023-05-01"
},
{
"column": "ordered_at",
"operator": "<",
"value": "2023-06-01"
},
{
"column": "delivered_at",
"operator": ">",
"value": {
"column_name": "expected_delivery_date"
}
}
],
"order_by": "asc"
}
New options for the response_format parameter
Developers can now choose whether they want a schema-specific output via a new json_schema option in response_format.This is useful when the model does not call a tool, but instead responds to the user in a structured manner.This feature is available for the latest GPT-4o models: gpt-4o-2024-08-06 and gpt-4o-mini-2024-07-18, released today.When setting response_format to strict: true , the model output will match the provided schema.{
"steps": [
{
"explanation": "Subtract 31 from both sides to isolate the term with x.",
"output": "8x + 31 - 31 = 2 - 31"
},
{
"explanation": "This simplifies to 8x = -29.",
"output": "8x = -29"
},
{
"explanation": "Divide both sides by 8 to solve for x.",
"output": "x = -29 / 8"
}
],
"final_answer": "x = -29 / 8"
}
Developers can generate answers step by step using structured outputs to guide towards the expected output.According to OpenAI, developers don’t need to validate or retry malformed responses, and the feature allows for simpler prompts.Native SDK support
OpenAI said their Python and Node SDKs have been updated to natively support structured output.Providing a schema or response format to a tool is as simple as providing a Pydantic or Zod object, and OpenAI’s SDK will convert data types to supported JSON schemas, automatically deserialize JSON responses into typed data structures, and parse rejections.Also, native structured output support is available for response_format.Other Use Cases
Developers frequently use OpenAI’s models to generate structured data for a variety of use cases.Some other examples include:-Dynamically generate user interface based on user intentDevelopers can use the structured output to create code or UI generation applications.Using the same response_format, you can generate different UIs based on user input.For example, to create a "gardener's login interface":It is generated with the following code:{
"type": "div",
"label": "",
"children": [
{
"type": "header",
"label": "",
"children": [
{
"type": "div",
"label": "Green Thumb Gardening",
"children": [],
"attributes": [{ "name": "className", "value": "site-title" }]
},
{
"type": "div",
"label": "Bringing Life to Your Garden",
"children": [],
"attributes": [{ "name": "className", "value": "site-tagline" }]
}
],
"attributes": [{ "name": "className", "value": "header" }]
},
{
"type": "section",
"label": "",
"children": [
{
"type": "div",
"label": "",
"children": [
{
"type": "div",
"label": "About Us",
"children": [
{
"type": "div",
"label": "At Green Thumb Gardening, we specialize in transforming your outdoor spaces into beautiful, thriving gardens. Our team has decades of experience in horticulture and landscape design.",
"children": [],
"attributes": [
{ "name": "className", "value": "about-description" }
]
}
],
"attributes": [{ "name": "className", "value": "about-section" }]
}
],
"attributes": [{ "name": "className", "value": "content" }]
}
],
"attributes": [{ "name": "className", "value": "about-container" }]
},
{
"type": "section",
"label": "",
"children": [
{
"type": "div",
"label": "",
"children": [
{
"type": "div",
"label": "Our Services",
"children": [
{
"type": "div",
"label": "Garden Design",
"children": [],
"attributes": [
{ "name": "className", "value": "service-item" }
]
},
{
"type": "div",
"label": "Plant Care & Maintenance",
"children": [],
"attributes": [
{ "name": "className", "value": "service-item" }
]
},
{
"type": "div",
"label": "Seasonal Cleanup",
"children": [],
"attributes": [
{ "name": "className", "value": "service-item" }
]
},
{
"type": "div",
"label": "Custom Landscaping",
"children": [],
"attributes": [
{ "name": "className", "value": "service-item" }
]
}
],
"attributes": [{ "name": "className", "value": "services-list" }]
}
],
"attributes": [{ "name": "className", "value": "content" }]
}
],
"attributes": [{ "name": "className", "value": "services-container" }]
}
],
"attributes": [{ "name": "className", "value": "landing-page" }]
}
- Separate the final answer from the supporting reasoning or additional commentsProviding the model with a separate thought chain field can improve the final quality of the response.{
"reasoning_steps": [
"First step is to compare the numbers 9.11 and 9.9.",
"Both numbers have the same whole number part, which is 9.",
"To compare the decimal parts, convert them to the same number of decimal places.",
"9.11 has two decimal places: it is 9.11.",
"9.9 has one decimal place: it can be rewritten as 9.90.",
"Now, compare 9.11 and 9.90 by looking at the decimal parts.",
"Compare 11 with 90.",
"90 is greater than 11, so 9.90 is greater than 9.11."
],
"answer": "9.9 is bigger than 9.11."
}
- Extract structured data from unstructured dataFor example, instruct the model to extract content such as to-do items, deadlines, and assignments from meeting notes.{
"action_items": [
{
"description": "Collaborate on optimizing the path planning algorithm",
"due_date": "2024-06-30",
"owner": "Jason Li"
},
{
"description": "Reach out to industry partners for additional datasets",
"due_date": "2024-06-25",
"owner": "Aisha Patel"
},
{
"description": "Explore alternative LIDAR sensor configurations and report findings",
"due_date": "2024-06-27",
"owner": "Kevin Nguyen"
},
{
"description": "Schedule extended stress tests for the integrated navigation system",
"due_date": "2024-06-28",
"owner": "Emily Chen"
},
{
"description": "Retest the system after bug fixes and update the team",
"due_date": "2024-07-01",
"owner": "David Park"
}
]
}
Safety is a top priority for OpenAI — the new structured output functions will adhere to OpenAI’s existing safety policies and will still allow models to reject unsafe requests.To make development easier, there is a new refusal string value on API responses, which allows developers to programmatically detect if a model is generating output that rejects instead of matching the schema.When the response does not contain a rejection and the model's response does not terminate prematurely (as indicated by finish_reason), the model's response will reliably produce valid JSON that matches the provided schema.
