Support structured and manual JSON output_type modes in addition to tool calls

[DouweM]

• Fixes Structured outputs as an alternative to Tool Calling #582
• Fixes Support for returning response directly from tool #1189
• Replaces Add support for the use of response_format to force a particular json schema for the response #959

Right now, this only implements response_format=json_schema structured output for OpenAI chat completions in non-streaming mode.

• Wrap non-object output_type in a dict (as we do with output tools), because while OpenAI supports them, our schema generation doesn't
• Make ToolOutput and StructuredOutput not preferred but forced, erroring if model doesn't support it
• Rename StructuredOutput
• Address the many, many TODOs
• response_format=json_object for older OpenAI models?
• Manual JSON mode
• Streaming
• Other models
  • Anthropic
  • Gemini Support "responseMimeType" as part of generationConfig for Gemini #1288
• Notes on structured output strategies #1615
  • Callables Support for returning response directly from tool #1189 Support functions as output_type, as well as lists of functions and other types #1785 -- to be merged first
• All the tests
• Documentation

This test illustrates the behavior, including the fact that OpenAI supports tool calls alongside structured output (as noted in #582 (comment)):

async def test_openai_structured_output(allow_model_requests: None, openai_api_key: str):
    m = OpenAIModel('gpt-4o', provider=OpenAIProvider(api_key=openai_api_key))

    class CityLocation(BaseModel):
        city: str
        country: str

    agent = Agent(m, output_type=StructuredOutput(type_=CityLocation))

    @agent.tool_plain
    async def get_user_country() -> str:
        return 'Mexico'

    result = await agent.run('What is the largest city in the user country?')
    assert result.output == snapshot(CityLocation(city='Mexico City', country='Mexico'))

    assert result.all_messages() == snapshot(
        [
            ModelRequest(
                parts=[
                    UserPromptPart(
                        content='What is the largest city in the user country?',
                        timestamp=IsDatetime(),
                    )
                ]
            ),
            ModelResponse(
                parts=[ToolCallPart(tool_name='get_user_country', args='{}', tool_call_id=IsStr())],
                model_name='gpt-4o-2024-08-06',
                timestamp=IsDatetime(),
            ),
            ModelRequest(
                parts=[
                    ToolReturnPart(
                        tool_name='get_user_country',
                        content='Mexico',
                        tool_call_id=IsStr(),
                        timestamp=IsDatetime(),
                    )
                ]
            ),
            ModelResponse(
                parts=[StructuredOutputPart(content='{"city":"Mexico City","country":"Mexico"}')],
                model_name='gpt-4o-2024-08-06',
                timestamp=IsDatetime(),
            ),
        ]
    )

[github-actions]

Docs Preview

commit:	7974df0
Preview URL:	https://30f14b49-pydantic-ai-previews.pydantic.workers.dev

[dmontagu]

It seems like this code is assuming that you don't have both tool calls and structured outputs, but I would think that this is possible. Obviously you aren't going to have output tool calls, but you could have non-output tool calls right? My intuition is that we should rework it so that we process any tool calls and then handle the structured outputs.

If it's impossible to use tool calls while also using structured output then maybe this is fine but I don't see why that should be the case in principle. (But maybe it's the case in all existing APIs? Or maybe it isn't?)

[dmontagu]

unfortunate exception name now

[dmontagu]

Could we use ModelRetry directly here? Maybe that's hard/annoying if things are set up to convert those into ToolRetryError (because it was assumed to be handled in a tool). I don't remember the details..