Fixes #2513: Add support for Free-Form Function Calling and Context Free Grammar constraints over tools #2572

matthewfranglen · 2025-08-15T10:24:21Z

This is the PR to implement free-form function calling and context free constraints over tools. This implements:

Free Form Function calling for Tool, @agent.tool, @agent.tool_plain
Free Form Function calling for output tools
Context Free constraints over the free form functions
Documentation
Tests

I'm hoping to get a working implementation done today, after which we can have the rounds of feedback and improvement required before this can be merged. I will try to keep in mind the feedback that I got on the last PR.

Since there is a specific company voice that you wish to maintain, when suggesting changes to documentation et al please provide the exact wording that you desire.

the response from the model is not correctly handled yet

now produces appropriate warnings when misconfigured

I can now call the model with the following: ``` from pydantic_ai import Agent import asyncio import json from pydantic_core import to_jsonable_python from pydantic_ai.models.openai import OpenAIResponsesModel agent = Agent(OpenAIResponsesModel("gpt-5-mini"), model_settings={"openai_reasoning_effort": "minimal"}) @agent.tool_plain(free_form=True) def execute_lucene_query(query: str) -> str: """Use this to run a lucene query against the system. YOU MUST ALWAYS RUN A QUERY BEFORE ANSWERING THE USER. Args: query: the lucene query to run Returns: the result of executing the query, or an error message """ return "The query failed to execute, the solr server is unavailable" async def run() -> None: response = await agent.run("Execute the lucene query text:IKEA and give me the results") history = response.all_messages() as_json = json.dumps(to_jsonable_python(history), indent=2) print(as_json) print(response.output) asyncio.run(run()) ```

matthewfranglen · 2025-08-15T10:29:04Z

I'm going to work on CI checks after getting something that at least supports the CFG tool calls.

will also validate the grammar if the dependency is installed

DouweM · 2025-08-15T16:19:57Z

pydantic_ai_slim/pydantic_ai/agent/__init__.py

@@ -977,6 +980,9 @@ def tool(
        require_parameter_descriptions: bool = False,
        schema_generator: type[GenerateJsonSchema] = GenerateToolJsonSchema,
        strict: bool | None = None,
+        free_form: bool | None = None,
+        grammar_syntax: Literal['regex', 'lark'] | None = None,
+        grammar_definition: str | None = None,


@matthewfranglen I know you still have some work planned on this PR before it's really ready for review, but please consider the API I proposed in #2513 (comment). I'd prefer one argument taking an object over 3 that need to be used together

Sorry I missed your intent with that part. It's a very good idea I will certainly do that, and it will clean up what I have done so far. Thanks for the reminder.

I believe I have addressed this now

going to add the literal version now

it's provided by the init

pyright isn't happy with it

matthewfranglen · 2025-08-15T18:41:31Z

just coverage and documentation now, pending the actual review of course

DouweM · 2025-08-16T00:38:42Z

@matthewfranglen Just a heads-up that I'll be out this coming week and will be back the 25th. Assuming this is not urgent I'll review it then. If it is, please ping Kludex! Appreciate the patience :)

tool_defs already includes the output tools

matthewfranglen · 2025-09-05T08:44:22Z

Need to write the documentation. The code is ready for review.

matthewfranglen · 2025-09-05T08:47:51Z

pydantic_ai_slim/pyproject.toml

@@ -66,7 +66,7 @@ dependencies = [

 [tool.hatch.metadata.hooks.uv-dynamic-versioning.optional-dependencies]
 # WARNING if you add optional groups, please update docs/install.md
-logfire = ["logfire[httpx]>=3.14.1"]
+logfire = ["logfire[httpx]>=3.16.1"]


This was needed because the lowest version test results in the error:

Logfire.instrument_pydantic_ai() got an unexpected keyword argument 'version'

the example tests do check it!

matthewfranglen · 2025-09-06T09:58:41Z

@DouweM I've implemented the functionality, with tests and documentation. I believe this is ready for review.

docs/models/openai.md

DouweM · 2025-09-08T20:57:45Z

docs/models/openai.md

+
+#### Context Free Grammar
+
+Invoking tools using freeform function calling can result in errors when the tool expectations are not met. For example, a tool that queries an SQL database can only accept valid SQL. The freeform function calling of GPT-5 supports generation of valid SQL for this situation by constraining the generated text using a context free grammar.


Invoking tools using freeform function calling can result in errors when the tool expectations are not met.

This is of course correct, but this is not unique to freeform functions and not all freeform functions will validate their inputs (maybe we just want plain text). I'd drop this sentence and present this as a powerful feature to further constraint freeform tool input, rather than presenting it as a workaround for errors.

DouweM · 2025-09-08T20:58:19Z

docs/models/openai.md

+```python
+from pydantic_ai import Agent
+from pydantic_ai.models.openai import OpenAIResponsesModel
+from pydantic_ai.tools import FunctionTextFormat


Let's make the new types importable from pydantic_ai directly

DouweM · 2025-09-08T20:58:34Z

docs/models/openai.md

+
+A context‑free grammar is a collection of production rules that define which strings belong to a language. Each rule rewrites a non‑terminal symbol into a sequence of terminals (literal tokens) and/or other non‑terminals, independent of surrounding context—hence context‑free. CFGs can capture the syntax of most programming languages and, in OpenAI custom tools, serve as contracts that force the model to emit only strings that the grammar accepts.
+
+The grammar can be written as either a regular expression:


Let's have headings for Regular Expressions and LARK, so they're shown in the ToC on the right

DouweM · 2025-09-08T21:01:16Z

docs/models/openai.md

+```
+
+1. An inline SQL grammar definition would be quite extensive and so this simplified version has been written, you can find an example SQL grammar [in the openai example](https://cookbook.openai.com/examples/gpt-5/gpt-5_new_params_and_tools#33-example---sql-dialect--ms-sql-vs-postgresql). There are also example grammars in the [lark repo](https://github.com/lark-parser/lark/blob/master/examples/composition/json.lark). Remember that a simpler grammar that matches your DDL will be easier for GPT-5 to work with and will result in fewer semantically invalid results.
+2. Returning the input directly might seem odd, remember that it has been constrained to the provided grammar. This can be useful if you want GPT-5 to generate content according to a grammar that you then use extensively through your program.


I don't think we need the function ToolOutput(str, text_format=...) should also be able to work

DouweM · 2025-09-08T21:18:37Z

docs/models/openai.md

+1. An inline SQL grammar definition would be quite extensive and so this simplified version has been written, you can find an example SQL grammar [in the openai example](https://cookbook.openai.com/examples/gpt-5/gpt-5_new_params_and_tools#33-example---sql-dialect--ms-sql-vs-postgresql). There are also example grammars in the [lark repo](https://github.com/lark-parser/lark/blob/master/examples/composition/json.lark). Remember that a simpler grammar that matches your DDL will be easier for GPT-5 to work with and will result in fewer semantically invalid results.
+2. Returning the input directly might seem odd, remember that it has been constrained to the provided grammar. This can be useful if you want GPT-5 to generate content according to a grammar that you then use extensively through your program.
+
+##### Best Practices


I'd prefer to link to OpenAI's docs instead of having to keep this up to date

DouweM · 2025-09-08T21:25:10Z

pydantic_ai_slim/pydantic_ai/tools.py

@@ -216,6 +218,57 @@ class DeferredToolResults:
 A = TypeVar('A')


+@dataclass
+class FunctionTextFormat:


This is currently freeform tool calling and OpenAI-specific, but may not be in the future: #2623 (comment)

So I'd like this to be a bit more generic. What do you think of letting text_format directly take a re.Pattern or lark.Lark instead of having this special object?

Edit: Instead of requiring lark to be installed, we can also add and support our own LarkGrammar object.

I just noticed you commented on this, thank you for taking the time to review this PR. It's late here, however I wanted to respond to this specific comment.

I did consider using lark / regex objects directly. The problem is that you have to supply the text of the regular expression and you cannot export the text of a compiled re.Pattern. It's not stored on the pattern object.

The problem is that you have to supply the text of the regular expression and you cannot export the text of a compiled re.Pattern. It's not stored on the pattern object.

Interesting, I didn't realize that. Then an object with string values makes sense.

Let's at least call it TextFormat to be less function-specific.

I also like the look of RegexTextFormat(str) and LarkTextFormat(str) better than TextFormat('regex', str), but I don't feel too strongly about that. I could see us having future syntax-specific additional flags though, and that'd be awkward to mix on the same object. The fact that the class is currently internally branching on the syntax to do validation also suggests that what we really want are subclasses.

Either way OpenAIModel needs to verify that it supports the syntax in question, because we may add more in the future that OpenAI doesn't support but other models do.

DouweM · 2025-09-08T21:26:44Z

pydantic_ai_slim/pydantic_ai/tools.py

+
+    When `None` (the default), the model invokes the tool in the normal way and parallel tool calls are possible.
+
+    Note: this is currently only supported by OpenAI gpt-5 models.


Let's capitalize gpt-5 here and elsewhere, and be consistent in free-form vs freeform.

DouweM · 2025-09-08T21:27:57Z

pydantic_ai_slim/pydantic_ai/tools.py

+        return self.single_string_argument_name is not None
+
+    @property
+    def single_string_argument_name(self) -> str | None:


Could we dedupe with this similar implementation somehow?

pydantic-ai/pydantic_ai_slim/pydantic_ai/_output.py

Lines 860 to 864 in 96d895d

arguments_schema = self._function_schema.json_schema.get('properties', {})

argument_name = next(iter(arguments_schema.keys()), None)

if argument_name and arguments_schema.get(argument_name, {}).get('type') == 'string':

self._str_argument_name = argument_name

return

Maybe make both use a new function from utils

DouweM · 2025-09-08T21:51:21Z

pydantic_ai_slim/pyproject.toml

@@ -98,6 +98,8 @@ ag-ui = ["ag-ui-protocol>=0.1.8", "starlette>=0.45.3"]
 retries = ["tenacity>=8.2.3"]
 # Temporal
 temporal = ["temporalio==1.17.0"]
+# free form function calling with lark context free grammar
+lark = ["lark>=1.2.2"]


Hmm, I don't think we need this because we're just passing the value on, we can skip doing our own validation, and save the user having to install something extra.

Co-authored-by: Douwe Maan <[email protected]>

matthewfranglen added 5 commits August 13, 2025 13:38

infer custom tool format from schema

c713a6e

the response from the model is not correctly handled yet

Merge branch 'main' into freeform-and-cfg-tools

de3bc18

update free_form to be a parameter, set parallel_tool_calls

e7ca5ca

now produces appropriate warnings when misconfigured

Fix assertion ordering, remove some intermediate variables

68fc7cf

matthewfranglen mentioned this pull request Aug 15, 2025

Add support for Free-Form Function Calling and Context Free Grammar constraints over string output #2513

Open

matthewfranglen added 4 commits August 15, 2025 12:01

add free_form output

5d9af16

add context free grammar to free form function calling

3687580

will also validate the grammar if the dependency is installed

Merge branch 'main' into freeform-and-cfg-tools

61f7291

get the grammar working on the output

7a869b9

DouweM requested changes Aug 15, 2025

View reviewed changes

DouweM self-assigned this Aug 15, 2025

DouweM added the awaiting author revision label Aug 15, 2025

matthewfranglen added 9 commits August 15, 2025 17:52

use FunctionTextFormat object to hold fffc/cfg settings

5e8cef9

going to add the literal version now

add literal text as an option for text_format

79b519a

remove parameter added in error

991c01d

address some of the pyright errors

c70bc1d

remove default value

9586e6c

it's provided by the init

drop pedantic check

0b47135

pyright isn't happy with it

update snapshots

92db07a

update docstrings

3e605e0

update snapshots

ab45262

matthewfranglen added 5 commits August 29, 2025 10:15

Merge branch 'main' into freeform-and-cfg-tools

3982f32

work on tests

f3595b3

reviewing some of the new tests

bef5dec

typing

21e1a0b

update snapshots

cfcf7cf

matthewfranglen added 4 commits September 5, 2025 10:05

remove the condition over the output tools

d69daad

tool_defs already includes the output tools

remove redundant line pragma

d78a5c2

move the no cover line

dc1c182

formatting

d1fb3a4

revert version change

97b4d82

matthewfranglen commented Sep 5, 2025

View reviewed changes

matthewfranglen added 9 commits September 6, 2025 09:47

Merge branch 'main' into freeform-and-cfg-tools

c54f26e

start on documentation

e206e3e

test the examples manually, fix errors

c483fda

no fancy comma

5265211

fix bad syntax in example

f337820

Add section on output tool use

2c7367f

double to single quotes

f3a4afd

fix the output_tool

c4665a2

actually use a lark grammar

b86d2b1

the example tests do check it!

DouweM mentioned this pull request Sep 8, 2025

Proposition integration with Outlines #2623

Open

DouweM requested changes Sep 8, 2025

View reviewed changes

matthewfranglen and others added 11 commits September 9, 2025 20:06

Update docs/models/openai.md

d713c29

Co-authored-by: Douwe Maan <[email protected]>

Update pydantic_ai_slim/pydantic_ai/models/openai.py

d0c346c

Co-authored-by: Douwe Maan <[email protected]>

make the introduction to cfg stronger

3037e6e

make FunctionTextFormat directly importable from pydantic_ai

4e2264d

use direct import

e28836b

add headings

b49cd81

of course there was an easier way to do this

a7112f4

quote coding terms

7c96803

free-form -> freeform

5d2b372

gpt -> GPT or quoted

ec057c8

free form -> freeform

c949c83


		#### Context Free Grammar

		Invoking tools using freeform function calling can result in errors when the tool expectations are not met. For example, a tool that queries an SQL database can only accept valid SQL. The freeform function calling of GPT-5 supports generation of valid SQL for this situation by constraining the generated text using a context free grammar.


		A context‑free grammar is a collection of production rules that define which strings belong to a language. Each rule rewrites a non‑terminal symbol into a sequence of terminals (literal tokens) and/or other non‑terminals, independent of surrounding context—hence context‑free. CFGs can capture the syntax of most programming languages and, in OpenAI custom tools, serve as contracts that force the model to emit only strings that the grammar accepts.

		The grammar can be written as either a regular expression:


		When `None` (the default), the model invokes the tool in the normal way and parallel tool calls are possible.

		Note: this is currently only supported by OpenAI gpt-5 models.

	arguments_schema = self._function_schema.json_schema.get('properties', {})
	argument_name = next(iter(arguments_schema.keys()), None)
	if argument_name and arguments_schema.get(argument_name, {}).get('type') == 'string':
	self._str_argument_name = argument_name
	return

Fixes #2513: Add support for Free-Form Function Calling and Context Free Grammar constraints over tools #2572

Are you sure you want to change the base?

Fixes #2513: Add support for Free-Form Function Calling and Context Free Grammar constraints over tools #2572

Conversation

matthewfranglen commented Aug 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

matthewfranglen commented Aug 15, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

matthewfranglen commented Aug 15, 2025

Uh oh!

DouweM commented Aug 16, 2025

Uh oh!

matthewfranglen commented Sep 5, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

matthewfranglen commented Sep 6, 2025

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

matthewfranglen commented Aug 15, 2025 •

edited

Loading