Skip to content

Add builtin_tools to Agent #2102

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 65 commits into
base: main
Choose a base branch
from

Conversation

mattbrandman
Copy link

@mattbrandman mattbrandman commented Jun 30, 2025

Fixes test and merge conflicts for #1722

Closes #840

Kludex and others added 30 commits May 14, 2025 11:00
- Added builtin_tools field to ModelRequestParameters
- Merged new output_mode and output_object fields from main
- Updated test snapshots to include all fields
- Resolved import conflicts to include both builtin tools and profiles

city: str
country: str
region: str
Copy link
Contributor

@dmontagu dmontagu Jul 16, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what is region?

More generally, are the required contents (of this field and the others on this class) vendor-specific in any way? Should we include examples?

part: ServerToolCallPart
"""The server tool call to make."""

event_kind: Literal['server_tool_call'] = 'server_tool_call'
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it makes me sad that some of our discriminators are snake_case (here) and some are kebab-case (part_kind). I guess you probably didn't introduce this inconsistency in this PR, but it feels bad. Maybe we should change it in v1 and just do value normalization during validation (i.e., replace any _ with - or vice versa).

if part.executable_code is not None:
items.append(ServerToolCallPart(args=part.executable_code.model_dump(), tool_name='code_execution'))
elif part.code_execution_result is not None:
# TODO(Marcelo): Is the idea to generate the tool_call_id on the `executable_code`, and then pass it here?
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I feel like we can/should answer this question before merging?

@mattbrandman
Copy link
Author

@Kludex should we include code interpreter from openai as its accessible on the responses API or save that for a follow up given its a fairly complex set of types

@mattbrandman
Copy link
Author

@Kludex unless we want to break these out into individual classes this feels like its in a pretty good spot for a first pass and to prevent it already feels like a giant PR

if tool == 'web-search':
self._builtin_tools.append(WebSearchTool())
else:
self._builtin_tools.append(tool)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we should either support only passing an AbstractBuiltinTool, or include all built-in (to Pydantic AI) built-in tools as strings, meaning also code-execution. If we support strings, I think we should have a dict of name to class in builtin_tools.py, and use that here.

"""


class UserLocation(TypedDict, total=False):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we call this WebSearchUserLocation or something so it's clear it belongs to that class?


@dataclass(repr=False)
class ServerToolReturnPart(BaseToolReturnPart):
"""A tool return message from a server tool."""
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's a bit confusing we're calling them "built-in tools" where the user passes them, but "server tools" here. Is there a specific reason for that, or can we standardize on one name? I prefer BuiltinToolReturnPart.

tool_call_id=item.id,
)
)
elif isinstance(item, BetaCodeExecutionToolResultBlock):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we merge this with the elif isinstance(item, BetaWebSearchToolResultBlock): above?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes.

@@ -307,6 +311,8 @@ def __init__(
output_retries: The maximum number of retries to allow for output validation, defaults to `retries`.
tools: Tools to register with the agent, you can also register tools via the decorators
[`@agent.tool`][pydantic_ai.Agent.tool] and [`@agent.tool_plain`][pydantic_ai.Agent.tool_plain].
builtin_tools: The builtin tools that the agent will use. This depends on the model, as some models may not
support certain tools. On models that don't support certain tools, the tool will be ignored.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I feel like silently ignoring built-in tools is unexpected, because the model is not going to behave as intended if it can't search the web or execute code. I'd prefer to raise an error from the model class if it sees an unsupported built-in tool, similarly to how we do for unsupported file/binary content types.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure it's the same case. I expect builtin tools to fail far more than the different content inputs.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Kludex I agree, but wouldn't you want the user to realize that they asked for something the model can't actually do?

@@ -734,6 +754,7 @@ async def _responses_create(
) -> responses.Response | AsyncStream[responses.ResponseStreamEvent]:
tools = self._get_tools(model_request_parameters)
tools = list(model_settings.get('openai_builtin_tools', [])) + tools
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we deprecate this setting and push people to use the new builtin_tools?

This setting currently supports FileSearchToolParam and ComputerToolParam as well, should we implement those as AbstractBuiltinTools?

Should we have a way to build an AbstractBuiltinTool with arbitrary built-in tool JSON supported by a given model, so users can start using them without having to wait for us to add support to the model class?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we deprecate this setting and push people to use the new builtin_tools?

Yes.

This setting currently supports FileSearchToolParam and ComputerToolParam as well, should we implement those as AbstractBuiltinTools?

I don't think they work properly right now, but the idea is to support them yes.

Should we have a way to build an AbstractBuiltinTool with arbitrary built-in tool JSON supported by a given model, so users can start using them without having to wait for us to add support to the model class?

Yes, but that needs to be model specific, so I left it out of this PR. I think we should have that possibility tho.

@@ -286,4 +286,4 @@ skip = '.git*,*.svg,*.lock,*.css,*.yaml'
check-hidden = true
# Ignore "formatting" like **L**anguage
ignore-regex = '\*\*[A-Z]\*\*[a-z]+\b'
ignore-words-list = 'asend,aci'
ignore-words-list = 'asend,aci,Hemishpere,synchonizing'
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we need those? Both of those clearly look like typos

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The LLM generated, but I think I removed them already - We can remove them from here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

GoogleSearchTool for GeminiModel and VertexAIModel
9 participants