Skip to content

Add builtin_tools to Agent #2102

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 64 commits into
base: main
Choose a base branch
from

Conversation

mattbrandman
Copy link

@mattbrandman mattbrandman commented Jun 30, 2025

Fixes test and merge conflicts for #1722

Closes #840

Kludex and others added 30 commits May 14, 2025 11:00
- Added builtin_tools field to ModelRequestParameters
- Merged new output_mode and output_object fields from main
- Updated test snapshots to include all fields
- Resolved import conflicts to include both builtin tools and profiles

city: str
country: str
region: str
Copy link
Contributor

@dmontagu dmontagu Jul 16, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what is region?

More generally, are the required contents (of this field and the others on this class) vendor-specific in any way? Should we include examples?

part: ServerToolCallPart
"""The server tool call to make."""

event_kind: Literal['server_tool_call'] = 'server_tool_call'
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it makes me sad that some of our discriminators are snake_case (here) and some are kebab-case (part_kind). I guess you probably didn't introduce this inconsistency in this PR, but it feels bad. Maybe we should change it in v1 and just do value normalization during validation (i.e., replace any _ with - or vice versa).

if part.executable_code is not None:
items.append(ServerToolCallPart(args=part.executable_code.model_dump(), tool_name='code_execution'))
elif part.code_execution_result is not None:
# TODO(Marcelo): Is the idea to generate the tool_call_id on the `executable_code`, and then pass it here?
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I feel like we can/should answer this question before merging?

@mattbrandman
Copy link
Author

@Kludex should we include code interpreter from openai as its accessible on the responses API or save that for a follow up given its a fairly complex set of types

@mattbrandman
Copy link
Author

@Kludex unless we want to break these out into individual classes this feels like its in a pretty good spot for a first pass and to prevent it already feels like a giant PR

@@ -259,7 +262,7 @@ def __init__(
history_processors: Sequence[HistoryProcessor[AgentDepsT]] | None = None,
) -> None: ...

def __init__(
def __init__( # noqa: C901 adding builtin tools
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
def __init__( # noqa: C901 adding builtin tools
def __init__( # noqa: C901

if tool == 'web-search':
self._builtin_tools.append(WebSearchTool())
else:
self._builtin_tools.append(tool)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we should either support only passing an AbstractBuiltinTool, or include all built-in (to Pydantic AI) built-in tools as strings, meaning also code-execution. If we support strings, I think we should have a dict of name to class in builtin_tools.py, and use that here.

"""


class UserLocation(TypedDict, total=False):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we call this WebSearchUserLocation or something so it's clear it belongs to that class?


@dataclass(repr=False)
class ServerToolReturnPart(BaseToolReturnPart):
"""A tool return message from a server tool."""
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's a bit confusing we're calling them "built-in tools" where the user passes them, but "server tools" here. Is there a specific reason for that, or can we standardize on one name? I prefer BuiltinToolReturnPart.

tool_call_id=item.id,
)
)
elif isinstance(item, BetaCodeExecutionToolResultBlock):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we merge this with the elif isinstance(item, BetaWebSearchToolResultBlock): above?

@@ -307,6 +311,8 @@ def __init__(
output_retries: The maximum number of retries to allow for output validation, defaults to `retries`.
tools: Tools to register with the agent, you can also register tools via the decorators
[`@agent.tool`][pydantic_ai.Agent.tool] and [`@agent.tool_plain`][pydantic_ai.Agent.tool_plain].
builtin_tools: The builtin tools that the agent will use. This depends on the model, as some models may not
support certain tools. On models that don't support certain tools, the tool will be ignored.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I feel like silently ignoring built-in tools is unexpected, because the model is not going to behave as intended if it can't search the web or execute code. I'd prefer to raise an error from the model class if it sees an unsupported built-in tool, similarly to how we do for unsupported file/binary content types.

tool_call_id = generate_tool_call_id()
items.append(
ServerToolCallPart(
tool_name=tool.type, args=tool.arguments, model_name='groq', tool_call_id=tool_call_id
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What do we need the model_name for?

@@ -734,6 +754,7 @@ async def _responses_create(
) -> responses.Response | AsyncStream[responses.ResponseStreamEvent]:
tools = self._get_tools(model_request_parameters)
tools = list(model_settings.get('openai_builtin_tools', [])) + tools
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we deprecate this setting and push people to use the new builtin_tools?

This setting currently supports FileSearchToolParam and ComputerToolParam as well, should we implement those as AbstractBuiltinTools?

Should we have a way to build an AbstractBuiltinTool with arbitrary built-in tool JSON supported by a given model, so users can start using them without having to wait for us to add support to the model class?

@@ -286,4 +286,4 @@ skip = '.git*,*.svg,*.lock,*.css,*.yaml'
check-hidden = true
# Ignore "formatting" like **L**anguage
ignore-regex = '\*\*[A-Z]\*\*[a-z]+\b'
ignore-words-list = 'asend,aci'
ignore-words-list = 'asend,aci,Hemishpere,synchonizing'
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we need those? Both of those clearly look like typos

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

GoogleSearchTool for GeminiModel and VertexAIModel
9 participants