Add builtin_tools to Agent
#2102
Conversation
Co-authored-by: Marcelo Trylesinski <[email protected]>
…dantic#1752) Co-authored-by: Marcelo Trylesinski <[email protected]>
- Added builtin_tools field to ModelRequestParameters - Merged new output_mode and output_object fields from main - Updated test snapshots to include all fields - Resolved import conflicts to include both builtin tools and profiles
|
|
||
| city: str | ||
| country: str | ||
| region: str |
There was a problem hiding this comment.
what is region?
More generally, are the required contents (of this field and the others on this class) vendor-specific in any way? Should we include examples?
| part: ServerToolCallPart | ||
| """The server tool call to make.""" | ||
|
|
||
| event_kind: Literal['server_tool_call'] = 'server_tool_call' |
There was a problem hiding this comment.
it makes me sad that some of our discriminators are snake_case (here) and some are kebab-case (part_kind). I guess you probably didn't introduce this inconsistency in this PR, but it feels bad. Maybe we should change it in v1 and just do value normalization during validation (i.e., replace any _ with - or vice versa).
| if part.executable_code is not None: | ||
| items.append(ServerToolCallPart(args=part.executable_code.model_dump(), tool_name='code_execution')) | ||
| elif part.code_execution_result is not None: | ||
| # TODO(Marcelo): Is the idea to generate the tool_call_id on the `executable_code`, and then pass it here? |
There was a problem hiding this comment.
I feel like we can/should answer this question before merging?
|
@Kludex should we include code interpreter from openai as its accessible on the responses API or save that for a follow up given its a fairly complex set of types |
Co-authored-by: David Montague <[email protected]>
mattbrandman
force-pushed
the
builtin-tool
branch
from
c4aa1ef to
4376fc2
Compare
|
@Kludex unless we want to break these out into individual classes this feels like its in a pretty good spot for a first pass and to prevent it already feels like a giant PR |
| if tool == 'web-search': | ||
| self._builtin_tools.append(WebSearchTool()) | ||
| else: | ||
| self._builtin_tools.append(tool) |
There was a problem hiding this comment.
I think we should either support only passing an AbstractBuiltinTool, or include all built-in (to Pydantic AI) built-in tools as strings, meaning also code-execution. If we support strings, I think we should have a dict of name to class in builtin_tools.py, and use that here.
| """ | ||
|
|
||
|
|
||
| class UserLocation(TypedDict, total=False): |
There was a problem hiding this comment.
Can we call this WebSearchUserLocation or something so it's clear it belongs to that class?
|
|
||
| @dataclass(repr=False) | ||
| class ServerToolReturnPart(BaseToolReturnPart): | ||
| """A tool return message from a server tool.""" |
There was a problem hiding this comment.
It's a bit confusing we're calling them "built-in tools" where the user passes them, but "server tools" here. Is there a specific reason for that, or can we standardize on one name? I prefer BuiltinToolReturnPart.
| tool_call_id=item.id, | ||
| ) | ||
| ) | ||
| elif isinstance(item, BetaCodeExecutionToolResultBlock): |
There was a problem hiding this comment.
Can we merge this with the elif isinstance(item, BetaWebSearchToolResultBlock): above?
| @@ -307,6 +311,8 @@ def __init__( | |||
| output_retries: The maximum number of retries to allow for output validation, defaults to `retries`. | |||
| tools: Tools to register with the agent, you can also register tools via the decorators | |||
| [`@agent.tool`][pydantic_ai.Agent.tool] and [`@agent.tool_plain`][pydantic_ai.Agent.tool_plain]. | |||
| builtin_tools: The builtin tools that the agent will use. This depends on the model, as some models may not | |||
| support certain tools. On models that don't support certain tools, the tool will be ignored. | |||
There was a problem hiding this comment.
I feel like silently ignoring built-in tools is unexpected, because the model is not going to behave as intended if it can't search the web or execute code. I'd prefer to raise an error from the model class if it sees an unsupported built-in tool, similarly to how we do for unsupported file/binary content types.
There was a problem hiding this comment.
I'm not sure it's the same case. I expect builtin tools to fail far more than the different content inputs.
There was a problem hiding this comment.
@Kludex I agree, but wouldn't you want the user to realize that they asked for something the model can't actually do?
| @@ -734,6 +754,7 @@ async def _responses_create( | |||
| ) -> responses.Response | AsyncStream[responses.ResponseStreamEvent]: | |||
| tools = self._get_tools(model_request_parameters) | |||
| tools = list(model_settings.get('openai_builtin_tools', [])) + tools | |||
There was a problem hiding this comment.
Should we deprecate this setting and push people to use the new builtin_tools?
This setting currently supports FileSearchToolParam and ComputerToolParam as well, should we implement those as AbstractBuiltinTools?
Should we have a way to build an AbstractBuiltinTool with arbitrary built-in tool JSON supported by a given model, so users can start using them without having to wait for us to add support to the model class?
There was a problem hiding this comment.
Should we deprecate this setting and push people to use the new builtin_tools?
Yes.
This setting currently supports FileSearchToolParam and ComputerToolParam as well, should we implement those as AbstractBuiltinTools?
I don't think they work properly right now, but the idea is to support them yes.
Should we have a way to build an AbstractBuiltinTool with arbitrary built-in tool JSON supported by a given model, so users can start using them without having to wait for us to add support to the model class?
Yes, but that needs to be model specific, so I left it out of this PR. I think we should have that possibility tho.
| @@ -286,4 +286,4 @@ skip = '.git*,*.svg,*.lock,*.css,*.yaml' | |||
| check-hidden = true | |||
| # Ignore "formatting" like **L**anguage | |||
| ignore-regex = '\*\*[A-Z]\*\*[a-z]+\b' | |||
| ignore-words-list = 'asend,aci' | |||
| ignore-words-list = 'asend,aci,Hemishpere,synchonizing' | |||
There was a problem hiding this comment.
Why do we need those? Both of those clearly look like typos
There was a problem hiding this comment.
The LLM generated, but I think I removed them already - We can remove them from here.
Co-authored-by: Douwe Maan <[email protected]>
Fixes test and merge conflicts for #1722
Closes #840