OpenAI Built In Code Interpreter not returning image files from OpenAI Response #2391
Description
Initial Checks
- I confirm that I'm using the latest version of Pydantic AI
- I confirm that I searched for my issue in https://github.com/pydantic/pydantic-ai/issues before opening this issue
Description
I’m using the OpenAI built in code interpreter in my agent through the OpenAIResponsesModel. My agent is able to write and execute code through it. However, if my agent asks the Code Interpreter to make a visual element like a chart, I’m not getting the file back through pydantic AI.
Looking at the code in the _process_response function in the OpenAIResponsesModel seems like the data for the file is being returned as item.type = 'code_interpreter_call'
This item type case doesn’t seem to be handled here, so it's being ignored. I dug into the item and I can see information on the container and the filename so the code interpreter seems to be doing the right thing.
Does code need to be added here to handle this or is there a different way to access this image data?
Example Code
from pydantic_ai import Agent
from pydantic_ai.models.openai import OpenAIResponsesModel, OpenAIResponsesModelSettings
from os import getenv
model_settings = OpenAIResponsesModelSettings(
temperature=0.0,
openai_builtin_tools=[{'type': 'code_interpreter', 'container': {'type': 'auto'}}],
)
model = OpenAIResponsesModel('gpt-4o')
ad_hoc_agent = Agent(
model=model,
model_settings=model_settings,
)
@ad_hoc_agent.system_prompt
async def get_system_prompt():
return """You are a helpful assistant that uses the python code interpreter to run python code."""
result = ad_hoc_agent.run_sync("Create a chart of y=x^2 for x=-5 to 5")
print(result.output)
# Here's the plot of \( y = x^2 \) for \( x \) ranging from -5 to 5. Let me know if you need any further assistance!Python, Pydantic AI & LLM client version
Python 3.10.17
pydantic==2.11.7
pydantic-ai-slim==0.4.10
pydantic-evals==0.4.10
pydantic-graph==0.4.10