Skip to content

BREAKING CHANGE: Support printing reasons in the console output for pydantic-evals #2163

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 9 commits into from
Aug 4, 2025
Merged

BREAKING CHANGE: Support printing reasons in the console output for pydantic-evals #2163

merged 9 commits into from
Aug 4, 2025
+77 −12

Conversation

dmontagu
Copy link
Contributor

@dmontagu dmontagu commented Jul 9, 2025

This adds an include_reasons kwarg to the rendering methods for evaluation reports, to allow you to include reasons in the console output.

Not sure whether we should put reasons in their own columns or just next to the values like we do in this PR. It remains off by default because it can be a bit visually loud and viewing the results in Logfire is probably a better experience most of the time, but with this PR you can opt into this behavior.

Looking for some user feedback before we add tests and merge this.

@github-actions GitHub Actions
Copy link

github-actions bot commented Jul 9, 2025 •
edited
Loading

Docs Preview

commit: 7b07fc3
Preview URL: https://1f7e3cb7-pydantic-ai-previews.pydantic.workers.dev

@dmontagu dmontagu changed the title Support printing reasons in the console output for pydantic-evals BREAKING CHANGE: Support printing reasons in the console output for pydantic-evals Aug 4, 2025
@hyperlint-ai Hyperlint AI
Copy link
Contributor

hyperlint-ai bot commented Aug 4, 2025

PR Change Summary

Enhanced console output for evaluation reports by introducing an option to include reasons alongside values.

  • Added include_reasons keyword argument to rendering methods for evaluation reports.
  • Reasons can be displayed in the console output, enhancing visibility of evaluation results.
  • Feature is off by default to maintain visual clarity in outputs.

Modified Files

  • docs/changelog.md
How can I customize these reviews?

Check out the Hyperlint AI Reviewer docs for more information on how to customize the review.

If you just want to ignore it on this PR, you can add the hyperlint-ignore label to the PR. Future changes won't trigger a Hyperlint review.

Note specifically for link checks, we only check the first 30 links in a file and we cache the results for several hours (for instance, if you just added a page, you might experience this). Our recommendation is to add hyperlint-ignore to the PR to ignore the link check for this PR.

@dmontagu dmontagu merged commit 74d94ec into main Aug 4, 2025
20 checks passed
@dmontagu dmontagu deleted the dmontagu/support-including-reasons branch August 4, 2025 22:49
ethanabrooks added a commit to reflectionai/pydantic-ai that referenced this pull request Aug 20, 2025
@cvanderschuere @akenarsari
@tonyxwz @DouweM @ethan01x @xjose97x @Kludex @strawgate @dmontagu @GuillermoBlasco @HamzaFarhan @medaminezghal @Tiksagol @mattbrandman @claude @AlexEnrique @jerry-heygen @claude @spike-spiegel-21 @alexmojaki @jerry-reevo @ryx2 @kauabh @Viicos @ethanabrooks @eballesteros
Update to v0.7.2 (#9)
f5a0a15 
* Add `priority` `service_tier` to `OpenAIModelSettings` and respect it in `OpenAIResponsesModel` (pydantic#2368)

* Add an example of using RunContext to pass data among tools (pydantic#2316)

Co-authored-by: Douwe Maan <[email protected]>

* Rename gemini-2.5-flash-lite-preview-06-17 to gemini-2.5-flash-lite as it's out of preview (pydantic#2387)

* Fix toggleable toolset example so toolset state is not shared across agent runs (pydantic#2396)

* Support custom thinking tags specified on the model profile (pydantic#2364)

Co-authored-by: jescudero <[email protected]>
Co-authored-by: Douwe Maan <[email protected]>

* Add convenience functions to handle AG-UI requests with request-specific deps (pydantic#2397)

* docs: add missing optional packages in `install.md` (pydantic#2412)

* Include default values in tool arguments JSON schema (pydantic#2418)

* Fix "test_download_item_no_content_type test fails on macOS" (pydantic#2404)

* Allow string format, pattern and others in OpenAI strict JSON mode (pydantic#2420)

* Let more `BaseModel`s use OpenAI strict JSON mode by defaulting to `additionalProperties=False` (pydantic#2419)

* BREAKING CHANGE: Change type of 'source' field on EvaluationResult (pydantic#2388)

Co-authored-by: Douwe Maan <[email protected]>

* Fix ImageUrl, VideoUrl, AudioUrl and DocumentUrl not being serializable (pydantic#2422)

* BREAKING CHANGE: Support printing reasons in the console output for pydantic-evals (pydantic#2163)

* Document performance implications of async vs sync tools (pydantic#2298)

Co-authored-by: Douwe Maan <[email protected]>

* Mention that tools become toolset internally (pydantic#2395)

Co-authored-by: Douwe Maan <[email protected]>

* Fix tests for Logfire>=3.22.0 (pydantic#2346)

* tests: speed up the test suite (pydantic#2414)

* google: add more information about schema on union (pydantic#2426)

* typo in output docs (pydantic#2427)

* Deprecate `GeminiModel` in favor of `GoogleModel` (pydantic#2416)

* Use `httpx` on `GoogleProvider` (pydantic#2438)

* Remove older deprecated models and add new model of Anthropic (pydantic#2435)

* Remove `next()` method from `Graph` (pydantic#2440)

* BREAKING CHANGE: Remove `data` from `FinalResult` (pydantic#2443)

* BREAKING CHANGE: Remove `get_data` and `validate_structured_result` from `StreamedRunResult` (pydantic#2445)

* docs: add `griffe_warnings_deprecated` (pydantic#2444)

* BREAKING CHANGE: Remove `format_as_xml` module (pydantic#2446)

* BREAKING CHANGE: Remove `result_type` parameter and similar from `Agent` (pydantic#2441)

* Deprecate `GoogleGLAProvider` and `GoogleVertexProvider` (pydantic#2450)

* BREAKING CHANGE: drop 4 months old deprecation warnings (pydantic#2451)

* Automatically use OpenAI strict mode for strict-compatible native output types (pydantic#2447)

* Make `InlineDefsJsonSchemaTransformer` public (pydantic#2455)

* Send `ThinkingPart`s back to Anthropic used through Bedrock (pydantic#2454)

* Bump boto3 to support `AWS_BEARER_TOKEN_BEDROCK` API key env var (pydantic#2456)

* Add new Heroku models (pydantic#2459)

* Add `builtin_tools` to `Agent` (pydantic#2102)

Co-authored-by: Marcelo Trylesinski <[email protected]>
Co-authored-by: Douwe Maan <[email protected]>

* Bump mcp-run-python (pydantic#2470)

* Remove fail_under from top-level coverage config so <100% html-coverage step doesn't end CI run (pydantic#2475)

* Add AbstractAgent, WrapperAgent, Agent.event_stream_handler, Toolset.id, Agent.override(tools=...) in preparation for Temporal (pydantic#2458)

* Let toolsets be built dynamically based on run context (pydantic#2366)

Co-authored-by: Douwe Maan <[email protected]>

* Add ToolsetFunc to API docs (fix CI) (pydantic#2486)

* tests: change time of evals example (pydantic#2501)

* ci: remove html and xml reports (pydantic#2491)

* fix: Add gpt-5 models to reasoning model detection for temperature parameter handling (pydantic#2483)

Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com>
Co-authored-by: Douwe Maan <[email protected]>
Co-authored-by: Marcelo Trylesinski <[email protected]>

* History processor replaces message history (pydantic#2324)

Co-authored-by: Marcelo Trylesinski <[email protected]>

* ci: split test suite (pydantic#2436)

Co-authored-by: Douwe Maan <[email protected]>

* ci: use the right install command (pydantic#2506)

* Update config.yaml (pydantic#2514)

* Skip testing flaky evals example (pydantic#2518)

* Fix error when parsing usage details for video without audio track in Google models (pydantic#2507)

* Make OpenAIResponsesModelSettings.openai_builtin_tools work again (pydantic#2520)

* Let Agent be run in a Temporal workflow by moving model requests, tool calls, and MCP to Temporal activities (pydantic#2225)

* Install only dev in CI (pydantic#2523)

* Improve CLAUDE.md (pydantic#2524)

* Add best practices regarding to coverage to CLAUDE.md (pydantic#2527)

* Add support for `"openai-responses"` model inference string (pydantic#2528)

Co-authored-by: Claude <[email protected]>

* docs: Confident AI (pydantic#2529)

* chore: mention what to do with the documentation when deprecating a class (pydantic#2530)

* chore: drop hyperlint (pydantic#2531)

* ci: improve matrix readability (pydantic#2532)

* Add pip to dev deps for PyCharm (pydantic#2533)

Co-authored-by: Marcelo Trylesinski <[email protected]>

* Add genai-prices to dev deps and a basic test (pydantic#2537)

* Add `--durations=100` to all pytest calls in CI (pydantic#2534)

* Cleanup snapshot in test_evaluate_async_logfire (pydantic#2538)

* Make some minor tweaks to the temporal docs (pydantic#2522)

Co-authored-by: Douwe Maan <[email protected]>

* Add new OpenAI GPT-5 models (pydantic#2503)

* Fix `FallbackModel` to respect each model's model settings (pydantic#2540)

* Add support for OpenAI verbosity parameter in Responses API (pydantic#2493)

Co-authored-by: Claude <[email protected]>
Co-authored-by: Douwe Maan <[email protected]>

* Add `UsageLimits.count_tokens_before_request` using Gemini `count_tokens` API (pydantic#2137)

Co-authored-by: Douwe Maan <[email protected]>

* chore: Fix uv.lock (pydantic#2546)

* Stop calling MCP server `get_tools` ahead of `agent run` span (pydantic#2545)

* Disable instrumentation by default in tests (pydantic#2535)

Co-authored-by: Marcelo Trylesinski <[email protected]>

* Only wrap necessary parts of type aliases in forward annotations (pydantic#2548)

* Remove anthropic-beta default header set in `AnthropicModel` (pydantic#2544)

Co-authored-by: Marcelo Trylesinski <[email protected]>

* docs: Clarify why AG-UI example links are on localhost (pydantic#2549)

* chore: Fix path to agent class in CLAUDE.md (pydantic#2550)

* Ignore leading whitespace when streaming from Qwen or DeepSeek (pydantic#2554)

* Ask model to try again if it produced a response without text or tool calls, only thinking (pydantic#2556)

Co-authored-by: Douwe Maan <[email protected]>

* chore: Improve Temporal test to check trace as tree instead of list (pydantic#2559)

* Fix: Forward max_uses parameter to Anthropic WebSearchTool (pydantic#2561)

* Let message history end on ModelResponse and execute pending tool calls (pydantic#2562)

* Fix type issues

* skip tests requiring API keys

* add `google-genai` dependency

* add other provider deps

* add pragma: no cover for untested logic

---------

Co-authored-by: akenar <[email protected]>
Co-authored-by: Tony Woland <[email protected]>
Co-authored-by: Douwe Maan <[email protected]>
Co-authored-by: Yi-Chen Lin <[email protected]>
Co-authored-by: José I. Escudero <[email protected]>
Co-authored-by: jescudero <[email protected]>
Co-authored-by: Marcelo Trylesinski <[email protected]>
Co-authored-by: William Easton <[email protected]>
Co-authored-by: David Montague <[email protected]>
Co-authored-by: Guillermo <[email protected]>
Co-authored-by: Hamza Farhan <[email protected]>
Co-authored-by: Mohamed Amine Zghal <[email protected]>
Co-authored-by: Yinon Ehrlich <[email protected]>
Co-authored-by: Matthew Brandman <[email protected]>
Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com>
Co-authored-by: Douwe Maan <[email protected]>
Co-authored-by: Alex Enrique <[email protected]>
Co-authored-by: Jerry Yan <[email protected]>
Co-authored-by: Claude <[email protected]>
Co-authored-by: Mayank <[email protected]>
Co-authored-by: Alex Hall <[email protected]>
Co-authored-by: Jerry Lin <[email protected]>
Co-authored-by: Raymond Xu <[email protected]>
Co-authored-by: kauabh <[email protected]>
Co-authored-by: Victorien <[email protected]>
Co-authored-by: Ethan Brooks <[email protected]>
Co-authored-by: eballesteros <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@DouweM DouweM DouweM approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@dmontagu @DouweM