Skip to content

Conversation

Chibionos
Copy link
Contributor

Summary

Fixes issues with evaluation reporting to the backend and adds debugging improvements for coded evaluators.

Key fixes:

  • Updated API endpoints to use new coded eval paths (/coded/evalRun, /coded/evalSetRun)
  • Added support for custom eval backend URL via UIPATH_EVAL_BACKEND_URL for local testing
  • Fixed field mapping to match new API schema (assertionRuns → evaluatorRuns)
  • Improved handling of both legacy and coded evaluators in progress reporter
  • Added --list-runs flag to view previous evaluation runs

Changes:

  • Progress reporter now correctly builds evaluator run data for the new API
  • CLI eval command supports listing previous runs with uipath eval --list-runs
  • Evaluator snapshot includes proper assertion type and output key

This ensures coded evaluators properly report results to the backend and makes local debugging easier.

🤖 Generated with Claude Code

Co-Authored-By: Claude [email protected]

@github-actions github-actions bot added test:uipath-langchain Triggers tests in the uipath-langchain-python repository test:uipath-llamaindex Triggers tests in the uipath-llamaindex-python repository labels Oct 17, 2025
@Chibionos Chibionos force-pushed the fix/llm-judge-evaluator-config-validation branch 2 times, most recently from 3cfb6c9 to 2fe10b8 Compare October 20, 2025 03:11
fix: updating the mocks and evaluator llms
@Chibionos Chibionos force-pushed the fix/llm-judge-evaluator-config-validation branch from 2fe10b8 to 6e2b9a4 Compare October 20, 2025 03:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

test:uipath-langchain Triggers tests in the uipath-langchain-python repository test:uipath-llamaindex Triggers tests in the uipath-llamaindex-python repository

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant