Skip to content

Conversation

radu-mocanu
Copy link
Contributor

@radu-mocanu radu-mocanu commented Oct 6, 2025

  • wire up new evaluator schemas to eval runtime
  • implemented contains evaluator
    example evaluator spec:
{
  "version": "1.0",
  "id": "DenialCodeContains",
  "description": "Checks if the response text includes the expected denial code.",
  "evaluatorTypeId": "uipath-contains",
  "evaluatorConfig": {
    "name": "Denial Code Contains",
    "targetOutputKey": "report",
    "negated": false,
    "ignoreCase": false,
    "defaultEvaluationCriteria": {
      "searchText": "mock report"
    }
  }
}

example evaluation-set spec:

{
  "version": "1.0",
  "id": "ClaimDenialReview",
  "name": "Claim Denial Review",
  "evaluatorRefs": [
    "DenialCodeContains"
  ],
  "evaluations": [
    {
      "id": "denial-default",
      "name": "Respond with default denial code",
      "inputs": {
        "topic": "Customer asks for the denial code on claim XFC-01."
      },
      "evaluationCriterias": {
        "DenialCodeContains": null
      }
    },
    {
      "id": "denial-override",
      "name": "Respond with override denial code",
      "inputs": {
        "topic": "Customer asks if claim XFC-02 was denied and why."
      },
      "evaluationCriterias": {
        "DenialCodeContains": {
          "searchText": "denial code XFC02"
        }
      }
    },
    {
      "id": "denial-skip",
      "name": "Skip denial code check",
      "inputs": {
        "topic": "Customer checks status of claim XFC-03 with no denial expected."
      },
      "evaluationCriterias": {}
    }
  ]
}

Development Package

  • Add this package as a dependency in your pyproject.toml:
[project]
dependencies = [
  # Exact version:
  "uipath==2.1.68.dev1006461466",

  # Any version from PR
  "uipath>=2.1.68.dev1006460000,<2.1.68.dev1006470000"
]

[[tool.uv.index]]
name = "testpypi"
url = "https://test.pypi.org/simple/"
publish-url = "https://test.pypi.org/legacy/"
explicit = true

[tool.uv.sources]
uipath = { index = "testpypi" }

@github-actions github-actions bot added test:uipath-langchain Triggers tests in the uipath-langchain-python repository test:uipath-llamaindex Triggers tests in the uipath-llamaindex-python repository labels Oct 6, 2025
@radu-mocanu radu-mocanu force-pushed the feat/new-eval-schema branch from 040f882 to 58f7bfc Compare October 6, 2025 16:51
@radu-mocanu radu-mocanu force-pushed the feat/new-eval-schema branch from 58f7bfc to 9a89c2e Compare October 6, 2025 16:53
@radu-mocanu radu-mocanu added the build:dev Create a dev build from the pr label Oct 6, 2025
@andrei-rusu andrei-rusu force-pushed the dev/andreiru/coded_evaluators_revamp branch from 20d2a18 to a55f704 Compare October 9, 2025 15:35
@radu-mocanu radu-mocanu force-pushed the dev/andreiru/coded_evaluators_revamp branch from a55f704 to 85ba576 Compare October 10, 2025 13:41
Base automatically changed from dev/andreiru/coded_evaluators_revamp to release/revamped-evals October 10, 2025 13:42
@radu-mocanu radu-mocanu force-pushed the release/revamped-evals branch from bf67bef to 2e24fe0 Compare October 10, 2025 13:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

build:dev Create a dev build from the pr test:uipath-langchain Triggers tests in the uipath-langchain-python repository test:uipath-llamaindex Triggers tests in the uipath-llamaindex-python repository

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants