[Feature] GEPA does not need to assert that the score returned by feedback func matches the rollout score

[LakshyAAAgrawal]

What feature would you like to see?

Reference: https://discord.com/channels/1161519468141355160/1161519642985111593/1408060002631028837

Would you like to contribute?

• Yes, I'd like to help implement this.
• No, I just want to request it.

Additional Context

No response

[Lucas-Fernandes-Martins]

@LakshyAAAgrawal Would it make sense to have two separate metrics: one for mini-batch evaluation, one for full eval rollouts?

Curious to know what design you have in mind. If by any chance you'd like a helping hand either investigating alternatives or implementing this I'd be happy to take a stab at it.

[LakshyAAAgrawal]

Hi @Lucas-Fernandes-Martins ,

I think having even the current 5-arg metric function is a bit complex, and by default I strive to go towards simplicity, without sacrificing flexibility. Currently, pred_name argument can be a str or None. It is None when GEPA is running a full eval, and doesn't need feedback, and it is set to a str only when GEPA is seeking feedback text. So, you can just check:

if pred_name is None:
    <do not call LLM-as-judge>
else:
    <call LLM-as-judge to generate feedback>

Please let me know if this abstraction doesn't fit your usecase, and I would be glad to brainstorm alternatives!

Also, regarding the issue, I would be glad if you could make a PR addressing the issue. It should be very simple, few line changes, in gepa_utils.py, where the DSPyAdapter is located.

[Lucas-Fernandes-Martins]

@LakshyAAAgrawal thank you very much for the detailed answer! Indeed, your solution fixes this use case quite gracefully, I hadn't considered it.

I've created a simple PR to solve the issue. As of now, I only removed the assert on line 217 of teleprompt/gepa/gepa_utils.py

As far as I tested, this seems to solve the issue and does not cause any other bugs. I've tested with a custom metric_function that always sends different scores for feedback fun and rollout score and found no issues.

Let me know if I'm missing out anything, more than happy to change the PR if needed.

Thank you very much :)