Skip to content

Commit 781cca5

Browse files
committed
docs(gepa): revise ReAct metric example to be general and extensible
Replace prescriptive 'minimize tool calls' example with educational progression that shows users how to write effective metrics without forcing specific objectives. Changes: - Show simple metric first (just correctness feedback) - Then show trajectory-based metric (accessing agent execution) - Use clear for-loop instead of list comprehension for readability - Follow DSPy docs conventions: answer_match variable, example/pred naming - Remove 'minimize tool calls' directive - let users decide their objectives - Add bullet points explaining what trajectory can reveal (tool selection, reasoning quality, efficiency) without prescribing how to use it - Rename section to 'Writing Metrics for ReAct Optimization' (more actionable) This aligns with GEPA's philosophy: provide general, extensible patterns that users can adapt to their specific needs. Detailed examples can be shown in tutorials rather than API documentation. Addresses PR comment #5 about prescriptive objectives in documentation.
1 parent bd4cdac commit 781cca5

File tree

1 file changed

+45
-24
lines changed

1 file changed

+45
-24
lines changed

docs/docs/api/optimizers/GEPA/GEPA_Advanced.md

Lines changed: 45 additions & 24 deletions
Original file line numberDiff line numberDiff line change
@@ -507,41 +507,62 @@ class GenerateImprovedReActDescriptionsFromFeedback(dspy.Signature):
507507

508508
The reflection LM receives all current components and execution traces, then decides which components to improve. Tool-specific fields (`improved_tool_{name}_desc`, `improved_tool_{name}_arg_{param}_desc`) are generated dynamically for each tool and parameter.
509509

510-
**Example: Writing Effective Metrics**
510+
**Writing Metrics for ReAct Optimization**
511511

512-
To help GEPA optimize ReAct modules, write metrics that provide trajectory feedback:
512+
GEPA optimizes ReAct modules more effectively when metrics provide feedback about the agent's execution. Here's how to write metrics that help:
513513

514514
```python
515-
def react_metric(example, prediction, trace=None, pred_name=None, pred_trace=None):
516-
"""Metric that provides trajectory feedback for ReAct optimization."""
517-
correct = prediction.answer == example.answer
518-
score = 1.0 if correct else 0.0
515+
def react_metric(example, pred, trace=None, pred_name=None, pred_trace=None):
516+
"""Evaluate ReAct agent performance with trajectory feedback."""
517+
# Check if the answer is correct
518+
answer_match = pred.answer == example.answer
519+
score = 1.0 if answer_match else 0.0
519520

520-
# Extract tool calls from trajectory
521-
trajectory = getattr(prediction, 'trajectory', {})
522-
tool_calls = [
523-
trajectory[key]
524-
for key in trajectory
525-
if key.startswith('tool_name_') and trajectory[key] != 'finish'
526-
]
521+
# Provide feedback to help GEPA understand what happened
522+
feedback = "Correct answer" if answer_match else "Incorrect answer"
527523

528-
if tool_calls:
529-
all_tool_names = ', '.join(tool_calls)
530-
num_calls = len(tool_calls)
531-
feedback = f"{'Correct Answer' if correct else 'Wrong Answer'}. Used {num_calls} tool calls: {all_tool_names}. Try to minimize tool calls."
524+
return dspy.Prediction(score=score, feedback=feedback)
525+
```
526+
527+
You can make feedback more informative by examining the trajectory:
528+
529+
```python
530+
def react_metric_with_trajectory(example, pred, trace=None, pred_name=None, pred_trace=None):
531+
"""Evaluate with trajectory analysis."""
532+
# Check if the answer is correct
533+
answer_match = pred.answer == example.answer
534+
score = 1.0 if answer_match else 0.0
535+
536+
# Access the ReAct trajectory to understand agent behavior
537+
trajectory = getattr(pred, 'trajectory', {})
538+
539+
# Extract tool names from trajectory (excluding 'finish')
540+
tools_used = []
541+
for key in trajectory:
542+
if key.startswith('tool_name_'):
543+
tool_name = trajectory[key]
544+
if tool_name != 'finish':
545+
tools_used.append(tool_name)
546+
547+
# Build feedback message
548+
if answer_match:
549+
feedback = "Correct answer"
532550
else:
533-
feedback = "Correct Answer" if correct else "Wrong Answer"
551+
feedback = "Incorrect answer"
552+
553+
if tools_used:
554+
feedback += f". Tools: {', '.join(tools_used)}"
534555

535556
return dspy.Prediction(score=score, feedback=feedback)
536557
```
537558

538-
This produces feedback like:
539-
```
540-
Correct Answer. Used 2 tool calls: web_search, summarize. Try to minimize tool calls.
541-
Wrong Answer. Used 5 tool calls: web_search, web_search, read_file, web_search, read_file. Try to minimize tool calls.
542-
```
559+
The trajectory contains the agent's step-by-step execution. Use it to provide feedback about:
560+
561+
- **Tool selection**: Were appropriate tools chosen?
562+
- **Reasoning quality**: Did the agent think through the problem?
563+
- **Efficiency**: Were there unnecessary steps?
543564

544-
This feedback helps GEPA learn to reduce unnecessary tool calls while maintaining correct outputs. The reflection LM uses these insights to jointly improve react instructions, tool descriptions, and extraction logic.
565+
The reflection LM uses your feedback to jointly improve react instructions, tool descriptions, and extraction logic.
545566

546567
### How It Works
547568

0 commit comments

Comments
 (0)