improve: update multi turn math example with concat as default method

[dhh1995]

Description

The multi-turn math example was outdated.
This PR branches from the latest gsm8k grpo example and extends it into a multi-turn math example.
With #668 enabled, the concat mode is now supported and can produce stable training results.

Related Issue

Type of Change

• Bug fix (non-breaking change that fixes an issue)
• New feature (non-breaking change that adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not
  work as expected)
• Documentation update
• Code refactoring (no functional changes)
• Performance improvement
• Test coverage improvement

Checklist

• I have read the Contributing Guide
• I have run formatting tools (pre-commit or manual)
• I have run relevant unit tests and they pass
• I have added tests for new functionality
• I have updated documentation if needed
• My branch is up to date with main
• This PR introduces breaking changes (if yes, fill out details below)
• If this PR changes documentation, I have built and previewed it locally with
  jb build docs
• No critical issues raised by AI reviewers (/gemini review)

Breaking Change Details (if applicable):

Additional Context

---

Need help? Check the Contributing Guide or ask in
GitHub Discussions!

[gemini-code-assist]

Summary of Changes

Hello @dhh1995, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request significantly improves the multi-turn math example by bringing it up to date with the latest GRPO framework and adopting the 'concat' mode for data handling. These changes aim to enhance the stability and performance of training results, providing a more robust and efficient foundation for developing multi-turn math agents. The update also includes a model upgrade and introduces an evaluation workflow to ensure comprehensive assessment of the agent's capabilities.

Highlights

• Multi-Turn Math Example Update: The multi-turn math example has been updated to branch from the latest GSM8K GRPO example, ensuring it is current and leverages recent improvements.
• Default to 'concat' Export Style: The example now defaults to using the 'concat' export style for organizing training data and discounting rewards, which is enabled by a recent feature (fix: fix bug and refact interaction cache #668) and has shown to produce stable training results.
• Model and Configuration Updates: The example now utilizes the Qwen/Qwen2.5-1.5B-Instruct model and includes updated configuration parameters, such as increased n_samples in gconfig and refined reward_norm settings, to optimize performance.
• Code Refactoring and Evaluation Workflow: The core Python script (gsm8k_rl_mt.py) has been refactored to remove outdated configurations, streamline agent and workflow initialization, and introduce a dedicated evaluation workflow for more robust testing.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request updates the multi-turn math example to use concat as the default method, which provides more stable training. The changes involve refactoring the configuration and the training script for better organization and clarity. I've identified a couple of areas for improvement: one to reduce code duplication in the main function and another to remove an unused variable in the MultiTurnMathAgent. Overall, the changes are a good improvement.

[garrett4wade]

Please respect gemini's comment as well

[nuzant]

LGTM.