[Feature] Add WenetSpeech #829
Conversation
| from lmms_eval.llm_judge import ServerConfig, get_server | ||
|
|
||
| API_TYPE = os.getenv("API_TYPE", "openai") | ||
| # Use JUDGE_MODEL_VERSION instead of MODEL_VERSION | ||
| JUDGE_MODEL_VERSION = os.getenv("JUDGE_MODEL_VERSION", "gpt-4o-mini") | ||
|
|
||
| server_config = ServerConfig( | ||
| model_name=JUDGE_MODEL_VERSION, | ||
| ) | ||
| server = get_server(server_name=API_TYPE, config=server_config) |
|
Hi, Thank you for the PR. I review the wenet part and I think most of the part LGTM. I think the commits is a bit chaos and the file changes include the file changes from your last PR. Do you mind only include the commit that contains the changes for wenet speech? Thanks! You can do that by checkout from a new main and then cherry pick the commit. Thanks! |
Ahh sorry for this, I will make a new one! |
This PR adds comprehensive support for parts of WenetSpeech datasets to lmms-eval.
🔧 Core Implementation
New Dataset: WenetSpeech
2 New Splits with around 25h testing audio data
📊 Example Usage
TASK="wenet_speech"
TASK_SUFFIX="${TASK//,/_}"
🧩 File Changes
New:
wenet_speech/utils.py - Evaluation methods for each subset
4 YAML files - Detailed evaluation information for each task & task group