"Accuracy" metric for LLM model(s)

[freedomtan]

Which "accuracy" metric(s) should we use for LLM benchmarking?

• MMLU: the first item people choose. it covers several field with multiple-choice questions.
  • @mohitmundhragithub please point where/how MLPerf Client use this
  • mostly running a full-set of this is gonna take several hours on Android devices
  • hence, other choice is TinyMMLU
  • TinyMMLU: 100 questions only
• Other tasks such as summarization, Q/A

[mohitmundhragithub]

In the MLPerf client group, they have used the MMLU score for accuracy computation.
They have a separate set of scripts, which processes the output logs generated during the inference to compute the mmlu score. it's an offline process.
You are right... it takes several hours to run the whole suite. In that sense TinyMMLU makes more sense.

The scripts that are used in the client group are shared here: https://github.com/mlcommons/mlperf_client_dev/tree/main/tools/mmlu

But, sadly, there is no public repository containing all the mmlu computation scripts.

One big problem I see is that MMLU score doesn't measure the tasks used for performance. As, its measuring the accuracy of only the 1st token. Basically, we are not measuring the whole pipeline.
Even in the client group, they are looking for other options.

In the inference group, the metric adopted is RogueN. May be we can explore that as well.

[freedomtan]

Let's check if we can run tinyBenchmarks (100 row validation sets maybe) with llama 3.2 1B and 3B and get acceptable numbers first.

@Mostelk @AhmedTElthakeb and @mohitmundhragithub

[mohitmundhragithub]

Let's check if we can run tinyBenchmarks (100 row validation sets maybe) with llama 3.2 1B and 3B and get acceptable numbers first.

@Mostelk @AhmedTElthakeb and @mohitmundhragithub

I think we discussed llama-3.1-8B and llama-3.2-3B in the meeting... right?

[freedomtan]

Let's check if we can run tinyBenchmarks (100 row validation sets maybe) with llama 3.2 1B and 3B and get acceptable numbers first.
@Mostelk @AhmedTElthakeb and @mohitmundhragithub

I think we discussed llama-3.1-8B and llama-3.2-3B in the meeting... right?

right, my bad.

[freedomtan]

baseline numbers

llama 3.2 3B
with lm_eval --model hf --model_args pretrained=meta-llama/Llama-3.2-3B --tasks tinyBenchmarks

Tasks	Version	Filter	n-shot	Metric		Value		Stderr
tinyBenchmarks	N/A
- tinyArc	0	none	25	acc_norm	↑	0.5055	±	N/A
- tinyGSM8k	0	flexible-extract	5	exact_match	↑	0.3170	±	N/A
		strict-match	5	exact_match	↑	0.3170	±	N/A
- tinyHellaswag	0	none	10	acc_norm	↑	0.7742	±	N/A
- tinyMMLU	0	none	0	acc_norm	↑	0.5910	±	N/A
- tinyTruthfulQA	0	none	0	acc	↑	0.4042	±	N/A
- tinyWinogrande	0	none	5	acc_norm	↑	0.6875	±	N/A

llama 3.2 3B Instruct
with lm_eval --model hf --model_args pretrained=meta-llama/Llama-3.2-3B-Instruct --tasks tinyBenchmarks

Tasks	Version	Filter	n-shot	Metric		Value		Stderr
tinyBenchmarks	N/A
- tinyArc	0	none	25	acc_norm	↑	0.5670	±	N/A
- tinyGSM8k	0	flexible-extract	5	exact_match	↑	0.6233	±	N/A
		strict-match	5	exact_match	↑	0.5994	±	N/A
- tinyHellaswag	0	none	10	acc_norm	↑	0.7494	±	N/A
- tinyMMLU	0	none	0	acc_norm	↑	0.6244	±	N/A
- tinyTruthfulQA	0	none	0	acc	↑	0.4795	±	N/A
- tinyWinogrande	0	none	5	acc_norm	↑	0.6298	±	N/A

llama 3.1 8B
with lm_eval --model hf --model_args pretrained=meta-llama/Llama-3.1-8B --tasks tinyBenchmarks

Tasks	Version	Filter	n-shot	Metric		Value		Stderr
tinyBenchmarks	N/A
- tinyArc	0	none	25	acc_norm	↑	0.5733	±	N/A
- tinyGSM8k	0	flexible-extract	5	exact_match	↑	0.5037	±	N/A
		strict-match	5	exact_match	↑	0.5037	±	N/A
- tinyHellaswag	0	none	10	acc_norm	↑	0.8344	±	N/A
- tinyMMLU	0	none	0	acc_norm	↑	0.6335	±	N/A
- tinyTruthfulQA	0	none	0	acc	↑	0.4776	±	N/A
- tinyWinogrande	0	none	5	acc_norm	↑	0.7507	±	N/A

llama 3.1 8B Instruct
with lm_eval --model hf --model_args pretrained=meta-llama/Llama-3.1-8B-Instruct --tasks tinyBenchmarks

Tasks	Version	Filter	n-shot	Metric		Value		Stderr
tinyBenchmarks	N/A
- tinyArc	0	none	25	acc_norm	↑	0.6533	±	N/A
- tinyGSM8k	0	flexible-extract	5	exact_match	↑	0.7610	±	N/A
		strict-match	5	exact_match	↑	0.7289	±	N/A
- tinyHellaswag	0	none	10	acc_norm	↑	0.8054	±	N/A
- tinyMMLU	0	none	0	acc_norm	↑	0.6413	±	N/A
- tinyTruthfulQA	0	none	0	acc	↑	0.5409	±	N/A
- tinyWinogrande	0	none	5	acc_norm	↑	0.7263	±	N/A

[freedomtan]

gemma 3 1b

Tasks	Version	Filter	n-shot	Metric		Value		Stderr
tinyBenchmarks	N/A
- tinyArc	0	none	25	acc_norm	↑	0.4571	±	N/A
- tinyGSM8k	0	flexible-extract	5	exact_match	↑	0.2647	±	N/A
		strict-match	5	exact_match	↑	0.2514	±	N/A
- tinyHellaswag	0	none	10	acc_norm	↑	0.5312	±	N/A
- tinyMMLU	0	none	0	acc_norm	↑	0.4935	±	N/A
- tinyTruthfulQA	0	none	0	acc	↑	0.3836	±	N/A
- tinyWinogrande	0	none	5	acc_norm	↑	0.6210	±	N/A

gemma 3 1b qat q4_0 unquantized

(https://huggingface.co/google/gemma-3-4b-it-qat-q4_0-unquantized)

Tasks	Version	Filter	n-shot	Metric		Value		Stderr
tinyBenchmarks	N/A
- tinyArc	0	none	25	acc_norm	↑	0.4125	±	N/A
- tinyGSM8k	0	flexible-extract	5	exact_match	↑	0.1290	±	N/A
		strict-match	5	exact_match	↑	0.1290	±	N/A
- tinyHellaswag	0	none	10	acc_norm	↑	0.5563	±	N/A
- tinyMMLU	0	none	0	acc_norm	↑	0.4550	±	N/A
- tinyTruthfulQA	0	none	0	acc	↑	0.3881	±	N/A
- tinyWinogrande	0	none	5	acc_norm	↑	0.6292	±	N/A

gemma 3 4b

Tasks	Version	Filter	n-shot	Metric		Value		Stderr
tinyBenchmarks	N/A
- tinyArc	0	none	25	acc_norm	↑	0.6002	±	N/A
- tinyGSM8k	0	flexible-extract	5	exact_match	↑	0.7760	±	N/A
		strict-match	5	exact_match	↑	0.7760	±	N/A
- tinyHellaswag	0	none	10	acc_norm	↑	0.7045	±	N/A
- tinyMMLU	0	none	0	acc_norm	↑	0.6005	±	N/A
- tinyTruthfulQA	0	none	0	acc	↑	0.4619	±	N/A
- tinyWinogrande	0	none	5	acc_norm	↑	0.6961	±	N/A

gemma 3 4b qat q4_0 unquantized

Tasks	Version	Filter	n-shot	Metric		Value		Stderr
tinyBenchmarks	N/A
- tinyArc	0	none	25	acc_norm	↑	0.6137	±	N/A
- tinyGSM8k	0	flexible-extract	5	exact_match	↑	0.7574	±	N/A
		strict-match	5	exact_match	↑	0.7574	±	N/A
- tinyHellaswag	0	none	10	acc_norm	↑	0.6894	±	N/A
- tinyMMLU	0	none	0	acc_norm	↑	0.6036	±	N/A
- tinyTruthfulQA	0	none	0	acc	↑	0.4426	±	N/A
- tinyWinogrande	0	none	5	acc_norm	↑	0.7046	±	N/A

[Mostelk]

Let us try to quantize these and report accuracy llama 3.1 8B Instruct, llama 3.2 3B Instruct

[Aswinoss]

After some exploration, the use cases we are trying to enable (say summarization, context generation etc.,) are not properly captured by the datasets used in TinyBenchmark. Most of the datasets have single-token outputs (except for GSM8k and AlpacaEval) and none of them capture summarization task in particular.

@Mostelk should we still go with above task of quantization and accuracy collection for these datasets ?

We are also looking into Open-Orca dataset which captures all the tasks we discussed on last meet.

We are also looking whether ROGUE-n score could be a viable accuracy metric for Open-Orca dataset.

[freedomtan]

After some exploration, the use cases we are trying to enable (say summarization, context generation etc.,) are not properly captured by the datasets used in TinyBenchmark. Most of the datasets have single-token outputs (except for GSM8k and AlpacaEval) and none of them capture summarization task in particular.

@Mostelk should we still go with above task of quantization and accuracy collection for these datasets ?

We are also looking into Open-Orca dataset which captures all the tasks we discussed on last meet.

We are also looking whether ROGUE-n score could be a viable accuracy metric for Open-Orca dataset.

@Aswinoss I am open to other benchmarks. However, I don't agree that benchmarks used by the TinyBenchmarks are single-token outputs. They are not.

[Aswinoss]

@freedomtan From the tinyBenchmarks page, the ones I have marked as Single-Token had descriptions similar to single-token outputs (PFB). We have not run it, to be exactly sure. But it looks like, the mentioned benchmarks won't have stream of text or content stream as output (except for GSM8k and AlpacaEval).

For example, TinyTruthfulQA had input/output description in this way, where we ask a question and the LLM responds with label / label list, where the true option has 1 and rest all 0s.

One more thing,
we are also okay with any of these accuracy benchmarks. But based on the discussion we had in the last engineering meeting, we thought that these datasets may not be representative enough.

[freedomtan]

@freedomtan From the tinyBenchmarks page, the ones I have marked as Single-Token had descriptions similar to single-token outputs (PFB). We have not run it, to be exactly sure. But it looks like, the mentioned benchmarks won't have stream of text or content stream as output (except for GSM8k and AlpacaEval).

For example, TinyTruthfulQA had input/output description in this way, where we ask a question and the LLM responds with label / label list, where the true option has 1 and rest all 0s.

One more thing, we are also okay with any of these accuracy benchmarks. But based on the discussion we had in the last engineering meeting, we thought that these datasets may not be representative enough.

@Aswinoss it's quite easy to check outputs of those benchmarks. For example, we can

• feed the inputs and get outputs and check them, or
• add --output_path and --log_samples as described at https://github.com/EleutherAI/lm-evaluation-harness?tab=readme-ov-file#saving--caching-results then we can check those outputs

The example you show is kinda internal representation of the dataset.

[freedomtan]

How about quantized model from Meta folks, we know they are available on Huggingface too

• https://huggingface.co/meta-llama/Llama-3.2-3B-Instruct-QLORA_INT4_EO8
• https://huggingface.co/meta-llama/Llama-3.2-3B-Instruct-SpinQuant_INT4_EO8

Well, they are not in Huggingface safetensor format, that is, we can evaluate them with lm_ev --model hf --model_args pretrained=.... It seems to me that we have to use ExecuTorch (see https://github.com/pytorch/executorch/blob/main/examples/models/llama/README.md)

[freedomtan]

@mohitmundhragithub @Aswinoss and @Mostelk OpenOrca is a dataset, not a benchmark.

[Mostelk]

@mohitmundhragithub @freedomtan I agree let us consider other metrics as well, but I prefer one metric that is publicly published by Meta, for example I see here they use SQUAD for reading comprehension,
Also we need to decide whether to use general model or instruction tuned model, looks reading comprehension results below are using the original general model
https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct

Given we used SQUAD before in MobileBert, I don;t think should be hard here as well?
Does Meta report numbers for summarization task?

Reading comprehension

| SQuAD | 1 | em | 76.4 | 77.0 | 85.6 | 81.8 | 89.3
-- | -- | -- | -- | -- | -- | -- | -- | --
QuAC (F1) | 1 | f1 | 44.4 | 44.9 | 51.1 | 51.1 | 53.6
BoolQ | 0 | acc_char | 75.7 | 75.0 | 79.0 | 79.4 | 80.0
DROP (F1) | 3 | f1 | 58.4 | 59.5 | 79.7 | 79.6 | 84.8

[Mostelk]

@mohitmundhragithub @freedomtan please check these 3.1 8B quantized models, with their evaluation, looks using lm-eval, so may be we adopt some tiny lm-eval ( appears using static weight quantization, but dynamic activation quantization)
https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct
https://huggingface.co/RedHatAI/Meta-Llama-3.1-8B-Instruct-quantized.w8a8
https://huggingface.co/RedHatAI/Meta-Llama-3.1-8B-Instruct-quantized.w4a16