Qwen3 MoE CPU is slower than full precision

[SearchSavior]

Hello!

Thank you team for the awesome work on getting Qwen3 support out.

Did some testing today against full precision and got some pretty terrible results compared to what I think we should expect from an MoE.

Test code:

import time
from threading import Thread
from transformers import AutoTokenizer, TextIteratorStreamer
# from optimum.intel.openvino import OVModelForCausalLM
from transformers import AutoModelForCausalLM


model_id = "" # Can be a local path or an HF id
#ov_config = {"PERFORMANCE_HINT": "LATENCY", "NUM_STREAMS": 4}

print("Loading model...")
load_time = time.perf_counter()
# model = OVModelForCausalLM.from_pretrained(
model = AutoModelForCausalLM.from_pretrained(
    model_id,
   # export=False,
   # ov_config=ov_config,
    device_map="auto",
    torch_dtype="auto"
)
tokenizer = AutoTokenizer.from_pretrained(model_id)
load_time = time.perf_counter() - load_time
print(f"Model loaded in {load_time:.3f} seconds.") 

text_prompt = "We really should join the OpenArc Discord"
conversation = [
    {
        "role": "user",
        "content": text_prompt
    }
]
text_prompt_templated = tokenizer.apply_chat_template(conversation, add_generation_prompt=True, tokenize=False)
inputs = tokenizer(text=text_prompt_templated, return_tensors="pt")
input_token_count = inputs['input_ids'].shape[1]

streamer = TextIteratorStreamer(tokenizer, skip_prompt=True, skip_special_tokens=True)
generation_kwargs = dict(inputs, streamer=streamer, max_new_tokens=128)
thread = Thread(target=model.generate, kwargs=generation_kwargs)

first_token_received = False
generate_start = 0.0
first_token = 0.0
ttft = 0.0
generated_text = ""

generate_start = time.perf_counter()
thread.start()

for new_text in streamer:
    if not first_token_received:
        first_token = time.perf_counter()
        ttft = first_token - generate_start
        first_token_received = True

    print(new_text, end='', flush=True)
    generated_text += new_text

thread.join()
generate_end = time.perf_counter()

generation_time = generate_end - generate_start

num_tokens_generated = len(tokenizer.encode(generated_text))

if generation_time > 0 and num_tokens_generated > 0:
    tokens_per_second = num_tokens_generated / generation_time
    average_token_latency = generation_time / num_tokens_generated

print("\nPerformance Report:")
print("-"*50)
print(f"Input Tokens    : {input_token_count:>9}")
print(f"Output Tokens   : {num_tokens_generated:>9}")
print("")
print(f"Load Time       : {load_time:>9.3f} sec (Model Load Time)")
print(f"TTFT            : {ttft:>9.3f} sec (Time To First Token)")
print(f"Generation Time : {generation_time:>9.3f} sec (Total Generation Time)")
print(f"Throughput      : {tokens_per_second:>9.2f} t/s (Tokens Per Second)")
print(f"Avg Latency     : {average_token_latency:>9.3f} sec (Average Token Latency)")
print("-"*50)

OpenVINO Results:

Model: https://huggingface.co/Echo9Zulu/Qwen3-30B-A3B-int8_asym-ov/tree/main
Converted with:

optimum-cli export openvino -m "" --task text-generation-with-past --weight-format int8_asym ""

Input Tokens    :        16
Output Tokens   :       128

Load Time       :   863.396 sec (Model Load Time)
TTFT            :     6.474 sec (Time To First Token)
Generation Time :    86.857 sec (Total Generation Time)
Throughput      :      1.47 t/s (Tokens Per Second)
Avg Latency     :     0.679 sec (Average Token Latency)

Model: https://huggingface.co/Echo9Zulu/Qwen3-30B-A3B-nf4-ov/tree/main
Converted with:

optimum-cli export openvino -m "" --task text-generation-with-past --weight-format nf4 --ratio 1 --group-size 128 --backup-precision int8_asym --sensitivity-metric weight_quantization_error ""

Input Tokens    :        16

Output Tokens   :       128

Load Time       :   963.818 sec (Model Load Time)
TTFT            :    15.691 sec (Time To First Token)
Generation Time :   119.668 sec (Total Generation Time)
Throughput      :      1.07 t/s (Tokens Per Second)
Avg Latency     :     0.935 sec (Average Token Latency)

Full Precision

Input Tokens    :        16
Output Tokens   :       128

Load Time       :    21.417 sec (Model Load Time)
TTFT            :     3.609 sec (Time To First Token)
Generation Time :   133.330 sec (Total Generation Time)
Throughput      :      0.96 t/s (Tokens Per Second)
Avg Latency     :     1.042 sec (Average Token Latency)

Some Notes

Machine:
2x Xeon 6242
768gb DDR4 ECC
Bare Metal
Ubuntu 22.04

• This test was only one-shot. Timing more generations may benefit performance by keeping model in memory.
• TTFT is faster for int8_asym and nf4 both cases as expected but throughput should be faster
• All generations reached 128 because model did not complete CoT so no eos token
• Maybe its worth testing on Tiber to see Max 1550 performance (keep model in memory
• Later on I will be checking out the openvino_model.xml to see if maybe the nncf algos nuked the model by quanting layers which should be in a different weight format?

I want to help diagnose this but don't know where to start. In #1214 a 'tiny' model was mentioned. How are those used, and are they on the hub? My feeling is that we might need a more nuanced custom export config but I'm not sure- there is probably something simpler to try first.

Again, thank you team for the awesome work!!!

[maxnick]

@SearchSavior, hello, and thanks for raising this issue.
From the results you provided we see that the total generation time is the lowest for int8 (86.857 sec), nf4 is in the middle (119.668 sec) and the native precision is the longest (133.330 sec). So this is rather expected, as nf4 implementation on CPU has certain hardware limitations, which impose a substantial overhead on the nf4 decoding. The same thing we observe for the throughput int8 is the fastest (1.47 t/s) as the more tokens per sec the better, nf4 is in the middle with 1.07 t/s and the native precision gives only 0.96 t/s.
Could you please elaborate a little on the following point?

TTFT is faster for int8_asym and nf4 both cases as expected but throughput should be faster

Do you mean that the absolute throughput value is expected to be higher, or relative throughput ratio between low precision and full precision is expected to be higher?

[SearchSavior]

Thanks for responding!!
 

nf4 implementation on CPU has certain hardware limitations, which impose a substantial overhead on the nf4 decoding

Good to know. Does this mean the performance would be better on future hardware or with features Xeon 6242 doesn't have?

Do you mean that the absolute throughput value is expected to be higher, or relative throughput ratio between low precision and full precision is expected to be higher?

Yes. Both should be true. I agree that the proportion between precisions seems to match the ratio we expect on most cases for CPU only with weight only compression precisions vs native. Similar quants in other frameworks perform closer to what we would expect.

To get a better baseline for the issue I made a fresh quant at int4_asym for both Qwen3-30B and Qwen3-32B and compared to ollama.

NOTE: I dont have time to tinker with ollama/llama.cpp on this machine to limit to one socket, so the ollama results are only toy examples but for MoE seem to demonstrate the performance benefit.

Qwen3-30B-A3B-int4_asym-ov

INFO:nncf:Statistics of the bitwidth distribution:

│ Weight compression mode   │ % all parameters (layers)   │ % ratio-defining parameters (layers)   │

│ int8_asym                 │ 2% (2 / 18674)              │ 0% (0 / 18672)                         │

│ int4_asym                 │ 98% (18672 / 18674)         │ 100% (18672 / 18672)                   │

Input Tokens    :        17
Output Tokens   :       128

Load Time       :   810.459 sec (Model Load Time)
TTFT            :     6.393 sec (Time To First Token)
Generation Time :    80.699 sec (Total Generation Time)
Throughput      :      1.71 t/s (Tokens Per Second)
Avg Latency     :     0.585 sec (Average Token Latency)

Observations:

• Does the bitwidth distribution seem high- does it reflect sparsity? I am not sure how this is configured. Maybe someone from NNCF can provide some insight if that's even relevant?

If we compare to Ollama with q4_km: (ollama run qwen3:30b)

total duration:       9.412261148s
load duration:        17.202602ms
prompt eval count:    17 token(s)
prompt eval duration: 329.45845ms
prompt eval rate:     51.60 tokens/s
eval count:           170 token(s)
eval duration:        9.06485558s
eval rate:            18.75 tokens/s

• We see results that accurately reflect the speedup offered by MoE vs dense, at least for a small prompt.

To enrich our baseline I converted Qwen3-32B

Qwen3-32B-int4_asym-ov

INFO:nncf:Statistics of the bitwidth distribution:

│ Weight compression mode   │ % all parameters (layers)   │ % ratio-defining parameters (layers)   │

│ int8_asym                 │ 5% (2 / 450)                │ 0% (0 / 448)                           │

│ int4_asym                 │ 95% (448 / 450)             │ 100% (448 / 448)                       │


Input Tokens    :        17
Output Tokens   :       128

Load Time       :     7.349 sec (Model Load Time)
TTFT            :     7.208 sec (Time To First Token)
Generation Time :    41.658 sec (Total Generation Time)
Throughput      :      3.69 t/s (Tokens Per Second)
Avg Latency     :     0.271 sec (Average Token Latency)

Qwen3-32B (ollama run qwen3:32b)

total duration:       38.027058636s
load duration:        12.352321579s
prompt eval count:    10 token(s)
prompt eval duration: 1.149058754s
prompt eval rate:     8.70 tokens/s
eval count:           92 token(s)
eval duration:        24.524947296s
eval rate:            3.75 tokens/s

Do these results seem surprising? Can you shed some insight on maybe what size models are intended to work well with OpenVINO in its current state? Anecdotally performance against llama.cpp specifically has always been superior. Perhaps there are high level details I haven't been able to figure out, like how, over time, I have noticed that ipex-llm seems to be the solution for multi-gpu scenarios over OpenVINO, see #1204 . My intuition so far has been that state management between CPU and GPU poses significant challenge to how we manage kv cache

Anyway thanks for getting back to me and I appreciate your work on OpenVINO.

[maxnick]

Hi,
Thank you for the detailed exploration

Good to know. Does this mean the performance would be better on future hardware or with features Xeon 6242 doesn't have?

Modern CPUs that have already been released don't have any special support for NF4, so for the CPU it's better to use int4/int8 to achieve the best performance.

As for the NNCF statistics. It basically indicates that only 2 layers where compressed to int8 (likely embeddings and the last layer) and all the other were compressed to int4 (98% of all parameters). The fact that all the weights available for compression were actually compressed to the target precision (int4) is reflected in the % ratio-defining parameters (layers) column, which is 100%, so in this context it has nothing to do with sparsity, it simply indicates that all the weights are compressed to int4 except two layers, which have to remain in int8.

As for the performance expectations, I think we need to take a look at these results on our side. Stay tuned. We will get back with our findings soon.

[SearchSavior]

Fantastic! Looking forward to your findings.

[luo-cheng2021]

@SearchSavior The PR #31081 should improve the loading and inference speed for qwen3 moe model. Could you please have a try? Thanks.

[SearchSavior]

@luo-cheng2021 Absolutely! Should I do a fresh export?

[luo-cheng2021]

The following has been tested on CPU:
--task text-generation-with-past --weight-format int4
--task text-generation-with-past --weight-format int8
--task text-generation-with-past --weight-format fp16

It is recommended to re-export to reduce mismatches.

[SearchSavior]

Sounds good. I haven't build openvino from source before.

Are these the right docs to reference for building from src? https://github.com/openvinotoolkit/openvino/blob/master/docs/dev/build_linux ? Dated 9 months ago

Anyway, I will test again tonight EST.

Also, what kind of performance uplift were you able to create? Glad I reached out, as from reading this PR I don't know OpenVINO nearly well enough yet to contribute something so exhaustive. So thanks again.

[luo-cheng2021]

Yes. The performance for throughput should be 10x+ better.

[SearchSavior]

@luo-cheng2021 After building from src with a similar script, on the same machine with INT4_ASYM I get:

Input Tokens : 21
Output Tokens : 128

Load Time : 64.088 sec (Model Load Time)
TTFT : 21.082 sec (Time To First Token)
Generation Time : 31.079 sec (Total Generation Time)
Throughput : 12.70 t/s (Tokens Per Second)
Avg Latency : 0.079 sec (Average Token Latency)

Fantastic results! Thank you for your work. Will be exciting to see how this performs with B60.