[Release]: Release checklist for v0.10.1rc1

[MengqingCao]

Release Checklist

Release Version: v0.10.1.1rc1
Release Branch: main
Release Date:
Release Manager: @MengqingCao

Prepare Release Note

• Create a new issue for release feedback [v0.10.1rc1] FAQ / Feedback | 问题/反馈 #2630

• Write the release note PR. [ReleaseNote] Add Release Note for v0.10.1rc1 #2635

  • Update the feedback issue link in docs/source/faqs.md

  • Add release note to docs/source/user_guide/release_notes.md

  • Update version info in docs/source/community/versioning_policy.md

  • Update contributor info in docs/source/community/contributors.md

  • Update package version in docs/conf.py

PR need Merge

• [Bugfix]Support Qwen3-MOE on aclgraph mode in sizes capture and add new ut [Bugfix]Support Qwen3-MOE on aclgraph mode in sizes capture and add new ut #2511
• [Fix] Add operations in _dummy_run to maintain synchronization with _process_reqs, resolving a service hang [Fix] Add operations in _dummy_run to maintain synchronization with _process_reqs, resolving a service hang #2454
• [Bug]: test_lm_eval_correctness failed: ImportError: cannot import name 'CUDAGraphMode' from 'vllm.config' #2522
• [main][bugfix] Fix MatmulNZ format bug on some machines #2549
• [Aclgraph] Update compilation config in check_and_update_config #2540
• [bugfix] fix torchair runtime error caused by configuration mismtaches and file missing #2532
• fix the bug with torchair + dp [Bugfix] Fix the bug of cos invalid shape when dp #2558
• [main][bugfix] Fix bugs and refactor cached mask generation logic #2442
• [Bugfix] Fix aclgraph not enabled by default #2590
• [Bugfix][LoRA][Patch] Fix the LoRA inference bug after upstream vLLM codebase changed #2560
• Support v0.10.1 #2584
• [Bugfix] Fix mc2 operator error in aclgraph + ep<16 scenario #2609
• [V1][BUGFIX][0.10.1] FIX mtp on main branch #2632
• [BugFix][MLA] Fix attn_mask bug for ring mla #2704
• [Bugfix] Fix qwen2.5-vl-without-padding #2623
• [Bugfix] Keep prefill node using prefill code path only in disaggregation mode #2660
• [P/D]mooncake_connector adapted to 0.10.1 #2664
• bugfix: fix initialization error for mooncake in k8s #2541
• [Bugfix][APC] Fix accuracy issue on prefix caching with AscendScheduler #2714

Functional Test

Bug needs to be fixed:

• issue on mc2: [Bug]: Deepseek runs failed with ep>=16 for graph mode #2523 [Aclgraph] Update compilation config in check_and_update_config #2540 @MengqingCao
• [Bug]: Qwen2.5-7B The process exits for this inner error, and the current working operator name is SelfAttentionOperation #2239 @leo-pony
• GLM4.5 long seq @shen-shanshan [Bugfix] Fix long context seq accuracy problem for GLM4.5 #2601
• GLM4V @Yikun will track on [Bug]: ZhipuAI/GLM-4.5V inference failed in eager mode and start failed in aclgraph mode #2516
• DS 400 QPM @Potabk
  No feature regression
• Qwen3 235B aclgraph @MengqingCao
• gpt-oss: Add gpt oss #2436 depends on [2/N][Feat] Add MC2 communication method for MoE layers #2469
• logits preprocessor @Potabk
• [Bug]: perf test failed due to AttributeError: '_OpNamespace' '_C' object has no attribute 'apply_repetition_penalties_' #2533 @Potabk online test

Doc Test

• Tutorial is updated.
• User Guide is updated.
• Developer Guide is updated.

Prepare Artifacts

• Docker image is ready.
• Wheel package is ready.

Release Step

• Release note PR is merged.
• Post the release on GitHub release page.
• Generate official doc page on https://app.readthedocs.org/dashboard/
• Wait for the wheel package to be available on https://pypi.org/project/vllm-ascend
• Wait for the docker image to be available on https://quay.io/ascend/vllm-ascend
• Upload 310p wheel to Github release page
• Broadcast the release news (By message, blog , etc)
• Close this issue

[Potabk]

#2568 add logits processor example and online serve test case

[Potabk]

#2560 this should fix #2533

[shen-shanshan]

Long Seq Accuracy Problem

❌ Testing GLM4 long seq scenario (max_tokens=1000) gets an broken error (both in A2 and A3):

logs

[rank0]:[E828 01:40:01.863734538 compiler_depend.ts:429] RopeOperation setup failed!
Exception raised from OperationSetup at build/third_party/op-plugin/op_plugin/CMakeFiles/op_plugin_atb.dir/compiler_depend.ts:151 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0xd4 (0xffff7add3ea4 in /home/sss/miniconda3/envs/vllm/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 0xe4 (0xffff7ad73e44 in /home/sss/miniconda3/envs/vllm/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #2: atb::OperationSetup(atb::VariantPack, atb::Operation*, atb::Context*) + 0x254 (0xffff5c56ac24 in /home/sss/miniconda3/envs/vllm/lib/python3.10/site-packages/torch_npu/lib/libop_plugin_atb.so)
frame #3: <unknown function> + 0x8b7bc (0xffff5c56b7bc in /home/sss/miniconda3/envs/vllm/lib/python3.10/site-packages/torch_npu/lib/libop_plugin_atb.so)
frame #4: <unknown function> + 0x2288114 (0xffff6d808114 in /home/sss/miniconda3/envs/vllm/lib/python3.10/site-packages/torch_npu/lib/libtorch_npu.so)
frame #5: <unknown function> + 0x8faad0 (0xffff6be7aad0 in /home/sss/miniconda3/envs/vllm/lib/python3.10/site-packages/torch_npu/lib/libtorch_npu.so)
frame #6: <unknown function> + 0x8fce64 (0xffff6be7ce64 in /home/sss/miniconda3/envs/vllm/lib/python3.10/site-packages/torch_npu/lib/libtorch_npu.so)
frame #7: <unknown function> + 0x8f978c (0xffff6be7978c in /home/sss/miniconda3/envs/vllm/lib/python3.10/site-packages/torch_npu/lib/libtorch_npu.so)
frame #8: <unknown function> + 0xe1ae0 (0xffff7ab91ae0 in /home/sss/miniconda3/envs/vllm/bin/../lib/libstdc++.so.6)
frame #9: <unknown function> + 0x8595c (0xffff8718595c in /lib/aarch64-linux-gnu/libc.so.6)
frame #10: <unknown function> + 0xeb7dc (0xffff871eb7dc in /lib/aarch64-linux-gnu/libc.so.6)

(VllmWorker TP1 pid=6032) ERROR 08-28 01:40:01 [multiproc_executor.py:602] WorkerProc hit an exception.
(VllmWorker TP1 pid=6032) ERROR 08-28 01:40:01 [multiproc_executor.py:602] Traceback (most recent call last):
(VllmWorker TP1 pid=6032) ERROR 08-28 01:40:01 [multiproc_executor.py:602]   File "/home/sss/github/vllm/vllm/v1/executor/multiproc_executor.py", line 597, in worker_busy_loop
(VllmWorker TP1 pid=6032) ERROR 08-28 01:40:01 [multiproc_executor.py:602]     output = func(*args, **kwargs)
(VllmWorker TP1 pid=6032) ERROR 08-28 01:40:01 [multiproc_executor.py:602]   File "/home/sss/github/vllm-ascend/vllm_ascend/worker/worker_v1.py", line 161, in determine_available_memory
(VllmWorker TP1 pid=6032) ERROR 08-28 01:40:01 [multiproc_executor.py:602]     self.model_runner.profile_run()
(VllmWorker TP1 pid=6032) ERROR 08-28 01:40:01 [multiproc_executor.py:602]   File "/home/sss/github/vllm-ascend/vllm_ascend/worker/model_runner_v1.py", line 2095, in profile_run
(VllmWorker TP1 pid=6032) ERROR 08-28 01:40:01 [multiproc_executor.py:602]     hidden_states = self._dummy_run(self.max_num_tokens,
(VllmWorker TP1 pid=6032) ERROR 08-28 01:40:01 [multiproc_executor.py:602]   File "/home/sss/miniconda3/envs/vllm/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
(VllmWorker TP1 pid=6032) ERROR 08-28 01:40:01 [multiproc_executor.py:602]     return func(*args, **kwargs)
(VllmWorker TP1 pid=6032) ERROR 08-28 01:40:01 [multiproc_executor.py:602]   File "/home/sss/github/vllm-ascend/vllm_ascend/worker/model_runner_v1.py", line 2069, in _dummy_run
(VllmWorker TP1 pid=6032) ERROR 08-28 01:40:01 [multiproc_executor.py:602]     hidden_states = self._generate_dummy_run_hidden_states(
(VllmWorker TP1 pid=6032) ERROR 08-28 01:40:01 [multiproc_executor.py:602]   File "/home/sss/github/vllm-ascend/vllm_ascend/worker/model_runner_v1.py", line 1923, in _generate_dummy_run_hidden_states
(VllmWorker TP1 pid=6032) ERROR 08-28 01:40:01 [multiproc_executor.py:602]     hidden_states = self.model(input_ids=input_ids,
(VllmWorker TP1 pid=6032) ERROR 08-28 01:40:01 [multiproc_executor.py:602]   File "/home/sss/miniconda3/envs/vllm/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
(VllmWorker TP1 pid=6032) ERROR 08-28 01:40:01 [multiproc_executor.py:602]     return self._call_impl(*args, **kwargs)
(VllmWorker TP1 pid=6032) ERROR 08-28 01:40:01 [multiproc_executor.py:602]   File "/home/sss/miniconda3/envs/vllm/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
(VllmWorker TP1 pid=6032) ERROR 08-28 01:40:01 [multiproc_executor.py:602]     return forward_call(*args, **kwargs)
(VllmWorker TP1 pid=6032) ERROR 08-28 01:40:01 [multiproc_executor.py:602]   File "/home/sss/github/vllm/vllm/model_executor/models/chatglm.py", line 476, in forward
(VllmWorker TP1 pid=6032) ERROR 08-28 01:40:01 [multiproc_executor.py:602]     hidden_states = self.transformer(input_ids, positions,
(VllmWorker TP1 pid=6032) ERROR 08-28 01:40:01 [multiproc_executor.py:602]   File "/home/sss/github/vllm/vllm/compilation/decorators.py", line 223, in __call__
(VllmWorker TP1 pid=6032) ERROR 08-28 01:40:01 [multiproc_executor.py:602]     return self.forward(*args, **kwargs)
(VllmWorker TP1 pid=6032) ERROR 08-28 01:40:01 [multiproc_executor.py:602]   File "/home/sss/github/vllm/vllm/model_executor/models/chatglm.py", line 356, in forward
(VllmWorker TP1 pid=6032) ERROR 08-28 01:40:01 [multiproc_executor.py:602]     hidden_states = self.encoder(
(VllmWorker TP1 pid=6032) ERROR 08-28 01:40:01 [multiproc_executor.py:602]   File "/home/sss/miniconda3/envs/vllm/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
(VllmWorker TP1 pid=6032) ERROR 08-28 01:40:01 [multiproc_executor.py:602]     return self._call_impl(*args, **kwargs)
(VllmWorker TP1 pid=6032) ERROR 08-28 01:40:01 [multiproc_executor.py:602]   File "/home/sss/miniconda3/envs/vllm/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
(VllmWorker TP1 pid=6032) ERROR 08-28 01:40:01 [multiproc_executor.py:602]     return forward_call(*args, **kwargs)
(VllmWorker TP1 pid=6032) ERROR 08-28 01:40:01 [multiproc_executor.py:602]   File "/home/sss/github/vllm/vllm/model_executor/models/chatglm.py", line 285, in forward
(VllmWorker TP1 pid=6032) ERROR 08-28 01:40:01 [multiproc_executor.py:602]     hidden_states = layer(hidden_states=hidden_states,
(VllmWorker TP1 pid=6032) ERROR 08-28 01:40:01 [multiproc_executor.py:602]   File "/home/sss/miniconda3/envs/vllm/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
(VllmWorker TP1 pid=6032) ERROR 08-28 01:40:01 [multiproc_executor.py:602]     return self._call_impl(*args, **kwargs)
(VllmWorker TP1 pid=6032) ERROR 08-28 01:40:01 [multiproc_executor.py:602]   File "/home/sss/miniconda3/envs/vllm/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
(VllmWorker TP1 pid=6032) ERROR 08-28 01:40:01 [multiproc_executor.py:602]     return forward_call(*args, **kwargs)
(VllmWorker TP1 pid=6032) ERROR 08-28 01:40:01 [multiproc_executor.py:602]   File "/home/sss/github/vllm/vllm/model_executor/models/chatglm.py", line 240, in forward
(VllmWorker TP1 pid=6032) ERROR 08-28 01:40:01 [multiproc_executor.py:602]     output = self.mlp(layernorm_output) + residual
(VllmWorker TP1 pid=6032) ERROR 08-28 01:40:01 [multiproc_executor.py:602]   File "/home/sss/miniconda3/envs/vllm/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
(VllmWorker TP1 pid=6032) ERROR 08-28 01:40:01 [multiproc_executor.py:602]     return self._call_impl(*args, **kwargs)
(VllmWorker TP1 pid=6032) ERROR 08-28 01:40:01 [multiproc_executor.py:602]   File "/home/sss/miniconda3/envs/vllm/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
(VllmWorker TP1 pid=6032) ERROR 08-28 01:40:01 [multiproc_executor.py:602]     return forward_call(*args, **kwargs)
(VllmWorker TP1 pid=6032) ERROR 08-28 01:40:01 [multiproc_executor.py:602]   File "/home/sss/github/vllm/vllm/model_executor/models/chatglm.py", line 164, in forward
(VllmWorker TP1 pid=6032) ERROR 08-28 01:40:01 [multiproc_executor.py:602]     intermediate_parallel = self.activation_func(intermediate_parallel)
(VllmWorker TP1 pid=6032) ERROR 08-28 01:40:01 [multiproc_executor.py:602]   File "/home/sss/miniconda3/envs/vllm/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
(VllmWorker TP1 pid=6032) ERROR 08-28 01:40:01 [multiproc_executor.py:602]     return self._call_impl(*args, **kwargs)
(VllmWorker TP1 pid=6032) ERROR 08-28 01:40:01 [multiproc_executor.py:602]   File "/home/sss/miniconda3/envs/vllm/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
(VllmWorker TP1 pid=6032) ERROR 08-28 01:40:01 [multiproc_executor.py:602]     return forward_call(*args, **kwargs)
(VllmWorker TP1 pid=6032) ERROR 08-28 01:40:01 [multiproc_executor.py:602]   File "/home/sss/github/vllm/vllm/model_executor/custom_op.py", line 44, in forward
(VllmWorker TP1 pid=6032) ERROR 08-28 01:40:01 [multiproc_executor.py:602]     return self._forward_method(*args, **kwargs)
(VllmWorker TP1 pid=6032) ERROR 08-28 01:40:01 [multiproc_executor.py:602]   File "/home/sss/github/vllm-ascend/vllm_ascend/ops/activation.py", line 41, in forward_oot
(VllmWorker TP1 pid=6032) ERROR 08-28 01:40:01 [multiproc_executor.py:602]     out = torch_npu.npu_swiglu(x)
(VllmWorker TP1 pid=6032) ERROR 08-28 01:40:01 [multiproc_executor.py:602]   File "/home/sss/miniconda3/envs/vllm/lib/python3.10/site-packages/torch/_ops.py", line 1158, in __call__
(VllmWorker TP1 pid=6032) ERROR 08-28 01:40:01 [multiproc_executor.py:602]     return self._op(*args, **(kwargs or {}))
(VllmWorker TP1 pid=6032) ERROR 08-28 01:40:01 [multiproc_executor.py:602] RuntimeError: The Inner error is reported as above. The process exits for this inner error, and the current working operator name is RopeOperation.
(VllmWorker TP1 pid=6032) ERROR 08-28 01:40:01 [multiproc_executor.py:602] Since the operator is called asynchronously, the stacktrace may be inaccurate. If you want to get the accurate stacktrace, please set the environment variable ASCEND_LAUNCH_BLOCKING=1.
(VllmWorker TP1 pid=6032) ERROR 08-28 01:40:01 [multiproc_executor.py:602] Note: ASCEND_LAUNCH_BLOCKING=1 will force ops to run in synchronous mode, resulting in performance degradation. Please unset ASCEND_LAUNCH_BLOCKING in time after debugging.

❌ Testing GLM4.5 also get this cyclic output problem:

logs

Prompt: 'The president of the United States is', Generated text: ' the head of state and head of government of the United States, indirectly elected to a four-year term by the American people through the Electoral College. The officeholder leads the executive branch of the federal government and is the commander-in-chief of the United States Armed Forces.\nSince the office was established in 1789, 45 men have served as president. The first was George Washington, who won unanimous support from the Electoral College. Since then, there have been 44 other presidents, including Grover Cleveland, who served two non-consecutive terms and is therefore counted as both the 22nd and 24th president of the United States.\nThe current president is Joe Biden, who was inaugurated on January 20, 2021. He is the 46th president of the United States.\nThe president of the United States is the head of state and head of government of the United States, as well as the commander-in-chief of the United States Armed Forces. The officeholder leads the executive branch of the federal government and is responsible for enforcing federal laws and appointing federal officials, including members of the Cabinet and Supreme Court justices.\nThe president is elected to a four-year term by the Electoral College, a body of electors chosen by the people of each state. The president is not directly elected by the people, but rather by the electors of each state. The number of electors in each state is equal to the number of members of Congress from that state, plus three electors for the District of Columbia.\nThe president is elected to a four-year term, and may be re-elected once. The president is not term-limited, but may serve no more than two terms in office.\nThe president is the head of state and head of government of the United States. As head of state, the president is the chief diplomat of the United States and represents the country on the world stage. As head of government, the president is the chief executive of the federal government and is responsible for enforcing federal laws and appointing federal officials.\nThe president is also the commander-in-chief of the United States Armed Forces. In this role, the president is responsible for the military and for leading the country into war.\nThe president is elected to a four-year term by the Electoral College, a body of electors chosen by the people of each state. The president is not directly elected by the people, but rather by the electors of each state. The number of electors in each state is equal to the number of members of Congress from that state, plus three for the District of Columbia.\nThe president is elected to a four-year term, and may be re-elected once. The president is not term-limited, but may serve no more than two terms in office.\nThe president is the head of state and head of government of the United States. As head of state, the president is the chief diplomat of the United States and represents the country on the world stage. As head of government, the president is the chief executive of the federal government and is responsible for enforcing federal laws and appointing federal officials.\nThe president is also the commander-in-chief of the United States Armed Forces. In this role, the president is responsible for the military and for leading the country in times of war.\nThe president is elected to a four-year term by the Electoral College, a body of electors chosen by the people of each state. The president is not directly elected by the people, but rather by the electors of each state. The number of electors in each state is equal to the number of members of Congress, plus three for the District of Columbia.\nThe president is elected to a four-year term, and may be re-elected once. The president is not term-limited, but may serve no more than two terms in office.\nThe president is the head of state and head of government of the United States. As head of state, the president is the chief diplomat of the United States and represents the country on the world stage. As head of government, the president is the chief executive of the federal government and is responsible for enforcing federal laws and appointing federal officials.\nThe president is also the commander-in-chief of the United States Armed Forces. In this role, the president is responsible for the military and for leading the country in times of war.\nThe president is elected to a four-year term by the Electoral College, a body of electors chosen by the people of each state. The president is not directly elected by the people, but rather by the electors of each state. The number of electors in each state is equal to the number of members of Congress, plus three for the District of Columbia.\nThe president is elected to a four-year term, and may be re-elected once. The president is not term-limited, but may serve no more than two terms in office.\nThe president is the head of state and head of government of the United States. As'
Prompt: 'The capital of France is', Generated text: ' Paris. It is the largest city in France and also the capital. The city is known for its beautiful architecture, art, fashion, and cuisine. It is one of the most popular tourist destinations in the world, attracting millions of visitors each year.\nParis is divided into 20 districts, known as arrondissements, each with its own unique character and charm. The city is home to many famous landmarks, including the Eiffel Tower, the Louvre Museum, and the Notre-Dame Cathedral. The Eiffel Tower is perhaps the most iconic symbol of Paris, attracting millions of visitors each year.\nThe Louvre Museum is the largest art museum in the world and is home to the Mona Lisa, among other famous works of art. The Notre-Dame Cathedral is another iconic landmark of Paris, known for its stunning Gothic architecture.\nParis is also known for its fashion and cuisine. The city is home to some of the most famous fashion houses in the world, including Chanel, Dior, and Yves Saint Laurent. French cuisine is also renowned around the world, with dishes such as escargot, coq au vin, and crème brûlée being popular choices.\nIn addition to its cultural attractions, Paris is also a major center for business, finance, and education. The city is home to several universities and grandes écoles, which are among the most prestigious in the world.\nOverall, Paris is a city of contrasts, where ancient history meets modern innovation, and where the arts are celebrated in all their forms. It is a city that truly has something for everyone, and one that is sure to leave a lasting impression on all who visit. 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 '
Prompt: 'The future of AI is', Generated text: ' here, and it’s called ChatGPT. This revolutionary technology is changing the way we interact with machines, and it’s only getting started. In this article, we’ll explore what ChatGPT is, how it works, and what the future holds for this exciting technology.\nWhat is ChatGPT?\nChatGPT is a large language model that was trained by OpenAI. It’s based on the GPT-3.5 architecture, and it’s designed to be a conversational AI. That means it can understand and respond to natural language input, just like a human would.\nHow does ChatGPT work?\nChatGPT uses a deep learning model called a transformer to process natural language input. The transformer is made up of multiple layers of neural networks, and it’s trained on a massive amount of text data. That’s how it learns to understand and generate human-like responses.\nWhat can you do with ChatGPT?\nYou can use ChatGPT to have natural language conversations on a wide range of topics. It can answer questions, provide recommendations, and even tell jokes. The possibilities are endless!\nWhat’s next for ChatGPT?\nThe future of ChatGPT is bright. OpenAI is constantly working to improve the model and make it more capable. In the future, we can expect to see even more advanced features, like the ability to understand and generate images, videos, and other types of media.\nConclusion\nChatGPT is a revolutionary technology that’s changing the way we interact with machines. It’s a powerful tool that can be used for a wide range of applications, and it’s only going to get better with time. If you’re interested in learning more about ChatGPT, be sure to check out the official OpenAI website. https://openai.com/blog/chatgpt/ https://openai.com/blog/chatgpt/ https://openai.com/blog/chatgpt/ https://openai.com/blog/chatgpt/ https://openai.com/blog/chatgpt/ https://openai.com/blog/chatgpt/ https://openai.com/blog/chatgpt/ https://openai.com/blog/chatgpt/ https://openai.com/blog/chatgpt/ https://openai.com/blog/chatgpt/ https://openai.com/blog/chatgpt/ https://openai.com/blog/chatgpt/ https://openai.com/blog/chatgpt/ https://openai.com/blog/chatgpt/ https://openai.com/blog/chatgpt/ https://openai.com/blog/chatgpt/ https://openai.com/blog/chatgpt/ https://openai.com/blog/chatgpt/ https://openai.com/blog/chatgpt/ https://openai.com/blog/chatgpt/ https://openai.com/blog/chatgpt/ https://openai.com/blog/chatgpt/ https://openai.com/blog/chatgpt/ https://openai.com/blog/chatgpt/ https://openai.com/blog/chatgpt/ https://openai.com/blog/chatgpt/ https://openai.com/blog/chatgpt/ https://openai.com/blog/chatgpt/ https://openai.com/blog/chatgpt/ https://openai.com/blog/chatgpt/ https://openai.com/blog/chatgpt/ https://openai.com/blog/chatgpt/ https://openai.com/blog/chatgpt/ https://openai.com/blog/chatgpt/ https://openai.com/blog/chatgpt/ https://openai.com/blog/chatgpt/ https://openai.com/blog/chatgpt/ https://openai.com/blog/chatgpt/ https://openai.com/blog/chatgpt/ https://openai.com/blog/chatgpt/ https://openai.com/blog/chatgpt/ https://openai.com/blog/chatgpt/ https://openai.com/blog/chatgpt/ https://openai.com/blog/chatgpt/ https://openai.com/blog/chatgpt/ https://openai.com/blog/chatgpt/ https://openai.com/blog/chatgpt/ https://openai.com/blog/chatgpt/ https://openai.com/blog/chatgpt/ https://openai.com/blog/chatgpt/ https://openai.com/blog/chatgpt/ https://openai.com/blog/chatgpt/ https://openai.com/blog/chatgpt/ https://openai.com/blog/chatgpt/ https://openai.com/blog/chatgpt/ https://openai.com/blog/chatgpt/ https://openai.com/blog/chatgpt/ https://openai.com/blog/chatgpt/ https://openai.com/blog/chatgpt/ https://openai.com/blog/chatgpt/ https://openai.com/blog/chatgpt/ https://openai.com/blog/chatgpt/ https://openai.com/blog/chatgpt/ https://openai.com/blog/chatgpt/ https://open'

Fix:

# ops.rotary_embedding.py
        if self.rotary_dim < self.head_size:
            print("-" * 50)
            num_tokens = query.shape[0]
            query = query.view(num_tokens, -1, self.head_size)
            key = key.view(num_tokens, -1, self.head_size)
            q_rot = query[..., :self.rotary_dim]
            q_pass = query[..., self.rotary_dim:]
            k_rot = key[..., :self.rotary_dim]
            k_pass = key[..., self.rotary_dim:]
            q_rot = q_rot.contiguous().view(num_tokens, -1)
            k_rot = k_rot.contiguous().view(num_tokens, -1)
            torch_npu._npu_rotary_embedding(
                positions,
                q_rot,
                k_rot,
                self.head_size,
                self.cos_sin_cache,
                neox_style,
            )
            q_rot = q_rot.view(num_tokens, -1, self.rotary_dim)
            k_rot = k_rot.view(num_tokens, -1, self.rotary_dim)
            q = torch.cat((q_rot, q_pass), dim=-1).reshape(query_shape)
            k = torch.cat((k_rot, k_pass), dim=-1).reshape(key_shape)
            return q, k

✅ After adding this fix, GLM4.5 seems worked well.

logs

Processed prompts: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 4/4 [03:03<00:00, 45.97s/it, est. speed input: 0.12 toks/s, output: 18.22 toks/s]
Prompt: 'Hello, my name is', Generated text: '***** and I have over 20 years of experience as a veterinarian. I am sorry to hear that your fellow has a lump on his side that you are concerned about.\nLumps can be caused by many different things, some benign and some more serious. A benign cyst or an abscess can feel very much like a tumor. The only way to know for sure what it is and whether it needs to be removed is to sample it.\nThere are a few ways to do that. The easiest and least invasive is to do a fine needle aspirate. A needle is inserted into the lump and cells are aspirated, then placed on a slide and examined by a pathologist. This is usually done while the dog is awake and is very quick. It is not 100% accurate but it will give the veterinarian an idea of what type of lump this is.\nThe other way is to take a biopsy of the lump. That requires sedation or anesthesia and a piece of the lump is sent to the pathologist for examination. This is much more accurate but is also more invasive and expensive.\nIf the lump is very small sometimes the best thing is to remove it and send the entire lump in for biopsy.\nI recommend that you have your fellow examined by his veterinarian and have a fine needle aspirate performed to get an idea of what this is. That way you will know whether it can be left alone or if it should be removed.\nIn the meantime monitor the lump. If it is growing very quickly, seems painful, is red or hot or seems to have a discharge then it is more of a concern and should be checked sooner rather than later.\nBest of luck with your fellow, please let me know if you have any further questions.'
Prompt: 'The president of the United States is', Generated text: " the head of state and head of government of the United States, indirectly elected to a four-year term by the American people through the Electoral College. The officeholder leads the executive branch of the federal government and is the commander-in-chief of the United States Armed Forces.\nSince the office was established in 1789, 45 men have served in 46 presidencies. The first president, George Washington, won a unanimous vote of the Electoral College. Grover Cleveland served two non-consecutive terms in office (the only president to have done so), and is therefore counted as the 22nd and 24th president of the United States; the 45th and current president is Donald Trump (since January 20, 2017). There are currently four living former presidents. The most recent former president to die was George H. W. Bush (1989–1993), on November 30, 2018.\nPresidents\nThe presidency of William Henry Harrison, who died 31 days after taking office in 1841, was the shortest in American history. Franklin D. Roosevelt served the longest, over twelve years, before dying early in his fourth term in 1945. He is the only U.S. president to have served more than two terms. Since the ratification of the Twenty-second Amendment to the United States Constitution in 1951, no person may be elected president more than twice, and no one who has served more than two years of a term to which someone else was elected may be elected more than once.\nOf the individuals elected as president, four died in office of natural causes (William Henry Harrison, Zachary Taylor, Warren G. Harding, and Franklin D. Roosevelt), four were assassinated (Abraham Lincoln, James A. Garfield, William McKinley and John F. Kennedy), and one resigned (Richard Nixon). John Tyler was the first vice president to assume the presidency during a presidential term, and set the precedent that a vice president who does so becomes the fully functioning president with his own presidency, as opposed to a caretaker president. The Twenty-fifth Amendment to the Constitution put Tyler's precedent into law in 1967. It also established a mechanism by which a mid-term vacancy in the vice presidency could be filled. Richard Nixon was the first president to fill a vacancy under this provision when he appointed Gerald Ford to the office. Later, Ford became the second to do so when he appointed Nelson Rockefeller to replace him. Previously, a mid-term vacancy was left unfilled.\nThroughout most of its history, American politics has been dominated by political parties. The Constitution is silent on the issue of political parties, and at the time it came into force in 1789, there were no parties. Soon after the 1st Congress convened, factions began rallying around dominant Washington Administration officials, such as Alexander Hamilton and Thomas Jefferson. Greatly concerned about the very real capacity of political parties to destroy the fragile unity holding the nation together, Washington remained unaffiliated with any political faction or party throughout his eight-year presidency. He was, and remains, the only U.S. president never to be affiliated with a political party. Since Washington, every president has been affiliated with a political party at the time of assuming the presidency.\nNumber of presidents living\nThe following table shows the number of living presidents and vice presidents from the inauguration of the first president (1789) to the present time. At times there have been co-presidents or co-vice presidents, as in the case of Andrew Johnson and Benjamin Wade, or when a vice president has resigned (John C. Calhoun and Spiro Agnew) or been assassinated (Andrew Johnson). The table also shows the duration of each president's and vice president's tenure.\nLiving former presidents\nCurrently, there are four living former presidents. The most recent former president to die was George H. W. Bush (1989–1993), on November 30, 2018. The living former presidents, in order of service, are:\nJimmy Carter (1977–1981)\nBill Clinton (1993–2001)\nGeorge W. Bush (2001–2009)\nBarack Obama (2009–2017)\nPresidential election timeline\nThe following timeline depicts the progression of the presidential election season, the major political events during the time, and the dates of the presidential debates. The first debate was held on September 26, 2016, and the last debate was held on October 19, 2016. This was also the first election since 1952 in which neither major party candidate was an incumbent president or vice president.\nPresidential election results\nThe following table shows the results of the 2016 presidential election by state. The winner in each state is indicated by bold text. The number of electoral votes for each state is based on the 2010 Census.\nTimeline of presidents\nThe following is"
Prompt: 'The capital of France is', Generated text: ' Paris. Paris is located in the north-central part of the country, on the banks of the Seine River. It is the largest city in France and one of the most famous cities in the world. Paris is known for its art, fashion, gastronomy, and culture. It is home to many famous landmarks, including the Eiffel Tower, the Louvre Museum, and the Notre-Dame Cathedral. Paris is also a major center for business, finance, and education. It is a global city and a popular tourist destination.\nThe capital of France is Paris. Paris is located in the north-central part of the country, on the banks of the Seine River. It is the largest city in France and one of the most famous cities in the world. Paris is known for its art, fashion, gastronomy, and culture. It is home to many famous landmarks, including the Eiffel Tower, the Louvre Museum, and the Notre-Dame Cathedral. Paris is also a major center for business, finance, and education. It is a global city and a popular tourist destination.\nThe capital of France is Paris. Paris is located in the north-central part of the country, on the banks of the Seine River. It is the largest city in France and one of the most famous cities in the world. Paris is known for its art, fashion, gastronomy, and culture. It is home to many famous landmarks, including the Eiffel Tower, the Louvre Museum, and the Notre-Dame Cathedral. Paris is also a major center for business, finance, and education. It is a global city and a popular tourist destination.\nThe capital of France is Paris. Paris is located in the north-central part of the country, on the banks of the Seine River. It is the largest city in France and one of the most famous cities in the world. Paris is known for its art, fashion, gastronomy, and culture. It is home to many famous landmarks, including the Eiffel Tower, the Louvre Museum, and the Notre-Dame Cathedral. Paris is also a major center for business, finance, and education. It is a global city and a popular tourist destination.\nThe capital of France is Paris. Paris is located in the north-central part of the country, on the banks of the Seine River. It is the largest city in France and one of the most famous cities in the world. Paris is known for its art, fashion, gastronomy, and culture. It is home to many famous landmarks, including the Eiffel Tower, the Louvre Museum, and the Notre-Dame Cathedral. Paris is also a major center for business, finance, and education. It is a global city and a popular tourist destination.\nThe capital of France is Paris. Paris is located in the north-central part of the country, on the banks of the Seine River. It is the largest city in France and one of the most famous cities in the world. Paris is known for its art, fashion, gastronomy, and culture. It is home to many famous landmarks, including the Eiffel Tower, the Louvre Museum, and the Notre-Dame Cathedral. Paris is also a major center for business, finance, and education. It is a global city and a popular tourist destination.\nThe capital of France is Paris. Paris is located in the north-central part of the country, on the banks of the Seine River. It is the largest city in France and one of the most famous cities in the world. Paris is known for its art, fashion, gastronomy, and culture. It is home to many famous landmarks, including the Eiffel Tower, the Louvre Museum, and the Notre-Dame Cathedral. Paris is also a major center for business, finance, and education. It is a global city and a popular tourist destination.\nThe capital of France is Paris. Paris is located in the north-central part of the country, on the banks of the Seine River. It is the largest city in France and one of the most famous cities in the world. Paris is known for its art, fashion, gastronomy, and culture. It is home to many famous landmarks, including the Eiffel Tower, the Louvre Museum, and the Notre-Dame Cathedral. Paris is also a major center for business, finance, and education. It is a global city and a popular tourist destination.\nThe capital of France is Paris. Paris is located in the north-central part of the country, on the banks of the Seine River. It is the largest city in France and one of the most famous cities in the world. Paris is known for its art, fashion, gastronomy, and culture. It is home to many famous landmarks, including the Eiffel Tower, the Louvre Museum, and the Notre-Dame Cathedral. Paris is also a major center for business, finance, and education. It is a global city and a popular'
Prompt: 'The future of AI is', Generated text: ' here, and it’s not just about chatbots and image generators. The next wave of artificial intelligence is multimodal, capable of understanding and interacting with the world through text, images, audio, and video. At the forefront of this revolution is Google’s Gemini, a family of powerful AI models that are redefining what’s possible.\nBut what exactly is Gemini, and why should you care? This comprehensive guide will break down everything you need to know, from its core capabilities to its real-world applications and the ethical questions it raises.\nWhat is Google Gemini?\nAt its heart, Google Gemini is a set of large language models (LLMs) designed from the ground up to be multimodal. Unlike previous models that primarily focused on text, Gemini was built to natively understand and reason across different types of information.\nThink of it this way: older AI models were like specialists who could only read and write. Gemini is a polymath who can read, write, see images, hear sounds, and watch videos, all at the same time, to gain a deeper understanding of any given topic.\nThis native multimodality is Gemini’s superpower. It allows the model to make connections and draw insights that would be impossible for a text-only model. For example, you could show Gemini a diagram of a complex machine and ask it to explain how it works in simple terms, or you could play it a song and ask it to analyze the musical structure and emotional tone.\nThe Gemini Family: A Model for Every Need\nGoogle didn’t just create one AI model; it developed a family of models optimized for different tasks and computational requirements. This ensures that the power of Gemini can be deployed efficiently, from massive data centers to the smartphone in your pocket.\nThe three main models in the Gemini family are:\n1. Gemini Ultra\nThis is the flagship model, the most powerful and capable of the bunch. Gemini Ultra is designed for highly complex tasks that require deep reasoning and advanced multimodal understanding. It excels in areas like:\nScientific research\nSophisticated problem-solving\nAdvanced content creation\nGemini Ultra is the model that achieved groundbreaking results, becoming the first AI model to outperform human experts on the massive multitask language understanding (MMLU) benchmark. This benchmark covers 57 subjects, including math, physics, history, law, medicine, and ethics, testing both world knowledge and problem-solving abilities.\n2. Gemini Pro\nThis is the workhorse of the family, offering a great balance of performance, speed, and cost-effectiveness. Gemini Pro is designed to scale across a wide range of tasks and is the model that powers many of Google’s AI products, including the Bard chatbot (now Gemini).\nIt’s capable of handling everything from drafting emails and summarizing documents to analyzing images and generating code, making it a versatile tool for both developers and end-users.\n3. Gemini Nano\nThis is the lightweight, efficient model designed to run directly on devices, like smartphones. Gemini Nano comes in two sizes (Nano-1 and Nano-2) with 1.8 billion and 3.25 billion parameters, respectively.\nIts on-device nature is crucial for two main reasons:\nSpeed: It provides fast responses without needing to send data to the cloud.\nPrivacy: Sensitive data can be processed locally on the device, enhancing user privacy.\nGemini Nano powers features on the Google Pixel 8 Pro, such as Summarize in the Recorder app and Smart Reply in Gboard.\nKey Capabilities: What Can Gemini Do?\nGemini’s multimodal nature unlocks a vast range of capabilities. Here are some of the most impressive:\n1. Advanced Reasoning and Problem Solving\nGemini can analyze complex information, identify patterns, and solve difficult problems. For example, it can look at a chart filled with data and a text-based question, then reason through the information to provide a correct, well-explained answer. This goes beyond simple pattern matching; it demonstrates a genuine ability to understand context and logic.\n2. Sophisticated Multimodal Understanding\nThis is where Gemini truly shines. It can seamlessly integrate information from different sources. Imagine showing it a picture of a messy kitchen and asking it to generate a recipe based on the visible ingredients. Or, you could give it a physics problem written out in text and a diagram of the setup, and it could solve the problem by combining its understanding of both.\n3. Code Generation and Debugging\nGemini is a powerful tool for developers. It can generate code in various programming languages (like Python, Java, C++, and Go) from simple text prompts. It can also explain existing code, translate code from one language to another, and even help debug errors by analyzing the code and suggesting fixes.\n4. Creative Content Generation\nFrom writing poems and scripts to creating marketing copy, Gemini can generate high-quality, creative text content. Its multimodal abilities extend to this area as well; you could'

Fix PR need merge: #2601.