[EP] bug fixes #1586
[EP] bug fixes #1586
Conversation
meta-cla
bot
added
the
CLA Signed
| @@ -52,6 +52,7 @@ tensor_parallel_degree = 1 | |||
| enable_async_tensor_parallel = false | |||
| pipeline_parallel_degree = 1 | |||
| pipeline_parallel_schedule = "1F1B" | |||
| context_parallel_degree = 1 | |||
There was a problem hiding this comment.
For deepseek-v3, I remember CP brings the loss Nan issue. Are you trying to support CP here? Or need to revert this change?
There was a problem hiding this comment.
I think putting it in debug_model.toml is fine.
If we would like to warn users, we should error out in code. Hiding it from toml doesn't help too much, IMO.
There was a problem hiding this comment.
Thank you, yeah let me error out CP in deepseek-v3
| local_rank = device_mesh.get_local_rank() | ||
| token_indices_experts_sorted += num_tokens // device_mesh.size() * local_rank | ||
| token_indices_experts_sorted += ( |
There was a problem hiding this comment.
qq: Numerically why this is a bug? Seems that the calculation are still the same but making the num_tokens a class parameter
There was a problem hiding this comment.
The shape of num_tokens are different.
For input, the shape is [bs * slen, topk].
For output, the shape is [bs * slen * topk,], hence the bug.
fixes bug introduced in #1555