Add DeepSeek-R1 B200 Recipes #85
Conversation
Signed-off-by: Pavani Majety <[email protected]>
| @@ -0,0 +1,459 @@ | |||
| # DSR1 Status with vLLM: Aggregated Serving on B200 | |||
|
|
|||
| **Overall Health**: Most paths work. DP Attention is failing in combination with Flashinfer MOE Kernels | |||
There was a problem hiding this comment.
could we link this to a GitHub issue?
| **How to Invoke:** | ||
|
|
||
| FlashInfer: | ||
| - Automatic on SM100 (requires flashinfer installed) |
There was a problem hiding this comment.
What does "Automatic" mean here? it automatically uses FlashInfer gemm on SM100? But this seems to conflict with line 26 which says DeepGemm is the default?
| - `VLLM_USE_DEEP_GEMM_E8M0=1` (default) | ||
|
|
||
| CUTLASS BlockScale: | ||
| - Automatic fallback (requires CUDA 12.8+ for SM100) |
There was a problem hiding this comment.
What does "Automatic fallback" mean? Fall back from what to what?
| - `VLLM_USE_FLASHINFER_MOE_FP8=1 VLLM_FLASHINFER_MOE_BACKEND=throughput` | ||
|
|
||
| CUTLASS BlockScale: | ||
| - Default on SM100 (auto-selected with block quant) |
There was a problem hiding this comment.
If either FlashInfer TRTLLM-Gen or DeepGemm is the most performant, why do we default to CUTLASS? Should we just use DeepGemm by default?
| **Available Backends:** | ||
|
|
||
| TP/EP: | ||
| - Flashinfer TRTLLM-Gen |
There was a problem hiding this comment.
This should be Flashinfer TRTLLM-Gen and CUTLASS, right? (according to the "How to Invoke" section)
There was a problem hiding this comment.
oh, maybe CUTLASS does not support this yet.
| CUDA_VISIBLE_DEVICES=0,1,2,3 \ | ||
| VLLM_USE_STANDALONE_COMPILE=0 \ | ||
| VLLM_USE_FLASHINFER_MOE_FP4=1 \ | ||
| VLLM_FLASHINFER_MOE_BACKEND="latency" \ |
There was a problem hiding this comment.
if we recommend using latency mode, should we just make it the default?
| VLLM_USE_FLASHINFER_MOE_FP4=1 \ | ||
| VLLM_FLASHINFER_MOE_BACKEND="latency" \ | ||
| vllm serve nvidia/DeepSeek-R1-FP4 | ||
| --quantization="modelopt_fp4" \ |
| VLLM_FLASHINFER_MOE_BACKEND="latency" \ | ||
| vllm serve nvidia/DeepSeek-R1-FP4 | ||
| --quantization="modelopt_fp4" \ | ||
| --trust-remote-code \ |
| --quantization="modelopt_fp4" \ | ||
| --trust-remote-code \ | ||
| --max-model-len=2048 \ | ||
| --block-size=128 \ |
| --enable-expert-parallel \ | ||
| --gpu-memory-utilization=0.8 \ | ||
| --tensor-parallel-size=1 \ | ||
| --data-parallel-size=4 |
There was a problem hiding this comment.
Maybe we should also disable prefix caching for perf benchmarking?
2 participants
WIP. Will try merging into existing DeepSeek