[Bug]: MXFP4 models still fall back to the Marlin kernel for RTX PRO 6000 (Blackwell SM120)

[heungwing]

Your current environment

python collect_env.py
--2025-12-05 21:50:34-- https://raw.githubusercontent.com/vllm-project/vllm/main/vllm/collect_env.py
Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.110.133, 185.199.108.133, 185.199.109.133, ...
Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.110.133|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 28050 (27K) [text/plain]
Saving to: ‘collect_env.py’

collect_env.py 100%[====================================================================================================>] 27.39K --.-KB/s in 0.01s

2025-12-05 21:50:35 (2.06 MB/s) - ‘collect_env.py’ saved [28050/28050]

Collecting environment information...
uv is set

    System Info

==============================
OS : Ubuntu 24.04.3 LTS (x86_64)
GCC version : (Ubuntu 13.3.0-6ubuntu2~24.04) 13.3.0
Clang version : Could not collect
CMake version : Could not collect
Libc version : glibc-2.39

==============================
PyTorch Info

PyTorch version : 2.9.0+cu130
Is debug build : False
CUDA used to build PyTorch : 13.0
ROCM used to build PyTorch : N/A

==============================
Python Environment

Python version : 3.12.3 (main, Nov 6 2025, 13:44:16) [GCC 13.3.0] (64-bit runtime)
Python platform : Linux-6.6.87.2-microsoft-standard-WSL2-x86_64-with-glibc2.39

==============================
CUDA / GPU Info

Is CUDA available : True
CUDA runtime version : 13.0.88
CUDA_MODULE_LOADING set to :
GPU models and configuration : GPU 0: NVIDIA RTX PRO 6000 Blackwell Workstation Edition
Nvidia driver version : 581.80
cuDNN version : Could not collect
HIP runtime version : N/A
MIOpen runtime version : N/A
Is XNNPACK available : True

==============================
CPU Info

Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Address sizes: 48 bits physical, 48 bits virtual
Byte Order: Little Endian
CPU(s): 32
On-line CPU(s) list: 0-31
Vendor ID: AuthenticAMD
Model name: AMD Ryzen 9 9950X3D 16-Core Processor
CPU family: 26
Model: 68
Thread(s) per core: 2
Core(s) per socket: 16
Socket(s): 1
Stepping: 0
BogoMIPS: 8583.73
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl tsc_reliable nonstop_tsc cpuid extd_apicid tsc_known_freq pni pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy svm cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw topoext perfctr_core ssbd ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 erms invpcid avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves avx_vnni avx512_bf16 clzero xsaveerptr arat npt nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold v_vmsave_vmload avx512vbmi umip avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg avx512_vpopcntdq rdpid fsrm avx512_vp2intersect
Virtualization: AMD-V
Hypervisor vendor: Microsoft
Virtualization type: full
L1d cache: 768 KiB (16 instances)
L1i cache: 512 KiB (16 instances)
L2 cache: 16 MiB (16 instances)
L3 cache: 96 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-31
Vulnerability Gather data sampling: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2: Mitigation; Retpolines; IBPB conditional; IBRS_FW; STIBP always-on; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected
Vulnerability Srbds: Not affected
Vulnerability Tsx async abort: Not affected

==============================
Versions of relevant libraries

[pip3] flashinfer-python==0.5.3
[pip3] numpy==2.1.2
[pip3] nvidia-cublas==13.0.0.19
[pip3] nvidia-cuda-cupti==13.0.48
[pip3] nvidia-cuda-nvrtc==13.0.48
[pip3] nvidia-cuda-runtime==13.0.48
[pip3] nvidia-cudnn-cu13==9.13.0.50
[pip3] nvidia-cudnn-frontend==1.16.0
[pip3] nvidia-cufft==12.0.0.15
[pip3] nvidia-cufile==1.15.0.42
[pip3] nvidia-curand==10.4.0.35
[pip3] nvidia-cusolver==12.0.3.29
[pip3] nvidia-cusparse==12.6.2.49
[pip3] nvidia-cusparselt-cu13==0.8.0
[pip3] nvidia-cutlass-dsl==4.3.2
[pip3] nvidia-ml-py==13.580.82
[pip3] nvidia-nccl-cu13==2.27.7
[pip3] nvidia-nvjitlink==13.0.39
[pip3] nvidia-nvshmem-cu13==3.3.24
[pip3] nvidia-nvtx==13.0.39
[pip3] pyzmq==27.1.0
[pip3] torch==2.9.0+cu130
[pip3] torch-c-dlpack-ext==0.1.3
[pip3] torchaudio==2.9.0+cu130
[pip3] torchvision==0.24.0+cu130
[pip3] transformers==4.57.3
[pip3] triton==3.5.0
[conda] Could not collect

==============================
vLLM Info

ROCM Version : Could not collect
vLLM Version : 0.12.0
vLLM Build Flags:
CUDA Archs: Not Set; ROCm: Disabled
GPU Topology:
GPU0 CPU Affinity NUMA Affinity GPU NUMA ID
GPU0 X N/A

Legend:

X = Self
SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
PIX = Connection traversing at most a single PCIe bridge
NV# = Connection traversing a bonded set of # NVLinks

==============================
Environment Variables

CUDA_VERSION=130
LD_LIBRARY_PATH=/usr/local/cuda-13.0/lib64:/usr/lib/wsl/lib:/usr/local/cuda-13.0/lib64:/usr/local/cuda-13.0/lib64:
PYTORCH_NVML_BASED_CUDA_CHECK=1
TORCHINDUCTOR_COMPILE_THREADS=1

(.venv312) heungwing@DESKTOP-L5KJORG:~/py312cuda130$

🐛 Describe the bug

Just tested OpenAI model on V 0.12.0, seems MXFP4 model will fall back to Marlin kernel.
Does anyone know how to add MXFP4 (FlashInfer) support for the RTX PRO 6000?

[marlin_utils_fp4.py:226] Your GPU does not have native support for FP4 computation but FP4 quantization is being used. Weight-only FP4 compression will be used leveraging the Marlin kernel. This may degrade performance for compute-heavy workloads.

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[Gadflyii]

exact same issue for me

[ducviet00-h2]

same issue on dgxspark

[AdamLouly]

Same for dgx spark, you can use nvidia's docker image for now: https://build.nvidia.com/spark/vllm/instructions

[jman0815]

same here

[gittb]

I can also confirm on RTX Pro 6000 WS.

[khoon485]

FlashInfer dose not support FP4 yet. llama.cpp only support Fp4 in sm_120

[Gadflyii]

I was able to get this working.

Do a freash pull, and build from source, or make sure you have these two recently merged PRs:

PR #29711: Runtime SM dispatch fix for SM120
PR #29242: NVFP4 MoE CUTLASS support for SM120

Run Varibles:

VLLM_USE_FLASHINFER_MOE_MXFP4_MXFP8_CUTLASS=1 # Use CUTLASS backend (SM120 compatible)
VLLM_FLASH_ATTN_VERSION=2 # FA3 Broken ?
CUDA_VISIBLE_DEVICES=0 # Single GPU (TP=2 broken due to NCCL?)

Build commands:

pip install torch==2.9.0 --index-url https://download.pytorch.org/whl/cu128
MAX_JOBS=128 TORCH_CUDA_ARCH_LIST="12.0" pip install -e ".[flashinfer]"
pip install triton==3.5.1
pip uninstall -y triton-kernels # conflicts with vLLM's bundled version

Getting about 153-155 tps with GPT-OSS-120B on a single RTX Pro 6000 with the full context.

[Gadflyii]

FlashInfer dose not support FP4 yet. llama.cpp only support Fp4 in sm_120

that isn't true