Naming convention of operator in etdump

[yujiaoliang]

After executing a quantized model, the operator name recorded by etdump is 'Fully Connected (NC, QD8, F32, QB4W) GEMM #1'. Could you please tell me which kernel this corresponds to and how to find it?

[GregoryComer]

Are you looking for the actual kernel implementation (in code)? Or the corresponding op in the original graph?

If the question is in regards to the kernel implementation, the kernel code lives in the XNNPACK library. The naming convention for XNNPACK ops in this case is (DimOrder, Activation DType, Output Activation DType, and Weight DType). QD8 means dynamically quantized activations, 8-bit. Output activation dtype is f32. And weights are 4-bit blockwise (groupwise) quantized.

The actual kernels in XNNPACK live under https://github.com/google/XNNPACK/tree/master/src/qd8-f32-qb4w-gemm. Which specific qb4w kernel depends on your hardware at runtime, with the kernel selection logic living in https://github.com/google/XNNPACK/blob/master/src/configs/gemm-config.c#L2334.

[yujiaoliang]

Thank you for your reply. I am trying to find the actual kernel that is being used, and then replace the gemm computation within it with my custom gemm library. There might be a better approach, but I'm not sure. Anyway, thank you very much！

[GregoryComer]

Makes sense. It may be hard to do this with the XNNPACK implementation. @kimishpatel @metascroy do you have any suggestions on the best way to use a custom quantized gemm kernel?

[metascroy]

Could you register your gemm kernel as a custom PyTorch op, and call it that way? You don't need to use XNNPACK to do this.

[yujiaoliang]

Could you register your gemm kernel as a custom PyTorch op, and call it that way? You don't need to use XNNPACK to do this.
Are you suggesting that I should set up a new backend to handle quantized operations using my gemm kernel, or perhaps create my own Quantizer? I'm wondering if such changes would be quite invasive, or if there's a more streamlined way to achieve this in your approach. Could you kindly point me to any relevant documentation? Thanks so much。