[Vulkan] Improve LLM Prefill Performance

[digantdesai]

Quantized Llama3.2 1B on par decode, >1x prefill performance on S24 compared to ET + XNNPACK (4-bit quantized).

cc @SS-JIA @manuelcandales @cbilgin

[SS-JIA]

Current status:

On Samsung Galaxy S24:

• With [ET-VK][qlinear] Faster weight only quantized linear gemv kernel #12444, observe ~49 tok/s with no batched prefill
• With Faster weight only quantized gemm #12862, observe ~260 tok/s prefill and ~34 tok/s decode

Note that batched prefill requires dynamic shapes, which currently incurs the cost of re-encoding a command buffer for each token.

Next steps:

• Reduce command buffer encode overhead by:
  • Break up into smaller command buffers and submit eagerly to improve GPU/CPU parallelism
  • Caching shader dispatch artifacts more aggressively
  • Only re-encode command buffer when "required" (required = new input shapes are larger than the old ones by a certain threshold)