vmdriver(krunkit): Use Vulkan GPU inside container to run AI models #4271
+178
−146
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
unsuman
force-pushed
the
fix/krunkit-gpu
branch
from
0a0c2b8 to
a70d9dc
Compare
AkihiroSuda
reviewed
Oct 31, 2025
| - GPU (Vulkan via Venus) | ||
| - Recommended distro: Fedora 40+ (smoothest Mesa/Vulkan setup; uses COPR “slp/mesa-krunkit” for patched mesa-vulkan-drivers). | ||
| - Start from the krunkit template and follow the logs to complete GPU setup. | ||
| Start a krunkit VM with rootful containerd (required for granting containers access to `/dev/dri` and the Vulkan API): |
There was a problem hiding this comment.
Why rootless can't access dri ?
There was a problem hiding this comment.
Oh, I remember because of group permission issues on Ubuntu
TU: error: ../src/freedreno/vulkan/tu_knl.cc:385: failed to open device /dev/dri/renderD128 (VK_ERROR_INCOMPATIBLE_DRIVER)
MESA: error: Opening /dev/dri/renderD128 failed: Permission denied
ggml_vulkan: No devices found.
There was a problem hiding this comment.
I can confirm that on Fedora, instances can also work with rootless but not on Ubuntu 🥲:
ansumansahoo@lima-default:/Users/ansumansahoo/Documents/GOLANG/lima2$ nerdctl run --rm -ti --device /dev/dri quay.io/slopezpa/fedora-vgpu-llama
[root@eea2e7854dc4 /]# vulkaninfo --summary
'DISPLAY' environment variable not set... skipping surface info
error: XDG_RUNTIME_DIR is invalid or not set in the environment.
TU: error: ../src/freedreno/vulkan/tu_knl.cc:385: failed to open device /dev/dri/renderD128 (VK_ERROR_INCOMPATIBLE_DRIVER)
WARNING: [../src/panfrost/vulkan/panvk_physical_device.c:56] Code 0 : failed to open device /dev/dri/renderD128 (VK_ERROR_INCOMPATIBLE_DRIVER)
MESA: error: Opening /dev/dri/renderD128 failed: Permission denied
==========
VULKANINFO
==========
Vulkan Instance Version: 1.4.313
........
Devices:
========
GPU0:
apiVersion = 1.4.305
driverVersion = 0.0.1
vendorID = 0x10005
deviceID = 0x0000
deviceType = PHYSICAL_DEVICE_TYPE_CPU
deviceName = llvmpipe (LLVM 20.1.7, 128 bits)
driverID = DRIVER_ID_MESA_LLVMPIPE
driverName = llvmpipe
driverInfo = Mesa 25.0.7 (LLVM 20.1.7)
conformanceVersion = 1.3.1.1
deviceUUID = 6d657361-3235-2e30-2e37-000000000000
driverUUID = 6c6c766d-7069-7065-5555-494400000000
There was a problem hiding this comment.
Can we just let the user join video and render groups?
$ ls -la /dev/dri/
total 0
drwxr-xr-x 3 root root 100 Nov 1 07:46 .
drwxr-xr-x 16 root root 4000 Nov 1 07:46 ..
drwxr-xr-x 2 root root 80 Nov 1 07:46 by-path
crw-rw---- 1 root video 226, 0 Nov 1 07:46 card0
crw-rw---- 1 root render 226, 128 Nov 1 07:46 renderD128There was a problem hiding this comment.
Alternatively you can just chmod or chown it
AkihiroSuda
reviewed
Oct 31, 2025
| - To verify GPU/Vulkan in the guest, use tools like `vulkaninfo` after running the install script. | ||
| - `Libkrun` and [`Ramalama`](https://github.com/containers/ramalama)(a tool that simplifies running AI models locally) use CPU inferencing as of **July 2, 2025** and are actively working to support GPU inferencing. [More info](https://developers.redhat.com/articles/2025/07/02/supercharging-ai-isolation-microvms-ramalama-libkrun#current_limitations_and_future_directions__gpu_enablement). | ||
| - Rootful containerd (`--containerd=system`) is necessary to pass through /dev/dri to containers. | ||
| - To verify GPU/Vulkan in the guest container, use tools like `vulkaninfo`. |
There was a problem hiding this comment.
It should still be possible to use GPU without containers.
The script no longer need to be embedded in the driver code, but can still remain as a doc
There was a problem hiding this comment.
Do you mean to support both?
There was a problem hiding this comment.
Both containers and non-containers
AkihiroSuda
reviewed
Oct 31, 2025
|
|
||
| ```bash | ||
| sudo install-vulkan-gpu.sh | ||
| llama-cli -m /models/YourModel.gguf -b 512 -ngl 99 -p "Introduce yourself" |
There was a problem hiding this comment.
I have only checked with tinyllama-1.1b-chat-v1.0.Q8_0.gguf and Qwen3-1.7B-Q8_0.gguf
To download:
curl -LO 'https://huggingface.co/TheBloke/TinyLlama-1.1B-Chat-v1.0-GGUF/resolve/main/tinyllama-1.1b-chat-v1.0.Q8_0.gguf?download=true'
curl -LO 'https://huggingface.co/Qwen/Qwen3-1.7B-GGUF/resolve/main/Qwen3-1.7B-Q8_0.gguf'
There was a problem hiding this comment.
IMO, tinyllama-1.1b-chat-v1.0.Q8_0.gguf works best
There was a problem hiding this comment.
Can we add that gguf to the document as an example?
There was a problem hiding this comment.
Qwen3-1.7B-Q8_0.gguf seems better for the "Introduce yourself" prompt.
tinyllama-1.1b-chat-v1.0.Q8_0.gguf
I'm Emily, a freelance writer and digital marketing consultant based in Los Angeles, California. I've been working in the industry for over a decade, specializing in helping businesses increase their online presence and drive traffic to their websites.
I started my career as a content writer, creating engaging and informative blog posts for small businesses and non-profits. Over time, I became more interested in marketing and began working as a digital marketing consultant, helping clients optimize their online strategies and increase their website traffic.
My areas of expertise include SEO, social media marketing, paid search advertising, email marketing, and content marketing. I've helped clients across a wide range of industries, from healthcare to technology to hospitality.
I'm passionate about helping businesses of all sizes achieve their online goals, whether that's increasing sales, driving leads, or building brand awareness. I strive to provide a personalized approach to my work, tailoring my strategies to fit each client's specific needs and goals.
I hold a bachelor's degree in English literature from the University of California, Los Angeles and a master's degree in business administration from the University of Southern California. I'm a member of the Association of Digital Marketing and Advertising Professionals and a certified Google Analytics Adwords professional.
In my free time, I enjoy traveling, reading, and spending time with my family and friends. I'm also a lover of all things pop culture, with a particular fondness for movies, TV shows, and music.
Qwen3-1.7B-Q8_0.gguf
<think>
Okay, the user wants me to introduce myself. Let me start by being friendly and welcoming. I should mention my name and what I do. I'm an AI assistant, so I need to clarify that. Maybe mention that I can help with various tasks like answering questions, writing, or providing information. I should also highlight that I'm here to assist and that I'm always ready to help. Keep it simple and concise. Avoid any technical jargon. Make sure to invite the user to ask questions or share their needs. Let me put that all together in a natural, conversational way.
</think>Hello! I'm your AI assistant, and I'm here to help you with anything you need. Whether you're asking for information, writing help, or just want to chat, I'm ready to support you. Feel free to ask me anything, and I'll do my best to assist! 😊
unsuman
force-pushed
the
fix/krunkit-gpu
branch
2 times, most recently
from
b6a8a1c to
cd60d8d
Compare
AkihiroSuda
reviewed
Oct 31, 2025
| sudo nerdctl run --rm -ti \ | ||
| --device /dev/dri \ | ||
| -v ~/models:/models \ | ||
| quay.io/slopezpa/fedora-vgpu-llama |
There was a problem hiding this comment.
Who built this image ?
Where is the Containerfile ?
There was a problem hiding this comment.
AkihiroSuda
reviewed
Oct 31, 2025
| - GPU mode requires a Fedora image/template; Fedora 40+ recommended for Mesa/Vulkan (Venus). | ||
| - To verify GPU/Vulkan in the guest, use tools like `vulkaninfo` after running the install script. | ||
| - `Libkrun` and [`Ramalama`](https://github.com/containers/ramalama)(a tool that simplifies running AI models locally) use CPU inferencing as of **July 2, 2025** and are actively working to support GPU inferencing. [More info](https://developers.redhat.com/articles/2025/07/02/supercharging-ai-isolation-microvms-ramalama-libkrun#current_limitations_and_future_directions__gpu_enablement). | ||
| - Rootful containerd (`--containerd=system`) is necessary to pass through /dev/dri to containers. |
There was a problem hiding this comment.
This doesn't seem needed on Fedora?
Even on other distros, chmod or chown should just work?
AkihiroSuda
reviewed
Oct 31, 2025
| dnf clean all | ||
|
|
||
| echo "Krunkit GPU(Venus) setup complete. Verify Vulkan installation by running 'vulkaninfo --summary'." | ||
| echo "Krunkit GPU(Venus) setup complete. Verify Vulkan installation by running 'vulkaninfo --summary'." |
unsuman
force-pushed
the
fix/krunkit-gpu
branch
from
cd60d8d to
78677a7
Compare
|
CI failing |
AkihiroSuda
reviewed
Nov 4, 2025
|
|
||
| You can now chat with the model. | ||
|
|
||
| ### 2) Run models without containers (slow) |
There was a problem hiding this comment.
Maybe you meant
Suggested change
| ### 2) Run models without containers (slow) | |
| ### 2) Run models without containers (hard way) |
AkihiroSuda
reviewed
Nov 4, 2025
|
|
||
| ### 2) Run models without containers (slow) | ||
|
|
||
| This path builds and installs dependencies (which can take some time. For faster builds, allocate more CPUs and memory to the VM. See [`options`](../../reference/limactl_start/#options)). Use Fedora and make sure your home mount is writable. |
There was a problem hiding this comment.
Home doesn't need to be writable?
AkihiroSuda
reviewed
Nov 4, 2025
| - `Libkrun` and [`Ramalama`](https://github.com/containers/ramalama)(a tool that simplifies running AI models locally) use CPU inferencing as of **July 2, 2025** and are actively working to support GPU inferencing. [More info](https://developers.redhat.com/articles/2025/07/02/supercharging-ai-isolation-microvms-ramalama-libkrun#current_limitations_and_future_directions__gpu_enablement). | ||
| - To verify GPU/Vulkan in the guest container or VM, use tools like `vulkaninfo --summary`. | ||
| - AI models on containers can run on any Linux distribution but without containers Fedora is required. | ||
| - For more information about usage of `llama-cli`. See [llama.cpp](https://github.com/ggml-org/llama.cpp?tab=readme-ov-file#llamacpp) docs. |
There was a problem hiding this comment.
Suggested change
| - For more information about usage of `llama-cli`. See [llama.cpp](https://github.com/ggml-org/llama.cpp?tab=readme-ov-file#llamacpp) docs. | |
| - For more information about usage of `llama-cli`. See [llama.cpp](https://github.com/ggml-org/llama.cpp) docs. |
AkihiroSuda
reviewed
Nov 4, 2025
| nerdctl run --rm -ti \ | ||
| --device /dev/dri \ | ||
| -v $(pwd)/models:/models \ | ||
| quay.io/slopezpa/fedora-vgpu-llama |
There was a problem hiding this comment.
Link https://sinrega.org/2024-03-06-enabling-containers-gpu-macos/ as the reference
AkihiroSuda
reviewed
Nov 4, 2025
There was a problem hiding this comment.
Probably this just can be a part of doc, not an embedded script
There was a problem hiding this comment.
Sorry? Do you mean to add the steps to krunkit.md and remove the hack?
There was a problem hiding this comment.
Will this not become tedious for the user because the script has a lot commands to run?
There was a problem hiding this comment.
They can just copy paste the script from a web browser.
Anyway no strong opinion from me. Feel free to retain it as an embedded script if you like.
AkihiroSuda
reviewed
Nov 4, 2025
| echo "Installing llama.cpp with Vulkan support..." | ||
| # Build and install llama.cpp with Vulkan support | ||
| dnf install -y git cmake clang curl-devel glslc vulkan-devel virglrenderer | ||
| git clone https://github.com/ggml-org/llama.cpp |
There was a problem hiding this comment.
This should use the guest home, not the mounted host home
unsuman
force-pushed
the
fix/krunkit-gpu
branch
3 times, most recently
from
bdc4396 to
e93ea35
Compare
AkihiroSuda
reviewed
Nov 4, 2025
| curl -LO --output-dir models 'https://huggingface.co/Qwen/Qwen3-1.7B-GGUF/resolve/main/Qwen3-1.7B-Q8_0.gguf' | ||
| ``` | ||
|
|
||
| ### 1) Run models using containers (fastest) |
There was a problem hiding this comment.
Suggested change
| ### 1) Run models using containers (fastest) | |
| ### 1) Run models using containers (easiest) |
AkihiroSuda
reviewed
Nov 4, 2025
| cmake -B build -DGGML_VULKAN=ON -DGGML_CCACHE=OFF -DCMAKE_INSTALL_PREFIX=/usr | ||
| cmake --build build --config Release -j8 | ||
| cmake --install build | ||
| cd .. && rm -fr llama.cpp && cd PWD |
There was a problem hiding this comment.
Suggested change
| cd .. && rm -fr llama.cpp && cd PWD | |
| cd .. && rm -fr llama.cpp |
AkihiroSuda
reviewed
Nov 4, 2025
| echo "Installing llama.cpp with Vulkan support..." | ||
| # Build and install llama.cpp with Vulkan support | ||
| dnf install -y git cmake clang curl-devel glslc vulkan-devel virglrenderer | ||
| PWD=$(pwd) |
Signed-off-by: Ansuman Sahoo <[email protected]>
unsuman
force-pushed
the
fix/krunkit-gpu
branch
from
e93ea35 to
a6ea2fa
Compare
AkihiroSuda
reviewed
Nov 4, 2025
| cmake --install build | ||
| cd .. && rm -fr llama.cpp | ||
|
|
||
| echo "Successfully installed llama.cpp with Vulkan support. Use 'llama-cli' app with .gguf models." |
There was a problem hiding this comment.
Doesn't work on my MacBook Pro 2024 (M4 Max, macOS 26.0.1, krunkit v1.1.1). Does it work for you?
-- ARM detected
-- Performing Test GGML_COMPILER_SUPPORTS_FP16_FORMAT_I3E
-- Performing Test GGML_COMPILER_SUPPORTS_FP16_FORMAT_I3E - Failed
-- ARM -mcpu not found, -mcpu=native will be used
-- Performing Test GGML_MACHINE_SUPPORTS_dotprod
-- Performing Test GGML_MACHINE_SUPPORTS_dotprod - Failed
-- Performing Test GGML_MACHINE_SUPPORTS_nodotprod
-- Performing Test GGML_MACHINE_SUPPORTS_nodotprod - Failed
-- Performing Test GGML_MACHINE_SUPPORTS_i8mm
-- Performing Test GGML_MACHINE_SUPPORTS_i8mm - Failed
-- Performing Test GGML_MACHINE_SUPPORTS_noi8mm
-- Performing Test GGML_MACHINE_SUPPORTS_noi8mm - Failed
-- Performing Test GGML_MACHINE_SUPPORTS_sve
-- Performing Test GGML_MACHINE_SUPPORTS_sve - Failed
-- Performing Test GGML_MACHINE_SUPPORTS_nosve
-- Performing Test GGML_MACHINE_SUPPORTS_nosve - Failed
-- Performing Test GGML_MACHINE_SUPPORTS_sme
-- Performing Test GGML_MACHINE_SUPPORTS_sme - Failed
-- Performing Test GGML_MACHINE_SUPPORTS_nosme
-- Performing Test GGML_MACHINE_SUPPORTS_nosme - Failed
cc1: sorry, unimplemented: no support for ‘sme’ without ‘sve2’
CMake Warning at ggml/src/ggml-cpu/CMakeLists.txt:163 (message):
Failed to get ARM features
Call Stack (most recent call first):
ggml/src/CMakeLists.txt:310 (ggml_add_cpu_backend_variant_impl)
-- Adding CPU backend variant ggml-cpu: -mcpu=native
-- Found Vulkan: /lib64/libvulkan.so (found version "1.4.321") found components: glslc missing components: glslangValidator
-- Vulkan found
-- GL_KHR_cooperative_matrix supported by glslc
-- GL_NV_cooperative_matrix2 supported by glslc
-- GL_EXT_integer_dot_product supported by glslc
-- GL_EXT_bfloat16 supported by glslc
-- Including Vulkan backend
-- Found CURL: /usr/lib64/libcurl.so (found version "8.15.0")
-- Configuring done (0.9s)
-- Generating done (0.1s)
-- Build files have been written to: /root/llama.cpp/build
[ 1%] Creating directories for 'vulkan-shaders-gen'
[ 1%] Generating build details from Git
[ 1%] Building C object ggml/src/CMakeFiles/ggml-base.dir/ggml.c.o
[ 2%] Building CXX object tools/mtmd/CMakeFiles/llama-llava-cli.dir/deprecation-warning.cpp.o
[ 3%] Building C object examples/gguf-hash/CMakeFiles/xxhash.dir/deps/xxhash/xxhash.c.o
[ 4%] Building C object examples/gguf-hash/CMakeFiles/sha1.dir/deps/sha1/sha1.c.o
[ 4%] Building CXX object tools/mtmd/CMakeFiles/llama-gemma3-cli.dir/deprecation-warning.cpp.o
-- Found Git: /usr/sbin/git (found version "2.51.1")
[ 4%] Building C object examples/gguf-hash/CMakeFiles/sha256.dir/deps/sha256/sha256.c.o
[ 4%] No download step for 'vulkan-shaders-gen'
[ 4%] No update step for 'vulkan-shaders-gen'
[ 5%] No patch step for 'vulkan-shaders-gen'
[ 5%] Performing configure step for 'vulkan-shaders-gen'
[ 5%] Building CXX object common/CMakeFiles/build_info.dir/build-info.cpp.o
-- The C compiler identification is GNU 15.2.1
[ 5%] Built target build_info
[ 5%] Building CXX object tools/mtmd/CMakeFiles/llama-minicpmv-cli.dir/deprecation-warning.cpp.o
[ 5%] Built target sha1
[ 5%] Linking CXX executable ../../bin/llama-llava-cli
[ 5%] Linking CXX executable ../../bin/llama-gemma3-cli
[ 6%] Building CXX object tools/mtmd/CMakeFiles/llama-qwen2vl-cli.dir/deprecation-warning.cpp.o
[ 6%] Built target sha256
[ 6%] Building CXX object ggml/src/CMakeFiles/ggml-base.dir/ggml.cpp.o
-- The CXX compiler identification is GNU 15.2.1
-- Detecting C compiler ABI info
[ 6%] Built target llama-llava-cli
[ 7%] Building C object ggml/src/CMakeFiles/ggml-base.dir/ggml-alloc.c.o
[ 8%] Linking CXX executable ../../bin/llama-minicpmv-cli
[ 8%] Built target llama-gemma3-cli
[ 8%] Building CXX object ggml/src/CMakeFiles/ggml-base.dir/ggml-backend.cpp.o
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/sbin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
[ 8%] Built target llama-minicpmv-cli
[ 9%] Building CXX object ggml/src/CMakeFiles/ggml-base.dir/ggml-opt.cpp.o
[ 9%] Linking CXX executable ../../bin/llama-qwen2vl-cli
[ 9%] Built target llama-qwen2vl-cli
[ 9%] Building CXX object ggml/src/CMakeFiles/ggml-base.dir/ggml-threading.cpp.o
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/sbin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
[ 9%] Building C object ggml/src/CMakeFiles/ggml-base.dir/ggml-quants.c.o
[ 10%] Building CXX object ggml/src/CMakeFiles/ggml-base.dir/gguf.cpp.o
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Success
-- Found Threads: TRUE
-- Enabling coopmat glslc support
-- Enabling coopmat2 glslc support
-- Enabling dot glslc support
-- Enabling bfloat16 glslc support
-- Configuring done (0.5s)
-- Generating done (0.0s)
-- Build files have been written to: /root/llama.cpp/build/ggml/src/ggml-vulkan/vulkan-shaders-gen-prefix/src/vulkan-shaders-gen-build
[ 11%] Performing build step for 'vulkan-shaders-gen'
[ 50%] Building CXX object CMakeFiles/vulkan-shaders-gen.dir/vulkan-shaders-gen.cpp.o
[100%] Linking CXX executable /root/llama.cpp/build/bin/vulkan-shaders-gen
[100%] Built target vulkan-shaders-gen
[ 11%] Built target vulkan-shaders-gen-build
[ 11%] Performing install step for 'vulkan-shaders-gen'
-- Install configuration: ""
[ 11%] Built target vulkan-shaders-gen-install
[ 11%] Completed 'vulkan-shaders-gen'
[ 11%] Built target vulkan-shaders-gen
[ 11%] Built target xxhash
[ 11%] Linking CXX shared library ../../bin/libggml-base.so
[ 11%] Built target ggml-base
[ 11%] Generate vulkan shaders
ggml_vulkan: Generating and compiling shaders to SPIR-V
[ 12%] Building C object ggml/src/CMakeFiles/ggml-cpu.dir/ggml-cpu/ggml-cpu.c.o
[ 12%] Building CXX object ggml/src/CMakeFiles/ggml-cpu.dir/ggml-cpu/repack.cpp.o
[ 13%] Building CXX object ggml/src/CMakeFiles/ggml-cpu.dir/ggml-cpu/hbm.cpp.o
[ 13%] Building CXX object ggml/src/CMakeFiles/ggml-cpu.dir/ggml-cpu/ggml-cpu.cpp.o
cc1: sorry, unimplemented: no support for ‘sme’ without ‘sve2’
cc1plus: sorry, unimplemented: no support for ‘sme’ without ‘sve2’
[ 13%] Building C object ggml/src/CMakeFiles/ggml-cpu.dir/ggml-cpu/quants.c.o
cc1plus: sorry, unimplemented: no support for ‘sme’ without ‘sve2’
cc1plus: sorry, unimplemented: no support for ‘sme’ without ‘sve2’
gmake[2]: *** [ggml/src/CMakeFiles/ggml-cpu.dir/build.make:79: ggml/src/CMakeFiles/ggml-cpu.dir/ggml-cpu/ggml-cpu.c.o] Error 1
gmake[2]: *** Waiting for unfinished jobs....
gmake[2]: *** [ggml/src/CMakeFiles/ggml-cpu.dir/build.make:93: ggml/src/CMakeFiles/ggml-cpu.dir/ggml-cpu/ggml-cpu.cpp.o] Error 1
gmake[2]: *** [ggml/src/CMakeFiles/ggml-cpu.dir/build.make:107: ggml/src/CMakeFiles/ggml-cpu.dir/ggml-cpu/repack.cpp.o] Error 1
gmake[2]: *** [ggml/src/CMakeFiles/ggml-cpu.dir/build.make:121: ggml/src/CMakeFiles/ggml-cpu.dir/ggml-cpu/hbm.cpp.o] Error 1
cc1: sorry, unimplemented: no support for ‘sme’ without ‘sve2’
gmake[2]: *** [ggml/src/CMakeFiles/ggml-cpu.dir/build.make:135: ggml/src/CMakeFiles/ggml-cpu.dir/ggml-cpu/quants.c.o] Error 1
gmake[1]: *** [CMakeFiles/Makefile2:2196: ggml/src/CMakeFiles/ggml-cpu.dir/all] Error 2
gmake[1]: *** Waiting for unfinished jobs....
Error executing command for flash_attn_f32_f16_q6_k_cm2: Failed to fork process
Error executing command for flash_attn_f32_f16_iq1_s_cm2: Failed to fork process
Error executing command for flash_attn_f32_f16_iq1_m_cm2: Failed to fork process
Error executing command for flash_attn_f32_f16_iq2_xxs_cm2: Failed to fork process
Error executing command for flash_attn_f32_f16_iq2_xs_cm2: Failed to fork process
Error executing command for flash_attn_f32_f16_iq2_s_cm2: Failed to fork process
Error executing command for flash_attn_f32_f16_iq3_xxs_cm2: Failed to fork process
Error executing command for flash_attn_f32_f16_iq3_s_cm2: Failed to fork process
Error executing command for flash_attn_f32_f16_iq4_xs_cm2: Failed to fork process
Error executing command for flash_attn_f32_f16_iq4_nl_cm2: Failed to fork process
Error executing command for flash_attn_f32_f16_f16_f16acc_cm2: Failed to fork process
Error executing command for flash_attn_f32_f16_f16_f16acc_cm1: Failed to fork process
Error executing command for flash_attn_f32_f16_f16_f16acc: Failed to fork process
Error executing command for flash_attn_f32_f16_q4_0_f16acc_cm2: Failed to fork process
Error executing command for flash_attn_f32_f16_q4_0_f16acc_cm1: Failed to fork process
Error executing command for flash_attn_f32_f16_q4_0_f16acc: Failed to fork process
Error executing command for flash_attn_f32_f16_q4_1_f16acc_cm2: Failed to fork process
Error executing command for flash_attn_f32_f16_q5_0_f16acc_cm2: Failed to fork process
Error executing command for flash_attn_f32_f16_q5_1_f16acc_cm2: Failed to fork process
Error executing command for flash_attn_f32_f16_q8_0_f16acc_cm2: Failed to fork process
Error executing command for flash_attn_f32_f16_q8_0_f16acc_cm1: Failed to fork process
Error executing command for flash_attn_f32_f16_q8_0_f16acc: Failed to fork process
Error executing command for flash_attn_f32_f16_q2_k_f16acc_cm2: Failed to fork process
Error executing command for flash_attn_f32_f16_q3_k_f16acc_cm2: Failed to fork process
Error executing command for flash_attn_f32_f16_q4_k_f16acc_cm2: Failed to fork process
Error executing command for flash_attn_f32_f16_q5_k_f16acc_cm2: Failed to fork process
Error executing command for flash_attn_f32_f16_q6_k_f16acc_cm2: Failed to fork process
Error executing command for flash_attn_f32_f16_iq1_s_f16acc_cm2: Failed to fork process
Error executing command for flash_attn_f32_f16_iq1_m_f16acc_cm2: Failed to fork process
Error executing command for flash_attn_f32_f16_iq2_xxs_f16acc_cm2: Failed to fork process
Error executing command for flash_attn_f32_f16_iq2_xs_f16acc_cm2: Failed to fork process
Error executing command for flash_attn_f32_f16_iq2_s_f16acc_cm2: Failed to fork process
Error executing command for flash_attn_f32_f16_iq3_xxs_f16acc_cm2: Failed to fork process
Error executing command for flash_attn_f32_f16_iq3_s_f16acc_cm2: Failed to fork process
Error executing command for flash_attn_f32_f16_iq4_xs_f16acc_cm2: Failed to fork process
Error executing command for flash_attn_f32_f16_iq4_nl_f16acc_cm2: Failed to fork process
Error executing command for mul_mat_vec_f32_f32_f32: Failed to fork process
Error executing command for mul_mat_vec_f32_f16_f32: Failed to fork process
Error executing command for mul_mat_vec_id_f32_f32: Failed to fork process
Error executing command for dequant_f32: Failed to fork process
Error executing command for get_rows_f32: Failed to fork process
Error executing command for get_rows_f32_f32: Failed to fork process
Error executing command for mul_mat_vec_f16_f32_f32: Failed to fork process
Error executing command for mul_mat_vec_f16_f16_f32: Failed to fork process
Error executing command for mul_mat_vec_id_f16_f32: Failed to fork process
Error executing command for get_rows_f16: Failed to fork process
Error executing command for get_rows_f16_f32: Failed to fork process
Error executing command for mul_mat_vec_q4_0_f32_f32: Failed to fork process
Error executing command for mul_mat_vec_q4_0_f16_f32: Failed to fork process
Error executing command for mul_mat_vec_id_q4_0_f32: Failed to fork process
Error executing command for dequant_q4_0: Failed to fork process
Error executing command for get_rows_q4_0: Failed to fork process
Error executing command for get_rows_q4_0_f32: Failed to fork process
Error executing command for mul_mat_vec_q4_1_f32_f32: Failed to fork process
Error executing command for mul_mat_vec_q4_1_f16_f32: Failed to fork process
Error executing command for mul_mat_vec_id_q4_1_f32: Failed to fork process
Error executing command for get_rows_q4_1: Failed to fork process
Error executing command for get_rows_q4_1_f32: Failed to fork process
Error executing command for mul_mat_vec_q5_0_f32_f32: Failed to fork process
Error executing command for mul_mat_vec_q5_0_f16_f32: Failed to fork process
Error executing command for mul_mat_vec_id_q5_0_f32: Failed to fork process
Error executing command for dequant_q5_0: Failed to fork process
Error executing command for get_rows_q5_0: Failed to fork process
Error executing command for get_rows_q5_0_f32: Failed to fork process
Error executing command for mul_mat_vec_q5_1_f32_f32: Failed to fork process
Error executing command for mul_mat_vec_q5_1_f16_f32: Failed to fork process
Error executing command for dequant_q4_1: Failed to fork process
Error executing command for dequant_q5_1: Failed to fork process
Error executing command for get_rows_q5_1: Failed to fork process
Error executing command for get_rows_q5_1_f32: Failed to fork process
Error executing command for mul_mat_vec_q8_0_f32_f32: Failed to fork process
Error executing command for mul_mat_vec_q8_0_f16_f32: Failed to fork process
Error executing command for mul_mat_vec_id_q8_0_f32: Failed to fork process
Error executing command for dequant_q8_0: Failed to fork process
Error executing command for get_rows_q8_0: Failed to fork process
Error executing command for get_rows_q8_0_f32: Failed to fork process
Error executing command for mul_mat_vec_q2_k_f32_f32: Failed to fork process
Error executing command for mul_mat_vec_q2_k_f16_f32: Failed to fork process
Error executing command for mul_mat_vec_id_q2_k_f32: Failed to fork process
Error executing command for dequant_q2_k: Failed to fork process
Error executing command for mul_mat_vec_q3_k_f32_f32: Failed to fork process
Error executing command for mul_mat_vec_q3_k_f16_f32: Failed to fork process
Error executing command for mul_mat_vec_id_q3_k_f32: Failed to fork process
Error executing command for dequant_q3_k: Failed to fork process
Error executing command for mul_mat_vec_q4_k_f32_f32: Failed to fork process
Error executing command for mul_mat_vec_q4_k_f16_f32: Failed to fork process
Error executing command for mul_mat_vec_id_q4_k_f32: Failed to fork process
Error executing command for dequant_q4_k: Failed to fork process
Error executing command for mul_mat_vec_q5_k_f32_f32: Failed to fork process
Error executing command for mul_mat_vec_q5_k_f16_f32: Failed to fork process
Error executing command for mul_mat_vec_id_q5_1_f32: Failed to fork process
Error executing command for dequant_q5_k: Failed to fork process
Error executing command for mul_mat_vec_q6_k_f32_f32: Failed to fork process
Error executing command for mul_mat_vec_q6_k_f16_f32: Failed to fork process
Error executing command for mul_mat_vec_id_q6_k_f32: Failed to fork process
Error executing command for dequant_q6_k: Failed to fork process
Error executing command for mul_mat_vec_iq1_s_f32_f32: Failed to fork process
Error executing command for mul_mat_vec_iq1_s_f16_f32: Failed to fork process
Error executing command for mul_mat_vec_id_iq1_s_f32: Failed to fork process
Error executing command for dequant_iq1_s: Failed to fork process
Error executing command for get_rows_iq1_s: Failed to fork process
Error executing command for get_rows_iq1_s_f32: Failed to fork process
Error executing command for mul_mat_vec_iq1_m_f32_f32: Failed to fork process
Error executing command for mul_mat_vec_iq1_m_f16_f32: Failed to fork process
Error executing command for mul_mat_vec_id_iq1_m_f32: Failed to fork process
Error executing command for dequant_iq1_m: Failed to fork process
Error executing command for get_rows_iq1_m: Failed to fork process
Error executing command for get_rows_iq1_m_f32: Failed to fork process
Error executing command for mul_mat_vec_iq2_xxs_f16_f32: Failed to fork process
Error executing command for mul_mat_vec_id_iq2_xxs_f32: Failed to fork process
Error executing command for mul_mat_vec_id_q5_k_f32: Failed to fork process
Error executing command for dequant_iq2_xxs: Failed to fork process
Error executing command for get_rows_iq2_xxs_f32: Failed to fork process
Error executing command for mul_mat_vec_iq2_xs_f32_f32: Failed to fork process
Error executing command for mul_mat_vec_iq2_xs_f16_f32: Failed to fork process
Error executing command for mul_mat_vec_id_iq2_xs_f32: Failed to fork process
Error executing command for dequant_iq2_xs: Failed to fork process
Error executing command for get_rows_iq2_xs: Failed to fork process
Error executing command for get_rows_iq2_xs_f32: Failed to fork process
Error executing command for get_rows_iq2_xxs: Failed to fork process
Error executing command for mul_mat_vec_iq2_s_f32_f32: Failed to fork process
Error executing command for mul_mat_vec_iq2_s_f16_f32: Failed to fork process
Error executing command for mul_mat_vec_iq2_xxs_f32_f32: Failed to fork process
Error executing command for dequant_iq2_s: Failed to fork process
Error executing command for mul_mat_vec_iq3_xxs_f32_f32: Failed to fork process
Error executing command for mul_mat_vec_iq3_xxs_f16_f32: Failed to fork process
Error executing command for mul_mat_vec_id_iq2_s_f32: Failed to fork process
Error executing command for mul_mat_vec_id_iq3_xxs_f32: Failed to fork process
Error executing command for get_rows_iq3_xxs: Failed to fork process
Error executing command for get_rows_iq3_xxs_f32: Failed to fork process
Error executing command for mul_mat_vec_iq3_s_f32_f32: Failed to fork process
Error executing command for mul_mat_vec_iq3_s_f16_f32: Failed to fork process
Error executing command for dequant_iq3_xxs: Failed to fork process
Error executing command for mul_mat_vec_id_iq3_s_f32: Failed to fork process
Error executing command for dequant_iq3_s: Failed to fork process
Error executing command for get_rows_iq3_s: Failed to fork process
Error executing command for get_rows_iq3_s_f32: Failed to fork process
Error executing command for mul_mat_vec_iq4_xs_f32_f32: Failed to fork process
Error executing command for mul_mat_vec_id_iq4_xs_f32: Failed to fork process
Error executing command for mul_mat_vec_iq4_xs_f16_f32: Failed to fork process
Error executing command for get_rows_iq4_xs: Failed to fork process
Error executing command for dequant_iq4_xs: Failed to fork process
Error executing command for get_rows_iq4_xs_f32: Failed to fork process
Error executing command for mul_mat_vec_iq4_nl_f16_f32: Failed to fork process
Error executing command for mul_mat_vec_iq4_nl_f32_f32: Failed to fork process
Error executing command for mul_mat_vec_id_iq4_nl_f32: Failed to fork process
Error executing command for get_rows_iq4_nl: Failed to fork process
Error executing command for dequant_iq4_nl: Failed to fork process
Error executing command for get_rows_iq4_nl_f32: Failed to fork process
Error executing command for mul_mat_vec_bf16_f16_f32: Failed to fork process
Error executing command for mul_mat_vec_bf16_f32_f32: Failed to fork process
Error executing command for mul_mat_vec_id_bf16_f32: Failed to fork process
Error executing command for get_rows_bf16_f32: Failed to fork process
Error executing command for get_rows_bf16: Failed to fork process
Error executing command for mul_mat_vec_p021_f16_f32_subgroup_add: Failed to fork process
Error executing command for mul_mat_vec_nc_f16_f32: Failed to fork process
Error executing command for norm_f32: Failed to fork process
Error executing command for group_norm_f32: Failed to fork process
Error executing command for mul_mat_vec_p021_f16_f32: Failed to fork process
Error executing command for rms_norm_f32: Failed to fork process
Error executing command for l2_norm_f32: Failed to fork process
Error executing command for rms_norm_back_f32: Failed to fork process
Error executing command for cpy_f32_f16: Failed to fork process
Error executing command for cpy_f16_f16: Failed to fork process
Error executing command for cpy_f16_f32: Failed to fork process
Error executing command for cpy_f32_bf16: Failed to fork process
Error executing command for contig_cpy_f32_f32: Failed to fork process
Error executing command for contig_cpy_f32_f16: Failed to fork process
Error executing command for contig_cpy_f16_f16: Failed to fork process
Error executing command for contig_cpy_f16_f32: Failed to fork process
Error executing command for cpy_f32_f32: Failed to fork process
Error executing command for contig_cpy_f32_bf16: Failed to fork process
Error executing command for cpy_f32_q4_0_rte: Failed to fork process
Error executing command for cpy_f32_q4_0: Failed to fork process
Error executing command for cpy_q4_0_f32: Failed to fork process
Error executing command for cpy_f32_q4_1: Failed to fork process
Error executing command for get_rows_iq2_s_f32: Failed to fork process
Error executing command for cpy_f32_q5_0: Failed to fork process
Error executing command for cpy_f32_q5_0_rte: Failed to fork process
Error executing command for cpy_q5_0_f32: Failed to fork process
Error executing command for cpy_f32_q5_1: Failed to fork process
Error executing command for cpy_f32_q5_1_rte: Failed to fork process
Error executing command for cpy_q5_1_f32: Failed to fork process
Error executing command for cpy_f32_q8_0: Failed to fork process
Error executing command for cpy_f32_q8_0_rte: Failed to fork process
Error executing command for cpy_q8_0_f32: Failed to fork process
Error executing command for cpy_f32_iq4_nl: Failed to fork process
Error executing command for get_rows_iq2_s: Failed to fork process
Error executing command for cpy_f32_iq4_nl_rte: Failed to fork process
Error executing command for add_f32_f32_f32: Failed to fork process
Error executing command for add_f32_f32_f16: Failed to fork process
Error executing command for add_f32_f16_f32: Failed to fork process
Error executing command for add_f32_f16_f16: Failed to fork process
Error executing command for add_f16_f32_f32: Failed to fork process
Error executing command for add_f16_f32_f16: Failed to fork process
Error executing command for add_f16_f16_f32: Failed to fork process
Error executing command for add_f16_f16_f16: Failed to fork process
Error executing command for sub_f32_f32_f32: Failed to fork process
Error executing command for sub_f32_f32_f16: Failed to fork process
Error executing command for sub_f32_f16_f32: Failed to fork process
Error executing command for cpy_q4_1_f32: Failed to fork process
Error executing command for cpy_f32_q4_1_rte: Failed to fork process
Error executing command for sub_f16_f32_f32: Failed to fork process
Error executing command for sub_f16_f32_f16: Failed to fork process
Error executing command for sub_f16_f16_f16: Failed to fork process
Error executing command for sub_f16_f16_f32: Failed to fork process
Error executing command for mul_f32_f32_f32: Failed to fork process
Error executing command for mul_f32_f16_f32: Failed to fork process
Error executing command for mul_f32_f16_f16: Failed to fork process
Error executing command for mul_f16_f32_f32: Failed to fork process
Error executing command for mul_f16_f32_f16: Failed to fork process
Error executing command for mul_f16_f16_f32: Failed to fork process
Error executing command for cpy_iq4_nl_f32: Failed to fork process
Error executing command for mul_f16_f16_f16: Failed to fork process
Error executing command for div_f32_f32_f32: Failed to fork process
Error executing command for div_f32_f16_f32: Failed to fork process
Error executing command for div_f32_f16_f16: Failed to fork process
Error executing command for div_f16_f32_f32: Failed to fork process
Error executing command for div_f16_f32_f16: Failed to fork process
Error executing command for div_f16_f16_f32: Failed to fork process
Error executing command for div_f16_f16_f16: Failed to fork process
Error executing command for sub_f32: Failed to fork process
Error executing command for acc_f32: Failed to fork process
Error executing command for split_k_reduce: Failed to fork process
Error executing command for fa_split_k_reduce: Failed to fork process
Error executing command for quantize_q8_1: Failed to fork process
Error executing command for sub_f32_f16_f16: Failed to fork process
Error executing command for mul_f32_f32_f16: Failed to fork process
Error executing command for div_f32: Failed to fork process
Error executing command for repeat_f32: Failed to fork process
Error executing command for scale_f32: Failed to fork process
Error executing command for sqr_f32: Failed to fork process
Error executing command for sin_f32: Failed to fork process
Error executing command for mul_f32: Failed to fork process
Error executing command for repeat_back_f32: Failed to fork process
Error executing command for clamp_f32: Failed to fork process
Error executing command for cos_f32: Failed to fork process
Error executing command for pad_f32: Failed to fork process
Error executing command for concat_f32: Failed to fork process
Error executing command for div_f32_f32_f16: Failed to fork process
Error executing command for concat_f16: Failed to fork process
Error executing command for upscale_f32: Failed to fork process
Error executing command for concat_i32: Failed to fork process
Error executing command for gelu_f16: Failed to fork process
Error executing command for gelu_quick_f16: Failed to fork process
Error executing command for silu_f32: Failed to fork process
Error executing command for relu_f16: Failed to create pipes
Error executing command for relu_f32: Failed to create pipes
Error executing command for tanh_f16: Failed to create pipes
Error executing command for tanh_f32: Failed to create pipes
Error executing command for sigmoid_f16: Failed to create pipes
Error executing command for sigmoid_f32: Failed to create pipes
Error executing command for leaky_relu_f32: Failed to create pipes
Error executing command for silu_back_f32: Failed to create pipes
Error executing command for diag_mask_inf_f32: Failed to create pipes
Error executing command for soft_max_f32: Failed to create pipes
Error executing command for soft_max_f32_f16: Failed to create pipes
Error executing command for soft_max_back_f32: Failed to create pipes
Error executing command for gelu_quick_f32: Failed to fork process
Error executing command for gelu_f32: Failed to fork process
Error executing command for silu_f16: Failed to fork process
Error executing command for rope_neox_f32: Failed to create pipes
Error executing command for rope_norm_f32: Failed to create pipes
Error executing command for rope_neox_f16_rte: Failed to create pipes
Error executing command for rope_neox_f16: Failed to create pipes
Error executing command for rope_multi_f16: Failed to create pipes
Error executing command for rope_multi_f16_rte: Failed to create pipes
Error executing command for rope_vision_f32: Failed to create pipes
Error executing command for rope_vision_f16: Failed to create pipes
Error executing command for rope_vision_f16_rte: Failed to create pipes
Error executing command for argsort_f32: Failed to create pipes
Error executing command for rope_norm_f16: Failed to create pipes
Error executing command for argmax_f32: Failed to create pipes
Error executing command for sum_rows_f32: Failed to create pipes
Error executing command for count_equal_i32: Failed to create pipes
Error executing command for rope_norm_f16_rte: Failed to create pipes
Error executing command for im2col_f32: Failed to create pipes
Error executing command for im2col_f32_f16_rte: Failed to create pipes
Error executing command for timestep_embedding_f32: Failed to create pipes
Error executing command for conv_transpose_1d_f32: Failed to create pipes
Error executing command for rwkv_wkv6_f32: Failed to create pipes
Error executing command for rwkv_wkv7_f32: Failed to create pipes
Error executing command for opt_step_adamw_f32: Failed to create pipes
Error executing command for conv2d_dw_whcn_f32: Failed to create pipes
Error executing command for rope_multi_f32: Failed to create pipes
Error executing command for pool2d_f32: Failed to create pipes
Error executing command for conv2d_dw_cwhn_f32: Failed to create pipes
Error executing command for im2col_f32_f16: Failed to create pipes
[ 13%] Building CXX object ggml/src/ggml-vulkan/CMakeFiles/ggml-vulkan.dir/ggml-vulkan.cpp.o
[ 14%] Building CXX object ggml/src/ggml-vulkan/CMakeFiles/ggml-vulkan.dir/ggml-vulkan-shaders.cpp.o
/root/llama.cpp/ggml/src/ggml-vulkan/ggml-vulkan.cpp: In function ‘void ggml_vk_load_shaders(vk_device&)’:
/root/llama.cpp/ggml/src/ggml-vulkan/ggml-vulkan.cpp:2014:181: error: ‘flash_attn_f32_f16_f16_f16acc_len’ was not declared in this scope; did you mean ‘flash_attn_f32_f16_f16_cm2_len’?
2014 | f16_D ## D ## SUFFIX[TYPE][0][0][0], "flash_attn_f32_f16_D" #D "_f16acc" #NAMELC #SUFFIX, flash_attn_f32_f16_ ## NAMELC ## _f16acc ## SUFFIX ## _len, flash_attn_f32_f16_ ## NAMELC ## _f16acc ## SUFFIX ## _data, "main", 5, sizeof(vk_flash_attn_push_constants), fa_wg_denoms(FAPATH, D,1,TYPE,false), fa_spec_constants(FAPATH, D,1,TYPE,false), 1, true, FAPATH==FA_COOPMAT1, (FAPATH==FA_COOPMAT1 ? 32 : 0)); \
| ^~~~~~~~~~~~~~~~~~~
/root/llama.cpp/ggml/src/ggml-vulkan/ggml-vulkan.cpp:2024:9: note: in expansion of macro ‘CREATE_FA2’
2024 | CREATE_FA2(TYPE, NAMELC, FAPATH, SUFFIX, 64) \
| ^~~~~~~~~~
/root/llama.cpp/ggml/src/ggml-vulkan/ggml-vulkan.cpp:2031:5: note: in expansion of macro ‘CREATE_FA’
2031 | CREATE_FA(GGML_TYPE_F16, f16, FA_SCALAR, )
| ^~~~~~~~~
/root/llama.cpp/build/ggml/src/ggml-vulkan/ggml-vulkan-shaders.cpp:1783463:39: error: ‘add_f32_f32_f32_data’ was not declared in this scope; did you mean ‘matmul_f32_f32_fp32_data’?
1783463 | unsigned char *add_data[2][2][2] = {{{add_f32_f32_f32_data, add_f32_f32_f16_data}, {add_f32_f16_f32_data, add_f32_f16_f16_data}}, {{add_f16_f32_f32_data, add_f16_f32_f16_data}, {add_f16_f16_f32_data, add_f16_f16_f16_data}}};
| ^~~~~~~~~~~~~~~~~~~~
| matmul_f32_f32_fp32_data
[...]
There was a problem hiding this comment.
This has also happened with me as well! Try re-running the script
There was a problem hiding this comment.
But make sure you remove the failed llama.cpp directory first
There was a problem hiding this comment.
This has also happened with me as well! Try re-running the script
Did it work?
There was a problem hiding this comment.
Retried, no success.
Sorry for going back and forth, but probably this script is not matured well, and should be moved from the built-in code to a document, with a note that the script may not work depending on environment
There was a problem hiding this comment.
No worries, I'll do it in sometime!
There was a problem hiding this comment.
Pushed a commit krunkit: move install-vulkan-gpu.sh to docs
The script seems immature to be included in the driver package. The script didn't work for me on my MacBook Pro 2024 with Apple M4 Max: ``` cc1: sorry, unimplemented: no support for ‘sme’ without ‘sve2’ ``` Signed-off-by: Akihiro Suda <[email protected]>
AkihiroSuda
force-pushed
the
fix/krunkit-gpu
branch
from
9db83f3 to
618d1ec
Compare
AkihiroSuda
approved these changes
Nov 4, 2025
|
Really thanks! I will try to reproduce this issue and come up with a fix later but for now it's ok to have the script in docs🙏🏻 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR changes the requirement of Fedora to run AI models inside the Krunkit VM. Now AI models can be run inside the VM using containers on any linux distro or without containers using Fedora as the distro.
Now VMs can be launched directly using:
limactl start default --vm-type=krunkitOr if without containers:
limactl start --vm-type=krunkit template://fedora