You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
docker run -it --rm -v "$(pwd):/app:Z" --device /dev/dri/renderD128:/dev/dri/renderD128 --device /dev/dri/card1:/dev/dri/card1 llama-cpp-vulkan -m "/app/models/YOUR_MODEL_FILE" -p "Building a website can be done in 10 simple steps:" -n 400 -e -ngl 33
386
386
```
387
387
388
-
**Without docker**:
388
+
### For Linux users:
389
389
390
-
Firstly, you need to make sure you have installed [Vulkan SDK](https://vulkan.lunarg.com/doc/view/latest/linux/getting_started_ubuntu.html)
390
+
First, follow the the official [Getting Started with the Linux Tarball Vulkan SDK](https://vulkan.lunarg.com/doc/sdk/latest/linux/getting_started.html) guide.
391
391
392
-
For example, on Ubuntu 22.04 (jammy), use the command below:
392
+
> [!IMPORTANT]
393
+
> After completing the first step, ensure that you have used the `source` command on the `setup_env.sh` file inside of the Vulkan SDK in your current terminal session. Otherwise, the build won't work. Additionally, if you close out of your terminal, you must perform this step again if you intend to perform a build. However, there are ways to make this persistent. Refer to the Vulkan SDK guide linked in the first step for more information about any of this.
393
394
395
+
Second, after verifying that you have done everything in the Vulkan SDK guide provided in the first step, run the following command to verify that everything is set up correctly:
# To verify the installation, use the command below:
400
397
vulkaninfo
401
398
```
402
399
403
-
Alternatively your package manager might be able to provide the appropriate libraries.
404
-
For example for Ubuntu 22.04 you can install `libvulkan-dev` instead.
405
-
For Fedora 40, you can install `vulkan-devel`, `glslc` and `glslang` packages.
406
-
407
-
Then, build llama.cpp using the cmake command below:
408
-
400
+
Then, assuming you have `cd` into your llama.cpp folder and there are no errors with running `vulkaninfo`, you can proceed to build llama.cpp using the CMake commands below:
409
401
```bash
410
402
cmake -B build -DGGML_VULKAN=1
411
403
cmake --build build --config Release
404
+
```
405
+
406
+
Finally, after finishing your build, you should be able to do this:
407
+
```bash
412
408
# Test the output binary (with "-ngl 33" to offload all layers to GPU)
413
-
./bin/llama-cli -m "PATH_TO_MODEL" -p "Hi you how are you" -n 50 -e -ngl 33 -t 4
409
+
./build/bin/llama-cli -m "PATH_TO_MODEL" -p "Hi you how are you" -n 50 -e -ngl 33 -t 4
414
410
415
411
# You should see in the output, ggml_vulkan detected your GPU. For example:
0 commit comments