Running on iGPUs with Vulkan can be fun - llama.cpp OOM error debugging

[geerlingguy]

My current goal is to run DeepSeek R1 671B on a 4-node cluster of servers with 128 GB of shared RAM on each system (with an iGPU that works with Vulkan).

The problem is it seems I can only use up to like 43 GB of space on each node (of the 128 GB). I'm guessing it's a limitation of shared VRAM, to prevent resource contention? Not sure. In the past all my llama.cpp work has involved dedicated GPUs, and either only VRAM or shared VRAM + system memory.

Running llama.cpp with rpc on three nodes but not main node:

load_tensors: loading model tensors, this can take a while... (mmap = false)
load_tensors: offloading 61 repeating layers to GPU
load_tensors: offloading output layer to GPU
load_tensors: offloaded 62/62 layers to GPU
load_tensors:      Vulkan0 model buffer size = 96507.91 MiB
load_tensors: RPC[10.0.2.242:50052] model buffer size = 97323.94 MiB
load_tensors: RPC[10.0.2.223:50052] model buffer size = 103503.00 MiB
load_tensors: RPC[10.0.2.209:50052] model buffer size = 87857.67 MiB
load_tensors:          CPU model buffer size =   497.11 MiB
radv/amdgpu: Not enough memory for command submission.
llama_model_load: error loading model: vk::Queue::submit: ErrorDeviceLost
llama_model_load_from_file_impl: failed to load model
common_init_from_params: failed to load model 'models/DeepSeek-R1-Q4_K_M.gguf'
main: error: unable to load model

RPC on four nodes including main node results in OOM on the main node since llama.cpp is trying a local Vulkan runner plus the RPC runner:

print_info: max token length = 256
load_tensors: loading model tensors, this can take a while... (mmap = false)
Killed

Setting:

export GGML_VK_PREFER_HOST_MEMORY=1

Only on main node:

........................./opt/llama.cpp/ggml/src/ggml-rpc/ggml-rpc.cpp:579: Remote RPC server crashed or returned malformed response
[New LWP 11588]
[New LWP 11587]
[New LWP 11586]

This GDB supports auto-downloading debuginfo from the following URLs:
  <https://debuginfod.fedoraproject.org/>
Enable debuginfod for this session? (y or [n]) [answered N; input not from terminal]
Debuginfod has been disabled.
To make this setting permanent, add 'set debuginfod enabled off' to .gdbinit.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
0x00007f00fe8876c2 in __syscall_cancel_arch () from /lib64/libc.so.6
#0  0x00007f00fe8876c2 in __syscall_cancel_arch () from /lib64/libc.so.6
#1  0x00007f00fe87b9da in __internal_syscall_cancel () from /lib64/libc.so.6
#2  0x00007f00fe87ba24 in __syscall_cancel () from /lib64/libc.so.6
#3  0x00007f00fe8eb5ef in wait4 () from /lib64/libc.so.6
#4  0x00007f0100b6c263 in ggml_print_backtrace () from /opt/llama.cpp/build/bin/libggml-base.so
#5  0x00007f0100b6c3af in ggml_abort () from /opt/llama.cpp/build/bin/libggml-base.so
#6  0x00007f0100e33c1a in ggml_backend_rpc_buffer_set_tensor(ggml_backend_buffer*, ggml_tensor*, void const*, unsigned long, unsigned long) () from /opt/llama.cpp/build/bin/libggml-rpc.so
#7  0x00007f0100c8a234 in llama_model_loader::load_all_data(ggml_context*, std::unordered_map<unsigned int, ggml_backend_buffer*, std::hash<unsigned int>, std::equal_to<unsigned int>, std::allocator<std::pair<unsigned int const, ggml_backend_buffer*> > >&, std::vector<std::unique_ptr<llama_mlock, std::default_delete<llama_mlock> >, std::allocator<std::unique_ptr<llama_mlock, std::default_delete<llama_mlock> > > >*, bool (*)(float, void*), void*) () from /opt/llama.cpp/build/bin/libllama.so
#8  0x00007f0100ccdb21 in llama_model::load_tensors(llama_model_loader&) () from /opt/llama.cpp/build/bin/libllama.so
#9  0x00007f0100c1b26d in llama_model_load_from_file_impl(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&, llama_model_params) () from /opt/llama.cpp/build/bin/libllama.so
#10 0x00007f0100c1bffc in llama_model_load_from_file () from /opt/llama.cpp/build/bin/libllama.so
#11 0x000000000051399f in common_init_from_params(common_params&) ()
#12 0x000000000042464d in main ()
[Inferior 1 (process 11585) detached]
Aborted (core dumped

[geerlingguy]

Using GGML_VK_PREFER_HOST_MEMORY=1 on all nodes (set in the systemd settings file for llama-rpc results in:

print_info: max token length = 256
load_tensors: loading model tensors, this can take a while... (mmap = false)
load_tensors: offloading 61 repeating layers to GPU
load_tensors: offloading output layer to GPU
load_tensors: offloaded 62/62 layers to GPU
load_tensors:      Vulkan0 model buffer size = 96507.91 MiB
load_tensors: RPC[10.0.2.242:50052] model buffer size = 97323.94 MiB
load_tensors: RPC[10.0.2.223:50052] model buffer size = 103503.00 MiB
load_tensors: RPC[10.0.2.209:50052] model buffer size = 87857.67 MiB
load_tensors:          CPU model buffer size =   497.11 MiB
....................................................................................................
llama_context: constructing llama_context
llama_context: non-unified KV cache requires ggml_set_rows() - forcing unified KV cache
llama_context: n_seq_max     = 1
llama_context: n_ctx         = 4096
llama_context: n_ctx_per_seq = 4096
llama_context: n_batch       = 2048
llama_context: n_ubatch      = 512
llama_context: causal_attn   = 1
llama_context: flash_attn    = 0
llama_context: kv_unified    = true
llama_context: freq_base     = 10000.0
llama_context: freq_scale    = 0.025
llama_context: n_ctx_per_seq (4096) < n_ctx_train (163840) -- the full capacity of the model will not be utilized
llama_context: Vulkan_Host  output buffer size =     0.49 MiB
llama_kv_cache_unified:    Vulkan0 KV buffer size =  4480.00 MiB
radv/amdgpu: Not enough memory for command submission.
llama_init_from_model: failed to initialize the context: vk::Queue::submit: ErrorDeviceLost
common_init_from_params: failed to create context with model 'models/DeepSeek-R1-Q4_K_M.gguf'
main: error: unable to load model

[geerlingguy]

Running rpc-server without the environment variable:

$ ./build/bin/rpc-server --host 10.0.2.209 --port 50052 --cache
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = Radeon 8060S Graphics (RADV GFX1151) (radv) | uma: 1 | fp16: 1 | bf16: 0 | warp size: 64 | shared memory: 65536 | int dot: 1 | matrix cores: KHR_coopmat

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
WARNING: Host ('10.0.2.209') is != '127.0.0.1'
         Never expose the RPC server to an open network!
         This is an experimental feature and is not secure!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

create_backend: using Vulkan0 backend
Starting RPC server v2.0.0
  endpoint       : 10.0.2.209:50052
  local cache    : /home/jgeerling/.cache/llama.cpp/rpc/
  backend memory : 43034 MB

With GGML_VK_PREFER_HOST_MEMORY=1 I'm still seeing 43034 MB...

If I override with the setting --mem {{ ansible_memtotal_mb - 5120 }} in the rpc command, I see the following in the logs:

Aug 01 20:51:32 node-2 rpc-server[396129]: create_backend: using Vulkan0 backend
Aug 01 20:51:32 node-2 rpc-server[396129]: Starting RPC server v2.0.0
Aug 01 20:51:32 node-2 rpc-server[396129]:   endpoint       : 10.0.2.209:50052
Aug 01 20:51:32 node-2 rpc-server[396129]:   local cache    : /home/jgeerling/.cache/llama.cpp/rpc/
Aug 01 20:51:32 node-2 rpc-server[396129]:   backend memory : 122960 MB

However, running llama-cli on the three RPC hosts and the local host, I see:

$ build/bin/llama-cli -m models/DeepSeek-R1-Q4_K_M.gguf -p "Hello, my name is" -ngl 99 --no-mmap --rpc 10.0.2.209:50052,10.0.2.242:50052,10.0.2.223:50052
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = Radeon 8060S Graphics (RADV GFX1151) (radv) | uma: 1 | fp16: 1 | bf16: 0 | warp size: 64 | shared memory: 65536 | int dot: 1 | matrix cores: KHR_coopmat
build: 6060 (9c35706b) with cc (GCC) 15.1.1 20250521 (Red Hat 15.1.1-2) for x86_64-redhat-linux
main: llama backend init
main: load the model and apply lora adapter, if any
llama_model_load_from_file_impl: using device RPC[10.0.2.209:50052] (RPC[10.0.2.209:50052]) - 122960 MiB free
llama_model_load_from_file_impl: using device RPC[10.0.2.242:50052] (RPC[10.0.2.242:50052]) - 122959 MiB free
llama_model_load_from_file_impl: using device RPC[10.0.2.223:50052] (RPC[10.0.2.223:50052]) - 122960 MiB free
llama_model_load_from_file_impl: using device Vulkan0 (Radeon 8060S Graphics (RADV GFX1151)) - 43034 MiB free

So how do I override on local? When I just run llama-cli I see:

ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = Radeon 8060S Graphics (RADV GFX1151) (radv) | uma: 1 | fp16: 1 | bf16: 0 | warp size: 64 | shared memory: 65536 | int dot: 1 | matrix cores: KHR_coopmat

Radeontop is only showing 64026M GTT... so maybe a BIOS setting? How do I get at all 128 MB?

[geerlingguy]

It seems like on the rpc servers I'm able to use the full 125 GB of RAM, according to top and btop. And the llama-cli output indicates the three RPC servers are storing over 100 GiB each for model buffers:

load_tensors: loading model tensors, this can take a while... (mmap = false)
load_tensors: offloading 61 repeating layers to GPU
load_tensors: offloading output layer to GPU
load_tensors: offloaded 62/62 layers to GPU
load_tensors:      Vulkan0 model buffer size = 36258.32 MiB
load_tensors: RPC[10.0.2.242:50052] model buffer size = 116788.73 MiB
load_tensors: RPC[10.0.2.223:50052] model buffer size = 124823.01 MiB
load_tensors: RPC[10.0.2.209:50052] model buffer size = 107322.46 MiB
load_tensors:          CPU model buffer size =   497.11 MiB
.............................

The loading model tensors, this can take a while... task does take a long while—and it seems like on the main node it's running on, it hits swap sometimes? I noticed the oomd getting some CPU time.

Eventually the main node also hits 128 GB of total memory utilization. So I'm wondering how to force llama-cli to break up the model appropriately among the nodes? Waiting for another run to complete... but it will probably fail. If it does, I'll paste the output below (starting with the ...... line above):

......................................................./opt/llama.cpp/ggml/src/ggml-rpc/ggml-rpc.cpp:566: Remote RPC server crashed or returned malformed response
[New LWP 7260]
[New LWP 7259]
[New LWP 7258]

This GDB supports auto-downloading debuginfo from the following URLs:
  <https://debuginfod.fedoraproject.org/>
Enable debuginfod for this session? (y or [n]) [answered N; input not from terminal]
Debuginfod has been disabled.
To make this setting permanent, add 'set debuginfod enabled off' to .gdbinit.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
0x00007f70c41996c2 in __syscall_cancel_arch () from /lib64/libc.so.6
#0  0x00007f70c41996c2 in __syscall_cancel_arch () from /lib64/libc.so.6
#1  0x00007f70c418d9da in __internal_syscall_cancel () from /lib64/libc.so.6
#2  0x00007f70c418da24 in __syscall_cancel () from /lib64/libc.so.6
#3  0x00007f70c41fd5ef in wait4 () from /lib64/libc.so.6
#4  0x00007f70c68a7263 in ggml_print_backtrace () from /opt/llama.cpp/build/bin/libggml-base.so
#5  0x00007f70c68a73af in ggml_abort () from /opt/llama.cpp/build/bin/libggml-base.so
#6  0x00007f70c693ec54 in ggml_backend_rpc_buffer_set_tensor(ggml_backend_buffer*, ggml_tensor*, void const*, unsigned long, unsigned long) () from /opt/llama.cpp/build/bin/libggml-rpc.so
#7  0x00007f70c668a234 in llama_model_loader::load_all_data(ggml_context*, std::unordered_map<unsigned int, ggml_backend_buffer*, std::hash<unsigned int>, std::equal_to<unsigned int>, std::allocator<std::pair<unsigned int const, ggml_backend_buffer*> > >&, std::vector<std::unique_ptr<llama_mlock, std::default_delete<llama_mlock> >, std::allocator<std::unique_ptr<llama_mlock, std::default_delete<llama_mlock> > > >*, bool (*)(float, void*), void*) () from /opt/llama.cpp/build/bin/libllama.so
#8  0x00007f70c66cdb21 in llama_model::load_tensors(llama_model_loader&) () from /opt/llama.cpp/build/bin/libllama.so
#9  0x00007f70c661b26d in llama_model_load_from_file_impl(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&, llama_model_params) () from /opt/llama.cpp/build/bin/libllama.so
#10 0x00007f70c661bffc in llama_model_load_from_file () from /opt/llama.cpp/build/bin/libllama.so
#11 0x000000000051399f in common_init_from_params(common_params&) ()
#12 0x000000000042464d in main ()
[Inferior 1 (process 7257) detached]
Aborted (core dumped)

Checking on the nodes, it looks like the llama-rpc service restarted on node 4. Checking on that node's logs:

Aug 01 21:31:52 node-4 rpc-server[5002]: Accepted client connection, free_mem=123564195840, total_mem=123564195840
Aug 01 21:45:12 node-4 systemd[1]: llama-rpc.service: A process of this unit has been killed by the OOM killer.
Aug 01 21:45:15 node-4 systemd[1]: llama-rpc.service: Main process exited, code=killed, status=9/KILL
Aug 01 21:45:15 node-4 systemd[1]: llama-rpc.service: Failed with result 'oom-kill'.
Aug 01 21:45:15 node-4 systemd[1]: llama-rpc.service: Consumed 3min 39.722s CPU time, 61.5G memory peak, 6.1G memory swap peak.
Aug 01 21:45:20 node-4 systemd[1]: llama-rpc.service: Scheduled restart job, restart counter is at 1.
Aug 01 21:45:20 node-4 systemd[1]: Started llama-rpc.service - "llama RPC Server".
...

Tried again, and it's node 4 again...

Aug 01 22:04:07 node-4 systemd[1]: llama-rpc.service: Failed with result 'oom-kill'.
Aug 01 22:04:07 node-4 systemd[1]: llama-rpc.service: Consumed 3min 35.921s CPU time, 55.9G memory peak, 7.8G memory swap peak.
Aug 01 22:04:12 node-4 systemd[1]: llama-rpc.service: Scheduled restart job, restart counter is at 2.

[geerlingguy]

Going to try with a slightly smaller model, Llama 3.1 405B Q4_K_M, which clocks in (file size) at 230 GB(ish).

...

load_tensors: loading model tensors, this can take a while... (mmap = false)
load_tensors: offloading 99 repeating layers to GPU
load_tensors: offloaded 99/127 layers to GPU
load_tensors:      Vulkan0 model buffer size = 19287.50 MiB
load_tensors: RPC[10.0.2.242:50052] model buffer size = 51561.25 MiB
load_tensors: RPC[10.0.2.223:50052] model buffer size = 54364.50 MiB
load_tensors: RPC[10.0.2.209:50052] model buffer size = 53490.00 MiB
load_tensors:  Vulkan_Host model buffer size = 51971.22 MiB
load_tensors:          CPU model buffer size =  1127.25 MiB
radv/amdgpu: Not enough memory for command submission.
llama_model_load: error loading model: vk::Queue::submit: ErrorDeviceLost
llama_model_load_from_file_impl: failed to load model
common_init_from_params: failed to load model 'models/Llama-3.1-405B-Q4_K_M.gguf'
main: error: unable to load model

Trying again with export GGML_VK_PREFER_HOST_MEMORY=1 before (oops, I should probably just set that globally...):

load_tensors: loading model tensors, this can take a while... (mmap = false)
load_tensors: offloading 99 repeating layers to GPU
load_tensors: offloaded 99/127 layers to GPU
load_tensors:      Vulkan0 model buffer size = 19287.50 MiB
load_tensors: RPC[10.0.2.242:50052] model buffer size = 51561.25 MiB
load_tensors: RPC[10.0.2.223:50052] model buffer size = 54364.50 MiB
load_tensors: RPC[10.0.2.209:50052] model buffer size = 53490.00 MiB
load_tensors:          CPU model buffer size =  1127.25 MiB
load_tensors:  Vulkan_Host model buffer size = 51971.22 MiB
....................................................................................................
llama_context: constructing llama_context
llama_context: non-unified KV cache requires ggml_set_rows() - forcing unified KV cache
llama_context: n_seq_max     = 1
llama_context: n_ctx         = 4096
llama_context: n_ctx_per_seq = 4096
llama_context: n_batch       = 2048
llama_context: n_ubatch      = 512
llama_context: causal_attn   = 1
llama_context: flash_attn    = 0
llama_context: kv_unified    = true
llama_context: freq_base     = 500000.0
llama_context: freq_scale    = 1
llama_context: n_ctx_per_seq (4096) < n_ctx_train (131072) -- the full capacity of the model will not be utilized
llama_context:        CPU  output buffer size =     0.49 MiB
llama_kv_cache_unified:    Vulkan0 KV buffer size =   160.00 MiB
radv/amdgpu: Not enough memory for command submission.
llama_init_from_model: failed to initialize the context: vk::Queue::submit: ErrorDeviceLost
common_init_from_params: failed to create context with model 'models/Llama-3.1-405B-Q4_K_M.gguf'
main: error: unable to load model

[geerlingguy]

I also tried using --device CPU on the rpc servers in their startup options, but running the llama-cli commands it would still try to allocate on the local GPU via Vulkan and hit OOM all the time.

Server:

Aug 01 23:47:22 node-1 rpc-server[22998]: create_backend: using CPU backend

Main llama node:

$ build/bin/llama-cli -m models/Llama-3.1-405B-Q4_K_M.gguf -p "Hello, my name is" -ngl 99 --no-mmap --rpc 10.0.2.233:50052,10.0.2.209:50052,10.0.2.242:50052,10.0.2.223:50052
...
print_info: max token length = 256
load_tensors: loading model tensors, this can take a while... (mmap = false)
Killed

[geerlingguy]

From this guide on running LLMs on Strix Halo with RPC, I found it showed a more conservative RAM alotment per node, around 108 GB instead of 128 (using, in that example, -m 108000). I was giving 10 GB of padding, so I'll up it to 20.

Also mentioned this:

When running llama-cli by default it will add itself to nodes and you don’t have to run a separate RPC instance. When using llama-bench it does not and you should run an RPC server on the host machine as well.

So... I'm trying llama-bench now, CPU-only, and it's running and transferring data over the network. Going to see if that works any better, then switch RPC servers back to Vulkan instead of CPU if it does.

That person was also using HIP (C++ runtime for CUDA portability) instead of Vulkan, it seems... maybe HIP / ROCm works better? I was having some trouble getting all the stars to align for ROCm to work earlier.

[geerlingguy]

CPU-only seemed to work there, so now I've lowered the mem to 108000 like the guide I linked above, and Vulkan-based run is working, but quite slowly so far... it doesn't seem to be network overhead, either?

$ build/bin/llama-bench -m models/Llama-3.1-405B-Q4_K_M.gguf -n 128 -p 512 -pg 512,128 -ngl 99 -fa 1 -r 2 --rpc 10.0.2.233:50052,10.0.2.209:50052,10.0.2.242:50052,10.0.2.223:50052
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = Radeon 8060S Graphics (RADV GFX1151) (radv) | uma: 1 | fp16: 1 | bf16: 0 | warp size: 64 | shared memory: 65536 | int dot: 1 | matrix cores: KHR_coopmat
| model                          |       size |     params | backend    | ngl | fa |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | -: | --------------: | -------------------: |
| llama ?B Q4_K - Medium         | 226.37 GiB |   405.85 B | Vulkan,RPC |  99 |  1 |           pp512 |          3.99 ± 0.21 |
| llama ?B Q4_K - Medium         | 226.37 GiB |   405.85 B | Vulkan,RPC |  99 |  1 |           tg128 |          0.70 ± 0.00 |
| llama ?B Q4_K - Medium         | 226.37 GiB |   405.85 B | Vulkan,RPC |  99 |  1 |     pp512+tg128 |          2.17 ± 0.00 |

It seems like while it's running a benchmark, llama-bench on node 1 is consuming 400-800% CPU according to atop:

So perhaps something being run on CPU still? nvtop across all nodes only shows about 20% GPU utilization (with little CPU utilization) for the rpc-server command.

[geerlingguy]

It does the same thing all the time, it seems — uses just 20% of the GPU power on each node. Maybe something weird with the way the tensors are split?

It briefly spiked each node to 100% GPU utilization while prepping the last bit of the run.

[geerlingguy]

Trying again with DeepSeek R1, still getting core dumped. Going to lower memory limits again...

llama_kv_cache_unified:    Vulkan0 KV buffer size =   200.00 MiB
llama_kv_cache_unified: RPC[10.0.2.209:50052] KV buffer size =   600.00 MiB
/opt/llama.cpp/ggml/src/ggml-rpc/ggml-rpc.cpp:615: Remote RPC server crashed or returned malformed response
[New LWP 4890]
[New LWP 4889]

This GDB supports auto-downloading debuginfo from the following URLs:
  <https://debuginfod.fedoraproject.org/>
Enable debuginfod for this session? (y or [n]) [answered N; input not from terminal]
Debuginfod has been disabled.
To make this setting permanent, add 'set debuginfod enabled off' to .gdbinit.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
0x00007ffb657996c2 in __syscall_cancel_arch () from /lib64/libc.so.6
#0  0x00007ffb657996c2 in __syscall_cancel_arch () from /lib64/libc.so.6
#1  0x00007ffb6578d9da in __internal_syscall_cancel () from /lib64/libc.so.6
#2  0x00007ffb6578da24 in __syscall_cancel () from /lib64/libc.so.6
#3  0x00007ffb657fd5ef in wait4 () from /lib64/libc.so.6
#4  0x00007ffb67e85263 in ggml_print_backtrace () from /opt/llama.cpp/build/bin/libggml-base.so
#5  0x00007ffb67e853af in ggml_abort () from /opt/llama.cpp/build/bin/libggml-base.so
#6  0x00007ffb67f1c851 in ggml_backend_rpc_buffer_clear(ggml_backend_buffer*, unsigned char) () from /opt/llama.cpp/build/bin/libggml-rpc.so
#7  0x00007ffb67c7048c in llama_kv_cache_unified::llama_kv_cache_unified(llama_model const&, std::function<bool (int)>&&, ggml_type, ggml_type, bool, bool, bool, unsigned int, unsigned int, unsigned int, unsigned int, llama_swa_type) () from /opt/llama.cpp/build/bin/libllama.so
#8  0x00007ffb67c98d65 in llama_model::create_memory(llama_memory_params const&, llama_cparams&) const () from /opt/llama.cpp/build/bin/libllama.so
#9  0x00007ffb67c379b8 in llama_context::llama_context(llama_model const&, llama_context_params) () from /opt/llama.cpp/build/bin/libllama.so
#10 0x00007ffb67c38741 in llama_init_from_model () from /opt/llama.cpp/build/bin/libllama.so
#11 0x0000000000405a57 in main ()
[Inferior 1 (process 4888) detached]
Aborted (core dumped)

[geerlingguy]

Regarding Llama 3.1 405B, it seems very slow, even parallelized among the four nodes. I read through Llama.cpp RPC over Ethernet strangely slow, but monitoring Ethernet, I tried both at 2.5 and 5 Gbps wired among all the machines, and the throughput never passes 5-10 Mbps while doing inference.

atop was showing only a few hundred pps of Ethernet going between nodes:

The llama-bench process on the main node was between 100-800% CPU, so it seems either some inference is done on the CPU still, or it's spending a lot of time coordinating things among the other nodes on the cluster.

I may try out Exo or distributed-llama next, to see if it's a llama.cpp-specific issue.

There are moments when I see the iGPU being fully utilized, but it's only for a brief period, seemingly while spooling up inference. Once that starts, it just uses 20% or so of the GPU's capacity, according to nvtop:

[deftdawg]

This is random, so not sure if it will be helpful, but I've observed optimistic VRAM reporting...

Got an 8845hs (Aoostar WTR Max) w/ 96GB of RAM last week, VRAM allocation can be set in the BIOS via:
AMD CMS -> NBIO Common Options -> GFX Configuration

• Setting iGPU Configuration to:
  • UMA_GAME_OPTIMIZED -> 32 GB of VRAM
  • UMA_WORKSTATION_OPTIMIZED -> 42 GB of VRAM

(as reported by LM Studio and Steam)

In the case of the UMA_WORKSTATION_OPTIMIZED / 42 GB number both of these also report that I have 62 GB of RAM; this means there's ~6GB of over allocation happening (since 42+62 = 104GB).

LACT and CPU-X both report only 32GB of VRAM and 64GB of RAM... As does the kernel:

sudo cat /sys/class/drm/card*/device/mem_info_vram_total 2>/dev/null
34359738368

[geerlingguy]

Over in #4 for distributed-llama, I've been playing with GTT size, upping it to 108 GB via kernel parameters:

# To remove an arg: `--remove-args`
# Calculation: `([size in GB] * 1024 * 1024) / 4.096`
sudo grubby --update-kernel=ALL --args='amdttm.pages_limit=27648000'
sudo grubby --update-kernel=ALL --args='amdttm.page_pool_size=27648000'
sudo reboot

This seems to have worked, but Vulkan still sees 72 GB instead of 108 GB... see #4 (comment)