Skip to content

metal : attach residency sets to queue #17766

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 3 commits into
base: master
Choose a base branch
from
Open

metal : attach residency sets to queue #17766

wants to merge 3 commits into from
+125 −0

Conversation

@ggerganov
Copy link
Member

@ggerganov ggerganov commented Dec 4, 2025

cont #11427
ref #10119

So something changed in MacOS recently because the fix from #11427 no longer works - the memory wiring/unwiring (a.k.a. throttling) after 1 second of being idle is back. Maybe this happened with the update to MacOS Tahoe - not sure.

Here are the results on master:

make -j && ./bin/llama-idle -m ../models/llama-3.1-70b/ggml-model-f16.gguf
0.06.147.450 I ggml_metal_init: allocating
0.06.147.477 I ggml_metal_init: found device: Apple M2 Ultra
0.06.147.483 I ggml_metal_init: picking default device: Apple M2 Ultra
0.06.147.486 I ggml_metal_init: use fusion         = true
0.06.147.486 I ggml_metal_init: use concurrency    = true
0.06.147.487 I ggml_metal_init: use graph optimize = true
0.06.147.505 I llama_context:        CPU  output buffer size =     0.49 MiB
0.06.151.959 I llama_kv_cache:      Metal KV buffer size =   160.00 MiB
0.06.159.411 I llama_kv_cache: size =  160.00 MiB (   512 cells,  80 layers,  1/1 seqs), K (f16):   80.00 MiB, V (f16):   80.00 MiB
0.06.160.621 I llama_context: Flash Attention was auto, set to enabled
0.06.170.950 I llama_context:      Metal compute buffer size =   282.50 MiB
0.06.170.952 I llama_context:        CPU compute buffer size =    17.01 MiB
0.06.170.952 I llama_context: graph nodes  = 2487
0.06.170.952 I llama_context: graph splits = 2
  - decode time:   193.05 ms
  - decode time:   192.43 ms
  - decode time:   192.13 ms
  - decode time:   192.62 ms
  - decode time:   193.71 ms
  - decode time:   192.02 ms
  - decode time:   191.88 ms
  - decode time:   193.06 ms
  - decode time:   192.33 ms
  - decode time:   191.80 ms
iters:   10, pause:   200 ms, avg decode time:   192.50 +/- 0.61 ms
  - decode time:   192.67 ms
  - decode time:   192.49 ms
  - decode time:   191.15 ms
  - decode time:   192.62 ms
  - decode time:   192.98 ms
  - decode time:   191.24 ms
  - decode time:   191.12 ms
  - decode time:   191.50 ms
  - decode time:   191.59 ms
  - decode time:   192.20 ms
iters:   10, pause:   400 ms, avg decode time:   191.95 +/- 0.71 ms
  - decode time:   191.18 ms
  - decode time:   191.54 ms
  - decode time:   190.83 ms
  - decode time:   191.12 ms
  - decode time:   190.08 ms
  - decode time:   190.78 ms
  - decode time:   191.07 ms
  - decode time:   191.04 ms
  - decode time:   196.06 ms
  - decode time:   192.10 ms
iters:   10, pause:   600 ms, avg decode time:   191.58 +/- 1.66 ms
  - decode time:   192.49 ms
  - decode time:   191.13 ms
  - decode time:   191.46 ms
  - decode time:   191.73 ms
  - decode time:   191.92 ms
  - decode time:   193.68 ms
  - decode time:   192.04 ms
  - decode time:   191.72 ms
  - decode time:   192.06 ms
  - decode time:   191.81 ms
iters:   10, pause:   800 ms, avg decode time:   192.01 +/- 0.69 ms
  - decode time:   193.58 ms
  - decode time:   194.51 ms
  - decode time:   192.86 ms
  - decode time:   433.47 ms
  - decode time:   190.99 ms
  - decode time:   193.74 ms
  - decode time:   199.78 ms
  - decode time:   190.71 ms
  - decode time:   191.84 ms
  - decode time:   193.31 ms
iters:   10, pause:  1000 ms, avg decode time:   217.48 +/- 75.93 ms
  - decode time:   412.50 ms
  - decode time:   309.28 ms
  - decode time:   360.40 ms
  - decode time:   387.33 ms
  - decode time:   370.75 ms
  - decode time:   376.98 ms
  - decode time:  1377.47 ms
  - decode time:  1300.97 ms
  - decode time:  1298.22 ms
  - decode time:  1327.39 ms
iters:   10, pause:  1200 ms, avg decode time:   752.13 +/- 495.04 ms
  - decode time:  1067.61 ms
  - decode time:  1125.08 ms
  - decode time:  1119.94 ms
  - decode time:  1315.78 ms
  - decode time:  1087.12 ms
  - decode time:  1091.60 ms
  - decode time:  1087.27 ms
  - decode time:  1084.48 ms
  - decode time:  1087.62 ms
  - decode time:  1087.20 ms
iters:   10, pause:  1400 ms, avg decode time:  1115.37 +/- 72.44 ms
  - decode time:  1598.81 ms
  - decode time:  1659.83 ms
  - decode time:  1704.90 ms
  - decode time:  1576.85 ms
  - decode time:  1682.92 ms
  - decode time:  1660.09 ms
  - decode time:  1685.85 ms
  - decode time:  1616.58 ms
  - decode time:  1660.40 ms
  - decode time:  1683.09 ms
iters:   10, pause:  1600 ms, avg decode time:  1652.93 +/- 41.88 ms
  - decode time:  1451.54 ms
  - decode time:  1429.19 ms
  - decode time:  1448.97 ms
  - decode time:  1472.25 ms
  - decode time:  1437.39 ms
  - decode time:  1455.94 ms
  - decode time:  1419.29 ms
  - decode time:  1471.90 ms
  - decode time:  1430.76 ms
  - decode time:  1461.42 ms
iters:   10, pause:  1800 ms, avg decode time:  1447.86 +/- 18.27 ms
2.36.527.021 I ggml_metal_free: deallocating

And here are the results with this PR:

0.06.159.343 I ggml_metal_init: allocating
0.06.159.377 I ggml_metal_init: found device: Apple M2 Ultra
0.06.159.383 I ggml_metal_init: picking default device: Apple M2 Ultra
0.06.159.386 I ggml_metal_init: use fusion         = true
0.06.159.386 I ggml_metal_init: use concurrency    = true
0.06.159.387 I ggml_metal_init: use graph optimize = true
0.06.159.403 I llama_context:        CPU  output buffer size =     0.49 MiB
0.06.164.393 I llama_kv_cache:      Metal KV buffer size =   160.00 MiB
0.06.171.847 I llama_kv_cache: size =  160.00 MiB (   512 cells,  80 layers,  1/1 seqs), K (f16):   80.00 MiB, V (f16):   80.00 MiB
0.06.173.129 I llama_context: Flash Attention was auto, set to enabled
0.06.183.473 I llama_context:      Metal compute buffer size =   282.50 MiB
0.06.183.482 I llama_context:        CPU compute buffer size =    17.01 MiB
0.06.183.482 I llama_context: graph nodes  = 2487
0.06.183.484 I llama_context: graph splits = 2
  - decode time:   191.36 ms
  - decode time:   191.88 ms
  - decode time:   192.17 ms
  - decode time:   190.04 ms
  - decode time:   191.18 ms
  - decode time:   191.01 ms
  - decode time:   192.39 ms
  - decode time:   192.18 ms
  - decode time:   193.01 ms
  - decode time:   192.78 ms
iters:   10, pause:   200 ms, avg decode time:   191.80 +/- 0.90 ms
  - decode time:   191.12 ms
  - decode time:   191.23 ms
  - decode time:   191.20 ms
  - decode time:   192.33 ms
  - decode time:   190.71 ms
  - decode time:   192.59 ms
  - decode time:   192.66 ms
  - decode time:   192.14 ms
  - decode time:   190.35 ms
  - decode time:   191.60 ms
iters:   10, pause:   400 ms, avg decode time:   191.59 +/- 0.80 ms
  - decode time:   193.46 ms
  - decode time:   194.52 ms
  - decode time:   191.91 ms
  - decode time:   190.64 ms
  - decode time:   194.11 ms
  - decode time:   191.17 ms
  - decode time:   192.81 ms
  - decode time:   192.62 ms
  - decode time:   192.73 ms
  - decode time:   191.18 ms
iters:   10, pause:   600 ms, avg decode time:   192.51 +/- 1.29 ms
  - decode time:   199.77 ms
  - decode time:   195.51 ms
  - decode time:   192.30 ms
  - decode time:   190.35 ms
  - decode time:   193.33 ms
  - decode time:   190.30 ms
  - decode time:   193.18 ms
  - decode time:   192.25 ms
  - decode time:   192.48 ms
  - decode time:   193.67 ms
iters:   10, pause:   800 ms, avg decode time:   193.31 +/- 2.74 ms
  - decode time:   193.74 ms
  - decode time:   194.80 ms
  - decode time:   192.59 ms
  - decode time:   192.08 ms
  - decode time:   194.69 ms
  - decode time:   191.29 ms
  - decode time:   191.55 ms
  - decode time:   196.23 ms
  - decode time:   191.95 ms
  - decode time:   192.46 ms
iters:   10, pause:  1000 ms, avg decode time:   193.14 +/- 1.64 ms
  - decode time:   221.25 ms
  - decode time:   222.57 ms
  - decode time:   219.05 ms
  - decode time:   218.60 ms
  - decode time:   191.75 ms
  - decode time:   223.81 ms
  - decode time:   217.56 ms
  - decode time:   192.10 ms
  - decode time:   219.01 ms
  - decode time:   217.57 ms
iters:   10, pause:  1200 ms, avg decode time:   214.33 +/- 11.99 ms
  - decode time:   227.78 ms
  - decode time:   218.82 ms
  - decode time:   212.44 ms
  - decode time:   222.10 ms
  - decode time:   225.00 ms
  - decode time:   229.71 ms
  - decode time:   226.76 ms
  - decode time:   218.06 ms
  - decode time:   222.88 ms
  - decode time:   220.53 ms
iters:   10, pause:  1400 ms, avg decode time:   222.41 +/- 5.19 ms
  - decode time:   226.02 ms
  - decode time:   227.07 ms
  - decode time:   224.31 ms
  - decode time:   221.87 ms
  - decode time:   224.10 ms
  - decode time:   226.71 ms
  - decode time:   225.18 ms
  - decode time:   228.20 ms
  - decode time:   224.98 ms
  - decode time:   221.39 ms
iters:   10, pause:  1600 ms, avg decode time:   224.98 +/- 2.18 ms
  - decode time:   225.10 ms
  - decode time:   225.45 ms
  - decode time:   229.04 ms
  - decode time:   229.66 ms
  - decode time:   229.84 ms
  - decode time:   221.65 ms
  - decode time:   232.24 ms
  - decode time:   229.59 ms
  - decode time:   227.63 ms
  - decode time:   227.49 ms
iters:   10, pause:  1800 ms, avg decode time:   227.77 +/- 3.03 ms
1.55.608.188 I ggml_metal_free: deallocating

It seems that attaching the residency sets to the Metal queue mostly eliminates the unwiring of the memory. Although, every now and then, it still seems to occur - not sure if this was the case before on MacOS Sequoia.

@loci-dev loci-dev mentioned this pull request Dec 4, 2025
Open
@ggerganov
Copy link
Member Author

ggerganov commented Dec 4, 2025

Yes, definitely something happened with the new MacOS Tahoe. It keep unwiring the memory, even with the patch from this PR. The only way to force it to not unwire it is by running:

sudo sysctl iogpu.disable_wired_collector=1

But this is quite inconvenient. Bummer..

  • MacOS Sequoia 15.5 (no unwiring)

    image

  • MacOS Tahoe 26.1 (memory gets unwired 1 second after model usage)

    image

  • MacOS Tahoe 26.1 (with iogpu.disable_wired_collector=1 there is no unwiring)

    image

@github-actions github-actions bot added examples ggml changes relating to the ggml tensor library for machine learning Apple Metal https://en.wikipedia.org/wiki/Metal_(API) labels Dec 4, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

Apple Metal https://en.wikipedia.org/wiki/Metal_(API) examples ggml changes relating to the ggml tensor library for machine learning

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@ggerganov