[Feature Request]: Mooncake Store Speed up large-size segment mounting

[xiaguan]

Describe your feature request

We've noticed that mounting very large memory segments (e.g., at the TB scale) can take several minutes, which hinders practical deployment.

The overhead mainly comes from two parts:

• Actual physical allocation, where memset (or equivalent) might benefit from parallelization to speed it up.
  Registration with RDMA — this could be accelerated by using pre-allocated large pages (e.g., 2MB or 1GB page sizes), which might also improve runtime performance.

• Using large pages and parallel init could significantly reduce mounting time for such large segments.

Suggestions :

• Consider supporting libhugetlbfs or explicit mmap with hugepage hints.
• Maybe add a config flag to enable parallel zero-initialization for big segments.

Also, if we’re adding more environment variables, it feels like we’re piling up too many already — makes the user experience pretty clunky. Maybe it’s time we support a builder pattern in the pyclient for more readable and maintainable config construction?

Before submitting a new issue...

• Make sure you already searched for relevant issues and read the documentation

[staryxchen]

Hi @xiaguan
We encountered the same issue previously and implemented some optimizations. Test results show that the time required to register 500GB of memory has been reduced from over 87 seconds to just over 11 seconds (assuming memory is pre-allocated). I will submit a PR later.

[staryxchen]

Hi @xiaguan We encountered the same issue previously and implemented some optimizations. Test results show that the time required to register 500GB of memory has been reduced from over 87 seconds to just over 11 seconds (assuming memory is pre-allocated). I will submit a PR later.

Hi @xiaguan @stmatengss
I have submitted the PR #855 , PTAL

[xiaguan]

• For large segments (e.g., 32 GiB), doing a read-only “touch per page” right after allocation (read 1 byte per page) builds the page tables without allocating physical pages, eliminating
  first‑touch minor faults for later reads.
• This “touch-read” warmup is much faster than write‑based warmup because it doesn’t dirty pages:
  • touch-read: 16 threads → ~2.33 s (13.7 GB/s), 48 threads → ~1.28 s (25.1 GB/s)
  • touch-per-page (write 1 byte/page, allocates pages): ~2.76–3.89 s (8.2–11.6 GB/s)
  • memset 32 GiB: ~3.01–4.65 s (6.9–10.6 GB/s); with MAP_POPULATE the cost shifts to mmap and memset rises to ~15.2 GB/s

[staryxchen]

• For large segments (e.g., 32 GiB), doing a read-only “touch per page” right after allocation (read 1 byte per page) builds the page tables without allocating physical pages, eliminating
  first‑touch minor faults for later reads.

• This “touch-read” warmup is much faster than write‑based warmup because it doesn’t dirty pages:

  • touch-read: 16 threads → ~2.33 s (13.7 GB/s), 48 threads → ~1.28 s (25.1 GB/s)
  • touch-per-page (write 1 byte/page, allocates pages): ~2.76–3.89 s (8.2–11.6 GB/s)
  • memset 32 GiB: ~3.01–4.65 s (6.9–10.6 GB/s); with MAP_POPULATE the cost shifts to mmap and memset rises to ~15.2 GB/s

Perhaps we could touch-read all memory before calling ibv_reg_mr in the register_memory function? However, I'm concerned that when memory is insufficient, this might cause the process to be killed due to OOM. Since the stack belongs to functions within the Mooncake Transfer Engine, this could lead to some misunderstandings.

[xiaguan]

Sure, we could do it at the store level first —we’re allocating a ton of memory for the storage service anyway.

[staryxchen]

Sure, we could do it at the store level first —we’re allocating a ton of memory for the storage service anyway.

After further consideration, I believe implicit pre-allocated memory within mooncake remains feasible, as ibv_reg_mr also pre-allocate memory. Since not all applications preallocate memory, enabling parallel registration without prior allocation could actually degrade performance.

[ykwd]

I made a series of benchmarks, the results of which may give some insights into how to speed up the mounting.

In all the benchmarks:

• The page size is 4KB
• The mmap, which is used to trigger page fault when allocating the memory, is revoked as the following code:

void* ptr = mmap(nullptr, size, PROT_READ | PROT_WRITE,
                 MAP_PRIVATE | MAP_ANONYMOUS | MAP_POPULATE, -1, 0);

First, register 4 GB segment:

No pre-touch:
register time taken: 2550 ms

Pre-touch with 1 thread:
Touch time taken: 1884 ms
register time taken: 1562 ms

Pre-touch with 4 threads:
Touch time taken: 1869 ms
register time taken: 1373 ms

mmap
Alloc time taken: 1261 ms
register time taken: 1463 ms

Next, register 40 GB segment:

No pre-touch:
register time taken: 21825 ms

Pre-touch with 1 thread:
Touch time taken: 16318 ms
register time taken: 12898 ms

Pre-touch with 4 threads:
Touch time taken: 4996 ms
register time taken: 13565 ms

mmap:
Alloc time taken: 11287 ms
register time taken: 12672 ms

Register 160 GB segment:

No pre-touch:
register time taken: 76314 ms

Pre-touch with 1 thread:
Touch time taken: 64724 ms
register time taken: 35149 ms

Pre-touch with 4 threads:
Touch time taken: 23619 ms
register time taken: 36703 ms

Pre-touch with 16 threads:
Touch time taken: 17830 ms
register time taken: 36892 ms

mmap:
Alloc time taken: 44055 ms
register time taken: 35009 ms

From the above data, we can observe:

• With only a single thread, the mmap method is faster than pre-touching, since pre-touching repeatedly switches between user mode and kernel mode. In fact, in this case, directly registering the memory without either mmap or pre-touching is the fastest approach.
• When the segment is sufficiently large, multi-threaded pre-touching becomes faster, as it can fully utilize the available memory bandwidth.
• However, using too many threads does not improve performance, since the process becomes memory-bandwidth bound once a certain number of threads is reached.

Conclusion:

• Skip pre-touching when the segment size is small or when only a few threads are available.
• Otherwise, use a fixed number of threads for pre-touching.

[ykwd]

The benchmark of RdmaTransport::registerLocalMemory is conducted on a server with 8 mlx5 RDMA NICs.

The register memory region time is recorded for each NIC.

Register 4 GB segment:

Without pre-touch
Register memory region time taken: 1307 ms
Register memory region time taken: 107 ms
Register memory region time taken: 106 ms
Register memory region time taken: 105 ms
Register memory region time taken: 103 ms
Register memory region time taken: 110 ms
Register memory region time taken: 105 ms
Register memory region time taken: 107 ms
Get memory location time taken: 216 ms
Add local memory buffer time taken: 1 ms
Total time taken: 2273 ms

With pre-touch:
Register memory region time taken: 161 ms
Register memory region time taken: 159 ms
Register memory region time taken: 159 ms
Register memory region time taken: 158 ms
Register memory region time taken: 159 ms
Register memory region time taken: 161 ms
Register memory region time taken: 157 ms
Register memory region time taken: 155 ms
Get memory location time taken: 265 ms
Add local memory buffer time taken: 1 ms
Total time taken: 1541 ms

Without pre-touch, parallel register:
Register memory region time taken: 7645 ms
Register memory region time taken: 7645 ms
Register memory region time taken: 7645 ms
Register memory region time taken: 7645 ms
Register memory region time taken: 7643 ms
Register memory region time taken: 7647 ms
Register memory region time taken: 7644 ms
Register memory region time taken: 7646 ms
Get memory location time taken: 266 ms
Add local memory buffer time taken: 1 ms
Total time taken: 7925 ms

With pre-touch, parallel register:
Register memory region time taken: 202 ms
Register memory region time taken: 202 ms
Register memory region time taken: 202 ms
Register memory region time taken: 202 ms
Register memory region time taken: 202 ms
Register memory region time taken: 202 ms
Register memory region time taken: 202 ms
Register memory region time taken: 202 ms
Get memory location time taken: 228 ms
Add local memory buffer time taken: 2 ms
Total time taken: 438 ms

Register 40 GB segment:

Without pre-touch
Register memory region time taken: 12045 ms
Register memory region time taken: 1082 ms
Register memory region time taken: 1084 ms
Register memory region time taken: 1090 ms
Register memory region time taken: 1072 ms
Register memory region time taken: 1105 ms
Register memory region time taken: 1220 ms
Register memory region time taken: 1213 ms
Get memory location time taken: 1636 ms
Add local memory buffer time taken: 1 ms
Total time taken: 21634 ms

With pre-touch:
Register memory region time taken: 1339 ms
Register memory region time taken: 1257 ms
Register memory region time taken: 1245 ms
Register memory region time taken: 1246 ms
Register memory region time taken: 1261 ms
Register memory region time taken: 1281 ms
Register memory region time taken: 1282 ms
Register memory region time taken: 1253 ms
Get memory location time taken: 1851 ms
Add local memory buffer time taken: 1 ms
Total time taken: 12083 ms

Without pre-touch, parallel register:
Register memory region time taken: 85478 ms
Register memory region time taken: 85479 ms
Register memory region time taken: 85484 ms
Register memory region time taken: 85493 ms
Register memory region time taken: 85494 ms
Register memory region time taken: 85499 ms
Register memory region time taken: 85498 ms
Register memory region time taken: 85502 ms
Get memory location time taken: 1825 ms
Add local memory buffer time taken: 1 ms
Total time taken: 87416 ms

With pre-touch, parallel register:
Register memory region time taken: 1665 ms
Register memory region time taken: 1667 ms
Register memory region time taken: 1676 ms
Register memory region time taken: 1680 ms
Register memory region time taken: 1681 ms
Register memory region time taken: 1684 ms
Register memory region time taken: 1689 ms
Register memory region time taken: 1704 ms
Get memory location time taken: 1878 ms
Add local memory buffer time taken: 2 ms
Total time taken: 3651 ms

Register 160 GB segment:

Without pre-touch
Register memory region time taken: 46193 ms
Register memory region time taken: 3310 ms
Register memory region time taken: 3308 ms
Register memory region time taken: 3311 ms
Register memory region time taken: 3278 ms
Register memory region time taken: 3246 ms
Register memory region time taken: 3246 ms
Register memory region time taken: 3269 ms
Get memory location time taken: 5805 ms
Add local memory buffer time taken: 1 ms
Total time taken: 75290 ms

With pre-touch:
Register memory region time taken: 3608 ms
Register memory region time taken: 3517 ms
Register memory region time taken: 3490 ms
Register memory region time taken: 3476 ms
Register memory region time taken: 3488 ms
Register memory region time taken: 3491 ms
Register memory region time taken: 3519 ms
Register memory region time taken: 3481 ms
Get memory location time taken: 5869 ms
Add local memory buffer time taken: 1 ms
Total time taken: 34223 ms

Without pre-touch, parallel register:
Not test

With pre-touch, parallel register:
Register memory region time taken: 4052 ms
Register memory region time taken: 4146 ms
Register memory region time taken: 4200 ms
Register memory region time taken: 4214 ms
Register memory region time taken: 4214 ms
Register memory region time taken: 4269 ms
Register memory region time taken: 4322 ms
Register memory region time taken: 4322 ms
Get memory location time taken: 5674 ms
Add local memory buffer time taken: 1 ms
Total time taken: 10261 ms

From the above data, we can observe:

• Without pre-touching, registering on the first NIC takes a very long time due to page faults. In contrast, with pre-touching enabled, registering across all NICs takes roughly the same amount of time.
• Without pre-touching, concurrent registration slows down significantly—even much slower than sequential registration. In contrast, with pre-touching, concurrent registration can greatly reduce the total time.

Conclusion:

• Perform concurrent registration only when pre-touching is used.

Combining this with the earlier pre-touching tests, we can conclude two practical strategies:

• When the segment size is small or only a few threads are available, skip pre-touching and register sequentially.
• Otherwise, use a fixed number of threads for pre-touching, and then use num_nic + 1 threads to concurrently register the memory region and get memory locations.

cc. @alogfans (Background: mooncake store needs to register hundreds or even 1TB memory; it will take several minutes to register. We want to speed up this process.)

[alogfans]

@ykwd Thanks for your experiments. Pre-touching essentially reduce the overhead of page allocation (which may constraits by giant locks). Can allocating 2MB or even 2GB hugepages further reduce latency?

[ykwd]

Solved in this pr: #875