Skip to content

Repo size - Submodules / Parallel Building Memory Usage #75

@danieldjewell

Description

@danieldjewell

Two issues:

  1. With all the submodules, the git pack files alone are 3.7GiB and with everything checked out it's in the neighborhood of 6 GiB... It would be nice to try and slim that down if possible. (Can git submodules be set to do a shallow fetch?)

  2. Building in parallel (make -j) on a VM with 32 cores exposed (the physical hardware is 2x Xeon 8-core with hyperthreading, so 16 threads/CPU) results in ~64 parallel jobs running - at least in the llvm/bolt phase. The trouble with this is:

  • Massive memory usage (something like 20GB)
  • Probably slower build times (depending on what I'm building, I find that 24-30 jobs is actually fastest ... because the hyperthreading isn't actually another core.)
  • The normal make options like -j16 aren't respected - when I try this, bolt and llvm appear to build in parallel but using one thread each.
  • I haven't tested the I/O impact specifically, but using all the cores can definitely run into interesting situations where the build is I/O limited and not processor limited. (I'm not using NVMe drives in this case...)

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions