find and document automation to dump/load hosts /nix/store db

[thkoch2001]

The hosts /nix/store is mounted inside the microvm. However nix inside the VM does not use objects from /nix/store. It does not know these objects since they are not in the Nix database (/nix/var/nix/db).

Probably the best solution is currently worked on as [RFC 0152] local-overlay store.

A workaround is to use nix-store --dump-db on the host and nix-store --load-db in the guest VM.

Of course this is a hack. But for my use case of local project development VMs it should work fine.

I've not yet figured out a nice automation for this. Ideally, a db-dump would be made on microvm boot, copied in the VM and loaded inside the VM.

[thkoch2001]

Documenting my current implementation here before I forget any details.

I'm running my microvm as a user systemd service, templated by the vm name. There is a dependency on a virtiofsd service. Before starting the VM, the microvm service does:

[Service]
ExecStartPre=/usr/bin/sh -c "[ /nix/var/nix/db/db.sqlite -ot %h/.local/share/microvm/nix-store-db-dump ] || nix-store --dump-db >%h/.local/share/microvm/nix-store-db-dump"
ExecStartPre=ln -f -t %h/.local/share/microvm/vms/%i/persistent/ %h/.local/share/microvm/nix-store-db-dump

The first line dumps the hosts nix database to ~/.local/share/microvm/nix-store-db-dump if the db.sqlite is not older than the dump file.
The second line hard-links the dump into a folder that gets mounted into the VM.

In the microvm flake.nix I got:

              systemd.services.loadnixdb = {
                description = "import hosts nix database";
                path = [pkgs.nix];
                wantedBy = ["multi-user.target"];
                requires = ["nix-daemon.service"];
                script = "cat ${persistencePath}/nix-store-db-dump|nix-store --load-db";
              };

[Dummyc0m]

It looks like local-overlay stores have been merged as an experimental feature. https://nix.dev/manual/nix/2.22/store/types/experimental-local-overlay-store

I'm curious if there's any interest/development in the VM guest use case... If no one has tried it yet I might spend a weekend working on it.

[astro]

I'd love to have an example for this!

[Ramblurr]

I looked into this, and have some news. For my use case the local-overlay-store feature is no good because it comes with the limitation that the underlying lowerdir store cannot be garbage collected, which is no good in my case. So local-overlay-store will have to wait for someone else to come along and explore it.

However the RFC mentioned an "overlay everything" method, which at first I thought was the microvm.nix solution, but as I read the RFC, I realized microvm.nix is missing a crucial piece, as @thkoch2001 has identified, the guest does not know about packages in the shared store because its db is different.

The true "overlay everything" method involves also a separate overlayfs mount for /nix/var/nix/db. Turns out it is fairly simple to add this extra overlay, it is mostly a copy paste job of what Astro already gave us with the ro-store.

Here is a self contained solution you can drop it into your config using microvm.nix and:

https://gist.github.com/Ramblurr/ea69233ad7c76dcd234ed7373bf6ff6e

With this config you can start a guest vm and attempt to run/build a derivation, and if it is in the hosts store it works as you would expect.

It comes with the restriction that anything added to the hosts store will not ever be visible to the guest, because the dbs diverged. I personally run this with "ephemeral" microvms where the rootfs, rw-store and and rw-var-nix-db are created fresh on every boot.