nix flake check: Skip substitutable derivations

Since `nix flake check` doesn't produce a `result` symlink, it doesn't
actually need to build/substitute derivations that are already known
to have succeeded, i.e. that are substitutable.

This can speed up CI jobs in cases where the derivations have already
been built by other jobs. For instance, a command like

  nix flake check github:NixOS/hydra/aa62c7f7db31753f0cde690f8654dd1907fc0ce2

should no longer build anything because the outputs are already in
cache.nixos.org.

Based-on: DeterminateSystems#134
Based-on: https://gerrit.lix.systems/c/lix/+/3841
Co-authored-by: Eelco Dolstra <[email protected]>

Author: 2 people

doc/manual/rl-next/faster-nix-flake-check.md

@@ -0,0 +1,9 @@
+---
+synopsis: "`nix flake check` now skips derivations that can be substituted"
+prs: [13574]
+---
+
+Previously, `nix flake check` would evaluate and build/substitute all
+derivations. Now, it will skip downloading derivations that can be substituted.
+This can drastically decrease the time invocations take in environments where
+checks may already be cached (like in CI).

src/nix/flake.cc

@@ -786,8 +786,29 @@ struct CmdFlakeCheck : FlakeCommand
         }
 
         if (build && !drvPaths.empty()) {
-            Activity act(*logger, lvlInfo, actUnknown, fmt("running %d flake checks", drvPaths.size()));
-            store->buildPaths(drvPaths);
+            // TODO the pruning is temp hacky fix until https://github.com/NixOS/nix/issues/5025 is implemented.
+            // At that point this should be replaced by code thats sets up a union store.
+            // Note that to exactly preserve behavior, we would want to look not a single destination store,
+            // but an "any of" union of stores as the destination store,
+            // to support different check results being cached with different substituters.
+
+            // This command doesn't need to actually substitute derivation outputs
+            // if they're missing but substitutable.
+            auto missing = store->queryMissing(drvPaths);
+            // Only occurs if `drvPaths` contains a `DerivedPath::Opaque`, which should never happen
+            assert(missing.unknown.empty());
+
+            std::vector<DerivedPath> toBuild;
+            for (auto & path : missing.willBuild) {
+                toBuild.emplace_back(
+                    DerivedPath::Built{
+                        .drvPath = makeConstantStorePathRef(path),
+                        .outputs = OutputsSpec::All{},
+                    });
+            }
+
+            Activity act(*logger, lvlInfo, actUnknown, fmt("running %d flake checks", toBuild.size()));
+            store->buildPaths(toBuild);
         }
         if (hasErrors)
             throw Error("some errors were encountered during the evaluation");

tests/functional/flakes/check.sh

@@ -167,3 +167,23 @@ EOF
 if !isTestOnNixOS && $NIX_REMOTE != daemon; then
     expectStderr 100 nix flake check "$flakeDir" | grepQuiet 'builder failed with exit code 1'
 fi
+
+# Ensure non-substitutable (read: usually failed) checks are actually run
+# https://github.com/NixOS/nix/pull/13574
+cp "$config_nix" $flakeDir/
+cat > $flakeDir/flake.nix <<EOF
+{
+  outputs = { self }: with import ./config.nix; {
+    checks.${system}.expectedToFail = derivation {
+      name = "expected-to-fail";
+      inherit system;
+      builder = "not-a-real-file";
+    };
+  };
+}
+EOF
+
+# NOTE: Regex pattern is used for compatibility with older daemon versions
+# We also can't expect a specific status code. Earlier daemons return 1, but as of 2.31, we return 100
+checkRes=$(nix flake check $flakeDir 2>&1 && fail "nix flake check should have failed" || true)
+echo "$checkRes" | grepQuiet -E "builder( for .*)? failed with exit code 1"