From 8db8371a486ec3a074e79f6729d47c09842f01ef Mon Sep 17 00:00:00 2001 From: Valentin Gagarin Date: Thu, 10 Apr 2025 13:51:14 +0200 Subject: [PATCH 001/218] glossary: re-introduce "derivation" The "derivation" is one of the key concepts and captures the most distinctive aspect of Nix: that we work with a certain type data (linked files) in a certain manner (using pure functions). Here we finally arrange all the important pieces to show how they belong together, while referring to the respective reference documentation for details. This change also unbreaks downstream links to `//glossary#gloss-derivation`, which had been broken by removing the glossary entry --- doc/manual/source/glossary.md | 21 ++++++++++++++++----- doc/manual/source/language/index.md | 2 +- 2 files changed, 17 insertions(+), 6 deletions(-) diff --git a/doc/manual/source/glossary.md b/doc/manual/source/glossary.md index 6a7501200d6..787978502be 100644 --- a/doc/manual/source/glossary.md +++ b/doc/manual/source/glossary.md @@ -31,9 +31,23 @@ The industry term for storage and retrieval systems using [content addressing](#gloss-content-address). A Nix store also has [input addressing](#gloss-input-addressed-store-object), and metadata. +- [derivation]{#gloss-derivation} + + A derivation can be thought of as a [pure function](https://en.wikipedia.org/wiki/Pure_function) that produces new [store objects][store object] from existing store objects. + + Derivations are implemented as [operating system processes that run in a sandbox](@docroot@/store/building.md#builder-execution). + This sandbox by default only allows reading from store objects specified as inputs, and only allows writing to designated [outputs][output] to be [captured as store objects](@docroot@/store/building.md#processing-outputs). + How exactly to invoke such an isolated process is encoded in a [store derivation]. + + A derivation is typically specified as a [derivation expression] in the [Nix language], and [instantiated][instantiate] to a store derivation. + There are multiple ways of obtaining store objects from store derivatons, collectively called [realisation][realise]. + + [derivation]: #gloss-derivation + - [store derivation]{#gloss-store-derivation} - A single build task. + A [derivation] represented as a [store object]. + See [Store Derivation](@docroot@/store/derivation/index.md#store-derivation) for details. [store derivation]: #gloss-store-derivation @@ -50,10 +64,7 @@ - [derivation expression]{#gloss-derivation-expression} - A description of a [store derivation] in the Nix language. - The output(s) of a derivation are store objects. - Derivations are typically specified in Nix expressions using the [`derivation` primitive](./language/derivations.md). - These are translated into store layer *derivations* (implicitly by `nix-env` and `nix-build`, or explicitly by `nix-instantiate`). + A description of a [store derivation] using the [`derivation` primitive](./language/derivations.md) in the [Nix language]. [derivation expression]: #gloss-derivation-expression diff --git a/doc/manual/source/language/index.md b/doc/manual/source/language/index.md index 2bfdbb8a0c6..b7630235a7c 100644 --- a/doc/manual/source/language/index.md +++ b/doc/manual/source/language/index.md @@ -1,6 +1,6 @@ # Nix Language -The Nix language is designed for conveniently creating and composing *derivations* – precise descriptions of how contents of existing files are used to derive new files. +The Nix language is designed for conveniently creating and composing [derivations](@docroot@/glossary.md#gloss-derivation) – precise descriptions of how contents of existing files are used to derive new files. > **Tip** > From 82d598282d9addd280df08dedd3fb175b3950722 Mon Sep 17 00:00:00 2001 From: Valentin Gagarin Date: Tue, 29 Apr 2025 10:55:38 +0200 Subject: [PATCH 002/218] don't emphasize 'store derivation' --- doc/manual/source/glossary.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/doc/manual/source/glossary.md b/doc/manual/source/glossary.md index 787978502be..371ac320fd7 100644 --- a/doc/manual/source/glossary.md +++ b/doc/manual/source/glossary.md @@ -37,9 +37,8 @@ Derivations are implemented as [operating system processes that run in a sandbox](@docroot@/store/building.md#builder-execution). This sandbox by default only allows reading from store objects specified as inputs, and only allows writing to designated [outputs][output] to be [captured as store objects](@docroot@/store/building.md#processing-outputs). - How exactly to invoke such an isolated process is encoded in a [store derivation]. - A derivation is typically specified as a [derivation expression] in the [Nix language], and [instantiated][instantiate] to a store derivation. + A derivation is typically specified as a [derivation expression] in the [Nix language], and [instantiated][instantiate] to a [store derivation]. There are multiple ways of obtaining store objects from store derivatons, collectively called [realisation][realise]. [derivation]: #gloss-derivation From 3a1301cd6db698a212a0c036e40ad402bd8a2a12 Mon Sep 17 00:00:00 2001 From: Sergei Zimmerman Date: Tue, 6 May 2025 21:58:52 +0000 Subject: [PATCH 003/218] libstore: Use `boost::regex` for GC root discovery MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit As it turns out using `std::regex` is actually the bottleneck for root discovery. Just substituting `std::` -> `boost::` makes root discovery twice as fast (3x if counting only userspace time). Some rather ad-hoc measurements to motivate the switch: (On master) ``` nix build github:nixos/nix/1e822bd4149a8bce1da81ee2ad9404986b07914c#nix-cli --out-link result-1e822bd4149a8bce1da81ee2ad9404986b07914c taskset -c 2,3 hyperfine "result-1e822bd4149a8bce1da81ee2ad9404986b07914c/bin/nix store gc --dry-run --max 0" Benchmark 1: result-1e822bd4149a8bce1da81ee2ad9404986b07914c/bin/nix store gc --dry-run --max 0 Time (mean ± σ): 481.6 ms ± 3.9 ms [User: 336.2 ms, System: 142.0 ms] Range (min … max): 474.6 ms … 487.7 ms 10 runs ``` (After this patch) ``` taskset -c 2,3 hyperfine "result/bin/nix store gc --dry-run --max 0" Benchmark 1: result/bin/nix store gc --dry-run --max 0 Time (mean ± σ): 254.7 ms ± 9.7 ms [User: 111.1 ms, System: 141.3 ms] Range (min … max): 246.5 ms … 281.3 ms 10 runs ``` `boost::regex` is a drop-in replacement for `std::regex`, but much faster. Doing a simple before/after comparison doesn't surface any change in behavior: ``` result/bin/nix store gc --dry-run -vvvvv --max 0 |& grep "got additional" | wc -l result-1e822bd4149a8bce1da81ee2ad9404986b07914c/bin/nix store gc --dry-run -vvvvv --max 0 |& grep "got additional" | wc -l ``` --- src/libstore/gc.cc | 29 +++++++++++++++-------------- 1 file changed, 15 insertions(+), 14 deletions(-) diff --git a/src/libstore/gc.cc b/src/libstore/gc.cc index f6a4124ff57..e9fe72afe2a 100644 --- a/src/libstore/gc.cc +++ b/src/libstore/gc.cc @@ -13,10 +13,11 @@ # include "nix/util/processes.hh" #endif +#include + #include #include #include -#include #include #include @@ -331,8 +332,8 @@ static void readProcLink(const std::filesystem::path & file, UncheckedRoots & ro static std::string quoteRegexChars(const std::string & raw) { - static auto specialRegex = std::regex(R"([.^$\\*+?()\[\]{}|])"); - return std::regex_replace(raw, specialRegex, R"(\$&)"); + static auto specialRegex = boost::regex(R"([.^$\\*+?()\[\]{}|])"); + return boost::regex_replace(raw, specialRegex, R"(\$&)"); } #ifdef __linux__ @@ -354,12 +355,12 @@ void LocalStore::findRuntimeRoots(Roots & roots, bool censor) auto procDir = AutoCloseDir{opendir("/proc")}; if (procDir) { struct dirent * ent; - auto digitsRegex = std::regex(R"(^\d+$)"); - auto mapRegex = std::regex(R"(^\s*\S+\s+\S+\s+\S+\s+\S+\s+\S+\s+(/\S+)\s*$)"); - auto storePathRegex = std::regex(quoteRegexChars(storeDir) + R"(/[0-9a-z]+[0-9a-zA-Z\+\-\._\?=]*)"); + static const auto digitsRegex = boost::regex(R"(^\d+$)"); + static const auto mapRegex = boost::regex(R"(^\s*\S+\s+\S+\s+\S+\s+\S+\s+\S+\s+(/\S+)\s*$)"); + auto storePathRegex = boost::regex(quoteRegexChars(storeDir) + R"(/[0-9a-z]+[0-9a-zA-Z\+\-\._\?=]*)"); while (errno = 0, ent = readdir(procDir.get())) { checkInterrupt(); - if (std::regex_match(ent->d_name, digitsRegex)) { + if (boost::regex_match(ent->d_name, digitsRegex)) { try { readProcLink(fmt("/proc/%s/exe" ,ent->d_name), unchecked); readProcLink(fmt("/proc/%s/cwd", ent->d_name), unchecked); @@ -386,15 +387,15 @@ void LocalStore::findRuntimeRoots(Roots & roots, bool censor) std::filesystem::path mapFile = fmt("/proc/%s/maps", ent->d_name); auto mapLines = tokenizeString>(readFile(mapFile.string()), "\n"); for (const auto & line : mapLines) { - auto match = std::smatch{}; - if (std::regex_match(line, match, mapRegex)) + auto match = boost::smatch{}; + if (boost::regex_match(line, match, mapRegex)) unchecked[match[1]].emplace(mapFile.string()); } auto envFile = fmt("/proc/%s/environ", ent->d_name); auto envString = readFile(envFile); - auto env_end = std::sregex_iterator{}; - for (auto i = std::sregex_iterator{envString.begin(), envString.end(), storePathRegex}; i != env_end; ++i) + auto env_end = boost::sregex_iterator{}; + for (auto i = boost::sregex_iterator{envString.begin(), envString.end(), storePathRegex}; i != env_end; ++i) unchecked[i->str()].emplace(envFile); } catch (SystemError & e) { if (errno == ENOENT || errno == EACCES || errno == ESRCH) @@ -413,12 +414,12 @@ void LocalStore::findRuntimeRoots(Roots & roots, bool censor) // Because of this we disable lsof when running the tests. if (getEnv("_NIX_TEST_NO_LSOF") != "1") { try { - std::regex lsofRegex(R"(^n(/.*)$)"); + boost::regex lsofRegex(R"(^n(/.*)$)"); auto lsofLines = tokenizeString>(runProgram(LSOF, true, { "-n", "-w", "-F", "n" }), "\n"); for (const auto & line : lsofLines) { - std::smatch match; - if (std::regex_match(line, match, lsofRegex)) + boost::smatch match; + if (boost::regex_match(line, match, lsofRegex)) unchecked[match[1].str()].emplace("{lsof}"); } } catch (ExecError & e) { From f3090ef7033c9bdc04beacfbb128c688cfa40fee Mon Sep 17 00:00:00 2001 From: Sergei Zimmerman Date: Tue, 13 May 2025 08:47:24 +0000 Subject: [PATCH 004/218] packaging/dependencies: Use boost without enableIcu This reduces the closure size on master by 40MiB. ``` $ nix build github:nixos/nix/1e822bd4149a8bce1da81ee2ad9404986b07914c#nix-store --out-link closure-on-master $ nix build .#nix-store -L --out-link closure-without-icu $ nix path-info --closure-size -h ./closure-on-master /nix/store/8gwr38m5h6p7245ji9jv28a2a11w1isx-nix-store-2.29.0pre 124.4 MiB $ nix path-info --closure-size -h ./closure-without-icu /nix/store/k0gwfykjqpnmaqbwh23nk55lhanc9g24-nix-store-2.29.0pre 86.6 MiB ``` --- packaging/dependencies.nix | 1 + 1 file changed, 1 insertion(+) diff --git a/packaging/dependencies.nix b/packaging/dependencies.nix index a90ef1b4ab7..7ce3bf1259c 100644 --- a/packaging/dependencies.nix +++ b/packaging/dependencies.nix @@ -63,6 +63,7 @@ scope: { "--with-coroutine" "--with-iostreams" ]; + enableIcu = false; }).overrideAttrs (old: { # Need to remove `--with-*` to use `--with-libraries=...` From 18a5589f9a6d710fe1f70e694cee513589c1c11c Mon Sep 17 00:00:00 2001 From: Sergei Zimmerman Date: Tue, 13 May 2025 08:51:46 +0000 Subject: [PATCH 005/218] libstore: Depend on boost_regex explicitly --- src/libstore/meson.build | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/libstore/meson.build b/src/libstore/meson.build index 255f83f74cd..a8781b457c3 100644 --- a/src/libstore/meson.build +++ b/src/libstore/meson.build @@ -94,7 +94,7 @@ subdir('nix-meson-build-support/libatomic') boost = dependency( 'boost', - modules : ['container'], + modules : ['container', 'regex'], include_type: 'system', ) # boost is a public dependency, but not a pkg-config dependency unfortunately, so we From 6591a4119c894a242b696fd270e561e618d15337 Mon Sep 17 00:00:00 2001 From: John Ericson Date: Wed, 14 May 2025 19:34:05 -0400 Subject: [PATCH 006/218] Bump version --- .version | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.version b/.version index f01356823fd..6a6900382e2 100644 --- a/.version +++ b/.version @@ -1 +1 @@ -2.29.0 +2.30.0 From 95e1f463c8a85e9c3fd8a0877dbabca30d3eb36a Mon Sep 17 00:00:00 2001 From: John Ericson Date: Wed, 14 May 2025 19:36:32 -0400 Subject: [PATCH 007/218] .mergify.yml: Add backport 2.29-maintenance entry --- .mergify.yml | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/.mergify.yml b/.mergify.yml index 36ffe6e8b91..8941711a993 100644 --- a/.mergify.yml +++ b/.mergify.yml @@ -139,3 +139,14 @@ pull_request_rules: labels: - automatic backport - merge-queue + + - name: backport patches to 2.29 + conditions: + - label=backport 2.29-maintenance + actions: + backport: + branches: + - "2.29-maintenance" + labels: + - automatic backport + - merge-queue From 99cb85cd37cf2c086bb9c01ec4b99077e5b866e5 Mon Sep 17 00:00:00 2001 From: John Ericson Date: Tue, 13 May 2025 13:37:31 -0400 Subject: [PATCH 008/218] Revert "If a substitute closure is incomplete, build dependencies, then retry the substituter" As summarized in https://github.com/NixOS/nix/issues/77#issuecomment-2843228280 the motivation is that the complicated retry logic this introduced was making the cleanup task #12628 harder to accomplish. It was not easy to ascertain just what policy / semantics the extra control-flow was implementing, in order to figure out a different way to implementing it either. After talking to Eelco about it, he decided we could just....get rid of the feature entirely! It's a bit scary removing a decade+ old feature, but I think he is right. See the release notes for more explanation. This reverts commit 299141ecbd08bae17013226dbeae71e842b4fdd7. Co-authored-by: Eelco Dolstra --- doc/manual/rl-next/revert-77.md | 17 +++++++++ src/libstore/build/derivation-goal.cc | 36 ++----------------- .../build/drv-output-substitution-goal.cc | 2 +- src/libstore/build/goal.cc | 6 ++-- src/libstore/build/substitution-goal.cc | 2 +- .../nix/store/build/derivation-goal.hh | 26 -------------- src/libstore/include/nix/store/build/goal.hh | 8 +---- tests/functional/binary-cache.sh | 7 ++-- 8 files changed, 29 insertions(+), 75 deletions(-) create mode 100644 doc/manual/rl-next/revert-77.md diff --git a/doc/manual/rl-next/revert-77.md b/doc/manual/rl-next/revert-77.md new file mode 100644 index 00000000000..c2cef74d3e4 --- /dev/null +++ b/doc/manual/rl-next/revert-77.md @@ -0,0 +1,17 @@ +--- +synopsis: Revert incomplete closure mixed download and build feature +issues: [77, 12628] +prs: [13176] +--- + +Since Nix 1.3 (299141ecbd08bae17013226dbeae71e842b4fdd7 in 2013) Nix has attempted to mix together upstream fresh builds and downstream substitutions when remote substuters contain an "incomplete closure" (have some store objects, but not the store objects they reference). +This feature is now removed. + +Worst case, removing this feature could cause more building downstream, but it should not cause outright failures, since this is not happening for opaque store objects that we don't know how to build if we decide not to substitute. +In practice, however, we doubt even the more building is very likely to happen. +Remote stores that are missing dependencies in arbitrary ways (e.g. corruption) don't seem to be very common. + +On the contrary, when remote stores fail to implement the [closure property](@docroot@/store/store-object.md#closure-property), it is usually an *intentional* choice on the part of the remote store, because it wishes to serve as an "overlay" store over another store, such as `https://cache.nixos.org`. +If an "incomplete closure" is encountered in that situation, the right fix is not to do some sort of "franken-building" as this feature implemented, but instead to make sure both substituters are enabled in the settings. + +(In the future, we should make it easier for remote stores to indicate this to clients, to catch settings that won't work in general before a missing dependency is actually encountered.) diff --git a/src/libstore/build/derivation-goal.cc b/src/libstore/build/derivation-goal.cc index 4d437112869..3931579387b 100644 --- a/src/libstore/build/derivation-goal.cc +++ b/src/libstore/build/derivation-goal.cc @@ -285,40 +285,13 @@ Goal::Co DerivationGoal::haveDerivation() assert(!drv->type().isImpure()); - if (nrFailed > 0 && nrFailed > nrNoSubstituters + nrIncompleteClosure && !settings.tryFallback) { + if (nrFailed > 0 && nrFailed > nrNoSubstituters && !settings.tryFallback) { co_return done(BuildResult::TransientFailure, {}, Error("some substitutes for the outputs of derivation '%s' failed (usually happens due to networking issues); try '--fallback' to build derivation from source ", worker.store.printStorePath(drvPath))); } - /* If the substitutes form an incomplete closure, then we should - build the dependencies of this derivation, but after that, we - can still use the substitutes for this derivation itself. - - If the nrIncompleteClosure != nrFailed, we have another issue as well. - In particular, it may be the case that the hole in the closure is - an output of the current derivation, which causes a loop if retried. - */ - { - bool substitutionFailed = - nrIncompleteClosure > 0 && - nrIncompleteClosure == nrFailed; - switch (retrySubstitution) { - case RetrySubstitution::NoNeed: - if (substitutionFailed) - retrySubstitution = RetrySubstitution::YesNeed; - break; - case RetrySubstitution::YesNeed: - // Should not be able to reach this state from here. - assert(false); - break; - case RetrySubstitution::AlreadyRetried: - debug("substitution failed again, but we already retried once. Not retrying again."); - break; - } - } - - nrFailed = nrNoSubstituters = nrIncompleteClosure = 0; + nrFailed = nrNoSubstituters = 0; if (needRestart == NeedRestartForMoreOutputs::OutputsAddedDoNeed) { needRestart = NeedRestartForMoreOutputs::OutputsUnmodifedDontNeed; @@ -456,11 +429,6 @@ Goal::Co DerivationGoal::gaveUpOnSubstitution() co_return done(BuildResult::DependencyFailed, {}, Error(msg)); } - if (retrySubstitution == RetrySubstitution::YesNeed) { - retrySubstitution = RetrySubstitution::AlreadyRetried; - co_return haveDerivation(); - } - /* Gather information necessary for computing the closure and/or running the build hook. */ diff --git a/src/libstore/build/drv-output-substitution-goal.cc b/src/libstore/build/drv-output-substitution-goal.cc index c553eeedb95..80a6e060ed3 100644 --- a/src/libstore/build/drv-output-substitution-goal.cc +++ b/src/libstore/build/drv-output-substitution-goal.cc @@ -139,7 +139,7 @@ Goal::Co DrvOutputSubstitutionGoal::realisationFetched(Goals waitees, std::share if (nrFailed > 0) { debug("The output path of the derivation output '%s' could not be substituted", id.to_string()); - co_return amDone(nrNoSubstituters > 0 || nrIncompleteClosure > 0 ? ecIncompleteClosure : ecFailed); + co_return amDone(nrNoSubstituters > 0 ? ecNoSubstituters : ecFailed); } worker.store.registerDrvOutput(*outputInfo); diff --git a/src/libstore/build/goal.cc b/src/libstore/build/goal.cc index d2feb34c79f..c9294a1a4d1 100644 --- a/src/libstore/build/goal.cc +++ b/src/libstore/build/goal.cc @@ -151,7 +151,7 @@ Goal::Done Goal::amDone(ExitCode result, std::optional ex) trace("done"); assert(top_co); assert(exitCode == ecBusy); - assert(result == ecSuccess || result == ecFailed || result == ecNoSubstituters || result == ecIncompleteClosure); + assert(result == ecSuccess || result == ecFailed || result == ecNoSubstituters); exitCode = result; if (ex) { @@ -170,12 +170,10 @@ Goal::Done Goal::amDone(ExitCode result, std::optional ex) goal->trace(fmt("waitee '%s' done; %d left", name, goal->waitees.size())); - if (result == ecFailed || result == ecNoSubstituters || result == ecIncompleteClosure) ++goal->nrFailed; + if (result == ecFailed || result == ecNoSubstituters) ++goal->nrFailed; if (result == ecNoSubstituters) ++goal->nrNoSubstituters; - if (result == ecIncompleteClosure) ++goal->nrIncompleteClosure; - if (goal->waitees.empty()) { worker.wakeUp(goal); } else if (result == ecFailed && !settings.keepGoing) { diff --git a/src/libstore/build/substitution-goal.cc b/src/libstore/build/substitution-goal.cc index 2b4b8ac25f4..b745548a14c 100644 --- a/src/libstore/build/substitution-goal.cc +++ b/src/libstore/build/substitution-goal.cc @@ -169,7 +169,7 @@ Goal::Co PathSubstitutionGoal::tryToRun(StorePath subPath, nix::ref sub, if (nrFailed > 0) { co_return done( - nrNoSubstituters > 0 || nrIncompleteClosure > 0 ? ecIncompleteClosure : ecFailed, + nrNoSubstituters > 0 ? ecNoSubstituters : ecFailed, BuildResult::DependencyFailed, fmt("some references of path '%s' could not be realised", worker.store.printStorePath(storePath))); } diff --git a/src/libstore/include/nix/store/build/derivation-goal.hh b/src/libstore/include/nix/store/build/derivation-goal.hh index 485a34ec454..f6f4da249d7 100644 --- a/src/libstore/include/nix/store/build/derivation-goal.hh +++ b/src/libstore/include/nix/store/build/derivation-goal.hh @@ -73,32 +73,6 @@ struct DerivationGoal : public Goal */ NeedRestartForMoreOutputs needRestart = NeedRestartForMoreOutputs::OutputsUnmodifedDontNeed; - /** - * See `retrySubstitution`; just for that field. - */ - enum RetrySubstitution { - /** - * No issues have yet arose, no need to restart. - */ - NoNeed, - /** - * Something failed and there is an incomplete closure. Let's retry - * substituting. - */ - YesNeed, - /** - * We are current or have already retried substitution, and whether or - * not something goes wrong we will not retry again. - */ - AlreadyRetried, - }; - - /** - * Whether to retry substituting the outputs after building the - * inputs. This is done in case of an incomplete closure. - */ - RetrySubstitution retrySubstitution = RetrySubstitution::NoNeed; - /** * The derivation stored at drvPath. */ diff --git a/src/libstore/include/nix/store/build/goal.hh b/src/libstore/include/nix/store/build/goal.hh index 9be27f6b392..399b5f82fde 100644 --- a/src/libstore/include/nix/store/build/goal.hh +++ b/src/libstore/include/nix/store/build/goal.hh @@ -61,7 +61,7 @@ private: Goals waitees; public: - typedef enum {ecBusy, ecSuccess, ecFailed, ecNoSubstituters, ecIncompleteClosure} ExitCode; + typedef enum {ecBusy, ecSuccess, ecFailed, ecNoSubstituters} ExitCode; /** * Backlink to the worker. @@ -85,12 +85,6 @@ public: */ size_t nrNoSubstituters = 0; - /** - * Number of substitution goals we are/were waiting for that - * failed because they had unsubstitutable references. - */ - size_t nrIncompleteClosure = 0; - /** * Name of this goal for debugging purposes. */ diff --git a/tests/functional/binary-cache.sh b/tests/functional/binary-cache.sh index ff39ab3b7dc..2c102df0771 100755 --- a/tests/functional/binary-cache.sh +++ b/tests/functional/binary-cache.sh @@ -151,8 +151,11 @@ nix-build --substituters "file://$cacheDir" --no-require-sigs dependencies.nix - grepQuiet "don't know how to build" "$TEST_ROOT/log" grepQuiet "building.*input-1" "$TEST_ROOT/log" grepQuiet "building.*input-2" "$TEST_ROOT/log" -grepQuiet "copying path.*input-0" "$TEST_ROOT/log" -grepQuiet "copying path.*top" "$TEST_ROOT/log" + +# Removed for now since 299141ecbd08bae17013226dbeae71e842b4fdd7 / issue #77 is reverted + +#grepQuiet "copying path.*input-0" "$TEST_ROOT/log" +#grepQuiet "copying path.*top" "$TEST_ROOT/log" # Create a signed binary cache. From bc85e20fb98a4170b2f832692298f57fe30dffd5 Mon Sep 17 00:00:00 2001 From: Eelco Dolstra Date: Thu, 15 May 2025 11:10:21 +0200 Subject: [PATCH 009/218] Remove otherNixes.nix_2_18 Nixpkgs no longer has Nix 2.18, so this fails to evaluate. --- tests/nixos/default.nix | 7 ------- 1 file changed, 7 deletions(-) diff --git a/tests/nixos/default.nix b/tests/nixos/default.nix index 3e2d20a715f..f0b1a886565 100644 --- a/tests/nixos/default.nix +++ b/tests/nixos/default.nix @@ -94,13 +94,6 @@ let ); }; - otherNixes.nix_2_18.setNixPackage = - { lib, pkgs, ... }: - { - imports = [ checkOverrideNixVersion ]; - nix.package = lib.mkForce pkgs.nixVersions.nix_2_18; - }; - in { From 2dd214275459b52994941867df185d425ec6a4c7 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?J=C3=B6rg=20Thalheim?= Date: Thu, 15 May 2025 11:22:37 +0200 Subject: [PATCH 010/218] rename StoreDirConfigItself to StoreDirConfigBase context: https://github.com/NixOS/nix/pull/13154#discussion_r2081904653 --- src/libstore/include/nix/store/store-dir-config.hh | 8 ++++---- src/libstore/store-dir-config.cc | 2 +- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/src/libstore/include/nix/store/store-dir-config.hh b/src/libstore/include/nix/store/store-dir-config.hh index 40a71e446c2..6bf9ebf1431 100644 --- a/src/libstore/include/nix/store/store-dir-config.hh +++ b/src/libstore/include/nix/store/store-dir-config.hh @@ -104,7 +104,7 @@ struct MixStoreDirMethods * Need to make this a separate class so I can get the right * initialization order in the constructor for `StoreDirConfig`. */ -struct StoreDirConfigItself : Config +struct StoreDirConfigBase : Config { using Config::Config; @@ -118,12 +118,12 @@ struct StoreDirConfigItself : Config }; /** - * The order of `StoreDirConfigItself` and then `MixStoreDirMethods` is - * very important. This ensures that `StoreDirConfigItself::storeDir_` + * The order of `StoreDirConfigBase` and then `MixStoreDirMethods` is + * very important. This ensures that `StoreDirConfigBase::storeDir_` * is initialized before we have our one chance (because references are * immutable) to initialize `MixStoreDirMethods::storeDir`. */ -struct StoreDirConfig : StoreDirConfigItself, MixStoreDirMethods +struct StoreDirConfig : StoreDirConfigBase, MixStoreDirMethods { using Params = std::map; diff --git a/src/libstore/store-dir-config.cc b/src/libstore/store-dir-config.cc index 191926be638..ec65013ef2a 100644 --- a/src/libstore/store-dir-config.cc +++ b/src/libstore/store-dir-config.cc @@ -5,7 +5,7 @@ namespace nix { StoreDirConfig::StoreDirConfig(const Params & params) - : StoreDirConfigItself(params) + : StoreDirConfigBase(params) , MixStoreDirMethods{storeDir_} { } From 3ba49d7ec204c2985ef0cffd6e8ceefab448e475 Mon Sep 17 00:00:00 2001 From: Eelco Dolstra Date: Thu, 15 May 2025 11:31:34 +0200 Subject: [PATCH 011/218] flake.lock: Update MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Flake lock file updates: • Updated input 'nixpkgs': 'github:NixOS/nixpkgs/f02fddb8acef29a8b32f10a335d44828d7825b78?narHash=sha256-IgBWhX7A2oJmZFIrpRuMnw5RAufVnfvOgHWgIdds%2Bhc%3D' (2025-05-01) → 'github:NixOS/nixpkgs/adaa24fbf46737f3f1b5497bf64bae750f82942e?narHash=sha256-qhFMmDkeJX9KJwr5H32f1r7Prs7XbQWtO0h3V0a0rFY%3D' (2025-05-13) --- flake.lock | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/flake.lock b/flake.lock index b8ff29a0c83..3075eabc233 100644 --- a/flake.lock +++ b/flake.lock @@ -63,11 +63,11 @@ }, "nixpkgs": { "locked": { - "lastModified": 1746141548, - "narHash": "sha256-IgBWhX7A2oJmZFIrpRuMnw5RAufVnfvOgHWgIdds+hc=", + "lastModified": 1747179050, + "narHash": "sha256-qhFMmDkeJX9KJwr5H32f1r7Prs7XbQWtO0h3V0a0rFY=", "owner": "NixOS", "repo": "nixpkgs", - "rev": "f02fddb8acef29a8b32f10a335d44828d7825b78", + "rev": "adaa24fbf46737f3f1b5497bf64bae750f82942e", "type": "github" }, "original": { From d626348f42c60a9a3192b43b13dd27ebb6252ad8 Mon Sep 17 00:00:00 2001 From: Eelco Dolstra Date: Thu, 15 May 2025 12:55:08 +0200 Subject: [PATCH 012/218] Fix nix-copy-closure VM test https://hydra.nixos.org/build/297112538 --- tests/nixos/nix-copy-closure.nix | 2 -- 1 file changed, 2 deletions(-) diff --git a/tests/nixos/nix-copy-closure.nix b/tests/nixos/nix-copy-closure.nix index 34e3a2c7de7..d24930de060 100644 --- a/tests/nixos/nix-copy-closure.nix +++ b/tests/nixos/nix-copy-closure.nix @@ -61,12 +61,10 @@ in "${pkgs.openssh}/bin/ssh-keygen", "-t", "ed25519", "-f", "key", "-N", "" ], capture_output=True, check=True) - client.succeed("mkdir -m 700 /root/.ssh") client.copy_from_host("key", "/root/.ssh/id_ed25519") client.succeed("chmod 600 /root/.ssh/id_ed25519") # Install the SSH key on the server. - server.succeed("mkdir -m 700 /root/.ssh") server.copy_from_host("key.pub", "/root/.ssh/authorized_keys") server.wait_for_unit("sshd") server.wait_for_unit("multi-user.target") From c1085ce84926c08b371921d69cd692498071c4cc Mon Sep 17 00:00:00 2001 From: John Ericson Date: Tue, 13 May 2025 19:24:35 -0400 Subject: [PATCH 013/218] Get rid of virtual `Goal::init()` Now, each class provides the initial coroutine by value. This avoids some sketchy virtual function stuff, and will also be further put to good use in the next commit. --- src/libstore/build/derivation-goal.cc | 4 ++-- .../build/drv-output-substitution-goal.cc | 2 +- src/libstore/build/substitution-goal.cc | 2 +- .../nix/store/build/derivation-goal.hh | 2 +- .../build/drv-output-substitution-goal.hh | 2 +- src/libstore/include/nix/store/build/goal.hh | 19 ++----------------- .../nix/store/build/substitution-goal.hh | 2 +- 7 files changed, 9 insertions(+), 24 deletions(-) diff --git a/src/libstore/build/derivation-goal.cc b/src/libstore/build/derivation-goal.cc index 3931579387b..48031ac3d16 100644 --- a/src/libstore/build/derivation-goal.cc +++ b/src/libstore/build/derivation-goal.cc @@ -25,7 +25,7 @@ namespace nix { DerivationGoal::DerivationGoal(const StorePath & drvPath, const OutputsSpec & wantedOutputs, Worker & worker, BuildMode buildMode) - : Goal(worker) + : Goal(worker, init()) , useDerivation(true) , drvPath(drvPath) , wantedOutputs(wantedOutputs) @@ -43,7 +43,7 @@ DerivationGoal::DerivationGoal(const StorePath & drvPath, DerivationGoal::DerivationGoal(const StorePath & drvPath, const BasicDerivation & drv, const OutputsSpec & wantedOutputs, Worker & worker, BuildMode buildMode) - : Goal(worker) + : Goal(worker, init()) , useDerivation(false) , drvPath(drvPath) , wantedOutputs(wantedOutputs) diff --git a/src/libstore/build/drv-output-substitution-goal.cc b/src/libstore/build/drv-output-substitution-goal.cc index 80a6e060ed3..e87a796f6b5 100644 --- a/src/libstore/build/drv-output-substitution-goal.cc +++ b/src/libstore/build/drv-output-substitution-goal.cc @@ -12,7 +12,7 @@ DrvOutputSubstitutionGoal::DrvOutputSubstitutionGoal( Worker & worker, RepairFlag repair, std::optional ca) - : Goal(worker) + : Goal(worker, init()) , id(id) { name = fmt("substitution of '%s'", id.to_string()); diff --git a/src/libstore/build/substitution-goal.cc b/src/libstore/build/substitution-goal.cc index b745548a14c..9ffc8219d97 100644 --- a/src/libstore/build/substitution-goal.cc +++ b/src/libstore/build/substitution-goal.cc @@ -9,7 +9,7 @@ namespace nix { PathSubstitutionGoal::PathSubstitutionGoal(const StorePath & storePath, Worker & worker, RepairFlag repair, std::optional ca) - : Goal(worker) + : Goal(worker, init()) , storePath(storePath) , repair(repair) , ca(ca) diff --git a/src/libstore/include/nix/store/build/derivation-goal.hh b/src/libstore/include/nix/store/build/derivation-goal.hh index f6f4da249d7..4cfeb592524 100644 --- a/src/libstore/include/nix/store/build/derivation-goal.hh +++ b/src/libstore/include/nix/store/build/derivation-goal.hh @@ -166,7 +166,7 @@ struct DerivationGoal : public Goal /** * The states. */ - Co init() override; + Co init(); Co haveDerivation(); Co gaveUpOnSubstitution(); Co tryToBuild(); diff --git a/src/libstore/include/nix/store/build/drv-output-substitution-goal.hh b/src/libstore/include/nix/store/build/drv-output-substitution-goal.hh index a00de41adfc..0176f001ab6 100644 --- a/src/libstore/include/nix/store/build/drv-output-substitution-goal.hh +++ b/src/libstore/include/nix/store/build/drv-output-substitution-goal.hh @@ -33,7 +33,7 @@ public: typedef void (DrvOutputSubstitutionGoal::*GoalState)(); GoalState state; - Co init() override; + Co init(); Co realisationFetched(Goals waitees, std::shared_ptr outputInfo, nix::ref sub); void timedOut(Error && ex) override { unreachable(); }; diff --git a/src/libstore/include/nix/store/build/goal.hh b/src/libstore/include/nix/store/build/goal.hh index 399b5f82fde..669b58e3e46 100644 --- a/src/libstore/include/nix/store/build/goal.hh +++ b/src/libstore/include/nix/store/build/goal.hh @@ -338,17 +338,6 @@ protected: */ std::optional top_co; - /** - * The entry point for the goal - */ - virtual Co init() = 0; - - /** - * Wrapper around @ref init since virtual functions - * can't be used in constructors. - */ - inline Co init_wrapper(); - /** * Signals that the goal is done. * `co_return` the result. If you're not inside a coroutine, you can ignore @@ -376,8 +365,8 @@ public: */ std::optional ex; - Goal(Worker & worker) - : worker(worker), top_co(init_wrapper()) + Goal(Worker & worker, Co init) + : worker(worker), top_co(std::move(init)) { // top_co shouldn't have a goal already, should be nullptr. assert(!top_co->handle.promise().goal); @@ -440,7 +429,3 @@ template struct std::coroutine_traits { using promise_type = nix::Goal::promise_type; }; - -nix::Goal::Co nix::Goal::init_wrapper() { - co_return init(); -} diff --git a/src/libstore/include/nix/store/build/substitution-goal.hh b/src/libstore/include/nix/store/build/substitution-goal.hh index 7b68b08219e..b61706840f2 100644 --- a/src/libstore/include/nix/store/build/substitution-goal.hh +++ b/src/libstore/include/nix/store/build/substitution-goal.hh @@ -64,7 +64,7 @@ public: /** * The states. */ - Co init() override; + Co init(); Co gotInfo(); Co tryToRun(StorePath subPath, nix::ref sub, std::shared_ptr info, bool & substituterFailed); Co finished(); From 01207fd1017f24a3c869be8a906c11d49d2ea23a Mon Sep 17 00:00:00 2001 From: John Ericson Date: Tue, 13 May 2025 15:34:13 -0400 Subject: [PATCH 014/218] Remove `useDerivation` Try to make `DerivationGoal` care less whether we're working from an in-memory derivation or not. It's a clean-up in its own right, but it will also help with other cleanups under the umbrella of #12628. --- src/libstore/build/derivation-goal.cc | 76 ++++++++++--------- .../nix/store/build/derivation-goal.hh | 7 +- 2 files changed, 40 insertions(+), 43 deletions(-) diff --git a/src/libstore/build/derivation-goal.cc b/src/libstore/build/derivation-goal.cc index 48031ac3d16..089c51305ad 100644 --- a/src/libstore/build/derivation-goal.cc +++ b/src/libstore/build/derivation-goal.cc @@ -25,8 +25,7 @@ namespace nix { DerivationGoal::DerivationGoal(const StorePath & drvPath, const OutputsSpec & wantedOutputs, Worker & worker, BuildMode buildMode) - : Goal(worker, init()) - , useDerivation(true) + : Goal(worker, loadDerivation()) , drvPath(drvPath) , wantedOutputs(wantedOutputs) , buildMode(buildMode) @@ -43,8 +42,7 @@ DerivationGoal::DerivationGoal(const StorePath & drvPath, DerivationGoal::DerivationGoal(const StorePath & drvPath, const BasicDerivation & drv, const OutputsSpec & wantedOutputs, Worker & worker, BuildMode buildMode) - : Goal(worker, init()) - , useDerivation(false) + : Goal(worker, haveDerivation()) , drvPath(drvPath) , wantedOutputs(wantedOutputs) , buildMode(buildMode) @@ -143,10 +141,10 @@ void DerivationGoal::addWantedOutputs(const OutputsSpec & outputs) } -Goal::Co DerivationGoal::init() { - trace("init"); +Goal::Co DerivationGoal::loadDerivation() { + trace("local derivation"); - if (useDerivation) { + { /* The first thing to do is to make sure that the derivation exists. If it doesn't, it may be created through a substitute. */ @@ -355,7 +353,7 @@ Goal::Co DerivationGoal::gaveUpOnSubstitution() std::map, GoalPtr, value_comparison> inputGoals; - if (useDerivation) { + { std::function, const DerivedPathMap::ChildNode &)> addWaiteeDerivedPath; addWaiteeDerivedPath = [&](ref inputDrv, const DerivedPathMap::ChildNode & inputNode) { @@ -375,7 +373,7 @@ Goal::Co DerivationGoal::gaveUpOnSubstitution() childNode); }; - for (const auto & [inputDrvPath, inputNode] : dynamic_cast(drv.get())->inputDrvs.map) { + for (const auto & [inputDrvPath, inputNode] : drv->inputDrvs.map) { /* Ensure that pure, non-fixed-output derivations don't depend on impure derivations. */ if (experimentalFeatureSettings.isEnabled(Xp::ImpureDerivations) && !drv->type().isImpure() && !drv->type().isFixed()) { @@ -417,8 +415,6 @@ Goal::Co DerivationGoal::gaveUpOnSubstitution() trace("all inputs realised"); if (nrFailed != 0) { - if (!useDerivation) - throw Error("some dependencies of '%s' are missing", worker.store.printStorePath(drvPath)); auto msg = fmt( "Cannot build '%s'.\n" "Reason: " ANSI_RED "%d %s failed" ANSI_NORMAL ".", @@ -435,8 +431,8 @@ Goal::Co DerivationGoal::gaveUpOnSubstitution() /* Determine the full set of input paths. */ /* First, the input derivations. */ - if (useDerivation) { - auto & fullDrv = *dynamic_cast(drv.get()); + { + auto & fullDrv = *drv; auto drvType = fullDrv.type(); bool resolveDrv = std::visit(overloaded { @@ -918,7 +914,12 @@ Goal::Co DerivationGoal::repairClosure() derivation is responsible for which path in the output closure. */ StorePathSet inputClosure; - if (useDerivation) worker.store.computeFSClosure(drvPath, inputClosure); + + /* If we're working from an in-memory derivation with no in-store + `*.drv` file, we cannot do this part. */ + if (worker.store.isValidPath(drvPath)) + worker.store.computeFSClosure(drvPath, inputClosure); + std::map outputsToDrv; for (auto & i : inputClosure) if (i.isDerivation()) { @@ -1133,7 +1134,9 @@ HookReply DerivationGoal::tryBuildHook() #ifdef _WIN32 // TODO enable build hook on Windows return rpDecline; #else - if (settings.buildHook.get().empty() || !worker.tryBuildHook || !useDerivation) return rpDecline; + /* This should use `worker.evalStore`, but per #13179 the build hook + doesn't work with eval store anyways. */ + if (settings.buildHook.get().empty() || !worker.tryBuildHook || !worker.store.isValidPath(drvPath)) return rpDecline; if (!worker.hook) worker.hook = std::make_unique(); @@ -1399,33 +1402,32 @@ void DerivationGoal::flushLine() std::map> DerivationGoal::queryPartialDerivationOutputMap() { assert(!drv->type().isImpure()); - if (!useDerivation || drv->type().hasKnownOutputPaths()) { - std::map> res; - for (auto & [name, output] : drv->outputs) - res.insert_or_assign(name, output.path(worker.store, drv->name, name)); - return res; - } else { - for (auto * drvStore : { &worker.evalStore, &worker.store }) - if (drvStore->isValidPath(drvPath)) - return worker.store.queryPartialDerivationOutputMap(drvPath, drvStore); - assert(false); - } + + for (auto * drvStore : { &worker.evalStore, &worker.store }) + if (drvStore->isValidPath(drvPath)) + return worker.store.queryPartialDerivationOutputMap(drvPath, drvStore); + + /* In-memory derivation will naturally fall back on this case, where + we do best-effort with static information. */ + std::map> res; + for (auto & [name, output] : drv->outputs) + res.insert_or_assign(name, output.path(worker.store, drv->name, name)); + return res; } OutputPathMap DerivationGoal::queryDerivationOutputMap() { assert(!drv->type().isImpure()); - if (!useDerivation || drv->type().hasKnownOutputPaths()) { - OutputPathMap res; - for (auto & [name, output] : drv->outputsAndOptPaths(worker.store)) - res.insert_or_assign(name, *output.second); - return res; - } else { - for (auto * drvStore : { &worker.evalStore, &worker.store }) - if (drvStore->isValidPath(drvPath)) - return worker.store.queryDerivationOutputMap(drvPath, drvStore); - assert(false); - } + + for (auto * drvStore : { &worker.evalStore, &worker.store }) + if (drvStore->isValidPath(drvPath)) + return worker.store.queryDerivationOutputMap(drvPath, drvStore); + + // See comment in `DerivationGoal::queryPartialDerivationOutputMap`. + OutputPathMap res; + for (auto & [name, output] : drv->outputsAndOptPaths(worker.store)) + res.insert_or_assign(name, *output.second); + return res; } diff --git a/src/libstore/include/nix/store/build/derivation-goal.hh b/src/libstore/include/nix/store/build/derivation-goal.hh index 4cfeb592524..dc65b494135 100644 --- a/src/libstore/include/nix/store/build/derivation-goal.hh +++ b/src/libstore/include/nix/store/build/derivation-goal.hh @@ -33,11 +33,6 @@ void runPostBuildHook( */ struct DerivationGoal : public Goal { - /** - * Whether to use an on-disk .drv file. - */ - bool useDerivation; - /** The path of the derivation. */ StorePath drvPath; @@ -166,7 +161,7 @@ struct DerivationGoal : public Goal /** * The states. */ - Co init(); + Co loadDerivation(); Co haveDerivation(); Co gaveUpOnSubstitution(); Co tryToBuild(); From 7bd9eef772540c37c7a1681c4fa2ed2493906484 Mon Sep 17 00:00:00 2001 From: John Ericson Date: Thu, 15 May 2025 16:21:02 -0400 Subject: [PATCH 015/218] Deduplicate the goal creation functions The weak reference logic is the same in both these cases, and if/when I get rid `addWantedOutputs`, also in the `DerivationGoal` case. --- src/libstore/build/worker.cc | 28 ++++++++----------- .../include/nix/store/build/worker.hh | 3 ++ 2 files changed, 15 insertions(+), 16 deletions(-) diff --git a/src/libstore/build/worker.cc b/src/libstore/build/worker.cc index dd3692f4179..6b1e3c0f40f 100644 --- a/src/libstore/build/worker.cc +++ b/src/libstore/build/worker.cc @@ -41,6 +41,16 @@ Worker::~Worker() assert(expectedNarSize == 0); } +template +std::shared_ptr Worker::initGoalIfNeeded(std::weak_ptr & goal_weak, Args && ...args) +{ + if (auto goal = goal_weak.lock()) return goal; + + auto goal = std::make_shared(args...); + goal_weak = goal; + wakeUp(goal); + return goal; +} std::shared_ptr Worker::makeDerivationGoalCommon( const StorePath & drvPath, @@ -79,27 +89,13 @@ std::shared_ptr Worker::makeBasicDerivationGoal(const StorePath std::shared_ptr Worker::makePathSubstitutionGoal(const StorePath & path, RepairFlag repair, std::optional ca) { - std::weak_ptr & goal_weak = substitutionGoals[path]; - auto goal = goal_weak.lock(); // FIXME - if (!goal) { - goal = std::make_shared(path, *this, repair, ca); - goal_weak = goal; - wakeUp(goal); - } - return goal; + return initGoalIfNeeded(substitutionGoals[path], path, *this, repair, ca); } std::shared_ptr Worker::makeDrvOutputSubstitutionGoal(const DrvOutput& id, RepairFlag repair, std::optional ca) { - std::weak_ptr & goal_weak = drvOutputSubstitutionGoals[id]; - auto goal = goal_weak.lock(); // FIXME - if (!goal) { - goal = std::make_shared(id, *this, repair, ca); - goal_weak = goal; - wakeUp(goal); - } - return goal; + return initGoalIfNeeded(drvOutputSubstitutionGoals[id], id, *this, repair, ca); } diff --git a/src/libstore/include/nix/store/build/worker.hh b/src/libstore/include/nix/store/build/worker.hh index 7e03a0c2fe6..adbaa402786 100644 --- a/src/libstore/include/nix/store/build/worker.hh +++ b/src/libstore/include/nix/store/build/worker.hh @@ -196,6 +196,9 @@ public: * @ref DerivationGoal "derivation goal" */ private: + template + std::shared_ptr initGoalIfNeeded(std::weak_ptr & goal_weak, Args && ...args); + std::shared_ptr makeDerivationGoalCommon( const StorePath & drvPath, const OutputsSpec & wantedOutputs, std::function()> mkDrvGoal); From 67535263a577699002b9b0d05c2eea3f9615dd73 Mon Sep 17 00:00:00 2001 From: Sergei Zimmerman Date: Thu, 15 May 2025 21:13:13 +0000 Subject: [PATCH 016/218] dev-shell: Drop bear dependency Since the autotools-based build system has been removed and meson already generates compile database there's no need to have it in the devshell. --- packaging/dev-shell.nix | 3 --- 1 file changed, 3 deletions(-) diff --git a/packaging/dev-shell.nix b/packaging/dev-shell.nix index be760496af3..8d3fa38527a 100644 --- a/packaging/dev-shell.nix +++ b/packaging/dev-shell.nix @@ -119,9 +119,6 @@ pkgs.nixComponents2.nix-util.overrideAttrs ( (pkgs.writeScriptBin "pre-commit-hooks-install" modular.pre-commit.settings.installationScript) pkgs.buildPackages.nixfmt-rfc-style ] - # TODO: Remove the darwin check once - # https://github.com/NixOS/nixpkgs/pull/291814 is available - ++ lib.optional (stdenv.cc.isClang && !stdenv.buildPlatform.isDarwin) pkgs.buildPackages.bear ++ lib.optional (stdenv.cc.isClang && stdenv.hostPlatform == stdenv.buildPlatform) ( lib.hiPrio pkgs.buildPackages.clang-tools ) From 4711720efe3e1cf27d0e80f818b5d02ca1075457 Mon Sep 17 00:00:00 2001 From: Sergei Zimmerman Date: Thu, 15 May 2025 22:28:41 +0000 Subject: [PATCH 017/218] libutil: Add `LRUCache::getOrNullptr` For heavier objects it doesn't make sense to return a std::optional with the copy of the data, when it can be used by const reference. --- src/libutil/include/nix/util/lru-cache.hh | 46 +++++++++++++++++------ 1 file changed, 35 insertions(+), 11 deletions(-) diff --git a/src/libutil/include/nix/util/lru-cache.hh b/src/libutil/include/nix/util/lru-cache.hh index c9bcd7ee002..0834a8e7496 100644 --- a/src/libutil/include/nix/util/lru-cache.hh +++ b/src/libutil/include/nix/util/lru-cache.hh @@ -33,6 +33,18 @@ private: Data data; LRU lru; + /** + * Move this item to the back of the LRU list. + */ + void promote(LRU::iterator it) + { + /* Think of std::list iterators as stable pointers to the list node, + * which never get invalidated. Thus, we can reuse the same lru list + * element and just splice it to the back of the list without the need + * to update its value in the key -> list iterator map. */ + lru.splice(/*pos=*/lru.end(), /*other=*/lru, it); + } + public: LRUCache(size_t capacity) @@ -83,7 +95,9 @@ public: /** * Look up an item in the cache. If it exists, it becomes the most * recently used item. - * */ + * + * @returns corresponding cache entry, std::nullopt if it's not in the cache + */ template std::optional get(const K & key) { @@ -91,20 +105,30 @@ public: if (i == data.end()) return {}; - /** - * Move this item to the back of the LRU list. - * - * Think of std::list iterators as stable pointers to the list node, - * which never get invalidated. Thus, we can reuse the same lru list - * element and just splice it to the back of the list without the need - * to update its value in the key -> list iterator map. - */ auto & [it, value] = i->second; - lru.splice(/*pos=*/lru.end(), /*other=*/lru, it.it); - + promote(it.it); return value; } + /** + * Look up an item in the cache. If it exists, it becomes the most + * recently used item. + * + * @returns mutable pointer to the corresponding cache entry, nullptr if + * it's not in the cache + */ + template + Value * getOrNullptr(const K & key) + { + auto i = data.find(key); + if (i == data.end()) + return nullptr; + + auto & [it, value] = i->second; + promote(it.it); + return &value; + } + size_t size() const noexcept { return data.size(); From 5ea81f5b8f36f351981f53d27e85c6e4733c2b3e Mon Sep 17 00:00:00 2001 From: Sergei Zimmerman Date: Thu, 15 May 2025 23:07:25 +0000 Subject: [PATCH 018/218] libexpr: Actually cache line information in PosTable MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Previous code had a sneaky bug due to which no caching actually happened: ```cpp auto linesForInput = (*lines)[origin->offset]; ``` That should have been: ```cpp auto & linesForInput = (*lines)[origin->offset]; ``` See [1]. Now that it also makes sense to make the cache bound in side in order not to memoize all the sources without freeing any memory. The default cache size has been chosen somewhat arbitrarily to be ~64k origins. For reference, 25.05 nixpkgs has ~50k .nix files. Simple benchmark: ```nix let pkgs = import { }; in builtins.foldl' (acc: el: acc + el.line) 0 ( builtins.genList (x: builtins.unsafeGetAttrPos "gcc" pkgs) 10000 ) ``` (After) ``` $ hyperfine "result/bin/nix eval -f ./test.nix" Benchmark 1: result/bin/nix eval -f ./test.nix Time (mean ± σ): 292.7 ms ± 3.9 ms [User: 131.0 ms, System: 120.5 ms] Range (min … max): 288.1 ms … 300.5 ms 10 runs ``` (Before) ``` hyperfine "nix eval -f ./test.nix" Benchmark 1: nix eval -f ./test.nix Time (mean ± σ): 666.7 ms ± 6.4 ms [User: 428.3 ms, System: 191.2 ms] Range (min … max): 659.7 ms … 681.3 ms 10 runs ``` If the origin happens to be a `all-packages.nix` or similar in size then the difference is much more dramatic. [1]: https://www.github.com/lix-project/lix/commit/22e3f0e9875082be7f4eec8e3caeb134a7f1c05f --- src/libutil/include/nix/util/pos-table.hh | 18 +++++++++- src/libutil/pos-table.cc | 40 +++++++++++++++-------- 2 files changed, 44 insertions(+), 14 deletions(-) diff --git a/src/libutil/include/nix/util/pos-table.hh b/src/libutil/include/nix/util/pos-table.hh index ef170e0f14b..f64466c2124 100644 --- a/src/libutil/include/nix/util/pos-table.hh +++ b/src/libutil/include/nix/util/pos-table.hh @@ -4,6 +4,7 @@ #include #include +#include "nix/util/lru-cache.hh" #include "nix/util/pos-idx.hh" #include "nix/util/position.hh" #include "nix/util/sync.hh" @@ -37,10 +38,20 @@ public: }; private: + /** + * Vector of byte offsets (in the virtual input buffer) of initial line character's position. + * Sorted by construction. Binary search over it allows for efficient translation of arbitrary + * byte offsets in the virtual input buffer to its line + column position. + */ using Lines = std::vector; + /** + * Cache from byte offset in the virtual buffer of Origins -> @ref Lines in that origin. + */ + using LinesCache = LRUCache; std::map origins; - mutable Sync> lines; + + mutable Sync linesCache; const Origin * resolve(PosIdx p) const { @@ -56,6 +67,11 @@ private: } public: + PosTable(std::size_t linesCacheCapacity = 65536) + : linesCache(linesCacheCapacity) + { + } + Origin addOrigin(Pos::Origin origin, size_t size) { uint32_t offset = 0; diff --git a/src/libutil/pos-table.cc b/src/libutil/pos-table.cc index 5a61ffbc5e7..e50b1287317 100644 --- a/src/libutil/pos-table.cc +++ b/src/libutil/pos-table.cc @@ -15,21 +15,35 @@ Pos PosTable::operator[](PosIdx p) const const auto offset = origin->offsetOf(p); Pos result{0, 0, origin->origin}; - auto lines = this->lines.lock(); - auto linesForInput = (*lines)[origin->offset]; - - if (linesForInput.empty()) { - auto source = result.getSource().value_or(""); - const char * begin = source.data(); - for (Pos::LinesIterator it(source), end; it != end; it++) - linesForInput.push_back(it->data() - begin); - if (linesForInput.empty()) - linesForInput.push_back(0); + auto linesCache = this->linesCache.lock(); + + /* Try the origin's line cache */ + const auto * linesForInput = linesCache->getOrNullptr(origin->offset); + + auto fillCacheForOrigin = [](std::string_view content) { + auto contentLines = Lines(); + + const char * begin = content.data(); + for (Pos::LinesIterator it(content), end; it != end; it++) + contentLines.push_back(it->data() - begin); + if (contentLines.empty()) + contentLines.push_back(0); + + return contentLines; + }; + + /* Calculate line offsets and fill the cache */ + if (!linesForInput) { + auto originContent = result.getSource().value_or(""); + linesCache->upsert(origin->offset, fillCacheForOrigin(originContent)); + linesForInput = linesCache->getOrNullptr(origin->offset); } - // as above: the first line starts at byte 0 and is always present - auto lineStartOffset = std::prev(std::upper_bound(linesForInput.begin(), linesForInput.end(), offset)); - result.line = 1 + (lineStartOffset - linesForInput.begin()); + assert(linesForInput); + + // as above: the first line starts at byte 0 and is always present + auto lineStartOffset = std::prev(std::upper_bound(linesForInput->begin(), linesForInput->end(), offset)); + result.line = 1 + (lineStartOffset - linesForInput->begin()); result.column = 1 + (offset - *lineStartOffset); return result; } From cb16cd707c17db7179d1bad6efb08d92ed8cc7fd Mon Sep 17 00:00:00 2001 From: Peder Bergebakken Sundt Date: Fri, 16 May 2025 09:20:55 +0200 Subject: [PATCH 019/218] docs: remove repeated "allowedReferences" This is what write-good lints as a "lexical illusion" --- doc/manual/source/language/advanced-attributes.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/manual/source/language/advanced-attributes.md b/doc/manual/source/language/advanced-attributes.md index 4031f763a7f..a939847e1aa 100644 --- a/doc/manual/source/language/advanced-attributes.md +++ b/doc/manual/source/language/advanced-attributes.md @@ -73,7 +73,7 @@ Derivations can declare some infrequently used optional attributes. > **Warning** > - > If set to `true`, other advanced attributes such as [`allowedReferences`](#adv-attr-allowedReferences), [`allowedReferences`](#adv-attr-allowedReferences), [`allowedRequisites`](#adv-attr-allowedRequisites), + > If set to `true`, other advanced attributes such as [`allowedReferences`](#adv-attr-allowedReferences), [`allowedRequisites`](#adv-attr-allowedRequisites), [`disallowedReferences`](#adv-attr-disallowedReferences) and [`disallowedRequisites`](#adv-attr-disallowedRequisites), maxSize, and maxClosureSize. will have no effect. From ea5302c4a28e254d6c72b842a6dd469c929f7f94 Mon Sep 17 00:00:00 2001 From: Peder Bergebakken Sundt Date: Fri, 16 May 2025 09:52:03 +0200 Subject: [PATCH 020/218] docs: remove lexical illusions detected with write-good I made this this non-markdown aware tool somewhat behave with some cursed fd+pandoc invocations --- .../source/store/derivation/outputs/content-address.md | 6 +++--- doc/manual/source/store/derivation/outputs/index.md | 2 +- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/doc/manual/source/store/derivation/outputs/content-address.md b/doc/manual/source/store/derivation/outputs/content-address.md index 7fc689fb318..4d51303480d 100644 --- a/doc/manual/source/store/derivation/outputs/content-address.md +++ b/doc/manual/source/store/derivation/outputs/content-address.md @@ -23,7 +23,7 @@ The output spec for an output with a fixed content addresses additionally contai > **Design note** > > In principle, the output spec could also specify the references the store object should have, since the references and file system objects are equally parts of a content-addressed store object proper that contribute to its content-addressed. -> However, at this time, the references are not not done because all fixed content-addressed outputs are required to have no references (including no self-reference). +> However, at this time, the references are not done because all fixed content-addressed outputs are required to have no references (including no self-reference). > > Also in principle, rather than specifying the references and file system object data with separate hashes, a single hash that constraints both could be used. > This could be done with the final store path's digest, or better yet, the hash that will become the store path's digest before it is truncated. @@ -116,7 +116,7 @@ Because the derivation output is not fixed (just like with [input addressing]), > (The "environment", in this case, consists of attributes such as the Operating System Nix runs atop, along with the operating-system-specific privileges that Nix has been granted. > Because of how conventional operating systems like macos, Linux, etc. work, granting builders *fewer* privileges may ironically require that Nix be run with *more* privileges.) -That said, derivations producing floating content-addressed outputs may declare their builders as impure (like the builders of derivations producing producing fixed outputs). +That said, derivations producing floating content-addressed outputs may declare their builders as impure (like the builders of derivations producing fixed outputs). This is provisionally supported as part of the [`impure-derivations`][xp-feature-impure-derivations] experimental feature. ### Compatibility negotiation @@ -144,7 +144,7 @@ A *deterministic* content-addressing derivation should produce outputs with the The choice of provisional store path can be thought of as an impurity, since it is an arbitrary choice. If provisional outputs paths are deterministically chosen, we are in the first branch of part (1). - The builder the data it produces based on it in arbitrary ways, but this gets us closer to to [input addressing]. + The builder the data it produces based on it in arbitrary ways, but this gets us closer to [input addressing]. Deterministically choosing the provisional path may be considered "complete sandboxing" by removing an impurity, but this is unsatisfactory Significant changes should add the following header, which moves them to the top. diff --git a/doc/manual/substitute.py b/doc/manual/substitute.py index a8b11d93250..6e27c338818 100644 --- a/doc/manual/substitute.py +++ b/doc/manual/substitute.py @@ -57,6 +57,9 @@ def recursive_replace(data: dict[str, t.Any], book_root: Path, search_path: Path ).replace( '@docroot@', ("../" * len(path_to_chapter.parent.parts) or "./")[:-1] + ).replace( + '@_at_', + '@' ), sub_items = [ recursive_replace(sub_item, book_root, search_path) From 7d89e46f65fb5cafa6528c3ff39e4a53d83e99ec Mon Sep 17 00:00:00 2001 From: Eelco Dolstra Date: Wed, 14 May 2025 14:24:57 +0200 Subject: [PATCH 023/218] Sync: Support moving out of another Sync --- src/libutil/include/nix/util/sync.hh | 1 + 1 file changed, 1 insertion(+) diff --git a/src/libutil/include/nix/util/sync.hh b/src/libutil/include/nix/util/sync.hh index 0c3e1f52836..4b9d546d2b7 100644 --- a/src/libutil/include/nix/util/sync.hh +++ b/src/libutil/include/nix/util/sync.hh @@ -39,6 +39,7 @@ public: SyncBase() { } SyncBase(const T & data) : data(data) { } SyncBase(T && data) noexcept : data(std::move(data)) { } + SyncBase(SyncBase && other) noexcept : data(std::move(*other.lock())) { } template class Lock From efcb9e36a9b5d6e31ce3513d5fd0571757c41c9b Mon Sep 17 00:00:00 2001 From: Eelco Dolstra Date: Wed, 14 May 2025 14:17:57 +0200 Subject: [PATCH 024/218] Remove global fetcher cache The cache is now part of fetchers::Settings. --- src/libcmd/common-eval-args.cc | 20 ++++++++++++++++--- src/libcmd/installable-value.cc | 2 +- src/libexpr/eval.cc | 3 ++- src/libexpr/primops.cc | 1 + src/libexpr/primops/fetchTree.cc | 3 ++- src/libfetchers/cache.cc | 9 ++++++--- src/libfetchers/fetch-to-store.cc | 12 ++++++----- src/libfetchers/fetchers.cc | 2 +- src/libfetchers/git-utils.cc | 7 ++++--- src/libfetchers/git.cc | 16 +++++++-------- src/libfetchers/github.cc | 10 +++++----- src/libfetchers/include/nix/fetchers/cache.hh | 2 -- .../include/nix/fetchers/fetch-settings.hh | 9 +++++++++ .../include/nix/fetchers/fetch-to-store.hh | 1 + .../include/nix/fetchers/git-utils.hh | 4 ++-- .../include/nix/fetchers/tarball.hh | 1 + src/libfetchers/mercurial.cc | 10 +++++----- src/libfetchers/path.cc | 3 ++- src/libfetchers/registry.cc | 2 +- src/libfetchers/tarball.cc | 16 +++++++++------ src/libflake/flake.cc | 4 ++-- src/nix-channel/nix-channel.cc | 8 +++++--- src/nix/flake.cc | 2 +- 23 files changed, 93 insertions(+), 54 deletions(-) diff --git a/src/libcmd/common-eval-args.cc b/src/libcmd/common-eval-args.cc index e087678c26f..d275beb12c3 100644 --- a/src/libcmd/common-eval-args.cc +++ b/src/libcmd/common-eval-args.cc @@ -34,7 +34,12 @@ EvalSettings evalSettings { auto flakeRef = parseFlakeRef(fetchSettings, std::string { rest }, {}, true, false); debug("fetching flake search path element '%s''", rest); auto [accessor, lockedRef] = flakeRef.resolve(state.store).lazyFetch(state.store); - auto storePath = nix::fetchToStore(*state.store, SourcePath(accessor), FetchMode::Copy, lockedRef.input.getName()); + auto storePath = nix::fetchToStore( + state.fetchSettings, + *state.store, + SourcePath(accessor), + FetchMode::Copy, + lockedRef.input.getName()); state.allowPath(storePath); return state.storePath(storePath); }, @@ -177,7 +182,11 @@ SourcePath lookupFileArg(EvalState & state, std::string_view s, const Path * bas state.store, state.fetchSettings, EvalSettings::resolvePseudoUrl(s)); - auto storePath = fetchToStore(*state.store, SourcePath(accessor), FetchMode::Copy); + auto storePath = fetchToStore( + state.fetchSettings, + *state.store, + SourcePath(accessor), + FetchMode::Copy); return state.storePath(storePath); } @@ -185,7 +194,12 @@ SourcePath lookupFileArg(EvalState & state, std::string_view s, const Path * bas experimentalFeatureSettings.require(Xp::Flakes); auto flakeRef = parseFlakeRef(fetchSettings, std::string(s.substr(6)), {}, true, false); auto [accessor, lockedRef] = flakeRef.resolve(state.store).lazyFetch(state.store); - auto storePath = nix::fetchToStore(*state.store, SourcePath(accessor), FetchMode::Copy, lockedRef.input.getName()); + auto storePath = nix::fetchToStore( + state.fetchSettings, + *state.store, + SourcePath(accessor), + FetchMode::Copy, + lockedRef.input.getName()); state.allowPath(storePath); return state.storePath(storePath); } diff --git a/src/libcmd/installable-value.cc b/src/libcmd/installable-value.cc index d9ac3a29e7a..e92496347e0 100644 --- a/src/libcmd/installable-value.cc +++ b/src/libcmd/installable-value.cc @@ -45,7 +45,7 @@ ref InstallableValue::require(ref installable) std::optional InstallableValue::trySinglePathToDerivedPaths(Value & v, const PosIdx pos, std::string_view errorCtx) { if (v.type() == nPath) { - auto storePath = fetchToStore(*state->store, v.path(), FetchMode::Copy); + auto storePath = fetchToStore(state->fetchSettings, *state->store, v.path(), FetchMode::Copy); return {{ .path = DerivedPath::Opaque { .path = std::move(storePath), diff --git a/src/libexpr/eval.cc b/src/libexpr/eval.cc index ab7f33d5f89..4c3aec977e6 100644 --- a/src/libexpr/eval.cc +++ b/src/libexpr/eval.cc @@ -2423,6 +2423,7 @@ StorePath EvalState::copyPathToStore(NixStringContext & context, const SourcePat ? *dstPathCached : [&]() { auto dstPath = fetchToStore( + fetchSettings, *store, path.resolveSymlinks(SymlinkResolution::Ancestors), settings.readOnlyMode ? FetchMode::DryRun : FetchMode::Copy, @@ -3125,7 +3126,7 @@ std::optional EvalState::resolveLookupPathPath(const LookupPath::Pat store, fetchSettings, EvalSettings::resolvePseudoUrl(value)); - auto storePath = fetchToStore(*store, SourcePath(accessor), FetchMode::Copy); + auto storePath = fetchToStore(fetchSettings, *store, SourcePath(accessor), FetchMode::Copy); return finish(this->storePath(storePath)); } catch (Error & e) { logWarning({ diff --git a/src/libexpr/primops.cc b/src/libexpr/primops.cc index 535a9a50117..d13314ffba9 100644 --- a/src/libexpr/primops.cc +++ b/src/libexpr/primops.cc @@ -2545,6 +2545,7 @@ static void addPath( if (!expectedHash || !state.store->isValidPath(*expectedStorePath)) { auto dstPath = fetchToStore( + state.fetchSettings, *state.store, path.resolveSymlinks(), settings.readOnlyMode ? FetchMode::DryRun : FetchMode::Copy, diff --git a/src/libexpr/primops/fetchTree.cc b/src/libexpr/primops/fetchTree.cc index 745705e04c1..f262507dffa 100644 --- a/src/libexpr/primops/fetchTree.cc +++ b/src/libexpr/primops/fetchTree.cc @@ -537,11 +537,12 @@ static void fetch(EvalState & state, const PosIdx pos, Value * * args, Value & v auto storePath = unpack ? fetchToStore( + state.fetchSettings, *state.store, fetchers::downloadTarball(state.store, state.fetchSettings, *url), FetchMode::Copy, name) - : fetchers::downloadFile(state.store, *url, name).storePath; + : fetchers::downloadFile(state.store, state.fetchSettings, *url, name).storePath; if (expectedHash) { auto hash = unpack diff --git a/src/libfetchers/cache.cc b/src/libfetchers/cache.cc index d369d213f51..9a2531ba526 100644 --- a/src/libfetchers/cache.cc +++ b/src/libfetchers/cache.cc @@ -1,4 +1,5 @@ #include "nix/fetchers/cache.hh" +#include "nix/fetchers/fetch-settings.hh" #include "nix/util/users.hh" #include "nix/store/sqlite.hh" #include "nix/util/sync.hh" @@ -162,10 +163,12 @@ struct CacheImpl : Cache } }; -ref getCache() +ref Settings::getCache() const { - static auto cache = std::make_shared(); - return ref(cache); + auto cache(_cache.lock()); + if (!*cache) + *cache = std::make_shared(); + return ref(*cache); } } diff --git a/src/libfetchers/fetch-to-store.cc b/src/libfetchers/fetch-to-store.cc index f1b02f4e0a8..f7ab32322ef 100644 --- a/src/libfetchers/fetch-to-store.cc +++ b/src/libfetchers/fetch-to-store.cc @@ -1,13 +1,14 @@ #include "nix/fetchers/fetch-to-store.hh" #include "nix/fetchers/fetchers.hh" +#include "nix/fetchers/fetch-settings.hh" namespace nix { fetchers::Cache::Key makeFetchToStoreCacheKey( - const std::string &name, - const std::string &fingerprint, + const std::string & name, + const std::string & fingerprint, ContentAddressMethod method, - const std::string &path) + const std::string & path) { return fetchers::Cache::Key{"fetchToStore", { {"name", name}, @@ -19,6 +20,7 @@ fetchers::Cache::Key makeFetchToStoreCacheKey( } StorePath fetchToStore( + const fetchers::Settings & settings, Store & store, const SourcePath & path, FetchMode mode, @@ -34,7 +36,7 @@ StorePath fetchToStore( if (!filter && path.accessor->fingerprint) { cacheKey = makeFetchToStoreCacheKey(std::string{name}, *path.accessor->fingerprint, method, path.path.abs()); - if (auto res = fetchers::getCache()->lookupStorePath(*cacheKey, store)) { + if (auto res = settings.getCache()->lookupStorePath(*cacheKey, store)) { debug("store path cache hit for '%s'", path); return res->storePath; } @@ -56,7 +58,7 @@ StorePath fetchToStore( debug(mode == FetchMode::DryRun ? "hashed '%s'" : "copied '%s' to '%s'", path, store.printStorePath(storePath)); if (cacheKey && mode == FetchMode::Copy) - fetchers::getCache()->upsert(*cacheKey, store, {}, storePath); + settings.getCache()->upsert(*cacheKey, store, {}, storePath); return storePath; } diff --git a/src/libfetchers/fetchers.cc b/src/libfetchers/fetchers.cc index e0161dee8fe..35201e8aef1 100644 --- a/src/libfetchers/fetchers.cc +++ b/src/libfetchers/fetchers.cc @@ -198,7 +198,7 @@ std::pair Input::fetchToStore(ref store) const try { auto [accessor, result] = getAccessorUnchecked(store); - auto storePath = nix::fetchToStore(*store, SourcePath(accessor), FetchMode::Copy, result.getName()); + auto storePath = nix::fetchToStore(*settings, *store, SourcePath(accessor), FetchMode::Copy, result.getName()); auto narHash = store->queryPathInfo(storePath)->narHash; result.attrs.insert_or_assign("narHash", narHash.to_string(HashFormat::SRI, true)); diff --git a/src/libfetchers/git-utils.cc b/src/libfetchers/git-utils.cc index 58ba398896b..60d8edee06b 100644 --- a/src/libfetchers/git-utils.cc +++ b/src/libfetchers/git-utils.cc @@ -1,6 +1,7 @@ #include "nix/fetchers/git-utils.hh" #include "nix/fetchers/git-lfs-fetch.hh" #include "nix/fetchers/cache.hh" +#include "nix/fetchers/fetch-settings.hh" #include "nix/util/finally.hh" #include "nix/util/processes.hh" #include "nix/util/signals.hh" @@ -610,18 +611,18 @@ struct GitRepoImpl : GitRepo, std::enable_shared_from_this throw Error("Commit signature verification on commit %s failed: %s", rev.gitRev(), output); } - Hash treeHashToNarHash(const Hash & treeHash) override + Hash treeHashToNarHash(const fetchers::Settings & settings, const Hash & treeHash) override { auto accessor = getAccessor(treeHash, false, ""); fetchers::Cache::Key cacheKey{"treeHashToNarHash", {{"treeHash", treeHash.gitRev()}}}; - if (auto res = fetchers::getCache()->lookup(cacheKey)) + if (auto res = settings.getCache()->lookup(cacheKey)) return Hash::parseAny(fetchers::getStrAttr(*res, "narHash"), HashAlgorithm::SHA256); auto narHash = accessor->hashPath(CanonPath::root); - fetchers::getCache()->upsert(cacheKey, fetchers::Attrs({{"narHash", narHash.to_string(HashFormat::SRI, true)}})); + settings.getCache()->upsert(cacheKey, fetchers::Attrs({{"narHash", narHash.to_string(HashFormat::SRI, true)}})); return narHash; } diff --git a/src/libfetchers/git.cc b/src/libfetchers/git.cc index 492867dc404..f716094b6ae 100644 --- a/src/libfetchers/git.cc +++ b/src/libfetchers/git.cc @@ -480,11 +480,11 @@ struct GitInputScheme : InputScheme return repoInfo; } - uint64_t getLastModified(const RepoInfo & repoInfo, const std::filesystem::path & repoDir, const Hash & rev) const + uint64_t getLastModified(const Settings & settings, const RepoInfo & repoInfo, const std::filesystem::path & repoDir, const Hash & rev) const { Cache::Key key{"gitLastModified", {{"rev", rev.gitRev()}}}; - auto cache = getCache(); + auto cache = settings.getCache(); if (auto res = cache->lookup(key)) return getIntAttr(*res, "lastModified"); @@ -496,11 +496,11 @@ struct GitInputScheme : InputScheme return lastModified; } - uint64_t getRevCount(const RepoInfo & repoInfo, const std::filesystem::path & repoDir, const Hash & rev) const + uint64_t getRevCount(const Settings & settings, const RepoInfo & repoInfo, const std::filesystem::path & repoDir, const Hash & rev) const { Cache::Key key{"gitRevCount", {{"rev", rev.gitRev()}}}; - auto cache = getCache(); + auto cache = settings.getCache(); if (auto revCountAttrs = cache->lookup(key)) return getIntAttr(*revCountAttrs, "revCount"); @@ -678,12 +678,12 @@ struct GitInputScheme : InputScheme Attrs infoAttrs({ {"rev", rev.gitRev()}, - {"lastModified", getLastModified(repoInfo, repoDir, rev)}, + {"lastModified", getLastModified(*input.settings, repoInfo, repoDir, rev)}, }); if (!getShallowAttr(input)) infoAttrs.insert_or_assign("revCount", - getRevCount(repoInfo, repoDir, rev)); + getRevCount(*input.settings, repoInfo, repoDir, rev)); printTalkative("using revision %s of repo '%s'", rev.gitRev(), repoInfo.locationToArg()); @@ -799,7 +799,7 @@ struct GitInputScheme : InputScheme input.attrs.insert_or_assign("rev", rev.gitRev()); input.attrs.insert_or_assign("revCount", - rev == nullRev ? 0 : getRevCount(repoInfo, repoPath, rev)); + rev == nullRev ? 0 : getRevCount(*input.settings, repoInfo, repoPath, rev)); verifyCommit(input, repo); } else { @@ -818,7 +818,7 @@ struct GitInputScheme : InputScheme input.attrs.insert_or_assign( "lastModified", repoInfo.workdirInfo.headRev - ? getLastModified(repoInfo, repoPath, *repoInfo.workdirInfo.headRev) + ? getLastModified(*input.settings, repoInfo, repoPath, *repoInfo.workdirInfo.headRev) : 0); return {accessor, std::move(input)}; diff --git a/src/libfetchers/github.cc b/src/libfetchers/github.cc index 4fcad5f2631..511afabe979 100644 --- a/src/libfetchers/github.cc +++ b/src/libfetchers/github.cc @@ -265,7 +265,7 @@ struct GitArchiveInputScheme : InputScheme input.attrs.erase("ref"); input.attrs.insert_or_assign("rev", rev->gitRev()); - auto cache = getCache(); + auto cache = input.settings->getCache(); Cache::Key treeHashKey{"gitRevToTreeHash", {{"rev", rev->gitRev()}}}; Cache::Key lastModifiedKey{"gitRevToLastModified", {{"rev", rev->gitRev()}}}; @@ -407,7 +407,7 @@ struct GitHubInputScheme : GitArchiveInputScheme auto json = nlohmann::json::parse( readFile( store->toRealPath( - downloadFile(store, url, "source", headers).storePath))); + downloadFile(store, *input.settings, url, "source", headers).storePath))); return RefInfo { .rev = Hash::parseAny(std::string { json["sha"] }, HashAlgorithm::SHA1), @@ -481,7 +481,7 @@ struct GitLabInputScheme : GitArchiveInputScheme auto json = nlohmann::json::parse( readFile( store->toRealPath( - downloadFile(store, url, "source", headers).storePath))); + downloadFile(store, *input.settings, url, "source", headers).storePath))); if (json.is_array() && json.size() >= 1 && json[0]["id"] != nullptr) { return RefInfo { @@ -551,7 +551,7 @@ struct SourceHutInputScheme : GitArchiveInputScheme std::string refUri; if (ref == "HEAD") { auto file = store->toRealPath( - downloadFile(store, fmt("%s/HEAD", base_url), "source", headers).storePath); + downloadFile(store, *input.settings, fmt("%s/HEAD", base_url), "source", headers).storePath); std::ifstream is(file); std::string line; getline(is, line); @@ -567,7 +567,7 @@ struct SourceHutInputScheme : GitArchiveInputScheme std::regex refRegex(refUri); auto file = store->toRealPath( - downloadFile(store, fmt("%s/info/refs", base_url), "source", headers).storePath); + downloadFile(store, *input.settings, fmt("%s/info/refs", base_url), "source", headers).storePath); std::ifstream is(file); std::string line; diff --git a/src/libfetchers/include/nix/fetchers/cache.hh b/src/libfetchers/include/nix/fetchers/cache.hh index 5b9319d774b..6ac693183f9 100644 --- a/src/libfetchers/include/nix/fetchers/cache.hh +++ b/src/libfetchers/include/nix/fetchers/cache.hh @@ -91,6 +91,4 @@ struct Cache Store & store) = 0; }; -ref getCache(); - } diff --git a/src/libfetchers/include/nix/fetchers/fetch-settings.hh b/src/libfetchers/include/nix/fetchers/fetch-settings.hh index 54c42084344..7bd04db53b0 100644 --- a/src/libfetchers/include/nix/fetchers/fetch-settings.hh +++ b/src/libfetchers/include/nix/fetchers/fetch-settings.hh @@ -3,6 +3,8 @@ #include "nix/util/types.hh" #include "nix/util/configuration.hh" +#include "nix/util/ref.hh" +#include "nix/util/sync.hh" #include #include @@ -11,6 +13,8 @@ namespace nix::fetchers { +struct Cache; + struct Settings : public Config { Settings(); @@ -110,6 +114,11 @@ struct Settings : public Config When empty, disables the global flake registry. )", {}, true, Xp::Flakes}; + + ref getCache() const; + +private: + mutable Sync> _cache; }; } diff --git a/src/libfetchers/include/nix/fetchers/fetch-to-store.hh b/src/libfetchers/include/nix/fetchers/fetch-to-store.hh index 44c33c147ed..a52d567ecfb 100644 --- a/src/libfetchers/include/nix/fetchers/fetch-to-store.hh +++ b/src/libfetchers/include/nix/fetchers/fetch-to-store.hh @@ -15,6 +15,7 @@ enum struct FetchMode { DryRun, Copy }; * Copy the `path` to the Nix store. */ StorePath fetchToStore( + const fetchers::Settings & settings, Store & store, const SourcePath & path, FetchMode mode, diff --git a/src/libfetchers/include/nix/fetchers/git-utils.hh b/src/libfetchers/include/nix/fetchers/git-utils.hh index 1506f8509e4..2926deb4f44 100644 --- a/src/libfetchers/include/nix/fetchers/git-utils.hh +++ b/src/libfetchers/include/nix/fetchers/git-utils.hh @@ -5,7 +5,7 @@ namespace nix { -namespace fetchers { struct PublicKey; } +namespace fetchers { struct PublicKey; struct Settings; } /** * A sink that writes into a Git repository. Note that nothing may be written @@ -115,7 +115,7 @@ struct GitRepo * Given a Git tree hash, compute the hash of its NAR * serialisation. This is memoised on-disk. */ - virtual Hash treeHashToNarHash(const Hash & treeHash) = 0; + virtual Hash treeHashToNarHash(const fetchers::Settings & settings, const Hash & treeHash) = 0; /** * If the specified Git object is a directory with a single entry diff --git a/src/libfetchers/include/nix/fetchers/tarball.hh b/src/libfetchers/include/nix/fetchers/tarball.hh index 691142091fa..2c5ea209f01 100644 --- a/src/libfetchers/include/nix/fetchers/tarball.hh +++ b/src/libfetchers/include/nix/fetchers/tarball.hh @@ -26,6 +26,7 @@ struct DownloadFileResult DownloadFileResult downloadFile( ref store, + const Settings & settings, const std::string & url, const std::string & name, const Headers & headers = {}); diff --git a/src/libfetchers/mercurial.cc b/src/libfetchers/mercurial.cc index 74e9fd08998..0b63876deae 100644 --- a/src/libfetchers/mercurial.cc +++ b/src/libfetchers/mercurial.cc @@ -253,13 +253,13 @@ struct MercurialInputScheme : InputScheme }}; if (!input.getRev()) { - if (auto res = getCache()->lookupWithTTL(refToRevKey)) + if (auto res = input.settings->getCache()->lookupWithTTL(refToRevKey)) input.attrs.insert_or_assign("rev", getRevAttr(*res, "rev").gitRev()); } /* If we have a rev, check if we have a cached store path. */ if (auto rev = input.getRev()) { - if (auto res = getCache()->lookupStorePath(revInfoKey(*rev), *store)) + if (auto res = input.settings->getCache()->lookupStorePath(revInfoKey(*rev), *store)) return makeResult(res->value, res->storePath); } @@ -309,7 +309,7 @@ struct MercurialInputScheme : InputScheme /* Now that we have the rev, check the cache again for a cached store path. */ - if (auto res = getCache()->lookupStorePath(revInfoKey(rev), *store)) + if (auto res = input.settings->getCache()->lookupStorePath(revInfoKey(rev), *store)) return makeResult(res->value, res->storePath); Path tmpDir = createTempDir(); @@ -326,9 +326,9 @@ struct MercurialInputScheme : InputScheme }); if (!origRev) - getCache()->upsert(refToRevKey, {{"rev", rev.gitRev()}}); + input.settings->getCache()->upsert(refToRevKey, {{"rev", rev.gitRev()}}); - getCache()->upsert(revInfoKey(rev), *store, infoAttrs, storePath); + input.settings->getCache()->upsert(revInfoKey(rev), *store, infoAttrs, storePath); return makeResult(infoAttrs, std::move(storePath)); } diff --git a/src/libfetchers/path.cc b/src/libfetchers/path.cc index 670397cb6b1..9239fd27466 100644 --- a/src/libfetchers/path.cc +++ b/src/libfetchers/path.cc @@ -4,6 +4,7 @@ #include "nix/fetchers/store-path-accessor.hh" #include "nix/fetchers/cache.hh" #include "nix/fetchers/fetch-to-store.hh" +#include "nix/fetchers/fetch-settings.hh" namespace nix::fetchers { @@ -149,7 +150,7 @@ struct PathInputScheme : InputScheme auto fp = getFingerprint(store, input); if (fp) { auto cacheKey = makeFetchToStoreCacheKey(input.getName(), *fp, method, "/"); - fetchers::getCache()->upsert(cacheKey, *store, {}, *storePath); + input.settings->getCache()->upsert(cacheKey, *store, {}, *storePath); } /* Trust the lastModified value supplied by the user, if diff --git a/src/libfetchers/registry.cc b/src/libfetchers/registry.cc index bfaf9569a4e..335935f53af 100644 --- a/src/libfetchers/registry.cc +++ b/src/libfetchers/registry.cc @@ -156,7 +156,7 @@ static std::shared_ptr getGlobalRegistry(const Settings & settings, re } if (!isAbsolute(path)) { - auto storePath = downloadFile(store, path, "flake-registry.json").storePath; + auto storePath = downloadFile(store, settings, path, "flake-registry.json").storePath; if (auto store2 = store.dynamic_pointer_cast()) store2->addPermRoot(storePath, getCacheDir() + "/flake-registry.json"); path = store->toRealPath(storePath); diff --git a/src/libfetchers/tarball.cc b/src/libfetchers/tarball.cc index 1bd7e3e5947..b0822cc3301 100644 --- a/src/libfetchers/tarball.cc +++ b/src/libfetchers/tarball.cc @@ -9,11 +9,13 @@ #include "nix/fetchers/store-path-accessor.hh" #include "nix/store/store-api.hh" #include "nix/fetchers/git-utils.hh" +#include "nix/fetchers/fetch-settings.hh" namespace nix::fetchers { DownloadFileResult downloadFile( ref store, + const Settings & settings, const std::string & url, const std::string & name, const Headers & headers) @@ -25,7 +27,7 @@ DownloadFileResult downloadFile( {"name", name}, }}}; - auto cached = getCache()->lookupStorePath(key, *store); + auto cached = settings.getCache()->lookupStorePath(key, *store); auto useCached = [&]() -> DownloadFileResult { @@ -92,7 +94,7 @@ DownloadFileResult downloadFile( key.second.insert_or_assign("url", url); assert(!res.urls.empty()); infoAttrs.insert_or_assign("url", *res.urls.rbegin()); - getCache()->upsert(key, *store, infoAttrs, *storePath); + settings.getCache()->upsert(key, *store, infoAttrs, *storePath); } return { @@ -104,13 +106,14 @@ DownloadFileResult downloadFile( } static DownloadTarballResult downloadTarball_( + const Settings & settings, const std::string & url, const Headers & headers, const std::string & displayPrefix) { Cache::Key cacheKey{"tarball", {{"url", url}}}; - auto cached = getCache()->lookupExpired(cacheKey); + auto cached = settings.getCache()->lookupExpired(cacheKey); auto attrsToResult = [&](const Attrs & infoAttrs) { @@ -196,7 +199,7 @@ static DownloadTarballResult downloadTarball_( /* Insert a cache entry for every URL in the redirect chain. */ for (auto & url : res->urls) { cacheKey.second.insert_or_assign("url", url); - getCache()->upsert(cacheKey, infoAttrs); + settings.getCache()->upsert(cacheKey, infoAttrs); } // FIXME: add a cache entry for immutableUrl? That could allow @@ -341,7 +344,7 @@ struct FileInputScheme : CurlInputScheme the Nix store directly, since there is little deduplication benefit in using the Git cache for single big files like tarballs. */ - auto file = downloadFile(store, getStrAttr(input.attrs, "url"), input.getName()); + auto file = downloadFile(store, *input.settings, getStrAttr(input.attrs, "url"), input.getName()); auto narHash = store->queryPathInfo(file.storePath)->narHash; input.attrs.insert_or_assign("narHash", narHash.to_string(HashFormat::SRI, true)); @@ -373,6 +376,7 @@ struct TarballInputScheme : CurlInputScheme auto input(_input); auto result = downloadTarball_( + *input.settings, getStrAttr(input.attrs, "url"), {}, "«" + input.to_string() + "»"); @@ -390,7 +394,7 @@ struct TarballInputScheme : CurlInputScheme input.attrs.insert_or_assign("lastModified", uint64_t(result.lastModified)); input.attrs.insert_or_assign("narHash", - getTarballCache()->treeHashToNarHash(result.treeHash).to_string(HashFormat::SRI, true)); + getTarballCache()->treeHashToNarHash(*input.settings, result.treeHash).to_string(HashFormat::SRI, true)); return {result.accessor, input}; } diff --git a/src/libflake/flake.cc b/src/libflake/flake.cc index 987c9f610af..3277cc05bad 100644 --- a/src/libflake/flake.cc +++ b/src/libflake/flake.cc @@ -30,7 +30,7 @@ static StorePath copyInputToStore( const fetchers::Input & originalInput, ref accessor) { - auto storePath = fetchToStore(*state.store, accessor, FetchMode::Copy, input.getName()); + auto storePath = fetchToStore(*input.settings, *state.store, accessor, FetchMode::Copy, input.getName()); state.allowPath(storePath); @@ -276,7 +276,7 @@ static Flake readFlake( state.symbols[setting.name], std::string(state.forceStringNoCtx(*setting.value, setting.pos, ""))); else if (setting.value->type() == nPath) { - auto storePath = fetchToStore(*state.store, setting.value->path(), FetchMode::Copy); + auto storePath = fetchToStore(state.fetchSettings, *state.store, setting.value->path(), FetchMode::Copy); flake.config.settings.emplace( state.symbols[setting.name], state.store->printStorePath(storePath)); diff --git a/src/nix-channel/nix-channel.cc b/src/nix-channel/nix-channel.cc index 9db13022081..bc1164a4345 100644 --- a/src/nix-channel/nix-channel.cc +++ b/src/nix-channel/nix-channel.cc @@ -4,9 +4,11 @@ #include "nix/store/filetransfer.hh" #include "nix/store/store-open.hh" #include "nix/cmd/legacy.hh" +#include "nix/cmd/common-eval-args.hh" #include "nix/expr/eval-settings.hh" // for defexpr #include "nix/util/users.hh" #include "nix/fetchers/tarball.hh" +#include "nix/fetchers/fetch-settings.hh" #include "self-exe.hh" #include "man-pages.hh" @@ -114,7 +116,7 @@ static void update(const StringSet & channelNames) // We want to download the url to a file to see if it's a tarball while also checking if we // got redirected in the process, so that we can grab the various parts of a nix channel // definition from a consistent location if the redirect changes mid-download. - auto result = fetchers::downloadFile(store, url, std::string(baseNameOf(url))); + auto result = fetchers::downloadFile(store, fetchSettings, url, std::string(baseNameOf(url))); auto filename = store->toRealPath(result.storePath); url = result.effectiveUrl; @@ -128,9 +130,9 @@ static void update(const StringSet & channelNames) if (!unpacked) { // Download the channel tarball. try { - filename = store->toRealPath(fetchers::downloadFile(store, url + "/nixexprs.tar.xz", "nixexprs.tar.xz").storePath); + filename = store->toRealPath(fetchers::downloadFile(store, fetchSettings, url + "/nixexprs.tar.xz", "nixexprs.tar.xz").storePath); } catch (FileTransferError & e) { - filename = store->toRealPath(fetchers::downloadFile(store, url + "/nixexprs.tar.bz2", "nixexprs.tar.bz2").storePath); + filename = store->toRealPath(fetchers::downloadFile(store, fetchSettings, url + "/nixexprs.tar.bz2", "nixexprs.tar.bz2").storePath); } } // Regardless of where it came from, add the expression representing this channel to accumulated expression diff --git a/src/nix/flake.cc b/src/nix/flake.cc index ea83bb54c84..fdeedf9c1ff 100644 --- a/src/nix/flake.cc +++ b/src/nix/flake.cc @@ -1488,7 +1488,7 @@ struct CmdFlakePrefetch : FlakeCommand, MixJSON auto originalRef = getFlakeRef(); auto resolvedRef = originalRef.resolve(store); auto [accessor, lockedRef] = resolvedRef.lazyFetch(store); - auto storePath = fetchToStore(*store, accessor, FetchMode::Copy, lockedRef.input.getName()); + auto storePath = fetchToStore(getEvalState()->fetchSettings, *store, accessor, FetchMode::Copy, lockedRef.input.getName()); auto hash = store->queryPathInfo(storePath)->narHash; if (json) { From b37d1cdd8ecb25724d52efc67dc26e4249c28801 Mon Sep 17 00:00:00 2001 From: Graham Christensen Date: Mon, 7 Apr 2025 09:19:58 -0400 Subject: [PATCH 025/218] Rename `nix profile install` to `nix profile add`. --- src/libcmd/installables.cc | 2 +- src/nix/profile-add.md | 37 ++++++++++++++++++++++++++++++ src/nix/profile-install.md | 34 ---------------------------- src/nix/profile.cc | 34 ++++++++++++++-------------- tests/functional/nix-profile.sh | 40 ++++++++++++++++----------------- 5 files changed, 75 insertions(+), 72 deletions(-) create mode 100644 src/nix/profile-add.md delete mode 100644 src/nix/profile-install.md diff --git a/src/libcmd/installables.cc b/src/libcmd/installables.cc index 2d60248901e..836aded96f8 100644 --- a/src/libcmd/installables.cc +++ b/src/libcmd/installables.cc @@ -849,7 +849,7 @@ RawInstallablesCommand::RawInstallablesCommand() void RawInstallablesCommand::applyDefaultInstallables(std::vector & rawInstallables) { if (rawInstallables.empty()) { - // FIXME: commands like "nix profile install" should not have a + // FIXME: commands like "nix profile add" should not have a // default, probably. rawInstallables.push_back("."); } diff --git a/src/nix/profile-add.md b/src/nix/profile-add.md new file mode 100644 index 00000000000..0bb65d8e696 --- /dev/null +++ b/src/nix/profile-add.md @@ -0,0 +1,37 @@ +R""( + +# Examples + +- Add a package from Nixpkgs: + + ```console + # nix profile add nixpkgs#hello + ``` + +- Add a package from a specific branch of Nixpkgs: + + ```console + # nix profile add nixpkgs/release-20.09#hello + ``` + +- Add a package from a specific revision of Nixpkgs: + + ```console + # nix profile add nixpkgs/d73407e8e6002646acfdef0e39ace088bacc83da#hello + ``` + +- Add a specific output of a package: + + ```console + # nix profile add nixpkgs#bash^man + ``` + +# Description + +This command adds [_installables_](./nix.md#installables) to a Nix profile. + +> **Note** +> +> `nix profile install` is an alias for `nix profile add` in Determinate Nix. + +)"" diff --git a/src/nix/profile-install.md b/src/nix/profile-install.md deleted file mode 100644 index 4c0f82c09e5..00000000000 --- a/src/nix/profile-install.md +++ /dev/null @@ -1,34 +0,0 @@ -R""( - -# Examples - -* Install a package from Nixpkgs: - - ```console - # nix profile install nixpkgs#hello - ``` - -* Install a package from a specific branch of Nixpkgs: - - ```console - # nix profile install nixpkgs/release-20.09#hello - ``` - -* Install a package from a specific revision of Nixpkgs: - - ```console - # nix profile install nixpkgs/d73407e8e6002646acfdef0e39ace088bacc83da#hello - ``` - -* Install a specific output of a package: - - ```console - # nix profile install nixpkgs#bash^man - ``` - - -# Description - -This command adds [*installables*](./nix.md#installables) to a Nix profile. - -)"" diff --git a/src/nix/profile.cc b/src/nix/profile.cc index 51de2f789fe..dbaae42fa63 100644 --- a/src/nix/profile.cc +++ b/src/nix/profile.cc @@ -338,14 +338,14 @@ builtPathsPerInstallable( return res; } -struct CmdProfileInstall : InstallablesCommand, MixDefaultProfile +struct CmdProfileAdd : InstallablesCommand, MixDefaultProfile { std::optional priority; - CmdProfileInstall() { + CmdProfileAdd() { addFlag({ .longName = "priority", - .description = "The priority of the package to install.", + .description = "The priority of the package to add.", .labels = {"priority"}, .handler = {&priority}, }); @@ -353,13 +353,13 @@ struct CmdProfileInstall : InstallablesCommand, MixDefaultProfile std::string description() override { - return "install a package into a profile"; + return "add a package to a profile"; } std::string doc() override { return - #include "profile-install.md" + #include "profile-add.md" ; } @@ -415,7 +415,7 @@ struct CmdProfileInstall : InstallablesCommand, MixDefaultProfile && existingSource->originalRef == elementSource->originalRef && existingSource->attrPath == elementSource->attrPath ) { - warn("'%s' is already installed", elementName); + warn("'%s' is already added", elementName); continue; } } @@ -462,15 +462,15 @@ struct CmdProfileInstall : InstallablesCommand, MixDefaultProfile "\n" " nix profile remove %3%\n" "\n" - "The new package can also be installed next to the existing one by assigning a different priority.\n" + "The new package can also be added next to the existing one by assigning a different priority.\n" "The conflicting packages have a priority of %5%.\n" "To prioritise the new package:\n" "\n" - " nix profile install %4% --priority %6%\n" + " nix profile add %4% --priority %6%\n" "\n" "To prioritise the existing package:\n" "\n" - " nix profile install %4% --priority %7%\n", + " nix profile add %4% --priority %7%\n", originalConflictingFilePath, newConflictingFilePath, originalEntryName, @@ -708,16 +708,14 @@ struct CmdProfileUpgrade : virtual SourceExprCommand, MixDefaultProfile, MixProf if (!element.source) { warn( - "Found package '%s', but it was not installed from a flake, so it can't be checked for upgrades!", - element.identifier() - ); + "Found package '%s', but it was not added from a flake, so it can't be checked for upgrades!", + element.identifier()); continue; } if (element.source->originalRef.input.isLocked()) { warn( - "Found package '%s', but it was installed from a locked flake reference so it can't be upgraded!", - element.identifier() - ); + "Found package '%s', but it was added from a locked flake reference so it can't be upgraded!", + element.identifier()); continue; } @@ -787,7 +785,7 @@ struct CmdProfileList : virtual EvalCommand, virtual StoreCommand, MixDefaultPro { std::string description() override { - return "list installed packages"; + return "list packages in the profile"; } std::string doc() override @@ -978,7 +976,7 @@ struct CmdProfile : NixMultiCommand : NixMultiCommand( "profile", { - {"install", []() { return make_ref(); }}, + {"add", []() { return make_ref(); }}, {"remove", []() { return make_ref(); }}, {"upgrade", []() { return make_ref(); }}, {"list", []() { return make_ref(); }}, @@ -986,6 +984,8 @@ struct CmdProfile : NixMultiCommand {"history", []() { return make_ref(); }}, {"rollback", []() { return make_ref(); }}, {"wipe-history", []() { return make_ref(); }}, + // 2025-04-05 Deprecated in favor of "add" + {"install", []() { return make_ref(); }}, }) { } diff --git a/tests/functional/nix-profile.sh b/tests/functional/nix-profile.sh index 7cf5fcb7456..b1cfef6b0b2 100755 --- a/tests/functional/nix-profile.sh +++ b/tests/functional/nix-profile.sh @@ -52,7 +52,7 @@ cp "${config_nix}" $flake1Dir/ # Test upgrading from nix-env. nix-env -f ./user-envs.nix -i foo-1.0 nix profile list | grep -A2 'Name:.*foo' | grep 'Store paths:.*foo-1.0' -nix profile install $flake1Dir -L +nix profile add $flake1Dir -L nix profile list | grep -A4 'Name:.*flake1' | grep 'Locked flake URL:.*narHash' [[ $($TEST_HOME/.nix-profile/bin/hello) = "Hello World" ]] [ -e $TEST_HOME/.nix-profile/share/man ] @@ -64,12 +64,12 @@ nix profile diff-closures | grep 'env-manifest.nix: ε → ∅' # Test XDG Base Directories support export NIX_CONFIG="use-xdg-base-directories = true" nix profile remove flake1 2>&1 | grep 'removed 1 packages' -nix profile install $flake1Dir +nix profile add $flake1Dir [[ $($TEST_HOME/.local/state/nix/profile/bin/hello) = "Hello World" ]] unset NIX_CONFIG -# Test conflicting package install. -nix profile install $flake1Dir 2>&1 | grep "warning: 'flake1' is already installed" +# Test conflicting package add. +nix profile add $flake1Dir 2>&1 | grep "warning: 'flake1' is already added" # Test upgrading a package. printf NixOS > $flake1Dir/who @@ -132,16 +132,16 @@ nix profile history | grep 'foo: 1.0 -> ∅' nix profile diff-closures | grep 'Version 3 -> 4' # Test installing a non-flake package. -nix profile install --file ./simple.nix '' +nix profile add --file ./simple.nix '' [[ $(cat $TEST_HOME/.nix-profile/hello) = "Hello World!" ]] nix profile remove simple 2>&1 | grep 'removed 1 packages' -nix profile install $(nix-build --no-out-link ./simple.nix) +nix profile add $(nix-build --no-out-link ./simple.nix) [[ $(cat $TEST_HOME/.nix-profile/hello) = "Hello World!" ]] # Test packages with same name from different sources mkdir $TEST_ROOT/simple-too cp ./simple.nix "${config_nix}" simple.builder.sh $TEST_ROOT/simple-too -nix profile install --file $TEST_ROOT/simple-too/simple.nix '' +nix profile add --file $TEST_ROOT/simple-too/simple.nix '' nix profile list | grep -A4 'Name:.*simple' | grep 'Name:.*simple-1' nix profile remove simple 2>&1 | grep 'removed 1 packages' nix profile remove simple-1 2>&1 | grep 'removed 1 packages' @@ -160,13 +160,13 @@ nix profile history | grep "packages.$system.default: 1.0, 1.0-man -> 3.0, 3.0-m nix profile remove flake1 2>&1 | grep 'removed 1 packages' printf 4.0 > $flake1Dir/version printf Utrecht > $flake1Dir/who -nix profile install $flake1Dir +nix profile add $flake1Dir [[ $($TEST_HOME/.nix-profile/bin/hello) = "Hello Utrecht" ]] [[ $(nix path-info --json $(realpath $TEST_HOME/.nix-profile/bin/hello) | jq -r .[].ca) =~ fixed:r:sha256: ]] # Override the outputs. nix profile remove simple flake1 -nix profile install "$flake1Dir^*" +nix profile add "$flake1Dir^*" [[ $($TEST_HOME/.nix-profile/bin/hello) = "Hello Utrecht" ]] [ -e $TEST_HOME/.nix-profile/share/man ] [ -e $TEST_HOME/.nix-profile/include ] @@ -179,7 +179,7 @@ nix profile upgrade flake1 [ -e $TEST_HOME/.nix-profile/include ] nix profile remove flake1 2>&1 | grep 'removed 1 packages' -nix profile install "$flake1Dir^man" +nix profile add "$flake1Dir^man" (! [ -e $TEST_HOME/.nix-profile/bin/hello ]) [ -e $TEST_HOME/.nix-profile/share/man ] (! [ -e $TEST_HOME/.nix-profile/include ]) @@ -193,9 +193,9 @@ printf World > $flake1Dir/who cp -r $flake1Dir $flake2Dir printf World2 > $flake2Dir/who -nix profile install $flake1Dir +nix profile add $flake1Dir [[ $($TEST_HOME/.nix-profile/bin/hello) = "Hello World" ]] -expect 1 nix profile install $flake2Dir +expect 1 nix profile add $flake2Dir diff -u <( nix --offline profile install $flake2Dir 2>&1 1> /dev/null \ | grep -vE "^warning: " \ @@ -214,31 +214,31 @@ error: An existing package already provides the following file: nix profile remove flake1 - The new package can also be installed next to the existing one by assigning a different priority. + The new package can also be added next to the existing one by assigning a different priority. The conflicting packages have a priority of 5. To prioritise the new package: - nix profile install path:${flake2Dir}#packages.${system}.default --priority 4 + nix profile add path:${flake2Dir}#packages.${system}.default --priority 4 To prioritise the existing package: - nix profile install path:${flake2Dir}#packages.${system}.default --priority 6 + nix profile add path:${flake2Dir}#packages.${system}.default --priority 6 EOF ) [[ $($TEST_HOME/.nix-profile/bin/hello) = "Hello World" ]] -nix profile install $flake2Dir --priority 100 +nix profile add $flake2Dir --priority 100 [[ $($TEST_HOME/.nix-profile/bin/hello) = "Hello World" ]] -nix profile install $flake2Dir --priority 0 +nix profile add $flake2Dir --priority 0 [[ $($TEST_HOME/.nix-profile/bin/hello) = "Hello World2" ]] -# nix profile install $flake1Dir --priority 100 +# nix profile add $flake1Dir --priority 100 # [[ $($TEST_HOME/.nix-profile/bin/hello) = "Hello World" ]] # Ensure that conflicts are handled properly even when the installables aren't # flake references. # Regression test for https://github.com/NixOS/nix/issues/8284 clearProfiles -nix profile install $(nix build $flake1Dir --no-link --print-out-paths) -expect 1 nix profile install --impure --expr "(builtins.getFlake ''$flake2Dir'').packages.$system.default" +nix profile add $(nix build $flake1Dir --no-link --print-out-paths) +expect 1 nix profile add --impure --expr "(builtins.getFlake ''$flake2Dir'').packages.$system.default" # Test upgrading from profile version 2. clearProfiles From fd407141e16ea04fa4fa2b83ad222fcf751af0d3 Mon Sep 17 00:00:00 2001 From: Eelco Dolstra Date: Thu, 10 Apr 2025 18:42:04 +0200 Subject: [PATCH 026/218] Make `nix profile install` an alias of `nix profile add` --- doc/manual/meson.build | 1 - src/nix/profile.cc | 8 +++++--- 2 files changed, 5 insertions(+), 4 deletions(-) diff --git a/doc/manual/meson.build b/doc/manual/meson.build index 6e5a610f463..cd46f1050d4 100644 --- a/doc/manual/meson.build +++ b/doc/manual/meson.build @@ -248,7 +248,6 @@ nix3_manpages = [ 'nix3-print-dev-env', 'nix3-profile-diff-closures', 'nix3-profile-history', - 'nix3-profile-install', 'nix3-profile-list', 'nix3-profile', 'nix3-profile-remove', diff --git a/src/nix/profile.cc b/src/nix/profile.cc index dbaae42fa63..2c593729f49 100644 --- a/src/nix/profile.cc +++ b/src/nix/profile.cc @@ -984,10 +984,12 @@ struct CmdProfile : NixMultiCommand {"history", []() { return make_ref(); }}, {"rollback", []() { return make_ref(); }}, {"wipe-history", []() { return make_ref(); }}, - // 2025-04-05 Deprecated in favor of "add" - {"install", []() { return make_ref(); }}, }) - { } + { + aliases = { + {"install", { AliasStatus::Deprecated, {"add"}}}, + }; + } std::string description() override { From 5f75738fd6ffbc9768e8f106edd7d9ff1b6c5751 Mon Sep 17 00:00:00 2001 From: Eelco Dolstra Date: Sat, 17 May 2025 21:18:04 +0200 Subject: [PATCH 027/218] Install 'nix profile add' manpage --- doc/manual/meson.build | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/doc/manual/meson.build b/doc/manual/meson.build index cd46f1050d4..6fe2374a764 100644 --- a/doc/manual/meson.build +++ b/doc/manual/meson.build @@ -246,10 +246,11 @@ nix3_manpages = [ 'nix3-nar', 'nix3-path-info', 'nix3-print-dev-env', + 'nix3-profile', + 'nix3-profile-add', 'nix3-profile-diff-closures', 'nix3-profile-history', 'nix3-profile-list', - 'nix3-profile', 'nix3-profile-remove', 'nix3-profile-rollback', 'nix3-profile-upgrade', From 46beb9af76284d131b43f2657296303390f9a585 Mon Sep 17 00:00:00 2001 From: Matt Sturgeon Date: Mon, 12 May 2025 12:16:40 +0100 Subject: [PATCH 028/218] Use correct parent `outPath` for relative path inputs Ensure relative path inputs are relative to the parent node's _actual_ `outPath`, instead of the subtly different `sourceInfo.outPath`. Additionally, non-flake inputs now also have a `sourceInfo` attribute. This fixes the relationship between `self.outPath` and `self.sourceInfo.outPath` in some edge cases. Fixes #13164 --- src/libflake/call-flake.nix | 26 +++++++++++--------------- 1 file changed, 11 insertions(+), 15 deletions(-) diff --git a/src/libflake/call-flake.nix b/src/libflake/call-flake.nix index fe326291f1f..ed7947e0601 100644 --- a/src/libflake/call-flake.nix +++ b/src/libflake/call-flake.nix @@ -39,24 +39,16 @@ let allNodes = mapAttrs ( key: node: let + hasOverride = overrides ? ${key}; + isRelative = node.locked.type or null == "path" && builtins.substring 0 1 node.locked.path != "/"; parentNode = allNodes.${getInputByPath lockFile.root node.parent}; - flakeDir = - let - dir = overrides.${key}.dir or node.locked.path or ""; - parentDir = parentNode.flakeDir; - in - if node ? parent then parentDir + ("/" + dir) else dir; - sourceInfo = - if overrides ? ${key} then + if hasOverride then overrides.${key}.sourceInfo - else if node.locked.type == "path" && builtins.substring 0 1 node.locked.path != "/" then + else if isRelative then parentNode.sourceInfo - // { - outPath = parentNode.sourceInfo.outPath + ("/" + flakeDir); - } else # FIXME: remove obsolete node.info. # Note: lock file entries are always final. @@ -64,7 +56,11 @@ let subdir = overrides.${key}.dir or node.locked.dir or ""; - outPath = sourceInfo + ((if subdir == "" then "" else "/") + subdir); + outPath = + if !hasOverride && isRelative then + parentNode.outPath + (if node.locked.path == "" then "" else "/" + node.locked.path) + else + sourceInfo.outPath + (if subdir == "" then "" else "/" + subdir); flake = import (outPath + "/flake.nix"); @@ -99,9 +95,9 @@ let assert builtins.isFunction flake.outputs; result else - sourceInfo; + sourceInfo // { inherit sourceInfo outPath; }; - inherit flakeDir sourceInfo; + inherit outPath sourceInfo; } ) lockFile.nodes; From eaee0b4740dcc82e3fec8482d5f803a45741594d Mon Sep 17 00:00:00 2001 From: Robert Hensing Date: Wed, 14 May 2025 22:48:56 +0200 Subject: [PATCH 029/218] tests/function/flakes/relative-paths: Test #13164 --- tests/functional/flakes/relative-paths.sh | 43 +++++++++++++++++++++++ 1 file changed, 43 insertions(+) diff --git a/tests/functional/flakes/relative-paths.sh b/tests/functional/flakes/relative-paths.sh index 4648ba98c63..560cc6aaa09 100644 --- a/tests/functional/flakes/relative-paths.sh +++ b/tests/functional/flakes/relative-paths.sh @@ -129,3 +129,46 @@ EOF # would fail: nix eval .#ok ) + +# https://github.com/NixOS/nix/issues/13164 +mkdir -p "$TEST_ROOT/issue-13164/nested-flake1/nested-flake2" +( + cd "$TEST_ROOT/issue-13164" + git init + git config --global user.email "you@example.com" + git config --global user.name "Your Name" + cat >flake.nix <nested-flake1/flake.nix <nested-flake1/nested-flake2/flake.nix < Date: Sun, 18 May 2025 00:18:46 +0100 Subject: [PATCH 030/218] tests/functional/flakes/non-flake-inputs: Test non-flake inputs having `sourceInfo` --- tests/functional/flakes/non-flake-inputs.sh | 46 +++++++++++++++++++++ 1 file changed, 46 insertions(+) diff --git a/tests/functional/flakes/non-flake-inputs.sh b/tests/functional/flakes/non-flake-inputs.sh index f5e12cd0141..05e65604226 100644 --- a/tests/functional/flakes/non-flake-inputs.sh +++ b/tests/functional/flakes/non-flake-inputs.sh @@ -37,11 +37,20 @@ cat > "$flake3Dir/flake.nix" < Date: Sun, 18 May 2025 00:37:23 +0100 Subject: [PATCH 031/218] Add release note for non-flake inputs having `sourceInfo` --- .../rl-next/outpath-and-sourceinfo-fixes.md | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) create mode 100644 doc/manual/rl-next/outpath-and-sourceinfo-fixes.md diff --git a/doc/manual/rl-next/outpath-and-sourceinfo-fixes.md b/doc/manual/rl-next/outpath-and-sourceinfo-fixes.md new file mode 100644 index 00000000000..479a2f5cb75 --- /dev/null +++ b/doc/manual/rl-next/outpath-and-sourceinfo-fixes.md @@ -0,0 +1,17 @@ +--- +synopsis: Non-flake inputs now contain a `sourceInfo` attribute +issues: 13164 +prs: 13170 +--- + +Flakes have always a `sourceInfo` attribute which describes the source of the flake. +The `sourceInfo.outPath` is often identical to the flake's `outPath`, however it can differ when the flake is located in a subdirectory of its source. + +Non-flake inputs (i.e. inputs with `flake = false`) can also be located at some path _within_ a wider source. +This usually happens when defining a relative path input within the same source as the parent flake, e.g. `inputs.foo.url = ./some-file.nix`. +Such relative inputs will now inherit their parent's `sourceInfo`. + +This also means it is now possible to use `?dir=subdir` on non-flake inputs. + +This iterates on the work done in 2.26 to improve relative path support ([#10089](https://github.com/NixOS/nix/pull/10089)), +and resolves a regression introduced in 2.28 relating to nested relative path inputs ([#13164](https://github.com/NixOS/nix/issues/13164)). From 6b4a86a9e337b3b90bb02c6fcb01a986e70da074 Mon Sep 17 00:00:00 2001 From: Sergei Zimmerman Date: Fri, 16 May 2025 21:53:06 +0000 Subject: [PATCH 032/218] libexpr: Add `EvalProfiler` This patch adds an EvalProfiler and MultiEvalProfiler that can be used to insert hooks into the evaluation for the purposes of function tracing (what function-trace currently does) or for flamegraph/tracy profilers. See the following commits for how this is supposed to be integrated into the evaluator and performance considerations. --- src/libexpr/eval-profiler.cc | 48 ++++++++ src/libexpr/include/nix/expr/eval-profiler.hh | 113 ++++++++++++++++++ .../include/nix/expr/function-trace.hh | 1 + src/libexpr/include/nix/expr/meson.build | 1 + src/libexpr/meson.build | 1 + 5 files changed, 164 insertions(+) create mode 100644 src/libexpr/eval-profiler.cc create mode 100644 src/libexpr/include/nix/expr/eval-profiler.hh diff --git a/src/libexpr/eval-profiler.cc b/src/libexpr/eval-profiler.cc new file mode 100644 index 00000000000..5374d7d9968 --- /dev/null +++ b/src/libexpr/eval-profiler.cc @@ -0,0 +1,48 @@ +#include "nix/expr/eval-profiler.hh" +#include "nix/expr/nixexpr.hh" + +namespace nix { + +void EvalProfiler::preFunctionCallHook( + const EvalState & state, const Value & v, std::span args, const PosIdx pos) +{ +} + +void EvalProfiler::postFunctionCallHook( + const EvalState & state, const Value & v, std::span args, const PosIdx pos) +{ +} + +void MultiEvalProfiler::preFunctionCallHook( + const EvalState & state, const Value & v, std::span args, const PosIdx pos) +{ + for (auto & profiler : profilers) { + if (profiler->getNeededHooks().test(Hook::preFunctionCall)) + profiler->preFunctionCallHook(state, v, args, pos); + } +} + +void MultiEvalProfiler::postFunctionCallHook( + const EvalState & state, const Value & v, std::span args, const PosIdx pos) +{ + for (auto & profiler : profilers) { + if (profiler->getNeededHooks().test(Hook::postFunctionCall)) + profiler->postFunctionCallHook(state, v, args, pos); + } +} + +EvalProfiler::Hooks MultiEvalProfiler::getNeededHooksImpl() const +{ + Hooks hooks; + for (auto & p : profilers) + hooks |= p->getNeededHooks(); + return hooks; +} + +void MultiEvalProfiler::addProfiler(ref profiler) +{ + profilers.push_back(profiler); + invalidateNeededHooks(); +} + +} diff --git a/src/libexpr/include/nix/expr/eval-profiler.hh b/src/libexpr/include/nix/expr/eval-profiler.hh new file mode 100644 index 00000000000..763b737f7a9 --- /dev/null +++ b/src/libexpr/include/nix/expr/eval-profiler.hh @@ -0,0 +1,113 @@ +#pragma once +/** + * @file + * + * Evaluation profiler interface definitions and builtin implementations. + */ + +#include "nix/util/ref.hh" + +#include +#include +#include +#include + +namespace nix { + +class EvalState; +class PosIdx; +struct Value; + +class EvalProfiler +{ +public: + enum Hook { + preFunctionCall, + postFunctionCall, + }; + + static constexpr std::size_t numHooks = Hook::postFunctionCall + 1; + using Hooks = std::bitset; + +private: + std::optional neededHooks; + +protected: + /** Invalidate the cached neededHooks. */ + void invalidateNeededHooks() + { + neededHooks = std::nullopt; + } + + /** + * Get which hooks need to be called. + * + * This is the actual implementation which has to be defined by subclasses. + * Public API goes through the needsHooks, which is a + * non-virtual interface (NVI) which caches the return value. + */ + virtual Hooks getNeededHooksImpl() const + { + return Hooks{}; + } + +public: + /** + * Hook called in the EvalState::callFunction preamble. + * Gets called only if (getNeededHooks().test(Hook::preFunctionCall)) is true. + * + * @param state Evaluator state. + * @param v Function being invoked. + * @param args Function arguments. + * @param pos Function position. + */ + virtual void + preFunctionCallHook(const EvalState & state, const Value & v, std::span args, const PosIdx pos); + + /** + * Hook called on EvalState::callFunction exit. + * Gets called only if (getNeededHooks().test(Hook::postFunctionCall)) is true. + * + * @param state Evaluator state. + * @param v Function being invoked. + * @param args Function arguments. + * @param pos Function position. + */ + virtual void + postFunctionCallHook(const EvalState & state, const Value & v, std::span args, const PosIdx pos); + + virtual ~EvalProfiler() = default; + + /** + * Get which hooks need to be invoked for this EvalProfiler instance. + */ + Hooks getNeededHooks() + { + if (neededHooks.has_value()) + return *neededHooks; + return *(neededHooks = getNeededHooksImpl()); + } +}; + +/** + * Profiler that invokes multiple profilers at once. + */ +class MultiEvalProfiler : public EvalProfiler +{ + std::vector> profilers; + + [[gnu::noinline]] Hooks getNeededHooksImpl() const override; + +public: + MultiEvalProfiler() = default; + + /** Register a profiler instance. */ + void addProfiler(ref profiler); + + [[gnu::noinline]] void + preFunctionCallHook(const EvalState & state, const Value & v, std::span args, const PosIdx pos) override; + [[gnu::noinline]] void + postFunctionCallHook(const EvalState & state, const Value & v, std::span args, const PosIdx pos) override; +}; + +} diff --git a/src/libexpr/include/nix/expr/function-trace.hh b/src/libexpr/include/nix/expr/function-trace.hh index dc92d4b5ca2..f9b82d8a5ba 100644 --- a/src/libexpr/include/nix/expr/function-trace.hh +++ b/src/libexpr/include/nix/expr/function-trace.hh @@ -2,6 +2,7 @@ ///@file #include "nix/expr/eval.hh" +#include "nix/expr/eval-profiler.hh" #include diff --git a/src/libexpr/include/nix/expr/meson.build b/src/libexpr/include/nix/expr/meson.build index 50ea8f3c22c..db902a61676 100644 --- a/src/libexpr/include/nix/expr/meson.build +++ b/src/libexpr/include/nix/expr/meson.build @@ -14,6 +14,7 @@ headers = [config_pub_h] + files( 'eval-error.hh', 'eval-gc.hh', 'eval-inline.hh', + 'eval-profiler.hh', 'eval-settings.hh', 'eval.hh', 'function-trace.hh', diff --git a/src/libexpr/meson.build b/src/libexpr/meson.build index 2b465b85a79..dc50d2b1935 100644 --- a/src/libexpr/meson.build +++ b/src/libexpr/meson.build @@ -140,6 +140,7 @@ sources = files( 'eval-cache.cc', 'eval-error.cc', 'eval-gc.cc', + 'eval-profiler.cc', 'eval-settings.cc', 'eval.cc', 'function-trace.cc', From fa6f69f9c5b2d753b7199161ab40ec661f0c190f Mon Sep 17 00:00:00 2001 From: Sergei Zimmerman Date: Fri, 16 May 2025 22:29:18 +0000 Subject: [PATCH 033/218] libexpr: Use `EvalProfiler` for `FunctionCallTrace` MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit This wires up the {pre,post}FunctionCallHook machinery in EvalState::callFunction and migrates FunctionCallTrace to use the new EvalProfiler mechanisms for tracing. Note that branches when the hook gets called are marked with [[unlikely]] as a hint to the compiler that this is not a hot path. For non-tracing evaluation this should be a 100% predictable branch, so the performance cost is nonexistent. Some measurements to prove support this point: ``` nix build .#nix-cli nix build github:nixos/nix/d692729759e4e370361cc5105fbeb0e33137ca9e#nix-cli --out-link before ``` (Before) ``` $ taskset -c 2,3 hyperfine "GC_INITIAL_HEAP_SIZE=16g before/bin/nix eval nixpkgs#gnome --no-eval-cache" --warmup 4 Benchmark 1: GC_INITIAL_HEAP_SIZE=16g before/bin/nix eval nixpkgs#gnome --no-eval-cache Time (mean ± σ): 2.517 s ± 0.032 s [User: 1.464 s, System: 0.476 s] Range (min … max): 2.464 s … 2.557 s 10 runs ``` (After) ``` $ taskset -c 2,3 hyperfine "GC_INITIAL_HEAP_SIZE=16g result/bin/nix eval nixpkgs#gnome --no-eval-cache" --warmup 4 Benchmark 1: GC_INITIAL_HEAP_SIZE=16g result/bin/nix eval nixpkgs#gnome --no-eval-cache Time (mean ± σ): 2.499 s ± 0.022 s [User: 1.448 s, System: 0.478 s] Range (min … max): 2.472 s … 2.537 s 10 runs ``` --- src/libexpr/eval.cc | 15 +++++++++++--- src/libexpr/function-trace.cc | 12 +++++++---- src/libexpr/include/nix/expr/eval.hh | 4 ++++ .../include/nix/expr/function-trace.hh | 20 +++++++++++++------ 4 files changed, 38 insertions(+), 13 deletions(-) diff --git a/src/libexpr/eval.cc b/src/libexpr/eval.cc index ab7f33d5f89..c171caa9905 100644 --- a/src/libexpr/eval.cc +++ b/src/libexpr/eval.cc @@ -372,6 +372,10 @@ EvalState::EvalState( ); createBaseEnv(settings); + + /* Register function call tracer. */ + if (settings.traceFunctionCalls) + profiler.addProfiler(make_ref()); } @@ -1526,9 +1530,14 @@ void EvalState::callFunction(Value & fun, std::span args, Value & vRes, { auto _level = addCallDepth(pos); - auto trace = settings.traceFunctionCalls - ? std::make_unique(positions[pos]) - : nullptr; + auto neededHooks = profiler.getNeededHooks(); + if (neededHooks.test(EvalProfiler::preFunctionCall)) [[unlikely]] + profiler.preFunctionCallHook(*this, fun, args, pos); + + Finally traceExit_{[&](){ + if (profiler.getNeededHooks().test(EvalProfiler::postFunctionCall)) [[unlikely]] + profiler.postFunctionCallHook(*this, fun, args, pos); + }}; forceValue(fun, pos); diff --git a/src/libexpr/function-trace.cc b/src/libexpr/function-trace.cc index 1dce5172688..993dd72d7d0 100644 --- a/src/libexpr/function-trace.cc +++ b/src/libexpr/function-trace.cc @@ -3,16 +3,20 @@ namespace nix { -FunctionCallTrace::FunctionCallTrace(const Pos & pos) : pos(pos) { +void FunctionCallTrace::preFunctionCallHook( + const EvalState & state, const Value & v, std::span args, const PosIdx pos) +{ auto duration = std::chrono::high_resolution_clock::now().time_since_epoch(); auto ns = std::chrono::duration_cast(duration); - printMsg(lvlInfo, "function-trace entered %1% at %2%", pos, ns.count()); + printMsg(lvlInfo, "function-trace entered %1% at %2%", state.positions[pos], ns.count()); } -FunctionCallTrace::~FunctionCallTrace() { +void FunctionCallTrace::postFunctionCallHook( + const EvalState & state, const Value & v, std::span args, const PosIdx pos) +{ auto duration = std::chrono::high_resolution_clock::now().time_since_epoch(); auto ns = std::chrono::duration_cast(duration); - printMsg(lvlInfo, "function-trace exited %1% at %2%", pos, ns.count()); + printMsg(lvlInfo, "function-trace exited %1% at %2%", state.positions[pos], ns.count()); } } diff --git a/src/libexpr/include/nix/expr/eval.hh b/src/libexpr/include/nix/expr/eval.hh index 6a6959bd821..ffbc84bcdaf 100644 --- a/src/libexpr/include/nix/expr/eval.hh +++ b/src/libexpr/include/nix/expr/eval.hh @@ -3,6 +3,7 @@ #include "nix/expr/attr-set.hh" #include "nix/expr/eval-error.hh" +#include "nix/expr/eval-profiler.hh" #include "nix/util/types.hh" #include "nix/expr/value.hh" #include "nix/expr/nixexpr.hh" @@ -903,6 +904,9 @@ private: typedef std::map FunctionCalls; FunctionCalls functionCalls; + /** Evaluation/call profiler. */ + MultiEvalProfiler profiler; + void incrFunctionCall(ExprLambda * fun); typedef std::map AttrSelects; diff --git a/src/libexpr/include/nix/expr/function-trace.hh b/src/libexpr/include/nix/expr/function-trace.hh index f9b82d8a5ba..9187cac6324 100644 --- a/src/libexpr/include/nix/expr/function-trace.hh +++ b/src/libexpr/include/nix/expr/function-trace.hh @@ -4,14 +4,22 @@ #include "nix/expr/eval.hh" #include "nix/expr/eval-profiler.hh" -#include - namespace nix { -struct FunctionCallTrace +class FunctionCallTrace : public EvalProfiler { - const Pos pos; - FunctionCallTrace(const Pos & pos); - ~FunctionCallTrace(); + Hooks getNeededHooksImpl() const override + { + return Hooks().set(preFunctionCall).set(postFunctionCall); + } + +public: + FunctionCallTrace() = default; + + [[gnu::noinline]] void + preFunctionCallHook(const EvalState & state, const Value & v, std::span args, const PosIdx pos) override; + [[gnu::noinline]] void + postFunctionCallHook(const EvalState & state, const Value & v, std::span args, const PosIdx pos) override; }; + } From 1290b7e53d03cc8b084aaa8e58baff177711ccb0 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?J=C3=B6rg=20Thalheim?= Date: Mon, 19 May 2025 09:25:34 +0200 Subject: [PATCH 034/218] libutil-tests/json-utils: fix -Werror=sign-compare error I am on a newer different nixpkgs branch, so I am getting this error --- src/libutil-tests/json-utils.cc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/libutil-tests/json-utils.cc b/src/libutil-tests/json-utils.cc index eae67b4b316..211f8bf1ee4 100644 --- a/src/libutil-tests/json-utils.cc +++ b/src/libutil-tests/json-utils.cc @@ -131,7 +131,7 @@ TEST(getString, wrongAssertions) { TEST(getIntegralNumber, rightAssertions) { auto simple = R"({ "int": 0, "signed": -1 })"_json; - ASSERT_EQ(getUnsigned(valueAt(getObject(simple), "int")), 0); + ASSERT_EQ(getUnsigned(valueAt(getObject(simple), "int")), 0u); ASSERT_EQ(getInteger(valueAt(getObject(simple), "int")), 0); ASSERT_EQ(getInteger(valueAt(getObject(simple), "signed")), -1); } From 5c512a4ee1ffc65baf517f2bee8f396e313618e0 Mon Sep 17 00:00:00 2001 From: Gwenn Le Bihan Date: Mon, 19 May 2025 16:58:16 +0200 Subject: [PATCH 035/218] docs: clarify attrset __functor --- doc/manual/source/language/syntax.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/doc/manual/source/language/syntax.md b/doc/manual/source/language/syntax.md index 08a64f68421..85162db747a 100644 --- a/doc/manual/source/language/syntax.md +++ b/doc/manual/source/language/syntax.md @@ -225,8 +225,8 @@ passed in first , e.g., ```nix let add = { __functor = self: x: x + self.x; }; - inc = add // { x = 1; }; -in inc 1 + inc = add // { x = 1; }; # inc is { x = 1; __functor = (...) } +in inc 1 # equivalent of `add.__functor add 1` i.e. `1 + self.x` ``` evaluates to `2`. This can be used to attach metadata to a function From 51151c2c28d5519206148945611812963dc0fb93 Mon Sep 17 00:00:00 2001 From: Gwenn Le Bihan Date: Mon, 19 May 2025 17:37:29 +0200 Subject: [PATCH 036/218] docs: add another equivalence for the implication operator the second equivalence, using a if-else expression, aligns much closer to how most humans think about implication, adding it might help some people :) --- doc/manual/source/language/operators.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/manual/source/language/operators.md b/doc/manual/source/language/operators.md index dbf2441cb7c..ab74e8a9999 100644 --- a/doc/manual/source/language/operators.md +++ b/doc/manual/source/language/operators.md @@ -196,7 +196,7 @@ All comparison operators are implemented in terms of `<`, and the following equi ## Logical implication -Equivalent to `!`*b1* `||` *b2*. +Equivalent to `!`*b1* `||` *b2* (or `if` *b1* `then` *b2* `else true`) [Logical implication]: #logical-implication From a4bfd559f1671f3bf9f0a8961d90bbd978cd3f08 Mon Sep 17 00:00:00 2001 From: Sergei Zimmerman Date: Mon, 19 May 2025 19:53:20 +0000 Subject: [PATCH 037/218] libexpr: Use `attrs.alreadySorted()` in primop_functionArgs Formals are already sorted as it's an invariant of `Formals`. --- src/libexpr/include/nix/expr/nixexpr.hh | 3 +++ src/libexpr/primops.cc | 12 +++++++++--- 2 files changed, 12 insertions(+), 3 deletions(-) diff --git a/src/libexpr/include/nix/expr/nixexpr.hh b/src/libexpr/include/nix/expr/nixexpr.hh index a5ce0fd8922..6ede91948e0 100644 --- a/src/libexpr/include/nix/expr/nixexpr.hh +++ b/src/libexpr/include/nix/expr/nixexpr.hh @@ -306,6 +306,9 @@ struct Formal struct Formals { typedef std::vector Formals_; + /** + * @pre Sorted according to predicate (std::tie(a.name, a.pos) < std::tie(b.name, b.pos)). + */ Formals_ formals; bool ellipsis; diff --git a/src/libexpr/primops.cc b/src/libexpr/primops.cc index d13314ffba9..c38eed26790 100644 --- a/src/libexpr/primops.cc +++ b/src/libexpr/primops.cc @@ -3148,10 +3148,16 @@ static void prim_functionArgs(EvalState & state, const PosIdx pos, Value * * arg return; } - auto attrs = state.buildBindings(args[0]->payload.lambda.fun->formals->formals.size()); - for (auto & i : args[0]->payload.lambda.fun->formals->formals) + const auto &formals = args[0]->payload.lambda.fun->formals->formals; + auto attrs = state.buildBindings(formals.size()); + for (auto & i : formals) attrs.insert(i.name, state.getBool(i.def), i.pos); - v.mkAttrs(attrs); + /* Optimization: avoid sorting bindings. `formals` must already be sorted according to + (std::tie(a.name, a.pos) < std::tie(b.name, b.pos)) predicate, so the following assertion + always holds: + assert(std::is_sorted(attrs.alreadySorted()->begin(), attrs.alreadySorted()->end())); + .*/ + v.mkAttrs(attrs.alreadySorted()); } static RegisterPrimOp primop_functionArgs({ From 8ee513379a6ffda4d05b187399985d4106b876ba Mon Sep 17 00:00:00 2001 From: Sergei Zimmerman Date: Mon, 19 May 2025 20:33:28 +0000 Subject: [PATCH 038/218] Use `StringMap` instead of `std::map` throughout the codebase --- src/libcmd/include/nix/cmd/command.hh | 2 +- src/libfetchers/attrs.cc | 4 ++-- src/libfetchers/fetchers.cc | 2 +- src/libfetchers/git-utils.cc | 2 +- src/libfetchers/include/nix/fetchers/attrs.hh | 2 +- src/libfetchers/include/nix/fetchers/fetchers.hh | 2 +- src/libflake/flakeref.cc | 2 +- src/libstore/build/derivation-goal.cc | 2 +- src/libstore/include/nix/store/store-dir-config.hh | 2 +- src/libstore/include/nix/store/store-reference.hh | 2 +- src/libstore/realisation.cc | 2 +- src/libstore/ssh.cc | 2 +- src/libstore/unix/build/derivation-builder.cc | 2 +- src/libutil-tests/strings.cc | 2 +- src/libutil-tests/url.cc | 4 ++-- src/libutil/archive.cc | 2 +- src/libutil/environment-variables.cc | 6 +++--- src/libutil/include/nix/util/environment-variables.hh | 4 ++-- src/libutil/include/nix/util/processes.hh | 2 +- src/libutil/include/nix/util/url.hh | 6 +++--- src/libutil/include/nix/util/xml-writer.hh | 2 +- src/libutil/linux/cgroup.cc | 4 ++-- src/libutil/linux/include/nix/util/cgroup.hh | 2 +- src/libutil/url.cc | 6 +++--- src/nix-channel/nix-channel.cc | 2 +- src/nix/develop.cc | 4 ++-- 26 files changed, 37 insertions(+), 37 deletions(-) diff --git a/src/libcmd/include/nix/cmd/command.hh b/src/libcmd/include/nix/cmd/command.hh index cb436cfdbbb..20cd1abc1c4 100644 --- a/src/libcmd/include/nix/cmd/command.hh +++ b/src/libcmd/include/nix/cmd/command.hh @@ -336,7 +336,7 @@ struct MixEnvironment : virtual Args StringSet keepVars; StringSet unsetVars; - std::map setVars; + StringMap setVars; bool ignoreEnvironment; MixEnvironment(); diff --git a/src/libfetchers/attrs.cc b/src/libfetchers/attrs.cc index 47f6aa8c55c..6808e8af1f6 100644 --- a/src/libfetchers/attrs.cc +++ b/src/libfetchers/attrs.cc @@ -89,9 +89,9 @@ bool getBoolAttr(const Attrs & attrs, const std::string & name) return *s; } -std::map attrsToQuery(const Attrs & attrs) +StringMap attrsToQuery(const Attrs & attrs) { - std::map query; + StringMap query; for (auto & attr : attrs) { if (auto v = std::get_if(&attr.second)) { query.insert_or_assign(attr.first, fmt("%d", *v)); diff --git a/src/libfetchers/fetchers.cc b/src/libfetchers/fetchers.cc index 35201e8aef1..9cb89660172 100644 --- a/src/libfetchers/fetchers.cc +++ b/src/libfetchers/fetchers.cc @@ -134,7 +134,7 @@ ParsedURL Input::toURL() const return scheme->toURL(*this); } -std::string Input::toURLString(const std::map & extraQuery) const +std::string Input::toURLString(const StringMap & extraQuery) const { auto url = toURL(); for (auto & attr : extraQuery) diff --git a/src/libfetchers/git-utils.cc b/src/libfetchers/git-utils.cc index 60d8edee06b..09508c6026c 100644 --- a/src/libfetchers/git-utils.cc +++ b/src/libfetchers/git-utils.cc @@ -368,7 +368,7 @@ struct GitRepoImpl : GitRepo, std::enable_shared_from_this if (git_config_iterator_glob_new(Setter(it), config.get(), "^submodule\\..*\\.(path|url|branch)$")) throw Error("iterating over .gitmodules: %s", git_error_last()->message); - std::map entries; + StringMap entries; while (true) { git_config_entry * entry = nullptr; diff --git a/src/libfetchers/include/nix/fetchers/attrs.hh b/src/libfetchers/include/nix/fetchers/attrs.hh index 1b757d71215..582abd14413 100644 --- a/src/libfetchers/include/nix/fetchers/attrs.hh +++ b/src/libfetchers/include/nix/fetchers/attrs.hh @@ -37,7 +37,7 @@ std::optional maybeGetBoolAttr(const Attrs & attrs, const std::string & na bool getBoolAttr(const Attrs & attrs, const std::string & name); -std::map attrsToQuery(const Attrs & attrs); +StringMap attrsToQuery(const Attrs & attrs); Hash getRevAttr(const Attrs & attrs, const std::string & name); diff --git a/src/libfetchers/include/nix/fetchers/fetchers.hh b/src/libfetchers/include/nix/fetchers/fetchers.hh index 3288ecc5ea5..1f8f6bdacd6 100644 --- a/src/libfetchers/include/nix/fetchers/fetchers.hh +++ b/src/libfetchers/include/nix/fetchers/fetchers.hh @@ -71,7 +71,7 @@ public: ParsedURL toURL() const; - std::string toURLString(const std::map & extraQuery = {}) const; + std::string toURLString(const StringMap & extraQuery = {}) const; std::string to_string() const; diff --git a/src/libflake/flakeref.cc b/src/libflake/flakeref.cc index 12bddf57852..d56f2858f8d 100644 --- a/src/libflake/flakeref.cc +++ b/src/libflake/flakeref.cc @@ -15,7 +15,7 @@ const static std::string subDirRegex = subDirElemRegex + "(?:/" + subDirElemRege std::string FlakeRef::to_string() const { - std::map extraQuery; + StringMap extraQuery; if (subdir != "") extraQuery.insert_or_assign("dir", subdir); return input.toURLString(extraQuery); diff --git a/src/libstore/build/derivation-goal.cc b/src/libstore/build/derivation-goal.cc index 3931579387b..ee848ec841b 100644 --- a/src/libstore/build/derivation-goal.cc +++ b/src/libstore/build/derivation-goal.cc @@ -974,7 +974,7 @@ void runPostBuildHook( fmt("running post-build-hook '%s'", settings.postBuildHook), Logger::Fields{store.printStorePath(drvPath)}); PushActivity pact(act.id); - std::map hookEnvironment = getEnv(); + StringMap hookEnvironment = getEnv(); hookEnvironment.emplace("DRV_PATH", store.printStorePath(drvPath)); hookEnvironment.emplace("OUT_PATHS", chomp(concatStringsSep(" ", store.printStorePathSet(outputPaths)))); diff --git a/src/libstore/include/nix/store/store-dir-config.hh b/src/libstore/include/nix/store/store-dir-config.hh index 6bf9ebf1431..ea0161d79c9 100644 --- a/src/libstore/include/nix/store/store-dir-config.hh +++ b/src/libstore/include/nix/store/store-dir-config.hh @@ -125,7 +125,7 @@ struct StoreDirConfigBase : Config */ struct StoreDirConfig : StoreDirConfigBase, MixStoreDirMethods { - using Params = std::map; + using Params = StringMap; StoreDirConfig(const Params & params); diff --git a/src/libstore/include/nix/store/store-reference.hh b/src/libstore/include/nix/store/store-reference.hh index 433a347aaca..750bd827510 100644 --- a/src/libstore/include/nix/store/store-reference.hh +++ b/src/libstore/include/nix/store/store-reference.hh @@ -41,7 +41,7 @@ namespace nix { */ struct StoreReference { - using Params = std::map; + using Params = StringMap; /** * Special store reference `""` or `"auto"` diff --git a/src/libstore/realisation.cc b/src/libstore/realisation.cc index 635fb6946bf..9a72422eb89 100644 --- a/src/libstore/realisation.cc +++ b/src/libstore/realisation.cc @@ -96,7 +96,7 @@ Realisation Realisation::fromJSON( std::map dependentRealisations; if (auto jsonDependencies = json.find("dependentRealisations"); jsonDependencies != json.end()) - for (auto & [jsonDepId, jsonDepOutPath] : jsonDependencies->get>()) + for (auto & [jsonDepId, jsonDepOutPath] : jsonDependencies->get()) dependentRealisations.insert({DrvOutput::parse(jsonDepId), StorePath(jsonDepOutPath)}); return Realisation{ diff --git a/src/libstore/ssh.cc b/src/libstore/ssh.cc index 97b75cba10a..d29734a3e4b 100644 --- a/src/libstore/ssh.cc +++ b/src/libstore/ssh.cc @@ -83,7 +83,7 @@ bool SSHMaster::isMasterRunning() { Strings createSSHEnv() { // Copy the environment and set SHELL=/bin/sh - std::map env = getEnv(); + StringMap env = getEnv(); // SSH will invoke the "user" shell for -oLocalCommand, but that means // $SHELL. To keep things simple and avoid potential issues with other diff --git a/src/libstore/unix/build/derivation-builder.cc b/src/libstore/unix/build/derivation-builder.cc index 58e8d8ba64b..f3847e60469 100644 --- a/src/libstore/unix/build/derivation-builder.cc +++ b/src/libstore/unix/build/derivation-builder.cc @@ -187,7 +187,7 @@ class DerivationBuilderImpl : public DerivationBuilder, DerivationBuilderParams typedef std::map PathsInChroot; // maps target path to source path PathsInChroot pathsInChroot; - typedef std::map Environment; + typedef StringMap Environment; Environment env; /** diff --git a/src/libutil-tests/strings.cc b/src/libutil-tests/strings.cc index f5af4e0ff77..bf1f66025eb 100644 --- a/src/libutil-tests/strings.cc +++ b/src/libutil-tests/strings.cc @@ -106,7 +106,7 @@ TEST(concatMapStringsSep, two) TEST(concatMapStringsSep, map) { - std::map strings; + StringMap strings; strings["this"] = "that"; strings["1"] = "one"; diff --git a/src/libutil-tests/url.cc b/src/libutil-tests/url.cc index 4c089c10622..c93a96d84b6 100644 --- a/src/libutil-tests/url.cc +++ b/src/libutil-tests/url.cc @@ -5,8 +5,8 @@ namespace nix { /* ----------- tests for url.hh --------------------------------------------------*/ - std::string print_map(std::map m) { - std::map::iterator it; + std::string print_map(StringMap m) { + StringMap::iterator it; std::string s = "{ "; for (it = m.begin(); it != m.end(); ++it) { s += "{ "; diff --git a/src/libutil/archive.cc b/src/libutil/archive.cc index 487873ce606..9069e4b495f 100644 --- a/src/libutil/archive.cc +++ b/src/libutil/archive.cc @@ -72,7 +72,7 @@ void SourceAccessor::dumpPath( /* If we're on a case-insensitive system like macOS, undo the case hack applied by restorePath(). */ - std::map unhacked; + StringMap unhacked; for (auto & i : readDirectory(path)) if (archiveSettings.useCaseHack) { std::string name(i.first); diff --git a/src/libutil/environment-variables.cc b/src/libutil/environment-variables.cc index 0b668f125c1..adae177347c 100644 --- a/src/libutil/environment-variables.cc +++ b/src/libutil/environment-variables.cc @@ -21,9 +21,9 @@ std::optional getEnvNonEmpty(const std::string & key) return value; } -std::map getEnv() +StringMap getEnv() { - std::map env; + StringMap env; for (size_t i = 0; environ[i]; ++i) { auto s = environ[i]; auto eq = strchr(s, '='); @@ -41,7 +41,7 @@ void clearEnv() unsetenv(name.first.c_str()); } -void replaceEnv(const std::map & newEnv) +void replaceEnv(const StringMap & newEnv) { clearEnv(); for (auto & newEnvVar : newEnv) diff --git a/src/libutil/include/nix/util/environment-variables.hh b/src/libutil/include/nix/util/environment-variables.hh index d6c7472fcf4..9b2fab4f487 100644 --- a/src/libutil/include/nix/util/environment-variables.hh +++ b/src/libutil/include/nix/util/environment-variables.hh @@ -34,7 +34,7 @@ std::optional getEnvNonEmpty(const std::string & key); /** * Get the entire environment. */ -std::map getEnv(); +StringMap getEnv(); #ifdef _WIN32 /** @@ -64,6 +64,6 @@ void clearEnv(); /** * Replace the entire environment with the given one. */ -void replaceEnv(const std::map & newEnv); +void replaceEnv(const StringMap & newEnv); } diff --git a/src/libutil/include/nix/util/processes.hh b/src/libutil/include/nix/util/processes.hh index ef7bddf2fef..ab5f23e49ad 100644 --- a/src/libutil/include/nix/util/processes.hh +++ b/src/libutil/include/nix/util/processes.hh @@ -103,7 +103,7 @@ struct RunOptions std::optional gid; #endif std::optional chdir; - std::optional> environment; + std::optional environment; std::optional input; Source * standardIn = nullptr; Sink * standardOut = nullptr; diff --git a/src/libutil/include/nix/util/url.hh b/src/libutil/include/nix/util/url.hh index ced846787b3..a509f06dacf 100644 --- a/src/libutil/include/nix/util/url.hh +++ b/src/libutil/include/nix/util/url.hh @@ -10,7 +10,7 @@ struct ParsedURL std::string scheme; std::optional authority; std::string path; - std::map query; + StringMap query; std::string fragment; std::string to_string() const; @@ -30,9 +30,9 @@ MakeError(BadURL, Error); std::string percentDecode(std::string_view in); std::string percentEncode(std::string_view s, std::string_view keep=""); -std::map decodeQuery(const std::string & query); +StringMap decodeQuery(const std::string & query); -std::string encodeQuery(const std::map & query); +std::string encodeQuery(const StringMap & query); ParsedURL parseURL(const std::string & url); diff --git a/src/libutil/include/nix/util/xml-writer.hh b/src/libutil/include/nix/util/xml-writer.hh index 74f53b7caa4..ae5a6ced7ef 100644 --- a/src/libutil/include/nix/util/xml-writer.hh +++ b/src/libutil/include/nix/util/xml-writer.hh @@ -10,7 +10,7 @@ namespace nix { -typedef std::map XMLAttrs; +typedef std::map> XMLAttrs; class XMLWriter diff --git a/src/libutil/linux/cgroup.cc b/src/libutil/linux/cgroup.cc index 4acfe82f179..c82fdc11cdd 100644 --- a/src/libutil/linux/cgroup.cc +++ b/src/libutil/linux/cgroup.cc @@ -31,9 +31,9 @@ std::optional getCgroupFS() } // FIXME: obsolete, check for cgroup2 -std::map getCgroups(const Path & cgroupFile) +StringMap getCgroups(const Path & cgroupFile) { - std::map cgroups; + StringMap cgroups; for (auto & line : tokenizeString>(readFile(cgroupFile), "\n")) { static std::regex regex("([0-9]+):([^:]*):(.*)"); diff --git a/src/libutil/linux/include/nix/util/cgroup.hh b/src/libutil/linux/include/nix/util/cgroup.hh index 6a41c6b4457..eb49c341986 100644 --- a/src/libutil/linux/include/nix/util/cgroup.hh +++ b/src/libutil/linux/include/nix/util/cgroup.hh @@ -10,7 +10,7 @@ namespace nix { std::optional getCgroupFS(); -std::map getCgroups(const Path & cgroupFile); +StringMap getCgroups(const Path & cgroupFile); struct CgroupStats { diff --git a/src/libutil/url.cc b/src/libutil/url.cc index eaa2b0682a8..b7286072dac 100644 --- a/src/libutil/url.cc +++ b/src/libutil/url.cc @@ -70,9 +70,9 @@ std::string percentDecode(std::string_view in) return decoded; } -std::map decodeQuery(const std::string & query) +StringMap decodeQuery(const std::string & query) { - std::map result; + StringMap result; for (const auto & s : tokenizeString(query, "&")) { auto e = s.find('='); @@ -108,7 +108,7 @@ std::string percentEncode(std::string_view s, std::string_view keep) return res; } -std::string encodeQuery(const std::map & ss) +std::string encodeQuery(const StringMap & ss) { std::string res; bool first = true; diff --git a/src/nix-channel/nix-channel.cc b/src/nix-channel/nix-channel.cc index bc1164a4345..c4a05865823 100644 --- a/src/nix-channel/nix-channel.cc +++ b/src/nix-channel/nix-channel.cc @@ -18,7 +18,7 @@ using namespace nix; -typedef std::map Channels; +typedef StringMap Channels; static Channels channels; static std::filesystem::path channelsList; diff --git a/src/nix/develop.cc b/src/nix/develop.cc index b525e5de384..37bce6ca078 100644 --- a/src/nix/develop.cc +++ b/src/nix/develop.cc @@ -55,12 +55,12 @@ struct BuildEnvironment using Array = std::vector; - using Associative = std::map; + using Associative = StringMap; using Value = std::variant; std::map vars; - std::map bashFunctions; + StringMap bashFunctions; std::optional> structuredAttrs; static BuildEnvironment fromJSON(const nlohmann::json & json) From 49d026083b52dd0f79cb46df65510d219bf999c6 Mon Sep 17 00:00:00 2001 From: Sergei Zimmerman Date: Mon, 19 May 2025 20:38:19 +0000 Subject: [PATCH 039/218] Use transparent comparators for `StringMap` and `StringPairs` --- src/libutil/include/nix/util/types.hh | 21 +++++++++++++++++++-- 1 file changed, 19 insertions(+), 2 deletions(-) diff --git a/src/libutil/include/nix/util/types.hh b/src/libutil/include/nix/util/types.hh index 5139256cad7..edb34f5e20f 100644 --- a/src/libutil/include/nix/util/types.hh +++ b/src/libutil/include/nix/util/types.hh @@ -12,8 +12,25 @@ namespace nix { typedef std::list Strings; -typedef std::map StringMap; -typedef std::map StringPairs; + +/** + * Alias to ordered std::string -> std::string map container with transparent comparator. + * + * Used instead of std::map to use C++14 N3657 [1] + * heterogenous lookup consistently across the whole codebase. + * Transparent comparators get rid of creation of unnecessary + * temporary variables when looking up keys by `std::string_view` + * or C-style `const char *` strings. + * + * [1]: https://www.open-std.org/jtc1/sc22/wg21/docs/papers/2013/n3657.htm + */ +using StringMap = std::map>; +/** + * Alias to an ordered map of std::string -> std::string. Uses transparent comparator. + * + * @see StringMap + */ +using StringPairs = StringMap; /** * Alias to ordered set container with transparent comparator. From 803c0086f375069679a3f28eccbbe033c66a363f Mon Sep 17 00:00:00 2001 From: Eelco Dolstra Date: Tue, 20 May 2025 15:39:40 +0200 Subject: [PATCH 040/218] Update src/nix/profile-add.md MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-authored-by: Jörg Thalheim --- src/nix/profile-add.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/nix/profile-add.md b/src/nix/profile-add.md index 0bb65d8e696..f1d5391efa7 100644 --- a/src/nix/profile-add.md +++ b/src/nix/profile-add.md @@ -32,6 +32,6 @@ This command adds [_installables_](./nix.md#installables) to a Nix profile. > **Note** > -> `nix profile install` is an alias for `nix profile add` in Determinate Nix. +> `nix profile install` is an alias for `nix profile add`. )"" From d1295448e08585dbb0f3b248166c2640c6e24f8d Mon Sep 17 00:00:00 2001 From: John Ericson Date: Mon, 28 Apr 2025 11:25:21 -0400 Subject: [PATCH 041/218] Copy files before split Same technique as 6c2a7fdc496b9558985e8971eadfe415887c356c. --- maintainers/flake-module.nix | 2 + .../build/derivation-building-goal.cc | 1563 +++++++++++++++++ .../store/build/derivation-building-goal.hh | 239 +++ 3 files changed, 1804 insertions(+) create mode 100644 src/libstore/build/derivation-building-goal.cc create mode 100644 src/libstore/include/nix/store/build/derivation-building-goal.hh diff --git a/maintainers/flake-module.nix b/maintainers/flake-module.nix index 6497b17c13f..224f4726839 100644 --- a/maintainers/flake-module.nix +++ b/maintainers/flake-module.nix @@ -276,6 +276,8 @@ ''^src/libstore/store-api\.cc$'' ''^src/libstore/include/nix/store/store-api\.hh$'' ''^src/libstore/include/nix/store/store-dir-config\.hh$'' + ''^src/libstore/build/derivation-building-goal\.cc$'' + ''^src/libstore/include/nix/store/build/derivation-building-goal\.hh$'' ''^src/libstore/build/derivation-goal\.cc$'' ''^src/libstore/include/nix/store/build/derivation-goal\.hh$'' ''^src/libstore/build/drv-output-substitution-goal\.cc$'' diff --git a/src/libstore/build/derivation-building-goal.cc b/src/libstore/build/derivation-building-goal.cc new file mode 100644 index 00000000000..82c350832ee --- /dev/null +++ b/src/libstore/build/derivation-building-goal.cc @@ -0,0 +1,1563 @@ +#include "nix/store/build/derivation-goal.hh" +#ifndef _WIN32 // TODO enable build hook on Windows +# include "nix/store/build/hook-instance.hh" +# include "nix/store/build/derivation-builder.hh" +#endif +#include "nix/util/processes.hh" +#include "nix/util/config-global.hh" +#include "nix/store/build/worker.hh" +#include "nix/util/util.hh" +#include "nix/util/compression.hh" +#include "nix/store/common-protocol.hh" +#include "nix/store/common-protocol-impl.hh" +#include "nix/store/local-store.hh" // TODO remove, along with remaining downcasts + +#include +#include +#include +#include + +#include + +#include "nix/util/strings.hh" + +namespace nix { + +DerivationGoal::DerivationGoal(const StorePath & drvPath, + const OutputsSpec & wantedOutputs, Worker & worker, BuildMode buildMode) + : Goal(worker, loadDerivation()) + , drvPath(drvPath) + , wantedOutputs(wantedOutputs) + , buildMode(buildMode) +{ + name = fmt( + "building of '%s' from .drv file", + DerivedPath::Built { makeConstantStorePathRef(drvPath), wantedOutputs }.to_string(worker.store)); + trace("created"); + + mcExpectedBuilds = std::make_unique>(worker.expectedBuilds); + worker.updateProgress(); +} + + +DerivationGoal::DerivationGoal(const StorePath & drvPath, const BasicDerivation & drv, + const OutputsSpec & wantedOutputs, Worker & worker, BuildMode buildMode) + : Goal(worker, haveDerivation()) + , drvPath(drvPath) + , wantedOutputs(wantedOutputs) + , buildMode(buildMode) +{ + this->drv = std::make_unique(drv); + + name = fmt( + "building of '%s' from in-memory derivation", + DerivedPath::Built { makeConstantStorePathRef(drvPath), drv.outputNames() }.to_string(worker.store)); + trace("created"); + + mcExpectedBuilds = std::make_unique>(worker.expectedBuilds); + worker.updateProgress(); + + /* Prevent the .chroot directory from being + garbage-collected. (See isActiveTempFile() in gc.cc.) */ + worker.store.addTempRoot(this->drvPath); +} + + +DerivationGoal::~DerivationGoal() +{ + /* Careful: we should never ever throw an exception from a + destructor. */ + try { killChild(); } catch (...) { ignoreExceptionInDestructor(); } +#ifndef _WIN32 // TODO enable `DerivationBuilder` on Windows + if (builder) { + try { builder->stopDaemon(); } catch (...) { ignoreExceptionInDestructor(); } + try { builder->deleteTmpDir(false); } catch (...) { ignoreExceptionInDestructor(); } + } +#endif + try { closeLogFile(); } catch (...) { ignoreExceptionInDestructor(); } +} + + +std::string DerivationGoal::key() +{ + /* Ensure that derivations get built in order of their name, + i.e. a derivation named "aardvark" always comes before + "baboon". And substitution goals always happen before + derivation goals (due to "b$"). */ + return "b$" + std::string(drvPath.name()) + "$" + worker.store.printStorePath(drvPath); +} + + +void DerivationGoal::killChild() +{ +#ifndef _WIN32 // TODO enable build hook on Windows + hook.reset(); +#endif +#ifndef _WIN32 // TODO enable `DerivationBuilder` on Windows + if (builder && builder->pid != -1) { + worker.childTerminated(this); + + /* If we're using a build user, then there is a tricky race + condition: if we kill the build user before the child has + done its setuid() to the build user uid, then it won't be + killed, and we'll potentially lock up in pid.wait(). So + also send a conventional kill to the child. */ + ::kill(-builder->pid, SIGKILL); /* ignore the result */ + + builder->killSandbox(true); + + builder->pid.wait(); + } +#endif +} + + +void DerivationGoal::timedOut(Error && ex) +{ + killChild(); + // We're not inside a coroutine, hence we can't use co_return here. + // Thus we ignore the return value. + [[maybe_unused]] Done _ = done(BuildResult::TimedOut, {}, std::move(ex)); +} + +void DerivationGoal::addWantedOutputs(const OutputsSpec & outputs) +{ + auto newWanted = wantedOutputs.union_(outputs); + switch (needRestart) { + case NeedRestartForMoreOutputs::OutputsUnmodifedDontNeed: + if (!newWanted.isSubsetOf(wantedOutputs)) + needRestart = NeedRestartForMoreOutputs::OutputsAddedDoNeed; + break; + case NeedRestartForMoreOutputs::OutputsAddedDoNeed: + /* No need to check whether we added more outputs, because a + restart is already queued up. */ + break; + case NeedRestartForMoreOutputs::BuildInProgressWillNotNeed: + /* We are already building all outputs, so it doesn't matter if + we now want more. */ + break; + }; + wantedOutputs = newWanted; +} + + +Goal::Co DerivationGoal::loadDerivation() { + trace("local derivation"); + + { + /* The first thing to do is to make sure that the derivation + exists. If it doesn't, it may be created through a + substitute. */ + + if (buildMode != bmNormal || !worker.evalStore.isValidPath(drvPath)) { + Goals waitees{upcast_goal(worker.makePathSubstitutionGoal(drvPath))}; + co_await await(std::move(waitees)); + } + + trace("loading derivation"); + + if (nrFailed != 0) { + co_return done(BuildResult::MiscFailure, {}, Error("cannot build missing derivation '%s'", worker.store.printStorePath(drvPath))); + } + + /* `drvPath' should already be a root, but let's be on the safe + side: if the user forgot to make it a root, we wouldn't want + things being garbage collected while we're busy. */ + worker.evalStore.addTempRoot(drvPath); + + /* Get the derivation. It is probably in the eval store, but it might be inthe main store: + + - Resolved derivation are resolved against main store realisations, and so must be stored there. + + - Dynamic derivations are built, and so are found in the main store. + */ + for (auto * drvStore : { &worker.evalStore, &worker.store }) { + if (drvStore->isValidPath(drvPath)) { + drv = std::make_unique(drvStore->readDerivation(drvPath)); + break; + } + } + assert(drv); + } + + co_return haveDerivation(); +} + + +Goal::Co DerivationGoal::haveDerivation() +{ + trace("have derivation"); + + if (auto parsedOpt = StructuredAttrs::tryParse(drv->env)) { + parsedDrv = std::make_unique(*parsedOpt); + } + try { + drvOptions = std::make_unique( + DerivationOptions::fromStructuredAttrs(drv->env, parsedDrv.get())); + } catch (Error & e) { + e.addTrace({}, "while parsing derivation '%s'", worker.store.printStorePath(drvPath)); + throw; + } + + if (!drv->type().hasKnownOutputPaths()) + experimentalFeatureSettings.require(Xp::CaDerivations); + + for (auto & i : drv->outputsAndOptPaths(worker.store)) + if (i.second.second) + worker.store.addTempRoot(*i.second.second); + + { + bool impure = drv->type().isImpure(); + + if (impure) experimentalFeatureSettings.require(Xp::ImpureDerivations); + + auto outputHashes = staticOutputHashes(worker.evalStore, *drv); + for (auto & [outputName, outputHash] : outputHashes) { + InitialOutput v{ + .wanted = true, // Will be refined later + .outputHash = outputHash + }; + + /* TODO we might want to also allow randomizing the paths + for regular CA derivations, e.g. for sake of checking + determinism. */ + if (impure) { + v.known = InitialOutputStatus { + .path = StorePath::random(outputPathName(drv->name, outputName)), + .status = PathStatus::Absent, + }; + } + + initialOutputs.insert({ + outputName, + std::move(v), + }); + } + + if (impure) { + /* We don't yet have any safe way to cache an impure derivation at + this step. */ + co_return gaveUpOnSubstitution(); + } + } + + { + /* Check what outputs paths are not already valid. */ + auto [allValid, validOutputs] = checkPathValidity(); + + /* If they are all valid, then we're done. */ + if (allValid && buildMode == bmNormal) { + co_return done(BuildResult::AlreadyValid, std::move(validOutputs)); + } + } + + Goals waitees; + + /* We are first going to try to create the invalid output paths + through substitutes. If that doesn't work, we'll build + them. */ + if (settings.useSubstitutes && drvOptions->substitutesAllowed()) + for (auto & [outputName, status] : initialOutputs) { + if (!status.wanted) continue; + if (!status.known) + waitees.insert( + upcast_goal( + worker.makeDrvOutputSubstitutionGoal( + DrvOutput{status.outputHash, outputName}, + buildMode == bmRepair ? Repair : NoRepair + ) + ) + ); + else { + auto * cap = getDerivationCA(*drv); + waitees.insert(upcast_goal(worker.makePathSubstitutionGoal( + status.known->path, + buildMode == bmRepair ? Repair : NoRepair, + cap ? std::optional { *cap } : std::nullopt))); + } + } + + co_await await(std::move(waitees)); + + trace("all outputs substituted (maybe)"); + + assert(!drv->type().isImpure()); + + if (nrFailed > 0 && nrFailed > nrNoSubstituters && !settings.tryFallback) { + co_return done(BuildResult::TransientFailure, {}, + Error("some substitutes for the outputs of derivation '%s' failed (usually happens due to networking issues); try '--fallback' to build derivation from source ", + worker.store.printStorePath(drvPath))); + } + + nrFailed = nrNoSubstituters = 0; + + if (needRestart == NeedRestartForMoreOutputs::OutputsAddedDoNeed) { + needRestart = NeedRestartForMoreOutputs::OutputsUnmodifedDontNeed; + co_return haveDerivation(); + } + + auto [allValid, validOutputs] = checkPathValidity(); + + if (buildMode == bmNormal && allValid) { + co_return done(BuildResult::Substituted, std::move(validOutputs)); + } + if (buildMode == bmRepair && allValid) { + co_return repairClosure(); + } + if (buildMode == bmCheck && !allValid) + throw Error("some outputs of '%s' are not valid, so checking is not possible", + worker.store.printStorePath(drvPath)); + + /* Nothing to wait for; tail call */ + co_return gaveUpOnSubstitution(); +} + + +/** + * Used for `inputGoals` local variable below + */ +struct value_comparison +{ + template + bool operator()(const ref & lhs, const ref & rhs) const { + return *lhs < *rhs; + } +}; + + +std::string showKnownOutputs(Store & store, const Derivation & drv) +{ + std::string msg; + StorePathSet expectedOutputPaths; + for (auto & i : drv.outputsAndOptPaths(store)) + if (i.second.second) + expectedOutputPaths.insert(*i.second.second); + if (!expectedOutputPaths.empty()) { + msg += "\nOutput paths:"; + for (auto & p : expectedOutputPaths) + msg += fmt("\n %s", Magenta(store.printStorePath(p))); + } + return msg; +} + + +/* At least one of the output paths could not be + produced using a substitute. So we have to build instead. */ +Goal::Co DerivationGoal::gaveUpOnSubstitution() +{ + /* At this point we are building all outputs, so if more are wanted there + is no need to restart. */ + needRestart = NeedRestartForMoreOutputs::BuildInProgressWillNotNeed; + + Goals waitees; + + std::map, GoalPtr, value_comparison> inputGoals; + + { + std::function, const DerivedPathMap::ChildNode &)> addWaiteeDerivedPath; + + addWaiteeDerivedPath = [&](ref inputDrv, const DerivedPathMap::ChildNode & inputNode) { + if (!inputNode.value.empty()) { + auto g = worker.makeGoal( + DerivedPath::Built { + .drvPath = inputDrv, + .outputs = inputNode.value, + }, + buildMode == bmRepair ? bmRepair : bmNormal); + inputGoals.insert_or_assign(inputDrv, g); + waitees.insert(std::move(g)); + } + for (const auto & [outputName, childNode] : inputNode.childMap) + addWaiteeDerivedPath( + make_ref(SingleDerivedPath::Built { inputDrv, outputName }), + childNode); + }; + + for (const auto & [inputDrvPath, inputNode] : drv->inputDrvs.map) { + /* Ensure that pure, non-fixed-output derivations don't + depend on impure derivations. */ + if (experimentalFeatureSettings.isEnabled(Xp::ImpureDerivations) && !drv->type().isImpure() && !drv->type().isFixed()) { + auto inputDrv = worker.evalStore.readDerivation(inputDrvPath); + if (inputDrv.type().isImpure()) + throw Error("pure derivation '%s' depends on impure derivation '%s'", + worker.store.printStorePath(drvPath), + worker.store.printStorePath(inputDrvPath)); + } + + addWaiteeDerivedPath(makeConstantStorePathRef(inputDrvPath), inputNode); + } + } + + /* Copy the input sources from the eval store to the build + store. + + Note that some inputs might not be in the eval store because they + are (resolved) derivation outputs in a resolved derivation. */ + if (&worker.evalStore != &worker.store) { + RealisedPath::Set inputSrcs; + for (auto & i : drv->inputSrcs) + if (worker.evalStore.isValidPath(i)) + inputSrcs.insert(i); + copyClosure(worker.evalStore, worker.store, inputSrcs); + } + + for (auto & i : drv->inputSrcs) { + if (worker.store.isValidPath(i)) continue; + if (!settings.useSubstitutes) + throw Error("dependency '%s' of '%s' does not exist, and substitution is disabled", + worker.store.printStorePath(i), worker.store.printStorePath(drvPath)); + waitees.insert(upcast_goal(worker.makePathSubstitutionGoal(i))); + } + + co_await await(std::move(waitees)); + + + trace("all inputs realised"); + + if (nrFailed != 0) { + auto msg = fmt( + "Cannot build '%s'.\n" + "Reason: " ANSI_RED "%d %s failed" ANSI_NORMAL ".", + Magenta(worker.store.printStorePath(drvPath)), + nrFailed, + nrFailed == 1 ? "dependency" : "dependencies"); + msg += showKnownOutputs(worker.store, *drv); + co_return done(BuildResult::DependencyFailed, {}, Error(msg)); + } + + /* Gather information necessary for computing the closure and/or + running the build hook. */ + + /* Determine the full set of input paths. */ + + /* First, the input derivations. */ + { + auto & fullDrv = *drv; + + auto drvType = fullDrv.type(); + bool resolveDrv = std::visit(overloaded { + [&](const DerivationType::InputAddressed & ia) { + /* must resolve if deferred. */ + return ia.deferred; + }, + [&](const DerivationType::ContentAddressed & ca) { + return !fullDrv.inputDrvs.map.empty() && ( + ca.fixed + /* Can optionally resolve if fixed, which is good + for avoiding unnecessary rebuilds. */ + ? experimentalFeatureSettings.isEnabled(Xp::CaDerivations) + /* Must resolve if floating and there are any inputs + drvs. */ + : true); + }, + [&](const DerivationType::Impure &) { + return true; + } + }, drvType.raw) + /* no inputs are outputs of dynamic derivations */ + || std::ranges::any_of( + fullDrv.inputDrvs.map.begin(), + fullDrv.inputDrvs.map.end(), + [](auto & pair) { return !pair.second.childMap.empty(); }); + + if (resolveDrv && !fullDrv.inputDrvs.map.empty()) { + experimentalFeatureSettings.require(Xp::CaDerivations); + + /* We are be able to resolve this derivation based on the + now-known results of dependencies. If so, we become a + stub goal aliasing that resolved derivation goal. */ + std::optional attempt = fullDrv.tryResolve(worker.store, + [&](ref drvPath, const std::string & outputName) -> std::optional { + auto mEntry = get(inputGoals, drvPath); + if (!mEntry) return std::nullopt; + + auto buildResult = (*mEntry)->getBuildResult(DerivedPath::Built{drvPath, OutputsSpec::Names{outputName}}); + if (!buildResult.success()) return std::nullopt; + + auto i = get(buildResult.builtOutputs, outputName); + if (!i) return std::nullopt; + + return i->outPath; + }); + if (!attempt) { + /* TODO (impure derivations-induced tech debt) (see below): + The above attempt should have found it, but because we manage + inputDrvOutputs statefully, sometimes it gets out of sync with + the real source of truth (store). So we query the store + directly if there's a problem. */ + attempt = fullDrv.tryResolve(worker.store, &worker.evalStore); + } + assert(attempt); + Derivation drvResolved { std::move(*attempt) }; + + auto pathResolved = writeDerivation(worker.store, drvResolved); + + auto msg = fmt("resolved derivation: '%s' -> '%s'", + worker.store.printStorePath(drvPath), + worker.store.printStorePath(pathResolved)); + act = std::make_unique(*logger, lvlInfo, actBuildWaiting, msg, + Logger::Fields { + worker.store.printStorePath(drvPath), + worker.store.printStorePath(pathResolved), + }); + + auto resolvedDrvGoal = worker.makeDerivationGoal( + pathResolved, wantedOutputs, buildMode); + { + Goals waitees{resolvedDrvGoal}; + co_await await(std::move(waitees)); + } + + trace("resolved derivation finished"); + + auto resolvedDrv = *resolvedDrvGoal->drv; + auto & resolvedResult = resolvedDrvGoal->buildResult; + + SingleDrvOutputs builtOutputs; + + if (resolvedResult.success()) { + auto resolvedHashes = staticOutputHashes(worker.store, resolvedDrv); + + StorePathSet outputPaths; + + for (auto & outputName : resolvedDrv.outputNames()) { + auto initialOutput = get(initialOutputs, outputName); + auto resolvedHash = get(resolvedHashes, outputName); + if ((!initialOutput) || (!resolvedHash)) + throw Error( + "derivation '%s' doesn't have expected output '%s' (derivation-goal.cc/resolve)", + worker.store.printStorePath(drvPath), outputName); + + auto realisation = [&]{ + auto take1 = get(resolvedResult.builtOutputs, outputName); + if (take1) return *take1; + + /* The above `get` should work. But sateful tracking of + outputs in resolvedResult, this can get out of sync with the + store, which is our actual source of truth. For now we just + check the store directly if it fails. */ + auto take2 = worker.evalStore.queryRealisation(DrvOutput { *resolvedHash, outputName }); + if (take2) return *take2; + + throw Error( + "derivation '%s' doesn't have expected output '%s' (derivation-goal.cc/realisation)", + worker.store.printStorePath(resolvedDrvGoal->drvPath), outputName); + }(); + + if (!drv->type().isImpure()) { + auto newRealisation = realisation; + newRealisation.id = DrvOutput { initialOutput->outputHash, outputName }; + newRealisation.signatures.clear(); + if (!drv->type().isFixed()) { + auto & drvStore = worker.evalStore.isValidPath(drvPath) + ? worker.evalStore + : worker.store; + newRealisation.dependentRealisations = drvOutputReferences(worker.store, *drv, realisation.outPath, &drvStore); + } + worker.store.signRealisation(newRealisation); + worker.store.registerDrvOutput(newRealisation); + } + outputPaths.insert(realisation.outPath); + builtOutputs.emplace(outputName, realisation); + } + + runPostBuildHook( + worker.store, + *logger, + drvPath, + outputPaths + ); + } + + auto status = resolvedResult.status; + if (status == BuildResult::AlreadyValid) + status = BuildResult::ResolvesToAlreadyValid; + + co_return done(status, std::move(builtOutputs)); + } + + /* If we get this far, we know no dynamic drvs inputs */ + + for (auto & [depDrvPath, depNode] : fullDrv.inputDrvs.map) { + for (auto & outputName : depNode.value) { + /* Don't need to worry about `inputGoals`, because + impure derivations are always resolved above. Can + just use DB. This case only happens in the (older) + input addressed and fixed output derivation cases. */ + auto outMap = [&]{ + for (auto * drvStore : { &worker.evalStore, &worker.store }) + if (drvStore->isValidPath(depDrvPath)) + return worker.store.queryDerivationOutputMap(depDrvPath, drvStore); + assert(false); + }(); + + auto outMapPath = outMap.find(outputName); + if (outMapPath == outMap.end()) { + throw Error( + "derivation '%s' requires non-existent output '%s' from input derivation '%s'", + worker.store.printStorePath(drvPath), outputName, worker.store.printStorePath(depDrvPath)); + } + + worker.store.computeFSClosure(outMapPath->second, inputPaths); + } + } + } + + /* Second, the input sources. */ + worker.store.computeFSClosure(drv->inputSrcs, inputPaths); + + debug("added input paths %s", worker.store.showPaths(inputPaths)); + + /* Okay, try to build. Note that here we don't wait for a build + slot to become available, since we don't need one if there is a + build hook. */ + co_await yield(); + co_return tryToBuild(); +} + +void DerivationGoal::started() +{ + auto msg = fmt( + buildMode == bmRepair ? "repairing outputs of '%s'" : + buildMode == bmCheck ? "checking outputs of '%s'" : + "building '%s'", worker.store.printStorePath(drvPath)); + fmt("building '%s'", worker.store.printStorePath(drvPath)); +#ifndef _WIN32 // TODO enable build hook on Windows + if (hook) msg += fmt(" on '%s'", machineName); +#endif + act = std::make_unique(*logger, lvlInfo, actBuild, msg, + Logger::Fields{worker.store.printStorePath(drvPath), +#ifndef _WIN32 // TODO enable build hook on Windows + hook ? machineName : +#endif + "", + 1, + 1}); + mcRunningBuilds = std::make_unique>(worker.runningBuilds); + worker.updateProgress(); +} + +Goal::Co DerivationGoal::tryToBuild() +{ + trace("trying to build"); + + /* Obtain locks on all output paths, if the paths are known a priori. + + The locks are automatically released when we exit this function or Nix + crashes. If we can't acquire the lock, then continue; hopefully some + other goal can start a build, and if not, the main loop will sleep a few + seconds and then retry this goal. */ + PathSet lockFiles; + /* FIXME: Should lock something like the drv itself so we don't build same + CA drv concurrently */ + if (dynamic_cast(&worker.store)) { + /* If we aren't a local store, we might need to use the local store as + a build remote, but that would cause a deadlock. */ + /* FIXME: Make it so we can use ourselves as a build remote even if we + are the local store (separate locking for building vs scheduling? */ + /* FIXME: find some way to lock for scheduling for the other stores so + a forking daemon with --store still won't farm out redundant builds. + */ + for (auto & i : drv->outputsAndOptPaths(worker.store)) { + if (i.second.second) + lockFiles.insert(worker.store.Store::toRealPath(*i.second.second)); + else + lockFiles.insert( + worker.store.Store::toRealPath(drvPath) + "." + i.first + ); + } + } + + if (!outputLocks.lockPaths(lockFiles, "", false)) + { + Activity act(*logger, lvlWarn, actBuildWaiting, + fmt("waiting for lock on %s", Magenta(showPaths(lockFiles)))); + + /* Wait then try locking again, repeat until success (returned + boolean is true). */ + do { + co_await waitForAWhile(); + } while (!outputLocks.lockPaths(lockFiles, "", false)); + } + + /* Now check again whether the outputs are valid. This is because + another process may have started building in parallel. After + it has finished and released the locks, we can (and should) + reuse its results. (Strictly speaking the first check can be + omitted, but that would be less efficient.) Note that since we + now hold the locks on the output paths, no other process can + build this derivation, so no further checks are necessary. */ + auto [allValid, validOutputs] = checkPathValidity(); + + if (buildMode != bmCheck && allValid) { + debug("skipping build of derivation '%s', someone beat us to it", worker.store.printStorePath(drvPath)); + outputLocks.setDeletion(true); + outputLocks.unlock(); + co_return done(BuildResult::AlreadyValid, std::move(validOutputs)); + } + + /* If any of the outputs already exist but are not valid, delete + them. */ + for (auto & [_, status] : initialOutputs) { + if (!status.known || status.known->isValid()) continue; + auto storePath = status.known->path; + debug("removing invalid path '%s'", worker.store.printStorePath(status.known->path)); + deletePath(worker.store.Store::toRealPath(storePath)); + } + + /* Don't do a remote build if the derivation has the attribute + `preferLocalBuild' set. Also, check and repair modes are only + supported for local builds. */ + bool buildLocally = + (buildMode != bmNormal || drvOptions->willBuildLocally(worker.store, *drv)) + && settings.maxBuildJobs.get() != 0; + + if (!buildLocally) { + switch (tryBuildHook()) { + case rpAccept: + /* Yes, it has started doing so. Wait until we get + EOF from the hook. */ + actLock.reset(); + buildResult.startTime = time(0); // inexact + started(); + co_await Suspend{}; + co_return hookDone(); + case rpPostpone: + /* Not now; wait until at least one child finishes or + the wake-up timeout expires. */ + if (!actLock) + actLock = std::make_unique(*logger, lvlWarn, actBuildWaiting, + fmt("waiting for a machine to build '%s'", Magenta(worker.store.printStorePath(drvPath)))); + outputLocks.unlock(); + co_await waitForAWhile(); + co_return tryToBuild(); + case rpDecline: + /* We should do it ourselves. */ + break; + } + } + + actLock.reset(); + + co_await yield(); + + if (!dynamic_cast(&worker.store)) { + throw Error( + R"( + Unable to build with a primary store that isn't a local store; + either pass a different '--store' or enable remote builds. + + For more information check 'man nix.conf' and search for '/machines'. + )" + ); + } + +#ifdef _WIN32 // TODO enable `DerivationBuilder` on Windows + throw UnimplementedError("building derivations is not yet implemented on Windows"); +#else + + // Will continue here while waiting for a build user below + while (true) { + + assert(!hook); + + unsigned int curBuilds = worker.getNrLocalBuilds(); + if (curBuilds >= settings.maxBuildJobs) { + outputLocks.unlock(); + co_await waitForBuildSlot(); + co_return tryToBuild(); + } + + if (!builder) { + /** + * Local implementation of these virtual methods, consider + * this just a record of lambdas. + */ + struct DerivationGoalCallbacks : DerivationBuilderCallbacks + { + DerivationGoal & goal; + + DerivationGoalCallbacks(DerivationGoal & goal, std::unique_ptr & builder) + : goal{goal} + {} + + ~DerivationGoalCallbacks() override = default; + + void childStarted(Descriptor builderOut) override + { + goal.worker.childStarted(goal.shared_from_this(), {builderOut}, true, true); + } + + void childTerminated() override + { + goal.worker.childTerminated(&goal); + } + + void noteHashMismatch() override + { + goal.worker.hashMismatch = true; + } + + void noteCheckMismatch() override + { + goal.worker.checkMismatch = true; + } + + void markContentsGood(const StorePath & path) override + { + goal.worker.markContentsGood(path); + } + + Path openLogFile() override { + return goal.openLogFile(); + } + void closeLogFile() override { + goal.closeLogFile(); + } + SingleDrvOutputs assertPathValidity() override { + return goal.assertPathValidity(); + } + void appendLogTailErrorMsg(std::string & msg) override { + goal.appendLogTailErrorMsg(msg); + } + }; + + /* If we have to wait and retry (see below), then `builder` will + already be created, so we don't need to create it again. */ + builder = makeDerivationBuilder( + worker.store, + std::make_unique(*this, builder), + DerivationBuilderParams { + drvPath, + buildMode, + buildResult, + *drv, + parsedDrv.get(), + *drvOptions, + inputPaths, + initialOutputs, + }); + } + + if (!builder->prepareBuild()) { + if (!actLock) + actLock = std::make_unique(*logger, lvlWarn, actBuildWaiting, + fmt("waiting for a free build user ID for '%s'", Magenta(worker.store.printStorePath(drvPath)))); + co_await waitForAWhile(); + continue; + } + + break; + } + + actLock.reset(); + + try { + + /* Okay, we have to build. */ + builder->startBuilder(); + + } catch (BuildError & e) { + outputLocks.unlock(); + builder->buildUser.reset(); + worker.permanentFailure = true; + co_return done(BuildResult::InputRejected, {}, std::move(e)); + } + + started(); + co_await Suspend{}; + + trace("build done"); + + auto res = builder->unprepareBuild(); + // N.B. cannot use `std::visit` with co-routine return + if (auto * ste = std::get_if<0>(&res)) { + outputLocks.unlock(); + co_return done(std::move(ste->first), {}, std::move(ste->second)); + } else if (auto * builtOutputs = std::get_if<1>(&res)) { + /* It is now safe to delete the lock files, since all future + lockers will see that the output paths are valid; they will + not create new lock files with the same names as the old + (unlinked) lock files. */ + outputLocks.setDeletion(true); + outputLocks.unlock(); + co_return done(BuildResult::Built, std::move(*builtOutputs)); + } else { + unreachable(); + } +#endif +} + + +Goal::Co DerivationGoal::repairClosure() +{ + assert(!drv->type().isImpure()); + + /* If we're repairing, we now know that our own outputs are valid. + Now check whether the other paths in the outputs closure are + good. If not, then start derivation goals for the derivations + that produced those outputs. */ + + /* Get the output closure. */ + auto outputs = queryDerivationOutputMap(); + StorePathSet outputClosure; + for (auto & i : outputs) { + if (!wantedOutputs.contains(i.first)) continue; + worker.store.computeFSClosure(i.second, outputClosure); + } + + /* Filter out our own outputs (which we have already checked). */ + for (auto & i : outputs) + outputClosure.erase(i.second); + + /* Get all dependencies of this derivation so that we know which + derivation is responsible for which path in the output + closure. */ + StorePathSet inputClosure; + + /* If we're working from an in-memory derivation with no in-store + `*.drv` file, we cannot do this part. */ + if (worker.store.isValidPath(drvPath)) + worker.store.computeFSClosure(drvPath, inputClosure); + + std::map outputsToDrv; + for (auto & i : inputClosure) + if (i.isDerivation()) { + auto depOutputs = worker.store.queryPartialDerivationOutputMap(i, &worker.evalStore); + for (auto & j : depOutputs) + if (j.second) + outputsToDrv.insert_or_assign(*j.second, i); + } + + Goals waitees; + + /* Check each path (slow!). */ + for (auto & i : outputClosure) { + if (worker.pathContentsGood(i)) continue; + printError( + "found corrupted or missing path '%s' in the output closure of '%s'", + worker.store.printStorePath(i), worker.store.printStorePath(drvPath)); + auto drvPath2 = outputsToDrv.find(i); + if (drvPath2 == outputsToDrv.end()) + waitees.insert(upcast_goal(worker.makePathSubstitutionGoal(i, Repair))); + else + waitees.insert(worker.makeGoal( + DerivedPath::Built { + .drvPath = makeConstantStorePathRef(drvPath2->second), + .outputs = OutputsSpec::All { }, + }, + bmRepair)); + } + + co_await await(std::move(waitees)); + + if (!waitees.empty()) { + trace("closure repaired"); + if (nrFailed > 0) + throw Error("some paths in the output closure of derivation '%s' could not be repaired", + worker.store.printStorePath(drvPath)); + } + co_return done(BuildResult::AlreadyValid, assertPathValidity()); +} + + +void runPostBuildHook( + Store & store, + Logger & logger, + const StorePath & drvPath, + const StorePathSet & outputPaths) +{ + auto hook = settings.postBuildHook; + if (hook == "") + return; + + Activity act(logger, lvlTalkative, actPostBuildHook, + fmt("running post-build-hook '%s'", settings.postBuildHook), + Logger::Fields{store.printStorePath(drvPath)}); + PushActivity pact(act.id); + StringMap hookEnvironment = getEnv(); + + hookEnvironment.emplace("DRV_PATH", store.printStorePath(drvPath)); + hookEnvironment.emplace("OUT_PATHS", chomp(concatStringsSep(" ", store.printStorePathSet(outputPaths)))); + hookEnvironment.emplace("NIX_CONFIG", globalConfig.toKeyValue()); + + struct LogSink : Sink { + Activity & act; + std::string currentLine; + + LogSink(Activity & act) : act(act) { } + + void operator() (std::string_view data) override { + for (auto c : data) { + if (c == '\n') { + flushLine(); + } else { + currentLine += c; + } + } + } + + void flushLine() { + act.result(resPostBuildLogLine, currentLine); + currentLine.clear(); + } + + ~LogSink() { + if (currentLine != "") { + currentLine += '\n'; + flushLine(); + } + } + }; + LogSink sink(act); + + runProgram2({ + .program = settings.postBuildHook, + .environment = hookEnvironment, + .standardOut = &sink, + .mergeStderrToStdout = true, + }); +} + + +void DerivationGoal::appendLogTailErrorMsg(std::string & msg) +{ + if (!logger->isVerbose() && !logTail.empty()) { + msg += fmt("\nLast %d log lines:\n", logTail.size()); + for (auto & line : logTail) { + msg += "> "; + msg += line; + msg += "\n"; + } + auto nixLogCommand = experimentalFeatureSettings.isEnabled(Xp::NixCommand) + ? "nix log" + : "nix-store -l"; + // The command is on a separate line for easy copying, such as with triple click. + // This message will be indented elsewhere, so removing the indentation before the + // command will not put it at the start of the line unfortunately. + msg += fmt("For full logs, run:\n " ANSI_BOLD "%s %s" ANSI_NORMAL, + nixLogCommand, + worker.store.printStorePath(drvPath)); + } +} + + +Goal::Co DerivationGoal::hookDone() +{ +#ifndef _WIN32 + assert(hook); +#endif + + trace("hook build done"); + + /* Since we got an EOF on the logger pipe, the builder is presumed + to have terminated. In fact, the builder could also have + simply have closed its end of the pipe, so just to be sure, + kill it. */ + int status = +#ifndef _WIN32 // TODO enable build hook on Windows + hook->pid.kill(); +#else + 0; +#endif + + debug("build hook for '%s' finished", worker.store.printStorePath(drvPath)); + + buildResult.timesBuilt++; + buildResult.stopTime = time(0); + + /* So the child is gone now. */ + worker.childTerminated(this); + + /* Close the read side of the logger pipe. */ +#ifndef _WIN32 // TODO enable build hook on Windows + hook->builderOut.readSide.close(); + hook->fromHook.readSide.close(); +#endif + + /* Close the log file. */ + closeLogFile(); + + /* Check the exit status. */ + if (!statusOk(status)) { + auto msg = fmt( + "Cannot build '%s'.\n" + "Reason: " ANSI_RED "builder %s" ANSI_NORMAL ".", + Magenta(worker.store.printStorePath(drvPath)), + statusToString(status)); + + msg += showKnownOutputs(worker.store, *drv); + + appendLogTailErrorMsg(msg); + + outputLocks.unlock(); + + /* TODO (once again) support fine-grained error codes, see issue #12641. */ + + co_return done(BuildResult::MiscFailure, {}, BuildError(msg)); + } + + /* Compute the FS closure of the outputs and register them as + being valid. */ + auto builtOutputs = + /* When using a build hook, the build hook can register the output + as valid (by doing `nix-store --import'). If so we don't have + to do anything here. + + We can only early return when the outputs are known a priori. For + floating content-addressing derivations this isn't the case. + */ + assertPathValidity(); + + StorePathSet outputPaths; + for (auto & [_, output] : builtOutputs) + outputPaths.insert(output.outPath); + runPostBuildHook( + worker.store, + *logger, + drvPath, + outputPaths + ); + + /* It is now safe to delete the lock files, since all future + lockers will see that the output paths are valid; they will + not create new lock files with the same names as the old + (unlinked) lock files. */ + outputLocks.setDeletion(true); + outputLocks.unlock(); + + co_return done(BuildResult::Built, std::move(builtOutputs)); +} + +HookReply DerivationGoal::tryBuildHook() +{ +#ifdef _WIN32 // TODO enable build hook on Windows + return rpDecline; +#else + /* This should use `worker.evalStore`, but per #13179 the build hook + doesn't work with eval store anyways. */ + if (settings.buildHook.get().empty() || !worker.tryBuildHook || !worker.store.isValidPath(drvPath)) return rpDecline; + + if (!worker.hook) + worker.hook = std::make_unique(); + + try { + + /* Send the request to the hook. */ + worker.hook->sink + << "try" + << (worker.getNrLocalBuilds() < settings.maxBuildJobs ? 1 : 0) + << drv->platform + << worker.store.printStorePath(drvPath) + << drvOptions->getRequiredSystemFeatures(*drv); + worker.hook->sink.flush(); + + /* Read the first line of input, which should be a word indicating + whether the hook wishes to perform the build. */ + std::string reply; + while (true) { + auto s = [&]() { + try { + return readLine(worker.hook->fromHook.readSide.get()); + } catch (Error & e) { + e.addTrace({}, "while reading the response from the build hook"); + throw; + } + }(); + if (handleJSONLogMessage(s, worker.act, worker.hook->activities, "the build hook", true)) + ; + else if (s.substr(0, 2) == "# ") { + reply = s.substr(2); + break; + } + else { + s += "\n"; + writeToStderr(s); + } + } + + debug("hook reply is '%1%'", reply); + + if (reply == "decline") + return rpDecline; + else if (reply == "decline-permanently") { + worker.tryBuildHook = false; + worker.hook = 0; + return rpDecline; + } + else if (reply == "postpone") + return rpPostpone; + else if (reply != "accept") + throw Error("bad hook reply '%s'", reply); + + } catch (SysError & e) { + if (e.errNo == EPIPE) { + printError( + "build hook died unexpectedly: %s", + chomp(drainFD(worker.hook->fromHook.readSide.get()))); + worker.hook = 0; + return rpDecline; + } else + throw; + } + + hook = std::move(worker.hook); + + try { + machineName = readLine(hook->fromHook.readSide.get()); + } catch (Error & e) { + e.addTrace({}, "while reading the machine name from the build hook"); + throw; + } + + CommonProto::WriteConn conn { hook->sink }; + + /* Tell the hook all the inputs that have to be copied to the + remote system. */ + CommonProto::write(worker.store, conn, inputPaths); + + /* Tell the hooks the missing outputs that have to be copied back + from the remote system. */ + { + StringSet missingOutputs; + for (auto & [outputName, status] : initialOutputs) { + // XXX: Does this include known CA outputs? + if (buildMode != bmCheck && status.known && status.known->isValid()) continue; + missingOutputs.insert(outputName); + } + CommonProto::write(worker.store, conn, missingOutputs); + } + + hook->sink = FdSink(); + hook->toHook.writeSide.close(); + + /* Create the log file and pipe. */ + [[maybe_unused]] Path logFile = openLogFile(); + + std::set fds; + fds.insert(hook->fromHook.readSide.get()); + fds.insert(hook->builderOut.readSide.get()); + worker.childStarted(shared_from_this(), fds, false, false); + + return rpAccept; +#endif +} + + +Path DerivationGoal::openLogFile() +{ + logSize = 0; + + if (!settings.keepLog) return ""; + + auto baseName = std::string(baseNameOf(worker.store.printStorePath(drvPath))); + + /* Create a log file. */ + Path logDir; + if (auto localStore = dynamic_cast(&worker.store)) + logDir = localStore->config->logDir; + else + logDir = settings.nixLogDir; + Path dir = fmt("%s/%s/%s/", logDir, LocalFSStore::drvsLogDir, baseName.substr(0, 2)); + createDirs(dir); + + Path logFileName = fmt("%s/%s%s", dir, baseName.substr(2), + settings.compressLog ? ".bz2" : ""); + + fdLogFile = toDescriptor(open(logFileName.c_str(), O_CREAT | O_WRONLY | O_TRUNC +#ifndef _WIN32 + | O_CLOEXEC +#endif + , 0666)); + if (!fdLogFile) throw SysError("creating log file '%1%'", logFileName); + + logFileSink = std::make_shared(fdLogFile.get()); + + if (settings.compressLog) + logSink = std::shared_ptr(makeCompressionSink("bzip2", *logFileSink)); + else + logSink = logFileSink; + + return logFileName; +} + + +void DerivationGoal::closeLogFile() +{ + auto logSink2 = std::dynamic_pointer_cast(logSink); + if (logSink2) logSink2->finish(); + if (logFileSink) logFileSink->flush(); + logSink = logFileSink = 0; + fdLogFile.close(); +} + + +bool DerivationGoal::isReadDesc(Descriptor fd) +{ +#ifdef _WIN32 // TODO enable build hook on Windows + return false; +#else + return + (hook && fd == hook->builderOut.readSide.get()) + || + (builder && fd == builder->builderOut.get()); +#endif +} + +void DerivationGoal::handleChildOutput(Descriptor fd, std::string_view data) +{ + // local & `ssh://`-builds are dealt with here. + auto isWrittenToLog = isReadDesc(fd); + if (isWrittenToLog) + { + logSize += data.size(); + if (settings.maxLogSize && logSize > settings.maxLogSize) { + killChild(); + // We're not inside a coroutine, hence we can't use co_return here. + // Thus we ignore the return value. + [[maybe_unused]] Done _ = done( + BuildResult::LogLimitExceeded, {}, + Error("%s killed after writing more than %d bytes of log output", + getName(), settings.maxLogSize)); + return; + } + + for (auto c : data) + if (c == '\r') + currentLogLinePos = 0; + else if (c == '\n') + flushLine(); + else { + if (currentLogLinePos >= currentLogLine.size()) + currentLogLine.resize(currentLogLinePos + 1); + currentLogLine[currentLogLinePos++] = c; + } + + if (logSink) (*logSink)(data); + } + +#ifndef _WIN32 // TODO enable build hook on Windows + if (hook && fd == hook->fromHook.readSide.get()) { + for (auto c : data) + if (c == '\n') { + auto json = parseJSONMessage(currentHookLine, "the derivation builder"); + if (json) { + auto s = handleJSONLogMessage(*json, worker.act, hook->activities, "the derivation builder", true); + // ensure that logs from a builder using `ssh-ng://` as protocol + // are also available to `nix log`. + if (s && !isWrittenToLog && logSink) { + const auto type = (*json)["type"]; + const auto fields = (*json)["fields"]; + if (type == resBuildLogLine) { + (*logSink)((fields.size() > 0 ? fields[0].get() : "") + "\n"); + } else if (type == resSetPhase && ! fields.is_null()) { + const auto phase = fields[0]; + if (! phase.is_null()) { + // nixpkgs' stdenv produces lines in the log to signal + // phase changes. + // We want to get the same lines in case of remote builds. + // The format is: + // @nix { "action": "setPhase", "phase": "$curPhase" } + const auto logLine = nlohmann::json::object({ + {"action", "setPhase"}, + {"phase", phase} + }); + (*logSink)("@nix " + logLine.dump(-1, ' ', false, nlohmann::json::error_handler_t::replace) + "\n"); + } + } + } + } + currentHookLine.clear(); + } else + currentHookLine += c; + } +#endif +} + + +void DerivationGoal::handleEOF(Descriptor fd) +{ + if (!currentLogLine.empty()) flushLine(); + worker.wakeUp(shared_from_this()); +} + + +void DerivationGoal::flushLine() +{ + if (handleJSONLogMessage(currentLogLine, *act, builderActivities, "the derivation builder", false)) + ; + + else { + logTail.push_back(currentLogLine); + if (logTail.size() > settings.logLines) logTail.pop_front(); + + act->result(resBuildLogLine, currentLogLine); + } + + currentLogLine = ""; + currentLogLinePos = 0; +} + + +std::map> DerivationGoal::queryPartialDerivationOutputMap() +{ + assert(!drv->type().isImpure()); + + for (auto * drvStore : { &worker.evalStore, &worker.store }) + if (drvStore->isValidPath(drvPath)) + return worker.store.queryPartialDerivationOutputMap(drvPath, drvStore); + + /* In-memory derivation will naturally fall back on this case, where + we do best-effort with static information. */ + std::map> res; + for (auto & [name, output] : drv->outputs) + res.insert_or_assign(name, output.path(worker.store, drv->name, name)); + return res; +} + +OutputPathMap DerivationGoal::queryDerivationOutputMap() +{ + assert(!drv->type().isImpure()); + + for (auto * drvStore : { &worker.evalStore, &worker.store }) + if (drvStore->isValidPath(drvPath)) + return worker.store.queryDerivationOutputMap(drvPath, drvStore); + + // See comment in `DerivationGoal::queryPartialDerivationOutputMap`. + OutputPathMap res; + for (auto & [name, output] : drv->outputsAndOptPaths(worker.store)) + res.insert_or_assign(name, *output.second); + return res; +} + + +std::pair DerivationGoal::checkPathValidity() +{ + if (drv->type().isImpure()) return { false, {} }; + + bool checkHash = buildMode == bmRepair; + auto wantedOutputsLeft = std::visit(overloaded { + [&](const OutputsSpec::All &) { + return StringSet {}; + }, + [&](const OutputsSpec::Names & names) { + return static_cast(names); + }, + }, wantedOutputs.raw); + SingleDrvOutputs validOutputs; + + for (auto & i : queryPartialDerivationOutputMap()) { + auto initialOutput = get(initialOutputs, i.first); + if (!initialOutput) + // this is an invalid output, gets catched with (!wantedOutputsLeft.empty()) + continue; + auto & info = *initialOutput; + info.wanted = wantedOutputs.contains(i.first); + if (info.wanted) + wantedOutputsLeft.erase(i.first); + if (i.second) { + auto outputPath = *i.second; + info.known = { + .path = outputPath, + .status = !worker.store.isValidPath(outputPath) + ? PathStatus::Absent + : !checkHash || worker.pathContentsGood(outputPath) + ? PathStatus::Valid + : PathStatus::Corrupt, + }; + } + auto drvOutput = DrvOutput{info.outputHash, i.first}; + if (experimentalFeatureSettings.isEnabled(Xp::CaDerivations)) { + if (auto real = worker.store.queryRealisation(drvOutput)) { + info.known = { + .path = real->outPath, + .status = PathStatus::Valid, + }; + } else if (info.known && info.known->isValid()) { + // We know the output because it's a static output of the + // derivation, and the output path is valid, but we don't have + // its realisation stored (probably because it has been built + // without the `ca-derivations` experimental flag). + worker.store.registerDrvOutput( + Realisation { + drvOutput, + info.known->path, + } + ); + } + } + if (info.known && info.known->isValid()) + validOutputs.emplace(i.first, Realisation { drvOutput, info.known->path }); + } + + // If we requested all the outputs, we are always fine. + // If we requested specific elements, the loop above removes all the valid + // ones, so any that are left must be invalid. + if (!wantedOutputsLeft.empty()) + throw Error("derivation '%s' does not have wanted outputs %s", + worker.store.printStorePath(drvPath), + concatStringsSep(", ", quoteStrings(wantedOutputsLeft))); + + bool allValid = true; + for (auto & [_, status] : initialOutputs) { + if (!status.wanted) continue; + if (!status.known || !status.known->isValid()) { + allValid = false; + break; + } + } + + return { allValid, validOutputs }; +} + + +SingleDrvOutputs DerivationGoal::assertPathValidity() +{ + auto [allValid, validOutputs] = checkPathValidity(); + if (!allValid) + throw Error("some outputs are unexpectedly invalid"); + return validOutputs; +} + + +Goal::Done DerivationGoal::done( + BuildResult::Status status, + SingleDrvOutputs builtOutputs, + std::optional ex) +{ + outputLocks.unlock(); + buildResult.status = status; + if (ex) + buildResult.errorMsg = fmt("%s", Uncolored(ex->info().msg)); + if (buildResult.status == BuildResult::TimedOut) + worker.timedOut = true; + if (buildResult.status == BuildResult::PermanentFailure) + worker.permanentFailure = true; + + mcExpectedBuilds.reset(); + mcRunningBuilds.reset(); + + if (buildResult.success()) { + auto wantedBuiltOutputs = filterDrvOutputs(wantedOutputs, std::move(builtOutputs)); + assert(!wantedBuiltOutputs.empty()); + buildResult.builtOutputs = std::move(wantedBuiltOutputs); + if (status == BuildResult::Built) + worker.doneBuilds++; + } else { + if (status != BuildResult::DependencyFailed) + worker.failedBuilds++; + } + + worker.updateProgress(); + + auto traceBuiltOutputsFile = getEnv("_NIX_TRACE_BUILT_OUTPUTS").value_or(""); + if (traceBuiltOutputsFile != "") { + std::fstream fs; + fs.open(traceBuiltOutputsFile, std::fstream::out); + fs << worker.store.printStorePath(drvPath) << "\t" << buildResult.toString() << std::endl; + } + + return amDone(buildResult.success() ? ecSuccess : ecFailed, std::move(ex)); +} + +} diff --git a/src/libstore/include/nix/store/build/derivation-building-goal.hh b/src/libstore/include/nix/store/build/derivation-building-goal.hh new file mode 100644 index 00000000000..dc65b494135 --- /dev/null +++ b/src/libstore/include/nix/store/build/derivation-building-goal.hh @@ -0,0 +1,239 @@ +#pragma once +///@file + +#include "nix/store/parsed-derivations.hh" +#include "nix/store/derivations.hh" +#include "nix/store/derivation-options.hh" +#include "nix/store/build/derivation-building-misc.hh" +#include "nix/store/outputs-spec.hh" +#include "nix/store/store-api.hh" +#include "nix/store/pathlocks.hh" +#include "nix/store/build/goal.hh" + +namespace nix { + +using std::map; + +#ifndef _WIN32 // TODO enable build hook on Windows +struct HookInstance; +struct DerivationBuilder; +#endif + +typedef enum {rpAccept, rpDecline, rpPostpone} HookReply; + +/** Used internally */ +void runPostBuildHook( + Store & store, + Logger & logger, + const StorePath & drvPath, + const StorePathSet & outputPaths); + +/** + * A goal for building some or all of the outputs of a derivation. + */ +struct DerivationGoal : public Goal +{ + /** The path of the derivation. */ + StorePath drvPath; + + /** + * The specific outputs that we need to build. + */ + OutputsSpec wantedOutputs; + + /** + * See `needRestart`; just for that field. + */ + enum struct NeedRestartForMoreOutputs { + /** + * The goal state machine is progressing based on the current value of + * `wantedOutputs. No actions are needed. + */ + OutputsUnmodifedDontNeed, + /** + * `wantedOutputs` has been extended, but the state machine is + * proceeding according to its old value, so we need to restart. + */ + OutputsAddedDoNeed, + /** + * The goal state machine has progressed to the point of doing a build, + * in which case all outputs will be produced, so extensions to + * `wantedOutputs` no longer require a restart. + */ + BuildInProgressWillNotNeed, + }; + + /** + * Whether additional wanted outputs have been added. + */ + NeedRestartForMoreOutputs needRestart = NeedRestartForMoreOutputs::OutputsUnmodifedDontNeed; + + /** + * The derivation stored at drvPath. + */ + std::unique_ptr drv; + + std::unique_ptr parsedDrv; + std::unique_ptr drvOptions; + + /** + * The remainder is state held during the build. + */ + + /** + * Locks on (fixed) output paths. + */ + PathLocks outputLocks; + + /** + * All input paths (that is, the union of FS closures of the + * immediate input paths). + */ + StorePathSet inputPaths; + + std::map initialOutputs; + + /** + * File descriptor for the log file. + */ + AutoCloseFD fdLogFile; + std::shared_ptr logFileSink, logSink; + + /** + * Number of bytes received from the builder's stdout/stderr. + */ + unsigned long logSize; + + /** + * The most recent log lines. + */ + std::list logTail; + + std::string currentLogLine; + size_t currentLogLinePos = 0; // to handle carriage return + + std::string currentHookLine; + +#ifndef _WIN32 // TODO enable build hook on Windows + /** + * The build hook. + */ + std::unique_ptr hook; + + std::unique_ptr builder; +#endif + + BuildMode buildMode; + + std::unique_ptr> mcExpectedBuilds, mcRunningBuilds; + + std::unique_ptr act; + + /** + * Activity that denotes waiting for a lock. + */ + std::unique_ptr actLock; + + std::map builderActivities; + + /** + * The remote machine on which we're building. + */ + std::string machineName; + + DerivationGoal(const StorePath & drvPath, + const OutputsSpec & wantedOutputs, Worker & worker, + BuildMode buildMode = bmNormal); + DerivationGoal(const StorePath & drvPath, const BasicDerivation & drv, + const OutputsSpec & wantedOutputs, Worker & worker, + BuildMode buildMode = bmNormal); + ~DerivationGoal(); + + void timedOut(Error && ex) override; + + std::string key() override; + + /** + * Add wanted outputs to an already existing derivation goal. + */ + void addWantedOutputs(const OutputsSpec & outputs); + + /** + * The states. + */ + Co loadDerivation(); + Co haveDerivation(); + Co gaveUpOnSubstitution(); + Co tryToBuild(); + Co hookDone(); + + /** + * Is the build hook willing to perform the build? + */ + HookReply tryBuildHook(); + + /** + * Open a log file and a pipe to it. + */ + Path openLogFile(); + + /** + * Close the log file. + */ + void closeLogFile(); + + bool isReadDesc(Descriptor fd); + + /** + * Callback used by the worker to write to the log. + */ + void handleChildOutput(Descriptor fd, std::string_view data) override; + void handleEOF(Descriptor fd) override; + void flushLine(); + + /** + * Wrappers around the corresponding Store methods that first consult the + * derivation. This is currently needed because when there is no drv file + * there also is no DB entry. + */ + std::map> queryPartialDerivationOutputMap(); + OutputPathMap queryDerivationOutputMap(); + + /** + * Update 'initialOutputs' to determine the current status of the + * outputs of the derivation. Also returns a Boolean denoting + * whether all outputs are valid and non-corrupt, and a + * 'SingleDrvOutputs' structure containing the valid outputs. + */ + std::pair checkPathValidity(); + + /** + * Aborts if any output is not valid or corrupt, and otherwise + * returns a 'SingleDrvOutputs' structure containing all outputs. + */ + SingleDrvOutputs assertPathValidity(); + + /** + * Forcibly kill the child process, if any. + */ + void killChild(); + + Co repairClosure(); + + void started(); + + Done done( + BuildResult::Status status, + SingleDrvOutputs builtOutputs = {}, + std::optional ex = {}); + + void appendLogTailErrorMsg(std::string & msg); + + StorePathSet exportReferences(const StorePathSet & storePaths); + + JobCategory jobCategory() const override { + return JobCategory::Build; + }; +}; + +} From 3b617e471ba90067d220a130eb68068c9a148c00 Mon Sep 17 00:00:00 2001 From: John Ericson Date: Mon, 21 Apr 2025 18:05:03 -0400 Subject: [PATCH 042/218] Split `DerivationGoal` in two This separation of concerns is generally good, but in particular sets up for removing `addWantedOutputs` next. --- .../build/derivation-building-goal.cc | 412 +----- src/libstore/build/derivation-goal.cc | 1104 +---------------- src/libstore/build/goal.cc | 2 +- src/libstore/build/worker.cc | 26 +- .../store/build/derivation-building-goal.hh | 55 +- .../nix/store/build/derivation-goal.hh | 111 +- src/libstore/include/nix/store/build/goal.hh | 21 + .../include/nix/store/build/worker.hh | 9 + src/libstore/include/nix/store/meson.build | 1 + src/libstore/meson.build | 1 + 10 files changed, 155 insertions(+), 1587 deletions(-) diff --git a/src/libstore/build/derivation-building-goal.cc b/src/libstore/build/derivation-building-goal.cc index 82c350832ee..4169903f654 100644 --- a/src/libstore/build/derivation-building-goal.cc +++ b/src/libstore/build/derivation-building-goal.cc @@ -1,3 +1,4 @@ +#include "nix/store/build/derivation-building-goal.hh" #include "nix/store/build/derivation-goal.hh" #ifndef _WIN32 // TODO enable build hook on Windows # include "nix/store/build/hook-instance.hh" @@ -23,47 +24,35 @@ namespace nix { -DerivationGoal::DerivationGoal(const StorePath & drvPath, - const OutputsSpec & wantedOutputs, Worker & worker, BuildMode buildMode) - : Goal(worker, loadDerivation()) +DerivationBuildingGoal::DerivationBuildingGoal(const StorePath & drvPath, const Derivation & drv_, + Worker & worker, BuildMode buildMode) + : Goal(worker, gaveUpOnSubstitution()) , drvPath(drvPath) - , wantedOutputs(wantedOutputs) , buildMode(buildMode) { - name = fmt( - "building of '%s' from .drv file", - DerivedPath::Built { makeConstantStorePathRef(drvPath), wantedOutputs }.to_string(worker.store)); - trace("created"); - - mcExpectedBuilds = std::make_unique>(worker.expectedBuilds); - worker.updateProgress(); -} - + drv = std::make_unique(drv_); -DerivationGoal::DerivationGoal(const StorePath & drvPath, const BasicDerivation & drv, - const OutputsSpec & wantedOutputs, Worker & worker, BuildMode buildMode) - : Goal(worker, haveDerivation()) - , drvPath(drvPath) - , wantedOutputs(wantedOutputs) - , buildMode(buildMode) -{ - this->drv = std::make_unique(drv); + if (auto parsedOpt = StructuredAttrs::tryParse(drv->env)) { + parsedDrv = std::make_unique(*parsedOpt); + } + try { + drvOptions = std::make_unique( + DerivationOptions::fromStructuredAttrs(drv->env, parsedDrv.get())); + } catch (Error & e) { + e.addTrace({}, "while parsing derivation '%s'", worker.store.printStorePath(drvPath)); + throw; + } - name = fmt( - "building of '%s' from in-memory derivation", - DerivedPath::Built { makeConstantStorePathRef(drvPath), drv.outputNames() }.to_string(worker.store)); + name = fmt("building of '%s' from in-memory derivation", worker.store.printStorePath(drvPath)); trace("created"); - mcExpectedBuilds = std::make_unique>(worker.expectedBuilds); - worker.updateProgress(); - /* Prevent the .chroot directory from being garbage-collected. (See isActiveTempFile() in gc.cc.) */ worker.store.addTempRoot(this->drvPath); } -DerivationGoal::~DerivationGoal() +DerivationBuildingGoal::~DerivationBuildingGoal() { /* Careful: we should never ever throw an exception from a destructor. */ @@ -78,17 +67,17 @@ DerivationGoal::~DerivationGoal() } -std::string DerivationGoal::key() +std::string DerivationBuildingGoal::key() { /* Ensure that derivations get built in order of their name, i.e. a derivation named "aardvark" always comes before "baboon". And substitution goals always happen before derivation goals (due to "b$"). */ - return "b$" + std::string(drvPath.name()) + "$" + worker.store.printStorePath(drvPath); + return "bd$" + std::string(drvPath.name()) + "$" + worker.store.printStorePath(drvPath); } -void DerivationGoal::killChild() +void DerivationBuildingGoal::killChild() { #ifndef _WIN32 // TODO enable build hook on Windows hook.reset(); @@ -112,7 +101,7 @@ void DerivationGoal::killChild() } -void DerivationGoal::timedOut(Error && ex) +void DerivationBuildingGoal::timedOut(Error && ex) { killChild(); // We're not inside a coroutine, hence we can't use co_return here. @@ -120,198 +109,6 @@ void DerivationGoal::timedOut(Error && ex) [[maybe_unused]] Done _ = done(BuildResult::TimedOut, {}, std::move(ex)); } -void DerivationGoal::addWantedOutputs(const OutputsSpec & outputs) -{ - auto newWanted = wantedOutputs.union_(outputs); - switch (needRestart) { - case NeedRestartForMoreOutputs::OutputsUnmodifedDontNeed: - if (!newWanted.isSubsetOf(wantedOutputs)) - needRestart = NeedRestartForMoreOutputs::OutputsAddedDoNeed; - break; - case NeedRestartForMoreOutputs::OutputsAddedDoNeed: - /* No need to check whether we added more outputs, because a - restart is already queued up. */ - break; - case NeedRestartForMoreOutputs::BuildInProgressWillNotNeed: - /* We are already building all outputs, so it doesn't matter if - we now want more. */ - break; - }; - wantedOutputs = newWanted; -} - - -Goal::Co DerivationGoal::loadDerivation() { - trace("local derivation"); - - { - /* The first thing to do is to make sure that the derivation - exists. If it doesn't, it may be created through a - substitute. */ - - if (buildMode != bmNormal || !worker.evalStore.isValidPath(drvPath)) { - Goals waitees{upcast_goal(worker.makePathSubstitutionGoal(drvPath))}; - co_await await(std::move(waitees)); - } - - trace("loading derivation"); - - if (nrFailed != 0) { - co_return done(BuildResult::MiscFailure, {}, Error("cannot build missing derivation '%s'", worker.store.printStorePath(drvPath))); - } - - /* `drvPath' should already be a root, but let's be on the safe - side: if the user forgot to make it a root, we wouldn't want - things being garbage collected while we're busy. */ - worker.evalStore.addTempRoot(drvPath); - - /* Get the derivation. It is probably in the eval store, but it might be inthe main store: - - - Resolved derivation are resolved against main store realisations, and so must be stored there. - - - Dynamic derivations are built, and so are found in the main store. - */ - for (auto * drvStore : { &worker.evalStore, &worker.store }) { - if (drvStore->isValidPath(drvPath)) { - drv = std::make_unique(drvStore->readDerivation(drvPath)); - break; - } - } - assert(drv); - } - - co_return haveDerivation(); -} - - -Goal::Co DerivationGoal::haveDerivation() -{ - trace("have derivation"); - - if (auto parsedOpt = StructuredAttrs::tryParse(drv->env)) { - parsedDrv = std::make_unique(*parsedOpt); - } - try { - drvOptions = std::make_unique( - DerivationOptions::fromStructuredAttrs(drv->env, parsedDrv.get())); - } catch (Error & e) { - e.addTrace({}, "while parsing derivation '%s'", worker.store.printStorePath(drvPath)); - throw; - } - - if (!drv->type().hasKnownOutputPaths()) - experimentalFeatureSettings.require(Xp::CaDerivations); - - for (auto & i : drv->outputsAndOptPaths(worker.store)) - if (i.second.second) - worker.store.addTempRoot(*i.second.second); - - { - bool impure = drv->type().isImpure(); - - if (impure) experimentalFeatureSettings.require(Xp::ImpureDerivations); - - auto outputHashes = staticOutputHashes(worker.evalStore, *drv); - for (auto & [outputName, outputHash] : outputHashes) { - InitialOutput v{ - .wanted = true, // Will be refined later - .outputHash = outputHash - }; - - /* TODO we might want to also allow randomizing the paths - for regular CA derivations, e.g. for sake of checking - determinism. */ - if (impure) { - v.known = InitialOutputStatus { - .path = StorePath::random(outputPathName(drv->name, outputName)), - .status = PathStatus::Absent, - }; - } - - initialOutputs.insert({ - outputName, - std::move(v), - }); - } - - if (impure) { - /* We don't yet have any safe way to cache an impure derivation at - this step. */ - co_return gaveUpOnSubstitution(); - } - } - - { - /* Check what outputs paths are not already valid. */ - auto [allValid, validOutputs] = checkPathValidity(); - - /* If they are all valid, then we're done. */ - if (allValid && buildMode == bmNormal) { - co_return done(BuildResult::AlreadyValid, std::move(validOutputs)); - } - } - - Goals waitees; - - /* We are first going to try to create the invalid output paths - through substitutes. If that doesn't work, we'll build - them. */ - if (settings.useSubstitutes && drvOptions->substitutesAllowed()) - for (auto & [outputName, status] : initialOutputs) { - if (!status.wanted) continue; - if (!status.known) - waitees.insert( - upcast_goal( - worker.makeDrvOutputSubstitutionGoal( - DrvOutput{status.outputHash, outputName}, - buildMode == bmRepair ? Repair : NoRepair - ) - ) - ); - else { - auto * cap = getDerivationCA(*drv); - waitees.insert(upcast_goal(worker.makePathSubstitutionGoal( - status.known->path, - buildMode == bmRepair ? Repair : NoRepair, - cap ? std::optional { *cap } : std::nullopt))); - } - } - - co_await await(std::move(waitees)); - - trace("all outputs substituted (maybe)"); - - assert(!drv->type().isImpure()); - - if (nrFailed > 0 && nrFailed > nrNoSubstituters && !settings.tryFallback) { - co_return done(BuildResult::TransientFailure, {}, - Error("some substitutes for the outputs of derivation '%s' failed (usually happens due to networking issues); try '--fallback' to build derivation from source ", - worker.store.printStorePath(drvPath))); - } - - nrFailed = nrNoSubstituters = 0; - - if (needRestart == NeedRestartForMoreOutputs::OutputsAddedDoNeed) { - needRestart = NeedRestartForMoreOutputs::OutputsUnmodifedDontNeed; - co_return haveDerivation(); - } - - auto [allValid, validOutputs] = checkPathValidity(); - - if (buildMode == bmNormal && allValid) { - co_return done(BuildResult::Substituted, std::move(validOutputs)); - } - if (buildMode == bmRepair && allValid) { - co_return repairClosure(); - } - if (buildMode == bmCheck && !allValid) - throw Error("some outputs of '%s' are not valid, so checking is not possible", - worker.store.printStorePath(drvPath)); - - /* Nothing to wait for; tail call */ - co_return gaveUpOnSubstitution(); -} - /** * Used for `inputGoals` local variable below @@ -343,12 +140,8 @@ std::string showKnownOutputs(Store & store, const Derivation & drv) /* At least one of the output paths could not be produced using a substitute. So we have to build instead. */ -Goal::Co DerivationGoal::gaveUpOnSubstitution() +Goal::Co DerivationBuildingGoal::gaveUpOnSubstitution() { - /* At this point we are building all outputs, so if more are wanted there - is no need to restart. */ - needRestart = NeedRestartForMoreOutputs::BuildInProgressWillNotNeed; - Goals waitees; std::map, GoalPtr, value_comparison> inputGoals; @@ -501,8 +294,9 @@ Goal::Co DerivationGoal::gaveUpOnSubstitution() worker.store.printStorePath(pathResolved), }); + // FIXME wanted outputs auto resolvedDrvGoal = worker.makeDerivationGoal( - pathResolved, wantedOutputs, buildMode); + pathResolved, OutputsSpec::All{}, buildMode); { Goals waitees{resolvedDrvGoal}; co_await await(std::move(waitees)); @@ -511,7 +305,10 @@ Goal::Co DerivationGoal::gaveUpOnSubstitution() trace("resolved derivation finished"); auto resolvedDrv = *resolvedDrvGoal->drv; - auto & resolvedResult = resolvedDrvGoal->buildResult; + auto resolvedResult = resolvedDrvGoal->getBuildResult(DerivedPath::Built{ + .drvPath = makeConstantStorePathRef(pathResolved), + .outputs = OutputsSpec::All{}, + }); SingleDrvOutputs builtOutputs; @@ -615,7 +412,7 @@ Goal::Co DerivationGoal::gaveUpOnSubstitution() co_return tryToBuild(); } -void DerivationGoal::started() +void DerivationBuildingGoal::started() { auto msg = fmt( buildMode == bmRepair ? "repairing outputs of '%s'" : @@ -637,7 +434,7 @@ void DerivationGoal::started() worker.updateProgress(); } -Goal::Co DerivationGoal::tryToBuild() +Goal::Co DerivationBuildingGoal::tryToBuild() { trace("trying to build"); @@ -773,15 +570,15 @@ Goal::Co DerivationGoal::tryToBuild() * Local implementation of these virtual methods, consider * this just a record of lambdas. */ - struct DerivationGoalCallbacks : DerivationBuilderCallbacks + struct DerivationBuildingGoalCallbacks : DerivationBuilderCallbacks { - DerivationGoal & goal; + DerivationBuildingGoal & goal; - DerivationGoalCallbacks(DerivationGoal & goal, std::unique_ptr & builder) + DerivationBuildingGoalCallbacks(DerivationBuildingGoal & goal, std::unique_ptr & builder) : goal{goal} {} - ~DerivationGoalCallbacks() override = default; + ~DerivationBuildingGoalCallbacks() override = default; void childStarted(Descriptor builderOut) override { @@ -826,7 +623,7 @@ Goal::Co DerivationGoal::tryToBuild() already be created, so we don't need to create it again. */ builder = makeDerivationBuilder( worker.store, - std::make_unique(*this, builder), + std::make_unique(*this, builder), DerivationBuilderParams { drvPath, buildMode, @@ -889,78 +686,6 @@ Goal::Co DerivationGoal::tryToBuild() } -Goal::Co DerivationGoal::repairClosure() -{ - assert(!drv->type().isImpure()); - - /* If we're repairing, we now know that our own outputs are valid. - Now check whether the other paths in the outputs closure are - good. If not, then start derivation goals for the derivations - that produced those outputs. */ - - /* Get the output closure. */ - auto outputs = queryDerivationOutputMap(); - StorePathSet outputClosure; - for (auto & i : outputs) { - if (!wantedOutputs.contains(i.first)) continue; - worker.store.computeFSClosure(i.second, outputClosure); - } - - /* Filter out our own outputs (which we have already checked). */ - for (auto & i : outputs) - outputClosure.erase(i.second); - - /* Get all dependencies of this derivation so that we know which - derivation is responsible for which path in the output - closure. */ - StorePathSet inputClosure; - - /* If we're working from an in-memory derivation with no in-store - `*.drv` file, we cannot do this part. */ - if (worker.store.isValidPath(drvPath)) - worker.store.computeFSClosure(drvPath, inputClosure); - - std::map outputsToDrv; - for (auto & i : inputClosure) - if (i.isDerivation()) { - auto depOutputs = worker.store.queryPartialDerivationOutputMap(i, &worker.evalStore); - for (auto & j : depOutputs) - if (j.second) - outputsToDrv.insert_or_assign(*j.second, i); - } - - Goals waitees; - - /* Check each path (slow!). */ - for (auto & i : outputClosure) { - if (worker.pathContentsGood(i)) continue; - printError( - "found corrupted or missing path '%s' in the output closure of '%s'", - worker.store.printStorePath(i), worker.store.printStorePath(drvPath)); - auto drvPath2 = outputsToDrv.find(i); - if (drvPath2 == outputsToDrv.end()) - waitees.insert(upcast_goal(worker.makePathSubstitutionGoal(i, Repair))); - else - waitees.insert(worker.makeGoal( - DerivedPath::Built { - .drvPath = makeConstantStorePathRef(drvPath2->second), - .outputs = OutputsSpec::All { }, - }, - bmRepair)); - } - - co_await await(std::move(waitees)); - - if (!waitees.empty()) { - trace("closure repaired"); - if (nrFailed > 0) - throw Error("some paths in the output closure of derivation '%s' could not be repaired", - worker.store.printStorePath(drvPath)); - } - co_return done(BuildResult::AlreadyValid, assertPathValidity()); -} - - void runPostBuildHook( Store & store, Logger & logger, @@ -1020,7 +745,7 @@ void runPostBuildHook( } -void DerivationGoal::appendLogTailErrorMsg(std::string & msg) +void DerivationBuildingGoal::appendLogTailErrorMsg(std::string & msg) { if (!logger->isVerbose() && !logTail.empty()) { msg += fmt("\nLast %d log lines:\n", logTail.size()); @@ -1042,7 +767,7 @@ void DerivationGoal::appendLogTailErrorMsg(std::string & msg) } -Goal::Co DerivationGoal::hookDone() +Goal::Co DerivationBuildingGoal::hookDone() { #ifndef _WIN32 assert(hook); @@ -1129,7 +854,7 @@ Goal::Co DerivationGoal::hookDone() co_return done(BuildResult::Built, std::move(builtOutputs)); } -HookReply DerivationGoal::tryBuildHook() +HookReply DerivationBuildingGoal::tryBuildHook() { #ifdef _WIN32 // TODO enable build hook on Windows return rpDecline; @@ -1244,7 +969,7 @@ HookReply DerivationGoal::tryBuildHook() } -Path DerivationGoal::openLogFile() +Path DerivationBuildingGoal::openLogFile() { logSize = 0; @@ -1282,7 +1007,7 @@ Path DerivationGoal::openLogFile() } -void DerivationGoal::closeLogFile() +void DerivationBuildingGoal::closeLogFile() { auto logSink2 = std::dynamic_pointer_cast(logSink); if (logSink2) logSink2->finish(); @@ -1292,7 +1017,7 @@ void DerivationGoal::closeLogFile() } -bool DerivationGoal::isReadDesc(Descriptor fd) +bool DerivationBuildingGoal::isReadDesc(Descriptor fd) { #ifdef _WIN32 // TODO enable build hook on Windows return false; @@ -1304,7 +1029,7 @@ bool DerivationGoal::isReadDesc(Descriptor fd) #endif } -void DerivationGoal::handleChildOutput(Descriptor fd, std::string_view data) +void DerivationBuildingGoal::handleChildOutput(Descriptor fd, std::string_view data) { // local & `ssh://`-builds are dealt with here. auto isWrittenToLog = isReadDesc(fd); @@ -1375,14 +1100,14 @@ void DerivationGoal::handleChildOutput(Descriptor fd, std::string_view data) } -void DerivationGoal::handleEOF(Descriptor fd) +void DerivationBuildingGoal::handleEOF(Descriptor fd) { if (!currentLogLine.empty()) flushLine(); worker.wakeUp(shared_from_this()); } -void DerivationGoal::flushLine() +void DerivationBuildingGoal::flushLine() { if (handleJSONLogMessage(currentLogLine, *act, builderActivities, "the derivation builder", false)) ; @@ -1399,7 +1124,7 @@ void DerivationGoal::flushLine() } -std::map> DerivationGoal::queryPartialDerivationOutputMap() +std::map> DerivationBuildingGoal::queryPartialDerivationOutputMap() { assert(!drv->type().isImpure()); @@ -1415,35 +1140,11 @@ std::map> DerivationGoal::queryPartialDeri return res; } -OutputPathMap DerivationGoal::queryDerivationOutputMap() -{ - assert(!drv->type().isImpure()); - - for (auto * drvStore : { &worker.evalStore, &worker.store }) - if (drvStore->isValidPath(drvPath)) - return worker.store.queryDerivationOutputMap(drvPath, drvStore); - - // See comment in `DerivationGoal::queryPartialDerivationOutputMap`. - OutputPathMap res; - for (auto & [name, output] : drv->outputsAndOptPaths(worker.store)) - res.insert_or_assign(name, *output.second); - return res; -} - - -std::pair DerivationGoal::checkPathValidity() +std::pair DerivationBuildingGoal::checkPathValidity() { if (drv->type().isImpure()) return { false, {} }; bool checkHash = buildMode == bmRepair; - auto wantedOutputsLeft = std::visit(overloaded { - [&](const OutputsSpec::All &) { - return StringSet {}; - }, - [&](const OutputsSpec::Names & names) { - return static_cast(names); - }, - }, wantedOutputs.raw); SingleDrvOutputs validOutputs; for (auto & i : queryPartialDerivationOutputMap()) { @@ -1452,9 +1153,7 @@ std::pair DerivationGoal::checkPathValidity() // this is an invalid output, gets catched with (!wantedOutputsLeft.empty()) continue; auto & info = *initialOutput; - info.wanted = wantedOutputs.contains(i.first); - if (info.wanted) - wantedOutputsLeft.erase(i.first); + info.wanted = true; if (i.second) { auto outputPath = *i.second; info.known = { @@ -1490,14 +1189,6 @@ std::pair DerivationGoal::checkPathValidity() validOutputs.emplace(i.first, Realisation { drvOutput, info.known->path }); } - // If we requested all the outputs, we are always fine. - // If we requested specific elements, the loop above removes all the valid - // ones, so any that are left must be invalid. - if (!wantedOutputsLeft.empty()) - throw Error("derivation '%s' does not have wanted outputs %s", - worker.store.printStorePath(drvPath), - concatStringsSep(", ", quoteStrings(wantedOutputsLeft))); - bool allValid = true; for (auto & [_, status] : initialOutputs) { if (!status.wanted) continue; @@ -1511,7 +1202,7 @@ std::pair DerivationGoal::checkPathValidity() } -SingleDrvOutputs DerivationGoal::assertPathValidity() +SingleDrvOutputs DerivationBuildingGoal::assertPathValidity() { auto [allValid, validOutputs] = checkPathValidity(); if (!allValid) @@ -1520,7 +1211,7 @@ SingleDrvOutputs DerivationGoal::assertPathValidity() } -Goal::Done DerivationGoal::done( +Goal::Done DerivationBuildingGoal::done( BuildResult::Status status, SingleDrvOutputs builtOutputs, std::optional ex) @@ -1534,13 +1225,10 @@ Goal::Done DerivationGoal::done( if (buildResult.status == BuildResult::PermanentFailure) worker.permanentFailure = true; - mcExpectedBuilds.reset(); mcRunningBuilds.reset(); if (buildResult.success()) { - auto wantedBuiltOutputs = filterDrvOutputs(wantedOutputs, std::move(builtOutputs)); - assert(!wantedBuiltOutputs.empty()); - buildResult.builtOutputs = std::move(wantedBuiltOutputs); + buildResult.builtOutputs = std::move(builtOutputs); if (status == BuildResult::Built) worker.doneBuilds++; } else { diff --git a/src/libstore/build/derivation-goal.cc b/src/libstore/build/derivation-goal.cc index 82c350832ee..5c8b75cb9de 100644 --- a/src/libstore/build/derivation-goal.cc +++ b/src/libstore/build/derivation-goal.cc @@ -1,4 +1,5 @@ #include "nix/store/build/derivation-goal.hh" +#include "nix/store/build/derivation-building-goal.hh" #ifndef _WIN32 // TODO enable build hook on Windows # include "nix/store/build/hook-instance.hh" # include "nix/store/build/derivation-builder.hh" @@ -57,24 +58,6 @@ DerivationGoal::DerivationGoal(const StorePath & drvPath, const BasicDerivation mcExpectedBuilds = std::make_unique>(worker.expectedBuilds); worker.updateProgress(); - /* Prevent the .chroot directory from being - garbage-collected. (See isActiveTempFile() in gc.cc.) */ - worker.store.addTempRoot(this->drvPath); -} - - -DerivationGoal::~DerivationGoal() -{ - /* Careful: we should never ever throw an exception from a - destructor. */ - try { killChild(); } catch (...) { ignoreExceptionInDestructor(); } -#ifndef _WIN32 // TODO enable `DerivationBuilder` on Windows - if (builder) { - try { builder->stopDaemon(); } catch (...) { ignoreExceptionInDestructor(); } - try { builder->deleteTmpDir(false); } catch (...) { ignoreExceptionInDestructor(); } - } -#endif - try { closeLogFile(); } catch (...) { ignoreExceptionInDestructor(); } } @@ -88,38 +71,6 @@ std::string DerivationGoal::key() } -void DerivationGoal::killChild() -{ -#ifndef _WIN32 // TODO enable build hook on Windows - hook.reset(); -#endif -#ifndef _WIN32 // TODO enable `DerivationBuilder` on Windows - if (builder && builder->pid != -1) { - worker.childTerminated(this); - - /* If we're using a build user, then there is a tricky race - condition: if we kill the build user before the child has - done its setuid() to the build user uid, then it won't be - killed, and we'll potentially lock up in pid.wait(). So - also send a conventional kill to the child. */ - ::kill(-builder->pid, SIGKILL); /* ignore the result */ - - builder->killSandbox(true); - - builder->pid.wait(); - } -#endif -} - - -void DerivationGoal::timedOut(Error && ex) -{ - killChild(); - // We're not inside a coroutine, hence we can't use co_return here. - // Thus we ignore the return value. - [[maybe_unused]] Done _ = done(BuildResult::TimedOut, {}, std::move(ex)); -} - void DerivationGoal::addWantedOutputs(const OutputsSpec & outputs) { auto newWanted = wantedOutputs.union_(outputs); @@ -188,20 +139,47 @@ Goal::Co DerivationGoal::haveDerivation() { trace("have derivation"); - if (auto parsedOpt = StructuredAttrs::tryParse(drv->env)) { - parsedDrv = std::make_unique(*parsedOpt); - } - try { - drvOptions = std::make_unique( - DerivationOptions::fromStructuredAttrs(drv->env, parsedDrv.get())); - } catch (Error & e) { - e.addTrace({}, "while parsing derivation '%s'", worker.store.printStorePath(drvPath)); - throw; - } + auto drvOptions = [&]() -> DerivationOptions { + auto parsedOpt = StructuredAttrs::tryParse(drv->env); + try { + return DerivationOptions::fromStructuredAttrs(drv->env, parsedOpt ? &*parsedOpt : nullptr); + } catch (Error & e) { + e.addTrace({}, "while parsing derivation '%s'", worker.store.printStorePath(drvPath)); + throw; + } + }(); if (!drv->type().hasKnownOutputPaths()) experimentalFeatureSettings.require(Xp::CaDerivations); + /* At least one of the output paths could not be + produced using a substitute. So we have to build instead. */ + auto gaveUpOnSubstitution = [&]() -> Goal::Co + { + auto g = worker.makeDerivationBuildingGoal(drvPath, *drv, buildMode); + + /* We will finish with it ourselves, as if we were the derivational goal. */ + g->preserveException = true; + + // TODO move into constructor + g->initialOutputs = initialOutputs; + + { + Goals waitees; + waitees.insert(g); + co_await await(std::move(waitees)); + } + + trace("outer build done"); + + buildResult = g->getBuildResult(DerivedPath::Built{ + .drvPath = makeConstantStorePathRef(drvPath), + .outputs = wantedOutputs, + }); + + co_return amDone(g->exitCode, g->ex); + }; + for (auto & i : drv->outputsAndOptPaths(worker.store)) if (i.second.second) worker.store.addTempRoot(*i.second.second); @@ -256,7 +234,7 @@ Goal::Co DerivationGoal::haveDerivation() /* We are first going to try to create the invalid output paths through substitutes. If that doesn't work, we'll build them. */ - if (settings.useSubstitutes && drvOptions->substitutesAllowed()) + if (settings.useSubstitutes && drvOptions.substitutesAllowed()) for (auto & [outputName, status] : initialOutputs) { if (!status.wanted) continue; if (!status.known) @@ -325,570 +303,6 @@ struct value_comparison }; -std::string showKnownOutputs(Store & store, const Derivation & drv) -{ - std::string msg; - StorePathSet expectedOutputPaths; - for (auto & i : drv.outputsAndOptPaths(store)) - if (i.second.second) - expectedOutputPaths.insert(*i.second.second); - if (!expectedOutputPaths.empty()) { - msg += "\nOutput paths:"; - for (auto & p : expectedOutputPaths) - msg += fmt("\n %s", Magenta(store.printStorePath(p))); - } - return msg; -} - - -/* At least one of the output paths could not be - produced using a substitute. So we have to build instead. */ -Goal::Co DerivationGoal::gaveUpOnSubstitution() -{ - /* At this point we are building all outputs, so if more are wanted there - is no need to restart. */ - needRestart = NeedRestartForMoreOutputs::BuildInProgressWillNotNeed; - - Goals waitees; - - std::map, GoalPtr, value_comparison> inputGoals; - - { - std::function, const DerivedPathMap::ChildNode &)> addWaiteeDerivedPath; - - addWaiteeDerivedPath = [&](ref inputDrv, const DerivedPathMap::ChildNode & inputNode) { - if (!inputNode.value.empty()) { - auto g = worker.makeGoal( - DerivedPath::Built { - .drvPath = inputDrv, - .outputs = inputNode.value, - }, - buildMode == bmRepair ? bmRepair : bmNormal); - inputGoals.insert_or_assign(inputDrv, g); - waitees.insert(std::move(g)); - } - for (const auto & [outputName, childNode] : inputNode.childMap) - addWaiteeDerivedPath( - make_ref(SingleDerivedPath::Built { inputDrv, outputName }), - childNode); - }; - - for (const auto & [inputDrvPath, inputNode] : drv->inputDrvs.map) { - /* Ensure that pure, non-fixed-output derivations don't - depend on impure derivations. */ - if (experimentalFeatureSettings.isEnabled(Xp::ImpureDerivations) && !drv->type().isImpure() && !drv->type().isFixed()) { - auto inputDrv = worker.evalStore.readDerivation(inputDrvPath); - if (inputDrv.type().isImpure()) - throw Error("pure derivation '%s' depends on impure derivation '%s'", - worker.store.printStorePath(drvPath), - worker.store.printStorePath(inputDrvPath)); - } - - addWaiteeDerivedPath(makeConstantStorePathRef(inputDrvPath), inputNode); - } - } - - /* Copy the input sources from the eval store to the build - store. - - Note that some inputs might not be in the eval store because they - are (resolved) derivation outputs in a resolved derivation. */ - if (&worker.evalStore != &worker.store) { - RealisedPath::Set inputSrcs; - for (auto & i : drv->inputSrcs) - if (worker.evalStore.isValidPath(i)) - inputSrcs.insert(i); - copyClosure(worker.evalStore, worker.store, inputSrcs); - } - - for (auto & i : drv->inputSrcs) { - if (worker.store.isValidPath(i)) continue; - if (!settings.useSubstitutes) - throw Error("dependency '%s' of '%s' does not exist, and substitution is disabled", - worker.store.printStorePath(i), worker.store.printStorePath(drvPath)); - waitees.insert(upcast_goal(worker.makePathSubstitutionGoal(i))); - } - - co_await await(std::move(waitees)); - - - trace("all inputs realised"); - - if (nrFailed != 0) { - auto msg = fmt( - "Cannot build '%s'.\n" - "Reason: " ANSI_RED "%d %s failed" ANSI_NORMAL ".", - Magenta(worker.store.printStorePath(drvPath)), - nrFailed, - nrFailed == 1 ? "dependency" : "dependencies"); - msg += showKnownOutputs(worker.store, *drv); - co_return done(BuildResult::DependencyFailed, {}, Error(msg)); - } - - /* Gather information necessary for computing the closure and/or - running the build hook. */ - - /* Determine the full set of input paths. */ - - /* First, the input derivations. */ - { - auto & fullDrv = *drv; - - auto drvType = fullDrv.type(); - bool resolveDrv = std::visit(overloaded { - [&](const DerivationType::InputAddressed & ia) { - /* must resolve if deferred. */ - return ia.deferred; - }, - [&](const DerivationType::ContentAddressed & ca) { - return !fullDrv.inputDrvs.map.empty() && ( - ca.fixed - /* Can optionally resolve if fixed, which is good - for avoiding unnecessary rebuilds. */ - ? experimentalFeatureSettings.isEnabled(Xp::CaDerivations) - /* Must resolve if floating and there are any inputs - drvs. */ - : true); - }, - [&](const DerivationType::Impure &) { - return true; - } - }, drvType.raw) - /* no inputs are outputs of dynamic derivations */ - || std::ranges::any_of( - fullDrv.inputDrvs.map.begin(), - fullDrv.inputDrvs.map.end(), - [](auto & pair) { return !pair.second.childMap.empty(); }); - - if (resolveDrv && !fullDrv.inputDrvs.map.empty()) { - experimentalFeatureSettings.require(Xp::CaDerivations); - - /* We are be able to resolve this derivation based on the - now-known results of dependencies. If so, we become a - stub goal aliasing that resolved derivation goal. */ - std::optional attempt = fullDrv.tryResolve(worker.store, - [&](ref drvPath, const std::string & outputName) -> std::optional { - auto mEntry = get(inputGoals, drvPath); - if (!mEntry) return std::nullopt; - - auto buildResult = (*mEntry)->getBuildResult(DerivedPath::Built{drvPath, OutputsSpec::Names{outputName}}); - if (!buildResult.success()) return std::nullopt; - - auto i = get(buildResult.builtOutputs, outputName); - if (!i) return std::nullopt; - - return i->outPath; - }); - if (!attempt) { - /* TODO (impure derivations-induced tech debt) (see below): - The above attempt should have found it, but because we manage - inputDrvOutputs statefully, sometimes it gets out of sync with - the real source of truth (store). So we query the store - directly if there's a problem. */ - attempt = fullDrv.tryResolve(worker.store, &worker.evalStore); - } - assert(attempt); - Derivation drvResolved { std::move(*attempt) }; - - auto pathResolved = writeDerivation(worker.store, drvResolved); - - auto msg = fmt("resolved derivation: '%s' -> '%s'", - worker.store.printStorePath(drvPath), - worker.store.printStorePath(pathResolved)); - act = std::make_unique(*logger, lvlInfo, actBuildWaiting, msg, - Logger::Fields { - worker.store.printStorePath(drvPath), - worker.store.printStorePath(pathResolved), - }); - - auto resolvedDrvGoal = worker.makeDerivationGoal( - pathResolved, wantedOutputs, buildMode); - { - Goals waitees{resolvedDrvGoal}; - co_await await(std::move(waitees)); - } - - trace("resolved derivation finished"); - - auto resolvedDrv = *resolvedDrvGoal->drv; - auto & resolvedResult = resolvedDrvGoal->buildResult; - - SingleDrvOutputs builtOutputs; - - if (resolvedResult.success()) { - auto resolvedHashes = staticOutputHashes(worker.store, resolvedDrv); - - StorePathSet outputPaths; - - for (auto & outputName : resolvedDrv.outputNames()) { - auto initialOutput = get(initialOutputs, outputName); - auto resolvedHash = get(resolvedHashes, outputName); - if ((!initialOutput) || (!resolvedHash)) - throw Error( - "derivation '%s' doesn't have expected output '%s' (derivation-goal.cc/resolve)", - worker.store.printStorePath(drvPath), outputName); - - auto realisation = [&]{ - auto take1 = get(resolvedResult.builtOutputs, outputName); - if (take1) return *take1; - - /* The above `get` should work. But sateful tracking of - outputs in resolvedResult, this can get out of sync with the - store, which is our actual source of truth. For now we just - check the store directly if it fails. */ - auto take2 = worker.evalStore.queryRealisation(DrvOutput { *resolvedHash, outputName }); - if (take2) return *take2; - - throw Error( - "derivation '%s' doesn't have expected output '%s' (derivation-goal.cc/realisation)", - worker.store.printStorePath(resolvedDrvGoal->drvPath), outputName); - }(); - - if (!drv->type().isImpure()) { - auto newRealisation = realisation; - newRealisation.id = DrvOutput { initialOutput->outputHash, outputName }; - newRealisation.signatures.clear(); - if (!drv->type().isFixed()) { - auto & drvStore = worker.evalStore.isValidPath(drvPath) - ? worker.evalStore - : worker.store; - newRealisation.dependentRealisations = drvOutputReferences(worker.store, *drv, realisation.outPath, &drvStore); - } - worker.store.signRealisation(newRealisation); - worker.store.registerDrvOutput(newRealisation); - } - outputPaths.insert(realisation.outPath); - builtOutputs.emplace(outputName, realisation); - } - - runPostBuildHook( - worker.store, - *logger, - drvPath, - outputPaths - ); - } - - auto status = resolvedResult.status; - if (status == BuildResult::AlreadyValid) - status = BuildResult::ResolvesToAlreadyValid; - - co_return done(status, std::move(builtOutputs)); - } - - /* If we get this far, we know no dynamic drvs inputs */ - - for (auto & [depDrvPath, depNode] : fullDrv.inputDrvs.map) { - for (auto & outputName : depNode.value) { - /* Don't need to worry about `inputGoals`, because - impure derivations are always resolved above. Can - just use DB. This case only happens in the (older) - input addressed and fixed output derivation cases. */ - auto outMap = [&]{ - for (auto * drvStore : { &worker.evalStore, &worker.store }) - if (drvStore->isValidPath(depDrvPath)) - return worker.store.queryDerivationOutputMap(depDrvPath, drvStore); - assert(false); - }(); - - auto outMapPath = outMap.find(outputName); - if (outMapPath == outMap.end()) { - throw Error( - "derivation '%s' requires non-existent output '%s' from input derivation '%s'", - worker.store.printStorePath(drvPath), outputName, worker.store.printStorePath(depDrvPath)); - } - - worker.store.computeFSClosure(outMapPath->second, inputPaths); - } - } - } - - /* Second, the input sources. */ - worker.store.computeFSClosure(drv->inputSrcs, inputPaths); - - debug("added input paths %s", worker.store.showPaths(inputPaths)); - - /* Okay, try to build. Note that here we don't wait for a build - slot to become available, since we don't need one if there is a - build hook. */ - co_await yield(); - co_return tryToBuild(); -} - -void DerivationGoal::started() -{ - auto msg = fmt( - buildMode == bmRepair ? "repairing outputs of '%s'" : - buildMode == bmCheck ? "checking outputs of '%s'" : - "building '%s'", worker.store.printStorePath(drvPath)); - fmt("building '%s'", worker.store.printStorePath(drvPath)); -#ifndef _WIN32 // TODO enable build hook on Windows - if (hook) msg += fmt(" on '%s'", machineName); -#endif - act = std::make_unique(*logger, lvlInfo, actBuild, msg, - Logger::Fields{worker.store.printStorePath(drvPath), -#ifndef _WIN32 // TODO enable build hook on Windows - hook ? machineName : -#endif - "", - 1, - 1}); - mcRunningBuilds = std::make_unique>(worker.runningBuilds); - worker.updateProgress(); -} - -Goal::Co DerivationGoal::tryToBuild() -{ - trace("trying to build"); - - /* Obtain locks on all output paths, if the paths are known a priori. - - The locks are automatically released when we exit this function or Nix - crashes. If we can't acquire the lock, then continue; hopefully some - other goal can start a build, and if not, the main loop will sleep a few - seconds and then retry this goal. */ - PathSet lockFiles; - /* FIXME: Should lock something like the drv itself so we don't build same - CA drv concurrently */ - if (dynamic_cast(&worker.store)) { - /* If we aren't a local store, we might need to use the local store as - a build remote, but that would cause a deadlock. */ - /* FIXME: Make it so we can use ourselves as a build remote even if we - are the local store (separate locking for building vs scheduling? */ - /* FIXME: find some way to lock for scheduling for the other stores so - a forking daemon with --store still won't farm out redundant builds. - */ - for (auto & i : drv->outputsAndOptPaths(worker.store)) { - if (i.second.second) - lockFiles.insert(worker.store.Store::toRealPath(*i.second.second)); - else - lockFiles.insert( - worker.store.Store::toRealPath(drvPath) + "." + i.first - ); - } - } - - if (!outputLocks.lockPaths(lockFiles, "", false)) - { - Activity act(*logger, lvlWarn, actBuildWaiting, - fmt("waiting for lock on %s", Magenta(showPaths(lockFiles)))); - - /* Wait then try locking again, repeat until success (returned - boolean is true). */ - do { - co_await waitForAWhile(); - } while (!outputLocks.lockPaths(lockFiles, "", false)); - } - - /* Now check again whether the outputs are valid. This is because - another process may have started building in parallel. After - it has finished and released the locks, we can (and should) - reuse its results. (Strictly speaking the first check can be - omitted, but that would be less efficient.) Note that since we - now hold the locks on the output paths, no other process can - build this derivation, so no further checks are necessary. */ - auto [allValid, validOutputs] = checkPathValidity(); - - if (buildMode != bmCheck && allValid) { - debug("skipping build of derivation '%s', someone beat us to it", worker.store.printStorePath(drvPath)); - outputLocks.setDeletion(true); - outputLocks.unlock(); - co_return done(BuildResult::AlreadyValid, std::move(validOutputs)); - } - - /* If any of the outputs already exist but are not valid, delete - them. */ - for (auto & [_, status] : initialOutputs) { - if (!status.known || status.known->isValid()) continue; - auto storePath = status.known->path; - debug("removing invalid path '%s'", worker.store.printStorePath(status.known->path)); - deletePath(worker.store.Store::toRealPath(storePath)); - } - - /* Don't do a remote build if the derivation has the attribute - `preferLocalBuild' set. Also, check and repair modes are only - supported for local builds. */ - bool buildLocally = - (buildMode != bmNormal || drvOptions->willBuildLocally(worker.store, *drv)) - && settings.maxBuildJobs.get() != 0; - - if (!buildLocally) { - switch (tryBuildHook()) { - case rpAccept: - /* Yes, it has started doing so. Wait until we get - EOF from the hook. */ - actLock.reset(); - buildResult.startTime = time(0); // inexact - started(); - co_await Suspend{}; - co_return hookDone(); - case rpPostpone: - /* Not now; wait until at least one child finishes or - the wake-up timeout expires. */ - if (!actLock) - actLock = std::make_unique(*logger, lvlWarn, actBuildWaiting, - fmt("waiting for a machine to build '%s'", Magenta(worker.store.printStorePath(drvPath)))); - outputLocks.unlock(); - co_await waitForAWhile(); - co_return tryToBuild(); - case rpDecline: - /* We should do it ourselves. */ - break; - } - } - - actLock.reset(); - - co_await yield(); - - if (!dynamic_cast(&worker.store)) { - throw Error( - R"( - Unable to build with a primary store that isn't a local store; - either pass a different '--store' or enable remote builds. - - For more information check 'man nix.conf' and search for '/machines'. - )" - ); - } - -#ifdef _WIN32 // TODO enable `DerivationBuilder` on Windows - throw UnimplementedError("building derivations is not yet implemented on Windows"); -#else - - // Will continue here while waiting for a build user below - while (true) { - - assert(!hook); - - unsigned int curBuilds = worker.getNrLocalBuilds(); - if (curBuilds >= settings.maxBuildJobs) { - outputLocks.unlock(); - co_await waitForBuildSlot(); - co_return tryToBuild(); - } - - if (!builder) { - /** - * Local implementation of these virtual methods, consider - * this just a record of lambdas. - */ - struct DerivationGoalCallbacks : DerivationBuilderCallbacks - { - DerivationGoal & goal; - - DerivationGoalCallbacks(DerivationGoal & goal, std::unique_ptr & builder) - : goal{goal} - {} - - ~DerivationGoalCallbacks() override = default; - - void childStarted(Descriptor builderOut) override - { - goal.worker.childStarted(goal.shared_from_this(), {builderOut}, true, true); - } - - void childTerminated() override - { - goal.worker.childTerminated(&goal); - } - - void noteHashMismatch() override - { - goal.worker.hashMismatch = true; - } - - void noteCheckMismatch() override - { - goal.worker.checkMismatch = true; - } - - void markContentsGood(const StorePath & path) override - { - goal.worker.markContentsGood(path); - } - - Path openLogFile() override { - return goal.openLogFile(); - } - void closeLogFile() override { - goal.closeLogFile(); - } - SingleDrvOutputs assertPathValidity() override { - return goal.assertPathValidity(); - } - void appendLogTailErrorMsg(std::string & msg) override { - goal.appendLogTailErrorMsg(msg); - } - }; - - /* If we have to wait and retry (see below), then `builder` will - already be created, so we don't need to create it again. */ - builder = makeDerivationBuilder( - worker.store, - std::make_unique(*this, builder), - DerivationBuilderParams { - drvPath, - buildMode, - buildResult, - *drv, - parsedDrv.get(), - *drvOptions, - inputPaths, - initialOutputs, - }); - } - - if (!builder->prepareBuild()) { - if (!actLock) - actLock = std::make_unique(*logger, lvlWarn, actBuildWaiting, - fmt("waiting for a free build user ID for '%s'", Magenta(worker.store.printStorePath(drvPath)))); - co_await waitForAWhile(); - continue; - } - - break; - } - - actLock.reset(); - - try { - - /* Okay, we have to build. */ - builder->startBuilder(); - - } catch (BuildError & e) { - outputLocks.unlock(); - builder->buildUser.reset(); - worker.permanentFailure = true; - co_return done(BuildResult::InputRejected, {}, std::move(e)); - } - - started(); - co_await Suspend{}; - - trace("build done"); - - auto res = builder->unprepareBuild(); - // N.B. cannot use `std::visit` with co-routine return - if (auto * ste = std::get_if<0>(&res)) { - outputLocks.unlock(); - co_return done(std::move(ste->first), {}, std::move(ste->second)); - } else if (auto * builtOutputs = std::get_if<1>(&res)) { - /* It is now safe to delete the lock files, since all future - lockers will see that the output paths are valid; they will - not create new lock files with the same names as the old - (unlinked) lock files. */ - outputLocks.setDeletion(true); - outputLocks.unlock(); - co_return done(BuildResult::Built, std::move(*builtOutputs)); - } else { - unreachable(); - } -#endif -} - - Goal::Co DerivationGoal::repairClosure() { assert(!drv->type().isImpure()); @@ -961,444 +375,6 @@ Goal::Co DerivationGoal::repairClosure() } -void runPostBuildHook( - Store & store, - Logger & logger, - const StorePath & drvPath, - const StorePathSet & outputPaths) -{ - auto hook = settings.postBuildHook; - if (hook == "") - return; - - Activity act(logger, lvlTalkative, actPostBuildHook, - fmt("running post-build-hook '%s'", settings.postBuildHook), - Logger::Fields{store.printStorePath(drvPath)}); - PushActivity pact(act.id); - StringMap hookEnvironment = getEnv(); - - hookEnvironment.emplace("DRV_PATH", store.printStorePath(drvPath)); - hookEnvironment.emplace("OUT_PATHS", chomp(concatStringsSep(" ", store.printStorePathSet(outputPaths)))); - hookEnvironment.emplace("NIX_CONFIG", globalConfig.toKeyValue()); - - struct LogSink : Sink { - Activity & act; - std::string currentLine; - - LogSink(Activity & act) : act(act) { } - - void operator() (std::string_view data) override { - for (auto c : data) { - if (c == '\n') { - flushLine(); - } else { - currentLine += c; - } - } - } - - void flushLine() { - act.result(resPostBuildLogLine, currentLine); - currentLine.clear(); - } - - ~LogSink() { - if (currentLine != "") { - currentLine += '\n'; - flushLine(); - } - } - }; - LogSink sink(act); - - runProgram2({ - .program = settings.postBuildHook, - .environment = hookEnvironment, - .standardOut = &sink, - .mergeStderrToStdout = true, - }); -} - - -void DerivationGoal::appendLogTailErrorMsg(std::string & msg) -{ - if (!logger->isVerbose() && !logTail.empty()) { - msg += fmt("\nLast %d log lines:\n", logTail.size()); - for (auto & line : logTail) { - msg += "> "; - msg += line; - msg += "\n"; - } - auto nixLogCommand = experimentalFeatureSettings.isEnabled(Xp::NixCommand) - ? "nix log" - : "nix-store -l"; - // The command is on a separate line for easy copying, such as with triple click. - // This message will be indented elsewhere, so removing the indentation before the - // command will not put it at the start of the line unfortunately. - msg += fmt("For full logs, run:\n " ANSI_BOLD "%s %s" ANSI_NORMAL, - nixLogCommand, - worker.store.printStorePath(drvPath)); - } -} - - -Goal::Co DerivationGoal::hookDone() -{ -#ifndef _WIN32 - assert(hook); -#endif - - trace("hook build done"); - - /* Since we got an EOF on the logger pipe, the builder is presumed - to have terminated. In fact, the builder could also have - simply have closed its end of the pipe, so just to be sure, - kill it. */ - int status = -#ifndef _WIN32 // TODO enable build hook on Windows - hook->pid.kill(); -#else - 0; -#endif - - debug("build hook for '%s' finished", worker.store.printStorePath(drvPath)); - - buildResult.timesBuilt++; - buildResult.stopTime = time(0); - - /* So the child is gone now. */ - worker.childTerminated(this); - - /* Close the read side of the logger pipe. */ -#ifndef _WIN32 // TODO enable build hook on Windows - hook->builderOut.readSide.close(); - hook->fromHook.readSide.close(); -#endif - - /* Close the log file. */ - closeLogFile(); - - /* Check the exit status. */ - if (!statusOk(status)) { - auto msg = fmt( - "Cannot build '%s'.\n" - "Reason: " ANSI_RED "builder %s" ANSI_NORMAL ".", - Magenta(worker.store.printStorePath(drvPath)), - statusToString(status)); - - msg += showKnownOutputs(worker.store, *drv); - - appendLogTailErrorMsg(msg); - - outputLocks.unlock(); - - /* TODO (once again) support fine-grained error codes, see issue #12641. */ - - co_return done(BuildResult::MiscFailure, {}, BuildError(msg)); - } - - /* Compute the FS closure of the outputs and register them as - being valid. */ - auto builtOutputs = - /* When using a build hook, the build hook can register the output - as valid (by doing `nix-store --import'). If so we don't have - to do anything here. - - We can only early return when the outputs are known a priori. For - floating content-addressing derivations this isn't the case. - */ - assertPathValidity(); - - StorePathSet outputPaths; - for (auto & [_, output] : builtOutputs) - outputPaths.insert(output.outPath); - runPostBuildHook( - worker.store, - *logger, - drvPath, - outputPaths - ); - - /* It is now safe to delete the lock files, since all future - lockers will see that the output paths are valid; they will - not create new lock files with the same names as the old - (unlinked) lock files. */ - outputLocks.setDeletion(true); - outputLocks.unlock(); - - co_return done(BuildResult::Built, std::move(builtOutputs)); -} - -HookReply DerivationGoal::tryBuildHook() -{ -#ifdef _WIN32 // TODO enable build hook on Windows - return rpDecline; -#else - /* This should use `worker.evalStore`, but per #13179 the build hook - doesn't work with eval store anyways. */ - if (settings.buildHook.get().empty() || !worker.tryBuildHook || !worker.store.isValidPath(drvPath)) return rpDecline; - - if (!worker.hook) - worker.hook = std::make_unique(); - - try { - - /* Send the request to the hook. */ - worker.hook->sink - << "try" - << (worker.getNrLocalBuilds() < settings.maxBuildJobs ? 1 : 0) - << drv->platform - << worker.store.printStorePath(drvPath) - << drvOptions->getRequiredSystemFeatures(*drv); - worker.hook->sink.flush(); - - /* Read the first line of input, which should be a word indicating - whether the hook wishes to perform the build. */ - std::string reply; - while (true) { - auto s = [&]() { - try { - return readLine(worker.hook->fromHook.readSide.get()); - } catch (Error & e) { - e.addTrace({}, "while reading the response from the build hook"); - throw; - } - }(); - if (handleJSONLogMessage(s, worker.act, worker.hook->activities, "the build hook", true)) - ; - else if (s.substr(0, 2) == "# ") { - reply = s.substr(2); - break; - } - else { - s += "\n"; - writeToStderr(s); - } - } - - debug("hook reply is '%1%'", reply); - - if (reply == "decline") - return rpDecline; - else if (reply == "decline-permanently") { - worker.tryBuildHook = false; - worker.hook = 0; - return rpDecline; - } - else if (reply == "postpone") - return rpPostpone; - else if (reply != "accept") - throw Error("bad hook reply '%s'", reply); - - } catch (SysError & e) { - if (e.errNo == EPIPE) { - printError( - "build hook died unexpectedly: %s", - chomp(drainFD(worker.hook->fromHook.readSide.get()))); - worker.hook = 0; - return rpDecline; - } else - throw; - } - - hook = std::move(worker.hook); - - try { - machineName = readLine(hook->fromHook.readSide.get()); - } catch (Error & e) { - e.addTrace({}, "while reading the machine name from the build hook"); - throw; - } - - CommonProto::WriteConn conn { hook->sink }; - - /* Tell the hook all the inputs that have to be copied to the - remote system. */ - CommonProto::write(worker.store, conn, inputPaths); - - /* Tell the hooks the missing outputs that have to be copied back - from the remote system. */ - { - StringSet missingOutputs; - for (auto & [outputName, status] : initialOutputs) { - // XXX: Does this include known CA outputs? - if (buildMode != bmCheck && status.known && status.known->isValid()) continue; - missingOutputs.insert(outputName); - } - CommonProto::write(worker.store, conn, missingOutputs); - } - - hook->sink = FdSink(); - hook->toHook.writeSide.close(); - - /* Create the log file and pipe. */ - [[maybe_unused]] Path logFile = openLogFile(); - - std::set fds; - fds.insert(hook->fromHook.readSide.get()); - fds.insert(hook->builderOut.readSide.get()); - worker.childStarted(shared_from_this(), fds, false, false); - - return rpAccept; -#endif -} - - -Path DerivationGoal::openLogFile() -{ - logSize = 0; - - if (!settings.keepLog) return ""; - - auto baseName = std::string(baseNameOf(worker.store.printStorePath(drvPath))); - - /* Create a log file. */ - Path logDir; - if (auto localStore = dynamic_cast(&worker.store)) - logDir = localStore->config->logDir; - else - logDir = settings.nixLogDir; - Path dir = fmt("%s/%s/%s/", logDir, LocalFSStore::drvsLogDir, baseName.substr(0, 2)); - createDirs(dir); - - Path logFileName = fmt("%s/%s%s", dir, baseName.substr(2), - settings.compressLog ? ".bz2" : ""); - - fdLogFile = toDescriptor(open(logFileName.c_str(), O_CREAT | O_WRONLY | O_TRUNC -#ifndef _WIN32 - | O_CLOEXEC -#endif - , 0666)); - if (!fdLogFile) throw SysError("creating log file '%1%'", logFileName); - - logFileSink = std::make_shared(fdLogFile.get()); - - if (settings.compressLog) - logSink = std::shared_ptr(makeCompressionSink("bzip2", *logFileSink)); - else - logSink = logFileSink; - - return logFileName; -} - - -void DerivationGoal::closeLogFile() -{ - auto logSink2 = std::dynamic_pointer_cast(logSink); - if (logSink2) logSink2->finish(); - if (logFileSink) logFileSink->flush(); - logSink = logFileSink = 0; - fdLogFile.close(); -} - - -bool DerivationGoal::isReadDesc(Descriptor fd) -{ -#ifdef _WIN32 // TODO enable build hook on Windows - return false; -#else - return - (hook && fd == hook->builderOut.readSide.get()) - || - (builder && fd == builder->builderOut.get()); -#endif -} - -void DerivationGoal::handleChildOutput(Descriptor fd, std::string_view data) -{ - // local & `ssh://`-builds are dealt with here. - auto isWrittenToLog = isReadDesc(fd); - if (isWrittenToLog) - { - logSize += data.size(); - if (settings.maxLogSize && logSize > settings.maxLogSize) { - killChild(); - // We're not inside a coroutine, hence we can't use co_return here. - // Thus we ignore the return value. - [[maybe_unused]] Done _ = done( - BuildResult::LogLimitExceeded, {}, - Error("%s killed after writing more than %d bytes of log output", - getName(), settings.maxLogSize)); - return; - } - - for (auto c : data) - if (c == '\r') - currentLogLinePos = 0; - else if (c == '\n') - flushLine(); - else { - if (currentLogLinePos >= currentLogLine.size()) - currentLogLine.resize(currentLogLinePos + 1); - currentLogLine[currentLogLinePos++] = c; - } - - if (logSink) (*logSink)(data); - } - -#ifndef _WIN32 // TODO enable build hook on Windows - if (hook && fd == hook->fromHook.readSide.get()) { - for (auto c : data) - if (c == '\n') { - auto json = parseJSONMessage(currentHookLine, "the derivation builder"); - if (json) { - auto s = handleJSONLogMessage(*json, worker.act, hook->activities, "the derivation builder", true); - // ensure that logs from a builder using `ssh-ng://` as protocol - // are also available to `nix log`. - if (s && !isWrittenToLog && logSink) { - const auto type = (*json)["type"]; - const auto fields = (*json)["fields"]; - if (type == resBuildLogLine) { - (*logSink)((fields.size() > 0 ? fields[0].get() : "") + "\n"); - } else if (type == resSetPhase && ! fields.is_null()) { - const auto phase = fields[0]; - if (! phase.is_null()) { - // nixpkgs' stdenv produces lines in the log to signal - // phase changes. - // We want to get the same lines in case of remote builds. - // The format is: - // @nix { "action": "setPhase", "phase": "$curPhase" } - const auto logLine = nlohmann::json::object({ - {"action", "setPhase"}, - {"phase", phase} - }); - (*logSink)("@nix " + logLine.dump(-1, ' ', false, nlohmann::json::error_handler_t::replace) + "\n"); - } - } - } - } - currentHookLine.clear(); - } else - currentHookLine += c; - } -#endif -} - - -void DerivationGoal::handleEOF(Descriptor fd) -{ - if (!currentLogLine.empty()) flushLine(); - worker.wakeUp(shared_from_this()); -} - - -void DerivationGoal::flushLine() -{ - if (handleJSONLogMessage(currentLogLine, *act, builderActivities, "the derivation builder", false)) - ; - - else { - logTail.push_back(currentLogLine); - if (logTail.size() > settings.logLines) logTail.pop_front(); - - act->result(resBuildLogLine, currentLogLine); - } - - currentLogLine = ""; - currentLogLinePos = 0; -} - - std::map> DerivationGoal::queryPartialDerivationOutputMap() { assert(!drv->type().isImpure()); @@ -1525,7 +501,6 @@ Goal::Done DerivationGoal::done( SingleDrvOutputs builtOutputs, std::optional ex) { - outputLocks.unlock(); buildResult.status = status; if (ex) buildResult.errorMsg = fmt("%s", Uncolored(ex->info().msg)); @@ -1535,7 +510,6 @@ Goal::Done DerivationGoal::done( worker.permanentFailure = true; mcExpectedBuilds.reset(); - mcRunningBuilds.reset(); if (buildResult.success()) { auto wantedBuiltOutputs = filterDrvOutputs(wantedOutputs, std::move(builtOutputs)); diff --git a/src/libstore/build/goal.cc b/src/libstore/build/goal.cc index c9294a1a4d1..8a8d7928326 100644 --- a/src/libstore/build/goal.cc +++ b/src/libstore/build/goal.cc @@ -155,7 +155,7 @@ Goal::Done Goal::amDone(ExitCode result, std::optional ex) exitCode = result; if (ex) { - if (!waiters.empty()) + if (!preserveException && !waiters.empty()) logError(ex->info()); else this->ex = std::move(*ex); diff --git a/src/libstore/build/worker.cc b/src/libstore/build/worker.cc index 6b1e3c0f40f..aaf4c149fbe 100644 --- a/src/libstore/build/worker.cc +++ b/src/libstore/build/worker.cc @@ -4,6 +4,7 @@ #include "nix/store/build/substitution-goal.hh" #include "nix/store/build/drv-output-substitution-goal.hh" #include "nix/store/build/derivation-goal.hh" +#include "nix/store/build/derivation-building-goal.hh" #ifndef _WIN32 // TODO Enable building on Windows # include "nix/store/build/hook-instance.hh" #endif @@ -87,6 +88,20 @@ std::shared_ptr Worker::makeBasicDerivationGoal(const StorePath } +std::shared_ptr Worker::makeDerivationBuildingGoal(const StorePath & drvPath, + const Derivation & drv, BuildMode buildMode) +{ + std::weak_ptr & goal_weak = derivationBuildingGoals[drvPath]; + auto goal = goal_weak.lock(); // FIXME + if (!goal) { + goal = std::make_shared(drvPath, drv, *this, buildMode); + goal_weak = goal; + wakeUp(goal); + } + return goal; +} + + std::shared_ptr Worker::makePathSubstitutionGoal(const StorePath & path, RepairFlag repair, std::optional ca) { return initGoalIfNeeded(substitutionGoals[path], path, *this, repair, ca); @@ -134,8 +149,9 @@ void Worker::removeGoal(GoalPtr goal) { if (auto drvGoal = std::dynamic_pointer_cast(goal)) nix::removeGoal(drvGoal, derivationGoals); - else - if (auto subGoal = std::dynamic_pointer_cast(goal)) + else if (auto drvBuildingGoal = std::dynamic_pointer_cast(goal)) + nix::removeGoal(drvBuildingGoal, derivationBuildingGoals); + else if (auto subGoal = std::dynamic_pointer_cast(goal)) nix::removeGoal(subGoal, substitutionGoals); else if (auto subGoal = std::dynamic_pointer_cast(goal)) nix::removeGoal(subGoal, drvOutputSubstitutionGoals); @@ -198,6 +214,9 @@ void Worker::childStarted(GoalPtr goal, const std::set 0); nrLocalBuilds--; break; + case JobCategory::Administration: + /* Intentionally not limited, see docs */ + break; default: unreachable(); } diff --git a/src/libstore/include/nix/store/build/derivation-building-goal.hh b/src/libstore/include/nix/store/build/derivation-building-goal.hh index dc65b494135..bff2e7a89a9 100644 --- a/src/libstore/include/nix/store/build/derivation-building-goal.hh +++ b/src/libstore/include/nix/store/build/derivation-building-goal.hh @@ -31,43 +31,11 @@ void runPostBuildHook( /** * A goal for building some or all of the outputs of a derivation. */ -struct DerivationGoal : public Goal +struct DerivationBuildingGoal : public Goal { /** The path of the derivation. */ StorePath drvPath; - /** - * The specific outputs that we need to build. - */ - OutputsSpec wantedOutputs; - - /** - * See `needRestart`; just for that field. - */ - enum struct NeedRestartForMoreOutputs { - /** - * The goal state machine is progressing based on the current value of - * `wantedOutputs. No actions are needed. - */ - OutputsUnmodifedDontNeed, - /** - * `wantedOutputs` has been extended, but the state machine is - * proceeding according to its old value, so we need to restart. - */ - OutputsAddedDoNeed, - /** - * The goal state machine has progressed to the point of doing a build, - * in which case all outputs will be produced, so extensions to - * `wantedOutputs` no longer require a restart. - */ - BuildInProgressWillNotNeed, - }; - - /** - * Whether additional wanted outputs have been added. - */ - NeedRestartForMoreOutputs needRestart = NeedRestartForMoreOutputs::OutputsUnmodifedDontNeed; - /** * The derivation stored at drvPath. */ @@ -125,7 +93,7 @@ struct DerivationGoal : public Goal BuildMode buildMode; - std::unique_ptr> mcExpectedBuilds, mcRunningBuilds; + std::unique_ptr> mcRunningBuilds; std::unique_ptr act; @@ -141,28 +109,18 @@ struct DerivationGoal : public Goal */ std::string machineName; - DerivationGoal(const StorePath & drvPath, - const OutputsSpec & wantedOutputs, Worker & worker, + DerivationBuildingGoal(const StorePath & drvPath, const Derivation & drv, + Worker & worker, BuildMode buildMode = bmNormal); - DerivationGoal(const StorePath & drvPath, const BasicDerivation & drv, - const OutputsSpec & wantedOutputs, Worker & worker, - BuildMode buildMode = bmNormal); - ~DerivationGoal(); + ~DerivationBuildingGoal(); void timedOut(Error && ex) override; std::string key() override; - /** - * Add wanted outputs to an already existing derivation goal. - */ - void addWantedOutputs(const OutputsSpec & outputs); - /** * The states. */ - Co loadDerivation(); - Co haveDerivation(); Co gaveUpOnSubstitution(); Co tryToBuild(); Co hookDone(); @@ -197,7 +155,6 @@ struct DerivationGoal : public Goal * there also is no DB entry. */ std::map> queryPartialDerivationOutputMap(); - OutputPathMap queryDerivationOutputMap(); /** * Update 'initialOutputs' to determine the current status of the @@ -218,8 +175,6 @@ struct DerivationGoal : public Goal */ void killChild(); - Co repairClosure(); - void started(); Done done( diff --git a/src/libstore/include/nix/store/build/derivation-goal.hh b/src/libstore/include/nix/store/build/derivation-goal.hh index dc65b494135..7118fd4a544 100644 --- a/src/libstore/include/nix/store/build/derivation-goal.hh +++ b/src/libstore/include/nix/store/build/derivation-goal.hh @@ -14,13 +14,6 @@ namespace nix { using std::map; -#ifndef _WIN32 // TODO enable build hook on Windows -struct HookInstance; -struct DerivationBuilder; -#endif - -typedef enum {rpAccept, rpDecline, rpPostpone} HookReply; - /** Used internally */ void runPostBuildHook( Store & store, @@ -73,73 +66,15 @@ struct DerivationGoal : public Goal */ std::unique_ptr drv; - std::unique_ptr parsedDrv; - std::unique_ptr drvOptions; - /** * The remainder is state held during the build. */ - /** - * Locks on (fixed) output paths. - */ - PathLocks outputLocks; - - /** - * All input paths (that is, the union of FS closures of the - * immediate input paths). - */ - StorePathSet inputPaths; - std::map initialOutputs; - /** - * File descriptor for the log file. - */ - AutoCloseFD fdLogFile; - std::shared_ptr logFileSink, logSink; - - /** - * Number of bytes received from the builder's stdout/stderr. - */ - unsigned long logSize; - - /** - * The most recent log lines. - */ - std::list logTail; - - std::string currentLogLine; - size_t currentLogLinePos = 0; // to handle carriage return - - std::string currentHookLine; - -#ifndef _WIN32 // TODO enable build hook on Windows - /** - * The build hook. - */ - std::unique_ptr hook; - - std::unique_ptr builder; -#endif - BuildMode buildMode; - std::unique_ptr> mcExpectedBuilds, mcRunningBuilds; - - std::unique_ptr act; - - /** - * Activity that denotes waiting for a lock. - */ - std::unique_ptr actLock; - - std::map builderActivities; - - /** - * The remote machine on which we're building. - */ - std::string machineName; + std::unique_ptr> mcExpectedBuilds; DerivationGoal(const StorePath & drvPath, const OutputsSpec & wantedOutputs, Worker & worker, @@ -147,9 +82,9 @@ struct DerivationGoal : public Goal DerivationGoal(const StorePath & drvPath, const BasicDerivation & drv, const OutputsSpec & wantedOutputs, Worker & worker, BuildMode buildMode = bmNormal); - ~DerivationGoal(); + ~DerivationGoal() = default; - void timedOut(Error && ex) override; + void timedOut(Error && ex) override { unreachable(); }; std::string key() override; @@ -163,33 +98,6 @@ struct DerivationGoal : public Goal */ Co loadDerivation(); Co haveDerivation(); - Co gaveUpOnSubstitution(); - Co tryToBuild(); - Co hookDone(); - - /** - * Is the build hook willing to perform the build? - */ - HookReply tryBuildHook(); - - /** - * Open a log file and a pipe to it. - */ - Path openLogFile(); - - /** - * Close the log file. - */ - void closeLogFile(); - - bool isReadDesc(Descriptor fd); - - /** - * Callback used by the worker to write to the log. - */ - void handleChildOutput(Descriptor fd, std::string_view data) override; - void handleEOF(Descriptor fd) override; - void flushLine(); /** * Wrappers around the corresponding Store methods that first consult the @@ -213,26 +121,15 @@ struct DerivationGoal : public Goal */ SingleDrvOutputs assertPathValidity(); - /** - * Forcibly kill the child process, if any. - */ - void killChild(); - Co repairClosure(); - void started(); - Done done( BuildResult::Status status, SingleDrvOutputs builtOutputs = {}, std::optional ex = {}); - void appendLogTailErrorMsg(std::string & msg); - - StorePathSet exportReferences(const StorePathSet & storePaths); - JobCategory jobCategory() const override { - return JobCategory::Build; + return JobCategory::Administration; }; }; diff --git a/src/libstore/include/nix/store/build/goal.hh b/src/libstore/include/nix/store/build/goal.hh index 669b58e3e46..577ce1e843e 100644 --- a/src/libstore/include/nix/store/build/goal.hh +++ b/src/libstore/include/nix/store/build/goal.hh @@ -50,6 +50,16 @@ enum struct JobCategory { * A substitution an arbitrary store object; it will use network resources. */ Substitution, + /** + * A goal that does no "real" work by itself, and just exists to depend on + * other goals which *do* do real work. These goals therefore are not + * limited. + * + * These goals cannot infinitely create themselves, so there is no risk of + * a "fork bomb" type situation (which would be a problem even though the + * goal do no real work) either. + */ + Administration, }; struct Goal : public std::enable_shared_from_this @@ -360,6 +370,17 @@ public: */ BuildResult getBuildResult(const DerivedPath &) const; + /** + * Hack to say that this goal should not log `ex`, but instead keep + * it around. Set by a waitee which sees itself as the designated + * continuation of this goal, responsible for reporting its + * successes or failures. + * + * @todo this is yet another not-nice hack in the goal system that + * we ought to get rid of. See #11927 + */ + bool preserveException = false; + /** * Exception containing an error message, if any. */ diff --git a/src/libstore/include/nix/store/build/worker.hh b/src/libstore/include/nix/store/build/worker.hh index adbaa402786..46c4d181edb 100644 --- a/src/libstore/include/nix/store/build/worker.hh +++ b/src/libstore/include/nix/store/build/worker.hh @@ -14,6 +14,7 @@ namespace nix { /* Forward definition. */ struct DerivationGoal; +struct DerivationBuildingGoal; struct PathSubstitutionGoal; class DrvOutputSubstitutionGoal; @@ -104,6 +105,7 @@ private: * same derivation / path. */ std::map> derivationGoals; + std::map> derivationBuildingGoals; std::map> substitutionGoals; std::map> drvOutputSubstitutionGoals; @@ -210,6 +212,13 @@ public: const StorePath & drvPath, const BasicDerivation & drv, const OutputsSpec & wantedOutputs, BuildMode buildMode = bmNormal); + /** + * @ref DerivationBuildingGoal "derivation goal" + */ + std::shared_ptr makeDerivationBuildingGoal( + const StorePath & drvPath, const Derivation & drv, + BuildMode buildMode = bmNormal); + /** * @ref PathSubstitutionGoal "substitution goal" */ diff --git a/src/libstore/include/nix/store/meson.build b/src/libstore/include/nix/store/meson.build index c5aa9b46146..a1843041760 100644 --- a/src/libstore/include/nix/store/meson.build +++ b/src/libstore/include/nix/store/meson.build @@ -13,6 +13,7 @@ headers = [config_pub_h] + files( 'binary-cache-store.hh', 'build-result.hh', 'build/derivation-goal.hh', + 'build/derivation-building-goal.hh', 'build/derivation-building-misc.hh', 'build/drv-output-substitution-goal.hh', 'build/goal.hh', diff --git a/src/libstore/meson.build b/src/libstore/meson.build index 672993bf05e..94b8951fdd9 100644 --- a/src/libstore/meson.build +++ b/src/libstore/meson.build @@ -254,6 +254,7 @@ sources = files( 'binary-cache-store.cc', 'build-result.cc', 'build/derivation-goal.cc', + 'build/derivation-building-goal.cc', 'build/drv-output-substitution-goal.cc', 'build/entry-points.cc', 'build/goal.cc', From 6d59fa03fd400965453564ffd9f6c1b048a60815 Mon Sep 17 00:00:00 2001 From: Eelco Dolstra Date: Tue, 20 May 2025 18:24:12 +0200 Subject: [PATCH 043/218] Add release note --- doc/manual/rl-next/nix-profile-add.md | 6 ++++++ 1 file changed, 6 insertions(+) create mode 100644 doc/manual/rl-next/nix-profile-add.md diff --git a/doc/manual/rl-next/nix-profile-add.md b/doc/manual/rl-next/nix-profile-add.md new file mode 100644 index 00000000000..a2fbcaa8da5 --- /dev/null +++ b/doc/manual/rl-next/nix-profile-add.md @@ -0,0 +1,6 @@ +--- +synopsis: "Rename `nix profile install` to `nix profile add`" +prs: [13224] +--- + +The command `nix profile install` has been renamed to `nix profile add` (though the former is still available as an alias). This is because the verb "add" is a better antonym for the verb "remove" (i.e. `nix profile remove`). Nix also does not have install hooks or general behavior often associated with "installing". From f627b8c721c92cdaa1a15687a181e73e947eb2ef Mon Sep 17 00:00:00 2001 From: Stefan Boca Date: Tue, 20 May 2025 22:04:52 -0700 Subject: [PATCH 044/218] nix-profile{,-daemon}.fish: format with fish_indent --- scripts/nix-profile-daemon.fish.in | 5 ++--- scripts/nix-profile.fish.in | 5 ++--- 2 files changed, 4 insertions(+), 6 deletions(-) diff --git a/scripts/nix-profile-daemon.fish.in b/scripts/nix-profile-daemon.fish.in index 3fb7271510a..f7cc5239e43 100644 --- a/scripts/nix-profile-daemon.fish.in +++ b/scripts/nix-profile-daemon.fish.in @@ -1,6 +1,5 @@ # Only execute this file once per shell. -if test -z "$HOME" || \ - test -n "$__ETC_PROFILE_NIX_SOURCED" +if test -z "$HOME" || test -n "$__ETC_PROFILE_NIX_SOURCED" exit end @@ -11,7 +10,7 @@ set --global __ETC_PROFILE_NIX_SOURCED 1 function add_path --argument-names new_path if type -q fish_add_path # fish 3.2.0 or newer - fish_add_path --prepend --global $new_path + fish_add_path --prepend --global $new_path else # older versions of fish if not contains $new_path $fish_user_paths diff --git a/scripts/nix-profile.fish.in b/scripts/nix-profile.fish.in index 3fb7271510a..f7cc5239e43 100644 --- a/scripts/nix-profile.fish.in +++ b/scripts/nix-profile.fish.in @@ -1,6 +1,5 @@ # Only execute this file once per shell. -if test -z "$HOME" || \ - test -n "$__ETC_PROFILE_NIX_SOURCED" +if test -z "$HOME" || test -n "$__ETC_PROFILE_NIX_SOURCED" exit end @@ -11,7 +10,7 @@ set --global __ETC_PROFILE_NIX_SOURCED 1 function add_path --argument-names new_path if type -q fish_add_path # fish 3.2.0 or newer - fish_add_path --prepend --global $new_path + fish_add_path --prepend --global $new_path else # older versions of fish if not contains $new_path $fish_user_paths From 6aed9d877cfbe1e649b69d2f15b6aeccbfbb0f3a Mon Sep 17 00:00:00 2001 From: Farid Zakaria Date: Tue, 20 May 2025 19:13:49 -0700 Subject: [PATCH 045/218] cherry-pick https://gerrit.lix.systems/c/lix/+/2100 Cherry-pick https://gerrit.lix.systems/c/lix/+/2100 This change fixes a potential concurrency failure when accessing random which is not thread safe. Co-authored-by: Lily Ballard --- src/libmain/shared.cc | 10 ------- src/libstore/build/derivation-goal.cc | 2 +- src/libstore/unix/build/derivation-builder.cc | 30 ++++++++++--------- src/libstore/unix/user-lock.cc | 1 + src/libutil/file-system.cc | 13 ++++---- src/libutil/include/nix/util/file-system.hh | 14 +++++---- src/libutil/source-accessor.cc | 2 +- 7 files changed, 35 insertions(+), 37 deletions(-) diff --git a/src/libmain/shared.cc b/src/libmain/shared.cc index 50d4991be8b..2da22a7731c 100644 --- a/src/libmain/shared.cc +++ b/src/libmain/shared.cc @@ -176,16 +176,6 @@ void initNix(bool loadConfig) now. In particular, store objects should be readable by everybody. */ umask(0022); - - /* Initialise the PRNG. */ - struct timeval tv; - gettimeofday(&tv, 0); -#ifndef _WIN32 - srandom(tv.tv_usec); -#endif - srand(tv.tv_usec); - - } diff --git a/src/libstore/build/derivation-goal.cc b/src/libstore/build/derivation-goal.cc index 82c350832ee..9763a5cd76c 100644 --- a/src/libstore/build/derivation-goal.cc +++ b/src/libstore/build/derivation-goal.cc @@ -9,7 +9,7 @@ #include "nix/util/util.hh" #include "nix/util/compression.hh" #include "nix/store/common-protocol.hh" -#include "nix/store/common-protocol-impl.hh" +#include "nix/store/common-protocol-impl.hh" // Don't remove is actually needed #include "nix/store/local-store.hh" // TODO remove, along with remaining downcasts #include diff --git a/src/libstore/unix/build/derivation-builder.cc b/src/libstore/unix/build/derivation-builder.cc index f3847e60469..f107f7a5d50 100644 --- a/src/libstore/unix/build/derivation-builder.cc +++ b/src/libstore/unix/build/derivation-builder.cc @@ -1,22 +1,14 @@ #include "nix/store/build/derivation-builder.hh" #include "nix/store/local-store.hh" #include "nix/util/processes.hh" -#include "nix/store/indirect-root-store.hh" -#include "nix/store/build/hook-instance.hh" -#include "nix/store/build/worker.hh" #include "nix/store/builtins.hh" -#include "nix/store/builtins/buildenv.hh" #include "nix/store/path-references.hh" #include "nix/util/finally.hh" #include "nix/util/util.hh" #include "nix/util/archive.hh" #include "nix/util/git.hh" -#include "nix/util/compression.hh" #include "nix/store/daemon.hh" #include "nix/util/topo-sort.hh" -#include "nix/util/callback.hh" -#include "nix/util/json-utils.hh" -#include "nix/util/current-process.hh" #include "nix/store/build/child.hh" #include "nix/util/unix-domain-socket.hh" #include "nix/store/posix-fs-canonicalise.hh" @@ -671,23 +663,33 @@ static void replaceValidPath(const Path & storePath, const Path & tmpPath) tmpPath (the replacement), so we have to move it out of the way first. We'd better not be interrupted here, because if we're repairing (say) Glibc, we end up with a broken system. */ - Path oldPath = fmt("%1%.old-%2%-%3%", storePath, getpid(), rand()); - if (pathExists(storePath)) + Path oldPath; + + if (pathExists(storePath)) { + // why do we loop here? + // although makeTempPath should be unique, we can't + // guarantee that. + do { + oldPath = makeTempPath(storePath, ".old"); + // store paths are often directories so we can't just unlink() it + // let's make sure the path doesn't exist before we try to use it + } while (pathExists(oldPath)); movePath(storePath, oldPath); - + } try { movePath(tmpPath, storePath); } catch (...) { try { // attempt to recover - movePath(oldPath, storePath); + if (!oldPath.empty()) + movePath(oldPath, storePath); } catch (...) { ignoreExceptionExceptInterrupt(); } throw; } - - deletePath(oldPath); + if (!oldPath.empty()) + deletePath(oldPath); } diff --git a/src/libstore/unix/user-lock.cc b/src/libstore/unix/user-lock.cc index 2bee277f9db..ef806af79b7 100644 --- a/src/libstore/unix/user-lock.cc +++ b/src/libstore/unix/user-lock.cc @@ -7,6 +7,7 @@ #include "nix/store/globals.hh" #include "nix/store/pathlocks.hh" #include "nix/util/users.hh" +#include "nix/util/logging.hh" namespace nix { diff --git a/src/libutil/file-system.cc b/src/libutil/file-system.cc index 90ec5eda53b..06c9a8c1f0c 100644 --- a/src/libutil/file-system.cc +++ b/src/libutil/file-system.cc @@ -8,12 +8,12 @@ #include "nix/util/util.hh" #include +#include #include #include #include #include #include -#include #include #include @@ -27,10 +27,6 @@ # include #endif -#include "nix/util/strings-inline.hh" - -#include "util-config-private.hh" - namespace nix { DirectoryIterator::DirectoryIterator(const std::filesystem::path& p) { @@ -641,6 +637,13 @@ std::pair createTempFile(const Path & prefix) return {std::move(fd), tmpl}; } +Path makeTempPath(const Path & root, const Path & suffix) +{ + // start the counter at a random value to minimize issues with preexisting temp paths + static std::atomic_uint_fast32_t counter(std::random_device{}()); + return fmt("%1%%2%-%3%-%4%", root, suffix, getpid(), counter.fetch_add(1, std::memory_order_relaxed)); +} + void createSymlink(const Path & target, const Path & link) { try { diff --git a/src/libutil/include/nix/util/file-system.hh b/src/libutil/include/nix/util/file-system.hh index b8fa4cfa0a7..02af5b37b6f 100644 --- a/src/libutil/include/nix/util/file-system.hh +++ b/src/libutil/include/nix/util/file-system.hh @@ -6,8 +6,6 @@ */ #include "nix/util/types.hh" -#include "nix/util/error.hh" -#include "nix/util/logging.hh" #include "nix/util/file-descriptor.hh" #include "nix/util/file-path.hh" @@ -18,12 +16,8 @@ #ifdef _WIN32 # include #endif -#include -#include #include -#include -#include #include /** @@ -335,6 +329,14 @@ Path defaultTempDir(); */ bool isExecutableFileAmbient(const std::filesystem::path & exe); +/** + * Return temporary path constructed by appending a suffix to a root path. + * + * The constructed path looks like `--`. To create a + * path nested in a directory, provide a suffix starting with `/`. + */ +Path makeTempPath(const Path & root, const Path & suffix = ".tmp"); + /** * Used in various places. */ diff --git a/src/libutil/source-accessor.cc b/src/libutil/source-accessor.cc index b9ebc82b6b1..fc9752456a1 100644 --- a/src/libutil/source-accessor.cc +++ b/src/libutil/source-accessor.cc @@ -1,5 +1,5 @@ +#include #include "nix/util/source-accessor.hh" -#include "nix/util/archive.hh" namespace nix { From 9e26549c2bea1d1e4ab088a55c12e160a2a1b3aa Mon Sep 17 00:00:00 2001 From: Farid Zakaria Date: Wed, 21 May 2025 09:01:49 -0700 Subject: [PATCH 046/218] Consolidate tempName and makeTempPath --- src/libstore/ssh.cc | 2 +- src/libstore/unix/build/derivation-builder.cc | 3 ++- src/libutil/file-system.cc | 19 ++----------------- src/libutil/include/nix/util/file-system.hh | 4 ++-- 4 files changed, 7 insertions(+), 21 deletions(-) diff --git a/src/libstore/ssh.cc b/src/libstore/ssh.cc index d29734a3e4b..c8fec52442e 100644 --- a/src/libstore/ssh.cc +++ b/src/libstore/ssh.cc @@ -34,7 +34,7 @@ SSHMaster::SSHMaster( throw Error("invalid SSH host name '%s'", host); auto state(state_.lock()); - state->tmpDir = std::make_unique(createTempDir("", "nix", true, true, 0700)); + state->tmpDir = std::make_unique(createTempDir("", "nix", 0700)); } void SSHMaster::addCommonSSHOpts(Strings & args) diff --git a/src/libstore/unix/build/derivation-builder.cc b/src/libstore/unix/build/derivation-builder.cc index f107f7a5d50..81e0a2584f6 100644 --- a/src/libstore/unix/build/derivation-builder.cc +++ b/src/libstore/unix/build/derivation-builder.cc @@ -1,4 +1,5 @@ #include "nix/store/build/derivation-builder.hh" +#include "nix/util/file-system.hh" #include "nix/store/local-store.hh" #include "nix/util/processes.hh" #include "nix/store/builtins.hh" @@ -879,7 +880,7 @@ void DerivationBuilderImpl::startBuilder() /* Create a temporary directory where the build will take place. */ - topTmpDir = createTempDir(settings.buildDir.get().value_or(""), "nix-build-" + std::string(drvPath.name()), false, false, 0700); + topTmpDir = createTempDir(settings.buildDir.get().value_or(""), "nix-build-" + std::string(drvPath.name()), 0700); #ifdef __APPLE__ if (false) { #else diff --git a/src/libutil/file-system.cc b/src/libutil/file-system.cc index 06c9a8c1f0c..103fe11b80e 100644 --- a/src/libutil/file-system.cc +++ b/src/libutil/file-system.cc @@ -578,26 +578,11 @@ std::string defaultTempDir() { return getEnvNonEmpty("TMPDIR").value_or("/tmp"); } -static Path tempName(Path tmpRoot, const Path & prefix, bool includePid, - std::atomic & counter) +Path createTempDir(const Path & tmpRoot, const Path & prefix, mode_t mode) { - tmpRoot = canonPath(tmpRoot.empty() ? defaultTempDir() : tmpRoot, true); - if (includePid) - return fmt("%1%/%2%-%3%-%4%", tmpRoot, prefix, getpid(), counter++); - else - return fmt("%1%/%2%-%3%", tmpRoot, prefix, counter++); -} - -Path createTempDir(const Path & tmpRoot, const Path & prefix, - bool includePid, bool useGlobalCounter, mode_t mode) -{ - static std::atomic globalCounter = 0; - std::atomic localCounter = 0; - auto & counter(useGlobalCounter ? globalCounter : localCounter); - while (1) { checkInterrupt(); - Path tmpDir = tempName(tmpRoot, prefix, includePid, counter); + Path tmpDir = makeTempPath(tmpRoot, prefix); if (mkdir(tmpDir.c_str() #ifndef _WIN32 // TODO abstract mkdir perms for Windows , mode diff --git a/src/libutil/include/nix/util/file-system.hh b/src/libutil/include/nix/util/file-system.hh index 02af5b37b6f..b4cc1567d5d 100644 --- a/src/libutil/include/nix/util/file-system.hh +++ b/src/libutil/include/nix/util/file-system.hh @@ -310,8 +310,8 @@ typedef std::unique_ptr AutoCloseDir; /** * Create a temporary directory. */ -Path createTempDir(const Path & tmpRoot = "", const Path & prefix = "nix", - bool includePid = true, bool useGlobalCounter = true, mode_t mode = 0755); +Path createTempDir(const Path & tmpRoot = "", const Path & prefix = "nix", + mode_t mode = 0755); /** * Create a temporary file, returning a file handle and its path. From 8fd15ac228af050f3a4ba0fbce8de2c7918612e8 Mon Sep 17 00:00:00 2001 From: Farid Zakaria Date: Wed, 21 May 2025 10:05:23 -0700 Subject: [PATCH 047/218] Add canon for path --- src/libutil/file-system.cc | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/src/libutil/file-system.cc b/src/libutil/file-system.cc index 103fe11b80e..0f86dd2b139 100644 --- a/src/libutil/file-system.cc +++ b/src/libutil/file-system.cc @@ -626,7 +626,8 @@ Path makeTempPath(const Path & root, const Path & suffix) { // start the counter at a random value to minimize issues with preexisting temp paths static std::atomic_uint_fast32_t counter(std::random_device{}()); - return fmt("%1%%2%-%3%-%4%", root, suffix, getpid(), counter.fetch_add(1, std::memory_order_relaxed)); + auto tmpRoot = canonPath(root.empty() ? defaultTempDir() : root, true); + return fmt("%1%/%2%-%3%-%4%", tmpRoot, suffix, getpid(), counter.fetch_add(1, std::memory_order_relaxed)); } void createSymlink(const Path & target, const Path & link) From 6c85a90b87ed307303d70894ef8bf72d0285306c Mon Sep 17 00:00:00 2001 From: Farid Zakaria Date: Wed, 21 May 2025 10:07:04 -0700 Subject: [PATCH 048/218] Change to atomic --- src/libutil/file-system.cc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/libutil/file-system.cc b/src/libutil/file-system.cc index 0f86dd2b139..ad8cef38c8c 100644 --- a/src/libutil/file-system.cc +++ b/src/libutil/file-system.cc @@ -625,7 +625,7 @@ std::pair createTempFile(const Path & prefix) Path makeTempPath(const Path & root, const Path & suffix) { // start the counter at a random value to minimize issues with preexisting temp paths - static std::atomic_uint_fast32_t counter(std::random_device{}()); + static std::atomic counter(std::random_device{}()); auto tmpRoot = canonPath(root.empty() ? defaultTempDir() : root, true); return fmt("%1%/%2%-%3%-%4%", tmpRoot, suffix, getpid(), counter.fetch_add(1, std::memory_order_relaxed)); } From b9ed3ae36e362120a480fb3a916d0cbc437b377a Mon Sep 17 00:00:00 2001 From: Stefan Boca Date: Tue, 20 May 2025 22:01:56 -0700 Subject: [PATCH 049/218] nix-profile{,-daemon}.fish: fix do not source twice Commit b36637c8f7ab7a2b93c6eae1139ea1c672700186 set `__ETC_PROFILE_NIX_SOURCED` globally, but this is not enough to prevent the script from being run again by child shells, because the variable was not exported and thus not inherited by any child process. Exporting the variable also agrees with the bash scripts. Notably, the old behavior broke `nix develop -c fish` in some cases, because the profile bin directory got prepended to the path, causing binaries from the profile to override binareis from the devshell. --- scripts/nix-profile-daemon.fish.in | 2 +- scripts/nix-profile.fish.in | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/scripts/nix-profile-daemon.fish.in b/scripts/nix-profile-daemon.fish.in index f7cc5239e43..fffa06293e1 100644 --- a/scripts/nix-profile-daemon.fish.in +++ b/scripts/nix-profile-daemon.fish.in @@ -3,7 +3,7 @@ if test -z "$HOME" || test -n "$__ETC_PROFILE_NIX_SOURCED" exit end -set --global __ETC_PROFILE_NIX_SOURCED 1 +set --global --export __ETC_PROFILE_NIX_SOURCED 1 # Local helpers diff --git a/scripts/nix-profile.fish.in b/scripts/nix-profile.fish.in index f7cc5239e43..fffa06293e1 100644 --- a/scripts/nix-profile.fish.in +++ b/scripts/nix-profile.fish.in @@ -3,7 +3,7 @@ if test -z "$HOME" || test -n "$__ETC_PROFILE_NIX_SOURCED" exit end -set --global __ETC_PROFILE_NIX_SOURCED 1 +set --global --export __ETC_PROFILE_NIX_SOURCED 1 # Local helpers From 5e74c0e4d6825d3531e33424bf7182a37901ef5d Mon Sep 17 00:00:00 2001 From: Sergei Zimmerman Date: Sun, 18 May 2025 22:24:14 +0000 Subject: [PATCH 050/218] libexpr: Add `SampleStack` stack-sampling profiler MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit This patch adds support for a native stack sampling profiler to the evaluator, which saves a collapsed stack profile information to a configurable location. Introduced options (in `EvalSettings`): - `eval-profile-file` - path to the collected profile file. - `eval-profiler-frequency` - sampling frequency. - `eval-profiler` - enumeration option for enabling the profiler. Currently only `flamegraph` is supported, but having this an enumeration rather than a boolean switch leaves the door open for other profiler variants (e.g. tracy). Profile includes the following information on best-effort basis (e.g. some lambdas might have an undefined name). Callstack information contains: - Call site location (where the function gets called). - Primop/lambda name of the function being called. - Functors/partial applications don't have a name attached to them unlike special-cased primops and lambads. For cases where callsite location isn't available we have to resort to providing the location where the lambda itself is defined. This removes some of the confusing `«none»:0` locations in the profile from previous attempts. Example usage with piping directly into zstd for compression: ``` nix eval --no-eval-cache nixpkgs#nixosTests.gnome \ --eval-profiler flamegraph \ --eval-profile-file >(zstd -of nix.profile.zstd) ``` Co-authored-by: Jörg Thalheim --- src/libexpr/eval-profiler-settings.cc | 49 ++++ src/libexpr/eval-profiler.cc | 265 ++++++++++++++++++ src/libexpr/eval.cc | 12 +- .../nix/expr/eval-profiler-settings.hh | 16 ++ src/libexpr/include/nix/expr/eval-profiler.hh | 4 + src/libexpr/include/nix/expr/eval-settings.hh | 22 ++ src/libexpr/include/nix/expr/eval.hh | 2 +- src/libexpr/include/nix/expr/meson.build | 1 + src/libexpr/meson.build | 1 + 9 files changed, 369 insertions(+), 3 deletions(-) create mode 100644 src/libexpr/eval-profiler-settings.cc create mode 100644 src/libexpr/include/nix/expr/eval-profiler-settings.hh diff --git a/src/libexpr/eval-profiler-settings.cc b/src/libexpr/eval-profiler-settings.cc new file mode 100644 index 00000000000..1a35d4a2d11 --- /dev/null +++ b/src/libexpr/eval-profiler-settings.cc @@ -0,0 +1,49 @@ +#include "nix/expr/eval-profiler-settings.hh" +#include "nix/util/configuration.hh" +#include "nix/util/logging.hh" /* Needs to be included before config-impl.hh */ +#include "nix/util/config-impl.hh" +#include "nix/util/abstract-setting-to-json.hh" + +#include + +namespace nix { + +template<> +EvalProfilerMode BaseSetting::parse(const std::string & str) const +{ + if (str == "disabled") + return EvalProfilerMode::disabled; + else if (str == "flamegraph") + return EvalProfilerMode::flamegraph; + else + throw UsageError("option '%s' has invalid value '%s'", name, str); +} + +template<> +struct BaseSetting::trait +{ + static constexpr bool appendable = false; +}; + +template<> +std::string BaseSetting::to_string() const +{ + if (value == EvalProfilerMode::disabled) + return "disabled"; + else if (value == EvalProfilerMode::flamegraph) + return "flamegraph"; + else + unreachable(); +} + +NLOHMANN_JSON_SERIALIZE_ENUM( + EvalProfilerMode, + { + {EvalProfilerMode::disabled, "disabled"}, + {EvalProfilerMode::flamegraph, "flamegraph"}, + }); + +/* Explicit instantiation of templates */ +template class BaseSetting; + +} diff --git a/src/libexpr/eval-profiler.cc b/src/libexpr/eval-profiler.cc index 5374d7d9968..f08b46e6bf8 100644 --- a/src/libexpr/eval-profiler.cc +++ b/src/libexpr/eval-profiler.cc @@ -1,5 +1,7 @@ #include "nix/expr/eval-profiler.hh" #include "nix/expr/nixexpr.hh" +#include "nix/expr/eval.hh" +#include "nix/util/lru-cache.hh" namespace nix { @@ -45,4 +47,267 @@ void MultiEvalProfiler::addProfiler(ref profiler) invalidateNeededHooks(); } +namespace { + +class PosCache : private LRUCache +{ + const EvalState & state; + +public: + PosCache(const EvalState & state) + : LRUCache(524288) /* ~40MiB */ + , state(state) + { + } + + Pos lookup(PosIdx posIdx) + { + auto posOrNone = LRUCache::get(posIdx); + if (posOrNone) + return *posOrNone; + + auto pos = state.positions[posIdx]; + upsert(posIdx, pos); + return pos; + } +}; + +struct LambdaFrameInfo +{ + ExprLambda * expr; + /** Position where the lambda has been called. */ + PosIdx callPos = noPos; + std::ostream & symbolize(const EvalState & state, std::ostream & os, PosCache & posCache) const; + auto operator<=>(const LambdaFrameInfo & rhs) const = default; +}; + +/** Primop call. */ +struct PrimOpFrameInfo +{ + const PrimOp * expr; + /** Position where the primop has been called. */ + PosIdx callPos = noPos; + std::ostream & symbolize(const EvalState & state, std::ostream & os, PosCache & posCache) const; + auto operator<=>(const PrimOpFrameInfo & rhs) const = default; +}; + +/** Used for functor calls (attrset with __functor attr). */ +struct FunctorFrameInfo +{ + PosIdx pos; + std::ostream & symbolize(const EvalState & state, std::ostream & os, PosCache & posCache) const; + auto operator<=>(const FunctorFrameInfo & rhs) const = default; +}; + +/** Fallback frame info. */ +struct GenericFrameInfo +{ + PosIdx pos; + std::ostream & symbolize(const EvalState & state, std::ostream & os, PosCache & posCache) const; + auto operator<=>(const GenericFrameInfo & rhs) const = default; +}; + +using FrameInfo = std::variant; +using FrameStack = std::vector; + +/** + * Stack sampling profiler. + */ +class SampleStack : public EvalProfiler +{ + /* How often stack profiles should be flushed to file. This avoids the need + to persist stack samples across the whole evaluation at the cost + of periodically flushing data to disk. */ + static constexpr std::chrono::microseconds profileDumpInterval = std::chrono::milliseconds(2000); + + Hooks getNeededHooksImpl() const override + { + return Hooks().set(preFunctionCall).set(postFunctionCall); + } + +public: + SampleStack(const EvalState & state, std::filesystem::path profileFile, std::chrono::nanoseconds period) + : state(state) + , sampleInterval(period) + , profileFd([&]() { + AutoCloseFD fd = toDescriptor(open(profileFile.c_str(), O_WRONLY | O_CREAT | O_TRUNC, 0660)); + if (!fd) + throw SysError("opening file %s", profileFile); + return fd; + }()) + , posCache(state) + { + } + + [[gnu::noinline]] void + preFunctionCallHook(const EvalState & state, const Value & v, std::span args, const PosIdx pos) override; + [[gnu::noinline]] void + postFunctionCallHook(const EvalState & state, const Value & v, std::span args, const PosIdx pos) override; + + void maybeSaveProfile(std::chrono::time_point now); + void saveProfile(); + FrameInfo getFrameInfoFromValueAndPos(const Value & v, PosIdx pos); + + SampleStack(SampleStack &&) = default; + SampleStack & operator=(SampleStack &&) = delete; + SampleStack(const SampleStack &) = delete; + SampleStack & operator=(const SampleStack &) = delete; + ~SampleStack(); + +private: + /** Hold on to an instance of EvalState for symbolizing positions. */ + const EvalState & state; + std::chrono::nanoseconds sampleInterval; + AutoCloseFD profileFd; + FrameStack stack; + std::map callCount; + std::chrono::time_point lastStackSample = + std::chrono::high_resolution_clock::now(); + std::chrono::time_point lastDump = std::chrono::high_resolution_clock::now(); + PosCache posCache; +}; + +FrameInfo SampleStack::getFrameInfoFromValueAndPos(const Value & v, PosIdx pos) +{ + /* NOTE: No actual references to garbage collected values are not held in + the profiler. */ + if (v.isLambda()) + return LambdaFrameInfo{.expr = v.payload.lambda.fun, .callPos = pos}; + else if (v.isPrimOp()) + return PrimOpFrameInfo{.expr = v.primOp(), .callPos = pos}; + else if (v.isPrimOpApp()) + /* Resolve primOp eagerly. Must not hold on to a reference to a Value. */ + return PrimOpFrameInfo{.expr = v.primOpAppPrimOp(), .callPos = pos}; + else if (state.isFunctor(v)) { + const auto functor = v.attrs()->get(state.sFunctor); + if (auto pos_ = posCache.lookup(pos); std::holds_alternative(pos_.origin)) + /* HACK: In case callsite position is unresolved. */ + return FunctorFrameInfo{.pos = functor->pos}; + return FunctorFrameInfo{.pos = pos}; + } else + /* NOTE: Add a stack frame even for invalid cases (e.g. when calling a non-function). This is what + * trace-function-calls does. */ + return GenericFrameInfo{.pos = pos}; +} + +[[gnu::noinline]] void SampleStack::preFunctionCallHook( + const EvalState & state, const Value & v, [[maybe_unused]] std::span args, const PosIdx pos) +{ + stack.push_back(getFrameInfoFromValueAndPos(v, pos)); + + auto now = std::chrono::high_resolution_clock::now(); + + if (now - lastStackSample > sampleInterval) { + callCount[stack] += 1; + lastStackSample = now; + } + + /* Do this in preFunctionCallHook because we might throw an exception, but + callFunction uses Finally, which doesn't play well with exceptions. */ + maybeSaveProfile(now); +} + +[[gnu::noinline]] void +SampleStack::postFunctionCallHook(const EvalState & state, const Value & v, std::span args, const PosIdx pos) +{ + + if (!stack.empty()) + stack.pop_back(); +} + +std::ostream & LambdaFrameInfo::symbolize(const EvalState & state, std::ostream & os, PosCache & posCache) const +{ + if (auto pos = posCache.lookup(callPos); std::holds_alternative(pos.origin)) + /* HACK: To avoid dubious «none»:0 in the generated profile if the origin can't be resolved + resort to printing the lambda location instead of the callsite position. */ + os << posCache.lookup(expr->getPos()); + else + os << pos; + if (expr->name) + os << ":" << state.symbols[expr->name]; + return os; +} + +std::ostream & GenericFrameInfo::symbolize(const EvalState & state, std::ostream & os, PosCache & posCache) const +{ + os << posCache.lookup(pos); + return os; +} + +std::ostream & FunctorFrameInfo::symbolize(const EvalState & state, std::ostream & os, PosCache & posCache) const +{ + os << posCache.lookup(pos) << ":functor"; + return os; +} + +std::ostream & PrimOpFrameInfo::symbolize(const EvalState & state, std::ostream & os, PosCache & posCache) const +{ + /* Sometimes callsite position can have an unresolved origin, which + leads to confusing «none»:0 locations in the profile. */ + auto pos = posCache.lookup(callPos); + if (!std::holds_alternative(pos.origin)) + os << posCache.lookup(callPos) << ":"; + os << *expr; + return os; +} + +void SampleStack::maybeSaveProfile(std::chrono::time_point now) +{ + if (now - lastDump >= profileDumpInterval) + saveProfile(); + else + return; + + /* Save the last dump timepoint. Do this after actually saving data to file + to not account for the time doing the flushing to disk. */ + lastDump = std::chrono::high_resolution_clock::now(); + + /* Free up memory used for stack sampling. This might be very significant for + long-running evaluations, so we shouldn't hog too much memory. */ + callCount.clear(); +} + +void SampleStack::saveProfile() +{ + auto os = std::ostringstream{}; + for (auto & [stack, count] : callCount) { + auto first = true; + for (auto & pos : stack) { + if (first) + first = false; + else + os << ";"; + + std::visit([&](auto && info) { info.symbolize(state, os, posCache); }, pos); + } + os << " " << count; + writeLine(profileFd.get(), std::move(os).str()); + /* Clear ostringstream. */ + os.str(""); + os.clear(); + } +} + +SampleStack::~SampleStack() +{ + /* Guard against cases when we are already unwinding the stack. */ + try { + saveProfile(); + } catch (...) { + ignoreExceptionInDestructor(); + } +} + +} // namespace + +ref +makeSampleStackProfiler(const EvalState & state, std::filesystem::path profileFile, uint64_t frequency) +{ + /* 0 is a special value for sampling stack after each call. */ + std::chrono::nanoseconds period = frequency == 0 + ? std::chrono::nanoseconds{0} + : std::chrono::nanoseconds{std::nano::den / frequency / std::nano::num}; + return make_ref(state, profileFile, period); +} + } diff --git a/src/libexpr/eval.cc b/src/libexpr/eval.cc index 1b87cf42f7d..ff7df249a02 100644 --- a/src/libexpr/eval.cc +++ b/src/libexpr/eval.cc @@ -376,8 +376,16 @@ EvalState::EvalState( /* Register function call tracer. */ if (settings.traceFunctionCalls) profiler.addProfiler(make_ref()); -} + switch (settings.evalProfilerMode) { + case EvalProfilerMode::flamegraph: + profiler.addProfiler(makeSampleStackProfiler( + *this, settings.evalProfileFile.get(), settings.evalProfilerFrequency)); + break; + case EvalProfilerMode::disabled: + break; + } +} EvalState::~EvalState() { @@ -2236,7 +2244,7 @@ bool EvalState::forceBool(Value & v, const PosIdx pos, std::string_view errorCtx } -bool EvalState::isFunctor(Value & fun) +bool EvalState::isFunctor(const Value & fun) const { return fun.type() == nAttrs && fun.attrs()->find(sFunctor) != fun.attrs()->end(); } diff --git a/src/libexpr/include/nix/expr/eval-profiler-settings.hh b/src/libexpr/include/nix/expr/eval-profiler-settings.hh new file mode 100644 index 00000000000..a94cde042ea --- /dev/null +++ b/src/libexpr/include/nix/expr/eval-profiler-settings.hh @@ -0,0 +1,16 @@ +#pragma once +///@file + +#include "nix/util/configuration.hh" + +namespace nix { + +enum struct EvalProfilerMode { disabled, flamegraph }; + +template<> +EvalProfilerMode BaseSetting::parse(const std::string & str) const; + +template<> +std::string BaseSetting::to_string() const; + +} diff --git a/src/libexpr/include/nix/expr/eval-profiler.hh b/src/libexpr/include/nix/expr/eval-profiler.hh index 763b737f7a9..9fee4ef3559 100644 --- a/src/libexpr/include/nix/expr/eval-profiler.hh +++ b/src/libexpr/include/nix/expr/eval-profiler.hh @@ -11,6 +11,7 @@ #include #include #include +#include namespace nix { @@ -110,4 +111,7 @@ public: postFunctionCallHook(const EvalState & state, const Value & v, std::span args, const PosIdx pos) override; }; +ref +makeSampleStackProfiler(const EvalState & state, std::filesystem::path profileFile, uint64_t frequency); + } diff --git a/src/libexpr/include/nix/expr/eval-settings.hh b/src/libexpr/include/nix/expr/eval-settings.hh index 8d3db59b3bb..f1367b9bb28 100644 --- a/src/libexpr/include/nix/expr/eval-settings.hh +++ b/src/libexpr/include/nix/expr/eval-settings.hh @@ -1,6 +1,7 @@ #pragma once ///@file +#include "nix/expr/eval-profiler-settings.hh" #include "nix/util/configuration.hh" #include "nix/util/source-path.hh" @@ -193,6 +194,27 @@ struct EvalSettings : Config `flamegraph.pl`. )"}; + Setting evalProfilerMode{this, EvalProfilerMode::disabled, "eval-profiler", + R"( + Enables evaluation profiling. The following modes are supported: + + * `flamegraph` stack sampling profiler. Outputs folded format, one line per stack (suitable for `flamegraph.pl` and compatible tools). + + Use [`eval-profile-file`](#conf-eval-profile-file) to specify where the profile is saved. + )"}; + + Setting evalProfileFile{this, "nix.profile", "eval-profile-file", + R"( + Specifies the file where [evaluation profile](#conf-eval-profiler) is saved. + )"}; + + Setting evalProfilerFrequency{this, 99, "eval-profiler-frequency", + R"( + Specifies the sampling rate in hertz for sampling evaluation profilers. + Use `0` to sample the stack after each function call. + See [`eval-profiler`](#conf-eval-profiler). + )"}; + Setting useEvalCache{this, true, "eval-cache", R"( Whether to use the flake evaluation cache. diff --git a/src/libexpr/include/nix/expr/eval.hh b/src/libexpr/include/nix/expr/eval.hh index ffbc84bcdaf..de865ae7bc4 100644 --- a/src/libexpr/include/nix/expr/eval.hh +++ b/src/libexpr/include/nix/expr/eval.hh @@ -731,7 +731,7 @@ public: */ void assertEqValues(Value & v1, Value & v2, const PosIdx pos, std::string_view errorCtx); - bool isFunctor(Value & fun); + bool isFunctor(const Value & fun) const; void callFunction(Value & fun, std::span args, Value & vRes, const PosIdx pos); diff --git a/src/libexpr/include/nix/expr/meson.build b/src/libexpr/include/nix/expr/meson.build index db902a61676..333490ee499 100644 --- a/src/libexpr/include/nix/expr/meson.build +++ b/src/libexpr/include/nix/expr/meson.build @@ -14,6 +14,7 @@ headers = [config_pub_h] + files( 'eval-error.hh', 'eval-gc.hh', 'eval-inline.hh', + 'eval-profiler-settings.hh', 'eval-profiler.hh', 'eval-settings.hh', 'eval.hh', diff --git a/src/libexpr/meson.build b/src/libexpr/meson.build index dc50d2b1935..f5adafae031 100644 --- a/src/libexpr/meson.build +++ b/src/libexpr/meson.build @@ -140,6 +140,7 @@ sources = files( 'eval-cache.cc', 'eval-error.cc', 'eval-gc.cc', + 'eval-profiler-settings.cc', 'eval-profiler.cc', 'eval-settings.cc', 'eval.cc', From 33141cd13394945985eb292e80b6e41eeeebc14e Mon Sep 17 00:00:00 2001 From: Sergei Zimmerman Date: Sun, 18 May 2025 22:52:48 +0000 Subject: [PATCH 051/218] tests/functional: Add tests for flamegraph profiler The tests are mostly based on existing `function-trace.sh` with some tests for corner cases. --- tests/functional/flamegraph-profiler.sh | 91 +++++++++++++++++++++++++ tests/functional/meson.build | 1 + 2 files changed, 92 insertions(+) create mode 100755 tests/functional/flamegraph-profiler.sh diff --git a/tests/functional/flamegraph-profiler.sh b/tests/functional/flamegraph-profiler.sh new file mode 100755 index 00000000000..0c35037a8d0 --- /dev/null +++ b/tests/functional/flamegraph-profiler.sh @@ -0,0 +1,91 @@ +#!/usr/bin/env bash + +source common.sh + +set +x + +expect_trace() { + expr="$1" + expect="$2" + actual=$( + nix-instantiate \ + --eval-profiler flamegraph \ + --eval-profiler-frequency 0 \ + --eval-profile-file /dev/stdout \ + --expr "$expr" | + grep "«string»" || true + ) + + echo -n "Tracing expression '$expr'" + msg=$( + diff -swB \ + <(echo "$expect") \ + <(echo "$actual") + ) && result=0 || result=$? + if [ "$result" -eq 0 ]; then + echo " ok." + else + echo " failed. difference:" + echo "$msg" + return "$result" + fi +} + +# lambda +expect_trace 'let f = arg: arg; in f 1' " +«string»:1:22:f 1 +" + +# unnamed lambda +expect_trace '(arg: arg) 1' " +«string»:1:1 1 +" + +# primop +expect_trace 'builtins.head [0 1]' " +«string»:1:1:primop head 1 +" + +# primop application +expect_trace 'let a = builtins.all (let f = x: x; in f); in a [1]' " +«string»:1:9:primop all 1 +«string»:1:47:primop all 1 +«string»:1:47:primop all;«string»:1:31:f 1 +" + +# functor +expect_trace '{__functor = x: arg: arg;} 1' " +«string»:1:1:functor 1 +«string»:1:1:functor;«string»:1:2 1 +" + +# failure inside a tryEval +expect_trace 'builtins.tryEval (throw "example")' " +«string»:1:1:primop tryEval 1 +«string»:1:1:primop tryEval;«string»:1:19:primop throw 1 +" + +# Missing argument to a formal function +expect_trace 'let f = ({ x }: x); in f { }' " +«string»:1:24:f 1 +" + +# Too many arguments to a formal function +expect_trace 'let f = ({ x }: x); in f { x = "x"; y = "y"; }' " +«string»:1:24:f 1 +" + +# Not enough arguments to a lambda +expect_trace 'let f = (x: y: x + y); in f 1' " +«string»:1:27:f 1 +" + +# Too many arguments to a lambda +expect_trace 'let f2 = (x: x); in f2 1 2' " +«string»:1:21:f2 1 +" + +# Not a function +expect_trace '1 2' " +«string»:1:1 1 +" diff --git a/tests/functional/meson.build b/tests/functional/meson.build index b2005d9d9ed..f5a19ac646a 100644 --- a/tests/functional/meson.build +++ b/tests/functional/meson.build @@ -133,6 +133,7 @@ suites = [ 'post-hook.sh', 'function-trace.sh', 'formatter.sh', + 'flamegraph-profiler.sh', 'eval-store.sh', 'why-depends.sh', 'derivation-json.sh', From 57348b677b50bf2696f76fea0b62eefe41d18c96 Mon Sep 17 00:00:00 2001 From: John Ericson Date: Wed, 14 May 2025 13:35:37 -0400 Subject: [PATCH 052/218] Restore dynamic derivations! This method does *not* create a new type of goal. We instead just make `DerivationGoal` more sophisticated, which is much easier to do now that `DerivationBuildingGoal` has been split from it (and so many fields are gone, or or local variables instead). This avoids the need for a secondarily trampoline goal that interacted poorly with `addWantedOutputs`. That, I hope, will mean the bugs from before do not reappear. There may in fact be a reason to introduce such a trampoline in the future, but it would only happen in conjunction with getting rid of `addWantedOutputs`. Restores the functionality (and tests) that was reverted in f4f28cdd0e01a9bfb0901e8a53ead719265fa9b8. --- .../build/derivation-building-goal.cc | 4 +- src/libstore/build/derivation-goal.cc | 100 ++++++++++++------ src/libstore/build/entry-points.cc | 2 +- src/libstore/build/worker.cc | 57 ++++++---- src/libstore/derived-path-map.cc | 4 + .../nix/store/build/derivation-goal.hh | 19 ++-- .../include/nix/store/build/worker.hh | 9 +- .../include/nix/store/derived-path-map.hh | 7 +- tests/functional/dyn-drv/build-built-drv.sh | 7 +- tests/functional/dyn-drv/dep-built-drv-2.sh | 4 +- tests/functional/dyn-drv/dep-built-drv.sh | 7 +- tests/functional/dyn-drv/failing-outer.sh | 4 +- 12 files changed, 146 insertions(+), 78 deletions(-) diff --git a/src/libstore/build/derivation-building-goal.cc b/src/libstore/build/derivation-building-goal.cc index 4169903f654..14c427b83a0 100644 --- a/src/libstore/build/derivation-building-goal.cc +++ b/src/libstore/build/derivation-building-goal.cc @@ -296,7 +296,7 @@ Goal::Co DerivationBuildingGoal::gaveUpOnSubstitution() // FIXME wanted outputs auto resolvedDrvGoal = worker.makeDerivationGoal( - pathResolved, OutputsSpec::All{}, buildMode); + makeConstantStorePathRef(pathResolved), OutputsSpec::All{}, buildMode); { Goals waitees{resolvedDrvGoal}; co_await await(std::move(waitees)); @@ -338,7 +338,7 @@ Goal::Co DerivationBuildingGoal::gaveUpOnSubstitution() throw Error( "derivation '%s' doesn't have expected output '%s' (derivation-goal.cc/realisation)", - worker.store.printStorePath(resolvedDrvGoal->drvPath), outputName); + resolvedDrvGoal->drvReq->to_string(worker.store), outputName); }(); if (!drv->type().isImpure()) { diff --git a/src/libstore/build/derivation-goal.cc b/src/libstore/build/derivation-goal.cc index 5c8b75cb9de..63dca308344 100644 --- a/src/libstore/build/derivation-goal.cc +++ b/src/libstore/build/derivation-goal.cc @@ -24,16 +24,16 @@ namespace nix { -DerivationGoal::DerivationGoal(const StorePath & drvPath, +DerivationGoal::DerivationGoal(ref drvReq, const OutputsSpec & wantedOutputs, Worker & worker, BuildMode buildMode) : Goal(worker, loadDerivation()) - , drvPath(drvPath) + , drvReq(drvReq) , wantedOutputs(wantedOutputs) , buildMode(buildMode) { name = fmt( "building of '%s' from .drv file", - DerivedPath::Built { makeConstantStorePathRef(drvPath), wantedOutputs }.to_string(worker.store)); + DerivedPath::Built { drvReq, wantedOutputs }.to_string(worker.store)); trace("created"); mcExpectedBuilds = std::make_unique>(worker.expectedBuilds); @@ -43,8 +43,8 @@ DerivationGoal::DerivationGoal(const StorePath & drvPath, DerivationGoal::DerivationGoal(const StorePath & drvPath, const BasicDerivation & drv, const OutputsSpec & wantedOutputs, Worker & worker, BuildMode buildMode) - : Goal(worker, haveDerivation()) - , drvPath(drvPath) + : Goal(worker, haveDerivation(drvPath)) + , drvReq(makeConstantStorePathRef(drvPath)) , wantedOutputs(wantedOutputs) , buildMode(buildMode) { @@ -52,7 +52,7 @@ DerivationGoal::DerivationGoal(const StorePath & drvPath, const BasicDerivation name = fmt( "building of '%s' from in-memory derivation", - DerivedPath::Built { makeConstantStorePathRef(drvPath), drv.outputNames() }.to_string(worker.store)); + DerivedPath::Built { drvReq, drv.outputNames() }.to_string(worker.store)); trace("created"); mcExpectedBuilds = std::make_unique>(worker.expectedBuilds); @@ -61,13 +61,24 @@ DerivationGoal::DerivationGoal(const StorePath & drvPath, const BasicDerivation } +static StorePath pathPartOfReq(const SingleDerivedPath & req) +{ + return std::visit( + overloaded{ + [&](const SingleDerivedPath::Opaque & bo) { return bo.path; }, + [&](const SingleDerivedPath::Built & bfd) { return pathPartOfReq(*bfd.drvPath); }, + }, + req.raw()); +} + + std::string DerivationGoal::key() { /* Ensure that derivations get built in order of their name, i.e. a derivation named "aardvark" always comes before "baboon". And substitution goals always happen before derivation goals (due to "b$"). */ - return "b$" + std::string(drvPath.name()) + "$" + worker.store.printStorePath(drvPath); + return "b$" + std::string(pathPartOfReq(*drvReq).name()) + "$" + drvReq->to_string(worker.store); } @@ -93,24 +104,46 @@ void DerivationGoal::addWantedOutputs(const OutputsSpec & outputs) Goal::Co DerivationGoal::loadDerivation() { - trace("local derivation"); + trace("need to load derivation from file"); { /* The first thing to do is to make sure that the derivation - exists. If it doesn't, it may be created through a - substitute. */ - - if (buildMode != bmNormal || !worker.evalStore.isValidPath(drvPath)) { - Goals waitees{upcast_goal(worker.makePathSubstitutionGoal(drvPath))}; + exists. If it doesn't, it may be built from another + derivation, or merely substituted. We can make goal to get it + and not worry about which method it takes to get the + derivation. */ + + if (auto optDrvPath = [this]() -> std::optional { + if (buildMode != bmNormal) + return std::nullopt; + + auto drvPath = StorePath::dummy; + try { + drvPath = resolveDerivedPath(worker.store, *drvReq); + } catch (MissingRealisation &) { + return std::nullopt; + } + auto cond = worker.evalStore.isValidPath(drvPath) || worker.store.isValidPath(drvPath); + return cond ? std::optional{drvPath} : std::nullopt; + }()) { + trace( + fmt("already have drv '%s' for '%s', can go straight to building", + worker.store.printStorePath(*optDrvPath), + drvReq->to_string(worker.store))); + } else { + trace("need to obtain drv we want to build"); + Goals waitees{worker.makeGoal(DerivedPath::fromSingle(*drvReq))}; co_await await(std::move(waitees)); } trace("loading derivation"); if (nrFailed != 0) { - co_return done(BuildResult::MiscFailure, {}, Error("cannot build missing derivation '%s'", worker.store.printStorePath(drvPath))); + co_return amDone(ecFailed, Error("cannot build missing derivation '%s'", drvReq->to_string(worker.store))); } + StorePath drvPath = resolveDerivedPath(worker.store, *drvReq); + /* `drvPath' should already be a root, but let's be on the safe side: if the user forgot to make it a root, we wouldn't want things being garbage collected while we're busy. */ @@ -129,13 +162,13 @@ Goal::Co DerivationGoal::loadDerivation() { } } assert(drv); - } - co_return haveDerivation(); + co_return haveDerivation(drvPath); + } } -Goal::Co DerivationGoal::haveDerivation() +Goal::Co DerivationGoal::haveDerivation(StorePath drvPath) { trace("have derivation"); @@ -221,11 +254,11 @@ Goal::Co DerivationGoal::haveDerivation() { /* Check what outputs paths are not already valid. */ - auto [allValid, validOutputs] = checkPathValidity(); + auto [allValid, validOutputs] = checkPathValidity(drvPath); /* If they are all valid, then we're done. */ if (allValid && buildMode == bmNormal) { - co_return done(BuildResult::AlreadyValid, std::move(validOutputs)); + co_return done(drvPath, BuildResult::AlreadyValid, std::move(validOutputs)); } } @@ -262,7 +295,7 @@ Goal::Co DerivationGoal::haveDerivation() assert(!drv->type().isImpure()); if (nrFailed > 0 && nrFailed > nrNoSubstituters && !settings.tryFallback) { - co_return done(BuildResult::TransientFailure, {}, + co_return done(drvPath, BuildResult::TransientFailure, {}, Error("some substitutes for the outputs of derivation '%s' failed (usually happens due to networking issues); try '--fallback' to build derivation from source ", worker.store.printStorePath(drvPath))); } @@ -271,16 +304,16 @@ Goal::Co DerivationGoal::haveDerivation() if (needRestart == NeedRestartForMoreOutputs::OutputsAddedDoNeed) { needRestart = NeedRestartForMoreOutputs::OutputsUnmodifedDontNeed; - co_return haveDerivation(); + co_return haveDerivation(std::move(drvPath)); } - auto [allValid, validOutputs] = checkPathValidity(); + auto [allValid, validOutputs] = checkPathValidity(drvPath); if (buildMode == bmNormal && allValid) { - co_return done(BuildResult::Substituted, std::move(validOutputs)); + co_return done(drvPath, BuildResult::Substituted, std::move(validOutputs)); } if (buildMode == bmRepair && allValid) { - co_return repairClosure(); + co_return repairClosure(std::move(drvPath)); } if (buildMode == bmCheck && !allValid) throw Error("some outputs of '%s' are not valid, so checking is not possible", @@ -303,7 +336,7 @@ struct value_comparison }; -Goal::Co DerivationGoal::repairClosure() +Goal::Co DerivationGoal::repairClosure(StorePath drvPath) { assert(!drv->type().isImpure()); @@ -313,7 +346,7 @@ Goal::Co DerivationGoal::repairClosure() that produced those outputs. */ /* Get the output closure. */ - auto outputs = queryDerivationOutputMap(); + auto outputs = queryDerivationOutputMap(drvPath); StorePathSet outputClosure; for (auto & i : outputs) { if (!wantedOutputs.contains(i.first)) continue; @@ -371,11 +404,11 @@ Goal::Co DerivationGoal::repairClosure() throw Error("some paths in the output closure of derivation '%s' could not be repaired", worker.store.printStorePath(drvPath)); } - co_return done(BuildResult::AlreadyValid, assertPathValidity()); + co_return done(drvPath, BuildResult::AlreadyValid, assertPathValidity(drvPath)); } -std::map> DerivationGoal::queryPartialDerivationOutputMap() +std::map> DerivationGoal::queryPartialDerivationOutputMap(const StorePath & drvPath) { assert(!drv->type().isImpure()); @@ -391,7 +424,7 @@ std::map> DerivationGoal::queryPartialDeri return res; } -OutputPathMap DerivationGoal::queryDerivationOutputMap() +OutputPathMap DerivationGoal::queryDerivationOutputMap(const StorePath & drvPath) { assert(!drv->type().isImpure()); @@ -407,7 +440,7 @@ OutputPathMap DerivationGoal::queryDerivationOutputMap() } -std::pair DerivationGoal::checkPathValidity() +std::pair DerivationGoal::checkPathValidity(const StorePath & drvPath) { if (drv->type().isImpure()) return { false, {} }; @@ -422,7 +455,7 @@ std::pair DerivationGoal::checkPathValidity() }, wantedOutputs.raw); SingleDrvOutputs validOutputs; - for (auto & i : queryPartialDerivationOutputMap()) { + for (auto & i : queryPartialDerivationOutputMap(drvPath)) { auto initialOutput = get(initialOutputs, i.first); if (!initialOutput) // this is an invalid output, gets catched with (!wantedOutputsLeft.empty()) @@ -487,9 +520,9 @@ std::pair DerivationGoal::checkPathValidity() } -SingleDrvOutputs DerivationGoal::assertPathValidity() +SingleDrvOutputs DerivationGoal::assertPathValidity(const StorePath & drvPath) { - auto [allValid, validOutputs] = checkPathValidity(); + auto [allValid, validOutputs] = checkPathValidity(drvPath); if (!allValid) throw Error("some outputs are unexpectedly invalid"); return validOutputs; @@ -497,6 +530,7 @@ SingleDrvOutputs DerivationGoal::assertPathValidity() Goal::Done DerivationGoal::done( + const StorePath & drvPath, BuildResult::Status status, SingleDrvOutputs builtOutputs, std::optional ex) diff --git a/src/libstore/build/entry-points.cc b/src/libstore/build/entry-points.cc index c934b0704ce..39fd471c4b2 100644 --- a/src/libstore/build/entry-points.cc +++ b/src/libstore/build/entry-points.cc @@ -30,7 +30,7 @@ void Store::buildPaths(const std::vector & reqs, BuildMode buildMod if (i->exitCode != Goal::ecSuccess) { #ifndef _WIN32 // TODO Enable building on Windows if (auto i2 = dynamic_cast(i.get())) - failed.insert(printStorePath(i2->drvPath)); + failed.insert(i2->drvReq->to_string(*this)); else #endif if (auto i2 = dynamic_cast(i.get())) diff --git a/src/libstore/build/worker.cc b/src/libstore/build/worker.cc index aaf4c149fbe..12d4aaa2d74 100644 --- a/src/libstore/build/worker.cc +++ b/src/libstore/build/worker.cc @@ -54,11 +54,11 @@ std::shared_ptr Worker::initGoalIfNeeded(std::weak_ptr & goal_weak, Args & } std::shared_ptr Worker::makeDerivationGoalCommon( - const StorePath & drvPath, + ref drvReq, const OutputsSpec & wantedOutputs, std::function()> mkDrvGoal) { - std::weak_ptr & goal_weak = derivationGoals[drvPath]; + std::weak_ptr & goal_weak = derivationGoals.ensureSlot(*drvReq).value; std::shared_ptr goal = goal_weak.lock(); if (!goal) { goal = mkDrvGoal(); @@ -71,18 +71,18 @@ std::shared_ptr Worker::makeDerivationGoalCommon( } -std::shared_ptr Worker::makeDerivationGoal(const StorePath & drvPath, +std::shared_ptr Worker::makeDerivationGoal(ref drvReq, const OutputsSpec & wantedOutputs, BuildMode buildMode) { - return makeDerivationGoalCommon(drvPath, wantedOutputs, [&]() -> std::shared_ptr { - return std::make_shared(drvPath, wantedOutputs, *this, buildMode); + return makeDerivationGoalCommon(drvReq, wantedOutputs, [&]() -> std::shared_ptr { + return std::make_shared(drvReq, wantedOutputs, *this, buildMode); }); } std::shared_ptr Worker::makeBasicDerivationGoal(const StorePath & drvPath, const BasicDerivation & drv, const OutputsSpec & wantedOutputs, BuildMode buildMode) { - return makeDerivationGoalCommon(drvPath, wantedOutputs, [&]() -> std::shared_ptr { + return makeDerivationGoalCommon(makeConstantStorePathRef(drvPath), wantedOutputs, [&]() -> std::shared_ptr { return std::make_shared(drvPath, drv, wantedOutputs, *this, buildMode); }); } @@ -118,10 +118,7 @@ GoalPtr Worker::makeGoal(const DerivedPath & req, BuildMode buildMode) { return std::visit(overloaded { [&](const DerivedPath::Built & bfd) -> GoalPtr { - if (auto bop = std::get_if(&*bfd.drvPath)) - return makeDerivationGoal(bop->path, bfd.outputs, buildMode); - else - throw UnimplementedError("Building dynamic derivations in one shot is not yet implemented."); + return makeDerivationGoal(bfd.drvPath, bfd.outputs, buildMode); }, [&](const DerivedPath::Opaque & bo) -> GoalPtr { return makePathSubstitutionGoal(bo.path, buildMode == bmRepair ? Repair : NoRepair); @@ -130,25 +127,45 @@ GoalPtr Worker::makeGoal(const DerivedPath & req, BuildMode buildMode) } +template +static void cullMap(std::map & goalMap, F f) +{ + for (auto i = goalMap.begin(); i != goalMap.end();) + if (!f(i->second)) + i = goalMap.erase(i); + else ++i; +} + + template static void removeGoal(std::shared_ptr goal, std::map> & goalMap) { /* !!! inefficient */ - for (auto i = goalMap.begin(); - i != goalMap.end(); ) - if (i->second.lock() == goal) { - auto j = i; ++j; - goalMap.erase(i); - i = j; - } - else ++i; + cullMap(goalMap, [&](const std::weak_ptr & gp) -> bool { + return gp.lock() != goal; + }); +} + +template +static void removeGoal(std::shared_ptr goal, std::map>::ChildNode> & goalMap); + +template +static void removeGoal(std::shared_ptr goal, std::map>::ChildNode> & goalMap) +{ + /* !!! inefficient */ + cullMap(goalMap, [&](DerivedPathMap>::ChildNode & node) -> bool { + if (node.value.lock() == goal) + node.value.reset(); + removeGoal(goal, node.childMap); + return !node.value.expired() || !node.childMap.empty(); + }); } void Worker::removeGoal(GoalPtr goal) { if (auto drvGoal = std::dynamic_pointer_cast(goal)) - nix::removeGoal(drvGoal, derivationGoals); + nix::removeGoal(drvGoal, derivationGoals.map); else if (auto drvBuildingGoal = std::dynamic_pointer_cast(goal)) nix::removeGoal(drvBuildingGoal, derivationBuildingGoals); else if (auto subGoal = std::dynamic_pointer_cast(goal)) @@ -297,7 +314,7 @@ void Worker::run(const Goals & _topGoals) topGoals.insert(i); if (auto goal = dynamic_cast(i.get())) { topPaths.push_back(DerivedPath::Built { - .drvPath = makeConstantStorePathRef(goal->drvPath), + .drvPath = goal->drvReq, .outputs = goal->wantedOutputs, }); } else diff --git a/src/libstore/derived-path-map.cc b/src/libstore/derived-path-map.cc index b785dddd93a..408d1a6b98f 100644 --- a/src/libstore/derived-path-map.cc +++ b/src/libstore/derived-path-map.cc @@ -52,6 +52,7 @@ typename DerivedPathMap::ChildNode * DerivedPathMap::findSlot(const Single // instantiations +#include "nix/store/build/derivation-goal.hh" namespace nix { template<> @@ -68,4 +69,7 @@ std::strong_ordering DerivedPathMap::ChildNode::operator <=> ( template struct DerivedPathMap::ChildNode; template struct DerivedPathMap; +template struct DerivedPathMap>; + + }; diff --git a/src/libstore/include/nix/store/build/derivation-goal.hh b/src/libstore/include/nix/store/build/derivation-goal.hh index 7118fd4a544..b422aec7ab5 100644 --- a/src/libstore/include/nix/store/build/derivation-goal.hh +++ b/src/libstore/include/nix/store/build/derivation-goal.hh @@ -27,7 +27,7 @@ void runPostBuildHook( struct DerivationGoal : public Goal { /** The path of the derivation. */ - StorePath drvPath; + ref drvReq; /** * The specific outputs that we need to build. @@ -62,7 +62,7 @@ struct DerivationGoal : public Goal NeedRestartForMoreOutputs needRestart = NeedRestartForMoreOutputs::OutputsUnmodifedDontNeed; /** - * The derivation stored at drvPath. + * The derivation stored at `drvReq`. */ std::unique_ptr drv; @@ -76,7 +76,7 @@ struct DerivationGoal : public Goal std::unique_ptr> mcExpectedBuilds; - DerivationGoal(const StorePath & drvPath, + DerivationGoal(ref drvReq, const OutputsSpec & wantedOutputs, Worker & worker, BuildMode buildMode = bmNormal); DerivationGoal(const StorePath & drvPath, const BasicDerivation & drv, @@ -97,15 +97,15 @@ struct DerivationGoal : public Goal * The states. */ Co loadDerivation(); - Co haveDerivation(); + Co haveDerivation(StorePath drvPath); /** * Wrappers around the corresponding Store methods that first consult the * derivation. This is currently needed because when there is no drv file * there also is no DB entry. */ - std::map> queryPartialDerivationOutputMap(); - OutputPathMap queryDerivationOutputMap(); + std::map> queryPartialDerivationOutputMap(const StorePath & drvPath); + OutputPathMap queryDerivationOutputMap(const StorePath & drvPath); /** * Update 'initialOutputs' to determine the current status of the @@ -113,17 +113,18 @@ struct DerivationGoal : public Goal * whether all outputs are valid and non-corrupt, and a * 'SingleDrvOutputs' structure containing the valid outputs. */ - std::pair checkPathValidity(); + std::pair checkPathValidity(const StorePath & drvPath); /** * Aborts if any output is not valid or corrupt, and otherwise * returns a 'SingleDrvOutputs' structure containing all outputs. */ - SingleDrvOutputs assertPathValidity(); + SingleDrvOutputs assertPathValidity(const StorePath & drvPath); - Co repairClosure(); + Co repairClosure(StorePath drvPath); Done done( + const StorePath & drvPath, BuildResult::Status status, SingleDrvOutputs builtOutputs = {}, std::optional ex = {}); diff --git a/src/libstore/include/nix/store/build/worker.hh b/src/libstore/include/nix/store/build/worker.hh index 46c4d181edb..c70c723774e 100644 --- a/src/libstore/include/nix/store/build/worker.hh +++ b/src/libstore/include/nix/store/build/worker.hh @@ -3,6 +3,7 @@ #include "nix/util/types.hh" #include "nix/store/store-api.hh" +#include "nix/store/derived-path-map.hh" #include "nix/store/build/goal.hh" #include "nix/store/realisation.hh" #include "nix/util/muxable-pipe.hh" @@ -104,7 +105,9 @@ private: * Maps used to prevent multiple instantiations of a goal for the * same derivation / path. */ - std::map> derivationGoals; + + DerivedPathMap> derivationGoals; + std::map> derivationBuildingGoals; std::map> substitutionGoals; std::map> drvOutputSubstitutionGoals; @@ -202,11 +205,11 @@ private: std::shared_ptr initGoalIfNeeded(std::weak_ptr & goal_weak, Args && ...args); std::shared_ptr makeDerivationGoalCommon( - const StorePath & drvPath, const OutputsSpec & wantedOutputs, + ref drvReq, const OutputsSpec & wantedOutputs, std::function()> mkDrvGoal); public: std::shared_ptr makeDerivationGoal( - const StorePath & drvPath, + ref drvReq, const OutputsSpec & wantedOutputs, BuildMode buildMode = bmNormal); std::shared_ptr makeBasicDerivationGoal( const StorePath & drvPath, const BasicDerivation & drv, diff --git a/src/libstore/include/nix/store/derived-path-map.hh b/src/libstore/include/nix/store/derived-path-map.hh index cad86d1b4e4..16ffeb05e69 100644 --- a/src/libstore/include/nix/store/derived-path-map.hh +++ b/src/libstore/include/nix/store/derived-path-map.hh @@ -21,8 +21,11 @@ namespace nix { * * @param V A type to instantiate for each output. It should probably * should be an "optional" type so not every interior node has to have a - * value. `* const Something` or `std::optional` would be - * good choices for "optional" types. + * value. For example, the scheduler uses + * `DerivedPathMap>` to + * remember which goals correspond to which outputs. `* const Something` + * or `std::optional` would also be good choices for + * "optional" types. */ template struct DerivedPathMap { diff --git a/tests/functional/dyn-drv/build-built-drv.sh b/tests/functional/dyn-drv/build-built-drv.sh index 647be945716..49d61c6ce26 100644 --- a/tests/functional/dyn-drv/build-built-drv.sh +++ b/tests/functional/dyn-drv/build-built-drv.sh @@ -18,4 +18,9 @@ clearStore drvDep=$(nix-instantiate ./text-hashed-output.nix -A producingDrv) -expectStderr 1 nix build "${drvDep}^out^out" --no-link | grepQuiet "Building dynamic derivations in one shot is not yet implemented" +# Store layer needs bugfix +requireDaemonNewerThan "2.30pre20250515" + +out2=$(nix build "${drvDep}^out^out" --no-link) + +test $out1 == $out2 diff --git a/tests/functional/dyn-drv/dep-built-drv-2.sh b/tests/functional/dyn-drv/dep-built-drv-2.sh index 3247720af76..0e4cc7c122d 100644 --- a/tests/functional/dyn-drv/dep-built-drv-2.sh +++ b/tests/functional/dyn-drv/dep-built-drv-2.sh @@ -3,7 +3,7 @@ source common.sh # Store layer needs bugfix -requireDaemonNewerThan "2.27pre20250205" +requireDaemonNewerThan "2.30pre20250515" TODO_NixOS # can't enable a sandbox feature easily @@ -13,4 +13,4 @@ restartDaemon NIX_BIN_DIR="$(dirname "$(type -p nix)")" export NIX_BIN_DIR -expectStderr 1 nix build -L --file ./non-trivial.nix --no-link | grepQuiet "Building dynamic derivations in one shot is not yet implemented" +nix build -L --file ./non-trivial.nix --no-link diff --git a/tests/functional/dyn-drv/dep-built-drv.sh b/tests/functional/dyn-drv/dep-built-drv.sh index 4f6e9b080fa..e9a8b6b832c 100644 --- a/tests/functional/dyn-drv/dep-built-drv.sh +++ b/tests/functional/dyn-drv/dep-built-drv.sh @@ -4,8 +4,11 @@ source common.sh out1=$(nix-build ./text-hashed-output.nix -A hello --no-out-link) +# Store layer needs bugfix +requireDaemonNewerThan "2.30pre20250515" + clearStore -expectStderr 1 nix-build ./text-hashed-output.nix -A wrapper --no-out-link | grepQuiet "Building dynamic derivations in one shot is not yet implemented" +out2=$(nix-build ./text-hashed-output.nix -A wrapper --no-out-link) -# diff -r $out1 $out2 +diff -r $out1 $out2 diff --git a/tests/functional/dyn-drv/failing-outer.sh b/tests/functional/dyn-drv/failing-outer.sh index fbad7070133..3feda74fbed 100644 --- a/tests/functional/dyn-drv/failing-outer.sh +++ b/tests/functional/dyn-drv/failing-outer.sh @@ -3,9 +3,7 @@ source common.sh # Store layer needs bugfix -requireDaemonNewerThan "2.27pre20250205" - -skipTest "dyn drv input scheduling had to be reverted for 2.27" +requireDaemonNewerThan "2.30pre20250515" expected=100 if [[ -v NIX_DAEMON_PACKAGE ]]; then expected=1; fi # work around the daemon not returning a 100 status correctly From 2190bf2006ed5d5a32c0491808778a760da0b4a1 Mon Sep 17 00:00:00 2001 From: Sergei Zimmerman Date: Wed, 21 May 2025 22:16:31 +0000 Subject: [PATCH 053/218] doc: Add `eval-profiler` documentation and release note --- doc/manual/rl-next/eval-profiler.md | 13 ++++++++ doc/manual/source/SUMMARY.md.in | 1 + .../source/advanced-topics/eval-profiler.md | 33 +++++++++++++++++++ 3 files changed, 47 insertions(+) create mode 100644 doc/manual/rl-next/eval-profiler.md create mode 100644 doc/manual/source/advanced-topics/eval-profiler.md diff --git a/doc/manual/rl-next/eval-profiler.md b/doc/manual/rl-next/eval-profiler.md new file mode 100644 index 00000000000..6b2bf2ce729 --- /dev/null +++ b/doc/manual/rl-next/eval-profiler.md @@ -0,0 +1,13 @@ +--- +synopsis: Add stack sampling evaluation profiler +prs: [13220] +--- + +Nix evaluator now supports stack sampling evaluation profiling via `--eval-profiler flamegraph` setting. +It collects collapsed call stack information to output file specified by +`--eval-profile-file` (`nix.profile` by default) in a format directly consumable +by `flamegraph.pl` and compatible tools like [speedscope](https://speedscope.app/). +Sampling frequency can be configured via `--eval-profiler-frequency` (99 Hz by default). + +Unlike existing `--trace-function-calls` this profiler includes the name of the function +being called when it's available. diff --git a/doc/manual/source/SUMMARY.md.in b/doc/manual/source/SUMMARY.md.in index 6711bd4bef4..8326a96e35c 100644 --- a/doc/manual/source/SUMMARY.md.in +++ b/doc/manual/source/SUMMARY.md.in @@ -57,6 +57,7 @@ - [Tuning Cores and Jobs](advanced-topics/cores-vs-jobs.md) - [Verifying Build Reproducibility](advanced-topics/diff-hook.md) - [Using the `post-build-hook`](advanced-topics/post-build-hook.md) + - [Evaluation profiler](advanced-topics/eval-profiler.md) - [Command Reference](command-ref/index.md) - [Common Options](command-ref/opt-common.md) - [Common Environment Variables](command-ref/env-common.md) diff --git a/doc/manual/source/advanced-topics/eval-profiler.md b/doc/manual/source/advanced-topics/eval-profiler.md new file mode 100644 index 00000000000..ed3848bb2db --- /dev/null +++ b/doc/manual/source/advanced-topics/eval-profiler.md @@ -0,0 +1,33 @@ +# Using the `eval-profiler` + +Nix evaluator supports [evaluation](@docroot@/language/evaluation.md) +[profiling]() +compatible with `flamegraph.pl`. The profiler samples the nix +function call stack at regular intervals. It can be enabled with the +[`eval-profiler`](@docroot@/command-ref/conf-file.md#conf-eval-profiler) +setting: + +```console +$ nix-instantiate "" -A hello --eval-profiler flamegraph +``` + +Stack sampling frequency and the output file path can be configured with +[`eval-profile-file`](@docroot@/command-ref/conf-file.md#conf-eval-profile-file) +and [`eval-profiler-frequency`](@docroot@/command-ref/conf-file.md#conf-eval-profiler-frequency). +By default the collected profile is saved to `nix.profile` file in the current working directory. + +The collected profile can be directly consumed by `flamegraph.pl`: + +```console +$ flamegraph.pl nix.profile > flamegraph.svg +``` + +The line information in the profile contains the location of the [call +site](https://en.wikipedia.org/wiki/Call_site) position and the name of the +function being called (when available). For example: + +``` +/nix/store/x9wnkly3k1gkq580m90jjn32q9f05q2v-source/pkgs/top-level/default.nix:167:5:primop import +``` + +Here `import` primop is called at `/nix/store/x9wnkly3k1gkq580m90jjn32q9f05q2v-source/pkgs/top-level/default.nix:167:5`. From 17eb2e8400b22635fbad74ae3cdfbd0a51d00233 Mon Sep 17 00:00:00 2001 From: Jeremy Fleischman Date: Wed, 7 May 2025 11:24:50 -0700 Subject: [PATCH 054/218] Expose flake directory to nix fmt as PRJ_ROOT env var This was discussed in https://github.com/NixOS/nix/issues/8034. I personally like `PRJ_ROOT`, which hopefully avoids some ambiguity around with subflakes. I only implemented this for `nix fmt` because it doesn't let you point at a flake not on your filesystem. macOS compilation fixes --- src/nix/formatter-run.md | 4 +++ src/nix/formatter.cc | 21 +++++++++++++-- src/nix/run.cc | 38 ++++++++++++++++++++++++---- src/nix/run.hh | 3 ++- tests/functional/formatter.sh | 33 +++++++++++++++++++++--- tests/functional/formatter.simple.sh | 2 +- 6 files changed, 88 insertions(+), 13 deletions(-) diff --git a/src/nix/formatter-run.md b/src/nix/formatter-run.md index db8583c9547..201cae92e4a 100644 --- a/src/nix/formatter-run.md +++ b/src/nix/formatter-run.md @@ -8,6 +8,10 @@ Flags can be forwarded to the formatter by using `--` followed by the flags. Any arguments will be forwarded to the formatter. Typically these are the files to format. +The environment variable `PRJ_ROOT` (according to [prj-spec](https://github.com/numtide/prj-spec)) +will be set to the absolute path to the directory containing the closest parent `flake.nix` +relative to the current directory. + # Example diff --git a/src/nix/formatter.cc b/src/nix/formatter.cc index 8b171b244ba..627fb362c83 100644 --- a/src/nix/formatter.cc +++ b/src/nix/formatter.cc @@ -1,8 +1,10 @@ #include "nix/cmd/command.hh" +#include "nix/cmd/installable-flake.hh" #include "nix/cmd/installable-value.hh" #include "nix/expr/eval.hh" #include "nix/store/local-fs-store.hh" #include "nix/cmd/installable-derived-path.hh" +#include "nix/util/environment-variables.hh" #include "run.hh" using namespace nix; @@ -72,10 +74,14 @@ struct CmdFormatterRun : MixFormatter, MixJSON auto evalState = getEvalState(); auto evalStore = getEvalStore(); - auto installable_ = parseInstallable(store, "."); + auto installable_ = parseInstallable(store, ".").cast(); auto & installable = InstallableValue::require(*installable_); auto app = installable.toApp(*evalState).resolve(evalStore, store); + auto maybeFlakeDir = installable_->flakeRef.input.getSourcePath(); + assert(maybeFlakeDir.has_value()); + auto flakeDir = maybeFlakeDir.value(); + Strings programArgs{app.program}; // Propagate arguments from the CLI @@ -83,11 +89,22 @@ struct CmdFormatterRun : MixFormatter, MixJSON programArgs.push_back(i); } + // Add the path to the flake as an environment variable. This enables formatters to format the entire flake even + // if run from a subdirectory. + StringMap env = getEnv(); + env["PRJ_ROOT"] = flakeDir; + // Release our references to eval caches to ensure they are persisted to disk, because // we are about to exec out of this process without running C++ destructors. evalState->evalCaches.clear(); - execProgramInStore(store, UseLookupPath::DontUse, app.program, programArgs); + execProgramInStore( + store, + UseLookupPath::DontUse, + app.program, + programArgs, + std::nullopt, // Use default system + env); }; }; diff --git a/src/nix/run.cc b/src/nix/run.cc index 0473c99b70b..3dae8ebc97d 100644 --- a/src/nix/run.cc +++ b/src/nix/run.cc @@ -10,6 +10,7 @@ #include "nix/util/finally.hh" #include "nix/util/source-accessor.hh" #include "nix/expr/eval.hh" +#include "nix/util/util.hh" #include #ifdef __linux__ @@ -19,6 +20,8 @@ #include +extern char ** environ __attribute__((weak)); + namespace nix::fs { using namespace std::filesystem; } using namespace nix; @@ -27,14 +30,37 @@ std::string chrootHelperName = "__run_in_chroot"; namespace nix { +/* Convert `env` to a list of strings suitable for `execve`'s `envp` argument. */ +Strings toEnvp(StringMap env) +{ + Strings envStrs; + for (auto & i : env) { + envStrs.push_back(i.first + "=" + i.second); + } + + return envStrs; +} + void execProgramInStore(ref store, UseLookupPath useLookupPath, const std::string & program, const Strings & args, - std::optional system) + std::optional system, + std::optional env) { logger->stop(); + char **envp; + Strings envStrs; + std::vector envCharPtrs; + if (env.has_value()) { + envStrs = toEnvp(env.value()); + envCharPtrs = stringsToCharPtrs(envStrs); + envp = envCharPtrs.data(); + } else { + envp = environ; + } + restoreProcessContext(); /* If this is a diverted store (i.e. its "logical" location @@ -54,7 +80,7 @@ void execProgramInStore(ref store, Strings helperArgs = { chrootHelperName, store->storeDir, store2->getRealStoreDir(), std::string(system.value_or("")), program }; for (auto & arg : args) helperArgs.push_back(arg); - execv(getSelfExe().value_or("nix").c_str(), stringsToCharPtrs(helperArgs).data()); + execve(getSelfExe().value_or("nix").c_str(), stringsToCharPtrs(helperArgs).data(), envp); throw SysError("could not execute chroot helper"); } @@ -64,10 +90,12 @@ void execProgramInStore(ref store, linux::setPersonality(*system); #endif - if (useLookupPath == UseLookupPath::Use) + if (useLookupPath == UseLookupPath::Use) { + // We have to set `environ` by hand because there is no `execvpe` on macOS. + environ = envp; execvp(program.c_str(), stringsToCharPtrs(args).data()); - else - execv(program.c_str(), stringsToCharPtrs(args).data()); + } else + execve(program.c_str(), stringsToCharPtrs(args).data(), envp); throw SysError("unable to execute '%s'", program); } diff --git a/src/nix/run.hh b/src/nix/run.hh index 9d95b8e7c64..5367c515c1f 100644 --- a/src/nix/run.hh +++ b/src/nix/run.hh @@ -14,6 +14,7 @@ void execProgramInStore(ref store, UseLookupPath useLookupPath, const std::string & program, const Strings & args, - std::optional system = std::nullopt); + std::optional system = std::nullopt, + std::optional env = std::nullopt); } diff --git a/tests/functional/formatter.sh b/tests/functional/formatter.sh index ea6f9e1ce13..6631dd6b87a 100755 --- a/tests/functional/formatter.sh +++ b/tests/functional/formatter.sh @@ -34,13 +34,38 @@ cat << EOF > flake.nix } EOF +mkdir subflake +cp ./simple.nix ./simple.builder.sh ./formatter.simple.sh "${config_nix}" "$TEST_HOME/subflake" + +cat << EOF > subflake/flake.nix +{ + outputs = _: { + formatter.$system = + with import ./config.nix; + mkDerivation { + name = "formatter"; + buildCommand = '' + mkdir -p \$out/bin + echo "#! ${shell}" > \$out/bin/formatter + cat \${./formatter.simple.sh} >> \$out/bin/formatter + chmod +x \$out/bin/formatter + ''; + }; + }; +} +EOF + # No arguments check -[[ "$(nix fmt)" = "Formatting(0):" ]] -[[ "$(nix formatter run)" = "Formatting(0):" ]] +[[ "$(nix fmt)" = "PRJ_ROOT=$TEST_HOME Formatting(0):" ]] +[[ "$(nix formatter run)" = "PRJ_ROOT=$TEST_HOME Formatting(0):" ]] # Argument forwarding check -nix fmt ./file ./folder | grep 'Formatting(2): ./file ./folder' -nix formatter run ./file ./folder | grep 'Formatting(2): ./file ./folder' +nix fmt ./file ./folder | grep "PRJ_ROOT=$TEST_HOME Formatting(2): ./file ./folder" +nix formatter run ./file ./folder | grep "PRJ_ROOT=$TEST_HOME Formatting(2): ./file ./folder" + +# test subflake +cd subflake +nix fmt ./file | grep "PRJ_ROOT=$TEST_HOME/subflake Formatting(1): ./file" # Build checks ## Defaults to a ./result. diff --git a/tests/functional/formatter.simple.sh b/tests/functional/formatter.simple.sh index e53f6c9be9a..355ff00efbb 100755 --- a/tests/functional/formatter.simple.sh +++ b/tests/functional/formatter.simple.sh @@ -1,2 +1,2 @@ #!/usr/bin/env bash -echo "Formatting(${#}):" "${@}" +echo "PRJ_ROOT=$PRJ_ROOT Formatting(${#}):" "${@}" From 90d1ff480590b56db202a20c3927df4bf05e4eac Mon Sep 17 00:00:00 2001 From: Sergei Zimmerman Date: Thu, 22 May 2025 23:08:59 +0000 Subject: [PATCH 055/218] libmain: Catch logger exceptions in `handleExceptions` Avoid std::terminate in case logging code also throws. --- src/libmain/shared.cc | 49 ++++++++++++++++++++++++------------------- 1 file changed, 27 insertions(+), 22 deletions(-) diff --git a/src/libmain/shared.cc b/src/libmain/shared.cc index 2da22a7731c..3b431f02135 100644 --- a/src/libmain/shared.cc +++ b/src/libmain/shared.cc @@ -317,29 +317,34 @@ int handleExceptions(const std::string & programName, std::function fun) std::string error = ANSI_RED "error:" ANSI_NORMAL " "; try { try { - fun(); - } catch (...) { - /* Subtle: we have to make sure that any `interrupted' - condition is discharged before we reach printMsg() - below, since otherwise it will throw an (uncaught) - exception. */ - setInterruptThrown(); - throw; + try { + fun(); + } catch (...) { + /* Subtle: we have to make sure that any `interrupted' + condition is discharged before we reach printMsg() + below, since otherwise it will throw an (uncaught) + exception. */ + setInterruptThrown(); + throw; + } + } catch (Exit & e) { + return e.status; + } catch (UsageError & e) { + logError(e.info()); + printError("Try '%1% --help' for more information.", programName); + return 1; + } catch (BaseError & e) { + logError(e.info()); + return e.info().status; + } catch (std::bad_alloc & e) { + printError(error + "out of memory"); + return 1; + } catch (std::exception & e) { + printError(error + e.what()); + return 1; } - } catch (Exit & e) { - return e.status; - } catch (UsageError & e) { - logError(e.info()); - printError("Try '%1% --help' for more information.", programName); - return 1; - } catch (BaseError & e) { - logError(e.info()); - return e.info().status; - } catch (std::bad_alloc & e) { - printError(error + "out of memory"); - return 1; - } catch (std::exception & e) { - printError(error + e.what()); + } catch (...) { + /* In case logger also throws just give up. */ return 1; } From 9f680874c5aa15304c3ab3b942170a743287f87b Mon Sep 17 00:00:00 2001 From: Eelco Dolstra Date: Fri, 14 Mar 2025 17:33:48 +0100 Subject: [PATCH 056/218] Make the JSON logger more robust We now ignore connection / write errors. --- src/libutil/logging.cc | 15 +++++++++++++-- 1 file changed, 13 insertions(+), 2 deletions(-) diff --git a/src/libutil/logging.cc b/src/libutil/logging.cc index 1d0d3fa2d70..21f13d3c0dd 100644 --- a/src/libutil/logging.cc +++ b/src/libutil/logging.cc @@ -204,6 +204,7 @@ struct JSONLogger : Logger { struct State { + bool enabled = true; }; Sync _state; @@ -216,8 +217,18 @@ struct JSONLogger : Logger { /* Acquire a lock to prevent log messages from clobbering each other. */ - auto state(_state.lock()); - writeLine(fd, line); + try { + auto state(_state.lock()); + if (state->enabled) + writeLine(fd, line); + } catch (...) { + bool enabled = false; + std::swap(_state.lock()->enabled, enabled); + if (enabled) { + ignoreExceptionExceptInterrupt(); + logger->warn("disabling JSON logger due to write errors"); + } + } } void log(Verbosity lvl, std::string_view s) override From 0087188d47f35778e1c347a2f853b9d2b1df39d0 Mon Sep 17 00:00:00 2001 From: Eelco Dolstra Date: Fri, 11 Apr 2025 16:11:57 +0200 Subject: [PATCH 057/218] Add convenience function for connecting to a Unix domain socket --- src/libstore/uds-remote-store.cc | 4 +--- src/libutil/include/nix/util/unix-domain-socket.hh | 7 +++++++ src/libutil/unix-domain-socket.cc | 7 +++++++ 3 files changed, 15 insertions(+), 3 deletions(-) diff --git a/src/libstore/uds-remote-store.cc b/src/libstore/uds-remote-store.cc index 9950e38b36c..c979b5e47c5 100644 --- a/src/libstore/uds-remote-store.cc +++ b/src/libstore/uds-remote-store.cc @@ -83,9 +83,7 @@ ref UDSRemoteStore::openConnection() auto conn = make_ref(); /* Connect to a daemon that does the privileged work for us. */ - conn->fd = createUnixDomainSocket(); - - nix::connect(toSocket(conn->fd.get()), config->path); + conn->fd = nix::connect(config->path); conn->from.fd = conn->fd.get(); conn->to.fd = conn->fd.get(); diff --git a/src/libutil/include/nix/util/unix-domain-socket.hh b/src/libutil/include/nix/util/unix-domain-socket.hh index 704999ec1d8..6885d740b12 100644 --- a/src/libutil/include/nix/util/unix-domain-socket.hh +++ b/src/libutil/include/nix/util/unix-domain-socket.hh @@ -9,6 +9,8 @@ #endif #include +#include + namespace nix { /** @@ -80,4 +82,9 @@ void bind(Socket fd, const std::string & path); */ void connect(Socket fd, const std::string & path); +/** + * Connect to a Unix domain socket. + */ +AutoCloseFD connect(const std::filesystem::path & path); + } diff --git a/src/libutil/unix-domain-socket.cc b/src/libutil/unix-domain-socket.cc index 8722c8f0557..0e8c21d668f 100644 --- a/src/libutil/unix-domain-socket.cc +++ b/src/libutil/unix-domain-socket.cc @@ -114,4 +114,11 @@ void connect(Socket fd, const std::string & path) bindConnectProcHelper("connect", ::connect, fd, path); } +AutoCloseFD connect(const std::filesystem::path & path) +{ + auto fd = createUnixDomainSocket(); + nix::connect(toSocket(fd.get()), path); + return fd; +} + } From 7240fb198f2ebc0e222969ba389979a45da2dc89 Mon Sep 17 00:00:00 2001 From: Eelco Dolstra Date: Fri, 11 Apr 2025 16:45:59 +0200 Subject: [PATCH 058/218] Add `json-log-path` setting This setting specifies a path (which can be a regular file or Unix domain socket) that receives a copy of all Nix log messages (in JSON format). --- doc/manual/rl-next/json-logger.md | 6 ++ src/libstore/daemon.cc | 2 + src/libutil/include/nix/util/logging.hh | 25 +++++- src/libutil/logging.cc | 53 +++++++++++- src/libutil/meson.build | 1 + src/libutil/tee-logger.cc | 107 ++++++++++++++++++++++++ src/nix/main.cc | 2 + tests/functional/logging.sh | 9 ++ 8 files changed, 200 insertions(+), 5 deletions(-) create mode 100644 doc/manual/rl-next/json-logger.md create mode 100644 src/libutil/tee-logger.cc diff --git a/doc/manual/rl-next/json-logger.md b/doc/manual/rl-next/json-logger.md new file mode 100644 index 00000000000..867b5d8b3fa --- /dev/null +++ b/doc/manual/rl-next/json-logger.md @@ -0,0 +1,6 @@ +--- +synopsis: "`json-log-path` setting" +prs: [13003] +--- + +New setting `json-log-path` that sends a copy of all Nix log messages (in JSON format) to a file or Unix domain socket. diff --git a/src/libstore/daemon.cc b/src/libstore/daemon.cc index 8f751427342..dfc068bc775 100644 --- a/src/libstore/daemon.cc +++ b/src/libstore/daemon.cc @@ -15,6 +15,7 @@ #include "nix/store/derivations.hh" #include "nix/util/args.hh" #include "nix/util/git.hh" +#include "nix/util/logging.hh" #ifndef _WIN32 // TODO need graceful async exit support on Windows? # include "nix/util/monitor-fd.hh" @@ -1050,6 +1051,7 @@ void processConnection( if (!recursive) { prevLogger_ = std::move(logger); logger = std::move(tunnelLogger_); + applyJSONLogger(); } unsigned int opCount = 0; diff --git a/src/libutil/include/nix/util/logging.hh b/src/libutil/include/nix/util/logging.hh index 9210229bf26..f2064b74d8a 100644 --- a/src/libutil/include/nix/util/logging.hh +++ b/src/libutil/include/nix/util/logging.hh @@ -6,6 +6,8 @@ #include "nix/util/file-descriptor.hh" #include "nix/util/finally.hh" +#include + #include namespace nix { @@ -49,6 +51,14 @@ struct LoggerSettings : Config Whether Nix should print out a stack trace in case of Nix expression evaluation errors. )"}; + + Setting jsonLogPath{ + this, "", "json-log-path", + R"( + A path to which JSON records of Nix's log output will be + written, in the same format as `--log-format internal-json` + (without the `@nix ` prefixes on each line). + )"}; }; extern LoggerSettings loggerSettings; @@ -196,7 +206,20 @@ extern std::unique_ptr logger; std::unique_ptr makeSimpleLogger(bool printBuildLogs = true); -std::unique_ptr makeJSONLogger(Descriptor fd); +/** + * Create a logger that sends log messages to `mainLogger` and the + * list of loggers in `extraLoggers`. Only `mainLogger` is used for + * writing to stdout and getting user input. + */ +std::unique_ptr makeTeeLogger( + std::unique_ptr mainLogger, + std::vector> && extraLoggers); + +std::unique_ptr makeJSONLogger(Descriptor fd, bool includeNixPrefix = true); + +std::unique_ptr makeJSONLogger(const std::filesystem::path & path, bool includeNixPrefix = true); + +void applyJSONLogger(); /** * @param source A noun phrase describing the source of the message, e.g. "the builder". diff --git a/src/libutil/logging.cc b/src/libutil/logging.cc index 21f13d3c0dd..a328a5c73e3 100644 --- a/src/libutil/logging.cc +++ b/src/libutil/logging.cc @@ -7,6 +7,7 @@ #include "nix/util/source-path.hh" #include "nix/util/position.hh" #include "nix/util/sync.hh" +#include "nix/util/unix-domain-socket.hh" #include #include @@ -182,8 +183,12 @@ void to_json(nlohmann::json & json, std::shared_ptr pos) struct JSONLogger : Logger { Descriptor fd; + bool includeNixPrefix; - JSONLogger(Descriptor fd) : fd(fd) { } + JSONLogger(Descriptor fd, bool includeNixPrefix) + : fd(fd) + , includeNixPrefix(includeNixPrefix) + { } bool isVerbose() override { return true; @@ -212,7 +217,7 @@ struct JSONLogger : Logger { void write(const nlohmann::json & json) { auto line = - "@nix " + + (includeNixPrefix ? "@nix " : "") + json.dump(-1, ' ', false, nlohmann::json::error_handler_t::replace); /* Acquire a lock to prevent log messages from clobbering each @@ -300,9 +305,49 @@ struct JSONLogger : Logger { } }; -std::unique_ptr makeJSONLogger(Descriptor fd) +std::unique_ptr makeJSONLogger(Descriptor fd, bool includeNixPrefix) { - return std::make_unique(fd); + return std::make_unique(fd, includeNixPrefix); +} + +std::unique_ptr makeJSONLogger(const std::filesystem::path & path, bool includeNixPrefix) +{ + struct JSONFileLogger : JSONLogger { + AutoCloseFD fd; + + JSONFileLogger(AutoCloseFD && fd, bool includeNixPrefix) + : JSONLogger(fd.get(), includeNixPrefix) + , fd(std::move(fd)) + { } + }; + + AutoCloseFD fd = + std::filesystem::is_socket(path) + ? connect(path) + : toDescriptor(open(path.c_str(), O_CREAT | O_APPEND | O_WRONLY, 0644)); + if (!fd) + throw SysError("opening log file %1%", path); + + return std::make_unique(std::move(fd), includeNixPrefix); +} + +void applyJSONLogger() +{ + if (!loggerSettings.jsonLogPath.get().empty()) { + try { + std::vector> loggers; + loggers.push_back(makeJSONLogger(std::filesystem::path(loggerSettings.jsonLogPath.get()), false)); + try { + logger = makeTeeLogger(std::move(logger), std::move(loggers)); + } catch (...) { + // `logger` is now gone so give up. + abort(); + } + } catch (...) { + ignoreExceptionExceptInterrupt(); + } + + } } static Logger::Fields getFields(nlohmann::json & json) diff --git a/src/libutil/meson.build b/src/libutil/meson.build index b0e82e46aaa..04ca06eee0c 100644 --- a/src/libutil/meson.build +++ b/src/libutil/meson.build @@ -147,6 +147,7 @@ sources = [config_priv_h] + files( 'strings.cc', 'suggestions.cc', 'tarfile.cc', + 'tee-logger.cc', 'terminal.cc', 'thread-pool.cc', 'union-source-accessor.cc', diff --git a/src/libutil/tee-logger.cc b/src/libutil/tee-logger.cc new file mode 100644 index 00000000000..55334a821fb --- /dev/null +++ b/src/libutil/tee-logger.cc @@ -0,0 +1,107 @@ +#include "nix/util/logging.hh" + +namespace nix { + +struct TeeLogger : Logger +{ + std::vector> loggers; + + TeeLogger(std::vector> && loggers) + : loggers(std::move(loggers)) + { + } + + void stop() override + { + for (auto & logger : loggers) + logger->stop(); + }; + + void pause() override + { + for (auto & logger : loggers) + logger->pause(); + }; + + void resume() override + { + for (auto & logger : loggers) + logger->resume(); + }; + + void log(Verbosity lvl, std::string_view s) override + { + for (auto & logger : loggers) + logger->log(lvl, s); + } + + void logEI(const ErrorInfo & ei) override + { + for (auto & logger : loggers) + logger->logEI(ei); + } + + void startActivity( + ActivityId act, + Verbosity lvl, + ActivityType type, + const std::string & s, + const Fields & fields, + ActivityId parent) override + { + for (auto & logger : loggers) + logger->startActivity(act, lvl, type, s, fields, parent); + } + + void stopActivity(ActivityId act) override + { + for (auto & logger : loggers) + logger->stopActivity(act); + } + + void result(ActivityId act, ResultType type, const Fields & fields) override + { + for (auto & logger : loggers) + logger->result(act, type, fields); + } + + void writeToStdout(std::string_view s) override + { + for (auto & logger : loggers) { + /* Let only the first logger write to stdout to avoid + duplication. This means that the first logger needs to + be the one managing stdout/stderr + (e.g. `ProgressBar`). */ + logger->writeToStdout(s); + break; + } + } + + std::optional ask(std::string_view s) override + { + for (auto & logger : loggers) { + auto c = logger->ask(s); + if (c) + return c; + } + return std::nullopt; + } + + void setPrintBuildLogs(bool printBuildLogs) override + { + for (auto & logger : loggers) + logger->setPrintBuildLogs(printBuildLogs); + } +}; + +std::unique_ptr +makeTeeLogger(std::unique_ptr mainLogger, std::vector> && extraLoggers) +{ + std::vector> allLoggers; + allLoggers.push_back(std::move(mainLogger)); + for (auto & l : extraLoggers) + allLoggers.push_back(std::move(l)); + return std::make_unique(std::move(allLoggers)); +} + +} diff --git a/src/nix/main.cc b/src/nix/main.cc index 05c5da27d6b..0e5ccf34b40 100644 --- a/src/nix/main.cc +++ b/src/nix/main.cc @@ -460,6 +460,8 @@ void mainWrapped(int argc, char * * argv) if (!args.helpRequested && !args.completions) throw; } + applyJSONLogger(); + if (args.helpRequested) { std::vector subcommand; MultiCommand * command = &args; diff --git a/tests/functional/logging.sh b/tests/functional/logging.sh index ddb1913adf2..83df9a45d7d 100755 --- a/tests/functional/logging.sh +++ b/tests/functional/logging.sh @@ -33,3 +33,12 @@ if isDaemonNewer "2.26"; then # Build works despite ill-formed structured build log entries. expectStderr 0 nix build -f ./logging/unusual-logging.nix --no-link | grepQuiet 'warning: Unable to handle a JSON message from the derivation builder:' fi + +# Test json-log-path. +if [[ "$NIX_REMOTE" != "daemon" ]]; then + clearStore + nix build -vv --file dependencies.nix --no-link --json-log-path "$TEST_ROOT/log.json" 2>&1 | grepQuiet 'building.*dependencies-top.drv' + jq < "$TEST_ROOT/log.json" + grep '{"action":"start","fields":\[".*-dependencies-top.drv","",1,1\],"id":.*,"level":3,"parent":0' "$TEST_ROOT/log.json" >&2 + (( $(grep '{"action":"msg","level":5,"msg":"executing builder .*"}' "$TEST_ROOT/log.json" | wc -l) == 5 )) +fi From 7cef4559fecc1b0ed1ebd0420773ac714b62bf64 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?J=C3=B6rg=20Thalheim?= Date: Fri, 23 May 2025 10:34:49 +0200 Subject: [PATCH 059/218] util/json-log-path: document unix sockets and concurrency issues --- src/libutil/include/nix/util/logging.hh | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/src/libutil/include/nix/util/logging.hh b/src/libutil/include/nix/util/logging.hh index f2064b74d8a..765975faa2a 100644 --- a/src/libutil/include/nix/util/logging.hh +++ b/src/libutil/include/nix/util/logging.hh @@ -55,9 +55,11 @@ struct LoggerSettings : Config Setting jsonLogPath{ this, "", "json-log-path", R"( - A path to which JSON records of Nix's log output will be + A file or unix socket to which JSON records of Nix's log output will be written, in the same format as `--log-format internal-json` (without the `@nix ` prefixes on each line). + Concurrent writes to the same file by multiple Nix processes are not supported and + may result in interleaved or corrupted log records. )"}; }; From d8da8f0cd6eafeee975e775d1fb74fa37d7012af Mon Sep 17 00:00:00 2001 From: Sergei Zimmerman Date: Fri, 23 May 2025 12:06:59 +0000 Subject: [PATCH 060/218] export/meson: Don't require `-std=c++2a` for -c libraries in `.pc` files --- nix-meson-build-support/export/meson.build | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/nix-meson-build-support/export/meson.build b/nix-meson-build-support/export/meson.build index b2409de8571..950bd954434 100644 --- a/nix-meson-build-support/export/meson.build +++ b/nix-meson-build-support/export/meson.build @@ -11,12 +11,18 @@ endforeach requires_public += deps_public extra_pkg_config_variables = get_variable('extra_pkg_config_variables', {}) + +extra_cflags = [] +if not meson.project_name().endswith('-c') + extra_cflags += ['-std=c++2a'] +endif + import('pkgconfig').generate( this_library, filebase : meson.project_name(), name : 'Nix', description : 'Nix Package Manager', - extra_cflags : ['-std=c++2a'], + extra_cflags : extra_cflags, requires : requires_public, requires_private : requires_private, libraries_private : libraries_private, From fa6e10ea6a87127ae813a708ccc97e708982f93f Mon Sep 17 00:00:00 2001 From: Eelco Dolstra Date: Fri, 23 May 2025 23:33:59 +0200 Subject: [PATCH 061/218] Don't use 'callback' object that we may have moved out of --- src/libstore/http-binary-cache-store.cc | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/src/libstore/http-binary-cache-store.cc b/src/libstore/http-binary-cache-store.cc index 2b591dda96e..e44d146b9ee 100644 --- a/src/libstore/http-binary-cache-store.cc +++ b/src/libstore/http-binary-cache-store.cc @@ -176,13 +176,13 @@ class HttpBinaryCacheStore : void getFile(const std::string & path, Callback> callback) noexcept override { + auto callbackPtr = std::make_shared(std::move(callback)); + try { checkEnabled(); auto request(makeRequest(path)); - auto callbackPtr = std::make_shared(std::move(callback)); - getFileTransfer()->enqueueFileTransfer(request, {[callbackPtr, this](std::future result) { try { @@ -198,7 +198,7 @@ class HttpBinaryCacheStore : }}); } catch (...) { - callback.rethrow(); + callbackPtr->rethrow(); return; } } From d877b0c0cc4795d17d10b9b9039f2de828152c55 Mon Sep 17 00:00:00 2001 From: Eelco Dolstra Date: Sat, 24 May 2025 00:14:32 +0200 Subject: [PATCH 062/218] fromStructuredAttrs(): Don't crash if exportReferencesGraph is a string Fixes error: [json.exception.type_error.302] type must be array, but is string and other crashes. Fixes #13254. --- src/libstore/derivation-options.cc | 9 +++++++-- src/libstore/misc.cc | 2 ++ 2 files changed, 9 insertions(+), 2 deletions(-) diff --git a/src/libstore/derivation-options.cc b/src/libstore/derivation-options.cc index e031f844757..f6bac2868fd 100644 --- a/src/libstore/derivation-options.cc +++ b/src/libstore/derivation-options.cc @@ -211,8 +211,13 @@ DerivationOptions::fromStructuredAttrs(const StringMap & env, const StructuredAt auto e = optionalValueAt(parsed->structuredAttrs, "exportReferencesGraph"); if (!e || !e->is_object()) return ret; - for (auto & [key, storePathsJson] : getObject(*e)) { - ret.insert_or_assign(key, storePathsJson); + for (auto & [key, value] : getObject(*e)) { + if (value.is_array()) + ret.insert_or_assign(key, value); + else if (value.is_string()) + ret.insert_or_assign(key, StringSet{value}); + else + throw Error("'exportReferencesGraph' value is not an array or a string"); } } else { auto s = getOr(env, "exportReferencesGraph", ""); diff --git a/src/libstore/misc.cc b/src/libstore/misc.cc index 967c91d72d3..dabae647fbb 100644 --- a/src/libstore/misc.cc +++ b/src/libstore/misc.cc @@ -225,6 +225,8 @@ void Store::queryMissing(const std::vector & targets, auto parsedDrv = StructuredAttrs::tryParse(drv->env); DerivationOptions drvOptions; try { + // FIXME: this is a lot of work just to get the value + // of `allowSubstitutes`. drvOptions = DerivationOptions::fromStructuredAttrs( drv->env, parsedDrv ? &*parsedDrv : nullptr); From c66eb9cef77c3462d0324b258d0c5e0b8e4f4e7f Mon Sep 17 00:00:00 2001 From: Eelco Dolstra Date: Sat, 24 May 2025 00:40:06 +0200 Subject: [PATCH 063/218] Add test --- tests/functional/structured-attrs-shell.nix | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/functional/structured-attrs-shell.nix b/tests/functional/structured-attrs-shell.nix index a819e39cdae..e9b9f1e3937 100644 --- a/tests/functional/structured-attrs-shell.nix +++ b/tests/functional/structured-attrs-shell.nix @@ -21,7 +21,7 @@ mkDerivation { "b" "c" ]; - exportReferencesGraph.refs = [ dep ]; + exportReferencesGraph.refs = dep; buildCommand = '' touch ''${outputs[out]}; touch ''${outputs[dev]} ''; From ca9696748a0ec17341dfa98c5ae5f823032c595b Mon Sep 17 00:00:00 2001 From: Eelco Dolstra Date: Sat, 24 May 2025 01:28:30 +0200 Subject: [PATCH 064/218] Make the S3 test more robust Waiting for the minio unit is apparently not reliable enough, so let's also wait for the port. --- tests/nixos/s3-binary-cache-store.nix | 1 + 1 file changed, 1 insertion(+) diff --git a/tests/nixos/s3-binary-cache-store.nix b/tests/nixos/s3-binary-cache-store.nix index fc55a27ae14..a22e4c2c28f 100644 --- a/tests/nixos/s3-binary-cache-store.nix +++ b/tests/nixos/s3-binary-cache-store.nix @@ -68,6 +68,7 @@ in # Create a binary cache. server.wait_for_unit("minio") server.wait_for_unit("network-addresses-eth1.service") + server.wait_for_open_port(9000) server.succeed("mc config host add minio http://localhost:9000 ${accessKey} ${secretKey} --api s3v4") server.succeed("mc mb minio/my-cache") From 751f50f4ad07929e3e0aaa3edfc143dab50b613b Mon Sep 17 00:00:00 2001 From: Stefan Boca Date: Tue, 20 May 2025 22:01:56 -0700 Subject: [PATCH 065/218] nix-profile{,-daemon}.fish: check for profile in XDG_DATA_HOME ...and also NIX_STATE_HOME in nix-profile.fish. This is directly translated from the bash scripts and makes the fish scripts equivalent in functionality to the bash scripts. Note that nix-profile.fish checks for NIX_STATE_HOME and nix-profile-daemon.fish does not, so the two scripts are no longer identical. --- scripts/nix-profile-daemon.fish.in | 28 ++++++++++++++++++++++++- scripts/nix-profile.fish.in | 33 +++++++++++++++++++++++++++++- 2 files changed, 59 insertions(+), 2 deletions(-) diff --git a/scripts/nix-profile-daemon.fish.in b/scripts/nix-profile-daemon.fish.in index 3fb7271510a..5bece7d7a29 100644 --- a/scripts/nix-profile-daemon.fish.in +++ b/scripts/nix-profile-daemon.fish.in @@ -24,7 +24,33 @@ end # Set up the per-user profile. -set --local NIX_LINK $HOME/.nix-profile +set --local NIX_LINK "$HOME/.nix-profile" +set --local NIX_LINK_NEW +if test -n "$XDG_STATE_HOME" + set NIX_LINK_NEW "$XDG_STATE_HOME/nix/profile" +else + set NIX_LINK_NEW "$HOME/.local/state/nix/profile" +end +if test -e "$NIX_LINK_NEW" + if test -t 2; and test -e "$NIX_LINK" + set --local warning "\033[1;35mwarning:\033[0m " + printf "$warning Both %s and legacy %s exist; using the former.\n" "$NIX_LINK_NEW" "$NIX_LINK" 1>&2 + + if test (realpath "$NIX_LINK") = (realpath "$NIX_LINK_NEW") + printf " Since the profiles match, you can safely delete either of them.\n" 1>&2 + else + # This should be an exceptionally rare occasion: the only way to get it would be to + # 1. Update to newer Nix; + # 2. Remove .nix-profile; + # 3. Set the $NIX_LINK_NEW to something other than the default user profile; + # 4. Roll back to older Nix. + # If someone did all that, they can probably figure out how to migrate the profile. + printf "$warning Profiles do not match. You should manually migrate from %s to %s.\n" "$NIX_LINK" "$NIX_LINK_NEW" 1>&2 + end + end + + set NIX_LINK "$NIX_LINK_NEW" +end # Set up environment. # This part should be kept in sync with nixpkgs:nixos/modules/programs/environment.nix diff --git a/scripts/nix-profile.fish.in b/scripts/nix-profile.fish.in index 3fb7271510a..4cb28ae55f5 100644 --- a/scripts/nix-profile.fish.in +++ b/scripts/nix-profile.fish.in @@ -24,7 +24,38 @@ end # Set up the per-user profile. -set --local NIX_LINK $HOME/.nix-profile +set --local NIX_LINK +if test -n "$NIX_STATE_HOME" + set NIX_LINK "$NIX_STATE_HOME/.nix-profile" +else + set NIX_LINK "$HOME/.nix-profile" + set --local NIX_LINK_NEW + if test -n "$XDG_STATE_HOME" + set NIX_LINK_NEW "$XDG_STATE_HOME/nix/profile" + else + set NIX_LINK_NEW "$HOME/.local/state/nix/profile" + end + if test -e "$NIX_LINK_NEW" + if test -t 2; and test -e "$NIX_LINK" + set --local warning "\033[1;35mwarning:\033[0m " + printf "$warning Both %s and legacy %s exist; using the former.\n" "$NIX_LINK_NEW" "$NIX_LINK" 1>&2 + + if test (realpath "$NIX_LINK") = (realpath "$NIX_LINK_NEW") + printf " Since the profiles match, you can safely delete either of them.\n" 1>&2 + else + # This should be an exceptionally rare occasion: the only way to get it would be to + # 1. Update to newer Nix; + # 2. Remove .nix-profile; + # 3. Set the $NIX_LINK_NEW to something other than the default user profile; + # 4. Roll back to older Nix. + # If someone did all that, they can probably figure out how to migrate the profile. + printf "$warning Profiles do not match. You should manually migrate from %s to %s.\n" "$NIX_LINK" "$NIX_LINK_NEW" 1>&2 + end + end + + set NIX_LINK "$NIX_LINK_NEW" + end +end # Set up environment. # This part should be kept in sync with nixpkgs:nixos/modules/programs/environment.nix From 128750225d8ad8c23364a1e1051799530e786ac0 Mon Sep 17 00:00:00 2001 From: Sergei Zimmerman Date: Sun, 25 May 2025 14:58:31 +0000 Subject: [PATCH 066/218] libexpr: Pass mutable `EvalState` to `EvalProfiler` Sometimes the profiler might want to do evaluation (e.g. for getting derivation names). This is not ideal, but is really necessary to make the profiler stack traces useful for end users. --- src/libexpr/eval-profiler.cc | 28 ++++++++----------- src/libexpr/function-trace.cc | 4 +-- src/libexpr/include/nix/expr/eval-profiler.hh | 13 ++++----- .../include/nix/expr/function-trace.hh | 4 +-- 4 files changed, 20 insertions(+), 29 deletions(-) diff --git a/src/libexpr/eval-profiler.cc b/src/libexpr/eval-profiler.cc index f08b46e6bf8..bf2442c31ae 100644 --- a/src/libexpr/eval-profiler.cc +++ b/src/libexpr/eval-profiler.cc @@ -5,18 +5,14 @@ namespace nix { -void EvalProfiler::preFunctionCallHook( - const EvalState & state, const Value & v, std::span args, const PosIdx pos) -{ -} +void EvalProfiler::preFunctionCallHook(EvalState & state, const Value & v, std::span args, const PosIdx pos) {} -void EvalProfiler::postFunctionCallHook( - const EvalState & state, const Value & v, std::span args, const PosIdx pos) +void EvalProfiler::postFunctionCallHook(EvalState & state, const Value & v, std::span args, const PosIdx pos) { } void MultiEvalProfiler::preFunctionCallHook( - const EvalState & state, const Value & v, std::span args, const PosIdx pos) + EvalState & state, const Value & v, std::span args, const PosIdx pos) { for (auto & profiler : profilers) { if (profiler->getNeededHooks().test(Hook::preFunctionCall)) @@ -25,7 +21,7 @@ void MultiEvalProfiler::preFunctionCallHook( } void MultiEvalProfiler::postFunctionCallHook( - const EvalState & state, const Value & v, std::span args, const PosIdx pos) + EvalState & state, const Value & v, std::span args, const PosIdx pos) { for (auto & profiler : profilers) { if (profiler->getNeededHooks().test(Hook::postFunctionCall)) @@ -126,7 +122,7 @@ class SampleStack : public EvalProfiler } public: - SampleStack(const EvalState & state, std::filesystem::path profileFile, std::chrono::nanoseconds period) + SampleStack(EvalState & state, std::filesystem::path profileFile, std::chrono::nanoseconds period) : state(state) , sampleInterval(period) , profileFd([&]() { @@ -140,9 +136,9 @@ class SampleStack : public EvalProfiler } [[gnu::noinline]] void - preFunctionCallHook(const EvalState & state, const Value & v, std::span args, const PosIdx pos) override; + preFunctionCallHook(EvalState & state, const Value & v, std::span args, const PosIdx pos) override; [[gnu::noinline]] void - postFunctionCallHook(const EvalState & state, const Value & v, std::span args, const PosIdx pos) override; + postFunctionCallHook(EvalState & state, const Value & v, std::span args, const PosIdx pos) override; void maybeSaveProfile(std::chrono::time_point now); void saveProfile(); @@ -156,7 +152,7 @@ class SampleStack : public EvalProfiler private: /** Hold on to an instance of EvalState for symbolizing positions. */ - const EvalState & state; + EvalState & state; std::chrono::nanoseconds sampleInterval; AutoCloseFD profileFd; FrameStack stack; @@ -191,7 +187,7 @@ FrameInfo SampleStack::getFrameInfoFromValueAndPos(const Value & v, PosIdx pos) } [[gnu::noinline]] void SampleStack::preFunctionCallHook( - const EvalState & state, const Value & v, [[maybe_unused]] std::span args, const PosIdx pos) + EvalState & state, const Value & v, [[maybe_unused]] std::span args, const PosIdx pos) { stack.push_back(getFrameInfoFromValueAndPos(v, pos)); @@ -208,9 +204,8 @@ FrameInfo SampleStack::getFrameInfoFromValueAndPos(const Value & v, PosIdx pos) } [[gnu::noinline]] void -SampleStack::postFunctionCallHook(const EvalState & state, const Value & v, std::span args, const PosIdx pos) +SampleStack::postFunctionCallHook(EvalState & state, const Value & v, std::span args, const PosIdx pos) { - if (!stack.empty()) stack.pop_back(); } @@ -300,8 +295,7 @@ SampleStack::~SampleStack() } // namespace -ref -makeSampleStackProfiler(const EvalState & state, std::filesystem::path profileFile, uint64_t frequency) +ref makeSampleStackProfiler(EvalState & state, std::filesystem::path profileFile, uint64_t frequency) { /* 0 is a special value for sampling stack after each call. */ std::chrono::nanoseconds period = frequency == 0 diff --git a/src/libexpr/function-trace.cc b/src/libexpr/function-trace.cc index 993dd72d7d0..cda3bc2db41 100644 --- a/src/libexpr/function-trace.cc +++ b/src/libexpr/function-trace.cc @@ -4,7 +4,7 @@ namespace nix { void FunctionCallTrace::preFunctionCallHook( - const EvalState & state, const Value & v, std::span args, const PosIdx pos) + EvalState & state, const Value & v, std::span args, const PosIdx pos) { auto duration = std::chrono::high_resolution_clock::now().time_since_epoch(); auto ns = std::chrono::duration_cast(duration); @@ -12,7 +12,7 @@ void FunctionCallTrace::preFunctionCallHook( } void FunctionCallTrace::postFunctionCallHook( - const EvalState & state, const Value & v, std::span args, const PosIdx pos) + EvalState & state, const Value & v, std::span args, const PosIdx pos) { auto duration = std::chrono::high_resolution_clock::now().time_since_epoch(); auto ns = std::chrono::duration_cast(duration); diff --git a/src/libexpr/include/nix/expr/eval-profiler.hh b/src/libexpr/include/nix/expr/eval-profiler.hh index 9fee4ef3559..21629eebc14 100644 --- a/src/libexpr/include/nix/expr/eval-profiler.hh +++ b/src/libexpr/include/nix/expr/eval-profiler.hh @@ -62,8 +62,7 @@ public: * @param args Function arguments. * @param pos Function position. */ - virtual void - preFunctionCallHook(const EvalState & state, const Value & v, std::span args, const PosIdx pos); + virtual void preFunctionCallHook(EvalState & state, const Value & v, std::span args, const PosIdx pos); /** * Hook called on EvalState::callFunction exit. @@ -74,8 +73,7 @@ public: * @param args Function arguments. * @param pos Function position. */ - virtual void - postFunctionCallHook(const EvalState & state, const Value & v, std::span args, const PosIdx pos); + virtual void postFunctionCallHook(EvalState & state, const Value & v, std::span args, const PosIdx pos); virtual ~EvalProfiler() = default; @@ -106,12 +104,11 @@ public: void addProfiler(ref profiler); [[gnu::noinline]] void - preFunctionCallHook(const EvalState & state, const Value & v, std::span args, const PosIdx pos) override; + preFunctionCallHook(EvalState & state, const Value & v, std::span args, const PosIdx pos) override; [[gnu::noinline]] void - postFunctionCallHook(const EvalState & state, const Value & v, std::span args, const PosIdx pos) override; + postFunctionCallHook(EvalState & state, const Value & v, std::span args, const PosIdx pos) override; }; -ref -makeSampleStackProfiler(const EvalState & state, std::filesystem::path profileFile, uint64_t frequency); +ref makeSampleStackProfiler(EvalState & state, std::filesystem::path profileFile, uint64_t frequency); } diff --git a/src/libexpr/include/nix/expr/function-trace.hh b/src/libexpr/include/nix/expr/function-trace.hh index 9187cac6324..ed1fc645203 100644 --- a/src/libexpr/include/nix/expr/function-trace.hh +++ b/src/libexpr/include/nix/expr/function-trace.hh @@ -17,9 +17,9 @@ public: FunctionCallTrace() = default; [[gnu::noinline]] void - preFunctionCallHook(const EvalState & state, const Value & v, std::span args, const PosIdx pos) override; + preFunctionCallHook(EvalState & state, const Value & v, std::span args, const PosIdx pos) override; [[gnu::noinline]] void - postFunctionCallHook(const EvalState & state, const Value & v, std::span args, const PosIdx pos) override; + postFunctionCallHook(EvalState & state, const Value & v, std::span args, const PosIdx pos) override; }; } From a76c76a9d54029a1485b668be56ca323a5135d70 Mon Sep 17 00:00:00 2001 From: Sergei Zimmerman Date: Sun, 25 May 2025 15:14:31 +0000 Subject: [PATCH 067/218] libexpr: Make `getAttr` a member function of `EvalState` --- src/libexpr/eval.cc | 10 ++++++++ src/libexpr/include/nix/expr/eval.hh | 5 ++++ src/libexpr/primops.cc | 35 ++++++++-------------------- 3 files changed, 25 insertions(+), 25 deletions(-) diff --git a/src/libexpr/eval.cc b/src/libexpr/eval.cc index ff7df249a02..af324a7defc 100644 --- a/src/libexpr/eval.cc +++ b/src/libexpr/eval.cc @@ -2243,6 +2243,16 @@ bool EvalState::forceBool(Value & v, const PosIdx pos, std::string_view errorCtx return v.boolean(); } +Bindings::const_iterator EvalState::getAttr(Symbol attrSym, const Bindings * attrSet, std::string_view errorCtx) +{ + auto value = attrSet->find(attrSym); + if (value == attrSet->end()) { + error("attribute '%s' missing", symbols[attrSym]) + .withTrace(noPos, errorCtx) + .debugThrow(); + } + return value; +} bool EvalState::isFunctor(const Value & fun) const { diff --git a/src/libexpr/include/nix/expr/eval.hh b/src/libexpr/include/nix/expr/eval.hh index de865ae7bc4..a4347b3e15a 100644 --- a/src/libexpr/include/nix/expr/eval.hh +++ b/src/libexpr/include/nix/expr/eval.hh @@ -542,6 +542,11 @@ public: std::string_view forceString(Value & v, NixStringContext & context, const PosIdx pos, std::string_view errorCtx, const ExperimentalFeatureSettings & xpSettings = experimentalFeatureSettings); std::string_view forceStringNoCtx(Value & v, const PosIdx pos, std::string_view errorCtx); + /** + * Get attribute from an attribute set and throw an error if it doesn't exist. + */ + Bindings::const_iterator getAttr(Symbol attrSym, const Bindings * attrSet, std::string_view errorCtx); + template [[gnu::noinline]] void addErrorTrace(Error & e, const Args & ... formatArgs) const; diff --git a/src/libexpr/primops.cc b/src/libexpr/primops.cc index c38eed26790..564df90c202 100644 --- a/src/libexpr/primops.cc +++ b/src/libexpr/primops.cc @@ -670,26 +670,12 @@ struct CompareValues typedef std::list> ValueList; - -static Bindings::const_iterator getAttr( - EvalState & state, - Symbol attrSym, - const Bindings * attrSet, - std::string_view errorCtx) -{ - auto value = attrSet->find(attrSym); - if (value == attrSet->end()) { - state.error("attribute '%s' missing", state.symbols[attrSym]).withTrace(noPos, errorCtx).debugThrow(); - } - return value; -} - static void prim_genericClosure(EvalState & state, const PosIdx pos, Value * * args, Value & v) { state.forceAttrs(*args[0], noPos, "while evaluating the first argument passed to builtins.genericClosure"); /* Get the start set. */ - auto startSet = getAttr(state, state.sStartSet, args[0]->attrs(), "in the attrset passed as argument to builtins.genericClosure"); + auto startSet = state.getAttr(state.sStartSet, args[0]->attrs(), "in the attrset passed as argument to builtins.genericClosure"); state.forceList(*startSet->value, noPos, "while evaluating the 'startSet' attribute passed as argument to builtins.genericClosure"); @@ -703,7 +689,7 @@ static void prim_genericClosure(EvalState & state, const PosIdx pos, Value * * a } /* Get the operator. */ - auto op = getAttr(state, state.sOperator, args[0]->attrs(), "in the attrset passed as argument to builtins.genericClosure"); + auto op = state.getAttr(state.sOperator, args[0]->attrs(), "in the attrset passed as argument to builtins.genericClosure"); state.forceFunction(*op->value, noPos, "while evaluating the 'operator' attribute passed as argument to builtins.genericClosure"); /* Construct the closure by applying the operator to elements of @@ -720,7 +706,7 @@ static void prim_genericClosure(EvalState & state, const PosIdx pos, Value * * a state.forceAttrs(*e, noPos, "while evaluating one of the elements generated by (or initially passed to) builtins.genericClosure"); - auto key = getAttr(state, state.sKey, e->attrs(), "in one of the attrsets generated by (or initially passed to) builtins.genericClosure"); + auto key = state.getAttr(state.sKey, e->attrs(), "in one of the attrsets generated by (or initially passed to) builtins.genericClosure"); state.forceValue(*key->value, noPos); if (!doneKeys.insert(key->value).second) continue; @@ -1203,7 +1189,7 @@ static void prim_derivationStrict(EvalState & state, const PosIdx pos, Value * * auto attrs = args[0]->attrs(); /* Figure out the name first (for stack backtraces). */ - auto nameAttr = getAttr(state, state.sName, attrs, "in the attrset passed as argument to builtins.derivationStrict"); + auto nameAttr = state.getAttr(state.sName, attrs, "in the attrset passed as argument to builtins.derivationStrict"); std::string drvName; try { @@ -1893,7 +1879,7 @@ static void prim_findFile(EvalState & state, const PosIdx pos, Value * * args, V if (i != v2->attrs()->end()) prefix = state.forceStringNoCtx(*i->value, pos, "while evaluating the `prefix` attribute of an element of the list passed to builtins.findFile"); - i = getAttr(state, state.sPath, v2->attrs(), "in an element of the __nixPath"); + i = state.getAttr(state.sPath, v2->attrs(), "in an element of the __nixPath"); NixStringContext context; auto path = state.coerceToString(pos, *i->value, context, @@ -2782,8 +2768,7 @@ void prim_getAttr(EvalState & state, const PosIdx pos, Value * * args, Value & v { auto attr = state.forceStringNoCtx(*args[0], pos, "while evaluating the first argument passed to builtins.getAttr"); state.forceAttrs(*args[1], pos, "while evaluating the second argument passed to builtins.getAttr"); - auto i = getAttr( - state, + auto i = state.getAttr( state.symbols.create(attr), args[1]->attrs(), "in the attribute set under consideration" @@ -2973,13 +2958,13 @@ static void prim_listToAttrs(EvalState & state, const PosIdx pos, Value * * args for (auto v2 : args[0]->listItems()) { state.forceAttrs(*v2, pos, "while evaluating an element of the list passed to builtins.listToAttrs"); - auto j = getAttr(state, state.sName, v2->attrs(), "in a {name=...; value=...;} pair"); + auto j = state.getAttr(state.sName, v2->attrs(), "in a {name=...; value=...;} pair"); auto name = state.forceStringNoCtx(*j->value, j->pos, "while evaluating the `name` attribute of an element of the list passed to builtins.listToAttrs"); auto sym = state.symbols.create(name); if (seen.insert(sym).second) { - auto j2 = getAttr(state, state.sValue, v2->attrs(), "in a {name=...; value=...;} pair"); + auto j2 = state.getAttr(state.sValue, v2->attrs(), "in a {name=...; value=...;} pair"); attrs.insert(sym, j2->value, j2->pos); } } @@ -4224,7 +4209,7 @@ static void prim_convertHash(EvalState & state, const PosIdx pos, Value * * args state.forceAttrs(*args[0], pos, "while evaluating the first argument passed to builtins.convertHash"); auto inputAttrs = args[0]->attrs(); - auto iteratorHash = getAttr(state, state.symbols.create("hash"), inputAttrs, "while locating the attribute 'hash'"); + auto iteratorHash = state.getAttr(state.symbols.create("hash"), inputAttrs, "while locating the attribute 'hash'"); auto hash = state.forceStringNoCtx(*iteratorHash->value, pos, "while evaluating the attribute 'hash'"); auto iteratorHashAlgo = inputAttrs->get(state.symbols.create("hashAlgo")); @@ -4232,7 +4217,7 @@ static void prim_convertHash(EvalState & state, const PosIdx pos, Value * * args if (iteratorHashAlgo) ha = parseHashAlgo(state.forceStringNoCtx(*iteratorHashAlgo->value, pos, "while evaluating the attribute 'hashAlgo'")); - auto iteratorToHashFormat = getAttr(state, state.symbols.create("toHashFormat"), args[0]->attrs(), "while locating the attribute 'toHashFormat'"); + auto iteratorToHashFormat = state.getAttr(state.symbols.create("toHashFormat"), args[0]->attrs(), "while locating the attribute 'toHashFormat'"); HashFormat hf = parseHashFormat(state.forceStringNoCtx(*iteratorToHashFormat->value, pos, "while evaluating the attribute 'toHashFormat'")); v.mkString(Hash::parseAny(hash, ha).to_string(hf, hf == HashFormat::SRI)); From 9e97ecabb6e283f5451957a29348bf6ebd45c581 Mon Sep 17 00:00:00 2001 From: Sergei Zimmerman Date: Sun, 25 May 2025 15:48:35 +0000 Subject: [PATCH 068/218] libexpr: Include derivation names in the call stack profile This makes the profiler much more useful by actually distiguishing different derivations being evaluated. This does make the implementation a bit more convoluted, but I think it's worth it. --- src/libexpr/eval-profiler.cc | 68 +++++++++++++++++++++---- tests/functional/flamegraph-profiler.sh | 10 ++++ 2 files changed, 68 insertions(+), 10 deletions(-) diff --git a/src/libexpr/eval-profiler.cc b/src/libexpr/eval-profiler.cc index bf2442c31ae..c72737f7394 100644 --- a/src/libexpr/eval-profiler.cc +++ b/src/libexpr/eval-profiler.cc @@ -95,6 +95,14 @@ struct FunctorFrameInfo auto operator<=>(const FunctorFrameInfo & rhs) const = default; }; +struct DerivationStrictFrameInfo +{ + PosIdx callPos = noPos; + std::string drvName; + std::ostream & symbolize(const EvalState & state, std::ostream & os, PosCache & posCache) const; + auto operator<=>(const DerivationStrictFrameInfo & rhs) const = default; +}; + /** Fallback frame info. */ struct GenericFrameInfo { @@ -103,7 +111,8 @@ struct GenericFrameInfo auto operator<=>(const GenericFrameInfo & rhs) const = default; }; -using FrameInfo = std::variant; +using FrameInfo = + std::variant; using FrameStack = std::vector; /** @@ -121,6 +130,8 @@ class SampleStack : public EvalProfiler return Hooks().set(preFunctionCall).set(postFunctionCall); } + FrameInfo getPrimOpFrameInfo(const PrimOp & primOp, std::span args, PosIdx pos); + public: SampleStack(EvalState & state, std::filesystem::path profileFile, std::chrono::nanoseconds period) : state(state) @@ -142,14 +153,13 @@ class SampleStack : public EvalProfiler void maybeSaveProfile(std::chrono::time_point now); void saveProfile(); - FrameInfo getFrameInfoFromValueAndPos(const Value & v, PosIdx pos); + FrameInfo getFrameInfoFromValueAndPos(const Value & v, std::span args, PosIdx pos); SampleStack(SampleStack &&) = default; SampleStack & operator=(SampleStack &&) = delete; SampleStack(const SampleStack &) = delete; SampleStack & operator=(const SampleStack &) = delete; ~SampleStack(); - private: /** Hold on to an instance of EvalState for symbolizing positions. */ EvalState & state; @@ -163,15 +173,41 @@ class SampleStack : public EvalProfiler PosCache posCache; }; -FrameInfo SampleStack::getFrameInfoFromValueAndPos(const Value & v, PosIdx pos) +FrameInfo SampleStack::getPrimOpFrameInfo(const PrimOp & primOp, std::span args, PosIdx pos) +{ + auto derivationInfo = [&]() -> std::optional { + /* Here we rely a bit on the implementation details of libexpr/primops/derivation.nix + and derivationStrict primop. This is not ideal, but is necessary for + the usefulness of the profiler. This might actually affect the evaluation, + but the cost shouldn't be that high as to make the traces entirely inaccurate. */ + if (primOp.name == "derivationStrict") { + try { + /* Error context strings don't actually matter, since we ignore all eval errors. */ + state.forceAttrs(*args[0], pos, ""); + auto attrs = args[0]->attrs(); + auto nameAttr = state.getAttr(state.sName, attrs, ""); + auto drvName = std::string(state.forceStringNoCtx(*nameAttr->value, pos, "")); + return DerivationStrictFrameInfo{.callPos = pos, .drvName = std::move(drvName)}; + } catch (...) { + /* Ignore all errors, since those will be diagnosed by the evaluator itself. */ + } + } + + return std::nullopt; + }(); + + return derivationInfo.value_or(PrimOpFrameInfo{.expr = &primOp, .callPos = pos}); +} + +FrameInfo SampleStack::getFrameInfoFromValueAndPos(const Value & v, std::span args, PosIdx pos) { /* NOTE: No actual references to garbage collected values are not held in the profiler. */ if (v.isLambda()) return LambdaFrameInfo{.expr = v.payload.lambda.fun, .callPos = pos}; - else if (v.isPrimOp()) - return PrimOpFrameInfo{.expr = v.primOp(), .callPos = pos}; - else if (v.isPrimOpApp()) + else if (v.isPrimOp()) { + return getPrimOpFrameInfo(*v.primOp(), args, pos); + } else if (v.isPrimOpApp()) /* Resolve primOp eagerly. Must not hold on to a reference to a Value. */ return PrimOpFrameInfo{.expr = v.primOpAppPrimOp(), .callPos = pos}; else if (state.isFunctor(v)) { @@ -186,10 +222,10 @@ FrameInfo SampleStack::getFrameInfoFromValueAndPos(const Value & v, PosIdx pos) return GenericFrameInfo{.pos = pos}; } -[[gnu::noinline]] void SampleStack::preFunctionCallHook( - EvalState & state, const Value & v, [[maybe_unused]] std::span args, const PosIdx pos) +[[gnu::noinline]] void +SampleStack::preFunctionCallHook(EvalState & state, const Value & v, std::span args, const PosIdx pos) { - stack.push_back(getFrameInfoFromValueAndPos(v, pos)); + stack.push_back(getFrameInfoFromValueAndPos(v, args, pos)); auto now = std::chrono::high_resolution_clock::now(); @@ -246,6 +282,18 @@ std::ostream & PrimOpFrameInfo::symbolize(const EvalState & state, std::ostream return os; } +std::ostream & +DerivationStrictFrameInfo::symbolize(const EvalState & state, std::ostream & os, PosCache & posCache) const +{ + /* Sometimes callsite position can have an unresolved origin, which + leads to confusing «none»:0 locations in the profile. */ + auto pos = posCache.lookup(callPos); + if (!std::holds_alternative(pos.origin)) + os << posCache.lookup(callPos) << ":"; + os << "primop derivationStrict:" << drvName; + return os; +} + void SampleStack::maybeSaveProfile(std::chrono::time_point now) { if (now - lastDump >= profileDumpInterval) diff --git a/tests/functional/flamegraph-profiler.sh b/tests/functional/flamegraph-profiler.sh index 0c35037a8d0..b074507e85f 100755 --- a/tests/functional/flamegraph-profiler.sh +++ b/tests/functional/flamegraph-profiler.sh @@ -89,3 +89,13 @@ expect_trace 'let f2 = (x: x); in f2 1 2' " expect_trace '1 2' " «string»:1:1 1 " + +# Derivation +expect_trace 'builtins.derivationStrict { name = "somepackage"; }' " +«string»:1:1:primop derivationStrict:somepackage 1 +" + +# Derivation without name attr +expect_trace 'builtins.derivationStrict { }' " +«string»:1:1:primop derivationStrict 1 +" From 114de63d883b54fac7085367a2a24cf894a683a1 Mon Sep 17 00:00:00 2001 From: Sergei Zimmerman Date: Sun, 25 May 2025 20:14:11 +0000 Subject: [PATCH 069/218] Fix various typos in source code This only touches code comments, class names, documentation, enumeration names and tests. --- .../include/nix/cmd/common-eval-args.hh | 6 +++--- src/libcmd/repl.cc | 2 +- src/libexpr-tests/error_traces.cc | 10 +++++----- src/libexpr-tests/trivial.cc | 2 +- src/libexpr/eval.cc | 6 +++--- src/libexpr/include/nix/expr/eval-settings.hh | 2 +- src/libexpr/lexer-helpers.hh | 2 +- src/libexpr/primops.cc | 20 +++++++++---------- src/libexpr/primops/fetchTree.cc | 2 +- src/libfetchers/git-utils.cc | 2 +- src/libfetchers/github.cc | 2 +- src/libflake-c/nix_api_flake.h | 2 +- src/libflake/flake.cc | 14 ++++++------- src/libflake/include/nix/flake/flake.hh | 2 +- src/libmain/progress-bar.cc | 4 ++-- src/libstore-tests/machines.cc | 2 +- src/libstore-tests/outputs-spec.cc | 4 ++-- .../build/derivation-building-goal.cc | 4 ++-- src/libstore/build/derivation-goal.cc | 6 +++--- src/libstore/derivations.cc | 4 ++-- src/libstore/gc.cc | 2 +- .../include/nix/store/build-result.hh | 2 +- .../store/build/derivation-building-misc.hh | 2 +- .../nix/store/build/derivation-goal.hh | 4 ++-- .../include/nix/store/common-protocol-impl.hh | 2 +- .../include/nix/store/common-protocol.hh | 4 ++-- .../include/nix/store/derivation-options.hh | 2 +- src/libstore/include/nix/store/derivations.hh | 2 +- src/libstore/include/nix/store/gc-store.hh | 2 +- .../include/nix/store/outputs-spec.hh | 4 ++-- src/libstore/include/nix/store/path-info.hh | 2 +- .../include/nix/store/serve-protocol-impl.hh | 2 +- src/libstore/include/nix/store/ssh.hh | 2 +- src/libstore/include/nix/store/store-api.hh | 4 ++-- .../include/nix/store/store-dir-config.hh | 2 +- .../include/nix/store/store-reference.hh | 2 +- .../include/nix/store/store-registration.hh | 4 ++-- .../nix/store/worker-protocol-connection.hh | 4 ++-- .../include/nix/store/worker-protocol-impl.hh | 2 +- .../include/nix/store/worker-protocol.hh | 2 +- src/libstore/profiles.cc | 2 +- src/libstore/ssh-store.cc | 2 +- src/libstore/store-api.cc | 6 +++--- src/libstore/unix/build/derivation-builder.cc | 2 +- src/libstore/windows/pathlocks.cc | 2 +- src/libutil-tests/logging.cc | 2 +- src/libutil/hash.cc | 2 +- src/libutil/hilite.cc | 2 +- src/libutil/include/nix/util/args.hh | 2 +- src/libutil/include/nix/util/comparator.hh | 2 +- .../include/nix/util/executable-path.hh | 4 ++-- .../include/nix/util/file-descriptor.hh | 2 +- .../include/nix/util/file-path-impl.hh | 4 ++-- src/libutil/include/nix/util/hash.hh | 4 ++-- src/libutil/include/nix/util/pos-idx.hh | 2 +- src/libutil/include/nix/util/serialise.hh | 2 +- .../include/nix/util/source-accessor.hh | 2 +- src/libutil/unix/file-descriptor.cc | 2 +- src/libutil/windows/file-descriptor.cc | 2 +- src/nix-store/nix-store.cc | 2 +- src/nix/flake.cc | 2 +- src/nix/meson.build | 4 ++-- src/nix/unix/daemon.cc | 2 +- 63 files changed, 104 insertions(+), 104 deletions(-) diff --git a/src/libcmd/include/nix/cmd/common-eval-args.hh b/src/libcmd/include/nix/cmd/common-eval-args.hh index 6f3367e58e9..71634042555 100644 --- a/src/libcmd/include/nix/cmd/common-eval-args.hh +++ b/src/libcmd/include/nix/cmd/common-eval-args.hh @@ -23,17 +23,17 @@ struct SourcePath; namespace flake { struct Settings; } /** - * @todo Get rid of global setttings variables + * @todo Get rid of global settings variables */ extern fetchers::Settings fetchSettings; /** - * @todo Get rid of global setttings variables + * @todo Get rid of global settings variables */ extern EvalSettings evalSettings; /** - * @todo Get rid of global setttings variables + * @todo Get rid of global settings variables */ extern flake::Settings flakeSettings; diff --git a/src/libcmd/repl.cc b/src/libcmd/repl.cc index f9ac59d36f5..07968fa4360 100644 --- a/src/libcmd/repl.cc +++ b/src/libcmd/repl.cc @@ -294,7 +294,7 @@ StringSet NixRepl::completePrefix(const std::string & prefix) } catch (BadURL & e) { // Quietly ignore BadURL flake-related errors. } catch (FileNotFound & e) { - // Quietly ignore non-existent file beeing `import`-ed. + // Quietly ignore non-existent file being `import`-ed. } } diff --git a/src/libexpr-tests/error_traces.cc b/src/libexpr-tests/error_traces.cc index a7522278d94..32e49efe6c9 100644 --- a/src/libexpr-tests/error_traces.cc +++ b/src/libexpr-tests/error_traces.cc @@ -458,7 +458,7 @@ namespace nix { HintFmt("expected a function but found %s: %s", "a list", Uncolored("[ ]")), HintFmt("while evaluating the first argument passed to builtins.filterSource")); - // Usupported by store "dummy" + // Unsupported by store "dummy" // ASSERT_TRACE2("filterSource (_: 1) ./.", // TypeError, @@ -636,7 +636,7 @@ namespace nix { HintFmt("expected a set but found %s: %s", "a list", Uncolored("[ ]")), HintFmt("while evaluating the second argument passed to builtins.mapAttrs")); - // XXX: defered + // XXX: deferred // ASSERT_TRACE2("mapAttrs \"\" { foo.bar = 1; }", // TypeError, // HintFmt("attempt to call something which is not a function but %s", "a string"), @@ -666,9 +666,9 @@ namespace nix { HintFmt("expected a set but found %s: %s", "an integer", Uncolored(ANSI_CYAN "1" ANSI_NORMAL)), HintFmt("while evaluating a value of the list passed as second argument to builtins.zipAttrsWith")); - // XXX: How to properly tell that the fucntion takes two arguments ? + // XXX: How to properly tell that the function takes two arguments ? // The same question also applies to sort, and maybe others. - // Due to lazyness, we only create a thunk, and it fails later on. + // Due to laziness, we only create a thunk, and it fails later on. // ASSERT_TRACE2("zipAttrsWith (_: 1) [ { foo = 1; } ]", // TypeError, // HintFmt("attempt to call something which is not a function but %s", "an integer"), @@ -877,7 +877,7 @@ namespace nix { HintFmt("expected a function but found %s: %s", "an integer", Uncolored(ANSI_CYAN "1" ANSI_NORMAL)), HintFmt("while evaluating the first argument passed to builtins.genList")); - // XXX: defered + // XXX: deferred // ASSERT_TRACE2("genList (x: x + \"foo\") 2 #TODO", // TypeError, // HintFmt("cannot add %s to an integer", "a string"), diff --git a/src/libexpr-tests/trivial.cc b/src/libexpr-tests/trivial.cc index 50a8f29f83d..6eabad6d7a4 100644 --- a/src/libexpr-tests/trivial.cc +++ b/src/libexpr-tests/trivial.cc @@ -143,7 +143,7 @@ namespace nix { // Usually Nix rejects duplicate keys in an attrset but it does allow // so if it is an attribute set that contains disjoint sets of keys. // The below is equivalent to `{a.b = 1; a.c = 2; }`. - // The attribute set `a` will be a Thunk at first as the attribuets + // The attribute set `a` will be a Thunk at first as the attributes // have to be merged (or otherwise computed) and that is done in a lazy // manner. diff --git a/src/libexpr/eval.cc b/src/libexpr/eval.cc index af324a7defc..1a067e75cdb 100644 --- a/src/libexpr/eval.cc +++ b/src/libexpr/eval.cc @@ -657,7 +657,7 @@ std::optional EvalState::getDoc(Value & v) Value & functor = *v.attrs()->find(sFunctor)->value; Value * vp[] = {&v}; Value partiallyApplied; - // The first paramater is not user-provided, and may be + // The first parameter is not user-provided, and may be // handled by code that is opaque to the user, like lib.const = x: y: y; // So preferably we show docs that are relevant to the // "partially applied" function returned by e.g. `const`. @@ -2495,7 +2495,7 @@ SourcePath EvalState::coerceToPath(const PosIdx pos, Value & v, NixStringContext } } - /* Any other value should be coercable to a string, interpreted + /* Any other value should be coercible to a string, interpreted relative to the root filesystem. */ auto path = coerceToString(pos, v, context, errorCtx, false, false, true).toOwned(); if (path == "" || path[0] != '/') @@ -2866,7 +2866,7 @@ bool EvalState::fullGC() { GC_gcollect(); // Check that it ran. We might replace this with a version that uses more // of the boehm API to get this reliably, at a maintenance cost. - // We use a 1K margin because technically this has a race condtion, but we + // We use a 1K margin because technically this has a race condition, but we // probably won't encounter it in practice, because the CLI isn't concurrent // like that. return GC_get_bytes_since_gc() < 1024; diff --git a/src/libexpr/include/nix/expr/eval-settings.hh b/src/libexpr/include/nix/expr/eval-settings.hh index f1367b9bb28..284e02a9b36 100644 --- a/src/libexpr/include/nix/expr/eval-settings.hh +++ b/src/libexpr/include/nix/expr/eval-settings.hh @@ -13,7 +13,7 @@ struct PrimOp; struct EvalSettings : Config { /** - * Function used to interpet look path entries of a given scheme. + * Function used to interpret look path entries of a given scheme. * * The argument is the non-scheme part of the lookup path entry (see * `LookupPathHooks` below). diff --git a/src/libexpr/lexer-helpers.hh b/src/libexpr/lexer-helpers.hh index d40f7b874e8..225eb157a96 100644 --- a/src/libexpr/lexer-helpers.hh +++ b/src/libexpr/lexer-helpers.hh @@ -2,7 +2,7 @@ #include -// inluding the generated headers twice leads to errors +// including the generated headers twice leads to errors #ifndef BISON_HEADER # include "lexer-tab.hh" # include "parser-tab.hh" diff --git a/src/libexpr/primops.cc b/src/libexpr/primops.cc index 564df90c202..c4e6feb2800 100644 --- a/src/libexpr/primops.cc +++ b/src/libexpr/primops.cc @@ -890,7 +890,7 @@ static void prim_ceil(EvalState & state, const PosIdx pos, Value * * args, Value auto arg = args[0]->integer(); auto res = v.integer(); if (arg != res) { - state.error("Due to a bug (see https://github.com/NixOS/nix/issues/12899) a loss of precision occured in previous Nix versions because the NixInt argument %1% was rounded to %2%.\n\tFuture Nix versions might implement the correct behavior.", arg, res).atPos(pos).debugThrow(); + state.error("Due to a bug (see https://github.com/NixOS/nix/issues/12899) a loss of precision occurred in previous Nix versions because the NixInt argument %1% was rounded to %2%.\n\tFuture Nix versions might implement the correct behavior.", arg, res).atPos(pos).debugThrow(); } } } @@ -931,7 +931,7 @@ static void prim_floor(EvalState & state, const PosIdx pos, Value * * args, Valu auto arg = args[0]->integer(); auto res = v.integer(); if (arg != res) { - state.error("Due to a bug (see https://github.com/NixOS/nix/issues/12899) a loss of precision occured in previous Nix versions because the NixInt argument %1% was rounded to %2%.\n\tFuture Nix versions might implement the correct behavior.", arg, res).atPos(pos).debugThrow(); + state.error("Due to a bug (see https://github.com/NixOS/nix/issues/12899) a loss of precision occurred in previous Nix versions because the NixInt argument %1% was rounded to %2%.\n\tFuture Nix versions might implement the correct behavior.", arg, res).atPos(pos).debugThrow(); } } } @@ -965,7 +965,7 @@ static void prim_tryEval(EvalState & state, const PosIdx pos, Value * * args, Va ReplExitStatus (* savedDebugRepl)(ref es, const ValMap & extraEnv) = nullptr; if (state.debugRepl && state.settings.ignoreExceptionsDuringTry) { - /* to prevent starting the repl from exceptions withing a tryEval, null it. */ + /* to prevent starting the repl from exceptions within a tryEval, null it. */ savedDebugRepl = state.debugRepl; state.debugRepl = nullptr; } @@ -2172,7 +2172,7 @@ static RegisterPrimOp primop_outputOf({ [input placeholder string](@docroot@/store/derivation/index.md#input-placeholder) if needed. - If the derivation has a statically-known output path (i.e. the derivation output is input-addressed, or fixed content-addresed), the output path will just be returned. + If the derivation has a statically-known output path (i.e. the derivation output is input-addressed, or fixed content-addressed), the output path will just be returned. But if the derivation is content-addressed or if the derivation is itself not-statically produced (i.e. is the output of another derivation), an input placeholder will be returned instead. *`derivation reference`* must be a string that may contain a regular store path to a derivation, or may be an input placeholder reference. @@ -2815,7 +2815,7 @@ static RegisterPrimOp primop_unsafeGetAttrPos(PrimOp { .fun = prim_unsafeGetAttrPos, }); -// access to exact position information (ie, line and colum numbers) is deferred +// access to exact position information (ie, line and column numbers) is deferred // due to the cost associated with calculating that information and how rarely // it is used in practice. this is achieved by creating thunks to otherwise // inaccessible primops that are not exposed as __op or under builtins to turn @@ -2827,7 +2827,7 @@ static RegisterPrimOp primop_unsafeGetAttrPos(PrimOp { // but each type of thunk has an associated runtime cost in the current evaluator. // as with black holes this cost is too high to justify another thunk type to check // for in the very hot path that is forceValue. -static struct LazyPosAcessors { +static struct LazyPosAccessors { PrimOp primop_lineOfPos{ .arity = 1, .fun = [] (EvalState & state, PosIdx pos, Value * * args, Value & v) { @@ -2843,7 +2843,7 @@ static struct LazyPosAcessors { Value lineOfPos, columnOfPos; - LazyPosAcessors() + LazyPosAccessors() { lineOfPos.mkPrimOp(&primop_lineOfPos); columnOfPos.mkPrimOp(&primop_columnOfPos); @@ -3623,7 +3623,7 @@ static void prim_genList(EvalState & state, const PosIdx pos, Value * * args, Va size_t len = size_t(len_); - // More strict than striclty (!) necessary, but acceptable + // More strict than strictly (!) necessary, but acceptable // as evaluating map without accessing any values makes little sense. state.forceFunction(*args[0], noPos, "while evaluating the first argument passed to builtins.genList"); @@ -4126,7 +4126,7 @@ static void prim_substring(EvalState & state, const PosIdx pos, Value * * args, } // Special-case on empty substring to avoid O(n) strlen - // This allows for the use of empty substrings to efficently capture string context + // This allows for the use of empty substrings to efficiently capture string context if (len == 0) { state.forceValue(*args[2], pos); if (args[2]->type() == nString) { @@ -4442,7 +4442,7 @@ void prim_split(EvalState & state, const PosIdx pos, Value * * args, Value & v) // Add a list for matched substrings. const size_t slen = match.size() - 1; - // Start at 1, beacause the first match is the whole string. + // Start at 1, because the first match is the whole string. auto list2 = state.buildList(slen); for (const auto & [si, v2] : enumerate(list2)) { if (!match[si + 1].matched) diff --git a/src/libexpr/primops/fetchTree.cc b/src/libexpr/primops/fetchTree.cc index f262507dffa..bdd65b4c667 100644 --- a/src/libexpr/primops/fetchTree.cc +++ b/src/libexpr/primops/fetchTree.cc @@ -305,7 +305,7 @@ static RegisterPrimOp primop_fetchTree({ - `"tarball"` Download a tar archive and extract it into the Nix store. - This has the same underyling implementation as [`builtins.fetchTarball`](@docroot@/language/builtins.md#builtins-fetchTarball) + This has the same underlying implementation as [`builtins.fetchTarball`](@docroot@/language/builtins.md#builtins-fetchTarball) - `url` (String, required) diff --git a/src/libfetchers/git-utils.cc b/src/libfetchers/git-utils.cc index 09508c6026c..1789cb97d8c 100644 --- a/src/libfetchers/git-utils.cc +++ b/src/libfetchers/git-utils.cc @@ -587,7 +587,7 @@ struct GitRepoImpl : GitRepo, std::enable_shared_from_this }); /* Evaluate result through status code and checking if public - key fingerprints appear on stderr. This is neccessary + key fingerprints appear on stderr. This is necessary because the git command might also succeed due to the commit being signed by gpg keys that are present in the users key agent. */ diff --git a/src/libfetchers/github.cc b/src/libfetchers/github.cc index 511afabe979..7a902d816d0 100644 --- a/src/libfetchers/github.cc +++ b/src/libfetchers/github.cc @@ -175,7 +175,7 @@ struct GitArchiveInputScheme : InputScheme return input; } - // Search for the longest possible match starting from the begining and ending at either the end or a path segment. + // Search for the longest possible match starting from the beginning and ending at either the end or a path segment. std::optional getAccessToken(const fetchers::Settings & settings, const std::string & host, const std::string & url) const override { auto tokens = settings.accessTokens.get(); diff --git a/src/libflake-c/nix_api_flake.h b/src/libflake-c/nix_api_flake.h index f5b9dc542b1..a1a7060a614 100644 --- a/src/libflake-c/nix_api_flake.h +++ b/src/libflake-c/nix_api_flake.h @@ -27,7 +27,7 @@ extern "C" { typedef struct nix_flake_settings nix_flake_settings; /** - * @brief Context and paramaters for parsing a flake reference + * @brief Context and parameters for parsing a flake reference * @see nix_flake_reference_parse_flags_free * @see nix_flake_reference_parse_string */ diff --git a/src/libflake/flake.cc b/src/libflake/flake.cc index 3277cc05bad..fc27df8874e 100644 --- a/src/libflake/flake.cc +++ b/src/libflake/flake.cc @@ -531,7 +531,7 @@ LockedFlake lockFlake( /* Resolve relative 'path:' inputs relative to the source path of the overrider. */ - auto overridenSourcePath = hasOverride ? i->second.sourcePath : sourcePath; + auto overriddenSourcePath = hasOverride ? i->second.sourcePath : sourcePath; /* Respect the "flakeness" of the input even if we override it. */ @@ -552,7 +552,7 @@ LockedFlake lockFlake( assert(input.ref); - auto overridenParentPath = + auto overriddenParentPath = input.ref->input.isRelative() ? std::optional(hasOverride ? i->second.parentInputAttrPath : inputAttrPathPrefix) : std::nullopt; @@ -561,8 +561,8 @@ LockedFlake lockFlake( { if (auto relativePath = input.ref->input.isRelative()) { return SourcePath { - overridenSourcePath.accessor, - CanonPath(*relativePath, overridenSourcePath.path.parent().value()) + overriddenSourcePath.accessor, + CanonPath(*relativePath, overriddenSourcePath.path.parent().value()) }; } else return std::nullopt; @@ -596,7 +596,7 @@ LockedFlake lockFlake( if (oldLock && oldLock->originalRef.canonicalize() == input.ref->canonicalize() - && oldLock->parentInputAttrPath == overridenParentPath + && oldLock->parentInputAttrPath == overriddenParentPath && !hasCliOverride) { debug("keeping existing input '%s'", inputAttrPathS); @@ -694,7 +694,7 @@ LockedFlake lockFlake( inputFlake.lockedRef, ref, true, - overridenParentPath); + overriddenParentPath); node->inputs.insert_or_assign(id, childNode); @@ -741,7 +741,7 @@ LockedFlake lockFlake( } }(); - auto childNode = make_ref(lockedRef, ref, false, overridenParentPath); + auto childNode = make_ref(lockedRef, ref, false, overriddenParentPath); nodePaths.emplace(childNode, path); diff --git a/src/libflake/include/nix/flake/flake.hh b/src/libflake/include/nix/flake/flake.hh index f90498b7cc3..ed34aa9c8db 100644 --- a/src/libflake/include/nix/flake/flake.hh +++ b/src/libflake/include/nix/flake/flake.hh @@ -129,7 +129,7 @@ struct LockedFlake /** * Source tree accessors for nodes that have been fetched in - * lockFlake(); in particular, the root node and the overriden + * lockFlake(); in particular, the root node and the overridden * inputs. */ std::map, SourcePath> nodePaths; diff --git a/src/libmain/progress-bar.cc b/src/libmain/progress-bar.cc index 23f5ff8f745..173ab876c2a 100644 --- a/src/libmain/progress-bar.cc +++ b/src/libmain/progress-bar.cc @@ -259,7 +259,7 @@ class ProgressBar : public Logger update(*state); } - /* Check whether an activity has an ancestore with the specified + /* Check whether an activity has an ancestor with the specified type. */ bool hasAncestor(State & state, ActivityType type, ActivityId act) { @@ -382,7 +382,7 @@ class ProgressBar : public Logger /** * Redraw, if the output has changed. * - * Excessive redrawing is noticable on slow terminals, and it interferes + * Excessive redrawing is noticeable on slow terminals, and it interferes * with text selection in some terminals, including libvte-based terminal * emulators. */ diff --git a/src/libstore-tests/machines.cc b/src/libstore-tests/machines.cc index 8873ff18337..f11866e0816 100644 --- a/src/libstore-tests/machines.cc +++ b/src/libstore-tests/machines.cc @@ -87,7 +87,7 @@ TEST(machines, getMachinesWithCommentsAndSemicolonSeparator) { TEST(machines, getMachinesWithFunnyWhitespace) { auto actual = Machine::parseConfig({}, - " # commment ; comment\n" + " # comment ; comment\n" " nix@scratchy.labs.cs.uu.nl ; nix@itchy.labs.cs.uu.nl \n" "\n \n" "\n ;;; \n" diff --git a/src/libstore-tests/outputs-spec.cc b/src/libstore-tests/outputs-spec.cc index a1c13d2f883..12f285e0d05 100644 --- a/src/libstore-tests/outputs-spec.cc +++ b/src/libstore-tests/outputs-spec.cc @@ -46,7 +46,7 @@ TEST(OutputsSpec, names_underscore) { ASSERT_EQ(expected.to_string(), str); } -TEST(OutputsSpec, names_numberic) { +TEST(OutputsSpec, names_numeric) { std::string_view str = "01"; OutputsSpec expected = OutputsSpec::Names { "01" }; ASSERT_EQ(OutputsSpec::parse(str), expected); @@ -126,7 +126,7 @@ TEST_DONT_PARSE(star_second, "^foo,*") #undef TEST_DONT_PARSE -TEST(ExtendedOutputsSpec, defeault) { +TEST(ExtendedOutputsSpec, default) { std::string_view str = "foo"; auto [prefix, extendedOutputsSpec] = ExtendedOutputsSpec::parse(str); ASSERT_EQ(prefix, "foo"); diff --git a/src/libstore/build/derivation-building-goal.cc b/src/libstore/build/derivation-building-goal.cc index 14c427b83a0..7335668e1ef 100644 --- a/src/libstore/build/derivation-building-goal.cc +++ b/src/libstore/build/derivation-building-goal.cc @@ -329,7 +329,7 @@ Goal::Co DerivationBuildingGoal::gaveUpOnSubstitution() auto take1 = get(resolvedResult.builtOutputs, outputName); if (take1) return *take1; - /* The above `get` should work. But sateful tracking of + /* The above `get` should work. But stateful tracking of outputs in resolvedResult, this can get out of sync with the store, which is our actual source of truth. For now we just check the store directly if it fails. */ @@ -1150,7 +1150,7 @@ std::pair DerivationBuildingGoal::checkPathValidity() for (auto & i : queryPartialDerivationOutputMap()) { auto initialOutput = get(initialOutputs, i.first); if (!initialOutput) - // this is an invalid output, gets catched with (!wantedOutputsLeft.empty()) + // this is an invalid output, gets caught with (!wantedOutputsLeft.empty()) continue; auto & info = *initialOutput; info.wanted = true; diff --git a/src/libstore/build/derivation-goal.cc b/src/libstore/build/derivation-goal.cc index 418a65f751e..b1f081115ce 100644 --- a/src/libstore/build/derivation-goal.cc +++ b/src/libstore/build/derivation-goal.cc @@ -86,7 +86,7 @@ void DerivationGoal::addWantedOutputs(const OutputsSpec & outputs) { auto newWanted = wantedOutputs.union_(outputs); switch (needRestart) { - case NeedRestartForMoreOutputs::OutputsUnmodifedDontNeed: + case NeedRestartForMoreOutputs::OutputsUnmodifiedDontNeed: if (!newWanted.isSubsetOf(wantedOutputs)) needRestart = NeedRestartForMoreOutputs::OutputsAddedDoNeed; break; @@ -303,7 +303,7 @@ Goal::Co DerivationGoal::haveDerivation(StorePath drvPath) nrFailed = nrNoSubstituters = 0; if (needRestart == NeedRestartForMoreOutputs::OutputsAddedDoNeed) { - needRestart = NeedRestartForMoreOutputs::OutputsUnmodifedDontNeed; + needRestart = NeedRestartForMoreOutputs::OutputsUnmodifiedDontNeed; co_return haveDerivation(std::move(drvPath)); } @@ -458,7 +458,7 @@ std::pair DerivationGoal::checkPathValidity(const StoreP for (auto & i : queryPartialDerivationOutputMap(drvPath)) { auto initialOutput = get(initialOutputs, i.first); if (!initialOutput) - // this is an invalid output, gets catched with (!wantedOutputsLeft.empty()) + // this is an invalid output, gets caught with (!wantedOutputsLeft.empty()) continue; auto & info = *initialOutput; info.wanted = wantedOutputs.contains(i.first); diff --git a/src/libstore/derivations.cc b/src/libstore/derivations.cc index 42de5ee0cbb..7c13b4f63bd 100644 --- a/src/libstore/derivations.cc +++ b/src/libstore/derivations.cc @@ -412,7 +412,7 @@ Derivation parseDerivation( expect(str, "rvWithVersion("); auto versionS = parseString(str); if (*versionS == "xp-dyn-drv") { - // Only verison we have so far + // Only version we have so far version = DerivationATermVersion::DynamicDerivations; xpSettings.require(Xp::DynamicDerivations); } else { @@ -553,7 +553,7 @@ static void unparseDerivedPathMapNode(const StoreDirConfig & store, std::string * derivation? * * In other words, does it on the output of derivation that is itself an - * ouput of a derivation? This corresponds to a dependency that is an + * output of a derivation? This corresponds to a dependency that is an * inductive derived path with more than one layer of * `DerivedPath::Built`. */ diff --git a/src/libstore/gc.cc b/src/libstore/gc.cc index 1469db3eca4..48467381d3f 100644 --- a/src/libstore/gc.cc +++ b/src/libstore/gc.cc @@ -775,7 +775,7 @@ void LocalStore::collectGarbage(const GCOptions & options, GCResults & results) deleteFromStore(path.to_string()); referrersCache.erase(path); } catch (PathInUse &e) { - // If we end up here, it's likely a new occurence + // If we end up here, it's likely a new occurrence // of https://github.com/NixOS/nix/issues/11923 printError("BUG: %s", e.what()); } diff --git a/src/libstore/include/nix/store/build-result.hh b/src/libstore/include/nix/store/build-result.hh index edc77a52350..088b057b65c 100644 --- a/src/libstore/include/nix/store/build-result.hh +++ b/src/libstore/include/nix/store/build-result.hh @@ -14,7 +14,7 @@ struct BuildResult { /** * @note This is directly used in the nix-store --serve protocol. - * That means we need to worry about compatability across versions. + * That means we need to worry about compatibility across versions. * Therefore, don't remove status codes, and only add new status * codes at the end of the list. */ diff --git a/src/libstore/include/nix/store/build/derivation-building-misc.hh b/src/libstore/include/nix/store/build/derivation-building-misc.hh index 915d891d709..3259c5e366d 100644 --- a/src/libstore/include/nix/store/build/derivation-building-misc.hh +++ b/src/libstore/include/nix/store/build/derivation-building-misc.hh @@ -1,6 +1,6 @@ #pragma once /** - * @file Misc type defitions for both local building and remote (RPC building) + * @file Misc type definitions for both local building and remote (RPC building) */ #include "nix/util/hash.hh" diff --git a/src/libstore/include/nix/store/build/derivation-goal.hh b/src/libstore/include/nix/store/build/derivation-goal.hh index b422aec7ab5..9d4257cb30a 100644 --- a/src/libstore/include/nix/store/build/derivation-goal.hh +++ b/src/libstore/include/nix/store/build/derivation-goal.hh @@ -42,7 +42,7 @@ struct DerivationGoal : public Goal * The goal state machine is progressing based on the current value of * `wantedOutputs. No actions are needed. */ - OutputsUnmodifedDontNeed, + OutputsUnmodifiedDontNeed, /** * `wantedOutputs` has been extended, but the state machine is * proceeding according to its old value, so we need to restart. @@ -59,7 +59,7 @@ struct DerivationGoal : public Goal /** * Whether additional wanted outputs have been added. */ - NeedRestartForMoreOutputs needRestart = NeedRestartForMoreOutputs::OutputsUnmodifedDontNeed; + NeedRestartForMoreOutputs needRestart = NeedRestartForMoreOutputs::OutputsUnmodifiedDontNeed; /** * The derivation stored at `drvReq`. diff --git a/src/libstore/include/nix/store/common-protocol-impl.hh b/src/libstore/include/nix/store/common-protocol-impl.hh index 18e63ac3374..e9c726a994d 100644 --- a/src/libstore/include/nix/store/common-protocol-impl.hh +++ b/src/libstore/include/nix/store/common-protocol-impl.hh @@ -4,7 +4,7 @@ * * Template implementations (as opposed to mere declarations). * - * This file is an exmample of the "impl.hh" pattern. See the + * This file is an example of the "impl.hh" pattern. See the * contributing guide. */ diff --git a/src/libstore/include/nix/store/common-protocol.hh b/src/libstore/include/nix/store/common-protocol.hh index 7887120b51d..1dc4aa7c569 100644 --- a/src/libstore/include/nix/store/common-protocol.hh +++ b/src/libstore/include/nix/store/common-protocol.hh @@ -89,12 +89,12 @@ DECLARE_COMMON_SERIALISER(std::map); * that the underlying types never serialize to the empty string. * * We do this instead of a generic std::optional instance because - * ordinal tags (0 or 1, here) are a bit of a compatability hazard. For + * ordinal tags (0 or 1, here) are a bit of a compatibility hazard. For * the same reason, we don't have a std::variant instances (ordinal * tags 0...n). * * We could the generic instances and then these as specializations for - * compatability, but that's proven a bit finnicky, and also makes the + * compatibility, but that's proven a bit finnicky, and also makes the * worker protocol harder to implement in other languages where such * specializations may not be allowed. */ diff --git a/src/libstore/include/nix/store/derivation-options.hh b/src/libstore/include/nix/store/derivation-options.hh index 16730b5c9cf..f61a43e6031 100644 --- a/src/libstore/include/nix/store/derivation-options.hh +++ b/src/libstore/include/nix/store/derivation-options.hh @@ -170,7 +170,7 @@ struct DerivationOptions /** * Parse this information from its legacy encoding as part of the * environment. This should not be used with nice greenfield formats - * (e.g. JSON) but is necessary for supporing old formats (e.g. + * (e.g. JSON) but is necessary for supporting old formats (e.g. * ATerm). */ static DerivationOptions diff --git a/src/libstore/include/nix/store/derivations.hh b/src/libstore/include/nix/store/derivations.hh index 46a9e2d0243..a813137bcba 100644 --- a/src/libstore/include/nix/store/derivations.hh +++ b/src/libstore/include/nix/store/derivations.hh @@ -214,7 +214,7 @@ struct DerivationType { /** * Impure derivation type * - * This is similar at buil-time to the content addressed, not standboxed, not fixed + * This is similar at build-time to the content addressed, not standboxed, not fixed * type, but has some restrictions on its usage. */ struct Impure { diff --git a/src/libstore/include/nix/store/gc-store.hh b/src/libstore/include/nix/store/gc-store.hh index cef6e8776e6..8b25ec8d4cb 100644 --- a/src/libstore/include/nix/store/gc-store.hh +++ b/src/libstore/include/nix/store/gc-store.hh @@ -89,7 +89,7 @@ struct GCResults * Some views have only a no-op temp roots even though others to the * same store allow triggering GC. For instance one can't add a root * over ssh, but that doesn't prevent someone from gc-ing that store - * accesed via SSH locally). + * accessed via SSH locally). * * - The derived `LocalFSStore` class has `LocalFSStore::addPermRoot`, * which is not part of this class because it relies on the notion of diff --git a/src/libstore/include/nix/store/outputs-spec.hh b/src/libstore/include/nix/store/outputs-spec.hh index b47f2654296..4e874a6f116 100644 --- a/src/libstore/include/nix/store/outputs-spec.hh +++ b/src/libstore/include/nix/store/outputs-spec.hh @@ -13,13 +13,13 @@ namespace nix { /** * An (owned) output name. Just a type alias used to make code more - * readible. + * readable. */ typedef std::string OutputName; /** * A borrowed output name. Just a type alias used to make code more - * readible. + * readable. */ typedef std::string_view OutputNameView; diff --git a/src/libstore/include/nix/store/path-info.hh b/src/libstore/include/nix/store/path-info.hh index 4691bfa9579..690f0f8134a 100644 --- a/src/libstore/include/nix/store/path-info.hh +++ b/src/libstore/include/nix/store/path-info.hh @@ -51,7 +51,7 @@ struct UnkeyedValidPathInfo Hash narHash; /** - * Other store objects this store object referes to. + * Other store objects this store object refers to. */ StorePathSet references; diff --git a/src/libstore/include/nix/store/serve-protocol-impl.hh b/src/libstore/include/nix/store/serve-protocol-impl.hh index 4ab16472177..4e66ca542ce 100644 --- a/src/libstore/include/nix/store/serve-protocol-impl.hh +++ b/src/libstore/include/nix/store/serve-protocol-impl.hh @@ -4,7 +4,7 @@ * * Template implementations (as opposed to mere declarations). * - * This file is an exmample of the "impl.hh" pattern. See the + * This file is an example of the "impl.hh" pattern. See the * contributing guide. */ diff --git a/src/libstore/include/nix/store/ssh.hh b/src/libstore/include/nix/store/ssh.hh index 40f2189d872..be9cf0c48b6 100644 --- a/src/libstore/include/nix/store/ssh.hh +++ b/src/libstore/include/nix/store/ssh.hh @@ -62,7 +62,7 @@ public: * * Current implementation is to use `fcntl` with `F_SETPIPE_SZ`, * which is Linux-only. For this implementation, `size` must - * convertable to an `int`. In other words, it must be within + * convertible to an `int`. In other words, it must be within * `[0, INT_MAX]`. */ void trySetBufferSize(size_t size); diff --git a/src/libstore/include/nix/store/store-api.hh b/src/libstore/include/nix/store/store-api.hh index 1648b13c1b2..72ca1f437fd 100644 --- a/src/libstore/include/nix/store/store-api.hh +++ b/src/libstore/include/nix/store/store-api.hh @@ -382,7 +382,7 @@ public: /** * Query the mapping outputName => outputPath for the given - * derivation. All outputs are mentioned so ones mising the mapping + * derivation. All outputs are mentioned so ones missing the mapping * are mapped to `std::nullopt`. */ virtual std::map> queryPartialDerivationOutputMap( @@ -809,7 +809,7 @@ protected: /** * Helper for methods that are not unsupported: this is used for - * default definitions for virtual methods that are meant to be overriden. + * default definitions for virtual methods that are meant to be overridden. * * @todo Using this should be a last resort. It is better to make * the method "virtual pure" and/or move it to a subclass. diff --git a/src/libstore/include/nix/store/store-dir-config.hh b/src/libstore/include/nix/store/store-dir-config.hh index ea0161d79c9..14e3e7db84e 100644 --- a/src/libstore/include/nix/store/store-dir-config.hh +++ b/src/libstore/include/nix/store/store-dir-config.hh @@ -89,7 +89,7 @@ struct MixStoreDirMethods /** * Read-only variant of addToStore(). It returns the store - * path for the given file sytem object. + * path for the given file system object. */ std::pair computeStorePath( std::string_view name, diff --git a/src/libstore/include/nix/store/store-reference.hh b/src/libstore/include/nix/store/store-reference.hh index 750bd827510..c1b681ba16d 100644 --- a/src/libstore/include/nix/store/store-reference.hh +++ b/src/libstore/include/nix/store/store-reference.hh @@ -9,7 +9,7 @@ namespace nix { /** * A parsed Store URI (URI is a slight misnomer...), parsed but not yet - * resolved to a specific instance and query parms validated. + * resolved to a specific instance and query params validated. * * Supported values are: * diff --git a/src/libstore/include/nix/store/store-registration.hh b/src/libstore/include/nix/store/store-registration.hh index 3f82ff51ceb..17298118e5a 100644 --- a/src/libstore/include/nix/store/store-registration.hh +++ b/src/libstore/include/nix/store/store-registration.hh @@ -7,7 +7,7 @@ * those implementations. * * Consumers of an arbitrary store from a URL/JSON configuration instead - * just need the defintions `nix/store/store-open.hh`; those do use this + * just need the definitions `nix/store/store-open.hh`; those do use this * but only as an implementation. Consumers of a specific extra type of * store can skip both these, and just use the definition of the store * in question directly. @@ -71,7 +71,7 @@ struct Implementations }; auto [it, didInsert] = registered().insert({TConfig::name(), std::move(factory)}); if (!didInsert) { - throw Error("Already registred store with name '%s'", it->first); + throw Error("Already registered store with name '%s'", it->first); } } }; diff --git a/src/libstore/include/nix/store/worker-protocol-connection.hh b/src/libstore/include/nix/store/worker-protocol-connection.hh index 11f112a7105..ce7e9aef47c 100644 --- a/src/libstore/include/nix/store/worker-protocol-connection.hh +++ b/src/libstore/include/nix/store/worker-protocol-connection.hh @@ -97,7 +97,7 @@ struct WorkerProto::BasicClientConnection : WorkerProto::BasicConnection /** * After calling handshake, must call this to exchange some basic - * information abou the connection. + * information about the connection. */ ClientHandshakeInfo postHandshake(const StoreDirConfig & store); @@ -157,7 +157,7 @@ struct WorkerProto::BasicServerConnection : WorkerProto::BasicConnection /** * After calling handshake, must call this to exchange some basic - * information abou the connection. + * information about the connection. */ void postHandshake(const StoreDirConfig & store, const ClientHandshakeInfo & info); }; diff --git a/src/libstore/include/nix/store/worker-protocol-impl.hh b/src/libstore/include/nix/store/worker-protocol-impl.hh index 908a9323e09..23e6068e9bb 100644 --- a/src/libstore/include/nix/store/worker-protocol-impl.hh +++ b/src/libstore/include/nix/store/worker-protocol-impl.hh @@ -4,7 +4,7 @@ * * Template implementations (as opposed to mere declarations). * - * This file is an exmample of the "impl.hh" pattern. See the + * This file is an example of the "impl.hh" pattern. See the * contributing guide. */ diff --git a/src/libstore/include/nix/store/worker-protocol.hh b/src/libstore/include/nix/store/worker-protocol.hh index 1b188806def..9630a88c063 100644 --- a/src/libstore/include/nix/store/worker-protocol.hh +++ b/src/libstore/include/nix/store/worker-protocol.hh @@ -89,7 +89,7 @@ struct WorkerProto struct BasicServerConnection; /** - * Extra information provided as part of protocol negotation. + * Extra information provided as part of protocol negotiation. */ struct ClientHandshakeInfo; diff --git a/src/libstore/profiles.cc b/src/libstore/profiles.cc index b5161b79fa9..09ef36705fa 100644 --- a/src/libstore/profiles.cc +++ b/src/libstore/profiles.cc @@ -331,7 +331,7 @@ Path getDefaultProfile() if (!pathExists(profileLink)) { replaceSymlink(profile, profileLink); } - // Backwards compatibiliy measure: Make root's profile available as + // Backwards compatibility measure: Make root's profile available as // `.../default` as it's what NixOS and most of the init scripts expect Path globalProfileLink = settings.nixStateDir + "/profiles/default"; if (isRootUser() && !pathExists(globalProfileLink)) { diff --git a/src/libstore/ssh-store.cc b/src/libstore/ssh-store.cc index 753256d483e..6992ae77462 100644 --- a/src/libstore/ssh-store.cc +++ b/src/libstore/ssh-store.cc @@ -120,7 +120,7 @@ std::string MountedSSHStoreConfig::doc() * store. * * MountedSSHStore is very similar to UDSRemoteStore --- ignoring the - * superficial differnce of SSH vs Unix domain sockets, they both are + * superficial difference of SSH vs Unix domain sockets, they both are * accessing remote stores, and they both assume the store will be * mounted in the local filesystem. * diff --git a/src/libstore/store-api.cc b/src/libstore/store-api.cc index b1360f8e591..296f2251adc 100644 --- a/src/libstore/store-api.cc +++ b/src/libstore/store-api.cc @@ -333,10 +333,10 @@ digraph graphname { node [shape=box] fileSource -> narSink narSink [style=dashed] - narSink -> unsualHashTee [style = dashed, label = "Recursive && !SHA-256"] + narSink -> unusualHashTee [style = dashed, label = "Recursive && !SHA-256"] narSink -> narHashSink [style = dashed, label = "else"] - unsualHashTee -> narHashSink - unsualHashTee -> caHashSink + unusualHashTee -> narHashSink + unusualHashTee -> caHashSink fileSource -> parseSink parseSink [style=dashed] parseSink-> fileSink [style = dashed, label = "Flat"] diff --git a/src/libstore/unix/build/derivation-builder.cc b/src/libstore/unix/build/derivation-builder.cc index 81e0a2584f6..08d4f021b68 100644 --- a/src/libstore/unix/build/derivation-builder.cc +++ b/src/libstore/unix/build/derivation-builder.cc @@ -1987,7 +1987,7 @@ void DerivationBuilderImpl::runChild() } for (auto & i : ss) { - // For backwards-compatibiliy, resolve all the symlinks in the + // For backwards-compatibility, resolve all the symlinks in the // chroot paths auto canonicalPath = canonPath(i, true); pathsInChroot.emplace(i, canonicalPath); diff --git a/src/libstore/windows/pathlocks.cc b/src/libstore/windows/pathlocks.cc index 0ba75853b3f..92a7cbcf9fd 100644 --- a/src/libstore/windows/pathlocks.cc +++ b/src/libstore/windows/pathlocks.cc @@ -127,7 +127,7 @@ bool PathLocks::lockPaths(const PathSet & paths, const std::string & waitMsg, bo } } - debug("lock aquired on '%1%'", lockPath); + debug("lock acquired on '%1%'", lockPath); struct _stat st; if (_fstat(fromDescriptorReadOnly(fd.get()), &st) == -1) diff --git a/src/libutil-tests/logging.cc b/src/libutil-tests/logging.cc index 494e9ce4cc8..5c9fcfe8f83 100644 --- a/src/libutil-tests/logging.cc +++ b/src/libutil-tests/logging.cc @@ -19,7 +19,7 @@ namespace nix { const char *one_liner = "this is the other problem line of code"; - TEST(logEI, catpuresBasicProperties) { + TEST(logEI, capturesBasicProperties) { MakeError(TestError, Error); ErrorInfo::programName = std::optional("error-unit-test"); diff --git a/src/libutil/hash.cc b/src/libutil/hash.cc index 015cc77a0f7..319eb795e6b 100644 --- a/src/libutil/hash.cc +++ b/src/libutil/hash.cc @@ -144,7 +144,7 @@ Hash Hash::parseSRI(std::string_view original) { auto rest = original; - // Parse the has type before the separater, if there was one. + // Parse the has type before the separator, if there was one. auto hashRaw = splitPrefixTo(rest, '-'); if (!hashRaw) throw BadHash("hash '%s' is not SRI", original); diff --git a/src/libutil/hilite.cc b/src/libutil/hilite.cc index cfadd6af9c9..6d4eb17a1ab 100644 --- a/src/libutil/hilite.cc +++ b/src/libutil/hilite.cc @@ -23,7 +23,7 @@ std::string hiliteMatches( auto m = *it; size_t start = m.position(); out.append(s.substr(last_end, m.position() - last_end)); - // Merge continous matches + // Merge continuous matches ssize_t end = start + m.length(); while (++it != matches.end() && (*it).position() <= end) { auto n = *it; diff --git a/src/libutil/include/nix/util/args.hh b/src/libutil/include/nix/util/args.hh index f1eb9667571..f3ab0b53249 100644 --- a/src/libutil/include/nix/util/args.hh +++ b/src/libutil/include/nix/util/args.hh @@ -252,7 +252,7 @@ protected: std::list processedArgs; /** - * Process some positional arugments + * Process some positional arguments * * @param finish: We have parsed everything else, and these are the only * arguments left. Used because we accumulate some "pending args" we might diff --git a/src/libutil/include/nix/util/comparator.hh b/src/libutil/include/nix/util/comparator.hh index 34ba6f4534c..c3af1758dff 100644 --- a/src/libutil/include/nix/util/comparator.hh +++ b/src/libutil/include/nix/util/comparator.hh @@ -16,7 +16,7 @@ /** * Awful hacky generation of the comparison operators by doing a lexicographic - * comparison between the choosen fields. + * comparison between the chosen fields. * * ``` * GENERATE_CMP(ClassName, me->field1, me->field2, ...) diff --git a/src/libutil/include/nix/util/executable-path.hh b/src/libutil/include/nix/util/executable-path.hh index 700d296d52d..cf6f3b25200 100644 --- a/src/libutil/include/nix/util/executable-path.hh +++ b/src/libutil/include/nix/util/executable-path.hh @@ -8,7 +8,7 @@ namespace nix { MakeError(ExecutableLookupError, Error); /** - * @todo rename, it is not just good for execuatable paths, but also + * @todo rename, it is not just good for executable paths, but also * other lists of paths. */ struct ExecutablePath @@ -51,7 +51,7 @@ struct ExecutablePath * * @param exe This must just be a name, and not contain any `/` (or * `\` on Windows). in case it does, per the spec no lookup should - * be perfomed, and the path (it is not just a file name) as is. + * be performed, and the path (it is not just a file name) as is. * This is the caller's respsonsibility. * * This is a pure function, except for the default `isExecutable` diff --git a/src/libutil/include/nix/util/file-descriptor.hh b/src/libutil/include/nix/util/file-descriptor.hh index 4f13a9a8fda..e2bcce2a283 100644 --- a/src/libutil/include/nix/util/file-descriptor.hh +++ b/src/libutil/include/nix/util/file-descriptor.hh @@ -68,7 +68,7 @@ static inline int fromDescriptorReadOnly(Descriptor fd) std::string readFile(Descriptor fd); /** - * Wrappers arount read()/write() that read/write exactly the + * Wrappers around read()/write() that read/write exactly the * requested number of bytes. */ void readFull(Descriptor fd, char * buf, size_t count); diff --git a/src/libutil/include/nix/util/file-path-impl.hh b/src/libutil/include/nix/util/file-path-impl.hh index d7c823fd0f1..1b4dd28f197 100644 --- a/src/libutil/include/nix/util/file-path-impl.hh +++ b/src/libutil/include/nix/util/file-path-impl.hh @@ -11,7 +11,7 @@ namespace nix { /** - * Unix-style path primives. + * Unix-style path primitives. * * Nix'result own "logical" paths are always Unix-style. So this is always * used for that, and additionally used for native paths on Unix. @@ -51,7 +51,7 @@ struct UnixPathTrait * often manipulating them converted to UTF-8 (*) using `char`. * * (Actually neither are guaranteed to be valid unicode; both are - * arbitrary non-0 8- or 16-bit bytes. But for charcters with specifical + * arbitrary non-0 8- or 16-bit bytes. But for characters with specifical * meaning like '/', '\\', ':', etc., we refer to an encoding scheme, * and also for sake of UIs that display paths a text.) */ diff --git a/src/libutil/include/nix/util/hash.hh b/src/libutil/include/nix/util/hash.hh index 0ae0a9564fd..71553745662 100644 --- a/src/libutil/include/nix/util/hash.hh +++ b/src/libutil/include/nix/util/hash.hh @@ -36,7 +36,7 @@ enum struct HashFormat : int { /// @brief Lowercase hexadecimal encoding. @see base16Chars Base16, /// @brief ":", format of the SRI integrity attribute. - /// @see W3C recommendation [Subresource Intergrity](https://www.w3.org/TR/SRI/). + /// @see W3C recommendation [Subresource Integrity](https://www.w3.org/TR/SRI/). SRI }; @@ -66,7 +66,7 @@ struct Hash /** * Parse a hash from a string representation like the above, except the - * type prefix is mandatory is there is no separate arguement. + * type prefix is mandatory is there is no separate argument. */ static Hash parseAnyPrefixed(std::string_view s); diff --git a/src/libutil/include/nix/util/pos-idx.hh b/src/libutil/include/nix/util/pos-idx.hh index c1749ba6935..0bf59301aff 100644 --- a/src/libutil/include/nix/util/pos-idx.hh +++ b/src/libutil/include/nix/util/pos-idx.hh @@ -8,7 +8,7 @@ namespace nix { class PosIdx { - friend struct LazyPosAcessors; + friend struct LazyPosAccessors; friend class PosTable; friend class std::hash; diff --git a/src/libutil/include/nix/util/serialise.hh b/src/libutil/include/nix/util/serialise.hh index d28c8e9a6b3..97fdddae301 100644 --- a/src/libutil/include/nix/util/serialise.hh +++ b/src/libutil/include/nix/util/serialise.hh @@ -564,7 +564,7 @@ struct FramedSink : nix::BufferedSink void writeUnbuffered(std::string_view data) override { - /* Don't send more data if an error has occured. */ + /* Don't send more data if an error has occurred. */ checkError(); to << data.size(); diff --git a/src/libutil/include/nix/util/source-accessor.hh b/src/libutil/include/nix/util/source-accessor.hh index 5ef66015015..92a9adc46e9 100644 --- a/src/libutil/include/nix/util/source-accessor.hh +++ b/src/libutil/include/nix/util/source-accessor.hh @@ -54,7 +54,7 @@ struct SourceAccessor : std::enable_shared_from_this * * @note Unlike Unix, this method should *not* follow symlinks. Nix * by default wants to manipulate symlinks explicitly, and not - * implictly follow them, as they are frequently untrusted user data + * implicitly follow them, as they are frequently untrusted user data * and thus may point to arbitrary locations. Acting on the targets * targets of symlinks should only occasionally be done, and only * with care. diff --git a/src/libutil/unix/file-descriptor.cc b/src/libutil/unix/file-descriptor.cc index e6d0c255dde..0051e8aa43c 100644 --- a/src/libutil/unix/file-descriptor.cc +++ b/src/libutil/unix/file-descriptor.cc @@ -15,7 +15,7 @@ namespace nix { namespace { // This function is needed to handle non-blocking reads/writes. This is needed in the buildhook, because -// somehow the json logger file descriptor ends up beeing non-blocking and breaks remote-building. +// somehow the json logger file descriptor ends up being non-blocking and breaks remote-building. // TODO: get rid of buildhook and remove this function again (https://github.com/NixOS/nix/issues/12688) void pollFD(int fd, int events) { diff --git a/src/libutil/windows/file-descriptor.cc b/src/libutil/windows/file-descriptor.cc index f451bc0d3ae..03d68232c37 100644 --- a/src/libutil/windows/file-descriptor.cc +++ b/src/libutil/windows/file-descriptor.cc @@ -47,7 +47,7 @@ void writeFull(HANDLE handle, std::string_view s, bool allowInterrupts) if (allowInterrupts) checkInterrupt(); DWORD res; #if _WIN32_WINNT >= 0x0600 - auto path = handleToPath(handle); // debug; do it before becuase handleToPath changes lasterror + auto path = handleToPath(handle); // debug; do it before because handleToPath changes lasterror if (!WriteFile(handle, s.data(), s.size(), &res, NULL)) { throw WinError("writing to file %1%:%2%", handle, path); } diff --git a/src/nix-store/nix-store.cc b/src/nix-store/nix-store.cc index 9acdf455448..0d758f44b1c 100644 --- a/src/nix-store/nix-store.cc +++ b/src/nix-store/nix-store.cc @@ -862,7 +862,7 @@ static void opServe(Strings opFlags, Strings opArgs) auto options = ServeProto::Serialise::read(*store, rconn); - // Only certain feilds get initialized based on the protocol + // Only certain fields get initialized based on the protocol // version. This is why not all the code below is unconditional. // See how the serialization logic in // `ServeProto::Serialise` matches diff --git a/src/nix/flake.cc b/src/nix/flake.cc index fdeedf9c1ff..0a70fa4b4e7 100644 --- a/src/nix/flake.cc +++ b/src/nix/flake.cc @@ -65,7 +65,7 @@ class FlakeCommand : virtual Args, public MixFlakeOptions std::vector getFlakeRefsForCompletion() override { return { - // Like getFlakeRef but with expandTilde calld first + // Like getFlakeRef but with expandTilde called first parseFlakeRef(fetchSettings, expandTilde(flakeUrl), std::filesystem::current_path().string()) }; } diff --git a/src/nix/meson.build b/src/nix/meson.build index 11c30914ba1..0ba8bdd4640 100644 --- a/src/nix/meson.build +++ b/src/nix/meson.build @@ -226,7 +226,7 @@ foreach linkname : nix_symlinks # TODO(Ericson2314): Don't do this once we have the `meson.override_find_program` working) build_by_default: true ) - # TODO(Ericson3214): Dosen't yet work + # TODO(Ericson3214): Doesn't yet work #meson.override_find_program(linkname, t) endforeach @@ -246,7 +246,7 @@ custom_target( # TODO(Ericson2314): Don't do this once we have the `meson.override_find_program` working) build_by_default: true ) -# TODO(Ericson3214): Dosen't yet work +# TODO(Ericson3214): Doesn't yet work #meson.override_find_program(linkname, t) localstatedir = nix_store.get_variable( diff --git a/src/nix/unix/daemon.cc b/src/nix/unix/daemon.cc index 301f8aa50ca..4132d5be2cd 100644 --- a/src/nix/unix/daemon.cc +++ b/src/nix/unix/daemon.cc @@ -481,7 +481,7 @@ static void processStdioConnection(ref store, TrustedFlag trustClient) * @param forceTrustClientOpt See `daemonLoop()` and the parameter with * the same name over there for details. * - * @param procesOps Whether to force processing ops even if the next + * @param processOps Whether to force processing ops even if the next * store also is a remote store and could process it directly. */ static void runDaemon(bool stdio, std::optional forceTrustClientOpt, bool processOps) From 0479db934c9412443aac5bf563e9dc4538eba7f1 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?J=C3=B6rg=20Thalheim?= Date: Mon, 26 May 2025 09:19:40 +0200 Subject: [PATCH 070/218] fetchGit: don't compute revCount on shallow repository This can never work and leads to a crash bug. --- src/libfetchers/git.cc | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/src/libfetchers/git.cc b/src/libfetchers/git.cc index f716094b6ae..f7e4894fc8c 100644 --- a/src/libfetchers/git.cc +++ b/src/libfetchers/git.cc @@ -798,8 +798,10 @@ struct GitInputScheme : InputScheme auto rev = repoInfo.workdirInfo.headRev.value_or(nullRev); input.attrs.insert_or_assign("rev", rev.gitRev()); - input.attrs.insert_or_assign("revCount", - rev == nullRev ? 0 : getRevCount(*input.settings, repoInfo, repoPath, rev)); + if (!getShallowAttr(input)) { + input.attrs.insert_or_assign("revCount", + rev == nullRev ? 0 : getRevCount(*input.settings, repoInfo, repoPath, rev)); + } verifyCommit(input, repo); } else { From b1ccfaa0801b08c0649aa7f6d113cfb73354b861 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?J=C3=B6rg=20Thalheim?= Date: Sun, 25 May 2025 16:27:51 +0200 Subject: [PATCH 071/218] git/revCount: improve error message when we have incomplete git history --- src/libfetchers/git-utils.cc | 13 +++++- tests/functional/fetchGit.sh | 14 +----- tests/functional/fetchGitShallow.sh | 67 +++++++++++++++++++++++++++++ tests/functional/meson.build | 1 + 4 files changed, 80 insertions(+), 15 deletions(-) create mode 100644 tests/functional/fetchGitShallow.sh diff --git a/src/libfetchers/git-utils.cc b/src/libfetchers/git-utils.cc index 09508c6026c..e93d58709bb 100644 --- a/src/libfetchers/git-utils.cc +++ b/src/libfetchers/git-utils.cc @@ -322,8 +322,17 @@ struct GitRepoImpl : GitRepo, std::enable_shared_from_this for (size_t n = 0; n < git_commit_parentcount(commit->get()); ++n) { git_commit * parent; - if (git_commit_parent(&parent, commit->get(), n)) - throw Error("getting parent of Git commit '%s': %s", *git_commit_id(commit->get()), git_error_last()->message); + if (git_commit_parent(&parent, commit->get(), n)) { + throw Error( + "Failed to retrieve the parent of Git commit '%s': %s. " + "This may be due to an incomplete repository history. " + "To resolve this, either enable the shallow parameter in your flake URL (?shallow=1) " + "or add set the shallow parameter to true in builtins.fetchGit, " + "or fetch the complete history for this branch.", + *git_commit_id(commit->get()), + git_error_last()->message + ); + } todo.push(Commit(parent)); } } diff --git a/tests/functional/fetchGit.sh b/tests/functional/fetchGit.sh index 5e5e8e61fb6..fcae8bf69aa 100755 --- a/tests/functional/fetchGit.sh +++ b/tests/functional/fetchGit.sh @@ -12,7 +12,7 @@ repo=$TEST_ROOT/./git export _NIX_FORCE_HTTP=1 -rm -rf $repo ${repo}-tmp $TEST_HOME/.cache/nix $TEST_ROOT/worktree $TEST_ROOT/shallow $TEST_ROOT/minimal +rm -rf $repo ${repo}-tmp $TEST_HOME/.cache/nix $TEST_ROOT/worktree $TEST_ROOT/minimal git init $repo git -C $repo config user.email "foobar@example.com" @@ -216,18 +216,6 @@ git -C $TEST_ROOT/minimal fetch $repo $rev2 git -C $TEST_ROOT/minimal checkout $rev2 [[ $(nix eval --impure --raw --expr "(builtins.fetchGit { url = $TEST_ROOT/minimal; }).rev") = $rev2 ]] -# Fetching a shallow repo shouldn't work by default, because we can't -# return a revCount. -git clone --depth 1 file://$repo $TEST_ROOT/shallow -(! nix eval --impure --raw --expr "(builtins.fetchGit { url = $TEST_ROOT/shallow; ref = \"dev\"; }).outPath") - -# But you can request a shallow clone, which won't return a revCount. -path6=$(nix eval --impure --raw --expr "(builtins.fetchTree { type = \"git\"; url = \"file://$TEST_ROOT/shallow\"; ref = \"dev\"; shallow = true; }).outPath") -[[ $path3 = $path6 ]] -[[ $(nix eval --impure --expr "(builtins.fetchTree { type = \"git\"; url = \"file://$TEST_ROOT/shallow\"; ref = \"dev\"; shallow = true; }).revCount or 123") == 123 ]] - -expectStderr 1 nix eval --expr 'builtins.fetchTree { type = "git"; url = "file:///foo"; }' | grepQuiet "'fetchTree' will not fetch unlocked input" - # Explicit ref = "HEAD" should work, and produce the same outPath as without ref path7=$(nix eval --impure --raw --expr "(builtins.fetchGit { url = \"file://$repo\"; ref = \"HEAD\"; }).outPath") path8=$(nix eval --impure --raw --expr "(builtins.fetchGit { url = \"file://$repo\"; }).outPath") diff --git a/tests/functional/fetchGitShallow.sh b/tests/functional/fetchGitShallow.sh new file mode 100644 index 00000000000..cf7e5fd4fbd --- /dev/null +++ b/tests/functional/fetchGitShallow.sh @@ -0,0 +1,67 @@ +#!/usr/bin/env bash + +# shellcheck source=common.sh +source common.sh + +requireGit + +# Create a test repo with multiple commits for all our tests +git init "$TEST_ROOT/shallow-parent" +git -C "$TEST_ROOT/shallow-parent" config user.email "foobar@example.com" +git -C "$TEST_ROOT/shallow-parent" config user.name "Foobar" + +# Add several commits to have history +echo "{ outputs = _: {}; }" > "$TEST_ROOT/shallow-parent/flake.nix" +echo "" > "$TEST_ROOT/shallow-parent/file.txt" +git -C "$TEST_ROOT/shallow-parent" add file.txt flake.nix +git -C "$TEST_ROOT/shallow-parent" commit -m "First commit" + +echo "second" > "$TEST_ROOT/shallow-parent/file.txt" +git -C "$TEST_ROOT/shallow-parent" commit -m "Second commit" -a + +echo "third" > "$TEST_ROOT/shallow-parent/file.txt" +git -C "$TEST_ROOT/shallow-parent" commit -m "Third commit" -a + +# Add a branch for testing ref fetching +git -C "$TEST_ROOT/shallow-parent" checkout -b dev +echo "branch content" > "$TEST_ROOT/shallow-parent/branch-file.txt" +git -C "$TEST_ROOT/shallow-parent" add branch-file.txt +git -C "$TEST_ROOT/shallow-parent" commit -m "Branch commit" + +# Make a shallow clone (depth=1) +git clone --depth 1 "file://$TEST_ROOT/shallow-parent" "$TEST_ROOT/shallow-clone" + +# Test 1: Fetching a shallow repo shouldn't work by default, because we can't +# return a revCount. +(! nix eval --impure --raw --expr "(builtins.fetchGit { url = \"$TEST_ROOT/shallow-clone\"; ref = \"dev\"; }).outPath") + +# Test 2: But you can request a shallow clone, which won't return a revCount. +path=$(nix eval --impure --raw --expr "(builtins.fetchTree { type = \"git\"; url = \"file://$TEST_ROOT/shallow-clone\"; ref = \"dev\"; shallow = true; }).outPath") +# Verify file from dev branch exists +[[ -f "$path/branch-file.txt" ]] +# Verify revCount is missing +[[ $(nix eval --impure --expr "(builtins.fetchTree { type = \"git\"; url = \"file://$TEST_ROOT/shallow-clone\"; ref = \"dev\"; shallow = true; }).revCount or 123") == 123 ]] + +# Test 3: Check unlocked input error message +expectStderr 1 nix eval --expr 'builtins.fetchTree { type = "git"; url = "file:///foo"; }' | grepQuiet "'fetchTree' will not fetch unlocked input" + +# Test 4: Regression test for revCount in worktrees derived from shallow clones +# Add a worktree to the shallow clone +git -C "$TEST_ROOT/shallow-clone" worktree add "$TEST_ROOT/shallow-worktree" + +# Prior to the fix, this would error out because of the shallow clone's +# inability to find parent commits. Now it should return an error. +if nix eval --impure --expr "(builtins.fetchGit { url = \"file://$TEST_ROOT/shallow-worktree\"; }).revCount" 2>/dev/null; then + echo "fetchGit unexpectedly succeeded on shallow clone" >&2 + exit 1 +fi + +# Also verify that fetchTree fails similarly +if nix eval --impure --expr "(builtins.fetchTree { type = \"git\"; url = \"file://$TEST_ROOT/shallow-worktree\"; }).revCount" 2>/dev/null; then + echo "fetchTree unexpectedly succeeded on shallow clone" >&2 + exit 1 +fi + +# Verify that we can shallow fetch the worktree +git -C "$TEST_ROOT/shallow-worktree" rev-list --count HEAD >/dev/null +nix eval --impure --raw --expr "(builtins.fetchGit { url = \"file://$TEST_ROOT/shallow-worktree\"; shallow = true; }).rev" diff --git a/tests/functional/meson.build b/tests/functional/meson.build index f5a19ac646a..cd1bc631978 100644 --- a/tests/functional/meson.build +++ b/tests/functional/meson.build @@ -73,6 +73,7 @@ suites = [ 'gc-runtime.sh', 'tarball.sh', 'fetchGit.sh', + 'fetchGitShallow.sh', 'fetchurl.sh', 'fetchPath.sh', 'fetchTree-file.sh', From 5419d82547bf25b99f3ef59f1f4f923c41837d52 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?J=C3=B6rg=20Thalheim?= Date: Sun, 25 May 2025 21:37:09 +0200 Subject: [PATCH 072/218] tests/fetchGit: work around something that looks a bash parsing quirk. Before we got something like this but only inside the VM test: vm-test-run-functional-tests-on-nixos_user> machine # fetchGit.sh: line 286: unexpected EOF while looking for matching `)' We now try to do not too much in a single line, so that the bash parser does not get confused. This also seems more readable and better quoted. --- tests/functional/fetchGit.sh | 13 ++++++++----- 1 file changed, 8 insertions(+), 5 deletions(-) diff --git a/tests/functional/fetchGit.sh b/tests/functional/fetchGit.sh index fcae8bf69aa..5832facda62 100755 --- a/tests/functional/fetchGit.sh +++ b/tests/functional/fetchGit.sh @@ -280,17 +280,20 @@ path11=$(nix eval --impure --raw --expr "(builtins.fetchGit ./.).outPath") empty="$TEST_ROOT/empty" git init "$empty" -emptyAttrs='{ lastModified = 0; lastModifiedDate = "19700101000000"; narHash = "sha256-pQpattmS9VmO3ZIQUFn66az8GSmB4IvYhTTCFn6SUmo="; rev = "0000000000000000000000000000000000000000"; revCount = 0; shortRev = "0000000"; submodules = false; }' - -[[ $(nix eval --impure --expr "builtins.removeAttrs (builtins.fetchGit $empty) [\"outPath\"]") = $emptyAttrs ]] +emptyAttrs="{ lastModified = 0; lastModifiedDate = \"19700101000000\"; narHash = \"sha256-pQpattmS9VmO3ZIQUFn66az8GSmB4IvYhTTCFn6SUmo=\"; rev = \"0000000000000000000000000000000000000000\"; revCount = 0; shortRev = \"0000000\"; submodules = false; }" +result=$(nix eval --impure --expr "builtins.removeAttrs (builtins.fetchGit $empty) [\"outPath\"]") +[[ "$result" = "$emptyAttrs" ]] echo foo > "$empty/x" -[[ $(nix eval --impure --expr "builtins.removeAttrs (builtins.fetchGit $empty) [\"outPath\"]") = $emptyAttrs ]] +result=$(nix eval --impure --expr "builtins.removeAttrs (builtins.fetchGit $empty) [\"outPath\"]") +[[ "$result" = "$emptyAttrs" ]] git -C "$empty" add x -[[ $(nix eval --impure --expr "builtins.removeAttrs (builtins.fetchGit $empty) [\"outPath\"]") = '{ lastModified = 0; lastModifiedDate = "19700101000000"; narHash = "sha256-wzlAGjxKxpaWdqVhlq55q5Gxo4Bf860+kLeEa/v02As="; rev = "0000000000000000000000000000000000000000"; revCount = 0; shortRev = "0000000"; submodules = false; }' ]] +expected_attrs="{ lastModified = 0; lastModifiedDate = \"19700101000000\"; narHash = \"sha256-wzlAGjxKxpaWdqVhlq55q5Gxo4Bf860+kLeEa/v02As=\"; rev = \"0000000000000000000000000000000000000000\"; revCount = 0; shortRev = \"0000000\"; submodules = false; }" +result=$(nix eval --impure --expr "builtins.removeAttrs (builtins.fetchGit $empty) [\"outPath\"]") +[[ "$result" = "$expected_attrs" ]] # Test a repo with an empty commit. git -C "$empty" rm -f x From 69914e4b3c91b81b34a3f482d8ae2a4d7a77c04b Mon Sep 17 00:00:00 2001 From: Eelco Dolstra Date: Mon, 26 May 2025 15:53:05 +0200 Subject: [PATCH 073/218] Remove `buildUser` from `DerivationBuilder` The use of a `buildUser` is an implementation detail of some types of sandboxes that shouldn't exposed. --- src/libstore/build/derivation-building-goal.cc | 2 +- src/libstore/unix/build/derivation-builder.cc | 7 ++++++- .../unix/include/nix/store/build/derivation-builder.hh | 5 ----- 3 files changed, 7 insertions(+), 7 deletions(-) diff --git a/src/libstore/build/derivation-building-goal.cc b/src/libstore/build/derivation-building-goal.cc index 7335668e1ef..2b5473b81fb 100644 --- a/src/libstore/build/derivation-building-goal.cc +++ b/src/libstore/build/derivation-building-goal.cc @@ -655,8 +655,8 @@ Goal::Co DerivationBuildingGoal::tryToBuild() builder->startBuilder(); } catch (BuildError & e) { + builder.reset(); outputLocks.unlock(); - builder->buildUser.reset(); worker.permanentFailure = true; co_return done(BuildResult::InputRejected, {}, std::move(e)); } diff --git a/src/libstore/unix/build/derivation-builder.cc b/src/libstore/unix/build/derivation-builder.cc index 08d4f021b68..45fdd5c442b 100644 --- a/src/libstore/unix/build/derivation-builder.cc +++ b/src/libstore/unix/build/derivation-builder.cc @@ -106,6 +106,11 @@ class DerivationBuilderImpl : public DerivationBuilder, DerivationBuilderParams private: + /** + * User selected for running the builder. + */ + std::unique_ptr buildUser; + /** * The cgroup of the builder, if any. */ @@ -264,7 +269,7 @@ class DerivationBuilderImpl : public DerivationBuilder, DerivationBuilderParams /** * Start building a derivation. */ - void startBuilder() override;; + void startBuilder() override; /** * Tear down build environment after the builder exits (either on diff --git a/src/libstore/unix/include/nix/store/build/derivation-builder.hh b/src/libstore/unix/include/nix/store/build/derivation-builder.hh index d6c40060a48..01266a49234 100644 --- a/src/libstore/unix/include/nix/store/build/derivation-builder.hh +++ b/src/libstore/unix/include/nix/store/build/derivation-builder.hh @@ -138,11 +138,6 @@ struct DerivationBuilderCallbacks */ struct DerivationBuilder : RestrictionContext { - /** - * User selected for running the builder. - */ - std::unique_ptr buildUser; - /** * The process ID of the builder. */ From 93ae95be83f4682f747edf7dfc50c27f152f7d99 Mon Sep 17 00:00:00 2001 From: Eelco Dolstra Date: Mon, 26 May 2025 15:55:58 +0200 Subject: [PATCH 074/218] Remove duplicate comments on DerivationBuilderImpl overriden methods Having the exact same doc comments isn't very useful/maintainable. --- src/libstore/unix/build/derivation-builder.cc | 31 ------------------- 1 file changed, 31 deletions(-) diff --git a/src/libstore/unix/build/derivation-builder.cc b/src/libstore/unix/build/derivation-builder.cc index 45fdd5c442b..e7412ed660b 100644 --- a/src/libstore/unix/build/derivation-builder.cc +++ b/src/libstore/unix/build/derivation-builder.cc @@ -256,30 +256,10 @@ class DerivationBuilderImpl : public DerivationBuilder, DerivationBuilderParams public: - /** - * Set up build environment / sandbox, acquiring resources (e.g. - * locks as needed). After this is run, the builder should be - * started. - * - * @returns true if successful, false if we could not acquire a build - * user. In that case, the caller must wait and then try again. - */ bool prepareBuild() override; - /** - * Start building a derivation. - */ void startBuilder() override; - /** - * Tear down build environment after the builder exits (either on - * its own or if it is killed). - * - * @returns The first case indicates failure during output - * processing. A status code and exception are returned, providing - * more information. The second case indicates success, and - * realisations for each output of the derivation are returned. - */ std::variant, SingleDrvOutputs> unprepareBuild() override; private: @@ -311,10 +291,6 @@ class DerivationBuilderImpl : public DerivationBuilder, DerivationBuilderParams public: - /** - * Stop the in-process nix daemon thread. - * @see startDaemon - */ void stopDaemon() override; private: @@ -346,15 +322,8 @@ class DerivationBuilderImpl : public DerivationBuilder, DerivationBuilderParams public: - /** - * Delete the temporary directory, if we have one. - */ void deleteTmpDir(bool force) override; - /** - * Kill any processes running under the build user UID or in the - * cgroup of the build. - */ void killSandbox(bool getStats) override; private: From 9cc8be26747a0206613421a1ba1c3b1f54212e8b Mon Sep 17 00:00:00 2001 From: Eelco Dolstra Date: Mon, 26 May 2025 23:30:16 +0200 Subject: [PATCH 075/218] Drop magic-nix-cache This no longer works, see https://determinate.systems/posts/magic-nix-cache-free-tier-eol/. --- .github/workflows/ci.yml | 4 ---- 1 file changed, 4 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 29cb33f56af..fb70fae871e 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -40,7 +40,6 @@ jobs: extra_nix_config: | sandbox = true max-jobs = 1 - - uses: DeterminateSystems/magic-nix-cache-action@main # Since ubuntu 22.30, unprivileged usernamespaces are no longer allowed to map to the root user: # https://ubuntu.com/blog/ubuntu-23-10-restricted-unprivileged-user-namespaces - run: sudo sysctl -w kernel.apparmor_restrict_unprivileged_userns=0 @@ -134,7 +133,6 @@ jobs: - uses: cachix/install-nix-action@v31 with: install_url: https://releases.nixos.org/nix/nix-2.20.3/install - - uses: DeterminateSystems/magic-nix-cache-action@main - run: echo NIX_VERSION="$(nix --experimental-features 'nix-command flakes' eval .\#nix.version | tr -d \")" >> $GITHUB_ENV - run: nix --experimental-features 'nix-command flakes' build .#dockerImage -L - run: docker load -i ./result/image.tar.gz @@ -176,7 +174,6 @@ jobs: steps: - uses: actions/checkout@v4 - uses: DeterminateSystems/nix-installer-action@main - - uses: DeterminateSystems/magic-nix-cache-action@main - run: | nix build -L \ .#hydraJobs.tests.functional_user \ @@ -202,5 +199,4 @@ jobs: repository: NixOS/flake-regressions-data path: flake-regressions/tests - uses: DeterminateSystems/nix-installer-action@main - - uses: DeterminateSystems/magic-nix-cache-action@main - run: nix build -L --out-link ./new-nix && PATH=$(pwd)/new-nix/bin:$PATH MAX_FLAKES=25 flake-regressions/eval-all.sh From d07852b5f33b4ef2bd19d7f542575502898f9f97 Mon Sep 17 00:00:00 2001 From: Tristan Ross Date: Mon, 26 May 2025 21:49:43 -0700 Subject: [PATCH 076/218] Remove propagated-build-inputs when static --- packaging/components.nix | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/packaging/components.nix b/packaging/components.nix index 8e3c21b7da5..b40bd45b0e4 100644 --- a/packaging/components.nix +++ b/packaging/components.nix @@ -166,6 +166,17 @@ let outputs = prevAttrs.outputs or [ "out" ] ++ [ "dev" ]; }; + fixupStaticLayer = finalAttrs: prevAttrs: { + postFixup = + prevAttrs.postFixup or "" + + lib.optionalString (stdenv.hostPlatform.isStatic) '' + # HACK: Otherwise the result will have the entire buildInputs closure + # injected by the pkgsStatic stdenv + # + rm -f $out/nix-support/propagated-build-inputs + ''; + }; + # Work around weird `--as-needed` linker behavior with BSD, see # https://github.com/mesonbuild/meson/issues/3593 bsdNoLinkAsNeeded = @@ -301,6 +312,7 @@ in scope.sourceLayer setVersionLayer mesonLayer + fixupStaticLayer scope.mesonComponentOverrides ]; mkMesonExecutable = mkPackageBuilder [ @@ -310,6 +322,7 @@ in setVersionLayer mesonLayer mesonBuildLayer + fixupStaticLayer scope.mesonComponentOverrides ]; mkMesonLibrary = mkPackageBuilder [ @@ -320,6 +333,7 @@ in setVersionLayer mesonBuildLayer mesonLibraryLayer + fixupStaticLayer scope.mesonComponentOverrides ]; From aaca9711fc4f745ebcfcec60dfc1ea4909a8db4a Mon Sep 17 00:00:00 2001 From: Eelco Dolstra Date: Mon, 26 May 2025 17:12:54 +0200 Subject: [PATCH 077/218] DerivationBuilderImpl: Drop std::optional from derivationType No point in computing this lazily, since it's pretty much the first thing the DerivationBuilder does. --- src/libstore/unix/build/derivation-builder.cc | 27 +++++++++---------- 1 file changed, 12 insertions(+), 15 deletions(-) diff --git a/src/libstore/unix/build/derivation-builder.cc b/src/libstore/unix/build/derivation-builder.cc index e7412ed660b..54173c16259 100644 --- a/src/libstore/unix/build/derivation-builder.cc +++ b/src/libstore/unix/build/derivation-builder.cc @@ -100,6 +100,7 @@ class DerivationBuilderImpl : public DerivationBuilder, DerivationBuilderParams : DerivationBuilderParams{std::move(params)} , store{store} , miscMethods{std::move(miscMethods)} + , derivationType{drv.type()} { } LocalStore & getLocalStore(); @@ -168,9 +169,9 @@ class DerivationBuilderImpl : public DerivationBuilder, DerivationBuilderParams /** * The sort of derivation we are building. * - * Just a cached value, can be recomputed from `drv`. + * Just a cached value, computed from `drv`. */ - std::optional derivationType; + const DerivationType derivationType; /** * Stuff we need to pass to initChild(). @@ -438,9 +439,6 @@ void DerivationBuilderImpl::killSandbox(bool getStats) bool DerivationBuilderImpl::prepareBuild() { - /* Cache this */ - derivationType = drv.type(); - /* Are we doing a chroot build? */ { if (settings.sandboxMode == smEnabled) { @@ -457,7 +455,7 @@ bool DerivationBuilderImpl::prepareBuild() else if (settings.sandboxMode == smDisabled) useChroot = false; else if (settings.sandboxMode == smRelaxed) - useChroot = derivationType->isSandboxed() && !drvOptions.noChroot; + useChroot = derivationType.isSandboxed() && !drvOptions.noChroot; } auto & localStore = getLocalStore(); @@ -594,11 +592,10 @@ std::variant, SingleDrvOutputs> Derivation return std::move(builtOutputs); } catch (BuildError & e) { - assert(derivationType); BuildResult::Status st = dynamic_cast(&e) ? BuildResult::NotDeterministic : statusOk(status) ? BuildResult::OutputRejected : - !derivationType->isSandboxed() || diskFull ? BuildResult::TransientFailure : + !derivationType.isSandboxed() || diskFull ? BuildResult::TransientFailure : BuildResult::PermanentFailure; return std::pair{std::move(st), std::move(e)}; @@ -1068,7 +1065,7 @@ void DerivationBuilderImpl::startBuilder() "nogroup:x:65534:\n", sandboxGid())); /* Create /etc/hosts with localhost entry. */ - if (derivationType->isSandboxed()) + if (derivationType.isSandboxed()) writeFile(chrootRootDir + "/etc/hosts", "127.0.0.1 localhost\n::1 localhost\n"); /* Make the closure of the inputs available in the chroot, @@ -1296,7 +1293,7 @@ void DerivationBuilderImpl::startBuilder() ProcessOptions options; options.cloneFlags = CLONE_NEWPID | CLONE_NEWNS | CLONE_NEWIPC | CLONE_NEWUTS | CLONE_PARENT | SIGCHLD; - if (derivationType->isSandboxed()) + if (derivationType.isSandboxed()) options.cloneFlags |= CLONE_NEWNET; if (usingUserNamespace) options.cloneFlags |= CLONE_NEWUSER; @@ -1502,7 +1499,7 @@ void DerivationBuilderImpl::initEnv() derivation, tell the builder, so that for instance `fetchurl' can skip checking the output. On older Nixes, this environment variable won't be set, so `fetchurl' will do the check. */ - if (derivationType->isFixed()) env["NIX_OUTPUT_CHECKED"] = "1"; + if (derivationType.isFixed()) env["NIX_OUTPUT_CHECKED"] = "1"; /* *Only* if this is a fixed-output derivation, propagate the values of the environment variables specified in the @@ -1513,7 +1510,7 @@ void DerivationBuilderImpl::initEnv() to the builder is generally impure, but the output of fixed-output derivations is by definition pure (since we already know the cryptographic hash of the output). */ - if (!derivationType->isSandboxed()) { + if (!derivationType.isSandboxed()) { auto & impureEnv = settings.impureEnv.get(); if (!impureEnv.empty()) experimentalFeatureSettings.require(Xp::ConfigurableImpureEnv); @@ -1863,7 +1860,7 @@ void DerivationBuilderImpl::runChild() userNamespaceSync.readSide = -1; - if (derivationType->isSandboxed()) { + if (derivationType.isSandboxed()) { /* Initialise the loopback interface. */ AutoCloseFD fd(socket(PF_INET, SOCK_DGRAM, IPPROTO_IP)); @@ -1939,7 +1936,7 @@ void DerivationBuilderImpl::runChild() /* Fixed-output derivations typically need to access the network, so give them access to /etc/resolv.conf and so on. */ - if (!derivationType->isSandboxed()) { + if (!derivationType.isSandboxed()) { // Only use nss functions to resolve hosts and // services. Don’t use it for anything else that may // be configured for this system. This limits the @@ -2172,7 +2169,7 @@ void DerivationBuilderImpl::runChild() #include "sandbox-defaults.sb" ; - if (!derivationType->isSandboxed()) + if (!derivationType.isSandboxed()) sandboxProfile += #include "sandbox-network.sb" ; From 80a429348637b5b2f76a93caa9e5a31936f1e834 Mon Sep 17 00:00:00 2001 From: zimbatm Date: Tue, 27 May 2025 11:56:45 +0200 Subject: [PATCH 078/218] nix flake archive: add --no-check-sigs option Allows to copy the archive to a remote host and not get error: cannot add path '/nix/store/01x2k4nlxcpyd85nnr0b9gm89rm8ff4x-source' because it lacks a signature by a trusted key --- src/nix/flake.cc | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-) diff --git a/src/nix/flake.cc b/src/nix/flake.cc index 0a70fa4b4e7..95cf856637f 100644 --- a/src/nix/flake.cc +++ b/src/nix/flake.cc @@ -1051,6 +1051,10 @@ struct CmdFlakeArchive : FlakeCommand, MixJSON, MixDryRun { std::string dstUri; + CheckSigsFlag checkSigs = CheckSigs; + + SubstituteFlag substitute = NoSubstitute; + CmdFlakeArchive() { addFlag({ @@ -1059,6 +1063,11 @@ struct CmdFlakeArchive : FlakeCommand, MixJSON, MixDryRun .labels = {"store-uri"}, .handler = {&dstUri}, }); + addFlag({ + .longName = "no-check-sigs", + .description = "Do not require that paths are signed by trusted keys.", + .handler = {&checkSigs, NoCheckSigs}, + }); } std::string description() override @@ -1122,7 +1131,8 @@ struct CmdFlakeArchive : FlakeCommand, MixJSON, MixDryRun if (!dryRun && !dstUri.empty()) { ref dstStore = dstUri.empty() ? openStore() : openStore(dstUri); - copyPaths(*store, *dstStore, sources); + + copyPaths(*store, *dstStore, sources, NoRepair, checkSigs, substitute); } } }; From 5653bf5e0a89ff84182a5da584730c27f00d582a Mon Sep 17 00:00:00 2001 From: Eelco Dolstra Date: Mon, 26 May 2025 19:42:07 +0200 Subject: [PATCH 079/218] DerivationBuilder: Move Linux/Darwin-specific code into subclasses --- .../unix/build/darwin-derivation-builder.cc | 181 +++ src/libstore/unix/build/derivation-builder.cc | 1046 ++++------------- .../unix/build/linux-derivation-builder.cc | 568 +++++++++ 3 files changed, 952 insertions(+), 843 deletions(-) create mode 100644 src/libstore/unix/build/darwin-derivation-builder.cc create mode 100644 src/libstore/unix/build/linux-derivation-builder.cc diff --git a/src/libstore/unix/build/darwin-derivation-builder.cc b/src/libstore/unix/build/darwin-derivation-builder.cc new file mode 100644 index 00000000000..3366403a76f --- /dev/null +++ b/src/libstore/unix/build/darwin-derivation-builder.cc @@ -0,0 +1,181 @@ +#ifdef __APPLE__ + +struct DarwinDerivationBuilder : DerivationBuilderImpl +{ + DarwinDerivationBuilder( + Store & store, std::unique_ptr miscMethods, DerivationBuilderParams params) + : DerivationBuilderImpl(store, std::move(miscMethods), std::move(params)) + { + useChroot = true; + } + + void execBuilder(const Strings & args, const Strings & envStrs) override + { + posix_spawnattr_t attrp; + + if (posix_spawnattr_init(&attrp)) + throw SysError("failed to initialize builder"); + + if (posix_spawnattr_setflags(&attrp, POSIX_SPAWN_SETEXEC)) + throw SysError("failed to initialize builder"); + + if (drv.platform == "aarch64-darwin") { + // Unset kern.curproc_arch_affinity so we can escape Rosetta + int affinity = 0; + sysctlbyname("kern.curproc_arch_affinity", NULL, NULL, &affinity, sizeof(affinity)); + + cpu_type_t cpu = CPU_TYPE_ARM64; + posix_spawnattr_setbinpref_np(&attrp, 1, &cpu, NULL); + } else if (drv.platform == "x86_64-darwin") { + cpu_type_t cpu = CPU_TYPE_X86_64; + posix_spawnattr_setbinpref_np(&attrp, 1, &cpu, NULL); + } + + posix_spawn( + NULL, drv.builder.c_str(), NULL, &attrp, stringsToCharPtrs(args).data(), stringsToCharPtrs(envStrs).data()); + } + + void setUser() override + { + DerivationBuilderImpl::setUser(); + + /* This has to appear before import statements. */ + std::string sandboxProfile = "(version 1)\n"; + + if (useChroot) { + + /* Lots and lots and lots of file functions freak out if they can't stat their full ancestry */ + PathSet ancestry; + + /* We build the ancestry before adding all inputPaths to the store because we know they'll + all have the same parents (the store), and there might be lots of inputs. This isn't + particularly efficient... I doubt it'll be a bottleneck in practice */ + for (auto & i : pathsInChroot) { + Path cur = i.first; + while (cur.compare("/") != 0) { + cur = dirOf(cur); + ancestry.insert(cur); + } + } + + /* And we want the store in there regardless of how empty pathsInChroot. We include the innermost + path component this time, since it's typically /nix/store and we care about that. */ + Path cur = store.storeDir; + while (cur.compare("/") != 0) { + ancestry.insert(cur); + cur = dirOf(cur); + } + + /* Add all our input paths to the chroot */ + for (auto & i : inputPaths) { + auto p = store.printStorePath(i); + pathsInChroot[p] = p; + } + + /* Violations will go to the syslog if you set this. Unfortunately the destination does not appear to be + * configurable */ + if (settings.darwinLogSandboxViolations) { + sandboxProfile += "(deny default)\n"; + } else { + sandboxProfile += "(deny default (with no-log))\n"; + } + + sandboxProfile += +# include "sandbox-defaults.sb" + ; + + if (!derivationType->isSandboxed()) + sandboxProfile += +# include "sandbox-network.sb" + ; + + /* Add the output paths we'll use at build-time to the chroot */ + sandboxProfile += "(allow file-read* file-write* process-exec\n"; + for (auto & [_, path] : scratchOutputs) + sandboxProfile += fmt("\t(subpath \"%s\")\n", store.printStorePath(path)); + + sandboxProfile += ")\n"; + + /* Our inputs (transitive dependencies and any impurities computed above) + + without file-write* allowed, access() incorrectly returns EPERM + */ + sandboxProfile += "(allow file-read* file-write* process-exec\n"; + + // We create multiple allow lists, to avoid exceeding a limit in the darwin sandbox interpreter. + // See https://github.com/NixOS/nix/issues/4119 + // We split our allow groups approximately at half the actual limit, 1 << 16 + const size_t breakpoint = sandboxProfile.length() + (1 << 14); + for (auto & i : pathsInChroot) { + + if (sandboxProfile.length() >= breakpoint) { + debug("Sandbox break: %d %d", sandboxProfile.length(), breakpoint); + sandboxProfile += ")\n(allow file-read* file-write* process-exec\n"; + } + + if (i.first != i.second.source) + throw Error( + "can't map '%1%' to '%2%': mismatched impure paths not supported on Darwin", + i.first, + i.second.source); + + std::string path = i.first; + auto optSt = maybeLstat(path.c_str()); + if (!optSt) { + if (i.second.optional) + continue; + throw SysError("getting attributes of required path '%s", path); + } + if (S_ISDIR(optSt->st_mode)) + sandboxProfile += fmt("\t(subpath \"%s\")\n", path); + else + sandboxProfile += fmt("\t(literal \"%s\")\n", path); + } + sandboxProfile += ")\n"; + + /* Allow file-read* on full directory hierarchy to self. Allows realpath() */ + sandboxProfile += "(allow file-read*\n"; + for (auto & i : ancestry) { + sandboxProfile += fmt("\t(literal \"%s\")\n", i); + } + sandboxProfile += ")\n"; + + sandboxProfile += drvOptions.additionalSandboxProfile; + } else + sandboxProfile += +# include "sandbox-minimal.sb" + ; + + debug("Generated sandbox profile:"); + debug(sandboxProfile); + + /* The tmpDir in scope points at the temporary build directory for our derivation. Some packages try different + mechanisms to find temporary directories, so we want to open up a broader place for them to put their files, + if needed. */ + Path globalTmpDir = canonPath(defaultTempDir(), true); + + /* They don't like trailing slashes on subpath directives */ + while (!globalTmpDir.empty() && globalTmpDir.back() == '/') + globalTmpDir.pop_back(); + + if (getEnv("_NIX_TEST_NO_SANDBOX") != "1") { + Strings sandboxArgs; + sandboxArgs.push_back("_GLOBAL_TMP_DIR"); + sandboxArgs.push_back(globalTmpDir); + if (drvOptions.allowLocalNetworking) { + sandboxArgs.push_back("_ALLOW_LOCAL_NETWORKING"); + sandboxArgs.push_back("1"); + } + char * sandbox_errbuf = nullptr; + if (sandbox_init_with_parameters( + sandboxProfile.c_str(), 0, stringsToCharPtrs(sandboxArgs).data(), &sandbox_errbuf)) { + writeFull( + STDERR_FILENO, + fmt("failed to configure sandbox: %s\n", sandbox_errbuf ? sandbox_errbuf : "(null)")); + _exit(1); + } + } + } +} + +#endif diff --git a/src/libstore/unix/build/derivation-builder.cc b/src/libstore/unix/build/derivation-builder.cc index 54173c16259..c082c02343d 100644 --- a/src/libstore/unix/build/derivation-builder.cc +++ b/src/libstore/unix/build/derivation-builder.cc @@ -85,8 +85,11 @@ MakeError(NotDeterministic, BuildError); * rather than incoming call edges that either should be removed, or * become (higher order) function parameters. */ -class DerivationBuilderImpl : public DerivationBuilder, DerivationBuilderParams +// FIXME: rename this to UnixDerivationBuilder or something like that. +class DerivationBuilderImpl : public DerivationBuilder, public DerivationBuilderParams { +protected: + Store & store; std::unique_ptr miscMethods; @@ -103,9 +106,7 @@ class DerivationBuilderImpl : public DerivationBuilder, DerivationBuilderParams , derivationType{drv.type()} { } - LocalStore & getLocalStore(); - -private: +protected: /** * User selected for running the builder. @@ -133,32 +134,16 @@ class DerivationBuilderImpl : public DerivationBuilder, DerivationBuilderParams */ Path tmpDirInSandbox; - /** - * Pipe for synchronising updates to the builder namespaces. - */ - Pipe userNamespaceSync; - - /** - * The mount namespace and user namespace of the builder, used to add additional - * paths to the sandbox as a result of recursive Nix calls. - */ - AutoCloseFD sandboxMountNamespace; - AutoCloseFD sandboxUserNamespace; - - /** - * On Linux, whether we're doing the build in its own user - * namespace. - */ - bool usingUserNamespace = true; - /** * Whether we're currently doing a chroot build. */ + // FIXME: remove bool useChroot = false; /** * The root of the chroot environment. */ + // FIXME: move Path chrootRootDir; /** @@ -212,9 +197,6 @@ class DerivationBuilderImpl : public DerivationBuilder, DerivationBuilderParams */ OutputPathMap scratchOutputs; - uid_t sandboxUid() { return usingUserNamespace ? (!buildUser || buildUser->getUIDCount() == 1 ? 1000 : 0) : buildUser->getUID(); } - gid_t sandboxGid() { return usingUserNamespace ? (!buildUser || buildUser->getUIDCount() == 1 ? 100 : 0) : buildUser->getGID(); } - const static Path homeDir; /** @@ -253,7 +235,10 @@ class DerivationBuilderImpl : public DerivationBuilder, DerivationBuilderParams /** * Whether we need to perform hash rewriting if there are valid output paths. */ - bool needsHashRewrite(); + virtual bool needsHashRewrite() + { + return true; + } public: @@ -263,6 +248,25 @@ class DerivationBuilderImpl : public DerivationBuilder, DerivationBuilderParams std::variant, SingleDrvOutputs> unprepareBuild() override; +protected: + + /** + * Called by prepareBuild() to do any setup in the parent to + * prepare for a sandboxed build. + */ + virtual void prepareSandbox(); + + /** + * Open the slave side of the pseudoterminal and use it as stderr. + */ + void openSlave(); + + /** + * Called by prepareBuild() to start the child process for the + * build. Must set `pid`. The child must call runChild(). + */ + virtual void startChild(); + private: /** @@ -270,11 +274,15 @@ class DerivationBuilderImpl : public DerivationBuilder, DerivationBuilderParams */ void initEnv(); +protected: + /** * Process messages send by the sandbox initialization. */ void processSandboxSetupMessages(); +private: + /** * Setup tmp dir location. */ @@ -298,6 +306,8 @@ class DerivationBuilderImpl : public DerivationBuilder, DerivationBuilderParams void addDependency(const StorePath & path) override; +protected: + /** * Make a file owned by the builder. */ @@ -308,6 +318,28 @@ class DerivationBuilderImpl : public DerivationBuilder, DerivationBuilderParams */ void runChild(); +private: + + /** + * Move the current process into the chroot, if any. Called early + * by runChild(). + */ + virtual void enterChroot() + { + } + + /** + * Change the current process's uid/gid to the build user, if + * any. Called by runChild(). + */ + virtual void setUser(); + + /** + * Execute the derivation builder process. Called by runChild() as + * its final step. Should not return unless there is an error. + */ + virtual void execBuilder(const Strings & args, const Strings & envStrs); + /** * Check that the derivation outputs all exist and register them * as valid. @@ -348,17 +380,6 @@ class DerivationBuilderImpl : public DerivationBuilder, DerivationBuilderParams StorePath makeFallbackPath(OutputNameView outputName); }; -std::unique_ptr makeDerivationBuilder( - Store & store, - std::unique_ptr miscMethods, - DerivationBuilderParams params) -{ - return std::make_unique( - store, - std::move(miscMethods), - std::move(params)); -} - void handleDiffHook( uid_t uid, uid_t gid, const Path & tryA, const Path & tryB, @@ -396,18 +417,7 @@ void handleDiffHook( const Path DerivationBuilderImpl::homeDir = "/homeless-shelter"; -inline bool DerivationBuilderImpl::needsHashRewrite() -{ -#ifdef __linux__ - return !useChroot; -#else - /* Darwin requires hash rewriting even when sandboxing is enabled. */ - return true; -#endif -} - - -LocalStore & DerivationBuilderImpl::getLocalStore() +static LocalStore & getLocalStore(Store & store) { auto p = dynamic_cast(&store); assert(p); @@ -439,45 +449,6 @@ void DerivationBuilderImpl::killSandbox(bool getStats) bool DerivationBuilderImpl::prepareBuild() { - /* Are we doing a chroot build? */ - { - if (settings.sandboxMode == smEnabled) { - if (drvOptions.noChroot) - throw Error("derivation '%s' has '__noChroot' set, " - "but that's not allowed when 'sandbox' is 'true'", store.printStorePath(drvPath)); -#ifdef __APPLE__ - if (drvOptions.additionalSandboxProfile != "") - throw Error("derivation '%s' specifies a sandbox profile, " - "but this is only allowed when 'sandbox' is 'relaxed'", store.printStorePath(drvPath)); -#endif - useChroot = true; - } - else if (settings.sandboxMode == smDisabled) - useChroot = false; - else if (settings.sandboxMode == smRelaxed) - useChroot = derivationType.isSandboxed() && !drvOptions.noChroot; - } - - auto & localStore = getLocalStore(); - if (localStore.storeDir != localStore.config->realStoreDir.get()) { - #ifdef __linux__ - useChroot = true; - #else - throw Error("building using a diverted store is not supported on this platform"); - #endif - } - - #ifdef __linux__ - if (useChroot) { - if (!mountAndPidNamespacesSupported()) { - if (!settings.sandboxFallback) - throw Error("this system does not support the kernel namespaces that are required for sandboxing; use '--no-sandbox' to disable sandboxing"); - debug("auto-disabling sandboxing because the prerequisite namespaces are not available"); - useChroot = false; - } - } - #endif - if (useBuildUsers()) { if (!buildUser) buildUser = acquireUserLock(drvOptions.useUidRange(drv) ? 65536 : 1, useChroot); @@ -493,6 +464,7 @@ bool DerivationBuilderImpl::prepareBuild() std::variant, SingleDrvOutputs> DerivationBuilderImpl::unprepareBuild() { + // FIXME: get rid of this, rely on RAII. Finally releaseBuildUser([&](){ /* Release the build user at the end of this function. We don't do it right away because we don't want another build grabbing this @@ -500,9 +472,6 @@ std::variant, SingleDrvOutputs> Derivation buildUser.reset(); }); - sandboxMountNamespace = -1; - sandboxUserNamespace = -1; - /* Since we got an EOF on the logger pipe, the builder is presumed to have terminated. In fact, the builder could also have simply have closed its end of the pipe, so just to be sure, @@ -678,7 +647,7 @@ bool DerivationBuilderImpl::cleanupDecideWhetherDiskFull() so, we don't mark this build as a permanent failure. */ #if HAVE_STATVFS { - auto & localStore = getLocalStore(); + auto & localStore = getLocalStore(store); uint64_t required = 8ULL * 1024 * 1024; // FIXME: make configurable struct statvfs st; if (statvfs(localStore.config->realStoreDir.get().c_str(), &st) == 0 && @@ -1015,118 +984,13 @@ void DerivationBuilderImpl::startBuilder() macOS 11+ has no /usr/lib/libSystem*.dylib */ pathsInChroot[i] = {i, true}; } - -#ifdef __linux__ - /* Create a temporary directory in which we set up the chroot - environment using bind-mounts. We put it in the Nix store - so that the build outputs can be moved efficiently from the - chroot to their final location. */ - auto chrootParentDir = store.Store::toRealPath(drvPath) + ".chroot"; - deletePath(chrootParentDir); - - /* Clean up the chroot directory automatically. */ - autoDelChroot = std::make_shared(chrootParentDir); - - printMsg(lvlChatty, "setting up chroot environment in '%1%'", chrootParentDir); - - if (mkdir(chrootParentDir.c_str(), 0700) == -1) - throw SysError("cannot create '%s'", chrootRootDir); - - chrootRootDir = chrootParentDir + "/root"; - - if (mkdir(chrootRootDir.c_str(), buildUser && buildUser->getUIDCount() != 1 ? 0755 : 0750) == -1) - throw SysError("cannot create '%1%'", chrootRootDir); - - if (buildUser && chown(chrootRootDir.c_str(), buildUser->getUIDCount() != 1 ? buildUser->getUID() : 0, buildUser->getGID()) == -1) - throw SysError("cannot change ownership of '%1%'", chrootRootDir); - - /* Create a writable /tmp in the chroot. Many builders need - this. (Of course they should really respect $TMPDIR - instead.) */ - Path chrootTmpDir = chrootRootDir + "/tmp"; - createDirs(chrootTmpDir); - chmod_(chrootTmpDir, 01777); - - /* Create a /etc/passwd with entries for the build user and the - nobody account. The latter is kind of a hack to support - Samba-in-QEMU. */ - createDirs(chrootRootDir + "/etc"); - if (drvOptions.useUidRange(drv)) - chownToBuilder(chrootRootDir + "/etc"); - - if (drvOptions.useUidRange(drv) && (!buildUser || buildUser->getUIDCount() < 65536)) - throw Error("feature 'uid-range' requires the setting '%s' to be enabled", settings.autoAllocateUids.name); - - /* Declare the build user's group so that programs get a consistent - view of the system (e.g., "id -gn"). */ - writeFile(chrootRootDir + "/etc/group", - fmt("root:x:0:\n" - "nixbld:!:%1%:\n" - "nogroup:x:65534:\n", sandboxGid())); - - /* Create /etc/hosts with localhost entry. */ - if (derivationType.isSandboxed()) - writeFile(chrootRootDir + "/etc/hosts", "127.0.0.1 localhost\n::1 localhost\n"); - - /* Make the closure of the inputs available in the chroot, - rather than the whole Nix store. This prevents any access - to undeclared dependencies. Directories are bind-mounted, - while other inputs are hard-linked (since only directories - can be bind-mounted). !!! As an extra security - precaution, make the fake Nix store only writable by the - build user. */ - Path chrootStoreDir = chrootRootDir + store.storeDir; - createDirs(chrootStoreDir); - chmod_(chrootStoreDir, 01775); - - if (buildUser && chown(chrootStoreDir.c_str(), 0, buildUser->getGID()) == -1) - throw SysError("cannot change ownership of '%1%'", chrootStoreDir); - - for (auto & i : inputPaths) { - auto p = store.printStorePath(i); - Path r = store.toRealPath(p); - pathsInChroot.insert_or_assign(p, r); - } - - /* If we're repairing, checking or rebuilding part of a - multiple-outputs derivation, it's possible that we're - rebuilding a path that is in settings.sandbox-paths - (typically the dependencies of /bin/sh). Throw them - out. */ - for (auto & i : drv.outputsAndOptPaths(store)) { - /* If the name isn't known a priori (i.e. floating - content-addressing derivation), the temporary location we use - should be fresh. Freshness means it is impossible that the path - is already in the sandbox, so we don't need to worry about - removing it. */ - if (i.second.second) - pathsInChroot.erase(store.printStorePath(*i.second.second)); - } - - if (cgroup) { - if (mkdir(cgroup->c_str(), 0755) != 0) - throw SysError("creating cgroup '%s'", *cgroup); - chownToBuilder(*cgroup); - chownToBuilder(*cgroup + "/cgroup.procs"); - chownToBuilder(*cgroup + "/cgroup.threads"); - //chownToBuilder(*cgroup + "/cgroup.subtree_control"); - } - -#else - if (drvOptions.useUidRange(drv)) - throw Error("feature 'uid-range' is not supported on this platform"); - #ifdef __APPLE__ - /* We don't really have any parent prep work to do (yet?) - All work happens in the child, instead. */ - #else - throw Error("sandboxing builds is not supported on this platform"); - #endif -#endif } else { if (drvOptions.useUidRange(drv)) throw Error("feature 'uid-range' is only supported in sandboxed builds"); } + prepareSandbox(); + if (needsHashRewrite() && pathExists(homeDir)) throw Error("home directory '%1%' exists; please remove it to assure purity of builds without sandboxing", homeDir); @@ -1205,194 +1069,52 @@ void DerivationBuilderImpl::startBuilder() if (unlockpt(builderOut.get())) throw SysError("unlocking pseudoterminal"); - /* Open the slave side of the pseudoterminal and use it as stderr. */ - auto openSlave = [&]() - { - AutoCloseFD builderOut = open(slaveName.c_str(), O_RDWR | O_NOCTTY); - if (!builderOut) - throw SysError("opening pseudoterminal slave"); - - // Put the pt into raw mode to prevent \n -> \r\n translation. - struct termios term; - if (tcgetattr(builderOut.get(), &term)) - throw SysError("getting pseudoterminal attributes"); - - cfmakeraw(&term); - - if (tcsetattr(builderOut.get(), TCSANOW, &term)) - throw SysError("putting pseudoterminal into raw mode"); - - if (dup2(builderOut.get(), STDERR_FILENO) == -1) - throw SysError("cannot pipe standard error into log file"); - }; - buildResult.startTime = time(0); - /* Fork a child to build the package. */ + /* Start a child process to build the derivation. */ + startChild(); -#ifdef __linux__ - if (useChroot) { - /* Set up private namespaces for the build: - - - The PID namespace causes the build to start as PID 1. - Processes outside of the chroot are not visible to those - on the inside, but processes inside the chroot are - visible from the outside (though with different PIDs). - - - The private mount namespace ensures that all the bind - mounts we do will only show up in this process and its - children, and will disappear automatically when we're - done. - - - The private network namespace ensures that the builder - cannot talk to the outside world (or vice versa). It - only has a private loopback interface. (Fixed-output - derivations are not run in a private network namespace - to allow functions like fetchurl to work.) - - - The IPC namespace prevents the builder from communicating - with outside processes using SysV IPC mechanisms (shared - memory, message queues, semaphores). It also ensures - that all IPC objects are destroyed when the builder - exits. - - - The UTS namespace ensures that builders see a hostname of - localhost rather than the actual hostname. - - We use a helper process to do the clone() to work around - clone() being broken in multi-threaded programs due to - at-fork handlers not being run. Note that we use - CLONE_PARENT to ensure that the real builder is parented to - us. - */ - - userNamespaceSync.create(); - - usingUserNamespace = userNamespacesSupported(); - - Pipe sendPid; - sendPid.create(); - - Pid helper = startProcess([&]() { - sendPid.readSide.close(); - - /* We need to open the slave early, before - CLONE_NEWUSER. Otherwise we get EPERM when running as - root. */ - openSlave(); - - try { - /* Drop additional groups here because we can't do it - after we've created the new user namespace. */ - if (setgroups(0, 0) == -1) { - if (errno != EPERM) - throw SysError("setgroups failed"); - if (settings.requireDropSupplementaryGroups) - throw Error("setgroups failed. Set the require-drop-supplementary-groups option to false to skip this step."); - } - - ProcessOptions options; - options.cloneFlags = CLONE_NEWPID | CLONE_NEWNS | CLONE_NEWIPC | CLONE_NEWUTS | CLONE_PARENT | SIGCHLD; - if (derivationType.isSandboxed()) - options.cloneFlags |= CLONE_NEWNET; - if (usingUserNamespace) - options.cloneFlags |= CLONE_NEWUSER; - - pid_t child = startProcess([&]() { runChild(); }, options); - - writeFull(sendPid.writeSide.get(), fmt("%d\n", child)); - _exit(0); - } catch (...) { - handleChildException(true); - _exit(1); - } - }); - - sendPid.writeSide.close(); - - if (helper.wait() != 0) { - processSandboxSetupMessages(); - // Only reached if the child process didn't send an exception. - throw Error("unable to start build process"); - } - - userNamespaceSync.readSide = -1; - - /* Close the write side to prevent runChild() from hanging - reading from this. */ - Finally cleanup([&]() { - userNamespaceSync.writeSide = -1; - }); - - auto ss = tokenizeString>(readLine(sendPid.readSide.get())); - assert(ss.size() == 1); - pid = string2Int(ss[0]).value(); - - if (usingUserNamespace) { - /* Set the UID/GID mapping of the builder's user namespace - such that the sandbox user maps to the build user, or to - the calling user (if build users are disabled). */ - uid_t hostUid = buildUser ? buildUser->getUID() : getuid(); - uid_t hostGid = buildUser ? buildUser->getGID() : getgid(); - uid_t nrIds = buildUser ? buildUser->getUIDCount() : 1; - - writeFile("/proc/" + std::to_string(pid) + "/uid_map", - fmt("%d %d %d", sandboxUid(), hostUid, nrIds)); + pid.setSeparatePG(true); + miscMethods->childStarted(builderOut.get()); - if (!buildUser || buildUser->getUIDCount() == 1) - writeFile("/proc/" + std::to_string(pid) + "/setgroups", "deny"); + processSandboxSetupMessages(); +} - writeFile("/proc/" + std::to_string(pid) + "/gid_map", - fmt("%d %d %d", sandboxGid(), hostGid, nrIds)); - } else { - debug("note: not using a user namespace"); - if (!buildUser) - throw Error("cannot perform a sandboxed build because user namespaces are not enabled; check /proc/sys/user/max_user_namespaces"); - } +void DerivationBuilderImpl::prepareSandbox() +{ + if (drvOptions.useUidRange(drv)) + throw Error("feature 'uid-range' is not supported on this platform"); +} - /* Now that we now the sandbox uid, we can write - /etc/passwd. */ - writeFile(chrootRootDir + "/etc/passwd", fmt( - "root:x:0:0:Nix build user:%3%:/noshell\n" - "nixbld:x:%1%:%2%:Nix build user:%3%:/noshell\n" - "nobody:x:65534:65534:Nobody:/:/noshell\n", - sandboxUid(), sandboxGid(), settings.sandboxBuildDir)); - - /* Save the mount- and user namespace of the child. We have to do this - *before* the child does a chroot. */ - sandboxMountNamespace = open(fmt("/proc/%d/ns/mnt", (pid_t) pid).c_str(), O_RDONLY); - if (sandboxMountNamespace.get() == -1) - throw SysError("getting sandbox mount namespace"); - - if (usingUserNamespace) { - sandboxUserNamespace = open(fmt("/proc/%d/ns/user", (pid_t) pid).c_str(), O_RDONLY); - if (sandboxUserNamespace.get() == -1) - throw SysError("getting sandbox user namespace"); - } +void DerivationBuilderImpl::openSlave() +{ + std::string slaveName = ptsname(builderOut.get()); - /* Move the child into its own cgroup. */ - if (cgroup) - writeFile(*cgroup + "/cgroup.procs", fmt("%d", (pid_t) pid)); + AutoCloseFD builderOut = open(slaveName.c_str(), O_RDWR | O_NOCTTY); + if (!builderOut) + throw SysError("opening pseudoterminal slave"); - /* Signal the builder that we've updated its user namespace. */ - writeFull(userNamespaceSync.writeSide.get(), "1"); + // Put the pt into raw mode to prevent \n -> \r\n translation. + struct termios term; + if (tcgetattr(builderOut.get(), &term)) + throw SysError("getting pseudoterminal attributes"); - } else -#endif - { - pid = startProcess([&]() { - openSlave(); - runChild(); - }); - } + cfmakeraw(&term); - /* parent */ - pid.setSeparatePG(true); - miscMethods->childStarted(builderOut.get()); + if (tcsetattr(builderOut.get(), TCSANOW, &term)) + throw SysError("putting pseudoterminal into raw mode"); - processSandboxSetupMessages(); + if (dup2(builderOut.get(), STDERR_FILENO) == -1) + throw SysError("cannot pipe standard error into log file"); } +void DerivationBuilderImpl::startChild() +{ + pid = startProcess([&]() { + openSlave(); + runChild(); + }); +} void DerivationBuilderImpl::processSandboxSetupMessages() { @@ -1570,7 +1292,7 @@ void DerivationBuilderImpl::startDaemon() auto store = makeRestrictedStore( [&]{ - auto config = make_ref(*getLocalStore().config); + auto config = make_ref(*getLocalStore(this->store).config); config->pathInfoCacheSize = 0; config->stateDir = "/no-such-path"; config->logDir = "/no-such-path"; @@ -1670,51 +1392,6 @@ void DerivationBuilderImpl::addDependency(const StorePath & path) if (isAllowed(path)) return; addedPaths.insert(path); - - /* If we're doing a sandbox build, then we have to make the path - appear in the sandbox. */ - if (useChroot) { - - debug("materialising '%s' in the sandbox", store.printStorePath(path)); - - #ifdef __linux__ - - Path source = store.Store::toRealPath(path); - Path target = chrootRootDir + store.printStorePath(path); - - if (pathExists(target)) { - // There is a similar debug message in doBind, so only run it in this block to not have double messages. - debug("bind-mounting %s -> %s", target, source); - throw Error("store path '%s' already exists in the sandbox", store.printStorePath(path)); - } - - /* Bind-mount the path into the sandbox. This requires - entering its mount namespace, which is not possible - in multithreaded programs. So we do this in a - child process.*/ - Pid child(startProcess([&]() { - - if (usingUserNamespace && (setns(sandboxUserNamespace.get(), 0) == -1)) - throw SysError("entering sandbox user namespace"); - - if (setns(sandboxMountNamespace.get(), 0) == -1) - throw SysError("entering sandbox mount namespace"); - - doBind(source, target); - - _exit(0); - })); - - int status = child.wait(); - if (status != 0) - throw Error("could not add path '%s' to sandbox", store.printStorePath(path)); - - #else - throw Error("don't know how to make path '%s' (produced by a recursive Nix call) appear in the sandbox", - store.printStorePath(path)); - #endif - - } } void DerivationBuilderImpl::chownToBuilder(const Path & path) @@ -1830,8 +1507,6 @@ void DerivationBuilderImpl::runChild() if (buildUser) throw; } - bool setUser = true; - /* Make the contents of netrc and the CA certificate bundle available to builtin:fetchurl (which may run under a different uid and/or in a sandbox). */ @@ -1850,234 +1525,7 @@ void DerivationBuilderImpl::runChild() } catch (SystemError &) { } } -#ifdef __linux__ - if (useChroot) { - - userNamespaceSync.writeSide = -1; - - if (drainFD(userNamespaceSync.readSide.get()) != "1") - throw Error("user namespace initialisation failed"); - - userNamespaceSync.readSide = -1; - - if (derivationType.isSandboxed()) { - - /* Initialise the loopback interface. */ - AutoCloseFD fd(socket(PF_INET, SOCK_DGRAM, IPPROTO_IP)); - if (!fd) throw SysError("cannot open IP socket"); - - struct ifreq ifr; - strcpy(ifr.ifr_name, "lo"); - ifr.ifr_flags = IFF_UP | IFF_LOOPBACK | IFF_RUNNING; - if (ioctl(fd.get(), SIOCSIFFLAGS, &ifr) == -1) - throw SysError("cannot set loopback interface flags"); - } - - /* Set the hostname etc. to fixed values. */ - char hostname[] = "localhost"; - if (sethostname(hostname, sizeof(hostname)) == -1) - throw SysError("cannot set host name"); - char domainname[] = "(none)"; // kernel default - if (setdomainname(domainname, sizeof(domainname)) == -1) - throw SysError("cannot set domain name"); - - /* Make all filesystems private. This is necessary - because subtrees may have been mounted as "shared" - (MS_SHARED). (Systemd does this, for instance.) Even - though we have a private mount namespace, mounting - filesystems on top of a shared subtree still propagates - outside of the namespace. Making a subtree private is - local to the namespace, though, so setting MS_PRIVATE - does not affect the outside world. */ - if (mount(0, "/", 0, MS_PRIVATE | MS_REC, 0) == -1) - throw SysError("unable to make '/' private"); - - /* Bind-mount chroot directory to itself, to treat it as a - different filesystem from /, as needed for pivot_root. */ - if (mount(chrootRootDir.c_str(), chrootRootDir.c_str(), 0, MS_BIND, 0) == -1) - throw SysError("unable to bind mount '%1%'", chrootRootDir); - - /* Bind-mount the sandbox's Nix store onto itself so that - we can mark it as a "shared" subtree, allowing bind - mounts made in *this* mount namespace to be propagated - into the child namespace created by the - unshare(CLONE_NEWNS) call below. - - Marking chrootRootDir as MS_SHARED causes pivot_root() - to fail with EINVAL. Don't know why. */ - Path chrootStoreDir = chrootRootDir + store.storeDir; - - if (mount(chrootStoreDir.c_str(), chrootStoreDir.c_str(), 0, MS_BIND, 0) == -1) - throw SysError("unable to bind mount the Nix store", chrootStoreDir); - - if (mount(0, chrootStoreDir.c_str(), 0, MS_SHARED, 0) == -1) - throw SysError("unable to make '%s' shared", chrootStoreDir); - - /* Set up a nearly empty /dev, unless the user asked to - bind-mount the host /dev. */ - Strings ss; - if (pathsInChroot.find("/dev") == pathsInChroot.end()) { - createDirs(chrootRootDir + "/dev/shm"); - createDirs(chrootRootDir + "/dev/pts"); - ss.push_back("/dev/full"); - if (store.config.systemFeatures.get().count("kvm") && pathExists("/dev/kvm")) - ss.push_back("/dev/kvm"); - ss.push_back("/dev/null"); - ss.push_back("/dev/random"); - ss.push_back("/dev/tty"); - ss.push_back("/dev/urandom"); - ss.push_back("/dev/zero"); - createSymlink("/proc/self/fd", chrootRootDir + "/dev/fd"); - createSymlink("/proc/self/fd/0", chrootRootDir + "/dev/stdin"); - createSymlink("/proc/self/fd/1", chrootRootDir + "/dev/stdout"); - createSymlink("/proc/self/fd/2", chrootRootDir + "/dev/stderr"); - } - - /* Fixed-output derivations typically need to access the - network, so give them access to /etc/resolv.conf and so - on. */ - if (!derivationType.isSandboxed()) { - // Only use nss functions to resolve hosts and - // services. Don’t use it for anything else that may - // be configured for this system. This limits the - // potential impurities introduced in fixed-outputs. - writeFile(chrootRootDir + "/etc/nsswitch.conf", "hosts: files dns\nservices: files\n"); - - /* N.B. it is realistic that these paths might not exist. It - happens when testing Nix building fixed-output derivations - within a pure derivation. */ - for (auto & path : { "/etc/resolv.conf", "/etc/services", "/etc/hosts" }) - if (pathExists(path)) - ss.push_back(path); - - if (settings.caFile != "") { - Path caFile = settings.caFile; - if (pathExists(caFile)) - pathsInChroot.try_emplace("/etc/ssl/certs/ca-certificates.crt", canonPath(caFile, true), true); - } - } - - for (auto & i : ss) { - // For backwards-compatibility, resolve all the symlinks in the - // chroot paths - auto canonicalPath = canonPath(i, true); - pathsInChroot.emplace(i, canonicalPath); - } - - /* Bind-mount all the directories from the "host" - filesystem that we want in the chroot - environment. */ - for (auto & i : pathsInChroot) { - if (i.second.source == "/proc") continue; // backwards compatibility - - #if HAVE_EMBEDDED_SANDBOX_SHELL - if (i.second.source == "__embedded_sandbox_shell__") { - static unsigned char sh[] = { - #include "embedded-sandbox-shell.gen.hh" - }; - auto dst = chrootRootDir + i.first; - createDirs(dirOf(dst)); - writeFile(dst, std::string_view((const char *) sh, sizeof(sh))); - chmod_(dst, 0555); - } else - #endif - doBind(i.second.source, chrootRootDir + i.first, i.second.optional); - } - - /* Bind a new instance of procfs on /proc. */ - createDirs(chrootRootDir + "/proc"); - if (mount("none", (chrootRootDir + "/proc").c_str(), "proc", 0, 0) == -1) - throw SysError("mounting /proc"); - - /* Mount sysfs on /sys. */ - if (buildUser && buildUser->getUIDCount() != 1) { - createDirs(chrootRootDir + "/sys"); - if (mount("none", (chrootRootDir + "/sys").c_str(), "sysfs", 0, 0) == -1) - throw SysError("mounting /sys"); - } - - /* Mount a new tmpfs on /dev/shm to ensure that whatever - the builder puts in /dev/shm is cleaned up automatically. */ - if (pathExists("/dev/shm") && mount("none", (chrootRootDir + "/dev/shm").c_str(), "tmpfs", 0, - fmt("size=%s", settings.sandboxShmSize).c_str()) == -1) - throw SysError("mounting /dev/shm"); - - /* Mount a new devpts on /dev/pts. Note that this - requires the kernel to be compiled with - CONFIG_DEVPTS_MULTIPLE_INSTANCES=y (which is the case - if /dev/ptx/ptmx exists). */ - if (pathExists("/dev/pts/ptmx") && - !pathExists(chrootRootDir + "/dev/ptmx") - && !pathsInChroot.count("/dev/pts")) - { - if (mount("none", (chrootRootDir + "/dev/pts").c_str(), "devpts", 0, "newinstance,mode=0620") == 0) - { - createSymlink("/dev/pts/ptmx", chrootRootDir + "/dev/ptmx"); - - /* Make sure /dev/pts/ptmx is world-writable. With some - Linux versions, it is created with permissions 0. */ - chmod_(chrootRootDir + "/dev/pts/ptmx", 0666); - } else { - if (errno != EINVAL) - throw SysError("mounting /dev/pts"); - doBind("/dev/pts", chrootRootDir + "/dev/pts"); - doBind("/dev/ptmx", chrootRootDir + "/dev/ptmx"); - } - } - - /* Make /etc unwritable */ - if (!drvOptions.useUidRange(drv)) - chmod_(chrootRootDir + "/etc", 0555); - - /* Unshare this mount namespace. This is necessary because - pivot_root() below changes the root of the mount - namespace. This means that the call to setns() in - addDependency() would hide the host's filesystem, - making it impossible to bind-mount paths from the host - Nix store into the sandbox. Therefore, we save the - pre-pivot_root namespace in - sandboxMountNamespace. Since we made /nix/store a - shared subtree above, this allows addDependency() to - make paths appear in the sandbox. */ - if (unshare(CLONE_NEWNS) == -1) - throw SysError("unsharing mount namespace"); - - /* Unshare the cgroup namespace. This means - /proc/self/cgroup will show the child's cgroup as '/' - rather than whatever it is in the parent. */ - if (cgroup && unshare(CLONE_NEWCGROUP) == -1) - throw SysError("unsharing cgroup namespace"); - - /* Do the chroot(). */ - if (chdir(chrootRootDir.c_str()) == -1) - throw SysError("cannot change directory to '%1%'", chrootRootDir); - - if (mkdir("real-root", 0500) == -1) - throw SysError("cannot create real-root directory"); - - if (pivot_root(".", "real-root") == -1) - throw SysError("cannot pivot old root directory onto '%1%'", (chrootRootDir + "/real-root")); - - if (chroot(".") == -1) - throw SysError("cannot change root directory to '%1%'", chrootRootDir); - - if (umount2("real-root", MNT_DETACH) == -1) - throw SysError("cannot unmount real root filesystem"); - - if (rmdir("real-root") == -1) - throw SysError("cannot remove real-root directory"); - - /* Switch to the sandbox uid/gid in the user namespace, - which corresponds to the build user or calling user in - the parent namespace. */ - if (setgid(sandboxGid()) == -1) - throw SysError("setgid failed"); - if (setuid(sandboxUid()) == -1) - throw SysError("setuid failed"); - - setUser = false; - } -#endif + enterChroot(); if (chdir(tmpDirInSandbox.c_str()) == -1) throw SysError("changing into '%1%'", tmpDir); @@ -2085,184 +1533,20 @@ void DerivationBuilderImpl::runChild() /* Close all other file descriptors. */ unix::closeExtraFDs(); -#ifdef __linux__ - linux::setPersonality(drv.platform); -#endif - /* Disable core dumps by default. */ struct rlimit limit = { 0, RLIM_INFINITY }; setrlimit(RLIMIT_CORE, &limit); // FIXME: set other limits to deterministic values? - /* Fill in the environment. */ - Strings envStrs; - for (auto & i : env) - envStrs.push_back(rewriteStrings(i.first + "=" + i.second, inputRewrites)); - - /* If we are running in `build-users' mode, then switch to the - user we allocated above. Make sure that we drop all root - privileges. Note that above we have closed all file - descriptors except std*, so that's safe. Also note that - setuid() when run as root sets the real, effective and - saved UIDs. */ - if (setUser && buildUser) { - /* Preserve supplementary groups of the build user, to allow - admins to specify groups such as "kvm". */ - auto gids = buildUser->getSupplementaryGIDs(); - if (setgroups(gids.size(), gids.data()) == -1) - throw SysError("cannot set supplementary groups of build user"); - - if (setgid(buildUser->getGID()) == -1 || - getgid() != buildUser->getGID() || - getegid() != buildUser->getGID()) - throw SysError("setgid failed"); - - if (setuid(buildUser->getUID()) == -1 || - getuid() != buildUser->getUID() || - geteuid() != buildUser->getUID()) - throw SysError("setuid failed"); - } - -#ifdef __APPLE__ - /* This has to appear before import statements. */ - std::string sandboxProfile = "(version 1)\n"; - - if (useChroot) { - - /* Lots and lots and lots of file functions freak out if they can't stat their full ancestry */ - PathSet ancestry; - - /* We build the ancestry before adding all inputPaths to the store because we know they'll - all have the same parents (the store), and there might be lots of inputs. This isn't - particularly efficient... I doubt it'll be a bottleneck in practice */ - for (auto & i : pathsInChroot) { - Path cur = i.first; - while (cur.compare("/") != 0) { - cur = dirOf(cur); - ancestry.insert(cur); - } - } - - /* And we want the store in there regardless of how empty pathsInChroot. We include the innermost - path component this time, since it's typically /nix/store and we care about that. */ - Path cur = store.storeDir; - while (cur.compare("/") != 0) { - ancestry.insert(cur); - cur = dirOf(cur); - } - - /* Add all our input paths to the chroot */ - for (auto & i : inputPaths) { - auto p = store.printStorePath(i); - pathsInChroot[p] = p; - } - - /* Violations will go to the syslog if you set this. Unfortunately the destination does not appear to be configurable */ - if (settings.darwinLogSandboxViolations) { - sandboxProfile += "(deny default)\n"; - } else { - sandboxProfile += "(deny default (with no-log))\n"; - } - - sandboxProfile += - #include "sandbox-defaults.sb" - ; - - if (!derivationType.isSandboxed()) - sandboxProfile += - #include "sandbox-network.sb" - ; - - /* Add the output paths we'll use at build-time to the chroot */ - sandboxProfile += "(allow file-read* file-write* process-exec\n"; - for (auto & [_, path] : scratchOutputs) - sandboxProfile += fmt("\t(subpath \"%s\")\n", store.printStorePath(path)); - - sandboxProfile += ")\n"; - - /* Our inputs (transitive dependencies and any impurities computed above) - - without file-write* allowed, access() incorrectly returns EPERM - */ - sandboxProfile += "(allow file-read* file-write* process-exec\n"; - - // We create multiple allow lists, to avoid exceeding a limit in the darwin sandbox interpreter. - // See https://github.com/NixOS/nix/issues/4119 - // We split our allow groups approximately at half the actual limit, 1 << 16 - const size_t breakpoint = sandboxProfile.length() + (1 << 14); - for (auto & i : pathsInChroot) { - - if (sandboxProfile.length() >= breakpoint) { - debug("Sandbox break: %d %d", sandboxProfile.length(), breakpoint); - sandboxProfile += ")\n(allow file-read* file-write* process-exec\n"; - } - - if (i.first != i.second.source) - throw Error( - "can't map '%1%' to '%2%': mismatched impure paths not supported on Darwin", - i.first, i.second.source); - - std::string path = i.first; - auto optSt = maybeLstat(path.c_str()); - if (!optSt) { - if (i.second.optional) - continue; - throw SysError("getting attributes of required path '%s", path); - } - if (S_ISDIR(optSt->st_mode)) - sandboxProfile += fmt("\t(subpath \"%s\")\n", path); - else - sandboxProfile += fmt("\t(literal \"%s\")\n", path); - } - sandboxProfile += ")\n"; - - /* Allow file-read* on full directory hierarchy to self. Allows realpath() */ - sandboxProfile += "(allow file-read*\n"; - for (auto & i : ancestry) { - sandboxProfile += fmt("\t(literal \"%s\")\n", i); - } - sandboxProfile += ")\n"; - - sandboxProfile += drvOptions.additionalSandboxProfile; - } else - sandboxProfile += - #include "sandbox-minimal.sb" - ; - - debug("Generated sandbox profile:"); - debug(sandboxProfile); - - /* The tmpDir in scope points at the temporary build directory for our derivation. Some packages try different mechanisms - to find temporary directories, so we want to open up a broader place for them to put their files, if needed. */ - Path globalTmpDir = canonPath(defaultTempDir(), true); - - /* They don't like trailing slashes on subpath directives */ - while (!globalTmpDir.empty() && globalTmpDir.back() == '/') - globalTmpDir.pop_back(); - - if (getEnv("_NIX_TEST_NO_SANDBOX") != "1") { - Strings sandboxArgs; - sandboxArgs.push_back("_GLOBAL_TMP_DIR"); - sandboxArgs.push_back(globalTmpDir); - if (drvOptions.allowLocalNetworking) { - sandboxArgs.push_back("_ALLOW_LOCAL_NETWORKING"); - sandboxArgs.push_back("1"); - } - char * sandbox_errbuf = nullptr; - if (sandbox_init_with_parameters(sandboxProfile.c_str(), 0, stringsToCharPtrs(sandboxArgs).data(), &sandbox_errbuf)) { - writeFull(STDERR_FILENO, fmt("failed to configure sandbox: %s\n", sandbox_errbuf ? sandbox_errbuf : "(null)")); - _exit(1); - } - } -#endif + setUser(); /* Indicate that we managed to set up the build environment. */ writeFull(STDERR_FILENO, std::string("\2\n")); sendException = false; - /* Execute the program. This should not return. */ + /* If this is a builtin builder, call it now. This should not return. */ if (drv.isBuiltin()) { try { logger = makeJSONLogger(getStandardError()); @@ -2284,7 +1568,7 @@ void DerivationBuilderImpl::runChild() } } - // Now builder is not builtin + /* It's not a builtin builder, so execute the program. */ Strings args; args.push_back(std::string(baseNameOf(drv.builder))); @@ -2292,31 +1576,11 @@ void DerivationBuilderImpl::runChild() for (auto & i : drv.args) args.push_back(rewriteStrings(i, inputRewrites)); -#ifdef __APPLE__ - posix_spawnattr_t attrp; - - if (posix_spawnattr_init(&attrp)) - throw SysError("failed to initialize builder"); - - if (posix_spawnattr_setflags(&attrp, POSIX_SPAWN_SETEXEC)) - throw SysError("failed to initialize builder"); - - if (drv.platform == "aarch64-darwin") { - // Unset kern.curproc_arch_affinity so we can escape Rosetta - int affinity = 0; - sysctlbyname("kern.curproc_arch_affinity", NULL, NULL, &affinity, sizeof(affinity)); - - cpu_type_t cpu = CPU_TYPE_ARM64; - posix_spawnattr_setbinpref_np(&attrp, 1, &cpu, NULL); - } else if (drv.platform == "x86_64-darwin") { - cpu_type_t cpu = CPU_TYPE_X86_64; - posix_spawnattr_setbinpref_np(&attrp, 1, &cpu, NULL); - } + Strings envStrs; + for (auto & i : env) + envStrs.push_back(rewriteStrings(i.first + "=" + i.second, inputRewrites)); - posix_spawn(NULL, drv.builder.c_str(), NULL, &attrp, stringsToCharPtrs(args).data(), stringsToCharPtrs(envStrs).data()); -#else - execve(drv.builder.c_str(), stringsToCharPtrs(args).data(), stringsToCharPtrs(envStrs).data()); -#endif + execBuilder(args, envStrs); throw SysError("executing '%1%'", drv.builder); @@ -2326,6 +1590,37 @@ void DerivationBuilderImpl::runChild() } } +void DerivationBuilderImpl::setUser() +{ + /* If we are running in `build-users' mode, then switch to the + user we allocated above. Make sure that we drop all root + privileges. Note that above we have closed all file + descriptors except std*, so that's safe. Also note that + setuid() when run as root sets the real, effective and + saved UIDs. */ + if (buildUser) { + /* Preserve supplementary groups of the build user, to allow + admins to specify groups such as "kvm". */ + auto gids = buildUser->getSupplementaryGIDs(); + if (setgroups(gids.size(), gids.data()) == -1) + throw SysError("cannot set supplementary groups of build user"); + + if (setgid(buildUser->getGID()) == -1 || + getgid() != buildUser->getGID() || + getegid() != buildUser->getGID()) + throw SysError("setgid failed"); + + if (setuid(buildUser->getUID()) == -1 || + getuid() != buildUser->getUID() || + geteuid() != buildUser->getUID()) + throw SysError("setuid failed"); + } +} + +void DerivationBuilderImpl::execBuilder(const Strings & args, const Strings & envStrs) +{ + execve(drv.builder.c_str(), stringsToCharPtrs(args).data(), stringsToCharPtrs(envStrs).data()); +} SingleDrvOutputs DerivationBuilderImpl::registerOutputs() { @@ -2758,7 +2053,7 @@ SingleDrvOutputs DerivationBuilderImpl::registerOutputs() } } - auto & localStore = getLocalStore(); + auto & localStore = getLocalStore(store); if (buildMode == bmCheck) { @@ -2835,7 +2130,7 @@ SingleDrvOutputs DerivationBuilderImpl::registerOutputs() paths referenced by each of them. If there are cycles in the outputs, this will fail. */ { - auto & localStore = getLocalStore(); + auto & localStore = getLocalStore(store); ValidPathInfos infos2; for (auto & [outputName, newInfo] : infos) { @@ -3056,5 +2351,70 @@ StorePath DerivationBuilderImpl::makeFallbackPath(const StorePath & path) Hash(HashAlgorithm::SHA256), path.name()); } +// FIXME: do this properly +#include "linux-derivation-builder.cc" +#include "darwin-derivation-builder.cc" + +std::unique_ptr makeDerivationBuilder( + Store & store, + std::unique_ptr miscMethods, + DerivationBuilderParams params) +{ + bool useSandbox = false; + + /* Are we doing a sandboxed build? */ + { + if (settings.sandboxMode == smEnabled) { + if (params.drvOptions.noChroot) + throw Error("derivation '%s' has '__noChroot' set, " + "but that's not allowed when 'sandbox' is 'true'", store.printStorePath(params.drvPath)); +#ifdef __APPLE__ + if (drvOptions.additionalSandboxProfile != "") + throw Error("derivation '%s' specifies a sandbox profile, " + "but this is only allowed when 'sandbox' is 'relaxed'", store.printStorePath(params.drvPath)); +#endif + useSandbox = true; + } + else if (settings.sandboxMode == smDisabled) + useSandbox = false; + else if (settings.sandboxMode == smRelaxed) + // FIXME: cache derivationType + useSandbox = params.drv.type().isSandboxed() && !params.drvOptions.noChroot; + } + + auto & localStore = getLocalStore(store); + if (localStore.storeDir != localStore.config->realStoreDir.get()) { + #ifdef __linux__ + useSandbox = true; + #else + throw Error("building using a diverted store is not supported on this platform"); + #endif + } + + #ifdef __linux__ + if (useSandbox) { + if (!mountAndPidNamespacesSupported()) { + if (!settings.sandboxFallback) + throw Error("this system does not support the kernel namespaces that are required for sandboxing; use '--no-sandbox' to disable sandboxing"); + debug("auto-disabling sandboxing because the prerequisite namespaces are not available"); + useSandbox = false; + } + } + + if (useSandbox) + return std::make_unique( + store, + std::move(miscMethods), + std::move(params)); + #endif + + if (useSandbox) + throw Error("sandboxing builds is not supported on this platform"); + + return std::make_unique( + store, + std::move(miscMethods), + std::move(params)); +} } diff --git a/src/libstore/unix/build/linux-derivation-builder.cc b/src/libstore/unix/build/linux-derivation-builder.cc new file mode 100644 index 00000000000..0700c582df9 --- /dev/null +++ b/src/libstore/unix/build/linux-derivation-builder.cc @@ -0,0 +1,568 @@ +#ifdef __linux__ + +struct LinuxDerivationBuilder : DerivationBuilderImpl +{ + /** + * Pipe for synchronising updates to the builder namespaces. + */ + Pipe userNamespaceSync; + + /** + * The mount namespace and user namespace of the builder, used to add additional + * paths to the sandbox as a result of recursive Nix calls. + */ + AutoCloseFD sandboxMountNamespace; + AutoCloseFD sandboxUserNamespace; + + /** + * On Linux, whether we're doing the build in its own user + * namespace. + */ + bool usingUserNamespace = true; + + LinuxDerivationBuilder( + Store & store, std::unique_ptr miscMethods, DerivationBuilderParams params) + : DerivationBuilderImpl(store, std::move(miscMethods), std::move(params)) + { + useChroot = true; + } + + uid_t sandboxUid() { return usingUserNamespace ? (!buildUser || buildUser->getUIDCount() == 1 ? 1000 : 0) : buildUser->getUID(); } + gid_t sandboxGid() { return usingUserNamespace ? (!buildUser || buildUser->getUIDCount() == 1 ? 100 : 0) : buildUser->getGID(); } + + bool needsHashRewrite() override + { + return false; + } + + void prepareSandbox() override + { + /* Create a temporary directory in which we set up the chroot + environment using bind-mounts. We put it in the Nix store + so that the build outputs can be moved efficiently from the + chroot to their final location. */ + auto chrootParentDir = store.Store::toRealPath(drvPath) + ".chroot"; + deletePath(chrootParentDir); + + /* Clean up the chroot directory automatically. */ + autoDelChroot = std::make_shared(chrootParentDir); + + printMsg(lvlChatty, "setting up chroot environment in '%1%'", chrootParentDir); + + if (mkdir(chrootParentDir.c_str(), 0700) == -1) + throw SysError("cannot create '%s'", chrootRootDir); + + chrootRootDir = chrootParentDir + "/root"; + + if (mkdir(chrootRootDir.c_str(), buildUser && buildUser->getUIDCount() != 1 ? 0755 : 0750) == -1) + throw SysError("cannot create '%1%'", chrootRootDir); + + if (buildUser && chown(chrootRootDir.c_str(), buildUser->getUIDCount() != 1 ? buildUser->getUID() : 0, buildUser->getGID()) == -1) + throw SysError("cannot change ownership of '%1%'", chrootRootDir); + + /* Create a writable /tmp in the chroot. Many builders need + this. (Of course they should really respect $TMPDIR + instead.) */ + Path chrootTmpDir = chrootRootDir + "/tmp"; + createDirs(chrootTmpDir); + chmod_(chrootTmpDir, 01777); + + /* Create a /etc/passwd with entries for the build user and the + nobody account. The latter is kind of a hack to support + Samba-in-QEMU. */ + createDirs(chrootRootDir + "/etc"); + if (drvOptions.useUidRange(drv)) + chownToBuilder(chrootRootDir + "/etc"); + + if (drvOptions.useUidRange(drv) && (!buildUser || buildUser->getUIDCount() < 65536)) + throw Error("feature 'uid-range' requires the setting '%s' to be enabled", settings.autoAllocateUids.name); + + /* Declare the build user's group so that programs get a consistent + view of the system (e.g., "id -gn"). */ + writeFile(chrootRootDir + "/etc/group", + fmt("root:x:0:\n" + "nixbld:!:%1%:\n" + "nogroup:x:65534:\n", sandboxGid())); + + /* Create /etc/hosts with localhost entry. */ + if (derivationType.isSandboxed()) + writeFile(chrootRootDir + "/etc/hosts", "127.0.0.1 localhost\n::1 localhost\n"); + + /* Make the closure of the inputs available in the chroot, + rather than the whole Nix store. This prevents any access + to undeclared dependencies. Directories are bind-mounted, + while other inputs are hard-linked (since only directories + can be bind-mounted). !!! As an extra security + precaution, make the fake Nix store only writable by the + build user. */ + Path chrootStoreDir = chrootRootDir + store.storeDir; + createDirs(chrootStoreDir); + chmod_(chrootStoreDir, 01775); + + if (buildUser && chown(chrootStoreDir.c_str(), 0, buildUser->getGID()) == -1) + throw SysError("cannot change ownership of '%1%'", chrootStoreDir); + + for (auto & i : inputPaths) { + auto p = store.printStorePath(i); + Path r = store.toRealPath(p); + pathsInChroot.insert_or_assign(p, r); + } + + /* If we're repairing, checking or rebuilding part of a + multiple-outputs derivation, it's possible that we're + rebuilding a path that is in settings.sandbox-paths + (typically the dependencies of /bin/sh). Throw them + out. */ + for (auto & i : drv.outputsAndOptPaths(store)) { + /* If the name isn't known a priori (i.e. floating + content-addressing derivation), the temporary location we use + should be fresh. Freshness means it is impossible that the path + is already in the sandbox, so we don't need to worry about + removing it. */ + if (i.second.second) + pathsInChroot.erase(store.printStorePath(*i.second.second)); + } + + if (cgroup) { + if (mkdir(cgroup->c_str(), 0755) != 0) + throw SysError("creating cgroup '%s'", *cgroup); + chownToBuilder(*cgroup); + chownToBuilder(*cgroup + "/cgroup.procs"); + chownToBuilder(*cgroup + "/cgroup.threads"); + //chownToBuilder(*cgroup + "/cgroup.subtree_control"); + } + } + + void startChild() override + { + /* Set up private namespaces for the build: + + - The PID namespace causes the build to start as PID 1. + Processes outside of the chroot are not visible to those + on the inside, but processes inside the chroot are + visible from the outside (though with different PIDs). + + - The private mount namespace ensures that all the bind + mounts we do will only show up in this process and its + children, and will disappear automatically when we're + done. + + - The private network namespace ensures that the builder + cannot talk to the outside world (or vice versa). It + only has a private loopback interface. (Fixed-output + derivations are not run in a private network namespace + to allow functions like fetchurl to work.) + + - The IPC namespace prevents the builder from communicating + with outside processes using SysV IPC mechanisms (shared + memory, message queues, semaphores). It also ensures + that all IPC objects are destroyed when the builder + exits. + + - The UTS namespace ensures that builders see a hostname of + localhost rather than the actual hostname. + + We use a helper process to do the clone() to work around + clone() being broken in multi-threaded programs due to + at-fork handlers not being run. Note that we use + CLONE_PARENT to ensure that the real builder is parented to + us. + */ + + userNamespaceSync.create(); + + usingUserNamespace = userNamespacesSupported(); + + Pipe sendPid; + sendPid.create(); + + Pid helper = startProcess([&]() { + sendPid.readSide.close(); + + /* We need to open the slave early, before + CLONE_NEWUSER. Otherwise we get EPERM when running as + root. */ + openSlave(); + + try { + /* Drop additional groups here because we can't do it + after we've created the new user namespace. */ + if (setgroups(0, 0) == -1) { + if (errno != EPERM) + throw SysError("setgroups failed"); + if (settings.requireDropSupplementaryGroups) + throw Error("setgroups failed. Set the require-drop-supplementary-groups option to false to skip this step."); + } + + ProcessOptions options; + options.cloneFlags = CLONE_NEWPID | CLONE_NEWNS | CLONE_NEWIPC | CLONE_NEWUTS | CLONE_PARENT | SIGCHLD; + if (derivationType.isSandboxed()) + options.cloneFlags |= CLONE_NEWNET; + if (usingUserNamespace) + options.cloneFlags |= CLONE_NEWUSER; + + pid_t child = startProcess([&]() { runChild(); }, options); + + writeFull(sendPid.writeSide.get(), fmt("%d\n", child)); + _exit(0); + } catch (...) { + handleChildException(true); + _exit(1); + } + }); + + sendPid.writeSide.close(); + + if (helper.wait() != 0) { + processSandboxSetupMessages(); + // Only reached if the child process didn't send an exception. + throw Error("unable to start build process"); + } + + userNamespaceSync.readSide = -1; + + /* Close the write side to prevent runChild() from hanging + reading from this. */ + Finally cleanup([&]() { + userNamespaceSync.writeSide = -1; + }); + + auto ss = tokenizeString>(readLine(sendPid.readSide.get())); + assert(ss.size() == 1); + pid = string2Int(ss[0]).value(); + + if (usingUserNamespace) { + /* Set the UID/GID mapping of the builder's user namespace + such that the sandbox user maps to the build user, or to + the calling user (if build users are disabled). */ + uid_t hostUid = buildUser ? buildUser->getUID() : getuid(); + uid_t hostGid = buildUser ? buildUser->getGID() : getgid(); + uid_t nrIds = buildUser ? buildUser->getUIDCount() : 1; + + writeFile("/proc/" + std::to_string(pid) + "/uid_map", + fmt("%d %d %d", sandboxUid(), hostUid, nrIds)); + + if (!buildUser || buildUser->getUIDCount() == 1) + writeFile("/proc/" + std::to_string(pid) + "/setgroups", "deny"); + + writeFile("/proc/" + std::to_string(pid) + "/gid_map", + fmt("%d %d %d", sandboxGid(), hostGid, nrIds)); + } else { + debug("note: not using a user namespace"); + if (!buildUser) + throw Error("cannot perform a sandboxed build because user namespaces are not enabled; check /proc/sys/user/max_user_namespaces"); + } + + /* Now that we now the sandbox uid, we can write + /etc/passwd. */ + writeFile(chrootRootDir + "/etc/passwd", fmt( + "root:x:0:0:Nix build user:%3%:/noshell\n" + "nixbld:x:%1%:%2%:Nix build user:%3%:/noshell\n" + "nobody:x:65534:65534:Nobody:/:/noshell\n", + sandboxUid(), sandboxGid(), settings.sandboxBuildDir)); + + /* Save the mount- and user namespace of the child. We have to do this + *before* the child does a chroot. */ + sandboxMountNamespace = open(fmt("/proc/%d/ns/mnt", (pid_t) pid).c_str(), O_RDONLY); + if (sandboxMountNamespace.get() == -1) + throw SysError("getting sandbox mount namespace"); + + if (usingUserNamespace) { + sandboxUserNamespace = open(fmt("/proc/%d/ns/user", (pid_t) pid).c_str(), O_RDONLY); + if (sandboxUserNamespace.get() == -1) + throw SysError("getting sandbox user namespace"); + } + + /* Move the child into its own cgroup. */ + if (cgroup) + writeFile(*cgroup + "/cgroup.procs", fmt("%d", (pid_t) pid)); + + /* Signal the builder that we've updated its user namespace. */ + writeFull(userNamespaceSync.writeSide.get(), "1"); + } + + void enterChroot() override + { + userNamespaceSync.writeSide = -1; + + if (drainFD(userNamespaceSync.readSide.get()) != "1") + throw Error("user namespace initialisation failed"); + + userNamespaceSync.readSide = -1; + + if (derivationType.isSandboxed()) { + + /* Initialise the loopback interface. */ + AutoCloseFD fd(socket(PF_INET, SOCK_DGRAM, IPPROTO_IP)); + if (!fd) + throw SysError("cannot open IP socket"); + + struct ifreq ifr; + strcpy(ifr.ifr_name, "lo"); + ifr.ifr_flags = IFF_UP | IFF_LOOPBACK | IFF_RUNNING; + if (ioctl(fd.get(), SIOCSIFFLAGS, &ifr) == -1) + throw SysError("cannot set loopback interface flags"); + } + + /* Set the hostname etc. to fixed values. */ + char hostname[] = "localhost"; + if (sethostname(hostname, sizeof(hostname)) == -1) + throw SysError("cannot set host name"); + char domainname[] = "(none)"; // kernel default + if (setdomainname(domainname, sizeof(domainname)) == -1) + throw SysError("cannot set domain name"); + + /* Make all filesystems private. This is necessary + because subtrees may have been mounted as "shared" + (MS_SHARED). (Systemd does this, for instance.) Even + though we have a private mount namespace, mounting + filesystems on top of a shared subtree still propagates + outside of the namespace. Making a subtree private is + local to the namespace, though, so setting MS_PRIVATE + does not affect the outside world. */ + if (mount(0, "/", 0, MS_PRIVATE | MS_REC, 0) == -1) + throw SysError("unable to make '/' private"); + + /* Bind-mount chroot directory to itself, to treat it as a + different filesystem from /, as needed for pivot_root. */ + if (mount(chrootRootDir.c_str(), chrootRootDir.c_str(), 0, MS_BIND, 0) == -1) + throw SysError("unable to bind mount '%1%'", chrootRootDir); + + /* Bind-mount the sandbox's Nix store onto itself so that + we can mark it as a "shared" subtree, allowing bind + mounts made in *this* mount namespace to be propagated + into the child namespace created by the + unshare(CLONE_NEWNS) call below. + + Marking chrootRootDir as MS_SHARED causes pivot_root() + to fail with EINVAL. Don't know why. */ + Path chrootStoreDir = chrootRootDir + store.storeDir; + + if (mount(chrootStoreDir.c_str(), chrootStoreDir.c_str(), 0, MS_BIND, 0) == -1) + throw SysError("unable to bind mount the Nix store", chrootStoreDir); + + if (mount(0, chrootStoreDir.c_str(), 0, MS_SHARED, 0) == -1) + throw SysError("unable to make '%s' shared", chrootStoreDir); + + /* Set up a nearly empty /dev, unless the user asked to + bind-mount the host /dev. */ + Strings ss; + if (pathsInChroot.find("/dev") == pathsInChroot.end()) { + createDirs(chrootRootDir + "/dev/shm"); + createDirs(chrootRootDir + "/dev/pts"); + ss.push_back("/dev/full"); + if (store.config.systemFeatures.get().count("kvm") && pathExists("/dev/kvm")) + ss.push_back("/dev/kvm"); + ss.push_back("/dev/null"); + ss.push_back("/dev/random"); + ss.push_back("/dev/tty"); + ss.push_back("/dev/urandom"); + ss.push_back("/dev/zero"); + createSymlink("/proc/self/fd", chrootRootDir + "/dev/fd"); + createSymlink("/proc/self/fd/0", chrootRootDir + "/dev/stdin"); + createSymlink("/proc/self/fd/1", chrootRootDir + "/dev/stdout"); + createSymlink("/proc/self/fd/2", chrootRootDir + "/dev/stderr"); + } + + /* Fixed-output derivations typically need to access the + network, so give them access to /etc/resolv.conf and so + on. */ + if (!derivationType.isSandboxed()) { + // Only use nss functions to resolve hosts and + // services. Don’t use it for anything else that may + // be configured for this system. This limits the + // potential impurities introduced in fixed-outputs. + writeFile(chrootRootDir + "/etc/nsswitch.conf", "hosts: files dns\nservices: files\n"); + + /* N.B. it is realistic that these paths might not exist. It + happens when testing Nix building fixed-output derivations + within a pure derivation. */ + for (auto & path : {"/etc/resolv.conf", "/etc/services", "/etc/hosts"}) + if (pathExists(path)) + ss.push_back(path); + + if (settings.caFile != "") { + Path caFile = settings.caFile; + if (pathExists(caFile)) + pathsInChroot.try_emplace("/etc/ssl/certs/ca-certificates.crt", canonPath(caFile, true), true); + } + } + + for (auto & i : ss) { + // For backwards-compatibility, resolve all the symlinks in the + // chroot paths. + auto canonicalPath = canonPath(i, true); + pathsInChroot.emplace(i, canonicalPath); + } + + /* Bind-mount all the directories from the "host" + filesystem that we want in the chroot + environment. */ + for (auto & i : pathsInChroot) { + if (i.second.source == "/proc") + continue; // backwards compatibility + +# if HAVE_EMBEDDED_SANDBOX_SHELL + if (i.second.source == "__embedded_sandbox_shell__") { + static unsigned char sh[] = { +# include "embedded-sandbox-shell.gen.hh" + }; + auto dst = chrootRootDir + i.first; + createDirs(dirOf(dst)); + writeFile(dst, std::string_view((const char *) sh, sizeof(sh))); + chmod_(dst, 0555); + } else +# endif + doBind(i.second.source, chrootRootDir + i.first, i.second.optional); + } + + /* Bind a new instance of procfs on /proc. */ + createDirs(chrootRootDir + "/proc"); + if (mount("none", (chrootRootDir + "/proc").c_str(), "proc", 0, 0) == -1) + throw SysError("mounting /proc"); + + /* Mount sysfs on /sys. */ + if (buildUser && buildUser->getUIDCount() != 1) { + createDirs(chrootRootDir + "/sys"); + if (mount("none", (chrootRootDir + "/sys").c_str(), "sysfs", 0, 0) == -1) + throw SysError("mounting /sys"); + } + + /* Mount a new tmpfs on /dev/shm to ensure that whatever + the builder puts in /dev/shm is cleaned up automatically. */ + if (pathExists("/dev/shm") + && mount( + "none", + (chrootRootDir + "/dev/shm").c_str(), + "tmpfs", + 0, + fmt("size=%s", settings.sandboxShmSize).c_str()) + == -1) + throw SysError("mounting /dev/shm"); + + /* Mount a new devpts on /dev/pts. Note that this + requires the kernel to be compiled with + CONFIG_DEVPTS_MULTIPLE_INSTANCES=y (which is the case + if /dev/ptx/ptmx exists). */ + if (pathExists("/dev/pts/ptmx") && !pathExists(chrootRootDir + "/dev/ptmx") + && !pathsInChroot.count("/dev/pts")) { + if (mount("none", (chrootRootDir + "/dev/pts").c_str(), "devpts", 0, "newinstance,mode=0620") == 0) { + createSymlink("/dev/pts/ptmx", chrootRootDir + "/dev/ptmx"); + + /* Make sure /dev/pts/ptmx is world-writable. With some + Linux versions, it is created with permissions 0. */ + chmod_(chrootRootDir + "/dev/pts/ptmx", 0666); + } else { + if (errno != EINVAL) + throw SysError("mounting /dev/pts"); + doBind("/dev/pts", chrootRootDir + "/dev/pts"); + doBind("/dev/ptmx", chrootRootDir + "/dev/ptmx"); + } + } + + /* Make /etc unwritable */ + if (!drvOptions.useUidRange(drv)) + chmod_(chrootRootDir + "/etc", 0555); + + /* Unshare this mount namespace. This is necessary because + pivot_root() below changes the root of the mount + namespace. This means that the call to setns() in + addDependency() would hide the host's filesystem, + making it impossible to bind-mount paths from the host + Nix store into the sandbox. Therefore, we save the + pre-pivot_root namespace in + sandboxMountNamespace. Since we made /nix/store a + shared subtree above, this allows addDependency() to + make paths appear in the sandbox. */ + if (unshare(CLONE_NEWNS) == -1) + throw SysError("unsharing mount namespace"); + + /* Unshare the cgroup namespace. This means + /proc/self/cgroup will show the child's cgroup as '/' + rather than whatever it is in the parent. */ + if (cgroup && unshare(CLONE_NEWCGROUP) == -1) + throw SysError("unsharing cgroup namespace"); + + /* Do the chroot(). */ + if (chdir(chrootRootDir.c_str()) == -1) + throw SysError("cannot change directory to '%1%'", chrootRootDir); + + if (mkdir("real-root", 0500) == -1) + throw SysError("cannot create real-root directory"); + + if (pivot_root(".", "real-root") == -1) + throw SysError("cannot pivot old root directory onto '%1%'", (chrootRootDir + "/real-root")); + + if (chroot(".") == -1) + throw SysError("cannot change root directory to '%1%'", chrootRootDir); + + if (umount2("real-root", MNT_DETACH) == -1) + throw SysError("cannot unmount real root filesystem"); + + if (rmdir("real-root") == -1) + throw SysError("cannot remove real-root directory"); + + // FIXME: move to LinuxDerivationBuilder + linux::setPersonality(drv.platform); + } + + void setUser() override + { + /* Switch to the sandbox uid/gid in the user namespace, + which corresponds to the build user or calling user in + the parent namespace. */ + if (setgid(sandboxGid()) == -1) + throw SysError("setgid failed"); + if (setuid(sandboxUid()) == -1) + throw SysError("setuid failed"); + } + + std::variant, SingleDrvOutputs> unprepareBuild() override + { + sandboxMountNamespace = -1; + sandboxUserNamespace = -1; + + return DerivationBuilderImpl::unprepareBuild(); + } + + void addDependency(const StorePath & path) override + { + if (isAllowed(path)) + return; + + addedPaths.insert(path); + + debug("materialising '%s' in the sandbox", store.printStorePath(path)); + + Path source = store.Store::toRealPath(path); + Path target = chrootRootDir + store.printStorePath(path); + + if (pathExists(target)) { + // There is a similar debug message in doBind, so only run it in this block to not have double messages. + debug("bind-mounting %s -> %s", target, source); + throw Error("store path '%s' already exists in the sandbox", store.printStorePath(path)); + } + + /* Bind-mount the path into the sandbox. This requires + entering its mount namespace, which is not possible + in multithreaded programs. So we do this in a + child process.*/ + Pid child(startProcess([&]() { + if (usingUserNamespace && (setns(sandboxUserNamespace.get(), 0) == -1)) + throw SysError("entering sandbox user namespace"); + + if (setns(sandboxMountNamespace.get(), 0) == -1) + throw SysError("entering sandbox mount namespace"); + + doBind(source, target); + + _exit(0); + })); + + int status = child.wait(); + if (status != 0) + throw Error("could not add path '%s' to sandbox", store.printStorePath(path)); + } +}; + +#endif From 67408807d895cf77b2ec5ed2b4f7019be7470529 Mon Sep 17 00:00:00 2001 From: Eelco Dolstra Date: Mon, 26 May 2025 20:30:36 +0200 Subject: [PATCH 080/218] Move pathsInChroot --- .../unix/build/darwin-derivation-builder.cc | 9 +- src/libstore/unix/build/derivation-builder.cc | 233 +++++++++--------- .../unix/build/linux-derivation-builder.cc | 15 +- 3 files changed, 140 insertions(+), 117 deletions(-) diff --git a/src/libstore/unix/build/darwin-derivation-builder.cc b/src/libstore/unix/build/darwin-derivation-builder.cc index 3366403a76f..cc23643902a 100644 --- a/src/libstore/unix/build/darwin-derivation-builder.cc +++ b/src/libstore/unix/build/darwin-derivation-builder.cc @@ -2,6 +2,8 @@ struct DarwinDerivationBuilder : DerivationBuilderImpl { + PathsInChroot pathsInChroot; + DarwinDerivationBuilder( Store & store, std::unique_ptr miscMethods, DerivationBuilderParams params) : DerivationBuilderImpl(store, std::move(miscMethods), std::move(params)) @@ -9,6 +11,11 @@ struct DarwinDerivationBuilder : DerivationBuilderImpl useChroot = true; } + void prepareSandbox() override + { + pathsInChroot = getPathsInSandbox(); + } + void execBuilder(const Strings & args, const Strings & envStrs) override { posix_spawnattr_t attrp; @@ -69,7 +76,7 @@ struct DarwinDerivationBuilder : DerivationBuilderImpl /* Add all our input paths to the chroot */ for (auto & i : inputPaths) { auto p = store.printStorePath(i); - pathsInChroot[p] = p; + pathsInChroot.insert_or_assign(p, p); } /* Violations will go to the syslog if you set this. Unfortunately the destination does not appear to be diff --git a/src/libstore/unix/build/derivation-builder.cc b/src/libstore/unix/build/derivation-builder.cc index c082c02343d..32d6f2380a2 100644 --- a/src/libstore/unix/build/derivation-builder.cc +++ b/src/libstore/unix/build/derivation-builder.cc @@ -149,6 +149,7 @@ class DerivationBuilderImpl : public DerivationBuilder, public DerivationBuilder /** * RAII object to delete the chroot directory. */ + // FIXME: move std::shared_ptr autoDelChroot; /** @@ -169,7 +170,6 @@ class DerivationBuilderImpl : public DerivationBuilder, public DerivationBuilder { } }; typedef std::map PathsInChroot; // maps target path to source path - PathsInChroot pathsInChroot; typedef StringMap Environment; Environment env; @@ -250,6 +250,17 @@ class DerivationBuilderImpl : public DerivationBuilder, public DerivationBuilder protected: + /** + * Return the paths that should be made available in the sandbox. + * This includes: + * + * * The paths specified by the `sandbox-paths` setting, and their closure in the Nix store. + * * The contents of the `__impureHostDeps` derivation attribute, if the sandbox is in relaxed mode. + * * The paths returned by the `pre-build-hook`. + * * The paths in the input closure of the derivation. + */ + PathsInChroot getPathsInSandbox(); + /** * Called by prepareBuild() to do any setup in the parent to * prepare for a sandboxed build. @@ -916,120 +927,11 @@ void DerivationBuilderImpl::startBuilder() } } - if (useChroot) { - - /* Allow a user-configurable set of directories from the - host file system. */ - pathsInChroot.clear(); - - for (auto i : settings.sandboxPaths.get()) { - if (i.empty()) continue; - bool optional = false; - if (i[i.size() - 1] == '?') { - optional = true; - i.pop_back(); - } - size_t p = i.find('='); - if (p == std::string::npos) - pathsInChroot[i] = {i, optional}; - else - pathsInChroot[i.substr(0, p)] = {i.substr(p + 1), optional}; - } - if (hasPrefix(store.storeDir, tmpDirInSandbox)) - { - throw Error("`sandbox-build-dir` must not contain the storeDir"); - } - pathsInChroot[tmpDirInSandbox] = tmpDir; - - /* Add the closure of store paths to the chroot. */ - StorePathSet closure; - for (auto & i : pathsInChroot) - try { - if (store.isInStore(i.second.source)) - store.computeFSClosure(store.toStorePath(i.second.source).first, closure); - } catch (InvalidPath & e) { - } catch (Error & e) { - e.addTrace({}, "while processing 'sandbox-paths'"); - throw; - } - for (auto & i : closure) { - auto p = store.printStorePath(i); - pathsInChroot.insert_or_assign(p, p); - } - - PathSet allowedPaths = settings.allowedImpureHostPrefixes; - - /* This works like the above, except on a per-derivation level */ - auto impurePaths = drvOptions.impureHostDeps; - - for (auto & i : impurePaths) { - bool found = false; - /* Note: we're not resolving symlinks here to prevent - giving a non-root user info about inaccessible - files. */ - Path canonI = canonPath(i); - /* If only we had a trie to do this more efficiently :) luckily, these are generally going to be pretty small */ - for (auto & a : allowedPaths) { - Path canonA = canonPath(a); - if (isDirOrInDir(canonI, canonA)) { - found = true; - break; - } - } - if (!found) - throw Error("derivation '%s' requested impure path '%s', but it was not in allowed-impure-host-deps", - store.printStorePath(drvPath), i); - - /* Allow files in drvOptions.impureHostDeps to be missing; e.g. - macOS 11+ has no /usr/lib/libSystem*.dylib */ - pathsInChroot[i] = {i, true}; - } - } else { - if (drvOptions.useUidRange(drv)) - throw Error("feature 'uid-range' is only supported in sandboxed builds"); - } - prepareSandbox(); if (needsHashRewrite() && pathExists(homeDir)) throw Error("home directory '%1%' exists; please remove it to assure purity of builds without sandboxing", homeDir); - if (useChroot && settings.preBuildHook != "") { - printMsg(lvlChatty, "executing pre-build hook '%1%'", settings.preBuildHook); - auto args = useChroot ? Strings({store.printStorePath(drvPath), chrootRootDir}) : - Strings({ store.printStorePath(drvPath) }); - enum BuildHookState { - stBegin, - stExtraChrootDirs - }; - auto state = stBegin; - auto lines = runProgram(settings.preBuildHook, false, args); - auto lastPos = std::string::size_type{0}; - for (auto nlPos = lines.find('\n'); nlPos != std::string::npos; - nlPos = lines.find('\n', lastPos)) - { - auto line = lines.substr(lastPos, nlPos - lastPos); - lastPos = nlPos + 1; - if (state == stBegin) { - if (line == "extra-sandbox-paths" || line == "extra-chroot-dirs") { - state = stExtraChrootDirs; - } else { - throw Error("unknown pre-build hook command '%1%'", line); - } - } else if (state == stExtraChrootDirs) { - if (line == "") { - state = stBegin; - } else { - auto p = line.find('='); - if (p == std::string::npos) - pathsInChroot[line] = line; - else - pathsInChroot[line.substr(0, p)] = line.substr(p + 1); - } - } - } - } - /* Fire up a Nix daemon to process recursive Nix calls from the builder. */ if (drvOptions.getRequiredSystemFeatures(drv).count("recursive-nix")) @@ -1080,6 +982,114 @@ void DerivationBuilderImpl::startBuilder() processSandboxSetupMessages(); } +DerivationBuilderImpl::PathsInChroot DerivationBuilderImpl::getPathsInSandbox() +{ + PathsInChroot pathsInChroot; + + /* Allow a user-configurable set of directories from the + host file system. */ + for (auto i : settings.sandboxPaths.get()) { + if (i.empty()) continue; + bool optional = false; + if (i[i.size() - 1] == '?') { + optional = true; + i.pop_back(); + } + size_t p = i.find('='); + if (p == std::string::npos) + pathsInChroot[i] = {i, optional}; + else + pathsInChroot[i.substr(0, p)] = {i.substr(p + 1), optional}; + } + if (hasPrefix(store.storeDir, tmpDirInSandbox)) + { + throw Error("`sandbox-build-dir` must not contain the storeDir"); + } + pathsInChroot[tmpDirInSandbox] = tmpDir; + + /* Add the closure of store paths to the chroot. */ + StorePathSet closure; + for (auto & i : pathsInChroot) + try { + if (store.isInStore(i.second.source)) + store.computeFSClosure(store.toStorePath(i.second.source).first, closure); + } catch (InvalidPath & e) { + } catch (Error & e) { + e.addTrace({}, "while processing 'sandbox-paths'"); + throw; + } + for (auto & i : closure) { + auto p = store.printStorePath(i); + pathsInChroot.insert_or_assign(p, p); + } + + PathSet allowedPaths = settings.allowedImpureHostPrefixes; + + /* This works like the above, except on a per-derivation level */ + auto impurePaths = drvOptions.impureHostDeps; + + for (auto & i : impurePaths) { + bool found = false; + /* Note: we're not resolving symlinks here to prevent + giving a non-root user info about inaccessible + files. */ + Path canonI = canonPath(i); + /* If only we had a trie to do this more efficiently :) luckily, these are generally going to be pretty small */ + for (auto & a : allowedPaths) { + Path canonA = canonPath(a); + if (isDirOrInDir(canonI, canonA)) { + found = true; + break; + } + } + if (!found) + throw Error("derivation '%s' requested impure path '%s', but it was not in allowed-impure-host-deps", + store.printStorePath(drvPath), i); + + /* Allow files in drvOptions.impureHostDeps to be missing; e.g. + macOS 11+ has no /usr/lib/libSystem*.dylib */ + pathsInChroot[i] = {i, true}; + } + + if (settings.preBuildHook != "") { + printMsg(lvlChatty, "executing pre-build hook '%1%'", settings.preBuildHook); + auto args = useChroot ? Strings({store.printStorePath(drvPath), chrootRootDir}) : + Strings({ store.printStorePath(drvPath) }); + enum BuildHookState { + stBegin, + stExtraChrootDirs + }; + auto state = stBegin; + auto lines = runProgram(settings.preBuildHook, false, args); + auto lastPos = std::string::size_type{0}; + for (auto nlPos = lines.find('\n'); nlPos != std::string::npos; + nlPos = lines.find('\n', lastPos)) + { + auto line = lines.substr(lastPos, nlPos - lastPos); + lastPos = nlPos + 1; + if (state == stBegin) { + if (line == "extra-sandbox-paths" || line == "extra-chroot-dirs") { + state = stExtraChrootDirs; + } else { + throw Error("unknown pre-build hook command '%1%'", line); + } + } else if (state == stExtraChrootDirs) { + if (line == "") { + state = stBegin; + } else { + auto p = line.find('='); + if (p == std::string::npos) + pathsInChroot[line] = line; + else + pathsInChroot[line.substr(0, p)] = line.substr(p + 1); + } + } + } + } + + return pathsInChroot; +} + void DerivationBuilderImpl::prepareSandbox() { if (drvOptions.useUidRange(drv)) @@ -2411,6 +2421,9 @@ std::unique_ptr makeDerivationBuilder( if (useSandbox) throw Error("sandboxing builds is not supported on this platform"); + if (params.drvOptions.useUidRange(params.drv)) + throw Error("feature 'uid-range' is only supported in sandboxed builds"); + return std::make_unique( store, std::move(miscMethods), diff --git a/src/libstore/unix/build/linux-derivation-builder.cc b/src/libstore/unix/build/linux-derivation-builder.cc index 0700c582df9..d26a781b5a3 100644 --- a/src/libstore/unix/build/linux-derivation-builder.cc +++ b/src/libstore/unix/build/linux-derivation-builder.cc @@ -20,6 +20,8 @@ struct LinuxDerivationBuilder : DerivationBuilderImpl */ bool usingUserNamespace = true; + PathsInChroot pathsInChroot; + LinuxDerivationBuilder( Store & store, std::unique_ptr miscMethods, DerivationBuilderParams params) : DerivationBuilderImpl(store, std::move(miscMethods), std::move(params)) @@ -102,12 +104,6 @@ struct LinuxDerivationBuilder : DerivationBuilderImpl if (buildUser && chown(chrootStoreDir.c_str(), 0, buildUser->getGID()) == -1) throw SysError("cannot change ownership of '%1%'", chrootStoreDir); - for (auto & i : inputPaths) { - auto p = store.printStorePath(i); - Path r = store.toRealPath(p); - pathsInChroot.insert_or_assign(p, r); - } - /* If we're repairing, checking or rebuilding part of a multiple-outputs derivation, it's possible that we're rebuilding a path that is in settings.sandbox-paths @@ -131,6 +127,13 @@ struct LinuxDerivationBuilder : DerivationBuilderImpl chownToBuilder(*cgroup + "/cgroup.threads"); //chownToBuilder(*cgroup + "/cgroup.subtree_control"); } + + pathsInChroot = getPathsInSandbox(); + + for (auto & i : inputPaths) { + auto p = store.printStorePath(i); + pathsInChroot.insert_or_assign(p, store.toRealPath(p)); + } } void startChild() override From 9f8f3968e3623a97480c973de98728084282ae3f Mon Sep 17 00:00:00 2001 From: Eelco Dolstra Date: Mon, 26 May 2025 21:25:56 +0200 Subject: [PATCH 081/218] Eliminate useChroot --- .../unix/build/darwin-derivation-builder.cc | 12 ++ src/libstore/unix/build/derivation-builder.cc | 131 +++++++----------- .../unix/build/linux-derivation-builder.cc | 113 ++++++++++++--- 3 files changed, 155 insertions(+), 101 deletions(-) diff --git a/src/libstore/unix/build/darwin-derivation-builder.cc b/src/libstore/unix/build/darwin-derivation-builder.cc index cc23643902a..2ba54ad97d1 100644 --- a/src/libstore/unix/build/darwin-derivation-builder.cc +++ b/src/libstore/unix/build/darwin-derivation-builder.cc @@ -1,5 +1,15 @@ #ifdef __APPLE__ +# include +# include +# include + +/* This definition is undocumented but depended upon by all major browsers. */ +extern "C" int +sandbox_init_with_parameters(const char * profile, uint64_t flags, const char * const parameters[], char ** errorbuf); + +namespace nix { + struct DarwinDerivationBuilder : DerivationBuilderImpl { PathsInChroot pathsInChroot; @@ -185,4 +195,6 @@ struct DarwinDerivationBuilder : DerivationBuilderImpl } } +} + #endif diff --git a/src/libstore/unix/build/derivation-builder.cc b/src/libstore/unix/build/derivation-builder.cc index 32d6f2380a2..497e0bf31db 100644 --- a/src/libstore/unix/build/derivation-builder.cc +++ b/src/libstore/unix/build/derivation-builder.cc @@ -52,15 +52,6 @@ # include "nix/store/personality.hh" #endif -#ifdef __APPLE__ -# include -# include -# include - -/* This definition is undocumented but depended upon by all major browsers. */ -extern "C" int sandbox_init_with_parameters(const char *profile, uint64_t flags, const char *const parameters[], char **errorbuf); -#endif - #include #include #include @@ -116,6 +107,7 @@ class DerivationBuilderImpl : public DerivationBuilder, public DerivationBuilder /** * The cgroup of the builder, if any. */ + // FIXME: move std::optional cgroup; /** @@ -134,18 +126,6 @@ class DerivationBuilderImpl : public DerivationBuilder, public DerivationBuilder */ Path tmpDirInSandbox; - /** - * Whether we're currently doing a chroot build. - */ - // FIXME: remove - bool useChroot = false; - - /** - * The root of the chroot environment. - */ - // FIXME: move - Path chrootRootDir; - /** * RAII object to delete the chroot directory. */ @@ -250,6 +230,14 @@ class DerivationBuilderImpl : public DerivationBuilder, public DerivationBuilder protected: + /** + * Acquire a build user lock. Return nullptr if no lock is available. + */ + virtual std::unique_ptr getBuildUser() + { + return acquireUserLock(1, false); + } + /** * Return the paths that should be made available in the sandbox. * This includes: @@ -261,12 +249,28 @@ class DerivationBuilderImpl : public DerivationBuilder, public DerivationBuilder */ PathsInChroot getPathsInSandbox(); + virtual void setBuildTmpDir() + { + tmpDir = topTmpDir; + tmpDirInSandbox = topTmpDir; + } + /** * Called by prepareBuild() to do any setup in the parent to * prepare for a sandboxed build. */ virtual void prepareSandbox(); + virtual Strings getPreBuildHookArgs() + { + return Strings({store.printStorePath(drvPath)}); + } + + virtual Path realPathInSandbox(const Path & p) + { + return store.toRealPath(p); + } + /** * Open the slave side of the pseudoterminal and use it as stderr. */ @@ -370,9 +374,13 @@ class DerivationBuilderImpl : public DerivationBuilder, public DerivationBuilder void killSandbox(bool getStats) override; +protected: + + virtual void cleanupBuild(); + private: - bool cleanupDecideWhetherDiskFull(); + bool decideWhetherDiskFull(); /** * Create alternative path calculated from but distinct from the @@ -462,11 +470,10 @@ bool DerivationBuilderImpl::prepareBuild() { if (useBuildUsers()) { if (!buildUser) - buildUser = acquireUserLock(drvOptions.useUidRange(drv) ? 65536 : 1, useChroot); + buildUser = getBuildUser(); - if (!buildUser) { + if (!buildUser) return false; - } } return true; @@ -528,7 +535,9 @@ std::variant, SingleDrvOutputs> Derivation /* Check the exit status. */ if (!statusOk(status)) { - diskFull |= cleanupDecideWhetherDiskFull(); + diskFull |= decideWhetherDiskFull(); + + cleanupBuild(); auto msg = fmt( "Cannot build '%s'.\n" @@ -582,6 +591,10 @@ std::variant, SingleDrvOutputs> Derivation } } +void DerivationBuilderImpl::cleanupBuild() +{ + deleteTmpDir(false); +} static void chmod_(const Path & path, mode_t mode) { @@ -644,10 +657,7 @@ static void replaceValidPath(const Path & storePath, const Path & tmpPath) deletePath(oldPath); } - - - -bool DerivationBuilderImpl::cleanupDecideWhetherDiskFull() +bool DerivationBuilderImpl::decideWhetherDiskFull() { bool diskFull = false; @@ -670,19 +680,6 @@ bool DerivationBuilderImpl::cleanupDecideWhetherDiskFull() } #endif - deleteTmpDir(false); - - /* Move paths out of the chroot for easier debugging of - build failures. */ - if (useChroot && buildMode == bmNormal) - for (auto & [_, status] : initialOutputs) { - if (!status.known) continue; - if (buildMode != bmCheck && status.known->isValid()) continue; - auto p = store.toRealPath(status.known->path); - if (pathExists(chrootRootDir + p)) - std::filesystem::rename((chrootRootDir + p), p); - } - return diskFull; } @@ -832,23 +829,9 @@ void DerivationBuilderImpl::startBuilder() /* Create a temporary directory where the build will take place. */ topTmpDir = createTempDir(settings.buildDir.get().value_or(""), "nix-build-" + std::string(drvPath.name()), 0700); -#ifdef __APPLE__ - if (false) { -#else - if (useChroot) { -#endif - /* If sandboxing is enabled, put the actual TMPDIR underneath - an inaccessible root-owned directory, to prevent outside - access. - - On macOS, we don't use an actual chroot, so this isn't - possible. Any mitigation along these lines would have to be - done directly in the sandbox profile. */ - tmpDir = topTmpDir + "/build"; - createDir(tmpDir, 0700); - } else { - tmpDir = topTmpDir; - } + setBuildTmpDir(); + assert(!tmpDir.empty()); + assert(!tmpDirInSandbox.empty()); chownToBuilder(tmpDir); for (auto & [outputName, status] : initialOutputs) { @@ -1053,14 +1036,12 @@ DerivationBuilderImpl::PathsInChroot DerivationBuilderImpl::getPathsInSandbox() if (settings.preBuildHook != "") { printMsg(lvlChatty, "executing pre-build hook '%1%'", settings.preBuildHook); - auto args = useChroot ? Strings({store.printStorePath(drvPath), chrootRootDir}) : - Strings({ store.printStorePath(drvPath) }); enum BuildHookState { stBegin, stExtraChrootDirs }; auto state = stBegin; - auto lines = runProgram(settings.preBuildHook, false, args); + auto lines = runProgram(settings.preBuildHook, false, getPreBuildHookArgs()); auto lastPos = std::string::size_type{0}; for (auto nlPos = lines.find('\n'); nlPos != std::string::npos; nlPos = lines.find('\n', lastPos)) @@ -1157,14 +1138,6 @@ void DerivationBuilderImpl::processSandboxSetupMessages() void DerivationBuilderImpl::initTmpDir() { - /* In a sandbox, for determinism, always use the same temporary - directory. */ -#ifdef __linux__ - tmpDirInSandbox = useChroot ? settings.sandboxBuildDir : tmpDir; -#else - tmpDirInSandbox = tmpDir; -#endif - /* In non-structured mode, set all bindings either directory in the environment or via a file, as specified by `DerivationOptions::passAsFile`. */ @@ -1653,14 +1626,6 @@ SingleDrvOutputs DerivationBuilderImpl::registerOutputs() for (auto & i : scratchOutputs) referenceablePaths.insert(i.second); for (auto & p : addedPaths) referenceablePaths.insert(p); - /* FIXME `needsHashRewrite` should probably be removed and we get to the - real reason why we aren't using the chroot dir */ - auto toRealPathChroot = [&](const Path & p) -> Path { - return useChroot && !needsHashRewrite() - ? chrootRootDir + p - : store.toRealPath(p); - }; - /* Check whether the output paths were created, and make all output paths read-only. Then get the references of each output (that we might need to register), so we can topologically sort them. For the ones @@ -1677,7 +1642,7 @@ SingleDrvOutputs DerivationBuilderImpl::registerOutputs() throw BuildError( "builder for '%s' has no scratch output for '%s'", store.printStorePath(drvPath), outputName); - auto actualPath = toRealPathChroot(store.printStorePath(*scratchOutput)); + auto actualPath = realPathInSandbox(store.printStorePath(*scratchOutput)); outputsToSort.insert(outputName); @@ -1786,7 +1751,7 @@ SingleDrvOutputs DerivationBuilderImpl::registerOutputs() auto output = get(drv.outputs, outputName); auto scratchPath = get(scratchOutputs, outputName); assert(output && scratchPath); - auto actualPath = toRealPathChroot(store.printStorePath(*scratchPath)); + auto actualPath = realPathInSandbox(store.printStorePath(*scratchPath)); auto finish = [&](StorePath finalStorePath) { /* Store the final path */ @@ -2361,10 +2326,14 @@ StorePath DerivationBuilderImpl::makeFallbackPath(const StorePath & path) Hash(HashAlgorithm::SHA256), path.name()); } +} + // FIXME: do this properly #include "linux-derivation-builder.cc" #include "darwin-derivation-builder.cc" +namespace nix { + std::unique_ptr makeDerivationBuilder( Store & store, std::unique_ptr miscMethods, diff --git a/src/libstore/unix/build/linux-derivation-builder.cc b/src/libstore/unix/build/linux-derivation-builder.cc index d26a781b5a3..3a8b79ba80e 100644 --- a/src/libstore/unix/build/linux-derivation-builder.cc +++ b/src/libstore/unix/build/linux-derivation-builder.cc @@ -1,5 +1,7 @@ #ifdef __linux__ +namespace nix { + struct LinuxDerivationBuilder : DerivationBuilderImpl { /** @@ -20,23 +22,56 @@ struct LinuxDerivationBuilder : DerivationBuilderImpl */ bool usingUserNamespace = true; + /** + * The root of the chroot environment. + */ + Path chrootRootDir; + PathsInChroot pathsInChroot; LinuxDerivationBuilder( Store & store, std::unique_ptr miscMethods, DerivationBuilderParams params) : DerivationBuilderImpl(store, std::move(miscMethods), std::move(params)) { - useChroot = true; } - uid_t sandboxUid() { return usingUserNamespace ? (!buildUser || buildUser->getUIDCount() == 1 ? 1000 : 0) : buildUser->getUID(); } - gid_t sandboxGid() { return usingUserNamespace ? (!buildUser || buildUser->getUIDCount() == 1 ? 100 : 0) : buildUser->getGID(); } + uid_t sandboxUid() + { + return usingUserNamespace ? (!buildUser || buildUser->getUIDCount() == 1 ? 1000 : 0) : buildUser->getUID(); + } + + gid_t sandboxGid() + { + return usingUserNamespace ? (!buildUser || buildUser->getUIDCount() == 1 ? 100 : 0) : buildUser->getGID(); + } bool needsHashRewrite() override { return false; } + std::unique_ptr getBuildUser() override + { + return acquireUserLock(drvOptions.useUidRange(drv) ? 65536 : 1, true); + } + + void setBuildTmpDir() override + { + /* If sandboxing is enabled, put the actual TMPDIR underneath + an inaccessible root-owned directory, to prevent outside + access. + + On macOS, we don't use an actual chroot, so this isn't + possible. Any mitigation along these lines would have to be + done directly in the sandbox profile. */ + tmpDir = topTmpDir + "/build"; + createDir(tmpDir, 0700); + + /* In a sandbox, for determinism, always use the same temporary + directory. */ + tmpDirInSandbox = settings.sandboxBuildDir; + } + void prepareSandbox() override { /* Create a temporary directory in which we set up the chroot @@ -59,7 +94,10 @@ struct LinuxDerivationBuilder : DerivationBuilderImpl if (mkdir(chrootRootDir.c_str(), buildUser && buildUser->getUIDCount() != 1 ? 0755 : 0750) == -1) throw SysError("cannot create '%1%'", chrootRootDir); - if (buildUser && chown(chrootRootDir.c_str(), buildUser->getUIDCount() != 1 ? buildUser->getUID() : 0, buildUser->getGID()) == -1) + if (buildUser + && chown( + chrootRootDir.c_str(), buildUser->getUIDCount() != 1 ? buildUser->getUID() : 0, buildUser->getGID()) + == -1) throw SysError("cannot change ownership of '%1%'", chrootRootDir); /* Create a writable /tmp in the chroot. Many builders need @@ -81,10 +119,12 @@ struct LinuxDerivationBuilder : DerivationBuilderImpl /* Declare the build user's group so that programs get a consistent view of the system (e.g., "id -gn"). */ - writeFile(chrootRootDir + "/etc/group", + writeFile( + chrootRootDir + "/etc/group", fmt("root:x:0:\n" "nixbld:!:%1%:\n" - "nogroup:x:65534:\n", sandboxGid())); + "nogroup:x:65534:\n", + sandboxGid())); /* Create /etc/hosts with localhost entry. */ if (derivationType.isSandboxed()) @@ -125,7 +165,7 @@ struct LinuxDerivationBuilder : DerivationBuilderImpl chownToBuilder(*cgroup); chownToBuilder(*cgroup + "/cgroup.procs"); chownToBuilder(*cgroup + "/cgroup.threads"); - //chownToBuilder(*cgroup + "/cgroup.subtree_control"); + // chownToBuilder(*cgroup + "/cgroup.subtree_control"); } pathsInChroot = getPathsInSandbox(); @@ -136,6 +176,18 @@ struct LinuxDerivationBuilder : DerivationBuilderImpl } } + Strings getPreBuildHookArgs() override + { + assert(!chrootRootDir.empty()); + return Strings({store.printStorePath(drvPath), chrootRootDir}); + } + + Path realPathInSandbox(const Path & p) override + { + // FIXME: why the needsHashRewrite() conditional? + return !needsHashRewrite() ? chrootRootDir + p : store.toRealPath(p); + } + void startChild() override { /* Set up private namespaces for the build: @@ -194,7 +246,8 @@ struct LinuxDerivationBuilder : DerivationBuilderImpl if (errno != EPERM) throw SysError("setgroups failed"); if (settings.requireDropSupplementaryGroups) - throw Error("setgroups failed. Set the require-drop-supplementary-groups option to false to skip this step."); + throw Error( + "setgroups failed. Set the require-drop-supplementary-groups option to false to skip this step."); } ProcessOptions options; @@ -226,9 +279,7 @@ struct LinuxDerivationBuilder : DerivationBuilderImpl /* Close the write side to prevent runChild() from hanging reading from this. */ - Finally cleanup([&]() { - userNamespaceSync.writeSide = -1; - }); + Finally cleanup([&]() { userNamespaceSync.writeSide = -1; }); auto ss = tokenizeString>(readLine(sendPid.readSide.get())); assert(ss.size() == 1); @@ -242,30 +293,32 @@ struct LinuxDerivationBuilder : DerivationBuilderImpl uid_t hostGid = buildUser ? buildUser->getGID() : getgid(); uid_t nrIds = buildUser ? buildUser->getUIDCount() : 1; - writeFile("/proc/" + std::to_string(pid) + "/uid_map", - fmt("%d %d %d", sandboxUid(), hostUid, nrIds)); + writeFile("/proc/" + std::to_string(pid) + "/uid_map", fmt("%d %d %d", sandboxUid(), hostUid, nrIds)); if (!buildUser || buildUser->getUIDCount() == 1) writeFile("/proc/" + std::to_string(pid) + "/setgroups", "deny"); - writeFile("/proc/" + std::to_string(pid) + "/gid_map", - fmt("%d %d %d", sandboxGid(), hostGid, nrIds)); + writeFile("/proc/" + std::to_string(pid) + "/gid_map", fmt("%d %d %d", sandboxGid(), hostGid, nrIds)); } else { debug("note: not using a user namespace"); if (!buildUser) - throw Error("cannot perform a sandboxed build because user namespaces are not enabled; check /proc/sys/user/max_user_namespaces"); + throw Error( + "cannot perform a sandboxed build because user namespaces are not enabled; check /proc/sys/user/max_user_namespaces"); } /* Now that we now the sandbox uid, we can write /etc/passwd. */ - writeFile(chrootRootDir + "/etc/passwd", fmt( - "root:x:0:0:Nix build user:%3%:/noshell\n" + writeFile( + chrootRootDir + "/etc/passwd", + fmt("root:x:0:0:Nix build user:%3%:/noshell\n" "nixbld:x:%1%:%2%:Nix build user:%3%:/noshell\n" "nobody:x:65534:65534:Nobody:/:/noshell\n", - sandboxUid(), sandboxGid(), settings.sandboxBuildDir)); + sandboxUid(), + sandboxGid(), + settings.sandboxBuildDir)); /* Save the mount- and user namespace of the child. We have to do this - *before* the child does a chroot. */ + *before* the child does a chroot. */ sandboxMountNamespace = open(fmt("/proc/%d/ns/mnt", (pid_t) pid).c_str(), O_RDONLY); if (sandboxMountNamespace.get() == -1) throw SysError("getting sandbox mount namespace"); @@ -528,6 +581,24 @@ struct LinuxDerivationBuilder : DerivationBuilderImpl return DerivationBuilderImpl::unprepareBuild(); } + void cleanupBuild() override + { + DerivationBuilderImpl::cleanupBuild(); + + /* Move paths out of the chroot for easier debugging of + build failures. */ + if (buildMode == bmNormal) + for (auto & [_, status] : initialOutputs) { + if (!status.known) + continue; + if (buildMode != bmCheck && status.known->isValid()) + continue; + auto p = store.toRealPath(status.known->path); + if (pathExists(chrootRootDir + p)) + std::filesystem::rename((chrootRootDir + p), p); + } + } + void addDependency(const StorePath & path) override { if (isAllowed(path)) @@ -568,4 +639,6 @@ struct LinuxDerivationBuilder : DerivationBuilderImpl } }; +} + #endif From b623fe8d143737b98c170261800a71808cf6b828 Mon Sep 17 00:00:00 2001 From: Eelco Dolstra Date: Mon, 26 May 2025 21:36:13 +0200 Subject: [PATCH 082/218] Move doBind() --- src/libstore/unix/build/derivation-builder.cc | 47 ----------------- .../unix/build/linux-derivation-builder.cc | 50 +++++++++++++++++++ 2 files changed, 50 insertions(+), 47 deletions(-) diff --git a/src/libstore/unix/build/derivation-builder.cc b/src/libstore/unix/build/derivation-builder.cc index 497e0bf31db..819658c8838 100644 --- a/src/libstore/unix/build/derivation-builder.cc +++ b/src/libstore/unix/build/derivation-builder.cc @@ -32,24 +32,13 @@ # include #endif -/* Includes required for chroot support. */ #ifdef __linux__ # include "linux/fchmodat2-compat.hh" -# include -# include -# include -# include -# include -# include -# include # include -# include "nix/util/namespaces.hh" # if HAVE_SECCOMP # include # endif -# define pivot_root(new_root, put_old) (syscall(SYS_pivot_root, new_root, put_old)) # include "nix/util/cgroup.hh" -# include "nix/store/personality.hh" #endif #include @@ -683,42 +672,6 @@ bool DerivationBuilderImpl::decideWhetherDiskFull() return diskFull; } - -#ifdef __linux__ -static void doBind(const Path & source, const Path & target, bool optional = false) { - debug("bind mounting '%1%' to '%2%'", source, target); - - auto bindMount = [&]() { - if (mount(source.c_str(), target.c_str(), "", MS_BIND | MS_REC, 0) == -1) - throw SysError("bind mount from '%1%' to '%2%' failed", source, target); - }; - - auto maybeSt = maybeLstat(source); - if (!maybeSt) { - if (optional) - return; - else - throw SysError("getting attributes of path '%1%'", source); - } - auto st = *maybeSt; - - if (S_ISDIR(st.st_mode)) { - createDirs(target); - bindMount(); - } else if (S_ISLNK(st.st_mode)) { - // Symlinks can (apparently) not be bind-mounted, so just copy it - createDirs(dirOf(target)); - copyFile( - std::filesystem::path(source), - std::filesystem::path(target), false); - } else { - createDirs(dirOf(target)); - writeFile(target, ""); - bindMount(); - } -}; -#endif - /** * Rethrow the current exception as a subclass of `Error`. */ diff --git a/src/libstore/unix/build/linux-derivation-builder.cc b/src/libstore/unix/build/linux-derivation-builder.cc index 3a8b79ba80e..9d475183976 100644 --- a/src/libstore/unix/build/linux-derivation-builder.cc +++ b/src/libstore/unix/build/linux-derivation-builder.cc @@ -1,7 +1,57 @@ #ifdef __linux__ +# ifdef __linux__ +# include +# include +# include +# include +# include +# include +# include +# include +# include "nix/util/namespaces.hh" +# if HAVE_SECCOMP +# include +# endif +# define pivot_root(new_root, put_old) (syscall(SYS_pivot_root, new_root, put_old)) +# include "nix/util/cgroup.hh" +# include "nix/store/personality.hh" +# endif + namespace nix { +static void doBind(const Path & source, const Path & target, bool optional = false) +{ + debug("bind mounting '%1%' to '%2%'", source, target); + + auto bindMount = [&]() { + if (mount(source.c_str(), target.c_str(), "", MS_BIND | MS_REC, 0) == -1) + throw SysError("bind mount from '%1%' to '%2%' failed", source, target); + }; + + auto maybeSt = maybeLstat(source); + if (!maybeSt) { + if (optional) + return; + else + throw SysError("getting attributes of path '%1%'", source); + } + auto st = *maybeSt; + + if (S_ISDIR(st.st_mode)) { + createDirs(target); + bindMount(); + } else if (S_ISLNK(st.st_mode)) { + // Symlinks can (apparently) not be bind-mounted, so just copy it + createDirs(dirOf(target)); + copyFile(std::filesystem::path(source), std::filesystem::path(target), false); + } else { + createDirs(dirOf(target)); + writeFile(target, ""); + bindMount(); + } +} + struct LinuxDerivationBuilder : DerivationBuilderImpl { /** From 9e2151d839713318981e77b227cda09643356815 Mon Sep 17 00:00:00 2001 From: Eelco Dolstra Date: Mon, 26 May 2025 21:44:51 +0200 Subject: [PATCH 083/218] Move seccomp code --- src/libstore/unix/build/derivation-builder.cc | 100 ------------- .../unix/build/linux-derivation-builder.cc | 133 +++++++++++++++--- 2 files changed, 117 insertions(+), 116 deletions(-) diff --git a/src/libstore/unix/build/derivation-builder.cc b/src/libstore/unix/build/derivation-builder.cc index 819658c8838..793439efde2 100644 --- a/src/libstore/unix/build/derivation-builder.cc +++ b/src/libstore/unix/build/derivation-builder.cc @@ -33,11 +33,6 @@ #endif #ifdef __linux__ -# include "linux/fchmodat2-compat.hh" -# include -# if HAVE_SECCOMP -# include -# endif # include "nix/util/cgroup.hh" #endif @@ -1337,95 +1332,6 @@ void DerivationBuilderImpl::chownToBuilder(const Path & path) throw SysError("cannot change ownership of '%1%'", path); } - -void setupSeccomp() -{ -#ifdef __linux__ - if (!settings.filterSyscalls) return; -#if HAVE_SECCOMP - scmp_filter_ctx ctx; - - if (!(ctx = seccomp_init(SCMP_ACT_ALLOW))) - throw SysError("unable to initialize seccomp mode 2"); - - Finally cleanup([&]() { - seccomp_release(ctx); - }); - - constexpr std::string_view nativeSystem = NIX_LOCAL_SYSTEM; - - if (nativeSystem == "x86_64-linux" && - seccomp_arch_add(ctx, SCMP_ARCH_X86) != 0) - throw SysError("unable to add 32-bit seccomp architecture"); - - if (nativeSystem == "x86_64-linux" && - seccomp_arch_add(ctx, SCMP_ARCH_X32) != 0) - throw SysError("unable to add X32 seccomp architecture"); - - if (nativeSystem == "aarch64-linux" && - seccomp_arch_add(ctx, SCMP_ARCH_ARM) != 0) - printError("unable to add ARM seccomp architecture; this may result in spurious build failures if running 32-bit ARM processes"); - - if (nativeSystem == "mips64-linux" && - seccomp_arch_add(ctx, SCMP_ARCH_MIPS) != 0) - printError("unable to add mips seccomp architecture"); - - if (nativeSystem == "mips64-linux" && - seccomp_arch_add(ctx, SCMP_ARCH_MIPS64N32) != 0) - printError("unable to add mips64-*abin32 seccomp architecture"); - - if (nativeSystem == "mips64el-linux" && - seccomp_arch_add(ctx, SCMP_ARCH_MIPSEL) != 0) - printError("unable to add mipsel seccomp architecture"); - - if (nativeSystem == "mips64el-linux" && - seccomp_arch_add(ctx, SCMP_ARCH_MIPSEL64N32) != 0) - printError("unable to add mips64el-*abin32 seccomp architecture"); - - /* Prevent builders from creating setuid/setgid binaries. */ - for (int perm : { S_ISUID, S_ISGID }) { - if (seccomp_rule_add(ctx, SCMP_ACT_ERRNO(EPERM), SCMP_SYS(chmod), 1, - SCMP_A1(SCMP_CMP_MASKED_EQ, (scmp_datum_t) perm, (scmp_datum_t) perm)) != 0) - throw SysError("unable to add seccomp rule"); - - if (seccomp_rule_add(ctx, SCMP_ACT_ERRNO(EPERM), SCMP_SYS(fchmod), 1, - SCMP_A1(SCMP_CMP_MASKED_EQ, (scmp_datum_t) perm, (scmp_datum_t) perm)) != 0) - throw SysError("unable to add seccomp rule"); - - if (seccomp_rule_add(ctx, SCMP_ACT_ERRNO(EPERM), SCMP_SYS(fchmodat), 1, - SCMP_A2(SCMP_CMP_MASKED_EQ, (scmp_datum_t) perm, (scmp_datum_t) perm)) != 0) - throw SysError("unable to add seccomp rule"); - - if (seccomp_rule_add(ctx, SCMP_ACT_ERRNO(EPERM), NIX_SYSCALL_FCHMODAT2, 1, - SCMP_A2(SCMP_CMP_MASKED_EQ, (scmp_datum_t) perm, (scmp_datum_t) perm)) != 0) - throw SysError("unable to add seccomp rule"); - } - - /* Prevent builders from using EAs or ACLs. Not all filesystems - support these, and they're not allowed in the Nix store because - they're not representable in the NAR serialisation. */ - if (seccomp_rule_add(ctx, SCMP_ACT_ERRNO(ENOTSUP), SCMP_SYS(getxattr), 0) != 0 || - seccomp_rule_add(ctx, SCMP_ACT_ERRNO(ENOTSUP), SCMP_SYS(lgetxattr), 0) != 0 || - seccomp_rule_add(ctx, SCMP_ACT_ERRNO(ENOTSUP), SCMP_SYS(fgetxattr), 0) != 0 || - seccomp_rule_add(ctx, SCMP_ACT_ERRNO(ENOTSUP), SCMP_SYS(setxattr), 0) != 0 || - seccomp_rule_add(ctx, SCMP_ACT_ERRNO(ENOTSUP), SCMP_SYS(lsetxattr), 0) != 0 || - seccomp_rule_add(ctx, SCMP_ACT_ERRNO(ENOTSUP), SCMP_SYS(fsetxattr), 0) != 0) - throw SysError("unable to add seccomp rule"); - - if (seccomp_attr_set(ctx, SCMP_FLTATR_CTL_NNP, settings.allowNewPrivileges ? 0 : 1) != 0) - throw SysError("unable to set 'no new privileges' seccomp attribute"); - - if (seccomp_load(ctx) != 0) - throw SysError("unable to load seccomp BPF program"); -#else - throw Error( - "seccomp is not supported on this platform; " - "you can bypass this error by setting the option 'filter-syscalls' to false, but note that untrusted builds can then create setuid binaries!"); -#endif -#endif -} - - void DerivationBuilderImpl::runChild() { /* Warning: in the child we should absolutely not make any SQLite @@ -1437,12 +1343,6 @@ void DerivationBuilderImpl::runChild() commonChildInit(); - try { - setupSeccomp(); - } catch (...) { - if (buildUser) throw; - } - /* Make the contents of netrc and the CA certificate bundle available to builtin:fetchurl (which may run under a different uid and/or in a sandbox). */ diff --git a/src/libstore/unix/build/linux-derivation-builder.cc b/src/libstore/unix/build/linux-derivation-builder.cc index 9d475183976..648f55adb4a 100644 --- a/src/libstore/unix/build/linux-derivation-builder.cc +++ b/src/libstore/unix/build/linux-derivation-builder.cc @@ -1,25 +1,123 @@ #ifdef __linux__ -# ifdef __linux__ -# include -# include -# include -# include -# include -# include -# include -# include -# include "nix/util/namespaces.hh" -# if HAVE_SECCOMP -# include -# endif -# define pivot_root(new_root, put_old) (syscall(SYS_pivot_root, new_root, put_old)) -# include "nix/util/cgroup.hh" -# include "nix/store/personality.hh" +# include "linux/fchmodat2-compat.hh" +# include +# include +# include +# include +# include +# include +# include +# include +# include "nix/util/namespaces.hh" +# if HAVE_SECCOMP +# include # endif +# define pivot_root(new_root, put_old) (syscall(SYS_pivot_root, new_root, put_old)) +# include "nix/util/cgroup.hh" +# include "nix/store/personality.hh" namespace nix { +static void setupSeccomp() +{ + if (!settings.filterSyscalls) + return; + +# if HAVE_SECCOMP + scmp_filter_ctx ctx; + + if (!(ctx = seccomp_init(SCMP_ACT_ALLOW))) + throw SysError("unable to initialize seccomp mode 2"); + + Finally cleanup([&]() { seccomp_release(ctx); }); + + constexpr std::string_view nativeSystem = NIX_LOCAL_SYSTEM; + + if (nativeSystem == "x86_64-linux" && seccomp_arch_add(ctx, SCMP_ARCH_X86) != 0) + throw SysError("unable to add 32-bit seccomp architecture"); + + if (nativeSystem == "x86_64-linux" && seccomp_arch_add(ctx, SCMP_ARCH_X32) != 0) + throw SysError("unable to add X32 seccomp architecture"); + + if (nativeSystem == "aarch64-linux" && seccomp_arch_add(ctx, SCMP_ARCH_ARM) != 0) + printError( + "unable to add ARM seccomp architecture; this may result in spurious build failures if running 32-bit ARM processes"); + + if (nativeSystem == "mips64-linux" && seccomp_arch_add(ctx, SCMP_ARCH_MIPS) != 0) + printError("unable to add mips seccomp architecture"); + + if (nativeSystem == "mips64-linux" && seccomp_arch_add(ctx, SCMP_ARCH_MIPS64N32) != 0) + printError("unable to add mips64-*abin32 seccomp architecture"); + + if (nativeSystem == "mips64el-linux" && seccomp_arch_add(ctx, SCMP_ARCH_MIPSEL) != 0) + printError("unable to add mipsel seccomp architecture"); + + if (nativeSystem == "mips64el-linux" && seccomp_arch_add(ctx, SCMP_ARCH_MIPSEL64N32) != 0) + printError("unable to add mips64el-*abin32 seccomp architecture"); + + /* Prevent builders from creating setuid/setgid binaries. */ + for (int perm : {S_ISUID, S_ISGID}) { + if (seccomp_rule_add( + ctx, + SCMP_ACT_ERRNO(EPERM), + SCMP_SYS(chmod), + 1, + SCMP_A1(SCMP_CMP_MASKED_EQ, (scmp_datum_t) perm, (scmp_datum_t) perm)) + != 0) + throw SysError("unable to add seccomp rule"); + + if (seccomp_rule_add( + ctx, + SCMP_ACT_ERRNO(EPERM), + SCMP_SYS(fchmod), + 1, + SCMP_A1(SCMP_CMP_MASKED_EQ, (scmp_datum_t) perm, (scmp_datum_t) perm)) + != 0) + throw SysError("unable to add seccomp rule"); + + if (seccomp_rule_add( + ctx, + SCMP_ACT_ERRNO(EPERM), + SCMP_SYS(fchmodat), + 1, + SCMP_A2(SCMP_CMP_MASKED_EQ, (scmp_datum_t) perm, (scmp_datum_t) perm)) + != 0) + throw SysError("unable to add seccomp rule"); + + if (seccomp_rule_add( + ctx, + SCMP_ACT_ERRNO(EPERM), + NIX_SYSCALL_FCHMODAT2, + 1, + SCMP_A2(SCMP_CMP_MASKED_EQ, (scmp_datum_t) perm, (scmp_datum_t) perm)) + != 0) + throw SysError("unable to add seccomp rule"); + } + + /* Prevent builders from using EAs or ACLs. Not all filesystems + support these, and they're not allowed in the Nix store because + they're not representable in the NAR serialisation. */ + if (seccomp_rule_add(ctx, SCMP_ACT_ERRNO(ENOTSUP), SCMP_SYS(getxattr), 0) != 0 + || seccomp_rule_add(ctx, SCMP_ACT_ERRNO(ENOTSUP), SCMP_SYS(lgetxattr), 0) != 0 + || seccomp_rule_add(ctx, SCMP_ACT_ERRNO(ENOTSUP), SCMP_SYS(fgetxattr), 0) != 0 + || seccomp_rule_add(ctx, SCMP_ACT_ERRNO(ENOTSUP), SCMP_SYS(setxattr), 0) != 0 + || seccomp_rule_add(ctx, SCMP_ACT_ERRNO(ENOTSUP), SCMP_SYS(lsetxattr), 0) != 0 + || seccomp_rule_add(ctx, SCMP_ACT_ERRNO(ENOTSUP), SCMP_SYS(fsetxattr), 0) != 0) + throw SysError("unable to add seccomp rule"); + + if (seccomp_attr_set(ctx, SCMP_FLTATR_CTL_NNP, settings.allowNewPrivileges ? 0 : 1) != 0) + throw SysError("unable to set 'no new privileges' seccomp attribute"); + + if (seccomp_load(ctx) != 0) + throw SysError("unable to load seccomp BPF program"); +# else + throw Error( + "seccomp is not supported on this platform; " + "you can bypass this error by setting the option 'filter-syscalls' to false, but note that untrusted builds can then create setuid binaries!"); +# endif +} + static void doBind(const Path & source, const Path & target, bool optional = false) { debug("bind mounting '%1%' to '%2%'", source, target); @@ -608,6 +706,9 @@ struct LinuxDerivationBuilder : DerivationBuilderImpl if (rmdir("real-root") == -1) throw SysError("cannot remove real-root directory"); + // FIXME: move to LinuxDerivationBuilder + setupSeccomp(); + // FIXME: move to LinuxDerivationBuilder linux::setPersonality(drv.platform); } From c9b55fa3f098a1ab01ddae534e5d65ee10f25061 Mon Sep 17 00:00:00 2001 From: Eelco Dolstra Date: Mon, 26 May 2025 22:23:59 +0200 Subject: [PATCH 084/218] Move autoDelChroot --- src/libstore/unix/build/derivation-builder.cc | 9 --------- src/libstore/unix/build/linux-derivation-builder.cc | 12 ++++++++++++ 2 files changed, 12 insertions(+), 9 deletions(-) diff --git a/src/libstore/unix/build/derivation-builder.cc b/src/libstore/unix/build/derivation-builder.cc index 793439efde2..037359bb02f 100644 --- a/src/libstore/unix/build/derivation-builder.cc +++ b/src/libstore/unix/build/derivation-builder.cc @@ -110,12 +110,6 @@ class DerivationBuilderImpl : public DerivationBuilder, public DerivationBuilder */ Path tmpDirInSandbox; - /** - * RAII object to delete the chroot directory. - */ - // FIXME: move - std::shared_ptr autoDelChroot; - /** * The sort of derivation we are building. * @@ -557,9 +551,6 @@ std::variant, SingleDrvOutputs> Derivation for (auto & i : redirectedOutputs) deletePath(store.Store::toRealPath(i.second)); - /* Delete the chroot (if we were using one). */ - autoDelChroot.reset(); /* this runs the destructor */ - deleteTmpDir(true); return std::move(builtOutputs); diff --git a/src/libstore/unix/build/linux-derivation-builder.cc b/src/libstore/unix/build/linux-derivation-builder.cc index 648f55adb4a..ee2e0c044d3 100644 --- a/src/libstore/unix/build/linux-derivation-builder.cc +++ b/src/libstore/unix/build/linux-derivation-builder.cc @@ -175,6 +175,11 @@ struct LinuxDerivationBuilder : DerivationBuilderImpl */ Path chrootRootDir; + /** + * RAII object to delete the chroot directory. + */ + std::shared_ptr autoDelChroot; + PathsInChroot pathsInChroot; LinuxDerivationBuilder( @@ -183,6 +188,13 @@ struct LinuxDerivationBuilder : DerivationBuilderImpl { } + void deleteTmpDir(bool force) override + { + autoDelChroot.reset(); /* this runs the destructor */ + + DerivationBuilderImpl::deleteTmpDir(force); + } + uid_t sandboxUid() { return usingUserNamespace ? (!buildUser || buildUser->getUIDCount() == 1 ? 1000 : 0) : buildUser->getUID(); From 774678b87f90148d507b0cdaf33f4d1194e068e5 Mon Sep 17 00:00:00 2001 From: Eelco Dolstra Date: Mon, 26 May 2025 22:29:08 +0200 Subject: [PATCH 085/218] Get rid of tmpDirInSandbox variable --- src/libstore/unix/build/derivation-builder.cc | 38 ++++++++++--------- .../unix/build/linux-derivation-builder.cc | 5 ++- 2 files changed, 24 insertions(+), 19 deletions(-) diff --git a/src/libstore/unix/build/derivation-builder.cc b/src/libstore/unix/build/derivation-builder.cc index 037359bb02f..217a677cda9 100644 --- a/src/libstore/unix/build/derivation-builder.cc +++ b/src/libstore/unix/build/derivation-builder.cc @@ -105,11 +105,6 @@ class DerivationBuilderImpl : public DerivationBuilder, public DerivationBuilder */ Path topTmpDir; - /** - * The path of the temporary directory in the sandbox. - */ - Path tmpDirInSandbox; - /** * The sort of derivation we are building. * @@ -230,7 +225,15 @@ class DerivationBuilderImpl : public DerivationBuilder, public DerivationBuilder virtual void setBuildTmpDir() { tmpDir = topTmpDir; - tmpDirInSandbox = topTmpDir; + } + + /** + * Return the path of the temporary directory in the sandbox. + */ + virtual Path tmpDirInSandbox() + { + assert(!topTmpDir.empty()); + return topTmpDir; } /** @@ -770,7 +773,6 @@ void DerivationBuilderImpl::startBuilder() topTmpDir = createTempDir(settings.buildDir.get().value_or(""), "nix-build-" + std::string(drvPath.name()), 0700); setBuildTmpDir(); assert(!tmpDir.empty()); - assert(!tmpDirInSandbox.empty()); chownToBuilder(tmpDir); for (auto & [outputName, status] : initialOutputs) { @@ -923,11 +925,11 @@ DerivationBuilderImpl::PathsInChroot DerivationBuilderImpl::getPathsInSandbox() else pathsInChroot[i.substr(0, p)] = {i.substr(p + 1), optional}; } - if (hasPrefix(store.storeDir, tmpDirInSandbox)) + if (hasPrefix(store.storeDir, tmpDirInSandbox())) { throw Error("`sandbox-build-dir` must not contain the storeDir"); } - pathsInChroot[tmpDirInSandbox] = tmpDir; + pathsInChroot[tmpDirInSandbox()] = tmpDir; /* Add the closure of store paths to the chroot. */ StorePathSet closure; @@ -1090,7 +1092,7 @@ void DerivationBuilderImpl::initTmpDir() Path p = tmpDir + "/" + fn; writeFile(p, rewriteStrings(i.second, inputRewrites)); chownToBuilder(p); - env[i.first + "Path"] = tmpDirInSandbox + "/" + fn; + env[i.first + "Path"] = tmpDirInSandbox() + "/" + fn; } } @@ -1098,16 +1100,16 @@ void DerivationBuilderImpl::initTmpDir() /* For convenience, set an environment pointing to the top build directory. */ - env["NIX_BUILD_TOP"] = tmpDirInSandbox; + env["NIX_BUILD_TOP"] = tmpDirInSandbox(); /* Also set TMPDIR and variants to point to this directory. */ - env["TMPDIR"] = env["TEMPDIR"] = env["TMP"] = env["TEMP"] = tmpDirInSandbox; + env["TMPDIR"] = env["TEMPDIR"] = env["TMP"] = env["TEMP"] = tmpDirInSandbox(); /* Explicitly set PWD to prevent problems with chroot builds. In particular, dietlibc cannot figure out the cwd because the inode of the current directory doesn't appear in .. (because getdents returns the inode of the mount point). */ - env["PWD"] = tmpDirInSandbox; + env["PWD"] = tmpDirInSandbox(); } @@ -1200,10 +1202,10 @@ void DerivationBuilderImpl::writeStructuredAttrs() writeFile(tmpDir + "/.attrs.sh", rewriteStrings(jsonSh, inputRewrites)); chownToBuilder(tmpDir + "/.attrs.sh"); - env["NIX_ATTRS_SH_FILE"] = tmpDirInSandbox + "/.attrs.sh"; + env["NIX_ATTRS_SH_FILE"] = tmpDirInSandbox() + "/.attrs.sh"; writeFile(tmpDir + "/.attrs.json", rewriteStrings(json.dump(), inputRewrites)); chownToBuilder(tmpDir + "/.attrs.json"); - env["NIX_ATTRS_JSON_FILE"] = tmpDirInSandbox + "/.attrs.json"; + env["NIX_ATTRS_JSON_FILE"] = tmpDirInSandbox() + "/.attrs.json"; } } @@ -1227,7 +1229,7 @@ void DerivationBuilderImpl::startDaemon() auto socketName = ".nix-socket"; Path socketPath = tmpDir + "/" + socketName; - env["NIX_REMOTE"] = "unix://" + tmpDirInSandbox + "/" + socketName; + env["NIX_REMOTE"] = "unix://" + tmpDirInSandbox() + "/" + socketName; daemonSocket = createUnixDomainSocket(socketPath, 0600); @@ -1339,7 +1341,7 @@ void DerivationBuilderImpl::runChild() different uid and/or in a sandbox). */ BuiltinBuilderContext ctx{ .drv = drv, - .tmpDirInSandbox = tmpDirInSandbox, + .tmpDirInSandbox = tmpDirInSandbox(), }; if (drv.isBuiltin() && drv.builder == "builtin:fetchurl") { @@ -1354,7 +1356,7 @@ void DerivationBuilderImpl::runChild() enterChroot(); - if (chdir(tmpDirInSandbox.c_str()) == -1) + if (chdir(tmpDirInSandbox().c_str()) == -1) throw SysError("changing into '%1%'", tmpDir); /* Close all other file descriptors. */ diff --git a/src/libstore/unix/build/linux-derivation-builder.cc b/src/libstore/unix/build/linux-derivation-builder.cc index ee2e0c044d3..5c6c80a0e72 100644 --- a/src/libstore/unix/build/linux-derivation-builder.cc +++ b/src/libstore/unix/build/linux-derivation-builder.cc @@ -226,10 +226,13 @@ struct LinuxDerivationBuilder : DerivationBuilderImpl done directly in the sandbox profile. */ tmpDir = topTmpDir + "/build"; createDir(tmpDir, 0700); + } + Path tmpDirInSandbox() override + { /* In a sandbox, for determinism, always use the same temporary directory. */ - tmpDirInSandbox = settings.sandboxBuildDir; + return settings.sandboxBuildDir; } void prepareSandbox() override From b27e684ca5ae84a3af68e5bd3f2da84c90e20e9f Mon Sep 17 00:00:00 2001 From: Eelco Dolstra Date: Mon, 26 May 2025 22:35:47 +0200 Subject: [PATCH 086/218] Inline initTmpDir() --- src/libstore/unix/build/derivation-builder.cc | 62 ++++++++----------- 1 file changed, 25 insertions(+), 37 deletions(-) diff --git a/src/libstore/unix/build/derivation-builder.cc b/src/libstore/unix/build/derivation-builder.cc index 217a677cda9..bc1ca959ad5 100644 --- a/src/libstore/unix/build/derivation-builder.cc +++ b/src/libstore/unix/build/derivation-builder.cc @@ -279,11 +279,6 @@ class DerivationBuilderImpl : public DerivationBuilder, public DerivationBuilder private: - /** - * Setup tmp dir location. - */ - void initTmpDir(); - /** * Write a JSON file containing the derivation attributes. */ @@ -1076,9 +1071,32 @@ void DerivationBuilderImpl::processSandboxSetupMessages() } } - -void DerivationBuilderImpl::initTmpDir() +void DerivationBuilderImpl::initEnv() { + env.clear(); + + /* Most shells initialise PATH to some default (/bin:/usr/bin:...) when + PATH is not set. We don't want this, so we fill it in with some dummy + value. */ + env["PATH"] = "/path-not-set"; + + /* Set HOME to a non-existing path to prevent certain programs from using + /etc/passwd (or NIS, or whatever) to locate the home directory (for + example, wget looks for ~/.wgetrc). I.e., these tools use /etc/passwd + if HOME is not set, but they will just assume that the settings file + they are looking for does not exist if HOME is set but points to some + non-existing path. */ + env["HOME"] = homeDir; + + /* Tell the builder where the Nix store is. Usually they + shouldn't care, but this is useful for purity checking (e.g., + the compiler or linker might only want to accept paths to files + in the store or in the build directory). */ + env["NIX_STORE"] = store.storeDir; + + /* The maximum number of cores to utilize for parallel building. */ + env["NIX_BUILD_CORES"] = fmt("%d", settings.buildCores); + /* In non-structured mode, set all bindings either directory in the environment or via a file, as specified by `DerivationOptions::passAsFile`. */ @@ -1110,36 +1128,6 @@ void DerivationBuilderImpl::initTmpDir() inode of the current directory doesn't appear in .. (because getdents returns the inode of the mount point). */ env["PWD"] = tmpDirInSandbox(); -} - - -void DerivationBuilderImpl::initEnv() -{ - env.clear(); - - /* Most shells initialise PATH to some default (/bin:/usr/bin:...) when - PATH is not set. We don't want this, so we fill it in with some dummy - value. */ - env["PATH"] = "/path-not-set"; - - /* Set HOME to a non-existing path to prevent certain programs from using - /etc/passwd (or NIS, or whatever) to locate the home directory (for - example, wget looks for ~/.wgetrc). I.e., these tools use /etc/passwd - if HOME is not set, but they will just assume that the settings file - they are looking for does not exist if HOME is set but points to some - non-existing path. */ - env["HOME"] = homeDir; - - /* Tell the builder where the Nix store is. Usually they - shouldn't care, but this is useful for purity checking (e.g., - the compiler or linker might only want to accept paths to files - in the store or in the build directory). */ - env["NIX_STORE"] = store.storeDir; - - /* The maximum number of cores to utilize for parallel building. */ - env["NIX_BUILD_CORES"] = fmt("%d", settings.buildCores); - - initTmpDir(); /* Compatibility hack with Nix <= 0.7: if this is a fixed-output derivation, tell the builder, so that for instance `fetchurl' From 352ca238a909fd2b81f5f5b58cac6892a03d5096 Mon Sep 17 00:00:00 2001 From: Eelco Dolstra Date: Mon, 26 May 2025 23:51:24 +0200 Subject: [PATCH 087/218] Move cgroup support --- .../build/derivation-building-goal.cc | 2 + src/libstore/unix/build/derivation-builder.cc | 85 +++---------------- .../unix/build/linux-derivation-builder.cc | 73 +++++++++++++++- 3 files changed, 83 insertions(+), 77 deletions(-) diff --git a/src/libstore/build/derivation-building-goal.cc b/src/libstore/build/derivation-building-goal.cc index 2b5473b81fb..19296fac315 100644 --- a/src/libstore/build/derivation-building-goal.cc +++ b/src/libstore/build/derivation-building-goal.cc @@ -86,6 +86,8 @@ void DerivationBuildingGoal::killChild() if (builder && builder->pid != -1) { worker.childTerminated(this); + // FIXME: move this into DerivationBuilder. + /* If we're using a build user, then there is a tricky race condition: if we kill the build user before the child has done its setuid() to the build user uid, then it won't be diff --git a/src/libstore/unix/build/derivation-builder.cc b/src/libstore/unix/build/derivation-builder.cc index bc1ca959ad5..142f75d1416 100644 --- a/src/libstore/unix/build/derivation-builder.cc +++ b/src/libstore/unix/build/derivation-builder.cc @@ -32,10 +32,6 @@ # include #endif -#ifdef __linux__ -# include "nix/util/cgroup.hh" -#endif - #include #include #include @@ -88,12 +84,6 @@ class DerivationBuilderImpl : public DerivationBuilder, public DerivationBuilder */ std::unique_ptr buildUser; - /** - * The cgroup of the builder, if any. - */ - // FIXME: move - std::optional cgroup; - /** * The temporary directory used for the build. */ @@ -236,6 +226,15 @@ class DerivationBuilderImpl : public DerivationBuilder, public DerivationBuilder return topTmpDir; } + /** + * Ensure that there are no processes running that conflict with + * `buildUser`. + */ + virtual void prepareUser() + { + killSandbox(false); + } + /** * Called by prepareBuild() to do any setup in the parent to * prepare for a sandboxed build. @@ -422,19 +421,7 @@ static LocalStore & getLocalStore(Store & store) void DerivationBuilderImpl::killSandbox(bool getStats) { - if (cgroup) { - #ifdef __linux__ - auto stats = destroyCgroup(*cgroup); - if (getStats) { - buildResult.cpuUser = stats.cpuUser; - buildResult.cpuSystem = stats.cpuSystem; - } - #else - unreachable(); - #endif - } - - else if (buildUser) { + if (buildUser) { auto uid = buildUser->getUID(); assert(uid != 0); killUser(uid); @@ -693,60 +680,10 @@ static void handleChildException(bool sendException) void DerivationBuilderImpl::startBuilder() { - if ((buildUser && buildUser->getUIDCount() != 1) - #ifdef __linux__ - || settings.useCgroups - #endif - ) - { - #ifdef __linux__ - experimentalFeatureSettings.require(Xp::Cgroups); - - /* If we're running from the daemon, then this will return the - root cgroup of the service. Otherwise, it will return the - current cgroup. */ - auto rootCgroup = getRootCgroup(); - auto cgroupFS = getCgroupFS(); - if (!cgroupFS) - throw Error("cannot determine the cgroups file system"); - auto rootCgroupPath = canonPath(*cgroupFS + "/" + rootCgroup); - if (!pathExists(rootCgroupPath)) - throw Error("expected cgroup directory '%s'", rootCgroupPath); - - static std::atomic counter{0}; - - cgroup = buildUser - ? fmt("%s/nix-build-uid-%d", rootCgroupPath, buildUser->getUID()) - : fmt("%s/nix-build-pid-%d-%d", rootCgroupPath, getpid(), counter++); - - debug("using cgroup '%s'", *cgroup); - - /* When using a build user, record the cgroup we used for that - user so that if we got interrupted previously, we can kill - any left-over cgroup first. */ - if (buildUser) { - auto cgroupsDir = settings.nixStateDir + "/cgroups"; - createDirs(cgroupsDir); - - auto cgroupFile = fmt("%s/%d", cgroupsDir, buildUser->getUID()); - - if (pathExists(cgroupFile)) { - auto prevCgroup = readFile(cgroupFile); - destroyCgroup(prevCgroup); - } - - writeFile(cgroupFile, *cgroup); - } - - #else - throw Error("cgroups are not supported on this platform"); - #endif - } - /* Make sure that no other processes are executing under the sandbox uids. This must be done before any chownToBuilder() calls. */ - killSandbox(false); + prepareUser(); /* Right platform? */ if (!drvOptions.canBuildLocally(store, drv)) { diff --git a/src/libstore/unix/build/linux-derivation-builder.cc b/src/libstore/unix/build/linux-derivation-builder.cc index 5c6c80a0e72..8d4d973e18d 100644 --- a/src/libstore/unix/build/linux-derivation-builder.cc +++ b/src/libstore/unix/build/linux-derivation-builder.cc @@ -1,6 +1,10 @@ #ifdef __linux__ +# include "nix/store/personality.hh" +# include "nix/util/cgroup.hh" +# include "nix/util/namespaces.hh" # include "linux/fchmodat2-compat.hh" + # include # include # include @@ -9,13 +13,12 @@ # include # include # include -# include "nix/util/namespaces.hh" + # if HAVE_SECCOMP # include # endif + # define pivot_root(new_root, put_old) (syscall(SYS_pivot_root, new_root, put_old)) -# include "nix/util/cgroup.hh" -# include "nix/store/personality.hh" namespace nix { @@ -182,6 +185,11 @@ struct LinuxDerivationBuilder : DerivationBuilderImpl PathsInChroot pathsInChroot; + /** + * The cgroup of the builder, if any. + */ + std::optional cgroup; + LinuxDerivationBuilder( Store & store, std::unique_ptr miscMethods, DerivationBuilderParams params) : DerivationBuilderImpl(store, std::move(miscMethods), std::move(params)) @@ -235,6 +243,51 @@ struct LinuxDerivationBuilder : DerivationBuilderImpl return settings.sandboxBuildDir; } + void prepareUser() override + { + if ((buildUser && buildUser->getUIDCount() != 1) || settings.useCgroups) { + experimentalFeatureSettings.require(Xp::Cgroups); + + /* If we're running from the daemon, then this will return the + root cgroup of the service. Otherwise, it will return the + current cgroup. */ + auto rootCgroup = getRootCgroup(); + auto cgroupFS = getCgroupFS(); + if (!cgroupFS) + throw Error("cannot determine the cgroups file system"); + auto rootCgroupPath = canonPath(*cgroupFS + "/" + rootCgroup); + if (!pathExists(rootCgroupPath)) + throw Error("expected cgroup directory '%s'", rootCgroupPath); + + static std::atomic counter{0}; + + cgroup = buildUser ? fmt("%s/nix-build-uid-%d", rootCgroupPath, buildUser->getUID()) + : fmt("%s/nix-build-pid-%d-%d", rootCgroupPath, getpid(), counter++); + + debug("using cgroup '%s'", *cgroup); + + /* When using a build user, record the cgroup we used for that + user so that if we got interrupted previously, we can kill + any left-over cgroup first. */ + if (buildUser) { + auto cgroupsDir = settings.nixStateDir + "/cgroups"; + createDirs(cgroupsDir); + + auto cgroupFile = fmt("%s/%d", cgroupsDir, buildUser->getUID()); + + if (pathExists(cgroupFile)) { + auto prevCgroup = readFile(cgroupFile); + destroyCgroup(prevCgroup); + } + + writeFile(cgroupFile, *cgroup); + } + } + + // Kill any processes left in the cgroup or build user. + DerivationBuilderImpl::prepareUser(); + } + void prepareSandbox() override { /* Create a temporary directory in which we set up the chroot @@ -747,6 +800,20 @@ struct LinuxDerivationBuilder : DerivationBuilderImpl return DerivationBuilderImpl::unprepareBuild(); } + void killSandbox(bool getStats) override + { + if (cgroup) { + auto stats = destroyCgroup(*cgroup); + if (getStats) { + buildResult.cpuUser = stats.cpuUser; + buildResult.cpuSystem = stats.cpuSystem; + } + return; + } + + DerivationBuilderImpl::killSandbox(getStats); + } + void cleanupBuild() override { DerivationBuilderImpl::cleanupBuild(); From b04962b33b01313fca1b0b36226db214459475b3 Mon Sep 17 00:00:00 2001 From: gustavderdrache Date: Fri, 23 May 2025 16:49:23 -0400 Subject: [PATCH 088/218] Make platform checks throw BuildError like other failures Co-authored-by: Cole Helbling --- src/libstore/unix/build/derivation-builder.cc | 25 +++++++++++-------- 1 file changed, 15 insertions(+), 10 deletions(-) diff --git a/src/libstore/unix/build/derivation-builder.cc b/src/libstore/unix/build/derivation-builder.cc index 54173c16259..4a777f8e105 100644 --- a/src/libstore/unix/build/derivation-builder.cc +++ b/src/libstore/unix/build/derivation-builder.cc @@ -836,17 +836,22 @@ void DerivationBuilderImpl::startBuilder() /* Right platform? */ if (!drvOptions.canBuildLocally(store, drv)) { + auto msg = fmt( + "Cannot build '%s'.\n" + "Reason: " ANSI_RED "required system or feature not available" ANSI_NORMAL "\n" + "Required system: '%s' with features {%s}\n" + "Current system: '%s' with features {%s}", + Magenta(store.printStorePath(drvPath)), + Magenta(drv.platform), + concatStringsSep(", ", drvOptions.getRequiredSystemFeatures(drv)), + Magenta(settings.thisSystem), + concatStringsSep(", ", store.config.systemFeatures)); + // since aarch64-darwin has Rosetta 2, this user can actually run x86_64-darwin on their hardware - we should tell them to run the command to install Darwin 2 - if (drv.platform == "x86_64-darwin" && settings.thisSystem == "aarch64-darwin") { - throw Error("run `/usr/sbin/softwareupdate --install-rosetta` to enable your %s to run programs for %s", settings.thisSystem, drv.platform); - } else { - throw Error("a '%s' with features {%s} is required to build '%s', but I am a '%s' with features {%s}", - drv.platform, - concatStringsSep(", ", drvOptions.getRequiredSystemFeatures(drv)), - store.printStorePath(drvPath), - settings.thisSystem, - concatStringsSep(", ", store.config.systemFeatures)); - } + if (drv.platform == "x86_64-darwin" && settings.thisSystem == "aarch64-darwin") + msg += fmt("\nNote: run `%s` to run programs for x86_64-darwin", Magenta("/usr/sbin/softwareupdate --install-rosetta")); + + throw BuildError(msg); } /* Create a temporary directory where the build will take From ce89c8c114254aa558c314a19b730a53cfe17785 Mon Sep 17 00:00:00 2001 From: gustavderdrache Date: Tue, 20 May 2025 13:46:19 -0400 Subject: [PATCH 089/218] Log warnings on IFD with new option Co-authored-by: Eelco Dolstra --- src/libexpr/include/nix/expr/eval-settings.hh | 10 ++++++++++ src/libexpr/primops.cc | 18 +++++++++++++----- 2 files changed, 23 insertions(+), 5 deletions(-) diff --git a/src/libexpr/include/nix/expr/eval-settings.hh b/src/libexpr/include/nix/expr/eval-settings.hh index 284e02a9b36..9ff73138a81 100644 --- a/src/libexpr/include/nix/expr/eval-settings.hh +++ b/src/libexpr/include/nix/expr/eval-settings.hh @@ -152,6 +152,16 @@ struct EvalSettings : Config )" }; + Setting traceImportFromDerivation{ + this, false, "trace-import-from-derivation", + R"( + By default, Nix allows [Import from Derivation](@docroot@/language/import-from-derivation.md). + + When this setting is `true`, Nix will log a warning indicating that it performed such an import. + This option has no effect if `allow-import-from-derivation` is disabled. + )" + }; + Setting enableImportFromDerivation{ this, true, "allow-import-from-derivation", R"( diff --git a/src/libexpr/primops.cc b/src/libexpr/primops.cc index c4e6feb2800..f3e3e129071 100644 --- a/src/libexpr/primops.cc +++ b/src/libexpr/primops.cc @@ -90,11 +90,19 @@ StringMap EvalState::realiseContext(const NixStringContext & context, StorePathS if (drvs.empty()) return {}; - if (isIFD && !settings.enableImportFromDerivation) - error( - "cannot build '%1%' during evaluation because the option 'allow-import-from-derivation' is disabled", - drvs.begin()->to_string(*store) - ).debugThrow(); + if (isIFD) { + if (!settings.enableImportFromDerivation) + error( + "cannot build '%1%' during evaluation because the option 'allow-import-from-derivation' is disabled", + drvs.begin()->to_string(*store) + ).debugThrow(); + + if (settings.traceImportFromDerivation) + warn( + "built '%1%' during evaluation due to an import from derivation", + drvs.begin()->to_string(*store) + ); + } /* Build/substitute the context. */ std::vector buildReqs; From d80f0fb15ac2c6970af22b226d645137a0702236 Mon Sep 17 00:00:00 2001 From: gustavderdrache Date: Wed, 21 May 2025 11:11:09 -0400 Subject: [PATCH 090/218] Add test for output warning to ensure stability Co-authored-by: Eelco Dolstra --- tests/functional/flakes/meson.build | 1 + tests/functional/flakes/trace-ifd.sh | 33 ++++++++++++++++++++++++++++ 2 files changed, 34 insertions(+) create mode 100644 tests/functional/flakes/trace-ifd.sh diff --git a/tests/functional/flakes/meson.build b/tests/functional/flakes/meson.build index 213c388a6d9..801fefc6f9a 100644 --- a/tests/functional/flakes/meson.build +++ b/tests/functional/flakes/meson.build @@ -33,6 +33,7 @@ suites += { 'debugger.sh', 'source-paths.sh', 'old-lockfiles.sh', + 'trace-ifd.sh', ], 'workdir': meson.current_source_dir(), } diff --git a/tests/functional/flakes/trace-ifd.sh b/tests/functional/flakes/trace-ifd.sh new file mode 100644 index 00000000000..4879b97322e --- /dev/null +++ b/tests/functional/flakes/trace-ifd.sh @@ -0,0 +1,33 @@ +#!/usr/bin/env bash + +source ./common.sh + +requireGit + +flake1Dir="$TEST_ROOT/flake" + +createGitRepo "$flake1Dir" +createSimpleGitFlake "$flake1Dir" + +cat > "$flake1Dir/flake.nix" <<'EOF' +{ + outputs = { self }: let inherit (import ./config.nix) mkDerivation; in { + drv = mkDerivation { + name = "drv"; + buildCommand = '' + echo drv >$out + ''; + }; + + ifd = mkDerivation { + name = "ifd"; + buildCommand = '' + echo ${builtins.readFile self.drv} >$out + ''; + }; + }; +} +EOF + +nix build --no-link "$flake1Dir#ifd" --option trace-import-from-derivation true 2>&1 \ + | grepQuiet 'warning: built .* during evaluation due to an import from derivation' From f9fdf94e12b79fd75de3a7069456380976bf0e05 Mon Sep 17 00:00:00 2001 From: Eelco Dolstra Date: Tue, 27 May 2025 15:25:51 +0200 Subject: [PATCH 091/218] Fix macOS build --- .../unix/build/darwin-derivation-builder.cc | 71 +++++++++++-------- src/libstore/unix/build/derivation-builder.cc | 20 ++++-- 2 files changed, 54 insertions(+), 37 deletions(-) diff --git a/src/libstore/unix/build/darwin-derivation-builder.cc b/src/libstore/unix/build/darwin-derivation-builder.cc index 2ba54ad97d1..5e06dbe5563 100644 --- a/src/libstore/unix/build/darwin-derivation-builder.cc +++ b/src/libstore/unix/build/darwin-derivation-builder.cc @@ -14,11 +14,20 @@ struct DarwinDerivationBuilder : DerivationBuilderImpl { PathsInChroot pathsInChroot; + /** + * Whether full sandboxing is enabled. Note that macOS builds + * always have *some* sandboxing (see sandbox-minimal.sb). + */ + bool useSandbox; + DarwinDerivationBuilder( - Store & store, std::unique_ptr miscMethods, DerivationBuilderParams params) + Store & store, + std::unique_ptr miscMethods, + DerivationBuilderParams params, + bool useSandbox) : DerivationBuilderImpl(store, std::move(miscMethods), std::move(params)) + , useSandbox(useSandbox) { - useChroot = true; } void prepareSandbox() override @@ -26,32 +35,6 @@ struct DarwinDerivationBuilder : DerivationBuilderImpl pathsInChroot = getPathsInSandbox(); } - void execBuilder(const Strings & args, const Strings & envStrs) override - { - posix_spawnattr_t attrp; - - if (posix_spawnattr_init(&attrp)) - throw SysError("failed to initialize builder"); - - if (posix_spawnattr_setflags(&attrp, POSIX_SPAWN_SETEXEC)) - throw SysError("failed to initialize builder"); - - if (drv.platform == "aarch64-darwin") { - // Unset kern.curproc_arch_affinity so we can escape Rosetta - int affinity = 0; - sysctlbyname("kern.curproc_arch_affinity", NULL, NULL, &affinity, sizeof(affinity)); - - cpu_type_t cpu = CPU_TYPE_ARM64; - posix_spawnattr_setbinpref_np(&attrp, 1, &cpu, NULL); - } else if (drv.platform == "x86_64-darwin") { - cpu_type_t cpu = CPU_TYPE_X86_64; - posix_spawnattr_setbinpref_np(&attrp, 1, &cpu, NULL); - } - - posix_spawn( - NULL, drv.builder.c_str(), NULL, &attrp, stringsToCharPtrs(args).data(), stringsToCharPtrs(envStrs).data()); - } - void setUser() override { DerivationBuilderImpl::setUser(); @@ -59,7 +42,7 @@ struct DarwinDerivationBuilder : DerivationBuilderImpl /* This has to appear before import statements. */ std::string sandboxProfile = "(version 1)\n"; - if (useChroot) { + if (useSandbox) { /* Lots and lots and lots of file functions freak out if they can't stat their full ancestry */ PathSet ancestry; @@ -101,7 +84,7 @@ struct DarwinDerivationBuilder : DerivationBuilderImpl # include "sandbox-defaults.sb" ; - if (!derivationType->isSandboxed()) + if (!derivationType.isSandboxed()) sandboxProfile += # include "sandbox-network.sb" ; @@ -193,7 +176,33 @@ struct DarwinDerivationBuilder : DerivationBuilderImpl } } } -} + + void execBuilder(const Strings & args, const Strings & envStrs) override + { + posix_spawnattr_t attrp; + + if (posix_spawnattr_init(&attrp)) + throw SysError("failed to initialize builder"); + + if (posix_spawnattr_setflags(&attrp, POSIX_SPAWN_SETEXEC)) + throw SysError("failed to initialize builder"); + + if (drv.platform == "aarch64-darwin") { + // Unset kern.curproc_arch_affinity so we can escape Rosetta + int affinity = 0; + sysctlbyname("kern.curproc_arch_affinity", NULL, NULL, &affinity, sizeof(affinity)); + + cpu_type_t cpu = CPU_TYPE_ARM64; + posix_spawnattr_setbinpref_np(&attrp, 1, &cpu, NULL); + } else if (drv.platform == "x86_64-darwin") { + cpu_type_t cpu = CPU_TYPE_X86_64; + posix_spawnattr_setbinpref_np(&attrp, 1, &cpu, NULL); + } + + posix_spawn( + NULL, drv.builder.c_str(), NULL, &attrp, stringsToCharPtrs(args).data(), stringsToCharPtrs(envStrs).data()); + } +}; } diff --git a/src/libstore/unix/build/derivation-builder.cc b/src/libstore/unix/build/derivation-builder.cc index 142f75d1416..1eff3487b80 100644 --- a/src/libstore/unix/build/derivation-builder.cc +++ b/src/libstore/unix/build/derivation-builder.cc @@ -308,8 +308,6 @@ class DerivationBuilderImpl : public DerivationBuilder, public DerivationBuilder */ void runChild(); -private: - /** * Move the current process into the chroot, if any. Called early * by runChild(). @@ -330,6 +328,8 @@ class DerivationBuilderImpl : public DerivationBuilder, public DerivationBuilder */ virtual void execBuilder(const Strings & args, const Strings & envStrs); +private: + /** * Check that the derivation outputs all exist and register them * as valid. @@ -2119,7 +2119,7 @@ std::unique_ptr makeDerivationBuilder( throw Error("derivation '%s' has '__noChroot' set, " "but that's not allowed when 'sandbox' is 'true'", store.printStorePath(params.drvPath)); #ifdef __APPLE__ - if (drvOptions.additionalSandboxProfile != "") + if (params.drvOptions.additionalSandboxProfile != "") throw Error("derivation '%s' specifies a sandbox profile, " "but this is only allowed when 'sandbox' is 'relaxed'", store.printStorePath(params.drvPath)); #endif @@ -2158,16 +2158,24 @@ std::unique_ptr makeDerivationBuilder( std::move(params)); #endif - if (useSandbox) - throw Error("sandboxing builds is not supported on this platform"); - if (params.drvOptions.useUidRange(params.drv)) throw Error("feature 'uid-range' is only supported in sandboxed builds"); + #ifdef __APPLE__ + return std::make_unique( + store, + std::move(miscMethods), + std::move(params), + useSandbox); + #else + if (useSandbox) + throw Error("sandboxing builds is not supported on this platform"); + return std::make_unique( store, std::move(miscMethods), std::move(params)); + #endif } } From cf9d962086a2fe4d1125353d187fa2193f8075ca Mon Sep 17 00:00:00 2001 From: Eelco Dolstra Date: Tue, 27 May 2025 17:53:56 +0200 Subject: [PATCH 092/218] Remove unused variable --- src/libstore/unix/build/derivation-builder.cc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/libstore/unix/build/derivation-builder.cc b/src/libstore/unix/build/derivation-builder.cc index 1eff3487b80..a498a0937b4 100644 --- a/src/libstore/unix/build/derivation-builder.cc +++ b/src/libstore/unix/build/derivation-builder.cc @@ -800,7 +800,7 @@ void DerivationBuilderImpl::startBuilder() printMsg(lvlVomit, "setting builder env variable '%1%'='%2%'", i.first, i.second); /* Create the log file. */ - [[maybe_unused]] Path logFile = miscMethods->openLogFile(); + miscMethods->openLogFile(); /* Create a pseudoterminal to get the output of the builder. */ builderOut = posix_openpt(O_RDWR | O_NOCTTY); From 625dce659af3c332aa44ef494665d1cf5654efdd Mon Sep 17 00:00:00 2001 From: John Ericson Date: Tue, 27 May 2025 14:51:39 -0400 Subject: [PATCH 093/218] Prepare for FreeBSD sandboxing support This is the utility changes from #9968, which were easier to rebase first. I (@Ericson2314) didn't write this code; I just rebased it. Co-Authored-By: Artemis Tosini Co-Authored-By: Audrey Dutcher --- maintainers/flake-module.nix | 2 +- src/libstore/filetransfer.cc | 2 +- src/libstore/globals.cc | 2 +- src/libstore/include/nix/store/globals.hh | 2 + .../unix/build/linux-derivation-builder.cc | 2 +- src/libstore/unix/user-lock.cc | 2 +- src/libutil/current-process.cc | 25 ++++++- src/libutil/file-system.cc | 66 +++++++++++++++++-- src/libutil/freebsd/freebsd-jail.cc | 52 +++++++++++++++ .../freebsd/include/nix/util/freebsd-jail.hh | 20 ++++++ .../freebsd/include/nix/util/meson.build | 7 ++ src/libutil/freebsd/meson.build | 5 ++ src/libutil/include/nix/util/file-system.hh | 15 ++++- .../{namespaces.hh => linux-namespaces.hh} | 0 .../linux/include/nix/util/meson.build | 2 +- .../{namespaces.cc => linux-namespaces.cc} | 1 + src/libutil/linux/meson.build | 2 +- src/libutil/meson.build | 4 ++ src/nix/main.cc | 2 +- 19 files changed, 198 insertions(+), 15 deletions(-) create mode 100644 src/libutil/freebsd/freebsd-jail.cc create mode 100644 src/libutil/freebsd/include/nix/util/freebsd-jail.hh create mode 100644 src/libutil/freebsd/include/nix/util/meson.build create mode 100644 src/libutil/freebsd/meson.build rename src/libutil/linux/include/nix/util/{namespaces.hh => linux-namespaces.hh} (100%) rename src/libutil/linux/{namespaces.cc => linux-namespaces.cc} (99%) diff --git a/maintainers/flake-module.nix b/maintainers/flake-module.nix index 224f4726839..6d8633c4f8b 100644 --- a/maintainers/flake-module.nix +++ b/maintainers/flake-module.nix @@ -359,7 +359,7 @@ ''^src/libutil/json-utils\.cc$'' ''^src/libutil/include/nix/util/json-utils\.hh$'' ''^src/libutil/linux/cgroup\.cc$'' - ''^src/libutil/linux/namespaces\.cc$'' + ''^src/libutil/linux/linux-namespaces\.cc$'' ''^src/libutil/logging\.cc$'' ''^src/libutil/include/nix/util/logging\.hh$'' ''^src/libutil/memory-source-accessor\.cc$'' diff --git a/src/libstore/filetransfer.cc b/src/libstore/filetransfer.cc index 8080fcfddc5..7e29d00e6c0 100644 --- a/src/libstore/filetransfer.cc +++ b/src/libstore/filetransfer.cc @@ -14,7 +14,7 @@ #endif #ifdef __linux__ -# include "nix/util/namespaces.hh" +# include "nix/util/linux-namespaces.hh" #endif #include diff --git a/src/libstore/globals.cc b/src/libstore/globals.cc index c2ecc496494..de512834783 100644 --- a/src/libstore/globals.cc +++ b/src/libstore/globals.cc @@ -85,7 +85,7 @@ Settings::Settings() builders = concatStringsSep("\n", ss); } -#if defined(__linux__) && defined(SANDBOX_SHELL) +#if (defined(__linux__) || defined(__FreeBSD__)) && defined(SANDBOX_SHELL) sandboxPaths = tokenizeString("/bin/sh=" SANDBOX_SHELL); #endif diff --git a/src/libstore/include/nix/store/globals.hh b/src/libstore/include/nix/store/globals.hh index 3a677216a73..5fae2be2395 100644 --- a/src/libstore/include/nix/store/globals.hh +++ b/src/libstore/include/nix/store/globals.hh @@ -682,7 +682,9 @@ public: description of the `size` option of `tmpfs` in mount(8). The default is `50%`. )"}; +#endif +#if defined(__linux__) || defined(__FreeBSD__) Setting sandboxBuildDir{this, "/build", "sandbox-build-dir", R"( *Linux only* diff --git a/src/libstore/unix/build/linux-derivation-builder.cc b/src/libstore/unix/build/linux-derivation-builder.cc index 8d4d973e18d..efbe4c8bb3b 100644 --- a/src/libstore/unix/build/linux-derivation-builder.cc +++ b/src/libstore/unix/build/linux-derivation-builder.cc @@ -2,7 +2,7 @@ # include "nix/store/personality.hh" # include "nix/util/cgroup.hh" -# include "nix/util/namespaces.hh" +# include "nix/util/linux-namespaces.hh" # include "linux/fchmodat2-compat.hh" # include diff --git a/src/libstore/unix/user-lock.cc b/src/libstore/unix/user-lock.cc index ef806af79b7..6a07cb7cc83 100644 --- a/src/libstore/unix/user-lock.cc +++ b/src/libstore/unix/user-lock.cc @@ -197,7 +197,7 @@ bool useBuildUsers() #ifdef __linux__ static bool b = (settings.buildUsersGroup != "" || settings.autoAllocateUids) && isRootUser(); return b; - #elif defined(__APPLE__) + #elif defined(__APPLE__) && defined(__FreeBSD__) static bool b = settings.buildUsersGroup != "" && isRootUser(); return b; #else diff --git a/src/libutil/current-process.cc b/src/libutil/current-process.cc index 4cc5a4218be..1afefbcb25b 100644 --- a/src/libutil/current-process.cc +++ b/src/libutil/current-process.cc @@ -16,7 +16,12 @@ #ifdef __linux__ # include # include "nix/util/cgroup.hh" -# include "nix/util/namespaces.hh" +# include "nix/util/linux-namespaces.hh" +#endif + +#ifdef __FreeBSD__ +# include +# include #endif namespace nix { @@ -115,6 +120,24 @@ std::optional getSelfExe() return buf; else return std::nullopt; + #elif defined(__FreeBSD__) + int sysctlName[] = { + CTL_KERN, + KERN_PROC, + KERN_PROC_PATHNAME, + -1, + }; + size_t pathLen = 0; + if (sysctl(sysctlName, sizeof(sysctlName) / sizeof(sysctlName[0]), nullptr, &pathLen, nullptr, 0) < 0) { + return std::nullopt; + } + + std::vector path(pathLen); + if (sysctl(sysctlName, sizeof(sysctlName) / sizeof(sysctlName[0]), path.data(), &pathLen, nullptr, 0) < 0) { + return std::nullopt; + } + + return Path(path.begin(), path.end()); #else return std::nullopt; #endif diff --git a/src/libutil/file-system.cc b/src/libutil/file-system.cc index ad8cef38c8c..283d19fe669 100644 --- a/src/libutil/file-system.cc +++ b/src/libutil/file-system.cc @@ -23,6 +23,11 @@ #include +#ifdef __FreeBSD__ +# include +# include +#endif + #ifdef _WIN32 # include #endif @@ -364,6 +369,13 @@ void syncParent(const Path & path) fd.fsync(); } +#ifdef __FreeBSD__ +#define MOUNTEDPATHS_PARAM , std::set &mountedPaths +#define MOUNTEDPATHS_ARG , mountedPaths +#else +#define MOUNTEDPATHS_PARAM +#define MOUNTEDPATHS_ARG +#endif void recursiveSync(const Path & path) { @@ -410,11 +422,19 @@ void recursiveSync(const Path & path) } -static void _deletePath(Descriptor parentfd, const std::filesystem::path & path, uint64_t & bytesFreed) +static void _deletePath(Descriptor parentfd, const std::filesystem::path & path, uint64_t & bytesFreed MOUNTEDPATHS_PARAM) { #ifndef _WIN32 checkInterrupt(); +#ifdef __FreeBSD__ + // In case of emergency (unmount fails for some reason) not recurse into mountpoints. + // This prevents us from tearing up the nullfs-mounted nix store. + if (mountedPaths.find(path) != mountedPaths.end()) { + return; + } +#endif + std::string name(baseNameOf(path.native())); struct stat st; @@ -468,7 +488,7 @@ static void _deletePath(Descriptor parentfd, const std::filesystem::path & path, checkInterrupt(); std::string childName = dirent->d_name; if (childName == "." || childName == "..") continue; - _deletePath(dirfd(dir.get()), path + "/" + childName, bytesFreed); + _deletePath(dirfd(dir.get()), path + "/" + childName, bytesFreed MOUNTEDPATHS_ARG); } if (errno) throw SysError("reading directory %1%", path); } @@ -484,7 +504,7 @@ static void _deletePath(Descriptor parentfd, const std::filesystem::path & path, #endif } -static void _deletePath(const std::filesystem::path & path, uint64_t & bytesFreed) +static void _deletePath(const std::filesystem::path & path, uint64_t & bytesFreed MOUNTEDPATHS_PARAM) { Path dir = dirOf(path.string()); if (dir == "") @@ -496,7 +516,7 @@ static void _deletePath(const std::filesystem::path & path, uint64_t & bytesFree throw SysError("opening directory '%1%'", path); } - _deletePath(dirfd.get(), path, bytesFreed); + _deletePath(dirfd.get(), path, bytesFreed MOUNTEDPATHS_ARG); } @@ -529,8 +549,20 @@ void createDirs(const std::filesystem::path & path) void deletePath(const std::filesystem::path & path, uint64_t & bytesFreed) { //Activity act(*logger, lvlDebug, "recursively deleting path '%1%'", path); +#ifdef __FreeBSD__ + std::set mountedPaths; + struct statfs *mntbuf; + int count; + if ((count = getmntinfo(&mntbuf, MNT_WAIT)) < 0) { + throw SysError("getmntinfo"); + } + + for (int i = 0; i < count; i++) { + mountedPaths.emplace(mntbuf[i].f_mntonname); + } +#endif bytesFreed = 0; - _deletePath(path, bytesFreed); + _deletePath(path, bytesFreed MOUNTEDPATHS_ARG); } @@ -572,6 +604,30 @@ void AutoDelete::reset(const std::filesystem::path & p, bool recursive) { ////////////////////////////////////////////////////////////////////// +#ifdef __FreeBSD__ +AutoUnmount::AutoUnmount() : del{false} {} + +AutoUnmount::AutoUnmount(Path &p) : path(p), del(true) {} + +AutoUnmount::~AutoUnmount() +{ + try { + if (del) { + if (unmount(path.c_str(), 0) < 0) { + throw SysError("Failed to unmount path %1%", path); + } + } + } catch (...) { + ignoreExceptionInDestructor(); + } +} + +void AutoUnmount::cancel() +{ + del = false; +} +#endif + ////////////////////////////////////////////////////////////////////// std::string defaultTempDir() { diff --git a/src/libutil/freebsd/freebsd-jail.cc b/src/libutil/freebsd/freebsd-jail.cc new file mode 100644 index 00000000000..575f9287e82 --- /dev/null +++ b/src/libutil/freebsd/freebsd-jail.cc @@ -0,0 +1,52 @@ +#ifdef __FreeBSD__ +# include "nix/util/freebsd-jail.hh" + +# include +# include +# include +# include + +# include "nix/util/error.hh" +# include "nix/util/util.hh" + +namespace nix { + +AutoRemoveJail::AutoRemoveJail() + : del{false} +{ +} + +AutoRemoveJail::AutoRemoveJail(int jid) + : jid(jid) + , del(true) +{ +} + +AutoRemoveJail::~AutoRemoveJail() +{ + try { + if (del) { + if (jail_remove(jid) < 0) { + throw SysError("Failed to remove jail %1%", jid); + } + } + } catch (...) { + ignoreExceptionInDestructor(); + } +} + +void AutoRemoveJail::cancel() +{ + del = false; +} + +void AutoRemoveJail::reset(int j) +{ + del = true; + jid = j; +} + +////////////////////////////////////////////////////////////////////// + +} +#endif diff --git a/src/libutil/freebsd/include/nix/util/freebsd-jail.hh b/src/libutil/freebsd/include/nix/util/freebsd-jail.hh new file mode 100644 index 00000000000..cb5abc511a5 --- /dev/null +++ b/src/libutil/freebsd/include/nix/util/freebsd-jail.hh @@ -0,0 +1,20 @@ +#pragma once +///@file + +#include "nix/util/types.hh" + +namespace nix { + +class AutoRemoveJail +{ + int jid; + bool del; +public: + AutoRemoveJail(int jid); + AutoRemoveJail(); + ~AutoRemoveJail(); + void cancel(); + void reset(int j); +}; + +} diff --git a/src/libutil/freebsd/include/nix/util/meson.build b/src/libutil/freebsd/include/nix/util/meson.build new file mode 100644 index 00000000000..561c8796c15 --- /dev/null +++ b/src/libutil/freebsd/include/nix/util/meson.build @@ -0,0 +1,7 @@ +# Public headers directory + +include_dirs += include_directories('../..') + +headers += files( + 'freebsd-jail.hh', +) diff --git a/src/libutil/freebsd/meson.build b/src/libutil/freebsd/meson.build new file mode 100644 index 00000000000..8ffdc283292 --- /dev/null +++ b/src/libutil/freebsd/meson.build @@ -0,0 +1,5 @@ +sources += files( + 'freebsd-jail.cc', +) + +subdir('include/nix/util') diff --git a/src/libutil/include/nix/util/file-system.hh b/src/libutil/include/nix/util/file-system.hh index b4cc1567d5d..0121745ab0e 100644 --- a/src/libutil/include/nix/util/file-system.hh +++ b/src/libutil/include/nix/util/file-system.hh @@ -310,7 +310,7 @@ typedef std::unique_ptr AutoCloseDir; /** * Create a temporary directory. */ -Path createTempDir(const Path & tmpRoot = "", const Path & prefix = "nix", +Path createTempDir(const Path & tmpRoot = "", const Path & prefix = "nix", mode_t mode = 0755); /** @@ -420,4 +420,17 @@ private: std::filesystem::directory_iterator it_; }; +#ifdef __FreeBSD__ +class AutoUnmount +{ + Path path; + bool del; +public: + AutoUnmount(Path&); + AutoUnmount(); + ~AutoUnmount(); + void cancel(); +}; +#endif + } diff --git a/src/libutil/linux/include/nix/util/namespaces.hh b/src/libutil/linux/include/nix/util/linux-namespaces.hh similarity index 100% rename from src/libutil/linux/include/nix/util/namespaces.hh rename to src/libutil/linux/include/nix/util/linux-namespaces.hh diff --git a/src/libutil/linux/include/nix/util/meson.build b/src/libutil/linux/include/nix/util/meson.build index 9587aa9166e..e28ad8e0595 100644 --- a/src/libutil/linux/include/nix/util/meson.build +++ b/src/libutil/linux/include/nix/util/meson.build @@ -4,5 +4,5 @@ include_dirs += include_directories('../..') headers += files( 'cgroup.hh', - 'namespaces.hh', + 'linux-namespaces.hh', ) diff --git a/src/libutil/linux/namespaces.cc b/src/libutil/linux/linux-namespaces.cc similarity index 99% rename from src/libutil/linux/namespaces.cc rename to src/libutil/linux/linux-namespaces.cc index 405866c0b56..93f299076a8 100644 --- a/src/libutil/linux/namespaces.cc +++ b/src/libutil/linux/linux-namespaces.cc @@ -1,3 +1,4 @@ +#include "nix/util/linux-namespaces.hh" #include "nix/util/current-process.hh" #include "nix/util/util.hh" #include "nix/util/finally.hh" diff --git a/src/libutil/linux/meson.build b/src/libutil/linux/meson.build index bfda8b1a6ac..b8053a5bb03 100644 --- a/src/libutil/linux/meson.build +++ b/src/libutil/linux/meson.build @@ -1,6 +1,6 @@ sources += files( 'cgroup.cc', - 'namespaces.cc', + 'linux-namespaces.cc', ) subdir('include/nix/util') diff --git a/src/libutil/meson.build b/src/libutil/meson.build index 04ca06eee0c..f5ad2b1f680 100644 --- a/src/libutil/meson.build +++ b/src/libutil/meson.build @@ -169,6 +169,10 @@ if host_machine.system() == 'linux' subdir('linux') endif +if host_machine.system() == 'freebsd' + subdir('freebsd') +endif + if host_machine.system() == 'windows' subdir('windows') else diff --git a/src/nix/main.cc b/src/nix/main.cc index 0e5ccf34b40..6144f746ffe 100644 --- a/src/nix/main.cc +++ b/src/nix/main.cc @@ -38,7 +38,7 @@ #endif #ifdef __linux__ -# include "nix/util/namespaces.hh" +# include "nix/util/linux-namespaces.hh" #endif #ifndef _WIN32 From d555d6b404124dbc95bfced3dc85cb321a3baae4 Mon Sep 17 00:00:00 2001 From: Sergei Zimmerman Date: Tue, 27 May 2025 22:28:13 +0000 Subject: [PATCH 094/218] tests/functional: Add more language tests for builtins.match These tests have been collected from nixpkgs f870c6ccc8951fc48aeb293cf3e98ade6ac42668 usage of builtins.match for x86_64-linux eval system. At most 2 matching and non-matching cases are included for each encountered regex. This should hopefully add more confidence when possibly trying to switch the regex implementation in the future. --- .../lang/eval-okay-regex-match2.exp | 1 + .../lang/eval-okay-regex-match2.nix | 938 ++++++++++++++++++ 2 files changed, 939 insertions(+) create mode 100644 tests/functional/lang/eval-okay-regex-match2.exp create mode 100644 tests/functional/lang/eval-okay-regex-match2.nix diff --git a/tests/functional/lang/eval-okay-regex-match2.exp b/tests/functional/lang/eval-okay-regex-match2.exp new file mode 100644 index 00000000000..b7fb4e05e16 --- /dev/null +++ b/tests/functional/lang/eval-okay-regex-match2.exp @@ -0,0 +1 @@ +[ null null null null null null null null null null [ ] [ ] null null null null [ "gnu" "m4/m4-1.4.19.tar.bz2" ] null [ "cpan" "src/5.0/perl-5.40.0.tar.gz" ] null null null [ "10" "" ] [ "11" "" ] [ "36" ] null [ "exec" ] [ ] null [ "26" ] null [ "26" ] null [ ] null null null null null [ "meson.patch?h=mingw-w64-xorgproto&id=7b817efc3144a50e6766817c4ca7242f8ce49307" ] null null [ "xmlto" ] null null [ "exec" ] null null [ ] [ ] null null [ "coconutbattery-4.0.2,152" ] [ "12" "0" ] [ "12" "8" ] [ "8" "9" "5" "30" ] [ "9" "7" "1" "26" ] null null [ ] [ ] null [ ] null [ ] null null [ null null "draupnir" ] [ ] [ ] null null [ null null "renderer" ] [ ] [ ] [ null ] null [ null ] null null null [ ] [ ] [ "p" ] [ "p" ] [ "systemtap" ] null [ ] null null [ ] null [ "20220722-71c783507536-b7eae18423ef" ] [ "20220726-bac6d66b5ca1-5b966f2f136c" ] [ ] [ "0.3.2308" ] null null null [ "17.0.14+" "7" ] null [ null ] [ null ] null null [ "21.0.7+" "6" ] null null [ ] [ ] [ "8u442" "06" ] [ ] [ "jna" "5.6.0" null null ] [ "jna" "5.6.0" null null ] [ ] [ ] null [ "2" ] null [ ] [ ] null null [ ] [ ] null null [ ] null [ ] [ "https://github.com/GRA0007/google-cloud-rs.git" null null null "4a2db92efd57a896e14d18877458c6ae43418aec" ] [ "https://github.com/GRA0007/google-cloud-rs.git" null null null "4a2db92efd57a896e14d18877458c6ae43418aec" ] null [ ] null [ "rejeep" "ansi.el" ] null [ "rejeep" "commander.el" ] null [ "2.2.4" "20231021.200112" "6" ] [ "2.2.4" "20231021.200112" "6" ] [ ] [ ] null null [ "" ] [ "" ".git" ] [ "" "\\.git" ] null null null [ "" "__pycache__" ] [ "" "__pycache__" ] null null null [ "" ] null null [ ] [ ] [ ] [ "8u442" "06" ] [ ] [ ] [ "simulator" ] null null null null null null [ "notify-send" ] [ "playlistmanager" ] [ ] [ ] null null null null [ "name" ] [ "name" ] null null null null [ "pypy" "27" ] [ "pypy" "310" ] [ "refs/heads/master" ] null [ "refs/heads/master" ] null null [ ] null null null null [ ] [ ] [ ] [ ] [ "b7eae18423ef" ] [ "20220726-bac6d66b5ca1-5b966f2f136c" ] null null [ ] [ ] [ ] [ ] [ ] [ ] [ ] [ ] [ ] [ ] [ "" ] [ ] ] diff --git a/tests/functional/lang/eval-okay-regex-match2.nix b/tests/functional/lang/eval-okay-regex-match2.nix new file mode 100644 index 00000000000..31a94423d86 --- /dev/null +++ b/tests/functional/lang/eval-okay-regex-match2.nix @@ -0,0 +1,938 @@ +# Derived from nixpkgs f870c6ccc8951fc48aeb293cf3e98ade6ac42668 by instrumenting +# builtins.match to collect at most 2 non-matching and 2 matching cases of every +# regex used when running: +# `nix-env --query --available --out-path --eval-system x86_64-linux`. + +builtins.map + ( + list: + let + re = builtins.head list; + str = builtins.elemAt list 1; + in + builtins.match re str + ) + [ + [ + ''(.*)e?abi.*'' + ''linux'' + ] + [ + ''(.*)e?abi.*'' + ''linux'' + ] + [ + ''.*-none.*'' + ''x86_64-unknown-linux-gnu'' + ] + [ + ''.*nvptx.*'' + ''x86_64-unknown-linux-gnu'' + ] + [ + ''.*switch.*'' + ''x86_64-unknown-linux-gnu'' + ] + [ + ''.*-uefi.*'' + ''x86_64-unknown-linux-gnu'' + ] + [ + ''.*-none.*'' + ''x86_64-unknown-linux-gnu'' + ] + [ + ''.*nvptx.*'' + ''x86_64-unknown-linux-gnu'' + ] + [ + ''.*switch.*'' + ''x86_64-unknown-linux-gnu'' + ] + [ + ''.*-uefi.*'' + ''x86_64-unknown-linux-gnu'' + ] + [ + ''[[:alnum:]+_?=-][[:alnum:]+._?=-]*'' + ''bootstrap-stage0-glibc-bootstrapFiles'' + ] + [ + ''[[:alnum:]+_?=-][[:alnum:]+._?=-]*'' + ''glibc-2.40-66'' + ] + [ + ''mirror://([a-z]+)/(.*)'' + ''https://github.com/madler/zlib/releases/download/v1.3.1/zlib-1.3.1.tar.gz'' + ] + [ + ''mirror://([a-z]+)/(.*)'' + ''https://git.savannah.gnu.org/cgit/config.git/plain/config.guess?id=948ae97ca5703224bd3eada06b7a69f40dd15a02'' + ] + [ + ''.*/.*'' + ''mktemp'' + ] + [ + ''.*/.*'' + ''rm'' + ] + [ + ''mirror://([a-z]+)/(.*)'' + ''mirror://gnu/m4/m4-1.4.19.tar.bz2'' + ] + [ + ''5\.[0-9]*[13579]\..+'' + ''5.40.0'' + ] + [ + ''mirror://([a-z]+)/(.*)'' + ''mirror://cpan/src/5.0/perl-5.40.0.tar.gz'' + ] + [ + ''5\.[0-9]*[13579]\..+'' + ''5.40.0'' + ] + [ + ''^([0-9][0-9\.]*)(.*)$'' + ''addons'' + ] + [ + ''^([0-9][0-9\.]*)(.*)$'' + ''extras'' + ] + [ + ''^([0-9][0-9\.]*)(.*)$'' + ''10'' + ] + [ + ''^([0-9][0-9\.]*)(.*)$'' + ''11'' + ] + [ + ''[[:space:]]*0*(-?[[:digit:]]+)[[:space:]]*'' + ''36'' + ] + [ + ''0+'' + ''36'' + ] + [ + ''/bin/([^/]+)'' + ''/bin/exec'' + ] + [ + ''[[:alnum:],._+:@%/-]+'' + ''/bin/exec'' + ] + [ + ''[[:alnum:],._+:@%/-]+'' + '''' + ] + [ + ''[[:space:]]*(-?[[:digit:]]+)[[:space:]]*'' + ''26'' + ] + [ + ''0[[:digit:]]+'' + ''26'' + ] + [ + ''[[:space:]]*(-?[[:digit:]]+)[[:space:]]*'' + ''26'' + ] + [ + ''0[[:digit:]]+'' + ''26'' + ] + [ + ''[[:alnum:],._+:@%/-]+'' + ''@tcl@'' + ] + [ + ''[[:alnum:],._+:@%/-]+'' + ''@[a-zA-Z_][0-9A-Za-z_'-]*@'' + ] + [ + ''.*pypy.*'' + ''/nix/store/8w718rm43x7z73xhw9d6vh8s4snrq67h-python3-3.12.10/bin/python3.12'' + ] + [ + ''(.*/)?\.\.(/.*)?'' + ''package.nix'' + ] + [ + ''/bin/([^/]+)'' + '''' + ] + [ + ''[[:alnum:]+_?=-][[:alnum:]+._?=-]*'' + ''meson.patch?h=mingw-w64-xorgproto&id=7b817efc3144a50e6766817c4ca7242f8ce49307'' + ] + [ + ''\.*(.*)'' + ''meson.patch?h=mingw-w64-xorgproto&id=7b817efc3144a50e6766817c4ca7242f8ce49307'' + ] + [ + ''/bin/([^/]+)'' + '''' + ] + [ + ''.*-rc.*'' + ''2.49.0'' + ] + [ + ''(.*)\.git'' + ''xmlto.git'' + ] + [ + ''[a-f0-9]*'' + ''0.0.29'' + ] + [ + ''.*-rc.*'' + ''2.49.0'' + ] + [ + ''/bin/([^/]+)'' + ''/bin/exec'' + ] + [ + ''.*-polly.*'' + ''/nix/store/0yxfdnfxbzczjxhgdpac81jnas194wfj-gnu-install-dirs.patch'' + ] + [ + ''.*-polly.*'' + ''/nix/store/jh2pda7psaasq85b2rrigmkjdbl8d0a1-llvm-lit-cfg-add-libs-to-dylib-path.patch'' + ] + [ + ''.*-polly.*'' + ''/nix/store/x868j4ih7wqiivf6wr9m4g424jav0hpq-gnu-install-dirs-polly.patch'' + ] + [ + ''.*-polly.*'' + ''/nix/store/gr73nf6sca9nyzl88x58y3qxrav04yhd-polly-lit-cfg-add-libs-to-dylib-path.patch'' + ] + [ + ''(.*/)?\.\.(/.*)?'' + ''package.nix'' + ] + [ + ''[[:alnum:]+_?=-][[:alnum:]+._?=-]*'' + ''coconutbattery-4.0.2,152'' + ] + [ + ''\.*(.*)'' + ''coconutbattery-4.0.2,152'' + ] + [ + ''^([[:digit:]]+)\.([[:digit:]]+)$'' + ''12.0'' + ] + [ + ''^([[:digit:]]+)\.([[:digit:]]+)$'' + ''12.8'' + ] + [ + ''^([[:digit:]]+)\.([[:digit:]]+)\.([[:digit:]]+)\.([[:digit:]]+)$'' + ''8.9.5.30'' + ] + [ + ''^([[:digit:]]+)\.([[:digit:]]+)\.([[:digit:]]+)\.([[:digit:]]+)$'' + ''9.7.1.26'' + ] + [ + ''^/.*'' + ''8.20'' + ] + [ + ''^/.*'' + ''8.20'' + ] + [ + ''^github.*'' + ''github.com'' + ] + [ + ''^github.*'' + ''github.com'' + ] + [ + ''^github.*'' + ''gitlab.inria.fr'' + ] + [ + ''^gitlab.*'' + ''gitlab.inria.fr'' + ] + [ + ''^github.*'' + ''gitlab.inria.fr'' + ] + [ + ''^gitlab.*'' + ''gitlab.inria.fr'' + ] + [ + ''^gitlab.*'' + ''sf.snu.ac.kr'' + ] + [ + ''^gitlab.*'' + ''sf.snu.ac.kr'' + ] + [ + ''^(@([^/]+)/)?([^/]+)$'' + ''draupnir'' + ] + [ + ''^[[:digit:]].*'' + ''0xproto'' + ] + [ + ''^[[:digit:]].*'' + ''3270'' + ] + [ + ''^[[:digit:]].*'' + ''adwaita-mono'' + ] + [ + ''^[[:digit:]].*'' + ''agave'' + ] + [ + ''^(@([^/]+)/)?([^/]+)$'' + ''renderer'' + ] + [ + ''[^[:space:]]*'' + ''900,906,908,1010,1012,1030'' + ] + [ + ''[^[:space:]]*'' + '''' + ] + [ + ''.*[0-9]_LIN(UX)?.sh'' + ''Wolfram_14.2.1_LIN.sh'' + ] + [ + ''.*[0-9]_LIN(UX)?.sh'' + ''Wolfram_14.2.1_LIN_Bndl.sh'' + ] + [ + ''.*[0-9]_LIN(UX)?.sh'' + ''Wolfram_14.2.0_LIN.sh'' + ] + [ + ''.*[0-9]_LIN(UX)?.sh'' + ''Wolfram_14.2.0_LIN_Bndl.sh'' + ] + [ + ''[A-Z]'' + ''b'' + ] + [ + ''[A-Z]'' + ''l'' + ] + [ + ''[A-Z]'' + ''E'' + ] + [ + ''[A-Z]'' + ''T'' + ] + [ + ''([0-9A-Za-z._])[0-9A-Za-z._-]*'' + ''pythoncheck.sh'' + ] + [ + ''([0-9A-Za-z._])[0-9A-Za-z._-]*'' + ''pythoncheck.sh'' + ] + [ + ''(.*)\.git'' + ''systemtap.git'' + ] + [ + ''[a-f0-9]*'' + ''release-5.2'' + ] + [ + ''[a-f0-9]*'' + ''b7a857659f8485ee3c6769c27a3e74b0af910746'' + ] + [ + ''.*pypy.*'' + ''/nix/store/8w718rm43x7z73xhw9d6vh8s4snrq67h-python3-3.12.10/bin/python3.12'' + ] + [ + ''(.*)\.git'' + ''gn'' + ] + [ + ''[a-f0-9]*'' + ''df98b86690c83b81aedc909ded18857296406159'' + ] + [ + ''.*-rc\..*'' + ''22.14.0'' + ] + [ + ''.*/linux-gecko-(.*).tar.bz2'' + ''https://static.replay.io/downloads/linux-gecko-20220722-71c783507536-b7eae18423ef.tar.bz2'' + ] + [ + ''.*/linux-node-(.*)'' + ''https://static.replay.io/downloads/linux-node-20220726-bac6d66b5ca1-5b966f2f136c'' + ] + [ + ''.*-DSQLITE_ENABLE_FTS3.*'' + ''-DSQLITE_ENABLE_COLUMN_METADATA -DSQLITE_ENABLE_DBSTAT_VTAB -DSQLITE_ENABLE_JSON1 -DSQLITE_ENABLE_FTS3 -DSQLITE_ENABLE_FTS3_PARENTHESIS -DSQLITE_ENABLE_FTS3_TOKENIZER -DSQLITE_ENABLE_FTS4 -DSQLITE_ENABLE_FTS5 -DSQLITE_ENABLE_PREUPDATE_HOOK -DSQLITE_ENABLE_RTREE -DSQLITE_ENABLE_SESSION -DSQLITE_ENABLE_STMT_SCANSTATUS -DSQLITE_ENABLE_UNLOCK_NOTIFY -DSQLITE_SOUNDEX -DSQLITE_SECURE_DELETE -DSQLITE_MAX_VARIABLE_NUMBER=250000 -DSQLITE_MAX_EXPR_DEPTH=10000'' + ] + [ + '' + [ + ]*(.*[^ + ])[ + ]*'' + '' + 0.3.2308 + '' + ] + [ + ''(.*)\.git'' + ''rtmpdump'' + ] + [ + ''.*;.*'' + ''Game'' + ] + [ + ''.*;.*'' + ''Game'' + ] + [ + ''(.+)+(.+)'' + ''17.0.14+7'' + ] + [ + ''^#(.*)$'' + ''20240715'' + ] + [ + ''[[:alpha:]_][[:alnum:]_]*(\.[[:alpha:]_][[:alnum:]_]*)*'' + ''external_deps_dirs'' + ] + [ + ''[[:alpha:]_][[:alnum:]_]*(\.[[:alpha:]_][[:alnum:]_]*)*'' + ''local_cache'' + ] + [ + ''^#(.*)$'' + ''20240715'' + ] + [ + ''.*-rc\..*'' + ''20.19.2'' + ] + [ + ''(.+)+(.+)'' + ''21.0.7+6'' + ] + [ + ''.*llvm-tblgen.*'' + ''-DLLVM_INSTALL_PACKAGE_DIR:STRING=/02qcpld1y6xhs5gz9bchpxaw0xdhmsp5dv88lh25r2ss44kh8dxz/lib/cmake/llvm'' + ] + [ + ''.*llvm-tblgen.*'' + ''-DLLVM_ENABLE_RTTI:BOOL=TRUE'' + ] + [ + ''.*llvm-tblgen.*'' + ''-DLLVM_TABLEGEN:STRING=/nix/store/xp9hkw8nsw9p81d69yvcg1yr6f7vh71c-llvm-tblgen-18.1.8/bin/llvm-tblgen'' + ] + [ + ''.*llvm-tblgen.*'' + ''-DLLVM_TABLEGEN_EXE:STRING=/nix/store/xp9hkw8nsw9p81d69yvcg1yr6f7vh71c-llvm-tblgen-18.1.8/bin/llvm-tblgen'' + ] + [ + ''(.+)-b(.+)'' + ''8u442-b06'' + ] + [ + ''.*-DSQLITE_ENABLE_FTS3.*'' + ''-DSQLITE_ENABLE_COLUMN_METADATA -DSQLITE_ENABLE_DBSTAT_VTAB -DSQLITE_ENABLE_JSON1 -DSQLITE_ENABLE_FTS3 -DSQLITE_ENABLE_FTS3_PARENTHESIS -DSQLITE_ENABLE_FTS3_TOKENIZER -DSQLITE_ENABLE_FTS4 -DSQLITE_ENABLE_FTS5 -DSQLITE_ENABLE_PREUPDATE_HOOK -DSQLITE_ENABLE_RTREE -DSQLITE_ENABLE_SESSION -DSQLITE_ENABLE_STMT_SCANSTATUS -DSQLITE_ENABLE_UNLOCK_NOTIFY -DSQLITE_SOUNDEX -DSQLITE_SECURE_DELETE -DSQLITE_MAX_VARIABLE_NUMBER=250000 -DSQLITE_MAX_EXPR_DEPTH=10000'' + ] + [ + ''([^/]*)/([^/]*)(/SNAPSHOT)?(/.*)?'' + ''jna/5.6.0'' + ] + [ + ''([^/]*)/([^/]*)(/SNAPSHOT)?(/.*)?'' + ''jna/5.6.0'' + ] + [ + ''[0-9]+'' + ''2'' + ] + [ + ''[0-9]+'' + ''3'' + ] + [ + ''[0-9]+'' + ''unstable'' + ] + [ + ''[[:space:]]*0*(-?[[:digit:]]+)[[:space:]]*'' + ''2'' + ] + [ + ''0+'' + ''2'' + ] + [ + ''0+'' + ''0'' + ] + [ + ''.*org/eclipse/jdt/ecj.*'' + ''https://repo.maven.apache.org/maven2/org/eclipse/jdt/ecj/maven-metadata.xml'' + ] + [ + ''.*[<>"'&].*'' + ''org.eclipse.jdt'' + ] + [ + ''.*[<>"'&].*'' + ''20241203050026'' + ] + [ + ''[a-zA-Z_][a-zA-Z0-9_'-]*'' + ''cpu'' + ] + [ + ''[a-zA-Z_][a-zA-Z0-9_'-]*'' + ''bits'' + ] + [ + ''armv[67]l-linux'' + ''x86_64-linux'' + ] + [ + ''armv[67]l-linux'' + ''x86_64-linux'' + ] + [ + ''0+'' + ''0'' + ] + [ + ''[0-9]+'' + ''rc'' + ] + [ + ''.*tensorflow_cpu.*'' + ''https://storage.googleapis.com/tensorflow/versions/2.19.0/tensorflow_cpu-2.19.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl'' + ] + [ + ''git\+([^?]+)(\?(rev|tag|branch)=(.*))?#(.*)'' + ''git+https://github.com/GRA0007/google-cloud-rs.git#4a2db92efd57a896e14d18877458c6ae43418aec'' + ] + [ + ''git\+([^?]+)(\?(rev|tag|branch)=(.*))?#(.*)'' + ''git+https://github.com/GRA0007/google-cloud-rs.git#4a2db92efd57a896e14d18877458c6ae43418aec'' + ] + [ + ''mpv[-_](.*)'' + ''detect-image'' + ] + [ + ''.*org/bouncycastle/bcutil-lts8on.*'' + ''https://plugins.gradle.org/m2/org/bouncycastle/bcutil-lts8on/maven-metadata.xml'' + ] + [ + ''^.*-unstable-([[:digit:]]{4})-([[:digit:]]{2})-([[:digit:]]{2})$'' + ''0.9.0'' + ] + [ + ''(.+)/(.+)'' + ''rejeep/ansi.el'' + ] + [ + ''^.*-unstable-([[:digit:]]{4})-([[:digit:]]{2})-([[:digit:]]{2})$'' + ''20230306.1823'' + ] + [ + ''(.+)/(.+)'' + ''rejeep/commander.el'' + ] + [ + ''mpv[-_](.*)'' + ''equalizer'' + ] + [ + ''(.*)-([^-]*)-([^-]*)'' + ''2.2.4-20231021.200112-6'' + ] + [ + ''(.*)-([^-]*)-([^-]*)'' + ''2.2.4-20231021.200112-6'' + ] + [ + ''.*com/badlogicgames/gdx-controllers/gdx-controllers-core.*'' + ''https://oss.sonatype.org/content/repositories/snapshots/com/badlogicgames/gdx-controllers/gdx-controllers-core/2.2.4-SNAPSHOT/maven-metadata.xml'' + ] + [ + ''.*com/badlogicgames/gdx-controllers/gdx-controllers-desktop.*'' + ''https://oss.sonatype.org/content/repositories/snapshots/com/badlogicgames/gdx-controllers/gdx-controllers-desktop/2.2.4-SNAPSHOT/maven-metadata.xml'' + ] + [ + ''^(#.*|$)'' + ''.git'' + ] + [ + ''^(#.*|$)'' + ''__pycache__'' + ] + [ + ''^(#.*|$)'' + '''' + ] + [ + ''^(!?)(.*)'' + ''.git'' + ] + [ + ''^(/?)(.*)'' + ''\.git'' + ] + [ + ''.+/.+'' + ''\.git'' + ] + [ + ''^(.*)/$'' + ''(^|.*/)\.git'' + ] + [ + ''(^|.*/)\.git'' + ''.flake8'' + ] + [ + ''^(!?)(.*)'' + ''__pycache__'' + ] + [ + ''^(/?)(.*)'' + ''__pycache__'' + ] + [ + ''.+/.+'' + ''__pycache__'' + ] + [ + ''^(.*)/$'' + ''(^|.*/)__pycache__'' + ] + [ + ''(^|.*/)__pycache__'' + ''.flake8'' + ] + [ + ''^(#.*|$)'' + '''' + ] + [ + ''(^|.*/)\.git'' + ''.gitignore'' + ] + [ + ''(^|.*/)__pycache__'' + ''.gitignore'' + ] + [ + ''^[a-fA-F0-9]{40}$'' + ''3a667bdb3d7f0955a5a51c8468eac83210c1439e'' + ] + [ + ''.*com/android/tools/build/gradle.*'' + ''https://repo.maven.apache.org/maven2/com/android/tools/build/gradle/maven-metadata.xml'' + ] + [ + ''^[a-fA-F0-9]{40}$'' + ''dc0a228a5544988d4a920cfb40be9cd28db41423'' + ] + [ + ''(.+)-b(.+)'' + ''8u442-b06'' + ] + [ + ''.*com/tobiasdiez/easybind.*'' + ''https://oss.sonatype.org/content/groups/public/com/tobiasdiez/easybind/2.2.1-SNAPSHOT/maven-metadata.xml'' + ] + [ + ''.*org/hamcrest/hamcrest.*'' + ''https://repo.maven.apache.org/maven2/org/hamcrest/hamcrest/maven-metadata.xml'' + ] + [ + ''^.*CONFIG_BOARD_DIRECTORY="([a-zA-Z0-9_]+)".*$'' + '' + # CONFIG_LOW_LEVEL_OPTIONS is not set + # CONFIG_MACH_AVR is not set + # CONFIG_MACH_ATSAM is not set + # CONFIG_MACH_ATSAMD is not set + # CONFIG_MACH_LPC176X is not set + # CONFIG_MACH_STM32 is not set + # CONFIG_MACH_HC32F460 is not set + # CONFIG_MACH_RPXXXX is not set + # CONFIG_MACH_PRU is not set + # CONFIG_MACH_AR100 is not set + # CONFIG_MACH_LINUX is not set + CONFIG_MACH_SIMU=y + CONFIG_BOARD_DIRECTORY="simulator" + CONFIG_CLOCK_FREQ=20000000 + CONFIG_SERIAL=y + CONFIG_SIMULATOR_SELECT=y + CONFIG_SERIAL_BAUD=250000 + CONFIG_USB_VENDOR_ID=0x1d50 + CONFIG_USB_DEVICE_ID=0x614e + CONFIG_USB_SERIAL_NUMBER="12345" + CONFIG_WANT_ADC=y + CONFIG_WANT_SPI=y + CONFIG_WANT_SOFTWARE_SPI=y + CONFIG_WANT_HARD_PWM=y + CONFIG_WANT_BUTTONS=y + CONFIG_WANT_TMCUART=y + CONFIG_WANT_NEOPIXEL=y + CONFIG_WANT_PULSE_COUNTER=y + CONFIG_WANT_ST7920=y + CONFIG_WANT_HD44780=y + CONFIG_WANT_ADXL345=y + CONFIG_WANT_LIS2DW=y + CONFIG_WANT_THERMOCOUPLE=y + CONFIG_WANT_HX71X=y + CONFIG_WANT_ADS1220=y + CONFIG_WANT_SENSOR_ANGLE=y + CONFIG_NEED_SENSOR_BULK=y + CONFIG_CANBUS_FREQUENCY=1000000 + CONFIG_INLINE_STEPPER_HACK=y + CONFIG_HAVE_GPIO=y + CONFIG_HAVE_GPIO_ADC=y + CONFIG_HAVE_GPIO_SPI=y + CONFIG_HAVE_GPIO_HARD_PWM=y + '' + ] + [ + ''[^.]*[.][^.]*-.*'' + ''5.15.183-rt85'' + ] + [ + ''[^.]*[.][^.]*-.*'' + ''6.1.134-rt51'' + ] + [ + ''^\.sw[a-z]$'' + ''package.nix'' + ] + [ + ''^\..*\.sw[a-z]$'' + ''package.nix'' + ] + [ + ''^\.sw[a-z]$'' + ''pyproject.toml'' + ] + [ + ''^\..*\.sw[a-z]$'' + ''pyproject.toml'' + ] + [ + ''mpv[-_](.*)'' + ''mpv-notify-send'' + ] + [ + ''mpv[-_](.*)'' + ''mpv-playlistmanager'' + ] + [ + ''.*ch/qos/logback/logback-core.*'' + ''https://repo.maven.apache.org/maven2/ch/qos/logback/logback-core/maven-metadata.xml'' + ] + [ + ''.*commons-codec/commons-codec.*'' + ''https://repo.maven.apache.org/maven2/commons-codec/commons-codec/maven-metadata.xml'' + ] + [ + ''/[0-9a-z]{52}'' + ''/run/opengl-driver'' + ] + [ + ''/[0-9a-z]{52}'' + ''/dev/dri'' + ] + [ + ''<(.*)>'' + ''_module'' + ] + [ + ''<(.*)>'' + ''args'' + ] + [ + ''<(.*)>'' + '''' + ] + [ + ''<(.*)>'' + '''' + ] + [ + ''[a-zA-Z_][a-zA-Z0-9_'-]*'' + ''2bwm'' + ] + [ + ''[a-zA-Z_][a-zA-Z0-9_'-]*'' + ''pm.max_children'' + ] + [ + ''(pypy|python)([[:digit:]]*)'' + ''override'' + ] + [ + ''(pypy|python)([[:digit:]]*)'' + ''overrideDerivation'' + ] + [ + ''(pypy|python)([[:digit:]]*)'' + ''pypy27'' + ] + [ + ''(pypy|python)([[:digit:]]*)'' + ''pypy310'' + ] + [ + ''^ref: (.*)$'' + ''ref: refs/heads/master'' + ] + [ + ''^ref: (.*)$'' + ''f870c6ccc8951fc48aeb293cf3e98ade6ac42668'' + ] + [ + ''^ref: (.*)$'' + ''ref: refs/heads/master'' + ] + [ + ''^ref: (.*)$'' + ''f870c6ccc8951fc48aeb293cf3e98ade6ac42668'' + ] + [ + ''.*\.post[0-9]+'' + ''1.7.2'' + ] + [ + ''.*tensorflow_cpu.*'' + ''https://storage.googleapis.com/tensorflow/versions/2.19.0/tensorflow_cpu-2.19.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl'' + ] + [ + ''.*tensorflow_cpu.*'' + ''https://storage.googleapis.com/tensorflow/versions/2.19.0/tensorflow-2.19.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl'' + ] + [ + ''.*\.post[0-9]+'' + ''1.7.2'' + ] + [ + ''.*darwin.*'' + ''i686-cygwin'' + ] + [ + ''.*darwin.*'' + ''x86_64-cygwin'' + ] + [ + ''.*darwin.*'' + ''x86_64-darwin'' + ] + [ + ''.*darwin.*'' + ''aarch64-darwin'' + ] + [ + ''.*com/badlogicgames/gdx-controllers/gdx-controllers-core.*'' + ''https://oss.sonatype.org/content/repositories/snapshots/com/badlogicgames/gdx-controllers/gdx-controllers-core/2.2.4-SNAPSHOT/maven-metadata.xml'' + ] + [ + ''.*com/badlogicgames/gdx-controllers/gdx-controllers-desktop.*'' + ''https://oss.sonatype.org/content/repositories/snapshots/com/badlogicgames/gdx-controllers/gdx-controllers-desktop/2.2.4-SNAPSHOT/maven-metadata.xml'' + ] + [ + ''.*/linux-recordreplay-(.*).tgz'' + ''https://static.replay.io/downloads/linux-recordreplay-b7eae18423ef.tgz'' + ] + [ + ''.*/linux-node-(.*)'' + ''https://static.replay.io/downloads/linux-node-20220726-bac6d66b5ca1-5b966f2f136c'' + ] + [ + ''.*-large-wordlist.*'' + ''hunspell-dict-cs-cz-libreoffice-6.3.0.4'' + ] + [ + ''.*-large-wordlist.*'' + ''hunspell-dict-da-dk-2.5.189'' + ] + [ + ''.*-large-wordlist.*'' + ''hunspell-dict-en-au-large-wordlist-2018.04.16'' + ] + [ + ''.*-large-wordlist.*'' + ''hunspell-dict-en-ca-large-wordlist-2018.04.16'' + ] + [ + ''.*com/fazecast/jSerialComm.*'' + ''https://oss.sonatype.org/content/repositories/snapshots/com/fazecast/jSerialComm/2.11.1-SNAPSHOT/maven-metadata.xml'' + ] + [ + ''.*net/java/dev/jna/jna-platform.*'' + ''https://oss.sonatype.org/content/repositories/snapshots/net/java/dev/jna/jna-platform/5.1.1-SNAPSHOT/maven-metadata.xml'' + ] + [ + ''.*net/java/dev/jna/jna-platform.*'' + ''https://oss.sonatype.org/content/repositories/snapshots/net/java/dev/jna/jna-platform/maven-metadata.xml'' + ] + [ + ''.*net/java/dev/jna/jna.*'' + ''https://oss.sonatype.org/content/repositories/snapshots/net/java/dev/jna/jna/5.1.1-SNAPSHOT/maven-metadata.xml'' + ] + [ + ''.*net/java/dev/jna/jna.*'' + ''https://oss.sonatype.org/content/repositories/snapshots/net/java/dev/jna/jna/maven-metadata.xml'' + ] + [ + ''.*org/java-websocket/Java-WebSocket.*'' + ''https://oss.sonatype.org/content/repositories/snapshots/org/java-websocket/Java-WebSocket/1.3.10-SNAPSHOT/maven-metadata.xml'' + ] + [ + ''.*org/java-websocket/Java-WebSocket.*'' + ''https://oss.sonatype.org/content/repositories/snapshots/org/java-websocket/Java-WebSocket/maven-metadata.xml'' + ] + [ + ''.*com/melloware/jintellitype.*'' + ''https://repo.maven.apache.org/maven2/com/melloware/jintellitype/maven-metadata.xml'' + ] + [ + ''[0-9.]*([a-z]*)'' + ''2025.1.1'' + ] + [ + ''.*com/velocitypowered/velocity-brigadier.*'' + ''https://repo.papermc.io/repository/maven-public/com/velocitypowered/velocity-brigadier/1.0.0-SNAPSHOT/maven-metadata.xml'' + ] + ] From 6686b540773ff308d9918abfa6cf2606951b6f34 Mon Sep 17 00:00:00 2001 From: Eelco Dolstra Date: Tue, 27 May 2025 23:44:47 +0200 Subject: [PATCH 095/218] Fix mingw build https://hydra.nixos.org/build/298331457 --- src/libcmd/repl-interacter.cc | 2 ++ src/libexpr/eval-profiler.cc | 2 +- src/libutil/include/nix/util/unix-domain-socket.hh | 2 +- src/libutil/logging.cc | 2 +- src/libutil/unix-domain-socket.cc | 7 ++----- src/libutil/windows/file-system.cc | 1 + src/nix/formatter.cc | 2 +- 7 files changed, 9 insertions(+), 9 deletions(-) diff --git a/src/libcmd/repl-interacter.cc b/src/libcmd/repl-interacter.cc index 769935efa8d..4de335dd5e5 100644 --- a/src/libcmd/repl-interacter.cc +++ b/src/libcmd/repl-interacter.cc @@ -2,6 +2,8 @@ #include +#include + #if USE_READLINE #include #include diff --git a/src/libexpr/eval-profiler.cc b/src/libexpr/eval-profiler.cc index c72737f7394..7053e4ec7d6 100644 --- a/src/libexpr/eval-profiler.cc +++ b/src/libexpr/eval-profiler.cc @@ -137,7 +137,7 @@ class SampleStack : public EvalProfiler : state(state) , sampleInterval(period) , profileFd([&]() { - AutoCloseFD fd = toDescriptor(open(profileFile.c_str(), O_WRONLY | O_CREAT | O_TRUNC, 0660)); + AutoCloseFD fd = toDescriptor(open(profileFile.string().c_str(), O_WRONLY | O_CREAT | O_TRUNC, 0660)); if (!fd) throw SysError("opening file %s", profileFile); return fd; diff --git a/src/libutil/include/nix/util/unix-domain-socket.hh b/src/libutil/include/nix/util/unix-domain-socket.hh index 6885d740b12..3aaaddf823d 100644 --- a/src/libutil/include/nix/util/unix-domain-socket.hh +++ b/src/libutil/include/nix/util/unix-domain-socket.hh @@ -80,7 +80,7 @@ void bind(Socket fd, const std::string & path); /** * Connect to a Unix domain socket. */ -void connect(Socket fd, const std::string & path); +void connect(Socket fd, const std::filesystem::path & path); /** * Connect to a Unix domain socket. diff --git a/src/libutil/logging.cc b/src/libutil/logging.cc index a328a5c73e3..9b4ecbda87a 100644 --- a/src/libutil/logging.cc +++ b/src/libutil/logging.cc @@ -324,7 +324,7 @@ std::unique_ptr makeJSONLogger(const std::filesystem::path & path, bool AutoCloseFD fd = std::filesystem::is_socket(path) ? connect(path) - : toDescriptor(open(path.c_str(), O_CREAT | O_APPEND | O_WRONLY, 0644)); + : toDescriptor(open(path.string().c_str(), O_CREAT | O_APPEND | O_WRONLY, 0644)); if (!fd) throw SysError("opening log file %1%", path); diff --git a/src/libutil/unix-domain-socket.cc b/src/libutil/unix-domain-socket.cc index 0e8c21d668f..2422caf14bb 100644 --- a/src/libutil/unix-domain-socket.cc +++ b/src/libutil/unix-domain-socket.cc @@ -29,7 +29,6 @@ AutoCloseFD createUnixDomainSocket() return fdSocket; } - AutoCloseFD createUnixDomainSocket(const Path & path, mode_t mode) { auto fdSocket = nix::createUnixDomainSocket(); @@ -100,7 +99,6 @@ static void bindConnectProcHelper( } } - void bind(Socket fd, const std::string & path) { unlink(path.c_str()); @@ -108,10 +106,9 @@ void bind(Socket fd, const std::string & path) bindConnectProcHelper("bind", ::bind, fd, path); } - -void connect(Socket fd, const std::string & path) +void connect(Socket fd, const std::filesystem::path & path) { - bindConnectProcHelper("connect", ::connect, fd, path); + bindConnectProcHelper("connect", ::connect, fd, path.string()); } AutoCloseFD connect(const std::filesystem::path & path) diff --git a/src/libutil/windows/file-system.cc b/src/libutil/windows/file-system.cc index a73fa223a36..f31c913f141 100644 --- a/src/libutil/windows/file-system.cc +++ b/src/libutil/windows/file-system.cc @@ -1,4 +1,5 @@ #include "nix/util/file-system.hh" +#include "nix/util/logging.hh" #ifdef _WIN32 namespace nix { diff --git a/src/nix/formatter.cc b/src/nix/formatter.cc index 627fb362c83..212bb8d7091 100644 --- a/src/nix/formatter.cc +++ b/src/nix/formatter.cc @@ -92,7 +92,7 @@ struct CmdFormatterRun : MixFormatter, MixJSON // Add the path to the flake as an environment variable. This enables formatters to format the entire flake even // if run from a subdirectory. StringMap env = getEnv(); - env["PRJ_ROOT"] = flakeDir; + env["PRJ_ROOT"] = flakeDir.string(); // Release our references to eval caches to ensure they are persisted to disk, because // we are about to exec out of this process without running C++ destructors. From d0a23238294198f6702e13d117f75af89dbeac62 Mon Sep 17 00:00:00 2001 From: Seth Flynn Date: Tue, 27 May 2025 22:20:53 -0400 Subject: [PATCH 096/218] lockFlake(): Allow registry lookups for overridden inputs Fixes #13144 --- src/libflake/flake.cc | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/src/libflake/flake.cc b/src/libflake/flake.cc index fc27df8874e..24252d710d7 100644 --- a/src/libflake/flake.cc +++ b/src/libflake/flake.cc @@ -570,7 +570,7 @@ LockedFlake lockFlake( /* Get the input flake, resolve 'path:./...' flakerefs relative to the parent flake. */ - auto getInputFlake = [&](const FlakeRef & ref) + auto getInputFlake = [&](const FlakeRef & ref, const fetchers::UseRegistries useRegistries) { if (auto resolvedPath = resolveRelativePath()) { return readFlake(state, ref, ref, ref, *resolvedPath, inputAttrPath); @@ -578,7 +578,7 @@ LockedFlake lockFlake( return getFlake( state, ref, - useRegistriesInputs, + useRegistries, inputAttrPath); } }; @@ -660,7 +660,7 @@ LockedFlake lockFlake( } if (mustRefetch) { - auto inputFlake = getInputFlake(oldLock->lockedRef); + auto inputFlake = getInputFlake(oldLock->lockedRef, useRegistriesInputs); nodePaths.emplace(childNode, inputFlake.path.parent()); computeLocks(inputFlake.inputs, childNode, inputAttrPath, oldLock, followsPrefix, inputFlake.path, false); @@ -685,10 +685,11 @@ LockedFlake lockFlake( nuked the next time we update the lock file. That is, overrides are sticky unless you use --no-write-lock-file. */ - auto ref = (input2.ref && explicitCliOverrides.contains(inputAttrPath)) ? *input2.ref : *input.ref; + auto inputIsOverride = explicitCliOverrides.contains(inputAttrPath); + auto ref = (input2.ref && inputIsOverride) ? *input2.ref : *input.ref; if (input.isFlake) { - auto inputFlake = getInputFlake(*input.ref); + auto inputFlake = getInputFlake(*input.ref, inputIsOverride ? fetchers::UseRegistries::All : useRegistriesInputs); auto childNode = make_ref( inputFlake.lockedRef, From a353b2f4b200df22dcd420f99ee13d226f566f0e Mon Sep 17 00:00:00 2001 From: John Ericson Date: Mon, 26 May 2025 13:54:04 -0400 Subject: [PATCH 097/218] Test hacky way of making structured attrs --- tests/functional/structured-attrs.sh | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/tests/functional/structured-attrs.sh b/tests/functional/structured-attrs.sh index 64d136e993a..fe9bcc0ee6b 100755 --- a/tests/functional/structured-attrs.sh +++ b/tests/functional/structured-attrs.sh @@ -40,3 +40,11 @@ jsonOut="$(nix print-dev-env -f structured-attrs-shell.nix --json)" test "$(<<<"$jsonOut" jq '.structuredAttrs|keys|.[]' -r)" = "$(printf ".attrs.json\n.attrs.sh")" test "$(<<<"$jsonOut" jq '.variables.outputs.value.out' -r)" = "$(<<<"$jsonOut" jq '.structuredAttrs.".attrs.json"' -r | jq -r '.outputs.out')" + +# Hacky way of making structured attrs. We should preserve for now for back compat, but also deprecate. + +hackyExpr='derivation { name = "a"; system = "foo"; builder = "/bin/sh"; __json = builtins.toJSON { a = 1; }; }' + +# Check it works with the expected structured attrs +hacky=$(nix-instantiate --expr "$hackyExpr") +nix derivation show "$hacky" | jq --exit-status '."'"$hacky"'".env.__json | fromjson | . == {"a": 1}' From 8d725fdcb0b57f9b2d7993201f88d36b3f917713 Mon Sep 17 00:00:00 2001 From: John Ericson Date: Wed, 28 May 2025 12:47:33 -0400 Subject: [PATCH 098/218] Fix FreeBSD builds --- src/libutil/package.nix | 2 ++ 1 file changed, 2 insertions(+) diff --git a/src/libutil/package.nix b/src/libutil/package.nix index 17c84ff1850..46f56e07e6d 100644 --- a/src/libutil/package.nix +++ b/src/libutil/package.nix @@ -37,6 +37,8 @@ mkMesonLibrary (finalAttrs: { ./include/nix/util/meson.build ./linux/meson.build ./linux/include/nix/util/meson.build + ./freebsd/meson.build + ./freebsd/include/nix/util/meson.build ./unix/meson.build ./unix/include/nix/util/meson.build ./windows/meson.build From 24f5d7a9c3dbeaee34858058b74b15f72ff27134 Mon Sep 17 00:00:00 2001 From: John Ericson Date: Wed, 28 May 2025 12:49:13 -0400 Subject: [PATCH 099/218] Fix warning when `HAVE_EMBEDDED_SANDBOX_SHELL` is not set Clang doesn't like the double indent that is needed for the `if...else` that is CPP'd away. Adding braces is fine in the `if...else...` case, and fine as a naked block in the CPP'd away case, and properly-indented both ways. --- src/libstore/unix/build/linux-derivation-builder.cc | 2 ++ 1 file changed, 2 insertions(+) diff --git a/src/libstore/unix/build/linux-derivation-builder.cc b/src/libstore/unix/build/linux-derivation-builder.cc index efbe4c8bb3b..23850c373ae 100644 --- a/src/libstore/unix/build/linux-derivation-builder.cc +++ b/src/libstore/unix/build/linux-derivation-builder.cc @@ -685,7 +685,9 @@ struct LinuxDerivationBuilder : DerivationBuilderImpl chmod_(dst, 0555); } else # endif + { doBind(i.second.source, chrootRootDir + i.first, i.second.optional); + } } /* Bind a new instance of procfs on /proc. */ From 7577d2d3ae8fa203604cc3b09e2ed81678163d46 Mon Sep 17 00:00:00 2001 From: John Ericson Date: Mon, 26 May 2025 13:54:04 -0400 Subject: [PATCH 100/218] Deprecate hacky way of making structured attrs The method tested for in the previous commit is now deprecated. Co-authored-by: Eelco Dolstra --- doc/manual/rl-next/deprecate__json.md | 11 +++++++++++ src/libexpr/eval.cc | 1 + src/libexpr/include/nix/expr/eval.hh | 2 +- src/libexpr/primops.cc | 2 ++ tests/functional/structured-attrs.sh | 3 +++ 5 files changed, 18 insertions(+), 1 deletion(-) create mode 100644 doc/manual/rl-next/deprecate__json.md diff --git a/doc/manual/rl-next/deprecate__json.md b/doc/manual/rl-next/deprecate__json.md new file mode 100644 index 00000000000..7fc05832f3b --- /dev/null +++ b/doc/manual/rl-next/deprecate__json.md @@ -0,0 +1,11 @@ +--- +synopsis: Deprecate manually making structured attrs with `__json = ...;` +prs: [13220] +--- + +The proper way to create a derivation using [structured attrs] in the Nix language is by using `__structuredAttrs = true` with [`builtins.derivation`]. +However, by exploiting how structured attrs are implementated, it has also been possible to create them by setting the `__json` environment variable to a serialized JSON string. +This sneaky alternative method is now deprecated, and may be disallowed in future versions of Nix. + +[structured attrs]: @docroot@/language/advanced-attributes.md#adv-attr-structuredAttrs +[`builtins.derivation`]: @docroot@/language/builtins.html#builtins-derivation diff --git a/src/libexpr/eval.cc b/src/libexpr/eval.cc index 1a067e75cdb..ce35901b137 100644 --- a/src/libexpr/eval.cc +++ b/src/libexpr/eval.cc @@ -212,6 +212,7 @@ EvalState::EvalState( , sRight(symbols.create("right")) , sWrong(symbols.create("wrong")) , sStructuredAttrs(symbols.create("__structuredAttrs")) + , sJson(symbols.create("__json")) , sAllowedReferences(symbols.create("allowedReferences")) , sAllowedRequisites(symbols.create("allowedRequisites")) , sDisallowedReferences(symbols.create("disallowedReferences")) diff --git a/src/libexpr/include/nix/expr/eval.hh b/src/libexpr/include/nix/expr/eval.hh index a4347b3e15a..27294d11403 100644 --- a/src/libexpr/include/nix/expr/eval.hh +++ b/src/libexpr/include/nix/expr/eval.hh @@ -213,7 +213,7 @@ public: const Symbol sWith, sOutPath, sDrvPath, sType, sMeta, sName, sValue, sSystem, sOverrides, sOutputs, sOutputName, sIgnoreNulls, sFile, sLine, sColumn, sFunctor, sToString, - sRight, sWrong, sStructuredAttrs, + sRight, sWrong, sStructuredAttrs, sJson, sAllowedReferences, sAllowedRequisites, sDisallowedReferences, sDisallowedRequisites, sMaxSize, sMaxClosureSize, sBuilder, sArgs, diff --git a/src/libexpr/primops.cc b/src/libexpr/primops.cc index f3e3e129071..01cfb3b0e16 100644 --- a/src/libexpr/primops.cc +++ b/src/libexpr/primops.cc @@ -1427,6 +1427,8 @@ static void derivationStrictInternal( else if (i->name == state.sOutputHashMode) handleHashMode(s); else if (i->name == state.sOutputs) handleOutputs(tokenizeString(s)); + else if (i->name == state.sJson) + warn("In derivation '%s': setting structured attributes via '__json' is deprecated, and may be disallowed in future versions of Nix. Set '__structuredAttrs = true' instead.", drvName); } } diff --git a/tests/functional/structured-attrs.sh b/tests/functional/structured-attrs.sh index fe9bcc0ee6b..465676b410f 100755 --- a/tests/functional/structured-attrs.sh +++ b/tests/functional/structured-attrs.sh @@ -45,6 +45,9 @@ test "$(<<<"$jsonOut" jq '.variables.outputs.value.out' -r)" = "$(<<<"$jsonOut" hackyExpr='derivation { name = "a"; system = "foo"; builder = "/bin/sh"; __json = builtins.toJSON { a = 1; }; }' +# Check for deprecation message +expectStderr 0 nix-instantiate --expr "$hackyExpr" --eval --strict | grepQuiet "In derivation 'a': setting structured attributes via '__json' is deprecated, and may be disallowed in future versions of Nix. Set '__structuredAttrs = true' instead." + # Check it works with the expected structured attrs hacky=$(nix-instantiate --expr "$hackyExpr") nix derivation show "$hacky" | jq --exit-status '."'"$hacky"'".env.__json | fromjson | . == {"a": 1}' From a65318492683bf2a5b0dbc2ad2e9f8eb530d08dc Mon Sep 17 00:00:00 2001 From: Sergei Zimmerman Date: Wed, 28 May 2025 21:42:33 +0000 Subject: [PATCH 101/218] Drop precompiled-headers.h Since the migration to meson precompiled-headers.h isn't actually used anymore and is just confusing. Meson can't handle shared pch across subprojects [1] and without that there's no performances benefit of PCH at all. Also rolling our own support for that isn't trivial. See [2] for an example of how that would look like. [1]: https://github.com/mesonbuild/meson/issues/4350 [2]: https://github.com/WayfireWM/wayfire/blob/22bc8b647351c45829baac570b828c8f852111dc/plugins/meson.build --- maintainers/flake-module.nix | 1 - precompiled-headers.h | 63 ------------------------------------ 2 files changed, 64 deletions(-) delete mode 100644 precompiled-headers.h diff --git a/maintainers/flake-module.nix b/maintainers/flake-module.nix index 6d8633c4f8b..d2bae32b6f2 100644 --- a/maintainers/flake-module.nix +++ b/maintainers/flake-module.nix @@ -81,7 +81,6 @@ # We haven't applied formatting to these files yet ''^doc/manual/redirects\.js$'' ''^doc/manual/theme/highlight\.js$'' - ''^precompiled-headers\.h$'' ''^src/build-remote/build-remote\.cc$'' ''^src/libcmd/built-path\.cc$'' ''^src/libcmd/include/nix/cmd/built-path\.hh$'' diff --git a/precompiled-headers.h b/precompiled-headers.h deleted file mode 100644 index e1a3f8cc031..00000000000 --- a/precompiled-headers.h +++ /dev/null @@ -1,63 +0,0 @@ -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include - -#include - -#include -#include -#include -#include -#include -#include -#include -#include - -#ifndef _WIN32 -# include -# include -# include -# include -# include -# include -# include -# include -# include -#endif - -#include From fba1bb0c137036adc5127afe4183f45ab3dde61d Mon Sep 17 00:00:00 2001 From: Sergei Zimmerman Date: Thu, 29 May 2025 19:35:12 +0000 Subject: [PATCH 102/218] Clear `displayPrefix` in `makeEmptySourceAccessor` Judging by the comment for `makeEmptySourceAccessor` the prefix has to be empty: > Return a source accessor that contains only an empty root directory. Fixes #13295. --- src/libutil/memory-source-accessor.cc | 4 ++++ tests/functional/pure-eval.sh | 12 ++++++++++++ 2 files changed, 16 insertions(+) diff --git a/src/libutil/memory-source-accessor.cc b/src/libutil/memory-source-accessor.cc index 7764ff946a2..5612c9454f0 100644 --- a/src/libutil/memory-source-accessor.cc +++ b/src/libutil/memory-source-accessor.cc @@ -187,6 +187,10 @@ void MemorySink::createSymlink(const CanonPath & path, const std::string & targe ref makeEmptySourceAccessor() { static auto empty = make_ref().cast(); + /* Don't forget to clear the display prefix, as the default constructed + SourceAccessor has the «unknown» prefix. Since this accessor is supposed + to mimic an empty root directory the prefix needs to be empty. */ + empty->setPathDisplay(""); return empty; } diff --git a/tests/functional/pure-eval.sh b/tests/functional/pure-eval.sh index 25038109982..45a65f9ab8f 100755 --- a/tests/functional/pure-eval.sh +++ b/tests/functional/pure-eval.sh @@ -34,3 +34,15 @@ rm -rf $TEST_ROOT/eval-out (! nix eval --store dummy:// --write-to $TEST_ROOT/eval-out --expr '{ "." = "bla"; }') (! nix eval --expr '~/foo') + +expectStderr 0 nix eval --expr "/some/absolute/path" \ + | grepQuiet "/some/absolute/path" + +expectStderr 0 nix eval --expr "/some/absolute/path" --impure \ + | grepQuiet "/some/absolute/path" + +expectStderr 0 nix eval --expr "some/relative/path" \ + | grepQuiet "$PWD/some/relative/path" + +expectStderr 0 nix eval --expr "some/relative/path" --impure \ + | grepQuiet "$PWD/some/relative/path" From b7fd872147ad1d3d91c8068686c37508199ba96e Mon Sep 17 00:00:00 2001 From: Eelco Dolstra Date: Wed, 28 May 2025 13:04:09 +0200 Subject: [PATCH 103/218] Cleanup --- src/libstore/unix/build/linux-derivation-builder.cc | 6 +----- 1 file changed, 1 insertion(+), 5 deletions(-) diff --git a/src/libstore/unix/build/linux-derivation-builder.cc b/src/libstore/unix/build/linux-derivation-builder.cc index 23850c373ae..fe62314f290 100644 --- a/src/libstore/unix/build/linux-derivation-builder.cc +++ b/src/libstore/unix/build/linux-derivation-builder.cc @@ -190,11 +190,7 @@ struct LinuxDerivationBuilder : DerivationBuilderImpl */ std::optional cgroup; - LinuxDerivationBuilder( - Store & store, std::unique_ptr miscMethods, DerivationBuilderParams params) - : DerivationBuilderImpl(store, std::move(miscMethods), std::move(params)) - { - } + using DerivationBuilderImpl::DerivationBuilderImpl; void deleteTmpDir(bool force) override { From 4dc419eaec84ca82abe0161a67f2596b04c8b75d Mon Sep 17 00:00:00 2001 From: Eelco Dolstra Date: Thu, 29 May 2025 21:57:25 +0200 Subject: [PATCH 104/218] Split LinuxDerivationBuilder This restores doing seccomp/personality initialization even when sandboxing is disabled. https://hydra.nixos.org/build/298482132 --- src/libstore/unix/build/derivation-builder.cc | 7 ++++++- .../unix/build/linux-derivation-builder.cc | 20 +++++++++++++------ 2 files changed, 20 insertions(+), 7 deletions(-) diff --git a/src/libstore/unix/build/derivation-builder.cc b/src/libstore/unix/build/derivation-builder.cc index 9935400b975..232a125e42f 100644 --- a/src/libstore/unix/build/derivation-builder.cc +++ b/src/libstore/unix/build/derivation-builder.cc @@ -2157,7 +2157,7 @@ std::unique_ptr makeDerivationBuilder( } if (useSandbox) - return std::make_unique( + return std::make_unique( store, std::move(miscMethods), std::move(params)); @@ -2172,6 +2172,11 @@ std::unique_ptr makeDerivationBuilder( std::move(miscMethods), std::move(params), useSandbox); + #elif defined(__linux__) + return std::make_unique( + store, + std::move(miscMethods), + std::move(params)); #else if (useSandbox) throw Error("sandboxing builds is not supported on this platform"); diff --git a/src/libstore/unix/build/linux-derivation-builder.cc b/src/libstore/unix/build/linux-derivation-builder.cc index fe62314f290..c27b87163ad 100644 --- a/src/libstore/unix/build/linux-derivation-builder.cc +++ b/src/libstore/unix/build/linux-derivation-builder.cc @@ -154,6 +154,18 @@ static void doBind(const Path & source, const Path & target, bool optional = fal } struct LinuxDerivationBuilder : DerivationBuilderImpl +{ + using DerivationBuilderImpl::DerivationBuilderImpl; + + void enterChroot() override + { + setupSeccomp(); + + linux::setPersonality(drv.platform); + } +}; + +struct ChrootLinuxDerivationBuilder : LinuxDerivationBuilder { /** * Pipe for synchronising updates to the builder namespaces. @@ -190,7 +202,7 @@ struct LinuxDerivationBuilder : DerivationBuilderImpl */ std::optional cgroup; - using DerivationBuilderImpl::DerivationBuilderImpl; + using LinuxDerivationBuilder::LinuxDerivationBuilder; void deleteTmpDir(bool force) override { @@ -772,11 +784,7 @@ struct LinuxDerivationBuilder : DerivationBuilderImpl if (rmdir("real-root") == -1) throw SysError("cannot remove real-root directory"); - // FIXME: move to LinuxDerivationBuilder - setupSeccomp(); - - // FIXME: move to LinuxDerivationBuilder - linux::setPersonality(drv.platform); + LinuxDerivationBuilder::enterChroot(); } void setUser() override From dd80f16376b8299fced19374ed64218ec34f0c36 Mon Sep 17 00:00:00 2001 From: Sergei Zimmerman Date: Thu, 29 May 2025 21:57:08 +0000 Subject: [PATCH 105/218] flake: Drop `pre-commit` override `pre-commit` builds fine with the flake's input nixpkgs on i686-linux. --- flake.nix | 13 ------------- 1 file changed, 13 deletions(-) diff --git a/flake.nix b/flake.nix index a8759d04257..46104dbdda3 100644 --- a/flake.nix +++ b/flake.nix @@ -179,19 +179,6 @@ }; nix = final.nixComponents2.nix-cli; - - # See https://github.com/NixOS/nixpkgs/pull/214409 - # Remove when fixed in this flake's nixpkgs - pre-commit = - if prev.stdenv.hostPlatform.system == "i686-linux" then - (prev.pre-commit.override (o: { - dotnet-sdk = ""; - })).overridePythonAttrs - (o: { - doCheck = false; - }) - else - prev.pre-commit; }; in From 4fa991a680ee8d479e23d1474de1ac5b331c3026 Mon Sep 17 00:00:00 2001 From: Sergei Zimmerman Date: Thu, 29 May 2025 22:35:50 +0000 Subject: [PATCH 106/218] flake: Restore `packaging-overriding` check The underlying bug seems to have been fixed in diffoscope 293 [1] [2]. Our nixpkgs input has 295. [1]: https://github.com/NixOS/nixpkgs/pull/393381#issuecomment-2766703347 [2]: https://diffoscope.org/news/diffoscope-292-released/ --- flake.nix | 40 ++++++++++++++++++---------------------- 1 file changed, 18 insertions(+), 22 deletions(-) diff --git a/flake.nix b/flake.nix index a8759d04257..de9244b5c4c 100644 --- a/flake.nix +++ b/flake.nix @@ -233,28 +233,24 @@ This shouldn't build anything significant; just check that things (including derivations) are _set up_ correctly. */ - # Disabled due to a bug in `testEqualContents` (see - # https://github.com/NixOS/nix/issues/12690). - /* - packaging-overriding = - let - pkgs = nixpkgsFor.${system}.native; - nix = self.packages.${system}.nix; - in - assert (nix.appendPatches [ pkgs.emptyFile ]).libs.nix-util.src.patches == [ pkgs.emptyFile ]; - if pkgs.stdenv.buildPlatform.isDarwin then - lib.warn "packaging-overriding check currently disabled because of a permissions issue on macOS" pkgs.emptyFile - else - # If this fails, something might be wrong with how we've wired the scope, - # or something could be broken in Nixpkgs. - pkgs.testers.testEqualContents { - assertion = "trivial patch does not change source contents"; - expected = "${./.}"; - actual = - # Same for all components; nix-util is an arbitrary pick - (nix.appendPatches [ pkgs.emptyFile ]).libs.nix-util.src; - }; - */ + packaging-overriding = + let + pkgs = nixpkgsFor.${system}.native; + nix = self.packages.${system}.nix; + in + assert (nix.appendPatches [ pkgs.emptyFile ]).libs.nix-util.src.patches == [ pkgs.emptyFile ]; + if pkgs.stdenv.buildPlatform.isDarwin then + lib.warn "packaging-overriding check currently disabled because of a permissions issue on macOS" pkgs.emptyFile + else + # If this fails, something might be wrong with how we've wired the scope, + # or something could be broken in Nixpkgs. + pkgs.testers.testEqualContents { + assertion = "trivial patch does not change source contents"; + expected = "${./.}"; + actual = + # Same for all components; nix-util is an arbitrary pick + (nix.appendPatches [ pkgs.emptyFile ]).libs.nix-util.src; + }; } // (lib.optionalAttrs (builtins.elem system linux64BitSystems)) { dockerImage = self.hydraJobs.dockerImage.${system}; From 908129eb2216863e4761730741779b5d21a886f5 Mon Sep 17 00:00:00 2001 From: Eelco Dolstra Date: Fri, 30 May 2025 11:54:54 +0200 Subject: [PATCH 107/218] Cleanup --- src/libstore/unix/build/derivation-builder.cc | 14 ++++++-------- 1 file changed, 6 insertions(+), 8 deletions(-) diff --git a/src/libstore/unix/build/derivation-builder.cc b/src/libstore/unix/build/derivation-builder.cc index 232a125e42f..eca01748733 100644 --- a/src/libstore/unix/build/derivation-builder.cc +++ b/src/libstore/unix/build/derivation-builder.cc @@ -2147,13 +2147,11 @@ std::unique_ptr makeDerivationBuilder( } #ifdef __linux__ - if (useSandbox) { - if (!mountAndPidNamespacesSupported()) { - if (!settings.sandboxFallback) - throw Error("this system does not support the kernel namespaces that are required for sandboxing; use '--no-sandbox' to disable sandboxing"); - debug("auto-disabling sandboxing because the prerequisite namespaces are not available"); - useSandbox = false; - } + if (useSandbox && !mountAndPidNamespacesSupported()) { + if (!settings.sandboxFallback) + throw Error("this system does not support the kernel namespaces that are required for sandboxing; use '--no-sandbox' to disable sandboxing"); + debug("auto-disabling sandboxing because the prerequisite namespaces are not available"); + useSandbox = false; } if (useSandbox) @@ -2163,7 +2161,7 @@ std::unique_ptr makeDerivationBuilder( std::move(params)); #endif - if (params.drvOptions.useUidRange(params.drv)) + if (!useSandbox && params.drvOptions.useUidRange(params.drv)) throw Error("feature 'uid-range' is only supported in sandboxed builds"); #ifdef __APPLE__ From 58e34a2d27a5264cc0eb36fceccc009e0ec205d0 Mon Sep 17 00:00:00 2001 From: Farid Zakaria Date: Sat, 31 May 2025 07:41:27 -0700 Subject: [PATCH 108/218] Overriding gtest with gmock How did this work before... * Added .direnv/ to gitignore --- .gitignore | 3 +++ src/libexpr-tests/meson.build | 4 ++-- 2 files changed, 5 insertions(+), 2 deletions(-) diff --git a/.gitignore b/.gitignore index 9c46912406f..4782bfbafd2 100644 --- a/.gitignore +++ b/.gitignore @@ -47,3 +47,6 @@ result-* .DS_Store flake-regressions + +# direnv +.direnv/ diff --git a/src/libexpr-tests/meson.build b/src/libexpr-tests/meson.build index f7822edfd9f..35ae8a9d068 100644 --- a/src/libexpr-tests/meson.build +++ b/src/libexpr-tests/meson.build @@ -32,8 +32,8 @@ deps_private += rapidcheck gtest = dependency('gtest') deps_private += gtest -gtest = dependency('gmock') -deps_private += gtest +gmock = dependency('gmock') +deps_private += gmock configdata = configuration_data() configdata.set_quoted('PACKAGE_VERSION', meson.project_version()) From 6badd21b6a22c6037a5152463343491e770a581d Mon Sep 17 00:00:00 2001 From: Jade Lynn Masker Date: Sat, 31 May 2025 17:50:46 -0400 Subject: [PATCH 109/218] add documentation of tarball-ttl to nix-channel --- doc/manual/source/command-ref/nix-channel.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/doc/manual/source/command-ref/nix-channel.md b/doc/manual/source/command-ref/nix-channel.md index 8b58392b7b5..4a7555c9058 100644 --- a/doc/manual/source/command-ref/nix-channel.md +++ b/doc/manual/source/command-ref/nix-channel.md @@ -53,6 +53,11 @@ This command has the following operations: Download the Nix expressions of subscribed channels and create a new generation. Update all channels if none is specified, and only those included in *names* otherwise. + > **Note** + > + > `--update` uses [fetchTarball](@docroot@/language/builtins.md#builtins-fetchTarball) under the hood, so it will cache channels. + > Use `--tarball-ttl` or the nix configuration option `tarball-ttl` to change this behavior. + - `--list-generations` Prints a list of all the current existing generations for the From 94bbaddb93988352645e9b4bb4bec6c5ed730747 Mon Sep 17 00:00:00 2001 From: Philipp Otterbein Date: Sun, 18 May 2025 03:53:50 +0200 Subject: [PATCH 110/218] symbol-table: reduce memory usage and use boost::unordered_flat_set --- src/libexpr/eval.cc | 8 +- .../include/nix/expr/print-ambiguous.hh | 1 + src/libexpr/include/nix/expr/symbol-table.hh | 228 ++++++++++++++---- src/libexpr/include/nix/expr/value.hh | 7 +- src/libexpr/nixexpr.cc | 2 +- 5 files changed, 193 insertions(+), 53 deletions(-) diff --git a/src/libexpr/eval.cc b/src/libexpr/eval.cc index 1a067e75cdb..123f0e86a68 100644 --- a/src/libexpr/eval.cc +++ b/src/libexpr/eval.cc @@ -2,6 +2,7 @@ #include "nix/expr/eval-settings.hh" #include "nix/expr/primops.hh" #include "nix/expr/print-options.hh" +#include "nix/expr/symbol-table.hh" #include "nix/util/exit.hh" #include "nix/util/types.hh" #include "nix/util/util.hh" @@ -918,6 +919,11 @@ void Value::mkStringMove(const char * s, const NixStringContext & context) mkString(s, encodeContext(context)); } +void Value::mkString(const SymbolStr & s) +{ + mkString(s.c_str(), nullptr); +} + void Value::mkPath(const SourcePath & path) { mkPath(&*path.accessor, makeImmutableString(path.path.abs())); @@ -3019,7 +3025,7 @@ void EvalState::printStatistics() // XXX: overrides earlier assignment topObj["symbols"] = json::array(); auto &list = topObj["symbols"]; - symbols.dump([&](const std::string & s) { list.emplace_back(s); }); + symbols.dump([&](std::string_view s) { list.emplace_back(s); }); } if (outPath == "-") { std::cerr << topObj.dump(2) << std::endl; diff --git a/src/libexpr/include/nix/expr/print-ambiguous.hh b/src/libexpr/include/nix/expr/print-ambiguous.hh index 09a849c498b..9e5a27e6d6e 100644 --- a/src/libexpr/include/nix/expr/print-ambiguous.hh +++ b/src/libexpr/include/nix/expr/print-ambiguous.hh @@ -1,6 +1,7 @@ #pragma once #include "nix/expr/value.hh" +#include "nix/expr/symbol-table.hh" namespace nix { diff --git a/src/libexpr/include/nix/expr/symbol-table.hh b/src/libexpr/include/nix/expr/symbol-table.hh index c04cc041b9f..994bddd6c24 100644 --- a/src/libexpr/include/nix/expr/symbol-table.hh +++ b/src/libexpr/include/nix/expr/symbol-table.hh @@ -1,16 +1,65 @@ #pragma once ///@file -#include -#include -#include - -#include "nix/util/types.hh" +#include +#include "nix/expr/value.hh" #include "nix/util/chunked-vector.hh" #include "nix/util/error.hh" +#include +#define USE_FLAT_SYMBOL_SET (BOOST_VERSION >= 108100) +#if USE_FLAT_SYMBOL_SET +# include +#else +# include +#endif + namespace nix { +class SymbolValue : protected Value +{ + friend class SymbolStr; + friend class SymbolTable; + + uint32_t size_; + uint32_t idx; + + SymbolValue() = default; + +public: + operator std::string_view() const noexcept + { + return {payload.string.c_str, size_}; + } +}; + +/** + * Symbols have the property that they can be compared efficiently + * (using an equality test), because the symbol table stores only one + * copy of each string. + */ +class Symbol +{ + friend class SymbolStr; + friend class SymbolTable; + +private: + uint32_t id; + + explicit Symbol(uint32_t id) noexcept : id(id) {} + +public: + Symbol() noexcept : id(0) {} + + [[gnu::always_inline]] + explicit operator bool() const noexcept { return id > 0; } + + auto operator<=>(const Symbol other) const noexcept { return id <=> other.id; } + bool operator==(const Symbol other) const noexcept { return id == other.id; } + + friend class std::hash; +}; + /** * This class mainly exists to give us an operator<< for ostreams. We could also * return plain strings from SymbolTable, but then we'd have to wrap every @@ -20,58 +69,124 @@ class SymbolStr { friend class SymbolTable; -private: - const std::string * s; + constexpr static size_t chunkSize{8192}; + using SymbolValueStore = ChunkedVector; - explicit SymbolStr(const std::string & symbol): s(&symbol) {} + const SymbolValue * s; + + struct Key + { + using HashType = boost::hash; + + SymbolValueStore & store; + std::string_view s; + std::size_t hash; + std::pmr::polymorphic_allocator & alloc; + + Key(SymbolValueStore & store, std::string_view s, std::pmr::polymorphic_allocator & stringAlloc) + : store(store) + , s(s) + , hash(HashType{}(s)) + , alloc(stringAlloc) {} + }; public: - bool operator == (std::string_view s2) const + SymbolStr(const SymbolValue & s) noexcept : s(&s) {} + + SymbolStr(const Key & key) + { + auto size = key.s.size(); + if (size >= std::numeric_limits::max()) { + throw Error("Size of symbol exceeds 4GiB and cannot be stored"); + } + // for multi-threaded implementations: lock store and allocator here + const auto & [v, idx] = key.store.add(SymbolValue{}); + if (size == 0) { + v.mkString("", nullptr); + } else { + auto s = key.alloc.allocate(size + 1); + memcpy(s, key.s.data(), size); + s[size] = '\0'; + v.mkString(s, nullptr); + } + v.size_ = size; + v.idx = idx; + this->s = &v; + } + + bool operator == (std::string_view s2) const noexcept { return *s == s2; } - const char * c_str() const + [[gnu::always_inline]] + const char * c_str() const noexcept { - return s->c_str(); + return s->payload.string.c_str; } - operator const std::string_view () const + [[gnu::always_inline]] + operator std::string_view () const noexcept { return *s; } friend std::ostream & operator <<(std::ostream & os, const SymbolStr & symbol); - bool empty() const + [[gnu::always_inline]] + bool empty() const noexcept { - return s->empty(); + return s->size_ == 0; } -}; -/** - * Symbols have the property that they can be compared efficiently - * (using an equality test), because the symbol table stores only one - * copy of each string. - */ -class Symbol -{ - friend class SymbolTable; + [[gnu::always_inline]] + size_t size() const noexcept + { + return s->size_; + } -private: - uint32_t id; + explicit operator Symbol() const noexcept + { + return Symbol{s->idx + 1}; + } + + struct Hash + { + using is_transparent = void; + using is_avalanching = std::true_type; - explicit Symbol(uint32_t id): id(id) {} + std::size_t operator()(SymbolStr str) const + { + return Key::HashType{}(*str.s); + } -public: - Symbol() : id(0) {} + std::size_t operator()(const Key & key) const noexcept + { + return key.hash; + } + }; - explicit operator bool() const { return id > 0; } + struct Equal + { + using is_transparent = void; - auto operator<=>(const Symbol other) const { return id <=> other.id; } - bool operator==(const Symbol other) const { return id == other.id; } + bool operator()(SymbolStr a, SymbolStr b) const noexcept + { + // strings are unique, so that a pointer comparison is OK + return a.s == b.s; + } - friend class std::hash; + bool operator()(SymbolStr a, const Key & b) const noexcept + { + return a == b.s; + } + + [[gnu::always_inline]] + bool operator()(const Key & a, SymbolStr b) const noexcept + { + return operator()(b, a); + } + }; }; /** @@ -82,29 +197,46 @@ class SymbolTable { private: /** - * Map from string view (backed by ChunkedVector) -> offset into the store. + * SymbolTable is an append only data structure. + * During its lifetime the monotonic buffer holds all strings and nodes, if the symbol set is node based. + */ + std::pmr::monotonic_buffer_resource buffer; + std::pmr::polymorphic_allocator stringAlloc{&buffer}; + SymbolStr::SymbolValueStore store{16}; + + /** + * Transparent lookup of string view for a pointer to a ChunkedVector entry -> return offset into the store. * ChunkedVector references are never invalidated. */ - std::unordered_map symbols; - ChunkedVector store{16}; +#if USE_FLAT_SYMBOL_SET + boost::unordered_flat_set symbols{SymbolStr::chunkSize}; +#else + using SymbolValueAlloc = std::pmr::polymorphic_allocator; + boost::unordered_set symbols{SymbolStr::chunkSize, {&buffer}}; +#endif public: /** * Converts a string into a symbol. */ - Symbol create(std::string_view s) - { + Symbol create(std::string_view s) { // Most symbols are looked up more than once, so we trade off insertion performance // for lookup performance. // FIXME: make this thread-safe. - auto it = symbols.find(s); - if (it != symbols.end()) - return Symbol(it->second + 1); + return [&](T && key) -> Symbol { + if constexpr (requires { symbols.insert(key); }) { + auto [it, _] = symbols.insert(key); + return Symbol(*it); + } else { + auto it = symbols.find(key); + if (it != symbols.end()) + return Symbol(*it); - const auto & [rawSym, idx] = store.add(s); - symbols.emplace(rawSym, idx); - return Symbol(idx + 1); + it = symbols.emplace(key).first; + return Symbol(*it); + } + }(SymbolStr::Key{store, s, stringAlloc}); } std::vector resolve(const std::vector & symbols) const @@ -118,12 +250,14 @@ public: SymbolStr operator[](Symbol s) const { - if (s.id == 0 || s.id > store.size()) + uint32_t idx = s.id - uint32_t(1); + if (idx >= store.size()) unreachable(); - return SymbolStr(store[s.id - 1]); + return store[idx]; } - size_t size() const + [[gnu::always_inline]] + size_t size() const noexcept { return store.size(); } @@ -147,3 +281,5 @@ struct std::hash return std::hash{}(s.id); } }; + +#undef USE_FLAT_SYMBOL_SET diff --git a/src/libexpr/include/nix/expr/value.hh b/src/libexpr/include/nix/expr/value.hh index e9cc1cd3ffa..3a96577a24d 100644 --- a/src/libexpr/include/nix/expr/value.hh +++ b/src/libexpr/include/nix/expr/value.hh @@ -5,7 +5,6 @@ #include #include "nix/expr/eval-gc.hh" -#include "nix/expr/symbol-table.hh" #include "nix/expr/value/context.hh" #include "nix/util/source-path.hh" #include "nix/expr/print-options.hh" @@ -65,6 +64,7 @@ struct ExprLambda; struct ExprBlackHole; struct PrimOp; class Symbol; +class SymbolStr; class PosIdx; struct Pos; class StorePath; @@ -331,10 +331,7 @@ public: void mkStringMove(const char * s, const NixStringContext & context); - inline void mkString(const SymbolStr & s) - { - mkString(s.c_str()); - } + void mkString(const SymbolStr & s); void mkPath(const SourcePath & path); void mkPath(std::string_view path); diff --git a/src/libexpr/nixexpr.cc b/src/libexpr/nixexpr.cc index 1a71096d41e..92071b22d39 100644 --- a/src/libexpr/nixexpr.cc +++ b/src/libexpr/nixexpr.cc @@ -606,7 +606,7 @@ void ExprLambda::setDocComment(DocComment docComment) { size_t SymbolTable::totalSize() const { size_t n = 0; - dump([&] (const std::string & s) { n += s.size(); }); + dump([&] (SymbolStr s) { n += s.size(); }); return n; } From ed4e512dcd4969836663611c57b6e9903c48c5b2 Mon Sep 17 00:00:00 2001 From: Philipp Otterbein Date: Sun, 1 Jun 2025 21:00:01 +0200 Subject: [PATCH 111/218] symbol-table: reference entries instead of allocating `Value`s --- src/libexpr/eval.cc | 10 +++++----- src/libexpr/include/nix/expr/symbol-table.hh | 6 ++++++ src/libexpr/include/nix/expr/value.hh | 7 +++++-- src/libexpr/primops.cc | 8 +++----- 4 files changed, 19 insertions(+), 12 deletions(-) diff --git a/src/libexpr/eval.cc b/src/libexpr/eval.cc index 123f0e86a68..d0069b32125 100644 --- a/src/libexpr/eval.cc +++ b/src/libexpr/eval.cc @@ -90,6 +90,11 @@ std::string printValue(EvalState & state, Value & v) return out.str(); } +Value * Value::toPtr(SymbolStr str) noexcept +{ + return const_cast(str.valuePtr()); +} + void Value::print(EvalState & state, std::ostream & str, PrintOptions options) { printValue(state, str, *this, options); @@ -919,11 +924,6 @@ void Value::mkStringMove(const char * s, const NixStringContext & context) mkString(s, encodeContext(context)); } -void Value::mkString(const SymbolStr & s) -{ - mkString(s.c_str(), nullptr); -} - void Value::mkPath(const SourcePath & path) { mkPath(&*path.accessor, makeImmutableString(path.path.abs())); diff --git a/src/libexpr/include/nix/expr/symbol-table.hh b/src/libexpr/include/nix/expr/symbol-table.hh index 994bddd6c24..1bf5b554340 100644 --- a/src/libexpr/include/nix/expr/symbol-table.hh +++ b/src/libexpr/include/nix/expr/symbol-table.hh @@ -145,6 +145,12 @@ public: return s->size_; } + [[gnu::always_inline]] + const Value * valuePtr() const noexcept + { + return s; + } + explicit operator Symbol() const noexcept { return Symbol{s->idx + 1}; diff --git a/src/libexpr/include/nix/expr/value.hh b/src/libexpr/include/nix/expr/value.hh index 3a96577a24d..febe36f80aa 100644 --- a/src/libexpr/include/nix/expr/value.hh +++ b/src/libexpr/include/nix/expr/value.hh @@ -172,6 +172,11 @@ private: public: + /** + * Never modify the backing `Value` object! + */ + static Value * toPtr(SymbolStr str) noexcept; + void print(EvalState &state, std::ostream &str, PrintOptions options = PrintOptions {}); // Functions needed to distinguish the type @@ -331,8 +336,6 @@ public: void mkStringMove(const char * s, const NixStringContext & context); - void mkString(const SymbolStr & s); - void mkPath(const SourcePath & path); void mkPath(std::string_view path); diff --git a/src/libexpr/primops.cc b/src/libexpr/primops.cc index c4e6feb2800..3017d259ace 100644 --- a/src/libexpr/primops.cc +++ b/src/libexpr/primops.cc @@ -2710,7 +2710,7 @@ static void prim_attrNames(EvalState & state, const PosIdx pos, Value * * args, auto list = state.buildList(args[0]->attrs()->size()); for (const auto & [n, i] : enumerate(*args[0]->attrs())) - (list[n] = state.allocValue())->mkString(state.symbols[i.name]); + list[n] = Value::toPtr(state.symbols[i.name]); std::sort(list.begin(), list.end(), [](Value * v1, Value * v2) { return strcmp(v1->c_str(), v2->c_str()) < 0; }); @@ -3170,9 +3170,8 @@ static void prim_mapAttrs(EvalState & state, const PosIdx pos, Value * * args, V auto attrs = state.buildBindings(args[1]->attrs()->size()); for (auto & i : *args[1]->attrs()) { - Value * vName = state.allocValue(); + Value * vName = Value::toPtr(state.symbols[i.name]); Value * vFun2 = state.allocValue(); - vName->mkString(state.symbols[i.name]); vFun2->mkApp(args[0], vName); attrs.alloc(i.name).mkApp(vFun2, i.value); } @@ -3236,8 +3235,7 @@ static void prim_zipAttrsWith(EvalState & state, const PosIdx pos, Value * * arg auto attrs = state.buildBindings(attrsSeen.size()); for (auto & [sym, elem] : attrsSeen) { - auto name = state.allocValue(); - name->mkString(state.symbols[sym]); + auto name = Value::toPtr(state.symbols[sym]); auto call1 = state.allocValue(); call1->mkApp(args[0], name); auto call2 = state.allocValue(); From 633d39109b292e88bdc98ff622f13153e866eab2 Mon Sep 17 00:00:00 2001 From: Jade Masker Date: Sun, 1 Jun 2025 15:55:49 -0400 Subject: [PATCH 112/218] remove overly verbose mention of fetchTarball Co-authored-by: Valentin Gagarin --- doc/manual/source/command-ref/nix-channel.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/manual/source/command-ref/nix-channel.md b/doc/manual/source/command-ref/nix-channel.md index 4a7555c9058..b1b2812d2ee 100644 --- a/doc/manual/source/command-ref/nix-channel.md +++ b/doc/manual/source/command-ref/nix-channel.md @@ -55,7 +55,7 @@ This command has the following operations: > **Note** > - > `--update` uses [fetchTarball](@docroot@/language/builtins.md#builtins-fetchTarball) under the hood, so it will cache channels. + > Downloaded channel contents are cached. > Use `--tarball-ttl` or the nix configuration option `tarball-ttl` to change this behavior. - `--list-generations` From c0ceaa2d5db4a2ad96c4b7ee475dc53834d6a502 Mon Sep 17 00:00:00 2001 From: Jade Masker Date: Sun, 1 Jun 2025 15:56:46 -0400 Subject: [PATCH 113/218] add reference to the tarball-ttl documentation Co-authored-by: Valentin Gagarin --- doc/manual/source/command-ref/nix-channel.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/manual/source/command-ref/nix-channel.md b/doc/manual/source/command-ref/nix-channel.md index b1b2812d2ee..ed9cbb41fbf 100644 --- a/doc/manual/source/command-ref/nix-channel.md +++ b/doc/manual/source/command-ref/nix-channel.md @@ -56,7 +56,7 @@ This command has the following operations: > **Note** > > Downloaded channel contents are cached. - > Use `--tarball-ttl` or the nix configuration option `tarball-ttl` to change this behavior. + > Use `--tarball-ttl` or the [`tarball-ttl` configuration option](@docroot@/command-ref/conf-file.md#conf-tarball-ttl) to change the validity period of cached downloads. - `--list-generations` From b73e706589d36a11b6b805e9866e006580b34ede Mon Sep 17 00:00:00 2001 From: Sergei Zimmerman Date: Sun, 1 Jun 2025 20:55:16 +0000 Subject: [PATCH 114/218] libutil: Use `std::shared_ptr` instead of `std::shared_ptr` There's actually no mutation happening so there's no point in using a mutable shared_ptr. Furthermore, this makes it much more evident to the reader that no actual mutation (especially in multithreaded case) is happening. Also get rid of redundant constructor that isn't actually used anywhere other than `Pos::operator std::shared_ptr` which just passes in &*this, (identical to just `this`), which can't be nullptr. --- src/libutil/error.cc | 4 ++-- src/libutil/include/nix/util/error.hh | 10 +++++----- src/libutil/include/nix/util/position.hh | 4 +--- src/libutil/logging.cc | 2 +- src/libutil/position.cc | 14 ++------------ 5 files changed, 11 insertions(+), 23 deletions(-) diff --git a/src/libutil/error.cc b/src/libutil/error.cc index 0ceaa4e76da..049555ea3fc 100644 --- a/src/libutil/error.cc +++ b/src/libutil/error.cc @@ -13,7 +13,7 @@ namespace nix { -void BaseError::addTrace(std::shared_ptr && e, HintFmt hint, TracePrint print) +void BaseError::addTrace(std::shared_ptr && e, HintFmt hint, TracePrint print) { err.traces.push_front(Trace { .pos = std::move(e), .hint = hint, .print = print }); } @@ -146,7 +146,7 @@ static bool printUnknownLocations = getEnv("_NIX_EVAL_SHOW_UNKNOWN_LOCATIONS").h * * @return true if a position was printed. */ -static bool printPosMaybe(std::ostream & oss, std::string_view indent, const std::shared_ptr & pos) { +static bool printPosMaybe(std::ostream & oss, std::string_view indent, const std::shared_ptr & pos) { bool hasPos = pos && *pos; if (hasPos) { oss << indent << ANSI_BLUE << "at " ANSI_WARNING << *pos << ANSI_NORMAL << ":"; diff --git a/src/libutil/include/nix/util/error.hh b/src/libutil/include/nix/util/error.hh index fa60d4c61a3..7c96112eac4 100644 --- a/src/libutil/include/nix/util/error.hh +++ b/src/libutil/include/nix/util/error.hh @@ -78,7 +78,7 @@ enum struct TracePrint { }; struct Trace { - std::shared_ptr pos; + std::shared_ptr pos; HintFmt hint; TracePrint print = TracePrint::Default; }; @@ -88,7 +88,7 @@ inline std::strong_ordering operator<=>(const Trace& lhs, const Trace& rhs); struct ErrorInfo { Verbosity level; HintFmt msg; - std::shared_ptr pos; + std::shared_ptr pos; std::list traces; /** * Some messages are generated directly by expressions; notably `builtins.warn`, `abort`, `throw`. @@ -172,7 +172,7 @@ public: err.status = status; } - void atPos(std::shared_ptr pos) { + void atPos(std::shared_ptr pos) { err.pos = pos; } @@ -182,12 +182,12 @@ public: } template - void addTrace(std::shared_ptr && e, std::string_view fs, const Args & ... args) + void addTrace(std::shared_ptr && e, std::string_view fs, const Args & ... args) { addTrace(std::move(e), HintFmt(std::string(fs), args...)); } - void addTrace(std::shared_ptr && e, HintFmt hint, TracePrint print = TracePrint::Default); + void addTrace(std::shared_ptr && e, HintFmt hint, TracePrint print = TracePrint::Default); bool hasTrace() const { return !err.traces.empty(); } diff --git a/src/libutil/include/nix/util/position.hh b/src/libutil/include/nix/util/position.hh index f9c98497695..90de0c0b2d4 100644 --- a/src/libutil/include/nix/util/position.hh +++ b/src/libutil/include/nix/util/position.hh @@ -46,12 +46,10 @@ struct Pos Pos(Pos & other) = default; Pos(const Pos & other) = default; Pos(Pos && other) = default; - Pos(const Pos * other); explicit operator bool() const { return line > 0; } - /* TODO: Why std::shared_ptr and not std::shared_ptr? */ - operator std::shared_ptr() const; + operator std::shared_ptr() const; /** * Return the contents of the source file. diff --git a/src/libutil/logging.cc b/src/libutil/logging.cc index 9b4ecbda87a..4dadf15501f 100644 --- a/src/libutil/logging.cc +++ b/src/libutil/logging.cc @@ -166,7 +166,7 @@ Activity::Activity(Logger & logger, Verbosity lvl, ActivityType type, logger.startActivity(id, lvl, type, s, fields, parent); } -void to_json(nlohmann::json & json, std::shared_ptr pos) +void to_json(nlohmann::json & json, std::shared_ptr pos) { if (pos) { json["line"] = pos->line; diff --git a/src/libutil/position.cc b/src/libutil/position.cc index dfe0e2abb80..a1d9460ed34 100644 --- a/src/libutil/position.cc +++ b/src/libutil/position.cc @@ -2,19 +2,9 @@ namespace nix { -Pos::Pos(const Pos * other) +Pos::operator std::shared_ptr() const { - if (!other) { - return; - } - line = other->line; - column = other->column; - origin = other->origin; -} - -Pos::operator std::shared_ptr() const -{ - return std::make_shared(&*this); + return std::make_shared(*this); } std::optional Pos::getCodeLines() const From cdb8567473999f6728584174961e99849c347fb2 Mon Sep 17 00:00:00 2001 From: Sergei Zimmerman Date: Sun, 1 Jun 2025 20:48:49 +0000 Subject: [PATCH 115/218] libutil: Don't explicitly default special member functions Since all of the member types are copyable/movable the compiler will generate all of those by default anyway. --- src/libutil/include/nix/util/position.hh | 3 --- 1 file changed, 3 deletions(-) diff --git a/src/libutil/include/nix/util/position.hh b/src/libutil/include/nix/util/position.hh index 90de0c0b2d4..34cf86392c1 100644 --- a/src/libutil/include/nix/util/position.hh +++ b/src/libutil/include/nix/util/position.hh @@ -43,9 +43,6 @@ struct Pos Pos() { } Pos(uint32_t line, uint32_t column, Origin origin) : line(line), column(column), origin(origin) { } - Pos(Pos & other) = default; - Pos(const Pos & other) = default; - Pos(Pos && other) = default; explicit operator bool() const { return line > 0; } From 9563b509ff6e30410cf3da2b1fc952119a99118a Mon Sep 17 00:00:00 2001 From: Sergei Zimmerman Date: Sun, 1 Jun 2025 21:16:01 +0000 Subject: [PATCH 116/218] libexpr: Deduplicate `Value::primOpAppPrimOp` `getPrimOp` function was basically identical to existing `Value::primOpAppPrimOp` modulo some trivial differences. Makes sense to reuse existing code for that. --- src/libexpr/eval.cc | 13 +++---------- 1 file changed, 3 insertions(+), 10 deletions(-) diff --git a/src/libexpr/eval.cc b/src/libexpr/eval.cc index ce35901b137..eacbf4c9b8b 100644 --- a/src/libexpr/eval.cc +++ b/src/libexpr/eval.cc @@ -94,15 +94,6 @@ void Value::print(EvalState & state, std::ostream & str, PrintOptions options) printValue(state, str, *this, options); } -const Value * getPrimOp(const Value &v) { - const Value * primOp = &v; - while (primOp->isPrimOpApp()) { - primOp = primOp->payload.primOpApp.left; - } - assert(primOp->isPrimOp()); - return primOp; -} - std::string_view showType(ValueType type, bool withArticle) { #define WA(a, w) withArticle ? a " " w : w @@ -133,7 +124,7 @@ std::string showType(const Value & v) case tPrimOp: return fmt("the built-in function '%s'", std::string(v.payload.primOp->name)); case tPrimOpApp: - return fmt("the partially applied built-in function '%s'", std::string(getPrimOp(v)->payload.primOp->name)); + return fmt("the partially applied built-in function '%s'", v.primOpAppPrimOp()->name); case tExternal: return v.external()->showType(); case tThunk: return v.isBlackhole() ? "a black hole" : "a thunk"; case tApp: return "a function application"; @@ -535,6 +526,8 @@ const PrimOp * Value::primOpAppPrimOp() const if (!left) return nullptr; + + assert(left->isPrimOp()); return left->primOp(); } From afd9c785081d93dca20f76cc1218691ce1a926b3 Mon Sep 17 00:00:00 2001 From: Philipp Otterbein Date: Sun, 1 Jun 2025 18:40:14 +0200 Subject: [PATCH 117/218] libexpr: fix various overflows and type mismatches --- src/libexpr-tests/primops.cc | 11 +++++++++++ src/libexpr/primops.cc | 31 +++++++++++++++++-------------- 2 files changed, 28 insertions(+), 14 deletions(-) diff --git a/src/libexpr-tests/primops.cc b/src/libexpr-tests/primops.cc index 7695a587ae3..2f864f2c25d 100644 --- a/src/libexpr-tests/primops.cc +++ b/src/libexpr-tests/primops.cc @@ -301,6 +301,7 @@ namespace nix { TEST_F(PrimOpTest, elemtAtOutOfBounds) { ASSERT_THROW(eval("builtins.elemAt [0 1 2 3] 5"), Error); + ASSERT_THROW(eval("builtins.elemAt [0] 4294967296"), Error); } TEST_F(PrimOpTest, head) { @@ -592,6 +593,16 @@ namespace nix { ASSERT_THAT(v, IsStringEq("n")); } + TEST_F(PrimOpTest, substringHugeStart){ + auto v = eval("builtins.substring 4294967296 5 \"nixos\""); + ASSERT_THAT(v, IsStringEq("")); + } + + TEST_F(PrimOpTest, substringHugeLength){ + auto v = eval("builtins.substring 0 4294967296 \"nixos\""); + ASSERT_THAT(v, IsStringEq("nixos")); + } + TEST_F(PrimOpTest, substringEmptyString){ auto v = eval("builtins.substring 1 3 \"\""); ASSERT_THAT(v, IsStringEq("")); diff --git a/src/libexpr/primops.cc b/src/libexpr/primops.cc index 01cfb3b0e16..9e8accd2717 100644 --- a/src/libexpr/primops.cc +++ b/src/libexpr/primops.cc @@ -423,7 +423,7 @@ void prim_exec(EvalState & state, const PosIdx pos, Value * * args, Value & v) "while evaluating the first element of the argument passed to builtins.exec", false, false).toOwned(); Strings commandArgs; - for (unsigned int i = 1; i < args[0]->listSize(); ++i) { + for (size_t i = 1; i < count; ++i) { commandArgs.push_back( state.coerceToString(pos, *elems[i], context, "while evaluating an element of the argument passed to builtins.exec", @@ -3106,7 +3106,7 @@ static void prim_catAttrs(EvalState & state, const PosIdx pos, Value * * args, V } auto list = state.buildList(found); - for (unsigned int n = 0; n < found; ++n) + for (size_t n = 0; n < found; ++n) list[n] = res[n]; v.mkList(list); } @@ -3319,7 +3319,7 @@ static void prim_elemAt(EvalState & state, const PosIdx pos, Value * * args, Val { NixInt::Inner n = state.forceInt(*args[1], pos, "while evaluating the second argument passed to 'builtins.elemAt'").value; state.forceList(*args[0], pos, "while evaluating the first argument passed to 'builtins.elemAt'"); - if (n < 0 || (unsigned int) n >= args[0]->listSize()) + if (n < 0 || std::make_unsigned_t(n) >= args[0]->listSize()) state.error( "'builtins.elemAt' called with index %d on a list of size %d", n, @@ -3442,11 +3442,12 @@ static void prim_filter(EvalState & state, const PosIdx pos, Value * * args, Val state.forceFunction(*args[0], pos, "while evaluating the first argument passed to builtins.filter"); - SmallValueVector vs(args[1]->listSize()); + auto len = args[1]->listSize(); + SmallValueVector vs(len); size_t k = 0; bool same = true; - for (unsigned int n = 0; n < args[1]->listSize(); ++n) { + for (size_t n = 0; n < len; ++n) { Value res; state.callFunction(*args[0], *args[1]->listElems()[n], res, noPos); if (state.forceBool(res, pos, "while evaluating the return value of the filtering function passed to builtins.filter")) @@ -3628,7 +3629,7 @@ static void prim_genList(EvalState & state, const PosIdx pos, Value * * args, Va { auto len_ = state.forceInt(*args[1], pos, "while evaluating the second argument passed to builtins.genList").value; - if (len_ < 0) + if (len_ < 0 || std::make_unsigned_t(len_) > std::numeric_limits::max()) state.error("cannot create list of size %1%", len_).atPos(pos).debugThrow(); size_t len = size_t(len_); @@ -3734,7 +3735,7 @@ static void prim_partition(EvalState & state, const PosIdx pos, Value * * args, ValueVector right, wrong; - for (unsigned int n = 0; n < len; ++n) { + for (size_t n = 0; n < len; ++n) { auto vElem = args[1]->listElems()[n]; state.forceValue(*vElem, pos); Value res; @@ -3847,7 +3848,7 @@ static void prim_concatMap(EvalState & state, const PosIdx pos, Value * * args, SmallTemporaryValueVector lists(nrLists); size_t len = 0; - for (unsigned int n = 0; n < nrLists; ++n) { + for (size_t n = 0; n < nrLists; ++n) { Value * vElem = args[1]->listElems()[n]; state.callFunction(*args[0], *vElem, lists[n], pos); state.forceList(lists[n], lists[n].determinePos(args[0]->determinePos(pos)), "while evaluating the return value of the function passed to builtins.concatMap"); @@ -3856,7 +3857,7 @@ static void prim_concatMap(EvalState & state, const PosIdx pos, Value * * args, auto list = state.buildList(len); auto out = list.elems; - for (unsigned int n = 0, pos = 0; n < nrLists; ++n) { + for (size_t n = 0, pos = 0; n < nrLists; ++n) { auto l = lists[n].listSize(); if (l) memcpy(out + pos, lists[n].listElems(), l * sizeof(Value *)); @@ -4121,19 +4122,17 @@ static RegisterPrimOp primop_toString({ non-negative. */ static void prim_substring(EvalState & state, const PosIdx pos, Value * * args, Value & v) { + using NixUInt = std::make_unsigned_t; NixInt::Inner start = state.forceInt(*args[0], pos, "while evaluating the first argument (the start offset) passed to builtins.substring").value; if (start < 0) state.error("negative start position in 'substring'").atPos(pos).debugThrow(); - NixInt::Inner len = state.forceInt(*args[1], pos, "while evaluating the second argument (the substring length) passed to builtins.substring").value; // Negative length may be idiomatically passed to builtins.substring to get // the tail of the string. - if (len < 0) { - len = std::numeric_limits::max(); - } + auto _len = std::numeric_limits::max(); // Special-case on empty substring to avoid O(n) strlen // This allows for the use of empty substrings to efficiently capture string context @@ -4145,10 +4144,14 @@ static void prim_substring(EvalState & state, const PosIdx pos, Value * * args, } } + if (len >= 0 && NixUInt(len) < _len) { + _len = len; + } + NixStringContext context; auto s = state.coerceToString(pos, *args[2], context, "while evaluating the third argument (the string) passed to builtins.substring"); - v.mkString((unsigned int) start >= s->size() ? "" : s->substr(start, len), context); + v.mkString(NixUInt(start) >= s->size() ? "" : s->substr(start, _len), context); } static RegisterPrimOp primop_substring({ From 6b6d3dcf34d8ccbeaec515d216509bcd5038b635 Mon Sep 17 00:00:00 2001 From: Eelco Dolstra Date: Fri, 30 May 2025 14:24:59 +0200 Subject: [PATCH 118/218] deletePath(): Keep going when encountering an undeletable file This should reduce the impact of #5207. --- src/libutil/file-system.cc | 20 ++++++++++++++++---- 1 file changed, 16 insertions(+), 4 deletions(-) diff --git a/src/libutil/file-system.cc b/src/libutil/file-system.cc index 283d19fe669..fee2945ffef 100644 --- a/src/libutil/file-system.cc +++ b/src/libutil/file-system.cc @@ -422,7 +422,7 @@ void recursiveSync(const Path & path) } -static void _deletePath(Descriptor parentfd, const std::filesystem::path & path, uint64_t & bytesFreed MOUNTEDPATHS_PARAM) +static void _deletePath(Descriptor parentfd, const std::filesystem::path & path, uint64_t & bytesFreed, std::exception_ptr & ex MOUNTEDPATHS_PARAM) { #ifndef _WIN32 checkInterrupt(); @@ -488,7 +488,7 @@ static void _deletePath(Descriptor parentfd, const std::filesystem::path & path, checkInterrupt(); std::string childName = dirent->d_name; if (childName == "." || childName == "..") continue; - _deletePath(dirfd(dir.get()), path + "/" + childName, bytesFreed MOUNTEDPATHS_ARG); + _deletePath(dirfd(dir.get()), path + "/" + childName, bytesFreed, ex MOUNTEDPATHS_ARG); } if (errno) throw SysError("reading directory %1%", path); } @@ -496,7 +496,14 @@ static void _deletePath(Descriptor parentfd, const std::filesystem::path & path, int flags = S_ISDIR(st.st_mode) ? AT_REMOVEDIR : 0; if (unlinkat(parentfd, name.c_str(), flags) == -1) { if (errno == ENOENT) return; - throw SysError("cannot unlink %1%", path); + try { + throw SysError("cannot unlink %1%", path); + } catch (...) { + if (!ex) + ex = std::current_exception(); + else + ignoreExceptionExceptInterrupt(); + } } #else // TODO implement @@ -516,7 +523,12 @@ static void _deletePath(const std::filesystem::path & path, uint64_t & bytesFree throw SysError("opening directory '%1%'", path); } - _deletePath(dirfd.get(), path, bytesFreed MOUNTEDPATHS_ARG); + std::exception_ptr ex; + + _deletePath(dirfd.get(), path, bytesFreed, ex MOUNTEDPATHS_ARG); + + if (ex) + std::rethrow_exception(ex); } From 0712339912a4b139efa87768fd3a38713e0e13ac Mon Sep 17 00:00:00 2001 From: Cole Helbling Date: Mon, 2 Jun 2025 14:28:43 -0700 Subject: [PATCH 119/218] `--keep-failed` with remote builders will keep the failed build directory on that builder --- src/build-remote/build-remote.cc | 9 ++++++++- tests/functional/build-remote.sh | 1 + 2 files changed, 9 insertions(+), 1 deletion(-) diff --git a/src/build-remote/build-remote.cc b/src/build-remote/build-remote.cc index 60247b73592..e79ee6a45ca 100644 --- a/src/build-remote/build-remote.cc +++ b/src/build-remote/build-remote.cc @@ -329,8 +329,15 @@ static int main_build_remote(int argc, char * * argv) drv.inputSrcs = store->parseStorePathSet(inputs); optResult = sshStore->buildDerivation(*drvPath, (const BasicDerivation &) drv); auto & result = *optResult; - if (!result.success()) + if (!result.success()) { + if (settings.keepFailed) { + warn( + "The failed build directory was kept on the remote builder due to `--keep-failed`. " + "If the build's architecture matches your host, you can re-run the command with `--builders ''` to disable remote building for this invocation." + ); + } throw Error("build of '%s' on '%s' failed: %s", store->printStorePath(*drvPath), storeUri, result.errorMsg); + } } else { copyClosure(*store, *sshStore, StorePathSet {*drvPath}, NoRepair, NoCheckSigs, substitute); auto res = sshStore->buildPathsWithResults({ diff --git a/tests/functional/build-remote.sh b/tests/functional/build-remote.sh index 3231341cbf6..2d3e874b1b4 100644 --- a/tests/functional/build-remote.sh +++ b/tests/functional/build-remote.sh @@ -85,6 +85,7 @@ out="$(nix-build 2>&1 failing.nix \ --arg busybox "$busybox")" || true [[ "$out" =~ .*"note: keeping build directory".* ]] +[[ "$out" =~ .*"The failed build directory was kept on the remote builder due to".* ]] build_dir="$(grep "note: keeping build" <<< "$out" | sed -E "s/^(.*)note: keeping build directory '(.*)'(.*)$/\2/")" [[ "foo" = $(<"$build_dir"/bar) ]] From 54aa73b19bd3bb3f86c1cfae7eaca669923c9313 Mon Sep 17 00:00:00 2001 From: Cole Helbling Date: Tue, 3 Jun 2025 08:38:04 -0700 Subject: [PATCH 120/218] fixup: only show "you can rerun" message if the derivation's platform is supported on this machine --- src/build-remote/build-remote.cc | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/src/build-remote/build-remote.cc b/src/build-remote/build-remote.cc index e79ee6a45ca..e0a6b2d8e10 100644 --- a/src/build-remote/build-remote.cc +++ b/src/build-remote/build-remote.cc @@ -332,8 +332,10 @@ static int main_build_remote(int argc, char * * argv) if (!result.success()) { if (settings.keepFailed) { warn( - "The failed build directory was kept on the remote builder due to `--keep-failed`. " - "If the build's architecture matches your host, you can re-run the command with `--builders ''` to disable remote building for this invocation." + "The failed build directory was kept on the remote builder due to `--keep-failed`.%s", + (settings.thisSystem == drv.platform || settings.extraPlatforms.get().count(drv.platform) > 0) + ? " You can re-run the command with `--builders ''` to disable remote building for this invocation." + : "" ); } throw Error("build of '%s' on '%s' failed: %s", store->printStorePath(*drvPath), storeUri, result.errorMsg); From cfc15d6921fbbea98df0ea1b5ddf139221da4ef7 Mon Sep 17 00:00:00 2001 From: PopeRigby Date: Tue, 3 Jun 2025 13:26:09 -0700 Subject: [PATCH 121/218] Modify docker.nix to use mapAttrsToList instead of mapAttrsFlatten The latter alias is deprecated in favor of the former, and produces a warning. --- docker.nix | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docker.nix b/docker.nix index a92d478d925..c418a9e6254 100644 --- a/docker.nix +++ b/docker.nix @@ -155,7 +155,7 @@ let nixConfContents = (lib.concatStringsSep "\n" ( - lib.mapAttrsFlatten ( + lib.mapAttrsToList ( n: v: let vStr = if builtins.isList v then lib.concatStringsSep " " v else v; From 4bce2d723d044fdf425626fa068f3bfd60be82aa Mon Sep 17 00:00:00 2001 From: Eelco Dolstra Date: Wed, 4 Jun 2025 21:37:17 +0200 Subject: [PATCH 122/218] GitSourceAccessor: Make thread-safe --- src/libfetchers/git-utils.cc | 91 ++++++++++++++++++++++-------------- 1 file changed, 55 insertions(+), 36 deletions(-) diff --git a/src/libfetchers/git-utils.cc b/src/libfetchers/git-utils.cc index c1bdf20974a..9fe271fe8ce 100644 --- a/src/libfetchers/git-utils.cc +++ b/src/libfetchers/git-utils.cc @@ -665,28 +665,40 @@ ref GitRepo::openRepo(const std::filesystem::path & path, bool create, struct GitSourceAccessor : SourceAccessor { - ref repo; - Object root; - std::optional lfsFetch = std::nullopt; + struct State + { + ref repo; + Object root; + std::optional lfsFetch = std::nullopt; + }; + + Sync state_; GitSourceAccessor(ref repo_, const Hash & rev, bool smudgeLfs) - : repo(repo_) - , root(peelToTreeOrBlob(lookupObject(*repo, hashToOID(rev)).get())) + : state_{ + State { + .repo = repo_, + .root = peelToTreeOrBlob(lookupObject(*repo_, hashToOID(rev)).get()), + .lfsFetch = smudgeLfs ? std::make_optional(lfs::Fetch(*repo_, hashToOID(rev))) : std::nullopt, + } + } { - if (smudgeLfs) - lfsFetch = std::make_optional(lfs::Fetch(*repo, hashToOID(rev))); } std::string readBlob(const CanonPath & path, bool symlink) { - const auto blob = getBlob(path, symlink); + auto state(state_.lock()); + + const auto blob = getBlob(*state, path, symlink); - if (lfsFetch) { - if (lfsFetch->shouldFetch(path)) { + if (state->lfsFetch) { + if (state->lfsFetch->shouldFetch(path)) { StringSink s; try { + // FIXME: do we need to hold the state lock while + // doing this? auto contents = std::string((const char *) git_blob_rawcontent(blob.get()), git_blob_rawsize(blob.get())); - lfsFetch->fetch(contents, path, s, [&s](uint64_t size){ s.s.reserve(size); }); + state->lfsFetch->fetch(contents, path, s, [&s](uint64_t size){ s.s.reserve(size); }); } catch (Error & e) { e.addTrace({}, "while smudging git-lfs file '%s'", path); throw; @@ -705,15 +717,18 @@ struct GitSourceAccessor : SourceAccessor bool pathExists(const CanonPath & path) override { - return path.isRoot() ? true : (bool) lookup(path); + auto state(state_.lock()); + return path.isRoot() ? true : (bool) lookup(*state, path); } std::optional maybeLstat(const CanonPath & path) override { + auto state(state_.lock()); + if (path.isRoot()) - return Stat { .type = git_object_type(root.get()) == GIT_OBJECT_TREE ? tDirectory : tRegular }; + return Stat { .type = git_object_type(state->root.get()) == GIT_OBJECT_TREE ? tDirectory : tRegular }; - auto entry = lookup(path); + auto entry = lookup(*state, path); if (!entry) return std::nullopt; @@ -741,6 +756,8 @@ struct GitSourceAccessor : SourceAccessor DirEntries readDirectory(const CanonPath & path) override { + auto state(state_.lock()); + return std::visit(overloaded { [&](Tree tree) { DirEntries res; @@ -758,7 +775,7 @@ struct GitSourceAccessor : SourceAccessor [&](Submodule) { return DirEntries(); } - }, getTree(path)); + }, getTree(*state, path)); } std::string readLink(const CanonPath & path) override @@ -772,7 +789,9 @@ struct GitSourceAccessor : SourceAccessor */ std::optional getSubmoduleRev(const CanonPath & path) { - auto entry = lookup(path); + auto state(state_.lock()); + + auto entry = lookup(*state, path); if (!entry || git_tree_entry_type(entry) != GIT_OBJECT_COMMIT) return std::nullopt; @@ -783,7 +802,7 @@ struct GitSourceAccessor : SourceAccessor std::unordered_map lookupCache; /* Recursively look up 'path' relative to the root. */ - git_tree_entry * lookup(const CanonPath & path) + git_tree_entry * lookup(State & state, const CanonPath & path) { auto i = lookupCache.find(path); if (i != lookupCache.end()) return i->second.get(); @@ -793,7 +812,7 @@ struct GitSourceAccessor : SourceAccessor auto name = path.baseName().value(); - auto parentTree = lookupTree(*parent); + auto parentTree = lookupTree(state, *parent); if (!parentTree) return nullptr; auto count = git_tree_entrycount(parentTree->get()); @@ -822,29 +841,29 @@ struct GitSourceAccessor : SourceAccessor return res; } - std::optional lookupTree(const CanonPath & path) + std::optional lookupTree(State & state, const CanonPath & path) { if (path.isRoot()) { - if (git_object_type(root.get()) == GIT_OBJECT_TREE) - return dupObject((git_tree *) &*root); + if (git_object_type(state.root.get()) == GIT_OBJECT_TREE) + return dupObject((git_tree *) &*state.root); else return std::nullopt; } - auto entry = lookup(path); + auto entry = lookup(state, path); if (!entry || git_tree_entry_type(entry) != GIT_OBJECT_TREE) return std::nullopt; Tree tree; - if (git_tree_entry_to_object((git_object * *) (git_tree * *) Setter(tree), *repo, entry)) + if (git_tree_entry_to_object((git_object * *) (git_tree * *) Setter(tree), *state.repo, entry)) throw Error("looking up directory '%s': %s", showPath(path), git_error_last()->message); return tree; } - git_tree_entry * need(const CanonPath & path) + git_tree_entry * need(State & state, const CanonPath & path) { - auto entry = lookup(path); + auto entry = lookup(state, path); if (!entry) throw Error("'%s' does not exist", showPath(path)); return entry; @@ -852,16 +871,16 @@ struct GitSourceAccessor : SourceAccessor struct Submodule { }; - std::variant getTree(const CanonPath & path) + std::variant getTree(State & state, const CanonPath & path) { if (path.isRoot()) { - if (git_object_type(root.get()) == GIT_OBJECT_TREE) - return dupObject((git_tree *) &*root); + if (git_object_type(state.root.get()) == GIT_OBJECT_TREE) + return dupObject((git_tree *) &*state.root); else - throw Error("Git root object '%s' is not a directory", *git_object_id(root.get())); + throw Error("Git root object '%s' is not a directory", *git_object_id(state.root.get())); } - auto entry = need(path); + auto entry = need(state, path); if (git_tree_entry_type(entry) == GIT_OBJECT_COMMIT) return Submodule(); @@ -870,16 +889,16 @@ struct GitSourceAccessor : SourceAccessor throw Error("'%s' is not a directory", showPath(path)); Tree tree; - if (git_tree_entry_to_object((git_object * *) (git_tree * *) Setter(tree), *repo, entry)) + if (git_tree_entry_to_object((git_object * *) (git_tree * *) Setter(tree), *state.repo, entry)) throw Error("looking up directory '%s': %s", showPath(path), git_error_last()->message); return tree; } - Blob getBlob(const CanonPath & path, bool expectSymlink) + Blob getBlob(State & state, const CanonPath & path, bool expectSymlink) { - if (!expectSymlink && git_object_type(root.get()) == GIT_OBJECT_BLOB) - return dupObject((git_blob *) &*root); + if (!expectSymlink && git_object_type(state.root.get()) == GIT_OBJECT_BLOB) + return dupObject((git_blob *) &*state.root); auto notExpected = [&]() { @@ -892,7 +911,7 @@ struct GitSourceAccessor : SourceAccessor if (path.isRoot()) notExpected(); - auto entry = need(path); + auto entry = need(state, path); if (git_tree_entry_type(entry) != GIT_OBJECT_BLOB) notExpected(); @@ -907,7 +926,7 @@ struct GitSourceAccessor : SourceAccessor } Blob blob; - if (git_tree_entry_to_object((git_object * *) (git_blob * *) Setter(blob), *repo, entry)) + if (git_tree_entry_to_object((git_object * *) (git_blob * *) Setter(blob), *state.repo, entry)) throw Error("looking up file '%s': %s", showPath(path), git_error_last()->message); return blob; From 0a87ba0e392ca890111d4ed7262e65c170751cb5 Mon Sep 17 00:00:00 2001 From: Eelco Dolstra Date: Thu, 5 Jun 2025 13:09:57 +0200 Subject: [PATCH 123/218] Prevent double copy of nixpkgs source tree --- docker.nix | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/docker.nix b/docker.nix index c418a9e6254..e3a0635d017 100644 --- a/docker.nix +++ b/docker.nix @@ -173,7 +173,12 @@ let channel = pkgs.runCommand "channel-nixos" { inherit bundleNixpkgs; } '' mkdir $out if [ "$bundleNixpkgs" ]; then - ln -s ${nixpkgs} $out/nixpkgs + ln -s ${ + builtins.path { + path = nixpkgs; + name = "source"; + } + } $out/nixpkgs echo "[]" > $out/manifest.nix fi ''; From e72a0ad8c338be5573a295db62748bc88d7ea4a4 Mon Sep 17 00:00:00 2001 From: h0nIg Date: Thu, 5 Jun 2025 23:28:47 +0200 Subject: [PATCH 124/218] docker: add docu references & remove duplicate code --- docker.nix | 27 +++++++++++---------------- 1 file changed, 11 insertions(+), 16 deletions(-) diff --git a/docker.nix b/docker.nix index c418a9e6254..c6905b24603 100644 --- a/docker.nix +++ b/docker.nix @@ -147,23 +147,11 @@ let "${k}:x:${toString gid}:${lib.concatStringsSep "," members}"; groupContents = (lib.concatStringsSep "\n" (lib.attrValues (lib.mapAttrs groupToGroup groups))); - defaultNixConf = { - sandbox = "false"; - build-users-group = "nixbld"; - trusted-public-keys = [ "cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=" ]; - }; - nixConfContents = - (lib.concatStringsSep "\n" ( - lib.mapAttrsToList ( - n: v: - let - vStr = if builtins.isList v then lib.concatStringsSep " " v else v; - in - "${n} = ${vStr}" - ) (defaultNixConf // nixConf) - )) - + "\n"; + pkgs.dockerTools.nixConf + { + build-users-group = "nixbld"; + }; userHome = if uid == 0 then "/root" else "/home/${uname}"; @@ -181,6 +169,8 @@ let name = "root-profile-env"; paths = defaultPkgs; }; + # doc/manual/source/command-ref/files/manifest.nix.md + # may get replaced by pkgs.buildEnv once manifest.json can get written manifest = pkgs.buildPackages.runCommand "manifest.nix" { } '' cat > $out < $out${userHome}/.nix-channels + # may get replaced by pkgs.dockerTools.binSh & pkgs.dockerTools.usrBinEnv mkdir -p $out/bin $out/usr/bin ln -s ${pkgs.coreutils}/bin/env $out/usr/bin/env ln -s ${pkgs.bashInteractive}/bin/bash $out/bin/sh From 3c9b9b13af52776ca08bbbb12b3a98f45de6083b Mon Sep 17 00:00:00 2001 From: Kevin Robert Stravers Date: Sat, 21 Sep 2024 02:56:04 -0400 Subject: [PATCH 125/218] nix repl: Print which variables are just loaded When we run `nix repl nixpkgs` we get "Added 6 variables". This is not useful as it doesn't tell us which variables the flake has exported to our global repl scope. This patch prints the name of each variable that was just loaded. We currently cap printing to 20 variables in order to avoid excessive prints. https://github.com/NixOS/nix/issues/11404 --- src/libcmd/repl.cc | 19 +++++++++++++++++++ tests/functional/repl.sh | 29 ++++++++++++++++++++++++++++- 2 files changed, 47 insertions(+), 1 deletion(-) diff --git a/src/libcmd/repl.cc b/src/libcmd/repl.cc index 07968fa4360..e5164b1378c 100644 --- a/src/libcmd/repl.cc +++ b/src/libcmd/repl.cc @@ -813,6 +813,25 @@ void NixRepl::addAttrsToScope(Value & attrs) staticEnv->sort(); staticEnv->deduplicate(); notice("Added %1% variables.", attrs.attrs()->size()); + + const int max_print = 20; + int counter = 0; + std::ostringstream loaded; + for (auto & i : attrs.attrs()->lexicographicOrder(state->symbols)) { + if (counter >= max_print) + break; + + if (counter > 0) + loaded << ", "; + + printIdentifier(loaded, state->symbols[i->name]); + counter += 1; + } + + notice("%1%", loaded.str()); + + if (attrs.attrs()->size() > max_print) + notice("... and %1% more", attrs.attrs()->size() - max_print); } diff --git a/tests/functional/repl.sh b/tests/functional/repl.sh index 762636e446e..7f5b269137a 100755 --- a/tests/functional/repl.sh +++ b/tests/functional/repl.sh @@ -157,6 +157,32 @@ foo + baz ' "3" \ ./flake ./flake\#bar --experimental-features 'flakes' +testReplResponse $' +:a { a = 1; b = 2; longerName = 3; "with spaces" = 4; } +' 'Added 4 variables. +a, b, longerName, "with spaces" +' + +cat < attribute-set.nix +{ + a = 1; + b = 2; + longerName = 3; + "with spaces" = 4; +} +EOF +testReplResponse ' +:l ./attribute-set.nix +' 'Added 4 variables. +a, b, longerName, "with spaces" +' + +testReplResponseNoRegex $' +:a builtins.foldl\' (x: y: x // y) {} (map (x: { ${builtins.toString x} = x; }) (builtins.genList (x: x) 23)) +' 'Added 23 variables. +"0", "1", "10", "11", "12", "13", "14", "15", "16", "17", "18", "19", "2", "20", "21", "22", "3", "4", "5", "6" +... and 3 more' + # Test the `:reload` mechansim with flakes: # - Eval `./flake#changingThing` # - Modify the flake @@ -322,7 +348,8 @@ runRepl () { -e "s@$testDir@/path/to/tests/functional@g" \ -e "s@$testDirNoUnderscores@/path/to/tests/functional@g" \ -e "s@$nixVersion@@g" \ - -e "s@Added [0-9]* variables@Added variables@g" \ + -e "/Added [0-9]* variables/{s@ [0-9]* @ @;n;d}" \ + -e '/\.\.\. and [0-9]* more/d' \ | grep -vF $'warning: you don\'t have Internet access; disabling some network-dependent features' \ ; } From 13e37043292df2b39c0252b1d476dad4c4aa8cce Mon Sep 17 00:00:00 2001 From: Kevin Robert Stravers Date: Sat, 21 Sep 2024 02:58:18 -0400 Subject: [PATCH 126/218] nix repl: Add `:ll` to show all recently loaded variables Invoking `:ll` will start a pager with all variables which have just been loaded by `:lf`, `:l`, or by a flake provided to `nix repl` as an argument. https://github.com/NixOS/nix/issues/11404 --- src/libcmd/repl.cc | 21 ++++++++++++++++++++- tests/functional/repl.sh | 4 ++-- 2 files changed, 22 insertions(+), 3 deletions(-) diff --git a/src/libcmd/repl.cc b/src/libcmd/repl.cc index e5164b1378c..54a002765d5 100644 --- a/src/libcmd/repl.cc +++ b/src/libcmd/repl.cc @@ -69,6 +69,7 @@ struct NixRepl const static int envSize = 32768; std::shared_ptr staticEnv; + Value lastLoaded; Env * env; int displ; StringSet varNames; @@ -95,6 +96,7 @@ struct NixRepl void loadFiles(); void loadFlakes(); void reloadFilesAndFlakes(); + void showLastLoaded(); void addAttrsToScope(Value & attrs); void addVarToScope(const Symbol name, Value & v); Expr * parseString(std::string s); @@ -378,6 +380,7 @@ ProcessLineResult NixRepl::processLine(std::string line) << " current profile\n" << " :l, :load Load Nix expression and add it to scope\n" << " :lf, :load-flake Load Nix flake and add it to scope\n" + << " :ll, :last-loaded Show most recently loaded variables added to scope\n" << " :p, :print Evaluate and print expression recursively\n" << " Strings are printed directly, without escaping.\n" << " :q, :quit Exit nix-repl\n" @@ -468,6 +471,10 @@ ProcessLineResult NixRepl::processLine(std::string line) loadFlake(arg); } + else if (command == ":ll" || command == ":last-loaded") { + showLastLoaded(); + } + else if (command == ":r" || command == ":reload") { state->resetFileCache(); reloadFilesAndFlakes(); @@ -760,6 +767,16 @@ void NixRepl::initEnv() varNames.emplace(state->symbols[i.first]); } +void NixRepl::showLastLoaded() +{ + RunPager pager; + + for (auto & i : *lastLoaded.attrs()) { + std::string_view name = state->symbols[i.name]; + logger->cout(name); + } +} + void NixRepl::reloadFilesAndFlakes() { @@ -814,6 +831,8 @@ void NixRepl::addAttrsToScope(Value & attrs) staticEnv->deduplicate(); notice("Added %1% variables.", attrs.attrs()->size()); + lastLoaded = attrs; + const int max_print = 20; int counter = 0; std::ostringstream loaded; @@ -831,7 +850,7 @@ void NixRepl::addAttrsToScope(Value & attrs) notice("%1%", loaded.str()); if (attrs.attrs()->size() > max_print) - notice("... and %1% more", attrs.attrs()->size() - max_print); + notice("... and %1% more; view with :ll", attrs.attrs()->size() - max_print); } diff --git a/tests/functional/repl.sh b/tests/functional/repl.sh index 7f5b269137a..7c4bae4ba8d 100755 --- a/tests/functional/repl.sh +++ b/tests/functional/repl.sh @@ -181,7 +181,7 @@ testReplResponseNoRegex $' :a builtins.foldl\' (x: y: x // y) {} (map (x: { ${builtins.toString x} = x; }) (builtins.genList (x: x) 23)) ' 'Added 23 variables. "0", "1", "10", "11", "12", "13", "14", "15", "16", "17", "18", "19", "2", "20", "21", "22", "3", "4", "5", "6" -... and 3 more' +... and 3 more; view with :ll' # Test the `:reload` mechansim with flakes: # - Eval `./flake#changingThing` @@ -349,7 +349,7 @@ runRepl () { -e "s@$testDirNoUnderscores@/path/to/tests/functional@g" \ -e "s@$nixVersion@@g" \ -e "/Added [0-9]* variables/{s@ [0-9]* @ @;n;d}" \ - -e '/\.\.\. and [0-9]* more/d' \ + -e '/\.\.\. and [0-9]* more; view with :ll/d' \ | grep -vF $'warning: you don\'t have Internet access; disabling some network-dependent features' \ ; } From d8b067b549f90fc57f295debe40eccb4145e795c Mon Sep 17 00:00:00 2001 From: Eelco Dolstra Date: Fri, 6 Jun 2025 14:04:44 +0200 Subject: [PATCH 127/218] repl: Don't wait on incomplete parses from imported file Fixes #13332. --- src/libcmd/repl.cc | 24 +++++++++++++----------- tests/functional/repl.sh | 6 ++++++ 2 files changed, 19 insertions(+), 11 deletions(-) diff --git a/src/libcmd/repl.cc b/src/libcmd/repl.cc index 07968fa4360..75a0baebc59 100644 --- a/src/libcmd/repl.cc +++ b/src/libcmd/repl.cc @@ -158,6 +158,8 @@ static std::ostream & showDebugTrace(std::ostream & out, const PosTable & positi return out; } +MakeError(IncompleteReplExpr, ParseError); + static bool isFirstRepl = true; ReplExitStatus NixRepl::mainLoop() @@ -205,16 +207,8 @@ ReplExitStatus NixRepl::mainLoop() default: unreachable(); } - } catch (ParseError & e) { - if (e.msg().find("unexpected end of file") != std::string::npos) { - // For parse errors on incomplete input, we continue waiting for the next line of - // input without clearing the input so far. - continue; - } else { - printMsg(lvlError, e.msg()); - } - } catch (EvalError & e) { - printMsg(lvlError, e.msg()); + } catch (IncompleteReplExpr &) { + continue; } catch (Error & e) { printMsg(lvlError, e.msg()); } catch (Interrupted & e) { @@ -837,7 +831,15 @@ Expr * NixRepl::parseString(std::string s) void NixRepl::evalString(std::string s, Value & v) { - Expr * e = parseString(s); + Expr * e; + try { + e = parseString(s); + } catch (ParseError & e) { + if (e.msg().find("unexpected end of file") != std::string::npos) + // For parse errors on incomplete input, we continue waiting for the next line of + // input without clearing the input so far. + throw IncompleteReplExpr(e.msg()); + } e->eval(*state, *env, v); state->forceValue(v, v.determinePos(noPos)); } diff --git a/tests/functional/repl.sh b/tests/functional/repl.sh index 762636e446e..015d296d5bd 100755 --- a/tests/functional/repl.sh +++ b/tests/functional/repl.sh @@ -277,6 +277,12 @@ testReplResponseNoRegex ' } ' +# Don't prompt for more input when getting unexpected EOF in imported files. +testReplResponse " +import $testDir/lang/parse-fail-eof-pos.nix +" \ +'.*error: syntax error, unexpected end of file.*' + # TODO: move init to characterisation/framework.sh badDiff=0 badExitCode=0 From 91b3573770d4628b1219577a381f79b8cb3fafe9 Mon Sep 17 00:00:00 2001 From: Eelco Dolstra Date: Fri, 6 Jun 2025 17:09:01 +0200 Subject: [PATCH 128/218] Rethrow non-EOF errors --- src/libcmd/repl.cc | 2 ++ 1 file changed, 2 insertions(+) diff --git a/src/libcmd/repl.cc b/src/libcmd/repl.cc index 75a0baebc59..de8558e5f99 100644 --- a/src/libcmd/repl.cc +++ b/src/libcmd/repl.cc @@ -839,6 +839,8 @@ void NixRepl::evalString(std::string s, Value & v) // For parse errors on incomplete input, we continue waiting for the next line of // input without clearing the input so far. throw IncompleteReplExpr(e.msg()); + else + throw; } e->eval(*state, *env, v); state->forceValue(v, v.determinePos(noPos)); From 2caccbed11ed5a517ed3fee86f1da3cdbff60211 Mon Sep 17 00:00:00 2001 From: h0nIg Date: Fri, 6 Jun 2025 23:54:15 +0200 Subject: [PATCH 129/218] docker: shrink code - use buildenv.manifest --- docker.nix | 19 +++++++++---------- 1 file changed, 9 insertions(+), 10 deletions(-) diff --git a/docker.nix b/docker.nix index c6905b24603..1401dc6c738 100644 --- a/docker.nix +++ b/docker.nix @@ -165,12 +165,7 @@ let echo "[]" > $out/manifest.nix fi ''; - rootEnv = pkgs.buildPackages.buildEnv { - name = "root-profile-env"; - paths = defaultPkgs; - }; # doc/manual/source/command-ref/files/manifest.nix.md - # may get replaced by pkgs.buildEnv once manifest.json can get written manifest = pkgs.buildPackages.runCommand "manifest.nix" { } '' cat > $out < Date: Tue, 10 Jun 2025 13:19:03 +0000 Subject: [PATCH 130/218] libexpr: Use `primOp` getter --- src/libexpr/eval.cc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/libexpr/eval.cc b/src/libexpr/eval.cc index 92516db3b1f..752938339a4 100644 --- a/src/libexpr/eval.cc +++ b/src/libexpr/eval.cc @@ -128,7 +128,7 @@ std::string showType(const Value & v) switch (v.internalType) { case tString: return v.payload.string.context ? "a string with context" : "a string"; case tPrimOp: - return fmt("the built-in function '%s'", std::string(v.payload.primOp->name)); + return fmt("the built-in function '%s'", std::string(v.primOp()->name)); case tPrimOpApp: return fmt("the partially applied built-in function '%s'", v.primOpAppPrimOp()->name); case tExternal: return v.external()->showType(); From 77f5f50ec2378c75fddfaa33c45e3b885df465e6 Mon Sep 17 00:00:00 2001 From: Sergei Zimmerman Date: Tue, 10 Jun 2025 13:22:20 +0000 Subject: [PATCH 131/218] libexpr: Use `context` getter --- src/libexpr/eval.cc | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/src/libexpr/eval.cc b/src/libexpr/eval.cc index 752938339a4..7346a76514c 100644 --- a/src/libexpr/eval.cc +++ b/src/libexpr/eval.cc @@ -126,7 +126,7 @@ std::string showType(const Value & v) #pragma GCC diagnostic push #pragma GCC diagnostic ignored "-Wswitch-enum" switch (v.internalType) { - case tString: return v.payload.string.context ? "a string with context" : "a string"; + case tString: return v.context() ? "a string with context" : "a string"; case tPrimOp: return fmt("the built-in function '%s'", std::string(v.primOp()->name)); case tPrimOpApp: @@ -2297,8 +2297,8 @@ std::string_view EvalState::forceString(Value & v, const PosIdx pos, std::string void copyContext(const Value & v, NixStringContext & context, const ExperimentalFeatureSettings & xpSettings) { - if (v.payload.string.context) - for (const char * * p = v.payload.string.context; *p; ++p) + if (v.context()) + for (const char * * p = v.context(); *p; ++p) context.insert(NixStringContextElem::parse(*p, xpSettings)); } From 408873c2f7e92063d380b9ed36334de6547e887e Mon Sep 17 00:00:00 2001 From: Sergei Zimmerman Date: Tue, 10 Jun 2025 13:23:12 +0000 Subject: [PATCH 132/218] libexpr: Use `c_str` getter --- src/libexpr/include/nix/expr/symbol-table.hh | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/libexpr/include/nix/expr/symbol-table.hh b/src/libexpr/include/nix/expr/symbol-table.hh index 1bf5b554340..20a05a09d35 100644 --- a/src/libexpr/include/nix/expr/symbol-table.hh +++ b/src/libexpr/include/nix/expr/symbol-table.hh @@ -29,7 +29,7 @@ class SymbolValue : protected Value public: operator std::string_view() const noexcept { - return {payload.string.c_str, size_}; + return {c_str(), size_}; } }; @@ -122,7 +122,7 @@ public: [[gnu::always_inline]] const char * c_str() const noexcept { - return s->payload.string.c_str; + return s->c_str(); } [[gnu::always_inline]] From 525078c59d8391b866f723338db1ee525a4a88cb Mon Sep 17 00:00:00 2001 From: Luc Perkins Date: Wed, 11 Jun 2025 08:52:04 -0700 Subject: [PATCH 133/218] Fix broken link in configuration description --- src/libstore/include/nix/store/globals.hh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/libstore/include/nix/store/globals.hh b/src/libstore/include/nix/store/globals.hh index 5fae2be2395..00d92923671 100644 --- a/src/libstore/include/nix/store/globals.hh +++ b/src/libstore/include/nix/store/globals.hh @@ -365,7 +365,7 @@ public: To build only on remote machines and disable local builds, set [`max-jobs`](#conf-max-jobs) to 0. - If you want the remote machines to use substituters, set [`builders-use-substitutes`](#conf-builders-use-substituters) to `true`. + If you want the remote machines to use substituters, set [`builders-use-substitutes`](#conf-builders-use-substitutes) to `true`. )", {}, false}; From 9eb46e9cc030016b1f4a073474a836bac1de3615 Mon Sep 17 00:00:00 2001 From: Eelco Dolstra Date: Wed, 11 Jun 2025 19:14:31 +0200 Subject: [PATCH 134/218] Make the repl test more robust Seen in https://github.com/DeterminateSystems/nix-src/actions/runs/15590867877/job/43909540271: nix-functional-tests> grep: repl_output: No such file or directory nix-functional-tests> +(repl.sh:174) cat repl_output This is because there is a small possibility that the `nix repl` child process hasn't created `repl_output` yet. So make sure it exists. --- tests/functional/repl.sh | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/tests/functional/repl.sh b/tests/functional/repl.sh index 5c699362166..6db9e2d3d63 100755 --- a/tests/functional/repl.sh +++ b/tests/functional/repl.sh @@ -189,7 +189,8 @@ testReplResponseNoRegex $' # - Re-eval it # - Check that the result has changed mkfifo repl_fifo -nix repl ./flake --experimental-features 'flakes' < repl_fifo > repl_output 2>&1 & +touch repl_output +nix repl ./flake --experimental-features 'flakes' < repl_fifo >> repl_output 2>&1 & repl_pid=$! exec 3>repl_fifo # Open fifo for writing echo "changingThing" >&3 From f42eaf2c8ea3c7e53d1fa15f694144f46006eaf5 Mon Sep 17 00:00:00 2001 From: John Ericson Date: Wed, 11 Jun 2025 16:11:38 -0400 Subject: [PATCH 135/218] Create test for #13293 It currently fails, before the fix. --- tests/functional/check.sh | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/tests/functional/check.sh b/tests/functional/check.sh index b213492889f..a1c6decf5b5 100755 --- a/tests/functional/check.sh +++ b/tests/functional/check.sh @@ -22,6 +22,11 @@ clearStore nix-build dependencies.nix --no-out-link nix-build dependencies.nix --no-out-link --check +# Make sure checking just one output works (#13293) +nix-build multiple-outputs.nix -A a --no-out-link +nix-store --delete "$(nix-build multiple-outputs.nix -A a.second --no-out-link)" +nix-build multiple-outputs.nix -A a.first --no-out-link --check + # Build failure exit codes (100, 104, etc.) are from # doc/manual/source/command-ref/status-build-failure.md From 52677184728a5ba21087a7536f9ee0c8eb3bc1b2 Mon Sep 17 00:00:00 2001 From: John Ericson Date: Wed, 28 May 2025 20:37:23 -0400 Subject: [PATCH 136/218] Fix #13293 We move the `assertPathValidity` to where we know what the wanted outputs are. --- src/libstore/build/derivation-building-goal.cc | 3 --- src/libstore/build/derivation-goal.cc | 7 +++++++ src/libstore/unix/build/derivation-builder.cc | 6 +++--- .../unix/include/nix/store/build/derivation-builder.hh | 8 -------- 4 files changed, 10 insertions(+), 14 deletions(-) diff --git a/src/libstore/build/derivation-building-goal.cc b/src/libstore/build/derivation-building-goal.cc index 19296fac315..9a91b3592ce 100644 --- a/src/libstore/build/derivation-building-goal.cc +++ b/src/libstore/build/derivation-building-goal.cc @@ -613,9 +613,6 @@ Goal::Co DerivationBuildingGoal::tryToBuild() void closeLogFile() override { goal.closeLogFile(); } - SingleDrvOutputs assertPathValidity() override { - return goal.assertPathValidity(); - } void appendLogTailErrorMsg(std::string & msg) override { goal.appendLogTailErrorMsg(msg); } diff --git a/src/libstore/build/derivation-goal.cc b/src/libstore/build/derivation-goal.cc index b1f081115ce..3fcc376ed95 100644 --- a/src/libstore/build/derivation-goal.cc +++ b/src/libstore/build/derivation-goal.cc @@ -210,6 +210,13 @@ Goal::Co DerivationGoal::haveDerivation(StorePath drvPath) .outputs = wantedOutputs, }); + if (buildMode == bmCheck) { + /* In checking mode, the builder will not register any outputs. + So we want to make sure the ones that we wanted to check are + properly there. */ + buildResult.builtOutputs = assertPathValidity(drvPath); + } + co_return amDone(g->exitCode, g->ex); }; diff --git a/src/libstore/unix/build/derivation-builder.cc b/src/libstore/unix/build/derivation-builder.cc index eca01748733..1f15cc9e95f 100644 --- a/src/libstore/unix/build/derivation-builder.cc +++ b/src/libstore/unix/build/derivation-builder.cc @@ -589,10 +589,10 @@ static void replaceValidPath(const Path & storePath, const Path & tmpPath) way first. We'd better not be interrupted here, because if we're repairing (say) Glibc, we end up with a broken system. */ Path oldPath; - + if (pathExists(storePath)) { // why do we loop here? - // although makeTempPath should be unique, we can't + // although makeTempPath should be unique, we can't // guarantee that. do { oldPath = makeTempPath(storePath, ".old"); @@ -1871,7 +1871,7 @@ SingleDrvOutputs DerivationBuilderImpl::registerOutputs() also a source for non-determinism. */ if (delayedException) std::rethrow_exception(delayedException); - return miscMethods->assertPathValidity(); + return {}; } /* Apply output checks. */ diff --git a/src/libstore/unix/include/nix/store/build/derivation-builder.hh b/src/libstore/unix/include/nix/store/build/derivation-builder.hh index 01266a49234..5ce38e034eb 100644 --- a/src/libstore/unix/include/nix/store/build/derivation-builder.hh +++ b/src/libstore/unix/include/nix/store/build/derivation-builder.hh @@ -97,14 +97,6 @@ struct DerivationBuilderCallbacks */ virtual void closeLogFile() = 0; - /** - * Aborts if any output is not valid or corrupt, and otherwise - * returns a 'SingleDrvOutputs' structure containing all outputs. - * - * @todo Probably should just be in `DerivationGoal`. - */ - virtual SingleDrvOutputs assertPathValidity() = 0; - virtual void appendLogTailErrorMsg(std::string & msg) = 0; /** From 93a42a597145a2ff0214ddc6a9828921056c3657 Mon Sep 17 00:00:00 2001 From: Sergei Zimmerman Date: Wed, 11 Jun 2025 22:08:03 +0000 Subject: [PATCH 137/218] flake: Add meson formatter This adds a meson.format file that mostly mirrors the projects meson style and a pre-commit hook to enforce this style. Some low-diff files are formatted. --- maintainers/flake-module.nix | 112 ++++++++++++++++++ meson.build | 10 +- meson.format | 7 ++ meson.options | 15 ++- .../windows-version/meson.build | 2 +- .../include/nix/expr/tests/meson.build | 8 +- src/libexpr/primops/meson.build | 2 +- .../linux/include/nix/store/meson.build | 4 +- src/libstore/linux/meson.build | 4 +- .../freebsd/include/nix/util/meson.build | 4 +- src/libutil/freebsd/meson.build | 4 +- src/libutil/include/nix/util/meson.build | 4 +- .../linux/include/nix/util/meson.build | 5 +- src/libutil/linux/meson.build | 5 +- .../windows/include/nix/util/meson.build | 6 +- src/perl/t/meson.build | 4 +- tests/functional/plugins/meson.build | 4 +- .../test-libstoreconsumer/meson.build | 4 +- 18 files changed, 152 insertions(+), 52 deletions(-) create mode 100644 meson.format diff --git a/maintainers/flake-module.nix b/maintainers/flake-module.nix index d2bae32b6f2..d443d1a741b 100644 --- a/maintainers/flake-module.nix +++ b/maintainers/flake-module.nix @@ -37,6 +37,118 @@ fi ''}"; }; + meson-format = { + enable = true; + files = "(meson.build|meson.options)$"; + entry = "${pkgs.writeScript "format-meson" '' + #!${pkgs.runtimeShell} + for file in "$@"; do + ${lib.getExe pkgs.meson} format -ic ${../meson.format} "$file" + done + ''}"; + excludes = [ + # We haven't applied formatting to these files yet + ''^doc/manual/meson.build$'' + ''^doc/manual/source/command-ref/meson.build$'' + ''^doc/manual/source/development/meson.build$'' + ''^doc/manual/source/language/meson.build$'' + ''^doc/manual/source/meson.build$'' + ''^doc/manual/source/release-notes/meson.build$'' + ''^doc/manual/source/store/meson.build$'' + ''^misc/bash/meson.build$'' + ''^misc/fish/meson.build$'' + ''^misc/launchd/meson.build$'' + ''^misc/meson.build$'' + ''^misc/systemd/meson.build$'' + ''^misc/zsh/meson.build$'' + ''^nix-meson-build-support/$'' + ''^nix-meson-build-support/big-objs/meson.build$'' + ''^nix-meson-build-support/common/meson.build$'' + ''^nix-meson-build-support/deps-lists/meson.build$'' + ''^nix-meson-build-support/export/meson.build$'' + ''^nix-meson-build-support/export-all-symbols/meson.build$'' + ''^nix-meson-build-support/generate-header/meson.build$'' + ''^nix-meson-build-support/libatomic/meson.build$'' + ''^nix-meson-build-support/subprojects/meson.build$'' + ''^scripts/meson.build$'' + ''^src/external-api-docs/meson.build$'' + ''^src/internal-api-docs/meson.build$'' + ''^src/libcmd/include/nix/cmd/meson.build$'' + ''^src/libcmd/meson.build$'' + ''^src/libcmd/nix-meson-build-support$'' + ''^src/libexpr/include/nix/expr/meson.build$'' + ''^src/libexpr/meson.build$'' + ''^src/libexpr/nix-meson-build-support$'' + ''^src/libexpr-c/meson.build$'' + ''^src/libexpr-c/nix-meson-build-support$'' + ''^src/libexpr-test-support/meson.build$'' + ''^src/libexpr-test-support/nix-meson-build-support$'' + ''^src/libexpr-tests/meson.build$'' + ''^src/libexpr-tests/nix-meson-build-support$'' + ''^src/libfetchers/include/nix/fetchers/meson.build$'' + ''^src/libfetchers/meson.build$'' + ''^src/libfetchers/nix-meson-build-support$'' + ''^src/libfetchers-c/meson.build$'' + ''^src/libfetchers-c/nix-meson-build-support$'' + ''^src/libfetchers-tests/meson.build$'' + ''^src/libfetchers-tests/nix-meson-build-support$'' + ''^src/libflake/include/nix/flake/meson.build$'' + ''^src/libflake/meson.build$'' + ''^src/libflake/nix-meson-build-support$'' + ''^src/libflake-c/meson.build$'' + ''^src/libflake-c/nix-meson-build-support$'' + ''^src/libflake-tests/meson.build$'' + ''^src/libflake-tests/nix-meson-build-support$'' + ''^src/libmain/include/nix/main/meson.build$'' + ''^src/libmain/meson.build$'' + ''^src/libmain/nix-meson-build-support$'' + ''^src/libmain-c/meson.build$'' + ''^src/libmain-c/nix-meson-build-support$'' + ''^src/libstore/include/nix/store/meson.build$'' + ''^src/libstore/meson.build$'' + ''^src/libstore/nix-meson-build-support$'' + ''^src/libstore/unix/include/nix/store/meson.build$'' + ''^src/libstore/unix/meson.build$'' + ''^src/libstore/windows/meson.build$'' + ''^src/libstore-c/meson.build$'' + ''^src/libstore-c/nix-meson-build-support$'' + ''^src/libstore-test-support/include/nix/store/tests/meson.build$'' + ''^src/libstore-test-support/meson.build$'' + ''^src/libstore-test-support/nix-meson-build-support$'' + ''^src/libstore-tests/meson.build$'' + ''^src/libstore-tests/nix-meson-build-support$'' + ''^src/libutil/meson.build$'' + ''^src/libutil/nix-meson-build-support$'' + ''^src/libutil/unix/include/nix/util/meson.build$'' + ''^src/libutil/unix/meson.build$'' + ''^src/libutil/windows/meson.build$'' + ''^src/libutil-c/meson.build$'' + ''^src/libutil-c/nix-meson-build-support$'' + ''^src/libutil-test-support/include/nix/util/tests/meson.build$'' + ''^src/libutil-test-support/meson.build$'' + ''^src/libutil-test-support/nix-meson-build-support$'' + ''^src/libutil-tests/meson.build$'' + ''^src/libutil-tests/nix-meson-build-support$'' + ''^src/nix/meson.build$'' + ''^src/nix/nix-meson-build-support$'' + ''^src/perl/lib/Nix/meson.build$'' + ''^src/perl/meson.build$'' + ''^tests/functional/ca/meson.build$'' + ''^tests/functional/common/meson.build$'' + ''^tests/functional/dyn-drv/meson.build$'' + ''^tests/functional/flakes/meson.build$'' + ''^tests/functional/git-hashing/meson.build$'' + ''^tests/functional/local-overlay-store/meson.build$'' + ''^tests/functional/meson.build$'' + ''^src/libcmd/meson.options$'' + ''^src/libexpr/meson.options$'' + ''^src/libstore/meson.options$'' + ''^src/libutil/meson.options$'' + ''^src/libutil-c/meson.options$'' + ''^src/nix/meson.options$'' + ''^src/perl/meson.options$'' + ]; + }; nixfmt-rfc-style = { enable = true; excludes = [ diff --git a/meson.build b/meson.build index 9f74711438a..0237919790f 100644 --- a/meson.build +++ b/meson.build @@ -1,13 +1,13 @@ # This is just a stub project to include all the others as subprojects # for development shell purposes -project('nix-dev-shell', 'cpp', +project( + 'nix-dev-shell', + 'cpp', version : files('.version'), subproject_dir : 'src', - default_options : [ - 'localstatedir=/nix/var', - ], - meson_version : '>= 1.1' + default_options : [ 'localstatedir=/nix/var' ], + meson_version : '>= 1.1', ) # Internal Libraries diff --git a/meson.format b/meson.format new file mode 100644 index 00000000000..4876dd962db --- /dev/null +++ b/meson.format @@ -0,0 +1,7 @@ +indent_by = ' ' +space_array = true +kwargs_force_multiline = false +wide_colon = true +group_arg_value = true +indent_before_comments = ' ' +use_editor_config = true diff --git a/meson.options b/meson.options index 329fe06bf15..30670902e62 100644 --- a/meson.options +++ b/meson.options @@ -1,13 +1,22 @@ # vim: filetype=meson -option('doc-gen', type : 'boolean', value : false, +option( + 'doc-gen', + type : 'boolean', + value : false, description : 'Generate documentation', ) -option('unit-tests', type : 'boolean', value : true, +option( + 'unit-tests', + type : 'boolean', + value : true, description : 'Build unit tests', ) -option('bindings', type : 'boolean', value : true, +option( + 'bindings', + type : 'boolean', + value : true, description : 'Build language bindings (e.g. Perl)', ) diff --git a/nix-meson-build-support/windows-version/meson.build b/nix-meson-build-support/windows-version/meson.build index 3a008e5df94..ed4caaa9a5c 100644 --- a/nix-meson-build-support/windows-version/meson.build +++ b/nix-meson-build-support/windows-version/meson.build @@ -2,5 +2,5 @@ if host_machine.system() == 'windows' # https://learn.microsoft.com/en-us/cpp/porting/modifying-winver-and-win32-winnt?view=msvc-170 # #define _WIN32_WINNT_WIN8 0x0602 # We currently don't use any API which requires higher than this. - add_project_arguments([ '-D_WIN32_WINNT=0x0602' ], language: 'cpp') + add_project_arguments([ '-D_WIN32_WINNT=0x0602' ], language : 'cpp') endif diff --git a/src/libexpr-test-support/include/nix/expr/tests/meson.build b/src/libexpr-test-support/include/nix/expr/tests/meson.build index 710bd8d4e3e..e44d2558234 100644 --- a/src/libexpr-test-support/include/nix/expr/tests/meson.build +++ b/src/libexpr-test-support/include/nix/expr/tests/meson.build @@ -1,9 +1,5 @@ # Public headers directory -include_dirs = [include_directories('../../..')] +include_dirs = [ include_directories('../../..') ] -headers = files( - 'libexpr.hh', - 'nix_api_expr.hh', - 'value/context.hh', -) +headers = files('libexpr.hh', 'nix_api_expr.hh', 'value/context.hh') diff --git a/src/libexpr/primops/meson.build b/src/libexpr/primops/meson.build index f910fe23768..b8abc6409af 100644 --- a/src/libexpr/primops/meson.build +++ b/src/libexpr/primops/meson.build @@ -1,6 +1,6 @@ generated_headers += gen_header.process( 'derivation.nix', - preserve_path_from: meson.project_source_root(), + preserve_path_from : meson.project_source_root(), ) sources += files( diff --git a/src/libstore/linux/include/nix/store/meson.build b/src/libstore/linux/include/nix/store/meson.build index a664aefa9f4..eb2ae25837c 100644 --- a/src/libstore/linux/include/nix/store/meson.build +++ b/src/libstore/linux/include/nix/store/meson.build @@ -1,5 +1,3 @@ include_dirs += include_directories('../..') -headers += files( - 'personality.hh', -) +headers += files('personality.hh') diff --git a/src/libstore/linux/meson.build b/src/libstore/linux/meson.build index 6fc193cf898..93084a8ae2b 100644 --- a/src/libstore/linux/meson.build +++ b/src/libstore/linux/meson.build @@ -1,5 +1,3 @@ -sources += files( - 'personality.cc', -) +sources += files('personality.cc') subdir('include/nix/store') diff --git a/src/libutil/freebsd/include/nix/util/meson.build b/src/libutil/freebsd/include/nix/util/meson.build index 561c8796c15..283c49b9510 100644 --- a/src/libutil/freebsd/include/nix/util/meson.build +++ b/src/libutil/freebsd/include/nix/util/meson.build @@ -2,6 +2,4 @@ include_dirs += include_directories('../..') -headers += files( - 'freebsd-jail.hh', -) +headers += files('freebsd-jail.hh') diff --git a/src/libutil/freebsd/meson.build b/src/libutil/freebsd/meson.build index 8ffdc283292..34508efd0ee 100644 --- a/src/libutil/freebsd/meson.build +++ b/src/libutil/freebsd/meson.build @@ -1,5 +1,3 @@ -sources += files( - 'freebsd-jail.cc', -) +sources += files('freebsd-jail.cc') subdir('include/nix/util') diff --git a/src/libutil/include/nix/util/meson.build b/src/libutil/include/nix/util/meson.build index e30b8dacd48..6c95a6cedaa 100644 --- a/src/libutil/include/nix/util/meson.build +++ b/src/libutil/include/nix/util/meson.build @@ -1,6 +1,6 @@ # Public headers directory -include_dirs = [include_directories('../..')] +include_dirs = [ include_directories('../..') ] headers = files( 'abstract-setting-to-json.hh', @@ -63,8 +63,8 @@ headers = files( 'source-path.hh', 'split.hh', 'std-hash.hh', - 'strings.hh', 'strings-inline.hh', + 'strings.hh', 'suggestions.hh', 'sync.hh', 'tarfile.hh', diff --git a/src/libutil/linux/include/nix/util/meson.build b/src/libutil/linux/include/nix/util/meson.build index e28ad8e0595..ac5f318f869 100644 --- a/src/libutil/linux/include/nix/util/meson.build +++ b/src/libutil/linux/include/nix/util/meson.build @@ -2,7 +2,4 @@ include_dirs += include_directories('../..') -headers += files( - 'cgroup.hh', - 'linux-namespaces.hh', -) +headers += files('cgroup.hh', 'linux-namespaces.hh') diff --git a/src/libutil/linux/meson.build b/src/libutil/linux/meson.build index b8053a5bb03..f3fe07c9cb1 100644 --- a/src/libutil/linux/meson.build +++ b/src/libutil/linux/meson.build @@ -1,6 +1,3 @@ -sources += files( - 'cgroup.cc', - 'linux-namespaces.cc', -) +sources += files('cgroup.cc', 'linux-namespaces.cc') subdir('include/nix/util') diff --git a/src/libutil/windows/include/nix/util/meson.build b/src/libutil/windows/include/nix/util/meson.build index 1bd56c4bd17..f0d4c37e956 100644 --- a/src/libutil/windows/include/nix/util/meson.build +++ b/src/libutil/windows/include/nix/util/meson.build @@ -2,8 +2,4 @@ include_dirs += include_directories('../..') -headers += files( - 'signals-impl.hh', - 'windows-async-pipe.hh', - 'windows-error.hh', -) +headers += files('signals-impl.hh', 'windows-async-pipe.hh', 'windows-error.hh') diff --git a/src/perl/t/meson.build b/src/perl/t/meson.build index dbd1139f327..cd98453c357 100644 --- a/src/perl/t/meson.build +++ b/src/perl/t/meson.build @@ -5,9 +5,7 @@ # src #--------------------------------------------------- -nix_perl_tests = files( - 'init.t', -) +nix_perl_tests = files('init.t') foreach f : nix_perl_tests diff --git a/tests/functional/plugins/meson.build b/tests/functional/plugins/meson.build index ae66e3036ac..7cbc935c9c5 100644 --- a/tests/functional/plugins/meson.build +++ b/tests/functional/plugins/meson.build @@ -1,8 +1,6 @@ libplugintest = shared_module( 'plugintest', 'plugintest.cc', - dependencies : [ - dependency('nix-expr'), - ], + dependencies : [ dependency('nix-expr') ], build_by_default : false, ) diff --git a/tests/functional/test-libstoreconsumer/meson.build b/tests/functional/test-libstoreconsumer/meson.build index e5a1cc18221..d27647ec460 100644 --- a/tests/functional/test-libstoreconsumer/meson.build +++ b/tests/functional/test-libstoreconsumer/meson.build @@ -1,8 +1,6 @@ libstoreconsumer_tester = executable( 'test-libstoreconsumer', 'main.cc', - dependencies : [ - dependency('nix-store'), - ], + dependencies : [ dependency('nix-store') ], build_by_default : false, ) From 57c72dee9b27b50b2a3b7a738cf33329acfb7236 Mon Sep 17 00:00:00 2001 From: Pol Dellaiera Date: Thu, 12 Jun 2025 09:41:37 +0200 Subject: [PATCH 138/218] docker: make sure `nix config check` works --- docker.nix | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/docker.nix b/docker.nix index e3a0635d017..99316fc84aa 100644 --- a/docker.nix +++ b/docker.nix @@ -280,7 +280,6 @@ let ln -s ${profile} $out/nix/var/nix/profiles/default-1-link ln -s /nix/var/nix/profiles/default-1-link $out/nix/var/nix/profiles/default - ln -s /nix/var/nix/profiles/default $out${userHome}/.nix-profile ln -s ${channel} $out/nix/var/nix/profiles/per-user/${uname}/channels-1-link ln -s /nix/var/nix/profiles/per-user/${uname}/channels-1-link $out/nix/var/nix/profiles/per-user/${uname}/channels @@ -332,7 +331,7 @@ pkgs.dockerTools.buildLayeredImageWithNixDb { ''; config = { - Cmd = [ "${userHome}/.nix-profile/bin/bash" ]; + Cmd = [ (lib.getExe pkgs.bashInteractive) ]; User = "${toString uid}:${toString gid}"; Env = [ "USER=${uname}" From ab10fddc6ee9c0ef205e9454d6992cdb7e7df801 Mon Sep 17 00:00:00 2001 From: Luc Perkins Date: Thu, 12 Jun 2025 08:24:39 -0700 Subject: [PATCH 139/218] Rework future tense in user-facing messages --- src/libcmd/installables.cc | 2 +- src/libexpr/include/nix/expr/eval-settings.hh | 20 +-- src/libexpr/primops.cc | 54 ++++---- src/libexpr/primops/context.cc | 2 +- src/libexpr/primops/fetchClosure.cc | 2 +- src/libexpr/primops/fetchTree.cc | 16 +-- .../include/nix/fetchers/fetch-settings.hh | 6 +- src/libflake/flake.cc | 4 +- src/libmain/plugin.cc | 8 +- .../include/nix/store/filetransfer.hh | 2 +- src/libstore/include/nix/store/globals.hh | 123 +++++++++--------- .../include/nix/store/local-fs-store.hh | 4 +- src/libstore/include/nix/store/local-store.hh | 2 +- .../nix/store/s3-binary-cache-store.hh | 6 +- src/libutil/experimental-features.cc | 2 +- src/libutil/include/nix/util/logging.hh | 2 +- src/nix-build/nix-build.cc | 2 +- src/nix/unix/daemon.cc | 2 +- tests/functional/fetchGit.sh | 2 +- tests/functional/flakes/unlocked-override.sh | 2 +- tests/nixos/github-flakes.nix | 22 ++-- 21 files changed, 141 insertions(+), 144 deletions(-) diff --git a/src/libcmd/installables.cc b/src/libcmd/installables.cc index 836aded96f8..49ffd82e1a3 100644 --- a/src/libcmd/installables.cc +++ b/src/libcmd/installables.cc @@ -199,7 +199,7 @@ SourceExprCommand::SourceExprCommand() .shortName = 'f', .description = "Interpret [*installables*](@docroot@/command-ref/new-cli/nix.md#installables) as attribute paths relative to the Nix expression stored in *file*. " - "If *file* is the character -, then a Nix expression will be read from standard input. " + "If *file* is the character -, then a Nix expression is read from standard input. " "Implies `--impure`.", .category = installablesCategory, .labels = {"file"}, diff --git a/src/libexpr/include/nix/expr/eval-settings.hh b/src/libexpr/include/nix/expr/eval-settings.hh index 9ff73138a81..f88f1387477 100644 --- a/src/libexpr/include/nix/expr/eval-settings.hh +++ b/src/libexpr/include/nix/expr/eval-settings.hh @@ -132,7 +132,7 @@ struct EvalSettings : Config Setting restrictEval{ this, false, "restrict-eval", R"( - If set to `true`, the Nix evaluator will not allow access to any + If set to `true`, the Nix evaluator doesn't allow access to any files outside of [`builtins.nixPath`](@docroot@/language/builtins.md#builtins-nixPath), or to URIs outside of @@ -157,7 +157,7 @@ struct EvalSettings : Config R"( By default, Nix allows [Import from Derivation](@docroot@/language/import-from-derivation.md). - When this setting is `true`, Nix will log a warning indicating that it performed such an import. + When this setting is `true`, Nix logs a warning indicating that it performed such an import. This option has no effect if `allow-import-from-derivation` is disabled. )" }; @@ -167,9 +167,9 @@ struct EvalSettings : Config R"( By default, Nix allows [Import from Derivation](@docroot@/language/import-from-derivation.md). - With this option set to `false`, Nix will throw an error when evaluating an expression that uses this feature, + With this option set to `false`, Nix throws an error when evaluating an expression that uses this feature, even when the required store object is readily available. - This ensures that evaluation will not require any builds to take place, + This ensures that evaluation doesn't require any builds to take place, regardless of the state of the store. )"}; @@ -188,8 +188,8 @@ struct EvalSettings : Config Setting traceFunctionCalls{this, false, "trace-function-calls", R"( - If set to `true`, the Nix evaluator will trace every function call. - Nix will print a log message at the "vomit" level for every function + If set to `true`, the Nix evaluator traces every function call. + Nix prints a log message at the "vomit" level for every function entrance and function exit. function-trace entered undefined position at 1565795816999559622 @@ -234,8 +234,8 @@ struct EvalSettings : Config Setting ignoreExceptionsDuringTry{this, false, "ignore-try", R"( - If set to true, ignore exceptions inside 'tryEval' calls when evaluating nix expressions in - debug mode (using the --debugger flag). By default the debugger will pause on all exceptions. + If set to true, ignore exceptions inside 'tryEval' calls when evaluating Nix expressions in + debug mode (using the --debugger flag). By default the debugger pauses on all exceptions. )"}; Setting traceVerbose{this, false, "trace-verbose", @@ -247,7 +247,7 @@ struct EvalSettings : Config Setting builtinsTraceDebugger{this, false, "debugger-on-trace", R"( If set to true and the `--debugger` flag is given, the following functions - will enter the debugger like [`builtins.break`](@docroot@/language/builtins.md#builtins-break). + enter the debugger like [`builtins.break`](@docroot@/language/builtins.md#builtins-break): * [`builtins.trace`](@docroot@/language/builtins.md#builtins-trace) * [`builtins.traceVerbose`](@docroot@/language/builtins.md#builtins-traceVerbose) @@ -269,7 +269,7 @@ struct EvalSettings : Config Setting builtinsAbortOnWarn{this, false, "abort-on-warn", R"( - If set to true, [`builtins.warn`](@docroot@/language/builtins.md#builtins-warn) will throw an error when logging a warning. + If set to true, [`builtins.warn`](@docroot@/language/builtins.md#builtins-warn) throws an error when logging a warning. This will give you a stack trace that leads to the location of the warning. diff --git a/src/libexpr/primops.cc b/src/libexpr/primops.cc index 48bc272f936..4c649e6deed 100644 --- a/src/libexpr/primops.cc +++ b/src/libexpr/primops.cc @@ -345,7 +345,7 @@ static RegisterPrimOp primop_import({ > } > ``` > - > then the following `foo.nix` will give an error: + > then the following `foo.nix` throws an error: > > ```nix > # foo.nix @@ -915,7 +915,7 @@ static RegisterPrimOp primop_ceil({ a NixInt and if `*number* < -9007199254740992` or `*number* > 9007199254740992`. If the datatype of *number* is neither a NixInt (signed 64-bit integer) nor a NixFloat - (IEEE-754 double-precision floating-point number), an evaluation error will be thrown. + (IEEE-754 double-precision floating-point number), an evaluation error is thrown. )", .fun = prim_ceil, }); @@ -1002,15 +1002,15 @@ static RegisterPrimOp primop_tryEval({ Try to shallowly evaluate *e*. Return a set containing the attributes `success` (`true` if *e* evaluated successfully, `false` if an error was thrown) and `value`, equalling *e* if - successful and `false` otherwise. `tryEval` will only prevent + successful and `false` otherwise. `tryEval` only prevents errors created by `throw` or `assert` from being thrown. - Errors `tryEval` will not catch are for example those created + Errors `tryEval` doesn't catch are, for example, those created by `abort` and type errors generated by builtins. Also note that this doesn't evaluate *e* deeply, so `let e = { x = throw ""; }; - in (builtins.tryEval e).success` will be `true`. Using + in (builtins.tryEval e).success` is `true`. Using `builtins.deepSeq` one can get the expected result: `let e = { x = throw ""; }; in - (builtins.tryEval (builtins.deepSeq e e)).success` will be + (builtins.tryEval (builtins.deepSeq e e)).success` is `false`. `tryEval` intentionally does not return the error message, because that risks bringing non-determinism into the evaluation result, and it would become very difficult to improve error reporting without breaking existing expressions. @@ -1108,7 +1108,7 @@ static RegisterPrimOp primop_trace({ If the [`debugger-on-trace`](@docroot@/command-ref/conf-file.md#conf-debugger-on-trace) option is set to `true` and the `--debugger` flag is given, the - interactive debugger will be started when `trace` is called (like + interactive debugger is started when `trace` is called (like [`break`](@docroot@/language/builtins.md#builtins-break)). )", .fun = prim_trace, @@ -1157,7 +1157,7 @@ static RegisterPrimOp primop_warn({ If the [`abort-on-warn`](@docroot@/command-ref/conf-file.md#conf-abort-on-warn) - option is set, the evaluation will be aborted after the warning is printed. + option is set, the evaluation is aborted after the warning is printed. This is useful to reveal the stack trace of the warning, when the context is non-interactive and a debugger can not be launched. )", .fun = prim_warn, @@ -1634,7 +1634,7 @@ static RegisterPrimOp primop_placeholder({ .name = "placeholder", .args = {"output"}, .doc = R"( - Return at + Return an [output placeholder string](@docroot@/store/derivation/index.md#output-placeholder) for the specified *output* that will be substituted by the corresponding [output path](@docroot@/glossary.md#gloss-output-path) @@ -1799,7 +1799,7 @@ static RegisterPrimOp primop_baseNameOf({ After this, the *base name* is returned as previously described, assuming `/` as the directory separator. (Note that evaluation must be platform independent.) - This is somewhat similar to the [GNU `basename`](https://www.gnu.org/software/coreutils/manual/html_node/basename-invocation.html) command, but GNU `basename` will strip any number of trailing slashes. + This is somewhat similar to the [GNU `basename`](https://www.gnu.org/software/coreutils/manual/html_node/basename-invocation.html) command, but GNU `basename` strips any number of trailing slashes. )", .fun = prim_baseNameOf, }); @@ -1998,9 +1998,9 @@ static RegisterPrimOp primop_findFile(PrimOp { > ] > ``` > - > and a *lookup-path* value `"nixos-config"` will cause Nix to try `/home/eelco/Dev/nixos-config` and `/etc/nixos` in that order and return the first path that exists. + > and a *lookup-path* value `"nixos-config"` causes Nix to try `/home/eelco/Dev/nixos-config` and `/etc/nixos` in that order and return the first path that exists. - If `path` starts with `http://` or `https://`, it is interpreted as the URL of a tarball that will be downloaded and unpacked to a temporary location. + If `path` starts with `http://` or `https://`, it is interpreted as the URL of a tarball to be downloaded and unpacked to a temporary location. The tarball must consist of a single top-level directory. The URLs of the tarballs from the official `nixos.org` channels can be abbreviated as `channel:`. @@ -2147,7 +2147,7 @@ static RegisterPrimOp primop_readDir({ Return the contents of the directory *path* as a set mapping directory entries to the corresponding file type. For instance, if directory `A` contains a regular file `B` and another directory - `C`, then `builtins.readDir ./A` will return the set + `C`, then `builtins.readDir ./A` returns the set ```nix { B = "regular"; C = "directory"; } @@ -2182,8 +2182,8 @@ static RegisterPrimOp primop_outputOf({ [input placeholder string](@docroot@/store/derivation/index.md#input-placeholder) if needed. - If the derivation has a statically-known output path (i.e. the derivation output is input-addressed, or fixed content-addressed), the output path will just be returned. - But if the derivation is content-addressed or if the derivation is itself not-statically produced (i.e. is the output of another derivation), an input placeholder will be returned instead. + If the derivation has a statically-known output path (i.e. the derivation output is input-addressed, or fixed content-addressed), the output path is returned. + But if the derivation is content-addressed or if the derivation is itself not-statically produced (i.e. is the output of another derivation), an input placeholder is returned instead. *`derivation reference`* must be a string that may contain a regular store path to a derivation, or may be an input placeholder reference. If the derivation is produced by a derivation, you must explicitly select `drv.outPath`. @@ -2196,7 +2196,7 @@ static RegisterPrimOp primop_outputOf({ "out" ``` - will return a input placeholder for the output of the output of `myDrv`. + returns an input placeholder for the output of the output of `myDrv`. This primop corresponds to the `^` sigil for [deriving paths](@docroot@/glossary.md#gloss-deriving-paths), e.g. as part of installable syntax on the command line. )", @@ -2582,12 +2582,12 @@ static RegisterPrimOp primop_filterSource({ > > `filterSource` should not be used to filter store paths. Since > `filterSource` uses the name of the input directory while naming - > the output directory, doing so will produce a directory name in + > the output directory, doing so produces a directory name in > the form of `--`, where `-` is > the name of the input directory. Since `` depends on the - > unfiltered directory, the name of the output directory will - > indirectly depend on files that are filtered out by the - > function. This will trigger a rebuild even when a filtered out + > unfiltered directory, the name of the output directory + > indirectly depends on files that are filtered out by the + > function. This triggers a rebuild even when a filtered out > file is changed. Use `builtins.path` instead, which allows > specifying the name of the output directory. @@ -2602,8 +2602,8 @@ static RegisterPrimOp primop_filterSource({ } ``` - However, if `source-dir` is a Subversion working copy, then all - those annoying `.svn` subdirectories will also be copied to the + However, if `source-dir` is a Subversion working copy, then all of + those annoying `.svn` subdirectories are also copied to the store. Worse, the contents of those directories may change a lot, causing lots of spurious rebuilds. With `filterSource` you can filter out the `.svn` directories: @@ -2623,8 +2623,8 @@ static RegisterPrimOp primop_filterSource({ `"regular"`, `"directory"`, `"symlink"` or `"unknown"` (for other kinds of files such as device nodes or fifos — but note that those cannot be copied to the Nix store, so if the predicate returns - `true` for them, the copy will fail). If you exclude a directory, - the entire corresponding subtree of *e2* will be excluded. + `true` for them, the copy fails). If you exclude a directory, + the entire corresponding subtree of *e2* is excluded. )", .fun = prim_filterSource, }); @@ -2698,7 +2698,7 @@ static RegisterPrimOp primop_path({ - sha256\ When provided, this is the expected hash of the file at the - path. Evaluation will fail if the hash is incorrect, and + path. Evaluation fails if the hash is incorrect, and providing a hash allows `builtins.path` to be used even when the `pure-eval` nix config option is on. )", @@ -4806,7 +4806,7 @@ void EvalState::createBaseEnv(const EvalSettings & evalSettings) .type = nInt, .doc = R"( Return the [Unix time](https://en.wikipedia.org/wiki/Unix_time) at first evaluation. - Repeated references to that name will re-use the initially obtained value. + Repeated references to that name re-use the initially obtained value. Example: @@ -4821,7 +4821,7 @@ void EvalState::createBaseEnv(const EvalSettings & evalSettings) 1683705525 ``` - The [store path](@docroot@/store/store-path.md) of a derivation depending on `currentTime` will differ for each evaluation, unless both evaluate `builtins.currentTime` in the same second. + The [store path](@docroot@/store/store-path.md) of a derivation depending on `currentTime` differs for each evaluation, unless both evaluate `builtins.currentTime` in the same second. )", .impureOnly = true, }); diff --git a/src/libexpr/primops/context.cc b/src/libexpr/primops/context.cc index 6a7284e051f..496678884af 100644 --- a/src/libexpr/primops/context.cc +++ b/src/libexpr/primops/context.cc @@ -240,7 +240,7 @@ static RegisterPrimOp primop_getContext({ The string context tracks references to derivations within a string. It is represented as an attribute set of [store derivation](@docroot@/glossary.md#gloss-store-derivation) paths mapping to output names. - Using [string interpolation](@docroot@/language/string-interpolation.md) on a derivation will add that derivation to the string context. + Using [string interpolation](@docroot@/language/string-interpolation.md) on a derivation adds that derivation to the string context. For example, ```nix diff --git a/src/libexpr/primops/fetchClosure.cc b/src/libexpr/primops/fetchClosure.cc index 4dd8b2606ca..ea6145f6f9e 100644 --- a/src/libexpr/primops/fetchClosure.cc +++ b/src/libexpr/primops/fetchClosure.cc @@ -214,7 +214,7 @@ static RegisterPrimOp primop_fetchClosure({ .doc = R"( Fetch a store path [closure](@docroot@/glossary.md#gloss-closure) from a binary cache, and return the store path as a string with context. - This function can be invoked in three ways, that we will discuss in order of preference. + This function can be invoked in three ways that we will discuss in order of preference. **Fetch a content-addressed store path** diff --git a/src/libexpr/primops/fetchTree.cc b/src/libexpr/primops/fetchTree.cc index bdd65b4c667..5b6dd65317b 100644 --- a/src/libexpr/primops/fetchTree.cc +++ b/src/libexpr/primops/fetchTree.cc @@ -191,7 +191,7 @@ static void fetchTree( input.to_string()); else state.error( - "in pure evaluation mode, '%s' will not fetch unlocked input '%s'", + "in pure evaluation mode, '%s' doesn't fetch unlocked input '%s'", fetcher, input.to_string()).atPos(pos).debugThrow(); } @@ -243,7 +243,7 @@ static RegisterPrimOp primop_fetchTree({ That is, `fetchTree` is idempotent. Downloads are cached in `$XDG_CACHE_HOME/nix`. - The remote source will be fetched from the network if both are true: + The remote source is fetched from the network if both are true: - A NAR hash is supplied and the corresponding store path is not [valid](@docroot@/glossary.md#gloss-validity), that is, not available in the store > **Note** @@ -338,7 +338,7 @@ static RegisterPrimOp primop_fetchTree({ > **Note** > - > If the URL points to a local directory, and no `ref` or `rev` is given, Nix will only consider files added to the Git index, as listed by `git ls-files` but use the *current file contents* of the Git working directory. + > If the URL points to a local directory, and no `ref` or `rev` is given, Nix only considers files added to the Git index, as listed by `git ls-files` but use the *current file contents* of the Git working directory. - `ref` (String, optional) @@ -681,7 +681,7 @@ static RegisterPrimOp primop_fetchGit({ This option has no effect once `shallow` cloning is enabled. By default, the `ref` value is prefixed with `refs/heads/`. - As of 2.3.0, Nix will not prefix `refs/heads/` if `ref` starts with `refs/`. + As of 2.3.0, Nix doesn't prefix `refs/heads/` if `ref` starts with `refs/`. - `submodules` (default: `false`) @@ -840,7 +840,7 @@ static RegisterPrimOp primop_fetchGit({ } ``` - Nix will refetch the branch according to the [`tarball-ttl`](@docroot@/command-ref/conf-file.md#conf-tarball-ttl) setting. + Nix refetches the branch according to the [`tarball-ttl`](@docroot@/command-ref/conf-file.md#conf-tarball-ttl) setting. This behavior is disabled in [pure evaluation mode](@docroot@/command-ref/conf-file.md#conf-pure-eval). @@ -851,9 +851,9 @@ static RegisterPrimOp primop_fetchGit({ ``` If the URL points to a local directory, and no `ref` or `rev` is - given, `fetchGit` will use the current content of the checked-out - files, even if they are not committed or added to Git's index. It will - only consider files added to the Git repository, as listed by `git ls-files`. + given, `fetchGit` uses the current content of the checked-out + files, even if they are not committed or added to Git's index. It + only considers files added to the Git repository, as listed by `git ls-files`. )", .fun = prim_fetchGit, }); diff --git a/src/libfetchers/include/nix/fetchers/fetch-settings.hh b/src/libfetchers/include/nix/fetchers/fetch-settings.hh index 7bd04db53b0..9cfd25e0b83 100644 --- a/src/libfetchers/include/nix/fetchers/fetch-settings.hh +++ b/src/libfetchers/include/nix/fetchers/fetch-settings.hh @@ -28,7 +28,7 @@ struct Settings : public Config space-separated `host=token` values. The specific token used is selected by matching the `host` portion against the "host" specification of the input. The `host` portion may - contain a path element which will match against the prefix + contain a path element which matches against the prefix URL for the input. (eg: `github.com/org=token`). The actual use of the `token` value is determined by the type of resource being accessed: @@ -95,11 +95,11 @@ struct Settings : public Config Setting trustTarballsFromGitForges{ this, true, "trust-tarballs-from-git-forges", R"( - If enabled (the default), Nix will consider tarballs from + If enabled (the default), Nix considers tarballs from GitHub and similar Git forges to be locked if a Git revision is specified, e.g. `github:NixOS/patchelf/7c2f768bf9601268a4e71c2ebe91e2011918a70f`. - This requires Nix to trust that the provider will return the + This requires Nix to trust that the provider returns the correct contents for the specified Git revision. If disabled, such tarballs are only considered locked if a diff --git a/src/libflake/flake.cc b/src/libflake/flake.cc index 24252d710d7..c178d076046 100644 --- a/src/libflake/flake.cc +++ b/src/libflake/flake.cc @@ -794,10 +794,10 @@ LockedFlake lockFlake( if (auto unlockedInput = newLockFile.isUnlocked(state.fetchSettings)) { if (lockFlags.failOnUnlocked) throw Error( - "Will not write lock file of flake '%s' because it has an unlocked input ('%s'). " + "Not writing lock file of flake '%s' because it has an unlocked input ('%s'). " "Use '--allow-dirty-locks' to allow this anyway.", topRef, *unlockedInput); if (state.fetchSettings.warnDirty) - warn("will not write lock file of flake '%s' because it has an unlocked input ('%s')", topRef, *unlockedInput); + warn("not writing lock file of flake '%s' because it has an unlocked input ('%s')", topRef, *unlockedInput); } else { if (!lockFlags.updateLockFile) throw Error("flake '%s' requires lock file changes but they're not allowed due to '--no-update-lock-file'", topRef); diff --git a/src/libmain/plugin.cc b/src/libmain/plugin.cc index db686a251ba..760a096ad21 100644 --- a/src/libmain/plugin.cc +++ b/src/libmain/plugin.cc @@ -43,9 +43,9 @@ struct PluginSettings : Config {}, "plugin-files", R"( - A list of plugin files to be loaded by Nix. Each of these files will - be dlopened by Nix. If they contain the symbol `nix_plugin_entry()`, - this symbol will be called. Alternatively, they can affect execution + A list of plugin files to be loaded by Nix. Each of these files is + dlopened by Nix. If they contain the symbol `nix_plugin_entry()`, + this symbol is called. Alternatively, they can affect execution through static initialization. In particular, these plugins may construct static instances of RegisterPrimOp to add new primops or constants to the expression language, RegisterStoreImplementation to add new store @@ -60,7 +60,7 @@ struct PluginSettings : Config itself, they must be DSOs compatible with the instance of Nix running at the time (i.e. compiled against the same headers, not linked to any incompatible libraries). They should not be linked to - any Nix libs directly, as those will be available already at load + any Nix libraries directly, as those are already available at load time. If an entry in the list is a directory, all files in the directory diff --git a/src/libstore/include/nix/store/filetransfer.hh b/src/libstore/include/nix/store/filetransfer.hh index 10c3ec7ef9b..745aeb29ee3 100644 --- a/src/libstore/include/nix/store/filetransfer.hh +++ b/src/libstore/include/nix/store/filetransfer.hh @@ -46,7 +46,7 @@ struct FileTransferSettings : Config )"}; Setting tries{this, 5, "download-attempts", - "How often Nix will attempt to download a file before giving up."}; + "The number of times Nix attempts to download a file before giving up."}; Setting downloadBufferSize{this, 64 * 1024 * 1024, "download-buffer-size", R"( diff --git a/src/libstore/include/nix/store/globals.hh b/src/libstore/include/nix/store/globals.hh index 00d92923671..b5157b4f46e 100644 --- a/src/libstore/include/nix/store/globals.hh +++ b/src/libstore/include/nix/store/globals.hh @@ -109,7 +109,7 @@ public: Setting tryFallback{ this, false, "fallback", R"( - If set to `true`, Nix will fall back to building from source if a + If set to `true`, Nix falls back to building from source if a binary substitute fails. This is equivalent to the `--fallback` flag. The default is `false`. )", @@ -127,11 +127,11 @@ public: MaxBuildJobsSetting maxBuildJobs{ this, 1, "max-jobs", R"( - Maximum number of jobs that Nix will try to build locally in parallel. + Maximum number of jobs that Nix tries to build locally in parallel. The special value `auto` causes Nix to use the number of CPUs in your system. Use `0` to disable local builds and directly use the remote machines specified in [`builders`](#conf-builders). - This will not affect derivations that have [`preferLocalBuild = true`](@docroot@/language/advanced-attributes.md#adv-attr-preferLocalBuild), which are always built locally. + This doesn't affect derivations that have [`preferLocalBuild = true`](@docroot@/language/advanced-attributes.md#adv-attr-preferLocalBuild), which are always built locally. > **Note** > @@ -146,8 +146,8 @@ public: this, 16, "max-substitution-jobs", R"( This option defines the maximum number of substitution jobs that Nix - will try to run in parallel. The default is `16`. The minimum value - one can choose is `1` and lower values will be interpreted as `1`. + tries to run in parallel. The default is `16`. The minimum value + one can choose is `1` and lower values are interpreted as `1`. )", {"substitution-max-jobs"}}; @@ -164,7 +164,7 @@ public: A very generic example using `derivation` and `xargs` may be more appropriate to explain the mechanism. Using `mkDerivation` as an example requires being aware of that there are multiple independent layers that are completely opaque here. --> - For instance, in Nixpkgs, if the attribute `enableParallelBuilding` for the `mkDerivation` build helper is set to `true`, it will pass the `-j${NIX_BUILD_CORES}` flag to GNU Make. + For instance, in Nixpkgs, if the attribute `enableParallelBuilding` for the `mkDerivation` build helper is set to `true`, it passes the `-j${NIX_BUILD_CORES}` flag to GNU Make. The value `0` means that the `builder` should use all available CPU cores in the system. @@ -186,7 +186,7 @@ public: this, NIX_LOCAL_SYSTEM, "system", R"( The system type of the current Nix installation. - Nix will only build a given [store derivation](@docroot@/glossary.md#gloss-store-derivation) locally when its `system` attribute equals any of the values specified here or in [`extra-platforms`](#conf-extra-platforms). + Nix only builds a given [store derivation](@docroot@/glossary.md#gloss-store-derivation) locally when its `system` attribute equals any of the values specified here or in [`extra-platforms`](#conf-extra-platforms). The default value is set when Nix itself is compiled for the system it will run on. The following system types are widely used, as Nix is actively supported on these platforms: @@ -292,28 +292,28 @@ public: > `i686-linux,x86_64-linux` 3. The SSH identity file to be used to log in to the remote machine. - If omitted, SSH will use its regular identities. + If omitted, SSH uses its regular identities. > **Example** > > `/home/user/.ssh/id_mac` - 4. The maximum number of builds that Nix will execute in parallel on the machine. + 4. The maximum number of builds that Nix executes in parallel on the machine. Typically this should be equal to the number of CPU cores. 5. The “speed factor”, indicating the relative speed of the machine as a positive integer. - If there are multiple machines of the right type, Nix will prefer the fastest, taking load into account. + If there are multiple machines of the right type, Nix prefers the fastest, taking load into account. 6. A comma-separated list of supported [system features](#conf-system-features). - A machine will only be used to build a derivation if all the features in the derivation's [`requiredSystemFeatures`](@docroot@/language/advanced-attributes.html#adv-attr-requiredSystemFeatures) attribute are supported by that machine. + A machine is only used to build a derivation if all the features in the derivation's [`requiredSystemFeatures`](@docroot@/language/advanced-attributes.html#adv-attr-requiredSystemFeatures) attribute are supported by that machine. 7. A comma-separated list of required [system features](#conf-system-features). - A machine will only be used to build a derivation if all of the machine’s required features appear in the derivation’s [`requiredSystemFeatures`](@docroot@/language/advanced-attributes.html#adv-attr-requiredSystemFeatures) attribute. + A machine is only used to build a derivation if all of the machine’s required features appear in the derivation’s [`requiredSystemFeatures`](@docroot@/language/advanced-attributes.html#adv-attr-requiredSystemFeatures) attribute. 8. The (base64-encoded) public host key of the remote machine. - If omitted, SSH will use its regular `known_hosts` file. + If omitted, SSH uses its regular `known_hosts` file. The value for this field can be obtained via `base64 -w0`. @@ -335,7 +335,7 @@ public: > nix@poochie.labs.cs.uu.nl i686-linux /home/nix/.ssh/id_scratchy 1 2 kvm benchmark > ``` > - > However, `poochie` will only build derivations that have the attribute + > However, `poochie` only builds derivations that have the attribute > > ```nix > requiredSystemFeatures = [ "benchmark" ]; @@ -348,7 +348,7 @@ public: > ``` > > `itchy` cannot do builds that require `kvm`, but `scratchy` does support such builds. - > For regular builds, `itchy` will be preferred over `scratchy` because it has a higher speed factor. + > For regular builds, `itchy` is preferred over `scratchy` because it has a higher speed factor. For Nix to use substituters, the calling user must be in the [`trusted-users`](#conf-trusted-users) list. @@ -372,15 +372,15 @@ public: Setting alwaysAllowSubstitutes{ this, false, "always-allow-substitutes", R"( - If set to `true`, Nix will ignore the [`allowSubstitutes`](@docroot@/language/advanced-attributes.md) attribute in derivations and always attempt to use [available substituters](#conf-substituters). + If set to `true`, Nix ignores the [`allowSubstitutes`](@docroot@/language/advanced-attributes.md) attribute in derivations and always attempt to use [available substituters](#conf-substituters). )"}; Setting buildersUseSubstitutes{ this, false, "builders-use-substitutes", R"( - If set to `true`, Nix will instruct [remote build machines](#conf-builders) to use their own [`substituters`](#conf-substituters) if available. + If set to `true`, Nix instructs [remote build machines](#conf-builders) to use their own [`substituters`](#conf-substituters) if available. - It means that remote build hosts will fetch as many dependencies as possible from their own substituters (e.g, from `cache.nixos.org`) instead of waiting for the local machine to upload them all. + It means that remote build hosts fetches as many dependencies as possible from their own substituters (e.g, from `cache.nixos.org`) instead of waiting for the local machine to upload them all. This can drastically reduce build times if the network connection between the local machine and the remote build host is slow. )"}; @@ -415,7 +415,7 @@ public: Setting useSubstitutes{ this, true, "substitute", R"( - If set to `true` (default), Nix will use binary substitutes if + If set to `true` (default), Nix uses binary substitutes if available. This option can be disabled to force building from source. )", @@ -432,11 +432,11 @@ public: since that would allow him/her to influence the build result. Therefore, if this option is non-empty and specifies a valid group, - builds will be performed under the user accounts that are a member + builds are performed under the user accounts that are a member of the group specified here (as listed in `/etc/group`). Those user accounts should not be used for any other purpose\! - Nix will never run two builds under the same user account at the + Nix never runs two builds under the same user account at the same time. This is to prevent an obvious security hole: a malicious user writing a Nix expression that modifies the build result of a legitimate Nix expression being built by another user. Therefore it @@ -448,7 +448,7 @@ public: by the Nix account, its group should be the group specified here, and its mode should be `1775`. - If the build users group is empty, builds will be performed under + If the build users group is empty, builds areperformed under the uid of the Nix process (that is, the uid of the caller if `NIX_REMOTE` is empty, the uid under which the Nix daemon runs if `NIX_REMOTE` is `daemon`). Obviously, this should not be used @@ -503,7 +503,7 @@ public: Setting keepLog{ this, true, "keep-build-log", R"( - If set to `true` (the default), Nix will write the build log of a + If set to `true` (the default), Nix writes the build log of a derivation (i.e. the standard output and error of its builder) to the directory `/nix/var/log/nix/drvs`. The build log can be retrieved using the command `nix-store -l path`. @@ -514,8 +514,8 @@ public: this, true, "compress-build-log", R"( If set to `true` (the default), build logs written to - `/nix/var/log/nix/drvs` will be compressed on the fly using bzip2. - Otherwise, they will not be compressed. + `/nix/var/log/nix/drvs` are compressed on the fly using bzip2. + Otherwise, they are not compressed. )", {"build-compress-log"}}; @@ -534,14 +534,14 @@ public: Setting gcKeepOutputs{ this, false, "keep-outputs", R"( - If `true`, the garbage collector will keep the outputs of - non-garbage derivations. If `false` (default), outputs will be + If `true`, the garbage collector keeps the outputs of + non-garbage derivations. If `false` (default), outputs are deleted unless they are GC roots themselves (or reachable from other roots). In general, outputs must be registered as roots separately. However, even if the output of a derivation is registered as a root, the - collector will still delete store paths that are used only at build + collector still deletes store paths that are used only at build time (e.g., the C compiler, or source tarballs downloaded from the network). To prevent it from doing so, set this option to `true`. )", @@ -550,9 +550,9 @@ public: Setting gcKeepDerivations{ this, true, "keep-derivations", R"( - If `true` (default), the garbage collector will keep the derivations - from which non-garbage store paths were built. If `false`, they will - be deleted unless explicitly registered as a root (or reachable from + If `true` (default), the garbage collector keeps the derivations + from which non-garbage store paths were built. If `false`, they are + deleted unless explicitly registered as a root (or reachable from other roots). Keeping derivation around is useful for querying and traceability @@ -582,7 +582,7 @@ public: If `true`, when you add a Nix derivation to a user environment, the path of the derivation is stored in the user environment. Thus, the - derivation will not be garbage-collected until the user environment + derivation isn't garbage-collected until the user environment generation is deleted (`nix-env --delete-generations`). To prevent build-time-only dependencies from being collected, you should also turn on `keep-outputs`. @@ -603,9 +603,9 @@ public: #endif , "sandbox", R"( - If set to `true`, builds will be performed in a *sandboxed + If set to `true`, builds are performed in a *sandboxed environment*, i.e., they’re isolated from the normal file system - hierarchy and will only see their dependencies in the Nix store, + hierarchy and only see their dependencies in the Nix store, the temporary build directory, private versions of `/proc`, `/dev`, `/dev/shm` and `/dev/pts` (on Linux), and the paths configured with the `sandbox-paths` option. This is useful to @@ -634,13 +634,13 @@ public: R"( A list of paths bind-mounted into Nix sandbox environments. You can use the syntax `target=source` to mount a path in a different - location in the sandbox; for instance, `/bin=/nix-bin` will mount + location in the sandbox; for instance, `/bin=/nix-bin` mounts the path `/nix-bin` as `/bin` inside the sandbox. If *source* is followed by `?`, then it is not an error if *source* does not exist; - for example, `/dev/nvidiactl?` specifies that `/dev/nvidiactl` will + for example, `/dev/nvidiactl?` specifies that `/dev/nvidiactl` only be mounted in the sandbox if it exists in the host filesystem. - If the source is in the Nix store, then its closure will be added to + If the source is in the Nix store, then its closure is added to the sandbox as well. Depending on how Nix was built, the default value for this option @@ -655,15 +655,15 @@ public: Setting requireDropSupplementaryGroups{this, isRootUser(), "require-drop-supplementary-groups", R"( Following the principle of least privilege, - Nix will attempt to drop supplementary groups when building with sandboxing. + Nix attempts to drop supplementary groups when building with sandboxing. However this can fail under some circumstances. For example, if the user lacks the `CAP_SETGID` capability. Search `setgroups(2)` for `EPERM` to find more detailed information on this. - If you encounter such a failure, setting this option to `false` will let you ignore it and continue. + If you encounter such a failure, setting this option to `false` enables you to ignore it and continue. But before doing so, you should consider the security implications carefully. - Not dropping supplementary groups means the build sandbox will be less restricted than intended. + Not dropping supplementary groups means the build sandbox is less restricted than intended. This option defaults to `true` when the user is root (since `root` usually has permissions to call setgroups) @@ -699,12 +699,12 @@ public: R"( The directory on the host, in which derivations' temporary build directories are created. - If not set, Nix will use the system temporary directory indicated by the `TMPDIR` environment variable. + If not set, Nix uses the system temporary directory indicated by the `TMPDIR` environment variable. Note that builds are often performed by the Nix daemon, so its `TMPDIR` is used, and not that of the Nix command line interface. This is also the location where [`--keep-failed`](@docroot@/command-ref/opt-common.md#opt-keep-failed) leaves its files. - If Nix runs without sandbox, or if the platform does not support sandboxing with bind mounts (e.g. macOS), then the [`builder`](@docroot@/language/derivations.md#attr-builder)'s environment will contain this directory, instead of the virtual location [`sandbox-build-dir`](#conf-sandbox-build-dir). + If Nix runs without sandbox, or if the platform does not support sandboxing with bind mounts (e.g. macOS), then the [`builder`](@docroot@/language/derivations.md#attr-builder)'s environment contains this directory instead of the virtual location [`sandbox-build-dir`](#conf-sandbox-build-dir). )"}; Setting allowedImpureHostPrefixes{this, {}, "allowed-impure-host-deps", @@ -745,12 +745,11 @@ public: 3. The path to the build's derivation - 4. The path to the build's scratch directory. This directory will - exist only if the build was run with `--keep-failed`. + 4. The path to the build's scratch directory. This directory + exists only if the build was run with `--keep-failed`. - The stderr and stdout output from the diff hook will not be - displayed to the user. Instead, it will print to the nix-daemon's - log. + The stderr and stdout output from the diff hook isn't + displayed to the user. Instead, it print to the nix-daemon's log. When using the Nix daemon, `diff-hook` must be set in the `nix.conf` configuration file, and cannot be passed at the command line. @@ -788,8 +787,8 @@ public: this, 60 * 60, "tarball-ttl", R"( The number of seconds a downloaded tarball is considered fresh. If - the cached tarball is stale, Nix will check whether it is still up - to date using the ETag header. Nix will download a new version if + the cached tarball is stale, Nix checks whether it is still up + to date using the ETag header. Nix downloads a new version if the ETag header is unsupported, or the cached ETag doesn't match. Setting the TTL to `0` forces Nix to always check if the tarball is @@ -824,7 +823,7 @@ public: R"( System types of executables that can be run on this machine. - Nix will only build a given [store derivation](@docroot@/glossary.md#gloss-store-derivation) locally when its `system` attribute equals any of the values specified here or in the [`system` option](#conf-system). + Nix only builds a given [store derivation](@docroot@/glossary.md#gloss-store-derivation) locally when its `system` attribute equals any of the values specified here or in the [`system` option](#conf-system). Setting this can be useful to build derivations locally on compatible machines: - `i686-linux` executables can be run on `x86_64-linux` machines (set by default) @@ -834,7 +833,7 @@ public: - `qemu-user` may be used to support non-native platforms (though this may be slow and buggy) - Build systems will usually detect the target platform to be the current physical system and therefore produce machine code incompatible with what may be intended in the derivation. + Build systems usually detect the target platform to be the current physical system and therefore produce machine code incompatible with what may be intended in the derivation. You should design your derivation's `builder` accordingly and cross-check the results when using this option against natively-built versions of your derivation. )", {}, @@ -924,7 +923,7 @@ public: this, 3600, "narinfo-cache-negative-ttl", R"( The TTL in seconds for negative lookups. - If a store path is queried from a [substituter](#conf-substituters) but was not found, there will be a negative lookup cached in the local disk cache database for the specified duration. + If a store path is queried from a [substituter](#conf-substituters) but was not found, a negative lookup is cached in the local disk cache database for the specified duration. Set to `0` to force updating the lookup cache. @@ -940,7 +939,7 @@ public: this, 30 * 24 * 3600, "narinfo-cache-positive-ttl", R"( The TTL in seconds for positive lookups. If a store path is queried - from a substituter, the result of the query will be cached in the + from a substituter, the result of the query is cached in the local disk cache database including some of the NAR metadata. The default TTL is a month, setting a shorter TTL for positive lookups can be useful for binary caches that have frequent garbage @@ -1026,7 +1025,7 @@ public: Setting netrcFile{ this, fmt("%s/%s", nixConfDir, "netrc"), "netrc-file", R"( - If set to an absolute path to a `netrc` file, Nix will use the HTTP + If set to an absolute path to a `netrc` file, Nix uses the HTTP authentication credentials in this file when trying to download from a remote host through HTTP or HTTPS. Defaults to `$NIX_CONF_DIR/netrc`. @@ -1052,7 +1051,7 @@ public: this, getDefaultSSLCertFile(), "ssl-cert-file", R"( The path of a file containing CA certificates used to - authenticate `https://` downloads. Nix by default will use + authenticate `https://` downloads. Nix by default uses the first of the following files that exists: 1. `/etc/ssl/certs/ca-certificates.crt` @@ -1084,7 +1083,7 @@ public: (Linux-specific.) By default, builders on Linux cannot acquire new privileges by calling setuid/setgid programs or programs that have file capabilities. For example, programs such as `sudo` or `ping` - will fail. (Note that in sandbox builds, no such programs are + should fail. (Note that in sandbox builds, no such programs are available unless you bind-mount them into the sandbox via the `sandbox-paths` option.) You can allow the use of such programs by enabling this option. This is impure and usually undesirable, but @@ -1108,7 +1107,7 @@ public: this, {}, "hashed-mirrors", R"( A list of web servers used by `builtins.fetchurl` to obtain files by - hash. Given a hash algorithm *ha* and a base-16 hash *h*, Nix will try to + hash. Given a hash algorithm *ha* and a base-16 hash *h*, Nix tries to download the file from *hashed-mirror*/*ha*/*h*. This allows files to be downloaded even if they have disappeared from their original URI. For example, given an example mirror `http://tarballs.nixos.org/`, @@ -1123,7 +1122,7 @@ public: Nix will attempt to download this file from `http://tarballs.nixos.org/sha256/2c26b46b68ffc68ff99b453c1d30413413422d706483bfa0f98a5e886266e7ae` - first. If it is not available there, if will try the original URI. + first. If it is not available there, it tries the original URI. )"}; Setting minFree{ @@ -1155,8 +1154,8 @@ public: Setting allowSymlinkedStore{ this, false, "allow-symlinked-store", R"( - If set to `true`, Nix will stop complaining if the store directory - (typically /nix/store) contains symlink components. + If set to `true`, Nix stops complaining if the store directory + (typically `/nix/store`) contains symlink components. This risks making some builds "impure" because builders sometimes "canonicalise" paths by resolving all symlink components. Problems @@ -1168,7 +1167,7 @@ public: Setting useXDGBaseDirectories{ this, false, "use-xdg-base-directories", R"( - If set to `true`, Nix will conform to the [XDG Base Directory Specification] for files in `$HOME`. + If set to `true`, Nix conforms to the [XDG Base Directory Specification] for files in `$HOME`. The environment variables used to implement this are documented in the [Environment Variables section](@docroot@/command-ref/env-common.md). [XDG Base Directory Specification]: https://specifications.freedesktop.org/basedir-spec/basedir-spec-latest.html @@ -1206,7 +1205,7 @@ public: If the user is trusted (see `trusted-users` option), when building a fixed-output derivation, environment variables set in this option - will be passed to the builder if they are listed in [`impureEnvVars`](@docroot@/language/advanced-attributes.md#adv-attr-impureEnvVars). + is passed to the builder if they are listed in [`impureEnvVars`](@docroot@/language/advanced-attributes.md#adv-attr-impureEnvVars). This option is useful for, e.g., setting `https_proxy` for fixed-output derivations and in a multi-user Nix installation, or diff --git a/src/libstore/include/nix/store/local-fs-store.hh b/src/libstore/include/nix/store/local-fs-store.hh index f9421b7febc..d5fafb0c61b 100644 --- a/src/libstore/include/nix/store/local-fs-store.hh +++ b/src/libstore/include/nix/store/local-fs-store.hh @@ -27,12 +27,12 @@ struct LocalFSStoreConfig : virtual StoreConfig PathSetting stateDir{this, rootDir.get() ? *rootDir.get() + "/nix/var/nix" : settings.nixStateDir, "state", - "Directory where Nix will store state."}; + "Directory where Nix stores state."}; PathSetting logDir{this, rootDir.get() ? *rootDir.get() + "/nix/var/log/nix" : settings.nixLogDir, "log", - "directory where Nix will store log files."}; + "directory where Nix stores log files."}; PathSetting realStoreDir{this, rootDir.get() ? *rootDir.get() + "/nix/store" : storeDir, "real", diff --git a/src/libstore/include/nix/store/local-store.hh b/src/libstore/include/nix/store/local-store.hh index efc59dc8cb7..9a118fcc517 100644 --- a/src/libstore/include/nix/store/local-store.hh +++ b/src/libstore/include/nix/store/local-store.hh @@ -54,7 +54,7 @@ struct LocalStoreConfig : std::enable_shared_from_this, virtua R"( Allow this store to be opened when its [database](@docroot@/glossary.md#gloss-nix-database) is on a read-only filesystem. - Normally Nix will attempt to open the store database in read-write mode, even for querying (when write access is not needed), causing it to fail if the database is on a read-only filesystem. + Normally Nix attempts to open the store database in read-write mode, even for querying (when write access is not needed), causing it to fail if the database is on a read-only filesystem. Enable read-only mode to disable locking and open the SQLite database with the [`immutable` parameter](https://www.sqlite.org/c3ref/open.html) set. diff --git a/src/libstore/include/nix/store/s3-binary-cache-store.hh b/src/libstore/include/nix/store/s3-binary-cache-store.hh index 9a123602e41..c38591e60f3 100644 --- a/src/libstore/include/nix/store/s3-binary-cache-store.hh +++ b/src/libstore/include/nix/store/s3-binary-cache-store.hh @@ -25,7 +25,7 @@ struct S3BinaryCacheStoreConfig : std::enable_shared_from_this **Note** > - > This endpoint must support HTTPS and will use path-based + > This endpoint must support HTTPS and uses path-based > addressing instead of virtual host based addressing. )"}; diff --git a/src/libutil/experimental-features.cc b/src/libutil/experimental-features.cc index 7dee1f5c765..88f3783f552 100644 --- a/src/libutil/experimental-features.cc +++ b/src/libutil/experimental-features.cc @@ -107,7 +107,7 @@ constexpr std::array xpFeatureDetails .name = "git-hashing", .description = R"( Allow creating (content-addressed) store objects which are hashed via Git's hashing algorithm. - These store objects will not be understandable by older versions of Nix. + These store objects aren't understandable by older versions of Nix. )", .trackingUrl = "https://github.com/NixOS/nix/milestone/41", }, diff --git a/src/libutil/include/nix/util/logging.hh b/src/libutil/include/nix/util/logging.hh index 765975faa2a..dabfac48390 100644 --- a/src/libutil/include/nix/util/logging.hh +++ b/src/libutil/include/nix/util/logging.hh @@ -55,7 +55,7 @@ struct LoggerSettings : Config Setting jsonLogPath{ this, "", "json-log-path", R"( - A file or unix socket to which JSON records of Nix's log output will be + A file or unix socket to which JSON records of Nix's log output are written, in the same format as `--log-format internal-json` (without the `@nix ` prefixes on each line). Concurrent writes to the same file by multiple Nix processes are not supported and diff --git a/src/nix-build/nix-build.cc b/src/nix-build/nix-build.cc index 80ebf6bfaba..3313c02aa61 100644 --- a/src/nix-build/nix-build.cc +++ b/src/nix-build/nix-build.cc @@ -474,7 +474,7 @@ static void main_nix_build(int argc, char * * argv) } catch (Error & e) { logError(e.info()); - notice("will use bash from your environment"); + notice("uses bash from your environment"); shell = "bash"; } } diff --git a/src/nix/unix/daemon.cc b/src/nix/unix/daemon.cc index 4132d5be2cd..a14632c2f0b 100644 --- a/src/nix/unix/daemon.cc +++ b/src/nix/unix/daemon.cc @@ -572,7 +572,7 @@ struct CmdDaemon : Command addFlag({ .longName = "force-untrusted", - .description = "Force the daemon to not trust connecting clients. The connection will be processed by the receiving daemon before forwarding commands.", + .description = "Force the daemon to not trust connecting clients. The connection is processed by the receiving daemon before forwarding commands.", .handler = {[&]() { isTrustedOpt = NotTrusted; }}, diff --git a/tests/functional/fetchGit.sh b/tests/functional/fetchGit.sh index 5832facda62..a41aa35c028 100755 --- a/tests/functional/fetchGit.sh +++ b/tests/functional/fetchGit.sh @@ -81,7 +81,7 @@ path2=$(nix eval --raw --expr "(builtins.fetchGit { url = file://$repo; rev = \" [[ $(nix eval --raw --expr "builtins.readFile (fetchGit { url = file://$repo; rev = \"$rev2\"; } + \"/hello\")") = world ]] # But without a hash, it fails. -expectStderr 1 nix eval --expr 'builtins.fetchGit "file:///foo"' | grepQuiet "'fetchGit' will not fetch unlocked input" +expectStderr 1 nix eval --expr 'builtins.fetchGit "file:///foo"' | grepQuiet "'fetchGit' doesn't fetch unlocked input" # Fetch again. This should be cached. mv $repo ${repo}-tmp diff --git a/tests/functional/flakes/unlocked-override.sh b/tests/functional/flakes/unlocked-override.sh index 512aca401d3..ed05440de03 100755 --- a/tests/functional/flakes/unlocked-override.sh +++ b/tests/functional/flakes/unlocked-override.sh @@ -33,7 +33,7 @@ echo 456 > "$flake1Dir"/x.nix # Dirty overrides require --allow-dirty-locks. expectStderr 1 nix flake lock "$flake2Dir" --override-input flake1 "$TEST_ROOT/flake1" | - grepQuiet "Will not write lock file.*because it has an unlocked input" + grepQuiet "Not writing lock file.*because it has an unlocked input" nix flake lock "$flake2Dir" --override-input flake1 "$TEST_ROOT/flake1" --allow-dirty-locks diff --git a/tests/nixos/github-flakes.nix b/tests/nixos/github-flakes.nix index 30ab1f3331d..3d7a77ec52b 100644 --- a/tests/nixos/github-flakes.nix +++ b/tests/nixos/github-flakes.nix @@ -1,8 +1,7 @@ -{ - lib, - config, - nixpkgs, - ... +{ lib +, config +, nixpkgs +, ... }: let pkgs = config.nodes.client.nixpkgs.pkgs; @@ -147,12 +146,11 @@ in }; client = - { - config, - lib, - pkgs, - nodes, - ... + { config + , lib + , pkgs + , nodes + , ... }: { virtualisation.writableStore = true; @@ -227,7 +225,7 @@ in # Fetching without a narHash should succeed if trust-github is set and fail otherwise. client.succeed(f"nix eval --raw --expr 'builtins.fetchTree github:github:fancy-enterprise/private-flake/{info['revision']}'") out = client.fail(f"nix eval --no-trust-tarballs-from-git-forges --raw --expr 'builtins.fetchTree github:github:fancy-enterprise/private-flake/{info['revision']}' 2>&1") - assert "will not fetch unlocked input" in out, "--no-trust-tarballs-from-git-forges did not fail with the expected error" + assert "doesn't fetch unlocked input" in out, "--no-trust-tarballs-from-git-forges did not fail with the expected error" # Shut down the web server. The flake should be cached on the client. github.succeed("systemctl stop httpd.service") From 5abaf361a41facb2baeee05aa58b9b5e3aa08f45 Mon Sep 17 00:00:00 2001 From: Pol Dellaiera Date: Thu, 12 Jun 2025 19:06:13 +0200 Subject: [PATCH 140/218] docker: reduce duplicates, use `coreutils-full` --- docker.nix | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docker.nix b/docker.nix index e3a0635d017..8262d1b7049 100644 --- a/docker.nix +++ b/docker.nix @@ -290,7 +290,7 @@ let echo "${channelURL} ${channelName}" > $out${userHome}/.nix-channels mkdir -p $out/bin $out/usr/bin - ln -s ${pkgs.coreutils}/bin/env $out/usr/bin/env + ln -s ${pkgs.coreutils-full}/bin/env $out/usr/bin/env ln -s ${pkgs.bashInteractive}/bin/bash $out/bin/sh '' From 5862f38d00b3d84ee9272f90983a52ba1b9c71ee Mon Sep 17 00:00:00 2001 From: Pol Dellaiera Date: Thu, 12 Jun 2025 19:07:20 +0200 Subject: [PATCH 141/218] docker: use `callPackage`, parametrise the image build --- docker.nix | 90 +++++++++++++++++++++++--------------- flake.nix | 3 +- tests/nixos/nix-docker.nix | 10 +---- 3 files changed, 58 insertions(+), 45 deletions(-) diff --git a/docker.nix b/docker.nix index 8262d1b7049..060dcd8f08a 100644 --- a/docker.nix +++ b/docker.nix @@ -1,6 +1,10 @@ { - pkgs ? import { }, - lib ? pkgs.lib, + # Core dependencies + pkgs, + lib, + runCommand, + buildPackages, + # Image configuration name ? "nix", tag ? "latest", bundleNixpkgs ? true, @@ -14,36 +18,52 @@ gid ? 0, uname ? "root", gname ? "root", + # Default Packages + nix, + bashInteractive, + coreutils-full, + gnutar, + gzip, + gnugrep, + which, + curl, + less, + wget, + man, + cacert, + findutils, + iana-etc, + git, + openssh, + # Other dependencies + shadow, }: let - defaultPkgs = - with pkgs; - [ - nix - bashInteractive - coreutils-full - gnutar - gzip - gnugrep - which - curl - less - wget - man - cacert.out - findutils - iana-etc - git - openssh - ] - ++ extraPkgs; + defaultPkgs = [ + nix + bashInteractive + coreutils-full + gnutar + gzip + gnugrep + which + curl + less + wget + man + cacert.out + findutils + iana-etc + git + openssh + ] ++ extraPkgs; users = { root = { uid = 0; - shell = "${pkgs.bashInteractive}/bin/bash"; + shell = lib.getExe bashInteractive; home = "/root"; gid = 0; groups = [ "root" ]; @@ -52,7 +72,7 @@ let nobody = { uid = 65534; - shell = "${pkgs.shadow}/bin/nologin"; + shell = lib.getExe' shadow "nologin"; home = "/var/empty"; gid = 65534; groups = [ "nobody" ]; @@ -63,7 +83,7 @@ let // lib.optionalAttrs (uid != 0) { "${uname}" = { uid = uid; - shell = "${pkgs.bashInteractive}/bin/bash"; + shell = lib.getExe bashInteractive; home = "/home/${uname}"; gid = gid; groups = [ "${gname}" ]; @@ -170,7 +190,7 @@ let baseSystem = let nixpkgs = pkgs.path; - channel = pkgs.runCommand "channel-nixos" { inherit bundleNixpkgs; } '' + channel = runCommand "channel-nixos" { inherit bundleNixpkgs; } '' mkdir $out if [ "$bundleNixpkgs" ]; then ln -s ${ @@ -182,11 +202,11 @@ let echo "[]" > $out/manifest.nix fi ''; - rootEnv = pkgs.buildPackages.buildEnv { + rootEnv = buildPackages.buildEnv { name = "root-profile-env"; paths = defaultPkgs; }; - manifest = pkgs.buildPackages.runCommand "manifest.nix" { } '' + manifest = buildPackages.runCommand "manifest.nix" { } '' cat > $out < $out${userHome}/.nix-channels mkdir -p $out/bin $out/usr/bin - ln -s ${pkgs.coreutils-full}/bin/env $out/usr/bin/env - ln -s ${pkgs.bashInteractive}/bin/bash $out/bin/sh + ln -s ${lib.getExe' coreutils-full "env"} $out/usr/bin/env + ln -s ${lib.getExe bashInteractive} $out/bin/sh '' + (lib.optionalString (flake-registry-path != null) '' @@ -300,7 +320,7 @@ let globalFlakeRegistryPath="$nixCacheDir/flake-registry.json" ln -s ${flake-registry-path} $out$globalFlakeRegistryPath mkdir -p $out/nix/var/nix/gcroots/auto - rootName=$(${pkgs.nix}/bin/nix --extra-experimental-features nix-command hash file --type sha1 --base32 <(echo -n $globalFlakeRegistryPath)) + rootName=$(${lib.getExe' nix "nix"} --extra-experimental-features nix-command hash file --type sha1 --base32 <(echo -n $globalFlakeRegistryPath)) ln -s $globalFlakeRegistryPath $out/nix/var/nix/gcroots/auto/$rootName '') ); @@ -332,7 +352,7 @@ pkgs.dockerTools.buildLayeredImageWithNixDb { ''; config = { - Cmd = [ "${userHome}/.nix-profile/bin/bash" ]; + Cmd = [ (lib.getExe bashInteractive) ]; User = "${toString uid}:${toString gid}"; Env = [ "USER=${uname}" diff --git a/flake.nix b/flake.nix index 7d7c4d4c2ae..69bd2a21adb 100644 --- a/flake.nix +++ b/flake.nix @@ -404,8 +404,7 @@ dockerImage = let pkgs = nixpkgsFor.${system}.native; - image = import ./docker.nix { - inherit pkgs; + image = pkgs.callPackage ./docker.nix { tag = pkgs.nix.version; }; in diff --git a/tests/nixos/nix-docker.nix b/tests/nixos/nix-docker.nix index c58a00cddbb..f1c218585a6 100644 --- a/tests/nixos/nix-docker.nix +++ b/tests/nixos/nix-docker.nix @@ -1,21 +1,15 @@ # Test the container built by ../../docker.nix. { - lib, config, - nixpkgs, - hostPkgs, ... }: let pkgs = config.nodes.machine.nixpkgs.pkgs; - nixImage = import ../../docker.nix { - inherit (config.nodes.machine.nixpkgs) pkgs; - }; - nixUserImage = import ../../docker.nix { - inherit (config.nodes.machine.nixpkgs) pkgs; + nixImage = pkgs.callPackage ../../docker.nix { }; + nixUserImage = pkgs.callPackage ../../docker.nix { name = "nix-user"; uid = 1000; gid = 1000; From 6eb4ee68556af97ab88f3f909f3158e57429f6af Mon Sep 17 00:00:00 2001 From: Pol Dellaiera Date: Thu, 12 Jun 2025 19:18:07 +0200 Subject: [PATCH 142/218] docker: replace `git` with `gitMinimal` --- docker.nix | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docker.nix b/docker.nix index 060dcd8f08a..e97ed175280 100644 --- a/docker.nix +++ b/docker.nix @@ -33,7 +33,7 @@ cacert, findutils, iana-etc, - git, + gitMinimal, openssh, # Other dependencies shadow, @@ -54,7 +54,7 @@ let cacert.out findutils iana-etc - git + gitMinimal openssh ] ++ extraPkgs; From 0f6cb33763e1114e738d7ae28baa35ed261bdfb0 Mon Sep 17 00:00:00 2001 From: Samuli Thomasson Date: Thu, 12 Jun 2025 21:27:30 +0200 Subject: [PATCH 143/218] fix throwing output paths out of sandbox paths It seems obvious that erasing any output paths from pathsInChroot needs to happen after getPathsInSandbox(), not before. Signed-off-by: Samuli Thomasson --- .../unix/build/linux-derivation-builder.cc | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/src/libstore/unix/build/linux-derivation-builder.cc b/src/libstore/unix/build/linux-derivation-builder.cc index c27b87163ad..b23c8003f5c 100644 --- a/src/libstore/unix/build/linux-derivation-builder.cc +++ b/src/libstore/unix/build/linux-derivation-builder.cc @@ -368,6 +368,13 @@ struct ChrootLinuxDerivationBuilder : LinuxDerivationBuilder if (buildUser && chown(chrootStoreDir.c_str(), 0, buildUser->getGID()) == -1) throw SysError("cannot change ownership of '%1%'", chrootStoreDir); + pathsInChroot = getPathsInSandbox(); + + for (auto & i : inputPaths) { + auto p = store.printStorePath(i); + pathsInChroot.insert_or_assign(p, store.toRealPath(p)); + } + /* If we're repairing, checking or rebuilding part of a multiple-outputs derivation, it's possible that we're rebuilding a path that is in settings.sandbox-paths @@ -391,13 +398,6 @@ struct ChrootLinuxDerivationBuilder : LinuxDerivationBuilder chownToBuilder(*cgroup + "/cgroup.threads"); // chownToBuilder(*cgroup + "/cgroup.subtree_control"); } - - pathsInChroot = getPathsInSandbox(); - - for (auto & i : inputPaths) { - auto p = store.printStorePath(i); - pathsInChroot.insert_or_assign(p, store.toRealPath(p)); - } } Strings getPreBuildHookArgs() override From 6587e7bcff6bfc7718cea96e64823fc2742d3d6e Mon Sep 17 00:00:00 2001 From: Sergei Zimmerman Date: Thu, 12 Jun 2025 19:42:50 +0000 Subject: [PATCH 144/218] libexpr: Add and use `lambda` getter --- src/libcmd/repl.cc | 2 +- src/libexpr-tests/primops.cc | 4 ++-- src/libexpr/eval-profiler.cc | 2 +- src/libexpr/eval.cc | 22 +++++++++++----------- src/libexpr/include/nix/expr/value.hh | 3 +++ src/libexpr/primops.cc | 4 ++-- src/libexpr/print.cc | 8 ++++---- src/libexpr/value-to-xml.cc | 12 ++++++------ src/libflake/flake.cc | 4 ++-- src/nix-build/nix-build.cc | 4 ++-- src/nix/flake.cc | 4 ++-- 11 files changed, 36 insertions(+), 33 deletions(-) diff --git a/src/libcmd/repl.cc b/src/libcmd/repl.cc index f01f128603b..8170bd579b9 100644 --- a/src/libcmd/repl.cc +++ b/src/libcmd/repl.cc @@ -484,7 +484,7 @@ ProcessLineResult NixRepl::processLine(std::string line) auto path = state->coerceToPath(noPos, v, context, "while evaluating the filename to edit"); return {path, 0}; } else if (v.isLambda()) { - auto pos = state->positions[v.payload.lambda.fun->pos]; + auto pos = state->positions[v.lambda().fun->pos]; if (auto path = std::get_if(&pos.origin)) return {*path, pos.line}; else diff --git a/src/libexpr-tests/primops.cc b/src/libexpr-tests/primops.cc index 2f864f2c25d..6c301f15762 100644 --- a/src/libexpr-tests/primops.cc +++ b/src/libexpr-tests/primops.cc @@ -667,8 +667,8 @@ namespace nix { auto v = eval("derivation"); ASSERT_EQ(v.type(), nFunction); ASSERT_TRUE(v.isLambda()); - ASSERT_NE(v.payload.lambda.fun, nullptr); - ASSERT_TRUE(v.payload.lambda.fun->hasFormals()); + ASSERT_NE(v.lambda().fun, nullptr); + ASSERT_TRUE(v.lambda().fun->hasFormals()); } TEST_F(PrimOpTest, currentTime) { diff --git a/src/libexpr/eval-profiler.cc b/src/libexpr/eval-profiler.cc index 7053e4ec7d6..b65bc3a4d45 100644 --- a/src/libexpr/eval-profiler.cc +++ b/src/libexpr/eval-profiler.cc @@ -204,7 +204,7 @@ FrameInfo SampleStack::getFrameInfoFromValueAndPos(const Value & v, std::spanpos; - case tLambda: return payload.lambda.fun->pos; + case tLambda: return lambda().fun->pos; case tApp: return payload.app.left->determinePos(pos); default: return pos; } @@ -610,7 +610,7 @@ std::optional EvalState::getDoc(Value & v) }; } if (v.isLambda()) { - auto exprLambda = v.payload.lambda.fun; + auto exprLambda = v.lambda().fun; std::ostringstream s; std::string name; @@ -1567,13 +1567,13 @@ void EvalState::callFunction(Value & fun, std::span args, Value & vRes, if (vCur.isLambda()) { - ExprLambda & lambda(*vCur.payload.lambda.fun); + ExprLambda & lambda(*vCur.lambda().fun); auto size = (!lambda.arg ? 0 : 1) + (lambda.hasFormals() ? lambda.formals->formals.size() : 0); Env & env2(allocEnv(size)); - env2.up = vCur.payload.lambda.env; + env2.up = vCur.lambda().env; Displacement displ = 0; @@ -1603,7 +1603,7 @@ void EvalState::callFunction(Value & fun, std::span args, Value & vRes, symbols[i.name]) .atPos(lambda.pos) .withTrace(pos, "from call site") - .withFrame(*fun.payload.lambda.env, lambda) + .withFrame(*fun.lambda().env, lambda) .debugThrow(); } env2.values[displ++] = i.def->maybeThunk(*this, env2); @@ -1630,7 +1630,7 @@ void EvalState::callFunction(Value & fun, std::span args, Value & vRes, .atPos(lambda.pos) .withTrace(pos, "from call site") .withSuggestions(suggestions) - .withFrame(*fun.payload.lambda.env, lambda) + .withFrame(*fun.lambda().env, lambda) .debugThrow(); } unreachable(); @@ -1825,14 +1825,14 @@ void EvalState::autoCallFunction(const Bindings & args, Value & fun, Value & res } } - if (!fun.isLambda() || !fun.payload.lambda.fun->hasFormals()) { + if (!fun.isLambda() || !fun.lambda().fun->hasFormals()) { res = fun; return; } - auto attrs = buildBindings(std::max(static_cast(fun.payload.lambda.fun->formals->formals.size()), args.size())); + auto attrs = buildBindings(std::max(static_cast(fun.lambda().fun->formals->formals.size()), args.size())); - if (fun.payload.lambda.fun->formals->ellipsis) { + if (fun.lambda().fun->formals->ellipsis) { // If the formals have an ellipsis (eg the function accepts extra args) pass // all available automatic arguments (which includes arguments specified on // the command line via --arg/--argstr) @@ -1840,7 +1840,7 @@ void EvalState::autoCallFunction(const Bindings & args, Value & fun, Value & res attrs.insert(v); } else { // Otherwise, only pass the arguments that the function accepts - for (auto & i : fun.payload.lambda.fun->formals->formals) { + for (auto & i : fun.lambda().fun->formals->formals) { auto j = args.get(i.name); if (j) { attrs.insert(*j); @@ -1850,7 +1850,7 @@ Nix attempted to evaluate a function as a top level expression; in this case it must have its arguments supplied either by default values, or passed explicitly with '--arg' or '--argstr'. See https://nixos.org/manual/nix/stable/language/constructs.html#functions.)", symbols[i.name]) - .atPos(i.pos).withFrame(*fun.payload.lambda.env, *fun.payload.lambda.fun).debugThrow(); + .atPos(i.pos).withFrame(*fun.lambda().env, *fun.lambda().fun).debugThrow(); } } } diff --git a/src/libexpr/include/nix/expr/value.hh b/src/libexpr/include/nix/expr/value.hh index febe36f80aa..26ec5ff3813 100644 --- a/src/libexpr/include/nix/expr/value.hh +++ b/src/libexpr/include/nix/expr/value.hh @@ -482,6 +482,9 @@ public: NixFloat fpoint() const { return payload.fpoint; } + + Lambda lambda() const + { return payload.lambda; } }; diff --git a/src/libexpr/primops.cc b/src/libexpr/primops.cc index 48bc272f936..a24695bcdbe 100644 --- a/src/libexpr/primops.cc +++ b/src/libexpr/primops.cc @@ -3138,12 +3138,12 @@ static void prim_functionArgs(EvalState & state, const PosIdx pos, Value * * arg if (!args[0]->isLambda()) state.error("'functionArgs' requires a function").atPos(pos).debugThrow(); - if (!args[0]->payload.lambda.fun->hasFormals()) { + if (!args[0]->lambda().fun->hasFormals()) { v.mkAttrs(&state.emptyBindings); return; } - const auto &formals = args[0]->payload.lambda.fun->formals->formals; + const auto &formals = args[0]->lambda().fun->formals->formals; auto attrs = state.buildBindings(formals.size()); for (auto & i : formals) attrs.insert(i.name, state.getBool(i.def), i.pos); diff --git a/src/libexpr/print.cc b/src/libexpr/print.cc index 06bae9c5c3a..f0e0eed2de3 100644 --- a/src/libexpr/print.cc +++ b/src/libexpr/print.cc @@ -453,13 +453,13 @@ class Printer if (v.isLambda()) { output << "lambda"; - if (v.payload.lambda.fun) { - if (v.payload.lambda.fun->name) { - output << " " << state.symbols[v.payload.lambda.fun->name]; + if (v.lambda().fun) { + if (v.lambda().fun->name) { + output << " " << state.symbols[v.lambda().fun->name]; } std::ostringstream s; - s << state.positions[v.payload.lambda.fun->pos]; + s << state.positions[v.lambda().fun->pos]; output << " @ " << filterANSIEscapes(toView(s)); } } else if (v.isPrimOp()) { diff --git a/src/libexpr/value-to-xml.cc b/src/libexpr/value-to-xml.cc index e26fff71ba4..54ff06f9e1a 100644 --- a/src/libexpr/value-to-xml.cc +++ b/src/libexpr/value-to-xml.cc @@ -126,18 +126,18 @@ static void printValueAsXML(EvalState & state, bool strict, bool location, break; } XMLAttrs xmlAttrs; - if (location) posToXML(state, xmlAttrs, state.positions[v.payload.lambda.fun->pos]); + if (location) posToXML(state, xmlAttrs, state.positions[v.lambda().fun->pos]); XMLOpenElement _(doc, "function", xmlAttrs); - if (v.payload.lambda.fun->hasFormals()) { + if (v.lambda().fun->hasFormals()) { XMLAttrs attrs; - if (v.payload.lambda.fun->arg) attrs["name"] = state.symbols[v.payload.lambda.fun->arg]; - if (v.payload.lambda.fun->formals->ellipsis) attrs["ellipsis"] = "1"; + if (v.lambda().fun->arg) attrs["name"] = state.symbols[v.lambda().fun->arg]; + if (v.lambda().fun->formals->ellipsis) attrs["ellipsis"] = "1"; XMLOpenElement _(doc, "attrspat", attrs); - for (auto & i : v.payload.lambda.fun->formals->lexicographicOrder(state.symbols)) + for (auto & i : v.lambda().fun->formals->lexicographicOrder(state.symbols)) doc.writeEmptyElement("attr", singletonAttrs("name", state.symbols[i.name])); } else - doc.writeEmptyElement("varpat", singletonAttrs("name", state.symbols[v.payload.lambda.fun->arg])); + doc.writeEmptyElement("varpat", singletonAttrs("name", state.symbols[v.lambda().fun->arg])); break; } diff --git a/src/libflake/flake.cc b/src/libflake/flake.cc index 24252d710d7..41141d41db9 100644 --- a/src/libflake/flake.cc +++ b/src/libflake/flake.cc @@ -252,8 +252,8 @@ static Flake readFlake( if (auto outputs = vInfo.attrs()->get(sOutputs)) { expectType(state, nFunction, *outputs->value, outputs->pos); - if (outputs->value->isLambda() && outputs->value->payload.lambda.fun->hasFormals()) { - for (auto & formal : outputs->value->payload.lambda.fun->formals->formals) { + if (outputs->value->isLambda() && outputs->value->lambda().fun->hasFormals()) { + for (auto & formal : outputs->value->lambda().fun->formals->formals) { if (formal.name != state.sSelf) flake.inputs.emplace(state.symbols[formal.name], FlakeInput { .ref = parseFlakeRef(state.fetchSettings, std::string(state.symbols[formal.name])) diff --git a/src/nix-build/nix-build.cc b/src/nix-build/nix-build.cc index 80ebf6bfaba..e302aca6787 100644 --- a/src/nix-build/nix-build.cc +++ b/src/nix-build/nix-build.cc @@ -387,8 +387,8 @@ static void main_nix_build(int argc, char * * argv) return false; } bool add = false; - if (v.type() == nFunction && v.payload.lambda.fun->hasFormals()) { - for (auto & i : v.payload.lambda.fun->formals->formals) { + if (v.type() == nFunction && v.lambda().fun->hasFormals()) { + for (auto & i : v.lambda().fun->formals->formals) { if (state->symbols[i.name] == "inNixShell") { add = true; break; diff --git a/src/nix/flake.cc b/src/nix/flake.cc index 95cf856637f..13f7363fcfd 100644 --- a/src/nix/flake.cc +++ b/src/nix/flake.cc @@ -492,8 +492,8 @@ struct CmdFlakeCheck : FlakeCommand if (!v.isLambda()) { throw Error("overlay is not a function, but %s instead", showType(v)); } - if (v.payload.lambda.fun->hasFormals() - || !argHasName(v.payload.lambda.fun->arg, "final")) + if (v.lambda().fun->hasFormals() + || !argHasName(v.lambda().fun->arg, "final")) throw Error("overlay does not take an argument named 'final'"); // FIXME: if we have a 'nixpkgs' input, use it to // evaluate the overlay. From 441fa86e82da65a71e01a5c45e3459287ab4d01e Mon Sep 17 00:00:00 2001 From: Sergei Zimmerman Date: Thu, 12 Jun 2025 19:48:42 +0000 Subject: [PATCH 145/218] libexpr: Add and use `thunk` getter --- src/libexpr/eval.cc | 10 +++++----- src/libexpr/include/nix/expr/eval-inline.hh | 4 ++-- src/libexpr/include/nix/expr/value.hh | 5 ++++- 3 files changed, 11 insertions(+), 8 deletions(-) diff --git a/src/libexpr/eval.cc b/src/libexpr/eval.cc index 51036a223f1..badb271f2af 100644 --- a/src/libexpr/eval.cc +++ b/src/libexpr/eval.cc @@ -160,10 +160,10 @@ bool Value::isTrivial() const internalType != tApp && internalType != tPrimOpApp && (internalType != tThunk - || (dynamic_cast(payload.thunk.expr) - && ((ExprAttrs *) payload.thunk.expr)->dynamicAttrs.empty()) - || dynamic_cast(payload.thunk.expr) - || dynamic_cast(payload.thunk.expr)); + || (dynamic_cast(thunk().expr) + && ((ExprAttrs *) thunk().expr)->dynamicAttrs.empty()) + || dynamic_cast(thunk().expr) + || dynamic_cast(thunk().expr)); } @@ -2163,7 +2163,7 @@ void EvalState::forceValueDeep(Value & v) try { // If the value is a thunk, we're evaling. Otherwise no trace necessary. auto dts = debugRepl && i.value->isThunk() - ? makeDebugTraceStacker(*this, *i.value->payload.thunk.expr, *i.value->payload.thunk.env, i.pos, + ? makeDebugTraceStacker(*this, *i.value->thunk().expr, *i.value->thunk().env, i.pos, "while evaluating the attribute '%1%'", symbols[i.name]) : nullptr; diff --git a/src/libexpr/include/nix/expr/eval-inline.hh b/src/libexpr/include/nix/expr/eval-inline.hh index 6e5759c0b44..97bf71a6b0f 100644 --- a/src/libexpr/include/nix/expr/eval-inline.hh +++ b/src/libexpr/include/nix/expr/eval-inline.hh @@ -89,9 +89,9 @@ Env & EvalState::allocEnv(size_t size) void EvalState::forceValue(Value & v, const PosIdx pos) { if (v.isThunk()) { - Env * env = v.payload.thunk.env; + Env * env = v.thunk().env; assert(env || v.isBlackhole()); - Expr * expr = v.payload.thunk.expr; + Expr * expr = v.thunk().expr; try { v.mkBlackhole(); //checkInterrupt(); diff --git a/src/libexpr/include/nix/expr/value.hh b/src/libexpr/include/nix/expr/value.hh index 26ec5ff3813..29f8ac37954 100644 --- a/src/libexpr/include/nix/expr/value.hh +++ b/src/libexpr/include/nix/expr/value.hh @@ -485,6 +485,9 @@ public: Lambda lambda() const { return payload.lambda; } + + ClosureThunk thunk() const + { return payload.thunk; } }; @@ -492,7 +495,7 @@ extern ExprBlackHole eBlackHole; bool Value::isBlackhole() const { - return internalType == tThunk && payload.thunk.expr == (Expr*) &eBlackHole; + return internalType == tThunk && thunk().expr == (Expr*) &eBlackHole; } void Value::mkBlackhole() From f07a9f863e16bb71ec330abf6c2ec5d5bff68788 Mon Sep 17 00:00:00 2001 From: Sergei Zimmerman Date: Thu, 12 Jun 2025 19:51:44 +0000 Subject: [PATCH 146/218] libexpr: Add and use `primOpApp` getter --- src/libexpr/eval.cc | 10 +++++----- src/libexpr/include/nix/expr/value.hh | 3 +++ 2 files changed, 8 insertions(+), 5 deletions(-) diff --git a/src/libexpr/eval.cc b/src/libexpr/eval.cc index badb271f2af..f9bff7b989d 100644 --- a/src/libexpr/eval.cc +++ b/src/libexpr/eval.cc @@ -525,9 +525,9 @@ std::ostream & operator<<(std::ostream & output, const PrimOp & primOp) const PrimOp * Value::primOpAppPrimOp() const { - Value * left = payload.primOpApp.left; + Value * left = primOpApp().left; while (left && !left->isPrimOp()) { - left = left->payload.primOpApp.left; + left = left->primOpApp().left; } if (!left) @@ -1702,7 +1702,7 @@ void EvalState::callFunction(Value & fun, std::span args, Value & vRes, Value * primOp = &vCur; while (primOp->isPrimOpApp()) { argsDone++; - primOp = primOp->payload.primOpApp.left; + primOp = primOp->primOpApp().left; } assert(primOp->isPrimOp()); auto arity = primOp->primOp()->arity; @@ -1718,8 +1718,8 @@ void EvalState::callFunction(Value & fun, std::span args, Value & vRes, Value * vArgs[maxPrimOpArity]; auto n = argsDone; - for (Value * arg = &vCur; arg->isPrimOpApp(); arg = arg->payload.primOpApp.left) - vArgs[--n] = arg->payload.primOpApp.right; + for (Value * arg = &vCur; arg->isPrimOpApp(); arg = arg->primOpApp().left) + vArgs[--n] = arg->primOpApp().right; for (size_t i = 0; i < argsLeft; ++i) vArgs[argsDone + i] = args[i]; diff --git a/src/libexpr/include/nix/expr/value.hh b/src/libexpr/include/nix/expr/value.hh index 29f8ac37954..797e3119190 100644 --- a/src/libexpr/include/nix/expr/value.hh +++ b/src/libexpr/include/nix/expr/value.hh @@ -488,6 +488,9 @@ public: ClosureThunk thunk() const { return payload.thunk; } + + FunctionApplicationThunk primOpApp() const + { return payload.primOpApp; } }; From c041d71406f706d971ee0ad53d555450de7ad03d Mon Sep 17 00:00:00 2001 From: Sergei Zimmerman Date: Thu, 12 Jun 2025 19:53:44 +0000 Subject: [PATCH 147/218] libexpr: Add and use `app` getter --- src/libexpr/eval.cc | 2 +- src/libexpr/include/nix/expr/eval-inline.hh | 2 +- src/libexpr/include/nix/expr/value.hh | 3 +++ 3 files changed, 5 insertions(+), 2 deletions(-) diff --git a/src/libexpr/eval.cc b/src/libexpr/eval.cc index f9bff7b989d..37018007fa2 100644 --- a/src/libexpr/eval.cc +++ b/src/libexpr/eval.cc @@ -148,7 +148,7 @@ PosIdx Value::determinePos(const PosIdx pos) const switch (internalType) { case tAttrs: return attrs()->pos; case tLambda: return lambda().fun->pos; - case tApp: return payload.app.left->determinePos(pos); + case tApp: return app().left->determinePos(pos); default: return pos; } #pragma GCC diagnostic pop diff --git a/src/libexpr/include/nix/expr/eval-inline.hh b/src/libexpr/include/nix/expr/eval-inline.hh index 97bf71a6b0f..7d13d7cc707 100644 --- a/src/libexpr/include/nix/expr/eval-inline.hh +++ b/src/libexpr/include/nix/expr/eval-inline.hh @@ -106,7 +106,7 @@ void EvalState::forceValue(Value & v, const PosIdx pos) } } else if (v.isApp()) - callFunction(*v.payload.app.left, *v.payload.app.right, v, pos); + callFunction(*v.app().left, *v.app().right, v, pos); } diff --git a/src/libexpr/include/nix/expr/value.hh b/src/libexpr/include/nix/expr/value.hh index 797e3119190..d25511c4557 100644 --- a/src/libexpr/include/nix/expr/value.hh +++ b/src/libexpr/include/nix/expr/value.hh @@ -491,6 +491,9 @@ public: FunctionApplicationThunk primOpApp() const { return payload.primOpApp; } + + FunctionApplicationThunk app() const + { return payload.app; } }; From e4df1891237b58050e7b10380f7f62551ade2783 Mon Sep 17 00:00:00 2001 From: Sergei Zimmerman Date: Thu, 12 Jun 2025 19:57:46 +0000 Subject: [PATCH 148/218] libexpr: Add and use `pathStr` getter --- src/libexpr-c/nix_api_value.cc | 2 +- src/libexpr/eval.cc | 6 +++--- src/libexpr/include/nix/expr/value.hh | 5 ++++- src/libexpr/primops.cc | 2 +- 4 files changed, 9 insertions(+), 6 deletions(-) diff --git a/src/libexpr-c/nix_api_value.cc b/src/libexpr-c/nix_api_value.cc index 298d9484598..8afe35a4b8b 100644 --- a/src/libexpr-c/nix_api_value.cc +++ b/src/libexpr-c/nix_api_value.cc @@ -252,7 +252,7 @@ const char * nix_get_path_string(nix_c_context * context, const nix_value * valu // We could use v.path().to_string().c_str(), but I'm concerned this // crashes. Looks like .path() allocates a CanonPath with a copy of the // string, then it gets the underlying data from that. - return v.payload.path.path; + return v.pathStr(); } NIXC_CATCH_ERRS_NULL } diff --git a/src/libexpr/eval.cc b/src/libexpr/eval.cc index 37018007fa2..66ac2e9fde9 100644 --- a/src/libexpr/eval.cc +++ b/src/libexpr/eval.cc @@ -2368,7 +2368,7 @@ BackedStringView EvalState::coerceToString( !canonicalizePath && !copyToStore ? // FIXME: hack to preserve path literals that end in a // slash, as in /foo/${x}. - v.payload.path.path + v.pathStr() : copyToStore ? store->printStorePath(copyPathToStore(context, v.path())) : std::string(v.path().path.abs()); @@ -2643,7 +2643,7 @@ void EvalState::assertEqValues(Value & v1, Value & v2, const PosIdx pos, std::st ValuePrinter(*this, v2, errorPrintOptions)) .debugThrow(); } - if (strcmp(v1.payload.path.path, v2.payload.path.path) != 0) { + if (strcmp(v1.pathStr(), v2.pathStr()) != 0) { error( "path '%s' is not equal to path '%s'", ValuePrinter(*this, v1, errorPrintOptions), @@ -2811,7 +2811,7 @@ bool EvalState::eqValues(Value & v1, Value & v2, const PosIdx pos, std::string_v return // FIXME: compare accessors by their fingerprint. v1.payload.path.accessor == v2.payload.path.accessor - && strcmp(v1.payload.path.path, v2.payload.path.path) == 0; + && strcmp(v1.pathStr(), v2.pathStr()) == 0; case nNull: return true; diff --git a/src/libexpr/include/nix/expr/value.hh b/src/libexpr/include/nix/expr/value.hh index d25511c4557..fc9ce1b39ed 100644 --- a/src/libexpr/include/nix/expr/value.hh +++ b/src/libexpr/include/nix/expr/value.hh @@ -445,7 +445,7 @@ public: assert(internalType == tPath); return SourcePath( ref(payload.path.accessor->shared_from_this()), - CanonPath(CanonPath::unchecked_t(), payload.path.path)); + CanonPath(CanonPath::unchecked_t(), pathStr())); } std::string_view string_view() const @@ -494,6 +494,9 @@ public: FunctionApplicationThunk app() const { return payload.app; } + + const char * pathStr() const + { return payload.path.path; } }; diff --git a/src/libexpr/primops.cc b/src/libexpr/primops.cc index a24695bcdbe..60f44ca62be 100644 --- a/src/libexpr/primops.cc +++ b/src/libexpr/primops.cc @@ -651,7 +651,7 @@ struct CompareValues // Note: we don't take the accessor into account // since it's not obvious how to compare them in a // reproducible way. - return strcmp(v1->payload.path.path, v2->payload.path.path) < 0; + return strcmp(v1->pathStr(), v2->pathStr()) < 0; case nList: // Lexicographic comparison for (size_t i = 0;; i++) { From bc6b52aff012592a9794df979c0617c85f46c2c3 Mon Sep 17 00:00:00 2001 From: Sergei Zimmerman Date: Thu, 12 Jun 2025 20:01:38 +0000 Subject: [PATCH 149/218] libexpr: Add and use `pathAccessor` getter --- src/libexpr/eval.cc | 4 ++-- src/libexpr/include/nix/expr/value.hh | 5 ++++- 2 files changed, 6 insertions(+), 3 deletions(-) diff --git a/src/libexpr/eval.cc b/src/libexpr/eval.cc index 66ac2e9fde9..ae422a3d4e7 100644 --- a/src/libexpr/eval.cc +++ b/src/libexpr/eval.cc @@ -2636,7 +2636,7 @@ void EvalState::assertEqValues(Value & v1, Value & v2, const PosIdx pos, std::st return; case nPath: - if (v1.payload.path.accessor != v2.payload.path.accessor) { + if (v1.pathAccessor() != v2.pathAccessor()) { error( "path '%s' is not equal to path '%s' because their accessors are different", ValuePrinter(*this, v1, errorPrintOptions), @@ -2810,7 +2810,7 @@ bool EvalState::eqValues(Value & v1, Value & v2, const PosIdx pos, std::string_v case nPath: return // FIXME: compare accessors by their fingerprint. - v1.payload.path.accessor == v2.payload.path.accessor + v1.pathAccessor() == v2.pathAccessor() && strcmp(v1.pathStr(), v2.pathStr()) == 0; case nNull: diff --git a/src/libexpr/include/nix/expr/value.hh b/src/libexpr/include/nix/expr/value.hh index fc9ce1b39ed..80bee59e971 100644 --- a/src/libexpr/include/nix/expr/value.hh +++ b/src/libexpr/include/nix/expr/value.hh @@ -444,7 +444,7 @@ public: { assert(internalType == tPath); return SourcePath( - ref(payload.path.accessor->shared_from_this()), + ref(pathAccessor()->shared_from_this()), CanonPath(CanonPath::unchecked_t(), pathStr())); } @@ -497,6 +497,9 @@ public: const char * pathStr() const { return payload.path.path; } + + SourceAccessor * pathAccessor() const + { return payload.path.accessor; } }; From 7b46eb9958e7681958020b68d9ea30a52be7cf09 Mon Sep 17 00:00:00 2001 From: Sergei Zimmerman Date: Thu, 12 Jun 2025 22:29:05 +0000 Subject: [PATCH 150/218] libexpr: Remove non-const overload of `listElems` This overload isn't actually necessary anywhere and doesn't make much sense. The pointers to `Value`s are themselves const, but the `Value`s are mutable. A non-const member function implies that the object itself can be modified but this doesn't make much sense considering the return type: `Value * const * `, which is a pointer to a constant array of pointers to mutable values. --- src/libexpr/include/nix/expr/value.hh | 5 ----- 1 file changed, 5 deletions(-) diff --git a/src/libexpr/include/nix/expr/value.hh b/src/libexpr/include/nix/expr/value.hh index 80bee59e971..fcc118c7e96 100644 --- a/src/libexpr/include/nix/expr/value.hh +++ b/src/libexpr/include/nix/expr/value.hh @@ -410,11 +410,6 @@ public: return internalType == tList1 || internalType == tList2 || internalType == tListN; } - Value * const * listElems() - { - return internalType == tList1 || internalType == tList2 ? payload.smallList : payload.bigList.elems; - } - std::span listItems() const { assert(isList()); From f8c1ac9515c03eedb79a7aa9e437d04428c4f736 Mon Sep 17 00:00:00 2001 From: John Ericson Date: Wed, 11 Jun 2025 15:15:00 -0400 Subject: [PATCH 151/218] Introduce top-level `structuredAttrs` field in JSON derivation format Makes the behavoral change of #13263 without the underlying refactor. Hopefully this clearly safe from a perf and GC perspective, and will make it easier to benchmark #13263. --- .../source/language/advanced-attributes.md | 24 +++----- .../source/protocols/json/derivation.md | 4 ++ doc/manual/source/store/derivation/index.md | 11 ++++ ...-attributes-structured-attrs-defaults.json | 12 +++- .../advanced-attributes-structured-attrs.json | 58 ++++++++++++++++++- ...-attributes-structured-attrs-defaults.json | 10 +++- .../advanced-attributes-structured-attrs.json | 56 +++++++++++++++++- src/libstore/derivations.cc | 17 +++++- tests/functional/structured-attrs.sh | 2 +- 9 files changed, 171 insertions(+), 23 deletions(-) diff --git a/doc/manual/source/language/advanced-attributes.md b/doc/manual/source/language/advanced-attributes.md index a939847e1aa..34c3b636b39 100644 --- a/doc/manual/source/language/advanced-attributes.md +++ b/doc/manual/source/language/advanced-attributes.md @@ -53,23 +53,13 @@ Derivations can declare some infrequently used optional attributes. - [`__structuredAttrs`]{#adv-attr-structuredAttrs}\ If the special attribute `__structuredAttrs` is set to `true`, the other derivation - attributes are serialised into a file in JSON format. The environment variable - `NIX_ATTRS_JSON_FILE` points to the exact location of that file both in a build - and a [`nix-shell`](../command-ref/nix-shell.md). This obviates the need for - [`passAsFile`](#adv-attr-passAsFile) since JSON files have no size restrictions, - unlike process environments. - - It also makes it possible to tweak derivation settings in a structured way; see - [`outputChecks`](#adv-attr-outputChecks) for example. - - As a convenience to Bash builders, - Nix writes a script that initialises shell variables - corresponding to all attributes that are representable in Bash. The - environment variable `NIX_ATTRS_SH_FILE` points to the exact - location of the script, both in a build and a - [`nix-shell`](../command-ref/nix-shell.md). This includes non-nested - (associative) arrays. For example, the attribute `hardening.format = true` - ends up as the Bash associative array element `${hardening[format]}`. + attributes are serialised into a file in JSON format. + + This obviates the need for [`passAsFile`](#adv-attr-passAsFile) since JSON files have no size restrictions, unlike process environments. + It also makes it possible to tweak derivation settings in a structured way; + see [`outputChecks`](#adv-attr-outputChecks) for example. + + See the [corresponding section in the derivation page](@docroot@/store/derivation/index.md#structured-attrs) for further details. > **Warning** > diff --git a/doc/manual/source/protocols/json/derivation.md b/doc/manual/source/protocols/json/derivation.md index 205b75fa507..04881776abc 100644 --- a/doc/manual/source/protocols/json/derivation.md +++ b/doc/manual/source/protocols/json/derivation.md @@ -91,3 +91,7 @@ is a JSON object with the following fields: * `env`: The environment passed to the `builder`. + +* `structuredAttrs`: + [Strucutured Attributes](@docroot@/store/derivation/index.md#structured-attrs), only defined if the derivation contains them. + Structured attributes are JSON, and thus embedded as-is. diff --git a/doc/manual/source/store/derivation/index.md b/doc/manual/source/store/derivation/index.md index 16ffc0923d9..147330df6ba 100644 --- a/doc/manual/source/store/derivation/index.md +++ b/doc/manual/source/store/derivation/index.md @@ -138,6 +138,17 @@ See [Wikipedia](https://en.wikipedia.org/wiki/Argv) for details. Environment variables which will be passed to the [builder](#builder) executable. +#### Structured Attributes {#structured-attrs} + +Nix also has special support for embedding JSON in the derivations. + +The environment variable `NIX_ATTRS_JSON_FILE` points to the exact location of that file both in a build and a [`nix-shell`](@docroot@/command-ref/nix-shell.md). + +As a convenience to Bash builders, Nix writes a script that initialises shell variables corresponding to all attributes that are representable in Bash. +The environment variable `NIX_ATTRS_SH_FILE` points to the exact location of the script, both in a build and a [`nix-shell`](@docroot@/command-ref/nix-shell.md). +This includes non-nested (associative) arrays. +For example, the attribute `hardening.format = true` ends up as the Bash associative array element `${hardening[format]}`. + ### Placeholders Placeholders are opaque values used within the [process creation fields] to [store objects] for which we don't yet know [store path]s. diff --git a/src/libstore-tests/data/derivation/ca/advanced-attributes-structured-attrs-defaults.json b/src/libstore-tests/data/derivation/ca/advanced-attributes-structured-attrs-defaults.json index 7d3c932b213..183148b29b3 100644 --- a/src/libstore-tests/data/derivation/ca/advanced-attributes-structured-attrs-defaults.json +++ b/src/libstore-tests/data/derivation/ca/advanced-attributes-structured-attrs-defaults.json @@ -5,7 +5,6 @@ ], "builder": "/bin/bash", "env": { - "__json": "{\"builder\":\"/bin/bash\",\"name\":\"advanced-attributes-structured-attrs-defaults\",\"outputHashAlgo\":\"sha256\",\"outputHashMode\":\"recursive\",\"outputs\":[\"out\",\"dev\"],\"system\":\"my-system\"}", "dev": "/02qcpld1y6xhs5gz9bchpxaw0xdhmsp5dv88lh25r2ss44kh8dxz", "out": "/1rz4g4znpzjwh1xymhjpm42vipw92pr73vdgl6xs1hycac8kf2n9" }, @@ -22,5 +21,16 @@ "method": "nar" } }, + "structuredAttrs": { + "builder": "/bin/bash", + "name": "advanced-attributes-structured-attrs-defaults", + "outputHashAlgo": "sha256", + "outputHashMode": "recursive", + "outputs": [ + "out", + "dev" + ], + "system": "my-system" + }, "system": "my-system" } diff --git a/src/libstore-tests/data/derivation/ca/advanced-attributes-structured-attrs.json b/src/libstore-tests/data/derivation/ca/advanced-attributes-structured-attrs.json index a421efea7ca..ec044d77877 100644 --- a/src/libstore-tests/data/derivation/ca/advanced-attributes-structured-attrs.json +++ b/src/libstore-tests/data/derivation/ca/advanced-attributes-structured-attrs.json @@ -5,7 +5,6 @@ ], "builder": "/bin/bash", "env": { - "__json": "{\"__darwinAllowLocalNetworking\":true,\"__impureHostDeps\":[\"/usr/bin/ditto\"],\"__noChroot\":true,\"__sandboxProfile\":\"sandcastle\",\"allowSubstitutes\":false,\"builder\":\"/bin/bash\",\"exportReferencesGraph\":{\"refs1\":[\"/164j69y6zir9z0339n8pjigg3rckinlr77bxsavzizdaaljb7nh9\"],\"refs2\":[\"/nix/store/qnml92yh97a6fbrs2m5qg5cqlc8vni58-bar.drv\"]},\"impureEnvVars\":[\"UNICORN\"],\"name\":\"advanced-attributes-structured-attrs\",\"outputChecks\":{\"bin\":{\"disallowedReferences\":[\"/0nyw57wm2iicnm9rglvjmbci3ikmcp823czdqdzdcgsnnwqps71g\"],\"disallowedRequisites\":[\"/07f301yqyz8c6wf6bbbavb2q39j4n8kmcly1s09xadyhgy6x2wr8\"]},\"dev\":{\"maxClosureSize\":5909,\"maxSize\":789},\"out\":{\"allowedReferences\":[\"/164j69y6zir9z0339n8pjigg3rckinlr77bxsavzizdaaljb7nh9\"],\"allowedRequisites\":[\"/0nr45p69vn6izw9446wsh9bng9nndhvn19kpsm4n96a5mycw0s4z\"]}},\"outputHashAlgo\":\"sha256\",\"outputHashMode\":\"recursive\",\"outputs\":[\"out\",\"bin\",\"dev\"],\"preferLocalBuild\":true,\"requiredSystemFeatures\":[\"rainbow\",\"uid-range\"],\"system\":\"my-system\"}", "bin": "/04f3da1kmbr67m3gzxikmsl4vjz5zf777sv6m14ahv22r65aac9m", "dev": "/02qcpld1y6xhs5gz9bchpxaw0xdhmsp5dv88lh25r2ss44kh8dxz", "out": "/1rz4g4znpzjwh1xymhjpm42vipw92pr73vdgl6xs1hycac8kf2n9" @@ -44,5 +43,62 @@ "method": "nar" } }, + "structuredAttrs": { + "__darwinAllowLocalNetworking": true, + "__impureHostDeps": [ + "/usr/bin/ditto" + ], + "__noChroot": true, + "__sandboxProfile": "sandcastle", + "allowSubstitutes": false, + "builder": "/bin/bash", + "exportReferencesGraph": { + "refs1": [ + "/164j69y6zir9z0339n8pjigg3rckinlr77bxsavzizdaaljb7nh9" + ], + "refs2": [ + "/nix/store/qnml92yh97a6fbrs2m5qg5cqlc8vni58-bar.drv" + ] + }, + "impureEnvVars": [ + "UNICORN" + ], + "name": "advanced-attributes-structured-attrs", + "outputChecks": { + "bin": { + "disallowedReferences": [ + "/0nyw57wm2iicnm9rglvjmbci3ikmcp823czdqdzdcgsnnwqps71g" + ], + "disallowedRequisites": [ + "/07f301yqyz8c6wf6bbbavb2q39j4n8kmcly1s09xadyhgy6x2wr8" + ] + }, + "dev": { + "maxClosureSize": 5909, + "maxSize": 789 + }, + "out": { + "allowedReferences": [ + "/164j69y6zir9z0339n8pjigg3rckinlr77bxsavzizdaaljb7nh9" + ], + "allowedRequisites": [ + "/0nr45p69vn6izw9446wsh9bng9nndhvn19kpsm4n96a5mycw0s4z" + ] + } + }, + "outputHashAlgo": "sha256", + "outputHashMode": "recursive", + "outputs": [ + "out", + "bin", + "dev" + ], + "preferLocalBuild": true, + "requiredSystemFeatures": [ + "rainbow", + "uid-range" + ], + "system": "my-system" + }, "system": "my-system" } diff --git a/src/libstore-tests/data/derivation/ia/advanced-attributes-structured-attrs-defaults.json b/src/libstore-tests/data/derivation/ia/advanced-attributes-structured-attrs-defaults.json index 473d006acb5..f5349e6c311 100644 --- a/src/libstore-tests/data/derivation/ia/advanced-attributes-structured-attrs-defaults.json +++ b/src/libstore-tests/data/derivation/ia/advanced-attributes-structured-attrs-defaults.json @@ -5,7 +5,6 @@ ], "builder": "/bin/bash", "env": { - "__json": "{\"builder\":\"/bin/bash\",\"name\":\"advanced-attributes-structured-attrs-defaults\",\"outputs\":[\"out\",\"dev\"],\"system\":\"my-system\"}", "dev": "/nix/store/8bazivnbipbyi569623skw5zm91z6kc2-advanced-attributes-structured-attrs-defaults-dev", "out": "/nix/store/f8f8nvnx32bxvyxyx2ff7akbvwhwd9dw-advanced-attributes-structured-attrs-defaults" }, @@ -20,5 +19,14 @@ "path": "/nix/store/f8f8nvnx32bxvyxyx2ff7akbvwhwd9dw-advanced-attributes-structured-attrs-defaults" } }, + "structuredAttrs": { + "builder": "/bin/bash", + "name": "advanced-attributes-structured-attrs-defaults", + "outputs": [ + "out", + "dev" + ], + "system": "my-system" + }, "system": "my-system" } diff --git a/src/libstore-tests/data/derivation/ia/advanced-attributes-structured-attrs.json b/src/libstore-tests/data/derivation/ia/advanced-attributes-structured-attrs.json index d68502d56be..b8d56646275 100644 --- a/src/libstore-tests/data/derivation/ia/advanced-attributes-structured-attrs.json +++ b/src/libstore-tests/data/derivation/ia/advanced-attributes-structured-attrs.json @@ -5,7 +5,6 @@ ], "builder": "/bin/bash", "env": { - "__json": "{\"__darwinAllowLocalNetworking\":true,\"__impureHostDeps\":[\"/usr/bin/ditto\"],\"__noChroot\":true,\"__sandboxProfile\":\"sandcastle\",\"allowSubstitutes\":false,\"builder\":\"/bin/bash\",\"exportReferencesGraph\":{\"refs1\":[\"/nix/store/p0hax2lzvjpfc2gwkk62xdglz0fcqfzn-foo\"],\"refs2\":[\"/nix/store/vj2i49jm2868j2fmqvxm70vlzmzvgv14-bar.drv\"]},\"impureEnvVars\":[\"UNICORN\"],\"name\":\"advanced-attributes-structured-attrs\",\"outputChecks\":{\"bin\":{\"disallowedReferences\":[\"/nix/store/r5cff30838majxk5mp3ip2diffi8vpaj-bar\"],\"disallowedRequisites\":[\"/nix/store/9b61w26b4avv870dw0ymb6rw4r1hzpws-bar-dev\"]},\"dev\":{\"maxClosureSize\":5909,\"maxSize\":789},\"out\":{\"allowedReferences\":[\"/nix/store/p0hax2lzvjpfc2gwkk62xdglz0fcqfzn-foo\"],\"allowedRequisites\":[\"/nix/store/z0rjzy29v9k5qa4nqpykrbzirj7sd43v-foo-dev\"]}},\"outputs\":[\"out\",\"bin\",\"dev\"],\"preferLocalBuild\":true,\"requiredSystemFeatures\":[\"rainbow\",\"uid-range\"],\"system\":\"my-system\"}", "bin": "/nix/store/33qms3h55wlaspzba3brlzlrm8m2239g-advanced-attributes-structured-attrs-bin", "dev": "/nix/store/wyfgwsdi8rs851wmy1xfzdxy7y5vrg5l-advanced-attributes-structured-attrs-dev", "out": "/nix/store/7cxy4zx1vqc885r4jl2l64pymqbdmhii-advanced-attributes-structured-attrs" @@ -41,5 +40,60 @@ "path": "/nix/store/7cxy4zx1vqc885r4jl2l64pymqbdmhii-advanced-attributes-structured-attrs" } }, + "structuredAttrs": { + "__darwinAllowLocalNetworking": true, + "__impureHostDeps": [ + "/usr/bin/ditto" + ], + "__noChroot": true, + "__sandboxProfile": "sandcastle", + "allowSubstitutes": false, + "builder": "/bin/bash", + "exportReferencesGraph": { + "refs1": [ + "/nix/store/p0hax2lzvjpfc2gwkk62xdglz0fcqfzn-foo" + ], + "refs2": [ + "/nix/store/vj2i49jm2868j2fmqvxm70vlzmzvgv14-bar.drv" + ] + }, + "impureEnvVars": [ + "UNICORN" + ], + "name": "advanced-attributes-structured-attrs", + "outputChecks": { + "bin": { + "disallowedReferences": [ + "/nix/store/r5cff30838majxk5mp3ip2diffi8vpaj-bar" + ], + "disallowedRequisites": [ + "/nix/store/9b61w26b4avv870dw0ymb6rw4r1hzpws-bar-dev" + ] + }, + "dev": { + "maxClosureSize": 5909, + "maxSize": 789 + }, + "out": { + "allowedReferences": [ + "/nix/store/p0hax2lzvjpfc2gwkk62xdglz0fcqfzn-foo" + ], + "allowedRequisites": [ + "/nix/store/z0rjzy29v9k5qa4nqpykrbzirj7sd43v-foo-dev" + ] + } + }, + "outputs": [ + "out", + "bin", + "dev" + ], + "preferLocalBuild": true, + "requiredSystemFeatures": [ + "rainbow", + "uid-range" + ], + "system": "my-system" + }, "system": "my-system" } diff --git a/src/libstore/derivations.cc b/src/libstore/derivations.cc index 7c13b4f63bd..0657a749901 100644 --- a/src/libstore/derivations.cc +++ b/src/libstore/derivations.cc @@ -1333,6 +1333,11 @@ nlohmann::json Derivation::toJSON(const StoreDirConfig & store) const res["args"] = args; res["env"] = env; + if (auto it = env.find("__json"); it != env.end()) { + res["env"].erase("__json"); + res["structuredAttrs"] = nlohmann::json::parse(it->second); + } + return res; } @@ -1396,7 +1401,17 @@ Derivation Derivation::fromJSON( res.platform = getString(valueAt(json, "system")); res.builder = getString(valueAt(json, "builder")); res.args = getStringList(valueAt(json, "args")); - res.env = getStringMap(valueAt(json, "env")); + + auto envJson = valueAt(json, "env"); + try { + res.env = getStringMap(envJson); + } catch (Error & e) { + e.addTrace({}, "while reading key 'env'"); + throw; + } + + if (auto structuredAttrs = get(json, "structuredAttrs")) + res.env.insert_or_assign("__json", structuredAttrs->dump()); return res; } diff --git a/tests/functional/structured-attrs.sh b/tests/functional/structured-attrs.sh index 465676b410f..2bd9b4aaf1b 100755 --- a/tests/functional/structured-attrs.sh +++ b/tests/functional/structured-attrs.sh @@ -50,4 +50,4 @@ expectStderr 0 nix-instantiate --expr "$hackyExpr" --eval --strict | grepQuiet " # Check it works with the expected structured attrs hacky=$(nix-instantiate --expr "$hackyExpr") -nix derivation show "$hacky" | jq --exit-status '."'"$hacky"'".env.__json | fromjson | . == {"a": 1}' +nix derivation show "$hacky" | jq --exit-status '."'"$hacky"'".structuredAttrs | . == {"a": 1}' From 699db04df3e755b9c3411567ca332709c39ba07d Mon Sep 17 00:00:00 2001 From: jayeshv Date: Fri, 13 Jun 2025 12:28:27 +0200 Subject: [PATCH 152/218] Fix a minor typo --- doc/manual/source/installation/installing-binary.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/manual/source/installation/installing-binary.md b/doc/manual/source/installation/installing-binary.md index 6a1a5ddcaff..21c15637437 100644 --- a/doc/manual/source/installation/installing-binary.md +++ b/doc/manual/source/installation/installing-binary.md @@ -25,7 +25,7 @@ This performs the default type of installation for your platform: We recommend the multi-user installation if it supports your platform and you can authenticate with `sudo`. -The installer can configured with various command line arguments and environment variables. +The installer can be configured with various command line arguments and environment variables. To show available command line flags: ```console From e27a06278376a3721d06a6cc88ae711bd0937406 Mon Sep 17 00:00:00 2001 From: Pol Dellaiera Date: Sat, 14 Jun 2025 10:37:36 +0200 Subject: [PATCH 153/218] docker: remove last use of `pkgs.` Follow-up of https://github.com/NixOS/nix/pull/13354 --- docker.nix | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docker.nix b/docker.nix index 598d1a12560..14648f0d125 100644 --- a/docker.nix +++ b/docker.nix @@ -2,6 +2,7 @@ # Core dependencies pkgs, lib, + dockerTools, runCommand, buildPackages, # Image configuration @@ -325,7 +326,7 @@ let ); in -pkgs.dockerTools.buildLayeredImageWithNixDb { +dockerTools.buildLayeredImageWithNixDb { inherit name From b2596a76158e20b07d9edac48fcb56e7cc8ec0b7 Mon Sep 17 00:00:00 2001 From: Sergei Zimmerman Date: Sun, 15 Jun 2025 16:51:38 +0000 Subject: [PATCH 154/218] libutil: Add custom PeekSort implementation Unlike std::sort and std::stable_sort, this implementation does not lead to out-of-bounds memory reads or other undefined behavior when the predicate is not strict weak ordering. This makes it possible to use this function in libexpr for builtins.sort, where an incorrectly implemented comparator in the user nix code currently can crash and burn the evaluator by invoking C++ UB. --- src/libutil-tests/meson.build | 1 + src/libutil-tests/sort.cc | 274 +++++++++++++++++++++ src/libutil/include/nix/util/meson.build | 1 + src/libutil/include/nix/util/sort.hh | 299 +++++++++++++++++++++++ 4 files changed, 575 insertions(+) create mode 100644 src/libutil-tests/sort.cc create mode 100644 src/libutil/include/nix/util/sort.hh diff --git a/src/libutil-tests/meson.build b/src/libutil-tests/meson.build index f2552550d3b..b3776e0940a 100644 --- a/src/libutil-tests/meson.build +++ b/src/libutil-tests/meson.build @@ -65,6 +65,7 @@ sources = files( 'position.cc', 'processes.cc', 'references.cc', + 'sort.cc', 'spawn.cc', 'strings.cc', 'suggestions.cc', diff --git a/src/libutil-tests/sort.cc b/src/libutil-tests/sort.cc new file mode 100644 index 00000000000..8eee961c8cd --- /dev/null +++ b/src/libutil-tests/sort.cc @@ -0,0 +1,274 @@ +#include +#include +#include "nix/util/sort.hh" + +#include +#include +#include +#include + +namespace nix { + +struct MonotonicSubranges : public ::testing::Test +{ + std::vector empty_; + std::vector basic_ = {1, 0, -1, -100, 10, 10, 20, 40, 5, 5, 20, 10, 10, 1, -5}; +}; + +TEST_F(MonotonicSubranges, empty) +{ + ASSERT_EQ(weaklyIncreasingPrefix(empty_.begin(), empty_.end()), empty_.begin()); + ASSERT_EQ(weaklyIncreasingSuffix(empty_.begin(), empty_.end()), empty_.begin()); + ASSERT_EQ(strictlyDecreasingPrefix(empty_.begin(), empty_.end()), empty_.begin()); + ASSERT_EQ(strictlyDecreasingSuffix(empty_.begin(), empty_.end()), empty_.begin()); +} + +TEST_F(MonotonicSubranges, basic) +{ + ASSERT_EQ(strictlyDecreasingPrefix(basic_.begin(), basic_.end()), basic_.begin() + 4); + ASSERT_EQ(strictlyDecreasingSuffix(basic_.begin(), basic_.end()), basic_.begin() + 12); + std::reverse(basic_.begin(), basic_.end()); + ASSERT_EQ(weaklyIncreasingPrefix(basic_.begin(), basic_.end()), basic_.begin() + 5); + ASSERT_EQ(weaklyIncreasingSuffix(basic_.begin(), basic_.end()), basic_.begin() + 11); +} + +template +class SortTestPermutations : public ::testing::Test +{ + std::vector initialData = {std::numeric_limits::max(), std::numeric_limits::min(), 0, 0, 42, 126, 36}; + std::vector vectorData; + std::list listData; + +public: + std::vector scratchVector; + std::list scratchList; + std::vector empty; + + void SetUp() override + { + vectorData = initialData; + std::sort(vectorData.begin(), vectorData.end()); + listData = std::list(vectorData.begin(), vectorData.end()); + } + + bool nextPermutation() + { + std::next_permutation(vectorData.begin(), vectorData.end()); + std::next_permutation(listData.begin(), listData.end()); + scratchList = listData; + scratchVector = vectorData; + return vectorData == initialData; + } +}; + +using SortPermutationsTypes = ::testing::Types; + +TYPED_TEST_SUITE(SortTestPermutations, SortPermutationsTypes); + +TYPED_TEST(SortTestPermutations, insertionsort) +{ + while (!this->nextPermutation()) { + auto & list = this->scratchList; + insertionsort(list.begin(), list.end()); + ASSERT_TRUE(std::is_sorted(list.begin(), list.end())); + auto & vector = this->scratchVector; + insertionsort(vector.begin(), vector.end()); + ASSERT_TRUE(std::is_sorted(vector.begin(), vector.end())); + } +} + +TYPED_TEST(SortTestPermutations, peeksort) +{ + while (!this->nextPermutation()) { + auto & vector = this->scratchVector; + peeksort(vector.begin(), vector.end()); + ASSERT_TRUE(std::is_sorted(vector.begin(), vector.end())); + } +} + +TEST(InsertionSort, empty) +{ + std::vector empty; + insertionsort(empty.begin(), empty.end()); +} + +struct RandomPeekSort : public ::testing::TestWithParam< + std::tuple> +{ + using ValueType = int; + std::vector data_; + std::mt19937 urng_; + std::uniform_int_distribution distribution_; + + void SetUp() override + { + auto [maxSize, min, max, iterations] = GetParam(); + urng_ = std::mt19937(GTEST_FLAG_GET(random_seed)); + distribution_ = std::uniform_int_distribution(min, max); + } + + auto regenerate() + { + auto [maxSize, min, max, iterations] = GetParam(); + std::size_t dataSize = std::uniform_int_distribution(0, maxSize)(urng_); + data_.resize(dataSize); + std::generate(data_.begin(), data_.end(), [&]() { return distribution_(urng_); }); + } +}; + +TEST_P(RandomPeekSort, defaultComparator) +{ + auto [maxSize, min, max, iterations] = GetParam(); + + for (std::size_t i = 0; i < iterations; ++i) { + regenerate(); + peeksort(data_.begin(), data_.end()); + ASSERT_TRUE(std::is_sorted(data_.begin(), data_.end())); + /* Sorting is idempotent */ + peeksort(data_.begin(), data_.end()); + ASSERT_TRUE(std::is_sorted(data_.begin(), data_.end())); + } +} + +TEST_P(RandomPeekSort, greater) +{ + auto [maxSize, min, max, iterations] = GetParam(); + + for (std::size_t i = 0; i < iterations; ++i) { + regenerate(); + peeksort(data_.begin(), data_.end(), std::greater{}); + ASSERT_TRUE(std::is_sorted(data_.begin(), data_.end(), std::greater{})); + /* Sorting is idempotent */ + peeksort(data_.begin(), data_.end(), std::greater{}); + ASSERT_TRUE(std::is_sorted(data_.begin(), data_.end(), std::greater{})); + } +} + +TEST_P(RandomPeekSort, brokenComparator) +{ + auto [maxSize, min, max, iterations] = GetParam(); + + /* This is a pretty nice way of modeling a worst-case scenario for a broken comparator. + If the sorting algorithm doesn't break in such case, then surely all deterministic + predicates won't break it. */ + auto comp = [&]([[maybe_unused]] const auto & lhs, [[maybe_unused]] const auto & rhs) -> bool { + return std::uniform_int_distribution(0, 1)(urng_); + }; + + for (std::size_t i = 0; i < iterations; ++i) { + regenerate(); + auto originalData = data_; + peeksort(data_.begin(), data_.end(), comp); + /* Check that the output is just a reordering of the input. This is the + contract of the implementation in regard to comparators that don't + define a strict weak order. */ + std::sort(data_.begin(), data_.end()); + std::sort(originalData.begin(), originalData.end()); + ASSERT_EQ(originalData, data_); + } +} + +TEST_P(RandomPeekSort, stability) +{ + auto [maxSize, min, max, iterations] = GetParam(); + + for (std::size_t i = 0; i < iterations; ++i) { + regenerate(); + std::vector> pairs; + + /* Assign sequential ids to objects. After the sort ids for equivalent + elements should be in ascending order. */ + std::transform( + data_.begin(), data_.end(), std::back_inserter(pairs), [id = std::size_t{0}](auto && val) mutable { + return std::pair{val, ++id}; + }); + + auto comp = [&]([[maybe_unused]] const auto & lhs, [[maybe_unused]] const auto & rhs) -> bool { + return lhs.first > rhs.first; + }; + + peeksort(pairs.begin(), pairs.end(), comp); + ASSERT_TRUE(std::is_sorted(pairs.begin(), pairs.end(), comp)); + + for (auto begin = pairs.begin(), end = pairs.end(); begin < end; ++begin) { + auto key = begin->first; + auto innerEnd = std::find_if_not(begin, end, [key](const auto & lhs) { return lhs.first == key; }); + ASSERT_TRUE(std::is_sorted(begin, innerEnd, [](const auto & lhs, const auto & rhs) { + return lhs.second < rhs.second; + })); + begin = innerEnd; + } + } +} + +using RandomPeekSortParamType = RandomPeekSort::ParamType; + +INSTANTIATE_TEST_SUITE_P( + PeekSort, + RandomPeekSort, + ::testing::Values( + RandomPeekSortParamType{128, std::numeric_limits::min(), std::numeric_limits::max(), 1024}, + RandomPeekSortParamType{7753, -32, 32, 128}, + RandomPeekSortParamType{11719, std::numeric_limits::min(), std::numeric_limits::max(), 64}, + RandomPeekSortParamType{4063, 0, 32, 256}, + RandomPeekSortParamType{771, -8, 8, 2048}, + RandomPeekSortParamType{433, 0, 1, 2048}, + RandomPeekSortParamType{0, 0, 0, 1}, /* empty case */ + RandomPeekSortParamType{ + 1, std::numeric_limits::min(), std::numeric_limits::max(), 1}, /* single element */ + RandomPeekSortParamType{ + 2, std::numeric_limits::min(), std::numeric_limits::max(), 2}, /* two elements */ + RandomPeekSortParamType{55425, std::numeric_limits::min(), std::numeric_limits::max(), 128})); + +template +struct SortProperty : public ::testing::Test +{}; + +using SortPropertyTypes = ::testing::Types; +TYPED_TEST_SUITE(SortProperty, SortPropertyTypes); + +RC_GTEST_TYPED_FIXTURE_PROP(SortProperty, peeksortSorted, (std::vector vec)) +{ + peeksort(vec.begin(), vec.end()); + RC_ASSERT(std::is_sorted(vec.begin(), vec.end())); +} + +RC_GTEST_TYPED_FIXTURE_PROP(SortProperty, peeksortSortedGreater, (std::vector vec)) +{ + auto comp = std::greater(); + peeksort(vec.begin(), vec.end(), comp); + RC_ASSERT(std::is_sorted(vec.begin(), vec.end(), comp)); +} + +RC_GTEST_TYPED_FIXTURE_PROP(SortProperty, insertionsortSorted, (std::vector vec)) +{ + insertionsort(vec.begin(), vec.end()); + RC_ASSERT(std::is_sorted(vec.begin(), vec.end())); +} + +RC_GTEST_PROP(SortProperty, peeksortStability, (std::vector> vec)) +{ + auto comp = [](auto lhs, auto rhs) { return lhs.first < rhs.first; }; + auto copy = vec; + std::stable_sort(copy.begin(), copy.end(), comp); + peeksort(vec.begin(), vec.end(), comp); + RC_ASSERT(copy == vec); +} + +RC_GTEST_TYPED_FIXTURE_PROP(SortProperty, peeksortSortedLinearComparisonComplexity, (std::vector vec)) +{ + peeksort(vec.begin(), vec.end()); + RC_ASSERT(std::is_sorted(vec.begin(), vec.end())); + std::size_t comparisonCount = 0; + auto countingComp = [&](auto lhs, auto rhs) { + ++comparisonCount; + return lhs < rhs; + }; + + peeksort(vec.begin(), vec.end(), countingComp); + + /* In the sorted case comparison complexify should be linear. */ + RC_ASSERT(comparisonCount <= vec.size()); +} + +} // namespace nix diff --git a/src/libutil/include/nix/util/meson.build b/src/libutil/include/nix/util/meson.build index 6c95a6cedaa..22438c1d075 100644 --- a/src/libutil/include/nix/util/meson.build +++ b/src/libutil/include/nix/util/meson.build @@ -59,6 +59,7 @@ headers = files( 'signals.hh', 'signature/local-keys.hh', 'signature/signer.hh', + 'sort.hh', 'source-accessor.hh', 'source-path.hh', 'split.hh', diff --git a/src/libutil/include/nix/util/sort.hh b/src/libutil/include/nix/util/sort.hh new file mode 100644 index 00000000000..0affdf3ce97 --- /dev/null +++ b/src/libutil/include/nix/util/sort.hh @@ -0,0 +1,299 @@ +#pragma once + +#include +#include +#include +#include +#include +#include + +/** + * @file + * + * In-house implementation of sorting algorithms. Used for cases when several properties + * need to be upheld regardless of the stdlib implementation of std::sort or + * std::stable_sort. + * + * PeekSort implementation is adapted from reference implementation + * https://github.com/sebawild/powersort licensed under the MIT License. + * + */ + +/* PeekSort attribution: + * + * MIT License + * + * Copyright (c) 2022 Sebastian Wild + * + * Permission is hereby granted, free of charge, to any person obtaining a copy + * of this software and associated documentation files (the "Software"), to deal + * in the Software without restriction, including without limitation the rights + * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the Software is + * furnished to do so, subject to the following conditions: + * + * The above copyright notice and this permission notice shall be included in all + * copies or substantial portions of the Software. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR + * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, + * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE + * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER + * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, + * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE + * SOFTWARE. + */ + +namespace nix { + +/** + * Merge sorted runs [begin, middle) with [middle, end) in-place [begin, end). + * Uses a temporary working buffer by first copying [begin, end) to it. + * + * @param begin Start of the first subrange to be sorted. + * @param middle End of the first sorted subrange and the start of the second. + * @param end End of the second sorted subrange. + * @param workingBegin Start of the working buffer. + * @param comp Comparator implementing an operator()(const ValueType& lhs, const ValueType& rhs). + * + * @pre workingBegin buffer must have at least std::distance(begin, end) elements. + * + * @note We can't use std::inplace_merge or std::merge, because their behavior + * is undefined if the comparator is not strict weak ordering. + */ +template< + std::forward_iterator Iter, + std::random_access_iterator BufIter, + typename Comparator = std::less>> +void mergeSortedRunsInPlace(Iter begin, Iter middle, Iter end, BufIter workingBegin, Comparator comp = {}) +{ + const BufIter workingMiddle = std::move(begin, middle, workingBegin); + const BufIter workingEnd = std::move(middle, end, workingMiddle); + + Iter output = begin; + BufIter workingLeft = workingBegin; + BufIter workingRight = workingMiddle; + + while (workingLeft != workingMiddle && workingRight != workingEnd) { + /* Note the inversion here !comp(...., ....). This is required for the merge to be stable. + If a == b where a if from the left part and b is the the right, then we have to pick + a. */ + *output++ = !comp(*workingRight, *workingLeft) ? std::move(*workingLeft++) : std::move(*workingRight++); + } + + std::move(workingLeft, workingMiddle, output); + std::move(workingRight, workingEnd, output); +} + +/** + * Simple insertion sort. + * + * Does not require that the std::iter_value_t is copyable. + * + * @param begin Start of the range to sort. + * @param end End of the range to sort. + * @comp Comparator the defines the ordering. Order of elements if the comp is not strict weak ordering + * is not specified. + * @throws Nothing. + * + * Note on exception safety: this function provides weak exception safety + * guarantees. To elaborate: if the comparator throws or move assignment + * throws (value type is not nothrow_move_assignable) then the range is left in + * a consistent, but unspecified state. + * + * @note This can't be implemented in terms of binary search if the strict weak ordering + * needs to be handled in a well-defined but unspecified manner. + */ +template>> +void insertionsort(Iter begin, Iter end, Comparator comp = {}) +{ + if (begin == end) + return; + for (Iter current = std::next(begin); current != end; ++current) { + for (Iter insertionPoint = current; + insertionPoint != begin && comp(*insertionPoint, *std::prev(insertionPoint)); + --insertionPoint) { + std::swap(*insertionPoint, *std::prev(insertionPoint)); + } + } +} + +/** + * Find maximal i <= end such that [begin, i) is strictly decreasing according + * to the specified comparator. + */ +template>> +Iter strictlyDecreasingPrefix(Iter begin, Iter end, Comparator && comp = {}) +{ + if (begin == end) + return begin; + while (std::next(begin) != end && /* *std::next(begin) < begin */ + comp(*std::next(begin), *begin)) + ++begin; + return std::next(begin); +} + +/** + * Find minimal i >= start such that [i, end) is strictly decreasing according + * to the specified comparator. + */ +template>> +Iter strictlyDecreasingSuffix(Iter begin, Iter end, Comparator && comp = {}) +{ + if (begin == end) + return end; + while (std::prev(end) > begin && /* *std::prev(end) < *std::prev(end, 2) */ + comp(*std::prev(end), *std::prev(end, 2))) + --end; + return std::prev(end); +} + +/** + * Find maximal i <= end such that [begin, i) is weakly increasing according + * to the specified comparator. + */ +template>> +Iter weaklyIncreasingPrefix(Iter begin, Iter end, Comparator && comp = {}) +{ + return strictlyDecreasingPrefix(begin, end, std::not_fn(std::forward(comp))); +} + +/** + * Find minimal i >= start such that [i, end) is weakly increasing according + * to the specified comparator. + */ +template>> +Iter weaklyIncreasingSuffix(Iter begin, Iter end, Comparator && comp = {}) +{ + return strictlyDecreasingSuffix(begin, end, std::not_fn(std::forward(comp))); +} + +/** + * Peeksort stable sorting algorithm. Sorts elements in-place. + * Allocates additional memory as needed. + * + * @details + * PeekSort is a stable, near-optimal natural mergesort. Most importantly, like any + * other mergesort it upholds the "Ord safety" property. Meaning that even for + * comparator predicates that don't satisfy strict weak ordering it can't result + * in infinite loops/out of bounds memory accesses or other undefined behavior. + * + * As a quick reminder, strict weak ordering relation operator< must satisfy + * the following properties. Keep in mind that in C++ an equvalence relation + * is specified in terms of operator< like so: a ~ b iff !(a < b) && !(b < a). + * + * 1. a < a === false - relation is irreflexive + * 2. a < b, b < c => a < c - transitivity + * 3. a ~ b, a ~ b, b ~ c => a ~ c, transitivity of equivalence + * + * @see https://www.wild-inter.net/publications/munro-wild-2018 + * @see https://github.com/Voultapher/sort-research-rs/blob/main/writeup/sort_safety/text.md#property-analysis + * + * The order of elements when comp is not strict weak ordering is not specified, but + * is not undefined. The output is always some permutation of the input, regardless + * of the comparator provided. + * Relying on ordering in such cases is erroneous, but this implementation + * will happily accept broken comparators and will not crash. + * + * @param begin Start of the range to be sorted. + * @param end End of the range to be sorted. + * @comp comp Comparator implementing an operator()(const ValueType& lhs, const ValueType& rhs). + * + * @throws std::bad_alloc if the temporary buffer can't be allocated. + * + * @return Nothing. + * + * Note on exception safety: this function provides weak exception safety + * guarantees. To elaborate: if the comparator throws or move assignment + * throws (value type is not nothrow_move_assignable) then the range is left in + * a consistent, but unspecified state. + * + */ +template>> +/* ValueType must be default constructible to create the temporary buffer */ + requires std::is_default_constructible_v> +void peeksort(Iter begin, Iter end, Comparator comp = {}) +{ + auto length = std::distance(begin, end); + + /* Special-case very simple inputs. This is identical to how libc++ does it. */ + switch (length) { + case 0: + [[fallthrough]]; + case 1: + return; + case 2: + if (comp(*--end, *begin)) /* [a, b], b < a */ + std::swap(*begin, *end); + return; + } + + using ValueType = std::iter_value_t; + auto workingBuffer = std::vector(length); + + /* + * sorts [begin, end), assuming that [begin, leftRunEnd) and + * [rightRunBegin, end) are sorted. + * Modified implementation from: + * https://github.com/sebawild/powersort/blob/1d078b6be9023e134c4f8f6de88e2406dc681e89/src/sorts/peeksort.h + */ + auto peeksortImpl = [&workingBuffer, + &comp](auto & peeksortImpl, Iter begin, Iter end, Iter leftRunEnd, Iter rightRunBegin) { + if (leftRunEnd == end || rightRunBegin == begin) + return; + + /* Dispatch to simpler insertion sort implementation for smaller cases + Cut-off limit is the same as in libstdc++ + https://github.com/gcc-mirror/gcc/blob/d9375e490072d1aae73a93949aa158fcd2a27018/libstdc%2B%2B-v3/include/bits/stl_algo.h#L4977 + */ + static constexpr std::size_t insertionsortThreshold = 16; + size_t length = std::distance(begin, end); + if (length <= insertionsortThreshold) + return insertionsort(begin, end, comp); + + Iter middle = std::next(begin, (length / 2)); /* Middle split between m and m - 1 */ + + if (middle <= leftRunEnd) { + /* |XXXXXXXX|XX X| */ + peeksortImpl(peeksortImpl, leftRunEnd, end, std::next(leftRunEnd), rightRunBegin); + mergeSortedRunsInPlace(begin, leftRunEnd, end, workingBuffer.begin(), comp); + return; + } else if (middle >= rightRunBegin) { + /* |XX X|XXXXXXXX| */ + peeksortImpl(peeksortImpl, begin, rightRunBegin, leftRunEnd, std::prev(rightRunBegin)); + mergeSortedRunsInPlace(begin, rightRunBegin, end, workingBuffer.begin(), comp); + return; + } + + /* Find middle run, i.e., run containing m - 1 */ + Iter i, j; + + if (!comp(*middle, *std::prev(middle)) /* *std::prev(middle) <= *middle */) { + i = weaklyIncreasingSuffix(leftRunEnd, middle, comp); + j = weaklyIncreasingPrefix(std::prev(middle), rightRunBegin, comp); + } else { + i = strictlyDecreasingSuffix(leftRunEnd, middle, comp); + j = strictlyDecreasingPrefix(std::prev(middle), rightRunBegin, comp); + std::reverse(i, j); + } + + if (i == begin && j == end) + return; /* single run */ + + if (middle - i < j - middle) { + /* |XX x|xxxx X| */ + peeksortImpl(peeksortImpl, begin, i, leftRunEnd, std::prev(i)); + peeksortImpl(peeksortImpl, i, end, j, rightRunBegin); + mergeSortedRunsInPlace(begin, i, end, workingBuffer.begin(), comp); + } else { + /* |XX xxx|x X| */ + peeksortImpl(peeksortImpl, begin, j, leftRunEnd, i); + peeksortImpl(peeksortImpl, j, end, std::next(j), rightRunBegin); + mergeSortedRunsInPlace(begin, j, end, workingBuffer.begin(), comp); + } + }; + + peeksortImpl(peeksortImpl, begin, end, /*leftRunEnd=*/begin, /*rightRunBegin=*/end); +} + +} From 351d898c43202ca9c3f94daf7d727b3b7bbd3daa Mon Sep 17 00:00:00 2001 From: Sergei Zimmerman Date: Sun, 15 Jun 2025 16:51:40 +0000 Subject: [PATCH 155/218] libexpr: Switch builtins.sort primop to use peeksort This prevents C++ level undefined behavior from affecting the evaluator. Stdlib implementation details should not affect eval, regardless of the build platform. Even erroneous usage of `builtins.sort` should not make it possible to crash the evaluator or produce results that depend on the host platform. --- src/libexpr/primops.cc | 13 +++++++++---- 1 file changed, 9 insertions(+), 4 deletions(-) diff --git a/src/libexpr/primops.cc b/src/libexpr/primops.cc index 60f44ca62be..75d7465dd5b 100644 --- a/src/libexpr/primops.cc +++ b/src/libexpr/primops.cc @@ -14,6 +14,7 @@ #include "nix/expr/value-to-xml.hh" #include "nix/expr/primops.hh" #include "nix/fetchers/fetch-to-store.hh" +#include "nix/util/sort.hh" #include #include @@ -3695,10 +3696,14 @@ static void prim_sort(EvalState & state, const PosIdx pos, Value * * args, Value return state.forceBool(vBool, pos, "while evaluating the return value of the sorting function passed to builtins.sort"); }; - /* FIXME: std::sort can segfault if the comparator is not a strict - weak ordering. What to do? std::stable_sort() seems more - resilient, but no guarantees... */ - std::stable_sort(list.begin(), list.end(), comparator); + /* NOTE: Using custom implementation because std::sort and std::stable_sort + are not resilient to comparators that violate strict weak ordering. Diagnosing + incorrect implementations is a O(n^3) problem, so doing the checks is much more + expensive that doing the sorting. For this reason we choose to use sorting algorithms + that are can't be broken by invalid comprators. peeksort (mergesort) + doesn't misbehave when any of the strict weak order properties is + violated - output is always a reordering of the input. */ + peeksort(list.begin(), list.end(), comparator); v.mkList(list); } From ddcfc81ff1fe01e4ab74cec80709cac8f77361d5 Mon Sep 17 00:00:00 2001 From: Sergei Zimmerman Date: Sun, 15 Jun 2025 16:51:42 +0000 Subject: [PATCH 156/218] libexpr: Document requirements for comparator passed to builtins.sort --- src/libexpr/primops.cc | 26 ++++++++++++++++++++++++++ 1 file changed, 26 insertions(+) diff --git a/src/libexpr/primops.cc b/src/libexpr/primops.cc index 75d7465dd5b..ba568e38d53 100644 --- a/src/libexpr/primops.cc +++ b/src/libexpr/primops.cc @@ -3725,6 +3725,32 @@ static RegisterPrimOp primop_sort({ This is a stable sort: it preserves the relative order of elements deemed equal by the comparator. + + *comparator* must impose a strict weak ordering on the set of values + in the *list*. This means that for any elements *a*, *b* and *c* from the + *list*, *comparator* must satisfy the following relations: + + 1. Transitivity + + ```nix + comparator a b && comparator b c -> comparator a c + ``` + + 1. Irreflexivity + + ```nix + comparator a a == false + ``` + + 1. Transitivity of equivalence + + ```nix + let equiv = a: b: (!comparator a b && !comparator b a); in + equiv a b && equiv b c -> equiv a c + ``` + + If the *comparator* violates any of these properties, then `builtins.sort` + reorders elements in an unspecified manner. )", .fun = prim_sort, }); From f9170a84f6eb6a162c800c39c17d3b34eff87dda Mon Sep 17 00:00:00 2001 From: Sergei Zimmerman Date: Sun, 15 Jun 2025 16:51:44 +0000 Subject: [PATCH 157/218] tests/functional/lang: Add sort stability test for lists langer than 16 elements libstdc++'s std::stable_sort and new builtins.sort implementation special-case ranges with length less than or equal to 16 and delegate to insertionsort. Having a larger e2e test would allow catching sort stability issues at functional level as well. --- tests/functional/lang/eval-okay-sort.exp | 2 +- tests/functional/lang/eval-okay-sort.nix | 74 ++++++++++++++++++++++++ 2 files changed, 75 insertions(+), 1 deletion(-) diff --git a/tests/functional/lang/eval-okay-sort.exp b/tests/functional/lang/eval-okay-sort.exp index 899119e20e3..fcb3b2224f6 100644 --- a/tests/functional/lang/eval-okay-sort.exp +++ b/tests/functional/lang/eval-okay-sort.exp @@ -1 +1 @@ -[ [ 42 77 147 249 483 526 ] [ 526 483 249 147 77 42 ] [ "bar" "fnord" "foo" "xyzzy" ] [ { key = 1; value = "foo"; } { key = 1; value = "fnord"; } { key = 2; value = "bar"; } ] [ [ ] [ ] [ 1 ] [ 1 4 ] [ 1 5 ] [ 1 6 ] [ 2 ] [ 2 3 ] [ 3 ] [ 3 ] ] ] +[ [ 42 77 147 249 483 526 ] [ 526 483 249 147 77 42 ] [ "bar" "fnord" "foo" "xyzzy" ] [ { key = 1; value = "foo"; } { key = 1; value = "fnord"; } { key = 2; value = "bar"; } ] [ { key = 1; value = "foo"; } { key = 1; value = "foo2"; } { key = 1; value = "foo3"; } { key = 1; value = "foo4"; } { key = 1; value = "foo5"; } { key = 1; value = "foo6"; } { key = 1; value = "foo7"; } { key = 1; value = "foo8"; } { key = 2; value = "bar"; } { key = 2; value = "bar2"; } { key = 2; value = "bar3"; } { key = 2; value = "bar4"; } { key = 2; value = "bar5"; } { key = 3; value = "baz"; } { key = 3; value = "baz2"; } { key = 3; value = "baz3"; } { key = 3; value = "baz4"; } { key = 4; value = "biz1"; } ] [ [ ] [ ] [ 1 ] [ 1 4 ] [ 1 5 ] [ 1 6 ] [ 2 ] [ 2 3 ] [ 3 ] [ 3 ] ] ] diff --git a/tests/functional/lang/eval-okay-sort.nix b/tests/functional/lang/eval-okay-sort.nix index 412bda4a09f..7a3b7f71b19 100644 --- a/tests/functional/lang/eval-okay-sort.nix +++ b/tests/functional/lang/eval-okay-sort.nix @@ -37,6 +37,80 @@ with builtins; value = "fnord"; } ]) + (sort (x: y: x.key < y.key) [ + { + key = 1; + value = "foo"; + } + { + key = 2; + value = "bar"; + } + { + key = 1; + value = "foo2"; + } + { + key = 2; + value = "bar2"; + } + { + key = 2; + value = "bar3"; + } + { + key = 2; + value = "bar4"; + } + { + key = 1; + value = "foo3"; + } + { + key = 3; + value = "baz"; + } + { + key = 3; + value = "baz2"; + } + { + key = 1; + value = "foo4"; + } + { + key = 3; + value = "baz3"; + } + { + key = 1; + value = "foo5"; + } + { + key = 1; + value = "foo6"; + } + { + key = 2; + value = "bar5"; + } + { + key = 3; + value = "baz4"; + } + { + key = 1; + value = "foo7"; + } + { + key = 4; + value = "biz1"; + } + { + key = 1; + value = "foo8"; + } + ]) (sort lessThan [ [ 1 From c1aaa970c7f444de14a7c03885f4f0b5a76eb980 Mon Sep 17 00:00:00 2001 From: Philipp Otterbein Date: Sun, 15 Jun 2025 00:32:36 +0200 Subject: [PATCH 158/218] libexpr: further removal of std::string copies --- src/libexpr/primops.cc | 22 +++++++++++----------- 1 file changed, 11 insertions(+), 11 deletions(-) diff --git a/src/libexpr/primops.cc b/src/libexpr/primops.cc index 60f44ca62be..9024f84d460 100644 --- a/src/libexpr/primops.cc +++ b/src/libexpr/primops.cc @@ -1179,7 +1179,7 @@ static void prim_second(EvalState & state, const PosIdx pos, Value * * args, Val static void derivationStrictInternal( EvalState & state, - const std::string & name, + std::string_view name, const Bindings * attrs, Value & v); @@ -1199,7 +1199,7 @@ static void prim_derivationStrict(EvalState & state, const PosIdx pos, Value * * /* Figure out the name first (for stack backtraces). */ auto nameAttr = state.getAttr(state.sName, attrs, "in the attrset passed as argument to builtins.derivationStrict"); - std::string drvName; + std::string_view drvName; try { drvName = state.forceStringNoCtx(*nameAttr->value, pos, "while evaluating the `name` attribute passed to builtins.derivationStrict"); } catch (Error & e) { @@ -1258,7 +1258,7 @@ static void checkDerivationName(EvalState & state, std::string_view drvName) static void derivationStrictInternal( EvalState & state, - const std::string & drvName, + std::string_view drvName, const Bindings * attrs, Value & v) { @@ -1898,7 +1898,7 @@ static void prim_findFile(EvalState & state, const PosIdx pos, Value * * args, V try { auto rewrites = state.realiseContext(context); - path = rewriteStrings(path, rewrites); + path = rewriteStrings(std::move(path), rewrites); } catch (InvalidPathError & e) { state.error( "cannot find '%1%', since path '%2%' is not valid", @@ -1908,8 +1908,8 @@ static void prim_findFile(EvalState & state, const PosIdx pos, Value * * args, V } lookupPath.elements.emplace_back(LookupPath::Elem { - .prefix = LookupPath::Prefix { .s = prefix }, - .path = LookupPath::Path { .s = path }, + .prefix = LookupPath::Prefix { .s = std::move(prefix) }, + .path = LookupPath::Path { .s = std::move(path) }, }); } @@ -2374,8 +2374,8 @@ static RegisterPrimOp primop_fromJSON({ static void prim_toFile(EvalState & state, const PosIdx pos, Value * * args, Value & v) { NixStringContext context; - std::string name(state.forceStringNoCtx(*args[0], pos, "while evaluating the first argument passed to builtins.toFile")); - std::string contents(state.forceString(*args[1], context, pos, "while evaluating the second argument passed to builtins.toFile")); + auto name = state.forceStringNoCtx(*args[0], pos, "while evaluating the first argument passed to builtins.toFile"); + auto contents = state.forceString(*args[1], context, pos, "while evaluating the second argument passed to builtins.toFile"); StorePathSet refs; @@ -2632,7 +2632,7 @@ static RegisterPrimOp primop_filterSource({ static void prim_path(EvalState & state, const PosIdx pos, Value * * args, Value & v) { std::optional path; - std::string name; + std::string_view name; Value * filterFun = nullptr; auto method = ContentAddressMethod::Raw::NixArchive; std::optional expectedHash; @@ -4562,12 +4562,12 @@ static void prim_replaceStrings(EvalState & state, const PosIdx pos, Value * * a "'from' and 'to' arguments passed to builtins.replaceStrings have different lengths" ).atPos(pos).debugThrow(); - std::vector from; + std::vector from; from.reserve(args[0]->listSize()); for (auto elem : args[0]->listItems()) from.emplace_back(state.forceString(*elem, pos, "while evaluating one of the strings to replace passed to builtins.replaceStrings")); - std::unordered_map cache; + std::unordered_map cache; auto to = args[1]->listItems(); NixStringContext context; From 18dc96269d01f7fa1e2e61086444a89f5ce18890 Mon Sep 17 00:00:00 2001 From: Pol Dellaiera Date: Sun, 15 Jun 2025 22:58:26 +0200 Subject: [PATCH 159/218] docker: add basics OpenContainers labels --- docker.nix | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/docker.nix b/docker.nix index 14648f0d125..74e970e4bfb 100644 --- a/docker.nix +++ b/docker.nix @@ -19,6 +19,13 @@ gid ? 0, uname ? "root", gname ? "root", + Labels ? { + "org.opencontainers.image.title" = "Nix"; + "org.opencontainers.image.source" = "https://github.com/NixOS/nix"; + "org.opencontainers.image.vendor" = "Nix project"; + "org.opencontainers.image.version" = nix.version; + "org.opencontainers.image.description" = "Nix container image"; + }, # Default Packages nix, bashInteractive, @@ -352,6 +359,7 @@ dockerTools.buildLayeredImageWithNixDb { ''; config = { + inherit Labels; Cmd = [ (lib.getExe bashInteractive) ]; User = "${toString uid}:${toString gid}"; Env = [ From bb44347fac3b97cc19f1d0ab0594478cc928cbd8 Mon Sep 17 00:00:00 2001 From: Pol Dellaiera Date: Sun, 15 Jun 2025 23:04:50 +0200 Subject: [PATCH 160/218] docker: expose `config.Cmd` as parameter --- docker.nix | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docker.nix b/docker.nix index 74e970e4bfb..6cfa7d55197 100644 --- a/docker.nix +++ b/docker.nix @@ -26,6 +26,7 @@ "org.opencontainers.image.version" = nix.version; "org.opencontainers.image.description" = "Nix container image"; }, + Cmd ? [ (lib.getExe bashInteractive) ], # Default Packages nix, bashInteractive, @@ -359,8 +360,7 @@ dockerTools.buildLayeredImageWithNixDb { ''; config = { - inherit Labels; - Cmd = [ (lib.getExe bashInteractive) ]; + inherit Cmd Labels; User = "${toString uid}:${toString gid}"; Env = [ "USER=${uname}" From d64c92216409b08ca40f43e601fbdae59114e812 Mon Sep 17 00:00:00 2001 From: Wolfgang Walther Date: Tue, 17 Jun 2025 08:45:29 +0200 Subject: [PATCH 161/218] libstore: fix race condition when creating state directories Running parallel nix in nix can lead to multiple instances trying to create the state directories and failing on the `createSymlink` step, because the link already exists. `replaceSymlink` is already idempotent, so let's use that. Resolves #2706 --- src/libstore/local-store.cc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/libstore/local-store.cc b/src/libstore/local-store.cc index 76fadba8649..e53cab2dcfc 100644 --- a/src/libstore/local-store.cc +++ b/src/libstore/local-store.cc @@ -133,7 +133,7 @@ LocalStore::LocalStore(ref config) Path gcRootsDir = config->stateDir + "/gcroots"; if (!pathExists(gcRootsDir)) { createDirs(gcRootsDir); - createSymlink(profilesDir, gcRootsDir + "/profiles"); + replaceSymlink(profilesDir, gcRootsDir + "/profiles"); } for (auto & perUserDir : {profilesDir + "/per-user", gcRootsDir + "/per-user"}) { From 59c7dac8675d4fd0ba607d9380a972b46d7804b6 Mon Sep 17 00:00:00 2001 From: Eelco Dolstra Date: Thu, 12 Jun 2025 15:54:34 +0200 Subject: [PATCH 162/218] Git fetcher: Do not consider a null revision (i.e. workdir) to be locked --- src/libfetchers/git.cc | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/src/libfetchers/git.cc b/src/libfetchers/git.cc index f7e4894fc8c..cf255c00183 100644 --- a/src/libfetchers/git.cc +++ b/src/libfetchers/git.cc @@ -883,7 +883,8 @@ struct GitInputScheme : InputScheme bool isLocked(const Input & input) const override { - return (bool) input.getRev(); + auto rev = input.getRev(); + return rev && rev != nullRev; } }; From 77f6b6532f582a9db2bd6317f4fd272c32a05c7d Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Vladim=C3=ADr=20=C4=8Cun=C3=A1t?= Date: Wed, 18 Jun 2025 10:05:02 +0200 Subject: [PATCH 163/218] tests: fixup with jq-1.8.0 --- tests/functional/flakes/flakes.sh | 2 +- tests/functional/flakes/relative-paths.sh | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/tests/functional/flakes/flakes.sh b/tests/functional/flakes/flakes.sh index e8b051198fd..ce695a6cbcd 100755 --- a/tests/functional/flakes/flakes.sh +++ b/tests/functional/flakes/flakes.sh @@ -160,7 +160,7 @@ expect 1 nix build -o "$TEST_ROOT/result" "$flake2Dir#bar" --no-update-lock-file nix build -o "$TEST_ROOT/result" "$flake2Dir#bar" --commit-lock-file [[ -e "$flake2Dir/flake.lock" ]] [[ -z $(git -C "$flake2Dir" diff main || echo failed) ]] -[[ $(jq --indent 0 . < "$flake2Dir/flake.lock") =~ ^'{"nodes":{"flake1":{"locked":{"lastModified":'.*',"narHash":"sha256-'.*'","ref":"refs/heads/master","rev":"'.*'","revCount":2,"type":"git","url":"file:///'.*'"},"original":{"id":"flake1","type":"indirect"}},"root":{"inputs":{"flake1":"flake1"}}},"root":"root","version":7}'$ ]] +[[ $(jq --indent 0 --compact-output . < "$flake2Dir/flake.lock") =~ ^'{"nodes":{"flake1":{"locked":{"lastModified":'.*',"narHash":"sha256-'.*'","ref":"refs/heads/master","rev":"'.*'","revCount":2,"type":"git","url":"file:///'.*'"},"original":{"id":"flake1","type":"indirect"}},"root":{"inputs":{"flake1":"flake1"}}},"root":"root","version":7}'$ ]] # Rerunning the build should not change the lockfile. nix build -o "$TEST_ROOT/result" "$flake2Dir#bar" diff --git a/tests/functional/flakes/relative-paths.sh b/tests/functional/flakes/relative-paths.sh index 5810fdebe0c..7480cd50458 100644 --- a/tests/functional/flakes/relative-paths.sh +++ b/tests/functional/flakes/relative-paths.sh @@ -69,7 +69,7 @@ git -C "$rootFlake" add flake.nix sub2/flake.nix git -C "$rootFlake" add sub2/flake.lock [[ $(nix eval "$subflake2#y") = 15 ]] -[[ $(jq --indent 0 . < "$subflake2/flake.lock") =~ ^'{"nodes":{"root":{"inputs":{"root":"root_2","sub1":"sub1"}},"root_2":{"inputs":{"sub0":"sub0"},"locked":{"path":"..","type":"path"},"original":{"path":"..","type":"path"},"parent":[]},"root_3":{"inputs":{"sub0":"sub0_2"},"locked":{"path":"../","type":"path"},"original":{"path":"../","type":"path"},"parent":["sub1"]},"sub0":{"locked":{"path":"sub0","type":"path"},"original":{"path":"sub0","type":"path"},"parent":["root"]},"sub0_2":{"locked":{"path":"sub0","type":"path"},"original":{"path":"sub0","type":"path"},"parent":["sub1","root"]},"sub1":{"inputs":{"root":"root_3"},"locked":{"path":"../sub1","type":"path"},"original":{"path":"../sub1","type":"path"},"parent":[]}},"root":"root","version":7}'$ ]] +[[ $(jq --indent 0 --compact-output . < "$subflake2/flake.lock") =~ ^'{"nodes":{"root":{"inputs":{"root":"root_2","sub1":"sub1"}},"root_2":{"inputs":{"sub0":"sub0"},"locked":{"path":"..","type":"path"},"original":{"path":"..","type":"path"},"parent":[]},"root_3":{"inputs":{"sub0":"sub0_2"},"locked":{"path":"../","type":"path"},"original":{"path":"../","type":"path"},"parent":["sub1"]},"sub0":{"locked":{"path":"sub0","type":"path"},"original":{"path":"sub0","type":"path"},"parent":["root"]},"sub0_2":{"locked":{"path":"sub0","type":"path"},"original":{"path":"sub0","type":"path"},"parent":["sub1","root"]},"sub1":{"inputs":{"root":"root_3"},"locked":{"path":"../sub1","type":"path"},"original":{"path":"../sub1","type":"path"},"parent":[]}},"root":"root","version":7}'$ ]] # Make sure there are no content locks for relative path flakes. (! grep "$TEST_ROOT" "$subflake2/flake.lock") From da76bc0cacd0fcaddbf3031e40f3a07d5a73ddf7 Mon Sep 17 00:00:00 2001 From: Nikita Krasnov Date: Wed, 18 Jun 2025 12:40:07 +0300 Subject: [PATCH 164/218] Fix broken link --- doc/manual/source/store/store-object.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/manual/source/store/store-object.md b/doc/manual/source/store/store-object.md index 115e107fbc9..bbf4803ddb5 100644 --- a/doc/manual/source/store/store-object.md +++ b/doc/manual/source/store/store-object.md @@ -18,7 +18,7 @@ In particular, the edge corresponding to a reference is from the store object th References other than a self-reference must not form a cycle. The graph of references excluding self-references thus forms a [directed acyclic graph]. -[directed acyclic graph]: @docroot@/glossary.md#gloss-directed acyclic graph +[directed acyclic graph]: @docroot@/glossary.md#gloss-directed-acyclic-graph We can take the [transitive closure] of the references graph, which any pair of store objects have an edge not if there is a single reference from the first to the second, but a path of one or more references from the first to the second. The *requisites* of a store object are all store objects reachable by paths of references which start with given store object's references. From 86dda9884ae5aa8674470df051c3954d4e6665c4 Mon Sep 17 00:00:00 2001 From: Nikita Krasnov Date: Wed, 18 Jun 2025 12:46:53 +0300 Subject: [PATCH 165/218] Fix typo --- doc/manual/source/store/store-object.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/manual/source/store/store-object.md b/doc/manual/source/store/store-object.md index 115e107fbc9..0cdf4ec9131 100644 --- a/doc/manual/source/store/store-object.md +++ b/doc/manual/source/store/store-object.md @@ -25,7 +25,7 @@ The *requisites* of a store object are all store objects reachable by paths of r [transitive closure]: https://en.wikipedia.org/wiki/Transitive_closure -We can also take the [transpose graph] ofthe references graph, where we reverse the orientation of all edges. +We can also take the [transpose graph] of the references graph, where we reverse the orientation of all edges. The *referrers* of a store object are the store objects that reference it. [transpose graph]: https://en.wikipedia.org/wiki/Transpose_graph From d2a25fbe5123645fe2426e44918b55b5c3cccef2 Mon Sep 17 00:00:00 2001 From: Luc Perkins Date: Wed, 18 Jun 2025 08:23:37 -0700 Subject: [PATCH 166/218] Fix Nix formatting changes --- tests/nixos/github-flakes.nix | 20 +++++++++++--------- 1 file changed, 11 insertions(+), 9 deletions(-) diff --git a/tests/nixos/github-flakes.nix b/tests/nixos/github-flakes.nix index 3d7a77ec52b..06142c2efda 100644 --- a/tests/nixos/github-flakes.nix +++ b/tests/nixos/github-flakes.nix @@ -1,7 +1,8 @@ -{ lib -, config -, nixpkgs -, ... +{ + lib, + config, + nixpkgs, + ... }: let pkgs = config.nodes.client.nixpkgs.pkgs; @@ -146,11 +147,12 @@ in }; client = - { config - , lib - , pkgs - , nodes - , ... + { + config, + lib, + pkgs, + nodes, + ... }: { virtualisation.writableStore = true; From 9b57573baea5abd242c5f62f537c7582c0097c3b Mon Sep 17 00:00:00 2001 From: Eelco Dolstra Date: Wed, 18 Jun 2025 18:06:24 +0200 Subject: [PATCH 167/218] Revert "Drop magic-nix-cache" This reverts commit 9cc8be26747a0206613421a1ba1c3b1f54212e8b since magic-nix-cache works again (thanks @jchv). --- .github/workflows/ci.yml | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index fb70fae871e..29cb33f56af 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -40,6 +40,7 @@ jobs: extra_nix_config: | sandbox = true max-jobs = 1 + - uses: DeterminateSystems/magic-nix-cache-action@main # Since ubuntu 22.30, unprivileged usernamespaces are no longer allowed to map to the root user: # https://ubuntu.com/blog/ubuntu-23-10-restricted-unprivileged-user-namespaces - run: sudo sysctl -w kernel.apparmor_restrict_unprivileged_userns=0 @@ -133,6 +134,7 @@ jobs: - uses: cachix/install-nix-action@v31 with: install_url: https://releases.nixos.org/nix/nix-2.20.3/install + - uses: DeterminateSystems/magic-nix-cache-action@main - run: echo NIX_VERSION="$(nix --experimental-features 'nix-command flakes' eval .\#nix.version | tr -d \")" >> $GITHUB_ENV - run: nix --experimental-features 'nix-command flakes' build .#dockerImage -L - run: docker load -i ./result/image.tar.gz @@ -174,6 +176,7 @@ jobs: steps: - uses: actions/checkout@v4 - uses: DeterminateSystems/nix-installer-action@main + - uses: DeterminateSystems/magic-nix-cache-action@main - run: | nix build -L \ .#hydraJobs.tests.functional_user \ @@ -199,4 +202,5 @@ jobs: repository: NixOS/flake-regressions-data path: flake-regressions/tests - uses: DeterminateSystems/nix-installer-action@main + - uses: DeterminateSystems/magic-nix-cache-action@main - run: nix build -L --out-link ./new-nix && PATH=$(pwd)/new-nix/bin:$PATH MAX_FLAKES=25 flake-regressions/eval-all.sh From 20ba6be7491c14978dba52991c7257342b9087e4 Mon Sep 17 00:00:00 2001 From: Eelco Dolstra Date: Thu, 19 Jun 2025 13:58:51 +0200 Subject: [PATCH 168/218] Improve the Rosetta installation hint The Nix daemon detects supported system types at start time, so it needs to be restarted to detect x86_64-darwin support. --- src/libstore/unix/build/derivation-builder.cc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/libstore/unix/build/derivation-builder.cc b/src/libstore/unix/build/derivation-builder.cc index 1f15cc9e95f..85b58637399 100644 --- a/src/libstore/unix/build/derivation-builder.cc +++ b/src/libstore/unix/build/derivation-builder.cc @@ -700,7 +700,7 @@ void DerivationBuilderImpl::startBuilder() // since aarch64-darwin has Rosetta 2, this user can actually run x86_64-darwin on their hardware - we should tell them to run the command to install Darwin 2 if (drv.platform == "x86_64-darwin" && settings.thisSystem == "aarch64-darwin") - msg += fmt("\nNote: run `%s` to run programs for x86_64-darwin", Magenta("/usr/sbin/softwareupdate --install-rosetta")); + msg += fmt("\nNote: run `%s` to run programs for x86_64-darwin", Magenta("/usr/sbin/softwareupdate --install-rosetta && launchctl stop org.nixos.nix-daemon")); throw BuildError(msg); } From 3132aba8e425628ec74de134cea02e1bc85cd7c1 Mon Sep 17 00:00:00 2001 From: Luc Perkins Date: Thu, 19 Jun 2025 15:23:10 -0700 Subject: [PATCH 169/218] Fix broken test --- tests/functional/fetchGitShallow.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/functional/fetchGitShallow.sh b/tests/functional/fetchGitShallow.sh index cf7e5fd4fbd..4c21bd7ac80 100644 --- a/tests/functional/fetchGitShallow.sh +++ b/tests/functional/fetchGitShallow.sh @@ -43,7 +43,7 @@ path=$(nix eval --impure --raw --expr "(builtins.fetchTree { type = \"git\"; url [[ $(nix eval --impure --expr "(builtins.fetchTree { type = \"git\"; url = \"file://$TEST_ROOT/shallow-clone\"; ref = \"dev\"; shallow = true; }).revCount or 123") == 123 ]] # Test 3: Check unlocked input error message -expectStderr 1 nix eval --expr 'builtins.fetchTree { type = "git"; url = "file:///foo"; }' | grepQuiet "'fetchTree' will not fetch unlocked input" +expectStderr 1 nix eval --expr 'builtins.fetchTree { type = "git"; url = "file:///foo"; }' | grepQuiet "'fetchTree' doesn't fetch unlocked input" # Test 4: Regression test for revCount in worktrees derived from shallow clones # Add a worktree to the shallow clone From 785f3867fd6c4f0dfa0863b215a30ae262187472 Mon Sep 17 00:00:00 2001 From: Nikita Krasnov Date: Fri, 20 Jun 2025 21:19:13 +0300 Subject: [PATCH 170/218] Update docs --- doc/manual/source/store/derivation/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/manual/source/store/derivation/index.md b/doc/manual/source/store/derivation/index.md index 147330df6ba..5ebbfa7c780 100644 --- a/doc/manual/source/store/derivation/index.md +++ b/doc/manual/source/store/derivation/index.md @@ -200,7 +200,7 @@ This ensures that there is a canonical [store path] used to refer to the derivat > **Note** > > Currently, the canonical encoding for every derivation is the "ATerm" format, -> but this is subject to change for types derivations which are not yet stable. +> but this is subject to change for the types of derivations which are not yet stable. Regardless of the format used, when serializing a derivation to a store object, that store object will be content-addressed. From 6ef683cb2ae64fa9bdddf54356613b2adf6844c6 Mon Sep 17 00:00:00 2001 From: Sergei Zimmerman Date: Fri, 20 Jun 2025 23:12:36 +0300 Subject: [PATCH 171/218] Restore multiline formatting of lists in meson files Applies a workaround to enforce multiline formatting of lists to reduce code churn introduced in 93a42a597145a2ff0214ddc6a9828921056c3657. --- meson.build | 5 ++++- .../include/nix/expr/tests/meson.build | 7 ++++++- src/libstore/linux/include/nix/store/meson.build | 5 ++++- src/libstore/linux/meson.build | 5 ++++- src/libutil/freebsd/include/nix/util/meson.build | 5 ++++- src/libutil/freebsd/meson.build | 5 ++++- src/libutil/linux/include/nix/util/meson.build | 6 +++++- src/libutil/linux/meson.build | 6 +++++- src/libutil/windows/include/nix/util/meson.build | 7 ++++++- src/perl/t/meson.build | 5 ++++- tests/functional/plugins/meson.build | 5 ++++- tests/functional/test-libstoreconsumer/meson.build | 5 ++++- 12 files changed, 54 insertions(+), 12 deletions(-) diff --git a/meson.build b/meson.build index 0237919790f..4a3a517fb2d 100644 --- a/meson.build +++ b/meson.build @@ -6,7 +6,10 @@ project( 'cpp', version : files('.version'), subproject_dir : 'src', - default_options : [ 'localstatedir=/nix/var' ], + default_options : [ + 'localstatedir=/nix/var', + # hack for trailing newline + ], meson_version : '>= 1.1', ) diff --git a/src/libexpr-test-support/include/nix/expr/tests/meson.build b/src/libexpr-test-support/include/nix/expr/tests/meson.build index e44d2558234..84ec401abad 100644 --- a/src/libexpr-test-support/include/nix/expr/tests/meson.build +++ b/src/libexpr-test-support/include/nix/expr/tests/meson.build @@ -2,4 +2,9 @@ include_dirs = [ include_directories('../../..') ] -headers = files('libexpr.hh', 'nix_api_expr.hh', 'value/context.hh') +headers = files( + 'libexpr.hh', + 'nix_api_expr.hh', + 'value/context.hh', + # hack for trailing newline +) diff --git a/src/libstore/linux/include/nix/store/meson.build b/src/libstore/linux/include/nix/store/meson.build index eb2ae25837c..c8e6a826863 100644 --- a/src/libstore/linux/include/nix/store/meson.build +++ b/src/libstore/linux/include/nix/store/meson.build @@ -1,3 +1,6 @@ include_dirs += include_directories('../..') -headers += files('personality.hh') +headers += files( + 'personality.hh', + # hack for trailing newline +) diff --git a/src/libstore/linux/meson.build b/src/libstore/linux/meson.build index 93084a8ae2b..5771cead501 100644 --- a/src/libstore/linux/meson.build +++ b/src/libstore/linux/meson.build @@ -1,3 +1,6 @@ -sources += files('personality.cc') +sources += files( + 'personality.cc', + # hack for trailing newline +) subdir('include/nix/store') diff --git a/src/libutil/freebsd/include/nix/util/meson.build b/src/libutil/freebsd/include/nix/util/meson.build index 283c49b9510..4b7d78624d9 100644 --- a/src/libutil/freebsd/include/nix/util/meson.build +++ b/src/libutil/freebsd/include/nix/util/meson.build @@ -2,4 +2,7 @@ include_dirs += include_directories('../..') -headers += files('freebsd-jail.hh') +headers += files( + 'freebsd-jail.hh', + # hack for trailing newline +) diff --git a/src/libutil/freebsd/meson.build b/src/libutil/freebsd/meson.build index 34508efd0ee..d9b91a03d8a 100644 --- a/src/libutil/freebsd/meson.build +++ b/src/libutil/freebsd/meson.build @@ -1,3 +1,6 @@ -sources += files('freebsd-jail.cc') +sources += files( + 'freebsd-jail.cc', + # hack for trailing newline +) subdir('include/nix/util') diff --git a/src/libutil/linux/include/nix/util/meson.build b/src/libutil/linux/include/nix/util/meson.build index ac5f318f869..ec7030c4995 100644 --- a/src/libutil/linux/include/nix/util/meson.build +++ b/src/libutil/linux/include/nix/util/meson.build @@ -2,4 +2,8 @@ include_dirs += include_directories('../..') -headers += files('cgroup.hh', 'linux-namespaces.hh') +headers += files( + 'cgroup.hh', + 'linux-namespaces.hh', + # hack for trailing newline +) diff --git a/src/libutil/linux/meson.build b/src/libutil/linux/meson.build index f3fe07c9cb1..230dd46f37c 100644 --- a/src/libutil/linux/meson.build +++ b/src/libutil/linux/meson.build @@ -1,3 +1,7 @@ -sources += files('cgroup.cc', 'linux-namespaces.cc') +sources += files( + 'cgroup.cc', + 'linux-namespaces.cc', + # hack for trailing newline +) subdir('include/nix/util') diff --git a/src/libutil/windows/include/nix/util/meson.build b/src/libutil/windows/include/nix/util/meson.build index f0d4c37e956..5d0ace9292a 100644 --- a/src/libutil/windows/include/nix/util/meson.build +++ b/src/libutil/windows/include/nix/util/meson.build @@ -2,4 +2,9 @@ include_dirs += include_directories('../..') -headers += files('signals-impl.hh', 'windows-async-pipe.hh', 'windows-error.hh') +headers += files( + 'signals-impl.hh', + 'windows-async-pipe.hh', + 'windows-error.hh', + # hack for trailing newline +) diff --git a/src/perl/t/meson.build b/src/perl/t/meson.build index cd98453c357..5e75920ac0a 100644 --- a/src/perl/t/meson.build +++ b/src/perl/t/meson.build @@ -5,7 +5,10 @@ # src #--------------------------------------------------- -nix_perl_tests = files('init.t') +nix_perl_tests = files( + 'init.t', + # hack for trailing newline +) foreach f : nix_perl_tests diff --git a/tests/functional/plugins/meson.build b/tests/functional/plugins/meson.build index 7cbc935c9c5..41050ffc16a 100644 --- a/tests/functional/plugins/meson.build +++ b/tests/functional/plugins/meson.build @@ -1,6 +1,9 @@ libplugintest = shared_module( 'plugintest', 'plugintest.cc', - dependencies : [ dependency('nix-expr') ], + dependencies : [ + dependency('nix-expr'), + # hack for trailing newline + ], build_by_default : false, ) diff --git a/tests/functional/test-libstoreconsumer/meson.build b/tests/functional/test-libstoreconsumer/meson.build index d27647ec460..ce566035fee 100644 --- a/tests/functional/test-libstoreconsumer/meson.build +++ b/tests/functional/test-libstoreconsumer/meson.build @@ -1,6 +1,9 @@ libstoreconsumer_tester = executable( 'test-libstoreconsumer', 'main.cc', - dependencies : [ dependency('nix-store') ], + dependencies : [ + dependency('nix-store'), + # hack for trailing newline + ], build_by_default : false, ) From 7226a116a0bbc1f304d8160615525cb0aeb46096 Mon Sep 17 00:00:00 2001 From: Raito Bezarius Date: Wed, 26 Mar 2025 01:04:12 +0100 Subject: [PATCH 172/218] libutil: guess or invent a path from file descriptors This is useful for certain error recovery paths (no pun intended) that does not thread through the original path name. Change-Id: I2d800740cb4f9912e64c923120d3f977c58ccb7e Signed-off-by: Raito Bezarius --- src/libutil/file-descriptor.cc | 26 +++++++++++++++++++ .../include/nix/util/file-descriptor.hh | 17 ++++++++++++ src/libutil/meson.build | 2 ++ 3 files changed, 45 insertions(+) diff --git a/src/libutil/file-descriptor.cc b/src/libutil/file-descriptor.cc index 9e0827442a1..9193a30bbde 100644 --- a/src/libutil/file-descriptor.cc +++ b/src/libutil/file-descriptor.cc @@ -1,5 +1,8 @@ #include "nix/util/serialise.hh" #include "nix/util/util.hh" +#include "nix/util/file-system.hh" + +#include "util-config-private.hh" #include #include @@ -74,6 +77,29 @@ Descriptor AutoCloseFD::get() const return fd; } +std::string guessOrInventPathFromFD(Descriptor fd) + { + assert(fd >= 0); + /* On Linux, there's no F_GETPATH available. + * But we can read /proc/ */ + #if defined(__linux__) + try { + return readLink(fmt("/proc/self/fd/%1%", fd)); + } catch (...) { + } + #elif defined (HAVE_F_GETPATH) && HAVE_F_GETPATH + std::string fdName(PATH_MAX, '\0'); + if (fcntl(fd, F_GETPATH, fdName.data()) != -1) { + fdName.resize(strlen(fdName.c_str())); + return fdName; + } + #else + #error "No implementation for retrieving file descriptors path." + #endif + + return fmt("", fd); + } + void AutoCloseFD::close() { diff --git a/src/libutil/include/nix/util/file-descriptor.hh b/src/libutil/include/nix/util/file-descriptor.hh index e2bcce2a283..35b359fb592 100644 --- a/src/libutil/include/nix/util/file-descriptor.hh +++ b/src/libutil/include/nix/util/file-descriptor.hh @@ -106,6 +106,14 @@ void drainFD( #endif ); + /* + * Will attempt to guess *A* path associated that might lead to the same file as used by this + * file descriptor. + * + * The returned string should NEVER be used as a valid path. + */ + std::string guessOrInventPathFromFD(Descriptor fd); + /** * Get [Standard Input](https://en.wikipedia.org/wiki/Standard_streams#Standard_input_(stdin)) */ @@ -160,6 +168,15 @@ public: AutoCloseFD& operator =(const AutoCloseFD & fd) = delete; AutoCloseFD& operator =(AutoCloseFD&& fd); Descriptor get() const; + + /** + * Will attempt to guess *A* path associated that might lead to the same file as used by this + * file descriptor. + * + * The returned string should NEVER be used as a valid path. + */ + std::string guessOrInventPath() const { return guessOrInventPathFromFD(fd); } + explicit operator bool() const; Descriptor release(); void close(); diff --git a/src/libutil/meson.build b/src/libutil/meson.build index f5ad2b1f680..2203e2294f8 100644 --- a/src/libutil/meson.build +++ b/src/libutil/meson.build @@ -37,6 +37,8 @@ foreach funcspec : check_funcs configdata.set(define_name, define_value, description: funcspec[1]) endforeach +configdata.set('HAVE_F_GETPATH', cxx.has_header_symbol('fcntl.h', 'F_GETPATH').to_int()) + subdir('nix-meson-build-support/libatomic') if host_machine.system() == 'windows' From 6a5b6ad3b75a1032ee495cec456e8d28b7e0595e Mon Sep 17 00:00:00 2001 From: Raito Bezarius Date: Wed, 26 Mar 2025 01:04:59 +0100 Subject: [PATCH 173/218] libstore: open build directory as a dirfd as well We now keep around a proper AutoCloseFD around the temporary directory which we plan to use for openat operations and avoiding the build directory being swapped out while we are doing something else. Change-Id: I18d387b0f123ebf2d20c6405cd47ebadc5505f2a Signed-off-by: Raito Bezarius --- src/libstore/unix/build/derivation-builder.cc | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/src/libstore/unix/build/derivation-builder.cc b/src/libstore/unix/build/derivation-builder.cc index 85b58637399..e2dfc586032 100644 --- a/src/libstore/unix/build/derivation-builder.cc +++ b/src/libstore/unix/build/derivation-builder.cc @@ -95,6 +95,11 @@ class DerivationBuilderImpl : public DerivationBuilder, public DerivationBuilder */ Path topTmpDir; + /** + * The file descriptor of the temporary directory. + */ + AutoCloseFD tmpDirFd; + /** * The sort of derivation we are building. * @@ -710,6 +715,13 @@ void DerivationBuilderImpl::startBuilder() topTmpDir = createTempDir(settings.buildDir.get().value_or(""), "nix-build-" + std::string(drvPath.name()), 0700); setBuildTmpDir(); assert(!tmpDir.empty()); + + /* The TOCTOU between the previous mkdir call and this open call is unavoidable due to + POSIX semantics.*/ + tmpDirFd = AutoCloseFD{open(tmpDir.c_str(), O_RDONLY | O_NOFOLLOW | O_DIRECTORY)}; + if (!tmpDirFd) + throw SysError("failed to open the build temporary directory descriptor '%1%'", tmpDir); + chownToBuilder(tmpDir); for (auto & [outputName, status] : initialOutputs) { From 002d202653a2750872ed7b943363ce4029af4dec Mon Sep 17 00:00:00 2001 From: Raito Bezarius Date: Wed, 26 Mar 2025 01:05:34 +0100 Subject: [PATCH 174/218] libstore: chown to builder variant for file descriptors We use it immediately for the build temporary directory. Change-Id: I180193c63a2b98721f5fb8e542c4e39c099bb947 Signed-off-by: Raito Bezarius --- src/libstore/unix/build/derivation-builder.cc | 17 ++++++++++++++++- 1 file changed, 16 insertions(+), 1 deletion(-) diff --git a/src/libstore/unix/build/derivation-builder.cc b/src/libstore/unix/build/derivation-builder.cc index e2dfc586032..30468d3b28a 100644 --- a/src/libstore/unix/build/derivation-builder.cc +++ b/src/libstore/unix/build/derivation-builder.cc @@ -305,9 +305,17 @@ class DerivationBuilderImpl : public DerivationBuilder, public DerivationBuilder /** * Make a file owned by the builder. + * + * SAFETY: this function is prone to TOCTOU as it receives a path and not a descriptor. + * It's only safe to call in a child of a directory only visible to the owner. */ void chownToBuilder(const Path & path); + /** + * Make a file owned by the builder addressed by its file descriptor. + */ + void chownToBuilder(const AutoCloseFD & fd); + /** * Run the builder's process. */ @@ -722,7 +730,7 @@ void DerivationBuilderImpl::startBuilder() if (!tmpDirFd) throw SysError("failed to open the build temporary directory descriptor '%1%'", tmpDir); - chownToBuilder(tmpDir); + chownToBuilder(tmpDirFd); for (auto & [outputName, status] : initialOutputs) { /* Set scratch path we'll actually use during the build. @@ -1267,6 +1275,13 @@ void DerivationBuilderImpl::chownToBuilder(const Path & path) throw SysError("cannot change ownership of '%1%'", path); } +void DerivationBuilderImpl::chownToBuilder(const AutoCloseFD & fd) +{ + if (!buildUser) return; + if (fchown(fd.get(), buildUser->getUID(), buildUser->getGID()) == -1) + throw SysError("cannot change ownership of file '%1%'", fd.guessOrInventPath()); +} + void DerivationBuilderImpl::runChild() { /* Warning: in the child we should absolutely not make any SQLite From 034f59bbb9a8c6a5febfed042fa9119410a3f123 Mon Sep 17 00:00:00 2001 From: Raito Bezarius Date: Wed, 26 Mar 2025 01:06:03 +0100 Subject: [PATCH 175/218] libutil: writeFile variant for file descriptors `writeFile` lose its `sync` boolean flag to make things simpler. A new `writeFileAndSync` function is created and all call sites are converted to it. Change-Id: Ib871a5283a9c047db1e4fe48a241506e4aab9192 Signed-off-by: Raito Bezarius --- src/libstore/local-store.cc | 4 +-- src/libutil/file-system.cc | 35 ++++++++++++++++----- src/libutil/include/nix/util/file-system.hh | 13 ++++++-- 3 files changed, 39 insertions(+), 13 deletions(-) diff --git a/src/libstore/local-store.cc b/src/libstore/local-store.cc index e53cab2dcfc..00c1ac6dd9f 100644 --- a/src/libstore/local-store.cc +++ b/src/libstore/local-store.cc @@ -247,7 +247,7 @@ LocalStore::LocalStore(ref config) else if (curSchema == 0) { /* new store */ curSchema = nixSchemaVersion; openDB(*state, true); - writeFile(schemaPath, fmt("%1%", curSchema), 0666, true); + writeFileAndSync(schemaPath, fmt("%1%", curSchema), 0666); } else if (curSchema < nixSchemaVersion) { @@ -298,7 +298,7 @@ LocalStore::LocalStore(ref config) txn.commit(); } - writeFile(schemaPath, fmt("%1%", nixSchemaVersion), 0666, true); + writeFileAndSync(schemaPath, fmt("%1%", nixSchemaVersion), 0666); lockFile(globalLock.get(), ltRead, true); } diff --git a/src/libutil/file-system.cc b/src/libutil/file-system.cc index fee2945ffef..4c3a56e659b 100644 --- a/src/libutil/file-system.cc +++ b/src/libutil/file-system.cc @@ -304,7 +304,7 @@ void readFile(const Path & path, Sink & sink, bool memory_map) } -void writeFile(const Path & path, std::string_view s, mode_t mode, bool sync) +void writeFile(const Path & path, std::string_view s, mode_t mode) { AutoCloseFD fd = toDescriptor(open(path.c_str(), O_WRONLY | O_TRUNC | O_CREAT // TODO @@ -314,20 +314,39 @@ void writeFile(const Path & path, std::string_view s, mode_t mode, bool sync) , mode)); if (!fd) throw SysError("opening file '%1%'", path); + + writeFile(fd, s, mode); + + /* Close explicitly to propagate the exceptions. */ + fd.close(); +} + +void writeFile(AutoCloseFD & fd, std::string_view s, mode_t mode) +{ + assert(fd); try { writeFull(fd.get(), s); } catch (Error & e) { - e.addTrace({}, "writing file '%1%'", path); + e.addTrace({}, "writing file '%1%'", fd.guessOrInventPath()); throw; } - if (sync) - fd.fsync(); - // Explicitly close to make sure exceptions are propagated. - fd.close(); - if (sync) - syncParent(path); } +void writeFileAndSync(const Path & path, std::string_view s, mode_t mode) +{ + { + AutoCloseFD fd{open(path.c_str(), O_WRONLY | O_TRUNC | O_CREAT | O_CLOEXEC, mode)}; + if (!fd) + throw SysError("opening file '%1%'", path); + + writeFile(fd, s, mode); + fd.fsync(); + /* Close explicitly to ensure that exceptions are propagated. */ + fd.close(); + } + + syncParent(path); +} void writeFile(const Path & path, Source & source, mode_t mode, bool sync) { diff --git a/src/libutil/include/nix/util/file-system.hh b/src/libutil/include/nix/util/file-system.hh index 0121745ab0e..77d5f2aa161 100644 --- a/src/libutil/include/nix/util/file-system.hh +++ b/src/libutil/include/nix/util/file-system.hh @@ -172,10 +172,10 @@ void readFile(const Path & path, Sink & sink, bool memory_map = true); /** * Write a string to a file. */ -void writeFile(const Path & path, std::string_view s, mode_t mode = 0666, bool sync = false); -static inline void writeFile(const std::filesystem::path & path, std::string_view s, mode_t mode = 0666, bool sync = false) +void writeFile(const Path & path, std::string_view s, mode_t mode = 0666); +static inline void writeFile(const std::filesystem::path & path, std::string_view s, mode_t mode = 0666) { - return writeFile(path.string(), s, mode, sync); + return writeFile(path.string(), s, mode); } void writeFile(const Path & path, Source & source, mode_t mode = 0666, bool sync = false); @@ -184,6 +184,13 @@ static inline void writeFile(const std::filesystem::path & path, Source & source return writeFile(path.string(), source, mode, sync); } +void writeFile(AutoCloseFD & fd, std::string_view s, mode_t mode = 0666 ); + +/** + * Write a string to a file and flush the file and its parents direcotry to disk. + */ +void writeFileAndSync(const Path & path, std::string_view s, mode_t mode = 0666); + /** * Flush a path's parent directory to disk. */ From 4e59d3fdb25335d7ab2612e72ff191e9bfbba8f4 Mon Sep 17 00:00:00 2001 From: Raito Bezarius Date: Wed, 26 Mar 2025 01:07:47 +0100 Subject: [PATCH 176/218] libstore: ensure that `passAsFile` is created in the original temp dir This ensures that `passAsFile` data is created inside the expected temporary build directory by `openat()` from the parent directory file descriptor. This avoids a TOCTOU which is part of the attack chain of CVE-????. Change-Id: Ie5273446c4a19403088d0389ae8e3f473af8879a Signed-off-by: Raito Bezarius --- src/libstore/unix/build/derivation-builder.cc | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/src/libstore/unix/build/derivation-builder.cc b/src/libstore/unix/build/derivation-builder.cc index 30468d3b28a..22445d54768 100644 --- a/src/libstore/unix/build/derivation-builder.cc +++ b/src/libstore/unix/build/derivation-builder.cc @@ -1070,8 +1070,11 @@ void DerivationBuilderImpl::initEnv() auto hash = hashString(HashAlgorithm::SHA256, i.first); std::string fn = ".attr-" + hash.to_string(HashFormat::Nix32, false); Path p = tmpDir + "/" + fn; - writeFile(p, rewriteStrings(i.second, inputRewrites)); - chownToBuilder(p); + AutoCloseFD passAsFileFd{openat(tmpDirFd.get(), fn.c_str(), O_WRONLY | O_TRUNC | O_CREAT | O_CLOEXEC | O_EXCL | O_NOFOLLOW, 0666)}; + if (!passAsFileFd) + throw SysError("opening `passAsFile` file in the sandbox '%1%'", p); + writeFile(passAsFileFd, rewriteStrings(i.second, inputRewrites)); + chownToBuilder(passAsFileFd); env[i.first + "Path"] = tmpDirInSandbox() + "/" + fn; } } From 5ec047f348d747d079bfb5d92a10a6c8d446089b Mon Sep 17 00:00:00 2001 From: Raito Bezarius Date: Wed, 26 Mar 2025 12:42:55 +0100 Subject: [PATCH 177/218] libutil: ensure that `_deletePath` does NOT use absolute paths with dirfds When calling `_deletePath` with a parent file descriptor, `openat` is made effective by using relative paths to the directory file descriptor. To avoid the problem, the signature is changed to resist misuse with an assert in the prologue of the function. Change-Id: I6b3fc766bad2afe54dc27d47d1df3873e188de96 Signed-off-by: Raito Bezarius --- src/libutil/file-system.cc | 23 +++++++++++++---------- 1 file changed, 13 insertions(+), 10 deletions(-) diff --git a/src/libutil/file-system.cc b/src/libutil/file-system.cc index 4c3a56e659b..12ada6d5c65 100644 --- a/src/libutil/file-system.cc +++ b/src/libutil/file-system.cc @@ -444,6 +444,8 @@ void recursiveSync(const Path & path) static void _deletePath(Descriptor parentfd, const std::filesystem::path & path, uint64_t & bytesFreed, std::exception_ptr & ex MOUNTEDPATHS_PARAM) { #ifndef _WIN32 + /* This ensures that `name` is an immediate child of `parentfd`. */ + assert(!path.empty() && path.string().find('/') == std::string::npos && "`name` is an immediate child to `parentfd`"); checkInterrupt(); #ifdef __FreeBSD__ @@ -454,13 +456,13 @@ static void _deletePath(Descriptor parentfd, const std::filesystem::path & path, } #endif - std::string name(baseNameOf(path.native())); + std::string name(path.native()); struct stat st; if (fstatat(parentfd, name.c_str(), &st, AT_SYMLINK_NOFOLLOW) == -1) { if (errno == ENOENT) return; - throw SysError("getting status of %1%", path); + throw SysError("getting status of '%1%' in directory '%2%'", name, guessOrInventPathFromFD(parentfd)); } if (!S_ISDIR(st.st_mode)) { @@ -491,23 +493,24 @@ static void _deletePath(Descriptor parentfd, const std::filesystem::path & path, /* Make the directory accessible. */ const auto PERM_MASK = S_IRUSR | S_IWUSR | S_IXUSR; if ((st.st_mode & PERM_MASK) != PERM_MASK) { - if (fchmodat(parentfd, name.c_str(), st.st_mode | PERM_MASK, 0) == -1) - throw SysError("chmod %1%", path); + if (fchmodat(parentfd, name.c_str(), st.st_mode | PERM_MASK, 0) == -1) { + throw SysError("chmod '%1%' in directory '%2%'", name, guessOrInventPathFromFD(parentfd)); + } } - int fd = openat(parentfd, path.c_str(), O_RDONLY); + int fd = openat(parentfd, name.c_str(), O_RDONLY | O_DIRECTORY | O_NOFOLLOW); if (fd == -1) - throw SysError("opening directory %1%", path); + throw SysError("opening directory '%1%' in directory '%2%'", name, guessOrInventPathFromFD(parentfd)); AutoCloseDir dir(fdopendir(fd)); if (!dir) - throw SysError("opening directory %1%", path); + throw SysError("opening directory '%1%' in directory '%2%'", name, guessOrInventPathFromFD(parentfd)); struct dirent * dirent; while (errno = 0, dirent = readdir(dir.get())) { /* sic */ checkInterrupt(); std::string childName = dirent->d_name; if (childName == "." || childName == "..") continue; - _deletePath(dirfd(dir.get()), path + "/" + childName, bytesFreed, ex MOUNTEDPATHS_ARG); + _deletePath(dirfd(dir.get()), childName, bytesFreed, ex MOUNTEDPATHS_ARG); } if (errno) throw SysError("reading directory %1%", path); } @@ -516,7 +519,7 @@ static void _deletePath(Descriptor parentfd, const std::filesystem::path & path, if (unlinkat(parentfd, name.c_str(), flags) == -1) { if (errno == ENOENT) return; try { - throw SysError("cannot unlink %1%", path); + throw SysError("cannot unlink '%1%' in directory '%2%'", name, guessOrInventPathFromFD(parentfd)); } catch (...) { if (!ex) ex = std::current_exception(); @@ -544,7 +547,7 @@ static void _deletePath(const std::filesystem::path & path, uint64_t & bytesFree std::exception_ptr ex; - _deletePath(dirfd.get(), path, bytesFreed, ex MOUNTEDPATHS_ARG); + _deletePath(dirfd.get(), path.filename(), bytesFreed, ex MOUNTEDPATHS_ARG); if (ex) std::rethrow_exception(ex); From 4ea4813753c895118b7377495647b48f240ff4d8 Mon Sep 17 00:00:00 2001 From: Raito Bezarius Date: Thu, 27 Mar 2025 12:22:26 +0100 Subject: [PATCH 178/218] libstore: ensure that temporary directory is always 0o000 before deletion In the case the deletion fails, we should ensure that the temporary directory cannot be used for nefarious purposes. Change-Id: I498a2dd0999a74195d13642f44a5de1e69d46120 Signed-off-by: Raito Bezarius --- src/libstore/unix/build/derivation-builder.cc | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/src/libstore/unix/build/derivation-builder.cc b/src/libstore/unix/build/derivation-builder.cc index 22445d54768..c61fe70013b 100644 --- a/src/libstore/unix/build/derivation-builder.cc +++ b/src/libstore/unix/build/derivation-builder.cc @@ -2093,6 +2093,15 @@ void DerivationBuilderImpl::checkOutputs(const std::map Date: Thu, 12 Jun 2025 11:15:58 +0200 Subject: [PATCH 179/218] Tweak comment --- src/libstore/unix/build/derivation-builder.cc | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/libstore/unix/build/derivation-builder.cc b/src/libstore/unix/build/derivation-builder.cc index c61fe70013b..a125676e567 100644 --- a/src/libstore/unix/build/derivation-builder.cc +++ b/src/libstore/unix/build/derivation-builder.cc @@ -2093,8 +2093,8 @@ void DerivationBuilderImpl::checkOutputs(const std::map Date: Thu, 12 Jun 2025 11:35:28 +0200 Subject: [PATCH 180/218] Drop guessOrInventPathFromFD() No need to do hacky stuff like that when we already know the original path. --- src/libstore/unix/build/derivation-builder.cc | 14 +++---- src/libutil/file-descriptor.cc | 26 ------------- src/libutil/file-system.cc | 39 +++++++++---------- .../include/nix/util/file-descriptor.hh | 17 -------- src/libutil/include/nix/util/file-system.hh | 4 +- src/libutil/meson.build | 2 - 6 files changed, 28 insertions(+), 74 deletions(-) diff --git a/src/libstore/unix/build/derivation-builder.cc b/src/libstore/unix/build/derivation-builder.cc index a125676e567..36405a8a2ff 100644 --- a/src/libstore/unix/build/derivation-builder.cc +++ b/src/libstore/unix/build/derivation-builder.cc @@ -314,7 +314,7 @@ class DerivationBuilderImpl : public DerivationBuilder, public DerivationBuilder /** * Make a file owned by the builder addressed by its file descriptor. */ - void chownToBuilder(const AutoCloseFD & fd); + void chownToBuilder(int fd, const Path & path); /** * Run the builder's process. @@ -730,7 +730,7 @@ void DerivationBuilderImpl::startBuilder() if (!tmpDirFd) throw SysError("failed to open the build temporary directory descriptor '%1%'", tmpDir); - chownToBuilder(tmpDirFd); + chownToBuilder(tmpDirFd.get(), tmpDir); for (auto & [outputName, status] : initialOutputs) { /* Set scratch path we'll actually use during the build. @@ -1073,8 +1073,8 @@ void DerivationBuilderImpl::initEnv() AutoCloseFD passAsFileFd{openat(tmpDirFd.get(), fn.c_str(), O_WRONLY | O_TRUNC | O_CREAT | O_CLOEXEC | O_EXCL | O_NOFOLLOW, 0666)}; if (!passAsFileFd) throw SysError("opening `passAsFile` file in the sandbox '%1%'", p); - writeFile(passAsFileFd, rewriteStrings(i.second, inputRewrites)); - chownToBuilder(passAsFileFd); + writeFile(passAsFileFd, p, rewriteStrings(i.second, inputRewrites)); + chownToBuilder(passAsFileFd.get(), p); env[i.first + "Path"] = tmpDirInSandbox() + "/" + fn; } } @@ -1278,11 +1278,11 @@ void DerivationBuilderImpl::chownToBuilder(const Path & path) throw SysError("cannot change ownership of '%1%'", path); } -void DerivationBuilderImpl::chownToBuilder(const AutoCloseFD & fd) +void DerivationBuilderImpl::chownToBuilder(int fd, const Path & path) { if (!buildUser) return; - if (fchown(fd.get(), buildUser->getUID(), buildUser->getGID()) == -1) - throw SysError("cannot change ownership of file '%1%'", fd.guessOrInventPath()); + if (fchown(fd, buildUser->getUID(), buildUser->getGID()) == -1) + throw SysError("cannot change ownership of file '%1%'", path); } void DerivationBuilderImpl::runChild() diff --git a/src/libutil/file-descriptor.cc b/src/libutil/file-descriptor.cc index 9193a30bbde..9e0827442a1 100644 --- a/src/libutil/file-descriptor.cc +++ b/src/libutil/file-descriptor.cc @@ -1,8 +1,5 @@ #include "nix/util/serialise.hh" #include "nix/util/util.hh" -#include "nix/util/file-system.hh" - -#include "util-config-private.hh" #include #include @@ -77,29 +74,6 @@ Descriptor AutoCloseFD::get() const return fd; } -std::string guessOrInventPathFromFD(Descriptor fd) - { - assert(fd >= 0); - /* On Linux, there's no F_GETPATH available. - * But we can read /proc/ */ - #if defined(__linux__) - try { - return readLink(fmt("/proc/self/fd/%1%", fd)); - } catch (...) { - } - #elif defined (HAVE_F_GETPATH) && HAVE_F_GETPATH - std::string fdName(PATH_MAX, '\0'); - if (fcntl(fd, F_GETPATH, fdName.data()) != -1) { - fdName.resize(strlen(fdName.c_str())); - return fdName; - } - #else - #error "No implementation for retrieving file descriptors path." - #endif - - return fmt("", fd); - } - void AutoCloseFD::close() { diff --git a/src/libutil/file-system.cc b/src/libutil/file-system.cc index 12ada6d5c65..3fd5ae669ba 100644 --- a/src/libutil/file-system.cc +++ b/src/libutil/file-system.cc @@ -315,19 +315,19 @@ void writeFile(const Path & path, std::string_view s, mode_t mode) if (!fd) throw SysError("opening file '%1%'", path); - writeFile(fd, s, mode); + writeFile(fd, path, s, mode); /* Close explicitly to propagate the exceptions. */ fd.close(); } -void writeFile(AutoCloseFD & fd, std::string_view s, mode_t mode) +void writeFile(AutoCloseFD & fd, const Path & origPath, std::string_view s, mode_t mode) { assert(fd); try { writeFull(fd.get(), s); } catch (Error & e) { - e.addTrace({}, "writing file '%1%'", fd.guessOrInventPath()); + e.addTrace({}, "writing file '%1%'", origPath); throw; } } @@ -339,7 +339,7 @@ void writeFileAndSync(const Path & path, std::string_view s, mode_t mode) if (!fd) throw SysError("opening file '%1%'", path); - writeFile(fd, s, mode); + writeFile(fd, path, s, mode); fd.fsync(); /* Close explicitly to ensure that exceptions are propagated. */ fd.close(); @@ -444,8 +444,6 @@ void recursiveSync(const Path & path) static void _deletePath(Descriptor parentfd, const std::filesystem::path & path, uint64_t & bytesFreed, std::exception_ptr & ex MOUNTEDPATHS_PARAM) { #ifndef _WIN32 - /* This ensures that `name` is an immediate child of `parentfd`. */ - assert(!path.empty() && path.string().find('/') == std::string::npos && "`name` is an immediate child to `parentfd`"); checkInterrupt(); #ifdef __FreeBSD__ @@ -456,13 +454,14 @@ static void _deletePath(Descriptor parentfd, const std::filesystem::path & path, } #endif - std::string name(path.native()); + std::string name(path.filename()); + assert(name != "." && name != ".." && !name.empty()); struct stat st; if (fstatat(parentfd, name.c_str(), &st, AT_SYMLINK_NOFOLLOW) == -1) { if (errno == ENOENT) return; - throw SysError("getting status of '%1%' in directory '%2%'", name, guessOrInventPathFromFD(parentfd)); + throw SysError("getting status of %1%", path); } if (!S_ISDIR(st.st_mode)) { @@ -493,24 +492,23 @@ static void _deletePath(Descriptor parentfd, const std::filesystem::path & path, /* Make the directory accessible. */ const auto PERM_MASK = S_IRUSR | S_IWUSR | S_IXUSR; if ((st.st_mode & PERM_MASK) != PERM_MASK) { - if (fchmodat(parentfd, name.c_str(), st.st_mode | PERM_MASK, 0) == -1) { - throw SysError("chmod '%1%' in directory '%2%'", name, guessOrInventPathFromFD(parentfd)); - } + if (fchmodat(parentfd, name.c_str(), st.st_mode | PERM_MASK, 0) == -1) + throw SysError("chmod %1%", path); } int fd = openat(parentfd, name.c_str(), O_RDONLY | O_DIRECTORY | O_NOFOLLOW); if (fd == -1) - throw SysError("opening directory '%1%' in directory '%2%'", name, guessOrInventPathFromFD(parentfd)); + throw SysError("opening directory %1%", path); AutoCloseDir dir(fdopendir(fd)); if (!dir) - throw SysError("opening directory '%1%' in directory '%2%'", name, guessOrInventPathFromFD(parentfd)); + throw SysError("opening directory %1%", path); struct dirent * dirent; while (errno = 0, dirent = readdir(dir.get())) { /* sic */ checkInterrupt(); std::string childName = dirent->d_name; if (childName == "." || childName == "..") continue; - _deletePath(dirfd(dir.get()), childName, bytesFreed, ex MOUNTEDPATHS_ARG); + _deletePath(dirfd(dir.get()), path / childName, bytesFreed, ex MOUNTEDPATHS_ARG); } if (errno) throw SysError("reading directory %1%", path); } @@ -519,7 +517,7 @@ static void _deletePath(Descriptor parentfd, const std::filesystem::path & path, if (unlinkat(parentfd, name.c_str(), flags) == -1) { if (errno == ENOENT) return; try { - throw SysError("cannot unlink '%1%' in directory '%2%'", name, guessOrInventPathFromFD(parentfd)); + throw SysError("cannot unlink %1%", path); } catch (...) { if (!ex) ex = std::current_exception(); @@ -535,19 +533,18 @@ static void _deletePath(Descriptor parentfd, const std::filesystem::path & path, static void _deletePath(const std::filesystem::path & path, uint64_t & bytesFreed MOUNTEDPATHS_PARAM) { - Path dir = dirOf(path.string()); - if (dir == "") - dir = "/"; + assert(path.is_absolute()); + assert(path.parent_path() != path); - AutoCloseFD dirfd = toDescriptor(open(dir.c_str(), O_RDONLY)); + AutoCloseFD dirfd = toDescriptor(open(path.parent_path().string().c_str(), O_RDONLY)); if (!dirfd) { if (errno == ENOENT) return; - throw SysError("opening directory '%1%'", path); + throw SysError("opening directory %s", path.parent_path()); } std::exception_ptr ex; - _deletePath(dirfd.get(), path.filename(), bytesFreed, ex MOUNTEDPATHS_ARG); + _deletePath(dirfd.get(), path, bytesFreed, ex MOUNTEDPATHS_ARG); if (ex) std::rethrow_exception(ex); diff --git a/src/libutil/include/nix/util/file-descriptor.hh b/src/libutil/include/nix/util/file-descriptor.hh index 35b359fb592..e2bcce2a283 100644 --- a/src/libutil/include/nix/util/file-descriptor.hh +++ b/src/libutil/include/nix/util/file-descriptor.hh @@ -106,14 +106,6 @@ void drainFD( #endif ); - /* - * Will attempt to guess *A* path associated that might lead to the same file as used by this - * file descriptor. - * - * The returned string should NEVER be used as a valid path. - */ - std::string guessOrInventPathFromFD(Descriptor fd); - /** * Get [Standard Input](https://en.wikipedia.org/wiki/Standard_streams#Standard_input_(stdin)) */ @@ -168,15 +160,6 @@ public: AutoCloseFD& operator =(const AutoCloseFD & fd) = delete; AutoCloseFD& operator =(AutoCloseFD&& fd); Descriptor get() const; - - /** - * Will attempt to guess *A* path associated that might lead to the same file as used by this - * file descriptor. - * - * The returned string should NEVER be used as a valid path. - */ - std::string guessOrInventPath() const { return guessOrInventPathFromFD(fd); } - explicit operator bool() const; Descriptor release(); void close(); diff --git a/src/libutil/include/nix/util/file-system.hh b/src/libutil/include/nix/util/file-system.hh index 77d5f2aa161..5f062412df9 100644 --- a/src/libutil/include/nix/util/file-system.hh +++ b/src/libutil/include/nix/util/file-system.hh @@ -173,18 +173,20 @@ void readFile(const Path & path, Sink & sink, bool memory_map = true); * Write a string to a file. */ void writeFile(const Path & path, std::string_view s, mode_t mode = 0666); + static inline void writeFile(const std::filesystem::path & path, std::string_view s, mode_t mode = 0666) { return writeFile(path.string(), s, mode); } void writeFile(const Path & path, Source & source, mode_t mode = 0666, bool sync = false); + static inline void writeFile(const std::filesystem::path & path, Source & source, mode_t mode = 0666, bool sync = false) { return writeFile(path.string(), source, mode, sync); } -void writeFile(AutoCloseFD & fd, std::string_view s, mode_t mode = 0666 ); +void writeFile(AutoCloseFD & fd, const Path & origPath, std::string_view s, mode_t mode = 0666); /** * Write a string to a file and flush the file and its parents direcotry to disk. diff --git a/src/libutil/meson.build b/src/libutil/meson.build index 2203e2294f8..f5ad2b1f680 100644 --- a/src/libutil/meson.build +++ b/src/libutil/meson.build @@ -37,8 +37,6 @@ foreach funcspec : check_funcs configdata.set(define_name, define_value, description: funcspec[1]) endforeach -configdata.set('HAVE_F_GETPATH', cxx.has_header_symbol('fcntl.h', 'F_GETPATH').to_int()) - subdir('nix-meson-build-support/libatomic') if host_machine.system() == 'windows' From a4b5584fb10e967f753a54b8c02cc5229b4cab8e Mon Sep 17 00:00:00 2001 From: Eelco Dolstra Date: Thu, 12 Jun 2025 12:14:04 +0200 Subject: [PATCH 181/218] Replace 'bool sync' with an enum for clarity And drop writeFileAndSync(). --- src/libstore/local-store.cc | 4 +-- src/libutil/file-content-address.cc | 2 +- src/libutil/file-system.cc | 32 +++++++-------------- src/libutil/include/nix/util/file-system.hh | 19 ++++++------ 4 files changed, 21 insertions(+), 36 deletions(-) diff --git a/src/libstore/local-store.cc b/src/libstore/local-store.cc index 00c1ac6dd9f..9e132426251 100644 --- a/src/libstore/local-store.cc +++ b/src/libstore/local-store.cc @@ -247,7 +247,7 @@ LocalStore::LocalStore(ref config) else if (curSchema == 0) { /* new store */ curSchema = nixSchemaVersion; openDB(*state, true); - writeFileAndSync(schemaPath, fmt("%1%", curSchema), 0666); + writeFile(schemaPath, fmt("%1%", curSchema), 0666, FsSync::Yes); } else if (curSchema < nixSchemaVersion) { @@ -298,7 +298,7 @@ LocalStore::LocalStore(ref config) txn.commit(); } - writeFileAndSync(schemaPath, fmt("%1%", nixSchemaVersion), 0666); + writeFile(schemaPath, fmt("%1%", nixSchemaVersion), 0666, FsSync::Yes); lockFile(globalLock.get(), ltRead, true); } diff --git a/src/libutil/file-content-address.cc b/src/libutil/file-content-address.cc index 142bc70d534..d957816918d 100644 --- a/src/libutil/file-content-address.cc +++ b/src/libutil/file-content-address.cc @@ -93,7 +93,7 @@ void restorePath( { switch (method) { case FileSerialisationMethod::Flat: - writeFile(path, source, 0666, startFsync); + writeFile(path, source, 0666, startFsync ? FsSync::Yes : FsSync::No); break; case FileSerialisationMethod::NixArchive: restorePath(path, source, startFsync); diff --git a/src/libutil/file-system.cc b/src/libutil/file-system.cc index 3fd5ae669ba..79e6cf3546c 100644 --- a/src/libutil/file-system.cc +++ b/src/libutil/file-system.cc @@ -304,7 +304,7 @@ void readFile(const Path & path, Sink & sink, bool memory_map) } -void writeFile(const Path & path, std::string_view s, mode_t mode) +void writeFile(const Path & path, std::string_view s, mode_t mode, FsSync sync) { AutoCloseFD fd = toDescriptor(open(path.c_str(), O_WRONLY | O_TRUNC | O_CREAT // TODO @@ -315,40 +315,28 @@ void writeFile(const Path & path, std::string_view s, mode_t mode) if (!fd) throw SysError("opening file '%1%'", path); - writeFile(fd, path, s, mode); + writeFile(fd, path, s, mode, sync); /* Close explicitly to propagate the exceptions. */ fd.close(); } -void writeFile(AutoCloseFD & fd, const Path & origPath, std::string_view s, mode_t mode) +void writeFile(AutoCloseFD & fd, const Path & origPath, std::string_view s, mode_t mode, FsSync sync) { assert(fd); try { writeFull(fd.get(), s); + + if (sync == FsSync::Yes) + fd.fsync(); + } catch (Error & e) { e.addTrace({}, "writing file '%1%'", origPath); throw; } } -void writeFileAndSync(const Path & path, std::string_view s, mode_t mode) -{ - { - AutoCloseFD fd{open(path.c_str(), O_WRONLY | O_TRUNC | O_CREAT | O_CLOEXEC, mode)}; - if (!fd) - throw SysError("opening file '%1%'", path); - - writeFile(fd, path, s, mode); - fd.fsync(); - /* Close explicitly to ensure that exceptions are propagated. */ - fd.close(); - } - - syncParent(path); -} - -void writeFile(const Path & path, Source & source, mode_t mode, bool sync) +void writeFile(const Path & path, Source & source, mode_t mode, FsSync sync) { AutoCloseFD fd = toDescriptor(open(path.c_str(), O_WRONLY | O_TRUNC | O_CREAT // TODO @@ -372,11 +360,11 @@ void writeFile(const Path & path, Source & source, mode_t mode, bool sync) e.addTrace({}, "writing file '%1%'", path); throw; } - if (sync) + if (sync == FsSync::Yes) fd.fsync(); // Explicitly close to make sure exceptions are propagated. fd.close(); - if (sync) + if (sync == FsSync::Yes) syncParent(path); } diff --git a/src/libutil/include/nix/util/file-system.hh b/src/libutil/include/nix/util/file-system.hh index 5f062412df9..c45cb55aa74 100644 --- a/src/libutil/include/nix/util/file-system.hh +++ b/src/libutil/include/nix/util/file-system.hh @@ -169,29 +169,26 @@ std::string readFile(const Path & path); std::string readFile(const std::filesystem::path & path); void readFile(const Path & path, Sink & sink, bool memory_map = true); +enum struct FsSync { Yes, No }; + /** * Write a string to a file. */ -void writeFile(const Path & path, std::string_view s, mode_t mode = 0666); +void writeFile(const Path & path, std::string_view s, mode_t mode = 0666, FsSync sync = FsSync::No); -static inline void writeFile(const std::filesystem::path & path, std::string_view s, mode_t mode = 0666) +static inline void writeFile(const std::filesystem::path & path, std::string_view s, mode_t mode = 0666, FsSync sync = FsSync::No) { - return writeFile(path.string(), s, mode); + return writeFile(path.string(), s, mode, sync); } -void writeFile(const Path & path, Source & source, mode_t mode = 0666, bool sync = false); +void writeFile(const Path & path, Source & source, mode_t mode = 0666, FsSync sync = FsSync::No); -static inline void writeFile(const std::filesystem::path & path, Source & source, mode_t mode = 0666, bool sync = false) +static inline void writeFile(const std::filesystem::path & path, Source & source, mode_t mode = 0666, FsSync sync = FsSync::No) { return writeFile(path.string(), source, mode, sync); } -void writeFile(AutoCloseFD & fd, const Path & origPath, std::string_view s, mode_t mode = 0666); - -/** - * Write a string to a file and flush the file and its parents direcotry to disk. - */ -void writeFileAndSync(const Path & path, std::string_view s, mode_t mode = 0666); +void writeFile(AutoCloseFD & fd, const Path & origPath, std::string_view s, mode_t mode = 0666, FsSync sync = FsSync::No); /** * Flush a path's parent directory to disk. From 9af4c267c6cf89ccd27c4f11a32b15533d20a20e Mon Sep 17 00:00:00 2001 From: Eelco Dolstra Date: Thu, 12 Jun 2025 12:30:32 +0200 Subject: [PATCH 182/218] Chown structured attr files safely --- src/libstore/unix/build/derivation-builder.cc | 33 ++++++++++++------- 1 file changed, 22 insertions(+), 11 deletions(-) diff --git a/src/libstore/unix/build/derivation-builder.cc b/src/libstore/unix/build/derivation-builder.cc index 36405a8a2ff..e2a148aeb17 100644 --- a/src/libstore/unix/build/derivation-builder.cc +++ b/src/libstore/unix/build/derivation-builder.cc @@ -316,6 +316,13 @@ class DerivationBuilderImpl : public DerivationBuilder, public DerivationBuilder */ void chownToBuilder(int fd, const Path & path); + /** + * Create a file in `tmpDir` owned by the builder. + */ + void writeBuilderFile( + const std::string & name, + std::string_view contents); + /** * Run the builder's process. */ @@ -1069,16 +1076,10 @@ void DerivationBuilderImpl::initEnv() } else { auto hash = hashString(HashAlgorithm::SHA256, i.first); std::string fn = ".attr-" + hash.to_string(HashFormat::Nix32, false); - Path p = tmpDir + "/" + fn; - AutoCloseFD passAsFileFd{openat(tmpDirFd.get(), fn.c_str(), O_WRONLY | O_TRUNC | O_CREAT | O_CLOEXEC | O_EXCL | O_NOFOLLOW, 0666)}; - if (!passAsFileFd) - throw SysError("opening `passAsFile` file in the sandbox '%1%'", p); - writeFile(passAsFileFd, p, rewriteStrings(i.second, inputRewrites)); - chownToBuilder(passAsFileFd.get(), p); + writeBuilderFile(fn, rewriteStrings(i.second, inputRewrites)); env[i.first + "Path"] = tmpDirInSandbox() + "/" + fn; } } - } /* For convenience, set an environment pointing to the top build @@ -1153,11 +1154,9 @@ void DerivationBuilderImpl::writeStructuredAttrs() auto jsonSh = StructuredAttrs::writeShell(json); - writeFile(tmpDir + "/.attrs.sh", rewriteStrings(jsonSh, inputRewrites)); - chownToBuilder(tmpDir + "/.attrs.sh"); + writeBuilderFile(".attrs.sh", rewriteStrings(jsonSh, inputRewrites)); env["NIX_ATTRS_SH_FILE"] = tmpDirInSandbox() + "/.attrs.sh"; - writeFile(tmpDir + "/.attrs.json", rewriteStrings(json.dump(), inputRewrites)); - chownToBuilder(tmpDir + "/.attrs.json"); + writeBuilderFile(".attrs.json", rewriteStrings(json.dump(), inputRewrites)); env["NIX_ATTRS_JSON_FILE"] = tmpDirInSandbox() + "/.attrs.json"; } } @@ -1285,6 +1284,18 @@ void DerivationBuilderImpl::chownToBuilder(int fd, const Path & path) throw SysError("cannot change ownership of file '%1%'", path); } +void DerivationBuilderImpl::writeBuilderFile( + const std::string & name, + std::string_view contents) +{ + auto path = std::filesystem::path(tmpDir) / name; + AutoCloseFD fd{openat(tmpDirFd.get(), name.c_str(), O_WRONLY | O_TRUNC | O_CREAT | O_CLOEXEC | O_EXCL | O_NOFOLLOW, 0666)}; + if (!fd) + throw SysError("creating file %s", path); + writeFile(fd, path, contents); + chownToBuilder(fd.get(), path); +} + void DerivationBuilderImpl::runChild() { /* Warning: in the child we should absolutely not make any SQLite From 88b7db1ba455926868dd61f5ea39e454e5fb0433 Mon Sep 17 00:00:00 2001 From: eldritch horrors Date: Sun, 30 Mar 2025 16:45:34 +0200 Subject: [PATCH 183/218] libstore: Don't default build-dir to temp-dir, store setting If a build directory is accessible to other users it is possible to smuggle data in and out of build directories. Usually this is only a build purity problem, but in combination with other issues it can be used to break out of a build sandbox. to prevent this we default to using a subdirectory of nixStateDir (which is more restrictive). (cherry picked from pennae Lix commit 55b416f6897fb0d8a9315a530a9b7f0914458ded) (store setting done by roberth) --- doc/manual/rl-next/build-dir-mandatory.md | 9 +++++ misc/systemd/nix-daemon.conf.in | 3 +- src/libstore/globals.cc | 1 + src/libstore/include/nix/store/globals.hh | 9 +---- src/libstore/include/nix/store/local-store.hh | 33 ++++++++++++++++++- src/libstore/local-store.cc | 12 +++++++ src/libstore/unix/build/derivation-builder.cc | 6 +++- .../build-remote-trustless-should-fail-0.sh | 1 - tests/functional/build-remote-trustless.sh | 1 - tests/functional/build-remote.sh | 1 - tests/functional/supplementary-groups.sh | 1 - 11 files changed, 62 insertions(+), 15 deletions(-) create mode 100644 doc/manual/rl-next/build-dir-mandatory.md diff --git a/doc/manual/rl-next/build-dir-mandatory.md b/doc/manual/rl-next/build-dir-mandatory.md new file mode 100644 index 00000000000..cb45a4315f1 --- /dev/null +++ b/doc/manual/rl-next/build-dir-mandatory.md @@ -0,0 +1,9 @@ +--- +synopsis: "`build-dir` no longer defaults to `$TMPDIR`" +--- + +The directory in which temporary build directories are created no longer defaults +to `TMPDIR` or `/tmp`, to avoid builders making their directories +world-accessible. This behavior allowed escaping the build sandbox and can +cause build impurities even when not used maliciously. We now default to `builds` +in `NIX_STATE_DIR` (which is `/nix/var/nix/builds` in the default configuration). diff --git a/misc/systemd/nix-daemon.conf.in b/misc/systemd/nix-daemon.conf.in index e7b264234ab..a0ddc401918 100644 --- a/misc/systemd/nix-daemon.conf.in +++ b/misc/systemd/nix-daemon.conf.in @@ -1 +1,2 @@ -d @localstatedir@/nix/daemon-socket 0755 root root - - +d @localstatedir@/nix/daemon-socket 0755 root root - - +d @localstatedir@/nix/builds 0755 root root 7d - diff --git a/src/libstore/globals.cc b/src/libstore/globals.cc index de512834783..721491defa8 100644 --- a/src/libstore/globals.cc +++ b/src/libstore/globals.cc @@ -6,6 +6,7 @@ #include "nix/util/abstract-setting-to-json.hh" #include "nix/util/compute-levels.hh" #include "nix/util/signals.hh" +#include "nix/util/types.hh" #include #include diff --git a/src/libstore/include/nix/store/globals.hh b/src/libstore/include/nix/store/globals.hh index b5157b4f46e..0ac689b55c1 100644 --- a/src/libstore/include/nix/store/globals.hh +++ b/src/libstore/include/nix/store/globals.hh @@ -697,14 +697,7 @@ public: Setting> buildDir{this, std::nullopt, "build-dir", R"( - The directory on the host, in which derivations' temporary build directories are created. - - If not set, Nix uses the system temporary directory indicated by the `TMPDIR` environment variable. - Note that builds are often performed by the Nix daemon, so its `TMPDIR` is used, and not that of the Nix command line interface. - - This is also the location where [`--keep-failed`](@docroot@/command-ref/opt-common.md#opt-keep-failed) leaves its files. - - If Nix runs without sandbox, or if the platform does not support sandboxing with bind mounts (e.g. macOS), then the [`builder`](@docroot@/language/derivations.md#attr-builder)'s environment contains this directory instead of the virtual location [`sandbox-build-dir`](#conf-sandbox-build-dir). + Override the `build-dir` store setting for all stores that have this setting. )"}; Setting allowedImpureHostPrefixes{this, {}, "allowed-impure-host-deps", diff --git a/src/libstore/include/nix/store/local-store.hh b/src/libstore/include/nix/store/local-store.hh index 9a118fcc517..e52d51f752f 100644 --- a/src/libstore/include/nix/store/local-store.hh +++ b/src/libstore/include/nix/store/local-store.hh @@ -34,7 +34,38 @@ struct OptimiseStats uint64_t bytesFreed = 0; }; -struct LocalStoreConfig : std::enable_shared_from_this, virtual LocalFSStoreConfig +struct LocalBuildStoreConfig : virtual LocalFSStoreConfig { + +private: + /** + Input for computing the build directory. See `getBuildDir()`. + */ + Setting> buildDir{this, std::nullopt, "build-dir", + R"( + The directory on the host, in which derivations' temporary build directories are created. + + If not set, Nix will use the `builds` subdirectory of its configured state directory. + + Note that builds are often performed by the Nix daemon, so its `build-dir` applies. + + Nix will create this directory automatically with suitable permissions if it does not exist. + Otherwise its permissions must allow all users to traverse the directory (i.e. it must have `o+x` set, in unix parlance) for non-sandboxed builds to work correctly. + + This is also the location where [`--keep-failed`](@docroot@/command-ref/opt-common.md#opt-keep-failed) leaves its files. + + If Nix runs without sandbox, or if the platform does not support sandboxing with bind mounts (e.g. macOS), then the [`builder`](@docroot@/language/derivations.md#attr-builder)'s environment will contain this directory, instead of the virtual location [`sandbox-build-dir`](#conf-sandbox-build-dir). + + > **Warning** + > + > `build-dir` must not be set to a world-writable directory. + > Placing temporary build directories in a world-writable place allows other users to access or modify build data that is currently in use. + > This alone is merely an impurity, but combined with another factor this has allowed malicious derivations to escape the build sandbox. + )"}; +public: + Path getBuildDir() const; +}; + +struct LocalStoreConfig : std::enable_shared_from_this, virtual LocalFSStoreConfig, virtual LocalBuildStoreConfig { using LocalFSStoreConfig::LocalFSStoreConfig; diff --git a/src/libstore/local-store.cc b/src/libstore/local-store.cc index 9e132426251..e25a802ec63 100644 --- a/src/libstore/local-store.cc +++ b/src/libstore/local-store.cc @@ -77,6 +77,18 @@ std::string LocalStoreConfig::doc() ; } +Path LocalBuildStoreConfig::getBuildDir() const +{ + if (settings.buildDir.get().has_value()) { + return *settings.buildDir.get(); + } + if (buildDir.get().has_value()) { + return *buildDir.get(); + } + + return stateDir.get() + "/builds"; +} + ref LocalStore::Config::openStore() const { return make_ref(ref{shared_from_this()}); diff --git a/src/libstore/unix/build/derivation-builder.cc b/src/libstore/unix/build/derivation-builder.cc index e2a148aeb17..a036c95f17b 100644 --- a/src/libstore/unix/build/derivation-builder.cc +++ b/src/libstore/unix/build/derivation-builder.cc @@ -725,9 +725,13 @@ void DerivationBuilderImpl::startBuilder() throw BuildError(msg); } + auto buildDir = getLocalStore(store).config->getBuildDir(); + + createDirs(buildDir); + /* Create a temporary directory where the build will take place. */ - topTmpDir = createTempDir(settings.buildDir.get().value_or(""), "nix-build-" + std::string(drvPath.name()), 0700); + topTmpDir = createTempDir(buildDir, "nix-build-" + std::string(drvPath.name()), 0700); setBuildTmpDir(); assert(!tmpDir.empty()); diff --git a/tests/functional/build-remote-trustless-should-fail-0.sh b/tests/functional/build-remote-trustless-should-fail-0.sh index 3401de1b073..e79527d7290 100755 --- a/tests/functional/build-remote-trustless-should-fail-0.sh +++ b/tests/functional/build-remote-trustless-should-fail-0.sh @@ -12,7 +12,6 @@ requiresUnprivilegedUserNamespaces [[ $busybox =~ busybox ]] || skipTest "no busybox" unset NIX_STORE_DIR -unset NIX_STATE_DIR # We first build a dependency of the derivation we eventually want to # build. diff --git a/tests/functional/build-remote-trustless.sh b/tests/functional/build-remote-trustless.sh index 9f91a91a969..6014b57bb1e 100644 --- a/tests/functional/build-remote-trustless.sh +++ b/tests/functional/build-remote-trustless.sh @@ -9,7 +9,6 @@ requiresUnprivilegedUserNamespaces [[ "$busybox" =~ busybox ]] || skipTest "no busybox" unset NIX_STORE_DIR -unset NIX_STATE_DIR remoteDir=$TEST_ROOT/remote diff --git a/tests/functional/build-remote.sh b/tests/functional/build-remote.sh index 765cd71b420..f396bc72e8f 100644 --- a/tests/functional/build-remote.sh +++ b/tests/functional/build-remote.sh @@ -8,7 +8,6 @@ requiresUnprivilegedUserNamespaces # Avoid store dir being inside sandbox build-dir unset NIX_STORE_DIR -unset NIX_STATE_DIR function join_by { local d=$1; shift; echo -n "$1"; shift; printf "%s" "${@/#/$d}"; } diff --git a/tests/functional/supplementary-groups.sh b/tests/functional/supplementary-groups.sh index 400333f7d4b..a667d3e998c 100755 --- a/tests/functional/supplementary-groups.sh +++ b/tests/functional/supplementary-groups.sh @@ -14,7 +14,6 @@ execUnshare < Date: Thu, 12 Jun 2025 11:04:07 +0200 Subject: [PATCH 184/218] Disallow the build directory having world-writable parents --- src/libstore/unix/build/derivation-builder.cc | 15 +++++++++++++++ tests/nixos/chroot-store.nix | 4 ++++ 2 files changed, 19 insertions(+) diff --git a/src/libstore/unix/build/derivation-builder.cc b/src/libstore/unix/build/derivation-builder.cc index a036c95f17b..b089a016915 100644 --- a/src/libstore/unix/build/derivation-builder.cc +++ b/src/libstore/unix/build/derivation-builder.cc @@ -698,6 +698,18 @@ static void handleChildException(bool sendException) } } +static bool checkNotWorldWritable(std::filesystem::path path) +{ + while (true) { + auto st = lstat(path); + if (st.st_mode & S_IWOTH) + return false; + if (path == path.parent_path()) break; + path = path.parent_path(); + } + return true; +} + void DerivationBuilderImpl::startBuilder() { /* Make sure that no other processes are executing under the @@ -729,6 +741,9 @@ void DerivationBuilderImpl::startBuilder() createDirs(buildDir); + if (buildUser && !checkNotWorldWritable(buildDir)) + throw Error("Path %s or a parent directory is world-writable or a symlink. That's not allowed for security.", buildDir); + /* Create a temporary directory where the build will take place. */ topTmpDir = createTempDir(buildDir, "nix-build-" + std::string(drvPath.name()), 0700); diff --git a/tests/nixos/chroot-store.nix b/tests/nixos/chroot-store.nix index f89a20bc4d5..0a4fff99222 100644 --- a/tests/nixos/chroot-store.nix +++ b/tests/nixos/chroot-store.nix @@ -41,5 +41,9 @@ in # Test that /nix/store is available via an overlayfs mount. machine.succeed("nix shell --store /tmp/nix ${pkgA} --command cowsay foo >&2") + + # Building in /tmp should fail for security reasons. + err = machine.fail("nix build --offline --store /tmp/nix --expr 'builtins.derivation { name = \"foo\"; system = \"x86_64-linux\"; builder = \"/foo\"; }' 2>&1") + assert "is world-writable" in err ''; } From 2e2fe4cb07c81f4af4798d1f67e0f6b8d6263e44 Mon Sep 17 00:00:00 2001 From: Eelco Dolstra Date: Thu, 12 Jun 2025 11:12:23 +0200 Subject: [PATCH 185/218] Cleanup --- src/libstore/globals.cc | 1 - src/libstore/include/nix/store/local-store.hh | 3 ++- src/libstore/local-store.cc | 14 ++++++-------- 3 files changed, 8 insertions(+), 10 deletions(-) diff --git a/src/libstore/globals.cc b/src/libstore/globals.cc index 721491defa8..de512834783 100644 --- a/src/libstore/globals.cc +++ b/src/libstore/globals.cc @@ -6,7 +6,6 @@ #include "nix/util/abstract-setting-to-json.hh" #include "nix/util/compute-levels.hh" #include "nix/util/signals.hh" -#include "nix/util/types.hh" #include #include diff --git a/src/libstore/include/nix/store/local-store.hh b/src/libstore/include/nix/store/local-store.hh index e52d51f752f..fd7e6fc3607 100644 --- a/src/libstore/include/nix/store/local-store.hh +++ b/src/libstore/include/nix/store/local-store.hh @@ -34,7 +34,8 @@ struct OptimiseStats uint64_t bytesFreed = 0; }; -struct LocalBuildStoreConfig : virtual LocalFSStoreConfig { +struct LocalBuildStoreConfig : virtual LocalFSStoreConfig +{ private: /** diff --git a/src/libstore/local-store.cc b/src/libstore/local-store.cc index e25a802ec63..0d2d96e6119 100644 --- a/src/libstore/local-store.cc +++ b/src/libstore/local-store.cc @@ -79,14 +79,12 @@ std::string LocalStoreConfig::doc() Path LocalBuildStoreConfig::getBuildDir() const { - if (settings.buildDir.get().has_value()) { - return *settings.buildDir.get(); - } - if (buildDir.get().has_value()) { - return *buildDir.get(); - } - - return stateDir.get() + "/builds"; + return + settings.buildDir.get().has_value() + ? *settings.buildDir.get() + : buildDir.get().has_value() + ? *buildDir.get() + : stateDir.get() + "/builds"; } ref LocalStore::Config::openStore() const From 37685b1c9c936da8644822d6ae60242b8ffcee8e Mon Sep 17 00:00:00 2001 From: Eelco Dolstra Date: Thu, 19 Jun 2025 13:26:10 +0200 Subject: [PATCH 186/218] Fix Darwin test failure in repl.sh MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Fixes error: … while processing sandbox path '/private/tmp/nix-shell.0MDgyx/nix-test/ca/repl/store/nix/var/nix/builds/nix-build-simple.drv-65916-3910734210' (/private/tmp/nix-shell.0MDgyx/nix-test/ca/repl/store) error: 'nix' is too short to be a valid store path which happened because we were now putting the build directory underneath the store directory. --- src/libstore/unix/build/derivation-builder.cc | 2 +- tests/functional/repl.sh | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/src/libstore/unix/build/derivation-builder.cc b/src/libstore/unix/build/derivation-builder.cc index b089a016915..fd62aa664a4 100644 --- a/src/libstore/unix/build/derivation-builder.cc +++ b/src/libstore/unix/build/derivation-builder.cc @@ -922,7 +922,7 @@ DerivationBuilderImpl::PathsInChroot DerivationBuilderImpl::getPathsInSandbox() store.computeFSClosure(store.toStorePath(i.second.source).first, closure); } catch (InvalidPath & e) { } catch (Error & e) { - e.addTrace({}, "while processing 'sandbox-paths'"); + e.addTrace({}, "while processing sandbox path '%s'", i.second.source); throw; } for (auto & i : closure) { diff --git a/tests/functional/repl.sh b/tests/functional/repl.sh index 6db9e2d3d63..bfe18c9e586 100755 --- a/tests/functional/repl.sh +++ b/tests/functional/repl.sh @@ -67,7 +67,7 @@ testRepl () { # Simple test, try building a drv testRepl # Same thing (kind-of), but with a remote store. -testRepl --store "$TEST_ROOT/store?real=$NIX_STORE_DIR" +testRepl --store "$TEST_ROOT/other-root?real=$NIX_STORE_DIR" # Remove ANSI escape sequences. They can prevent grep from finding a match. stripColors () { From df21f249877cead560782f5ccdd1796bc5682349 Mon Sep 17 00:00:00 2001 From: Egor Konovalov <73017521+egorkonovalov@users.noreply.github.com> Date: Mon, 23 Jun 2025 11:26:59 +0200 Subject: [PATCH 187/218] Fix link Remove extra `realise` --- doc/manual/source/store/derivation/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/manual/source/store/derivation/index.md b/doc/manual/source/store/derivation/index.md index 5ebbfa7c780..1687ad8c04d 100644 --- a/doc/manual/source/store/derivation/index.md +++ b/doc/manual/source/store/derivation/index.md @@ -173,7 +173,7 @@ There are two types of placeholder, corresponding to the two cases where this pr > **Explanation** > -> In general, we need to realise [realise] a [store object] in order to be sure to have a store object for it. +> In general, we need to [realise] a [store object] in order to be sure to have a store object for it. > But for these two cases this is either impossible or impractical: > > - In the output case this is impossible: From 55d12dfc5db53f7b2f90fef235e7115c9c17e8b2 Mon Sep 17 00:00:00 2001 From: Sergei Zimmerman Date: Wed, 25 Jun 2025 00:07:58 +0300 Subject: [PATCH 188/218] libstore-tests: Don't leak memory in tests We shouldn't leak memory in unit tests in order to make enabling ASAN easier. --- .../include/nix/store/tests/nix_api_store.hh | 8 ++++++-- src/libstore-tests/nix_api_store.cc | 9 +++++++-- 2 files changed, 13 insertions(+), 4 deletions(-) diff --git a/src/libstore-test-support/include/nix/store/tests/nix_api_store.hh b/src/libstore-test-support/include/nix/store/tests/nix_api_store.hh index 63f80cf91eb..e51be3dab5a 100644 --- a/src/libstore-test-support/include/nix/store/tests/nix_api_store.hh +++ b/src/libstore-test-support/include/nix/store/tests/nix_api_store.hh @@ -34,6 +34,8 @@ public: Store * store; std::string nixDir; std::string nixStoreDir; + std::string nixStateDir; + std::string nixLogDir; protected: void init_local_store() @@ -53,11 +55,13 @@ protected: #endif nixStoreDir = nixDir + "/my_nix_store"; + nixStateDir = nixDir + "/my_state"; + nixLogDir = nixDir + "/my_log"; // Options documented in `nix help-stores` const char * p1[] = {"store", nixStoreDir.c_str()}; - const char * p2[] = {"state", (new std::string(nixDir + "/my_state"))->c_str()}; - const char * p3[] = {"log", (new std::string(nixDir + "/my_log"))->c_str()}; + const char * p2[] = {"state", nixStateDir.c_str()}; + const char * p3[] = {"log", nixLogDir.c_str()}; const char ** params[] = {p1, p2, p3, nullptr}; diff --git a/src/libstore-tests/nix_api_store.cc b/src/libstore-tests/nix_api_store.cc index 4eb95360a6a..3d9f7908b3f 100644 --- a/src/libstore-tests/nix_api_store.cc +++ b/src/libstore-tests/nix_api_store.cc @@ -71,17 +71,21 @@ TEST_F(nix_api_store_test, ReturnsValidStorePath) ASSERT_NE(result, nullptr); ASSERT_STREQ("name", result->path.name().data()); ASSERT_STREQ(PATH_SUFFIX.substr(1).c_str(), result->path.to_string().data()); + nix_store_path_free(result); } TEST_F(nix_api_store_test, SetsLastErrCodeToNixOk) { - nix_store_parse_path(ctx, store, (nixStoreDir + PATH_SUFFIX).c_str()); + StorePath * path = nix_store_parse_path(ctx, store, (nixStoreDir + PATH_SUFFIX).c_str()); ASSERT_EQ(ctx->last_err_code, NIX_OK); + nix_store_path_free(path); } TEST_F(nix_api_store_test, DoesNotCrashWhenContextIsNull) { - ASSERT_NO_THROW(nix_store_parse_path(ctx, store, (nixStoreDir + PATH_SUFFIX).c_str())); + StorePath * path = nullptr; + ASSERT_NO_THROW(path = nix_store_parse_path(ctx, store, (nixStoreDir + PATH_SUFFIX).c_str())); + nix_store_path_free(path); } TEST_F(nix_api_store_test, get_version) @@ -119,6 +123,7 @@ TEST_F(nix_api_store_test, nix_store_is_valid_path_not_in_store) { StorePath * path = nix_store_parse_path(ctx, store, (nixStoreDir + PATH_SUFFIX).c_str()); ASSERT_EQ(false, nix_store_is_valid_path(ctx, store, path)); + nix_store_path_free(path); } TEST_F(nix_api_store_test, nix_store_real_path) From 7c0ea143d81559993d3f3005b97b925e28dee39c Mon Sep 17 00:00:00 2001 From: Eelco Dolstra Date: Thu, 12 Jun 2025 20:19:19 +0200 Subject: [PATCH 189/218] Fix deep overrides An override like inputs.foo.inputs.bar.inputs.nixpkgs.follows = "nixpkgs"; implicitly set `inputs.foo.inputs.bar` to `flake:bar`, which led to an unexpected error like error: cannot find flake 'flake:bar' in the flake registries We now no longer create a parent override (like for `foo.bar` in the example above) if it doesn't set an explicit ref or follows attribute. We only recursively apply its child overrides. Fixes https://github.com/NixOS/nix/issues/8325, https://github.com/DeterminateSystems/nix-src/issues/95, https://github.com/NixOS/nix/issues/12083, https://github.com/NixOS/nix/issues/5790. --- src/libflake/flake.cc | 35 ++++++++++++++++++++--------------- 1 file changed, 20 insertions(+), 15 deletions(-) diff --git a/src/libflake/flake.cc b/src/libflake/flake.cc index 3a9bdf43a82..2b73dbf0b20 100644 --- a/src/libflake/flake.cc +++ b/src/libflake/flake.cc @@ -101,7 +101,6 @@ static void parseFlakeInputAttr( static FlakeInput parseFlakeInput( EvalState & state, - std::string_view inputName, Value * value, const PosIdx pos, const InputAttrPath & lockRootAttrPath, @@ -171,9 +170,6 @@ static FlakeInput parseFlakeInput( input.ref = parseFlakeRef(state.fetchSettings, *url, {}, true, input.isFlake, true); } - if (!input.follows && !input.ref) - input.ref = FlakeRef::fromAttrs(state.fetchSettings, {{"type", "indirect"}, {"id", std::string(inputName)}}); - return input; } @@ -201,7 +197,6 @@ static std::pair, fetchers::Attrs> parseFlakeInput } else { inputs.emplace(inputName, parseFlakeInput(state, - inputName, inputAttr.value, inputAttr.pos, lockRootAttrPath, @@ -483,18 +478,27 @@ LockedFlake lockFlake( /* Get the overrides (i.e. attributes of the form 'inputs.nixops.inputs.nixpkgs.url = ...'). */ - for (auto & [id, input] : flakeInputs) { + std::function addOverrides; + addOverrides = [&](const FlakeInput & input, const InputAttrPath & prefix) + { for (auto & [idOverride, inputOverride] : input.overrides) { - auto inputAttrPath(inputAttrPathPrefix); - inputAttrPath.push_back(id); + auto inputAttrPath(prefix); inputAttrPath.push_back(idOverride); - overrides.emplace(inputAttrPath, - OverrideTarget { - .input = inputOverride, - .sourcePath = sourcePath, - .parentInputAttrPath = inputAttrPathPrefix - }); + if (inputOverride.ref || inputOverride.follows) + overrides.emplace(inputAttrPath, + OverrideTarget { + .input = inputOverride, + .sourcePath = sourcePath, + .parentInputAttrPath = inputAttrPathPrefix + }); + addOverrides(inputOverride, inputAttrPath); } + }; + + for (auto & [id, input] : flakeInputs) { + auto inputAttrPath(inputAttrPathPrefix); + inputAttrPath.push_back(id); + addOverrides(input, inputAttrPath); } /* Check whether this input has overrides for a @@ -550,7 +554,8 @@ LockedFlake lockFlake( continue; } - assert(input.ref); + if (!input.ref) + input.ref = FlakeRef::fromAttrs(state.fetchSettings, {{"type", "indirect"}, {"id", std::string(id)}}); auto overriddenParentPath = input.ref->input.isRelative() From 637c4f3ad75a41ff0f0a34301fdc76451e50f800 Mon Sep 17 00:00:00 2001 From: Eelco Dolstra Date: Thu, 12 Jun 2025 20:33:28 +0200 Subject: [PATCH 190/218] Add tests for deep overrides Taken from https://github.com/NixOS/nix/pull/6621. Co-authored-by: Sebastian Ullrich --- tests/functional/flakes/follow-paths.sh | 60 +++++++++++++++++++++++++ 1 file changed, 60 insertions(+) diff --git a/tests/functional/flakes/follow-paths.sh b/tests/functional/flakes/follow-paths.sh index a71d4c6d706..abc09dfc2e0 100755 --- a/tests/functional/flakes/follow-paths.sh +++ b/tests/functional/flakes/follow-paths.sh @@ -359,3 +359,63 @@ rm "$flakeFollowsCustomUrlA"/flake.lock json=$(nix flake metadata "$flakeFollowsCustomUrlA" --override-input B/C "$flakeFollowsCustomUrlD" --json) echo "$json" | jq .locks.nodes.C.original [[ $(echo "$json" | jq -r .locks.nodes.C.original.path) = './flakeC' ]] + +# Test deep overrides, e.g. `inputs.B.inputs.C.inputs.D.follows = ...`. + +cat < $flakeFollowsD/flake.nix +{ outputs = _: {}; } +EOF +cat < $flakeFollowsC/flake.nix +{ + inputs.D.url = "path:nosuchflake"; + outputs = _: {}; +} +EOF +cat < $flakeFollowsB/flake.nix +{ + inputs.C.url = "path:$flakeFollowsC"; + outputs = _: {}; +} +EOF +cat < $flakeFollowsA/flake.nix +{ + inputs.B.url = "path:$flakeFollowsB"; + inputs.D.url = "path:$flakeFollowsD"; + inputs.B.inputs.C.inputs.D.follows = "D"; + outputs = _: {}; +} +EOF + +nix flake lock $flakeFollowsA + +[[ $(jq -c .nodes.C.inputs.D $flakeFollowsA/flake.lock) = '["D"]' ]] + +# Test overlapping flake follows: B has D follow C/D, while A has B/C follow C + +cat < $flakeFollowsC/flake.nix +{ + inputs.D.url = "path:$flakeFollowsD"; + outputs = _: {}; +} +EOF +cat < $flakeFollowsB/flake.nix +{ + inputs.C.url = "path:nosuchflake"; + inputs.D.url = "path:nosuchflake"; + inputs.D.follows = "C/D"; + outputs = _: {}; +} +EOF +cat < $flakeFollowsA/flake.nix +{ + inputs.B.url = "path:$flakeFollowsB"; + inputs.C.url = "path:$flakeFollowsC"; + inputs.B.inputs.C.follows = "C"; + outputs = _: {}; +} +EOF + +# bug was not triggered without recreating the lockfile +nix flake lock $flakeFollowsA --recreate-lock-file + +[[ $(jq -c .nodes.B.inputs.D $flakeFollowsA/flake.lock) = '["B","C","D"]' ]] From b415faceca9ea32823ec3701580dc3c1d2530d82 Mon Sep 17 00:00:00 2001 From: Eelco Dolstra Date: Thu, 12 Jun 2025 20:38:51 +0200 Subject: [PATCH 191/218] Don't allow flake inputs to have both a flakeref and a follows Having both doesn't make sense so it's best to disallow it. If this causes issues we could turn into a warning. --- src/libflake/flake.cc | 3 +++ tests/functional/flakes/follow-paths.sh | 13 ++++++++++++- 2 files changed, 15 insertions(+), 1 deletion(-) diff --git a/src/libflake/flake.cc b/src/libflake/flake.cc index 2b73dbf0b20..e59649fecb9 100644 --- a/src/libflake/flake.cc +++ b/src/libflake/flake.cc @@ -170,6 +170,9 @@ static FlakeInput parseFlakeInput( input.ref = parseFlakeRef(state.fetchSettings, *url, {}, true, input.isFlake, true); } + if (input.ref && input.follows) + throw Error("flake input has both a flake reference and a follows attribute, at %s", state.positions[pos]); + return input; } diff --git a/tests/functional/flakes/follow-paths.sh b/tests/functional/flakes/follow-paths.sh index abc09dfc2e0..cf27681cbd5 100755 --- a/tests/functional/flakes/follow-paths.sh +++ b/tests/functional/flakes/follow-paths.sh @@ -401,7 +401,6 @@ EOF cat < $flakeFollowsB/flake.nix { inputs.C.url = "path:nosuchflake"; - inputs.D.url = "path:nosuchflake"; inputs.D.follows = "C/D"; outputs = _: {}; } @@ -419,3 +418,15 @@ EOF nix flake lock $flakeFollowsA --recreate-lock-file [[ $(jq -c .nodes.B.inputs.D $flakeFollowsA/flake.lock) = '["B","C","D"]' ]] + +# Check that you can't have both a flakeref and a follows attribute on an input. +cat < $flakeFollowsB/flake.nix +{ + inputs.C.url = "path:nosuchflake"; + inputs.D.url = "path:nosuchflake"; + inputs.D.follows = "C/D"; + outputs = _: {}; +} +EOF + +expectStderr 1 nix flake lock $flakeFollowsA --recreate-lock-file | grepQuiet "flake input has both a flake reference and a follows attribute" From eaced1e0d2a041668a07f34ea2c94c78466d08aa Mon Sep 17 00:00:00 2001 From: Eelco Dolstra Date: Thu, 26 Jun 2025 17:04:34 +0200 Subject: [PATCH 192/218] Move FlakeCommand into a header, allow separate registration of subcommands This allows us to start splitting up src/nix/flake.cc. --- .../include/nix/cmd/common-eval-args.hh | 3 +- src/nix/flake-command.hh | 27 ++++++ src/nix/flake.cc | 90 ++++++++----------- 3 files changed, 67 insertions(+), 53 deletions(-) create mode 100644 src/nix/flake-command.hh diff --git a/src/libcmd/include/nix/cmd/common-eval-args.hh b/src/libcmd/include/nix/cmd/common-eval-args.hh index 71634042555..88ede1ed7e7 100644 --- a/src/libcmd/include/nix/cmd/common-eval-args.hh +++ b/src/libcmd/include/nix/cmd/common-eval-args.hh @@ -5,6 +5,7 @@ #include "nix/util/canon-path.hh" #include "nix/main/common-args.hh" #include "nix/expr/search-path.hh" +#include "nix/expr/eval-settings.hh" #include @@ -15,10 +16,8 @@ class Store; namespace fetchers { struct Settings; } class EvalState; -struct EvalSettings; struct CompatibilitySettings; class Bindings; -struct SourcePath; namespace flake { struct Settings; } diff --git a/src/nix/flake-command.hh b/src/nix/flake-command.hh new file mode 100644 index 00000000000..36dfe44c632 --- /dev/null +++ b/src/nix/flake-command.hh @@ -0,0 +1,27 @@ +#pragma once + +#include "nix/cmd/command.hh" +#include "nix/cmd/installable-flake.hh" +#include "nix/flake/flake.hh" + +namespace nix { + +using namespace nix::flake; + +class FlakeCommand : virtual Args, public MixFlakeOptions +{ +protected: + std::string flakeUrl = "."; + +public: + + FlakeCommand(); + + FlakeRef getFlakeRef(); + + LockedFlake lockFlake(); + + std::vector getFlakeRefsForCompletion() override; +}; + +} diff --git a/src/nix/flake.cc b/src/nix/flake.cc index 13f7363fcfd..1d20add02ce 100644 --- a/src/nix/flake.cc +++ b/src/nix/flake.cc @@ -1,11 +1,9 @@ -#include "nix/cmd/command.hh" -#include "nix/cmd/installable-flake.hh" +#include "flake-command.hh" #include "nix/main/common-args.hh" #include "nix/main/shared.hh" #include "nix/expr/eval.hh" #include "nix/expr/eval-inline.hh" #include "nix/expr/eval-settings.hh" -#include "nix/flake/flake.hh" #include "nix/expr/get-drvs.hh" #include "nix/util/signals.hh" #include "nix/store/store-open.hh" @@ -33,43 +31,36 @@ using namespace nix::flake; using json = nlohmann::json; struct CmdFlakeUpdate; -class FlakeCommand : virtual Args, public MixFlakeOptions -{ -protected: - std::string flakeUrl = "."; - -public: - FlakeCommand() - { - expectArgs({ - .label = "flake-url", - .optional = true, - .handler = {&flakeUrl}, - .completer = {[&](AddCompletions & completions, size_t, std::string_view prefix) { - completeFlakeRef(completions, getStore(), prefix); - }} - }); - } +FlakeCommand::FlakeCommand() +{ + expectArgs({ + .label = "flake-url", + .optional = true, + .handler = {&flakeUrl}, + .completer = {[&](AddCompletions & completions, size_t, std::string_view prefix) { + completeFlakeRef(completions, getStore(), prefix); + }} + }); +} - FlakeRef getFlakeRef() - { - return parseFlakeRef(fetchSettings, flakeUrl, std::filesystem::current_path().string()); //FIXME - } +FlakeRef FlakeCommand::getFlakeRef() +{ + return parseFlakeRef(fetchSettings, flakeUrl, std::filesystem::current_path().string()); //FIXME +} - LockedFlake lockFlake() - { - return flake::lockFlake(flakeSettings, *getEvalState(), getFlakeRef(), lockFlags); - } +LockedFlake FlakeCommand::lockFlake() +{ + return flake::lockFlake(flakeSettings, *getEvalState(), getFlakeRef(), lockFlags); +} - std::vector getFlakeRefsForCompletion() override - { - return { - // Like getFlakeRef but with expandTilde called first - parseFlakeRef(fetchSettings, expandTilde(flakeUrl), std::filesystem::current_path().string()) - }; - } -}; +std::vector FlakeCommand::getFlakeRefsForCompletion() +{ + return { + // Like getFlakeRef but with expandTilde called first + parseFlakeRef(fetchSettings, expandTilde(flakeUrl), std::filesystem::current_path().string()) + }; +} struct CmdFlakeUpdate : FlakeCommand { @@ -1528,21 +1519,7 @@ struct CmdFlakePrefetch : FlakeCommand, MixJSON struct CmdFlake : NixMultiCommand { CmdFlake() - : NixMultiCommand( - "flake", - { - {"update", []() { return make_ref(); }}, - {"lock", []() { return make_ref(); }}, - {"metadata", []() { return make_ref(); }}, - {"info", []() { return make_ref(); }}, - {"check", []() { return make_ref(); }}, - {"init", []() { return make_ref(); }}, - {"new", []() { return make_ref(); }}, - {"clone", []() { return make_ref(); }}, - {"archive", []() { return make_ref(); }}, - {"show", []() { return make_ref(); }}, - {"prefetch", []() { return make_ref(); }}, - }) + : NixMultiCommand("flake", RegisterCommand::getCommandsFor({"flake"})) { } @@ -1566,3 +1543,14 @@ struct CmdFlake : NixMultiCommand }; static auto rCmdFlake = registerCommand("flake"); +static auto rCmdFlakeArchive = registerCommand2({"flake", "archive"}); +static auto rCmdFlakeCheck = registerCommand2({"flake", "check"}); +static auto rCmdFlakeClone = registerCommand2({"flake", "clone"}); +static auto rCmdFlakeInfo = registerCommand2({"flake", "info"}); +static auto rCmdFlakeInit = registerCommand2({"flake", "init"}); +static auto rCmdFlakeLock = registerCommand2({"flake", "lock"}); +static auto rCmdFlakeMetadata = registerCommand2({"flake", "metadata"}); +static auto rCmdFlakeNew = registerCommand2({"flake", "new"}); +static auto rCmdFlakePrefetch = registerCommand2({"flake", "prefetch"}); +static auto rCmdFlakeShow = registerCommand2({"flake", "show"}); +static auto rCmdFlakeUpdate = registerCommand2({"flake", "update"}); From 8fbc27af46e49265691ca664e5705b043639c7ca Mon Sep 17 00:00:00 2001 From: h0nIg Date: Thu, 26 Jun 2025 23:33:27 +0200 Subject: [PATCH 193/218] enhancements --- docker.nix | 16 +++++++++++----- 1 file changed, 11 insertions(+), 5 deletions(-) diff --git a/docker.nix b/docker.nix index 825ffff4f97..fff9672b2d2 100644 --- a/docker.nix +++ b/docker.nix @@ -176,11 +176,17 @@ let "${k}:x:${toString gid}:${lib.concatStringsSep "," members}"; groupContents = (lib.concatStringsSep "\n" (lib.attrValues (lib.mapAttrs groupToGroup groups))); - nixConfContents = - pkgs.dockerTools.nixConf - { - build-users-group = "nixbld"; - }; + toConf = with pkgs.lib.generators; toKeyValue { + mkKeyValue = mkKeyValueDefault { + mkValueString = v: if lib.isList v then lib.concatStringsSep " " v else mkValueStringDefault { } v; + } " = "; + }; + + nixConfContents = toConf { + sandbox = false; + build-users-group = "nixbld"; + trusted-public-keys = [ "cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=" ]; + }; userHome = if uid == 0 then "/root" else "/home/${uname}"; From ba12adc0f92396297b6c825690f3a3dfa8a9fbd5 Mon Sep 17 00:00:00 2001 From: h0nIg Date: Thu, 26 Jun 2025 23:37:39 +0200 Subject: [PATCH 194/218] format --- docker.nix | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-) diff --git a/docker.nix b/docker.nix index fff9672b2d2..c6e8e478e7e 100644 --- a/docker.nix +++ b/docker.nix @@ -176,11 +176,13 @@ let "${k}:x:${toString gid}:${lib.concatStringsSep "," members}"; groupContents = (lib.concatStringsSep "\n" (lib.attrValues (lib.mapAttrs groupToGroup groups))); - toConf = with pkgs.lib.generators; toKeyValue { - mkKeyValue = mkKeyValueDefault { - mkValueString = v: if lib.isList v then lib.concatStringsSep " " v else mkValueStringDefault { } v; - } " = "; - }; + toConf = + with pkgs.lib.generators; + toKeyValue { + mkKeyValue = mkKeyValueDefault { + mkValueString = v: if lib.isList v then lib.concatStringsSep " " v else mkValueStringDefault { } v; + } " = "; + }; nixConfContents = toConf { sandbox = false; From 75412ebc305761cf5a75d93d1e130cdb86bcfff0 Mon Sep 17 00:00:00 2001 From: Sergei Zimmerman Date: Fri, 27 Jun 2025 23:33:54 +0300 Subject: [PATCH 195/218] libflake: Remove unused maybeParseFlakeRef and maybeParseFlakeRefWithFragment These wrappers don't seem to be used anywhere in and out of tree. Also the declaration in the header has an incorrect function name `maybeParseFlake`. Closes #11948 --- src/libflake/flakeref.cc | 23 ---------------------- src/libflake/include/nix/flake/flakeref.hh | 16 --------------- 2 files changed, 39 deletions(-) diff --git a/src/libflake/flakeref.cc b/src/libflake/flakeref.cc index d56f2858f8d..37b7eff4ccb 100644 --- a/src/libflake/flakeref.cc +++ b/src/libflake/flakeref.cc @@ -57,18 +57,6 @@ FlakeRef parseFlakeRef( return flakeRef; } -std::optional maybeParseFlakeRef( - const fetchers::Settings & fetchSettings, - const std::string & url, - const std::optional & baseDir) -{ - try { - return parseFlakeRef(fetchSettings, url, baseDir); - } catch (Error &) { - return {}; - } -} - static std::pair fromParsedURL( const fetchers::Settings & fetchSettings, ParsedURL && parsedURL, @@ -261,17 +249,6 @@ std::pair parseFlakeRefWithFragment( } } -std::optional> maybeParseFlakeRefWithFragment( - const fetchers::Settings & fetchSettings, - const std::string & url, const std::optional & baseDir) -{ - try { - return parseFlakeRefWithFragment(fetchSettings, url, baseDir); - } catch (Error & e) { - return {}; - } -} - FlakeRef FlakeRef::fromAttrs( const fetchers::Settings & fetchSettings, const fetchers::Attrs & attrs) diff --git a/src/libflake/include/nix/flake/flakeref.hh b/src/libflake/include/nix/flake/flakeref.hh index 6184d2363c4..c0045fcf368 100644 --- a/src/libflake/include/nix/flake/flakeref.hh +++ b/src/libflake/include/nix/flake/flakeref.hh @@ -93,14 +93,6 @@ FlakeRef parseFlakeRef( bool isFlake = true, bool preserveRelativePaths = false); -/** - * @param baseDir Optional [base directory](https://nixos.org/manual/nix/unstable/glossary#gloss-base-directory) - */ -std::optional maybeParseFlake( - const fetchers::Settings & fetchSettings, - const std::string & url, - const std::optional & baseDir = {}); - /** * @param baseDir Optional [base directory](https://nixos.org/manual/nix/unstable/glossary#gloss-base-directory) */ @@ -112,14 +104,6 @@ std::pair parseFlakeRefWithFragment( bool isFlake = true, bool preserveRelativePaths = false); -/** - * @param baseDir Optional [base directory](https://nixos.org/manual/nix/unstable/glossary#gloss-base-directory) - */ -std::optional> maybeParseFlakeRefWithFragment( - const fetchers::Settings & fetchSettings, - const std::string & url, - const std::optional & baseDir = {}); - /** * @param baseDir Optional [base directory](https://nixos.org/manual/nix/unstable/glossary#gloss-base-directory) */ From fa3d7e6f6875758c2c6730b564666cf9629f8461 Mon Sep 17 00:00:00 2001 From: Philipp Otterbein Date: Sun, 15 Jun 2025 21:59:13 +0200 Subject: [PATCH 196/218] libexpr: don't allocate additional set in builtins.listToAttrs --- src/libexpr/primops.cc | 42 ++++++++++++++++++++++++++++++++---------- 1 file changed, 32 insertions(+), 10 deletions(-) diff --git a/src/libexpr/primops.cc b/src/libexpr/primops.cc index 60f44ca62be..8cad87f17f0 100644 --- a/src/libexpr/primops.cc +++ b/src/libexpr/primops.cc @@ -2961,25 +2961,47 @@ static void prim_listToAttrs(EvalState & state, const PosIdx pos, Value * * args { state.forceList(*args[0], pos, "while evaluating the argument passed to builtins.listToAttrs"); - auto attrs = state.buildBindings(args[0]->listSize()); + // Step 1. Sort the name-value attrsets in place using the memory we allocate for the result + size_t listSize = args[0]->listSize(); + auto & bindings = *state.allocBindings(listSize); + using ElemPtr = decltype(&bindings[0].value); - std::set seen; - - for (auto v2 : args[0]->listItems()) { + for (const auto & [n, v2] : enumerate(args[0]->listItems())) { state.forceAttrs(*v2, pos, "while evaluating an element of the list passed to builtins.listToAttrs"); auto j = state.getAttr(state.sName, v2->attrs(), "in a {name=...; value=...;} pair"); auto name = state.forceStringNoCtx(*j->value, j->pos, "while evaluating the `name` attribute of an element of the list passed to builtins.listToAttrs"); - auto sym = state.symbols.create(name); - if (seen.insert(sym).second) { - auto j2 = state.getAttr(state.sValue, v2->attrs(), "in a {name=...; value=...;} pair"); - attrs.insert(sym, j2->value, j2->pos); - } + + // (ab)use Attr to store a Value * * instead of a Value *, so that we can stabilize the sort using the Value * * + bindings[n] = Attr(sym, std::bit_cast(&v2)); } - v.mkAttrs(attrs); + std::sort(&bindings[0], &bindings[listSize], [](const Attr & a, const Attr & b) { + // Note that .value is actually a Value * * that corresponds to the position in the list + return a < b || (!(a > b) && std::bit_cast(a.value) < std::bit_cast(b.value)); + }); + + // Step 2. Unpack the bindings in place and skip name-value pairs with duplicate names + Symbol prev; + for (size_t n = 0; n < listSize; n++) { + auto attr = bindings[n]; + if (prev == attr.name) { + continue; + } + // Note that .value is actually a Value * *; see earlier comments + Value * v2 = *std::bit_cast(attr.value); + + auto j = state.getAttr(state.sValue, v2->attrs(), "in a {name=...; value=...;} pair"); + prev = attr.name; + bindings.push_back({prev, j->value, j->pos}); + } + // help GC and clear end of allocated array + for (size_t n = bindings.size(); n < listSize; n++) { + bindings[n] = Attr{}; + } + v.mkAttrs(&bindings); } static RegisterPrimOp primop_listToAttrs({ From 8708e9a52617a70ce1a85ff090e216e257c2ae93 Mon Sep 17 00:00:00 2001 From: Sergei Zimmerman Date: Mon, 30 Jun 2025 23:29:07 +0300 Subject: [PATCH 197/218] libutil: Use caching `directory_entry` API in `PosixSourceAccessor::readDirectory` MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Previous use of symlink_status() always translated into a stat call, leading to huge performance penalties for by-name-overlay in nixpkgs. The comment below references the possible caching, but that seemed to be erroneous, since the correct way to make use of the caching API is by calling a bunch of `is_*` functions [1]. For example, here's how libstdc++ does that [2], [3]. This translates to great nixpkgs eval performance improvements: ``` Benchmark 1: GC_INITIAL_HEAP_SIZE=4G result/bin/nix-instantiate ../nixpkgs -A hello --readonly-mode Time (mean ± σ): 186.7 ms ± 6.7 ms [User: 121.3 ms, System: 64.9 ms] Range (min … max): 179.4 ms … 201.6 ms 16 runs Benchmark 2: GC_INITIAL_HEAP_SIZE=4G nix-instantiate ../nixpkgs -A hello --readonly-mode Time (mean ± σ): 230.6 ms ± 5.0 ms [User: 126.9 ms, System: 103.1 ms] Range (min … max): 225.1 ms … 241.4 ms 13 runs ``` [1]: https://www.open-std.org/jtc1/sc22/wg21/docs/papers/2016/p0317r1.html [2]: https://github.com/gcc-mirror/gcc/blob/8ea555b7b4725dbc5d9286f729166cd54ce5b615/libstdc%2B%2B-v3/include/bits/fs_dir.h#L341-L348 [3]: https://github.com/gcc-mirror/gcc/blob/8ea555b7b4725dbc5d9286f729166cd54ce5b615/libstdc%2B%2B-v3/include/bits/fs_dir.h#L161-L163 --- src/libutil/posix-source-accessor.cc | 49 +++++++++++++++++----------- 1 file changed, 30 insertions(+), 19 deletions(-) diff --git a/src/libutil/posix-source-accessor.cc b/src/libutil/posix-source-accessor.cc index 773540e6a09..2ce7c88e4f8 100644 --- a/src/libutil/posix-source-accessor.cc +++ b/src/libutil/posix-source-accessor.cc @@ -141,33 +141,44 @@ SourceAccessor::DirEntries PosixSourceAccessor::readDirectory(const CanonPath & for (auto & entry : DirectoryIterator{makeAbsPath(path)}) { checkInterrupt(); auto type = [&]() -> std::optional { - std::filesystem::file_type nativeType; try { - nativeType = entry.symlink_status().type(); + /* WARNING: We are specifically not calling symlink_status() + * here, because that always translates to `stat` call and + * doesn't make use of any caching. Instead, we have to + * rely on the myriad of `is_*` functions, which actually do + * the caching. If you are in doubt then take a look at the + * libstdc++ implementation [1] and the standard proposal + * about the caching variations of directory_entry [2]. + + * [1]: https://github.com/gcc-mirror/gcc/blob/8ea555b7b4725dbc5d9286f729166cd54ce5b615/libstdc%2B%2B-v3/include/bits/fs_dir.h#L341-L348 + * [2]: https://www.open-std.org/jtc1/sc22/wg21/docs/papers/2016/p0317r1.html + */ + + /* Check for symlink first, because other getters follow symlinks. */ + if (entry.is_symlink()) + return tSymlink; + if (entry.is_regular_file()) + return tRegular; + if (entry.is_directory()) + return tDirectory; + if (entry.is_character_file()) + return tChar; + if (entry.is_block_file()) + return tBlock; + if (entry.is_fifo()) + return tFifo; + if (entry.is_socket()) + return tSocket; + return tUnknown; } catch (std::filesystem::filesystem_error & e) { // We cannot always stat the child. (Ideally there is no // stat because the native directory entry has the type // already, but this isn't always the case.) if (e.code() == std::errc::permission_denied || e.code() == std::errc::operation_not_permitted) return std::nullopt; - else throw; + else + throw; } - - // cannot exhaustively enumerate because implementation-specific - // additional file types are allowed. -#pragma GCC diagnostic push -#pragma GCC diagnostic ignored "-Wswitch-enum" - switch (nativeType) { - case std::filesystem::file_type::regular: return Type::tRegular; break; - case std::filesystem::file_type::symlink: return Type::tSymlink; break; - case std::filesystem::file_type::directory: return Type::tDirectory; break; - case std::filesystem::file_type::character: return Type::tChar; break; - case std::filesystem::file_type::block: return Type::tBlock; break; - case std::filesystem::file_type::fifo: return Type::tFifo; break; - case std::filesystem::file_type::socket: return Type::tSocket; break; - default: return tUnknown; - } -#pragma GCC diagnostic pop }(); res.emplace(entry.path().filename().string(), type); } From d16af1d0993bade1da1d26710f0fd5b43e03b506 Mon Sep 17 00:00:00 2001 From: Sergei Zimmerman Date: Wed, 2 Jul 2025 00:57:35 +0300 Subject: [PATCH 198/218] libfetchers: Add missing include guard to git-lfs-fetch.hh This is a publicly installed header without a header guard. Doesn't seem right. --- src/libfetchers/include/nix/fetchers/git-lfs-fetch.hh | 3 +++ 1 file changed, 3 insertions(+) diff --git a/src/libfetchers/include/nix/fetchers/git-lfs-fetch.hh b/src/libfetchers/include/nix/fetchers/git-lfs-fetch.hh index e701288cf3c..b59da391a05 100644 --- a/src/libfetchers/include/nix/fetchers/git-lfs-fetch.hh +++ b/src/libfetchers/include/nix/fetchers/git-lfs-fetch.hh @@ -1,3 +1,6 @@ +#pragma once +///@file + #include "nix/util/canon-path.hh" #include "nix/util/serialise.hh" #include "nix/util/url.hh" From ea32580c9b50b9f3dbe47ae9dd2ae28d9bcabca9 Mon Sep 17 00:00:00 2001 From: Sergei Zimmerman Date: Sat, 28 Jun 2025 15:30:38 +0300 Subject: [PATCH 199/218] libexpr: Format value.hh The following commits will touch this file significantly, so it's better to get the formatting out of the way first. --- maintainers/flake-module.nix | 1 - src/libexpr/include/nix/expr/value.hh | 244 +++++++++++++++++--------- 2 files changed, 158 insertions(+), 87 deletions(-) diff --git a/maintainers/flake-module.nix b/maintainers/flake-module.nix index d443d1a741b..1058d633473 100644 --- a/maintainers/flake-module.nix +++ b/maintainers/flake-module.nix @@ -256,7 +256,6 @@ ''^src/libexpr/include/nix/expr/value-to-json\.hh$'' ''^src/libexpr/value-to-xml\.cc$'' ''^src/libexpr/include/nix/expr/value-to-xml\.hh$'' - ''^src/libexpr/include/nix/expr/value\.hh$'' ''^src/libexpr/value/context\.cc$'' ''^src/libexpr/include/nix/expr/value/context\.hh$'' ''^src/libfetchers/attrs\.cc$'' diff --git a/src/libexpr/include/nix/expr/value.hh b/src/libexpr/include/nix/expr/value.hh index fcc118c7e96..b67f6380807 100644 --- a/src/libexpr/include/nix/expr/value.hh +++ b/src/libexpr/include/nix/expr/value.hh @@ -17,7 +17,6 @@ namespace nix { struct Value; class BindingsBuilder; - typedef enum { tUninitialized = 0, tInt = 1, @@ -54,7 +53,7 @@ typedef enum { nAttrs, nList, nFunction, - nExternal + nExternal, } ValueType; class Bindings; @@ -81,15 +80,15 @@ using NixFloat = double; */ class ExternalValueBase { - friend std::ostream & operator << (std::ostream & str, const ExternalValueBase & v); + friend std::ostream & operator<<(std::ostream & str, const ExternalValueBase & v); friend class Printer; - protected: +protected: /** * Print out the value */ virtual std::ostream & print(std::ostream & str) const = 0; - public: +public: /** * Return a simple string describing the type */ @@ -104,41 +103,44 @@ class ExternalValueBase * Coerce the value to a string. Defaults to uncoercable, i.e. throws an * error. */ - virtual std::string coerceToString(EvalState & state, const PosIdx & pos, NixStringContext & context, bool copyMore, bool copyToStore) const; + virtual std::string coerceToString( + EvalState & state, const PosIdx & pos, NixStringContext & context, bool copyMore, bool copyToStore) const; /** * Compare to another value of the same type. Defaults to uncomparable, * i.e. always false. */ - virtual bool operator ==(const ExternalValueBase & b) const noexcept; + virtual bool operator==(const ExternalValueBase & b) const noexcept; /** * Print the value as JSON. Defaults to unconvertable, i.e. throws an error */ - virtual nlohmann::json printValueAsJSON(EvalState & state, bool strict, - NixStringContext & context, bool copyToStore = true) const; + virtual nlohmann::json + printValueAsJSON(EvalState & state, bool strict, NixStringContext & context, bool copyToStore = true) const; /** * Print the value as XML. Defaults to unevaluated */ - virtual void printValueAsXML(EvalState & state, bool strict, bool location, - XMLWriter & doc, NixStringContext & context, PathSet & drvsSeen, + virtual void printValueAsXML( + EvalState & state, + bool strict, + bool location, + XMLWriter & doc, + NixStringContext & context, + PathSet & drvsSeen, const PosIdx pos) const; - virtual ~ExternalValueBase() - { - }; + virtual ~ExternalValueBase() {}; }; -std::ostream & operator << (std::ostream & str, const ExternalValueBase & v); - +std::ostream & operator<<(std::ostream & str, const ExternalValueBase & v); class ListBuilder { const size_t size; Value * inlineElems[2] = {nullptr, nullptr}; public: - Value * * elems; + Value ** elems; ListBuilder(EvalState & state, size_t size); // NOTE: Can be noexcept because we are just copying integral values and @@ -147,22 +149,28 @@ public: : size(x.size) , inlineElems{x.inlineElems[0], x.inlineElems[1]} , elems(size <= 2 ? inlineElems : x.elems) - { } + { + } - Value * & operator [](size_t n) + Value *& operator[](size_t n) { return elems[n]; } - typedef Value * * iterator; + typedef Value ** iterator; - iterator begin() { return &elems[0]; } - iterator end() { return &elems[size]; } + iterator begin() + { + return &elems[0]; + } + iterator end() + { + return &elems[size]; + } friend struct Value; }; - struct Value { private: @@ -177,21 +185,36 @@ public: */ static Value * toPtr(SymbolStr str) noexcept; - void print(EvalState &state, std::ostream &str, PrintOptions options = PrintOptions {}); + void print(EvalState & state, std::ostream & str, PrintOptions options = PrintOptions{}); // Functions needed to distinguish the type // These should be removed eventually, by putting the functionality that's // needed by callers into methods of this type // type() == nThunk - inline bool isThunk() const { return internalType == tThunk; }; - inline bool isApp() const { return internalType == tApp; }; + inline bool isThunk() const + { + return internalType == tThunk; + }; + inline bool isApp() const + { + return internalType == tApp; + }; inline bool isBlackhole() const; // type() == nFunction - inline bool isLambda() const { return internalType == tLambda; }; - inline bool isPrimOp() const { return internalType == tPrimOp; }; - inline bool isPrimOpApp() const { return internalType == tPrimOpApp; }; + inline bool isLambda() const + { + return internalType == tLambda; + }; + inline bool isPrimOp() const + { + return internalType == tPrimOp; + }; + inline bool isPrimOpApp() const + { + return internalType == tPrimOpApp; + }; /** * Strings in the evaluator carry a so-called `context` which @@ -215,26 +238,31 @@ public: * For canonicity, the store paths should be in sorted order. */ - struct StringWithContext { + struct StringWithContext + { const char * c_str; - const char * * context; // must be in sorted order + const char ** context; // must be in sorted order }; - struct Path { + struct Path + { SourceAccessor * accessor; const char * path; }; - struct ClosureThunk { + struct ClosureThunk + { Env * env; Expr * expr; }; - struct FunctionApplicationThunk { - Value * left, * right; + struct FunctionApplicationThunk + { + Value *left, *right; }; - struct Lambda { + struct Lambda + { Env * env; ExprLambda * fun; }; @@ -249,7 +277,8 @@ public: Path path; Bindings * attrs; - struct { + struct + { size_t size; Value * const * elems; } bigList; @@ -275,18 +304,35 @@ public: inline ValueType type(bool invalidIsThunk = false) const { switch (internalType) { - case tUninitialized: break; - case tInt: return nInt; - case tBool: return nBool; - case tString: return nString; - case tPath: return nPath; - case tNull: return nNull; - case tAttrs: return nAttrs; - case tList1: case tList2: case tListN: return nList; - case tLambda: case tPrimOp: case tPrimOpApp: return nFunction; - case tExternal: return nExternal; - case tFloat: return nFloat; - case tThunk: case tApp: return nThunk; + case tUninitialized: + break; + case tInt: + return nInt; + case tBool: + return nBool; + case tString: + return nString; + case tPath: + return nPath; + case tNull: + return nNull; + case tAttrs: + return nAttrs; + case tList1: + case tList2: + case tListN: + return nList; + case tLambda: + case tPrimOp: + case tPrimOpApp: + return nFunction; + case tExternal: + return nExternal; + case tFloat: + return nFloat; + case tThunk: + case tApp: + return nThunk; } if (invalidIsThunk) return nThunk; @@ -317,17 +363,17 @@ public: inline void mkInt(NixInt n) { - finishValue(tInt, { .integer = n }); + finishValue(tInt, {.integer = n}); } inline void mkBool(bool b) { - finishValue(tBool, { .boolean = b }); + finishValue(tBool, {.boolean = b}); } - inline void mkString(const char * s, const char * * context = 0) + inline void mkString(const char * s, const char ** context = 0) { - finishValue(tString, { .string = { .c_str = s, .context = context } }); + finishValue(tString, {.string = {.c_str = s, .context = context}}); } void mkString(std::string_view s); @@ -341,7 +387,7 @@ public: inline void mkPath(SourceAccessor * accessor, const char * path) { - finishValue(tPath, { .path = { .accessor = accessor, .path = path } }); + finishValue(tPath, {.path = {.accessor = accessor, .path = path}}); } inline void mkNull() @@ -351,7 +397,7 @@ public: inline void mkAttrs(Bindings * a) { - finishValue(tAttrs, { .attrs = a }); + finishValue(tAttrs, {.attrs = a}); } Value & mkAttrs(BindingsBuilder & bindings); @@ -359,26 +405,26 @@ public: void mkList(const ListBuilder & builder) { if (builder.size == 1) - finishValue(tList1, { .smallList = { builder.inlineElems[0] } }); + finishValue(tList1, {.smallList = {builder.inlineElems[0]}}); else if (builder.size == 2) - finishValue(tList2, { .smallList = { builder.inlineElems[0], builder.inlineElems[1] } }); + finishValue(tList2, {.smallList = {builder.inlineElems[0], builder.inlineElems[1]}}); else - finishValue(tListN, { .bigList = { .size = builder.size, .elems = builder.elems } }); + finishValue(tListN, {.bigList = {.size = builder.size, .elems = builder.elems}}); } inline void mkThunk(Env * e, Expr * ex) { - finishValue(tThunk, { .thunk = { .env = e, .expr = ex } }); + finishValue(tThunk, {.thunk = {.env = e, .expr = ex}}); } inline void mkApp(Value * l, Value * r) { - finishValue(tApp, { .app = { .left = l, .right = r } }); + finishValue(tApp, {.app = {.left = l, .right = r}}); } inline void mkLambda(Env * e, ExprLambda * f) { - finishValue(tLambda, { .lambda = { .env = e, .fun = f } }); + finishValue(tLambda, {.lambda = {.env = e, .fun = f}}); } inline void mkBlackhole(); @@ -387,7 +433,7 @@ public: inline void mkPrimOpApp(Value * l, Value * r) { - finishValue(tPrimOpApp, { .primOpApp = { .left = l, .right = r } }); + finishValue(tPrimOpApp, {.primOpApp = {.left = l, .right = r}}); } /** @@ -397,12 +443,12 @@ public: inline void mkExternal(ExternalValueBase * e) { - finishValue(tExternal, { .external = e }); + finishValue(tExternal, {.external = e}); } inline void mkFloat(NixFloat n) { - finishValue(tFloat, { .fpoint = n }); + finishValue(tFloat, {.fpoint = n}); } bool isList() const @@ -438,9 +484,7 @@ public: SourcePath path() const { assert(internalType == tPath); - return SourcePath( - ref(pathAccessor()->shared_from_this()), - CanonPath(CanonPath::unchecked_t(), pathStr())); + return SourcePath(ref(pathAccessor()->shared_from_this()), CanonPath(CanonPath::unchecked_t(), pathStr())); } std::string_view string_view() const @@ -455,54 +499,77 @@ public: return payload.string.c_str; } - const char * * context() const + const char ** context() const { return payload.string.context; } ExternalValueBase * external() const - { return payload.external; } + { + return payload.external; + } const Bindings * attrs() const - { return payload.attrs; } + { + return payload.attrs; + } const PrimOp * primOp() const - { return payload.primOp; } + { + return payload.primOp; + } bool boolean() const - { return payload.boolean; } + { + return payload.boolean; + } NixInt integer() const - { return payload.integer; } + { + return payload.integer; + } NixFloat fpoint() const - { return payload.fpoint; } + { + return payload.fpoint; + } Lambda lambda() const - { return payload.lambda; } + { + return payload.lambda; + } ClosureThunk thunk() const - { return payload.thunk; } + { + return payload.thunk; + } FunctionApplicationThunk primOpApp() const - { return payload.primOpApp; } + { + return payload.primOpApp; + } FunctionApplicationThunk app() const - { return payload.app; } + { + return payload.app; + } const char * pathStr() const - { return payload.path.path; } + { + return payload.path.path; + } SourceAccessor * pathAccessor() const - { return payload.path.accessor; } + { + return payload.path.accessor; + } }; - extern ExprBlackHole eBlackHole; bool Value::isBlackhole() const { - return internalType == tThunk && thunk().expr == (Expr*) &eBlackHole; + return internalType == tThunk && thunk().expr == (Expr *) &eBlackHole; } void Value::mkBlackhole() @@ -510,11 +577,16 @@ void Value::mkBlackhole() mkThunk(nullptr, (Expr *) &eBlackHole); } - typedef std::vector> ValueVector; -typedef std::unordered_map, std::equal_to, traceable_allocator>> ValueMap; -typedef std::map, traceable_allocator>> ValueVectorMap; - +typedef std::unordered_map< + Symbol, + Value *, + std::hash, + std::equal_to, + traceable_allocator>> + ValueMap; +typedef std::map, traceable_allocator>> + ValueVectorMap; /** * A value allocated in traceable memory. From 1a033ee4eee9804f5544a92a3fe8d66e77d3257c Mon Sep 17 00:00:00 2001 From: Sergei Zimmerman Date: Sat, 28 Jun 2025 15:30:40 +0300 Subject: [PATCH 200/218] libexpr: Use single `tSmallList` `Value` discriminator for small lists --- src/libexpr/include/nix/expr/value.hh | 16 +++++++--------- 1 file changed, 7 insertions(+), 9 deletions(-) diff --git a/src/libexpr/include/nix/expr/value.hh b/src/libexpr/include/nix/expr/value.hh index b67f6380807..658cb5a6af1 100644 --- a/src/libexpr/include/nix/expr/value.hh +++ b/src/libexpr/include/nix/expr/value.hh @@ -25,8 +25,7 @@ typedef enum { tPath, tNull, tAttrs, - tList1, - tList2, + tListSmall, tListN, tThunk, tApp, @@ -318,8 +317,7 @@ public: return nNull; case tAttrs: return nAttrs; - case tList1: - case tList2: + case tListSmall: case tListN: return nList; case tLambda: @@ -405,9 +403,9 @@ public: void mkList(const ListBuilder & builder) { if (builder.size == 1) - finishValue(tList1, {.smallList = {builder.inlineElems[0]}}); + finishValue(tListSmall, {.smallList = {builder.inlineElems[0], nullptr}}); else if (builder.size == 2) - finishValue(tList2, {.smallList = {builder.inlineElems[0], builder.inlineElems[1]}}); + finishValue(tListSmall, {.smallList = {builder.inlineElems[0], builder.inlineElems[1]}}); else finishValue(tListN, {.bigList = {.size = builder.size, .elems = builder.elems}}); } @@ -453,7 +451,7 @@ public: bool isList() const { - return internalType == tList1 || internalType == tList2 || internalType == tListN; + return internalType == tListSmall || internalType == tListN; } std::span listItems() const @@ -464,12 +462,12 @@ public: Value * const * listElems() const { - return internalType == tList1 || internalType == tList2 ? payload.smallList : payload.bigList.elems; + return internalType == tListSmall ? payload.smallList : payload.bigList.elems; } size_t listSize() const { - return internalType == tList1 ? 1 : internalType == tList2 ? 2 : payload.bigList.size; + return internalType == tListSmall ? (payload.smallList[1] == nullptr ? 1 : 2) : payload.bigList.size; } PosIdx determinePos(const PosIdx pos) const; From 810455f1b80959ce096ed8be5c7f3c50b58fc7d9 Mon Sep 17 00:00:00 2001 From: Sergei Zimmerman Date: Sat, 28 Jun 2025 15:30:43 +0300 Subject: [PATCH 201/218] libexpr: Simplify Value::is* methods by introducing isa function template --- src/libexpr/eval.cc | 9 +++--- src/libexpr/include/nix/expr/value.hh | 40 +++++++++++++++++---------- 2 files changed, 30 insertions(+), 19 deletions(-) diff --git a/src/libexpr/eval.cc b/src/libexpr/eval.cc index ae422a3d4e7..806b506a592 100644 --- a/src/libexpr/eval.cc +++ b/src/libexpr/eval.cc @@ -125,7 +125,7 @@ std::string showType(const Value & v) // Allow selecting a subset of enum values #pragma GCC diagnostic push #pragma GCC diagnostic ignored "-Wswitch-enum" - switch (v.internalType) { + switch (v.getInternalType()) { case tString: return v.context() ? "a string with context" : "a string"; case tPrimOp: return fmt("the built-in function '%s'", std::string(v.primOp()->name)); @@ -145,7 +145,7 @@ PosIdx Value::determinePos(const PosIdx pos) const // Allow selecting a subset of enum values #pragma GCC diagnostic push #pragma GCC diagnostic ignored "-Wswitch-enum" - switch (internalType) { + switch (getInternalType()) { case tAttrs: return attrs()->pos; case tLambda: return lambda().fun->pos; case tApp: return app().left->determinePos(pos); @@ -157,9 +157,8 @@ PosIdx Value::determinePos(const PosIdx pos) const bool Value::isTrivial() const { return - internalType != tApp - && internalType != tPrimOpApp - && (internalType != tThunk + !isa() + && (!isa() || (dynamic_cast(thunk().expr) && ((ExprAttrs *) thunk().expr)->dynamicAttrs.empty()) || dynamic_cast(thunk().expr) diff --git a/src/libexpr/include/nix/expr/value.hh b/src/libexpr/include/nix/expr/value.hh index 658cb5a6af1..0af9e1ea50b 100644 --- a/src/libexpr/include/nix/expr/value.hh +++ b/src/libexpr/include/nix/expr/value.hh @@ -175,6 +175,18 @@ struct Value private: InternalType internalType = tUninitialized; + InternalType getInternalType() const noexcept + { + return internalType; + } + + /** Check if currently stored value type is either of Vs. */ + template + bool isa() const noexcept + { + return ((getInternalType() == Vs) || ...); + } + friend std::string showType(const Value & v); public: @@ -193,26 +205,26 @@ public: // type() == nThunk inline bool isThunk() const { - return internalType == tThunk; + return isa(); }; inline bool isApp() const { - return internalType == tApp; + return isa(); }; inline bool isBlackhole() const; // type() == nFunction inline bool isLambda() const { - return internalType == tLambda; + return isa(); }; inline bool isPrimOp() const { - return internalType == tPrimOp; + return isa(); }; inline bool isPrimOpApp() const { - return internalType == tPrimOpApp; + return isa(); }; /** @@ -302,7 +314,7 @@ public: */ inline ValueType type(bool invalidIsThunk = false) const { - switch (internalType) { + switch (getInternalType()) { case tUninitialized: break; case tInt: @@ -351,7 +363,7 @@ public: */ inline bool isValid() const { - return internalType != tUninitialized; + return !isa(); } inline void mkInt(NixInt::Inner n) @@ -451,7 +463,7 @@ public: bool isList() const { - return internalType == tListSmall || internalType == tListN; + return isa(); } std::span listItems() const @@ -462,12 +474,12 @@ public: Value * const * listElems() const { - return internalType == tListSmall ? payload.smallList : payload.bigList.elems; + return isa() ? payload.smallList : payload.bigList.elems; } size_t listSize() const { - return internalType == tListSmall ? (payload.smallList[1] == nullptr ? 1 : 2) : payload.bigList.size; + return isa() ? (payload.smallList[1] == nullptr ? 1 : 2) : payload.bigList.size; } PosIdx determinePos(const PosIdx pos) const; @@ -481,19 +493,19 @@ public: SourcePath path() const { - assert(internalType == tPath); + assert(isa()); return SourcePath(ref(pathAccessor()->shared_from_this()), CanonPath(CanonPath::unchecked_t(), pathStr())); } std::string_view string_view() const { - assert(internalType == tString); + assert(isa()); return std::string_view(payload.string.c_str); } const char * c_str() const { - assert(internalType == tString); + assert(isa()); return payload.string.c_str; } @@ -567,7 +579,7 @@ extern ExprBlackHole eBlackHole; bool Value::isBlackhole() const { - return internalType == tThunk && thunk().expr == (Expr *) &eBlackHole; + return isa() && thunk().expr == (Expr *) &eBlackHole; } void Value::mkBlackhole() From c39cc004043b95d55a0c2c2bdba58d6d3e0db846 Mon Sep 17 00:00:00 2001 From: Sergei Zimmerman Date: Sat, 28 Jun 2025 16:18:18 +0300 Subject: [PATCH 202/218] libexpr: Factor out `Payload` union to a default implementation of `ValueStorage` This factors out most of the value representation into a mixin class. `finishValue` is now gone for good and replaced with a simple template function `setStorage` which derives the type information/disriminator from the type of the argument. Likewise, reading of the value goes through function template `getStorage`. An empty type `Null` is introduced to make the bijection InternalType <-> C++ type complete. --- src/libexpr/eval.cc | 6 +- src/libexpr/include/nix/expr/value.hh | 402 ++++++++++++++++---------- src/libexpr/primops.cc | 2 +- 3 files changed, 252 insertions(+), 158 deletions(-) diff --git a/src/libexpr/eval.cc b/src/libexpr/eval.cc index 806b506a592..fba1e666518 100644 --- a/src/libexpr/eval.cc +++ b/src/libexpr/eval.cc @@ -502,7 +502,7 @@ void EvalState::addConstant(const std::string & name, Value * v, Constant info) /* Install value the base environment. */ staticBaseEnv->vars.emplace_back(symbols.create(name), baseEnvDispl); baseEnv.values[baseEnvDispl++] = v; - getBuiltins().payload.attrs->push_back(Attr(symbols.create(name2), v)); + const_cast(getBuiltins().attrs())->push_back(Attr(symbols.create(name2), v)); } } @@ -540,7 +540,7 @@ const PrimOp * Value::primOpAppPrimOp() const void Value::mkPrimOp(PrimOp * p) { p->check(); - finishValue(tPrimOp, { .primOp = p }); + setStorage(p); } @@ -572,7 +572,7 @@ Value * EvalState::addPrimOp(PrimOp && primOp) else { staticBaseEnv->vars.emplace_back(envName, baseEnvDispl); baseEnv.values[baseEnvDispl++] = v; - getBuiltins().payload.attrs->push_back(Attr(symbols.create(primOp.name), v)); + const_cast(getBuiltins().attrs())->push_back(Attr(symbols.create(primOp.name), v)); } return v; diff --git a/src/libexpr/include/nix/expr/value.hh b/src/libexpr/include/nix/expr/value.hh index 0af9e1ea50b..642459463b3 100644 --- a/src/libexpr/include/nix/expr/value.hh +++ b/src/libexpr/include/nix/expr/value.hh @@ -3,6 +3,7 @@ #include #include +#include #include "nix/expr/eval-gc.hh" #include "nix/expr/value/context.hh" @@ -170,63 +171,15 @@ public: friend struct Value; }; -struct Value -{ -private: - InternalType internalType = tUninitialized; - - InternalType getInternalType() const noexcept - { - return internalType; - } - - /** Check if currently stored value type is either of Vs. */ - template - bool isa() const noexcept - { - return ((getInternalType() == Vs) || ...); - } - - friend std::string showType(const Value & v); - -public: - - /** - * Never modify the backing `Value` object! - */ - static Value * toPtr(SymbolStr str) noexcept; - - void print(EvalState & state, std::ostream & str, PrintOptions options = PrintOptions{}); - - // Functions needed to distinguish the type - // These should be removed eventually, by putting the functionality that's - // needed by callers into methods of this type - - // type() == nThunk - inline bool isThunk() const - { - return isa(); - }; - inline bool isApp() const - { - return isa(); - }; - inline bool isBlackhole() const; - - // type() == nFunction - inline bool isLambda() const - { - return isa(); - }; - inline bool isPrimOp() const - { - return isa(); - }; - inline bool isPrimOpApp() const - { - return isa(); - }; +namespace detail { +/** + * Implementation mixin class for defining the public types + * In can be inherited from by the actual ValueStorage implementations + * for free due to Empty Base Class Optimization (EBCO). + */ +struct ValueBase +{ /** * Strings in the evaluator carry a so-called `context` which * is a list of strings representing store paths. This is to @@ -261,6 +214,9 @@ public: const char * path; }; + struct Null + {}; + struct ClosureThunk { Env * env; @@ -272,39 +228,187 @@ public: Value *left, *right; }; + /** + * Like FunctionApplicationThunk, but must be a distinct type in order to + * resolve overloads to `tPrimOpApp` instead of `tApp`. + * This type helps with the efficient implementation of arity>=2 primop calls. + */ + struct PrimOpApplicationThunk + { + Value *left, *right; + }; + struct Lambda { Env * env; ExprLambda * fun; }; + using SmallList = std::array; + + struct List + { + size_t size; + Value * const * elems; + }; +}; + +template +struct PayloadTypeToInternalType +{}; + +/** + * All stored types must be distinct (not type aliases) for the purposes of + * overload resolution in setStorage. This ensures there's a bijection from + * InternalType <-> C++ type. + */ +#define NIX_VALUE_STORAGE_FOR_EACH_FIELD(MACRO) \ + MACRO(NixInt, integer, tInt) \ + MACRO(bool, boolean, tBool) \ + MACRO(ValueBase::StringWithContext, string, tString) \ + MACRO(ValueBase::Path, path, tPath) \ + MACRO(ValueBase::Null, null_, tNull) \ + MACRO(Bindings *, attrs, tAttrs) \ + MACRO(ValueBase::List, bigList, tListN) \ + MACRO(ValueBase::SmallList, smallList, tListSmall) \ + MACRO(ValueBase::ClosureThunk, thunk, tThunk) \ + MACRO(ValueBase::FunctionApplicationThunk, app, tApp) \ + MACRO(ValueBase::Lambda, lambda, tLambda) \ + MACRO(PrimOp *, primOp, tPrimOp) \ + MACRO(ValueBase::PrimOpApplicationThunk, primOpApp, tPrimOpApp) \ + MACRO(ExternalValueBase *, external, tExternal) \ + MACRO(NixFloat, fpoint, tFloat) + +#define NIX_VALUE_PAYLOAD_TYPE(T, FIELD_NAME, DISCRIMINATOR) \ + template<> \ + struct PayloadTypeToInternalType \ + { \ + static constexpr InternalType value = DISCRIMINATOR; \ + }; + +NIX_VALUE_STORAGE_FOR_EACH_FIELD(NIX_VALUE_PAYLOAD_TYPE) + +#undef NIX_VALUE_PAYLOAD_TYPE + +template +inline constexpr InternalType payloadTypeToInternalType = PayloadTypeToInternalType::value; + +} + +/** + * Discriminated union of types stored in the value. + * The union discriminator is @ref InternalType enumeration. + * + * This class can be specialized with a non-type template parameter + * of pointer size for more optimized data layouts on when pointer alignment + * bits can be used for storing the discriminator. + * + * All specializations of this type need to implement getStorage, setStorage and + * getInternalType methods. + */ +template +class ValueStorage : public detail::ValueBase +{ +protected: using Payload = union { - NixInt integer; - bool boolean; - - StringWithContext string; - - Path path; - - Bindings * attrs; - struct - { - size_t size; - Value * const * elems; - } bigList; - Value * smallList[2]; - ClosureThunk thunk; - FunctionApplicationThunk app; - Lambda lambda; - PrimOp * primOp; - FunctionApplicationThunk primOpApp; - ExternalValueBase * external; - NixFloat fpoint; +#define NIX_VALUE_STORAGE_DEFINE_FIELD(T, FIELD_NAME, DISCRIMINATOR) T FIELD_NAME; + NIX_VALUE_STORAGE_FOR_EACH_FIELD(NIX_VALUE_STORAGE_DEFINE_FIELD) +#undef NIX_VALUE_STORAGE_DEFINE_FIELD }; Payload payload; +private: + InternalType internalType = tUninitialized; + +public: +#define NIX_VALUE_STORAGE_GET_IMPL(K, FIELD_NAME, DISCRIMINATOR) \ + void getStorage(K & val) const noexcept \ + { \ + assert(internalType == DISCRIMINATOR); \ + val = payload.FIELD_NAME; \ + } + +#define NIX_VALUE_STORAGE_SET_IMPL(K, FIELD_NAME, DISCRIMINATOR) \ + void setStorage(K val) noexcept \ + { \ + payload.FIELD_NAME = val; \ + internalType = DISCRIMINATOR; \ + } + + NIX_VALUE_STORAGE_FOR_EACH_FIELD(NIX_VALUE_STORAGE_GET_IMPL) + NIX_VALUE_STORAGE_FOR_EACH_FIELD(NIX_VALUE_STORAGE_SET_IMPL) + +#undef NIX_VALUE_STORAGE_SET_IMPL +#undef NIX_VALUE_STORAGE_GET_IMPL +#undef NIX_VALUE_STORAGE_FOR_EACH_FIELD + + /** Get internal type currently occupying the storage. */ + InternalType getInternalType() const noexcept + { + return internalType; + } +}; + +struct Value : public ValueStorage +{ + friend std::string showType(const Value & v); + + template + bool isa() const noexcept + { + return ((getInternalType() == discriminator) || ...); + } + + template + T getStorage() const noexcept + { + if (getInternalType() != detail::payloadTypeToInternalType) [[unlikely]] + unreachable(); + T out; + ValueStorage::getStorage(out); + return out; + } + +public: + + /** + * Never modify the backing `Value` object! + */ + static Value * toPtr(SymbolStr str) noexcept; + + void print(EvalState & state, std::ostream & str, PrintOptions options = PrintOptions{}); + + // Functions needed to distinguish the type + // These should be removed eventually, by putting the functionality that's + // needed by callers into methods of this type + + // type() == nThunk + inline bool isThunk() const + { + return isa(); + }; + inline bool isApp() const + { + return isa(); + }; + inline bool isBlackhole() const; + + // type() == nFunction + inline bool isLambda() const + { + return isa(); + }; + inline bool isPrimOp() const + { + return isa(); + }; + inline bool isPrimOpApp() const + { + return isa(); + }; + /** * Returns the normal type of a Value. This only returns nThunk if * the Value hasn't been forceValue'd @@ -350,40 +454,34 @@ public: unreachable(); } - inline void finishValue(InternalType newType, Payload newPayload) - { - payload = newPayload; - internalType = newType; - } - /** * A value becomes valid when it is initialized. We don't use this * in the evaluator; only in the bindings, where the slight extra * cost is warranted because of inexperienced callers. */ - inline bool isValid() const + inline bool isValid() const noexcept { return !isa(); } - inline void mkInt(NixInt::Inner n) + inline void mkInt(NixInt::Inner n) noexcept { mkInt(NixInt{n}); } - inline void mkInt(NixInt n) + inline void mkInt(NixInt n) noexcept { - finishValue(tInt, {.integer = n}); + setStorage(NixInt{n}); } - inline void mkBool(bool b) + inline void mkBool(bool b) noexcept { - finishValue(tBool, {.boolean = b}); + setStorage(b); } - inline void mkString(const char * s, const char ** context = 0) + inline void mkString(const char * s, const char ** context = 0) noexcept { - finishValue(tString, {.string = {.c_str = s, .context = context}}); + setStorage(StringWithContext{.c_str = s, .context = context}); } void mkString(std::string_view s); @@ -395,55 +493,55 @@ public: void mkPath(const SourcePath & path); void mkPath(std::string_view path); - inline void mkPath(SourceAccessor * accessor, const char * path) + inline void mkPath(SourceAccessor * accessor, const char * path) noexcept { - finishValue(tPath, {.path = {.accessor = accessor, .path = path}}); + setStorage(Path{.accessor = accessor, .path = path}); } - inline void mkNull() + inline void mkNull() noexcept { - finishValue(tNull, {}); + setStorage(Null{}); } - inline void mkAttrs(Bindings * a) + inline void mkAttrs(Bindings * a) noexcept { - finishValue(tAttrs, {.attrs = a}); + setStorage(a); } Value & mkAttrs(BindingsBuilder & bindings); - void mkList(const ListBuilder & builder) + void mkList(const ListBuilder & builder) noexcept { if (builder.size == 1) - finishValue(tListSmall, {.smallList = {builder.inlineElems[0], nullptr}}); + setStorage(std::array{builder.inlineElems[0], nullptr}); else if (builder.size == 2) - finishValue(tListSmall, {.smallList = {builder.inlineElems[0], builder.inlineElems[1]}}); + setStorage(std::array{builder.inlineElems[0], builder.inlineElems[1]}); else - finishValue(tListN, {.bigList = {.size = builder.size, .elems = builder.elems}}); + setStorage(List{.size = builder.size, .elems = builder.elems}); } - inline void mkThunk(Env * e, Expr * ex) + inline void mkThunk(Env * e, Expr * ex) noexcept { - finishValue(tThunk, {.thunk = {.env = e, .expr = ex}}); + setStorage(ClosureThunk{.env = e, .expr = ex}); } - inline void mkApp(Value * l, Value * r) + inline void mkApp(Value * l, Value * r) noexcept { - finishValue(tApp, {.app = {.left = l, .right = r}}); + setStorage(FunctionApplicationThunk{.left = l, .right = r}); } - inline void mkLambda(Env * e, ExprLambda * f) + inline void mkLambda(Env * e, ExprLambda * f) noexcept { - finishValue(tLambda, {.lambda = {.env = e, .fun = f}}); + setStorage(Lambda{.env = e, .fun = f}); } inline void mkBlackhole(); void mkPrimOp(PrimOp * p); - inline void mkPrimOpApp(Value * l, Value * r) + inline void mkPrimOpApp(Value * l, Value * r) noexcept { - finishValue(tPrimOpApp, {.primOpApp = {.left = l, .right = r}}); + setStorage(PrimOpApplicationThunk{.left = l, .right = r}); } /** @@ -451,35 +549,35 @@ public: */ const PrimOp * primOpAppPrimOp() const; - inline void mkExternal(ExternalValueBase * e) + inline void mkExternal(ExternalValueBase * e) noexcept { - finishValue(tExternal, {.external = e}); + setStorage(e); } - inline void mkFloat(NixFloat n) + inline void mkFloat(NixFloat n) noexcept { - finishValue(tFloat, {.fpoint = n}); + setStorage(n); } - bool isList() const + bool isList() const noexcept { return isa(); } - std::span listItems() const + std::span listItems() const noexcept { assert(isList()); return std::span(listElems(), listSize()); } - Value * const * listElems() const + Value * const * listElems() const noexcept { - return isa() ? payload.smallList : payload.bigList.elems; + return isa() ? payload.smallList.data() : getStorage().elems; } - size_t listSize() const + size_t listSize() const noexcept { - return isa() ? (payload.smallList[1] == nullptr ? 1 : 2) : payload.bigList.size; + return isa() ? (getStorage()[1] == nullptr ? 1 : 2) : getStorage().size; } PosIdx determinePos(const PosIdx pos) const; @@ -493,85 +591,82 @@ public: SourcePath path() const { - assert(isa()); return SourcePath(ref(pathAccessor()->shared_from_this()), CanonPath(CanonPath::unchecked_t(), pathStr())); } - std::string_view string_view() const + std::string_view string_view() const noexcept { - assert(isa()); - return std::string_view(payload.string.c_str); + return std::string_view(getStorage().c_str); } - const char * c_str() const + const char * c_str() const noexcept { - assert(isa()); - return payload.string.c_str; + return getStorage().c_str; } - const char ** context() const + const char ** context() const noexcept { - return payload.string.context; + return getStorage().context; } - ExternalValueBase * external() const + ExternalValueBase * external() const noexcept { - return payload.external; + return getStorage(); } - const Bindings * attrs() const + const Bindings * attrs() const noexcept { - return payload.attrs; + return getStorage(); } - const PrimOp * primOp() const + const PrimOp * primOp() const noexcept { - return payload.primOp; + return getStorage(); } - bool boolean() const + bool boolean() const noexcept { - return payload.boolean; + return getStorage(); } - NixInt integer() const + NixInt integer() const noexcept { - return payload.integer; + return getStorage(); } - NixFloat fpoint() const + NixFloat fpoint() const noexcept { - return payload.fpoint; + return getStorage(); } - Lambda lambda() const + Lambda lambda() const noexcept { - return payload.lambda; + return getStorage(); } - ClosureThunk thunk() const + ClosureThunk thunk() const noexcept { - return payload.thunk; + return getStorage(); } - FunctionApplicationThunk primOpApp() const + PrimOpApplicationThunk primOpApp() const noexcept { - return payload.primOpApp; + return getStorage(); } - FunctionApplicationThunk app() const + FunctionApplicationThunk app() const noexcept { - return payload.app; + return getStorage(); } - const char * pathStr() const + const char * pathStr() const noexcept { - return payload.path.path; + return getStorage().path; } - SourceAccessor * pathAccessor() const + SourceAccessor * pathAccessor() const noexcept { - return payload.path.accessor; + return getStorage().accessor; } }; @@ -579,7 +674,7 @@ extern ExprBlackHole eBlackHole; bool Value::isBlackhole() const { - return isa() && thunk().expr == (Expr *) &eBlackHole; + return isThunk() && thunk().expr == (Expr *) &eBlackHole; } void Value::mkBlackhole() @@ -606,5 +701,4 @@ typedef std::shared_ptr RootValue; RootValue allocRootValue(Value * v); void forceNoNullByte(std::string_view s, std::function = nullptr); - } diff --git a/src/libexpr/primops.cc b/src/libexpr/primops.cc index 588b1fb6d51..40ff250a9af 100644 --- a/src/libexpr/primops.cc +++ b/src/libexpr/primops.cc @@ -5050,7 +5050,7 @@ void EvalState::createBaseEnv(const EvalSettings & evalSettings) /* Now that we've added all primops, sort the `builtins' set, because attribute lookups expect it to be sorted. */ - getBuiltins().payload.attrs->sort(); + const_cast(getBuiltins().attrs())->sort(); staticBaseEnv->sort(); From e73fcf7b5300c045a60e42c63e6d445b445426c4 Mon Sep 17 00:00:00 2001 From: Sergei Zimmerman Date: Wed, 2 Jul 2025 21:57:02 +0300 Subject: [PATCH 203/218] libexpr: Use proxy ListView for all Value list accesses This also makes it possible to make `payload` field private in the `ValueStorage` class template. --- src/libexpr-c/nix_api_value.cc | 2 +- src/libexpr-tests/primops.cc | 101 +++++++------ src/libexpr/attr-path.cc | 2 +- src/libexpr/eval-cache.cc | 2 +- src/libexpr/eval.cc | 14 +- src/libexpr/get-drvs.cc | 9 +- src/libexpr/include/nix/expr/value.hh | 199 ++++++++++++++++++++++++-- src/libexpr/primops.cc | 72 +++++----- src/libexpr/primops/context.cc | 2 +- src/libexpr/print-ambiguous.cc | 6 +- src/libexpr/print.cc | 4 +- src/libexpr/value-to-json.cc | 2 +- src/libexpr/value-to-xml.cc | 2 +- src/libflake/flake.cc | 2 +- src/nix-env/nix-env.cc | 2 +- src/nix/prefetch.cc | 4 +- 16 files changed, 309 insertions(+), 116 deletions(-) diff --git a/src/libexpr-c/nix_api_value.cc b/src/libexpr-c/nix_api_value.cc index 8afe35a4b8b..fb90e2872e6 100644 --- a/src/libexpr-c/nix_api_value.cc +++ b/src/libexpr-c/nix_api_value.cc @@ -324,7 +324,7 @@ nix_value * nix_get_list_byidx(nix_c_context * context, const nix_value * value, try { auto & v = check_value_in(value); assert(v.type() == nix::nList); - auto * p = v.listElems()[ix]; + auto * p = v.listView()[ix]; nix_gc_incref(nullptr, p); if (p != nullptr) state->state.forceValue(*p, nix::noPos); diff --git a/src/libexpr-tests/primops.cc b/src/libexpr-tests/primops.cc index 6c301f15762..9b5590d8d03 100644 --- a/src/libexpr-tests/primops.cc +++ b/src/libexpr-tests/primops.cc @@ -150,8 +150,8 @@ namespace nix { TEST_F(PrimOpTest, attrValues) { auto v = eval("builtins.attrValues { x = \"foo\"; a = 1; }"); ASSERT_THAT(v, IsListOfSize(2)); - ASSERT_THAT(*v.listElems()[0], IsIntEq(1)); - ASSERT_THAT(*v.listElems()[1], IsStringEq("foo")); + ASSERT_THAT(*v.listView()[0], IsIntEq(1)); + ASSERT_THAT(*v.listView()[1], IsStringEq("foo")); } TEST_F(PrimOpTest, getAttr) { @@ -250,8 +250,8 @@ namespace nix { TEST_F(PrimOpTest, catAttrs) { auto v = eval("builtins.catAttrs \"a\" [{a = 1;} {b = 0;} {a = 2;}]"); ASSERT_THAT(v, IsListOfSize(2)); - ASSERT_THAT(*v.listElems()[0], IsIntEq(1)); - ASSERT_THAT(*v.listElems()[1], IsIntEq(2)); + ASSERT_THAT(*v.listView()[0], IsIntEq(1)); + ASSERT_THAT(*v.listView()[1], IsIntEq(2)); } TEST_F(PrimOpTest, functionArgs) { @@ -320,7 +320,8 @@ namespace nix { TEST_F(PrimOpTest, tail) { auto v = eval("builtins.tail [ 3 2 1 0 ]"); ASSERT_THAT(v, IsListOfSize(3)); - for (const auto [n, elem] : enumerate(v.listItems())) + auto listView = v.listView(); + for (const auto [n, elem] : enumerate(listView)) ASSERT_THAT(*elem, IsIntEq(2 - static_cast(n))); } @@ -331,17 +332,17 @@ namespace nix { TEST_F(PrimOpTest, map) { auto v = eval("map (x: \"foo\" + x) [ \"bar\" \"bla\" \"abc\" ]"); ASSERT_THAT(v, IsListOfSize(3)); - auto elem = v.listElems()[0]; + auto elem = v.listView()[0]; ASSERT_THAT(*elem, IsThunk()); state.forceValue(*elem, noPos); ASSERT_THAT(*elem, IsStringEq("foobar")); - elem = v.listElems()[1]; + elem = v.listView()[1]; ASSERT_THAT(*elem, IsThunk()); state.forceValue(*elem, noPos); ASSERT_THAT(*elem, IsStringEq("foobla")); - elem = v.listElems()[2]; + elem = v.listView()[2]; ASSERT_THAT(*elem, IsThunk()); state.forceValue(*elem, noPos); ASSERT_THAT(*elem, IsStringEq("fooabc")); @@ -350,7 +351,7 @@ namespace nix { TEST_F(PrimOpTest, filter) { auto v = eval("builtins.filter (x: x == 2) [ 3 2 3 2 3 2 ]"); ASSERT_THAT(v, IsListOfSize(3)); - for (const auto elem : v.listItems()) + for (const auto elem : v.listView()) ASSERT_THAT(*elem, IsIntEq(2)); } @@ -367,7 +368,8 @@ namespace nix { TEST_F(PrimOpTest, concatLists) { auto v = eval("builtins.concatLists [[1 2] [3 4]]"); ASSERT_THAT(v, IsListOfSize(4)); - for (const auto [i, elem] : enumerate(v.listItems())) + auto listView = v.listView(); + for (const auto [i, elem] : enumerate(listView)) ASSERT_THAT(*elem, IsIntEq(static_cast(i)+1)); } @@ -405,7 +407,8 @@ namespace nix { auto v = eval("builtins.genList (x: x + 1) 3"); ASSERT_EQ(v.type(), nList); ASSERT_EQ(v.listSize(), 3u); - for (const auto [i, elem] : enumerate(v.listItems())) { + auto listView = v.listView(); + for (const auto [i, elem] : enumerate(listView)) { ASSERT_THAT(*elem, IsThunk()); state.forceValue(*elem, noPos); ASSERT_THAT(*elem, IsIntEq(static_cast(i)+1)); @@ -418,7 +421,8 @@ namespace nix { ASSERT_EQ(v.listSize(), 6u); const std::vector numbers = { 42, 77, 147, 249, 483, 526 }; - for (const auto [n, elem] : enumerate(v.listItems())) + auto listView = v.listView(); + for (const auto [n, elem] : enumerate(listView)) ASSERT_THAT(*elem, IsIntEq(numbers[n])); } @@ -429,17 +433,17 @@ namespace nix { auto right = v.attrs()->get(createSymbol("right")); ASSERT_NE(right, nullptr); ASSERT_THAT(*right->value, IsListOfSize(2)); - ASSERT_THAT(*right->value->listElems()[0], IsIntEq(23)); - ASSERT_THAT(*right->value->listElems()[1], IsIntEq(42)); + ASSERT_THAT(*right->value->listView()[0], IsIntEq(23)); + ASSERT_THAT(*right->value->listView()[1], IsIntEq(42)); auto wrong = v.attrs()->get(createSymbol("wrong")); ASSERT_NE(wrong, nullptr); ASSERT_EQ(wrong->value->type(), nList); ASSERT_EQ(wrong->value->listSize(), 3u); ASSERT_THAT(*wrong->value, IsListOfSize(3)); - ASSERT_THAT(*wrong->value->listElems()[0], IsIntEq(1)); - ASSERT_THAT(*wrong->value->listElems()[1], IsIntEq(9)); - ASSERT_THAT(*wrong->value->listElems()[2], IsIntEq(3)); + ASSERT_THAT(*wrong->value->listView()[0], IsIntEq(1)); + ASSERT_THAT(*wrong->value->listView()[1], IsIntEq(9)); + ASSERT_THAT(*wrong->value->listView()[2], IsIntEq(3)); } TEST_F(PrimOpTest, concatMap) { @@ -448,7 +452,8 @@ namespace nix { ASSERT_EQ(v.listSize(), 6u); const std::vector numbers = { 1, 2, 0, 3, 4, 0 }; - for (const auto [n, elem] : enumerate(v.listItems())) + auto listView = v.listView(); + for (const auto [n, elem] : enumerate(listView)) ASSERT_THAT(*elem, IsIntEq(numbers[n])); } @@ -682,7 +687,8 @@ namespace nix { ASSERT_THAT(v, IsListOfSize(4)); const std::vector strings = { "1", "2", "3", "git" }; - for (const auto [n, p] : enumerate(v.listItems())) + auto listView = v.listView(); + for (const auto [n, p] : enumerate(listView)) ASSERT_THAT(*p, IsStringEq(strings[n])); } @@ -772,12 +778,12 @@ namespace nix { auto v = eval("builtins.split \"(a)b\" \"abc\""); ASSERT_THAT(v, IsListOfSize(3)); - ASSERT_THAT(*v.listElems()[0], IsStringEq("")); + ASSERT_THAT(*v.listView()[0], IsStringEq("")); - ASSERT_THAT(*v.listElems()[1], IsListOfSize(1)); - ASSERT_THAT(*v.listElems()[1]->listElems()[0], IsStringEq("a")); + ASSERT_THAT(*v.listView()[1], IsListOfSize(1)); + ASSERT_THAT(*v.listView()[1]->listView()[0], IsStringEq("a")); - ASSERT_THAT(*v.listElems()[2], IsStringEq("c")); + ASSERT_THAT(*v.listView()[2], IsStringEq("c")); } TEST_F(PrimOpTest, split2) { @@ -785,17 +791,17 @@ namespace nix { auto v = eval("builtins.split \"([ac])\" \"abc\""); ASSERT_THAT(v, IsListOfSize(5)); - ASSERT_THAT(*v.listElems()[0], IsStringEq("")); + ASSERT_THAT(*v.listView()[0], IsStringEq("")); - ASSERT_THAT(*v.listElems()[1], IsListOfSize(1)); - ASSERT_THAT(*v.listElems()[1]->listElems()[0], IsStringEq("a")); + ASSERT_THAT(*v.listView()[1], IsListOfSize(1)); + ASSERT_THAT(*v.listView()[1]->listView()[0], IsStringEq("a")); - ASSERT_THAT(*v.listElems()[2], IsStringEq("b")); + ASSERT_THAT(*v.listView()[2], IsStringEq("b")); - ASSERT_THAT(*v.listElems()[3], IsListOfSize(1)); - ASSERT_THAT(*v.listElems()[3]->listElems()[0], IsStringEq("c")); + ASSERT_THAT(*v.listView()[3], IsListOfSize(1)); + ASSERT_THAT(*v.listView()[3]->listView()[0], IsStringEq("c")); - ASSERT_THAT(*v.listElems()[4], IsStringEq("")); + ASSERT_THAT(*v.listView()[4], IsStringEq("")); } TEST_F(PrimOpTest, split3) { @@ -803,36 +809,36 @@ namespace nix { ASSERT_THAT(v, IsListOfSize(5)); // First list element - ASSERT_THAT(*v.listElems()[0], IsStringEq("")); + ASSERT_THAT(*v.listView()[0], IsStringEq("")); // 2nd list element is a list [ "" null ] - ASSERT_THAT(*v.listElems()[1], IsListOfSize(2)); - ASSERT_THAT(*v.listElems()[1]->listElems()[0], IsStringEq("a")); - ASSERT_THAT(*v.listElems()[1]->listElems()[1], IsNull()); + ASSERT_THAT(*v.listView()[1], IsListOfSize(2)); + ASSERT_THAT(*v.listView()[1]->listView()[0], IsStringEq("a")); + ASSERT_THAT(*v.listView()[1]->listView()[1], IsNull()); // 3rd element - ASSERT_THAT(*v.listElems()[2], IsStringEq("b")); + ASSERT_THAT(*v.listView()[2], IsStringEq("b")); // 4th element is a list: [ null "c" ] - ASSERT_THAT(*v.listElems()[3], IsListOfSize(2)); - ASSERT_THAT(*v.listElems()[3]->listElems()[0], IsNull()); - ASSERT_THAT(*v.listElems()[3]->listElems()[1], IsStringEq("c")); + ASSERT_THAT(*v.listView()[3], IsListOfSize(2)); + ASSERT_THAT(*v.listView()[3]->listView()[0], IsNull()); + ASSERT_THAT(*v.listView()[3]->listView()[1], IsStringEq("c")); // 5th element is the empty string - ASSERT_THAT(*v.listElems()[4], IsStringEq("")); + ASSERT_THAT(*v.listView()[4], IsStringEq("")); } TEST_F(PrimOpTest, split4) { auto v = eval("builtins.split \"([[:upper:]]+)\" \" FOO \""); ASSERT_THAT(v, IsListOfSize(3)); - auto first = v.listElems()[0]; - auto second = v.listElems()[1]; - auto third = v.listElems()[2]; + auto first = v.listView()[0]; + auto second = v.listView()[1]; + auto third = v.listView()[2]; ASSERT_THAT(*first, IsStringEq(" ")); ASSERT_THAT(*second, IsListOfSize(1)); - ASSERT_THAT(*second->listElems()[0], IsStringEq("FOO")); + ASSERT_THAT(*second->listView()[0], IsStringEq("FOO")); ASSERT_THAT(*third, IsStringEq(" ")); } @@ -850,14 +856,14 @@ namespace nix { TEST_F(PrimOpTest, match3) { auto v = eval("builtins.match \"a(b)(c)\" \"abc\""); ASSERT_THAT(v, IsListOfSize(2)); - ASSERT_THAT(*v.listElems()[0], IsStringEq("b")); - ASSERT_THAT(*v.listElems()[1], IsStringEq("c")); + ASSERT_THAT(*v.listView()[0], IsStringEq("b")); + ASSERT_THAT(*v.listView()[1], IsStringEq("c")); } TEST_F(PrimOpTest, match4) { auto v = eval("builtins.match \"[[:space:]]+([[:upper:]]+)[[:space:]]+\" \" FOO \""); ASSERT_THAT(v, IsListOfSize(1)); - ASSERT_THAT(*v.listElems()[0], IsStringEq("FOO")); + ASSERT_THAT(*v.listView()[0], IsStringEq("FOO")); } TEST_F(PrimOpTest, match5) { @@ -874,7 +880,8 @@ namespace nix { // ensure that the list is sorted const std::vector expected { "a", "x", "y", "z" }; - for (const auto [n, elem] : enumerate(v.listItems())) + auto listView = v.listView(); + for (const auto [n, elem] : enumerate(listView)) ASSERT_THAT(*elem, IsStringEq(expected[n])); } diff --git a/src/libexpr/attr-path.cc b/src/libexpr/attr-path.cc index 722b57bbf39..111d04cf2c0 100644 --- a/src/libexpr/attr-path.cc +++ b/src/libexpr/attr-path.cc @@ -95,7 +95,7 @@ std::pair findAlongAttrPath(EvalState & state, const std::strin if (*attrIndex >= v->listSize()) throw AttrPathNotFound("list index %1% in selection path '%2%' is out of range", *attrIndex, attrPath); - v = v->listElems()[*attrIndex]; + v = v->listView()[*attrIndex]; pos = noPos; } diff --git a/src/libexpr/eval-cache.cc b/src/libexpr/eval-cache.cc index 0d732f59c4a..27d60d6ef49 100644 --- a/src/libexpr/eval-cache.cc +++ b/src/libexpr/eval-cache.cc @@ -721,7 +721,7 @@ std::vector AttrCursor::getListOfStrings() std::vector res; - for (auto & elem : v.listItems()) + for (auto elem : v.listView()) res.push_back(std::string(root->state.forceStringNoCtx(*elem, noPos, "while evaluating an attribute for caching"))); if (root->db) diff --git a/src/libexpr/eval.cc b/src/libexpr/eval.cc index fba1e666518..1321e00a5a5 100644 --- a/src/libexpr/eval.cc +++ b/src/libexpr/eval.cc @@ -2007,9 +2007,10 @@ void EvalState::concatLists(Value & v, size_t nrLists, Value * const * lists, co auto list = buildList(len); auto out = list.elems; for (size_t n = 0, pos = 0; n < nrLists; ++n) { - auto l = lists[n]->listSize(); + auto listView = lists[n]->listView(); + auto l = listView.size(); if (l) - memcpy(out + pos, lists[n]->listElems(), l * sizeof(Value *)); + memcpy(out + pos, listView.data(), l * sizeof(Value *)); pos += l; } v.mkList(list); @@ -2174,7 +2175,7 @@ void EvalState::forceValueDeep(Value & v) } else if (v.isList()) { - for (auto v2 : v.listItems()) + for (auto v2 : v.listView()) recurse(*v2); } }; @@ -2411,7 +2412,8 @@ BackedStringView EvalState::coerceToString( if (v.isList()) { std::string result; - for (auto [n, v2] : enumerate(v.listItems())) { + auto listView = v.listView(); + for (auto [n, v2] : enumerate(listView)) { try { result += *coerceToString(pos, *v2, context, "while evaluating one element of the list", @@ -2666,7 +2668,7 @@ void EvalState::assertEqValues(Value & v1, Value & v2, const PosIdx pos, std::st } for (size_t n = 0; n < v1.listSize(); ++n) { try { - assertEqValues(*v1.listElems()[n], *v2.listElems()[n], pos, errorCtx); + assertEqValues(*v1.listView()[n], *v2.listView()[n], pos, errorCtx); } catch (Error & e) { e.addTrace(positions[pos], "while comparing list element %d", n); throw; @@ -2818,7 +2820,7 @@ bool EvalState::eqValues(Value & v1, Value & v2, const PosIdx pos, std::string_v case nList: if (v1.listSize() != v2.listSize()) return false; for (size_t n = 0; n < v1.listSize(); ++n) - if (!eqValues(*v1.listElems()[n], *v2.listElems()[n], pos, errorCtx)) return false; + if (!eqValues(*v1.listView()[n], *v2.listView()[n], pos, errorCtx)) return false; return true; case nAttrs: { diff --git a/src/libexpr/get-drvs.cc b/src/libexpr/get-drvs.cc index f15ad4d7304..3c9ff9ff3c6 100644 --- a/src/libexpr/get-drvs.cc +++ b/src/libexpr/get-drvs.cc @@ -117,7 +117,7 @@ PackageInfo::Outputs PackageInfo::queryOutputs(bool withPaths, bool onlyOutputsT state->forceList(*i->value, i->pos, "while evaluating the 'outputs' attribute of a derivation"); /* For each output... */ - for (auto elem : i->value->listItems()) { + for (auto elem : i->value->listView()) { std::string output(state->forceStringNoCtx(*elem, i->pos, "while evaluating the name of an output of a derivation")); if (withPaths) { @@ -159,7 +159,7 @@ PackageInfo::Outputs PackageInfo::queryOutputs(bool withPaths, bool onlyOutputsT /* ^ this shows during `nix-env -i` right under the bad derivation */ if (!outTI->isList()) throw errMsg; Outputs result; - for (auto elem : outTI->listItems()) { + for (auto elem : outTI->listView()) { if (elem->type() != nString) throw errMsg; auto out = outputs.find(elem->c_str()); if (out == outputs.end()) throw errMsg; @@ -206,7 +206,7 @@ bool PackageInfo::checkMeta(Value & v) { state->forceValue(v, v.determinePos(noPos)); if (v.type() == nList) { - for (auto elem : v.listItems()) + for (auto elem : v.listView()) if (!checkMeta(*elem)) return false; return true; } @@ -400,7 +400,8 @@ static void getDerivations(EvalState & state, Value & vIn, } else if (v.type() == nList) { - for (auto [n, elem] : enumerate(v.listItems())) { + auto listView = v.listView(); + for (auto [n, elem] : enumerate(listView)) { std::string pathPrefix2 = addToPath(pathPrefix, fmt("%d", n)); if (getDerivation(state, *elem, pathPrefix2, drvs, done, ignoreAssertionFailures)) getDerivations(state, *elem, pathPrefix2, autoArgs, drvs, done, ignoreAssertionFailures); diff --git a/src/libexpr/include/nix/expr/value.hh b/src/libexpr/include/nix/expr/value.hh index 642459463b3..3f28aca0186 100644 --- a/src/libexpr/include/nix/expr/value.hh +++ b/src/libexpr/include/nix/expr/value.hh @@ -4,6 +4,7 @@ #include #include #include +#include #include "nix/expr/eval-gc.hh" #include "nix/expr/value/context.hh" @@ -317,12 +318,11 @@ protected: #undef NIX_VALUE_STORAGE_DEFINE_FIELD }; - Payload payload; - private: InternalType internalType = tUninitialized; + Payload payload; -public: +protected: #define NIX_VALUE_STORAGE_GET_IMPL(K, FIELD_NAME, DISCRIMINATOR) \ void getStorage(K & val) const noexcept \ { \ @@ -351,6 +351,189 @@ public: } }; +/** + * View into a list of Value * that is itself immutable. + * + * Since not all representations of ValueStorage can provide + * a pointer to a const array of Value * this proxy class either + * stores the small list inline or points to the big list. + */ +class ListView +{ + using SpanType = std::span; + using SmallList = detail::ValueBase::SmallList; + using List = detail::ValueBase::List; + + std::variant raw; + +public: + ListView(SmallList list) + : raw(list) + { + } + + ListView(List list) + : raw(list) + { + } + + Value * const * data() const & noexcept + { + return std::visit( + overloaded{ + [](const SmallList & list) { return list.data(); }, [](const List & list) { return list.elems; }}, + raw); + } + + std::size_t size() const noexcept + { + return std::visit( + overloaded{ + [](const SmallList & list) -> std::size_t { return list.back() == nullptr ? 1 : 2; }, + [](const List & list) -> std::size_t { return list.size; }}, + raw); + } + + Value * operator[](std::size_t i) const noexcept + { + return data()[i]; + } + + SpanType span() const & + { + return SpanType(data(), size()); + } + + /* Ensure that no dangling views can be created accidentally, as that + would lead to hard to diagnose bugs that only affect small lists. */ + SpanType span() && = delete; + Value * const * data() && noexcept = delete; + + /** + * Random-access iterator that only allows iterating over a constant range + * of mutable Value pointers. + * + * @note Not a pointer to minimize potential misuses and implicitly relying + * on the iterator being a pointer. + **/ + class iterator + { + public: + using value_type = Value *; + using pointer = const value_type *; + using reference = const value_type &; + using difference_type = std::ptrdiff_t; + using iterator_category = std::random_access_iterator_tag; + + private: + pointer ptr = nullptr; + + friend class ListView; + + iterator(pointer ptr) + : ptr(ptr) + { + } + + public: + iterator() = default; + + reference operator*() const + { + return *ptr; + } + + const value_type * operator->() const + { + return ptr; + } + + reference operator[](difference_type diff) const + { + return ptr[diff]; + } + + iterator & operator++() + { + ++ptr; + return *this; + } + + iterator operator++(int) + { + pointer tmp = ptr; + ++*this; + return iterator(tmp); + } + + iterator & operator--() + { + --ptr; + return *this; + } + + iterator operator--(int) + { + pointer tmp = ptr; + --*this; + return iterator(tmp); + } + + iterator & operator+=(difference_type diff) + { + ptr += diff; + return *this; + } + + iterator operator+(difference_type diff) const + { + return iterator(ptr + diff); + } + + friend iterator operator+(difference_type diff, const iterator & rhs) + { + return iterator(diff + rhs.ptr); + } + + iterator & operator-=(difference_type diff) + { + ptr -= diff; + return *this; + } + + iterator operator-(difference_type diff) const + { + return iterator(ptr - diff); + } + + difference_type operator-(const iterator & rhs) const + { + return ptr - rhs.ptr; + } + + std::strong_ordering operator<=>(const iterator & rhs) const = default; + }; + + using const_iterator = iterator; + + iterator begin() const & + { + return data(); + } + + iterator end() const & + { + return data() + size(); + } + + /* Ensure that no dangling iterators can be created accidentally, as that + would lead to hard to diagnose bugs that only affect small lists. */ + iterator begin() && = delete; + iterator end() && = delete; +}; + +static_assert(std::random_access_iterator); + struct Value : public ValueStorage { friend std::string showType(const Value & v); @@ -564,15 +747,9 @@ public: return isa(); } - std::span listItems() const noexcept - { - assert(isList()); - return std::span(listElems(), listSize()); - } - - Value * const * listElems() const noexcept + ListView listView() const noexcept { - return isa() ? payload.smallList.data() : getStorage().elems; + return isa() ? ListView(getStorage()) : ListView(getStorage()); } size_t listSize() const noexcept diff --git a/src/libexpr/primops.cc b/src/libexpr/primops.cc index 40ff250a9af..f9f834a62f0 100644 --- a/src/libexpr/primops.cc +++ b/src/libexpr/primops.cc @@ -415,7 +415,7 @@ void prim_importNative(EvalState & state, const PosIdx pos, Value * * args, Valu void prim_exec(EvalState & state, const PosIdx pos, Value * * args, Value & v) { state.forceList(*args[0], pos, "while evaluating the first argument passed to builtins.exec"); - auto elems = args[0]->listElems(); + auto elems = args[0]->listView(); auto count = args[0]->listSize(); if (count == 0) state.error("at least one argument to 'exec' required").atPos(pos).debugThrow(); @@ -660,8 +660,8 @@ struct CompareValues return false; } else if (i == v1->listSize()) { return true; - } else if (!state.eqValues(*v1->listElems()[i], *v2->listElems()[i], pos, errorCtx)) { - return (*this)(v1->listElems()[i], v2->listElems()[i], "while comparing two list elements"); + } else if (!state.eqValues(*v1->listView()[i], *v2->listView()[i], pos, errorCtx)) { + return (*this)(v1->listView()[i], v2->listView()[i], "while comparing two list elements"); } } default: @@ -689,7 +689,7 @@ static void prim_genericClosure(EvalState & state, const PosIdx pos, Value * * a state.forceList(*startSet->value, noPos, "while evaluating the 'startSet' attribute passed as argument to builtins.genericClosure"); ValueList workSet; - for (auto elem : startSet->value->listItems()) + for (auto elem : startSet->value->listView()) workSet.push_back(elem); if (startSet->value->listSize() == 0) { @@ -727,7 +727,7 @@ static void prim_genericClosure(EvalState & state, const PosIdx pos, Value * * a state.forceList(newElements, noPos, "while evaluating the return value of the `operator` passed to builtins.genericClosure"); /* Add the values returned by the operator to the work set. */ - for (auto elem : newElements.listItems()) { + for (auto elem : newElements.listView()) { state.forceValue(*elem, noPos); // "while evaluating one one of the elements returned by the `operator` passed to builtins.genericClosure"); workSet.push_back(elem); } @@ -1367,7 +1367,7 @@ static void derivationStrictInternal( command-line arguments to the builder. */ else if (i->name == state.sArgs) { state.forceList(*i->value, pos, context_below); - for (auto elem : i->value->listItems()) { + for (auto elem : i->value->listView()) { auto s = state.coerceToString(pos, *elem, context, "while evaluating an element of the argument list", true).toOwned(); @@ -1399,7 +1399,7 @@ static void derivationStrictInternal( /* Require ‘outputs’ to be a list of strings. */ state.forceList(*i->value, pos, context_below); Strings ss; - for (auto elem : i->value->listItems()) + for (auto elem : i->value->listView()) ss.emplace_back(state.forceStringNoCtx(*elem, pos, context_below)); handleOutputs(ss); } @@ -1882,7 +1882,7 @@ static void prim_findFile(EvalState & state, const PosIdx pos, Value * * args, V LookupPath lookupPath; - for (auto v2 : args[0]->listItems()) { + for (auto v2 : args[0]->listView()) { state.forceAttrs(*v2, pos, "while evaluating an element of the list passed to builtins.findFile"); std::string prefix; @@ -2920,7 +2920,7 @@ static void prim_removeAttrs(EvalState & state, const PosIdx pos, Value * * args // 64: large enough to fit the attributes of a derivation boost::container::small_vector names; names.reserve(args[1]->listSize()); - for (auto elem : args[1]->listItems()) { + for (auto elem : args[1]->listView()) { state.forceStringNoCtx(*elem, pos, "while evaluating the values of the second argument passed to builtins.removeAttrs"); names.emplace_back(state.symbols.create(elem->string_view()), nullptr); } @@ -2963,11 +2963,12 @@ static void prim_listToAttrs(EvalState & state, const PosIdx pos, Value * * args state.forceList(*args[0], pos, "while evaluating the argument passed to builtins.listToAttrs"); // Step 1. Sort the name-value attrsets in place using the memory we allocate for the result - size_t listSize = args[0]->listSize(); + auto listView = args[0]->listView(); + size_t listSize = listView.size(); auto & bindings = *state.allocBindings(listSize); using ElemPtr = decltype(&bindings[0].value); - for (const auto & [n, v2] : enumerate(args[0]->listItems())) { + for (const auto & [n, v2] : enumerate(listView)) { state.forceAttrs(*v2, pos, "while evaluating an element of the list passed to builtins.listToAttrs"); auto j = state.getAttr(state.sName, v2->attrs(), "in a {name=...; value=...;} pair"); @@ -3122,7 +3123,7 @@ static void prim_catAttrs(EvalState & state, const PosIdx pos, Value * * args, V SmallValueVector res(args[1]->listSize()); size_t found = 0; - for (auto v2 : args[1]->listItems()) { + for (auto v2 : args[1]->listView()) { state.forceAttrs(*v2, pos, "while evaluating an element in the list passed as second argument to builtins.catAttrs"); if (auto i = v2->attrs()->get(attrName)) res[found++] = i->value; @@ -3247,7 +3248,7 @@ static void prim_zipAttrsWith(EvalState & state, const PosIdx pos, Value * * arg state.forceFunction(*args[0], pos, "while evaluating the first argument passed to builtins.zipAttrsWith"); state.forceList(*args[1], pos, "while evaluating the second argument passed to builtins.zipAttrsWith"); - const auto listItems = args[1]->listItems(); + const auto listItems = args[1]->listView(); for (auto & vElem : listItems) { state.forceAttrs(*vElem, noPos, "while evaluating a value of the list passed as second argument to builtins.zipAttrsWith"); @@ -3346,8 +3347,8 @@ static void prim_elemAt(EvalState & state, const PosIdx pos, Value * * args, Val n, args[0]->listSize() ).atPos(pos).debugThrow(); - state.forceValue(*args[0]->listElems()[n], pos); - v = *args[0]->listElems()[n]; + state.forceValue(*args[0]->listView()[n], pos); + v = *args[0]->listView()[n]; } static RegisterPrimOp primop_elemAt({ @@ -3368,8 +3369,8 @@ static void prim_head(EvalState & state, const PosIdx pos, Value * * args, Value state.error( "'builtins.head' called on an empty list" ).atPos(pos).debugThrow(); - state.forceValue(*args[0]->listElems()[0], pos); - v = *args[0]->listElems()[0]; + state.forceValue(*args[0]->listView()[0], pos); + v = *args[0]->listView()[0]; } static RegisterPrimOp primop_head({ @@ -3394,7 +3395,7 @@ static void prim_tail(EvalState & state, const PosIdx pos, Value * * args, Value auto list = state.buildList(args[0]->listSize() - 1); for (const auto & [n, v] : enumerate(list)) - v = args[0]->listElems()[n + 1]; + v = args[0]->listView()[n + 1]; v.mkList(list); } @@ -3429,7 +3430,7 @@ static void prim_map(EvalState & state, const PosIdx pos, Value * * args, Value auto list = state.buildList(args[1]->listSize()); for (const auto & [n, v] : enumerate(list)) (v = state.allocValue())->mkApp( - args[0], args[1]->listElems()[n]); + args[0], args[1]->listView()[n]); v.mkList(list); } @@ -3470,9 +3471,9 @@ static void prim_filter(EvalState & state, const PosIdx pos, Value * * args, Val bool same = true; for (size_t n = 0; n < len; ++n) { Value res; - state.callFunction(*args[0], *args[1]->listElems()[n], res, noPos); + state.callFunction(*args[0], *args[1]->listView()[n], res, noPos); if (state.forceBool(res, pos, "while evaluating the return value of the filtering function passed to builtins.filter")) - vs[k++] = args[1]->listElems()[n]; + vs[k++] = args[1]->listView()[n]; else same = false; } @@ -3501,7 +3502,7 @@ static void prim_elem(EvalState & state, const PosIdx pos, Value * * args, Value { bool res = false; state.forceList(*args[1], pos, "while evaluating the second argument passed to builtins.elem"); - for (auto elem : args[1]->listItems()) + for (auto elem : args[1]->listView()) if (state.eqValues(*args[0], *elem, pos, "while searching for the presence of the given element in the list")) { res = true; break; @@ -3523,7 +3524,8 @@ static RegisterPrimOp primop_elem({ static void prim_concatLists(EvalState & state, const PosIdx pos, Value * * args, Value & v) { state.forceList(*args[0], pos, "while evaluating the first argument passed to builtins.concatLists"); - state.concatLists(v, args[0]->listSize(), args[0]->listElems(), pos, "while evaluating a value of the list passed to builtins.concatLists"); + auto listView = args[0]->listView(); + state.concatLists(v, args[0]->listSize(), listView.data(), pos, "while evaluating a value of the list passed to builtins.concatLists"); } static RegisterPrimOp primop_concatLists({ @@ -3561,7 +3563,8 @@ static void prim_foldlStrict(EvalState & state, const PosIdx pos, Value * * args if (args[2]->listSize()) { Value * vCur = args[1]; - for (auto [n, elem] : enumerate(args[2]->listItems())) { + auto listView = args[2]->listView(); + for (auto [n, elem] : enumerate(listView)) { Value * vs []{vCur, elem}; vCur = n == args[2]->listSize() - 1 ? &v : state.allocValue(); state.callFunction(*args[0], vs, *vCur, pos); @@ -3603,7 +3606,7 @@ static void anyOrAll(bool any, EvalState & state, const PosIdx pos, Value * * ar : "while evaluating the return value of the function passed to builtins.all"; Value vTmp; - for (auto elem : args[1]->listItems()) { + for (auto elem : args[1]->listView()) { state.callFunction(*args[0], *elem, vTmp, pos); bool res = state.forceBool(vTmp, pos, errorCtx); if (res == any) { @@ -3701,7 +3704,7 @@ static void prim_sort(EvalState & state, const PosIdx pos, Value * * args, Value auto list = state.buildList(len); for (const auto & [n, v] : enumerate(list)) - state.forceValue(*(v = args[1]->listElems()[n]), pos); + state.forceValue(*(v = args[1]->listView()[n]), pos); auto comparator = [&](Value * a, Value * b) { /* Optimization: if the comparator is lessThan, bypass @@ -3787,7 +3790,7 @@ static void prim_partition(EvalState & state, const PosIdx pos, Value * * args, ValueVector right, wrong; for (size_t n = 0; n < len; ++n) { - auto vElem = args[1]->listElems()[n]; + auto vElem = args[1]->listView()[n]; state.forceValue(*vElem, pos); Value res; state.callFunction(*args[0], *vElem, res, pos); @@ -3844,7 +3847,7 @@ static void prim_groupBy(EvalState & state, const PosIdx pos, Value * * args, Va ValueVectorMap attrs; - for (auto vElem : args[1]->listItems()) { + for (auto vElem : args[1]->listView()) { Value res; state.callFunction(*args[0], *vElem, res, pos); auto name = state.forceStringNoCtx(res, pos, "while evaluating the return value of the grouping function passed to builtins.groupBy"); @@ -3900,7 +3903,7 @@ static void prim_concatMap(EvalState & state, const PosIdx pos, Value * * args, size_t len = 0; for (size_t n = 0; n < nrLists; ++n) { - Value * vElem = args[1]->listElems()[n]; + Value * vElem = args[1]->listView()[n]; state.callFunction(*args[0], *vElem, lists[n], pos); state.forceList(lists[n], lists[n].determinePos(args[0]->determinePos(pos)), "while evaluating the return value of the function passed to builtins.concatMap"); len += lists[n].listSize(); @@ -3909,9 +3912,10 @@ static void prim_concatMap(EvalState & state, const PosIdx pos, Value * * args, auto list = state.buildList(len); auto out = list.elems; for (size_t n = 0, pos = 0; n < nrLists; ++n) { - auto l = lists[n].listSize(); + auto listView = lists[n].listView(); + auto l = listView.size(); if (l) - memcpy(out + pos, lists[n].listElems(), l * sizeof(Value *)); + memcpy(out + pos, listView.data(), l * sizeof(Value *)); pos += l; } v.mkList(list); @@ -4587,7 +4591,7 @@ static void prim_concatStringsSep(EvalState & state, const PosIdx pos, Value * * res.reserve((args[1]->listSize() + 32) * sep.size()); bool first = true; - for (auto elem : args[1]->listItems()) { + for (auto elem : args[1]->listView()) { if (first) first = false; else res += sep; res += *state.coerceToString(pos, *elem, context, "while evaluating one element of the list of strings to concat passed to builtins.concatStringsSep"); } @@ -4617,11 +4621,11 @@ static void prim_replaceStrings(EvalState & state, const PosIdx pos, Value * * a std::vector from; from.reserve(args[0]->listSize()); - for (auto elem : args[0]->listItems()) + for (auto elem : args[0]->listView()) from.emplace_back(state.forceString(*elem, pos, "while evaluating one of the strings to replace passed to builtins.replaceStrings")); std::unordered_map cache; - auto to = args[1]->listItems(); + auto to = args[1]->listView(); NixStringContext context; auto s = state.forceString(*args[2], context, pos, "while evaluating the third argument passed to builtins.replaceStrings"); diff --git a/src/libexpr/primops/context.cc b/src/libexpr/primops/context.cc index 496678884af..56962d6a872 100644 --- a/src/libexpr/primops/context.cc +++ b/src/libexpr/primops/context.cc @@ -312,7 +312,7 @@ static void prim_appendContext(EvalState & state, const PosIdx pos, Value * * ar name ).atPos(i.pos).debugThrow(); } - for (auto elem : attr->value->listItems()) { + for (auto elem : attr->value->listView()) { auto outputName = state.forceStringNoCtx(*elem, attr->pos, "while evaluating an output name within a string context"); context.emplace(NixStringContextElem::Built { .drvPath = makeConstantStorePathRef(namePath), diff --git a/src/libexpr/print-ambiguous.cc b/src/libexpr/print-ambiguous.cc index 0646783c268..2a0b009ebfb 100644 --- a/src/libexpr/print-ambiguous.cc +++ b/src/libexpr/print-ambiguous.cc @@ -50,11 +50,13 @@ void printAmbiguous( break; } case nList: - if (seen && v.listSize() && !seen->insert(v.listElems()).second) + /* Use pointer to the Value instead of pointer to the elements, because + that would need to explicitly handle the case of SmallList. */ + if (seen && v.listSize() && !seen->insert(&v).second) str << "«repeated»"; else { str << "[ "; - for (auto v2 : v.listItems()) { + for (auto v2 : v.listView()) { if (v2) printAmbiguous(*v2, symbols, str, seen, depth - 1); else diff --git a/src/libexpr/print.cc b/src/libexpr/print.cc index f0e0eed2de3..1f0c592c157 100644 --- a/src/libexpr/print.cc +++ b/src/libexpr/print.cc @@ -415,8 +415,8 @@ class Printer if (depth < options.maxDepth) { increaseIndent(); output << "["; - auto listItems = v.listItems(); - auto prettyPrint = shouldPrettyPrintList(listItems); + auto listItems = v.listView(); + auto prettyPrint = shouldPrettyPrintList(listItems.span()); size_t currentListItemsPrinted = 0; diff --git a/src/libexpr/value-to-json.cc b/src/libexpr/value-to-json.cc index 4c0667d9e50..a9b51afa074 100644 --- a/src/libexpr/value-to-json.cc +++ b/src/libexpr/value-to-json.cc @@ -73,7 +73,7 @@ json printValueAsJSON(EvalState & state, bool strict, case nList: { out = json::array(); int i = 0; - for (auto elem : v.listItems()) { + for (auto elem : v.listView()) { try { out.push_back(printValueAsJSON(state, strict, *elem, pos, context, copyToStore)); } catch (Error & e) { diff --git a/src/libexpr/value-to-xml.cc b/src/libexpr/value-to-xml.cc index 54ff06f9e1a..235ef262760 100644 --- a/src/libexpr/value-to-xml.cc +++ b/src/libexpr/value-to-xml.cc @@ -114,7 +114,7 @@ static void printValueAsXML(EvalState & state, bool strict, bool location, case nList: { XMLOpenElement _(doc, "list"); - for (auto v2 : v.listItems()) + for (auto v2 : v.listView()) printValueAsXML(state, strict, location, *v2, doc, context, drvsSeen, pos); break; } diff --git a/src/libflake/flake.cc b/src/libflake/flake.cc index e59649fecb9..322abaa4a52 100644 --- a/src/libflake/flake.cc +++ b/src/libflake/flake.cc @@ -289,7 +289,7 @@ static Flake readFlake( Explicit { state.forceBool(*setting.value, setting.pos, "") }); else if (setting.value->type() == nList) { std::vector ss; - for (auto elem : setting.value->listItems()) { + for (auto elem : setting.value->listView()) { if (elem->type() != nString) state.error("list element in flake configuration setting '%s' is %s while a string is expected", state.symbols[setting.name], showType(*setting.value)).debugThrow(); diff --git a/src/nix-env/nix-env.cc b/src/nix-env/nix-env.cc index 25ff39e38d7..fd48e67dce4 100644 --- a/src/nix-env/nix-env.cc +++ b/src/nix-env/nix-env.cc @@ -1265,7 +1265,7 @@ static void opQuery(Globals & globals, Strings opFlags, Strings opArgs) } else if (v->type() == nList) { attrs2["type"] = "strings"; XMLOpenElement m(xml, "meta", attrs2); - for (auto elem : v->listItems()) { + for (auto elem : v->listView()) { if (elem->type() != nString) continue; XMLAttrs attrs3; attrs3["value"] = elem->c_str(); diff --git a/src/nix/prefetch.cc b/src/nix/prefetch.cc index 9e5e3c09a78..96dcdb4e87a 100644 --- a/src/nix/prefetch.cc +++ b/src/nix/prefetch.cc @@ -46,7 +46,7 @@ std::string resolveMirrorUrl(EvalState & state, const std::string & url) if (mirrorList->value->listSize() < 1) throw Error("mirror URL '%s' did not expand to anything", url); - std::string mirror(state.forceString(*mirrorList->value->listElems()[0], noPos, "while evaluating the first available mirror")); + std::string mirror(state.forceString(*mirrorList->value->listView()[0], noPos, "while evaluating the first available mirror")); return mirror + (hasSuffix(mirror, "/") ? "" : "/") + s.substr(p + 1); } @@ -221,7 +221,7 @@ static int main_nix_prefetch_url(int argc, char * * argv) state->forceList(*attr->value, noPos, "while evaluating the urls to prefetch"); if (attr->value->listSize() < 1) throw Error("'urls' list is empty"); - url = state->forceString(*attr->value->listElems()[0], noPos, "while evaluating the first url from the urls list"); + url = state->forceString(*attr->value->listView()[0], noPos, "while evaluating the first url from the urls list"); /* Extract the hash mode. */ auto attr2 = v.attrs()->get(state->symbols.create("outputHashMode")); From 5a20a48f1367234b52dca9569955f825a0d626b0 Mon Sep 17 00:00:00 2001 From: Sergei Zimmerman Date: Mon, 30 Jun 2025 22:03:07 +0300 Subject: [PATCH 204/218] libexpr: Reduce the size of Value down to 16 bytes This shaves off a very significand amount of memory used for evaluation as well as reduces the GC-managed heap. Previously the union discriminator (InternalType) was stored as a separate field in the Value, which takes up whole 8 bytes due to padding needed for member alignment. This effectively wasted 7 whole bytes of memory. Instead of doing that InternalType is instead packed into pointer alignment niches. As it turns out, there's more than enough unused bits there for the bit packing to be effective. See the doxygen comment in the ValueStorage specialization for more details. This does not add any performance overhead, even though we now consistently assert the InternalType in all getters. This can also be made atomic with a double width compare-and-swap instruction on x86_64 (CMPXCHG16B instruction) for parallel evaluation. --- src/libexpr/eval-gc.cc | 8 + src/libexpr/include/nix/expr/value.hh | 311 +++++++++++++++++++++++++- 2 files changed, 310 insertions(+), 9 deletions(-) diff --git a/src/libexpr/eval-gc.cc b/src/libexpr/eval-gc.cc index bec6680017e..5a4ecf03575 100644 --- a/src/libexpr/eval-gc.cc +++ b/src/libexpr/eval-gc.cc @@ -4,6 +4,7 @@ #include "nix/util/config-global.hh" #include "nix/util/serialise.hh" #include "nix/expr/eval-gc.hh" +#include "nix/expr/value.hh" #include "expr-config-private.hh" @@ -52,6 +53,13 @@ static inline void initGCReal() GC_INIT(); + /* Register valid displacements in case we are using alignment niches + for storing the type information. This way tagged pointers are considered + to be valid, even when they are not aligned. */ + if constexpr (detail::useBitPackedValueStorage) + for (std::size_t i = 1; i < sizeof(std::uintptr_t); ++i) + GC_register_displacement(i); + GC_set_oom_fn(oomHandler); /* Set the initial heap size to something fairly big (25% of diff --git a/src/libexpr/include/nix/expr/value.hh b/src/libexpr/include/nix/expr/value.hh index 3f28aca0186..098effa29d1 100644 --- a/src/libexpr/include/nix/expr/value.hh +++ b/src/libexpr/include/nix/expr/value.hh @@ -19,23 +19,35 @@ namespace nix { struct Value; class BindingsBuilder; +/** + * Internal type discriminator, which is more detailed than `ValueType`, as + * it specifies the exact representation used (for types that have multiple + * possible representations). + * + * @warning The ordering is very significant. See ValueStorage::getInternalType() for details + * about how this is mapped into the alignment bits to save significant memory. + * This also restricts the number of internal types represented with distinct memory layouts. + */ typedef enum { tUninitialized = 0, + /* layout: Single/zero field payload */ tInt = 1, tBool, - tString, - tPath, tNull, + tFloat, + tExternal, + tPrimOp, tAttrs, + /* layout: Pair of pointers payload */ tListSmall, - tListN, - tThunk, + tPrimOpApp, tApp, + tThunk, tLambda, - tPrimOp, - tPrimOpApp, - tExternal, - tFloat + /* layout: Single untaggable field */ + tListN, + tString, + tPath, } InternalType; /** @@ -307,7 +319,7 @@ inline constexpr InternalType payloadTypeToInternalType = PayloadTypeToInternalT * All specializations of this type need to implement getStorage, setStorage and * getInternalType methods. */ -template +template class ValueStorage : public detail::ValueBase { protected: @@ -351,6 +363,287 @@ protected: } }; +namespace detail { + +/* Whether to use a specialization of ValueStorage that does bitpacking into + alignment niches. */ +template +inline constexpr bool useBitPackedValueStorage = (ptrSize == 8) && (__STDCPP_DEFAULT_NEW_ALIGNMENT__ >= 8); + +} // namespace detail + +/** + * Value storage that is optimized for 64 bit systems. + * Packs discriminator bits into the pointer alignment niches. + */ +template +class ValueStorage>> : public detail::ValueBase +{ + /* Needs a dependent type name in order for member functions (and + * potentially ill-formed bit casts) to be SFINAE'd out. + * + * Otherwise some member functions could possibly be instantiated for 32 bit + * systems and fail due to an unsatisfied constraint. + */ + template + struct PackedPointerTypeStruct + { + using type = std::uint64_t; + }; + + using PackedPointer = typename PackedPointerTypeStruct::type; + using Payload = std::array; + Payload payload = {}; + + static constexpr int discriminatorBits = 3; + static constexpr PackedPointer discriminatorMask = (PackedPointer(1) << discriminatorBits) - 1; + + /** + * The value is stored as a pair of 8-byte double words. All pointers are assumed + * to be 8-byte aligned. This gives us at most 6 bits of discriminator bits + * of free storage. In some cases when one double word can't be tagged the whole + * discriminator is stored in the first double word. + * + * The layout of discriminator bits is determined by the 3 bits of PrimaryDiscriminator, + * which are always stored in the lower 3 bits of the first dword of the payload. + * The memory layout has 3 types depending on the PrimaryDiscriminator value. + * + * PrimaryDiscriminator::pdSingleDWord - Only the second dword carries the data. + * That leaves the first 8 bytes free for storing the InternalType in the upper + * bits. + * + * PrimaryDiscriminator::pdListN - pdPath - Only has 3 available padding bits + * because: + * - tListN needs a size, whose lower bits we can't borrow. + * - tString and tPath have C-string fields, which don't necessarily need to + * be aligned. + * + * In this case we reserve their discriminators directly in the PrimaryDiscriminator + * bits stored in payload[0]. + * + * PrimaryDiscriminator::pdPairOfPointers - Payloads that consist of a pair of pointers. + * In this case the 3 lower bits of payload[1] can be tagged. + * + * The primary discriminator with value 0 is reserved for uninitialized Values, + * which are useful for diagnostics in C bindings. + */ + enum PrimaryDiscriminator : int { + pdUninitialized = 0, + pdSingleDWord, //< layout: Single/zero field payload + /* The order of these enumations must be the same as in InternalType. */ + pdListN, //< layout: Single untaggable field. + pdString, + pdPath, + pdPairOfPointers, //< layout: Pair of pointers payload + }; + + template + requires std::is_pointer_v + static T untagPointer(PackedPointer val) noexcept + { + return std::bit_cast(val & ~discriminatorMask); + } + + PrimaryDiscriminator getPrimaryDiscriminator() const noexcept + { + return static_cast(payload[0] & discriminatorMask); + } + + static void assertAligned(PackedPointer val) noexcept + { + assert((val & discriminatorMask) == 0 && "Pointer is not 8 bytes aligned"); + } + + template + void setSingleDWordPayload(PackedPointer untaggedVal) noexcept + { + /* There's plenty of free upper bits in the first dword, which is + used only for the discriminator. */ + payload[0] = static_cast(pdSingleDWord) | (static_cast(type) << discriminatorBits); + payload[1] = untaggedVal; + } + + template + void setUntaggablePayload(T * firstPtrField, U untaggableField) noexcept + { + static_assert(discriminator >= pdListN && discriminator <= pdPath); + auto firstFieldPayload = std::bit_cast(firstPtrField); + assertAligned(firstFieldPayload); + payload[0] = static_cast(discriminator) | firstFieldPayload; + payload[1] = std::bit_cast(untaggableField); + } + + template + void setPairOfPointersPayload(T * firstPtrField, U * secondPtrField) noexcept + { + static_assert(type >= tListSmall && type <= tLambda); + { + auto firstFieldPayload = std::bit_cast(firstPtrField); + assertAligned(firstFieldPayload); + payload[0] = static_cast(pdPairOfPointers) | firstFieldPayload; + } + { + auto secondFieldPayload = std::bit_cast(secondPtrField); + assertAligned(secondFieldPayload); + payload[1] = (type - tListSmall) | secondFieldPayload; + } + } + + template + requires std::is_pointer_v && std::is_pointer_v + void getPairOfPointersPayload(T & firstPtrField, U & secondPtrField) const noexcept + { + firstPtrField = untagPointer(payload[0]); + secondPtrField = untagPointer(payload[1]); + } + +protected: + /** Get internal type currently occupying the storage. */ + InternalType getInternalType() const noexcept + { + switch (auto pd = getPrimaryDiscriminator()) { + case pdUninitialized: + /* Discriminator value of zero is used to distinguish uninitialized values. */ + return tUninitialized; + case pdSingleDWord: + /* Payloads that only use up a single double word store the InternalType + in the upper bits of the first double word. */ + return InternalType(payload[0] >> discriminatorBits); + /* The order must match that of the enumerations defined in InternalType. */ + case pdListN: + case pdString: + case pdPath: + return static_cast(tListN + (pd - pdListN)); + case pdPairOfPointers: + return static_cast(tListSmall + (payload[1] & discriminatorMask)); + [[unlikely]] default: + unreachable(); + } + } + +#define NIX_VALUE_STORAGE_DEF_PAIR_OF_PTRS(TYPE, MEMBER_A, MEMBER_B) \ + \ + void getStorage(TYPE & val) const noexcept \ + { \ + getPairOfPointersPayload(val MEMBER_A, val MEMBER_B); \ + } \ + \ + void setStorage(TYPE val) noexcept \ + { \ + setPairOfPointersPayload>(val MEMBER_A, val MEMBER_B); \ + } + + NIX_VALUE_STORAGE_DEF_PAIR_OF_PTRS(SmallList, [0], [1]) + NIX_VALUE_STORAGE_DEF_PAIR_OF_PTRS(PrimOpApplicationThunk, .left, .right) + NIX_VALUE_STORAGE_DEF_PAIR_OF_PTRS(FunctionApplicationThunk, .left, .right) + NIX_VALUE_STORAGE_DEF_PAIR_OF_PTRS(ClosureThunk, .env, .expr) + NIX_VALUE_STORAGE_DEF_PAIR_OF_PTRS(Lambda, .env, .fun) + +#undef NIX_VALUE_STORAGE_DEF_PAIR_OF_PTRS + + void getStorage(NixInt & integer) const noexcept + { + /* PackedPointerType -> int64_t here is well-formed, since the standard requires + this conversion to follow 2's complement rules. This is just a no-op. */ + integer = NixInt(payload[1]); + } + + void getStorage(bool & boolean) const noexcept + { + boolean = payload[1]; + } + + void getStorage(Null & null) const noexcept {} + + void getStorage(NixFloat & fpoint) const noexcept + { + fpoint = std::bit_cast(payload[1]); + } + + void getStorage(ExternalValueBase *& external) const noexcept + { + external = std::bit_cast(payload[1]); + } + + void getStorage(PrimOp *& primOp) const noexcept + { + primOp = std::bit_cast(payload[1]); + } + + void getStorage(Bindings *& attrs) const noexcept + { + attrs = std::bit_cast(payload[1]); + } + + void getStorage(List & list) const noexcept + { + list.elems = untagPointer(payload[0]); + list.size = payload[1]; + } + + void getStorage(StringWithContext & string) const noexcept + { + string.context = untagPointer(payload[0]); + string.c_str = std::bit_cast(payload[1]); + } + + void getStorage(Path & path) const noexcept + { + path.accessor = untagPointer(payload[0]); + path.path = std::bit_cast(payload[1]); + } + + void setStorage(NixInt integer) noexcept + { + setSingleDWordPayload(integer.value); + } + + void setStorage(bool boolean) noexcept + { + setSingleDWordPayload(boolean); + } + + void setStorage(Null path) noexcept + { + setSingleDWordPayload(0); + } + + void setStorage(NixFloat fpoint) noexcept + { + setSingleDWordPayload(std::bit_cast(fpoint)); + } + + void setStorage(ExternalValueBase * external) noexcept + { + setSingleDWordPayload(std::bit_cast(external)); + } + + void setStorage(PrimOp * primOp) noexcept + { + setSingleDWordPayload(std::bit_cast(primOp)); + } + + void setStorage(Bindings * bindings) noexcept + { + setSingleDWordPayload(std::bit_cast(bindings)); + } + + void setStorage(List list) noexcept + { + setUntaggablePayload(list.elems, list.size); + } + + void setStorage(StringWithContext string) noexcept + { + setUntaggablePayload(string.context, string.c_str); + } + + void setStorage(Path path) noexcept + { + setUntaggablePayload(path.accessor, path.path); + } +}; + /** * View into a list of Value * that is itself immutable. * From eb97d8c17048b075acebcba7b574611151a79cd7 Mon Sep 17 00:00:00 2001 From: Eelco Dolstra Date: Fri, 4 Jul 2025 15:21:31 +0200 Subject: [PATCH 205/218] Fix indentation of "Unable to start any build" error message The use of R"(...)" added a bunch of unnecessary whitespace, e.g. error: Unable to start any build; either increase '--max-jobs' or enable remote builds. For more information run 'man nix.conf' and search for '/machines'. Now we get error: Unable to start any build; either increase '--max-jobs' or enable remote builds. For more information run 'man nix.conf' and search for '/machines'. --- src/libstore/build/worker.cc | 21 ++++++--------------- 1 file changed, 6 insertions(+), 15 deletions(-) diff --git a/src/libstore/build/worker.cc b/src/libstore/build/worker.cc index 12d4aaa2d74..a3419d9da45 100644 --- a/src/libstore/build/worker.cc +++ b/src/libstore/build/worker.cc @@ -362,23 +362,14 @@ void Worker::run(const Goals & _topGoals) else if (awake.empty() && 0U == settings.maxBuildJobs) { if (getMachines().empty()) throw Error( - R"( - Unable to start any build; - either increase '--max-jobs' or enable remote builds. - - For more information run 'man nix.conf' and search for '/machines'. - )" - ); + "Unable to start any build; either increase '--max-jobs' or enable remote builds.\n" + "\n" + "For more information run 'man nix.conf' and search for '/machines'."); else throw Error( - R"( - Unable to start any build; - remote machines may not have all required system features. - - For more information run 'man nix.conf' and search for '/machines'. - )" - ); - + "Unable to start any build; remote machines may not have all required system features.\n" + "\n" + "For more information run 'man nix.conf' and search for '/machines'."); } else assert(!awake.empty()); } From af05ce0f6d6543c48d1f182023888083359467bf Mon Sep 17 00:00:00 2001 From: Eelco Dolstra Date: Fri, 4 Jul 2025 15:39:47 +0200 Subject: [PATCH 206/218] queryMissing(): Return a struct ...instead of having a bunch of pass-by-reference arguments. --- src/libmain/shared.cc | 6 ++--- src/libstore/build/worker.cc | 4 +-- src/libstore/daemon.cc | 12 ++++----- .../include/nix/store/remote-store.hh | 4 +-- src/libstore/include/nix/store/store-api.hh | 16 +++++++++--- src/libstore/misc.cc | 26 ++++++++----------- src/libstore/remote-store.cc | 18 ++++++------- src/libstore/restricted-store.cc | 24 +++++++---------- src/libstore/store-api.cc | 9 +++---- src/nix-build/nix-build.cc | 7 ++--- src/nix-store/nix-store.cc | 12 +++------ 11 files changed, 59 insertions(+), 79 deletions(-) diff --git a/src/libmain/shared.cc b/src/libmain/shared.cc index 3b431f02135..75c9ebadd81 100644 --- a/src/libmain/shared.cc +++ b/src/libmain/shared.cc @@ -46,10 +46,8 @@ void printGCWarning() void printMissing(ref store, const std::vector & paths, Verbosity lvl) { - uint64_t downloadSize, narSize; - StorePathSet willBuild, willSubstitute, unknown; - store->queryMissing(paths, willBuild, willSubstitute, unknown, downloadSize, narSize); - printMissing(store, willBuild, willSubstitute, unknown, downloadSize, narSize, lvl); + auto missing = store->queryMissing(paths); + printMissing(store, missing.willBuild, missing.willSubstitute, missing.unknown, missing.downloadSize, missing.narSize, lvl); } diff --git a/src/libstore/build/worker.cc b/src/libstore/build/worker.cc index 12d4aaa2d74..8e10e948aed 100644 --- a/src/libstore/build/worker.cc +++ b/src/libstore/build/worker.cc @@ -324,9 +324,7 @@ void Worker::run(const Goals & _topGoals) } /* Call queryMissing() to efficiently query substitutes. */ - StorePathSet willBuild, willSubstitute, unknown; - uint64_t downloadSize, narSize; - store.queryMissing(topPaths, willBuild, willSubstitute, unknown, downloadSize, narSize); + store.queryMissing(topPaths); debug("entered goal loop"); diff --git a/src/libstore/daemon.cc b/src/libstore/daemon.cc index dfc068bc775..bf4a9d95906 100644 --- a/src/libstore/daemon.cc +++ b/src/libstore/daemon.cc @@ -948,14 +948,12 @@ static void performOp(TunnelLogger * logger, ref store, case WorkerProto::Op::QueryMissing: { auto targets = WorkerProto::Serialise::read(*store, rconn); logger->startWork(); - StorePathSet willBuild, willSubstitute, unknown; - uint64_t downloadSize, narSize; - store->queryMissing(targets, willBuild, willSubstitute, unknown, downloadSize, narSize); + auto missing = store->queryMissing(targets); logger->stopWork(); - WorkerProto::write(*store, wconn, willBuild); - WorkerProto::write(*store, wconn, willSubstitute); - WorkerProto::write(*store, wconn, unknown); - conn.to << downloadSize << narSize; + WorkerProto::write(*store, wconn, missing.willBuild); + WorkerProto::write(*store, wconn, missing.willSubstitute); + WorkerProto::write(*store, wconn, missing.unknown); + conn.to << missing.downloadSize << missing.narSize; break; } diff --git a/src/libstore/include/nix/store/remote-store.hh b/src/libstore/include/nix/store/remote-store.hh index dd2396fe32b..18c02456f4c 100644 --- a/src/libstore/include/nix/store/remote-store.hh +++ b/src/libstore/include/nix/store/remote-store.hh @@ -149,9 +149,7 @@ struct RemoteStore : void addSignatures(const StorePath & storePath, const StringSet & sigs) override; - void queryMissing(const std::vector & targets, - StorePathSet & willBuild, StorePathSet & willSubstitute, StorePathSet & unknown, - uint64_t & downloadSize, uint64_t & narSize) override; + MissingPaths queryMissing(const std::vector & targets) override; void addBuildLog(const StorePath & drvPath, std::string_view log) override; diff --git a/src/libstore/include/nix/store/store-api.hh b/src/libstore/include/nix/store/store-api.hh index 72ca1f437fd..e0a3e67d13b 100644 --- a/src/libstore/include/nix/store/store-api.hh +++ b/src/libstore/include/nix/store/store-api.hh @@ -71,6 +71,18 @@ struct KeyedBuildResult; typedef std::map> StorePathCAMap; +/** + * Information about what paths will be built or substituted, returned + * by Store::queryMissing(). + */ +struct MissingPaths +{ + StorePathSet willBuild; + StorePathSet willSubstitute; + StorePathSet unknown; + uint64_t downloadSize{0}; + uint64_t narSize{0}; +}; /** * About the class hierarchy of the store types: @@ -694,9 +706,7 @@ public: * derivations that will be built, and the set of output paths that * will be substituted. */ - virtual void queryMissing(const std::vector & targets, - StorePathSet & willBuild, StorePathSet & willSubstitute, StorePathSet & unknown, - uint64_t & downloadSize, uint64_t & narSize); + virtual MissingPaths queryMissing(const std::vector & targets); /** * Sort a set of paths topologically under the references diff --git a/src/libstore/misc.cc b/src/libstore/misc.cc index dabae647fbb..7c97dbc5717 100644 --- a/src/libstore/misc.cc +++ b/src/libstore/misc.cc @@ -98,23 +98,17 @@ const ContentAddress * getDerivationCA(const BasicDerivation & drv) return nullptr; } -void Store::queryMissing(const std::vector & targets, - StorePathSet & willBuild_, StorePathSet & willSubstitute_, StorePathSet & unknown_, - uint64_t & downloadSize_, uint64_t & narSize_) +MissingPaths Store::queryMissing(const std::vector & targets) { Activity act(*logger, lvlDebug, actUnknown, "querying info about missing paths"); - downloadSize_ = narSize_ = 0; - // FIXME: make async. ThreadPool pool(fileTransferSettings.httpConnections); struct State { std::unordered_set done; - StorePathSet & unknown, & willSubstitute, & willBuild; - uint64_t & downloadSize; - uint64_t & narSize; + MissingPaths res; }; struct DrvState @@ -125,7 +119,7 @@ void Store::queryMissing(const std::vector & targets, DrvState(size_t left) : left(left) { } }; - Sync state_(State{{}, unknown_, willSubstitute_, willBuild_, downloadSize_, narSize_}); + Sync state_; std::function doPath; @@ -143,7 +137,7 @@ void Store::queryMissing(const std::vector & targets, auto mustBuildDrv = [&](const StorePath & drvPath, const Derivation & drv) { { auto state(state_.lock()); - state->willBuild.insert(drvPath); + state->res.willBuild.insert(drvPath); } for (const auto & [inputDrv, inputNode] : drv.inputDrvs.map) { @@ -203,7 +197,7 @@ void Store::queryMissing(const std::vector & targets, if (!isValidPath(drvPath)) { // FIXME: we could try to substitute the derivation. auto state(state_.lock()); - state->unknown.insert(drvPath); + state->res.unknown.insert(drvPath); return; } @@ -282,7 +276,7 @@ void Store::queryMissing(const std::vector & targets, if (infos.empty()) { auto state(state_.lock()); - state->unknown.insert(bo.path); + state->res.unknown.insert(bo.path); return; } @@ -291,9 +285,9 @@ void Store::queryMissing(const std::vector & targets, { auto state(state_.lock()); - state->willSubstitute.insert(bo.path); - state->downloadSize += info->second.downloadSize; - state->narSize += info->second.narSize; + state->res.willSubstitute.insert(bo.path); + state->res.downloadSize += info->second.downloadSize; + state->res.narSize += info->second.narSize; } for (auto & ref : info->second.references) @@ -306,6 +300,8 @@ void Store::queryMissing(const std::vector & targets, pool.enqueue(std::bind(doPath, path)); pool.process(); + + return std::move(state_.lock()->res); } diff --git a/src/libstore/remote-store.cc b/src/libstore/remote-store.cc index 3151f319c00..1b8bad04807 100644 --- a/src/libstore/remote-store.cc +++ b/src/libstore/remote-store.cc @@ -855,9 +855,7 @@ void RemoteStore::addSignatures(const StorePath & storePath, const StringSet & s } -void RemoteStore::queryMissing(const std::vector & targets, - StorePathSet & willBuild, StorePathSet & willSubstitute, StorePathSet & unknown, - uint64_t & downloadSize, uint64_t & narSize) +MissingPaths RemoteStore::queryMissing(const std::vector & targets) { { auto conn(getConnection()); @@ -868,16 +866,16 @@ void RemoteStore::queryMissing(const std::vector & targets, conn->to << WorkerProto::Op::QueryMissing; WorkerProto::write(*this, *conn, targets); conn.processStderr(); - willBuild = WorkerProto::Serialise::read(*this, *conn); - willSubstitute = WorkerProto::Serialise::read(*this, *conn); - unknown = WorkerProto::Serialise::read(*this, *conn); - conn->from >> downloadSize >> narSize; - return; + MissingPaths res; + res.willBuild = WorkerProto::Serialise::read(*this, *conn); + res.willSubstitute = WorkerProto::Serialise::read(*this, *conn); + res.unknown = WorkerProto::Serialise::read(*this, *conn); + conn->from >> res.downloadSize >> res.narSize; + return res; } fallback: - return Store::queryMissing(targets, willBuild, willSubstitute, - unknown, downloadSize, narSize); + return Store::queryMissing(targets); } diff --git a/src/libstore/restricted-store.cc b/src/libstore/restricted-store.cc index 0485f558473..69435122a24 100644 --- a/src/libstore/restricted-store.cc +++ b/src/libstore/restricted-store.cc @@ -143,13 +143,7 @@ struct RestrictedStore : public virtual IndirectRootStore, public virtual GcStor unsupported("addSignatures"); } - void queryMissing( - const std::vector & targets, - StorePathSet & willBuild, - StorePathSet & willSubstitute, - StorePathSet & unknown, - uint64_t & downloadSize, - uint64_t & narSize) override; + MissingPaths queryMissing(const std::vector & targets) override; virtual std::optional getBuildLogExact(const StorePath & path) override { @@ -306,19 +300,14 @@ std::vector RestrictedStore::buildPathsWithResults( return results; } -void RestrictedStore::queryMissing( - const std::vector & targets, - StorePathSet & willBuild, - StorePathSet & willSubstitute, - StorePathSet & unknown, - uint64_t & downloadSize, - uint64_t & narSize) +MissingPaths RestrictedStore::queryMissing(const std::vector & targets) { /* This is slightly impure since it leaks information to the client about what paths will be built/substituted or are already present. Probably not a big deal. */ std::vector allowed; + StorePathSet unknown; for (auto & req : targets) { if (goal.isAllowed(req)) allowed.emplace_back(req); @@ -326,7 +315,12 @@ void RestrictedStore::queryMissing( unknown.insert(pathPartOfReq(req)); } - next->queryMissing(allowed, willBuild, willSubstitute, unknown, downloadSize, narSize); + auto res = next->queryMissing(allowed); + + for (auto & p : unknown) + res.unknown.insert(p); + + return res; } } diff --git a/src/libstore/store-api.cc b/src/libstore/store-api.cc index 296f2251adc..9aeab1d1f6b 100644 --- a/src/libstore/store-api.cc +++ b/src/libstore/store-api.cc @@ -790,15 +790,12 @@ void Store::substitutePaths(const StorePathSet & paths) for (auto & path : paths) if (!path.isDerivation()) paths2.emplace_back(DerivedPath::Opaque{path}); - uint64_t downloadSize, narSize; - StorePathSet willBuild, willSubstitute, unknown; - queryMissing(paths2, - willBuild, willSubstitute, unknown, downloadSize, narSize); + auto missing = queryMissing(paths2); - if (!willSubstitute.empty()) + if (!missing.willSubstitute.empty()) try { std::vector subs; - for (auto & p : willSubstitute) subs.emplace_back(DerivedPath::Opaque{p}); + for (auto & p : missing.willSubstitute) subs.emplace_back(DerivedPath::Opaque{p}); buildPaths(subs); } catch (Error & e) { logWarning(e.info()); diff --git a/src/nix-build/nix-build.cc b/src/nix-build/nix-build.cc index cde9d67424a..36e6f3eab4e 100644 --- a/src/nix-build/nix-build.cc +++ b/src/nix-build/nix-build.cc @@ -422,13 +422,10 @@ static void main_nix_build(int argc, char * * argv) auto buildPaths = [&](const std::vector & paths) { /* Note: we do this even when !printMissing to efficiently fetch binary cache data. */ - uint64_t downloadSize, narSize; - StorePathSet willBuild, willSubstitute, unknown; - store->queryMissing(paths, - willBuild, willSubstitute, unknown, downloadSize, narSize); + auto missing = store->queryMissing(paths); if (settings.printMissing) - printMissing(ref(store), willBuild, willSubstitute, unknown, downloadSize, narSize); + printMissing(ref(store), missing.willBuild, missing.willSubstitute, missing.unknown, missing.downloadSize, missing.narSize); if (!dryRun) store->buildPaths(paths, buildMode, evalStore); diff --git a/src/nix-store/nix-store.cc b/src/nix-store/nix-store.cc index 0d758f44b1c..c508029b56f 100644 --- a/src/nix-store/nix-store.cc +++ b/src/nix-store/nix-store.cc @@ -146,23 +146,19 @@ static void opRealise(Strings opFlags, Strings opArgs) for (auto & i : opArgs) paths.push_back(followLinksToStorePathWithOutputs(*store, i)); - uint64_t downloadSize, narSize; - StorePathSet willBuild, willSubstitute, unknown; - store->queryMissing( - toDerivedPaths(paths), - willBuild, willSubstitute, unknown, downloadSize, narSize); + auto missing = store->queryMissing(toDerivedPaths(paths)); /* Filter out unknown paths from `paths`. */ if (ignoreUnknown) { std::vector paths2; for (auto & i : paths) - if (!unknown.count(i.path)) paths2.push_back(i); + if (!missing.unknown.count(i.path)) paths2.push_back(i); paths = std::move(paths2); - unknown = StorePathSet(); + missing.unknown = StorePathSet(); } if (settings.printMissing) - printMissing(ref(store), willBuild, willSubstitute, unknown, downloadSize, narSize); + printMissing(ref(store), missing.willBuild, missing.willSubstitute, missing.unknown, missing.downloadSize, missing.narSize); if (dryRun) return; From 5d308ccca558230d374ea0579c80cf68d950feda Mon Sep 17 00:00:00 2001 From: Eelco Dolstra Date: Fri, 4 Jul 2025 16:27:08 +0200 Subject: [PATCH 207/218] printMissing(): Take a MissingPaths argument --- src/libmain/include/nix/main/shared.hh | 8 +++--- src/libmain/shared.cc | 34 +++++++++++++------------- src/nix-build/nix-build.cc | 2 +- src/nix-store/nix-store.cc | 2 +- 4 files changed, 24 insertions(+), 22 deletions(-) diff --git a/src/libmain/include/nix/main/shared.hh b/src/libmain/include/nix/main/shared.hh index 2ff57135b1b..4d4b816e714 100644 --- a/src/libmain/include/nix/main/shared.hh +++ b/src/libmain/include/nix/main/shared.hh @@ -35,15 +35,17 @@ void printVersion(const std::string & programName); void printGCWarning(); class Store; +struct MissingPaths; void printMissing( ref store, const std::vector & paths, Verbosity lvl = lvlInfo); -void printMissing(ref store, const StorePathSet & willBuild, - const StorePathSet & willSubstitute, const StorePathSet & unknown, - uint64_t downloadSize, uint64_t narSize, Verbosity lvl = lvlInfo); +void printMissing( + ref store, + const MissingPaths & missing, + Verbosity lvl = lvlInfo); std::string getArg(const std::string & opt, Strings::iterator & i, const Strings::iterator & end); diff --git a/src/libmain/shared.cc b/src/libmain/shared.cc index 75c9ebadd81..0982810d1a7 100644 --- a/src/libmain/shared.cc +++ b/src/libmain/shared.cc @@ -46,41 +46,41 @@ void printGCWarning() void printMissing(ref store, const std::vector & paths, Verbosity lvl) { - auto missing = store->queryMissing(paths); - printMissing(store, missing.willBuild, missing.willSubstitute, missing.unknown, missing.downloadSize, missing.narSize, lvl); + printMissing(store, store->queryMissing(paths), lvl); } -void printMissing(ref store, const StorePathSet & willBuild, - const StorePathSet & willSubstitute, const StorePathSet & unknown, - uint64_t downloadSize, uint64_t narSize, Verbosity lvl) +void printMissing( + ref store, + const MissingPaths & missing, + Verbosity lvl) { - if (!willBuild.empty()) { - if (willBuild.size() == 1) + if (!missing.willBuild.empty()) { + if (missing.willBuild.size() == 1) printMsg(lvl, "this derivation will be built:"); else - printMsg(lvl, "these %d derivations will be built:", willBuild.size()); - auto sorted = store->topoSortPaths(willBuild); + printMsg(lvl, "these %d derivations will be built:", missing.willBuild.size()); + auto sorted = store->topoSortPaths(missing.willBuild); reverse(sorted.begin(), sorted.end()); for (auto & i : sorted) printMsg(lvl, " %s", store->printStorePath(i)); } - if (!willSubstitute.empty()) { - const float downloadSizeMiB = downloadSize / (1024.f * 1024.f); - const float narSizeMiB = narSize / (1024.f * 1024.f); - if (willSubstitute.size() == 1) { + if (!missing.willSubstitute.empty()) { + const float downloadSizeMiB = missing.downloadSize / (1024.f * 1024.f); + const float narSizeMiB = missing.narSize / (1024.f * 1024.f); + if (missing.willSubstitute.size() == 1) { printMsg(lvl, "this path will be fetched (%.2f MiB download, %.2f MiB unpacked):", downloadSizeMiB, narSizeMiB); } else { printMsg(lvl, "these %d paths will be fetched (%.2f MiB download, %.2f MiB unpacked):", - willSubstitute.size(), + missing.willSubstitute.size(), downloadSizeMiB, narSizeMiB); } std::vector willSubstituteSorted = {}; - std::for_each(willSubstitute.begin(), willSubstitute.end(), + std::for_each(missing.willSubstitute.begin(), missing.willSubstitute.end(), [&](const StorePath &p) { willSubstituteSorted.push_back(&p); }); std::sort(willSubstituteSorted.begin(), willSubstituteSorted.end(), [](const StorePath *lhs, const StorePath *rhs) { @@ -93,10 +93,10 @@ void printMissing(ref store, const StorePathSet & willBuild, printMsg(lvl, " %s", store->printStorePath(*p)); } - if (!unknown.empty()) { + if (!missing.unknown.empty()) { printMsg(lvl, "don't know how to build these paths%s:", (settings.readOnlyMode ? " (may be caused by read-only store access)" : "")); - for (auto & i : unknown) + for (auto & i : missing.unknown) printMsg(lvl, " %s", store->printStorePath(i)); } } diff --git a/src/nix-build/nix-build.cc b/src/nix-build/nix-build.cc index 36e6f3eab4e..e2f4c03302d 100644 --- a/src/nix-build/nix-build.cc +++ b/src/nix-build/nix-build.cc @@ -425,7 +425,7 @@ static void main_nix_build(int argc, char * * argv) auto missing = store->queryMissing(paths); if (settings.printMissing) - printMissing(ref(store), missing.willBuild, missing.willSubstitute, missing.unknown, missing.downloadSize, missing.narSize); + printMissing(ref(store), missing); if (!dryRun) store->buildPaths(paths, buildMode, evalStore); diff --git a/src/nix-store/nix-store.cc b/src/nix-store/nix-store.cc index c508029b56f..3da7a8ac108 100644 --- a/src/nix-store/nix-store.cc +++ b/src/nix-store/nix-store.cc @@ -158,7 +158,7 @@ static void opRealise(Strings opFlags, Strings opArgs) } if (settings.printMissing) - printMissing(ref(store), missing.willBuild, missing.willSubstitute, missing.unknown, missing.downloadSize, missing.narSize); + printMissing(ref(store), missing); if (dryRun) return; From f039f6886a9880b006f78df2ab273a0d66f20db9 Mon Sep 17 00:00:00 2001 From: Eelco Dolstra Date: Fri, 4 Jul 2025 16:32:37 +0200 Subject: [PATCH 208/218] nix-build: Drop unnecessary call to queryMissing() This is already done by Worker::run(). --- src/nix-build/nix-build.cc | 6 +----- 1 file changed, 1 insertion(+), 5 deletions(-) diff --git a/src/nix-build/nix-build.cc b/src/nix-build/nix-build.cc index e2f4c03302d..7e0b4025254 100644 --- a/src/nix-build/nix-build.cc +++ b/src/nix-build/nix-build.cc @@ -420,12 +420,8 @@ static void main_nix_build(int argc, char * * argv) state->maybePrintStats(); auto buildPaths = [&](const std::vector & paths) { - /* Note: we do this even when !printMissing to efficiently - fetch binary cache data. */ - auto missing = store->queryMissing(paths); - if (settings.printMissing) - printMissing(ref(store), missing); + printMissing(ref(store), paths); if (!dryRun) store->buildPaths(paths, buildMode, evalStore); From 8a9e625ba5483f08aaef7c4b7074042bcc7f0f57 Mon Sep 17 00:00:00 2001 From: Eelco Dolstra Date: Mon, 7 Jul 2025 14:06:24 +0200 Subject: [PATCH 209/218] release notes: 2.30.0 --- doc/manual/rl-next/build-dir-mandatory.md | 9 -- doc/manual/rl-next/deprecate__json.md | 11 -- doc/manual/rl-next/eval-profiler.md | 13 --- doc/manual/rl-next/json-logger.md | 6 - doc/manual/rl-next/nix-profile-add.md | 6 - .../rl-next/outpath-and-sourceinfo-fixes.md | 17 --- doc/manual/rl-next/revert-77.md | 17 --- doc/manual/source/SUMMARY.md.in | 1 + doc/manual/source/release-notes/rl-2.30.md | 106 ++++++++++++++++++ .../data/release-credits-email-to-handle.json | 21 +++- .../data/release-credits-handle-to-name.json | 18 ++- 11 files changed, 144 insertions(+), 81 deletions(-) delete mode 100644 doc/manual/rl-next/build-dir-mandatory.md delete mode 100644 doc/manual/rl-next/deprecate__json.md delete mode 100644 doc/manual/rl-next/eval-profiler.md delete mode 100644 doc/manual/rl-next/json-logger.md delete mode 100644 doc/manual/rl-next/nix-profile-add.md delete mode 100644 doc/manual/rl-next/outpath-and-sourceinfo-fixes.md delete mode 100644 doc/manual/rl-next/revert-77.md create mode 100644 doc/manual/source/release-notes/rl-2.30.md diff --git a/doc/manual/rl-next/build-dir-mandatory.md b/doc/manual/rl-next/build-dir-mandatory.md deleted file mode 100644 index cb45a4315f1..00000000000 --- a/doc/manual/rl-next/build-dir-mandatory.md +++ /dev/null @@ -1,9 +0,0 @@ ---- -synopsis: "`build-dir` no longer defaults to `$TMPDIR`" ---- - -The directory in which temporary build directories are created no longer defaults -to `TMPDIR` or `/tmp`, to avoid builders making their directories -world-accessible. This behavior allowed escaping the build sandbox and can -cause build impurities even when not used maliciously. We now default to `builds` -in `NIX_STATE_DIR` (which is `/nix/var/nix/builds` in the default configuration). diff --git a/doc/manual/rl-next/deprecate__json.md b/doc/manual/rl-next/deprecate__json.md deleted file mode 100644 index 7fc05832f3b..00000000000 --- a/doc/manual/rl-next/deprecate__json.md +++ /dev/null @@ -1,11 +0,0 @@ ---- -synopsis: Deprecate manually making structured attrs with `__json = ...;` -prs: [13220] ---- - -The proper way to create a derivation using [structured attrs] in the Nix language is by using `__structuredAttrs = true` with [`builtins.derivation`]. -However, by exploiting how structured attrs are implementated, it has also been possible to create them by setting the `__json` environment variable to a serialized JSON string. -This sneaky alternative method is now deprecated, and may be disallowed in future versions of Nix. - -[structured attrs]: @docroot@/language/advanced-attributes.md#adv-attr-structuredAttrs -[`builtins.derivation`]: @docroot@/language/builtins.html#builtins-derivation diff --git a/doc/manual/rl-next/eval-profiler.md b/doc/manual/rl-next/eval-profiler.md deleted file mode 100644 index 6b2bf2ce729..00000000000 --- a/doc/manual/rl-next/eval-profiler.md +++ /dev/null @@ -1,13 +0,0 @@ ---- -synopsis: Add stack sampling evaluation profiler -prs: [13220] ---- - -Nix evaluator now supports stack sampling evaluation profiling via `--eval-profiler flamegraph` setting. -It collects collapsed call stack information to output file specified by -`--eval-profile-file` (`nix.profile` by default) in a format directly consumable -by `flamegraph.pl` and compatible tools like [speedscope](https://speedscope.app/). -Sampling frequency can be configured via `--eval-profiler-frequency` (99 Hz by default). - -Unlike existing `--trace-function-calls` this profiler includes the name of the function -being called when it's available. diff --git a/doc/manual/rl-next/json-logger.md b/doc/manual/rl-next/json-logger.md deleted file mode 100644 index 867b5d8b3fa..00000000000 --- a/doc/manual/rl-next/json-logger.md +++ /dev/null @@ -1,6 +0,0 @@ ---- -synopsis: "`json-log-path` setting" -prs: [13003] ---- - -New setting `json-log-path` that sends a copy of all Nix log messages (in JSON format) to a file or Unix domain socket. diff --git a/doc/manual/rl-next/nix-profile-add.md b/doc/manual/rl-next/nix-profile-add.md deleted file mode 100644 index a2fbcaa8da5..00000000000 --- a/doc/manual/rl-next/nix-profile-add.md +++ /dev/null @@ -1,6 +0,0 @@ ---- -synopsis: "Rename `nix profile install` to `nix profile add`" -prs: [13224] ---- - -The command `nix profile install` has been renamed to `nix profile add` (though the former is still available as an alias). This is because the verb "add" is a better antonym for the verb "remove" (i.e. `nix profile remove`). Nix also does not have install hooks or general behavior often associated with "installing". diff --git a/doc/manual/rl-next/outpath-and-sourceinfo-fixes.md b/doc/manual/rl-next/outpath-and-sourceinfo-fixes.md deleted file mode 100644 index 479a2f5cb75..00000000000 --- a/doc/manual/rl-next/outpath-and-sourceinfo-fixes.md +++ /dev/null @@ -1,17 +0,0 @@ ---- -synopsis: Non-flake inputs now contain a `sourceInfo` attribute -issues: 13164 -prs: 13170 ---- - -Flakes have always a `sourceInfo` attribute which describes the source of the flake. -The `sourceInfo.outPath` is often identical to the flake's `outPath`, however it can differ when the flake is located in a subdirectory of its source. - -Non-flake inputs (i.e. inputs with `flake = false`) can also be located at some path _within_ a wider source. -This usually happens when defining a relative path input within the same source as the parent flake, e.g. `inputs.foo.url = ./some-file.nix`. -Such relative inputs will now inherit their parent's `sourceInfo`. - -This also means it is now possible to use `?dir=subdir` on non-flake inputs. - -This iterates on the work done in 2.26 to improve relative path support ([#10089](https://github.com/NixOS/nix/pull/10089)), -and resolves a regression introduced in 2.28 relating to nested relative path inputs ([#13164](https://github.com/NixOS/nix/issues/13164)). diff --git a/doc/manual/rl-next/revert-77.md b/doc/manual/rl-next/revert-77.md deleted file mode 100644 index c2cef74d3e4..00000000000 --- a/doc/manual/rl-next/revert-77.md +++ /dev/null @@ -1,17 +0,0 @@ ---- -synopsis: Revert incomplete closure mixed download and build feature -issues: [77, 12628] -prs: [13176] ---- - -Since Nix 1.3 (299141ecbd08bae17013226dbeae71e842b4fdd7 in 2013) Nix has attempted to mix together upstream fresh builds and downstream substitutions when remote substuters contain an "incomplete closure" (have some store objects, but not the store objects they reference). -This feature is now removed. - -Worst case, removing this feature could cause more building downstream, but it should not cause outright failures, since this is not happening for opaque store objects that we don't know how to build if we decide not to substitute. -In practice, however, we doubt even the more building is very likely to happen. -Remote stores that are missing dependencies in arbitrary ways (e.g. corruption) don't seem to be very common. - -On the contrary, when remote stores fail to implement the [closure property](@docroot@/store/store-object.md#closure-property), it is usually an *intentional* choice on the part of the remote store, because it wishes to serve as an "overlay" store over another store, such as `https://cache.nixos.org`. -If an "incomplete closure" is encountered in that situation, the right fix is not to do some sort of "franken-building" as this feature implemented, but instead to make sure both substituters are enabled in the settings. - -(In the future, we should make it easier for remote stores to indicate this to clients, to catch settings that won't work in general before a missing dependency is actually encountered.) diff --git a/doc/manual/source/SUMMARY.md.in b/doc/manual/source/SUMMARY.md.in index 8326a96e35c..bfb921567c3 100644 --- a/doc/manual/source/SUMMARY.md.in +++ b/doc/manual/source/SUMMARY.md.in @@ -137,6 +137,7 @@ - [Contributing](development/contributing.md) - [Releases](release-notes/index.md) {{#include ./SUMMARY-rl-next.md}} + - [Release 2.30 (2025-07-07)](release-notes/rl-2.30.md) - [Release 2.29 (2025-05-14)](release-notes/rl-2.29.md) - [Release 2.28 (2025-04-02)](release-notes/rl-2.28.md) - [Release 2.27 (2025-03-03)](release-notes/rl-2.27.md) diff --git a/doc/manual/source/release-notes/rl-2.30.md b/doc/manual/source/release-notes/rl-2.30.md new file mode 100644 index 00000000000..b205bad5150 --- /dev/null +++ b/doc/manual/source/release-notes/rl-2.30.md @@ -0,0 +1,106 @@ +# Release 2.30.0 (2025-07-07) + +- `build-dir` no longer defaults to `$TMPDIR` + + The directory in which temporary build directories are created no longer defaults + to `TMPDIR` or `/tmp`, to avoid builders making their directories + world-accessible. This behavior allowed escaping the build sandbox and can + cause build impurities even when not used maliciously. We now default to `builds` + in `NIX_STATE_DIR` (which is `/nix/var/nix/builds` in the default configuration). + +- Deprecate manually making structured attrs with `__json = ...;` [#13220](https://github.com/NixOS/nix/pull/13220) + + The proper way to create a derivation using [structured attrs] in the Nix language is by using `__structuredAttrs = true` with [`builtins.derivation`]. + However, by exploiting how structured attrs are implementated, it has also been possible to create them by setting the `__json` environment variable to a serialized JSON string. + This sneaky alternative method is now deprecated, and may be disallowed in future versions of Nix. + + [structured attrs]: @docroot@/language/advanced-attributes.md#adv-attr-structuredAttrs + [`builtins.derivation`]: @docroot@/language/builtins.html#builtins-derivation + +- Add stack sampling evaluation profiler [#13220](https://github.com/NixOS/nix/pull/13220) + + Nix evaluator now supports stack sampling evaluation profiling via `--eval-profiler flamegraph` setting. + It collects collapsed call stack information to output file specified by + `--eval-profile-file` (`nix.profile` by default) in a format directly consumable + by `flamegraph.pl` and compatible tools like [speedscope](https://speedscope.app/). + Sampling frequency can be configured via `--eval-profiler-frequency` (99 Hz by default). + + Unlike existing `--trace-function-calls` this profiler includes the name of the function + being called when it's available. + +- `json-log-path` setting [#13003](https://github.com/NixOS/nix/pull/13003) + + New setting `json-log-path` that sends a copy of all Nix log messages (in JSON format) to a file or Unix domain socket. + +- Rename `nix profile install` to `nix profile add` [#13224](https://github.com/NixOS/nix/pull/13224) + + The command `nix profile install` has been renamed to `nix profile add` (though the former is still available as an alias). This is because the verb "add" is a better antonym for the verb "remove" (i.e. `nix profile remove`). Nix also does not have install hooks or general behavior often associated with "installing". + +- Non-flake inputs now contain a `sourceInfo` attribute [#13164](https://github.com/NixOS/nix/issues/13164) [#13170](https://github.com/NixOS/nix/pull/13170) + + Flakes have always a `sourceInfo` attribute which describes the source of the flake. + The `sourceInfo.outPath` is often identical to the flake's `outPath`, however it can differ when the flake is located in a subdirectory of its source. + + Non-flake inputs (i.e. inputs with `flake = false`) can also be located at some path _within_ a wider source. + This usually happens when defining a relative path input within the same source as the parent flake, e.g. `inputs.foo.url = ./some-file.nix`. + Such relative inputs will now inherit their parent's `sourceInfo`. + + This also means it is now possible to use `?dir=subdir` on non-flake inputs. + + This iterates on the work done in 2.26 to improve relative path support ([#10089](https://github.com/NixOS/nix/pull/10089)), + and resolves a regression introduced in 2.28 relating to nested relative path inputs ([#13164](https://github.com/NixOS/nix/issues/13164)). + +- Revert incomplete closure mixed download and build feature [#77](https://github.com/NixOS/nix/issues/77) [#12628](https://github.com/NixOS/nix/issues/12628) [#13176](https://github.com/NixOS/nix/pull/13176) + + Since Nix 1.3 (299141ecbd08bae17013226dbeae71e842b4fdd7 in 2013) Nix has attempted to mix together upstream fresh builds and downstream substitutions when remote substuters contain an "incomplete closure" (have some store objects, but not the store objects they reference). + This feature is now removed. + + Worst case, removing this feature could cause more building downstream, but it should not cause outright failures, since this is not happening for opaque store objects that we don't know how to build if we decide not to substitute. + In practice, however, we doubt even the more building is very likely to happen. + Remote stores that are missing dependencies in arbitrary ways (e.g. corruption) don't seem to be very common. + + On the contrary, when remote stores fail to implement the [closure property](@docroot@/store/store-object.md#closure-property), it is usually an *intentional* choice on the part of the remote store, because it wishes to serve as an "overlay" store over another store, such as `https://cache.nixos.org`. + If an "incomplete closure" is encountered in that situation, the right fix is not to do some sort of "franken-building" as this feature implemented, but instead to make sure both substituters are enabled in the settings. + + (In the future, we should make it easier for remote stores to indicate this to clients, to catch settings that won't work in general before a missing dependency is actually encountered.) + + +# Contributors + + +This release was made possible by the following 32 contributors: + +- Robert Hensing [**(@roberth)**](https://github.com/roberth) +- Jörg Thalheim [**(@Mic92)**](https://github.com/Mic92) +- Egor Konovalov [**(@egorkonovalov)**](https://github.com/egorkonovalov) +- PopeRigby [**(@poperigby)**](https://github.com/poperigby) +- Peder Bergebakken Sundt [**(@pbsds)**](https://github.com/pbsds) +- Farid Zakaria [**(@fzakaria)**](https://github.com/fzakaria) +- Gwenn Le Bihan [**(@gwennlbh)**](https://github.com/gwennlbh) +- Jade Masker [**(@donottellmetonottellyou)**](https://github.com/donottellmetonottellyou) +- Nikita Krasnov [**(@synalice)**](https://github.com/synalice) +- tomberek [**(@tomberek)**](https://github.com/tomberek) +- Wolfgang Walther [**(@wolfgangwalther)**](https://github.com/wolfgangwalther) +- Samuli Thomasson [**(@SimSaladin)**](https://github.com/SimSaladin) +- h0nIg [**(@h0nIg)**](https://github.com/h0nIg) +- Valentin Gagarin [**(@fricklerhandwerk)**](https://github.com/fricklerhandwerk) +- Vladimír Čunát [**(@vcunat)**](https://github.com/vcunat) +- Graham Christensen [**(@grahamc)**](https://github.com/grahamc) +- kstrafe [**(@kstrafe)**](https://github.com/kstrafe) +- gustavderdrache [**(@gustavderdrache)**](https://github.com/gustavderdrache) +- Matt Sturgeon [**(@MattSturgeon)**](https://github.com/MattSturgeon) +- John Ericson [**(@Ericson2314)**](https://github.com/Ericson2314) +- Tristan Ross [**(@RossComputerGuy)**](https://github.com/RossComputerGuy) +- jayeshv [**(@jayeshv)**](https://github.com/jayeshv) +- Eelco Dolstra [**(@edolstra)**](https://github.com/edolstra) +- pennae [**(@pennae)**](https://github.com/pennae) +- Luc Perkins [**(@lucperkins)**](https://github.com/lucperkins) +- Cole Helbling [**(@cole-h)**](https://github.com/cole-h) +- Pol Dellaiera [**(@drupol)**](https://github.com/drupol) +- Sergei Zimmerman [**(@xokdvium)**](https://github.com/xokdvium) +- Seth Flynn [**(@getchoo)**](https://github.com/getchoo) +- Jonas Chevalier [**(@zimbatm)**](https://github.com/zimbatm) +- Stefan Boca [**(@stefanboca)**](https://github.com/stefanboca) +- Jeremy Fleischman [**(@jfly)**](https://github.com/jfly) +- Philipp Otterbein +- Raito Bezarius diff --git a/maintainers/data/release-credits-email-to-handle.json b/maintainers/data/release-credits-email-to-handle.json index c77fe468414..48e8685e6d9 100644 --- a/maintainers/data/release-credits-email-to-handle.json +++ b/maintainers/data/release-credits-email-to-handle.json @@ -166,5 +166,24 @@ "the-tumultuous-unicorn-of-darkness@gmx.com": "TheTumultuousUnicornOfDarkness", "dev@rodney.id.au": "rvl", "pe@pijul.org": "P-E-Meunier", - "yannik@floxdev.com": "ysndr" + "yannik@floxdev.com": "ysndr", + "73017521+egorkonovalov@users.noreply.github.com": "egorkonovalov", + "raito@lix.systems": null, + "nikita.nikita.krasnov@gmail.com": "synalice", + "lucperkins@gmail.com": "lucperkins", + "vladimir.cunat@nic.cz": "vcunat", + "walther@technowledgy.de": "wolfgangwalther", + "jayesh.mail@gmail.com": "jayeshv", + "samuli.thomasson@pm.me": "SimSaladin", + "kevin@stravers.net": "kstrafe", + "poperigby@mailbox.org": "poperigby", + "cole.helbling@determinate.systems": "cole-h", + "donottellmetonottellyou@gmail.com": "donottellmetonottellyou", + "getchoo@tuta.io": "getchoo", + "alex.ford@determinate.systems": "gustavderdrache", + "stefan.r.boca@gmail.com": "stefanboca", + "gwenn.lebihan7@gmail.com": "gwennlbh", + "hey@ewen.works": "gwennlbh", + "matt@sturgeon.me.uk": "MattSturgeon", + "pbsds@hotmail.com": "pbsds" } \ No newline at end of file diff --git a/maintainers/data/release-credits-handle-to-name.json b/maintainers/data/release-credits-handle-to-name.json index 734476078c3..a6352c44b22 100644 --- a/maintainers/data/release-credits-handle-to-name.json +++ b/maintainers/data/release-credits-handle-to-name.json @@ -146,5 +146,21 @@ "ajlekcahdp4": "Alexander Romanov", "Valodim": "Vincent Breitmoser", "rvl": "Rodney Lorrimar", - "whatsthecraic": "Dean De Leo" + "whatsthecraic": "Dean De Leo", + "gwennlbh": "Gwenn Le Bihan", + "donottellmetonottellyou": "Jade Masker", + "kstrafe": null, + "synalice": "Nikita Krasnov", + "poperigby": "PopeRigby", + "MattSturgeon": "Matt Sturgeon", + "lucperkins": "Luc Perkins", + "gustavderdrache": null, + "SimSaladin": "Samuli Thomasson", + "getchoo": "Seth Flynn", + "stefanboca": "Stefan Boca", + "wolfgangwalther": "Wolfgang Walther", + "pbsds": "Peder Bergebakken Sundt", + "egorkonovalov": "Egor Konovalov", + "jayeshv": "jayeshv", + "vcunat": "Vladim\u00edr \u010cun\u00e1t" } \ No newline at end of file From 19c4e78d9726fa64bcb4060ea968eb885f0e95c7 Mon Sep 17 00:00:00 2001 From: Eelco Dolstra Date: Mon, 7 Jul 2025 14:19:57 +0200 Subject: [PATCH 210/218] Typo --- doc/manual/source/release-notes/rl-2.30.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/doc/manual/source/release-notes/rl-2.30.md b/doc/manual/source/release-notes/rl-2.30.md index b205bad5150..3111eddc6aa 100644 --- a/doc/manual/source/release-notes/rl-2.30.md +++ b/doc/manual/source/release-notes/rl-2.30.md @@ -38,8 +38,8 @@ - Non-flake inputs now contain a `sourceInfo` attribute [#13164](https://github.com/NixOS/nix/issues/13164) [#13170](https://github.com/NixOS/nix/pull/13170) - Flakes have always a `sourceInfo` attribute which describes the source of the flake. - The `sourceInfo.outPath` is often identical to the flake's `outPath`, however it can differ when the flake is located in a subdirectory of its source. + Flakes have always had a `sourceInfo` attribute which describes the source of the flake. + The `sourceInfo.outPath` is often identical to the flake's `outPath`. However, it can differ when the flake is located in a subdirectory of its source. Non-flake inputs (i.e. inputs with `flake = false`) can also be located at some path _within_ a wider source. This usually happens when defining a relative path input within the same source as the parent flake, e.g. `inputs.foo.url = ./some-file.nix`. From f5312492c1280f01914ee2a5bd025eed7b5305d9 Mon Sep 17 00:00:00 2001 From: Eelco Dolstra Date: Mon, 7 Jul 2025 14:53:11 +0200 Subject: [PATCH 211/218] Add manual release notes --- doc/manual/source/release-notes/rl-2.30.md | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) diff --git a/doc/manual/source/release-notes/rl-2.30.md b/doc/manual/source/release-notes/rl-2.30.md index 3111eddc6aa..05e1d7fcb1e 100644 --- a/doc/manual/source/release-notes/rl-2.30.md +++ b/doc/manual/source/release-notes/rl-2.30.md @@ -1,5 +1,25 @@ # Release 2.30.0 (2025-07-07) +- Reduce the size of value from 24 to 16 bytes [#13407](https://github.com/NixOS/nix/pull/13407) + + This shaves off a very significant amount of memory used for evaluation (~20% percent reduction in maximum heap size and ~17% in total bytes). + +- `builtins.sort` uses PeekSort [#12623](https://github.com/NixOS/nix/pull/12623) + + Previously it used libstdc++'s `std::stable_sort()`. However, that implementation is not reliable if the user-supplied comparison function is not a strict weak ordering. + +- `nix repl` prints which variables were loaded [#11406](https://github.com/NixOS/nix/pull/11406) + + Instead of `Added variables` it now prints the first 10 variables that were added to the global scope. + +- `nix flake archive`: add `--no-check-sigs` option [#13277](https://github.com/NixOS/nix/pull/13277) + + This is useful when using `nix flake archive` with the destination set to a remote store. + +- Emit warnings for IFDs with `trace-import-from-derivation` option [#13279](https://github.com/NixOS/nix/pull/13279) + + While we have the setting `allow-import-from-derivation` to deny import-from-derivation (IFD), sometimes users would like to observe IFDs during CI processes to gradually phase out the idiom. The new setting `trace-import-from-derivation`, when set, logs a simple warning to the console. + - `build-dir` no longer defaults to `$TMPDIR` The directory in which temporary build directories are created no longer defaults From 2c0343ec51ef83c2cafda021e68046b35cb67c12 Mon Sep 17 00:00:00 2001 From: Eelco Dolstra Date: Mon, 7 Jul 2025 15:00:49 +0200 Subject: [PATCH 212/218] # Contributors -> ## Contributors --- doc/manual/source/release-notes/rl-2.24.md | 2 +- doc/manual/source/release-notes/rl-2.25.md | 2 +- doc/manual/source/release-notes/rl-2.26.md | 2 +- doc/manual/source/release-notes/rl-2.27.md | 2 +- doc/manual/source/release-notes/rl-2.28.md | 2 +- doc/manual/source/release-notes/rl-2.29.md | 2 +- maintainers/release-notes | 2 +- 7 files changed, 7 insertions(+), 7 deletions(-) diff --git a/doc/manual/source/release-notes/rl-2.24.md b/doc/manual/source/release-notes/rl-2.24.md index bac0a59984d..d4af3cb5174 100644 --- a/doc/manual/source/release-notes/rl-2.24.md +++ b/doc/manual/source/release-notes/rl-2.24.md @@ -269,7 +269,7 @@ e.g. `--warn-large-path-threshold 100M`. -# Contributors +## Contributors This release was made possible by the following 43 contributors: diff --git a/doc/manual/source/release-notes/rl-2.25.md b/doc/manual/source/release-notes/rl-2.25.md index 29e3e509cf0..cfde8b1ef8f 100644 --- a/doc/manual/source/release-notes/rl-2.25.md +++ b/doc/manual/source/release-notes/rl-2.25.md @@ -77,7 +77,7 @@ `` is also known as the builtin derivation builder `builtin:fetchurl`. It's not to be confused with the evaluation-time function `builtins.fetchurl`, which was not affected by this issue. -# Contributors +## Contributors This release was made possible by the following 58 contributors: diff --git a/doc/manual/source/release-notes/rl-2.26.md b/doc/manual/source/release-notes/rl-2.26.md index d2a890eb694..0c3df828f79 100644 --- a/doc/manual/source/release-notes/rl-2.26.md +++ b/doc/manual/source/release-notes/rl-2.26.md @@ -76,7 +76,7 @@ - Evaluation caching now works for dirty Git workdirs [#11992](https://github.com/NixOS/nix/pull/11992) -# Contributors +## Contributors This release was made possible by the following 45 contributors: diff --git a/doc/manual/source/release-notes/rl-2.27.md b/doc/manual/source/release-notes/rl-2.27.md index 3643f747638..34da6252578 100644 --- a/doc/manual/source/release-notes/rl-2.27.md +++ b/doc/manual/source/release-notes/rl-2.27.md @@ -47,7 +47,7 @@ blake3-34P4p+iZXcbbyB1i4uoF7eWCGcZHjmaRn6Y7QdynLwU= ``` -# Contributors +## Contributors This release was made possible by the following 21 contributors: diff --git a/doc/manual/source/release-notes/rl-2.28.md b/doc/manual/source/release-notes/rl-2.28.md index 6da09546efe..93ea2cfdedf 100644 --- a/doc/manual/source/release-notes/rl-2.28.md +++ b/doc/manual/source/release-notes/rl-2.28.md @@ -82,7 +82,7 @@ This completes the infrastructure overhaul for the [RFC 132](https://github.com/ Although this change is not as critical, we figured it would be good to do this API change at the same time, also. Also note that we try to keep the C API compatible, but we decided to break this function because it was young and likely not in widespread use yet. This frees up time to make important progress on the rest of the C API. -# Contributors +## Contributors This earlier-than-usual release was made possible by the following 16 contributors: diff --git a/doc/manual/source/release-notes/rl-2.29.md b/doc/manual/source/release-notes/rl-2.29.md index ad63fff2fe5..b59d6d6f03f 100644 --- a/doc/manual/source/release-notes/rl-2.29.md +++ b/doc/manual/source/release-notes/rl-2.29.md @@ -111,7 +111,7 @@ This fact is counterbalanced by the fact that most of those changes are bug fixe This in particular prevents parts of GCC 14's diagnostics from being improperly filtered away. -# Contributors +## Contributors This release was made possible by the following 40 contributors: diff --git a/maintainers/release-notes b/maintainers/release-notes index 6586b22dc27..5bb492227b7 100755 --- a/maintainers/release-notes +++ b/maintainers/release-notes @@ -157,7 +157,7 @@ section_title="Release $version_full ($DATE)" if ! $IS_PATCH; then echo - echo "# Contributors" + echo "## Contributors" echo VERSION=$version_full ./maintainers/release-credits fi From a492493d97d0d2beceb8fc6d20b4ea2d2459d2a5 Mon Sep 17 00:00:00 2001 From: Eelco Dolstra Date: Mon, 7 Jul 2025 15:01:18 +0200 Subject: [PATCH 213/218] Rearrange release notes --- doc/manual/source/release-notes/rl-2.30.md | 66 ++++++++++++---------- 1 file changed, 37 insertions(+), 29 deletions(-) diff --git a/doc/manual/source/release-notes/rl-2.30.md b/doc/manual/source/release-notes/rl-2.30.md index 05e1d7fcb1e..c5b741a1f02 100644 --- a/doc/manual/source/release-notes/rl-2.30.md +++ b/doc/manual/source/release-notes/rl-2.30.md @@ -1,24 +1,6 @@ # Release 2.30.0 (2025-07-07) -- Reduce the size of value from 24 to 16 bytes [#13407](https://github.com/NixOS/nix/pull/13407) - - This shaves off a very significant amount of memory used for evaluation (~20% percent reduction in maximum heap size and ~17% in total bytes). - -- `builtins.sort` uses PeekSort [#12623](https://github.com/NixOS/nix/pull/12623) - - Previously it used libstdc++'s `std::stable_sort()`. However, that implementation is not reliable if the user-supplied comparison function is not a strict weak ordering. - -- `nix repl` prints which variables were loaded [#11406](https://github.com/NixOS/nix/pull/11406) - - Instead of `Added variables` it now prints the first 10 variables that were added to the global scope. - -- `nix flake archive`: add `--no-check-sigs` option [#13277](https://github.com/NixOS/nix/pull/13277) - - This is useful when using `nix flake archive` with the destination set to a remote store. - -- Emit warnings for IFDs with `trace-import-from-derivation` option [#13279](https://github.com/NixOS/nix/pull/13279) - - While we have the setting `allow-import-from-derivation` to deny import-from-derivation (IFD), sometimes users would like to observe IFDs during CI processes to gradually phase out the idiom. The new setting `trace-import-from-derivation`, when set, logs a simple warning to the console. +## Backward-incompatible changes and deprecations - `build-dir` no longer defaults to `$TMPDIR` @@ -28,7 +10,7 @@ cause build impurities even when not used maliciously. We now default to `builds` in `NIX_STATE_DIR` (which is `/nix/var/nix/builds` in the default configuration). -- Deprecate manually making structured attrs with `__json = ...;` [#13220](https://github.com/NixOS/nix/pull/13220) +- Deprecate manually making structured attrs using the `__json` attribute [#13220](https://github.com/NixOS/nix/pull/13220) The proper way to create a derivation using [structured attrs] in the Nix language is by using `__structuredAttrs = true` with [`builtins.derivation`]. However, by exploiting how structured attrs are implementated, it has also been possible to create them by setting the `__json` environment variable to a serialized JSON string. @@ -37,6 +19,20 @@ [structured attrs]: @docroot@/language/advanced-attributes.md#adv-attr-structuredAttrs [`builtins.derivation`]: @docroot@/language/builtins.html#builtins-derivation +- Rename `nix profile install` to `nix profile add` [#13224](https://github.com/NixOS/nix/pull/13224) + + The command `nix profile install` has been renamed to `nix profile add` (though the former is still available as an alias). This is because the verb "add" is a better antonym for the verb "remove" (i.e. `nix profile remove`). Nix also does not have install hooks or general behavior often associated with "installing". + +## Performance improvements + +This release has a number performance improvements, in particular: + +- Reduce the size of value from 24 to 16 bytes [#13407](https://github.com/NixOS/nix/pull/13407) + + This shaves off a very significant amount of memory used for evaluation (~20% percent reduction in maximum heap size and ~17% in total bytes). + +## Features + - Add stack sampling evaluation profiler [#13220](https://github.com/NixOS/nix/pull/13220) Nix evaluator now supports stack sampling evaluation profiling via `--eval-profiler flamegraph` setting. @@ -48,13 +44,21 @@ Unlike existing `--trace-function-calls` this profiler includes the name of the function being called when it's available. -- `json-log-path` setting [#13003](https://github.com/NixOS/nix/pull/13003) +- `nix repl` prints which variables were loaded [#11406](https://github.com/NixOS/nix/pull/11406) - New setting `json-log-path` that sends a copy of all Nix log messages (in JSON format) to a file or Unix domain socket. + Instead of `Added variables` it now prints the first 10 variables that were added to the global scope. -- Rename `nix profile install` to `nix profile add` [#13224](https://github.com/NixOS/nix/pull/13224) +- `nix flake archive`: Add `--no-check-sigs` option [#13277](https://github.com/NixOS/nix/pull/13277) - The command `nix profile install` has been renamed to `nix profile add` (though the former is still available as an alias). This is because the verb "add" is a better antonym for the verb "remove" (i.e. `nix profile remove`). Nix also does not have install hooks or general behavior often associated with "installing". + This is useful when using `nix flake archive` with the destination set to a remote store. + +- Emit warnings for IFDs with `trace-import-from-derivation` option [#13279](https://github.com/NixOS/nix/pull/13279) + + While we have the setting `allow-import-from-derivation` to deny import-from-derivation (IFD), sometimes users would like to observe IFDs during CI processes to gradually phase out the idiom. The new setting `trace-import-from-derivation`, when set, logs a simple warning to the console. + +- `json-log-path` setting [#13003](https://github.com/NixOS/nix/pull/13003) + + New setting `json-log-path` that sends a copy of all Nix log messages (in JSON format) to a file or Unix domain socket. - Non-flake inputs now contain a `sourceInfo` attribute [#13164](https://github.com/NixOS/nix/issues/13164) [#13170](https://github.com/NixOS/nix/pull/13170) @@ -70,13 +74,19 @@ This iterates on the work done in 2.26 to improve relative path support ([#10089](https://github.com/NixOS/nix/pull/10089)), and resolves a regression introduced in 2.28 relating to nested relative path inputs ([#13164](https://github.com/NixOS/nix/issues/13164)). +## Miscellaneous changes + +- `builtins.sort` uses PeekSort [#12623](https://github.com/NixOS/nix/pull/12623) + + Previously it used libstdc++'s `std::stable_sort()`. However, that implementation is not reliable if the user-supplied comparison function is not a strict weak ordering. + - Revert incomplete closure mixed download and build feature [#77](https://github.com/NixOS/nix/issues/77) [#12628](https://github.com/NixOS/nix/issues/12628) [#13176](https://github.com/NixOS/nix/pull/13176) Since Nix 1.3 (299141ecbd08bae17013226dbeae71e842b4fdd7 in 2013) Nix has attempted to mix together upstream fresh builds and downstream substitutions when remote substuters contain an "incomplete closure" (have some store objects, but not the store objects they reference). This feature is now removed. - Worst case, removing this feature could cause more building downstream, but it should not cause outright failures, since this is not happening for opaque store objects that we don't know how to build if we decide not to substitute. - In practice, however, we doubt even the more building is very likely to happen. + In the worst case, removing this feature could cause more building downstream, but it should not cause outright failures, since this is not happening for opaque store objects that we don't know how to build if we decide not to substitute. + In practice, however, we doubt even more building is very likely to happen. Remote stores that are missing dependencies in arbitrary ways (e.g. corruption) don't seem to be very common. On the contrary, when remote stores fail to implement the [closure property](@docroot@/store/store-object.md#closure-property), it is usually an *intentional* choice on the part of the remote store, because it wishes to serve as an "overlay" store over another store, such as `https://cache.nixos.org`. @@ -84,9 +94,7 @@ (In the future, we should make it easier for remote stores to indicate this to clients, to catch settings that won't work in general before a missing dependency is actually encountered.) - -# Contributors - +## Contributors This release was made possible by the following 32 contributors: From 8c71de202fc7b110cf1b7a28d20241fcab96f392 Mon Sep 17 00:00:00 2001 From: Eelco Dolstra Date: Mon, 7 Jul 2025 15:49:12 +0200 Subject: [PATCH 214/218] Add link --- doc/manual/source/release-notes/rl-2.30.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/doc/manual/source/release-notes/rl-2.30.md b/doc/manual/source/release-notes/rl-2.30.md index c5b741a1f02..cb96988463b 100644 --- a/doc/manual/source/release-notes/rl-2.30.md +++ b/doc/manual/source/release-notes/rl-2.30.md @@ -35,13 +35,13 @@ This release has a number performance improvements, in particular: - Add stack sampling evaluation profiler [#13220](https://github.com/NixOS/nix/pull/13220) - Nix evaluator now supports stack sampling evaluation profiling via `--eval-profiler flamegraph` setting. - It collects collapsed call stack information to output file specified by + The Nix evaluator now supports [stack sampling evaluation profiling](@docroot@/advanced-topics/eval-profiler.md) via the `--eval-profiler flamegraph` setting. + It outputs collapsed call stack information to the file specified by `--eval-profile-file` (`nix.profile` by default) in a format directly consumable by `flamegraph.pl` and compatible tools like [speedscope](https://speedscope.app/). Sampling frequency can be configured via `--eval-profiler-frequency` (99 Hz by default). - Unlike existing `--trace-function-calls` this profiler includes the name of the function + Unlike the existing `--trace-function-calls`, this profiler includes the name of the function being called when it's available. - `nix repl` prints which variables were loaded [#11406](https://github.com/NixOS/nix/pull/11406) From 9e7655f440bf85c659042c8539ee65ab6f3c84eb Mon Sep 17 00:00:00 2001 From: Thomas Bereknyei Date: Mon, 7 Jul 2025 10:13:40 -0400 Subject: [PATCH 215/218] fix: make setuid tests use new build-dir location /nix/var/nix/builds --- tests/nixos/user-sandboxing/default.nix | 20 +++++++++++--------- 1 file changed, 11 insertions(+), 9 deletions(-) diff --git a/tests/nixos/user-sandboxing/default.nix b/tests/nixos/user-sandboxing/default.nix index 028efd17f1c..3f6b575b035 100644 --- a/tests/nixos/user-sandboxing/default.nix +++ b/tests/nixos/user-sandboxing/default.nix @@ -104,15 +104,16 @@ in # Wait for the build to be ready # This is OK because it runs as root, so we can access everything - machine.wait_for_file("/tmp/nix-build-open-build-dir.drv-0/build/syncPoint") + machine.wait_until_succeeds("stat /nix/var/nix/builds/nix-build-open-build-dir.drv-*/build/syncPoint") + dir = machine.succeed("ls -d /nix/var/nix/builds/nix-build-open-build-dir.drv-*").strip() # But Alice shouldn't be able to access the build directory - machine.fail("su alice -c 'ls /tmp/nix-build-open-build-dir.drv-0/build'") - machine.fail("su alice -c 'touch /tmp/nix-build-open-build-dir.drv-0/build/bar'") - machine.fail("su alice -c 'cat /tmp/nix-build-open-build-dir.drv-0/build/foo'") + machine.fail(f"su alice -c 'ls {dir}/build'") + machine.fail(f"su alice -c 'touch {dir}/build/bar'") + machine.fail(f"su alice -c 'cat {dir}/build/foo'") # Tell the user to finish the build - machine.succeed("echo foo > /tmp/nix-build-open-build-dir.drv-0/build/syncPoint") + machine.succeed(f"echo foo > {dir}/build/syncPoint") with subtest("Being able to execute stuff as the build user doesn't give access to the build dir"): machine.succeed(r""" @@ -124,16 +125,17 @@ in args = [ (builtins.storePath "${create-hello-world}") ]; }' >&2 & """.strip()) - machine.wait_for_file("/tmp/nix-build-innocent.drv-0/build/syncPoint") + machine.wait_until_succeeds("stat /nix/var/nix/builds/nix-build-innocent.drv-*/build/syncPoint") + dir = machine.succeed("ls -d /nix/var/nix/builds/nix-build-innocent.drv-*").strip() # The build ran as `nixbld1` (which is the only build user on the # machine), but a process running as `nixbld1` outside the sandbox # shouldn't be able to touch the build directory regardless - machine.fail("su nixbld1 --shell ${pkgs.busybox-sandbox-shell}/bin/sh -c 'ls /tmp/nix-build-innocent.drv-0/build'") - machine.fail("su nixbld1 --shell ${pkgs.busybox-sandbox-shell}/bin/sh -c 'echo pwned > /tmp/nix-build-innocent.drv-0/build/result'") + machine.fail(f"su nixbld1 --shell ${pkgs.busybox-sandbox-shell}/bin/sh -c 'ls {dir}/build'") + machine.fail(f"su nixbld1 --shell ${pkgs.busybox-sandbox-shell}/bin/sh -c 'echo pwned > {dir}/build/result'") # Finish the build - machine.succeed("echo foo > /tmp/nix-build-innocent.drv-0/build/syncPoint") + machine.succeed(f"echo foo > {dir}/build/syncPoint") # Check that the build was not affected machine.succeed(r""" From 58e07c3291074af6fb3716c90a6b7b54bdd3de7c Mon Sep 17 00:00:00 2001 From: Eelco Dolstra Date: Mon, 7 Jul 2025 16:17:06 +0200 Subject: [PATCH 216/218] Sort contributors --- doc/manual/source/release-notes/rl-2.30.md | 46 +++++++++++----------- 1 file changed, 23 insertions(+), 23 deletions(-) diff --git a/doc/manual/source/release-notes/rl-2.30.md b/doc/manual/source/release-notes/rl-2.30.md index cb96988463b..9c9e63acb9f 100644 --- a/doc/manual/source/release-notes/rl-2.30.md +++ b/doc/manual/source/release-notes/rl-2.30.md @@ -98,37 +98,37 @@ This release has a number performance improvements, in particular: This release was made possible by the following 32 contributors: -- Robert Hensing [**(@roberth)**](https://github.com/roberth) -- Jörg Thalheim [**(@Mic92)**](https://github.com/Mic92) +- Cole Helbling [**(@cole-h)**](https://github.com/cole-h) +- Eelco Dolstra [**(@edolstra)**](https://github.com/edolstra) - Egor Konovalov [**(@egorkonovalov)**](https://github.com/egorkonovalov) -- PopeRigby [**(@poperigby)**](https://github.com/poperigby) -- Peder Bergebakken Sundt [**(@pbsds)**](https://github.com/pbsds) - Farid Zakaria [**(@fzakaria)**](https://github.com/fzakaria) +- Graham Christensen [**(@grahamc)**](https://github.com/grahamc) +- gustavderdrache [**(@gustavderdrache)**](https://github.com/gustavderdrache) - Gwenn Le Bihan [**(@gwennlbh)**](https://github.com/gwennlbh) -- Jade Masker [**(@donottellmetonottellyou)**](https://github.com/donottellmetonottellyou) -- Nikita Krasnov [**(@synalice)**](https://github.com/synalice) -- tomberek [**(@tomberek)**](https://github.com/tomberek) -- Wolfgang Walther [**(@wolfgangwalther)**](https://github.com/wolfgangwalther) -- Samuli Thomasson [**(@SimSaladin)**](https://github.com/SimSaladin) - h0nIg [**(@h0nIg)**](https://github.com/h0nIg) -- Valentin Gagarin [**(@fricklerhandwerk)**](https://github.com/fricklerhandwerk) -- Vladimír Čunát [**(@vcunat)**](https://github.com/vcunat) -- Graham Christensen [**(@grahamc)**](https://github.com/grahamc) +- Jade Masker [**(@donottellmetonottellyou)**](https://github.com/donottellmetonottellyou) +- jayeshv [**(@jayeshv)**](https://github.com/jayeshv) +- Jeremy Fleischman [**(@jfly)**](https://github.com/jfly) +- John Ericson [**(@Ericson2314)**](https://github.com/Ericson2314) +- Jonas Chevalier [**(@zimbatm)**](https://github.com/zimbatm) +- Jörg Thalheim [**(@Mic92)**](https://github.com/Mic92) - kstrafe [**(@kstrafe)**](https://github.com/kstrafe) -- gustavderdrache [**(@gustavderdrache)**](https://github.com/gustavderdrache) +- Luc Perkins [**(@lucperkins)**](https://github.com/lucperkins) - Matt Sturgeon [**(@MattSturgeon)**](https://github.com/MattSturgeon) -- John Ericson [**(@Ericson2314)**](https://github.com/Ericson2314) -- Tristan Ross [**(@RossComputerGuy)**](https://github.com/RossComputerGuy) -- jayeshv [**(@jayeshv)**](https://github.com/jayeshv) -- Eelco Dolstra [**(@edolstra)**](https://github.com/edolstra) +- Nikita Krasnov [**(@synalice)**](https://github.com/synalice) +- Peder Bergebakken Sundt [**(@pbsds)**](https://github.com/pbsds) - pennae [**(@pennae)**](https://github.com/pennae) -- Luc Perkins [**(@lucperkins)**](https://github.com/lucperkins) -- Cole Helbling [**(@cole-h)**](https://github.com/cole-h) +- Philipp Otterbein - Pol Dellaiera [**(@drupol)**](https://github.com/drupol) +- PopeRigby [**(@poperigby)**](https://github.com/poperigby) +- Raito Bezarius +- Robert Hensing [**(@roberth)**](https://github.com/roberth) +- Samuli Thomasson [**(@SimSaladin)**](https://github.com/SimSaladin) - Sergei Zimmerman [**(@xokdvium)**](https://github.com/xokdvium) - Seth Flynn [**(@getchoo)**](https://github.com/getchoo) -- Jonas Chevalier [**(@zimbatm)**](https://github.com/zimbatm) - Stefan Boca [**(@stefanboca)**](https://github.com/stefanboca) -- Jeremy Fleischman [**(@jfly)**](https://github.com/jfly) -- Philipp Otterbein -- Raito Bezarius +- tomberek [**(@tomberek)**](https://github.com/tomberek) +- Tristan Ross [**(@RossComputerGuy)**](https://github.com/RossComputerGuy) +- Valentin Gagarin [**(@fricklerhandwerk)**](https://github.com/fricklerhandwerk) +- Vladimír Čunát [**(@vcunat)**](https://github.com/vcunat) +- Wolfgang Walther [**(@wolfgangwalther)**](https://github.com/wolfgangwalther) From 9f8df6878faec8d0992d64f7a41f90adf6003896 Mon Sep 17 00:00:00 2001 From: Robert Hensing Date: Mon, 7 Jul 2025 16:13:53 +0200 Subject: [PATCH 217/218] doc: Add more links Mostly in the 2.30 release notes --- doc/manual/source/release-notes/rl-2.30.md | 53 +++++++++++++------ src/libexpr/include/nix/expr/eval-settings.hh | 2 + 2 files changed, 38 insertions(+), 17 deletions(-) diff --git a/doc/manual/source/release-notes/rl-2.30.md b/doc/manual/source/release-notes/rl-2.30.md index 9c9e63acb9f..34d3e5bab4c 100644 --- a/doc/manual/source/release-notes/rl-2.30.md +++ b/doc/manual/source/release-notes/rl-2.30.md @@ -2,7 +2,7 @@ ## Backward-incompatible changes and deprecations -- `build-dir` no longer defaults to `$TMPDIR` +- [`build-dir`] no longer defaults to `$TMPDIR` The directory in which temporary build directories are created no longer defaults to `TMPDIR` or `/tmp`, to avoid builders making their directories @@ -19,9 +19,9 @@ [structured attrs]: @docroot@/language/advanced-attributes.md#adv-attr-structuredAttrs [`builtins.derivation`]: @docroot@/language/builtins.html#builtins-derivation -- Rename `nix profile install` to `nix profile add` [#13224](https://github.com/NixOS/nix/pull/13224) +- Rename `nix profile install` to [`nix profile add`] [#13224](https://github.com/NixOS/nix/pull/13224) - The command `nix profile install` has been renamed to `nix profile add` (though the former is still available as an alias). This is because the verb "add" is a better antonym for the verb "remove" (i.e. `nix profile remove`). Nix also does not have install hooks or general behavior often associated with "installing". + The command `nix profile install` has been renamed to [`nix profile add`] (though the former is still available as an alias). This is because the verb "add" is a better antonym for the verb "remove" (i.e. `nix profile remove`). Nix also does not have install hooks or general behavior often associated with "installing". ## Performance improvements @@ -33,39 +33,39 @@ This release has a number performance improvements, in particular: ## Features -- Add stack sampling evaluation profiler [#13220](https://github.com/NixOS/nix/pull/13220) +- Add [stack sampling evaluation profiler] [#13220](https://github.com/NixOS/nix/pull/13220) - The Nix evaluator now supports [stack sampling evaluation profiling](@docroot@/advanced-topics/eval-profiler.md) via the `--eval-profiler flamegraph` setting. + The Nix evaluator now supports [stack sampling evaluation profiling](@docroot@/advanced-topics/eval-profiler.md) via the [`--eval-profiler flamegraph`] setting. It outputs collapsed call stack information to the file specified by - `--eval-profile-file` (`nix.profile` by default) in a format directly consumable + [`--eval-profile-file`] (`nix.profile` by default) in a format directly consumable by `flamegraph.pl` and compatible tools like [speedscope](https://speedscope.app/). - Sampling frequency can be configured via `--eval-profiler-frequency` (99 Hz by default). + Sampling frequency can be configured via [`--eval-profiler-frequency`] (99 Hz by default). - Unlike the existing `--trace-function-calls`, this profiler includes the name of the function + Unlike the existing [`--trace-function-calls`], this profiler includes the name of the function being called when it's available. -- `nix repl` prints which variables were loaded [#11406](https://github.com/NixOS/nix/pull/11406) +- [`nix repl`] prints which variables were loaded [#11406](https://github.com/NixOS/nix/pull/11406) Instead of `Added variables` it now prints the first 10 variables that were added to the global scope. -- `nix flake archive`: Add `--no-check-sigs` option [#13277](https://github.com/NixOS/nix/pull/13277) +- `nix flake archive`: Add [`--no-check-sigs`] option [#13277](https://github.com/NixOS/nix/pull/13277) - This is useful when using `nix flake archive` with the destination set to a remote store. + This is useful when using [`nix flake archive`] with the destination set to a remote store. -- Emit warnings for IFDs with `trace-import-from-derivation` option [#13279](https://github.com/NixOS/nix/pull/13279) +- Emit warnings for IFDs with [`trace-import-from-derivation`] option [#13279](https://github.com/NixOS/nix/pull/13279) - While we have the setting `allow-import-from-derivation` to deny import-from-derivation (IFD), sometimes users would like to observe IFDs during CI processes to gradually phase out the idiom. The new setting `trace-import-from-derivation`, when set, logs a simple warning to the console. + While we have the setting [`allow-import-from-derivation`] to deny import-from-derivation (IFD), sometimes users would like to observe IFDs during CI processes to gradually phase out the idiom. The new setting `trace-import-from-derivation`, when set, logs a simple warning to the console. - `json-log-path` setting [#13003](https://github.com/NixOS/nix/pull/13003) - New setting `json-log-path` that sends a copy of all Nix log messages (in JSON format) to a file or Unix domain socket. + New setting [`json-log-path`] that sends a copy of all Nix log messages (in JSON format) to a file or Unix domain socket. - Non-flake inputs now contain a `sourceInfo` attribute [#13164](https://github.com/NixOS/nix/issues/13164) [#13170](https://github.com/NixOS/nix/pull/13170) Flakes have always had a `sourceInfo` attribute which describes the source of the flake. The `sourceInfo.outPath` is often identical to the flake's `outPath`. However, it can differ when the flake is located in a subdirectory of its source. - Non-flake inputs (i.e. inputs with `flake = false`) can also be located at some path _within_ a wider source. + Non-flake inputs (i.e. inputs with [`flake = false`]) can also be located at some path _within_ a wider source. This usually happens when defining a relative path input within the same source as the parent flake, e.g. `inputs.foo.url = ./some-file.nix`. Such relative inputs will now inherit their parent's `sourceInfo`. @@ -76,13 +76,13 @@ This release has a number performance improvements, in particular: ## Miscellaneous changes -- `builtins.sort` uses PeekSort [#12623](https://github.com/NixOS/nix/pull/12623) +- [`builtins.sort`] uses PeekSort [#12623](https://github.com/NixOS/nix/pull/12623) Previously it used libstdc++'s `std::stable_sort()`. However, that implementation is not reliable if the user-supplied comparison function is not a strict weak ordering. - Revert incomplete closure mixed download and build feature [#77](https://github.com/NixOS/nix/issues/77) [#12628](https://github.com/NixOS/nix/issues/12628) [#13176](https://github.com/NixOS/nix/pull/13176) - Since Nix 1.3 (299141ecbd08bae17013226dbeae71e842b4fdd7 in 2013) Nix has attempted to mix together upstream fresh builds and downstream substitutions when remote substuters contain an "incomplete closure" (have some store objects, but not the store objects they reference). + Since Nix 1.3 ([commit `299141e`] in 2013) Nix has attempted to mix together upstream fresh builds and downstream substitutions when remote substuters contain an "incomplete closure" (have some store objects, but not the store objects they reference). This feature is now removed. In the worst case, removing this feature could cause more building downstream, but it should not cause outright failures, since this is not happening for opaque store objects that we don't know how to build if we decide not to substitute. @@ -132,3 +132,22 @@ This release was made possible by the following 32 contributors: - Valentin Gagarin [**(@fricklerhandwerk)**](https://github.com/fricklerhandwerk) - Vladimír Čunát [**(@vcunat)**](https://github.com/vcunat) - Wolfgang Walther [**(@wolfgangwalther)**](https://github.com/wolfgangwalther) + + +[stack sampling evaluation profiler]: @docroot@/advanced-topics/eval-profiler.md +[`--eval-profiler`]: @docroot@/command-ref/conf-file.md#conf-eval-profiler +[`--eval-profiler flamegraph`]: @docroot@/command-ref/conf-file.md#conf-eval-profiler +[`--trace-function-calls`]: @docroot@/command-ref/conf-file.md#conf-trace-function-calls +[`--eval-profile-file`]: @docroot@/command-ref/conf-file.md#conf-eval-profile-file +[`--eval-profiler-frequency`]: @docroot@/command-ref/conf-file.md#conf-eval-profiler-frequency +[`build-dir`]: @docroot@/command-ref/conf-file.md#conf-build-dir +[`nix profile add`]: @docroot@/command-ref/new-cli/nix3-profile-add.md +[`nix repl`]: @docroot@/command-ref/new-cli/nix3-repl.md +[`nix flake archive`]: @docroot@/command-ref/new-cli/nix3-flake-archive.md +[`json-log-path`]: @docroot@/command-ref/conf-file.md#conf-json-log-path +[`trace-import-from-derivation`]: @docroot@/command-ref/conf-file.md#conf-trace-import-from-derivation +[`allow-import-from-derivation`]: @docroot@/command-ref/conf-file.md#conf-allow-import-from-derivation +[`builtins.sort`]: @docroot@/language/builtins.md#builtins-sort +[`flake = false`]: @docroot@/command-ref/new-cli/nix3-flake.md?highlight=false#flake-inputs +[`--no-check-sigs`]: @docroot@/command-ref/new-cli/nix3-flake-archive.md#opt-no-check-sigs +[commit `299141e`]: https://github.com/NixOS/nix/commit/299141ecbd08bae17013226dbeae71e842b4fdd7 diff --git a/src/libexpr/include/nix/expr/eval-settings.hh b/src/libexpr/include/nix/expr/eval-settings.hh index f88f1387477..eee3b0f0e76 100644 --- a/src/libexpr/include/nix/expr/eval-settings.hh +++ b/src/libexpr/include/nix/expr/eval-settings.hh @@ -211,6 +211,8 @@ struct EvalSettings : Config * `flamegraph` stack sampling profiler. Outputs folded format, one line per stack (suitable for `flamegraph.pl` and compatible tools). Use [`eval-profile-file`](#conf-eval-profile-file) to specify where the profile is saved. + + See [Using the `eval-profiler`](@docroot@/advanced-topics/eval-profiler.md). )"}; Setting evalProfileFile{this, "nix.profile", "eval-profile-file", From 812e0693022c77a3ce77b1342be1d0aef850f2d8 Mon Sep 17 00:00:00 2001 From: Eelco Dolstra Date: Mon, 7 Jul 2025 17:36:13 +0200 Subject: [PATCH 218/218] Mark official release --- flake.nix | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/flake.nix b/flake.nix index 69bd2a21adb..c884fb0ff5b 100644 --- a/flake.nix +++ b/flake.nix @@ -32,7 +32,7 @@ let inherit (nixpkgs) lib; - officialRelease = false; + officialRelease = true; linux32BitSystems = [ "i686-linux" ]; linux64BitSystems = [