Merge pull request NixOS#13569 from Mic92/benchmark

add derivation parser benchmark

Author: Ericson2314

doc/manual/source/SUMMARY.md.in

@@ -128,6 +128,7 @@
 - [Development](development/index.md)
   - [Building](development/building.md)
   - [Testing](development/testing.md)
+  - [Benchmarking](development/benchmarking.md)
   - [Debugging](development/debugging.md)
   - [Documentation](development/documentation.md)
   - [CLI guideline](development/cli-guideline.md)

doc/manual/source/development/benchmarking.md

@@ -0,0 +1,187 @@
+# Running Benchmarks
+
+This guide explains how to build and run performance benchmarks in the Nix codebase.
+
+## Overview
+
+Nix uses the [Google Benchmark](https://github.com/google/benchmark) framework for performance testing. Benchmarks help measure and track the performance of critical operations like derivation parsing.
+
+## Building Benchmarks
+
+Benchmarks are disabled by default and must be explicitly enabled during the build configuration. For accurate results, use a debug-optimized release build.
+
+### Development Environment Setup
+
+First, enter the development shell which includes the necessary dependencies:
+
+```bash
+nix develop .#native-ccacheStdenv
+```
+
+### Configure Build with Benchmarks
+
+From the project root, configure the build with benchmarks enabled and optimization:
+
+```bash
+cd build
+meson configure -Dbenchmarks=true -Dbuildtype=debugoptimized
+```
+
+The `debugoptimized` build type provides:
+- Compiler optimizations for realistic performance measurements
+- Debug symbols for profiling and analysis
+- Balance between performance and debuggability
+
+### Build the Benchmarks
+
+Build the project including benchmarks:
+
+```bash
+ninja
+```
+
+This will create benchmark executables in the build directory. Currently available:
+- `build/src/libstore-tests/nix-store-benchmarks` - Store-related performance benchmarks
+
+Additional benchmark executables will be created as more benchmarks are added to the codebase.
+
+## Running Benchmarks
+
+### Basic Usage
+
+Run benchmark executables directly. For example, to run store benchmarks:
+
+```bash
+./build/src/libstore-tests/nix-store-benchmarks
+```
+
+As more benchmark executables are added, run them similarly from their respective build directories.
+
+### Filtering Benchmarks
+
+Run specific benchmarks using regex patterns:
+
+```bash
+# Run only derivation parser benchmarks
+./build/src/libstore-tests/nix-store-benchmarks --benchmark_filter="derivation.*"
+
+# Run only benchmarks for hello.drv
+./build/src/libstore-tests/nix-store-benchmarks --benchmark_filter=".*hello.*"
+```
+
+### Output Formats
+
+Generate benchmark results in different formats:
+
+```bash
+# JSON output
+./build/src/libstore-tests/nix-store-benchmarks --benchmark_format=json > results.json
+
+# CSV output
+./build/src/libstore-tests/nix-store-benchmarks --benchmark_format=csv > results.csv
+```
+
+### Advanced Options
+
+```bash
+# Run benchmarks multiple times for better statistics
+./build/src/libstore-tests/nix-store-benchmarks --benchmark_repetitions=10
+
+# Set minimum benchmark time (useful for micro-benchmarks)
+./build/src/libstore-tests/nix-store-benchmarks --benchmark_min_time=2
+
+# Compare against baseline
+./build/src/libstore-tests/nix-store-benchmarks --benchmark_baseline=baseline.json
+
+# Display time in custom units
+./build/src/libstore-tests/nix-store-benchmarks --benchmark_time_unit=ms
+```
+
+## Writing New Benchmarks
+
+To add new benchmarks:
+
+1. Create a new `.cc` file in the appropriate `*-tests` directory
+2. Include the benchmark header:
+   ```cpp
+   #include <benchmark/benchmark.h>
+   ```
+
+3. Write benchmark functions:
+   ```cpp
+   static void BM_YourBenchmark(benchmark::State & state)
+   {
+       // Setup code here
+
+       for (auto _ : state) {
+           // Code to benchmark
+       }
+   }
+   BENCHMARK(BM_YourBenchmark);
+   ```
+
+4. Add the file to the corresponding `meson.build`:
+   ```meson
+   benchmarks_sources = files(
+       'your-benchmark.cc',
+       # existing benchmarks...
+   )
+   ```
+
+## Profiling with Benchmarks
+
+For deeper performance analysis, combine benchmarks with profiling tools:
+
+```bash
+# Using Linux perf
+perf record ./build/src/libstore-tests/nix-store-benchmarks
+perf report
+```
+
+### Using Valgrind Callgrind
+
+Valgrind's callgrind tool provides detailed profiling information that can be visualized with kcachegrind:
+
+```bash
+# Profile with callgrind
+valgrind --tool=callgrind ./build/src/libstore-tests/nix-store-benchmarks
+
+# Visualize the results with kcachegrind
+kcachegrind callgrind.out.*
+```
+
+This provides:
+- Function call graphs
+- Instruction-level profiling
+- Source code annotation
+- Interactive visualization of performance bottlenecks
+
+## Continuous Performance Testing
+
+```bash
+# Save baseline results
+./build/src/libstore-tests/nix-store-benchmarks --benchmark_format=json > baseline.json
+
+# Compare against baseline in CI
+./build/src/libstore-tests/nix-store-benchmarks --benchmark_baseline=baseline.json
+```
+
+## Troubleshooting
+
+### Benchmarks not building
+
+Ensure benchmarks are enabled:
+```bash
+meson configure build | grep benchmarks
+# Should show: benchmarks true
+```
+
+### Inconsistent results
+
+- Ensure your system is not under heavy load
+- Disable CPU frequency scaling for consistent results
+- Run benchmarks multiple times with `--benchmark_repetitions`
+
+## See Also
+
+- [Google Benchmark documentation](https://github.com/google/benchmark/blob/main/docs/user_guide.md)

meson.options

@@ -20,3 +20,10 @@ option(
   value : true,
   description : 'Build language bindings (e.g. Perl)',
 )
+
+option(
+  'benchmarks',
+  type : 'boolean',
+  value : false,
+  description : 'Build benchmarks (requires gbenchmark)',
+)

packaging/dev-shell.nix

@@ -126,7 +126,8 @@ pkgs.nixComponents2.nix-util.overrideAttrs (
       ++ lib.optional stdenv.hostPlatform.isLinux pkgs.buildPackages.mold-wrapped;
 
     buildInputs =
-      attrs.buildInputs or [ ]
+      [ pkgs.gbenchmark ]
+      ++ attrs.buildInputs or [ ]
       ++ pkgs.nixComponents2.nix-util.buildInputs
       ++ pkgs.nixComponents2.nix-store.buildInputs
       ++ pkgs.nixComponents2.nix-store-tests.externalBuildInputs

src/libstore-tests/data/derivation/firefox.drv

@@ -0,0 +1 @@
+Derive([("out","/nix/store/hhg83gh653wjw4ny49xn92f13v2j1za4-hello-2.12.2","","")],[("/nix/store/1xz4avqqrxqsxw7idz119vdzw837p1n1-version-check-hook.drv",["out"]),("/nix/store/bsv47sbqcar3205il55spxqacxp8j0fj-hello-2.12.2.tar.gz.drv",["out"]),("/nix/store/s4b8yadif84kiv8gyr9nxdi6zbg69b4g-bash-5.2p37.drv",["out"]),("/nix/store/sc2pgkzc1s6zp5dp8j7wsd4msilsnijn-stdenv-linux.drv",["out"])],["/nix/store/shkw4qm9qcw5sc5n1k5jznc83ny02r39-default-builder.sh","/nix/store/vj1c3wf9c11a0qs6p3ymfvrnsdgsdcbq-source-stdenv.sh"],"x86_64-linux","/nix/store/p79bgyzmmmddi554ckwzbqlavbkw07zh-bash-5.2p37/bin/bash",["-e","/nix/store/vj1c3wf9c11a0qs6p3ymfvrnsdgsdcbq-source-stdenv.sh","/nix/store/shkw4qm9qcw5sc5n1k5jznc83ny02r39-default-builder.sh"],[("NIX_MAIN_PROGRAM","hello"),("__structuredAttrs",""),("buildInputs",""),("builder","/nix/store/p79bgyzmmmddi554ckwzbqlavbkw07zh-bash-5.2p37/bin/bash"),("cmakeFlags",""),("configureFlags",""),("depsBuildBuild",""),("depsBuildBuildPropagated",""),("depsBuildTarget",""),("depsBuildTargetPropagated",""),("depsHostHost",""),("depsHostHostPropagated",""),("depsTargetTarget",""),("depsTargetTargetPropagated",""),("doCheck","1"),("doInstallCheck","1"),("mesonFlags",""),("name","hello-2.12.2"),("nativeBuildInputs","/nix/store/fxzn6kr5anxn5jgh511x56wrg8b3a99a-version-check-hook"),("out","/nix/store/hhg83gh653wjw4ny49xn92f13v2j1za4-hello-2.12.2"),("outputs","out"),("patches",""),("pname","hello"),("postInstallCheck","stat \"${!outputBin}/bin/hello\"\n"),("propagatedBuildInputs",""),("propagatedNativeBuildInputs",""),("src","/nix/store/dw402azxjrgrzrk6j0p66wkqrab5mwgw-hello-2.12.2.tar.gz"),("stdenv","/nix/store/a13rl87yjhzqrbkc4gb0mrwz2mfkivcf-stdenv-linux"),("strictDeps",""),("system","x86_64-linux"),("version","2.12.2")])

src/libstore-tests/data/derivation/hello.drv

@@ -0,0 +1,45 @@
+#include <benchmark/benchmark.h>
+#include "nix/store/derivations.hh"
+#include "nix/store/store-api.hh"
+#include "nix/util/experimental-features.hh"
+#include "nix/store/store-open.hh"
+#include "nix/store/globals.hh"
+#include <fstream>
+#include <sstream>
+
+using namespace nix;
+
+// Benchmark parsing real derivation files
+static void BM_ParseRealDerivationFile(benchmark::State & state, const std::string & filename)
+{
+    // Read the file once
+    std::ifstream file(filename);
+    std::stringstream buffer;
+    buffer << file.rdbuf();
+    std::string content = buffer.str();
+
+    auto store = openStore("dummy://");
+    ExperimentalFeatureSettings xpSettings;
+
+    for (auto _ : state) {
+        auto drv = parseDerivation(*store, std::string(content), "test", xpSettings);
+        benchmark::DoNotOptimize(drv);
+    }
+    state.SetBytesProcessed(state.iterations() * content.size());
+}
+
+// Register benchmarks for actual test derivation files if they exist
+BENCHMARK_CAPTURE(BM_ParseRealDerivationFile, hello, std::string(NIX_UNIT_TEST_DATA) + "/derivation/hello.drv");
+BENCHMARK_CAPTURE(BM_ParseRealDerivationFile, firefox, std::string(NIX_UNIT_TEST_DATA) + "/derivation/firefox.drv");
+
+// Custom main to initialize Nix before running benchmarks
+int main(int argc, char ** argv)
+{
+    // Initialize libstore
+    nix::initLibStore(false);
+
+    // Initialize and run benchmarks
+    ::benchmark::Initialize(&argc, argv);
+    ::benchmark::RunSpecifiedBenchmarks();
+    return 0;
+}

src/libstore-tests/derivation-parser-bench.cc

@@ -105,3 +105,19 @@ test(
   },
   protocol : 'gtest',
 )
+
+# Build benchmarks if enabled
+if get_option('benchmarks')
+  gbenchmark = dependency('benchmark', required : true)
+
+  benchmark_exe = executable(
+    'nix-store-benchmarks',
+    'derivation-parser-bench.cc',
+    config_priv_h,
+    dependencies : deps_private_subproject + deps_private + deps_other + [gbenchmark],
+    include_directories : include_dirs,
+    link_args: linker_export_flags,
+    install : false,
+    cpp_args : ['-DNIX_UNIT_TEST_DATA="' + meson.current_source_dir() + '/data"'],
+  )
+endif

src/libstore-tests/meson.build

@@ -0,0 +1,9 @@
+# vim: filetype=meson
+
+option(
+  'benchmarks',
+  type : 'boolean',
+  value : false,
+  description : 'Build benchmarks (requires gbenchmark)',
+  yield : true,
+)

src/libstore-tests/meson.options

@@ -35,7 +35,7 @@ mkMesonExecutable (finalAttrs: {
     ../../.version
     ./.version
     ./meson.build
-    # ./meson.options
+    ./meson.options
     (fileset.fileFilter (file: file.hasExt "cc") ./.)
     (fileset.fileFilter (file: file.hasExt "hh") ./.)
   ];