From 5b80d53c7c7c30846bf8caa143686639cb55a81e Mon Sep 17 00:00:00 2001 From: Samuel Evans-Powell Date: Thu, 10 Feb 2022 10:56:24 +0800 Subject: [PATCH 1/4] Add documentation for Flakes - Add some beginning resources for Nix Flakes. --- docs/code/Flakes/Explanation/Why.md | 46 ++++++ docs/code/Flakes/Reference/CommonErrors.md | 10 ++ docs/code/Flakes/Reference/Invocations.md | 50 ++++++ docs/code/Flakes/Reference/Schema.md | 112 +++++++++++++ docs/code/Flakes/Tutorials/Hacking.md | 32 ++++ docs/code/Flakes/Tutorials/Installation.md | 40 +++++ docs/code/Flakes/Tutorials/NewProject.md | 182 +++++++++++++++++++++ 7 files changed, 472 insertions(+) create mode 100644 docs/code/Flakes/Explanation/Why.md create mode 100644 docs/code/Flakes/Reference/CommonErrors.md create mode 100644 docs/code/Flakes/Reference/Invocations.md create mode 100644 docs/code/Flakes/Reference/Schema.md create mode 100644 docs/code/Flakes/Tutorials/Hacking.md create mode 100644 docs/code/Flakes/Tutorials/Installation.md create mode 100644 docs/code/Flakes/Tutorials/NewProject.md diff --git a/docs/code/Flakes/Explanation/Why.md b/docs/code/Flakes/Explanation/Why.md new file mode 100644 index 0000000..eff1378 --- /dev/null +++ b/docs/code/Flakes/Explanation/Why.md @@ -0,0 +1,46 @@ +--- +title: Why Flakes? +date: 2021-02-10 +--- + +Nix Flakes are a new feature of the "nix package manager". They unify the typical ways of declaring inputs and ouputs of a project under a common syntax. + +But why use them? + +## Why? + + - Unify the management of inputs (make redundant tools like `niv`). + - Remove sources of impurity that existed in the previous model. + - Standardize the output format of repositories. + - A Nix expression typically creates a "package" for a repository, for example, the cardano-wallet expression creates a binary package which is available at "cardano-wallet". The location of the attribute in the output attribute set is completely arbitrary. Another repository might put their "package" at "packages.cardano-doohickey". + - Nix flakes attempt to standardize the typical output locations of a Nix expression, so that packages are always available at the same location. + +## Concepts + +From an abstract point of view, a derivation in Nix is a function from inputs to ouputs. Inputs are typically libraries and executables, and the output is typically another library or executable. + +Flakes extend the input/output model of derivations to the entire repository. A single file sits at the root of the repository called `flake.nix`, that might look something like this: + +``` +{ + description = "Example flake"; + + inputs = { + cardano-db-sync.url = "github:input-output-hk/cardano-db-sync?ref=refs/tags/12.0.0"; + haskellNix.url = "github:input-output-hk/haskell.nix"; + # ... other inputs + }; + + outputs = { cardano-db-sync, haskellNix }: { + packages.x86_64-linux.linode-cli = pkgs.python3Packages.callPackage ./pkgs/linode-cli.nix {}; + # ... other outputs + }; +} +``` + +It's conceptually quite simple. Given these inputs, this repository produces a set of outputs. + +IOHK previously made use of `niv`, `niv` was just a different method of pinning inputs that wrote to the file "sources.json". The flakes model isn't necessarily "better" than `niv`, but flakes have other benefits. + +By running `nix flake lock`, the inputs of the repository are pinned down to specific revisions and SHAs and written to `flake.lock`. + diff --git a/docs/code/Flakes/Reference/CommonErrors.md b/docs/code/Flakes/Reference/CommonErrors.md new file mode 100644 index 0000000..195d8e4 --- /dev/null +++ b/docs/code/Flakes/Reference/CommonErrors.md @@ -0,0 +1,10 @@ +--- +title: Common Errors +date: 2021-02-09 +--- + +``` +error: getting status of '/nix/store/1dg75ahm07ah26phnd1jy1jq06s680ps-source/some-file.nix': No such file or directory +``` + +Make sure the file is staged to version control. Flakes can't see files that are not tracked by version control. diff --git a/docs/code/Flakes/Reference/Invocations.md b/docs/code/Flakes/Reference/Invocations.md new file mode 100644 index 0000000..c9687aa --- /dev/null +++ b/docs/code/Flakes/Reference/Invocations.md @@ -0,0 +1,50 @@ +--- +title: Invocations and Pathing +date: 2021-02-10 +--- + +## Invocations + +`nix-build` is now `nix build` + +`nix-shell` is now `nix develop` + +`nix-repl default.nix` is now: + +``` +$ nix repl +> :lf . +# Get some information: +> legacyPackages.x86_64-linux.pkgs.stdenv.hostPlatform.libc +# Build cardano-wallet: +> :b packages.x86_64-linux.cardano-wallet +``` + +## Pathing + +`nix-build -A cardano-wallet` is now `nix build .#packages.x86_64-linux.cardano-wallet` + +i.e. + +``` +nix build $FLAKE_PATH +$FLAKE_PATH = $PATH_TO_FLAKE_FILE#$ATTRIBUTE +$FLAKE_PATH = .#packages.x86_64-linux.cardano-cli +``` + +The `$PATH_TO_FLAKE_FILE` can be quite complicated: + +``` + · .: The flake in the current directory. + · /home/alice/src/patchelf: A flake in some other directory. + · nixpkgs: The nixpkgs entry in the flake registry. + · nixpkgs/a3a3dda3bacf61e8a39258a0ed9c924eeca8e293: The nixpkgs entry in the flake registry, with its Git revision overridden to a specific value. + · github:NixOS/nixpkgs: The master branch of the NixOS/nixpkgs repository on GitHub. + · github:NixOS/nixpkgs/nixos-20.09: The nixos-20.09 branch of the nixpkgs repository. + · github:NixOS/nixpkgs/a3a3dda3bacf61e8a39258a0ed9c924eeca8e293: A specific revision of the nixpkgs repository. + · github:edolstra/nix-warez?dir=blender: A flake in a subdirectory of a GitHub repository. + · git+https://github.com/NixOS/patchelf: A Git repository. + · git+https://github.com/NixOS/patchelf?ref=master: A specific branch of a Git repository. + · git+https://github.com/NixOS/patchelf?ref=master&rev=f34751b88bd07d7f44f5cd3200fb4122bf916c7e: A specific branch and revision of a Git repository. + · https://github.com/NixOS/patchelf/archive/master.tar.gz: A tarball flake. +``` diff --git a/docs/code/Flakes/Reference/Schema.md b/docs/code/Flakes/Reference/Schema.md new file mode 100644 index 0000000..5aa06c0 --- /dev/null +++ b/docs/code/Flakes/Reference/Schema.md @@ -0,0 +1,112 @@ +--- +title: Nix Flake Schema +date: 2021-02-09 +--- + +The `flake.nix` has the following schema: + +``` +{ + description = "A string"; + inputs = *input_schema; + outputs = *output_schema: +} +``` + +## Input schema + +An example input attrset: + +``` +{ + # A GitHub repository. + import-cargo = { + type = "github"; + owner = "edolstra"; + repo = "import-cargo"; + }; + + import-cargo-alt.url = github:edolstra/import-cargo; + + nixpkgs.url = "nixpkgs"; + + cardano-db-sync.url = "github:input-output-hk/cardano-db-sync?ref=refs/tags/12.0.0"; + + nixpkgs-alt.url = "github:NixOS/nixpkgs/nixos-21.11"; +} +``` + + +## Output schema + +There is a standard schema for the outputs of a flake: + +``` +{ self, cardano-db-sync, nixpkgs, import-cargo, ... } @ inputs: { + # Packages built by the repository. + # Executed by `nix build .#` + packages.""."" = derivation; + + # Executed by `nix build .` + defaultPackage."" = derivation; + + # A default development shell for the project. + # Used by `nix develop`. + devShell."" = derivation; + + # A development shell for the project. + # Used by `nix develop .#` + devShells.""."" = derivation; + + # Hydra build jobs + hydraJobs.""."" = derivation; + + # Automated checks. + # Executed by `nix flake check` + checks.""."" = derivation; + + # Used for nixpkgs packages, also accessible via `nix build .#` + # Typically the repository will provide the version of nixpkgs used in the repository here: + # E.g. + # legacyPackages = import nixpkgs { + # inherit system overlays; + # }; + legacyPackages.""."" = derivation; + + # Used by `nix flake init -t #` + templates."" = { path = ""; description = ""; }; + + # Default overlay, consumed by other flakes. Overlays are typically overlaid onto nixpkgs to add extra packages to the package set. Potentially for another project or for a system configuration. + overlay = final: prev: { }; + + # Same idea as overlay but a list or attrset of them. + overlays = {}; + + # Default module, consumed by other flakes. NixOS modules are used by NixOS configurations to "modularize" configuration. + # For example, by importing the "cardano-node.nixosModule" I can run a simple node on my computer by enabling the option: "services.cardano-node.enable = true". + nixosModule = { config }: { options = {}; config = {}; }; + + # Same idea as nixosModule but a list or attrset of them. + nixosModules = {}; + + # Used with `nixos-rebuild --flake .#` + # nixosConfigurations."".config.system.build.toplevel must be a derivation + nixosConfigurations."" = {}; + + # Used by `nix flake init -t ` + defaultTemplate = { + path = ""; + description = "template description goes here?"; + }; + + # Executed by `nix run .#` + apps.""."" = { + type = "app"; + program = ""; + }; + + # Default application + # Executed by `nix run . -- ` + defaultApp."" = { type = "app"; program = "..."; }; +} +``` diff --git a/docs/code/Flakes/Tutorials/Hacking.md b/docs/code/Flakes/Tutorials/Hacking.md new file mode 100644 index 0000000..2bb0551 --- /dev/null +++ b/docs/code/Flakes/Tutorials/Hacking.md @@ -0,0 +1,32 @@ +--- +title: 03-Create a Development Environment +date: 2021-02-09 +--- + +## Hacking + +`nix-shell` has been replaced by `nix develop`, and by default it builds the attribute `devShell.x86_64-linux`, so a simple python development environment would look like this: + +`flake.nix` +``` +{ + description = "A very basic flake"; + + inputs = { + nixpkgs.url = "github:NixOS/nixpkgs/nixos-21.11"; + }; + + outputs = { self, nixpkgs }: + let + pkgs = nixpkgs.legacyPackages.x86_64-linux + in { + devShell.x86_64-linux = pkgs.mkShell { + nativeBuildInputs = [ + pkgs.python3 + ]; + }; + }; +} +``` + +We could then run `nix develop` and we would enter a shell with access to python. diff --git a/docs/code/Flakes/Tutorials/Installation.md b/docs/code/Flakes/Tutorials/Installation.md new file mode 100644 index 0000000..69ea491 --- /dev/null +++ b/docs/code/Flakes/Tutorials/Installation.md @@ -0,0 +1,40 @@ +--- +title: 01-Installation +date: 2021-02-09 +--- + +Presuming nix is already installed, Nix flakes are available from Nix version 2.4 and up. + +## Non-NixOS + +``` +$ nix-env -iA nixpkgs.nixUnstable # or nixpkgs.nix_2_4 + # or nixpkgs.nix_2_5 + # or nixpkgs.nix_2_6, etc. +``` + +Then edit either `~/.config/nix/nix.conf` or `/etc/nix/nix.conf` and add: + +``` +experimental-features = nix-command flakes +``` + +## NixOS + +Add the following to your NixOS configuration: + +`/etc/nixos/configuration.nix` +``` + nix = { + package = pkgs.nixUnstable; # or pkgs.nix_2_4 + # or pkgs.nix_2_5 + # or pkgs.nix_2_6, etc. + extraOptions = '' + experimental-features = nix-command flakes + ''; + }; +``` + +Then + +`nixos-rebuild switch` diff --git a/docs/code/Flakes/Tutorials/NewProject.md b/docs/code/Flakes/Tutorials/NewProject.md new file mode 100644 index 0000000..9fdd475 --- /dev/null +++ b/docs/code/Flakes/Tutorials/NewProject.md @@ -0,0 +1,182 @@ +--- +title: 02-Create a New Flakes Project +date: 2021-02-09 +--- + +## Creating an example project + +``` +$ git init flake-tutorial +$ cd flake-tutorial +$ nix flake init +``` + +This will create the minimum required for a Nix flake-based project, a single file `flake.nix`: + +`flake.nix` +``` +{ + description = "A very basic flake"; + + outputs = { self, nixpkgs }: { + + packages.x86_64-linux.hello = nixpkgs.legacyPackages.x86_64-linux.hello; + + defaultPackage.x86_64-linux = self.packages.x86_64-linux.hello; + + }; +} +``` + +Ensure that this file is staged in version control, otherwise Nix won't be able to find it. + +This flake takes no inputs (or more accurately, a single input `nixpkgs`, which is implicit), and produces two outputs which can be built with the following commands: + +`nix build .#packages.x86_64-linux.hello` + +and + +`nix build .` (builds the `defaultPackage`) + +Note that the two outputs are for the same package (`hello` in this case). + +The `hello` binary can be run with: + +`./result/bin/hello` + +## The lock file + +You might have noticed that a `flake.lock` file has been generated in your repository. + +This file pins the flake inputs to specific revisions. You should commit this file to version control. + +In this case our input, `nixpkgs`, was implicit, so an arbitrary revision of nixpkgs was chosen (probably based on the version of Nix your system is running). + +If we wanted more control over what version of nixpkgs is used, we could edit our flake.nix file like so: + +`flake.nix` +``` +{ + description = "A very basic flake"; + + inputs = { + nixpkgs.url = "github:NixOS/nixpkgs/nixos-21.11"; + }; + + outputs = { self, nixpkgs }: { + + packages.x86_64-linux.hello = nixpkgs.legacyPackages.x86_64-linux.hello; + + defaultPackage.x86_64-linux = self.packages.x86_64-linux.hello; + + }; +} +``` + +If we then run `nix flake lock`, our nixpkgs revision will be updated to the head of the `nixos-21.11` branch. + +Subsequent builds will use nixpkgs v21.11 as the base set of packages. + +## Building a python application + +Instead of building `hello`, let's build a python application not available in `nixpkgs`: `linode-cli` (more accurately, it is available, but is out-of-date). + +First copy and paste the following linode-cli derivation (the derivation is out-of-scope for this tutorial): + +`linode-cli.nix`: +``` +{ lib +, buildPythonApplication +, fetchFromGitHub +, fetchpatch +, fetchurl +, terminaltables +, colorclass +, requests +, pyyaml +, setuptools +}: + +let + + spec = fetchurl { + url = "https://raw.githubusercontent.com/linode/linode-api-docs/v4.111.0/openapi.yaml"; + sha256 = "0j1i4ig1gwvwg2vfydpkh5skdirmbbfqbrznaq6v7sz35bk7carl"; + }; + +in + +buildPythonApplication rec { + pname = "linode-cli"; + version = "5.13.2"; + + src = fetchFromGitHub { + owner = "linode"; + repo = pname; + rev = version; + sha256 = "10mlkkprky7qqjrkv43v1lzmlgdjpkzy3729k9xxdm5mpq5bjdwj"; + }; + + # remove need for git history + prePatch = '' + substituteInPlace setup.py \ + --replace "version=get_version()," "version='${version}'," + ''; + + propagatedBuildInputs = [ + terminaltables + colorclass + requests + pyyaml + setuptools + ]; + + postConfigure = '' + python3 -m linodecli bake ${spec} --skip-config + cp data-3 linodecli/ + ''; + + # requires linode access token for unit tests, and running executable + doCheck = false; + + meta = with lib; { + homepage = "https://github.com/linode/linode-cli"; + description = "The Linode Command Line Interface"; + license = licenses.bsd3; + maintainers = with maintainers; [ ryantm ]; + }; + +} +``` + +Make sure to stage this file! + +Modify your `flake.nix` file to build the python package: + +`flake.nix`: +``` +{ + description = "A very basic flake"; + + inputs = { + nixpkgs.url = "github:NixOS/nixpkgs/nixos-21.11"; + }; + + outputs = { self, nixpkgs }: { + + packages.x86_64-linux.linode-cli = + nixpkgs.legacyPackages.x86_64-linux.python3Packages.callPackage ./linode-cli.nix {}; + + defaultPackage.x86_64-linux = self.packages.x86_64-linux.linode-cli; + + }; +} +``` + +We can now build the package with: + +`nix build .` or `nix build .#packages.x86_64-linux.linode-cli` + +And run the package with: + +`./result/bin/linode-cli` From 6ddddfc738a998d563ce6eeff17f4ff66353d208 Mon Sep 17 00:00:00 2001 From: Samuel Evans-Powell Date: Wed, 16 Feb 2022 12:46:05 +0800 Subject: [PATCH 2/4] Add another common error --- docs/code/Flakes/Reference/CommonErrors.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/docs/code/Flakes/Reference/CommonErrors.md b/docs/code/Flakes/Reference/CommonErrors.md index 195d8e4..b3c5d2d 100644 --- a/docs/code/Flakes/Reference/CommonErrors.md +++ b/docs/code/Flakes/Reference/CommonErrors.md @@ -8,3 +8,9 @@ error: getting status of '/nix/store/1dg75ahm07ah26phnd1jy1jq06s680ps-source/som ``` Make sure the file is staged to version control. Flakes can't see files that are not tracked by version control. + +``` +warning: Git tree '/home/rodney/iohk/cardano-wallet' is dirty +``` + +It can be hidden (if desired) by using the allow-dirty = true option in nix.conf. From b642519773f09c827041500fb10d3112f28e23f1 Mon Sep 17 00:00:00 2001 From: Samuel Evans-Powell Date: Wed, 16 Feb 2022 12:47:53 +0800 Subject: [PATCH 3/4] Use 'order' property for ordering --- docs/code/Flakes/Tutorials/Hacking.md | 3 ++- docs/code/Flakes/Tutorials/Installation.md | 3 ++- docs/code/Flakes/Tutorials/NewProject.md | 3 ++- 3 files changed, 6 insertions(+), 3 deletions(-) diff --git a/docs/code/Flakes/Tutorials/Hacking.md b/docs/code/Flakes/Tutorials/Hacking.md index 2bb0551..60ecc89 100644 --- a/docs/code/Flakes/Tutorials/Hacking.md +++ b/docs/code/Flakes/Tutorials/Hacking.md @@ -1,6 +1,7 @@ --- -title: 03-Create a Development Environment +title: Create a Development Environment date: 2021-02-09 +order: 3 --- ## Hacking diff --git a/docs/code/Flakes/Tutorials/Installation.md b/docs/code/Flakes/Tutorials/Installation.md index 69ea491..0fdfe2a 100644 --- a/docs/code/Flakes/Tutorials/Installation.md +++ b/docs/code/Flakes/Tutorials/Installation.md @@ -1,6 +1,7 @@ --- -title: 01-Installation +title: Installation date: 2021-02-09 +order: 1 --- Presuming nix is already installed, Nix flakes are available from Nix version 2.4 and up. diff --git a/docs/code/Flakes/Tutorials/NewProject.md b/docs/code/Flakes/Tutorials/NewProject.md index 9fdd475..22bd434 100644 --- a/docs/code/Flakes/Tutorials/NewProject.md +++ b/docs/code/Flakes/Tutorials/NewProject.md @@ -1,6 +1,7 @@ --- -title: 02-Create a New Flakes Project +title: Create a New Flakes Project date: 2021-02-09 +order: 2 --- ## Creating an example project From 47779b6bc9a25fde920c98d0641b8ae190e9d17d Mon Sep 17 00:00:00 2001 From: Samuel Evans-Powell Date: Wed, 16 Feb 2022 12:49:30 +0800 Subject: [PATCH 4/4] Add link to nix develop man page --- docs/code/Flakes/Tutorials/Hacking.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/docs/code/Flakes/Tutorials/Hacking.md b/docs/code/Flakes/Tutorials/Hacking.md index 60ecc89..aadc21f 100644 --- a/docs/code/Flakes/Tutorials/Hacking.md +++ b/docs/code/Flakes/Tutorials/Hacking.md @@ -31,3 +31,7 @@ order: 3 ``` We could then run `nix develop` and we would enter a shell with access to python. + +## Reference + +[nix develop man page](https://nixos.org/manual/nix/stable/command-ref/new-cli/nix3-develop.html)