Self-contained Scripts With Nix

“Nix is a tool that takes a unique approach to package management and system configuration.” - nixos.org

The goal here is pretty simple: help teams create reproducible environments where they can focus on creating value through developing new features.

Reproducible environments abstract some of the complexity of setting up a project, which is usually detailed in the first page of its repository, and explains how to set up the machine before doing any work. This configuration might very well conflict with that of other projects, be outdated, or simply not work on a particular architecture or OS (Operating System).

With this in mind, we are introducing Nix in our projects without developers even noticing it - their workflow stays the same after the introduction of Nix. This incremental approach increases the abstraction level with every step, improving the parity between the environments that an application goes through in the development process.

Nix has different use cases, but in the context of this blog post, we would like to focus on self-contained scripts as a quick introduction to some of the benefits offered by Nix, which revolve around the ideas of isolation and reproducibility.

Introduction to Nix

Before we go any further, let's get started with a brief introduction of Nix, which usually refers to a bunch of things like the package manager, the language, or the Nixpkgs (Nix packages) library. These concepts will be described in the following sections:

Nix Package Manager

The Nix package manager is fully functional and has a few benefits over common ones like apt or Homebrew. For example, being multi-platform, supporting side-by-side installation of multiple versions of a package or being hermetic, which prevents packages from breaking during installation and upgrades.

While most operating systems expect packages in /usr/local/bin or other global paths, Nix places them in the Nix store, which usually lives under /nix/store, and then plays around with the PATH variable to make those tools accessible from everywhere on the system.

A package example could be: /nix/store/<ID>-firefox-33.1, where ID is a cryptographic hash unique for this particular version of the Firefox package that captures all its dependencies. This enables Nix to do a few things:

• Hold multiple versions of the same package as every little change to the package will generate a completely different hash, preventing collisions.
• Perform atomic installs and updates, where nothing is overridden. Nix will only change paths when the package is fully installed. This no-override feature facilitates that different packages use different versions of their dependencies and simplifies rollbacks as nothing gets deleted.

Nix language

The Nix language is used to declare packages and configurations to be built by Nix. It is also purely functional and lazily evaluated.

Nixpkgs library

Finally, Nixpkgs is a repository that holds every package available in the Nix ecosystem. The branches are called channels which group different packages and versions together based on their stability, amount of packages, etc. Those packages are defined using the Nix language and have been built from source with a full dependency tree. After being built, resulting binaries are hosted on a binary cache allowing end users to download and use them without the need of rebuilding them locally.

Self-contained scripts

After this introduction, let’s dive into the main topic of this post and one of the most basic use cases for Nix, self-contained scripts.

Scripts are useful to automate tasks that otherwise would have to be executed by hand. These scripts can be written in a variety of languages, but even the most basic Bash scripts depend on certain commands to be present at run-time. Some of these basic utilities are available in most operating systems, however, differences in versions or flavors of the same tool (GNU vs non-GNU tools) present a challenge for maintainers who want to create portable scripts.

As you might imagine, this looks like a recipe for disaster, and the most common way to solve this problem is by either adding a lot of complexity to a script to try to handle all different edge cases or relying on extensive documentation that is overlooked most of the time. To make things worse, even if we are able to capture the full dependency tree of the script and pin every package to a specific version, the next user will have a hard time replicating that environment, as the average package manager is not flexible enough to pinpoint an exact version for every package.

The end result of all this scripting is a very fragile piece of code which “works on my machine”, at least until you update one of its mutable dependencies, breaking it for every user. Let’s see how self-contained scripts with Nix can help us to avoid this issue with an example of a regular Bash script:

#!/usr/bin/env bash
echo "---AWS CLI---"
echo "Version:" "$(aws --version)"
echo "Location:" "$(which aws)"
echo "---jq---"
echo "Version:" "$(jq --version)"
echo "Location:" "$(which jq)"

Now check the Nix version of it:

#!/usr/bin/env nix-shell
#!nix-shell -i bash -p awscli2 jq
#!nix-shell -I nixpkgs=https://github.com/NixOS/nixpkgs/archive/nixos-22.11.tar.gz
echo "---AWS CLI---"
echo "Version:" "$(aws --version)"
echo "Location:" "$(which aws)"
echo "---jq---"
echo "Version:" "$(jq --version)"
echo "Location:" "$(which jq)"

Assuming you have Nix installed, you can run the above script as you would do with any other bash script and it will use its own, self-contained dependencies to execute.

This is possible thanks to the first three lines, which individually do the following:

• #!/usr/bin/env nix-shell is a shebang, which tells the system to run the script with nix-shell, one of Nix’s tools
• #!nix-shell -i bash -p awscli2 jq is doing a few things:
  • i bash sets the real interpreter, which is the one ultimately running the script
  • p awscli2 jq sets which packages will be included in the shell
• The last line is pinning a specific nixpkgs channel, in this case, nixos-22.11

After this, we have our usual bash code which will make use of every package available in the system, but will give priority to those specified in the nix-shell. This is a source of “impurity”, but our code only uses these two packages (awscli2 and jq), which are properly isolated from the rest, including their own dependencies.

Another point worth mentioning is the ability to pin a specific revision of Nixpkgs, as with the current example, package versions will change over time as the channel is updated. To do that, we would replace the last line with:

#!nix-shell -I

nixpkgs=https://github.com/NixOS/nixpkgs/archive/3c75992f01290979c9c1f997e40efa77845bef1a.tar.gz

Compared to the line in the example script, this one has a specific commit hash, preventing packages from changing over time unless we manually update the channel. The downside of this approach is that old package versions may be removed from the remote binary cache, forcing us to compile them locally from source, causing extensive build times. Depending on the use case we might choose one option or the other.

The other major language for scripting, Python, is also compatible with this kind of shell:

#!/usr/bin/env nix-shell
#!nix-shell -i python -p python37 pythonPackages.prettytable
#!nix-shell -I nixpkgs=https://github.com/NixOS/nixpkgs/archive/nixos-22.11.tar.gz
import prettytable
# Print a simple table
t = prettytable.PrettyTable(["N", "N^2"])
for n in range(1, 10): t.add_row([n, n * n])
print(t)

In this case, we are including a specific version of python (3.7), and some python packages (prettytable) which we would normally install with pip. This way we skip the usual steps of:

• Using pyenv to select python 3.7
• Creating a virtual environment with venv
• Activating this virtual environment
• Installing dependencies with pip
• Running the script
• Deactivating the environment and changing python version again

Conclusions

As we can see, introducing these self-contained scripts in our workflows requires very little effort from the development team, which only needs to have Nix installed, but brings lots of benefits in terms of reproducibility, isolation and ease of use. We can tweak these parameters to accommodate the project’s needs, for example, by being more strict with the pinning or allowing a higher level of impurity. Most advanced use cases of Nix, like development environments, bring the same benefits to the table but instead of just a script, they encapsulate an entire setup, from dependencies to environment variables and everything in between, helping developers reduce the overhead of configuring and switching between different projects while ripping the benefits of reproducibility, isolation and portability.