How to package my software in nix or write my own package derivation for nixpkgs

Question

How can I write a simple derivation to package a program for nix and how can I create a PR to include it in nixpkgs?

(I am writing this as I can't find simple explanations)

Answer 1

NB: this answer is not yet fully complete, but it's already a good starting point. I plan to add more language-specific stuff later (or maybe to create one question per language too keep this answer… """reasonably""" short).

Here are a few references:

• Quickstart in the manual : we will go in more details here
• The Nix Pills (and the section 6 in particular) : they are great but take a bottom-up approach by first explaining all the internals of nix that I find confusing… you may not need to learn all of that to write your first derivation. Here we will take a top-bottom approach, i.e. give the function that most users want to use directly and explain how it works after.
• The documentation for stdenv which is quite nice but contains certainly many information and few (incomplete) examples
• Another answer of mine, but specific to building pre-compiled binaries

Your first derivation

A derivation is, informally, a recipe to build a program. When you cook, you need some ingredients (a.k.a. sources and dependencies in nix) and some steps to combine you ingredients into a cake (a.k.a. programs…).

A simple C program

So let's start with a simple example, the simplest C program that I can imagine. You can write it in a file program.c (for this example) in any folder you like:

#include <stdio.h>

int main() {
   printf("Hello, World!\n");
   return 0;
}

The derivation

Then, we need to say to nix how to compile this program. So create a file derivation.nix:

{ stdenv }:
stdenv.mkDerivation rec {
  name = "program-${version}";
  version = "1.0";

  src = ./.;

  nativeBuildInputs = [ ];
  buildInputs = [ ];

  buildPhase = ''
    gcc program.c -o myprogram
  '';

  installPhase = ''
    mkdir -p $out/bin
    cp myprogram $out/bin
  '';
}

This describes a function, where the inputs are the dependencies (the "ingredients" of the cake; here only stdenv that provides useful utilities) and that outputs a derivation thanks to stdenv.mkDerivation: informally, you can imagine that this process will output a folder with all the compiled files. I provided to mkDerivation some informations:

• the source: here the sources are in the current folder, but you can also take sources from the web as we will see later
• the dependencies nativeBuildInputs needed to compile the program (gcc is always included by default… so you don't need to specify anything here)
• the dependencies buildInputs needed to run the program (you will typically put the libraries here)
• the instructions buildPhase to build the program (it is a bash script). At the beginning of this phase you are dropped in a folder containing the sources
• the instructions installPhase to describe how to "install" the program (see below).

There are actually many more phases (to uncompress the sources, to patch, to configure…) but we don't need them for this example.

What is the installPhase doing?

The install phase is here to say where the final executable/libraries/assets/… should be located. In a typical Linux environment, the binaries are usually copied in /bin, /usr/bin or /usr/local/bin, the libraries in /lib or /lib64, the assets in /share… and it can quickly be a mess when all programs put there own stuff at the same place.

In Nix all programs have there own folder in a path like /nix/store/someUniqueHash-programName-version (the value of this path being set to $out in the installPhase) and the binaries then go to $out/bin, the libraries to $out/lib, the assets to $out/share… reproducing the typical Linux folder hierarchy. So if you are not sure where you should put a file, you surely want to check where you would put it in a normal linux distribution and prepend $out/ to the path (there are few exceptions, like we use $out/bin instead of $out/usr/local/bin since there is no more reasons to have a local folder). Note that many build systems (cmake…) have a variable like PREFIX to say where the program should be installed.:PREFIX might typically be / or /usr/local, and this will install binaries to PREFIX/bin etc. In this case, we can often simply set PREFIX=$out and run the usual compilation commands. When you install the program, Nix will then do the job of properly creating links to the files of the installed softwares, for instance in NixOs the binaries installed globally are linked in /run/current-system/sw/bin

$ ls /run/current-system/sw/bin -al | grep firefox
lrwxrwxrwx 1 root root   70 janv.  1  1970 firefox -> /nix/store/152drilm2qhjimzfx8mch0hmqvr27p29-firefox-99.0.1/bin/firefox

Therefore in our example, in the install phase we just need to create the folder $out/bin and copy the binary obtained during the copy phase… And it's exactly what we did!

How can I try it??

To try it, you still need to specify where the dependencies should be obtained (similarly when you cook a cake you should first visit your favorite farmer to buy some eggs). So create another file default.nix (the name is important here as nix will look for this file first) containing

{ pkgs ? import <nixpkgs> {} }:
pkgs.callPackage ./derivation.nix {}

Here you basically tell nix to use the channel <nixpkgs> to get the dependencies, and callPackage will properly populate the inputs of derivation.nix.

Then, just run

$ nix-build

At the end, you should have a new folder result present, and this folder is linked to the $out folder of your derivation:

$ ls -al | grep result
lrwxrwxrwx  1 leo  users  55 sept. 13 20:59 result -> /nix/store/xi0hx472hzykl6xjw0hnmh0zjyp6sc52-program-1.0

You can then execute the binary using:

$ ./result/bin/myprogram
Hello, World!

Congratulations, you made your first derivation!

We will see below how to package more complex applications. But before, let's see how to install the package and contribute to nixpkgs.

Can I install it on my system?

You can of course install this derivation. Copy your files (except default.nix, it is not needed) in /etc/nixos and change your list of installed packages into:

environment.systemPackages = with pkgs; [
  (callPackage ./derivation.nix {})
]

Here you go!

You can also install it imperatively on any system using

$ nix-env -i -f default.nix

How can I submit my derivation to nixpkgs?

All package expressions in the nixpkgs project are located in https://github.com/NixOS/nixpkgs and you can add your own package there! To do so, first fork (to do Pull Requests) and clone your repository. Then copy the derivation.nix in pkgs/CATEGORY/PACKAGE/default.nix where CATEGORY is appropriately chosen depending on the range of application of your program and PACKAGE is the name of your program.

Of course, the nixpkgs repo does not contain the sources of the program, so you should change the source attribute to point to an external source (see below).

Then, the list of all programs available in nixpkgs is located in pkgs/top-level/all-packages.nix so you should add a line:

  myprogram = callPackage ../CATEGORY/PACKAGE { };

in this file (programs are sorted alphabetically). To test it, go to the root of the repo and call

$ nix-build -A myprogram

it should compile your program and create a result folder to test it as before.

Once it is done, commit and submit your work as a pull request!

If you are not familiar with git or want more details, you might like this thread https://discourse.nixos.org/t/how-to-find-needed-librarys-for-closed-source-bin-applications/39118/43?u=tobiasbora

What if the sources are online?

Most of the time you will be trying to download sources that are hosted online. No problem, just change your src attribute for instance if you download from github (see the list of fetchers here):

{ stdenv, lib, fetchFromGitHub }:
stdenv.mkDerivation rec {
  name = "program-${version}";
  version = "1.0";

  # For https://github.com/myuser/myexample
  src = fetchFromGitHub {
    owner = "myuser";
    repo = "myexample";
    rev = "v${version}"; # If there is a release like v1.0, otherwise put the commit directly 
    sha256 = ""; # <-- dummy hash: after the first compilation this line will give an error and the correct hash. Replace lib.fakeSha256 with "givenhash". Or use nix-prefetch-git. On older nix, this might fail, use sha256 = lib.fakeSha256; instead.
  };

  buildPhase = ''
    gcc program.c -o myprogram
  '';

  installPhase = ''
    mkdir -p $out/bin
    cp myprogram $out/bin
  '';
}

Make sure to change the sha256 line with your own hash (needed to verify that the downloaded files are correct). lib.fakeSha256 is a dummy hash, so the first time you compile it will gave an error saying that the hash is wrong and that it is truehash. So replace the hash with this value (there are also tools like nix-prefetch-git but I have to admit that I don't use them). WARNING: if you use instead the hash of another program already in the cache, it will not give any error, instead it well peak the source of the other package!

Note also that nix will automatically try to do the right thing with the source, in particular it will unpack automatically compressed files downloaded with

  src = fetchurl {
    url = "http://example.org/libfoo-source-${version}.tar.bz2";
    sha256 = "0x2g1jqygyr5wiwg4ma1nd7w4ydpy82z9gkcv8vh2v8dn3y58v5m";
  };

How to use libraries

Now, let us complicate a bit the program by using a library, ncurses for this example. We will use the ncurses hello-world program:

#include <ncurses.h>

int main(int argc, char ** argv)
{
    initscr(); // init screen and sets up screen
    printw("Hello World"); // print to screen
    refresh(); // refreshes the screen
    getch(); // pause the screen output
    endwin(); // deallocates memory and ends ncurses
    return 0;
}

If you compile this program direcly as we did above you will get an error

program.c:1:10: fatal error: ncurses.h: No such file or directory

Which is expected as we have not added ncurses as a dependency. To do that, add in the (space separated) list buildInputs the library ncurses (you must also add it in the first line in the input dependencies): doing so will make sure that the binaries of the programs in buildInputs are available, that the compiler searches for the header files in the include subdirectory… Also update the compilation command with -lncurses:

{ stdenv, ncurses }:
stdenv.mkDerivation rec {
  name = "program-${version}";
  version = "1.0";

  src = ./.;

  buildInputs = [
    ncurses
  ];

  buildPhase = ''
    gcc -lncurses program.c -o myprogram
  '';

  installPhase = ''
    mkdir -p $out/bin
    cp myprogram $out/bin
  '';
}

Compile and run the program as before, that's it!

Debug a program, part 1: nix-shell

It can sometimes be annoying to debug a program using nix-build as nix-build will not cache the compilation: every time it fails, it restarts from scratch the compilation the next time (this is needed to ensure reproducibility). However in practice this can be a bit annoying… nix-shell has been created (also) to solve this problem. If you run the gcc command to compile the above file it will fail directly as gcc and the libraries ncurses are not installed globally (and it's a feature, for instance it allows multiple projects to use different versions of the same library). To create a shell in which this is installed, just run nix-shell, it will automatically check what are the dependencies of the program:

$ nix-shell
$ gcc -lncurses program.c -o myprogram
$ ./myprogram

We will see later more advanced usages of nix-shell.

Save time using the default phases and hooks

The recipe to compile a program is often the same, as many programs are compiled simply using:

$ ./configure --prefix=$out
$ make
$ make install

So nix will by default try the above commands (and more as it tries to patch, test…), that's why many programs in nixpkgs do not really bother writing any phase.

Most of the phases are really configurable: you can for instance enable/disable some parts of the phases, provide some parameters like makeFlags = [ "PREFIX=$(out)" ]; to add flags to the makefile… The whole documentation of these phases is provided in the manual, more specifically in this subsection. If you really want to check what is being run, you can check the function genericBuild in the file pkgs/stdenv/generic/setup.sh that calls then the default phases written above in the file, unless they are overwritten by the derivation. You can also directly read the used code from the nix-shell as we will see later.

Note that these default phases can also be overwritten by dependencies. For instance, if your program uses cmake, adding nativeBuildInputs = [ cmake ]; will automatically adapt the configure phase to use cmake (this can also be configured as documented here). Similar behavior will occur with scons, ninja, meson… More generally, nix defines many "hooks" that will run before or after a given phase, in order to modify the build process. Just including them in nativeBuildInputs should be enough to trigger them. Most hooks are documented here, among others you have:

• autoPatchelfHook that automatically patches (often proprietary) binaries to make them usable in nix (see also my other answer here)

For instance, we can use CMake in our (ncurse) program as follows: create a file CMakeLists.txt containing the usual cmake rules to compile a program:

cmake_minimum_required(VERSION 3.10)

# set the project name
project(myprogram)

# Configure curses as a dependency
find_package(Curses REQUIRED)
include_directories(${CURSES_INCLUDE_DIR})

# add the executable
add_executable(myprogram program.c)

# Link the curses library
target_link_libraries(myprogram ${CURSES_LIBRARIES})

# Explains how to install the program
install(TARGETS myprogram DESTINATION bin)

It is now possible to simplify a lot our derivation.nix:

{ stdenv, ncurses, cmake }:
stdenv.mkDerivation rec {
  name = "program-${version}";
  version = "1.0";

  src = ./.;

  buildInputs = [
    ncurses
    cmake
  ];

}

Debug a program with nix-shell and internals of nix: part 2

NB: this section is not necessary to understand the rest, you can skip it safely.

We saw above how nix-shell could be used to drop us in a shell with all the required dependencies to save compilation time by exploiting caching. In this shell, one can of course run the usual commands to compile a program as before, but it is sometime good to run the exact same commands as the one run by the nix builder.

This is also the opportunity to learn a bit more things about the internals of nix (we also refer to the Nix pills for more details and to the wiki). When you write a derivation, nix will derive from it a .drv file that explains in a simple json format how to build the package.

To see that file, you can run:

$ nix-shell
# (or "nix-shell -A myprogram" if you run it from nixpkgs)
$ nix show-derivation $(nix-instantiate | sed 's/!.*//')
{
  "/nix/store/4ja3vvab4wswalczr7k0lw17dxb69nf7-program-1.0.drv": {
    "outputs": {
      "out": {
        "path": "/nix/store/qv8s0lm7w0az90xjc90dy7rvjqmic9zz-program-1.0"
      }
    },
    "inputSrcs": [
      "/nix/store/9krlzvny65gdc8s7kpb6lkx8cd02c25b-default-builder.sh",
      "/nix/store/zrpp5wmrq39ylqy73pbk3plvw5sx59vh-example"
    ],
    "inputDrvs": {
      "/nix/store/1av43alhcb8a894sz2cnnf9aldfdyb0h-stdenv-linux.drv": [
        "out"
      ],
      "/nix/store/6pj63b323pn53gpw3l5kdh1rly55aj15-bash-5.1-p16.drv": [
        "out"
      ],
      "/nix/store/p6y4zvhi9vjg8h7hli0ix9jxkl225ahk-ncurses-6.3-p20220507.drv": [
        "dev"
      ],
      "/nix/store/w6jf92i16rghx0jr4ix33snq4d237l8i-cmake-3.24.0.drv": [
        "out"
      ]
    },
    "system": "x86_64-linux",
    "builder": "/nix/store/1b9p07z77phvv2hf6gm9f28syp39f1ag-bash-5.1-p16/bin/bash",
    "args": [
      "-e",
      "/nix/store/9krlzvny65gdc8s7kpb6lkx8cd02c25b-default-builder.sh"
    ],
    "env": {
      "buildInputs": "/nix/store/kn8gbpi8bfxkzg6slyskz4y0d2pkl0xk-ncurses-6.3-p20220507-dev /nix/store/xjg2fzw513iig1cghd4mvcq5fh2cyv4y-cmake-3.24.0",
      "builder": "/nix/store/1b9p07z77phvv2hf6gm9f28syp39f1ag-bash-5.1-p16/bin/bash",
      "cmakeFlags": "",
      "configureFlags": "",
      "depsBuildBuild": "",
      "depsBuildBuildPropagated": "",
      "depsBuildTarget": "",
      "depsBuildTargetPropagated": "",
      "depsHostHost": "",
      "depsHostHostPropagated": "",
      "depsTargetTarget": "",
      "depsTargetTargetPropagated": "",
      "doCheck": "",
      "doInstallCheck": "",
      "mesonFlags": "",
      "name": "program-1.0",
      "nativeBuildInputs": "",
      "out": "/nix/store/qv8s0lm7w0az90xjc90dy7rvjqmic9zz-program-1.0",
      "outputs": "out",
      "patches": "",
      "propagatedBuildInputs": "",
      "propagatedNativeBuildInputs": "",
      "src": "/nix/store/zrpp5wmrq39ylqy73pbk3plvw5sx59vh-example",
      "stdenv": "/nix/store/bj5n3k01mq8bysw0rcdm7jxvhc620pd3-stdenv-linux",
      "strictDeps": "",
      "system": "x86_64-linux",
      "version": "1.0"
    }
  }
}

The exact output is not really important, but note that there are a few important parts: first, the derivation specifies the output folder, the sources and dependencies, some environment variables that will be available during the build and that nix-shell automatically populated for us: see the "out": …? you already have it properly configured thanks to nix-shell:

$ echo $out
/nix/store/qv8s0lm7w0az90xjc90dy7rvjqmic9zz-program-1.0

and more importantly these lines:

    "builder": "/nix/store/1b9p07z77phvv2hf6gm9f28syp39f1ag-bash-5.1-p16/bin/bash",
    "args": [
      "-e",
      "/nix/store/9krlzvny65gdc8s7kpb6lkx8cd02c25b-default-builder.sh"
    ],

This means that to produce the outputs, nix will simply run the builder /nix/store/…/bin/bash (here it's simply the bash interpreter) with the arguments -e /nix/store/9krlzvny65gdc8s7kpb6lkx8cd02c25b-default-builder.sh

This file is quite simple:

$ cat /nix/store/9krlzvny65gdc8s7kpb6lkx8cd02c25b-default-builder.sh
source $stdenv/setup
genericBuild

And if you type

$ cat $stdenv/setup

you will realize that it is exactly equal to the pkgs/stdenv/generic/setup.sh file that configured the default phases!

Therefore, in the nix-shell, you can run all the phases at once using something like that (creating a different $out folder allows you not to write in read-only /nix/store):

cd empty_directory # important to make sure "source" folder is not existing, otherwise you get an error like "unpacker appears to have produced no directories". Sources will be unpacked in a subdirectory, and it must be removed every time you restart the download process (otherwise we get the above error).
export out=/tmp/out # Create a temporary folder to put the output of the derivation
set -x # Optional: to display all the command lines, useful to debug sometimes
source $stdenv/setup # In order to load the default phase of the derivation
set +e # Do not quit the shell on error/Ctrl-C ($stdenv/setup adds a "set -e")
genericBuild # start the build process.

You can also just specify a few phases to run by replacing the last line with:

phases="buildPhase" genericBuild

To get the list of phases, you can do:

echo "$phases"

If it is empty, then the default is given for instance via

$ typeset -f genericBuild | grep 'phases='
phases="${prePhases:-} unpackPhase patchPhase ${preConfigurePhases:-}             configurePhase ${preBuildPhases:-} buildPhase checkPhase             ${preInstallPhases:-} installPhase ${preFixupPhases:-} fixupPhase installCheckPhase             ${preDistPhases:-} distPhase ${postPhases:-}"

Packaging other languages

The instructions provided above certainly work for many languages and cases, but some languages provide some other tools to deal with there own requirements in term of environment variables and dependencies (for instance we can't really use pip to install python dependencies). It is hard to list on this page all the existing languages, so here are some generic advices to follow:

• The nixpkgs manual contains basically one section per language, and it is therefore usually a good place to start
• The nixos wiki contains sometimes additional information for a given language. Check if it helps
• The nixpkgs repo contains thousands of programs… someone certainly packaged before a program like the one you are trying to package. Get inspiration, using search online (sometimes it seems to miss some entries) or rg (a nicer grep) to search in your local copy to find derivations using the tools you want to use.

For simplicity I will however put below some cases that you may often encounter.

How to package (often proprietary) binaries

I already made a quite extensive answer here. You are certainly interested by solution 4 (autoPatchElf) or 5-6 (buildFHSUserEnv)… Basically copy your binaries to $out/bin and if your are lucky adding autoPatchelfHook in your nativeBuildInputs should be enough (if the program has assets you can also copy it to $out/opt and put in $out/bin some links or scripts that call the programs in $out/opt).

How to package shell scripts

Let's consider the file myshellscript.sh:

#!/usr/bin/bash

echo "Hello, world"

Just use

{ stdenv }:
stdenv.mkDerivation rec {
  name = "program-${version}";
  version = "1.0";

  src = ./.;

  installPhase = ''
    mkdir -p $out/bin
    cp myshellscript.sh $out/bin
    chmod +x $out/bin/myshellscript.sh # not needed if the file is already executable
  '';
}

and the bash script will automatically be patched by the patchShebangsAuto hook that is present by default in the fixup phase.

Read further to see how to use trivial builders to make this derivation even smaller!

Wrappers, or how to add executables

Let's say that our package needs some executables to work, say cowsay. Because nix tries to maintain "hermiticity" (a.k.a. purity) between packages to limit conflicts as beautifully explained here (maybe different programs need different versions ofcowsay), you cannot assume that cowsay will be "available", i.e. present in the $PATH environment variable. Therefore you need to add cowsay to this variable right before calling your program. This is done via a so-called "wrapper" replacing the original program, that will setup $PATH (and more environment variables if needed) before calling the actual program.

Note that we will see later tools that make this step even simpler for simple bash scripts, but wrappers are useful in many contexts and it's surely not a waste of time to learn how to use them now.

So let's package this myshellscript.sh script:

#!/usr/bin/bash

cowsay "My first wrapper!"

using this derivation.nix:

{ lib, stdenv, cowsay, makeBinaryWrapper}:
stdenv.mkDerivation rec {
  name = "program-${version}";
  version = "1.0";

  src = ./.;
  nativeBuildInputs = [
    makeBinaryWrapper # You can also use makeWrapper to use a bash wrapper, but this won't be compatible with MacOs that expects binary loaders
  ];
  buildInputs = [
    cowsay
  ];

  installPhase = ''
    mkdir -p $out/bin
    cp myshellscript.sh $out/bin
    chmod +x $out/bin/myshellscript.sh
    wrapProgram $out/bin/myshellscript.sh \
      --prefix PATH : ${lib.makeBinPath [ cowsay ]}
  '';
}

Note how we added the input cowsay and how we created the wrapper using:

    wrapProgram $out/bin/myshellscript.sh \
      --prefix PATH : ${lib.makeBinPath [ cowsay ]}
  '';

in order to add cowsay to the path. Now if you nix-build (do not forget the usual default.nix file) you can see that the ./result/bin/myshellscript.sh is now a binary file (that you can still somehow read with less)… since it is hard to see what this file is exactly doing you may want to use makeWrapper instead of makeBinWrapper but be aware that it won't work in MacOs for "security" reasons. Here you would read something like:

$ cat result/bin/myshellscript.sh
#! /nix/store/1b9p07z77phvv2hf6gm9f28syp39f1ag-bash-5.1-p16/bin/bash -e
PATH=${PATH:+':'$PATH':'}
PATH=${PATH/':''/nix/store/mrl0n0kphz0xwvv8qbk2xyz2x1pr2f76-cowsay-3.04/bin'':'/':'}
PATH='/nix/store/mrl0n0kphz0xwvv8qbk2xyz2x1pr2f76-cowsay-3.04/bin'$PATH
PATH=${PATH#':'}
PATH=${PATH%':'}
export PATH
exec -a "$0" "/nix/store/xrz4cv51nd8n1bawfw5i6vd4yizzmajb-program-1.0/bin/.myshellscript.sh-wrapped"  "$@" 

This code is a bit complicated but what it does basically is adding the binary of cowsay at the beginning of the path, and then it execute the shell file that has been moved to $out/bin/.myshellscript.sh-wrapped by the wrapProgram tool.

It is time to test it now:

$ ./result/bin/myshellscript.sh
 ___________________ 
< My first wrapper! >
 ------------------- 
        \   ^__^
         \  (oo)\_______
            (__)\       )\/\
                ||----w |
                ||     ||

Good! Note that you can find the various options to modify the wrappers here.

Even shorter derivations thanks to trivial builders

Sometimes it can be a bit annoying to write a stdenv.mkDerivation with the install phase, the wrappers etc… so trivial builder have been created to wrap stdenv.mkDerivation into a simpler function. They are documented here in the manual. We won't go through all of them (there are some to create new files, scripts, merge derivations…) but we will use it to simplify our code running cowsay.

This way we can simply use this derivation:

{ lib, stdenv, cowsay, writeShellApplication }:
writeShellApplication {
  name = "mycowsay";

  runtimeInputs = [ cowsay ];

  text = ''
    cowsay "My first wrapper!"
  '';
}

and it will create a bash script in $out/bin/mycowsay with the appropriate $PATH based on runtimeInputs. If you prefer instead to write the script in an external file as before, you can do instead:

text = builtins.readFile ./myshellscript.sh;

How to package python scripts

TODO, but see the related topics:

• for a single script, you can use: https://stackoverflow.com/questions/43837691/how-to-package-a-single-python-script-with-nix
• for a whole python application and/or module, you certainly want to use buildPythonPackage (that is based on mkDerivation with few extra stuff), and write a proper setup.py file, see e.g. the entry I added in the wiki.
• if you also package bash scripts that might call python, you also want to wrap it like:

  postFixup = ''
    wrapProgram "$out/bin/mssql-cli" \
      --prefix PYTHONPATH : "$PYTHONPATH"
  '';

Inside a buildPythonPackage, binaries (but not scripts) should (to test) be automatically wrapped. (I'm not a big fan of this, but wrapPythonPrograms only works for binary files I think, so I created this issue)

How to package GTK applications

TODO

How to package QT applications

TODO