14

How do I diagnose a nix build failing?

Currently I see output as:

nix build -v
warning: dumping very large path (> 256 MiB); this may run out of memory
building '/nix/store/fdrm6kbm68vld3bhfjizv684ck725lyf-blog.drv'...
builder for '/nix/store/fdrm6kbm68vld3bhfjizv684ck725lyf-blog.drv' failed with exit code 1; last 5 log lines:
  unpacking sources
  unpacking source archive /nix/store/s7r5vlvp49ad6a9d5hqhsiaxw691iyhf-Blog
  source root is Blog
  patching sources
  configuring
[0 built (1 failed), 0.0 MiB DL]
error: build of '/nix/store/fdrm6kbm68vld3bhfjizv684ck725lyf-blog.drv' failed

I'd expect to find some logs / errors of why it failed to build?


Following from https://stackoverflow.com/a/47264375/1663462:

I've tried adding build-cache-failures = true; to the default.nix however I still see no output from:

nix-store --read-log

And nix-store --query-failed-paths results in:

error: no operation specified
Try 'nix-store --help' for more information.

4 Answers 4

7

see also:

to debug the build process of a package:

instead of nix-build

nix-build -E 'with import <nixpkgs> {}; callPackage ./. {}'

run nix-shell to enter the build environment

nix-shell -E 'with import <nixpkgs> {}; callPackage ./. {}'

and then run the build phases

# run build in tempdir
cd $(mktemp -d)

# dont install to /nix/store
for n in $outputs; do eval export $n=$PWD/result-$n; done

# https://github.com/NixOS/nixpkgs/blob/master/pkgs/stdenv/generic/setup.sh
phases="${prePhases[*]:-} unpackPhase patchPhase ${preConfigurePhases[*]:-} \
    configurePhase ${preBuildPhases[*]:-} buildPhase checkPhase \
    ${preInstallPhases[*]:-} installPhase ${preFixupPhases[*]:-} fixupPhase installCheckPhase \
    ${preDistPhases[*]:-} distPhase ${postPhases[*]:-}";

echo writing nix-build-phases.txt
for curPhase in $phases; do
    echo "eval \"\${$curPhase:-$curPhase}\""
    if [ "$curPhase" = unpackPhase ]; then
        echo '[ -n "${sourceRoot:-}" ] && chmod +x "${sourceRoot}"'
        echo 'cd "${sourceRoot:-.}"'
    fi
done >nix-build-phases.txt

echo "please manually run the build phases in nix-build-phases.txt"
echo
cat nix-build-phases.txt

example nix-build-phases.txt

eval "${unpackPhase:-unpackPhase}"
[ -n "${sourceRoot:-}" ] && chmod +x "${sourceRoot}"
cd "${sourceRoot:-.}"
eval "${patchPhase:-patchPhase}"
eval "${updateAutotoolsGnuConfigScriptsPhase:-updateAutotoolsGnuConfigScriptsPhase}"
eval "${configurePhase:-configurePhase}"
eval "${buildPhase:-buildPhase}"
eval "${checkPhase:-checkPhase}"
eval "${installPhase:-installPhase}"
eval "${fixupPhase:-fixupPhase}"
eval "${installCheckPhase:-installCheckPhase}"
eval "${distPhase:-distPhase}"

in this build environment, you can

  • find the cause of build errors (what build phase is failing?)
  • modify the build commands, for example redefine the buildPhase: run type buildPhase to print the old phase, copy the old phase, modify the phase, paste the new phase
  • run the modified build commands. this can be faster than a full rebuild

note: builders can execute different build phases. the actual build phases are stored in variables like prePhases or preConfigurePhases

problem: the build script runs with set -e so it will exit on the first error, but that would exit the interative shell. this is a limitation of the bash shell, so ideally, we would run the build script in a bash debugger, to stop execution on the first error, modify commands, step back and repeat execution of commands, and to allow inspection of state.

# run build in tempdir
cd $(mktemp -d)

# dont install to /nix/store
for n in $outputs; do eval export $n=$PWD/result-$n; done

# https://github.com/NixOS/nixpkgs/blob/master/pkgs/stdenv/generic/setup.sh
phases="${prePhases[*]:-} unpackPhase patchPhase ${preConfigurePhases[*]:-} \
    configurePhase ${preBuildPhases[*]:-} buildPhase checkPhase \
    ${preInstallPhases[*]:-} installPhase ${preFixupPhases[*]:-} fixupPhase installCheckPhase \
    ${preDistPhases[*]:-} distPhase ${postPhases[*]:-}";

# test
#patchPhase="echo test error in patchPhase; false"

donePhases=()
missingPhases=($phases)
for curPhase in $phases; do
    # no. genericBuild does not "return 1" on error
    #phases="$curPhase" genericBuild || break
    echo
    echo "# running $curPhase"
    # FIXME this also traps suspend (Ctrl-Z)
    trap '
        echo "FIXME error in $curPhase"
        echo "done phases:" ${donePhases[@]}
        echo "missing phases:" ${missingPhases[@]}
        echo "hit Enter to close the nix-shell"
        read
    ' EXIT
    set -e # exit on error
    set -x # trace
    eval "${!curPhase:-$curPhase}"
    set +x
    set +e
    if [ "$curPhase" = unpackPhase ]; then
        [ -n "${sourceRoot:-}" ] && chmod +x "${sourceRoot}"
        cd "${sourceRoot:-.}"
    fi
    donePhases+=($curPhase)
    missingPhases=(${missingPhases[@]:1})
done
8
  • I'm not sure how this relates to the question? Oct 16, 2021 at 15:35
  • added more text from the nixos wiki. clearer now?
    – milahu
    Oct 16, 2021 at 16:33
  • Sorry I'm a few years late, but yes this finally does make sense now though I think it's missing some context to highlight that these commands or phases relate to building a derivation. It would be useful to know that these commands are likely exactly the ones being run indirectly via nix-build and hence you get the full output with no truncation or aggregation. Jul 28, 2023 at 11:15
  • nix-build can execute different build phases, for example in python3.pkgs.buildPythonPackage. the actual build phases are stored in the variable $phases, so in the nix-shell, see echo $phases. docs: Building a stdenv package in nix-shell, issues: docs: fix nix-shell commands, docs: Building a stdenv package in nix-shell
    – milahu
    Jul 28, 2023 at 11:54
  • Using this approach when running buildPhase results in: no Makefile, doing nothing. Other commands seem to fail with mkdir: cannot create directory ‘/nix/store/kk9irdv2ppca45fzwwz7422vsmbpgga1-HaskellNixCabalStarter-0.1.0.0’: Read-only file system. Jan 14 at 16:26
5

Reading log should work. Did you specify the derivation? nix-store --read-log /nix/store/fdrm6kbm68vld3bhfjizv684ck725lyf-blog.drv

5
  • 4
    If by that you mean nix-store --read-log /nix/store/fdrm6kbm68vld3bhfjizv684ck725lyf-blog.drv then yes this seems to add additional info. Feb 8, 2019 at 18:36
  • 4
    Sometimes it's also useful to build with --keep-failed to retain the temporary directory of the build. Feb 8, 2019 at 19:19
  • Is this the same functionality that is implemented here github.com/NixOS/nix/pull/2810 with a new --print-build-logs? Jun 20, 2020 at 10:07
  • What I posted reads logs of past builds. What you link appears to be "only" about showing logs of the builds that are currently happening. Jun 22, 2020 at 16:06
  • nix log /nix/store/fdrm6kbm68vld3bhfjizv684ck725lyf-blog.drv Jul 12, 2022 at 2:09
3

There is a --debug option as well as a --print-build-logs option (on nix build but not nix-build) that can provide additional info.

Regarding the specific issue - it might be this bug: https://github.com/NixOS/nix/issues/2176

More information

https://releases.nixos.org/nix/nix-0.16/build/565033/download/1/manual/index.html#id494024

2
  • 1
    From the link: At the beginning of each phase, the set of all shell variables is written to the file env-vars at the top-level build directory. This is useful for debugging: it allows you to recreate the environment in which a build was performed. For instance, if a build fails, then assuming you used the -K flag, you can go to the output directory and “switch” to the environment of the builder
    – Ben Creasy
    Sep 23, 2019 at 1:28
  • 1
    thank you! --debug is exactly what I was looking for :)
    – hraban
    Jun 23, 2022 at 7:40
1

Adding breakpointHook to nativeBuildInputs of the failing derivation is also a powerful technique ― it allows you to connect with cntr and explore the environment as it was at the point that the derivation's build failed.

Thanks to jamesbrock on the NixOS Discourse, the short of it is:

Add the breakpointHook package to the nativeBuildInputs of a mkDerivation.

nativeBuildInputs = [ breakpointHook ];

Install cntr

nix-env -i cntr

Run your Nix build command, and when it fails, the breakpointHook will output a message like this:

build failed in installPhase with exit code 1
To attach install cntr and run the following command as root:

   cntr attach -t command cntr-/nix/store/6vwxqrwq5h1fd3nw4mc61wgk7rppn2qw-jupyterlab-extended

So we run that command as root. The root user doesn’t have a PATH to the cntr command which we installed in our Nix profile, so give it the full path to our single-user-installed cntr.

sudo /home/$USER/.nix-profile/bin/cntr attach -t command cntr-/nix/store/6vwxqrwq5h1fd3nw4mc61wgk7rppn2qw-jupyterlab-extended

And now we’re in the sleeping Nix container.
Note that paths like $out and $TMPDIR are available at this stage along with your usual tools, however, they're under the current directory (/var/lib/cntr), instead of /.
(You can use them like cd $(echo $out | cut -c2-).)

From here, run cntr exec to fully load the build environment.

/home/$USER/.nix-profile/bin/cntr exec

The $TMPDIR directory is where Nix has created the temporary directory for the build 10.

cd $TMPDIR

Now we are fully inside the context of the build, and any commands which we enter in the shell will be as if we had entered that command on a line in the phase (for example, buildPhase) of our derivation when the derivation failed.


Mentioned here in the NixOS Discourse:

I want to also add a few details regarding breakpointHook. It is a really powerful debug hook by @Mic92.

It is enough to put pkgs.breakpointHook into your buildInputs to get it working. In the case of a failure there is an exact “connection” command printed out to the console. You can use this command (cntr) to attach a shell into the build environment. To use it you have to be able to run sudo along with cntr (so you should install cntr package before ;-)).

You can find more interesting details in the @Mic92 NixCon presentation: Jörg 'Mic92' Thalheim - About Nix sandboxes and breakpoints (NixCon 2018)

Here's a link to the relevant section of the manual: https://nixos.org/manual/nixpkgs/stable/#breakpointhook

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .