8

I'm working with a copy of Raspbian, installed with pi-gen. Pi-gen runs in a Docker container with a volume for the filesystem, running debootstrap and custom scripts inside a chroot to the volume.

I'm running a shell inside the Raspbian filesystem using chroot and qemu-arm-static, but without Docker.

I noticed that the mkinitramfs script was not working. I traced the problem back to dash, which the script is running in.

For some reason dash is not expanding filename wildcards in commands:

# echo /*
/*
# ls /
bin boot dev etc home lib media mnt opt proc root run sbin sys tmp usr var

This happens in all folders inside the chroot and also in scripts. This breaks a lot of stuff.

However, wildcard expansion works normally in filesystems bind-mounted inside the chroot, such as /proc and /run. Also, path expansion using the same dash binary works inside a different chroot.

I've already tried set +f and set +o noglob with no luck. The noglob option is definitely not on:

# set -o
Current option settings
errexit         off
noglob          off
ignoreeof       off
interactive     on
monitor         on
noexec          off
stdin           on
xtrace          off
verbose         off
vi              off
emacs           off
noclobber       off
allexport       off
notify          off
nounset         off
nolog           off
debug           off

I'm running version 0.5.8-2.4 of the dash package from http://raspbian.raspberrypi.org/raspbian stretch/main armhf. The host machine is running Kali Linux 2019.1 with kernel 4.19.0-kali4-amd64.

Has anyone seen a similar problem before? What could I use as a workaround?

Update: The following is the relevant part of the strace dump in a working chroot:

read(0, "echo /*\n", 8192)              = 8
openat(AT_FDCWD, "/", O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = 3
fstat(3, {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
getdents(3, /* 11 entries */, 32768)    = 264
getdents(3, /* 0 entries */, 32768)     = 0
close(3)                                = 0
write(1, "/bin /dev /etc /lib /pls /proc /"..., 46) = 46

The same in the non-working chroot:

read(0, "echo /*\n", 8192)              = 8
openat(AT_FDCWD, "/", O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = 3
fstat(3, {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
getdents64(3, /* 20 entries */, 32768)  = 488
close(3)                                = 0
write(1, "/*\n", 3)                     = 3
  • After failing to expand /* when used with echo, have you verified with ls / that there are in fact visible names there? – Kusalananda Jul 4 '19 at 11:20
  • @Kusalananda Yes. Regardless of folder, expansion fails and ls sees the files. Also happens in scripts. – PurkkaKoodari Jul 4 '19 at 11:20
  • does case /foo in /*) echo yes;; /foo) echo no; esac print yes or no? dash is not using the system's glob(3) function by default, but is using the systems fnmatch(3) for both glob expansion and case pattern matching. – mosvy Jul 4 '19 at 11:49
  • @mosvy It prints yes, as expected. – PurkkaKoodari Jul 4 '19 at 12:08
  • If you have strace, you can try comparing the output of strace -ve trace=getdents,getdents64 >/dev/null dash -c 'echo /*' with that of ... ls /. – mosvy Jul 4 '19 at 12:29
4

I have the exact same symptoms on my machine, even though I am working on a different project (based on docker, not on QEMU), so I cannot prove that it is the issue in your case, but it probably is.

I have a build which is sometimes working, and sometimes failing, because the wildcard does not always get expanded (I am doing ls /tmp/linux*deb).

Strace shows that for the failing case, a call to getdents is returning -1 EOVERFLOW (Value too large for defined data type)

Working case:

[pid 29791] open("/tmp", O_RDONLY|O_NONBLOCK|O_LARGEFILE|O_DIRECTORY|O_CLOEXEC) = 3
[pid 29791] fstat64(3, {st_mode=S_IFDIR|S_ISVTX|0777, st_size=4096, ...}) = 0
[pid 29791] getdents(3, /* 3 entries */, 32768) = 60
[pid 29791] getdents(3, /* 0 entries */, 32768) = 0
[pid 29791] close(3)                    = 0

Failing case:

[pid 25606] open("/tmp", O_RDONLY|O_NONBLOCK|O_LARGEFILE|O_DIRECTORY|O_CLOEXEC) = 3
[pid 25606] fstat64(3, {st_mode=S_IFDIR|S_ISVTX|0777, st_size=100, ...}) = 0
[pid 25606] getdents(3, /* 1 entries */, 32768) = 16
[pid 25606] getdents(3, 0x585a831c, 32768) = -1 EOVERFLOW (Value too large for defined data type)
[pid 25606] close(3)                    = 0

I found a couple of bug-reports referencing the issue (getdents returns a 64 bits value, but the caller expects a 32 bits value in this case):

| improve this answer | |
  • 1
    Thanks for doing the research! I'll self-answer with what I can gather from those links, have an upvote in the meantime. – PurkkaKoodari Apr 24 at 15:22
2
+50

Cause

As Étienne found out (please, upvote them for the research effort), the root cause seems to be a change in glibc 2.28.

  • getdents64 on 64-bit Linux can return d_off values that don't fit in a 32-bit int. On 32-bit systems it only returns 32-bit values.
  • qemu-user that emulates a 32-bit system on 64-bit Linux just passes the syscall through, so a 32-bit userland process can get overflowing values.
  • glibc 2.28 has a change that makes readdir always use getdents64 and error out if d_off doesn't fit. On real 32-bit systems, this must not be causing problems because glibc went through with it. (Previously it called getdents on 32-bit systems, this returns 32-bit values and the kernel does any truncating.)

Fix (or the lack thereof)

This glibc bug is currently tracking progress on this issue. It'll take its time to be fixed and eventually deployed to your distribution.

Workarounds

  • The simplest workaround I can see is running these builds on a system with a 32-bit Linux kernel.
  • If you can downgrade the glibc version (in the chroot!) to 2.27 that would fix the problem, but that might cause dependency problems and will alter the image you're building.
  • One can also apply one of the many patches in the linked threads, but that will require compiling (and probably detaching from your package manager) either your kernel, glibc or (probably the easiest) QEMU.
| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.