Subscripted variable expansion on NUL delimited words in zsh

Question

I have data produced by file, which is identifying the mime-types of all the files in a directory (or directories). The result is a list, with commands on the left and mime-types on the right. By default, these two values (or strings?) are delimited by a colon (:). I want to instead delimit the commands from their mime-types with the NUL character. So,

files=( ${(f)"$(file --print0 --mime-type **/*)"} )

This seems to work.

print -l ${files[@]} gives:

cs:           text/x-shellscript
da:           text/x-shellscript
dns-test:     text/x-shellscript

Or, print -l ${(V)files[@]} gives:

cs^@:           text/x-shellscript
da^@:           text/x-shellscript
dns-test^@:     text/x-shellscript

But, when using subscripting to match the first word, the output is the first character instead. So,

for f in "${files[@]}"; do
  print "${f[(pws:\0:)1]}"
done

This only prints the first character of each array element (each line outputted by file):

c
d
d

If I instead omit the specification of (ps:\0:), thus using the default of separating on white-space, then I do get the first word of each element. So,

for f in "${files[@]}"; do
  print "${f[(w)1]}"
done

This works:

cs
da
dns-test

But I don't want to rely on white-space. I want to use a safer delimiter like NUL. Note that I also tried the subscript [(pws:\000:)1], but this also only produced the first character.

How can I identify words delimited by NUL when working with subscripts?

Answer 1

File paths can contain newline characters. That's why several utilities that report files or things about files offer you to delimit records with NULs instead of newline as 0 is the only byte value that cannot occur in a file path.

If you're splitting the output on newline as you do with ${(f)"$(file...)"}, you're defeating the purpose.

file is a bit of an odd one in that it doesn't use NUL as its record delimiter, but uses it to delimit the file path in its output record (which remains loosely defined).

$ ls
':\0:'$'\n'':\1:'  'a'$'\n''b'
$ file --print0 --mime-type -- ./* | sed -n l
./:\\0:$
:\\1:\000: application/x-xz$
./a$
b\000:       application/zip$

So its record is <file-path><NUL><colon><some-spaces><mime-type><newline> which makes it unnecessarily more difficult to parse (even impossible if the text contains newlines which in the case of --mime-type thankfully should not be the case).

Here, you could do:

typeset -A mime_type
file --print0 --mime-type --no-pad --separator '' ./**/* |
  while
    IFS= read -rd $'\0' file &&
      IFS=' ' read -r type
  do
    mime_type[$file]=$type
  done

Where the first read with NUL as -delimiter reads the NUL-delimited file path and the second reads the newline-delimited mime-type (trimming the leading whitespace with IFS processing).

The --no-pad turns the <some-spaces> into a single space, --separator '' removes the colon, so we have <file-path><NUL><space><mime-type><newline>

In the loop, we fill the $mime_type associative array, so you can thereafter get the type of a given file with $mime_type[./given/file] or list the files of a given type with print -rC1 -- ${(k)mime_type[(R)text/plain]}.

---

As to why $scalar[(pws[\0])1] doesn't split on NULs, that's a bug in current versions of zsh. The behaviour is the same as in $array[(pws[])1] suggesting zsh fails to escape the NUL as it otherwise usually does so you end up splitting on the empty string. That missing escaping affects other separators like all those containing byte values 0x83 to 0xa2 (such as á which in UTF-8 is encoded as 0xc3 0xa1).

Without s[separator], I find it splits on $IFS characters (not whitespace, another bug, a documentation one this time), NUL being in the default value of $IFS, so you could do: IFS=$'\0'; first_non_empty_NUL_separated_field=$scalar[(w)1], but you could also use the 0 (short for ps[\0]) parameter expansion flag:

non_empty_NUL_separated_fields=( ${(0)scalar} )

NUL_separated_fields=( "${(0@)scalar}" )

(and then first=$NUL_separated_fields[1]).

Or in one go first=${${(0)scalar}[1]}.

Or you can use the ${scalar%%pattern} Korn shell operator:

before_first_NUL=${scalar%%$'\0'*}

Or IFS-splitting:

IFS=$'\0'
NUL_separated_fields=( $=scalar )

Or ksh93-style ${param/pattern[/replacement]}:

first=${scalar/$'\0'*}

---

Some comments on your code:

• in file --print0 --mime-type **/*, if any of the file paths starts with -, that would be taken as an option to file. You could change it to file --print0 --mime-type -- **/* to avoid that but that still doesn't work properly if there's a file called - in the current working directory. Using ./**/* avoids both problems (and further problems down the line).
• @, whether as a parameter expansion flag or as [@] or $@ only really makes sense when quoted, to prevent removal of empty elements. ${files[@]} in list context is the same as $files or $files[@] or $files[*] and expands to the non-empty elements in the array. Note that in Korn-like shells, you do also need the quotes, to prevent empty-removal but also to prevent split+glob there. Korn-like shells (contrary to most other shells or languages) also require the braces, and $files alone expand to the element of indice 0, not all the elements.
• With print, you generally want to use the -r option without which print does some backslash processing, and you almost always want to use the -- or - option delimiter without which you introduce command injection vulnerabilities. print $var is an ACE vulnerability. Like in the Korn shell where that builtin comes from, you want print -r - $var or print -r -- $var (though in ksh, you'd need print -r - "$var").
• print -rC1 -- $list is generally better than print -rl -- $list to print a list raw on 1 Column as the latter prints an empty line if passed no argument.