I have an array of values v that I want to remove from the file f. f is NUL delimited. How do I proceed? I tried using sd, but it didn't work.


I have this file:

# cat -v $attic
^@this is 1.
this is 2.
this is 3.
^@blue boy

And this variable $i:

# cat -v <<<"$i"
this is 1.
this is 2.
this is 3.

I want a command that removes $i from that file, resulting in:

^@blue boy

I have tried FROM="$i" perl -pi -e 's/\0\Q$ENV{FROM}\E//g' "$attic", but it doesn't work if $i is multiline. I tried FROM="$i" perl -pi0 -e 's/\0\Q$ENV{FROM}\E//g' "$attic" but this didn't do anything.

  • Please give us an example. There are loads of tools that can deal with NULLs, but we can't help you parse data you don't show us. Edit your question and add a few lines from your input file and the output you need from that input. – terdon Aug 7 at 8:48
  • @terdon Is this good enough? – HappyFace Aug 7 at 9:17
  • Not really :). Please paste the actual file, with the nulls so we can use it to test our solutions. Then show us the output you would want from that file. – terdon Aug 7 at 9:21
  • @terdon now? That is the actual file, btw:)) – HappyFace Aug 7 at 9:24
  • @terdon (^@ means the NUL char) – HappyFace Aug 7 at 9:25

Since you mention zsh, you could do:

zmodload zsh/mapfile
  • $mapfile in the zsh/mapfile module is a special associative array that maps file names to their content.
  • "${(0@)var}": splits $var on NULs (and with @ inside quotes preserves empty elements like "$@").
  • ${array:#pattern}. removes the elements that match the pattern. Here the content of $i is taken literally, you'd need $~i for it to be taken as a pattern (or enable the globsubst option).
  • ${(j:string:)array}: joins the elements of the array with string. With p, \0 is converted to NUL. (note that the 0 parameter expansion flag above can also be written ps:\0:).

You can get something similar with perl with:

FROM=$i perl -0lni -e 'print if $_ ne $ENV{FROM}' -- "$attic"

The difference being that perl will add a NUL at the end if there was not one already (and the usual problems with-i (breaking links, not preserving all metadata...)).

A closer equivalent would be:

FROM=$i perl -0777 -F'\0' -pi -e '
  $_ = join "\0", grep {$_ ne $ENV{FROM}} @F' -- "$attic"


FROM="$i" perl -pi0 -e 's/\0\Q$ENV{FROM}\E//g' "$attic"

doesn't work because:

  • in -pi0: the 0 is taken as the argument to -i (the backup filename suffix)
  • even if you had written -0pi, that couldn't have worked as that tells perl to process NUL-terminated records, so the record ($_) will have the NUL at the end, not beginning. Use -0777 for perl to process the input as one record containing the whole input.
  • Thanks! Your perl solution works flawlessly. I gather -l0 makes it process NUL-started records? Does this load the whole file into memory? And why doesn't my ` FROM="$i" perl -0777 -pi -e 's/(\0|\A)\Q$ENV{FROM}\E(?<sep>\0|\Z)/$+{sep}/gm' "$attic"` replace all occurrences of $i? (It replaces some, but not all. I can't figure out how it decides how many to delete.) – HappyFace Aug 8 at 10:07
  • btw I had to use perl -pi -e 's/\0\Z//' "$attic" to clean up the extra NUL it sometimes left. – HappyFace Aug 8 at 10:14
  • 1
    @HappyFace because each replacement eats both the NUL before and after the match, so when perl resumes searching for the next match, it won't find the immediately following match because of NUL before it missing. You'd need to use a look-ahead operator that don't eat the NULs but still check they're there. – Stéphane Chazelas Aug 8 at 10:32
  • 1
    @HappyFace, -l0 would set the output record separator to \0. Same as $\ = "\0";, but here, I'm using -0l. -0 sets the input record delimiter ($/ = "\0") and -l sets the output record separator to the same value as the input record separator ($\ = $/) and also causes the IRS to be stripped on input. – Stéphane Chazelas Aug 8 at 10:35
  • 1
    @HappyFace The -0l approach reads one NUL delimited record at a time so won't load the whole file in memory if it's larger than perl's internal buffer size. -0777 is the slurp mode so will load the whole file in memory (like that sd does btw). – Stéphane Chazelas Aug 8 at 10:36

Based on your input data, and assuming your file is small enough to fit in memory, this might work for you:

$ export i
$ perl -0777 -pe 's/\Q$ENV{i}\E\n?//g' file 
blue boy

The -0777 causes perl to slurp the whole file into memory. The $ENV{var} is perl's way of accessing exported environment variables. So $ENV{i} will get the value for the exported variable i. The s/old/new/g will replace old with new globally. The \Q and \E ensure that the pattern isn't interpreted as a regular expression. Finally, the \n? is needed since the shell will eat newlines from the ends of variables when assigning the output of a command substitution (e.g. var=$(printf 'foo\n')), so $i might not actually include the final newline.

Note that this will also match substrings. So if i is foo and your file contains foolish, then the foo will be removed leaving ish. If you don't want that, you can use:

perl -0777 -pe 's/\Q$ENV{i}\E(\n|\b)//g' file 

Testing on your example (after replacing ^@ with \0):

$ cat -v file
^@this is 1.
this is 2.
this is 3.
^@blue boy

$ export i="$(printf 'this is 1.\nthis is 2.\nthis is 3.\n')"

$ perl -0777 -pe 's/\Q$ENV{i}\n?\E//g' file 
blue boy

Of course, this assumes that $i has no trailing newline. I can't tell if it does or not since cat <<<"$i" would add one even if it didn't.

If you need to do this for a shell array, you can do:

for i in "${foo[@]}"; do 
    export i
    perl -0777 -i -pe 's/\b\Q$ENV{i}\E(\n|\b)//g' file 

Important: note the -i in the example above. This edits the file in place, so back it up before testing.

  • Thanks! How do I find -007 in perldoc perlrun? BTW, I didn't know pasting preserves NUL. $i also does contain \n, and it needs to match the whole string between two NULs. I'm currently using FROM="$i" perl -007 -pi -e 's/(\0|\A)\Q$ENV{FROM}\E//gm' "$attic" , but this has the bug that it doesn't check for the ending \0. – HappyFace Aug 7 at 10:36
  • @HappyFace yeah, that's kind of why it's so important to give examples that accurately reproduce your issue. I can't debug stuff I cannot see :/ If you edit your question with an actual example that we can use for testing I can try and play with it some more. – terdon Aug 7 at 10:38
  • 1
    Thanks, I debugged it myself to reach the perfect (:-? :D) FROM="$i" perl -007 -pi -e 's/(\0|\A)\Q$ENV{FROM}\E(?<sep>\0|\Z)/$+{sep}/gm' "$attic". – HappyFace Aug 7 at 10:41
  • How do I found the docs for -007?😅 – HappyFace Aug 7 at 10:53
  • 1
    The special value 00 will cause Perl to slurp files in paragraph mode. Any value 0400 or above will cause Perl to slurp files whole, but by convention the value 0777 is the one normally used for this purpose. ? – HappyFace Aug 7 at 10:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.