2

I have the following script which gets the absolute paths of some directories and pipes them into GNU parallel for zipping. I keep getting a signal 13 error and I'm not sure why.

find $directory -maxdepth 1 | \
grep -v "^${directory}$" | \
xargs realpath | \
parallel -j 4 zip -r ${new_directory}/{/.}.zip {}

The error is terminated with signal 13; aborting. Sometimes the error is not raised but no zip file is present in the output, new_directory.

Any help would be appreciated.

2
  • The first three lines of your script may be combined into find "$(realpath "$directory")" -maxdepth 1 -mindepth 1. That way realpath is also only called once.
    – FelixJN
    Jul 5, 2023 at 14:19
  • 1
    @FelixJN I think he wants to run the realpath for each one of the files, not just for the top directory
    – aviro
    Jul 5, 2023 at 14:39

1 Answer 1

3

Before getting to the actual failure that you're having, there are few things you need to fix in your command to handle some corner cases

find "$directory" -maxdepth 1 -mindepth 1 -print0 | \
xargs -0 realpath | \
parallel -j 4 zip -r "$new_directory"/{/.}.zip {}

The changes in find are:

  • -mindepth 1 - this will exclude your top directory, so you won't need your grep -v command.
  • -print0 - This will solve a possible problem case any of you files include spaces or any escape characters. All the files will be delimited by the the null character instead of a new line.

That's why you also need to add the -0 to your xargs command, so it would read the input delimited by the null characters.

Further troubleshooting

In case it doesn't solve your issue, you'll need to debug it further.

  1. First, remove the parallel command altogether and check if the input for this command looks as expected.

  2. If it seems ok, add verbosity to the parallel command to capture exactly what it's doing:

parallel -t -j 4 zip -r ${new_directory}/{/.}.zip {} 2> parallel.log
  • Then you'll have the list of commands it's running in the parallel.log file. Check if the zip commands seem to be generated correctly.
  1. If you still don't see anything unusual at the list of zip commands in the log file, try the commands in the file:
bash -x parallel.log

At some during the process you'll have to see at what stage you receive the error.

5
  • +1. You can eliminate the pipe to xargs by replacing the -print0 with -exec realpath -z {} +. realpath's -z option tells it to produce NUL-separated output. Also, both $directory and $new_directory need to be double-quoted, and there's no need for the curly braces around $new_directory (see $VAR vs ${VAR} and to quote or not to quote). Finally, GNU parallel needs a -0 or --null option for NUL separated input.
    – cas
    Jul 6, 2023 at 14:41
  • All together, that's find "$directory" -maxdepth 1 -mindepth 1 -exec realpath -z {} + | parallel -0 -t -j 4 zip -r "$new_directory"/{/.}.zip {} 2> parallel.log
    – cas
    Jul 6, 2023 at 14:42
  • @cas, if the number of files in the directory is relatively small, I prefer using xargs, because then realpath will run once for all the files at once instead of for each file. If there are many files and the parallel command will need to wait until all the files are found, it might be better to add -n (max-args) to xargs with some reasonable number.
    – aviro
    Jul 12, 2023 at 12:56
  • 1
    Using + instead of \; to terminate the -exec causes find to fit as many filename args into the command line as it can - just like xargs. xargs is useful when you're not using find (i.e. when you have something else generating your list of arguments, e.g. when the args aren't filenames), but when you're using find, it's redundant...-exec is better, find's own built-in xargs. BTW, parallel has its own -N option for max args, similar to xargs' -n option.
    – cas
    Jul 12, 2023 at 14:14
  • Also worth noting: if you're going to use xargs anyway, xargs has a -P / --max-procs option for parallel execution. It's not as flexible as parallel, but it is much simpler to use and should be good enough for zipping some directories. IMO & IME, a task like that isn't well suited to parallel execution, anyway, because it's a mostly I/O-bound job (especially on any modernish CPU, say <= 20 years old)...and running zip multiple times in parallel is just going to make that worse, with each of the jobs contending with each other for read and write disk I/O.
    – cas
    Jul 12, 2023 at 14:21

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .