Uncompressed .lzo files in parallel in both the folders simultaneously and then delete the original .lzo files

Question

So I have .lzo files in /test01/primary folder which I need to uncompress and then delete all the .lzo files. Same thing I need to do in /test02/secondary folder as well. I will have around 150 .lzo files in both folders so total around 300 .lzo files.

From a command line I was running like this to uncomressed one file lzop -d file_name.lzo.

What is the fastest way to uncompressed all .lzo files and then delete all .lzo files from both folders simultaneously. Below is the code I have:

#!/bin/bash

set -e

export PRIMARY=/test01/primary
export SECONDARY=/test02/secondary

parallel lzop -dU -- ::: {"$PRIMARY","$SECONDARY"}/*.lzo

I want to uncompress and delete .lzo files parallelly in both PRIMARY and SECONDARY folder simultaneously. With my above code, it does in PRIMARY first and then in SECONDARY folder. How can I achieve parallellism both in PRIMARY and SECONDARY simultaneously?

Also does it uncompress all the files and then delete later on or uncompress one file and then delete that file and then move to next one?

I tried with this but it doesn't work. It just works on first 40 files and after that it doesn't work at all.

#!/bin/bash

set -e

export PRIMARY=/test01/primary
export SECONDARY=/test02/secondary

parallel -j 40 lzop -dU -- ::: "$PRIMARY"/*.lzo &
parallel -j 40 lzop -dU -- ::: "$SECONDARY"/*.lzo &
wait

Answer 1

If I understand correctly, you want to parallelize both directories, but balance the load between (presumably) two storage devices?

The easiest solution is probably to interleave the file names in the input to parallel:

paste -d'\n' <(ls "$PRIMARY"/*.lzo) <(ls "$SECONDARY"/*.lzo) | parallel lzop -dU --

• The paste command takes one line from each file and normally puts them side-by-side in an output file, but here we're using \n as the delimiter, so they end up with the lines interleaved.

• <(...) is bash syntax to "pipe" a command output into another command as if it were a file (I think it uses temporary named pipes), which allows us to pipe in two at once.

• parallel reads the filenames from stdin, if ::: is not given, and runs one job per CPU if -j is not given. It will work its way down the input list in a linear manner, so the interleaving should spread the load how you want it.