0

So I have .lzo files in /test01/primary folder which I need to uncompress and then delete all the .lzo files. Same thing I need to do in /test02/secondary folder as well. I will have around 150 .lzo files in both folders so total around 300 .lzo files.

From a command line I was running like this to uncomressed one file lzop -d file_name.lzo.

What is the fastest way to uncompressed all .lzo files and then delete all .lzo files from both folders simultaneously. Below is the code I have:

#!/bin/bash

set -e

export PRIMARY=/test01/primary
export SECONDARY=/test02/secondary

parallel lzop -dU -- ::: {"$PRIMARY","$SECONDARY"}/*.lzo

I want to uncompress and delete .lzo files parallelly in both PRIMARY and SECONDARY folder simultaneously. With my above code, it does in PRIMARY first and then in SECONDARY folder. How can I achieve parallellism both in PRIMARY and SECONDARY simultaneously?

Also does it uncompress all the files and then delete later on or uncompress one file and then delete that file and then move to next one?

I tried with this but it doesn't work. It just works on first 40 files and after that it doesn't work at all.

#!/bin/bash

set -e

export PRIMARY=/test01/primary
export SECONDARY=/test02/secondary

parallel -j 40 lzop -dU -- ::: "$PRIMARY"/*.lzo &
parallel -j 40 lzop -dU -- ::: "$SECONDARY"/*.lzo &
wait
  • It deletes each lzo file as soon as it has successfully decompressed it, it's in the manual excerpt in my answer to your other question. – don_crissti Oct 9 '15 at 23:49
  • aahh sure.. understood that part.. – user1950349 Oct 9 '15 at 23:58
  • Why do you have 80 lzop processes running all at the same time? – Ryan Babchishin Oct 10 '15 at 17:09
  • I just came up with some number so started using 40. I want to parallelly uncompressed and delete .lzo files both in PRIMARY and SECONDARY folder simultaneously so that's why I came up with that example. – user1950349 Oct 10 '15 at 17:33
  • Write a script, no shebang, just a file e.g. p_script with two lines: parallel lzop -dU -- ::: path1/*.lzo and parallel lzop -dU -- ::: path2/*.lzo make it executable and then run parallel ::: ./p_script (you should further optimize with -j...) – don_crissti Oct 10 '15 at 17:40
0

If I understand correctly, you want to parallelize both directories, but balance the load between (presumably) two storage devices?

The easiest solution is probably to interleave the file names in the input to parallel:

paste -d'\n' <(ls "$PRIMARY"/*.lzo) <(ls "$SECONDARY"/*.lzo) | parallel lzop -dU --
  • The paste command takes one line from each file and normally puts them side-by-side in an output file, but here we're using \n as the delimiter, so they end up with the lines interleaved.

  • <(...) is bash syntax to "pipe" a command output into another command as if it were a file (I think it uses temporary named pipes), which allows us to pipe in two at once.

  • parallel reads the filenames from stdin, if ::: is not given, and runs one job per CPU if -j is not given. It will work its way down the input list in a linear manner, so the interleaving should spread the load how you want it.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.