2

So I have .lzo files in /test01/primary folder which I need to uncompress and then delete all the .lzo files. Same thing I need to do in /test02/secondary folder as well. I will have around 150.lzo files in both folders so total around 300 files.

From a command line I was running like this to uncomressed one file lzop -d file_name.lzo
What is the fastest way to uncompressed all .lzo files and then delete all .lzo files from both folders. I came up with below code.

#!/bin/bash

set -e

export PRIMARY=/test01/primary
export SECONDARY=/test02/secondary

lzop -d $PRIMARY/* & lzop -d $SECONDARY/*
find $PRIMARY $SECONDARY -name '*.lzo' -delete

May be we can decompress the .lzo files in parallel and then delete all .lzo file in both the folders simultaneously to speed up the process. Is there a better way?

1

Per the man page:

   -U, --unlink, --delete
              Delete input files after succesful compression or decompression.

so you could simply run

lzop -dU -- {"$PRIMARY","$SECONDARY"}/*.lzo

to delete each lzo file as soon as it's successfully decompressed.
lzop is single-threaded so if you want parallel processing you could use gnu parallel:

parallel lzop -dU -- ::: {"$PRIMARY","$SECONDARY"}/*.lzo
  • thanks for suggestion. I have updated the question with details. You mean something like that? And that's all? It will delete automatically? – user1950349 Oct 9 '15 at 20:31
  • Will it uncompressed and delete in parallel in both the folders simultaneously? Or it will uncompressed PRIMARY first and then delete PRIMARY .lzo and then uncompressed SECONDARY and then delete SECONDARY .lzo? – user1950349 Oct 9 '15 at 20:34
  • Yeah I was trying to understand to see that's what you meant or something else. Ok gnu parallel looks fine.. Now let's say if I have 150 files in PRIMARY and 150 files in SECONDARY, then with your parallel suggestion it will start uncompressing and deleting on all 150 files both in PRIMARY and SECONDARY folder simultaneously? Or can we control like do 40 files at a time and then move to next 40? I am trying to see whether there will be any issue if it starts uncompressing and deleting on all 150 files both in PRIMARY and SECONDARY at same time. – user1950349 Oct 9 '15 at 20:46
  • 1
    @user1950349 - no, parallel processing doesn't mean all your files will be processed simultaneously but as many as possible. Sure, you can fine tune it, see this post by the author and read the manual & tutorial (linked in the same post). – don_crissti Oct 9 '15 at 20:56
1

GNU Parallel spawns the jobs in the order they are given to GNU Parallel. So if you first list the $PRIMARY files and then the $SECONDARY files, then the $PRIMARY files will be processed first.

Unless --shuf is given. --shuf shuffles the arguments before running them. You will still not get one $PRIMARY file, then a $SECONDARY file every time, but at least the order will be random.

  • Thanks Ole for suggestion. I have opened another question here in which I am trying to paralelly uncompress and delete .lzo files both in PRIMARY and SECONDARY folders simultaneously but not able to make it work so far. – user1950349 Oct 10 '15 at 18:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.