I have a bash extraction function that I am trying to parallelize. Its job is to find and extract nested archives. Ideally, I want the if evaluation and all of its actions to be sent to the background. The catch is that the action items of the if evaluation need to be done in order, so I can't just add an "&". to the commands in the if. Is there a way to encapsulate the entire if evaluation into a single background job and have the commands execute in order?

Here is the current working function:

extract () {
for ext in zip rar tar.gz tar.bz2 tbz tgz 7z tar; do
    while [ "`find . -type f -iname "*.$ext" | wc -l`" -gt 0 ]; do
        for z in `find . -type f -iname "*.$ext"`; do
            if [ ! -d "`echo "$z" | rev | cut -c$(expr ${#ext} + 2)- | rev`" ]; then
                echo "Extracting `basename "$z"` ..."
                mkdir -p `echo "$z" | rev | cut -c$(expr ${#ext} + 2)- | rev`
                if [[ "$z" =~ ^.*\.7z$ ]]; then 7z x "$z" -o"`echo "$z" | rev | cut -c$(expr ${#ext} + 2)- | rev`" > /dev/null
                elif [[ "$z" =~ ^.*\.zip$ ]]; then unzip -uoLq "$z" -d `echo "$z" | rev | cut -c$(expr ${#ext} + 2)- | rev` 2>&1 | grep -ive warning
                elif [[ "$z" =~ ^.*\.tar\.xz$ ]] || [[ "$z" =~ ^.*\.tar\.gz$ ]] || [[ "$z" =~ ^.*\.tar\.bz2$ ]] || [[ "$z" =~ ^.*\.tgz$ ]] || [[ "$z" =~ ^.*\.tbz$ ]] || [[ "$z" =~ ^.*\.tar$ ]] ; then tar -xaf "$z" -C `echo "$z" | rev | cut -c$(expr ${#ext} + 2)- | rev` 
                elif [[ "$z" =~ ^.*\.rar$ ]]; then unrar x -y -o+ "$z" `echo "$z" | rev | cut -c$(expr ${#ext} + 2)- | rev`
                rm -f "$z"
            else echo "Omitting `basename "$z"`, directory with that name already exists."; rm -f "$z"

Also, I am curious if there is any way to perform the extraction without deleting the source archives. I currently do this to prevent an endless loop. For now, the function is reliable enough to not loose any data, but I would like to avoid deleting anything for safety.

  • You should at least store strings that you reuse (for instance, mydir=$(echo "$z" | rev | cut -c$(expr ${#ext} + 2)- | rev) would save time and make it easier to read). Others have already pointed out that you can find everything by a simple regular expression, I'd just like to add that ${z%.*} cuts off the extension (.tar.gz may need special handling, for instance, if you know it's in this form, use ${z%.tar.gz}). – orion Aug 5 '14 at 19:11
  • Unless you have extremely high disk bandwidth, parallelizing an I/O-bound operation would only make it slower. – Gilles Aug 5 '14 at 22:43

Why do you run the same find command multiple times, twice for each extension? You can just generate a single find command that will only traverse the directory tree once:

find . -regextype posix-egrep -iregex $EXT_REGEX

Now, you don't need nested loops at all, and you certainly don't need the while that caused your infinite loop problem.

Secondly, your code is broken for filenames with spaces in. You can fix that by adding


(to stop for z in ... splitting the output on whitespace).

Finally, if you stick an & at the end of each of your if/elif branches, they'll run in parallel.

BTW, what are all the echo "$z" | rev supposed to accomplish? Were you somehow getting multi-line file names?

  • Actually, the parent script sets IFS=$'\n', so the spaces think hasn't been a problem, I probably should have included that. The echo and rev logic cuts the extension off the archive filename and creates a directory named after it for the contents to be extracted into. Because the destination directory needs to be created for the contents to be extracted into, I need the commands in the if to run in order, hence why I cannot just add an & to them. Or did I miss something? – MrDrMcCoy Aug 5 '14 at 17:24
  • As for eliminating the while loop, I don't see how this can be done while preserving its original function, as I am expecting to re-run the find command to discover and extract new zip files that were nested in the source. – MrDrMcCoy Aug 5 '14 at 17:30
  • Thanks for the reminder on using regex directly in find. That will at least allow me to eliminate the first loop. – MrDrMcCoy Aug 5 '14 at 17:31
  • Just realized something: I'm not sure sure I can eliminate the first for loop using regex in find, as the bits that create the destination directory depend on that to do their naming. I expect there is a more clever way to do this, but I'm not quite sure how. – MrDrMcCoy Aug 5 '14 at 17:39
  • @nakedhitman The reason the bits that depend on that depend on anything is because everything depends on everything! I really don't like saying this, but your script is really bad. And it is that - it is a script, not a function. A function performs a function - your thing does 1000 per line. You need to do... for ext in tar zip etc ; do $ext & ; done where each $ext is also the name of a function, and starts with a check like set -- *$myext ; [ -e "$1" ] || return. Your thing... is a monster. – mikeserv Aug 6 '14 at 4:14

Thanks to advice from @Useless and @Orion, I now have beaten the function into submission. It now spawns all the extractions in the background, no longer deletes source files, and is over 25% faster for me than its predecessor. @Gilles noted that the parallelization is not for everyone, as this is rather storage-expensive. It was better for me though, and if you find that you can use this script, I will provide it below:

extract () { # Extracts all archives and any nested archives of specified directory into a new child directory named after the archive.
trap "rm $skipfiles ; exit" SIGINT SIGTERM
shopt -s nocasematch # Allows case-insensitive regex matching
echo -e "\n=====Extracting files====="
skipfiles=`mktemp` ; echo -e '\e' > $skipfiles # This creates a temp file to keep track of files that are already processed. Because of how it is read by grep, it needs an initial search string to omit from the found files. I opted for a literal escape character because who would name a file with that?
while [ "`find "$1/" -type f -regextype posix-egrep -iregex '^.*\.(tar\.gz|tar\.bz2|tar\.xz|tar|tbz|tgz|zip|rar|7z)$' | grep -ivf $skipfiles | wc -l`" -gt 0 ]; do #The while loop ensures that nested archives will be extracted. Its find operation needs to be separate from the find for the for loop below because it will change.
    for z in `find "$1/" -type f -regextype posix-egrep -iregex '^.*\.(tar\.gz|tar\.bz2|tar\.xz|tar|tbz|tgz|zip|rar|7z)$' | grep -ivf $skipfiles`; do
        destdir=`echo "$z" | sed -r 's/\.(tar\.gz|tar\.bz2|tar\.xz|tar|tbz|tgz|zip|rar|7z)$//'` # This removes the extension from the source filename so we can extract the files to a new directory named after the archive.
        if [ ! -d "$destdir" ]; then
            echo "Extracting `basename $z` into `basename $destdir` ..."
            mkdir -p "$destdir"
            if [[ "$z" =~ ^.*\.7z$ ]]; then 7z x "$z" -o"$destdir" > /dev/null & 
            elif [[ "$z" =~ ^.*\.rar$ ]]; then unrar x -y -o+ "$z" "$destdir" &
            elif [[ "$z" =~ ^.*\.zip$ ]]; then unzip -uoLq "$z" -d "$destdir" 2>/dev/null &
            elif [[ "$z" =~ ^.*\.(tar\.gz|tar\.bz2|tar\.xz|tar|tbz|tgz)$ ]] ; then tar -xaf "$z" -C "$destdir" &
            echo `basename "$z"` >> $skipfiles # This adds the name of the extracted file to the omission list for the next pass.
        else echo "Omitting `basename $z`, directory with that name already exists."; echo `basename "$z"` >> $skipfiles # Same as last line
    wait # This will wait for all files in this pass to finish extracting before the next one.
rm "$skipfiles" # Removes temporary file

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.