2

I have a directory that contains many (>5000) folders:

folder1
folder2
folder3
...

All of these folders have many subdirectories. In a specific subdirectory of each folder, there might be a tar.gz archive. If a folder contains a tar.gz archive, it will only be one, and it will be in a specific subdirectory.

For example:

folder1/foo/baz.tar.gz
folder2/bar/qux.tar.gz
folder3 [no tar.gz file in this folder]
...

I need to write a bash script as to accomplish the following:

  • I want to loop through each folder, find the tar.gz archive if it exists, and untar its contents in a different directory (which is the same for all found archives).
  • Once every tar.gz archive is found, I need to also move more files in the same directory as the tar.gz archive, maybe needing to store the path of the archive.

I am able to list all of the archives with:

find . -name "*tar.gz"

I am wondering whether manipulating the obtained command is the most optimal solution, or looping through each directory would be better...

What would be the most optimal approach? How should it be carried out?

  • 1) you wrote "I need to store the path" - how every path should be stored? 2) that different directory - is it common for all extracted files that to be moved? – RomanPerekhrest Feb 15 '18 at 11:06
  • Question updated. And yes, the different directory is the same. – erik Feb 15 '18 at 11:09
2

You can perform basically any single operation or list of operations within find's -exec option, so why not untar each archive directly within find -exec? For complex commands, it's common to use the -exec feature to invoke a shell, and to use the shell command's -c option to pass it the actual commands that you want to run. For example (in practice there is a much simpler way to perform this actual example, but it is for the purpose of showing the idea):

-exec sh -c 'mv "$1" "~/$1"' sh {} ';'

This would start a shell for each file found, and have it move the file to your $HOME directory. Note that the {} serves to pass the found file name as a shell positional parameter, ie. $1, so within the shell command, $1 is being used, not {}. In your case, a solution of this type would be something like:

-exec sh -c 'tar xvf "$1" -C "$(dirname $1)"' sh {} ';' 

The idea is that this idiom brings all of your favorite shell's features within the find command. (And, yes, you could use bash or zsh instead of sh, just be aware that sh loads much faster and when you're processing many files, that could add up).

If this is something that you do repeatedly, or you expect in advance that the operation will take a long time, AND you have a multi-core cpu, then it may be beneficial for you to consider a second option - piping your list of files ro GNU parallel and have it perform the untar operation simultaneously on all cores. Try this for starters:

find . -name "*tar.gz" -type f -print0 |
  parallel -0 tar xvf {} -C {//}

As an additional efficiency, the above answer no longer needs to use the external dirname command because GNU parallel can do it more efficiently itself. That's what the {//} is doing.

WARNING: I'm no expert on parallel and am offering that option without actual experience using it, so if others can chime in on whether that's the correct way, that would be great.

0

You could use the find command to pass the tarball paths to a script which could untar them (I haven't tested this):

$ cat script
#!/bin/bash --
tarball="${1}"
dir="$(dirname ${tarball})"
tar xvf "${tarball}" -C "${dir}"

Then call the script with find:

$ find . -type f -name '*.tar.gz' -exec ./script "{}" \;

Or in one find command (quickly tested):

find . -type f -name '*.tar.gz' -exec sh -c 'dir="$(dirname ''"{}"'')"; tar xvf "{}" -C "${dir}"' \;

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.