How to gzip and copy files keeping its directory structure?

Question

How can I gzip and copy files to another folder keeping its directory structure with one command in Linux?

For example, I have:

/dir1
/dir1/file1.fit
/dir1/file2.fit
/dir1/file3.fit
/dir1/dir2/file1.fit
/dir1/dir2/file2.fit
/dir1/dir2/file3.fit

After I use a command (Lets we say I copy /dir1 to /another_dir), I want to get:

/another_dir/dir1
/another_dir/dir1/file1.fit.gz
/another_dir/dir1/file2.fit.gz
/another_dir/dir1/file3.fit.gz
/another_dir/dir1/dir2/file1.fit.gz
/another_dir/dir1/dir2/file2.fit.gz
/another_dir/dir1/dir2/file3.fit.gz

Here /another_dir is actually another hard drive. Since no enough space in this target drive (it is a data of 2TB!), please do not suggest me to copy the files first and then gzip all (or vice-versa). Similarly, the gz files should not remain in the source folder after the operation.

Answer 1

Assuming you're in the root folder where are all directories for compression (in your case /), you can use find along with xargs command, e.g.

find dir1/ -name "*.fit" -print0 | xargs -i% -r0 sh -c 'mkdir -vp "$(dirname "/another_dir/%")" && gzip -vc "%" | tee "/another_dir/%".gz > /dev/null && rm -v "%"'

^{Note: You can also replace | tee "/another_dir/%".gz > /dev/null with > "/another_dir/%".gz.}

This will find all .fit files in dir1/ and pass them to xargs command for parsing where % is replaced with each of your file.

The xargs command will:

• create the empty folder (mkdir) with its parents (-p) as a placeholder,
• compress given file (%) into standard output (-c) and redirect compressed output to tee,
• tee will save the compressed input into .gz file (since tee by default prints the input to the terminal screen, sending it to /dev/null will suppress it, but it'll still save the content into the given file).

After successful compression, remove the original (rm). You can always remove that part, in order to remove them manually after verifying your compressed files.

It is important that you're in relative folder to your dir1/, so all paths returned by find are relative to the current folder, so you don't have to convert absolute paths into relative (this still can be done by realpath, e.g. realpath --relative-to=$absolute $current, but it will just overcomplicate the above command).

On macOS, to use -r argument for xargs, you need to install GNU xargs (brew install xargs) and use gxargs command instead. Similar on other BSD systems.

Related question: gzip several files in different directories and copy to new directory.

Answer 2

I am not aware of single command to do it, but over on ServerFault is example of simple script that implements the same. Note that it does not preserve file permissions or access/modification times, and may have issues if file names contain spaces or similar.