Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I have a large data directory (20-30Gb) on my Ubuntu 10.10 desktop machine that consists of many raw data files, processed data files, and assorted scripts, tables, figures etc. generated from the processed data. The data directory has accumulated over many years, and is very poorly structured - "one day" I will sort it out, but there are always more important things to do.

I'm now switching to an online backup service, and in order to reduce both the time taken to backup and the online storage required i'd like to split out the raw data, which takes up a lot of space but is easily replaced as is already archived elsewhere, while retaining its general position in the directory structure. In other words, I want to go from something like:

... etc.


... etc.


... etc.

So the raw data files swap from /data to /raw_data but otherwise retain their position in the directory structure, while the processed data and associated files remain in the same place. The overall file structure is much more complex and disordered than this, but the saving grace is that all raw data can be identified by filetype (mainly .fits and .sdf).

I'm sure this is trivial with the right combination of commands and/or a few lines of bash script, but my command-line knowledge is limited to the basics and I would rather ask than risk messing it up :)

And, as an aside, is there a simple way to look for duplicates in the raw data - will have identical filename + size, not necessarily timestamp which gets reset as the data is downloaded from the archive, although to be completely sure I need to pipe each duplicate candidate through dfits and grep the timestamp in the fits header.

share|improve this question
Thanks - .fits and .sdf are right, I was just trying to keep the description of what I was trying to do as simple as possible – strmqm Aug 2 '11 at 8:37

One way to do this would be to use rsync with some specially crafted include/exclude rules and the option to remove the sources files after syncing like this:

rsync -av --include "*/" --include='*.fits' --include='*.sdf' \
    --exclude='*' --remove-source-files /data/ /raw_data/

If you wanted to move step by step ina loop so that you could potentially include other actions, you will need a script that does something like this:


find "$DIR1" -type f \( -iname '*.fits' -or -iname '*.sdf' \) -print0 |
    while read -d $'\0' file; do
        mkdir -p "$DIR2/$(basename "$file")"
        mv "$file" "$DIR2/$(basename "$file")"
share|improve this answer
@strmqm The rsync solution is better than the one I suggested because it's more general. It will work for any directory structure whereas mine assumes and hardcodes a fixed 1 level layout like in the example. You can skip the find ... -delete and do it all in one step with rsync's --remove-source-files option. – jw013 Aug 2 '11 at 12:48
@jw013: I'd forgotten about --remove-source-files, thanks. – Caleb Aug 2 '11 at 13:24

There are a bunch of file copying tools that allow constructing a target directory name with sufficiently flexible rules (zcp, rsync, pax, …). Unfortunately, few of them allow both moving (as opposed to copying) and creating target directories on demand. So I'll show some ways do it in two passes: first create all potentially necessary target directories, then perform the move.

Perl rename

The Perl rename program shipped by Debian and Ubuntu can create the target directory when needed, if you write the necessary bit of Perl.

shopt -s globstar       # make **/ traverse directories recursively (requires bash 4)
rename 'BEGIN {use File::Path}
        m!(.*)/!; mkpath($1)' /data/**/*.raw

In zsh, omit the shopt -s globstar line; ** means recursive traversal by default. In shells other than bash and zsh, you need to use find for recursive traversals (see examples below). Don't worry about all this if you have a single level of directories.

Creating the target directories

In zsh (explanation: the / glob qualifier means to match only directories, and the e glob qualifier applies the transformation given afterwards to each name):

mkdir /data/**/*(/e\''REPLY=${REPLY/data/raw_data}'\')

With other shells:

find /data -type d \
     -exec sh -c 'for d; do mkdir "/raw_data${d#/data}"; done' _ {} +

If you only have one level of subdirectories, it's a lot simpler:

for d in /data/*/; do mkdir "/raw_data${d#/data}"; done

Moving the files (zsh)

autoload zmv
zmv -Q '/data/(**/)(*.raw)(.)' '/raw_data/$1$2'

Moving the files (portable)

find /data -name '*.raw' \
     -exec sh -c 'for x; do mv "$x" "/raw_data${x#/data}"; done' _ {} +
share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.