I have a large data directory (20-30Gb) on my Ubuntu 10.10 desktop machine that consists of many raw data files, processed data files, and assorted scripts, tables, figures etc. generated from the processed data. The data directory has accumulated over many years, and is very poorly structured - "one day" I will sort it out, but there are always more important things to do.
I'm now switching to an online backup service, and in order to reduce both the time taken to backup and the online storage required i'd like to split out the raw data, which takes up a lot of space but is easily replaced as is already archived elsewhere, while retaining its general position in the directory structure. In other words, I want to go from something like:
/data/A/raw1.data /data/A/raw2.data /data/A/raw3.data /data/A/processed.txt /data/A/figure.eps /data/A/plot.gnu /data/B/raw4.data /data/B/processed.txt ... etc.
/data/A/processed.txt /data/A/figure.eps /data/A/plot.gnu /data/B/processed.txt ... etc.
/raw_data/A/raw1.data /raw_data/A/raw2.data /raw_data/A/raw3.data /raw_data/B/raw4.data ... etc.
So the raw data files swap from /data to /raw_data but otherwise retain their position in the directory structure, while the processed data and associated files remain in the same place. The overall file structure is much more complex and disordered than this, but the saving grace is that all raw data can be identified by filetype (mainly .fits and .sdf).
I'm sure this is trivial with the right combination of commands and/or a few lines of bash script, but my command-line knowledge is limited to the basics and I would rather ask than risk messing it up :)
And, as an aside, is there a simple way to look for duplicates in the raw data - will have identical filename + size, not necessarily timestamp which gets reset as the data is downloaded from the archive, although to be completely sure I need to pipe each duplicate candidate through dfits and grep the timestamp in the fits header.