I wish to know a command I can use to grab and archive all files for a year at a time. I am working on a directory that has 72000 files, with files dated from 2007 to current. And I wish to make that more manageable by tar-bziping older files in one-year groups. Thus, all files from 1-1-2007 to 12-31-2007, etc. Thanks for the help!

  • What shell do you have at your disposal? Does the directory have subdirectories, and if so should the files be archived individually or should each directory be archived as a block? – Gilles 'SO- stop being evil' Jul 9 '15 at 23:49
  • thx. It is on a web server host, running linux (centos I believe). My usual access is SSH terminal via PuTTY 064 . No subdirectories. It is just one directory itself with that unmanageable 72,000 files in it. Collecting data there since 2007. Each group of one year can be in a 'block'. thus I was thinking something that could pipe to a tar-bzip command. – Bruce Jul 9 '15 at 23:51
  • Are the file names reflective of date, or is it going by creation/modification date? – Drav Sloan Jul 9 '15 at 23:52
  • file names are relatively randomly assigned my some black-box process. thus no way to tell by looking at file name. Operation will thus have to be dependent on the actual file date – Bruce Jul 9 '15 at 23:53
  • example file name: Abcd510x510x1Cdeceffc000-40x40x3cff00C000\@66,344.gif And the only difference with other files is the final five or six digits. I ma not sure why they are named such, with such long fixed part, and with the "\@" and only the last 5 or 6 digits being unique.... but anyway we need to clear them out, saving to an archive for safety. – Bruce Jul 9 '15 at 23:56

You can use the find command to select files based on their modification time.

cd /path/to/directory
find -type f -newermt 2007-01-01 ! -newermt 2008-01-01

Note that this selects files created in 2007 in your time zone, and files timestamped exactly at midnight on year change are grouped with the preceding year.

If your version of CentOS has an old version of find that doesn't support -newermt, you can use -newer with fencepost files instead.

touch -d 2007-01-01 /tmp/2007-01-01
touch -d 2008-01-01 /tmp/2008-01-01
find -type f -newer /tmp/2007-01-01 ! -newer /tmp/2008-01-01

Assuming that the file names don't contain newlines, you can pipe the resulting listing as input for tar with the -T option (-T - means to read file names from standard input).

cd /path/to/directory
find -type f -newermt 2007-01-01 ! -newermt 2008-01-01 |
tar -cjf /elsewhere/2007.tar.bz2 -T -

(Consider tar -cJf /elsewhere/2007.tar.xz instead, xz is faster to decompress and compresses better, at the expense of requiring more memory when compressing.)

  • Thanks Gilles, re Centos it is ` rpm -q centos-release ==> centos-release-6-6.el6.centos.12.1.x86_64` (I cant figure out the trick to making line breaks in these comments) – Bruce Jul 10 '15 at 1:23
  • I have been experimenting with find and listing, to verify that the selected files are correct. Seems the only way I've been able to do that is to use find fencepost style for the year, which results in a file list with 5527 lines but only file name and no date info. Then tried similar find but pipe to ls as: ` find ./ -newer start.dat \! -newer stop.dat -exec ls -hltr > 2007list3.txt {} \; ` which works, has date and file name details, but results are not date sorted. Yet it does show the same 5527 lines, and they all have 2007 dates. So the pipe to tar -cJf looks like it should work. – Bruce Jul 10 '15 at 1:29
  • My one sub question before I do that, is there a way to verify my selection by creating a text file with my selected one year results, with date detail info (which I have from my above comment command), but have this results text file in date sequence? – Bruce Jul 10 '15 at 1:30
  • I'm giving it a go. I will post back after my work (which may be a day or so given time constraints) to let youall know my steps. Assuming anyone ever encounters a similar situation... I know I will, given the strangeness of the site I have inherited maintaining ;-) . . . Thanks. – Bruce Jul 10 '15 at 4:45
  • 1
    @Bruce If you add -ls at the end of the find … command, you'll get a file listing with file dates. If you want them sorted, add -exec ls -l --full-time {} + | sort -k 6,7 (the simpler |xargs ls -lt may or may not work: it only sorts correctly if there's a single call to ls, which may not be the case because the total length of the file names may exceed the command line length limit). – Gilles 'SO- stop being evil' Jul 10 '15 at 7:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.