3

I have few files in a directory, I need to pick the oldest one for processing.

For Exp: I have below files in /tmp directory:

Sample_0000237826-001_xyz.dat
Sample_0000437564-001_xyz.dat
Sample_0000237826-002_xyz.dat
Sample_0000137294-003_xyz.dat

So my script should first sort the files on the basis of 1st set of 10 digit numbers in the file name I.e. nnnnnnnnnn, and in case for the first set of 10 digit number, if it gets multiple entries, then it should sort on the next set of 3 digit numbers I.e. nnn , and it should pick the oldest one.

Once I have picked up that file for processing, I will delete it from the directory. So it won't appear during the next run.

Hence for 1st run, I should pick the file: Sample_0000137294-003_xyz.dat

For 2nd run, I should pick: Sample_0000237826-001_xyz.dat

For 3rd run, I should pick: Sample_0000237826-002_xyz.dat

and so on.

Guys forgot to add one more thing, these files will be copied from remote server to my local server one by one as mentioned above, hence for copying I am using "rsync". Hence, please let me know if the below provided solution will work with rsync command as well or not.

3
  • Welcome to Unix & Linux. There is no need to have your name twice under your post, and you should read at least the help tour. Please provide us with the awk script you have so far, where it goes wrong and what the error message is (if applicable).
    – Anthon
    Mar 3, 2015 at 13:50
  • Do you need to parse the file names or can you use the file's creation date?
    – terdon
    Mar 3, 2015 at 13:53
  • 1
    If the prefix (Sample_) is the same for every file, globbing should sort in that order (so for f in *.dat would do). Mar 3, 2015 at 14:00

5 Answers 5

3

With zsh:

extract_numbers() REPLY=${(SM)REPLY##<->-<->}
for file (*<->-<->*.dat(no+extract_numbers)) {
  do-what-you-will-with $file
}

We define a sorting function (extract_numbers) that returns the Substring of the file name that Matches <->-<-> (<-> matching any positive decimal number).

We use that function for numeric ordering of the *<->.<->*.dat glob.

1

assuming

  • all files do not begin with Sample
  • numeric field are sperated by _
  • you can't relay on file time

I would use

sort -t_ -k2n

which yield for the four files given in sample

Sample_0000137294-003_xyz.dat
Sample_0000237826-001_xyz.dat
Sample_0000237826-002_xyz.dat
Sample_0000437564-001_xyz.dat
1
  • forgot to add one more thing, these files will be copied from remote server to my local server one by one as mentioned above, hence for copying I am using "rsync". Hence, please let me know if the below provided solution will work with rsync command as well or not.
    – Niraj
    Mar 3, 2015 at 15:44
1
#!/bin/bash

ls /tmp/*dat > filelist

while read fn
do 
    #do something with /tmp/$fn
done < filelist
3
  • 1
    ls will already sort the filelist for you, a separate sort invocation is not necessary. I'd also do while read fn; do whatever; done < filelist instead of continually editing the list.
    – wurtel
    Mar 3, 2015 at 13:58
  • Yes, good comments. I edited my answer.
    – user236012
    Mar 3, 2015 at 14:02
  • 1
    You don't really need to use a file there (you don't even need ls). Just use a pipe: ls | while read fn; ... done. Mar 3, 2015 at 16:47
1

You can just do a simple sort. Assuming your file names contain no whitespace or other strange characters, this command will always give you the oldest file:

find . -name '*dat' | sort -r | tail -n1

If you need to deal with strange characters, use

find . -name '*dat' -print0 | sort -zr | tr '\0' '\n' | tail -n1

So, you could do something along the lines of

file=$(find . -name '*dat' -print0 | sort -zr | tr '\0' '\n' | tail -n1)
while [ -e "$file" ]; do
    if [ -e "$file" ]; then 
        echo "$file"; ## Process the file here
        rm "$file"; 
    fi;
file=$(find . -name '*dat' -print0 | sort -zr | tr '\0' '\n' | tail -n1)
done

If you really need to sort on only the 1st set of 10 digits, you could do something like (assuming what you want to sort on is between the first two _):

find . -name '*dat' -print0 | sort -zr -t "_" -k 2 | tr '\0' '\n' | tail -n1

Also note that in the case you show where all files start with the same characters and only the numbers change, you don't need to sort at all:

for file in *dat; do
   echo "$file" ## process the file here
   rm "$file"
done
2
  • forgot to add one more thing, these files will be copied from remote server to my local server one by one as mentioned above, hence for copying I am using "rsync". Hence, please let me know if the below provided solution will work with rsync command as well or not.
    – Niraj
    Mar 3, 2015 at 15:44
  • @Niraj yes, in all the examples above, just replace echo "$file" with whatever you actually want to do.
    – terdon
    Mar 3, 2015 at 17:05
0

This command will give the filename of only one file sorted by the numeric fields, assuming the pattern is Sample_nnnnnnnnnn-ppp_xyz.dat:

ls Sample_$(ls -l *.dat | awk -F ' |_' {'print $10'} |sort -n| head -1)*.dat
2
  • forgot to add one more thing, these files will be copied from remote server to my local server one by one as mentioned above, hence for copying I am using "rsync". Hence, please let me know if the below provided solution will work with rsync command as well or not.
    – Niraj
    Mar 3, 2015 at 15:45
  • I think it will.
    – jcbermu
    Mar 3, 2015 at 15:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.