BASH: Grouping files by name

Question

I have over one million files. And I have to proceed over them.
My files' directory hierarchies just like below

source=/opt/output/renamed/
target=/opt/output/combine
send=/opt/output/send/combined 

Firstly I have to loop 1000 files from source (/opt/output/renamed) directory and group them by filename.

Filename ==> ORACLE_gprtcp_201209221454_312312.log.gz

first and second colun is not important so much. But I have to group them by third field which is a timestamp.

And they have to be grouped by thirty minutes. For example,there are two files which like these

1.ORACLE_gprtcp_201209231632_987546.log.gz 
2.ORACLE_gprtcp_201209231612_123876.log.gz 
3.ORACLE_gprtcp_201209231602_987546.log.gz 
4.ORACLE_gprtcp_201209231644_987546.log.gz  
5.ORACLE_gprtcp_201209231647_987546.log.gz 
6.ORACLE_gprtcp_201209231601_987546.log.gz 

the first group's time interval must be in thirty minutes

for example, first goruped files are

2, 3 and 6 files (they are in first thirty minutes) 1,4 and 5 files(they are in last thirty minutes)

I have tried to write a script just like this

#!/bin/bash 
sourceFolder="/opt/rename/"
limitCount=10 # Limit for send file count for per process
renamed="/opt/combine"
target="/opt/send/combined/"

    for sf in ${sourceFolder}; do
    fileList=$(find ${sf} -type f -name "*.gz"  | sort -t '_' -k3 | head -${limitCount} ) 
    for filePath in $(echo "${fileList}"); do 
      fileName=$(basename ${filePath}) # source file name
      dirName=$(dirname ${filePath}) # source dir name
      #timeRef=$(echo ${fileName} | cut -d '_' -f 3 |  sed 's/\(.\{11\}\).*/\1/') 
      timeRef=$(echo ${fileName} | cut -d '_' -f 3 |  cut -c-11) 
      #time ref : ORACLE_gprtcp_20120923012703_3431593.log.gz

        if [ "${sf}" == "/opt/rename/" ]; then #####  combine
        #Move files to under /opt/combine/ to process files in the fastest way
        mv ${filePath} ${renamed}
        timeRef30="${group} | cut -d '_' -f 3 |  sed 's/\(.\{10\}\).*/\1/')"
        echo  $timeRef30
        for files in $(find ${renamed} -name "*${timeRef}*" | uniq)
        do
         fileGroup=$(echo $files | sort -t '_' -k 3 )
         first=$(echo ${fileGroup} | head -1 | cut -d '_' -f 4 | cut -d '.' -f 1)
         last=$(echo ${fileGroup}  | tail -1 | cut -d '_' -f 4 | cut -d '.' -f 1)           
          for group in ${fileGroup}
          do
           timeInt=$(echo ${group} | cut -d '_' -f 3 |  sed 's/\(.\{10\}\).*/\1/')
           zcatBaseName=$(dirname ${group}) #/opt/rename/
           zcatName=$(basename ${group}) 
           zcatUniq=$(echo ${group}| cut -d '_' -f 4 | cut -d '.' -f 1)
           newName=$(echo ${targetNAT}/ORACLE_gprtcp_${timeInt}000_${first}${last}.log)
           sleep 1
           echo "starting to zcat all files ${fileGroup}"
           zcat -f $(echo ${fileGroup}) >> "/opt/combine/ORACLE_gprtcp_${timeInt}000_${first}${last}.log"
           gzip "/opt/infolog/output/iotest/24/combine/ORACLE_gprtcp_${timeInt}000_${first}${last}.log"
           rm -f $(echo ${fileGroup})
           sleep 4                              
          done
         done
         fi 
done 
done

Is there anyone that can give me a suggestion about how can I success to group files between thirty minutes and zcat them to new file ?

Thanks in advance

Answer 1

Unfortunately I don't have time to give you a full answer but only some hints that might help.

I'd go about it just printing out the relevant files and sort them according to Unix time (found that works better than normal / human readable time):

find $PWD -type f -printf '%T@ %p\n' | sort -nb

then you could store the Unix time of the first member of a 30-min group as reference point as to when the 30 mins start, calculate difference to current file's Unix timestamp if > 1800 then create new group else add to current group. Something along those lines:

#!/bin/bash
#1800 s = 30 min
#unix time 86400s = 1 day

fileList=$(find $PWD -type f -printf '%T@ %p\n' | sort -nb)
## for debugging:
# fileList=$(find $PWD -type f -printf '%T@ %t %p\n' | sort -nb)

org_IFS=$IFS
IFS=$'\n'
group_start_time=0
for line in $fileList; do
    current_time=$(echo $line | awk '{print $1}')
    if [ $group_start_time -eq 0 ] ; then
        group_start_time=$current_time
    else
        delta=$(($current_time - $group_start_time))
        #echo $delta
        if [ $delta -lt 1801 ] ; then
            echo $line
        else
            echo -e "\nnew group:\n$line"
            group_start_time=$current_time
        fi
    fi
done
IFS=$org_IFS

from there you can just do a redirect of the file path to whatever file you want (using >>). and later run mv on that list of files to their respective directories.

I hope that helps somehow. :)

Edit: I've modified the script such that it writes the groups of log.gz files to the files in source ( your /opt/rename/ ) in the target directory (which I assumed was your /opt/send/combined/ directory). Below is the modified code:

#!/bin/bash
#1800 s = 30 min
#unix time 86400s = 1 day

sourceFolder="/opt/rename/"
target="/opt/send/combined/"

path_to_file=$target
current_file="ORACLE_gprtcp_000.log.gz"

fileList=$(find $sourceFolder -type f -name '*.log.gz' -printf '%T@ %p\n' | sort -nb)
## for debugging:
# fileList=$(find $PWD -type f -printf '%T@ %t %p\n' | sort -nb)

echo ${fileList[0]}

org_IFS=$IFS
IFS=$'\n'
group_start_time=0

for line in $fileList; do
    current_time=$(echo $line | awk '{print $1}')
    if [ $group_start_time -eq 0 ] ; then
        group_start_time=$current_time
        hr_time=$( date -d @$current_time +%F_%0k%0M )
        current_file="ORACLE_gprtcp_"$hr_time".log.gz"
    else
        delta=$(($current_time - $group_start_time))
        #echo $delta
        if [ $delta -lt 1801 ] ; then
            # just append file path to current_file
            echo $line | awk '{print $2}' >> $path_to_file"/"$current_file
            echo $line
        else
            # construct new filename based on time of the first member of the group
            hr_time=$( date -d @$current_time +%F_%0k%0M )
            current_file="ORACLE_gprtcp_"$hr_time".log.gz"

            # create file, append file path to current_file
            echo $line | awk '{print $2}' >> $path_to_file"/"$current_file
            echo -e "\nnew group:\n$line"

            group_start_time=$current_time
        fi
    fi
done

IFS=$org_IFS

Answer 2

Assuming that there is no "\n" character in file names :

find . -name '*_*_[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]_*.gz' | perl -le '
    use strict;
    use warnings;
    my %hash;
    while(<>) {
        chomp;
        my($group)=/^([^_]+_[^_]+_[0-9]{11})/;
        $group=~s/[0-2]$/00/;
        $group=~s/[3-5]$/30/;
        push @{$hash{$group}},$_;
    }
    while(my($group,$files_arr_ref)=each%hash) {
        print "processing group $group";
        for my$file (sort @{$files_arr_ref}) {
            print "processing file $file";
            # do system command calls here; for example
            # system "gzip -cd \"$file\" >> $group.txt";
        }
    }
' 

Edit: a few changes after Craig suggestions. First idea was just to use perl for arrays and hashes, finally it's more clear to do all. @ARGV is the list of path to pass to find. For example if name of script is script.pl :

script.pl ${sourceFolder}

#!/usr/bin/perl

use strict;
use warnings;
use File::Find;

my %hash;

sub wanted {
    return unless /^([^_]+_[^_]+_[0-9]{11})/;
    my$group=$1;
    $group=~s/[0-2]$/00/;
    $group=~s/[3-5]$/30/;
    push @{$hash{$group}},$_;
}

File::Find::find(\&wanted, @ARGV);

while(my($group,$files_arr_ref)=each%hash) {
    print "processing group $group\n";
    ### do system command calls here; for example
    # system "rm $group.txt";
    ### or just use perl
    # unlink $group.'.txt';
    for my$file (sort @{$files_arr_ref}) {
         print "processing file $file\n";
         ### and other system command calls here; for example
         # system "gzip -cd $file >> $group.txt";
    }
    ### and here; for example
    # system "gzip $group.txt";
}