Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I have over one million files. And I have to proceed over them.
My files' directory hierarchies just like below

source=/opt/output/renamed/
target=/opt/output/combine
send=/opt/output/send/combined 

Firstly I have to loop 1000 files from source (/opt/output/renamed) directory and group them by filename.

Filename ==> ORACLE_gprtcp_201209221454_312312.log.gz

first and second colun is not important so much. But I have to group them by third field which is a timestamp.

And they have to be grouped by thirty minutes. For example,there are two files which like these

1.ORACLE_gprtcp_201209231632_987546.log.gz 
2.ORACLE_gprtcp_201209231612_123876.log.gz 
3.ORACLE_gprtcp_201209231602_987546.log.gz 
4.ORACLE_gprtcp_201209231644_987546.log.gz  
5.ORACLE_gprtcp_201209231647_987546.log.gz 
6.ORACLE_gprtcp_201209231601_987546.log.gz 

the first group's time interval must be in thirty minutes

for example, first goruped files are

2, 3 and 6 files (they are in first thirty minutes) 1,4 and 5 files(they are in last thirty minutes)

I have tried to write a script just like this

#!/bin/bash 
sourceFolder="/opt/rename/"
limitCount=10 # Limit for send file count for per process
renamed="/opt/combine"
target="/opt/send/combined/"

    for sf in ${sourceFolder}; do
    fileList=$(find ${sf} -type f -name "*.gz"  | sort -t '_' -k3 | head -${limitCount} ) 
    for filePath in $(echo "${fileList}"); do 
      fileName=$(basename ${filePath}) # source file name
      dirName=$(dirname ${filePath}) # source dir name
      #timeRef=$(echo ${fileName} | cut -d '_' -f 3 |  sed 's/\(.\{11\}\).*/\1/') 
      timeRef=$(echo ${fileName} | cut -d '_' -f 3 |  cut -c-11) 
      #time ref : ORACLE_gprtcp_20120923012703_3431593.log.gz

        if [ "${sf}" == "/opt/rename/" ]; then #####  combine
        #Move files to under /opt/combine/ to process files in the fastest way
        mv ${filePath} ${renamed}
        timeRef30="${group} | cut -d '_' -f 3 |  sed 's/\(.\{10\}\).*/\1/')"
        echo  $timeRef30
        for files in $(find ${renamed} -name "*${timeRef}*" | uniq)
        do
         fileGroup=$(echo $files | sort -t '_' -k 3 )
         first=$(echo ${fileGroup} | head -1 | cut -d '_' -f 4 | cut -d '.' -f 1)
         last=$(echo ${fileGroup}  | tail -1 | cut -d '_' -f 4 | cut -d '.' -f 1)           
          for group in ${fileGroup}
          do
           timeInt=$(echo ${group} | cut -d '_' -f 3 |  sed 's/\(.\{10\}\).*/\1/')
           zcatBaseName=$(dirname ${group}) #/opt/rename/
           zcatName=$(basename ${group}) 
           zcatUniq=$(echo ${group}| cut -d '_' -f 4 | cut -d '.' -f 1)
           newName=$(echo ${targetNAT}/ORACLE_gprtcp_${timeInt}000_${first}${last}.log)
           sleep 1
           echo "starting to zcat all files ${fileGroup}"
           zcat -f $(echo ${fileGroup}) >> "/opt/combine/ORACLE_gprtcp_${timeInt}000_${first}${last}.log"
           gzip "/opt/infolog/output/iotest/24/combine/ORACLE_gprtcp_${timeInt}000_${first}${last}.log"
           rm -f $(echo ${fileGroup})
           sleep 4                              
          done
         done
         fi 
done 
done

Is there anyone that can give me a suggestion about how can I success to group files between thirty minutes and zcat them to new file ?

Thanks in advance

share|improve this question
2  
I think you've left out part of the script, or you're using ${group} in timeRef30="${group}..." before you've defined it. Anyway, you probably want to extract the timestamp from the filename (cut -d_ -f3 is fine), and then get the last two digits of that, will be 01-30 or 31-59. use that to decide whether to push the filename into a $first array or $last array. hope that helps, it's 00:45 am here now, too late for me to think too hard or long about it. –  cas Oct 2 '12 at 14:43
2  
Can you explain in human language what you are trying to do? It's hard to tell what you want from a non-functional script. My best guess is you want to take all the ORACLE_??????_YYYYmmddHHMM_######.log.gz files from the same half hour and zcat those files (in time order, I presume?) to a single file with roughly the same name, but with the time rounded down to a whole half hour and with some extra stuff on the end (you'll have to describe this as well). –  jw013 Oct 2 '12 at 14:56
add comment

2 Answers 2

Unfortunately I don't have time to give you a full answer but only some hints that might help.

I'd go about it just printing out the relevant files and sort them according to Unix time (found that works better than normal / human readable time):

find $PWD -type f -printf '%T@ %p\n' | sort -nb

then you could store the Unix time of the first member of a 30-min group as reference point as to when the 30 mins start, calculate difference to current file's Unix timestamp if > 1800 then create new group else add to current group. Something along those lines:

#!/bin/bash
#1800 s = 30 min
#unix time 86400s = 1 day

fileList=$(find $PWD -type f -printf '%T@ %p\n' | sort -nb)
## for debugging:
# fileList=$(find $PWD -type f -printf '%T@ %t %p\n' | sort -nb)

org_IFS=$IFS
IFS=$'\n'
group_start_time=0
for line in $fileList; do
    current_time=$(echo $line | awk '{print $1}')
    if [ $group_start_time -eq 0 ] ; then
        group_start_time=$current_time
    else
        delta=$(($current_time - $group_start_time))
        #echo $delta
        if [ $delta -lt 1801 ] ; then
            echo $line
        else
            echo -e "\nnew group:\n$line"
            group_start_time=$current_time
        fi
    fi
done
IFS=$org_IFS

from there you can just do a redirect of the file path to whatever file you want (using >>). and later run mv on that list of files to their respective directories.

I hope that helps somehow. :)

Edit: I've modified the script such that it writes the groups of log.gz files to the files in source ( your /opt/rename/ ) in the target directory (which I assumed was your /opt/send/combined/ directory). Below is the modified code:

#!/bin/bash
#1800 s = 30 min
#unix time 86400s = 1 day

sourceFolder="/opt/rename/"
target="/opt/send/combined/"

path_to_file=$target
current_file="ORACLE_gprtcp_000.log.gz"

fileList=$(find $sourceFolder -type f -name '*.log.gz' -printf '%T@ %p\n' | sort -nb)
## for debugging:
# fileList=$(find $PWD -type f -printf '%T@ %t %p\n' | sort -nb)

echo ${fileList[0]}

org_IFS=$IFS
IFS=$'\n'
group_start_time=0

for line in $fileList; do
    current_time=$(echo $line | awk '{print $1}')
    if [ $group_start_time -eq 0 ] ; then
        group_start_time=$current_time
        hr_time=$( date -d @$current_time +%F_%0k%0M )
        current_file="ORACLE_gprtcp_"$hr_time".log.gz"
    else
        delta=$(($current_time - $group_start_time))
        #echo $delta
        if [ $delta -lt 1801 ] ; then
            # just append file path to current_file
            echo $line | awk '{print $2}' >> $path_to_file"/"$current_file
            echo $line
        else
            # construct new filename based on time of the first member of the group
            hr_time=$( date -d @$current_time +%F_%0k%0M )
            current_file="ORACLE_gprtcp_"$hr_time".log.gz"

            # create file, append file path to current_file
            echo $line | awk '{print $2}' >> $path_to_file"/"$current_file
            echo -e "\nnew group:\n$line"

            group_start_time=$current_time
        fi
    fi
done

IFS=$org_IFS
share|improve this answer
add comment

Assuming that there is no "\n" character in file names :

find . -name '*_*_[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]_*.gz' | perl -le '
    use strict;
    use warnings;
    my %hash;
    while(<>) {
        chomp;
        my($group)=/^([^_]+_[^_]+_[0-9]{11})/;
        $group=~s/[0-2]$/00/;
        $group=~s/[3-5]$/30/;
        push @{$hash{$group}},$_;
    }
    while(my($group,$files_arr_ref)=each%hash) {
        print "processing group $group";
        for my$file (sort @{$files_arr_ref}) {
            print "processing file $file";
            # do system command calls here; for example
            # system "gzip -cd \"$file\" >> $group.txt";
        }
    }
' 

Edit: a few changes after Craig suggestions. First idea was just to use perl for arrays and hashes, finally it's more clear to do all. @ARGV is the list of path to pass to find. For example if name of script is script.pl :

script.pl ${sourceFolder}

#!/usr/bin/perl

use strict;
use warnings;
use File::Find;

my %hash;

sub wanted {
    return unless /^([^_]+_[^_]+_[0-9]{11})/;
    my$group=$1;
    $group=~s/[0-2]$/00/;
    $group=~s/[3-5]$/30/;
    push @{$hash{$group}},$_;
}

File::Find::find(\&wanted, @ARGV);

while(my($group,$files_arr_ref)=each%hash) {
    print "processing group $group\n";
    ### do system command calls here; for example
    # system "rm $group.txt";
    ### or just use perl
    # unlink $group.'.txt';
    for my$file (sort @{$files_arr_ref}) {
         print "processing file $file\n";
         ### and other system command calls here; for example
         # system "gzip -cd $file >> $group.txt";
    }
    ### and here; for example
    # system "gzip $group.txt";
}
share|improve this answer
    
+1. Nice, but you could use the File::Find module ( included with perl these days, don't even need to fetch it from CPAN) instead of find or you could open a pipe from find e.g. open(FIND,'-|','/usr/bin/find','.','name',...); (see perlipc man-page). Either way you'd have a perl script rather than a minimal shell wrapper around perl (so you can use perl -w -c to syntax-check it, and vim syntax highlighting would work properly, etc). BTW, i don't think the perl -l is doing anything useful here...harmless, but not necessary. –  cas Oct 3 '12 at 10:37
    
What an insane collection of [0-9]s! I would probably have preferred a find -regex instead in the first line of the first solution (while remembering that I would have to prepend a path specification, since find -regex is not a search, but a match). –  syntaxerror Aug 20 '13 at 9:01
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.