-1

I wrote a shell script which reads some files and copy all the data from these files to other consolidated file.

This happens multiple times and previously copied data will not be copied again. For this i am saving the last copied line in a separate file and comparing it from the second time.

Flow of logic :

  • For the first time, read individual files and copy the whole data except last line into the consolidated file. Saving the last copied line into a temporary file.

sed '$d' $file >> $consolidatedCSVFile

tail -2 $file | head -1 > $consolidatedCSVFile$lastLines/$(basename $file)$lastLine

  • From second time, read last line from the temporary file and search for that line in individual file. When found, take all the lines from the next line and append to consolidated file.

Full script:

#!/bin/bash

consolidatedCSVFile=$1
endTime=$2

curdate=$(date +%s)

CSVFiles=${@:3}
CSVFiles=${CSVFiles[@]}

lastLines=_LASTLINES
lastLine=_LASTLINE

newMerge=false

# Returns 1 if Consolidated file is empty
[ -s $consolidatedCSVFile  ]
if [ $? == 1  ]
    then
        mkdir $consolidatedCSVFile$lastLines
        newMerge=true
fi

testInProcess=false

if [ $endTime -ge $curdate  ]
    then
        testInProcess=true
fi

for file in $CSVFiles
do
if $newMerge ;
    then
        touch $consolidatedCSVFile$1astLines5(basename $file)$lastLine
        sed $d $file >> $consolidatedCSVFile
        tail -2 $file | head -1 >   $consolidatedCSVFile$1astLines5(basename $file)$lastLine
    else
        availableLastLine="`cat $consolidatedCSVFile$1astLines/$(basename $file)$lastLine`"
        if $testInProcess
            then
                sed '1,/^'"$availableLastLine"'$/d' $file | sed '$d' >> $consolidatedCSVFile
                tail -2 $file | head -1 > $consolidatedCSVFile$1astLines5(basename $file)$lastLine
            else
                sed '1,/^'"$availableLastLine"'$/d' $file >> $consolidatedCSVFile
        fi
fi

done

if  !  $testInProcess ;
then
    rm -rf $consolidatedCSVFile$lastLines
fi

Sometimes, i get error as sed: -e expression #1, char 26: unknown command: 'X'

My guess

I am trying to combine the files generated by JMeter. Sometimes, the files are generating data like 1439459065780,5,SOAP/XML-RPC Request,200,OK,ws test 1-3,text,true,267,3,3,5

I think the problem is while sed '1,/^'"$availableLastLine"'$/d' $file | sed '$d' >> $consolidatedCSVFile at SOAP/XML. Slash(/) could be problem here.

0

worked after modifying availableLastLine to availableLastLine=$(sed 's/\//\\\//' $consolidatedCSVFile$lastLines/$(basename $file)$lastLine).

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.