0

I want to perform some mathematical calculation on some csv files present inside a directory, using them as input. The files will be random so using static names is out of option. Also, the requirement is to generate an actual sequence of number using the 1st and last number in the csv sheet, read the actual count of rows in the sheet and then calculate the percentage of missing sequence. I'm using a For loop to read files, generate sequence, do a wc -l to count the rows, store it in a temp file (total.csv) and then perform percentage operation. At times the sheet might have just one row so that has to be taken into consideration and if that is the case then missing percentage should be 0, hence using If condition inside For loop. In the end i'm removing the temp (total.csv) file and the loop will continue for all csv files in directory. Here's what i have done.

FILES=$dir/*.csv
for f in $FILES;
do
        var1=$(cat $f | wc -l)
        var2=$(cut -d ',' -f7 $f | head -n 1)
        var3=$(cut -d ',' -f1 $f | head -n 1)
        seq1=$(head -n 3 $f)
        seq2=$(tail -n 3 $f)
        if [ "$seq2" -gt "$seq1" ]; then
        seq $seq1 1 $seq2 | cat > $dir/../total.csv
        total=$(cat $dir/../total | wc -l)
        actual=$(cat $f | wc -l)
        missing=$(expr $total - $actual)
        percentage=$((100*missing/total))
        echo "missing percentage of var1 and var2 is $percentage:"
        rm $dir/total.csv
        else
        echo "Loss percentage is 0"
        fi

done

Above script reads the files, but while generating sequence it throws error. The numbers in this error are the 1st line of my input csv file.

./test.sh: line 22: [: 9,-60,18240,28,0,1517400128,2356,0,0,0,0,551: integer expression expected

I'm stuck with this and unable to figure out the issue. Is there any better mechanism to accomplish the requirement stated above? Thanks in advance.

  • 1
    What are you expecting the result of [ "$seq2" -gt "$seq1" ] to be? – steeldriver Feb 14 '18 at 1:34
  • That if condition is basically to check if the files contain more than 1 row and if yes then generate sequence based on 1st and last value of a particular column which i have stored as variables. If the seq2 is less than seq1, then in any case the calculation will exit with a message. – rajanan Feb 14 '18 at 1:46
  • Aren't the first and last (or, at least, first and seventh) values $var2 and $var3 though, not $seq1 and $seq2? – steeldriver Feb 14 '18 at 1:53
  • Sincere apologies if what i stated is not clear somehow. Actually var2 and var3 are just the first column values in sheet which i'll be using as identifiers to echo results. $seq1 and $seq are the first and last values from 3rd column in the sheet. There might be some values missing between seq1 and seq2 in actual input sheet, hence i'm generating a sequence using those 2 values to derive actual count of numbers between 1st and last values. – rajanan Feb 14 '18 at 2:05
  • 1
    I think you may be confused about what head -n 3 does: it will return the first three lines of the file (not the 3rd column of the first line, for example). Likewise for tail -n 3. – steeldriver Feb 14 '18 at 2:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.