2

I was writing a shell script to generate a string in a particular format so that it can be used as a input in one of the XML I am working with.

Given a input file in the format <attribute field>,<data_type>,<size>

instanceid,varchar,256
sysdate,date
status,number
notes,varchar,4000
created_on,date

I want to store in a variable the "check sum" like md5( INSTANCEID || STATUS || NOTES). That is I want all the attribute fields except the field having date as it's type Or'd.

The script I had written is this

IFS=$'\n'
file=$(cat source.txt)
line_number=$(cat source.txt | wc -l)
checksum="md5( "
for line in $file
do
let line_number=line_number-1
data_field=$(echo $line | cut -f1 -d','| tr "a-z" "A-Z")
data_type=$(echo $line | cut -f2 -d',' | tr "a-z" "A-Z")
if [ $data_type != "DATE" ]  && [ $line_number -gt 0 ]
  then checksum+="$data_field || "
elif [ $data_type != "DATE" ] && [ $line_number -eq 0 ]
  then checksum+=" $data_field "
fi
done
checksum+=")"
echo $checksum

This script works fine with all the input scenarios except when the last line has a attribute with date as it's type.

In which case the variable has a value likemd5( INSTANCEID || STATUS || NOTES || )

I tried to check if the last line was a date using tail command, but this again would fail if the last few lines had it's type as date.

How can I do away with the || which appears in the end?

3

The quick answer is checksum="${checksum% || })" instead of checksum+=")". Just unconditionally add the || string in each step and then strip off the last unnecessary one at the very end (so the line_number computation is no longer needed).

A better way to do this is

awk -F, 'BEGIN { printf "md5( " } 
         toupper($2) != "DATE" { printf "%s%s", sep, toupper($1); sep = " || " }
         END { print ")" }' source.txt
1
  • It’s remarkable how rarely cat is useful in a shell script.  $(cat source.txt | wc -l) is a classic useless use of cat; if you needed to count the lines in a file, $(wc -l < source.txt) is a much cleaner ways of doing it.
  • But you don’t need to count the lines in source.txt.
  • file=$(cat source.txt) is an ugly way to read a file;
    while read …
    do
        ︙
    done < filename
    is better.  read has the benefit that it can split lines into fields for you.
  • It’s silly to run tr twice for each line of the file when you need only to run it once on the entire file.  In some situations,
    tr … < filename | while read …
    do
        ︙
    done
    works nicely.  But there’s a problem with this: the while loop runs in a subshell, so changes that you make to shell variables (e.g., checksum) won’t be visible after the loop ends.  Terdon shows one way to work around that problem; here’s another one:
    tr … < filename | { while read …
    do
            ︙
        commands that potentially change checksum.
            ︙
    done
        ︙
    commands that use $checksum.
        ︙
    }
  • As you’ve discovered, identifying the last occurrence of something can be difficult.  It’s often easier to identify the first:

    checksum="md5("
    first=1
    tr "a-z" "A-Z" < source.txt | { while IFS=, read data_field data_type size
    do
        if [ "$data_type" != "DATE" ]
        then
            if [ "$first" ]
            then
                first=
            else
                checksum+=" || "
            fi
            checksum+="$data_field"
        fi
    done
    checksum+=")"
    echo "$checksum"
    }
    

    Note that you really don’t need to test if [ "$data_type" != "DATE" ] twice.
    Also note that you should always quote references to shell variables (e.g., "$data_type") unless you have a good reason not to and you’re sure you know what you’re doing.

  • As a further optimization, you can eliminate the first variable and simply use checksum itself to identify your first iteration through the loop:

    checksum=
    tr "a-z" "A-Z" < source.txt | { while IFS=, read data_field data_type size
    do
        if [ "$data_type" != "DATE" ]
        then
            if [ "$checksum" != "" ]
            then
                checksum+=" || "
            fi
            checksum+="$data_field"
        fi
    done
    checksum="md5($checksum)"
    echo "$checksum"
    }
    
0

You don't need anything half as complicated as what you've written. You could just do:

#!/usr/bin/env bash

checksum="md5("
## Read each line into the fields array (read -a fields), with fields
## separated by commas (IFS=,)
while IFS=, read -a fields
do
    ## If the 2nd element of the array is not "DATE"
    if [ ${fields[1]} != "DATE" ]
    then
        ## Add this to $checksum
        checksum+="${fields[0]} || "
    fi
## The tr is making everything upper case and then feeds
## directly into the while loop.
done < <(tr "a-z" "A-Z" < "$1")
## Get rid of the last || and add the closing ")"
checksum="${checksum% || })"
printf "OUT is: %s\n" "$checksum"

You then run the script with your file as input:

$ foo.sh file
OUT is: md5(INSTANCEID || STATUS || NOTES)
  • (1) “The script you posted only stores the details of the first line of input.”  Huh?  file=$(cat source.txt) reads the entire file, and for line in $file loops through the lines.  (2) I’m shocked that < "$@" isn’t a syntax error.  If there are two or more parameters, it is equivalent to < "$*". – G-Man May 20 '15 at 4:37
  • @G-Man 1) yes, I had originally misunderstood the requirements. That sentence was left over from my first draft of the answer, thanks for pointing it out. 2) You're absolutely right, I don't know what I was thinking. Fixed now. – terdon May 20 '15 at 11:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.