0

I have 2 csv files whose contents are-

expo1.csv:

102,GREAT,adjective,ENG,p1_0,no,p2_1,no,p3,no,4,yes,p5_2,no,p6,yes....,su1,amb,su_09,no

104,BHAAG,verb,HIN,p1,yes,p2,no,p3_7,amb,p4,no,p5,no,p6_9,yes....,sg4_3,yes,su119,amb

110,.......,su11_0,amb

and impo1.csv:

104,p1,no
102,p2,yes
104,p10,no
110,su11,no

Basically expo1.csv is a file on the server, and impo1.csv is a file I created to update expo1.csv. A script makes the changes in expo1.csv as specified in impo1.csv after performing slight processing in the impo1 data (eg. The line 102,p2,yes from impo1.csv is processed and then an update is made to expo1.csv - p2_1,yes.)

expo1.csv after changes:

102,GREAT,adjective,ENG,p1_0,no,p2_1,yes,p3,no,4,yes,p5_2,no,p6,yes....,su1,amb,su_09,no

104,BHAAG,verb,HIN,p1,no,p2,no,p3_7,amb,p4,no,p5,no,p6_9,yes....,sg4_3,yes,su119,amb

110,.........,su11_0,no

Now after the script makes the changes, we need to validate if the changes are done properly by comparing the impo1 and expo1 files. This is where i'm stuck.

So far I could isolate the data between the commas in impo1.csv separately into variables using awk:

Sno=104 102 104
Posw=p1 p2 p10
cho=no yes no

Now the question is, how do I check this? The impo1.csv files contains around 3000 updates. If I grep p1 expo1.csv|grep no expo1.csv, obviously it will not return the correct result as the file has many 'no' strings. I have tried using a for loop to separate the data using awk into separate variables and then grep using a wildcard - grep sno expo1.csv|grep '/<$posw.*,$cho>/' expo1.csv - but it doesn't work.

Using GNU bash 4.1.2.

EDIT - Should have mentioned this earlier, my bad - There are no clear patterns in the impo1.csv file which I can use to check the expo1 file. I have made corrections to the sample file contents which illustrate my point.

0

The solution is rather simple. You just need to create a pattern from each line of impo1.csv and then grep it from expo1.csv after updated

validate() {
    # $1 ~ impo1.csv
    # $2 ~ expo1.csv after changes
    while read pattern; do
        grep -q "^$pattern" $2 || return 1
    done < <(sed "s/,/,.*/" $1 )
}
  • Thanks, but I wasn't clear enough in my explanation, sorry. Getting patterns from the impo1.csv file is very difficult. I have added the necessary explanation to the question. – Shashank K R Apr 25 '17 at 11:00
  • @ShashankKR Does each line of impo1.csv contains only one update? – linerd Apr 25 '17 at 11:09
  • Yes@linerd one line contains one update. – Shashank K R Apr 25 '17 at 11:12
  • Thanks @linerd, I've decided to use a while loop to do this, but instead of using read pattern i'm using read line. – Shashank K R Apr 26 '17 at 4:52
0
awk -F, '
    NR==FNR{
        for(i=1;i<NF;i)
            DATA[$1 SUBSEP $++i] = $++i;
        next
    }
    DATA[$1 SUBSEP $2] != $3
    ' expo1.csv impo1.csv

will print lines in impo1.csv which are different to data in expo1.csv

  • Note this shows in LQP review queue, probably because of the lack of explanation. – fedorqui Apr 25 '17 at 10:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.