Hi I have a csv file with data more than l lakhs and seperated by pipe it look like

3||||dfrg||DFgfg||FDGRFG|FGB|FG|4546|@#$|FGFDG|DGFDFG|||FGfg||DGF |||GF |||
3||||dfrg||DFgfg||FDGRFG|FGB|FG|4546|@#$|FGFDG|DGFDFG|||FGfg||DGF |||GF |||
3||||dfrg||DFgfg||FDGRFG|FGB|FG|4546|@#$|FGFDG|DGFDFG|||FGfg||DGF |||GF |||
3||||dfrg||DFgfg||FDGRFG|FGB|FG|4546|@#$|FGFDG|DGFDFG|||FGfg||DGF |||GF |||

I want to check repetion of data in specific column each time i execute my script

For example i want to check if column 1,7,12,14 in all the CSV file contain same data or not if yes then display only those data which are repeting

I have tried

awk -F"|" '{
if (x[$'"$ColumnNo1"'$'"$ColumnNo2"'$'"$ColumnNo3"'$'"ColumnNo4"'])
print $0;
print x[$'"$ColumnNo1"'$'"$ColumnNo2"'$'"$ColumnNo3"'$'"ColumnNo4"']}}
x[$'"$ColumnNo1"'$'"$ColumnNo2"'$'"$ColumnNo3"'$'"ColumnNo4"']=$0}' csvfilename.csv

but i am not getting any output

$ColumnNo1,$ColumnNo2,$ColumnNo3 are shell script variable

Please help :)


Apart from there are few specilized tools to work with csv (e.g. csvtool)

awk -F"|" '
        r = $w SUBSEP $x SUBSEP $y SUBSEP $z #prepare index from 4 fields data
    R[r]{                                    #if index present in array already
        if ( R[r] != 1){                     #if it is a first repetition
            print R[r]                       #print line stored in array
            R[r] = 1                         #mark element «not a first time»
        print                                #print present line
        next                                 #pass rest of code(goto next line)
        R[r] = $0                            #store line in array (first time only)
    ' w=$ColumnNo1 x=$ColumnNo2 y=$ColumnNo3 z=$ColumnNo4 file.csv
  • can you please explain how it will work i.e what is SUBSEP – SinghChan Nov 28 '16 at 9:13
  • I have tried above code and its not working – SinghChan Nov 28 '16 at 9:26
  • @SinghChan I have tested it on your data — it works well. To find reason supply as with the sample data on which code is not working – Costas Nov 28 '16 at 11:25
  • @SinghChan LESS=+"@/^\s*SUBSEP" man awk helps you – Costas Nov 28 '16 at 11:41

Try this (note that your shell variables MUST be integers):

awk -v C1="$ColumnNo1" -v C2="$ColumnNo2" -v C3="$ColumnNo3" -v C4="$ColumnNo4" -F'|' '
       { a1[$C1]++; a2[$C2]++; a3[$C3]++; a4[$C4]++}
       END {
       printf "Non-unique entries in column %d\n", C1 
       for (key in a1) {              
         if (a1[key] > 1) print key
       printf "Non-unique entries in column %d\n", C2
       for (key in a2) {              
         if (a2[key] > 1) print key
       printf "Non-unique entries in column %d\n", C3
       for (key in a3) {               
         if (a3[key] > 1) print key
       printf "Non-unique entries in column %d\n", C4
       for (key in a4) {               
         if (a4[key] > 1) print key
       }}' <myfile.csv

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.