2

by the following command we can print the duplicate lines from file

uniq -d string file.txt

but how we can do it on csv file ?

we need to print the duplicate lines only on fields 1,2 from csv file - not include field 3

FS - ","

for example:

 spark2-thrift-sparkconf,spark.history.fs.logDirectory,{{spark_history_dir}}
 spark2-thrift-sparkconf,spark.history.fs.logDirectory,true
 spark2-thrift-sparkconf,spark.history.Log.logDirectory,true
 spark2-thrift-sparkconf,spark.history.DF.logDirectory,true

expected results:

 spark2-thrift-sparkconf,spark.history.fs.logDirectory,{{spark_history_dir}}
 spark2-thrift-sparkconf,spark.history.fs.logDirectory,true

second:

how exclude the duplicate lines from the csv file ( I mean to delete only the duplicate lines on fields 1,2

expected output:

 spark2-thrift-sparkconf,spark.history.Log.logDirectory,true
 spark2-thrift-sparkconf,spark.history.DF.logDirectory,true
1
$ awk -F, 'NR==FNR{a[$1,$2]++; next} a[$1,$2]>1' file.txt file.txt 
spark2-thrift-sparkconf,spark.history.fs.logDirectory,{{spark_history_dir}}
spark2-thrift-sparkconf,spark.history.fs.logDirectory,true

Two file processing using same input file twice

  • NR==FNR{a[$1,$2]++; next} using first two fields as key, save number of occurrences
  • a[$1,$2]>1 print only if count is greater than 1 during second pass


For the opposite case, simple matter of changing condition check

$ awk -F, 'NR==FNR{a[$1,$2]++; next} a[$1,$2]==1' file.txt file.txt 
spark2-thrift-sparkconf,spark.history.Log.logDirectory,true
spark2-thrift-sparkconf,spark.history.DF.logDirectory,true
  • clear and elegant syntax – yael Aug 22 '17 at 9:01
  • is it possible to delete the lines without to create another file copy ? as "sed -i or perl -i , etc " – yael Aug 22 '17 at 9:02
  • can't think of a way to do it with two file processing like this... you can always do it manually by adding > tmp && mv tmp file.txt – Sundeep Aug 22 '17 at 9:26
0

cut the first two fields, uniq them as you suggested and use the result to grep the lines in the original file:

cut -d, -f1,2 file.cvs |uniq -d|grep -Ff - file.cvs

If the file is not yet sorted, you have to sort it before the uniq:

cut -d, -f1,2 file.cvs |sort|uniq -d|grep -Ff - file.cvs

For the second question (inverted result), use option -u instead of -d:

cut -d, -f1,2 file.cvs |sort|uniq -u|grep -Ff - file.cvs
  • nice , is it possible to add another question - we need also to exclude the duplicate lines from the csv ( I mean to delete only the duplicate lines on fields 1,2 ) – yael Aug 22 '17 at 8:15
  • update question – yael Aug 22 '17 at 8:20
  • answer updated. – Philippos Aug 22 '17 at 8:24
  • from some reason after we created the new file without duplicate lines , i see still duplicate lines – yael Aug 22 '17 at 8:34
  • example - ams-hbase-site,hbase.zookeeper.quorum,{{zookeeper_quorum_hosts}} core-site,fs.defaultFS,true ams-hbase-site,hbase.zookeeper.quorum,true ams-hbase-site,hbase.zookeeper.quorum,{{zookeeper_quorum_hosts}} core-site,fs.defaultFS,hdfs://hdfsha core-site,fs.defaultFS,true – yael Aug 22 '17 at 8:34
0

awk solution:

-- to print the duplicate lines only on fields 1,2 from csv file

awk -F, '$1==f1 && $2==f2{ printf "%s,%s,%s\n%s\n", f1,f2,f3,$0 }
        { f1=$1; f2=$2; f3=$3 }' <(sort -t, -k1,1 -k2,2 file.txt)

The output:

spark2-thrift-sparkconf,spark.history.fs.logDirectory,{{spark_history_dir}}
spark2-thrift-sparkconf,spark.history.fs.logDirectory,true

-- exclude the duplicate lines from the csv file (on fields 1,2)

awk -F, 'f1 && f2{ if($1!=f1 || $2!=f2) { printf "%s,%s,%s\n", f1,f2,f3 } else next }
        { f1=$1; f2=$2; f3=$3 }' <(sort -t, -k1,1 -k2,2 file.txt)

The output:

spark2-thrift-sparkconf,spark.history.DF.logDirectory,true
spark2-thrift-sparkconf,spark.history.fs.logDirectory,{{spark_history_dir}}

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.