0

Input csv file :

key,c1,c2,c3......,cn
1,car,phone,cat,.....,kite
2,abc,def,hij,.......,pot
1,yes,no,is,.........,hello
2,hello,yes,no,......,help

Output csv file:

Key,c1,c2,c3,.......,cn
1,caryes,phoneno,catis,.....,kitehello
2,abchello,defyes,hijno,....,pothelp

The input file has 14 million rows. Can someone help with an efficient way of doing this? Thanks in advance.

  • There are always exactly two lines with the same key and each line has n+1 elements? – Hauke Laging May 16 '18 at 22:04
  • No. It was just a sample. There could be 1000s of lines with the same key as there are about 14 million rows. There are 5550801 unique keys in the dataset. All the rows have the same number of columns, yes. – Meghana M May 16 '18 at 22:22
0

I don't know how efficient it will be, but you could do something like this with a hash of anonymous arrays, using a map to apply the string concatenation assignment .= to each element of each hash value:

perl -F, -nle '
  $k = shift @F;
  map { $h{$k}[$_] .= $F[$_] } 0..$#F 
  }{ 
  for $k (sort { $a <=> $b } keys %h) {
    print join ",", $k, @{ $h{$k} }
  }' file
key,c1,c2,c3......,cn
1,caryes,phoneno,catis,..............,kitehello
2,abchello,defyes,hijno,.............,pothelp

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.