-1

I have two columns (or could separate as two files) in a huge file and i need to remove the duplicate city name in the state/area column.

los angeles      los angeles ca, usa, west
new york         new york ny, usa, east
vancouver        vancouver bc can, west
...

can i somehow pipe column one as a variable and pattern match and remove from column 2?

  • Have you tried anything ? (this should be quite easy if you're familiar with regex). Apparently the columns are tabs/spaces separated, right ? – don_crissti Mar 12 '16 at 12:58
  • is it quite easy? how do you know what to pattern match on? – Azeli Mar 12 '16 at 13:04
  • 1. Yes. 2. You use something called "backreference"... – don_crissti Mar 12 '16 at 13:07
0

using awk, with input file

los angeles     los angeles ca, usa, west
new york        new york ny, usa, east
vancouver       vancouver bc can, west
paris   france paris, europe
berlin  germany berlin
mardrid ola, spain

where columns are tab separated properly.

awk -F\\t '{if (i=index($2,$1)) $2=substr($2,0,i-1) substr($2,i+length($1)) ; print }' u

where

  • -F\\t use tab as deleimiter
  • i=index($2,$1) search occurence of $1 in $2
  • if ( ) if found ..
  • $2=substr($2,0,i-1) substr($2,i+length($1)) replace $2 by substring where $1 is stripped (this assume only one occurence of $1)
  • print print result

result

los angeles  ca, usa, west
new york  ny, usa, east
vancouver  bc can, west
paris france , europe
berlin germany
mardrid ola, spain
  • beware of resritictive condition, if lines are extraced from excel (.tsv), that should be OK, but if not ... – Archemar Mar 12 '16 at 13:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.