0

using bash(awk), how do I remove duplicates from a pipe delimited file based on multiple columns but columns need to be passed as input arguments. I have below data in the file

Insert|val1|val2|val3|val4 
Insert|val1a|val2|val3a|val4

I want to remove duplicates based on input argument. Sometimes I want rows dropped using position 1 and position 3 and sometimes using position2 and position3. I am using this

 awk -F, '!seen[$1,$3 ]++ '

but don't know how to parameterize $1,$2

1
  • ITYM -F'|', not -F,.
    – Ed Morton
    Mar 29, 2022 at 11:52

1 Answer 1

0

This might be what you're trying to do:

$ awk -v c='1 3' -F'|' 'BEGIN{split(c,k," ")} {key=""; for (i in k) key=key FS $(k[i])} !seen[key]++' file
Insert|val1|val2|val3|val4
3
  • Yes, this works. I just need figure out how make '1 3' as argument for the script. Thanks
    – vks009
    Mar 30, 2022 at 15:24
  • script 1 3 then inside awk -v c="$*" '...' file.
    – Ed Morton
    Mar 30, 2022 at 15:50
  • I am facing another issue.plz check this <unix.stackexchange.com/questions/698261/…>.
    – vks009
    Apr 6, 2022 at 19:56

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .