-1

I have a file that has 2 columns like the following:

apple pear
banana pizza
spoon fork
pizza plate
sausage egg

If a word appears on multiple lines i want to delete all lines that the repeating word appears, as you can see 'pizza' appears twice so 2 lines should be deleted, the following is the required output:

apple pear
spoon fork
sausage egg

I am aware of using :

awk '!seen[$1]++' 

However this only removes the lines when the string appears in one column, i require a command that will check both columns. How can i achieve this?

1

2 Answers 2

1

A basic two-step approach would be to create a list of non-unique words, and pipe that to grep -v.

For the first part, you could use sort and uniq -d (after serializing the words - perhaps with tr) or - if you want to continue to use awk, something like:

awk -vRS='[ \t\n]' 'seen[$0]++' file

which splits records on whitespace and prints those that have been seen before. So

$ awk -vRS='[ \t\n]' 'seen[$0]++' file | grep -Fvf- file
apple pear
spoon fork
sausage egg
1
  • You should mention that requires GNU awk for multi-char RS and for no space between -v and RS=. It'll also delete grapefruit if grape appears multiple times.
    – Ed Morton
    Jul 8, 2019 at 23:55
0

Tried with below command

for i in `cat filename| perl -pne "s/ /\n/g"| sort| uniq`; do j=`grep -c $i p.txt`; if [[ $j -eq 2 ]]; then sr=`echo $i`; else sed -n '/'$i'/p' p.txt| sed -n '/'$sr'/!p'; fi; done| awk '{if (!seen[$0]++)print $0}'

output

apple pear
sausage egg
spoon fork

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .