0

I have a file where strings are separated by a special string (not a comma or delimiter) e.g. <vvv>. I want to check if all strings in the first field are unique. If duplicate rows for the same field found, I want to remove any repeated rows (leave the first occurrence).

Example:

aaa<vvv>bbb<vvv>ccc
xxx<vvv>yyy<vvv>zzz
aaa<vvv>new<vvv>new2
111<vvv>222<vvv>333

I want to get:

aaa<vvv>bbb<vvv>ccc
xxx<vvv>yyy<vvv>zzz
111<vvv>222<vvv>333

We removed aaa<vvv>new<vvv>new2 because aaa has appeared already.

I do not prefer to us awk unless if it is the only solution. Its syntax is a bit complex to me as an unfamiliar to Linux.

  • Is the ordering of the lines in the result important? – Kusalananda Apr 19 at 10:25
  • Yes. I prefer to keep everything as is. Just remove duplicates. – user9371654 Apr 19 at 10:26
  • 1
    @user9371654L Why tag awk if you can't use it? It is as-straightforward as it could get, if used with awk – Inian Apr 19 at 10:39
2

Without using awk very much:

$ awk -v OFS="<" '{ print NR, $0 }' file | sort -t '<' -u -k2,2 | sort -t '<' -k1,1n | cut -d '<' -f 2-
aaa<vvv>bbb<vvv>ccc
xxx<vvv>yyy<vvv>zzz
111<vvv>222<vvv>333

This only uses awk to insert a line number followed by < into the original data. We do this to be able to keep track of the ordering of the original lines. We use < as the delimiter between the line number and the rest of the line because this also appears as the delimiter between the original first field and the rest of the line.

After the first stage of the pipeline, in which awk is used to insert the line numbers, the data will look like

1<aaa<vvv>bbb<vvv>ccc
2<xxx<vvv>yyy<vvv>zzz
3<aaa<vvv>new<vvv>new2
4<111<vvv>222<vvv>333

The next step of the pipeline sorts this on the second field (the first original field), removing duplicates. The result will be

4<111<vvv>222<vvv>333
1<aaa<vvv>bbb<vvv>ccc
2<xxx<vvv>yyy<vvv>zzz

The second sort restores the original line order by sorting the lines on the first field numerically, and we get

1<aaa<vvv>bbb<vvv>ccc
2<xxx<vvv>yyy<vvv>zzz
4<111<vvv>222<vvv>333

The cut then removes the numbers from the first field (and the inserted delimiter).


A solution that gives ordered output without using awk at all would look like

$ sort -t '<' -u -k1,1 file
111<vvv>222<vvv>333
aaa<vvv>bbb<vvv>ccc
xxx<vvv>yyy<vvv>zzz

This is essentially the second step in the above pipeline and it sorts the file on the first field while removing duplicates.


An awk solution would look like

$ awk -F '<' '!seen[$1]++' file
aaa<vvv>bbb<vvv>ccc
xxx<vvv>yyy<vvv>zzz
111<vvv>222<vvv>333

This stores the first field as a key in an associative array called seen and post-increments the associated value. If the value in the array for the given key is zero (i.e., this first field has not been seen before), the line is printed.

0

Or equivalently with neither awk nor cut, but with sed:

$ sed '=' file \
      | sed 'N;s/\n/</' \
      | sort -t"<" -u -k2,2 \
      | sort -t"<" -k1,1 \
      | sed 's/^[0-9]*<//'
aaa<vvv>bbb<vvv>ccc
xxx<vvv>yyy<vvv>zzz
111<vvv>222<vvv>333

But this is very clunky. @Kusalananda's last (awk-based) solution is much better.


Just for pedagogy's sake, the two first sed blocks above, are equivalent to Kusalananda's more compact awk cmd :

  • sed '=' file, prints line numbers for future ordering
  • sed 'N;s/\n/</', appends the next input line in pattern space (i.e. "joins current and next lines") and substitutes the end-of-line \n with <.

The 3rd and last sed tidbit, sed 's/^[0-9]*<//', substitutes the line number and "<" placed earlier at the beginning of each line, with nothing.


For more details on sed, issue $ info sed in your console.

0

Using GNU sed we can do the task as given :

$ sed -Ene '
   G
   /^([^<]+)<vvv>.*\n\1(\n|$)/d
   P;s/<vvv>.*//;H
 ' input.txt

Store the first field in hold space and compare it to the current line's first field. Only when they differ, update the hold as also print the current line.

0

Tried with below 2 methods

Method1

 awk -F "<" '{if (!seen[$1]++)print }' filename

Method2

awk -F "<" '!a[$1]++' filename

output

aaa<vvv>bbb<vvv>ccc
xxx<vvv>yyy<vvv>zzz
111<vvv>222<vvv>333

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.