1

I am working with sample data, over 10k lines, similar to the following:

hxxp://google.com 
hxxp://google.com "Seen"
hxxp://yahoo.com "Check again"
hxxp://yahoo.com 
hxxp://about.com
hxxp://x.com
hxxp://y.com
hxxp://z.com
hxxp://reddit.com
hxxp://a.com "good"
hxxp://a.com
hxxp://b.com "good"
hxxp://c.com
hxxp://c.com "good"
hxxp://c.com

I've been trying to find a way to achieve these results:

hxxp://google.com "Seen"
hxxp://yahoo.com "Check again"
hxxp://about.com
hxxp://x.com
hxxp://y.com
hxxp://z.com
hxxp://reddit.com
hxxp://a.com "good"
hxxp://b.com "good"
hxxp://c.com "good"

In the sample data, there might be a duplicate row. If there is a duplicate row, delete the row that is missing a second column. The delimiter is space.

awk -F' ' '!seen[$1]++' dupe.txt > clean.txt

That doesn't seem to be the right one liner for me.

2
  • Do you need to keep the order of the lines?
    – choroba
    Commented Jul 27, 2018 at 15:43
  • No, I plan to sort the list alphabetical once I am done.
    – mernix
    Commented Jul 27, 2018 at 15:47

4 Answers 4

2
$ LC_ALL=C sort -r <file | LC_ALL=C sort -k1,1 -us
hxxp://a.com "good"
hxxp://about.com
hxxp://b.com "good"
hxxp://c.com "good"
hxxp://google.com "Seen"
hxxp://reddit.com
hxxp://x.com
hxxp://y.com
hxxp://yahoo.com "Check again"
hxxp://z.com

The first sort sorts the file in reverse order, using the whole line as the sorting key. This will result in the intermediate result

hxxp://z.com
hxxp://yahoo.com "Check again"
hxxp://yahoo.com
hxxp://y.com
hxxp://x.com
hxxp://reddit.com
hxxp://google.com "Seen"
hxxp://google.com
hxxp://c.com "good"
hxxp://c.com
hxxp://c.com
hxxp://b.com "good"
hxxp://about.com
hxxp://a.com "good"
hxxp://a.com

given the example data. Notice how the lines containing extra annotation always comes before the corresponding lines with no extra annotation.

The second sort only produces lines whose sorting key is unique. We use only the first field (the URL) as the sorting key. We also ask sort to use a "stable" sorting algorithm with -s. This means that the order of lines with identical keys will not change from what's in the input.

The -u and -s combination gives you only the lines with extra annotation for the URLs that are repeated.

The LC_ALL=C bit is to makes sure sort gets a sane locale so that the lines sorts correctly for this to work.

0
1

if your example contains all states of yours and assumes your file named Sample.data:

grep " " Sample.data > Result
cut -d' ' -f1 Result > FirstCol
grep Sample.data -v -f FirstCol >> Result
0

You could do this:

awk '
    !($1 in a) || NF > 1 {a[$1] = $0} 
    END {for (url in a) print a[url]}
' dupe.txt

The order of the lines will not be preserved.

This will keep the order but requires 2 passes over the file:

awk '
    NR == FNR {
        if (!($1 in a) || NF > 1) {a[$1] = $0} 
        next
    }
    $0 == a[$1]
' dupe.txt dupe.txt
3
  • Ran that and output to clean.txt. Then "sort - u clean.txt > final.txt" and I noticed that there are duplicates (two rows with only 1 second column).
    – mernix
    Commented Jul 27, 2018 at 15:51
  • 1
    I don't understand your comment. Can you update your sample data to help reproduce the problem? Commented Jul 27, 2018 at 15:54
  • Updated with more sample data. From my previous note, I ran your argument and noticed it left duplicate entries.
    – mernix
    Commented Jul 27, 2018 at 17:04
0
perl -lane '
    push @h, $F[0] if ! exists $h{$F[0]};
    $h{$F[0]} = $_ if ! exists $h{$F[0]} || @F > 1;
    }{ print $h{$_} for @h;
' dupe.txt > clean.txt

Explanation:

  • @h stores the order in which the keys, viz., the first fields are encountered.
  • %h is the hash which is keyed on the first field, $F[0] and the corresponding values are the lines in toto.
  • The values of the hash %h are updated everytime a new key is seen OR when we see an old key but the line has more than one field, in which case it is ripe for updation.
  • At the eof, we simply loop over the keys stored in the array @h and pull out the corresponding values and take to the stdout.

Output:

xxp://google.com "Seen"
hxxp://yahoo.com "Check again"
hxxp://about.com
hxxp://x.com
hxxp://y.com
hxxp://z.com
hxxp://reddit.com
hxxp://a.com "good"
hxxp://b.com "good"
hxxp://c.com "good"

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .