0

I have a file with data as follows:

"google1|yoo|dummy|yes|wow|/" + VARIABLE + "/"
"google2|hub|lab|dummy|yes|/" + VARIABLE + "/"
"google3|short|lab|yoo|/" + VARIABLE + "/"
"google4|hello|good-guy|bad-girl|lol|dummy|/" + VARIABLE + "/"
"google5|good-guy|a4-123|yoo|/" + VARIABLE + "/"
"google6|bad-girl|b4-124|hub|/" + VARIABLE + "/"

Now, I want to get a list of strings between delimiter "|" (pipe).

Output should be as

yoo
dummy
yes
wow
hub
hello
good-guy
bad-girl
a4-123
b4-124
dummy
lol
short
lab

Basically, I want to have unique values from the list of strings after delimiter filter. I tried using awk as

awk -F"|" '{gsub(/\).*/,"",$2);print $2}' file

But, I get wrong data.

  • there is a typo between input file and expected output.. good-boy should be good-guy – Sundeep Sep 30 '16 at 13:38
3

If you have grep with pcre option:

$ grep -oP '\|\K[^|]+(?=\|)' ip.txt | sort -u
a4-123
b4-124
bad-girl
dummy
good-guy
hello
hub
lab
lol
short
wow
yes
yoo
  • -o print only matching pattern
  • -P use pcre regex
  • \|\K positive lookbehind to see if | is there before our string to be extracted
    • similarly, (?=\|) positive lookahead to see if there is | after our string to be extracted
  • [^|]+ string to be extracted - simply negate | and get one or more of such character
  • sort -u to get unique value

If you want to preserve order in which these strings are found:

$ grep -oP '\|\K[^|]+(?=\|)' ip.txt | awk '!seen[$0]++'
yoo
dummy
yes
wow
hub
lab
short
hello
good-guy
bad-girl
lol
a4-123
b4-124
  • 1
    +1 was just about to suggest the same but with (?<=\|).*?(?=\|) – steeldriver Sep 30 '16 at 13:41
  • 1
    yes.. Better option and works perfectly :-) – Kasino Sep 30 '16 at 13:50
3

what about the following ?

cut file -d'|' -f2,3,4 | tr '|' '\n'

The above command will print fixed number of columns (3). If you want to print variable number of columns, up to first occurrence of /, you could use something like:

cut -d'/' -f1 file | cut  -d'|' -f2- | tr '|' '\n'
  • This works.. But -f2,3,4 (here I dont know how many maximum will be there) but atleast not more than 20. Is there a way to give range ? – Kasino Sep 30 '16 at 13:27
  • Thanks and works.. But extra spaces.. so i did at the end as | grep '[^[:blank:]]'| sort -u :-) – Kasino Sep 30 '16 at 13:50
2

If you don't care about order, you could use a perl hash to ensure uniqueness e.g.

$ perl -lne '$h{$_}++ for /(?<=\|).*?(?=\|)/g; END{print for keys %h}' file
short
b4-124
lol
yes
bad-girl
lab
yoo
good-guy
hub
dummy
hello
a4-123
wow

See creating a hash with regex matches in perl

  • how about perl -lne 'for (/(?<=\|).*?(?=\|)/g) {print if !$h{$_}++}' file to print in same order as found... – Sundeep Sep 30 '16 at 14:21
  • 1
    @Sundeep doh! of course... much better - thanks! – steeldriver Sep 30 '16 at 14:22
0

Your output has "dummy" repeating. This is what I get with the script below --

   awk -f f1.awk /tmp/f1
    short
    hub
    wow
    hello
    a4-123
    b4-124
    yes
    yoo
    lol
    bad-girl
    good-guy
    lab
    dummy

    cat f1.awk 
    {
      n=split($1,a,"|")

      for(i=2; i<n; i++) {
        arr[a[i]] = a[i] 
      } 
    }   
    END{
      for (var in arr) 
        print(var)  
    }

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.