6

I want to grep a regex. The pattern I am searching for may appear multiple times in a line. If the pattern appeared many times, I want to separate each occurrence by a comma and print the match only not the full line in a new file. If it did not appear in a line I want to print n.a.

Example. I want to use this regex to find numbers in the pattern: [12.123.1.3].

grep -oh "\[\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}\]" 'filename'

input file (input.txt)

blabla [11.335.2.33] xyuoeretrete [43.22.11.88] jfdfjkfbs [55.66.77.88]
blabla [66.223.44.33]
foo bar
blabla [1.2.33.3] xyuoeretrete [42] bla[1.32.2.4]

intended result in a new file (output.csv):

11.335.2.33,43.22.11.88,55.66.77.88
66.223.44.33
n.a.
1.2.33.3,1.32.2.4

Note: I use Ubuntu

2
  • 1
    As you already see from answers, it is quite far from grep knowledge. grep can select lines, but you want text processing... you need a "real langue" for that.
    – redseven
    Jun 24 at 13:51
  • Are you asking for dotted-octet decimal IPv4 address notation? With each of the 4 numbers taking a value between 0 and 255? Jul 2 at 4:54

5 Answers 5

4

With perl:

perl -lne '
  if (@ips = /\[(\d{1,3}(?:\.\d{1,3}){3})\]/g) {
    print join ",", @ips;
  } else {
    print "n.a.";
  }'

Or using the regexp for dotted-quad-decimal IPv4 addresses from Regexp::Common (libregexp-common-perl package on Debian-based systems such as Ubuntu):

perl -MRegexp::Common=net -lne '
  if (@ips = /\[($RE{net}{IPv4})\]/g)
    print join ",", @ips;
  } else {
    print "n.a.";
  }'

With -n, input can be given either on stdin or read from files whose paths are given as extra arguments or from the output of some commands if passed arguments like some commands|. perl's print prints to stdout by default, which from your shell you could redirect into a file with redirection operators such as >, >> (append), or 1<> (like > except the file is not truncated first, and is opened in read+write mode), and possibly more depending on your shell.

You can also add the -i option for the output to end up replacing the contents of the input files (whose paths must then be given as arguments).

Here, to take the input from a file called input.txt, and overwrite or create a output.csv file with the output:

< input.txt perl... > output.csv
3

Using GNU awk for the FPAT:

awk -v FPAT='\\[([0-9]{1,3}[.]){3}[0-9]{1,3}\\]' -v OFS=, '
{
    $1=$1; print (gsub(/[][]/, "")?$0:"N/A")
}' <infile >output

or with any POSIX awk (which all supports the {x,y} RE intervals):

awk '
{
    bkup=$0;
    gsub(/\[([0-9]{1,3}[.]){3}[0-9]{1,3}\]/, "|")
    gsub(/[][()\\.{}?+*$^]/, "\\\\&")
    n=split(bkup, tmp, $0)
    for(i=1; i<=n; i++){
        if(tmp[i]!=""){
            gsub(/[][]/, "", tmp[i])
            printf ("%s", (sep?",":"") tmp[i])
            sep=","
        }
    }; print (sep?"":"N/A"); sep=""
}' <infile >output

Output is written into the file output.

$ cat output
11.335.2.33,43.22.11.88,55.66.77.88
66.223.44.33
N/A
1.2.33.3,1.32.2.4

Note that your input should not contain | and & characters for the second approach.


Same code with inline explanation:

awk '
{
    #backup from the current record
    bkup=$0;

    #replace desired pattern all with "|" characters 
    #to build regexp patterns of everything other than our desired pattern
    gsub(/\[([0-9]{1,3}[.]){3}[0-9]{1,3}\]/, "|")

    #escape all regexp operators except "|"
    gsub(/[][()\\.{}?+*$^]/, "\\\\&")

    #split the original record (from bkup) into tmp on regexp
    # from the result of the first gsub() above
    n=split(bkup, tmp, $0)

    #loop through the splitted fields on the tmp array
    for(i=1; i<=n; i++){

        #if the current field is not empty
        if(tmp[i]!=""){

            #remove the ], [ characters from it
            gsub(/[][]/, "", tmp[i])

            #and print it (will add comma when it is the second or the next one)
            printf ("%s", (sep?",":"") tmp[i])

            #set comma as the field seperator when at least one field was printed
            sep=","
        }

    #print "N/A" in case there was no field and var "sep" did not set above
    # and then unset the "sep" var
    }; print (sep?"":"N/A"); sep=""

}' <infile >output
0
1

Executable awk file filter.awk:

#! /usr/bin/awk -f
{
    ret = ""
    line = $0
    while (match(line, /\[([[:digit:]]{1,3}\.){3}[[:digit:]]{1,3}\]/) > 0) {
        if (ret != "") {
            ret = ret ","
        }
        ret = ret substr(line, RSTART, RLENGTH)
        line = substr(line, RSTART + RLENGTH)
    }
    if (ret != "") {
        print ret
    }
}

Executed like that:

./filter.awk filename
3
  • when I tried to run I get: sudo: ./filter.awk: command not found. I used sudo I my terminal path is the same of the filter.awk
    – randomname
    Jun 24 at 11:27
  • 1
    The filter.awk file must be executable with command like : chmod 755 filter.awk. Check if the awk path is really /usr/bin with this command : which awk. Fix shebank (first awk file line) if necessary. When you use sudo, like crontab, it's better to use absolute path. Jun 24 at 14:03
  • 2
    @randomname use sudo sparingly and only in very specific situations when absolutely required for very specific reasons. This isn't one of them.
    – Ed Morton
    Jun 25 at 2:32
0

Using Perl in sed mode (-p), (-l) shall remove LF (\n) from every record read and also append to every record printed.

perl -plse '
  $_ = "@{[/\[\K\d{1,3}(?:\.\d{1,3}){3}(?=])/g]}" || "n.a.";
' -- -\"=, file

GNU sed in extended regex mode (-E)

sed -E '
  s/\[([0-9]{1,3}(\.[0-9]{1,3}){3})]/\n\1\n/g

  y/\n_/_\n/
    s/[^_]*_([^_]*_)/\1/g
    s/_?[^_]*$//
  y/\n_/_,/

  /./!c n.a.
' file

Using awk with progressive matching of the regex

_m0="n.a." \
awk -v OFS=, '
BEGIN {
  msg = ENVIRON["_m0"]
  _d = "[[:digit:]]{1,3}"
  fmt = sprintf("%s%%s([.]%%s){%%d}%s", "[[]", "[]]")
  ip_re = sprintf(fmt, _d, _d, 3)
}
{
  t=$0;$0=""
  for (i=1; match(t,ip_re)>0; i++) {
    $i = substr(t, RSTART+1, RLENGTH-2)
    t = substr(t, RSTART+RLENGTH)
  }
}
!NF{print msg};NF' file

Output:

11.335.2.33,43.22.11.88,55.66.77.88
66.223.44.33
n.a.
1.2.33.3,1.32.2.4

0

Using Raku (formerly known as Perl_6)

No value-checking:

raku -ne 'if m:g/ ( \d**1..3 )**4 % "." / { $/.join(",").put } else {"n.a.".say};'    

OR

raku -ne 'm:g/ ( \d**1..3 )**4 % "." / ?? $/.join(",").put !! "n.a.".say;'  

Sample Input:

blabla [11.335.2.33] xyuoeretrete [43.22.11.88] jfdfjkfbs [55.66.77.88]
blabla [66.223.44.33]
foo bar
blabla [1.2.33.3] xyuoeretrete [42] bla[1.32.2.4]

Sample Output (both examples):

11.335.2.33,43.22.11.88,55.66.77.88
66.223.44.33
n.a.
1.2.33.3,1.32.2.4

At least in Perl-family languages, what you're asking for is a match, not grep. hence use of the m/.../ match operator, made 'global' as in m:g/.../ to return multiple instances of the match [which differs from grep, which returns the full element (such as a line, etc.), containing a match].

Briefly, clusters of 1-to-3 digits ( \d**1..3 ) are searched for, with these digits **4 repeated 4 times, and having % "." a period between each instance, and this regex match is searched for globally (m:global or m:g), meaning to get all instances of the match per element (line, etc.), not just the first match. First example: if found (if) output the match contained in the $/ match variable, else say n.a.. Second example: same match conditional used in Raku's ternary operator, i.e. condition ?? True !! False. Thus if the conditional is ?? (True), output the match contained in the $/ match variable, if the conditional is !! False say n.a..


Below WITH value-checking:

raku -ne 'if m:g/ ( \d**1..3 <?{ $/ < 256 }> )**4 % "." / { $/.join(",").put } else {"n.a.".say};'    

OR

raku -ne 'm:g/ ( \d**1..3 <?{ $/ < 256 }> )**4 % "." / ?? $/.join(",").put !! "n.a.".say;'  

Sample Input: same as above

Sample Output (both examples):

43.22.11.88,55.66.77.88
66.223.44.33
n.a.
1.2.33.3,1.32.2.4

Immediately above, Raku code is shown that checks each 1-to-3 digit cluster to ensure it is an integer less than 256. The additional regex element, <?{ $/ < 256 }> , is a positive assertion containing a {...} code-block that checks if $/ match-variable is less than 256. Reference here.

https://raku.org

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.