2

Let's say I have a file like following

1,2,3-5,6
1,2,3-5,6,
1
1-3
1,2,3-,4,5-7
1,2,3-,4,5-7,
1,2,-3,4,5
1,2,-,3,4
1,2,,,3,4
,1,2,3

Only combination of following rules should be considered as valid:

  1. Ranges [0-9]+-[0-9]+
  2. Groups [0-9]+,[0-9]+
  3. Single Numbers [0-9]+

The lines could ending with comma should also be considered valid

I want to extract only

1,2,3-5,6
1,2,3-5,6,
1
1-3

As the other lines shown below do not match the rules

1,2,3-,4,5-7
1,2,3-,4,5-7,
1,2,-3,4,5
1,2,-,3,4
1,2,,,3,4
,1,2,3

Because some lines have incomplete ranges, some have missing numbers in groups


P.S: A PCRE compatible grep only solution would be awesome, but other solutions are also welcome

4
  • 1
    Do you allow negative integers, e.g. -1,-2--1? Or "backwards" ranges as in 9-2?
    – Kusalananda
    Jul 25, 2021 at 12:33
  • 2
    Regarding The lines could ending with comma should also be considered valid – should 1,,,6, or 1,2,3,, be considered valid?
    – rowboat
    Jul 25, 2021 at 14:16
  • 1
    @Kusalananda Good observation.. In my case, neither. No backwards, no negatives. Sorry for the late response, was out the whole day after posting Jul 25, 2021 at 22:08
  • 1
    @rowboat No, neither of the cases you mentioned should be considered valid because of the ambiguity issue. Sorry for the late response.. Jul 25, 2021 at 22:11

5 Answers 5

7

The full pcre that will match the strings you listed (and those that start with a ,) might be:

grep -P '^([0-9]+(-[0-9]+)?(,|$))+$'

How have we got there?

The most basic element to match is a digit, lets assume that [0-9], or the simpler \d in PCRE, is a correct regex for a English (ASCII) digit. Which might as well not be. It could match Devanagari numerals, for example. Then you would need to write: [0123456789] to be precise.

Then, a run of digits would be matched by [0-9]+.

After a number (1 or 3 or 26) ther could be a dash '-' followed by one or several digits ( a number again ):

[0-9]+(-[0-9]+)?

Where the ? makes the dash-number sequence optional.

Then, each of those numbers: 3 (or number ranges: 4-9) should be followed by a comma , (several times):

([0-9]+(-[0-9]+)?,)+

Except that the last comma might be missing:

([0-9]+(-[0-9]+)?(,|$))+

And, if required, a leading comma might be present:

(^|,)([0-9]+(-[0-9]+)?(,|$))+

It is a very good idea to anchor the regex to the beginning and end of the text tested:

^((^|,)([0-9]+(-[0-9]+)?(,|$))+)$

You may test and edit the PCRE regex in this site

If the leading comma should be rejected, use:

^(([0-9]+(-[0-9]+)?(,|$))+)$

That leaves no optional interpretations to the regex machine. All must be matched, and anything that is not matched gets rejected.

It may be written as an (GNU) extended regex:

grep -E '^(([0-9]+(-[0-9]+)?(,|$))+)$'

As a Basic Regular Expression (BRE):

grep '^\(\([0-9]\{1,\}\(-[0-9]\{1,\}\)\{0,1\},\{0,1\}\)\{1,\}\)$'

Where the comma , is optional {0,1}, the regex engine might take some decisions about what to match.


Descriptive Regex?

A more descriptive regex, with spaces and comments might be had by starting it with (?x) in pcregrep

pcregrep '(?x)                  # tell the regex engine to allow
                                 # white space and comments.
           (?(DEFINE)            # subroutines that will be used. 
             (?<nrun> [0-9]+)    # run of digits (n-run).

              # define a range pair. A number run followed by
              # an optional ( dash and another number run )
             (?<range> (?&nrun)  (-(?&nrun))? )    # range pair.
             
             (?<sep> ,)          # separator used.
           )                     # end of definitions.

         # Actual regex to use:
         # (range) that ends in a (sep) 
         # or is at the end of the line,
         # several times (+).

         ^(  (?&range)  ((?&sep)|$)  )+$

        ' file

This regex (once compiled) is exactly equivalent to the original one and will run equally fast. Of course, there is an (negligible) additional time used to compile the regex.

Test example is here

0
5

Using awk to break down each line into comma-delimited fields, and then splitting those fields on dashes into sub-fields, while discarding lines that contains unwanted fields or sub-fields:

BEGIN { FS = "," }

{
    for (i = 1; i <= NF; ++i) {
        # Only the 1st field is allowed to be
        # empty, but only if there are further
        # fields (avoids empty lines).

        if ($i == "" && (i != 1 || NF == 1)) next

        # If the field is split on dashes, it
        # should split into no more than two
        # elements.

        if ((n = split($i, a, "-")) > 2) next

        # Each split-up element needs to be made
        # up of decimal digits only.

        for (j = 1; j <= n; ++j)
            if (a[j] !~ "^[[:digit:]]+$") next
    }

    # The current line is ok to print.

    print
}

This would be used like

awk -f script file

where script holds the awk program.

Or, as "once-liner":

awk -F, '{for(i=1;i<=NF;++i){if(($i==""&&(i!=1||NF==1))||((n=split($i,a,"-"))>2))next;for(j=1;j<=n;++j)if(a[j]!~"^[[:digit:]]+$")next}};1' file

You could easily add a check for "backward ranges" (e.g. 5-2) after the j loop:

if (n == 2 && a[1] > a[2]) next
4
  • Because you went for a program as a solution. I'd like to ask if there are any downsides to using a regex as mentioned by @ImHere instead of a program like yours ? Jul 25, 2021 at 22:51
  • 2
    @GypsyCosmonaut I don't really like finding negative things to say about other's solutions when there's nothing wrong with them. The main issue with large regular expressions is that they are "write only", i.e. it's difficult to read and understand them, and consequently it's difficult to modify and extend them, and to correct bugs in them. It comes down to who you're writing your code for. I often write code that need to be understood by colleagues and by random people on the internet, and I tend to prefer readability to tightly packed code nuggets.
    – Kusalananda
    Jul 25, 2021 at 23:29
  • @GypsyCosmonaut A regex runs faster than the program, it is compiled once, used many times.
    – user232326
    Sep 11, 2021 at 3:33
  • @Kusalananda If you like desciptive code and comments, you can use a regex that ignore whitespace and allow comments. The PCRE flavor of grep allows that if you start it with (?x). I'll add an example in my answer.
    – user232326
    Sep 11, 2021 at 3:37
4
$ perl -n -e 'print if /^((\d+(-\d+)?)(,|$))+$/g' input.txt 
1,2,3-5,6
1,2,3-5,6,
1
1-3

or, same thing with with GNU grep:

$ grep -P '^((\d+(-\d+)?)(,|$))+$' input.txt 
1,2,3-5,6
1,2,3-5,6,
1
1-3
0
2

Antipattern:

grep -Ev '^,|,-|-,|,,' file
1,2,3-5,6
1,2,3-5,6,
1
1-3
grep -E '^,|,-|-,|,,' file
1,2,3-,4,5-7
1,2,3-,4,5-7,
1,2,-3,4,5
1,2,-,3,4
1,2,,,3,4
,1,2,3
4
  • 1
    Logically, the string should not end with a comma: '^,|,-|-,|,,|,$'
    – nezabudka
    Jul 25, 2021 at 18:05
  • 2
    This will allow abc2-23ght, for example. It is difficult to list all invalid patterns. An initial test for [0-9,-]+ for the whole string might help. Like grep -xE '[0-9,-]+' file | grep -vE '^,|,-|-,|,,'
    – user232326
    Jul 25, 2021 at 22:23
  • @nezabudka Not an antipattern, golang allows trailing commas while declaring arrays Jul 25, 2021 at 22:30
  • 1
    As does perl. It allows them deliberately so you can add, delete, or move array elements in the declaration without having to worry about which one is (or used to be) the last element. same for hash declarations. This is especially useful when using an anonymous hash as args to a function (which effectively gives you "free" option processing for the function - no parsing, no getopts). I expect golang does it for pretty much the same reason.
    – cas
    Jul 26, 2021 at 1:00
0

Using Raku (formerly known as Perl_6)

raku -ne '.put if m:g/^^ [ [\d+ [\-\d+]?] [\,|$$] ]+ $$/;'

Sample Input:

1,2,3-5,6
1,2,3-5,6,
1
1-3
1,2,3-,4,5-7
1,2,3-,4,5-7,
1,2,-3,4,5
1,2,-,3,4
1,2,,,3,4
,1,2,3

Sample Output:

1,2,3-5,6
1,2,3-5,6,
1
1-3

An advantage of using Raku is whitespace tolerance within the matcher. This makes for more readable code. Secondly, modifiers of the basic regex engine like :global acquire a leading colon and appear at the head of the m/.../ match construct. This also makes the regex more readable.

Reading the above regex literally, it says: 'Find one-or-more digits followed by an optional (zero-or-one) dash-one-or-more digits, followed by either a comma or end-of-line ($$), the entire preceding pattern repeated one-or-more times.'

Now you might be asking yourself, "So what? It looks just like Perl5." That's because the code above is almost a direct translation of Perl5/PCRE. In fact, Raku (i.e. Perl6) has a new "separator" idiom (i.e. 'modified quantifier') that can be used to solve common regex problems. To wit:

raku -ne '.put if m:g/^^ [\d+ [\-\d+]? ]+ % "," $$/;'

OR

raku -ne '.put if m:g/^^ [\d+ [\-\d+]? ]+ %% "," $$/;'

The first line using % detects matches wherein a comma separator is interposed between the pattern to the left. The second line using %% does the same--but also allows a trailing comma. Your choice.

Perl guru Damian Conway states that Raku (i.e. Perl6) represents an entirely new flavor of Regular Expressions: if you try it out you may be inclined to agree.

https://docs.raku.org/language/regexes#Modified_quantifier:_%,_%%
https://youtu.be/ubvSjW6Nyqk
https://raku.org/

0

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .