2

Consider a file file2.txt having the following content:

P 89 24 -1.5388040474568784e+01 7.4421775186012660e+00 -1.3143195543234219e+03 1.3168884860257754e+03 8.0419002445999993e+01 44 0 0 -97 0
P 122 -4 -1.4869334602986523e+01 5.7316939411954255e+00 -1.3144161801429666e+03 1.3169704096915282e+03 8.0419002445999993e+01 44 0 0 -370 0
P 493 -24 -1.4690576431881317e+01 7.3848907323212831e+00 -1.3144620647251766e+03 1.3170224315489374e+03 8.0419002445999993e+01 62 0 0 -499 0
E 3 -1 -1.0000000000000000e+00 -1.0000000000000000e+00 -1.0000000000000000e+00 9999 0 970 1 2 0 7 1.7003962000000002e+05 8.5019810000000018e-01 8.5019810000000018e-01 8.5019810000000018e-01 3.0000000000000000e+01 3.8153441026312507e+01 1.0000000000000000e+11
E 4 -1 -1.0000000000000000e+00 -1.0000000000000000e+00 -1.0000000000000000e+00 9999 0 818 1 2 0 7 1.7003962000000002e+05 8.5019810000000018e-01 8.5019810000000018e-01 8.5019810000000018e-01 3.0000000000000000e+01 3.2509364886711985e+01 1.0000000000000000e+11
P 5 2 0 0 3.7531787088999999e+02 3.8383684055052936e+02 8.0419002445999993e+01 22 0 0 -6 0
P 8 24 7.0195398693654170e+00 3.1543502387874696e+01 5.5989200759599044e+01 1.0318077843755555e+02 8.0419002445999993e+01 44 0 0 -50 0
P 67 28 5.8271676589304882e+00 3.3476871962084061e+01 5.6723118833601163e+01 1.0411236719963519e+02 8.0419002445999993e+01 44 0 0 -168 0
P 219 13 6.0328453988772415e+00 3.3531592253635168e+01 5.6777179460595200e+01 1.0417114266715717e+02 8.0419002445999993e+01 44 0 0 -329 0
P 444 -24 6.4646967953734418e+00 3.4909545978243479e+01 5.7879920796889749e+01 1.0525098522544691e+02 8.0419002445999993e+01 62 0 0 -452 0
E 5 -1 -1.0000000000000000e+00 -1.0000000000000000e+00 -1.0000000000000000e+00 9999 0 598 1 2 0 7 1.7003962000000002e+05 0 0 8.5019810000000018e-01 3.0000000000000000e+01 6.8997318544430456e+01 1.0000000000000000e+11

I want to extract only the strings P ... 24 ... or P ... -24 .... This is what I do:

cat file2.txt | grep -E '(P [0-9]+ 24 | P [0-9] + -24 |P [0-9][0-9]+ 24 | P [0-9][0-9] + -24 |P [0-9][0-9][0-9] + 24 | P [0-9][0-9][0-9] + -24 |P [0-9][0-9][0-9][0-9]+ 24 | P [0-9][0-9][0-9][0-9] + -24 )' &> file3.txt

But the resulting file3.txt contains only the strings P ... 24. Could you please tell me what I am doing wrong?

4
  • 1
    Why can't you use awk or cut instead?
    – k.Cyborg
    Feb 12, 2023 at 16:29
  • 13
    P [0-9] + -24 I believe there should not be a space between [0-9] and +. The purpose of + is to say "one or more occurrences of the previous pattern". But if you have [0-9] +, then the plus applies to the space, not [0-9]. Looks like you have made this mistake in several places. Feb 12, 2023 at 21:40
  • Is the original data really space delimited or tab delimited? If it is tab delimited, you can literally paste it into a spreadsheet (online works) and just copy out the columns you need... Even if it is just space delimited, convert them to tabs and drop it into a spreadsheet.
    – Nelson
    Feb 13, 2023 at 3:33
  • I can always recommend regex101. Here, you can see and debug your attempt at catching: regex101.com/r/QISWQp/1 and here is @bu5hman's answer: regex101.com/r/QISWQp/2
    – Therkel
    Feb 13, 2023 at 10:44

9 Answers 9

13

.... what am I doing wrong? ... apart from making it far more complicated that it needs to be ... you are trying to match multiple spaces and leading spaces that are not in your strings in all of the cases for -24 and in some of other cases too ....

P [0-9]+ 24 |

is fine P then then a series of digits [0-9]+, then and 24 followed by a space

| P [0-9] + -24 |

here there is a before the P and after the digits you had one or more spaces + followed by another which fails to match because of the additional spaces

|P [0-9][0-9]+ 24 |

fine again though all matches are already caught in the first pattern and so it is redundant

| P [0-9][0-9] + -24 |

extra spaces, the same as -24 above ... no match

|P [0-9][0-9][0-9] + 24 |

extra space before the + so it is looking for 2 or more again...

| P [0-9][0-9][0-9] + -24 |

leading space before the P and 2 or more before -24 again.

|P [0-9][0-9][0-9][0-9]+ 24 |

fine but redundant

| P [0-9][0-9][0-9][0-9] + -24

leading space before the P and 2 or more before -24 again.

While @gillesquenot has a far more elegant solution, yours "works" if you lose the additional spaces...

grep -E '(P [0-9]+ 24 |P [0-9]+ -24 |P [0-9][0-9]+ 24 |P [0-9][0-9]+ -24 |P [0-9][0-9][0-9] + 24 |P [0-9][0-9][0-9]+ -24 |P [0-9][0-9][0-9][0-9]+ 24 |P [0-9][0-9][0-9][0-9]+ -24 )'

And if you have the possibility of multiple spaces

grep -E '^P +[0-9]+ +-?24'

edit

This is a useful resource for seeing what matches and where in any string

12

Seems a task for ! This is DSL ¹: exactly the case where awk is the way to go.
Simple, re-usable, efficient, and quick.

without regex:

awk '$1 == "P" && ($3 == "-24" || $3 == "24")' file2.txt 

or using regex:

awk '$1 == "P" && $3 ~ /^-?24$/' file2.txt 

Here, in the regex /^-?24$/, the dash is optional, which permit the use of this short regex.


The regular expression matches as follows:
Node Explanation
^ the beginning of the string
-? '-' (optional (matching the most amount possible))
24 '24'
$ before an optional \n, and the end of the string

¹ Question to the most known chat

Q:what means DSL for a developer?

DSL (Domain-Specific Language) is a type of computer language designed to make coding specific tasks easier. It is tailored to a particular application domain, allowing developers to write code quickly and efficiently. Common usage of DSL include database queries, processing text and scientific simulations. DSLs allow developers to quickly build applications for a particular purpose, offering a more intuitive approach to programming that is less reliant on syntax

2
  • @MarcusMüller Hmm... awk is specifically not a DSL, which makes it versatile enough to deal with these kind of problems. A DSL would be aware of the type of simulation or measurement data that we see in the question and would be able to attach meaning to the individual fields of the P and E records, possibly even providing a way of solving this issue without having to deal with two types of records or having to single out a space-delimited field from the others, etc.
    – Kusalananda
    Feb 13, 2023 at 17:15
  • 3
    @Kusalananda I think I don't agree with you on that one – the intended purposes might not be that narrow for awk, but: awk is a DSL for processing of string-oriented data; that's its domain. It's really not useful for writing other programs. And: you often, and usually, find awk programs embedded into more general-purpose language programs to solve problems specific to that very domain and nothing else. Feb 13, 2023 at 17:30
9

Let's simplify a bit the regex:

grep -E '^P [0-9]+ -?24\b' file2.txt 

The main issue from your regex (apart non needed complexity and readability) is that you use extra erroneous spaces.

No need:

If you think you can have more space than just one in your input, you can use one of:

  • +
  • [[:space:]]+ (POSIX character class)
  • \s+ require -P aka PCRE switch of grep

The regular expression matches as follows:

Node Explanation
^ start of the line anchor
P 'P' + space
[0-9]+ any character of: '0' to '9' (1 or more times (matching the most amount possible))
a space
-? '-' (optional (matching the most amount possible))
24 '24'
\b word boundary
0
7

Your regular expression seems to have random spaces inserted into it. A space in a regular expression will match a space in the text you apply it to. Also note that [0-9]+ will match the same strings as [0-9][0-9]+ and [0-9][0-9][0-9]+ (the reverse is not true though).

You should therefore be able to use

grep -E '^P [0-9]+ -?24 ' file2.txt

Note the use of the anchor to ensure that the P is matched at the start of the line, and -? to match an optional dash before 24.

This is identical to what's suggested in another answer, apart from in the end, where I use a simple space in place of the non-standard word-boundary pattern \b. I can do that if I know that there is a space to be matched in the data, which the example seems to suggest there is.


With Miller (mlr), treating the input as a file with implicitly-integer-indexed fields ("nidx") on each line, filtering out (extracting) the lines with a P in the first field and a third field with an absolute value of exactly 24:

$ mlr --nidx filter '$1 == "P" && abs($3) == 24' file2.txt
P 89 24 -1.5388040474568784e+01 7.4421775186012660e+00 -1.3143195543234219e+03 1.3168884860257754e+03 8.0419002445999993e+01 44 0 0 -97 0
P 493 -24 -1.4690576431881317e+01 7.3848907323212831e+00 -1.3144620647251766e+03 1.3170224315489374e+03 8.0419002445999993e+01 62 0 0 -499 0
P 8 24 7.0195398693654170e+00 3.1543502387874696e+01 5.5989200759599044e+01 1.0318077843755555e+02 8.0419002445999993e+01 44 0 0 -50 0
P 444 -24 6.4646967953734418e+00 3.4909545978243479e+01 5.7879920796889749e+01 1.0525098522544691e+02 8.0419002445999993e+01 62 0 0 -452 0
7
  • non-standard word-boundary pattern \b ? Supported via busybox, *BSD... Debian for decade(s). And the OP explicitly use ERE with -E switch Feb 12, 2023 at 19:09
  • @GillesQuenot Widely supported does not make \b standard. There are three competing ways to match word boundaries: \b...\b, \<...\>, and [[:<:]]...[[:>:]]. The -E option is standard though, which is why I'm not saying anything about it.
    – Kusalananda
    Feb 12, 2023 at 19:57
  • Do you know a case where regex don't support \b in real life? Feb 12, 2023 at 20:13
  • @GillesQuenot Plan9 grep, but as I said, being widely supported does not make it standard. The POSIX standard does not have \b as a pattern for world boundaries. In fact, it does not have any way to match a word boundary.
    – Kusalananda
    Feb 12, 2023 at 20:16
  • Yeah, not found it in opengroup Feb 12, 2023 at 20:18
3

um, you're relying on only that one field possibly having value 24. That seems like a bad design – there's a lot of tools that could just extract the first and third columns from such a file, for example cut, for further analysis.

Honestly, though: This is a job for a tool that deals with tabular numerical data, not for a string engine like sed or cut or grep.

In all honesty, a handful of lines of Python (or Perl, or… whatever programming language you like) would solve this; and: I'm 100% certain you'll deal with the results here programmatically anyways, so just use that language to begin with.

Just to illustrate:

#!/usr/bin/env python3
import sys

with open(sys.argv[1], "r", encoding="ascii") as inputfile:
  linecounter = 0
  for line in inputfile:
    linecounter += 1
    try:
      if not line.startswith("P "):
         continue # skip this line alltogether
      thirdpos = int(line.split(" ")[2])
      if not (thirdpos == 24 or thirdpos == -24):
         continue
      print(line) # or actually, do something useful with the line
    except Exception as e:
      print("Exception occurred on line {linecounter}, '{line}':\n{e}", sys.stderr)

in a text file, chmod 755 thatfile, then running /path/to/thatfile /path/to/input_file would do.

2

If you want to extract the full line and match -24/24 only in the third field then use awk:

$ awk '{if (($1=="P") && ($3==24 || $3==-24)) print}' infile

P 89 24 -1.5388040474568784e+01 7.4421775186012660e+00 -1.3143195543234219e+03 1.3168884860257754e+03 8.0419002445999993e+01 44 0 0 -97 0
P 493 -24 -1.4690576431881317e+01 7.3848907323212831e+00 -1.3144620647251766e+03 1.3170224315489374e+03 8.0419002445999993e+01 62 0 0 -499 0
P 8 24 7.0195398693654170e+00 3.1543502387874696e+01 5.5989200759599044e+01 1.0318077843755555e+02 8.0419002445999993e+01 44 0 0 -50 0
P 444 -24 6.4646967953734418e+00 3.4909545978243479e+01 5.7879920796889749e+01 1.0525098522544691e+02 8.0419002445999993e+01 62 0 0 -452 0
0
1

Another way to do it:

grep -P "P [^ ]+ -?24 " FILE 

which means grep for P, followed for a blank, some non blanks followed by another blank, optionally followed by a minus sign and 24, then again a blank.

If the blank might as well be a tab or more than one of both:

grep -P "P[\t ]+[^ \t]+[\t ]+-?24[\t ]+" FILE 

With -P while using gnu-grep, you enable Perl style regexes which allow to search for tab-characters by \t.

A leading caret isn't necessary, if the discriminating letter is always in column 1.

If column 2 is always positive, [0-9]+ is sufficient. -?[0-9] would work too for negative numbers.

4
  • 1
    [^ ] is just one non-blank, you need [^ ]+ to match one or more.
    – ilkkachu
    Feb 12, 2023 at 20:23
  • Yes, thanks, messed it up in the 1st command. Feb 12, 2023 at 21:32
  • 1
    Note that [\t ] might match a backslash, t, or space, depending on the system's regular expression library (FreeBSD, Ubuntu, Alpine Linux, and OpenBSD would do this). If you want to match a space or a tab, either use a literal tab character instead of \t, or use something like [[:blank:]] which matches a space or tab character in the C or POSIX locales.
    – Kusalananda
    Feb 12, 2023 at 22:24
  • Thanks @Kusalananda, I corrected my example to explicitly use the -P switch (perl style regex). Feb 13, 2023 at 17:50
0

The issue with the regular expression used in the grep command is that there is an extra space after the second set of digits in the pattern. For example, "P 89 24" will match but "P 89 -24" will not match because of the extra space.

To fix this, remove the extra space between the second set of digits and the next character in the pattern:

cat file2.txt | grep -E 'P [0-9]+ (24|-24) | P [0-9][0-9]+ (24|-24) | P [0-9][0-9][0-9]+ (24|-24) | P [0-9][0-9][0-9][0-9]+ (24|-24)' > file3.txt

This should match both "P ... 24" and "P ... -24" patterns.

Another way to simplify the regular expression is to use the optional "?" operator to match the "-" symbol:

cat file2.txt | grep -E 'P [0-9]+ -?24 | P [0-9][0-9]+ -?24 | P [0-9][0-9][0-9]+ -?24 | P [0-9][0-9][0-9][0-9]+ -?24' > file3.txt

This will match both "P ... 24" and "P ... -24" patterns with a single regular expression.

0

Put the content in the file file.txt, then: Work with grep:

grep -E 'P\s+\S+\s+([+-]24|\b24)\b' file.txt

For regex, try post it in regex101.com, with regex:

/^P\s+\d+\s+(24|\-24)\s+[\d\.\+\-e]+\s+[\d\.\+\-e]+\s+[\d\.\+\-e]+\s+[\d\.\+\-e]+\s+[\d\.\+\-e]+\s+\d+\s+\d+\s+\d+\s+\-?\d+\s+\d+$
/gm

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .