2

I'm trying to find a way to grep information in the file that located in the different lines and one of the pattern is contain pattern of date and time.

below is the input

unwantedtext unwantedtext unwantedtext unwantedtext 8/1/2022 6:15 (1st required pattern using date and time format and the date and time can be changed)

unwantedtext unwantedtext unwantedtext unwantedtext 
unwantedtext unwantedtext INC-220721-00007628 (2nd required pattern)

also in the second file some the second pattern not always located in the 3rd lines, however definitely located in the different lines from the date and time information.

second patterns always have INC in the beginning and then followed by the year month and date info

INC

Always have this INC in the first 3 Digit

INC-YYMMDD

After INC followed by Date and time

INC-YYMMDD-00000000

after YYMMDD followed by random 8 digits.

-

Hypen Char is always as separator for the 2nd pattern

and I'm expecting the following output

INC-220721-00007628,8/1/2022 6:15 

for testing I'm using windows CYGWYN and for the large set of file I'm using CENTOS 7

thank you in advance for all the suggestion

3
  • Please edit your question and clarify what you need. What is the actual pattern and what can change? Will the date always be 8/1/2022 6:15? Do you need to find any date and time? How far into the future or past? And how do we recognize the next pattern? Will it always be INC- followed by two sets of numbers separated by a -? Will it always be that exact number? Please edit and tell us what can change and what will be always the same so we can use it to find what you need. Finally, please also tell us what operating system you are using.
    – terdon
    Aug 10 at 10:05
  • @terdon thanks, already editing my question and the detail that missing in the first place. Aug 10 at 10:21
  • I think asking for a solution for "arbitrary mess" (that's how those lines look for me) is not of general interest, so maybe explain what it's all about. Most likely grep is simply the wrong tool to use.
    – U. Windl
    Aug 10 at 21:46

5 Answers 5

4

Assuming the values are always in pairs, you could use grep to find them and paste to combine them:

$ grep -oP '\b\d{1,2}/\d{1,2}/\d{4} \d{1,2}:\d{2}|INC-\d{6}-\d{8}\b' file | 
  paste -d "," - - 
8/1/2022 6:15,INC-220721-00007628
2

You can't do this with grep alone. You might be able to do it in sed but it'll be a PITA and require far more knowledge about pattern and hold spaces than you probably have (and such knowledge generally isn't worth learning because it's much easier to do in other languages).

In short, you'll need to use or . For example:

$ perl -lne 'BEGIN { $, = "," };

             if (m=(\b\d{1,2}/\d{1,2}/\d{4} \d{1,2}:\d{1,2}\b)=) {
               $dt = $1
             } elsif (m/(INC-\d+-\d+)/) {
               print $1, $dt;
               #$dt = ""; # uncomment to clear $dt before next input line
             }' input.txt 
INC-220721-00007628,8/1/2022 6:15

This perl one-liner script uses perl's -l option to enable automatic handling of line-endings while reading the input and printing output (e.g. \n on unix or \r\n on windows), the -n option to make perl work like sed -n. The -e option indicates that the next argument is the script to be run by perl.

First, this script sets the output field separator ($,) to a comma. This variable is documented in man perlvar. It does this in a BEGIN {...} block so that it runs only once, when the script is started, rather than once for each input line read.

BTW, you can optionally use perl's English module if you don't like or can't remember the cryptic single-character variable names (with use English; inside the script or -MEnglish with a one-liner), this gives you access to both long descriptive English aliases and awk-like equivalents where appropriate for the short variables. e.g. with use English you can use either $,, $OUTPUT_FIELD_SEPARATOR, or the awk-like $OFS - they all mean the same thing and refer to the same variable.

The script uses perl's regex match operator m to match (and capture, due to the parentheses in the regex) the required patterns. See man perlop and search for "m/PATTERN". The first time I use m, I'm using = as the regex delimiter so I don't have to escape the /s in the date pattern. The second-time, I'm using the more familiar /.

See also the man pages for perlre, as well as perlrequick and perlretut.

For each input line read, it tries to match the required date and time pattern and, if successful, stores the captured date and time (from perl's $1 sub-pattern match variable, which is similar to \1 in sed - see man perlvar and search for the section headed "Variables related to regular expressions") into a variable called $dt.

If the previous match was not successful, it tries to match the INC-\d+-\d+ pattern. If successful, it prints the captured pattern and the $dt variable.

All other input is ignored.

or with awk:

$ awk -v OFS=, '
  match($0,/\<[[:digit:]]{1,2}\/[[:digit:]]{1,2}\/[[:digit:]]{4} [[:digit:]]{1,2}:[[:digit:]]{1,2}\>/,a) {
    dt = a[0]; next
  };
  match($0,/INC-[[:digit:]]+-[[:digit:]]+/,a) {
    print a[0], dt
  }' input.txt 
INC-220721-00007628,8/1/2022 6:15

awk has a handy -v option to set awk variables, so we don't need a BEGIN block to set OFS.

This awk one-liner is pretty much a direct translation of the perl version, but uses awk's match() function to do the match-and-capture test. It capture any matches into array a.

It also uses [[:digit:]] which is the equivalent to perl's \d. In many locales, you can get away with using [0-9] instead, but both [[:digit:]] and perl's \d will work in any locale.

6
  • this is worked like charm, would you please also help to explain your detail script? also if I want to put the input as variable how can I change input.txt? I tried to change as $0 not working, and $1 is not working either... Aug 10 at 11:22
  • awk solution also perfect, and using this I can place $1 as an input instead of put the input file manually. Aug 10 at 11:28
  • input.txt is a placeholder for this example - you can use any filename(s), or just pipe your data into either the awk or perl version. If, by "place $1 as an input" you mean the first argument to a shell script, you can replace input.txt with $1 for either the awk or perl version. Remember to double-quote it ("$1"), just in case it contains problematic characters like white-space or globs.
    – cas
    Aug 10 at 11:33
  • Hi @cas, thanks a lot for the solution. I have tried to add the 3rd pattern by adding the script as follow but it did not worked, any suggestion and help to revise the following code? 'awk -v OFS=, ' match($0,/\<[[:digit:]]{1,2}\/[[:digit:]]{1,2}\/[[:digit:]]{4} [[:digit:]]{1,2}:[[:digit:]]{1,2}\>/,a) {dt = a[0]; next}; match($0,/3rd pattern/,b); match($0,/INC-[[:digit:]]+-[[:digit:]]+/,a) {print a[0], dt; print b[0]}' input.txt' Aug 30 at 4:12
  • Is it possible that your third pattern is on the same line as the first pattern? If so, the first match will extract it and then skip to the next input line. I'd suggest either reworking the script entirely to suit the new requirements or, maybe, adding something like if (match($0,/third pattern/,b) { b0 = b[0] } inside the code block for the first match (before the next statement). Then print b0. Remember to set b0 to an empty string after printing it so that it doesn't persist on following lines (or set b0 to a default value in a BEGIN{} block, and reset it to that value after printing)
    – cas
    Aug 30 at 4:56
2
grep -Eo 'INC-[[:digit:]]{6}-[[:digit:]]+|[[:digit:]]{1,2}/[[:digit:]]{1,2}/[[:digit:]]{4} [[:digit:]]{1,2}:[[:digit:]]{1,2}' inputfile.txt |
 tr '\n' ',' | 
 sed 's/,$//'

Notes:

  • grep:
    • -o: get only matched expressions
    • -E: accept extended RE
      • The patterns are separated by a pipe (|), boolean operator "or"

Result is :

8/1/2022
INC-220721-00007628

The order is the same than into the input file.

  • tr ...: translate line feed (\n) to coma (,)

Result is:

8/1/2022 6:15,INC-220721-00007628,
  • sed ...: search and replace/remove last coma added by tr command

Result is:

8/1/2022 6:15,INC-220721-00007628
1

Using any awk in any shell on every Unix box:

$ awk -v OFS=',' '
    match($0,"[0-9]+/[0-9]+/[0-9]+ [0-9]+:[0-9]+") { dt = substr($0,RSTART,RLENGTH) } 
    match($0,/INC-[0-9]+-[0-9]+/) { print substr($0,RSTART,RLENGTH), dt }
' file
INC-220721-00007628,8/1/2022 6:15

I'm using neither the 3rg arg to match(), nor word boundaries, nor RE intervals, nor character classes as those aren't portable to all awks. For this task you don't need the 3rd arg to match(), nor do you need character classes, nor RE intervals, and you almost certainly don't need word boundaries to resolve potential false matches on similar-looking substrings but if you do then there are simple ways to implement the equivalent functionality in any awk on a case by case basis, e.g. , e.g. match($0,"(^|[^0-9])[0-9]+/[0-9]+/[0-9]+ [0-9]+:[0-9]+([^0-9]|$)") { dt = substr($0,RSTART,RLENGTH); sub(/^[^0-9]/,"",dt); sub(/[^0-9]$/,"",dt) }

1

With pcregrep's multi-line mode:

$ pcregrep -M -o2 -o1 --om-separator=, '(?s) (\d+/\d+/\d+ \d+:\d+) .*?(INC-\d{6}-\d{8})' your-file
INC-220721-00007628,8/1/2022 6:15
2
  • Why (?s) and .*? are used? I would be very thankful for that. And that -on can be used without muli line mode. Nov 4 at 14:29
  • 1
    @PrabhjotSingh, (?s) enables the s flag like in perl's /regexp/s without which . doesn't match on newline characters. .*? matches 0 or more characters but as few as possible so that it's the first occurrence of INC-... after the timestamp that is reported, not the last in the file. See man pcrepattern for the PCRE syntax (which emulates that of perl, see perldoc perlre for the perl regexps). Nov 4 at 14:34

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .