1

I have a datafile, with dates in 2nd column

# cat datafile
-;20210106;-;-;-;
-;20210112;-;-;-;
-;20210112;-;-;-;
-;20210112;-;-;-;
...
-;20210217;-;-;-;
-;20210217;-;-;-;
-;20210217;-;-;-;
-;20210217;-;-;-;
-;20210219;-;-;-;
-;20210219;-;-;-;
-;20210221;-;-;*20210219*;

the hyphen '-' represent random text data, and dots '...' represent more lines of data, '*' represents random text in same column. All I want is data between 20210112 & 20210219 based on 2nd column.

I want to avoid sed/grep as both will grep similar patterns in other columns as well.

# sed -n '/20210112/,/20210219/p' datafile
-;20210112;-;-;-;
-;20210112;-;-;-;
-;20210112;-;-;-;
...
-;20210217;-;-;-;
-;20210217;-;-;-;
-;20210217;-;-;-;
-;20210217;-;-;-;
-;20210219;-;-;-;
-;20210219;-;-;-;
-;20210221;-;-;*20210219*;

It will match some other text as well in other non-relevant lines. So, I guess AWK is a better candidate here, but I noticed awk prints only between 1st match of 1st pattern to 1st match of 2nd pattern

# awk -F';' '$2 ~ /20210112/,$2 ~ /20210219/' datafile
-;20210112;-;-;-;
-;20210112;-;-;-;
-;20210112;-;-;-;
...
-;20210217;-;-;-;
-;20210217;-;-;-;
-;20210217;-;-;-;
-;20210217;-;-;-;
-;20210219;-;-;-;

Whereas, I'd like to bring all lines till last match of 2nd pattern.

Desired

-;20210112;-;-;-;
-;20210112;-;-;-;
-;20210112;-;-;-;
...
-;20210217;-;-;-;
-;20210217;-;-;-;
-;20210217;-;-;-;
-;20210217;-;-;-;
-;20210219;-;-;-;
-;20210219;-;-;-;
4
  • Is it ensured that the stop date is actually present in the data, or could it be that you want to print everything until 20210219 but the entries "jump" from 20210217 directly to 20210220, making the entries with 20210217 the last things to pring?
    – AdminBee
    Nov 15, 2021 at 10:20
  • 2
    It's not that awk prints only between ...., it's that the awk script you wrote does that, other awk scripts will do different things.
    – Ed Morton
    Nov 15, 2021 at 12:59
  • You haven't stated what to do in the most common use cases for script like this, e.g. what if the start and/or end date isn't present in the input but dates within the range delimited by them are? What if the start date is but not the end date? What if the end date is but not the start? What if neither but there are dates within the range delimited by them? Please edit your question to state your requirements for more than just the sunny day case where both the start and end dates are present.
    – Ed Morton
    Nov 16, 2021 at 15:44
  • @EdMorton I usually would put date range that is in file. And thanks for your help. Your oneliner is what I needed.
    – Sollosa
    Nov 17, 2021 at 6:11

6 Answers 6

6

This is how I would do it:

BEGIN {FS = ";"}

$2 == 20210112 {capture = 1}
capture == 1   {buffer = buffer $0 "\n"}
$2 == 20210219 {printf ("%s", buffer); buffer = ""}

After seeing the first occurrence of the first pattern, it starts putting lines into a buffer. For each occurrence of the second pattern, it prints the buffer, and resets the buffer to an empty string again.

1
  • The OP didn't state their needs but usually in a case like this if the ending date isn't present the requirement is to print from the first date to the end of the file which your script wouldn't do. Easy enough to handle by adding END{printf "%s", buffer if that's what's required though.
    – Ed Morton
    Nov 16, 2021 at 15:34
4

If we ignore the ... line and assume all of your dates are in increasing $2 order as shown otherwise in your example then all you need is:

$ awk -F';' '(20210112 <= $2) && ($2 <= 20210219)' file
-;20210112;-;-;-;
-;20210112;-;-;-;
-;20210112;-;-;-;
-;20210217;-;-;-;
-;20210217;-;-;-;
-;20210217;-;-;-;
-;20210217;-;-;-;
-;20210219;-;-;-;
-;20210219;-;-;-;

or more efficiently since it exits once past the range:

$ awk -F';' '20210112 <= $2{f=1} $2 > 20210219{exit} f' file
-;20210112;-;-;-;
-;20210112;-;-;-;
-;20210112;-;-;-;
...
-;20210217;-;-;-;
-;20210217;-;-;-;
-;20210217;-;-;-;
-;20210217;-;-;-;
-;20210219;-;-;-;
-;20210219;-;-;-;

That above also assumes that if the end date is absent from the input then you want to print from the start date to the end of the file and if the start date is absent then you want to print from the first date greater than the start date to the end date, etc.

3

Determining when the last occurrence of a pattern has been seen typically requires remembering lines before that and only outputting them once it has been determined that no further occurrence of the pattern will be seen.

To avoid having to do that manually, you can chop off the start of the file, reverse it, chop off the new start of the file, and reverse it again:

awk -F\; '$2 == 20210112,0' | tac | awk -F\; '$2 == 20210219,0' | tac
0
2

Assuming you have a string date ordering, i.e. there can be no "stray" 20210220 between two 20210219 lines, the following will work:

awk -F';' -v start="20210112" -v stop="20210219" '$2==start{p=1} $2==stop{p=0} p||$2==stop' input.dat

This will pass the start and stop dates as awk variables start and stop to the program. Once we find the start date, we set the flag p (for "print") to 1. If we find the stop date, we reset it to 0. Lines will be printed as long as either p is 1, or the current line date is equal to the stop date. That way, we print until the last occurence of 20210219.

Note that this will only work if both the start and stop dates are explicitly found in the file. If not, you can make use of the fact that your dates are in ISO order:

awk -F';' -v start="20210112" -v stop="20210219" '$2>=start&&$2<=stop' input.dat

This will print as long as the second field is equal or greater than the start date, and equal or lesser than the stop date. This will work with dates formatted the way you showed, but not in case of $2 content that cannot be ordered easily using the < or > comparison operators.

2

You can do it in awk, you just need a slightly more complicated approach. Note the use of == instead of ~, this is to avoid matching cases where the field contains 20210112 or 20210219 as a substring (e.g. 20210219123):

$ awk -F';' '$2==20210112{a=1}; $2==20210219 && a{b=1} a && b && $2!=20210219{exit}; a ' datafile 
-;20210112;-;-;-;
-;20210112;-;-;-;
-;20210112;-;-;-;
...
-;20210217;-;-;-;
-;20210217;-;-;-;
-;20210217;-;-;-;
-;20210217;-;-;-;
-;20210219;-;-;-;
-;20210219;-;-;-;

Or, more verbosely, but easier to understand:

$ awk -F';' '{
              if($2==20210112){ a=1 }
              if($2==20210219 && a){ b=1 } 
              if(a && b && $2!=20210219){ exit }; 
              if(a){ print }
             }' datafile 

Alternatively, just do a numerical comparison:

awk -F';' '$2>=20210112 && $2<=20210219' datafile 

As an aside, you can still use sed for things like this if you have to, just by anchor the pattern so it only matches in the second field:

$ sed -n '/^[^;]*;20210112/,/^[^;]*;20210219/p' datafile

Not useful in this case since it will stop at the first match, but at least it won't match on other fields.

0
1

I want to avoid sed/grep as both will grep similar patterns in other columns as well.

Well no, not if you do it right. Example:

sed -n '/^[^;]*;20210112;/! d; :1; H; /^[^;]*;20210219;/ { s/.*//; x; s/^.//; p }; n; b1' \
  datafile

That is modeled on @Abigail's answer. It skips all lines up to the first match to the range start (/^[^;]*;20210112;/! d). Then it accumulates lines in sed's hold space, printing the accumulated lines and clearing the hold space each time a line matching the range end is encountered. It selects the specific wanted field by matching the right number (1) of preceding field delimiters.

awk is cleaner for this because it has a built-in sense of separating input lines into fields, and because it provides for user-defined variables. And awk generally affords code that is more human-readable. Nevertheless, sed can do it.

On the other hand grep cannot do it by itself at all. It can discriminate between matches in the second field and those elsewhere, but it has no way to remember or take account of any matches to previous lines.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.