I am trying to read a text file that has the following data:

Smith Jane Chem  Date:[somedate]
Doe Joanne Victoria ENG Date:[today]

In this example [somedate] means any date that is not today, [today] means today's date. The idea is that I use awk to filter out all the students that register today. The Date format would look like this: 2022-06-21.

Any help would be appreciated

  • 1
    If it doesn't have to be awk, grep is a better choice: grep -v $(date +'%F') filename
    – MDeBusk
    Sep 22 at 16:47
  • 1
    @EdgarMagallon "To filter" and "to filter out" are equivalent in English. "The "out" is redundant because filtering implies removing something from something.
    – MDeBusk
    Sep 22 at 17:44
  • 1
    @EdgarMagallon If I were to ask for only those students who registered today, I'd ask that they be filtered FOR, not OUT. I'm wondering if I've misunderstood the intend of the questioner.
    – MDeBusk
    Sep 22 at 21:42
  • 1
    @EdgarMagallon I think "filter by" denotes something different than "filter for," but that may be idiomatic to me. I'd say "filter by the contents of the third field" (or some variable or type) and "filter for today's date", that is to say, something specific.
    – MDeBusk
    Sep 22 at 22:48
  • 1
    @EdgarMagallon There are several articles that could be applied after the work filter. The general goal of a filter is to differentiate some particular attribute of what is being filtered. Thinking of rocks, for example, you can filter by size, or filter by color, or filter by weight. Then, after applying some filter the requested output could be to accept that which is being filtered (or to filter in) or could be to reject what is being filtered (or to filter out). Sep 23 at 5:20

4 Answers 4

grep -v "Date:$(date +%F)\$" < file

Would return the lines that don't end in Date: followed by the current date in YYYY-MM-DD format.

If using zsh, you can do that without having to invoke date with:

grep -v Date:${(%):-%D{%F}}$ < file

The GNU implementation of awk can also get the current date by itself:

gawk 'BEGIN {search = strftime("Date:%F")}
      $NF != search' < file

Would return the lines whose last blank delimited field is not Date:[today].

Change to:

gawk 'BEGIN {search = strftime("Date:%F")}
      NR == 1 || $NF != search' < file

To also print the header.

On systems where awk is not the GNU awk, you can do:

awk -v search="$(date +Date:%F)" '
  NR == 1 || $NF != search' < file


awk 'BEGIN {"date +Date:%F" | getline search}
     NR == 1 || $NF != search' < file

(though that one runs an extra shell invocation to interpret that date +Date:%F shell code; that approach would only be useful in a #! /usr/bin/awk -f script).

If the input can be parsed as some sort of csv with space as separator, you could also use csvkit's csvgrep:

<file csvgrep -d ' ' -c DATE -r "$(date +'^Date:%F$')"

Here matching on the DATE field.

The output is comma-separated-value though. You can change the delimiter back to space instead of comma by piping to csvformat -D ' '.

  • Why are you escaping the $? What grep/shell combination will choke on grep "...$"? I am guessing there is a condition where the $" is treated as a variable by the shell?
    – terdon
    Sep 22 at 17:24
  • @terdon, IIRC, POSIX allows $" to be handled as a special parameter, so "whatever$" is unspecified per POSIX, while "whatever\$" is fully specified (to expand to whatever$). In practice though, I don't know of any shell where the backslash is necessary (note that ksh93 and bash have a $"..." form of quote that makes use of the fact that POSIX leaves $" unspecified). More generally, you want to escape $ so it be taken literally, outside of single quotes. Sep 22 at 18:08
  • First off thank you! I had to update the example. Sep 22 at 18:24
  • Thank you for helping, but I only have access to awk not gawk. Sep 22 at 20:09
  • 1
    Thanks @OlivierDulac, the initial version of the answer filtered in instead of out, and it looks like I missed some bits when reversing. Fixed now. Sep 23 at 13:26

Get current date with with command substitution and compare $NF:

awk -F, -v date="$(date +%F)" '$NF == date' FILE

This works for me:

awk -F'[ :]' -v dt="$(date +'%F')" '$NF != dt' filename
  • -F'[ :]' tells awk to use either the space or the colon as the field separator
  • -v tells awk to use the following as a variable
  • dt=" $(date +'%F')" sets dt to the current date in YYYY-MM-DD format
  • '$NF != dt' says "ignore this line if the last field equals the current date"
  • @user9980782 Your alteration to the file format in your question is so severe that most of the answers don't work any longer. I've edited my answer to reflect this.
    – MDeBusk
    Sep 22 at 21:27
  • Thanks, when I run this I get: Smith Jane Chem Date:[somedate] Doe Joanne Victoria ENG Date:[today] [blank line] It doesn't filter just prints all the data Sep 23 at 12:43
  • @user9980782 : is the line in the log files always : ...... yyyy-mm-dd ? (with a space before yyyy) ? which is what this answer is looking for? or if your file looks different please show a true exemple Sep 23 at 13:19
  • @user9980782 I copied and pasted your example data file, replaced [somedate] with 2022-09- and an arbitrary two-digit number < 30, and replaced [today] with the output of $date +'%F'. It worked for me. I'm running GNU awk 5.1.0.
    – MDeBusk
    Sep 23 at 16:14

In general, to filter the last field in AWK, you should use $NF == "text" or, to make it even more general: $NF == value where value is a variable with the appropriate content. That could be applied to any field by changing $NF to $3, for example (to filter on column 3). That (the ==) would filter in (accept) all students that registered today. To reject them, you can use $NF != value or !($NF == value) both provide the same logical result.

The value to match could be [today] or Date:[today] or Date:2022-06-21 or anything that is required. Just make the content of the variable value exactly equal to what you need and this awk command will "filter out" lines with that value. Assuming the field separator is a tab:

awk -v value="Date:[2022-06-21]" '$NF != value' infile

Add a -F"," if the field delimiter is a comma.

To get today's date you can use the shell if using bash (4.3+) or ksh (93):

printf -v value 'Date:[%(%F)T]'
awk -v value="$value" '$NF != value' infile

If the first line of the script is a header, and you want it to be printed, add:

printf -v value 'Date:[%(%F)T]'
awk -v value="$value" '($NF != value) || (NR==1)' infile

Or, in any shell:

value="Date:[$(date +'%F')]"
awk -v value="$value" '($NF != value) || (NR==1)' infile

In some versions of awk, it is possible to get the date directly:

awk 'BEGIN{ value = "Date:[" strftime("%F") "]" } ($NF != value) || (NR == 1)' infile

The == would make a string comparison which needs to be exact. A related test is a regex match ~ (or the negated !~) which could allow less strict matches ($NF ~ /2022/ will match any string that contains a 2022 anywhere, which could probably be used to match "this year", for example).

  • when I run it I get: awk: calling undefined function strftime source line number 1 Sep 24 at 23:04
  • What do you mean by it? That you are trying the last option and it fails? Well, the line just above declares that this option works only in some versions of awk, not all. Have you tried other options from the answer? The before-last solution should work correctly in any awk. Sep 25 at 1:46
  • Hi when I copy and past the command you wrote w/my file that's it. Sep 27 at 0:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.