sed (or awk): print captured group or placeholder if it doesn't exist

Question

So since GitHub removed the insights tab for non-premium accounts, I'm trying to locally list insertions and deletions to my git repository per day.

I figured out this way of printing what I want:

git log --pretty="@%ad" --date=short --shortstat |  tr "\n" " " | tr "@" "\n"

That produces this kind of output:

2024-06-13   7 files changed, 400 insertions(+), 406 deletions(-) 
2024-06-12   3 files changed, 145 insertions(+) 
2024-06-12   5 files changed, 638 deletions(-) 
2024-06-12   1 file changed, 1 insertion(+), 1 deletion(-) 

Notice the plurals in file(s), insertion(s) and deletion(s). Another problem is that a commit might not have insertions or deletions (or both, but let's ignore this case).

So I'm almost there, I just need to extract the date, insertions and deletions and group by date. That will produce some sort of "work done per day" graph.

I made this regex to capture the fields dealing with all the optionals:

/^([0-9]{4}-[0-9]{2}-[0-9]{2})\s{3}[0-9]+\sfile(s)?\schanged,\s(([0-9]+)\sinsertion(s)?\(\+\))?(,\s)?(([0-9]+)\sdeletion(s)?\(\-\))?\s$/gm

Now I need to get the 1st, 4th and 8th groups, for instance with sed:

echo "2024-06-13   7 files changed, 400 insertions(+), 406 deletions(-) " |
    sed -r 's/^([0-9]{4}-[0-9]{2}-[0-9]{2})\s{3}[0-9]+\sfile(s)?\schanged,\s(([0-9]+)\sinsertion(s)?\(\+\))?(,\s)?(([0-9]+)\sdeletion(s)?\(\-\))?\s$/\1 \4 \8/gm'

That produces the correct output:

2024-06-13 400 406

But if the input string doesn't have insertions or deletions, sed just prints nothing for that captured group. Eg.:

2024-06-13 400

And I have no way to tell if the single number are insertions or deletions.

Is there anyway to extract the groups from each line, but print a "0" as placeholder if the group doesn't exist? (not necessarily with sed alone, and not necessarily in a single command).

Answer 1

There are always many solutions possible. Here is one. I like a single regexp that does everything, but sometimes it is best to keep it simple and step by step:

sed -r '
/^([0-9]{4}-[0-9]{2}-[0-9]{2}) /{
  s/$/XXX C0 I0 D0/
  s/([0-9]+)( files? changed.*)C0/\2C\1/
  s/([0-9]+)( insertion.*)I0/\2I\1/
  s/([0-9]+)( deletion.*)D0/\2D\1/
  s/ .*XXX//
  s/[CDI]//g
}
'

This matches the date, then adds to the end of the line the placeholder string "XXX C0 I0 D0". The next 3 commands then replace the appropriate 0 by the real number of changed, inserted, or deleted items. The whole line from the end of the date until the placeholder is removed, and the C, D, and I letters are removed, leaving just the updated numbers attached to them.

Answer 2

If the intermediate placeholder is empty you'll have a double space, which you can fill with a zero. Likewise if you have no last item your line will end with a space.

echo "2024-06-13   7 files changed, 400 insertions(+), 406 deletions(-) " |
    sed -r \
        -e 's/^([0-9]{4}-[0-9]{2}-[0-9]{2})\s{3}[0-9]+\sfile(s)?\schanged,\s(([0-9]+)\sinsertion(s)?\(\+\))?(,\s)?(([0-9]+)\sdeletion(s)?\(\-\))?\s$/\1 \4 \8/gm' \
        -e 's/  / 0 /' -e 's/ $/ 0/'

Answer 3

It might be more reliable if you use --numstat and calculate the required stats yourself. Using --numstat (e.g., in git log --numstat --pretty="@%ad" --date=short) results in output like:

@2024-06-07

1       1       plugin/supertab.vim
@2024-06-07

2       34      plugin/supertab.vim
@2024-06-06

7       1       plugin/supertab.vim
@2021-04-22

4       0       .gitignore
17      0       Makefile
210     0       README.rst
473     0       doc/supertab.txt
61      0       ftplugin/html.vim
45      0       ftplugin/xml.vim
1144    0       plugin/supertab.vim

Then you can use something like the following awk code:

/^@[0-9]+-[0-9]+-[0-9]+/ { # new commit info starting with `@<date>`
  if (date) {  # print previous commit's stats
    print date, inserts, deletes
  };
  date = $0;
  getline;
  inserts = deletes = 0;  # reset counts
  next
}
{
  inserts += $1;
  deletes += $2;
}
END {
  if (date) {  # Final (initial?) commit
    print date, inserts, deletes
  }
}

Example output:

% git log --numstat --pretty="@%ad" --date=short | awk '/^@[0-9]+-[0-9]+-[0-9]+/ {if (date) {print date, i, d}; date=$0; getline; i=d=0; next} {i+=$1; d+=$2; } END {if (date) {print date, i, d}}' | head
@2024-06-07 1 1
@2024-06-07 2 34
@2024-06-06 7 1
@2021-04-22 1954 0

If you want to combine the stats for all commits on the same day, you can instead use something like:

if (date == $0) {
  getline; next
}
if (date) {
  # print previous commit's stats
  #...
}

So:

% git log --numstat --pretty="@%ad" --date=short | awk '/^@[0-9]+-[0-9]+-[0-9]+/ {if (date == $0) { getline; next } if (date) {print date, i, d}; date=$0; getline; i=d=0; next} {i+=$1; d+=$2; } END {if (date) {print date, i, d}}' | head
@2024-06-07 3 35
@2024-06-06 7 1
@2021-04-22 1954 0

Answer 4

We can insert 0 insertions after "changed" if it's missing:

/insertion/!s/changed,/& 0 insertions(+),/

Similarly, we can append 0 deletions if that's necessary:

/deletion/!s/$/, deletions(-)/

---

Complete script

Pipe the output of git log --pretty=%as --shortstat into this small program (no need for @ in format or for tr commands):

#!/usr/bin/sed -Ef

# Store date line in hold space
/^[0-9]{4}-[0-9]{2}/h

# Ignore everything except shortstat lines
/changed/!d

# Add insertions if absent
/insertion/!s/changed,/& 0,/
# Add deletions if absent
/deletion/!s/$/, 0/

# Reformat
s/.* changed,//
s/[^0-9,]+//g

# Prepend the saved date
H
x
y/,\n/  /

Answer 5

Using a single call to any awk in any shell on every Unix box, this

git log --pretty="@%ad" --date=short --shortstat |
awk '
    sub(/^@/,"") { date = $0 }
    gsub(/s?\([+-]),?/,"") {
        delete val
        val[$5] = $4
        val[$7] = $6
        print date, val["insertion"]+0, val["deletion"]+0
    }
'

will produce this output:

2024-06-13 400 406
2024-06-12 145 0
2024-06-12 0 638
2024-06-12 1 1

Here's a commented version of the above script to help explain what it does:

sub(/^@/,"") {          # Try to remove a leading `@` from the current line.
                        # If that succeeded then a leading `@` was present so
                        # this must be a date line so enter this subsequent
                        # action block.

    date = $0           # Save the date, i.e. whats left on the line once
                        # the above `sub()` removed the leading `@`.
}

gsub(/s?\([+-]),?/,"") {#  Try to remove a string that contains `(+)` or `(-)`
                        # optionally preceded by `s` or followed by `,` so
                        # `insertion(+)`, `insertions(+),` or similar become
                        # `insertion` and `deletions(-),` etc. become `deletion`.
                        # If that succeeded then this is a line containing
                        # insertion and deletion values to process them.

    delete val          # Remove the `val` array to ensure no it cannot contain
                        # leftover valus from processing the previous line.

    val[$5] = $4        # $5 contains `insertion` or `deletion` or is empty
                        # so save the mapping from that to the value thats
                        # in $4 so we have e.g. `val["insertion"] = 400`.

    val[$7] = $6        # Ditto for $7 and $6.

    print date, val["insertion"]+0, val["deletion"]+0
                        # Print the saved values in the order we want them
                        # to be output. Adding `0` means if `insertion` and/or
                        # deletion were missing from the input and so `val[]`
                        # wasnt populated for them on this line, wed still
                        # print `0` instead of a null string.
}

I see from your answer though that you actually want to sum the values per date - if so then this, again using any awk:

git log --pretty="@%ad" --date=short --shortstat |
awk '
    sub(/^@/,"") {
        date = $0
        if ( (date != prev) && (prev != "") ) {
            print prev, val["insertion"]+0, val["deletion"]+0
            delete val
        }
        prev = date
    }
    gsub(/s?\([+-]),?/,"") {
        val[$5] += $4
        val[$7] += $6
    }
    END {
        print prev, val["insertion"]+0, val["deletion"]+0
    }
'

will produce this output:

2024-06-13 400 406
2024-06-12 146 639

without having to store all values in memory at once.

The above assumes that this:

git log --pretty="@%ad" --date=short --shortstat

produces output like this:

@2024-06-13

 7 files changed, 400 insertions(+), 406 deletions(-)

@2024-06-12

 3 files changed, 145 insertions(+)

@2024-06-12

 5 files changed, 638 deletions(-)

@2024-06-12

 1 file changed, 1 insertion(+), 1 deletion(-)

Answer 6

Complete solution based on @meuh 's regex, but ignoring the number of changed files:

git log --pretty="@%ad" --date=short --shortstat |  tr "\n" " " | tr "@" "\n" |
 tail -n +2 |
 sed -r '
/^([0-9]{4}-[0-9]{2}-[0-9]{2}) /{
  s/$/XXX I0 D0/
  s/([0-9]+)( insertion.*)I0/\2I\1/
  s/([0-9]+)( deletion.*)D0/\2D\1/
  s/ .*XXX//
  s/[DI]//g
}' |
 awk '{a[$1]+=$2; b[$1]+=$3} END { for (i in a) print i,a[i],b[i] }' |
 sort -k 1

The tail command in the second line is needed to remove a first empty line that the tr replacements introduce in the first line.

After that, the sed command removes everything except date, inserted lines and deleted lines, accounting for commits whith no insertions or no deletions.

Then the awk command sums the 2nd and 3rd columns grouping by the first column (date). But because awk's array keys are not ordered, we need to sort the result with a final sort by the first column, which produces the list of unique daily stats in descending order by date, eg:

2024-05-29 1234 2
2024-05-30 350 76
2024-06-03 405 68
2024-06-06 483 36
2024-06-11 447 59
2024-06-12 795 12
2024-06-13 522 436
...

Answer 7

In Perl, this would be relatively simple, since you can either use integer formatting %d with printf, which will print empty values as zeroes, and/or you can use the // operator to set a "default" value:

% git ... | perl -0777 -ne '
    while (/^([0-9]{4}-[0-9]{2}-[0-9]{2})\s{3}[0-9]+\sfile(s)?\schanged,\s(([0-9]+)\sinsertion(s)?\(\+\))?(,\s)?(([0-9]+)\sdeletion(s)?\(\-\))?\s$/gm) {
        printf "%s %d %d\n", $1, $4//0, $8//0;
    }'  
2024-06-13 400 406
2024-06-12 145 0
2024-06-12 0 638
2024-06-12 1 1

or with s///e, which allows Perl code in the replacement part:

% git ... | perl -0777 -pe 's#^([0-9]{4}-[0-9]{2}-[0-9]{2})\s{3}[0-9]+\sfile(s)?\schanged,\s(([0-9]+)\sinsertion(s)?\(\+\))?(,\s)?(([0-9]+)\sdeletion(s)?\(\-\))?\s$# sprintf("%s %d %d", $1, $4//0, $8//0) #gme'
2024-06-13 400 406
2024-06-12 145 0
2024-06-12 0 638
2024-06-12 1 1

Though, as muru already mentioned, the output from --numstat might be easier to parse.

---

On the other hand, it may just be easier to pick each value separately, that would help avoid the gory regex. E.g. something like this, which does the daily summing at the same time:

% git ... | perl -ne '
    next unless /^([-0-9]+)\s+(\d+) files? changed/;
    $date = $1;
    $ins{$date} += $1 if /(\d+) insert/;
    $del{$date} += $1 if /(\d+) delet/;

    END {
        printf "date: %s total insertions: %d deletions: %d\n", $_, $ins{$_}, $del{$_} for sort keys %ins;
    }
'

date: 2024-06-12 total insertions: 146 deletions: 639
date: 2024-06-13 total insertions: 400 deletions: 406

Incidentally, that's rather easy to adjust so that the tr isn't needed, just set @ as the record separator:

git log --pretty="@%ad" --date=short --shortstat | perl -ne '
    BEGIN { $/ = "@" }
    next unless /^([-0-9]+)\s+\d+ files? changed/;
    $date = $1;
    $ins{$date} += $1 if /(\d+) insert/;
    $del{$date} += $1 if /(\d+) delet/;
    
    END {
        printf "date: %s total insertions: %d deletions: %d\n", $_, $ins{$_}, $del{$_} for sort keys %ins;
    }
'

(Note that the \s in your regex is Perl-ism. While GNU supports it for some reason, it's nonstandard and not supported by the standard tools on other systems.)