4

I want to track the number of additions and deletions that exclude moved lines. So if a commit has 10 additions, 5 deletions, and 3 moved lines, then I have 7 additions, 2 deletions excluding moved lines. The 10 and 5 are given by the following code. I need to generate the 3 (I only need weak moved line detection, e.g. any line that appears identically as simultaneously deleted from one location and added to another location within the same commit).

I am using the following to track number of additions and deletions for an important file in my git repository.

git log --since=2014-08-01 --date=short --pretty=format:"%ad%x09" --numstat -- file.tex

This yields the following, where the first number is additions and the second number is deletions.

2014-08-19      
72      0       file.tex

2014-08-19      
211     290     file.tex

...

I want to add a third column, call it moved lines. The moved lines per commit can be found by doing the following in a loop for every commit:

  1. Grep changeset for lines starting with + or -
  2. Strip leading + or -
  3. sort
  4. uniq -d
  5. wc -l

Is there an fast and elegant way to run this pseudocode or do I just need to dump and parse a bunch of full git diffs to get what I need?

0

I wrote the following implementation in the TXR language. At first, I used your algorithm for determining the moved count. However, I noticed that it produces results which are not useful: for instance it was identifying positive values for "moved lines" in changes that contained nothing but + lines, simply because some of the + lines were duplicates of each other. The new algorithm is discussed in Notes at the end.

The complete program:

#!/usr/bin/env txr
@(bind option-spec
       @(list (opt nil "since" :str
                   "Specifies the starting date (passed \
                   \ through to git); it is mandatory.")
              (opt nil "help" :bool
                   "Prints this help text")))
@(bind parsed-opts @(getopts option-spec *args*))
@(if (or [parsed-opts "help"] (not [parsed-opts "since"])))
@  (output)

usage: @{self-path} --since=<date> -- git arguments
@  (end)
@  (do (opthelp option-spec)
       (exit 0))
@(end)
@(do
   (defun histogram (strings)
     [group-reduce (hash :equal-based) identity (op succ @1) strings 0])

   (defun moved (a b)
     (let* ((hist-a (histogram a))
            (hist-b (histogram b))
            (isec [hash-isec hist-a hist-b min]))
       [reduce-left + (hash-values isec) 0])))
@(next (open-command `git log --since=@[parsed-opts "since"] \
                     \ --date=short --pretty=format:"%H:%ad%x09" \
                     \ --numstat @{parsed-opts.out-args}`))
@(repeat)
@sha:@date@\t
@  (collect :gap 0)
@added@\t@removed@\t@rawpath
@    (next :string rawpath)
@    (cases)
@pro/{@before => @after}/@epi
@      (bind path `@pro/@after/@epi`)
@    (or)
@before => @after
@      (bind path after)
@    (or)
@      (bind path rawpath)
@    (end)
@    (next (open-command `git show -p @sha -- @path`))
@    (collect :vars ((+line nil) (-line nil)))
@      (cases)
+@{+line}
@      (or)
-@{-line}
@      (end)
@    (end)
@    (flatten -line +line)
@    (bind moved @(moved +line -line))
@  (end)
@  (output)
@date@\t
@    (repeat)
@added@\t@removed@\t@moved@\t@rawpath
@    (end)

@  (end)
@(end)

I have this in a marked executable file called movedlines.txr and example usage is:

$ ./movedlines.txr --since=2017-01-01 path/to

The --since option is mandatory; the path/to is an optional argument passed through to git. If you don't specify the mandatory option, or you specify --help, then the program prints a help summary and quits.

Notes:

  • I changed the output format of your example git command slightly to include the SHA to the left of the date, separated by a colon: see the %H. The program parses that out and then it can use the SHA to do a git show -p on each file in each set. The SHA is omitted when the imitation of the output is regurgitated by the program with the additional moved column.

  • There is a complication in that the output of git shows renames. The syntax falls into three cases, which are clearly and easily parsed by cases using the @(cases) construct. If an entire path is renamed, the rename is from => to. If just some components are renamed, it is be/fore/{from => to}/after. I don't know if multiple curly brace syntax occurs in one path; I haven't seen it. We must munge this syntax and convert it to a plain path, because git doesn't understand it. I.e. we must convert be/fore/{from => to}/after into be/fore/to/after. Maybe there is a git option to have the paths output that way without the notation; I didn't bother looking.

  • The script is not robust robust against spaces in filenames because of the use of open-command. For that we need open-process which takes an argument list.

  • The algorithm calculates separate frequency histograms of the diff - and + lines (minus their leading - or +). Then it calculates the intersection of these sets, which keeps only those histogram entries which occur in both histograms. The join function for the intersection is min. So for instance suppose line abc was added 5 times and removed 3 times. (min 3 5) is 3 and that's the number of moved abc lines. This commutes. If 3 occurrences of abc are removed, and 5 added, that implies 3 moves. This is by no means posited as some sort of perfect algorithm for detecting moves. It has obvious pitfalls like not detecting lines which de facto move but also undergo a trivial whitespace change, like indentation.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.