0

Question

I have a git repository full of files, mostly text.

I want to know that there are

  • n lines of code in files ending in .py
  • m lines of code in files ending in .md
  • o lines of code in files ending in .yaml
  • p lines of code in files with no extension
  • etc

Notes:

  • I want this to be recursive, checking inside folders
  • I want to exclude the .git directory in the top level folder
  • I want to ignore binary files is possible (noting that I have some text files with no extension, and some binary files with no extension)
  • I want to be case insensitive. Group .csv with .CSV
  • I want to ignore empty (or only whitespace) lines
  • If there's a file like myfile.yaml.j2 I do not care whether it's counted in the .j2 group or .yaml.j2.
5
  • What about empy/blank lines? do they count too?
    – Jetchisel
    May 1, 2020 at 4:52
  • I want to exclude them. The solution I wrote below includes them. I thought there were options to pass to wc for this, but I can't see them. May 1, 2020 at 5:34
  • 2
    Utility for counting lines of code is cloc(linux.die.net/man/1/cloc). Study the manual and see, what it offers. It will probably not fulfill all the requirements you have, but maybe you will find cloc output informative, and you will find out that it is what you need.
    – nobody
    May 1, 2020 at 6:02
  • Could you say that you want to ignore all hidden directories, or just .git? May 1, 2020 at 16:51
  • Ah yes, all hidden directories would be nice May 2, 2020 at 8:29

3 Answers 3

3

Try this

find ./ -not -path "./.git/*" -type f -exec wc -l {} + |
    awk '{print tolower($0)}' |
    sed -e '$ d' | 
    sed -e "s#/.*/##g" |
    sed -e "s/\./ \./g" |
    awk '
        { if ( NF <= 2 ) { count["none"] += $1 } else { count[$NF] += $1 } }
        { next }
        END { for (group in count) printf("%d%s%s\n", count[group], OFS, group) }
    ' |
    sort -n

Broken down:

  • find ./ find objects under this directory, recursively
  • -not -path "./.git/*" excluding .git
  • -type f files not directories
  • -exec wc -l {} + for each file, run the word count utility (wc). This includes empty lines, so doesn't meet all of the quesiton's requirements.
  • awk '{print tolower($0)}' make lowercase
  • sed -e '$ d' delete the last line, which is the sum of lines over all files
  • sed -e "s#/.*/##g" remove the path of the file, e.g. a/something.egg/blah should count as no extension, not .egg/blah extension
  • sed -e "s/\./ \./g" search/replace.with .`, so the file extension is its own word
  • awk '{ if ( NF <= 2 ) { count["none"] += $1 } else { count[$NF] += $1 } } { next } END { for (group in count) printf("%d%s%s\n", count[group], OFS, group) }' this is a big one. awk is powerful, but not super clear
    • count is a dictionary
    • if (NF <= 2) if there are fewer than 3 'words', i.e. there is no extension
    • count["none"] += $1 increment an element in the dictionary, key is string literal none, increment it by adding the number of lines in this file, which is the first word, which is $1
    • count[$NF] += $1 increment an element in the dictionary, key is $NF which is the last word in the line, which is the extension, by $1 which is the first word in the line, which is the number of lines in this file
    • { next } repeat for all lines, do what's after next only once
    • for (group in count) a for loop, inline
    • printf(...) format the output string, as 123 .abc (if there are 123 lines in files ending with .abc)
  • sort -n sort the results in ascending order, -n means sort as a number not string
2
  • You don't need {next} in your awk script as that's implied
    – roaima
    May 1, 2020 at 7:21
  • -not is not an operator supported by find. I guess that you intended to use !.
    – schily
    May 1, 2020 at 7:34
2

If I understand you correctly, and my tests are good, I propose this (assuming you want to skip hidden directories and files, tell me if that's not the case):

shopt -s globstar

declare -A arr
for f in test/**; do
  # if a directory, skip
  [[ -d "$f" ]] && continue
  lines=0
  # strip the extension
  ext="${f##*.}"
  # convert it to lowercase
  ext="${ext,,}"
  # if no dot in the name, extension is "empty"
  [[ ! $(basename "$f") =~ \. ]] && ext="empty"
  # count the lines
  lines=$(wc -l "$f"| cut -d' ' -f1)
  # if lines equals to 0, skip
  [[ $lines -eq 0 ]] && continue
  # append the number of line to the array
  lines=$(( "${arr[$ext]}"+$lines ))
  arr[$ext]=$lines 
done

# loop over the array
for n in ${!arr[@]}; do
  echo "files $n: total lines ${arr[$n]}"
done

Output (from my example files):

files yaml: total lines 3
files md: total lines 3
files empty: total lines 4
files csv: total lines 6
files py: total lines 5
3
  • It looks good, but I get bash: ""+23 : syntax error: operand expected (error token is """+23 "). Are you using bash? Do I have to invoke like . script.sh or just normal ./script.sh? May 2, 2020 at 8:34
  • @falsePockets For now (I deleted al my tests files), try with the line [[ ! -z "${arr[$ext]}" ]] && lines=$(( "${arr[$ext]}"+$lines )) May 2, 2020 at 14:45
  • I will recreate the files and let you know May 2, 2020 at 14:47
1

I broke things up into functions to try to make it easier to understand:

#!/bin/bash

# For the next two functions, we will use "-print0", which will print out \0 instead of \n.
# This will help prevent whitespace problems when piping the filenames into xargs.

find_extension()
{
    find "$1" -type f -name "*.$2" -print0 2>/dev/null
}

find_no_extension()
{
    find "$1" -type f -regex '^.*/[^.]+$' -print0 2>/dev/null
}

concat_files()
{
    xargs -0 cat
}

delete_empty_lines()
{
    sed -E '/^[[:space:]]*$/d'
}

line_count_of_files()
{
    concat_files | delete_empty_lines | wc -l
}

print_usage()
{
    echo "Usage: $0 [EXTENSION]... [SEARCH_DIRECTORY]";
}

NUMBER_OF_EXTENSIONS=$(($# - 1))
SEARCH_DIR="${*: -1}"

if [ $# -lt 2 ];
then
    echo "Not enough parameters.";
    print_usage;
    exit 1;
fi

if ! [ -d "$SEARCH_DIR" ];
then
    echo "$SEARCH_DIR does not exist, or is not a directory."
    print_usage;
    exit 1;
fi

for EXTENSION in "${@:1:$NUMBER_OF_EXTENSIONS}";
do
    printf ".$EXTENSION: %s\n" $(find_extension "$SEARCH_DIR" "$EXTENSION" | line_count_of_files)
done


printf "No extension: %s\n" $(find_no_extension "$SEARCH_DIR" | line_count_of_files)

This is more of a generic script that lets you specify arbitrary file extensions to search for. It will always search for files with no extension though.

You should save this into a file, give it executable permission, and put it in your PATH. Let's say you name it count_lines.sh. You can call it like this: count_lines.sh py md yaml ~/Code. This will search in the directory ~/Code for files ending in .py, .md, and .yaml, as well as files without an extension at all. You can choose any number of extensions to search for, just make sure you have at least one.

3
  • 1
    xargs -0 is non-portable, find -exec + however is
    – schily
    May 1, 2020 at 7:36
  • @schily wasn't aware of that. If you know how to fix it to use the portable solution, please make an edit. May 1, 2020 at 7:40
  • Probably by using arguments with the command, given to the functions in case you like to use functions.
    – schily
    May 1, 2020 at 7:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.