# Sum the total number of lines of code in all tar files in all subdirectories

I would like to calculate the sum of the total number of lines of all tar files in all subdirectories.

Consider the following example directory, `se-example`:

``````se-example
├── directory1
│   ├── archive1.tar
│   └── archive2.tar
└── directory2
├── archive3.tar
└── directory3
└── archive4.tar
``````

To recreate this example, consider the following:

``````echo "create and enter example"
mkdir se-example
cd se-example

echo "create example directory 1 (w/ two tar files)"
mkdir directory1
cd directory1

echo "create example tar 1 [+7 lines]"
mkdir archive1
printf "Line 1 \n Line 2" >> archive1/README.txt
printf "Line 1 \n Line 2" >> archive1/code1.py
printf "Line 1 \n Line 2 \n Line 3" >> archive1/code2.py
tar -cf archive1.tar archive1
rm -rf archive1

echo "create example tar 2 [+5 lines]"
mkdir archive2
printf "Line 1 \n Line 2 \n Line 3" >> archive2/code1.py
printf "Line 1 \n Line 2" >> archive2/code2.py
tar -cf archive2.tar archive2
rm -rf archive2

cd ..

echo "create example directory 1 (w/ subdirectory, readme, and two tar files)"
mkdir directory2
cd directory2

echo "create example readme [+0 lines]"
printf "Line 1 \n Line 2" >> README.txt

echo "create example tar 3 [+1 line]"
mkdir archive3
printf "Line 1" >> archive3/code1.py
tar -cf archive3.tar archive3
rm -rf archive3

echo "create example subdirectory (w/ one tar file)"
mkdir directory3
cd directory3

echo "create example tar 4 [+5 lines]"
mkdir archive4
printf "Line 1 \n Line 2 \n Line 3" >> archive4/code1.py
printf "Line 1 \n Line 2" >> archive4/code2.py
tar -cf archive4.tar archive4
rm -rf archive4

cd ..
cd ..
cd ..

echo "done creating example"
``````

In this example, there are four tar files with a total of 18 lines (7 lines in tar 1, 5 lines in tar 2, 1 line in tar 3, and 5 lines in tar 4). The correct answer would give 18.

I know how to count the number of tar files (suggested here):

``````find . -maxdepth 1 -mindepth 1 -type d | while read dir; do
printf "%-25.25s : " "\$dir"
find "\$dir" -name "*.tar" | wc -l
done
``````

I know how to count the number of lines of code in individual tar files (suggested here):

``````tar -tf se-example/directory1/archive1.tar | wc -l
``````

How do I combine these two commands, or is there an efficient alternative that will solve this problem?

• Why not just `find . -maxdepth 1 -mindepth 1 -type f -name *.tar | xargs -n1 tar tf >> file.txt` and then `awk 'END{print NR}' file.txt`? It will give an error message but it will still append all of the lines to the file. Commented Mar 7, 2020 at 0:23
• Maybe I am not understanding correctly. Executing `find . -maxdepth 1 -mindepth 1 -type f -name "*.tar" | xargs -n1 tar tf >> file.txt` yields an empty file. Commented Mar 7, 2020 at 0:25
• Perhaps your environment is different. When I run that same command, it prints the lines to the file with an error at the end(I was running it before without `-name *.tar`. I had to escape the asterisk to get it to expand and this command: `find . -mindepth 1 -maxdepth 1 -type f -name \*.tar | xargs -n1 tar tf >> file.txt` outputs all of the lines to file.txt and `awk 'END{print NR}' file.txt` prints the number of records or lines just like the first command but with no errors. Commented Mar 7, 2020 at 0:41

You're missing the newline `\n` for the last line in each of your test files, so the count of `wc` is actually less:

``````\$ find . \( -name "*.py" -o -name "README*" \) -print0 | xargs -0 wc -l
1 ./directory1/archive1/code1.py
2 ./directory1/archive1/code2.py
2 ./directory1/archive2/code1.py
1 ./directory1/archive2/code2.py
0 ./directory2/archive3/code1.py
2 ./directory2/directory3/archive4/code1.py
1 ./directory2/directory3/archive4/code2.py
11 total
``````

Since `/directory2/README.txt` is not in an archive, we have 10 lines in total.

1. List the files in the archives:

``````\$ find . -name "*.tar" -print0 | xargs -0I{} tar tf {}
archive1/
archive1/code1.py
archive1/code2.py
archive2/
archive2/code1.py
archive2/code2.py
archive3/
archive3/code1.py
archive4/
archive4/code1.py
archive4/code2.py
``````
2. Extract the files to stdout with the `-O` / `--to-stdout` option:

``````\$ find . -name "*.tar" -print0 | xargs -0I{} tar xOf {}
Line 1
Line 2Line 1
Line 2Line 1
Line 2
Line 3Line 1
Line 2
Line 3Line 1
Line 2Line 1Line 1
Line 2
Line 3Line 1
Line 2
``````

(The last line `Line 2` has no newline)

3. Count the lines with `wc`:

``````\$ find . -name "*.tar" -print0 | xargs -0I{} tar xOf {} | wc -l
10
``````
• Works at about 20 minutes to read 76615873 lines. Thanks! Commented Mar 7, 2020 at 1:31
• What if we wanted to only count the lines in the python files? Commented Mar 7, 2020 at 1:46
• GNU `tar` has a `--wildcards` option to match member names of the archive: `... | xargs -0I{} tar xOf {} --wildcards '*.py' | wc -l` Commented Mar 7, 2020 at 2:58