1

I would like to calculate the sum of the total number of lines of all tar files in all subdirectories.

Consider the following example directory, se-example:

se-example
├── directory1
│   ├── archive1.tar
│   └── archive2.tar
└── directory2
    ├── README.txt
    ├── archive3.tar
    └── directory3
        └── archive4.tar

To recreate this example, consider the following:

echo "create and enter example"
mkdir se-example
cd se-example

echo "create example directory 1 (w/ two tar files)"
mkdir directory1
cd directory1

echo "create example tar 1 [+7 lines]"
mkdir archive1
printf "Line 1 \n Line 2" >> archive1/README.txt
printf "Line 1 \n Line 2" >> archive1/code1.py
printf "Line 1 \n Line 2 \n Line 3" >> archive1/code2.py
tar -cf archive1.tar archive1
rm -rf archive1 

echo "create example tar 2 [+5 lines]"
mkdir archive2 
printf "Line 1 \n Line 2 \n Line 3" >> archive2/code1.py
printf "Line 1 \n Line 2" >> archive2/code2.py
tar -cf archive2.tar archive2
rm -rf archive2

cd ..

echo "create example directory 1 (w/ subdirectory, readme, and two tar files)"
mkdir directory2
cd directory2

echo "create example readme [+0 lines]"
printf "Line 1 \n Line 2" >> README.txt

echo "create example tar 3 [+1 line]"
mkdir archive3 
printf "Line 1" >> archive3/code1.py
tar -cf archive3.tar archive3
rm -rf archive3

echo "create example subdirectory (w/ one tar file)"
mkdir directory3
cd directory3

echo "create example tar 4 [+5 lines]"
mkdir archive4
printf "Line 1 \n Line 2 \n Line 3" >> archive4/code1.py
printf "Line 1 \n Line 2" >> archive4/code2.py
tar -cf archive4.tar archive4
rm -rf archive4

cd ..
cd ..
cd ..

echo "done creating example"

In this example, there are four tar files with a total of 18 lines (7 lines in tar 1, 5 lines in tar 2, 1 line in tar 3, and 5 lines in tar 4). The correct answer would give 18.

I know how to count the number of tar files (suggested here):

find . -maxdepth 1 -mindepth 1 -type d | while read dir; do
  printf "%-25.25s : " "$dir"
  find "$dir" -name "*.tar" | wc -l
done

I know how to count the number of lines of code in individual tar files (suggested here):

tar -tf se-example/directory1/archive1.tar | wc -l

How do I combine these two commands, or is there an efficient alternative that will solve this problem?

3
  • Why not just find . -maxdepth 1 -mindepth 1 -type f -name *.tar | xargs -n1 tar tf >> file.txt and then awk 'END{print NR}' file.txt? It will give an error message but it will still append all of the lines to the file. Mar 7, 2020 at 0:23
  • Maybe I am not understanding correctly. Executing find . -maxdepth 1 -mindepth 1 -type f -name "*.tar" | xargs -n1 tar tf >> file.txt yields an empty file. Mar 7, 2020 at 0:25
  • Perhaps your environment is different. When I run that same command, it prints the lines to the file with an error at the end(I was running it before without -name *.tar. I had to escape the asterisk to get it to expand and this command: find . -mindepth 1 -maxdepth 1 -type f -name \*.tar | xargs -n1 tar tf >> file.txt outputs all of the lines to file.txt and awk 'END{print NR}' file.txt prints the number of records or lines just like the first command but with no errors. Mar 7, 2020 at 0:41

1 Answer 1

3

You're missing the newline \n for the last line in each of your test files, so the count of wc is actually less:

$ find . \( -name "*.py" -o -name "README*" \) -print0 | xargs -0 wc -l
  1 ./directory1/archive1/README.txt
  1 ./directory1/archive1/code1.py
  2 ./directory1/archive1/code2.py
  2 ./directory1/archive2/code1.py
  1 ./directory1/archive2/code2.py
  1 ./directory2/README.txt
  0 ./directory2/archive3/code1.py
  2 ./directory2/directory3/archive4/code1.py
  1 ./directory2/directory3/archive4/code2.py
 11 total

Since /directory2/README.txt is not in an archive, we have 10 lines in total.

  1. List the files in the archives:

    $ find . -name "*.tar" -print0 | xargs -0I{} tar tf {}
    archive1/
    archive1/README.txt
    archive1/code1.py
    archive1/code2.py
    archive2/
    archive2/code1.py
    archive2/code2.py
    archive3/
    archive3/code1.py
    archive4/
    archive4/code1.py
    archive4/code2.py
    
  2. Extract the files to stdout with the -O / --to-stdout option:

    $ find . -name "*.tar" -print0 | xargs -0I{} tar xOf {}
    Line 1
     Line 2Line 1
     Line 2Line 1
     Line 2
     Line 3Line 1
     Line 2
     Line 3Line 1
     Line 2Line 1Line 1
     Line 2
     Line 3Line 1
     Line 2
    

    (The last line Line 2 has no newline)

  3. Count the lines with wc:

    $ find . -name "*.tar" -print0 | xargs -0I{} tar xOf {} | wc -l
    10
    
3
  • Works at about 20 minutes to read 76615873 lines. Thanks! Mar 7, 2020 at 1:31
  • What if we wanted to only count the lines in the python files? Mar 7, 2020 at 1:46
  • 1
    GNU tar has a --wildcards option to match member names of the archive: ... | xargs -0I{} tar xOf {} --wildcards '*.py' | wc -l
    – Freddy
    Mar 7, 2020 at 2:58

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .