1

I would like to calculate the sum of the total number of lines of all tar files in all subdirectories.

Consider the following example directory, se-example:

se-example
├── directory1
│   ├── archive1.tar
│   └── archive2.tar
└── directory2
    ├── README.txt
    ├── archive3.tar
    └── directory3
        └── archive4.tar

To recreate this example, consider the following:

echo "create and enter example"
mkdir se-example
cd se-example

echo "create example directory 1 (w/ two tar files)"
mkdir directory1
cd directory1

echo "create example tar 1 [+7 lines]"
mkdir archive1
printf "Line 1 \n Line 2" >> archive1/README.txt
printf "Line 1 \n Line 2" >> archive1/code1.py
printf "Line 1 \n Line 2 \n Line 3" >> archive1/code2.py
tar -cf archive1.tar archive1
rm -rf archive1 

echo "create example tar 2 [+5 lines]"
mkdir archive2 
printf "Line 1 \n Line 2 \n Line 3" >> archive2/code1.py
printf "Line 1 \n Line 2" >> archive2/code2.py
tar -cf archive2.tar archive2
rm -rf archive2

cd ..

echo "create example directory 1 (w/ subdirectory, readme, and two tar files)"
mkdir directory2
cd directory2

echo "create example readme [+0 lines]"
printf "Line 1 \n Line 2" >> README.txt

echo "create example tar 3 [+1 line]"
mkdir archive3 
printf "Line 1" >> archive3/code1.py
tar -cf archive3.tar archive3
rm -rf archive3

echo "create example subdirectory (w/ one tar file)"
mkdir directory3
cd directory3

echo "create example tar 4 [+5 lines]"
mkdir archive4
printf "Line 1 \n Line 2 \n Line 3" >> archive4/code1.py
printf "Line 1 \n Line 2" >> archive4/code2.py
tar -cf archive4.tar archive4
rm -rf archive4

cd ..
cd ..
cd ..

echo "done creating example"

In this example, there are four tar files with a total of 18 lines (7 lines in tar 1, 5 lines in tar 2, 1 line in tar 3, and 5 lines in tar 4). The correct answer would give 18.

I know how to count the number of tar files (suggested here):

find . -maxdepth 1 -mindepth 1 -type d | while read dir; do
  printf "%-25.25s : " "$dir"
  find "$dir" -name "*.tar" | wc -l
done

I know how to count the number of lines of code in individual tar files (suggested here):

tar -tf se-example/directory1/archive1.tar | wc -l

How do I combine these two commands, or is there an efficient alternative that will solve this problem?

3
  • Why not just find . -maxdepth 1 -mindepth 1 -type f -name *.tar | xargs -n1 tar tf >> file.txt and then awk 'END{print NR}' file.txt? It will give an error message but it will still append all of the lines to the file. Mar 7, 2020 at 0:23
  • Maybe I am not understanding correctly. Executing find . -maxdepth 1 -mindepth 1 -type f -name "*.tar" | xargs -n1 tar tf >> file.txt yields an empty file. Mar 7, 2020 at 0:25
  • Perhaps your environment is different. When I run that same command, it prints the lines to the file with an error at the end(I was running it before without -name *.tar. I had to escape the asterisk to get it to expand and this command: find . -mindepth 1 -maxdepth 1 -type f -name \*.tar | xargs -n1 tar tf >> file.txt outputs all of the lines to file.txt and awk 'END{print NR}' file.txt prints the number of records or lines just like the first command but with no errors. Mar 7, 2020 at 0:41

1 Answer 1

3

You're missing the newline \n for the last line in each of your test files, so the count of wc is actually less:

$ find . \( -name "*.py" -o -name "README*" \) -print0 | xargs -0 wc -l
  1 ./directory1/archive1/README.txt
  1 ./directory1/archive1/code1.py
  2 ./directory1/archive1/code2.py
  2 ./directory1/archive2/code1.py
  1 ./directory1/archive2/code2.py
  1 ./directory2/README.txt
  0 ./directory2/archive3/code1.py
  2 ./directory2/directory3/archive4/code1.py
  1 ./directory2/directory3/archive4/code2.py
 11 total

Since /directory2/README.txt is not in an archive, we have 10 lines in total.

  1. List the files in the archives:

    $ find . -name "*.tar" -print0 | xargs -0I{} tar tf {}
    archive1/
    archive1/README.txt
    archive1/code1.py
    archive1/code2.py
    archive2/
    archive2/code1.py
    archive2/code2.py
    archive3/
    archive3/code1.py
    archive4/
    archive4/code1.py
    archive4/code2.py
    
  2. Extract the files to stdout with the -O / --to-stdout option:

    $ find . -name "*.tar" -print0 | xargs -0I{} tar xOf {}
    Line 1
     Line 2Line 1
     Line 2Line 1
     Line 2
     Line 3Line 1
     Line 2
     Line 3Line 1
     Line 2Line 1Line 1
     Line 2
     Line 3Line 1
     Line 2
    

    (The last line Line 2 has no newline)

  3. Count the lines with wc:

    $ find . -name "*.tar" -print0 | xargs -0I{} tar xOf {} | wc -l
    10
    
3
  • Works at about 20 minutes to read 76615873 lines. Thanks! Mar 7, 2020 at 1:31
  • What if we wanted to only count the lines in the python files? Mar 7, 2020 at 1:46
  • 1
    GNU tar has a --wildcards option to match member names of the archive: ... | xargs -0I{} tar xOf {} --wildcards '*.py' | wc -l
    – Freddy
    Mar 7, 2020 at 2:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.