Bash script to compress nested directories not traversing correctly

Question

I have a folder structure that looks like this:

0/1/2 and the numbers goes from {0..9}

I tried writing a bash shell script so that I can compress each of the folders like this:

#!/bin/bash
for i in {0..9};
  do
  cd $i
    for j in {0..9};
      do
      cd $j
        for k in {0..9};
         do
           tar -czvf $i$j$k.tar.gz $k;
         done
      done
 done

It seems to be able to create the names for the .tar.gz files ok, but it won't traverse through the directory structure to be able to create them for each of the folders inside the folders.

I'm not 100% sure what I am doing wrong with this script.

What I want it to do is to go to 0/0/ and then create the .tar.gz file for folders {0..9}.

And then go to 0/1/ and do it again.

And then go to 0/2/ and do it again, etc.

Until all of the folders have been cycled through, and the .tar.gz files have been created.

It seems to just saying in 0/0/ and just keep compressing the same files over and over again, and it won't move on to the next folder.

Any help is greatly appreciated.

(Disclaimer: I'm not a programmer nor a developer, so this is new territory for me.)

(Background: this is for me to back up my Photoprism data storage (as it has about 2.5 million pictures and takes up about 1.6 TB of space or so). I tried sending it straight to LTO-8 tape and I guess that because there are so many small files, LTFS ran into an out of memory exception error such that when I eject the tape and try to load it back in, it automatically spits the tape back out because of said out-of-memory error; but that's a whole 'nother question. So I figure that if I compress the files, then it will have fewer files that LTFS is dealing with, and therefore; that should help fix the LTFS/LTO-8 tape drive error.)

Thank you.

Answer 1

Instead of cd, use the -C option of tar:

Something like (untested):

#!/bin/bash
for i in {0..9}
  do
    for j in {0..9}
      do
        for k in {0..9}
         do
           [ -d $i/$j/$k ] && tar -C $i/$j -czvf $i$j$k.tar.gz $k
         done
      done
 done

I added a [ -d ... ] test in case some of these directories don't exist.

Note: Conveniently, the .tar.gz files are created in the current directory, not in the directory where tar switches to using -C! So you have them all in one place.

Solution in TXR Lisp:

(each ((digits "000".."999"))
  (match `@{i 1}@{j 1}@k` digits
    (if (path-dir-p `@i/@j/@k`)
      (sh `tar -C @i/@j -czvf @i@[email protected] @k`))))

Let string generation generate the digits. Destructure them into variables with match; then the same core logic: does the i-j-k directory exist; if so, tell tar to switch to i/j, make the ijk archive out of k.

Alternative without the match stuff; with each-prod we can condense nested loops:

(each-prod ((i "0".."9") (j "0".."9") (k "0".."9"))
  (if (path-dir-p `@i/@j/@k`)
    (sh `tar -C @i/@j -czv0f @i@[email protected] @k`)))

Answer 2

Your script, starting from /start, does

• cd 0, after which you land in /start/0, then
• cd 0, after which you land in /start/0/0, then
• cd 1, after which you land in /start/0/0/1, not /start/0/1, then
• cd 2, which likely fails as there's no /start/0/0/1/2.

If I had to use bash, I'd do:

#! /bin/bash -
shopt -s nullglob
for i in [0123456789]/[0123456789]/; do
  (
    CDPATH= cd "$i" && 
      for j in [0123456789]/; do
        j=${j%/}
        tar zcf "${i//\/}$j.tar.gz" "$j"
      done
  )
done

Note:

• the (...) which starts a subshell, so cd only changes the current directory of that subshell
• the && so we don't carry on if the cd failed
• the CDPATH= in case there's a CDPATH in the environment which could make you land somewhere completely different.
• the usage of globbing ([0123456789]) instead of zsh-style {0..9} brace expansion, so it only expands to directories that actually exists.
• the trailing / (which we strip afterwards with ${j%/}, or ${i//\/} along with the other /s) to restrict the globbing to directories.