1

I have many hundreds of thousands of files in a directory. These files are named as follows:

left-00001.tiff
left-00002.tiff
...
left-99999.tiff
left-100000.tiff
...
left-245000.tiff

I would like to rename the files as follows:

left-000001.tiff
...
left-099999.tiff
...
left-245000.tiff

I found elegant solution to this problem here.

This solution implements a bash script called zeropad.sh. the bash is coded as follows:

#!/bin/bash
num=`expr match "$1" '[^0-9]*\([0-9]\+\).*'`
paddednum=`printf "%06d" $num`
echo ${1/$num/$paddednum}

and can be applied iteratively using for loop as follows:

for i in *.tiff;do mv $i `./zeropad.sh $i`; done

However, this solution takes a very long time because it does much unnecessary work renaming all the files which are already properly padded. i.e. as %06d type numbers. For my own purposes this is solution is very slow.

I have two questions:

1- How can I modify the iterator to only apply zeropad.sh on files which need to be zero padded?

2- How can I use the command touch in a for loop to generate test data? It is crucial to verify that this script works before I apply it on the original data.

  • 1
    It is slow because you are calling your shell script for every file. It would be quicker if you did the looping and renaming in the same script. – Kusalananda Sep 5 '18 at 19:19
  • 2
    Posting as comment because question asks for bash, but I think Python is a better fit: from pathlib import Path; for f in Path().glob('left-?????.tiff'): f.rename(f.name.replace('-', '-0')) Note that touch left-{00001..99999}.tiff can be used to quickly generate files. – BoppreH Sep 5 '18 at 21:43
3

The bulk of the time spent by your loop is probably in calling your zeropad.sh script.

Instead, do it all in one script:

#!/bin/bash

for filename in left-*.tiff; do
    if [[ "$filename" =~ ^left-0*([1-9]?[0-9]+)\.tiff$ ]]; then
        num=${BASH_REMATCH[1]}
        newname="left-$( printf '%06d' "$num" ).tiff"
        if [ "$filename" != "$newname" ] && [ ! -e "$newname" ]; then
            echo mv "$filename" "$newname"
        fi
    fi
done

Remove the echo once you have verified that the script is doing the correct thing.

| improve this answer | |
4

This is how I usually do it (manually on the shell):

rename left- left-0 left-?.png    # for 0-9
rename left- left-0 left-??.png   # for 00-99
rename left- left-0 left-???.png  # for 000-999
# result left-0000.png - left-9999.png

This is easy to do in an interactive shell session... just repeat the last command with one additional ? added.

However, with a large number of files, you'll eventually end up with a too long list of arguments. And obviously it's not the most efficient choice, as it ends up renaming the same file multiple times (left-1.png -> left-01.png -> left-001.png -> ...).

Also there are two flavors for rename about, one with perl regular expressions and one without. Depending on distro you end up with rename.ul or perl-rename or other names for them. Basically it renders any script using the rename command unportable since you never know what to expect.

I'm using the the util-linux rename and your question is actually one of their examples, from the man page:

EXAMPLES

Given the files foo1, ..., foo9, foo10, ..., foo278, the commands

    rename foo foo00 foo?
    rename foo foo0 foo??

will turn them into foo001, ..., foo009, foo010, ..., foo278.

Which is the more efficient method (each file renamed only once) but you have to figure out the correct distribution of 000 vs ??? or you'll end up with a wrong result.

To me, the inefficient method is the more practical one, on an interactive shell, when dealing with a reasonable small set of files.


The advantage of rename over scripting it yourself is that it doesn't have to spawn one mv process for each file, or as in your case, a sub-script just to figure out a filename. It's not clear what has more overhead, the process spawning, or repeated renaming, and I'm too lazy to benchmark it.

Actually the answer you linked already contains the "optimal" solution at the very end... using perl-rename:

rename 's/\d+/sprintf("%04d",$&)/e' *.png

Well, one can argue about the regular expression, but the point is, it's possible to do it all in one go, without unnecessary mv, or spawning of processes. If you still need to improve on that, write a tool that reads directory contents directly instead of using shell globbing (which sorts, which is slow) and performs the renaming as needed.

Maybe that's actually the answer you linked to and maybe that's why you're getting downvoted. ;)

| improve this answer | |
4

What's costly is to fork so many processes and run so many commands for each file.

With zsh:

zmodload zsh/files # make mv builtin to speed things up
autoload zmv
zmv -n '(*-)(<->)(.tiff)' '$1${(l:6::0:)2}$3'

(remove -n when happy)

That's all with builtins, so doesn't fork any process nor execute any file.

Or with perl's rename:

rename -n 's/\d+(?=\.tiff\z)/sprintf "%06d", $&/e' ./*[0-9].tiff
| improve this answer | |
2

For the first part, consider:

for i in left-?????.tiff left-????.tiff left-???.tiff left-??.tiff left-?.tiff ...`

If that generates too many files, then break it up into sections:

for i in left-?????.tiff ...`

...

for i in left-????.tiff ...`

...

The above works by using the ? glob character to substitute any single character whenever it appears. Here, I've specifically requested 5, 4, 3, 2, and then 1 digits after the leading left-.

For the second part, one option is:

dir=$(mktemp)
cd "$dir"
for i in $(seq 10); do touch $(printf 'left-%05d.tiff' $((RANDOM % 10000))); done

Adjust the seq 10 to generate more or fewer filenames. Adjust the % 10000 to generate smaller or larger numbers. Note that bash's $RANDOM generates numbers between 0 and 32,767.

| improve this answer | |
  • Thanks Jeff. I will try this as soon as I'm back to my pc – kevinkayaks Sep 5 '18 at 19:03
  • Thakns jeff, I used the dummy data generator you suggested. The renaming script led to a generating too many files bug and I didn't really troubleshoot. Sorry I didn't accept it. @kusalanda's solution is just easier to understand with my limited unix knowledge – kevinkayaks Sep 5 '18 at 23:18
1

I love Perl one-liners:

ls left-*.tiff | perl -ne 'if(m/(\S+)-(\d+).tiff/){chomp;printf "mv $_ left-%06d.tiff\n", $2}' | bash

PS, make sure to double check the output before piping into bash. Just to be safe.

| improve this answer | |
  • 2
    perl has a rename function built-in. Printing out bash commands is always going to be slower. Or better, use the existing rename wrapper script like Stephane's answer. – Peter Cordes Sep 6 '18 at 6:54
1

You can rename all the files in parallel. Do the following trivial changes on the same slow code that you had provided in your question, as follows:

cd data_folder  # cd the folder where you put the *.tiff files
for i in *.tiff;do 
{
mv $i `./zeropad.sh $i`;
}&

This will rename all the files at once. Please be aware that you must have enough memory resources on your workstation before you run this code inside the folder that contain the *.tiff files. No enough memory resources may lead to memory crash. But given that the process is only renaming files you should be fine!

In order to take into acccount memory resources on your workstation. Save the following code in a file called code, give it permissions then run it:

mem=$(free -m | awk 'NR==2{printf "Memory Usage: %s/%sMB (%.2f%%)\n", $3,$2,$3*100/$2 }' | grep Memory | awk '{print $3}' | tr -d "()%MB" | cut -d / -f 2  )

for i in *.tiff;do 
{
mv $i `./zeropad.sh $i`;
}&

if [ $mem -lt 100000 ] 
 then 
   if  (( "$i" % 75 == 0 ))
    then 
      sleep 4
   fi
 fi
if [ $mem -gt 100000 ] 
 then 
   if  (( "$i" % 300 == 0 ))
    then 
      sleep 3
   fi
 fi
done

When you run code, it will check memory resources on your workstation using the variable mem. If memory is less 100000MB, then it will rename 75 files at once. If memory resources are more than 100000MB, then it will rename up to 300 files at once. However, you can adjust all the variables as you want.

| improve this answer | |
  • Hi TNT: I did not try this because @Kusalananda's solution was simply easier for me to understand with limited knowledge. However, how much memory would this require? Does it actually hold the files in RAM? The folder of files is of the order of ~500GB – kevinkayaks Sep 5 '18 at 23:19
  • @kevinkayaks. 500GB is to much. Please see my edits. I had updated the code. Now it will sense how much memory resources are available and then it will rename the files accordingly. You can adjust the numbers in the code as you want! – user90704 Sep 6 '18 at 0:15
  • 1
    @kevinkayaks: Renaming a file within the same directory (or on the same filesystem) doesn't require touch the file's data at all. Renaming a 1TB file is as fast as renaming an empty file; the rename(2) system call just has to modify directory entries (and update the ctime in the file's own inode). The memory consumption here doesn't come from filesize, it just comes from the couple MB of RAM for mv, but mostly the zeropad.sh in parallel many times. This doesn't optimize away the bash startup cost of running zeropad.sh once per file. – Peter Cordes Sep 6 '18 at 6:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.