Issue expanding variable with multiple wildcards in bash shell script with mv / rename

Question

I wrote a bash shell script for Linux to move files in a static folder according to parameters specified in a data file.

I've reduced that script to a minimal reproducible example to demonstrate an issue I'm having with it:

#!/bin/bash
while IFS=" " read -r fldr matchStr
do
    perlExp=\'s/^/\\/home\\/test\\/alpha\\/"$fldr"\\//\'
    fullmatchStr=testfile\ -\ "$matchStr"\ -\ *.txt
    rename --verbose "$perlExp" ${fullmatchStr}
done < test.dat

(Please note that I originally started out using mv instead of rename, and I'm fine with using either command.)

This script reads test.dat, which is a text file of the syntax:

folderName matchStr
...

A simple example of this test.dat file:

test001 *.foo.bar
test002 *.foo.baz
test003 bar.*.baz

It is essential that wildcards are allowed in this data file and are supported by the script. Also, support for spaces within filenames is required.

Examples of filenames of files to be moved:

testfile - linux.foo.bar - 10101010 10101010.txt
testfile - unix.foo.bar - 01010101 10101010.txt
testfile - ubuntu.foo.baz - 10101010 01010101.txt
testfile - debian.foo.baz - 10101010 00000000.txt
testfile - bar.linus.baz - 11111111 01010101.txt

Given the above test.dat file, those files need to be moved, respectively, to these folders:

/home/test/alpha/test001
/home/test/alpha/test001
/home/test/alpha/test002
/home/test/alpha/test002
/home/test/alpha/test003

I read several relevant QAs, including:

• mv: cannot stat No such file or directory in shell script
• https://stackoverflow.com/questions/8748831/when-do-we-need-curly-braces-around-shell-variables
• Why does my shell script choke on whitespace or other special characters?

but I'm still not getting the results I desire. No errors are presented, but no files are moved.

How can this piece of code be improved to make it work as desired?

Answer 1

If you're using a Perl-based rename (which seems to be the case), then it's best to not build Perl code using shell strings. Just use Perl code directly to do what you would have accomplished using shell code.

In this case, in the loop, I'd do something like:

IFS=
target="/home/test/alpha/$fldr/" rename --verbose 's/^/$ENV{target}/' "testfile - "$matchStr" - "*.txt

Here, the replacement is the value of the environment variable target, which we set for the rename command, allowing it to easily get it using Perl code. Additionally, setting IFS to the empty string disables word splitting, so we can safely use $matchStr for globbing. (However, it's probably not needed here and we can just use the unquoted $matchStr without bothering about word splitting since it is the result of IFS=" " read -r, but in the unlikely case that it contains tabs...)

Answer 2

Mostly to focus on why you get no effect and no errors, this:

perlExp=\'s/^/\\/home\\/test\\/alpha\\/"$fldr"\\//\'
rename --verbose "$perlExp" ${fullmatchStr}

assigns to perlExp and passes to perl the string 's/^/\/home\/test\/alpha\/...\//' with the single quotes. That's a valid Perl expression, containing a single string constant. It just has no effect, so rename makes no changes.

That is, the quotes embedded in a shell variable are just data, and the shell doesn't parse the result of the variable expansion again. Hence, you should not put additional quotes inside your values. Consider how foo="'foo bar'"; echo "$foo" is the same as echo "'foo bar'" and prints 'foo bar'.

---

As another issue, fullmatchStr=testfile\ -\ "$matchStr"\ -\ *.txt sets the variable to the string testfile - ... - *.txt, and with both whitespace and glob characters in it, the unquoted expansion will first split it into testfile, -, ..., . and *.txt and only then will the glob expand, matching any filenames that end in .txt. To fix that, you'd need to disable word splitting by setting IFS to the empty string.

With Perl's s///, you can also use a different character as the separator, and that may be better with paths since you get to avoid the leaning toothpicks effect.

---

So, something like this might work:

#!/bin/bash

# disable word splitting in the script due to the unquoted expansion
IFS=

# the IFS assignment below only applies to 'read'
while IFS=" " read -r fldr matchStr
do
    perlExp='s#^#/tmp/test/alpha/'"$fldr"'/#'
    fullmatchStr="testfile - $matchStr - *.txt"
    rename --verbose "$perlExp" ${fullmatchStr}
done < test.dat

---

But in all honesty, muru's solution of passing the string through the environment is better in that it avoids the care and two-level quoting required when embedding data in the code and especially you don't need to care if the directory names you get from the file contain characters that are special to Perl.