4

I wrote a bash shell script for Linux to move files in a static folder according to parameters specified in a data file.

I've reduced that script to a minimal reproducible example to demonstrate an issue I'm having with it:

#!/bin/bash
while IFS=" " read -r fldr matchStr
do
    perlExp=\'s/^/\\/home\\/test\\/alpha\\/"$fldr"\\//\'
    fullmatchStr=testfile\ -\ "$matchStr"\ -\ *.txt
    rename --verbose "$perlExp" ${fullmatchStr}
done < test.dat

(Please note that I originally started out using mv instead of rename, and I'm fine with using either command.)

This script reads test.dat, which is a text file of the syntax:

folderName matchStr
...

A simple example of this test.dat file:

test001 *.foo.bar
test002 *.foo.baz
test003 bar.*.baz

It is essential that wildcards are allowed in this data file and are supported by the script. Also, support for spaces within filenames is required.

Examples of filenames of files to be moved:

testfile - linux.foo.bar - 10101010 10101010.txt
testfile - unix.foo.bar - 01010101 10101010.txt
testfile - ubuntu.foo.baz - 10101010 01010101.txt
testfile - debian.foo.baz - 10101010 00000000.txt
testfile - bar.linus.baz - 11111111 01010101.txt

Given the above test.dat file, those files need to be moved, respectively, to these folders:

/home/test/alpha/test001
/home/test/alpha/test001
/home/test/alpha/test002
/home/test/alpha/test002
/home/test/alpha/test003

I read several relevant QAs, including:

but I'm still not getting the results I desire. No errors are presented, but no files are moved.

How can this piece of code be improved to make it work as desired?

1
  • fullmatchStr="testfile - $matchStr - *.txt" would be a much easier-to-read way to write the exact assignment your code already contains. Mind. if you want to actually replace that glob with a set of matching names, you should probably just store the names directly: names=( "testfile - $matchStr - "*.txt ), which will give you an array which can be expanded with "${names[@]}". Apr 3 at 16:28

2 Answers 2

7

If you're using a Perl-based rename (which seems to be the case), then it's best to not build Perl code using shell strings. Just use Perl code directly to do what you would have accomplished using shell code.

In this case, in the loop, I'd do something like:

IFS=
target="/home/test/alpha/$fldr/" rename --verbose 's/^/$ENV{target}/' "testfile - "$matchStr" - "*.txt

Here, the replacement is the value of the environment variable target, which we set for the rename command, allowing it to easily get it using Perl code. Additionally, setting IFS to the empty string disables word splitting, so we can safely use $matchStr for globbing. (However, it's probably not needed here and we can just use the unquoted $matchStr without bothering about word splitting since it is the result of IFS=" " read -r, but in the unlikely case that it contains tabs...)

5
  • 1
    @ilkkachu my focus is mainly on building any code using strings, like with building shell command lines by concatenating strings
    – muru
    Apr 2 at 14:31
  • Thank you for this answer. I appreciate it. It's odd that this answer received so many upvotes, as I don't think anyone tried it. It contained an error that prevented it from working. A terminating forward slash was missing in the assignment of the variable target. I've submitted an edit to correct it. Apr 4 at 8:38
  • @EndAnti-SemiticHate not particularly. Most people here aren't going to run rename, they will run rename -n which just shows what renames will be done, instead of making actual changes. Because otherwise we'd have to make the full /home/whatever paths and nobody's going to bother with that. And of course it's easy to miss a typo in the output then.
    – muru
    Apr 4 at 8:40
  • Can IFS= be taken outside of the while loop, as in ilkkachu's answer? Apr 4 at 8:47
  • @EndAnti-SemiticHate sure.
    – muru
    Apr 4 at 8:53
6

Mostly to focus on why you get no effect and no errors, this:

perlExp=\'s/^/\\/home\\/test\\/alpha\\/"$fldr"\\//\'
rename --verbose "$perlExp" ${fullmatchStr}

assigns to perlExp and passes to perl the string 's/^/\/home\/test\/alpha\/...\//' with the single quotes. That's a valid Perl expression, containing a single string constant. It just has no effect, so rename makes no changes.

That is, the quotes embedded in a shell variable are just data, and the shell doesn't parse the result of the variable expansion again. Hence, you should not put additional quotes inside your values. Consider how foo="'foo bar'"; echo "$foo" is the same as echo "'foo bar'" and prints 'foo bar'.


As another issue, fullmatchStr=testfile\ -\ "$matchStr"\ -\ *.txt sets the variable to the string testfile - ... - *.txt, and with both whitespace and glob characters in it, the unquoted expansion will first split it into testfile, -, ..., . and *.txt and only then will the glob expand, matching any filenames that end in .txt. To fix that, you'd need to disable word splitting by setting IFS to the empty string.

With Perl's s///, you can also use a different character as the separator, and that may be better with paths since you get to avoid the leaning toothpicks effect.


So, something like this might work:

#!/bin/bash

# disable word splitting in the script due to the unquoted expansion
IFS=

# the IFS assignment below only applies to 'read'
while IFS=" " read -r fldr matchStr
do
    perlExp='s#^#/tmp/test/alpha/'"$fldr"'/#'
    fullmatchStr="testfile - $matchStr - *.txt"
    rename --verbose "$perlExp" ${fullmatchStr}
done < test.dat

But in all honesty, muru's solution of passing the string through the environment is better in that it avoids the care and two-level quoting required when embedding data in the code and especially you don't need to care if the directory names you get from the file contain characters that are special to Perl.

1
  • Thanks. With your help, I found the error. I appreciate your help. Apr 4 at 8:37

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .