Moving a list of files by matching them to a destination index

Question

Say I have two text files src.txt and dest.txt, where src.txt contains a list of filenames (some of which include spaces) in /src/dir/ and dest.txt contains, in random order, a list of the complete file paths (again with spaces) where they belong. For example:

src.txt:

file 1.jpg
file_2.html
file 3.jpg

dest.txt:

/dest/dir 1/file 3.jpg
/dest/file4.txt
/dest/file 5.txt
/dest/dir 2/file 1.jpg
/dest/file_2.html

How can I perform this batch move operation from the shell? I have been working with a while read loop over the source file, and I am pretty sure I need to use the mv command, but I'm not sure if grep or sed are needed here. I keep running into cannot stat... and space-character resolution errors.

Answer 1

With zsh:

src=(${(f)"$(<src.txt)"})
for f (${(f)"$(<dest.txt)"})
(($src[(Ie)$f:t])) && mv /src/dir/$f:t $f

This reads each file in an array and then, for each element in the "dest" array, if the basename (:t is a zsh modifier which removes all leading pathname components) is also in the "src" array then it moves the file. To perform a dry-run replace mv with printf '"%s" -> "%s"\n'.

---

Now, you could also run (still in zsh):

for f (${(f)"$(grep -Ff src.txt dest.txt)"})
mv /src/dir/$f:t $f

which works fine as long as none of the file names in src.txt matches any of the directory names (or part of that name) in the list of paths in dest.txt (e.g. a file name data1 in src.txt and a path like /path/data1_dir/some_file in dest.txt would give a false positive). To avoid that you could pass the file names to grep as patterns (i.e. using regex like /filename$) instead of Fixed strings so as to match only the last component of the paths in dest.txt. Though that requires escaping all special characters (if any) in file names in src.txt, e.g. this time with bash (4):

readarray -t files < <(sed 's|[[\.*^$/]|\\&|g;s|.*|/&$|' src.txt | grep -f- dest.txt)
for f in "${files[@]}"; do mv /src/dir/"${f##*/}" "$f"; done

Answer 2

If a newline is an acceptable delimiter then the following should be pretty robust in an POSIX shell:

IFS='
';set -f
for   f in $(cat <"$destfile")
do    [ -e "./${f##*/}" ] ||
      [ -h "./${f##*/}" ] &&
      mv   "./${f##*/}"  "$f"
done

There are two possible problems with that solution that I can imagine:

• The input file size is simply too large to split out in a single go like that.

  • On my system this doesn't really even bear seriously considering until input approaches many tens of thousands of lines.

• A filename in $destfile might exist in the current directory and yet should not be moved anyway.

  • Because this solution foregoes comparing the two input files entirely and only checks each last pathname component in $destfile for existence in the current directory, if any filenames might match unintentionally it should not be considered.

If only the first problem needs handling:

sed -ne"s|'|'"'\\&&|g' <"$destfile"    \
    -e "s|.*/\([^/].*\)|_mv './\1' '&'|p" | 
sh  -c '_mv(){ [ -e "$1" ]||[ -h "$1" ]&& mv "$@";};. /dev/fd/0'

If your sh is dash you might drop the . /dev/fd/0 at the end and use:

sed ... | sh -cs '_mv(){ ...;}'

...because dash strangely handles both the command-line and stdin invocation options in concert and without complaint. That wouldn't be very portable, but . /dev/fd/0 - while pretty portable - isn't strictly standards-compliant either.

If the second issue is a concern:

export  LC_ALL=C 
sed  -ne'\|/$|!s|.*/\(.*\)|\1/&|p' <"$destfile" |
sort -t/ -k1,1 - ./"$srcfile"  |  cut  -d/ -f2- |
sed  -e "\|/|!N;\|\n.*/|!d"    \
     -e "s|'|'"'\\&&|g'        \
     -e "s|\n|' '|;s|.*|mv './&'|" | sh

...that should handle it very nicely so long as all of the filenames in ./"$srcfile" are properly and identically accounted for at the tail-end of some path in "$destfile". sort will always float the shorter of two otherwise identical comparisons to the top, and so when only the first field matters, and the filename is prepended to the head of each pathname from "$destfile" then a merged sort operation of both files will output sequences like:

$srcfile:  no /
$destfile: match
$destfile: unique
$destfile: unique
...
$srcfile:  no /
$destfile: match
$destfile: unique

...and so you only need to concern yourself with pairs of lines beginning with one that doesn't match /.

Answer 3

while read i; do echo cp \""$i"\" \"$(grep "/$i$" dst.txt)\"; done < src.txt

This will print what would have been done. Just get rid of the echo to actually copy the files.

Answer 4

A one-liner script generates a script which generates a script.

In this example, we use a first invocation of sed on src.txt to generate a second sed script that will run on dest.txt to generate a shell script to copy the files.

Here's the one-liner:

$ sed -n "$(sed 's,\(..*\),/\\/\1$/ { s/^/cp "\1" "/; s/$/";/; p; },' src.txt)" dest.txt #| sh -x

and the output:

cp "file 3.jpg" "/dest/dir 1/file 3.jpg";
cp "file 1.jpg" "/dest/dir 2/file 1.jpg";
cp "file_2.html" "/dest/file_2.html";

Note the comment #| sh at the end of the command. This way, you can try the command and see what it will do, and if it's good, uncomment the pipe to sh and really copy the files.

The inner sed command builds a sed script out of src.txt. The first line of the generated script looks like this:

/\/file 1.jpg$/ { s/^/cp file 1.jpg /; p; }

This is how it works:

Input:

    $ cat src.txt
    file 1.jpg
    file_2.html
    file 3.jpg

    $ cat dest.txt
    /dest/dir 1/file 3.jpg
    /dest/file4.txt
    /dest/file 5.txt
    /dest/dir 2/file 1.jpg
    /dest/file_2.html

First sed invocation. This shows the generated script that will be interpreted by the second invocation of sed:

$ sed 's,\(..*\),/\\/\1$/ { s/^/cp "\1" "/; s/$/";/; p; },' src.txt
/\/file 1.jpg$/ { s/^/cp "file 1.jpg" "/; s/$/";/; p; }
/\/file_2.html$/ { s/^/cp "file_2.html" "/; s/$/";/; p; }
/\/file 3.jpg$/ { s/^/cp "file 3.jpg" "/; s/$/";/; p; }

Use shell command substition to use the output of the first sed command as a script on the command line passed to the second invocation of sed:

$ sed -n "$(sed 's,\(..*\),/\\/\1$/ { s/^/cp "\1" "/; s/$/";/; p; },' src.txt)" dest.txt
cp "file 3.jpg" "/dest/dir 1/file 3.jpg";
cp "file 1.jpg" "/dest/dir 2/file 1.jpg";
cp "file_2.html" "/dest/file_2.html";

Now, pipe the output of sed to the shell, with the xtrace option (sh -x). I don't have any of the files, hence the errors:

$ sed -n "$(sed 's,\(..*\),/\\/\1$/ { s/^/cp "\1" "/; s/$/";/; p; },' src.txt)" dest.txt  | sh -x
+ cp file 3.jpg /dest/dir 1/file 3.jpg
cp: cannot stat ‘file 3.jpg’: No such file or directory
+ cp file 1.jpg /dest/dir 2/file 1.jpg
cp: cannot stat ‘file 1.jpg’: No such file or directory
+ cp file_2.html /dest/file_2.html
cp: cannot stat ‘file_2.html’: No such file or directory