0

I have made a mistake and dumped files together into the same directory. Luckily I can sort them based on the filename: ''' 2019-02-19 20.18.58.ndpi_2_2688_2240.jpg ''' Where the # bit or 2 in this specific case is the location information, as it were. The range is 0-9 and the filenames are all identical length so that number will always be in the same position of the filename (26th character inlucing spaces, flanked by underscores). I found this great link: https://askubuntu.com/questions/621063/command-to-find-files-by-searching-only-part-of-their-names

However, I can't pipe the output into the move command. I tried to loop the output into a variable but that didn't seem to work either:

for f in find . -name '*_0_*' ; do  mv "$f" /destination/directory ; done

Based on this link, I might have put the * or some quotes in the wrong place:mv: cannot stat No such file or directory in shell script

That said, I have many directories and I would like to sort them in to an identical directory structure somewhere else:

-Flowers (to be sorted)          -Flowers-buds               -Flowers-stems
     -Roses                          -Roses                      -Roses
        buds.jpg       ===>             buds.jpg     ===>           stems.jpg
        stems.jpg
        petals.jpg
     -Daisies                       -Daisies                    -Daisies
        buds.jpgs                       buds.jpg                   stems.jpg
        stems.jpg
        petals.jpg
     -Tulips                        -Tulips                     -Tulip
        buds.jpgs                       buds.jpg                  stems.jpg
        stems.jpg
        petals.jpg

...and more based on that number (#). Is this something that's practical do in bash? I'm running MacOS in the terminal with coreutils installed so the tools should behave like GNU linux, instead of darwin (BSD).

1

The shell is splitting input on whitespace. You can use bash globbing with the recursive ** to get the filenames split properly:

shopt -s globstar
for f in **/*_0_*; do mv "$f" /dest/; done

If they're all going to the same place, the loop is not needed:

shopt -s globstar
mv **/*_0_*  /dest/

... or use find -exec, which passes filenames directly from find to an exec() call, without the shell's involvement:

find . -name '*_0_*' -exec mv {} /dest/ \;

Or use read plus find -print0 to split on null. It's overkill for this problem, useful when globbing and find -exec aren't up to the task:

while IFS=  read -r -d ''; do mv "$REPLY" /dest/; done < <( find . -name '*_0_*' -print0 )

To change the top-level destination based on the file name, as in your example, cut off pieces of the filename using ${var#remove_from_start} and ${var%remove_from_end}. Wildcards in the remove patterns will remove as little as possible; to remove as much as possible, double the operation character, as in ${…##…} or ${…%%…}:

    shopt -s globstar
    for f in **/*_0_*; do            # Flowers/Roses/stems.jpg 
      file=${f##*/}                  # stems.jpg
      base=${file%.*}                # stems
      path=${f%/*}                   # Flowers/Roses
      toppath=${path%%/*}            # Flowers
      subpath=${path#*/}             # Roses
      dest=$toppath-$base/$subpath/  # Flowers-stems/Roses/
      [[ -d $dest ]] || mkdir -p "$dest"
      mv "$f" "$dest"
    done

  • The files found in one directory will all be going to the same destination, but I was hoping to scan all the subdirectories and move them over. I mean if files were found in folder1, they should be moved to folder1 in a different branch, branch1 say. Then the script would scan folder2 and then move the files found in tthat directory to folder2 in branch1. – SciGuy Jun 1 at 19:39
  • see edit for an example of how to do that – Oh My Goodness Jun 1 at 20:17
  • Sorry for the delay. Finally back to civilization. I tried your suggestion which looks good. Unfortunately, I get a mv: cannot stat '**/*_0_*': No such file or directory error. It turns out my shopt command doesn't have globstar, which I suspect is the problem. I have dotglob, extglob, failglob, and nullglob, but no globstar – SciGuy Jun 8 at 18:59
  • 1
    globstar was introduced in bash v4, circa 2010. If upgrading is not possible, a read loop (4th example above) can replace the ** glob loop. – Oh My Goodness Jun 9 at 1:19
3

You can do this with find and mv, but it isn't the simplest approach here. You're on macOS, so zsh is preinstalled. Zsh comes with a neat file mass-renaming tool called zmv. Run zsh in a terminal, then in zsh run something like:

autoload zmv
zmv -n '[^_]#_([0-9])_*' '/elsewhere/${1}/${f}'

Explanations:

  • -n tells zmv to display what it would do, but not actually do anything. When you're happy with what it shows, run the command again without -n.
  • The first non-option argument is a wildcard pattern. zmv will act on the matching files (and ignore non-matching files).
  • [^_]# means zero or more characters other than _. Zsh gives you access to the same wildcards as bash and more. # is a zsh-only feature (available in bash with a different syntax) that means “any number of the previous character”. The pattern [^_]#_[0-9]_* matches any file name that contains a single digit between two underscores, and no other underscore before that digit.
  • The parentheses around [0-9] make the digit available as $1 in the replacement text.
  • The replacement text can use ${1}, ${2}, etc. to refer to parenthesized fragments in the source pattern, and ${f} to refer to the full original name.

For example, the snippet above moves 2019-02-19 20.18.58.ndpi_2_2688_2240.jpg to /elsewhere/2/2019-02-19 20.18.58.ndpi_2_2688_2240.jpg. Your question doesn't describe what needs to be moved and where it needs to be moved to very precisely, but you should be able to get what you want by tweaking the command above. If you can't figure it out, edit your question to add specific examples of what must and must not be matched.

If you have files in subdirectories, you can use */ at the beginning of the pattern. If you need to match the directory to refer to it with ${NUM}, you need to put parentheses around the *: you can't put parentheses around a slash. If you want to traverse the directory recursively and match files at any debt, use **/ for recursive globbing. As an exception, you need to use (**/) to access the directory path in the replacement text as ${NUM}. In cases like this one, it's often easier to not use parentheses, and instead use modifiers on ${f} to extract parts of the original path as a whole. For example, do the same renaming as above, but moving files from the current directory to a parallel structure under /elsewhere, but with an extra level 0, 1, etc. just before the file name:

autoload zmv
zmv -n '**/[^_]#_([0-9])_*' '/elsewhere/${f:h}/${1}/${f:t}'

If it's difficult to match the files based on their name, a different approach would be to match them based on their change time¹. If you just copied a bunch of files to a directory which hadn't changed in a while, the new files are the ones that have a recent change time. You can match files by change time in zsh with the glob qualifier c. For example, to list the files last changed in the past hour:

ls -lctr *(ch-1)

Instead of finding a cutoff time, you can refer to the N most recently changed files with the glob qualifiers oc (to sort by increasing ctime) and [1,N] (to keep only the first N matches). For example, if you know that you just moved 42 files into that directory and haven't changed anything in it since then:

ls -lctr *(oc[1,42])

If you want to use glob qualifiers with zmv, you need to pass the -Q option. For example, to move files to a directory based on their name as above, but ignore all files that haven't changed in the past hour:

zmv -n -Q '[^_]#_([0-9])_*(ch-1)' '/elsewhere/${1}/${f}'

zmv has some safeties to check that it won't overwrite files, and that there are no collisions (two source files with the same destination name). If you have a huge number of files, these safeties may take time. If zmv is too slow and the structure of your renaming guarantees that there won't be any collisions, you can hard-code the specific renaming you're doing in a loop. Zsh tends to beat bash even if you don't use zmv thanks to its nicer globbing and parameter expansion mechanisms. For performance, you can dynamically load a module that contains builtins for file manipulation.

For example, to move regular files from under the current directory to a whole new hierarchy if their name contains _0_:

zmodload -m -F zsh/files 'b:zf_*'
for x in **/*_0_*(.); do
  zf_mkdir -p /destination/directory/$x:h
  zf_mv ./$x /destination/directory/$x
done

(:h is yet another zsh feature: a history modifier.)

To take the first number that is between two underscores in the file name, and put files in a directory named for that number, you'll need to extract the number. zmv does this automatically for parenthesized groups, here we need to do it by hand.

zmodload -m -F zsh/files 'b:zf_*'
for source in **/*_<->_*(.); do
  suffix=${${source:t}#*_<->_}
  n=${${source%$suffix}##*_}
  destination=${source:h}/$n/${source:t}
  zf_mkdir -p /destination/directory/$destination:h
  zf_mv ./$source /destination/directory/$destination
done

If you want to learn about other methods to mass-rename files, you can browse questions tagged rename on this site.

¹ The inode change time of a file, called “change time” or “ctime” for short, is the time the file was last moved or last had its attributes changed such as permissions. It is different from the modification time (mtime).

  • Is there a way to speed up zmv? I'm moving 100k's if not 1M files at a time and it seems to be quite slow. – SciGuy Jun 8 at 22:55
  • 1
    @SciGuy zmv plays it safe: it determines the complete list of files that it's going to move, and checks that there are no collisions. Usually you wouldn't notice it unless it points out a problem, but with a million files, the checks may well be very slow indeed. I think if you pass the -f option it bypasses some of these checks. If it's still too slow, then you may be better off writing a loop manually that handles your specific case. I've added examples to my answer. You'll have to adjust them because I don't know exactly what you do and do not want to rename. – Gilles Jun 9 at 0:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.