Converting `for file in` to `find` so that my script can apply recursively

Question

I have this idea of running a bash script to check some conditions and using ffmpeg to convert all the videos in my directory from any format to .mkv and it is working great!

The thing is, I did not know that a for file in loop does not work recursively (https://stackoverflow.com/questions/4638874/how-to-loop-through-a-directory-recursively)

But I barely understand "piping" and am looking forward to seeing an example and clearing some uncertainties.

I have this scenario in mind that I think would help me a lot to understand.

Suppose I have this bash script snippet:

for file in *.mkv *avi *mp4 *flv *ogg *mov; do
target="${file%.*}.mkv"
    ffmpeg -i "$file" "$target" && rm -rf "$file"
done

What it does is, for the current directory, search for any *.mkv *avi *mp4 *flv *ogg *mov then declare the output to have its extension to be .mkv then afterwards delete the original file, then the output should be saved to the very same folder the original video is in.

1. How can I convert this to run recursively? if I use find, where to declare the variable $file? And where should you declare $target? Are all find just really one-liners? I really need to pass the file to a variable $file, because I will still need to run the condition check.

2. And, assuming that (1) is successful, how to make sure that the requirement "then the output should be save to the very same folder the original video is in" is satisfied?

Answer 1

With POSIX find:

find . \( -name '*.mkv' -o -name '*avi' -o -name '*mp4' -o -name '*flv' -o \
          -name '*ogg' -o -name '*mov' \) -exec sh -c '
  for file do
    target="${file%.*}.mkv"
    echo ffmpeg -i "$file" "$target"
  done' sh {} +

Replace echo with whatever command you want to use.

If you have GNU find or BSD find, you can use -regex:

find . -regex '.*\.\(mkv\|avi\|mp4\|flv\|ogg\|mov\)'

Answer 2

You've got this code:

for file in *.mkv *avi *mp4 *flv *ogg *mov; do
target="${file%.*}.mkv"
    ffmpeg -i "$file" "$target" && rm -rf "$file"
done

which runs in the current directory. To turn it into a recursive process you have a couple of choices. The easiest (IMO) is to use find as you suggested. The syntax for find is very "un-UNIX-like" but the principle here is that each argument can be applied with AND or OR conditions. Here, we're going to say "If this-filename matches OR that-filename matches Then print-it". The filename patterns are quoted so that the shell can't get hold of them (remember that the shell is responsible for expanding all unquoted patterns, so if you had an unquoted pattern of *.mp4 and you had janeeyre.mp4 in your current directory, the shell would replace *.mp4 with the match, and find would see -name janeeyre.mp4 instead of your desired -name *.mp4; it gets worse if *.mp4 matches multiple names...). The brackets are prefixed with \ also to keep the shell from trying to action them as subshell markers (we could quote the brackets instead, if preferred: '(').

find . \( -name '*.mkv' -o -name '*avi' -o -name '*mp4' -o -name '*flv' -o -name '*ogg' -o -name '*mov' \) -print

The output of this needs to be fed into the input of a while loop that processes each file in turn:

while IFS= read file    ## IFS= prevents "read" stripping whitespace
do
    target="${file%.*}.mkv"
    ffmpeg -i "$file" "$target" && rm -rf "$file"
done

Now all that's left is to join the two parts together with a pipe | so that the output of the find becomes the input of the while loop.

While you're testing this code I'd recommend you prefix both ffmpeg and rm with echo so you can see what would be executed - and with what paths.

Here is the final result, including the echo statements I recommend for testing:

find . \( -name '*.mkv' -o -name '*avi' -o -name '*mp4' -o -name '*flv' -o -name '*ogg' -o -name '*mov' \) -print |
    while IFS= read file    ## IFS= prevents "read" stripping whitespace
        do
            target="${file%.*}.mkv"
            echo ffmpeg -i "$file" "$target" && echo rm -rf "$file"
        done

Answer 3

Example snippet without piping (assumes you are giving the path as argument):

#!/bin/bash

backup_dir=/backup/

OIFS="$IFS"
IFS=$'\n'

files="$(find "$1" -type f -name '*.mkv' -or -name '*.avi' -or -name '*.mp4' -or -name '*.ogg' -or -name '*.mov' -or -name '*.flv')"

for f in $files; do
    # get path
    d="${f%/*}"
    # get filename
    b="$(basename "$f")"
    ttarget="${b%.*}.mkv"

    # this is your final target
    target="$d/$ttarget"
    echo $target
    # mv $f "$backup_dir" 
done

IFS="$OIFS"

The shell reads the IFS variable, which is set to (space, tab, newline) by default. Then it looks at each character in the output of find. So if it finds space it thinks it is end of the filename (file containing spaces for example "Sin City.avi" ill be treated as two files "Sin" and "City.avi"). So with IFS=$'\n' we are telling to split the input on newlines. And finally we restore old (default) IFS which is saved in $OIFS variable.
Or as suggested in comments may be better approach could be:

#!/bin/bash

backup_dir=/backup/

find "$1" -type f \( -name '*.mkv' -or -name '*.avi' -or -name '*.mp4' -or -name '*.ogg' -or -name '*.mov' -or -name '*.flv' \) -print0 | while IFS= read -r -d '' f
do
    # get path
    d="${f%/*}"
    # get filename
    b="$(basename "$f")"
    ttarget="${b%.*}.mkv"

    # this is your final target
    target="$d/$ttarget"
    echo $target
    # mv $f "$backup_dir"
done

Answer 4

Welcome to Unix :)

To answer some of your minor questions that answers to the main question didn't cover:

Shell scripting certainly has some rough edges, since a lot of things break on file names with spaces. And almost everything breaks on filenames with newlines (fortunately, nobody makes those on purpose). Filenames containing glob characters like [, ], and * are sometimes a problem, too. Sometimes it's just not worth writing hard-to-read shell code up to the standards of wooledge's BashGuide, for your own use, or for a one-off where you know your filenames aren't weird.

where to declare the variable:

Shell variables don't need to be declared. In bash, you can shopt -o nounset to make it an error to reference and unSET variable, but that's not quite the same thing as undeclared. Unsetting a variable can be useful. In a shell function, it is good practice to declare all your temporaries with local foo bar baz;, so you don't litter the shell environment with variables, or worse, step on the caller's variable of the same name.

I barely understand "piping".

When working with the shell, a lot of data passing happens by printing the data to stdout. Pipes send that data to another program, which reads it on stdin (and usually prints something on stdout). You can capture output into shell variables using command substitution, $(). e.g. for i in $( locate foo | grep bar );do echo "$i"; done. (this will break on filenames with spaces in them, like a lot of shell code if you aren't careful. Use read if you want to write scripts that are reliable.) locate prints, grep reads and prints, and the shell reads the output of grep. (The shell gets its hands on the output of grep by starting grep with its output connected to the input side of a pipe which the shell created. The shell reads the output side of the pipe.)

A pipe is just a way for programs to work like they're writing to a file, but actually they're writing into a small buffer. A process reading from a pipe will have its read(2) system call return when there's data available, which only happens when something writes to the other end of the pipe.

The shell's |, $(), and some other syntax elements are how you tell the shell how to set up the plumbing connecting programs to each other, and to the shell.

It's easy to learn bad idioms for shell programming, because a lot of the obvious things and old ways of doing things have hidden pitfalls which break on odd filenames. See for example http://mywiki.wooledge.org/BashFAQ/001.

Better to learn safe ways of scripting from the start, rather than learning ways that break on odd filenames, as long as they're not too clunky to type. :)

A lot of GNU utils have a -0 option, to use ASCII NUL (the 0 byte, can't be present in filenames or text) as a record separator. This lets you pipe data between find and sort, for example, without any possibility of having one "line" of find output turned into multiple lines of sort input. This ends up not being super useful when you want to get the data into a shell variable, because bash doesn't have a way to read \0-delimited lines. (I don't think that's a valid value for IFS.)

Anyway, avoiding having the shell treat data as code is the reason to always double-quote everything you can, unless you really WANT word-splitting. If you ever want to make your brain hurt looking at complex shell code, just look at the bash-completion code. (It handles the programmable completion that does clever things like completing ls --colo => --color, or only completing *.zip files for unzip.) set -x and hit tab :P. (set +x to turn off execution tracing.)

re: your for loop: with *.mkv as one of your patterns, you're going to have source = dest for those input files. ffmpeg will prompt you to overwrite the output file for each one.

Also, do you really need to transcode the audio? -c:a copy might be a good idea. Video bitrate is usually a bigger deal. And you might want to use -preset slow (or slower, or even veryslow) to get more quality per bitrate, at the cost of more CPU usage. There's also -crf 20 (default 23). https://trac.ffmpeg.org/wiki/Encode/H.264. You hopefully already knew this, and left it out because it wasn't relevant to the bash scripting, but just in case... :P -c:v libx264 is the default when outputting to mkv, so that's good.