8

This is a minor issue, but it is a bit annoying. I have looked and looked at this, but I cannot quite figure out why it is happening or how to stop it. I have several thousand video files which I am slowly converting from MPEG II format to h.264 in MP4 containers for several reasons. Each .mpg file has a .txt file and a .jpg file associated with it, and these files need to be changed from "xxxxxxxxxxxxxx.mpg.txt and xxxxxxxxxxxxxx.mpg.jpg to xxxxxxxxxxxxxx.mp4.txt and xxxxxxxxxxxxxx.mp4.jpg, respectively, then the old .mpg files need to be deleted. I use the following script named mpgscan.sh to do that very thing:

#! /bin/bash

suffix=".mpg"
mpegname="$@"
filename=${mpegname%"$suffix"}
mp4name="$filename"".mp4"
if [ -s "$mp4name" ]
then
    if lsof "$mp4name" > /dev/null || lsof "$mpegname" > /dev/null
    then
        echo "$filename in use"
    else
        if [[ -f "$mpegname"".txt" ]]
        then
            rename -f 's/\.mpg.txt$/.mp4.txt/' "$mpegname"".txt"
        fi
        if [[ -f "$mpegname"".jpg" ]] 
        then
            rename -f 's/\.mpg.jpg$/.mp4.jpg/' "$mpegname"".jpg"
        fi
        echo "$filename removed"
        rm "$mpegname"
    fi
fi

The conversion of the files is going on in the background, so every once in a while I update the associated files in a batch by entering the following command:

find /RAID/Recordings -name "*.mpg" -exec mpegscan.sh {} \;

It works pretty well, but instead of getting an expected output like:

/RAID/Recordings/Movies/K/K-9 (Recorded Tue Apr 01, 2008, HBOHD) removed
/RAID/Recordings/Movies/K/Kate & Leopold (Recorded Sat Sep 11, 2010, ENCR1H) removed
/RAID/Recordings/Movies/K/Kindergarten Cop (Recorded Tue May 06, 2008, HBOHD) removed
/RAID/Recordings/Movies/K/King Ralph (Recorded Wed Jul 16, 2008, HBOHD) in use

I get:

/RAID/Recordings/Movies/K/K-9 (Recorded Tue Apr 01, 2008, HBOHD) removed
find: '/RAID/Recordings/Movies/K/K-9 (Recorded Tue Apr 01, 2008, HBOHD).mpg.jpg': No such file or directory
find: '/RAID/Recordings/Movies/K/K-9 (Recorded Tue Apr 01, 2008, HBOHD).mpg.txt': No such file or directory
/RAID/Recordings/Movies/K/Kate & Leopold (Recorded Sat Sep 11, 2010, ENCR1H) removed
find: '/RAID/Recordings/Movies/K/Kate & Leopold (Recorded Sat Sep 11, 2010, ENCR1H).mpg.jpg': No such file or directory
find: '/RAID/Recordings/Movies/K/Kate & Leopold (Recorded Sat Sep 11, 2010, ENCR1H).mpg.txt': No such file or directory
/RAID/Recordings/Movies/K/Kindergarten Cop (Recorded Tue May 06, 2008, HBOHD) removed
find: '/RAID/Recordings/Movies/K/Kindergarten Cop (Recorded Tue May 06, 2008, HBOHD).mpg.jpg': No such file or directory
find: '/RAID/Recordings/Movies/K/Kindergarten Cop (Recorded Tue May 06, 2008, HBOHD).mpg.txt': No such file or directory
/RAID/Recordings/Movies/K/King Ralph (Recorded Wed Jul 16, 2008, HBOHD) in use

I have tried both the rename and mv commands with the same result. I am flummoxed. First of all, how / why is the error being reported by the find command? The find command isn't issued against either the .jpg or .txt files, it is issued against .mpg files. More to the point, why is there any sort of error at all? The files being reported do exist and they are being properly processed by the mv / rename command. Not only that, but I even check in the script to make sure they do exist before renaming them. What am I missing? The script runs perfectly with no errors if I run it manually against any .mpg file.

5
  • Note: you have minor flaw in your mpegscan.sh. The line mpegname="$@" accepts all command line arguments, not just first one. It will break on mpegscan.sh 1.mpg 2.mpg which will make it try fo look for file "1.mpg 2.mpg.txt" (with space). You might actually want to make mpegscan.sh to accept multiple arguments (which is quite easy) because with it you can change \; to + in your find invocation and make it collect more file names before sending them to mpegscan.sh which will help with race condition issue @frostschutz told you.
    – legolegs
    Commented Dec 11, 2023 at 11:46
  • Proposed change (one liner because of SO comments limitations): suffix=".mpg"; while [ -n "$1" ]; do mpegname="$1"; shift; .... rest of the script ...; done Invocation: find /RAID/Recordings -name "*.mpg" -exec mpegscan.sh {} +
    – legolegs
    Commented Dec 11, 2023 at 11:46
  • True, but since the find sends one file name with spaces as the argument, its not really an issue. It also means I can send a filename with spaces and not have to worry about adding quotes around it.
    – LesRhorer
    Commented Dec 13, 2023 at 1:57
  • As I mentioned, the find could send multiple arguments at once using -exec ... {} + syntax which is often beneficial (it always handle spaces correctly no matter what). Also I do not advise to rely on supplying filenames-with-spaces as unquoted arguments, it is against common practices and also could fail on double spaces (I know people who use multiple spaces in file name because it is easier to read for them).
    – legolegs
    Commented Dec 14, 2023 at 13:34
  • I don't recommend it, either. I did not create the files, or the filenames. When I said "I", I meant the script.
    – LesRhorer
    Commented Dec 15, 2023 at 14:14

1 Answer 1

12

find reads a list of directory entries and processes them one by one. For each entry that matches your conditions, it -executes your shell script.

Your script then renames files, thus changing the list of directory entries that find is still processing. So find still has filenames on its to-do list which you already removed/renamed.

When find tries to process them, it complains about the file no longer existing.

Example:

# create files in random order
$ shuf -e {a..z} {a..z}.foo {a..z}.bar | xargs touch

# rename using find -exec mv
$ busybox find -name "[a-z]" -exec mv {}.foo {}.baz \;
find: ./i.foo: No such file or directory
find: ./t.foo: No such file or directory
find: ./d.foo: No such file or directory
...

I'm using busybox find here because GNU findutils find is already smart enough to handle this case. You might be using an older version, or even the busybox variant if this is an embedded shell on some NAS of some kind.

You can ignore this error if the missing files would have been ignored by your conditions anyway; alternatively, let find build the list of files first and process this list afterwards.

Sometimes it's enough to use find -print0 | xargs -0 pattern instead of find -exec pattern. It helps if your shellscript can process multiple arguments instead of just one.

8
  • Of course I could ignore the errors (tons of them)! That is not the point.
    – LesRhorer
    Commented Dec 10, 2023 at 16:42
  • No, it is not busybox or a NAS, nor is it older. It is a server running Debian Bookworm - the latest release. It is GNU findutils find 4.9.0, and apparently it isn't smart enough. It's difficult to see what is going on - the non-errors, that is, with all the detritus in the way.
    – LesRhorer
    Commented Dec 10, 2023 at 16:56
  • copy a small subset of your files someplace "safe" and do your testing there. It's too bad find doesn't have a -vvv options (superdebug).
    – shellter
    Commented Dec 10, 2023 at 19:28
  • 1
    find does have -D all or you could strace it. I did try with Debian Bookworm but unable to reproduce it. Is there something special about the filesystem? fuse or network mount perhaps? Sometimes these can trigger odd behavior as well. Commented Dec 10, 2023 at 19:33
  • @frostschutz : doah! -D ... (lots of fine grain options or all) is excellent! Tnx!
    – shellter
    Commented Dec 10, 2023 at 20:57

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .