2

I have a cronjob that runs every 5 minutes, looking for wavs and mp3s of the same name in a source directory and if it finds a corresponding pair, it moves both to a destination directory.

The script is working very smoothly as far as I can tell, however some MP3s are being left in the source directory as well as appearing in the destination. Both copies are full.

The source directory is also the output of another process that churns out wavs and mp3s, and I tested it while writing the script to make sure that it wouldn't take files before they had been fully written. It appears that it holds the progressively written wav in a temp location until it is complete, at which point it moves it to the directory. So the output appears complete at a single instant. The wavs however, are written progressively. I was under the impression that the mp3s are written before the wav, in which case the script would work fine, since it wouldn't proceed until both wav and mp3 were present.

[edit: The process is as follows.

  1. Audio machine processes audio files and spits out a wav and a corresponding mp3 (with same name).
  2. These land in $INPUT although I can't be sure of the relative order in which they appear.
  3. My script (below) finds both wav and mp3, generates a hashsum for the wav, and moves both to the $OUTPUT. ]

Is it possible – if some mp3s are being written after the wav (entirely possible and unpredictable with our audio software) – that the mv command is taking not yet fully written mp3s, dropping them in the destination, but then continuing to update them?

I'm just confused as to why both source and destination copies are complete.

Here's the relevant bit of my script:

cd "$INPUT"
for x in *.wav
do if [ ! -e $x ]                   
then exit 0
else if [ -e ${x%.wav}.mp3 ]                
    then    md5sum $x > "$MD5DIR"/$x.md5 &&     
            mv $x "$OUTPUT" &&          
            mv ${x%.wav}.mp3 "$OUTPUT"      
    fi
fi

Result (for example):

$ ls $INPUT/
1.mp3
4.mp3

$ ls $OUTPUT/
1.wav    1.mp3
2.wav    2.mp3
3.wav    3.mp3
4.wav    4.mp3

[Edit 2014-03-12:

I think I know what is happening.

The earlier process that produced the wavs and mp3 files appears to write the id3 metadata tags last, after it has written the entire audio content of the MP3 file. My script then presumably took the MP3 after the audio content had been written but before the ID3 tags had been written. Comparing the files in an audio editor shows no difference, but comparing their id3 tags shows that only the copies left in the input directory had complete tags.

I see no way around this, but I discovered that the audio processing machine also produces a result log file for each audio file produced. So as a workaround, the presence of this results file is now the trigger for my script to run.

]

9
  • I think I just had a brainwave. Is it the case that mv creates a hardlink in the new location if the file is busy (i.e., currently being written to disk)? If so, then this behaviour would be predicted. I end up the same file in both source and destination.
    – Jangari
    Mar 7, 2014 at 0:15
  • You might want to read through this and clarify a bit. Things like The wavs however, are written [...] written before the wav are confusing. What is producing the wavs and mp3s? I am guessing the two are somehow related, is the mp3 made from the wav?
    – terdon
    Mar 7, 2014 at 3:47
  • Yes, sorry about that. I fixed up those two sentences (and added a bit explaining the order of events). What I was concerned about is the order in which the two files arrive. If it is mp3 then wav, then all is fine. If the reverse, I'm worried that mv is taking the mp3 before it's fully written to disk and causing this duplication of the mp3 file.
    – Jangari
    Mar 7, 2014 at 5:33
  • And you've confirmed that the duplicate mp3s are identical as well as complete?
    – terdon
    Mar 7, 2014 at 5:41
  • They have different hashes, but they appear to contain the same audio content.
    – Jangari
    Mar 7, 2014 at 5:55

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.