How to change integer in bash script according to written .dat filenames

Question

In my bash script, I want to update a string according to new written filenames with .dat format. Here is what I am trying to do:

• For example, I use a file named blabla_3200.dat in my bash script, then run a case with this script.
• After this case is done, a new blabla_3300.dat is written in the same directory
• For the next run, I want to use this last blabla_3300.dat in the same bash script.

Therefore, I have to search for the largest integer in the file names .dat, and then use sed to update my bash script like:

sed -i 's/3200/max/g' mybash.sh

then run a new case.

Any help will be appreciated Have a good day!

Clearifying Note

I should be more clear:

Let's suppose i have blabla_3200.dat, submission.sh, bash.sh in same directory.

I told the program in submission.sh; read data blabla_3200.dat then start running.

To call this submission.sh file to slurm machine, i command sbatch submission.sh in bash.sh

Then end the end of run, program writes output file blabla_4500.dat in same directory(it is unknown what is going to write, it might blabla_8254.dat for example).

What i want this; the code in bash.sh should update read data command in submission.sh after each new output came. Now in submission.sh, read data blabla_4500.dat shoud be commanded.

Answer 1

Use a soft link ln -s blabla_3300.dat .next.

Tell your script to read .next, then rm .next;ln -s blabla3400.dat .next.

Answer 2

I came up with a simple example that calculates the SHA256 hash of each file as they appear in the directory. Harmless and easy to verify if it got that wrong. You could use any other processing.

#!/bin/sh
inotifywait -m "$@" -e create | while read dir event file; do
    if [ -f "${dir}${file}" ]; then
        sha256sum "${dir}${file}"
    fi
done

There are a few problems with this, not all of them are easy to solve.

• What if there are newlines in the filename? It will process them as 2 separate elements, and worse, whoever is writing those files can force the script to work on some arbitrary other file. Theoretically you could use \0 - the only character that cannot appear in a filename - as the separator but I couldn't get it to work with $IFS and the --format option. Maybe it would be better to not use -m but repeatedly call inotifywait so you know that one output = one file. But then you'll miss many events!
• Creating and quickly deleting the file quickly will miss events. It's actually worse than that. Someone could create a regular file, then your script executes [ -f ${dir}${file} ], then they delete it and create a symlink instead, then your script does its thing to an arbitrary file. So even if I got the \0 thing working, it wouldn't have mattered.
• Slow processing. If processing takes longer than creating a file, there will be a large delay, making the above problems much worse. This can be easily fixed with a well placed & :)
• Reading half-written files. This could be avoided by using the close_write event, or by creating the files somewhere else and moving them to the monitored directory when they are done.

I think this is one of those tasks that should not be done in a shell script. There needs to be some sort of synchronization between the reader and the writer here, and there may or may not be a trust boundary.