0

In my bash script, I want to update a string according to new written filenames with .dat format. Here is what I am trying to do:

  • For example, I use a file named blabla_3200.dat in my bash script, then run a case with this script.
  • After this case is done, a new blabla_3300.dat is written in the same directory
  • For the next run, I want to use this last blabla_3300.dat in the same bash script.

Therefore, I have to search for the largest integer in the file names .dat, and then use sed to update my bash script like:

sed -i 's/3200/max/g' mybash.sh

then run a new case.

Any help will be appreciated Have a good day!

Clearifying Note

I should be more clear:

Let's suppose i have blabla_3200.dat, submission.sh, bash.sh in same directory.

I told the program in submission.sh; read data blabla_3200.dat then start running.

To call this submission.sh file to slurm machine, i command sbatch submission.sh in bash.sh

Then end the end of run, program writes output file blabla_4500.dat in same directory(it is unknown what is going to write, it might blabla_8254.dat for example).

What i want this; the code in bash.sh should update read data command in submission.sh after each new output came. Now in submission.sh, read data blabla_4500.dat shoud be commanded.

5
  • 3
    Why do you have file names hard coded in your script? Why don't you just pass the file name as an argument? – terdon Feb 24 at 17:25
  • 1
    Or have the script itself search for the highest number, store it in a variable, and use that. – Gordon Davisson Feb 24 at 17:35
  • @terdon because i use this files (blabla_3300.dat) as an input for my program – hll Feb 25 at 5:13
  • @GordonDavisson Yeap i figured out what i should do, but i cannot have enough experience to do so. Bash script language (python) is very strange to me – hll Feb 25 at 5:15
  • "Bash script language (python)" shell script and python are totally different. Which one are you using? – roaima Feb 25 at 8:10
0

Use a soft link ln -s blabla_3300.dat .next.

Tell your script to read .next, then rm .next;ln -s blabla3400.dat .next.

2
  • It may be blabla_4300 though, i cannot be sure what is going to be written in next. It should be check the last .dat file itself – hll Feb 25 at 5:22
  • "Last" as in most recent? Please add that to your question because it adds valuable detail – roaima Feb 25 at 8:11
0

I came up with a simple example that calculates the SHA256 hash of each file as they appear in the directory. Harmless and easy to verify if it got that wrong. You could use any other processing.

#!/bin/sh
inotifywait -m "$@" -e create | while read dir event file; do
    if [ -f "${dir}${file}" ]; then
        sha256sum "${dir}${file}"
    fi
done

There are a few problems with this, not all of them are easy to solve.

  • What if there are newlines in the filename? It will process them as 2 separate elements, and worse, whoever is writing those files can force the script to work on some arbitrary other file. Theoretically you could use \0 - the only character that cannot appear in a filename - as the separator but I couldn't get it to work with $IFS and the --format option. Maybe it would be better to not use -m but repeatedly call inotifywait so you know that one output = one file. But then you'll miss many events!
  • Creating and quickly deleting the file quickly will miss events. It's actually worse than that. Someone could create a regular file, then your script executes [ -f ${dir}${file} ], then they delete it and create a symlink instead, then your script does its thing to an arbitrary file. So even if I got the \0 thing working, it wouldn't have mattered.
  • Slow processing. If processing takes longer than creating a file, there will be a large delay, making the above problems much worse. This can be easily fixed with a well placed & :)
  • Reading half-written files. This could be avoided by using the close_write event, or by creating the files somewhere else and moving them to the monitored directory when they are done.

I think this is one of those tasks that should not be done in a shell script. There needs to be some sort of synchronization between the reader and the writer here, and there may or may not be a trust boundary.

1
  • Thanks for this genius method. But i am afraid i could not use this method because i am not good enough. A simple but clear for loops would be more helpful for me i suppose :) – hll Feb 25 at 5:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.