0

The following bash command reads the input file then prints each line with a 15 second pause between them:

IFS=$'\n';for line in $(cat file.txt | tail -n +2); do echo $line && sleep 15; done;

However, since there are long delays between each line, I might modify the input file while this command is running.

For example, I might modify the fifth line while the command has only printed the first three lines. 30 seconds later when the command gets to the fifth line and prints it, I want it to show the updated line, instead of the file as it was when the command was first executed.


Example input file.txt

Continents
Africa
Antarctica
Asia
Australia
Europe
N.America
S.America

There is no order in the input file

Modified input file.txt
Africa
Antarctica
Europe
Asia
Australia
N.America
S.America

1

You'll have to read the file each time it runs, try this:

num=1
while true; do
if [[ $num < $(wc -l file.txt) ]]; then
    awk "NR==$num" file.txt && let "num++"
else
    break
fi
sleep 15
done

This will print one line, sleep for 15 seconds then continue, it will exit when there are no more lines. NR is a built in awk variable of Number of Records or line number.

1

Use the following and only append to the file (or be very careful how you edit it)

awk '{ system("echo "$0"; sleep 15") }' file.txt

The Command

The main problem you are having is with how you are reading the file $(cat file.txt | tail -n +2) will get executed first—read the whole file and produce a list of lines to loop over before the loop even starts. At which point it does not matter what happens to the file as you are done reading it. You then loop over these lines with your expensive command.

Instead you want an application to read and process the lines one at a time. You can do this with awk and the system command.

awk '{ system("echo "$0"; sleep 15") }' file.txt

Here we use awk to open the file and process it line by line. On each line it will execute the command "echo "$0"; sleep 15" in a shell. Note the quotes are important $0 must be outside the quotes for sed to replace it with the current line, otherwise it gets treated as part of the command.

Modifying the file

The command however is likely not your only problem. Under the hood most applications modify files by writing to a completely new file and then deleting the old and renaming the new file when you click save. The do this for safety - if they crash half way through writing the file it does not matter as the original file remains intact, only after they have successfully written a file do they delete the old file in an very fast operation making it look like the file was modified in place. This means that awk (or what ever command you might use) needs to close and open the file again in order to start reading the modifications.

Unfortunately most commands that modify files do it in this way. You can however safely append to a file such as

echo "Test" >> file.txt

And the above command will start processing it once it has finished all the other countries.

All of this is actually does for good reason and you are glossing over a huge amount of race conditions and corner cases that would be possible if it were simple to do what you require. For one, what happens if you modify a file before the line you want? File position is tracked by byte position not line number so if the line you modify is a different length then your application will lose where it is in the file. Also you must consider that even if an application can modify a file in place it is actually writing the whole file again from the start rather than just updating the bytes that have changed (again due to the line you are adding/moving might be of different size to the original).

Alternatively you can read the file each time and keep track of the line number you are one - this resolves some of the issues above but there are still some corner cases to consider such as what happens if you add a line before the line that is currently being processed? In a trivial solution you will process the current line twice which may or may not be an issue. Your absolute best move is to write your own application to read the file, reread it if it gets modified/deleted from under you and processes the changes in the file manually to resolve any discrepancies but this would involve a lot more work and a simple append only method like above might be all that is required for your situation.

  • 1
    This doesn't actually work. Not even when only appending to the file (which isn't what the OP asked for) since awk has already read the input, despite not having processed it. Presumably, it uses some sort of buffer. Also, why would you use a system call and echo of all things? You're already using awk, just have it do the printing! – terdon Oct 26 '17 at 9:21
  • awk does not read the entire file at once, it processes it line by line for efficiency else it would not handle very large files at all and from trying it locally it worked fine when appending with bash redirection (with awk version 4.1.4 on archlinux) it works as described. I used the systemcall as I assumed that the echo $line; sleep 15 was a mock for a larger more complex script in spirit of the original question. – Michael Daffin Oct 26 '17 at 17:28
  • I stand corrected. I had also tested and the new line was not printed. I did the same thing now, however, and it was. Perhaps it depends on the versions used? I tested on 2 different machines. But I didn't mean that awk will read the entire file, I was suggesting that it might read a few lines at a time into a buffer, process those line by line and then read the next. – terdon Oct 26 '17 at 17:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.