Backing up a large file

Question

I have a test running which crashes. I need to find out how far it reached before crashing. It will restart again after crashing and wipe the log file. This wiping takes 10 minutes though. So If I'm saving the log file every 9 minutes, I should be able to get the log file.

The problem is that how do I write a bash script to copy a log file to a specific location and overwrite an existing file (if any) only if the file being copied is larger than the file at the destination. Else I want to leave it alone.

 25       if [ -f $testFile ]; then
 26         COUNT=`cat $testFile|wc -l`
 27                 if [ $COUNT -gt 0 ];then
 28                     ARGS=`head -1 $testFile`
 29                     echo "Executing test for ARGS"
 30                     sed -i '1d' $testFile
 31                     cd /testCode; sleep 600

At this point, the code sleeps for 10 minutes, as mentioned

 32                     /testCode/startTest.sh $ARGS

This starts the test which removes the log file. If the test did not crash, then when the test is done it goes to the next command. Else it starts again from the top

 33                     `wall "System is going down for reboot in 600 seconds. Please save your work"`
 34                     sleep 600
 35                     /sbin/reboot &
 36                     exit 0
 37                 fi
 38 fi

UPDATE #1

I figured out a way. So when the test starts, I save the logs from the previous run. Then I have enough time to check the logs while the test has restarted. The test usually crashes after an hour or so with segfaults. That gives me ample time to check the logs.

It is not a really clean way, but it will work. @Alexandre Alves also gave a good answer. That can work too. @Slm gave a good answer, but unfortunately that will not work in large tests like mine since that will clutter up the log file.

Also my issue is saving the log file, not logging per se.

Answer 1

I would turn on debugging for the line that starts the test where it's crashing. You can use one of the techniques outlined below:

1. Debug entire script from calling script

   bash -x /testCode/startTest.sh $ARGS
   

2. Debug the script itself

   Or if you have control of this script you could add the shebang line at the top like this:

   #!/bin/bash -x
   

3. Debugging specific sections of code

   Or you can do this around sections of the script where you want more verbosity:

   line1
   line2
   
   set -x
   line3
   line4
   set +x
   
   line5
   

Answer 2

I'm guessing that you cannot change the way the test logs it's logfile.

But I can suggest two ways of doing that:

1. If you really want a bash script then

   #!/bin/bash
   source_size=`du -b file1|cut -f 1`
   destin_size=`du -b file2|cut -f 1`
   if [ $source_size -gt $destin_size ]; then
       mv file1 file1.bck
       touch file1
       cp file1.bck /destination/file2
       rm file1.bck
   fi
   

   Note that I'm doing the move and then copy the file because if you are moving the file between different file systems it will have to copy then delete. This mean that you could run into the situation of the file being overwritten by the test while being moved.

2. Using tee

   If you know when the problem is about to happen then you could do

   tail -f logfile |tee newlog
   

   This will show to the stout the logfile while writing at the same time to the newlog file.

3. Change the way the test creates the logfiles to perform logfile rotation.