0

Let me preface this by saying that I am new to bash scripting. I was tasked with trying to redirect the content that is being written to a text file called "kiltslog.txt" to another directory on our network. Note: I did not write the following script: and it runs every 10 minutes ... what I don't know is how the above log file is being generated/updated from the following script (if at all). One more thing ... the "kiltslog.txt" file is being updated every 10 minutes and all of the "echo" statement starting with the "echo StartExec:" are written to that above log file. Also, I don't see any references to the "kiltslog.txt" file anywhere in the script.

#!/bin/bash

start_ts=`date +%s`
out_date=$(date -d @$start_ts --rfc-3339 ns)
echo StartExec: $out_date $start_ts

# min free space, in KB
# equal to 4TB
minSpaceThreshold=4294967296
echo "Checking free space..."

myUsed=$(df -k /nielsen_extracts  | tail -1 | awk '{print $3}')

## this can be collapsed into one awk function here:
df -k /nielsen_extracts  | tail -1 | /bin/awk '{if  (int ($3)>4294967296) print "Enough space: " $3;else print "Not enough space " $3 }'

if [[ $myUsed -gt $minSpaceThreshold ]] ;
then
        echo "We have enough free space:" $myUsed
else
        echo "We do not have enough free space:" $myUsed
        exit
fi


start_ts=`date +%s`
out_date=$(date -d @$start_ts --rfc-3339 ns)
echo DEVStartExec: $out_date $start_ts



# Use a lockfile containing the pid of the running process
# If script crashes and leaves lockfile around, it will have a different pid so
# will not prevent script running again.
#
lf=/tmp/pidLockFileKiltsProd
# create empty lock file if none exists
cat /dev/null >> $lf
read lastPID < $lf
# if lastPID is not null and a process with that pid exists , exit
[ ! -z "$lastPID" -a -d /proc/$lastPID ] && exit
echo not running
# save my pid in the lock file
echo $$ > $lf

nielsen_extracts_root=/nielsen_extracts
app_data_root=/nielsen_extracts/KiltsFilesRequests/AppData
requests_root=/nielsen_extracts/KiltsFilesRequests/AppData/Requests
queue_root=/nielsen_extracts/KiltsFilesRequests/AppData/Requests/queue
processing_root=/nielsen_extracts/KiltsFilesRequests/AppData/Requests/processing

complete_root=/nielsen_extracts/KiltsFilesRequests/AppData/Requests/complete
request_name=
request_id=
request_user=
request_parent_path=

tmp_curl_url=
tmp_curl_param_NewStatus=

##ls $requests_root

# check queue for requests
##echo $(date +Y%m%d)

# tar request in scratch space
#find $requests_root/queue -maxdepth 1 -type d  -print
DIRECTORIES=$(find $queue_root -mindepth 1 -type d)
for d in $DIRECTORIES
do
    echo "Processing $d directory..."
    echo "Moving $d from queue to processing directory..."  
    #cat $d/requestinfo | awk 'NR==2' | awk BEGIN { FS = ': " };
    request_id=$(cat $d/requestinfo | awk 'BEGIN { FS = ":" } ; { print $2 }' | awk 'NR==1' | sed -e 's/^ *//g' -e 's/ *$//g')
    request_name=$(cat $d/requestinfo | awk 'BEGIN { FS = ":" } ; { print $2 }' | awk 'NR==2' | sed -e 's/^ *//g' -e 's/ *$//g')
    request_user=$(cat $d/requestinfo | awk 'BEGIN {FS = ":" } ; {print $2}' | awk 'NR==3' | sed -e 's/^ *//g' -e 's/ *$//g')
    request_parent_path=$(cat $d/requestinfo | awk 'BEGIN {FS = ":" } ; {print $2}' | awk 'NR==4' | sed -e 's/^ *//g' -e 's/ *$//g')
    request_id="$(echo $request_id | tr '[a-z'] '[A-Z]')"
    echo "request_id: $request_id"
    echo "request_name: $request_name"
    echo "request_user: $request_user"
    echo "request_parent_path: $request_parent_path"
    echo "Updating status on front-end to PROCESSING..."
    tmp_curl_url="https://kiltsfiles.chicagobooth.edu/Services/UpdateRequestStatus.aspx?RequestID="
    tmp_curl_url="$tmp_curl_url$request_id"
    tmp_curl_url="$tmp_curl_url&NewStatus=P"
    echo "curling $tmp_curl_url ..."
    curl $tmp_curl_url | grep updated
    #exit
    cd $d
    pwd
    cd ..
    pwd
    echo "Moving request data to processing..."
    mv $request_id/ ../processing 
    #mv $d $processing_root
    # create the subfolder in scratch
    echo "Creating subdirectory in scratch /mnt/kiltGlobus/scratch/$request_id" 
    mkdir -p  /mnt/kiltGlobus/scratch/$request_id
    #tar the file list
    echo "Running tar process with cvz args on request..."
    tar cvz -T /nielsen_extracts/KiltsFilesRequests/AppData/Requests/processing/$request_id/filelist -f /mnt/kiltGlobus/scratch/$request_id/$request_name.tgz
    #exit
    echo "tar complete"
    echo "Moving request data to complete..."
    cd $processing_root
    pwd
    mv $request_id/ ../complete
    #move to the globus endpoint
    mkdir /mnt/kiltGlobus/RMS/$request_user
    echo "Moving file to Globus endpoint (RMS)..."
    mv /mnt/kiltGlobus/scratch/$request_id/$request_name.tgz /mnt/kiltGlobus/RMS/$request_user
    #finish with email notification and front-end update
    echo "Updating status on front-end to COMPLETE..."
    tmp_curl_url="https://kiltsfiles.chicagobooth.edu/Services/UpdateRequestStatus.aspx?RequestID="
        tmp_curl_url="$tmp_curl_url$request_id"
        tmp_curl_url="$tmp_curl_url&NewStatus=C"
        echo "curling $tmp_curl_url ..."
        curl $tmp_curl_url | grep updated
    echo "Cleaning up scratch dir..."
    rm -rf /mnt/kiltGlobus/scratch/$request_id
done

end_ts=`date +%s`
out_date=$(date -d @$end_ts --rfc-3339 ns)
echo EndExec: $out_date $end_ts

ts_diff=$(($end_ts-$start_ts))
echo ExecTime: $ts_diff
  • 1
    How is the script started every 10 minutes? By cron? – choroba Mar 31 '18 at 23:11
  • Thanks for your response ... I am going to meet with the administrator on Tuesday, to get some more answers. Is there some way to check this cron? – Paul T. Rykiel Mar 31 '18 at 23:16
  • in the etc directory there appears to be a cron.d, cron.hourly, cron.daily etc.. – Paul T. Rykiel Mar 31 '18 at 23:20
  • It seems this script is started by cron in this manner: /path/to/script > /path/to/kiltslog.txt or /path/to/script 1>/path/to/kiltslog.txt. One more thing is necessary for redirect to several files is using tee (copy stdout to stdin or file(s). For example: /path/to/script | tee /path/to/kiltslog.txt /path/to/another_file.txt | 1>/dev/null. – Yurij Goncharuk Mar 31 '18 at 23:26
  • See in /etc/crontab too. Maybe it's located there. – Yurij Goncharuk Mar 31 '18 at 23:30
1

The script is likely with its standard output rerouted to a file:

thescript >/path/to/kiltslog.txt

So you have to change whatever calls it (cron job or else).

  • Thank you, I am going to meet with our system admin, as I don't have root user access ... and I will mark this up once I find out and learn more about this. – Paul T. Rykiel Mar 31 '18 at 23:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.