1

I have the following script that will schedule a job based on input from the file "schedule.csv". The file contains 7 fields and the $f7 field contains a flag that either has "NOT_S" for a job that needs to be scheduled and "YES" if the job has already been scheduled.

#!/bin/bash
filename='/home/opc/3A-Lab/schedule.csv'

i=1
while IFS=, read -r f1 f2 f3 f4 f5 f6 f7; do

        case "$f7" in

        NOT_S)
                test  $i -eq 1 && ((i=i+1)) && continue
                echo "/home/opc/3A-Lab/3ALab.sh $f5 start" | at  now
                echo "/home/opc/3A-Lab/3ALab.sh $f5 stop"  | at  now + "$f6" " " "hours"
                ;;
        YES)
                echo "Already Scheduled"
                ;;
esac
done < $filename

echo "something is wrong"

When I run the script, I get the output of "something is wrong" which is something I added a troubleshooting step to alert me that for some reason the logic I have is not working.

What I want is the jobs that have the flag "NOT_S" to be scheduled and the ones that have "YES" to be ignored. Without the decision logic, the jobs will get scheduled as desired. However, the problem with this is that I then have no way of excluding the jobs that have already be scheduled once the script runs again. The file it's reading from is generated by a Google Form and I'm downloading the responses from Google Sheets via wget. I currently have Google Sheets setting the flag in column $f7 so that after a period of time, when the file is downloaded again and the script is run again on the server, it will only process the jobs that have been added to the sheet that have the "NOT_S" flag and ignore the other ones.

What am I doing wrong in my script as to why its not picking up on the value of $f7 and executing the commands to schedule the jobs? Thanks for any help you can provide.

EDIT:

Here is sample of what the schedule.csv file looks like:

Timestamp,Email Address,First Name,Last Name,Workstation Name,Duration (in hours),Schedule Flag
,,,,no workstation,,YES
1/1/2021 14:52:11,john.doe@domain.com,Jone Doe,no workstation,2,NOT_S
1/1/2021 15:39:48,jane.sue@domain.com,jane,sue,no workstation,2,NOT_S
1/1/2021 15:40:26,fred.flintstone,Fred,Flintsone,no workstation,5,YES
,,,,no workstation,,YES
,,,,no workstation,,
,,,,no workstation,,

So let me explain the file: A student comes to the Google form to book time for a virtual workstation. The form automatically retrieves their email address. Based on their email address, the sheet will populate with a workstation name associated with the student. Then the form asks them how much time do they want to reserve on the workstation. They can select between 1 to 5 hours from a drop down menu. Once that is complete, the form saves their entry onto the google sheet.

The sheet has a formula that will determine that uses the timestamp time +10 minutest to set the flag in field 7 to either NOT_S or YES. Once the 10 minutes are up, the flag gets set to YES. At the time of scheduling, it gets set to NOT_S.

Here is a sample output of the script scheduling the jobs without the conditional logic:

job 13984 at Sat Jan  2 14:41:00 2021
syntax error. Last token seen: hours
Garbled time
job 13985 at Sat Jan  2 14:41:00 2021
job 13986 at Sat Jan  2 16:41:00 2021
job 13987 at Sat Jan  2 14:41:00 2021
job 13988 at Sat Jan  2 16:41:00 2021
job 13989 at Sat Jan  2 14:41:00 2021
job 13990 at Sat Jan  2 19:41:00 2021
job 13991 at Sat Jan  2 14:41:00 2021
syntax error. Last token seen: hours
Garbled time
job 13992 at Sat Jan  2 14:41:00 2021
syntax error. Last token seen: hours
Garbled time
[opc@vm-control-server 3A-Lab]$ 

2nd EDIT:

Here is the output using the commands Gordon suggested:

[opc@vm-control-server 3A-Lab]$ file ./schedule.csv 
./schedule.csv: ASCII text, with CRLF line terminators
[opc@vm-control-server 3A-Lab]$ LC_ALL=C cat -vet ./schedule.csv 
Timestamp,Email Address,First Name,Last Name,Workstation Name,Duration (in hours),Schedule Flag^M$
,,,,no workstation,,YES^M$
1/1/2021 14:52:11,name@domain.com,Philip,Monroe,no workstation,2,NOT_S^M$
1/1/2021 15:39:48,name@domain.comi,John,Crocket,no workstation,2,NOT_S^M$
1/1/2021 15:40:26,name@domain.com,Eddie,Reed,no workstation,5,YES^M$
,,,,no workstation,,YES^M$
,,,,no workstation,,^M$
,,,,no workstation,,[opc@vm-control-server 3A-Lab]$ 

Edit 3:

Here is the updated file I'm running the script on:

[opc@vm-control-server 3A-Lab]$ LC_ALL=C cat -vet ./schedule.csv 
Timestamp,Email Address,First Name,Last Name,Workstation Name,Duration (in hours),Schedule Flag^M$
,,,,no workstation,,YES^M$
1/1/2021 14:52:11,name@domain.com,Philip,Monroe,no workstation,2,NOT_S^M$
1/1/2021 15:39:48,name@domain.com,John,Crocket,no workstation,2,NOT_S^M$
1/1/2021 15:40:26,name@domain.com,Fred,Brown,no workstation,5,YES^M$
,,,,no workstation,,YES^M$
,,,,no workstation,,^M$
,,,,no workstation,,[opc@vm-control-server 3A-Lab]$ 

Output of the script. Keep in mind, this running without any decision logic, so it's not taking into account f7 field so it's scheduling all jobs on the sheet.

[opc@vm-control-server 3A-Lab]$ ./final2.sh
job 13993 at Sat Jan  2 15:38:00 2021
syntax error. Last token seen: hours
Garbled time
job 13994 at Sat Jan  2 15:38:00 2021
job 13995 at Sat Jan  2 17:38:00 2021
job 13996 at Sat Jan  2 15:38:00 2021
job 13997 at Sat Jan  2 17:38:00 2021
job 13998 at Sat Jan  2 15:38:00 2021
job 13999 at Sat Jan  2 20:38:00 2021
job 14000 at Sat Jan  2 15:38:00 2021
syntax error. Last token seen: hours
Garbled time
job 14001 at Sat Jan  2 15:38:00 2021
syntax error. Last token seen: hours
Garbled time
[opc@vm-control-server 3A-Lab]$ 

Edit 4:

Olivier's modifications are working great. However, I need to solve another problem that I hope you all can help me with. So the google sheet that I'm collecting the responses from the google form has a formula on it that checks the timestamp of the requested entry + 10 minutes to see if its >= to the now() time and if so, it changes the flag to YES. Well it works great on the form and ther flag gets changed, however, when my cron job pulls the next wget of the file, it doesn't recognize the changes to the f7 field and still has the entry as NOT_S so my script will continue to schedule jobs that have already been scheduled.

What I ultimately want to do is as soon as the job is scheduled, immediately set the flag to YES so that once the file gets read again, it will get ignored.

I was trying to figure out how to use wget to only pull in the changes to the sheets file and not have to download the whole thing each time. I want to set up something where I can write out to a local file on the Linux server and just have the script update that local file with the flag change and only take in the new entries from the google sheet. I would appreciate some help with this. Thanks.

9
  • Why are you expecting the final echo to not be executed? Could you possibly show us the schedule.csv file and the output from running the code? Also, was the CSV file created on a Windows or DOS system with CRLF newlines? – Kusalananda Jan 2 at 19:10
  • Sure I will post what the file looks like. It's a comma-delimited file. The file is generated by a Google form I have setup on the web. The file is downloaded every minute to a linux server. I then have the scripts on the server to process the file/jobs. – Eddie Reed Jan 2 at 19:20
  • 2
    Is the CSV file a DOS text file with CRLF newlines? If so, you may want to convert it to a Unix text file using dos2unix. – Kusalananda Jan 2 at 19:27
  • Please add output of this command to your question: file /home/opc/3A-Lab/schedule.csv – Cyrus Jan 2 at 19:40
  • 1
    @EddieReed That doesn't mean it's not in DOS/Windows format. Check it with the file command. Also, try printing it with LC_ALL=C cat -vet /home/opc/3A-Lab/schedule.csv -- this should print it normally except that there'll be a "$" at the end of each line. If there's anything else different about the output (like spaces before the "$", or a "^m", or whatever), that should indicate there's unexpected content in the file that you'll need to take into account. – Gordon Davisson Jan 2 at 20:06
2

Your input files have ^M (cr) characters before the unix newline (lf). Therefore, your f7 field is not what is expected.

Pass the file through: dos2unix < oldfile > newfile

or : tr -d '\015' < oldfile > newfile

And try again.

Make also sure:

  • all input lines have 7 arguments (empty or not, ie have 6 commas): awk -F',' '(NF != 7) { print "line: " NR " has " NF " arguments : " $0 }' /home/opc/3A-Lab/schedule.csv
  • add a default line for your case to notify which lines do not correspond to either cases: *) printf "Something is wrong: We have a line with an unexpected f7='%s'\n" "$f7" ;; # before the esac.
  • delete the last echo : it would be always displayed, whatever the while does. or just modify it to say : echo "End of the script."
  • and the line with hours has an extra empty arguments: change at now + "$f6" " " "hours" into at now + "$f6" hours

So the script becomes:

#!/bin/bash
filename='/home/opc/3A-Lab/schedule.csv'

i=0
while IFS=, read -r f1 f2 f3 f4 f5 f6 f7; do
    i=$(( i + 1 ))
    if [ "$i" = "1" ]; then printf "Bypass first line\n" ; continue ; fi
    if [ -z "$f1" ]; then printf "Bypass line: %s, empty field f1\n" "$i" ; continue ; fi
    case "$f7" in
    NOT_S)
            printf "/home/opc/3A-Lab/3ALab.sh %s start\n" "$f5" | at  now
            printf "/home/opc/3A-Lab/3ALab.sh %s stop\n"  "$f5" | at  now + "$f6" hours
            printf "scheduled start and stop for line: %s\n" "$i"
            ;;
    YES)
            printf "line %s: YES : Already Scheduled\n" "%i"
            ;;
    "")     printf "Warning: I see an empty f7... for line: %s =  '%s,%s,%s,%s,%s,%s,%s'\n" "$i" "$f1" "$f2" "$f3" "$f4" "$f5" "$f6" "$f7"
            ;;
    *)
            printf "Something is wrong: line %s: we see f7='%s'\n No action taken.\n" "$i" "$f7"
            ;;
    esac
done < $filename

printf "End of the Script.\n"

I am also puzzled as to why you don't want to do anything with the first NOT_S line encountered (ie, where i=1) ... But you probably have your reasons.

edit1: changed echo to printf everywhere, good habit to take (printf is portable, echo isn't.) edit2: changed the script following the comment about bypassing title and comments (and the reason for $i)

9
  • so the idea with the i=1 line is to skip the header line and the 2nd line in the sheet has all my formulas for my sheet so entries to my sheet don't start until line 3. If you have a better way skipping those lines, I would appreciate it. Thanks. – Eddie Reed Jan 2 at 21:32
  • @EddieReed : note that you only increase i in case you are in a "NOT_S" line AND i==1. So in effect it : bypasses the first "NOT_S" line. – Olivier Dulac Jan 2 at 21:38
  • @EddieReed : I updated the proposed script in the end to : bypass the first (title) line, and bypass any lines where $f1 is empty (you can modify those tests if you need better ones). – Olivier Dulac Jan 2 at 21:43
  • 1
    I made the initial change of converting the file to Unix format and ran the script with my basic logic and it works as intended. Thank you very much. I will update the script with your enhancements and run to see how it works. – Eddie Reed Jan 2 at 22:40
  • 1
    Olivier, the script works great! Thank you. One change had to make was to add ;; at the end of the case block for ""). Can you update your code on this post so that its 100% correct so that users that need to reference this solution will get code that is complete? Thanks again for your awesome assistance. – Eddie Reed Jan 3 at 1:49
1

As has been explained elsewhere, your file's lines are terminated with Windows-style CRLF endings. This means that your last field always ends with a CR character, so your literal NOT_S and YES can never match.

You can either pass the file through a convertor, or more easily just ignore trailing characters on the end of the last field:

case "$f7" in
    NOT_S*) ... ;;
    YES*) ... ;;
esac
2
  • Thank you for this code. Makes it so I don't have to convert the file before processing. – Eddie Reed Jan 2 at 22:55
  • 1
    Or you can use f7=${f7%$'\r'} to remove a trailing CR. – glenn jackman Jan 3 at 3:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.