0

I have a server which runs our backend scripts for our database (ETL processes, etc). However, for two hours a day we dedicate it to sending our e-mails. The e-mails are the top priority, however, sometimes they become very memory intensive and all processes on the server crash. Due to this I would want to temporarily pause all of the ETL scripts (written in Python) while the e-mail system is running. Once it has completed, I would then want it to resume those scripts once the e-mail system completes.

Let's say that the e-mail system runs on three crons with the following executables:

/usr/bin/php /home/Email/main.php --action=set1
/usr/bin/php /home/Email/main.php --action=set2
/usr/bin/php /home/Email/main.php --action=set3

I can wrap these three into a script file. However, in that script I need to pause and start all Python scripts. From searching on stack exchange it seems that I can run the following:

kill -TSTP [pid]
/usr/bin/php /home/Email/main.php --action=set1 &
/usr/bin/php /home/Email/main.php --action=set2 &
/usr/bin/php /home/Email/main.php --action=set3 &
wait
kill -CONT [pid]

where pid is the process ids of my Python scripts (there are multiple scripts). My question is, how can I adjust this to retrieve the pid's of all running Python scripts and loop through them in this? Note: this is being run on Ubuntu 14.04.2. Any other suggestions would also be appreciated. Thanks.

Solution

The solution on my system was the following:

pkill -STOP python
/usr/bin/php /home/Email/main.php --action=set1
/usr/bin/php /home/Email/main.php --action=set2
/usr/bin/php /home/Email/main.php --action=set3
pkill -CONT python
  • Look into renice to lower the priority of the etl jobs temporarily – Jeff Schaller Feb 27 '16 at 15:22
2

To find out pids of processes You can use pgrep

In Your case could use pkill.

pkill -TSTP Email 
pkill -CONT Email 

allthough probably better matching of process name would be wise

  • I've tried running pkill -TSTP python to pause all of the python processes, but they still seem to be running (they're still constantly writing to log files). Is there someway to debug this? – user2694306 Feb 27 '16 at 15:50
  • Could you try with SIGSTOP? I don't know the specific case, but Python is slow on handling signals (see SIGINT with Control-C in python). STOP cannot be intercepted by python. – Giacomo Catenazzi Feb 27 '16 at 16:39
  • The solution for me was to use "pkill -STOP python" and "pkill -CONT python" – user2694306 Feb 27 '16 at 18:55
2

Check killall or pkill (pgrep).

But I think the best way for our case: setup a simple send-only MTA which queue messages, and set it as high priority (with nice), so that your PHP code doesn't interrupt, and the sending (slow connection creating on network, slow initial connection with MTA on the other side [anti-spam feature]) can go in parallel, without slowing down the rest.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.