1

I need to run a python script using a specific anaconda environment, since I need to execute this script several times per day I would like to do it using the linux crontab file (/etc/crontab).

When I launch the script from the linux command line it works fine:

~/anaconda2/envs/py36/bin/python ~/production/web_scraping.py

I added the process to the crontab file:

SHELL=/bin/sh
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
# m h dom mon dow user  command
57 08 * * * user ~/anaconda2/envs/py36/bin/python ~/production/web_scraping.py | touch ~/test.txt

The line is correctly executed by cron, this is the cron log:

user@exmachina:~$ sudo cat /var/log/syslog | grep web_*
Jan  2 08:57:02 localhost CRON[10854]: (user) CMD (~/anaconda2/envs/py36/bin/python ~/web_scraping.py | touch ~/test.txt) 

I added | touch test.txt to the cron line to ensure the line is executed, and the test.txt file is generated.

The python script uses Selenium to extract data from a web page, by means of the chrome webdriver. Can this be the reason the script runs fine from the command line but not from the cron file?

Thanks for your help!

  • I would suggest using the full path and not tilde in cron jobs. – Jeight Oct 1 at 21:59
0

Instead of | touch ~/test.txt it would be more informative to use | tee ~/test.txt (or tee -a).

Consider incorporating env|sort; id; date in the output you log.

You were quite vague in describing the selenium script, and the way in which it "does not run fine" from cron. Did all imports succeed, or is PYTHONPATH perhaps not correct? Would it help to run conda env update -f environment.yml and then have the cron job source activate xxx so the job pulls in appropriate library versions?

A few variables, including PATH and LD_LIBRARY_PATH, can be important for your browser, and when you view the logged values you may see they differ from the interactive environment where selenium is working fine. Part of setting up a cron job is copying aspects of the environment so the job will succeed.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.