Run python code from cron in an Anaconda environment

Question

I need to run a python script using a specific anaconda environment, since I need to execute this script several times per day I would like to do it using the linux crontab file (/etc/crontab).

When I launch the script from the linux command line it works fine:

~/anaconda2/envs/py36/bin/python ~/production/web_scraping.py

I added the process to the crontab file:

SHELL=/bin/sh
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
# m h dom mon dow user  command
57 08 * * * user ~/anaconda2/envs/py36/bin/python ~/production/web_scraping.py | touch ~/test.txt

The line is correctly executed by cron, this is the cron log:

user@exmachina:~$ sudo cat /var/log/syslog | grep web_*
Jan  2 08:57:02 localhost CRON[10854]: (user) CMD (~/anaconda2/envs/py36/bin/python ~/web_scraping.py | touch ~/test.txt) 

I added | touch test.txt to the cron line to ensure the line is executed, and the test.txt file is generated.

The python script uses Selenium to extract data from a web page, by means of the chrome webdriver. Can this be the reason the script runs fine from the command line but not from the cron file?

Thanks for your help!

Answer 1

Instead of | touch ~/test.txt it would be more informative to use | tee ~/test.txt (or tee -a).

Consider incorporating env|sort; id; date in the output you log.

You were quite vague in describing the selenium script, and the way in which it "does not run fine" from cron. Did all imports succeed, or is PYTHONPATH perhaps not correct? Would it help to run conda env update -f environment.yml and then have the cron job source activate xxx so the job pulls in appropriate library versions?

A few variables, including PATH and LD_LIBRARY_PATH, can be important for your browser, and when you view the logged values you may see they differ from the interactive environment where selenium is working fine. Part of setting up a cron job is copying aspects of the environment so the job will succeed.