0

I have a Anaconda Python Virtual Environment set up and if I run my project while that virutal environment is activated everything runs great.

But I have a cronjob configured to run it every hour. I piped the output to a log because it wasn't running correctly.

crontab -e:

10 * * * * bash /work/sql_server_etl/src/python/run_parallel_workflow.sh >> /home/etlservice/cronlog.log 2>&1

I get this error in the cronlog.log:

Traceback (most recent call last):
  File "__parallel_workflow.py", line 10, in <module>
    import yaml
ImportError: No module named yaml

That is indicative of the cronjob somehow not running the file without the virtual environment activated.

To remedy this I added a line to the /home/user/.bash_profile file:

conda activate ~/anaconda3/envs/sql_server_etl/

Now when I login the environment is activated automatically.

However, the problem persists.

I tried one more thing. I changed the cronjob, (and I also tried this in the bash file the cronjob runs) to explicitly manually activate the environment each time it runs, but to no avail:

10 * * * * conda activate ~/anaconda3/envs/sql_server_etl/ && bash /work/sql_server_etl/src/python/run_parallel_workflow.sh >> /home/etlservice/cronlog.log 2>&1

Of course, nothing I've tried has fixed it. I really know nothing about linux so maybe there's something obvious I need to change.

So, is there anyway to specify that the cronjob should run under a virutal environment?

1

Found answer on stack over flow:

https://stackoverflow.com/questions/3287038/cron-and-virtualenv

The solution is to reference the python executable within the virtual environment itself. In my case, I changed the bash file to run this executable:

/home/etlservice/anaconda3/envs/sql_server_etl/bin/python

  • 1
    This is not a fully proper solution. It might appear to work, but just using the python executable for the given conda environment is not the same as activating the conda environment and running the python executable. Environment activation performs several additional steps besides just setting PATH (for gdal users, the GDAL_DATA env variable gets properly set). – Ben Root May 31 at 19:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.