14

I have a Anaconda Python Virtual Environment set up and if I run my project while that virutal environment is activated everything runs great.

But I have a cronjob configured to run it every hour. I piped the output to a log because it wasn't running correctly.

crontab -e:

10 * * * * bash /work/sql_server_etl/src/python/run_parallel_workflow.sh >> /home/etlservice/cronlog.log 2>&1

I get this error in the cronlog.log:

Traceback (most recent call last):
  File "__parallel_workflow.py", line 10, in <module>
    import yaml
ImportError: No module named yaml

That is indicative of the cronjob somehow not running the file without the virtual environment activated.

To remedy this I added a line to the /home/user/.bash_profile file:

conda activate ~/anaconda3/envs/sql_server_etl/

Now when I login the environment is activated automatically.

However, the problem persists.

I tried one more thing. I changed the cronjob, (and I also tried this in the bash file the cronjob runs) to explicitly manually activate the environment each time it runs, but to no avail:

10 * * * * conda activate ~/anaconda3/envs/sql_server_etl/ && bash /work/sql_server_etl/src/python/run_parallel_workflow.sh >> /home/etlservice/cronlog.log 2>&1

Of course, nothing I've tried has fixed it. I really know nothing about linux so maybe there's something obvious I need to change.

So, is there anyway to specify that the cronjob should run under a virutal environment?

1
11

Posted a working solution (on Ubuntu 18.04) with detailed reasoning on SO.

The short form is:

1. Copy snippet appended by Anaconda in ~/.bashrc (at the end of the file) to a separate file ~/.bashrc_conda

As of Anaconda 2020.02 installation, the snippet reads as follows:

# >>> conda initialize >>>
# !! Contents within this block are managed by 'conda init' !!
__conda_setup="$('/home/USERNAME/anaconda3/bin/conda' 'shell.bash' 'hook' 2> /dev/null)"
if [ $? -eq 0 ]; then
    eval "$__conda_setup"
else
    if [ -f "/home/USERNAME/anaconda3/etc/profile.d/conda.sh" ]; then
        . "/home/USERNAME/anaconda3/etc/profile.d/conda.sh"
    else
        export PATH="/home/USERNAME/anaconda3/bin:$PATH"
    fi
fi
unset __conda_setup
# <<< conda initialize <<<

Make sure that:

  • The path /home/USERNAME/anaconda3/ is correct.
  • The user running the cronjob has read permissions for ~/.bashrc_conda (and no other user can write to this file).

2. In crontab -e add lines to run cronjobs on bash and to source ~/.bashrc_conda

Run crontab -e and insert the following before the cronjob:

SHELL=/bin/bash
BASH_ENV=~/.bashrc_conda

3. In crontab -e include at beginning of the cronjob conda activate my_env; as in example

Example of entry for a script that would execute at noon 12:30 each day on the Python interpreter within the conda environment:

30 12 * * * conda activate my_env; python /path/to/script.py; conda deactivate

And that's it.

You may want to check from time to time that the snippet in ~/.bashrc_conda is up to date in case conda updates its snippet in ~/.bashrc.

5
  • I'm curious, why should one copy the anaconda part from .bashrc to a separate file instead of just setting BASH_ENV=~/.bashrc in crontab? Oct 4 '20 at 15:17
  • Because in Ubuntu, there is a line in .bashrc that exits if it's not in an interactive shell (after the comment # If not running interactively, don't do anything). I've mentioned this in the longer post on SO. Also mentioned here
    – Jean Monet
    Oct 5 '20 at 8:14
  • 1
    I see. I'm on Mac OS and have a .bash_profile file instead of .bashrc and don't see any interactive-only parts in it, so I'll use it directly for now (to prevent any deviations between the conda initialize blocks). Oct 5 '20 at 8:54
  • About that option to use conda activate, shouldn't the commands be connected by && instead of ;? Feb 5 at 5:49
  • @ApiwatChantawibul ; works great. && should work as well, I haven't tested.
    – Jean Monet
    Feb 5 at 8:21
4

Found answer on stack over flow:

https://stackoverflow.com/questions/3287038/cron-and-virtualenv

The solution is to reference the python executable within the virtual environment itself. In my case, I changed the bash file to run this executable:

/home/etlservice/anaconda3/envs/sql_server_etl/bin/python

2
  • 2
    This is not a fully proper solution. It might appear to work, but just using the python executable for the given conda environment is not the same as activating the conda environment and running the python executable. Environment activation performs several additional steps besides just setting PATH (for gdal users, the GDAL_DATA env variable gets properly set).
    – Ben Root
    May 31 '19 at 19:51
  • 1
    I found a better answer here
    – Wassadamo
    Oct 15 '19 at 22:50
3

This is a solution that has worked well for me.

source /root/miniconda3/etc/profile.d/conda.sh && \
conda activate <your_env> && \
python <your_application> &

I am using miniconda with Conda version 4.7.12 on a Ubuntu 18.04.3 LTS.

I am able to place the above inside a script and run it via crontab as well without any trouble.

3
  • Any idea how to do it for conda ? Mar 15 '20 at 18:57
  • 1
    I would assume that the only change will be to the conda.sh path. Kindly search for conda.sh in your installation and point to it. Mar 18 '20 at 20:17
  • it worked after I included #!/bin/bash and rand it using bash, but when tried with #!/bin/sh and executing it with sh it gave an error source not found Jul 6 at 11:37
2

This solution works to me.

Try to use

source /path/to/conda/bin/activate my_env

instead of

conda activate my_env

in your bash script run_parallel_workflow.sh.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.