1

I recently got a new DigitalOcean LEMP environment with Ubuntu. I tried making an SH script that uses CasperJS to scrape data from an external website to a JSON file, and a PHP script to parse the JSON and update a MySQL database. The SH is executed by a cron job. My crontab is as follows, meant to run every 90 minutes.

# m h  dom mon dow   command
0 0,3,6,9,12,15,18,21 * * * /bin/sh /usr/share/nginx/private/mpd_calls_for_service/scrape_mpd_calls_cron.sh
30 1,4,7,10,13,16,19,22 * * * /bin/sh /usr/share/nginx/private/mpd_calls_for_service/scrape_mpd_calls_cron.sh

And the SH file in question is.

#!/bin/sh
/usr/local/bin/casperjs /usr/share/nginx/private/mpd_calls_for_service/scrape_mpd_calls.js
/usr/bin/php /usr/share/nginx/private/mpd_calls_for_service/parse_json_file.php

Executing these lines of code individually via Terminal in SSH works fine. However, when running the shell script manually via SSH, or the Cron, this is the result.

Unable to open file: /usr/share/nginx/private/mpd_calls_for_service/scrape_mpd_calls.js Unsafe JavaScript attempt to access frame with URL about:blank from frame with URL file:///usr/local/lib/node_modules/casperjs/bin/bootstrap.js. Domains, protocols and ports must match.

Unsafe JavaScript attempt to access frame with URL about:blank from frame with URL file:///usr/local/lib/node_modules/casperjs/bin/bootstrap.js. Domains, protocols and ports must match.

Could not open input file: /usr/share/nginx/private/mpd_calls_for_service/parse_json_file.php

The solutions I have tried over the past two days have included

  1. CHMOD 777 for all affected files.
  2. Editing the crontab-e while sudo.
  3. Reinstalling PhantomJS and CasperJS via npm -g.

After multiple attempts though, I'm at a dead end and ready to start pulling my hair out. Any assistance from the community would be appreciated.

  • 1
    cron executes commands in a different environment that your login shell. Check your login environment for variables that might be getting set (that would need to be set inside the shell script when run under cron) – Jeff Schaller Jun 22 '15 at 18:04
0

I ended up just running the commands in a single crontab line with && and inside of a bash -l -c statement, it is working now!

0

I had to figure this out as well but at the very least executing casperjs requires you to have variables available that are loaded when you load a shell as a log-in shell. In my case I didn't feel like hard-coding the path to casperJS in my script because it was already set in the $PATH on my mashine. This then lead to the discovery that just spawning a shell from crontab (or in my case, ANT) does not load the .rc files , so there was no path. Beyond that, casperJS may require even more variables that are provided in a log-in shell. so bash -l -c is the way to go to load the bash as a log-in shell.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.