1

i am quit new to webscraping, i am building a personal app with python on debian buster and firefox geckodriver as follow, the server is online

firefox_binary = FirefoxBinary('/usr/bin/firefox')
opts = FirefoxOptions()
opts.add_argument("--headless")
driver = webdriver.Firefox(options=opts, firefox_binary=firefox_binary)
driver.get(url)

i aim to triggered this app by a ajax call in a web page, it's a nginx server so the py script is called in a php exec command :

exec ('/usr/bin/python3 webdriver.py');

it works fine in command line in debian if i am logged in root or another user. when i call the script using web interface, the user is www-data and i encounter this error : connection refused (os error 111)

i ve been searching for 3 days in all internet sources i can find and tried all i could, i haven't any solution for now

would you have any idea of the way to solve this issue?

Edit :

  • selenium version 3.141.0

  • Mozilla Firefox 78.8.0esr

  • geckodriver 0.29

  • geckodriver.log for www-data user (with error):

    • 1614703595351 geckodriver INFO Listening on 127.0.0.1:53899
    • 1614703595556 mozrunner::runner INFO Running command: "/usr/bin/firefox" "--marionette" "--headless" "-foreground" "-no-remote" "-profile" "/tmp/rust_mozprofile22XajP"
    • *** You are running in headless mode.
  • geckodriver.log for for admin (without error) :

    • 1614702846828 geckodriver INFO Listening on 127.0.0.1:46897
    • 1614702847019 mozrunner::runner INFO Running command: "/usr/bin/firefox" "--marionette" "--headless" "-foreground" "-no-remote" "-profile" "/tmp/rust_mozprofile7jTA4y"
    • *** You are running in headless mode.
    • 1614702859057 Marionette INFO Listening on port 45145
    • 1614702859109 Marionette WARN TLS certificate errors will be ignored for this session
    • JavaScript warning: https://..........................js?v=1.2.7199.0, line 119: unreachable code after return statement
    • JavaScript warning: https://..........................js?v=1.2.7199.0, line 119: unreachable code after return statement
    • 1614702872140 Marionette WARN TimedPromise timed out after 500 ms: stacktrace:
    • TimedPromise/<@chrome://marionette/content/sync.js:245:13
    • TimedPromise@chrome://marionette/content/sync.js:230:10
    • Interaction.flushEventLoop@chrome://marionette/content/interaction.js:416:10 webdriverClickElement@chrome://marionette/content/interaction.js:182:31
    • 1614702900530 Marionette INFO Stopped listening on port 45145
  • geckodriver.log for www-data with geckodriver 0.21 :

    • 1614706982655 geckodriver INFO geckodriver 0.21.0

    • 1614706982680 geckodriver INFO Listening on 127.0.0.1:33947

    • 1614706983553 mozrunner::runner INFO Running command: "/usr/bin/firefox" "-marionette" "--headless" "-foreground" "-no-remote" "-profile" "/tmp/rust_mozprofile.p37a7r0syAmR"

    • *** You are running in headless mode.

    • (firefox-esr:22409): dconf-CRITICAL **: 17:44:03.845: unable to create directory '/var/www/.cache/dconf': Permission denied. dconf will not work properly.

3
  • Welcome to Stackexchange! Your question doesn't tell which address and port your program is trying to access. "Connection refused" often means that there is no listener at the given address and port combination. It can also mean that a firewall actively refuses your connection attempt. When launched from the command line, your program may use a different address/port combination than when launched from the web page. It is your job to find out which address/port it uses in both cases. Or the networking environment of the web page is different from the Debian command line. – berndbausch Mar 1 at 0:04
  • thanks for your answer, i should have precise the geckodriver.log – madj Mar 2 at 18:11
  • I'm having the exact same issue on an Ubuntu server. It looks like Marionette doesn't start when using the www-data user but there is no error either. You can get rid of the permission denied issue by creating /var/www/.cache and giving it to www-data but that won't help your problem. – sdragnev Mar 2 at 21:11
1

It's a permissions problem. www-data's home dir is /var/www which is usually owned by root.

You can either give /var/www to www-data or, to limit permission changes, you can create /var/www/.mozilla and change its ownership to www-data. Then try running your script again.

What seems to be happening is Firefox is trying to save the user preferences and failing. It reports no error nor does it fully crash its main process but is still being limited enough to not be able to communicate with geckodriver.

Source: https://answers.launchpad.net/ubuntu/+question/695824

1
  • thank you very much for these answers, sure it's a permission problem. digging it.. – madj Mar 6 at 15:17
0

yeaaa it works fine thanks a lot

i just created the following directories: /var/www/.cache /var/www/.mozilla

i gave www-data permission to write in it.

i m wondering if ther is not a security problem with this. maybe should be better to specify a working directory for mozilla somewhere else..

1
  • I think you're fine security-wise because those dirs will be specific to your selenium scripts and not shared with other users/processes. If you want to be extra safe, you can delete the contents of the .cache directory after each run in case firefox stores something sensitive there (depending on what you're doing). – sdragnev Mar 7 at 23:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.