connection refused (os error 111) using python selenium & geckodriver

Question

i am quit new to webscraping, i am building a personal app with python on debian buster and firefox geckodriver as follow, the server is online

firefox_binary = FirefoxBinary('/usr/bin/firefox')
opts = FirefoxOptions()
opts.add_argument("--headless")
driver = webdriver.Firefox(options=opts, firefox_binary=firefox_binary)
driver.get(url)

i aim to triggered this app by a ajax call in a web page, it's a nginx server so the py script is called in a php exec command :

exec ('/usr/bin/python3 webdriver.py');

it works fine in command line in debian if i am logged in root or another user. when i call the script using web interface, the user is www-data and i encounter this error : connection refused (os error 111)

i ve been searching for 3 days in all internet sources i can find and tried all i could, i haven't any solution for now

would you have any idea of the way to solve this issue?

Edit :

• selenium version 3.141.0

• Mozilla Firefox 78.8.0esr

• geckodriver 0.29

• geckodriver.log for www-data user (with error):

  • 1614703595351 geckodriver INFO Listening on 127.0.0.1:53899
  • 1614703595556 mozrunner::runner INFO Running command: "/usr/bin/firefox" "--marionette" "--headless" "-foreground" "-no-remote" "-profile" "/tmp/rust_mozprofile22XajP"
  • *** You are running in headless mode.

• geckodriver.log for for admin (without error) :

  • 1614702846828 geckodriver INFO Listening on 127.0.0.1:46897
  • 1614702847019 mozrunner::runner INFO Running command: "/usr/bin/firefox" "--marionette" "--headless" "-foreground" "-no-remote" "-profile" "/tmp/rust_mozprofile7jTA4y"
  • *** You are running in headless mode.
  • 1614702859057 Marionette INFO Listening on port 45145
  • 1614702859109 Marionette WARN TLS certificate errors will be ignored for this session
  • JavaScript warning: https://..........................js?v=1.2.7199.0, line 119: unreachable code after return statement
  • JavaScript warning: https://..........................js?v=1.2.7199.0, line 119: unreachable code after return statement
  • 1614702872140 Marionette WARN TimedPromise timed out after 500 ms: stacktrace:
  • TimedPromise/<@chrome://marionette/content/sync.js:245:13
  • TimedPromise@chrome://marionette/content/sync.js:230:10
  • Interaction.flushEventLoop@chrome://marionette/content/interaction.js:416:10 webdriverClickElement@chrome://marionette/content/interaction.js:182:31
  • 1614702900530 Marionette INFO Stopped listening on port 45145

• geckodriver.log for www-data with geckodriver 0.21 :

  • 1614706982655 geckodriver INFO geckodriver 0.21.0

  • 1614706982680 geckodriver INFO Listening on 127.0.0.1:33947

  • 1614706983553 mozrunner::runner INFO Running command: "/usr/bin/firefox" "-marionette" "--headless" "-foreground" "-no-remote" "-profile" "/tmp/rust_mozprofile.p37a7r0syAmR"

  • *** You are running in headless mode.

  • (firefox-esr:22409): dconf-CRITICAL **: 17:44:03.845: unable to create directory '/var/www/.cache/dconf': Permission denied. dconf will not work properly.

Answer 1

It's a permissions problem. www-data's home dir is /var/www which is usually owned by root.

You can either give /var/www to www-data or, to limit permission changes, you can create /var/www/.mozilla and change its ownership to www-data. Then try running your script again.

What seems to be happening is Firefox is trying to save the user preferences and failing. It reports no error nor does it fully crash its main process but is still being limited enough to not be able to communicate with geckodriver.

Source: https://answers.launchpad.net/ubuntu/+question/695824

Answer 2

yeaaa it works fine thanks a lot

i just created the following directories: /var/www/.cache /var/www/.mozilla

i gave www-data permission to write in it.

i m wondering if ther is not a security problem with this. maybe should be better to specify a working directory for mozilla somewhere else..

Answer 3

The main problem is .cache and .mozilla folders, which need permissions. The data is generated in the profile of the user who execute the script.

This solution worked for me:

1. Identify the user who needs to save the cache. In my case it was www-data (the user of Symfony). I checked it in the folder /tmp/. You can see the owner of folders like /tmp/Temp-276cddf8-d7c5-4a09... These folders are generated by Firefox.
2. The user needs bash permissions, so you have to adjust the file
   /etc/passwd from www-data:33:33:www-data:/usr/sbin:/usr/sbin/nologin to www-data:x:33:33:www-data:**/var/www**:**/bin/bash**
3. Create the folders in the path, in my case is /var/www/ (this path is located in the user path specified in the file /etc/passwd).

mkdir /var/www/.cache

mkdir /var/www/.mozilla

chown www-data:www-data .cache

chown www-data:www-data .mozilla

---

@sdragnev answered correctly, but in my case I still needed to give bash permission to the user.