1

Overview:

Selenium scraper works perfectly in headless mode. Spawning a virtual display shows no errors via XVFB:

from xvfbwrapper import Xvfb

vdisplay = Xvfb()
vdisplay.start()
vdisplay.stop()

But when I try to run them together, it errors out with:

[ERROR] SessionNotCreatedException: Message: session not created from disconnected: Unable to receive message from renderer (Session info: chrome=96.0.4664.0)

Traceback:

Traceback (most recent call last):
  File "/var/task/slack_main.py", line 34, in handler
    scrape_price(asin_list)
  File "/var/task/slack_main.py", line 58, in scrape_price
    driver = webdriver.Chrome("/opt/chromedriver",options=options)
  File "/var/lang/lib/python3.9/site-packages/selenium/webdriver/chrome/webdriver.py", line 70, in __init__
    super(WebDriver, self).__init__(DesiredCapabilities.CHROME['browserName'], "goog",
  File "/var/lang/lib/python3.9/site-packages/selenium/webdriver/chromium/webdriver.py", line 92, in __init__
    RemoteWebDriver.__init__(
  File "/var/lang/lib/python3.9/site-packages/selenium/webdriver/remote/webdriver.py", line 275, in __init__
    self.start_session(capabilities, browser_profile)
  File "/var/lang/lib/python3.9/site-packages/selenium/webdriver/remote/webdriver.py", line 365, in start_session
    response = self.execute(Command.NEW_SESSION, parameters)
  File "/var/lang/lib/python3.9/site-packages/selenium/webdriver/remote/webdriver.py", line 430, in execute
    self.error_handler.check_response(response)
  File "/var/lang/lib/python3.9/site-packages/selenium/webdriver/remote/errorhandler.py", line 247, in check_response
    raise exception_class(message, screen, stacktrace)

Configuration:

Below is my complete Selenium and XVFB configuration:

from selenium import webdriver
from selenium_stealth import stealth
from xvfbwrapper import Xvfb

vdisplay = Xvfb()
vdisplay.start()
options = webdriver.ChromeOptions()
prefs = {"browser.downloads.dir": "//tmp//", "download.default_directory": "//tmp//", "directory_upgrade": True}
options.add_experimental_option("prefs", prefs)
options.binary_location = '/opt/chrome/chrome'
#options.add_argument('--headless') #toggled on and off when running with or without XVFB
options.add_argument('--no-sandbox')
options.add_argument("--disable-gpu")
options.add_argument("--window-size=1280x1696")
options.add_argument("--single-process")
options.add_argument("--disable-dev-shm-usage")
options.add_argument("--disable-dev-tools")
options.add_argument("--no-zygote")
options.set_capability('unhandledPromptBehavior', 'ignore')
options.add_argument("download.default_directory=/tmp")
driver = webdriver.Chrome("/opt/chromedriver",options=options)
stealth(driver,
        languages=["en-US", "en"],
        vendor="Google Inc.",
        platform="Win32",
        fix_hairline=True,
        )
vdisplay.stop()
driver.close()

Why is it not connecting to the display? My guess is it has something to do with the '--headless' toggle?

Versions and Tools:

  • Selenium version 3.141.0
  • xvfbwrapper version 0.2.9
  • Docker is used to compile and push to AWS Lambda, base image used (no changes are made in docker file with or without XVFB)

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.