3

Problem

I am trying to automate a task in my Firefox browser with xdotool.

First, I open a new tab in my browser with:

firefox -new-tab "www.domain.tld"

Then (after the page www.domain.tld has finished loading) I want to perform a task:

if [ <page has fully loaded> ]
then
    <commands>
fi

How can I detect whether the page has finished loading in bash?

Workaround

At the moment I use sleep 5 (wait 5 seconds until calling the next command) which is a a bit hacky because some pages load really fast and others don't.

Improve this question
6
  • 1
    Wouldn't it be simpler to use something designed for browser automation? – terdon♦ Jan 13 at 14:02
  • Normally yes. In my case no. I want to save several (hundred) web pages as pdf and the only tool that gives me the desired result was Firefox. The problem is that there are svg images which are rendered on each page load on every page and all other tools I tried break them. – Sr. Schneider Jan 13 at 14:05
  • But why do you need a GUI then? This feels like an XY problem. I suggest you explain what you are trying to do and ask for solutions to that instead of trying to force bash to do something it is really not designed to do. – terdon♦ Jan 13 at 17:34
  • 1
    Please post your workaround as an answer. That's a really clever trick, changing the window title via javascript! – terdon♦ Jan 14 at 10:23
  • 1
    I had a similar problem for a site I frequent, which I previously used to scrape using wget and now cloudflare blocks all non-browser accesses. I found an extension called "SingleFile" that does the job. Conveniently it also seems to have the "after DOM content is loaded" piece built-in (and enabled via the add-on's preferences). – sitaram Jan 19 at 6:20
2

You could use a traffic monitoring service like iftop. This tool shows connections based on the hostname (or IP if you wish).

 #!/bin/bash
 while ( iftop -t -s 5 2>/dev/null | grep www.domain.ltd >/dev/null ) ; do
   echo "still loading"
 done

Limitations:

  • needs root to run
  • assumes proper hostname resolution (will fail e.g. with youtube, where they use all sorts of hostnames but not youtube)
  • not sure about IPv6 support in hostname resolution
  • needs a few seconds to properly see traffic
  • what about websites that constantly reload some element?

Alternatively nethogs will do a per-process analysis and shows sent and received. E.g. for 2 counts with 2-second delay:

 #!/bin/bash
 while ( nethogs -t -c 2 -d 2 2>/dev/null | grep firefox >/dev/null ) ; do
   echo "still loading"
 done

Limitations:

  • needs root to run
  • watches a process: if the web browser has other tabs constantly loading data, it would fail. (e.g. music from a website)
  • needs a few seconds to properly see traffic

Or tcpdump, here limited to incoming TCP packets and stopped by timeout

while ( timeout  3 tcpdump 'tcp' -Q in -q 2>/dev/null |
        grep www.domain.ltd >/dev/null) ; do
  echo "still loading"
done

Limitations: see iftop

Conclusion: all methods are based on network traffic monitoring, this means they all need a few seconds to analyze said traffic and might actually help when it comes to make sure big websites are loaded, but would not speed up the whole thing in case of small websites.

Improve this answer
1

I accepted @Fiximan's answer because it solves the problem mentioned in the question. In my particular case though (I try to print a website with a lot of MathJax formulas) his solution doesn't work because after loading the page the formulas are still rendering. If you have a similar case where you need to wait until the page is loaded AND some rendering processes are finished then you can use the following solution: You have to add some JavaScript to the page source to make it work. So either the site is yours or you have to download the page, manipulate the source code and start some local server.

General solution

If you want to call a bash script after a page has finished loading (incl. javascript):

HTML

<!DOCTYPE html>
<html>
<head>
  <title>This is the page title</title>
  <script>
      // Save the page title in a variable
      let pageTitle = document.title;
      // Set the page title to something like...
      document.title = "Page is loading..."
      // If page has finished loading
      document.addEventListener("DOMContentLoaded", function(event) {
          // Set page title to original page title
          document.title = pageTitle;
      });
  </script>
</head>
<body>
<h1>Hello world!</h1>
</body>
</html>

Bash

# Open new tab
firefox -new-tab "www.domain.tld"

# Wait 1 second
sleep 1

# Wait until page has finished loading...
  while (xdotool search --name "Page is loading... - Mozilla Firefox" > /dev/null 2>&1) ; do
    sleep 0.1
  done

# Then execute new command
# < some command >

MathJax 3 solution

In my case I want to wait until the page has finished loading and MathJax has finished rendering. So I have to move the part "Set page title to original page title" to the MathJax configuration:

<script>
  window.MathJax = {
    // ...
    // < your configurations >
    // ...
    startup: {
      // If page has finished loading
      pageReady() {
        // If MathJax has finished typesetting
        return MathJax.startup.defaultPageReady().then(function() {
          // Set page title to original page title
          document.title = "This is the page title";
        });
      }
    }
    {{ end }}
  };
</script>
Improve this answer
1
  • 1
    Perhaps this could be a Tampermonkey script. I wonder if the printing can also be accomplished via JS, in which case the Tampermonkey script could do the whole job. – muru Jan 14 at 10:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.