# Bash: How can I check if a website has finished loading?

Problem

I am trying to automate a task in my Firefox browser with xdotool.

First, I open a new tab in my browser with:

firefox -new-tab "www.domain.tld"


if [ <page has fully loaded> ]
then
<commands>
fi


How can I detect whether the page has finished loading in bash?

Workaround

At the moment I use sleep 5 (wait 5 seconds until calling the next command) which is a a bit hacky because some pages load really fast and others don't.

• Wouldn't it be simpler to use something designed for browser automation? – terdon Jan 13 at 14:02
• Normally yes. In my case no. I want to save several (hundred) web pages as pdf and the only tool that gives me the desired result was Firefox. The problem is that there are svg images which are rendered on each page load on every page and all other tools I tried break them. – Sr. Schneider Jan 13 at 14:05
• But why do you need a GUI then? This feels like an XY problem. I suggest you explain what you are trying to do and ask for solutions to that instead of trying to force bash to do something it is really not designed to do. – terdon Jan 13 at 17:34
• Please post your workaround as an answer. That's a really clever trick, changing the window title via javascript! – terdon Jan 14 at 10:23
• I had a similar problem for a site I frequent, which I previously used to scrape using wget and now cloudflare blocks all non-browser accesses. I found an extension called "SingleFile" that does the job. Conveniently it also seems to have the "after DOM content is loaded" piece built-in (and enabled via the add-on's preferences). – sitaram Jan 19 at 6:20

You could use a traffic monitoring service like iftop. This tool shows connections based on the hostname (or IP if you wish).

 #!/bin/bash
while ( iftop -t -s 5 2>/dev/null | grep www.domain.ltd >/dev/null ) ; do
done


Limitations:

• needs root to run
• assumes proper hostname resolution (will fail e.g. with youtube, where they use all sorts of hostnames but not youtube)
• not sure about IPv6 support in hostname resolution
• needs a few seconds to properly see traffic

Alternatively nethogs will do a per-process analysis and shows sent and received. E.g. for 2 counts with 2-second delay:

 #!/bin/bash
while ( nethogs -t -c 2 -d 2 2>/dev/null | grep firefox >/dev/null ) ; do
done


Limitations:

• needs root to run
• watches a process: if the web browser has other tabs constantly loading data, it would fail. (e.g. music from a website)
• needs a few seconds to properly see traffic

Or tcpdump, here limited to incoming TCP packets and stopped by timeout

while ( timeout  3 tcpdump 'tcp' -Q in -q 2>/dev/null |
grep www.domain.ltd >/dev/null) ; do
done


Limitations: see iftop

Conclusion: all methods are based on network traffic monitoring, this means they all need a few seconds to analyze said traffic and might actually help when it comes to make sure big websites are loaded, but would not speed up the whole thing in case of small websites.

I accepted @Fiximan's answer because it solves the problem mentioned in the question. In my particular case though (I try to print a website with a lot of MathJax formulas) his solution doesn't work because after loading the page the formulas are still rendering. If you have a similar case where you need to wait until the page is loaded AND some rendering processes are finished then you can use the following solution: You have to add some JavaScript to the page source to make it work. So either the site is yours or you have to download the page, manipulate the source code and start some local server.

General solution

If you want to call a bash script after a page has finished loading (incl. javascript):

HTML

<!DOCTYPE html>
<html>
<title>This is the page title</title>
<script>
// Save the page title in a variable
let pageTitle = document.title;
// Set the page title to something like...
// Set page title to original page title
document.title = pageTitle;
});
</script>
<body>
<h1>Hello world!</h1>
</body>
</html>


Bash

# Open new tab
firefox -new-tab "www.domain.tld"

# Wait 1 second
sleep 1

while (xdotool search --name "Page is loading... - Mozilla Firefox" > /dev/null 2>&1) ; do
sleep 0.1
done

# Then execute new command
# < some command >


MathJax 3 solution

In my case I want to wait until the page has finished loading and MathJax has finished rendering. So I have to move the part "Set page title to original page title" to the MathJax configuration:

<script>
window.MathJax = {
// ...
// ...
startup: {