1

Is there a recommendable way to get the bytes size of an URL with all referenced images, CSS and javascript files (the number of bytes a normal visitor has to receive before the page has fully loaded in the browser) by means of command line utilities? Something like (fictious):

wget --output-format=mozilla-archive-format --with-referenced-content --compression=0       \
    'http://www.example.com/' > temp.maff && stat -c%s temp.mapp ; rm temp.maff

I would aggree to limitations like, not including references that are dynamically generated and loaded by scripts, …. Just the basic information, getting 4,000,000 when there is a 15,000 bytes HTML page with a 3,985,000 bytes image in it, would be already worth looking at. I currently have a Perl script doing the very basic:

#!/usr/bin/perl
use URI;
$html = `wget -qO - '$ARGV[0]'`;
$sum = length($html);
while($html =~ /\bsrc="(.*?)"/g){
    $url = URI->new_abs($1, $ARGV[0]);
    $head = `curl -I --silent '$url'`;
    if($head =~ /Content-Length: ?(\d+)/){
        $size = scalar($1);
    }else{
        $size = length(`wget -qO - '$url'`);
    }
    $sum += $size;
}
print "$sum\n";

However, this has many drawbacks. Only one level of links being followed, no script evaluation, …. Any better approach, maybe a one-liner using mature tools, would be appreciated.

0

The relevant wget option is -p/--page-requisites:

This option causes Wget to download all the files that are necessary to properly display a given HTML page. This includes such things as inlined images, sounds, and referenced stylesheets.

-H/--span-hosts is also recommended if the page uses anything from another site (gravatar, google APIs).

-O/--output-document will output it all to a single file, without any markup or structure. This is not very helpful most of the time, but good enough if you only need to know its size.

wget -p -H 'http://www.example.com/' -O tempfile
wc -c < tempfile

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.