Website monitoring

Question

The goal that I am trying to accomplish is to pull HTML code from a website, write it to a file. Once this is done, I want to loop it to do it again and write it to another file. After the second file is written I want it to compare the two files together to see if anything has changed. Here is what I have so far, and it does not work:

#!/bin/bash
echo "Hopefully this will do everything at once!"

while true;
do 

wget -q -O - http://website.com > websitebaseline.txt

if -e websitebaseline.txt
then
    wget -q -O - http://www.website.com > websitechange.txt
    echo "Update to websitechange.txt has been made"

    if !-e websitebaseline.txt
    then
        wget -q -O - http://www.website.com > webbaseline.txt
        echo "Baseline has been created"

if -e websitebaseline.txt websitechange.txt
then diff -y websitebaseline.txt websitechange.txt --supress-common-lines > Changeinsite.txt
    if !-e websitebaseline.txt
    then 
        wget -q -O - http://www.website.com > websitebaseline.txt   echo "Baseline has been created"
    elif !-e websitechange.txt
    then
        wget -q- O - http://websitename.com > websitenamechange.txt
        echo "Update has been made"

sleep 100;
done

Answer 1

Way overcomplicating things.

#!/bin/bash
left=$(mktemp)
right=$(mktemp)
url="http://url.example.com/"
trap 'rm -f "$left" "$right"' EXIT
for file in "$left" "$right"; do
    wget -q -O "$file" "$url"
done
if diff "$left" "$right" > /dev/null 2>&1; then
    echo "Changes detected in successive retrievals of '$url'."
fi

A similar regime could be used to incrementally note changes over time:

left=$(mktemp)
right=$(mktemp)
url="http://url.example.com/"
trap 'rm -f "$left" "$right"' EXIT
# Establish the "baseline":
wget -q -O "$left" "$url"

# Okay, now check for updates forever:
while sleep 30; do
    wget -q -O "$right" "$url"
    if diff "$left" "$right" > /dev/null 2>&1; then
        echo "$(date) - Changes detected in '$url'."
        cp "$right" "$left"
    fi
done

Answer 2

The easiest way to detect a difference between two files, no matter how small the change may be, is to compare their checksums. To demonstrate here I just used the "md5sum" command to generate the md5 hash of each request.

#!/bin/bash

wget -q website.com -O site.txt
baseline=$(md5sum site.txt)
echo first request checksum: $baseline
rm site.txt

wget -q website.com -O site.txt
change=$(md5sum site.txt)
echo second request checksum: $change
rm site.txt

The output of this script will be the md5 hash from each request, and you'll very easily be able to see if the hashes are the same or different.