0

The goal that I am trying to accomplish is to pull HTML code from a website, write it to a file. Once this is done, I want to loop it to do it again and write it to another file. After the second file is written I want it to compare the two files together to see if anything has changed. Here is what I have so far, and it does not work:

#!/bin/bash
echo "Hopefully this will do everything at once!"

while true;
do 

wget -q -O - http://website.com > websitebaseline.txt

if -e websitebaseline.txt
then
    wget -q -O - http://www.website.com > websitechange.txt
    echo "Update to websitechange.txt has been made"

    if !-e websitebaseline.txt
    then
        wget -q -O - http://www.website.com > webbaseline.txt
        echo "Baseline has been created"

if -e websitebaseline.txt websitechange.txt
then diff -y websitebaseline.txt websitechange.txt --supress-common-lines > Changeinsite.txt
    if !-e websitebaseline.txt
    then 
        wget -q -O - http://www.website.com > websitebaseline.txt   echo "Baseline has been created"
    elif !-e websitechange.txt
    then
        wget -q- O - http://websitename.com > websitenamechange.txt
        echo "Update has been made"

sleep 100;
done
  • 1
    Rather than multiple copies of the file, you could use revision control.... – drewbenn May 2 '17 at 20:44
1

Way overcomplicating things.

#!/bin/bash
left=$(mktemp)
right=$(mktemp)
url="http://url.example.com/"
trap 'rm -f "$left" "$right"' EXIT
for file in "$left" "$right"; do
    wget -q -O "$file" "$url"
done
if diff "$left" "$right" > /dev/null 2>&1; then
    echo "Changes detected in successive retrievals of '$url'."
fi

A similar regime could be used to incrementally note changes over time:

left=$(mktemp)
right=$(mktemp)
url="http://url.example.com/"
trap 'rm -f "$left" "$right"' EXIT
# Establish the "baseline":
wget -q -O "$left" "$url"

# Okay, now check for updates forever:
while sleep 30; do
    wget -q -O "$right" "$url"
    if diff "$left" "$right" > /dev/null 2>&1; then
        echo "$(date) - Changes detected in '$url'."
        cp "$right" "$left"
    fi
done
  • Can this be used to create a baseline and detect if changes happen over a long period of time? – Tweak May 2 '17 at 20:54
  • It'd be a simple enough change, either to do it incrementally (i. e. detect every change over time) or statically (i. e. detect every instance that differs from a baseline that would be maintained separately). – DopeGhoti May 2 '17 at 21:00
  • What would it look like for the static detection? I am very new at this and this is really only my second script. And is there a way to write the changes to a file for documentation? – Tweak May 2 '17 at 21:02
  • See the new example added to the answer above. – DopeGhoti May 2 '17 at 21:07
  • Consider cmp -s rather than diff... – roaima May 2 '17 at 21:13
0

The easiest way to detect a difference between two files, no matter how small the change may be, is to compare their checksums. To demonstrate here I just used the "md5sum" command to generate the md5 hash of each request.

#!/bin/bash

wget -q website.com -O site.txt
baseline=$(md5sum site.txt)
echo first request checksum: $baseline
rm site.txt

wget -q website.com -O site.txt
change=$(md5sum site.txt)
echo second request checksum: $change
rm site.txt

The output of this script will be the md5 hash from each request, and you'll very easily be able to see if the hashes are the same or different.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.