while IFS= read -r url; do
curl -o "$file.new" "$url"
if ! cmp -s "$file" "$file.new"
printf '%s\n' "$url"
mv -f "$file.new" "$file"
i=$(( i + 1 ))
This would read the URLs from
url-list.txt, line by line, and use
curl to fetch each, saving the output in a file called
N is an integer (the URL ordinal number in the file).
If there is no old
data-N.out file, or if this file differs from
data-N.out.new, then the URL is printed to standard output.
The fetched data file is then renamed for when you run the script again.
The first time you run the script, all URLs will be outputted as they have never been seen before.
Reordering the URLs, or adding new URLs at the top would make the URLs be flagged as changed as the contents of the corresponding data file has changed. You could fix this by using e.g. the base64-encoded URL as part of the output filename instead of
Whether you use
wget or some other Web client is essentially unimportant.