I want to read webpage list and check if some of them have updated. Is it better to use wget or curl and how should I do that?

Webpage list is in a simple text file. If the contents of a webpage is the same it will not print anything. If contents changed from the last time that the script ran, then it will type(stdout) the webpage address.

  • Please don't re-post the same question. I have now reopened this one and closed your first one as a duplicate. – terdon Nov 23 '18 at 12:28
  • Βάσω, please edit your question and show us a few lines of your file with the webpages so we know what we're dealing with. And do you keep the previous versions of the webpages somewhere? Are they simple html files? More complex pages which are generated on the fly? – terdon Nov 23 '18 at 12:28
  • Also, please look at the comments on your original question and edit this one to provide the details requested. – terdon Nov 23 '18 at 12:29

while IFS= read -r url; do

    curl -o "$file.new" "$url"

    if ! cmp -s "$file" "$file.new"
        printf '%s\n' "$url"

    mv -f "$file.new" "$file"

    i=$(( i + 1 ))
done <url-list.txt

This would read the URLs from url-list.txt, line by line, and use curl to fetch each, saving the output in a file called data-N.out.new where N is an integer (the URL ordinal number in the file).

If there is no old data-N.out file, or if this file differs from data-N.out.new, then the URL is printed to standard output.

The fetched data file is then renamed for when you run the script again.

The first time you run the script, all URLs will be outputted as they have never been seen before.

Reordering the URLs, or adding new URLs at the top would make the URLs be flagged as changed as the contents of the corresponding data file has changed. You could fix this by using e.g. the base64-encoded URL as part of the output filename instead of $i.

Whether you use curl or wget or some other Web client is essentially unimportant.

  • This question is a duplicate. Vote to open the old one, and put answer there. (And maybe edit question to make it clear.) – ctrl-alt-delor Nov 23 '18 at 12:03
  • Yes that would work. Note there are some edits of the other question (just grammar, punctuation). – ctrl-alt-delor Nov 23 '18 at 12:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.