3

I'm trying to write a script that notifies me when a static web page has changed. To do it, I'm using wget to download the web page, and diff to check whether it has changed or not. I'm running an Ubuntu 20.04 LTS virtual machine. Here is the example:

$ wget --quiet https://twiki.di.uniroma1.it/twiki/view/Reti_Avanzate/InternetOfThings2021 -O file1
$ wget --quiet https://twiki.di.uniroma1.it/twiki/view/Reti_Avanzate/InternetOfThings2021 -O file2
$ diff -q file1 file2
Files file1 and file2 differ

As you can see, diff reports differences between the two files. Why? Even if I try to compare them with diff -y they look the same for me.

UPDATE

Looking for differences with git diff --color-words -- file1 file2 gave the following result:

Differences

Apparently, there's a field in which the timestamp is added, and in one of the two files there's a <!--GENERATED_HEADERS--> which is absent in the other.

Any idea on how to solve it?

3
  • 2
    Have you tried a different viewer? git diff --color-words -- file1 file2 might be more illustrative than a line-based view.
    – ojdo
    Jun 1 at 8:48
  • @ojdo thank you, updated Jun 1 at 9:03
  • 1
    wow, that's a weird piece of software. There's a bunch of links there that look like <a href="/twiki/..." class="twikiLink">, and the href and class attributes are in the opposite order in another between two downloads I got. You could use some tool to just extract the text content (but whitespace might still be a problem). Or, since it looks to be a wiki with version control, look at the revision number/date. It says "Topic revision: r170 - 2021-05-22" at the bottom of the page.
    – ilkkachu
    Jun 1 at 9:21
5

You can solve this problem by using w3m with -dump option that ignores tags while rendering the page.

$ w3m -dump  https://twiki.di.uniroma1.it/twiki/view/Reti_Avanzate/InternetOfThings2021 > file1
$ w3m -dump  https://twiki.di.uniroma1.it/twiki/view/Reti_Avanzate/InternetOfThings2021 > file2
$ if cmp -s file1 file2; then echo "Files are not different"; fi
  Files are not different                                                              
$
2

One more "answer" not on how to solve this, but what causes this: the used version of (apparently) TiddlyWiki seems to generate its output with random tag order in HTML elements. Examples:

--- file1
+++ file2
@@ -703,881 +703,881 @@
[...]
-<td valign="top" bgcolor="#ffffff" class="twikiTableCol2" align="left"> r1 </td>
[...]
+<td bgcolor="#ffffff" class="twikiTableCol2" align="left" valign="top"> r1 </td>
[...]

This repeats for attributes in tables, images, links.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.