0

I am trying to create a script to monitor a web page and send a telegram notification whenever there is a change in that page, I am using diff to accomplish this task.

The script seems to work well but some web pages insert a kind of random ID in the content of the page, this ID changes every time I download the page and I need to solve this for the diff to work correctly.

I need to find some way to remove/edit this randomly generated ID, in short I need to edit the string of this ID deleting almost all letters, spaces, hyphens, numbers, etc. and save only the data without ID.

Example, I need to modify only the information that is enclosed in the quotation marks " ":

<path d="M0 0h7v7h-7zM9 0h1v2h-1zM12 0h1v4h-2v-1h-1v-1h1v-1h1zM16 0h1v3h-1v-1h-1v-1h1zM18 0h4v1h-1v1h1v1h-2v-2h-1v1h-1zM23 0h1v1h-1zM26 0h7v7h-7zM1 1v5h5v-5zM22 1h1v1h-1zM27 1v5h5v-5zM2 2h3v3h-3zM8 2h1v1h1v1h1v1h1v1h-1v1h-1v-1h-1v1h1v1h-2zM14 2h1v1h-1zM23 2h1v2h1v3h-1v-2h-4v-1h3zM28 2h3v3h-3zM15 3h1v1h2v2h-1v-1h-2v2h-1v-1h-1v-2h2zM18 3h1v1h-1zM19 5h1v1h-1zM12 6h1v2h-2v-1h1zM16 6h1v2h1v-2h1v1h1v-1h1v1h1v1h-2v1h1v1h1v1h1v1h-3v1h-1v-1h-2v1h-1v-2h-2v1h-1v-4h2v1h-1v1h2zM22 6h1v1h-1zM23 7h1v1h-1zM0 8h1v1h1v-1h5v1h-3v1h3v1h-1v1h-1v-1h-2v-1h-1v1h-1v1h-1zM22 8h1v1h-1zM24 8h1v1h-1zM26 8h5v2h1v2h-2v1h3v1h-1v1h-1v1h-1v-2h-1v-1h-1v-3h1v1h1v-1h-1v-1h-1v1h-2zM9 9h1v1h-1zM23 9h1v1h-1zM32 9h1v1h-1zM8 10h1v1h-1zM18 10v1h2v-1zM10 11h1v1h-1zM25 11h1v1h-1zM3 12h2v1h-1v2h-2v-1h1zM6 12h3v1h-1v1h1v1h-1v1h-1v-1h-1v1h1v1h-1v1h-2v1h-1v-1h-3v-5h2v1h-1v2h1v1h1v-1h2v-2h2v-1h-1zM11 12h1v2h3v-1h1v1h1v1h-1v2h-1v-2h-1v1h-3zM14 12h1v1h-1zM17 13h2v1h-2zM22 13h6v1h-1v2h-1v1h-1v-1h-1v-2h-2zM20 14h2v1h1v1h-2v-1h-1zM9 15h1v1h-1zM28 15h1v2h-1v1h1v1h1v-1h-1v-1h2v1h1v1h1v3h-1v-1h-1v-1h-1v3h-1v-2h-1v-1h-2v-1h1v-1h-1v-1h1v-1h1zM10 16h1v1h-1zM17 16h1v1h-1zM32 16h1v2h-1zM8 17h2v1h-1v1h-1v1h2v3h-1v-1h-1v1h-2v1h2v1h-3v-1h-1v1h-1v-1h-1v-2h1v1h2v-1h2v-1h-1v-1h1v-1h-1v-1h2zM11 17h3v2h1v-1h1v1h1v1h-1v1h1v1h-2v-2h-3v-1h1v-1h-1v1h-1v1h-1v-2h1zM16 17h1v1h-1zM19 17h1v1h-1zM21 17h1v1h-1zM23 17h1v1h-1zM18 18h1v1h-1zM20 18h1v1h1v1h-1v1h-1v1h-1v-1h-1v-1h2zM22 18h1v1h-1zM24 18h2v1h-1v1h-1zM1 19h2v1h2v1h-3v-1h-1zM5 19h1v1h-1zM11 20h1v1h1v1h-1v1h-1zM23 20h1v1h4v2h-2v1h4v1h-1v2h1v-2h1v1h1v1h-1v1h-1v3h1v1h-1v1h-1v-1h-1v-1h1v-1h-1v-1h-4v-1h-1v-2h1v-4h-1v1h-1v-2h1zM0 21h2v1h-1v3h-1zM31 22h1v1h1v1h-3v-1h1zM10 23h1v1h-1zM13 23h1v1h-1zM16 23h1v1h-1zM21 23h1v1h-1zM9 24h1v1h1v-1h2v2h-1v1h-1v1h-1v-1h-1v-1h-1v-1h1zM14 24h1v2h-1zM17 24h1v3h2v-1h-1v-2h1v1h2v1h-1v1h1v1h-1v1h-1v1h-1v1h-3v2h-2v-1h1v-1h-4v-2h5v1h2v-1h1v-1h-2v1h-1v-2h-1v-1h1v-1h1zM22 24h1v1h-1zM25 25v3h3v-3zM32 25h1v1h-1zM0 26h7v7h-7zM26 26h1v1h-1zM1 27v5h5v-5zM8 27h1v1h1v3h1v2h-1v-1h-1v-1h-1zM12 27h1v1h-1zM2 28h3v3h-3zM31 28h2v2h-2zM21 29h2v1h-2zM20 30h1v1h-1zM23 30h1v2h-1v1h-1v-1h-1v-1h2zM26 30h2v1h-2zM8 32h1v1h-1zM17 32h3v1h-3zM24 32h1v1h-1zM26 32h2v1h-2zM31 32h1v1h-1z"/>

Result I need:

<path d = ""/>

or anything like these examples:

<path d="0"/>
<path d="CLEAN"/>
<path d=""/>
<path d=/>

I believe that it may be possible to resolve this issue using sed, but due to the complexity of the string, there are many characters, spaces, hyphens, numbers, etc, I am having a lot of difficulties to get to the ideal command

Example of the script I am using:

#! /bin/bash

page_mofication="$(cat /opt/pagename/listing/latest_modifications/latest_modifications.log)"
fileold=/opt/pagename/latest_modifications/latest_modifications_old
filenew=/opt/pagename/latest_modifications/latest_modifications_new
log=/opt/pagename/listing/latest_modifications/latest_modifications.log
logold=/opt/pagename/oldfiles/latest_modifications/latest_modifications.log

mv $log $logold-`date +%d-%m-%Y_%H:%M:%S`
wget https://www.pagename.com -O $filenew


diff $fileold $filenew >> $log    
message=$'\n'"$page_mofication"
/etc/scripts/telegram-send.sh "$message"

cp $filenew $fileold
exit 0

Any ideas on how I can solve this problem?

4
  • Why don't you just hard code the nulling of path_d variable?
    – guest_7
    May 4 at 0:29
  • Always paste your script into https://shellcheck.net, a syntax checker, or install shellcheck locally. Make using shellcheck part of your development process.
    – waltinator
    May 4 at 20:07
  • @guest_7 - Honestly I am a novice user in shell script, I do not know how to do this "nulling of path_d", I would appreciate any help, if you can help me I will be grateful.
    – Doom
    May 5 at 2:46
  • @waltinator - Thank you very much for the tip of the shellcheck, for beginners it seems to me to be a very interesting tool.
    – Doom
    May 5 at 2:49
1

Assuming you are on GNU sed version of the sed editor. Try to null the path d data in both fileold and filenew files before diffing. So you could do something along these lines:

sed -i '
/<path d=/c\
<path d=/>
'  -- "$fileold" "$filenew";

Or if you must need to make sure that the characters in between the quotes are only alphanumerics , hyphens, horizontal whitespaces

sed -Ei  '
s|(<path d)="[\t a-zA-Z0-9-]+"/>|\1=/>|
' -- "$fileold" "$filenew";
1
  • The second option worked perfectly, thank you very much! I have some more questions and I will update the initial post.
    – Doom
    May 5 at 2:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.