1

Background: my particular problem, which gave rise to this question, is as follows. I'm a slackware linux user, and on 23-March-2019 mirrored their distribution with the following command

wget -r -np -R "index.html*" https://mirror.slackbuilds.org/slackware/slackware64-current/

Then recently, 29-Aug-2019, I refreshed/updated my local mirror simply by adding the -N option to the above command. But that resulted in my mirror containing many, many "duplicate" older and newer versions of the same packages, just with different version numbers, e.g.,

SDL2_mixer-2.0.4-x86_64-1.txz
SDL2_mixer-2.0.4-x86_64-2.txz
libcddb-1.3.2-x86_64-5.txz
libcddb-1.3.2-x86_64-6.txz

etc (and I mean lots of etc's:)

Question: So what I really want to do is something like

rsync -av --delete https://mirror.slackbuilds.org/slackware/slackware64-current/ my-slackware64-mirror-directory/

That rsync --delete would have automatically deleted all the older versions from my mirror-directory that no longer exist on slackbuilds.org. However, I don't have any kind of account on slackbuilds, and therefore can't (as far as I know) run rsync to get files from it. Is there any wget way to accomplish the same thing? Or any way at all? Thanks.


Edit:   lengthy reply to @roaima's comment...
    Thanks for the suggestion, @roaima. And now that you mention it, yup, there is such a file in the top-level directory, predictably named FILELIST.TXT   But I'm not sure how to use it as input to some procedure that would --delete the older files not in the current filelist. Could you point me to a manpage, or whatever, that describes how to do that? Thanks again.

Also, that FILELIST.TXT is in an   ls -al   format that might not be the easiest for canned procedures to parse (although I can probably write a small C program to convert it to any suitable format). A few typical lines from the file are

-rw-r--r--  1 root root   1637708 2019-08-15 18:06 ./slackware64/a/bash-5.0.009-x86_64-1.txz
-rw-r--r--  1 root root       163 2019-08-15 18:06 ./slackware64/a/bash-5.0.009-x86_64-1.txz.asc
-rw-r--r--  1 root root       226 2018-10-17 03:06 ./slackware64/a/bin-11.1-x86_64-3.txt
-rw-r--r--  1 root root     39576 2018-10-17 03:06 ./slackware64/a/bin-11.1-x86_64-3.txz
-rw-r--r--  1 root root       163 2018-10-17 03:06 ./slackware64/a/bin-11.1-x86_

And very relevant to your apparently-prescient suggestion, the first few lines of the file are the remark

Wed Aug 28 21:44:15 UTC 2019
Here is the file list for this directory.  If you are using a 
mirror site and find missing or extra files in the disk 
subdirectories, please have the archive administrator refresh
the mirror.
3
  • 1
    Is there a list of current packages that you could use to trim the set that you have copied?
    – roaima
    Aug 30 '19 at 5:07
  • @roaima Thanks for the suggestion. I've added an Edit at the bottom of the question with a lengthy reply. In short, yes, there's a list. But what's the procedure to use that list to sync up the actual files with it? Thanks again. Aug 30 '19 at 5:36
  • @roaima googling found a couple of ways to use such a list, e.g., unix.stackexchange.com/questions/127268 and askubuntu.com/questions/830776 But none that I found seem to --recurse below the pwd. And that slackware mirror has lots of subdirectories, with the filelist containing full paths to them. Aug 30 '19 at 5:57
3

Disclaimer: I can't currently test any code I'm suggesting. As always, please test carefully before building this code into a proper script.

Given it's a package list (and therefore having simple filenames without spaces, etc.), you can probably pick out the filenames with some very straightforward code

# Get last item from each line of FILELIST
awk '{print $NF}' FILELIST | sort >weblist

# Generate a list of your files
find -type f -print | sort >mylist

# Compare results
comm -23 mylist weblist >diffs

# Remove old files
xargs -r echo rm -fv <diffs

As you can see, you'll need to remove the final echo to allow rm to do its work

Furthermore, next time you want to update your mirror, you can modify the comm line (by swapping the two file arguments) to find the set of files you don't have, and feed those to wget.

4
  • 1
    Thanks again, roaima. Yeah, that all (each step separately) seems to work, though I haven't tested the last actual rm step yet (and will eventually do it against a copy of the mirror). There's a little bit of symlink goofiness (they're in the original FILELIST, but not the -type f list), but I can straighten that out for myself. Your basic four steps seem right on the money (but you only get a "check":). And it still strikes me as a little strange that there isn't a straightforward wget --option for this. They've got --mirror, but not the necessary rsync-like --delete for an -N update mirror. Aug 30 '19 at 7:32
  • 1
    P.S. Your roaima.co.uk site seems to be down. And with respect to bellringing.org, where the heck's your link to poets.org/poem/bells ???:) Aug 30 '19 at 7:54
  • 1
    PS reply: I haven't run roaima.co.uk for a few years; I'll remove the link. I ought to link to Dorothy Sayers too then :-)
    – roaima
    Aug 30 '19 at 10:14
  • No need to remove the echo, just pipe the output to sh. That way, running ./myscript | less will always show you what files it's going to delete, but not actually do it. A nice dry-run safety feature. If the user agrees, the user then runs ./myscript | sh and the files go away. +1 btw.
    – Jim L.
    Aug 30 '19 at 17:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.