2

I am mirroring a website using wget. On the server, there is a dump of the find / command, which includes filenames in the form :

/dir1/ /dir1/page1.txt /dir1/page2.txt ...

The mirrorring is very large (100GB+) and I would like to compare the local output of find to the text file containing the server-side dump, so as to have an idea of how many files are left.

I was thinking of using a grep -v ... or awk but I'm not quite sure how one would go about doing that.

Can anyone help me please?

Thanks !

|improve this question
2

You can do this with comm (using bash process substitution):

comm -13 <(find / | sort) <(sort server_dump)

This will show the files which are unique to the server. For files that are unique to the local system:

comm -23 <(find / | sort) <(sort server_dump)

You may also wan to add the -xdev option to find to stop it from comparing files on other filesystems, such as /proc and /sys.

|improve this answer
  • This is pretty good in theory, but it doesn't work in practice – is there a way to make it more flexible ? E.g. that the files don't need to be exactly, line by line, in the same order, and for example the out put of find is in an absolute format /pathToMirrorDirectory/dir1... ; so there is a mismatch. – Alexandre Cassagne Apr 11 '14 at 19:40
  • This ended up being the way to go. I just dupplicated the dump and appended /PathToMirrorDirectory/ before each line. Kind of ugly – but it worked. Thank you ! – Alexandre Cassagne Apr 11 '14 at 19:50
  • @AlexandreCassagne, unfortunately sorting is necessary for comm. You can always use -printf '%P\n' with find to strip the location argument from you path or worst case add a sed process to alter the paths. It depends on the exact commands you are actually using though. – Graeme Apr 11 '14 at 19:58
2

I don't think you need to use wget's recursive downloading options because you already have a list of the files that need downloading:

wget --input-file=./path/to/your/list --base=URL

In this way you would just follow the list in the first place, and so every file downloaded would be just the next in the list. So you could always find your progress by merely checking against the last file downloaded and its position in the list.

But if that doesn't work:

{ find . ; sed 's/.*/.&/' list.file ; } | 
    sort | uniq -u
|improve this answer
  • Thanks, but I already launched the mirror. Besides – the recursive download is useful, in case the dump wasn't updated or something ! :) – Alexandre Cassagne Apr 11 '14 at 19:51
  • @AlexandreCassagne I put another solution up, and I'd be curious just to know if it worked. – mikeserv Apr 11 '14 at 20:15
  • I upvoted because this seems like a good way to tackle this, but the code doesn't run atm; after searching a bit why it seems to say the Mac OS X (which I'm trying to run this on) doesn't have the same sed as the GNU standard. But thanks :) – Alexandre Cassagne Apr 11 '14 at 22:40
  • @AlexandreCassagne I think its because of /dev/stdin - either that or you're right and the problem is -f - which breaks the streamed model - or s:. You can - and probably should - get gnu tools for the mac - in my opinion. – mikeserv Apr 11 '14 at 22:43
  • @AlexandreCassagne if the process substitution in Graeme's answer works you could try: sed -f <(find /./path/from/root | sed 's:/./path/from/root/to/base/dir(.*):\\|\1|d:') ./path/to/list but you already have a working solution, so it might not be worth the effort. – mikeserv Apr 11 '14 at 22:51
0

Why not just dump the output of find to a file, on both hosts, and then do a diff on the two files? To get the number of files, you could count using wc.

|improve this answer
0

It isn't clear from the question if there is SSH access available to the server. If there is, I would simply use rsync for the job.

|improve this answer
  • Would be useful — but the mirrored server isn't under my control. This is clandestine to say the least, haha – Alexandre Cassagne Apr 13 '14 at 16:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.