Tag Info

New answers tagged

4

It's because wget is newely invoked for each url. Use the -i option to feed a list of URLs to wget: $ wget -i urls.txt --wait=30 From the manual: -i file --input-file=file Read URLs from a local or external file. If - is specified as file, URLs are read from the standard input. (Use ./- to read from a file literally named -.) If this function is used, ...


2

You are missing --html-extension flag. I would suggest using this syntax instead: wget \ --recursive \ --no-clobber \ --page-requisites \ --html-extension \ --convert-links \ --restrict-file-names=windows \ --domains website.org \ --no-parent \ www.website.org


3

What could be causing this problem? Have you checked that the actual site always returns documents with an extension? There is no protocol which requires this and it is completely normal to not use them; the document type is determined by the http headers and not any suffix on the address. Of course, your file browser does use file extensions to ...


0

It isn't clear from the question if there is SSH access available to the server. If there is, I would simply use rsync for the job.


2

I don't think you need to use wget's recursive downloading options because you already have a list of the files that need downloading: wget --input-file=./path/to/your/list --base=URL In this way you would just follow the list in the first place, and so every file downloaded would be just the next in the list. So you could always find your progress by ...


2

You can do this with comm (using bash process substitution): comm -13 <(find / | sort) <(sort server_dump) This will show the files which are unique to the server. For files that are unique to the local system: comm -23 <(find / | sort) <(sort server_dump) You may also wan to add the -xdev option to find to stop it from comparing files on ...


0

Why not just dump the output of find to a file, on both hosts, and then do a diff on the two files? To get the number of files, you could count using wc.


0

Have you tried wait{!}? It will wait for all background processes to finish before proceeding.


3

wait$! is a syntax error, you're missing a space. This happens to have no impact here because you never actually start a background job, so $! remains empty and the command is wait with no argument, which waits for all background jobs — and since there are none, it returns immediately. There's a difference between using wget --background … and wget … &. ...


2

As suggested, you could both try the command wait #instead of wait$! because I really doubt that there's any command named wait$! maybe what you want here is using wait $! or as it's a cron script you could maybe disable the option of wget --background because you want to wait for the end of this command and not put it in the background.



Top 50 recent answers are included