Tag Info

New answers tagged

0

Please make sure that file should be in .html format containing URL please refer wget man page . -i file --input-file=file Read URLs from file, in which case no URLs need to be on the command line. If there are URLs both on the command line and in an input file, those on the command lines will be the first ones to be retrieved. The file need not be an HTML ...


1

It seems to be caused by a bug in wget that makes it fail on long URLs, or on writing to file names it had derived from long ursl. See wget starts downloading then stops “cannot write to” This but looks related, for example: bug #21714: File name too long The problem may be solved already in the current version of wget - let us know the version you use ...


1

You should give your URLs with http:// Your url.txt should have http://www.google.com http://www.yahoo.com If you don't have the protocol included you'll get messages stating the following: $ wget --content-disposition -i url.txt url.txt: Invalid URL www.google.com: Scheme missing url.txt: Invalid URL www.yahoo.com: Scheme missing


0

You can use the curl command to download curl -O https://github.com/downloads/SpiderLabs/ModSecurity/modsecurity-apache_2.7.1.tar.gz


24

Usually, we see that when we have stopped a download and the continued/resumed with it again. That way, we are downloading only portion which has not been downloaded already. This happens when you use the -c switch. For example $ wget https://help.ubuntu.com/10.04/serverguide/serverguide.pdf 53% [=======================> ] 531,834 ...


0

Here's a slightly simpler (one-line) approach based on janmoesen's solution. export no_proxy=`echo 10.1.1.{1..255} | sed 's/ /,/g'` Unfortunately, sed chokes on all the arguments passed from the expansion of 10.1.{1..255}.{1..255}, either in his code or mine. So if you really need to expand to 256 * 256 IP addresses, you'll need a different approach, ...


0

What the guys mentioned is true but from the user's point of view there is only one difference: wget is used to download things curl is used to see/test what response you get from a machine For example wget http://www.google.com will download the HTML file located at www.google.com, while curl http://www.google.com will just print onscreen the ...


0

The file name you need can not be derived by wget itself, so it needs to be handeled by a shell script: $ url='http://www.example.com/123/def/ghi/jkl.mno' $ outFile=$(echo "$url" | cut -d / -f 5- | tr / _) $ echo $outFile def_ghi_jkl.mno $ wget "$url" -O "$outFile" Or if you like it as one line: wget "$url" -O "$(echo "$url" | cut -d / -f 5- | tr / ...


2

You can use an external bufffer, and pipe the output through it. For example, the program buffer allows you to buffer up to 1GB, and you can specify at which fill level percentage it should start to write : To buffer 10 blocks of 512 kB (5MB) and write out to the file when the buffer is filled to 85%: wget example.com -O- | buffer -s 512k -b 10 -p 85 > ...


1

Personally, I would just keep the UniProt ACs in the file: $ cat names P32234 P05552 P07701 You can then use the same file for various operations. For example, to download the corresponding flat file from UniProt, feed it into a loop: while read prot; do wget http://www.uniprot.org/uniprot/"$prot".txt -O "$prot".flat done < names Since your ...


2

with w3m: echo 'http://unix.stackexchange.com/questions/148670/save-html-to-text-file' | tee - - - | xargs -n1 w3m -dump | sed '/Save html/!d;N;N;N;N;N;N;N' It seems to me that xargs shouldn't even be necessary - surely there's a setting for multiple urls at once, but I can't grok it at the moment. In any case, xargs works: Save html to text file ...


1

There are two other methods: wget $(<file) and while read -r link; do wget "$link"; done < file


0

I was able to download a public shared file like this: wget --no-check-certificate 'https://docs.google.com/uc?export=download&id=FILEID' -O FILENAME Where FILEID must be replaced by the actual file ID. FILENAME is the path/filename where download will be stored. Note you cannot use a folderid instead of fileid. I have used view source in a folder ...


6

Use -i option: wget -i ./url.txt From man wget: -i file --input-file=file Read URLs from a local or external file. If - is specified as file, URLs are read from the standard input. (Use ./- to read from a file literally named -.) If this function is used, no URLs need be present on the command line. If there are URLs both on the ...


5

wget has an option for doing exactly this: wget --input-file url.txt will read one URL per line out of url.txt and download them into the current directory sequentially. More generally, you can use xargs for this sort of thing, combined with wget or curl: xargs wget < url.txt xargs curl -O < url.txt xargs reads each line of its input and ...


2

I would do this in the shell with wget. while read y; do wget "$y" done < url.txt


0

One way to solve this problem is to collect all the links in a plain text file and do the following:- while read line; do echo "Downloading ${line}" wget $line & done < $1 Save this file as script.sh and make it executable and run it as $ ./script.sh A better way to permanently solve this problem would be to rewrite wget to be ...


0

Use grep with it: wget url -rqO - | grep -oE '[a-z]+://[^[:space:]"]+'


1

It is working for me. --backups is a new option, for example my RHEL6.5 machines do not have it (wget 1.12), and my local Fedora machine does have it (wget 1.14). Have you tried a simple test without the other options? This is what works for me: >mkdir test; cd test >wget -q --backups=4 http://ddg.gg/ >ls index.html >wget -q --backups=4 ...


0

Following the question strict: mycurl() { START=$(date +%s) curl -s "http://some_url_here/"$1 > $1.txt END=$(date +%s) DIFF=$(( $END - $START )) echo "It took $DIFF seconds" } export -f mycurl seq 100000 | parallel -j0 mycurl Shorter if you do not need the boilerplate text around the timings: seq 100000 | parallel -j0 --joblog ...



Top 50 recent answers are included