260

The main differences are: wget's major strong side compared to curl is its ability to download recursively. wget is command line only. There's no lib or anything, but curl's features are powered by libcurl. curl supports FTP, FTPS, HTTP, HTTPS, SCP, SFTP, TFTP, TELNET, DICT, LDAP, LDAPS, FILE, POP3, IMAP, SMTP, RTMP and RTSP. wget supports HTTP, HTTPS and ...


127

Yes both wget and curl support limiting your download rate. Both options are directly mentioned in the man page. curl --limit-rate <speed> Specify the maximum transfer rate you want curl to use. This feature is useful if you have a limited pipe and you'd like your transfer not to use your entire bandwidth. ...


60

The common protocols HTTP, FTP and SFTP support range requests, so you can request part of a file. Note that this also requires server support, so it might or might not work in practice. You can use curl and the -r or --range option to specify the range and eventually just catting the files together. Example: curl -r 0-104857600 -o distro1.iso '...


56

In a few words: wget is a tool to download files from servers curl is a tool that let's you exchange requests/responses with a server wget Wget solely lets you download files from an HTTP/HTTPS or FTP server. You give it a link and it automatically downloads the file where the link points to. It builds the request automatically. curl Curl in contrast to ...


50

Since you didn't specify, I'm assuming you are using wget to download the file. If this is the case, try using it with the -c option (e.g. wget -c <URL>). Please notice that in case the protocol used is ftp (the URL looks like ftp://...) there is a chance the remote server uses an old/ancient ftp daemon which doesn't support resuming downloads (newer ...


47

I was able to download a public shared file using this command: $ wget --no-check-certificate 'https://docs.google.com/uc?export=download&id=FILEID' -O FILENAME Where FILEID must be replaced by the actual file ID. FILENAME is the path/filename where download will be stored. Note you cannot use a folderid instead of fileid. I have used view source in ...


33

By default, wget writes to a file whose name is the last component of the URL that you pass to it. Many servers redirect URLs like http://www.url1.com/app?q=123&gibb=erish&gar=ble to a different URL with a nice-looking file name like http://download.url1.com/files/something.pdf. You can tell wget to use the name from the redirected URL (i.e. ...


33

lftp (Wikipedia) is good for that. It supports a number of protocols, can download files using several concurrent parallel connections (useful where there's a lot of packet loss not caused by congestion), and can automatically resume downloads. It's also scriptable. Here including the fine-tuning you came up with (credits to you): lftp -c 'set net:idle 10 ...


30

Use --download-only: sudo apt-get install --download-only pppoe This will download pppoe and any dependencies you need, and place them in /var/cache/apt/archives. That way a subsequent apt-get install pppoe will be able to complete without any extra downloads.


25

From man wget: -x, --force-directories: [...] create a hierarchy of directories, even if one would not have been created otherwise. E.g. wget -x http://fly.srk.fer.hr/robots.txt will save the downloaded file to fly.srk.fer.hr/robots.txt.


22

For google drive files that are too big, and you get a "Google cannot provide a virus-scan" error message, here is the solution I used: First, attempt to download the file here (follow the steps above): $ wget --save-cookies cookies.txt --keep-session-cookies --no-check-certificate 'https://docs.google.com/uc?export=download&id=FILEID' -O- | sed -rn '...


20

You could specify a list of allowed resp. disallowed filename patterns: Allowed: -A LIST --accept LIST Disallowed: -R LIST --reject LIST LIST is comma-separated list of filename patterns/extensions. You can use the following reserved characters to specify patterns: * ? [ ] Examples: only download PNG files: -A png don't download CSS files: -R css ...


18

Unix as a standalone entity doesn't exist as a modern operating system. As indicated by the comments for unix-derivatives: Several systems started with Unix source code, but this was written out over time so that no original Unix code remains. The best known examples are OpenSolaris, FreeBSD, NetBSD and OpenBSD. FreeBSD 5.3 2004-08 OpenBSD 4.6 2009-...


17

Actually, the major difference is that curl includes a library (libcurl), and that library is widely used by other applications. wget is standalone.


16

Use this command: sudo apt-get download pppoe this command let you download the package and requirements into the current directory. Also for accessing to installed .deb files, you can check this path: /var/cache/apt/archives


15

To get the structure you are asking for, I would suggest using -nH as well as -x. This will remove the hostname as well as create the expected directory structure. e.g. wget -x -nH http://xyz.abc.com/Dir1/Dir3/sds.exe - 'Dir1/Dir3/sds.exe' saved [1234] From the man page: -nH --no-host-directories Disable generation of host-prefixed directories. By ...


14

You are confusing a few things. "Onclick" actions refer to JavaScript and are client-side. You would have to examine what the JavaScript hook on those links does to unravel the URL. However, there are no onclick actions in play here. What the web site in question does is referrer checking, also known as "hotlink protection". The browser sends a referrer ...


14

If you follow your description the command becomes "obvious" (at least, when you see it...). You really weren't far off with your attempt, though: ssh user@remote tar czf - /my/directory/ > /my/local/destination/archive.tgz (The "duplicate" link cited by Marco provides a cleaner solution. To create this command here I've simply followed the style of ...


13

There is a -nc (--no-clobber) option for wget.


13

Launch it with javaws (java webstart), e.g javaws ~/Downloads/ContestAppletProd.jnlp


13

I wouldn't call it multithreading as such but you could simply launch 70 jobs in the background: for i in {1..70}; do wget http://www.betaservice.domain.host.com/web/hasChanged?ver=0 2>/dev/null & done That will result in 70 wget processes running at once. You can also do something more sophisticated like this little script: #!/usr/bin/env bash ...


12

There is one way that works for me (based on this tutorial: https://www.marstranslation.com/blog/how-to-make-direct-link-of-onedrive-files) Right-click on the file you are interested in download (from web interface), and choose Embed. Press "Generate HTML code to embed this file". Copy the part contained in the "" of src is your link. This will look like . ...


12

I can't test this for you in your situation, but you should not be using --range with -C -. Here's what the man page has to say on the subject: Use -C - to tell curl to automatically find out where/how to resume the transfer. It then uses the given output/input files to figure that out. Try this instead: curl -s --retry 9999 --retry-delay 3 --speed-...


11

This kind of site tries to make it difficult not to use a graphical web browser, because if you use wget you'll be missing all these ads that pay for the bandwidth. Some sites don't make advanced checks and can be tricked easily: tell wget to pretend that it's really Mozilla and that it's coming from the download site. wget --user-agent='Mozilla/5.0 (...


11

If you want to be hard core, you can run a V7 Unix on a PDP-11 emulator. I've done this, and it had better performance than VAX running 4.2BSD I used in college.


11

Check out transmission-cli. The usage is as simple as running transmission-cli <torrent-file>, but you can obviously tune it to your needs with several options. Just a side comment: Actually you could use many other options, apart from transmission-cli and there will probably appear many other suggestions here (like deluge by Benjamin B. in the ...


11

You may use the wget utility. It has a really simple syntax, and all what do you need is to: wget http://link.to.file and it will be stored in the same directory where do you run wget. If you'd like to store a downloaded file somewhere else, you may use -P option, e.g. wget -P /path/to/store http://link.to.file


11

it's generally a mistake in tech to mistake ones own fundamental ignorance for a flaw with the technology one is completely ignorant of. I tested this, and found the issue immediately: wget respects robots.txt unless explicitly told not to. wget -r http://www.comp.brad.ac.uk/research/GIP/tutorials/index.html --2015-12-31 12:29:52-- http://www.comp.brad....


11

you can use curl -L -O --retry 999 --retry-max-time 0 -C - http://url --retry 999 : retrying so many times --retry-max-time 0 : prevent it from timing out on retrying or curl -L -o 'filename' -C - http://url Update export ec=18; while [ $ec -eq 18 ]; do /usr/bin/curl -O -C - "http://www.example.com/big-archive.zip"; export ec=$?; done ...


10

check iftop and nload iftop does for network usage what top(1) does for CPU usage. It listens to network traffic on a named interface and displays a table of current bandwidth usage by pairs of hosts. Handy for answering the question "why is our ADSL link so slow?". nload is a console application which monitors network traffic and bandwidth usage in ...


Only top voted, non community-wiki answers of a minimum length are eligible