0

I am trying to download a streaming mp3 using wget. this is my basic command:

wget http://sj128.hnux.com/sj128.mp3 -c --timeout=1 --waitretry=0 --tries=0 -O "file.mp3"

i have been doing this in a script (which lets this run for 1 hour), but what i have been infuriatingly finding is that my file would end up truncated and incomplete. for example, i would expect the file to be, say around 30MB and it would only be something like 13MB.

i didn't understand what was happening until i ran this command directly from the CLI and saw that eventually i'd always run into a "read timeout". this shouldn't be a show stopper. the -c and infinite retries should handle this FINE.

but instead, after a "read timeout" and a new retry, my file would stop growing even though the download continued.

why does the download continue but the file not continue to grow as expected? i went so far as to create an elaborate script which started a completely new wget under a completely different file name to avoid a "file" type of conflict and even though ALL OUTPUT showed a completely different file name with a completely new process, IT STILL DIDN'T WRITE A NEW FILE!

in this case, why does the download appear to commence and my new file does't even show up!?

  • Have you tried running it without the —timeout parameter? Also, -c isn’t necessary if you start a fresh download each time and don’t need to continue a partial download. – Peschke Feb 7 at 9:09
  • You need to provide more details here. I'm going to assume that you're on some very flaky internet connection. Could you please share the output you see? And preferably your script as well. Also, the version of Wget you're using would be useful. – darnir Feb 7 at 9:32
  • You've stated this is a streaming MP3. Streams have strange behaviour and a "retry" might not behave as wget expects. I don't know of any options to enable wget to stream anything. My first suggestion would be to try again with "curl" instead of wget and see if you have any more luck. – Philip Couling Feb 7 at 9:45
  • darnir, take a look at Kamil and bu5hman's posts. because they truly understand wget, they both were able to recognize that i provided enough information to analyze the problem and provide meaningful solutions. – Low Information Voter Feb 9 at 16:31
1

It's a realtime stream. The whole concept of "resuming" doesn't apply since there is neither beginning nor any fixed position to resume from. You just get whatever data is streamed at the moment.

But wget doesn't know that. After a network failure a resume attempt looks like this:

  1. wget knows how large the file is. If the server supported resuming, wget would asks it to resume from the middle of the source file; but there is no such file on the server side, the server doesn't support resuming and this approach fails.
  2. Attempt to resume fails, so wget believes it receives the same data from the very beginning. It discards data until the discarded amount reaches the old file size. Then it start appending new data to the file. This is the moment your file begins to grow.

In effect you miss not only some fragment of the stream when there are problems with the connection; you additionally miss data you otherwise can save, only because wget assumes it receives the same data for the second time.

To overcome this, start (and keep restarting if needed) the following:

wget http://sj128.hnux.com/sj128.mp3 -O - >> "file.mp3"

(with additional options if you wish). Anything wget receives will be appended to the file. If you miss some fragments of the stream, the resulting file will obviously store "shredded" content. In my tests VLC had no technical problems playing such file though.

Note: at any time you can truncate the file to zero size with : > file.mp3. This will work even if wget is running, because >> always seeks to the end of the given file (see this).


started a completely new wget under a completely different file name to avoid a "file" type of conflict and […] it still didn't write a new file!

Unable to reproduce. My wget does write a new file.

  • Thank you for your understanding! Here's my test code and a video of the results. I'm pretty sure this will be the accepted answer once I test the code you provided, even though i don't understand that extra hyphen after the -O argument.. – Low Information Voter Feb 9 at 17:06
  • @LowInformationVoter Extra hyphen is explained in the manual entry for -O option. It makes wget write to its stdout. – Kamil Maciorowski Feb 9 at 18:48
  • even though your solution works perfectly, any clue why a new file isn't written in my demonstration? – Low Information Voter Feb 9 at 23:24
0

In the absence of anything better to do I expended a little of my employers bandwidth in play.

The file you are trying to download is in excess of 230MiB. That's as far as it got before before I shut it off.

The download is throttled to 15kBps at the other end so if you are on a 'flaky' connection, as it appears you are, then the connection is going to drop.

When wget tries to resume it simply appears that the server at the other end does not support resume and so wget just goes back to the beginning and starts again.

This is wholly expected behaviour. From the man page for wget under certain conditions.

 Note that -c only works with FTP servers and with HTTP servers that support the 
"Range" header.

The file doesn't get bigger because

  1. Your connection drops after every (I would guess, 30,000MiB / 15kBps) = half an hour(ish)

  2. wget has to start from the beginning of the file every time your connection drops. Your file manager just sees the max used the last time and reports that size while wget simply overwrites the original file piecemeal from the beginning until your connection drops....again.....and again....and..

Run wget for 10 secs, stop it and then run it for 20 secs. On the second run you will see the file size increasing again once the downloaded size exceeds the previous.

Although not specifically documented with -c (at least I cant find it) this behaviour is expected under certain conditions. Read the man wget section on -nc

  • Thank you for taking your time and for understanding this far better than I do! I ignored #1 because true or not, it is simply catalyst to the real issue: my file not growing after a reconnection. I was absolutely able to reproduce #2. They edited my post to remove that this was going to make me jump out of a window, but the easily reproducible knowledge in #2 helps avert that (now edited) catastrophe! i just now need to figure out why starting a new file doesn't work for me... – Low Information Voter Feb 9 at 16:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.