I'm constantly using wget to download many files over 1GB each. I'm getting concerned about how this may be straining my HDD. Can wget's internal buffer size be modified so that it writes to my HDD less often? What about curl?

  • In what sense is it going to be "straining your HDD" -- do you want this stuff saved on disk, or not? – goldilocks Aug 10 '14 at 13:31
  • @goldilocks I would rather have it write to disk every 1 MB rather than every 1 KB for example. That way it will only write to HDD 1 time rather than about 1000 times. – Matias Casado Aug 10 '14 at 14:03
  • 1
    As long as you have free memory, writes are buffered by the OS disk cache anyway (watch it grow when you do this). I won't say there's anything wrong with what you want to do, but I doubt it is going to make any ultimate difference to the disk's lifespan. – goldilocks Aug 10 '14 at 14:22
  • Yeah, you would probably only do this to keep it from swamping the queue but there are block-level controls for that sort of behavior. The requests are going to be merged by the scheduler anyways. – Bratchley Aug 10 '14 at 14:28
  • Also, hard disks have their own buffer which is at least a few MB: en.wikipedia.org/wiki/Disk_buffer I'm sure there are more than a few engineers that have given some thought to this issue already. – goldilocks Aug 10 '14 at 14:40

You can use an external bufffer, and pipe the output through it.

For example, the program buffer allows you to buffer up to 1GB, and you can specify at which fill level percentage it should start to write :

To buffer 10 blocks of 512 kB (5MB) and write out to the file when the buffer is filled to 85%:

wget example.com -O- | buffer -s 512k -b 10 -p 85 > ./outfile.txt

(may need apt-get install buffer)

But, other than directly answering the question, this does not look like something you should need to do manually. Maybe you can configure your file system to cache write operations a little longer before actually writing to the disk? Depends on what problem you want to solve exactly.

|improve this answer|||||
  • +1 I'm guessing buffer is so obscure because it mostly replicates dd, only with the added -m, -p, and -b features that make what you are suggesting possible. Unfortunately I can't find the upstream source; while it's in debian it's not in e.g. fedora. – goldilocks Aug 10 '14 at 14:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.