Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I'm constantly using wget to download many files over 1GB each. I'm getting concerned about how this may be straining my HDD. Can wget's internal buffer size be modified so that it writes to my HDD less often? What about curl?

share|improve this question
    
In what sense is it going to be "straining your HDD" -- do you want this stuff saved on disk, or not? –  goldilocks Aug 10 at 13:31
    
@goldilocks I would rather have it write to disk every 1 MB rather than every 1 KB for example. That way it will only write to HDD 1 time rather than about 1000 times. –  Matias Casado Aug 10 at 14:03
1  
As long as you have free memory, writes are buffered by the OS disk cache anyway (watch it grow when you do this). I won't say there's anything wrong with what you want to do, but I doubt it is going to make any ultimate difference to the disk's lifespan. –  goldilocks Aug 10 at 14:22
    
Yeah, you would probably only do this to keep it from swamping the queue but there are block-level controls for that sort of behavior. The requests are going to be merged by the scheduler anyways. –  Joel Davis Aug 10 at 14:28
    
Also, hard disks have their own buffer which is at least a few MB: en.wikipedia.org/wiki/Disk_buffer I'm sure there are more than a few engineers that have given some thought to this issue already. –  goldilocks Aug 10 at 14:40

1 Answer 1

up vote 2 down vote accepted

You can use an external bufffer, and pipe the output through it.

For example, the program buffer allows you to buffer up to 1GB, and you can specify at which fill level percentage it should start to write :

To buffer 10 blocks of 512 kB (5MB) and write out to the file when the buffer is filled to 85%:

wget example.com -O- | buffer -s 512k -b 10 -p 85 > ./outfile.txt

(may need apt-get install buffer)


But, other than directly answering the question, this does not look like something you should need to do manually. Maybe you can configure your file system to cache write operations a little longer before actually writing to the disk? Depends on what problem you want to solve exactly.

share|improve this answer
    
+1 I'm guessing buffer is so obscure because it mostly replicates dd, only with the added -m, -p, and -b features that make what you are suggesting possible. Unfortunately I can't find the upstream source; while it's in debian it's not in e.g. fedora. –  goldilocks Aug 10 at 14:34

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.