3

I use MappedByteBuffer to write file in linux.

File file = new File("testFile");
RandomAccessFile raf = new RandomAccessFile(file, "rw");
FileChannel fc = raf.getChannel();
MappedByteBuffer mbf = fc.map(FileChannel.MapMode.READ_WRITE, 0, file.length());
mbf.put(buffer);

If testFile has write 500MB, and write the 500MB data twice, it takes 1s.but when I rm testFile, write the 500MB data, it takes 4s.

Why override a file is faster than write a new file? How can I write a new file as faster as override a file?

  • 2
    I never used MappedByteBuffer, but if I had to blindly guess maybe your comparison is not valid because in the second case you are including the allocation time in the write time. – andreatsh Jan 11 '17 at 19:50
2

Whether overwriting or creating a new file is faster depends on the filesystem type. Many filesystems overwrite file data in place; then overwriting is faster because it only requires writing the data, whereas creating a new file requires first allocating space and then writing the data in the newly allocated space. I wouldn't expect a large difference though. Some filesystems don't overwrite an existing block (to allow a write to be undone), and then overwriting an existing file is done by writing the new data followed by deleting the old data. I wouldn't expect a large difference in either case though.

The underlying layers can have similar effects to make one operation more costly than the other. For example, overwriting on a system that keeps snapshots keeps the old data around so that the snapshot can be restored. Flash media can only be erased in bulk so new data is written to free sectors but overwriting some data eventually does lead to it being freed which takes time.

By far the thing with the biggest effect on read and write timing is buffering and caching. Make sure that you're doing your benchmarks in a known cache configuration (you should probably flush the disk cache before starting each benchmarked operation) and ends with buffers all written (finish by calling sync) unless you want to measure warm-cache/buffers timings. For example, doing two consecutive writes where the first write only writes to memory buffers won't cost much more than doing a single write.

In any case, if it takes 4s to do the operation you want then it takes 4s. There's no magical way to make it 4 times faster.

| improve this answer | |
1

A quick test using dd shows that appending data to a file is not faster:

My test filesize is 1024MB. The final appended filesize is 2048MB.

creating a new file
real    0m3,052s
user    0m0,523s
sys     0m0,578s

overwriting existing file
real    0m3,510s
user    0m0,695s
sys     0m0,867s

appending to existing file
real    0m3,226s
user    0m0,602s
sys     0m0,594s

deleting file
real    0m0,273s
user    0m0,086s
sys     0m0,195s

test.sh:

#!/usr/bin/env bash
printf "creating a new file\n"
time dd if=/dev/zero of=./test.img count=1024 bs=1M && sync -f

printf "\noverwriting existing file\n"
time dd if=/dev/zero of=./test.img count=1024 bs=1M && sync -f

printf "\nappending to existing file\n"
time dd if=/dev/zero bs=1M count=1024 >> ./test.img && sync -f

printf "\ndeleting file\n"
[[ -f ./test.img ]] && time rm ./test.img && sync -f
| improve this answer | |
  • You are using a different method than the author, and you may be using a different file system. – Joe Zack Feb 27 at 0:29
  • Question answered. – Michael D. Feb 28 at 15:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.