7

I want to copy a number of files onto a flash drive such that each individual file is contiguous. They do not need to be contiguous with each other, and can be in any order. The files will be very large – hundreds of megabytes, to gigabytes. No other files on the drive should be defragmented; doing so would waste time and cause unnecessary wear on the flash media.

I want to be able to do this for at least FAT32, but methods for *nix filesystems will also be appreciated.

Essentially, there are 2 approaches to this:

  • Copy then defragment the files.
  • Defrag enough contiguous free space for each file, then copy each file into its place.

The 2nd option would generally be far quicker than the 1st, and would avoid causing unnecessary wear on the flash media, so the 2nd option would be very much preferable.

I don't mind solutions that work offline, but obviously online is preferable.

9
  • 4
    What exactly are you trying to achieve? Flash memory is random access and therefore has no seek times, so the file being contiguous doesn't matter. – j883376 Jun 20 '13 at 2:32
  • @j883376: I'm well aware of that. It's to use my flash drive as a boot disk for many ISOs, without having to partition/dd and therefore duplicate each one. – James Haigh Jun 20 '13 at 2:40
  • 1
    I've never heard of this before - doesn't make it not possible - but I did find this Q&A on SO: stackoverflow.com/questions/10071962/… – slm Jun 20 '13 at 2:43
  • 1
    You cannot prevent files being interrupted by the metadata which ext4 writes in certain distances on the disk. I don't know how to calculate this, though. If you cannot accept that then you need to write to a raw disk, avoiding a file system. This can relatively easy be done with LVM (LVs can be fragmented, too! So check with dmsetup). Of course, you need to store the length of the file somewhere in that case. – Hauke Laging Jun 20 '13 at 5:36
  • possibly related unix.stackexchange.com/questions/68189/… – frostschutz Jun 20 '13 at 10:32
3

I have to admit that I don't really understand what you're trying to do but I don't think that matters. I found this tool called defragfs which purports to do what you're asking (if I understand what you're trying to do).

http://defragfs.sourceforge.net/index.html

I think the thing with this tool though, is that it defrags after, where I think you're looking to essentially guarantee that as you write the file to disk, that it's done so in a sequential manner. According to this StackOverflow Q&A titled: how to store a file in continuous disk block in linux, it doesn't sound like's is possible given the architecture of how the writes are done.

Just to round out this Q&A I did find the following resources which sound like they might be useful.

2

It is possible at least for FAT32, however I don't recall the name of the tool.

If I remember right is based on a first call for a full length hole and assign it to the new stream, then the file is read and write on that stream.

It is also possible with NTFS via pre-allocate copy if I remember right.

But, for NTFS compressed files there is no way... since Windows first write the file (non-compressed) and after that it compress the file in chunks... to tell the truth it does that in a pipeline... that is the reason the file get so fragmented (a file of 10GiB can create more than one hundred thousand fragments).

I do not know any tool / command able to avoid that when NTFS compression is enabled. I used a tool a long time ago that had a nice GUI to copy without causing fragmentation in Windows, but I can't seem to recall the name.

So the best I can tell you is to use on windows the command line utility XCOPY with the /J flag.

It tries to not fragment, ideal for big files if NTFS compression is disabled or you are using FAT32.

Explanation for NTFS compression pipeline fragmentation:

  1. Block N will be written on Cluster N
  2. Clusters arround N+# will be compressed and stored somewhere else, so clusters between N and N+# will be free, aka, file will be fragmented, very fragmented. 10GiB = 100000 fragments, assuming a compression of 50%, that is why NTFS compression is very BAD. If the file would be compressed first on RAM then sent to disk no fragmentation would occur or at least it can be avoided.

Another side effect of this way of doing things is assume we have 5Gib of free space and I want to write a file of 6GiB that after compressing will take only 3GiB. This is not possible, but if you first move a 2GiB (non-compressible) to another place, then free space will be 7GiB, write 6GiB, compressing makes it only 3GiB, free space will be 4GiB, then put back the 2GiB of original data that we moved and voila all is there and 2GiB free. Point being that if there is not enough space for uncompressed file, it will not be possible to be copied on NTFS and it does not matter if after it would be NTFS compressed there is enough space. It needs all because it first write it without compression, then it applies the compression. Lastly NTFS drivers do that in pipeline so in theory it would still be possible, but they did not change the check free size part.

Windows does not write the file "after" compressing it, windows "compress" the file "after" saving it uncompressed, so for each cluster the hard disk will see two write attempts, first with non compressed data, second with compressed data. Modern pipeline NTFS controllers avoid HDD to see both writes but on first NTFS version the HDD writes the whole file uncompressed, then it compresses the file. It was very noticeable with very large and well compressible files. Writing a file of 10GiB that NTFS compressed down to only a few MiB (A file filled with zeros) took longer to write than writing the non-compressed version of the file. Nowadays pipeline has break that, and now it took a very little amount of time but the prior enough free size is still there.

Hopefully one day the NTFS compression method will do it to a row, so we could get non-fragmented NTFS compressed files without the need to de-fragment them after we write them. Untill then the best option is XCOPY with the /J flag and CONTIG or that GUI tool I can't remember the name of. It was while Windows was only up to XP, and had options to pause, to copy in parallel from HDD1 to HDD2 and HDD3 to HDD4, and of course the one wished, to pre-allocate. Hopefully this helps!

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.