How to deliberately fragment a file

Question

I am looking for a way to fragment an existing file in order to evaluate the performance of some tools. I found a solution for NTFS file system called MyFragmenter as described in this thread. However I can't find anything for ext2/3/4... I guest I can develop my own file fragmenter but due to time constraint I would like to find a faster solution. I found some tool like HJ-Split which split a file in smaller bits but I doubt this will simulate file fragmentation.

Is their any solution available for my problem?

Answer 1

If you want to ensure fragmentation but not prevent it (so you only have partial control over what happens), and you don't care about the specifics of the fragmentation, here's a quick & dirty way of doing things.

To create a file of n blocks in at least two fragments:

1. Open the file with synchronous writes, write m < n blocks.
2. Open another file. Add to it until there are at most n - m blocks free on disk. Don't make it sparse by mistake!
3. Write the remaining n - m blocks to the first file.
4. Close and unlink the second file.

You can fragment in more pieces by interlacing more files.

This assumes the filesystem is available for this sort of torture, i.e. not in a multi-user or mission-critical environment. It also assumes the filesystem has no reserved blocks, or the reserved blocks are reserved for your UID, or you're root.

There's no direct way to ensure fragmentation, because Unix systems employ filesystem abstraction, so you never talk to the raw filesystem.

Also, ensuring filesystem-level fragmentation tells you nothing about what happens at lower levels. LVM, software and hardware RAID, hardware-level sector remapping and other abstraction layers can play havoc with your expectations (and measurements).

Answer 2

I have yet to meet a general-purpose file system on Linux that would sacrifice write throughput over having contiguous files. That is, every file system fragments if the pieces are written in a non-sequential order, especially with sparse files.

The easy way: Run a file through a torrent client -- preferably something that does not pre-allocate the file. BitTornado or rtorrent fit this bill. (The former has configurable allocation modes)

The hard way: Split the source file into pieces of some KB in size, shuffle them. Open the destination file. For each piece, seek to its correct position and write it.

Here's a Perl script that does it:

#!/usr/bin/perl

use List::Util qw/shuffle/;
use IO::Handle;

use constant BLOCK_SIZE => 4096;

my ($src, $dst) = @ARGV;

my $size = (stat($src))[7];
my @blocks = shuffle(0 .. ($size / BLOCK_SIZE));

my ($srcfh, $dstfh);
open $srcfh, "<", $src or die "cannot open $src: $!";
open $dstfh, ">", $dst or die "cannot open $dst: $!";
truncate $dstfh, $size; # undefined behaviour

my $buf;
for my $blockno (@blocks) {
  seek $_, $blockno * BLOCK_SIZE, 0 for ($srcfh, $dstfh);
  read $srcfh, $buf, BLOCK_SIZE;
  print $dstfh $buf;
  $dstfh->flush;
}

close $dstfh;
close $srcfh;

You can check for fragmentation with the filefrag command, contained in the e2fsprogs package.

Here's an example of what a torrent does:

# ls -sh amd64memstick-5.1.2.fs.gz
239M amd64memstick-5.1.2.fs.gz
# filefrag amd64memstick-5.1.2.fs.gz
amd64memstick-5.1.2.fs.gz: 585 extents found

Here's what I got with my script (on ext3):

$ ls -sh source.tar
42M source.tar
$ perl fragment.pl source.tar fragmented.tar
$ md5sum fragmented.tar source.tar
f77fdd7ab526ede434f416f9787fa9b3  fragmented.tar
f77fdd7ab526ede434f416f9787fa9b3  source.tar
# filefrag fragmented.tar
fragmented.tar: 395 extents found

EDIT: Never mind, it does not appear to work that well after all, except for largish files (an 1,5 GB file fragments for sure.)

The VM system probably is caching and postpones/reorders too small writes. This is why torrent clients manage to fragment (since they usually don't download at >10MB/s) but my script doesn't. I think it can be tweaked by lowering the vm thresholds. See /proc/sys/vm/dirty_*