Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I am looking for a way to fragment an existing file in order to evaluate the performance of some tools. I found a solution for NTFS file system called MyFragmenter as described in this thread. However I can't find anything for ext2/3/4... I guest I can develop my own file fragmenter but due to time constraint I would like to find a faster solution. I found some tool like HJ-Split which split a file in smaller bits but I doubt this will simulate file fragmentation.

Is their any solution available for my problem?

share|improve this question
add comment

2 Answers

up vote 6 down vote accepted

If you want to ensure fragmentation but not prevent it (so you only have partial control over what happens), and you don't care about the specifics of the fragmentation, here's a quick & dirty way of doing things.

To create a file of n blocks in at least two fragments:

  1. Open the file with synchronous writes, write m < n blocks.
  2. Open another file. Add to it until there are at most n - m blocks free on disk. Don't make it sparse by mistake!
  3. Write the remaining n - m blocks to the first file.
  4. Close and unlink the second file.

You can fragment in more pieces by interlacing more files.

This assumes the filesystem is available for this sort of torture, i.e. not in a multi-user or mission-critical environment. It also assumes the filesystem has no reserved blocks, or the reserved blocks are reserved for your UID, or you're root.

There's no direct way to ensure fragmentation, because Unix systems employ filesystem abstraction, so you never talk to the raw filesystem.

Also, ensuring filesystem-level fragmentation tells you nothing about what happens at lower levels. LVM, software and hardware RAID, hardware-level sector remapping and other abstraction layers can play havoc with your expectations (and measurements).

share|improve this answer
    
Thanks for the answer! So it seems that I will not be able to do it on a running system. However, as I'm going to use a disk image, I can access the raw data when I've done the acquisition. Thus, I would bypass LVM and such. Now, I'll see if I can do something similar to your proposition on a disk image. –  Flanfl Mar 18 '12 at 18:04
    
Out of curiosity, what are you trying to do on a high level? –  Alexios Mar 18 '12 at 19:01
    
I'am trying to evaluate file carvers such as Scalpel link and file fragmentation is a main issue for these tools. –  Flanfl Mar 18 '12 at 19:09
    
Well, you don't necessarily need a real filesystem for this: you can make up a disk image with interspersed noise and signal blocks from various sources. This gives you proper, full control over the experimental methodology. –  Alexios Mar 18 '12 at 19:16
    
You are right, however the disk image will be used to evaluate other tools. And it would be nice to use one disk image for all the tools (instead of creating specific one). I can use a running OS to generate user activities and such. Plus, a requirements of the project is to use realistic disk image. So fragmenting the file in the running system would be better but if I can't (or the results are too unreliable) I'll have to find a workaround, like working on the raw disk image. Another solution would be to move to Windows... –  Flanfl Mar 18 '12 at 19:42
add comment

I have yet to meet a general-purpose file system on Linux that would sacrifice write throughput over having contiguous files. That is, every file system fragments if the pieces are written in a non-sequential order, especially with sparse files.

The easy way: Run a file through a torrent client -- preferably something that does not pre-allocate the file. BitTornado or rtorrent fit this bill. (The former has configurable allocation modes)

The hard way: Split the source file into pieces of some KB in size, shuffle them. Open the destination file. For each piece, seek to its correct position and write it.

Here's a Perl script that does it:

#!/usr/bin/perl

use List::Util qw/shuffle/;
use IO::Handle;

use constant BLOCK_SIZE => 4096;

my ($src, $dst) = @ARGV;

my $size = (stat($src))[7];
my @blocks = shuffle(0 .. ($size / BLOCK_SIZE));

my ($srcfh, $dstfh);
open $srcfh, "<", $src or die "cannot open $src: $!";
open $dstfh, ">", $dst or die "cannot open $dst: $!";
truncate $dstfh, $size; # undefined behaviour

my $buf;
for my $blockno (@blocks) {
  seek $_, $blockno * BLOCK_SIZE, 0 for ($srcfh, $dstfh);
  read $srcfh, $buf, BLOCK_SIZE;
  print $dstfh $buf;
  $dstfh->flush;
}

close $dstfh;
close $srcfh;

You can check for fragmentation with the filefrag command, contained in the e2fsprogs package.

Here's an example of what a torrent does:

# ls -sh amd64memstick-5.1.2.fs.gz
239M amd64memstick-5.1.2.fs.gz
# filefrag amd64memstick-5.1.2.fs.gz
amd64memstick-5.1.2.fs.gz: 585 extents found

Here's what I got with my script (on ext3):

$ ls -sh source.tar
42M source.tar
$ perl fragment.pl source.tar fragmented.tar
$ md5sum fragmented.tar source.tar
f77fdd7ab526ede434f416f9787fa9b3  fragmented.tar
f77fdd7ab526ede434f416f9787fa9b3  source.tar
# filefrag fragmented.tar
fragmented.tar: 395 extents found

EDIT: Never mind, it does not appear to work that well after all, except for largish files (an 1,5 GB file fragments for sure.)

The VM system probably is caching and postpones/reorders too small writes. This is why torrent clients manage to fragment (since they usually don't download at >10MB/s) but my script doesn't. I think it can be tweaked by lowering the vm thresholds. See /proc/sys/vm/dirty_*

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.