Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

For benchmark and testing purposes I need to be able to allocate a file at a specific offset from the start of the partition. When I create a new file normally, its blocks are placed wherever the file system decides, but I want to control that. In other words, I want to manually pick which blocks are assigned to a file.

I've looked at debugfs, but I can't find any way to do what I want. Though I can mark blocks as allocated and modify the inode, this only works for the first 12 blocks. After that I need to be able to create indirect and double indirect blocks as well, which it doesn't look like debugfs has any capability for.

Is there any way to do this? Any tool that could help me? You may assume that the file system is either ext3 or ext4 and that it has been freshly formatted (no other files exist).

Thanks in advance.

share|improve this question
1  
What's the point of the benchmark? Couldn't you write directly to the device without using a fs? –  Stéphane Gimenez Sep 19 '11 at 10:39
1  
Could you not just repartition the drive to place you test partition where you want it, and the create a file that fills it? –  Kusalananda Sep 19 '11 at 13:30
    
The problem with writing directly to the device is that it overwrites the FS's data structures, which is dangerous if the device is mounted. If it's not mounted, the IO scheduler behaves different invalidating the results (because the behavior of the IO scheduler is one of the factors I wish to determine). The partitioning scheme is one I've used but it's not very flexible. I think I've come up with a solution, though, so if no one comes up with a better way I'll post that myself (later, after I've gotten some sleep). –  Sven Sep 19 '11 at 16:14

2 Answers 2

up vote 2 down vote accepted

I have managed to find a way to do this. It uses a python script which first uses debugfs to find the necesssary number of blocks (including indirect blocks) that the file will need. It then manually writes the indirect blocks to the disk, and invokes debugfs again to mark the blocks as used and to update the file's inode.

The only issue is that debugfs apparently doesn't update the free block count of the block group when you use setb. Although I can set that parameter manually, there doesn't appear to be any way to print the current value so I can't calculate the correct value. As far as I can tell it doesn't have any real negative consequences, and fsck.ext3 can be used to correct the values if needed, so for benchmark purposes it'll do.

If there's any other file system consistency issue I've missed, please let me know, but since fsck.ext3 reports nothing besides the incorrect free block count I should be safe.

import sys
import tempfile
import struct
import subprocess

SECTOR_SIZE = 512
BLOCK_SIZE = 4096
DIRECT_BLOCKS = 12
BLOCKS_PER_INDIRECT_BLOCK = BLOCK_SIZE / 4

def write_indirect_block(device, indirect_block, blocks):
    print "writing indirect block ", indirect_block
    dev = open(device, "wb")
    dev.seek(indirect_block * BLOCK_SIZE)
    # Write blocks
    for block in blocks:
        bin_block = struct.pack("<I", int(block))
        dev.write(bin_block)
    zero = struct.pack("<I", 0)
    # Zero out the rest of the block
    for x in range(len(blocks), BLOCKS_PER_INDIRECT_BLOCK):
        dev.write(zero)
    dev.close()

def main(argv):
    if len(argv) < 5:
        print "Usage: ext3allocfile.py [device] [file] [sizeInMB] [offsetInMB]"
        return

    device = argv[1] # device containing the ext3 file system, e.g. "/dev/sdb1"
    file = argv[2] # file name relative to the root of the device, e.g. "/myfile"
    size = int(argv[3]) * 1024 * 1024 # Size in MB
    offset = int(argv[4]) * 1024 * 1024 # Offset from the start of the device in MB

    if size > 0xFFFFFFFF:
        # Supporting this requires two things: triple indirect block support, and proper handling of size_high when changing the inode
        print "Unable to allocate files over 4GB."
        return

    # Because size is specified in MB, it should always be exactly divisable by BLOCK_SIZE.
    size_blocks = size / BLOCK_SIZE
    # We need 1 indirect block for each 1024 blocks over 12 blocks.
    ind_blocks = (size_blocks - DIRECT_BLOCKS) / BLOCKS_PER_INDIRECT_BLOCK
    if (size_blocks - DIRECT_BLOCKS) % BLOCKS_PER_INDIRECT_BLOCK != 0:
        ind_blocks += 1
    # We need a double indirect block if we have more than one indirect block
    has_dind_block = ind_blocks > 1
    total_blocks = size_blocks + ind_blocks
    if has_dind_block:
        total_blocks += 1

    # Find free blocks we can use at the offset
    offset_block = offset / BLOCK_SIZE
    print "Finding ", total_blocks, " free blocks from block ", offset_block
    process = subprocess.Popen(["debugfs", device, "-R", "ffb %d %d" % (total_blocks, offset_block)], stdout=subprocess.PIPE)
    output = process.stdout
    # The first three entries after splitting are "Free", "blocks", "found:", so we skip those.
    blocks = output.readline().split(" ")[3:]
    output.close()
    # The last entry may contain a line-break. Removing it this way to be safe.
    blocks = filter(lambda x: len(x.strip(" \n")) > 0, blocks)
    if len(blocks) != total_blocks:
        print "Not enough free blocks found for the file."
        return

    # The direct blocks in the inode are blocks 0-11
    # Write the first indirect block, listing the blocks for file blocks 12-1035 (inclusive)
    if ind_blocks > 0:
        write_indirect_block(device, int(blocks[DIRECT_BLOCKS]), blocks[DIRECT_BLOCKS + 1 : DIRECT_BLOCKS + 1 + BLOCKS_PER_INDIRECT_BLOCK])

    if has_dind_block:
        dind_block_index = DIRECT_BLOCKS + 1 + BLOCKS_PER_INDIRECT_BLOCK
        dind_block = blocks[dind_block_index]
        ind_block_indices = [dind_block_index+1+(i*(BLOCKS_PER_INDIRECT_BLOCK+1)) for i in range(ind_blocks-1)]
        # Write the double indirect block, listing the blocks for the remaining indirect block
        write_indirect_block(device, int(dind_block), [blocks[i] for i in ind_block_indices])
        # Write the remaining indirect blocks, listing the relevant file blocks
        for i in ind_block_indices:
            write_indirect_block(device, int(blocks[i]), blocks[i+1:i+1+BLOCKS_PER_INDIRECT_BLOCK])

    # Time to generate a script for debugfs
    script = tempfile.NamedTemporaryFile(mode = "w", delete = False)
    # Mark all the blocks as in-use
    for block in blocks:
        script.write("setb %s\n" % (block,))

    # Change direct blocks in the inode
    for i in range(DIRECT_BLOCKS):
        script.write("sif %s block[%d] %s\n" % (file, i, blocks[i]))

    # Change indirect block in the inode
    if size_blocks > DIRECT_BLOCKS:
        script.write("sif %s block[IND] %s\n" % (file, blocks[DIRECT_BLOCKS]))

    # Change double indirect block in the inode
    if has_dind_block:
        script.write("sif %s block[DIND] %s\n" % (file, dind_block))

    # Set total number of blocks in the inode (this value seems to actually be sectors
    script.write("sif %s blocks %d\n" % (file, total_blocks * (BLOCK_SIZE / SECTOR_SIZE)))
    # Set file size in the inode
    # TODO: Need support of size_high for large files
    script.write("sif %s size %d\n" % (file, size))
    script.close()

    # execute the script
    print "Modifying file"
    subprocess.call(["debugfs", "-w", device, "-f", script.name])
    script.unlink(script.name)

if __name__ == "__main__":
    main(sys.argv)

The script can be used as follows to create a 1GB file at offset 200GB (you need to be root):

touch /mount/point/myfile
sync
python ext3allocfile.py /dev/sdb1 /myfile 1024 204800
umount /dev/sdb1
mount /dev/sdb1

The umount/mount combo is necessary to get the system to recognize the change. You can unmount before invoking the script but that makes invoking debugfs slower.

If anyone wants to use this: I don't guarantee it'll work right, I don't take responsibility if you lose any data. In general, don't use it on a file system that contains anything important.

share|improve this answer
    
Kool that you are sharing your work here. Do you mind asking for improvements on codereview.stackexchange.com (from a few second of reading, I've already spotted 2 issues with the code). –  Tshepang Feb 27 '12 at 10:01
    
@Tshepang: this was throwaway code for a problem that is long since irrelevant to me, so I'm not really interested in improving it at this point. –  Sven Feb 27 '12 at 11:57

This is not the answer you want,and I realize it. but the method is file system independent.

ONE TIME ONLY

find the block size; loop around creating files of that block size from beginning to end; once done; delete all files except the one(s) you want.

Once done, do a raw copy of the device to a compressed file

dd if=/dev/sdp1 |bzip2 -9 > /tmp/my-fs-image.bz2

Voila! A file system image thats not real big and has just one block in the where-ever you want. allocated.

to restore, create a partition with the exact same number/size of physical disk blocks

bzip2 -d < /tmp/my-fs-image.bz2|dd of=/dev/sdq1

That first go-around, it's going to be a hassle, tho.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.