Linux EXT 4: How to list files that occupy a block group? I believe a file can span multiple block groups. Given a block group, how do i enumerate the paths of all files contained in it?

migrated from serverfault.com Sep 13 '13 at 2:18

This question came from our site for system and network administrators.

  • Yes there are blocks, but "block groups"? – jordanm Sep 13 '13 at 2:51
  • Oh, we's gots block groups, flexible block groups, meta block groups, and probably even twerk block groups for all I know. – msw Sep 13 '13 at 3:00

With great difficulty. The e2fsprogs package has the rudiments that you'll need, particularly debugfs which can be used to walk around the filesystem and look at block groups and file allocations. Here are some excerpts from the stats and extent commands of debugfs:

debugfs:   stats
 Group  0: block bitmap at 64, inode bitmap at 80, inode table at 96
           28663 free blocks, 5777 free inodes, 3 used directories, 5777 unused inodes
           [Checksum 0xe713]
 Group  1: block bitmap at 65, inode bitmap at 81, inode table at 596
           0 free blocks, 8000 free inodes, 0 used directories, 8000 unused inodes
           [Inode not init, Checksum 0x9416]
 Group  2: block bitmap at 66, inode bitmap at 82, inode table at 1096
           0 free blocks, 8000 free inodes, 0 used directories, 8000 unused inodes
debugfs:  extents bigfile
Level Entries       Logical        Physical Length Flags
 0/ 1   1/  1     0 - 62499 120569           62500
 1/ 1   1/  6     0 - 12287 133120 - 145407  12288 
 1/ 1   2/  6 12288 - 12499 131524 - 131735    212 
 1/ 1   3/  6 12500 - 24575 145408 - 157483  12076 
 1/ 1   4/  6 24576 - 24999 131736 - 132159    424 
 1/ 1   5/  6 25000 - 30719 157484 - 163203   5720 
 1/ 1   6/  6 30720 - 62499 165888 - 197667  31780 

So you can piece together the information you desire, although a roadmap of ext4 disk layout will come in handy.

I cannot help but wonder why you might want to grovel through a file system in that fashion if you don't have to, but perhaps I don't really want to know.

added in response to comment:

A big design motivation for block groups is to minimize the seek penalties that you are trying to measure. That is, the file system distributes the meta-data of free lists, inode tables and data blocks into block groups across the drive so that the head doesn't have to jump from edge to edge as much as it might if there were only one block group as with FAT filesystems. I don't know if NTFS does it old FAT-style or is smarter like the BSD-FFS and its ext descendants.

A much more reliable way to make the comparison you seek would be through drive partitioning. If, for example, you partitioned a drive with outer, middle, and inner partitions you could force seeking by copying from the outer to inner partition. If you try to use block-groups to force this, the filesystem will "fight" your efforts because it is designed to keep drive access local.

Even in the case of multiple partitions, you can expect the system-wide block cache to conspire with the device driver to delay writes to locations far away using tricks like the elevator algorithm. In short, you're trying to break a big pile of optimization throughout the system all designed to minimize the exact phenomenon you are trying to measure. You might also find that the drive controller electronics get involved in the seek-reduction game and much of that will be completely hidden from you.

You could open a raw device and through the right ioctls force the drive to do the sort of unoptimized writing that you are trying to do, but that experiment would have so little to do with real performance to be mostly useless. If you are going to that trouble you might as well just read the manufacturer's maximum head seek time from the hardware spec and forget about testing. Or you could just run DOS instead.

second response to comments:

Your fseek test likely didn't show any head seek effects because of the block-cache which will often read more than you ask for in the expectation that you'll ask for the following bits soon.

To reiterate: various bits of the system are designed to fight your efforts to impose a seek-penalty on you. At this point, I'd ask if your interest is theoretical or practical. If theoretical, I think you have your answer. If practical, you are stuck contriving a test that will — by definition ­— not be representative of real load.

What question are you really seeking an answer for? Please open a new question; this has gotten overlong already.

  • Thanks for the answer. I did the same kind of mapping mentioned in the answer. I am trying to see the performance difference when copying the contents of one file to other file in 2 scenarios: 1. When the target file is in the same group. 2. when the target is in different(preferably very far away block group (I am not sure if the difference in block group number is what I should worry about)). – Ram Sep 13 '13 at 4:28
  • @Ram see "added" section in my answer. – msw Sep 13 '13 at 9:10
  • Thanks again. I completely agree with you. To be more clear,this is what I am trying to do "Generate 2 file system workloads 1 that has very good performance and the other which is significantly poor than the other."I understood(misunderstood?) from the FFS paper that block access spreading block groups should be slow compared to accessing blocks that stay on the same group. I tried a simple experiment by f-seeking forth and back between blocks that reside on different groups but couldn't measure significant differences. Now, I am wondering that if my first assumption was itself flawed. – Ram Sep 13 '13 at 15:19
  • If that is the case, is there any other way that I can use to generate the desired pattern? – Ram Sep 13 '13 at 15:20
  • @Ram see "second" in my answer. – msw Sep 13 '13 at 21:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.