Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

For everything in this question pretend the system only has one disk and filesystem. (we are not writing to different partitions, disks or filesystems)

I am working on a project that cats very large .MTS files into one huge .MTS file. This requires reading each small file and writing them to a new bigger file then deleting the small files. This takes a very long time with files this big.

My understanding - cp takes longer than mv because cp reads the file and writes it to a different place on the disk. mv on the other hand doesn't copy or move the file. mv removes the reference to the file and creates a new one at the new location. For instance mv /tmp/foo /tmp/bar leaves the file as is on disk and removes the reference that directs /tmp/foo to the file on disk and adds the new reference that points /tmp/bar to the file on disk.

The Question:

cat is like cp because it copies the file to the new location. With such large files and no need for the smaller files when I am done, is there something similar to cat that uses mv instead of cp?

Theory (I may have it wrong)

It is already common for files to be stored scattered about the drive. For instance a 2GB file might have several smaller chunks stored in different parts of the drive. This way when a 5K files is deleted it can be overwritten with part of a 20MB file. If we left the 2GB files where they are and just reference all the parts it seems like we could make the same effect as cat foo/* >> bar/bigfile.MTS; rm foo/* in a fraction of the time.

If there is nothing out there that does this and it is a bad idea, can anyone give me example of why? Is it bad to encourage mucking up the disk with scattered file chunks?

share|improve this question
    
I'm fairly certain that doing mv doesn't even remove and create a new reference, it simply changes the name (and possibly the parent, if a directory change is involved) of the existing reference. Of course, once you cross file systems, mv is just as slow as cp is. –  Alex Jul 17 at 22:39
    
I don't ever work with filesystems so I don't know a lot about how they work. I was just assuming based off of my experience. I wanted to say more to point out my ignorance on the topic but didn't want to get off topic in the question. –  DutGRIFF Jul 18 at 0:40
    
@Alex I changed 'partition' to 'filesystem' above. –  DutGRIFF Jul 18 at 0:45
    
Hey, no worries, I wasn't trying to harp on your ignorance of file systems or anything, I was simply trying to point out that mv is probably even more efficient than you first imagined (again, assuming file system boundaries aren't crossed). :) –  Alex Jul 18 at 7:14

2 Answers 2

up vote 5 down vote accepted

The largest obstacle to a tool like this existing is that unless each file's size (except the last one) being concatenated is exactly divisible by the block size (I'm a little uncertain about the right terminology here), you'll end up with "gaps" with garbage data between your concatenated files in the final file.

This is because file data is typically stored in blocks with specific sizes on the file system, such that a 618 byte file stored on a file system using 32 byte blocks would take up 618 / 32 = 19.3125 blocks, i.e. 19 full blocks, and about 1/3 of an additional block.

Assuming you wanted to combine two files like this without regarding my obstacle, you'd simply point the "new file" to the blocks of the first file, plus the blocks of the second file, right?

With that naïve approach, you'd end up with a file of 40 blocks, with its block 20 being 1/3 sensible and 2/3 garbage, and block 21 starting the second file's data.

With some file formats, you might be able to do some clever calculations and manipulations of file headers to basically tell the application that will be using the file to skip the garbage parts, but that's more of a band-aid solution than a proper one.

share|improve this answer
    
TBH, a filesystem that really wanted to get around this problem could simply add internal support for the concept of partially filled blocks. In fact, it looks as though Btrfs might actually already support it at the system call level using clone ranges. I didn't confirm whether or not you could really use this to clone ranges from multiple source files into a single destination file, but if it isn't actually implemented then nothing prevents it in theory. Someone even wanted to add support for clone range in cp! –  Celada Jul 17 at 22:30
    
Right, a filesystem that wanted to accomplish anything could feasibly have support for it. That wouldn't be of much use in the generic case, however, like a "file combiner" tool that would theoretically work on all file systems. I mean, my answer makes some assumptions already, there's no rule saying a filesystem has to be block-based in the first place... –  Alex Jul 17 at 22:33
1  
If it is pioneered first by a single type of file system, it could become generic later. For example after Btrfs added support for cloning, someone became interested in supporting the same thing via the same system call in a fuse filesystem. Also, your remark "there's no rule saying a filesystem has to be block-based in the first place..." is quite on the mark. I guess all is possible as long as we're theorizing about potential future filesystem features! :-) –  Celada Jul 17 at 22:41

Here is a similar question from stackoverflow http://stackoverflow.com/questions/5893531/fast-concatenate-multiple-files-on-linux

The simple answer is this-

It could only work if all the files (except the last) were guaranteed to have a size that is a multiple of the filesystem's block size.

Because the filesystem needs to read through all the blocks until the end of the file.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.