16

So I'm trying to split a 64MB file FileCarve.001 into 512 byte segments (each block is 512 bytes long). I need to make sure the file has the same data when split into smaller files, so I cat all the files to standard out and pipe it into sha256sum (there's a lot of files, so I need to do this with find and xargs).

Splitting the file in 512 byte segments seems to garble the data when the output gets split by the split command.

$ dd if=FileCarve.001 bs=512 | split -b512 - splits/img
131072+0 records in
131072+0 records out
67108864 bytes (67 MB, 64 MiB) copied, 4.10824 s, 16.3 MB/s
$ sha256sum FileCarve.001 
3e64100044099b10060f5ca3194d4d60414941c7cb26437330aba532852a60cd  FileCarve.001
$ find splits/ -type f -print0 | xargs -0 cat | sha256sum
25b37f28204895e5d0b1cb160c5fa599d15188baf7e529ccc92a10fdb3f0515a  -

But splitting the file in 1 kilobyte segments (1000 bytes) seems to work just fine.

$ dd if=FileCarve.001 bs=512 | split -b1k - splits/img
131072+0 records in
131072+0 records out
67108864 bytes (67 MB, 64 MiB) copied, 2.06029 s, 32.6 MB/s
$ sha256sum FileCarve.001 
3e64100044099b10060f5ca3194d4d60414941c7cb26437330aba532852a60cd  FileCarve.001
$ find splits/ -type f -print0 | xargs -0 cat | sha256sum
3e64100044099b10060f5ca3194d4d60414941c7cb26437330aba532852a60cd  -

Why are they different? Is there something I don't understand about the way blocks work on a storage device?

In response to a comment: I did clear out the split/ directory on each run.

0

1 Answer 1

40

The order find processes the files is not deterministic. It may be just the same order as the underlying system call gives, which probably depends on the underlying filesystem structure and can be essentially random. Some implementations might process the list somehow, but don't expect it to be sorted.

Let's try with a smaller file. cat frag* reproduces the right file, since shell globs do sort the filenames:

$ split -b512 orig.bin frag
$ cat frag* > new.bin
$ sha256sum orig.bin new.bin 
8d12b42623eeefee872f123bd0dc85d535b00df4d42e865f993c40f7bfc92b1e  orig.bin
8d12b42623eeefee872f123bd0dc85d535b00df4d42e865f993c40f7bfc92b1e  new.bin

But find doesn't, so we get a different file:

$ find . -name 'frag*' -exec cat {} + > second.bin
$ sha256sum second.bin 
821325739ca65d1cb568ecf3a16bd2e01ac4eef1419b4d714834fab07d2f135c  second.bin

Just running find to print the names reveals this nicely:

$ find . -name 'frag*' |head -5
./fragzbgv
./fragzbmg
./fragvt
./fragyd
./fragzayc

That was on Linux and ext4. I think it uses some sort of hashing and trees to store the filenames, thus producing a random-looking order. On tmpfs, I got the list in reverse creation order, which isn't that random but still would mess up this case.

Explicitly sorting the list of filenames should help:

$ find . -name 'frag*' -print0 | sort -z | xargs -0 cat > third.bin
$ sha256sum third.bin 
8d12b42623eeefee872f123bd0dc85d535b00df4d42e865f993c40f7bfc92b1e  third.bin

The fact that it worked for you with 1k blocks, is probably an accident...


As an aside, I'm not sure why you're using dd bs=512 for the input there instead of just cat or giving split the filename directly. The thing dd does is reads and writes with a particular block size, but the pipe between dd and sort doesn't conserve block sizes, it's just a stream of bytes, and the filesystem really shouldn't care what blocksize you use to read the file, be it 512 (2^8) or 521 (a prime) bytes.

3
  • Your suggestion works... I did need to sort the output of find. Should I change the title of the post? Commented Nov 4, 2021 at 19:56
  • 8
    @FlashDaggerX, well, that's part of the SE problem. If you think it's easier to find for any future readers with another phrasing, and care to edit, then go ahead. But it describes the behaviour you saw, and that's ok too.
    – ilkkachu
    Commented Nov 4, 2021 at 20:08
  • 4
    About It's the same order as the underlying system call gives. That's not necessarily true. Some are known to sort the list of files by inode for instance as an optimisation (to minimise the number of HDD head seeks or take advantage of prefetching and caching of the inode table). (in any case, it will still appear as random) Commented Nov 5, 2021 at 11:24

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .