17

I have a bunch of Class 10 UHS-1 SDHC SD cards from different manufacturers. They are all partitioned as follows

 $ sudo fdisk -l /dev/sdj
Disk /dev/sdj: 14.9 GiB, 15931539456 bytes, 31116288 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x0000de21

Device     Boot   Start      End  Sectors  Size Id Type
/dev/sdj1          2048  1050623  1048576  512M  c W95 FAT32 (LBA)
/dev/sdj2       1050624  2099199  1048576  512M 83 Linux
/dev/sdj3       2099200  3147775  1048576  512M 83 Linux
/dev/sdj4       3147776 31116287 27968512 13.3G 83 Linux

I used a memory card duplicator to copy the images. All cards have the same content.

When I mount the second partition of any two SD cards and compare the content, they are exactly the same.

 $ sudo mount -o ro /dev/sdg2 /mnt/system-a/
 $ sudo mount -o ro /dev/sdj2 /mnt/system-b/
 $ diff -r --no-derefence /mnt/system-a /mnt/system-b/
 $ # prints nothing^

However, if I compare the sha1sum of the partitions, they sometimes differ

 $ sudo dd if=/dev/sdg2 | sha1sum
1048576+0 records in
1048576+0 records out
536870912 bytes (537 MB) copied, 12.3448 s, 43.5 MB/s
ee7a16a8d7262ccc6a2e6974e8026f78df445e72  -

 $ sudo dd if=/dev/sdj2 | sha1sum
1048576+0 records in
1048576+0 records out
536870912 bytes (537 MB) copied, 12.6412 s, 42.5 MB/s
4bb6e3e5f3e47dc6cedc6cf8ed327ca2ca7cd7c4  -

Stranger, if I compare these two drives using a binary diffing tool like radiff2, I see the following

 $ sudo dd if=/dev/sdg2 of=sdg2.img
1048576+0 records in
1048576+0 records out
536870912 bytes (537 MB) copied, 12.2378 s, 43.9 MB/s

 $ sudo dd if=/dev/sdj2 of=sdj2.img
1048576+0 records in
1048576+0 records out
536870912 bytes (537 MB) copied, 12.2315 s, 43.9 MB/s

 $ radiff2 -c sdg2.img sdj2.img
767368

767368 changes, even though diff didn't see any differences in the content!

And for sanity, if I compare two partitions that had the same sha1sum, I see the following

 $ radiff2 -c sdj2.img sdf2.img
0

0 changes!

Here is a breakdown of the different sha1sums I see from different cards. It seems like the manufacturer of the card has a large affect on what sha1sum I get when I use dd to read the drive.

enter image description here

Despite differences in sha1sums, all these cards work for my purposes. However, it is making integrety checking difficult because I cannot compare sha1sums.

How is it possible two SD card partitions could have different sha1sums, yet have the exact same content when mounted?

Answer: So now it works as expected. To clear things up, the inconsistency was caused by the SySTOR duplicator I was using. The copy setting I had it use copied partition information and files, but it did not necessary dd the bits to ensure there was a one-to-one match.

Improve this question
2
  • 3
    What kind of testing are you doing with such a bunch of cards? :)
    – h.j.k.
    Dec 18, 2015 at 8:43
  • If you are comparing them after you mount them, therein is your problem.
    – David Hoelzer
    Dec 18, 2015 at 11:47

2 Answers 2

Reset to default
18

Did you compare their contents immediately after writing the duplicated contents? If yes, they should come out exactly the same. For example,

# Duplicate
dd bs=16M if=/dev/sdg of=/dev/sdk

# Comparing should produce no output
cmp /dev/sdg /dev/sdk
# Compare, listing each byte difference; also no output
cmp -l /dev/sdg /dev/sdk

This is only true if the cards have exactly the same size. Sometimes, even different batches of cards that are the same manufacturer and model come out with slightly different sizes. Use blockdev --getsize64 to get the exact size of the device.

Also, if both cards have exactly identical sizes but you wrote an image to both cards that was smaller than the capacity of the cards, then the garbage that comes after the end of the image may cause differences to be reported.

Once you mount any filesystem on the device, you will start to see differences. The filesystem implementation will write various things to the filesystem, such as an empty journal, or a flag/timestamp to mark the filesystem as clean, and then you won't see identical content anymore. I believe this can be the case under some circumstances even if you mount the filesystem read-only.

Improve this answer
3
  • Does the OP need to use blockdev --getsize64?  It looks like dd is announcing the amount of data that it reads.
    – G-Man Says 'Reinstate Monica'
    Dec 18, 2015 at 5:16
  • 3
    EIBTI. Querying the size makes it really clear. dd will report how much it copied. In case of size mismatches between an image file, the size of one device, and the size of another device, etc... that may be the size of the source, the desination, or both.
    – Celada
    Dec 18, 2015 at 5:32
  • You're right. They should be and they are exactly the same. After looking into this further, I found the inconsistency was caused by the copy setting on my SySTOR duplicator. When I dd the SD cards from my computer (as I did with the master image for the duplicator), all the shasums match. I changed the settings on the SySTOR from "systems and files data only" to "whole media" and now all duplicated cards have matching shasums
    – peskal
    Dec 18, 2015 at 22:14
8

To build upon Celada's answer:  On the one hand, you're doing a diff (recursive) between two mounted filesystems.  On the other hand, you're doing a binary compare between devices that have filesystems on them — apparently, after you have mounted the filesystems.  That's apples and pomegranates.

The operation at the mounted filesystem level can see only the data content of the files in the filesystems.  The binary compare between the devices looks at the data and the metadata.  I'm a little surprised by the 767368 differences, but I can guess at a few:

  • When you mount a filesystem, the kernel writes the current time into the filesystem superblock as the "mount time".  If you have mounted both devices (and not at the exact same time), the "mount times" in the superblocks will be different.
  • If you do the device-level binary compare after the recursive filesystem diff, every file on each device will have had its access time (in the inode) updated.

P.S. Do you need to use dd so much?  What happens if you do radiff2 -c /dev/sdg2 /dev/sdj2 or sha1sum /dev/sdg2?

Improve this answer
3
  • Does this apply even when mounting the drive as read-only? I've done the shasum comparison before mounting too and they still differ. I've also never seen the shasum change after mounting as read only. -- Also you're right, I should win a useless use of dd award :p
    – peskal
    Dec 18, 2015 at 17:53
  • (1) No, as you suspect (i.e., consistent with your experience), mounting a filesystem as ro (read-only) should not cause (or allow) any modification. (Although I have seen one or two cases of software doing something other than what it should do.) (2) After reading your comments (one on each answer, at the time of this writing), I still don’t quite understand what happened. Will you please either edit your question or post an answer explaining the circumstances under which you got comparison failure (i.e., it found differences) immediately after duplicating (before mounting),  … (Cont’d)
    – G-Man Says 'Reinstate Monica'
    Dec 19, 2015 at 10:18
  • (Cont’d) …  and what you did to resolve it? (3) I like it, but should it be called “UUOD”, “UUODD”, or “UUDD”? I vote for “UUDD”, but we should probably take this up on Meta.  :-)  ⁠
    – G-Man Says 'Reinstate Monica'
    Dec 19, 2015 at 10:19

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .