20

I often see a checksum given next to a file available for download. The purpose of this practice eludes me. It is obviously to detect corrupt files, but what could be the cause of this corruption and is it at all likely?

Surely the file will not be damaged by transmission errors since those are detected by the network protocol. And surely any attacker who could alter the file for malicious purposes could likewise alter the given checksum. Are we checking for hard drive errors? Are those more likely to happen when writing then when reading? Am I missing something important?

Improve this question
3
  • 2
    And surely any attacker who could alter the file for malicious purposes could likewise alter the given checksum. - Agreed, a checksum does not guarantee authenticity if it's not served over HTTPS, or you aren't sure that the SSL certificate belongs to the creator of the software.
    – Mihai
    Jul 13, 2015 at 16:07
  • 1
    The TCP checksum is actually pretty lousy: it's only 16 bits. If you're serving up large files to thousands of people (think: installation DVD images), it's virtually certain that some of those downloads will be undetectably corrupted.
    – Mark
    Jul 14, 2015 at 7:55
  • @Mihai Of course, it probably does decrease the risk a bit, though. For example, if your server is infected by a virus that automatically modifies all binary responses (or just replaces all executables you download). It's not perfect, but it can help in some cases.
    – Luaan
    Jul 14, 2015 at 11:53

7 Answers 7

Reset to default
10

And surely any attacker who could alter the file for malicious purposes could likewise alter the given checksum.

Not always.

You could have a content link along with a checksum served on HTTPS. The link could be a nonencrypted link -- plain HTTP or FTP, or something else.

On the downside, the unencrypted connection can get easily middle-manned, on the upside, it can be faster or more convenient for the webmaster (less computing resources needed and opportunities the network to cache that stuff).

If the checksum is served on an unbroken trusted connection and the payload matches the checksum, you get the best of both worlds (provided the checksum is cryptographically secure).

That said, you've reminded me that there are distros out there that claim to be "secure" and yet their website is on HTTP only, as are the links to their images.

Examples:

It's kind of funny because you can't possibly get more insecure that that. Even if they aren't malicious themselves, any ISP could easily replace both the website and the image with fakes, and getting someone to install a rigged operating system while making it seem like they're getting a "secure" Linux distro is the ultimate pwnage.

Improve this answer
1
  • 1
    There are a lot of things less secure than unauthenticated HTTP, which requires an active MITM to subvert.
    – user253751
    Jul 14, 2015 at 1:53
9

To detect corruption is not entirely correct. To ascertain the integrity of the software would be a more correct usage. Normally a software is not distributed from a single server. The same software may be distributed from many servers. So when you download a particular software, the server closest to your destination is chosen as the download source to increase the download speed. However, these 'non-official' (third party) servers cannot be always trusted. They might/can include trojans/viruses/adware/backdoors into the program which is not good.

So to ensure that the software downloaded is exactly same as that of the 'official' software released by the concerned organisation, the checksum is used. The algorithms used for generating checksums are such that even a slight change in the program results in an entirely different checksum.

Example taken from Practical Unix and Internet Security

MD5(There is $1500 in the blue box.) = 05f8cfc03f4e58cbee731aa4a14b3f03

MD5(There is $1100 in the blue box.) = d6dee11aae89661a45eb9d21e30d34cb

The messages, which differ by only a single character (and, within that character, by only a single binary bit), have completely different message digests.

If the downloaded file has the same checksum as the checksum given on the 'official' website, then the software can be assumed to be not modified.

Side Note: In theory, two different files CAN have the same hash value. For the Hash/checksum algorithm to be considered secure, it should be computationally very expensive to find another file which produces the same checksum.

Improve this answer
4
  • 1
    So if the file and the checksum are provided by the same host, it's somewhat useless?
    – Karolis Juodelė
    Jul 13, 2015 at 14:49
  • May be. The checksum is only a means of ascertaining the integrity. Say in a particular scenario, if an attacker gains access to the FTP server of the organisation, he might alter the software. But you can still use the same checksum to ascertain the integrity IF AND ONLY IF the attacker has not broken into the HTTP server. So If both are under the control of the attacker, he can easily alter both and you wouldn't know the difference.
    – Aswin P J
    Jul 13, 2015 at 15:02
  • 1
    Another situation where the checksum may be relevant is to detect situations where a file transfer is resumed after a hiccup but the file has been changed in the interim.
    – supercat
    Jul 13, 2015 at 19:18
  • @KarolisJuodelė The download link might be in the same website/host. But Where it resolves to might be different based on which server is nearest. Also note that, the checksum page should be https while the download can be any protocol http or ftp
    – balki
    Aug 10, 2015 at 20:51
4

As far as why the TCP/IP error checking doesn't catch everything: From https://stackoverflow.com/a/17083365/2551539

There are different errors that can occur (that TCP will detect) [pointed out by Jacob Krall]:

  • Incorrect order of the packets
  • Loss of packets
  • Corrupt data inside the packet
  • Phantom packets (receiver gets packets that have never been sent)

Edit with some additional info:

Page 9 of this study: http://paperhub.s3.amazonaws.com/8ff1e4414c070e900da8ab3885593085.pdf suggests that there are errors that can go undetected by TCP. My understanding is that it happens when an erroneous datagram (called a "bad twin" in the study) has the same checksum as the intended datagram (called a "good twin" in the study).

Improve this answer
1
  • 2
    Read that answer more carefully - those are all errors that are corrected by TCP.
    – Jacob Krall
    Jul 13, 2015 at 20:28
4

Transmission errors can happen. Link-layer protocols typically contain checksums or error-correcting codes to avoid them, but they aren't perfect: there's a small chance that an error will go uncorrected. TCP packets also contain a checksum, which reduces the probability of errors by 2^16. That makes a very small, but nonzero probability of a transmission error. It's the sort of thing that most people will never unknowingly encounter in their lifetime, but it's not at the never-in-a-billion-years likelihood range of cryptographic checksums.

A hardware error on the client, such as disk corruption, is unlikely to be detected by checking right after downloading, because the checksum will be calculated from the cached copy. Checking boot media for corruption if they've failed to boot is useful, on the other hand — you're really testing the media, and you have a presupposition that the hardware may be bad.

The real reason to compute checksums is in fact to detect software-level errors. These do happen. Possible errors include:

  • A file was partially downloaded. Web servers and browsers tend to be bad at detecting interrupted connections and cleaning up partial files. The error could be during your download, or it could have been during upload, it adds up.
  • There was some corruption along the way. For example, some intermediate node in the distribution of the file decided to apply a text encoding conversion to a binary file. Or some misconfigured server served an error message instead of the content.
  • A variant: the wrong file was uploaded.
  • Rare, but can be useful to protect against: an adversary changed the file but was unable to change the reference checksum. Security infrastructures tend to make it more difficult for an attacker to propagate an invalid checksum than an invalid file. For example, large files are often distributed through mirrors, whereas the checksums are served by a central site with fewer opportunities for tampering (server access only to project leaders, distribution through HTTPS).

In practice, checking the size of the downloaded file catches the most common errors, which are truncated or invalidly-converted files. Checksums have the advantage that they detect strictly more problems.

Improve this answer
2

In theory, the network would deliver every single segment properly and they'd be assembled properly on the disk and nothing would go wrong.

In reality, computers are machines and software, both of which are designed and built by fallible humans. In the event that a download does somehow not come down right for one reason or another, such as the download being through some intermediary device whether innocuous or nefarious which mangles the data, it's nice to have a way to check that the file almost certainly was downloaded as an accurate replica of the file on the provider's side.

A high-quality checksum is a reliable method for validating the integrity of the data.

Improve this answer
0

No checksum can be 100% reliable because many files map to the same checksum.

When we add another checksum to the train we multiply the probability of detecting an error.

There is so much traffic on the internet that errors are actually quite common.

Improve this answer
2
  • There's also bit rot.
    – Deer Hunter
    Jul 13, 2015 at 17:33
  • Which should be detected by the storage Hardware itself, but checksumming being a key feature of ZFS and btrfs, I doubt it's working perfectly.
    – bot47
    Jul 14, 2015 at 7:43
0

Checksum will also help to prevent corrupted download due to the following situation:

The server has an internal error while serving the download hence the download is terminated.

When that happens, there are a few possible outcomes:

  • Good server - the server's implementation of Chunked transfer encoding is not buggy:
    • Good client (like cURL, wget) will be able to inform you this is a bad download since the terminating chunk has never been sent from the server.
    • Bad client will think the download has completed since no more data is being received from the server.
  • Bad server - the server's implementation of Chunked transfer encoding is buggy that it sends the terminating chunk for this bad download:
    • Any client will think this download has completed successfully.

I've seen these behaviours among popular client tools and server frameworks, so when you don't use checksum, then in the case of "good server + bad client" or "bad server + any client", you corrupted download will be unnoticed.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.