19

The GNU md5sum command has two modes: binary mode and text mode. I guess the difference is only on how newline characters are handled? Am I right?

On GNU/Linux, the two modes always produce same result, so the only use of -b and -t options is to indicate the flag (* or ) used before the file name?

In what circumstances can the modes produce different results? On Windows/MacOS systems? (Versions for these platforms available?)

1

3 Answers 3

13

On GNU/Linux, the two modes always produce same result

Yes, explicitly. From man md5sum:

Note: There is no difference between binary and text mode option on [sic] GNU system.

This is from the md5sum implementation that ships with GNU coreutils 8.21; I notice an older version (8.12) does not have this notice but I presume the same would be true anyway.

Although AFAICT md5sum is not officially standardized (e.g., by POSIX), it is available on various platforms in various implementations and there is obviously some effort to make these compliant with one another for ease of use across systems.

In relation to this, the ISO/ANSI C Standard includes high level stream functions for accessing files. As part of the standard, these are available on any operating system which implements ISO C via a shared library or a compiler. Since pretty much all operating systems have this available (and are themselves most often written in C), it is a sort of universal language used to implement potentially very portable software.

Considering what it does, it would be totally feasible to write an md5sum that would compile and work on any operating system. I am not claiming this is true of the GNU coreutils version, but one of the high level file stream functions mentioned earlier is fopen(), which is mandated by ISO C to include a b switch used in opening a file to indicate it is being opened "as binary file". What that may mean or require of the system isn't stipulated by the standard, it's just required to exist so it can be used on system where there may be some (any) reason for it.

There is no such reason on linux/POSIX/*nix-style operating systems, so the switch does nothing. From the POSIX spec (a superset of ISO C) for fopen():

The character 'b' shall have no effect, but is allowed for ISO C standard conformance.

So, a completely portable md5sum implementation might use the ISO high level file stream functions, since there are no other methods for accessing files in ISO C (most platforms, including POSIX complaint ones, have their own lower level methods as well, but using these would not be portable because they are not in ISO C), and it should also implement the -b and -t flags to add or not add the b option to fopen() when it reads the file. On systems where that is meaningless, it won't make any difference.

Again, I'm not saying GNU's md5sum is written in such a completely portable way or derived from one that is, but obviously it is trying to comply, in its operability, with one that is. Note that having a flag which does nothing is not the same as not having the flag -- in the former case, it is specified to be okay but do nothing, whereas in the later case using it could be an error or lead to undefined behaviour.

4
  • 2
    Why do these options are added if they are useless? Commented May 6, 2014 at 14:31
  • 2
    I've added some paragraphs above about this (TL;DR -> to make it complaint with versions of md5sum users may be familiar with from other operating systems, and batch code, etc. written for such).
    – goldilocks
    Commented May 6, 2014 at 15:05
  • For all readers about the use of checksums on digital preservation: the binary mode is the standard. Commented Aug 7, 2020 at 7:53
  • 1
    md5sum, sha256sum man page says: Note: There is no difference between binary mode and text mode on GNU systems.
    – yurenchen
    Commented Jan 23, 2023 at 1:12
3

As others have already stated, on GNU systems and on modern Windows, these options have no effect whatsoever.

Looking at the source code, when md5sum is used with a file, the text/binary options determine if the mode parameter used in the fopen(const char *pathname, const char *mode) will be "r" or "rb". On all POSIX conforming systems, the b is simply ignored and has no effect.

When md5sum is reading standard input coreutils will default to text mode if and only if all of the following is true:

  • the compile-time macro O_BINARY is defined (i.e. there is a difference between text and binary modes),
  • input is coming from a terminal
  • and the mode is not overridden by a command-line option.

Otherwise, binary mode is assumed (unless overridden by command-line option) and if O_BINARY is defined, xset_binary_mode() is called for standard input.

xset_binary_mode() is a wrapper defined in xbinary-io.h of gnulib. If O_BINARY is undefined, it will be an dummy function which will get optimized away by any good C compiler - so again, text vs. binary ends up having no effect even if the function would be called for some reason.

But when O_BINARY is defined, xset_binary_mode() will be a wrapper for set_binary_mode(), declared in binary-io.h of gnulib. From here, we'll find first clues about programming environments for which the "binary mode" actually makes a difference:

#if O_BINARY
# if defined __EMX__ || defined __DJGPP__ || defined __CYGWIN__
#  include <io.h> /* declares setmode() */
#  define __gl_setmode setmode
# else
#  define __gl_setmode _setmode
#  undef fileno
#  define fileno _fileno
# endif
#else
  /* On reasonable systems, binary I/O is the only choice.  */
  /* Use a function rather than a macro, to avoid gcc warnings
     "warning: statement with no effect".  */
BINARY_IO_INLINE int
__gl_setmode (int fd _GL_UNUSED, int mode _GL_UNUSED)
{
  return O_BINARY;
}
#endif

Apparently, __EMX__ is Eberhard Mattes eXtender, a 32-bit mode programming environment for MS-DOS and OS/2. Likewise, __DJGPP__ refers to the DJGPP 32-bit development system for MS-DOS. And then there's also Cygwin. All these programming environments rely on setmode() declared in <io.h>.

For other programming environments that define O_BINARY, _setmode() (as defined by the respective programming environment) will be used.

One example of operating systems where text mode vs. binary mode can be important would be OpenVMS. It also knows the Unix-style way of storing text as simply a stream of characters. In the OpenVMS world, this is apparently known as the Stream_LF format - and the fact that it has a specific name should be a hint that it is not the only possible way to format text files.

For the curious: http://neilrieck.net/docs/openvms_notes_text_files.html

To summarize, in the OpenVMS, the non-stream simple text file format contains zero or more records (lines) whose maximum size is defined in file metadata (can be up to 32767 bytes). Each line is prefixed by a 16-bit value indicating the number of bytes in each line. There is no such thing as "end-of-line character": a line ends when the specified number of bytes has been read. If the number of bytes on a line is odd, a padding 0x00 byte is added to have the beginning of the next line at a word-aligned address. This padding byte is never counted in line length.

GNU Coreutils has certainly been ported to OpenVMS.

1

As supplement I'd add info that on Windows md5sum has also no difference in calculation in --text or --binary mode. Tested on md5sum (GNU coreutils) 8.31 for Windows.

c:\temp\file-fingerprint-test\work-copy1\file-fingerprint-eol (master -> origin)
2021-05-30 - 7:35:52 PM
λ xxd unix-eol.txt
00000000: 7468 6973 2066 696c 6520 6861 7320 756e  this file has un
00000010: 6978 206c 696e 6520 656e 6469 6e67 730a  ix line endings.
00000020: 7468 6973 206d 6561 6e20 6974 2068 6173  this mean it has
00000030: 206f 6e6c 7920 4c46 0a                    only LF.

c:\temp\file-fingerprint-test\work-copy1\file-fingerprint-eol (master -> origin)
2021-05-30 - 7:36:08 PM
λ xxd win-eol.txt
00000000: 7468 6973 2066 696c 6520 6861 7320 7769  this file has wi
00000010: 6e20 206c 696e 6520 656e 6469 6e67 730d  n  line endings.
00000020: 0a74 6869 7320 6d65 616e 2069 7420 6861  .this mean it ha
00000030: 7320 2020 4352 204c 460d 0a              s   CR LF..

c:\temp\file-fingerprint-test\work-copy1\file-fingerprint-eol (master -> origin)
2021-05-30 - 7:36:15 PM
λ md5sum.exe --text *.txt
c8a8c7bb97ab554cff96a76b2a8f89fa  unix-eol.txt
03b3b1458cddff2cf1c15819b1255af3  win-eol.txt

c:\temp\file-fingerprint-test\work-copy1\file-fingerprint-eol (master -> origin)
2021-05-30 - 7:36:46 PM
λ md5sum.exe --bin *.txt
c8a8c7bb97ab554cff96a76b2a8f89fa *unix-eol.txt
03b3b1458cddff2cf1c15819b1255af3 *win-eol.txt

c:\temp\file-fingerprint-test\work-copy1\file-fingerprint-eol (master -> origin)
2021-05-30 - 7:36:57 PM
λ md5sum.exe --version
md5sum (GNU coreutils) 8.31
Copyright (C) 2019 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Written by Ulrich Drepper, Scott Miller, and David Madore.

1
  • Welcome to the site. Please don't post screenshots of console output. They are often difficult to read, the content will not show up in search engine results, and the OP will need to type-copy command examples when trying to apply your solution. Instead, paste it into the question using code formatting.
    – AdminBee
    Commented May 21, 2021 at 9:42

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .