22

Why md5sum is prepending "\" in front of the checksum when finding the checksum of a file with "\" in the name?

$ md5sum /tmp/test\\test
\d41d8cd98f00b204e9800998ecf8427e  /tmp/test\\test

The same is noted for every other utility.

  • Just for reference, the other *sum utilities (of the same family as md5sum, e,g, sha1sum etc.) in GNU coreutils does the same. – Kusalananda Feb 16 '18 at 14:45
  • I don't see this behaviour, what's the version of the utility: md5sum --version – Kiwy Feb 16 '18 at 14:46
  • @Kusalananda This may be coreutils version specific; on CentOS 7 cksum doesn't; eg % cksum test\\test 3915528286 4 test\test – Stephen Harris Feb 16 '18 at 14:47
  • @StephenHarris That probably is because cksum is a POSIX utility and its spec. does not allow it. – Kusalananda Feb 16 '18 at 14:49
33

This is documented, for Coreutils’ md5sum:

If file contains a backslash or newline, the line is started with a backslash, and each problematic character in the file name is escaped with a backslash, making the output unambiguous even in the presence of arbitrary file names.

(file is the filename, not the file’s contents).

b2sum, sha1sum, and the various SHA-2 tools behave in the same way as md5sum. sum and cksum don’t; sum is only provided for backwards-compatibility (and its ancestors don’t produce quoted output), and cksum is specified by POSIX and doesn’t allow this type of output.

This behaviour was introduced in November 2015 and released in version 8.25 (January 2016), with the following NEWS entry:

md5sum now ensures a single line per file for status on standard output, by using a '\' at the start of the line, and replacing any newlines with '\n'. This also affects sha1sum, sha224sum, sha256sum, sha384sum and sha512sum.

The backslash at the start of the line serves as a flag: escapes in filenames are only processed if the line starts with a backslash. (Unescaping can’t be the default behaviour: it would break sums generated with older versions of Coreutils containing \\ or \n in the stored filenames.)

  • 30
    It's a shame something completely unintuitive like this isn't documented in the man pages, though. (And yes, I'm aware GNU wants everyone to read their highly convoluted info pages instead.) – roaima Feb 16 '18 at 15:14
  • 3
    @msouth the backslash at the start of the line serves as a flag indicating that backslashes in the filename are escapes; otherwise you wouldn’t know whether to process \n etc. as literals or escapes. – Stephen Kitt Feb 16 '18 at 21:36
  • 3
    @msouth if it’s at the start of the filename, you’ve got no way of knowing whether it’s the flag, or a filename genuinely starting with a backslash... – Stephen Kitt Feb 16 '18 at 21:46
  • 1
    @StephenKitt I don't think the leading \ is there for disambiguation. There is no ambiguity if the output is documented as always escaping backslashes and newlines. It's there so that de-escaping doesn't have to be done if not necessary. You can of course debate whether this is worth it (personally I think it isn't but I am not a coreutils contributor). – TypeIA Feb 16 '18 at 21:52
  • 1
    The documentation's phrase "each problematic character in the file name is escaped with a backslash" is wrong; replacing a newline with \n is not the same as escaping a newline with a backslash! – ruakh Feb 17 '18 at 21:39
17

Stephen Kitt's answer covers the what and I will try to cover why this change was implemented. First, someone observed that a filename containing newlines1 could result in ambiguous output. For example, consider this output:

d41d8cd98f00b204e9800998ecf8427e  foo
25af89c92254a806b2e93fffd8ac1814  bar

Does this mean there were two files foo and bar, or only one file whose filename is "foo\n25af89c92254a806b2e93fffd8ac1814 bar"? Granted, this latter possibility is highly unlikely, but it is possible. To resolve the ambiguity the developers chose to escape newlines with a backslash (\). The output then becomes distinguishable. However, then there is a further ambiguity:

764efa883dda1e11db47671c4a3bbd9e  foo\nbar

Does this file's name contain a newline, or a backslash followed by an n? To resolve this we need to escape backslashes too, so that the latter case becomes:

764efa883dda1e11db47671c4a3bbd9e  foo\\nbar

Finally, they elected to prepend each output line which contains such escapes with a \\ to make it easy for a parser to detect whether escaping has been done. Presumably this was done to allow parsers to handle output both from escaping versions of md5sum and from non-escaping versions (non-GNU). The flag also means that "costly" un-escaping does not need to be done when not necessary. You can see an example of this parsing in action in md5sum.c itself (line 382 in the linked version).


1 By newline I mean the character \n which is sometimes also specifically referred to as a linefeed or LF; see md5sum.c.

  • 1
    Of course the sane behaviour would be to completely ban every file containing a newline. Just refuse to process them. – pipe Feb 17 '18 at 14:03
  • 1
    @pipe it's insane behavior. POSIX does allow such file names, and the utilities intentionally refusing to work with legitimate files are bad and must be killed with fire. – Ruslan Feb 17 '18 at 14:09
  • 2
    @Ruslan The point is to protest against POSIX for allowing such antisocial names. Allowing such characters has likely caused a large amount of security issues and code bloat just to handle such special cases. – pipe Feb 17 '18 at 14:17
  • @pipe while LF in a file name is indeed antisocial, other things mentioned in your link are much more debatable — like spaces, non-latin letters etc.. – Ruslan Feb 17 '18 at 14:54
  • Classic over-engineering by engineers. Lesson (yet again): do not allow engineers to drive requirements. They will find the most obscure and convoluted case and elevate it to the dominate case and confuse everyone. – user56041 Feb 17 '18 at 15:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.