5

I want to pipe in a command to sed like so:

md5sum input.txt | sed 's/^\(....\).*/\1/;q'

This works by only outputting the first 4 characters of the checksum. However, I want to output the first 4 characters, but also have an x in the place of every other characters (redacting info). I'm so lost now.

  • If the filename contains conflicting characters (like a newline) the output of md5sum starts with a backslash \. I shall assume that such starting (optional) character should be allowed, right? – Isaac Oct 27 at 0:58
6

With GNU Sed,

md5sum input.txt | sed 's/./x/5g'

This simply skips substituting the 4 first characters of the string and performs the substitution for all other characters.

A POSIX alternative with Awk (although there is probably something simpler),

md5sum xad | awk '{
  four=substr($0, 1, 4)
  rest=substr($0, 5)
  gsub(/./, "x", rest)
  print four, rest
}' OFS=""
| improve this answer | |
  • What if I wanted to make it recursive solely using the .sed script. Is that possible? – That Guy Oct 23 at 1:36
  • New questions should be posted as their own questions... – Jeff Schaller Oct 23 at 1:52
  • Nice try, but: Note that the sed command will change also the spaces and the file name to x's. Not only the 128 bit file hash. – Isaac Oct 27 at 0:35
  • @Isaac Sure, I'm answering to the question, which reads "Replace all characters except the first four characters". You are thinking of unix.stackexchange.com/q/615942 – Quasímodo Oct 27 at 11:46
4

POSIXly (I think), you could use a sed loop to repeatedly replace the first non-x character following the 4-character prefix:

$ md5sum input.txt | sed '
:a
s/^\(....x*\)[^x]/\1x/
ta
'

Replace [^x] with [^x ] if you only want to do the substitution in the first field (the checksum).

| improve this answer | |
4

With perl if GNU sed isn't available:

md5sum input.txt | perl -pe 's/^.{4}(*SKIP)(*F)|./x/g'

^.{4}(*SKIP)(*F) will prevent replacement of first four characters

|. specifies the alternate pattern that has to be replaced


To change only the checksum:

md5sum ip.txt | perl -pe 's/(^.{4}|\h.*$)(*SKIP)(*F)|./x/g'

If the md5sum output starts with a \ (for ex: if filename has a newline character), then you can use ^\\?.{4} instead of ^.{4} to allow first five characters to be left unmasked.

| improve this answer | |
2

Another perl variant to replace all but the first 4 bytes of the checksum (and the checksum only) with x:

$ md5sum input.txt | perl -pe 's{.{4}\K\S+}{$& =~ s/./x/gr}e'
d632xxxxxxxxxxxxxxxxxxxxxxxxxxxx  input.txt
| improve this answer | |
  • This answers exactly Replace each characters until specifc character seen. Consider putting this answer there. – Quasímodo Oct 24 at 18:09
  • I shall assume that an optional starting backslash \ (for filenames with conflicting characters (like a newline)) should be allowed. So, maybe: 's{\\?.{4}\K\S+}{$& =~ s/./x/gr}e' ? – Isaac Oct 27 at 0:42
1

The problem with Quasímodo's answer is that it also replaces the filename with x's. OP posted a followup question about that. Here's a sed solution that stops at the space:

md5sum always produces a 32-character output for the hash. Instead of detecting a space, you could look for 32-characters then a space and replace the last 28 characters with an X.

md5sum input.txt | sed 's/^\([a-zA-Z0-9]\{4\}\)[a-zA-Z0-9]\{28\} /\1xxxxxxxxxxxxxxxxxxxxxxxxxxxx /g'
35c9xxxxxxxxxxxxxxxxxxxxxxxxxxxx  input.txt

Breaking down the statement:

's/^\([a-zA-Z0-9]\{4\}\)[a-zA-Z0-9]\{28\} /\1xxxxxxxxxxxxxxxxxxxxxxxxxxxx /g'

's/ A                                     / B                             /g'
we're substituting patterns matching A with B globally

's/   [a-zA-Z0-9]       [a-zA-Z0-9]       /                               /g'
we're looking for two groups of alphanumeric  characters

's/   [a-zA-Z0-9]\{4\}  [a-zA-Z0-9]\{28\} /                               /g'
The first group has exactly four characters
The second group has exactly twenty-eight characters

's/ \([a-zA-Z0-9]\{4\}\)[a-zA-Z0-9]\{28\} /                               /g'
The first group is a "capture group" which we can reference later

's/ \([a-zA-Z0-9]\{4\}\)[a-zA-Z0-9]\{28\} /\1                             /g'
We will print out the first group verbatim in the output

's/ \([a-zA-Z0-9]\{4\}\)[a-zA-Z0-9]\{28\} /\1xxxxxxxxxxxxxxxxxxxxxxxxxxxx /g'
We will print x's followed by a space for the next 28 characters

's/^\([a-zA-Z0-9]\{4\}\)[a-zA-Z0-9]\{28\} /\1xxxxxxxxxxxxxxxxxxxxxxxxxxxx /g'
The statement must appear at the start of a line and have a space at the end.
| improve this answer | |
  • (1) An s command that is anchored by ^ or $ cannot work more than once per line, so specifying the g option is pointless and potentially confusing.  (2) As long as you’re sure what the input looks like, you could use \(.\{4\}\).\{28\}  (or even \(....\).\{28\} ) as your from pattern. – Scott Oct 24 at 5:59
1

The MD5 hash is always 32 characters long. The first four characters may be had by cutting with cut -c -4, and the 28 x required to pad out the rest of the hash may be added with printf:

$ md5sum somefile
d68610fdffd770de94818268899d6abb  somefile
$ printf '%sxxxxxxxxxxxxxxxxxxxxxxxxxxxx\n' $(md5sum somefile | cut -c -4)
d686xxxxxxxxxxxxxxxxxxxxxxxxxxxx

You may also use %.4s instead of %s in the printf format string to get rid of the cut:

$ printf '%.4sxxxxxxxxxxxxxxxxxxxxxxxxxxxx\n' "$(md5sum somefile)"
d686xxxxxxxxxxxxxxxxxxxxxxxxxxxx

... although this would only work for formatting the hashes of a single file at a time.

| improve this answer | |
  • The initial word is not always 32 characters long. If the filename requires encoding (like a file name with a newline), the first word starts with a \. Try echo 'a test' >$'file\nname' ; md5sum $'file\nname' – Isaac Oct 27 at 1:11
0

With standard POSIX sed you can do it, although it's more complicated:

md5sum input.txt | sed -E 'h;s/^(.{4}).*$/\1/;x;s/^.{4}(.*)$/\1/;s/./x/g;H;x;s/\n//'

Here is what the sed script does:

h                   copy the pattern buffer into the hold buffer
s/^(.{4}).*$/\1/    keep just the first four characters in the pattern buffer
x                   exchange the pattern buffer with the hold buffer
s/^.{4}(.*)$/\1/    keep all but the first four characters in the pattern buffer
s/./x/g             replace each character with an x
H                   append newline and x's to hold buffer (which has the first four)
x                   exchange again; the pattern buffer has an extra newline, though
s/\n//              remove the newline from the pattern buffer

The pattern buffer now has the first four characters plus x's for all the remaining characters; the cycle ends, and it is printed out.

Note: the -E switch enables extended regex syntax. It isn't strictly necessary here; instead, a backslash could be placed before each open and close parenthesis, but I think it's hard enough to read without extra backslashes in it.

| improve this answer | |
  • Nice try, but: Note that this command will change also the spaces and the file name to x's. Not only the 128 bit file hash. – Isaac Oct 26 at 23:52
  • @Isaac So will the accepted answer. As will the method in the question. – David Conrad Oct 27 at 0:24
  • Yes, the accepted answer suffers of the same issue, but I haven't got around to comment it. In any case, the note from my command would be useful to everyone, IMO. – Isaac Oct 27 at 0:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.