1

Trying to calculate md5 hash of all *.txt files in directory and store each in same filename but with extension replaced from .txt to .md5.

For file: foo.txt i want to create file: foo.md5 with content:

9fa74bee342222aadd999b2e8b2bef62 *foo.txt

I have managed to get needed files with command below, but I wonder is there some much shorter alternative, or at least a way to drop one or two awk commands?

time find *.txt | parallel md5sum | awk '{ print $1, $2, $2 }' | awk '{$1 $2 $3 > gsub(".txt", ".md5", $3); print}' | awk '{ print $1, "*"$2 > $3 }'

I have found some alternatives, but they don't use parallel and the just add .md5 to filename, not replace it. Also they store only md5 hash, I need this *filename modification.

2
  • 2
    Your script will fail on spaces or newlines in filename. And if there are same filenames with only different extension (okay, you only use .txt, I would still go with file.ext.md5 instead of file.md5)
    – pLumo
    May 9 at 13:32
  • You are correct, script is not for general purpose. But this files are machine generated so I know there are no spaces and no other extensions + I need to keep backward compatibility so this is OK. May 10 at 8:39

2 Answers 2

3

Create a function and run with parallel:

_md5(){
  md5sum -b "$1" > "$1.md5"
}
export -f _md5

parallel -j8 _md5 ::: *.txt

Use "${1%.*}".md5 to replace the extension with md5 instead of appending it.

5
  • This will create 9fa74bee342222aadd999b2e8b2bef62 foo.txt instead of 9fa74bee342222aadd999b2e8b2bef62 *foo.txt but OK. Don't ask why but backward compatibility sucks. May 10 at 8:42
  • 1
    FWIW, the md5sum I have on Debian adds the * if it's told to read the file in binary mode, i.e. md5sum -b.
    – ilkkachu
    May 10 at 10:14
  • @ikkacu. You're hero, this is also stated in man md5sum: The default mode is to print a line with checksum, a space, a character indicating input mode ('*' for binary, ' ' for text or where binary is insignificant), and name for each FILE.
    – pLumo
    May 10 at 10:17
  • Added -b to md5sum in the function!
    – pLumo
    May 10 at 10:18
  • right, "[space] for text or when binary is insignificant". As in all Unixen, basically...
    – ilkkachu
    May 10 at 10:20
1

With zsh:

autoload zargs
m5() md5sum < $1 > $1:r.md5
zargs -l1 -rP8 -- **/*.txt(ND.) -- m5

Or with GNU utilities and any shell:

LC_ALL=C find . -name '*.txt' -type f -print0 |
  xargs -r0 -l1 -P8 sh -c 'exec md5sum < "$1" > "${1%.*}.md5"' sh
1
  • Cool alternatives, will need to try this and decode all cmd params when I catch some time :) May 10 at 8:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.