1

Trying to calculate md5 hash of all *.txt files in directory and store each in same filename but with extension replaced from .txt to .md5.

For file: foo.txt i want to create file: foo.md5 with content:

9fa74bee342222aadd999b2e8b2bef62 *foo.txt

I have managed to get needed files with command below, but I wonder is there some much shorter alternative, or at least a way to drop one or two awk commands?

time find *.txt | parallel md5sum | awk '{ print $1, $2, $2 }' | awk '{$1 $2 $3 > gsub(".txt", ".md5", $3); print}' | awk '{ print $1, "*"$2 > $3 }'

I have found some alternatives, but they don't use parallel and the just add .md5 to filename, not replace it. Also they store only md5 hash, I need this *filename modification.

2
  • 2
    Your script will fail on spaces or newlines in filename. And if there are same filenames with only different extension (okay, you only use .txt, I would still go with file.ext.md5 instead of file.md5)
    – pLumo
    May 9, 2022 at 13:32
  • You are correct, script is not for general purpose. But this files are machine generated so I know there are no spaces and no other extensions + I need to keep backward compatibility so this is OK. May 10, 2022 at 8:39

2 Answers 2

3

Create a function and run with parallel:

_md5(){
  md5sum -b "$1" > "$1.md5"
}
export -f _md5

parallel -j8 _md5 ::: *.txt

Use "${1%.*}".md5 to replace the extension with md5 instead of appending it.

5
  • This will create 9fa74bee342222aadd999b2e8b2bef62 foo.txt instead of 9fa74bee342222aadd999b2e8b2bef62 *foo.txt but OK. Don't ask why but backward compatibility sucks. May 10, 2022 at 8:42
  • 1
    FWIW, the md5sum I have on Debian adds the * if it's told to read the file in binary mode, i.e. md5sum -b.
    – ilkkachu
    May 10, 2022 at 10:14
  • @ikkacu. You're hero, this is also stated in man md5sum: The default mode is to print a line with checksum, a space, a character indicating input mode ('*' for binary, ' ' for text or where binary is insignificant), and name for each FILE.
    – pLumo
    May 10, 2022 at 10:17
  • Added -b to md5sum in the function!
    – pLumo
    May 10, 2022 at 10:18
  • right, "[space] for text or when binary is insignificant". As in all Unixen, basically...
    – ilkkachu
    May 10, 2022 at 10:20
1

With zsh:

autoload zargs
m5() md5sum < $1 > $1:r.md5
zargs -l1 -rP8 -- **/*.txt(ND.) -- m5

Or with GNU utilities and any shell:

LC_ALL=C find . -name '*.txt' -type f -print0 |
  xargs -r0 -l1 -P8 sh -c 'exec md5sum < "$1" > "${1%.*}.md5"' sh
1
  • Cool alternatives, will need to try this and decode all cmd params when I catch some time :) May 10, 2022 at 8:47

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .