2

I have a directory full of files. Each file will be copied to a specific type of destination host.

I want to calculate an MD5 sum for each file in the directory, and store that md5 sum in a file that matches the name of the file that generated the sum, but with .md5 appended.

So, for instance, if I have a directory with:

a.bin
b.bin
c.bin

The final result should be:

a.bin
a.bin.md5     # a.bin's calculated checksum
b.bin
b.bin.md5     # b.bin's calculated checksum
c.bin
c.bin.md5     # c.bin's calculated checksum

I have attempted this with find exec, and with xargs.

With find, I tried this command:

find . -type f -exec md5sum {} + > {}.md5

Using xargs, I tried this command:

find . -type f | xargs -I {} md5sum {} > {}.md5

In either case, I end up with a file called {}.txt, which isn't really what I am looking for.

Could anyone point out how to tweak these to generate the md5 files I am looking to generate?

  • 1
    Dumping all the output to a single file will let you use md5sum -c later to verify all files at once. – Ignacio Vazquez-Abrams Nov 6 '17 at 20:21
  • The problem is we won't be copying all files to the destinations. Only one file will be copied. So, we'd end up with a monolithic file listing files that mostly aren't present on the destination hosts. Using the example above, some hosts get a.bin, some get b.bin. The a.bin hosts don't need md5sum info for b.bin and c.bin. I hope that makes sense. :) – Matthew Nov 6 '17 at 20:38
2
cd /path/to/files &&
for file in *; do
    if [[ -f "$file" ]]; then
        md5sum -- "$file" > "${file}.md5"
    fi
done
3

Redirection is an operator of the shell.

In:

find some args  > {}.md5

find is run with some args and its stdout redirected to {}.md5.

Here, you'd need find to start a shell to perform one redirection for each file:

find . -type f -exec sh -c '
  for file do
    md5sum "$file" > "$file.md5"
  done' sh {} +

If you want to avoid the directory component in the .md5 part, you can either use the -execdir alternatives supported by many find implementations (though with some you'll still get a ./ directory component, and with some, that will still run one shell per file even with {} +:

find . -type f -execdir sh -c '
  for file do
    md5sum "$file" > "$file.md5"
  done' sh {} +

Or you can do it as:

find . -type f -exec sh -c '
  for file do
    (cd "${file%/*" && exec md5sum -- "${file##*/}") > "$file.md5"
  done' sh {} +
0

I know there are side effects to piping into a while loop, but I like the simplicilty, readability and extensibilty of this solution

find . -type f ! -name \*.md5 |\
  while read a; do md5sum $a > $a.md5; done

To use this scipplet to check sums simply:

find . -type f ! -name \*.md5 |\
  while read a; do echo $a; md5sum -c $a.md5; done
-1
find . -name '*.iso' -type f -exec sh -c "ls -lh {}; pv {} | md5sum > {}.md5" \;

optional

ls -lh {};

to show processing file, and

pv {}

to show progressbar on big (slow) iso file in my case

  • Looks OK, but you're restricting it to .iso files. – telcoM Jan 20 '18 at 17:50
-1
cd /path/to/files && ls -1| xargs md5sum |  xargs -l bash -c 'echo $0"  "$1 > $1.md5'

first xargs generate , second xargs picks up and save to .md5

If you need filter file patter, you can replace ls with find or add grep afterwards.

  • Parses output of ls and uses unquoted variable expansions. – Kusalananda Feb 1 '18 at 17:43
  • Can you explain your comments? you can replace ls with find or any other command. – Ming Jia Feb 1 '18 at 19:12
  • Assume spaces in any of the names in the files directory, for example. Also consider what would happen if one of the names is a directory. – Kusalananda Feb 2 '18 at 6:33
  • @MingJia Parsing output of ls is considered generally bad practice (see this ). Even if there aren't any difficult filenames, that doesn't mean one shouldn't anticipate them. Second big problem is the $0 and $1 should be double-quoted. Because of "word splitting", if you have file with spaces or tabs in filename, ` > $1` will have ambiguous redirect error. Try var="with space.txt"; echo "hi" > $var and then var="with space.txt"; echo "hi" > "$var" – Sergiy Kolodyazhnyy Feb 2 '18 at 6:55
  • @MingJia Don't get discouraged, you'll learn to pick up on these things if you stick around on the site, with time of course. Welcome, to the site and consider re-working your answer. – Sergiy Kolodyazhnyy Feb 2 '18 at 6:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.