10

My problem (in a script with #!/bin/sh) is as follows: I try to checksum all files in a directory for archival purposes. The checksum (in my case sha1) file with all filenames should reside in the same directory. Lets say we have a directory ~/test with files f1 and f2: .

mkdir ~/test
cd ~/test
echo "hello" > f1
echo "world" > f2

Now calculating the checksums with

find -maxdepth 1 -type f -printf '%P\n' | xargs shasum

does exactly what I want, it lists all files of the current directory only and calculates the sha1 sums (maxdepth may be changed later). The output on STDOUT is:

f572d396fae9206628714fb2ce00f72e94f2258f  f1
9591818c07e900db7e1e0bc4b884c945e6a61b24  f2

Unfortunately, when trying to save this to a file with

find -maxdepth 1 -type f -printf '%P\n' | xargs shasum > sums.sha1

the resulting file displays the checksum for itself:

da39a3ee5e6b4b0d3255bfef95601890afd80709  sums.sha1
f572d396fae9206628714fb2ce00f72e94f2258f  f1
9591818c07e900db7e1e0bc4b884c945e6a61b24  f2  

and therefore fails at a later shasum --check, because of the obvious problem of additional file modification when saving the last sum.

I looked around and by using -p flag for xargs, I found out that it somehow creates the output file before even executing the find command, therefore the additional file is found and will be checksummed...

I know that as a workaround I could save the checksum to another location (temp directory via mktemp) or exclude it in find specifically, but I'd like to understand why it behaves the way it does - which is in my eyes not that useful, for example if the first command would check if the output file is already on disk, it would never get the correct answer...

  • 8
    It is not xargs, it is the shell itself that creates this file, because before any command is executed firstly shell redirects all input, output and pipes, so that when find starts the output file already exists. Use -exec instead: find -maxdepth 1 -type f -exec sh -c 'shasum "$@" > sums.sha1' {} + – jimmij Jun 29 '15 at 12:29
  • @jimmij, that's not guaranteed to work either if several sh invocations are necessary. Note that you need an argument for $0 before {}. – Stéphane Chazelas Jun 29 '15 at 12:53
  • @jimmij Your other answer that suggested tee has vanished? I tried it and it works fine, I also suppressed STDOUT with adding of 1>/dev/null. Was there something wrong with the answer or was it a bug? – user121391 Jun 29 '15 at 13:06
  • @user121391 Stephane pointed out that sometimes there can be race condition problem, what seem true. I undeleted it for a while so that you can look, but if you have many files on the list that command could go wrong. – jimmij Jun 29 '15 at 13:10
  • @jimmij ah, I see. It might be helpful if you prefixed it with a warning about the issues, because I think it is not so well known that this can happen. Otherwise, I would have accepted your answer for the cases if recurring runs include the old file and Anthon's for cases where it should be overwritten. – user121391 Jun 29 '15 at 13:17
12

You can prevent the file from reaching xargs using:

find . -maxdepth 1 -type f ! -name sums.sha1 -printf '%P\n' |
  xargs -r shasum -- > sums.sha1

To prevent problems with filename that have blanks or newlines or quotes or backslashes, I would however use:

find . -maxdepth 1 -type f ! -name sums.sha1 -printf '%P\0' |
  xargs -r0 shasum -- > sums.sha1

instead.

The -- is to avoid problems with file names that start with -. It will however not help for a file called -. Had you used -print0 instead of -printf '%P\0', you wouldn't have needed the -- and would not have had a problem with the - file.

  • Your solution is what I ended up using. I especially like that subsequent runs do not rehash the checksum file and inflat the directory. Also, in my script I used basename to get the sums.sha1 filename from the given full path (this was not included in the question, but it might help others). – user121391 Jun 29 '15 at 13:32
7

Since you're using -maxdepth 1, I assume you don't want recursion. If so, just do it in the shell instead:

for f in ~/test/*; do
    shasum -- "$f"
done > sums.sha1

To skip directories, you can do:

for f in ~/test/*; do
    [ ! -d "$f" ] && shasum -- "$f"
done > sums.sha1

If you do need recursion and are using bash, do:

shopt -s globstar
for f in ~/test/**; do
    [ ! -d "$f" ] && shasum -- "$f"
done > sums.sha1

Note that all these approaches have the benefit of working on arbitrary file names, including those with spaces, newlines or anything else.

  • I think you would mention that this solves any issues the OP would have with file names with newlines in them as well. On the other hand if the sums.sha1 is already there (from a previous run) your solution will incorporate it. – Anthon Jun 29 '15 at 12:51
  • Sorry, I did not clarify before: the maxdepth was only used in this example, I use a function where the user/script can supply any values, although currently I only need depth 1. – user121391 Jun 29 '15 at 13:01
  • @user121391 see updated answer for a recursive approach. – terdon Jun 29 '15 at 13:09
  • Note that it will also try to checksum other types of non-regular files like pipes, devices... (and symlinks to them). – Stéphane Chazelas Jun 29 '15 at 13:14
  • Thank you, personally I am using sh, but your answer might help others. – user121391 Jun 29 '15 at 13:14
4

with zsh:

shasum -- *(D.) > sums.sha1

The glob will be expanded before the redirection is made, so the sums.sha1 will not be included if it was not there in the first place.

D is to include dot-files (hidden files) as find would. . is to select only regular files (like your -type f).

To exclude the sums.sha1 anyway in case it was there in the first place:

setopt extendedglob # best in ~/.zshrc
shasum -- ^sums.sha1(D.) > sums.sha1

Note that those run one shasum command, so you may end up seeing a "Arg list too long" error if the list is huge. To work around that:

autoload zargs
zargs -e/ -- *(D.) / shasum > sums.sha1

I would recommend using ./* instead of * to avoid potential problems with a file called -.

  • I edited the question with type of shell, but your answer reminds me that I wanted to switch to zsh some time ago... ;) – user121391 Jun 29 '15 at 13:20
1

As the other answers already stated the issue is that the shell opens and creates the sums.sha1 file, before executing your pipeline. You can use the program sponge which is part of the moreutils package of many distributions. In contrast to the shell redirection sponge will wait until it received everything, before opening the file. It is generally used when you want to write a file you read in the same pipeline.

In your case it is used like this:

$ find -maxdepth 1 -type f -printf '%P\n' |xargs shasum |sponge sums.sha1
$ cat sums.sha1
31836aeaab22dc49555a97edb4c753881432e01d  B
7d157d7c000ae27db146575c08ce30df893d3a64  A
0

As an alternative to the find/xargs etc you might want sha1deep. It is probably in a different package though - on my box it comes in the md5deep package.

As others have said the sums.sha1 is created by the shell even before find starts. A trick with ! -name sums.sha1 to find will work, as will

find -maxdepth 1 -type f -printf '%P\n' | xargs shasum | grep -v ' sums\.sha1$' > sums.sha1

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.