2

In AIX system (v 7.1) sha1sum is calculating different hash codes when its piped directly to the output of tar compared to when it reads a file.

What are the reasons for this ? Are there ways to workaround this and get the hash code directly from the tar piped output ?

(In others systems like Debian and Ubuntu, piped sha1sum to tar output works well)

folder=myfolder
tarfile=myfolder.tar

tar -cf $tarfile $folder && sha1sum $tarfile
fe2dcba2b25d4bbb35460309c8bb87a1d2514d7d  myfolder.tar

tar -cf $tarfile $folder && sha1sum $tarfile
fe2dcba2b25d4bbb35460309c8bb87a1d2514d7d  myfolder.tar

tar -cf - $folder > $tarfile && sha1sum $tarfile
fe2dcba2b25d4bbb35460309c8bb87a1d2514d7d  myfolder.tar

tar -cf - $folder > $tarfile && sha1sum $tarfile
fe2dcba2b25d4bbb35460309c8bb87a1d2514d7d  myfolder.tar

tar -cf - $folder | sha1sum
f1dd1a0c4e82dd5c441664869b656c7bce799270  -

tar -cf - $folder | sha1sum
f1dd1a0c4e82dd5c441664869b656c7bce799270  -
6
  • Test with sha1sum again but on a really tiny file (tarred) to make sure it's still behaving in the same way. Then replace sha1sum with od ("octal dump") and compare visually to figure out what the differences are. I believe that there's also a sha1 utility on AIX that you can try in place of GNU coreutil's sha1sum.
    – Kusalananda
    Aug 16, 2017 at 19:15
  • @Kusalananda: The issue looks to be with a sort of flushing of tar output. I tried these: tar -cf - smallfile | wc -c vs tar -cf - smallfile > tarfile && wc -c tarfile, and I got respectively: 4096 vs 10240. Where smallfile has a "hi" without newline. (btw, about 10K minimum output to tar only 2 bytes)
    – Luciano
    Aug 16, 2017 at 19:36
  • Try with another tar implementation, for example gtar (the one from the GNU people), if you have access to it.
    – Kusalananda
    Aug 16, 2017 at 19:41
  • appears that tar writes extra nulls when writing to a file vs stdout
    – Jeff Schaller
    Aug 16, 2017 at 19:43
  • 1
    look into the -b parameter; seems to affect the output here
    – Jeff Schaller
    Aug 16, 2017 at 19:47

1 Answer 1

4

The reason for that problem is the command tar. It has internal records made of a fixed number of 512 bytes blocks. The number of blocks per record can be set with the parameter -b.

Some implementations can adjust the amount of blocks automatically according to the file descriptor, if its a tape device, a regular file, or a pipe.

Fixing the amount of blocks with the -b parameter fixed the problem. Like as: tar -b1 -cf - $folder | sha1sum. But to match the default blocks predefined in the first two commands of the question I had to use -b20 (10240 byte records is the default for archives stored in regular files):

$ tar -b20 -cf - $folder > $tarfile && sha1sum $tarfile
fe2dcba2b25d4bbb35460309c8bb87a1d2514d7d  myfolder.tar

$ tar -b20 -cf - $folder |sha1sum                      
fe2dcba2b25d4bbb35460309c8bb87a1d2514d7d 

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .