2

I create a .tgz/tarball with $(npm pack).

I then run:

sha1sum oresoftware-npp-0.0.1001.tgz 

and I get:

77c58da68593dcdcd14bb16a37f5f63ef42bab63  oresoftware-npp-0.0.1001.tgz

I want to compare that shasum against another tarball on a remote server. I can query for a shasum for a tarball on the NPM registry, with:

 npm view @oresoftware/npp@latest dist.shasum

which yields:

3c2e7328110ba57e530c9938708b35bde941c419

this shasum is different than the other one above, but that's expected, since I changed the contents of the .tgz tarball file.

my question is 3 fold:

  1. When I generated a sha1sum of the .tgz file resulting from npm pack, is that the right way to do it? To generate the sha1sum after the tar file is created?

  2. I assume that the the sha1sum would be identical if the tarballs had identical contents? Would they differ if the files were created/modified at different times even if they have otherwise the same contents?

  3. Is there a better way to check if two tarballs have the same contents? That's all I am trying to do.

  • A tar archive contains information about ownership and timestamps on files, so to compare two archives generated on two separate machines would likely not work. You would be better off extracting the content of the two archives and comparing that. I'm not writing an answer because I don't know, off the top of my head, of a simple way of doing that. – Kusalananda Aug 4 '18 at 18:41
  • For comparing two archives, there is no software sulution. Since 25 years, star however supports to compare an archive with the filesystem and allows to configure what to compare while doing that. – schily Aug 4 '18 at 21:39
  • @schily what about this: tar -xOzf foo.tgz | sort | sha1sum – Alexander Mills Aug 5 '18 at 2:00
  • Since -O usually tells tar to create archives compatible to the 1977 format, you seem to seem to use a different program. Let me assume that you have GNU tar in mind. With that tar clone and that command, the content of the files is extracted to stdout. If you did omit sort, you would get a definite value to compare tar archives with the same order of files inside. With the sort call, the result does not help you as you could get the same final result even in case there are differences. – schily Aug 5 '18 at 8:20
2

The checksums available from the NPM registry provide two features: they allow you to verify that your download hasn’t been corrupted, and if you can verify the checksums out of band, that the downloaded files haven’t been altered. Unless NPM archives are built reproducibly, the checksums don’t allow you to verify that an archive you build yourself using npm pack contains what it’s supposed to.

The issue with tarballs is that they contain metadata: the ownership, permissions and timestamps of the stored files, as stored by tar, and on top of that, compression metadata. If values for all of these are pre-agreed, they can be specified to override the values obtained from the file system, but that requires pre-agreement.

To compare the contents of two arbitrary tarballs, the only reliable way is to extract their contents and compare that.

  • 1
    Thanks Stephen, so yes say I go through all the files in the tarball and compare them - name of file and size of file, can you maybe explain a good place to start with that? I don't care about file modification date, but I do care about file permissions. I figure if the file count across the tarballs is the same, the file names are the same, and all the file sizes are the same, then probably good enough. – Alexander Mills Aug 4 '18 at 19:10
  • 1
    I reckon you should also compare the files’ contents; I wouldn’t consider comparing file names and sizes as sufficient. – Stephen Kitt Aug 4 '18 at 21:00
  • In my testing, the shasum thing seems to work. I publish the tarball to NPM, then I retrieve the shasum from NPM. Then I copied the files with rsync to another local folder, and then generated a new tarball with npm pack and got the shasum, and it was the same as the one from the remote registry. Not sure how meaningful that is, but maybe the shasum works in this case. I have to read more about how sha1sum works. – Alexander Mills Aug 4 '18 at 22:05
  • As an example, if you have a .tgz file in a Git repo, Git can tell you if that .tgz file changed, I just am not sure how exactly. – Alexander Mills Aug 4 '18 at 22:07
  • something like this perhaps: stackoverflow.com/questions/51689992/… – Alexander Mills Aug 4 '18 at 22:28
0

Whether this checksum comparison is useful for you depends on what tar command you are using and that tar archive type is used.

star e.g. includes all three time stamps since 1986 and this causes tar archives to differ even with the same content, as the last acess time differs.

With the new POSIX tar enhancements from 2001, there are also all three time stamps - depending on whether the tar implementation implements the original standard or the later change.

If you don't care about time stamps, comparing archive checksums is not the right way to go.

Comparing two tar archives and making statements from that is thus usually impossible.

If you however may unpack one of the two archives, there is a nice way to compare content and meta data with a configurable set of meta data that is used for the comparison. Use star -diff, see http://schilytools.sourceforge.net/man/man1/star.1.html

If you e.g. call:

star -diff -v diffopts=!times < archive.tar.gz

the file content and all meta data except for all time stamps is compared.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.