3

So I have a bash script that in part does the following:

file -b --mime-type "$1"

I then have logic to check if it is the expected type of file.

if [[ $type == "application/gzip" ]]; then
    <stuff>
    exit 0
fi

# basically the else
echo "File type $type not supported"

So I built this script on a Debian system and everything with the logic works great. However, when a colleague asked if he could use it on some systems that he was working with (which are CentOS based), he keeps getting the following output in error:

File type application/x-gzip not supported

The script was intended to be portable, hence using bash and file and such. I tried to look at the /usr/share/magic on the CentOS system, which appears to be plain text. However, on Debian, it is in a binary format and is much harder to interact with.

So my understanding was that by calling ---mime-type that this would display the same mime type each time, on each operating system. I understand that I can add the application/x-gzip to the if statement but if there is a way to make the output consistent I would rather have that as an option. In addition, if I add other parts to this script in the future, or use it as a template for future work, I would prefer to have a way to get consistent output across operating systems so that I do not have to worry about many edge cases.

I am fine changing to something other than file if there is a way to get consistent answers.

Thanks in advance.

EDIT #00

I had looked at IANA initially as a reference but later realized that there are many types that IANA does not recognize. Which of course makes things worse.

As for your suggestion on version difference, I think that might be the root cause here, as the Debian systems are in and around:

redacted@redacted ~$ file -v
file-5.35
magic file from /etc/magic:/usr/share/misc/magic

while the CentOS systems are at:

redacted@redacted ~$ file -v
file-5.11
magic file from /etc/magic:/usr/share/misc/magic

As for the test from strace, it does appear (as well as being backed up above by both environments) to be reading a file at /etc/magic, and in the case of CentOS, that is in plain text. The strace looks like:

strace -e trace=stat,open,openat file -i bob.txt
open("/usr/lib64/tls/x86_64/libmagic.so.1", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat("/usr/lib64/tls/x86_64", 0x7fff0427d1d0) = -1 ENOENT (No such file or directory)
open("/usr/lib64/tls/libmagic.so.1", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat("/usr/lib64/tls", {st_mode=S_IFDIR|0555, st_size=4096, ...}) = 0
open("/usr/lib64/x86_64/libmagic.so.1", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat("/usr/lib64/x86_64", 0x7fff0427d1d0) = -1 ENOENT (No such file or directory)
open("/usr/lib64/libmagic.so.1", O_RDONLY|O_CLOEXEC) = 3
open("/usr/lib64/tls/libz.so.1", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/lib64/libz.so.1", O_RDONLY|O_CLOEXEC) = 3
open("/usr/lib64/tls/libc.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/lib64/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
open("/usr/lib/locale/locale-archive", O_RDONLY|O_CLOEXEC) = 3
stat("/root/.magic.mgc", 0x7fff0427e750) = -1 ENOENT (No such file or directory)
stat("/root/.magic", 0x7fff0427e750)    = -1 ENOENT (No such file or directory)
stat("/etc/sysconfig/64bit_strstr_via_64bit_strstr_sse2_unaligned", 0x7fff0427dcb0) = -1 ENOENT (No such file or directory)
open("/etc/magic.mgc", O_RDONLY)        = -1 ENOENT (No such file or directory)
stat("/etc/magic", {st_mode=S_IFREG|0644, st_size=111, ...}) = 0
open("/etc/magic", O_RDONLY)            = 3
open("/usr/share/misc/magic.mgc", O_RDONLY) = 3
open("/usr/lib64/gconv/gconv-modules.cache", O_RDONLY) = 3
stat("bob.txt", {st_mode=S_IFREG|0644, st_size=19, ...}) = 0
open("bob.txt", O_RDONLY)               = 3
bob.txt: text/troff; charset=us-ascii
+++ exited with 0 +++

Don't mind that the troff and .txt don't match, this was done on purpose. TROFF was chosen so it would not just be plain text...

I will see if the version of file can be updated on the CentOS boxes by my colleague and see if the results are better.

EDIT #01

So looking into another tool, xdg-mime, I found that in that program (at least), there appears to be no functional difference between x-gzip and gzip.

Source code for xdg-mime type, ll. 2427-35

It appears to be considered an alias in that program (which I was researching to see if there would be independent magic storage in order to get around this this issue). So it would appear that it is common to have either variation in magic files.

1
  • As for the trace: it reads both the plain-text and the .mgc file. As for version that can be somewhat sketchy as the magic definitions are updated by the system. My version for example is 5.32 with install date 2017, but the magic file is from 2020 and has updates such as for gzip. As for xdg-mime that would give the same "problem" as it also uses definition files which can differ between systems. When it comes to the use of x- in of itself it depends on use. If one are to match against input and then where data for that is originating or produce output and then for what etc. – ibuprofen Jul 21 at 2:42
1

IANA has application/gzip defined as of 2012.

but i.e. for HTTP one have this note:

[…] or compatibility with previous implementations of HTTP, applications SHOULD consider "x-gzip" and "x-compress" to be equivalent to "gzip" and "compress" respectively.

But that is for HTTP/1.1 and rather (very) old - and a different game all depending on what one use it for.

gzip in of itself is from the early 90s, and back in those days one also said x- could be used for unregistered sub-types.

In general both has been and is being used. If one can choose I would have used the non x- variant. But 100% consistency for a thing like this would be very hard due to the nature of the beast. Just like file changes with the times, other tools that do the same, changes with the times as well.

As for standards etc. RFC 6838 (Jan. 2013) strongly discourage the use of x- types.


If one look at the file command in of itself, the x-gzip was replaced with gzip in 2019 (from the looks of it).

Note that there is a BGZF (Blocked gzip) with x-gzip as well. That is "more" understandable as it is a variant of gzip, though conflicting with the RFC mentioned above. Then again these things take a very long time to get into place. It is used in, who knows, how many applications etc.


Only way to make sure it is consistent would be to make sure the same database is used across systems.

file uses the magic pattern file and one could of course distribute the magic.mgc with the script and say one should put it in $HOME/.magic or the like - or use the -m option in the script (use this/these magic file(s)). One could get issues if different versions of file is used - but not sure by what version etc.

The magic files are written in text then compiled into mgc.

file -C -m my-magic-file

but one can also use it as plain text. Perhaps what is happening on the CentOS system? A strace should reveal what happens, e.g.

strace -e trace=stat,open,openat file -i foo.ext

Sorry if this got very inconsistent and all over the place.

5
  • Do you know why "x-*" is used at all? – thePinochleKid Jul 19 at 23:21
  • @thePinochleKid gzip was not registered with IANA until 2012, but has been around since the early 90s. The HTTP/1.1 link above is later, but also before 2012. Also see en.wikipedia.org/wiki/Media_type#Unregistered_tree – ibuprofen Jul 19 at 23:42
  • I understand it took a while to be recognized, my question is more do you know of any reason why the "x-*" is used for some of these forms? My question is more can I potentially standardize by just using sed to remove all leading "x-*"'s or do they mean something and therefore negate that as an option – thePinochleKid Jul 19 at 23:56
  • @thePinochleKid Well problem with that is that you would merge sub types into one (xample with bgzf above) + that some x- types are used, i.e. x-shellscript which is not listed at IANA but widely used on *nix. – ibuprofen Jul 20 at 0:10
  • Email headers and mime types etc starting with X- or x- indicates that they are unofficial or "not yet standard". The tradition is that "X-" is a wild, unregulated namespace...applications can use it however they like but devs shouldn't complain if someone else uses the same label for something completely different. So application/x-gzip is old, from before it became an official recognised mime-type. application/gzip is the official standard. – cas Jul 20 at 3:15
0

Ibuprofen has already done a good job explaining how file and its magic file works, so I'll add a pragmatic "how do I make it work" answer:

Use a regex match (=~) that matches both variants instead of an exact (==) match.

if [[ $type =~ ^application/(x-)?gzip$ ]]; then
...
fi

or (if you don't mind matching anything that contains the pattern "gzip", which may be more than you want or expect - e.g. /etc/mime.types on my Debian system lists "application/tlsrpt+gzip" which seems to be for some kind of gzipped SMTP TLS report...i've never seen one in the wild, but there's a draft IETF standard for it):

if [[ $type =~ gzip ]]; then
...
fi

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.