0

I'm coding a shell script, that processes log-files. Sometimes the files may be compressed -- and have the appropriate suffix in the file name.

I'd like to add the decompression to the file-processing pipeline in such cases:

case "$file" in:
*.gz)  READER="gzip -cd" ;;
*.bz2) READER="bzip2 -cd" ;;
...
*)     READER="cat" ;;
esac
$READER < $file | processing

This works, but I do not like it, because, when the file is uncompressed, cat is invoked uselessly. I'd like processing to open the file directly in this case:

< $file processing

I tried using eval and making the | part of the $READER, but that messes up the processing pipeline as eval is applied to that as well...

Any suggestions?

4
  • You should double quote your variables. and define READER to be an array rather than a single string. e.g. *.gz) READER=(gzip -cd) ;; ..... "${READER[@]}" < "$file" | processing
    – cas
    Dec 25, 2017 at 10:28
  • Thank you, but this looks like a bashism -- my script can not rely on bash being available.
    – Mikhail T.
    Dec 25, 2017 at 15:05
  • something I've used several times in the past is to rely on the fact that lesspipe already implements all of this (and more). So if you have GNU less installed, and don't want to/can't install ucat, you could replace all of the code in your example with eval $(lesspipe) and less "$file" | processing
    – cas
    Dec 25, 2017 at 15:21
  • Unfortunately, no. Can't rely on any of the "non-standard" tools being present -- the script has to work with stock sh and not only on BSD and Linux, but on dinasaurs like HP-UX as well.
    – Mikhail T.
    Dec 25, 2017 at 17:12

1 Answer 1

-1

My suggestion is that there is not even the slightest reason to care about a Useless Use Of Cat ("UUOC").

In this case, it's not even a "useless" use - cat is serving the very useful purpose of being the default filter so that you don't have to write extra lines of code to handle a special case.

Unnecessary complexity is a far worse programming "crime" than a UUOC.

BTW, if the cat really bothers you, you can use ucat from the unp tool set and avoid re-inventing the wheel. unp doesn't seem to have a home page, but it's packaged for debian and ubuntu at least (currently maintained by debian developer Eduard Block).

Package: unp
Version: 2.0~pre7+nmu1
Installed-Size: 133
Maintainer: Eduard Bloch <[email protected]>
Architecture: all
Suggests: bzip2, unrar | unrar-free, xdms, p7zip | p7zip-full, unzip, cabextract,
orange
Description-en: unpack (almost) everything with one command
 unp is a small perl script which makes extraction of any archive files
 a bit easier. It support several compressors and archiver programs,
 chooses the right one(s) automatically and extracts one or more files
 in one go.
 .
 You may also want to install some non-free packages like "unace-nonfree",
 "unrar-nonfree" and "lha" to extract archives of these types.
7
  • By Moore's law over the last 15 years, for example, computers have become 1024 times more powerful -- faster cores, more of them, bigger caches, wider pipes. The computing experience -- not so much. Because programmers "ate" a vast share of the improvements for their own convenience and to avoid "unnecessary" complexity...
    – Mikhail T.
    Dec 25, 2017 at 15:04
  • if you're writing extra code just to avoid something trivial like a UUOC then that's unnecessary complexity. You've add complexity (and the potential for more bugs) just to avoid something that isn't a problem and doesn't consume noticeable resources (certainly less than the extra code needed to avoid running cat). Forking cat thousands of times in a loop would be bad. Forking it one extra time is completely trivial and irrelevant.
    – cas
    Dec 25, 2017 at 15:10
  • btw, my computer is many times faster than it was 15 years ago - even after all the greedy time-wasting programmer convenience code-crimes. Also BTW, I/O speeds (even PCI-e x4 SSDs and 10+gigait NICs) haven't kept up with CPU/RAM/Cache/etc improvements - they're they bottleneck. They'll probably always be the bottleneck.
    – cas
    Dec 25, 2017 at 15:13
  • Yes, your computer is much faster. Maybe, even a 100 times. But nowhere near the 1024 times... But, whatever. As is obvious, I already know, how to implement it using cat. My entire question is how to avoid the extra memory-copying involved in this method. Telling me, I should not try to avoid it, is not an answer.
    – Mikhail T.
    Dec 25, 2017 at 17:10
  • You asked for suggestions, I gave you a suggestion (in fact, I've given multiple suggestions - you don't like any of them). "There's no need to care about UUOC" IS an answer - and probably the best you'll get - even if you don't like it.
    – cas
    Dec 26, 2017 at 3:37

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .