Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

We are writing scripts for Linux systems, there has been some debate over what would be the most universally Linux present scripting language to use. Bash, SH, Posix? What?

share|improve this question

migrated from stackoverflow.com Sep 25 '12 at 23:11

This question came from our site for professional and enthusiast programmers.

3  
I'd guess sh. –  Blender Sep 25 '12 at 22:52
    
Do you have a list of target distros? Or compulsory/desirable distros it must run on? –  TaninDirect Sep 25 '12 at 22:55
2  
"sh"? What "sh"? The Thomson shell? the Bourne shell? bash, ksh, pdksh, ash, zsh as found as /bin/sh on system X or Y? The POSIX sh specification, The SUSv3, SUSv4 sh specification, the LSB sh specification? "sh" by itself doesn't mean anything. –  Stéphane Chazelas Sep 26 '12 at 10:23
1  
if this is for software builds and install scripts, you might want to check out Autotools, which attempts to solve cross-system compilation issues. –  Lie Ryan Sep 26 '12 at 10:44
    
@sch The most portable common intersection, "obviously". In case people unfamiliar with Unix shells get misled by your comment, the POSIX spec is implemented by nearly all contemporary shells that dare to go by the name sh, and the non-conforming sh that don't are ancient and rare nowadays (e.g. Bourne). One can rattle off an ever growing list of extensions and variations but if the goal is "universal" or portability, one should be going in the opposite direction. Gilles' answer covers the details in more depth. –  jw013 Sep 26 '12 at 14:28
show 2 more comments

8 Answers 8

There are two programming environments that are available on every unix-like operating system, that are Turing-complete and that are able to call other programs: awk, and sh, the Bourne/POSIX shell family. AWK is oriented towards text processing (it complements more specialized utilities), while sh is oriented towards being a glue language to put programs together. Sh is the universal scripting language on Linux and across the unix world.

The POSIX standard defines mandatory features of sh itself and associated utilities. Most unix-like systems comply with POSIX 1003.1-2004 (a.k.a. Single Unix v3, a.k.a. the Open Group Base Specification issue 6); the latest version of that standard is POSIX 1003.1-2008 (a.k.a. Single Unix v4, a.k.a. the Open Group Base Specification Issue 7).

Every Linux and unix or Unix-like system has a Bourne-style shell at the path /bin/sh, and any non-antique system has a POSIX-compliant shell (barring the occasional bug). Every modern unix-like system (including Linux) supports shebangs, so it automatically runs scripts in /bin/sh if the first line is #!/bin/sh. There are POSIX systems where sh is located in another place (typically emulation layers on OSes you wouldn't think of as being really Unix-like).

Embedded Linux systems may have a stripped-down BusyBox system that doesn't implement all POSIX features. BusyBox has a large number of compile-time options to accommodate small-footprint systems, so it's difficult to know what to expect in advance, you have to tailor your scripts to a particular device. BusyBox is the most common small-footprint implementation of sh and assorted utilities; another one you might encounter is the extremely reduced shell environment in Android (later versions are less anemic).

Non-embedded Linux systems almost always have either dash or bash as /bin/sh. Dash is a small and fast shell that implements little more than POSIX features. Bash is a larger shell with more features.

Non-embedded Linux systems almost always have Bash installed as /bin/bash. Hence, for portability on non-embedded Linux systems, you can assume that bash is available. Among the useful additional features of bash are arrays, the ability to cope with dot files conveniently, the pipestatus variable to obtain the return status of all the commands in a pipeline, additional comparison operators for file times, and (in recent versions) regular expression matching.

One of the characteristics of shell programming is that you aren't just using the sh program, you're also using a number of utilities. Most of the file manipulation and text processing utilities on Linux are the GNU coreutils (on embedded systems, they're usually from BusyBox).

If you need portability beyond Linux, your best bet is to stick to POSIX. Other unix variants may not have bash installed (bash is part of the standard installation on OSX, but an optional package on *BSD and most commercial unices). Almost all unix variants other than Linux and OSX (i.e. *BSD and commercial unices) have some version of the Korn shell, at least pdksh. Many of bash's convenient extensions are from ksh, so it can be useful to write scripts that can run under both, but detecting where bash or ksh is located on an unknown system can be a bit of a pain.

The shell can't do everything. If you need a more sophisticated language, the two more common choices are Perl and Python (anything else is far behind as a unix scripting language). Perl is the traditional scripting language, and few non-embedded Linux systems lack it, but Python is gaining ground (boosted in part by being the recommended scripting language for Ubuntu). In the non-Linux world, Perl is part of the base installation on OSX and OpenBSD; it's optional but very commonly installed on FreeBSD, and optional but often installed on NetBSD.

share|improve this answer
1  
"boosted in part by..." That and being all-but-required in Fedora and RHEL systems. –  Ignacio Vazquez-Abrams Sep 26 '12 at 1:27
    
Mostly agree with all of this. Just some different emphases: * some distros are BusyBox-based but not necessarily embedded (I use one, Alpine). (Granted, the answer does say "almost always".) * BSDs are a large class of Unix-like systems where you can't assume bash by default. * dash can do almost everything bash can, it just sometimes demands more care. * If you need a more sophisticated language, yes Perl and Python are most common choices, but awk is even more ubiquitous and is adequate for many purposes. It's also commonly underestimated. But it's lighter-weight than Perl and Python. –  dubiousjim Sep 26 '12 at 7:33
2  
A fair warning, though -- FreeBSD dropped Perl from their default install some time ago. Aside from that, I don't know of any other distro who doesn't have Perl in their base install. –  TC1 Sep 26 '12 at 8:56
    
@TC1 Perl was always optional on NetBSD. It's in the base system on OpenBSD. –  Gilles Sep 26 '12 at 10:04
    
@dubiousjim Only Linux (non-embedded, or a good enough approximation in practice) and OSX have bash in their default installation; *BSD have pdksh or mksh, and commercial unices have ATT ksh. –  Gilles Sep 26 '12 at 10:06
show 2 more comments

In availability order:

  1. sh, but stick to POSIX-specified facilities.
  2. bash, but don't forget to explicitly specify it in the shebang or you might get dash instead.
  3. Python. Almost everyone uses it.
  4. Perl. But you get to write it.

After that, no one really cares since there isn't much you can't do with just those.

share|improve this answer
10  
I'd put PERL before python, it is installed by default in most linux systems. –  terdon Sep 25 '12 at 23:23
4  
Perl is #3. And you get to write it, bonus! :) –  Warren Young Sep 26 '12 at 1:42
1  
I agree with @ignacio perl is #4 and python is #3. The reason is obvious. I think Python is evolution of perl. –  Naai Sekar Sep 26 '12 at 7:03
4  
@ashwin: nope, python is not an evolution of, or even like, perl. perl is a language by sysadmins for sysadmins. python is a language by programmers for programmers. That difference is crucial. They both have their uses and while there can be a lot of overlap in use-cases, for some tasks perl is the better choice, and for others python is clearly superior. –  cas Sep 26 '12 at 9:56
1  
Ruby and PHP were inspired by Perl. Python is the result of a physics experiment wherein they created anti-Perls in a supercollider. (I have nothing against Python. Plusses and minuses, plusses and minuses.) –  Warren Young Sep 26 '12 at 21:00
show 1 more comment

Ordinarily, I'd say sh....but since you specified Linux, I'll say bash - it's guaranteed to be on every Linux system around (well, excluding the obscure pedantically-tiny ones that fetishize minimalism :).

If you don't have to care about non-linux portability (and if you don't need to run on tiny distros or in embedded Linux devices like plastic-box routers), then you may as well make use of the significant enhancements that it offers over plain sh. Otherwise use sh.

After bash (and sh), the next most "universal" scripting language for Linux would be some dialect of awk - usually either mawk or gawk. If you stick with plain awk and avoid gawkisms your script should run fine on almost any Linux system (it may be missing on tiny distros or embedded devices). Most Linux systems will have both mawk and gawk available, but on some distros (debian, for example) mawk is installed by default and you have to install gawk yourself if you want it.

Next would be perl. AFAIK, the base perl language is installed by default on all common Linux distros, so that makes it a good choice. Even more fortunately, there is very little version incompatibility with perl5 releases (although perl 5.12 or it might have been 5.14 finally got around to removing some obscure features that had been deprecated for about 15 years...ample warning not to use them) so unless your coding style is truly weird and you like to ignore over a decade's worth of "don't do that" warnings, your perl scripts will run just fine nearly anywhere. The language is robust and powerful and can do everything that awk and sed can do and more. With a little effort it can do the things that sh is traditionally good for too (e.g. running external commands and using/piping the output). The standard perl libraries are pretty comprehensive, too - covering more than just the basics.

The only catch with perl is that there is also an enormous library of CPAN modules to do just about anything you can think of (and lots more that may never occur to you) - and not all of them will be available on every system with perl. They're usually of very high quality so it's easy to get into the habit of just using them - but if you do make use of them, you'll have to make sure they're installed. Many CPAN modules are pre-packaged for Linux, and the remainder are easily installed with the cpan tool (or dh-make-perl on debian/ubuntu/etc to turn a CPAN module into a .deb package)

I'd like to be able to say python next, but I really can't. There's a lot to like about python but it's not included by default on many Linux systems and, frankly, it's version compatibility (both for python itself and its allegedly "standard" libs) is a complete shambles. Some distros make an excellent effort at sorting out the mess, and some don't. Neither are helped by the fact that python is basically a language written for programmers by programmers (as opposed to sysadmins) and they don't seem to think that the system their code will be installed on is at all important....their code is really super special so they don't need to concern themselves with boring stuff like integration into existing systems.

(Don't get the wrong idea from my sarcasm here - I really like python as a language, I just hate the fact that version and dependency management is a PITA. It's like going back 20+ years to the era of manually hunting for obscure bits of code and patches to get something compiled and running on your proprietary *nix)

share|improve this answer
    
I know this is an old post, but the best way to circumvent this is to specify which version in the shebang (e.g. /usr/bin/python2, /usr/bin/python3). –  impinball Jun 19 at 22:50
add comment

I'd suggest follow ksh93 or even POSIX flavour and you can always use bash/zsh to run.

And Debian-based distro use mawk not gawk as default awk. So yes avoid gawk additions since mawk is way faster.

share|improve this answer
add comment

Not bash. Write to a close-to-POSIX sh like dash or ash. That's going to be most universal. I don't think there's anything else that's even a close competitor. (And this seems to me a factual question, not an "opinion" as one of the commentors complains.)

If you need something a bit more powerful than sh (for example, if you want real associative arrays), use awk. (Avoiding the gawk extensions. There are many versions of awk, but there's a largely shared core.) That should be available almost as broadly as sh is.

share|improve this answer
1  
awk-on-Linux is universally gawk; I can't think of any distro that has another variant by default. –  Ignacio Vazquez-Abrams Sep 25 '12 at 23:04
5  
Which variants of Linux do not support bash? –  Jonathan Leffler Sep 25 '12 at 23:24
1  
Certainly bash can be installed and run on (nearly) any Linux box, but many debian users will install only dash and prefer not to install bash. –  William Pursell Sep 26 '12 at 0:50
6  
@WilliamPursell The bash package is tagged Essential on Debian (i.e. you have to go through hoops to uninstall, with ample warnings that this will hose your system). Bash is just about universal on non-embedded Linux systems. –  Gilles Sep 26 '12 at 1:05
1  
@JonathanLeffler: Embedded ones conversely may only have Busybox. –  Mechanical snail Sep 26 '12 at 1:53
show 1 more comment

I would say the availability would rank somewhere in this order:

  1. sh
  2. awk
  3. Perl (I've yet to see a *nix without it, including BSDs)

Although Python sounds like a good language, I've not used any OS that came with it in a base installation, though apparently Ubuntu does, maybe?

share|improve this answer
add comment

I believe the experts here have already provided great suggestions that should help you decide. The usability and availability of the different shells have been nicely described in the other posts.

On a different note, if I were you, I would also consider the goal that I want to achieve with the scripts. Every tool has its own merits and demerits. So, although I largely use Python, I won't use that in every case.

I can think of a few scenarios and mention some tools useful for them.

FTP files to-and-fro; process them; send email notifications; and so on

In this case it would be better to stick with the shell (e.g., Bash) and use the various utilities (e.g., ftp, cron and mail). This is a typical use case in many large companies.

Quick text processing

Once again, the shell. The utilities like grep, awk, sed, paste and others are very handy in this case.

Build

In the C/C++ domain, the universal tool for this is make. The Java world prefers Apache ANT.

Deployment and remote control

Either the shell or Python. For example, scp and rsync, respectively, would be quite useful in copying the file(s) or synchronizing the files.

Python, on the other hand, has a very useful module named fabric. This would be useful for more complex operations for e.g., copy files, stop some process, make the server up and likewise. Moreover, Python also provides a module, pip, which can solve the specified dependencies by downloading and installing the relevant packages.

(Note that the suggestions above does not focus on the shell features much, but rather on the different utilities available.)

share|improve this answer
add comment

Perl is not in FreeBSD, NetBSD or DragonflyBSD base, sorry. OpenBSD has Perl in the base installation. OS X is nowadays a certified UNIX and may be thought as a some sort of BSD variant at some level and it has Perl, Python and Ruby in base.

Many proprietary UNIXen don't have Perl in their base, for example Solaris, AFAIK neither HP-UX or IBM AIX too...

share|improve this answer
    
Your statement about Solaris missing Perl is incorrect. Perl is a mandatory core component of Solaris at least since 2005 (Solaris 10). –  jlliagre Aug 31 '13 at 19:48
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.