17

Does anybody know why bash still has history substitution enabled by default? My .bashrc has included set +H for many many years but some other people are still getting bitten by this feature.

Given that pretty much everybody are using terminals with copy-paste features and bash compiled with readline library and history substitution is enabled by default only in interactive shells, is there really any reason to have the feature at all? None of the existing scripts would be broken even if this was disabled by default for all shells.

Try this if you do not know why history substitution is broken:

$ set +H # disable feature history substitution
$ echo "WTF???!?!!?"
WTF???!?!!?
$ set -H # enable feature history substitution
$ echo "WTF???!?!!?"
echo WTF???echo WTF???!?!!?
WTF???echo WTF???!?!!?

(Clearly the feature has major issues if it's disabled by default for all scripting and a feature exists to verify the results before executing: shopt -s histverify.)

See also:

closed as primarily opinion-based by Jeff Schaller, Wouter Verhelst, Arkadiusz Drabczyk, Hauke Laging, chepner Apr 9 '18 at 21:59

Many good questions generate some degree of opinion based on expert experience, but answers to this question will tend to be almost entirely based on opinions, rather than facts, references, or specific expertise. If this question can be reworded to fit the rules in the help center, please edit the question.

  • 16
    You seem to think that people don't use history substitution. I use it every day. I find to quicker to follow ls -l foo/bar/baz/weeble.cpp with less !$ than recall the command and edit it. – Martin Bonner Apr 9 '18 at 12:55
  • 2
    @MartinBonner So you can enable it. The question is about why it's enabled by default, not why it still exists. – Barmar Apr 9 '18 at 15:35
  • 5
    @Barmar : Sure. But my point was to challenge the assumption that the question is predicated on. – Martin Bonner Apr 9 '18 at 15:38
  • 5
    If we follow your reasoning, anything that requires escaping or quoting should be disabled by default. Why should I need to escape or quote an URL containing &? Why should I need to escape or quote a filename containing a ?? There are UIs for people who think this is a problem. – jcaron Apr 9 '18 at 16:23
  • 3
    I use !$ several times a day, and !! quite often as well. I have to admit I don't use other history substitutions as much, but I would definitely be unhappy if the default behaviour of the shell I have been using for years suddenly changed. – jcaron Apr 9 '18 at 16:35
32

If you are already familiar with bash, then dealing with history substitution patterns is not much more likely to bite you than handling any other characters that are special to this shell. However, if one is unfamiliar with the shell or just never used its history substitution features, it will obviously be a surprise when seemingly innocuous unquoted or double-quoted strings triggers it.

In an interactive shell with history substitutions enabled, the ! character is special in pretty much the same way as the $ character is special, i.e. everywhere unless escaped with \ or in single-quoted strings.

As opposed to $ through, history substitutions do not expand in here-documents, and since they are line-oriented, they additionally will happen on lines where the substitution falls within an unquoted context or a double quoted context (in that line when scanned separately). See this bug report for more info.

History substitution is disabled in non-interactive shells (scripts) because the shell's command history capability is not needed there, not because the feature has "major issues". In a script, saving every command to $HISTFILE makes no sense, and history substitution likewise is not something you'd want to rely on in a script.

Whether or not it should be enabled by default or not in interactive shells can be debated (though I'm not entirely convinced that a debate here would matter much to the bash developers). You seem to think that most bash users are having problems with history expansions, but neither one of you and me know how common it is to use them.

Unix shells allow one to modify the shell's behaviour to fit one's personal needs and taste. If you want to turn off history substitutions for all your interactive shells, continue doing what you are doing with using set +H in your ~/.bashrc file, or lobby the bash developers to change the default (which, I believe, would upset and confuse more people than it would help).

  • 2
    @MikkoRantalainen You seem to think that history substitution is not something that majority of bash users are using. I don't think either of us knows how common it is. I'd suggest that if you feel strongly about this, you submit a feature/bug request to the bash mailing list. See savannah.gnu.org/mail/?group=bash – Kusalananda Apr 9 '18 at 12:05
  • 8
    Right. The problem is not actually about history substitution at all, but about the fact that double “quotes” in Bash aren't really quotes in the sense other languages use the term, but rather work like a special kind of parentheses. @MikkoRantalainen, make it a habit to put anything that you don't want Bash to interpret in any special way in single quotes! – leftaroundabout Apr 9 '18 at 13:39
  • 5
    @leftaroundabout It depends on what you mean by interpreting. Most languages interpret control characters in strings for output (but granted, not in the same way as in the shell when just shuffling strings around). Perl's quoting rules additionally works pretty much like the shell's (variable interpolation). As with any programming language, it helps if one is able to remember what language one is currently working in. – Kusalananda Apr 9 '18 at 14:05
  • 5
    "my point is that ! is so rarely used special character" I'd say it's the other way around... the ! is so rarely used as a normal character that – echoing WTFs aside – the number of people getting caught out is far outweighed by the number of people who still use it to substitute history. – TripeHound Apr 9 '18 at 15:46
  • 3
    -1 for deflecting the blame onto improper quoting. Since ! is not special to most Bourne-like shells, nor in bash scripts, there are many paths by which people can reasonably learn that "$my_var some text!!" will expand the $my_var but not the !!. I got familiar with Bourne-like shell on busybox ash, and my first encounter with bash history substitution was me being bitten by it when I was trying to do some quick one-liner in an interactive shell. People proficient in portable and scripting use of Bourne-like shells are often bitten by it at least once when using bash. – mtraceur Apr 9 '18 at 17:14
9

History substitution is useful. Take for example

% make-me-a-sandwich
make-me-a-sandwich: Permission denied
% sudo !!
Ok.
  • 2
    So? If you find it useful you can enable it in your .bashrc. – Barmar Apr 9 '18 at 15:37
  • 3
    xkcd.com/149 – Walter Tross Apr 9 '18 at 17:03
  • 3
    @Barmar and if you find it un-useful you can disable it. Funny how symmetry works. – hobbs Apr 9 '18 at 19:39
  • 3
    @hobbs If you don't know that it exists, how would you know to disable it? – Barmar Apr 9 '18 at 19:48
  • 2
    This answer does a good job of explaining one reason why the feature is useful. I think OP wants to understand why this usefulness is sufficient to justify this behavior, even though OP (and others, such as the people asking several of the related questions on this stackexchange) find this behavior surprising/unexpected. – mtraceur Apr 10 '18 at 16:46
4

Social/cultural inertia.

This question is in the how-humans-work problem space, so I'm going to answer from that angle, without putting forth any opinion about whether or not the feature ought to be on by default.

To start, to make sure you understand the other side, consider that the annoyance you feel about having to go out of your way to turn off the feature, is the annoyance they would feel if they had to go out of their way to turn on the feature.

Combine the above with the fact that enough bash users do use the feature, suggestions of removing it or turning it off by default are met with resistance from the people who are already comfortable with it being there by default.

Also, bash is the default shell for many people (not just in the default login or system shell sense, but in a psychological sense). If your reference frame for shell quoting is bash, if that's the shell you learned first, the fact that ! is a special shell character will feel natural and automatic to you (or at least, when you first learn it, it'll be just part of the way the shell is, just one quirk to accept among many).

And if you think about it, a lot of bash users probably encounter the history substitution syntax in a positive context: they read about it or someone shows it to them and they see the possible usefulness of it, when they're first learning bash.

It's only coming from the peripheral world of other Bourne-like shells, that you'd be bitten by ! being special and thus be inclined to view it negatively: Because if you're used to shells where you never had the feature, then your first exposure to it will be when it screws you when you're trying to get something done in a hurry.

TL;DR: Most users probably don't strongly care what the default is either way, some users like the feature and have the strong advantage of it already being that way, and there hasn't been enough people actively advocating against the feature to overcome that.

  • 2
    No, it's not only when coming from other shells. I've known this feature for years in bash and still get bitten by surprise, as single quoting doesn't always work on ! because of the broken bashimplementation. – Philippos Apr 27 '18 at 11:19
  • @Philippos That's terrifying (well, not exactly terrifying... but some sort of distressing negative qualia). Thanks for pointing this out. I will try to revisit this answer later to work your point into my answer once I've thought of a good way to integrate it. – mtraceur Apr 27 '18 at 20:32
2

Why is bash history substitution still enabled by default?

Because many people use it, and people using an interactive bash shell probably should know the rules to avoid problems and will generally find it helps more than it hurts.

My .bashrc has included set +H for many many years but some other people are still getting bitten by this feature.

So it's a feature you don't use, that doesn't mean a majority of users don't use it. You can petition for the default to be changed, but you have to look at A) the portion of people that care, and B) the ratio of people that prefer it your way. The people that care and don't like it probably already have it disabled. Changing it would help when they get an account on a new computer. People that care and do like it would have to update their settings on every computer they use and every account they get in the future.

Given that pretty much everybody are using terminals with copy-paste features

A clunkier option in my opinion...

is there really any reason to have the feature at all? None of the existing scripts would be broken even if this was disabled by default for all shells.

Yes, people find it useful. What's the use in having a remote control given that you used to be able to just get up and change the channel?

(Clearly the feature has major issues if it's disabled by default for all scripting and a feature exists to verify the results before executing: shopt -s histverify.)

History doesn't really make sense in scripts, but more importantly it could cause security issues. In your case you could avoid the problem by using single quotes. I don't remember this ever causing an issue for me, so I don't know how you can say it has 'major issues'. Has this caused an actual problem for you, or were you annoyed at having to set your defaults on a new computer?

I don't see how it is any different than having to escape or use single quotes in this if you actually want to get some money:

$ echo "Give me $50 or the cat gets it"
Give me $0 or the cat gets it
  • 2
    See the related questions (right column); most questions are hitting this feature by accident and assume that the shell is broken. I personally learned about this feature a long time ago due the fact that I accidentally used character sequence that caused wrong output. I would have been saved by shopt -s histverify but that was not on by default. In addition, figuring the cause by oneself is pretty hard because the error message is cryptic ("event not found") and the character sequence that triggers it is hard to google for. – Mikko Rantalainen Apr 10 '18 at 12:32
  • 1
    People get genuinely bitten by this feature because the "universal" shell syntax features like $ being special inside "..." are one of the first things people learn about shell, while the fact that ! is special inside "..." (but only in interactive mode) is less broadly known, typically not taught as immediately/prominently, and is bash-specific. Also, Bourne-like shells mostly have a symmetry between text-in-a-script and text-on-the-interactive-shell, so you can generally copaste from one to the other and have it work the same - and this feature creates one of the only exceptions. – mtraceur Apr 10 '18 at 16:19
  • 1
    I don't see how it is any different than having to escape or use single quotes True for zsh, when single-quoting ! always works, but not for bash, where the implementation produces unexpected results. You say, we should know the rules, well, did you know that linked rule? – Philippos Apr 27 '18 at 11:28

Not the answer you're looking for? Browse other questions tagged or ask your own question.