12

POSIX sed documentation said:

A function can be preceded by one or more '!' characters, in which case the function shall be applied if the addresses do not select the pattern space. Zero or more <blank> characters shall be accepted before the first '!' character. It is unspecified whether <blank> characters can follow a '!' character, and conforming applications shall not follow a '!' character with <blank> characters.

So, with any POSIX sed, we can:

sed -e '/pattern/!d' file

It's the same as writing:

sed -e '/pattern/!!d' file

And !!!d and n of exclamation marks are still be fine (Tested with three sed version from heirloom toolchest). I don't see any benefit between multiple instead of one exclamation.

Why did the spec allow that syntax and how is it useful in real world application?


It seems that GNU sed is not compliant in this case, it will complain if we use multiple exclamations:

$ sed -e '/pattern/!!d' file
sed: -e expression #1, char 11: multiple `!'s
  • 2
    FWIW: On OpenBSD ! acts as a toggle, /pattern/!! is the same as /pattern/, and /pattern/!!! is the same as /pattern/!. On FreeBSD multiple ! are the same as a single one. – lcd047 Jun 22 '15 at 15:33
  • 2
    The point of a lot of stuff in the spec is that sed scripts can be generated. Given a POSIX sed, it should be a realtively simple matter to to script the writing of a sed script. And so if you had some trigger for some case which should mark an address ! not worthy of whatever your action was, you might even trigger that multiple times for the same one and still come out with the same results. – mikeserv Jun 23 '15 at 1:54
  • @cuonglm No, only FreeBSD's is. The GNU, OpenBSD, and NetBSD seds aren't. – lcd047 Jun 23 '15 at 11:02
  • @lcd047: yeah, of course. Sorry for my bad English. I mean it isn't compliant, is it. It's good to know that. But the main point in my question is how that syntax can be useful in real world, with POSIX sed? – cuonglm Jun 23 '15 at 11:19
  • 1
    FWIW: a fix for this has been committed in OpenBSD-current. – lcd047 Jun 24 '15 at 3:23
5

sed's API is primitive - and this is by design. At least, it has remained primitive by design - whether it was designed primitively at inception I cannot say. In most cases the writing of a sed script which, when run, will output another sed script is a simple matter indeed. sed is very often applied in this way by macro preprocessors such as m4 and/or make.

(What follows is a highly hypothetical use case: it is a problem engineered to suit a solution. If it feels like a stretch to you, then that is probably because it is, but that doesn't necessarily make it any less valid.)


Consider the following input file:

cat <<"" >./infile
camel
cat dog camel
dog cat
switch
upper
lower

If we wanted to write a sed script which would append the word -case to the tail of each appropriate word in the above input file only if it could be found on a line in appropriate context, and we desired to do so as efficiently as possible (as should be our goal, for example, during a compile operation) then we should prefer to avoid applying /regexp/s as much as possible.

One thing we might do is pre-edit the file on our system right now, and never call sed at all during compilation. But if any of those words in the file should or should not be included based on local settings and/or compile-time options, then doing so would likely not be a desirable alternative.

Another thing we might do is process the file now against regexps. We can produce - and include in our compilation - a sed script which can apply edits according to line number - which is typically a far more efficient route in the long-run.

For example:

n=$(printf '\\\n\t')
grep -En 'camel|upper|lower' <infile |
sed "   1i${n%?}#!/usr/heirloom/bin/posix2001/sed -nf
        s/[^:]*/:&$n&!n;&!b&$n&/;s/://2;\$a${n%?}q"'
        s/ *cat/!/g;s/ *dog/!/g
        s| *\([cul][^ ]*\).*|s/.*/\1-case/p|'

...which writes output in the form of a sed script and which looks like...

#!/usr/heirloom/bin/posix2001/sed -nf
:1
    1!n;1!b1
    1s/.*/camel-case/p
:2
    2!n;2!b2
    2!!s/.*/camel-case/p
:5
    5!n;5!b5
    5s/.*/upper-case/p
:6
    6!n;6!b6
    6s/.*/lower-case/p
q

When that output is saved to an executable text file on my machine named ./bang.sed and run like ./bang.sed ./infile, the output is:

camel-case
upper-case
lower-case

Now you might ask me... Why would I want to do that? Why would I not just anchor grep's matches? Who uses camel-case anyway? And to each question I could only reply, I have no idea... because I don't. Before reading this question I had never personally noticed the multi-! parsing requirement in the spec - I think it's a pretty neat catch.

The multi-! thing did immediately make sense to me, though - much of the sed specification is geared toward simply parsed and simply generated sed scripts. You'll probably find the required \newline delimiters for [wr:bt{] make a lot more sense in that context, and if you keep that idea in mind you might make better sense of some other aspects of the spec - (such as : accepting no addresses, and q refusing to accept any more than 1).

In the example above I write out a certain form of sed script which can only ever be read once. If you look hard at it you might notice that as sed reads the edit-file it progresses from one command-block to the next - it never branches away from or completes its edit-script until it is completely through with its edit-file.

I consider that multi-! addresses might be more useful in that context than in some others, but, in honesty, I can't think of a single case in which I might have put it to very good use - and I sed a lot. I also think it noteworthy that GNU/BSD seds both fail to handle it as specified - this is probably not an aspect of the spec which is in much demand, and so if an implementation overlooks it I doubt very seriously their bugs@ box will suffer terribly as a result.

That said, failure to handle this as specified is a bug for any implementation which pretends to compliance, and so I think shooting an email to the relevant dev boxes is called-for here, and I intend to do so if you don't.

  • 1
    It's now fixed in OpenBSD-current. – lcd047 Jun 24 '15 at 3:31
  • 1
    Multiple ! is going to be removed in next spec, what's going on here! – cuonglm Jun 24 '15 at 8:27
  • @cuonglm - too little too late, i guess. maybe i was nearer to the mark than i thought. – mikeserv Jun 24 '15 at 19:04
  • @cuonglm - well, ok, but what does that... Accepted as Marked even mean? – mikeserv Jun 25 '15 at 6:50
  • 1
    @mikeserv: the answer explained my wonder and gave me another view with sed API. It make sense to me! – cuonglm Jun 25 '15 at 7:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.