Implement inclusion of another file into a stream (read from stdin)

Question

Background

I'm facing the following problem: I insist on using m4_include([some/file.m4]) in my configure.ac file, where some/file.m4 then contains patterns like 'AC_INIT' and 'LT_INIT' that are necessary for libtoolize to work. In otherwords, I need configure.ac to be preprocessed by m4 before libtoollize feeds it to sed looking for those patterns.

I do NOT want to generate configure.ac (if I can avoid it). The only hook that libtoolize provides that can help me is that they look for the environment variable SED and use that instead of /bin/sed when found.

I wrote a shell script that analyses sed arguments and passes them on to the real sed. Unfortunately, libtoolize concatenates m4local.ac and configure.ac and feeds that to $SED at some point, so looking for an input file named 'configure.ac' fails. Nevertheless, after analyzing the arguments passed to $SED I can just concatenate all input files and/or stdin, because in the end it has to work by reading from a stream (stdin) anyway.

Remaining problem

The problem that I am facing therefore is the following:

My script must read from stdin and write to stdout, until it encounters a line like m4_include([cwm4/configure_ac_top.m4]) which it then should swallow and instead insert the contents of cwm4/configure_ac_top.m4, no other m4 preprocessing is necessary, just the includes will do. In fact, as sed is also used for countless other little things it would be downright dangerous to do any other preprocessing than the m4_include's. However I need it to be recursive as well: if an included file contains a line m4_sinclude([lt_init.m4]) then also that file needs to be included.

I found multiple question and their answers on this site that ask how to insert one file into another, but then the file name of the to-be inserted file is known (not read from the original file), it isn't recursive and in many cases it wasn't reading from stdin (but from the first, known, file by name).

Answer 1

In the end I solved this by using m4. In order to make any 'sed' command that might be thrown at this work, m4 should not do anything except processing m4_include() macro's, although once it does that it may assume we're actually processing m4 input and doing more, like removing quotes and of course recursive inclusion etc, is ok.

So, what I do is replace all occurrences of 'm4_' in the input stream with some magic string (xyzzy), change the quotes to none by inserting a m4_changequote(,) at the beginning and pipe the result into m4. This guarantees that m4 will do absolutely nothing with the input. However, I want it to process m4_include, so before piping it into m4 I replace xyzzyinclude([\1]) with m4_changequote([,])m4_include([\1])m4_changequote(,) causing m4 to process the included file with the proper quoting. Finally, the output of m4 -P is processed by sed again, replacing the magic string back with 'm4_'.

magic="xyzzy"
cat first.txt | /bin/sed "s/m4_/$magic/g;s/$magic""include(\[\([^]]*\)\])/m4_changequote([,])m4_include(\1)m4_changequote(,)/g;1s/^/m4_changequote(,)/" | m4 -P - | sed "s/$magic/m4_/g"

Note that the restrictions on the magic string are: it may not contain (the substring) m4_, it may not begin with an underscore or with 4_ nor end on an m or m4. It also should not end on the same character as that it starts with. Finally, if it occurs in the input stream, it should not harm the purpose of the script when it is replaced with 'm4_'.

Having the following input:

A1 m4_dnl Nothing should be changed.
A2 m4___file__
m4_include([second.txt])
A4 m4___line__
A5 [` Unmatched quoting: no problem.

and

This is second.txt, an m4 file.
The wizard said xyzzy,
and nothing happened.

Using a 'magic' string of 'xyzzy', we get the result:

A1 m4_dnl Nothing should be changed.
A2 m4___file__
This is second.txt, an m4 file.
The wizard said m4_,
and nothing happened.

A4 m4___line__
A5 [` Unmatched quoting: no problem.

For the purpose of libtoolize, using a configure.ac that uses m4_include's, this is sufficient. For the full SED script, please see https://github.com/CarloWood/cwm4/blob/master/scripts/SED.sh

Edit: it turned out not to be sufficient; I also had to replace m4 as a hack with https://github.com/CarloWood/cwm4/blob/master/scripts/M4.sh