5

So, I am trying to create an output file from a source file that includes other files.

(My use-case is actually YAML for Kubernetes/OpenShift but these .txt files show the aim.)

For example:

% cat source.txt
This is the source files it will include two other files.
The first will be here:
#INCLUDE ./first-included-file.txt
And the second file will be inserted here:
#INCLUDE ./second-included-file.txt
And this is the end of source.txt

And if the included files are:

% cat first-included-file.txt
This is FIRST
End of FIRST

% cat second-included-file.txt
This is SECOND
End of SECOND

Then the output will be:

This is the source files it will include two other files.
The first will be here:
This is FIRST
End of FIRST
And the second file will be inserted here:
This is SECOND
End of SECOND
And this is the end of source.txt

Other answers use

sed '/#INCLUDE/ r insertfile'

but is there a generic solution that can find the filename from the value in the source?

I guess a Bash script that read and parsed each line might do the job but, perhaps, awk or something else could do this?

1
  • 2
    While you could write something in bash, IMO it's a better route to use an existing templating system. For k8s manifest templating specifically, jsonnet and helm are popular choices. Also while it is not templating, kustomize is officially supported by k8s and aims to eliminate the need to template manifests.
    – jordanm
    Commented Jul 6, 2023 at 17:51

5 Answers 5

8

cpp - The C preprocessor command can be used. This command is usually included in the gcc or cpp packages in most Linux distributions.

Although it's called a C preprocessor, it can be used on other files as well, and you can use standard C preprocessor directives such as #include, #define, #ifdef and so on.

For example:

source.txt

This is the source files it will include two other files.
The first will be here:
#include "first-included-file.txt"
And the second file will be inserted here:
#include "second-included-file.txt"
And this is the end of source.txt

first-included-file.txt

This is FIRST
End of FIRST

second-included-file.txt

This is SECOND
End of SECOND

output of cpp -P source.txt

$ cpp -P source.txt
This is the source files it will include two other files.
The first will be here:
This is FIRST
End of FIRST
And the second file will be inserted here:
This is SECOND
End of SECOND
And this is the end of source.txt

notes:

  • The -P flag inhibits generation of linemarkers in the output from the preprocessor.
  • manual page
  • Regarding "you can use standard C preprocessor directives such as #include, #define, #ifdef and so on." - but if you don't want that and instead just want files included as-is then in addition to converting each #INCLUDE to #include and wrapping your included file names in quotes, you must do something to disable any occurrences of the strings #include, #define, #ifdef, etc. that might just happen to be in your input before running cpp otherwise it'll undesirably transform your text based on those strings appearing.
  • You may also get unexpected results if the file to be included is missing from the local directory (assuming your implementation of cpp does search the local directory first, as is usually the case) but cpp can find a file of the same name in one of the other implementation-defined directories it searches for include files.
1
  • 2
    @EdMorton Thanks for the information. Would you mind editing this answer?
    – memchr
    Commented Jul 7, 2023 at 12:34
1

Using Raku (formerly known as Perl_6)

Both Perl and Raku have a quoting construct known as heredocs (i.e. quote "doc :to here"), which can provide the desired output:

source.txt

my $file1 = '/path/to/first-included-file.txt'.IO.lines.join("\n");
my $file2 = '/path/to/second-included-file.txt'.IO.lines.join("\n");

my $pre-processed = qs:to/TERMINATOR/;
This is the source files it will include two other files.
The first will be here:
$file1
And the second file will be inserted here:
$file2
And this is the end of source.txt
TERMINATOR

put $pre-processed;

first-included-file.txt

This is FIRST
End of FIRST

second-included-file.txt

This is SECOND
End of SECOND

output of raku source.txt

This is the source files it will include two other files.
The first will be here:
This is FIRST
End of FIRST
And the second file will be inserted here:
This is SECOND
End of SECOND
And this is the end of source.txt

You don't need to learn the entirety of the Raku language to only use this quoting language, known as "Q-lang" (or slang) in Raku lingo. Above, the minimal interpolation is invoked using the qs:to/TERMINATOR/; code. Here, qs tells Raku to only interpolate "scalar" or $-sigiled variables (using qq instead will interpolate all Raku variables). Other (intermediate) interpolation options can be found below.

https://docs.raku.org/syntax/heredocs%20%3Ato
https://docs.raku.org/language/quoting
https://stackoverflow.com/a/76624379/7270649
https://raku.org

1

With GNU sed:

sed 's/^#INCLUDE /cat /e' source.txt

The e flag will execute the contents of the pattern space as a shell command and replace it with the output.

1
  • 2
    It's worth mentioning that that'll only work at 1 level so if source.txt contains #INCLUDE foo and foo contains #INCLUDE bar, bar won't be expanded in the output, it'll still show #INCLUDE bar.
    – Ed Morton
    Commented Jul 15, 2023 at 19:48
0

Tweaking "d) Recursive-descent parsing of an input file or files, e.g.:" in http://awk.freeshell.org/AllAboutGetline to match on #INCLUDE instead of include and work even if your file names include non-newline white spaces you could do the following with any awk:

awk '
    function read(file) {
        while ( (getline < file) > 0) {
            if ( sub(/^#INCLUDE[ \t]+/,"") ) {
                read($0)
            } else {
                print
            }
        }
        close(file)
     }
     BEGIN {
         read(ARGV[1])
     }
' source.txt
This is the source files it will include two other files.
The first will be here:
This is FIRST
End of FIRST
And the second file will be inserted here:
This is SECOND
End of SECOND
And this is the end of source.txt

The above assumes you don't have recursive includes - if you do then add whatever logic is necessary to implement whatever you want that recursion to mean.

0

The "m4" macro (name starts with 'm' and has 4 following letters) program will do includes. HOWEVER, the form of the include is: "include (`file')". See "info m4".

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .