5

I have a file where I want to replace some variables with data from a shell script.

-A INPUT -i lo -s @LOCAL_IP@ -j ACCEPT

Here I want to replace @LOCAL_IP@ with an IP address, I use the following:

... | sed -e 's/@LOCAL_IP@/192.168.1.1/' | ...

I may have 192.168.1.1 in a variable in my shell script, but this is to give the basic idea.

Now, I have another rule that looks like this:

-A INPUT -i @PUBLIC_INTERFACE@ -p tcp -m tcp --dport 22
            -d @PUBLIC_IP@ -s @ADMIN_IPS@ --syn -j ACCEPT

(this is one long line, broken up here for readability)

The real input file would be many lines like that in the input file, most without the @ADMIN_IPS@, for example:

-A INPUT -i @PUBLIC_INTERFACE@ -p tcp -m tcp --dport 80 -d @PUBLIC_IP@ --syn -j ACCEPT
-A INPUT -i @PUBLIC_INTERFACE@ -p tcp -m tcp --dport 22 -d @PUBLIC_IP@ -s @ADMIN_IPS@ --syn -j ACCEPT
-A INPUT -i @PUBLIC_INTERFACE@ -p tcp -m tcp --dport 443 -d @PUBLIC_IP@ --syn -j ACCEPT

In this case, I can easily replace @PUBLIC_INTERFACE@ and @PUBLIC_IP@. This is just the same as the line before. However, @ADMIN_IPS@ is a list of 2 or 3 IP addresses. What the result should be is something like this:

-A INPUT -i eth0 -p tcp -m tcp --dport 80 -d 8.8.8.8 --syn -j ACCEPT
-A INPUT -i eth0 -p tcp -m tcp --dport 22 -d 8.8.8.8 -s 8.8.10.1 --syn -j ACCEPT
-A INPUT -i eth0 -p tcp -m tcp --dport 22 -d 8.8.8.8 -s 8.8.10.2 --syn -j ACCEPT
-A INPUT -i eth0 -p tcp -m tcp --dport 22 -d 8.8.8.8 -s 8.8.10.3 --syn -j ACCEPT
-A INPUT -i eth0 -p tcp -m tcp --dport 80 -d 8.8.8.8 --syn -j ACCEPT

Is there a relatively easy way to transform a single line into multiple lines with a tool such as sed in Linux? (Ubuntu 16.04) In this case, the three ADMIN_IPS addresses would be defined in a variable as in:

ADMIN_IPS=8.8.10.1,8.8.10.2,8.8.10.3

It could be space separated too or even an array.

  • 1
    If this is your only use-case, then the solutions suggested here should work fine. If you find yourself using this approach in many config. files, may I suggest looking at a configuration management tool such as Puppet or Chef? They live to handle cases like this (among other things) – Joseph R. Jul 6 '16 at 20:28
  • 1
    as well as puppet/chef/ansible/etc, when your templating requirements grow beyond simple replacement of single values, I strongly recommend you use one of the exitsting templating tools (like perl's Text::Template or module or perl's Template Toolkit) instead of re-inventing the wheel. Even just looking at the docs for such tools will give you ideas on how to implement things that you thought were impossible or too hard, or that had never occurred to you before. – cas Jul 7 '16 at 7:30
3

Here's an ugly awk solution; someone else may have a smarter way to do it:

awk -v pi=1.2.3.4 -v pint=eth0 -v pip=8.8.8 -vaips="8.8.10.1 8.8.10.2 8.8.10.3"  \
  'BEGIN{
        split(aips, array)
   }
   {
        gsub(/@PUBLIC_INTERFACE@/, pint);
        gsub(/@PUBLIC_IP@/, pi);
        if (/@ADMIN_IPS@/) {
                copy=$0
                for (i in array)  {
                  gsub(/@ADMIN_IPS@/, array[i], copy)
                  print copy
                  copy=$0
                }
        } else {
          print;
        }
   }' input

This passes the variables into awk (using space-separation for the admin IPs); before any input is read, awk will split that passed-in "array" into a bona fide awk array named array. For each line of input, awk will then globally search & replace the various singleton variables, then if @ADMIN_IPS@ is present on the line, go through a loop over the admin IP's and print out copies of the input line with the corresponding admin IP's replaced.

  • Okay, that looks good as it makes it possible to pipe everything (i.e. I currently have two variables with multiple IPs, so I can have two awk scripts in my pipes to do the job). – Alexis Wilke Jul 6 '16 at 20:37
  • Yeah, I think awk is a better tool for this job. For the record, you definitely can do this with sed but it's not "an easy way"... Basically, you have to prepare a sed script so e.g. assuming you had those values in an array named adminips then something like { printf '%s\n' '/@ADMIN_IPS@/{h';c=1; until [ "$c" -gt "${#adminips[@]}" ]; do printf '%s\n' "s/@ADMIN_IPS@/${adminips[c]}/gp;g";((c++));done;printf '%s\n' 'd;}'; } | sed -f- infile . Note I use zsh (array indexing starts from 1), if you're using bash then set c=0 before the until loop and use -eq instead of -gt. – don_crissti Jul 6 '16 at 23:49
1

Please try:::::

given::
echo $ADMIN_IPS 

8.8.10.1,8.8.10.2,8.8.10.3

and Input file /tmp/myvar given with contents as below::::

-A INPUT -i @PUBLIC_INTERFACE@ -p tcp -m tcp --dport 80 -d @PUBLIC_IP@ --syn -j ACCEPT

-A INPUT -i @PUBLIC_INTERFACE@ -p tcp -m tcp --dport 22 -d @PUBLIC_IP@ -s @ADMIN_IPS@ --syn -j ACCEPT

-A INPUT -i @PUBLIC_INTERFACE@ -p tcp -m tcp --dport 443 -d @PUBLIC_IP@ --syn -j ACCEPT

while read j; do savedline=$(echo "$j"|sed 's/@PUBLIC_INTERFACE@/eth0/'|sed 's/@PUBLIC_IP@/1.1.1.1/'); for i in $(echo $ADMIN_IPS |tr ',' '\n'); do echo $savedline|sed "s/@ADMIN_IPS@/$i/";done|uniq;done < /tmp/myvar

Try and let us know.

  • That will work as expected for an input file with just that one line, but my file would have many lines, some of which would need to have their @ADMIN_IPS@ entry updated and the line repeated... – Alexis Wilke Jul 6 '16 at 20:02
  • modified my script line, to take care of a whole file of many lines. – mkzia Jul 6 '16 at 22:13
  • sort is definitely not possible. Firewall rules have to be kept in order they were defined. Also I think that your loop would duplicate each line, not just those that include @ADMIN_IPS@. – Alexis Wilke Jul 6 '16 at 22:45
  • fixed it now, please try Sir! – mkzia Jul 6 '16 at 23:33
1

Here's a perl solution using Text::Template. The script reads the template from a string variable ($tstr), performs all the replacements (including some embedded perl code to loop over the @ADMIN_IPS array, and then prints out the result:

(on a debian system, this requires the libtext-template-perl package to be installed)

#! /usr/bin/perl

use strict;
use Text::Template;

# The template, in a string ($tstr):
my $tstr='-A INPUT -i {$PUBLIC_INTERFACE} -p tcp -m tcp --dport 80 -d {$PUBLIC_IP} --syn -j ACCEPT

{
  foreach $a (@ADMIN_IPS) {
    $OUT .= "-A INPUT -i $PUBLIC_INTERFACE -p tcp -m tcp --dport 22 -d $PUBLIC_IP -s $a --syn -j ACCEPT\n";
  }
}
-A INPUT -i {$PUBLIC_INTERFACE} -p tcp -m tcp --dport 443 -d {$PUBLIC_IP} --syn -j ACCEPT
';

# create the Text::Template object ($tt)
my $tt = Text::Template->new(TYPE => 'STRING', SOURCE => $tstr);

# define a hash reference to hold all the replacements:
my $vars = { PUBLIC_INTERFACE => 'eth0',
             PUBLIC_IP => '8.8.8.8',
             ADMIN_IPS => [ '8.8.10.1', '8.8.10.2', '8.8.10.3' ],
           };

# fill in the template
my $text = $tt->fill_in(HASH => $vars);

print $text;

Output:

-A INPUT -i eth0 -p tcp -m tcp --dport 80 -d 8.8.8.8 --syn -j ACCEPT

-A INPUT -i eth0 -p tcp -m tcp --dport 22 -d 8.8.8.8 -s 8.8.10.1 --syn -j ACCEPT
-A INPUT -i eth0 -p tcp -m tcp --dport 22 -d 8.8.8.8 -s 8.8.10.2 --syn -j ACCEPT
-A INPUT -i eth0 -p tcp -m tcp --dport 22 -d 8.8.8.8 -s 8.8.10.3 --syn -j ACCEPT

-A INPUT -i eth0 -p tcp -m tcp --dport 443 -d 8.8.8.8 --syn -j ACCEPT

For most light templating like this, scalar (single-value) variables and the occasional list (aka array) or hash (aka associative array) is all you need.

The above script can be adapted endlessly for all sorts of similar jobs - just change the template and the $vars hash reference. And it's not at all difficult to, e.g., load the template from one file and the variables (scalars and arrays) from another, so you could have a tiny re-usable script that takes two file arguments (template and vars).

Apart from the template itself and the $vars setup there's only about 5 lines of code. Maybe 6 or 7, depending on how you count the for loop in the template.

Templates can be read from a filename, an array of lines, an already opened file-handle (incl. stdin) or a string as in this example.

You can do calculations, table lookups, database queries (e.g. with the DBI module), extraction of data from a CSV file, fetch and process the output of subroutines and external programs, and more within a template. Anything you can do with perl.

For simple scalar variables, all you need to do is embed the variable name in the template inside curly brackets (e.g. with variable $foo, use {$foo}). For arrays and hashes and calculations etc you'd need to embed some perl code in the template.


Here's a version that reads in a template filename and a config variable filename from the first two args on the command line:

(on a debian system, this requires the libconfig-simple-perl and libtext-template-perl packages to be installed)

#! /usr/bin/perl

use strict;
use Text::Template;
use Config::Simple;

# create the Text::Template object ($tt)
my $tt = Text::Template->new(TYPE => 'FILE', SOURCE => $ARGV[0]);

# read the config file into the `%vars` hash.    
my $cfg = new Config::Simple();
$cfg->read($ARGV[1]);
my %vars = $cfg->vars();

# strip "default." from key names.
%vars = map { s/^default\.//r => $vars{$_} } keys(%vars);

# fill in the template
my $text = $tt->fill_in(HASH => \%vars);

print $text;

NOTE: the script needs to strip default. from the beginning of each hash key name because the config file is very much like a .INI file and can have [sections] just like them. Any config variables not in a section are presumed to be in the default section. Writing the template with variables like {$default.PUBLIC_INTERFACE} would be tedious, so the solution is to fix the keys of the %vars hash.

BTW, these .INI-like [sections] aren't necessarily a problem. It is possible to make good use of them in a template. But the default. prefix is just pointless and annoying when used with Text::Template.

Anyway, with this template file:

$ cat iptables.tpl 
-A INPUT -i {$PUBLIC_INTERFACE} -p tcp -m tcp --dport 80 -d {$PUBLIC_IP} --syn -j ACCEPT
{
  my @o=();
  foreach $a (@ADMIN_IPS) {
    push @o, "-A INPUT -i $PUBLIC_INTERFACE -p tcp -m tcp --dport 22 -d $PUBLIC_IP -s $a --syn -j ACCEPT";
    $OUT .= join("\n",@o);
  }
}
-A INPUT -i {$PUBLIC_INTERFACE} -p tcp -m tcp --dport 443 -d {$PUBLIC_IP} --syn -j ACCEPT

NOTE: this template is slightly improved as it uses an array (@o) and join() to avoid adding unwanted extra newlines - did you notice how I "cheated" in $tstr in the first version by adding one more newline so that the array output lines were in a separate paragraph?

and this file containing the variables:

$ cat iptables.var 
PUBLIC_INTERFACE=eth0
PUBLIC_IP=8.8.8.8
ADMIN_IPS=8.8.10.1, 8.8.10.2, 8.8.10.3

That file would work exactly the same if it had [default] inserted as the first line.

Also, unlike most forms of .INI files, this one has a very easy way of defining array variables: just separate them with a comma, with optional extra whitespace, which is exactly what you asked for in your question.

BTW, whitespace is ignored in the variable definitions unless you put it in quotes. See man Config::Simple for more details about the config file(s).

Run it like this:

$ ./alexis.pl iptables.tpl iptables.var 
-A INPUT -i eth0 -p tcp -m tcp --dport 80 -d 8.8.8.8 --syn -j ACCEPT
-A INPUT -i eth0 -p tcp -m tcp --dport 22 -d 8.8.8.8 -s 8.8.10.1 --syn -j ACCEPT
-A INPUT -i eth0 -p tcp -m tcp --dport 22 -d 8.8.8.8 -s 8.8.10.2 --syn -j ACCEPT
-A INPUT -i eth0 -p tcp -m tcp --dport 22 -d 8.8.8.8 -s 8.8.10.3 --syn -j ACCEPT
-A INPUT -i eth0 -p tcp -m tcp --dport 443 -d 8.8.8.8 --syn -j ACCEPT

It's intentionally minimalist (i.e. very quick and dirty, e.g. doesn't even try to validate the existence of the files) and there are many ways it could be improved, but it's a fully-functional example of how to do the basic job.

  • btw, like perl, python also has a large selection of templating modules – cas Jul 7 '16 at 13:01
  • doh. dunno what i was thinking. chomp $OUT; (to trim the trailing newline) immediately after the end of the foreach block would have been much simpler than using @o and join(). – cas Jul 8 '16 at 12:54
0

If you have a chance to attach all the line for each ip, I can suggest the following

#!/bin/bash

first="-A INPUT -i eth0 -p tcp -m tcp --dport 22 -d 8.8.8.8 -s 10.2.2.1 --syn -j ACCEPT"
second="-A INPUT -i eth0 -p tcp -m tcp --dport 22 -d 8.8.8.8 -s 10.2.2.2 --syn -j ACCEPT"
third="-A INPUT -i eth0 -p tcp -m tcp --dport 22 -d 8.8.8.8 -s 10.2.2.3 --syn -j ACCEPT"

sed 's/.*@ADMIN_IPS@.*/${first}\n${second}\n${third}/g' file

It may not be the most clean, but can be a solution

give a try

  • I do not see how that would work. In the output, I need that one line repeated N times (N being the number of items int he ADMIN_IPS array, could be zero, although we could have a special case for that). – Alexis Wilke Jul 6 '16 at 19:56
  • @AlexisWilke I don't get what you want. How I understood your Q is how to replace @ADMIN_IPS@ with ip addresses in a variable, in this case ADMIN_IPS. Can you please update your Q with a more specific expected output? – tachomi Jul 6 '16 at 20:03
  • To better show the point, I now show a line before and a line after with other rules. The input file usually has 100 or so rules... some of which need duplication if ADMIN_IPS (and other variables) have multiple entries. – Alexis Wilke Jul 6 '16 at 20:08
  • @AlexisWilke I updated, hope it's what you're looking for – tachomi Jul 6 '16 at 20:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.