Append something to each list in a file

Question

I have a file, lists.txt, that looks like this:

// stuff at beginning of file

var list1 = new Array();
i = 0;
list1[i++] = 'a';
list1[i++] = 'b';
...
list1[i++] = 'z';

var list2 = new Array();
i = 0;
list2[i++] = 'a';
list2[i++] = 'b';
...
list2[i++] = 'z';

// other stuff at end of file

I need to append to each of these lists (there are more than two of them) and end up with something like this:

var list1 = new Array();
i = 0;
list1[i++] = 'a';
list1[i++] = 'b';
...
list1[i++] = 'z';
list1[i++] = 'something new';

var list2 = new Array();
i = 0;
list2[i++] = 'a';
list2[i++] = 'b';
...
list2[i++] = 'z';
list2[i++] = 'another thing';

// other stuff at end of file

I've been wracking my brain on this for a while. I know how to get the last occurrence of each list:

list1_last=$(grep "list1\[i++\]" lists.txt | tail -1)
list2_last=$(grep "list2\[i++\]" lists.txt | tail -1)

I know how to get everything between the start of the first list and the start of the second list (inclusive):

list1=$(sed -n '/var list1/,/var list2/p' lists.txt)

I know I can get list1 without the first line of list2 with this Perl one-liner or this crazy sed script.

But I'm having a hard time putting all the pieces together. How should I do this?

Edit

The additional values I want to append are in another file, additional-values.txt, which for example contains:

list1[i++] = 'something new';
list2[i++] = 'another thing';

I guess you could say I'm trying to merge the two files.

Edit 2

The actual file looks more like this:

// comment
// comment
// ...
var foo = "bar";

// comment
// comment
// ...
var i= 0;

// comment
// comment
// ...
var GoodDomains = new Array();
i=0;
GoodDomains[i++] = "anything.com";  // comment
GoodDomains[i++] = "something.com"; // comment
...
GoodDomains[i++] = "lastthing.com"; // comment
// THIS IS WHERE I WANT TO INSERT SOMETHING

// comment
// comment
// ...
var BadDomains = new Array();
i=0;
BadDomains[i++] = "anything.com";  // comment
BadDomains[i++] = "something.com"; // comment
...
BadDomains[i++] = "lastthing.com"; // comment
// THIS IS WHERE I WANT TO INSERT SOMETHING

// more lists, including GoodHosts, GoodURLs, etc.

// comment
// comment
// ...
for (i in GoodDomains) {
    ...
}

// loop through BadDomains, GoodHosts, GoodURLs, etc.

// comment
// comment
// ...
function IsNumIpAddr(host) {
    ...
}

I originally posted a simplified version because

1. I'm not sure if the actual file will always follow this format (comments at the top, variable declarations, more comments, list definitions, functions, etc.)
2. I'd like to find a generic solution to the problem (appending stuff to lists in the middle of a file)

Sorry if this was misleading.

Answer 1

If you reverse the file, you can add a line the first time you see something:

tac lists.txt |
awk -v l1="list1" -v val1="something new" \
    -v l2="list2" -v val2="another thing" '
          index($0, l1"[i++]") && !found1 {
              printf "%s[i++] = \"%s\";\n", l1, val1
              found1 = 1
          }
          index($0, l2"[i++]") && !found2 { 
              printf "%s[i++] = \"%s\";\n", l2, val2
              found2 = 1
          }
          {print}
' |
tac > lists.txt.new

It's a bit un-DRY, but it'll do.

I missed that there was "additional-values.txt". It's way better this way:

tac lists.txt | 
awk '
    NR == FNR {additional[$1] = $0; next}
    $1 in additional && !found[$1] {print additional[$1]; found[$1] = 1}
    {print}
' additional-values.txt - | 
tac > newfile

Answer 2

Since you're trying with sed ranges, here's one possible way to do it. The lines in your additional-values.txt follow the same pattern:

KEY[i++] = 'VALUE'; //etc

and as far as I can tell, each line should be inserted in a range that is always delimited by

var KEY = new Array();

and an empty line

so you could process additional-values.txt and turn it into a sed script that for each line does:

/^var KEY = new Array();/,/^$/{
/^$/ i\
KEY[i++] = 'VALUE'; // etc
}

that is, in /^var KEY = new Array();/,/^$/ range, insert line KEY[i++] = 'VALUE'; // etc before the empty line. You then use the script to process lists.txt:

sed 's/\\/&&/g' additional-values.txt | \
sed 's|^\([^[]*\).*|/^var \1 = new Array();/,/^$/{\
/^$/ i\\\
&\
}|' | sed -f - lists.txt

The first sed escapes any backslashes, the second sed processes additional-values.txt turning it into a script that is used by the third sed (via -f) to process lists.txt.
e.g. sample additional-values.txt content:

GoodDomains[i++] = '^stuff/here/'; \
BadDomains[i++] = '%XYZ+=?\\<>';
GoodNetworks[i++] = '|*{};:\'; // Malware\\
BadDomains[i++] = '\$.|&$@"#"!||';

the result of:

sed 's/\\/&&/g' additional-values.txt | \
sed 's|^\([^[]*\).*|/^var \1 = new Array();/,/^$/{\
/^$/ i\\\
&\
}|'

is

/^var GoodDomains = new Array();/,/^$/{
/^$/ i\
GoodDomains[i++] = '^stuff/here/'; \\
}
/^var BadDomains = new Array();/,/^$/{
/^$/ i\
BadDomains[i++] = '%XYZ+=?\\\\<>';
}
/^var GoodNetworks = new Array();/,/^$/{
/^$/ i\
GoodNetworks[i++] = '|*{};:\\'; // Malware\\\\
}
/^var BadDomains = new Array();/,/^$/{
/^$/ i\
BadDomains[i++] = '\\$.|&$@"#"!||'; 
}

this is then passed to sed -f - lists.txt so with e.g. sample lists.txt:

// Counter Variable to initalize the arrays.
var i= 0;

var GoodDomains = new Array();
i=0;
GoodDomains[i++] = 'aba.com'; // Phish - 2010-02-05

var GoodNetworks = new Array();
i=0;
GoodNetworks[i++] = '10.0.0.0, 255.0.0.0';  // NRIP
// GoodNetworks[i++] = "63.140.35.160"; // DNSWCD 2o7

var BadDomains = new Array();
i=0;
BadDomains[i++] = '.0catch.com'; // AdServer - 2009-06-16

//var BadDomains = new Array();

running:

sed 's/\\/&&/g' additional-values.txt | \
sed 's|^\([^[]*\).*|/^var \1 = new Array();/,/^$/{\
/^$/ i\\\
&\
}|' | sed -f - lists.txt

outputs:

// Counter Variable to initalize the arrays.
var i= 0;

var GoodDomains = new Array();
i=0;
GoodDomains[i++] = 'aba.com'; // Phish - 2010-02-05
GoodDomains[i++] = '^stuff/here/'; \

var GoodNetworks = new Array();
i=0;
GoodNetworks[i++] = '10.0.0.0, 255.0.0.0';  // NRIP
// GoodNetworks[i++] = "63.140.35.160"; // DNSWCD 2o7
GoodNetworks[i++] = '|*{};:\'; // Malware\\

var BadDomains = new Array();
i=0;
BadDomains[i++] = '.0catch.com'; // AdServer - 2009-06-16
BadDomains[i++] = '%XYZ+=?\\<>';
BadDomains[i++] = '\$.|&$@"#"!||'; 

//var BadDomains = new Array();

---

If you prefer gnu sed and process substitution:

sed -E 's|^([^[]*).*|/^var \1 = new Array();/,/^$/{/^$/ i\\\n&\
}|' <(sed 's/\\/&&/g' additional-values.txt) | sed -f - lists.txt

Answer 3

If the lists in your input file is separated by a blank line, you can use a tool that lets you set the record separator (what defines a "line") to consecutive newlines. For example, in Perl (assuming your replacements are in a file called additions):

perl -ne 'BEGIN{## Open the additions file
                open($fh,"additions"); 
                while(<$fh>){ 
                  ## Get the name of the current list
                  /list./; 
                  ## save this replacement in the %f hash
                  $f{$&}=$_;
                }
                ## Set the record separator to consecutive newlines.
                $/="\n\n";
               }
          ## Now that the BEGIN{} block is finished, process the
          ## input file.

         ## Does this line match "list."? 
         if(/list./){
            chomp; ## remove trailing newlines. 
            ## Add the addition to this "line"
            $_.= "\n$f{$&}\n\n"; 
          } 
         ## print each input line
         print ' file 

The above can be condensed to:

perl -ne 'BEGIN{open($fh,"additions"); while(<$fh>){/list./;$f{$&}=$_;}$/="\n\n";}
         if(/list./){chomp;$_.= "\n$f{$&}\n\n"; }; print ' file 

Answer 4

I finally came up with something that works:

# print from beginning of file to "var list1" (exclusive)                                                                                 
sed "/var list1/,\$d" lists.txt > merged.txt

# print from "var list1" to last member of array
lastlist1=$(grep -n "list1\[i++\]" lists.txt | tail -1 | cut -f1 -d:)
sed -n "/var list1/,$(echo $lastlist1)p" lists.txt >> merged.txt
grep "^list1" additional-values.txt >> merged.txt

# print from "var list2" to last member of array
lastlist2=$(grep -n "list2\[i++\]" lists.txt | tail -1 | cut -f1 -d:)
sed -n "/var list2/,$(echo $lastlist2)p" lists.txt >> merged.txt
grep "^list2" additional-values.txt >> merged.txt

# do this for list3, list4,... listn

# print from last member of listn (exclusive) to end of file
sed "1,$(echo $lastlistn)d" lists.txt >> merged.txt

This is pretty tedious and can probably be improved but at least I understand it.

Answer 5

Given that your lists are separated by new lines such as this

var list1 = new Array();
i = 0;
list1[i++] = 'a';
list1[i++] = 'b';
list1[i++] = 'z';

var list2 = new Array();
i = 0;
list2[i++] = 'a';
list2[i++] = 'b';
list2[i++] = 'z';\n

And if additional-lists.txt looks like:

list1[i++] = 'something new';
list2[i++] = 'another thing';

Then this bash/sed script will produce the desired output:

#! /bin/bash
a="lists.txt"
b="additional-values.txt"
while read line; do
    list=$(expr match "$line" '\(.*\[\)')   
    list=${list::-1}
    sed -i "/$list\[i++\]/{:loop; n; /^$/{s/^$/$line\n/; b}; b loop;}" $a
done < $b

It does this by reading each line of additional-values.txt and getting the substring of the line up to the [ (we are assuming that additional-lists.txt is of the format name[i++]...), eg "list1[", then it removes the last character to get the list name. Then it starts a sed script that matches the list name (notice the use of double quotes to use the bash variable), and then starts a loop that terminates when you reach a blank line. Finally it replaces the blank line with the line from additional values (and a newline). The -i option means edit in place.

Output:

 $ cat lists.txt
 var list1 = new Array();
 i = 0;
 list1[i++] = 'a';
 list1[i++] = 'b';
 list1[i++] = 'z';
 list1[i++] = 'something new';

 var list2 = new Array();
 i = 0;
 list2[i++] = 'a';
 list2[i++] = 'b';
 list2[i++] = 'z';
 list2[i++] = 'another thing';

Answer 6

Uses awk's Record Separator RS set to: the last line of a list, plus the following blank line.

How it works:

Initially, RS is \n (default) — for the input file named in 1st arg: additional-values.txt
Immediately after reading in the first file, awk changes the value of RS to the value in 2nd arg.
The second file lists.txt, named in 3rd, arg has the RS defined by 2nd arg

line 1: Build Additional Values array
line 2: 1st field of split is current list's key — via RT (the text as per RS)
line 3: print record + record-separator (less one \n) + additional values

 awk 'RS == "\n" { addval[$1] = addval[$1] $0 "\n"; next }
   { split(RT,crskey) 
     print $0 gensub(/\n/,"","",RT) addval[ crskey[1] ] 
   }' additional-values.txt \
      RS='[^[\n]+[[]i[+][+][]] = [^;\n]+;\n\n' \
      lists.txt