4

I have a file, lists.txt, that looks like this:

// stuff at beginning of file

var list1 = new Array();
i = 0;
list1[i++] = 'a';
list1[i++] = 'b';
...
list1[i++] = 'z';

var list2 = new Array();
i = 0;
list2[i++] = 'a';
list2[i++] = 'b';
...
list2[i++] = 'z';

// other stuff at end of file

I need to append to each of these lists (there are more than two of them) and end up with something like this:

var list1 = new Array();
i = 0;
list1[i++] = 'a';
list1[i++] = 'b';
...
list1[i++] = 'z';
list1[i++] = 'something new';

var list2 = new Array();
i = 0;
list2[i++] = 'a';
list2[i++] = 'b';
...
list2[i++] = 'z';
list2[i++] = 'another thing';

// other stuff at end of file

I've been wracking my brain on this for a while. I know how to get the last occurrence of each list:

list1_last=$(grep "list1\[i++\]" lists.txt | tail -1)
list2_last=$(grep "list2\[i++\]" lists.txt | tail -1)

I know how to get everything between the start of the first list and the start of the second list (inclusive):

list1=$(sed -n '/var list1/,/var list2/p' lists.txt)

I know I can get list1 without the first line of list2 with this Perl one-liner or this crazy sed script.

But I'm having a hard time putting all the pieces together. How should I do this?

Edit

The additional values I want to append are in another file, additional-values.txt, which for example contains:

list1[i++] = 'something new';
list2[i++] = 'another thing';

I guess you could say I'm trying to merge the two files.

Edit 2

The actual file looks more like this:

// comment
// comment
// ...
var foo = "bar";

// comment
// comment
// ...
var i= 0;

// comment
// comment
// ...
var GoodDomains = new Array();
i=0;
GoodDomains[i++] = "anything.com";  // comment
GoodDomains[i++] = "something.com"; // comment
...
GoodDomains[i++] = "lastthing.com"; // comment
// THIS IS WHERE I WANT TO INSERT SOMETHING

// comment
// comment
// ...
var BadDomains = new Array();
i=0;
BadDomains[i++] = "anything.com";  // comment
BadDomains[i++] = "something.com"; // comment
...
BadDomains[i++] = "lastthing.com"; // comment
// THIS IS WHERE I WANT TO INSERT SOMETHING

// more lists, including GoodHosts, GoodURLs, etc.

// comment
// comment
// ...
for (i in GoodDomains) {
    ...
}

// loop through BadDomains, GoodHosts, GoodURLs, etc.

// comment
// comment
// ...
function IsNumIpAddr(host) {
    ...
}

I originally posted a simplified version because

  1. I'm not sure if the actual file will always follow this format (comments at the top, variable declarations, more comments, list definitions, functions, etc.)
  2. I'd like to find a generic solution to the problem (appending stuff to lists in the middle of a file)

Sorry if this was misleading.

6
  • And what is // stuff at beginning of file? How can we know when we've reached the part of the file that should be edited?
    – terdon
    May 25, 2015 at 17:33
  • @terdon comments and variable declarations. You've identified part of the problem. See securemecca.com/Downloads/proxy_en.txt May 25, 2015 at 18:18
  • Oh, wow, that's a very different file from what you show. My perl solution will probably work but I suggest you include the actual file in your question. I would probably solve this by writing a script that sets a counter to 1 or 0 depending on where in the file we are.
    – terdon
    May 25, 2015 at 18:52
  • @terdon sorry, I thought I had included the pertinent parts of the script D: May 25, 2015 at 18:54
  • You did indeed, pretty much. However, the fact that the comments can be multiple lines is important as is the fact that we're not dealing with listN. Also, it is very important that the rest of the file is completely different.
    – terdon
    May 25, 2015 at 18:58

6 Answers 6

4

If you reverse the file, you can add a line the first time you see something:

tac lists.txt |
awk -v l1="list1" -v val1="something new" \
    -v l2="list2" -v val2="another thing" '
          index($0, l1"[i++]") && !found1 {
              printf "%s[i++] = \"%s\";\n", l1, val1
              found1 = 1
          }
          index($0, l2"[i++]") && !found2 { 
              printf "%s[i++] = \"%s\";\n", l2, val2
              found2 = 1
          }
          {print}
' |
tac > lists.txt.new

It's a bit un-DRY, but it'll do.

I missed that there was "additional-values.txt". It's way better this way:

tac lists.txt | 
awk '
    NR == FNR {additional[$1] = $0; next}
    $1 in additional && !found[$1] {print additional[$1]; found[$1] = 1}
    {print}
' additional-values.txt - | 
tac > newfile
5
  • Short, sweet, and functional. I like it. May 26, 2015 at 3:06
  • 1
    It currently doesn't handle more than one (the last encountered) additional entry in additional-values.txt
    – Peter.O
    May 26, 2015 at 8:36
  • @Peter.O; Why do you think so? (Glenn's code seems reasonable - it uses additional depending on $1! - and certainly works for me.)
    – Janis
    May 26, 2015 at 12:01
  • He's quite right. Because I'm storing the additional values in a simple associative array, I can only store one line for each variable. To be able to store multiple lines, I'd switch to Perl where hash-of-arrays is easy. May 26, 2015 at 12:09
  • 2
    To cater for multiple additonal lines per list, change 1) additional[$1] = $0; to additional[$1] = $0 "\n" additional[$1]; and 2) print additional[$1]; to printf "%s",additional[$1];
    – Peter.O
    May 26, 2015 at 15:41
3

Since you're trying with sed ranges, here's one possible way to do it. The lines in your additional-values.txt follow the same pattern:

KEY[i++] = 'VALUE'; //etc

and as far as I can tell, each line should be inserted in a range that is always delimited by

var KEY = new Array();

and an empty line


so you could process additional-values.txt and turn it into a sed script that for each line does:

/^var KEY = new Array();/,/^$/{
/^$/ i\
KEY[i++] = 'VALUE'; // etc
}

that is, in /^var KEY = new Array();/,/^$/ range, insert line KEY[i++] = 'VALUE'; // etc before the empty line. You then use the script to process lists.txt:

sed 's/\\/&&/g' additional-values.txt | \
sed 's|^\([^[]*\).*|/^var \1 = new Array();/,/^$/{\
/^$/ i\\\
&\
}|' | sed -f - lists.txt

The first sed escapes any backslashes, the second sed processes additional-values.txt turning it into a script that is used by the third sed (via -f) to process lists.txt.
e.g. sample additional-values.txt content:

GoodDomains[i++] = '^stuff/here/'; \
BadDomains[i++] = '%XYZ+=?\\<>';
GoodNetworks[i++] = '|*{};:\'; // Malware\\
BadDomains[i++] = '\$.|&$@"#"!||';

the result of:

sed 's/\\/&&/g' additional-values.txt | \
sed 's|^\([^[]*\).*|/^var \1 = new Array();/,/^$/{\
/^$/ i\\\
&\
}|'

is

/^var GoodDomains = new Array();/,/^$/{
/^$/ i\
GoodDomains[i++] = '^stuff/here/'; \\
}
/^var BadDomains = new Array();/,/^$/{
/^$/ i\
BadDomains[i++] = '%XYZ+=?\\\\<>';
}
/^var GoodNetworks = new Array();/,/^$/{
/^$/ i\
GoodNetworks[i++] = '|*{};:\\'; // Malware\\\\
}
/^var BadDomains = new Array();/,/^$/{
/^$/ i\
BadDomains[i++] = '\\$.|&$@"#"!||'; 
}

this is then passed to sed -f - lists.txt so with e.g. sample lists.txt:

// Counter Variable to initalize the arrays.
var i= 0;

var GoodDomains = new Array();
i=0;
GoodDomains[i++] = 'aba.com'; // Phish - 2010-02-05

var GoodNetworks = new Array();
i=0;
GoodNetworks[i++] = '10.0.0.0, 255.0.0.0';  // NRIP
// GoodNetworks[i++] = "63.140.35.160"; // DNSWCD 2o7

var BadDomains = new Array();
i=0;
BadDomains[i++] = '.0catch.com'; // AdServer - 2009-06-16

//var BadDomains = new Array();

running:

sed 's/\\/&&/g' additional-values.txt | \
sed 's|^\([^[]*\).*|/^var \1 = new Array();/,/^$/{\
/^$/ i\\\
&\
}|' | sed -f - lists.txt

outputs:

// Counter Variable to initalize the arrays.
var i= 0;

var GoodDomains = new Array();
i=0;
GoodDomains[i++] = 'aba.com'; // Phish - 2010-02-05
GoodDomains[i++] = '^stuff/here/'; \

var GoodNetworks = new Array();
i=0;
GoodNetworks[i++] = '10.0.0.0, 255.0.0.0';  // NRIP
// GoodNetworks[i++] = "63.140.35.160"; // DNSWCD 2o7
GoodNetworks[i++] = '|*{};:\'; // Malware\\

var BadDomains = new Array();
i=0;
BadDomains[i++] = '.0catch.com'; // AdServer - 2009-06-16
BadDomains[i++] = '%XYZ+=?\\<>';
BadDomains[i++] = '\$.|&$@"#"!||'; 

//var BadDomains = new Array();

If you prefer gnu sed and process substitution:

sed -E 's|^([^[]*).*|/^var \1 = new Array();/,/^$/{/^$/ i\\\n&\
}|' <(sed 's/\\/&&/g' additional-values.txt) | sed -f - lists.txt
6
  • 1
    My brain just exploded into a million pieces. May 26, 2015 at 1:20
  • @Koveras - put it back together - this is good. See this for how/why it works. It's better than my answer - which i'm about to delete. i do think the best way to go is to put separate appended lists in separate files - you don't need all of that stuff in the same one like that - that's what preprocessors are for. The fs is your friend. Don - kudos. I love it when code writes code clean.
    – mikeserv
    May 26, 2015 at 3:36
  • `/^$/ i\\\n\1\2` would be good for indenting the code - rather than a hard linebreak,
    – Peter.O
    May 26, 2015 at 8:44
  • @Koveras - :) I'll take that as a compliment... I've added a gnu sed alternative that does the same thing (other seds might work too if they support ERE and i\\\n)... May 26, 2015 at 13:47
  • It was meant as one. Can you explain the difference between the GNU version and the normal script? May 27, 2015 at 18:38
2

If the lists in your input file is separated by a blank line, you can use a tool that lets you set the record separator (what defines a "line") to consecutive newlines. For example, in Perl (assuming your replacements are in a file called additions):

perl -ne 'BEGIN{## Open the additions file
                open($fh,"additions"); 
                while(<$fh>){ 
                  ## Get the name of the current list
                  /list./; 
                  ## save this replacement in the %f hash
                  $f{$&}=$_;
                }
                ## Set the record separator to consecutive newlines.
                $/="\n\n";
               }
          ## Now that the BEGIN{} block is finished, process the
          ## input file.

         ## Does this line match "list."? 
         if(/list./){
            chomp; ## remove trailing newlines. 
            ## Add the addition to this "line"
            $_.= "\n$f{$&}\n\n"; 
          } 
         ## print each input line
         print ' file 

The above can be condensed to:

perl -ne 'BEGIN{open($fh,"additions"); while(<$fh>){/list./;$f{$&}=$_;}$/="\n\n";}
         if(/list./){chomp;$_.= "\n$f{$&}\n\n"; }; print ' file 
3
  • I think the only lines that would need to be modified in the case that list1 is actually GoodDomains, list2 is actually BadDomains, etc. are 5 /list./; and 16 if(/list./). Maybe something like ^\([^[]*\)\(\[.*\) from @don_crissti's answer to capture everything up to the open bracket? May 26, 2015 at 1:54
  • By the way, the condensed version has different parens at the beginning of the second line: if(/(list.)/) vs if(/list./). Does that affect anything? May 26, 2015 at 1:56
  • @Koveras no, they're irrelevant. I had originally thought of capturing the pattern instead of using $& (whatever was matched). I have ow removed them.
    – terdon
    May 26, 2015 at 17:15
1

I finally came up with something that works:

# print from beginning of file to "var list1" (exclusive)                                                                                 
sed "/var list1/,\$d" lists.txt > merged.txt

# print from "var list1" to last member of array
lastlist1=$(grep -n "list1\[i++\]" lists.txt | tail -1 | cut -f1 -d:)
sed -n "/var list1/,$(echo $lastlist1)p" lists.txt >> merged.txt
grep "^list1" additional-values.txt >> merged.txt

# print from "var list2" to last member of array
lastlist2=$(grep -n "list2\[i++\]" lists.txt | tail -1 | cut -f1 -d:)
sed -n "/var list2/,$(echo $lastlist2)p" lists.txt >> merged.txt
grep "^list2" additional-values.txt >> merged.txt

# do this for list3, list4,... listn

# print from last member of listn (exclusive) to end of file
sed "1,$(echo $lastlistn)d" lists.txt >> merged.txt

This is pretty tedious and can probably be improved but at least I understand it.

1

Given that your lists are separated by new lines such as this

var list1 = new Array();
i = 0;
list1[i++] = 'a';
list1[i++] = 'b';
list1[i++] = 'z';

var list2 = new Array();
i = 0;
list2[i++] = 'a';
list2[i++] = 'b';
list2[i++] = 'z';\n

And if additional-lists.txt looks like:

list1[i++] = 'something new';
list2[i++] = 'another thing';

Then this bash/sed script will produce the desired output:

#! /bin/bash
a="lists.txt"
b="additional-values.txt"
while read line; do
    list=$(expr match "$line" '\(.*\[\)')   
    list=${list::-1}
    sed -i "/$list\[i++\]/{:loop; n; /^$/{s/^$/$line\n/; b}; b loop;}" $a
done < $b

It does this by reading each line of additional-values.txt and getting the substring of the line up to the [ (we are assuming that additional-lists.txt is of the format name[i++]...), eg "list1[", then it removes the last character to get the list name. Then it starts a sed script that matches the list name (notice the use of double quotes to use the bash variable), and then starts a loop that terminates when you reach a blank line. Finally it replaces the blank line with the line from additional values (and a newline). The -i option means edit in place.

Output:

 $ cat lists.txt
 var list1 = new Array();
 i = 0;
 list1[i++] = 'a';
 list1[i++] = 'b';
 list1[i++] = 'z';
 list1[i++] = 'something new';

 var list2 = new Array();
 i = 0;
 list2[i++] = 'a';
 list2[i++] = 'b';
 list2[i++] = 'z';
 list2[i++] = 'another thing';
6
  • Thanks I have made it generic by no longer searching for the test "list"
    – bkmoney
    May 25, 2015 at 23:36
  • @G-Man this solution can be abstracted. Please see my latest edit. May 26, 2015 at 0:39
  • You're right about the bracket. And can you post the exact input and output? It shouldn't have anything to do with an underscore I think
    – bkmoney
    May 26, 2015 at 2:21
  • It was still matching the previous line so I changed it to match also the i++. Should be working now
    – bkmoney
    May 26, 2015 at 3:15
  • I had a bug with the conditional branching of sed, it returns to branch if the last substitution was successful. So I just added a substitution that replaces the line with itself if the line is not a blank line.
    – bkmoney
    May 26, 2015 at 5:09
1

Uses awk's Record Separator RS set to: the last line of a list, plus the following blank line.

How it works:

Initially, RS is \n (default) — for the input file named in 1st arg: additional-values.txt
Immediately after reading in the first file, awk changes the value of RS to the value in 2nd arg.
The second file lists.txt, named in 3rd, arg has the RS defined by 2nd arg

line 1: Build Additional Values array
line 2: 1st field of split is current list's key — via RT (the text as per RS)
line 3: print record + record-separator (less one \n) + additional values

 awk 'RS == "\n" { addval[$1] = addval[$1] $0 "\n"; next }
   { split(RT,crskey) 
     print $0 gensub(/\n/,"","",RT) addval[ crskey[1] ] 
   }' additional-values.txt \
      RS='[^[\n]+[[]i[+][+][]] = [^;\n]+;\n\n' \
      lists.txt
2
  • I take it file is lists.txt. How do the additional values get inserted? May 26, 2015 at 1:44
  • The code has been ammended
    – Peter.O
    May 27, 2015 at 6:47

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .