4

I have a file, lists.txt, that looks like this:

// stuff at beginning of file

var list1 = new Array();
i = 0;
list1[i++] = 'a';
list1[i++] = 'b';
...
list1[i++] = 'z';

var list2 = new Array();
i = 0;
list2[i++] = 'a';
list2[i++] = 'b';
...
list2[i++] = 'z';

// other stuff at end of file

I need to append to each of these lists (there are more than two of them) and end up with something like this:

var list1 = new Array();
i = 0;
list1[i++] = 'a';
list1[i++] = 'b';
...
list1[i++] = 'z';
list1[i++] = 'something new';

var list2 = new Array();
i = 0;
list2[i++] = 'a';
list2[i++] = 'b';
...
list2[i++] = 'z';
list2[i++] = 'another thing';

// other stuff at end of file

I've been wracking my brain on this for a while. I know how to get the last occurrence of each list:

list1_last=$(grep "list1\[i++\]" lists.txt | tail -1)
list2_last=$(grep "list2\[i++\]" lists.txt | tail -1)

I know how to get everything between the start of the first list and the start of the second list (inclusive):

list1=$(sed -n '/var list1/,/var list2/p' lists.txt)

I know I can get list1 without the first line of list2 with this Perl one-liner or this crazy sed script.

But I'm having a hard time putting all the pieces together. How should I do this?

Edit

The additional values I want to append are in another file, additional-values.txt, which for example contains:

list1[i++] = 'something new';
list2[i++] = 'another thing';

I guess you could say I'm trying to merge the two files.

Edit 2

The actual file looks more like this:

// comment
// comment
// ...
var foo = "bar";

// comment
// comment
// ...
var i= 0;

// comment
// comment
// ...
var GoodDomains = new Array();
i=0;
GoodDomains[i++] = "anything.com";  // comment
GoodDomains[i++] = "something.com"; // comment
...
GoodDomains[i++] = "lastthing.com"; // comment
// THIS IS WHERE I WANT TO INSERT SOMETHING

// comment
// comment
// ...
var BadDomains = new Array();
i=0;
BadDomains[i++] = "anything.com";  // comment
BadDomains[i++] = "something.com"; // comment
...
BadDomains[i++] = "lastthing.com"; // comment
// THIS IS WHERE I WANT TO INSERT SOMETHING

// more lists, including GoodHosts, GoodURLs, etc.

// comment
// comment
// ...
for (i in GoodDomains) {
    ...
}

// loop through BadDomains, GoodHosts, GoodURLs, etc.

// comment
// comment
// ...
function IsNumIpAddr(host) {
    ...
}

I originally posted a simplified version because

  1. I'm not sure if the actual file will always follow this format (comments at the top, variable declarations, more comments, list definitions, functions, etc.)
  2. I'd like to find a generic solution to the problem (appending stuff to lists in the middle of a file)

Sorry if this was misleading.

  • And what is // stuff at beginning of file? How can we know when we've reached the part of the file that should be edited? – terdon May 25 '15 at 17:33
  • @terdon comments and variable declarations. You've identified part of the problem. See securemecca.com/Downloads/proxy_en.txt – David Kennedy May 25 '15 at 18:18
  • Oh, wow, that's a very different file from what you show. My perl solution will probably work but I suggest you include the actual file in your question. I would probably solve this by writing a script that sets a counter to 1 or 0 depending on where in the file we are. – terdon May 25 '15 at 18:52
  • @terdon sorry, I thought I had included the pertinent parts of the script D: – David Kennedy May 25 '15 at 18:54
  • You did indeed, pretty much. However, the fact that the comments can be multiple lines is important as is the fact that we're not dealing with listN. Also, it is very important that the rest of the file is completely different. – terdon May 25 '15 at 18:58
3

Since you're trying with sed ranges, here's one possible way to do it. The lines in your additional-values.txt follow the same pattern:

KEY[i++] = 'VALUE'; //etc

and as far as I can tell, each line should be inserted in a range that is always delimited by

var KEY = new Array();

and an empty line


so you could process additional-values.txt and turn it into a sed script that for each line does:

/^var KEY = new Array();/,/^$/{
/^$/ i\
KEY[i++] = 'VALUE'; // etc
}

that is, in /^var KEY = new Array();/,/^$/ range, insert line KEY[i++] = 'VALUE'; // etc before the empty line. You then use the script to process lists.txt:

sed 's/\\/&&/g' additional-values.txt | \
sed 's|^\([^[]*\).*|/^var \1 = new Array();/,/^$/{\
/^$/ i\\\
&\
}|' | sed -f - lists.txt

The first sed escapes any backslashes, the second sed processes additional-values.txt turning it into a script that is used by the third sed (via -f) to process lists.txt.
e.g. sample additional-values.txt content:

GoodDomains[i++] = '^stuff/here/'; \
BadDomains[i++] = '%XYZ+=?\\<>';
GoodNetworks[i++] = '|*{};:\'; // Malware\\
BadDomains[i++] = '\$.|&$@"#"!||';

the result of:

sed 's/\\/&&/g' additional-values.txt | \
sed 's|^\([^[]*\).*|/^var \1 = new Array();/,/^$/{\
/^$/ i\\\
&\
}|'

is

/^var GoodDomains = new Array();/,/^$/{
/^$/ i\
GoodDomains[i++] = '^stuff/here/'; \\
}
/^var BadDomains = new Array();/,/^$/{
/^$/ i\
BadDomains[i++] = '%XYZ+=?\\\\<>';
}
/^var GoodNetworks = new Array();/,/^$/{
/^$/ i\
GoodNetworks[i++] = '|*{};:\\'; // Malware\\\\
}
/^var BadDomains = new Array();/,/^$/{
/^$/ i\
BadDomains[i++] = '\\$.|&$@"#"!||'; 
}

this is then passed to sed -f - lists.txt so with e.g. sample lists.txt:

// Counter Variable to initalize the arrays.
var i= 0;

var GoodDomains = new Array();
i=0;
GoodDomains[i++] = 'aba.com'; // Phish - 2010-02-05

var GoodNetworks = new Array();
i=0;
GoodNetworks[i++] = '10.0.0.0, 255.0.0.0';  // NRIP
// GoodNetworks[i++] = "63.140.35.160"; // DNSWCD 2o7

var BadDomains = new Array();
i=0;
BadDomains[i++] = '.0catch.com'; // AdServer - 2009-06-16

//var BadDomains = new Array();

running:

sed 's/\\/&&/g' additional-values.txt | \
sed 's|^\([^[]*\).*|/^var \1 = new Array();/,/^$/{\
/^$/ i\\\
&\
}|' | sed -f - lists.txt

outputs:

// Counter Variable to initalize the arrays.
var i= 0;

var GoodDomains = new Array();
i=0;
GoodDomains[i++] = 'aba.com'; // Phish - 2010-02-05
GoodDomains[i++] = '^stuff/here/'; \

var GoodNetworks = new Array();
i=0;
GoodNetworks[i++] = '10.0.0.0, 255.0.0.0';  // NRIP
// GoodNetworks[i++] = "63.140.35.160"; // DNSWCD 2o7
GoodNetworks[i++] = '|*{};:\'; // Malware\\

var BadDomains = new Array();
i=0;
BadDomains[i++] = '.0catch.com'; // AdServer - 2009-06-16
BadDomains[i++] = '%XYZ+=?\\<>';
BadDomains[i++] = '\$.|&$@"#"!||'; 

//var BadDomains = new Array();

If you prefer gnu sed and process substitution:

sed -E 's|^([^[]*).*|/^var \1 = new Array();/,/^$/{/^$/ i\\\n&\
}|' <(sed 's/\\/&&/g' additional-values.txt) | sed -f - lists.txt
  • 1
    My brain just exploded into a million pieces. – David Kennedy May 26 '15 at 1:20
  • @Koveras - put it back together - this is good. See this for how/why it works. It's better than my answer - which i'm about to delete. i do think the best way to go is to put separate appended lists in separate files - you don't need all of that stuff in the same one like that - that's what preprocessors are for. The fs is your friend. Don - kudos. I love it when code writes code clean. – mikeserv May 26 '15 at 3:36
  • `/^$/ i\\\n\1\2` would be good for indenting the code - rather than a hard linebreak, – Peter.O May 26 '15 at 8:44
  • @Koveras - :) I'll take that as a compliment... I've added a gnu sed alternative that does the same thing (other seds might work too if they support ERE and i\\\n)... – don_crissti May 26 '15 at 13:47
  • It was meant as one. Can you explain the difference between the GNU version and the normal script? – David Kennedy May 27 '15 at 18:38
4

If you reverse the file, you can add a line the first time you see something:

tac lists.txt |
awk -v l1="list1" -v val1="something new" \
    -v l2="list2" -v val2="another thing" '
          index($0, l1"[i++]") && !found1 {
              printf "%s[i++] = \"%s\";\n", l1, val1
              found1 = 1
          }
          index($0, l2"[i++]") && !found2 { 
              printf "%s[i++] = \"%s\";\n", l2, val2
              found2 = 1
          }
          {print}
' |
tac > lists.txt.new

It's a bit un-DRY, but it'll do.

I missed that there was "additional-values.txt". It's way better this way:

tac lists.txt | 
awk '
    NR == FNR {additional[$1] = $0; next}
    $1 in additional && !found[$1] {print additional[$1]; found[$1] = 1}
    {print}
' additional-values.txt - | 
tac > newfile
  • Short, sweet, and functional. I like it. – David Kennedy May 26 '15 at 3:06
  • 1
    It currently doesn't handle more than one (the last encountered) additional entry in additional-values.txt – Peter.O May 26 '15 at 8:36
  • @Peter.O; Why do you think so? (Glenn's code seems reasonable - it uses additional depending on $1! - and certainly works for me.) – Janis May 26 '15 at 12:01
  • He's quite right. Because I'm storing the additional values in a simple associative array, I can only store one line for each variable. To be able to store multiple lines, I'd switch to Perl where hash-of-arrays is easy. – glenn jackman May 26 '15 at 12:09
  • 2
    To cater for multiple additonal lines per list, change 1) additional[$1] = $0; to additional[$1] = $0 "\n" additional[$1]; and 2) print additional[$1]; to printf "%s",additional[$1]; – Peter.O May 26 '15 at 15:41
2

If the lists in your input file is separated by a blank line, you can use a tool that lets you set the record separator (what defines a "line") to consecutive newlines. For example, in Perl (assuming your replacements are in a file called additions):

perl -ne 'BEGIN{## Open the additions file
                open($fh,"additions"); 
                while(<$fh>){ 
                  ## Get the name of the current list
                  /list./; 
                  ## save this replacement in the %f hash
                  $f{$&}=$_;
                }
                ## Set the record separator to consecutive newlines.
                $/="\n\n";
               }
          ## Now that the BEGIN{} block is finished, process the
          ## input file.

         ## Does this line match "list."? 
         if(/list./){
            chomp; ## remove trailing newlines. 
            ## Add the addition to this "line"
            $_.= "\n$f{$&}\n\n"; 
          } 
         ## print each input line
         print ' file 

The above can be condensed to:

perl -ne 'BEGIN{open($fh,"additions"); while(<$fh>){/list./;$f{$&}=$_;}$/="\n\n";}
         if(/list./){chomp;$_.= "\n$f{$&}\n\n"; }; print ' file 
  • I think the only lines that would need to be modified in the case that list1 is actually GoodDomains, list2 is actually BadDomains, etc. are 5 /list./; and 16 if(/list./). Maybe something like ^\([^[]*\)\(\[.*\) from @don_crissti's answer to capture everything up to the open bracket? – David Kennedy May 26 '15 at 1:54
  • By the way, the condensed version has different parens at the beginning of the second line: if(/(list.)/) vs if(/list./). Does that affect anything? – David Kennedy May 26 '15 at 1:56
  • @Koveras no, they're irrelevant. I had originally thought of capturing the pattern instead of using $& (whatever was matched). I have ow removed them. – terdon May 26 '15 at 17:15
1

I finally came up with something that works:

# print from beginning of file to "var list1" (exclusive)                                                                                 
sed "/var list1/,\$d" lists.txt > merged.txt

# print from "var list1" to last member of array
lastlist1=$(grep -n "list1\[i++\]" lists.txt | tail -1 | cut -f1 -d:)
sed -n "/var list1/,$(echo $lastlist1)p" lists.txt >> merged.txt
grep "^list1" additional-values.txt >> merged.txt

# print from "var list2" to last member of array
lastlist2=$(grep -n "list2\[i++\]" lists.txt | tail -1 | cut -f1 -d:)
sed -n "/var list2/,$(echo $lastlist2)p" lists.txt >> merged.txt
grep "^list2" additional-values.txt >> merged.txt

# do this for list3, list4,... listn

# print from last member of listn (exclusive) to end of file
sed "1,$(echo $lastlistn)d" lists.txt >> merged.txt

This is pretty tedious and can probably be improved but at least I understand it.

1

Given that your lists are separated by new lines such as this

var list1 = new Array();
i = 0;
list1[i++] = 'a';
list1[i++] = 'b';
list1[i++] = 'z';

var list2 = new Array();
i = 0;
list2[i++] = 'a';
list2[i++] = 'b';
list2[i++] = 'z';\n

And if additional-lists.txt looks like:

list1[i++] = 'something new';
list2[i++] = 'another thing';

Then this bash/sed script will produce the desired output:

#! /bin/bash
a="lists.txt"
b="additional-values.txt"
while read line; do
    list=$(expr match "$line" '\(.*\[\)')   
    list=${list::-1}
    sed -i "/$list\[i++\]/{:loop; n; /^$/{s/^$/$line\n/; b}; b loop;}" $a
done < $b

It does this by reading each line of additional-values.txt and getting the substring of the line up to the [ (we are assuming that additional-lists.txt is of the format name[i++]...), eg "list1[", then it removes the last character to get the list name. Then it starts a sed script that matches the list name (notice the use of double quotes to use the bash variable), and then starts a loop that terminates when you reach a blank line. Finally it replaces the blank line with the line from additional values (and a newline). The -i option means edit in place.

Output:

 $ cat lists.txt
 var list1 = new Array();
 i = 0;
 list1[i++] = 'a';
 list1[i++] = 'b';
 list1[i++] = 'z';
 list1[i++] = 'something new';

 var list2 = new Array();
 i = 0;
 list2[i++] = 'a';
 list2[i++] = 'b';
 list2[i++] = 'z';
 list2[i++] = 'another thing';
  • Thanks I have made it generic by no longer searching for the test "list" – bkmoney May 25 '15 at 23:36
  • @G-Man this solution can be abstracted. Please see my latest edit. – David Kennedy May 26 '15 at 0:39
  • You're right about the bracket. And can you post the exact input and output? It shouldn't have anything to do with an underscore I think – bkmoney May 26 '15 at 2:21
  • It was still matching the previous line so I changed it to match also the i++. Should be working now – bkmoney May 26 '15 at 3:15
  • I had a bug with the conditional branching of sed, it returns to branch if the last substitution was successful. So I just added a substitution that replaces the line with itself if the line is not a blank line. – bkmoney May 26 '15 at 5:09
1

Uses awk's Record Separator RS set to: the last line of a list, plus the following blank line.

How it works:

Initially, RS is \n (default) — for the input file named in 1st arg: additional-values.txt
Immediately after reading in the first file, awk changes the value of RS to the value in 2nd arg.
The second file lists.txt, named in 3rd, arg has the RS defined by 2nd arg

line 1: Build Additional Values array
line 2: 1st field of split is current list's key — via RT (the text as per RS)
line 3: print record + record-separator (less one \n) + additional values

 awk 'RS == "\n" { addval[$1] = addval[$1] $0 "\n"; next }
   { split(RT,crskey) 
     print $0 gensub(/\n/,"","",RT) addval[ crskey[1] ] 
   }' additional-values.txt \
      RS='[^[\n]+[[]i[+][+][]] = [^;\n]+;\n\n' \
      lists.txt
  • I take it file is lists.txt. How do the additional values get inserted? – David Kennedy May 26 '15 at 1:44
  • The code has been ammended – Peter.O May 27 '15 at 6:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.