I have a web server which stores configurations to PostgreSQL, and I plan to retrieve information from a certain table and then later parse it into csv format.


<?xml version="1.0" encoding="UTF-8" standalone="yes"?><rule roleDefinition="false" id="1168" type="COMMON" enabled="true" owner="admin" buildingBlock="false" scope="LOCAL"><name>Recon: Remote Windows Server Scanner</name><notes>Reports a remote host attempting reconnaissance or suspicious connections on common Windows server ports to more than 60 hosts in 10 minutes.</notes><testDefinitions><test uid="4" id="80" name="com.q1labs.semsources.cre.tests.ThresholdFunction_Test" group="Functions"><text>when any of these &lt;a href='javascript:editParameter("4", "1")' class='dynamic'&gt;BB:CategoryDefinition: Recon Events, BB:CategoryDefinition: Suspicious Events&lt;/a&gt; with the same &lt;a href='javascript:editParameter("4", "2")' class='dynamic'&gt;source IP&lt;/a&gt; more than &lt;a href='javascript:editParameter("4", "3")' class='dynamic'&gt;5&lt;/a&gt; times, across &lt;a href='javascript:editParameter("4", "4")' class='dynamic'&gt;more than&lt;/a&gt; &lt;a href='javascript:editParameter("4", "5")' class='dynamic'&gt;59&lt;/a&gt; &lt;a href='javascript:editParameter("4", "6")' class='dynamic'&gt;destination IP&lt;/a&gt; within &lt;a href='javascript:editParameter("4", "7")' class='dynamic'&gt;10&lt;/a&gt; &lt;a href='javascript:editParameter("4", "8")' class='dynamic'&gt;minutes&lt;/a&gt;</text>


perl -nle '
my @enabled = /(?<=enabled="").*?(?<=\"\")/g;
my @group = /(?<=group="").*?(?<=\"\")/g;
my @name = /(?<=<name>).*?(?=<\/name>)/g;chomp(@name);
my @text = /(?<=<text>).*?(?=<\/text>)/g;chomp(@text);
print "@enabled;@group;@name;@text"
' Rules.csv |sed 's/\&lt\;/</g'|sed 's/\&gt\;/>/g' | sed -e 's/<*>//g' -e '/ *$/d' > Rules2.csv

Rules2.csv shows no content, and empty file, however when I break the command I get some very positive results, please see pastebin (as the code) is bit too long. The pastebin analysis would show that somewhere in command section sed -e 's/<*>//g' -e '/ *$/d' it results into empty my Rules2.csv file. But I don't know why or what is causing this behavior.

Any help that helps me achieve the desired output as shown in pastebin would be wonderful.

  • 2
    Exact duplicate to superuser.com/questions/699955/… – erik Jan 10 '14 at 21:07
  • You're very welcome. Please try to insert a couple of newlines into your XML payload as well if that wouldn't interfere with your intended question. Also, welcome to U&L :) – Joseph R. Jan 10 '14 at 21:07
  • @JosephR. Let me do it. Thanks for mentioning. – user3066819 Jan 10 '14 at 21:10

Although you should not parse XML with regexes, this works with your sample input:

perl -ple '
    ($enabled) = /(?<=enabled=").*?(?=")/g;
    ($group)   = /(?<=group=").*?(?=")/g;
    ($name)    = /(?<=<name>).*?(?=<\/name>)/g;
    ($text)    = /(?<=<text>).*?(?=<\/text>)/g;
    $_= "$enabled;$group;$name;$text";
' Rules.csv
true;Functions;Recon: Remote Windows Server Scanner;when any of these BB:CategoryDefinition: Recon Events, BB:CategoryDefinition: Suspicious Events with the same source IP more than 5 times, across more than 59 destination IP within 10 minutes


  • your "enabled" and "group" lookarounds were broken
  • if you're working in perl, you don't need to pipe into sed
  • the reason s/<*>//g erases all output is that you're matching zero or more open brackets up to the last close bracket -- this matches your entire string.
    • you probably intended s/<[^>]*>//g
  • thanks for making me realize my error by explicit mentioning them.I will run this code now. – user3066819 Jan 10 '14 at 21:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.