2

I want to separate 3 columns, the first with a single digit 4, 5 or 6. The second with an ID and the third with its description.

Sample input:

%ASA-4-105505: (Primary|Secondary) Failed to connect to peer unit peer-ip:port    
%ASA-4-105524: (Primary|Secondary) Transitioning to Negotiating state due to the presence of another Active HA unit    
%ASA-4-105553: (Primary|Secondary) Detected another Active HA unit    
%ASA-4-106023: Deny protocol src [interface_name:source_address/source_port] [([idfw_user|FQDN_string], sg_info)] dst interface_name:dest_address/dest_port [([idfw_user|FQDN_string], sg_info)] [type {string}, code {code}] by access_group acl_ID [0x8ed66b60, 0xf8852875]
%ASA-4-106027: Deny src [source address] dst [destination address] by access-group “access-list name”.
%ASA-4-106103: access-list acl_ID denied protocol for user username interface_name/source_address source_port interface_name/dest_address dest_port hit-cnt number first hit hash codes
%ASA-4-108004: action_class: action ESMTP req_resp from src_ifc:sip|sport to dest_ifc:dip|dport;further_info, page 1-23
%ASA-4-109017: User at IP_address exceeded auth proxy connection limit (max)
%ASA-4-109022: exceeded HTTPS proxy process limit
%ASA-4-109027: [aaa protocol] Unable to decipher response message Server = server_IP_address, User = user
%ASA-4-109028: aaa bypassed for same-security traffic from ingress_ interface:source_address/source_port to egress_interface:dest_address/dest_port
%ASA-4-109030: Autodetect ACL convert wildcard did not convert ACL access_list source | dest netmask netmask.

Expected output:

4   105505 (Primary|Secondary) Failed to connect to peer unit peer-ip:port
4   105524 (Primary|Secondary) Transitioning to Negotiating state due to the presence of another Active HA unit
4   105553 (Primary|Secondary) Detected another Active HA unit
4   106023 Deny protocol src [interface_name:source_address/source_port] [([idfw_user|FQDN_string], sg_info)] dst interface_name:dest_address/dest_port [([idfw_user|FQDN_string], sg_info)] [type {string}, code {code}] by access_group acl_ID [0x8ed66b60, 0xf8852875]
4   106027 Deny src [source address] dst [destination address] by access-group “access-list name”.
4   106103 access-list acl_ID denied protocol for user username interface_name/source_address source_port interface_name/dest_address dest_port hit-cnt number first hit hash codes
4   108004 action_class: action ESMTP req_resp from src_ifc:sip|sport to dest_ifc:dip|dport;further_info, page 1-23
4   109017 User at IP_address exceeded auth proxy connection limit (max)
4   109022 exceeded HTTPS proxy process limit
4   109027 [aaa protocol] Unable to decipher response message Server = server_IP_address, User = user
4   109028 aaa bypassed for same-security traffic from ingress_ interface:source_address/source_port to egress_interface:dest_address/dest_port
4   109030 Autodetect ACL convert wildcard did not convert ACL access_list source | dest netmask netmask.

I was able to extract the 2nd and 3rd columns with cat rawSyslog.txt | awk -F '[-:]' '{print $2 "\t" $3}' but the last column has a lot of special characters that mess with the output fields. How do I extract the final column?

2
  • Will the first column ever be anything other than 4 or 5 or 6 ? Mar 1, 2022 at 6:33
  • 1
    Yes, but it will be a natural number. Mar 1, 2022 at 15:22

9 Answers 9

4

Using GNU awk for gensub():

$ awk -F'[:-]' -v OFS='\t' '{print $2, $3, gensub(/[^ ]* /,"",1)}' file
4       105505  (Primary|Secondary) Failed to connect to peer unit peer-ip:port
4       105524  (Primary|Secondary) Transitioning to Negotiating state due to the presence of another Active HA unit
4       105553  (Primary|Secondary) Detected another Active HA unit
4       106023  Deny protocol src [interface_name:source_address/source_port] [([idfw_user|FQDN_string], sg_info)] dst interface_name:dest_address/dest_port [([idfw_user|FQDN_string], sg_info)] [type {string}, code {code}] by access_group acl_ID [0x8ed66b60, 0xf8852875]
4       106027  Deny src [source address] dst [destination address] by access-group “access-list name”.
4       106103  access-list acl_ID denied protocol for user username interface_name/source_address source_port interface_name/dest_address dest_port hit-cnt number first hit hash codes
4       108004  action_class: action ESMTP req_resp from src_ifc:sip|sport to dest_ifc:dip|dport;further_info, page 1-23
4       109017  User at IP_address exceeded auth proxy connection limit (max)
4       109022  exceeded HTTPS proxy process limit
4       109027  [aaa protocol] Unable to decipher response message Server = server_IP_address, User = user
4       109028  aaa bypassed for same-security traffic from ingress_ interface:source_address/source_port to egress_interface:dest_address/dest_port
4       109030  Autodetect ACL convert wildcard did not convert ACL access_list source | dest netmask netmask.

or using any awk:

$ awk -F'[:-]' -v OFS='\t' '{x=$0; sub(/[^ ]* /,"",x); print $2, $3, x}' file
4       105505  (Primary|Secondary) Failed to connect to peer unit peer-ip:port
4       105524  (Primary|Secondary) Transitioning to Negotiating state due to the presence of another Active HA unit
4       105553  (Primary|Secondary) Detected another Active HA unit
4       106023  Deny protocol src [interface_name:source_address/source_port] [([idfw_user|FQDN_string], sg_info)] dst interface_name:dest_address/dest_port [([idfw_user|FQDN_string], sg_info)] [type {string}, code {code}] by access_group acl_ID [0x8ed66b60, 0xf8852875]
4       106027  Deny src [source address] dst [destination address] by access-group “access-list name”.
4       106103  access-list acl_ID denied protocol for user username interface_name/source_address source_port interface_name/dest_address dest_port hit-cnt number first hit hash codes
4       108004  action_class: action ESMTP req_resp from src_ifc:sip|sport to dest_ifc:dip|dport;further_info, page 1-23
4       109017  User at IP_address exceeded auth proxy connection limit (max)
4       109022  exceeded HTTPS proxy process limit
4       109027  [aaa protocol] Unable to decipher response message Server = server_IP_address, User = user
4       109028  aaa bypassed for same-security traffic from ingress_ interface:source_address/source_port to egress_interface:dest_address/dest_port
4       109030  Autodetect ACL convert wildcard did not convert ACL access_list source | dest netmask netmask.
6
  • I need to be more thorough with regexps.. From what I understood: the first argument /[^ ]* / is looking for the leftmost, first occurrence of 0 or more characters that are not spaces, the second argument is replacing them with "" (nothing), and it the 3rd arg (target) is the 0th field delimited by either : or - Is that correct? Mar 1, 2022 at 15:13
  • You're correct for the first 2 args to both *sub() functions but idk what you mean by the 0th field - are you referring to the call to gensub() or to sub()?
    – Ed Morton
    Mar 1, 2022 at 15:21
  • I'm referring to sub(), when you initalized x = $0 Mar 1, 2022 at 15:23
  • 1
    $0 is the current record (a line in your case), not any of the fields of the record, so x=$0 is saving the current line in x and then sub(/[^ ]* /,"",x) removes the string that matches [^ ]* (i.e. the first string of non-blanks followed by a blank such as %ASA-4-105505: ) from the front of x thereby just leaving everything after the first blank in x so after the sub() x contains (Primary|Secondary) Failed to connect to peer unit peer-ip:port. You could add print x statements to the code to see what x contains at each step.
    – Ed Morton
    Mar 1, 2022 at 15:26
  • 1
    To be clear: not the current record, the contents of x which is a copy of the current record. If I trimmed the current record ($0) then I'd no longer have $2 and $3 available for printing which is why I'm saving a copy of the current record, $0, in x and then trimming x instead of trimming $0. I could alternatively have done y=$2; z=$3; sub(/[^ ]* /,""); print y, z, $0 which WOULD be trimming the current record (the default 3rd arg for sub() is $0).
    – Ed Morton
    Mar 1, 2022 at 15:35
2

A Perl approach:

$ perl -lne 'print join "\t",$1,$2,$3 if /-(\d)-(\d+):\s+(.*)/' file 
4   105505  (Primary|Secondary) Failed to connect to peer unit peer-ip:port    
4   105524  (Primary|Secondary) Transitioning to Negotiating state due to the presence of another Active HA unit    
4   105553  (Primary|Secondary) Detected another Active HA unit    
4   106023  Deny protocol src [interface_name:source_address/source_port] [([idfw_user|FQDN_string], sg_info)] dst interface_name:dest_address/dest_port [([idfw_user|FQDN_string], sg_info)] [type {string}, code {code}] by access_group acl_ID [0x8ed66b60, 0xf8852875]
4   106027  Deny src [source address] dst [destination address] by access-group “access-list name”.
4   106103  access-list acl_ID denied protocol for user username interface_name/source_address source_port interface_name/dest_address dest_port hit-cnt number first hit hash codes
4   108004  action_class: action ESMTP req_resp from src_ifc:sip|sport to dest_ifc:dip|dport;further_info, page 1-23
4   109017  User at IP_address exceeded auth proxy connection limit (max)
4   109022  exceeded HTTPS proxy process limit
4   109027  [aaa protocol] Unable to decipher response message Server = server_IP_address, User = user
4   109028  aaa bypassed for same-security traffic from ingress_ interface:source_address/source_port to egress_interface:dest_address/dest_port
4   109030  Autodetect ACL convert wildcard did not convert ACL access_list source | dest netmask netmask.
2

Using Raku (formerly known as Perl_6)

raku -ne 'put ($0,$1,$2).join("\t") if / \- (<[456]>) \- (\d+) \: \s+ (.*) /;'  

OR

raku -ne '.split(/<[:-]>/, 4).skip.join("\t").put;'  

The first answer generally follows @terdon's Perl5 code, while the second answer generally follows @glenn_jackman's Perl5 code.

Note in Raku:

  1. Raku captures start from $0,

  2. Raku doesn't make you guess when escaping characters within regexes: if it isn't <alnum> it needs to be escaped,

  3. Enumerated character classes are created in Raku with <[ ... ]>,

  4. A leading . (like .split) is short for $_., indicating the function call applies to the $_ topic variable.

Sample Input:

%ASA-4-105505: (Primary|Secondary) Failed to connect to peer unit peer-ip:port    
%ASA-4-105524: (Primary|Secondary) Transitioning to Negotiating state due to the presence of another Active HA unit    
%ASA-4-105553: (Primary|Secondary) Detected another Active HA unit    
%ASA-4-106023: Deny protocol src [interface_name:source_address/source_port] [([idfw_user|FQDN_string], sg_info)] dst interface_name:dest_address/dest_port [([idfw_user|FQDN_string], sg_info)] [type {string}, code {code}] by access_group acl_ID [0x8ed66b60, 0xf8852875]
%ASA-4-106027: Deny src [source address] dst [destination address] by access-group “access-list name”.
%ASA-4-106103: access-list acl_ID denied protocol for user username interface_name/source_address source_port interface_name/dest_address dest_port hit-cnt number first hit hash codes
%ASA-4-108004: action_class: action ESMTP req_resp from src_ifc:sip|sport to dest_ifc:dip|dport;further_info, page 1-23
%ASA-4-109017: User at IP_address exceeded auth proxy connection limit (max)
%ASA-4-109022: exceeded HTTPS proxy process limit
%ASA-4-109027: [aaa protocol] Unable to decipher response message Server = server_IP_address, User = user
%ASA-4-109028: aaa bypassed for same-security traffic from ingress_ interface:source_address/source_port to egress_interface:dest_address/dest_port
%ASA-4-109030: Autodetect ACL convert wildcard did not convert ACL access_list source | dest netmask netmask.

Sample Output:

4   105505   (Primary|Secondary) Failed to connect to peer unit peer-ip:port    
4   105524   (Primary|Secondary) Transitioning to Negotiating state due to the presence of another Active HA unit    
4   105553   (Primary|Secondary) Detected another Active HA unit    
4   106023   Deny protocol src [interface_name:source_address/source_port] [([idfw_user|FQDN_string], sg_info)] dst interface_name:dest_address/dest_port [([idfw_user|FQDN_string], sg_info)] [type {string}, code {code}] by access_group acl_ID [0x8ed66b60, 0xf8852875]
4   106027   Deny src [source address] dst [destination address] by access-group “access-list name”.
4   106103   access-list acl_ID denied protocol for user username interface_name/source_address source_port interface_name/dest_address dest_port hit-cnt number first hit hash codes
4   108004   action_class: action ESMTP req_resp from src_ifc:sip|sport to dest_ifc:dip|dport;further_info, page 1-23
4   109017   User at IP_address exceeded auth proxy connection limit (max)
4   109022   exceeded HTTPS proxy process limit
4   109027   [aaa protocol] Unable to decipher response message Server = server_IP_address, User = user
4   109028   aaa bypassed for same-security traffic from ingress_ interface:source_address/source_port to egress_interface:dest_address/dest_port
4   109030   Autodetect ACL convert wildcard did not convert ACL access_list source | dest netmask netmask.

https://raku.org

1

I was able to split on - as field separator and then, using index and substr functions as follows"

awk -F '-' '{i=index($3,":"); print $2 "\t" substr($3,1,i-1) "\t" substr($3,i+1)}'
1
  • 1
    This works for the most part, but messes up descriptions with a - in them Feb 28, 2022 at 18:39
1

You can use sed:

$ sed 's/^.\{5\}\([[:digit:]]\+\)-\([[:digit:]]\+\):[[:blank:]]*\(.*\)$/\1\t\2\t\3/' file 
4       105505  (Primary|Secondary) Failed to connect to peer unit peer-ip:port
4       105524  (Primary|Secondary) Transitioning to Negotiating state due to the presence of another Active HA unit
4       105553  (Primary|Secondary) Detected another Active HA unit
4       106023  Deny protocol src [interface_name:source_address/source_port] [([idfw_user|FQDN_string], sg_info)] dst interface_name:dest_address/dest_port [([idfw_user|FQDN_string], sg_info)] [type {string}, code {code}] by access_group acl_ID [0x8ed66b60, 0xf8852875]
4       106027  Deny src [source address] dst [destination address] by access-group “access-list name”.
4       106103  access-list acl_ID denied protocol for user username interface_name/source_address source_port interface_name/dest_address dest_port hit-cnt number first hit hash codes
4       108004  action_class: action ESMTP req_resp from src_ifc:sip|sport to dest_ifc:dip|dport;further_info, page 1-23
4       109017  User at IP_address exceeded auth proxy connection limit (max)
4       109022  exceeded HTTPS proxy process limit
4       109027  [aaa protocol] Unable to decipher response message Server = server_IP_address, User = user
4       109028  aaa bypassed for same-security traffic from ingress_ interface:source_address/source_port to egress_interface:dest_address/dest_port
4       109030  Autodetect ACL convert wildcard did not convert ACL access_list source | dest netmask netmask.
1

With GNU awk, we can use the 3-argument match function:

gawk -v OFS='\t' '
    match($0, /-([456])-([^:]+): (.*)/, m) {print m[1], m[2], m[3]}
' file

See String Functions in the manual


Or perl where we can set an upper limit for how many pieces the split function returns:

perl -ne '
    @fields = split(/[:-]/, $_, 4);
    shift @fields;
    print join "\t", @fields;
' file
0

Using sed

$ sed 's/[^0-9]*\([0-9]\)-\([^:]*\):/\1 \2/' file
4 105505 (Primary|Secondary) Failed to connect to peer unit peer-ip:port
4 105524 (Primary|Secondary) Transitioning to Negotiating state due to the presence of another Active HA unit
4 105553 (Primary|Secondary) Detected another Active HA unit
4 106023 Deny protocol src [interface_name:source_address/source_port] [([idfw_user|FQDN_string], sg_info)] dst interface_name:dest_address/dest_port [([idfw_user|FQDN_string], sg_info)] [type {string}, code {code}] by access_group acl_ID [0x8ed66b60, 0xf8852875]

Using awk

$ awk -F: 'BEGIN{OFS=""}{sub(/%ASA-/,"",$1);sub(/-/," ",$1)}1' file
4 105505 (Primary|Secondary) Failed to connect to peer unit peer-ipport
4 105524 (Primary|Secondary) Transitioning to Negotiating state due to the presence of another Active HA unit
4 105553 (Primary|Secondary) Detected another Active HA unit
4 106023 Deny protocol src [interface_namesource_address/source_port] [([idfw_user|FQDN_string], sg_info)] dst interface_namedest_address/dest_port [([idfw_user|FQDN_string], sg_info)] [type {string}, code {code}] by access_group acl_ID [0x8ed66b60, 0xf8852875]
0

You can use other awk functions to separate data. Perhaps:

awk '
NF {
    j = index ($0, ": ");
    split (substr ($0, 1, j - 1), V, "-");
    printf ("%s   %s %s\n", V[2], V[3], substr ($0, j + 2));
}' rawSyslog.txt

It assumes your input data is reliable, but ignores the blank lines. I used spaces for the output separators, but you may have intended tabs.

0

Split it in two chunk the first from the % character up to the : character, and the second anythings comes after. Then the first field will be splitted again at each - to get the digit and the id. The second chunk will be reassamble in the for loop if another occurence of the FS is encounterd. If you are sure that no occurence of FS can occur you can omi the for loop and just print a[2], a[3], $2.

awk -F': ' -v OFS='\t' '{
    split($1, a, "-")
    s = ""
    for(i = 2; i <= NF; i++) 
        s = s ? s FS $i : $i
    print a[2], a[3], s
}' test

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .