2

I have more than a hundred firewall rules for many different customers in discrete files in a folder.

Suppose the filenames are:

1.wawa.com_firewall_rule

2.newa.com_firewall_rule

Each file contains the IPs of the customers. What I need is to extract the IPs for each customer from the file, and append it to a SINGLE new file in the following format:

wawa.com
192.168.1.1
newa.com
8.8.8.8
9.9.9.9

Right now, I have been able to extract the IPs and save them in new files. But I need to save all in a single file. (as a report) - see my code below

#!/bin/bash
cd /opt/zeus/zxtm/conf/rules/sba_filter/test
for f in *; do 
#echo $f > hosts.txt
grep -E -o "([0-9]{1,3}[\.]){3}[0-9]{1,3}" "$f" > "$f.txt"
done
5
  • try this loop: for i in *_firewall_rule; do echo ""$i"$(<"$i")"| sed s/_firewall_rule//g > ips.txt; done
    – malyy
    Jun 7, 2016 at 13:09
  • 1
    Well, isn't that simply grep -E -o "([0-9]{1,3}[\.]){3}[0-9]{1,3}" * > report? Jun 7, 2016 at 13:43
  • Is the host name in the file or do you want to extract it from the file name?
    – fd0
    Jun 7, 2016 at 13:48
  • 2
    Please edit your question and show us an example of your input files. What else is there apart from IPs?
    – terdon
    Jun 7, 2016 at 14:01
  • Also, tell us what you want the output to look like.  I know you show the output file “format”, but your question title says “separated by a line,” and I don’t understand how the output that you show reflects that requirement. Jun 11, 2016 at 3:07

3 Answers 3

3

Preserving your work, a possible solution could be:

#!/bin/bash
cd /home/prova
for f in *; do 
  basename $f '_firewall_rule' >> output.txt
  printf "\n" >> output.txt
  grep -E -o "([0-9]{1,3}[\.]){3}[0-9]{1,3}" "$f" | while read -r line ; do
    echo $line >> output.txt
    printf "\n" >> output.txt
  done
done

See optimization contributions in comments

4
  • 2
    I haven't profiled, but I would guess that it'd be more efficient, and certainly less error prone to put hte redirection into output.txt on the for loop itself instead of having each line open the file to append Jun 7, 2016 at 14:27
  • Instead of the while loop you could write it like this - grep -E -o "([0-9]{1,3}[\.]){3}[0-9]{1,3}" "$f" | xargs printf '%s\n\n'
    – fd0
    Jun 7, 2016 at 15:11
  • (1) You can prevent a lot of noise by saying for f in *_firewall_rule instead of for f in *. (2) You should always quote your shell variable references (e.g., "$f" and "$line") unless you have a good reason not to, and you’re sure you know what you’re doing. (3) I’m not convinced that you even need to add the blank lines that you’re adding — the OP’s example output doesn’t have blank lines. (I admit that the “separated by a line” qualifier in the question title is puzzling.) … (Cont’d) Jun 11, 2016 at 3:08
  • (Cont’d) …  (4) As fd0 points out, there are much better ways of adding blank lines to a data stream than while read -r line; do echo "$line"; printf "\n"; done.  I recommend sed 's/$/\n/'.  If you want to be cryptic, you could do paste -d'\n' - /dev/null. If you insist on using a while loop, you could at least trim it down to while read -r line; do printf "%s\n\n" "$line"; done. (Or you could use fd0’s suggestion.)  By the way, your current code (with echo "$line") could fail for filenames beginning with a dash (-). Jun 11, 2016 at 3:10
2
find your_dir -name "*firewall_rule"
-exec basename {} \;
-exec echo \;
-exec grep -P '(\d{1,3}\.){3}\d{1,3}' {} \;
-exec echo \;
> your_output_file

This will (line by line):

  • Find all files in your_dir ending in "firewall_rule"
  • Output a blank line
  • Output all [0-999].[0-999].[0-999].[0-999] IP addresses
  • Output another blank line
  • Send the output to a new file called your_output_file
1
  • nice solution Michael, just missing the new line between the ips
    – lese
    Jun 7, 2016 at 13:35
2

You could do it all with awk in a single command if you don't have too many files for the command line to handle:

awk --posix 'FNR==1 {f=FILENAME; sub("_firewall_rule", "", f); print f "\n"} /([0-9]{1,3}\.){3}[0-9]{1,3}/ {print $0 "\n"}' *_firewall_rule

for my version of gawk I needed the --posix to get the braces to work right in the regex.

How it works

Whenever a new file is started FNR will be 1, since it's the first record in the file, so we'll get the filename from the builtin FILENAME variable, strip off the _firewall_rule part and print what's left with a blank line

Then any line that looks roughly like an IPv4 address we'll print that line with an extra newline on it.

If you didn't want the extra blank lines between all the IPs and the names and such you can remove the \n parts of the print, and actually simplify the pattern part like so:

awk --posix 'FNR==1 {f=FILENAME; sub("_firewall_rule", "", f); print f} /([0-9]{1,3}\.){3}[0-9]{1,3}/ ' *_firewall_rule
2
  • This is probably the best answer. But note that the OP’s use of the -o option to grep raises the possibility that the input files have other text on the lines that contain the IP addresses — text that should be removed (and not output). Jun 11, 2016 at 3:12
  • 1
    /pattern/1; prints all lines; drop the 1 and preferably also the semicolon. @G-Man: -o allows multiple IPs and other stuff. {while(match($0,/patt/){print substr($0,RSTART,RLENGTH); $0=substr($0,RSTART+RLENGTH)}} can handle this, but I'd stick with grep. Jun 25, 2016 at 17:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.