Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

Need some pointing in right direction on script to fetch and regex or sed.

Site24x7 provides a URL with a CSV list of their source IP's used for monitoring. (they also provide other formats, CSV seems the least messed up as their structure leaves a lot to be desired. https://www.site24x7.com/multi-location-web-site-monitoring.html )

Like so:

Country,City,IP Address External
Australia,Sydney,"101.0.67.53"
Australia,Melbourne,"125.214.65.59"
Belgium,Brussels,"87.238.165.164"
Brazil,São Paulo,"200.170.83.170"
Brazil,Rio de Janeiro,"201.20.20.237"
Canada,Toronto,"208.69.56.166,
208.69.56.171,
208.69.56.172 "
Canada,Montreal,"199.204.45.153,
199.204.45.154,
199.204.45.155,
199.204.45.156"

I need to save it as an Allow include file in apache. Like so:

Allow from \
72.5.230.111 \
72.5.230.65 \
72.5.230.84
share

2 Answers 2

up vote 1 down vote accepted
#!/bin/bash

webpage="https://www.site24x7.com/multi-location-web-site-monitoring.html"
csv=$(curl -s "$webpage" | grep 'title="CSV"' | \
    sed 's/^.*href="\(http[^ ]*\)".*$/\1/')

echo -e "Allow from \\"
curl -s "$csv" | egrep -o '[0-9.]{7,15}' | paste -s | sed 's/\t/ \\\n/g'
share
    
Looks Jedi. I'll test in the AM. Thanks, more than expected. –  8None1 Feb 10 '12 at 5:15

Quick and dirty and not very robust sed(1) one-liner:

echo 'Country,City,IP Address External
Australia,Sydney,"101.0.67.53"
Australia,Melbourne,"125.214.65.59"
Belgium,Brussels,"87.238.165.164"
Brazil,São Paulo,"200.170.83.170"
Brazil,Rio de Janeiro,"201.20.20.237"
Canada,Toronto,"208.69.56.166,
208.69.56.171,
208.69.56.172 "
Canada,Montreal,"199.204.45.153,
199.204.45.154,
199.204.45.155,
199.204.45.156"' | 
    sed -e 1d -e 's/"$//' -e 's/^[^"]*"/Allow from /' -e 's/,$/ \\/'

Output:

Allow from 101.0.67.53
Allow from 125.214.65.59
Allow from 87.238.165.164
Allow from 200.170.83.170
Allow from 201.20.20.237
Allow from 208.69.56.166 \
208.69.56.171 \
208.69.56.172 
Allow from 199.204.45.153 \
199.204.45.154 \
199.204.45.155 \
199.204.45.156

Assumptions:

  • Every IP address group is surrounded with the double quote (") character
  • The " character cannot occur anywhere else except to delimit IP groups.
  • A comma (,) on the end of a line means multiple IP addresses are continued on the next line.
share

This site is currently not accepting new answers.

Not the answer you're looking for? Browse other questions tagged .