i have a large AWS ELB logfile from which i want to remove the private ip, for example:

2018-06-03T23:59:57.000Z, [03/Jun/2018:23:59:57 +0000] "GET http://www.domain0.com/latest-news/dubai-duty-free-sales-rise-11-q1-12-04-2018/ HTTP/1.1" 200 16669 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
2018-06-03T23:59:58.000Z, [03/Jun/2018:23:59:58 +0000] "GET http://www.domain3.com/feed HTTP/1.1" 200 17144 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.95 Safari/537.11"
2018-06-03T23:59:58.000Z, [03/Jun/2018:23:59:58 +0000] "GET http://www.domain1.com/feed/ HTTP/1.1" 200 18532 "https://www.domain1.com/feed" "Mozilla/5.0 (compatible; SemrushBot/1.0~bm; +http://www.semrush.com/bot.html)"
2018-06-03T23:59:58.000Z, [03/Jun/2018:23:59:58 +0000] "GET http://www.domain.com/blogs/gadget-master/general/brightsparks-trophies-make-mark-2017-05/https/blogs/distribution-world/page/9/ HTTP/1.1" 200 16339 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
2018-06-03T23:59:59.000Z, [03/Jun/2018:23:59:59 +0000] "GET http://www.domain.com/electro-ramblings/tag/blog/feed/ HTTP/1.1" 200 1664 "-" "Mozilla/5.0 (X11; Linux i686) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/30.0.1599.66 Safari/537.36"`, "\n")

gets me the regex

but when i run it

cat logs.txt | sed '/T/s/T/:/1' | sed -E 's/,10.0/ /g'

but this only returns

2018-06-03:23:59:57.000Z .1.81 

how do i remove the private ip with the , from the file using regex in sed?

any advice is much appreciated

3 Answers 3


As the requirement was to remove "private" IPs (10...) only, the regex given by ender.qa could be refined to

sed -E '/T/s/T/:/1; s/,10\.([0-9]{1,3}.){2}[0-9]{1,3}//g' file

or, as that IP would not contain spaces, even shorter:

sed -E '/T/s/T/:/1; s/,10\.[^ ]*//g' file

It looks like you want your 'data before':

2018-06-03T23:59:57.000Z, [03/Jun/2018:23:59:57 +0000] ...

to be 'data after':

2018-06-03:23:59:57.000Z [03/Jun/2018:23:59:57 +0000] ...

Is that correct?

If yes:

cat logs.txt | sed '/T/s/T/:/1' | sed -e 's/,.*\[//g'

Also, you don't need the cat:

sed '/T/s/T/:/1' logs.txt | sed -e 's/,.*\[/ \[/g'

Basically, you are removing any characters (.*) between the comma (,) and the left bracket ([), which needs to be escaped.

If you only want to remove the IP address

sed '/T/s/T/:/1' logs.txt | sed -re 's/,([0-9]{1,3}\.){3}[0-9]{1,3}//g'


sed '/T/s/T/:/1' logs.txt | sed -e 's/,\([0-9]\{1,3\}\.\)\{3\}[0-9]\{1,3\}//g'

Reference: https://stackoverflow.com/a/14928879/4635198


Not quite what you asked for, but this shows how to use grepcidr to filter out (remove) lines containing IP addresses in the private address range in the file file:

grepcidr -v '' file

The same thing, but using CIDR network block syntax:

grepcidr -v '' file

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .