1

I have a script that sends out email to sysadmin group when a server is powered on or off. The email contains dump of some important logs like /var/log/messages, /var/log/secure, /var/log/boot.log . The /var/log/boot.log contains some special characters which when sent in email looks garbled as it contains some special formatting chars for coloring and tabs. How do I remove these special characters to make it readable?

I know I can use sed to remove the characters but I am looking for an easy and elegant solution.

(The /var/log/boot.log is from CentOS 6.x)

Here's a dump of my /var/log/boot/.log:

[root@vagrant ~]# cat -v /var/log/boot.log 
^[%G            Welcome to ^[[0;36mCentOS^[[0;39m ^M
Starting udev: ^[%G^[[60G[^[[0;32m  OK  ^[[0;39m]^M^M
Setting hostname vagrant:  ^[[60G[^[[0;32m  OK  ^[[0;39m]^M^M
Setting up Logical Volume Management:   5 logical volume(s) in volume group "vgdynamic" now active^M
  3 logical volume(s) in volume group "vgstatic" now active^M
^[[60G[^[[0;32m  OK  ^[[0;39m]^M^M
Checking filesystems^M
/dev/mapper/vgstatic-lvroot: clean, 8102/884736 files, 175959/3537920 blocks^M
/dev/sda1: clean, 44/32768 files, 17226/131072 blocks^M
/dev/mapper/vgdynamic-lvhome: clean, 10280/196608 files, 74141/786432 blocks^M
/dev/mapper/vgdynamic-lvopt: clean, 932/655360 files, 104046/2620416 blocks^M
/dev/mapper/vgstatic-lvtmp: clean, 12/131072 files, 25386/524288 blocks^M
/dev/mapper/vgstatic-lvusr: clean, 41785/262144 files, 236524/1048576 blocks^M
/dev/mapper/vgdynamic-lvvar: clean, 1989/393216 files, 93057/1572864 blocks^M
/dev/mapper/vgdynamic-lvvarlog: clean, 55/49152 files, 8030/196608 blocks^M
/dev/mapper/vgdynamic-lvaudit: clean, 14/65536 files, 73366/262144 blocks^M
^[[60G[^[[0;32m  OK  ^[[0;39m]^M^M
Remounting root filesystem in read-write mode:  ^[[60G[^[[0;32m  OK  ^[[0;39m]^M^M
Mounting local filesystems:  ^[[60G[^[[0;32m  OK  ^[[0;39m]^M^M
Enabling /etc/fstab swaps:  ^[[60G[^[[0;32m  OK  ^[[0;39m]^M^M
Entering non-interactive startup^M
Calling the system activity data collector (sadc)... ^M
Starting monitoring for VG vgdynamic:   5 logical volume(s) in volume group "vgdynamic" monitored^M
^[[60G[^[[0;32m  OK  ^[[0;39m]^M^M
Starting monitoring for VG vgstatic:   3 logical volume(s) in volume group "vgstatic" monitored^M
^[[60G[^[[0;32m  OK  ^[[0;39m]^M^M
Bringing up loopback interface:  ^[[60G[^[[0;32m  OK  ^[[0;39m]^M^M
Bringing up interface eth0:  ^M
Determining IP information for eth0... done.^M
^[[60G[^[[0;32m  OK  ^[[0;39m]^M^M
Starting auditd: ^[[60G[^[[0;32m  OK  ^[[0;39m]^M^M
Starting system logger: ^[[60G[^[[0;32m  OK  ^[[0;39m]^M^M
Starting lwsmd: ^[[60G[^[[0;32m  OK  ^[[0;39m]^M^[[60G[^[[0;32m  OK  ^[[0;39m]^M^M
Mounting filesystems:  ^[[60G[^[[0;32m  OK  ^[[0;39m]^M^M
Retrigger failed udev events^[[60G[^[[0;32m  OK  ^[[0;39m]^M^M
Starting the VirtualBox Guest Additions ^[[60G[^[[0;32m  OK  ^[[0;39m]^M^M
Starting VirtualBox Guest Addition service ^[[60G[^[[0;32m  OK  ^[[0;39m]^M^M
Starting sshd: ^[[60G[^[[0;32m  OK  ^[[0;39m]^M^M
Starting ntpd: ^[[60G[^[[0;32m  OK  ^[[0;39m]^M^M
Starting crond: ^[[60G[^[[0;32m  OK  ^[[0;39m]^M^M
Starting atd: ^[[60G[^[[0;32m  OK  ^[[0;39m]^M^M
[root@vagrant ~]# 
  • Is sed not easy or elegant? – jimmij Oct 26 '14 at 4:51
  • Maybe there is a command line utility that automagically removes the control chars? – GMaster Oct 26 '14 at 4:52
  • There is [:cntrl:] pattern and indeed sed 's/[[:cntrl:]]//g' file1 > file2 is first what comes to my mind. You can use also tr, awk, pure bash... but hard to judge which tool is more easy or elegant. – jimmij Oct 26 '14 at 4:59
  • 1
    Oh, from your update I see, that the problem is not with control characters but with characters which are interpreted as control by shell as \e[0m (colors), although I'm not sure if ^M is control or literal in you log. – jimmij Oct 26 '14 at 5:02
  • ^M is control char in the log. I tried your sed, it does remove the control chars but not the other bits like [60G[[0;32m – GMaster Oct 26 '14 at 5:11
1

You should start with the more complex (longer) patterns first, if you start with only the control characters, the remaining might match patterns in the normal text as well. If speed is not the highest priority, split the processing over multiple statements, that way they are easier to debug and you can more easily control the order of removal.

The colour escape sequences start with ^[ and end in m. You can remove them using sed

sed -r "s/\x1B\[([0-9]{1,2}(;[0-9]{1,2})?)?[m//g"

Other patterns start with ^[ and end in G (position cursor):

sed -r "s/\x1B\[.*G//g"

Any remaining control characters should go with:

sed 's/[[:cntrl:]]//g' 

Combined into:

sed -r "s/\x1B\[([0-9]{1,2}(;[0-9]{1,2})?)?m//g" /var/log/boot.log | \
   sed -r "s/\x1B\[.*G//g" | \
   sed 's/[[:cntrl:]]//g'

¹ Whether that is elegant or elephant is in the eye of the beholder.

  • Well, it did not quite get what I wanted: [root@vagrant ~]# sed -r "s/\x1B\[([0-9]{1,2}(;[0-9]{1,2})?)?[m|K]//g" /var/log/boot.log > /tmp/dump [root@vagrant ~]# cat -v /tmp/dump ^[%G Welcome to CentOS ^M Starting udev: ^[%G^[[60G[ OK ]^M^M Setting hostname vagrant: ^[[60G[ OK ]^M^M – GMaster Oct 26 '14 at 6:17
  • Add G and plain cntrl: sed -r "s/\x1B\[([0-9]{1,2}(;[0-9]{1,2})?)?[m|K|G]//g;s/[[:cntrl:]]//g". – jimmij Oct 26 '14 at 7:27
  • A liitle be shorter sed -E 's/\^\[[^ m]+[mGK]\]?|[[:cntrl:]]//g' (not sure about K, but you have included). Explanation:start with ^[ then some any symbols exept m and ` ` and finish with m or G or K and possible ]. Or just ^M. It is working with example but may be subject to correction. – Costas Oct 26 '14 at 8:35
  • But the question: if ^M is control char so ^[ shoul be control char too? – Costas Oct 26 '14 at 8:48
  • I finally made a not so elegant solution: sed -r "s/\x1B\[([0-9]{1,2}(;[0-9]{1,2})?)?[m|K]//g" /var/log/boot.log | sed 's/[[:cntrl:]]//g' | sed -r 's/(\[60G\[|\]$|%G)//g' combining from the comments made on this page – GMaster Oct 26 '14 at 9:39
0

Maybe the command dos2unix is what you are looking for.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.