3

I have a unix csv file as pipeline "|" separator . But while I am opening in vi editor there are some extra characters are coming as ~G . But while I am doing cat , I could not see any ~G characters .

enter image description here

453136~G|OORAHASS0343136~G|Generic Box Access~G|NMBLDD~G|/shelf=0/slot=1/port=7~G|20Mbit/s~G|80Mbit/s~G|IS

How to remove ~G characters .

I have already tried below steps but no luck .

sed -e 's/[^ -~]//g' file_in > file_out

or

grep -c '[^ -~]' file_in 

or

sed -i 's/\~H//g;s/\~G//g' file_in
10

cat -e rendering them as M-^G suggests they are 0x87 bytes (0207 in octal). As its documentation1 says, vim renders byte 0x87 as ~G when in locales using single-byte charsets or when the encoding is Unicode and the ESA character is encoded as a valid UTF-8 multibyte sequence, and renders the byte as <87> when the encoding option is Unicode and the character does not form part of a valid UTF-8 sequence. (It renders ^G for 0x7, the ASCII BEL character.)

That's G (0x47 in ASCII) with bit 7 (meta) set to 1 and bit 6 set to 0 (control). That byte doesn't form a valid character in UTF-8 and is typically the code for a control character (ESA) in the C1 set in ISO8859-x charsets.

To get rid of it, you can do:

tr -d '\207' < file > file.new

With GNU sed and a shell like ksh93/zsh/bash with support for $'...':

sed -i $'s/\207//g' file

Your

sed 's/[^ -~]//g'

would have done it, but only in the C locale. What character ranges match in other locales is pretty random. So:

LC_ALL=C sed 's/[^ -~]//g' < file > file.new

(note that it would delete all other control characters including tabulation and CR (but not LF) and non-ASCII characters).

0x87 is ‡ in the windows-1252 character set (sometimes improperly refereed to as latin1 or iso8859-1).

If you wanted those 0x87 to be converted to ‡ (because for instance those files come from the Windows world and that's what those 0x87 were intended to be) in your locale's charset (assuming it has such a character), you could use:

iconv -f windows-1252 < file > file.new

1 Bram Moolenaar (2011-03-22). 'isprint'. "options". VIM Reference Manual.

  • Yes its working now . thanks for sharing the knowledge gives more inputs on shell . – chinmaya Das Jun 7 '18 at 14:40
  • @JdeBP, thanks for the edit, however note that I still see <87> when editing a UTF-8 file containing the UTF-8 encoding of U+0087 (0xc2 0x87) in a locale using UTF-8 as its charset which seems to contradict the text you added. – Stéphane Chazelas Jun 7 '18 at 18:54
  • I was trying to provide a rather more explicit explanation than the rather terse one in the manual. It might have gained errors in translation. (-: Note that the manual says that this depends from the encoding, which it in turn explains isn't necessarily the locale. If the manual turns out to be wrong, please let the VIM and NeoVIM people know. – JdeBP Jun 8 '18 at 14:02
2

Using tools from coreutils only:

# Generate a test file
printf 'head\207\nsome text\207\nnew line' > /tmp/test.cchar

# And filter with tr
tr -d "\207" < /tmp/test.cchar > /tmp/test.filtered 
  • You, too, have got the wrong control character. – JdeBP Jun 7 '18 at 14:35
0

The ~Gis a bell character which is ASCII 007. An easy way to remove it and update your file, inplace is:

perl -pi -e 's/\007//' file_in

See also the ASCII table

A more convoluted sed solution is to use shell substitution:

sed -i 's/'`echo "\007"`'//' file_in

When using cat, add the -e option to show non-printing characters.

  • HI I have used cat -e then I can see the junk character as below cat -e abc.csv 592619M-^G|OORAHASS0343136-^G|Generic Ethernet AccessM-^G|NAAMINM-^G|/shelf=0/slot=3/port=11M-^G|2Mbit/sM-^G|40Mbit/sM-^G|IS$ but the above commands are not able to remove those characters :( – chinmaya Das Jun 7 '18 at 13:12
  • 1
    Note that not all echo implementation will expand that \007. Some need a -e for that. printf '\7' or printf '\a' would be more portable. See also sed -i $'s/\7//g' with ksh93/zsh/bash and a few other shells. – Stéphane Chazelas Jun 7 '18 at 13:25
  • 1
    No, that is not how vi/VIM/NeoVIM display the BEL character. It is how they display the ESA character. See :help isprint in VIM or NeoVIM. – JdeBP Jun 7 '18 at 14:22
  • @JdeBP what was actually displayed was not clear in the original OP question. It wasn't until the OP ran cat -e that the problem became clearer. – JRFerguson Jun 7 '18 at 14:44
  • That would be a reason for not realizing that it is a control character, or exactly which control character. It isn't a reason for an incorrect statement about what vi/VIM/NeoVIM display for the BEL character, which isn't dependent from information in the question. – JdeBP Jun 7 '18 at 15:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.