3

I have read nearly every answer about this topic on this website or Stackoverflow but didn't manage to solve the issue below.

When I copy the text from a PDF file and paste it into a text file file.txt) , the text looks normal but when I use cat command:

cat -v file.txt

The output is:

vbox = None
M-BM- M-BM- M-BM- M-BM- def __init__(self, title="Error!", parent=None,
M-BM- M-BM- M-BM- M-BM- M-BM- M-BM- M-BM- M-BM- M-BM- M-BM- M-BM- M-BM- M-BM- M-BM- M-BM- M-BM- M-BM- M-BM- flags=Gtk.DialogFlags.MODAL, buttons=("NO",
M-BM- M-BM- M-BM- M-BM- M-BM- M-BM- M-BM- M-BM- M-BM- M-BM- M-BM- M-BM- M-BM- M-BM- M-BM- M-BM- M-BM- M-BM- Gtk.ResponseType.NO, "_YES",
M-BM- M-BM- M-BM- M-BM- M-BM- M-BM- M-BM- M-BM- M-BM- M-BM- M-BM- M-BM- M-BM- M-BM- M-BM- M-BM- M-BM- M-BM- M-BM- M-BM- M-BM- M-BM- M-BM- M-BM- M-BM- M-BM- M-BM- Gtk.ResponseType.YES)):
M-BM- M-BM- M-BM- M-BM- M-BM- M-BM- M-BM- M-BM- super().__init__(title=title, parent=parent, flags=flags,
M-BM- M-BM- M-BM- M-BM- M-BM- M-BM- M-BM- M-BM- M-BM- M-BM- M-BM- M-BM- M-BM- M-BM- M-BM- M-BM- M-BM- M-BM- M-BM- M-BM- M-BM- M-BM- M-BM- M-BM- M-BM- buttons=buttons)
M-BM- M-BM- M-BM- M-BM- M-BM- M-BM- M-BM- M-BM- self.vbox = self.get_content_area()
M-BM- M-BM- M-BM- M-BM- M-BM- M-BM- M-BM- M-BM- self.hbox = Gtk.Box(orientation=Gtk.Orientation.HORIZONTAL,
spacing=5)
M-BM- M-BM- M-BM- M-BM- M-BM- M-BM- M-BM- M-BM- icon_theme = Gtk.IconTheme.get_default()
M-BM- M-BM- M-BM- M-BM- M-BM- M-BM- M-BM- M-BM- icon = icon_theme.load_icon("dialog-question", 48,
M-BM- M-BM- M-BM- M-BM- M-BM- M-BM- M-BM- M-BM- M-BM- M-BM- M-BM- M-BM- M-BM- M-BM- M-BM- M-BM- M-BM- M-BM- M-BM- M-BM- M-BM- M-BM- M-BM- M-BM- M-BM- M-BM- M-BM- M-BM- M-BM- M-BM- M-BM- M-BM- M-BM- M-BM- M-BM- M-BM- Gtk.IconLookupFlags.FORCE_SVG)
M-BM- M-BM- M-BM- M-BM- M-BM- M-BM- M-BM- M-BM- image = Gtk.Image.new_from_pixbuf(icon)
M-BM- M-BM- M-BM- M-BM- M-BM- M-BM- M-BM- M-BM- self.hbox.pack_start(image, False, False, 5)
M-BM- M-BM- M-BM- M-BM- M-BM- M-BM- M-BM- M-BM- self.vbox.add(self.hbox)
M-BM- M-BM- M-BM- M-BM- def set_message(self, message, add_msg=None):
M-BM- M-BM- M-BM- M-BM- M-BM- M-BM- M-BM- M-BM- self.hbox.pack_start(Gtk.Label(message), False, False, 5)
M-BM- M-BM- M-BM- M-BM- M-BM- M-BM- M-BM- M-BM- if add_msg != None:

Or when I use bat command:

bat -A file.txt

The output is:

vbox•=•None␊
\u{a0}\u{a0}\u{a0}\u{a0}def•__init__(self,•title="Error!",•parent=None,␊
\u{a0}\u{a0}\u{a0}\u{a0}\u{a0}\u{a0}\u{a0}\u{a0}\u{a0}\u{a0}\u{a0}\u{a0}\u{a0}\u{a0}\u{a0}\u{a0}\u{a0}\u{a0}flags=Gtk.DialogFlags.MODAL,•buttons=("NO",␊
\u{a0}\u{a0}\u{a0}\u{a0}\u{a0}\u{a0}\u{a0}\u{a0}\u{a0}\u{a0}\u{a0}\u{a0}\u{a0}\u{a0}\u{a0}\u{a0}\u{a0}\u{a0}Gtk.ResponseType.NO,•"_YES",␊
\u{a0}\u{a0}\u{a0}\u{a0}\u{a0}\u{a0}\u{a0}\u{a0}\u{a0}\u{a0}\u{a0}\u{a0}\u{a0}\u{a0}\u{a0}\u{a0}\u{a0}\u{a0}\u{a0}\u{a0}\u{a0}\u{a0}\u{a0}\u{a0}\u{a0}\u{a0}\u{a0}Gtk.ResponseType.YES)):␊
\u{a0}\u{a0}\u{a0}\u{a0}\u{a0}\u{a0}\u{a0}\u{a0}super().__init__(title=title,•parent=parent,•flags=flags,␊
\u{a0}\u{a0}\u{a0}\u{a0}\u{a0}\u{a0}\u{a0}\u{a0}\u{a0}\u{a0}\u{a0}\u{a0}\u{a0}\u{a0}\u{a0}\u{a0}\u{a0}\u{a0}\u{a0}\u{a0}\u{a0}\u{a0}\u{a0}\u{a0}\u{a0}buttons=buttons)␊
\u{a0}\u{a0}\u{a0}\u{a0}\u{a0}\u{a0}\u{a0}\u{a0}self.vbox•=•self.get_content_area()␊
\u{a0}\u{a0}\u{a0}\u{a0}\u{a0}\u{a0}\u{a0}\u{a0}self.hbox•=•Gtk.Box(orientation=Gtk.Orientation.HORIZONTAL,␊
spacing=5)␊
\u{a0}\u{a0}\u{a0}\u{a0}\u{a0}\u{a0}\u{a0}\u{a0}icon_theme•=•Gtk.IconTheme.get_default()␊
\u{a0}\u{a0}\u{a0}\u{a0}\u{a0}\u{a0}\u{a0}\u{a0}icon•=•icon_theme.load_icon("dialog-question",•48,␊
\u{a0}\u{a0}\u{a0}\u{a0}\u{a0}\u{a0}\u{a0}\u{a0}\u{a0}\u{a0}\u{a0}\u{a0}\u{a0}\u{a0}\u{a0}\u{a0}\u{a0}\u{a0}\u{a0}\u{a0}\u{a0}\u{a0}\u{a0}\u{a0}\u{a0}\u{a0}\u{a0}\u{a0}\u{a0}\u{a0}\u{a0}\u{a0}\u{a0}\u{a0}\u{a0}\u{a0}Gtk.IconLookupFlags.FORCE_SVG)␊
\u{a0}\u{a0}\u{a0}\u{a0}\u{a0}\u{a0}\u{a0}\u{a0}image•=•Gtk.Image.new_from_pixbuf(icon)␊
\u{a0}\u{a0}\u{a0}\u{a0}\u{a0}\u{a0}\u{a0}\u{a0}self.hbox.pack_start(image,•False,•False,•5)␊
\u{a0}\u{a0}\u{a0}\u{a0}\u{a0}\u{a0}\u{a0}\u{a0}self.vbox.add(self.hbox)␊
\u{a0}\u{a0}\u{a0}\u{a0}def•set_message(self,•message,•add_msg=None):␊
\u{a0}\u{a0}\u{a0}\u{a0}\u{a0}\u{a0}\u{a0}\u{a0}self.hbox.pack_start(Gtk.Label(message),•False,•False,•5)␊
\u{a0}\u{a0}\u{a0}\u{a0}\u{a0}\u{a0}\u{a0}\u{a0}if•add_msg•!=•None:␊

On Visual studio code, when I hover on those characters, I get:

The character U+00a0 is not a basic ASCII character.

enter image description here

How can I use sed command to replace those characters with normal "space" characters?

2 Answers 2

5

Looks like the UTF-8 encoding of the non-breaking space (U+00A0), the bytes are c2 a0 in hex. Something like sed -e 's/\xc2\xa0/ /g' in GNU sed should work to replace them with regular spaces.

7
  • I have tried sed -e 's/[[:cntrl:]]/ /g' file.txt but I wonder why it doesn't work.
    – Amazigh_05
    Jan 15, 2022 at 20:08
  • 1
    @OK-Validation cntrl doesn't include the necessary characters. See regular-expressions.info/posixbrackets.html Jan 15, 2022 at 20:25
  • 2
    You could probably use [^[:graph:][:blank:]] to match any non-printing, non-space, non-tab, character. But I don't know how it plays with locales. Matching specific bytes is probably better, but do you need LC_ALL=C?
    – Kusalananda
    Jan 16, 2022 at 10:02
  • 1
    @SebMa if you feed UTF-8 text to that sed running in the C locale, it would trash any non-ASCII character, starting with e.g. é and ä. That would be a pretty big problem for a few languages.
    – ilkkachu
    May 18, 2022 at 21:37
  • 1
    @SebMa, yeah. The non-breaking space is a bit hard to catch with the character classes anyway, it's in [:punct:] along with :-,. etc. on GNU, and (so I hear) in [:blank:] along with the space on BSDs. Changing all of [:blank:] to spaces might make sense, but trashing punctuation doesn't seem too useful. Then of course there's piles of other Unicode space-like characters too (but they're more likely to be in [:blank:]).
    – ilkkachu
    May 18, 2022 at 22:00
1

Using Raku (formerly known as Perl_6)

raku -pe 's:g/\c[NO-BREAK SPACE]/ /;'  file.txt

Here are solutions coded in Raku, a member of the Perl-family of programming languages. An advantage of Raku is high-level support for Unicode out-of-the-box, no external libraries/packages/modules necessary.

Above I just use the 'official' Unicode name for &nbsp, which can be entered into the familiar s/// substitution operator as \c[NO-BREAK SPACE] (see https://www.unicode.org/charts/PDF/U0080.pdf).

But the \c[NBSP] name-alias below also works:

raku -pe 's:g/\c[NBSP]/ /;'  file.txt

Finally, entering hex also works:

raku -pe 's:g/\x00A0/ /;' file.txt

For further information on Unicode support in Raku, see below.

https://www.codesections.com/blog/raku-unicode/
https://docs.raku.org/language/unicode

2
  • Is \x00A0 the UTF-16 equivalent of \xC2\xA0 in UTF-8 ?
    – SebMa
    May 19, 2022 at 7:44
  • @SebMa Using Raku, calling .uniname on \xA0 returns NO-BREAK SPACE, same as \x00A0. And calling .uniname on either \xC2 or \x00C2 returns LATIN CAPITAL LETTER A WITH CIRCUMFLEX. May 27, 2022 at 6:19

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .