0

I've written a Python module to dump objects. When using it with objects that hold binary data (e.g. numpy.ndarray) in a linux terminal (TERM=linux) though, printing out the data results in the terminal's font breaking - apparently, some characters are treated as terminal control sequences. In Windows, printing works fine, even in Cygwin's mintty terminal (it has TERM=xterm though).

The same happens when I cat a binary file.

I can fix that with reset, of course, but at the cost of losing the output, and it's generally inconvenient. While I do know that most, if not all control characters have alternative graphical representations in fonts (e.g. for CR, it's ♪).

So, is there some way to alter the raw stream to make the linux terminal treat special characters that were in it like literals? Basically, I wish to see something like this:

sample binary data printout

I'm primarily interested in a programmatic way (=what needs to be done from terminal's standpoint and an implementation in common system libraries if there is one); a way in shell would be a plus.

Python's repr() doesn't fit my needs: it expands any non-printable-ASCII characters and into variable-length sequences, including national letters, while the module's design goal is for the dump printout to be concise and readable.

Improve this question
3
  • @drewbenn "the module's design goal is for the dump printout to be concise and readable". The image is PC's (and, consequently, DOS' and Windows') native way to represent binary data, and it's as close to that goal as possible: it always represents one byte with one character (so I readily see how many bytes everything takes, thus offsets), has almost 1-to-1 glyph-to-value mapping, and allows to quickly identify any embedded readable data (even UTF-8 in it, while not readable, has a very distinctive pattern). Basically, if it's there and it does the job, why invent anything else?
    – ivan_pozdeev
    Dec 2, 2016 at 21:12
  • @drewbenn I used that module in IronPython, Windows and Cygwin CPython for years, and only in Linux proper, ran into problems. It's only natural I wish to get around them with as little difference in behaviour as possible.
    – ivan_pozdeev
    Dec 2, 2016 at 21:19
  • @drewbenn I can show its usual output and let you decide for yourself if xxd or hexdump-style output would be out of place there.
    – ivan_pozdeev
    Dec 2, 2016 at 21:23

2 Answers 2

Reset to default
3

The showconsolefont program can display 256 different (or 512 different...) glyphs at once on the Linux console. But it does this using a system call (which happens to only work for connections to the console device). Its manual page doesn't mention that.

But glyphs (which are used to display characters) aren't the same thing as characters. You would display a character by printing it on the terminal, and the terminal maps that to a glyph. There's no escape sequence which can tell the Linux console to treat control characters as printable.

For instance, showconsolefont doesn't actually write control characters for cells 0-31. It maps printable characters into the range 0-31 using (you guessed it) a system call.

Further reading:

Improve this answer
5
  • linux.die.net/man/4/console_codes is a reference of linux terminal control codes.
    – ivan_pozdeev
    Dec 2, 2016 at 21:43
  • If there had been a suitable escape sequence, I would have mentioned it.
    – Thomas Dickey
    Dec 2, 2016 at 21:44
  • Okay, I see that Linux doesn't use PC's native format due to its incompatible non-PC legacy, and, apparently, no one cares enough to suggest changing that. Despite having been developed for PC in the first place - that's what I cannot believe. How crazy one must be to throw that away without an adequate alternative?! Perhaps because since everything is open-source, the ability to readably and accurately represent unknown binary data isn't so critical here.
    – ivan_pozdeev
    Dec 2, 2016 at 22:34
  • The canonical "concise" format appears to be what less and vim use: ^<character>. Now, all that leaves is "implementation in common system libraries" - which library functionality encodes/displays characters like that? Is highlighting them in blue (vim) or white background (less) a part of it or their private logic?
    – ivan_pozdeev
    Dec 2, 2016 at 22:41
  • This is sounding like a new question. The original question was how to get the terminal to treat all characters as literals. Applications doing their own conversion is a different (and simpler) question.
    – Thomas Dickey
    Dec 2, 2016 at 22:49
0

I'm not aware of any way to change the terminal to accept all characters. The control characters are a feature of the terminal, and it's usually the duty of the program to pay attention to the terminal type, produce the right control characters for terminal features it wants to use, and escape any control characters it wants to print.

Information how to change a Python program to do this can be found for example in this stackoverflow question.

In the shell, you can use e.g. tr to convert control characters to other ASCII characters (though not unicode characters). See this question for alternatives that can use unicode characters.

Unicode defines a Control Picture group to display control characters, for example, carriage return is ␍. If never heard about ♪ representing CR, and if so, that's purely accidental in some font you happen to use.

And in the shell, you wouldn't just cat a binary file, but use hexdump -C or similar to examine it.

Improve this answer
12
  • 2
    Should mention cat -v (assuming that you have that option).
    – icarus
    Nov 30, 2016 at 7:41
  • @icarus: I actually didn't know about cat -v. Yes, that's also an option.
    – dirkt
    Nov 30, 2016 at 7:47
  • Really, I didn't think there's anyone out there who didn't see the signature "binary data" printout at least once to recognize it in description... Clarified the question.
    – ivan_pozdeev
    Nov 30, 2016 at 11:33
  • The problem is that there's no "standard way" to do "binary data printout". ASCII control characters are called "control characters" for a reason: They don't have any standard printable representation. If your font happens to have one, that's an accident. Same goes for the characters with high bit set, that's highly dependent on the encoding (and funny things happen with utf8). So you first have to explain which characters you'd actually like to see for those values, in which encoding. Then you can try to add some conversion to actually print them (possibly via utf8).
    – dirkt
    Nov 30, 2016 at 11:54
  • "ASCII control characters are called "control characters" for a reason: They don't have any standard printable representation" - oh, really? Built into and the same across all PCs in the world and throughout history - that's standard enough for me.
    – ivan_pozdeev
    Nov 30, 2016 at 11:58

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .