0

I've written a Python module to dump objects. When using it with objects that hold binary data (e.g. numpy.ndarray) in a linux terminal (TERM=linux) though, printing out the data results in the terminal's font breaking - apparently, some characters are treated as terminal control sequences. In Windows, printing works fine, even in Cygwin's mintty terminal (it has TERM=xterm though).

The same happens when I cat a binary file.

I can fix that with reset, of course, but at the cost of losing the output, and it's generally inconvenient. While I do know that most, if not all control characters have alternative graphical representations in fonts (e.g. for CR, it's ).

So, is there some way to alter the raw stream to make the linux terminal treat special characters that were in it like literals? Basically, I wish to see something like this:

sample binary data printout

I'm primarily interested in a programmatic way (=what needs to be done from terminal's standpoint and an implementation in common system libraries if there is one); a way in shell would be a plus.

Python's repr() doesn't fit my needs: it expands any non-printable-ASCII characters and into variable-length sequences, including national letters, while the module's design goal is for the dump printout to be concise and readable.

  • It's pretty different from your screenshot but the classic method is a hex editor view, something like hexdump -C, to show the actual data bytes as well as the printable ASCII characters off to the side. Would a redesign like that be close enough to what you want (or maybe you can explain what problem you are solving by displaying the non-printable ASCII characters like that)? – drewbenn Dec 2 '16 at 20:38
  • @drewbenn "the module's design goal is for the dump printout to be concise and readable". The image is PC's (and, consequently, DOS' and Windows') native way to represent binary data, and it's as close to that goal as possible: it always represents one byte with one character (so I readily see how many bytes everything takes, thus offsets), has almost 1-to-1 glyph-to-value mapping, and allows to quickly identify any embedded readable data (even UTF-8 in it, while not readable, has a very distinctive pattern). Basically, if it's there and it does the job, why invent anything else? – ivan_pozdeev Dec 2 '16 at 21:12
  • @drewbenn I used that module in IronPython, Windows and Cygwin CPython for years, and only in Linux proper, ran into problems. It's only natural I wish to get around them with as little difference in behaviour as possible. – ivan_pozdeev Dec 2 '16 at 21:19
  • @drewbenn I can show its usual output and let you decide for yourself if xxd or hexdump-style output would be out of place there. – ivan_pozdeev Dec 2 '16 at 21:23
3

The showconsolefont program can display 256 different (or 512 different...) glyphs at once on the Linux console. But it does this using a system call (which happens to only work for connections to the console device). Its manual page doesn't mention that.

But glyphs (which are used to display characters) aren't the same thing as characters. You would display a character by printing it on the terminal, and the terminal maps that to a glyph. There's no escape sequence which can tell the Linux console to treat control characters as printable.

For instance, showconsolefont doesn't actually write control characters for cells 0-31. It maps printable characters into the range 0-31 using (you guessed it) a system call.

Further reading:

  • linux.die.net/man/4/console_codes is a reference of linux terminal control codes. – ivan_pozdeev Dec 2 '16 at 21:43
  • If there had been a suitable escape sequence, I would have mentioned it. – Thomas Dickey Dec 2 '16 at 21:44
  • Okay, I see that Linux doesn't use PC's native format due to its incompatible non-PC legacy, and, apparently, no one cares enough to suggest changing that. Despite having been developed for PC in the first place - that's what I cannot believe. How crazy one must be to throw that away without an adequate alternative?! Perhaps because since everything is open-source, the ability to readably and accurately represent unknown binary data isn't so critical here. – ivan_pozdeev Dec 2 '16 at 22:34
  • The canonical "concise" format appears to be what less and vim use: ^<character>. Now, all that leaves is "implementation in common system libraries" - which library functionality encodes/displays characters like that? Is highlighting them in blue (vim) or white background (less) a part of it or their private logic? – ivan_pozdeev Dec 2 '16 at 22:41
  • This is sounding like a new question. The original question was how to get the terminal to treat all characters as literals. Applications doing their own conversion is a different (and simpler) question. – Thomas Dickey Dec 2 '16 at 22:49
0

I'm not aware of any way to change the terminal to accept all characters. The control characters are a feature of the terminal, and it's usually the duty of the program to pay attention to the terminal type, produce the right control characters for terminal features it wants to use, and escape any control characters it wants to print.

Information how to change a Python program to do this can be found for example in this stackoverflow question.

In the shell, you can use e.g. tr to convert control characters to other ASCII characters (though not unicode characters). See this question for alternatives that can use unicode characters.

Unicode defines a Control Picture group to display control characters, for example, carriage return is . If never heard about representing CR, and if so, that's purely accidental in some font you happen to use.

And in the shell, you wouldn't just cat a binary file, but use hexdump -C or similar to examine it.

  • 2
    Should mention cat -v (assuming that you have that option). – icarus Nov 30 '16 at 7:41
  • @icarus: I actually didn't know about cat -v. Yes, that's also an option. – dirkt Nov 30 '16 at 7:47
  • Really, I didn't think there's anyone out there who didn't see the signature "binary data" printout at least once to recognize it in description... Clarified the question. – ivan_pozdeev Nov 30 '16 at 11:33
  • The problem is that there's no "standard way" to do "binary data printout". ASCII control characters are called "control characters" for a reason: They don't have any standard printable representation. If your font happens to have one, that's an accident. Same goes for the characters with high bit set, that's highly dependent on the encoding (and funny things happen with utf8). So you first have to explain which characters you'd actually like to see for those values, in which encoding. Then you can try to add some conversion to actually print them (possibly via utf8). – dirkt Nov 30 '16 at 11:54
  • "ASCII control characters are called "control characters" for a reason: They don't have any standard printable representation" - oh, really? Built into and the same across all PCs in the world and throughout history - that's standard enough for me. – ivan_pozdeev Nov 30 '16 at 11:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.