A shell-like environment for binary processing

Question

This question came to me a couple of times before, now in response to the question Loop through binary data chunks from stdin in Bash Answers given in https://stackoverflow.com/questions/993434/what-language-is-to-binary-as-perl-is-to-text were also not satisfactory.

I'm looking for a scripting environment suited specifically to handle I/O with binary files. I know I can use one of the fully-fledged programming languages (c/Python/...) but they have an enormous initialization and coding overhead (allocation and fread/fwrite in c, bitstrings in Python...) not to mention they are less suited for scripting (calling other applications from it). Perl is no better with its unpack functions, string-oriented operation and goofy syntax.

Something like od, but as a language.

What I expect:

1. set or change endianness with a single switch/command.
2. simple specification of requested type (something like extending bash read var with int32 var, float var etc.).
3. handling of binary through pipes, skipping of specified number of bytes.
4. standard scripting flow control (for/if/...) that we are used to.

I'd like to process raw data (photography, scientific data, unknown and poorly documented formats) with the same ease and insight that you get when inspecting ASCII files. I'm using c now, but it's not optimal for ad-hoc scripting, and can't be interactive.

Does anyone know a tool like that? No clicky GUI software, please, it needs to work over ssh, from other scripts and so on. "Does not exist" is an acceptable, but depressing answer.

Answer 1

I do have the exact same problem than you for years as well.

For simple non-interactive uses, I like to use the binary block editor BBE. BBE is to binary as SED is to text, including its archaic syntax and simplicity, however, it has a lot of features missing from what I often need, so I have to combine it with other tools. So, BBE is only a partial solution. Also note that BBE hasn't had any updates or improvements for years.

Of course one can use xxd before and xxd -r after editing the data with text-based tools, but that won't work when the data in question is large and random access is required, for example when processing block devices.

(Note: For Windows, there is at least the costly, proprietary WinHex scripting language, but that won't get us anywhere.)

For more complicated binary editing, I usually fall back to Python as well, even though it sometimes is too slow for large files, which is it's main drawback. I hope Pyston (Python employing LLVM to compile to optimized machine code) will someday mature enough to be usable, or even better, someone will design and implement a free compact, fast and versatile binary processing scripting language, which AFAIK doesn't exist for U*IX like systems yet.

UPDATE

I also happen to use the homebrew, open source Intel x86 assembler flat assembler, or fasm for short, that evolved into much more than just an assembler.

It has a powerful, textblock-based macro preprocessor (itself a turing complete language) with a syntax in the tradition of the borland turbo assembler macro language, but much more advanced.

Also, it has a data manipulation language, which allows to binary include arbitrary files, do all kinds of binary and arithmetic manipulation on it (integer only) at "compile time" and write the result into an output file. This data manipulation language has control strutures and is also turing complete.

It is much easier to use than writing a program that does some binary manipulation in C and probably even in python. Plus, it loads blindingly fast, as it is a small sized executable with almost no external dependecies (There are 2 versions: either it only requires libc or it can run as a static executable directly on the Linux kernel ABI).

It does have some ruff edges, like

1. not supporting concurrency

2. being writting in 32 bit x86 assembly (works on x86_64 though), you probably need qemu or a similar emulator if you want to run it on anything else than x86 or x86_64

3. it's powerful macro preprocessor language is turing complete, this means you better have some experience with languages like Lisp, Haskell, XSLT, or probably M4 would be the best choice.

4. all data that is to be written into the output file are performed in a "flat" buffer in memory, and this buffer can grow but not shrink until the output file has been written and fasm terminated. This means that one can only generate files at most as large as you have main memory left in a single run of fasm.

5. data can only be written into a single output file for each run of fasm

6. yeah, it is homebrew, a really neat and clever one though

Answer 2

You don't necessarily have to "make peace" with Perl's unpack... one of the great things about perl is how you can abuse the parser and symbol table to make your own language, in a custom package.

Is this basically what you're looking for?

use MyBinLib;
my $struct= struct(
  pack => 8,
  size => 400,
  fields => [int32('foo','bar','baz'), float32('x1','x2','x3','x4'), int8, int8, int16('z')]
);
while (my $rec= $struct->read(<STDIN>)) {
  printf "x1 = %d, x2 = $d\n", $rec->x1, $rec->x2;
}

The exercise then is to learn enough perl to write the MyBinLib package. Ask in a Perl forum and people would probably be happy to help.

Answer 3

Have you come across beav it has macros but I couldn't find scripting,

apt-cache show beav extract :

With beav, you can edit a file in HEX, ASCII, EBCDIC, OCTAL, DECIMAL, and BINARY. You can display but not edit data in FLOAT mode. You can search or search and replace in any of these modes. Data can be displayed in BYTE, WORD, or DOUBLE WORD formats. While displaying WORDS or DOUBLE WORDS the data can be displayed in INTEL's or MOTOROLA's byte ordering. Data of any length can be inserted at any point in the file. The source of this data can be the keyboard, another buffer, or a file. Any data that is being displayed can be sent to a printer in the displayed format. Files that are bigger than memory can be handled.

Then there's xxd which converts to/from binary/ascii display mode and could be combined together with sed or vi, but hasn't got the byte swapping feature.

Answer 4

You could always go for the gold and drop down into C or ASM. If you are working with raw binary, just bounce it straight off the register. You are 'already there'.