15

This question came to me a couple of times before, now in response to the question Loop through binary data chunks from stdin in Bash Answers given in https://stackoverflow.com/questions/993434/what-language-is-to-binary-as-perl-is-to-text were also not satisfactory.

I'm looking for a scripting environment suited specifically to handle I/O with binary files. I know I can use one of the fully-fledged programming languages (c/Python/...) but they have an enormous initialization and coding overhead (allocation and fread/fwrite in c, bitstrings in Python...) not to mention they are less suited for scripting (calling other applications from it). Perl is no better with its unpack functions, string-oriented operation and goofy syntax.

Something like od, but as a language.

What I expect:

  1. set or change endianness with a single switch/command.
  2. simple specification of requested type (something like extending bash read var with int32 var, float var etc.).
  3. handling of binary through pipes, skipping of specified number of bytes.
  4. standard scripting flow control (for/if/...) that we are used to.

I'd like to process raw data (photography, scientific data, unknown and poorly documented formats) with the same ease and insight that you get when inspecting ASCII files. I'm using c now, but it's not optimal for ad-hoc scripting, and can't be interactive.

Does anyone know a tool like that? No clicky GUI software, please, it needs to work over ssh, from other scripts and so on. "Does not exist" is an acceptable, but depressing answer.

|improve this question
  • 2
    It does not take away the pain of startup time, but I find the bytes from Python 3.3, together with plumbum very workable: chain = ls["-a"] | grep["-v", "\\.py"] | wc["-l"]; chain() Have you looked at that? – Anthon Mar 10 '14 at 12:33
  • You could take the C code you have now and turn it into a set of command-line tools you could use in a bash script. While you can't put binary into a shell variable, you can stash it in named ('fifo') pipes; their contents are in held in memory until you want to read them. – goldilocks Mar 10 '14 at 13:52
  • 1
    There's a serious flaw in your reasoning WRT python and perl, BTW. While individual command line tools are compiled, shell scripts aren't and involve much forking (if you want expensive, forking is it). Your discussion, other question, etc, imply you would be fine w/ using bash here if it could handle binary. Python and perl scripts are both pre-compiled. If you benchmark a reasonably complex python or perl script vs. a parallel bash script, the perl or python will be an order of magnitude faster. If you don't believe me, you are welcome to search the web for evidence to the contrary. – goldilocks Mar 10 '14 at 14:14
  • I'm not looking for a tool that runs fast, I'm looking for something I can code fast. For instance, if I have a strange program that outputs a binary int for array size of structs (int,float,float) that follow after it, I'd like to quickly read the array size and loop over the array, possibly calculating some cumulative or maximum of some components, or just print one component as ascii column for gnuplot processing. Anthon: thank you, I didn't know about this, it will be useful. goldilocks: I'm trying to avoid that but I may just write my own tool at the end :) – orion Mar 10 '14 at 15:15
  • 2
    Sounds like you need a tutorial on how to use perl's unpack (ᵔᴥᵔ) – Stéphane Chazelas Mar 10 '14 at 21:19
2

I do have the exact same problem than you for years as well.

For simple non-interactive uses, I like to use the binary block editor BBE. BBE is to binary as SED is to text, including its archaic syntax and simplicity, however, it has a lot of features missing from what I often need, so I have to combine it with other tools. So, BBE is only a partial solution. Also note that BBE hasn't had any updates or improvements for years.

Of course one can use xxd before and xxd -r after editing the data with text-based tools, but that won't work when the data in question is large and random access is required, for example when processing block devices.

(Note: For Windows, there is at least the costly, proprietary WinHex scripting language, but that won't get us anywhere.)

For more complicated binary editing, I usually fall back to Python as well, even though it sometimes is too slow for large files, which is it's main drawback. I hope Pyston (Python employing LLVM to compile to optimized machine code) will someday mature enough to be usable, or even better, someone will design and implement a free compact, fast and versatile binary processing scripting language, which AFAIK doesn't exist for U*IX like systems yet.

UPDATE

I also happen to use the homebrew, open source Intel x86 assembler flat assembler, or fasm for short, that evolved into much more than just an assembler.

It has a powerful, textblock-based macro preprocessor (itself a turing complete language) with a syntax in the tradition of the borland turbo assembler macro language, but much more advanced.

Also, it has a data manipulation language, which allows to binary include arbitrary files, do all kinds of binary and arithmetic manipulation on it (integer only) at "compile time" and write the result into an output file. This data manipulation language has control strutures and is also turing complete.

It is much easier to use than writing a program that does some binary manipulation in C and probably even in python. Plus, it loads blindingly fast, as it is a small sized executable with almost no external dependecies (There are 2 versions: either it only requires libc or it can run as a static executable directly on the Linux kernel ABI).

It does have some ruff edges, like

  1. not supporting concurrency

  2. being writting in 32 bit x86 assembly (works on x86_64 though), you probably need qemu or a similar emulator if you want to run it on anything else than x86 or x86_64

  3. it's powerful macro preprocessor language is turing complete, this means you better have some experience with languages like Lisp, Haskell, XSLT, or probably M4 would be the best choice.

  4. all data that is to be written into the output file are performed in a "flat" buffer in memory, and this buffer can grow but not shrink until the output file has been written and fasm terminated. This means that one can only generate files at most as large as you have main memory left in a single run of fasm.

  5. data can only be written into a single output file for each run of fasm

  6. yeah, it is homebrew, a really neat and clever one though

|improve this answer
2

You don't necessarily have to "make peace" with Perl's unpack... one of the great things about perl is how you can abuse the parser and symbol table to make your own language, in a custom package.

Is this basically what you're looking for?

use MyBinLib;
my $struct= struct(
  pack => 8,
  size => 400,
  fields => [int32('foo','bar','baz'), float32('x1','x2','x3','x4'), int8, int8, int16('z')]
);
while (my $rec= $struct->read(<STDIN>)) {
  printf "x1 = %d, x2 = $d\n", $rec->x1, $rec->x2;
}

The exercise then is to learn enough perl to write the MyBinLib package. Ask in a Perl forum and people would probably be happy to help.

|improve this answer
1

Have you come across beav it has macros but I couldn't find scripting,

apt-cache show beav extract :

With beav, you can edit a file in HEX, ASCII, EBCDIC, OCTAL, DECIMAL, and BINARY. You can display but not edit data in FLOAT mode. You can search or search and replace in any of these modes. Data can be displayed in BYTE, WORD, or DOUBLE WORD formats. While displaying WORDS or DOUBLE WORDS the data can be displayed in INTEL's or MOTOROLA's byte ordering. Data of any length can be inserted at any point in the file. The source of this data can be the keyboard, another buffer, or a file. Any data that is being displayed can be sent to a printer in the displayed format. Files that are bigger than memory can be handled.

Then there's xxd which converts to/from binary/ascii display mode and could be combined together with sed or vi, but hasn't got the byte swapping feature.

|improve this answer
0

You could always go for the gold and drop down into C or ASM. If you are working with raw binary, just bounce it straight off the register. You are 'already there'.

|improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.