Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

This question came to me a couple of times before, now in response to the question Loop through binary data chunks from stdin in Bash Answers given in http://stackoverflow.com/questions/993434/what-language-is-to-binary-as-perl-is-to-text were also not satisfactory.

I'm looking for a scripting environment suited specifically to handle I/O with binary files. I know I can use one of the fully-fledged programming languages (c/Python/...) but they have an enormous initialization and coding overhead (allocation and fread/fwrite in c, bitstrings in Python...) not to mention they are less suited for scripting (calling other applications from it). Perl is no better with its unpack functions, string-oriented operation and goofy syntax.

Something like od, but as a language.

What I expect:

  1. set or change endianness with a single switch/command.
  2. simple specification of requested type (something like extending bash read var with int32 var, float var etc.).
  3. handling of binary through pipes, skipping of specified number of bytes.
  4. standard scripting flow control (for/if/...) that we are used to.

I'd like to process raw data (photography, scientific data, unknown and poorly documented formats) with the same ease and insight that you get when inspecting ASCII files. I'm using c now, but it's not optimal for ad-hoc scripting, and can't be interactive.

Does anyone know a tool like that? No clicky GUI software, please, it needs to work over ssh, from other scripts and so on. "Does not exist" is an acceptable, but depressing answer.

share|improve this question
2  
It does not take away the pain of startup time, but I find the bytes from Python 3.3, together with plumbum very workable: chain = ls["-a"] | grep["-v", "\\.py"] | wc["-l"]; chain() Have you looked at that? –  Anthon Mar 10 at 12:33
    
You could take the C code you have now and turn it into a set of command-line tools you could use in a bash script. While you can't put binary into a shell variable, you can stash it in named ('fifo') pipes; their contents are in held in memory until you want to read them. –  goldilocks Mar 10 at 13:52
1  
There's a serious flaw in your reasoning WRT python and perl, BTW. While individual command line tools are compiled, shell scripts aren't and involve much forking (if you want expensive, forking is it). Your discussion, other question, etc, imply you would be fine w/ using bash here if it could handle binary. Python and perl scripts are both pre-compiled. If you benchmark a reasonably complex python or perl script vs. a parallel bash script, the perl or python will be an order of magnitude faster. If you don't believe me, you are welcome to search the web for evidence to the contrary. –  goldilocks Mar 10 at 14:14
    
I'm not looking for a tool that runs fast, I'm looking for something I can code fast. For instance, if I have a strange program that outputs a binary int for array size of structs (int,float,float) that follow after it, I'd like to quickly read the array size and loop over the array, possibly calculating some cumulative or maximum of some components, or just print one component as ascii column for gnuplot processing. Anthon: thank you, I didn't know about this, it will be useful. goldilocks: I'm trying to avoid that but I may just write my own tool at the end :) –  orion Mar 10 at 15:15
2  
Sounds like you need a tutorial on how to use perl's unpack (ᵔᴥᵔ) –  Stéphane Chazelas Mar 10 at 21:19

4 Answers 4

Have you come across beav it has macros but I couldn't find scripting,

apt-cache show beav extract :

With beav, you can edit a file in HEX, ASCII, EBCDIC, OCTAL, DECIMAL, and BINARY. You can display but not edit data in FLOAT mode. You can search or search and replace in any of these modes. Data can be displayed in BYTE, WORD, or DOUBLE WORD formats. While displaying WORDS or DOUBLE WORDS the data can be displayed in INTEL's or MOTOROLA's byte ordering. Data of any length can be inserted at any point in the file. The source of this data can be the keyboard, another buffer, or a file. Any data that is being displayed can be sent to a printer in the displayed format. Files that are bigger than memory can be handled.

Then there's xxd which converts to/from binary/ascii display mode and could be combined together with sed or vi, but hasn't got the byte swapping feature.

share|improve this answer

I do have the exact same problem than you for years as well.

For simple non-interactive uses, I like to use the binary block editor BBE. BBE is to binary as SED is to text, including its archaic syntax and simplicity, however, it has a lot of features missing from what I often need, so I have to combine it with other tools. So, BBE is only a partial solution. Also note that BBE hasn't had any updates or improvements for years.

Of course one can use xxd before and xxd -r after editing the data with text-based tools, but that won't work when the data in question is large and random access is required, for example when processing block devices.

(Note: For Windows, there is at least the costly, proprietary WinHex scripting language, but that won't get us anywhere.)

For more complicated binary editing, I usually fall back to Python as well, even though it sometimes is too slow for large files, which is it's main drawback. I hope Pyston (Python employing LLVM to compile to optimized machine code) will someday mature enough to be usable, or even better, someone will design and implement a free compact, fast and versatile binary processing scripting language, which AFAIK doesn't exist for U*IX like systems yet.

share|improve this answer

You could always go for the gold and drop down into C or ASM. If you are working with raw binary, just bounce it straight off the register. You are 'already there'.

share|improve this answer

You don't necessarily have to "make peace" with Perl's unpack... one of the great things about perl is how you can abuse the parser and symbol table to make your own language, in a custom package.

Is this basically what you're looking for?

use MyBinLib;
my $struct= struct(
  pack => 8,
  size => 400,
  fields => [int32('foo','bar','baz'), float32('x1','x2','x3','x4'), int8, int8, int16('z')]
);
while (my $rec= $struct->read(<STDIN>)) {
  printf "x1 = %d, x2 = $d\n", $rec->x1, $rec->x2;
}

The exercise then is to learn enough perl to write the MyBinLib package. Ask in a Perl forum and people would probably be happy to help.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.