5

Example: I have the file "mybinaryfile", and the contents in hex are:

A0 01 00 FF 77 01 77 01 A0

I need to know how many A0 bytes there are in this file, how many 01, and so on. The result could be:

A0: 2
01: 3
00: 1
FF: 1
77: 2

Is there some way to make this count directly in shell or do I need to write a program in whatever language to do this specific task?

  • 1
    Looking at answers, this seems to be a worthy codegolf ;) – val Jun 29 at 15:31
18

This uses od to show one hex value per line, then sorts and counts:

od -t x1 -w1 -v -An mybinaryfile | sort | uniq -c

(-w1 is an extension, it’s not mandated by POSIX.)

  • Alternatives to od are: xxd -c1 -p file and/or hexdump -v -e '/1 "%02X \n"' file. – Isaac Jun 29 at 21:00
5

Using Perl to unpack the slurped file to a byte array and then use a hash to count unique bytes:

printf '\xA0\x01\x00\xFF\x77\x01\x77\x01\xA0' | 
  perl -0777 -nE '
    @bytes = unpack("C*",$_) 
    }{ 
    $counts{$_}++ for @bytes; 
    for $k (sort { $a <=> $b } keys %counts) {
      printf "%02X: %d\n", $k, $counts{$k}
    }
 '
00: 1
01: 3
77: 2
A0: 2
FF: 1

If a sufficiently recent version of List::MoreUtils is available, you may be able to simplify the counting by using its frequency function.

0

Quick Python solution:

#!/usr/bin/env python3
import sys, itertools, collections
print(
    *itertools.starmap(
        "{:02X}: {:d}".format,
        collections.Counter(sys.stdin.detach().read()).items()),
    sep="\n")

One-liner:

python3 -c 'import sys, itertools, collections; print(*itertools.starmap("{:02X}: {:d}".format, collections.Counter(sys.stdin.detach().read()).items()), sep="\n")' \
    < input.bin

Options and caveats

  • If you want to the output sorted by frequency in descending order, replace .items() with .most_common(). Alternatively or for other sorting schemes, use the built-in sorted() function or post-process the output with the sort(1) program.

  • In its current state, the programs slurps the entire standard-input data into a byte buffer which is fine for relatively small files. For larger files, the program needs to be rewritten to read files in chunks.

0
  • < my_binary_file xxd -p | fold -w 2 | sort | uniq -c

       1 00
       3 01
       2 77
       2 a0
       1 ff
    
  • < my_binary_file xxd -p | fold -w 2 | sort | uniq -c | awk '{print $2": "$1}'

    00: 1
    01: 3
    77: 2
    a0: 2
    ff: 1
    

Explanation

  • < my_binary_file passes the contents of my_binary_file to the standard input of the xxd command.
  • xxd -p converts the data read from its standard input in a hexadecimal dump, and the modifier -p (plain) tells the program to output only the digits without offsets nor textual representation.
  • fold -w 2 inserts a newline character every two characters (-w 2), converting the input stream to a newline-separated byte list.
  • sort, as the name suggests, sorts the lines grouping the byte values.
  • uniq -c counts the occurrences of each value in the input data.
  • Optionally, a bit of awk magic converts the output format to the requested in the original post.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.