15

I was solving a challenge where I found a data file with no file extension. The file command shows that it is a data file (application/octet-stream). The hd command shows GNP. in the last line. So if I reverse this file then I will get the .PNG format file, I searched everywhere but I didn't find a solution explaining how to reverse the content of a binary file.

0

5 Answers 5

Reset to default
20

With xxd (from vim) and tac (from GNU coreutils, also tail -r on some systems):

< file.gnp xxd -p -c1 | tac | xxd -p -r > file.png
3
  • Is there any way for this to be combined with vi.stackexchange.com/a/2237/10649? I tried all kind of combinations with no luck :(
    – Iulian Onofrei
    May 12, 2018 at 17:26
  • 1
    This is not a solution because it will mirror all the file.
    – Philippe Delteil
    Nov 21, 2018 at 3:33
  • 1
    @PhilippeDelteil, mirroring all the file was what the OP is asking for here? What else would you want it to do?
    – Stéphane Chazelas
    Nov 29, 2019 at 16:11
5

In zsh (the only shell that can internally deal with binary data (unless you want to consider ksh93's base64 encoding approach)):

zmodload zsh/mapfile
(LC_ALL=C; printf %s ${(s::Oa)mapfile[file.gnp]} > file.png)
  • LC_ALL=C: characters are bytes
  • $mapfile[file.gnp]: content of file.gnp file
  • s::: split the string into its byte constituents
  • Oa: reverse Order on array subscript that array
1
  • 1
    zsh is not the only shell that can handle binary data.
    – fpmurphy
    Jan 12, 2018 at 14:21
3

With perl:

perl -0777pe '$_=reverse $_'  [input_file]

Performance test:

dd if=/dev/urandom of=/tmp/a bs=1M count=1
LC_ALL=C tac -rs $'.\\|\n' /tmp/a > /tmp/r

time perl -0777pe '$_=reverse $_' /tmp/a         | diff -q - /tmp/r
time xxd -p -c1 /tmp/a | tac | xxd -p -r         | diff -q - /tmp/r
time perl -0777 -F -ape '$_=reverse@F' /tmp/a    | diff -q - /tmp/r
time LC_ALL=C tac -rs $'.\\|\n' /tmp/a           | diff -q - /tmp/r

Result:

  • Tested locally: my solution is the fastest, perl -0777 -F is the slowest.
  • Tested on Try it online!: my solution is the fastest, xxd is the slowest.

Note: the time diff runs should be the same for all solutions, as the output should be the same.

1
  • 1
    I've deleted my perl one. I hadn't realised at the time reverse could reverse strings as well, so doing that splitting didn't make much sense and your version is much much better.
    – Stéphane Chazelas
    Sep 22, 2019 at 17:37
2

Here is one way of reversing a binary file using ksh93. I have left the code "loose" to make it easier to understand.

#!/bin/ksh93

typeset -b byte

redirect 3< image.gpj || exit 1

eof=$(3<#((EOF)))

read -r -u 3 -N 1 byte
printf "%B" byte > image.jpg
3<#((CUR - 1))

while (( $(3<#) > 0 ))
do
    read -r -u 3 -N 1 byte
    printf "%B" byte >> image.jpg
    3<#((CUR - 2))
done

read -r -u 3 -N 1 byte
printf "%B" byte >> image.jpg

redirect 3<&- || echo 'cannot close FD 3'

exit 0
5
  • nice. That's the only answer so far that doesn't involve storing the whole file in memory. However, it's terribly inefficient in that it makes several system calls for each byte of the file (and conversions to/from base64), so wouldn't be suitable for files that don't fit in memory either. On my machine, it processes files at about 10KB/s
    – Stéphane Chazelas
    Jan 12, 2018 at 15:51
  • Note that the first read above should read nothing as it's done at the end of the file.
    – Stéphane Chazelas
    Jan 12, 2018 at 15:58
  • Trying to understand why it was so slow, I tried running it under strace and ksh93 seems to be behaving very weirdly, where it seeks all over the place within the file and reads large amounts at the time. Maybe a variant of github.com/att/ast/issues/15
    – Stéphane Chazelas
    Jan 12, 2018 at 16:00
  • @StéphaneChazelas. No mystery as to why it is relatively slow. Within the loop it has to seek backwards each time it reads a byte. This can easily be significantly reduced by a factor of 20 or even more by reading and writing more than one byte at a time. The write side of things can similarly be optimized. Lots of other techniques are available to further speed things up. I will leave that exercise up to you.
    – fpmurphy
    Jan 13, 2018 at 5:49
  • Try strace on the script to see what I mean. ksh93 reads the files thousands of times over. For instance, before reading the first byte, it seeks 64KiB off the end of the file, reads 64KiB, then seeks before the last byte and reads 1 byte and does something similar for every byte. Note that what you can do with those base64 encoded strings is limited, so if you read more than one byte at a time, it's going to be more difficult to extract the individual bytes of that.
    – Stéphane Chazelas
    Jan 13, 2018 at 9:23
1

I tried the following:

tac -rs '.' input.gnp > output.png

The idea is to force 'tac' using any character as separator. I tried that on a binary file and it seemed to work but any confirmation would be appreciated.

Main advantage is that it does not load file into memory.

2
  • Doesn't work for me (here with GNU tac 8.28) when the input contains newline characters. printf '1\n2' | tac -rs . | od -vAn -tc outputs \n 2 1 instead of 2 \n 1. You'd also need LC_ALL=C or . could match multi-byte characters.
    – Stéphane Chazelas
    Nov 15, 2018 at 20:23
  • 4
    LC_ALL=C tac -rs $'.\\|\n' seems to work though.
    – Stéphane Chazelas
    Nov 15, 2018 at 20:25

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .