Extract data between two matched patterns in a binary file

Question

I am trying to extract a jpeg image from a binary text file. I want to extract all data between 0xFF 0xD8 (start of image) and 0xFF 0xD9 (end of image) inclusive. Earlier, I have successfully run the following command to get the desired image.jpg from a single paragraph file received.txt:

sed 's/.*\xFF\xD8/\xFF\xD8/; s/\xFF\xD9.*/\xFF\xD9/' received.txt > image.jpg

But when I tried to run the same operation on a different file, it didn't work. I also tried using

sed -n '/\xFF\xD8/,/\xFF\xD9/p' received.txt > temp.txt
sed 's/.*\xFF\xD8/\xFF\xD8/; s/\xFF\xD9.*/\xFF\xD9/' temp.txt > image.jpg

to remove any lines before or after the matched lines but got no success.

Although the file was too large, I pasted the hex dump of the relevant portion below:

0a 55 57 5d 50 cf ff d8 ff fe ff ff ff d9 df 47 fe e7 c9 3b e9 9b 6b 55 c4 57 9b 98 73 fd 15 f7 77 7e f7 95 dd 55 f7 55 05 cc 55 97 55 dd 62 d1 1f 51 ef f1 ef fb e9 bf ed 5f bf f2 9d 75 af fe 6b fb bf 8f f7 f7 7e ff d3 bf 8e d5 5f df 57 75 fe 77 7b bf d7 af df 5d fb 0a 47 de d5 ff c1 23 9b 20 08 20 65 3c 06 83 11 05 30 50 a0 20 55 20 84 41 04 c2 59 50 89 64 44 44 10 05 20 87 28 1d a9

The hex dump of the desired output in this case is:

ff d8 ff fe ff ff ff d9

---

Update

While trying to resolve the issue, I found that the sed command removes all the characters before or after a matched pattern upto the non-ASCII character (0x80 - 0xFF) but not go beyond that non-ASCII character. As an example, if we try:

echo 55 57 5d 50 cf 50 65 7f ff d8 ff fe ff ff ff d9 | xxd -r -p | sed 's/.*\xFF\xD8/\xFF\xD8/' > output

The hex dump of the output can be seen as:

xxd output

which is:

55 57 5d 50 cf ff d8 ff fe ff ff ff d9

As can be seen, the characters between the non-ASCII character and matched pattern are removed but the characters before the non-ASCII character are not.

---

Alternative Solution (not perfect)

I used the following commands to somewhat resolve the problem:

sed 's/\xFF\xD8/\x0A\xFF\xD8/; s/\xFF\xD9/\xFF\xD9\x0A/' received.txt > temp.txt

then run the following command (which will work if there is no new line character (0x0A) somewhere between 0xFF 0xD8 and 0xFF 0xD9):

sed -n '/\xFF\xD8/{/\xFF\xD9/p}' temp.txt > image.jpg

but if image.jpg file is empty (after execution of the above command), then run the following command:

sed -n '/\xFF\xD8/,/\xFF\xD9/p' temp.txt > image.jpg

These commands will do the desired job except that it puts 0x0A at the end of the image.jpg file (i.e., after 0xFF 0xD9). In my case, it did not create any issue as JPEG file automatically discards data after 0xFF 0xD9 marker.

I was stuck at the implementation of 'if image file is empty' condition when @chaos came up with a perfect solution. So, I am now following his solution. Thanks a lot @chaos!

---

Notes:

Here is how you can get the actual data from its hex dump which you can pipe to sed command:

echo 0a 55 57 5d 50 cf ff d8 ff fe ff ff ff d9 df 47 fe e7 c9 3b e9 9b 6b 55 c4 57 9b 98 73 fd 15 f7 77 7e f7 95 dd 55 f7 55 05 cc 55 97 55 dd 62 d1 1f 51 ef f1 ef fb e9 bf ed 5f bf f2 9d 75 af fe 6b fb bf 8f f7 f7 7e ff d3 bf 8e d5 5f df 57 75 fe 77 7b bf d7 af df 5d fb 0a 47 de d5 ff c1 23 9b 20 08 20 65 3c 06 83 11 05 30 50 a0 20 55 20 84 41 04 c2 59 50 89 64 44 44 10 05 20 87 28 1d a9 | xxd -r -p

and you can see the hex dump of a file by:

xxd file.txt

Answer 1

With your example data and grep with perl regular expressions (PCRE) activated (-P):

grep -oP '\xFF\xD8.*\xFF\xD9' input >image.jpeg

The -o flag says grep to only print the matching part. The test after looks promising:

$ file image.jpeg
image.jpeg: JPEG image data

---

Edit: If the above doesn't work, and it has to be sed, we have to convert the data to text:

hexdump -ve '1/1 "%.2X"' input | sed 's/.*\(FFD8.*FFD9\).*/\1/' | xxd -r -p >image.jpeg

• With hexdump the input file is converted to a sequence similar to the one in your question.
  • -e specifies the format
  • 1/1 means apply the format 1 times (iteration count) and the 1 after the / specifies the number of bytes to be interpreted by each iteration (byte count).
  • %.2X is the format: a two-digit hex value.
• Then sed removes everything before FFD8 and after FFD9 from the dump.
  • The brackets \(...\) specify a subpattern which we want to save for later
  • Replace everything with \1, which is the content of the subpattern from above.
• At least, xxd reverses the hexdump to a binary format.

---

The test succeeds, when using the example in your question:

$ echo 0a 55 57 5d 50 cf ff d8 ff fe ff ff ff d9 df 47 fe e7 c9 3b e9 9b 6b 55 c4 57 9b 98 73 fd 15 f7 77 7e f7 95 dd 55 f7 55 05 cc 55 97 55 dd 62 d1 1f 51 ef f1 ef fb e9 bf ed 5f bf f2 9d 75 af fe 6b fb bf 8f f7 f7 7e ff d3 bf 8e d5 5f df 57 75 fe 77 7b bf d7 af df 5d fb 0a 47 de d5 ff c1 23 9b 20 08 20 65 3c 06 83 11 05 30 50 a0 20 55 20 84 41 04 c2 59 50 89 64 44 44 10 05 20 87 28 1d a9 | \
  xxd -r -p | \
  hexdump -ve '1/1 "%.2X"' | \
  sed 's/.*\(FFD8.*FFD9\).*/\1/' | \
  xxd -r -p >image.jpeg
$
$ file image.jpeg
image.jpeg: JPEG image data
$ xxd image.jpeg
0000000: ffd8 fffe ffff ffd9                      ........

Answer 2

Just want to add little more to @chaos solution

hexdump -ve '1/1 "%.2X "' input | sed 's/.*\(FF D8.*FF D9\).*/\1/' | xxd -r -p > image.jpeg

I have just added space after %.2X and, between FFD8 and FFD9. This is to avoid matching the shifted pattern such as:

0f fd 80 ... 0f fd 90