5

I have a large (2GB) file that looks like this:

^%%-=-=-=-=-=-=-=-=-=-=-=-=-=-%%^
<binary data>
^%%-=-=-=-=-=-=-=-=-=-=-=-=-=-%%^ 
<binary data>
^%%-=-=-=-=-=-=-=-=-=-=-=-=-=-%%^
<binary data>
...

The ^%%-=-=-=-=-=-=-=-=-=-=-=-=-=-%%^ lines are separators. The binary segments are large. There are about fifty of them in the file.

I am trying to extract the binary parts of this file. Each binary segment needs to go into its own file.

I tried using csplit,

csplit --digits=2 --prefix=out stu.ear '/\^%%-=-=-=-=-=-=-=-=-=-=-=-=-=-%%\^/'

but received the following output and two out?? files,

1
2097951144

Is there a tool for this job (a csplit implementation that works with binary files, perhaps?)

2
  • 1
    did you try awk '/\^%%-=-=-=-=-=-=-=-=-=-=-=-=-=-%%\^/{n++}{print >"out" n ".ear" }' stu.ear, This should work but I am not sure about the awk data input limits
    – Raza
    Jul 1 '13 at 1:32
  • To only split big files into smaller ones is not sufficient. I asked a similar question but with simultaneous progress information here stackoverflow.com/q/31021731/54964 I think the main tool here is with dclfdd. Jun 24 '15 at 8:41
5

I wrote a little python tool to do this. https://github.com/mypalmike/csplitb

csplitb.py --prefix X --suffix Y --number Z XXXXXXXX input-file.extension
X = Name at beginning of output filename
Y = Desired output file extension
Z = Number of digits used to differentiate output files
XXXXXXXX = Starting hex of each binary file to be split out of the input file
input-file.extension = The file being split

Example:
csplitb.py --prefix photo --suffix .png --number 4 89504e47 block-file.raw

Output:

photo0000.png
photo0001.png
photo0002.png
.............
3
  • 1
    Welcome to Unix & Linux Stack Exchange! Whilst this may theoretically answer the question, it would be preferable to include the essential parts of the answer here, and provide the link for reference.
    – slm
    Nov 15 '13 at 1:47
  • 1
    You should at the least show an example of how to use the tool you are recommending.
    – jordanm
    Nov 15 '13 at 3:57
  • Thanks, but it's not huge-files-friendly. Traceback (most recent call last): File "./csplitb/csplitb.py", line 64, in <module> main() File "./csplitb/csplitb.py", line 61, in main return csplitb.run() File "./csplitb/csplitb.py", line 30, in run self.finish() File "./csplitb/csplitb.py", line 42, in finish self.write(self.mm[self.last_idx:]) MemoryError
    – Paul
    Nov 5 '16 at 8:54
3

The following will work:

      awk '/\^%%-=-=-=-=-=-=-=-=-=-=-=-=-=-%%\^/{n++}{print >"out" n ".ear" }
1
  • Have you tried dcfldd command? Awk solution may become problematic in some situations. Jun 24 '15 at 8:19
1

You told csplit to split the file at one location, at the first occurrence of the text ^%%-=-=-=-=-=-=-=-=-=-=-=-=-=-%%^. So you naturally got two pieces: one that contains the first byte of the file (is there a newline or blank before the first separator?) and one that contains everything from the first separator onwards.

If you want to split into separate files, you'll have to repeat the pattern as many times as you want pieces minus one. Portably, you need to count the pieces.

csplit --digits=2 --prefix=out stu.ear '/\^%%-=-=-=-=-=-=-=-=-=-=-=-=-=-%%\^/'"{$(grep -c '\^%%-=-=-=-=-=-=-=-=-=-=-=-=-=-%%\^' stu.ear)}"

GNU csplit has an extension that lets you repeat a pattern an indefinite number of times:

csplit --digits=2 --prefix=out stu.ear '/\^%%-=-=-=-=-=-=-=-=-=-=-=-=-=-%%\^/ {*}'

But this doesn't do what you want, because the separator is included in the output. You can remove it from the files afterward; it would be a little easier if you arrange to have the separators at the end of the files, using % rather than / as the pattern delimiter (so csplit … '%\^\%\%-=-=-=-=-=-=-=-=-=-=-=-=-=-\%\%\^% {*}'). But you might as well resign yourself to the fact that csplit, while cute, has a very narrow use case, and yours doesn't fit. Use a better-suited tool such as awk.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy