5

I'm working on this big file (DATA.DAT, ~900MB) which contains several other files. It's from a PS2 game.

Sound samples (which are in .AIFF format), precisely what I'm after, make up most of its size.

After searching the web for PS2 .DAT extractors I found out that they're basically developer dependent and since this game/tool is rather obscure and not finding much about it online, I thought about automating the process myself.

Inspecting the file on a hex editor I came across some .AIFF headers, cloned the chunks to new .AIFF files and without any further work, they were playable.

Having spent a while getting the rust out of my VERY limited bash knowledge and having read similar questions here, I came up with this expression:

gcsplit -f "sample-" -b "%04d.aif" DATA.DAT /FORM/ '{*}'

(I'm on OSX using coreutils, hence the g- prefix on csplit)

Given that .AIFF files start with the string "FORM" and given that basically all samples in the file are next to each other (spaced apart by disregardable amounts of data that won't generate unwanted end noise on the samples), I thought that the regexp

/FORM/

would suffice to split the files up.

However, every split file is being output with junk data that sits in between sound samples before the .AIFF header, rendering it unplayable.

Screenshots of the hex data of a split sound sample below:

bad split

This actual sample begins roughly around the 1500 bytes mark:

sample

What's making this expression split the files with an offset?

2
  • hmm, the csplit docs talk a lot about lines, so I'm not so sure it's good for binary files, as those only have lines by accident – thrig Nov 26 '17 at 4:41
  • +1 for pointing me to gcsplit (I actually needed it for text). csplit was giving me fits. – Luke Gedeon Feb 20 '19 at 1:16
2

Csplit is a text utility. It is line-based. A pattern /FORM/ means “a line containing FORM”. A line is a sequence of bytes other than LF (line feed, also known as newline, which can be written \n, ^J, …), followed by an LF byte (or by the end of the file, with GNU utilities). Thus the “junk” that you observe is whatever is between the previous LF character and the FORM substring.

The man page and the --help short description assume that you already know what the command does, so they just mention “pieces” without explanation. You need to read the full documentation to get a description of what the pieces are.

You can't do what you want with csplit. You can do it with GNU awk. (Other versions of awk may not have the requisite features — support of arbitrary record separators and coping with null bytes.) Untested:

gawk -v RS='FORM' -v ORS='' '{
    print "FORM" $0 >sprintf("sample-%04d.aif", n++)
}' DATA.DAT

But this may cut at spurious places if the compressed data just happens to contain the four bytes FORM. This may be good enough for one-time operation with manual review, but you would be better off with a format-aware tool if you need something reliable.

3
  • Well, this is cutting up nicely but it's leaving out the /FORM/s... how can I include them in each file? – João Nov 26 '17 at 17:45
  • also, thank you for the explanation. very concise! – João Nov 26 '17 at 17:51
  • 1
    @João Awk cuts the delimiter. I meant to add them back with print "FORM" … but I made a typo that happened to silently print nothing (awk saw print FORM …, and since the variable FORM is undefined it evaluates to an empty string). I've fixed it now. – Gilles 'SO- stop being evil' Nov 26 '17 at 20:17
0

A text-based utility is not appropriate for manipulating binary files.

You're likely to obtain better results with Lib/aifc, PySoundFile, or the ffmpeg command line app.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.