Psychonauts .PKG File Format Specs

[Benny]

After finding yet another texture id (6) I can now parse all .dds' in .ppf archives for both the pc and xbox version. Thanks for all your help JustinRoad.

 

Now I have to write some code to generate a dds header and hopefully that should be it. Hopefully being the operative word

[JustinRoad]

When performing my own file writes I usually just use a static global structure to write the header out.

 

eg. (C++ code)

 

char gacDDSSignature[] = {'D', 'D', 'S', ' '};

 

DDSURFACEDESC2 gsDDSHeader =

{

0, // dwFlags

0, // dwWidth

...

..

etc..

}

 

EBool WriteDDS(EStream *psStream, EPicture *psPicture)

{

gsDDSHeader.dwWidth = psPicture->iWidth;

gsDDSHeader.dwHeight = psPicture->iHeight;

gsDDSHeader.dwLinearSize = psPicture->iDataSize;

// etc...

 

psStream->Write(gacDDSSignature, 4);

psStream->Write(&gsDDSHeader, sizeof(DDSURFACEDESC2));

psStream->Write(psPicture->pcData, psPicture->iDataSize);

return ETRUE;

}

 

And then when I want to write a header I setup only the image specifics (in this case dwFlags, dwHeight, dwWidth, pixel format and caps) and write the static header. It saves a crapload of time

[john_doe]

Nice work you two are doing

 

While searching for more information on how to decompile Lua 4 I came across this exact thread as the first hit

The deleted posts are still in Google's cache in case you want to copy them:

 

http://216.239.59.104/search?q=cache:Lc3D2jUiYBcJ:www.lucasforums.com/showthread.php%3Fs%3D%26threadid%3D148943+lua+4+OP_CALL&hl=en&client=firefox-a

[Benny]

Excellent, good old google cache

I kept experiencing that too, when searching for information on Xbox ADPCM and Xbox wavebanks I kept coming across my own posts.

 

I've mostly completed the dds' stuff. It now displays almost all of the dds' in the .ppf archives.

 

There's still a couple of obscure texture id's that need supporting, but I'm not sure what dds format they correspond to so I'll probably leave them for now. I'll post complete dds specs soon.

 

I added the final version of your adpcm decoder to Psychonauts Explorer and it works perfectly with the Pc Audio. It works for a lot of the Xbox Audio too. Some of the Xbox audio is marked as Xbox ADPCM but it appears to be slightly different to the standard. With the old method of using the Xbox codec the sounds wouldnt play, even though the wav header was correct. Now, with your decoder they do play, but with distortion. I'll upload some examples later and perhaps you could take a look at them sometime? Those are pretty much the only audio files that cant be played now

[john_doe]

Yes, I'll have a look at them later.

 

I also finally found out why the OP_CALL opcode in my Lua decompiler didn't always work...turns out it worked in functions without parameters. As soon as there were parameters I didn't add the to the stack so the OP_CALL argument pointed to some weird stack location...now it works

Still need to get this if/loop-stuff done...

[JustinRoad]

I downloaded the PC demo of Psychonauts and played through it a couple of times and I'm pretty sure it uses bumpmapping. Which means if it compresses these normalmaps there is a good chance that they use one of the 2 major normalmap compression schemes: 3DC and RXGB.

 

3DC is just 16 bytes per block, but it encodes only 2 channels of the normalmap (each 8 byte block is a DXT5alpha channel style encoding of the Y, X components of the normal in the normalmap respectively. Z is calculated in the pixel shader from the x,y via z=sqrt(1.0 - x*x + y*y)). I'm not sure about whether it uses 3DC on the XBox (as 3DC is an ATI technology)

 

RXGB is Doom3 style normalmap compression which is just DXT5 compression but with the red channel moved into the alpha channel and the red channel set to 0 . This gives better encoding fidelity for normalmaps provided you can unswizzle in the pixel shader (ie. texcol.r = texcol.a)

 

Hopefully this can help with your obscure texture ids.

 

Oh, by the way. I do have decoders written for all the major DXT formats as well as 3DC, RXGB. If you're interested I could run the 'unknown' texture format data through my decoders and see if they produce anything

[john_doe]

Originally posted by JustinRoad

I'm not sure about whether it uses 3DC on the XBox (as 3DC is an ATI technology)

 

Afaik nVidia licensed the 3DC technology from ATI so it could be possible.

It's worth a try at least

[Benny]

I've uploaded a zip here with some examples of the distorted Xbox Audio and the original data. All the information about the files is in info.txt.

 

JustinRoad - yes that would be helpful. I'll upload some examples of the texture id's later.

 

The current wip version of the tool can be found here. Note that if you dump a raw dds file, this includes the 44 (or 72) byte header. So far texture id's 0 (usually rgb8), 9 (dxt1), 10 (dxt3?) and 11 (dxt5) are supported. 'Animated' dds' (id 0), and id's 6, 12 and 14 are not yet supported.

[john_doe]

I've had a (real) quick look the two files. It seems there's 8-bit audio data at the end of the file, the first blocks seem to be Adpcm-compressed. Strange....

[Benny]

Psychonauts .PPF File Format: 
By Benjamin Gorman (bgbennyboy) and Benjamin Haisch (John_Doe) 

4 bytes: Header 'PPAK' 

Texture section 
Contains .dds files without the standard header.
2 bytes - Number of DDS files 

Repeat for each file:
40 bytes - ?
2 bytes - Filename length
X bytes - Filename
44 byte header:
2 bytes - ID number
2 bytes - ID2 number
2 bytes - Texture ID
10 bytes - ?
4 bytes - Texture width
4 bytes - Texture height
4 bytes - Number of mipmaps
4 bytes - ?
2 bytes - New Texture ID (used only if 'animated' image, see below)
10 bytes - ?
X bytes - Texture data

Size of texture data depends on the Texture ID:
0:  Size=(Width*Height)*4
6:  Size=(Width*Height)
9:  Size=(Width * Height) div 2
10: Size=(Width * Height)
11: Size=(Width * Height)
12: Size=(Width*Height)*2
14: Size=(Width*Height) (+1024 if has a palette)

If the file is a cubemap (see the filename) then:
Size=Size*6

However:
The 44 byte header is larger for some files:

If ID2=0 and TextureID=0 then the file is an animation.
It contains multiple dds images within one file:

Number of frames=ID1
Texture ID=New Texture ID
After the 44 byte header:
4 bytes - Texture width
4 bytes - Texture height
4 bytes - Number of mipmaps
16 bytes - ??
Essentially this format is just a container. To parse this format you use
exactly the same methods as below.
After the first block of texture data there follows 'Number of frames' -1
dds files that are set out as normal, with a 44 byte header and then the data.
Also note that the frames within this file do not count as seperate dds files
(in the texture section header).


If Texture ID=14 then:
2 bytes - HasPalette
If HasPalette = 1 or 256 then image has a palette.
Palette has 256 entries so is always 1024 bytes in size.
After the palette data comes the texture data as usual.


The file size also depends on whether the image has mipmaps.
Each mipmap is one-fourth the size of the previous. So the size of each mipmap
has to be worked out and added to the total size. The method of working out
the size depends on the texture id.
So if texture ID was 0 with 8 mipmaps and size of 256*256 then:
262144
65536
16384
4096
1024
256
64
16
total size = 349520


If the TextureID = 9, 10 or 11 things are slightly different:
These use DXT compression so when working out the size:
If TextureID = 9 {DXT1} then minimum mipmaps size = 8
If TextureID = 10 or 11 {DXT3 + DXT5} then minimum mipmaps size = 16

Furthermore if TextureID = 9, 10 or 11 and the image is non-square (eg 256*128) then
the following method must be used at each mipmap level:
Size = max(1,width ÷ 4)x max(1,height ÷ 4) x 8(If ID 9) or x16 (If ID 10 + 11)

Annoyingly there's a further layer of complexity:
If textureID = 0 or 14 then the number of mipmaps reported by the file may be wrong.
So for these 2 ID's the number of mipmaps needs to be corrected:

 if (mipmaps=0) or (width <> height) then
 begin
   temp:=min(width, height);
   case temp of
     512:  mipmaps:=10;
     256:  mipmaps:=9;
     128:  mipmaps:=8;
     64:   mipmaps:=7;
     32:   mipmaps:=6;
     16:   mipmaps:=5;
     8:    mipmaps:=4;
     4:    mipmaps:=3;
     2:    mipmaps:=2;
     1:    mipmaps:=1
     else
       Mipmaps:=1;
   end;
 end;

Texture ID's Correspond to:
0:  8:8:8: argb
6:  ??
9:   DXT1
10: DXT3
11: DXT5
12: 8:8:8: rgb ???
14: ??


Model section. 
4 bytes - 'MPAK' 
2 bytes - Number of model files 
Repeat for each file: 
*2 bytes - Filename length 
*X bytes - Filename 
*2 bytes -?? 
*4 bytes - Size of model data 
*X bytes - Model Data 

Named scripts section. 
Repeat for each file: 
*2 bytes - Filename length 
*X bytes - Filename 
*4 bytes - Size of script data 
*X bytes - Script data 

Unnamed scripts section. 
*4 bytes - Size of script data 
*X bytes - Script data 

Level section? 
Not present in some files (eg common.ppf) 
Size = remaining file data 

Its a bit of a mess, I'll try and clean it up sometime

[Benny]

Attached to this post are examples of some of the other texture id's. The type of texture is in the filename: eg. (6)arial_swz.dds

 

ID 6 is xbox only and seems (only?) to be used in the font dds files.

ID14 are the palleted files

ID12 are (only?) in the 'animated' dds' - as the individual 'frames'. I think this format is probably a 16 bit rgb format - its the right size for it at least.

 

All the files in the zip have the standard 44 byte header, although the file for ID 12 has the 72 byte header.

textureids.zip

[JustinRoad]

Just to let you know that I won't be able to look at those files for a while. I am in crunch mode at work at the moment.

 

If only you'd posted them earlier

[Benny]

Ok, no problem.