Psychonauts .PKG File Format Specs

[Benny]

I've just finished my Psychonauts .PKG Dumper so here are the specs for the .PKG file format.

 

Psychonauts .PKG File Format:
-----------------------------

Offsets are relative to the beginning of the file unless otherwise stated.

In the xbox version the .pkg file is zlib compressed. There is a 16 byte
header then a compressed zlib archive. The format of this header is:

4 bytes: Header 'ZLIB'
4 bytes: Version
4 bytes: Size of file when decompressed
4 bytes: Size of compressed file (minus the 16 byte header)

Structure of file:
------------------
Section		      Size (in bytes)

Header:               512 bytes
File records:         16 * Number of Files
Unknown data:         Offset of Name Directory - End of File Records
Name directory:       Offset of File Types - Offset of Name Directory
File Types directory: End of File Types Dir - Offset of File Types Dir
File data:            Rest of file


File Header:
------------
4 bytes: Header 'ZPKG'
4 bytes: Version
4 bytes: End of File Types Directory
4 bytes: Number of files
4 bytes: End of File Records
4 bytes: ?
4 bytes: Offset of Name Directory
4 bytes: Offset of File Types Directory


File Records:
-------------
Each file record is 16 bytes long:
4 bytes: Identifier
4 bytes: Offset of name in Name Directory (relative to start of name dir)
4 bytes: Offset of file data
4 bytes: File data size

The identifier identifies the file type:
256:   .asd
1280:  .atx
2304:  .cam
3328:  .dds
4352:  .dfs
5376:  .eve
6400:  .h
6912:  .hlps
8192:  .ini
9216:  .jan
10240: .lpf
11264: .lua
12288: .pba
13312: .plb
14336: .psh
15360: .vsh

I'm fairly sure that the identifiers match up to these file extensions.
These file extensions are found in the File Types Directory.


Name Directory:
---------------
This contains the name of every file (without the file extension).
Each filename is null terminated (00)
To get the filename, read the 'Offset of name in Name Directory' value
from the file record and seek to this value + 'Offset of Name Directory'.
Be aware that some files share the same filename.


File Types Directory:
---------------------
This lists the file extensions of the various files in the .PKG.
Each file extension is null terminated.


Dumping files:
--------------
To dump a file:
1) Read the file record
2) Seek to (Offset of name in Name Dir + Start offset of Name Dir)
3) Read the filename and add the correct file extension according to the
identifier
4) Seek to 'Offset of file data'
5) Copy out 'File data size' bytes

pkg_file_format.txt

[john_doe]

Very nice!

 

I've had a look at the other files:

 

jan are the animation files, plb seem to be the models.

hlps, psh and vsh are Pixel/Vertex Shader programs.

 

If time permits I'll try to work on the plb files

[Benny]

Cool! If you or anyone else could figure out the format of the level pack files that'd be great too. The .apf files are the same on both pc and xbox, but the .ppf ones differ (when decompressed).

 

I'm not sure about the .ppf's at all, I dont *think* they are based on chunks but then that would probably mean that the indexes of the files were stored in the .apf files, but this cant be right because these are the same on pc and xbox, whereas the .ppf files differ.

 

In common.ppf the first file block refers to textures\icons\ui_icons\ui_joystick_01.dds and if you compare that 'block' to 'ui_joystick_01.dds' from the main .pkg file - its the same data, but without the .dds header data.

 

In conclusion, I dont really know how the level pack files work.

[netmonkey]

I'll be waiting for PsychonautsRev...

[ThunderPeel2001]

I'm still waiting for SCUMMRev (well, one that work properly! )

[john_doe]

I've had a closer look at the Ppf files. It seems there are several 'sections' of data

- textures (at the beginning of the file, a lot of dds files, "PPAK")

- models (starts with "MPAK" section)

- scripts

- more scripts

- another model.

 

The texture section is right at the beginning and contains dds textures with a special header which has yet to be analyzed. I couln't find an obvious value in this section which says how big the following texture data is.

 

The model group starts with the marker "MPAK" and has a rather simple structure, after the marker comes an unsigned short telling the number of model files, then for each file an unsigned short with the length of the filename, then the filename and an unsigned long with the size of the data, after this the model data (same format as the plb files from the main pkg).

 

Next comes the script section, similar in strucuture to the model section, the only obvious difference that there are compiled Lua scripts here

 

After this section come more scripts, but without the filename.

 

The last section is some model file, maybe the level itself. It has some data in the "PRCS" section to specify events and Lua functions to be called/used or similar.

[Benny]

LF has lost the last month's worth of posts.

 

I've made some progress with the .dds' in the .ppf's, I can now parse many of them, but there are some that still give me headaches. I'll post about what I've found out/what I'm stuck with later

[Benny]

DDS files inside PPF archives:

'PPAK' - header
2 bytes - Number of DDS files

*Repeat for each DDS*
40 bytes - ?
2 bytes - Filename length
X bytes - Filename
2 bytes - ID number
2 bytes - Bigger ID number
2 bytes - Texture ID
10 bytes - ?
4 bytes - Texture width
4 bytes - Texture height
4 bytes - Number of mipmaps
16 bytes - ?
X bytes - Texture data

 

I used the information on dds' as set out here.

TextureID's:

0: Size= (Width*Height)*4

9: Size=(Width * Height) div 2

10: Size=(Width * Height)

11: Size=(Width * Height)

 

However if the file is a cubemap (it says in the filename) then size:= Size*6

 

If there are mipmaps then the size of the mipmaps must be taken into account and if the texture is not square then the size must be calculated slightly differently. See the attached file and this for more information.

 

So far so good, this parses a good few of the files. But problems have occured with 'lightmap000.dds' in asco.ppf:

 

Texture ID = 0

Width = 256

Height = 128

Mipmaps = 0

Name= textures\lightmaps\asco\lightmap000.dds

Actual size = 174760

Size that tool gives - 131072

 

The file has no mipmaps, so according to the formula its size should be (256*128)*4. I cant work out a method for getting the size of this texture, the only thing I can think of is that it might perhaps be a volume texture

 

So as things stand, I've made some progress, but am now stuck. The tool always reaches a file with texture id 0 thats either a lightmap or norm (normal map?) and exits.

 

[Edit] It wouldnt let me attach a file, so i've uploaded it here

[JustinRoad]

Actual size = 174760

Size that tool gives - 131072

 

I don't actually own psychonauts and haven't looked at the data in question, but I'm a console programmer and this kinda thing really interests me

 

Even though the DDS doesn't report any mipmaps, you are aware that 17460 is very, very close to the total size of the image with mipmaps?

 

131072 (base level image size) +

32768+

8192+

2048+

512+

128+

32+

8 (the smallest image size as far as I'm aware)

 

= 174760

 

EDIT: just to check my math

[Benny]

Its great to have someone else's input

 

I'm just trying to get my head around all this again, its been a week or so since I was messing with this.

 

For texture id 0 I've assumed that the minimum mipmap size is 1 because I dont *think* that its one of the DXTx compressions. That could be wrong of course but it works for parsing this file:

 

textures\leveltextures\as_asylum\as_in_tilefloor_norm.dds

ID = 442

ID 2 = 45388

Texture ID = 0

Width = 256

Height = 256

Mipmaps = 9

totalsize= 349524

 

Now I think about it, I actually use the DXTx method for computing the size with textureID 0 - if this really isnt one of the DXTx compressions then the size should be calculated the normal way I think.

 

This post probably makes no sense, I'm just trying to get my head round my messy code again. 12 bytes is very close, perhaps with lightmaps there are always 8 or 9 mipmaps, even if the mipmap number isnt specified.

 

 

 

[Edit] Ignore all that. It looks like youve got it!

[Benny]

Right then it seems I was doing two things wrong:

 

I was using the DXTx method for calculating the size of non-square textures when (I think) textureID 0 isnt DXTx compression.

 

When I hardcode 8 mipmaps it parses the first lightmap fine, then computes the size of the next one slightly wrong. I suspect that this is because the next lightmap has 1 less mipmap. Some method of calculating the number of mipmaps from the texture size is probably needed.

 

Well done JustinRoad! Thank you

[Benny]

Ok so the next 2 lightmaps:

 

textures\lightmaps\asco\lightmap001.dds

ID = 483

ID 2 = 45429

Texture ID = 0

Width = 32

Height = 16

Mipmaps = 0

totalsize= 2732

actualsize=2728

=5 mipmaps

 

textures\lightmaps\asco\lightmap002.dds

ID = 484

ID 2 = 45430

Texture ID = 0

Width = 32

Height = 32

Mipmaps = 0

totalsize= 5456

actualsize=5460

=6 mipmaps

 

I wouldnt expect lightmap002 to have 6 mipmaps, I'd expect 5 really, so it might be tricky to work out a formula for computing the no of mipmaps.

 

[Edit] Whoops, 6 mipmaps even

[Edit 2] My mistake - it looks like the no of mipmaps is as on the MSDN page - just with non-square textures its 1 less mipmap.

[JustinRoad]

12 bytes is very close,

 

I know. I made a bit of an oopsie in my additions. The 12 bytes is non-existant. The mip size is spot on! (I edited my post to reflect the changes )

 

wouldnt expect lightmap002 to have 4 mipmaps, I'd expect 5 really, so it might be tricky to work out a formula for computing the no of mipmaps.

 

 

As far as I can figure out, the general formula for the number of mips is:

NumMips = 1 + LogBase2(Min(width,height))

 

so, for a 32x32 you will have:

NumMips = 1 + LogBase2(32)

= 1 + 5

= 6 mipmaps

 

The mips are listed here:

-> 32x32x4 = 4096

-> 16x16x4 = 1024

-> 8x8x4 = 256

-> 4x4x4 = 64

-> 2x2x4 = 16

-> 1x1x4 = 4

 

The sum of which is 5460...

 

for a 32x16 you wil have

NumMips = 1 + LogBase2(16)

= 1 + 4

= 5

 

The mips are listed here:

-> 32x16x4 = 2048

-> 16x8x4 = 512

-> 8x4x4 = 128

-> 4x2x4 = 32

-> 2x1x4 = 8

 

The sum of which is: 2728

[Benny]

Thanks but I'm not sure exactly what LogBase2() means. I have very little maths knowledge.

 

Instead I have done this, which seems to work

 

if mipmaps=0 then

begin

temp:=max(width, height);

case temp of

512: mipmaps:=10;

256: mipmaps:=9;

128: mipmaps:=8;

64: mipmaps:=7;

32: mipmaps:=6;

16: mipmaps:=5;

8: mipmaps:=4;

4: mipmaps:=3;

2: mipmaps:=2;

1: mipmaps:=1;

end;

end;

 

if width <> height then dec(mipmaps, 1);

 

 

This seems to work with the files that I've tried so far

Now it errors with 'flames' textures that all have a width of 1097859072. Nothings every easy...

[JustinRoad]

Okay, LogBase2 is pretty much what your 'if' statement does, but it does it in a pure 'mathematical' way.

 

If you know anything about logarithms, you can use the natural log: eg. LogBase2(x) = Log(x) / Log(2);

 

Your version will work okay for cases where 1 axis is no more than 2 times the size of another axis.

 

What if your texture was 512x128? According to you it will be 10 mipmaps when in reality it will be 8 mipmaps.

 

What you really need to do is something like this:

 

if mipmaps=0 then

begin

case min(width,height) of

512: mipmaps:=10;

256: mipmaps:=9;

128: mipmaps:=8;

64: mipmaps:=7;

32: mipmaps:=6;

16: mipmaps:=5;

8: mipmaps:=4;

4: mipmaps:=3;

2: mipmaps:=2;

1: mipmaps:=1;

end;

end;

 

where min(x,y) simply returns the smaller of the 2 numbers.

eg. (I haven't done pascal in a while, so excuse poor code)

 

function min(x : Integer; y: Integer) : Integer;

begin

if (x < y) then

result := x;

else

result := y;

end;

[Benny]

Thanks for explaining

 

As you can see, I edited the old post while you were typing that one, but I used max instead of min().

 

I've just tried min() and it computes the size incorrectly, are you sure you meant min and not max?

 

[Edit] No, you're right, it is min. I didnt remove the 'if width <> height then dec(mipmaps, 1);' part when I tried it

[JustinRoad]

Well, I'm fairly certain it's min(x,y)

 

For example: The only possible mipmaps on 512x128 are:

 

512x128

256x64

128x32

64x16

32x8

16x4

8x2

4x1

 

There are 8 of them. If you were to use max(512,128) your function will return 10, which is incorrect in this case

 

The number that this returns includes the base mipmap level.

 

So to re-iterate. to calculate the number of mipmaps, this is what my functions would look like (just in case I've confused the issue )

 

function LogBase2(Value: Integer) : Integer

begin

case Value of

512: LogBase2 := 9;

256: LogBase2 := 8;

128: LogBase2 := 7;

64: LogBase2 := 6;

32: LogBase2 := 5;

16: LogBase2 := 4;

8: LogBase2 := 3;

4: LogBase2 := 2;

2: LogBase2 := 1;

1: LogBase2 := 0;

default: { what is the default handler for a case statement? I forgot! }

ReportError("This function should only be called with power of 2 values!");

end;

 

end;

 

function GetNumMips(Width, Height : integer) : Integer

begin

Smallest : Integer;

NumMips: Integer;

 

Smallest := min(Width, Height);

 

GetNumMips := 1 + LogBase2(Smallest);

 

end;

 

Did that help?

[Benny]

I should make new posts rather than keep editing my old ones. This is what I have at the moment, and it seems to work ok.

        if (mipmaps=0) or (mipmaps > 20) then

        begin

          temp:=min(width, height);

          case temp of

            512:  mipmaps:=10;

            256:  mipmaps:=9;

            128:  mipmaps:=8;

            64:   mipmaps:=7;

            32:   mipmaps:=6;

            16:   mipmaps:=5;

            8:    mipmaps:=4;

            4:    mipmaps:=3;

            2:    mipmaps:=2;

            1:    mipmaps:=1

            else

              Mipmaps:=1;

          end;

        end;


        total:=GetDDSSize(1, Mipmaps,  (Width * Height) * 4);

[/Code]

 

It looks like it does the same as yours, thank you for all your help.

Now it seems that some files have their width/height specified within the dds data, not within the header. Which either means that they have a different (and larger) header to the other files, or..they are just weird :~

[Benny]

All texture format 0:

 

common.ppf

(1st file)

joystick01.dds

64*64

actual size=14628

 

In almost every other .ppf:

 

textures\fxtextures\flames\flickera_00.dds

64*64

actual size=60836

 

flames_00.dds

64*128

actual size=88016

 

Their second ID No is always 0 and mipmaps always =1. What makes these files special is that their width/height comes straight after the 44 byte header. This means that the 'actual size' values I've used here may be out by 8+ byes as the header may be bigger.

 

I cant see a way of getting the size from these values. As its texture id 0 it should be (w*h)*4 but this is obviously wrong. For now I just hardcode the sizes for these particular files and seek past them, hopefully there wont be any other files of this type.

 

[Edit] Unfortunately it seems like quite a few files use this 'other' format.

[JustinRoad]

thank you for all your help.

 

You're welcome. It's kewl what you're doing. Keep it up

 

textures\fxtextures\flames\flickera_00.dds

64*64

actual size=60836

 

I have a suspicion that this format is an 'animated' format - meaning there are several frames of animation in the actual size there. Can you confirm this? ie. is there any reference to flickera_01.dds anywhere in the stream?

 

In your header might be some sort of animation frame count. I suspect you're looking for either a '0x0d' or '0x03' (depending on the format of the picture)

 

edit: Also, just to be sure. Is there maybe a number in the stream/header that looks like it's very very close to the 'actual size'. This may help particularly as DDS files have a field dwLinearSize which, if not set to zero, could easily help you skip these files.

 

It may also be worth looking at the DDS file format. If you have access to the D3DSDK, lookup DDSURFACEDESC2. A DDS file is simply the bytes 'DDS ' followed by a DDSURFACEDESC2 structure followed by the image data. Check out the dwFlags to check whether the CAPS fields, PIXELFORMAT field and (most importantly) the dwFourCC field are set. There is a possibility that this type of file also contains different surfaces (Eg. zbuffer stuff). You may have to start parsing DDS files. If the dwFourCC is not still something like 'DXT1','DXT3','DXT5' you need to find out what it is.

 

Failing that, you could always post the chunk of data in question somewhere and I could take a look at it

[Benny]

It looks like they are animations like you said. But the frames are all within the same file. I've only looked at

ui_joystick_01.dds right now but:

 

The file sizes in my previous post were wrong, in this other format the header is 72 bytes not 44 as before.

 

Header:

2 bytes - No of frames ?

2 Bytes - ID2 - (always 0)

40 bytes ?

4 bytes - width

4 bytes - height

4 bytes - mipmaps

16 bytes - ?

 

Then comes a normal block of texture data - the size of this is worked out from the texture ID as usual. I havent yet worked out where in the header this is stored. For ui_joystick_01.dds its texture id 9.

 

Then for the remaining frames:

A normal 44 byte header

Texture data

[Benny]

I can now parse all the .dds's (except a few in common.ppf which I will fix later). The next step is to clean up my messy code and then work out which texture id corresponds to which dds format so I can recreate the dds'

[Benny]

I spoke too soon. I can parse all the dds' in the pc version, but the xbox version has another texture id - 14.

 

In asco.ppf (xbox):

 

textures\lightmaps\asco\lightmap000.dds (yes that file again).

ID = 313

ID 2 = 33119

Texture ID = 14

Width = 256

Height = 128

Mipmaps = 9

Actual size = 44716

 

 

I'm having problems working out the size with this texture id.

If its not a DTXx compression:

256 by 128 = 32768

128-by-64 =8192

64-by-32 = 2048

32-by-16 = 512

16-by-8 = 128

8-by-4 = 32

4-by-2 = 8

2-by-1 = 2

1-by-1 = 1

total=43691

 

 

If its a DXT compression then it should use the non-square dxt method ( max(1,width ÷ 4)x max(1,height ÷ 4)x 8 (DXT1) or 16 (DXT2-5) ):

256*128 = 32768

128*64 = 512 * 16 = 8192

64*32 = 128 * 16 = 2048

32*16 = 32 * 16 = 512

16*8 = 8 * 16 = 128

8*4 = 2 * 16 = 32

4*2 = 1 * 16 = 16

2*1 = 1 * 16 = 16

1*1 = 1 * 16 = 16

total=43728

 

Its not massively off with either method, but obviously its not correct. Maybe JustinRoad has an idea?

 

I've attached the file to this post.

xboxascolightmap.zip

[JustinRoad]

As far as I can figure out, it is a paletted image.

 

The breakdown is as follows:

 

at 0x007e in the file is a 16 bit number. Not sure if this specifies whether there is a palette or not. In the data you gave me this is 0x0001.

 

at 0x0080 is where the palette begins. Each entry is 4 bytes (r,g,b,a) and there are 256 entries. After this is where the index data begins (I think )

 

So, your actual size is comprised as follows:

44716 =

 

2 bytes for 16 bit number (is there a palette?)

1024 bytes for palette

32768 bytes for 1st mip

8192 bytes for 2nd mip

2048 bytes for 3rd mip

512 bytes for 4th mip

128 bytes for 5th mip

32 bytes for 6th mip

8 bytes for 7th mip

2 bytes for 8th mip

(there is no 9th mip)

 

At least I don't think there is a 9th mip. A 256 by 128 image will only have 8 mips so that 9 must mean something else.

 

How did I figure this out? I looked at the data in my favourite hex editor (Visual C ) and found the palette chunk quite remarkable (with all those 0xFF's in the alpha channel for all the palette entries).

 

Let me know if it works. I haven't actually done the extraction myself and checked it out - that's what you're for

[Benny]

Nice one!

 

I've not tested it with all the xbox files yet but I can parse through Xbox Asco.ppf now.

 

The 2 byte value at comes directly after the usual 44 byte header.

If it = 256 or 1 - then there's the 1024 byte palette, otherwise there isnt.

 

The number of mipmaps is still correct for the most part, however (as with texture id 0) you need to run a check if its a non-square image:

         if (mipmaps=0) or (width<>height) then
         begin
           temp:=min(width, height);
           case temp of
             512:  mipmaps:=10;
             256:  mipmaps:=9;
             128:  mipmaps:=8;
             64:   mipmaps:=7;
             32:   mipmaps:=6;
             16:   mipmaps:=5;
             8:    mipmaps:=4;
             4:    mipmaps:=3;
             2:    mipmaps:=2;
             1:    mipmaps:=1
             else
               Mipmaps:=1;
           end;
         end;

 

Thank you for this, I wouldnt have thought about a palette. I'll start testing with the other files now.

 

I've also made progress on checking which texture id corresponds to which DDS type.

Texture ID:

0 = 32 bit A8R8G8B8 (8:8:8: argb)

9 = DXT1

10 = DXT3 or DXT5

11 = DXT3 or DXT5

12 = ??

14 =

I'm not sure which of the 2 DXT's 10 and 11 correspond to. When I dumped to both DXT3 and DXT5, the image looked the same in both cases.