Jade Empire modding (see first post for info summary)

[stoffe]

Index - useful resource links:

 

JE Modding tools (so far):

• TalkEd v1.1.1a - TLK V4.0 editor for Jade Empire (for modifying all game text in dialog.tlk)
   
   
  
• ERF/RIM Edit v0.3.6a - ERF/MOD/RIM Editor for Jade Empire (with correct file type associations)
   
   
  
• JEFindRes v0.3a - Search tool to look for files with names matching the search pattern within all the game's BIF, MOD and RIM files (and the override folder for completeness). Allows extraction of found files and converting images from TXB format to TGA. (See this post for more info)
   
   
  
• Txb2Tga v0.1a - Converts TXB textures to a TGA and TXI file to allow viewing and modifying textures and images.
   
   
  
• JE DLGEditor v2.3.0a2 - Dialog editor for Jade Empire, alpha release (See this post for more info )
   
   
  
• Jade Empire Savegame Editor v0.6 - A basic savegame editor for Jade Empire (see this post for more info)
   
   
  
• SAC Utils - A save.sac ripper and mender in case you want to try manual GFF editing of your savegame.
  

 

Other Modding tools that work with JE:

• Scripts can be compiled with the KOTOR-adapted versions of nwnnsscomp using nwscript.nss from JE's override folder.
   
   
  
• 2DAs can be edited with any text editor if they're in 2DA V2.0 format. If they are in the 2DA v2.b binary format they can be edited with KotorTool or tk102's Excel Add-in or converted to 2DA V2.0 with my Convert2da utility.
   
   
  
• K-GFF v1.2.0 and above seems to work to edit the GFF files from Jade Empire. Make sure you don't use an older version since the new data type JE uses would not be supported then, resulting in data corruption when saving.
  

 

 

Modding info, tutorial and how-to posts:

• Making new gems, effects and techniques
  
• Modifying the combat system
  
• Model format information
  

 

 

Released JE Mods in this thread:

• stoffe's "White" Radiant Jen-Zi reskin
   
  
• sekan's "Three Brothers Gems"
   
  
• sekan's "Elephant Demon Style & Moon Light Weapon Style"
  

 

 

 

[align=center] * * * [/align]

 

 

Original post:

 

Since I just got Jade Empire for PC I thought I'd try to update my RIM/ERF editor to handle the Jade Empire files to make modding it a little easier. The ERF and RIM file formats seems to be mostly the same, though the ResType <--> File extension lookup table seems to have changed, both changing the meaning of some existing ResTypes, and adding a bunch of new file types to it.

 

I've been looking through various RIM and MOD files so far trying to figure out what the different files are. So far I've only been able to figure out this fairly short list. Do anyone know what other file type <--> ResType associations there are, or what the new files are used for?

 

Edit: Found a list, so hopefully the table should be complete now. Still need to figure out the purpose of some of the files, and what format they are in though. (Some may be leftovers from Neverwinter Nights / Knights of the Old Republic and unused in Jade Empire.)

 

ResType Type    Comment/speculation
------- ----    ---------------------------------------
0       RES     ???
1       BMP     Bitmap image
2       MVE     ???
3       TGA     Targa image file
4       WAV     Wave sound
5       WFX     Wave extra info
7       INI     Configuration file (plaintext)
8       MP3     MP3 sound
9       MPG     MPG video
10      TXT     Text file
11      PVD     Binary data (Pointer to Void Data).

2000    PLH     Polyhedron
2001    TEX     Texture
2002    MDL     Model data (MDL) [used along with MDX file with the same name]
2003    THG     Thing (???)
2005    FNT     Font
2007    LUA     Lua script
2008    SLT     Slot file
2009    NSS     Script sourcecode (plaintext)
2010    NCS     Script bytecode (NCS)
2011    MOD     Module (ERF)
2012    ARE     Static area info (GFF)
2013    SET     Tileset file
2014    IFO     Module information (GFF)
2016    WOK     Area Walkmesh
2017    2DA     Data table (plaintext or 2DA V2.b binary)
2018    TLK     Talk table file (TLK)
2022    TXI     Extra Texture Info (plaintext)
2023    GIT     Game Instance File (dynamic area info) (GFF)
2024    ITM     Item template (GFF)
2026    CRE     Creature template (GFF)
2029    DLG     Dialog file (GFF)
2030    PAL     Palette (GFF)
2031    TRG     Triggers (GFF)
2033    DDS     DXTC Compressed texture
2034    SND     Sounds (???)
2036    LTR     ???
2037    GFF     Generic GFF file (GFF... duh)
2038    FAC     Faction file (GFF)
2039    ENC     Encounter template (GFF)
2040    CON     Container template (GFF)
2041    DOR     Door template (GFF)
2043    PLA     Placeable template (GFF)
2045    DFT     ???
2046    GIC     Game Instance Comments (area editor comment data)
2047    GUI     Graphical User Interface definition file. (GFF)
2048    CSS     Conditional Source Script
2049    CCS     Conditional Compiled Script
2050    MER     Merchant/store template (GFF)
2052    DWK     Door walkmesh
2053    PWK     Placeable walkmesh (plaintext)
2054    GEN     Random Item Generator
2056    JRL     Journal file (GFF)
2057    SAV     Area game state data (GFF)
2058    WAY     waypoint template (GFF)
2059    4PC     Custom 16-bit RGBA texture format
2060    SSF     sound set file (SSF)
2061    HAK     HAK file (ERF)
2062    NWM     NWN Official Campaign module (ERF)
2063    BIK     BINK movie (Bink video)
2064    MIN     MiniGame data (GFF)
2065    NDB     Script debugger data (toolset)
2066    XWB     XACT WaveBank
2067    XSB     XACT SoundBank
2068    BIN     XBOX DSP filter

3000    LYT     Area model layout (plaintext)
3001    VIS     Area visibility graph (plaintext)
3002    RIM     Resource Image File (RIM)
3004    LIP     Lipsync Visemes data
3005    XMV     ???
3006    WMA     ???
3007    CWD     Crowd game object template (GFF)
3008    PRO     Projectile static resource type
3009    AOE     Area of Effect static resouce type
3010    MAT     material file (plaintext)
3011    MAB     material file (binary)
3012    QST     Quest/journal data (GFF)
3013    STO     Store static resource file (GFF) [used with MER file?]
3014    APL     Artist placeable
3015    HEX     Hex grid (plaintext) [appears to be map-related in some way]
3016    MDX     Extra model data (binary) [used along with MDL file]
3017    TXB     Compressed image with TXI data
3018    TPC     Compressed image with TXI data
3019    CAM     ???
3020    SPT     ???
3021    STYLE   ???
3022    FSM     Finite State Machine (FSM) data
3023    ART     [Weather/Light data for area?] (INI/plaintext)
3024    AMP     ??? (binary)
3025    CWA     Crowd Attribute (GFF)
3026    XLS      MS Excel spreadsheet
3027    SPF     NGF: Style profiles
3028    BIP     PATL: Binary LIP. - [lipsync/Sound file?]

9996    IDS     Legacy BG Script File
9997    ERF     Encapsulated file
9998    BIF     BIFF file
9999    KEY     Key file

[oldflash]

Since I just got Jade Empire for PC I thought I'd try to update my RIM/ERF editor to handle the Jade Empire files to make modding it a little easier. The ERF and RIM file formats seems to be the same, though the ResID <--> File extension lookup table seems to have changed, both changing the meaning of some existing ResIDs, and adding a bunch of new file types to it.

That's a good news.

I've done few things like editing some 2da's and make some scripts.

For that I have use 2DA Converter/Merger Tool, ERF/RIM Editor and KotorTool.

Maybe you have a look into .tlk files? All I want is to set description for few items.

[stoffe]

Maybe you have a look into .tlk files? All I want is to set description for few items.

 

I've made a new version of TalkEd that should handle the TLK V4.0 format used by Jade Empire. It seems to work to read and modify the English TLK file with it from my quick testing. Though please keep an eye out for bugs if you use it since I've only had time to do very limited testing. You can download it here for now. (This version only works with the Jade Empire TLK files, not the KOTOR ones.)

 

There is one data field in the TLK file I don't know the purpose of currently. It seems to be a 32 bit unsigned integer which is set to 0xFFFFFFFF (-1 I suppose) for the majority of the TLK entries, though a fair number of them (mostly dialog related strings) has some other value set here, and it seems each value (except 0xFFFFFFFF) used is unique and set for just that one entry.

 

TalkEd just lets you set this number directly (using hexadecimal notation) for the time being until someone can figure out what the purpose of this data is.

 

Edit: I suspect the unknown numeric value refers to names of the numbered BIP/sound files found in for example audio.bif, though I haven't been able to verify this yet.

[tk102]

stoffe, have you tried poking around in the .key/.bif files to fill in the gaps for erf extensions? I found

this thread on Bioware's forums that references a tool that recognizes most of the key/bif extensions for Jade Empire. Just a thought.

[stoffe]

stoffe, have you tried poking around in the .key/.bif files to fill in the gaps for erf extensions? I found

[url=http://jade.bioware.com/forums/viewtopic.html?topic=554641&forum=108&highlight=erf

this thread[/url] on Bioware's forums that references a tool that recognizes most of the key/bif extensions for Jade Empire. Just a thought.

 

I haven't, but the problem I can see with that is that the KEY/BIF format doesn't store file extensions either, as far as I am aware. So if the lookup table for ResID <--> File type has changed for Jade Empire tools made for other games would still output data with the wrong file extension, or no valid extension at all, since it has no way of knowing what those ResID's are supposed to be.

 

That the data files seems to be an even more disorganized mess than KOTOR, with files or the same type or files belonging to the same area scattered all over the place, doesn't make it easier to find all the file types that are used and how they fit together

[tk102]

It was my hope that the tool referenced above (in a broken url link, now fixed) would shed the light on some unknown extensions. A long shot I'm sure, but just a different angle to attack from.

[stoffe]

It was my hope that the tool referenced above (in a broken url link, now fixed) would shed the light on some unknown extensions. A long shot I'm sure, but just a different angle to attack from.

 

Poked around the BIF files a bit, and while they do suffer from the same problem of incorrect/unknown file extensions I managed to figure out a few more things (updated the list in the first post).

 

I've made a JE version of my ERF/RIM Editor that uses the above file extension/ResType table (minus the couple of unknowns in the list) so far. While the list isn't complete it's easier to spot which files aren't properly identified when it doesn't use the KOTOR file associations.

 

If anyone is interested you can download it from the link in the first post in this thread. If you notice any file types that are listed starting with a # followed by a number those are of unknown types. Please let me know what file it it and where (in what file) you found them in those cases.

[oldflash]

That fiels is for sound ("0xFFFFFFFF" value = no sound). I test this by replacing values and it works. However the sound file must be loaded in that module or to be present in override because if I set a sound which is used in other module will do nothing. I haven't test append new entry but I'm sure is no problem with that. This tool is perfect for what I need to finish my mod.

Many thanks stoffe.

Edit: I just found dialog tool and I should test-it tonight.

I like how easy is to attach scripts for diferite events by editing ar_*.2da files.

[stoffe]

That fiels is for sound ("0xFFFFFFFF" value = no sound). I test this by replacing values and it works. However the sound file must be loaded in that module or to be present in override because if I set a sound which is used in other module will do nothing.

 

Ok, thanks for the confirmation. I've updated TalkEd to handle the values as sound reference numbers and not display them in hexadecimal notation.

 

I like how easy is to attach scripts for diferite events by editing ar_*.2da files.

 

Haven't had much time to look at how things fit together yet since I've only had the game installed for roughly a day so far, but that may be very useful if they don't have any debug console in the game (haven't managed to find one)

[stoffe]

While poking around in the RIM and MOD files I found some more file types I managed to figure out with reasonably certainty what they should be. I've updated the list in the first post and uploaded another version of ERF/RIM Edit with the recent changes, see the link in the first post in this thread to download.

 

Any other file types I've not managed to find yet?

[RedHawke]

Thanks for all the effort stoffe! =)

 

Now I just need to get off my lazy behind and install the darn game (DDO is evil... just Evil!)

[mjpb3]

Kind of on topic here, but if someone could find a way to open the PC & NPC textures (they're in txb form) I would be ever so grateful.

 

(If this doesn't fit in here, please excuse me. )

[oldflash]

@stoffe: many thanks again for all this tools.

I have no idea how to use a debug console or what is that

.ndb files are debug information for the script debugger and not necessary for the game to function.

... if this info good for anything.

However, I belive to be possible to send mesajes to log or display by #include debug.

 

@mjpb3: I also look for this and for a decomp. for mdl files

[stoffe]

Kind of on topic here, but if someone could find a way to open the PC & NPC textures (they're in txb form) I would be ever so grateful.

 

The TXB format has some similarities to the TPC format used for the KOTOR games. The header is different, but like TPC it contains compressed bitmaps and optionally TXI data at the end.

 

Don't know if I'm good enough at this kind of programming to be able to make a converter that converts TXB --> TGA directly, but I'll give it a try and see how it goes.

 

 

I have no idea how to use a debug console or what is that

 

I was just looking for an in-game debug console (similar to the yellow "> " you can type commands at in KOTOR1, and the invisible one in TSL), but it doesn't seem like Jade Empire has one. Hopefully most of that can be handled via scripts instead though.

[tk102]

I built a TPC Perl mdoule based on eiz's tpc2tga source code. If you want any assistance let me know.

[stoffe]

I built a TPC Perl mdoule based on eiz's tpc2tga source code. If you want any assistance let me know.

 

Thanks, that will probably be needed if this is going to go anywhere at all. I had a look at the tpc2tga source to get an idea of how to decode the DXT data and was immediately reminded why I strongly dislike C++ (couldn't even get that code to compile, so I'm doing it in Delphi instead).

 

Edit: Nevermind, I was just stupid.

[mjpb3]

Whatever you can come up with would be excellent for me. That's the main thing I ever wanted to mod in Jade Empire--the textures. Thanks for anything you can accomplish.

[tk102]

sub reverse_bytes {
my $in=shift;
return unpack('V',pack('C4',reverse (unpack('C4',pack('V',$in)))));
}

It is indeed a reversal of bytes (0xAABBCCDD becomes 0xDDCCBBAA for example).

 

This is so the $cpx variable in decode_dxt1 can be right shifted 2 bits at a time and disregarded as the new $px is calculated. The blended array is set up in such a way to make sense of the two bits (which are now reversed).

 

see:

for (my $y=0; $y<4; $y++) {
 for (my $x=0; $x<4; $x++) {
   my $px = $cpx & 3;
   $pixels[($ty-4+$y)*$pitch +($tx+$x)]=$blended[$px];
   $cpx >>=2;
 }
}

 

 

I don't have the tpc2tga source in front of me at the moment so I'm just working off the tpc.pm, btw -- the Perl code was a direct (more or less) translation of the C++.

 

Edit: crossed out the part about the 2 bits being reversed -- they aren't reversed -- their position within their bytes are unaltered but the bytes themselves are reordered.

[stoffe]

sub reverse_bytes {
my $in=shift;
return unpack('V',pack('C4',reverse (unpack('C4',pack('V',$in)))));
}

It is indeed a reversal of bytes (0xAABBCCDD becomes 0xDDCCBBAA for example).

 

That reminds me why I'm not too fond of Perl either. I haven't done anything with Perl for over 6 years, so I didn't understand much of that code. (I prefer high-level "almost english" languages like Delphi, C#, PHP, so it feels like I'm in way over my head with this.)

 

The unpack() function takes a string and splits it by token into an array, like explode() in PHP? While pack() takes an array and spits out a token-separated string from it?

 

What does your reverse_bytes() function receive as parameter, a token-separated string (using "V" as separator) containing the three values a texel-struct holds?

 

If so you convert the string to an array, immediately explode the array into a string again (this time using "C4" as token separator instead), reverse-sort the string, turn it into an array again (wouldn't "C4" be "4C" when you reverse the string?), and immediately turns the array back into a string (this time separated with "V" again) which you return? Wouldn't that return an array of 3 values?

 

Huh? I don't follow what this accomplishes. It obviously works, but I still don't understand what it does.

 

(As you don't have the source: The C++ code takes a struct as function parameter containing two 16 bit unsigned ints and one 32 bit unsigned int value... does something with those three values which I have absolutely no idea what, and then spits out a single 32 bit unsigned integer as result.)

 

* * *

 

Another question ... this is called in a loop that loops up to the Y-size of the bitmap while writing the TGA (i is the incrementing value in the loop):

fwrite(file->pixels+pitch*(ysz-i-1), 4, pitch, fp)

 

file->pixels is an array of 32 bit unsigned integers (the RGBA data), pitch is the X-dimension of the image (in pixels), ysz is the Y-dimension of the image

 

What exactly is it writing? file->pixels is an array, but it doesn't supply any array index in that function call... Does the file->pixels+pitch*(ysz-i-1) add pitch*(ysz-i-1) to the memory pointer of the pixels array and then write the four bytes found in that location? Why do this instead of using an array index?

 

Weird lower-level languages I haven't used for years confuse me. Use of pointers always confuse me.

[tk102]

The unpack() function takes a string and splits it by token into an array, like explode() in PHP? While pack() takes an array and spits out a token-separated string from it?

 

No, The pack and unpack functions are Perl's way forcing data to/from certain interpretations like an unsigned long (template: V) or a unsigned byte (template: C). This is needed because of Perl's weak typing mechanisms. pack takes data forces it into a datatype while unpack translates coerced data back into perl scalars.

 

 

What does your reverse_bytes() function receive as parameter, a token-separated string (using "V" as separator) containing the three values a texel-struct holds?

reverse_bytes accepts a perl scalar (which itself was populated from an unpack function with template V). It then packs this value into an unsigned long again, unpacks it into 4 unsigned bytes, reverses the array of bytes, repacks the array of bytes, and then unpacks the result as unsigned long.

 

(As you don't have the source: The C++ code takes a struct as function parameter containing two 16 bit unsigned ints and one 32 bit unsigned int value... does something with those three values which I have absolutely no idea what, and then spits out a single 32 bit unsigned integer as result.)

Anyway the more elegant C++ code appears as this:

inline u32 reverse_bytes(u32 i, int2type<4>) {
return    (i&0xFF000000) >> 24 
	| (i&0x00FF0000) >>  8
	| (i&0x0000FF00) <<  8
	| (i&0x000000FF) << 24;
}

Here you can see better the masking and data shifting involved. The perl code accomplishes the same thing but uses a temporary array of bytes that it reverses and repacks into a long then unpacks to get the numeric value.

 

* * *

[tk102]

Another question ... What exactly is it writing? file->pixels is an array, but it doesn't supply any array index in that function call... Does the file->pixels+pitch*(ysz-i-1) add pitch*(ysz-i-1) to the memory pointer of the pixels array and then write the four bytes found in that location? Why do this instead of using an array index?

Yes that is exactly what it is doing, byte by byte. I believe this syntax was used because in the tpc.cpp file, you can see file->pixels+pitch*(ysz-i-1) is used to assign the values byte-by-byte. I think eiz just used copy/paste of the code as a mnuemonic so he could easily recognize what exactly he looping on.

[tk102]

The fwrite function is writing row by row starting at the end of the array. Think of the array as the image mirrored vertically. (Left-right is maintained but top-bottom is reversed.) I'm sorry for confusing you by saying byte-by-byte earlier.

 

Here's the declaration for fwrite

size_t fwrite(const void *ptr, size_t size, size_t nitems,
   FILE *stream);

We are using size=4, and nitems=pitch (the width in pixels of the iamge), so each fwrite is writing 4*pitch bytes. The fwrite function returns the number of items (not bytes) which should be equal to pitch in our case. The array's memory address begins at file->pixels, so at the beginning of the loop to write the last row from the array the offset would be (pitch*(ysz-1)). The next to last row would begin at (pitch*(ysz-2)).

 

HTH and sorry for not having the concentration earlier.

[oldflash]

I was just looking for an in-game debug console (similar to the yellow "> " you can type commands at in KOTOR1, and the invisible one in TSL), but it doesn't seem like Jade Empire has one. Hopefully most of that can be handled via scripts instead though.

Ah, stupid me... I was thinking to something more complex... I saw a console_font file somewhere.

 

Great news about textures and I must thank you all for efforts.

[stoffe]

The fwrite function is writing row by row starting at the end of the array. Think of the array as the image mirrored vertically. (Left-right is maintained but top-bottom is reversed.) I'm sorry for confusing you by saying byte-by-byte earlier.

 

But still something does not seem right with how I'm understanding this... In the above example each line of the image has 256 pixels... and each pixel is represented by a dword (4 byte integer) holding its RGBA value. In other words each line contains 4 * 256 = 1024 bytes, which you read from the array and write to the file.

 

But each iteration of the loop still only seems to back off pitch (=256) bytes. Now can it back up one row if it only backs up 25% of the data that is being read/written? Shouldn't it back up pitch*4 to get to the next line of data? Does the memory pointer voodoo deal with array elements and not byte positions in memory? I.e. it backs up pitch (=256) elements and not bytes?

 

Edit again: Yes! That seems to have done the trick. Seems like the utility works to convert at least those TXB files to TGA I have tested it on.

[tk102]

\o/ Yay for stoffe! Did you answer your own question then? The *4 part seems like it should be there to me too... but then again I'm not a C++ guru. I would've expected the use of brackets here to denote element iteration. (Don't like having to think at this level with my programming either, stoffe.) I remember this is the area I fumbled through with trial and error also until I got something recognizable. Good job.