The Kotor lightmap format

[magnusll]

So, yesterday I was adding a few random odd and ends to the 0.2 version of KAuroraEditor, and after that I started to clean up a few bits of code who were messed up due to several rewritings. Among those bits was the definitive handling of one tiny little field in the .mdl trimesh header which I had previously almost decoded, identifying the values it could take and noting how it depended on textures and a few other things.

 

The field I'm talking about is the one immediately after the size of the MDX data structure in the header; you'll find it as field nr. 43 in CChargin's doc; mdlops doesn't have any specific name for it while reading, and when writing it chooses between the values "33" if you have no texture on the node, and "35" if you have one. While this is good enough for textures, it is not good enough for lightmaps.

 

That field is a bitmap which says which data you're going to find in the MDX file for that particular node. The .mdx files in Kotor contain an array of structures, with one element for each vertex of each node in the corresponding .mdl file. Since you can have different kind of nodes in an .mdl, you need to know for each node what kind of additional data has been written to the .mdx. That is what that field is all about. It goes something like this:

 

  1    2    4    8   16   32   64  128
  X                                        mdx has vertex coordinates
       X                                   mdx has texture 1 UV coordinates
            X                              mdx has texture 2 UV coordinates
                 X                         mdx has texture 3 UV coordinates
                      X                    mdx has texture 4 UV coordinates
                           X               mdx has vertex normals coordinates
                                X          ? never used
                                     X     mdx has unknown struct (32 bytes)

 

You see, each Kotor model can have from 0 to 4 textures assigned to it. To the best of my knowledge, there's no model in Kotor I using more than 2 textures, so textures 3 and 4 are never present. I have no access to Kotor II files from my development PC, but I doubt Kotor II models use the last two textures, either.

 

Texture nr. 2 is always used as the lightmap (when present). Note that each texture is independent of each other - you can have a lightmap (texture 2) without having the normal texture (nr 1). When both are present, the Kotor engine simply combines them by multiplying the RGB values of them to get the vertex final color - and subsequently interpolates colors for each pixel between vertices.

 

Yesterday, after having cleaned up that particular bit of code, I also modified the code I was using to read in UV coordinates from the .mdx file, taking advantage of the newly discovered info. Lo and behold... my little renderer was finally able to visualize area models in much better ways than before; now the colors actually make sense and strongly resemble the ones you get in the game.

 

Bottom line: the Kotor lightmap format is actually ridiculously simple; it is just an additional set of UV coordinates in the .MDX file. Nothing less, nothing more. E.g. assuming you have a node with all the above mentioned data present, you'd have something like this for each of the node's vertices in the mdx file:

 

0x000 vertex coordinates (3 float)

0x00C vertex normal coordinates (3 float)

0x018 texture 1 UV coordinates (2 float)

0x020 texture 2 (lightmap) UV coordinates (2 float)

0x028 unknown 32-bytes struct (definitely looks like 8 float)

 

And this is it, basically. If you have a lightmap ready for your node, all you need to do is to set the 3rd bit on the field, and fill-in the UV coordinates in the MDX file.

 

Now for the bad news:

 

A) one problem I have is that I can't generate the lightmaps in the first place. Lightmapping belongs to rendering, and all the rendering capabilities were stripped off from 3DSMax when it was converted to GMax. So while 3DSMax is perfectly able to create and handle lightmaps, GMax is not... and I'm certainly not going to fork thousands of dollars to purchase the bigger brother.

 

I'll look into the possibility of working with Yafray, a free rendering engine, to create and import lightmaps into Kotor models using only free products. Meanwhile, if anyone knows the format used by NWMax for 3DSMax to export / import models having lightmaps in them, post (or send me) a brief example and I'll see what I can do to write something usable with it.

 

B) I don't think this is the problem which makes scratch-built areas black. As I said in the other thread, there are many area models which do not have lightmaps associated with them, and yet you can see them normally in the game. So there has to be something else which creates the problem.

[Gargoyle King]

Wow, so after all that the lightmap format was surprisingly really simple? Good find magnusll. Now all we need is someone to actually "create" a lightmap and see if the theory works. Very good find, this is indeed a great day for the community, keep up the hard work on your nifty little app!

[H2O]

I must say, this is really exciting stuff. It feels like the last few 'untouchable' file formats in KOTOR are about to become moddable...

 

The main reason for my post is that, by coincidence, I was browsing around on this stuff just this evening. Here's some of what I found, perhaps you may find it helpful for your 'Problem A'.

 

I was particularly looking at Blender and its ability to import/export KOTOR files. Blender is a massive open source 3D modelling software project. From what I can tell, it's really only been mentioned here and there as a 'potential' KOTOR model editor, I don't think it's found much practical modding use (if any at all).

 

Although I'm still very new at Blender, my thought was that it could potentially be used to edit MDLs and MDXs (perhaps even WOKs and AREs) in a more direct and integrated way. There seems to be a lot of messing-about with files to get around GMAX's limitations; Blender is designed to be expandable, and so readily accepts python scripts for importing and exporting.

 

After reading your last post, a quick Google led me to this page, which confirmed that Blender can handle lightmapping. The only difficulties would be loading the mdl/mdx data into it, and learning how to use Blender.

 

It seems quite conceivable that Blender could be used, especially with scripts like NeverBlender already existing. However it is quite different to GMAX, and with these kinds of software unfamiliarity is usually quite an issue.

 

Anyway, just thought you might be interested

[Gargoyle King]

Good find H20, i'm sure magnusll will look into this. Besides, any alternative rather than having to fork out thousands of your hard earned cash for one program is definately a bonus. Oh, and welcome to LF anyways.

[Quanon]

Blender is BIG .

 

And is used by lots of people , I remember it was used a lot to model for Morrowind .

 

I think the main Script/Program language is called Phyton .

 

The only "downside" to Blender is his render engine .

 

Max has MentalRay and for example RenderBoy or V-Ray , note you need to pay for these last two renders , but they're really amazing on how to creat reallistic reflections ; daylight and such . Not that Kotor uses any of this stuff.

 

Allthough I'm sure Blender could offer you a good enough Renderer for Kotor .

 

And it is supported by a big community , lots of tutorials can be found .

 

I think this would be an exellent choice , allthough you might not find a NW-blender tool .

 

Though you can ealsy export your model as a .obj file with G-Max or Max , put it in Blender, run the Renderer + somesort of Script wich gives you the Needed file/files .

 

It's a bit of a détour .

[Gargoyle King]

Though you can ealsy export your model as a .obj file with G-Max or Max , put it in Blender, run the Renderer + somesort of Script wich gives you the Needed file/files .

 

It's a bit of a détour .

So Blender does actually have the ability to render new lightmaps with aid of G-max etc, if so this is indeed good news as AuroraEditor 0.2 could have the ability to create workable custom built areas in Kotor & TSL.

[magnusll]

I do know about Blender. The problem is that GMax comes with all the import/export scripts already done thanks to NWMax, while Blender doesn't and I don't want to rewrite all that from scratch. So it isn't a matter of familiarity, it's a matter of laziness

 

I didn't know about NeverBlender, and it certainly could have helped... but looking at the web site, there are no updates since 2005, and in the FAQ it states that he couldn't get it to work with Windows... so I would probably end up spending more time trying to fix it than rewrite everything from scratch.

 

This is why I was looking at Yafray (which, BTW, being a dedicated render engine looks like it can kick serious ass in the lightmapping department... though people more knowledgeable than me can give better opinions). The idea was to use it just to create lightmaps so I only had to write the exporter/importer for those, leaving everything else untouched. Besides, it's already half-integrated with GMax thanks to YAGMax, a GMax utility designed to bridge the two (or so I understand... I still haven't used them though).

[Gargoyle King]

This is why I was looking at Yafray (which, BTW, being a dedicated render engine looks like it can kick serious ass in the lightmapping department... though people more knowledgeable than me can give better opinions). The idea was to use it just to create lightmaps so I only had to write the exporter/importer for those, leaving everything else untouched. Besides, it's already half-integrated with GMax thanks to YAGMax, a GMax utility designed to bridge the two (or so I understand... I still haven't used them though).

Well whatever is easier for you magnusll as you are the one making all this possible!

 

It seems that Yafray would indeed be the better option then. If the lightmapping works out magnusll will your app be able to export full custom areas into kotor, i understand there will be bugs since the apps in it's alpha stages but if an area can be exported into kotor, even a buggy one would still be pretty awesome IMO.

[Quanon]

@magnusll : It's your program , go at what you feel is the best choice .

 

I just knew Blender is an open project with lot of support , but hey if you can build some-sort of Render into Kaurora that would make things lots easier and even user-friendlier .

 

Setting up your render in Max isn't that easy going, I even know the but a small part of it or what it is exactly for .

 

I got V-Ray , but almost never use it , as the options for this renderer are so overwhelming .

 

And I'm sure most people prefer a simple program with "clear" and understandble options .

 

And just curious if you know something about the shaders Kotor uses .

 

I played around with Shaders for Jedi Academy ( Quake3 engine based) , you can create some neat looking lights with those or pretty realistic looking suns.

 

But so far the only shader I know of for Kotor is CM_BareMetal , and I haven't found a list , if there exists one .

 

Cause knowing the shaders, could help to create nice new areas

[stoffe]

But so far the only shader I know of for Kotor is CM_BareMetal , and I haven't found a list , if there exists one .

 

Cause knowing the shaders, could help to create nice new areas

 

As far as I know CM_Baremetal isn't a shader, it's an environment map texture (whose TPC file you can find in one of the texture packs along with a few others).

 

Here is some info about the TXI data, it's for Neverwinter Nights but it mostly works the same in KOTOR from what I know.

[magnusll]

If the lightmapping works out magnusll will your app be able to export full custom areas into kotor, i understand there will be bugs since the apps in it's alpha stages but if an area can be exported into kotor, even a buggy one would still be pretty awesome IMO.

 

I'll repeat what I wrote before: as far as I can see, you do not really need lightmaps to have a working scratch-built area. Lightmaps are not necessary; they are just a nice addition. Something else is causing the "blackness" problem; most likely some mismatch due to importing differing geometries into areas which where calibrated on the original models. The current best bet to get a working scratchbuilt area is to copy the three main GFF files (.are, .ifo and .git) from some very simple area like the Rakatan mind prison, modify them to point at your newly-created .mdl, .mdx and .wok files (which should be done with a somewhat similar overall size but different geometry to the original), and see what happens.

 

As things stand, I duplicated most of mdlops "reading binaries" functionality in my utility; now I have to at least write corresponding "reading ascii" and "writing binaries" functions which will allow me to import Gmax models into Kotor. Then I'll be able to

 

A) put the newly-found mdl knowledge to use

B) see for myself what happens

 

So far things are looking good, but let's not jump ahead of ourselves until there's at least one working new area produced...

[southern_fox]

So far things are looking good, but let's not jump ahead of ourselves until there's at least one working new area produced...

 

I entirely agree here.

 

Outstanding work.

[Quanon]

Thanks Stoffe for the info

 

Was wondering what caused these effects in the game , in most games they use shaders and such .

EDIT : Is there some sort of effect you can apply in the .txi to push your texture to infinity?

 

 

About the new area , I'm doing something wrong in the Hex-editing .

Kotor keeps freezing / crashing when I warp to my new area .

 

He tries to save , but just freezes in the proces .

[Miltiades]

About the new area , I'm doing something wrong in the Hex-editing .

Kotor keeps freezing / crashing when I warp to my new area .

 

He tries to save , but just freezes in the proces .

 

Could be something else. I have this sometimes when loading my own module, but that's because I forgot to rename the .are files back to the name of my module (When editing with moduel editor, you have to use the original name of the .are used in the game).

 

But yeah, it's probably Hex-editing that went wrong somewhere, I can imagine Hex-editing to be very difficult when making a new area.

[Gargoyle King]

I'll repeat what I wrote before: as far as I can see, you do not really need lightmaps to have a working scratch-built area. Lightmaps are not necessary; they are just a nice addition. Something else is causing the "blackness" problem; most likely some mismatch due to importing differing geometries into areas which where calibrated on the original models. The current best bet to get a working scratchbuilt area is to copy the three main GFF files (.are, .ifo and .git) from some very simple area like the Rakatan mind prison, modify them to point at your newly-created .mdl, .mdx and .wok files (which should be done with a somewhat similar overall size but different geometry to the original), and see what happens.

 

As things stand, I duplicated most of mdlops "reading binaries" functionality in my utility; now I have to at least write corresponding "reading ascii" and "writing binaries" functions which will allow me to import Gmax models into Kotor. Then I'll be able to

 

A) put the newly-found mdl knowledge to use

B) see for myself what happens

 

So far things are looking good, but let's not jump ahead of ourselves until there's at least one working new area produced...

Agreed!

I suppose lightmaps aren't needed as such as the Rakata mind prison has noe, yet that is a walkable area.