[Feedback needed] merging TLK files or not

[K4thos]

This time no poll since the discussion requires some knowledge regarding how the engine and mods work. I'm wondering what are the actual benefits of using SAY command for whole game resource importation rather than simply merging 2 TLK files into one. Here are my thoughts.

Benefits of using SAY:
- smaller TLK size (in vanilla EET case that would be 16,50 MB vs 24,76, so 8,26 MB difference)
- SAY command automatically tries to attach already present TLK entry (which most of the time is useless due to slight differences in descriptions between BG:EE and BG2:EE
- ? Can't think about anything else

Benefits of merging TLK files together:
- when it comes to vanilla content we would always have valid TLK entries. Which means the game would behave exactly like vanilla games - you can share vanilla EET saves and they will always work as it is for everyone (even if someone did installation with different language)
- modders could reference actual BG1 quest entries rather than including them in TRA files since the TLK number would be always the same. Same for updating items etc. Probably not much of benefit considering mods works with multiple platforms at once, so it's unlikely that someone would care about such a thing
- well, dunno...

Did I missed something important? Does larger TLK file affects the engine in negative way? If no are there any real benefit in using SAY command in EET? Form the code perspective I can switch to merged approach in a matter of minutes (just swap SAY with code that adds amount of TLK entries in BG2:EE to existing BG:EE TLK value). And I can't really see any negatives when it comes to mod compatibility (probably no one would ever notice it if not for this topic). So, is it worth it?

---------

Regardless of the above stuff whatever system will be used in the end (since it doesn't really changes anything important) I think for content importation it's a good idea to prepare an array that would automatically redirect at least common descriptions from BG1 to always use BG2 TLK entries (so we don't have 2 slightly different versions of the same string of let's say Bastard Sword that are not related to lore but simply different people responsible for updating general descriptions or even same people but at different time - pretty much every single item description in EE games has been adjusted by Beamdog one way or another). I can generate such array automatically via weidu code using the same base items from both games and later use them all the time by comparing BG:EE TLK string with value existing in array. Unless again someone think that keeping SAY implementation for general descriptions too should be kept.

Please, share your thoughts.

 

edit: there is one more SAY command benefit - it does quite a good job at matching string references in spell effects (for those strings that are not yet recognized by EET automatically, although the lack of noticeable problems in game may be related to new EE commands that can provide protection for whole spell including messages and remove both at once too). I'm planning to prepare more accurate spell effects patching either way using a function that will automatically add missing strings and swap existing entries using long ass array listing all common effects from game matched with particular opcode (similar approach to what lolfixer does for missing spell effect icons when you install it). Just in case since this will require quite a lot of manual work - no one ever did something like this, right?

[argent77]

This appears to be primarily an internal design decision for EET. Does any of the two approaches cause additional compatibility issues with mods?

Edit: Would the merge option allow you to install EET for all available game languages at the same time (provided they are present in both games)?

[Roxanne]

I'm planning to prepare more accurate spell effects patching either way using a function that will automatically add missing strings and swap existing entries using long ass array listing all common effects from game matched with particular opcode (similar approach to what lolfixer does for missing spell effect icons when you install it).

This was a bad idea way back with lolfixer in BGT as it was not just correction but a tweak disguised as fix - however, the issue was that lolfixer did it at the very end of BGT thus also *correcting* items/spells/effects that were intentionally designed in a certain way by modders. Hopefully you do not intend something similar,

[K4thos]

This appears to be primarily an internal design decision for EET. Does any of the two approaches cause additional compatibility issues with mods?

nothing I can think of. Let's see if someone else will be able to find a problem with this approach.

 

Edit: Would the merge option allow you to install EET for all available game languages at the same time (provided they are present in both games)?

well, yes, extracting TLK to TRA and creating TLK back is a matter of seconds (and the only thing we do with TLK is appending it as it is at the end of BG2:EE TLK file). There shouldn't be even a problem with appending parts of TLK from one language to another language in case of missing strings if that's what you need. But what would be a purpose of it? If there is a good reason for it than it could done by default since it wouldn't really slow down the installation.

 

This was a bad idea way back with lolfixer in BGT as it was not just correction but a tweak disguised as fix - however, the issue was that lolfixer did it at the very end of BGT thus also *correcting* items/spells/effects that were intentionally designed in a certain way by modders. Hopefully you do not intend something similar,

not sure what's wrong with lolfixer implementation (it fixed like 100 of such issues in DSotSC). His implementation seems even more useful for EE games where you can disable all effects of the spell with a single opcode. Either way handling strings asociated with status effects etc. in a way I've described, lasting exactly as long as that effect opcode shouldn't have negative effects. Why would this be a problem? This way we are always sure that what's imported by EET from BG:EE follows exactly BG2 TLK references for opcode stuff, which is expected by mods installed on BG2:EE.

[Roxanne]

ot sure what's wrong with lolfixer implementation

The problem was that in BGT lolfixer was installed similar to your EET_end. This way it was as well affecting all mods that were installed after the games were merged and was erroneously *correcting* things that were correct, i.e. mods specifically designed to work on top of the merged games.

So the problem was mostly due to install order (which was defining the scope of what the tool was changing.) No problem if EET does it in its first part, to align BGEE with BG2EE, may repeat the lolfixer problem if done in EET_end.

[Jarno Mikkola]

may repeat the lolfixer problem if done in EET_end.

Are you sure the problem was not that the fact that he was just a one mod maker with the idea, and that there was like a 1000 other mods already installed. Trouble shooting is a biach in that situation.

[K4thos]

 

ot sure what's wrong with lolfixer implementation

The problem was that in BGT lolfixer was installed similar to your EET_end. This way it was as well affecting all mods that were installed after the games were merged and was erroneously *correcting* things that were correct, i.e. mods specifically designed to work on top of the merged games.

So the problem was mostly due to install order (which was defining the scope of what the tool was changing.) No problem if EET does it in its first part, to align BGEE with BG2EE, may repeat the lolfixer problem if done in EET_end.

 

ah ok make sense. This is indeed intended only to handle BG1 resources during core component installation.

[GrimLefourbe]

Wouldn't appending tlk pose problems for mods that try to modify description or mods that should apply to Bg1 and bg2 ? If I understand correctly, this would make for a huge number of duplicate entries and every mod that wants to change one of those would have to change 2 versions of the same string ? They could maybe just add new strings and reattach them, however it would leave a lot of unused strings in the tlk.

 

I'm no expert though so it's just a quick question for people with more knowledge.

[Roxanne]

Wouldn't appending tlk pose problems for mods that try to modify description or mods that should apply to Bg1 and bg2 ? If I understand correctly, this would make for a huge number of duplicate entries and every mod that wants to change one of those would have to change 2 versions of the same string ? They could maybe just add new strings and reattach them, however it would leave a lot of unused strings in the tlk.

 

I'm no expert though so it's just a quick question for people with more knowledge.

When developping and testing my mod, I continously install/re-install parts of the game because I have added dialogue lines, corrected typos etc, Whenever I do this, a new entry in dialogue tlk is added, and the incorrect ones become now unused (nothing is deleted from the file in the process). I always use very large installations with a very large dialogue.tlk file. i have never in all years encountered issues that could be accosted to the size or number of unused stuff in that file.

So the only reason to deal with the issue (in my opinion) would be consistency.

[K4thos]

Wouldn't appending tlk pose problems for mods that try to modify description or mods that should apply to Bg1 and bg2 ? If I understand correctly, this would make for a huge number of duplicate entries and every mod that wants to change one of those would have to change 2 versions of the same string ? They could maybe just add new strings and reattach them, however it would leave a lot of unused strings in the tlk.

 

I'm no expert though so it's just a quick question for people with more knowledge.

All common descriptions for items that exists in both games are meant to use BG2 string numbers assigned via pre-made array, as mentioned in the first post, so it would be very compatible with mods that simply use STRING_SET to alter general descriptions. GUI already uses BG2 strings. Spell effects will use BG2 strings (current implementation with SAY is not always reliable). So what's left? Actual dialogue and text content. In which you are able to use STRING_SET, GET_STRREF or even reference dialogue strings with # reliably since you know exactly the BG1 TLK string value in relation to BG:EE, so you could for example update entire BG1 translation of the game even though the game has been imported to EET. Can't see any problems here. You would need to show some mod as an example how it would negatively affect it.

[GrimLefourbe]

I don't know actually, I was just trying to find issues but it seems there is already good solutions for these so I haven't got any good example. I was a bit worried about issues similar to the ones item revision has with the list based components. It sounds like they would only happen in case of archaic mod code which shouldn't be coded around imo.

[Mike1072]

One thing that springs to mind if you do merge the TLK files is that the string references may not be completely static if you append the BGEE ones after BG2EE - there's the potential that a new patch for BG2EE could add additional strings. Still, I like the idea.

[Jarno Mikkola]

One thing that springs to mind if you do merge the TLK files is that the string references may not be completely static if you append the BGEE ones after BG2EE - there's the potential that a new patch for BG2EE could add additional strings. Still, I like the idea.

And at that point you'll also need to reinstall all the mods, including EET. So...

[K4thos]

One thing that springs to mind if you do merge the TLK files is that the string references may not be completely static if you append the BGEE ones after BG2EE - there's the potential that a new patch for BG2EE could add additional strings. Still, I like the idea.

That's indeed a valid concern considering patch 2.x added 167 new strings to BG2 (related to new kit, spells, story / legacy of bhaal modes and item importation from SoD). Currently BG2:EE with latest patch has 103213 TLK strings. In order to ensure future compatibility with saves we would need to assign some dummy entries at the end of file to reserve strings for future official changes in BG2:EE TLK. Not sure how many though. If there won't be SoD like expansion released for BG2:EE than I think we could automatically reserve strings for let's say up to 103499. So BG:EE TLK file would always start from string 103500. Or we could reserve even more if 286 is not safe enough prediction of what may be needed in future by BG2:EE.

[subtledoctor]

Out of curiosity, why reserve up to 103,499? Why not start the BGEE strings at a nice round number like 200,001? Then any changes modders have to make to BG1-specific strings can simply be changed to add a 2 and some zeros before the old string.

 

I know that's kind of trivial, but that kind of trivial standardization can actually be helpful to modders. (Also it would leave tons of room for the small chance that Beamdog does add some kind of content expansion to BG2EE.)

 

Is there a performance or other reason to try to keep the strref range low?