Jump to content

Photo

Immutability and encapsulation in mod design


41 replies to this topic

#1 DavidW

DavidW
  • Gibberlings
  • 4707 posts
  • Gender:Male

Posted 18 September 2018 - 05:46 PM

This is a note on a couple of design principles I've been using in my mods in the last few years (since roughly v22 of SCS, I think) and that I thought might be of wider interest, at least for discussion. A lot of this is probably overkill for something like a quest or NPC mod; it's mostly relevant for big complicated multi-component mods that work on multiple game systems. This note assumes you're already familiar with the basics of WEIDU.

The two design principles (both of which, from an academic perspective, come from functional programming) are:

  • Encapsulation: each component of the mod needs to be kept entirely separate from the other components. Nothing that happens in any component should have any effects on anything in another component (except through the effects it has on in-game files). In particular, variables set in one component should be forgotten as soon as the installer moves on to the next component.
  • Immutability: Nothing in your mod folder should change at all when the mod is installed.

(Quite a few mods use encapsulation to some degree, though I don't think many use it systematically. I think I'm the only person who cares about immutability.)

 

Why care about encapsulation?

 

I learned to care about encapsulation the hard way, way back in the prehistory of SCSII testing. Odd situations occurred where the mod would install fine on my computer, and then would throw errors on other people's computers. It turned out that I was installing one component at a time, so that variables set by each component were being forgotten, but my testers were (reasonably enough) installing several at once. Encapsulation is your guarantee that if a component works in isolation, it won't be confused by - or confuse - the wider environment in which it operates.

 

Why care about immutability?

 

This is subtler, but the big benefits of immutability (to me, at least) are:

  • You don't risk accidentally distributing things you didn't intend to, like partly-built files, your backup directory, or UTF-8 tra files. I used to do this all the time in the early days of SCS.
  • It's one fewer source of bug reports, given that users have been known to copy the mod folder from one game to another.
  • You can have the mod simultaneously present on multiple versions of the game. When I'm developing SCS I have one copy of the mod folder, with virtual directory links to every IE game on my harddrive. So I can change a file and have it instantly reflected in all the virtual "copies" - but I can install part of SCS on (say) BGT without that affecting my BG2EE install. This only works given immutability, because otherwise folders in the mod (like your backup folder) keep track of what's installed and you'll hopelessly confuse things if you try the virtual-folder trick. Before SCS was immutable, I used to have to carefully copy it from game to game, and not infrequently had sync errors. (Come to think of it, in the git era there are other ways around it, but the most straightforward require an internet connection, and I often mod on long flights.)
  • It's easier to use online backup services. That "one copy of the mod folder" lives in my Dropbox folder, so gets backed up, and synced between my desktop and my laptop, in real time. That gets very cumbersome if Dropbox has to copy several thousand files out of the Backup directory every time I install a component; more importantly, I can have different install states on different machines without confusing things.

In the next few posts I'll talk about how to do these things.

 



#2 DavidW

DavidW
  • Gibberlings
  • 4707 posts
  • Gender:Male

Posted 18 September 2018 - 06:03 PM

How to encapsulate

 

The simplest way to encapsulate is to wrap the entire component in a function with no output. Since functions forget everything once they exit, that guarantees nothing gets held onto.

 

So a TP2 component, in this paradigm, might look like

[.. mod preamble]

ALWAYS
   OUTER_SPRINT scsroot mymod
   INCLUDE "%scsroot%/lib/always.tph"
END


/////////////////////////////////////////////////////
///// Awesome component of awesomeness
/////////////////////////////////////////////////////
BEGIN @1000 DESIGNATED 400 GROUP @5
REQUIRE_COMPONENT "setup-stratagems.tp2" 5900 @10
REQUIRE_PREDICATE GAME_IS "bgee bg2ee eet" @20

INCLUDE "%scsroot%/lib/awesome_component.tpa"
LAF awesome_component END

Note that after the usual REQUIRE_xxx declarations in the component preamble, there is basically no code here at all - just an INCLUDE, and a function call. (The INCLUDE has a variable in - scsroot - that is set to your mod name in the ALWAYS block, but you don't need to do that - I just do it that way to help with portability.)

 

In turn, the INCLUDEd file should consist only of function declarations. (If you put raw code in it, it spoils the whole point.) The main function should contain all the code for the component. You can also define subsidiary functions there if you want them only for that component.

 

You also want to keep an eye on your ALWAYS block. In the code here, I've basically outsourced all its content to one INCLUDE. That INCLUDE can't be a function (else its effects will be lost) but you mostly want to program it to run once only, so that it can't affect itself on subsequent calls. A general shape might be:

/// Always block

/// Stuff to do once

ACTION_IF !VARIABLE_IS_SET always_block_called_already BEGIN
    OUTER_SET always_block_called_already=1

    [INCLUDE function and macro definitions, set variables, read in game data, or whatever] 


END


// stuff to do for every component 


[As little as possible! - because everything you put here violates encapsulation]


#3 DavidW

DavidW
  • Gibberlings
  • 4707 posts
  • Gender:Male

Posted 18 September 2018 - 06:13 PM

How to make your mod folder immutable

 

There are three likely ways in which a mod violates immutability.

  1. Most obviously and most commonly, standard WEIDU practice is to put the backup folder inside your mod folder. 
  2. Complicated mods often build files during the install process. (SCS puts together hundreds of scripts, for instance.) Those files tend to need to live somewhere in the process, and that "somewhere" is usually inside the mod itself - either in a "workspace" directory, or just scattered around.
  3. More recently, a new violation of immutability has come about as a consequence of the Enhanced Editions. EE and non-EE versions of the game require their tra files to be coded differently; in the dawn of the EE era modders just shipped EE and non-EE copies of the tra file, but standard practice these days is to use HANDLE_CHARSETS, which (in its normal usage) builds the EE versions at install time inside your mod folder.

 

1 and 2 are easy to solve (SCS has been using this solution for years):

 

1. Put your backup folder somewhere else. SCS backs up to stratagems_external/backup/stratagems. Wheels of Prophecy backs up to stratagems_external/backup/wheels. My (hypothetical) secret project mod backs up to stratagems_external/backup/secret_project. And if you want to adopt this approach, you're welcome to back up to stratagems_external/backup/mymod. (And if someone wants to try this but balks at having their backup folder SCS-branded, I'm quite happy to shift to a new name - weidu_external, say.

 

2. Do all your assembly in an external directory. SCS has

OUTER_SPRINT workspace "stratagems_external/workspace"

in its ALWAYS block, and then just copies any file that's being worked on to %workspace%. (SCS and Wheels of Prophecy share a workspace directory, since the working assumption is that all its contents can be discarded at any time.)

 

3. is a newer problem and trickier to solve; I'll give my solution in the next post.

 



#4 DavidW

DavidW
  • Gibberlings
  • 4707 posts
  • Gender:Male

Posted 18 September 2018 - 06:25 PM

Immutability and HANDLE_CHARSETS

 

The HANDLE_CHARSETS problem is new since I last did any serious modding. HANDLE_CHARSETS is a huge improvement on the old methods of EE modding, but it does seem to require you to have your mod edit itself at install time, in violation of immutability. Fortunately, it turns out that there's a way around that: copy your tra files to an external location before calling HANDLE_CHARSETS. Here's the code I've worked out to do it (and I'd welcome feedback from experts if they see any problem). This lives in the ALWAYS block (in %scsroot%/lib/always.tph if you follow my strategy of putting the meat of ALWAYS somewhere else).

   ACTION_IF GAME_IS bgee bg2ee eet iwdee BEGIN
       OUTER_SPRINT scs_tra_loc "%workspace%/lang"
       ACTION_IF !FILE_EXISTS "override/dw#%scsroot%_languages_installed.mrk"  BEGIN
          MKDIR "%workspace%"
          MKDIR "%workspace%/lang/english"
          MKDIR "%workspace%/lang/%LANGUAGE%"
          COPY_EXISTING "misc01.itm" "override/dw#%scsroot%_languages_installed.mrk"
          ACTION_BASH_FOR "%scsroot%/lang/%LANGUAGE%" ".*\.tra" BEGIN
             COPY "%BASH_FOR_FILESPEC%" "%scs_tra_loc%/%LANGUAGE%"
          END
          ACTION_MATCH "%LANGUAGE%" WITH
          english BEGIN END
          DEFAULT
            ACTION_BASH_FOR "%scsroot%/lang/english" ".*\.tra" BEGIN
                COPY "%BASH_FOR_FILESPEC%" "%scs_tra_loc%/english"
            END
          END

          LAF HANDLE_CHARSETS INT_VAR infer_charsets = 1 STR_VAR iconv_path = "%scsroot%/lang/iconv" tra_path = ~%scs_tra_loc%~  END
       END
   END ELSE BEGIN
      OUTER_SPRINT scs_tra_loc "%scsroot%/lang"
   END
   LOAD_TRA "%scs_tra_loc%/english/setup.tra" "%scs_tra_loc%/english/shared.tra" "%scs_tra_loc%/%LANGUAGE%/setup.tra" "%scs_tra_loc%/%LANGUAGE%/shared.tra"

At the end of that process:

  • On an EE install, the TRA files in your chosen language (and in English, if that's not your chosen language) have been copied over to a space in %workspace% and HANDLE_CHARSETS has UTF-8-ified them
  • The variable scs_tra_loc has been set to the location of the correct TRA files - either the lang directory in your mod folder (for non-EE installs) or %workspace%/lang (for EE installs)
  • The tra files setup.tra (where I assume your component names etc live) and shared.tra (where you put any strings that you want to use in multiple places) are loaded. (Obviously you can load other tra files the same way in ALWAYS.)

(The process only runs once, because it puts a marker file in the override to note that it's done so.)

 

Now any component can include the tra files it needs just by using a LOAD_TRA or WITH_TRA that's relative to scs_tra_loc.

 

In the last post I'll show how you can put this together reasonably neatly.


Edited by DavidW, 18 September 2018 - 11:21 PM.


#5 DavidW

DavidW
  • Gibberlings
  • 4707 posts
  • Gender:Male

Posted 18 September 2018 - 06:33 PM

Laziness and the 'run' function

 

To make all this a bit more streamlined, I define a function "run", like this, in the always block or in some function library that block INCLUDEs (actually my version does a few more tasks, but this is the simplified version):

DEFINE_ACTION_FUNCTION run
       STR_VAR file=""
               file_loc="" 
               tra=""
               version=""
BEGIN
           PRINT ~Including and running function %file%~
           INCLUDE ~%scsroot%/%file_loc%/%file%.tpa~ 
           WITH_TRA "%scs_tra_loc%/english/%tra%.tra" "%scs_tra_loc%/%LANGUAGE%/%tra%.tra" BEGIN
              LAF ~%file%~ STR_VAR version END
           END
       END
END

The TP2 component now streamlines to something like

/////////////////////////////////////////////////////
///// Awesome component of awesomeness
/////////////////////////////////////////////////////
BEGIN @1000 DESIGNATED 400 GROUP @5
REQUIRE_COMPONENT "setup-stratagems.tp2" 5900 @10
REQUIRE_PREDICATE GAME_IS "bgee bg2ee eet" @20

LAF run STR_VAR file=awesome_component file_loc=lib tra=awesome END

That function call:

  • INCLUDEs "mymod/lib/awesome_component.tpa" (I'm still assuming that you've set scsroot to your mod directory in the ALWAYS
  • Loads the tra file "awesome.tra" from the appropriate place. (If there's no such tra file, nothing will be loaded; this is harmless.)
  • Executes the function

You can use the "version" field if you want to use the same function for multiple components - the most natural use case is when a component has multiple subcomponents. You can also use "run" itself inside the component function, to encapsulate chunks of your component to their own function with their own tra file.

 

---

 

Anyway - this is as much documenting things for my own interest as for anything else, and as I say is probably overkill for the majority of mods, but it might be of use or interest to someone. Feedback welcomed.


Edited by DavidW, 18 September 2018 - 06:36 PM.


#6 jastey

jastey
  • Gibberlings
  • 7547 posts
  • Gender:Female

Posted 18 September 2018 - 09:42 PM

Very interesting. I hate to admit I only understand 85% or your code examples but I do grab the concept.

 

Immutability for the utf-8 text transformation is also very important if you tend to work on your texts while it is installed to an EE game. My experience was that any change was lost after deinstallation. This is the reason why I used the principle of copying the texts to another folder for transformation in most of my mods, too.

 

Having the mod folder at one location and working with virtual directory links in all the game folders is one of those brilliant ideas I wonder why I didn't do this already. I guess it's because I have no idea how to make such a virtual link for Windows. I know this is not the actual scope of this tutorial, but if you or someone else would tell me how to do this I would be very obliged.

 

Great fan of the concept(s)! Thank you for sharing.



#7 DavidW

DavidW
  • Gibberlings
  • 4707 posts
  • Gender:Male

Posted 18 September 2018 - 11:19 PM

Having the mod folder at one location and working with virtual directory links in all the game folders is one of those brilliant ideas I wonder why I didn't do this already. I guess it's because I have no idea how to make such a virtual link for Windows. I know this is not the actual scope of this tutorial, but if you or someone else would tell me how to do this I would be very obliged.

 

On Windows, open a command prompt window (on recent versions, go to the Start menu and start typing "command prompt"), navigate to your BG2EE (or whatever) game directory (i.e, type: cd <the directory path>), and type

mklink /j "stratagems" "c:\documents\mymods\stratagems"

replacing "stratagems" with the name of your mod folder, and "c:/documents/mymods/stratagems" with the path to your mod folder.

 

(But it's not a panacea if your mod is non-immutable. At the least, you probably want to relocate your backup folder outside your mod folder.)



#8 ALIENQuake

ALIENQuake
  • Modders
  • 631 posts
  • Gender:Male
  • Location:Poland

Posted 19 September 2018 - 01:23 AM

Yet another fantastic work. Many thanks for providing such valuable guide. I've mention the "users have been known to copy the mod folder from one game to another" problem here: http://forums.pocket....msg338652.html and people ensure me that it's one of the problems which won't fix, even if weidu would add hardcoded prefix to the backup path. Now I'm not so sure.

 

Does enforcing hardcoded part (let's say "<gamefolder>\weidu_backup") would help fix this problem for all mods at once without need to modify/fix tp2 code?

 

 

Regarding directory Junction, Powershell 5.1 way:

New-Item -ItemType Junction -Path G:\Gry\Beamdog\00766\stratagems -Value D:\OneDrive\EET-BWS\Mods\stratagems
You can also use very nice and helpful application: http://schinagl.priv...n.html#download it has "Right-Click Menu", I really encourge anyone who is dealing with Junction/Symlinks to try it!

Edited by ALIENQuake, 19 September 2018 - 01:36 AM.


#9 jastey

jastey
  • Gibberlings
  • 7547 posts
  • Gender:Female

Posted 19 September 2018 - 01:30 AM

(But it's not a panacea if your mod is non-immutable. At the least, you probably want to relocate your backup folder outside your mod folder.)

Thank you very much for the explanation! I am very thrilled because this - with appropriate immutability - is the solution to the "I have x copies of my mod in different game folders and need to keep track which one is the most recent and have to copy it around if testing the installs" problem, in the past leading to Gremlins taking over my mod folders from time to time with vanishing changes or me working with outdated copies.

This is awesome advice!



#10 DavidW

DavidW
  • Gibberlings
  • 4707 posts
  • Gender:Male

Posted 19 September 2018 - 06:57 AM

 

Yet another fantastic work. Many thanks for providing such valuable guide. I've mention the "users have been known to copy the mod folder from one game to another" problem here: http://forums.pocket....msg338652.html and people ensure me that it's one of the problems which won't fix, even if weidu would add hardcoded prefix to the backup path. Now I'm not so sure.

 

Does enforcing hardcoded part (let's say "<gamefolder>\weidu_backup") would help fix this problem for all mods at once without need to modify/fix tp2 code?

 

I think I agree with the skeptical consensus in that thread, unfortunately. It takes a reasonable amount of design care to make SCS immutable - moving the backup folder is not the main problem - and I don't think you could enforce it from outside a mod in an effective way. (And that being the case, Wisp is right to be skeptical about adding a hardcoded backup prefix.)

 

 

(But it's not a panacea if your mod is non-immutable. At the least, you probably want to relocate your backup folder outside your mod folder.)

Thank you very much for the explanation! I am very thrilled because this - with appropriate immutability - is the solution to the "I have x copies of my mod in different game folders and need to keep track which one is the most recent and have to copy it around if testing the installs" problem, in the past leading to Gremlins taking over my mod folders from time to time with vanishing changes or me working with outdated copies.

This is awesome advice!

 

If this is sufficiently useful to someone, I can turn the code fragments here into a template and upload it somewhere.



#11 jastey

jastey
  • Gibberlings
  • 7547 posts
  • Gender:Female

Posted 19 September 2018 - 08:29 AM

If this is sufficiently useful to someone, I can turn the code fragments here into a template and upload it somewhere.

I'd like to take you up on that offer. :)

#12 CamDawg

CamDawg

    ALL GLORY TO THE HYPNOTOAD

  • Gibberling Poobah
  • 9763 posts
  • Gender:Not Telling

Posted 19 September 2018 - 10:14 AM

Thanks for this. It's got me thinking about structuring mods in a new way, which is always a good thing, and I see a lot of advantages here. I'm not sure I'd rate these as priorities over maintainability or readability, but I think there's room to accommodate all. 
 

My (hypothetical) secret project mod


We already know it's a BG version of Blonde Imoen.


Why is this Hypnotoad video so popu... ALL GLORY TO THE HYPNOTOAD.


#13 DavidW

DavidW
  • Gibberlings
  • 4707 posts
  • Gender:Male

Posted 19 September 2018 - 10:35 AM

Thanks for this. It's got me thinking about structuring mods in a new way, which is always a good thing, and I see a lot of advantages here. I'm not sure I'd rate these as priorities over maintainability or readability, but I think there's room to accommodate all.


Absolutely. There are lots of different good ways to structure mods; I wouldn't want to imply that everyone has to do it this way. (And different people have different levels of tolerance for abstraction; I work on the conceptual foundations of modern physics, so mine is probably atypical!)

#14 Wisp

Wisp
  • Modders
  • 1142 posts
  • Gender:Male

Posted 19 September 2018 - 12:22 PM

3. is a newer problem and trickier to solve

I've added an option for controlling where the output ends up:
  • out_path, defaults to %tra_path%
  • If %out_path% is different from %tra_path%, no TRA files in %tra_path% are changed and no marker files or other new files are created
  • If converted TRA files are arranged in a directory structure in %tra_path%, a mirrored structure is created in %out_path%
  • Created files in %out_path% are uninstalled, but any new directory structures that may be created during install remain (because uninstalls do not reverse-create directories)
If you use %out_path%, HANDLE_CHARSETS is obviously no longer transparent, so any references to TRA files you may have in your TP2 need to take this into account, say by LOAD_TRAing ~%tra_folder%/%LANGUAGE%/%etc%~, where %tra_folder% effectively is %tra_path% on non-EE games and %out_path% on EE games. Similarly for USING, etc.

The conventional way of using HANDLE_CHARSETS seems to be to do a mass conversion in the ALWAYS block. In legacy mode, the function uses marker files to make sure conversions happen exactly once, but with %out_path%, there is no non-tenuous analogue (the presence of files in %out_path% does not, to my mind, imply the redundancy of conversion), so suggestion is that the mod author will have to handle it verself, or the ALWAYS HANDLE_CHARSETS will be run for every component (which is probably undesirable).

#15 DavidW

DavidW
  • Gibberlings
  • 4707 posts
  • Gender:Male

Posted 19 September 2018 - 12:56 PM

There is now a template version on GitHub at https://github.com/G...ings3/immutable .

@Wisp: cool - is that in the current WEIDU or queued for the next one?



Reply to this topic



  


0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users