Jump to content

Photo

Mod metadata source - brainstorming


31 replies to this topic

#1 ALIENQuake

ALIENQuake
  • Modders
  • 609 posts
  • Gender:Male
  • Location:Poland

Posted 16 September 2018 - 03:21 AM

Hi,

 

let me start from make clear statement that NOTHING IS REQUIRED. Did I say that nothing is required? Yes, that's true, nothing is required. You don't have to do anything. My proposition/design also aims for be "less selfish as possible" aka "don't require anything and if, require minimal amount". I hope that's clear now.

 

The main idea behind this is to give back the control of mod metadata in the hands of modders. Who else know the current and most up to date mod name, description or the working link to the place where players could receive support? That's right, the modder itself.

 

The first idea was to have a website where modders could provide mod metadata. Creation of such website is very simple and can be done in one day. But then there is a problem when some other people will provide 'malformed' info about other people mods. Trolling is an art. It's impossible to counter without some sophistic login/verification method. And ofc "yet another website" will be instantly boycotted, even if it would be offer a Genie in a bottle.

 

So without website, similar to "mod configuration file", a mod could provide few important metadata like:

 

- Full name of the mod itself, without version number

- Mod description

- a direct download link to the latest version of the mod (which is intend to be used by players so no ALPHA/BETA/DEV)

 

VERSION is dynamically read from tp2. All other missing info I can read from tp2: AUTHOR/README, but if they are inside INI, INI will have preference over tp2.

 

This is my take on this matter:

 

0. File encoding: It can be ANSI if it won't offer to write Unicode strings. UTF8 no BOM, if it does.
1. File format of such file must be 'industry standard', one for all mods. And please forget about xml  :p I can't write a custom parser for each mod only because it's author like to put spaces before and after '=' character. Or use TAB to separate key and values. Or include weidu commands which the tool can't understand. It would be too much work and errors. So let's choose one, modern 'industry standard' file format. I vote would be for INI as a first choice and HJSON/JSON5/JSON (JSON doesn't' allow comments) for more advanced modders who understand benefits of such modern format.
 
2. File extension of 'mod metadata file': Almost irrelevant, follow the file format definition associated extension. Those files shouldn't be modified by players anyway.
 
3. Filename of such 'mod metadata file'. There is one, very simple solution: <modtp2name>.ini/HJSON/JSON5/JSON. This is what is currently implemented.
- it doesn't require file header / ini sections adde [Mod] to avoid false detection
- it prevents people from making multiple, outdated version of this file
 
Other solutions require file headers/scanning mod directory and parsing files etc.
 
4. File content: nothing is required, you can even put one, single line Key=value pair with only mod full name. One single line.

 

INI:

Name=NPCs Enhanced for Everyone
Description=This is intended to be a successor to the fantastic "Level 1 NPCs" mod. L1NPCs was never updated for the EE games, and its author is no longer active, and it is an incredible, complicated piece of work that is difficult for someone else to update. So I made this.
Download=http://lynxlynx.info/ie/modhub.php?UnearthedArcana/NPC_EE

Remarks:

For some modders, tool could use industry standard JSON file or JSON5/HJSON, it has very strict rules, only for modders who wan't to benefit from the advantages of the JSON file format itself. Order of the preference: JSON > JSON5/HJSON > INI . It means that if you provide only INI, everything works. Later, if someone wan't to switch to more advanced metadata format, mod metadata will be read from them. I think that INI would be enough but I don't want to limit modders.

 
5. Presentation to user
- mod metadata will be displayed inside "Mod info textbox"
- in addition, mod name will be displayed at treeview/mod list
- mod downoad link would be used to download new mod version (if it is supported)
 
 
Example of partial INI and tp2 combination:
 
tp2:
AUTHOR ~SubtleD~

ini:

Name=Scales of Balance
Download=http://lynxlynx.info/ie/modhub.php?UnearthedArcana/Scales_of_Balance
Information=http://forum.baldursgate.com/discussion/33657/scales-of-balance-post-hac-kits-and-tweaks
Collected data:
Name=Scales of Balance
Author=SubtleD (extracted from tp2)
Information=http://forum.baldursgate.com/discussion/33657/scales-of-balance-post-hac-kits-and-tweaks
Download=http://lynxlynx.info/ie/modhub.php?UnearthedArcana/Scales_of_Balance
 
 
Example of INI with all metadata provided:
tp2:
AUTHOR ~email@example.com~

ini:

Author=subtledoctor (tp2 AUTHOR is ignored)
Name=Scales of Balance
Information=http://forum.baldursgate.com/discussion/33657/scales-of-balance-post-hac-kits-and-tweaks
Download=http://lynxlynx.info/ie/modhub.php?UnearthedArcana/Scales_of_Balance

Collected data:

Name=Scales of Balance
Author=subtledoctor (tp2 AUTHOR is ignored)
Information=http://forum.baldursgate.com/discussion/33657/scales-of-balance-post-hac-kits-and-tweaks
Download=http://lynxlynx.info/ie/modhub.php?UnearthedArcana/Scales_of_Balance

 

 

That's everything which comes to my mind. I'm sure that everyone will have their own vision and ideas of how it's should be done etc. If I miss some keywords/mod metadata, please let me know.

Edited by ALIENQuake, 04 October 2018 - 05:44 AM.


#2 Sam.

Sam.
  • Modders
  • 134 posts
  • Gender:Male
  • Location:USA

Posted 16 September 2018 - 07:18 AM

Where does the key "Information" point to for a mod that has a readme, a forum, and a website all hosted online, and not necessarily at the same base URL?

Also, if you mandate everyone conform exactly to "an industry standard", you must point to strict documentation of that standard. Preferably, the standard and documentation would be rigorously maintained by a recognized authority. Ideally, the format chosen would be established and widespread enough to be *natively* supported by many programming languages, and IMO be human-readable.
"Ok, I've just about had my FILL of riddle asking, quest assigning, insult throwing, pun hurling, hostage taking, iron mongering, smart-arsed fools, freaks, and felons that continually test my will, mettle, strength, intelligence, and most of all, patience! If you've got a straight answer ANYWHERE in that bent little head of yours, I want to hear it pretty damn quick or I'm going to take a large blunt object roughly the size of Elminster AND his hat, and stuff it lengthwise into a crevice of your being so seldom seen that even the denizens of the nine hells themselves wouldn't touch it with a twenty-foot rusty halberd! Have I MADE myself perfectly CLEAR?!"

-- to Portalbendarwinden

--------------------

Posted Image
___________Old pen and paper modules of the 70s and 80s.___________

CA Forums CA Homepage


#3 ALIENQuake

ALIENQuake
  • Modders
  • 609 posts
  • Gender:Male
  • Location:Poland

Posted 16 September 2018 - 07:58 AM

Where does the key "Information" point to for a mod that has a readme, a forum, and a website all hosted online, and not necessarily at the same base URL?

Also, if you mandate everyone conform exactly to "an industry standard", you must point to strict documentation of that standard. Preferably, the standard and documentation would be rigorously maintained by a recognized authority. Ideally, the format chosen would be established and widespread enough to be *natively* supported by many programming languages, and IMO be human-readable.

 

Ad 1.

It display the the value. So if you enter homepage, it will display homepage. You want three separate keys for online readme, online forum/post and homepage? No problem :) You have to define extra ini keys, I would need to add them to parser and "Mod Info box" it will display all of them.

 

Ad 2. Good point:

INI: https://en.wikipedia.org/wiki/INI_file

JSON: Definition https://www.json.org, Validator https://jsonlint.com/

JSON5: Definition https://json5.org/, it's just the same as JSON with possible comments and less rules, no online validator yet :/

HJSON: Definition: it's just the same as JSON with possible comments and less rules, online validator http://hjson.org/try.html

 

All of those formats are "human-readable".


Edited by ALIENQuake, 16 September 2018 - 08:02 AM.


#4 DavidW

DavidW
  • Gibberlings
  • 4512 posts
  • Gender:Male

Posted 16 September 2018 - 08:01 AM

It would be a very bad idea to write this on the assumption that [mymod].ini is metadata. Doing so would be to reserve a chunk of namespace inside people's mod folders that has no particular connection to the purpose (you can't tell from the fact that it's called [mymod].ini that it's metadata) and that conflicts with already-existing usage in at least some mods (notably SCS).

This is bad practice in an abstract space, but more importantly it runs a serious risk of false positives, i.e. of a file being parsed by your autoinstaller as if it's metadata (or, worse, configuration options) when it was not intended to be parsed that way by the writer. (If you had implemented this proposal without me being aware of it, you might well have ended up offering SCS's and Wheels of Prophecy's debug options to the end user as installation choices.)

If you want to do this (and it sounds sensible) then you need conventions whereby files only get treated as processable metadata if they are explicitly and intentionally marked as such by the mod maintainer (or, I guess, by a BW Fixpack change if we're talking about not-currently-maintained mods). There are lots of ways to do that:

1) My preferred option, as I said on the other thread, is to embed a keyword in the comment field of the file, as in
;;;IE_METADATA_FILE;;;
2) Alternatively, you could mark a file through a particular key-value pair, as in
treat_this_as_ie_metadata=1

3) If you really want to insist on doing this through namespace, pick a bit of namespace that you can be really confident isn't going to be used except for the intended purpose. metadata.ini is the obvious choice; using a .metadata suffix would also work. Theoretically this is less safe than either of the first two proposals; in practice I doubt you're going to get anyone who's already (e.g.) using "metadata.ini" to indicate something that's not metadata.

#5 ALIENQuake

ALIENQuake
  • Modders
  • 609 posts
  • Gender:Male
  • Location:Poland

Posted 16 September 2018 - 08:18 AM

It would be a very bad idea to write this on the assumption that [mymod].ini is metadata. Doing so would be to reserve a chunk of namespace inside people's mod folders that has no particular connection to the purpose (you can't tell from the fact that it's called [mymod].ini that it's metadata) and that conflicts with already-existing usage in at least some mods (notably SCS).

This is bad practice in an abstract space, but more importantly it runs a serious risk of false positives, i.e. of a file being parsed by your autoinstaller as if it's metadata (or, worse, configuration options) when it was not intended to be parsed that way by the writer. (If you had implemented this proposal without me being aware of it, you might well have ended up offering SCS's and Wheels of Prophecy's debug options to the end user as installation choices.)

 

You concerns about 'false-positive detection" are valid. I might have been gone to far in terms of "require nothing". That is why I wanted to receive feedback from modders. You "metadata.ini" is a good suggestion. The name reflects it's purpose. With one minor defect: It would be impossible to store multiple metadata files inside one directory. So how about <*-metadata>.ini ? So it could be "modtp2name-metadata.ini" but it doesn't have to be. The important pat is thee 'metadata.ini" part. I think it's very safe and no modder will create such file by accident.
 
As an alternative to you comment/key-value, how about using a INI "Section", let's call it [Mod] or [Metadata] inside the ini file of any given name, which will be inside main mod folder, where tp2 file is:
[Mod]
Name=NPCs Enhanced for Everyone
Description=This is intended to be a successor to the fantastic "Level 1 NPCs" mod. L1NPCs was never updated for the EE games, and its author is no longer active, and it is an incredible, complicated piece of work that is difficult for someone else to update. So I made this.
Download=http://lynxlynx.info/ie/modhub.php?UnearthedArcana/NPC_EE
Such method can also be applicable to JSON/HJSON/JSON5.
 
I believe that combining those two requirements could be very, very safe and no false-positive detection could occur. Unless there will be a major opposition to use INI Section, and a clear message from other modders that "filename schema will be enough" I would like to go for it.

Edited by ALIENQuake, 16 September 2018 - 08:31 AM.


#6 DavidW

DavidW
  • Gibberlings
  • 4512 posts
  • Gender:Male

Posted 16 September 2018 - 08:28 AM

That looks great. (& yes, I agree, blah-metadata.ini is totally safe.) Many thanks.

#7 Sam.

Sam.
  • Modders
  • 134 posts
  • Gender:Male
  • Location:USA

Posted 16 September 2018 - 11:20 AM

Ad 2. Good point:
INI: https://en.wikipedia.org/wiki/INI_file
JSON: Definition https://www.json.org, Validator https://jsonlint.com/
JSON5: Definition https://json5.org/, it's just the same as JSON with possible comments and less rules, no online validator yet :/
HJSON: Definition: it's just the same as JSON with possible comments and less rules, online validator http://hjson.org/try.html
 
All of those formats are "human-readable".

The wikipedia page for INI doesn't mention it, but I'm pretty sure any language relying on Windows API to read INI files will limit the length of a key's value to 65,535 characters.  This presumably includes AutoIt (and so BWS) and AutoHotkey.  Just a tidbit to keep in mind in the event of rather verbose descriptions...


"Ok, I've just about had my FILL of riddle asking, quest assigning, insult throwing, pun hurling, hostage taking, iron mongering, smart-arsed fools, freaks, and felons that continually test my will, mettle, strength, intelligence, and most of all, patience! If you've got a straight answer ANYWHERE in that bent little head of yours, I want to hear it pretty damn quick or I'm going to take a large blunt object roughly the size of Elminster AND his hat, and stuff it lengthwise into a crevice of your being so seldom seen that even the denizens of the nine hells themselves wouldn't touch it with a twenty-foot rusty halberd! Have I MADE myself perfectly CLEAR?!"

-- to Portalbendarwinden

--------------------

Posted Image
___________Old pen and paper modules of the 70s and 80s.___________

CA Forums CA Homepage


#8 Jarno Mikkola

Jarno Mikkola

    The Imp

  • Modders
  • 6787 posts
  • Gender:Male
  • Location:The town where the dead haven't keeled over, yet. In Finland.

Posted 16 September 2018 - 11:24 AM

Such method can also be applicable to JSON/HJSON/JSON5.

You forgot that the file needs to work in weidu to be read from ... now, of course if you can get Wisp to allow such bastardizations...

Edited by Jarno Mikkola, 16 September 2018 - 11:27 AM.

Welcome to the sanity, you are free to search for the limit, it's out there, we drew it in the sand.
Here's how to install all the ... mods you ever really could want to Infinity Engine games. I removed the stable word from there as Roxanne began to add BS mods that are likely to break compatibility from the BWS.

#9 ALIENQuake

ALIENQuake
  • Modders
  • 609 posts
  • Gender:Male
  • Location:Poland

Posted 16 September 2018 - 11:25 AM

Forget about BWS and it's limitations, I'm not loading the content of the INI using 80's win32 API. But you really would need to put whole book into "Description" value  :p


Edited by ALIENQuake, 16 September 2018 - 11:25 AM.


#10 lynx

lynx
  • Modders
  • 3139 posts
  • Gender:Male
  • Location:Ljubljana, Slovenija

Posted 17 September 2018 - 12:29 AM

In what case would a mod want to include more than one metadata file? Sounds like an unnecessary complication.

 

Since you're already parsing weidu — why not store all of it there, just like author?


GemRB - IE anywhere.
Mages needed! Looking for Planescape: Torment testers
Play android version IS NOT SUPPORTED ANYMORE: reported bugs will be ignored! Still looking for builders ...


#11 ALIENQuake

ALIENQuake
  • Modders
  • 609 posts
  • Gender:Male
  • Location:Poland

Posted 17 September 2018 - 12:57 AM

Two metadata files can coexist to provide different kind of metadata: for a mod information and for mod dependencies ;)

 

I requested new keywords for tp2 long time ago. The wisp conclusion was that the tp2 code is not a good place for those information. Metadata files are much better suit for such data. Code vs Data separation: modification of the metadata it can be added/modified without touching tp2 file. They are industry standard, parser and tools are very solid etc.


Edited by ALIENQuake, 17 September 2018 - 01:03 AM.


#12 Jarno Mikkola

Jarno Mikkola

    The Imp

  • Modders
  • 6787 posts
  • Gender:Male
  • Location:The town where the dead haven't keeled over, yet. In Finland.

Posted 17 September 2018 - 01:05 AM

In what case would a mod want to include more than one metadata file? Sounds like an unnecessary complication.
 
Since you're already parsing weidu — why not store all of it there, just like author?

Good question and my answer, of course, is that you wouldn't... well, perhaps in the case of Level 1 NPCs NPC-components, you could see it, as that's per NPC, but other wise, nope to me.
I intend for the file to contain all the variables I use as the inserts ... so it needs to complete as a weidu's .tpa file, which is just a part of a .tp2 injected in, so it needs to follow the regular .tp2 coding. Dah... which is what I refered in the above post.
We don't care if another program can get something out from it.
Welcome to the sanity, you are free to search for the limit, it's out there, we drew it in the sand.
Here's how to install all the ... mods you ever really could want to Infinity Engine games. I removed the stable word from there as Roxanne began to add BS mods that are likely to break compatibility from the BWS.

#13 lynx

lynx
  • Modders
  • 3139 posts
  • Gender:Male
  • Location:Ljubljana, Slovenija

Posted 17 September 2018 - 08:06 AM

Still sounds like needless proliferation.

With a fixed name it's also always obvious what's inside (and less chances for misspelling/miscasing).

 

What problem does it actually solve? If the metadata is stored with the source, one always has to download it to check if it changed.


GemRB - IE anywhere.
Mages needed! Looking for Planescape: Torment testers
Play android version IS NOT SUPPORTED ANYMORE: reported bugs will be ignored! Still looking for builders ...


#14 ALIENQuake

ALIENQuake
  • Modders
  • 609 posts
  • Gender:Male
  • Location:Poland

Posted 19 September 2018 - 03:56 AM

Still sounds like needless proliferation.

With a fixed name it's also always obvious what's inside (and less chances for misspelling/miscasing).

 

Well, that's valid too, but unless you convince DavidW, I need to keep "<modname>-metadata.ini" scheme.

 

 

 

What problem does it actually solve? If the metadata is stored with the source, one always has to download it to check if it changed.

 

The source of the mod metadata is the mod itself, not some external database.


Edited by ALIENQuake, 19 September 2018 - 04:36 AM.


#15 DavidW

DavidW
  • Gibberlings
  • 4512 posts
  • Gender:Male

Posted 19 September 2018 - 07:01 AM

 

Still sounds like needless proliferation.

With a fixed name it's also always obvious what's inside (and less chances for misspelling/miscasing).

 

Well, that's valid too, but unless you convince DavidW, I need to keep "<modname>-metadata.ini" scheme.

 

What do you need to convince me of? I'm neutral as to whether you need multiple metadata files or just one - going from "metadata.ini" to "xx-metadata.ini" was your idea. I'm also fine if you want to read metadata out of mymod.ini provided there is a completely unambiguous way to designate it as metadata that won't lead to false positives - your suggestion of using .ini section headers is less hacky than my ideas.

 

I don't think I'd seen "store the metadata in the tp2" mentioned before, but I'm strongly against it - code/data separation, please.





Reply to this topic



  


0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users