Jump to content


Photo

[solved] Updating entries in large 2DA files


9 replies to this topic

#1 argent77

argent77
  • Modders
  • 648 posts
  • Gender:Male

Posted 19 May 2017 - 03:53 PM

I'd like to update several values in a 2DA file with a great number of lines (over 1000). I'm currently using this code. However, it is very slow and takes several seconds to complete.

OUTER_SET strref_1 = RESOLVE_STR_REF(@10000)
OUTER_SET strref_2 = RESOLVE_STR_REF(@10001)
OUTER_SET strref_3 = RESOLVE_STR_REF(@10002)
OUTER_SET strref_4 = RESOLVE_STR_REF(@10003)
ACTION_DEFINE_ASSOCIATIVE_ARRAY strings BEGIN
  ~STRREF_FEEDBACK_SNEAK_ATTACK~          => ~%strref_1%~
  ~STRREF_FEEDBACK_SNEAK_AVOIDED~         => ~%strref_2%~
  ~STRREF_FEEDBACK_CRIPPLING_STRIKE~      => ~%strref_3%~
  ~STRREF_FEEDBACK_SNEAK_UNUSABLE_WEAPON~ => ~%strref_4%~
END
COPY_EXISTING ~enginest.2da~ ~override~
  COUNT_2DA_ROWS 2 numRows
  PHP_EACH strings AS name => value BEGIN
    FOR (row = 0; row < numRows; row += 1) BEGIN
      READ_2DA_ENTRY row 0 2 curName
      PATCH_IF (~%curName%~ STR_EQ ~%name%~) BEGIN
        SET_2DA_ENTRY row 1 2 value
        SET row = numRows
      END
    END
  END
BUT_ONLY

 
I'd like to use the 2DA batch functions READ_2DA_ENTRIES_NOW/READ_2DA_ENTRY_FORMER and SET_2DA_ENTRIES_NOW/SET_2DA_ENTRY_LATER (which are hopefully faster), but I'm not very experienced with them. A first test didn't result into any changes. Any idea how to use these functions properly?


Edited by argent77, 20 May 2017 - 04:25 AM.


#2 kjeron

kjeron
  • Members
  • 51 posts

Posted 19 May 2017 - 05:08 PM

COPY_EXISTING ~enginest.2da~ ~override~
  COUNT_2DA_COLS numCols
  READ_2DA_ENTRIES_NOW ~prefix_read_enginest~ numCols
  FOR (row = 0; row < prefix_read_enginest; row += 1) BEGIN
    READ_2DA_ENTRY_FORMER ~prefix_read_enginest~ row 0 curName
    PATCH_MATCH ~%curName%~ WITH
      ~STRREF_FEEDBACK_SNEAK_ATTACK~  BEGIN
        SET_2DA_ENTRY_LATER ~prefix_set_enginest~ row 1 RESOLVE_STR_REF(@10000)
      END
      ~STRREF_FEEDBACK_SNEAK_AVOIDED~  BEGIN
        SET_2DA_ENTRY_LATER ~prefix_set_enginest~ row 1 RESOLVE_STR_REF(@10001)
      END
      ~STRREF_FEEDBACK_CRIPPLING_STRIKE~  BEGIN
        SET_2DA_ENTRY_LATER ~prefix_set_enginest~ row 1 RESOLVE_STR_REF(@10002)
      END
      ~STRREF_FEEDBACK_SNEAK_UNUSABLE_WEAPON~  BEGIN
        SET_2DA_ENTRY_LATER ~prefix_set_enginest~ row 1 RESOLVE_STR_REF(@10003)
      END
      DEFAULT
    END
  END
  SET_2DA_ENTRIES_NOW ~prefix_set_enginest~ numCols
BUT_ONLY

It's definitely faster, but I don't understand them well enough to explain beyond example.



#3 argent77

argent77
  • Modders
  • 648 posts
  • Gender:Male

Posted 20 May 2017 - 03:21 AM

Thanks. I tried it out, but the code triggered the following error:

ERROR: [enginest.2da] -> [override] Patching Failed (COPY) (Failure("ERROR: READ_2DA_ENTRY_FORMER failed on variable %a7_read_enginest_1333_0%"))
Stopping installation because of error.
Stopping installation because of error.
Stopping installation because of error.

ERROR Installing [Install Hidden Gameplay Options -> All available options], rolling back to previous state
Unable to Unlink [HiddenGameplayOptions/backup/0/OTHER.0]: Unix.Unix_error(1, "unlink", "HiddenGameplayOptions/backup/0/OTHER.0")
[HiddenGameplayOptions/backup/0/UNSETSTR.0] SET_STRING uninstall info not found
Will uninstall  15 files for [HIDDENGAMEPLAYOPTIONS/SETUP-HIDDENGAMEPLAYOPTIONS.TP2] component 0.
Uninstalled     15 files for [HIDDENGAMEPLAYOPTIONS/SETUP-HIDDENGAMEPLAYOPTIONS.TP2] component 0.
Unable to Unlink [HiddenGameplayOptions/backup/0/READLN.0]: Unix.Unix_error(20, "unlink", "HiddenGameplayOptions/backup/0/READLN.0")
Unable to Unlink [HiddenGameplayOptions/backup/0/READLN.0.TEXT]: Unix.Unix_error(20, "unlink", "HiddenGameplayOptions/backup/0/READLN.0.TEXT")
ERROR: Failure("ERROR: READ_2DA_ENTRY_FORMER failed on variable %a7_read_enginest_1333_0%")

Which is strange since the file in question has only 1333 matching lines (line 0 .. 1332). I'll test some more. (Too bad that these commands are so badly documented in the WeiDU docs.)



#4 kjeron

kjeron
  • Members
  • 51 posts

Posted 20 May 2017 - 03:56 AM

Thanks. I tried it out, but the code triggered the following error:
[/code]
Which is strange since the file in question has only 1333 matching lines (line 0 .. 1332). I'll test some more. (Too bad that these commands are so badly documented in the WeiDU docs.)

I tested it before posting. Did you use different string-variable for read and set, that error usually comes up when I forget and use the same string for both.
READ_2DA_ENTRIES_NOW ~prefix_read_enginest~ ...
READ_2DA_ENTRY_FORMER ~prefix_read_enginest~ ...
SET_2DA_ENTRY_LATER ~prefix_set_enginest~ ...
SET_2DA_ENTRIES_NOW ~prefix_set_enginest~ ...

#5 argent77

argent77
  • Modders
  • 648 posts
  • Gender:Male

Posted 20 May 2017 - 04:25 AM

That's it, thank you! I was using the same variable for read and set operations. It works now as expected. Moreover, processing time has been reduced significantly. Using READ_2DA_ENTRY/SET_2DA_ENTRY took about two seconds to finish while the *_FORMER/*_LATER variants take only a few milliseconds.

Btw, I've noticed a minor difference between the two methods. The first method appears to preserve line breaks while the second method converts everything into unix-style line breaks.
 



#6 Wisp

Wisp
  • Modders
  • 1084 posts
  • Gender:Male

Posted 20 May 2017 - 05:15 AM

(Too bad that these commands are so badly documented in the WeiDU docs.)

What parts are unclear? I don't personally like the documentation-by-tutorial approach that is employed in this case, but I guess others do?

Using the same prefix name for READ and SET causes an error like this because reading uses its prefix for one thing and writing uses its prefix for something fundamentally different. The details of the implementations allow you to use SET_2DA_ENTRY_LATER/NOW without needing to first read the contents of the file.

Essentially, SET_2DA_ENTRY_LATER queues up changes to be written to file and SET_2DA_ENTRIES_NOW reads the file, makes the queued changes and returns. By comparison, SET_2DA_ENTRY reads the file, makes a single change and returns every time you call it. READ_2DA_ENTRIES_NOW kind of works in reverse, in that it reads the file in one swoop and stores it in memory. READ_2DA_ENTRY reads the whole file for every 2DA entry it looks up. (Technically, it does not quite work like this, because the actual file is read and written by COPY and the patches just deal in buffers (strings); the overhead you save is mostly "parsing" the 2DA string into discrete values and reconstituting it into a buffer.)
 

Btw, I've noticed a minor difference between the two methods. The first method appears to preserve line breaks while the second method converts everything into unix-style line breaks.

Not quite. They both rewrite the contents of the file, but SET_2DA_ENTRY rewrites it with CRLF newlines and SET_2DA_ENTRIES_NOW uses LF.

Edited by Wisp, 20 May 2017 - 05:20 AM.


#7 argent77

argent77
  • Modders
  • 648 posts
  • Gender:Male

Posted 20 May 2017 - 07:40 AM

 

(Too bad that these commands are so badly documented in the WeiDU docs.)

What parts are unclear? I don't personally like the documentation-by-tutorial approach that is employed in this case, but I guess others do?

 

I would like to see a short description of the required parameters in the syntax table. Currently each command is merely referring to a tutorial.

The tutorials themselves are structured somewhat complicated. They are first using several code examples without proper explanation while the commands themselves are described more formally only further below. Moreover, I think the choice of variable names with the unusual prefix "_#_#_#" gives the false impression that it might be a formatting code or could be otherwise relevant to the SET_2DA_* / READ_2DA_* commands.

The tutorials are also using full name (e.g. READ_2DA_ENTRIES_NOW) and an alternate name (e.g. R_2_E_N) interchangeably. It would be more readable to stick to a single naming convention and mention the alternate version in the formal description part. (Is the alternate name even supported by WeiDU?)

I have also found an error in the SET_2DA tutorial (chapter 10.15). It states SET_2DA_ENTRIES_NOW ~string~ row_count. "row_count" should probably be "col_count".



#8 Wisp

Wisp
  • Modders
  • 1084 posts
  • Gender:Male

Posted 21 May 2017 - 12:59 AM

The documentation has been rewritten, hopefully for the better.



#9 argent77

argent77
  • Modders
  • 648 posts
  • Gender:Male

Posted 21 May 2017 - 02:36 AM

It's definitely better now. The command descriptions are informative, and the tutorials are much cleaner.

Some small nitpicks though. SET_2DA_ENTRY_LATER definition states:

... The change is stored is stored in variables derived from the first parameter, string.


The following line in the same description doesn't contain a link to the tutorial:

... See the SET_2DA_ENTRY_LATER tutorial.


The same applies to both READ_2DA_ENTRIES_NOW and READ_2DA_ENTRY_FORMER descriptions:

... See the READ_2DA_ENTRIES_NOW tutorial.

... See the READ_2DA_ENTRY_FORMER tutorial.


The tutorial "10.16 READ_2DA_ENTRIES_NOW and READ_2DA_ENTRY_FORMER" mentions a misspelled version of SET_2DA_ENTRY_LATER and a minor spelling error:

... In is specifically not safe to use the same string you use for READ_2DA_ENTRIES_NOW as you are using for SET_2DA_ENTRIES_LATER.


There are also several links not working (although this is an issue for some time now).

#10 Wisp

Wisp
  • Modders
  • 1084 posts
  • Gender:Male

Posted 21 May 2017 - 06:24 AM

Yeah, I removed the links since they were not working anyway. I'll fix the errors.





Reply to this topic



  


0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users