From a703d8906e570494031b706f38879fff0cbe0287 Mon Sep 17 00:00:00 2001 From: Frotty Date: Tue, 18 Aug 2026 11:38:00 +0200 Subject: [PATCH 1/4] Write down the natively keyed store design before building it Agreed with the repo owner: a Lua table is already a hash map, so a bounded store should lower to t[key] there rather than hashing and probing, which also gets string keys off StringHash - case insensitive, collapsing partial multibyte slices to one constant, changed between patches, and not emulated by the interpreter, so the test covering them says nothing about the game. The part worth writing down is why the type class decides it rather than the container. A Lua table matches by raw identity, so an instance whose equals is structural would have Jass treat two equal-valued keys as one and Lua treat them as two, from one program and without saying so. A second bound states which keys are identity-keyed, and the native path is taken only for those. Also records the groundwork, so the next attempt does not rediscover it: where intrinsics are declared and recognised, that isLuaTarget() is available during lowering and an isLua guard is not enough because folding runs later, that the Lua backend already emits t[i] for an array access so only the index type blocks this, and that imError covers the Jass lowering. The write is left open on purpose: it has to become a statement, and whether that is an ImStatementExpr or an AST expansion like wurstMapFields is the thing to settle first. This blocks WurstStdlib2#468 deliberately, rather than shipping an API the Lua path would make meaningless on that target. --- BACKLOG.md | 56 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 56 insertions(+) diff --git a/BACKLOG.md b/BACKLOG.md index a97772baa..b56a1c49b 100644 --- a/BACKLOG.md +++ b/BACKLOG.md @@ -17,6 +17,62 @@ The container these bounds were added for now compiles and runs against the stan Jass, and compiles on Lua (#1239). What is left is generality around it rather than the feature itself, and one gap in what the suite can see. +27. **A natively keyed store on Lua, selected by a type class.** Decided with the repo owner: + **the native path is taken only where the key's equality is identity**, and a type class says + which keys those are. Everything else keeps today's probing on both targets. + + Why it is worth doing. `FastHashMap` does its own hashing and linear probing on both targets, but + a Lua table already is a hash map: `t[key] = value` would let Lua hash, and would lift the fixed + `FASTHASHMAP_CAPACITY`/`FASTHASHMAP_MAX_INSTANCES` limits there entirely. It would also get string + keys off `StringHash`, which this library documents as unusable for the purpose - case insensitive + (`String.wurst:81`, so `a` and `A` share a key), collapsing every partial multibyte slice to one + constant (`MultibyteDiagnostics`), undocumented and changed between game versions, and **not + emulated by the interpreter**, so `FastHashMapTests.testStringKeys` passing says nothing about the + game. Other containers - a set, a memo cache, adjacency maps - then build on the one store. + + Why the type class is load bearing rather than incidental. A Lua table matches keys by raw + identity, which for a class is reference identity. An instance whose `equals` is structural - + `Hashable` comparing components - would therefore have Jass treat two equal-valued keys as + one key and Lua treat them as two, silently, from one program. So the native path is sound only + for `int`, `real`, `string`, `boolean` and reference-keyed classes. A second bound states that: + + public interface RawKeyed // no requirements; a promise that equality is identity + + class FastHashMap // probing on both targets, any key + class FastHashMap // t[k] on Lua + + which is what the `and` bound is for, and what makes the choice of representation a property of + the key type rather than of the container. + + Groundwork already established, so the next attempt does not have to find it again: + + - Intrinsics are declared in Wurst with `@compilerintrinsic` in `wurst/_wurst/MagicFunctions.wurst` + and bounds parse there; `wurstNewInstance() returns T` is the shape to copy. Recognition is + by name plus `!AttrFuncDef.hasApplicableUserFunction(call)` in `CompilerIntrinsics`. + - `ImTranslator.isLuaTarget()` is available during Wurst-to-IM lowering, which is what makes this + feasible: the intrinsic can lower differently per target rather than needing dead-branch + elimination to have happened first. An `if isLua` guard alone does not help, because folding + runs after translation and both branches are lowered. + - A Wurst array access already lowers to a plain `t[i]` on Lua + (`lua.translation.ExprTranslation.translateArrayAccessRaw` builds `LuaExprArrayAccess`). Nothing + in the backend needs changing; the only obstacle is that the language requires an `int` index, + and typechecking is target independent, so relaxing it for Lua alone would let a program compile + for one target and fail on the other. + - `ImTranslator.imError(trace, message)` gives a runtime error call, for the Jass lowering of an + intrinsic that has no Jass meaning. `ImStatementExpr` is available for pairing statements with a + value. + + Left to settle. The read is straightforward - `wurstKeyedRead(store, key)` lowering to + `store[key]`. The write is the open question: as an `ExprFunctionCall` it must lower to an + `ImExpr`, while what it wants to be is an `ImSet` to an array access. Either wrap it in an + `ImStatementExpr` with a discarded value, or expand it at AST level after validation the way + `wurstMapFields` assigns back to fields, which is a different mechanism and may be the cleaner + one. Settle that before writing the surface. + + Blocks `WurstStdlib2#468`, deliberately: shipping `FastHashMap` first would commit + `FASTHASHMAP_CAPACITY`, `isFull()` and `Hashable.hash` to the public API when the Lua path makes + all three meaningless on that target. + 22. **The library's own tests do not run on Lua.** They run on the interpreter now — all 460 of them, collected by importing every package in the checkout whose name ends in `Tests`. That half is done; this is the other one. From a24036216166aa0c9bd0607e4e7cbd9bb562a5ea Mon Sep 17 00:00:00 2001 From: Frotty Date: Tue, 18 Aug 2026 12:00:46 +0200 Subject: [PATCH 2/4] Correct the selection mechanism and the claim about the interpreter Two things the review caught, both worth having right in a note whose whole purpose is that the next attempt does not start from scratch. Wurst does not overload a type on its bounds, so the two FastHashMap declarations this sketched cannot coexist - every mention of the name would be ambiguous before any instance is considered. Records that the selection is unsettled, and the two shapes it could take, rather than a spelling which does not compile. The interpreter does emulate StringHash: StringProvider delegates to Wc3StringHash and Wc3StringHashTest checks parity with the Lua shim. The case insensitivity and the multibyte collapse are real and are the reason not to use it, but the claim that a test could not see them was wrong. Also records that null is a value of any class type and lowers to nil, and that t[nil] is a runtime error in Lua while probing accepts it - so identity is not on its own enough to admit a reference-keyed class to the native path. --- BACKLOG.md | 24 ++++++++++++++++-------- 1 file changed, 16 insertions(+), 8 deletions(-) diff --git a/BACKLOG.md b/BACKLOG.md index b56a1c49b..7e549e9be 100644 --- a/BACKLOG.md +++ b/BACKLOG.md @@ -26,9 +26,11 @@ itself, and one gap in what the suite can see. `FASTHASHMAP_CAPACITY`/`FASTHASHMAP_MAX_INSTANCES` limits there entirely. It would also get string keys off `StringHash`, which this library documents as unusable for the purpose - case insensitive (`String.wurst:81`, so `a` and `A` share a key), collapsing every partial multibyte slice to one - constant (`MultibyteDiagnostics`), undocumented and changed between game versions, and **not - emulated by the interpreter**, so `FastHashMapTests.testStringKeys` passing says nothing about the - game. Other containers - a set, a memo cache, adjacency maps - then build on the one store. + constant (`MultibyteDiagnostics`), and undocumented and changed between game versions. It *is* + emulated - `StringProvider` delegates to `Wc3StringHash` and `Wc3StringHashTest` checks parity with + the Lua shim - so a test does see the real behaviour; an earlier draft of this entry said otherwise + and was wrong. What makes it unusable is the behaviour itself, not the fidelity of the emulation. + Other containers - a set, a memo cache, adjacency maps - then build on the one store. Why the type class is load bearing rather than incidental. A Lua table matches keys by raw identity, which for a class is reference identity. An instance whose `equals` is structural - @@ -38,11 +40,17 @@ itself, and one gap in what the suite can see. public interface RawKeyed // no requirements; a promise that equality is identity - class FastHashMap // probing on both targets, any key - class FastHashMap // t[k] on Lua - - which is what the `and` bound is for, and what makes the choice of representation a property of - the key type rather than of the container. + How the two are then selected is unsettled, and the obvious spelling does not work: Wurst does not + overload a type on its bounds, so declaring `FastHashMap` beside + `FastHashMap` makes every mention of the name ambiguous before any + instance is considered. Either the native variant is a separate type - `RawHashMap`, say, with the + bound as its entry condition - or one type carries both strategies and picks per operation, which + costs a branch and gives up the limits being lifted. Decide that before writing any of it. + + A further constraint on admitting reference-keyed classes: `null` is a value of any class type and + lowers to `nil`, and `t[nil]` is a runtime error in Lua while the probing implementation accepts + it wherever `Hashable` does. So identity alone is not sufficient for the native path - null keys + have to be rejected or special-cased. Groundwork already established, so the next attempt does not have to find it again: From d26c0dc1a07bb7896e1706d1afc5cc2f206e46ac Mon Sep 17 00:00:00 2001 From: Frotty Date: Tue, 18 Aug 2026 13:25:34 +0200 Subject: [PATCH 3/4] Record which target the container is for, and the hashing package Both from discussion, and both change what this entry leads to. The owner reads FastHashMap as really a Lua container. That is right about speed - on Jass a native Table hashes outside the script, so probing arrays with a hash computed in Wurst will not beat HashMap - and wrong about usefulness, since HashMap cannot take a vec2, a tuple or anything not castable, and two keys casting to one int collide there. That niche is target independent. Keeping both while making the native table primary rather than an optimisation also settles the selection question, because the two variants stop being peers. The hashing package wanted alongside it runs into the same divergence. fmix32, xxHash and FxHash are xor, shift and wrapping multiply; bwXor32 extracts eight bytes and does four table lookups per xor, so fmix32 costs roughly twenty four divisions and twelve lookups per integer on Jass - worse than the mix it would replace, for avalanche nobody sees at mod 32. On Lua the ops are native and it is the right answer outright. It also needs 32 bit wrapping multiply, which Jass has and Lua does not, so matching values across targets costs masking. One surface split by target on the same lowering, which makes the package a second consumer of this work rather than separate work. Both say to measure bwXor32 first. Its cost is the argument and I read it rather than timed it. --- BACKLOG.md | 26 ++++++++++++++++++++++++++ 1 file changed, 26 insertions(+) diff --git a/BACKLOG.md b/BACKLOG.md index 7e549e9be..462608ac6 100644 --- a/BACKLOG.md +++ b/BACKLOG.md @@ -77,6 +77,32 @@ itself, and one gap in what the suite can see. `wurstMapFields` assigns back to fields, which is a different mechanism and may be the cleaner one. Settle that before writing the surface. + **Two things this now has to answer, from discussion.** + + *Which target is `FastHashMap` for.* The owner's read is that it is really a Lua container. That + is right about speed and not about usefulness. On Jass a native `Table` does the hashing outside + the script, so probing arrays with a hash computed in Wurst will not beat `HashMap` - the name + only earns itself on Lua. But `HashMap` cannot take a `vec2`, a tuple or anything else not + castable, and two keys which cast to one int collide there, which is the niche `FastHashMap` + exists for and which is target independent. So: keep it working on both, take the native table on + Lua, and stop presenting the Jass path as the fast one. If that holds, the native table becomes + the primary implementation rather than an optimisation, and the capacity limits and probing become + the Jass fallback - which also settles the selection question above, since the two variants stop + being peers. + + *A `Hashing` package.* Wanted so instances and future containers stop hand-rolling a mix each. + The modern choices - MurmurHash3's `fmix32`, xxHash, FxHash - are all xor, shift and wrapping + multiply, and that is where the targets diverge sharply. `Bitwise.bwXor32` extracts eight bytes + and does four table lookups per xor, so `fmix32` would cost roughly twenty four divisions and + twelve lookups per integer on Jass, worse than the arithmetic mix it would replace and for + avalanche nobody observes at `mod 32`. On Lua those ops are native and `fmix32` is the right + answer outright. `fmix32` also depends on 32 bit wrapping multiply, which Jass has and Lua does + not, so identical values across targets need masking - two more `and32`, another sixteen + divisions. Conclusion: one surface, `hashInt`/`hashString`/`combine`, split by target on the same + `isLuaTarget()` lowering this entry is about, which makes the package a second consumer of it + rather than separate work. Measure `bwXor32` before committing to any of this; its cost is the + whole argument and it was read from the source rather than timed. + Blocks `WurstStdlib2#468`, deliberately: shipping `FastHashMap` first would commit `FASTHASHMAP_CAPACITY`, `isFull()` and `Hashable.hash` to the public API when the Lua path makes all three meaningless on that target. From de23cc360b2dc1965f60ac5fcb662b60e4d58300 Mon Sep 17 00:00:00 2001 From: Frotty Date: Tue, 18 Aug 2026 13:58:02 +0200 Subject: [PATCH 4/4] Record the two Lua emission costs and what would remove them Both found by reading the emitted script rather than by profiling, and both entries say to measure first: a call count is not what the game's interpreter charges. Every primitive array read becomes an ensureInt/ensureBool/ensureReal call, and it survives the optimiser, so slotFor makes two Lua calls per probe step where it wants two table indexes. The shim is needed - an untouched key is nil where Jass reads 0 or false - but the default belongs on the table rather than at every read site. A metatable whose __index returns the default makes a present key a raw index with no call, and runs only on a miss, which is the rare case; three shared metatables cover the primitive types. A method with an override stops being a direct call and becomes an instance miss plus an __index hop. The tables are flat, so it is one hop rather than a walk, but adding an override anywhere silently converts every call site of that slot. Class hierarchy analysis at the call site fixes it where a slot has one reachable implementation, and getSubMethods already holds what that needs. --- BACKLOG.md | 46 ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 46 insertions(+) diff --git a/BACKLOG.md b/BACKLOG.md index 462608ac6..50e44015b 100644 --- a/BACKLOG.md +++ b/BACKLOG.md @@ -107,6 +107,52 @@ itself, and one gap in what the suite can see. `FASTHASHMAP_CAPACITY`, `isFull()` and `Hashable.hash` to the public API when the Lua path makes all three meaningless on that target. +28. **Two costs the Lua emission pays which look avoidable.** Found by reading the emitted script + for `FastHashMapTests_fastHashMapRuntimeLua`, not by profiling. **Measure both before touching + either**; each proposal below is a structural argument and a call count, which is not the same as + knowing what the game's interpreter charges for it. + + *Every primitive array read is a function call.* `LuaNativeLowering.lowerPrimitiveArrayEnsure` + wraps each non-lvalue primitive array access in `ensureInt`/`ensureBool`/`ensureReal`, and the + wrapper survives the optimiser into the emitted script: + + if not(__wurst_ensureBool(FastHashMap_used[s])) then + return ((s2 >= this5.FastHashMap_base) and __wurst_ensureBool(FastHashMap_used[s2])) + + Writes are raw, lvalues being skipped, so only reads pay. `slotFor` reads `used` and `dead` every + probe step, which is two Lua calls per step in the hottest loop the container has. The shim itself + is needed: an untouched table key is `nil` where Jass reads `0` or `false`, and class and handle + typed arrays already skip it because `nil` is the right default for them. + + The fix is to move the default off the access site and onto the table, which is the ordinary Lua + idiom for exactly this: + + __wurst_default_false = ({__index = function() return false end}) + setmetatable(FastHashMap_used, __wurst_default_false) + + A present key is then a raw table index with no call at all, and the function runs only on a miss + - which is the rare case, a read of a slot never written. One metatable per primitive array global + replaces a wrapper at every read site. Three shared metatables cover int, bool and real. `pairs` + is unaffected by `__index`, so nothing which iterates an array changes. What has to be checked: + where the `setmetatable` calls are emitted (globals init, before any read), that a `0` or `false` + actually stored still reads back rather than falling through, and whether anything relies on the + stacktrace argument `callWithStacktrace` adds to the current wrapper. + + *A method with an override stops being a direct call.* Without one, a bounded generic's method + lowers to a direct call - `Box_render_specialized(Box_new_Box(), 42)`. Add a subclass which + overrides it and the same call becomes `b:Box_size_specialized_integer(1)`, an instance miss + followed by an `__index` hop to the class table. The tables are flat rather than chained - + `SubBox.Box_size_specialized_integer` is assigned on `SubBox` itself, so it is one hop and not a + walk up the hierarchy - so the cost is that hop plus a dynamic lookup against a global function + call, and adding an override anywhere silently converts every call site of that slot. + + The fix is class hierarchy analysis at the call site: if the receiver's static type has exactly one + reachable implementation of the slot, emit the direct call. `ImMethod.getSubMethods()` already + holds what that question needs, and `RemoveGarbage` already establishes reachability, so this is + reading data which exists rather than computing anything new. It also composes with the erasure + model in item 23: fewer virtual slots means fewer of the naming and binding problems items 15 and + 26 came from. + 22. **The library's own tests do not run on Lua.** They run on the interpreter now — all 460 of them, collected by importing every package in the checkout whose name ends in `Tests`. That half is done; this is the other one.