Conversation
Thirty-third table off Lift Mapper. No injector sits in front of the provider - NewStyle called MappedApiProductAttributesProvider directly - so this moves the object itself to DoobieApiProductAttributesProvider and updates the one call site. ApiProductAttributeTrait moves into the provider file, since nothing else declared it. Nothing in the suite exercised this table before this change; ApiProductAttributesProviderTest is written first and confirmed against the Mapper version. createOrUpdateApiProductAttribute keeps its exact lookup shape: by apiProductAttributeId, not by (bankId, apiProductCode) - a bank/product pair can carry more than one attribute with the same name at once, so the unique index (and this lookup) is on the id alone, and a supplied id with no matching row falls back to create. The unique index on apiProductAttributeId is carried over and added to the guard test, for the same reason as every table so far: FlywayBaselineExport does not emit dbIndexes-declared unique indexes even though Schemifier creates them.
… the schema Thirty-fourth table off Lift Mapper - the SCA-style challenge/answer flow for updating a user auth context. Already partially prepared for this the same way MappedBadLoginAttempt was: DoobieUserAuthContextUpdateQueries existed with an atomic conditional UPDATE for checkAnswer's TOCTOU fix (CONCURRENCY_HAZARDS.md hazard H2, exercised by ConcurrentConsentStatusRaceTest), used only for the status-transition path while find/create/delete still went through the Mapper entity. This finishes the table without touching that fix. createUserAuthContextUpdates never set challenge explicitly - the Mapper version relied on mChallenge's field default (SecureRandomUtil.csprng.nextInt(99999999), an up-to-8-digit numeric OTP) firing on an unset field. That default is now generated explicitly at the call site rather than implicitly by a field's defaultValue override, since there is no field to override. ConcurrentConsentStatusRaceTest's H2 scenario used the Mapper entity directly for its own fixture setup and status readback; those two helpers move to the same Doobie queries the production code now uses. Its H1/H3/M5 scenarios exercise MappedConsent, a separate table not touched here, and are left alone. Migration.scala's alterTableMappedUserAuthContextUpdate() derived its migration's log-entry name via nameOf(MappedUserAuthContextUpdate) - a compile-time macro over the now-deleted object. That name is the key already recorded in migration_script_log on every environment that has run this migration, so it becomes the literal string the macro produced rather than a fresh name. MigrationOfMappedUserAuthContextUpdate itself moves from DbFunction.tableExists(MetaMapper) to tableExistsByName, same as the other historical migrations already ported. No unique index - only the primary key, matching Schemifier's real output (the entity's own dbIndexes was `super.dbIndexes`, adding nothing). MigrationOfMappedUserAuthContextUpdate drops a legacy index that predates this and was already gone before this migration.
Replace the Lift Mapper card-attribute entity with a Doobie-backed provider (thirty-fifth table off Lift Mapper). No unique index exists on this table - only plain indexes on mCardId and mCardAttributeId, matching the entity's own dbIndexes declaration and confirmed against a booted instance's information_schema.indexes. createOrUpdateCardAttribute preserves the exact find-by-cardAttributeId then update-or-create shape, including the nullable bankId/cardId fallback behaviour on create. Also fills in five migrated tables (mappedfxrate, migrationscriptlog, transactionrequestreasons, apiproductattribute, mappeduserauthcontextupdate) that were missing from MigratedTablesExistTest's existence-check list from earlier migrations in this series.
Replace the Lift Mapper ATM-attribute entity with a Doobie-backed provider (thirty-sixth table off Lift Mapper). No unique index exists on this table - only a plain composite index on (BankId, AtmId), matching the entity's own dbIndexes and confirmed against a booted instance's information_schema.indexes. The Type column is stored as type_c, since Lift Mapper suffixes reserved SQL words and TYPE collides with H2's reserved keyword. The entity type leaked into public signatures across Connector.scala, NewStyle.scala, LocalMappedConnector.scala, JSONFactory5.1.0.scala and Http4s510.scala; all of those now use the existing obp-commons AtmAttributeTrait instead of the concrete Mapper class. AtmTest's direct AtmAttribute.findAll() row-count assertion moves to a raw SQL count query.
Replace the Lift Mapper bank-attribute entity with a Doobie-backed provider (thirty-seventh table off Lift Mapper). No unique index exists on this table - only a plain index on bankid_, matching the entity's own dbIndexes and confirmed against a booted instance's information_schema.indexes. The BankId_ Mapper field has no dbColumnName override, so the column keeps the trailing underscore (bankid_) rather than being renamed like AtmAttribute's BankId_/AtmId_ were. The Type column is stored as type_c for the same reserved-word reason as AtmAttribute. The entity type leaked into public signatures across Connector.scala, NewStyle.scala and LocalMappedConnector.scala; those now use the existing obp-commons BankAttributeTrait instead of the concrete Mapper class.
Replace the Lift Mapper counterparty-attribute entity with a Doobie-backed provider (thirty-eighth table off Lift Mapper). No unique index exists on this table - only a plain index on counterpartyid, matching the entity's own dbIndexes and confirmed against a booted instance's information_schema.indexes. The Type column is stored as type_c for the same reserved-word reason as AtmAttribute/BankAttribute. Unlike those two, this entity's callers already went through code.api.util.newstyle.CounterpartyAttributeNewStyle, which was already typed against the obp-commons CounterpartyAttributeTrait, so only the provider trait itself and Boot.scala needed updating.
Replace the Lift Mapper regulated-entity-attribute entity with a Doobie-backed provider (thirty-ninth table off Lift Mapper). No unique index exists on this table - only a plain index on regulatedentityid, matching the entity's own dbIndexes and confirmed against a booted instance's information_schema.indexes. The Type column is stored as type_c for the same reserved-word reason as the other *Attribute tables migrated so far. MappedRegulatedEntity.attributes (still Mapper-backed, migrates separately) read this table directly via a cross-table Mapper query; it now calls DoobieRegulatedEntityAttributeProvider's synchronous helper instead.
Replace the Lift Mapper product-attribute entity with a Doobie-backed provider (fortieth table off Lift Mapper). No unique index exists on this table - only plain indexes on mBankId and mProductAttributeId, confirmed against a booted instance's information_schema.indexes. Unlike AtmAttribute/BankAttribute/CounterpartyAttribute/ RegulatedEntityAttribute, the Type column here (mType) does not collide with H2's reserved TYPE keyword, so no reserved-word renaming applies. Four call sites read or wrote this table directly through the Mapper entity and now go through DoobieProductAttributeProvider instead: - LocalMappedConnector.getProducts's attribute-filter query, ported to a Doobie Fragment that reproduces the same OR-across-attributes row match semantics as the original BySql filter (exercised by ProductTest's "getProducts by url parameters" scenario). - deletion.DeleteProductCascade's cascade delete. - MappedProductCollectionItemProvider.getProductCollectionItemsTree's read of a product's attributes. - MigrationOfProductAttribute, a historical one-time backfill of the isActive column, switched to raw SQL via the tableExistsByName/ makeBackUpOfTableByName overloads already used by other historical migrations in this series.
Replace the Lift Mapper customer-attribute entity with a Doobie-backed provider (forty-first table off Lift Mapper). No unique index exists on this table - only plain indexes on mCustomerId and mCustomerAttributeId, confirmed against a booted instance's information_schema.indexes. mBankId is stored under the column mbankidid, a historical typo baked into the entity's own dbColumnName override, preserved as-is. getCustomerIdsByAttributeNameValues previously built a Mapper BySql(...) filter via AttributeQueryTrait's getSqlParametersFilter/ getParameters; it now builds the equivalent Doobie Fragment directly, reproducing the same OR-across-attributes row match semantics. No endpoint test exercised this path, so a provider-level characterization test (CustomerAttributeProviderTest) was added and confirmed green against the pristine Mapper entity before the migration, then again against the Doobie provider. Two other call sites read/wrote this table directly through the Mapper entity and now go through DoobieCustomerAttributeProvider or raw SQL: deletion.DeleteCustomerCascade's cascade delete, and MigrationOfCustomerAttributes's historical column-width migration (switched to the tableExistsByName overload).
Replace the Lift Mapper account-attribute entity with a Doobie-backed provider (forty-second table off Lift Mapper). No unique index exists on this table - only plain indexes on mAccountId and mAccountAttributeId, confirmed against a booted instance's information_schema.indexes. getAccountAttributesByAccountCanBeSeenOnView and getAccountAttributesByAccountsCanBeSeenOnView still read AttributeDefinition (a separate, not-yet-migrated Mapper entity) directly and join in plain Scala, exactly as before - only the AccountAttribute-table reads moved to Doobie, including the ByList(mAccountId, ...) multi-account read via Fragments.in. getAccountIdsByParams previously built a Mapper BySql(...) filter via AttributeQueryTrait; it now builds the equivalent Doobie Fragment directly, reproducing the same OR-across-attributes row match semantics. This path backs getFirehoseAccounts filtering and several other endpoints across v3-v6 with no direct endpoint test coverage, so a provider-level characterization test (AccountAttributeProviderTest) covering CRUD, the filter semantics, and both view-visibility methods was added and confirmed green against the pristine Mapper entity before the migration, then again against the Doobie provider. Two cascade-delete call sites move to the new provider: deletion.DeleteBankCascade's "customer_number" attribute lookup and deletion.DeleteAccountCascade's cascade delete.
Replace the Lift Mapper transaction-attribute entity with a Doobie-backed provider (forty-third table off Lift Mapper). No unique index exists on this table - only plain indexes on mTransactionId and mTransactionAttributeId, confirmed against a booted instance's information_schema.indexes. getTransactionAttributesCanBeSeenOnView and getTransactionsAttributesCanBeSeenOnView still read AttributeDefinition (a separate, not-yet-migrated Mapper entity) directly and join in plain Scala, exactly as before - only the TransactionAttribute-table reads moved to Doobie, including the multi-transaction ByList read via Fragments.in. getTransactionIdsByAttributeNameValues previously built a Mapper BySql(...) filter via AttributeQueryTrait; it now builds the equivalent Doobie Fragment directly, reproducing the same OR-across-attributes row match semantics. No endpoint test exercised this filter path or the multi-transaction view-visibility method, so a provider-level characterization test (TransactionAttributeProviderTest) covering CRUD, the filter semantics, and both view-visibility methods was added and confirmed green against the pristine Mapper entity before the migration, then again against the Doobie provider. deletion.DeleteTransactionCascade's cascade delete and V400ServerSetup's shared "no related data left" test helper both move to the new provider.
Follow-up to the previous commit - the new provider test file was written and run but not staged.
Replace the Lift Mapper transaction-request-attribute entity with a Doobie-backed provider (forty-fourth table off Lift Mapper). No unique index exists on this table - only plain indexes on transactionrequestid and transactionrequestattributeid, confirmed against a booted instance's information_schema.indexes. The Type column is stored as type_c for the same reserved-word reason as the other *Attribute tables; Value is unbounded CHARACTER VARYING (same pattern as V015's connectormethod.methodbody), since Open Corridor promise evidence stores a full preimage JSON that exceeds any fixed varchar bound. Two pre-existing quirks in the Mapper version are preserved verbatim rather than fixed: getTransactionRequestAttributesCanBeSeenOnView filters AttributeDefinition by AttributeCategory.Account instead of .TransactionRequest, and getByAttributeNameValues always queries WHERE ispersonal = true regardless of the isPersonal argument it receives. Existing coverage (TransactionRequestTest's "getProducts by url parameters"-equivalent scenario, TransactionRequestAttributesTest, and Http4s700RoutesTest's Open Corridor promise/settlement scenarios) exercises the filter path and the two direct-query call sites in OpenCorridorSettlement (hasPromiseEvidence, coveredTrIds), so no new characterization test was needed this round. MigrationOfTransactionRequestAttributeValueType, a historical migration, switches to the tableExistsByName overload used by the other historical migrations in this series.
The previous commit rewired the provider to Doobie and removed the Mapper entity from Boot.scala's ToSchemify list, but missed deleting the entity class itself - it lived in its own file, separate from the provider file that got deleted. No remaining references; full suite still green.
Replace the Lift Mapper tax-residence entity with a Doobie-backed provider (forty-fifth table off Lift Mapper). mCustomerId is a MappedLongForeignKey pointing at mappedcustomer.id (the customer's internal BIGINT primary key, not their UUID customerId) - the Doobie provider resolves customerId back to the UUID via a MappedCustomer lookup by id, falling back to the raw long id as a string if the customer row is missing, matching the Mapper entity's own getter exactly. Two indexes carried over: a plain index on mcustomerid (from the foreign-key field) and a UNIQUE INDEX on (mcustomerid, mdomain, mtaxnumber), confirmed against a booted instance's information_schema.indexes - caught a first attempt at the migration script that used ALTER TABLE ... ADD CONSTRAINT ... UNIQUE, which H2 backs with an auto-suffixed index name rather than the literal constraint name; switched to CREATE UNIQUE INDEX to match Schemifier's actual output, per MigratedTablesExistTest. deletion.DeleteCustomerCascade's cascade delete moves to raw SQL against the new table.
Replace the Lift Mapper customer-link entity with a Doobie-backed provider (forty-sixth table off Lift Mapper). Unique index on customerlinkid; plain indexes on customerid and othercustomerid, confirmed against a booted instance's information_schema.indexes. No test exercised this table's provider or the connector methods wired to it (bank-to-bank customer relationships, e.g. spouse/parent at another bank), so a characterization test (CustomerLinkProviderTest) covering full CRUD plus bulkDelete was added and confirmed green against the pristine Mapper entity before the migration, then again against the Doobie provider.
Replace the Lift Mapper counterparty-limit entity with a Doobie-backed provider (forty-seventh table off Lift Mapper). Two unique indexes: one on counterpartylimitid, one on the composite (bankid, accountid, viewid, counterpartyid) - at most one limit per tuple - confirmed against a booted instance's information_schema.indexes. Amount fields are NUMERIC(16,10); count fields default to -1 and amount fields to 0 at the application layer on create, matching the Mapper fields' own defaultValue overrides. toJValue moves onto the new CounterpartyLimitRow case class verbatim, since CounterpartyLimitTrait extends JsonAble. MigrationOfCounterpartyLimitFieldType, a historical migration, switches to the tableExistsByName overload used by the other historical migrations in this series.
Replace the Lift Mapper customer-account-link entity with a Doobie-backed provider (forty-eighth table off Lift Mapper). Two unique indexes: one on customeraccountlinkid, one on the composite (accountid, customerid) - a customer has at most one link per account - confirmed against a booted instance's information_schema.indexes. createAgentAccountLink builds its own AgentAccountLinkTraitCommons from the CustomerAccountLinkTrait result fields (documented in the entity as "customer and agent share the same model"), so the row type only needs to implement CustomerAccountLinkTrait. The endpoint test covers create/read/update/delete, but not getOrCreateCustomerAccountLink, the unfiltered getCustomerAccountLinks, or bulkDeleteCustomerAccountLinks, so a characterization test (CustomerAccountLinkProviderTest) covering those three was added and confirmed green against the pristine Mapper entity before the migration, then again against the Doobie provider. Two cascade-delete call sites move to the new provider: deletion.DeleteBankCascade's account-scoped lookup (filters by accountId only, no bankId) and deletion.DeleteCustomerCascade's cascade delete. A third direct reference in LocalMappedConnector.getBankAccountsForUser also moves over.
Replace the Lift Mapper user-customer-link entity with a Doobie-backed provider (forty-ninth table off Lift Mapper). Two unique indexes: one on musercustomerlinkid, one on the composite (muserid, mcustomerid) - a user has at most one link per customer - confirmed against a booted instance's information_schema.indexes. mdateinserted is a separate column from the createdat/updatedat pair the CreatedUpdated mixin also adds; the trait's dateInserted getter reads the former. getOCreateUserCustomerLink preserves the Mapper version's find-then-insert-with-retry-on-conflict shape exactly, including the scala.util.Try wrapping around the insert: ConcurrentDuplicateCreationTest scenario L races 8 concurrent calls for the same (userId, customerId) and asserts no exception and exactly one row, relying on the unique index to reject the losing insert so it can be caught and retried as a re-fetch. Confirmed green against the pristine Mapper entity first, then again after the migration. Two direct callers by name (not through the DI seam) move to the new object: deletion.DeleteCustomerCascade's cascade delete and MappedCustomerProvider.getCustomersByUserId.
Replace the Lift Mapper CRM-event entity with a Doobie-backed provider (fiftieth table off Lift Mapper). Three indexes: UNIQUE on mcrmeventid, plain on mbankid, plus a plain index on muserid that the entity's own dbIndexes never declared - Lift auto-indexes MappedLongForeignKey columns, confirmed against a booted instance's information_schema.indexes. mUserId stores ResourceUser's internal BIGINT primary key; the row's user accessor resolves it back to a live ResourceUser the same way the Mapper entity's getter did, including throwing if the id doesn't resolve. The sandbox importer (LocalMappedConnectorDataImport) built a not-yet-saved MappedCrmEvent via Mapper's create/.validate/Saveable pattern; it now builds a CrmEventCreateParams (a transient CrmEventTrait implementation whose user accessor throws, matching what the Mapper version would have thrown for the same reason - the importer never sets mUserId) and saves it through DoobieCrmEventProvider.createEvent via a new SaveableCrmEvent, mirroring the SaveableAtm pattern already used for the Atm table. MappedCrmEventProviderTest, which called the Mapper entity directly, was rewritten to go through DoobieCrmEventProvider's create/get methods; confirmed green against the pristine Mapper entity before the migration under its original form. SandboxDataLoadingTest's "should create CRM Events ok" scenario exercises the sandbox-import path end-to-end.
Table 51/140 in the Lift Mapper to Doobie strangler migration. Replaces MappedUserRefreshes with DoobieUserRefreshesProvider, backed by a Flyway migration matching the probed schema (muserid varchar(44), createdat/updatedat timestamps, unique index on muserid). This table drives the refresh_user.interval login flow that periodically forces a fresh credential check. AuthUserTest exercises this table's row-count assertions extensively across its login scenarios; rewrote its direct MappedUserRefreshes calls (findAll().length, bulkDelete_!!()) to the new provider's count()/bulkDelete() helpers, added specifically to support those assertions. All 5 AuthUserTest scenarios pass unchanged against the new backing store, along with the full suite (3638 tests, 0 failures).
Table 52/140 in the Lift Mapper to Doobie strangler migration. MappedThing was never wired into Boot.scala's ToSchemify list, so its table was never created by Schemifier, and Thing.thingProvider had no caller anywhere in the codebase - a leftover example/tutorial entity with zero live callers and zero test coverage. There is no data to migrate and no table to create a Flyway script for, so the correct move is deleting the scaffold rather than building a Doobie provider nothing would ever invoke. Removes the code.examplething package entirely and its exemption entry in MappedClassNameTest. Full suite passes unchanged (3638 tests, 0 failures).
Table 53/140 in the Lift Mapper to Doobie strangler migration. Replaces MappedPayeeLookupProvider with DoobiePayeeLookupProvider, backed by a Flyway migration matching the probed schema (unique index on lookupid, plain index on expiresat). This is the short-lived lookup cache behind the mobile-wallet payee-lookup flow in Http4s700 - createPayeeLookup writes a row with a ttl, later reads filter out expired rows in application code without deleting them, preserved exactly as before. Covered by the existing Http4s700RoutesTest suite; no new characterization test needed. Full suite passes unchanged (3638 tests, 0 failures).
Table 54/140 in the Lift Mapper to Doobie strangler migration. MetricsArchiveRun's concrete Mapper type leaked into two cross-file signatures - RunCompleted(run: MetricsArchiveRun) in MetricsArchiveScheduler and metricsArchiveRunToJson(r: MetricsArchiveRun) in JSONFactory7.0.0 - so this introduces MetricsArchiveRunTrait with plain getters and widens both call sites to it, backed by MetricsArchiveRunRow. The plain-object API (recordRun, lastRun, lastSuccessfulRun, pruneToMostRecent, maxRowsToKeep, count, bulkDelete_!!) keeps its original names so only field-level access (PascalCase MappedField.get -> lowercase trait getter) needed to change at call sites, in both production code and MetricsArchiveSchedulerTest. Flyway migration matches the probed schema: unique index on runid, plain index on startedat. Full suite passes unchanged (3638 tests, 0 failures).
Table 55/140 in the Lift Mapper to Doobie strangler migration. Introduces OpenCorridorFeeAccrualTrait so the plain-object API (accrue, unswept, find, markSwept) can keep returning domain values instead of the Mapper entity. The one behavioural translation is the sweep's per-row mutation - Lift called `accrual.FeeSettlementId(id).saveMe()` on each row in place; the Doobie replacement is `OpenCorridorFeeAccrual.markSwept(trId, id)`, an explicit UPDATE keyed on the unique transaction_request_id, called once per accrual from OpenCorridorFees.sweep exactly as before. Flyway migration matches the probed schema: unique index on transaction_request_id (idempotent accrual per covered promise), plain indexes on debtor_bank_id and fee_settlement_id. Covered by the existing Http4s700RoutesTest fee-sweep scenario; no new characterization test needed. Full suite passes unchanged (3638 tests, 0 failures).
Table 56/140 in the Lift Mapper to Doobie strangler migration. Replaces MappedUtilityPaymentCallbackProvider with DoobieUtilityPaymentCallbackProvider behind the existing UtilityPaymentCallbackProvider trait, so UtilityCallbackDispatcher and the injector wiring in UtilityPaymentCallbacks are unaffected. This is the one-shot callback registry for UTILITY transaction-requests that supply a callback_url - distinct from the standing account-event webhook system. Flyway migration matches the probed schema: unique index on callbackid, plain index on transactionrequestid. Covered by the existing Http4s700RoutesTest scenarios for createUtilityVendResult and createTransactionRequestUtility. Full suite passes (3638 tests, 0 failures) on a clean, uncontaminated run; an earlier sharded run's single failure in ConcurrentBackoffCounterSelfHealTest (an unrelated in-memory timing test with no dependency on this table) turned out to be a flake introduced by running an isolated retest concurrently against the same target/ directory - confirmed by both an isolated rerun and this clean full-suite rerun passing.
Table 57/140 in the Lift Mapper to Doobie strangler migration. MappedWebUiPropsProvider is referenced directly by name (no DI injector indirection) from roughly fifteen call sites across AuthUser, I18NUtil, Glossary, APIUtil, and several APIMethods/Http4s version files, all going through the provider-interface methods (getAll/getByName/createOrUpdate/delete/getWebUiPropsValue) rather than raw Mapper fields - so the object keeps its name and only its backing implementation moves to Doobie, avoiding a large-radius rename. createOrUpdate preserves the original's find-by-name-then-upsert shape, including the quirk that a newly created row's webUiPropsId is always a freshly generated UUID - the caller's own webUiPropsId field, if any, is ignored on create, matching Lift's MappedUUID default-value behaviour where .create() never had that field set explicitly. Flyway migration matches the probed schema: unique indexes on both webuipropsid and name. Covered by three dedicated WebUiPropsTest suites (v3.1.0/v5.1.0/v6.0.0, 37 scenarios). Full suite passes (3638 tests, 0 failures).
Table 58/140 in the Lift Mapper to Doobie strangler migration. MappedUserScope had zero callers anywhere in the codebase - no endpoint, no test, nothing reaching UserScope.userScope.vend. Its sibling table Scope/MappedScope (plural, backing the actual v4.0.0 scopes endpoints) is unrelated and stays untouched. There is no data to migrate for a table nothing ever writes to or reads from, so the scaffold is deleted rather than given a Doobie provider nothing would invoke. Removes the code.scope.UserScope and code.scope.MappedUserScopeProvider files entirely, their Boot.scala ToSchemify entry, and the MappedClassNameTest exemption. Full suite passes unchanged (3638 tests, 0 failures).
Table 59/140 in the Lift Mapper to Doobie strangler migration. Backs the v6.0.0 management/groups CRUD endpoints (a named bundle of roles that can be granted to a user's entitlement in one shot, scoped to a bank or system-wide). Had zero existing test coverage - no endpoint test ever exercised group creation, retrieval, update, or deletion - so this adds GroupTest.scala (13 scenarios covering create/get/list/update/delete, role-gating for bank-scoped vs system-level groups, and 404s) as a characterization suite, confirmed green against the pristine Mapper implementation before migrating, then confirmed green again against DoobieGroupProvider. Flyway migration matches the probed schema: two plain (non-unique) indexes on groupid and bankid, matching the entity's own Index(...)/Index(...) declaration. Full suite passes (3651 tests, 0 failures).
Table 60/140 in the Lift Mapper to Doobie strangler migration. Replaces MappedOrganisationProvider with DoobieOrganisationProvider behind the existing OrganisationProvider trait, so the injector wiring in Organisations and every call site through Organisations.organisation.vend is unaffected. updateOrganisation preserves the original's partial-update semantics: each Option field overwrites only when present, and lastupdate is stamped on every successful update. Note that website and logoUrl use orElse against the existing value rather than getOrElse, matching Lift's behaviour where a None in the request left the stored column untouched rather than blanking it. Flyway migration matches the probed schema: one unique index on organisationid. Covered by the existing Http4s700RoutesTest organisation CRUD scenarios. Full suite passes (3651 tests, 0 failures).
serializationNamespace exists so two builds whose Kryo encodings differ cannot address each
other's entries. It derived its discriminator from scala.util.Properties.versionNumberString,
which reads the STANDARD LIBRARY - and Scala 3 compiles against the 2.13 one, so it answered
"2.13" here too. This branch therefore produced byte-identical keys to develop, on exactly the
upgrade the namespace was added to protect: measured as the prefix "obpser1-scala2.13" in this
branch's own golden-key test output, on a build whose scala.compiler is 3.3.8.
The failure that follows is the one already documented above the value: an entry written by one
chill/Scala combination decodes under the other into a different collection type, the decode
succeeds, and the call site whose signature says List gets a ClassCastException - a 500 for the
whole TTL, since a read that throws does not evict.
The compiler generation is not in any version string the runtime exposes, so it is a class probe:
scala.runtime.Scala3RunTime ships in scala3-library and does not exist in scala-library 2.13. Both
halves are kept ("3-lib2.13"), because the encoding depends on the compiler that produced the
classes and on the library they were compiled against. A 2.13 build keeps develop's spelling
exactly, so only this side moves and no one else cold-starts a cache.
The test probes with a different Scala-3-only class (scala.runtime.LazyVals$) than the
implementation uses: repeating the production probe would make the test agree with it however
wrong both were.
`CurrentNamespace should include("3")` was meant to say the namespace names the compiler
generation. It says nothing: "obpser1-scala2.13" contains a '3' as well, so the assertion held in
precisely the state the test exists to reject. The real work was being done by the line above it,
and this one only added false confidence. It now looks for "scala3".
Two comments corrected alongside it, both wrong in ways a reader would act on:
Redis.scala - the block explaining what the namespace is for had a second doc comment placed
between it and `serializationNamespace`, so it documented nothing and the value it explains was
left bare. The probe moves above it.
db.changelog-develop-merge.yaml - the precondition's comment said character_maximum_length is NULL
for an unbounded type. That is Postgres and MySQL; H2 reports 1000000000 and SQL Server -1. The
changeset is correct either way because it counts columns still at exactly 255, which is what the
comment now says.
getDistinctParentIds and getParentIdWithAttributes were written for AttributeQueryTrait.getParentIdByParams and NewAttributeQueryTrait.getParentIdByParams. Both traits are dead code with zero mixers anywhere in the tree, removed in the next commit as part of the net.liftweb.mapper cleanup - and neither ever called into these two methods either, so their removal leaves this file's only remaining function, getDistinctProviders, unaffected. Found while auditing the mapper cleanup's blast radius: the doc comments on both methods asserted a caller that had not existed since the traits were deleted, which is worse than no comment at all.
First step of unbundling lift-persistence: the fork's mapper package is 46.5% of its
lines, has no upstream Scala 3 port (Lift itself deleted persistence rather than
porting it), and OBP-API has had zero live Mapper entities since the Doobie migration
completed (ToSchemify.models = Nil). This removes the last obp-api references to it,
without touching the dependency itself - obp-api still pulls in lift-persistence for
common/util/db, same as before.
Deleted outright (all confirmed zero external references, not just zero imports):
AttributeQueryTrait/NewAttributeQueryTrait (self: BaseMetaMapper, no mixers anywhere),
CommonFunctions (validUri/validUrl, zero call sites), MappedAccountNumber/
DefaultStringField/MappedUUID/UUIDString (MappedString subclasses with no entity left
to use them), and MappedClassNameTest - which asserted over classOf[Mapper[_]]
subtypes, a set that has been permanently empty since the last entity was moved to
Doobie. It is the same "assertion that could not fail" shape as the CacheKeyFormatTest
fix earlier on this branch.
Two deletions needed care because nothing importing net.liftweb.mapper pointed at
them - they are reachable only through a class-name string, so removing the jar
without removing these would compile clean and then fail at runtime:
- JsonSerializers.MapperSerializer: ReflectUtils.forType("net.liftweb.mapper.Mapper")
inside an eager val, wired into the json4s Formats chain. Deleting the object
without also dropping it from the `serializers ::` list would leave a reference
to a name that no longer exists.
- ClassScanUtils.getMappers: Class.forName("net.liftweb.mapper.LongKeyedMapper")
inside a try/catch that logs and returns Nil on any Exception - the failure mode
a `net.liftweb.mapper`-string grep cannot see and a deleted jar would hit silently.
Zero callers, confirmed before deletion.
LocalMappedConnectorDataImport.MappedSaveable (zero instantiations) is deleted the
same way, with the historical comments at its three call-alike sites updated to say
"the now-removed MappedSaveable" rather than describing a type that no longer exists.
Same treatment for two comments in DoobieQueries.scala that credited
AttributeQueryTrait/NewAttributeQueryTrait as callers of getDistinctParentIds/
getParentIdWithAttributes - untrue even before this commit, since those two methods
already had zero callers (deleted separately, previous commit).
Remaining touches are narrowing, not removal: 18 migration scripts had a dead `DB`
import alongside the `Schemifier` one they actually use (`Schemifier.infoF` as a
logging callback - handled in the next commit), and three files had a dead
`import code.util.{MappedUUID, UUIDString}` left over from before those types moved to
Doobie-native construction.
Verification: mvn -Pprod -DskipTests clean install clean on first pass (deletions are
self-checking - the compiler is the reachability proof). H2 Surefire audit: 4073/0/0
(4075 - the 2 MappedClassNameTest scenarios, the only test-file change here).
Postgres was flakier to pin down and worth recording. Three concurrent 4-shard runs
and one 6-shard run all failed, but never on a real assertion:
- Shard 2 hit `run_tests_parallel.sh`'s 1200s per-shard timeout in every attempt,
once at 19m51s - a hair under the cap. The JVM's own shutdown hooks fired cleanly
mid-scenario each time, with zero exceptions, zero OOM/jetsam events, zero
Postgres connection errors in any log. This machine had a second worktree's
orphaned scalatest fork alive for >40h during every attempt (a stray
forkMode=once JVM this repo's own comments already document as a known
reparenting hazard) plus this session's own earlier background work, pushing
load average past 7 - not something to kill blindly (not owned by this session),
so shard 2's package set was instead run standalone (own Postgres database, own
ports, 1800s budget, no sibling shards competing for CPU): 1197 succeeded, 0
failed, 0 exceptions.
- Shard 3 failed once, on ResourceDocsTest's v4.0.0 scenarios, with a
scala.xml.XML.loadString error on a literal "<random-string>" placeholder inside
an existing (untouched by this commit) v4.0.0 endpoint description -
`resource_docs.take(3).foreach(doc => stringToNodeSeq(doc.description))` only
XML-validates the first three docs returned, so whether this fires depends on
resource-doc ordering, not on anything this commit changed. The suite passed
standalone (63/63) and again as part of shard 3's full package set run the same
isolated way as shard 2: 1008 succeeded, 0 failed.
Both isolated runs together cover every package the 4-shard split runs; shards 1 and 4
were clean across all three concurrent attempts. That is full Postgres coverage, green,
just not all four shards inside one concurrent invocation this particular machine could
sustain today.
…local ones
Second step of unbundling lift-persistence, following the mapper-surface deletions in
the previous commit. Four symbols were still genuinely called (not just imported) from
obp-api, none of them mapper-specific in behaviour - Schemifier's logging callback and
schema-name lookup both operate purely on net.liftweb.db types, and DB/
DefaultConnectionIdentifier under the mapper package are forwarders to the db/util
originals, not distinct implementations. Decompiled the shipped jar (javap) to copy
each one exactly rather than guess:
Schemifier.infoF(msg: => AnyRef): Unit = logger.info(msg) - unwrapped, verbatim
Schemifier.getDefaultSchemaName(conn: SuperConnection): String =
conn.schemaName.or(conn.driverType.defaultSchemaName).or(DB.globalDefaultSchemaName)
.openOr(conn.getMetaData.getUserName) - unwrapped, verbatim
Both now live on Migration.DbFunction, next to the tableExistsByName/
makeBackUpOfTableByName helpers that already carried the "copied from
net.liftweb.mapper.Schemifier" comment for the same reason. 62 call sites across 41
migration scripts and StoredProcedureUtils.scala move from `Schemifier.infoF _` to
`DbFunction.infoF _` - a mechanical substitution, verified uniform first: every one of
those 41 files used Schemifier for infoF and nothing else, and every one already
imported DbFunction unqualified for other Migration helpers, so the now-dead
`import net.liftweb.mapper.Schemifier` line comes out alongside each substitution.
`net.liftweb.mapper.DB` becomes `net.liftweb.db.DB` in Migration.scala (11 call sites)
and `net.liftweb.mapper.DefaultConnectionIdentifier` becomes
`net.liftweb.util.DefaultConnectionIdentifier` in DBUtil.scala - both confirmed
identical singletons by decompiling: `mapper.DB` is `object DB extends db.DB1`, and
`mapper.DefaultConnectionIdentifier` is a one-line forwarder to `util.DefaultConnectionIdentifier`.
Migration.DbFunction.tableExists(BaseMetaMapper, ...) and makeBackUpOfTable(BaseMetaMapper)
are deleted outright: both were the last two consumers of BaseMetaMapper, both had zero
callers (confirmed by grep before deletion - the only remaining hits are doc comments in
other migration scripts that already say the entity behind them is gone), and both have
had *ByName successors in active use for a while.
Two call sites intentionally untouched: Boot.scala:540 and
MockedRabbitMqAdapter.scala:3322 still call Schemifier.schemify(true, Schemifier.infoF _,
ToSchemify.models: _*) on an empty list - a no-op, but the whole call and its
ToSchemify.models plumbing come out in the next commit along with Boot's remaining
Schemifier-adjacent setup, rather than half-migrating a call this commit does not also
delete.
net.liftweb.mapper now has zero live references from obp-api (grep -rn
"net\.liftweb\.mapper" obp-api/src/main | grep -v '^\s*//' turns up only the two
Boot.scala/MockedRabbitMqAdapter.scala schemify calls and pre-existing commented-out
Lift-era files this refactor does not touch).
Verification: clean compile in one pass. H2 Surefire audit: 4073/0/0, unchanged from
the previous commit (no test files touched here).
Postgres: given the previous commit's documented machine-load flakiness on concurrent
shards, went straight to isolating each of the 4-shard split's package sets against its
own database and ports rather than re-running the concurrent layout first - two pairs
run concurrently (shard 1 with shard 4, then shard 2 with shard 3) for a bounded total
runtime without reintroducing the contention that caused the earlier timeouts. All four
green: shard 1 535/0, shard 2 1197/0, shard 3 1008/0, shard 4 1285/0 - full coverage of
the 4-shard layout, all BUILD SUCCESS, zero FAILED markers anywhere.
Third and final step of removing net.liftweb.mapper from obp-api. The previous two
commits took every reference down to two schemify calls, both already no-ops
(Schemifier.schemify(true, Schemifier.infoF _, ToSchemify.models: _*) on an empty
list), plus a MapperRules setting and a MetaMapper-typed field that fed them. All
four come out here, along with the last wildcard mapper import.
- Boot.scala:173's MapperRules.createForeignKeys_? assignment: the only reader was
Schemifier, and Schemifier's argument was always Nil, so this configured a
foreign-key policy for a schema-creation pass that never created anything. The
mapper_rules.create_foreign_keys prop it read is retired (release_notes.md, both
props templates).
- Boot.scala:539's schemifyAll(), renamed createDefaultChatRoom() with the
Schemifier.schemify line removed - it kept exactly one live side effect
(getOrCreateDefaultRoom()) and the name should say so, not describe schema work
that stopped happening once ToSchemify.models went to Nil.
- MockedRabbitMqAdapter.scala:3322's identical schemify call, and its now-dead
net.liftweb.mapper.Schemifier / bootstrap.liftweb.ToSchemify imports.
- ToSchemify.models itself: not just emptied, deleted. The object stays (it also
starts the optional gRPC server and registers a JVM shutdown hook, unrelated to
schema). Its four remaining "importers" - ServerSetup, LocalMappedConnectorTestSetup,
TestConnectorSetupWithStandardPermissions, SandboxDataLoadingTest - never actually
read the field; each import was dead weight left over from when their reset loops
iterated it. Removing them is confirmed safe by the same evidence that made the
field safe to delete: obp-api has had zero live Mapper entities since the Doobie
migration finished.
- Boot.scala:64's `import net.liftweb.mapper.{DefaultConnectionIdentifier => _, _}` -
the wildcard that supplied MapperRules, Schemifier and MetaMapper to this file.
Nothing else in it needed anything from that package.
LiquibaseSchemaSetupTest asserted `ToSchemify.models shouldBe empty` as half of pinning
"liquibase.enabled defaults to true because nothing else creates a table." That
assertion doesn't compile once the field is gone, and doesn't need to: the invariant it
protected is now enforced by the compiler rather than by a runtime check, since there
is no Schemifier.schemify call left anywhere in obp-api to accidentally un-empty a list
that no longer exists. Rewrote the test and the doc comments in LiquibaseSchemaSetup.scala
and LiquibaseOnExistingSchemaTest.scala that described the old mechanism, so none of them
keep pointing at a symbol that isn't there.
One more comment turned out to be stale independently of this refactor, caught only
because it was about to become more obviously wrong: AtmTableResetIsolationTest.scala's
doc comment said MappedAtm was "still in Boot.ToSchemify.models" and reset "happens for
free" via that list's bulkDelete_!! loop - checked, and all four reset paths it lists
already carry an explicit `DELETE FROM mappedatm` (ServerSetup:150 and the same line
number pattern in the other three). MappedAtm moved to Doobie a while ago; the comment
was never updated to say so. Corrected to describe the current mechanism instead of a
superseded one.
obp-api/pom.xml's comment on the lift-persistence dependency said Scala 3 doesn't exist
"see docs/scala3-lift-mapper-blocker.md" as if obp-api's own code were still blocked by
it. It isn't, any more - grep -rn "net\.liftweb\.mapper" across obp-api and obp-commons
main sources now turns up only comments and the pre-existing entirely-commented-out
Lift-era files this refactor doesn't touch. What is still pinned to _2.13 is the
ARTIFACT: lift-persistence bundles common+db+mapper+proto+util as one jar, and no
Scala 3 build of the bundle exists because mapper can't compile under Scala 3. Reworded
to say that rather than implying obp-api's own mapper usage is the blocker.
Verification: clean compile in one pass. H2 Surefire audit: 4073/0/0, unchanged (the
4 dead-import deletions and the LiquibaseSchemaSetupTest rewrite add or remove no
scenarios). Postgres: same isolated-per-shard-pair strategy as the previous commit,
same numbers - shard 1 535/0, shard 2 1197/0, shard 3 1008/0, shard 4 1285/0, all
BUILD SUCCESS, zero FAILED anywhere.
Also did the one check the test suite cannot: a real production-mode boot
(flushall_build_and_run.sh, backed by an isolated in-memory H2 rather than any
suite's shared setup) reached `Ember-Server service bound to address: 127.0.0.1:8080`
with no ExceptionInInitializerError and no Schemifier line anywhere in the log, then
served two live requests against it - GET /obp/v5.1.0/root (200) and
GET /obp/v5.1.0/resource-docs/v5.1.0/obp (200, 3.5MB, 599 resource_docs entries) - the
second one specifically to drive the json4s Formats chain end to end now that
MapperSerializer is gone from it (removed two commits ago), on a real multi-megabyte
payload rather than a test fixture.
End state: grep -rn "net\.liftweb\.mapper" obp-api/src obp-commons/src, filtered to
non-comment lines, returns nothing. obp-api's dependency on net.liftweb.mapper is zero.
…ook never ran
Found by code review of the previous three commits (the net.liftweb.mapper removal):
deleting ToSchemify.models and the Schemifier.schemify(...) call that read it removed the
only thing in the whole codebase that ever touched the ToSchemify object. Scala objects
initialize their entire body - every val and top-level statement - on first access to any
member, not at class-load time. Before this session's earlier commits, Boot.scala's
schemifyAll() reading ToSchemify.models was that first access; once schemifyAll() was
renamed to createDefaultChatRoom() and stopped touching ToSchemify, and models itself was
deleted, nothing else in the tree ever referenced it again (confirmed by grep - zero live
code hits, only comments).
The object's body is not just schema-adjacent bookkeeping: it starts the optional gRPC
server (grpc.server.enabled) and registers the JVM's one ORDERED shutdown hook - added,
per its own comment, specifically to fix a race between two previously-concurrent hooks
(gRPC could still be serving a request while the DB pool closed underneath it). With the
object never initializing, both silently stop happening: grpc.server.enabled=true starts
no server and logs no error, and - regardless of that flag - the app stops gracefully
closing the Hikari pool and Redis on shutdown at all.
Verified live, not just by reading the bytecode-initialization rule: booted the packaged
jar and sent it SIGTERM. Before this fix, no HikariPool shutdown log line appeared at all.
After renaming the object to ProcessLifecycle and adding an explicit
ProcessLifecycle.start() call in Boot.boot() (with a comment explaining why an explicit
call is required rather than relying on incidental access), the same test produces
"HikariPool-1 - Shutdown initiated..." / "Shutdown completed." from the shutdown-hook
thread.
Also renamed for the same reason the earlier commits already applied to schemifyAll(): the
object's name described work it no longer does (nothing about it is "to schemify" any
more - the schema half left when models did), and that mismatch is very likely part of why
nothing noticed it had gone silently unreachable.
Swept the doc comments the same review flagged as referencing renamed/deleted symbols by
name, in the files most likely to be read while debugging boot order or writing a new
migration:
- Boot.scala: the comment above the executeScripts calls still said "AFTER schemifyAll()
above", read right next to the createDefaultChatRoom() call it was talking about.
- Migration.scala's `database` object doc comment named `schemifyAll()` and
`tableExists(ResourceUser)` - the latter is the exact Mapper-typed overload the
previous commit deleted; a reader copying that comment's example would write code that
no longer compiles.
- Two migration scripts' historical comments (MigrationOfConsentAuthContextDropIndex,
MigrationOfMappedUserAuthContext) named the same deleted overload as "what this used to
call" without saying the overload itself is gone, not just unused.
Verification: clean compile. H2 Surefire audit: 4073/0/0, unchanged (no test files
touched). Postgres: same isolated-per-shard-pair strategy as the prior three commits -
shard 1 535/0, shard 2 1197/0, shard 3 1008/0, shard 4 1285/0, all BUILD SUCCESS, zero
FAILED anywhere.
Second of two cleanups from the code review of the net.liftweb.mapper removal.
DbFunction.maybeWrite took a `logFunc: (=> AnyRef) => Unit` because Schemifier did - it
was a general-purpose library where a caller might reasonably want its own logger. Here it
never was: all 64 call sites across the 41 migration scripts and StoredProcedureUtils
passed the identical `DbFunction.infoF _`, and infoF was never anything but
`logger.info(msg)`. Checked before removing: 64 occurrences of the call, 64 of them that
exact shape, and infoF had no other reader.
A parameter with zero call-site variance is not an abstraction, so `logger.info(ct)` moves
inline and infoF goes. This also finishes what the previous commits started - they already
deleted the Mapper-typed overloads (tableExists, makeBackUpOfTable) once each had a single
call shape, and leaving this one pluggable was inconsistent with that.
Also corrects an overstatement the same review flagged in LiquibaseSchemaSetupTest: the
comment replacing the retired `ToSchemify.models shouldBe empty` assertion claimed the
invariant is now "enforced by the compiler". That holds only for the exact regression it
replaced - repopulating a field that no longer exists. It does not hold for the wider claim
the surrounding doc makes ("nothing else creates a table"): lift-persistence still ships
net.liftweb.mapper, so a new Mapper entity plus a fresh Schemifier.schemify call would
compile and run, and no test in the suite boots Boot.scala to notice one running beside
Liquibase. The comment now says what is actually guaranteed and what is not.
Verification: clean compile. H2 Surefire audit 4073/0/0, unchanged. Postgres, isolated per
shard pair as before: shard 1 535/0, shard 2 1197/0, shard 3 1008/0, shard 4 1285/0, all
BUILD SUCCESS.
recordConnectorTrace called APIUtil.getCorrelationId() to fill the correlationId column, but that function is a stub returning "" since the Lift teardown (it used to read Lift's container session). The matching connectormetric row is written with the correlation id routeToConnector already extracted from the CallContext for exactly this purpose - trace rows just never received it, so with write_connector_trace enabled every row was written with correlationid = '' and could not be looked up by OBPCorrelationId or joined to its metric row. Pass the already-extracted correlationId into recordConnectorTrace instead of re-deriving it.
scopeFor keyed the dedup lock/response cache on (consumer or Authorization
header, operation id) alone. Two gaps:
- No user in the scope. One Consumer (API Explorer, the Portal, a bank's
mobile app) serves many users, so a second user reusing a key the first
user had already used against the same operation was served the first
user's cached response - their own request never ran.
- No concrete path in the scope. The operation id is the ResourceDoc
template ("OBPv5.1.0-deleteAtm"), never substituted with the real
BANK_ID/ATM_ID. Without the path, one key covered every resource under
an operation - deleting atm-1 then atm-2 under the same key deleted only
the first and replayed its 204 for the second, since a DELETE's body
hash is sha256("") for both requests and gave no other discriminator.
Add both to the scope key. Also add guaranteeCase to the lock acquired in
runAndCache: it was released only on the two normal completion paths, but
ResourceDocMiddleware wraps every endpoint in a timeout that CANCELS the
fiber holding the lock, so a slow POST answered 504 left the lock held for
its full 60s TTL and told the client's well-behaved retry "operation
already in flight" when nothing was.
IdempotencyMiddlewareTest's "in flight" scenario hard-codes the scope hash
the middleware computes; updated it to the new four-part formula.
TokenBinding.verifyTokenBinding compared a bound access token's cnf.x5t#S256 claim against whatever PeerTrust resolved as the caller's certificate, without checking how that certificate was resolved. On the out-of-the-box configuration (mtls.enabled unset, no trusted proxies), PeerTrust.trustForwardedHeaderWithoutTls defaults to true, so an unauthenticated PSD2-CERT header is enough to name "the caller" - by design, for endpoints that only need some certificate to attribute a request to. RFC 8705 sender-constraining needs more than that: a certificate is public information (a QWAC is not a secret), so an attacker who replays a stolen bound access token alongside the victim's own public certificate in that header would pass ENFORCE/REQUIRED verification even though nothing proved they hold the matching private key. Add PeerTrust.UnauthenticatedHopDetail as the named marker for that one resolution (already used internally, just not exposed for a caller to check), and have TokenBinding treat it as equivalent to no certificate at all via a new callerCertificateForBinding - reading cc.certificateTrust / certificateTrustDetail rather than re-deriving anything from the raw header, so it can never disagree with what PeerTrust actually decided.
allStaticResourceDocs deduplicated the union with distinctBy(_.operationId), which keeps the FIRST occurrence in iteration order. Every other consumer of this ordering (Http4s600's top-apis/popular-apis, JSONFactory6.0.0's metrics, and this registry's own sortKey docstring) is built on the opposite convention: the standard sorted LATER wins a name it shares with one sorted earlier, via a `.toMap` where the last entry wins. This silently broke the Berlin Group v1.3 alias safeguard sortKey already implements. The alias re-stamps the canonical BG v1.3 docs with implementedInApiVersion.copy(apiStandard = doc.implementedInApiVersion.apiStandard), so with the natural configuration (berlin_group_v1_3_alias_path ending in "v1.3") its operation ids are byte-identical to the canonical ones, and sortKey ranks the alias first specifically so the union's last-wins dedup keeps the canonical entry. distinctBy's first-wins direction handed the win to the alias instead, replacing all 55 canonical BG v1.3 docs with copies whose URL prefix is the alias path. Switch to reverse/distinctBy/reverse: same last-wins direction as every other consumer, while preserving the relative order of what survives.
1. The mappedconsent join had no guard against the empty-string sentinel. opt() in MappedConsent.scala stores an empty consent_reference_id as '' rather than NULL, and every non-consent metric row also defaults to consent_reference_id = ''. An unguarded `ON m.consent_reference_id = c.consent_reference_id` therefore joined ALL non-consent rows to a single legacy/blanked consent row whenever one existed, and COALESCE(c.muserid, ...) attributed the estate's entire non-consent traffic to that one unrelated user. Add `AND m.consent_reference_id <> ''` to the join condition in buildAggregateMetricsQuery and buildTopUsersQuery - the same fix the NULLIF(..., '') calls already apply on the read side. 2. buildFilterConditions' user_id filter always bound to the raw metric.userid column, even in the two queries whose SELECT/GROUP BY attributes a consent-borne call to the granting human via COALESCE(c.muserid, m.userid). Filtering by a human's user_id excluded exactly the consent-borne calls the endpoint claims to attribute to them (their metric.userid is the consent's own shadow user), while filtering by the shadow user's id returned rows displayed under a different (the human's) identity. Add a resolvedUserIdExpr parameter, defaulting to the previous behaviour for callers with no consent resolution in play, and pass the COALESCE expression from buildAggregateMetricsQuery and buildTopUsersQuery.
Two independent gaps in updateMyMobilePhoneNumber and the mobile number on POST /users: - The regex character class is a union, not a required sequence, so " " (five spaces), "((.))" and "-.-.-" all matched despite the ResourceDoc promising "5 to 50 digits, spaces, dashes, dots or parentheses". A digit-free string would be stored as the user's mobile number with nothing for the later validation/SMS flow to send to. Require at least five actual digits alongside the shape check. - updateMyMobilePhoneNumber wrote straight to the authenticated principal with no check for a consent user. Under a Consent, cc.user is the consent's own shadow ResourceUser by default; letting that identity overwrite the mobile number - an authentication channel used for validation codes and SMS OTP - would let an agent repoint the granting human's second factor. Refuse it outright rather than silently redirecting to the resolved human, since a silent redirect here would let the agent change a security-relevant field the caller has no reason to believe they don't have permission to change.
getTopUsers and getTopConsumers called createQueriesByHttpParamsFuture directly on the raw request params instead of going through APIMetrics.applyMetricsFromDateDefault the way every other metrics-reading endpoint does. With no from_date, APIUtil.getFromDate substitutes the epoch, which makes MappedMetrics.determineMetricsCacheTTL classify the query as "only stable data" and pick the 24-hour TTL - so the default, no-parameter call an operator dashboard would make froze for a day while traffic kept arriving, and the first miss of that day scanned the whole metric table since 1970. Corrected the two ResourceDoc descriptions to match (they claimed "defaults to one year ago" / "the current date", which was never the actual range).
createAccountJSON's Links.Self does list.head.AccountId unconditionally. getAccount builds that list by filtering the caller's own private accounts down to the requested accountId, which is legitimately empty for an id that does not exist (or belongs to someone else) - the same shape a real TPP integration hits on a typo or a stale id. That empty list reached list.head and threw NoSuchElementException, answering 500 instead of the 404 UK Open Banking's spec calls for. Found by extending the endpoint auth/crash sweep to cover Berlin Group and UK Open Banking (previously OBP-standard only) - FailureSweepTest calls every endpoint with a nonexistent id and asserts none of them 5xx.
…n Banking
EndpointCatalog.all was Http4s700.allResourceDocs - the OBP-standard
aggregation only. AuthSweepTest, SuccessSweepTest and FailureSweepTest all
read their coverage from it, so every Berlin Group and UK Open Banking
endpoint was silently outside the anonymous-401/crash sweep: a doc in
those standards missing AuthenticatedUserIsRequired with empty roles would
let anonymous callers reach account data and nothing would catch it.
Switching to ResourceDocRegistry.allStaticResourceDocs (the same
cross-standard union APIUtil.getAllResourceDocs already exposes) needed
three follow-on fixes, all specific to a catalog that now spans multiple
independent route trees rather than one:
- EndpointCatalog.concretePath hard-coded "/obp/" + apiShortVersion.
Berlin Group and UK Open Banking routes match on Root / urlPrefix /
apiShortVersion with no "/obp" segment at all (see e.g.
Http4sBGv13AIS.bgV13Prefix) - urlPrefix is "obp" for the OBP standard by
construction (ApiVersion.setUrlPrefix patches it to the configured
apiPathZero at boot), so using implementedInApiVersion.urlPrefix
uniformly reproduces the old OBP behaviour while giving BG/UK their own
real prefix instead of a path that 404s before reaching any route.
- AuthSweepTest.messageOf only read the top-level "message" field. Berlin
Group requests get a PSD2-mandated {"tppMessages": [{"text": ...}]}
envelope instead (ErrorResponseConverter.toBgErrorBody) - the endpoint
was correctly answering 401, the sweep just could not see the message
text to compare it against. Fall back to tppMessages[0].text, which
carries the identical string the OBP envelope would have.
- SweepCoverageTest's "deduplicated by (url, verb)" check assumed one
route shape maps to one operation, true for OBP but not for Berlin
Group: several SCA sub-steps (e.g. updatePsuAuthentication /
selectPsuAuthenticationMethod / transactionAuthorisation) legitimately
share one URL and verb, disambiguated by request body rather than path.
Replaced with a check on operationId uniqueness, which is the union's
actual by-construction guarantee and still catches a genuine duplicate
(e.g. two ResourceDoc objects registered under the same operation id).
Two categories of endpoint answer non-2xx to SuccessSweepTest's
fully-entitled-but-consentless caller and are documented in
expectedNon2xx rather than treated as failures: Berlin Group AIS and UK
Open Banking account-read endpoints both require an established,
standard-tagged consent regardless of role, which the sweep's generic
fixture (grants every role, creates no consent) does not provide. The
403 in both cases is the endpoint correctly refusing, not a defect - the
anonymous case is what AuthSweepTest already covers independently.
…ie stores
develop's 73 commits here are four features and one large teardown.
The features: maker/checker for runtime-supplied code (a Dynamic Change Request holds a
proposed create/update/delete of a dynamic resource doc, connector method, message doc or ABAC
rule until a second person approves its payload hash, and the runtime executes only rows whose
body hash equals the approved one); API Product Subscriptions (one Consumer holds one product
for a period, with a status machine and the Scope and RateLimiting rows it created); Dynamic
Glossary Items, which an operator adds at runtime and which may shadow a static item; and the
on-behalf-of attribution framework, which renames accountableUserId to onBehalfOfUserId, pulls
the resolver into Users.onBehalfOfUserIdOf, and states in UserReference.scala, per column,
whether a write by a consent user is recorded against the consent user or the human it acts for.
The teardown: the 12 OBPAPIx_y_z aggregator objects and the VersionedOBPApis trait are gone,
along with the APIMethodsXYZ.scala shims. Version enumeration is now explicit in
Http4sResourceDocAggregation.allVersions, and the Lift ResourceDoc text those files carried as
comments was exported first to scripts/resource_doc_baseline/*.json, which is what the parity
audit now reads.
All four features were written against Lift Mapper entities and created their tables through
Schemifier. This branch has neither: ToSchemify.models is empty, so a Mapper column here would
compile and then not exist. The resolution carries the behaviour across instead of restoring the
entities - DynamicChangeRequest, DynamicGlossaryItem, ApiProductSubscription, its Scope join
table and its Attribute table are case classes over Doobie stores, MakerChecker's 33 Mapper call
sites read through the Doobie providers, and the five tables plus the approvedhash/isactive
columns on the four target types and dynamicentity.authmode come from a new
db.changelog-develop-merge-2.yaml.
Three of develop's entries in the new attribution policy name entities this branch does not
have, and one names a column it does not have:
- 16 of the 65 classes UserReference.scala names are Lift Mapper entities this branch already
moved to Doobie. They are only ever read to build the log line in Users.attributionOf, but
AgentDelegationTest resolves each one, so they are repointed at the class that actually owns
the write here, with the real column names beside them.
- PemUsageLastUser and UserScopeUser are dropped: 6bfcb4b deleted the PemUsage scaffold (its
table was never created in any environment) and ac72b13 deleted MappedUserScope (no
callers). A policy entry for a table that does not exist is not a policy.
- abacrule already carries isactive from the baseline, unlike the three dynamic-code tables, so
its maker/checker changeset adds only approvedhash. Liquibase aborted the boot on the
duplicate column, which reads as a test-infrastructure failure rather than a schema one.
Two defects were introduced while resolving conflicts and are fixed here rather than left for CI:
The attribution policy was not actually applied to dynamic data. MapppedDynamicDataProvider
defines ownerOf and never calls it - save, update, get, getAll and existsData all passed the raw
caller id - and MapppedDynamicEntityProvider.createOrUpdate stored dynamicEntity.userId
unresolved. Taking our side of those conflicts kept the Doobie method bodies and dropped the
wrapping develop had put around them, so a consent user's rows and definitions were owned by the
agent rather than by the human it acts for. That is the whole point of the feature, and it is
silent: nothing fails, the row is simply attributed to the wrong user.
Four resource-doc examples published a dangling $ref to Object. A field of an erased type takes
its published type from its example value, so ProvenanceJsonV700.is_active and three of
RateLimiterLimitJsonV700's windows, left as None by develop's examples, had no type to recover.
DynamicChangeRequestJsonV700.current_payload is a separate case: it is a bare JValue and is
genuinely JNothing for a CREATE, which json4s omits from the body entirely - that now publishes
as {"type":"object"}, which is what the null branch already answered for the same field, while
every other declared type with an empty example still fails. buildSwaggerSchema's own failures
named neither the class nor the field, so translateEntity now adds that; it is how the offending
field was found at all.
Alongside: the four table-count assertions move from 147 to 152 for the five new tables, and
run_tests_parallel.sh's per-shard hard kill becomes SHARD_TIMEOUT_SECONDS, defaulting to 2400s
on Postgres. That kill exists to stop a JVM whose Pekko threads hang after the tests finish, not
to bound how long tests may take, and at 1200s three of the four Postgres shards were killed
mid-run - each still emitting test output at the moment it died. A killed shard reports zero
failures while dropping ~800 tests from the audit line, so the run reads as a pass unless the
per-shard durations are checked.
Suite: 4233 tests, 0 failures on H2 and the same 4233 on Postgres, where the three shards now
finish in 20:43, 21:58 and 22:39 - which is what the old 20-minute kill was cutting off.
… their roles ResourceDocMatcher picked the first doc whose template matched, which is registration order. Two docs can share (verb, version, segment count) and both match a request when one spells a segment as an all-caps wildcard: v4's catch-all /banks/BANK_ID/accounts/ACCOUNT_ID/GRANT_VIEW_ID/transaction-request-types/TRANSACTION_REQUEST_TYPE/transaction-requests also matches .../FREE_FORM/..., and it is registered in the object body while the per-type alias docs are added later by initBatch9AliasResourceDocs(). The catch-all declares no roles, so ResourceDocMiddleware - which reads roles from the matched doc only - ran no role check at all, and the FREE_FORM doc's Some(List(canCreateAnyTransactionRequest)) was never enforced. A user holding only the owner view could create a FREE_FORM transaction request and get 201; Lift returned 403, and Lift had no wildcard doc for this URL at all - all nine were literal per-type entries, so this was introduced by the migration. Two changes, both at the mechanism rather than at the call site: findResourceDoc now picks the match with the most literal segments instead of the first one. sortBy is stable, so docs of equal specificity keep their registration order and nothing else moves. literalAllCapsSegments gains the seven transaction-request types that were missing from it - BULK, HOLD, MOBILE_WALLET, UTILITY, CARDANO, ETH_SEND_TRANSACTION, ETH_SEND_RAW_TRANSACTION. Specificity ordering cannot separate two wildcards, and MOBILE_WALLET was shadowing UTILITY and OPEN_CORRIDOR_PROMISE in v7 (HOLD and CARDANO shadowing the two ETH_* docs in v6) the same way, which attached the wrong operationId to those requests - so api_disabled_endpoints, the JSON schema and auth-type allow-lists, and the metric rows all keyed on the wallet endpoint. Its own doc comment already said to add a value here when a path uses a new all-caps literal. TRANSACTION_REQUEST_TYPE stays a wildcard: it is the only one of the eighteen that is a placeholder rather than a type.
… not the human createTransactionRequestImpl210 is the only writer of on_behalf_of_user_id. develop resolved it with Users.attributionOf(userId, UserReference.TransactionRequest); porting the method to the Doobie store kept the Doobie body and dropped the attribution block, leaving onBehalfOfUserId = callContext.flatMap(cc => cc.onBehalfOfUser.or(cc.consenter).map(_.userId)) CallContext.onBehalfOfUser is consentCreator.or(consenter).or(user) - a plain field fallback. It does not consult the consent chain; the resolver that does lives in the String sibling (ApiSession.onBehalfOfUserId -> Users.onBehalfOfUserIdOf), whose own doc describes three steps. The expression above did step 1 and then jumped to step 3, so for a consent user whose CallContext carries only `user` - the ordinary case, and the one AgentDelegationTest asserts is resolved through ResourceUser.createdByConsentId - the row was written with userId == onBehalfOfUserId == the agent's id. The request succeeds and the attribution WARN is never logged, so nothing surfaces: the payment is simply owned by the wrong user. Restores the attributionOf call, keeping develop's precedence rule that a consentCreator or consenter populated by a middleware still wins over the database chain. (.or(cc.consenter) in the old expression was dead in any case - consenter is already inside onBehalfOfUser's chain.) The other five attributionOf call sites survived the port; this was the one that did not. UserReference.TransactionRequest still declared policy UseOnBehalfOfUserId with no callers left.
…same id deleteAtm issued DELETE FROM mappedatm WHERE matmid = ?, keyed on the ATM id alone. The unique index mappedatm_mbankid_matmid is on the pair, so two banks may legitimately hold the same atm_id, and every other statement in this store - getAtmFromProvider, the createOrUpdateAtm re-read, updateColumn - filters on both columns. deleteAtm was the only one that did not, though atm.bankId.value was in hand. The Lift code this replaced was also bank-unscoped, but it deleted through find(By(mAtmId, ...)), which returns at most one row - so the blast radius was one wrong row rather than all of them. Widening that to a set-based DELETE is what the migration introduced. Note deleteAtmAttributesByAtmId, called immediately after this in Http4s510, is unscoped for the same reason and atmattribute does carry a bankid column - but its signature comes from the Connector trait, so fixing it changes a connector method signature and is left out of this commit deliberately.
…mn, a dead tryo MetricsArchiveRun.pruneToMostRecent built DELETE ... WHERE id < (SELECT MIN(id) FROM (...)) with no alias on the derived table. PostgreSQL required one until version 16, so the statement is a syntax error on any PG 15 or older deployment. Nothing catches it: CI pins postgres:16-alpine and H2 never required an alias. The consequence is worse than a failed prune, because MetricsArchiveScheduler calls recordRun again from inside its own catch - the second throw is unguarded and escapes runOnce, so the tick ends in an exception and the table this class documents as self-capping grows without bound. buildTopConsumersQuery reused buildFilterConditions, whose consumerId fragment is the unqualified `consumerid`. That query selects FROM metric, consumer and both tables carry the column, so ?consumer_id=X on /management/metrics/top-consumers fails with an ambiguous column reference. getTopConsumersByConsumerId avoided the reuse for exactly this reason and says so in a comment. buildFilterConditions already took resolvedUserIdExpr so a caller could qualify userid; consumerIdExpr does the same for this one, and buildTopApisQuery - which selects FROM metric alone - keeps the bare default. getAllAggregateMetricsBox and getTopConsumersFuture wrapped an already-evaluated val in tryo. The query ran on the line above, so tryo had nothing to catch and a failure threw straight past it, bypassing the caller's unboxFullOrFail and its OBP error code. The query now runs inside the tryo, which is how the other four memoized methods in this file were already written.
Every other version reaches the chain through gate(), which resolves to HttpRoutes.empty when APIUtil.versionIsAllowed is false. The three Berlin Group trees were spliced in raw. Neither BG wrapper re-checks the prop, and ResourceDocMiddleware deliberately does not re-check version-level enablement per request - it says so at isEndpointEnabled, because the check is meant to happen once at startup in exactly this place. So Berlin Group had no version gate at any layer. The asymmetry is what makes it a problem rather than an omission: with api_disabled_versions=BGv1.3 set, GET /obp/v7.0.0/api/versions reports BG as inactive while /berlin-group/v1.3/accounts/ACCOUNT_ID/transactions keeps returning consent-scoped account data, and UKv3.1 disabled by the same prop does go dark.
…eware skipped them ResourceDocMatcher indexes on (verb, apiVersion, segment count). elasticSearchWarehouse's template was "/search/warehouse" - two segments - while its route is `prefixPath / "search" / "warehouse" / queryString`, three. A real call therefore looked up a key the doc was not filed under and found nothing, and there is no prefix or longest-prefix fallback. elasticSearchMetrics had the same mismatch. With no doc, ResourceDocMiddleware takes its unmatched branch, which calls resolveCallerWithoutRateLimiting instead of anonymousAccess and never runs validateOnly. Four things silently did not happen for these endpoints: the Consumer rate limiter never counted the call, so this Elasticsearch proxy could be driven unthrottled and any per-endpoint limit row for it was inert; api_disabled_endpoints could not disable them; auth-type and JSON-schema validation never ran; and the metric row carried an empty operationId. Authentication and the canSearchWarehouse / canSearchMetrics checks are inline in the handlers, so this was a control bypass rather than data exposure. The templates gain a third segment. SEARCH_QUERY is outside literalAllCapsSegments so it stays a wildcard. That diverges from the Lift baseline, which is the documented placeholder-rename case: both are recorded in parity_allowlist.json with the digests allowlist_helper.py computed.
resetDatabaseForTestClass states that every table is listed explicitly, since no entity is a Lift Mapper any more and there is no model loop to clear them. The five tables db.changelog-develop-merge-2.yaml creates were not in the list, so that statement was false for them and their rows survived into later test classes - the suite runs forkMode=once against one h2:mem database with DB_CLOSE_DELAY=10, so the leak spans the whole JVM. Nothing failed because of it: the suites that write these tables use UUID-suffixed ids, and the one fixed glossary title is deleted by every scenario that creates it. That is luck rather than isolation, and check_test_isolation.py would not catch the next table added without it - it checks setPropsValues placement, not table coverage.
…em were bare
check_nullable_column_reads.py exists to prove that no store reads a nullable column into a
non-nullable Scala type, because doobie's Get throws NonNullableColumnRead on a SQL NULL and
fails the whole query rather than the one row - one legacy row turns a listing endpoint into a
500. It printed "OK: every nullable column is read through Option" while structurally unable to
see most of the code it was pointed at. Two independent blind spots:
The scan matched only `type XxxRow = (...)` alias declarations. About a third of the stores are
written that way; the rest write the row type inline at the `.query[(...)]` call site or use a
case class, and neither was ever examined. Adding those two shapes - plus comment stripping,
without which this script's own scaladoc example (a `case class TopApi` beside a
`SELECT ... FROM metric`) matched itself and reported three violations against a util class that
runs no queries - takes the count from 0 to 597 across 47 stores.
read_ddl broke out of a createTable block the moment it saw `tableName:`. That works only
because Liquibase's writer emits keys alphabetically and so puts tableName after columns; a
hand-written changelog puts it first, and the scan then registered the table with no columns at
all. Empty is worse than absent here: find_violations skips any column not in the map, so every
column of those tables was silently exempt. Six tables from the develop merge were in that
state, addColumn changesets were never read, and flow-style `- column: {name: x}` entries were
not parsed. The block is now read by indentation, in either key order and either style.
The 597 reads are fixed the way the script's own docstring prescribes - bound as Option and
collapsed to what Mapper's reader returned: String to null, Boolean to false, numerics to the
field's default, Timestamp to a null Date. Three shapes appear across the stores and each keeps
its callers unchanged: a tuple consumed as `r._N`, a `case (...)` destructuring (binders renamed
and given a collapsing val), and a case-class row (fields become `xRaw: Option[T]` with a
defaulted `def x: T` accessor). Where a row class is constructed by name outside its store -
DoobieInvestigationQueries.CustomerRow is built by Http4s600 - the class keeps its shape and the
collapse happens at the query instead.
Spot-checked rather than taken on faith: customerlink declares every column except its primary
key nullable, and the store read eight of them bare at six query sites.
The attribute providers bind the free-text columns that come straight from the request body - name and value - as bare String. Doobie's Put for a non-nullable type calls unsafeSetNonNullable, which throws on a null, and the create branch is not inside tryo, so the exception escapes the Future and the caller gets a 500 with no OBP frame in the trace. A null does reach the bind. The JSON case classes declare value as a plain String, both of this repo's custom String deserializers explicitly exclude JNull, and json4s 4.1.1's default ExtractionNullStrategy is Keep - so extraction yields null rather than raising, and no upstream validation rejects it: the endpoint validates `type` and passes `value` through untouched. Lift's MappedString did not override setFilter, so it stored SQL NULL and returned 201. Binding as Option restores that. The same wrapping lands on the name filter in the attribute WHERE clauses, which is the behaviour Lift's By(field, null) had: it renders `= NULL` and matches nothing, rather than throwing at bind time. Ten providers, 48 binds.
AuthSweepTest keeps a signed-off list of endpoints whose observed authentication differs from what their ResourceDoc declares, and a scenario that fails when an entry stops being needed - an exemption nobody removes is read by the next person as still true. createTransactionRequestFreeForm was on that list for answering 400 rather than the 403 its doc declares. The written reason - that the endpoint does no upfront role check and leaves the decision to the connector - described the symptom rather than the cause: the doc the middleware matched was v4's wildcard catch-all, which declares no roles at all, so no role check ran for any transaction-request type. With the matcher now preferring the most specific template, the per-type doc wins and its canCreateAnyTransactionRequest is enforced. The sweep caught this on its own, which is the whole point of that scenario.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Migrates OBP-API to Scala 3, replacing the two frameworks that blocked it. 287 commits on top of
3df73fe11; 1027 files, +66009/-33154.Opened for review of the whole line. It does not merge cleanly onto
developyet — the branchis 26 commits behind and conflicts in 12 files, all of them where
developextended a Lift Mapperentity this branch had already moved to Doobie. That merge is in progress separately and will be
pushed here; the conflicts are listed at the end so reviewers know what is coming rather than
discovering it from a red merge box.
What changed
Scala 3. Lift Mapper cannot compile under Scala 3 — the compiler crashes on the
object X extends class Xshape every Mapper entity uses (reduced to a 5-line case). Both of itsconsumers had to go first.
Lift Web → http4s, complete.
net.liftweb.httpno longer appears in any.scalasource. Thereis no Lift fallback in the request chain: an unmatched
/obp/*path returns a JSON 404 fromnotFoundCatchAll. API versions are unchanged by this — a framework migration happens in placeinside the existing version file, and a version bump still means a changed API signature.
Lift Mapper → Doobie, complete.
ToSchemify.modelsis nowNil; Schemifier creates nothing.Flyway → Liquibase. Flyway needed one hand-written script set per vendor: 118 for H2, 118 for
Postgres, and nothing for the three other drivers its
vendorFolderwould have booted againstsilently, with no tables. One changelog now describes each change once and Liquibase emits the
dialect. The baseline is generated from a Postgres database the Flyway scripts built, not
hand-written, and is regenerated with
scripts/GenerateChangelog.javaplus a normaliser ratherthan edited.
Defects found and fixed along the way
Migrating the data layer surfaced behaviour that lived in Mapper's field types rather than in the
entities, and which a column-by-column port drops silently:
MappedBooleanreads back asfalsewhateverdefaultValuedeclares, a NULLMappedLongas the declared default — read as a hardcoded-1, six call-limit columns turned"the configured limit" into "no limit", and that value is what the rate limiter enforces from
MappedEmaillowercases and trims on every set and validates on save;authuserlost all of itMappedMetrics, reachable from read-only roles, giving a boolean-blindoracle over the whole database
Security fixes on top: a locked account could still authenticate through OIDC and Keycloak (both
read
v_oidc_usersover JDBC and never callverify-credentials);consents.sca.enabled=falseaccepted any SCA answer in production; the dynamic-code dependency validation inspected an empty
list because its scan was gated on an unrelated diagnostic prop; and
compileScalaCodenow refusesto run when no SecurityManager can be installed (JEP 486) unless the operator says so explicitly.
Testing
3876 scenarios, 0 failures, on both H2 and Postgres. Postgres is not optional here: H2 tolerates
things Postgres does not, and the schema is generated from the changelog at boot on both.
Conflicts with current develop
developaddedcreatedByUserId/updatedByUserId/ a method-body hash to DynamicResourceDoc,DynamicMessageDoc and ConnectorMethod, and one new Mapper entity (
ChatEmailDigestState). Thisbranch had already moved the first three to Doobie, so the resolution ports the new fields into the
Doobie stores and the changelog rather than restoring the Mapper entities;
ChatEmailDigestStateneeds the same treatment, since with
models = Nilits table would otherwise never be created.