Migration and vulnerabilities

.github/workflows/build_pull_request.yml

@@ -235,6 +235,12 @@ jobs:
           echo consents.allowed=true >> obp-api/src/main/resources/props/test.default.props
           echo hikari.maximumPoolSize=20 >> obp-api/src/main/resources/props/test.default.props
           echo write_metrics=false >> obp-api/src/main/resources/props/test.default.props
+          # Log emails instead of opening a real SMTP socket: without this,
+          # LocalMappedConnector.sendCustomerNotification's EMAIL branch calls
+          # CommonsEmailWrapper.sendTextEmail which throws ConnectException because
+          # there's no mail server in CI. That surfaces as 500 in any test that
+          # hits an endpoint triggering the notification (v5 consent flows, etc.).
+          echo mail.test.mode=true >> obp-api/src/main/resources/props/test.default.props
 
       - name: Run tests — shard ${{ matrix.shard }} (${{ matrix.name }})
         run: |

LIFT_HTTP4S_MIGRATION.md

@@ -29,7 +29,7 @@ New API versions are implemented as native http4s routes and do not pass through
 
 ### Priority routing
 
-Routes are tried in order: `corsHandler` (OPTIONS) → `AppsPage` → `StatusPage` → `Http4s500` → `Http4s700` → `Http4sBGv2` → `Http4s300` → `Http4s220` → `Http4s210` → `Http4s200` → `Http4s140` → `Http4s130` → `Http4s121` → `Http4sLiftWebBridge` (Lift fallback). Unhandled `/obp/v7.0.0/*` paths fall through silently to Lift — they do not 404.
+Routes are tried in order: `corsHandler` (OPTIONS) → `AppsPage` → `StatusPage` → `Http4s500` → `Http4s700` → `Http4sBGv2` → `Http4s400` → `Http4s310` → `Http4s300` → `Http4s220` → `Http4s210` → `Http4s200` → `Http4s140` → `Http4s130` → `Http4s121` → `Http4sLiftWebBridge` (Lift fallback). Unhandled `/obp/v7.0.0/*` paths fall through silently to Lift — they do not 404.
 
 ```
 HTTP Request
@@ -119,8 +119,8 @@ Bottom-up — each version depends on the one below it being done.
 | 5 | `APIMethods210` | 28 | **Done** — `Http4s210.scala`: 25 own endpoints + path-rewriting bridge to `Http4s200`; all 79 v2.1.0 tests pass |
 | 6 | `APIMethods220` | 19 | **Done** — `Http4s220.scala`: 18 own endpoints + path-rewriting bridge to `Http4s210`; all 27 v2.2.0 tests pass |
 | 7 | `APIMethods300` | 47 | **Done** — `Http4s300.scala`: 47 own endpoints + path-rewriting bridge to `Http4s220`; all 86 v3.0.0 tests pass |
-| 8 | `APIMethods310` | 102 | |
-| 9 | `APIMethods400` | ~258 total | Largest file; may need splitting into sub-traits |
+| 8 | `APIMethods310` | 102 | **Done** — `Http4s310.scala` has all 100 functional endpoints (42 GET, 10 DELETE, 19 POST, 25 PUT, 1 GET-shaped revoke, 3 SCA aliases) + path-rewriting bridge to `Http4s300`; 181 v3.1.0 tests pass. Two endpoints tracked separately in "Per-version Lift leftovers" (`getMessageDocsSwagger`, `getObpConnectorLoopback`) — they retire via the Resource-docs workstream / bridge-removal PR, not as v3.1.0 follow-up. |
+| 9 | `APIMethods400` | ~258 total | **In progress (47/258 endpoints)** — `Http4s400.scala` scaffolded with `staticResourceDocs`/`resourceDocs` split + bridge to `Http4s310`. **Dynamic-entity family complete** (11/11), **dynamic-endpoint family complete** (12/12), **mainstream batch 1** (`getMapperDatabaseInfo`, `getLogoutLink`, `getBanks`, `getBank`, `ibanChecker`, `callsLimit`, `createBank`, `root`). **Override audit started** (13/35 v4-over-older overrides migrated: `getBanks`, `getBank`, `createBank`, `root`, `getAtms`, `getAtm`, `createAtm`, `getProducts`, `getProduct`, `createProduct`, `createProductAttribute`, `updateProductAttribute`, `callsLimit`). Tests passing: BankTests, BankAttributeTests, MapperDatabaseInfoTest, RateLimitingTest, AtmsTest, ProductTest, DynamicEntityTest, DynamicEndpointsTest et al. **Bridge-cascade hijack gotcha** (see CLAUDE.md): v4 endpoints that *override* a same-URL endpoint from an earlier version must be migrated to `Http4s400` own-routes *before* relying on the bridge — otherwise the bridge cascade rewrites the path down to the older version's handler (which has different behaviour). 22 overrides remain to migrate. |
 | 10 | `APIMethods500` | 37 | |
 | 11 | `APIMethods510` | 111 | |
 | 12 | `APIMethods600` | ~244 total | Final Lift endpoint file |
@@ -136,8 +136,9 @@ Resource-docs endpoints are **version-polymorphic**: `GET /obp/v6.0.0/resource-d
 Add one service to `Http4sApp` (above the Lift bridge, before any per-version service) that handles:
 
 ```
-GET  /obp/*/resource-docs/API_VERSION/obp   → version-dispatch via getResourceDocsList
+GET  /obp/*/resource-docs/API_VERSION/obp           → version-dispatch via getResourceDocsList
 GET  /obp/*/resource-docs/API_VERSION/openapi.yaml
+GET  /obp/*/message-docs/CONNECTOR/swagger2.0       → absorbs APIMethods310.getMessageDocsSwagger
 ```
 
 The wildcard prefix means all resource-doc requests are intercepted regardless of which version prefix the client uses. This workstream is **independent of the per-version migration order** — it can land at any time and immediately removes all resource-docs traffic from the Lift bridge.
@@ -159,7 +160,8 @@ Currently served via a raw Lift `serve { case Req(..., "openapi.yaml", ...) }` b
 1. Fix aggregation bug in `getResourceDocsObpV700` → make `V7ResourceDocsAggregationTest` pass.
 2. Extract shared handler logic into `Http4sResourceDocs` service; wire into `Http4sApp`.
 3. Add `openapi.yaml` route to the same service.
-4. Remove resource-docs from the per-version Lift objects (`ResourceDocs140`–`ResourceDocs600`) once the centralized service covers them.
+4. Port `getMessageDocsSwagger` from `APIMethods310` into the same service (currently still served by the Lift bridge — see "Per-version Lift leftovers" below).
+5. Remove resource-docs from the per-version Lift objects (`ResourceDocs140`–`ResourceDocs600`) once the centralized service covers them.
 
 ---
 
@@ -178,6 +180,19 @@ These are the last hard dependency on Lift Web in the request path. The Lift bri
 
 ---
 
+## Per-version Lift leftovers
+
+An `APIMethods{version}` file is marked **done** in the progress table when every *functional* endpoint is on http4s and the version's test suite is green. A small number of endpoints are deliberately *not* migrated inline because they belong to a different workstream or have no behaviour worth porting. They continue to be served by the Lift bridge until the workstream that owns them lands; they do **not** create new follow-up work on the per-version file.
+
+| Endpoint | Origin | Why on Lift | Retired by |
+|---|---|---|---|
+| `getMessageDocsSwagger` (`GET /message-docs/CONNECTOR/swagger2.0`) | `APIMethods310` | Same shape as `getResourceDocsObpV700` / `openapi.yaml` — runtime Swagger generation with shared caching | The **Http4sResourceDocs** workstream (step 4) |
+| `getObpConnectorLoopback` (`GET /connector/loopback`) | `APIMethods310` | Deprecated stub that unconditionally throws `IllegalStateException(NotImplemented)`; no functional behaviour | Either a 3-line native http4s route that throws the same exception or outright deletion, decided when the Lift bridge is removed |
+
+Track new leftovers here when later version files are migrated — the bridge-removal milestone in "Done Criteria" only requires the per-version files to be **done** in this table's sense (functional endpoints migrated, tests green). Leftovers folded into the Resource-docs or Auth-stack workstreams retire via those workstreams.
+
+---
+
 ## Server Chain After Full Migration
 
 ```
@@ -206,10 +221,10 @@ corsHandler
 
 | Milestone | Condition |
 |---|---|
-| Version file done | All endpoints are `HttpRoutes[IO]`; `OBPRestHelper` removed from the file; existing tests pass |
-| Lift bridge removable | All 12 APIMethods files done + auth stack done |
-| Lift Web removed | `lift-webkit` removed from `pom.xml`; `Boot.scala` reduced to DB init + scheduler startup |
-| `lift-mapper` | Separate long-term effort — not in scope here |
+| Version file done | All *functional* endpoints are `HttpRoutes[IO]`; the version's test suite is green. Endpoints folded into the Resource-docs / Auth-stack workstreams or marked as non-functional stubs are listed in "Per-version Lift leftovers" rather than blocking the file's done status. |
+| Lift bridge removable | All 12 APIMethods files done (per the row above) + auth stack done + Resource-docs workstream done. Any remaining stubs from "Per-version Lift leftovers" are ported or deleted in the bridge-removal PR. |
+| Lift Web removed | `lift-webkit` removed from `pom.xml`; `Boot.scala` reduced to DB init + scheduler startup. |
+| `lift-mapper` | Separate long-term effort — not in scope here. |
 
 ---
 
@@ -245,7 +260,7 @@ Binds to `hostname` / `dev.port` from your props file (defaults: `127.0.0.1:8080
 | `APIMethods210` | done — `Http4s210.scala` (25 own endpoints; path-rewriting bridge to Http4s200) |
 | `APIMethods220` | done — `Http4s220.scala` (18 own endpoints; path-rewriting bridge to Http4s210) |
 | `APIMethods300` | done — `Http4s300.scala` (47 own endpoints; path-rewriting bridge to Http4s220; all 86 v3.0.0 tests pass) |
-| `APIMethods310` | todo |
+| `APIMethods310` | done — `Http4s310.scala` (100 own endpoints; path-rewriting bridge to Http4s300; 2 endpoints intentionally left on Lift: `getMessageDocsSwagger`, `getObpConnectorLoopback`) |
 | `APIMethods400` | todo |
 | `APIMethods500` | todo |
 | `APIMethods510` | todo |

obp-api/pom.xml

@@ -52,7 +52,7 @@
     <dependency>
       <groupId>org.bouncycastle</groupId>
       <artifactId>bcpg-jdk18on</artifactId>
-      <version>1.78.1</version>
+      <version>1.84</version>
     </dependency>
     <dependency>
       <groupId>org.http4s</groupId>
@@ -67,7 +67,7 @@
     <dependency>
       <groupId>org.bouncycastle</groupId>
       <artifactId>bcpkix-jdk18on</artifactId>
-      <version>1.78.1</version>
+      <version>1.84</version>
     </dependency>
     <!-- https://mvnrepository.com/artifact/org.apache.commons/commons-lang3 -->
     <dependency>
@@ -82,7 +82,7 @@
     <dependency>
       <groupId>org.postgresql</groupId>
       <artifactId>postgresql</artifactId>
-      <version>42.7.7</version>
+      <version>42.7.11</version>
     </dependency>
     <!-- https://mvnrepository.com/artifact/com.oracle.database.jdbc/ojdbc8-production -->
     <dependency>
@@ -122,11 +122,12 @@
       <version>1.1.10.4</version>
     </dependency>
     <!-- Pin commons-beanutils to override the 1.9.2 pulled in transitively via
-         everit json-schema → commons-validator. Fixes CVE-2025-48734, CVE-2019-10086. -->
+         everit json-schema → commons-validator. Fixes CVE-2025-48734, CVE-2019-10086;
+         1.11.0 adds the Improper-Access-Control fix (GHSA on top of those). -->
     <dependency>
       <groupId>commons-beanutils</groupId>
       <artifactId>commons-beanutils</artifactId>
-      <version>1.10.1</version>
+      <version>1.11.0</version>
     </dependency>
     <!-- Pin msal4j to override the older version pulled in transitively by
          mssql-jdbc → azure-identity. Fixes CVE-2024-35255 (elevation of
@@ -138,17 +139,19 @@
       <version>1.24.1</version>
     </dependency>
     <!-- Pin log4j-api / log4j-core to override the 2.19.0 pulled in transitively
-         by elasticsearch:8.14.0. Fixes CVE-2025-68161, CVE-2026-34477, CVE-2026-34479,
-         CVE-2026-34480, CVE-2026-34481. (Not Log4Shell — that one was patched long ago.) -->
+         by elasticsearch (the project's logger is logback; log4j2 is only present
+         for ES to load its own appenders). 2.26.0 also fixes the Rfc5424Layout log
+         injection, XmlLayout XML-1.0 silent drop, Socket-Appender TLS hostname
+         non-verification, and verifyHostName silent-ignore advisories that hit 2.24.3. -->
     <dependency>
       <groupId>org.apache.logging.log4j</groupId>
       <artifactId>log4j-api</artifactId>
-      <version>2.24.3</version>
+      <version>2.26.0</version>
     </dependency>
     <dependency>
       <groupId>org.apache.logging.log4j</groupId>
       <artifactId>log4j-core</artifactId>
-      <version>2.24.3</version>
+      <version>2.26.0</version>
     </dependency>
 
     <dependency>
@@ -281,7 +284,7 @@
     <dependency>
       <groupId>com.nimbusds</groupId>
       <artifactId>nimbus-jose-jwt</artifactId>
-      <version>9.37.2</version>
+      <version>10.5</version>
     </dependency>
     <dependency>
       <groupId>com.github.OpenBankProject</groupId>
@@ -299,7 +302,7 @@
     <dependency>
       <groupId>com.nimbusds</groupId>
       <artifactId>oauth2-oidc-sdk</artifactId>
-      <version>9.27</version>
+      <version>11.37.1</version>
     </dependency>
     <!-- ********** flexmark START ********** -->
     <!-- Library flexmark-all v0.40.8 is replaced with used modules -->
@@ -391,22 +394,22 @@
     <dependency>
       <groupId>io.grpc</groupId>
       <artifactId>grpc-netty-shaded</artifactId>
-      <version>1.68.3</version>
+      <version>1.75.0</version>
     </dependency>
     <dependency>
       <groupId>io.grpc</groupId>
       <artifactId>grpc-protobuf</artifactId>
-      <version>1.68.3</version>
+      <version>1.75.0</version>
     </dependency>
     <dependency>
       <groupId>io.grpc</groupId>
       <artifactId>grpc-stub</artifactId>
-      <version>1.68.3</version>
+      <version>1.75.0</version>
     </dependency>
     <dependency>
       <groupId>io.grpc</groupId>
       <artifactId>grpc-services</artifactId>
-      <version>1.68.3</version>
+      <version>1.75.0</version>
     </dependency>
     <dependency>
       <groupId>org.asynchttpclient</groupId>
@@ -542,10 +545,13 @@
       <scope>test</scope>
     </dependency>
 
+    <!-- com.sun.mail line is abandoned (last release 2.0.2, still vulnerable to
+         CVE-2025-7962 SMTP injection). Migrated to org.eclipse.angus, the official
+         Eclipse successor; jakarta.mail.* API is identical. -->
     <dependency>
-        <groupId>com.sun.mail</groupId>
+        <groupId>org.eclipse.angus</groupId>
         <artifactId>jakarta.mail</artifactId>
-        <version>2.0.1</version>
+        <version>2.0.5</version>
     </dependency>
     <dependency>
         <groupId>jakarta.activation</groupId>

obp-api/src/main/scala/code/api/util/CertificateUtil.scala

@@ -4,7 +4,7 @@ import code.api.CertificateConstants
 import code.api.util.CryptoSystem.CryptoSystem
 import code.api.util.SelfSignedCertificateUtil.generateSelfSignedCert
 import code.util.Helper.MdcLoggable
-import com.nimbusds.jose._
+import com.nimbusds.jose.{EncryptionMethod, JWEAlgorithm, JWEHeader, JWSAlgorithm, JWSHeader, JWSSigner}
 import com.nimbusds.jose.crypto.{MACSigner, RSAEncrypter, RSASSASigner}
 import com.nimbusds.jose.util.X509CertUtils
 import com.nimbusds.jwt.{EncryptedJWT, JWTClaimsSet}

obp-api/src/main/scala/code/api/util/JwtUtil.scala

@@ -262,7 +262,6 @@ object JwtUtil extends MdcLoggable {
   def validateIdToken(idToken: String, remoteJWKSetUrl: String): Box[IDTokenClaimsSet] = {
     import java.net._
 
-    import com.nimbusds.jose._
     import com.nimbusds.oauth2.sdk.id._
     import com.nimbusds.openid.connect.sdk.validators._
 

obp-api/src/main/scala/code/api/util/http4s/Http4sApp.scala

@@ -3,6 +3,7 @@ package code.api.util.http4s
 import cats.data.{Kleisli, OptionT}
 import cats.effect.IO
 import code.api.util.APIUtil
+import code.api.util.http4s.Http4sRequestAttributes
 import com.openbankproject.commons.util.{ApiVersion, ScannedApiVersion}
 import org.http4s._
 import org.typelevel.ci.CIString
@@ -67,27 +68,58 @@ object Http4sApp {
   private val v210Routes: HttpRoutes[IO] = gate(ApiVersion.v2_1_0, code.api.v2_1_0.Http4s210.wrappedRoutesV210Services)
   private val v220Routes: HttpRoutes[IO] = gate(ApiVersion.v2_2_0, code.api.v2_2_0.Http4s220.wrappedRoutesV220Services)
   private val v300Routes: HttpRoutes[IO] = gate(ApiVersion.v3_0_0, code.api.v3_0_0.Http4s300.wrappedRoutesV300Services)
+  private val v310Routes: HttpRoutes[IO] = gate(ApiVersion.v3_1_0, code.api.v3_1_0.Http4s310.wrappedRoutesV310Services)
+  private val v400Routes: HttpRoutes[IO] = gate(ApiVersion.v4_0_0, code.api.v4_0_0.Http4s400.wrappedRoutesV400Services)
   private val v500Routes: HttpRoutes[IO] = gate(ApiVersion.v5_0_0, code.api.v5_0_0.Http4s500.wrappedRoutesV500Services)
+  private val v510Routes: HttpRoutes[IO] = gate(ApiVersion.v5_1_0, code.api.v5_1_0.Http4s510.wrappedRoutesV510Services)
   private val v700Routes: HttpRoutes[IO] = gate(ApiVersion.v7_0_0, code.api.v7_0_0.Http4s700.wrappedRoutesV700Services)
 
   /**
-   * Build the base HTTP4S routes with priority-based routing
+   * Build the base HTTP4S routes with priority-based routing.
+   *
+   * Body caching: http4s request bodies are single-shot streams. The first version's
+   * `ResourceDocMiddleware.fromRequest` consumes the body to build CallContext; any later
+   * bridge hop (v400→v310→v300→…→v210) that re-reads `req.bodyText` gets an empty stream
+   * and the eventual handler returns 500 because JSON parsing fails. We pre-read the body
+   * here and stash it in `cachedBodyKey`, so every downstream `fromRequest` reads from the
+   * attribute instead of the (now-drained) stream. GETs/DELETEs/HEADs/OPTIONS skip this.
    */
+  private val noBodyMethods: Set[Method] = Set(Method.GET, Method.DELETE, Method.HEAD, Method.OPTIONS)
+
+  private def cacheBodyOnce(req: Request[IO]): IO[Request[IO]] = {
+    if (req.attributes.lookup(Http4sRequestAttributes.cachedBodyKey).isDefined) IO.pure(req)
+    else if (noBodyMethods.contains(req.method)) IO.pure(req.withAttribute(Http4sRequestAttributes.cachedBodyKey, Option.empty[String]))
+    else req.body.compile.to(Array).map { bytes =>
+      val cached: Option[String] = if (bytes.isEmpty) None else Some(new String(bytes, "UTF-8"))
+      // Replay the bytes on every subsequent stream read so the Lift fallback and any
+      // handler that still reads req.body sees the same payload. fs2.Stream.emits is
+      // pure — re-evaluating it yields a fresh stream of the same bytes.
+      req
+        .withBodyStream(fs2.Stream.emits(bytes).covary[IO])
+        .withAttribute(Http4sRequestAttributes.cachedBodyKey, cached)
+    }
+  }
+
   private def baseServices: HttpRoutes[IO] = Kleisli[HttpF, Request[IO], Response[IO]] { req: Request[IO] =>
-    corsHandler.run(req)
-      .orElse(AppsPage.routes.run(req))
-      .orElse(StatusPage.routes.run(req))
-      .orElse(v500Routes.run(req))
-      .orElse(v700Routes.run(req))
-      .orElse(code.api.berlin.group.v2.Http4sBGv2.wrappedRoutes.run(req))
-      .orElse(v300Routes.run(req))
-      .orElse(v220Routes.run(req))
-      .orElse(v210Routes.run(req))
-      .orElse(v200Routes.run(req))
-      .orElse(v140Routes.run(req))
-      .orElse(v130Routes.run(req))
-      .orElse(v121Routes.run(req))
-      .orElse(Http4sLiftWebBridge.routes.run(req))
+    OptionT.liftF(cacheBodyOnce(req)).flatMap { req =>
+      corsHandler.run(req)
+        .orElse(AppsPage.routes.run(req))
+        .orElse(StatusPage.routes.run(req))
+        .orElse(v510Routes.run(req))
+        .orElse(v500Routes.run(req))
+        .orElse(v700Routes.run(req))
+        .orElse(code.api.berlin.group.v2.Http4sBGv2.wrappedRoutes.run(req))
+        .orElse(v400Routes.run(req))
+        .orElse(v310Routes.run(req))
+        .orElse(v300Routes.run(req))
+        .orElse(v220Routes.run(req))
+        .orElse(v210Routes.run(req))
+        .orElse(v200Routes.run(req))
+        .orElse(v140Routes.run(req))
+        .orElse(v130Routes.run(req))
+        .orElse(v121Routes.run(req))
+        .orElse(Http4sLiftWebBridge.routes.run(req))
+    }
   }
 
   /**