feat(proxy): enforce shared scheduled network maintenance

docs/superpowers/plans/2026-10-19-network-maintenance.md

@@ -0,0 +1,8 @@
+# Implementation plan
+
+1. Config Service: multi-document least-privilege grants; atomic create authorization; tests and verification.
+2. Proxy: pure validated schedule engine and bypass admission/kick tests; Config polling; LoginEvent and server-list integration; permission manifest and localization; build.
+3. Forge: strict typed contract; bounded private Config client/CAS and atomic create; environment-specific RBAC routes; tests/typecheck.
+4. Portal: protected environment-aware network page; schedule/whitelist/announcements editor, explicit bypass kick behavior, MOTD editor and safe preview; tests/typecheck/build.
+5. Infra: exact grants and NATS publisher subjects, explicit environment connections; relevant checks.
+6. Strong primary security/correctness review of all domains, cross-proxy behavior, conflicts, unavailable Production, UUID handling and bypass kicks. Create reviewable PRs and record deployment prerequisites. Never turn maintenance on to test.

docs/superpowers/specs/2026-10-19-network-maintenance-design.md

@@ -0,0 +1,41 @@
+# Network maintenance and MOTD
+
+Approved scope: Config Service backed maintenance, scheduled start, optional automatic end, separately estimated end, permission/UUID whitelist bypass, kick existing non-bypass players, configurable announcements, all proxies/Java/Bedrock, Portal maintenance and MOTD editors. Never enable live maintenance as a deployment test.
+
+## Contract
+
+Config app `velocity`, environment `stage` or `prod`, documents `maintenance/active` and existing `motd/active`. Stage and Production are separate connections, never fall back across environments.
+
+Maintenance JSON (strict at Forge, compatible explicit parser in proxy):
+```
+{
+  "schemaVersion": 1,
+  "enabled": false,
+  "id": "UUID",
+  "startsAt": "2026-10-19T20:00:00.000Z",
+  "endsAt": null,
+  "estimatedEndAt": null,
+  "reason": "Wartungsarbeiten",
+  "joinMessage": "<red>Wartungsarbeiten</red>\n{reason}\nVoraussichtliches Ende: {end}",
+  "motd": "<red>Wartungsarbeiten</red>\n<gray>Voraussichtliches Ende: {end}</gray>",
+  "kickExisting": true,
+  "whitelist": [],
+  "announcementSeconds": [10800,7200,3600,1800,600,300,120,60],
+  "announcementMessage": "<yellow>Wartungsarbeiten in {remaining}: {reason}</yellow>"
+}
+```
+Required fields above. UUID canonical, no UUID-version constraint (Floodgate support). Dates ISO UTC. endsAt/estimatedEndAt nullable, if set > startsAt. endsAt automatically disables effective maintenance; estimatedEndAt only displays ETA. Limits: reason 500 chars, messages/MOTD 4096 chars, whitelist 1000 UUIDs, offsets <=32 distinct integers 1..604800. Preserve ID on edits; create a new ID when intentionally starting a new episode. Offset threshold local announcements never replay after bootstrap/restart. Whitelist OR `grounds.maintenance.bypass` exempts joins AND existing-player kicks. Kick only on transition/episode entry, never merely because a config refresh changed metadata. Local player enumeration and delivery on every proxy ensures network-wide behavior without duplicate cross-proxy broadcasts.
+
+MOTD JSON preserves existing shape: {text:string,maxPlayers?:integer,source?:string,updatedBy?:string,updatedAt?:string}. text 1..4096, maxPlayers 1..1000000. Forge stamps actor/time. Preview is safely rendered React text, supported-format preview may be partial and must say so.
+
+Forge routes:
+- GET /v1/admin/network/:environment => {environment,available,maintenance:Document|null,motd:Document|null}. Config missing document returns defaults in Document.settings and version:null. Unconfigured environment => available:false, both null.
+- PUT /v1/admin/network/:environment/maintenance or /motd body {expectedVersion:string|null,settings:...} => Document.
+- Document = {version:string|null,settings:...,updatedAt:string|null,updatedBy:string|null}.
+- Decimal version strings preserve BIGINT precision. null means atomic create-if-absent using Config POST; existing updates require If-Match. Conflict 409. No unconditional writes or arbitrary config paths.
+- Permissions ADMIN_AREA_ACCESS plus NETWORK_STAGE_VIEW/NETWORK_STAGE_MANAGE or NETWORK_PRODUCTION_VIEW/NETWORK_PRODUCTION_MANAGE. Manage implies ability to view via either-permission check. Service-account Forge JWTs cannot enter these routes.
+- Forge dedicated env NETWORK_CONFIG_STAGE_URL/TOKEN_FILE and NETWORK_CONFIG_PRODUCTION_URL/TOKEN_FILE. Re-read projected token per request. Fixed paths, 5s timeout, no redirects, bounded responses, sanitized errors.
+
+Proxy caches valid documents, polls every 5s and refreshes on NATS invalidation hints. Pings/logins never do remote IO. Keep last valid on failure, deny ordinary new joins once read freshness >30s; bypass still allowed. No mass kick caused by config outages. HTTP 404 is known disabled initial configuration. LoginEvent uses verified UUID and permission provider. Maintenance ping overrides normal MOTD; Java may advertise Maintenance status with incompatible protocol while actual LoginEvent enforces admission. Bedrock sees MOTD and login denial.
+
+Infrastructure: Forge SA receives exactly existing network/stage/resourcepacks/global plus velocity/stage/{motd,maintenance}/active via pipe-separated grants. Config create endpoint accepts exact writers consistent with PUT. NATS Stage Config publisher adds config.velocity.stage.changed to existing subject. Preserve live bindings/flags and never change unrelated networking enforcement.

velocity/src/main/kotlin/gg/grounds/proxy/velocity/GroundsProxyPlugin.kt

@@ -2,7 +2,10 @@ package gg.grounds.proxy.velocity
 
 import com.google.gson.JsonParser
 import com.google.inject.Inject
+import com.velocitypowered.api.event.PostOrder
+import com.velocitypowered.api.event.ResultedEvent
 import com.velocitypowered.api.event.Subscribe
+import com.velocitypowered.api.event.connection.LoginEvent
 import com.velocitypowered.api.event.connection.PostLoginEvent
 import com.velocitypowered.api.event.player.ServerPostConnectEvent
 import com.velocitypowered.api.event.proxy.ProxyInitializeEvent
@@ -12,6 +15,7 @@ import com.velocitypowered.api.plugin.Plugin
 import com.velocitypowered.api.proxy.Player
 import com.velocitypowered.api.proxy.ProxyServer
 import com.velocitypowered.api.proxy.messages.MinecraftChannelIdentifier
+import com.velocitypowered.api.scheduler.ScheduledTask
 import gg.grounds.BuildInfo
 import gg.grounds.i18n.Translations
 import gg.grounds.presentation.VelocitySystemMessages
@@ -31,6 +35,9 @@ import gg.grounds.proxy.velocity.command.RegionCommand
 import gg.grounds.proxy.velocity.handler.NatsHandler
 import gg.grounds.proxy.velocity.listener.LobbyCountsListener
 import gg.grounds.proxy.velocity.listener.ProxyHopListener
+import gg.grounds.proxy.velocity.maintenance.MaintenanceAccess
+import gg.grounds.proxy.velocity.maintenance.MaintenanceConfigStore
+import gg.grounds.proxy.velocity.maintenance.MaintenanceEngine
 import gg.grounds.proxy.velocity.metrics.MetricsConfig
 import gg.grounds.proxy.velocity.metrics.ProxyMetrics
 import gg.grounds.proxy.velocity.metrics.ProxySnapshot
@@ -44,6 +51,8 @@ import gg.grounds.proxy.velocity.tab.TabList
 import io.nats.client.Subscription
 import java.time.Duration
 import java.util.concurrent.TimeUnit
+import java.util.concurrent.atomic.AtomicBoolean
+import net.kyori.adventure.text.minimessage.MiniMessage
 import net.kyori.adventure.text.serializer.gson.GsonComponentSerializer
 import org.slf4j.Logger
 
@@ -207,6 +216,12 @@ constructor(private val proxy: ProxyServer, private val logger: Logger) {
      */
     @Volatile private var motdManager: MotdManager? = null
     private var motdStore: MotdConfigStore? = null
+    @Volatile private var maintenanceStore: MaintenanceConfigStore? = null
+    private val maintenance = MaintenanceEngine()
+    private val maintenanceRefreshRunning = AtomicBoolean()
+    private val maintenanceClosed = AtomicBoolean()
+    private var maintenanceRefreshTask: ScheduledTask? = null
+    private var maintenanceApplyTask: ScheduledTask? = null
 
     @Subscribe
     fun onInitialize(event: ProxyInitializeEvent) {
@@ -278,6 +293,7 @@ constructor(private val proxy: ProxyServer, private val logger: Logger) {
             }
 
         startMotd()
+        startMaintenance()
 
         subscribeForProxy()
 
@@ -417,6 +433,78 @@ constructor(private val proxy: ProxyServer, private val logger: Logger) {
         )
     }
 
+    /** Starts the shared maintenance reader. Every proxy polls and only acts on its own players. */
+    private fun startMaintenance() {
+        val target = resolveConfigServiceTarget() ?: return
+        val app = env("CONFIG_APP") ?: DEFAULT_CONFIG_APP
+        val configEnv = env("CONFIG_ENV") ?: env("GROUNDS_PERMISSION_ENVIRONMENT") ?: return
+        maintenanceStore = MaintenanceConfigStore.open(app, configEnv, target)
+        // Hints shorten propagation after Portal saves; polling remains the correctness fallback.
+        crossProxySubscriptions +=
+            natsHandler.subscribe("config.$app.$configEnv.changed") {
+                if (!maintenanceClosed.get()) {
+                    proxy.scheduler
+                        .buildTask(this, Runnable { refreshMaintenanceConfiguration() })
+                        .schedule()
+                }
+            }
+        maintenanceRefreshTask =
+            proxy.scheduler
+                .buildTask(this, Runnable { refreshMaintenanceConfiguration() })
+                .repeat(5, TimeUnit.SECONDS)
+                .schedule()
+        // The local schedule still advances while Config is unavailable: end times and countdowns
+        // are decisions from the cached signed document, not from a successful request this tick.
+        maintenanceApplyTask =
+            proxy.scheduler
+                .buildTask(this, Runnable { applyMaintenance() })
+                .repeat(1, TimeUnit.SECONDS)
+                .schedule()
+    }
+
+    private fun refreshMaintenanceConfiguration() {
+        if (maintenanceClosed.get()) return
+        val store = maintenanceStore ?: return
+        if (!maintenanceRefreshRunning.compareAndSet(false, true)) return
+        try {
+            val document = store.read()
+            if (!maintenanceClosed.get()) maintenance.adopt(document)
+        } catch (failure: Exception) {
+            maintenance.refreshFailed()
+            logger.warn(
+                "Maintenance refresh failed (reason={}); retaining last valid state",
+                failure.message,
+            )
+        } finally {
+            maintenanceRefreshRunning.set(false)
+        }
+    }
+
+    private fun applyMaintenance() {
+        if (maintenanceClosed.get()) return
+        val now = java.time.Instant.now()
+        maintenance.announcementsAt(now).forEach {
+            maintenance.announcementMessage(now)?.let { message ->
+                val component = MiniMessage.miniMessage().deserialize(message)
+                proxy.allPlayers.forEach { player -> player.sendMessage(component) }
+            }
+        }
+        proxy.allPlayers.forEach { player ->
+            if (
+                MaintenanceAccess.shouldKick(
+                    maintenance,
+                    player.uniqueId,
+                    { player.hasPermission(MaintenanceAccess.BYPASS_PERMISSION) },
+                    now,
+                )
+            ) {
+                player.disconnect(
+                    MiniMessage.miniMessage().deserialize(maintenance.denialMessage(now))
+                )
+            }
+        }
+    }
+
     private fun env(name: String): String? = System.getenv(name)?.trim()?.takeIf { it.isNotEmpty() }
 
     /**
@@ -438,7 +526,8 @@ constructor(private val proxy: ProxyServer, private val logger: Logger) {
     fun onProxyPing(event: ProxyPingEvent) {
         val count = networkPlayerCount
         val motd = motdManager
-        if (count == null && motd == null) return
+        val maintenanceMotd = maintenanceStore?.let { maintenance.motd() }
+        if (count == null && motd == null && maintenanceMotd == null) return
 
         val builder = event.ping.asBuilder()
         count?.let { builder.onlinePlayers(it) }
@@ -450,9 +539,30 @@ constructor(private val proxy: ProxyServer, private val logger: Logger) {
                 builder.description(it)
             }
         }
+        maintenanceMotd?.let { builder.description(MiniMessage.miniMessage().deserialize(it)) }
         event.ping = builder.build()
     }
 
+    @Subscribe(order = PostOrder.LAST)
+    fun onLogin(event: LoginEvent) {
+        if (!event.result.isAllowed) return
+        if (maintenanceStore == null) return
+        val player = event.player
+        val admission =
+            MaintenanceAccess.admission(
+                maintenance,
+                player.uniqueId,
+                { player.hasPermission(MaintenanceAccess.BYPASS_PERMISSION) },
+                java.time.Instant.now(),
+            )
+        if (!admission.allowed) {
+            event.result =
+                ResultedEvent.ComponentResult.denied(
+                    MiniMessage.miniMessage().deserialize(admission.message!!)
+                )
+        }
+    }
+
     @Subscribe
     fun onLogin(event: PostLoginEvent) {
         bedrockRoster?.refresh()
@@ -477,6 +587,11 @@ constructor(private val proxy: ProxyServer, private val logger: Logger) {
         crossProxySubscriptions.forEach { natsHandler.unsubscribe(it) }
         crossProxySubscriptions.clear()
         motdStore?.close()
+        maintenanceClosed.set(true)
+        maintenanceRefreshTask?.cancel()
+        maintenanceApplyTask?.cancel()
+        maintenanceStore?.close()
+        maintenanceStore = null
         metrics?.let { runCatching { it.close() } }
         metrics = null
         if (this::natsHandler.isInitialized) {

velocity/src/main/kotlin/gg/grounds/proxy/velocity/maintenance/MaintenanceAccess.kt

@@ -0,0 +1,33 @@
+package gg.grounds.proxy.velocity.maintenance
+
+import java.time.Instant
+import java.util.UUID
+
+/**
+ * Keeps Velocity permission lookup at the listener boundary and makes whitelist precedence
+ * explicit.
+ */
+object MaintenanceAccess {
+    const val BYPASS_PERMISSION = "grounds.maintenance.bypass"
+
+    fun admission(
+        engine: MaintenanceEngine,
+        uuid: UUID,
+        hasPermission: () -> Boolean,
+        now: Instant,
+    ): MaintenanceEngine.Admission =
+        engine.admission(uuid, bypass(engine, uuid, hasPermission), now)
+
+    fun shouldKick(
+        engine: MaintenanceEngine,
+        uuid: UUID,
+        hasPermission: () -> Boolean,
+        now: Instant,
+    ): Boolean = engine.shouldKick(uuid, bypass(engine, uuid, hasPermission), now)
+
+    private fun bypass(
+        engine: MaintenanceEngine,
+        uuid: UUID,
+        hasPermission: () -> Boolean,
+    ): Boolean = engine.whitelisted(uuid) || runCatching(hasPermission).getOrDefault(false)
+}