Make Hot-Reload Transactional with Reliable Last-Known-Good Rollback

[aaronburtle]

Design: Transactional Hot-Reload and Reliable Last-Known-Good Rollback

Status

Proposed design

How to Read the Source References

Links under Current code touchpoints identify existing source files whose current behavior is being described and which will likely need to be refactored during implementation. They are not example implementations of the proposed design.

The proposed classes, interfaces, algorithms, and code fragments in this document describe the new implementation. The linked source files show where the current behavior lives so the future implementation can be grounded in the existing architecture.

---

Problem Statement

The current hot-reload pipeline treats a configuration as last-known-good as soon as configuration validation succeeds, before all dependent runtime services have successfully refreshed.

The current sequence is approximately:

1. Parse the new configuration.
2. Replace the loader's active RuntimeConfig.
3. Validate the new configuration.
4. Save it as LastValidRuntimeConfig.
5. Raise ordered hot-reload events.
6. Each dependent service mutates its live state.

Current code touchpoints

• FileSystemRuntimeConfigLoader.cs loads a new configuration and begins the hot-reload sequence.
• RuntimeConfigLoader.cs stores active/LKG state and raises the ordered component events.
• RuntimeConfigProvider.cs validates the newly loaded configuration and currently marks it as LKG before component refresh finishes.
• HotReloadEventHandler.cs dispatches the existing string-named refresh events.

This creates two problems:

1. A dependent service can fail after the new configuration has already become the LKG.
2. Services that refreshed before the failure are not restored to their previous state.

RestoreLkgConfig() restores only the configuration reference. It does not restore metadata providers, authorization maps, query engines, GraphQL schemas, or other cached state.

---

Example Failure

Assume configuration A is active and fully functional.

Configuration B:

• passes ordinary runtime configuration validation;
• changes one or more entities;
• causes a later runtime component to fail while rebuilding.

An MCP-specific example would be two entities that normalize to the same tool name, such as ReadRecords colliding with the built-in read_records tool.

The existing sequence can result in:

• active RuntimeConfig: B;
• LastValidRuntimeConfig: B;
• query managers: B;
• metadata providers: B;
• authorization state: B;
• MCP registry: A because its refresh failed;
• other components: potentially either A or B.

If the next edit is malformed, restoring the LKG restores B rather than the last completely operational configuration A.

This is a general hot-reload problem, not an MCP-specific problem.

---

Goals

1. Treat a configuration as active only after every critical runtime component can apply it.
2. Define LKG as the last configuration successfully applied to the entire runtime.
3. Build replacement component state without mutating live state.
4. Publish prepared state only after every participant has prepared successfully.
5. Restore all already-committed components if an unexpected commit failure occurs.
6. Keep the previous runtime completely usable when parsing, validation, or preparation fails.
7. Serialize overlapping file-change notifications.
8. Signal configuration change tokens only after a successful commit.
9. Distinguish critical transactional work from noncritical post-commit notifications.
10. Produce clear structured logs for each reload generation and failure stage.

---

Non-Goals

The initial implementation does not need to provide:

• dynamic ASP.NET endpoint mapping;
• hot-reload of endpoint paths;
• hot-enabling services that were not registered at startup;
• perfect generation isolation for requests already in flight;
• rollback of external mutations performed outside DAB;
• persistence of LKG state across process restarts.

Settings that require endpoint or middleware reconstruction must be rejected as restart-required changes during hot-reload.

---

Required Invariants

After a hot-reload attempt completes, exactly one of these must be true:

Successful reload

• active configuration is the candidate;
• LKG is the candidate;
• every critical component contains candidate state;
• change-token subscribers are notified;
• noncritical post-commit observers are invoked.

Failed reload

• active configuration remains the previous configuration;
• LKG remains the previous configuration;
• every critical component contains previous state;
• candidate resources are disposed;
• no configuration change token is signaled.

A configuration that merely passes schema validation must not become LKG.

---

Proposed Architecture

1. Separate Candidate Configuration From Active Configuration

Loading a file for hot-reload must produce a candidate without immediately replacing the active configuration.

Introduce an immutable value such as:

public sealed record RuntimeConfigCandidate(
    long Generation,
    RuntimeConfig Config,
    string SourcePath,
    DateTimeOffset DetectedAt);

Add a candidate-loading API to RuntimeConfigLoader, for example:

public bool TryLoadCandidate(
    string path,
    [NotNullWhen(true)] out RuntimeConfigCandidate? candidate,
    bool isDevelopmentMode,
    DeserializationVariableReplacementSettings replacementSettings);

Candidate loading may:

• read the file;
• deserialize it;
• replace environment and Key Vault references;
• preserve startup-bound values where required;
• perform parse-level validation.

Candidate loading must not:

• replace the active RuntimeConfig;
• modify the LKG;
• signal a change token;
• invoke component refresh events.

Remove the hot-reload dependence on:

• IsNewConfigDetected;
• IsNewConfigValidated;
• DoesConfigNeedValidation().

These flags exist because the current loader mutates active state before validation. They should no longer be necessary once candidates are isolated.

2. Make RuntimeConfigProvider the Active State Owner

RuntimeConfigProvider is documented as the owner of active runtime configuration, but the mutable state currently lives primarily in RuntimeConfigLoader.

Move active-state publication behind RuntimeConfigProvider.

Suggested state:

private RuntimeConfig _activeConfig;
private RuntimeConfig _lastAppliedConfig;
private long _activeGeneration;

Expose internal operations such as:

internal void CommitCandidate(RuntimeConfigCandidate candidate);
internal RuntimeConfigSnapshot GetActiveSnapshot();

GetConfig() and TryGetConfig() must return only committed state.

Use atomic reference publication, such as Volatile.Read() and Interlocked.Exchange(), where appropriate.

Rename LastValidRuntimeConfig to LastAppliedRuntimeConfig or equivalent. “Valid” is ambiguous because schema-valid does not mean successfully applied.

3. Introduce a Hot-Reload Coordinator

Add a singleton service in Core:

public sealed class RuntimeConfigHotReloadCoordinator

Responsibilities:

1. Serialize reload attempts.
2. Load or receive the latest candidate.
3. Validate hot-reload compatibility.
4. Validate the candidate configuration.
5. Prepare every transactional participant.
6. Commit every prepared change.
7. Publish the candidate through RuntimeConfigProvider.
8. Update the LKG.
9. Signal configuration change tokens.
10. Run post-commit observers.
11. Roll back committed participants on unexpected commit failure.
12. Dispose candidate or retired state correctly.

Use a SemaphoreSlim or a single-reader Channel to prevent concurrent reload transactions.

The file-watcher callback should enqueue a reload request and return quickly. It should not perform database initialization or other long-running work directly on the file-watcher thread.

Multiple notifications for the same file save should be coalesced. If a new notification arrives during a reload, process the latest file contents once the current attempt finishes.

4. Add Transactional Participant Contracts

Replace critical string-based hot-reload events with explicit transactional participants.

Suggested contracts:

public interface IHotReloadParticipant
{
    string Name { get; }

    HotReloadStage Stage { get; }

    ValueTask<IPreparedHotReloadChange> PrepareAsync(
        HotReloadContext context,
        CancellationToken cancellationToken);
}

public interface IPreparedHotReloadChange : IAsyncDisposable
{
    void Commit();

    void Rollback();
}

PrepareAsync() may perform:

• validation;
• database metadata discovery;
• construction of replacement dictionaries;
• construction of replacement clients or engines;
• schema generation;
• any operation that can reasonably fail.

PrepareAsync() must not mutate live service state.

Commit() must:

• perform only in-memory publication;
• preferably be an Interlocked.Exchange() or equivalent reference swap;
• perform no file, database, or network I/O;
• be designed not to throw.

Rollback() must:

• restore the exact previous state captured before commit;
• perform only in-memory publication;
• be safe when called after a partial commit sequence.

Resource ownership rules:

1. Before commit, the prepared change owns candidate resources.
2. After commit, the live service owns candidate resources.
3. Previous resources must remain available until the complete transaction succeeds.
4. After successful completion, previous resources may be disposed.
5. After failed preparation, candidate resources must be disposed.
6. After rollback, candidate resources must be disposed and previous resources retained.

5. Define Explicit Reload Stages

Replace ordering hidden in event invocation order with a typed stage model.

Suggested initial stages:

public enum HotReloadStage
{
    QueryManagers = 100,
    MetadataProviders = 200,
    QueryEngines = 300,
    MutationEngines = 400,
    Documentation = 500,
    Authorization = 600,
    McpRegistry = 700,
    GraphQLSchema = 800,
    AuthenticationOptions = 900,
    Logging = 1000
}

The coordinator prepares participants in stage order.

Rollback occurs in reverse commit order.

Participants in the same stage should run sequentially initially. Parallel preparation can be considered later only when dependencies are explicitly understood.

6. Add a Transaction-Local Preparation Context

Some candidate components depend on other candidate components. For example, candidate metadata providers may require candidate query-manager state.

HotReloadContext should contain:

public sealed class HotReloadContext
{
    public RuntimeConfigSnapshot Current { get; }

    public RuntimeConfigCandidate Candidate { get; }

    public HotReloadPreparedState PreparedState { get; }
}

HotReloadPreparedState may provide typed transaction-local state:

public void Add<T>(T state);
public T GetRequired<T>();

This is not application DI. It exists only for one reload transaction and allows later preparation stages to consume state prepared by earlier stages without publishing it globally.

Dependencies between stages must be documented and covered by tests.

---

Transaction Workflow

Phase 1: Detect and Load

1. File watcher detects a change.
2. Reload request is queued.
3. Coordinator takes the reload serialization lock.
4. Loader reads the newest file contents.
5. Loader creates a RuntimeConfigCandidate.
6. Active configuration remains unchanged.

Phase 2: Compatibility Validation

Compare the current and candidate configurations.

Reject changes to settings that cannot safely reload, such as:

• endpoint paths;
• middleware shape;
• initially unregistered services;
• host mode;
• other startup-bound settings discovered during implementation.

Return a clear restart-required error rather than accepting a configuration that does not match the running endpoint table.

Create a dedicated component such as:

public sealed class HotReloadCompatibilityValidator

The validator should report every incompatible property in one result where practical.

Phase 3: Candidate Validation

Validate the candidate directly.

Add candidate-aware validation APIs so RuntimeConfigValidator does not need the candidate to become globally active before validating it.

Separate validation into:

• static configuration validation;
• runtime preparation validation.

Validation that requires constructing metadata providers or contacting a database belongs in participant preparation.

Do not call SetLkgConfig() during this phase.

Phase 4: Prepare

For each participant in stage order:

1. Call PrepareAsync().
2. Record the returned prepared change.
3. Make any required prepared state available to later stages.

If a participant fails:

1. Stop preparation.
2. Dispose all prepared changes in reverse order.
3. Keep active configuration and live services unchanged.
4. Log the participant and exception.
5. Do not signal change tokens.

Phase 5: Commit

Once every participant has prepared successfully:

1. Enter the short commit section.
2. Call Commit() for each prepared change in stage order.
3. Publish the candidate through RuntimeConfigProvider.
4. Set the candidate as LastAppliedRuntimeConfig.
5. Increment the active generation.
6. Exit the commit section.

No validation, database work, or schema generation should happen during commit.

Phase 6: Unexpected Commit Failure

Although commits must be designed not to throw, handle unexpected failures defensively.

If commit fails:

1. Do not publish the candidate configuration.
2. Call Rollback() on already-committed changes in reverse order.
3. Dispose candidate state.
4. Keep the previous configuration and LKG.
5. Log a critical transaction failure.

If rollback itself fails, the runtime state is indeterminate. In that case:

• report unhealthy status;
• log at critical severity;
• stop the application gracefully rather than continue with unknown mixed state.

Phase 7: Post-Commit

Only after the transaction succeeds:

1. Signal RuntimeConfigProvider change tokens.
2. Refresh IOptionsMonitor consumers.
3. Send MCP list-change notifications.
4. Emit informational logs and telemetry.
5. Dispose retired previous-generation resources.

Post-commit observer failure must be logged but must not roll back an already committed runtime. Any observer required for correctness belongs in the transactional participant set instead.

---

Change-Token Semantics

RuntimeConfigProvider.GetChangeToken() currently signals before the ordered component refresh pipeline completes.

Change this behavior so that the token means:

A complete new runtime generation was successfully committed.

The token must not mean merely:

A new file was detected or parsed.

This ensures JWT options and other change-token consumers never observe a candidate that later fails to apply.

---

Components to Convert

Query Manager Factory

Refactor QueryManagerFactory so it can build replacement dictionaries without assigning them to live fields.

Prepared state should include:

• query builders;
• query executors;
• database exception parsers.

Commit should atomically swap one containing state object instead of assigning three fields independently.

Current code touchpoint

• QueryManagerFactory.cs

Metadata Provider Factory

Do not clear the live provider dictionary before replacement providers initialize.

Preparation should:

1. Build a separate provider dictionary.
2. Initialize all providers.
3. Collect initialization errors.
4. Fail without touching live providers if initialization fails.

Commit should swap the complete provider snapshot.

Current code touchpoint

• MetadataProviderFactory.cs

Query and Mutation Engine Factories

Build complete replacement factory state during preparation.

Commit each complete state with one reference swap.

Current code touchpoints

• QueryEngineFactory.cs
• MutationEngineFactory.cs

Authorization Resolver

Build the candidate permission map and explicit-role set separately.

Publish both inside one immutable authorization snapshot.

Current code touchpoint

• AuthorizationResolver.cs

OpenAPI Documentor

Generate candidate documentation without replacing the currently served document.

Publish only during commit.

Current code touchpoint

• OpenApiDocumentor.cs

GraphQL Schema

Combine eviction, creation, and refresh into one transactional participant.

Preparation should build and validate the replacement schema without evicting the live schema.

Commit should publish the prepared schema or executor.

Do not evict the active schema until the replacement is known to be valid.

Current code touchpoints

• GraphQLSchemaCreator.cs
• Startup.cs

Logging

Parse and validate candidate log-level configuration during preparation.

Apply it after critical runtime state commits.

Preserve existing production behavior where only supported logging changes are honored.

Authentication Options

Build and validate candidate JWT/authentication options before commit.

Signal the options change token only after the runtime transaction succeeds.

Current code touchpoint

• JwtBearerOptionsChangeTokenSource.cs

MCP Tool Registry

The MCP registry should become a participant once MCP registry hot-reload is implemented.

Preparation should build and validate a complete registry snapshot.

Commit should atomically swap the snapshot.

A failed registry preparation must leave the previous registry active and cause the complete candidate reload to be rejected.

Other Cached Services

Before implementation, inventory every type that:

• subscribes to HotReloadEventHandler;
• subscribes to RuntimeConfigLoadedHandlers;
• subscribes to the runtime change token;
• caches values from RuntimeConfig;
• is constructed conditionally from startup configuration.

For each type, classify it as:

1. transactional participant;
2. post-commit observer;
3. direct dynamic reader requiring no refresh;
4. startup-bound setting requiring restart;
5. unsupported hot-reload behavior that must be documented.

This inventory is an explicit deliverable of the implementation.

---

Dependency Injection Registration

A participant must use the same singleton instance as the live service.

Avoid registering separate instances for the service interface and participant interface.

Use a pattern equivalent to:

services.AddSingleton<QueryManagerFactory>();

services.AddSingleton<IAbstractQueryManagerFactory>(
    provider => provider.GetRequiredService<QueryManagerFactory>());

services.AddSingleton<IHotReloadParticipant>(
    provider => provider.GetRequiredService<QueryManagerFactory>());

Alternatively, register a lightweight participant adapter that targets the existing singleton.

---

Handling Requests During Commit

Preparation may take time, but it does not affect live state.

The commit section should be very short because it consists only of reference swaps.

Optionally add a HotReloadCommitBarrier:

• requests beginning during the commit window wait for commit or rollback to finish;
• requests already in flight may finish using state they already resolved;
• the barrier is released after either successful commit or completed rollback.

Perfect per-request generation isolation is not required for the initial implementation, but no partial state may remain after the transaction finishes.

---

File-Watcher and Reload Queue Behavior

Implement a hosted reload-processing service or equivalent single-reader queue.

Required behavior:

• file-watcher callbacks enqueue work only;
• duplicate notifications are coalesced;
• only one reload transaction runs at a time;
• if another change arrives while processing, rerun against the newest file;
• shutdown cancels queued work;
• reload exceptions never terminate the file-watcher callback thread;
• every attempt receives a generation or correlation ID.

---

Late Configuration

The configuration endpoint uses RuntimeConfigLoadedHandlers, which has similar partial-application behavior.

The first implementation may focus on file hot-reload, but the long-term design should route late configuration through the same coordinator.

The coordinator must support:

• no active configuration, for first-time late configuration;
• an existing active configuration, for replacement;
• the same prepare/commit guarantees in both cases.

Do not maintain two independent application pipelines indefinitely.

---

Existing Event System

HotReloadEventHandler may remain temporarily for compatibility, but critical state mutation must move to transactional participants.

During migration:

• critical events must not run before commit;
• noncritical events may be invoked as post-commit observers;
• event exceptions must be isolated and logged;
• string event names should eventually be removed for transactional components.

Do not mix old mutating event handlers with new participants for the same component.

---

Logging and Telemetry

Log one structured record for each stage:

• candidate detected;
• candidate parsed;
• compatibility validation passed or failed;
• configuration validation passed or failed;
• participant preparation started;
• participant preparation completed;
• participant preparation failed;
• transaction commit started;
• transaction committed;
• rollback started;
• rollback completed;
• post-commit observer failed.

Include:

• reload generation;
• configuration path;
• participant name;
• stage;
• elapsed time;
• previous active generation;
• candidate generation;
• failure exception;
• final transaction result.

Expose the latest reload status to health diagnostics where practical.

---

Failure Semantics

Failure stage	Active config	LKG	Component state	Change token
File read or parse	Previous	Previous	Previous	Not signaled
Compatibility validation	Previous	Previous	Previous	Not signaled
Config validation	Previous	Previous	Previous	Not signaled
Participant preparation	Previous	Previous	Previous	Not signaled
Commit with successful rollback	Previous	Previous	Previous	Not signaled
Commit with failed rollback	Indeterminate; stop app	Previous	Indeterminate	Not signaled
Post-commit observer	Candidate	Candidate	Candidate	Already signaled

---

Testing Plan

Candidate Loading Tests

• Parsing a candidate does not replace active configuration.
• Invalid JSON leaves active configuration and LKG unchanged.
• Environment-variable replacement occurs in the candidate only.
• Production-mode filtering creates an effective candidate without mutating active state.

Coordinator Tests

• Participants prepare in stage order.
• Participants commit in stage order.
• Preparation failure prevents every commit.
• Prepared changes are disposed in reverse order after failure.
• Commit failure rolls back committed participants in reverse order.
• Active config is published only after all participant commits.
• LKG updates only after complete success.
• Change token fires exactly once after success.
• Change token does not fire after any failure.
• Post-commit failure does not roll back committed state.
• Concurrent reload requests are serialized.
• Duplicate file events are coalesced.

Participant Tests

For each participant:

• preparation does not mutate live state;
• successful commit publishes all related fields atomically;
• rollback restores the exact prior snapshot;
• failed preparation disposes candidate resources;
• previous resources are retained until transaction completion.

Integration Tests

1. Start with configuration A.
2. Hot-reload valid configuration B.
3. Verify every runtime component reflects B.
4. Hot-reload malformed configuration C.
5. Verify every runtime component still reflects B.
6. Hot-reload schema-valid configuration D that causes a participant preparation failure.
7. Verify every runtime component and LKG still reflect B.
8. Correct D and verify the next reload succeeds.
9. Trigger rapid consecutive file writes and verify the final file wins.
10. Verify restart-required property changes are rejected without altering active state.

Fault-Injection Tests

Add test participants that fail during:

• preparation;
• first commit;
• middle commit;
• rollback;
• disposal;
• post-commit notification.

Use these tests to prove coordinator state-machine behavior independently of databases.

---

Implementation Order

1. Add characterization tests demonstrating the existing LKG/partial-state problem.
2. Add candidate-loading APIs that do not mutate active state.
3. Move active configuration ownership into RuntimeConfigProvider.
4. Add HotReloadCompatibilityValidator.
5. Add participant and prepared-change contracts.
6. Add the serialized coordinator and reload queue.
7. Convert query-manager state.
8. Convert metadata-provider state.
9. Convert query and mutation engines.
10. Convert authorization state.
11. Convert OpenAPI and GraphQL state.
12. Convert authentication and logging behavior.
13. Move change-token signaling to post-commit.
14. Route file-watcher reloads through the coordinator.
15. Convert or isolate remaining event subscribers.
16. Add MCP registry participation when its hot-reload implementation is available.
17. Route late configuration through the same coordinator.
18. Remove obsolete validation flags and mutating hot-reload event paths.
19. Update the hot-reload design documentation.

Do not switch production hot-reload orchestration to the coordinator until every currently critical event subscriber has either become a participant or been explicitly classified as post-commit/restart-bound.

---

Acceptance Criteria

The work is complete when:

1. A candidate cannot become active or LKG before all critical participants prepare successfully.
2. A participant preparation failure leaves the complete previous runtime operational.
3. An unexpected commit failure restores every already-committed participant.
4. RestoreLkgConfig() is no longer the sole rollback mechanism.
5. The LKG always represents the last fully applied runtime generation.
6. Critical components no longer clear or mutate live state during preparation.
7. Change tokens are emitted only after successful transaction commit.
8. Unsupported startup-bound changes are rejected with a restart-required diagnostic.
9. Overlapping file notifications cannot run overlapping transactions.
10. Integration and fault-injection tests cover parse, validation, preparation, commit, rollback, and post-commit failures.

---

Key Design Principle

Hot-reload must be treated as a state transition:

Current Generation -> Prepared Candidate -> Committed Generation

It must not be treated as a sequence of independent mutation callbacks.

Validation determines whether a candidate is structurally acceptable.

Preparation determines whether the complete runtime can support it.

Commit makes it active.

Only a successfully committed generation is last-known-good.