diff --git a/CHANGELOG.md b/CHANGELOG.md index e52ea65..3bcd8dd 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,29 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). --- +## v26.06.77 (2026-06-08) + +### Docs (feature sync — v26.06.57-76 are now documented) + +A sweep bringing the docs (and the companion plugin skills) in line with everything shipped since +v26.06.57, verified key-by-key against the source: + +- **Scheduling** — distributed-lock providers (`pyfly.scheduling.lock.provider` = none|memory|redis| + **postgres** advisory lock) and the pluggable executor (`pyfly.scheduling.executor.type` = + asyncio|thread). +- **Session** — concurrency registry backends (`pyfly.session.concurrency.registry` = memory|redis| + **postgres**). +- **Security** — persistent OAuth2 token store (`pyfly.security.oauth2.token-store.provider` = + memory|redis|postgres) for multi-instance auth servers. +- **Config** — `Config.reload_from_sources()` + `POST /actuator/refresh` now re-reads config files. +- **Data/CLI** — run-on-startup Alembic migrations (`pyfly.data.relational.migrations.enabled`). +- **Web** — access-log opt-out (`pyfly.web.request-logging.enabled`). +- **Observability** — the `MetricsRecorder` port + `NoOpMetricsRecorder`. +- **DI** — the public container SPI (`register_instance` / `contains_type` / `get_registration` / + `registered_types` / `reset_instance`). +- Fixed the broken `@mongo_transactional(client)` example in the MongoDB adapter doc and the + stale `/actuator/refresh` description (now reflect the unified `@transactional` + config reload). + ## v26.06.76 (2026-06-08) ### Tested (data — MongoDB query/projection execution coverage) diff --git a/README.md b/README.md index 1a961d2..519b40c 100644 --- a/README.md +++ b/README.md @@ -11,7 +11,7 @@ Firefly Framework Python 3.12+ License: Apache 2.0 - Version: 26.06.76 + Version: 26.06.77 Type Checked: mypy strict Code Style: Ruff Async First diff --git a/docs/adapters/mongodb.md b/docs/adapters/mongodb.md index f30cf67..cf75a53 100644 --- a/docs/adapters/mongodb.md +++ b/docs/adapters/mongodb.md @@ -77,20 +77,29 @@ Wires compiled query methods onto `MongoRepository` subclasses at startup — id ### Transactions -Use `@mongo_transactional` for multi-document transactions. The decorator -requires a `Motor` client (injected from the auto-configured bean): +Use the unified **`@transactional`** (from `pyfly.data`) for multi-document transactions — the +same annotation as the relational backend. On a service exposing a Motor client as +`self._motor_client`, it opens a session + transaction and injects it as the `session` keyword +argument (requires a MongoDB replica set): ```python -from pyfly.data.document.mongodb import mongo_transactional -from motor.motor_asyncio import AsyncIOMotorClient +from pyfly.container import service +from pyfly.data import transactional -client: AsyncIOMotorClient = ... # injected by DI -@mongo_transactional(client) -async def transfer(from_id: str, to_id: str, amount: float) -> None: - ... +@service +class AccountService: + def __init__(self, motor_client) -> None: + self._motor_client = motor_client # selects the MongoDB transaction manager + + @transactional() + async def transfer(self, from_id: str, to_id: str, amount: float, *, session=None) -> None: + ... ``` +> `from pyfly.data.document.mongodb import mongo_transactional` still works but is a **deprecated +> alias** of `@transactional`. + --- ## Testing diff --git a/docs/cli.md b/docs/cli.md index 97302f2..6d1afe1 100644 --- a/docs/cli.md +++ b/docs/cli.md @@ -21,6 +21,7 @@ The PyFly CLI provides command-line tools for project scaffolding, application m - [pyfly db migrate](#pyfly-db-migrate) - [pyfly db upgrade](#pyfly-db-upgrade) - [pyfly db downgrade](#pyfly-db-downgrade) + - [Running Migrations on Startup](#running-migrations-on-startup) - [pyfly license](#pyfly-license) - [pyfly sbom](#pyfly-sbom) - [Development Workflow](#typical-development-workflow) @@ -758,6 +759,25 @@ pyfly db downgrade abc123 # Revert to specific revision pyfly db downgrade base # Revert all migrations ``` +### Running Migrations on Startup + +Instead of (or in addition to) running `pyfly db upgrade` by hand, PyFly can apply migrations **automatically on application startup** — the Flyway-style auto-migrate equivalent. This reuses the same Alembic environment created by `pyfly db init`, so the CLI commands above continue to work unchanged. + +It is **opt-in** via configuration, not a CLI flag. Enable it in `pyfly.yaml`: + +```yaml +pyfly: + data: + relational: + url: postgresql+asyncpg://user:pass@localhost:5432/app + migrations: + enabled: true # run `alembic upgrade head` on startup + config: alembic.ini # Alembic config path (default: alembic.ini) + revision: head # target revision (default: head) +``` + +When enabled, the app runs `alembic upgrade ` against the same datasource (`pyfly.data.relational.url`) during startup. If `alembic.ini` is not found, the migration step is skipped with a warning suggesting you run `pyfly db init` — startup is not aborted. See [Data Relational — Run Migrations on Startup](modules/data-relational.md#run-migrations-on-startup-flyway-style) for the full configuration reference. + --- ## pyfly license diff --git a/docs/modules/actuator.md b/docs/modules/actuator.md index dbd0087..c77c58d 100644 --- a/docs/modules/actuator.md +++ b/docs/modules/actuator.md @@ -998,10 +998,11 @@ first, matching `traceback.extract_stack()` order. **Endpoint:** `POST /actuator/refresh` -The refresh endpoint mirrors Spring Cloud's `POST /actuator/refresh`. It triggers a -context refresh — evicting all refresh-scoped beans and resetting -`@config_properties` beans so they re-bind against the live `Config` (which -re-reads environment variables and `${...}` placeholders) on next resolution. +The refresh endpoint mirrors Spring Cloud's `POST /actuator/refresh`. It first calls +`Config.reload_from_sources()` to **re-read the configuration files/profiles** (the exact merge +`from_sources` performed), then triggers a context refresh — evicting all refresh-scoped beans and +resetting `@config_properties` beans so they re-bind against the freshly-reloaded `Config` (which +also re-reads environment variables and `${...}` placeholders) on next resolution. A `RefreshScopeRefreshedEvent` is published, and the response lists the cache keys of the beans that were refreshed: diff --git a/docs/modules/configuration.md b/docs/modules/configuration.md index 9f3bba3..73af60e 100644 --- a/docs/modules/configuration.md +++ b/docs/modules/configuration.md @@ -15,36 +15,38 @@ and the full reference of framework defaults. - [get()](#get) - [get_section()](#get_section) - [bind()](#bind) -3. [YAML Configuration](#yaml-configuration) -4. [TOML Configuration](#toml-configuration) -5. [Profile System](#profile-system) + - [reload_from_sources()](#reload_from_sources) +3. [Runtime Configuration Refresh](#runtime-configuration-refresh) +4. [YAML Configuration](#yaml-configuration) +5. [TOML Configuration](#toml-configuration) +6. [Profile System](#profile-system) - [Activating Profiles](#activating-profiles) - [Profile-Specific Files](#profile-specific-files) - [Profile Expressions in Beans](#profile-expressions-in-beans) -6. [Configuration Layering](#configuration-layering) +7. [Configuration Layering](#configuration-layering) - [Layer 1: Framework Defaults](#layer-1-framework-defaults) - [Layer 2: User Configuration File](#layer-2-user-configuration-file) - [Layer 3: Profile Overlays](#layer-3-profile-overlays) - [Layer 4: Environment Variables](#layer-4-environment-variables) - [Deep Merge Behavior](#deep-merge-behavior) - [Remote Config Import (Config Server)](#remote-config-import-config-server) -7. [Environment Variable Overrides](#environment-variable-overrides) +8. [Environment Variable Overrides](#environment-variable-overrides) - [Naming Convention](#naming-convention) - [Type Coercion](#type-coercion) - [Examples](#environment-variable-examples) -8. [@config_properties](#config_properties) +9. [@config_properties](#config_properties) - [Defining a Config Class](#defining-a-config-class) - [Binding at Runtime](#binding-at-runtime) - [Type Coercion in bind()](#type-coercion-in-bind) -9. [@Value (Field-Level Config Injection)](#value-field-level-config-injection) - - [Expression Syntax](#expression-syntax) - - [Usage in Beans](#usage-in-beans) - - [@Value vs @config_properties](#value-vs-config_properties) -10. [SpEL-lite Expressions](#spel-lite-expressions) +10. [@Value (Field-Level Config Injection)](#value-field-level-config-injection) + - [Expression Syntax](#expression-syntax) + - [Usage in Beans](#usage-in-beans) + - [@Value vs @config_properties](#value-vs-config_properties) +11. [SpEL-lite Expressions](#spel-lite-expressions) - [The `#{ ... }` Form](#the-spel-form) - [`@conditional_on_expression`](#conditional_on_expression) - [Safety Model](#safety-model) -11. [Framework Defaults Reference](#framework-defaults-reference) +12. [Framework Defaults Reference](#framework-defaults-reference) - [Application](#application-defaults) - [Profiles](#profiles-defaults) - [Banner](#banner-defaults) @@ -58,7 +60,7 @@ and the full reference of framework defaults. - [Admin](#admin-defaults) - [Security](#security-defaults) - [Observability](#observability-defaults) -12. [Complete Example: Multi-Environment Setup](#complete-example-multi-environment-setup) +13. [Complete Example: Multi-Environment Setup](#complete-example-multi-environment-setup) --- @@ -216,6 +218,120 @@ print(web.port) # 8080 Raises `ValueError` if the class is not decorated with `@config_properties`. +### reload_from_sources() + +```python +def reload_from_sources(self) -> bool: +``` + +Re-reads the original configuration sources and **atomically swaps in** the freshly +merged result, so a running application picks up edits to the config files and profile +overlays without a restart (Spring Cloud config refresh). It replays the exact merge +recorded by [`from_sources()`](#from_sources) — framework defaults, starter defaults, +`config/` files, project-root files, and profile overlays — under an internal lock, then +rebinds `_data` in a single assignment. Because `get()` reads that single attribute, +concurrent readers always see a consistent snapshot (the old tree or the new one, never a +half-merged mix). + +Returns: + +| Return | Meaning | +|---|---| +| `True` | The sources were re-read and the merged config was swapped in. | +| `False` | The instance was **not** built via `from_sources()` (e.g. a dict-constructed `Config`), so there is nothing to reload — a no-op. | + +```python +config = Config.from_sources(".", active_profiles=["prod"]) +# ... edit pyfly.yaml / pyfly-prod.yaml on disk ... +config.reload_from_sources() # True — files re-read, get() now returns new values +Config({"a": 1}).reload_from_sources() # False — dict-constructed, nothing to reload +``` + +> **Note:** environment-variable overrides and `${...}` placeholders are always resolved at +> *read time* in `get()`, so they reflect the current process environment regardless of +> reloading. `reload_from_sources()` is specifically about re-reading the **files**. + +This method is invoked automatically by `POST /actuator/refresh` — see +[Runtime Configuration Refresh](#runtime-configuration-refresh) below. + +--- + +## Runtime Configuration Refresh + +PyFly supports Spring Cloud-style runtime configuration refresh: you can change config +files (or profile overlays) on disk and have a running application pick up the changes — +no restart — by issuing a single management request. + +```bash +curl -X POST http://localhost:8080/actuator/refresh +# {"refreshed": ["FeatureFlags-singleton", "PricingProperties-singleton"]} +``` + +### What happens on POST /actuator/refresh + +The endpoint resolves the injectable `ContextRefresher` and calls its `refresh()`, which +performs the following steps in order (`src/pyfly/context/refresh.py`): + +1. **Re-reads the config sources.** It calls `Config.reload_from_sources()`, which replays + the original multi-source merge and atomically swaps in the new tree. This is the step + that lets edits to `pyfly.yaml` / `pyfly-{profile}.yaml` take effect at runtime. (For a + dict-constructed `Config` this is a no-op.) +2. **Evicts all refresh-scoped beans** (`@refresh_scope`). Each cached instance is dropped + so the next resolution rebuilds it — re-running constructor/field injection and + re-reading `@Value` placeholders against the now-refreshed `Config`. +3. **Resets `@config_properties` singletons.** Their backing instances are cleared so they + re-`bind()` from the live `Config` (which now reflects the re-read files, env-var + overrides, and resolved `${...}` placeholders) on next resolution. +4. **Publishes a `RefreshScopeRefreshedEvent`** on the application event bus. + +The response is `{"refreshed": [...]}`, listing the cache keys of the evicted +refresh-scoped beans (an empty list when none are registered). + +### Picking up file changes in your beans + +Because step 1 now re-reads the files before steps 2–3 rebuild the affected beans, both +`@refresh_scope` and `@config_properties` beans see the **new file values** after a +refresh: + +```python +from dataclasses import dataclass +from pyfly.container import component +from pyfly.core import config_properties +from pyfly.container.refresh_scope import refresh_scope +from pyfly.core.value import Value + + +@config_properties(prefix="pyfly.pricing") +@dataclass +class PricingProperties: + base_rate: float = 1.0 # edit pyfly.yaml + POST /actuator/refresh -> re-bound + + +@component +@refresh_scope +class FeatureFlags: + new_checkout: bool = Value("${features.new-checkout:false}") + # next resolution after refresh re-reads ${features.new-checkout} from the live Config +``` + +### Exposure (opt-in) + +Like Spring Boot, the actuator is secure-by-default: only `health` and `info` are +reachable over HTTP. The `refresh` endpoint is registered whenever the actuator is enabled +with a context, but it is **not mounted** until you add it to the exposure include list: + +```yaml +pyfly: + management: + endpoints: + web: + exposure: + include: "health,info,refresh" # or "*" to expose every enabled endpoint +``` + +With the default `health,info`, `POST /actuator/refresh` returns `404`. See the +[Actuator guide](actuator.md#refresh-endpoint) for the full endpoint reference. + --- ## YAML Configuration diff --git a/docs/modules/core.md b/docs/modules/core.md index 3aa38d4..f01c229 100644 --- a/docs/modules/core.md +++ b/docs/modules/core.md @@ -23,6 +23,7 @@ depends on them. - [Reading Values: get()](#reading-values-get) - [Reading Sections: get_section()](#reading-sections-get_section) - [Typed Binding: bind()](#typed-binding-bind) + - [Runtime Reload: reload_from_sources()](#runtime-reload-reload_from_sources) 5. [@config_properties Decorator](#config_properties-decorator) 6. [Configuration Layering](#configuration-layering) - [Framework Defaults](#framework-defaults) @@ -373,6 +374,41 @@ It reads the prefix from the decorator, fetches the effective config section (wi `${...}` placeholders resolved, env overrides applied, and env-only keys injected), and constructs the dataclass with relaxed key matching and type coercion. +### Runtime Reload: reload_from_sources() + +```python +def reload_from_sources(self) -> bool: +``` + +Re-reads the original configuration sources and **atomically swaps in** the freshly merged +result, so a running application can pick up edits to its config files / profile overlays +without a restart (Spring Cloud config refresh). It replays the exact merge recorded by +`from_sources()` (framework defaults, starter defaults, `config/` files, project-root +files, and profile overlays) under an internal lock, then rebinds the backing data in a +single assignment — `get()` reads that one attribute, so concurrent readers always see a +consistent snapshot. + +Returns `True` when the sources were re-read and swapped in; returns `False` for instances +not built via `from_sources()` (e.g. a dict-constructed `Config`), where there is nothing +to reload (a no-op). + +```python +config = Config.from_sources(".", active_profiles=["prod"]) +# ... edit pyfly.yaml / pyfly-prod.yaml on disk ... +config.reload_from_sources() # True — get() now returns the new file values +``` + +Environment-variable overrides and `${...}` placeholders are resolved at read time in +`get()`, so they already track the live process environment; `reload_from_sources()` is +specifically about re-reading the **files**. + +This is what `POST /actuator/refresh` calls under the hood: the `ContextRefresher` +(`pyfly.context.refresh`) reloads config from sources *before* it evicts refresh-scoped +beans and resets `@config_properties` singletons, so those beans re-bind against the +re-read files on next resolution. See the +[Configuration guide](configuration.md#runtime-configuration-refresh) and the +[Actuator guide](actuator.md#refresh-endpoint). + --- ## @config_properties Decorator diff --git a/docs/modules/data-relational.md b/docs/modules/data-relational.md index 77a02ca..61ef9b0 100644 --- a/docs/modules/data-relational.md +++ b/docs/modules/data-relational.md @@ -41,6 +41,7 @@ PyFly Data Relational implements the Repository pattern with Spring Data-style d - [Paginated Queries](#paginated-queries) - [Paginated Specification Queries](#paginated-specification-queries) - [Transaction Management](#transaction-management) +- [Run Migrations on Startup (Flyway-Style)](#run-migrations-on-startup-flyway-style) - [Read/Write Routing (Read Replicas)](#readwrite-routing-read-replicas) - [Multiple Named Datasources](#multiple-named-datasources) - [NamedDataSources](#nameddatasources) @@ -567,6 +568,46 @@ await transfer_funds("acc-1", "acc-2", 100.0) --- +## Run Migrations on Startup (Flyway-Style) + +By default schema migrations are applied with the [`pyfly db`](../cli.md#pyfly-db) CLI commands. PyFly can also apply them **automatically on application startup** — the equivalent of Spring Boot's Flyway/Liquibase auto-migrate. This is **opt-in** and reuses the existing Alembic environment created by `pyfly db init`; the CLI commands keep working exactly as before. + +Enable it with `pyfly.data.relational.migrations.enabled`: + +```yaml +pyfly: + data: + relational: + url: postgresql+asyncpg://user:pass@primary:5432/app + migrations: + enabled: true # apply `alembic upgrade head` on startup + config: alembic.ini # path to the Alembic config (default: alembic.ini) + revision: head # target revision (default: head) +``` + +When enabled, `MigrationAutoConfiguration` registers a `MigrationRunner` bean. `MigrationRunner` implements the `start()` / `stop()` lifecycle, so the `ApplicationContext` auto-discovers it as an infrastructure adapter and calls `start()` once during startup. On `start()` it runs `alembic upgrade ` against the **same datasource** the app uses (it forwards `pyfly.data.relational.url` into Alembic's `sqlalchemy.url`, so there is a single source of truth for the connection string). + +The upgrade runs in a worker thread (`asyncio.to_thread`) because the generated async `alembic/env.py` calls `asyncio.run` internally, which must not be nested inside the running event loop. + +If the Alembic config file is not found, startup migration is **skipped with a warning** (rather than failing) telling you to run `pyfly db init` first: + +``` +pyfly.data.relational.migrations.enabled is true but alembic.ini was not found — +run 'pyfly db init' to create the Alembic environment; skipping migrations. +``` + +| Config key | Default | Description | +|------------|---------|-------------| +| `pyfly.data.relational.migrations.enabled` | `false` (absent) | Apply migrations on startup when `true`. | +| `pyfly.data.relational.migrations.config` | `alembic.ini` | Path to the Alembic config file. | +| `pyfly.data.relational.migrations.revision` | `head` | Target revision passed to `alembic upgrade`. | + +> **Migrations vs. `ddl-auto`:** startup migrations are independent of the `engine_lifecycle` `ddl-auto` schema strategy. For an Alembic-managed database, set `pyfly.data.relational.ddl-auto: none` so the engine does not also create tables from `Base.metadata`, and let migrations own the schema. + +**Source:** `src/pyfly/data/relational/migrations.py` (`MigrationRunner`) · `src/pyfly/data/relational/auto_configuration.py` (`MigrationAutoConfiguration`) + +--- + ## Read/Write Routing (Read Replicas) PyFly can route read-only work to a database **read replica** while keeping writes on the primary — the equivalent of Spring's `AbstractRoutingDataSource` driven by `@Transactional(readOnly = true)`. Routing is **opt-in**: with no replica configured, every session goes to the primary, so behavior is unchanged for existing apps. diff --git a/docs/modules/dependency-injection.md b/docs/modules/dependency-injection.md index c4f122e..b24c6b9 100644 --- a/docs/modules/dependency-injection.md +++ b/docs/modules/dependency-injection.md @@ -17,6 +17,7 @@ a multi-layer application where every component is managed, wired, and lifecycle - [resolve_by_name()](#resolve_by_name) - [resolve_all()](#resolve_all) - [contains()](#contains) + - [Introspection / registration SPI](#introspection--registration-spi) 3. [Stereotypes](#stereotypes) - [@component](#component) - [@service](#service) @@ -267,6 +268,49 @@ if container.contains("cache_adapter"): cache = container.resolve_by_name("cache_adapter") ``` +### Introspection / registration SPI + +`Container` exposes a small **public SPI** for installing pre-built instances and +inspecting registrations, so callers (refresh/config-reload, tests, tooling) never have +to reach into the private `_registrations` dict. All five methods live on `Container` +(from `pyfly.container`): + +```python +def register_instance(self, cls: type, instance: Any, *, name: str = "") -> None: +def contains_type(self, cls: type) -> bool: +def get_registration(self, cls: type) -> Registration | None: +def registered_types(self) -> list[type]: +def reset_instance(self, cls: type) -> Any | None: +``` + +| Method | Behaviour | +|---|---| +| `register_instance(cls, instance, *, name="")` | Register an already-constructed object as a `SINGLETON` bean (Spring's `registerSingleton`). The supported way to install a pre-built instance — preferred over mutating registration internals. | +| `contains_type(cls)` | `True` if a bean is registered under **exactly** `cls`. (Compare with `contains(name)`, which checks the named-bean store.) | +| `get_registration(cls)` | The `Registration` for `cls`, or `None` if unregistered. | +| `registered_types()` | A snapshot `list[type]` of every registered bean type. | +| `reset_instance(cls)` | Drop the cached `SINGLETON` instance of `cls` so it is rebuilt on the next `resolve()`. Returns the evicted instance (or `None`). Used by refresh / config-reload to force re-creation without reaching into registration internals. | + +```python +from pyfly.container import Container + +container = Container() + +# Install a pre-built singleton (optionally named): +container.register_instance(Clock, SystemClock(), name="system_clock") + +# Inspect registrations: +container.contains_type(Clock) # True +reg = container.get_registration(Clock) # Registration | None +all_types = container.registered_types() # [Clock, ...] + +# Force re-creation on next resolve (refresh / config reload): +previous = container.reset_instance(Clock) # returns the evicted instance or None +fresh = container.resolve(Clock) # rebuilt +``` + +**Source:** `src/pyfly/container/container.py` + --- ## Stereotypes diff --git a/docs/modules/observability.md b/docs/modules/observability.md index 54ac7c0..670181b 100644 --- a/docs/modules/observability.md +++ b/docs/modules/observability.md @@ -10,6 +10,7 @@ logging -- along with a health check system for readiness and liveness probes. 1. [Introduction](#introduction) 2. [Metrics](#metrics) + - [MetricsRecorder Port and NoOpMetricsRecorder](#metricsrecorder-port-and-noopmetricsrecorder) - [MetricsRegistry](#metricsregistry) - [Counter Metrics](#counter-metrics) - [Histogram Metrics](#histogram-metrics) @@ -55,6 +56,7 @@ The core observability utilities come from two packages: ```python # Metrics and tracing from pyfly.observability import ( + MetricsRecorder, NoOpMetricsRecorder, # Metrics port + dependency-free adapter MetricsRegistry, timed, counted, # Metrics (requires pyfly[observability]) span, # Tracing (requires opentelemetry) ) @@ -72,6 +74,53 @@ from pyfly.actuator import ( ## Metrics +### MetricsRecorder Port and NoOpMetricsRecorder + +`MetricsRecorder` is the **port** that framework and application instrumentation depend +on, so metric-emitting code is not hard-coupled to Prometheus. It is a +`@runtime_checkable` `Protocol` with three factory methods that each return a backend +metric handle: + +```python +from typing import Any, Protocol, runtime_checkable + +@runtime_checkable +class MetricsRecorder(Protocol): + def counter(self, name: str, description: str, labels: list[str] | None = None) -> Any: ... + def histogram( + self, + name: str, + description: str, + labels: list[str] | None = None, + buckets: tuple[float, ...] | None = None, + ) -> Any: ... + def gauge(self, name: str, description: str, labels: list[str] | None = None) -> Any: ... +``` + +`MetricsRegistry` (below) is the default Prometheus-backed adapter for this port. +`NoOpMetricsRecorder` is a **dependency-free** adapter for tests and for deployments +that disable metrics — every method returns a shared no-op metric handle, so +instrumentation code can always hold a recorder instead of guarding `None`: + +```python +from pyfly.observability import MetricsRecorder, NoOpMetricsRecorder + +recorder: MetricsRecorder = NoOpMetricsRecorder() + +# Each factory returns a chainable no-op handle that accepts every Prometheus-style +# operation (.labels(...).inc(), .observe(...), .set(...), and the .time()/async +# context-manager forms) and does nothing. +recorder.counter("orders_total", "Total orders").labels(status="created").inc() +recorder.histogram("latency_seconds", "Latency").observe(0.042) +``` + +Both `MetricsRecorder` and `NoOpMetricsRecorder` are exported from +`pyfly.observability` (and live in `pyfly.observability.ports`). Because +`NoOpMetricsRecorder` has no `prometheus_client` dependency, it never raises +`ImportError`, unlike constructing a `MetricsRegistry`. + +**Source:** `src/pyfly/observability/ports.py` + ### MetricsRegistry `MetricsRegistry` is a thin wrapper around the `prometheus_client` library. It diff --git a/docs/modules/scheduling.md b/docs/modules/scheduling.md index 61fcad5..5d7d9ad 100644 --- a/docs/modules/scheduling.md +++ b/docs/modules/scheduling.md @@ -33,8 +33,13 @@ PyFly scheduling module. 6. [AsyncIOTaskExecutor](#asynciotaskexecutor) 7. [ThreadPoolTaskExecutor](#threadpooltaskexecutor) 8. [Distributed Locking with DistributedLock](#distributed-locking-with-distributedlock) + - [Built-in Lock Providers](#built-in-lock-providers) + - [Cross-Process Coordination (custom)](#cross-process-coordination-custom) + - [Registering a DistributedLock Bean](#registering-a-distributedlock-bean) 9. [The @async_method Decorator](#the-async_method-decorator) 10. [Configuration](#configuration) + - [Selecting the Executor](#selecting-the-executor) + - [Selecting the Lock Provider](#selecting-the-lock-provider) 11. [Auto-Configuration](#auto-configuration) 12. [Complete Example](#complete-example) @@ -56,9 +61,12 @@ The module is built around a hexagonal architecture: - **TaskScheduler** is the engine that discovers decorated methods, creates execution loops, and manages their lifecycle. - **TaskExecutorPort** is the outbound port abstraction, allowing you to swap - execution strategies. -- **AsyncIOTaskExecutor** and **ThreadPoolTaskExecutor** are the built-in - adapters. + execution strategies; **AsyncIOTaskExecutor** and **ThreadPoolTaskExecutor** + are the built-in adapters, selectable via `pyfly.scheduling.executor.type`. +- **DistributedLock** coordinates `@scheduled(lock=...)` jobs across instances; + **LocalLock**, **InProcessDistributedLock**, **RedisDistributedLock**, and + **PostgresAdvisoryLock** are the built-in providers, selectable via + `pyfly.scheduling.lock.provider`. All public types are available from a single import: @@ -71,9 +79,13 @@ from pyfly.scheduling import ( TaskExecutorPort, DistributedLock, LocalLock, + InProcessDistributedLock, ) from pyfly.scheduling.adapters.asyncio_executor import AsyncIOTaskExecutor from pyfly.scheduling.adapters.thread_executor import ThreadPoolTaskExecutor +# Built-in cluster-coordination lock adapters (normally selected via config): +from pyfly.scheduling.adapters.redis_lock import RedisDistributedLock +from pyfly.scheduling.adapters.postgres_lock import PostgresAdvisoryLock ``` --- @@ -212,8 +224,10 @@ class ImportService: `lock` works with all three trigger types (`cron`, `fixed_rate`, `fixed_delay`). Out of the box the scheduler uses an in-process `LocalLock` -that always acquires — so single-instance behavior is unchanged. For true -cross-process coordination you must register a `DistributedLock` bean; see +that always acquires — so single-instance behavior is unchanged. For real +coordination, select a built-in lock provider with +`pyfly.scheduling.lock.provider` (`memory`, `redis`, or `postgres`) — no custom +code required — or register your own `DistributedLock` bean. See [Distributed Locking with DistributedLock](#distributed-locking-with-distributedlock). ### Decorator Metadata @@ -568,11 +582,65 @@ from pyfly.scheduling import TaskScheduler, LocalLock scheduler = TaskScheduler(lock=LocalLock()) # default behavior ``` -### Cross-Process Coordination +### Built-in Lock Providers -To actually serialize a job across instances, implement `DistributedLock` -against a shared store (Redis, a database row, etc.) and pass it to the -scheduler. Any object with conforming `try_acquire` / `release` coroutines +You do **not** need to write a lock to coordinate across instances. The +`SchedulingAutoConfiguration.distributed_lock` bean selects a provider from +`pyfly.scheduling.lock.provider`, and the auto-wired `TaskScheduler` uses it for +every `@scheduled(lock=...)` job: + +| `pyfly.scheduling.lock.provider` | Implementation | Scope | Extra infra | +|---|---|---|---| +| `none` *(default)* | `LocalLock` | single instance (always acquires) | none | +| `memory` | `InProcessDistributedLock` | one process (real mutual exclusion within the process) | none | +| `redis` | `RedisDistributedLock` | cross-process / cluster | Redis | +| `postgres` | `PostgresAdvisoryLock` | cross-process / cluster | none beyond an existing Postgres | + +```yaml +# pyfly.yaml +pyfly: + scheduling: + lock: + provider: postgres # none | memory | redis | postgres +``` + +- **`none`** — `LocalLock`; `try_acquire` always returns `True`. Single-instance + default; `lock=` declarations are effectively no-ops. +- **`memory`** — `InProcessDistributedLock`; real mutual exclusion **within one + process** (with a TTL self-heal so a crashed/never-released name auto-frees + after `lock_ttl`). Prevents a slow tick from overlapping its next tick in the + same process, but does **not** coordinate across processes. +- **`redis`** — `RedisDistributedLock`; cross-process via an atomic Redis + `SET key value NX PX `, with an owner-token compare-and-delete release + (an instance only releases a lock it still owns). The async Redis client is + built by the auto-config from `pyfly.scheduling.lock.redis.url` (default + `redis://localhost:6379/0`) and injected — the adapter never imports `redis` + itself. Selected only when `redis.asyncio` is importable; otherwise the bean + falls back to `LocalLock`. Keys are prefixed `pyfly:schedlock:`. +- **`postgres`** — `PostgresAdvisoryLock`; cross-process via Postgres + **session-level advisory locks** (`pg_try_advisory_lock` / + `pg_advisory_unlock`). For apps already on Postgres this gives cluster-safe + coordination with **no extra infrastructure**. The lock name is mapped to a + stable signed 64-bit key (blake2b, deterministic across processes). The + `AsyncEngine` is resolved lazily from the container on first acquire (so + bean-ordering does not matter). Note there is **no TTL** for this provider: + the advisory lock lives with the holding connection and is auto-released when + the connection closes — including when the process dies, which is the + crash-safety mechanism in lieu of `lock_ttl`. + +**When to use which:** + +- Single instance, no cluster → leave the default `none` (or `memory` if you + want to prevent in-process overlap of a slow job). +- Multiple instances and you already run Redis → `redis`. +- Multiple instances and you already run Postgres (but no Redis) → `postgres`, + to avoid standing up new infrastructure just for scheduling. + +### Cross-Process Coordination (custom) + +If none of the built-in providers fit, implement `DistributedLock` against any +shared store and pass it to the scheduler (or register it as a bean — see +below). Any object with conforming `try_acquire` / `release` coroutines satisfies the protocol: ```python @@ -599,10 +667,12 @@ scheduler = TaskScheduler(lock=RedisLock(redis_client)) ### Registering a DistributedLock Bean -With auto-configuration, the auto-wired `TaskScheduler` automatically looks up a -`DistributedLock` bean from the container and uses it for `@scheduled(lock=...)` -coordination. If none is registered, it falls back to `LocalLock`. Just declare -your implementation as a bean of type `DistributedLock`: +The built-in providers above (`pyfly.scheduling.lock.provider`) are themselves +registered as the `distributed_lock` bean, so in most cases a YAML setting is +all you need. If you want a fully custom lock, declare your own bean of type +`DistributedLock`; the auto-wired `TaskScheduler` looks it up from the container +and uses it for `@scheduled(lock=...)` coordination (falling back to `LocalLock` +if none is registered): ```python from pyfly.container.bean import bean @@ -649,26 +719,75 @@ The framework picks this up and routes the call through the configured ## Configuration -Scheduling behavior can be configured in `pyfly.yaml`: +Scheduling behavior is configured in `pyfly.yaml`. The auto-configured +`task_scheduler` and `distributed_lock` beans read these keys to pick the +executor and lock backend: ```yaml pyfly: scheduling: enabled: true - thread-pool: - max-workers: 4 + executor: + type: asyncio # asyncio | thread + max-workers: 4 # thread-pool size when type=thread + lock: + provider: none # none | memory | redis | postgres + redis: + url: redis://localhost:6379/0 # used when provider=redis ``` | Key | Description | Default | |---|---|---| -| `pyfly.scheduling.enabled` | Enable or disable the scheduling subsystem | `true` | -| `pyfly.scheduling.thread-pool.max-workers` | Max threads for `ThreadPoolTaskExecutor` | `4` | - -When `enabled` is `false`, the `TaskScheduler` will not start any loops and -`@scheduled` methods will be ignored. +| `pyfly.scheduling.enabled` | Convention flag set by the `application`/`data` starters (see [Auto-Configuration](#auto-configuration)) | `true` | +| `pyfly.scheduling.executor.type` | Executor backend: `asyncio` (in-loop) or `thread` (`ThreadPoolTaskExecutor`) | `asyncio` | +| `pyfly.scheduling.executor.max-workers` | Thread-pool size when `executor.type=thread` | `4` | +| `pyfly.scheduling.lock.provider` | Distributed-lock backend: `none` / `memory` / `redis` / `postgres` | `none` | +| `pyfly.scheduling.lock.redis.url` | Redis URL when `lock.provider=redis` | `redis://localhost:6379/0` | **Requires:** `uv add "pyfly[scheduling]"` (installs `croniter` for cron -expression parsing) +expression parsing). The `redis` lock provider additionally needs +`redis.asyncio` importable; the `postgres` provider needs a SQLAlchemy +`AsyncEngine` bean. + +### Selecting the Executor + +The scheduler submits each run through a `TaskExecutorPort`. The auto-config +chooses the adapter from `pyfly.scheduling.executor.type`: + +- `asyncio` *(default)* — `AsyncIOTaskExecutor`. Ideal for I/O-bound + `async`/`await` tasks; runs work on the event loop. +- `thread` — `ThreadPoolTaskExecutor(max_workers=pyfly.scheduling.executor.max-workers)`. + Offloads blocking/CPU-bound jobs to a worker-thread pool. + +```yaml +pyfly: + scheduling: + executor: + type: thread + max-workers: 8 +``` + +This is equivalent to constructing +`TaskScheduler(executor=ThreadPoolTaskExecutor(max_workers=8))` yourself (see +[ThreadPoolTaskExecutor](#threadpooltaskexecutor)). To swap in a fully custom +executor, override the `task_scheduler` bean (see +[Overriding the Auto-Configured Scheduler](#overriding-the-auto-configured-scheduler)). + +### Selecting the Lock Provider + +`pyfly.scheduling.lock.provider` chooses the `distributed_lock` bean used for +`@scheduled(lock=...)` coordination. See +[Built-in Lock Providers](#built-in-lock-providers) for the full matrix and +guidance on `none` / `memory` / `redis` / `postgres`. + +```yaml +pyfly: + scheduling: + lock: + provider: redis + redis: + url: redis://cache:6379/0 +``` --- @@ -682,7 +801,8 @@ When `croniter` is installed, PyFly automatically registers a `TaskScheduler` be | Bean | Type | Description | |------|------|-------------| -| `task_scheduler` | `TaskScheduler` | Container-managed scheduler that discovers and runs `@scheduled` methods | +| `distributed_lock` | `DistributedLock` | Lock backend for `@scheduled(lock=...)`, selected by `pyfly.scheduling.lock.provider` (`none`/`memory`/`redis`/`postgres`) | +| `task_scheduler` | `TaskScheduler` | Container-managed scheduler that discovers and runs `@scheduled` methods; uses the executor from `pyfly.scheduling.executor.type` and resolves the `distributed_lock` bean | With auto-configuration, you no longer need a `SchedulerManager` service. The `ApplicationContext` automatically: @@ -725,7 +845,11 @@ The `TaskScheduler` is auto-wired as a container bean and the `ApplicationContex ### Overriding the Auto-Configured Scheduler -Provide your own `TaskScheduler` bean to override the auto-configured one: +For the common cases — switching the executor to a thread pool, or picking a +lock provider — you do **not** need a custom bean; just set +`pyfly.scheduling.executor.type` / `pyfly.scheduling.lock.provider` in +`pyfly.yaml` (see [Configuration](#configuration)). Provide your own +`TaskScheduler` bean only when you need a fully custom executor or lock: ```python from pyfly.container.bean import bean diff --git a/docs/modules/security.md b/docs/modules/security.md index 3f21f15..2b2834c 100644 --- a/docs/modules/security.md +++ b/docs/modules/security.md @@ -1208,7 +1208,44 @@ class TokenStore(Protocol): async def revoke(self, token_id: str) -> None: ... ``` -`InMemoryTokenStore` is the built-in adapter for development and testing. In production, implement `TokenStore` with Redis or a database backend. +`InMemoryTokenStore` is the built-in adapter for development and testing. PyFly also ships persistent Redis and Postgres adapters for production use (see below). + +#### Persistent Token Stores (multi-instance authorization server) + +`InMemoryTokenStore` keeps refresh tokens in a process-local dict — it is fine for a single-instance dev/test server, but it **loses all tokens on restart** and is not shared across instances (a refresh issued on one node is unknown to another, and revocation does not propagate). To run the authorization server across multiple instances, select a persistent token store via configuration. `OAuth2AuthorizationServerAutoConfiguration._build_token_store()` reads `pyfly.security.oauth2.token-store.provider` (case-insensitive) and wires the matching adapter: + +| Provider | Adapter | Persistence | When to use | +|---|---|---|---| +| `memory` (default) | `InMemoryTokenStore` (`pyfly.security.oauth2.authorization_server`) | Process-local; **lost on restart**, not shared across instances | Development and testing only — single instance | +| `redis` | `RedisTokenStore` (`pyfly.security.adapters.redis_token_store`) | Cross-instance, fast distributed revocation; tokens self-evict at the refresh-token TTL | Multi-instance servers wanting fast revocation | +| `postgres` | `PostgresTokenStore` (`pyfly.security.adapters.postgres_token_store`) | Durable + auditable in a SQL table, no Redis required | Multi-instance servers needing durable, auditable storage | + +Selecting `redis` or `postgres` fixes multi-instance authorization servers: refresh tokens and revocations are shared across all nodes, so a token issued or revoked on one instance is honoured everywhere, and tokens survive a restart (postgres) or persist until their TTL (redis). + +```yaml +pyfly: + security: + oauth2: + authorization-server: + enabled: true + secret: "${OAUTH2_SECRET}" + refresh-token-ttl: 86400 # also the Redis token TTL + token-store: + provider: redis # memory (default) | redis | postgres + redis: + url: "redis://localhost:6379/0" # falls back to pyfly.session.redis.url +``` + +**Configuration keys:** + +| Key | Default | Description | +|---|---|---| +| `pyfly.security.oauth2.token-store.provider` | `memory` | `memory`, `redis`, or `postgres` (matched case-insensitively) | +| `pyfly.security.oauth2.token-store.redis.url` | falls back to `pyfly.session.redis.url`, then `redis://localhost:6379/0` | Redis connection URL (redis provider only) | + +**Redis adapter.** When `provider=redis` and the `redis.asyncio` driver is available, the composition root builds an async client with `redis.asyncio.from_url(url)` and injects it into `RedisTokenStore(client, ttl=refresh_ttl)`. Tokens are stored as JSON under the `pyfly:oauth2:token:` key prefix with an `ex` equal to the refresh-token TTL, so expired tokens self-evict. If the driver is unavailable the store falls back to `InMemoryTokenStore`. + +**Postgres adapter.** When `provider=postgres`, the composition root resolves a SQLAlchemy `AsyncEngine` from the container (so an `AsyncEngine` bean must be present) and injects an engine factory into `PostgresTokenStore`. The adapter lazily and idempotently creates a `pyfly_oauth2_tokens` table (`token_id TEXT PRIMARY KEY, data TEXT NOT NULL`) on first use and upserts refresh tokens with `ON CONFLICT (token_id) DO UPDATE`. Both adapters are hexagonal: they never import their driver at module scope — the client/engine is injected by the auto-configuration. #### Error Codes diff --git a/docs/modules/session.md b/docs/modules/session.md index 8c9b192..f756e27 100644 --- a/docs/modules/session.md +++ b/docs/modules/session.md @@ -228,11 +228,61 @@ pyfly: pending session and responds with HTTP `401` and body `{"error": "max_sessions", ...}`. +### Registry Backends + +The cap is enforced against a `SessionRegistry` — a per-principal index of live +session ids, kept separate from the `SessionStore`. Three backends ship out of +the box, selected by `pyfly.session.concurrency.registry`: + +| `registry` | Implementation | Scope | Requirements | +|---|---|---|---| +| `memory` (default) | `InMemorySessionRegistry` | Single process only | none | +| `redis` | `RedisSessionRegistry` | Cross-process / multi-instance | `redis.asyncio` installed | +| `postgres` | `PostgresSessionRegistry` | Cross-process / durable | SQLAlchemy `AsyncEngine` bean | + +- **`memory`** — in-process index guarded by an `asyncio.Lock` (mirrors + `InMemorySessionStore`). Each app instance counts only its own sessions, so + the cap is **not** enforced across multiple processes. Suitable for + single-node deployments, development, and testing. State is lost on restart. +- **`redis`** — a cross-process index shared by all app instances. Each + principal's live sessions are stored in a Redis sorted set (score = + `created_at`, member = `session_id`), so `list_sessions` is naturally + oldest-first. Requires `redis.asyncio`; if it is unavailable the + auto-configuration falls back to the in-memory registry. The connection URL + comes from `pyfly.session.concurrency.redis.url`, falling back to + `pyfly.session.redis.url`, then `redis://localhost:6379/0`. +- **`postgres`** — a durable, queryable, cross-process index for + relational-only deployments (no Redis required). Session ids are stored in a + Postgres table (`session_id` PK, `principal`, `created_at`), created lazily + and idempotently on first use. Resolves a SQLAlchemy `AsyncEngine` bean from + the container; this requires the data module / an `AsyncEngine` to be + configured. + +### Configuration (Registry Backend) + +```yaml +pyfly: + session: + concurrency: + enabled: true + max-sessions: 1 + strategy: evict-oldest + registry: redis # memory (default) | redis | postgres + redis: + url: redis://localhost:6379/0 # optional; falls back to pyfly.session.redis.url +``` + +| Key | Default | Description | +|-----|---------|-------------| +| `pyfly.session.concurrency.registry` | `memory` | Registry backend: `memory`, `redis`, or `postgres` (case-insensitive) | +| `pyfly.session.concurrency.redis.url` | falls back to `pyfly.session.redis.url`, then `redis://localhost:6379/0` | Redis connection URL (used when `registry=redis`) | + ### Auto-Configuration When `pyfly.session.concurrency.enabled=true`, `SessionConcurrencyAutoConfiguration` registers a -`SessionConcurrencyController` bean backed by an `InMemorySessionRegistry`. +`SessionConcurrencyController` bean backed by the registry selected via +`pyfly.session.concurrency.registry` (`InMemorySessionRegistry` by default). The OAuth2 login auto-configuration resolves this bean (if present) and passes it to `OAuth2LoginHandler`, so no manual wiring is required: @@ -240,6 +290,10 @@ it to `OAuth2LoginHandler`, so no manual wiring is required: |---|---|---| | `SessionConcurrencyAutoConfiguration` | `session_concurrency_controller` | `pyfly.session.concurrency.enabled=true` | +The Redis client and SQLAlchemy `AsyncEngine` are obtained in the +auto-configuration (the composition root) and injected into the adapters — the +adapters never import their driver at module scope (hexagonal wiring). + The controller's `session_deleter` is wired to `SessionStore.delete`, so an evicted session is purged from whichever store backend is active (memory or Redis). @@ -267,9 +321,12 @@ policy = ConcurrencyControlPolicy(max_sessions=1, strategy="reject-new") | `strategy` | `"evict-oldest"` | `"evict-oldest"` or `"reject-new"` | `SessionRegistry` is a `runtime_checkable` Protocol — a per-principal index of -live session ids, kept separate from the `SessionStore`: +live session ids, kept separate from the `SessionStore`. It is also exported +from `pyfly.session.ports`: ```python +from pyfly.session.ports import SessionRegistry + class SessionRegistry(Protocol): async def register(self, principal: str, session_id: str, created_at: float) -> None: ... async def deregister(self, principal: str, session_id: str) -> None: ... @@ -278,8 +335,27 @@ class SessionRegistry(Protocol): ``` `InMemorySessionRegistry` is the in-process implementation (guarded by an -`asyncio.Lock`), the default used by auto-configuration. Provide your own -`SessionRegistry` implementation for a distributed registry. +`asyncio.Lock`), the default used by auto-configuration when +`registry=memory`. Two cross-process implementations ship as adapters; both +have their driver/engine injected by the composition root: + +```python +from pyfly.session.adapters.redis_registry import RedisSessionRegistry +from pyfly.session.adapters.postgres_registry import PostgresSessionRegistry +``` + +`RedisSessionRegistry(client, *, key_prefix="pyfly:session:user:", ttl=86400)` +stores each principal's sessions in a Redis sorted set (oldest-first by +`created_at`). The `ttl` (seconds) bounds orphan growth and slides forward on +each `register`. Used when `registry=redis`. + +`PostgresSessionRegistry(engine_factory, *, table="pyfly_session_registry")` +stores sessions in a Postgres table. `engine_factory` is a zero-arg callable +returning a SQLAlchemy `AsyncEngine` (resolved lazily on first use); the table +name is validated as a SQL identifier. Used when `registry=postgres`. + +You may still provide your own `SessionRegistry` bean to override the +auto-configured one entirely. `SessionConcurrencyController` enforces the policy: diff --git a/docs/modules/web-filters.md b/docs/modules/web-filters.md index 2c43ce1..559f7eb 100644 --- a/docs/modules/web-filters.md +++ b/docs/modules/web-filters.md @@ -172,7 +172,22 @@ class RequestLoggingFilter(OncePerRequestFilter): Logged fields: `method`, `path`, `status_code`, `duration_ms`, `transaction_id`. Failed requests are logged at `error` level with `error` and `error_type` fields. +**Disabling per-request access logging.** The `RequestLoggingFilter` is on by default, +but it is the costliest filter (the `structlog` emit runs on every request). To shave +that per-request footprint, set `pyfly.web.request-logging.enabled` to `false`; the +filter is then omitted from the chain entirely. The default is `true`, and the toggle +is read by `create_app()` (Starlette **and** FastAPI adapters) when an +`ApplicationContext` is supplied. + +```yaml +pyfly: + web: + request-logging: + enabled: false # default: true — omits RequestLoggingFilter from the chain +``` + **Source:** `src/pyfly/web/adapters/starlette/filters/request_logging_filter.py` +(toggle wired in `src/pyfly/web/adapters/starlette/app.py` and `.../fastapi/app.py`) ### SecurityHeadersFilter diff --git a/pyproject.toml b/pyproject.toml index dc4ee62..23b3b2b 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -7,7 +7,7 @@ name = "pyfly" # CalVer YY.MM.PATCH — package metadata uses PEP 440 normalized form (26.5.4); # git tag, GitHub release and human-readable display use leading-zero form # (v26.05.04) to match the Java/.NET/Go siblings. -version = "26.6.76" +version = "26.6.77" description = "The official Python implementation of the Firefly Framework — DI, CQRS, EDA, hexagonal architecture, and more." readme = "README.md" license = "Apache-2.0" diff --git a/src/pyfly/__init__.py b/src/pyfly/__init__.py index 0b7300e..52d8186 100644 --- a/src/pyfly/__init__.py +++ b/src/pyfly/__init__.py @@ -13,4 +13,4 @@ # limitations under the License. """PyFly — Enterprise Python Framework.""" -__version__ = "26.06.76" +__version__ = "26.06.77" diff --git a/uv.lock b/uv.lock index 66e6063..68547c0 100644 --- a/uv.lock +++ b/uv.lock @@ -1981,7 +1981,7 @@ wheels = [ [[package]] name = "pyfly" -version = "26.6.76" +version = "26.6.77" source = { editable = "." } dependencies = [ { name = "pydantic" },