From 10830c350064ef69b58871e9ba0d4b2edf2d4df6 Mon Sep 17 00:00:00 2001 From: Tim Taplin Date: Thu, 29 Oct 2026 09:23:44 -0500 Subject: [PATCH] README: explain the deferred mysql-nio rewrite in place of an internal note The Status section linked Documentation/mysql-nio-integration-plan.md, an internal working note with local paths and session references. The README now gives the decision and its reasons itself, and the note is removed (it stays in history at 944ef30). Co-Authored-By: Claude Opus 5.5 --- Documentation/mysql-nio-integration-plan.md | 439 -------------------- README.md | 9 +- 2 files changed, 6 insertions(+), 442 deletions(-) delete mode 100644 Documentation/mysql-nio-integration-plan.md diff --git a/Documentation/mysql-nio-integration-plan.md b/Documentation/mysql-nio-integration-plan.md deleted file mode 100644 index c5f63de..0000000 --- a/Documentation/mysql-nio-integration-plan.md +++ /dev/null @@ -1,439 +0,0 @@ -> Received 2026-07-14 from a separate Claude Desktop agent session -> (`local-agent-mode-sessions/.../Perfect-MySQL-NIO-Integration-Plan.md`), -> saved here for the record. - -## Decision (2026-07-14, corrected 2026-07-16): Not now — deferred to an explicit trigger, not flatly declined - -**Correction**: an earlier version of this note cited a since-superseded -bridge (`Sources/LassoPerfectServer/AsyncBridge.swift`, deleted) as the -reason MySQL doesn't need this. That's stale. The actual, authoritative -answer to this exact question already exists, produced independently -(by Tim + a separate Claude Code session) the same day this plan -arrived, with far more rigor than either that stale note or this plan's -own analysis: - -**`Documentation/async-mysql-research-findings.md` in -`/Users/timtaplin/Perfect-Lasso/`** (the Lasso Adapter project — moved/ -renamed from `Documents/Perfect resurrection` since this plan was first -filed here). Read that document in full before acting on this one; the -summary below is not a substitute for it. - -What actually happened in that project, for context: Phase 1 converted -its render pipeline to native `async throws` end-to-end (no bridge -needed anymore, for any backend); Phase 2 put `PerfectCRUDLassoExecutor`'s -blocking MySQL calls on `@concurrent` (SE-0461) to keep them off the -cooperative pool; Phase 3 is the findings doc above, which asked this -plan's exact question — "should the MySQL side genuinely go async -instead?" — and answered with real evidence, not general reasoning: - -- The libmysqlclient build this ecosystem actually links (Oracle - 9.6.0, verified via `otool -L`) has a nonblocking API, but **it has no - nonblocking prepared-statement calls at all** — and Perfect-MySQL's - entire CRUD path is prepared statements. Dead end confirmed by - grepping the actual linked header, not assumed. -- mysql-nio/MySQLKit **cannot back Perfect-CRUD's existing typed - connector protocol** (`SQLExeDelegate` is synchronous pull-based, - `Select: Sequence` can't `await`) without a breaking redesign of - Perfect-CRUD itself — a materially larger and more invasive change - than this plan's "additive DatabaseService layer" framing suggests. - It *could* back the Lasso Adapter's narrower `DynamicQuery`/ - `DynamicMutation`/`DynamicSQL` handler seam directly, bypassing - Perfect-CRUD for that one consumer — a much smaller, additive change - than this plan's full 8-week Perfect-MySQL replacement. -- At this project's actual scale (one legacy site, no connection - pooling, not hundreds-to-thousands of concurrent queries), the - research concludes **connection pooling is the larger, more certain - latency win** — and Phase 2's thread-offload approach can deliver that - without adopting any new client. - -**Ranked disposition** (from the findings doc, this plan's real answer): -1. Thread-pool offload + pooled persistent connections — the actual - plan going forward, already adopted in the Lasso Adapter project. -2. mysql-nio/MySQLKit behind the Lasso Adapter's handler seam - specifically (not a Perfect-MySQL replacement) — kept as an - explicit escape hatch, revisited only if real, *measured* cooperative- - pool starvation is observed under real load. Not ruled out forever; - just not justified by "nicer async style" alone, and not scoped as - "replace Perfect-MySQL for every consumer" the way this plan proposes. -3. Hand-rolling the C-library nonblocking API (MariaDB's connector has - one; Oracle's doesn't cover prepared statements) — worst cost/benefit - of the three, not recommended. - -This plan (the 8-week Perfect-MySQL → mysql-nio replacement below) is -**not being pursued** on that basis: it targets a bigger blast radius -(Perfect-MySQL itself, used by PerfectCRUD/Perfect-Session/PerfectTemplate/ -the Lasso Adapter) than the real problem needs, for a benefit -(async-native style) the actual evidence says isn't the binding -constraint at this project's scale. Perfect-MySQL stays as-is. Kept here -for reference, not deleted — revisit only under the specific trigger -above (measured cooperative-pool starvation), not speculatively. - ---- - -# Perfect Swift + mysql-nio Integration Plan - -## Executive Summary - -Migrate Perfect's MySQL connectivity from the aging libmysqlclient wrapper (Perfect-MySQL) to **mysql-nio**, a modern, pure-Swift async MySQL driver. This enables true async/nonblocking prepared statements while maintaining Perfect's lightweight philosophy and eliminating C API dependencies. - -## Current State Analysis - -### Perfect-MySQL (Current Solution) -- **Foundation**: Wraps Oracle's libmysqlclient library -- **Async Support**: Limited; nonblocking API does NOT support prepared statements -- **Problem**: Binary choice between prepared statements (sync only) or async queries (no prepared statements) -- **Maintenance**: Community-maintained, minimal recent updates -- **Dependencies**: Requires libmysqlclient system library installation - -### The Core Constraint -As documented in Oracle's libmysqlclient 9.6.0: -- Nonblocking API functions: `mysql_real_connect_nonblocking()`, `mysql_real_query_nonblocking()`, `mysql_fetch_row_nonblocking()`, etc. -- **No nonblocking variants for**: `mysql_stmt_prepare()`, `mysql_stmt_execute()`, `mysql_stmt_bind_param()` -- **Workaround in libmysqlclient**: Use multiple connections with polling (inefficient, complex state management) - -## Proposed Solution: mysql-nio - -### What is mysql-nio? -- **Type**: Pure Swift async MySQL client library -- **Owner**: Vapor organization (GitHub: vapor/mysql-nio) -- **License**: Open source -- **Dependencies**: Swift NIO (Apple), swift-nio-ssl, swift-log, swift-crypto — NO Vapor framework required -- **Maturity**: Production-ready (stable releases, active maintenance) -- **Architecture**: Event-driven nonblocking I/O via Swift NIO - -### Why mysql-nio Solves This -✅ True async/await prepared statements (no libmysqlclient limitations) -✅ Pure Swift implementation (no C FFI complexity) -✅ Built on Swift NIO (event-driven, efficient resource usage) -✅ Zero Vapor framework dependencies -✅ Seamless Perfect integration (both are Swift-native) -✅ Connection pooling support built-in -✅ Parameter binding for SQL injection prevention - -### Functional Parity -| Feature | Perfect-MySQL | mysql-nio | Notes | -|---------|---------------|-----------|-------| -| Text queries | ✅ | ✅ | Basic SELECT/INSERT/UPDATE/DELETE | -| Prepared statements | ✅ | ✅ | Both support parameter binding | -| Async queries | ✅ | ✅ | EventLoop-based concurrency | -| Async prepared statements | ❌ | ✅ | **mysql-nio advantage** | -| Connection pooling | Limited | ✅ | Built-in connection management | -| TLS/SSL | ✅ | ✅ | Encryption support | -| Transaction support | ✅ | ✅ | Transaction control | - -## Implementation Architecture - -### High-Level Design -``` -Perfect HTTP Handler - ↓ -[Perfect Router/Request Handler] - ↓ -[Database Service Layer - NEW] - ↓ -[mysql-nio Connection Pool] - ↓ -MySQL Server -``` - -### Layer Responsibilities - -**Database Service Layer** (NEW abstraction) -- Manages mysql-nio connection pool -- Exposes async query/prepared statement methods -- Handles connection lifecycle and error handling -- Provides transaction management -- Acts as bridge between Perfect handlers and mysql-nio - -**Perfect Handler Updates** -- Replace Perfect-MySQL imports with Database Service Layer -- Use `async/await` instead of `.wait()` where possible -- Minimal changes to existing logic (service layer handles compatibility) - -## Implementation Plan - -### Phase 1: Foundation (Weeks 1-2) - -**1.1 Create Database Service Layer** -- New module: `Sources/DatabaseService/` -- File: `MySQLConnection.swift` - - Wraps mysql-nio connection - - Exposes: `query()`, `preparedQuery()`, `execute()`, `transaction()` - - All methods are `async throws` -- File: `MySQLPool.swift` - - Connection pool management using mysql-nio - - Configurable pool size, timeout, idle connection handling -- File: `DatabaseConfig.swift` - - Configuration struct (host, port, user, password, database) - - Load from environment or config file - -**1.2 Add mysql-nio Dependency** -- Update `Package.swift`: - ```swift - .package(url: "https://github.com/vapor/mysql-nio.git", from: "1.7.0") - ``` -- No libmysqlclient system dependency required - -**1.3 Integration Tests** -- Test basic connection pool creation -- Test single query execution -- Test prepared statement with parameters -- Test connection timeout/recovery - -### Phase 2: Core Functionality (Weeks 3-4) - -**2.1 Query Execution Methods** -- Implement `async func query(_ sql: String) -> [[String: Any]]` -- Implement `async func preparedQuery(_ sql: String, params: [MySQLData]) -> [[String: Any]]` -- Add result row parsing (convert MySQLRow to dictionary/Codable) - -**2.2 Prepared Statement Support** -- Parameter binding with type safety -- Support for common types: Int, String, Double, Bool, Date, nil -- Example usage: - ```swift - let results = try await db.preparedQuery( - "SELECT * FROM users WHERE id = ? AND status = ?", - params: [MySQLData(int: 42), MySQLData(string: "active")] - ) - ``` - -**2.3 Transaction Support** -- Implement `async func transaction(_ block: @escaping () async throws -> T) -> T` -- Handle COMMIT/ROLLBACK -- Ensure connection state consistency - -### Phase 3: Migration (Weeks 5-6) - -**3.1 Identify Perfect-MySQL Usage** -- Audit codebase for all Perfect-MySQL imports -- Document current usage patterns -- Categorize: text queries, prepared statements, connection handling - -**3.2 Incremental Migration** -- Start with one route/handler at a time -- Replace Perfect-MySQL calls with DatabaseService Layer -- Run tests after each module update -- Maintain feature parity during migration - -**3.3 Error Handling Alignment** -- Map mysql-nio errors to application error types -- Maintain existing error messages for backwards compatibility -- Add logging for debugging - -### Phase 4: Optimization & Cleanup (Weeks 7-8) - -**4.1 Connection Pool Tuning** -- Benchmark connection pool size vs throughput -- Configure idle connection timeout -- Add metrics/monitoring - -**4.2 Remove Perfect-MySQL** -- Remove dependency from Package.swift -- Remove old wrapper code -- Clean up any libmysqlclient build configuration - -**4.3 Documentation** -- Document DatabaseService Layer API -- Update Perfect deployment guide (no more libmysqlclient requirement) -- Add examples for common queries - -## Code Examples - -### Current Approach (Perfect-MySQL) - Problematic -```swift -import PerfectMySQL - -let mysql = MySQL() -mysql.setOption(.MYSQL_OPT_RECONNECT, value: true) -mysql.connect(host: "localhost", user: "root", password: "password", db: "mydb") - -// Prepared statements must be synchronous -if mysql.queryStatement("SELECT * FROM users WHERE id = ?") { - // Can't use async here... -} -mysql.close() -``` - -### New Approach (mysql-nio) - Async Throughout -```swift -import MySQLNIO - -class DatabaseService { - let pool: MySQLConnectionPool - - init(config: DatabaseConfig) { - self.pool = MySQLConnectionPool( - host: config.host, - port: config.port, - username: config.username, - password: config.password, - database: config.database, - on: eventLoopGroup - ) - } - - // Async prepared statement - fully nonblocking - func getUserById(_ id: Int) async throws -> [String: Any]? { - let rows = try await pool.execute( - "SELECT * FROM users WHERE id = ?", - [MySQLData(int: id)] - ) - return rows.first - } - - // Transaction support - func transferFunds(from: Int, to: Int, amount: Decimal) async throws { - try await pool.transaction { conn in - try await conn.execute( - "UPDATE accounts SET balance = balance - ? WHERE id = ?", - [MySQLData(decimal: amount), MySQLData(int: from)] - ) - try await conn.execute( - "UPDATE accounts SET balance = balance + ? WHERE id = ?", - [MySQLData(decimal: amount), MySQLData(int: to)] - ) - } - } -} - -// Usage in Perfect handler -func handleUserRequest(_ req: HTTPRequest) async throws -> HTTPResponse { - let db = try getDatabase() // Inject or retrieve from app state - let user = try await db.getUserById(42) - - return HTTPResponse(status: .ok, body: encodeJSON(user)) -} -``` - -## Technical Considerations - -### EventLoop Management -- mysql-nio requires a Swift NIO EventLoopGroup -- Perfect can create/manage this centrally in app initialization -- One EventLoopGroup per app, shared across all connections - -### Type Conversion -- MySQLData handles type marshaling (Swift type ↔ MySQL wire format) -- Build type-safe wrapper for common conversions -- Support: Int, Int64, String, Double, Bool, Date, Decimal, UUID, JSON, nil - -### Connection Pooling Strategy -- Recommended pool size: number of CPU cores × 2-4 -- Idle timeout: 30-60 seconds -- Max connections: 20-100 depending on load -- Use metrics to tune (via swift-log integration) - -### Error Handling Strategy -```swift -enum DatabaseError: Error { - case connectionFailed(String) - case queryFailed(String) - case parameterBindingFailed(String) - case transactionFailed(String) -} -``` - -### Thread Safety -- mysql-nio connections are NOT thread-safe -- Connection pool ensures 1 connection per async task -- No manual lock management needed (EventLoop handles it) - -## Risk Mitigation - -| Risk | Mitigation | -|------|-----------| -| Breaking existing code | Use adapter layer; migrate incrementally | -| Connection pool exhaustion | Add monitoring, configure timeouts | -| MySQL version compatibility | Test against MySQL 5.7, 8.0, 8.4 | -| Performance regression | Benchmark before/after; optimize pool settings | -| Concurrency bugs in transition | Comprehensive test suite for each phase | - -## Testing Strategy - -### Unit Tests -- Mock DatabaseService for handler testing -- Test parameter binding type conversion -- Test error handling paths -- Test transaction rollback on failure - -### Integration Tests -- Real MySQL database (Docker recommended for CI) -- Test connection pool behavior under load -- Test prepared statement execution -- Test transaction isolation -- Test connection recovery after network failure - -### Load Testing -- Compare performance: Perfect-MySQL vs mysql-nio -- Measure throughput (queries/second) -- Monitor memory usage under sustained load -- Test connection pool saturation handling - -## Deployment Implications - -### System Dependencies -**Before (Perfect-MySQL)** -- Requires libmysqlclient installed on server -- Version conflicts possible -- Platform-specific installation complexity - -**After (mysql-nio)** -- Zero system dependencies (pure Swift) -- Simpler deployment (Swift binary + config) -- Easier containerization (no C library compilation) - -### Configuration Changes -- Environment variables: `MYSQL_HOST`, `MYSQL_USER`, `MYSQL_PASSWORD`, `MYSQL_DATABASE` -- Optional: `MYSQL_PORT`, `MYSQL_POOL_SIZE`, `MYSQL_IDLE_TIMEOUT` -- Backward compatible with existing Perfect environment setup - -### Performance Expectations -- **Prepared statements**: 10-30% faster than text queries (SQL parsing avoidance) -- **Async overhead**: Minimal (event-loop based, not thread-based) -- **Memory**: Slightly lower (no libmysqlclient overhead) -- **Throughput**: Same or better (efficient connection pooling) - -## Success Criteria - -✅ All text queries execute with async/await -✅ All prepared statements execute with async/await -✅ No prepared statement query blocked on sync calls -✅ Connection pool functions under sustained load (>100 concurrent requests) -✅ Error handling consistent with existing Perfect patterns -✅ Zero breaking changes to Perfect API (backwards compatible layer if needed) -✅ Deployment guide updated, no libmysqlclient required -✅ Test coverage >85% for database layer - -## Timeline Summary - -| Phase | Duration | Deliverable | -|-------|----------|-------------| -| Phase 1: Foundation | 2 weeks | DatabaseService abstraction, tests | -| Phase 2: Core Features | 2 weeks | Query execution, prepared statements, transactions | -| Phase 3: Migration | 2 weeks | Codebase fully migrated, old code removed | -| Phase 4: Optimization | 2 weeks | Performance tuned, documentation complete | -| **Total** | **8 weeks** | Production-ready mysql-nio integration | - -## Next Steps - -1. **Review with Claude Code agents** ← You are here -2. Validate architecture and API design -3. Create initial DatabaseService skeleton -4. Set up integration test database (Docker MySQL) -5. Begin Phase 1 implementation -6. Establish acceptance criteria per phase - -## References - -- [mysql-nio GitHub](https://github.com/vapor/mysql-nio) -- [Swift NIO Documentation](https://apple.github.io/swift-nio/docs/current/NIO/index.html) -- [Perfect Swift Server Framework](https://perfect.org/) -- [MySQL Binary Protocol Documentation](https://dev.mysql.com/doc/internals/en/client-server-protocol.html) -- [Swift Concurrency: async/await](https://developer.apple.com/videos/play/wwdc2021/10132/) - ---- - -**Document Version**: 1.0 -**Date**: July 2026 -**Status**: Ready for Architecture Review diff --git a/README.md b/README.md index 2455a45..d80f44e 100644 --- a/README.md +++ b/README.md @@ -10,10 +10,13 @@ A Swift 6 wrapper around the MySQL client library (libmysqlclient), providing bo ## Status -Perfect-MySQL is core, actively-used infrastructure, depended on directly by Perfect-NIO, -Perfect-Session, and PerfectTemplate — not a standalone/experimental library. +Perfect-NIO, Perfect-Session and PerfectTemplate depend on this package. It's a synchronous, blocking wrapper +around libmysqlclient. -An `mysql-nio`-based async rewrite was considered and deliberately deferred — see [`Documentation/mysql-nio-integration-plan.md`](Documentation/mysql-nio-integration-plan.md) for the tradeoffs. This package remains the synchronous, blocking libmysqlclient wrapper described below. +An async rewrite on [mysql-nio](https://github.com/vapor/mysql-nio) was considered and deferred. +libmysqlclient has no nonblocking calls for prepared statements, which the Perfect-CRUD layer is built on, +and mysql-nio can't back Perfect-CRUD's synchronous connector protocol without a redesign of Perfect-CRUD. +It may be revisited if blocking calls are shown to starve Swift's cooperative thread pool under real load. The pre-Swift-6 version of this package (3.x, last released as 3.5.1) is preserved on the [`legacy`](https://github.com/PerfectlySoft/Perfect-MySQL/tree/legacy) branch. Stay on 3.5.1 if you can't move to Swift 6.2 yet.