feat: Lanes B/C/D — Source/Destination lifecycle, config/record codec, acceptance harness + example

CHANGELOG.md

@@ -17,5 +17,61 @@ doc's Upgrade/rollback section for the pre-1.0 caveat).
   (`SourcePlugin`, `DestinationPlugin`, `SpecifierPlugin`).
 - Handshake implementation (`_handshake.py`): magic-cookie check, protocol
   version negotiation, stdout handshake line.
+- `Source`/`Destination` ABCs (`source.py`/`destination.py`) with dual sync/
+  async method dispatch (`_dispatch.py`), backing onto the generated
+  `SourcePlugin`/`DestinationPlugin` gRPC servicers.
+- `serve()` entry point (`serve.py`): handshake validation, `grpc.aio` server
+  bootstrap, `grpc.health.v1` health registration, and a hand-written
+  `GRPCController.Shutdown` service (`_grpc/_controller.py`) for graceful
+  go-plugin teardown.
+- Hung-event-loop watchdog (`_ShutdownCoordinator` in `serve.py`): an
+  independent `threading.Timer`-based force-exit deadline after `SIGTERM`,
+  bounding a genuinely wedged event loop's shutdown time.
+- `BaseConfig`/`Field`/`to_parameters()` (`config.py`): pydantic-v2-based
+  config with introspection-driven `config.Parameter` mapping (no codegen).
+- `Record`/`Change`/`Operation`/`Metadata` (`record.py`) and the proto
+  (de)serialization adapters (`_grpc/adapters.py`), including the documented
+  `google.protobuf.Struct` int→float fidelity boundary (B3).
+- `BackoffRetry`/`BatchWriteError`/`ConnectorError` (`errors.py`), including
+  `BatchWriteError`'s construction-time-validated, exhaustive per-index
+  accounting (the B1 partial-batch-write fix).
+- Acceptance-test harness (`testing/acceptance.py`, `testing/fixtures.py`):
+  `AcceptanceTestDriver` Protocol, `ConfigurableAcceptanceTestDriver`,
+  `AcceptanceTestSuite` (contract version `2026-07.v1`).
+- Worked example connector (`examples/http-poll-source/`), exercised
+  end-to-end by the acceptance suite in this repo's own test suite.
+- Go-duration config support (`config.py`): `datetime.timedelta` fields now
+  map to `config.Parameter.TYPE_DURATION`, serializing/parsing Go's
+  `time.Duration` string syntax (`"5s"`, `"1h30m"`, `"500ms"`) via the new
+  public `format_go_duration()`/`parse_go_duration()` functions. Closes the
+  previously-`NotImplementedError` Duration A-gap.
+- `BatchWriteError.partial(batch_size, written=N, cause=exc)` (`errors.py`):
+  the recommended constructor for the common partial-batch-write case,
+  carrying the real exception that caused the failure instead of a generic
+  placeholder message.
+- `Configure` RPC handlers now catch `pydantic.ValidationError` explicitly
+  and abort with `INVALID_ARGUMENT` plus a per-field detail message
+  (`errors.format_validation_error()`), rather than relying on `grpc.aio`'s
+  generic `UNKNOWN`-status wrapping of an uncaught exception.
+- Lifecycle hook rename: `Source`/`Destination`'s `lifecycle_on_created`/
+  `lifecycle_on_updated`/`lifecycle_on_deleted` are now `on_created`/
+  `on_updated`/`on_deleted` (the `lifecycle_` prefix was redundant).
+- `conduit-connector-sdk build` (`_build.py`/`_cli.py`, new console-script
+  entry point): packages a connector project into a single, directly
+  executable artifact with an absolute-interpreter-path shebang (no `PATH`
+  lookup at exec time, per design doc §1.1.6) and every third-party
+  dependency vendored in, including compiled-extension dependencies
+  (`grpcio`, `pydantic-core`) via an extract-on-first-run bootstrap.
+
+### Fixed
+
+- `serve()`'s SIGTERM-triggered graceful shutdown path (`_sigterm_shutdown`
+  in `serve.py`) no longer silently hangs until the hung-loop watchdog's
+  deadline if `teardown()` itself raises (e.g. a connector's `teardown()`
+  running before `Open` was ever called) — `shutdown_requested` is now set
+  unconditionally, with a clear diagnostic distinguishing "teardown raised"
+  from "loop genuinely wedged." Found via a real end-to-end SIGTERM test
+  while building `conduit-connector-sdk build`'s test coverage. The example
+  connector's own `teardown()` is also now defensive against this case.
 
 No release has been tagged yet; nothing here is installable from PyPI.

README.md

@@ -48,13 +48,44 @@ src/conduit/
   destination.py       # Destination ABC
   serve.py              # handshake + gRPC server bootstrap
   _handshake.py          # magic cookie, protocol negotiation, stdout line
-  _grpc/                  # generated protobuf/grpc stubs (buf generate output)
-  testing/                 # acceptance-test harness (fast-follow)
-examples/http-poll-source/  # worked example connector
-docs/design/                 # design docs for this repo
-tests/                         # unit tests
+  _build.py                # `conduit-connector-sdk build` implementation
+  _cli.py                   # `conduit-connector-sdk` console-script entry point
+  _grpc/                      # generated protobuf/grpc stubs (buf generate output)
+  testing/                     # acceptance-test harness (acceptance.py, fixtures.py)
+examples/http-poll-source/      # worked example connector
+docs/design/                     # design docs for this repo
+tests/                             # unit tests
 ```
 
+## Building a standalone connector artifact
+
+Conduit launches a standalone connector as a subprocess with a **clean
+environment** — no inherited `PATH` (design doc §1.1.6). A `pip
+install`-then-shebang-script connector (`#!/usr/bin/env python3`, or an
+activated venv) cannot launch this way: there's no `PATH` for `env` to
+search. `conduit-connector-sdk build` closes that gap:
+
+```shell
+conduit-connector-sdk build examples/http-poll-source -o http-poll-source.pyz
+./http-poll-source.pyz   # directly executable — no `python` prefix, no venv activation
+```
+
+This produces one file with an **absolute** interpreter shebang (resolved
+at build time, never looked up via `PATH`), bundling every third-party
+dependency your connector needs — including compiled-extension
+dependencies like `grpcio`/`pydantic`'s `pydantic-core`, which a plain
+[`zipapp`](https://docs.python.org/3/library/zipapp.html) can't load
+in-place: the artifact extracts itself to a per-build cache directory on
+first run (the same fundamental approach `shiv`/`pex` use), then executes
+your connector's real entry point from those extracted files. Later
+launches of the same build reuse the cache.
+
+**Precondition:** run `build` from an environment where your connector's
+own dependencies are already installed (however you installed them — pip,
+uv, poetry) — it vendors from what's already resolved, not a fresh
+`pip install`. See `conduit/_build.py`'s module docstring for the full
+rationale and known limitations.
+
 ## Contributing
 
 See [CONTRIBUTING.md](CONTRIBUTING.md). This SDK sits on Conduit's data path —

docs/design/20260707-python-connector-sdk.md

@@ -417,15 +417,17 @@ is enough to pick the right oneof branch. **`Data = bytes | Mapping[str, Any]`.*
 ```python
 @dataclass
 class Change:
-    before: Data | None = None   # update/delete only
-    after: Data | None = None    # all ops except delete
+    before: Data | None = None  # update/delete only
+    after: Data | None = None  # all ops except delete
+
 
 class Operation(enum.Enum):
     CREATE = 1
     UPDATE = 2
     DELETE = 3
     SNAPSHOT = 4
 
+
 @dataclass
 class Record:
     position: bytes
@@ -548,6 +550,7 @@ metadata is auto-populated.
 
 ```python
 """A minimal Conduit source connector: polls an HTTP endpoint for new rows."""
+
 from __future__ import annotations
 
 import asyncio
@@ -916,13 +919,18 @@ replacement for, the ordinary (non-wedged) SIGTERM-mid-write drain test.
    interaction with process-exit signal handling needs care. **This risk is
    now split into two distinct sub-cases, per ▶ MUST-FIX 3 above**: (a) the
    ordinary case — an event loop that's mid-`await` when SIGTERM arrives must
-   let in-flight writes drain before exit (covered by the ordinary
-   SIGTERM-mid-write test), and (b) the **hung-loop** case — SIGTERM arrives
-   while the loop is wedged and cannot process the signal handler at all,
-   which has no Go analog and needs the bounded watchdog mitigation and its
-   own dedicated test. Treating these as one risk understates (b), which is
-   the one that can genuinely hang a shutdown indefinitely without the
-   watchdog.
+   let in-flight reads/writes drain before `teardown()` runs, enforced by
+   `_SourceServicer.drain`/`_DestinationServicer.drain` (`conduit/source.py`,
+   `conduit/destination.py`) and wired into `conduit/serve.py`'s
+   `_sigterm_shutdown`; covered by
+   `tests/test_serve.py::TestSigtermDrainsInFlightOperation`'s
+   `test_sigterm_mid_write_drains_before_teardown` and
+   `test_sigterm_mid_read_drains_before_teardown` — and (b) the **hung-loop**
+   case — SIGTERM arrives while the loop is wedged and cannot process the
+   signal handler at all, which has no Go analog and needs the bounded
+   watchdog mitigation and its own dedicated test. Treating these as one risk
+   understates (b), which is the one that can genuinely hang a shutdown
+   indefinitely without the watchdog.
 4. **Performance vs. Go is unknown and unclaimed.** No benchmark exists yet;
    per CLAUDE.md, no performance claim should be made about this SDK until a
    `benchi` run is committed to the repo.

examples/http-poll-source/README.md

@@ -1,12 +1,31 @@
-# http-poll-source (not started)
+# http-poll-source
 
 The Phase-1 worked example connector from
 `docs/design/20260707-python-connector-sdk.md` §2.7 — a minimal source that
-polls an HTTP endpoint for new rows. This is **Lane D** in the v0.19
-workstream, depending on Lane B (`Source`/`Destination` ABCs, `BaseConfig`,
-the OpenCDC record model) — none of which exists yet as of this scaffold.
+polls an HTTP endpoint for new rows. Implemented in `main.py`, fully runnable
+(`python main.py`, launched by a go-plugin host such as Conduit — it is not
+meant to be run as a bare script; see `conduit._handshake`).
 
-This will become both the worked example *and* the source for
-`conduit connector new --lang python`'s scaffolded template (Lane E), per the
-design doc's §11 reasoning for reusing one connector for both rather than
+Exercised end to end, in-process, by this repo's own
+`tests/test_example_http_poll_source.py`, which runs the versioned
+acceptance suite (`conduit.testing.acceptance`) against this exact file
+against a real local HTTP server — not a mock of `httpx`, not a stub of the
+connector.
+
+This is both the worked example *and* will become the source for
+`conduit connector new --lang python`'s scaffolded template (Phase 3), per
+the design doc's reasoning for reusing one connector for both rather than
 building a separate template repo before the SDK API has stabilized.
+
+## Notes for connector authors
+
+- **You don't need to override `Source.ack()`** unless you're also
+  acknowledging against the source system itself (e.g. committing a Kafka
+  consumer offset, deleting a queue message, marking a row processed
+  upstream). This example doesn't override it -- Conduit's own
+  position-based resume (via `open(position)`) is enough for an HTTP
+  polling source with no upstream ack concept.
+- Build a standalone, directly-executable artifact for this connector with
+  `conduit-connector-sdk build examples/http-poll-source -o http-poll-source`
+  -- see the root [`README.md`](../../README.md#building-a-standalone-connector-artifact)
+  for why this is required (not just convenient) for Conduit to launch it.

examples/http-poll-source/main.py

@@ -0,0 +1,101 @@
+"""A minimal Conduit source connector: polls an HTTP endpoint for new rows.
+
+The Phase-1 worked example from
+``docs/design/20260707-python-connector-sdk.md`` §2.7 -- the SDK's own
+"hello world," made fully runnable rather than illustrative-only. It expects
+an HTTP endpoint that accepts ``?since=<cursor>`` and returns a JSON list of
+rows (each with an ``id`` field) newer than that cursor, oldest-first.
+
+Run it directly with ``python main.py`` under a real go-plugin host
+(Conduit); it is not meant to be run as a plain script otherwise -- see
+``conduit._handshake`` for why (this process expects
+``CONDUIT_PLUGIN_MAGIC_COOKIE``/``PLUGIN_PROTOCOL_VERSIONS`` to be set by
+the launching host).
+
+See ``tests/test_example_http_poll_source.py`` (this repo's own test suite,
+not this directory) for the versioned acceptance suite run against this
+exact file, in-process, against a real local HTTP server -- proving this
+example is a fully working connector, not just a doc snippet.
+"""
+
+from __future__ import annotations
+
+from datetime import UTC, datetime
+
+import httpx
+
+from conduit import BackoffRetry, Change, Metadata, Operation, Record, Source, serve
+from conduit.config import BaseConfig, Field, Specification
+
+
+class Config(BaseConfig):
+    """Configuration for :class:`HTTPPollSource`."""
+
+    url: str = Field(description="HTTP endpoint to poll, expects ?since=<cursor>.")
+    poll_interval_ms: int = Field(
+        default=1000, ge=100, description="Delay between empty polls (paced by the SDK itself)."
+    )
+
+
+class HTTPPollSource(Source[Config]):
+    """Polls ``config.url?since=<cursor>`` for new rows, oldest-first."""
+
+    _client: httpx.AsyncClient | None = None
+    """``None`` until :meth:`open` runs. Checked in :meth:`teardown` -- see
+    that method's docstring for why this matters even though a normal
+    Conduit-driven lifecycle always calls ``Open`` before ``Teardown``."""
+
+    async def open(self, position: bytes | None) -> None:
+        """Open the HTTP client and resume from ``position`` (or the beginning).
+
+        Per invariant 2: ``position`` is the last cursor this connector (or
+        a previous run of it) successfully emitted -- resuming means
+        requesting strictly newer rows, never replaying it.
+        """
+        self._client = httpx.AsyncClient()
+        self._since = position.decode() if position else "0"
+
+    async def read(self) -> Record:
+        """Fetch the next row past ``self._since``, or signal there's nothing yet.
+
+        Raises:
+            conduit.errors.BackoffRetry: the endpoint returned no new rows;
+                the SDK's own read loop paces retries -- this does not
+                sleep itself, avoiding a double backoff (design doc §2.7).
+        """
+        resp = await self._client.get(self.config.url, params={"since": self._since})
+        rows = resp.json()
+        if not rows:
+            raise BackoffRetry()
+
+        row = rows[0]
+        self._since = str(row["id"])
+        metadata: dict[str, str] = {}
+        Metadata.set_read_at(metadata, int(datetime.now(UTC).timestamp() * 1e9))
+        return Record(
+            position=self._since.encode(),
+            operation=Operation.CREATE,
+            key={"id": row["id"]},
+            payload=Change(after=row),
+            metadata=metadata,
+        )
+
+    async def teardown(self) -> None:
+        """Close the HTTP client, if one was ever opened.
+
+        Guards against ``teardown()`` running before ``open()`` ever did --
+        e.g. SIGTERM arriving immediately after the subprocess starts, before
+        Conduit calls ``Configure``/``Open``. A normal Conduit-driven
+        lifecycle always calls ``Open`` before ``Teardown``, but SIGTERM can
+        arrive at any point (design doc's SIGTERM-mid-write failure mode),
+        and an unguarded ``self._client.aclose()`` here would raise
+        ``AttributeError`` in that case -- found via a real end-to-end
+        SIGTERM test while building ``conduit-connector-sdk build``'s test
+        coverage (see ``tests/test_build.py``).
+        """
+        if self._client is not None:
+            await self._client.aclose()
+
+
+if __name__ == "__main__":
+    serve(Specification(name="http-poll", version="0.1.0", author="you"), source=HTTPPollSource)

examples/http-poll-source/pyproject.toml

@@ -0,0 +1,22 @@
+[build-system]
+requires = ["hatchling"]
+build-backend = "hatchling.build"
+
+[project]
+name = "http-poll-source"
+version = "0.1.0"
+description = "Example Conduit source connector: polls an HTTP endpoint for new rows."
+requires-python = ">=3.11"
+# This mirrors what `conduit connector new --lang python` will scaffold
+# (Phase 3, design doc §3/§4) -- a standalone connector project depending
+# on the published `conduit-connector-sdk` package. Pinned as a local path
+# dependency here since this example lives inside the SDK's own repo and is
+# exercised by the SDK's own test suite (../../tests/test_example_http_poll_source.py),
+# not published to PyPI itself.
+dependencies = [
+    "conduit-connector-sdk",
+    "httpx>=0.27,<1",
+]
+
+[tool.uv.sources]
+conduit-connector-sdk = { path = "../..", editable = true }