feat(connectors): add sink_template and source_template starting points - #3965
Open
ryerraguntla wants to merge 3 commits into
Open
feat(connectors): add sink_template and source_template starting points#3965ryerraguntla wants to merge 3 commits into
ryerraguntla wants to merge 3 commits into
Conversation
There was a problem hiding this comment.
Pull request overview
Warning
Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.
Adds ready-to-copy “template” sink and source connector crates (plus docs and skill checklists) to standardize connector implementations and reduce recurring review feedback (secrets handling, retries/circuit-breaker, config validation, cursor staging, and canonical tests).
Changes:
- Introduces
sink_templateandsource_templatecrates implementing the expected connector scaffolding withTODO(ConnectorDeveloper)placeholders for backend-specific logic. - Adds example runtime configs and README/docs entries to make the templates discoverable and easy to adopt.
- Updates
.claude/skillsguidance (including newconnector-pr-review) to encode recurring review blockers and conventions.
Reviewed changes
Copilot reviewed 21 out of 22 changed files in this pull request and generated 7 comments.
Show a summary per file
| File | Description |
|---|---|
| core/connectors/sources/source_template/src/lib.rs | New source template implementation + cursor staging + tests |
| core/connectors/sources/source_template/config.toml | Example source template connector config |
| core/connectors/sources/source_template/README.md | Source template usage guidance and expectations |
| core/connectors/sources/source_template/Cargo.toml | Source template crate metadata/deps for workspace + cdylib |
| core/connectors/sources/README.md | Registers source_template in sources index |
| core/connectors/sinks/sink_template/src/lib.rs | New sink template implementation + batching/circuit breaker + tests |
| core/connectors/sinks/sink_template/config.toml | Example sink template connector config |
| core/connectors/sinks/sink_template/README.md | Sink template usage guidance and expectations |
| core/connectors/sinks/sink_template/Cargo.toml | Sink template crate metadata/deps for workspace + cdylib |
| core/connectors/sinks/README.md | Registers sink_template in sinks index |
| core/connectors/runtime/example_config/connectors/source_template.toml | Runtime example config for source template |
| core/connectors/runtime/example_config/connectors/sink_template.toml | Runtime example config for sink template |
| core/connectors/BLOG_POST.md | Draft blog post announcing the templates |
| Cargo.toml | Adds both template crates to workspace members |
| .claude/skills/connectors-overview/SKILL.md | Updates connector overview skill (incl. review checklist pointers) |
| .claude/skills/connector-source/TEMPLATE.md | Reworks source template kit content to align with ACK/NACK + patterns |
| .claude/skills/connector-source/SKILL.md | Documents ACK/NACK contract and staging rules more explicitly |
| .claude/skills/connector-sink/TEMPLATE.md | Reworks sink template kit content and pre-flight guidance |
| .claude/skills/connector-sink/SKILL.md | Adds pointers to PR pre-flight skill |
| .claude/skills/connector-runtime/SKILL.md | Updates metrics naming guidance |
| .claude/skills/connector-pr-review/SKILL.md | New PR review checklist skill for connector PRs |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+371
to
+381
| let mut messages = Vec::with_capacity(records.len()); | ||
| let mut new_cursor = cursor; | ||
| for record in records { | ||
| new_cursor = Some(record.cursor_value); | ||
| let Ok(payload) = serde_json::to_vec(&record.payload) else { | ||
| error!( | ||
| "Failed to serialize a record fetched by {CONNECTOR_NAME} connector with ID: {}", | ||
| self.id | ||
| ); | ||
| continue; | ||
| }; |
Comment on lines
+300
to
+301
| self.config.retry_max_delay.as_deref(), | ||
| DEFAULT_RETRY_MAX_DELAY, |
| // and unknown keys are rejected outright so a typo in a TOML file fails at | ||
| // load time instead of silently doing nothing. | ||
|
|
||
| #[derive(Debug, Clone, Serialize, Deserialize)] |
Comment on lines
+92
to
+93
| #[serde(serialize_with = "iggy_common::serde_secret::serialize_secret")] | ||
| pub connection_string: SecretString, |
Comment on lines
+100
to
+104
| #[serde( | ||
| default, | ||
| serialize_with = "iggy_common::serde_secret::serialize_optional_secret" | ||
| )] | ||
| pub auth_token: Option<SecretString>, |
Comment on lines
+571
to
+581
| #[tokio::test] | ||
| async fn poll_returns_empty_without_error_when_circuit_is_open() { | ||
| let source = TemplateSource::new(1, test_config(), None); | ||
| source.circuit_breaker.record_failure().await; | ||
| source.circuit_breaker.record_failure().await; | ||
| source.circuit_breaker.record_failure().await; | ||
| assert!(source.circuit_breaker.is_open().await); | ||
|
|
||
| let result = source | ||
| .poll() | ||
| .await |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #3965 +/- ##
============================================
- Coverage 84.11% 84.04% -0.08%
Complexity 1358 1358
============================================
Files 1214 1215 +1
Lines 170054 169991 -63
Branches 137611 137748 +137
============================================
- Hits 143045 142862 -183
+ Misses 23305 23293 -12
- Partials 3704 3836 +132
🚀 New features to boost your workflow:
|
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #3956
What
Adds two ready-to-copy connector crates under
core/connectors/:core/connectors/sinks/sink_template/core/connectors/sources/source_template/Each implements every framework-level requirement already expected of a production connector (config parsing with unknown-field rejection,
SecretString-wrapped credentials, connectivity validation inopen(), retry/backoff + circuit breaker viaiggy_connector_sdk::retry, batch processing / Ack-Nack cursor staging, and a canonical test suite), leaving only the backend-specific bits markedTODO(ConnectorDeveloper):Also included:
core/connectors/docs/authoring-sinks-and-sources.md— a written guide walking through the required patterns and why each one exists, meant to be cross-linked from the connector READMEs and reused as the basis for an Apache Iggy blog post..claude/skills/connector-*skill definitions, including a newconnector-pr-reviewskill intended to catch the same recurring review issues (missing retry/circuit-breaker, plaintext secrets, unvalidated config, missing cursor staging) automatically on future connector PRs.Cargo.toml,Cargo.lock) and README entries (core/connectors/sinks/README.md,core/connectors/sources/README.md,core/connectors/README.md) for both new crates.core/connectors/runtime/example_config/connectors/{sink,source}_template.toml).Why
Per the discussion in #3956: connector PRs have been hitting the same review comments repeatedly (improperly typed secrets, missing state-staging, inconsistent error handling, thin test coverage). Rather than repeating that feedback PR after PR, these templates make the required pattern the path of least resistance — a new connector starts from working, reviewed code and only needs the backend-specific logic filled in.
Local Execution
Passed
Pre-commit hooks ran
Testing
cargo fmt --allcargo clippy --all-targets --all-features -- -D warningscargo build -p iggy_connector_template_sink -p iggy_connector_template_sourcecargo test -p iggy_connector_template_sink -p iggy_connector_template_sourcecargo machetecargo sort --workspacetypos./scripts/ci/license-headers.sh --checkAI Usage
Claude
Implementation was AI-generated with human review
Each line of the code is reviewed and can be explained by developer