feat: support logship to sync logs to cloudwatch

[kerthcet]

What this PR does / why we need it

Which issue(s) this PR fixes

Fixes #

Special notes for your reviewer

Does this PR introduce a user-facing change?

[Copilot]

🟡 Changes recommended

The Modal log follower retries transient failures without any backoff and can emit zero timestamps, both of which can cause avoidable operational issues and/or silent log loss.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

This PR introduces a new optional “component” model under components/, and adds the first component (logship) that continuously streams Modal sandbox stdout/stderr into the existing cluster log pipeline so logs are retained in CloudWatch beyond Modal’s retention window (without directly calling AWS APIs).

Changes:

• Add root Make/CI plumbing to discover and run lint/test targets across nested Go modules under components/.
• Add components/logship as its own Go module with build, Docker, deploy/probe/verify scripts, and a full implementation of watch → supervise → ship → emit for log shipping.
• Add supporting documentation and gitignore updates for component outputs.

File summaries

File	Description
Makefile	Add components-% delegation target to run named targets across all components.
components/README.md	Document the “components are separate Go modules” contract and expectations.
components/logship/README.md	Usage and operational notes for running/deploying/probing logship.
components/logship/Makefile	Component-local build/test/lint/docker/deploy targets (reusing root tooling).
components/logship/internal/watch/watch.go	Watch Nebula Pods and translate them into shippable instances.
components/logship/internal/watch/watch_test.go	Unit tests for instance selection and watch→fleet driving.
components/logship/internal/watch/client.go	Build kube client (in-cluster first, kubeconfig fallback).
components/logship/internal/supervise/supervisor.go	Supervisor for stream lifecycles, retries, stats, and shutdown.
components/logship/internal/supervise/supervisor_test.go	Tests for idempotency, retry policy, stats folding, shutdown behavior.
components/logship/internal/ship/ship.go	Chunk→line assembly (Assembler) for stream data.
components/logship/internal/ship/ship_test.go	Tests for line assembly semantics (cursor, timestamps, CR/LF, bounds).
components/logship/internal/ship/record.go	Record formatting to match consumer schema (nested JSON log envelope).
components/logship/internal/ship/port.go	Define Source/Sink ports for adapters.
components/logship/internal/ship/pipeline.go	Pipeline wiring (source→assembler→batcher→sink) with buffering and retry hooks.
components/logship/internal/ship/pipeline_test.go	Tests for drain, resume cursor, interval flush, dropping behavior, retry rules.
components/logship/internal/ship/batch.go	Batcher with size limits, per-event overhead, JSON-safe splitting.
components/logship/internal/ship/batch_test.go	Tests for caps, splitting correctness, JSON validity, rune boundaries, clamping.
components/logship/internal/modal/source.go	Bind Modal sandbox+FD to ship.Source, plus descriptor naming/parsing.
components/logship/internal/modal/source_test.go	Tests for descriptor round-tripping and descriptor list validity.
components/logship/internal/modal/pool.go	Connection pool to avoid HTTP/2 concurrent stream queuing pitfalls.
components/logship/internal/modal/pool_test.go	Tests for sizing, distribution, and URL validation.
components/logship/internal/modal/logstream.go	Cursor-aware Modal log stream follower with reconnection logic.
components/logship/internal/modal/logstream_test.go	Fake-server-backed tests for cursor semantics, reconnection, retry behavior.
components/logship/internal/modal/fake_server_test.go	bufconn-backed fake Modal gRPC server for logstream tests.
components/logship/internal/modal/client.go	Dial Modal with SDK-matching headers/keepalive; env-based creds.
components/logship/internal/modal/client_test.go	Tests for sdkVersion pin check, env creds, and URL parsing.
components/logship/internal/emit/emit.go	Stdout sink and limits sized for CRI chunking and agent handoff.
components/logship/internal/emit/emit_test.go	Tests for write atomicity, ctx-ignoring, limits, and record preservation.
components/logship/hack/verify.sh	Verify that shipped records arrived and queries behave as expected.
components/logship/hack/probegen/main.go	Generate probe records via the real formatter (shape + size).
components/logship/hack/probe.sh	Deploy a probe pod and validate nested record behavior in CloudWatch.
components/logship/hack/deploy.sh	Deploy logship (RBAC + Deployment) with single-replica Recreate strategy.
components/logship/go.mod	New component module definition and dependency pins (with root replace).
components/logship/Dockerfile	Build logship with repo-root context (to satisfy replace).
components/logship/design.md	Design doc: rationale, architecture, handoff mechanics, failure modes.
components/logship/cmd/main.go	CLI entrypoint: cluster shipping mode + tail mode.
components/logship/cmd/build.go	Build pipelines per instance/descriptor with record identity formatting.
components/logship/cmd/build_test.go	Tests for label carry-through and descriptor validation.
.gitignore	Ignore components/*/bin/ so component build outputs aren’t committed.
.github/workflows/test.yml	Add discovered component matrix to run tests per component module.
.github/workflows/lint.yml	Add discovered component matrix to lint each component module.

Review details

Suppressed comments (1)

components/logship/internal/modal/logstream.go:121

• When Recv() fails with a retryable (non-EOF) error, the code immediately reopens the stream with no delay. Adding a small ctx-aware backoff avoids tight retry loops during transient outages.

				if !errors.Is(err, io.EOF) {
					if !retryable(err) || retries == 0 {
						return fmt.Errorf("reading log stream for sandbox %s: %w", sandboxID, err)
					}
					retries--

• Files reviewed: 40/42 changed files
• Comments generated: 2
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

🟡 Changes recommended

components/logship/Makefile’s build target writes to bin/logship without ensuring bin/ exists, which will break make build on a clean checkout.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

• Files reviewed: 41/43 changed files
• Comments generated: 1
• Review effort level: Lite

[Copilot]

🔵 Needs a closer look

It introduces a large new subsystem (k8s informers + gRPC log streaming + new module/CI plumbing), which warrants final human validation despite tests and generally sound structure.

Review details

• Files reviewed: 41/43 changed files
• Comments generated: 1
• Review effort level: Lite

[Copilot]

🟡 Changes recommended

The new component make build target writes to bin/logship without ensuring bin/ exists, so it will fail on a clean checkout.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

Suppressed comments (1)

components/logship/Makefile:61

• make build will fail on a fresh checkout because bin/ is not created before writing -o bin/logship (Go does not create parent directories for -o). Add mkdir -p bin (or a bin prerequisite target) before the build output is written.

.PHONY: build
build: fmt vet ## Build the logship binary.
	go build -o bin/logship ./cmd

• Files reviewed: 41/43 changed files
• Comments generated: 1
• Review effort level: Lite

[Copilot]

🔵 Needs a closer look

It introduces a substantial new operational component (in-cluster watching + multi-stream shipping) and new CI/workflow behavior that warrants final human validation despite limited concrete review findings.

Review details

Suppressed comments (1)

Previously missed (1) — in code that hasn't changed since the last review.

.github/workflows/ci.yml:40

• The root test job runs go mod tidy but doesn't fail if it changes go.mod/go.sum, which can let dependency state drift without being reflected in the PR. The component test jobs already enforce this via git diff --exit-code; doing the same here will catch untidied root module deps early.

• Files reviewed: 41/43 changed files
• Comments generated: 0 new
• Review effort level: Lite

[Copilot]

🟡 Changes recommended

CI robustness and correctness issues (component discovery failing on empty matches and missing tidy diff enforcement) plus an internal doc/behavior mismatch should be addressed before approval.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

Suppressed comments (2)

Previously missed (1) — in code that hasn't changed since the last review.

.github/workflows/ci.yml:43

• The root test job runs go mod tidy but doesn't fail if it modifies go.mod/go.sum, which can let CI pass while leaving the repo untidy. The component test job already enforces a clean diff; consider doing the same here for consistency.

.github/workflows/ci.yml:65

• The discover step uses ls -d components/*/Makefile, which exits non-zero when the glob matches nothing (e.g., a repo with no components yet), failing the entire workflow. Consider handling the empty case and emitting an empty JSON array instead so CI remains robust.

        run: |
          names=$(ls -d components/*/Makefile | xargs -n1 dirname | xargs -n1 basename | jq -R . | jq -sc .)
          echo "components=$names" >> "$GITHUB_OUTPUT"

• Files reviewed: 41/43 changed files
• Comments generated: 1
• Review effort level: Lite

[Copilot]

🔵 Needs a closer look

It introduces a substantial new shipping component plus CI/build/deploy plumbing changes that warrant final human review for operational fit and rollout safety.

Review details

• Files reviewed: 49/51 changed files
• Comments generated: 0 new
• Review effort level: Lite

[Copilot]

🔵 Needs a closer look

It introduces a large new production component and new CI/build plumbing, so it needs careful human validation of operational behavior and integration assumptions.

Review details

Suppressed comments (2)

Previously missed (2) — in code that hasn't changed since the last review.

.github/workflows/ci.yml:44

• The root-module test job runs go mod tidy but never verifies that it produced no diff. That can let an untidy go.mod/go.sum slip through while CI quietly rewrites the files in the runner (components do enforce this with git diff --exit-code). Consider failing the job if tidy changes anything, for consistency and to keep PRs reproducible.
  components/logship/internal/watch/watch.go:117
• The Watcher.Resync field comment says this is a "relist interval", but the package-level DefaultResync comment explicitly notes this is not a relist (it re-delivers cached objects as updates). This inconsistency can mislead readers about informer behavior and failure modes.

• Files reviewed: 50/52 changed files
• Comments generated: 0 new
• Review effort level: Lite