Add authenticated, idempotent email-document intake into private DataOps storage

[alexeygrigorev]

Add authenticated, idempotent email-document intake into private DataOps storage

Status: pending
Tags: enhancement, backend, data, infra, P0
Depends on: None
Blocks: None

Scope

Add a machine-to-machine POST /api/v1/intake/email-documents endpoint for the Zapier-replacement sender to submit one email envelope plus zero or more document descriptors. A descriptor represents either an original email attachment or a PDF rendered by the sender from an invoice-email body. The request contains metadata and a stable private S3 source reference; attachment bytes, raw email bodies, credentials, and presigned URLs are never accepted in JSON.

The endpoint must reuse the durable intake model delivered in #31 and the existing generic artifact model. It creates one sensitive email intake item, imports each referenced object into approved DataOps-managed private storage, creates a sensitive artifact for each successfully imported document, and links those artifact references to the intake item. A metadata-only TODO email is valid and creates an intake item with no artifacts. No parallel email-document table or public object path is introduced.

This issue is API/backend/storage work only. The sender remains responsible for receiving email, rendering HTML email bodies to PDF, calculating SHA-256, and placing source objects in the configured private transfer location before calling DataOps.

API Contract

• The route is explicitly versioned and accepts JSON containing: contract version; source Message-ID; configured recipient route; sender; subject; received-at timestamp; and documents (maximum 25).
• Each document contains: kind (attachment or rendered-email-pdf), stable private s3:// source URI, sanitized original filename, declared media type, positive byte size, and sha256:<64 lowercase hex> checksum. Rendered-email artifacts must be PDF. A document may be at most 25 MiB.
• Reject malformed JSON, unknown contract versions, unknown top-level/document fields, invalid or unbounded strings, impossible timestamps, duplicate document descriptors, zero/oversized objects, non-S3 or tokenized URLs, embedded bytes/base64/raw-body fields, and sources outside configured bucket/prefix allowlists. Reject the encoded request body above 256 KiB before JSON parsing.
• Return JSON with stable machine-readable status/error codes and no stack traces or AWS exception text: 202 accepted, 200 duplicate for an exact completed replay, 207 partial-failure with safe per-document indexes/codes, 400 validation-error, 401 unauthorized, 409 idempotency-conflict, 413 payload-too-large, 429 rate-limited, and 503 configuration-error.
• Document the exact request/response schema, retry rules, limits, source-object prerequisites, private-download behavior, and a sanitized example. The API contract is public-safe; it must contain no real addresses, bucket names, account IDs, secret ARNs, object keys, or operational email content.

Authentication and Authorization

• This route does not use an interactive portal session. It has dedicated machine authentication whose credential is supplied only in a header and resolved at runtime from a pre-created AWS Secrets Manager secret configured by stack parameter/environment reference.
• Compare credentials without timing leaks. Missing, malformed, and wrong credentials return the same non-revealing 401 response. Never accept credentials in query strings or bodies and never log them.
• Support credential rotation without a code change or redeploy. Any cache must expire in at most 60 seconds and a failed comparison must be able to refresh so rotation does not leave the endpoint accepting only a stale value.
• Apply a configurable per-credential request rate limit (default 60 requests per minute) and return Retry-After on 429. Structured audit logs may include a non-secret credential identifier, contract version, outcome, counts, and correlation/idempotency hash, but not sender, subject, Message-ID, filenames, S3 paths, checksums, tokens, document bytes, or raw exception messages.
• The public Function URL remains AuthType: NONE only because the application enforces this route-specific credential before any write. Tests must prove the route cannot fall through to portal auth bypass or reach persistence when machine auth fails.

Storage and Privacy

• Accept source references only from explicit configured bucket/prefix allowlists. Before registration, use least-privilege S3 access to verify each source object exists and its actual size/media/checksum metadata agrees with the descriptor; mismatch is a document failure and must not create an artifact.
• Copy verified objects into a DataOps-managed private artifact location before persisting the artifact. Persist only the durable destination s3:// URI; never persist a source presigned URL, source credentials, attachment bytes, raw email body, or signed download URL in DynamoDB.
• The managed bucket/prefix is declared in SAM/CloudFormation, encrypted at rest, blocks all public access, enables versioning, enforces TLS, and uses retention-safe resource policies (DeletionPolicy and UpdateReplacePolicy). Lambda permissions are limited to the configured transfer prefixes and managed destination prefix; no account-wide S3 access.
• Artifacts and intake items are marked sensitive, start in needs-review/new state, and use deterministic IDs. Access to the object remains through the existing authenticated, short-lived private-download mechanism; the endpoint does not return object bytes, a public URL, or a signed URL.
• Existing portable exports/backups retain the intake/artifact metadata needed to reconcile records after restore; private document bytes remain protected in retained/versioned storage and are not embedded in DynamoDB exports.
• Document same-account and cross-account transfer IAM/KMS prerequisites, credential rotation, source allowlist configuration, destination ownership, retry/cleanup behavior, and operator remediation for a blocked partial intake. Documentation must not contain credentials-adjacent live setup values.

Idempotency and Failure Semantics

• The idempotency identity is a normalized (recipientRoute, source Message-ID) pair. Creation must be concurrency-safe (conditional write or equivalent), so simultaneous deliveries cannot overwrite one another or create duplicate intake/artifact records.
• Artifact identity is deterministic from the intake identity plus immutable document checksum/kind, with a documented tie-breaker for two distinct files with the same content. Replaying the exact completed request returns the existing intake/artifact IDs and writes no duplicate records or history noise.
• A retry of the same immutable request after 207 resumes only missing/failed copies and links; it must reuse successful objects/artifacts and finish without duplication.
• Reusing the same idempotency identity with a changed immutable envelope or document manifest returns 409 idempotency-conflict and does not mutate the original intake item.
• A partial failure leaves the intake item visibly blocked, preserves successful links, records safe failure codes in private state/history, and can be remediated by an exact retry. A successful retry returns the item to new and clears the blocked reason.

Acceptance Criteria

• The authenticated versioned endpoint accepts a metadata-only TODO email and creates exactly one sensitive intake item with no artifact.
• One attachment, one rendered-email PDF, and multiple attachments are verified, copied into DataOps-managed private storage, registered as sensitive needs-review artifacts, and linked to the intake item.
• Exact replay and concurrent delivery are idempotent; partial retry resumes; conflicting replay is rejected without mutation.
• Validation and response behavior matches the contract and limits above, including source allowlisting, object metadata verification, payload/rate limits, safe error bodies, and no persistence on authentication/validation failure.
• Secret rotation, least-privilege S3/Secrets Manager permissions, managed-bucket privacy/retention controls, configuration, and outputs are declared in infra/template.full.yaml without hardcoded live credentials, account IDs, secret ARNs, or bucket names.
• Tests inspect stored records and logs to prove sensitive classifications and the absence of raw bodies, bytes, tokens, signed URLs, credentials, and sensitive document metadata in logs/responses.
• Public-safe operator/integrator documentation describes the sanitized contract, retries, IAM/KMS shape, rotation, private access, and partial-failure remediation.
• Automate private Mailchimp account exports and attach them to recurring work #108 and Add authenticated, idempotent email-document intake into private DataOps storage #109 remain independently testable and committable: shared backend/src/router.ts, backend/src/handler.ts, infra/template.full.yaml, and infrastructure tests contain no implicit dependency on mailing-export modules or configuration.
• [HUMAN] In the deployed AWS sandbox, an operator uses a temporary private source object and rotated test credential to verify accepted, exact duplicate, unauthorized-after-rotation, and authenticated private-download behavior, then removes the temporary source object and revokes the test credential.

Test Scenarios

Scenario: metadata-only TODO email

Given: valid machine authentication and a configured TODO recipient route
When: the sender submits a valid envelope with no documents
Then: the API returns 202 accepted and one sensitive intake item exists with no artifact references

Scenario: attachment and rendered invoice import

Given: allowlisted private source objects whose size, media type, and SHA-256 metadata match the descriptors
When: the sender submits one attachment, one rendered PDF, and then a request with multiple attachments
Then: each object is copied once to managed private storage, each artifact is sensitive and needs-review, and all references are linked to the correct intake item

Scenario: exact replay, concurrency, and conflict

Given: a completed request
When: it is replayed exactly, submitted concurrently, or replayed with a changed envelope/document manifest
Then: exact/concurrent deliveries return the same IDs without extra records/history, while the changed request returns 409 and leaves prior state unchanged

Scenario: partial failure and recovery

Given: a multi-document request where one S3 verification/copy or artifact/link write fails
When: the request returns 207 and is retried unchanged after the dependency recovers
Then: successful work is reused, missing work completes exactly once, the blocked state is cleared, and no orphan/duplicate artifact is created

Scenario: rejected and oversized input

Given: missing/wrong credentials or a malformed, oversized, over-limit, unknown-field, embedded-byte, signed-URL, non-allowlisted, metadata-mismatched, or unsupported-version request
When: the route receives it
Then: it returns the specified safe 4xx response and creates/copies nothing

Scenario: secret rotation and throttling

Given: a cached old credential, a newly rotated credential, and a configured per-credential limit
When: callers use the new credential and then exceed the limit
Then: the endpoint refreshes authentication safely, rejects the old credential, and returns 429 with Retry-After without logging either credential

Scenario: infrastructure privacy

Given: the SAM/CloudFormation template
When: static infrastructure tests and sam validate run
Then: public access is blocked, encryption/versioning/TLS/retention are present, IAM is prefix-scoped, runtime configuration references pre-created secrets and allowlists, and no live identifier is hardcoded

Required Verification

• npm --prefix backend test (including focused endpoint tests for every scenario above)
• npm --prefix backend run typecheck
• npm --prefix backend run build
• uv run --with pytest python -m pytest tests/infra
• make sam-validate
• No browser UI changes are in scope, so Playwright screenshots are not required. If implementation adds or changes any operator UI, Tester must also run npm --prefix backend run test:e2e and capture/read screenshots of every changed state.

Out of Scope

• Receiving or polling mail, configuring inbox providers, or replacing the upstream email automation itself.
• Rendering HTML/email bodies to PDF inside DataOps, parsing/OCR/classifying documents, matching them to bookkeeping transactions, or auto-approving artifacts.
• Uploading the attached historical invoice ZIP archives or migrating historical documents; that is separate migration intake.
• A new operator UI, changes to the existing generic intake/artifact review UI, or a second document database.
• Mailing-list provider exports and schedules tracked in Automate private Mailchimp account exports and attach them to recurring work #108. Neither issue may require the other's service, table, bucket, secret, schedule, or module to run.
• Source-repository changes in ../dtc-operations, ../datatasks, ../podcast-assistant, or any Zapier-plan repository.

Dependencies

• Implement raw intake inbox for operational inputs #31 (closed) supplies the durable intake model and Add private bookkeeping ledger and migrate the historical workbook into AWS sandbox #107 (closed) supplies private-document/download patterns that may be reused without coupling email intake to bookkeeping records.
• Automate private Mailchimp account exports and attach them to recurring work #108 is parallel work, not a prerequisite. It currently overlaps shared router/handler/SAM/infrastructure-test files; engineers must coordinate integration and stage only issue-specific hunks so each issue can pass its required verification independently.
• Deployment requires an operator-created Secrets Manager credential and configured private transfer allowlist. Cross-account sources additionally require narrowly scoped source-bucket and KMS grants to the DataOps runtime role; no broad or hardcoded grant is acceptable.

[alexeygrigorev]

PM GROOMED

Rewrote this into an agent-ready API/backend/storage contract after reviewing the existing intake/artifact models, private bookkeeping document patterns, infrastructure, and the current uncommitted email-document implementation. The uncommitted implementation was treated only as evidence: it does not establish approved behavior and currently falls short of this issue in areas such as source allowlisting/object verification, DataOps-owned copying, conflicting-replay protection, concurrency safety, bounded rotation refresh, throttling, log redaction, hardcoded live identifiers, and full privacy/retention controls.

Key product decisions: metadata-only TODO emails are supported; document bytes and raw bodies remain outside JSON/DynamoDB; verified documents are copied into retained/versioned private DataOps storage; idempotency is route plus Message-ID with exact-replay, partial-retry, and conflict semantics; the M2M route has dedicated rotatable authentication and limits; no UI or bookkeeping auto-linking is included.

#108 remains parallel rather than a dependency. Both issues overlap router/handler/SAM/infra-test files, so implementation must coordinate and stage issue-specific hunks while keeping each service, configuration, and verification path independently committable.

Labels set: enhancement, backend, data, infra, P0. needs grooming is absent/removed. Ready for Software Engineer assignment.

[alexeygrigorev]

SOFTWARE ENGINEER — READY FOR TEST

Implemented the groomed email-document intake contract without committing or pushing.

Implementation:

• Added strict versioned request validation, 256 KiB/25-document/25 MiB limits, configured route/source allowlists, stable safe error codes, and route-specific authentication before any portal/session bypass or persistence.
• Added constant-time Secrets Manager credential authentication with <=60-second cache, mismatch refresh for rotation, per-credential throttling, Retry-After, and redacted structured audit events.
• Added conditional deterministic intake/artifact reservations, immutable request fingerprints, exact/concurrent replay behavior, conflict rejection, deterministic same-content tie-breakers, partial recovery, and safe blocked-state history.
• Added S3 HeadObject size/media/SHA-256 verification, deterministic copy into a DataOps-owned destination, sensitive needs-review artifacts, and an authenticated five-minute generic artifact download.
• Added retained/versioned/KMS-encrypted/private/TLS-only storage plus prefix-scoped S3/KMS/Secrets Manager runtime configuration. Removed the hardcoded live secret ARN.
• Expanded public-safe integrator/operator documentation for the schema, responses, retry/remediation, rotation, same/cross-account IAM/KMS shape, retention, restore reconciliation, and private access.
• Preserved the parallel Automate private Mailchimp account exports and attach them to recurring work #108 mailing-export code and unrelated frontend changes in the shared checkout.

Issue-owned files:

• backend/src/routes/emailDocuments.ts
• backend/tests/api-email-documents.test.ts
• docs/email-document-intake.md

Shared supporting files (only #109-specific additions):

• backend/src/db/intake.ts
• backend/src/db/artifacts.ts
• backend/src/routes/artifacts.ts
• backend/src/router.ts
• infra/template.full.yaml
• tests/infra/test_template.py

Verification:

• PASS: focused email-document endpoint suite — 9 tests
• PASS: npm --prefix backend test — 1173 tests, 170 suites
• PASS: npm --prefix backend run typecheck
• PASS: npm --prefix backend run build
• PASS: uv run --with pytest python -m pytest tests/infra — 17 tests
• PASS: make sam-validate
• PASS: git diff --check

Acceptance criteria:

• All agent-verifiable criteria are implemented and checked in the issue body.
• The explicitly marked HUMAN deployed-sandbox credential rotation/source-object/private-download exercise remains for the post-merge operator stage.