[PMerlet]

fixes PRD-1670

Adds @forestadmin/datasource-dynamodb: a datasource for Amazon DynamoDB with the capabilities of the Cosmos DB one (introspection, manual schema, composite keys, relations, search, aggregation, charts, arrays as collections, native queries) plus single-table design. DynamoDB cannot sort, count or aggregate server-side, so the datasource reads by key whenever a filter allows it and does the rest on the items it reads; scanPolicy (warn by default) and maxScannedItems keep a request that no key serves from reading a whole table unnoticed.

What changes

1. Before: a DynamoDB table could not be used as a datasource. After: createDynamoDBDataSource({ region }) introspects the tables (keys and indexes from DescribeTable, attributes from a sample), or takes a manual schema; the partition and sort keys are the composite primary key.
2. Before: n/a. After: each request takes the cheapest path: Query on the table or an index when an equality pins a partition key, BatchGetItem when the filter names whole keys (what relation emulation sends), parallel Scan otherwise. Operators DynamoDB lacks (ILike, EndsWith, date intervals), sorts on non-key attributes, counts, aggregations and charts are computed on the items read, streamed so memory stays flat.
3. Before: one collection per table. After: several collections can share a table by sort key prefix or discriminator attribute, and an array of maps can be exposed as a collection keyed by its parent and its index.
4. Before: the CI knows no DynamoDB. After: datasource-dynamodb joins the test matrix, whose job starts DynamoDB Local (docker-compose.yml in the package) and sets DYNAMODB_ENDPOINT for that package only.

Tests

Unit tests use a fake client and assert the requests sent. An integration suite (22 tests, skipped without DYNAMODB_ENDPOINT) runs against DynamoDB Local, since a fake cannot tell whether DynamoDB accepts an expression: it is what validates the key-attribute and sort-key rules of Query filters, the nested array writes and the fan-out of an In on a partition key. A real agent over seeded DynamoDB Local data was also exercised by hand (list, count, paging, search, relations, chart, writes).

Not delivered, per the ticket's own list of additions: collections backed by a secondary index, and optimistic updates.

Decisions

• PRD-1670: 8 functional, 2 architectural · 7 unticked

Definition of Done

General

• Write an explicit title for the Pull Request, following Conventional Commits specification
• Test manually the implemented changes
• Validate the code quality (indentation, syntax, style, simplicity, readability)

Security

• Consider the security impact of the changes made

Squash message

feat(datasource-dynamodb): add a DynamoDB datasource

DynamoDB reads fast by key and scans otherwise, and cannot sort, count or aggregate on the server. The datasource reads by key whenever a filter allows it (Query, a Query per partition of an In, BatchGetItem, parallel Scan otherwise) and does the rest on the items it reads. It introspects tables or takes a manual schema, exposes the keys as a composite primary key, shares a table between collections, exposes arrays of maps (nested too) as collections, and runs PartiQL SELECT native queries. scanPolicy and maxScannedItems keep a request that no key serves from reading a table unnoticed.

🤖 Generated with Claude Code

Note

Add a DynamoDB datasource package

• Adds @forestadmin/datasource-dynamodb in packages/datasource-dynamodb/src. It supports table introspection (sampling up to 100 items, depth 5), a manual-schema mode, a chainable collection builder, and virtual "array collections" that expose array elements as their own collections.
• Plans reads as table queries, GSI queries, scans, or BatchGet key lookups, and pushes filters to DynamoDB where possible; the rest is processed in memory. Sorting, pagination (with an offset cache), aggregations, and binary/set/date value handling are included.
• Native queries run PartiQL SELECT statements only, with maxScannedItems enforcement and a 1,000-page cap. Default client retries are 10 attempts with adaptive mode.
• Includes a DynamoDB Local integration suite via docker-compose.yml and the build workflow matrix entry.
• Risk: introspection version is hardcoded to 1 in types.ts and rejected otherwise by validateManualSchema in manual-schema.ts; new packages land as a first release, so breaking changes can still be made freely before adoption.

^{Macroscope summarized c6eb113.}

[linear-code]

PRD-1670

[qltysh]

75 new issues

Tool	Category	Rule	Count
qlty	Structure	Function with high complexity (count = 16): constructor	26
qlty	Structure	Function with many parameters (count = 4): aggregate	16
qlty	Structure	Function with many returns (count = 5): computeAggregation	14
qlty	Structure	Deeply nested control flow (level = 4)	9
qlty	Structure	Complex binary expression	6
qlty	Structure	High total complexity (count = 154)	4

[qltysh]

Coverage Impact

This PR will not change total coverage.

Modified Files with Diff Coverage (24)

Rating	File	% Diff	Uncovered Line #s
	packages/datasource-dynamodb/src/schema/manual-schema.ts	97.1%	40, 278-279
	packages/datasource-dynamodb/src/utils/sort.ts	100.0%
	packages/datasource-dynamodb/src/utils/concurrency.ts	100.0%
	packages/datasource-dynamodb/src/collection.ts	98.4%	101-102, 507-508, 741
	packages/datasource-dynamodb/src/utils/aggregator.ts	97.4%	29, 35
	packages/datasource-dynamodb/src/datasource.ts	97.1%	89
	packages/datasource-dynamodb/src/introspection/introspector.ts	100.0%
	packages/datasource-dynamodb/src/errors.ts	100.0%
	packages/datasource-dynamodb/src/types.ts	100.0%
	packages/datasource-dynamodb/src/utils/type-converter.ts	97.6%	72
	packages/datasource-dynamodb/src/virtual-collections.ts	100.0%
	packages/datasource-dynamodb/src/utils/executor.ts	100.0%
	packages/datasource-dynamodb/src/client.ts	100.0%
	packages/datasource-dynamodb/src/utils/partiql.ts	100.0%
	packages/datasource-dynamodb/src/settings.ts	100.0%
	packages/datasource-dynamodb/src/utils/pagination-cache.ts	96.6%	67
	packages/datasource-dynamodb/src/index.ts	100.0%
	packages/datasource-dynamodb/src/utils/access-path.ts	98.3%	136, 162
	packages/datasource-dynamodb/src/array-collection.ts	96.0%	415, 438, 470-472...
	packages/datasource-dynamodb/src/utils/serializer.ts	96.7%	138-139
	packages/datasource-dynamodb/src/utils/compare.ts	100.0%
	packages/datasource-dynamodb/src/utils/expression-builder.ts	100.0%
	packages/datasource-dynamodb/src/introspection/builder.ts	100.0%
	packages/datasource-dynamodb/src/schema/schema-converter.ts	100.0%
	Total	98.2%

🤖 Increase coverage with AI coding...

In the `feature/prd-1670-dynamodb-datasource-for-the-nodejs-agent-parity-with-cosmos` branch, add test coverage for this new code:

- `packages/datasource-dynamodb/src/array-collection.ts` -- Lines 415, 438, 470-472, 493-495, 521, 576, 588, and 610
- `packages/datasource-dynamodb/src/collection.ts` -- Lines 101-102, 507-508, and 741
- `packages/datasource-dynamodb/src/datasource.ts` -- Line 89
- `packages/datasource-dynamodb/src/schema/manual-schema.ts` -- Lines 40 and 278-279
- `packages/datasource-dynamodb/src/utils/access-path.ts` -- Lines 136 and 162
- `packages/datasource-dynamodb/src/utils/aggregator.ts` -- Lines 29 and 35
- `packages/datasource-dynamodb/src/utils/pagination-cache.ts` -- Line 67
- `packages/datasource-dynamodb/src/utils/serializer.ts` -- Line 138-139
- `packages/datasource-dynamodb/src/utils/type-converter.ts` -- Line 72

🚦 See full report on Qlty Cloud »

🛟 Help

• Diff Coverage: Coverage for added or modified lines of code (excludes deleted files). Learn more.

• Total Coverage: Coverage for the whole repository, calculated as the sum of all File Coverage. Learn more.

• File Coverage: Covered Lines divided by Covered Lines plus Missed Lines. (Excludes non-executable lines including blank lines and comments.)

  • Indirect Changes: Changes to File Coverage for files that were not modified in this PR. Learn more.

[macroscopeapp]

🟠 High src/array-collection.ts:557

The rewrite fallbacks can erase successful concurrent writes, including appends and updates to unrelated parent attributes. replaceElements and appendByRewriting put the entire item snapshot read by get, while their conditions only check item existence and entity membership; because PutItem replaces the whole item, changes made after the read are lost. Restrict writes to the affected path or condition the replacement on the snapshot and retry on conflict.

🚀 Reply "fix it for me" or copy this AI Prompt for your agent:

In file @packages/datasource-dynamodb/src/array-collection.ts around line 557:

The rewrite fallbacks can erase successful concurrent writes, including appends and updates to unrelated parent attributes. `replaceElements` and `appendByRewriting` put the entire item snapshot read by `get`, while their conditions only check item existence and entity membership; because `PutItem` replaces the whole item, changes made after the read are lost. Restrict writes to the affected path or condition the replacement on the snapshot and retry on conflict.

[PMerlet]

Claude Sonnet 5.5 (claude-sonnet-5-5): Not fixed here: a rewrite fallback can overwrite a write that lands between its read and its put, and closing that needs optimistic locking on a version attribute, which is a feature of its own.

Why

The read is strongly consistent and the put is conditioned on the item existing, so only a concurrent write inside that window is lost. Tracked in PRD-1673, which covers these fallbacks and replaceOne.

[PMerlet]

Spec (PRD-1670): 7 unvalidated decisions: not final. One option name differs from the ticket, see the finding on index.ts.

[arnaud-moncel]

Let's see for a v1