diff --git a/packages/mongodb-schema/README.md b/packages/mongodb-schema/README.md index ee5eacf3e..6c599d553 100644 --- a/packages/mongodb-schema/README.md +++ b/packages/mongodb-schema/README.md @@ -163,6 +163,86 @@ A high-level view of the schema tree structure is as follows: ![](./docs/mongodb-schema_diagram.png) +## Using a collection's schema validator + +If a collection has a [`$jsonSchema` validator][json-schema-validator] set on it, that +validator already describes the collection's shape, so you can read it directly instead of +sampling documents. How far the existing documents actually match it depends on the +collection's `validationLevel` (see [below](#validation-levels)). +`convertMongoDBJSONSchemaToSimplified` turns a `$jsonSchema` into the +same simplified schema `getSimplifiedSchema` returns: + +```javascript +const { + convertMongoDBJSONSchemaToSimplified, + getSimplifiedSchema, +} = require('@mongodb-js/mongodb-schema'); + +const [collInfo] = await database.listCollections({ name: 'data' }).toArray(); +const jsonSchema = collInfo?.options?.validator?.$jsonSchema; + +const schema = jsonSchema + ? convertMongoDBJSONSchemaToSimplified(jsonSchema) + : await getSimplifiedSchema(database.collection('data').find()); +``` + +The function takes the `$jsonSchema` subdocument itself, not the enclosing `validator` +document. A validator can combine `$jsonSchema` with other query operators, e.g. +`{ $and: [{ $jsonSchema: {...} }, { status: { $in: [...] } }] }`; extracting it from those +is left to the caller, as the example above does only for the top-level case. This is +distinct from `anyOf`/`oneOf`/`allOf` _inside_ the `$jsonSchema`, which are handled (see +below). + +A validator constrains documents rather than describing them, so the conversion is +intentionally lossy and never throws. Value-level constraints (`enum`, `minimum`, `pattern`, +`maxLength`, ...) are ignored, since the simplified schema records BSON types only, as are +`patternProperties` and `additionalProperties`. Beyond that: + +- Keywords at the same level all apply, so the result is the types every one of them + permits. `anyOf` and `oneOf` contribute the union of their branches, intersected with the + schema's own types: `{ bsonType: ['null', 'int'], anyOf: [{ bsonType: 'string' }, { bsonType: 'int' }] }` + permits only `int`. `allOf` intersects its branches. Document fields and array members are + intersected the same way, and a field no value can satisfy is omitted. `oneOf` is treated + like `anyOf`, since exclusivity can't be expressed in the simplified schema. +- A branch with no type of its own constrains the types of the schema it belongs to, so + `{ bsonType: 'object', oneOf: [{ properties: { a } }, { properties: { b } }] }` describes + one document with fields `a` and `b`. This applies at the root too. +- A subschema with no `bsonType` or `type` is read as a document if it has `properties`, and + as an array if it has `items`. +- `bsonType: 'number'` (and `type: 'number'`) expands to `Int32`, `Long`, `Double` and + `Decimal128`. +- `additionalItems` only applies alongside a tuple-form `items`, as in JSON Schema. Since it + defaults to permitting anything and the simplified schema can't record tuple positions, a + tuple's members are reported as unconstrained (an empty `types` list) unless + `additionalItems` is `false` or a schema. The same holds for a union in which any array + branch leaves its members unconstrained. +- Fields encrypted with client-side field level encryption (`encrypt`) are reported as + `Binary`, which is how they are stored. +- A document with `$ref` and `$id` properties is reported as `DBRef`, as the driver + deserializes one. Unless the validator also lists both as `required`, it can still hold + plain documents, so it is reported as both `Document` and `DBRef`. +- A field whose subschema says nothing about its type is omitted from the result. + +### Validation levels + +A validator only describes the documents it has actually been enforced on, and the +collection's `validationLevel` and `validationAction` decide which ones those are: + +- `constraint` (MongoDB 9.0+): every document in the collection is guaranteed to match. + The server checks existing documents when the level is set, and rejects + `bypassDocumentValidation` writes. +- `strict`: all inserts and updates are validated, but documents that were already in the + collection before the validator was set, or that were written with + `bypassDocumentValidation`, may not match. +- `moderate`: updates to documents that don't already match are not validated, so + non-matching documents can stay that way. +- `off`, or a `validationAction` of `warn`: nothing is enforced, and the validator may + describe the intended shape rather than the actual one. + +Even with `constraint`, a validator need not cover every field in the collection (most +leave out `_id`, for example) unless it sets `additionalProperties: false`. Where +completeness matters, prefer inferring the schema from documents. + ## BSON Types `mongodb-schema` supports all [BSON types][bson-types]. @@ -348,6 +428,7 @@ npm test Apache 2.0 [bson-types]: http://docs.mongodb.org/manual/reference/bson-types/ +[json-schema-validator]: https://www.mongodb.com/docs/manual/core/schema-validation/specify-json-schema/ [tests]: https://github.com/mongodb-js/devtools-shared/tree/main/packages/mongodb-schema/test [npm_img]: https://img.shields.io/npm/v/@mongodb-js/mongodb-schema.svg [npm_url]: https://www.npmjs.org/package/@mongodb-js/mongodb-schema diff --git a/packages/mongodb-schema/src/index.ts b/packages/mongodb-schema/src/index.ts index 7d95f4289..f9ab82f33 100644 --- a/packages/mongodb-schema/src/index.ts +++ b/packages/mongodb-schema/src/index.ts @@ -21,6 +21,7 @@ import type { import { convertInternalToExpanded } from './schema-converters/internal-to-expanded'; import { convertInternalToMongodb } from './schema-converters/internal-to-mongodb'; import { convertInternalToStandard } from './schema-converters/internal-to-standard'; +import { convertMongoDBJSONSchemaToSimplified } from './schema-converters/mongodb-to-simplified'; import * as schemaStats from './stats'; import type { AnyIterable, @@ -102,6 +103,7 @@ export { analyzeDocuments, getSchemaPaths, getSimplifiedSchema, + convertMongoDBJSONSchemaToSimplified, SchemaAnalyzer, schemaStats, toTypescriptTypeDefinition, diff --git a/packages/mongodb-schema/src/schema-analyzer.ts b/packages/mongodb-schema/src/schema-analyzer.ts index 3c3c60486..48630bc0c 100644 --- a/packages/mongodb-schema/src/schema-analyzer.ts +++ b/packages/mongodb-schema/src/schema-analyzer.ts @@ -28,7 +28,8 @@ type TypeCastMap = { Decimal128: Decimal128; Double: Double; Int32: Int32; - Int64: Long; + // Keyed by the BSON `_bsontype` value, which is what `getBSONType` reports. + Long: Long; MaxKey: MaxKey; MinKey: MinKey; Null: null; @@ -121,7 +122,11 @@ export type Schema = { fields: SchemaField[]; }; -type SchemaBSONType = Exclude | 'Document'; +// Note: not fully exhaustive of what `getBSONType` can return, since it reports +// the raw `_bsontype`. `DBRef` is listed because it has no `TypeCastMap` entry; +// `Number`, `RegExp` and `Symbol` are also reachable at runtime but absent here. +export type SchemaBSONType = + Exclude | 'Document' | 'DBRef'; type SchemaAnalysisBaseType = { name: string; diff --git a/packages/mongodb-schema/src/schema-converters/mongodb-to-simplified.test.ts b/packages/mongodb-schema/src/schema-converters/mongodb-to-simplified.test.ts new file mode 100644 index 000000000..beee359b7 --- /dev/null +++ b/packages/mongodb-schema/src/schema-converters/mongodb-to-simplified.test.ts @@ -0,0 +1,1247 @@ +import assert from 'assert'; +import type { SimplifiedSchema } from '..'; +import { analyzeDocuments, getSimplifiedSchema } from '..'; +import type { JSONSchema } from '../types'; +import { allBSONTypesDoc } from '../../test/all-bson-types-fixture'; +import { convertMongoDBJSONSchemaToSimplified } from './mongodb-to-simplified'; + +describe('convertMongoDBJSONSchemaToSimplified', function () { + describe('bsonType mapping', function () { + it('maps flat scalar properties to their inference type names', function () { + const result = convertMongoDBJSONSchemaToSimplified({ + bsonType: 'object', + properties: { + str: { bsonType: 'string' }, + int: { bsonType: 'int' }, + lng: { bsonType: 'long' }, + dbl: { bsonType: 'double' }, + dec: { bsonType: 'decimal' }, + bool: { bsonType: 'bool' }, + oid: { bsonType: 'objectId' }, + dt: { bsonType: 'date' }, + nul: { bsonType: 'null' }, + rx: { bsonType: 'regex' }, + sym: { bsonType: 'symbol' }, + js: { bsonType: 'javascript' }, + jsws: { bsonType: 'javascriptWithScope' }, + bin: { bsonType: 'binData' }, + ts: { bsonType: 'timestamp' }, + mink: { bsonType: 'minKey' }, + maxk: { bsonType: 'maxKey' }, + undef: { bsonType: 'undefined' }, + }, + }); + + assert.deepEqual(result, { + str: { types: [{ bsonType: 'String' }] }, + int: { types: [{ bsonType: 'Int32' }] }, + lng: { types: [{ bsonType: 'Long' }] }, + dbl: { types: [{ bsonType: 'Double' }] }, + dec: { types: [{ bsonType: 'Decimal128' }] }, + bool: { types: [{ bsonType: 'Boolean' }] }, + oid: { types: [{ bsonType: 'ObjectId' }] }, + dt: { types: [{ bsonType: 'Date' }] }, + nul: { types: [{ bsonType: 'Null' }] }, + rx: { types: [{ bsonType: 'BSONRegExp' }] }, + sym: { types: [{ bsonType: 'BSONSymbol' }] }, + js: { types: [{ bsonType: 'Code' }] }, + jsws: { types: [{ bsonType: 'CodeWScope' }] }, + bin: { types: [{ bsonType: 'Binary' }] }, + ts: { types: [{ bsonType: 'Timestamp' }] }, + mink: { types: [{ bsonType: 'MinKey' }] }, + maxk: { types: [{ bsonType: 'MaxKey' }] }, + undef: { types: [{ bsonType: 'Undefined' }] }, + }); + }); + + it('expands the numeric "number" alias to every BSON numeric type', function () { + const result = convertMongoDBJSONSchemaToSimplified({ + bsonType: 'object', + properties: { n: { bsonType: 'number' } }, + }); + + assert.deepEqual(result, { + n: { + types: [ + { bsonType: 'Int32' }, + { bsonType: 'Long' }, + { bsonType: 'Double' }, + { bsonType: 'Decimal128' }, + ], + }, + }); + }); + + it('deduplicates "number" against the numeric types it covers', function () { + const result = convertMongoDBJSONSchemaToSimplified({ + bsonType: 'object', + properties: { n: { bsonType: ['double', 'number', 'null'] } }, + }); + + assert.deepEqual(result, { + n: { + types: [ + { bsonType: 'Double' }, + { bsonType: 'Int32' }, + { bsonType: 'Long' }, + { bsonType: 'Decimal128' }, + { bsonType: 'Null' }, + ], + }, + }); + }); + + it('produces one type entry per alias when bsonType is an array', function () { + const result = convertMongoDBJSONSchemaToSimplified({ + bsonType: 'object', + properties: { mixed: { bsonType: ['string', 'int', 'null'] } }, + }); + + assert.deepEqual(result, { + mixed: { + types: [ + { bsonType: 'String' }, + { bsonType: 'Int32' }, + { bsonType: 'Null' }, + ], + }, + }); + }); + + it('deduplicates repeated aliases within a bsonType array', function () { + const result = convertMongoDBJSONSchemaToSimplified({ + bsonType: 'object', + properties: { n: { bsonType: ['int', 'int'] } }, + }); + + assert.deepEqual(result, { n: { types: [{ bsonType: 'Int32' }] } }); + }); + + it('maps dbPointer to DBRef, inverting internalToMongoDB', function () { + const result = convertMongoDBJSONSchemaToSimplified({ + bsonType: 'object', + properties: { ptr: { bsonType: 'dbPointer' } }, + }); + + assert.deepEqual(result, { ptr: { types: [{ bsonType: 'DBRef' }] } }); + }); + + it('drops unrecognised bsonType aliases without throwing', function () { + const result = convertMongoDBJSONSchemaToSimplified({ + bsonType: 'object', + properties: { weird: { bsonType: ['nonsense', 'string'] } }, + }); + + assert.deepEqual(result, { weird: { types: [{ bsonType: 'String' }] } }); + }); + }); + + describe('standard "type" keyword fallback', function () { + it('falls back to type when bsonType is absent', function () { + const result = convertMongoDBJSONSchemaToSimplified({ + bsonType: 'object', + properties: { + s: { type: 'string' }, + n: { type: 'number' }, + i: { type: 'integer' }, + b: { type: 'boolean' }, + nul: { type: 'null' }, + o: { type: 'object', properties: { a: { type: 'string' } } }, + a: { type: 'array', items: { type: 'string' } }, + }, + }); + + assert.deepEqual(result, { + s: { types: [{ bsonType: 'String' }] }, + n: { + types: [ + { bsonType: 'Int32' }, + { bsonType: 'Long' }, + { bsonType: 'Double' }, + { bsonType: 'Decimal128' }, + ], + }, + i: { types: [{ bsonType: 'Int32' }, { bsonType: 'Long' }] }, + b: { types: [{ bsonType: 'Boolean' }] }, + nul: { types: [{ bsonType: 'Null' }] }, + o: { + types: [ + { + bsonType: 'Document', + fields: { a: { types: [{ bsonType: 'String' }] } }, + }, + ], + }, + a: { + types: [{ bsonType: 'Array', types: [{ bsonType: 'String' }] }], + }, + }); + }); + + it('prefers bsonType over type when both are present', function () { + const result = convertMongoDBJSONSchemaToSimplified({ + bsonType: 'object', + properties: { n: { bsonType: 'long', type: 'number' } }, + }); + + assert.deepEqual(result, { n: { types: [{ bsonType: 'Long' }] } }); + }); + }); + + describe('documents', function () { + it('recurses into nested properties', function () { + const result = convertMongoDBJSONSchemaToSimplified({ + bsonType: 'object', + properties: { + outer: { + bsonType: 'object', + properties: { + inner: { + bsonType: 'object', + properties: { leaf: { bsonType: 'int' } }, + }, + }, + }, + }, + }); + + assert.deepEqual(result, { + outer: { + types: [ + { + bsonType: 'Document', + fields: { + inner: { + types: [ + { + bsonType: 'Document', + fields: { leaf: { types: [{ bsonType: 'Int32' }] } }, + }, + ], + }, + }, + }, + ], + }, + }); + }); + + it('emits an empty fields map for an object with no properties', function () { + const result = convertMongoDBJSONSchemaToSimplified({ + bsonType: 'object', + properties: { o: { bsonType: 'object' } }, + }); + + assert.deepEqual(result, { + o: { types: [{ bsonType: 'Document', fields: {} }] }, + }); + }); + + it('returns an empty schema for a root with no properties', function () { + assert.deepEqual( + convertMongoDBJSONSchemaToSimplified({ bsonType: 'object' }), + {}, + ); + assert.deepEqual(convertMongoDBJSONSchemaToSimplified({}), {}); + }); + + it('uses null-prototype field maps, as inference does', function () { + // Matches `simplifiedSchema` in schema-analyzer.ts, so that field names + // like `__proto__` or `constructor` cannot collide with object internals. + const result = convertMongoDBJSONSchemaToSimplified({ + bsonType: 'object', + properties: { + nested: { + bsonType: 'object', + properties: { a: { bsonType: 'int' } }, + }, + }, + }); + + assert.strictEqual(Object.getPrototypeOf(result), null); + const nested = result.nested.types[0] as { + fields: Record; + }; + assert.strictEqual(Object.getPrototypeOf(nested.fields), null); + }); + + it('handles a field literally named __proto__', function () { + // Built via JSON.parse because `__proto__` in an object literal invokes + // the prototype setter instead of creating an own property. A validator + // read off a collection arrives as an own property, as here. + const result = convertMongoDBJSONSchemaToSimplified( + JSON.parse( + '{"bsonType":"object","properties":{"__proto__":{"bsonType":"string"}}}', + ) as JSONSchema, + ); + + assert.deepEqual(Object.keys(result), ['__proto__']); + assert.deepEqual( + Object.getOwnPropertyDescriptor(result, '__proto__')?.value, + { types: [{ bsonType: 'String' }] }, + ); + }); + }); + + describe('arrays', function () { + it('reads member types from a single items schema', function () { + const result = convertMongoDBJSONSchemaToSimplified({ + bsonType: 'object', + properties: { a: { bsonType: 'array', items: { bsonType: 'string' } } }, + }); + + assert.deepEqual(result, { + a: { types: [{ bsonType: 'Array', types: [{ bsonType: 'String' }] }] }, + }); + }); + + it('unions member types across a closed tuple items form', function () { + const result = convertMongoDBJSONSchemaToSimplified({ + bsonType: 'object', + properties: { + a: { + bsonType: 'array', + items: [{ bsonType: 'string' }, { bsonType: 'int' }], + additionalItems: false, + }, + }, + }); + + assert.deepEqual(result, { + a: { + types: [ + { + bsonType: 'Array', + types: [{ bsonType: 'String' }, { bsonType: 'Int32' }], + }, + ], + }, + }); + }); + + it('leaves members unconstrained for a tuple items form open to additional items', function () { + const result = convertMongoDBJSONSchemaToSimplified({ + bsonType: 'object', + properties: { + implicit: { + bsonType: 'array', + items: [{ bsonType: 'string' }, { bsonType: 'int' }], + }, + explicit: { + bsonType: 'array', + items: [{ bsonType: 'string' }], + additionalItems: true, + }, + }, + }); + + assert.deepEqual(result, { + implicit: { types: [{ bsonType: 'Array', types: [] }] }, + explicit: { types: [{ bsonType: 'Array', types: [] }] }, + }); + }); + + it('emits an empty member type list for an array with no items', function () { + const result = convertMongoDBJSONSchemaToSimplified({ + bsonType: 'object', + properties: { a: { bsonType: 'array' } }, + }); + + assert.deepEqual(result, { + a: { types: [{ bsonType: 'Array', types: [] }] }, + }); + }); + + it('recurses through nested arrays of documents', function () { + const result = convertMongoDBJSONSchemaToSimplified({ + bsonType: 'object', + properties: { + matrix: { + bsonType: 'array', + items: { + bsonType: 'array', + items: { + bsonType: 'object', + properties: { x: { bsonType: 'int' } }, + }, + }, + }, + }, + }); + + assert.deepEqual(result, { + matrix: { + types: [ + { + bsonType: 'Array', + types: [ + { + bsonType: 'Array', + types: [ + { + bsonType: 'Document', + fields: { x: { types: [{ bsonType: 'Int32' }] } }, + }, + ], + }, + ], + }, + ], + }, + }); + }); + }); + + describe('unions', function () { + it('flattens anyOf into a type union', function () { + const result = convertMongoDBJSONSchemaToSimplified({ + bsonType: 'object', + properties: { + n: { anyOf: [{ bsonType: 'int' }, { bsonType: 'long' }] }, + }, + }); + + assert.deepEqual(result, { + n: { types: [{ bsonType: 'Int32' }, { bsonType: 'Long' }] }, + }); + }); + + it('treats oneOf as a union', function () { + const result = convertMongoDBJSONSchemaToSimplified({ + bsonType: 'object', + properties: { + n: { oneOf: [{ bsonType: 'int' }, { bsonType: 'string' }] }, + }, + }); + + assert.deepEqual(result, { + n: { types: [{ bsonType: 'Int32' }, { bsonType: 'String' }] }, + }); + }); + + it('intersects allOf branches', function () { + const result = convertMongoDBJSONSchemaToSimplified({ + bsonType: 'object', + properties: { + n: { + allOf: [ + { bsonType: ['int', 'string', 'null'] }, + { bsonType: ['string', 'int'] }, + { minimum: 0 }, + ], + }, + }, + }); + + assert.deepEqual(result, { + n: { types: [{ bsonType: 'Int32' }, { bsonType: 'String' }] }, + }); + }); + + it('omits a field whose allOf branches permit no common type', function () { + const result = convertMongoDBJSONSchemaToSimplified({ + bsonType: 'object', + properties: { + kept: { bsonType: 'int' }, + n: { allOf: [{ bsonType: 'int' }, { bsonType: 'string' }] }, + }, + }); + + assert.deepEqual(Object.keys(result), ['kept']); + }); + + it('intersects the fields of allOf document branches', function () { + const result = convertMongoDBJSONSchemaToSimplified({ + bsonType: 'object', + properties: { + d: { + allOf: [ + { + bsonType: 'object', + properties: { + a: { bsonType: ['int', 'null'] }, + b: { bsonType: 'string' }, + }, + }, + { + bsonType: 'object', + properties: { + a: { bsonType: ['int', 'long'] }, + c: { bsonType: 'bool' }, + }, + }, + ], + }, + }, + }); + + assert.deepEqual(result, { + d: { + types: [ + { + bsonType: 'Document', + fields: { + a: { types: [{ bsonType: 'Int32' }] }, + b: { types: [{ bsonType: 'String' }] }, + c: { types: [{ bsonType: 'Boolean' }] }, + }, + }, + ], + }, + }); + }); + + it('intersects the member types of allOf array branches', function () { + const result = convertMongoDBJSONSchemaToSimplified({ + bsonType: 'object', + properties: { + a: { + allOf: [ + { bsonType: 'array', items: { bsonType: ['int', 'string'] } }, + { bsonType: 'array', items: { bsonType: ['int', 'null'] } }, + { bsonType: 'array' }, + ], + }, + }, + }); + + assert.deepEqual(result, { + a: { types: [{ bsonType: 'Array', types: [{ bsonType: 'Int32' }] }] }, + }); + }); + + it('merges the field maps of two document branches into one Document type', function () { + const result = convertMongoDBJSONSchemaToSimplified({ + bsonType: 'object', + properties: { + d: { + anyOf: [ + { bsonType: 'object', properties: { a: { bsonType: 'int' } } }, + { bsonType: 'object', properties: { b: { bsonType: 'string' } } }, + ], + }, + }, + }); + + assert.deepEqual(result, { + d: { + types: [ + { + bsonType: 'Document', + fields: { + a: { types: [{ bsonType: 'Int32' }] }, + b: { types: [{ bsonType: 'String' }] }, + }, + }, + ], + }, + }); + }); + + it('merges the type unions of a field present in both document branches', function () { + const result = convertMongoDBJSONSchemaToSimplified({ + bsonType: 'object', + properties: { + d: { + anyOf: [ + { bsonType: 'object', properties: { a: { bsonType: 'int' } } }, + { bsonType: 'object', properties: { a: { bsonType: 'string' } } }, + ], + }, + }, + }); + + assert.deepEqual(result, { + d: { + types: [ + { + bsonType: 'Document', + fields: { + a: { types: [{ bsonType: 'Int32' }, { bsonType: 'String' }] }, + }, + }, + ], + }, + }); + }); + + it('keeps array members unconstrained when any union branch leaves them so', function () { + const result = convertMongoDBJSONSchemaToSimplified({ + bsonType: 'object', + properties: { + first: { + anyOf: [ + { bsonType: 'array' }, + { bsonType: 'array', items: { bsonType: 'string' } }, + ], + }, + last: { + anyOf: [ + { bsonType: 'array', items: { bsonType: 'string' } }, + { bsonType: 'array' }, + ], + }, + }, + }); + + assert.deepEqual(result, { + first: { types: [{ bsonType: 'Array', types: [] }] }, + last: { types: [{ bsonType: 'Array', types: [] }] }, + }); + }); + + it('merges the member types of two array branches into one Array type', function () { + const result = convertMongoDBJSONSchemaToSimplified({ + bsonType: 'object', + properties: { + a: { + anyOf: [ + { bsonType: 'array', items: { bsonType: 'int' } }, + { bsonType: 'array', items: { bsonType: 'string' } }, + ], + }, + }, + }); + + assert.deepEqual(result, { + a: { + types: [ + { + bsonType: 'Array', + types: [{ bsonType: 'Int32' }, { bsonType: 'String' }], + }, + ], + }, + }); + }); + + it('intersects an anyOf with a bsonType at the same position', function () { + const result = convertMongoDBJSONSchemaToSimplified({ + bsonType: 'object', + properties: { + f: { + bsonType: ['null', 'int'], + anyOf: [{ bsonType: 'string' }, { bsonType: 'int' }], + }, + }, + }); + + assert.deepEqual(result, { f: { types: [{ bsonType: 'Int32' }] } }); + }); + + it('intersects anyOf and oneOf at the same position', function () { + const result = convertMongoDBJSONSchemaToSimplified({ + bsonType: 'object', + properties: { + f: { + anyOf: [{ bsonType: 'string' }, { bsonType: 'int' }], + oneOf: [{ bsonType: 'int' }, { bsonType: 'null' }], + }, + }, + }); + + assert.deepEqual(result, { f: { types: [{ bsonType: 'Int32' }] } }); + }); + + it('treats an anyOf with an unconstrained branch as no constraint', function () { + const result = convertMongoDBJSONSchemaToSimplified({ + bsonType: 'object', + properties: { + f: { anyOf: [{ bsonType: 'int' }, { minimum: 5 }] }, + g: { + bsonType: ['int', 'string'], + anyOf: [{ bsonType: 'int' }, { minimum: 5 }], + }, + }, + }); + + assert.deepEqual(Object.keys(result), ['g']); + assert.deepEqual(result.g, { + types: [{ bsonType: 'Int32' }, { bsonType: 'String' }], + }); + }); + + it('recurses into unions nested inside array items', function () { + const result = convertMongoDBJSONSchemaToSimplified({ + bsonType: 'object', + properties: { + a: { + bsonType: 'array', + items: { anyOf: [{ bsonType: 'int' }, { bsonType: 'string' }] }, + }, + }, + }); + + assert.deepEqual(result, { + a: { + types: [ + { + bsonType: 'Array', + types: [{ bsonType: 'Int32' }, { bsonType: 'String' }], + }, + ], + }, + }); + }); + }); + + describe('constructs that carry no type information', function () { + it('omits a field whose subschema yields no types', function () { + const result = convertMongoDBJSONSchemaToSimplified({ + bsonType: 'object', + properties: { + kept: { bsonType: 'string' }, + enumOnly: { enum: ['a', 'b'] }, + rangeOnly: { minimum: 0, maximum: 10 }, + patternOnly: { pattern: '^a' }, + empty: {}, + notOnly: { not: { bsonType: 'string' } }, + }, + }); + + // Asserted explicitly rather than via deepEqual, which treats an + // `undefined`-valued key as absent. + assert.deepEqual(Object.keys(result), ['kept']); + assert.deepEqual(result, { + kept: { types: [{ bsonType: 'String' }] }, + }); + }); + + it('ignores value constraints alongside a bsonType', function () { + const result = convertMongoDBJSONSchemaToSimplified({ + bsonType: 'object', + properties: { + s: { + bsonType: 'string', + minLength: 1, + maxLength: 5, + pattern: '^a', + enum: ['aa'], + }, + n: { bsonType: 'int', minimum: 0, maximum: 10, multipleOf: 2 }, + }, + }); + + assert.deepEqual(result, { + s: { types: [{ bsonType: 'String' }] }, + n: { types: [{ bsonType: 'Int32' }] }, + }); + }); + + it('ignores required, title and description', function () { + const result = convertMongoDBJSONSchemaToSimplified({ + bsonType: 'object', + title: 'A thing', + description: 'described', + required: ['a', 'missingFromProperties'], + properties: { + a: { bsonType: 'string', title: 'A', description: 'd' }, + }, + }); + + assert.deepEqual(result, { a: { types: [{ bsonType: 'String' }] } }); + }); + + it('ignores patternProperties, whose field names are unknowable', function () { + const result = convertMongoDBJSONSchemaToSimplified({ + bsonType: 'object', + properties: { + tags: { + bsonType: 'object', + patternProperties: { '^t': { bsonType: 'string' } }, + }, + }, + }); + + assert.deepEqual(result, { + tags: { types: [{ bsonType: 'Document', fields: {} }] }, + }); + }); + + it('ignores additionalProperties', function () { + const result = convertMongoDBJSONSchemaToSimplified({ + bsonType: 'object', + properties: { + o: { + bsonType: 'object', + properties: { a: { bsonType: 'int' } }, + additionalProperties: false, + }, + }, + }); + + assert.deepEqual(result, { + o: { + types: [ + { + bsonType: 'Document', + fields: { a: { types: [{ bsonType: 'Int32' }] } }, + }, + ], + }, + }); + }); + }); + + describe('root', function () { + it('reads the root as a document when it is untyped', function () { + const result = convertMongoDBJSONSchemaToSimplified({ + properties: { a: { bsonType: 'int' } }, + }); + + assert.deepEqual(result, { a: { types: [{ bsonType: 'Int32' }] } }); + }); + + it('merges the properties of root anyOf branches', function () { + const result = convertMongoDBJSONSchemaToSimplified({ + anyOf: [ + { + properties: { + kind: { bsonType: 'string' }, + a: { bsonType: 'int' }, + }, + }, + { + properties: { + kind: { bsonType: 'string' }, + b: { bsonType: 'string' }, + }, + }, + ], + }); + + assert.deepEqual(result, { + kind: { types: [{ bsonType: 'String' }] }, + a: { types: [{ bsonType: 'Int32' }] }, + b: { types: [{ bsonType: 'String' }] }, + }); + }); + + it('merges root allOf branches into the root properties', function () { + const result = convertMongoDBJSONSchemaToSimplified({ + bsonType: 'object', + properties: { a: { bsonType: ['int', 'null'] } }, + allOf: [ + { properties: { b: { bsonType: 'string' } } }, + { + bsonType: 'object', + properties: { a: { bsonType: ['int', 'long'] } }, + }, + ], + }); + + assert.deepEqual(result, { + a: { types: [{ bsonType: 'Int32' }] }, + b: { types: [{ bsonType: 'String' }] }, + }); + }); + + it('keeps a null-prototype result when built from branches', function () { + const result = convertMongoDBJSONSchemaToSimplified({ + oneOf: [{ properties: { a: { bsonType: 'int' } } }], + }); + + assert.strictEqual(Object.getPrototypeOf(result), null); + }); + }); + + describe('untyped subschemas', function () { + it('implies a Document from properties', function () { + const result = convertMongoDBJSONSchemaToSimplified({ + bsonType: 'object', + properties: { + addr: { properties: { city: { bsonType: 'string' } } }, + }, + }); + + assert.deepEqual(result, { + addr: { + types: [ + { + bsonType: 'Document', + fields: { city: { types: [{ bsonType: 'String' }] } }, + }, + ], + }, + }); + }); + + it('implies an Array from items', function () { + const result = convertMongoDBJSONSchemaToSimplified({ + bsonType: 'object', + properties: { tags: { items: { bsonType: 'string' } } }, + }); + + assert.deepEqual(result, { + tags: { + types: [{ bsonType: 'Array', types: [{ bsonType: 'String' }] }], + }, + }); + }); + + it('merges untyped branch properties into the parent Document', function () { + const result = convertMongoDBJSONSchemaToSimplified({ + bsonType: 'object', + properties: { + d: { + bsonType: 'object', + properties: { shared: { bsonType: 'bool' } }, + oneOf: [ + { properties: { a: { bsonType: 'int' } } }, + { properties: { b: { bsonType: 'string' } } }, + ], + }, + }, + }); + + assert.deepEqual(result, { + d: { + types: [ + { + bsonType: 'Document', + fields: { + shared: { types: [{ bsonType: 'Boolean' }] }, + a: { types: [{ bsonType: 'Int32' }] }, + b: { types: [{ bsonType: 'String' }] }, + }, + }, + ], + }, + }); + }); + + it('does not invent a Document from an untyped branch of a scalar', function () { + const result = convertMongoDBJSONSchemaToSimplified({ + bsonType: 'object', + properties: { + s: { + bsonType: 'string', + anyOf: [{ properties: { a: { bsonType: 'int' } } }], + }, + }, + }); + + assert.deepEqual(result, { s: { types: [{ bsonType: 'String' }] } }); + }); + + it('refines only the matching type of a multi-typed parent', function () { + const result = convertMongoDBJSONSchemaToSimplified({ + bsonType: 'object', + properties: { + f: { + bsonType: ['object', 'null'], + allOf: [{ properties: { a: { bsonType: 'int' } } }], + }, + }, + }); + + assert.deepEqual(result, { + f: { + types: [ + { + bsonType: 'Document', + fields: { a: { types: [{ bsonType: 'Int32' }] } }, + }, + { bsonType: 'Null' }, + ], + }, + }); + }); + }); + + describe('validator-specific constructs', function () { + it('maps CSFLE encrypt fields to Binary', function () { + const result = convertMongoDBJSONSchemaToSimplified({ + bsonType: 'object', + properties: { + ssn: { + encrypt: { + bsonType: 'string', + keyId: [], + algorithm: 'AEAD_AES_256_CBC_HMAC_SHA_512-Deterministic', + }, + }, + }, + }); + + assert.deepEqual(result, { ssn: { types: [{ bsonType: 'Binary' }] } }); + }); + + it('includes additionalItems in the array member types', function () { + const result = convertMongoDBJSONSchemaToSimplified({ + bsonType: 'object', + properties: { + t: { + bsonType: 'array', + items: [{ bsonType: 'int' }], + additionalItems: { bsonType: 'string' }, + }, + }, + }); + + assert.deepEqual(result, { + t: { + types: [ + { + bsonType: 'Array', + types: [{ bsonType: 'Int32' }, { bsonType: 'String' }], + }, + ], + }, + }); + }); + + it('ignores additionalItems for a single-schema items', function () { + const result = convertMongoDBJSONSchemaToSimplified({ + bsonType: 'object', + properties: { + t: { + bsonType: 'array', + items: { bsonType: 'int' }, + additionalItems: { bsonType: 'string' }, + }, + }, + }); + + assert.deepEqual(result, { + t: { types: [{ bsonType: 'Array', types: [{ bsonType: 'Int32' }] }] }, + }); + }); + + it('does not imply an Array from additionalItems alone', function () { + const result = convertMongoDBJSONSchemaToSimplified({ + bsonType: 'object', + properties: { + t: { additionalItems: { bsonType: 'string' } }, + }, + }); + + assert.deepEqual(result, {}); + }); + + it('ignores a boolean additionalItems', function () { + const result = convertMongoDBJSONSchemaToSimplified({ + bsonType: 'object', + properties: { + t: { + bsonType: 'array', + items: [{ bsonType: 'int' }], + additionalItems: false, + }, + }, + }); + + assert.deepEqual(result, { + t: { types: [{ bsonType: 'Array', types: [{ bsonType: 'Int32' }] }] }, + }); + }); + + it('maps a document requiring $ref and $id to DBRef, as js-bson deserialises it', function () { + const dbRefShape = { + bsonType: 'object', + required: ['$ref', '$id'], + properties: { + $ref: { bsonType: 'string' }, + $id: { bsonType: 'objectId' }, + }, + }; + const result = convertMongoDBJSONSchemaToSimplified({ + bsonType: 'object', + properties: { + ref: dbRefShape, + refs: { bsonType: 'array', items: dbRefShape }, + both: { anyOf: [dbRefShape, { bsonType: 'dbPointer' }] }, + }, + }); + + assert.deepEqual(result, { + ref: { types: [{ bsonType: 'DBRef' }] }, + refs: { + types: [{ bsonType: 'Array', types: [{ bsonType: 'DBRef' }] }], + }, + both: { types: [{ bsonType: 'DBRef' }] }, + }); + }); + + it('reports both Document and DBRef when $ref and $id are optional', function () { + const result = convertMongoDBJSONSchemaToSimplified({ + bsonType: 'object', + properties: { + r: { + bsonType: 'object', + required: ['$ref'], + properties: { + $ref: { bsonType: 'string' }, + $id: { bsonType: 'objectId' }, + }, + }, + }, + }); + + assert.deepEqual(result, { + r: { + types: [ + { + bsonType: 'Document', + fields: { + $ref: { types: [{ bsonType: 'String' }] }, + $id: { types: [{ bsonType: 'ObjectId' }] }, + }, + }, + { bsonType: 'DBRef' }, + ], + }, + }); + }); + + it('narrows a Document to DBRef when an allOf branch requires the DBRef shape', function () { + const result = convertMongoDBJSONSchemaToSimplified({ + bsonType: 'object', + properties: { + r: { + bsonType: 'object', + allOf: [ + { + required: ['$ref', '$id'], + properties: { + $ref: { bsonType: 'string' }, + $id: { bsonType: 'objectId' }, + }, + }, + ], + }, + }, + }); + + assert.deepEqual(result, { r: { types: [{ bsonType: 'DBRef' }] } }); + }); + + it('keeps a document with only $ref as a Document', function () { + const result = convertMongoDBJSONSchemaToSimplified({ + bsonType: 'object', + properties: { + r: { + bsonType: 'object', + properties: { $ref: { bsonType: 'string' } }, + }, + }, + }); + + assert.deepEqual(result, { + r: { + types: [ + { + bsonType: 'Document', + fields: { $ref: { types: [{ bsonType: 'String' }] } }, + }, + ], + }, + }); + }); + }); + + describe('robustness', function () { + it('falls back to type when every bsonType alias is unrecognised', function () { + const result = convertMongoDBJSONSchemaToSimplified({ + bsonType: 'object', + properties: { f: { bsonType: 'nonsense', type: 'string' } }, + }); + + assert.deepEqual(result, { f: { types: [{ bsonType: 'String' }] } }); + }); + + it('does not resolve aliases to Object.prototype members', function () { + const result = convertMongoDBJSONSchemaToSimplified({ + bsonType: 'object', + properties: { + a: { bsonType: 'constructor' }, + b: { bsonType: ['toString', 'int'] }, + }, + }); + + assert.deepEqual(Object.keys(result), ['b']); + assert.deepEqual(result.b, { types: [{ bsonType: 'Int32' }] }); + }); + + it('skips malformed subschemas without throwing', function () { + const result = convertMongoDBJSONSchemaToSimplified( + JSON.parse( + JSON.stringify({ + bsonType: 'object', + properties: { + nullSchema: null, + boolSchema: true, + nullBranch: { anyOf: [null, { bsonType: 'int' }] }, + objectAnyOf: { bsonType: 'string', anyOf: { bsonType: 'int' } }, + badItems: { + bsonType: 'array', + items: [null, 'x', { bsonType: 'int' }], + additionalItems: false, + }, + badProperties: { bsonType: 'object', properties: ['a'] }, + }, + }), + ) as JSONSchema, + ); + + assert.deepEqual(result, { + nullBranch: { types: [{ bsonType: 'Int32' }] }, + objectAnyOf: { types: [{ bsonType: 'String' }] }, + badItems: { + types: [{ bsonType: 'Array', types: [{ bsonType: 'Int32' }] }], + }, + badProperties: { types: [{ bsonType: 'Document', fields: {} }] }, + }); + }); + + it('tolerates a non-object root', function () { + assert.deepEqual( + convertMongoDBJSONSchemaToSimplified(null as unknown as JSONSchema), + {}, + ); + }); + }); + + // The load-bearing test: pins this converter's output vocabulary to the one + // inference produces, so a consumer reading a validator sees the same type + // names it would have got from sampling documents. + describe('round trip through inference', function () { + let inferred: SimplifiedSchema; + let viaValidator: SimplifiedSchema; + + before(async function () { + const accessor = await analyzeDocuments([allBSONTypesDoc]); + inferred = await getSimplifiedSchema([allBSONTypesDoc]); + viaValidator = convertMongoDBJSONSchemaToSimplified( + await accessor.getMongoDBJsonSchema(), + ); + }); + + it('describes exactly the same fields', function () { + assert.deepEqual( + Object.keys(viaValidator).sort(), + Object.keys(inferred).sort(), + ); + }); + + it('reproduces the inferred schema, bar one irreducible difference', function () { + // `array: [1, 2, 3]` holds plain JS numbers, which inference names + // `Number`. internalToMongoDB collapses both `Number` and `Double` onto + // `double`, so this cannot be inverted; `Double` is what a validator + // saying `double` means, and is what comes back. + const expected = { + ...inferred, + array: { + types: [{ bsonType: 'Array', types: [{ bsonType: 'Double' }] }], + }, + }; + + assert.deepEqual(viaValidator, expected); + }); + + it('round trips DBRef, which internalToMongoDB writes as dbPointer', function () { + assert.deepEqual(inferred.dbRef, { types: [{ bsonType: 'DBRef' }] }); + assert.deepEqual(viaValidator.dbRef, { types: [{ bsonType: 'DBRef' }] }); + }); + }); +}); diff --git a/packages/mongodb-schema/src/schema-converters/mongodb-to-simplified.ts b/packages/mongodb-schema/src/schema-converters/mongodb-to-simplified.ts new file mode 100644 index 000000000..f1388db24 --- /dev/null +++ b/packages/mongodb-schema/src/schema-converters/mongodb-to-simplified.ts @@ -0,0 +1,456 @@ +/** + * Transforms MongoDB's $jsonSchema - as set in a collection validator - to the + * simplified schema. + * + * This deliberately does not go via the internal schema: that representation is + * probabilistic (count, probability, hasDuplicates, sample values) and a + * validator contains none of those, so populating it would mean inventing + * statistics. The simplified schema carries types only, which a validator can + * describe honestly. + * + * A validator constrains documents rather than describing them, so the mapping + * is intentionally lossy: value-level constraints (enum, minimum, pattern, ...) + * are ignored, as are `patternProperties` and `additionalProperties`. Nothing + * throws - constructs that carry no type information, and malformed subschemas, + * simply contribute nothing. + */ +import type { + SchemaBSONType, + SimplifiedSchema, + SimplifiedSchemaArrayType, + SimplifiedSchemaDocumentType, + SimplifiedSchemaType, +} from '../schema-analyzer'; +import type { JSONSchema, MongoDBJSONSchema } from '../types'; + +const NUMERIC_TYPES: SchemaBSONType[] = [ + 'Int32', + 'Long', + 'Double', + 'Decimal128', +]; + +/** + * BSON type aliases accepted by $jsonSchema's `bsonType`, mapped to the type + * names schema inference produces. Note this is not a straight inversion of + * `InternalTypeToBsonTypeMap`, which is many-to-one (both `Number` and `Double` + * map to `double`, both `RegExp` and `BSONRegExp` to `regex`), so where that map + * collapses two names we pick the BSON-wrapper one a validator would mean. + */ +// The rule's `__proto__: null` fix does not typecheck against `Record`. +// Lookups take keys from the validator, so they go through `mapAliases`, +// which only accepts own properties. +// eslint-disable-next-line @mongodb-js/devtools/no-plain-object-records +export const BSONTypeAliasToSimplifiedType: Record< + string, + SchemaBSONType | SchemaBSONType[] +> = { + double: 'Double', + string: 'String', + object: 'Document', + array: 'Array', + binData: 'Binary', + undefined: 'Undefined', + objectId: 'ObjectId', + bool: 'Boolean', + date: 'Date', + null: 'Null', + regex: 'BSONRegExp', + javascript: 'Code', + javascriptWithScope: 'CodeWScope', + symbol: 'BSONSymbol', + int: 'Int32', + timestamp: 'Timestamp', + long: 'Long', + decimal: 'Decimal128', + minKey: 'MinKey', + maxKey: 'MaxKey', + // The inverse of `InternalTypeToBsonTypeMap`'s `DBRef: 'dbPointer'`. + dbPointer: 'DBRef', + // `number` accepts every BSON numeric type. + number: NUMERIC_TYPES, +}; + +/** + * $jsonSchema also accepts the standard JSON Schema `type` keyword, used as a + * fallback when `bsonType` is absent. + */ +// The rule's `__proto__: null` fix does not typecheck against `Record`. +// Lookups take keys from the validator, so they go through `mapAliases`, +// which only accepts own properties. +// eslint-disable-next-line @mongodb-js/devtools/no-plain-object-records +export const JSONSchemaTypeToSimplifiedType: Record< + string, + SchemaBSONType | SchemaBSONType[] +> = { + object: 'Document', + array: 'Array', + string: 'String', + number: NUMERIC_TYPES, + boolean: 'Boolean', + null: 'Null', + // MongoDB rejects `type: 'integer'` inside $jsonSchema, but accepting it here + // costs nothing and makes this usable for plain JSON Schema input. + integer: ['Int32', 'Long'], +}; + +const UNION_KEYWORDS = ['anyOf', 'oneOf'] as const; + +/** + * The types a subschema permits, or `undefined` when it says nothing about + * types and so permits any. An empty array means it permits no value at all. + */ +type TypeSet = SimplifiedSchemaType[] | undefined; + +function toArray(value: T | T[] | undefined): T[] { + if (value === undefined) return []; + return Array.isArray(value) ? value : [value]; +} + +/** + * Validators are checked by the server when set, but this is also usable on + * arbitrary input, so every subschema is checked before it is read. + */ +function isSchema(value: unknown): value is MongoDBJSONSchema { + return typeof value === 'object' && value !== null && !Array.isArray(value); +} + +function isDocumentType( + type: SimplifiedSchemaType, +): type is SimplifiedSchemaDocumentType { + return type.bsonType === 'Document'; +} + +function isArrayType( + type: SimplifiedSchemaType, +): type is SimplifiedSchemaArrayType { + return type.bsonType === 'Array'; +} + +/** + * Adds a type to a union, merging into the existing entry if one already has + * this bsonType. Inference emits at most one entry per bsonType per field, so + * collapsing duplicates keeps the output shape identical to inferred schemas - + * two `object` branches of an anyOf become one Document with merged fields. + */ +function mergeTypeInto( + types: SimplifiedSchemaType[], + incoming: SimplifiedSchemaType, +): void { + const existing = types.find((type) => type.bsonType === incoming.bsonType); + if (!existing) { + types.push(incoming); + return; + } + + if (isDocumentType(existing) && isDocumentType(incoming)) { + mergeFieldsInto(existing.fields, incoming.fields); + } else if (isArrayType(existing) && isArrayType(incoming)) { + // An empty member list means unconstrained members, which absorbs any + // constrained branch in a union. + if (existing.types.length === 0) return; + if (incoming.types.length === 0) { + existing.types = []; + return; + } + mergeTypesInto(existing.types, incoming.types); + } +} + +function mergeTypesInto( + types: SimplifiedSchemaType[], + incoming: SimplifiedSchemaType[], +): void { + for (const type of incoming) { + mergeTypeInto(types, type); + } +} + +function mergeFieldsInto( + fields: SimplifiedSchema, + incoming: SimplifiedSchema, +): void { + for (const name of Object.keys(incoming)) { + const existing = fields[name]; + if (!existing) { + fields[name] = incoming[name]; + continue; + } + mergeTypesInto(existing.types, incoming[name].types); + } +} + +function mapAliases( + value: unknown, + map: Record, +): SchemaBSONType[] { + const types: SchemaBSONType[] = []; + for (const key of toArray(value)) { + // Own-property check, so that aliases like `constructor` do not resolve to + // `Object.prototype` members. + if ( + typeof key === 'string' && + Object.prototype.hasOwnProperty.call(map, key) + ) { + types.push(...toArray(map[key])); + } + } + return types; +} + +/** + * The types a subschema names outright. Unrecognised aliases contribute + * nothing, so a `bsonType` made up only of those falls through to `type`. + */ +function explicitTypes(schema: MongoDBJSONSchema): SchemaBSONType[] { + const bsonTypes = mapAliases(schema.bsonType, BSONTypeAliasToSimplifiedType); + if (bsonTypes.length > 0) return bsonTypes; + + const jsonTypes = mapAliases(schema.type, JSONSchemaTypeToSimplifiedType); + if (jsonTypes.length > 0) return jsonTypes; + + // CSFLE fields are declared with `encrypt` in place of `bsonType`. The value + // is stored as BinData subtype 6, which is what inference would observe. + if (isSchema(schema.encrypt)) return ['Binary']; + + return []; +} + +/** + * The types an untyped subschema's structural keywords imply. Strictly, JSON + * Schema applies `properties` only if the value is an object, but a validator + * author leaving out `bsonType: 'object'` almost always means one. + */ +function impliedTypes(schema: MongoDBJSONSchema): SchemaBSONType[] { + const types: SchemaBSONType[] = []; + if (isSchema(schema.properties) || isSchema(schema.patternProperties)) { + types.push('Document'); + } + // Not `additionalItems`, which has no effect without a tuple `items`. + if (schema.items !== undefined) { + types.push('Array'); + } + return types; +} + +/** + * Whether two types can describe the same value. A DBRef is a document as far + * as the validator is concerned, so it intersects with Document. + */ +function intersectKind( + a: SchemaBSONType, + b: SchemaBSONType, +): SchemaBSONType | undefined { + if (a === b) return a; + if ( + (a === 'DBRef' && b === 'Document') || + (a === 'Document' && b === 'DBRef') + ) { + return 'DBRef'; + } + return undefined; +} + +/** + * The values both type sets permit. Matching documents keep the fields of + * both sides, with the types of a field present in both intersected; a field + * no value can satisfy is dropped. Matching arrays intersect their members. + */ +function intersectTypes(a: TypeSet, b: TypeSet): TypeSet { + if (a === undefined) return b; + if (b === undefined) return a; + + const types: SimplifiedSchemaType[] = []; + for (const left of a) { + for (const right of b) { + const bsonType = intersectKind(left.bsonType, right.bsonType); + if (bsonType === undefined) continue; + + if (bsonType === 'Document') { + mergeTypeInto(types, { + bsonType, + fields: intersectFields( + (left as SimplifiedSchemaDocumentType).fields, + (right as SimplifiedSchemaDocumentType).fields, + ), + }); + } else if (bsonType === 'Array') { + const leftMembers = (left as SimplifiedSchemaArrayType).types; + const rightMembers = (right as SimplifiedSchemaArrayType).types; + mergeTypeInto(types, { + bsonType, + // An empty member list means the members are unconstrained. + types: + leftMembers.length === 0 + ? rightMembers + : rightMembers.length === 0 + ? leftMembers + : (intersectTypes(leftMembers, rightMembers) ?? []), + }); + } else { + mergeTypeInto(types, { bsonType }); + } + } + } + return types; +} + +function intersectFields( + a: SimplifiedSchema, + b: SimplifiedSchema, +): SimplifiedSchema { + const fields: SimplifiedSchema = Object.create(null); + for (const name of Object.keys(a)) { + if (!(name in b)) { + fields[name] = a[name]; + continue; + } + const types = intersectTypes(a[name].types, b[name].types); + if (types && types.length > 0) fields[name] = { types }; + } + for (const name of Object.keys(b)) { + if (!(name in a)) fields[name] = b[name]; + } + return fields; +} + +/** + * The values any of the type sets permit. If one branch is unconstrained, so + * is the union. + */ +function unionTypes(sets: TypeSet[]): TypeSet { + const types: SimplifiedSchemaType[] = []; + for (const set of sets) { + if (set === undefined) return undefined; + mergeTypesInto(types, set); + } + return types; +} + +function isRequired(schema: MongoDBJSONSchema, name: string): boolean { + return Array.isArray(schema.required) && schema.required.includes(name); +} + +function buildTypes( + bsonType: SchemaBSONType, + schema: MongoDBJSONSchema, +): SimplifiedSchemaType[] { + if (bsonType === 'Document') { + const fields = collectFields(schema.properties); + if (!('$ref' in fields && '$id' in fields)) { + return [{ bsonType, fields }]; + } + // js-bson deserialises any embedded document with `$ref` and `$id` into a + // DBRef, so that is what inference reports for one. Unless the validator + // requires both, plain documents without them are permitted too. + if (isRequired(schema, '$ref') && isRequired(schema, '$id')) { + return [{ bsonType: 'DBRef' }]; + } + return [{ bsonType, fields }, { bsonType: 'DBRef' }]; + } + + if (bsonType === 'Array') { + // `additionalItems` only applies past a tuple `items` form, where it + // defaults to `true`: unless it is `false` or a schema, members past the + // tuple can be anything, and positions aren't representable here, so the + // whole member list is unconstrained. + if ( + Array.isArray(schema.items) && + schema.additionalItems !== false && + !isSchema(schema.additionalItems) + ) { + return [{ bsonType, types: [] }]; + } + const memberSchemas = ( + Array.isArray(schema.items) + ? [...schema.items, schema.additionalItems] + : [schema.items] + ).filter(isSchema); + const members = + memberSchemas.length > 0 + ? unionTypes(memberSchemas.map((items) => collectTypes(items))) + : undefined; + // An empty member list stands for unconstrained members. + return [{ bsonType, types: members ?? [] }]; + } + + return [{ bsonType }]; +} + +/** + * The types a single subschema permits. Keywords at the same level all apply, + * so `anyOf`/`oneOf` contribute the union of their branches intersected with + * the schema's own types, and each `allOf` branch is intersected in turn. + * + * `scope` is the types of the enclosing schema, when this is one of its + * branches. An untyped branch constrains those types rather than introducing + * its own, so `{ bsonType: 'object', oneOf: [{ properties }] }` merges the + * branch's properties into the parent's Document, and a `properties`-only + * branch of a `string` does not invent a Document. + */ +function collectTypes(schema: unknown, scope?: SchemaBSONType[]): TypeSet { + if (!isSchema(schema)) return undefined; + + const explicit = explicitTypes(schema); + const own = explicit.length > 0 ? explicit : (scope ?? impliedTypes(schema)); + + let types: TypeSet; + if (own.length > 0) { + types = []; + for (const bsonType of own) { + mergeTypesInto(types, buildTypes(bsonType, schema)); + } + } + + const branchScope = own.length > 0 ? own : undefined; + for (const keyword of UNION_KEYWORDS) { + // Malformed branches are skipped rather than read as permitting anything, + // and an empty branch list (itself invalid) as no constraint. + const branches: unknown[] = Array.isArray(schema[keyword]) + ? schema[keyword].filter(isSchema) + : []; + if (branches.length === 0) continue; + types = intersectTypes( + types, + unionTypes(branches.map((branch) => collectTypes(branch, branchScope))), + ); + } + + if (Array.isArray(schema.allOf)) { + for (const branch of schema.allOf) { + types = intersectTypes(types, collectTypes(branch, branchScope)); + } + } + + return types; +} + +function collectFields(properties: unknown): SimplifiedSchema { + // Null prototype, matching `simplifiedSchema` in schema-analyzer, so that + // field names like `__proto__` cannot collide with object internals. + const fields: SimplifiedSchema = Object.create(null); + if (!isSchema(properties)) return fields; + + const subschemas = properties as Record; + for (const name of Object.keys(subschemas)) { + const types = collectTypes(subschemas[name]); + // A field with no type information is omitted rather than emitted with an + // empty `types` array: inference cannot produce the latter, since a field + // appears there only because a value was observed for it. A field no value + // can satisfy is omitted too, since it can only ever be absent. + if (types && types.length > 0) { + fields[name] = { types }; + } + } + + return fields; +} + +export function convertMongoDBJSONSchemaToSimplified( + jsonSchema: JSONSchema, +): SimplifiedSchema { + // The root always describes a document, so it is read as one even when + // untyped - including when its shape lives entirely in union branches. + const root = collectTypes(jsonSchema, ['Document'])?.find(isDocumentType); + return root?.fields ?? Object.create(null); +} diff --git a/packages/mongodb-schema/src/types.ts b/packages/mongodb-schema/src/types.ts index d555a7640..ad5b4e814 100644 --- a/packages/mongodb-schema/src/types.ts +++ b/packages/mongodb-schema/src/types.ts @@ -11,6 +11,13 @@ export type MongoDBJSONSchema = Pick< properties?: Record; items?: MongoDBJSONSchema | MongoDBJSONSchema[]; anyOf?: MongoDBJSONSchema[]; + // Not produced by this library, but accepted by $jsonSchema validators. + type?: StandardJSONSchema['type']; + oneOf?: MongoDBJSONSchema[]; + allOf?: MongoDBJSONSchema[]; + patternProperties?: Record; + additionalItems?: boolean | MongoDBJSONSchema; + encrypt?: Record; }; export type ExpandedJSONSchema = StandardJSONSchema & { diff --git a/packages/mql-typescript/src/testGenerator/test-generator.ts b/packages/mql-typescript/src/testGenerator/test-generator.ts index 82839d62e..d94caa184 100644 --- a/packages/mql-typescript/src/testGenerator/test-generator.ts +++ b/packages/mql-typescript/src/testGenerator/test-generator.ts @@ -14,7 +14,7 @@ import * as bson from 'bson'; type TestType = NonNullable[number]; -type SchemaBSONType = SimplifiedSchemaBaseType['bsonType'] | 'Long' | 'Number'; +type SchemaBSONType = SimplifiedSchemaBaseType['bsonType'] | 'Number'; export class TestGenerator extends GeneratorBase { private schemaBsonTypeToTS(type: SchemaBSONType): string { @@ -36,7 +36,6 @@ export class TestGenerator extends GeneratorBase { case 'Int32': return 'bson.Int32 | number'; case 'Long': - case 'Int64': return 'bson.Long'; case 'MaxKey': return 'bson.MaxKey'; @@ -44,6 +43,8 @@ export class TestGenerator extends GeneratorBase { return 'bson.MinKey'; case 'Null': return 'null'; + case 'DBRef': + return 'bson.DBRef'; case 'ObjectId': return 'bson.ObjectId'; case 'BSONRegExp': diff --git a/packages/saslprep/src/code-points-data.ts b/packages/saslprep/src/code-points-data.ts index ad121e4c2..0e5fb08a7 100644 --- a/packages/saslprep/src/code-points-data.ts +++ b/packages/saslprep/src/code-points-data.ts @@ -2,7 +2,7 @@ import { gunzipSync } from 'zlib'; export default gunzipSync( Buffer.from( - 'H4sIAAAAAAACA+3dTYgcaRkA4LemO9Mhxm0FITnE9Cwr4jHgwgZ22B6YywqCJ0HQg5CL4sGTuOjCtGSF4CkHEW856MlTQHD3EJnWkU0Owh5VxE3LHlYQdNxd2U6mU59UV/d09fw4M2EySSXPAzNdP1/9fX/99bzVNZEN4jisRDulVFnQmLxm1aXF9Id/2/xMxNJ4XZlg576yuYlGt9gupV6xoFf8jhu9YvulVrFlp5XSx+lfvYhORGPXvqIRWSxERKtIm8bKFd10WNfKDS5Fo9jJWrq2+M2IlW+8uHgl/+BsROfPF4v5L7148Ur68Sha6dqZpYiVVy8tvLCWXo80Sf/lS89dGX2wHGvpzoXVn75/YWH5wmqe8uika82ViJXTy83Ve2k5Urozm38wm4/ls6t5uT6yfsTSJ7J3T0VKt8c5ExEXI8aFkH729c3eT+7EC6ca8cVULZUiYacX0R5PNWNxlh9L1y90q5kyzrpyy+9WcvOV6URntqw7La9sNVstXyczWVaWYbaaTYqzOHpr7pyiNT3/YzKuT63Z/FqKZlFTiuXtFM2vVOtIq7jiyKJbWZaOWD0euz0yoV2Z7kY0xq2x0YhfzVpmM5px9nTEH7JZ0ot5u39p0ma75Z472/s/H+2yr2inYyuq7fMvJivH2rM72N/Z3lyL31F2b1ya1P0zn816k2KP6JU9UzseucdQH5YqVeH/lFajSN2udg+TLJ9rksNxlvV2lki19rXKI43TPLejFu4ov7k3nMbhyhfY3Xb37f8BAGCf0eMTOH5szf154KmnNgKcnLb+Fzi2AfXktbN7fJelwTAiO/W5uQ2KINXRYu+znqo/WTAdLadURHmy3qciazd3bra4T3w16/f7t7Ms9U5gfJu10955sx1r3vmhBAAAAAAAgId20J1iZbDowNvIjuH427Gr5l/eiC+8OplZON8sVjx/qr9y+Pj+YRItT+NqAM+kkZs3AAAAAID6yfx1FwCAI97/dCh1/ub6SA0AAAAAAAAAgNoT/wcAAAAAAACA+hP/BwAAAAAAAID6E/8HAAAAAAAAgPoT/wcAAAAAAACA+hP/BwAAAAAAAID6E/8HAAAAAAAAgPoT/wcAAAAAAACA+hP/BwAAAAAAAID6E/8HAAAAAAAAgPoT/wcAAAAAAACA+hutp5SiQpYAAAAAAAAAQO2MIpZiT804flnAE2fhwjOeAZXr76kOAAAAAAAA8FjNf4N/l0NE3U/vuVQskLpSd4/Yh2xu9xTu0tFeeNYsLI2f/VMdNxTzj6Je9E/+6pp6Nn3awW3A54goe4Bss6v+PGsjQGMAAAAAAOBp5XEgwH6e7J7rwEQHRb/XvAMAAAAAAAA8yzoDeQDwVGjIAgAAAAAAAACoPfF/AAAAAAAAAKg/8X8AAAAAAAAAqD/xfwAAAAAAAACoP/F/AAAAAAAAAKg/8X8AAAAAAAAAqD/xfwAAAAAAAACoP/F/AAAAAAAAAKg/8X8AAAAAAAAAqD/xfwAAAAAAAACoP/F/AAAAAAAAAKg/8X8AAAAAAAAAqL/GSkSkClkCAAAAAAAAALXTSAAAAAAAAABA3Y1kAQAAAAAAAADUX8RSXZ9dsHC9+M8Fg2Ex/em1lAZpEBGttcrVjZqLEa+k0XpKw9mG4zWx4ukPUMhkAQAAAAAAABzBqbSe3//rXOS9HxGdo4TqR2XkutCdBu+LaPZw/lBbO7cbHnh2C7N7AIo4evEznllqLqWUp/LnYOtpM2bnOH66wI1+9GO4sOuISwv/TOlumu56FDv3NZhc4mR9v7zYIrafr40j/Cccvj9Xns3t3mu99E7qxUv3bqS0/ouNH/08++RGemfQ+nsx/5uNXsQPGulynPvv3ZTW37zd+1ovrqaYpP/122X6Xpx779Z3zr/3YOPKW1lkaRDf31pPaf3j/msRsVGkL+d/f+/m4sJsPm1cfSsr16e8m9Ldj/KsnyIuR3nXw83Is3EhxLd/2V773ks3m/cj/THKUummdP9qKhIOImuOU0Xjwb3y+oqt735rpTetVbF9n8R4x9crRfO77TKqVOZpDclv5bfK18lMnk+q0K18UpxF/RrGXE0Zxtqx3tWSj+vxbL4XaasfKb0dRbtLW73JsfPGg177H+OmGKlfvS1msllt7JEJm9XOJqXR+Fkfo1H66uy5H1v3Xx5+uJmGLw9jro2u7Loj4PnuR6+f+e3d261+eazNhzrL7X83MohoHpS4PddV8ki1it61//pw1g7z6p1U/26Nm2llST57B5rUvuG0XqSU/rPd7jYrqWcbd+beJQ77BgPMDwn37/8BAGCf0eMTOH4cPlufv9VGgJOzqf8Fjm1APXkd7B7f5dF57GPMaWy/MTvjvNvtXj6h8W2+GXvnzXaseeeHEgAAAAAAAB7aQXeKlcGiadBoEOeLb2dtpGOL2MyOtf391a3P/zD96c3JzIP3t4oV797vrh8+vn+YRL5bBuj/AQAAAABqJvfHXQAAHkX82zfXAQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAACeAgkAAAAAAAAAqLuRLAAAAAAAAACA2hv9D1iu/VAYaAYA', + 'H4sIAAAAAAACE+3dTYgcaRkA4LemO9Mhxm0FITnE9Cwr4jHgwgZ22B6YywqCJ0HQg5CL4sGTuOjCtGSF4CkHEW856MlTQHD3EJnWkU0Owh5VxE3LHlYQdNxd2U6mU59UV/d09fw4M2EySSXPAzNdP1/9fX/99bzVNZEN4jisRDulVFnQmLxm1aXF9Id/2/xMxNJ4XZlg576yuYlGt9gupV6xoFf8jhu9YvulVrFlp5XSx+lfvYhORGPXvqIRWSxERKtIm8bKFd10WNfKDS5Fo9jJWrq2+M2IlW+8uHgl/+BsROfPF4v5L7148Ur68Sha6dqZpYiVVy8tvLCWXo80Sf/lS89dGX2wHGvpzoXVn75/YWH5wmqe8uika82ViJXTy83Ve2k5Urozm38wm4/ls6t5uT6yfsTSJ7J3T0VKt8c5ExEXI8aFkH729c3eT+7EC6ca8cVULZUiYacX0R5PNWNxlh9L1y90q5kyzrpyy+9WcvOV6URntqw7La9sNVstXyczWVaWYbaaTYqzOHpr7pyiNT3/YzKuT63Z/FqKZlFTiuXtFM2vVOtIq7jiyKJbWZaOWD0euz0yoV2Z7kY0xq2x0YhfzVpmM5px9nTEH7JZ0ot5u39p0ma75Z472/s/H+2yr2inYyuq7fMvJivH2rM72N/Z3lyL31F2b1ya1P0zn816k2KP6JU9UzseucdQH5YqVeH/lFajSN2udg+TLJ9rksNxlvV2lki19rXKI43TPLejFu4ov7k3nMbhyhfY3Xb37f8BAGCf0eMTOH5szf154KmnNgKcnLb+Fzi2AfXktbN7fJelwTAiO/W5uQ2KINXRYu+znqo/WTAdLadURHmy3qciazd3bra4T3w16/f7t7Ms9U5gfJu10955sx1r3vmhBAAAAAAAgId20J1iZbDowNvIjuH427Gr5l/eiC+8OplZON8sVjx/qr9y+Pj+YRItT+NqAM+kkZs3AAAAAID6yfx1FwCAI97/dCh1/ub6SA0AAAAAAAAAgNoT/wcAAAAAAACA+hP/BwAAAAAAAID6E/8HAAAAAAAAgPoT/wcAAAAAAACA+hP/BwAAAAAAAID6E/8HAAAAAAAAgPoT/wcAAAAAAACA+hP/BwAAAAAAAID6E/8HAAAAAAAAgPoT/wcAAAAAAACA+hutp5SiQpYAAAAAAAAAQO2MIpZiT804flnAE2fhwjOeAZXr76kOAAAAAAAA8FjNf4N/l0NE3U/vuVQskLpSd4/Yh2xu9xTu0tFeeNYsLI2f/VMdNxTzj6Je9E/+6pp6Nn3awW3A54goe4Bss6v+PGsjQGMAAAAAAOBp5XEgwH6e7J7rwEQHRb/XvAMAAAAAAAA8yzoDeQDwVGjIAgAAAAAAAACoPfF/AAAAAAAAAKg/8X8AAAAAAAAAqD/xfwAAAAAAAACoP/F/AAAAAAAAAKg/8X8AAAAAAAAAqD/xfwAAAAAAAACoP/F/AAAAAAAAAKg/8X8AAAAAAAAAqD/xfwAAAAAAAACoP/F/AAAAAAAAAKg/8X8AAAAAAAAAqL/GSkSkClkCAAAAAAAAALXTSAAAAAAAAABA3Y1kAQAAAAAAAADUX8RSXZ9dsHC9+M8Fg2Ex/em1lAZpEBGttcrVjZqLEa+k0XpKw9mG4zWx4ukPUMhkAQAAAAAAABzBqbSe3//rXOS9HxGdo4TqR2XkutCdBu+LaPZw/lBbO7cbHnh2C7N7AIo4evEznllqLqWUp/LnYOtpM2bnOH66wI1+9GO4sOuISwv/TOlumu56FDv3NZhc4mR9v7zYIrafr40j/Cccvj9Xns3t3mu99E7qxUv3bqS0/ouNH/08++RGemfQ+nsx/5uNXsQPGulynPvv3ZTW37zd+1ovrqaYpP/122X6Xpx779Z3zr/3YOPKW1lkaRDf31pPaf3j/msRsVGkL+d/f+/m4sJsPm1cfSsr16e8m9Ldj/KsnyIuR3nXw83Is3EhxLd/2V773ks3m/cj/THKUummdP9qKhIOImuOU0Xjwb3y+oqt735rpTetVbF9n8R4x9crRfO77TKqVOZpDclv5bfK18lMnk+q0K18UpxF/RrGXE0Zxtqx3tWSj+vxbL4XaasfKb0dRbtLW73JsfPGg177H+OmGKlfvS1msllt7JEJm9XOJqXR+Fkfo1H66uy5H1v3Xx5+uJmGLw9jro2u7Loj4PnuR6+f+e3d261+eazNhzrL7X83MohoHpS4PddV8ki1it61//pw1g7z6p1U/26Nm2llST57B5rUvuG0XqSU/rPd7jYrqWcbd+beJQ77BgPMDwn37/8BAGCf0eMTOH4cPlufv9VGgJOzqf8Fjm1APXkd7B7f5dF57GPMaWy/MTvjvNvtXj6h8W2+GXvnzXaseeeHEgAAAAAAAB7aQXeKlcGiadBoEOeLb2dtpGOL2MyOtf391a3P/zD96c3JzIP3t4oV797vrh8+vn+YRL5bBuj/AQAAAABqJvfHXQAAHkX82zfXAQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAACeAgkAAAAAAAAAqLuRLAAAAAAAAACA2hv9D1iu/VAYaAYA', 'base64', ), );