diff --git a/.gitignore b/.gitignore index 496ee2ca..eaec4a94 100644 --- a/.gitignore +++ b/.gitignore @@ -1 +1,6 @@ -.DS_Store \ No newline at end of file +.DS_Store + +# MkDocs build output (generated by `mkdocs build` / `mkdocs serve`) +/site/* +!/site/index.html +!/site/versions.json diff --git a/docs/Reference/Console/MongoDB/collections.md b/docs/Reference/Console/MongoDB/collections.md new file mode 100644 index 00000000..1060e086 --- /dev/null +++ b/docs/Reference/Console/MongoDB/collections.md @@ -0,0 +1,37 @@ +# Collections and Indexes + +Console uses a fixed database name, `consoledb`. Collections and their unique +indexes are created at startup by `ensureIndexes` — creating an index +materializes the collection too. Collection names mirror the SQL table names, +and every collection carries a `tenantid` field. Most unique indexes are +scoped per tenant; the one exception is called out below the table. + +| Collection | Unique index | +|---|---| +| `devices` | `(guid, tenantid)` | +| `profiles` | `(profilename, tenantid)` | +| `ciraconfigs` | `(configname, tenantid)` | +| `domains` | `(profilename, tenantid)` | +| `domains` | `(domainsuffix, tenantid)` | +| `domains` | `(profilename, domainsuffix)` — case-insensitive collation | +| `ieee8021xconfigs` | `(profilename, tenantid)` | +| `wirelessconfigs` | `(profilename, tenantid)` | +| `profiles_wirelessconfigs` | `(profilename, wirelessprofilename, priority, tenantid)` | + +These indexes reproduce the SQL `UNIQUE` and `PRIMARY KEY` constraints. Index +creation is idempotent — restarting Console against an existing `consoledb` +with the same index definitions is safe and is a no-op. + +Field names shown above are the MongoDB BSON keys (taken from each entity +struct's `bson:"…"` tags). The SQL schema uses different column names — for +example, the `domains` table has columns `name`, `domain_suffix`, and +`tenant_id`, not `profilename`, `domainsuffix`, and `tenantid`. See Console's +[migrations directory][migrations] for the SQL DDL. + +!!! note "Cross-tenant uniqueness on `domains`" + The `(profilename, domainsuffix)` index on `domains` omits `tenantid` by + design — it mirrors the SQL `lower_name_suffix_idx` constraint. Two + different tenants cannot register a domain with the same + `(profilename, domainsuffix)` pair, compared case-insensitively. + +[migrations]: https://github.com/device-management-toolkit/console/tree/main/internal/app/migrations diff --git a/docs/Reference/Console/MongoDB/extending.md b/docs/Reference/Console/MongoDB/extending.md new file mode 100644 index 00000000..b5a48b64 --- /dev/null +++ b/docs/Reference/Console/MongoDB/extending.md @@ -0,0 +1,89 @@ +# Adding a New NoSQL DB + +`internal/usecase/nosqldb/` is a parent directory by design — it holds the +MongoDB backend today and is the home for future NoSQL siblings such as +Cassandra, DynamoDB, or Couchbase. The design is **closed for modification, +open for extension**: none of the existing use-case or HTTP code changes — you +add a new package and wire it in. + +!!! note "Illustrative placeholder" + `cassandra` below is a placeholder used to illustrate the pattern; it is + not a supported backend. Substitute the backend you are actually adding. + +## What you change + +| # | Step | File(s) | +|---|---|---| +| 1 | Create the package — one file per repository, plus shared infrastructure (`client.go`, `errors.go`, `fields.go`) | new `internal/usecase/nosqldb//` | +| 2 | Implement each `Repository` interface, with a compile-time guard per repo | the new package | +| 3 | Add a provider constant — `ProviderCassandra = "cassandra"` | `internal/app/repos.go` | +| 4 | Add a `buildRepos(cfg, log) (*usecase.Repos, error)` constructor that dials the driver and registers a `Closer` for graceful shutdown | `internal/app/repos.go` | +| 5 | Add `case Provider:` to the `buildRepos` switch | `internal/app/repos.go` | +| 6 | Extend the migration-skip guard — NoSQL backends manage their own schema | `internal/app/migrate.go` | +| 7 | Add a `build-` CI job mirroring `build-mongo` (starts the DB in Docker, runs the Postman collection) — the authoritative end-to-end parity check | `.github/workflows/api-test.yml` | +| 8 | Unit tests against the driver's wire-level mock when one ships (e.g., `drivertest.NewMockDeployment` for `mongo-driver/v2`); step 7 covers end-to-end | new `internal/usecase/nosqldb//*_test.go` | + +## What you don't change + +| File | Why | +|---|---| +| `internal/usecase/usecase.go` | Purely interface-based — never touched. | +| `internal/usecase//…` | Use cases consume interfaces, not concrete drivers. | +| Any HTTP handler | Calls use cases, not repositories. | +| `internal/usecase/sqldb/…` | The SQL backend is entirely orthogonal. | + +## Package layout (step 1) + +```text +internal/usecase/nosqldb/cassandra/ + client.go # Connect/Disconnect, index or schema setup, database name + device.go # implements devices.Repository + domain.go # implements domains.Repository + profile.go # implements profiles.Repository + ciraconfig.go + ieee8021xconfig.go + wificonfig.go + profilewificonfig.go + errors.go # reuse repoerrors via consoleerrors.CreateConsoleError + fields.go # centralize column / field name constants +``` + +## Wiring snippets (steps 2–6) + +**Step 2 — repository guard** (one per repo file under `internal/usecase/nosqldb//`) + +```go +var _ devices.Repository = (*DeviceRepo)(nil) +``` + +**Step 3 — provider constant** (`internal/app/repos.go`) + +```go +const ProviderCassandra = "cassandra" +``` + +**Step 4 — constructor** (`internal/app/repos.go`) + +```go +func buildCassandraRepos(cfg *config.Config, log logger.Interface) (*usecase.Repos, error) { + // dial the driver, optionally run schema setup, then return + // &usecase.Repos{ Devices: ..., Domains: ..., Closer: usecase.CloserFunc(...) } +} +``` + +**Step 5 — switch case** (`internal/app/repos.go`, inside `buildRepos`) + +```go +case ProviderCassandra: + return buildCassandraRepos(cfg, log) +``` + +**Step 6 — migration skip** (`internal/app/migrate.go`) + +The current check is a single equality (`cfg.Provider == ProviderMongo`); adding a backend turns it into an OR (or a small set). + +```go +if cfg.Provider == ProviderMongo || cfg.Provider == ProviderCassandra { + return nil // NoSQL backends manage their own schema +} +``` diff --git a/docs/Reference/Console/MongoDB/overview.md b/docs/Reference/Console/MongoDB/overview.md new file mode 100644 index 00000000..fdbfde55 --- /dev/null +++ b/docs/Reference/Console/MongoDB/overview.md @@ -0,0 +1,50 @@ +# MongoDB + +MongoDB is an alternative storage backend for Console — a drop-in replacement +for the default SQLite / PostgreSQL SQL backends. Switching is a configuration +change only: no code change, no schema migration. The repository interfaces are +identical across backends, so the use-case and HTTP layers are unaware of which +one is in use. + +!!! info "API compatibility" + The MongoDB backend is fully API-compatible with the SQL backends. + Everything available through the Console UI or REST API behaves the same on + either backend. Data does **not** migrate automatically between backends. + +## Selecting the backend + +Set `DB_PROVIDER` (or `db.provider` in `config.yml`): + +| `DB_PROVIDER` | Backend | `DB_URL` example | +|---|---|---| +| `sqlite` *(default; also when unset)* | Embedded SQLite | *none — uses a local file* | +| `postgres` | PostgreSQL | `postgres://user:pass@host:5432/dbname` | +| `mongo` | MongoDB | `mongodb://user:pass@host:27017/?authSource=admin` | + +Notes: + +- `DB_POOL_MAX` is honored only by the SQL backends. The MongoDB driver manages + its own connection pool, configured through `DB_URL` options such as + `maxPoolSize`. See [MongoDB connection string options][mongo-conn-opts] for + the full list. + +[mongo-conn-opts]: https://www.mongodb.com/docs/manual/reference/connection-string-options/ + +!!! note "Source paths in this section" + File paths and CI workflow references in the pages that follow (e.g., + `internal/usecase/nosqldb/mongo/`, `internal/app/repos.go`, + `.github/workflows/api-test.yml`) refer to the + [Console source repository][console-src], not to this docs repo. + +[console-src]: https://github.com/device-management-toolkit/console + +## Next steps + +- [Quick Start with Docker Compose](quickStart.md) — bring up the bundled + `mongo` service, point Console at it, and verify the deployment. +- [Collections and Indexes](collections.md) — what Console creates inside the + `consoledb` database. +- [Schema Changes](schemaChanges.md) — the bookkeeping each backend needs + when an entity changes. +- [Adding a New NoSQL DB](extending.md) — extending the `nosqldb` package with + another backend. diff --git a/docs/Reference/Console/MongoDB/quickStart.md b/docs/Reference/Console/MongoDB/quickStart.md new file mode 100644 index 00000000..6789d617 --- /dev/null +++ b/docs/Reference/Console/MongoDB/quickStart.md @@ -0,0 +1,91 @@ +# Quick Start with Docker Compose + +!!! note "Before you start" + This walkthrough uses the `docker-compose.yml` bundled with Console and the + `mongo` provider. See the [Overview](overview.md) for how `DB_PROVIDER` and + `DB_URL` select the backend. + +## 1. Start the stack + +The bundled `docker-compose.yml` includes a profile-gated `mongo` service. +Bring it up with the MongoDB environment variables set: + +```bash +DB_PROVIDER=mongo \ +DB_URL="mongodb://mongoadmin:admin123@mongo:27017/?authSource=admin" \ +docker compose --profile mongo up -d +``` + +| Setting | Effect | +|---|---| +| `--profile mongo` | Activates the `mongo` service (it stays idle otherwise). | +| `DB_PROVIDER=mongo` | Selects the MongoDB repository implementation. | +| `DB_URL` | Connection string; `mongo` is the Compose service hostname. | + +## 2. Check the startup log + +The `app` service listens on port `8181`. On startup it logs that it reached +MongoDB and ensured the unique indexes, for example: + +```text +mongo connected: db=consoledb +mongo unique indexes ensured (9 total) +``` + +## 3. Inspect the live database + +!!! tip "Prefer the UI?" + Console also ships with a web UI. The bundled Compose stack sets + `AUTH_DISABLED=true` on the `app` service, so browsing to + [http://localhost:8181](http://localhost:8181) opens the **Devices** view + directly — no login prompt. If the list renders without an error banner, + the MongoDB round-trip is working end to end. Continue with the `mongosh` + steps below to inspect the database directly. + +The `mongo` service from step 1 ships with the official MongoDB shell +([`mongosh`](https://www.mongodb.com/docs/mongodb-shell/)). The quickest way +to confirm Console created `consoledb` and its indexes from the command line +is to open an interactive `mongosh` session inside that container: + +```bash +docker compose exec mongo mongosh \ + -u mongoadmin -p admin123 --authenticationDatabase admin consoledb +``` + +| Part of the command | Effect | +|---|---| +| `docker compose exec mongo` | Runs the next command inside the running `mongo` service container. | +| `mongosh` | The MongoDB shell, already installed in the official `mongo` image. | +| `-u mongoadmin -p admin123 --authenticationDatabase admin` | Logs in with the dev credentials baked into the bundled service. | +| `consoledb` | Selects the database Console uses — the prompt switches to `consoledb>`. | + +Once you see the `consoledb>` prompt, run: + +```js +show collections +db.devices.countDocuments() +db.devices.getIndexes() +``` + +- `show collections` lists the collections Console manages — see + [Collections and Indexes](collections.md) for the full set. +- `db.devices.countDocuments()` returns `0` on a fresh deployment. +- `db.devices.getIndexes()` should include the unique compound index on + `(guid, tenantid)`, evidence that `ensureIndexes` ran successfully against + the live database. + +Type `exit` (or press **Ctrl+D**) to leave the shell. + +## Production notes + +The bundled stack is for local development only. Console connects to whatever +MongoDB `DB_URL` points at — it does not provision, secure, or back up the +database itself. For production, run a hardened MongoDB deployment following +the [MongoDB security checklist][mongo-sec], then point `DB_URL` at it, for +example: + +```text +mongodb://user:pass@host:27017/?authSource=admin&tls=true&replicaSet=rs0 +``` + +[mongo-sec]: https://www.mongodb.com/docs/manual/administration/security-checklist/ diff --git a/docs/Reference/Console/MongoDB/schemaChanges.md b/docs/Reference/Console/MongoDB/schemaChanges.md new file mode 100644 index 00000000..21ee800c --- /dev/null +++ b/docs/Reference/Console/MongoDB/schemaChanges.md @@ -0,0 +1,54 @@ +# Schema Changes + +The Go entity in `internal/entity/.go` is the single source of truth. +Both backends derive their behavior from it, but each requires distinct +bookkeeping when a field is added, removed, or changed. + +| Aspect | SQL backend (schema-on-write) | MongoDB backend (schema-on-read) | +|---|---|---| +| Shape enforcement | database (DDL) | application (repo code) | +| Where the shape is declared | `internal/app/migrations/*.sql` | nowhere — no DDL, no `ALTER TABLE` | +| Repo write obligation | name every column in `Insert` / `Update` / `Scan` | include every field in the `bson.M{"$set": …}` document | +| Missed field at runtime | SQL error: `unknown column` (loud) | field silently dropped from the document (silent) | + +!!! warning "Missed fields in `$set` are silent" + Forget a field in `Update`'s `$set` map and the MongoDB driver writes the + document without it — no error is returned. The api-test workflow's + `build-mongo` job (running the full Postman collection against a real + MongoDB) is the authoritative parity check that surfaces this drift. + +## Add a nullable field + +| Step | SQL backend | MongoDB backend | +|---|---|---| +| 1. Entity struct | add the field to `internal/entity/.go` | same file — add a `bson:""` struct tag | +| 2. Schema | add `_.up.sql` and `.down.sql` under `internal/app/migrations/` | none — schemaless | +| 3. `Insert` | add to `squirrel.Insert(...).Columns(...)` / `.Values(...)` in `internal/usecase/sqldb/.go` | none — `InsertOne(ctx, e)` serializes the struct by its `bson` tags | +| 4. `Update` | add to the `squirrel.Update(...).Set(...)` chain | add the key to the `bson.M{"$set": …}` map in `internal/usecase/nosqldb/mongo/.go` | +| 5. `Get` / `Scan` | add to the projection and `rows.Scan(...)` targets | none — `cur.All(...)` / `Decode(...)` populates any field present | +| 6. Index (optional) | migration with `CREATE INDEX` | new `IndexModel` in `ensureIndexes` (`client.go`) | + +## Variations + +Each row lists only the delta from the procedure above. + +| Change | SQL backend | MongoDB backend | +|---|---|---| +| `NOT NULL` with default | migration must include `DEFAULT` to backfill existing rows — e.g., `ALTER TABLE profiles ADD COLUMN uefi_wifi_sync_enabled BOOLEAN NOT NULL DEFAULT FALSE` | existing documents lack the field; reads return the Go zero value. Optional backfill: `db.profiles.updateMany({uefiwifisyncenabled: {$exists: false}}, {$set: {uefiwifisyncenabled: false}})` | +| Unique constraint | migration with `CREATE UNIQUE INDEX ... ON profiles (profilename, tenantid)`; existing duplicates must be removed first | new unique `IndexModel` in `ensureIndexes` (optionally collated); existing duplicates block startup | +| Drop a field | migration `ALTER TABLE devices DROP COLUMN mpspassword` (`.down.sql` re-adds) | stop writing it in `Insert` / `Update`; old documents retain it. Optional cleanup: `db.devices.updateMany({}, {$unset: {mpspassword: 1}})` | +| Rename or change type | dual-write → backfill → switch reads → drop the old shape, across multiple releases | identical multi-release pattern | +| Foreign-key relationship | `FOREIGN KEY` constraint in the migration | no native foreign keys — enforce in the use-case layer (parent lookup before child insert) | + +## Files to change together + +A change to `internal/entity/.go` typically touches: + +```text +internal/app/migrations/_.up.sql +internal/app/migrations/_.down.sql +internal/usecase/sqldb/.go +internal/usecase/nosqldb/mongo/.go +internal/usecase/nosqldb/mongo/fields.go # if used as a filter or sort key +internal/usecase/nosqldb/mongo/client.go # if it needs a unique index +``` diff --git a/docs/Reference/Console/configuration.md b/docs/Reference/Console/configuration.md index fd7a0cc9..c106bdfd 100644 --- a/docs/Reference/Console/configuration.md +++ b/docs/Reference/Console/configuration.md @@ -35,8 +35,9 @@ Console can also be configured using environment variables. These `.env` variabl | HTTP_ALLOWED_ORIGINS | http.allowed_origins | `*` | Allowed origins for CORS policy. | | HTTP_ALLOWED_HEADERS | http.allowed_headers | `*` | Allowed headers for CORS policy. | | LOG_LEVEL | logger.log_level | `info` | Controls the level of logging. Options: `error`, `warn`, `info`, `debug`, `fatal`. | -| DB_POOL_MAX | db.pool_max | `2` | Maximum number of database connections in the pool. | -| DB_URL | db.url | No Value | By default, Console uses a SQLite database to store device data locally. Users can optionally configure this variable to provide a PostgreSQL connection string, enabling the use of an external PostgreSQL database for data storage. This allows for greater scalability and centralized database management. | +| DB_PROVIDER | db.provider | `sqlite` | Selects the storage backend. Valid values: `postgres`, `sqlite` (default), `mongo`. See [MongoDB](MongoDB/overview.md) for the NoSQL option. | +| DB_POOL_MAX | db.pool_max | `2` | Maximum number of database connections in the pool. Honored by the SQL backends only; the MongoDB driver manages its own pool via the `DB_URL`. See [MongoDB](MongoDB/overview.md) for connection-string pool options. | +| DB_URL | db.url | No Value | Connection string for the selected backend. Examples: `postgres://user:pass@host:5432/dbname` for Postgres; `mongodb://user:pass@host:27017/?authSource=admin` for MongoDB. Leave empty to use the embedded SQLite database. | | EA_URL | ea.url | `http://localhost:8000` | URL for the Enterprise Assistant service. | | EA_USERNAME | ea.username | No Value | Username for the Enterprise Assistant service. | | EA_PASSWORD | ea.password | No Value | Password for the Enterprise Assistant service. | diff --git a/mkdocs.yml b/mkdocs.yml index 8614ecbb..4af9bcdb 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -128,6 +128,12 @@ nav: - Console: - Overview: Reference/Console/overview.md - Configuration: Reference/Console/configuration.md + - MongoDB: + - Overview: Reference/Console/MongoDB/overview.md + - Quick Start: Reference/Console/MongoDB/quickStart.md + - Collections and Indexes: Reference/Console/MongoDB/collections.md + - Schema Changes: Reference/Console/MongoDB/schemaChanges.md + - Adding a New NoSQL DB: Reference/Console/MongoDB/extending.md - Upgrade: Reference/Console/upgrade.md - Security Information: Reference/Console/securityConsole.md - Features: