-
Notifications
You must be signed in to change notification settings - Fork 11
docs: add nosqldb for console #807
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
madhavilosetty-intel
wants to merge
1
commit into
main
Choose a base branch
from
mongodb
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1 +1,6 @@ | ||
| .DS_Store | ||
| .DS_Store | ||
|
|
||
| # MkDocs build output (generated by `mkdocs build` / `mkdocs serve`) | ||
| /site/* | ||
| !/site/index.html | ||
| !/site/versions.json |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,37 @@ | ||
| # Collections and Indexes | ||
|
|
||
| Console uses a fixed database name, `consoledb`. Collections and their unique | ||
| indexes are created at startup by `ensureIndexes` — creating an index | ||
| materializes the collection too. Collection names mirror the SQL table names, | ||
| and every collection carries a `tenantid` field. Most unique indexes are | ||
| scoped per tenant; the one exception is called out below the table. | ||
|
|
||
| | Collection | Unique index | | ||
| |---|---| | ||
| | `devices` | `(guid, tenantid)` | | ||
| | `profiles` | `(profilename, tenantid)` | | ||
| | `ciraconfigs` | `(configname, tenantid)` | | ||
| | `domains` | `(profilename, tenantid)` | | ||
| | `domains` | `(domainsuffix, tenantid)` | | ||
| | `domains` | `(profilename, domainsuffix)` — case-insensitive collation | | ||
| | `ieee8021xconfigs` | `(profilename, tenantid)` | | ||
| | `wirelessconfigs` | `(profilename, tenantid)` | | ||
| | `profiles_wirelessconfigs` | `(profilename, wirelessprofilename, priority, tenantid)` | | ||
|
|
||
| These indexes reproduce the SQL `UNIQUE` and `PRIMARY KEY` constraints. Index | ||
| creation is idempotent — restarting Console against an existing `consoledb` | ||
| with the same index definitions is safe and is a no-op. | ||
|
|
||
| Field names shown above are the MongoDB BSON keys (taken from each entity | ||
| struct's `bson:"…"` tags). The SQL schema uses different column names — for | ||
| example, the `domains` table has columns `name`, `domain_suffix`, and | ||
| `tenant_id`, not `profilename`, `domainsuffix`, and `tenantid`. See Console's | ||
| [migrations directory][migrations] for the SQL DDL. | ||
|
|
||
| !!! note "Cross-tenant uniqueness on `domains`" | ||
| The `(profilename, domainsuffix)` index on `domains` omits `tenantid` by | ||
| design — it mirrors the SQL `lower_name_suffix_idx` constraint. Two | ||
| different tenants cannot register a domain with the same | ||
| `(profilename, domainsuffix)` pair, compared case-insensitively. | ||
|
|
||
| [migrations]: https://github.com/device-management-toolkit/console/tree/main/internal/app/migrations | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,89 @@ | ||
| # Adding a New NoSQL DB | ||
|
|
||
| `internal/usecase/nosqldb/` is a parent directory by design — it holds the | ||
| MongoDB backend today and is the home for future NoSQL siblings such as | ||
| Cassandra, DynamoDB, or Couchbase. The design is **closed for modification, | ||
| open for extension**: none of the existing use-case or HTTP code changes — you | ||
| add a new package and wire it in. | ||
|
|
||
| !!! note "Illustrative placeholder" | ||
| `cassandra` below is a placeholder used to illustrate the pattern; it is | ||
| not a supported backend. Substitute the backend you are actually adding. | ||
|
|
||
| ## What you change | ||
|
|
||
| | # | Step | File(s) | | ||
| |---|---|---| | ||
| | 1 | Create the package — one file per repository, plus shared infrastructure (`client.go`, `errors.go`, `fields.go`) | new `internal/usecase/nosqldb/<name>/` | | ||
| | 2 | Implement each `Repository` interface, with a compile-time guard per repo | the new package | | ||
| | 3 | Add a provider constant — `ProviderCassandra = "cassandra"` | `internal/app/repos.go` | | ||
| | 4 | Add a `build<Name>Repos(cfg, log) (*usecase.Repos, error)` constructor that dials the driver and registers a `Closer` for graceful shutdown | `internal/app/repos.go` | | ||
| | 5 | Add `case Provider<Name>:` to the `buildRepos` switch | `internal/app/repos.go` | | ||
| | 6 | Extend the migration-skip guard — NoSQL backends manage their own schema | `internal/app/migrate.go` | | ||
| | 7 | Add a `build-<name>` CI job mirroring `build-mongo` (starts the DB in Docker, runs the Postman collection) — the authoritative end-to-end parity check | `.github/workflows/api-test.yml` | | ||
|
madhavilosetty-intel marked this conversation as resolved.
|
||
| | 8 | Unit tests against the driver's wire-level mock when one ships (e.g., `drivertest.NewMockDeployment` for `mongo-driver/v2`); step 7 covers end-to-end | new `internal/usecase/nosqldb/<name>/*_test.go` | | ||
|
|
||
| ## What you don't change | ||
|
|
||
| | File | Why | | ||
| |---|---| | ||
| | `internal/usecase/usecase.go` | Purely interface-based — never touched. | | ||
| | `internal/usecase/<feature>/…` | Use cases consume interfaces, not concrete drivers. | | ||
| | Any HTTP handler | Calls use cases, not repositories. | | ||
| | `internal/usecase/sqldb/…` | The SQL backend is entirely orthogonal. | | ||
|
|
||
| ## Package layout (step 1) | ||
|
|
||
| ```text | ||
| internal/usecase/nosqldb/cassandra/ | ||
| client.go # Connect/Disconnect, index or schema setup, database name | ||
| device.go # implements devices.Repository | ||
| domain.go # implements domains.Repository | ||
| profile.go # implements profiles.Repository | ||
| ciraconfig.go | ||
| ieee8021xconfig.go | ||
| wificonfig.go | ||
| profilewificonfig.go | ||
| errors.go # reuse repoerrors via consoleerrors.CreateConsoleError | ||
| fields.go # centralize column / field name constants | ||
| ``` | ||
|
|
||
| ## Wiring snippets (steps 2–6) | ||
|
|
||
| **Step 2 — repository guard** (one per repo file under `internal/usecase/nosqldb/<name>/`) | ||
|
|
||
| ```go | ||
| var _ devices.Repository = (*DeviceRepo)(nil) | ||
| ``` | ||
|
|
||
| **Step 3 — provider constant** (`internal/app/repos.go`) | ||
|
|
||
| ```go | ||
| const ProviderCassandra = "cassandra" | ||
| ``` | ||
|
|
||
| **Step 4 — constructor** (`internal/app/repos.go`) | ||
|
|
||
| ```go | ||
| func buildCassandraRepos(cfg *config.Config, log logger.Interface) (*usecase.Repos, error) { | ||
| // dial the driver, optionally run schema setup, then return | ||
| // &usecase.Repos{ Devices: ..., Domains: ..., Closer: usecase.CloserFunc(...) } | ||
| } | ||
| ``` | ||
|
|
||
| **Step 5 — switch case** (`internal/app/repos.go`, inside `buildRepos`) | ||
|
|
||
| ```go | ||
| case ProviderCassandra: | ||
| return buildCassandraRepos(cfg, log) | ||
| ``` | ||
|
|
||
| **Step 6 — migration skip** (`internal/app/migrate.go`) | ||
|
|
||
| The current check is a single equality (`cfg.Provider == ProviderMongo`); adding a backend turns it into an OR (or a small set). | ||
|
|
||
| ```go | ||
| if cfg.Provider == ProviderMongo || cfg.Provider == ProviderCassandra { | ||
| return nil // NoSQL backends manage their own schema | ||
| } | ||
| ``` | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,50 @@ | ||
| # MongoDB | ||
|
|
||
| MongoDB is an alternative storage backend for Console — a drop-in replacement | ||
| for the default SQLite / PostgreSQL SQL backends. Switching is a configuration | ||
| change only: no code change, no schema migration. The repository interfaces are | ||
| identical across backends, so the use-case and HTTP layers are unaware of which | ||
| one is in use. | ||
|
|
||
| !!! info "API compatibility" | ||
| The MongoDB backend is fully API-compatible with the SQL backends. | ||
| Everything available through the Console UI or REST API behaves the same on | ||
| either backend. Data does **not** migrate automatically between backends. | ||
|
|
||
| ## Selecting the backend | ||
|
|
||
| Set `DB_PROVIDER` (or `db.provider` in `config.yml`): | ||
|
|
||
| | `DB_PROVIDER` | Backend | `DB_URL` example | | ||
| |---|---|---| | ||
| | `sqlite` *(default; also when unset)* | Embedded SQLite | *none — uses a local file* | | ||
| | `postgres` | PostgreSQL | `postgres://user:pass@host:5432/dbname` | | ||
| | `mongo` | MongoDB | `mongodb://user:pass@host:27017/?authSource=admin` | | ||
|
|
||
| Notes: | ||
|
|
||
| - `DB_POOL_MAX` is honored only by the SQL backends. The MongoDB driver manages | ||
| its own connection pool, configured through `DB_URL` options such as | ||
| `maxPoolSize`. See [MongoDB connection string options][mongo-conn-opts] for | ||
| the full list. | ||
|
|
||
| [mongo-conn-opts]: https://www.mongodb.com/docs/manual/reference/connection-string-options/ | ||
|
|
||
| !!! note "Source paths in this section" | ||
| File paths and CI workflow references in the pages that follow (e.g., | ||
| `internal/usecase/nosqldb/mongo/`, `internal/app/repos.go`, | ||
| `.github/workflows/api-test.yml`) refer to the | ||
| [Console source repository][console-src], not to this docs repo. | ||
|
|
||
| [console-src]: https://github.com/device-management-toolkit/console | ||
|
|
||
| ## Next steps | ||
|
|
||
| - [Quick Start with Docker Compose](quickStart.md) — bring up the bundled | ||
| `mongo` service, point Console at it, and verify the deployment. | ||
| - [Collections and Indexes](collections.md) — what Console creates inside the | ||
| `consoledb` database. | ||
| - [Schema Changes](schemaChanges.md) — the bookkeeping each backend needs | ||
| when an entity changes. | ||
| - [Adding a New NoSQL DB](extending.md) — extending the `nosqldb` package with | ||
| another backend. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,91 @@ | ||
| # Quick Start with Docker Compose | ||
|
|
||
| !!! note "Before you start" | ||
| This walkthrough uses the `docker-compose.yml` bundled with Console and the | ||
| `mongo` provider. See the [Overview](overview.md) for how `DB_PROVIDER` and | ||
| `DB_URL` select the backend. | ||
|
|
||
| ## 1. Start the stack | ||
|
|
||
| The bundled `docker-compose.yml` includes a profile-gated `mongo` service. | ||
| Bring it up with the MongoDB environment variables set: | ||
|
|
||
| ```bash | ||
| DB_PROVIDER=mongo \ | ||
| DB_URL="mongodb://mongoadmin:admin123@mongo:27017/?authSource=admin" \ | ||
| docker compose --profile mongo up -d | ||
| ``` | ||
|
|
||
| | Setting | Effect | | ||
| |---|---| | ||
| | `--profile mongo` | Activates the `mongo` service (it stays idle otherwise). | | ||
| | `DB_PROVIDER=mongo` | Selects the MongoDB repository implementation. | | ||
| | `DB_URL` | Connection string; `mongo` is the Compose service hostname. | | ||
|
|
||
| ## 2. Check the startup log | ||
|
|
||
| The `app` service listens on port `8181`. On startup it logs that it reached | ||
| MongoDB and ensured the unique indexes, for example: | ||
|
|
||
| ```text | ||
| mongo connected: db=consoledb | ||
| mongo unique indexes ensured (9 total) | ||
| ``` | ||
|
|
||
| ## 3. Inspect the live database | ||
|
|
||
| !!! tip "Prefer the UI?" | ||
| Console also ships with a web UI. The bundled Compose stack sets | ||
| `AUTH_DISABLED=true` on the `app` service, so browsing to | ||
| [http://localhost:8181](http://localhost:8181) opens the **Devices** view | ||
| directly — no login prompt. If the list renders without an error banner, | ||
| the MongoDB round-trip is working end to end. Continue with the `mongosh` | ||
| steps below to inspect the database directly. | ||
|
|
||
| The `mongo` service from step 1 ships with the official MongoDB shell | ||
| ([`mongosh`](https://www.mongodb.com/docs/mongodb-shell/)). The quickest way | ||
| to confirm Console created `consoledb` and its indexes from the command line | ||
| is to open an interactive `mongosh` session inside that container: | ||
|
|
||
| ```bash | ||
| docker compose exec mongo mongosh \ | ||
| -u mongoadmin -p admin123 --authenticationDatabase admin consoledb | ||
| ``` | ||
|
|
||
| | Part of the command | Effect | | ||
| |---|---| | ||
| | `docker compose exec mongo` | Runs the next command inside the running `mongo` service container. | | ||
| | `mongosh` | The MongoDB shell, already installed in the official `mongo` image. | | ||
| | `-u mongoadmin -p admin123 --authenticationDatabase admin` | Logs in with the dev credentials baked into the bundled service. | | ||
| | `consoledb` | Selects the database Console uses — the prompt switches to `consoledb>`. | | ||
|
|
||
| Once you see the `consoledb>` prompt, run: | ||
|
|
||
| ```js | ||
| show collections | ||
| db.devices.countDocuments() | ||
| db.devices.getIndexes() | ||
| ``` | ||
|
|
||
| - `show collections` lists the collections Console manages — see | ||
| [Collections and Indexes](collections.md) for the full set. | ||
| - `db.devices.countDocuments()` returns `0` on a fresh deployment. | ||
| - `db.devices.getIndexes()` should include the unique compound index on | ||
| `(guid, tenantid)`, evidence that `ensureIndexes` ran successfully against | ||
| the live database. | ||
|
|
||
| Type `exit` (or press **Ctrl+D**) to leave the shell. | ||
|
|
||
| ## Production notes | ||
|
|
||
| The bundled stack is for local development only. Console connects to whatever | ||
| MongoDB `DB_URL` points at — it does not provision, secure, or back up the | ||
| database itself. For production, run a hardened MongoDB deployment following | ||
| the [MongoDB security checklist][mongo-sec], then point `DB_URL` at it, for | ||
| example: | ||
|
|
||
| ```text | ||
| mongodb://user:pass@host:27017/?authSource=admin&tls=true&replicaSet=rs0 | ||
| ``` | ||
|
|
||
| [mongo-sec]: https://www.mongodb.com/docs/manual/administration/security-checklist/ |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,54 @@ | ||
| # Schema Changes | ||
|
|
||
| The Go entity in `internal/entity/<entity>.go` is the single source of truth. | ||
| Both backends derive their behavior from it, but each requires distinct | ||
| bookkeeping when a field is added, removed, or changed. | ||
|
|
||
| | Aspect | SQL backend (schema-on-write) | MongoDB backend (schema-on-read) | | ||
| |---|---|---| | ||
| | Shape enforcement | database (DDL) | application (repo code) | | ||
| | Where the shape is declared | `internal/app/migrations/*.sql` | nowhere — no DDL, no `ALTER TABLE` | | ||
| | Repo write obligation | name every column in `Insert` / `Update` / `Scan` | include every field in the `bson.M{"$set": …}` document | | ||
| | Missed field at runtime | SQL error: `unknown column` (loud) | field silently dropped from the document (silent) | | ||
|
|
||
| !!! warning "Missed fields in `$set` are silent" | ||
| Forget a field in `Update`'s `$set` map and the MongoDB driver writes the | ||
| document without it — no error is returned. The api-test workflow's | ||
| `build-mongo` job (running the full Postman collection against a real | ||
| MongoDB) is the authoritative parity check that surfaces this drift. | ||
|
madhavilosetty-intel marked this conversation as resolved.
|
||
|
|
||
| ## Add a nullable field | ||
|
|
||
| | Step | SQL backend | MongoDB backend | | ||
| |---|---|---| | ||
| | 1. Entity struct | add the field to `internal/entity/<entity>.go` | same file — add a `bson:"<name>"` struct tag | | ||
| | 2. Schema | add `<timestamp>_<name>.up.sql` and `.down.sql` under `internal/app/migrations/` | none — schemaless | | ||
| | 3. `Insert` | add to `squirrel.Insert(...).Columns(...)` / `.Values(...)` in `internal/usecase/sqldb/<entity>.go` | none — `InsertOne(ctx, e)` serializes the struct by its `bson` tags | | ||
| | 4. `Update` | add to the `squirrel.Update(...).Set(...)` chain | add the key to the `bson.M{"$set": …}` map in `internal/usecase/nosqldb/mongo/<entity>.go` | | ||
| | 5. `Get` / `Scan` | add to the projection and `rows.Scan(...)` targets | none — `cur.All(...)` / `Decode(...)` populates any field present | | ||
| | 6. Index (optional) | migration with `CREATE INDEX` | new `IndexModel` in `ensureIndexes` (`client.go`) | | ||
|
|
||
| ## Variations | ||
|
|
||
| Each row lists only the delta from the procedure above. | ||
|
|
||
| | Change | SQL backend | MongoDB backend | | ||
| |---|---|---| | ||
| | `NOT NULL` with default | migration must include `DEFAULT` to backfill existing rows — e.g., `ALTER TABLE profiles ADD COLUMN uefi_wifi_sync_enabled BOOLEAN NOT NULL DEFAULT FALSE` | existing documents lack the field; reads return the Go zero value. Optional backfill: `db.profiles.updateMany({uefiwifisyncenabled: {$exists: false}}, {$set: {uefiwifisyncenabled: false}})` | | ||
| | Unique constraint | migration with `CREATE UNIQUE INDEX ... ON profiles (profilename, tenantid)`; existing duplicates must be removed first | new unique `IndexModel` in `ensureIndexes` (optionally collated); existing duplicates block startup | | ||
| | Drop a field | migration `ALTER TABLE devices DROP COLUMN mpspassword` (`.down.sql` re-adds) | stop writing it in `Insert` / `Update`; old documents retain it. Optional cleanup: `db.devices.updateMany({}, {$unset: {mpspassword: 1}})` | | ||
| | Rename or change type | dual-write → backfill → switch reads → drop the old shape, across multiple releases | identical multi-release pattern | | ||
| | Foreign-key relationship | `FOREIGN KEY` constraint in the migration | no native foreign keys — enforce in the use-case layer (parent lookup before child insert) | | ||
|
|
||
| ## Files to change together | ||
|
|
||
| A change to `internal/entity/<entity>.go` typically touches: | ||
|
|
||
| ```text | ||
| internal/app/migrations/<timestamp>_<name>.up.sql | ||
| internal/app/migrations/<timestamp>_<name>.down.sql | ||
| internal/usecase/sqldb/<entity>.go | ||
| internal/usecase/nosqldb/mongo/<entity>.go | ||
| internal/usecase/nosqldb/mongo/fields.go # if used as a filter or sort key | ||
| internal/usecase/nosqldb/mongo/client.go # if it needs a unique index | ||
| ``` | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.