Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 6 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
@@ -1 +1,6 @@
.DS_Store
.DS_Store

# MkDocs build output (generated by `mkdocs build` / `mkdocs serve`)
/site/*
!/site/index.html
!/site/versions.json
37 changes: 37 additions & 0 deletions docs/Reference/Console/MongoDB/collections.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
# Collections and Indexes

Console uses a fixed database name, `consoledb`. Collections and their unique
indexes are created at startup by `ensureIndexes` — creating an index
materializes the collection too. Collection names mirror the SQL table names,
and every collection carries a `tenantid` field. Most unique indexes are
scoped per tenant; the one exception is called out below the table.

| Collection | Unique index |
|---|---|
| `devices` | `(guid, tenantid)` |
| `profiles` | `(profilename, tenantid)` |
| `ciraconfigs` | `(configname, tenantid)` |
| `domains` | `(profilename, tenantid)` |
| `domains` | `(domainsuffix, tenantid)` |
| `domains` | `(profilename, domainsuffix)` — case-insensitive collation |
Comment thread
madhavilosetty-intel marked this conversation as resolved.
| `ieee8021xconfigs` | `(profilename, tenantid)` |
| `wirelessconfigs` | `(profilename, tenantid)` |
| `profiles_wirelessconfigs` | `(profilename, wirelessprofilename, priority, tenantid)` |

These indexes reproduce the SQL `UNIQUE` and `PRIMARY KEY` constraints. Index
creation is idempotent — restarting Console against an existing `consoledb`
with the same index definitions is safe and is a no-op.

Field names shown above are the MongoDB BSON keys (taken from each entity
struct's `bson:"…"` tags). The SQL schema uses different column names — for
example, the `domains` table has columns `name`, `domain_suffix`, and
`tenant_id`, not `profilename`, `domainsuffix`, and `tenantid`. See Console's
[migrations directory][migrations] for the SQL DDL.

!!! note "Cross-tenant uniqueness on `domains`"
The `(profilename, domainsuffix)` index on `domains` omits `tenantid` by
design — it mirrors the SQL `lower_name_suffix_idx` constraint. Two
different tenants cannot register a domain with the same
`(profilename, domainsuffix)` pair, compared case-insensitively.

[migrations]: https://github.com/device-management-toolkit/console/tree/main/internal/app/migrations
89 changes: 89 additions & 0 deletions docs/Reference/Console/MongoDB/extending.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,89 @@
# Adding a New NoSQL DB

`internal/usecase/nosqldb/` is a parent directory by design — it holds the
MongoDB backend today and is the home for future NoSQL siblings such as
Cassandra, DynamoDB, or Couchbase. The design is **closed for modification,
open for extension**: none of the existing use-case or HTTP code changes — you
add a new package and wire it in.

!!! note "Illustrative placeholder"
`cassandra` below is a placeholder used to illustrate the pattern; it is
not a supported backend. Substitute the backend you are actually adding.

## What you change

| # | Step | File(s) |
|---|---|---|
| 1 | Create the package — one file per repository, plus shared infrastructure (`client.go`, `errors.go`, `fields.go`) | new `internal/usecase/nosqldb/<name>/` |
| 2 | Implement each `Repository` interface, with a compile-time guard per repo | the new package |
| 3 | Add a provider constant — `ProviderCassandra = "cassandra"` | `internal/app/repos.go` |
| 4 | Add a `build<Name>Repos(cfg, log) (*usecase.Repos, error)` constructor that dials the driver and registers a `Closer` for graceful shutdown | `internal/app/repos.go` |
| 5 | Add `case Provider<Name>:` to the `buildRepos` switch | `internal/app/repos.go` |
| 6 | Extend the migration-skip guard — NoSQL backends manage their own schema | `internal/app/migrate.go` |
| 7 | Add a `build-<name>` CI job mirroring `build-mongo` (starts the DB in Docker, runs the Postman collection) — the authoritative end-to-end parity check | `.github/workflows/api-test.yml` |
Comment thread
madhavilosetty-intel marked this conversation as resolved.
| 8 | Unit tests against the driver's wire-level mock when one ships (e.g., `drivertest.NewMockDeployment` for `mongo-driver/v2`); step 7 covers end-to-end | new `internal/usecase/nosqldb/<name>/*_test.go` |

## What you don't change

| File | Why |
|---|---|
| `internal/usecase/usecase.go` | Purely interface-based — never touched. |
| `internal/usecase/<feature>/…` | Use cases consume interfaces, not concrete drivers. |
| Any HTTP handler | Calls use cases, not repositories. |
| `internal/usecase/sqldb/…` | The SQL backend is entirely orthogonal. |

## Package layout (step 1)

```text
internal/usecase/nosqldb/cassandra/
client.go # Connect/Disconnect, index or schema setup, database name
device.go # implements devices.Repository
domain.go # implements domains.Repository
profile.go # implements profiles.Repository
ciraconfig.go
ieee8021xconfig.go
wificonfig.go
profilewificonfig.go
errors.go # reuse repoerrors via consoleerrors.CreateConsoleError
fields.go # centralize column / field name constants
```

## Wiring snippets (steps 2–6)

**Step 2 — repository guard** (one per repo file under `internal/usecase/nosqldb/<name>/`)

```go
var _ devices.Repository = (*DeviceRepo)(nil)
```

**Step 3 — provider constant** (`internal/app/repos.go`)

```go
const ProviderCassandra = "cassandra"
```

**Step 4 — constructor** (`internal/app/repos.go`)

```go
func buildCassandraRepos(cfg *config.Config, log logger.Interface) (*usecase.Repos, error) {
// dial the driver, optionally run schema setup, then return
// &usecase.Repos{ Devices: ..., Domains: ..., Closer: usecase.CloserFunc(...) }
}
```

**Step 5 — switch case** (`internal/app/repos.go`, inside `buildRepos`)

```go
case ProviderCassandra:
return buildCassandraRepos(cfg, log)
```

**Step 6 — migration skip** (`internal/app/migrate.go`)

The current check is a single equality (`cfg.Provider == ProviderMongo`); adding a backend turns it into an OR (or a small set).

```go
if cfg.Provider == ProviderMongo || cfg.Provider == ProviderCassandra {
return nil // NoSQL backends manage their own schema
}
```
50 changes: 50 additions & 0 deletions docs/Reference/Console/MongoDB/overview.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,50 @@
# MongoDB

MongoDB is an alternative storage backend for Console — a drop-in replacement
for the default SQLite / PostgreSQL SQL backends. Switching is a configuration
change only: no code change, no schema migration. The repository interfaces are
identical across backends, so the use-case and HTTP layers are unaware of which
one is in use.

!!! info "API compatibility"
The MongoDB backend is fully API-compatible with the SQL backends.
Everything available through the Console UI or REST API behaves the same on
either backend. Data does **not** migrate automatically between backends.

## Selecting the backend

Set `DB_PROVIDER` (or `db.provider` in `config.yml`):

| `DB_PROVIDER` | Backend | `DB_URL` example |
|---|---|---|
| `sqlite` *(default; also when unset)* | Embedded SQLite | *none — uses a local file* |
| `postgres` | PostgreSQL | `postgres://user:pass@host:5432/dbname` |
| `mongo` | MongoDB | `mongodb://user:pass@host:27017/?authSource=admin` |

Notes:

- `DB_POOL_MAX` is honored only by the SQL backends. The MongoDB driver manages
its own connection pool, configured through `DB_URL` options such as
`maxPoolSize`. See [MongoDB connection string options][mongo-conn-opts] for
the full list.

[mongo-conn-opts]: https://www.mongodb.com/docs/manual/reference/connection-string-options/

!!! note "Source paths in this section"
File paths and CI workflow references in the pages that follow (e.g.,
`internal/usecase/nosqldb/mongo/`, `internal/app/repos.go`,
`.github/workflows/api-test.yml`) refer to the
[Console source repository][console-src], not to this docs repo.

[console-src]: https://github.com/device-management-toolkit/console

## Next steps

- [Quick Start with Docker Compose](quickStart.md) — bring up the bundled
`mongo` service, point Console at it, and verify the deployment.
- [Collections and Indexes](collections.md) — what Console creates inside the
`consoledb` database.
- [Schema Changes](schemaChanges.md) — the bookkeeping each backend needs
when an entity changes.
- [Adding a New NoSQL DB](extending.md) — extending the `nosqldb` package with
another backend.
91 changes: 91 additions & 0 deletions docs/Reference/Console/MongoDB/quickStart.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,91 @@
# Quick Start with Docker Compose

!!! note "Before you start"
This walkthrough uses the `docker-compose.yml` bundled with Console and the
`mongo` provider. See the [Overview](overview.md) for how `DB_PROVIDER` and
`DB_URL` select the backend.

## 1. Start the stack

The bundled `docker-compose.yml` includes a profile-gated `mongo` service.
Bring it up with the MongoDB environment variables set:

```bash
DB_PROVIDER=mongo \
DB_URL="mongodb://mongoadmin:admin123@mongo:27017/?authSource=admin" \
docker compose --profile mongo up -d
```

| Setting | Effect |
|---|---|
| `--profile mongo` | Activates the `mongo` service (it stays idle otherwise). |
| `DB_PROVIDER=mongo` | Selects the MongoDB repository implementation. |
| `DB_URL` | Connection string; `mongo` is the Compose service hostname. |

## 2. Check the startup log

The `app` service listens on port `8181`. On startup it logs that it reached
MongoDB and ensured the unique indexes, for example:

```text
mongo connected: db=consoledb
mongo unique indexes ensured (9 total)
```

## 3. Inspect the live database

!!! tip "Prefer the UI?"
Console also ships with a web UI. The bundled Compose stack sets
`AUTH_DISABLED=true` on the `app` service, so browsing to
[http://localhost:8181](http://localhost:8181) opens the **Devices** view
directly — no login prompt. If the list renders without an error banner,
the MongoDB round-trip is working end to end. Continue with the `mongosh`
steps below to inspect the database directly.

The `mongo` service from step 1 ships with the official MongoDB shell
([`mongosh`](https://www.mongodb.com/docs/mongodb-shell/)). The quickest way
to confirm Console created `consoledb` and its indexes from the command line
is to open an interactive `mongosh` session inside that container:

```bash
docker compose exec mongo mongosh \
-u mongoadmin -p admin123 --authenticationDatabase admin consoledb
```

| Part of the command | Effect |
|---|---|
| `docker compose exec mongo` | Runs the next command inside the running `mongo` service container. |
| `mongosh` | The MongoDB shell, already installed in the official `mongo` image. |
| `-u mongoadmin -p admin123 --authenticationDatabase admin` | Logs in with the dev credentials baked into the bundled service. |
| `consoledb` | Selects the database Console uses — the prompt switches to `consoledb>`. |

Once you see the `consoledb>` prompt, run:

```js
show collections
db.devices.countDocuments()
db.devices.getIndexes()
```

- `show collections` lists the collections Console manages — see
[Collections and Indexes](collections.md) for the full set.
- `db.devices.countDocuments()` returns `0` on a fresh deployment.
- `db.devices.getIndexes()` should include the unique compound index on
`(guid, tenantid)`, evidence that `ensureIndexes` ran successfully against
the live database.

Type `exit` (or press **Ctrl+D**) to leave the shell.

## Production notes

The bundled stack is for local development only. Console connects to whatever
MongoDB `DB_URL` points at — it does not provision, secure, or back up the
database itself. For production, run a hardened MongoDB deployment following
the [MongoDB security checklist][mongo-sec], then point `DB_URL` at it, for
example:

```text
mongodb://user:pass@host:27017/?authSource=admin&tls=true&replicaSet=rs0
```

[mongo-sec]: https://www.mongodb.com/docs/manual/administration/security-checklist/
54 changes: 54 additions & 0 deletions docs/Reference/Console/MongoDB/schemaChanges.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,54 @@
# Schema Changes

The Go entity in `internal/entity/<entity>.go` is the single source of truth.
Both backends derive their behavior from it, but each requires distinct
bookkeeping when a field is added, removed, or changed.

| Aspect | SQL backend (schema-on-write) | MongoDB backend (schema-on-read) |
|---|---|---|
| Shape enforcement | database (DDL) | application (repo code) |
| Where the shape is declared | `internal/app/migrations/*.sql` | nowhere — no DDL, no `ALTER TABLE` |
| Repo write obligation | name every column in `Insert` / `Update` / `Scan` | include every field in the `bson.M{"$set": …}` document |
| Missed field at runtime | SQL error: `unknown column` (loud) | field silently dropped from the document (silent) |

!!! warning "Missed fields in `$set` are silent"
Forget a field in `Update`'s `$set` map and the MongoDB driver writes the
document without it — no error is returned. The api-test workflow's
`build-mongo` job (running the full Postman collection against a real
MongoDB) is the authoritative parity check that surfaces this drift.
Comment thread
madhavilosetty-intel marked this conversation as resolved.

## Add a nullable field

| Step | SQL backend | MongoDB backend |
|---|---|---|
| 1. Entity struct | add the field to `internal/entity/<entity>.go` | same file — add a `bson:"<name>"` struct tag |
| 2. Schema | add `<timestamp>_<name>.up.sql` and `.down.sql` under `internal/app/migrations/` | none — schemaless |
| 3. `Insert` | add to `squirrel.Insert(...).Columns(...)` / `.Values(...)` in `internal/usecase/sqldb/<entity>.go` | none — `InsertOne(ctx, e)` serializes the struct by its `bson` tags |
| 4. `Update` | add to the `squirrel.Update(...).Set(...)` chain | add the key to the `bson.M{"$set": …}` map in `internal/usecase/nosqldb/mongo/<entity>.go` |
| 5. `Get` / `Scan` | add to the projection and `rows.Scan(...)` targets | none — `cur.All(...)` / `Decode(...)` populates any field present |
| 6. Index (optional) | migration with `CREATE INDEX` | new `IndexModel` in `ensureIndexes` (`client.go`) |

## Variations

Each row lists only the delta from the procedure above.

| Change | SQL backend | MongoDB backend |
|---|---|---|
| `NOT NULL` with default | migration must include `DEFAULT` to backfill existing rows — e.g., `ALTER TABLE profiles ADD COLUMN uefi_wifi_sync_enabled BOOLEAN NOT NULL DEFAULT FALSE` | existing documents lack the field; reads return the Go zero value. Optional backfill: `db.profiles.updateMany({uefiwifisyncenabled: {$exists: false}}, {$set: {uefiwifisyncenabled: false}})` |
| Unique constraint | migration with `CREATE UNIQUE INDEX ... ON profiles (profilename, tenantid)`; existing duplicates must be removed first | new unique `IndexModel` in `ensureIndexes` (optionally collated); existing duplicates block startup |
| Drop a field | migration `ALTER TABLE devices DROP COLUMN mpspassword` (`.down.sql` re-adds) | stop writing it in `Insert` / `Update`; old documents retain it. Optional cleanup: `db.devices.updateMany({}, {$unset: {mpspassword: 1}})` |
| Rename or change type | dual-write → backfill → switch reads → drop the old shape, across multiple releases | identical multi-release pattern |
| Foreign-key relationship | `FOREIGN KEY` constraint in the migration | no native foreign keys — enforce in the use-case layer (parent lookup before child insert) |

## Files to change together

A change to `internal/entity/<entity>.go` typically touches:

```text
internal/app/migrations/<timestamp>_<name>.up.sql
internal/app/migrations/<timestamp>_<name>.down.sql
internal/usecase/sqldb/<entity>.go
internal/usecase/nosqldb/mongo/<entity>.go
internal/usecase/nosqldb/mongo/fields.go # if used as a filter or sort key
internal/usecase/nosqldb/mongo/client.go # if it needs a unique index
```
5 changes: 3 additions & 2 deletions docs/Reference/Console/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,8 +35,9 @@ Console can also be configured using environment variables. These `.env` variabl
| HTTP_ALLOWED_ORIGINS | http.allowed_origins | `*` | Allowed origins for CORS policy. |
| HTTP_ALLOWED_HEADERS | http.allowed_headers | `*` | Allowed headers for CORS policy. |
| LOG_LEVEL | logger.log_level | `info` | Controls the level of logging. Options: `error`, `warn`, `info`, `debug`, `fatal`. |
| DB_POOL_MAX | db.pool_max | `2` | Maximum number of database connections in the pool. |
| DB_URL | db.url | No Value | By default, Console uses a SQLite database to store device data locally. Users can optionally configure this variable to provide a PostgreSQL connection string, enabling the use of an external PostgreSQL database for data storage. This allows for greater scalability and centralized database management. |
| DB_PROVIDER | db.provider | `sqlite` | Selects the storage backend. Valid values: `postgres`, `sqlite` (default), `mongo`. See [MongoDB](MongoDB/overview.md) for the NoSQL option. |
| DB_POOL_MAX | db.pool_max | `2` | Maximum number of database connections in the pool. Honored by the SQL backends only; the MongoDB driver manages its own pool via the `DB_URL`. See [MongoDB](MongoDB/overview.md) for connection-string pool options. |
| DB_URL | db.url | No Value | Connection string for the selected backend. Examples: `postgres://user:pass@host:5432/dbname` for Postgres; `mongodb://user:pass@host:27017/?authSource=admin` for MongoDB. Leave empty to use the embedded SQLite database. |
| EA_URL | ea.url | `http://localhost:8000` | URL for the Enterprise Assistant service. |
| EA_USERNAME | ea.username | No Value | Username for the Enterprise Assistant service. |
| EA_PASSWORD | ea.password | No Value | Password for the Enterprise Assistant service. |
Expand Down
6 changes: 6 additions & 0 deletions mkdocs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -128,6 +128,12 @@ nav:
- Console:
- Overview: Reference/Console/overview.md
- Configuration: Reference/Console/configuration.md
- MongoDB:
- Overview: Reference/Console/MongoDB/overview.md
- Quick Start: Reference/Console/MongoDB/quickStart.md
- Collections and Indexes: Reference/Console/MongoDB/collections.md
- Schema Changes: Reference/Console/MongoDB/schemaChanges.md
- Adding a New NoSQL DB: Reference/Console/MongoDB/extending.md
- Upgrade: Reference/Console/upgrade.md
- Security Information: Reference/Console/securityConsole.md
- Features:
Expand Down
Loading