diff --git a/.agents/skills/README.md b/.agents/skills/README.md index 988e7cb1f469e..dd0940cb132a7 100644 --- a/.agents/skills/README.md +++ b/.agents/skills/README.md @@ -45,3 +45,14 @@ choosing which framework to use, start with **mz-test**. | Skill | When to use | What it does | |---|---|---| | **mz-adapter-guide** | Working on or asking about the adapter layer | Correctness invariants and architectural notes for the coordinator, pgwire, peek paths, timestamp oracle, and related crates | + +## Demos + +| Skill | When to use | What it does | +|---|---|---| +| **mz-demo-data** | Standing up live, realistic data to show Materialize off | Builds continuously-updating synthetic data entirely in SQL, no Kafka or external load generator. Ships six domains (auctions, ecommerce, banking, IoT, clickstream, zoo) and a rubric for designing new ones | + +Unlike the skills above, `mz-demo-data` is aimed at people evaluating +Materialize rather than developing it. Its assets live in `misc/demo-data/`, +with `.agents/skills/mz-demo-data` as a symlink, so the directory can be +lifted out on its own. diff --git a/.agents/skills/mz-demo-data b/.agents/skills/mz-demo-data new file mode 120000 index 0000000000000..e2d82f34120fa --- /dev/null +++ b/.agents/skills/mz-demo-data @@ -0,0 +1 @@ +../../misc/demo-data \ No newline at end of file diff --git a/misc/demo-data/README.md b/misc/demo-data/README.md new file mode 100644 index 0000000000000..226310b0ba384 --- /dev/null +++ b/misc/demo-data/README.md @@ -0,0 +1,50 @@ +# mz-demo-data + +Continuously-updating, realistic synthetic data for Materialize demos — +generated entirely in SQL, no external load generator. + +Based on [this blog post](https://github.com/frankmcsherry/blog/blob/master/posts/2024-05-19.md). + +## Quickstart + +```sh +PSQL="psql -p 6875 -h localhost -U materialize" + +$PSQL -f assets/scaffold.sql # moments + random (24h window, 1s tick) +$PSQL -f assets/common/people.sql # shared 256-identity pool +$PSQL -f assets/domains/auctions.sql # or any other domain +``` + +Then in a psql session: + +```sql +COPY (SUBSCRIBE (SELECT COUNT(*) FROM auctions) WITH (progress = true)) TO STDOUT; +``` + +To change the retention window or tick: + +```sql +\set retention '6 hours' +\set tick '1 second' +\i assets/scaffold.sql +``` + +Teardown: `$PSQL -f assets/teardown.sql`. + +## Domains + +* **auctions** — marketplace; auctions with lifecycle and bids +* **ecommerce** — orders, line items, totals (joins shared `people`) +* **banking** — double-entry transactions; `SUM(balances) = 0` invariant +* **iot** — devices, readings, threshold alerts +* **clickstream** — sessions, page views, conversion funnel +* **zoo** — zoo visits, ratings, shipments; four invariants at once + +Load more than one to see Materialize keep multiple data products in sync +over the same shared identity space. + +## Designing a new domain + +See [`SKILL.md`](SKILL.md). The short version: derive PKs from `moment`'s +random bytes, derive FKs by re-hashing the parent's bytes, control +distributions with byte masks, and bake an invariant into the construction. diff --git a/misc/demo-data/SKILL.md b/misc/demo-data/SKILL.md new file mode 100644 index 0000000000000..61910ef084c02 --- /dev/null +++ b/misc/demo-data/SKILL.md @@ -0,0 +1,379 @@ +--- +name: mz-demo-data +description: > + Trigger: "demo data", "synthetic data", "live operational data", "generate + fake data", "create a demo", "load generator alternative", or wants + realistic continuously-updating data in Materialize for a demo. Also "I + need streaming data to show off X" or "build a demo schema for ". + Use this to stand up auctions/bids, ecommerce, banking, IoT, or + clickstream demos — or to design a new domain in the same style. +--- + +# Live operational demo data, generated entirely in SQL + +This skill builds continuously-updating, realistic synthetic data inside a +running Materialize instance — no Kafka, no external load generator, no +seed scripts. Everything is plain views over `mz_now()`. + +The technique comes from [this blog post][blog]. Use the catalog at the +bottom for an existing domain; use the rubric to design new ones. + +[blog]: https://github.com/frankmcsherry/blog/blob/master/posts/2024-05-19.md + +## Quickstart + +```sh +psql -p 6875 -h localhost -U materialize -f assets/scaffold.sql +psql -p 6875 -h localhost -U materialize -f assets/common/people.sql +psql -p 6875 -h localhost -U materialize -f assets/domains/auctions.sql +``` + +Re-running `scaffold.sql` or `people.sql` is safe — both short-circuit if +already loaded. Re-running a domain file errors on the second `CREATE +VIEW`; run `assets/teardown.sql` first to rebuild. + +Then in a psql session: + +```sql +COPY (SUBSCRIBE (SELECT COUNT(*) FROM auctions) WITH (progress = true)) TO STDOUT; +``` + +You should see a heartbeat tick once per second. The count grows for the first +24 hours, then stabilizes at retention/tick (default 86,400). + +To change the window or tick rate, set them before `\i`: + +```sql +\set retention '6 hours' +\set tick '1 second' +\i assets/scaffold.sql +``` + +To tear everything down: `psql -f assets/teardown.sql`. + +## First contact: walking a fresh user from zero to demo + +**Read this section first if you're an agent and a user shows up wanting a +demo.** The skill is meant to feel guided, not like a SQL reference. Drive +the user through these turns: + +### Turn 1 — establish the connection + +Ask once if it isn't already in context: + +> What's the connection? A `psql` command, a `MATERIALIZE_URL`, or +> host/port/user/db? + +Accept any of: a connection string, env var, or the four flags. Don't +proceed until you can run `psql -c 'SELECT mz_version();'` and get a row +back. If they say "localhost defaults", use +`psql -p 6875 -h localhost -U materialize -d materialize`. + +### Turn 2 — pick a domain + +If the user named one ("show me a banking demo"), check the catalog at the +bottom of this file. If it matches, jump to Turn 5 with that file. + +If they have a domain in mind that isn't in the catalog ("I work in +logistics", "we do telecom billing", "give me healthcare"), continue to +Turn 3 — we'll build a new domain together. **Do not** force-fit their +domain onto one in the catalog; that's worse than a fresh one. + +If they don't have a domain in mind, suggest 2–3 from the catalog with a +one-line hook each and let them pick. + +### Turn 3 — propose the model in plain language + +**Not SQL yet.** Show the user the entities and relationships in English, +plus the invariant you'd bake in. Format: + +> Here's what I'd build: +> - **\** (one per moment) — fields: a, b, c +> - **\** (3–5 per top-level) — fields: x, y +> - **\** (static, N rows) — purpose +> +> **Invariant baked in:** \ +> +> **Joins to people?** \ +> +> Sound right, or different entities? + +Wait for their reaction. **Don't write SQL until they sign off.** They will +usually correct one of: an entity name, a missing field, the invariant, or +the cardinality. Cheap to iterate here, expensive later. + +### Turn 4 — show the byte budget + +Once entities are confirmed, show the byte allocation explicitly: + +> Per top-level row, 16 random bytes: +> ``` +> [0..2] id 24-bit +> [3] person_id mod 256, FK to people +> [4] kind mod N +> [5] n_children 1..K +> [6] time offset for due_at +> [7..] free +> ``` +> Children re-hash `(parent.random || child_index)` for their own bytes. + +This is the last cheap-to-change step. If the user wants higher cardinality +in some field, swap bytes here, not later. + +### Turn 5 — write and load + +Copy `assets/domains/_template.sql` to `assets/domains/.sql` and +fill it in following the byte budget. Then: + +```sh +# scaffold is idempotent — safe to run even if already loaded. +psql ... -f assets/scaffold.sql +psql ... -f assets/common/people.sql # only if domain joins people +psql ... -f assets/domains/.sql +``` + +If a previous load of THIS domain is present, scaffold will skip silently +but the domain file will fail on `CREATE VIEW ... already exists`. In that +case, run `assets/teardown.sql` first to wipe and reload. + +### Turn 6 — prove it works, hand over the wheel + +Show the user three queries: + +1. **Heartbeat** — confirms data is flowing: + ```sql + COPY (SUBSCRIBE (SELECT COUNT(*) FROM ) WITH (progress = true)) TO STDOUT; + ``` +2. **Invariant** — confirms correctness; should return the expected fixed + value (usually 0). +3. **One "cool" query** — the demo payoff. For aggregations, + `SELECT ... GROUP BY ...`; for cross-domain, a join through `people`. + +Then offer: "Want to layer another domain on top? You'll see joins through +the shared `people` table stay live too." + +### Things to *not* do during first contact + +- Don't dump the catalog up front. The user came with a domain or wants + guidance — give them one path, not a menu. +- Don't ship SQL without the plain-language proposal first. +- Don't skip the byte budget. It's where misalignments hide (e.g., two + fields sharing byte 5 by accident). +- Don't claim "it's done" until the heartbeat ticks and the invariant + query returns its expected value against THEIR Materialize instance. +- Don't add columns the user didn't ask for. The template's free bytes + exist; leave them free unless the user names a use. + +## The two-layer model + +Every demo built with this skill has exactly two layers: + +1. **Scaffold** (stable). The `moments` view — a sliding window of timestamps + — and the `random` view — MD5(moment) producing 16 deterministic bytes per + moment. **Do not edit per-domain.** Always loaded first; the same scaffold + serves every domain. Lives in `assets/scaffold.sql`. + +2. **Domain** (creative). Views that turn `random` bytes into entities, + relationships, and aggregates. One file per domain in `assets/domains/`. + Domains can compose: load multiple and they cross-join naturally via the + shared `people` table. + +Knowing where the line is matters. If you find yourself wanting to edit the +scaffold from a domain file, you almost certainly want to add another view +on top instead. + +## The four design rules for a domain + +Every domain in the catalog follows these four rules. Apply them when +designing a new one. + +### 1. PKs are derived deterministically from `moment` + +A row's primary key is a function of its moment's random bytes: + +```sql +get_byte(random, 0) + get_byte(random, 1) * 256 + get_byte(random, 2) * 65536 AS id +``` + +**Why:** the same moment always yields the same id, so re-derivation is stable +even though the underlying view is a sliding window. PKs naturally vanish from +the system when their moment falls out of retention. + +### 2. FKs come from re-derivation, not declared constraints + +A child row is generated by re-hashing the parent's random bytes: + +```sql +WITH expanded AS ( + SELECT id AS parent_id, ... + digest(random::text || generate_series(1, n_children)::text, 'md5') AS random + FROM parent_core +) +SELECT ... FROM expanded; +``` + +**Why:** when the parent's moment falls out of retention, every child derived +from it vanishes simultaneously. Referential integrity for free, no +`FOREIGN KEY` declarations needed. This is the key insight that makes the +whole approach work. + +### 3. Distributions are byte-mask choices + +Cardinality is controlled by which bytes you read: + +| Pattern | Cardinality | +|----------------------------------------|------------------| +| `get_byte(random, 0)` | 256 | +| `get_byte(random, 0) + get_byte(random, 1) * 256` | 65,536 | +| `mod(get_byte(random, 0), 5)` | 5 | +| `get_byte(random, 0) < 80` | ~31% boolean | + +**Why:** explicit and predictable. A 256-account banking demo uses one byte +mod 64 for from-account; a 16M-id auction marketplace uses three bytes. Pick +the cardinality you need. + +### 4. Evolution is `moment + interval` + +Time-relative fields are derived from the moment plus a random offset: + +```sql +moment + (get_byte(random, 6)::text || ' minutes')::interval AS end_time +``` + +**Why:** the field is monotone in the moment, so downstream views can filter +on it sensibly. Combined with rule #1, this gives you a temporal lifecycle +(start, end, expiry) tied to the entity's identity. + +## Invariant-by-construction patterns + +The strongest demos rely on invariants that **cannot be violated** because of +how the data is constructed. Two patterns: + +**Sum-to-zero (double entry).** Each event emits two child rows with opposite +signs (`banking.sql`). `SUM(ledger_entries.amount) = 0` holds at every +consistent timestamp, regardless of concurrent transaction volume. This is +the headline Materialize correctness demo and it's almost impossible to fake +with off-the-shelf data. + +**Count matches declared fanout.** A parent row declares `n_items`, and uses +`generate_series(1, n_items)` to emit children. `COUNT(children) per parent = +parent.n_items` is invariant (`ecommerce.sql`). + +When designing a new domain, look for an invariant of this form. A demo +without one is weaker — it shows speed but not correctness. + +## Invariants vs. id collisions + +The blog's construction derives primary keys from 24 bits of MD5 entropy +(`get_byte(random, 0..2)` packed into an int). At default settings — 86,400 +moments in retention — the birthday paradox guarantees roughly 230–500 +id collisions in steady state. **This is intentional and expected.** The +blog accepts it; this skill keeps it for fidelity to the blog and to keep +parent ids in a tractable range. + +Collisions affect which invariant shapes you can honestly claim: + +**Survives collisions (claim freely):** +- Aggregate equalities. `COUNT(children) = SUM(n_children over all parents)`. +- Sum-to-zero. `SUM(ledger_entries.amount) = 0`. +- Subset / FK by re-derivation. `every child.parent_id has matching parent` — + re-derivation guarantees this even when multiple parents share an id (the + child joins to *some* row with that id). +- Distribution-shape claims. `~10% of page_views are /checkout`. + +**Does NOT survive collisions (don't claim, or restate as aggregate):** +- Per-parent fanout equality. `for every parent, COUNT(children with this id) = parent.n_children`. + Two parents sharing an id will sum their children, breaking the equality. +- Per-id uniqueness. `every id appears exactly once`. +- Per-row temporal monotonicity that depends on id grouping. + +When the user asks "what's the invariant," default to the aggregate form +unless the construction makes per-id claims structurally true (e.g. +double-entry, where each transaction emits exactly two entries no matter +what id arithmetic does). + +If a demo *requires* unique ids (e.g. a customer-facing dashboard that +treats id as a key), the fix is to either widen the id space (use 4+ +bytes), use `EXTRACT(EPOCH FROM moment)::bigint` as the id (collision-free +by construction), or accept ~0.5% noise as a feature ("real data is +messy"). All three are reasonable; the catalog uses the blog's 24-bit +form for fidelity. + +## Adapting to a new domain + +When walking a user through this, follow the **First contact** protocol +above — propose-then-confirm, don't write SQL upfront. The mechanical +checklist is below. + +To add `assets/domains/.sql`, work through these steps in order: + +1. **List the entities.** Top-level (one per moment), child rows (per parent + via fanout), shared lookups (static). +2. **Pick the byte budget for each top-level entity.** Random gives you 16 + bytes per moment. Allocate bytes to fields by cardinality: + id (3 bytes), FK to people (1 byte mod 256), category (1 byte mod N), + timing offsets (1 byte), free for amount/value (2–3 bytes). +3. **Decide which existing shared tables you reference.** Today: `people`. If + you reference people, your domain joins the cross-domain "same person + appears in multiple data products" story automatically. +4. **Identify the invariant.** What sum, count, or equality must hold by + construction? Bake it in via fanout or two-legged emission, not as a + `CHECK` constraint. +5. **Copy the template.** `cp assets/domains/_template.sql assets/domains/.sql` + and fill in the TODOs. The template already encodes the standard + `_core` view → public MV → child MV → aggregate VIEW pattern and an + idempotency guard. Keep that structure. +6. **Add two validation queries** at the bottom: a `SUBSCRIBE` heartbeat + and an invariant query that should always return 0 (or a fixed value). + +## Validation + +For every domain, two queries: + +```sql +-- Heartbeat: confirms the domain is live and producing rows. +COPY (SUBSCRIBE (SELECT COUNT(*) FROM ) WITH (progress = true)) TO STDOUT; + +-- Invariant: should return a fixed value (usually 0). +SELECT ... ; +``` + +If the heartbeat ticks but the invariant drifts, the byte budget is +misaligned — likely two entities sharing the same byte for different +purposes. Re-allocate. + +## The catalog + +| File | Domain | Highlights | Joins `people`? | +|---|---|---|---| +| `domains/auctions.sql` | Auctions & bids | Marketplace; lifecycle (end_time); winning-bid demo | No | +| `domains/ecommerce.sql` | Orders, line items, totals | Multi-row child fanout; `order_totals` aggregate-as-view | Yes | +| `domains/banking.sql` | Accounts, double-entry txns | **`SUM(balances) = 0`** invariant. Strongest correctness demo. | Yes | +| `domains/iot.sql` | Devices, readings, alerts | High cardinality; threshold alerts; per-site rollup | No | +| `domains/clickstream.sql`| Sessions, page views, funnel | Funnel analytics; conversion rate by channel | Yes | +| `domains/zoo.sql` | Zoo visits, ratings, shipments | Four invariants at once, richest domain. Front-of-house ratings correlate with back-of-house skim by construction. | Yes | + +Loading more than one ecommerce/banking/clickstream domain at once gives you +the cross-product demo: "show me Person 042's orders, her transactions, and +her browsing session — all live, all consistent." + +## File map + +``` +misc/demo-data/ +├── SKILL.md ← you are here +├── README.md short human intro +└── assets/ + ├── scaffold.sql moments + random; do not edit per-domain + ├── teardown.sql drops everything + ├── common/ + │ └── people.sql 256-identity pool, shared across domains + └── domains/ + ├── _template.sql ← copy this to start a new domain + ├── auctions.sql + ├── ecommerce.sql + ├── banking.sql + ├── iot.sql + ├── clickstream.sql + └── zoo.sql +``` diff --git a/misc/demo-data/assets/common/people.sql b/misc/demo-data/assets/common/people.sql new file mode 100644 index 0000000000000..f43d531b4068c --- /dev/null +++ b/misc/demo-data/assets/common/people.sql @@ -0,0 +1,40 @@ +-- Copyright Materialize, Inc. and contributors. All rights reserved. +-- +-- Use of this software is governed by the Business Source License +-- included in the LICENSE file at the root of this repository. +-- +-- As of the Change Date specified in that file, in accordance with +-- the Business Source License, use of this software will be governed +-- by the Apache License, Version 2.0. + +-- ============================================================================= +-- Shared identity pool: 256 deterministic people, used as customers / holders +-- / users across multiple domains. Letting ecommerce + banking + clickstream +-- all reference the same id space is the "data products staying in sync" demo: +-- "show me Person 042's orders and her recent transactions, live." +-- +-- 256 is chosen so a single `get_byte(random, N)` picks a person uniformly +-- without modulus. +-- +-- Static (not moment-driven). Names/emails are deterministic for repeatability. +-- +-- Load with: \i common/people.sql (after scaffold.sql) +-- ============================================================================= + +SELECT EXISTS (SELECT 1 FROM mz_views WHERE name = 'people') AS already_loaded \gset +\if :already_loaded +\echo people already loaded; skipping. +\else + +CREATE VIEW people AS +SELECT + id::int AS id, + 'Person ' || lpad(id::text, 3, '0') AS name, + 'person' || lpad(id::text, 3, '0') || '@example.com' AS email, + (ARRAY['US-W','US-E','EU','APAC','LATAM'])[1 + mod(id, 5)] AS region, + digest('person:' || id::text, 'md5') AS attrs +FROM generate_series(0, 255) AS id; + +CREATE DEFAULT INDEX ON people; + +\endif diff --git a/misc/demo-data/assets/domains/_template.sql b/misc/demo-data/assets/domains/_template.sql new file mode 100644 index 0000000000000..30d041ab959fa --- /dev/null +++ b/misc/demo-data/assets/domains/_template.sql @@ -0,0 +1,148 @@ +-- Copyright Materialize, Inc. and contributors. All rights reserved. +-- +-- Use of this software is governed by the Business Source License +-- included in the LICENSE file at the root of this repository. +-- +-- As of the Change Date specified in that file, in accordance with +-- the Business Source License, use of this software will be governed +-- by the Apache License, Version 2.0. + +-- ============================================================================= +-- TEMPLATE — copy to .sql, fill in the TODOs, delete this header. +-- +-- This file is a fill-in-the-blank skeleton for a new domain. It encodes the +-- four design rules from SKILL.md: +-- +-- 1. PKs derived deterministically from `random` bytes +-- 2. FKs by re-derivation (re-hash parent random || child index) +-- 3. Distributions by byte mask +-- 4. Time-relative fields via `moment + interval` +-- +-- Prerequisites: scaffold.sql (always). Add `\i common/people.sql` to the +-- prereq list if your domain references people (recommended for cross-domain +-- demos). +-- ============================================================================= + +-- TODO: replace `domain` with your domain's name (e.g. `trip`, `shipment`). +SELECT EXISTS (SELECT 1 FROM mz_views WHERE name = 'TODO_core') AS already_loaded \gset +\if :already_loaded +\echo TODO domain already loaded; skipping. +\else + +-- ----------------------------------------------------------------------------- +-- (Optional) Static lookups. Use for small enumerations: product types, +-- statuses, depots, channels. Skip if not needed. +-- ----------------------------------------------------------------------------- +-- CREATE VIEW TODO_kinds (id, name) AS VALUES +-- (0, 'Kind A'), +-- (1, 'Kind B'), +-- (2, 'Kind C'); + +-- ----------------------------------------------------------------------------- +-- The `_core` view: raw fields extracted directly from random bytes. +-- +-- Byte budget (16 bytes available; allocate by cardinality): +-- [0..2] entity id 24-bit → ~16M space +-- [3] foreign key to people mod 256 +-- [4] lookup index mod N +-- [5] quantity / count 1 + mod(., K) for fanout +-- [6] minor field / jitter +-- [7..] free +-- +-- Edit the SELECT below to match your byte plan. +-- ----------------------------------------------------------------------------- +CREATE VIEW TODO_core AS +SELECT + moment, + random, + get_byte(random, 0) + + get_byte(random, 1) * 256 + + get_byte(random, 2) * 65536 AS id, + get_byte(random, 3) AS person_id, -- FK into people + mod(get_byte(random, 4)::int, 3) AS kind_id, -- FK into TODO_kinds + 1 + mod(get_byte(random, 5)::int, 5) AS n_children, -- fanout for child rows + moment + (get_byte(random, 6)::text || ' minutes')::interval AS due_at -- time-relative field +FROM random; + +-- ----------------------------------------------------------------------------- +-- The public top-level view. Materialize and join lookups for fast reads. +-- This is what demos and validation queries hit. +-- ----------------------------------------------------------------------------- +CREATE MATERIALIZED VIEW TODO AS +SELECT + c.id, + c.person_id, + p.name AS person_name, + -- k.name AS kind, + c.moment AS started_at, + c.due_at +FROM TODO_core c +JOIN people p ON p.id = c.person_id; +-- LEFT JOIN TODO_kinds k ON k.id = c.kind_id; + +-- ----------------------------------------------------------------------------- +-- (Optional) Child rows via fanout. The pattern: +-- 1. generate_series(1, parent.n_children) to expand +-- 2. digest(parent.random || child_index) to re-hash for each child +-- 3. extract child fields from the new random +-- +-- This is rule #2 — FK by re-derivation. When the parent moment falls out +-- of retention, every child generated from it vanishes simultaneously. +-- ----------------------------------------------------------------------------- +CREATE MATERIALIZED VIEW TODO_children AS +WITH expanded AS ( + SELECT + id AS parent_id, + moment AS parent_moment, + generate_series(1, n_children) AS child_no, + digest(random::text || generate_series(1, n_children)::text, 'md5') AS random + FROM TODO_core +) +SELECT + parent_id, + child_no, + get_byte(random, 0) AS field_a, + get_byte(random, 1) + get_byte(random, 2) * 256 AS field_b, + parent_moment + + (child_no::text || ' seconds')::interval AS occurred_at +FROM expanded; + +-- ----------------------------------------------------------------------------- +-- (Optional) Aggregates. The classic Materialize moment: a view that stays +-- correct as data flows in. Group by some FK and sum/count/max. +-- ----------------------------------------------------------------------------- +CREATE VIEW TODO_rollup AS +SELECT + parent_id, + COUNT(*) AS n, + SUM(field_a) AS total_a, + MAX(field_b) AS max_b +FROM TODO_children +GROUP BY parent_id; + +\endif + +-- ----------------------------------------------------------------------------- +-- Validation queries — paste these into a session to confirm the domain works. +-- ----------------------------------------------------------------------------- +-- +-- Heartbeat — should tick continuously: +-- COPY (SUBSCRIBE (SELECT COUNT(*) FROM TODO) WITH (progress = true)) TO STDOUT; +-- +-- Invariant: every child references an existing parent (should be 0). +-- SELECT COUNT(*) FROM TODO_children c +-- LEFT JOIN TODO t ON t.id = c.parent_id WHERE t.id IS NULL; +-- +-- Invariant: total child rows = total declared fanout (aggregate form). +-- A per-parent version of this looks tempting but fails because the 24-bit +-- random `id` space has birthday collisions at 86k+ rows — see SKILL.md +-- "Invariants vs. id collisions". The aggregate form is unaffected. +-- SELECT (SELECT COUNT(*) FROM TODO_children) = +-- (SELECT SUM(n_children) FROM TODO_core) AS fanout_balances; +-- +-- Add at least one *domain-specific* invariant of your own — that's what +-- makes a demo land. Pick a shape that survives id collisions: sum-to-zero +-- over all rows, total count = declared total, monotone timestamps, +-- mutually-exclusive states, subset relationships (X ⊆ Y). Avoid invariants +-- that depend on synthetic ids being unique. +-- ----------------------------------------------------------------------------- diff --git a/misc/demo-data/assets/domains/auctions.sql b/misc/demo-data/assets/domains/auctions.sql new file mode 100644 index 0000000000000..78493d04f7424 --- /dev/null +++ b/misc/demo-data/assets/domains/auctions.sql @@ -0,0 +1,106 @@ +-- Copyright Materialize, Inc. and contributors. All rights reserved. +-- +-- Use of this software is governed by the Business Source License +-- included in the LICENSE file at the root of this repository. +-- +-- As of the Change Date specified in that file, in accordance with +-- the Business Source License, use of this software will be governed +-- by the Apache License, Version 2.0. + +-- ============================================================================= +-- Auctions & Bids. The canonical example from the blog post. +-- +-- Demonstrates: +-- * deterministic PK derivation from moment hash +-- * FK by re-derivation: bids re-hash their auction's random bytes, so when +-- an auction's moment falls out of the retention window, its bids vanish +-- with it. Referential integrity for free. +-- * temporal lifecycle: auctions have an end_time computed from random bytes. +-- +-- Standalone domain: does not reference `people`. (The seller/buyer ids are a +-- larger space than the shared pool, by design — auctions are typically a +-- long-tail marketplace.) +-- +-- Load with: \i scaffold.sql +-- \i domains/auctions.sql +-- ============================================================================= + +-- Item-type lookup. Five categories cycled via `auction.item % 5`. +CREATE VIEW items (id, item) AS VALUES + (0, 'Signed Memorabilia'), + (1, 'City Bar Crawl'), + (2, 'Best Pizza in Town'), + (3, 'Gift Basket'), + (4, 'Custom Art'); + +-- Raw auction stream. One auction per moment. +-- Byte budget for `random` (16 bytes): +-- [0..2] id (24-bit ⇒ ~16M space, sparse → unique with high prob) +-- [3..4] seller (16-bit ⇒ 65k sellers) +-- [5] item type (used both as item lookup and as bid-count fanout below) +-- [6] auction duration in minutes (0..255) +CREATE VIEW auctions_core AS +SELECT + moment, + random, + get_byte(random, 0) + + get_byte(random, 1) * 256 + + get_byte(random, 2) * 65536 AS id, + get_byte(random, 3) + + get_byte(random, 4) * 256 AS seller, + get_byte(random, 5) AS item, + moment + (get_byte(random, 6)::text || ' minutes')::interval AS end_time +FROM random; + +-- The published auctions table — joined to items, materialized for fast joins. +CREATE MATERIALIZED VIEW auctions AS +SELECT auctions_core.id, seller, items.item, end_time +FROM auctions_core, items +WHERE auctions_core.item % 5 = items.id; + +-- Bids. Each auction spawns up to 255 bids via `generate_series`, where the +-- number is itself a random byte (item / bid-count share a byte by design — +-- popular categories get more bids). +-- +-- Each bid re-hashes (auction.random || bid_index) to get its own 16 bytes. +CREATE MATERIALIZED VIEW bids AS +WITH prework AS ( + SELECT + id AS auction_id, + moment AS auction_start, + end_time AS auction_end, + digest(random::text || generate_series(1, get_byte(random, 5))::text, 'md5') AS random + FROM auctions_core +) +SELECT + get_byte(random, 0) + + get_byte(random, 1) * 256 + + get_byte(random, 2) * 65536 AS id, + get_byte(random, 3) + + get_byte(random, 4) * 256 AS buyer, + auction_id, + get_byte(random, 5)::numeric AS amount, + auction_start + (get_byte(random, 6)::text || ' minutes')::interval AS bid_time +FROM prework; + +-- ----------------------------------------------------------------------------- +-- Validation. Paste either of these into a fresh session to confirm liveness: +-- ----------------------------------------------------------------------------- +-- +-- Heartbeat — should tick continuously, count stays near retention/tick: +-- COPY (SUBSCRIBE (SELECT COUNT(*) FROM auctions) WITH (progress = true)) TO STDOUT; +-- +-- Winning bid per auction (the classic Materialize demo): +-- SELECT auction_id, MAX(amount) AS winning_bid +-- FROM bids GROUP BY auction_id; +-- +-- Invariant: every bid references an existing auction (FK by re-derivation). +-- Should always return 0. +-- SELECT COUNT(*) FROM bids b +-- LEFT JOIN auctions a ON b.auction_id = a.id WHERE a.id IS NULL; +-- +-- Note on bid_time: the bid's time offset and the auction's duration come +-- from INDEPENDENT random bytes, so ~50% of bids fire after their auction +-- ends. That's intentional — late-bid attempts are realistic, and showing +-- Materialize handle them as a "filter to live auctions" view is a fine +-- demo on its own. diff --git a/misc/demo-data/assets/domains/banking.sql b/misc/demo-data/assets/domains/banking.sql new file mode 100644 index 0000000000000..29d46eea516b2 --- /dev/null +++ b/misc/demo-data/assets/domains/banking.sql @@ -0,0 +1,113 @@ +-- Copyright Materialize, Inc. and contributors. All rights reserved. +-- +-- Use of this software is governed by the Business Source License +-- included in the LICENSE file at the root of this repository. +-- +-- As of the Change Date specified in that file, in accordance with +-- the Business Source License, use of this software will be governed +-- by the Apache License, Version 2.0. + +-- ============================================================================= +-- Banking: accounts + double-entry transactions +-- +-- Demonstrates: +-- * the "invariant by construction" pattern: every transaction emits TWO +-- ledger entries summing to zero, so SUM(entries.amount) is trivially 0 +-- at every consistent timestamp. This is Materialize's headline correctness +-- property and is hard to demo with synthetic data any other way. +-- * cross-domain join to `people` for account holders +-- +-- Prerequisites: scaffold.sql, common/people.sql +-- ============================================================================= + +-- Static account directory: 64 accounts. Each held by someone from `people`. +-- account_id is 0..63 so a single byte mod 64 picks one uniformly. +CREATE VIEW accounts AS +SELECT + id::int AS id, + 'ACCT-' || lpad(id::text, 4, '0') AS number, + -- Hash the account id with a salt to pick a holder, so the assignment + -- isn't a trivial id == holder_id mapping. + get_byte(digest('acct:' || id::text, 'md5'), 0) AS holder_id, + (ARRAY['Checking','Savings','Credit'])[1 + mod(id, 3)] AS account_type +FROM generate_series(0, 63) AS id; + +CREATE DEFAULT INDEX ON accounts; + +-- One transaction per moment. +-- Byte budget: +-- [0..2] transaction_id +-- [3] from_account, mod 64 +-- [4] to_account: derived from `from` + offset in [1..63] (mod 64) so the +-- two accounts are guaranteed distinct. +-- [5..6] amount in cents (16-bit ⇒ up to $655.35 per txn) +CREATE VIEW transactions_core AS +SELECT + moment, + random, + get_byte(random, 0) + + get_byte(random, 1) * 256 + + get_byte(random, 2) * 65536 AS id, + mod(get_byte(random, 3)::int, 64) AS from_account, + mod( + mod(get_byte(random, 3)::int, 64) + + 1 + mod(get_byte(random, 4)::int, 63), + 64 + ) AS to_account, + ((get_byte(random, 5) + get_byte(random, 6) * 256))::numeric + / 100.0 AS amount +FROM random; + +CREATE MATERIALIZED VIEW transactions AS +SELECT + t.id, + t.moment AS posted_at, + t.from_account, + t.to_account, + t.amount +FROM transactions_core t; + +-- Double-entry: each transaction emits two ledger rows that sum to zero. +-- generate_series(1,2) fans out each txn; we pick from vs. to by row index. +CREATE MATERIALIZED VIEW ledger_entries AS +WITH expanded AS ( + SELECT id, moment, from_account, to_account, amount, + generate_series(1, 2) AS leg + FROM transactions_core +) +SELECT + id AS transaction_id, + moment AS posted_at, + CASE WHEN leg = 1 THEN from_account ELSE to_account END AS account_id, + CASE WHEN leg = 1 THEN -amount ELSE amount END AS amount +FROM expanded; + +-- Running balance per account. The headline view: stays correct under +-- arbitrary concurrent reads + writes because of strict serializability. +CREATE VIEW account_balances AS +SELECT + a.id, + a.number, + a.account_type, + p.name AS holder_name, + p.region AS holder_region, + COALESCE(SUM(le.amount), 0) AS balance +FROM accounts a +LEFT JOIN ledger_entries le ON le.account_id = a.id +LEFT JOIN people p ON p.id = a.holder_id +GROUP BY a.id, a.number, a.account_type, p.name, p.region; + +-- ----------------------------------------------------------------------------- +-- Validation: +-- +-- Heartbeat (count of in-flight transactions): +-- COPY (SUBSCRIBE (SELECT COUNT(*) FROM transactions) WITH (progress = true)) TO STDOUT; +-- +-- THE invariant: total of all balances is exactly zero. This is the demo: +-- you can hit this query repeatedly while millions of transactions land, +-- and it will return 0 every single time. +-- SELECT SUM(balance) FROM account_balances; +-- +-- Per-holder net position (joins to people, useful for cross-domain demos): +-- SELECT holder_name, SUM(balance) FROM account_balances GROUP BY holder_name; +-- ----------------------------------------------------------------------------- diff --git a/misc/demo-data/assets/domains/clickstream.sql b/misc/demo-data/assets/domains/clickstream.sql new file mode 100644 index 0000000000000..e5859d7102909 --- /dev/null +++ b/misc/demo-data/assets/domains/clickstream.sql @@ -0,0 +1,131 @@ +-- Copyright Materialize, Inc. and contributors. All rights reserved. +-- +-- Use of this software is governed by the Business Source License +-- included in the LICENSE file at the root of this repository. +-- +-- As of the Change Date specified in that file, in accordance with +-- the Business Source License, use of this software will be governed +-- by the Apache License, Version 2.0. + +-- ============================================================================= +-- Clickstream: sessions → page_views → conversions +-- +-- Demonstrates: +-- * funnel analytics as a maintained view +-- * window-like behavior via per-session offsets within the moment +-- * cross-domain join to `people` for user identity +-- +-- Prerequisites: scaffold.sql, common/people.sql +-- ============================================================================= + +-- One session per moment. +-- Byte budget: +-- [0..2] session_id +-- [3] user_id (mod 256 ⇒ people pool) +-- [4] n_views (clamped 3..15) +-- [5] source channel index +-- [6] device-type index +CREATE VIEW sessions_core AS +SELECT + moment, + random, + get_byte(random, 0) + + get_byte(random, 1) * 256 + + get_byte(random, 2) * 65536 AS id, + get_byte(random, 3) AS user_id, + 3 + mod(get_byte(random, 4)::int, 13) AS n_views, + (ARRAY['organic','paid','referral','direct','email']) + [1 + mod(get_byte(random, 5)::int, 5)] AS source, + (ARRAY['desktop','mobile','tablet']) + [1 + mod(get_byte(random, 6)::int, 3)] AS device +FROM random; + +CREATE MATERIALIZED VIEW sessions AS +SELECT + s.id, + s.user_id, + p.name AS user_name, + p.region AS user_region, + s.moment AS started_at, + s.source, + s.device, + s.n_views +FROM sessions_core s +JOIN people p ON p.id = s.user_id; + +-- Page views: each session emits n_views events. Pages are chosen by hash and +-- skewed so /checkout is rarer than /landing (the funnel narrows). +-- Page distribution: low byte values → /landing, /product, /search (common), +-- high byte values → /cart, /checkout (rare). This produces a realistic funnel. +CREATE MATERIALIZED VIEW page_views AS +WITH expanded AS ( + SELECT + id AS session_id, + moment AS session_start, + generate_series(1, n_views) AS view_no, + digest(random::text || generate_series(1, n_views)::text, 'md5') AS random + FROM sessions_core +) +SELECT + session_id, + view_no, + -- Each view offset within the session, in seconds. View N happens at + -- session_start + (N + small jitter) seconds. + session_start + + (view_no::text || ' seconds')::interval + + (mod(get_byte(random, 1)::int, 30)::text || ' seconds')::interval + AS viewed_at, + -- Skewed page picker: bias towards top-of-funnel. + CASE + WHEN get_byte(random, 0) < 80 THEN '/landing' + WHEN get_byte(random, 0) < 150 THEN '/product' + WHEN get_byte(random, 0) < 200 THEN '/search' + WHEN get_byte(random, 0) < 230 THEN '/cart' + ELSE '/checkout' + END AS path +FROM expanded; + +-- Conversions: sessions that reached /checkout. +CREATE VIEW conversions AS +SELECT DISTINCT + pv.session_id, + s.user_id, + s.user_name, + s.source, + s.started_at +FROM page_views pv +JOIN sessions s ON s.id = pv.session_id +WHERE pv.path = '/checkout'; + +-- Funnel: pageview counts per stage, the classic clickstream dashboard. +CREATE VIEW funnel AS +SELECT path, COUNT(*) AS views +FROM page_views +GROUP BY path; + +-- Conversion rate by source channel. A useful "stays correct under load" demo. +CREATE VIEW conversion_by_source AS +SELECT + s.source, + COUNT(DISTINCT s.id) AS sessions, + COUNT(DISTINCT c.session_id) AS conversions, + COUNT(DISTINCT c.session_id)::float + / NULLIF(COUNT(DISTINCT s.id), 0) AS conversion_rate +FROM sessions s +LEFT JOIN conversions c ON c.session_id = s.id +GROUP BY s.source; + +-- ----------------------------------------------------------------------------- +-- Validation: +-- +-- Heartbeat: +-- COPY (SUBSCRIBE (SELECT COUNT(*) FROM sessions) WITH (progress = true)) TO STDOUT; +-- +-- Current funnel: +-- SELECT * FROM funnel ORDER BY views DESC; +-- +-- Cross-domain demo (requires ecommerce.sql also loaded): +-- Which users converted AND placed an order in the window? +-- SELECT c.user_name FROM conversions c +-- WHERE EXISTS (SELECT 1 FROM orders o WHERE o.customer_id = c.user_id); +-- ----------------------------------------------------------------------------- diff --git a/misc/demo-data/assets/domains/ecommerce.sql b/misc/demo-data/assets/domains/ecommerce.sql new file mode 100644 index 0000000000000..354d8947ea3ce --- /dev/null +++ b/misc/demo-data/assets/domains/ecommerce.sql @@ -0,0 +1,130 @@ +-- Copyright Materialize, Inc. and contributors. All rights reserved. +-- +-- Use of this software is governed by the Business Source License +-- included in the LICENSE file at the root of this repository. +-- +-- As of the Change Date specified in that file, in accordance with +-- the Business Source License, use of this software will be governed +-- by the Apache License, Version 2.0. + +-- ============================================================================= +-- E-commerce: orders → line_items → order_totals +-- +-- Demonstrates: +-- * multi-row child generation via generate_series fanout +-- * cross-domain join to shared `people` (customer FK) +-- * aggregate-as-view (`order_totals`) — the classic MV use case +-- +-- Prerequisites: scaffold.sql, common/people.sql +-- ============================================================================= + +-- Static product catalog. Sixteen products keeps line_items tractable. +CREATE VIEW products (id, name, category) AS VALUES + ( 0, 'Coffee Beans 1lb', 'Grocery'), + ( 1, 'Olive Oil 500ml', 'Grocery'), + ( 2, 'Dish Soap', 'Household'), + ( 3, 'Paper Towels 6pk', 'Household'), + ( 4, 'Notebook A5', 'Stationery'), + ( 5, 'Ballpoint Pen 12pk', 'Stationery'), + ( 6, 'USB-C Cable 1m', 'Electronics'), + ( 7, 'Wireless Mouse', 'Electronics'), + ( 8, 'T-Shirt Plain', 'Apparel'), + ( 9, 'Wool Socks', 'Apparel'), + (10, 'Yoga Mat', 'Fitness'), + (11, 'Resistance Band Set', 'Fitness'), + (12, 'Cast Iron Skillet', 'Kitchen'), + (13, 'Chef Knife 8in', 'Kitchen'), + (14, 'Houseplant Pothos', 'Home'), + (15, 'Candle Lavender', 'Home'); + +-- One order per moment. +-- Byte budget for `random`: +-- [0..2] order id (24-bit space) +-- [3] customer_id (mod 256 ⇒ aligns with people pool) +-- [6] n_items, clamped to 1..8 +CREATE VIEW orders_core AS +SELECT + moment, + random, + get_byte(random, 0) + + get_byte(random, 1) * 256 + + get_byte(random, 2) * 65536 AS id, + get_byte(random, 3) AS customer_id, + 1 + mod(get_byte(random, 6)::int, 8) AS n_items +FROM random; + +CREATE MATERIALIZED VIEW orders AS +SELECT + o.id, + o.customer_id, + p.name AS customer_name, + p.region AS customer_region, + o.moment AS placed_at, + o.n_items +FROM orders_core o +JOIN people p ON p.id = o.customer_id; + +-- Line items: each order spawns n_items via generate_series, each line +-- re-hashes (order.random || line_no) for its own bytes. +-- Line-item byte budget: +-- [0..1] product_id (mod 16) +-- [2] qty (1..8) +-- [3..5] unit_price_cents (24-bit ⇒ up to ~$167k; realistic skew via mask) +CREATE MATERIALIZED VIEW line_items AS +WITH expanded AS ( + SELECT + id AS order_id, + generate_series(1, n_items) AS line_no, + digest(random::text || generate_series(1, n_items)::text, 'md5') AS random + FROM orders_core +) +SELECT + order_id, + line_no, + mod(get_byte(random, 0) + get_byte(random, 1) * 256, 16) AS product_id, + 1 + mod(get_byte(random, 2)::int, 8) AS qty, + -- Prices skewed small: 24-bit cents, but we keep low byte dominant + -- so most items are cheap and a few are pricey. + (get_byte(random, 3) + + get_byte(random, 4) * 256 + + get_byte(random, 5) * 4)::numeric / 100.0 AS unit_price +FROM expanded; + +-- Aggregate-as-view. THIS is the demo: the running sum stays consistent with +-- line_items in real time, no batch job, no consistency window. +CREATE VIEW order_totals AS +SELECT + li.order_id, + SUM(li.qty * li.unit_price) AS total +FROM line_items li +GROUP BY li.order_id; + +-- Customer running spend across the retention window. Shows people-side joins. +CREATE VIEW customer_spend AS +SELECT + o.customer_id, + o.customer_name, + COUNT(DISTINCT o.id) AS orders_placed, + SUM(li.qty * li.unit_price) AS total_spent +FROM orders o +JOIN line_items li ON li.order_id = o.id +GROUP BY o.customer_id, o.customer_name; + +-- ----------------------------------------------------------------------------- +-- Validation: +-- +-- Heartbeat: +-- COPY (SUBSCRIBE (SELECT COUNT(*) FROM orders) WITH (progress = true)) TO STDOUT; +-- +-- Invariant: every line_item references an existing order (should be 0). +-- SELECT COUNT(*) FROM line_items li +-- LEFT JOIN orders o ON o.id = li.order_id WHERE o.id IS NULL; +-- +-- Invariant: total line-items = total declared fanout, in aggregate. +-- (A per-order version of this fails because the 24-bit `id` space has +-- birthday-paradox collisions at 86k+ rows — see SKILL.md "Invariants vs. +-- id collisions". The aggregate form is unaffected since both sides count +-- rows regardless of id.) +-- SELECT (SELECT COUNT(*) FROM line_items) = +-- (SELECT SUM(n_items) FROM orders_core) AS fanout_balances; +-- ----------------------------------------------------------------------------- diff --git a/misc/demo-data/assets/domains/iot.sql b/misc/demo-data/assets/domains/iot.sql new file mode 100644 index 0000000000000..401e045a668db --- /dev/null +++ b/misc/demo-data/assets/domains/iot.sql @@ -0,0 +1,114 @@ +-- Copyright Materialize, Inc. and contributors. All rights reserved. +-- +-- Use of this software is governed by the Business Source License +-- included in the LICENSE file at the root of this repository. +-- +-- As of the Change Date specified in that file, in accordance with +-- the Business Source License, use of this software will be governed +-- by the Apache License, Version 2.0. + +-- ============================================================================= +-- IoT: devices + readings + alerts +-- +-- Demonstrates: +-- * high-cardinality fan-out (multiple readings per moment) +-- * per-device aggregates over a sliding window (running avg, max) +-- * threshold-based alerting as a continuously-maintained view +-- +-- Standalone domain: no `people` join. Devices are their own identity space. +-- +-- Prerequisites: scaffold.sql +-- ============================================================================= + +-- Static device fleet of 128 devices. Each has a sensor type and a per-device +-- alert threshold. Threshold derived from the device id so it's stable. +CREATE VIEW devices AS +SELECT + id::int AS id, + 'dev-' || lpad(id::text, 4, '0') AS device_name, + (ARRAY['temperature','pressure','humidity','vibration']) + [1 + mod(id, 4)] AS sensor_type, + -- Site picked deterministically from the id; 16 sites of ~8 devices each. + 'site-' || lpad((mod(id, 16))::text, 2, '0') AS site, + -- Threshold: most devices ~200, a few aggressive ones ~150. + 150 + mod(id * 7, 100) AS threshold +FROM generate_series(0, 127) AS id; + +CREATE DEFAULT INDEX ON devices; + +-- Readings. Each moment emits 8 readings, each one re-hashing to pick a +-- device and a value. Total cardinality ≈ retention_seconds × 8. +-- Reading byte budget (per re-hash): +-- [0] device_id (mod 128) +-- [1..2] reading value (16-bit, scaled into roughly the threshold range) +CREATE MATERIALIZED VIEW readings AS +WITH expanded AS ( + SELECT + moment, + digest(random::text || generate_series(1, 8)::text, 'md5') AS random, + generate_series(1, 8) AS slot + FROM random +) +SELECT + moment AS observed_at, + slot, + mod(get_byte(random, 0)::int, 128) AS device_id, + (get_byte(random, 1) + get_byte(random, 2) * 256) + ::numeric / 100.0 AS value +FROM expanded; + +-- Per-device running aggregates over the live retention window. This is the +-- IoT demo: a streaming dashboard that stays current. +CREATE VIEW device_stats AS +SELECT + d.id AS device_id, + d.device_name, + d.sensor_type, + d.site, + COUNT(*) AS readings_in_window, + AVG(r.value) AS avg_value, + MAX(r.value) AS max_value, + MIN(r.value) AS min_value +FROM devices d +JOIN readings r ON r.device_id = d.id +GROUP BY d.id, d.device_name, d.sensor_type, d.site; + +-- Alerts: readings exceeding their device's threshold. A continuously +-- maintained, joinable, streaming alert table. +CREATE MATERIALIZED VIEW alerts AS +SELECT + r.observed_at, + r.device_id, + d.device_name, + d.sensor_type, + d.site, + r.value, + d.threshold, + r.value - d.threshold AS overshoot +FROM readings r +JOIN devices d ON d.id = r.device_id +WHERE r.value > d.threshold; + +-- Per-site alert load. Useful for "which site is hot right now" dashboards. +CREATE VIEW site_alert_load AS +SELECT + site, + COUNT(*) AS active_alerts, + MAX(overshoot) AS worst_overshoot +FROM alerts +GROUP BY site; + +-- ----------------------------------------------------------------------------- +-- Validation: +-- +-- Heartbeat: +-- COPY (SUBSCRIBE (SELECT COUNT(*) FROM readings) WITH (progress = true)) TO STDOUT; +-- +-- Top noisy sites right now: +-- SELECT * FROM site_alert_load ORDER BY active_alerts DESC LIMIT 5; +-- +-- Invariant: every alert is reachable from a reading (FK integrity). 0 expected: +-- SELECT COUNT(*) FROM alerts a +-- LEFT JOIN readings r ON r.observed_at = a.observed_at AND r.device_id = a.device_id +-- WHERE r.device_id IS NULL; +-- ----------------------------------------------------------------------------- diff --git a/misc/demo-data/assets/domains/zoo.sql b/misc/demo-data/assets/domains/zoo.sql new file mode 100644 index 0000000000000..a3a27535fae10 --- /dev/null +++ b/misc/demo-data/assets/domains/zoo.sql @@ -0,0 +1,411 @@ +-- Copyright Materialize, Inc. and contributors. All rights reserved. +-- +-- Use of this software is governed by the Business Source License +-- included in the LICENSE file at the root of this repository. +-- +-- As of the Change Date specified in that file, in accordance with +-- the Business Source License, use of this software will be governed +-- by the Apache License, Version 2.0. + +-- ============================================================================= +-- Zoo of Disappointing Animals (a front for mostly crime) +-- +-- Two layers, one schema: +-- Front of house: visits, ratings of 64 disappointing animals. +-- Back of house: shipments with consistent skim, "consultations" that only +-- happen on skeptical-mood visits. +-- +-- The correlation between disappointment and crime is baked in: skeptical +-- guests rate harsher AND trigger consultations, so hours with high average +-- disappointment also have higher envelope totals. The cover IS the tell. +-- +-- Invariants: +-- 1. COUNT(ratings) per visit = visits.n_animals_seen (fanout) +-- 2. SUM(shipments.declared - actual) is monotone non-positive (skim) +-- 3. consultations.visit_id ⊆ visits where mood = 'skeptical' (subset) +-- 4. consultations.client_id is always a real visitor (re-derivation) +-- +-- Prerequisites: scaffold.sql, common/people.sql +-- ============================================================================= + +SELECT EXISTS (SELECT 1 FROM mz_views WHERE name = 'visits_core') AS already_loaded \gset +\if :already_loaded +\echo zoo already loaded; skipping. +\else + +-- ----------------------------------------------------------------------------- +-- Static lookups +-- ----------------------------------------------------------------------------- + +CREATE VIEW moods (id, name) AS VALUES + (0, 'hopeful'), + (1, 'skeptical'), + (2, 'dragged_along'), + (3, 'field_trip'); + +CREATE VIEW crate_labels (id, name) AS VALUES + (0, 'fresh hay'), + (1, 'veterinary supplies'), + (2, 'enrichment toys'), + (3, 'do not open'); + +CREATE VIEW service_codes (id, name) AS VALUES + (0, 'tax advisory'), + (1, 'logistics'), + (2, 'asset relocation'), + (3, 'reputation management'), + (4, 'witness preparation'), + (5, 'inventory adjustment'), + (6, 'jurisdictional planning'), + (7, 'unspecified consulting'); + +-- Species roster (the brochure version). +CREATE VIEW species (id, name, expected_disappointment) AS VALUES + (0, 'Capybara', 2), + (1, 'Sleeping Lion', 5), + (2, 'Pigeon (Allegedly Eagle)', 5), + (3, 'Sloth', 3), + (4, 'Empty Enclosure', 5), + (5, 'Goat', 2), + (6, 'Tiger (Painted Mule)', 5), + (7, 'Lizard in a Jar', 4), + (8, 'Bear (Costume)', 4), + (9, 'Penguin (Plastic)', 5), + (10, 'Rabbit', 1), + (11, 'Snake (Garden Hose)', 5); + +-- 64 animals, each with a name, species, and a specific letdown reason. +-- animal_id is 0..63 so one byte mod 64 picks uniformly. +CREATE VIEW animals AS +SELECT + id::int AS id, + 'ANI-' || lpad(id::text, 3, '0') AS tag, + (ARRAY[ + 'Bartholomew','Mister Whiskers','Doctor Paws','Sir Naps-a-Lot', + 'Greg','The Disappointment','Beatrice','Captain Snore', + 'Marquis de Lounge','Pebbles','Twiggy','Mayor McMolt', + 'Lord Featherbottom','Janet','Cinder','Pudding', + 'Stumpy','Princess Vacant','Reginald','The Smell', + 'Ottoman','Doorstop','Ms. Hibernation','Mister Allegedly', + 'The Glistener','Frank','Fenestra','Dim Bulb', + 'Sergeant Nope','Roy','Crouton','The Refunder', + 'Hazelnut','Hugo','Mavis','Crinkle', + 'The Audit','Snickers','Phyllis','Buttercup', + 'Vince','Mister Eyes','Lump','The Suggestion', + 'Pretzel','Boris','Calamity','Tuffet', + 'Dr. Disappointment','The Husk','Marbles','The Receipt', + 'Twitch','The Bystander','Norbert','Cricket', + 'The Witness','Edna','Lasagna','The Defendant', + 'Pinto','Whimsy','The Inheritance','The Final Straw' + ])[1 + id] AS name, + mod(get_byte(digest('species:' || id::text, 'md5'), 0)::int, 12) AS species_id, + (ARRAY[ + 'sleeps 22 hours a day', + 'is actually a rock with eyes glued on', + 'smaller than the brochure photo', + 'technically a pigeon', + 'enclosure is just a mirror', + 'has not moved since 2019', + 'turns out to be a costume', + 'enclosure card reads "TBD"', + 'visible only on tuesdays', + 'painted to look like another animal', + 'is a taxidermy with a fan blowing on it', + 'smells worse than expected', + 'enclosure contains only a sign reading SOON', + 'absent due to "training"', + 'considerably damper than promised' + ])[1 + mod(get_byte(digest('reason:' || id::text, 'md5'), 0)::int, 15)] AS letdown_reason +FROM generate_series(0, 63) AS id; + +CREATE DEFAULT INDEX ON animals; + +-- ----------------------------------------------------------------------------- +-- Visits: one per moment. +-- +-- Byte budget on `random`: +-- [0..2] visit_id +-- [3] visitor_id mod 256 (FK to people) +-- [4] mood mod 4 +-- [5] n_animals_seen = (byte mod 5) + 1 +-- [6] arrival offset in seconds +-- [7] consultation count seed (mod 3, gated by mood = skeptical) +-- ----------------------------------------------------------------------------- +CREATE VIEW visits_core AS +SELECT + moment, + random, + get_byte(random, 0) + + get_byte(random, 1) * 256 + + get_byte(random, 2) * 65536 AS id, + get_byte(random, 3) AS visitor_id, + mod(get_byte(random, 4)::int, 4) AS mood_id, + 1 + mod(get_byte(random, 5)::int, 5) AS n_animals_seen, + moment + (get_byte(random, 6)::text || ' seconds')::interval AS arrived_at, + mod(get_byte(random, 7)::int, 3) AS consultation_seed +FROM random; + +CREATE MATERIALIZED VIEW visits AS +SELECT + v.id, + v.visitor_id, + p.name AS visitor_name, + p.region AS visitor_region, + m.name AS mood, + v.n_animals_seen, + v.arrived_at, + -- Consultations only happen when the guest is "skeptical". + CASE WHEN m.name = 'skeptical' THEN v.consultation_seed ELSE 0 END + AS n_consultations +FROM visits_core v +JOIN people p ON p.id = v.visitor_id +JOIN moods m ON m.id = v.mood_id; + +-- ----------------------------------------------------------------------------- +-- Ratings: one row per (visit, animal seen). Re-hash random with the rating +-- index. Skeptical guests rate +2 stars harsher (clamped to 5). That single +-- nudge produces the correlation between front-of-house disappointment and +-- back-of-house activity. +-- +-- Child random bytes: +-- [0] animal_id mod 64 +-- [1] base stars = (byte mod 5) + 1; bumped to +2 if parent mood skeptical +-- [2] viewed_at offset (seconds after arrival) +-- ----------------------------------------------------------------------------- +CREATE MATERIALIZED VIEW ratings AS +WITH expanded AS ( + SELECT + v.id AS visit_id, + v.arrived_at, + v.mood_id, + generate_series(1, v.n_animals_seen) AS rating_no, + digest(v.random::text || 'rating' || + generate_series(1, v.n_animals_seen)::text, 'md5') AS random + FROM visits_core v +) +SELECT + visit_id, + rating_no, + mod(get_byte(random, 0)::int, 64) AS animal_id, + LEAST( + 5, + 1 + mod(get_byte(random, 1)::int, 5) + + CASE WHEN mood_id = 1 THEN 2 ELSE 0 END + ) AS stars, + arrived_at + (get_byte(random, 2)::text || ' seconds')::interval + AS viewed_at +FROM expanded; + +-- ----------------------------------------------------------------------------- +-- Shipments: one per moment, re-hashing `random || 'shipment'`. Independent +-- entropy from visits so the two streams are not trivially correlated. +-- +-- Byte budget: +-- [0..2] shipment_id +-- [3] handler_id mod 256 (FK to people) +-- [4] crate_label mod 4 +-- [5] declared_weight = byte + 50 (50..305 kg) +-- [6] skim = byte mod 8 (subtracted, always) +-- ----------------------------------------------------------------------------- +CREATE VIEW shipments_core AS +SELECT + moment, + digest(random::text || 'shipment', 'md5') AS random +FROM random; + +CREATE MATERIALIZED VIEW shipments AS +SELECT + moment AS arrived_at, + get_byte(random, 0) + + get_byte(random, 1) * 256 + + get_byte(random, 2) * 65536 AS id, + get_byte(random, 3) AS handler_id, + p.name AS handler_name, + cl.name AS crate_label, + (get_byte(random, 5) + 50)::numeric AS declared_weight_kg, + ((get_byte(random, 5) + 50) - mod(get_byte(random, 6)::int, 8))::numeric + AS actual_weight_kg, + mod(get_byte(random, 6)::int, 8)::numeric AS skim_kg +FROM shipments_core +JOIN people p ON p.id = get_byte(random, 3) +JOIN crate_labels cl ON cl.id = mod(get_byte(random, 4)::int, 4); + +-- ----------------------------------------------------------------------------- +-- Consultations: 0..2 child rows per visit, gated by mood = skeptical. +-- client_id is CARRIED from the parent visit (not re-rolled) — that's how +-- "the guest meeting" maps to "the visitor in the cover story". +-- +-- Child random bytes: +-- [0..2] consultation_id +-- [4] service_code mod 8 +-- [5] envelope_mm = byte (0..255) +-- ----------------------------------------------------------------------------- +CREATE MATERIALIZED VIEW consultations AS +WITH gated AS ( + SELECT + v.id AS visit_id, + v.visitor_id AS client_id, + v.arrived_at, + v.consultation_seed, + v.random + FROM visits_core v + JOIN moods m ON m.id = v.mood_id + WHERE m.name = 'skeptical' + AND v.consultation_seed > 0 +), +expanded AS ( + SELECT + visit_id, + client_id, + arrived_at, + generate_series(1, consultation_seed) AS meeting_no, + digest(random::text || 'consult' || + generate_series(1, consultation_seed)::text, 'md5') AS random + FROM gated +) +SELECT + e.visit_id, + e.meeting_no, + get_byte(e.random, 0) + + get_byte(e.random, 1) * 256 + + get_byte(e.random, 2) * 65536 AS id, + e.client_id, + p.name AS client_name, + sc.name AS service_code, + get_byte(e.random, 5)::int AS envelope_mm, + e.arrived_at + (e.meeting_no * 7 || ' minutes')::interval AS met_at +FROM expanded e +JOIN people p ON p.id = e.client_id +JOIN service_codes sc ON sc.id = mod(get_byte(e.random, 4)::int, 8); + +-- ----------------------------------------------------------------------------- +-- Aggregates: the demos. +-- ----------------------------------------------------------------------------- + +-- The headline join. Per-hour: how disappointing was the cover, and how busy +-- was the back of house? High avg_disappointment should track high envelope +-- totals, because both ride the skeptical-mood bytes. +CREATE VIEW cover_quality AS +WITH visit_disappointment AS ( + SELECT v.id AS visit_id, v.arrived_at, AVG(r.stars)::numeric(10,2) AS avg_stars + FROM visits v + JOIN ratings r ON r.visit_id = v.id + GROUP BY v.id, v.arrived_at +), +hourly_visits AS ( + SELECT + date_trunc('hour', arrived_at) AS hour, + COUNT(*) AS visits, + AVG(avg_stars)::numeric(10,2) AS avg_disappointment + FROM visit_disappointment + GROUP BY date_trunc('hour', arrived_at) +), +hourly_shipments AS ( + SELECT + date_trunc('hour', arrived_at) AS hour, + SUM(skim_kg) AS skim_kg + FROM shipments + GROUP BY date_trunc('hour', arrived_at) +), +hourly_consults AS ( + SELECT + date_trunc('hour', met_at) AS hour, + COUNT(*) AS consultations, + SUM(envelope_mm) AS envelope_mm_total + FROM consultations + GROUP BY date_trunc('hour', met_at) +) +SELECT + v.hour, + v.visits, + v.avg_disappointment, + COALESCE(s.skim_kg, 0) AS skim_kg, + COALESCE(c.consultations, 0) AS consultations, + COALESCE(c.envelope_mm_total, 0) AS envelope_mm_total +FROM hourly_visits v +LEFT JOIN hourly_shipments s ON s.hour = v.hour +LEFT JOIN hourly_consults c ON c.hour = v.hour; + +-- Which animals attract the worst reviews? (The leaderboard of letdowns.) +CREATE VIEW animal_disappointment AS +SELECT + a.tag, + a.name, + sp.name AS species, + a.letdown_reason, + COUNT(*) AS sightings, + AVG(r.stars)::numeric(10,2) AS avg_stars +FROM ratings r +JOIN animals a ON a.id = r.animal_id +JOIN species sp ON sp.id = a.species_id +GROUP BY a.tag, a.name, sp.name, a.letdown_reason; + +-- Which services run hottest, and under how much cover? +CREATE VIEW business_efficiency AS +WITH visit_cover AS ( + SELECT v.id AS visit_id, AVG(r.stars)::numeric(10,2) AS cover_score + FROM visits v JOIN ratings r ON r.visit_id = v.id + GROUP BY v.id +) +SELECT + c.service_code, + COUNT(*) AS meetings, + AVG(c.envelope_mm)::numeric(10,2) AS avg_envelope_mm, + SUM(c.envelope_mm) AS total_envelope_mm, + AVG(vc.cover_score)::numeric(10,2) AS avg_cover_disappointment +FROM consultations c +JOIN visit_cover vc ON vc.visit_id = c.visit_id +GROUP BY c.service_code; + +-- Cross-domain hook: per-person rap sheet. Joins client_id back through +-- people for use with banking / ecommerce demos. +CREATE VIEW client_activity AS +SELECT + p.id AS person_id, + p.name AS person_name, + p.region, + COUNT(*) AS consultations, + SUM(c.envelope_mm) AS envelope_mm_total +FROM consultations c +JOIN people p ON p.id = c.client_id +GROUP BY p.id, p.name, p.region; + +\endif + +-- ----------------------------------------------------------------------------- +-- Validation queries: +-- +-- Heartbeat: +-- COPY (SUBSCRIBE (SELECT COUNT(*) FROM visits) WITH (progress = true)) TO STDOUT; +-- +-- Invariant 1: total ratings = total declared animals seen (aggregate fanout). +-- SELECT (SELECT COUNT(*) FROM ratings) = +-- (SELECT SUM(n_animals_seen) FROM visits) AS fanout_balances; +-- +-- Invariant 2: skim is monotone non-positive in aggregate. +-- SELECT SUM(declared_weight_kg - actual_weight_kg) AS total_skim_kg FROM shipments; +-- -- should always be >= 0 (we're skimming, not adding) +-- +-- Invariant 3: count of consultations equals total declared meetings from +-- skeptical visits. Aggregate form — survives 24-bit id collisions. +-- (The per-row JOIN version reports false positives when two visits share an +-- id and one is skeptical and the other isn't; see SKILL.md "Invariants vs. +-- id collisions".) +-- SELECT (SELECT COUNT(*) FROM consultations) = +-- (SELECT SUM(CASE WHEN m.name = 'skeptical' +-- THEN mod(get_byte(v.random, 7)::int, 3) ELSE 0 END) +-- FROM visits_core v JOIN moods m ON m.id = v.mood_id) AS consultations_match; +-- +-- Demo: hours where the cover ran best AND the back of house was busy. +-- SELECT * FROM cover_quality ORDER BY hour DESC LIMIT 20; +-- +-- Demo: the leaderboard of letdowns. +-- SELECT * FROM animal_disappointment ORDER BY avg_stars DESC, sightings DESC LIMIT 10; +-- +-- Demo (with banking loaded): people who are both clients AND have large +-- round-number transactions. +-- SELECT ca.person_name, ca.envelope_mm_total, ab.balance +-- FROM client_activity ca +-- JOIN account_balances ab ON ab.holder_name = ca.person_name +-- ORDER BY ca.envelope_mm_total DESC LIMIT 20; +-- ----------------------------------------------------------------------------- diff --git a/misc/demo-data/assets/scaffold.sql b/misc/demo-data/assets/scaffold.sql new file mode 100644 index 0000000000000..dd3d75b0f1ecb --- /dev/null +++ b/misc/demo-data/assets/scaffold.sql @@ -0,0 +1,113 @@ +-- Copyright Materialize, Inc. and contributors. All rights reserved. +-- +-- Use of this software is governed by the Business Source License +-- included in the LICENSE file at the root of this repository. +-- +-- As of the Change Date specified in that file, in accordance with +-- the Business Source License, use of this software will be governed +-- by the Apache License, Version 2.0. + +-- ============================================================================= +-- Moments scaffold: a sliding window of timestamps you can hash for entropy. +-- +-- This file is stable boilerplate. Domains build on top of `moments` and +-- `random`; they should never need to edit anything here. +-- +-- The trick: build a 130-year `generate_series` of years, but filter so only +-- the years near `mz_now()` survive. Cascade through days/hours/minutes/seconds. +-- The `UNION ALL SELECT * FROM empty` at each level blocks the optimizer from +-- inlining and pre-computing the whole timeline. +-- +-- Load with: \i scaffold.sql +-- ============================================================================= + +-- Knobs. Override before \i, e.g.: +-- \set retention '6 hours' +-- \i scaffold.sql +\if :{?retention} \else \set retention '1 day' \endif +\if :{?tick} \else \set tick '1 second' \endif + +\echo Scaffold: retention=:retention, tick=:tick + +-- Idempotency: if `empty` already exists in the current database, assume +-- the scaffold is loaded and skip. Run teardown.sql first to rebuild. +SELECT EXISTS (SELECT 1 FROM mz_tables WHERE name = 'empty') AS already_loaded \gset +\if :already_loaded +\echo Scaffold already loaded; skipping. (Run teardown.sql first to rebuild.) +\else + +CREATE TABLE empty (e TIMESTAMP); + +-- Each year-long interval of interest. +CREATE VIEW years AS +SELECT * +FROM generate_series( + '1970-01-01 00:00:00+00', + '2099-01-01 00:00:00+00', + '1 year') year +WHERE mz_now() BETWEEN year AND year + '1 year' + '1 day'; + +-- Each day-long interval of interest. +CREATE VIEW days AS +SELECT * FROM ( + SELECT generate_series(year, year + '1 year' - '1 day'::interval, '1 day') AS day + FROM years + UNION ALL SELECT * FROM empty +) +WHERE mz_now() BETWEEN day AND day + '1 day' + '1 day'; + +-- Each hour-long interval of interest. +CREATE VIEW hours AS +SELECT * FROM ( + SELECT generate_series(day, day + '1 day' - '1 hour'::interval, '1 hour') AS hour + FROM days + UNION ALL SELECT * FROM empty +) +WHERE mz_now() BETWEEN hour AND hour + '1 hour' + '1 day'; + +-- Each minute-long interval of interest. +CREATE VIEW minutes AS +SELECT * FROM ( + SELECT generate_series(hour, hour + '1 hour' - '1 minute'::interval, '1 minute') AS minute + FROM hours + UNION ALL SELECT * FROM empty +) +WHERE mz_now() BETWEEN minute AND minute + '1 minute' + '1 day'; + +-- Each second-long interval of interest. +CREATE VIEW seconds AS +SELECT * FROM ( + SELECT generate_series(minute, minute + '1 minute' - '1 second'::interval, '1 second') AS second + FROM minutes + UNION ALL SELECT * FROM empty +) +WHERE mz_now() BETWEEN second AND second + '1 second' + '1 day'; + +-- Indexes in order. Each level depends on the prior being indexed so the +-- expansion fires incrementally rather than re-scanning the cascade. +CREATE DEFAULT INDEX ON years; +CREATE DEFAULT INDEX ON days; +CREATE DEFAULT INDEX ON hours; +CREATE DEFAULT INDEX ON minutes; +CREATE DEFAULT INDEX ON seconds; + +-- Public surface #1: a sliding-window stream of timestamps. +-- Cardinality = retention / tick. Defaults give 86,400 rows (24h of seconds). +-- The `mod(...)` clause thins to the requested tick rate. Tick must be a +-- whole number of seconds. +CREATE VIEW moments AS +SELECT second AS moment FROM seconds +WHERE mz_now() >= second + AND mz_now() < second + :'retention'::interval + AND mod(EXTRACT(EPOCH FROM second)::bigint, + EXTRACT(EPOCH FROM :'tick'::interval)::bigint) = 0; + +-- Public surface #2: deterministic pseudorandom bytes per moment. +-- Use `get_byte(random, N)` in domains to pull 0..255 values for fields, +-- foreign keys, distributions. Re-hashing `random || something` lets a row +-- spawn many child rows that stay stable across re-derivation. +CREATE VIEW random AS +SELECT moment, digest(moment::text, 'md5') AS random +FROM moments; + +\endif diff --git a/misc/demo-data/assets/teardown.sql b/misc/demo-data/assets/teardown.sql new file mode 100644 index 0000000000000..4da2f23f2281d --- /dev/null +++ b/misc/demo-data/assets/teardown.sql @@ -0,0 +1,24 @@ +-- Copyright Materialize, Inc. and contributors. All rights reserved. +-- +-- Use of this software is governed by the Business Source License +-- included in the LICENSE file at the root of this repository. +-- +-- As of the Change Date specified in that file, in accordance with +-- the Business Source License, use of this software will be governed +-- by the Apache License, Version 2.0. + +-- Drop everything created by the scaffold and any loaded domains. +-- Idempotent: safe to run repeatedly, safe to run when only some domains +-- were loaded. Order matters: drop `empty CASCADE` first to kill the +-- moments chain and everything derived from it, then drop the static +-- lookup tables that don't transitively depend on `empty`. + +-- Scaffold + everything reachable from a moment. +DROP TABLE IF EXISTS empty CASCADE; + +-- Static lookups (have no dep on `empty`, must be dropped explicitly). +DROP VIEW IF EXISTS people CASCADE; -- common +DROP VIEW IF EXISTS items CASCADE; -- auctions +DROP VIEW IF EXISTS products CASCADE; -- ecommerce +DROP VIEW IF EXISTS accounts CASCADE; -- banking +DROP VIEW IF EXISTS devices CASCADE; -- iot