stashgen refuses every number, date and boolean EQL type — produce them in EQL v4 using ADR-0002's encodings

[coderdan]

Background

EQL (Encrypt Query Language) is the SQL library we install into a customer's PostgreSQL database. It stores encrypted values and lets the database search them without decrypting. Each EQL type (for example TextEq, IntegerOrd, TextMatch) fixes what a column stores: the ciphertext, plus the search terms that type needs. A search term is a small encrypted value the database can compare, such as an HMAC (a keyed hash, used for equality) or an ORE term (order-revealing encryption, used for < and >).

Stack Encrypt (packages/stack-encrypt) is our Rust encryption engine. Since #1095, a plan field can name an EQL type as its target: the engine then writes the exact value that EQL type stores. The Go SDK uses this through its WASI guest (a WebAssembly build of the engine) and the stashgen code generator.

A target type is producible when the engine can write it. The engine runs the type's own Rust plan, which is the EncryptFrom derive on the type's generated struct. eql-codegen adds that derive only to the types listed in ENCRYPTION_DOMAINS (packages/eql/crates/eql-codegen/src/bindings.rs:316), and today that list is only ("text", "eq"). So of the 51 EQL types, only TextEq is producible. The generated table packages/eql/crates/eql-bindings/src/v3/targets.rs records a reason for each of the other 50, and stashgen refuses them with that reason. #1062 scoped this on purpose to TextEq only. This issue tracks one group of the remaining types.

Problem

No numeric, date, timestamp or boolean EQL type can be written by the engine. That is 42 of the 51 types: every type in the integer, smallint, bigint, date, timestamp, numeric, boolean, real and double families (for example Integer, IntegerEq, IntegerOrd, BigintOrdOre, Date, Boolean).

What happens:

1. A Go developer declares a field such as Age int32 with the EQL type IntegerOrd.
2. stashgen asks the engine which types it can produce, and refuses the field with: "how this family encodes a plaintext for the stack-encrypt producer profile is not specified; only the text family's encoding is".
3. The developer has no way to store an encrypted, searchable number or date that EQL can query from Go or from Rust's data plans.

The reason is not missing code. It is a missing decision: how each family turns its plaintext into bytes before the engine encrypts it and derives its terms. For text that encoding is fixed, so TextEq's ciphertext and HMAC match what EQL and the other producers expect. For numbers, dates and booleans nobody has written it down yet (byte order, width, sign handling, how a date or timestamp is represented, how numeric keeps its precision). If the engine picked an encoding without a spec, its terms would not match terms written by another producer for the same value, and queries would silently return no rows.

Nothing catches this today except the refusal itself, which is the intended behaviour until the encoding exists.

Proposal

1. Implement the encodings decided in ADR-0002 (docs(adr): one value model, three encodings in vitaminc, EQL v4 by producer #1139): the new value kinds and one canonical form per kind, in vitaminc 0.6.0 (aead-value 0.6: add 8-, 16- and 128-bit integers, Date, Timestamp and Decimal as value kinds vitaminc#372, vitaminc-prf: equality terms for the new kinds, floats and decimal, from one canonical form per kind vitaminc#373, New vitaminc-ore crate: one order encoding per kind, with block ORE, CLLW ORE and CLLW OPE as pluggable schemes vitaminc#374), then stack-encrypt (stack-encrypt: delete dynamic::Value and Scalar, derive terms through vitaminc, add block ORE, drop Bool order terms #1140).
2. Produce these types into EQL v4 (eql_v4_* domains, EQL: write the same SQL out as eql_v3 and eql_v4, so Stack Encrypt columns are their own Postgres types #1141, @cipherstash/eql 4.0: ship the v3 and v4 bundles together and move Stack Encrypt's targets to v4 #1142), which are separate Postgres types from the eql_v3_* domains other producers write. Matching another producer's term bytes is not a goal: a column belongs to one producer, and mixing them fails loudly. boolean stays storage-only (a ciphertext with no terms).
3. Add the families to ENCRYPTION_DOMAINS one at a time, starting with the equality types (IntegerEq, BigintEq, DateEq, …), which need only the ciphertext and an HMAC. Each addition regenerates targets.rs with producible: true and lets stashgen accept it.
4. Ordering types follow, but they also depend on stashgen refuses TextMatch, TextSearch, TextOrd and TextOrdOpe — the engine derives their terms, but no EQL type is built from them #1131 (OPE) and stashgen refuses TextOrdOre and TextSearchOre — EQL stores block ORE terms and the engine derives CLLW ORE #1132 (ORE): the same term-format questions apply to numbers.

A cheap first step: IntegerEq and BigintEq alone. An integer's encoding is the least ambiguous, and it unblocks the most common Go field type.

Relationship to other work

• Go and Rust plans cannot target EQL domains, so Go writes rows EQL cannot query — make domains plan field targets #1062: made EQL types plan targets, with TextEq only. This issue continues it for the non-text families.
• stashgen refuses TextMatch, TextSearch, TextOrd and TextOrdOpe — the engine derives their terms, but no EQL type is built from them #1131 (match and OPE text types), stashgen refuses TextOrdOre and TextSearchOre — EQL stores block ORE terms and the engine derives CLLW ORE #1132 (block ORE vs CLLW ORE, which also blocks every numeric and date *OrdOre type) and stashgen refuses the storage-only Text type — it is held back until TextEq is proven end to end #1133 (storage-only Text) cover the text types.
• stack-encrypt has no searchable JSON — add a Json index that seals a document's entries under one data key #1060: the JSON index, which SteVecDocument needs. Not covered here.

[coderdan]

ADR-0002 (#1139) settles the spec this issue asks for in step 1, and changes
step 2:

• The encoding is specified. New value kinds (8-, 16- and 128-bit
  integers, Date, Timestamp, Decimal) and one canonical form per kind
  for equality and order terms, implemented in vitaminc (vitaminc-prf and
  the new vitaminc-ore).
• Step 2 no longer applies. These types become producible in EQL v4
  (eql_v4_* domains), which are separate Postgres types from the eql_v3_*
  domains other producers write. Matching another producer's bytes isn't a
  goal; mixing producers in one column now fails loudly.
• boolean: stays storage-only. A ciphertext with no equality or order
  term, because a two-value domain leaks through any term.

Work: #1140 (stack-encrypt) and #1142 (EQL 4.0).