Background
EQL (Encrypt Query Language) is the SQL library we install into a customer's PostgreSQL database. It stores encrypted values and lets the database search them without decrypting. Each EQL type (for example TextEq, IntegerOrd, TextMatch) fixes what a column stores: the ciphertext, plus the search terms that type needs. A search term is a small encrypted value the database can compare, such as an HMAC (a keyed hash, used for equality) or an ORE term (order-revealing encryption, used for < and >).
Stack Encrypt (packages/stack-encrypt) is our Rust encryption engine. Since #1095, a plan field can name an EQL type as its target: the engine then writes the exact value that EQL type stores. The Go SDK uses this through its WASI guest (a WebAssembly build of the engine) and the stashgen code generator.
A target type is producible when the engine can write it. The engine runs the type's own Rust plan, which is the EncryptFrom derive on the type's generated struct. eql-codegen adds that derive only to the types listed in ENCRYPTION_DOMAINS (packages/eql/crates/eql-codegen/src/bindings.rs:316), and today that list is only ("text", "eq"). So of the 51 EQL types, only TextEq is producible. The generated table packages/eql/crates/eql-bindings/src/v3/targets.rs records a reason for each of the other 50, and stashgen refuses them with that reason. #1062 scoped this on purpose to TextEq only. This issue tracks one group of the remaining types.
Problem
Two text types cannot be written because the engine and EQL use different ORE schemes:
| EQL type |
Terms EQL stores |
TextOrdOre |
HMAC + ORE |
TextSearchOre |
HMAC + ORE + bloom filter |
EQL's ore term is block ORE: the SQL compares it with the ore_block functions (for example packages/eql/src/v3/common.sql and the *_ord_ore_functions.sql files). The engine derives CLLW ORE terms, a different scheme. The two are not interchangeable: a CLLW term cannot be compared with block-ORE functions. The generated table refuses both types with: "EQL stores a block ORE term and the engine derives a CLLW ORE term; the two are different algorithms". The engine's test target resolver says the same (packages/stack-encrypt/src/dynamic/target.rs:454).
The same mismatch blocks every *OrdOre type in the numeric and date families (for example IntegerOrdOre, DateOrdOre), once their plaintext encoding exists.
Impact: a developer who needs range queries on text with ORE (for example because their column already uses TextOrdOre) cannot write it from Go or a Rust data plan. Today they can only use the OPE types, once those are producible.
Proposal
Decide which side moves. The options:
- The engine derives block-ORE terms. Add a block-ORE index to Stack Encrypt that produces exactly the bytes EQL's
ore_block functions compare, then add TextOrdOre and TextSearchOre to ENCRYPTION_DOMAINS.
- EQL accepts CLLW ORE. Add CLLW ORE comparison to EQL (new SQL functions, or a new ORE domain), keeping block ORE for existing data.
- Do not produce ORE types from the engine. Mark them permanently unproducible and point users at the OPE types (
TextOrdOpe, TextSearch), which do not have this problem.
Whatever is chosen, prove it end to end in PostgreSQL with a fixture that pins the term bytes, as TextEq was.
Relationship to other work
Background
EQL (Encrypt Query Language) is the SQL library we install into a customer's PostgreSQL database. It stores encrypted values and lets the database search them without decrypting. Each EQL type (for example
TextEq,IntegerOrd,TextMatch) fixes what a column stores: the ciphertext, plus the search terms that type needs. A search term is a small encrypted value the database can compare, such as an HMAC (a keyed hash, used for equality) or an ORE term (order-revealing encryption, used for<and>).Stack Encrypt (
packages/stack-encrypt) is our Rust encryption engine. Since #1095, a plan field can name an EQL type as itstarget: the engine then writes the exact value that EQL type stores. The Go SDK uses this through its WASI guest (a WebAssembly build of the engine) and thestashgencode generator.A target type is producible when the engine can write it. The engine runs the type's own Rust plan, which is the
EncryptFromderive on the type's generated struct.eql-codegenadds that derive only to the types listed inENCRYPTION_DOMAINS(packages/eql/crates/eql-codegen/src/bindings.rs:316), and today that list is only("text", "eq"). So of the 51 EQL types, onlyTextEqis producible. The generated tablepackages/eql/crates/eql-bindings/src/v3/targets.rsrecords a reason for each of the other 50, andstashgenrefuses them with that reason. #1062 scoped this on purpose toTextEqonly. This issue tracks one group of the remaining types.Problem
Two text types cannot be written because the engine and EQL use different ORE schemes:
TextOrdOreTextSearchOreEQL's
oreterm is block ORE: the SQL compares it with theore_blockfunctions (for examplepackages/eql/src/v3/common.sqland the*_ord_ore_functions.sqlfiles). The engine derives CLLW ORE terms, a different scheme. The two are not interchangeable: a CLLW term cannot be compared with block-ORE functions. The generated table refuses both types with: "EQL stores a block ORE term and the engine derives a CLLW ORE term; the two are different algorithms". The engine's test target resolver says the same (packages/stack-encrypt/src/dynamic/target.rs:454).The same mismatch blocks every
*OrdOretype in the numeric and date families (for exampleIntegerOrdOre,DateOrdOre), once their plaintext encoding exists.Impact: a developer who needs range queries on text with ORE (for example because their column already uses
TextOrdOre) cannot write it from Go or a Rust data plan. Today they can only use the OPE types, once those are producible.Proposal
Decide which side moves. The options:
ore_blockfunctions compare, then addTextOrdOreandTextSearchOretoENCRYPTION_DOMAINS.TextOrdOpe,TextSearch), which do not have this problem.Whatever is chosen, prove it end to end in PostgreSQL with a fixture that pins the term bytes, as
TextEqwas.Relationship to other work
TextEqonly.*_ord_oredomains #629: an ORE SQL coverage suite for the*_ord_oredomains. It tests EQL's SQL; it does not make ORE producible.*OrdOretypes also need this decided.