Engine config split, file-format registry, and error/pipeline fixes - #164
Merged
Conversation
`ErrorResponse::with_message` appended the caller's message to the kind's default instead of replacing it, so an endpoint whose message matched its kind default returned the text twice — e.g. login returned "Invalid credentials. Invalid credentials". Make `with_message` replace the default, since a caller-supplied message is the authoritative human-facing text. Also make error messages uniform on punctuation: drop the trailing period from the ErrorResponse default presets and the one caller message that carried one, so every client-facing message follows the same no-trailing-period convention. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The pipeline name and description limits disagreed between layers: the request DTO validated name 3-100 and description <=500, while the DB CHECK constraints allowed name 1-255 and description <=4096, and the constraint error messages quoted the DB numbers. Align all three on a single contract — name 2-128, description <=500 — in the DTO validation, the migration CHECKs and column comments, and the error messages, so the enforced limit and the reported limit match. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Move infrastructure config out of the per-pipeline definition and into an operator-owned engine config file, leaving the pipeline definition as pure detection/governance intent. Server-wide (engine config TOML at ENGINE_CONFIG_FILEPATH): - enrichers (language/OCR/STT) — uniform across the deployment. - deduplication: calibration (operator-only) plus fallback defaults for merging / tiebreaker / min_confidence a pipeline inherits per-field. - NER/LLM recognizer lineups, expressed as [[recognizers.ner]] / [[recognizers.llm]] to mirror a pipeline's recognizers selectors. Per-pipeline (PipelineDefinition): recognizers (pattern + ner/llm selectors), deduplication intent (optional, overrides the config default), label catalog, default scope, policy slugs. The merge lives on the engine service: EngineService::analyzer_params takes the pipeline definition + request scope and returns a ready AnalyzerParams, merging in its private defaults and rejecting a pipeline that enables NER/LLM this deployment has no lineup for. The runs handler no longer sees the deployment defaults at all. The engine service module is split into config.rs (the TOML file types + load/parse), error.rs (UnknownFormatToken), and mod.rs (the service). Adds the toml dependency, a committed docker/engine.example.toml, and the .env / docker-compose wiring for ENGINE_CONFIG_FILEPATH. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Drop the hand-maintained FileFormat enum (which duplicated and drifted from elide's format set) in favor of the engine's authoritative codec registry. The file list filter now resolves format tokens against it: - ?formats= takes file extensions (pdf, png); each expands to its format's full extension set (so jpg also matches jpeg). - ?modality= takes modality keywords (text, tabular, image, audio), expanding to every extension of that modality's formats. - The two are separate facets combined with AND (intersection); an unknown token rejects with 400. The postgres layer stays registry-free: FileFilter carries a resolved Option<Vec<String>> of extensions (None = no constraint, Some(empty) = match nothing), and the server does the token->extension expansion via EngineService::resolve_extensions / resolve_modalities. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
martsokha
added
feat
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Four related changes to the redaction engine's configuration surface and a couple of correctness fixes, landed together.
1. Error message fix + uniformity
ErrorResponse::with_messageappended the caller message to the kind's default, so an endpoint whose message matched its default returned it twice (login returned "Invalid credentials. Invalid credentials"). It now replaces the default. Error messages are also made uniform on punctuation (no trailing period across presets + caller messages).2. Pipeline name/description bounds reconciled
The name/description limits disagreed between the DTO, the DB CHECK, and the error messages. Aligned all three: name 2–128, description ≤500.
3. Engine config split — server-wide defaults vs pipeline intent
Infrastructure config moves out of the per-pipeline definition into an operator-owned engine TOML file (
ENGINE_CONFIG_FILEPATH):[[recognizers.ner]]/[[recognizers.llm]]).EngineService::analyzer_params(&definition, scope)— it owns the private defaults, does the fallback merge, and rejects a pipeline that enables NER/LLM this deployment lacks. The runs handler never touches deployment defaults.config.rs/error.rs/mod.rs. Adds thetomldep, a committeddocker/engine.example.toml, and.env/ docker-compose wiring.4. FileFormat enum → engine codec registry,
?modality=Dropped the hand-maintained
FileFormatenum (drifted from elide's format set) for the engine's authoritative registry:?formats== extensions (pdf, png),?modality== modality keywords (text/tabular/image/audio); separate facets combined with AND; unknown token → 400.FileFiltercarries resolved extensions;None= no constraint,Some(empty)= match nothing).Migration note
Pipeline
definitionblobs are re-derived from the shrunken schema; existing pipeline rows should be wiped (pre-release, no legacy). Migrations regenerated for the name/description bounds.Verification
cargo check --all-features --workspace✅cargo clippy --all-targets --all-features --workspace -- -D warnings✅cargo +nightly fmt --all -- --check✅RUSTDOCFLAGS="-D warnings" cargo doc --no-deps --all-features --workspace✅cargo test --all-features --workspace✅ (all suites)🤖 Generated with Claude Code