refactor!: replace the pipeline with a design of our own

CHANGELOG.md

@@ -106,15 +106,53 @@ described below. Everything here is relative to `deepl_diff` 2.2.0.
   configuration still setting `deepl_host` raises `NoMethodError` on
   `TranslationDiff.configure`. Rename it.
 - A provider returning the wrong number of translations now raises
-  `TranslationDiff::ResponseError`, not `TranslationDiff::Request::Error`.
-  `Request::Error` still exists, and still means "`from:` is missing and the
-  provider cannot detect"; a `rescue TranslationDiff::Request::Error` written
-  to catch a short response no longer catches one. Both are
-  `TranslationDiff::Error`, so a rescue of the base class is unaffected.
+  `TranslationDiff::ResponseError`, not the error that used to live on
+  `Request`; a `rescue` written to catch a short response that way no longer
+  catches one. Both are `TranslationDiff::Error`, so a rescue of the base
+  class is unaffected.
 - A provider returning a well-formed response that carries no translation for
   one input -- Azure answers 200 for a batch where a single string failed --
   also raises `TranslationDiff::ResponseError`, naming the position. It
-  previously reached `Spacing.restore` and died there as `NoMethodError`.
+  previously reached the spacing step and died there as `NoMethodError`.
+- **The translation pipeline is new code.** `TranslationDiff::Linearizer`,
+  `Spacing`, `Chunker`, `Tokenizer`, `Cache` and `Request` are gone as public
+  constants. What replaces them: `Document` and `Leaves` (walking the
+  caller's structure), `Passage`, `Fragment` and `Segment` (markup and prose,
+  cut into sentences), `Markup` (entity references and a `<` that opens no
+  tag), `Batch` (packing sentences into provider requests), `SentenceCache`
+  (the cache key, read and write) and `Translator` (the coordinator
+  `TranslationDiff.translate` and `Context#translate` now build). See
+  [How it works](docs/how-it-works.md). If you referenced any of the six by
+  name, that reference is now a `NameError`.
+- `TranslationDiff::Request::Error` is now `TranslationDiff::Translator::Error`
+  and `TranslationDiff::Cache::Error` is now
+  `TranslationDiff::SentenceCache::Error`. There is no alias for either: this
+  gem has never been published under the name `translation_diff` with those
+  constants in it. `TranslationDiff::Chunker::Error` is gone with no
+  replacement -- a single sentence too large to send now raises
+  `TranslationDiff::Batch::Error`. All three remain
+  `TranslationDiff::Error`, so a rescue of the base class is unaffected.
+- **`TranslationDiff.translate` and `Context#translate` raise `ArgumentError`
+  when `to:` is missing or `nil`.** The keyword still defaults to `nil` in the
+  signature, and the message names it. Previously a `nil` target compared
+  equal to a `nil` source, the call short-circuited as "same language" and
+  your values came back untranslated, silently. If you have a caller reading
+  `to:` out of a configuration that can be blank, it has been a no-op and will
+  now raise.
+- **The `cache` instrumentation event fires once per `translate` call, not
+  once per chunk.** The cache is now consulted for every sentence in one
+  `read_multi` before anything is batched. `hits` and `misses` still sum to
+  the same totals over a call, so a counter that adds them up is unaffected;
+  a counter of *events*, or a histogram of per-chunk hit ratios, will see the
+  cardinality drop. `request` and `rate_limit` still fire once per batch sent.
+- **Two more cache keys move, beyond the entity and `<` fixes below.** A
+  sentence padded with Unicode whitespace -- a non-breaking space, say -- now
+  keys as the bare sentence: the pipeline uses one Unicode-aware definition
+  of padding everywhere, where the key used to be built with ASCII `strip`,
+  which leaves a `U+00A0` in place. And the whole document key format is
+  otherwise unmoved: it is pinned by test against recorded values, and every
+  other input in the corpus this rewrite was judged against produces the same
+  key it did before.
 
 ### Removed
 
@@ -297,6 +335,29 @@ described below. Everything here is relative to `deepl_diff` 2.2.0.
 - DeepL's batch limit was declared as 300 sentences per request; DeepL
   documents 50. The request-size limit (1,700 escaped characters) was
   already correct and is unchanged.
+- **An entity reference no longer reaches the provider raw.** `Salt &amp;
+  pepper.` was sent to the provider as the six characters `&amp;`, so the
+  provider translated the entity's spelling as if it were words -- and was
+  billed for it. It is now sent as `Salt & pepper.`, the text the document
+  actually says, and re-encoded on the way out. Named entities, `&#38;` and
+  `&#x26;` alike are decoded; anything neither decoder knows is left as it
+  arrived.
+- **A bare `<` no longer swallows the rest of the sentence.** `if a < b then
+  stop. Fine.` was parsed by `ox` as prose followed by an unclosed tag, so
+  only `if a` was ever sent for translation and everything after the `<` came
+  back untranslated. A `<` that no element name, closing name, declaration or
+  instruction follows is now escaped before parsing and restored after, so the
+  whole sentence is translated. `5 < 6 and 7 > 6. True.` was sent as three
+  fragments and is now sent as two sentences.
+- **These two fixes move the cache key for the documents they affect.** A
+  document containing an entity reference, or a `<` that opens no tag, will
+  miss the cache once and be re-translated. That is the point: what was cached
+  for it was translated from the wrong text.
+- **A known remaining limit: `&lt;` still reaches a provider undecoded**, and
+  `a <b then stop. Fine.` still loses everything after the `<`. Both fall out
+  of escaping a bare `<` as `&lt;` to work around `ox` rather than replacing
+  it with a lexer of this gem's own, which is out of scope here. `<b` cannot
+  be told apart from a tag without one. Every other entity is decoded.
 
 ## [2.2.0] - 2026-09-07
 

README.md

@@ -2,8 +2,6 @@
 
 A translation cache that helps translate only changes between revisions of long texts.
 
-**TranslationDiff** based on [GoogleTranslateDiff](https://github.com/gzigzigzeo/google_translate_diff)
-
 ## Why TranslationDiff?
 
 Assume your project contains a significant amount of products descriptions which:

docs/caching.md

@@ -20,6 +20,35 @@ once.
 Both read and write the same cache, keyed per provider, so switching one
 never serves you the other's translations.
 
+`TranslationDiff::SentenceCache` is the class that builds the key and does
+both the read and the write.
+
+## The options digest is lossy, on purpose
+
+The per-call options are canonicalised to a string before they are digested,
+and that canonical form flattens more than it distinguishes. Nesting is not
+recorded, so `["x", ["y", "z"]]` and `["x", "y", "z"]` canonicalise
+identically; neither is emptiness typed, so `tags: []` and `tags: {}` do
+too. Two calls whose options differ only in one of those ways share a cache
+entry.
+
+This is a known property, not an oversight. The pipeline this replaced
+collides on exactly the same inputs -- that was checked, not assumed -- so
+reproducing it was the choice that left every warm cache warm. Fixing it
+would give those calls new keys and re-translate everything already cached
+under the old ones, for a distinction no provider option this gem ships
+actually makes. If you pass an option where that distinction matters, give
+the configuration its own `cache_namespace`.
+
+A value the canonical form cannot render at all -- anything that is not a
+String, Symbol, Numeric, `true`, `false`, `nil`, or an Array or Hash of
+those -- raises `TranslationDiff::SentenceCache::Error` rather than being
+guessed at. A key that is silently wrong costs you the whole cache and tells
+you nothing.
+
+No options at all contributes no field to the key, which is the four-field
+key every already-warm cache is keyed on.
+
 ## The cache store contract
 
 `config.cache` accepts either a registered name (`:redis`, `:memory`) or an

docs/errors.md

@@ -23,13 +23,13 @@ TranslationDiff::Error
 │                                                # that returned no translation for an input
 ├── TranslationDiff::InvalidProviderError       # a class registered without inheriting
 │                                                # TranslationDiff::Provider
-├── TranslationDiff::Request::Error             # from: missing and the provider cannot
+├── TranslationDiff::Translator::Error          # from: missing and the provider cannot
 │                                                # detect, cache_key missing on an
 │                                                # assigned provider object
-├── TranslationDiff::Cache::Error               # provider options have no stable
+├── TranslationDiff::SentenceCache::Error       # provider options have no stable
 │                                                # serialisation for the cache key
-├── TranslationDiff::Chunker::Error             # a single value is larger than the
-│                                                # provider's declared max_request_size
+├── TranslationDiff::Batch::Error               # one sentence, once escaped, is larger
+│                                                # than the provider's declared limit
 ├── TranslationDiff::Segmenters::Pragmatic::Error
 │                                                # Pragmatic computed offsets that
 │                                                # violate its own postcondition --
@@ -43,6 +43,24 @@ and `#status` (the HTTP status code), so a caller can log or branch on which
 service and which response caused the failure without parsing the message.
 
 `TranslationDiff::Registry` -- which backs the provider, cache store and
-segmenter registries -- also raises `TranslationDiff::Error` directly (not a
+segmenter registries -- raises `TranslationDiff::Error` directly (not a
 dedicated subclass) for an unknown name, listing what is actually
-registered.
+registered. `TranslationDiff::Batch::Error` is its own class rather than a
+direct `TranslationDiff::Error`, so a caller can catch "this sentence is too
+long for this provider" without also catching an unrelated registry miss; it
+is raised when one sentence is larger once escaped than the provider's
+declared `max_request_size` or `max_text_size` and so could never be sent
+even in a batch of its own. The message names a short prefix of the
+offending text and both numbers.
+
+`ArgumentError`, not a `TranslationDiff::Error`, is what
+`TranslationDiff.translate` and `Context#translate` raise when `to:` is
+missing or `nil`. It is a caller's mistake before it is a translation, and
+the message names the keyword.
+
+**Renamed in 3.1.0.** `TranslationDiff::Request::Error` is now
+`TranslationDiff::Translator::Error` and `TranslationDiff::Cache::Error` is
+now `TranslationDiff::SentenceCache::Error`; both classes they hung off are
+gone. `TranslationDiff::Chunker::Error` is gone with no replacement -- the
+condition it named now raises `TranslationDiff::Batch::Error`. A
+`rescue TranslationDiff::Error` catches all three exactly as before.

docs/how-it-works.md

@@ -1,11 +1,59 @@
 # How it works
 
-- Text nodes are extracted from HTML.
-- Every text node is split into sentences by `config.segmenter` (see
-  [The segmenter contract](contracts.md#the-segmenter-contract)).
-- Cache is checked for the presence of each sentence (using language couple and a hash of string).
-- Missing sentences are translated via the provider and cached.
-- Original HTML is recombined from translations and cache data.
+A call to `TranslationDiff.translate` walks a value, cuts the prose in it
+into sentences, translates only the sentences no cache already holds, and
+puts the value back together in the shape it arrived in.
+
+`TranslationDiff::Translator` is the one class that coordinates all of it.
+Everything below is a collaborator it drives.
+
+## The steps
+
+1. **The caller's structure is walked, not flattened.**
+   `TranslationDiff::Document` visits every leaf `String` of a String, Array
+   or deep Hash and can rebuild the same shape from new leaves.
+   `TranslationDiff::Leaves` holds the two promises the structure itself does
+   not: a nested `nil` comes back as `""`, and the `values` count in an event
+   payload is every leaf the caller wrote, translatable or not.
+
+2. **Each leaf becomes a passage of markup and prose.**
+   `TranslationDiff::Passage` parses the string with `ox` and records where
+   every construct begins, so each run of the source is either markup --
+   tags, comments, CDATA, doctypes, processing instructions, `<script>` and
+   `<style>` bodies, and anything inside `class="notranslate"` -- or prose.
+   Each run becomes a `TranslationDiff::Fragment`, and a fragment is always a
+   slice of the source, never a rebuilt string.
+
+3. **Prose is cut into sentences.**
+   A prose fragment is split by `config.segmenter` (see [The segmenter
+   contract](contracts.md#the-segmenter-contract)) into
+   `TranslationDiff::Segment`s. A segment keeps the whitespace it was found
+   in: its `#core` is the text a provider sees, with entity references
+   decoded to the characters they mean, and its `#render` is markup again.
+
+4. **The cache is consulted once, for every sentence at once.**
+   `TranslationDiff::SentenceCache` builds one key per sentence and reads
+   them all in a single `read_multi`. Sentences it answers for are already
+   done; the rest are misses. See [Caching](caching.md).
+
+5. **The misses are packed into requests.**
+   `TranslationDiff::Batch.pack` groups the missing sentences into batches
+   that fit inside the provider's declared `max_batch_size` and
+   `max_request_size` -- its `TranslationDiff::Capabilities` (see
+   [Providers](providers.md)). Each batch is sent, and the reply is
+   applied back onto the very segments that produced it -- no step ever
+   correlates a translation to a sentence by position after the fact.
+
+6. **What came back is written home, and the value is rebuilt.**
+   Only sentences that actually got a translation are cached. Then each
+   passage renders itself -- markup fragments byte-exact, translated
+   sentences re-encoded as HTML text -- and `Document` puts the renders back
+   into the caller's shape.
+
+`TranslationDiff::Markup` is the small module underneath steps 2, 3 and 6: it
+decodes entity references on the way to a provider, encodes `&` and `<` again
+on the way out, and escapes a `<` that opens no tag so `ox` cannot read the
+rest of the sentence as markup.
 
 *NOTE:* if `:from` is not specified or equal to nil, then the provider's `#detect` will be called once with a sample of text up to 100 characters long to determine the language, and `#translate` will be called separately with the entire text.
         Try to specify `:from` explicitly to save the extra call -- it also improves segmentation, since the segmenter only sees a language when `:from` is given (see [The segmenter contract](contracts.md#the-segmenter-contract)).
@@ -22,7 +70,13 @@ TranslationDiff.translate(
 )
 ```
 
-See `TranslationDiff::Linearizer` for details.
+A leaf that is not a `String` -- a number, a symbol, `true` -- is handed back
+untouched. A nested `nil` comes back as `""`. See
+`TranslationDiff::Document` and `TranslationDiff::Leaves` for details.
+
+`to:` is not optional. It defaults to `nil` in the signature and a `nil`
+target raises `ArgumentError` naming the keyword, rather than silently
+handing your values back untranslated.
 
 ## HTML
 

docs/instrumentation.md

@@ -11,9 +11,16 @@ A translation emits up to four events, each named `<name>.translation_diff`:
 | Event | Fired | Payload |
 | --- | --- | --- |
 | `translate` | Once per `translate` call that reaches the provider, wrapping the whole thing. A call whose source and target languages are the same, or whose values hold no translatable text at all, returns early and emits no events. | `from`, `to`, `provider`, `values` (number of texts) |
-| `cache` | Once per chunk, after checking the cache. | `provider`, `hits`, `misses` |
-| `request` | Once per chunk actually sent to the provider (skipped entirely on a full cache hit). | `provider`, `batch` (values sent), `characters` |
-| `rate_limit` | Once per chunk sent to the provider, only when a rate limiter is configured. | `provider`, `characters` |
+| `cache` | Once per `translate` call that reaches the provider, after checking the cache for every sentence at once. | `provider`, `hits`, `misses` |
+| `request` | Once per batch actually sent to the provider (skipped entirely on a full cache hit). | `provider`, `batch` (values sent), `characters` |
+| `rate_limit` | Once per batch sent to the provider, only when a rate limiter is configured. | `provider`, `characters` |
+
+**`cache` fires once per call as of 3.1.0, not once per chunk.** The cache is
+now consulted for every sentence in one `read_multi` before anything is
+batched, so there is one event where there used to be one per chunk. `hits`
+and `misses` still sum to the same totals over a call, so a counter that adds
+them up is unaffected; a counter of *events*, or a histogram of per-chunk hit
+ratios, will see the cardinality drop.
 
 **Instrumentation payloads never contain the text being translated, its
 translation, or a credential.** This is a guarantee, not an implementation

docs/providers.md

@@ -146,7 +146,7 @@ subtag is its own. A provider of your own gets the same rule from
 `TranslationDiff::Provider#language`; declare `def self.language_case =
 :upcase` if your vendor wants upper case.
 
-"Request size" is what `Chunker` measures: the URL-escaped form of each
+"Request size" is what `TranslationDiff::Batch` measures: the URL-escaped form of each
 string (`CGI.escape(text).size`), which is never smaller than its UTF-8 byte
 count. "HTML support" names the provider option that turns HTML handling on
 -- every vendor spells it differently, which is exactly what

lib/translation_diff.rb

@@ -13,6 +13,14 @@
 require "translation_diff/translation/request"
 require "translation_diff/translation/response"
 require "translation_diff/registry"
+require "translation_diff/document"
+require "translation_diff/leaves"
+require "translation_diff/markup"
+require "translation_diff/segment"
+require "translation_diff/batch"
+require "translation_diff/fragment"
+require "translation_diff/passage"
+require "translation_diff/sentence_cache"
 require "translation_diff/configuration"
 require "translation_diff/configuration/provider_option_owners"
 
@@ -31,17 +39,12 @@
 require "translation_diff/segmenters"
 require "translation_diff/segmenters/simple"
 require "translation_diff/segmenters/pragmatic"
-require "translation_diff/tokenizer"
-require "translation_diff/linearizer"
-require "translation_diff/chunker"
-require "translation_diff/spacing"
-require "translation_diff/cache"
 require "translation_diff/stores"
 require "translation_diff/memory_cache_store"
 require "translation_diff/redis_cache_store"
 require "translation_diff/redis_rate_limiter"
 require "translation_diff/instrumentation"
-require "translation_diff/request"
+require "translation_diff/translator"
 require "translation_diff/context"
 
 module TranslationDiff
@@ -58,7 +61,7 @@ def context(&) = Context.new(config.copy.tap(&))
 
     # `provider:` and `config:` are reserved; every other keyword is forwarded to the provider.
     def translate(values, from: nil, to: nil, provider: nil, **)
-      Request.new(values, from: from, to: to, provider: provider, config: config, **).call
+      Translator.new(values, from: from, to: to, provider: provider, config: config, **).call
     end
   end
 end