Skip to content

feat: ship language data, report usage, stop inspect leaking keys - #21

Merged
Halvanhelv merged 3 commits into
mainfrom
feat/languages-and-usage
Sep 10, 2026
Merged

feat: ship language data, report usage, stop inspect leaking keys#21
Halvanhelv merged 3 commits into
mainfrom
feat/languages-and-usage

Conversation

@Halvanhelv

Copy link
Copy Markdown
Owner

Subsystem C, the last of three. A (providers and transport) and B (the pipeline) are merged.

Three changes that share a theme: the gem knew things it never told anyone, and told one thing it should not.

The usage event

Every provider built a Translation::Usage and nothing read it. It is now a
fifth instrumentation event, usage.translation_diff, beside translate,
cache, request and rate_limit:

key meaning
provider the provider's cache key, as in every other payload
characters what this library sent, counted locally
billed_characters what the provider said it charged for, or nil
reported whether the provider reports billing at all
model the model the provider named, when it names one

reported is what makes a nil honest: three of the six providers report
billing, so a subscriber summing billed_characters without checking it
gets a total that is quietly too low.

The language registry

One JSON file per provider under data/languages/, captured live from the
vendors on 2026-09-10, each naming the date and the endpoint it came from.
Translator consults it before every translation and raises
UnsupportedLanguageError before any request is made.

Data ships for DeepL, Google, Azure and ModernMT. Amazon ships none — its
list needs signed credentials — and LibreTranslate ships none either, because
it is self-hosted, so the language set belongs to the instance. A provider
with no data refuses nothing: Languages.supports? answers nil, and nil is
not a refusal.

Codes match downcased on the primary subtag, so en-GB and en match each
other. Deliberately permissive: the registry catches a wrong language, not a
wrong regional variant. One extra rule handles a mismatch the vendors create
themselves — ModernMT publishes pes where applications write fa — and it
holds in one direction only, because a vendor publishing zh has promised
nothing about cmn.

rake languages:refresh re-fetches the lists. A provider whose fetch fails,
or which answers with an empty list, keeps its previous file, and failures
are reported rather than silently blanking good data.

Breaking: validation is on by default, so an application translating a
pair the shipped data does not list will begin raising. Two escapes:
assume_supported: true per call, config.validate_languages = false
globally. Both are named in the error message.

Redacted inspect

Configuration#inspect and Provider#inspect printed API keys verbatim,
walking around every protection the earlier subsystems built. They now print
[FILTERED], and the filtered set is derived from the provider registry
rather than from a list someone has to remember to update. A credential
embedded in a URL — the usual shape of a managed Redis URL — has its
userinfo redacted while the host stays readable.

Along the way

Translator's dispatch seam moved into its own Dispatcher: batching,
throttling, the wire call and three of the five events. Translator keeps
document assembly, provider resolution, source-language settling, validation
and the sentence cache. The Metrics/ClassLength exclude it had acquired is
gone.

Cache keys are untouched. assume_supported: is a reserved keyword, so it
never reaches @options and never reaches a provider's payload — an
upgrading application keeps its cache.

567 runs, 0 failures. Rubocop clean across 94 files.

Not in this branch

No price list. No vendor exposes prices through an API, a list price is wrong
for exactly the people who count money, and a stale number in money is worse
than a stale sentence in a README. The character count is here; the rate is
the reader's own.

@Halvanhelv
Halvanhelv merged commit 90a3260 into main Sep 10, 2026
3 checks passed
@Halvanhelv
Halvanhelv deleted the feat/languages-and-usage branch September 10, 2026 13:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant