A limitation we observe with the translate API is that it processes one input at a time.
Our investigation shows that batched inference significantly improves performance on CPU, we see a 2‑fold performance gain with a batch size greater than 32, which we believe is due to lower kernel dispatch, scheduling overhead, better tiling, and improved hardware utilization.
From a developer perspective, use cases are translating content across multiple unrelated elements or several individual user-generated comments on a page require passing a collection of strings (with independent context) and translating them together with the source sentence identity preserved so that they can be replaced back in the DOM.
Concretely, the proposal is to support two overloads on the translate, translateStreaming API.
Promise<sequence<DOMString>> translate(
sequence<DOMString> inputs,
optional TranslatorTranslateOptions options = {}
);
ReadableStream translateStreaming(
sequence<DOMString> input,
optional TranslatorTranslateOptions options = {}
);
The readable stream at each step will return a Sequence of DOMString.
A limitation we observe with the translate API is that it processes one input at a time.
Our investigation shows that batched inference significantly improves performance on CPU, we see a 2‑fold performance gain with a batch size greater than 32, which we believe is due to lower kernel dispatch, scheduling overhead, better tiling, and improved hardware utilization.
From a developer perspective, use cases are translating content across multiple unrelated elements or several individual user-generated comments on a page require passing a collection of strings (with independent context) and translating them together with the source sentence identity preserved so that they can be replaced back in the DOM.
Concretely, the proposal is to support two overloads on the translate, translateStreaming API.
The readable stream at each step will return a Sequence of DOMString.