Update from code changes: document StreamingDataset transform_parallelism

[mintlify]

Summary

The Python StreamingDataset gains two related capabilities that the training data-loading page now covers. First, a keyword-only transform_parallelism argument caps how many transform batches run concurrently, defaulting to one worker per CPU (falling back to one when the CPU count is unknown). Second, a new on_transform_error option lets iteration skip rows whose transform raises, with a companion merge_state_dicts helper for elastic resume when different ranks skip different rows. The docs page now walks through both options, calls out the synchronous-distributed-training footgun, and points readers to the filter parameter for predictable bad data.

Changes

• Update the Transformation section to describe transform_parallelism, its default fallback, and its memory/throughput trade-off.
• Add a "Handling bad rows" subsection covering on_transform_error ("raise", "skip", "warn", callable handlers), the rows_skipped counter, row-granular skipping semantics, and the sync-training deadlock warning.
• Add an "Elastic resume when rows are skipped" subsection covering merge_state_dicts for combining per-rank checkpoints across topology changes.

Update history

• Initial: documented the new transform_parallelism option for lancedb/lancedb#3699.
• Appended on_transform_error fault-tolerance guidance and merge_state_dicts elastic-resume coverage for lancedb/lancedb#3763.

[mintlify]

Preview deployment for your docs. Learn more about Mintlify Previews.

Project	Status	Preview	Updated (UTC)
lancedb-bcbb4faf	🟢 Ready	View Preview	Jul 28, 2026, 10:40 PM

[mintlify]

Update from code changes updated this PR.

Added on_transform_error skip handling and merge_state_dicts elastic-resume guidance

Source PRs

• lancedb/lancedb#3763 — feat(python): add on_transform_error fault tolerance to StreamingDataset