Skip to content

fix(export): stream batched SQLite table dumps via ReadableStream (#59) - #330

Closed
jeffrey-graham wants to merge 1 commit into
outerbase:mainfrom
jeffrey-graham:patch-1
Closed

jeffrey-graham wants to merge 1 commit into
outerbase:mainfrom
jeffrey-graham:patch-1

Conversation

@jeffrey-graham

Copy link
Copy Markdown

Summary

Fixes #59
/claim #59

Problem

dumpDatabaseRoute in src/export/dump.ts previously executed SELECT * FROM ${table}; for every table and concatenated the entire SQL dump in memory into a single dumpContent string before wrapping it in new Blob([dumpContent]). On large Cloudflare Durable Object SQLite databases, this exceeds Worker memory/response limits and fails.

Solution

  • Added fetchRowsBatches(tableName, dataSource, config) async generator querying SELECT * FROM ${tableName} LIMIT 1000 OFFSET ${offset}; in batches of ROW_BATCH_SIZE = 1000.
  • Streamed table schemas and batched INSERT INTO statements through a ReadableStream<Uint8Array> directly into new Response(stream, { headers }) so memory usage remains bounded regardless of database size.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Database dumps do not work on large databases

1 participant