Repository navigation
Layout F: reads in fewer round trips - #66
Merged
Merged
Conversation
#65 bounded F's reads at a cost in round trips, which showed on staging: an fsck of a 204k-record F version ran past six minutes (its job was canceled) where today's layout's took a few. - An index read skips layers whose keys miss its own (from the manifest, no reads), then takes 16 at a time: a collection's usual ~10 runs in one round. - A leaf's index pages: 8 in flight, not one ahead. - Pack reads take up to 64 blocks a request (16). - Readers of a whole leaf (bodyLineStream, fsck, sync, copyRecords) take a few MB of records a lookup (leafChunks), not 256 records: a leaf of small records is one lookup.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
#65 bounded F's reads at a cost in storage round trips (layout-f-pr.md "Watch for", items 1–3). That cost showed on staging. An fsck of
f-test-b, a 204k-record F version, ran 381 s of wall time on 19 s of CPU and was then canceled. Today's layout's fsck of the same records took a few minutes.bodyLineStream, fsck, sync,copyRecords) take up to 4 MB of records a lookup (leafChunks) instead of 256 records. A leaf of small records is one index lookup instead of four.Memory stays bounded by these constants. The 400k memory gate (
memory.ts --layout f --records 400000 --small-every 2000 --read-ms 1 --max-mb 60) passes: a whole-version read peaks at 16.7 MB (12.0 before),layout.maintainat 54.3 MB, and every job is under 60 MB.Protocol 258 and server 566 (both layouts) pass.
Deploy: both Workers, no migration. Then rerun fsck on
qa-review-2/f-test-b.