From 25c66411734cf89796b4f8a6950807595bdf513d Mon Sep 17 00:00:00 2001 From: Ivan Despot <66276597+g-despot@users.noreply.github.com> Date: Tue, 4 Aug 2026 17:51:32 +0200 Subject: [PATCH 1/2] docs: fold HNSW snapshots into concepts/storage for v1.39 Review feedback on #475 was that the page reads as too alarmist and documents too much of the pre-v1.39 approach. Since v1.39 leaves nothing to configure, the standalone page no longer earns its own URL: its content moves into the storage concepts page and the per-variable reference stays in the env-vars table, which already carried all five variables in full. - Drop the "snapshots are now required to start" caution in favour of plain prose: checksummed blocks, the write-then-delete ordering, and the commit logs on disk being the delta since the last snapshot. - Correct the failure radius. An unreadable snapshot fails the whole shard, not just one vector index. Lazy-loaded shards leave the node up and error per request; eagerly loaded shards fail node startup, and eager is the default for single-tenant collections. - Note the disk-usage reduction on upgrade, attributed to the snapshot replacing the logs it covers rather than to a new snapshot format -- the V3 format predates v1.39. Scope it to loaded shards after the compactor's first cycles, and keep the headroom caveat, since peak usage still rises while a snapshot is written. - Delete the deprecated-env-var section, keeping the two facts that lived nowhere else: the variables are still recognised but ignored, and the equivalent camelCase fields in a --config-file are accepted silently, with no startup warning. - Collapse the pre-v1.39 configuration detail behind a
, with the anchor on the heading above it. Docusaurus's Details component has no hashchange handling, so an anchor inside would strand the five inbound env-var links on a collapsed summary. - Redirect the old path to the concepts anchor; the v1.31 release blog and a live kapa answer both cite it. Also fixes the long-standing broken anchor on the vector-cache env var: APITable row ids are the variable name verbatim, so the link needs upper case. --- docs/deploy/configuration/env-vars/index.md | 10 +-- docs/deploy/configuration/persistence.md | 1 + docs/weaviate/concepts/storage.md | 41 ++++++--- docs/weaviate/configuration/hnsw-snapshots.md | 89 ------------------- docs/weaviate/configuration/index.mdx | 7 -- netlify.toml | 5 ++ sidebars.js | 1 - 7 files changed, 40 insertions(+), 114 deletions(-) delete mode 100644 docs/weaviate/configuration/hnsw-snapshots.md diff --git a/docs/deploy/configuration/env-vars/index.md b/docs/deploy/configuration/env-vars/index.md index 74f83efef..703178285 100644 --- a/docs/deploy/configuration/env-vars/index.md +++ b/docs/deploy/configuration/env-vars/index.md @@ -83,11 +83,11 @@ import APITable from '@site/src/components/APITable'; | `OPERATIONAL_MODE` | Sets the [mode of operation](../status.md#operational-modes) for the instance. Options: `READ_WRITE` (default), `READ_ONLY`, `WRITE_ONLY`, `SCALE_OUT`. Limits available operations based on the mode selected. | `string` | `READ_WRITE` | | `ORIGIN` | Set the http(s) origin for Weaviate | `string - HTTP origin` | `https://my-weaviate-deployment.com` | | `PERSISTENCE_DATA_PATH` | Path to the Weaviate data store.
[Note about file systems and performance](/weaviate/concepts/resources.md#file-system). | `string - file path` | `/var/lib/weaviate`
Defaults to `./data`| -| `PERSISTENCE_HNSW_DISABLE_SNAPSHOTS` | **Deprecated in `v1.39`.** Ignored, and logs a startup warning. [In `v1.31` through `v1.38`](/weaviate/configuration/hnsw-snapshots.md#pre-v1-39-configuration): if set, HNSW snapshotting will be disabled. Default: `false` (enabled) as of `v1.36`; `true` (disabled) in `v1.31` through `v1.35`
Added in `v1.31` | `boolean` | `false` | -| `PERSISTENCE_HNSW_SNAPSHOT_INTERVAL_SECONDS` | **Deprecated in `v1.39`.** Ignored, and logs a startup warning. [In `v1.31` through `v1.38`](/weaviate/configuration/hnsw-snapshots.md#pre-v1-39-configuration): the minimum time in seconds that must pass before the next snapshot is created. Default: `21600` seconds (6 hours)
Added in `v1.31` | `string - number` | `3600` | -| `PERSISTENCE_HNSW_SNAPSHOT_MIN_DELTA_COMMITLOGS_NUMBER` | **Deprecated in `v1.39`.** Ignored, and logs a startup warning. [In `v1.31` through `v1.38`](/weaviate/configuration/hnsw-snapshots.md#pre-v1-39-configuration): the minimum number of new commit log files created since the last snapshot. Default: `1`
Added in `v1.31` | `string - number` | `100` | -| `PERSISTENCE_HNSW_SNAPSHOT_MIN_DELTA_COMMITLOGS_SIZE_PERCENTAGE` | **Deprecated in `v1.39`.** Ignored, and logs a startup warning. [In `v1.31` through `v1.38`](/weaviate/configuration/hnsw-snapshots.md#pre-v1-39-configuration): the minimum total size of new commit logs (as a percentage of the previous snapshot's size) required to trigger a new snapshot. Default: `5` (meaning 5% of the previous snapshot's size in new commit logs)
Added in `v1.31` | `string - number` | `15` | -| `PERSISTENCE_HNSW_SNAPSHOT_ON_STARTUP` | **Deprecated in `v1.39`.** Ignored, and logs a startup warning. [In `v1.31` through `v1.38`](/weaviate/configuration/hnsw-snapshots.md#pre-v1-39-configuration): if set, Weaviate will try to create a new snapshot during startup if there are changes in the commit log since the last snapshot. If there are no changes, then the existing snapshot will be loaded. Default: `true`
Added in `v1.31` | `boolean` | `false` | +| `PERSISTENCE_HNSW_DISABLE_SNAPSHOTS` | **Deprecated in `v1.39`.** Ignored, and logs a startup warning. In `v1.31` through `v1.38`: if set, HNSW snapshotting is disabled. Default: `false` (enabled) as of `v1.36`; `true` (disabled) in `v1.31` through `v1.35`. See [snapshot configuration before `v1.39`](/weaviate/concepts/storage.md#pre-v1-39-configuration).
Added in `v1.31` | `boolean` | `false` | +| `PERSISTENCE_HNSW_SNAPSHOT_INTERVAL_SECONDS` | **Deprecated in `v1.39`.** Ignored, and logs a startup warning. In `v1.31` through `v1.38`: the minimum time in seconds that must pass before the next snapshot is created. Default: `21600` seconds (6 hours). See [snapshot configuration before `v1.39`](/weaviate/concepts/storage.md#pre-v1-39-configuration).
Added in `v1.31` | `string - number` | `3600` | +| `PERSISTENCE_HNSW_SNAPSHOT_MIN_DELTA_COMMITLOGS_NUMBER` | **Deprecated in `v1.39`.** Ignored, and logs a startup warning. In `v1.31` through `v1.38`: the minimum number of new commit log files created since the last snapshot. Default: `1`. See [snapshot configuration before `v1.39`](/weaviate/concepts/storage.md#pre-v1-39-configuration).
Added in `v1.31` | `string - number` | `100` | +| `PERSISTENCE_HNSW_SNAPSHOT_MIN_DELTA_COMMITLOGS_SIZE_PERCENTAGE` | **Deprecated in `v1.39`.** Ignored, and logs a startup warning. In `v1.31` through `v1.38`: the minimum total size of new commit logs (as a percentage of the previous snapshot's size) required to trigger a new snapshot. Default: `5` (meaning 5% of the previous snapshot's size in new commit logs). See [snapshot configuration before `v1.39`](/weaviate/concepts/storage.md#pre-v1-39-configuration).
Added in `v1.31` | `string - number` | `15` | +| `PERSISTENCE_HNSW_SNAPSHOT_ON_STARTUP` | **Deprecated in `v1.39`.** Ignored, and logs a startup warning. In `v1.31` through `v1.38`: if set, Weaviate tries to create a new snapshot during startup if there are changes in the commit log since the last snapshot. If there are no changes, the existing snapshot is loaded. Default: `true`. See [snapshot configuration before `v1.39`](/weaviate/concepts/storage.md#pre-v1-39-configuration).
Added in `v1.31` | `boolean` | `false` | | `PERSISTENCE_HNSW_MAX_LOG_SIZE` | Maximum size of the HNSW [write-ahead-log](/weaviate/concepts/storage.md#hnsw-vector-index-storage). Increase this to improve log compaction efficiency, or decrease to reduce memory requirements. Default: 500MiB | `string` | `4GiB` (IEC units), `4GB` (SI units), `4000000000` (bytes) | | `PERSISTENCE_LSM_ACCESS_STRATEGY` | Function used to access disk data in virtual memory. Default: `mmap` | `string` | `mmap` or `pread` | | `PERSISTENCE_LSM_MAX_SEGMENT_SIZE` | Maximum size of a segment in the [LSM store](/weaviate/concepts/storage.md#object-and-inverted-index-store). Set this to limit disk usage spikes during compaction to ~2x the segment size. Default: no limit | `string` | `4GiB` (IEC units), `4GB` (SI units), `4000000000` (bytes) | diff --git a/docs/deploy/configuration/persistence.md b/docs/deploy/configuration/persistence.md index e503b049e..074904dcb 100644 --- a/docs/deploy/configuration/persistence.md +++ b/docs/deploy/configuration/persistence.md @@ -107,6 +107,7 @@ You can configure automatic deletion of objects after a specified time period us ## Related pages - [Configuration: Backups](/deploy/configuration/backups.md) +- [Concepts: Storage - HNSW snapshots](/weaviate/concepts/storage.md#hnsw-snapshots) ## Questions and feedback diff --git a/docs/weaviate/concepts/storage.md b/docs/weaviate/concepts/storage.md index 69112ec0f..b358a4705 100644 --- a/docs/weaviate/concepts/storage.md +++ b/docs/weaviate/concepts/storage.md @@ -1,7 +1,7 @@ --- title: Storage sidebar_position: 18 -description: "Persistent, fault-tolerant storage architecture for objects, vectors, and inverted index management." +description: "Persistent, fault-tolerant storage architecture for objects, vectors, and inverted index management, including HNSW snapshots and commit log compaction." image: og/docs/concepts.jpg # tags: ['architecture', 'storage'] --- @@ -79,7 +79,7 @@ This change improves reliability during rolling restarts and upgrades. Eager loa #### Vector cache prefill behavior -The [`HNSW_STARTUP_WAIT_FOR_VECTOR_CACHE`](/deploy/configuration/env-vars#hnsw_startup_wait_for_vector_cache) environment variable controls whether vector cache prefill is synchronous (blocking) or asynchronous (background) at startup. Its default changed to `true` in v1.36.6. +The [`HNSW_STARTUP_WAIT_FOR_VECTOR_CACHE`](/deploy/configuration/env-vars#HNSW_STARTUP_WAIT_FOR_VECTOR_CACHE) environment variable controls whether vector cache prefill is synchronous (blocking) or asynchronous (background) at startup. Its default changed to `true` in v1.36.6. For collections where lazy shard loading is active, vector cache prefill is always **asynchronous** — the `HNSW_STARTUP_WAIT_FOR_VECTOR_CACHE` value is overridden to `false` regardless of the configured value. For eagerly-loaded collections, the configured value applies (default: `true`, meaning synchronous prefill). @@ -101,7 +101,7 @@ For the HNSW vector index, the Write-Ahead-Log (WAL) is a critical component for The entire HNSW index state can be reconstructed by replaying these WAL entries. -For very large indexes of tens or hundreds of millions of objects, this can be time-consuming. To avoid replaying the entire WAL on every restart, Weaviate writes **[HNSW snapshots](../configuration/hnsw-snapshots.md)**. +For very large indexes of tens or hundreds of millions of objects, this can be time-consuming. To avoid replaying the entire commit log on every restart, Weaviate writes **[HNSW snapshots](#hnsw-snapshots)**. ### HNSW snapshots @@ -109,27 +109,44 @@ import HnswSnapshots from '/_includes/feature-notes/hnsw-snapshots.mdx'; -For very large HNSW vector indexes, HNSW snapshots significantly reduce the startup time. - A snapshot represents a point-in-time state of the HNSW index. When Weaviate starts, it loads the most recent snapshot and replays only the commit log entries written after it. This significantly reduces startup time, because the number of entries that have to be replayed no longer grows with the age of the index. The commit log still persists every change immediately, guaranteeing that any acknowledged write is durable. Even with a fresh snapshot, the server typically still has to load at least one subsequent commit log file. -Starting in `v1.39`, snapshots are part of how the vector index is stored rather than an optional speedup, and Weaviate manages them automatically. A background compactor owns the on-disk lifecycle of the index: it compacts newly flushed commit logs, merges them together, and writes a new snapshot when doing so is worthwhile. Snapshots and commit logs live in the same directory, and a snapshot replaces the commit logs it covers rather than duplicating them, which keeps the disk footprint proportional to the size of the index. +Starting in `v1.39`, snapshots are part of how the vector index is stored rather than an optional speedup. A background process called the commit log compactor owns the on-disk lifecycle of the index: it compacts newly flushed commit logs, merges them together, and writes a new snapshot when doing so is worthwhile. Snapshots and commit logs live in the same directory, and a snapshot replaces the commit logs it covers rather than duplicating them, which keeps the disk footprint proportional to the size of the index. + +Upgrading to `v1.39` therefore reduces the disk space the vector index uses, in some cases substantially. Earlier versions keep the full commit log alongside the snapshot, and a snapshot is a more compact representation of the same index than the commit logs it replaces, because compaction keeps only the final state of each vector's connections instead of every change made to them. + +Two caveats apply. The saving appears once the compactor has run its first cycles on each loaded shard rather than at the moment you upgrade, and inactive tenants do not shrink until they are next activated. Keep planning for the same headroom as before, because peak disk usage still rises while a snapshot is being written. Because the merge reads its inputs as sorted streams from disk, only the most recently flushed commit log is processed in memory. That step has a fixed cost regardless of how large the graph is, so snapshot creation no longer requires enough memory to hold the previous snapshot plus the commit log delta. Weaviate protects this on-disk state in several ways. New files are written to a temporary path and atomically renamed into place, so an interrupted write can never be mistaken for a complete file, and orphaned temporary files are cleaned up on the next startup. -Commit logs, including compacted ones, are self-healing. If one cannot be read in full, whether because a crash tore its tail or because the file was damaged on disk, it is truncated back to its last valid entry. The entries written before the damage are retained and the file becomes valid again for later compaction, so only the damaged tail is lost. +Commit logs, including compacted ones, are self-healing. If one cannot be read in full, whether because a crash cut it short or because the file was damaged on disk, it is truncated back to its last valid entry. The entries written before the damage are retained and the file becomes valid again for later compaction, so only the damaged tail is lost. -Snapshots are handled differently. A snapshot is stored in a checksummed block format, and it is never truncated or repaired. If the current snapshot cannot be read, the affected vector index fails to start rather than loading partial data. Because the commit logs a snapshot covers are deleted once it has been written, nothing remains on the node to replay in its place, so recovering that shard means restoring the data, for example from a [backup](/deploy/configuration/backups.md). +Snapshots are handled differently. A snapshot is stored in a checksummed block format and every block is verified when it is read, but unlike a commit log, a snapshot is not truncated or repaired. -See **[the HNSW snapshots configuration](../configuration/hnsw-snapshots.md)** for version-specific details. +If the current snapshot cannot be read, Weaviate does not load a partial index. The shard that owns the snapshot fails to load, and so does every other vector index on that shard. Because the commit logs a snapshot covers are deleted once the snapshot has been written, the index cannot be rebuilt from the node's remaining files. If that shard uses [dynamic lazy shard loading](#dynamic-lazy-shard-loading), the node stays up and requests to the shard return an error. If the shard is loaded eagerly, which is the default for single-tenant collections and for multi-tenant collections below the auto-detection thresholds, node startup fails instead. Restore the affected data from a [backup](/deploy/configuration/backups.md), which includes the snapshot. -:::note Behavior in `v1.31` through `v1.38` -In these versions, snapshots are an optional feature layered on top of the commit log, and are configured with the `PERSISTENCE_HNSW_SNAPSHOT_*` environment variables. They are enabled by default starting in `v1.36`, and disabled by default in `v1.31` through `v1.35`. Weaviate creates a snapshot at startup if the commit log changed since the last snapshot, and periodically once a configured time interval has passed and enough new commit log data has accumulated. If a snapshot cannot be loaded, it is removed and Weaviate falls back to loading the full commit log from the beginning. -::: +The environment variables that configured snapshots before `v1.39` are deprecated. That version and later still recognize `PERSISTENCE_HNSW_DISABLE_SNAPSHOTS` and the `PERSISTENCE_HNSW_SNAPSHOT_*` variables, so an existing deployment starts without a configuration error, but their values are ignored. For each of these variables that is set, Weaviate logs a warning at startup stating that the variable has no effect and will be removed in a future version; variables that are not set produce no warning. If your deployment passes a configuration file with `--config-file` (`weaviate.conf.json` by default), the equivalent camelCase fields in that file, such as `hnswDisableSnapshots`, are accepted and ignored as well, and unlike the environment variables they produce no startup warning. Remove the variables from your deployment configuration, and the equivalent fields from any configuration file, to clear the warnings. + +#### Snapshot configuration before `v1.39` {#pre-v1-39-configuration} + +In `v1.31` through `v1.38`, snapshots are an optional feature layered on top of the commit log rather than part of it, and the `PERSISTENCE_HNSW_SNAPSHOT_*` environment variables control when Weaviate creates them. Snapshots are enabled by default starting in `v1.36`, and disabled by default in `v1.31` through `v1.35`. Weaviate creates one at startup if the commit log changed since the last snapshot, and periodically thereafter. If a snapshot cannot be read in these versions, it is discarded and Weaviate replays the full commit log instead. For the variables themselves, including their defaults and deprecation status, see [`PERSISTENCE_HNSW_DISABLE_SNAPSHOTS`](/deploy/configuration/env-vars/index.md#PERSISTENCE_HNSW_DISABLE_SNAPSHOTS) and the rows that follow it. + +
+ Periodic snapshot conditions and memory requirements + +Periodic snapshot creation is governed by three variables, and **all** of the following conditions must be met before Weaviate creates a snapshot: + +- `PERSISTENCE_HNSW_SNAPSHOT_INTERVAL_SECONDS` — the minimum time since the previous snapshot has elapsed. +- `PERSISTENCE_HNSW_SNAPSHOT_MIN_DELTA_COMMITLOGS_NUMBER` — enough new commit log files have been created since the last snapshot. +- `PERSISTENCE_HNSW_SNAPSHOT_MIN_DELTA_COMMITLOGS_SIZE_PERCENTAGE` — the new commit logs are large enough, measured as a percentage of the previous snapshot's size. + +Before creating a new snapshot, Weaviate loads the previous snapshot and the commit log difference into memory, so the node needs enough memory to accommodate both. This requirement does not apply in `v1.39` and later, where the merge streams its inputs from disk. + +
## Conclusions diff --git a/docs/weaviate/configuration/hnsw-snapshots.md b/docs/weaviate/configuration/hnsw-snapshots.md deleted file mode 100644 index ec0f392f7..000000000 --- a/docs/weaviate/configuration/hnsw-snapshots.md +++ /dev/null @@ -1,89 +0,0 @@ ---- -title: HNSW Snapshots -sidebar_position: 47 -sidebar_label: HNSW Snapshots -description: Learn how Weaviate uses HNSW snapshots for faster startup times, and how they were configured in versions before v1.39. ---- - -import HnswSnapshots from '/_includes/feature-notes/hnsw-snapshots.mdx'; - - - -HNSW (Hierarchical Navigable Small World) snapshots significantly reduce startup times for instances with large vector indexes. - -:::info Concepts: HNSW snapshots -See this [concepts page](../concepts/storage.md#hnsw-snapshots) for a detailed description. -::: - -## Snapshots in `v1.39` and later - -Starting in `v1.39`, HNSW snapshots are always enabled and are not configurable. - -The commit log compactor owns the on-disk lifecycle of the vector index, and snapshots are one of the file formats it produces. Weaviate creates and maintains them automatically, so there is nothing to enable, disable, schedule, or tune. - -:::caution Snapshots are now required to start -Because a snapshot replaces the commit logs it covers, it is the only copy of the index state up to that point. If the current snapshot cannot be read, the affected vector index fails to start rather than loading partial data, and it cannot be rebuilt from the node's remaining files. Recovering that shard means restoring the data, for example from a [backup](/deploy/configuration/backups.md). In `v1.31` through `v1.38`, an unreadable snapshot is discarded and Weaviate replays the full commit log instead. -::: - -### Environment variables that no longer have an effect - -The following environment variables are still recognized, so existing deployments continue to start without a configuration error. However, they are ignored: - -- `PERSISTENCE_HNSW_DISABLE_SNAPSHOTS` -- `PERSISTENCE_HNSW_SNAPSHOT_INTERVAL_SECONDS` -- `PERSISTENCE_HNSW_SNAPSHOT_ON_STARTUP` -- `PERSISTENCE_HNSW_SNAPSHOT_MIN_DELTA_COMMITLOGS_NUMBER` -- `PERSISTENCE_HNSW_SNAPSHOT_MIN_DELTA_COMMITLOGS_SIZE_PERCENTAGE` - -If any of these variables is set, Weaviate logs a warning at startup that names the variable and states that it has no effect and will be removed in a future version. Variables that are not set produce no warning. To clear the warnings, remove the variables from your deployment configuration. - -The same settings in a Weaviate configuration file (`weaviate.conf.json` by default) are accepted and ignored too, but they produce no startup warning. - -## Configuring snapshot creation in versions before `v1.39` {#pre-v1-39-configuration} - -:::note Applies to `v1.31` through `v1.38` -In `v1.39` and later, the environment variables below are ignored. -::: - -In these versions, HNSW snapshotting is an optional feature layered on top of the commit log, and the following environment variables control it. - -HNSW snapshotting is **enabled by default** starting in `v1.36`. To disable it, set `PERSISTENCE_HNSW_DISABLE_SNAPSHOTS` to `true`. In versions prior to `v1.36`, HNSW snapshotting is disabled by default. Set `PERSISTENCE_HNSW_DISABLE_SNAPSHOTS` to `false` to enable it. - -:::note -Before creating a new snapshot, the previous snapshot and the commit log difference need to be loaded into memory. Make sure you have enough memory to accommodate this process. This requirement does not apply in `v1.39` and later, where snapshots are merged from disk as a stream. -::: - -### Snapshot on startup - -Enable or disable snapshot creation on startup: - -- `PERSISTENCE_HNSW_SNAPSHOT_ON_STARTUP`: If `true`, Weaviate will try to create a new snapshot during startup if there are changes in the commit log since the last snapshot. If there are no changes, then the existing snapshot will be loaded. - - **Default:** `true` - -### Periodic snapshots - -Set the following to configure periodic snapshot creation. Note **all** of the following conditions must be met to trigger a snapshot: - -1. **A time interval has passed:** - - - `PERSISTENCE_HNSW_SNAPSHOT_INTERVAL_SECONDS`: The minimum time in seconds since the previous snapshot. - - **Default:** `21600` seconds (6 hours) - -2. **Sufficient new commit logs (by number):** - - - `PERSISTENCE_HNSW_SNAPSHOT_MIN_DELTA_COMMITLOGS_NUMBER`: The minimum number of new commit log files created since the last snapshot. - - **Default:** `1` - -3. **Sufficient new commit logs (by size percentage):** - - `PERSISTENCE_HNSW_SNAPSHOT_MIN_DELTA_COMMITLOGS_SIZE_PERCENTAGE`: The minimum total size of new commit logs (as a percentage of the previous snapshot's size) required to trigger a new snapshot. - - **Default:** `5` (meaning 5% of the previous snapshot's size in new commit logs). For example, if the previous snapshot was 1000MB, at least 50MB of new commit log data is required. - -## Further resources - -- [Concepts: Storage - Persistence and Crash Recovery](../concepts/storage.md#persistence-and-crash-recovery) - -## Questions and feedback - -import DocsFeedback from '/\_includes/docs-feedback.mdx'; - - diff --git a/docs/weaviate/configuration/index.mdx b/docs/weaviate/configuration/index.mdx index 3177eca48..b28eff66e 100644 --- a/docs/weaviate/configuration/index.mdx +++ b/docs/weaviate/configuration/index.mdx @@ -25,13 +25,6 @@ export const configOpsData = [ link: "/weaviate/configuration/compression", icon: "fas fa-compress-alt", }, - { - title: "HNSW Snapshots", - description: - "Speed up startup and recovery for instances with large vector indexes.", - link: "/weaviate/configuration/hnsw-snapshots", - icon: "fas fa-camera", - }, { title: "Modules", description: diff --git a/netlify.toml b/netlify.toml index 4aecc23e2..233964326 100644 --- a/netlify.toml +++ b/netlify.toml @@ -506,6 +506,11 @@ from = "/weaviate/configuration/persistence" to = "/deploy/configuration/persistence" status = 301 +[[redirects]] +from = "/weaviate/configuration/hnsw-snapshots" +to = "/weaviate/concepts/storage#hnsw-snapshots" +status = 301 + [[redirects]] from = "/weaviate/configuration/monitoring" to = "/deploy/configuration/monitoring" diff --git a/sidebars.js b/sidebars.js index b328f000a..146d3abb0 100644 --- a/sidebars.js +++ b/sidebars.js @@ -533,7 +533,6 @@ const sidebars = { "weaviate/configuration/compression/multi-vectors", ], }, - "weaviate/configuration/hnsw-snapshots", "weaviate/configuration/modules", { type: "doc", From 89b1085cd9bc7641925f681789d1ce7ff3fe6f2e Mon Sep 17 00:00:00 2001 From: Ivan Despot <66276597+g-despot@users.noreply.github.com> Date: Tue, 11 Aug 2026 17:39:23 +0200 Subject: [PATCH 2/2] docs: correct seven inaccurate claims about snapshots and the commit log An audit against the v1.39.0 tag found several statements in this section that the code does not support. Four predate this branch and are live on v1-39/main; three were introduced while reworking it. - The HNSW commit log does not fsync per write. It is buffered, flushed as batches are processed, and synced when a log file is rotated. The same applies to the LSM store WAL, which is written but not synced before a request is acknowledged; scope the ack guarantee accordingly, and say the two logs differ in when they sync rather than when they are written. - Snapshot creation still loads the snapshot it supersedes into memory. Only the commit log delta and the new snapshot are streamed. Drop the claims that no snapshot is held and that the in-memory step is fixed cost -- it grows with the node ID space. - Peak disk usage during snapshot creation is above the steady state, since the previous snapshot, the merge inputs and the new snapshot coexist. Do not tell operators to plan for unchanged headroom. - Only snapshots and compacted commit logs are written atomically. Raw commit logs are appended in place, which is why they self-heal from a torn tail. - Self-healing recovers a torn tail, not arbitrary on-disk damage. - A snapshot is created at startup once enough new commit log data has accumulated, not on any change, and only rotated files count. Align the env-var reference with the same wording. Also restores two things the earlier rework lost: that the periodic conditions are conjunctive with their defaults spelled out, and that these options set through a configuration file are ignored silently, with no startup warning -- the one path where an operator gets no signal at all. --- docs/deploy/configuration/env-vars/index.md | 2 +- docs/weaviate/concepts/storage.md | 42 +++++++++++---------- 2 files changed, 24 insertions(+), 20 deletions(-) diff --git a/docs/deploy/configuration/env-vars/index.md b/docs/deploy/configuration/env-vars/index.md index 703178285..0efd4366c 100644 --- a/docs/deploy/configuration/env-vars/index.md +++ b/docs/deploy/configuration/env-vars/index.md @@ -87,7 +87,7 @@ import APITable from '@site/src/components/APITable'; | `PERSISTENCE_HNSW_SNAPSHOT_INTERVAL_SECONDS` | **Deprecated in `v1.39`.** Ignored, and logs a startup warning. In `v1.31` through `v1.38`: the minimum time in seconds that must pass before the next snapshot is created. Default: `21600` seconds (6 hours). See [snapshot configuration before `v1.39`](/weaviate/concepts/storage.md#pre-v1-39-configuration).
Added in `v1.31` | `string - number` | `3600` | | `PERSISTENCE_HNSW_SNAPSHOT_MIN_DELTA_COMMITLOGS_NUMBER` | **Deprecated in `v1.39`.** Ignored, and logs a startup warning. In `v1.31` through `v1.38`: the minimum number of new commit log files created since the last snapshot. Default: `1`. See [snapshot configuration before `v1.39`](/weaviate/concepts/storage.md#pre-v1-39-configuration).
Added in `v1.31` | `string - number` | `100` | | `PERSISTENCE_HNSW_SNAPSHOT_MIN_DELTA_COMMITLOGS_SIZE_PERCENTAGE` | **Deprecated in `v1.39`.** Ignored, and logs a startup warning. In `v1.31` through `v1.38`: the minimum total size of new commit logs (as a percentage of the previous snapshot's size) required to trigger a new snapshot. Default: `5` (meaning 5% of the previous snapshot's size in new commit logs). See [snapshot configuration before `v1.39`](/weaviate/concepts/storage.md#pre-v1-39-configuration).
Added in `v1.31` | `string - number` | `15` | -| `PERSISTENCE_HNSW_SNAPSHOT_ON_STARTUP` | **Deprecated in `v1.39`.** Ignored, and logs a startup warning. In `v1.31` through `v1.38`: if set, Weaviate tries to create a new snapshot during startup if there are changes in the commit log since the last snapshot. If there are no changes, the existing snapshot is loaded. Default: `true`. See [snapshot configuration before `v1.39`](/weaviate/concepts/storage.md#pre-v1-39-configuration).
Added in `v1.31` | `boolean` | `false` | +| `PERSISTENCE_HNSW_SNAPSHOT_ON_STARTUP` | **Deprecated in `v1.39`.** Ignored, and logs a startup warning. In `v1.31` through `v1.38`: if set, Weaviate tries to create a new snapshot during startup when enough new commit log data has accumulated since the last snapshot. Otherwise, it loads the existing snapshot. Default: `true`. See [snapshot configuration before `v1.39`](/weaviate/concepts/storage.md#pre-v1-39-configuration).
Added in `v1.31` | `boolean` | `false` | | `PERSISTENCE_HNSW_MAX_LOG_SIZE` | Maximum size of the HNSW [write-ahead-log](/weaviate/concepts/storage.md#hnsw-vector-index-storage). Increase this to improve log compaction efficiency, or decrease to reduce memory requirements. Default: 500MiB | `string` | `4GiB` (IEC units), `4GB` (SI units), `4000000000` (bytes) | | `PERSISTENCE_LSM_ACCESS_STRATEGY` | Function used to access disk data in virtual memory. Default: `mmap` | `string` | `mmap` or `pread` | | `PERSISTENCE_LSM_MAX_SEGMENT_SIZE` | Maximum size of a segment in the [LSM store](/weaviate/concepts/storage.md#object-and-inverted-index-store). Set this to limit disk usage spikes during compaction to ~2x the segment size. Default: no limit | `string` | `4GiB` (IEC units), `4GB` (SI units), `4000000000` (bytes) | diff --git a/docs/weaviate/concepts/storage.md b/docs/weaviate/concepts/storage.md index b358a4705..55846463a 100644 --- a/docs/weaviate/concepts/storage.md +++ b/docs/weaviate/concepts/storage.md @@ -13,7 +13,7 @@ The components mentioned on this page aid Weaviate in creating some of its uniqu * Each write operation is immediately persisted and also tolerant to application and system crashes. * On a vector search query, Weaviate returns the entire object (in other databases sometimes called a "document"), not just a reference, such as an ID. * When combining structured search with vector search, filters are applied prior to performing the vector search. This means that you will always receive the specified number of elements as opposed to post-filtering when the final result count is unpredictable. -* Objects and their vectors can be updated or deleted at will; even while reading from the database. +* Objects and their vectors can be updated or deleted at will, even while reading from the database. ## Logical Storage Units: Indexes, Shards, Stores @@ -35,7 +35,7 @@ Weaviate periodically merges smaller, older segments to make larger segments. Si Considerations -Object storage and inverted index storage implement the LSM algorithm; they use segmentation. The vector index uses a different storage algorithm. The vector index does not use segmentation. +Object storage and inverted index storage implement the LSM algorithm, they use segmentation. The vector index uses a different storage algorithm. The vector index does not use segmentation. Weaviate versions before `v1.5.0` use a B+Tree storage mechanism. The LSM method is faster, it works in constant time, and it improves write performance. @@ -93,7 +93,7 @@ Prior to v1.36.6, lazy shard loading was enabled by default for all collections. Both the LSM stores used for object and inverted storage, as well as the HNSW vector index store make use of memory at some point of the ingestion journey. To prevent data loss on a crash, each operation is additionally written into a **[Write-Ahead-Log (WAL)](https://martinfowler.com/articles/patterns-of-distributed-systems/wal.html)** (also known as a *commit log*). WALs are append-only files that are very efficient to write to and that are rarely a bottleneck for ingestion. -By the time Weaviate has responded with a successful status to your ingestion request, a WAL entry will have been created. If a WAL entry could not be created - for example because the disks are full - Weaviate will respond with an error to the insert or update request. +By the time Weaviate has responded with a successful status to your ingestion request, an LSM store WAL entry will have been created. If a WAL entry could not be created - for example because the disks are full - Weaviate will respond with an error to the insert or update request. The HNSW vector index keeps its own commit log, described [below](#hnsw-snapshots). It is written on the same request path, and the two differ in when they are synced to disk. The LSM stores will try to flush a segment on an orderly shutdown. Only if the operation is successful, will the WAL be marked as "complete". This means that if an unexpected crash happens and Weaviate encounters an "incomplete" WAL, it will recover from it. As part of the recovery process, Weaviate will flush a new segment based on the WAL and mark it as complete. As a result, future restarts will no longer have to recover from this WAL. @@ -111,46 +111,50 @@ import HnswSnapshots from '/_includes/feature-notes/hnsw-snapshots.mdx'; A snapshot represents a point-in-time state of the HNSW index. When Weaviate starts, it loads the most recent snapshot and replays only the commit log entries written after it. This significantly reduces startup time, because the number of entries that have to be replayed no longer grows with the age of the index. -The commit log still persists every change immediately, guaranteeing that any acknowledged write is durable. Even with a fresh snapshot, the server typically still has to load at least one subsequent commit log file. +The commit log records every change to the index as it happens. Entries are written to the log as batches are processed, and a log file is synced to disk when it is rotated. Even with a fresh snapshot, Weaviate typically still has to load at least one subsequent commit log file. -Starting in `v1.39`, snapshots are part of how the vector index is stored rather than an optional speedup. A background process called the commit log compactor owns the on-disk lifecycle of the index: it compacts newly flushed commit logs, merges them together, and writes a new snapshot when doing so is worthwhile. Snapshots and commit logs live in the same directory, and a snapshot replaces the commit logs it covers rather than duplicating them, which keeps the disk footprint proportional to the size of the index. +Starting in `v1.39`, snapshots are part of how the vector index is stored rather than an optional speedup. A background process called the commit log compactor owns the on-disk lifecycle of the index: it compacts newly flushed commit logs, merges them together, and writes a new snapshot when doing so is worthwhile. Snapshots and commit logs live in the same directory, and a snapshot replaces the commit logs it covers rather than duplicating them, so the commit logs left on disk hold only the delta since the last snapshot. This keeps the disk footprint proportional to the size of the index. Snapshots are also written as a stream. Weaviate still loads the snapshot it supersedes into memory, but the commit log delta and the new snapshot itself are streamed rather than also held there, as they were before `v1.39`. -Upgrading to `v1.39` therefore reduces the disk space the vector index uses, in some cases substantially. Earlier versions keep the full commit log alongside the snapshot, and a snapshot is a more compact representation of the same index than the commit logs it replaces, because compaction keeps only the final state of each vector's connections instead of every change made to them. +Upgrading to `v1.39` reduces the disk space the vector index uses, in some cases substantially. Earlier versions keep the full commit log alongside the snapshot, and a snapshot is a more compact representation of the same index than the commit logs it replaces, because compaction keeps only the final state of each vector's connections instead of every change made to them. -Two caveats apply. The saving appears once the compactor has run its first cycles on each loaded shard rather than at the moment you upgrade, and inactive tenants do not shrink until they are next activated. Keep planning for the same headroom as before, because peak disk usage still rises while a snapshot is being written. +A few caveats apply. The saving appears once the compactor has run its first cycles on each loaded shard rather than at the moment you upgrade, and inactive tenants do not shrink until they are next activated. Plan headroom for the peak rather than the steady state: while a snapshot is being written, the directory transiently holds the previous snapshot, the files being merged, and the new snapshot as it is assembled, so disk usage during snapshot creation is meaningfully above the size the index settles at. -Because the merge reads its inputs as sorted streams from disk, only the most recently flushed commit log is processed in memory. That step has a fixed cost regardless of how large the graph is, so snapshot creation no longer requires enough memory to hold the previous snapshot plus the commit log delta. +Weaviate protects this on-disk state in several ways. Snapshots and compacted commit logs are written to a temporary path and atomically renamed into place, so an interrupted write can never be mistaken for a complete file, and orphaned temporary files are cleaned up on the next startup. When a new snapshot is written, the snapshot it supersedes and the commit logs it covers are removed only after the new one is durably on disk. -Weaviate protects this on-disk state in several ways. New files are written to a temporary path and atomically renamed into place, so an interrupted write can never be mistaken for a complete file, and orphaned temporary files are cleaned up on the next startup. - -Commit logs, including compacted ones, are self-healing. If one cannot be read in full, whether because a crash cut it short or because the file was damaged on disk, it is truncated back to its last valid entry. The entries written before the damage are retained and the file becomes valid again for later compaction, so only the damaged tail is lost. +Commit logs are self-healing. If a crash leaves the last entry of a log incomplete, the file is truncated back to its last valid entry. The entries written before the tear are retained and the file becomes valid again for later compaction, so only the incomplete tail is lost. Snapshots are handled differently. A snapshot is stored in a checksummed block format and every block is verified when it is read, but unlike a commit log, a snapshot is not truncated or repaired. -If the current snapshot cannot be read, Weaviate does not load a partial index. The shard that owns the snapshot fails to load, and so does every other vector index on that shard. Because the commit logs a snapshot covers are deleted once the snapshot has been written, the index cannot be rebuilt from the node's remaining files. If that shard uses [dynamic lazy shard loading](#dynamic-lazy-shard-loading), the node stays up and requests to the shard return an error. If the shard is loaded eagerly, which is the default for single-tenant collections and for multi-tenant collections below the auto-detection thresholds, node startup fails instead. Restore the affected data from a [backup](/deploy/configuration/backups.md), which includes the snapshot. +In the rare case that the current snapshot cannot be read, restore the affected data from a [backup](/deploy/configuration/backups.md), which includes the snapshot. Weaviate does not load a partial index, and because the commit logs the snapshot covers have already been removed, nothing remains on the node to replay in its place. + +That failure is scoped to the shard that owns the snapshot: the shard fails to load, and so does every other vector index on it. If that shard uses [dynamic lazy shard loading](#dynamic-lazy-shard-loading), the node stays up and requests to the shard return an error. If the shard is loaded eagerly, which is the default for single-tenant collections and for multi-tenant collections below the auto-detection thresholds, node startup fails instead. -The environment variables that configured snapshots before `v1.39` are deprecated. That version and later still recognize `PERSISTENCE_HNSW_DISABLE_SNAPSHOTS` and the `PERSISTENCE_HNSW_SNAPSHOT_*` variables, so an existing deployment starts without a configuration error, but their values are ignored. For each of these variables that is set, Weaviate logs a warning at startup stating that the variable has no effect and will be removed in a future version; variables that are not set produce no warning. If your deployment passes a configuration file with `--config-file` (`weaviate.conf.json` by default), the equivalent camelCase fields in that file, such as `hnswDisableSnapshots`, are accepted and ignored as well, and unlike the environment variables they produce no startup warning. Remove the variables from your deployment configuration, and the equivalent fields from any configuration file, to clear the warnings. +Weaviate creates and maintains snapshots automatically, so there is nothing to enable, disable, schedule, or tune. [`PERSISTENCE_HNSW_MAX_LOG_SIZE`](/deploy/configuration/env-vars/index.md#PERSISTENCE_HNSW_MAX_LOG_SIZE) still influences the size at which commit log files are rotated, and therefore how often there is new material to compact, but it does not configure snapshots. + +The environment variables that configured snapshots before `v1.39` are deprecated. That version and later still recognize `PERSISTENCE_HNSW_DISABLE_SNAPSHOTS` and the `PERSISTENCE_HNSW_SNAPSHOT_*` variables, so an existing deployment starts without a configuration error, but their values are ignored. For each of these variables that is set, Weaviate logs a warning at startup stating that the variable has no effect and will be removed in a future version. If these options are set through a configuration file rather than as environment variables, they are ignored in the same way, but no startup warning is logged. Remove the variables from your deployment configuration to clear the warnings. #### Snapshot configuration before `v1.39` {#pre-v1-39-configuration} -In `v1.31` through `v1.38`, snapshots are an optional feature layered on top of the commit log rather than part of it, and the `PERSISTENCE_HNSW_SNAPSHOT_*` environment variables control when Weaviate creates them. Snapshots are enabled by default starting in `v1.36`, and disabled by default in `v1.31` through `v1.35`. Weaviate creates one at startup if the commit log changed since the last snapshot, and periodically thereafter. If a snapshot cannot be read in these versions, it is discarded and Weaviate replays the full commit log instead. For the variables themselves, including their defaults and deprecation status, see [`PERSISTENCE_HNSW_DISABLE_SNAPSHOTS`](/deploy/configuration/env-vars/index.md#PERSISTENCE_HNSW_DISABLE_SNAPSHOTS) and the rows that follow it. +In `v1.31` through `v1.38`, snapshots are an optional feature layered on top of the commit log rather than part of it, and the `PERSISTENCE_HNSW_SNAPSHOT_*` environment variables control when Weaviate creates them. Snapshots are enabled by default starting in `v1.36`, and disabled by default in `v1.31` through `v1.35`. Weaviate can create one at startup and periodically thereafter, once enough new commit log data has accumulated since the last snapshot. Only commit log files that have been rotated count toward that threshold, so changes still in the active file are not considered until the next rotation. If a snapshot cannot be read in these versions, it is discarded and Weaviate replays the full commit log instead. For the variables themselves, including their defaults and deprecation status, see [`PERSISTENCE_HNSW_DISABLE_SNAPSHOTS`](/deploy/configuration/env-vars/index.md#PERSISTENCE_HNSW_DISABLE_SNAPSHOTS) and the rows that follow it.
Periodic snapshot conditions and memory requirements Periodic snapshot creation is governed by three variables, and **all** of the following conditions must be met before Weaviate creates a snapshot: -- `PERSISTENCE_HNSW_SNAPSHOT_INTERVAL_SECONDS` — the minimum time since the previous snapshot has elapsed. -- `PERSISTENCE_HNSW_SNAPSHOT_MIN_DELTA_COMMITLOGS_NUMBER` — enough new commit log files have been created since the last snapshot. -- `PERSISTENCE_HNSW_SNAPSHOT_MIN_DELTA_COMMITLOGS_SIZE_PERCENTAGE` — the new commit logs are large enough, measured as a percentage of the previous snapshot's size. +- `PERSISTENCE_HNSW_SNAPSHOT_INTERVAL_SECONDS` — the minimum time since the previous snapshot has elapsed (default `21600` seconds, or six hours). +- `PERSISTENCE_HNSW_SNAPSHOT_MIN_DELTA_COMMITLOGS_NUMBER` — enough new commit log files have been created since the last snapshot (default `1`). +- `PERSISTENCE_HNSW_SNAPSHOT_MIN_DELTA_COMMITLOGS_SIZE_PERCENTAGE` — the new commit logs are large enough, measured as a percentage of the previous snapshot's size (default `5`). This condition does not apply to the first snapshot, when there is no previous snapshot to measure against. + +Meeting these conditions makes a snapshot eligible rather than guaranteed. The background process that condenses and combines commit log files is also the one that writes the snapshot, so a snapshot can be created on a later pass than the one where the conditions are first met. -Before creating a new snapshot, Weaviate loads the previous snapshot and the commit log difference into memory, so the node needs enough memory to accommodate both. This requirement does not apply in `v1.39` and later, where the merge streams its inputs from disk. +In these versions, before creating a new snapshot, Weaviate loads the previous snapshot and the commit log difference into memory, so the node needs enough memory to accommodate both.
## Conclusions -This page introduced you to the storage mechanisms of Weaviate. It outlined how all writes are persisted immediately and outlined the patterns used within Weaviate to make datasets scale well. For structured data, Weaviate makes use of segmentation to keep the write times constant. For the HNSW vector index, Weaviate avoids segmentation to keep query times efficient. +This page introduced you to the storage mechanisms of Weaviate. It outlined how all writes are persisted to a log before they are acknowledged and outlined the patterns used within Weaviate to make datasets scale well. For structured data, Weaviate makes use of segmentation to keep the write times constant. For the HNSW vector index, Weaviate avoids segmentation to keep query times efficient. ## Questions and feedback