Add optional TTL for migrated Cassandra timeseries data - #12
Draft
hophead12 wants to merge 3 commits into
Draft
Conversation
Migrated data currently lives forever in Cassandra since no TTL is set on writes. Add an optional -ttl CLI argument (in days) that applies USING TTL to ts_kv_cf and ts_kv_partitions_cf inserts. ts_kv_latest_cf is intentionally left unaffected, matching ThingsBoard's own TTL semantics where only historical points expire, not the latest value. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Reject non-positive ttl values and ttl values exceeding Cassandra's hard-coded 20 year (7300 day) TTL maximum, instead of letting them fail later with an opaque error from CQLSSTableWriter or silently producing no TTL (ttl=0 means "no TTL" in Cassandra). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
hophead12
marked this pull request as draft
August 21, 2026 09:43
…estamp Previously -ttl was applied as a flat duration counted from migration time, so a row that was already 2 years old would get the full TTL and end up expiring 1 year later than it should for a 3 year retention policy. Now each ts_kv_cf row's TTL is computed as ttl - (migration_time - row_ts), using a single "now" captured once at startup for all rows. Rows whose age already exceeds the configured ttl are skipped instead of migrated. ts_kv_partitions_cf keeps a flat full-duration TTL (from migration time), which is intentional: it is always >= any individual row's remaining TTL, so partition bookkeeping can never expire before the data it references. Also fixes a bug caught while testing this: Cassandra's TTL bind parameter is a 32-bit int, not bigint - binding a Long threw a ClassCastException on every row, which went completely unnoticed during earlier testing because the packaged jar has no SLF4J binding and silently drops all log output (separate pre-existing issue, not fixed here). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
hophead12
marked this pull request as ready for review
August 21, 2026 09:56
hophead12
marked this pull request as draft
August 21, 2026 12:54
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Migrated timeseries data currently lives in Cassandra forever — none of the generated SSTables set a TTL, and the target tables have no
default_time_to_live. For teams that only need historical data for a limited retention window, this means migrated data has to be cleaned up manually afterwards, or it just accumulates indefinitely and wastes disk space.This PR adds an optional
-ttl <days>CLI argument so migrated data can expire automatically in Cassandra, the same way it would if it had been written with a TTL policy from the start.Why this matters
-ttl 90and Cassandra will expire the migrated rows automatically via native TTL, no follow-up cleanup step required.What changed
-ttl(integer, days). If omitted, behavior is unchanged — no TTL, data never expires.ts_kv_cf: each row's remaining TTL is computed from its own original timestamp, not from migration time:remaining_ttl = ttl - (migration_time - row_ts), using a single "now" captured once when the migrator starts. A row whose age already exceeds the configured-ttlis skipped rather than migrated. This matters because with a flat TTL counted from migration time, a 2-year-old row would incorrectly get the full retention period and expire a year late (for a 3-year policy).ts_kv_partitions_cf(partition bookkeeping) gets a flat, full-duration TTL counted from migration time. This is intentional and safe: it is always >= any individual row's remaining TTL, so the partition bookkeeping entry can never expire before the data it references.ts_kv_latest_cf(last known value per key) is intentionally not affected by-ttlat all, matching ThingsBoard's own TTL semantics where only historical points expire, not the current value.-ttlmust be a positive number of days and cannot exceed 7300 days (20 years) — Cassandra's hard-coded maximum TTL. Values outside this range are rejected up front with a clear error.Testing
Ran end-to-end migrations against a real ThingsBoard Postgres database (Docker, ~8.4M
ts_kvrows) using the documented dump commands.Run 1 — flat TTL sanity check (
-ttl 30, before the per-row calculation was added):SSTableExport/SSTableMetadataViewerthat every row gotttl = 2592000(30 days).related_entities.dmpare skipped by design).Run 2 — per-row calculation (
-ttl 10), after adding the age-aware logic:int, notbigint. Binding aLongthrew aClassCastExceptionon every row, which went completely unnoticed at first because the packaged jar has no SLF4J binding and silently drops all log output (SLF4J: Defaulting to no-operation (NOP) logger implementation) — the process still exited 0 withts_kv_cfcompletely empty. Reproduced standalone againstCQLSSTableWriterto confirm the root cause, fixed by binding anInteger, and reran.SSTableMetadataViewerthat TTL varies per row (e.g.TTL max: 527311~= 6.1 days for the freshest rows, consistent with10 days - ~3.9 days elapsed),ts_kv_partitions_cfstayed flat at864000(10 days) as designed, and exactly 12 rows were dropped as already-expired — matching an independent count of rows older than the 10-day cutoff directly from the source dump. Combined with the 324 orphaned-entity skips, this fully reconciles the migrated row count (8,431,293 - 336 = 8,430,957).mvn clean compile assembly:singlebuilds cleanly.Note (separate, pre-existing issue found during testing)
The packaged jar has no SLF4J binding (only
slf4j-api/log4j-over-slf4j), so all of the tool's logging (Lines processed,Lines migrated, errors) is silently dropped at runtime. This is unrelated to this change and not fixed here, but it's worth a follow-up: besides hiding progress, it also means real errors - like theClassCastExceptionabove - fail completely silently, with the process still exiting 0.