Skip to content

Add optional TTL for migrated Cassandra timeseries data - #12

Draft
hophead12 wants to merge 3 commits into
thingsboard:masterfrom
hophead12:feature/ts-ttl-option
Draft

Add optional TTL for migrated Cassandra timeseries data#12
hophead12 wants to merge 3 commits into
thingsboard:masterfrom
hophead12:feature/ts-ttl-option

Conversation

@hophead12

@hophead12 hophead12 commented Aug 18, 2026

Copy link
Copy Markdown

Summary

Migrated timeseries data currently lives in Cassandra forever — none of the generated SSTables set a TTL, and the target tables have no default_time_to_live. For teams that only need historical data for a limited retention window, this means migrated data has to be cleaned up manually afterwards, or it just accumulates indefinitely and wastes disk space.

This PR adds an optional -ttl <days> CLI argument so migrated data can expire automatically in Cassandra, the same way it would if it had been written with a TTL policy from the start.

Why this matters

  • Without this feature: every row written by this tool is permanent. If a customer only needs, say, 90 days of historical data available after migration, there is no way to enforce that at write time — it either has to be deleted manually with a separate maintenance job, or it silently stays forever and grows the cluster's storage footprint.
  • With this feature: operators can pass -ttl 90 and Cassandra will expire the migrated rows automatically via native TTL, no follow-up cleanup step required.

What changed

  • New optional CLI option -ttl (integer, days). If omitted, behavior is unchanged — no TTL, data never expires.
  • Per-row TTL for ts_kv_cf: each row's remaining TTL is computed from its own original timestamp, not from migration time: remaining_ttl = ttl - (migration_time - row_ts), using a single "now" captured once when the migrator starts. A row whose age already exceeds the configured -ttl is skipped rather than migrated. This matters because with a flat TTL counted from migration time, a 2-year-old row would incorrectly get the full retention period and expire a year late (for a 3-year policy).
  • ts_kv_partitions_cf (partition bookkeeping) gets a flat, full-duration TTL counted from migration time. This is intentional and safe: it is always >= any individual row's remaining TTL, so the partition bookkeeping entry can never expire before the data it references.
  • ts_kv_latest_cf (last known value per key) is intentionally not affected by -ttl at all, matching ThingsBoard's own TTL semantics where only historical points expire, not the current value.
  • Validation: -ttl must be a positive number of days and cannot exceed 7300 days (20 years) — Cassandra's hard-coded maximum TTL. Values outside this range are rejected up front with a clear error.
  • README updated with a new "Setting TTL for migrated data" section.

Testing

Ran end-to-end migrations against a real ThingsBoard Postgres database (Docker, ~8.4M ts_kv rows) using the documented dump commands.

Run 1 — flat TTL sanity check (-ttl 30, before the per-row calculation was added):

  • Exit code 0. Verified via the bundled SSTableExport/SSTableMetadataViewer that every row got ttl = 2592000 (30 days).
  • Row count check: 8,430,969 migrated vs 8,431,293 in source Postgres — the 324-row difference is expected (rows referencing entities missing from related_entities.dmp are skipped by design).

Run 2 — per-row calculation (-ttl 10), after adding the age-aware logic:

  • Exit code 0.
  • First attempt actually surfaced a real bug: Cassandra's TTL bind parameter is a 32-bit int, not bigint. Binding a Long threw a ClassCastException on every row, which went completely unnoticed at first because the packaged jar has no SLF4J binding and silently drops all log output (SLF4J: Defaulting to no-operation (NOP) logger implementation) — the process still exited 0 with ts_kv_cf completely empty. Reproduced standalone against CQLSSTableWriter to confirm the root cause, fixed by binding an Integer, and reran.
  • After the fix: verified via SSTableMetadataViewer that TTL varies per row (e.g. TTL max: 527311 ~= 6.1 days for the freshest rows, consistent with 10 days - ~3.9 days elapsed), ts_kv_partitions_cf stayed flat at 864000 (10 days) as designed, and exactly 12 rows were dropped as already-expired — matching an independent count of rows older than the 10-day cutoff directly from the source dump. Combined with the 324 orphaned-entity skips, this fully reconciles the migrated row count (8,431,293 - 336 = 8,430,957).
  • mvn clean compile assembly:single builds cleanly.

Note (separate, pre-existing issue found during testing)

The packaged jar has no SLF4J binding (only slf4j-api/log4j-over-slf4j), so all of the tool's logging (Lines processed, Lines migrated, errors) is silently dropped at runtime. This is unrelated to this change and not fixed here, but it's worth a follow-up: besides hiding progress, it also means real errors - like the ClassCastException above - fail completely silently, with the process still exiting 0.

Danylo Bosenko and others added 2 commits August 18, 2026 17:03
Migrated data currently lives forever in Cassandra since no TTL is set
on writes. Add an optional -ttl CLI argument (in days) that applies
USING TTL to ts_kv_cf and ts_kv_partitions_cf inserts. ts_kv_latest_cf
is intentionally left unaffected, matching ThingsBoard's own TTL
semantics where only historical points expire, not the latest value.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Reject non-positive ttl values and ttl values exceeding Cassandra's
hard-coded 20 year (7300 day) TTL maximum, instead of letting them
fail later with an opaque error from CQLSSTableWriter or silently
producing no TTL (ttl=0 means "no TTL" in Cassandra).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@hophead12
hophead12 marked this pull request as draft August 21, 2026 09:43
…estamp

Previously -ttl was applied as a flat duration counted from migration
time, so a row that was already 2 years old would get the full TTL
and end up expiring 1 year later than it should for a 3 year retention
policy. Now each ts_kv_cf row's TTL is computed as ttl - (migration_time
- row_ts), using a single "now" captured once at startup for all rows.
Rows whose age already exceeds the configured ttl are skipped instead
of migrated. ts_kv_partitions_cf keeps a flat full-duration TTL (from
migration time), which is intentional: it is always >= any individual
row's remaining TTL, so partition bookkeeping can never expire before
the data it references.

Also fixes a bug caught while testing this: Cassandra's TTL bind
parameter is a 32-bit int, not bigint - binding a Long threw a
ClassCastException on every row, which went completely unnoticed
during earlier testing because the packaged jar has no SLF4J binding
and silently drops all log output (separate pre-existing issue, not
fixed here).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@hophead12
hophead12 marked this pull request as ready for review August 21, 2026 09:56
@hophead12
hophead12 marked this pull request as draft August 21, 2026 12:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant