From 133d53d691575c3771c32718d7aefc50a1801031 Mon Sep 17 00:00:00 2001 From: wa0o Date: Wed, 2 Sep 2026 13:57:35 +0000 Subject: [PATCH 1/2] Record what the radio was doing, and turn PTT into a recording The recorder was already ported - WAV, continuous record and the replay ring, wired and unit-tested. What was missing is that a recording carried no provenance: filename in local time, no frequency, no mode. Every log worth matching against is UTC, so nothing downstream could join on it and "who was that" was unanswerable by construction. - Every recording now writes a .json sidecar: UTC start/end, frequency at start and end, mode, and the operator's overs. Manual, replay and auto alike. Provenance is pushed from the poll loop rather than pulled, because pulling took the poller's lock while holding the recorder's, from the poll thread. A replay's start time is derived from the sample count - the ring holds what happened before the press, so stamping it "now" would match it to the wrong contact. "overs": null means not tracked; an empty array would claim the operator never transmitted. - QsoRecorder ports the C# PTT auto-record: start on the rising edge, each over pushes the idle deadline out, stop on idle or a QSY measured from where the QSO started. It must not start on a tune, and a tune must not move the remembered PTT state. Off by default - it records whoever the operator is talking to, unasked. - RadioPoller::OnPoll is one fan-out point for everything that reacts to rig state, rather than a second loop competing for the CAT port. The UTC assertion was blind: this box runs UTC, so localtime_r and gmtime_r agree and the test passed against a local-time stamp. It now pins TZ to a POSIX string glibc parses without tzdata, asserts the two clocks actually differ, and compares the filename's local hour to the sidecar's UTC hour. Only then does localtime_r fail it. Same for the other three: each bug was reintroduced and watched to fail. Proven on the built host against the simulated rig, not only in tests: keyed, unkeyed, waited out the timer, got a 3.0 s WAV with rms 5656 and a sidecar carrying both UTC times, the frequency, the mode and one closed over. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01EqubrbjLXwz1GrVgmTcrf9 --- CMakeLists.txt | 5 +- WIP.md | 86 ++++++++++++ src/config.cpp | 6 + src/config.h | 7 + src/main.cpp | 19 +++ src/qso_record.cpp | 69 ++++++++++ src/qso_record.h | 65 +++++++++ src/radio.cpp | 2 + src/radio.h | 11 ++ src/recorder.cpp | 133 +++++++++++++++++- src/recorder.h | 43 +++++- tests/test_qso_record.cpp | 274 ++++++++++++++++++++++++++++++++++++++ 12 files changed, 711 insertions(+), 9 deletions(-) create mode 100644 src/qso_record.cpp create mode 100644 src/qso_record.h create mode 100644 tests/test_qso_record.cpp diff --git a/CMakeLists.txt b/CMakeLists.txt index 1899a27..5213c7c 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -38,7 +38,7 @@ FetchContent_Declare(nlohmann_json GIT_SHALLOW TRUE) FetchContent_MakeAvailable(nlohmann_json) -add_executable(hamdeck-host src/main.cpp src/log.cpp src/amp_tuner.cpp src/api.cpp src/transmit_routes.cpp src/cw_text.cpp src/http.cpp src/audio.cpp src/cat_sim.cpp src/serial_cat.cpp src/radio.cpp src/auth.cpp src/config.cpp src/tx_audio.cpp src/cat_proxy.cpp src/alsa_devices.cpp src/rig_cal.cpp src/alsa_audio.cpp src/tgxl.cpp src/recorder.cpp +add_executable(hamdeck-host src/main.cpp src/log.cpp src/amp_tuner.cpp src/api.cpp src/transmit_routes.cpp src/cw_text.cpp src/http.cpp src/audio.cpp src/cat_sim.cpp src/serial_cat.cpp src/radio.cpp src/auth.cpp src/config.cpp src/tx_audio.cpp src/cat_proxy.cpp src/alsa_devices.cpp src/rig_cal.cpp src/alsa_audio.cpp src/tgxl.cpp src/recorder.cpp src/qso_record.cpp src/session_stats.cpp) target_link_libraries(hamdeck-host PRIVATE civetweb-c-library OpenSSL::Crypto nlohmann_json::nlohmann_json asound) target_compile_options(hamdeck-host PRIVATE -Wall -Wextra) @@ -63,6 +63,9 @@ add_test(NAME audio_queue COMMAND test_audio_queue) add_executable(test_recorder tests/test_recorder.cpp src/recorder.cpp) add_test(NAME recorder COMMAND test_recorder) +add_executable(test_qso_record tests/test_qso_record.cpp src/qso_record.cpp src/recorder.cpp) +add_test(NAME qso_record COMMAND test_qso_record) + add_executable(test_session_stats tests/test_session_stats.cpp src/session_stats.cpp) add_test(NAME session_stats COMMAND test_session_stats) diff --git a/WIP.md b/WIP.md index 53a1005..5aba74b 100644 --- a/WIP.md +++ b/WIP.md @@ -1511,3 +1511,89 @@ one thing most likely to regress — if `OUTPUT_NAME` stops applying the bundle **Not yet proven on hardware.** CI checks structure; nobody has opened the renamed bundle in Finder or keyed up on a Mac. + +--- + +## 09/02/2026 — PTT auto-record, and provenance on every recording + +Started from the wrong premise ("port recording like the C# has") and measured before +building: `src/recorder.cpp` already does WAV writing, continuous record AND the replay ring, +wired at `main.cpp:213` and unit-tested. `AUDIT-CSHARP.md:42` had it as ✅ ported all along. + +**The real gap was that a recording carried no provenance.** Files were +`hamdeck-rec-.wav` and nothing else — no frequency, no mode, and the only +timestamp in local time while every log worth matching against is UTC. Nothing downstream +could join on that, so "who was it" was unanswerable by construction. + +### 1. Every recording now writes a `.json` sidecar +UTC start/end, rig-connected, frequency at start **and** end, mode, sample rate, and the +operator's overs. Written for manual, replay and auto recordings alike. + +⚠️ **Provenance is PUSHED from the poll loop, not pulled.** The first cut had the recorder +call back into `RadioPoller` for the current frequency — that takes the poller's lock while +holding the recorder's, from the poll thread, waiting on the one API path that nests them the +other way. `RadioPoller::OnPoll` hands the values over instead; `Recorder::UpdateProvenance` +keeps them under its own small lock. One fan-out point, no second poller competing for CAT. + +⚠️ **A replay clip's `started_utc` is in the PAST**, derived from the sample count. The ring +holds what happened *before* the press; stamping it "now" files the audio minutes after the +exchange it contains and matches it to the wrong QSO. + +⚠️ **`"overs": null` ≠ `"overs": []`.** Null means not tracked (a replay clip). An empty array +would claim the operator never transmitted. Different facts, and the second one is a lie. + +### 2. `src/qso_record.cpp` — PTT auto-record, ported from the C# behaviour +`Views/MainWindow.xaml.cs:420`, not invented: start on the PTT rising edge, every later over +pushes an idle deadline out, stop on idle (`ptt_record_seconds`, 60) or a QSY past +`ptt_record_qsy_khz` (10) **from where the QSO started** — a reading-to-reading comparison +never trips on someone tuning across the band in small steps. + +⚠️ **It must not start on a tune.** Keying the tuner is PTT to the rig; without the guard every +band change litters the directory with two-second files no log will ever match. A tune also +must not move the remembered PTT state, or its unkey closes an over that never opened. + +⚠️ **OFF by default** (`ptt_record_enabled`). It records whoever the operator is talking to, +unasked. That is a decision made once in the config, not a default that arrives with an update. + +### The gates — `tests/test_qso_record.cpp`, PROVEN to fail +| reintroduced | caught by | +|---|---| +| `NoteOver` before the file opens | `overs == 2` — the over that STARTS each QSO went unlisted while every later one was recorded | +| `localtime_r` for the sidecar | the local-vs-UTC hour comparison (see below) | +| replay stamped when saved | `age >= 19 && age < 60` | +| auto-record starting on a tune | `!q.active()` | + +⚠️⚠️ **THE UTC TEST WAS BLIND AND PASSED ANYWAY.** The build box runs UTC, so `localtime_r` +and `gmtime_r` return the same thing and the assertion held just as happily against a +local-time stamp. The test now sets `TZ` to a **POSIX string** (`CST6CDT,M3.2.0/2,M11.1.0/2`, +which glibc parses with no tzdata, so a bare container cannot silently drop it back to UTC), +asserts the two clocks actually differ before testing anything, and compares the filename's +local hour against the sidecar's UTC hour. Only then did swapping in `localtime_r` fail. + +### Proven on the built host, not only in tests +Ran `hamdeck-host` against the simulated rig with `ptt_record_seconds: 3`, keyed via +`/api/ptt/on`, unkeyed, waited out the idle timer: +`hamdeck-qso-09-02-2026-135540.wav` — 3.0 s, 1ch/16bit/22050, **rms 5656 peak 7997** (real +audio, not an empty header) — beside a sidecar carrying `trigger: idle`, both UTC times, +14074000 Hz, USB, and one closed over. + +⚠️ **`pkill -f hamdeck-host` KILLS THE SHELL THAT RUNS IT** — the pattern matches the +command line of the very shell issuing it. Cost a lost commit. Use `pkill -x hamdeck-host`. + +### Next, in order +1. **Re-measure MONI.** `CARRYOVER.md:207` says it cannot be captured (120 s of `/ws`, band + noise only); the C# says the opposite at `WsAudioClient.cs:205` and mutes RX while keyed + *because* the operator hears themselves. `RadioController.cs:687` is the likely + explanation — MON needs **`ML0001;` to enable AND `ML1;` to set the level**, and + enable-at-level-0 is on and silent, which reads exactly like "no transmission in it". + Gate: recorded RMS during a keyed window vs the same window unkeyed, into a dummy load. + ⚠️ It matters beyond convenience: recording the host's own `/ws/tx` PCM would have looked + perfect through every one of the six TX-chain bugs. MONI is the only source that proves + audio actually left the radio. +2. **Identification, layered** — Wavelog QSO in the window and band ⇒ the callsign, stated as + fact; otherwise the NetLogger roster for whatever net was up ⇒ **candidates**, stated as + candidates. Never the same kind of claim, and never "nobody" as a finding. + ⚠️ **NetLogger has a hard 7-day wall and no bulk history endpoint.** Past net rosters + cannot be fetched retroactively — the only source is `netlogger_poll.py`'s + `netlogger_checkin` table, and only for the period it has actually been polling. Anything + older is unattributable from the net side, permanently. diff --git a/src/config.cpp b/src/config.cpp index 2154813..13bf04e 100644 --- a/src/config.cpp +++ b/src/config.cpp @@ -88,6 +88,9 @@ bool Config::Load(const std::string& path, Config& out, std::string& error) { Get(j, "record_buffer_seconds", cfg.record_buffer_seconds); Get(j, "record_max_seconds", cfg.record_max_seconds); Get(j, "record_warning_seconds", cfg.record_warning_seconds); + Get(j, "ptt_record_enabled", cfg.ptt_record_enabled); + Get(j, "ptt_record_seconds", cfg.ptt_record_seconds); + Get(j, "ptt_record_qsy_khz", cfg.ptt_record_qsy_khz); Get(j, "api_port", cfg.api_port); Get(j, "cat_proxy_port", cfg.cat_proxy_port); Get(j, "dashboard_port", cfg.dashboard_port); @@ -173,6 +176,9 @@ bool Config::Save(const std::string& path, std::string& error) const { j["record_path"] = record_path; j["record_buffer_seconds"] = record_buffer_seconds; j["record_max_seconds"] = record_max_seconds; + j["ptt_record_enabled"] = ptt_record_enabled; + j["ptt_record_seconds"] = ptt_record_seconds; + j["ptt_record_qsy_khz"] = ptt_record_qsy_khz; j["alsa_capture_device"] = alsa_capture_device; j["alsa_playback_device"] = alsa_playback_device; j["api_port"] = api_port; diff --git a/src/config.h b/src/config.h index 1af306e..69e2d26 100644 --- a/src/config.h +++ b/src/config.h @@ -59,6 +59,13 @@ struct Config { int record_max_seconds = 10800; // hard ceiling; 0 disables the ceiling int record_warning_seconds = 300; + // PTT auto-record. ⚠️ OFF unless the operator turns it on: it records whoever + // they are talking to, unasked, and that is their call to make once rather + // than a default that arrives with an update. + bool ptt_record_enabled = false; + int ptt_record_seconds = 60; // idle time after the last over + int ptt_record_qsy_khz = 10; // QSY from the start freq that ends it + // API int api_port = 5001; // control, bound to loopback diff --git a/src/main.cpp b/src/main.cpp index 4eca1e3..17f744f 100644 --- a/src/main.cpp +++ b/src/main.cpp @@ -22,6 +22,7 @@ #include "auth.h" #include "alsa_audio.h" #include "cat_sim.h" +#include "qso_record.h" #include "recorder.h" #include "session_stats.h" #include "tgxl.h" @@ -219,6 +220,7 @@ int main(int argc, char** argv) { rx_audio.SetRecorder(&recorder); } + rx_audio.Start(); // TX audio. The null sink discards: the codec is on the reference host, so @@ -274,6 +276,23 @@ int main(int argc, char** argv) { AmpTuner amp(tgxl_rig); if (tgxl.configured()) std::cout << "TGXL: " << tgxl.Describe() << '\n' << std::flush; + // ⚠️ FED FROM THE POLL LOOP, NOT ITS OWN TIMER. QsoRecorder is what turns PTT + // into a recording, and it also hands the recorder the frequency and mode + // that go in every sidecar - so it is wired up even when auto-record is off. + // The tuner check is here rather than inside it because the amp and the TGXL + // are what know a tune is running, and keying for a tune is PTT to the rig. + QsoRecorder qso_record(&recorder, QsoRecorder::Options{ + config.ptt_record_enabled, config.ptt_record_seconds, + static_cast(config.ptt_record_qsy_khz) * 1000}); + poller.OnPoll([&](bool connected, long long freq, const std::string& mode, bool tx) { + const bool tuning = amp.IsActive() || tgxl.IsActive(); + qso_record.Observe(connected, freq, mode, tx, tuning); + }); + if (config.ptt_record_enabled && recorder.available()) { + std::cout << "ptt auto-record: on (" << config.ptt_record_seconds + << "s idle, " << config.ptt_record_qsy_khz << " kHz QSY)\n" << std::flush; + } + HostState host_state; ApiDeps deps; diff --git a/src/qso_record.cpp b/src/qso_record.cpp new file mode 100644 index 0000000..15eb78d --- /dev/null +++ b/src/qso_record.cpp @@ -0,0 +1,69 @@ +#include "qso_record.h" + +#include "recorder.h" + +QsoRecorder::QsoRecorder(Recorder* rec, Options opts, + std::function now) + : rec_(rec), opts_(opts), now_(std::move(now)) {} + +void QsoRecorder::Observe(bool connected, long long freq_hz, + const std::string& mode, bool tx, bool tuning) { + if (!rec_) return; + + // The poll loop is the only place that knows all of this at once, so it is + // also where the sidecar's frequency comes from - fed always, so a MANUAL + // recording gets provenance too. + rec_->UpdateProvenance(connected, freq_hz, mode); + + const bool rising = tx && !last_tx_; + const bool falling = !tx && last_tx_; + // ⚠️ A tune does not move the remembered PTT state at all (the C# does the + // same at MainWindow.xaml.cs:438). Letting it would mean the tune's unkey + // registers as the end of an over the operator never started. + if (!tuning) last_tx_ = tx; + + if (opts_.enabled && rising && !tuning) { + if (!active_) { + // ⚠️ active_ is set from whether the file OPENED, never from having + // decided to record - the same rule the Recorder itself follows. A full + // disk must not leave a state machine believing it is recording. + const auto r = rec_->Start("qso"); + if (!r.ok) return; + active_ = true; + start_freq_ = freq_hz; + } + deadline_ = now_() + std::chrono::seconds(opts_.idle_seconds); + } + + // ⚠️ AFTER the start, not before. Noting the over first drops the very over + // that began the recording: NoteOver does nothing when no file is open, so + // the first transmission of every QSO went unlisted while every later one + // was recorded. Caught by the test asserting two overs, not by reading it. + if (!tuning && (rising || falling)) rec_->NoteOver(tx); + + if (!active_) return; + + if (now_() > deadline_) { + Stop("idle"); + return; + } + // ⚠️ QSY is measured from where the QSO STARTED, not from the last reading. + // Tuning across the band in small steps would never trip a + // reading-to-reading comparison, and the recording would run until the idle + // timer caught it - filed under a frequency the operator left long ago. + if (start_freq_ > 0 && connected) { + const long long moved = freq_hz > start_freq_ ? freq_hz - start_freq_ + : start_freq_ - freq_hz; + if (moved > opts_.qsy_threshold_hz) Stop("qsy"); + } +} + +void QsoRecorder::Stop(const std::string& reason) { + // Stop() writes the sidecar with this reason in it, so a recording says how + // it ended rather than only when. + rec_->Stop(reason); + active_ = false; + start_freq_ = 0; + last_stop_ = reason; + ++stopped_; +} diff --git a/src/qso_record.h b/src/qso_record.h new file mode 100644 index 0000000..e42c300 --- /dev/null +++ b/src/qso_record.h @@ -0,0 +1,65 @@ +#pragma once + +// PTT auto-record: a recording that brackets a QSO without being asked for. +// +// Ported from the C# panel (Views/MainWindow.xaml.cs:420) rather than invented, +// because the behaviour is the operator's habit and not a design question: +// - the first time PTT goes down, start recording +// - every later press pushes an idle deadline out +// - stop when the operator has been quiet for idle_seconds, or has QSY'd +// further than qsy_threshold_hz from where the QSO started +// +// ⚠️ IT MUST NOT START ON A TUNE. Keying an antenna tuner is PTT as far as the +// rig is concerned, and a tune at the top of every band change would litter the +// directory with two-second files that no log will ever match. +// +// ⚠️ OFF BY DEFAULT. This writes audio of whoever the operator is talking to, +// unasked. That is a decision for the operator to make once, in the config, not +// something a version bump turns on for them. + +#include +#include +#include + +class Recorder; + +class QsoRecorder { + public: + using Clock = std::chrono::steady_clock; + + struct Options { + bool enabled = false; + int idle_seconds = 60; // C# PTTRecordSeconds default + long long qsy_threshold_hz = 10000; // C# PTTQSYThresholdKHz default, 10 kHz + }; + + // The clock is injectable so the idle timeout can be tested without waiting + // a minute for it - a test that sleeps for the real timeout gets deleted or + // shortened until it no longer tests the thing. + QsoRecorder(Recorder* rec, Options opts, + std::function now = [] { return Clock::now(); }); + + // Fed once per poll cycle, from the same place SessionStats is fed. + void Observe(bool connected, long long freq_hz, const std::string& mode, + bool tx, bool tuning); + + bool active() const { return active_; } + // Why the last automatic recording stopped: "idle", "qsy", or empty if none + // has. Reported so the operator can tell a finished QSO from a cut-off one. + std::string last_stop_reason() const { return last_stop_; } + int stopped_count() const { return stopped_; } + + private: + void Stop(const std::string& reason); + + Recorder* rec_; + Options opts_; + std::function now_; + + bool active_ = false; + bool last_tx_ = false; + long long start_freq_ = 0; + Clock::time_point deadline_{}; + std::string last_stop_; + int stopped_ = 0; +}; diff --git a/src/radio.cpp b/src/radio.cpp index e0bcb0f..6349d30 100644 --- a/src/radio.cpp +++ b/src/radio.cpp @@ -120,6 +120,7 @@ void RadioPoller::PollOnce() { auto id = cat_->Exchange("ID;"); if (!id.has_value()) { if (stats_) stats_->Observe(false, 0, "", false); + if (poll_cb_) poll_cb_(false, 0, "", false); std::lock_guard lock(mu_); snap_ = s; // connected = false snap_.taken = std::chrono::steady_clock::now(); @@ -159,6 +160,7 @@ void RadioPoller::PollOnce() { s.taken = std::chrono::steady_clock::now(); if (stats_) stats_->Observe(true, s.freq, s.mode, s.tx); + if (poll_cb_) poll_cb_(true, s.freq, s.mode, s.tx); CheckWatchdog(s.tx); std::lock_guard lock(mu_); snap_ = s; diff --git a/src/radio.h b/src/radio.h index e11951c..6099528 100644 --- a/src/radio.h +++ b/src/radio.h @@ -105,6 +105,16 @@ class RadioPoller { // the counts follow the RADIO rather than any one client. void SetSessionStats(SessionStats* stats) { stats_ = stats; } + // ⚠️ ONE FAN-OUT POINT, NOT A SECOND POLLER. Everything that needs to react + // to what the rig is doing - session stats, PTT auto-record - is fed from the + // one cycle that already asked. A feature that polls the rig on its own + // schedule competes for the CAT port with the loop that keeps the panel live. + // Called on the poll thread with no lock held; keep the callback short. + void OnPoll(std::function cb) { + poll_cb_ = std::move(cb); + } + void SetPttTimeoutSeconds(int seconds) { ptt_timeout_s_.store(seconds); } int PttTimeoutSeconds() const { return ptt_timeout_s_.load(); } @@ -168,6 +178,7 @@ class RadioPoller { std::atomic ptt_timeout_s_{kDefaultPttTimeoutSeconds}; SessionStats* stats_ = nullptr; + std::function poll_cb_; std::atomic watchdog_trips_{0}; std::function watchdog_cb_; std::chrono::steady_clock::time_point keyed_since_{}; diff --git a/src/recorder.cpp b/src/recorder.cpp index 67b6042..1dc5884 100644 --- a/src/recorder.cpp +++ b/src/recorder.cpp @@ -29,6 +29,29 @@ std::string Stamp() { return buf; } +// ⚠️ UTC, ISO 8601, ALWAYS - and deliberately not the same clock the FILENAME +// uses. The name is local time because that is how the operator reads a +// directory (house style, MM-DD-YYYY); the sidecar is UTC because that is what +// a log is in. Deriving one from the other is the whole class of timezone bug +// that has bitten this operator's other tooling. +std::string Utc(std::chrono::system_clock::time_point tp) { + const std::time_t t = std::chrono::system_clock::to_time_t(tp); + std::tm tm{}; + gmtime_r(&t, &tm); + char buf[32]; + std::strftime(buf, sizeof(buf), "%Y-%m-%dT%H:%M:%SZ", &tm); + return buf; +} + +std::string JsonEscape(const std::string& in) { + std::string out; + for (char c : in) { + if (c == '"' || c == '\\') { out += '\\'; out += c; } + else if (static_cast(c) >= 0x20) out += c; + } + return out; +} + } // namespace Recorder::Recorder(std::string dir, int sample_rate, int buffer_seconds, @@ -146,7 +169,7 @@ void Recorder::CloseFile() { file_ = nullptr; } -Recorder::Result Recorder::Start() { +Recorder::Result Recorder::Start(const std::string& tag) { Result r; if (!available_) { r.message = reason_; @@ -163,15 +186,20 @@ Recorder::Result Recorder::Start() { " s ago; stop it first"; return r; } - r = OpenFile("rec"); + r = OpenFile(tag); // ⚠️ recording_ is set from whether the file actually opened, never from // having been asked. That is the whole lesson of the route this replaces. recording_.store(r.ok); - if (r.ok) r.message = "recording"; + if (r.ok) { + file_started_ = std::chrono::system_clock::now(); + file_start_prov_ = Ask(); + overs_.clear(); + r.message = "recording"; + } return r; } -Recorder::Result Recorder::Stop() { +Recorder::Result Recorder::Stop(const std::string& trigger) { Result r; std::lock_guard lock(mu_); if (!file_) { @@ -180,6 +208,9 @@ Recorder::Result Recorder::Stop() { } r.filename = file_path_; const int secs = static_cast(file_frames_ / rate_); + // ⚠️ Before CloseFile(), which clears file_path_. The sidecar is written for + // the file that just closed, from the state that produced it. + WriteSidecar(file_path_, trigger, file_started_, file_start_prov_, &overs_); CloseFile(); recording_.store(false); r.ok = true; @@ -208,13 +239,103 @@ Recorder::Result Recorder::SaveReplay() { WriteWavHeader(f, static_cast(snap.size() * 2)); std::fwrite(snap.data(), sizeof(int16_t), snap.size(), f); std::fclose(f); + // ⚠️ THE REPLAY'S START TIME IS IN THE PAST. The whole point of the ring is + // that it holds what happened BEFORE the operator pressed anything, so + // stamping the sidecar with "now" would file the audio minutes after the + // exchange it contains and match it to the wrong QSO - or to none. Derived + // from the sample count, which is what the audio actually is. + const auto secs = static_cast(snap.size() / rate_); + const auto began = std::chrono::system_clock::now() - std::chrono::seconds(secs); + // ⚠️ Provenance is read at save time, not capture time: the frequency is + // where the rig is NOW, which is where it was during the buffer only if it + // has not moved. Good enough to match on, and the sidecar says start and end + // separately so a QSY between them is visible rather than hidden. + WriteSidecar(path, "replay", began, Ask(), nullptr); + r.ok = true; r.filename = path; - r.message = std::format("saved {} s from the buffer", - static_cast(snap.size() / rate_)); + r.message = std::format("saved {} s from the buffer", secs); return r; } +void Recorder::UpdateProvenance(bool connected, long long freq_hz, + const std::string& mode) { + std::lock_guard lock(prov_mu_); + prov_.connected = connected; + prov_.freq_hz = freq_hz; + prov_.mode = mode; +} + +Recorder::Provenance Recorder::Ask() const { + // ⚠️ NO GUESSING. Until the poller has handed over a connected reading, the + // sidecar says connected:false and carries no frequency - it does not carry a + // stale one. A recording filed under the wrong band is worse than one filed + // under none, because the wrong one gets matched to a QSO. + std::lock_guard lock(prov_mu_); + return prov_; +} + +void Recorder::NoteOver(bool keyed) { + std::lock_guard lock(mu_); + if (!file_) return; // overs only mean something inside a recording + const auto now = std::chrono::system_clock::now(); + if (keyed) { + if (!overs_.empty() && overs_.back().open) return; // already keyed + overs_.push_back({now, now, true}); + } else if (!overs_.empty() && overs_.back().open) { + overs_.back().end = now; + overs_.back().open = false; + } +} + +void Recorder::WriteSidecar(const std::string& wav_path, const std::string& trigger, + std::chrono::system_clock::time_point started, + const Provenance& at_start, + const std::vector* overs_in) const { + const Provenance at_end = Ask(); + const auto ended = std::chrono::system_clock::now(); + + std::string overs = "null"; + if (overs_in) { + overs = "["; + for (size_t i = 0; i < overs_in->size(); ++i) { + const auto& o = (*overs_in)[i]; + // An over still open at close is a recording that ended mid-transmission - + // reported as such rather than given an invented end. + overs += std::format(R"({}{{"start_utc":"{}","end_utc":{}}})", + i ? "," : "", Utc(o.start), + o.open ? "null" : ("\"" + Utc(o.end) + "\"")); + } + overs += "]"; + } + + const std::string path = wav_path + ".json"; + std::FILE* f = std::fopen(path.c_str(), "wb"); + if (!f) return; // the audio is saved; a missing sidecar must not lose it + const std::string json = std::format( + "{{\n" + R"( "file": "{}",)" "\n" + R"( "trigger": "{}",)" "\n" + R"( "started_utc": "{}",)" "\n" + R"( "ended_utc": "{}",)" "\n" + R"( "rig_connected": {},)" "\n" + R"( "freq_hz_start": {},)" "\n" + R"( "freq_hz_end": {},)" "\n" + R"( "mode": "{}",)" "\n" + R"( "sample_rate": {},)" "\n" + R"( "channels": 1,)" "\n" + R"( "overs": {})" "\n" + "}}\n", + JsonEscape(std::filesystem::path(wav_path).filename().string()), + JsonEscape(trigger), Utc(started), Utc(ended), + at_start.connected ? "true" : "false", + at_start.connected ? at_start.freq_hz : 0, + at_end.connected ? at_end.freq_hz : 0, + JsonEscape(at_start.mode), rate_, overs); + std::fwrite(json.data(), 1, json.size(), f); + std::fclose(f); +} + int Recorder::recorded_seconds() const { std::lock_guard lock(mu_); return file_ ? static_cast(file_frames_ / rate_) : 0; diff --git a/src/recorder.h b/src/recorder.h index e032be7..311dc5e 100644 --- a/src/recorder.h +++ b/src/recorder.h @@ -11,6 +11,7 @@ // If the file could not be opened, recording is false and the reason is said. #include +#include #include #include #include @@ -33,8 +34,28 @@ class Recorder { std::string message; }; - Result Start(); - Result Stop(); + // ⚠️ WHAT THE RADIO WAS DOING IS PART OF THE RECORDING. A .wav on its own + // cannot be matched to a log: the filename is local time by house style, and + // every log worth matching against (Wavelog QSOs, NetLogger check-ins) is UTC. + // So each file gets a .json sidecar carrying UTC start/end, frequency, mode + // and the operator's overs. Set by main.cpp from the rig poller; without it + // the sidecar still gets written, saying it did not know rather than guessing. + struct Provenance { + bool connected = false; + long long freq_hz = 0; + std::string mode; + }; + void UpdateProvenance(bool connected, long long freq_hz, const std::string& mode); + + // An "over" is one transmission. Fed from the same PTT edges the auto-record + // watches, so a long recording can be navigated by who was talking when - + // in a net recording the operator's overs are what bracket each exchange. + void NoteOver(bool keyed); + + Result Start(const std::string& tag = "rec"); + // The trigger is recorded in the sidecar: what stopped this recording is the + // difference between a QSO that ended and a disk limit that cut one off. + Result Stop(const std::string& trigger = "manual"); // ⚠️ Writes the ring buffer - the audio from BEFORE the operator pressed // anything. That is the entire point: you press it after hearing something, // not before. @@ -52,6 +73,16 @@ class Recorder { bool WriteWavHeader(std::FILE* f, uint32_t data_bytes) const; Result OpenFile(const std::string& tag); void CloseFile(); + Provenance Ask() const; // takes prov_mu_, never mu_ + // Assumes mu_ is held. + struct Over { std::chrono::system_clock::time_point start, end; bool open = false; }; + // ⚠️ overs == nullptr means NOT TRACKED, and the sidecar says null rather than + // [] - a replay clip has no over list, and an empty array would claim the + // operator never transmitted during it. Those are different facts. + void WriteSidecar(const std::string& wav_path, const std::string& trigger, + std::chrono::system_clock::time_point started, + const Provenance& at_start, + const std::vector* overs) const; std::string dir_; std::string reason_; @@ -67,6 +98,14 @@ class Recorder { std::string file_path_; uint32_t file_frames_ = 0; + // Its own lock, held only for the copy - never while mu_ is being taken. + mutable std::mutex prov_mu_; + Provenance prov_; + std::chrono::system_clock::time_point file_started_{}; + Provenance file_start_prov_; + // Each entry is one over: UTC start, and UTC end once it is unkeyed. + std::vector overs_; + std::atomic recording_{false}; std::atomic buffering_{false}; std::atomic frames_fed_{0}; diff --git a/tests/test_qso_record.cpp b/tests/test_qso_record.cpp new file mode 100644 index 0000000..c18208f --- /dev/null +++ b/tests/test_qso_record.cpp @@ -0,0 +1,274 @@ +// PTT auto-record, and the sidecar that makes a recording identifiable. +// +// ⚠️ THE POINT OF THIS FILE IS THAT NONE OF IT CAN BE CHECKED BY LOOKING. A +// recording that starts on a tune, or stops on the wrong edge, or carries a +// local-time stamp where a log expects UTC, produces a .wav that plays +// perfectly and matches the wrong contact - or none. Every assertion here is +// against what landed on disk. + +#include "check.h" + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "../src/qso_record.h" +#include "../src/recorder.h" + +namespace fs = std::filesystem; + +namespace { + +// A controllable clock, so a 60-second idle timeout costs no wall time. A test +// that has to sleep for the real timeout gets shortened until it stops testing +// the thing it was written for. +std::chrono::steady_clock::time_point g_now = std::chrono::steady_clock::now(); +void Advance(int seconds) { g_now += std::chrono::seconds(seconds); } + +std::vector Tone(size_t frames) { return std::vector(frames, 1000); } + +std::vector Wavs(const fs::path& dir) { + std::vector out; + for (const auto& e : fs::directory_iterator(dir)) + if (e.path().extension() == ".wav") out.push_back(e.path()); + return out; +} + +std::string Read(const fs::path& p) { + std::ifstream in(p); + std::ostringstream ss; + ss << in.rdbuf(); + return ss.str(); +} + +// Deliberately crude: no JSON library is linked here, and a real parser would +// hide a malformed sidecar behind a helpful error. This asserts on the bytes. +bool Has(const std::string& hay, const std::string& needle) { + return hay.find(needle) != std::string::npos; +} + +std::string Field(const std::string& json, const std::string& key) { + const auto k = "\"" + key + "\": "; + const auto at = json.find(k); + if (at == std::string::npos) return ""; + auto from = at + k.size(); + if (json[from] == '"') { + ++from; + return json.substr(from, json.find('"', from) - from); + } + const auto end = json.find_first_of(",\n", from); + return json.substr(from, end - from); +} + +} // namespace + +int main() { + // ⚠️ THIS TEST IS BLIND ON A UTC BOX, AND THE BUILD BOX IS ONE. With TZ=UTC, + // localtime_r and gmtime_r return the same thing, so the assertion that the + // sidecar is UTC passes just as happily against a local-time stamp. Verified: + // swapping gmtime_r for localtime_r in recorder.cpp did NOT fail this file + // until this line existed. + // + // A POSIX TZ string rather than "America/Chicago" on purpose: glibc parses it + // with no zoneinfo file, so the test keeps its teeth on a bare container that + // ships no tzdata instead of silently falling back to UTC. + setenv("TZ", "CST6CDT,M3.2.0/2,M11.1.0/2", 1); + tzset(); + { + const std::time_t t = std::time(nullptr); + std::tm l{}, g{}; + localtime_r(&t, &l); + gmtime_r(&t, &g); + // If these agree the test cannot tell UTC from local, and every timezone + // assertion below is decoration. Fail loudly rather than pass emptily. + CHECK(timegm(&l) != timegm(&g)); + } + + const fs::path dir = fs::temp_directory_path() / "hamdeck-qso-test"; + fs::remove_all(dir); + fs::create_directories(dir); + + const int rate = 8000; + auto make = [&](QsoRecorder::Options opts) { + return opts; + }; + + // ── A tune must not start a recording ───────────────────────────────────── + { + Recorder rec(dir.string(), rate, 5, 0, 0); + CHECK(rec.available()); + QsoRecorder q(&rec, make({true, 60, 10000}), [] { return g_now; }); + q.Observe(true, 7185000, "LSB", true, /*tuning=*/true); + rec.Feed(Tone(rate).data(), rate); + q.Observe(true, 7185000, "LSB", false, /*tuning=*/true); + CHECK(!q.active()); + CHECK(!rec.recording()); + CHECK(Wavs(dir).empty()); + std::cout << " ok a tune keys the rig and starts nothing\n"; + } + + // ── PTT starts it; a later over pushes the deadline out ─────────────────── + { + fs::remove_all(dir); fs::create_directories(dir); + Recorder rec(dir.string(), rate, 5, 0, 0); + QsoRecorder q(&rec, make({true, 60, 10000}), [] { return g_now; }); + + q.Observe(true, 7185000, "LSB", true, false); + CHECK(q.active()); + CHECK(rec.recording()); + rec.Feed(Tone(rate).data(), rate); + q.Observe(true, 7185000, "LSB", false, false); + + // 50 s of listening: not idle yet. + Advance(50); + q.Observe(true, 7185000, "LSB", false, false); + CHECK(q.active()); + + // A second over resets the clock, so 50 s more must still not end it. + q.Observe(true, 7185000, "LSB", true, false); + q.Observe(true, 7185000, "LSB", false, false); + Advance(50); + q.Observe(true, 7185000, "LSB", false, false); + CHECK(q.active()); + std::cout << " ok each over pushes the idle deadline out\n"; + + // Now let it go quiet past the timeout. + Advance(20); + q.Observe(true, 7185000, "LSB", false, false); + CHECK(!q.active()); + CHECK(!rec.recording()); + CHECK(q.last_stop_reason() == "idle"); + + const auto wavs = Wavs(dir); + CHECK(wavs.size() == 1); + CHECK(wavs[0].filename().string().starts_with("hamdeck-qso-")); + + // ── The sidecar ─────────────────────────────────────────────────────── + const fs::path side = wavs[0].string() + ".json"; + CHECK(fs::exists(side)); + const std::string json = Read(side); + + CHECK(Field(json, "trigger") == "idle"); + CHECK(Field(json, "freq_hz_start") == "7185000"); + CHECK(Field(json, "mode") == "LSB"); + CHECK(Field(json, "rig_connected") == "true"); + + // ⚠️ UTC, AND NOT DERIVED FROM THE FILENAME. The name is local time by + // house style; every log this will be matched against is UTC. A sidecar + // that quietly carried local time would match a QSO five hours away. + const std::string started = Field(json, "started_utc"); + CHECK(started.size() == 20); + CHECK(started.back() == 'Z'); + CHECK(started[10] == 'T'); + // It must actually BE UTC, not local time with a Z stapled on. + { + std::tm tm{}; + std::istringstream in(started); + in >> std::get_time(&tm, "%Y-%m-%dT%H:%M:%SZ"); + CHECK(!in.fail()); + const std::time_t parsed = timegm(&tm); + const std::time_t now = std::time(nullptr); + const double off = std::difftime(now, parsed); + CHECK(off >= 0 && off < 120); + } + // The filename is LOCAL time, the sidecar is UTC, and under a non-UTC TZ + // the two must disagree. This is the assertion that actually catches a + // sidecar written with localtime_r. + { + const std::string name = wavs[0].filename().string(); // hamdeck-qso-MM-DD-YYYY-HHMMSS.wav + const std::string local_hh = name.substr(name.size() - 10, 2); + const std::string utc_hh = started.substr(11, 2); + CHECK(local_hh != utc_hh); + } + std::cout << " ok the sidecar's start time is real UTC, not a relabelled local clock\n"; + + // Both overs are there, and both are closed. + CHECK(Has(json, "\"overs\": [")); + size_t overs = 0, at = 0; + while ((at = json.find("\"start_utc\"", at)) != std::string::npos) { ++overs; ++at; } + CHECK(overs == 2); + CHECK(!Has(json, "\"end_utc\":null")); + std::cout << " ok both overs are recorded and closed\n"; + } + + // ── A QSY ends it, and is measured from where the QSO STARTED ───────────── + { + fs::remove_all(dir); fs::create_directories(dir); + Recorder rec(dir.string(), rate, 5, 0, 0); + QsoRecorder q(&rec, make({true, 60, 10000}), [] { return g_now; }); + + q.Observe(true, 7185000, "LSB", true, false); + q.Observe(true, 7185000, "LSB", false, false); + CHECK(q.active()); + + // Small steps: no single one trips a reading-to-reading test. + q.Observe(true, 7190000, "LSB", false, false); // +5 kHz + CHECK(q.active()); + // ⚠️ EXACTLY at the threshold is NOT a QSY - the test is `moved > + // threshold`, so 10 kHz from a 10 kHz setting still counts as the same + // QSO. Asserted because it is the kind of off-by-one nobody notices until + // a recording splits in two. + q.Observe(true, 7195000, "LSB", false, false); // +10 kHz exactly + CHECK(q.active()); + q.Observe(true, 7200000, "LSB", false, false); // +15 kHz + CHECK(!q.active()); + CHECK(q.last_stop_reason() == "qsy"); + const auto wavs = Wavs(dir); + CHECK(wavs.size() == 1); + const std::string json = Read(fs::path(wavs[0].string() + ".json")); + CHECK(Field(json, "trigger") == "qsy"); + // Start and end frequencies are both there, so the move is visible. + CHECK(Field(json, "freq_hz_start") == "7185000"); + CHECK(Field(json, "freq_hz_end") == "7200000"); + std::cout << " ok a QSY in small steps still ends the recording\n"; + } + + // ── Off by default ──────────────────────────────────────────────────────── + { + fs::remove_all(dir); fs::create_directories(dir); + Recorder rec(dir.string(), rate, 5, 0, 0); + QsoRecorder q(&rec, QsoRecorder::Options{}, [] { return g_now; }); + CHECK(QsoRecorder::Options{}.enabled == false); + q.Observe(true, 7185000, "LSB", true, false); + q.Observe(true, 7185000, "LSB", false, false); + CHECK(!q.active()); + CHECK(Wavs(dir).empty()); + std::cout << " ok auto-record records nothing until it is turned on\n"; + } + + // ── A replay clip's sidecar is stamped when the AUDIO happened ──────────── + { + fs::remove_all(dir); fs::create_directories(dir); + Recorder rec(dir.string(), rate, 30, 0, 0); + rec.UpdateProvenance(true, 14074000, "USB"); + rec.Feed(Tone(rate * 20).data(), rate * 20); // 20 s in the ring + const auto r = rec.SaveReplay(); + CHECK(r.ok); + const std::string json = Read(fs::path(r.filename + ".json")); + CHECK(Field(json, "trigger") == "replay"); + CHECK(Field(json, "freq_hz_start") == "14074000"); + // ⚠️ The ring holds what happened BEFORE the press. Stamping it "now" would + // file 20 s of audio 20 s after the exchange it contains. + std::tm tm{}; + std::istringstream in(Field(json, "started_utc")); + in >> std::get_time(&tm, "%Y-%m-%dT%H:%M:%SZ"); + CHECK(!in.fail()); + const double age = std::difftime(std::time(nullptr), timegm(&tm)); + CHECK(age >= 19 && age < 60); + // An untracked over list must say null, not [] - "not recorded" is not + // the same claim as "the operator never transmitted". + CHECK(Has(json, "\"overs\": null")); + std::cout << " ok a replay is stamped when the audio happened, not when it was saved\n"; + } + + fs::remove_all(dir); + std::cout << "qso_record: all checks passed\n"; + return 0; +} From 016f54dca4c3778098443ce4b054eaf0767045a1 Mon Sep 17 00:00:00 2001 From: wa0o Date: Wed, 2 Sep 2026 14:06:01 +0000 Subject: [PATCH 2/2] Answer "who was that" from the log, in two layers that are not the same claim Takes a recording's sidecar and reports LOGGED (a fact - the operator wrote the callsign down) separately from ON THE NET (candidates - a check-in says present, not that they were the voice on the tape). Those must never be presented as the same kind of answer. The net name turns out to be in the log already, so this layer needs no API and works retroactively over the whole log: NetLogger-sourced QSOs carry it in COL_COMMENT in the two encodings qsl-queue measured over 29,573 rows. Bracketed text is not always a net, so only net-shaped tokens are taken as one - checked against the [New call sign May 2025] rows that would otherwise be reported as a net name. Matching is on time alone and the band is shown rather than filtered on: filtering would silently drop true matches whenever the sidecar's frequency is unreliable, and a dropped true match is invisible in a way a flagged odd one is not. A QSO on another band in the same window is marked, never hidden. A logged QSO is an instant, not a span - COL_TIME_OFF equals COL_TIME_ON on every row - so the timestamp is when it was logged, usually the end of the exchange. Hence the pad, and hence every match prints its offset into the recording so an edge match reads as one. "Nothing found" is never printed as "nobody". An unlogged QSO and a station heard but not worked look identical from here. Verified against the real log: a 26-minute window over the 07/26 80m net returned three callsigns with their offsets and the net name; the same window with the band claim changed flagged both rows as a different band; a window with no QSOs said nothing was found and said why that is not an absence. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01EqubrbjLXwz1GrVgmTcrf9 --- WIP.md | 37 ++++++ tools/identify_recording.py | 217 ++++++++++++++++++++++++++++++++++++ 2 files changed, 254 insertions(+) create mode 100755 tools/identify_recording.py diff --git a/WIP.md b/WIP.md index 5aba74b..bceb5bd 100644 --- a/WIP.md +++ b/WIP.md @@ -1597,3 +1597,40 @@ command line of the very shell issuing it. Cost a lost commit. Use `pkill -x ham cannot be fetched retroactively — the only source is `netlogger_poll.py`'s `netlogger_checkin` table, and only for the period it has actually been polling. Anything older is unattributable from the net side, permanently. + +### Identification, layer 1 — `tools/identify_recording.py` (09/02/2026) + +Takes a recording's sidecar and answers "who was that" from the log. Two layers that are +deliberately **not** the same kind of claim: **LOGGED** is a fact (the operator wrote the +callsign down); **ON THE NET** is a list of **candidates** (a check-in says present, not that +they were the voice on the tape). + +⚠️ **The net name is ALREADY in the log** and needs no API at all. NetLogger-sourced QSOs carry +it in `COL_COMMENT`, in the two encodings qsl-queue's README measured over 29,573 rows — +`OMISS 40m SSB Net` and `MT/HI [OMISS 40m SSB Net]`. So layer 1 works **retroactively over the +whole log**, unlike anything that depends on the NetLogger API. Bracket text is not always a +net (`[New call sign May 2025]` is in there), so a bracketed token is only taken as one when it +is net-shaped — verified against that exact row. + +⚠️ **Matching is on TIME ALONE, and the band is shown so a wrong match is visible.** Filtering +by band would silently drop true matches whenever the sidecar's frequency is unreliable (rig +disconnected, or a QSY between the exchange and the log entry), and a dropped true match is +invisible in a way a flagged odd one is not. A QSO on another band in the same window is marked +`⚠️ DIFFERENT BAND`, never hidden. Proven by re-running a real window with the band claim +changed and watching both rows flag. + +⚠️ **A logged QSO is an INSTANT, not a span** — `COL_TIME_OFF == COL_TIME_ON` on every row, so +the timestamp is when it was *logged*, usually the end of the exchange. Hence `--pad` (120 s +default) and hence each match prints its offset into the recording, so an edge match reads as +one instead of a bullseye. + +⚠️ **"Nothing found" is never printed as "nobody".** An unlogged QSO and a station heard but +not worked look identical to this tool. A false negative stated as a finding is worse than no +answer. + +Measured against the real log, not a fixture: a 26-minute window over the 07/26 80m net +returned **N4GTO / N4TTU / W4ETA**, each with its offset into the recording and the net name +parsed from the comment. + +⚠️ **Connection details come from the environment** (`WAVELOG_DB_*`, or `WAVELOG_DB_DOCKER`), +never from the file — same rule as the rest of this repo. diff --git a/tools/identify_recording.py b/tools/identify_recording.py new file mode 100755 index 0000000..0558d11 --- /dev/null +++ b/tools/identify_recording.py @@ -0,0 +1,217 @@ +#!/usr/bin/env python3 +"""Who was on a recording? + +Takes the .json sidecar a recording writes and answers it from the log, in two +layers that are NOT the same kind of claim: + + LOGGED a QSO in the log whose time falls inside the recording. The + callsign is a fact - the operator wrote it down. + ON THE NET everyone checked in to the net that was running. These are + CANDIDATES. A check-in says somebody was present, not that they + were the voice on the tape. + +⚠️ A LOOKUP FINDS MATCHES. IT NEVER PROVES AN ABSENCE. "Nothing logged in this +window" is not "nobody was there" - it is far more often a QSO that was never +logged, or a station heard and not worked. This tool says "nothing found" and +never "nobody", because a false negative printed as a finding is worse than no +answer at all. + +⚠️ A LOGGED QSO IS AN INSTANT, NOT A SPAN. COL_TIME_OFF equals COL_TIME_ON on +every row in this log, so the timestamp is when the operator logged it - usually +the end of the exchange, sometimes minutes after. Hence --pad, and hence the +window is matched generously and reported with the offset shown, so a match at +the edge is visible as one rather than presented as a bullseye. + +Connection comes from the environment, never from this file - the repo is not +where station details live: + WAVELOG_DB_HOST WAVELOG_DB_USER WAVELOG_DB_PASS WAVELOG_DB_NAME +or WAVELOG_DB_DOCKER= to shell into a local container instead. + +Usage: identify_recording.py [more.json ...] [--pad SECONDS] +""" +import json +import os +import re +import subprocess +import sys +from datetime import datetime, timedelta, timezone + +QSO_TABLE = "TABLE_HRD_CONTACTS_V01" + +# ⚠️ MATCHING IS ON TIME ALONE, ON PURPOSE, AND THE BAND IS SHOWN SO A WRONG +# MATCH IS VISIBLE. Filtering by band would silently drop true matches whenever +# the sidecar's frequency is unreliable - the rig disconnected, or a QSY between +# the exchange and the log entry - and a dropped true match is invisible in a +# way a flagged odd one is not. So every QSO in the window is listed, and one on +# a different band is marked rather than hidden. +BANDS_HZ = [ + ("160m", 1_800_000, 2_000_000), ("80m", 3_500_000, 4_000_000), + ("60m", 5_300_000, 5_450_000), ("40m", 7_000_000, 7_300_000), + ("30m", 10_100_000, 10_150_000), ("20m", 14_000_000, 14_350_000), + ("17m", 18_068_000, 18_168_000), ("15m", 21_000_000, 21_450_000), + ("12m", 24_890_000, 24_990_000), ("10m", 28_000_000, 29_700_000), + ("6m", 50_000_000, 54_000_000), ("2m", 144_000_000, 148_000_000), +] + + +def band_of(freq_hz): + for name, lo, hi in BANDS_HZ: + if lo <= freq_hz <= hi: + return name + return None + +# The net name in a QSO comment, in the two encodings that coexist in this log - +# see qsl-queue's README, measured over 29,573 rows. Bracket text is NOT always +# a net ("[New call sign May 2025]" is in there), so a bracketed token is +# reported as the net only when it looks like one. +NET_BRACKET = re.compile(r"\[([^\]]+)\]\s*$") +NET_SHAPED = re.compile(r"\bnet\b", re.IGNORECASE) + + +def run_sql(sql): + """Returns rows as lists of strings. Tab-separated, no header, NULL as \\N.""" + container = os.environ.get("WAVELOG_DB_DOCKER") + if container: + cmd = ["docker", "exec", container, "sh", "-lc", + 'mariadb -uroot -p"$MYSQL_ROOT_PASSWORD" -N -B ' + + os.environ.get("WAVELOG_DB_NAME", "wavelog") + + " -e " + shell_quote(sql)] + else: + need = ("WAVELOG_DB_HOST", "WAVELOG_DB_USER", "WAVELOG_DB_PASS") + missing = [k for k in need if not os.environ.get(k)] + if missing: + sys.exit("set " + ", ".join(missing) + " (or WAVELOG_DB_DOCKER)") + cmd = ["mariadb", "-h", os.environ["WAVELOG_DB_HOST"], + "-u", os.environ["WAVELOG_DB_USER"], + "-p" + os.environ["WAVELOG_DB_PASS"], "-N", "-B", + os.environ.get("WAVELOG_DB_NAME", "wavelog"), "-e", sql] + out = subprocess.run(cmd, capture_output=True, text=True) + if out.returncode != 0: + sys.exit("database query failed: " + (out.stderr.strip() or "no message")) + return [line.split("\t") for line in out.stdout.splitlines() if line] + + +def shell_quote(s): + return "'" + s.replace("'", "'\\''") + "'" + + +def sql_str(s): + return "'" + str(s).replace("\\", "\\\\").replace("'", "''") + "'" + + +def parse_utc(s): + return datetime.strptime(s, "%Y-%m-%dT%H:%M:%SZ").replace(tzinfo=timezone.utc) + + +def net_from_comment(comment): + if not comment or comment == "\\N": + return None + m = NET_BRACKET.search(comment) + if m: + return m.group(1).strip() if NET_SHAPED.search(m.group(1)) else None + return comment.strip() if NET_SHAPED.search(comment) else None + + +def netlogger_table_exists(): + rows = run_sql("show tables like 'netlogger_checkin';") + return bool(rows) + + +def identify(path, pad): + with open(path) as fh: + side = json.load(fh) + + start = parse_utc(side["started_utc"]) + end = parse_utc(side["ended_utc"]) + lo = (start - timedelta(seconds=pad)).strftime("%Y-%m-%d %H:%M:%S") + hi = (end + timedelta(seconds=pad)).strftime("%Y-%m-%d %H:%M:%S") + + print(f"\n{side.get('file', os.path.basename(path))}") + print(f" {side['started_utc']} → {side['ended_utc']} UTC" + f" {side.get('freq_hz_start', 0)/1e6:.4f} MHz {side.get('mode', '?')}" + f" ({len(side.get('overs') or [])} overs" + f"{', not tracked' if side.get('overs') is None else ''})") + + if not side.get("rig_connected"): + print(" ⚠️ the rig was not connected when this was recorded - no frequency to match on") + + rows = run_sql( + f"select COL_TIME_ON, COL_CALL, COL_BAND, COL_MODE, COL_FREQ, " + f"coalesce(COL_COMMENT,'') from {QSO_TABLE} " + f"where COL_TIME_ON between {sql_str(lo)} and {sql_str(hi)} " + f"order by COL_TIME_ON;") + + rec_band = band_of(side.get("freq_hz_start") or 0) if side.get("rig_connected") else None + + nets = set() + if rows: + print(f" LOGGED - {len(rows)} QSO(s) in the window (±{pad}s):") + for t, call, band, mode, freq, comment in rows: + when = datetime.strptime(t, "%Y-%m-%d %H:%M:%S").replace(tzinfo=timezone.utc) + if when < start: + off = f"{(start - when).total_seconds():.0f}s before the recording" + elif when > end: + off = f"{(when - end).total_seconds():.0f}s after it ended" + else: + off = f"{(when - start).total_seconds():.0f}s in" + net = net_from_comment(comment) + if net: + nets.add(net) + # A different band in the same window is almost certainly a + # coincidence of time, not this recording. Flagged, never dropped. + odd = " ⚠️ DIFFERENT BAND" if rec_band and band and band != rec_band else "" + print(f" {call:<10} {band:<5} {mode:<5} {int(freq)/1e6:>9.4f} MHz " + f"logged {off}" + (f" net: {net}" if net else "") + odd) + else: + # ⚠️ Wording matters here. See the module docstring. + print(f" LOGGED - nothing found in the window (±{pad}s). That is not evidence " + f"nobody was worked;\n an unlogged QSO and a station heard but " + f"not worked both look like this.") + + if not nets: + print(" ON THE NET - no net named on any matching QSO, so there is nothing to " + "look a roster up by.") + return + + if not netlogger_table_exists(): + print(f" ON THE NET - net(s) {', '.join(sorted(nets))}, but netlogger_checkin does " + f"not exist:\n the poller has never run, so no roster was ever " + f"captured. NetLogger keeps\n only 7 days and has no bulk " + f"history endpoint, so this window cannot be\n recovered later " + f"- only nets from the day the poller starts onward.") + return + + for net in sorted(nets): + rows = run_sql( + f"select c.callsign, coalesce(c.first_name,'') from netlogger_checkin c " + f"join netlogger_net n on n.server=c.server and n.net_id=c.net_id " + f"where n.net_name={sql_str(net)} and c.callsign<>'' " + f"and n.started between date_sub({sql_str(lo)}, interval 12 hour) " + f"and date_add({sql_str(hi)}, interval 12 hour) order by c.callsign;") + if not rows: + print(f" ON THE NET - {net}: no roster stored for this net on this date.") + continue + print(f" ON THE NET - {net}: {len(rows)} checked in. ⚠️ CANDIDATES, not " + f"identifications -\n a check-in says present, not that they " + f"were the voice on the tape.") + calls = [f"{c}{' (' + n + ')' if n and n != chr(92) + 'N' else ''}" for c, n in rows] + for i in range(0, len(calls), 4): + print(" " + " ".join(f"{c:<18}" for c in calls[i:i + 4]).rstrip()) + + +def main(): + args = [a for a in sys.argv[1:]] + pad = 120 + if "--pad" in args: + i = args.index("--pad") + pad = int(args[i + 1]) + del args[i:i + 2] + if not args: + sys.exit(__doc__) + for path in args: + identify(path, pad) + print() + + +if __name__ == "__main__": + main()