Retain cached JSONL history within requested date ranges

docs/adr/60361-retain-cached-jsonl-history-within-date-ranges.md

@@ -0,0 +1,50 @@
+# ADR-60361: Retain Cached JSONL History Within Date Ranges
+
+**Date**: 2026-09-12
+**Status**: Draft
+**Deciders**: gh-aw maintainers
+
+---
+
+### Context
+
+The `gh aw logs --cached-jsonl` flow reuses previously stored JSONL records and appends newly collected runs, but the PR description shows that cached run entries outside a requested `--start-date` / `--end-date` window were previously left in place. This made cached output inconsistent with the date-bounded query the user asked for, even though the command continued to preserve non-run metadata and append new results immediately. The diff adds post-collection filtering in both single-target and multi-target log orchestration, plus regression coverage for inclusive date boundaries and file-permission preservation. The architectural question is how cached JSONL state should behave when users request a bounded date range.
+
+### Decision
+
+We will keep `--cached-jsonl` as an append-first cache format, then atomically filter cached run records to the resolved requested date range after collection completes. We decided to preserve non-run, unknown-schema, and undated records while pruning only dated run entries outside the inclusive bounds, because the PR evidence shows the cache must remain reusable and forward-compatible without returning stale run history for date-scoped requests. We will apply the same lifecycle in both single-target and multi-target log collection paths so date-range semantics stay consistent across command modes.
+
+### Alternatives Considered
+
+#### Alternative 1: Leave all previously cached run records untouched
+
+The existing behavior effectively preserved the entire JSONL history and only appended newly collected results. This was considered because it keeps cache writes simple and maximizes retained history. It was not chosen because the PR description and tests show that users asking for a specific date range should not keep seeing stale cached run entries outside that range.
+
+#### Alternative 2: Rewrite the cache to contain only newly fetched records for the current invocation
+
+Another option would be to discard prior cache contents and rebuild the JSONL file solely from the current run's collected records. This was considered because it guarantees strict alignment with the current query window. It was not chosen because the PR evidence explicitly preserves workflow-list, rate-limit, unknown, and undated records, and the existing cache design values incremental append/reuse rather than full replacement.
+
+#### Alternative 3: Filter results only in memory and keep the on-disk cache broader than the requested range
+
+The command could have filtered the final rendered output while leaving the backing JSONL file unchanged. This was considered because it avoids an extra read/filter/write pass on the cache file. It was not chosen because the reported bug is specifically about cached JSONL state retaining out-of-range run history, which would continue to affect later reuse and make the on-disk cache diverge from requested date-range semantics.
+
+### Consequences
+
+#### Positive
+- Cached JSONL files now match inclusive `--start-date` and `--end-date` expectations for run records, reducing stale results during later reuse.
+- The cache continues to preserve non-run metadata, unknown records, and undated records, maintaining forward compatibility and diagnostic usefulness.
+- Single-target and multi-target logs flows share the same post-collection filtering lifecycle, reducing semantic drift between command modes.
+
+#### Negative
+- Each date-bounded cached run now incurs an additional full-file read and atomic rewrite step after collection, increasing I/O cost for large cache files.
+- Cache lifecycle behavior becomes more complex because correctness depends on append-first collection followed by deferred pruning.
+- Date-range correctness now depends on consistent date parsing and boundary resolution, creating more edge cases around timestamps and date-only end bounds.
+
+#### Neutral
+- The implementation keeps the JSONL file format unchanged and introduces behavior through orchestration and writer helpers rather than a new cache schema.
+- Help text and flag descriptions now document that date-bounded runs prune cached run entries after collection while retaining other record kinds.
+- Regression coverage explicitly checks preserved file permissions and retained non-run record types, clarifying the cache contract for future changes.
+
+---
+
+*ADR created by [adr-writer agent]. Review and finalize before changing status from Draft to Accepted.*

pkg/cli/logs_cached_json.go

@@ -364,6 +364,93 @@ func (w *cachedLogsJSONLWriter) appendRecord(record []byte) error {
 	return nil
 }
 
+func (w *cachedLogsJSONLWriter) filterDateRange(startDate, endDate string) error {
+	if w == nil || (startDate == "" && endDate == "") {
+		return nil
+	}
+	dateRange, err := newCachedLogsJSONLDateRange(startDate, endDate)
+	if err != nil {
+		return err
+	}
+	w.mu.Lock()
+	defer w.mu.Unlock()
+	data, err := os.ReadFile(w.path)
+	if errors.Is(err, os.ErrNotExist) {
+		return nil
+	}
+	if err != nil {
+		return fmt.Errorf("failed to read cached logs JSONL for date filtering: %w", err)
+	}
+	filtered := make([]byte, 0, len(data))
+	for _, line := range bytes.SplitAfter(data, []byte{'\n'}) {
+		trimmed := bytes.TrimSpace(line)
+		if len(trimmed) == 0 {
+			filtered = append(filtered, line...)
+			continue
+		}
+		var record cachedLogsJSONLRecord
+		if err := json.Unmarshal(trimmed, &record); err != nil ||
+			record.Kind != cachedLogsJSONLKindRun ||
+			record.SchemaVersion != cachedLogsJSONLSchemaVersion ||
+			record.Run == nil ||
+			record.Run.CreatedAt.IsZero() {
+			filtered = append(filtered, line...)
+			continue
+		}
+		if !dateRange.includes(record.Run.CreatedAt) {
+			continue
+		}
+		filtered = append(filtered, line...)
+	}
+	if bytes.Equal(data, filtered) {
+		return nil
+	}
+	if err := writeFileAtomically(w.path, filtered); err != nil {
+		return fmt.Errorf("failed to filter cached logs JSONL by date range: %w", err)
+	}
+	return nil
+}
+
+type cachedLogsJSONLDateRange struct {
+	start         time.Time
+	end           time.Time
+	endIsDateOnly bool
+}
+
+func newCachedLogsJSONLDateRange(startDate, endDate string) (cachedLogsJSONLDateRange, error) {
+	resolvedStart, resolvedEnd, err := resolveLogsDateRange(startDate, endDate, time.Now())
+	if err != nil {
+		return cachedLogsJSONLDateRange{}, err
+	}
+	dateRange := cachedLogsJSONLDateRange{endIsDateOnly: len(resolvedEnd) == len("2006-01-02")}
+	if resolvedStart != "" {
+		dateRange.start, err = parseFilterDate(resolvedStart)
+		if err != nil {
+			return cachedLogsJSONLDateRange{}, fmt.Errorf("failed to parse cached logs JSONL start date: %w", err)
+		}
+	}
+	if resolvedEnd != "" {
+		dateRange.end, err = parseFilterDate(resolvedEnd)
+		if err != nil {
+			return cachedLogsJSONLDateRange{}, fmt.Errorf("failed to parse cached logs JSONL end date: %w", err)
+		}
+	}
+	return dateRange, nil
+}
+
+func (r cachedLogsJSONLDateRange) includes(createdAt time.Time) bool {
+	if !r.start.IsZero() && createdAt.Before(r.start) {
+		return false
+	}
+	if r.end.IsZero() {
+		return true
+	}
+	if r.endIsDateOnly {
+		return createdAt.Before(r.end.AddDate(0, 0, 1))
+	}
+	return !createdAt.After(r.end)
+}
+
 func (request cachedWorkflowRunsRequest) key() (string, error) {
 	data, err := json.Marshal(request)
 	if err != nil {

pkg/cli/logs_cached_json_test.go

@@ -8,6 +8,7 @@ import (
 	"encoding/json"
 	"os"
 	"path/filepath"
+	"strings"
 	"sync"
 	"testing"
 	"time"
@@ -267,6 +268,46 @@ func TestPrepareCachedLogsJSONLLoadsOnceAndOnlyAppends(t *testing.T) {
 	assert.Equal(t, 2, bytes.Count(bytes.TrimSpace(data), []byte{'\n'})+1)
 }
 
+func TestCachedLogsJSONLWriterFiltersAppendedContentByDateRange(t *testing.T) {
+	path := filepath.Join(t.TempDir(), "logs.jsonl")
+	oldRun := `{"schema_version":2,"kind":"run","run":{"run_id":1,"created_at":"2026-08-31T23:59:59Z","future_field":"preserved"}}`
+	firstIncludedRun := `{"schema_version":2,"kind":"run","run":{"run_id":2,"created_at":"2026-09-01T00:00:00Z"}}`
+	lastIncludedRun := `{"schema_version":2,"kind":"run","run":{"run_id":3,"created_at":"2026-09-10T23:59:59Z"}}`
+	futureRun := `{"schema_version":2,"kind":"run","run":{"run_id":4,"created_at":"2026-09-11T00:00:00Z"}}`
+	workflowRuns := `{"schema_version":2,"kind":"workflow_runs","request":{"host":"github.com","repository":"github/gh-aw","args":["run","list"]},"payload":[{"databaseId":1}]}`
+	rateLimit := `{"schema_version":2,"kind":"github_api_rate_limit","rate_limit":{"host":"github.com"}}`
+	unknown := `{"schema_version":99,"kind":"future","value":"preserved"}`
+	withoutCreatedAt := `{"schema_version":2,"kind":"run","run":{"run_id":5}}`
+	futureSchemaRun := `{"schema_version":99,"kind":"run","run":{"run_id":7,"created_at":"2026-08-31T00:00:00Z"}}`
+	olderSchemaRun := `{"schema_version":1,"kind":"run","run":{"run_id":8,"created_at":"2026-08-31T00:00:00Z"}}`
+	previous := strings.Join([]string{oldRun, firstIncludedRun, lastIncludedRun, futureRun, workflowRuns, rateLimit, unknown, withoutCreatedAt, futureSchemaRun, olderSchemaRun}, "\n") + "\n"
+	require.NoError(t, os.WriteFile(path, []byte(previous), 0o600))
+	writer := newCachedLogsJSONLWriter(path)
+	appendedAt := time.Date(2026, time.September, 5, 12, 0, 0, 0, time.UTC)
+
+	require.NoError(t, writer.Append(ProcessedRun{Run: WorkflowRun{DatabaseID: 6, CreatedAt: appendedAt}}))
+	require.NoError(t, writer.filterDateRange("2026-09-01", "2026-09-10"))
+
+	data, err := os.ReadFile(path)
+	require.NoError(t, err)
+	content := string(data)
+	assert.NotContains(t, content, `"run_id":1`)
+	assert.Contains(t, content, firstIncludedRun)
+	assert.Contains(t, content, lastIncludedRun)
+	assert.NotContains(t, content, `"run_id":4`)
+	assert.Contains(t, content, workflowRuns)
+	assert.Contains(t, content, rateLimit)
+	assert.Contains(t, content, unknown)
+	assert.Contains(t, content, withoutCreatedAt)
+	assert.Contains(t, content, futureSchemaRun)
+	assert.Contains(t, content, olderSchemaRun)
+	assert.Contains(t, content, `"run_id":6`)
+	assert.Contains(t, content, appendedAt.Format(time.RFC3339))
+	info, err := os.Stat(path)
+	require.NoError(t, err)
+	assert.Equal(t, os.FileMode(0o600), info.Mode().Perm())
+}
+
 func TestCachedLogsJSONLStoresCompleteWorkflowRunsPayload(t *testing.T) {
 	path := filepath.Join(t.TempDir(), "logs.jsonl")
 	writer := newCachedLogsJSONLWriter(path)

pkg/cli/logs_command.go

@@ -152,8 +152,9 @@ Use --artifacts all to download all artifacts, or specify individual sets such a
 --artifacts agent,firewall to fetch only what you need.
 
 Use --cached-jsonl to reuse matching run records without downloading and processing their
-artifacts again. New results are appended immediately as JSON Lines. Aggregate analysis may be approximate when compact
-cached records omit detailed data.
+artifacts again. New results are appended immediately as JSON Lines. When a date range is specified,
+cached run records outside that range are removed after collection; other record types are retained.
+Aggregate analysis may be approximate when compact cached records omit detailed data.
 
 All available artifact sets: %s.
 
@@ -228,6 +229,8 @@ func loadStdinLogsOptions(cmd *cobra.Command) (StdinLogsOptions, error) {
 	}
 	return StdinLogsOptions{
 		OutputDir:         values.OutputDir,
+		StartDate:         values.StartDate,
+		EndDate:           values.EndDate,
 		Engine:            values.Engine,
 		Runtime:           values.Runtime,
 		RepoOverride:      values.RepoOverride,
@@ -634,7 +637,7 @@ func addLogsCommandFlags(logsCmd *cobra.Command, validArtifactSets string) {
 	logsCmd.Flags().String("drain3-weights", "", "Path to existing Drain3 weights JSON used to seed log pattern training")
 	logsCmd.Flags().String("format", "", "Output format: console (decorated tables), tsv (tab-separated), pretty (cross-run report), markdown (cross-run Markdown). Default: compact agent-optimized output")
 	logsCmd.Flags().String("report-file", "", "Write --format markdown output directly to this file path instead of stdout (creates parent directories as needed)")
-	logsCmd.Flags().String("cached-jsonl", "", "Path to cached logs JSONL to reuse for matching runs and append new results immediately")
+	logsCmd.Flags().String("cached-jsonl", "", "Path to cached logs JSONL to reuse, append new results, and retain runs in the requested date range")
 	logsCmd.Flags().Int("last", 0, "Alias for --count/-c: number of recent runs to download")
 	logsCmd.Flags().StringSlice("artifacts", []string{"info"}, "Artifact sets to download (default: info — compact workflow metadata). Use 'all' for everything, or comma-separate sets. Valid sets: "+validArtifactSets)
 	logsCmd.Flags().String("cache-before", "", "(Cache eviction) Evict locally cached run folders for runs before this date, prior to downloading. Accepts deltas like -1d, -1w, -1mo (or explicit day counts like -30d), or an absolute date YYYY-MM-DD. Unlike --start-date, this only clears local cache and does not filter which runs are fetched.")

pkg/cli/logs_multi.go

@@ -140,12 +140,12 @@ func logsTargetContinuationOptions(opts LogsDownloadOptions) continuationOptions
 // DownloadWorkflowLogsForTargets downloads several workflow reports concurrently
 // and renders one combined report. Each target gets an isolated output directory
 // so run IDs from different repositories cannot collide in the local cache.
-func DownloadWorkflowLogsForTargets(
+func DownloadWorkflowLogsForTargets( //nolint:largefunc // Keeps shared collection and final cache filtering in one lifecycle.
 	ctx context.Context,
 	opts LogsDownloadOptions,
 	targets []logsWorkflowTarget,
 	initialErrors []error,
-) error {
+) (err error) {
 	if len(targets) == 0 {
 		return errors.Join(initialErrors...)
 	}
@@ -159,6 +159,9 @@ func DownloadWorkflowLogsForTargets(
 	if err := prepareCachedLogsJSONL(&opts); err != nil {
 		return err
 	}
+	defer func() {
+		err = errors.Join(err, opts.cachedJSONLWriter.filterDateRange(opts.StartDate, opts.EndDate))
+	}()
 	allAPIRateLimits := startGitHubAPIRateLimitReports(activeCtx, logsTargetRateLimitHosts(targets))
 	results := collectLogsTargets(activeCtx, opts, targets)
 	processedRuns, continuations, timeoutReached, countLimitReached, storageLimitReached, allErrors := mergeLogsTargetResults(results, initialErrors)
@@ -296,7 +299,7 @@ type logsTargetSharedState struct {
 // collectSingleLogsTarget runs one workflow target's log collection, recovering
 // from panics and building a resumable continuation when the target is still
 // waiting for a worker slot when the shared deadline or cancellation fires.
-func collectSingleLogsTarget(ctx context.Context, opts LogsDownloadOptions, target logsWorkflowTarget, shared logsTargetSharedState) (targetResult logsTargetResult) {
+func collectSingleLogsTarget(ctx context.Context, opts LogsDownloadOptions, target logsWorkflowTarget, shared logsTargetSharedState) (targetResult logsTargetResult) { //nolint:largefunc // Existing target collection remains centralized.
 	defer func() {
 		if recovered := recover(); recovered != nil {
 			targetResult = logsTargetResult{target: target, err: fmt.Errorf("workflow collector panicked: %v", recovered)}

pkg/cli/logs_orchestrator.go

@@ -297,11 +297,14 @@ func buildContinuationIfNeeded(
 }
 
 // DownloadWorkflowLogs downloads and analyzes workflow logs with metrics
-func DownloadWorkflowLogs(ctx context.Context, opts LogsDownloadOptions) error {
+func DownloadWorkflowLogs(ctx context.Context, opts LogsDownloadOptions) (err error) {
 	logsOrchestratorLog.Printf("Downloading workflow logs: workflow=%q, count=%d, outputDir=%q", opts.WorkflowName, opts.Count, opts.OutputDir)
 	if err := prepareCachedLogsJSONL(&opts); err != nil {
 		return err
 	}
+	defer func() {
+		err = errors.Join(err, opts.cachedJSONLWriter.filterDateRange(opts.StartDate, opts.EndDate))
+	}()
 	apiRateLimit := startGitHubAPIRateLimitReport(ctx, logsRateLimitHost(opts.RepoOverride))
 	result, err := collectWorkflowLogs(ctx, opts)
 	if err != nil {

pkg/cli/logs_orchestrator_stdin.go

@@ -22,7 +22,7 @@ import (
 // DownloadWorkflowLogsFromStdin fetches and processes workflow run logs for runs
 // provided as IDs or URLs, bypassing the GitHub API run-discovery step.
 // This is used when the --stdin flag is passed to the logs command.
-func DownloadWorkflowLogsFromStdin(ctx context.Context, opts StdinLogsOptions) error { //nolint:largefunc // Existing stdin orchestration remains centralized.
+func DownloadWorkflowLogsFromStdin(ctx context.Context, opts StdinLogsOptions) (err error) { //nolint:largefunc // Existing stdin orchestration remains centralized.
 	logsOrchestratorLog.Printf("Starting stdin log download: runs=%d, outputDir=%s", len(opts.RunURLs), opts.OutputDir)
 
 	if err := ValidateArtifactSets(opts.ArtifactSets); err != nil {
@@ -50,6 +50,9 @@ func DownloadWorkflowLogsFromStdin(ctx context.Context, opts StdinLogsOptions) e
 		cachedRuns = nil
 	}
 	cachedJSONLWriter := newCachedLogsJSONLWriter(opts.CachedJSONL)
+	defer func() {
+		err = errors.Join(err, cachedJSONLWriter.filterDateRange(opts.StartDate, opts.EndDate))
+	}()
 
 	if err := ensureLogsGitignore(); err != nil {
 		logsOrchestratorLog.Printf("Failed to ensure logs .gitignore: %v", err)

pkg/cli/logs_orchestrator_types.go

@@ -84,6 +84,8 @@ type workflowLogsResult struct {
 type StdinLogsOptions struct {
 	RunURLs           []string
 	OutputDir         string
+	StartDate         string
+	EndDate           string
 	Engine            string
 	Runtime           string
 	RepoOverride      string

pkg/cli/logs_orchestrator_unit_test.go

@@ -805,3 +805,29 @@ func TestCurrentLogsGuardrailStatusReportsAllBoundaries(t *testing.T) {
 	assert.Positive(t, status.timeoutRemaining)
 	assert.LessOrEqual(t, status.timeoutRemaining, time.Minute)
 }
+
+// TestDownloadWorkflowLogsFromStdinFiltersCachedJSONLByDateRange verifies that
+// --stdin honors --start-date/--end-date by pruning out-of-range cached run
+// records, mirroring the discovery-mode behavior in DownloadWorkflowLogs.
+func TestDownloadWorkflowLogsFromStdinFiltersCachedJSONLByDateRange(t *testing.T) {
+	path := filepath.Join(t.TempDir(), "logs.jsonl")
+	outOfRangeRun := `{"schema_version":2,"kind":"run","run":{"run_id":1,"created_at":"2026-08-31T00:00:00Z"}}`
+	inRangeRun := `{"schema_version":2,"kind":"run","run":{"run_id":2,"created_at":"2026-09-05T00:00:00Z"}}`
+	previous := outOfRangeRun + "\n" + inRangeRun + "\n"
+	require.NoError(t, os.WriteFile(path, []byte(previous), 0o600))
+
+	opts := StdinLogsOptions{
+		OutputDir:   t.TempDir(),
+		CachedJSONL: path,
+		StartDate:   "2026-09-01",
+		EndDate:     "2026-09-10",
+	}
+
+	require.NoError(t, DownloadWorkflowLogsFromStdin(context.Background(), opts))
+
+	data, err := os.ReadFile(path)
+	require.NoError(t, err)
+	content := string(data)
+	assert.NotContains(t, content, `"run_id":1`)
+	assert.Contains(t, content, `"run_id":2`)
+}