Repository navigation
[Q1-P0] Add performance baseline tests for historical queries #15
Description
Problem
We don't know how long queries SHOULD take. The historical timeout bug happened because:
- 1 year query takes 67-85 seconds
- SDK timeout was 30 seconds
- No tests validated this mismatch
Impact
- Can't detect performance regressions
- Can't validate timeout configurations are appropriate
- No baseline for "what's normal"
Solution
Add performance test suite with baseline expectations:
# tests/performance/test_historical_benchmarks.py
@pytest.mark.performance
@pytest.mark.parametrize('days,max_duration', [
(1, 5), # 1 day query should be <5s
(7, 15), # 1 week query should be <15s
(30, 30), # 1 month query should be <30s
(365, 90), # 1 year query should be <90s
])
def test_query_performance_baseline(prod_client, days, max_duration):
"""Verify queries complete within expected time."""
start_date = (datetime.now() - timedelta(days=days)).date()
end_date = datetime.now().date()
start_time = time.time()
response = prod_client.historical.get(
commodity='WTI_USD',
start_date=start_date,
end_date=end_date,
interval='daily'
)
duration = time.time() - start_time
# FAIL if performance degrades
assert duration < max_duration, \
f"{days}-day query took {duration:.2f}s, expected <{max_duration}s"
assert len(response.data) > 0Test Cases
-
Response time baselines
- 1 day: <5s
- 1 week: <15s
- 1 month: <30s
- 1 year: <90s
-
Timeout appropriateness
- Verify calculated timeout > expected response time
- Verify timeout has reasonable margin (1.5x response time)
-
Performance regression detection
- Compare with previous run
- Alert if >20% slower
Implementation
# Store baseline results
tests/performance/
baselines/
historical_response_times.json # Baseline data
test_historical_benchmarks.py # Benchmark tests
conftest.py # Performance fixturesCI Integration
# Run performance tests weekly (not every commit - too slow)
on:
schedule:
- cron: '0 0 * * 0' # Weekly on Sunday
workflow_dispatch: # Manual trigger
jobs:
performance:
runs-on: ubuntu-latest
steps:
- name: Run performance tests
run: pytest tests/performance/ --performance -v
- name: Compare with baseline
run: python scripts/compare_performance.py
- name: Create issue if regression
if: failure()
run: gh issue create --title "Performance Regression Detected"Acceptance Criteria
- Performance test suite created
- Baselines established for all query types
- Tests fail if performance degrades >20%
- Tests run weekly in CI
- Results stored for trend analysis
- Automated alerts on regression
Estimated Effort
Time: 3-4 hours
Complexity: Medium (need baseline data collection)
Success Metrics
- Detect performance regressions before customers notice
- Validate timeout configurations are appropriate
- Track performance trends over time
Related
- Issue Refactor: Extract duplicate retry logic into shared strategy class #1 (Integration tests)
- Historical timeout bug
Activity
coderabbitai commentedon Dec 17, 2025
📝 CodeRabbit Plan Mode
Generate an implementation plan and prompts that you can use with your favorite coding agent.
Examples
🔗 Similar Issues
Related Issues
👤 Suggested Assignees
🧪 Issue enrichment is currently in open beta.
To disable automatic issue enrichment, add the following to your
.coderabbit.yaml:💬 Have feedback or questions? Drop into our discord or schedule a call!
karlwaldman commentedon Jul 13, 2026
Closing as complete — performance baseline tests exist and run weekly:
tests/integration/test_historical_endpoints.py::TestHistoricalPerformanceBaselines— 1-week (<30s), 1-month (<60s), 1-year (<120s) baselines, printing measured durations for monitoring; plusTestHistoricalTimeoutBehaviorcovering custom-timeout behavior (the exact v1.4.1 scenario).github/workflows/weekly-health.ymland locally viascripts/pre-release-validation.shNote: two of these tests were permanently red because they asserted >300/>1000 datapoints against the API's 100-point pagination cap — PR #53 fixes the assertions so the timing baselines (the real regression guards) do the work. Verified green against production locally (2026-07-13).