Skip to content

cli-reference.md

Latest commit

 

History

History
249 lines (183 loc) · 7.48 KB

cli-reference.md

File metadata and controls

249 lines (183 loc) · 7.48 KB
Raw

CLI Reference

The ml4t-data command-line tool provides unified access to data fetching, storage management, validation, and system administration. All commands support --verbose and --quiet flags for output control.

ml4t-data [--verbose | --quiet] <command> [options]

Core Commands

fetch

Fetch financial data from any configured provider.

# Single symbol
ml4t-data fetch -s AAPL --provider yahoo --start 2024-01-01 --end 2024-12-31

# Multiple symbols
ml4t-data fetch -s BTC -s ETH --provider cryptocompare \
    --start 2024-01-01 --end 2024-06-30

# From a symbols file (one symbol per line)
ml4t-data fetch -f symbols.txt --provider yahoo --start 2024-01-01 --end 2024-12-31

# Specific provider and frequency
ml4t-data fetch -s AAPL --provider yahoo --frequency daily \
    --start 2024-01-01 --end 2024-12-31

# Save output to file
ml4t-data fetch -s AAPL --provider yahoo \
    --start 2024-01-01 --end 2024-12-31 -o data.parquet

# Show progress bar for batch fetches
ml4t-data fetch -f sp500.txt --provider yahoo \
    --start 2024-01-01 --end 2024-12-31 --progress
Option Short Description
--symbol -s Symbol(s) to fetch (repeatable)
--symbols-file -f File containing symbols, one per line
--start Start date (YYYY-MM-DD), required
--end End date (YYYY-MM-DD), required
--frequency daily, hourly, or weekly (default: daily)
--provider -p Specific provider to use
--output -o Output file path (.parquet or .csv)
--config -c JSON configuration file
--progress Show progress bar for batch fetches

update

Perform incremental data updates for symbols already in storage.

# Incremental update of an existing dataset
ml4t-data update -s AAPL --storage-path ./data

# Use a longer overlap to revise recent observations
ml4t-data update -s AAPL --lookback-days 30 --storage-path ./data

# Set the date range used when the dataset does not exist yet
ml4t-data update -s AAPL --provider yahoo \
    --start 2020-01-01 --end 2020-12-31 --storage-path ./data

# Use the storage and provider settings from a configuration file
ml4t-data update -s AAPL --config ml4t-data.yaml
Option Description
--symbol, -s Symbol to update (required)
--frequency daily, hourly, or weekly
--asset-class equities, crypto, forex, or futures
--lookback-days Existing-data overlap used for revisions
--fill-gaps, --no-fill-gaps Enable or disable gap filling
--provider, -p Provider to use for fetching
--start, --end Date bounds used only for a first load
--initial-load-days First-load history when --start is omitted
--config Configuration file selecting storage and provider settings
--storage-path Hive storage directory (default: ./data)

validate

Validate data quality and integrity. Checks OHLC relationships, duplicates, and optionally runs anomaly detection.

# Validate a single symbol
ml4t-data validate -s AAPL --storage-path ./data

# Validate all stored symbols
ml4t-data validate --all --storage-path ./data

# Include anomaly detection
ml4t-data validate -s AAPL --anomalies --storage-path ./data

# Filter by severity and save report
ml4t-data validate -s AAPL --anomalies --severity error --save-report --config ml4t-data.yaml
Option Description
--symbol, -s Symbol to validate
--all Validate all symbols in storage
--anomalies Run anomaly detection (return outliers, volume spikes, staleness)
--save-report Save anomaly report to ./anomaly_reports/
--severity Minimum severity: info, warning, error, critical
--frequency, --asset-class Select the canonical dataset key
--config Configuration file selecting Hive or flat storage
--storage-path Hive storage directory (default: ./data)

status

Show system overview and health status of stored datasets.

ml4t-data status --storage-path ./data
ml4t-data status --detailed --stale-days 7 --storage-path ./data
ml4t-data status --config config.yaml --detailed

The --detailed flag shows each dataset's provider, row count, observation date, last update, and health status. Use --config to select a configured Hive or flat store.

export

Export stored data to CSV, JSON, or Parquet.

ml4t-data export -s AAPL -o aapl.csv --format csv --storage-path ./data
ml4t-data export -s AAPL -o aapl.json --format json --config ml4t-data.yaml
ml4t-data export -s BTC --asset-class crypto -o btc.parquet --format parquet

Use --frequency and --asset-class to select the dataset key. Use either --config for configured Hive or flat storage, or --storage-path for Hive storage.

list

List all stored datasets, organized by provider type.

ml4t-data list --config ml4t-data.yaml
ml4t-data list --storage-path ./data

Requires either --config (YAML file with storage.path) or --storage-path.

info

Show detailed information about a specific stored symbol, including a data preview.

ml4t-data info -s AAPL --storage-path ./data
ml4t-data info -s BTC --asset-class crypto --config ml4t-data.yaml

Batch Operations

update-all

Update all datasets defined in a YAML configuration file.

# Update everything
ml4t-data update-all -c ml4t-data.yaml

# Update a specific dataset group
ml4t-data update-all -c ml4t-data.yaml --dataset futures

# Preview what would be updated
ml4t-data update-all -c ml4t-data.yaml --dry-run

The configuration file defines datasets with inline symbols or file references:

storage:
  base_path: ~/ml4t-data
  partition_granularity: year

datasets:
  - name: equities
    provider: yahoo
    symbols: [AAPL, MSFT, GOOGL]
    frequency: daily
    asset_class: equity
    initial_load_days: 3650
  - name: sp500
    provider: yahoo
    symbols_file: sp500.txt
    start_date: "2010-01-01"

Futures Commands

download-futures

Download historical futures data from Databento (requires API key).

ml4t-data download-futures -c futures-config.yaml
ml4t-data download-futures -c futures-config.yaml --dry-run   # cost estimate only
ml4t-data download-futures -c futures-config.yaml -p ES -p NQ  # specific products
ml4t-data download-futures -c futures-config.yaml -j 4          # parallel downloads

update-futures

Update existing futures data with the latest available bars.

download-cot

Download CFTC Commitment of Traders data.

ml4t-data download-cot -p ES -p CL --start-year 2020
ml4t-data download-cot --list-products          # show available product codes
ml4t-data download-cot -c cot-config.yaml -o ~/ml4t-data/cot

System Commands

Command Description
ml4t-data version Show version and Python information
ml4t-data providers List available data providers with API key requirements
ml4t-data config Show current configuration
ml4t-data status --storage-path ./data Check stored dataset health
ml4t-data status --config config.yaml --detailed Show configured dataset status
ml4t-data show-completion bash Print shell completion script

Shell Completion

Enable tab completion for your shell:

# Bash
eval "$(_ML4T_DATA_COMPLETE=bash_source ml4t-data)"

# Zsh
eval "$(_ML4T_DATA_COMPLETE=zsh_source ml4t-data)"

# Fish
eval (env _ML4T_DATA_COMPLETE=fish_source ml4t-data)