Skip to content

fix(security): harden artifact serialization and loading - #2339

Open
XianBW (XianBW) wants to merge 15 commits into
mainfrom
security/harden-artifact-loading
Open

fix(security): harden artifact serialization and loading#2339
XianBW (XianBW) wants to merge 15 commits into
mainfrom
security/harden-artifact-loading

Conversation

@XianBW

Copy link
Copy Markdown
Collaborator

Summary

  • replace HIST stock-index object NPY metadata with validated JSON
  • replace pandas/numpy module-prefix trust with an explicit restricted-pickle global allowlist
  • make unrestricted MLflow artifact loading opt-in through trusted=True with a warning
  • constrain HighFreqProvider artifact paths to an explicit root and stop mutating input configs

Security impact

This blocks the reported HIST object-deserialization path and the pandas.read_pickle/namespace bypass in RestrictedUnpickler. Arbitrary Python pickle artifacts can still be loaded for compatibility, but only through an explicit trusted path.

Compatibility

  • HIST stock_index must now be JSON; the bundled mapping is migrated without changing its 735 entries.
  • callers loading arbitrary model objects from MLflow must explicitly pass trusted=True. DataFrame, Series, ndarray, and supported Qlib data artifacts continue through restricted loading.
  • HighFreqProvider paths default to the current working directory as artifact_root; callers using absolute paths outside it must set the intended root.

Validation

  • py_compile for changed Python files
  • git diff --check
  • verified migrated HIST mapping exactly matches the previous NPY mapping
  • verified ndarray, Series, and DataFrame restricted-pickle round trips
  • regression tests reject pandas.read_pickle, numpy.load, os.system, subprocess.Popen, and reduce payloads

Full project pytest is delegated to CI because the local base environment does not contain Qlib's complete optional dependency set.

@XianBW XianBW (XianBW) changed the title security: harden artifact serialization and loading fix(security): harden artifact serialization and loading Sep 2, 2026
@XianBW

Copy link
Copy Markdown
Collaborator Author

Local verification update:

  • 23 security regression tests passed
  • handler serialization plus full train/backtest pipeline: 4 passed
  • repository-wide Black, Flake8, and Pylint passed
  • HIST JSON mapping verified identical to all 735 entries in the previous object NPY
  • added explicit trusted=True to the pipeline test that intentionally restores an arbitrary model pickle

The branch also contains minimal CI compatibility fixes for current Node/commitlint, Plotly, MLflow, and Windows/Python 3.8 OSQP behavior.

@XianBW

Copy link
Copy Markdown
Collaborator Author

Final verification: all 76 GitHub checks pass on commit 9d8d63df/latest branch state, including title, CLA, pip compatibility, source builds, notebook execution, full pytest, and slow pytest matrices.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant