Fix CSI300 tutorial using shared EOD disk cache and Sina/Eastmoney feeds

docs/csi300-tutorial.md

@@ -0,0 +1,35 @@
+# CSI300 simulation tutorial
+
+This tutorial demonstrates AlgoQuant's historical-universe and simulation APIs. Its outputs illustrate the workflow and do not establish investment performance.
+
+## Index sources and verified coverage
+
+The demo uses `EastmoneyEODWebDownloader` by default. Select `SinaEODWebDownloader` with `-Dcsi300.indexSource=sina`. Both are Java HTTPS clients provided by the matching AlgoQuant library change. There is no Yahoo index dependency, Python process, cookie or API token in this path.
+
+Before implementation, live requests returned **1,944** CSI300 sessions from **2010-01-04** through **2017-12-29** from each provider, with identical date sets and at most **0.005** index-point close difference. The 2008–2017 Eastmoney response contained 2,434 observations for pre-simulation calibration. Evidence: https://github.com/nmltd/algoquant/actions/runs/39127258230 . The library diagnostic workflow additionally checks the actual Java HTTP clients.
+
+The original demo dates remain 2010-01-01 to 2017-01-01 (exclusive). The separate coverage check includes all of calendar 2017. The demo downloads calibration history starting one year and two weeks before the simulation and validates range endpoints before running any strategy.
+
+Constituent membership and adjusted-price ZIPs still come from `test-data/china/constituent_index_300.csv` and `test-data/china/20070104-20171027/adjust_prices/`. Provider symbols are mapped from the canonical `000300.SS`; the index remains the first product, as required by `IndexAwareOptimizer`. The report benchmark uses these same CSI300 levels. Risk-free rate is explicitly assumed zero. Index close is a price-index level, not a total-return series.
+
+## Run
+
+```sh
+mvn -Dexec.mainClass=tech.nmfin.algoquant.demo.model.portfoliooptimization.csi300.CSI300SimulationDemo compile exec:java
+
+# Alternative index source:
+mvn -Dexec.mainClass=tech.nmfin.algoquant.demo.model.portfoliooptimization.csi300.CSI300SimulationDemo \
+  -Dcsi300.indexSource=sina compile exec:java
+```
+
+Override local constituent paths with `-Dcsi300.constituents=/data/membership.csv` and `-Dcsi300.prices=/data/adjust_prices`. The default performs two simulations: buy-and-hold index and monthly equal-weight constituents. Opt into the original **990 optimizer simulations** with `-Dcsi300.fullGrid=true`. The grid runs sequentially to bound memory and preserve ordering. Reports and a PNG are saved under `log/`; add `-Dcsi300.showCharts=true` to open a window.
+
+The two-argument Java constructor accepts a caller-supplied `SequentialCacheFactory<ListedProduct, StockEOD>` to replay a saved authentic index response or use another feed for reproducible tests.
+
+## Dependency
+
+This tutorial uses `tech.nmfin:algoquant-dist:3.3.2`. The Sina/Eastmoney downloaders and shared disk cache are introduced in AlgoQuant 3.3.2 and are not present in 3.3.1. Publish the 3.3.2 library release before merging this consumer update. See the library release notes and `docs/eod-examples-compatibility.md` in the library repository for validation results and known demo limitations.
+
+### Reusing index downloads
+
+The demo wraps either `EastmoneyEODWebDownloader` or `SinaEODWebDownloader` in the shared `DiskCachedEODFactory`. Completed index histories are stored in `./cache/china-index/eastmoney-unadjusted` or `./cache/china-index/sina-unadjusted` and reused across runs for the same or contained date range. Override the root with `-Dcsi300.indexCache=/path/to/cache`. Wider requests fetch a new history; invalid cache files trigger a download. Empty/failed responses and unfinished current-day histories are not saved. Delete the provider cache directory when you want to pick up historical vendor revisions. The raw `*WebDownloader` classes remain network-only, just like Yahoo's web downloader; Yahoo's disk-first behavior is in its cache factory.

docs/demo-repair-validation.md

@@ -25,6 +25,6 @@ The [targeted run](https://github.com/nmltd/algoquant/actions/runs/39122217560)
 
 Yahoo returned no historical observations for **2010–2017** under either `000300.SS` or `399300.SZ` when checked on 18 October 2026. The repository's `indices/hs300.xlsx` contains constituent membership, not official index prices.
 
-CSI300SimulationDemo now validates the index history before its large research sweep. An empty or unavailable benchmark produces a clear data-prerequisite error, rather than endless retries, a null-pointer exception or an apparently successful flat index account. A genuine index history covering the calibration/simulation period is still required; an equal-weight constituent portfolio is not a substitute for that index.
+CSI300SimulationDemo now uses the new Eastmoney Java downloader by default, with Sina selectable via csi300.indexSource=sina. Both providers were tested before implementation and each returned 1,944 identical CSI300 dates for calendar 2010–2017, with at most 0.005 index-point close difference. The actual Java HTTP clients passed the same live cross-check. The demo validates calibration coverage, uses CSI300 as its report benchmark, and defaults to two baselines. The index-only baseline replays index events only. See [the CSI300 tutorial](csi300-tutorial.md) for source selection, run instructions and the library-publication dependency. Genuine index levels are used; an equal-weight constituent portfolio is not substituted for the official index.
 
 Generated CSVs, logs, charts and plotted-data downloads belong in versioned Releases and the Wiki, not in the source tree. Older releases remain available for audit and reproduction.

pom.xml

@@ -9,7 +9,7 @@
     </properties>
     <repositories><repository><id>github-nm-packages</id><url>https://maven.pkg.github.com/nmltd/maven-packages</url>
         <snapshots><enabled>true</enabled></snapshots></repository></repositories>
-    <dependencies><dependency><groupId>junit</groupId><artifactId>junit</artifactId><version>4.13.2</version><scope>test</scope></dependency><dependency><groupId>com.google.code.gson</groupId><artifactId>gson</artifactId><version>2.13.2</version></dependency><dependency><groupId>tech.nmfin</groupId><artifactId>algoquant-dist</artifactId><version>3.3.0-SNAPSHOT</version></dependency></dependencies>
+    <dependencies><dependency><groupId>junit</groupId><artifactId>junit</artifactId><version>4.13.2</version><scope>test</scope></dependency><dependency><groupId>com.google.code.gson</groupId><artifactId>gson</artifactId><version>2.13.2</version></dependency><dependency><groupId>tech.nmfin</groupId><artifactId>algoquant-dist</artifactId><version>3.3.2</version></dependency></dependencies>
     <build><plugins>
         <plugin><groupId>org.apache.maven.plugins</groupId><artifactId>maven-compiler-plugin</artifactId><version>3.14.1</version></plugin>
         <plugin><groupId>org.apache.maven.plugins</groupId><artifactId>maven-jar-plugin</artifactId><version>3.4.2</version>

src/main/java/tech/nmfin/algoquant/demo/model/portfoliooptimization/csi300/CSI300SimulationDemo.java

@@ -28,6 +28,19 @@
 import dev.nm.nmutils.plot.api.Chart;
 import dev.nm.solver.multivariate.constrained.general.penaltymethod.PenaltyMethodMinimizer;
 import java.io.File;
+import tech.nmfin.algoquant.data.cache.BufferCache;
+import tech.nmfin.algoquant.data.cache.SequentialCacheFactory;
+import tech.nmfin.algoquant.execution.datatype.StockEOD;
+import tech.nmfin.algoquant.execution.datatype.product.ListedProduct;
+import tech.nmfin.algoquant.execution.datatype.product.stock.SimpleStock;
+import tech.nmfin.algoquant.execution.datatype.product.stock.Stock;
+import tech.nmfin.algoquant.model.strategy.rebalance.RebalancePortfolio.ProductChooser;
+import tech.nmfin.algoquant.execution.datatype.product.stock.Exchange;
+import tech.nmfin.algoquant.data.historicaldata.AbstractEODWebDownloader;
+import tech.nmfin.algoquant.data.historicaldata.DiskCachedEODFactory;
+import tech.nmfin.algoquant.data.historicaldata.china.EastmoneyEODWebDownloader;
+import tech.nmfin.algoquant.data.historicaldata.china.SinaEODWebDownloader;
+import tech.nmfin.algoquant.data.historicaldata.china.zqf.stock.csi300.CSI300DataProvider;
 import java.io.IOException;
 import java.time.LocalDateTime;
 import java.time.Period;
@@ -37,7 +50,6 @@
 import org.slf4j.Logger;
 import tech.nmfin.algoquant.data.cache.SequentialCache;
 import tech.nmfin.algoquant.data.cache.TimedEntry;
-import tech.nmfin.algoquant.data.historicaldata.china.zqf.stock.csi300.CSI300DataProvider;
 import tech.nmfin.algoquant.execution.component.chart.ChartUtils;
 import tech.nmfin.algoquant.execution.report.SimulationResultsExporter;
 import tech.nmfin.algoquant.execution.strategy.NamedParam;
@@ -57,7 +69,15 @@
  * <p><b>Using this example.</b>
  * Configures CSI 300 constituents and local adjusted-price files for a portfolio simulation.
  * The constituent CSV and price directory are independent inputs and must cover compatible
- * dates. Paths in main are examples to adapt to your own dataset.
+ * dates. The index is downloaded from Eastmoney by default; select Sina with
+ * {@code -Dcsi300.indexSource=sina}. Optional properties {@code csi300.constituents}
+ * and {@code csi300.prices} override the local constituent-data paths. The index
+ * history is cached on disk in {@code ./cache/china-index} (override with
+ * {@code csi300.indexCache}) and validated for calibration coverage before any simulation begins.
+ * The default runs buy-and-hold and equal-weight baselines only; opt into the
+ * expensive 990-optimizer sweep with {@code -Dcsi300.fullGrid=true}.
+ * Reported risk-free rates are assumed zero. Set {@code csi300.showCharts=true}
+ * to open chart windows; PNG export is automatic.
  * </p>
  *
  * <p><b>Model and publication.</b>
@@ -89,32 +109,55 @@ public static void main(String[] args) throws IOException {
 
     private final LocalDateTimeInterval interval;
     private final CSI300DataProvider dataProvider;
+    private static final SimpleStock INDEX = new SimpleStock("000300.SS", Exchange.SS);
+    private BufferCache<Double> indexHistory;
 
     public CSI300SimulationDemo(LocalDateTimeInterval interval) throws IOException {
+        this(interval, selectIndexSource());
+    }
+
+    private static SequentialCacheFactory<ListedProduct, StockEOD> selectIndexSource() {
+        String sourceName = System.getProperty("csi300.indexSource", "eastmoney");
+        AbstractEODWebDownloader downloader;
+        if ("eastmoney".equalsIgnoreCase(sourceName)) downloader = new EastmoneyEODWebDownloader();
+        else if ("sina".equalsIgnoreCase(sourceName)) downloader = new SinaEODWebDownloader();
+        else throw new IllegalArgumentException("csi300.indexSource must be eastmoney or sina");
+        SequentialCacheFactory<ListedProduct, StockEOD> source = new DiskCachedEODFactory(
+            System.getProperty("csi300.indexCache", "./cache/china-index"), downloader);
+        LOGGER.info("CSI300 index source: {}; report risk-free rate assumed zero", sourceName);
+        return source;
+    }
+
+    /**
+     * Uses a caller-supplied index feed, for reproducible offline tests or another provider.
+     * @param interval simulation dates
+     * @param source daily CSI300 index observations including calibration history
+     * @throws IOException if constituent membership cannot be read
+     */
+    public CSI300SimulationDemo(LocalDateTimeInterval interval,
+        SequentialCacheFactory<ListedProduct, StockEOD> source) throws IOException {
         this.interval = interval;
-        this.dataProvider
-            = new CSI300DataProvider(
-                "./test-data/china/constituent_index_300.csv",
-                //                        "D:\\Data\\ZQF\\Daily\\1991_20180727", // TODO: change this to your own data folder
-                "./test-data/china/20070104-20171027/adjust_prices", // TODO: change this to your own data folder
-                true,
-                LocalDateTimeInterval.of(
-                    interval.start().minusYears(5),
-                    interval.end()),
-                false // offline
-            );
+        this.dataProvider = new CSI300DataProvider(
+            System.getProperty("csi300.constituents", "./test-data/china/constituent_index_300.csv"),
+            System.getProperty("csi300.prices", "./test-data/china/20070104-20171027/adjust_prices"),
+            true,
+            LocalDateTimeInterval.of(interval.start().minusYears(1).minusWeeks(2), interval.end()),
+            false,
+            source
+        );
     }
 
     public int run() throws IOException {
-        // A constituent workbook is not the official index price history.
-        // Validate the benchmark before launching the expensive parameter grid;
-        // never publish an empty-history buy-and-hold account as a CSI300 result.
-        dataProvider.getCache(new tech.nmfin.algoquant.execution.datatype.product.stock.SimpleStock(
-            "000300.SS", tech.nmfin.algoquant.execution.datatype.product.stock.Exchange.SS), interval);
+        validateIndexHistory();
+        LOGGER.info("CSI300 tutorial: full research grid enabled={}",
+            Boolean.getBoolean("csi300.fullGrid"));
         Map<NamedParam, Results> results = new LinkedHashMap<>();
 
         results.putAll(simulateBaseline(interval));
-        results.putAll(simulateAlgorithms(interval));
+        if (Boolean.getBoolean("csi300.fullGrid")) {
+            LOGGER.info("Starting full research grid: 990 optimizer simulations");
+            results.putAll(simulateAlgorithms(interval));
+        }
         LOGGER.info(String.format("simulating %d sets of parameters/strategies", results.size()));
         saveResults(results);
 
@@ -123,14 +166,41 @@ public int run() throws IOException {
 
         Collection<SequentialCache<Double>> set = profitCurves.values();
         for (SequentialCache<Double> cache : set) {
-            LOGGER.info(print(cache, true));
+            LOGGER.info("Profit curve contains {} observations", SequentialCache.size(cache));
         }
 
         plotCurves(profitCurves);
 
         return results.size();
     }
 
+    private void validateIndexHistory() throws IOException {
+        LocalDateTime requiredStart = interval.start().minusYears(1).minusWeeks(2);
+        indexHistory = BufferCache.fromSequentialCache(dataProvider.getCache(INDEX,
+            LocalDateTimeInterval.of(requiredStart, interval.end())));
+        if (indexHistory.size() < 2
+                || indexHistory.firstEntry().time().isAfter(requiredStart.plusDays(14))
+                || indexHistory.lastEntry().time().isBefore(interval.end().minusDays(14))) {
+            throw new IOException("CSI300 index history does not cover calibration/simulation: "
+                + requiredStart + " to " + interval.end());
+        }
+        for (TimedEntry<Double> observation : indexHistory) {
+            if (!Double.isFinite(observation.data()) || observation.data() <= 0) {
+                throw new IOException("Invalid CSI300 index value at " + observation.time());
+            }
+        }
+        LOGGER.info("Loaded {} CSI300 index observations: {} to {}", indexHistory.size(),
+            indexHistory.firstEntry().time(), indexHistory.lastEntry().time());
+    }
+
+    private double indexValueAt(LocalDateTime time) {
+        TimedEntry<Double> observation = indexHistory.get(time);
+        if (observation == null || observation.time().isBefore(time.minusDays(14))) {
+            throw new IllegalStateException("Missing or stale CSI300 index observation at " + time);
+        }
+        return observation.data();
+    }
+
     private Map<NamedParam, Results> simulateBaseline(LocalDateTimeInterval interval) {
         Map<NamedParam, Results> results = new LinkedHashMap<>();
         NamedParam param
@@ -144,10 +214,7 @@ private Map<NamedParam, Results> simulateBaseline(LocalDateTimeInterval interval
                 interval,
                 new NamedParam(param)
                     .set("algorithm", "buy-and-hold"),
-                new IndexAwareOptimizer(
-                    (returns, r, s, i) -> new DenseVector(returns.nCols()), // put 0% capital in stocks
-                    (a, b, c) -> 1. // put 100% capital in the index
-                )
+                (returns, weights, symbols, dates) -> new DenseVector(new double[]{1.0})
             )
         );
 
@@ -167,7 +234,7 @@ private Map<NamedParam, Results> simulateEqual(LocalDateTimeInterval interval) {
 
         Map<NamedParam, Results> results = new ConcurrentHashMap<>();
 
-        params.parallelStream().forEach(param
+        params.stream().forEach(param
             -> results.putAll(simulateEqual(interval, param))
         );
         return results;
@@ -200,7 +267,7 @@ private Map<NamedParam, Results> simulateAlgorithms(
                 .params();
 
         Map<NamedParam, Results> results = new ConcurrentHashMap<>();
-        params.parallelStream().forEach(param -> {
+        params.stream().forEach(param -> {
             results.putAll(simulateAlgorithms(interval, param));
         });
 
@@ -441,17 +508,33 @@ private Map<NamedParam, Results> simulateAlgorithm(
         PortfolioOptimizationAlgorithm algorithm
     ) {
         try {
+            String algoName = param.value("algorithm");
+            boolean indexOnly = "buy-and-hold".equals(algoName);
+            Collection<? extends Stock> products = indexOnly
+                ? Collections.singleton(INDEX) : dataProvider.allProducts(interval);
+            ProductChooser<Stock> chooser = indexOnly
+                ? new ProductChooser<Stock>() {
+                    @Override public Set<Stock> allProducts(LocalDateTimeInterval dates) {
+                        return Collections.singleton(INDEX);
+                    }
+                    @Override public List<Stock> chooseProducts(LocalDateTime time) {
+                        return Collections.singletonList(INDEX);
+                    }
+                } : dataProvider;
             OptimizationAlgorithmComparisonSimulation sim
                 = new OptimizationAlgorithmComparisonSimulation(
-                    dataProvider.allProducts(interval),
-                    dataProvider,
+                    products,
+                    chooser,
                     dataProvider,
                     param.value("rebalance"),
                     param.value("window"),
                     param.value("riskAverse"),
                     false
                 );
-            String algoName = param.value("algorithm");
+            // Use the actual CSI300 benchmark rather than the simulation's SP500/Yahoo defaults.
+            sim.setBenchmarkLookupFactory("CSI300", ignored -> this::indexValueAt);
+            // Explicit tutorial assumption: zero risk-free rate (not a downloaded US bill yield).
+            sim.setRiskFreeRateLookup(time -> 0.0);
             sim.addAlgorithm(algoName, algorithm);
             LOGGER.info("simulating {} for interval {}...", algoName, interval);
             return Collections.singletonMap(
@@ -491,8 +574,12 @@ private void plotCurves(
             new ArrayList<>(curves.values()),
             true
         );
-        chart.show();
-        chart.saveAsPNG(new File("./log/" + title + ".png"), 1200, 800);
+        if (Boolean.getBoolean("csi300.showCharts")) {
+            chart.show();
+        }
+        File output = new File("./log/" + title + ".png");
+        java.nio.file.Files.createDirectories(output.toPath().getParent());
+        chart.saveAsPNG(output, 1200, 800);
     }
 
     private LinkedHashMap<String, SequentialCache<Double>> extractProfitCurves(
@@ -526,3 +613,5 @@ private void saveResults(
     }
 
 }
+
+