History(barCount) returns ~17% fewer daily bars for lunch-break markets (HKFE) when called with the clock inside the break

[AlexCatarino]

Expected Behavior

History(symbol, barCount, Resolution.Daily) should return barCount bars whenever that
much history exists, regardless of where in the algorithm it is called from.

Actual Behavior

For a market whose trading day is split by a lunch break (HKFE, and any exchange with two
market segments per day), a bar-count history request issued while the algorithm clock sits
inside that break returns ~17% fewer bars than requested. Same symbol, same request, different
call site:

initialize (algo time 2018-02-01 00:00 New York = 13:00 Asia/Hong_Kong)
  History(HSI continuous, 1300, Daily) -> 1078 bars, first 2013-11-26
on_end_of_algorithm (2018-02-09 19:00 New York = 2018-02-10 08:00 Hong Kong)
  History(HSI continuous, 1300, Daily) -> 1300 bars, first 2013-01-29

The shortfall is a constant ratio, not a boundary effect (same run, called from initialize):

requested 100 -> 82 bars      requested 800  -> 661 bars
requested 200 -> 165 bars     requested 1078 -> 893 bars
requested 400 -> 328 bars     requested 1300 -> 1078 bars

No data is missing: a window request from the same call site,
History(HSI, datetime(2013,1,1), self.time, Resolution.Daily), returns 1312 bars from
2013-01-02, and exactly 1078 of them fall on or after 2013-11-26 - i.e. the request's computed
start time is what is wrong, not the data. ES (CME, no lunch break) returns 1300/1300 from
both call sites, and the extendedMarketHours argument makes no difference.

Potential Solution

Two things combine in Time.GetStartTimeForTradeBars (Common/Time.cs):

1. The loop counts 24-hour windows that contain any market segment, not trading sessions. It
   is only equivalent to a session count while the windows are aligned to the exchange's day
   boundary.
2. The round-down that produces that alignment is skipped here:

if (dailyPreciseEndTime && barSize == OneDay)
{
    if (!exchangeHours.IsDateOpen(current) ||
        exchangeHours.GetNextMarketClose(current.Date, extendedMarketHours) > current)
    {
        current = end.RoundDownInTimeZone(barSize, exchangeHours.TimeZone, dataTimeZone);
    }
}

SecurityExchangeHours.GetNextMarketClose(localDateTime, extendedMarketHours) calls the
overload with lastClose: false, so for HKFE it returns the first close of the day, 12:00,
not the session close at 16:30. With a reference time of 13:00 the condition
12:00 > 13:00 is false, the round-down does not happen, and the windows end up aligned to
13:00 - which splits every HKFE session across two windows and counts six windows per week
instead of five. 5/6 = 0.833, matching the observed 0.829 ratio.

Suggested fix: use the last close of the day for this check (the lastClose: true overload) so
the round-down happens whenever the current session has not finished, and/or count sessions
rather than 24-hour windows for barSize == OneDay.

Reproducing the problem

from AlgorithmImports import *

class ProbeHsiTiming(QCAlgorithm):
    def initialize(self):
        self.set_start_date(2018, 2, 1)
        self.set_end_date(2018, 2, 8)
        self.hsi = self.add_future('HSI', resolution=Resolution.DAILY, market=Market.HKFE,
            data_normalization_mode=DataNormalizationMode.BACKWARDS_RATIO,
            data_mapping_mode=DataMappingMode.LAST_TRADING_DAY, contract_depth_offset=0).symbol
        self.report('initialize')

    def on_end_of_algorithm(self):
        self.report('on_end_of_algorithm')

    def report(self, tag):
        df = self.history(self.hsi, 1300, Resolution.DAILY)
        dates = [idx[-1].date() for idx in df.index]
        self.log(f'{tag}: bars={len(dates)} first={dates[0]} last={dates[-1]}')

Reproduced on LEAN 2.5.0.0.18057. Warm-up code that calls History() from Initialize() is
the common way to hit this, since the algorithm clock is then at midnight in the algorithm's
time zone.

Reported via Intercom conversation 215475691979456.

[abhijeetvichare76]

Hi, I'd like to take this one. I've traced it to the daily precise end time round-down in Time.GetStartTimeForTradeBars calling the GetNextMarketClose overload that returns the first close of the day, so at 13:00 HKT the round-down is skipped and each session straddles two 24h windows. Plan: use the last close of the day for that check, add a TimeTests case with the HKFE hours at 13:00 local and one after the 16:30 close asserting the session count, and run the history and fill-forward callers. Could you assign it to me?