[ML] Stabilize uncertain trend forecasts

[valeriy42]

Summary

Fixes #2772, where an uncertain fitted trend could dominate a long forecast and produce implausible curvature, including values outside a percentage metric's natural range.

Mathematical rationale

The trend component returns a polynomial regression prediction f(x) and its covariance-derived forecast variance. Previously, a forecast evaluated the fitted polynomial directly at every future time. This means a small but poorly determined quadratic coefficient is multiplied by x^2; over a long horizon that term can become much larger than the local signal even when the model is highly uncertain about it. The existing prediction-horizon guard does not cover this case: the requested duration can be operationally reasonable while the selected trend is still statistically unreliable to extrapolate.

The fix keeps the model's near-term prediction but attenuates its change from the first forecast point according to the fraction of variance explained by the fitted trend:

w(t) = V_trend / (V_trend + V_extrapolation(t))
y_fixed(t) = f(t0) + w(t) * (f(t) - f(t0))

Here t0 is the first forecast time, V_trend is the component's long-term variance, and V_extrapolation(t) is the additional uncertainty incurred by projecting the fitted polynomial from the regression origin to t. Thus w(t) is close to one when the trend is reliable, preserving the original prediction; it falls toward zero when extrapolation uncertainty dominates, smoothly anchoring the forecast at the boundary instead of allowing an uncertain high-order term to diverge. This uses no new persisted state, settings, metric bounds, or customer-wide runtime cost.

The covariance calculation must use the regression basis. The trend model is parameterized at its regression origin, so the variance of the change f(t) - f(t0) uses the basis delta:

[0, t - t0, t^2 - t0^2]

Using [1, t, t^2] instead incorrectly includes the intercept and computes the variance of an absolute prediction rather than the extrapolated change. This PR corrects that basis.

Behaviour and coverage

• Exact zero extrapolation variance retains the original trend prediction (w = 1); this prevents a regression for deterministic affine trends.
• Invalid or negative numerical variances take the conservative anchored fallback.
• The forecast remains continuous at the first prediction point.
• Added tests cover the restored [ML] Forecasting trend is unexpected given data #2772 state, prefix continuity, affine trends, and exact zero-uncertainty trends.

This targets the intended observability workload: ordinary trend forecasts remain unchanged when their extrapolation is well supported, while temporary changes and weakly identified curvature cannot dominate a long-horizon forecast.

[elasticsearchmachine]

Pinging @elastic/ml-core (Team:ML)

[elasticsearchmachine]

Hi @valeriy42, I've created a changelog YAML for you.

[valeriy42]

old-vs-new-synthetic-forecast-comparison-10x-horizon.pdf

Synthetic examples before and after the change

[tveasey]

I like the strategy. I have a proposal for simplifying the edge case handling, but otherwise LGTM.

[tveasey]

I find this kind of hard to follow. How about the equivalent:

double extrapolationWeight{0.0};
if (extrapolationVariance == 0.0) {
    extrapolationWeight = 1.0;
} else if (extrapolationVariance > 0.0 && std::isfinite(extrapolationVariance) &&
           longTermVariance > 0.0 && std::isfinite(longTermVariance)) {
    // Preserve extrapolation while its uncertainty is comparable to (within 2x)
    // the variation observed in the series.
    extrapolationWeight = std::sqrt(
        std::min(2.0 * longTermVariance / extrapolationVariance, 1.0));
}

[valeriy42]

Applied the flattened exclusive-case form. Zero still maps to weight 1; invalid/negative still fall through to 0. Dropped the post-sqrt isfinite check because the argument is in [0, 1] given the guards.

[elastic-vault-github-plugin-prod]

💔 Some backports could not be created

Status	Branch	Result
❌	8.19	Backport failed because of merge conflicts
✅	9.4
✅	9.5

Manual backport

To create the backport manually run:

backport --pr 3189

Questions ?

Please refer to the Backport tool documentation and see the Github Action logs for details

[elastic-vault-github-plugin-prod]

💔 All backports failed

Status	Branch	Result
❌	8.19	Backport failed because of merge conflicts

Manual backport

To create the backport manually run:

backport --pr 3189

Questions ?

Please refer to the Backport tool documentation and see the Github Action logs for details

[valeriy42]

💚 All backports created successfully

Status	Branch	Result
✅	8.19

Questions ?

Please refer to the Backport tool documentation