Add Central Model demo on SMAP data

[Shadow-AI]

demos -> main

[Veggente]

In the presentation, the anomaly detection results between Ideal and No Sharing are the following:

	Ideal	No Sharing
F1 Score	0.35	0.19
TPR	25.5%	11.8%
FPR	1.5%	0.8%

Note No Sharing has lower FPR so it is not clear if Ideal is better than No Sharing at these operating points.

I wonder if the ROC or PR curves can be compared. Alternatively, we can at least show the F1 Score and TPR for the same FPR for a direct comparison.

[amad-person]

I'm not sure if the ROC curves will be "useful" because the anomaly detection classifier is returning hard labels 🤔

How the classifier returns the hard labels:

• 1 if current true reading of feature9 is outside a predicted range (predicted using the timeseries forecasting model)
• 0 otherwise

[Veggente]

I'm not sure if the ROC curves will be "useful" because the anomaly detection classifier is returning hard labels 🤔

How the classifier returns the hard labels:

* 1 if current true reading of feature9 is outside a predicted range (predicted using the timeseries forecasting model)

* 0 otherwise

There might be ways to predict soft labels or likelihoods based on the readings and the predicted ranges. Then we can use different threshold to sweep out the curves. Or there might be some parameters that directly control the tradeoff between TPR and FPR, which can be used for sweeping.

[amad-person]

Yeah, I was thinking distance(current_reading, predicted_range_max)? Forgot to mention that the classifier returns 1 if the current reading is greater than the max of the predicted range.

[Veggente]

Forgot to mention that the classifier returns 1 if the current reading is greater than the max of the predicted range.

Isn't that a case of "current true reading of feature9 is outside a predicted range"?

[amad-person]

Yes that's right, we don't say the current reading is an anomaly if it is below predicted_range_min. Wanted to clarify in case this would affect how we get the soft labels 😅

[wolfdancer]

We will look at Rockfish-Data/experiments#602 first as the next example, then I'll come back to this to see if we can salvage this for public consumption.