Events and metrics¶
A detector emits one label per sample. An operator thinks in outages. Most of the awkwardness in evaluating anomaly detection lives in the gap between those two views, so it is worth being exact about how this library crosses it.
What an event is¶
Events is a sorted, non-overlapping set of closed intervals \([a, b]\) in
UTC nanoseconds. Closed at both ends means an instant is representable as
\([t, t]\), and it fixes the duration as
The \(+1\) looks like an off-by-one and is not. Integer nanosecond bounds are inclusive, so \([a, b]\) occupies the continuous half-open span \([a, b+1)\): an instant lasts one nanosecond, the finest representable moment, rather than zero time. The payoff is that an event covering \(k\) consecutive samples spaced \(\Delta\) apart lasts exactly \(k\Delta\) — the duration ratio and the sample count agree, so an event-based metric and a point-based one measure the same thing.
Two intervals whose gap is exactly 1 ns are fused, for the same reason: a 1 ns gap is an artefact of inclusive endpoints, not a moment of normality. Overlapping and nested intervals collapse too, so the invariants always hold — sorted, disjoint, non-touching. Union and intersection are boundary sweeps that re-normalise through the same path.
Labels to intervals and back¶
to_events reads a label series as intervals. A label counts as anomalous when
it clips to exactly 1 — so NaN is not anomalous, and neither is 0.5.
Unknown never invents an event.
What each flagged sample becomes depends on whether the axis is regular:
Period semantics is what you want, and what you get by default when the sampling interval \(\Delta\) can be inferred: a flagged sample stands for the interval it opens. Six consecutive hourly flags then become one six-hour event rather than six one-hour ones, because consecutive intervals land exactly 1 ns apart and fuse. Under instant semantics the same six flags stay six events of 1 ns each.
\(\Delta\) is inferred only from an axis of at least three samples with a constant
step; otherwise it is None and instant semantics apply silently. as_periods=
overrides the inference in either direction, and as_periods=True on an
irregular axis raises rather than guessing.
to_labels goes back, marking every sample whose own period overlaps an event.
Bounds falling between samples are snapped outwards to the samples they
touch, so the round trip is lossless for events that to_events could have
produced on that axis, and lossy — necessarily — for arbitrary bounds.
expand_events(events, before=, after=) widens each interval and re-merges,
which is how a detection tolerance is expressed. A bare integer margin is
nanoseconds; "1h", timedelta and np.timedelta64 all work and are
clearer.
Point-based metrics¶
Pass label series and the four metrics count samples. With \(T\) and \(P\) the sets of true and predicted anomalous samples:
An empty denominator gives NaN, not zero — no true anomalies means recall is
undefined, and reporting 0.0 would let it be averaged into a summary as if it
were a failure.
Note the consequence of NaN labels reading as not-anomalous: an unknown region
never invents a detection, but it never counts as a correct negative either. It
simply does not appear in \(T\) or \(P\).
Event-based metrics¶
Pass Events and the same functions switch to intervals. An event counts as
detected when the fraction of its duration covered by the other set reaches
thresh:
with \(E\) the true events and \(F\) the predicted ones. The criterion is applied to
the event being scored — to true events for recall, to predicted events for
precision — never to both at once. \(\theta\) defaults to 0.5 and must lie in
\((0, 1]\); since it is strictly positive and every event lasts at least 1 ns, an
uncovered event can never pass.
IoU is the one metric with the same definition in both modes, which makes it the one to compare across them.
This is usually the mode that matters. An alert firing part-way through a six-hour outage is one page and one investigation — a hit, not a fraction of one. Conversely a detector that fires for one minute in the middle of that outage scores full recall on the event and would score \(1/360\) on the samples. Which of those is the honest number depends on what happens next: if the alert wakes someone up, event-based is right; if the labels feed a downstream aggregate, point-based is.
f1_score takes two thresholds, recall_thresh and precision_thresh,
because the two questions are genuinely different. How much of an outage must be
covered before you call it caught, and how much of an alert must be real before
you call it justified, need not be the same number.
The failure mode to watch for is comparing shapes that are not comparable. A
change-point detector produces short intervals around transitions; scored
against interval-shaped ground truth it will show near-zero recall while working
perfectly. The guide works through the
example;
the fix is to reduce the truth to change points and allow a tolerance with
expand_events.
How late the alert was¶
Precision and recall answer whether, never when. A detector that finds every
outage six hours in scores a perfect 1.0 on all four metrics above, and is
useless. detection_delays fills that gap with one number per true event: with
\(E_i = [a_i, b_i]\) the \(i\)-th true event and \(P\) the predicted set,
in nanoseconds, the library's internal time unit. Read one back as a duration
with np.timedelta64(int(d), "ns").
Three decisions in that definition are worth defending.
An undetected event is NaN, not zero and not its own duration. Zero would
claim an instant response to something nobody ever saw. The duration would be an
invented number that grows with how long the outage happened to run. A missed
event has no delay at all, and recall is the metric that counts it.
Firing early clamps to zero, and cannot go negative. The delay is measured from the first moment the prediction and the event coincide — the start of \(E_i \cap P\), which by construction is never before \(a_i\). An alert that opened an hour before the outage was not an hour early to it; it was a false positive that happened to run into one, and precision is where that shows up. Allowing \(-1\,\text{h}\) would let early noise cancel late detections in the mean, and a mean of signed delays is not a number anyone can act on.
Only detected events are reduced. detection_delay takes the mean,
median or max over the non-NaN entries, and is itself NaN when nothing
was detected at all.
A delay is only meaningful next to a recall. Dropping the misses is what
makes the statistic interpretable, and it is also what makes it gameable. A
detector that catches one outage out of fifty, quickly, posts a better delay than
one that catches all fifty a little more slowly. Quote the pair, always. The same
caveat applies to the max: it is the worst case among the ones you found.
Ranking without a threshold¶
Every metric so far needs labels, so scoring a Scorer with them means picking a
threshold first — and then you are measuring the threshold as much as the score.
average_precision and roc_auc need no threshold. They ask only whether the
anomalous samples are ranked above the normal ones. Both are sample-based.
Take the distinct values of the score in decreasing order as thresholds. At the \(k\)-th, let \(\mathrm{tp}_k\) and \(\mathrm{fp}_k\) count the anomalous and normal samples scoring at least that high, with \(n_P\) anomalous and \(n_N\) normal samples in total:
That is a right-hand rectangle sum, not a trapezoid: the curve is never interpolated between operating points, because the segment between two achievable points is generally not itself achievable and integrating it flatters the score. The area under the ROC curve is computed exactly too, but through midranks rather than by walking the curve. Rank the samples by score in increasing order from 1, giving every member of a tied block the average \(r_i\) of the ranks that block occupies; then
which is the probability that a random anomalous sample outranks a random normal
one, ties counting half. Both are checked against
sklearn.metrics.average_precision_score and roc_auc_score in the test suite,
including on scores that are almost entirely ties.
Ties are the trap. Tied scores carry no information about which sample is worse, so no ordering may be imposed on them. Taking distinct thresholds gives a tied block exactly one point on the PR curve; midranks give it one diagonal step of the ROC and therefore the trapezoid under that diagonal. Sort a tied block arbitrarily instead and a constant score — a scorer that has said nothing at all — can come out anywhere between 0 and 1, depending on how the sort happened to break the tie. Integer-valued and clipped scores tie constantly, so this is not a corner case.
A NaN score is excluded, not ranked last. The sample was never placed,
usually because a rolling window had not filled; ranking it at the bottom would
credit the scorer with a judgement it did not make. Note the asymmetry with
labels, which is deliberate: a NaN truth counts as normal, exactly as it does
for the point-based metrics, because an unknown label still describes a sample
that existed. NaN comes back when the answer is undefined — no positives, no
negatives, or nothing left once the unscored rows are dropped.
Read the two together. The false positive rate has the whole normal class in
its denominator, so on the rare-anomaly data this library is aimed at, a detector
can post an excellent roc_auc while being wrong far more often than it is
right. Average precision has \(\mathrm{tp}_k + \mathrm{fp}_k\) in its denominator
and notices.
There is deliberately no event-based average precision. It would need one score per interval, and there is no defensible way to pick it: the maximum over an interval rewards one lucky sample, the mean punishes a detector that is emphatically right for one minute of a six-hour outage, and the choice changes the ranking rather than just rescaling it. If events are what matter, threshold the score and use the event-based metrics above, where the thresholding decision is at least explicit.
Time-ordered splitting¶
split_train_test builds folds that never train on the future. Shuffled
\(k\)-fold would leak: a model fitted on Wednesday and validated on Tuesday has
seen the answer.
Four layouts, chosen by mode, with \(L\) the fold length:
mode |
Train | Test | Shape |
|---|---|---|---|
| 1 | first \(\theta\) of each block | rest of that block | disjoint consecutive blocks |
| 2 | \([0, \theta u_k)\) | \([\theta u_k, u_k)\) | nested, all starting at the beginning |
| 3 | \([0, c_k)\) | next \(L\) samples | expanding window, fixed test block |
| 4 | \([0, c_k)\) | everything after | expanding window, test to the end |
with \(u_k = (k+1)L\) and \(c_k = (k+1)L\). Modes 1 and 2 divide by n_splits;
modes 3 and 4 divide by n_splits + 1, since the last fold needs data left over
to test on. train_ratio applies to modes 1 and 2 only — in 3 and 4 the split
point is the fold boundary. The final fold absorbs the remainder rather than
dropping it, and every fold is checked to be non-empty on both sides.
Mode 3 is the usual choice for "does this still work as more history accumulates". Mode 1 is the one to reach for when the series is long enough that distant history is not representative.
Combining verdicts¶
An Aggregator reduces several label series to one, and the interesting part is
what it does with NaN. Unknown is a third state, and the combination rules are
Kleene's three-valued logic:
OrAggregator |
AndAggregator |
|
|---|---|---|
any input is 1 |
1 | NaN if any unknown, else 1 |
any input is 0 |
NaN if any unknown, else 0 |
0 |
| all unknown | NaN |
NaN |
The asymmetry is the point. For "or", one confident detection settles it — whatever the others could not say cannot make it less true. For "and", one confident normal settles it. Only where the missing verdict could change the answer does the result become unknown. Anything non-zero and known counts as anomalous, so these compose with scores as well as with labels.
VoteAggregator takes a fraction of the known inputs:
Unknowns abstain rather than voting against — a detector still inside its warm-up
window should not drag the vote down. The comparison is non-strict, so the
default \(\theta = 0.5\) is a simple majority with ties counting as anomalous, and
a row where every input is unknown is NaN rather than a division by zero.
Inputs are outer-joined on the time axis first, so a timestamp missing from one input arrives as unknown, and detectors on different but overlapping axes combine without alignment work.
Combining scores¶
Every aggregator above takes verdicts, which means each input was thresholded
before the ensemble saw it — and the threshold is where the knowledge of degree
was spent. ScoreAggregator combines the scores instead, so a point two scorers
nearly flagged stays distinguishable from one neither came close to flagging, and
the combination is still a ranking rather than a set. That is the reason to prefer
it: a ranking is what an operator works down, and it survives the ensemble only if
nothing binarised first.
The price is that scores from different scorers share no unit, so each input column has to be put on a common scale before the rows are reduced. Ranking is the default. Over the \(n\) observed values of input \(j\),
where \(\mathrm{pos}\) is the position in ascending order and \(T_{tj}\) is the set of values tied with \(x_{tj}\), so ties share their average position. The smallest observed value maps to 0 and the largest to 1, and nothing about the spacing of the original scores is used. That is the point rather than a limitation: an input cannot buy influence by being measured in larger numbers, nor by emitting one enormous outlier, and no distributional assumption is needed to compare it with the others. A column with no variation gives every value the midrank \((n+1)/2\) and hence \(\tilde{x} = 0.5\) throughout, which contributes nothing to any of the reductions — an input with no opinion is silent rather than loud.
The alternative keeps the spacing, as a robust z-score against the column's own median:
with \(m_j\) the column median and \(c\) the constant from Thresholds that puts a MAD on the scale of a standard deviation. A score ten deviations out therefore stays ten times as alarming as one deviation out, which ranking flattens away — but it is unbounded again, so one loud input can still carry a mean. Where \(\mathrm{MAD}_j = 0\) there is no unit to divide by and the column is centred but left unscaled, so a value on the median contributes exactly 0 and the excursions stay finite and ordered rather than becoming an infinity that would swallow every other input.
The reduction across the normalised inputs is a mean, a maximum or a median, and
the choice is the same trade-off the vote makes in discrete form: the mean asks
for consensus and dilutes a lone voice, the maximum loses nothing any input
noticed and nothing any input imagined either, and the median is unmoved by one
input in either direction. Unknowns abstain here as they do in the vote — a row is
reduced over whichever inputs are known, and is NaN only when none of them is.