API reference¶
Every public name, grouped by the module it lives in. The five base classes, the
TimeSeries boundary and the window engine are importable straight from
hazure; everything else from the module listed here.
hazure¶
hazure
¶
hazure — finding anomalies in time series without labelled examples.
The case this is built for: you have a metric, you suspect it occasionally misbehaves, and you have no record of when it did. There is nothing to train a classifier on, so the model has to describe what normal looks like and report departures from it.
Everything is built from five composable pieces::
Scorer series -> continuous score .score() .fit_score()
Threshold score -> binary labels .apply() .fit_apply()
Detector a Scorer and Threshold paired .detect() .fit_detect()
Aggregator several label series -> one .aggregate()
Transformer series -> series .transform() .fit_transform()
Asking "how unusual is this point" and asking "is that unusual enough to report" are different questions, so they are separate types. One threshold policy is then reusable across every scorer, a scorer can be swapped without revisiting the policy, and a score is useful on its own for ranking rather than flagging.
Any pandas, polars or pyarrow object with a time axis is accepted, and results come back in the flavour they went in as::
>>> import numpy as np, pandas as pd
>>> from hazure import SpikeDetector
>>> index = pd.date_range("2024-01-01", periods=200, freq="h")
>>> values = np.zeros(200)
>>> values[120] = 9.0
>>> flags = SpikeDetector(window=24).fit_detect(pd.Series(values, index=index))
>>> bool(flags.idxmax() == index[120])
True
Labels are 1.0 anomalous, 0.0 normal and NaN unknown — a point whose
score could not be computed is not quietly called normal.
Runtime dependencies are narwhals and numpy. SciPy, scikit-learn,
statsmodels, matplotlib, stumpy and ruptures are extras, imported only by the
components that need them.
Every public name is importable straight from this package, apart from the sample
data in :mod:hazure.datasets — a convenience for trying things out rather than
part of the detection API. Names are also grouped by subject, if you prefer to
import from there:
hazure.detection
Ready-made detectors, each a scorer paired with a threshold.
hazure.scoring
Continuous scores, for ranking or for pairing with your own threshold.
hazure.thresholds
Turning a score into labels.
hazure.features
Feature engineering: rolling aggregates, lags, decomposition, projections.
hazure.ensemble
Combining several verdicts into one.
hazure.compose
:class:Pipeline for a chain, :class:Graph for anything branching.
hazure.events
Moving between per-sample labels and anomalous intervals.
hazure.evaluation
Metrics — how much was caught, how late, how well ranked — and time-ordered
folds to compute them over.
hazure.methods
Further method families: spectral residual, Hampel filtering, change-point
segmentation, matrix profile discords, STL residuals.
hazure.streaming
:class:Stream, for driving a fitted component one observation at a time.
hazure.calibration
Choosing where to draw the line: from labelled events, or from an alert
budget.
hazure.datasets
Series to try things on — synthetic, or a benchmark fetched on demand.
hazure.plotting
One :func:~hazure.plotting.plot function, for looking at the result.
TimeSeries
dataclass
¶
TimeSeries(
time: NDArray[int64],
values: NDArray[float64],
columns: tuple[str, ...],
freq: int | None,
origin: Origin,
)
A time-indexed block of floating point data, backend-independent.
Instances are immutable; every operation returns a new object. Construct
them with :meth:from_any rather than calling TimeSeries(...) directly,
unless you already hold validated arrays.
Attributes:
| Name | Type | Description |
|---|---|---|
time |
NDArray[int64]
|
UTC nanoseconds since the epoch, strictly increasing. |
values |
NDArray[float64]
|
Shape |
columns |
tuple[str, ...]
|
Column names, unique and in the same order as |
freq |
int | None
|
Nanoseconds between consecutive samples, or None when the series is irregular or too short to tell. Calendar-based frequencies such as "month start" are irregular in nanoseconds and so report None. |
origin |
Origin
|
Provenance used to rebuild the caller's native type. |
from_any
classmethod
¶
from_any(
data: Any,
*,
time: str | None = None,
sort: bool = True,
drop_duplicates: bool = True,
) -> TimeSeries
Build a TimeSeries from any supported native object.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
data
|
Any
|
A pandas Series/DataFrame with a |
required |
time
|
str | None
|
Name of the column holding timestamps. Required only when a frame has more than one temporal column; otherwise it is detected. For pandas input this selects a column instead of the index. |
None
|
sort
|
bool
|
Sort by time when the input is not already ordered. |
True
|
drop_duplicates
|
bool
|
Keep the first observation at each timestamp. When False, duplicate timestamps raise instead. |
True
|
Returns:
| Type | Description |
|---|---|
TimeSeries
|
Validated series with a strictly increasing time axis. |
Raises:
| Type | Description |
|---|---|
TypeError
|
The object is not a recognised dataframe, or its time axis is not temporal. |
ValueError
|
Column names are duplicated, timestamps are missing, or duplicate
timestamps were found with |
Source code in src/hazure/_core/series.py
from_arrays
classmethod
¶
from_arrays(
time: NDArray[Any],
values: NDArray[Any],
columns: Sequence[str] | None = None,
*,
origin: Origin | None = None,
sort: bool = True,
drop_duplicates: bool = True,
) -> TimeSeries
Build a TimeSeries from raw numpy arrays.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
time
|
NDArray[Any]
|
|
required |
values
|
NDArray[Any]
|
1-D (one column) or 2-D |
required |
columns
|
Sequence[str] | None
|
Column names. Defaults to |
None
|
origin
|
Origin | None
|
Provenance for :meth: |
None
|
sort
|
bool
|
Sort by time when not already ordered. |
True
|
drop_duplicates
|
bool
|
Keep the first observation at each timestamp. |
True
|
Returns:
| Type | Description |
|---|---|
TimeSeries
|
Validated series. |
Source code in src/hazure/_core/series.py
wrap
¶
wrap(
values: NDArray[Any],
columns: Sequence[str] | None = None,
) -> TimeSeries
Return a new series on this time axis carrying values.
This is how scorers and transformers emit results: the time axis, freq and provenance are inherited, so only the numbers need supplying.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
values
|
NDArray[Any]
|
1-D or 2-D array with |
required |
columns
|
Sequence[str] | None
|
Names for the new columns. Defaults to this series' names when the
width is unchanged, otherwise |
None
|
Returns:
| Type | Description |
|---|---|
TimeSeries
|
Series sharing this time axis. |
Source code in src/hazure/_core/series.py
select
¶
select(columns: Sequence[str] | str) -> TimeSeries
Return a series restricted to columns, in the order given.
Provenance is carried through unchanged. Narrowing to one column does not turn a frame the caller passed into a series they did not: whether the result is emitted as a series depends on what arrived, not on how many columns happen to be left mid-computation.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
columns
|
Sequence[str] | str
|
One name, or a sequence of names. |
required |
Returns:
| Type | Description |
|---|---|
TimeSeries
|
Series carrying only the requested columns. |
Raises:
| Type | Description |
|---|---|
KeyError
|
A requested name is not present. |
Source code in src/hazure/_core/series.py
iter_columns
¶
iter_columns() -> Iterator[TimeSeries]
Yield each column as its own univariate TimeSeries.
Yields:
| Type | Description |
|---|---|
TimeSeries
|
One single-column series per column, in order. |
Source code in src/hazure/_core/series.py
join
¶
join(*others: TimeSeries) -> TimeSeries
Outer-join other series onto this one along the time axis.
This replaces pandas' implicit index alignment with something explicit and backend-independent: the union of all time axes, with NaN where a series had no observation.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
*others
|
TimeSeries
|
Series to merge in. Their columns are appended in order. |
()
|
Returns:
| Type | Description |
|---|---|
TimeSeries
|
Combined series over the union of every time axis. |
Raises:
| Type | Description |
|---|---|
ValueError
|
Two inputs contribute the same column name. |
Source code in src/hazure/_core/series.py
to_native
¶
Rebuild the caller's native object.
pandas input comes back with its DatetimeIndex restored, including
time zone and resolution. polars and pyarrow input come back with the
time column in its original position and name. A single-column series
that arrived as a pandas Series leaves as a pandas Series.
A series built by :meth:from_arrays has no native counterpart, so it
returns itself. Building a dataframe would mean importing a library the
caller never asked for — numpy is the only dependency such a series has,
and emitting it should not add one. Pass backend to opt in.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
backend
|
str | None
|
Emit into this backend instead of the original one, e.g.
|
None
|
Returns:
| Type | Description |
|---|---|
Any
|
A native dataframe or series, or this |
Source code in src/hazure/_core/series.py
to_numpy
¶
column_values
¶
Return one column as a 1-D float64 array.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
name
|
str
|
Column name. |
required |
Returns:
| Type | Description |
|---|---|
ndarray
|
The column's values. |
Raises:
| Type | Description |
|---|---|
KeyError
|
The name is not present. |
Source code in src/hazure/_core/series.py
Component
¶
Bases: Configurable, ABC
Shared machinery for scorers, thresholds, transformers and detectors.
Subclasses implement :meth:_compute, and :meth:_learn if they need
training. Everything else — accepting any backend, validating the time axis,
fanning out across columns, checking that training happened — is handled
here.
Attributes:
| Name | Type | Description |
|---|---|---|
multivariate |
bool
|
True when the algorithm needs every column at once, as PCA reconstruction does. False means the algorithm is per-series, and a multi-column input is handled by fanning out. |
trainable |
bool
|
False for algorithms with nothing to learn, such as a fixed threshold.
Those may be used without calling :meth: |
feature_names
property
¶
Columns seen during :meth:fit, or None before fitting.
fit
¶
Train on data and return self, for chaining.
A univariate component given a frame of k columns trains k independent copies of itself, one per column, so each learns its own normal range.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
data
|
Any
|
Any supported dataframe, series, or :class: |
required |
Returns:
| Type | Description |
|---|---|
Component
|
This component. |
Source code in src/hazure/_core/component.py
run
¶
run(ts: TimeSeries) -> TimeSeries
Apply this component to a :class:TimeSeries, returning one.
This is the composition entry point: pipelines and compound detectors
chain components through run so intermediate results never make a
round trip through a native dataframe.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
ts
|
TimeSeries
|
Input series. |
required |
Returns:
| Type | Description |
|---|---|
TimeSeries
|
The component's output. |
Raises:
| Type | Description |
|---|---|
RuntimeError
|
The component needs training and has not been trained. |
ValueError
|
The input is missing a column that training used, or a multivariate component was handed different columns than it learned from. |
Source code in src/hazure/_core/component.py
BaseScorer
¶
Bases: Component
Turns a series into a continuous anomaly score.
A score is "how unusual is this point", on whatever scale the algorithm
works in. Higher means more unusual. Scores are useful on their own for
ranking, and become labels when passed through a
:class:BaseThreshold.
score
¶
Score data.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
data
|
Any
|
Any supported dataframe, series, or :class: |
required |
Returns:
| Type | Description |
|---|---|
Any
|
Continuous scores, in the same flavour as the input. |
Source code in src/hazure/_core/component.py
fit_score
¶
Fit on data and score it in one step.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
data
|
Any
|
Any supported dataframe, series, or :class: |
required |
Returns:
| Type | Description |
|---|---|
Any
|
Continuous scores, in the same flavour as the input. |
Source code in src/hazure/_core/component.py
BaseThreshold
¶
Bases: Component
Turns continuous scores into binary labels.
Kept separate from scoring so one policy — a quantile, an inter-quartile range, a fixed cut-off — can be reused across every scorer, and swapped without touching the scorer.
apply
¶
Label scores.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
scores
|
Any
|
Continuous scores, as produced by a :class: |
required |
Returns:
| Type | Description |
|---|---|
Any
|
Labels as 1.0 for anomalous, 0.0 for normal and NaN for unknown, in the same flavour as the input. |
Source code in src/hazure/_core/component.py
fit_apply
¶
Fit on scores and label them in one step.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
scores
|
Any
|
Continuous scores, as produced by a :class: |
required |
Returns:
| Type | Description |
|---|---|
Any
|
Labels, in the same flavour as the input. |
Source code in src/hazure/_core/component.py
BaseDetector
¶
Bases: Component
Turns a series directly into binary anomaly labels.
Most detectors are a :class:BaseScorer paired with a
:class:BaseThreshold; this is the type that pairing presents itself as, so
that the common case stays a single object with familiar parameters.
detect
¶
Detect anomalies in data.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
data
|
Any
|
Any supported dataframe, series, or :class: |
required |
Returns:
| Type | Description |
|---|---|
Any
|
Labels as 1.0 for anomalous, 0.0 for normal and NaN for unknown, in the same flavour as the input. |
Source code in src/hazure/_core/component.py
fit_detect
¶
Fit on data and detect anomalies in it in one step.
This is the usual entry point for unsupervised use, where the same series both defines "normal" and is searched for departures from it.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
data
|
Any
|
Any supported dataframe, series, or :class: |
required |
Returns:
| Type | Description |
|---|---|
Any
|
Labels, in the same flavour as the input. |
Source code in src/hazure/_core/component.py
BaseTransformer
¶
Bases: Component
Turns a series into another series.
Feature engineering: rolling aggregates, lagging, seasonal decomposition. Transformers sit upstream of scorers in a pipeline.
transform
¶
Transform data.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
data
|
Any
|
Any supported dataframe, series, or :class: |
required |
Returns:
| Type | Description |
|---|---|
Any
|
The transformed series, in the same flavour as the input. |
Source code in src/hazure/_core/component.py
fit_transform
¶
Fit on data and transform it in one step.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
data
|
Any
|
Any supported dataframe, series, or :class: |
required |
Returns:
| Type | Description |
|---|---|
Any
|
The transformed series, in the same flavour as the input. |
Source code in src/hazure/_core/component.py
BaseAggregator
¶
Bases: Configurable, ABC
Combines several label series into one.
Aggregators sit outside the fit/apply hierarchy because they have nothing to learn and no single input: they take the outputs of several detectors and reduce them, which is how ensembling and multi-condition rules are built.
aggregate
¶
Combine label series.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
*label_sets
|
Any
|
Two or more label series, or a single frame whose columns are the series to combine. |
()
|
names
|
Iterable[str] | None
|
Names for the inputs, used to disambiguate identically named series.
Defaults to |
None
|
Returns:
| Type | Description |
|---|---|
Any
|
A single label series, in the same flavour as the first input. |
Raises:
| Type | Description |
|---|---|
ValueError
|
Nothing was passed, or a single input has only one column. |
Source code in src/hazure/_core/component.py
rolling
¶
rolling(
values: NDArray[float64],
window: Window,
agg: str = "mean",
*,
time: NDArray[int64] | None = None,
center: bool = False,
min_periods: int | None = None,
closed: Closed | None = None,
q: float | None = None,
) -> NDArray[float64]
Roll a window over a series and aggregate it.
Mirrors pandas.Series.rolling(...).agg(), extended with statistics
pandas exposes only via apply (nnz, nunique, iqr, idr)
and available for duration windows on irregular data.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
values
|
NDArray[float64]
|
1-D float array. NaN marks a missing observation and is skipped. |
required |
window
|
Window
|
Observations ( |
required |
agg
|
str
|
A name from :data: |
'mean'
|
time
|
NDArray[int64] | None
|
UTC nanoseconds. Required for duration windows. |
None
|
center
|
bool
|
Centre the window instead of trailing it. |
False
|
min_periods
|
int | None
|
Minimum non-missing observations for a result. Defaults to the full window for integer windows and to 1 for duration windows, as in pandas. |
None
|
closed
|
Closed | None
|
Which endpoints to include; defaults to |
None
|
q
|
float | None
|
Quantile in |
None
|
Returns:
| Type | Description |
|---|---|
ndarray
|
Float array the same length as |
Notes
Unlike pandas, agg="count" respects min_periods: pandas returns the
observation count even when it falls below min_periods, which makes
count behave unlike every other statistic.
Examples:
>>> import numpy as np
>>> rolling(np.array([1.0, 2.0, 3.0, 4.0]), 2, "mean")
array([nan, 1.5, 2.5, 3.5])
Source code in src/hazure/_core/window.py
double_rolling
¶
double_rolling(
values: NDArray[float64],
window: Window | tuple[Window, Window],
agg: str | tuple[str, str] = "median",
*,
time: NDArray[int64] | None = None,
diff: Literal[
"l1", "l2", "diff", "rel_diff", "abs_rel_diff"
] = "l1",
min_periods: int
| tuple[int | None, int | None]
| None = None,
q: float | None = None,
) -> NDArray[float64]
Compare the window before each point with the window after it.
This is the primitive behind spike and level-shift detection: aggregate the recent past and the near future separately, then measure how far apart they are. A large gap means the series changed character at that point.
The left window covers observations strictly before each row; the right window covers the row itself and what follows. Both are expressed directly as bounds, so unlike pandas there is no need to reverse the series or build a mirrored index to get a forward-looking window.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
values
|
NDArray[float64]
|
1-D float array. |
required |
window
|
Window | tuple[Window, Window]
|
One spec for both sides, or |
required |
agg
|
str | tuple[str, str]
|
One statistic for both sides, or |
'median'
|
time
|
NDArray[int64] | None
|
UTC nanoseconds. Required for duration windows. |
None
|
diff
|
Literal['l1', 'l2', 'diff', 'rel_diff', 'abs_rel_diff']
|
How to compare the two sides. |
'l1'
|
min_periods
|
int | tuple[int | None, int | None] | None
|
One value for both sides, or |
None
|
q
|
float | None
|
Quantile for |
None
|
Returns:
| Type | Description |
|---|---|
ndarray
|
Float array the same length as |
Raises:
| Type | Description |
|---|---|
ValueError
|
|
Examples:
A step change shows up as a spike in the output:
>>> import numpy as np
>>> series = np.array([0.0, 0.0, 0.0, 5.0, 5.0, 5.0])
>>> double_rolling(series, 2, "mean", diff="diff")
array([nan, nan, 2.5, 5. , 2.5, nan])
Source code in src/hazure/_core/window.py
614 615 616 617 618 619 620 621 622 623 624 625 626 627 628 629 630 631 632 633 634 635 636 637 638 639 640 641 642 643 644 645 646 647 648 649 650 651 652 653 654 655 656 657 658 659 660 661 662 663 664 665 666 667 668 669 670 671 672 673 674 675 676 677 678 679 680 681 682 683 684 685 686 687 688 689 690 691 692 693 694 695 696 697 698 699 700 701 702 703 704 705 706 707 708 709 710 711 712 713 714 715 716 717 718 719 720 721 722 723 724 725 726 727 728 729 730 731 732 733 | |
parse_duration
¶
Convert a duration to nanoseconds.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
spec
|
str | timedelta64 | timedelta
|
A string such as |
required |
Returns:
| Type | Description |
|---|---|
int
|
Length of the duration in nanoseconds. |
Raises:
| Type | Description |
|---|---|
ValueError
|
The string is unparseable, names a calendar unit, or is not positive. |
Examples:
Source code in src/hazure/_core/window.py
hazure.detection¶
Ready-made detectors: a series in, binary labels out.
detection
¶
Detection: series in, binary labels out.
A detector is a scorer and a threshold in one object, which is how anomaly
detection is usually wanted: fit_detect on a series, labels back. The parts
remain visible as .scorer and .threshold, so a detector can be pulled
apart to inspect the raw score, to reuse the scorer under a different rule, or
simply to read how it was assembled.
:class:ScoreDetector pairs any scorer with any threshold. The named detectors
are the pairings worth having ready, one per phenomenon: a value outside its
usual range, a spike, a level shift, a change in volatility, a break in a
seasonal pattern, a break in the series' own dynamics, and — for several columns
at once — a broken relationship between them.
Labels are 1.0 anomalous, 0.0 normal and NaN unknown. NaN is common
and meaningful: a window-based detector cannot judge the first few observations,
and saying so is more useful than calling them normal.
AutoregressionDetector
¶
AutoregressionDetector(
n_steps: int = 1,
step_size: int = 1,
regressor: Regressor | None = None,
factor: Factor = 3.0,
side: Side = "both",
)
Bases: SignedScoreDetector
Flag points their own recent past fails to predict.
Fits the relationship between each value and the values a few steps before it, and judges the signed residual. This asks a sharper question than whether a value is unusual: whether it is unusual given where the series just was. A break in the dynamics is caught even at a perfectly ordinary level.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
n_steps
|
int
|
Number of past values to regress on. |
1
|
step_size
|
int
|
Gap in observations between them. With |
1
|
regressor
|
Regressor | None
|
Any object with |
None
|
factor
|
Factor
|
Inter-quartile-range factor deciding how large a residual is too large. |
3.0
|
side
|
Side
|
|
'both'
|
Raises:
| Type | Description |
|---|---|
ValueError
|
|
Notes
The first n_steps * step_size points have an incomplete history and are
labelled NaN.
Examples:
>>> import numpy as np
>>> from hazure import TimeSeries
>>> values = np.tile([1.0, 3.0, 5.0, 3.0], 8)
>>> values[17] = 11.0
>>> time = np.arange("2024-01-01", "2024-02-02", dtype="datetime64[D]")
>>> ts = TimeSeries.from_arrays(time, values)
>>> labels = AutoregressionDetector(n_steps=3).fit_detect(ts)
>>> np.flatnonzero(labels.values.ravel() == 1.0)
array([17])
Source code in src/hazure/detection/autoregression.py
EsdDetector
¶
Bases: ScoreDetector
Flag values by the generalised extreme Studentized deviate test.
Sets the line by a significance level rather than by a factor, which is
useful when a false-positive rate is easier to justify than a multiple of a
spread. Assumes the values are approximately normal; where that is doubtful,
:class:IqrDetector asks less of the data.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
alpha
|
float
|
Significance level, in |
0.05
|
Raises:
| Type | Description |
|---|---|
ValueError
|
|
ImportError
|
SciPy is not installed. |
Examples:
>>> import numpy as np
>>> from hazure import TimeSeries
>>> rng = np.random.default_rng(1)
>>> values = rng.normal(loc=20.0, size=60)
>>> values[42] = 30.0
>>> time = np.arange("2024-01-01", "2024-03-01", dtype="datetime64[D]")
>>> labels = EsdDetector().fit_detect(TimeSeries.from_arrays(time, values))
>>> np.flatnonzero(labels.values.ravel() == 1.0)
array([42])
Source code in src/hazure/detection/esd.py
IqrDetector
¶
Bases: ScoreDetector
Flag values far outside the training inter-quartile range.
The box-plot rule, and a sound default when nothing is known about the distribution: because quartiles ignore the tails, the outliers being looked for do not widen the range that is supposed to exclude them.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
factor
|
FactorSpec
|
One factor for both tails, or |
3.0
|
Raises:
| Type | Description |
|---|---|
ValueError
|
A factor is negative, or the pair is not of length two. |
Examples:
>>> import numpy as np
>>> from hazure import TimeSeries
>>> values = np.array([10.0, 11, 12, 11, 10, 12, 11, 10, 11, 60])
>>> time = np.arange("2024-01-01", "2024-01-11", dtype="datetime64[D]")
>>> ts = TimeSeries.from_arrays(time, values)
>>> IqrDetector().fit_detect(ts).values.ravel()
array([0., 0., 0., 0., 0., 0., 0., 0., 0., 1.])
Source code in src/hazure/detection/iqr.py
LevelShiftDetector
¶
LevelShiftDetector(
window: Window | tuple[Window, Window],
factor: Factor = 6.0,
side: Side = "both",
min_periods: int
| tuple[int | None, int | None]
| None = None,
)
Bases: SignedScoreDetector
Flag the point at which the series settles at a new level.
Two windows of equal length, one either side of each point, are summarised and compared. Both being long is what separates a level shift from a spike: a single odd value barely moves the median of a wide window, while a genuine step moves one window's median entirely away from the other's.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
window
|
Window | tuple[Window, Window]
|
Size of each window, or |
required |
factor
|
Factor
|
Inter-quartile-range factor deciding how large a shift is too large. Set higher than for spike detection by default, because the difference of two window medians is a much quieter signal than a single point's departure. |
6.0
|
side
|
Side
|
|
'both'
|
min_periods
|
int | tuple[int | None, int | None] | None
|
Minimum non-missing observations per window, or |
None
|
Raises:
| Type | Description |
|---|---|
ValueError
|
|
Notes
The two windows overlap the shift for as long as it takes them to clear it,
so the score stays high for a run of points around the change and the
detector flags a short plateau rather than a single instant. Narrowing
window narrows the plateau.
Examples:
>>> import numpy as np
>>> from hazure import TimeSeries
>>> values = np.concatenate([np.zeros(20), np.full(20, 10.0)])
>>> time = np.arange("2024-01-01", "2024-02-10", dtype="datetime64[D]")
>>> ts = TimeSeries.from_arrays(time, values)
>>> labels = LevelShiftDetector(window=3).fit_detect(ts)
>>> np.flatnonzero(labels.values.ravel() == 1.0)
array([19, 20, 21])
Source code in src/hazure/detection/level_shift.py
MinClusterDetector
¶
Bases: MultivariateScoreDetector
Flag points that fall in the rarest cluster.
Clusters the observations, treating each as a point in as many dimensions as there are columns, and calls the smallest group anomalous. Nothing needs to be said about what anomalous looks like: the shape of the data decides, which makes this the detector to reach for when the failure mode is unknown but known to be rare.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
model
|
Any
|
A clustering model with |
required |
Raises:
| Type | Description |
|---|---|
ValueError
|
The model has no |
Examples:
>>> import numpy as np
>>> from hazure import TimeSeries
>>> class NearestOfTwo:
... def fit_predict(self, X):
... self.split_ = X.mean()
... return self.predict(X)
... def predict(self, X):
... return (X.mean(axis=1) > self.split_).astype(int)
>>> pairs = np.column_stack([[1.0, 1, 1, 1, 1, 9], [2.0, 2, 2, 2, 2, 9]])
>>> time = np.arange("2024-01-01", "2024-01-07", dtype="datetime64[D]")
>>> ts = TimeSeries.from_arrays(time, pairs, ["a", "b"])
>>> MinClusterDetector(NearestOfTwo()).fit_detect(ts).values.ravel()
array([0., 0., 0., 0., 0., 1.])
Source code in src/hazure/detection/min_cluster.py
MultivariateScoreDetector
¶
MultivariateScoreDetector(
scorer: BaseScorer | None, threshold: BaseThreshold
)
Bases: ScoreDetector
A pairing whose scorer needs every column at once.
Fitting sees the whole frame rather than one column at a time, so the model
can learn how the columns relate. The single label series is reported under
the column name anomaly, since it describes the frame as a whole and not
any one of its columns.
Source code in src/hazure/detection/score.py
MultivariateSignedScoreDetector
¶
MultivariateSignedScoreDetector(
scorer: BaseScorer | None,
threshold: BaseThreshold,
side: Side = "both",
)
Bases: SignedScoreDetector
A signed pairing whose scorer needs every column at once.
Combines the direction filter of :class:SignedScoreDetector with the
whole-frame view of :class:MultivariateScoreDetector.
Source code in src/hazure/detection/signed_score.py
OutlierDetector
¶
Bases: MultivariateScoreDetector
Flag points a general-purpose outlier model rejects.
Treats each observation as a point in as many dimensions as there are columns
and ignores the time axis entirely, which is the right trade when the anomaly
is a combination of readings rather than a moment in a sequence. Any outlier
model marking outliers with -1 can be used.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
model
|
Any
|
An outlier model with either |
required |
Examples:
>>> import numpy as np
>>> from hazure import TimeSeries
>>> class FarFromCentre:
... def fit(self, X):
... self.centre_ = np.median(X, axis=0)
... return self
... def predict(self, X):
... return np.where(np.abs(X - self.centre_).sum(axis=1) > 5.0, -1, 1)
>>> pairs = np.column_stack([[0.0, 1, 0, 1, 20], [0.0, 1, 1, 0, 20]])
>>> time = np.arange("2024-01-01", "2024-01-06", dtype="datetime64[D]")
>>> ts = TimeSeries.from_arrays(time, pairs, ["a", "b"])
>>> OutlierDetector(FarFromCentre()).fit_detect(ts).values.ravel()
array([0., 0., 0., 0., 1.])
Source code in src/hazure/detection/outlier.py
PcaDetector
¶
Bases: MultivariateScoreDetector
Flag points that have left the subspace the data lives in.
Correlated columns confine every observation to a low-dimensional subspace of
the space they nominally span. Principal component analysis finds that
subspace from the training data, and the squared distance from a point to it
measures how far the columns have stopped agreeing with each other. Unlike
:class:RegressionDetector this singles out no column: any one of them, or
several together, can be the one that drifted.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
k
|
int
|
Number of principal components to keep — how many directions of genuine variation the data has. Everything else counts as error. |
1
|
factor
|
Factor
|
Inter-quartile-range factor deciding how large a reconstruction error is too large. |
5.0
|
Raises:
| Type | Description |
|---|---|
ValueError
|
|
Notes
A reconstruction error is a squared distance and so never negative, which makes its lower tail meaningless. Only the upper tail is tested, so a very tightly clustered training set cannot produce a positive lower cut-off that would flag the best-reconstructed points as anomalies.
Examples:
>>> import numpy as np
>>> from hazure import TimeSeries
>>> base = np.arange(20.0)
>>> partner = 2.0 * base + 1.0
>>> partner[6] += 15.0
>>> time = np.arange("2024-01-01", "2024-01-21", dtype="datetime64[D]")
>>> ts = TimeSeries.from_arrays(
... time, np.column_stack([base, partner]), ["a", "b"]
... )
>>> labels = PcaDetector(k=1).fit_detect(ts)
>>> np.flatnonzero(labels.values.ravel() == 1.0)
array([6])
Source code in src/hazure/detection/pca.py
QuantileDetector
¶
Bases: ScoreDetector
Flag values in the tails of the training distribution.
Makes no assumption about the shape of that distribution, only about how much
of it is acceptable: high=0.99 means "the top one per cent of what we have
seen is worth a look".
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
low
|
float | None
|
Lower quantile in |
None
|
high
|
float | None
|
Upper quantile in |
None
|
Raises:
| Type | Description |
|---|---|
ValueError
|
Both quantiles are None, or one falls outside |
Examples:
>>> import numpy as np
>>> from hazure import TimeSeries
>>> values = np.array([1.0, 2, 3, 4, 5, 6, 7, 8, 9, 90])
>>> time = np.arange("2024-01-01", "2024-01-11", dtype="datetime64[D]")
>>> ts = TimeSeries.from_arrays(time, values)
>>> QuantileDetector(high=0.9).fit_detect(ts).values.ravel()
array([0., 0., 0., 0., 0., 0., 0., 0., 0., 1.])
Source code in src/hazure/detection/quantile.py
RegressionDetector
¶
RegressionDetector(
target: str,
regressor: Regressor | None = None,
factor: Factor = 3.0,
side: Side = "both",
)
Bases: MultivariateSignedScoreDetector
Flag points where one column stops matching the others.
Predicts the target column from the rest and judges the signed residual. Where the columns are physically linked — a valve and the flow it causes, a request rate and the CPU it burns — the regression captures the link, and a large residual means the link itself has broken. Both columns can be in their usual range and still be impossible together.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
target
|
str
|
Name of the column to predict. Every other column is a feature. |
required |
regressor
|
Regressor | None
|
Any object with |
None
|
factor
|
Factor
|
Inter-quartile-range factor deciding how large a residual is too large. |
3.0
|
side
|
Side
|
|
'both'
|
Raises:
| Type | Description |
|---|---|
ValueError
|
|
Examples:
>>> import numpy as np
>>> from hazure import TimeSeries
>>> drive = np.tile([1.0, 2.0, 3.0, 4.0], 5)
>>> follow = 3.0 * drive - 2.0
>>> follow[11] += 20.0
>>> time = np.arange("2024-01-01", "2024-01-21", dtype="datetime64[D]")
>>> ts = TimeSeries.from_arrays(
... time, np.column_stack([drive, follow]), ["drive", "follow"]
... )
>>> labels = RegressionDetector(target="follow").fit_detect(ts)
>>> list(labels.columns)
['anomaly']
>>> np.flatnonzero(labels.values.ravel() == 1.0)
array([11])
Source code in src/hazure/detection/regression.py
ScoreDetector
¶
ScoreDetector(
scorer: BaseScorer | None, threshold: BaseThreshold
)
Bases: BaseDetector
A scorer and a threshold, applied in that order.
Fitting fits the scorer, then fits the threshold on the scores the fitted scorer produces for the training data — so the threshold learns the scale the scorer actually works on, which is the whole reason the two are fitted together rather than independently.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
scorer
|
BaseScorer | None
|
The scorer to apply, or None when the series is already its own score, as it is for a plain value range. |
required |
threshold
|
BaseThreshold
|
The rule that turns those scores into labels. |
required |
Examples:
Any scorer pairs with any threshold:
>>> import numpy as np
>>> from hazure import TimeSeries
>>> from hazure.scoring import DeviationScorer
>>> from hazure.thresholds import MadThreshold
>>> values = np.array([5.0, 6.0, 5.0, 6.0, 5.0, 6.0, 40.0])
>>> ts = TimeSeries.from_arrays(
... np.arange("2024-01-01", "2024-01-08", dtype="datetime64[D]"), values
... )
>>> detector = ScoreDetector(DeviationScorer(), MadThreshold())
>>> detector.fit_detect(ts).values.ravel()
array([0., 0., 0., 0., 0., 0., 1.])
>>> detector.scorer.center_
6.0
Source code in src/hazure/detection/score.py
SeasonalDetector
¶
SeasonalDetector(
period: int | None = None,
factor: Factor = 3.0,
side: Side = "both",
trend: bool = False,
)
Bases: SignedScoreDetector
Flag points that break a repeating pattern.
A daily or weekly cycle is normal behaviour, so it is subtracted before anything is judged: what remains is the part of the series the pattern does not explain, and it is that remainder the threshold is applied to. A value perfectly ordinary for a Tuesday is therefore anomalous on a Sunday.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
period
|
int | None
|
Length of a cycle in observations. When None it is detected from the autocorrelation of the training series. |
None
|
factor
|
Factor
|
Inter-quartile-range factor deciding how large a residual is too large. |
3.0
|
side
|
Side
|
|
'both'
|
trend
|
bool
|
Remove a moving-average trend as well as the seasonal profile. Costs a NaN margin of half a period at each end, where the centred average has no window. |
False
|
Raises:
| Type | Description |
|---|---|
ValueError
|
|
Examples:
>>> import numpy as np
>>> from hazure import TimeSeries
>>> values = np.tile([1.0, 5.0, 3.0, 2.0], 8)
>>> values[13] = 12.0
>>> time = np.arange("2024-01-01", "2024-02-02", dtype="datetime64[D]")
>>> ts = TimeSeries.from_arrays(time, values)
>>> labels = SeasonalDetector(period=4).fit_detect(ts)
>>> np.flatnonzero(labels.values.ravel() == 1.0)
array([13])
Source code in src/hazure/detection/seasonal.py
SignedScoreDetector
¶
SignedScoreDetector(
scorer: BaseScorer | None,
threshold: BaseThreshold,
side: Side = "both",
)
Bases: ScoreDetector
A signed score, thresholded on magnitude and filtered by direction.
The threshold judges |score|, so "how big is too big" is asked once and
answered symmetrically. The sign of the score then says which way the series
moved, and side decides whether that direction is of interest. Detecting
only the increases is therefore not a different algorithm, just a filter on
the same one.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
scorer
|
BaseScorer | None
|
A scorer whose sign is meaningful. |
required |
threshold
|
BaseThreshold
|
The rule applied to the magnitude. A one-sided rule such as
|
required |
side
|
Side
|
|
'both'
|
Raises:
| Type | Description |
|---|---|
ValueError
|
|
Examples:
>>> import numpy as np
>>> from hazure import TimeSeries
>>> from hazure.scoring import DoubleRollingScorer
>>> from hazure.thresholds import IqrThreshold
>>> values = np.array([1.0, 1.0, 1.0, 9.0, 1.0, 1.0, 1.0, 1.0])
>>> ts = TimeSeries.from_arrays(
... np.arange("2024-01-01", "2024-01-09", dtype="datetime64[D]"), values
... )
>>> detector = SignedScoreDetector(
... DoubleRollingScorer(window=(3, 1), diff="diff"),
... IqrThreshold(factor=(None, 3.0)),
... side="positive",
... )
>>> detector.fit_detect(ts).values.ravel()
array([nan, nan, nan, 1., 0., 0., 0., 0.])
Source code in src/hazure/detection/signed_score.py
SpikeDetector
¶
SpikeDetector(
window: Window = 1,
factor: Factor = 3.0,
side: Side = "both",
min_periods: int | None = None,
agg: str = "median",
)
Bases: SignedScoreDetector
Flag points that depart sharply from the values just before them.
The window in front of each point is one observation wide and the window
behind it is window wide: a short right window catches the blip while the
long left window keeps a stable notion of recent normal, which is what makes
this asymmetry the shape of spike detection. Because the comparison is local,
a slow drift is invisible to it — which is the point.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
window
|
Window
|
Size of the preceding window: observations ( |
1
|
factor
|
Factor
|
Inter-quartile-range factor deciding how large a departure is too large. |
3.0
|
side
|
Side
|
|
'both'
|
min_periods
|
int | None
|
Minimum non-missing observations in the preceding window. |
None
|
agg
|
str
|
How to summarise the preceding window: |
'median'
|
Raises:
| Type | Description |
|---|---|
ValueError
|
|
Notes
With the default window=1 the preceding window is a single observation, so
a one-point spike changes the score twice — once on the way up and once on the
way back down — and side="both" flags both the spike and the point after
it. side="positive" isolates the spike itself. A wider window has a median
the spike cannot move, and then the spike alone scores.
Examples:
>>> import numpy as np
>>> from hazure import TimeSeries
>>> values = np.ones(20)
>>> values[12] = 9.0
>>> time = np.arange("2024-01-01", "2024-01-21", dtype="datetime64[D]")
>>> ts = TimeSeries.from_arrays(time, values)
>>> np.flatnonzero(SpikeDetector().fit_detect(ts).values.ravel() == 1.0)
array([12, 13])
>>> labels = SpikeDetector(side="positive").fit_detect(ts)
>>> np.flatnonzero(labels.values.ravel() == 1.0)
array([12])
Source code in src/hazure/detection/spike.py
ThresholdDetector
¶
Bases: ScoreDetector
Flag values outside a range the caller supplies.
The simplest possible detector, and the only one that learns nothing: use it when the acceptable range is known in advance. There is no scorer, because a value is already the quantity being judged.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
low
|
float | None
|
Values below this are anomalous. None leaves the lower side unbounded. |
None
|
high
|
float | None
|
Values above this are anomalous. None leaves the upper side unbounded. |
None
|
Raises:
| Type | Description |
|---|---|
ValueError
|
Both bounds are None, which would make the detector inert. |
Examples:
>>> import numpy as np
>>> from hazure import TimeSeries
>>> time = np.arange("2024-01-01", "2024-01-06", dtype="datetime64[D]")
>>> ts = TimeSeries.from_arrays(time, [20.0, 21.0, 45.0, 19.0, -5.0])
>>> ThresholdDetector(low=0.0, high=40.0).detect(ts).values.ravel()
array([0., 0., 1., 0., 1.])
Source code in src/hazure/detection/threshold.py
VolatilityShiftDetector
¶
VolatilityShiftDetector(
window: Window | tuple[Window, Window],
factor: Factor = 6.0,
side: Side = "both",
min_periods: int
| tuple[int | None, int | None]
| None = None,
agg: str = "std",
)
Bases: SignedScoreDetector
Flag the point at which the series becomes more or less erratic.
The same two symmetric windows as level-shift detection, with two changes that matter. The statistic measures spread rather than position, so the level can stay put while the noise around it changes. And the comparison is relative — the change in spread divided by the earlier spread — because a doubling of noise is equally significant on a quiet series and a loud one, which an absolute difference would not capture.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
window
|
Window | tuple[Window, Window]
|
Size of each window, or |
required |
factor
|
Factor
|
Inter-quartile-range factor deciding how large a relative change is too large. |
6.0
|
side
|
Side
|
|
'both'
|
min_periods
|
int | tuple[int | None, int | None] | None
|
Minimum non-missing observations per window, or |
None
|
agg
|
str
|
How to measure spread: |
'std'
|
Raises:
| Type | Description |
|---|---|
ValueError
|
|
Notes
A spread is never negative, so the sign of the relative change is the sign of
the change itself and side reads as expected. A window with no spread at
all makes the relative change undefined, and those points score NaN.
Two consequences of measuring spread over a window are worth knowing:
- A relative increase is unbounded while a relative decrease cannot pass
-1, so a fall in volatility produces a smaller score than the equivalent
rise. Detecting
side="negative"usually wants a smallerfactor. - A level shift falling inside a window inflates that window's spread, so a
step registers here as well as in :class:
LevelShiftDetector. Running both is how the two are told apart.
Examples:
>>> import numpy as np
>>> from hazure import TimeSeries
>>> rng = np.random.default_rng(0)
>>> quiet, loud = rng.normal(scale=0.1, size=40), rng.normal(scale=5.0, size=40)
>>> time = np.arange("2024-01-01", "2024-03-21", dtype="datetime64[D]")
>>> ts = TimeSeries.from_arrays(time, np.concatenate([quiet, loud]))
>>> labels = VolatilityShiftDetector(window=10).fit_detect(ts)
>>> bool(labels.values.ravel()[40] == 1.0)
True
Source code in src/hazure/detection/volatility_shift.py
hazure.scoring¶
The scorers those detectors are built from. A score is worth having on its own, for ranking.
scoring
¶
Scoring: series in, "how unusual is each point" out.
A score is a continuous number per observation, on whatever scale the algorithm
naturally works in, where a larger magnitude means more unusual. Scores are
useful on their own — for ranking, for plotting, for feeding a model — and become
labels when passed through a :class:hazure.BaseThreshold.
Several scorers are signed, and the sign carries information: which way the series moved. A detector can therefore use one scorer and still act on increases only, or decreases only, without a second pass over the data.
Univariate scorers handle one series at a time and fan out over the columns of a frame, each column learning its own normal. Multivariate ones need every column at once, which is what lets them see an anomaly that lives in the relationship between columns rather than in any one of them.
AutoregressionResidualScorer
¶
AutoregressionResidualScorer(
n_steps: int = 1,
step_size: int = 1,
regressor: Regressor | None = None,
)
Bases: BaseScorer
Score each point by what its own recent past fails to predict.
Many series are largely predictable from where they just were. Fitting that relationship and scoring the signed residual asks a sharper question than "is this value unusual": it asks whether the value is unusual given what came immediately before, which catches a break in the dynamics even at a perfectly ordinary level.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
n_steps
|
int
|
Number of past values to regress on. |
1
|
step_size
|
int
|
Gap in observations between them. With |
1
|
regressor
|
Regressor | None
|
Any object with |
None
|
Attributes:
| Name | Type | Description |
|---|---|---|
transformer_ |
RegressionResidual
|
The fitted regression stage, whose |
Raises:
| Type | Description |
|---|---|
ValueError
|
|
Notes
The regressor is deep-copied at :meth:fit time. A scorer handed a frame
fans out into one copy per column, and without the copy every column would
fit the same model object and only the last would survive. It also leaves the
caller's object untouched; the fitted one is reachable through
transformer_.regressor_.
The first n_steps * step_size points have an incomplete history and so
score NaN.
Examples:
>>> import numpy as np
>>> from hazure import TimeSeries
>>> values = np.tile([1.0, 2.0, 3.0], 6)
>>> values[10] = 9.0
>>> time = np.arange("2024-01-01", "2024-01-19", dtype="datetime64[D]")
>>> ts = TimeSeries.from_arrays(time, values)
>>> scores = AutoregressionResidualScorer(n_steps=3).fit_score(ts).values.ravel()
>>> int(np.nanargmax(np.abs(scores)))
10
Source code in src/hazure/scoring/autoregression_residual.py
DeviationScorer
¶
Bases: BaseScorer
Score each point by its signed distance from a learned centre.
A robust z-score: (x - centre) / scale, where both are learned once from
the training series. With a median centre and an inter-quartile-range scale,
neither estimate is moved by the outliers being looked for, which a mean and
a standard deviation both are — a single value a thousand times too large
inflates the scale enough to hide itself.
The score keeps its sign, so a detector can act on excursions in one direction only.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
center
|
Centre
|
Where normal sits: |
'median'
|
scale
|
Scale
|
What one unit of deviation is: |
'iqr'
|
Attributes:
| Name | Type | Description |
|---|---|---|
center_ |
float
|
The learned centre. |
scale_ |
float
|
The learned scale. Zero for a constant training series. |
Raises:
| Type | Description |
|---|---|
ValueError
|
|
Notes
A constant training series has no spread, so every point equal to the centre scores 0 and anything else scores infinity. That is the honest reading: with no observed variation, any change at all is unprecedented.
Examples:
>>> import numpy as np
>>> from hazure import TimeSeries
>>> time = np.arange("2024-01-01", "2024-01-08", dtype="datetime64[D]")
>>> ts = TimeSeries.from_arrays(time, [10.0, 12.0, 11.0, 13.0, 12.0, 10.0, 40.0])
>>> scorer = DeviationScorer().fit(ts)
>>> (scorer.center_, scorer.scale_)
(12.0, 2.0)
>>> scorer.score(ts).values.ravel()
array([-1. , 0. , -0.5, 0.5, 0. , -1. , 14. ])
Source code in src/hazure/scoring/deviation.py
DoubleRollingScorer
¶
DoubleRollingScorer(
window: Window | tuple[Window, Window],
agg: str | tuple[str, str] = "median",
diff: Diff = "l1",
min_periods: int
| tuple[int | None, int | None]
| None = None,
q: float | None = None,
)
Bases: TransformerScorer
Score each point by how much the series changes across it.
The window before each point is summarised, the window from it onwards is summarised, and the two are compared. One scorer covers three phenomena, and only the settings differ:
- spike — a long left window, a right window of 1, so a single blip is measured against a stable notion of recent normal;
- level shift — two windows of equal length, long enough that both sides are stable, so a persistent change registers and a lone spike does not;
- volatility shift — the same symmetric windows with a dispersion
statistic (
agg="std") and a relative comparison (diff="rel_diff"), because a doubling of noise matters equally whether the series is quiet or loud.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
window
|
Window | tuple[Window, Window]
|
One spec for both sides, or |
required |
agg
|
str | tuple[str, str]
|
One statistic for both sides, or |
'median'
|
diff
|
Diff
|
How to compare the two sides: |
'l1'
|
min_periods
|
int | tuple[int | None, int | None] | None
|
Minimum non-missing observations per side, or |
None
|
q
|
float | None
|
Quantile in |
None
|
Examples:
A step of 5 registers at the step, and the score is unavailable within a window of each end:
>>> import numpy as np
>>> from hazure import TimeSeries
>>> time = np.arange("2024-01-01", "2024-01-07", dtype="datetime64[D]")
>>> ts = TimeSeries.from_arrays(time, [0.0, 0.0, 0.0, 5.0, 5.0, 5.0])
>>> DoubleRollingScorer(window=2, diff="diff").score(ts).values.ravel()
array([nan, nan, 2.5, 5. , 2.5, nan])
Source code in src/hazure/scoring/double_rolling.py
MinClusterScorer
¶
Bases: BaseScorer
Score each point by whether it falls in the rarest cluster.
Clustering the observations and calling the smallest group anomalous needs no
labels and no notion of a threshold — the shape of the data decides. The
score is 1.0 for membership of the smallest cluster and 0.0 otherwise, so
pairing it with FixedThreshold(high=0.5) turns it into labels.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
model
|
Any
|
A clustering model with |
required |
Attributes:
| Name | Type | Description |
|---|---|---|
model_ |
object
|
The fitted clustering model. |
smallest_cluster_ |
int or None
|
Label of the cluster judged anomalous, or None when the training data formed a single cluster and so has no rare minority. |
Raises:
| Type | Description |
|---|---|
ValueError
|
The model has no |
Notes
Many clustering algorithms are transductive: they label the points they were
given and cannot place a new one. Those cannot be used here, and the check
happens at :meth:fit, where the fix — a model that generalises, such as
k-means or a Gaussian mixture — is still actionable.
Ties are broken towards the lowest cluster label, so the choice is deterministic when two clusters are equally small.
A clustering that finds only one group has no smallest group, and every point scores 0. Calling the single cluster anomalous would flag the entire series.
The model is deep-copied at :meth:fit time, leaving the caller's object
untouched.
Examples:
>>> import numpy as np
>>> from hazure import TimeSeries
>>> class TwoMeans:
... def fit_predict(self, X):
... self.split_ = X[:, 0].mean()
... return self.predict(X)
... def predict(self, X):
... return (X[:, 0] > self.split_).astype(int)
>>> ts = TimeSeries.from_arrays(
... np.arange("2024-01-01", "2024-01-08", dtype="datetime64[D]"),
... np.column_stack([[0.0, 0, 0, 0, 0, 0, 9], [1.0, 1, 1, 1, 1, 1, 9]]),
... ["a", "b"],
... )
>>> MinClusterScorer(TwoMeans()).fit_score(ts).values.ravel()
array([0., 0., 0., 0., 0., 0., 1.])
Source code in src/hazure/scoring/min_cluster.py
OutlierScorer
¶
Bases: BaseScorer
Score each point with a time-independent outlier detection model.
Treats every observation as a point in as many dimensions as there are
columns and asks a general-purpose outlier model about it, ignoring the time
axis entirely. The score is 1.0 for an outlier and 0.0 otherwise, so
FixedThreshold(high=0.5) turns it into labels.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
model
|
Any
|
An outlier model marking outliers with
|
required |
Attributes:
| Name | Type | Description |
|---|---|---|
model_ |
object
|
The fitted outlier model. |
Notes
The model is deep-copied at :meth:fit time, leaving the caller's object
untouched.
Examples:
>>> import numpy as np
>>> from hazure import TimeSeries
>>> class FarFromCentre:
... def fit(self, X):
... self.centre_ = np.median(X, axis=0)
... return self
... def predict(self, X):
... far = np.abs(X - self.centre_).sum(axis=1) > 5.0
... return np.where(far, -1, 1)
>>> ts = TimeSeries.from_arrays(
... np.arange("2024-01-01", "2024-01-06", dtype="datetime64[D]"),
... np.column_stack([[0.0, 1, 0, 1, 20], [0.0, 1, 1, 0, 20]]),
... ["a", "b"],
... )
>>> OutlierScorer(FarFromCentre()).fit_score(ts).values.ravel()
array([0., 0., 0., 0., 1.])
Source code in src/hazure/scoring/outlier.py
PcaReconstructionErrorScorer
¶
Bases: TransformerScorer
Score each point by how far it lies from the principal subspace.
Correlated columns confine every observation to a low-dimensional subspace of the space they nominally span. Principal component analysis finds that subspace, and the squared distance from a point to it says how far the columns have stopped agreeing with one another — which is exactly the kind of anomaly that hides from every column individually.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
k
|
int
|
Number of principal components to keep. The score is the squared
Euclidean distance to the best rank- |
1
|
Attributes:
| Name | Type | Description |
|---|---|---|
mean_ |
ndarray
|
Column means of the training data. |
components_ |
ndarray
|
The |
Examples:
Two columns on a line lie in a one-dimensional subspace, so a point knocked off that line stands out even though neither of its coordinates is extreme:
>>> import numpy as np
>>> from hazure import TimeSeries
>>> base = np.arange(8.0)
>>> partner = 2.0 * base + 1.0
>>> partner[3] = 2.0 * base[3] - 4.0
>>> ts = TimeSeries.from_arrays(
... np.arange("2024-01-01", "2024-01-09", dtype="datetime64[D]"),
... np.column_stack([base, partner]),
... ["a", "b"],
... )
>>> scores = PcaReconstructionErrorScorer(k=1).fit_score(ts)
>>> int(np.argmax(scores.values))
3
Source code in src/hazure/scoring/pca_reconstruction_error.py
RegressionResidualScorer
¶
RegressionResidualScorer(
target: str, regressor: Regressor | None = None
)
Bases: TransformerScorer
Score each point by what the other columns fail to predict about one.
Where columns are physically linked — a valve position and the flow it causes, a request rate and the CPU it burns — a regression captures the link and the signed residual is what the link cannot explain. A large residual means the relationship itself broke, which no single column would reveal.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
target
|
str
|
Name of the column to predict. Every other column is a feature. |
required |
regressor
|
Regressor | None
|
Any object with |
None
|
Attributes:
| Name | Type | Description |
|---|---|---|
transformer_ |
RegressionResidual
|
The fitted regression stage, whose |
Notes
The regressor is deep-copied at :meth:fit time, so the caller's object is
left unfitted and reusable.
Examples:
>>> import numpy as np
>>> from hazure import TimeSeries
>>> drive = np.arange(8.0)
>>> follow = 2.0 * drive + 1.0
>>> follow[5] += 10.0
>>> ts = TimeSeries.from_arrays(
... np.arange("2024-01-01", "2024-01-09", dtype="datetime64[D]"),
... np.column_stack([drive, follow]),
... ["drive", "follow"],
... )
>>> scores = RegressionResidualScorer(target="follow").fit_score(ts)
>>> int(np.argmax(np.abs(scores.values)))
5
Source code in src/hazure/scoring/regression_residual.py
RollingAggregateScorer
¶
RollingAggregateScorer(
window: Window,
agg: str = "mean",
center: bool = False,
min_periods: int | None = None,
closed: Closed | None = None,
q: float | None = None,
)
Bases: TransformerScorer
Score each point by a statistic of the window ending at it.
A feature-style score: not anomalous or not on its own, but a summary the threshold can be pointed at. Counting non-zero values in a rolling day, for instance, turns "the pump idled" into a number a quantile rule can judge.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
window
|
Window
|
Observations ( |
required |
agg
|
str
|
A name from :data: |
'mean'
|
center
|
bool
|
Centre the window on each point instead of trailing it. |
False
|
min_periods
|
int | None
|
Minimum non-missing observations for a result. Defaults to the full window for integer windows and to 1 for duration windows. |
None
|
closed
|
Closed | None
|
Which window endpoints to include; defaults to |
None
|
q
|
float | None
|
Quantile in |
None
|
Examples:
>>> import numpy as np
>>> from hazure import TimeSeries
>>> time = np.arange("2024-01-01", "2024-01-07", dtype="datetime64[D]")
>>> ts = TimeSeries.from_arrays(time, [1.0, 1.0, 1.0, 9.0, 1.0, 1.0])
>>> RollingAggregateScorer(window=2, agg="max").score(ts).values.ravel()
array([nan, 1., 1., 9., 9., 1.])
Source code in src/hazure/scoring/rolling_aggregate.py
SeasonalResidualScorer
¶
Bases: TransformerScorer
Score each point by what a repeating pattern fails to explain.
A daily or weekly cycle is normal behaviour, so it belongs in the model rather than in the anomalies. Classic additive decomposition removes it: the seasonal profile is the average shape of one cycle, optionally on top of a moving-average trend, and the residual is the signed remainder. The profile is learned once, so a later series is judged against the pattern that used to hold rather than against its own.
Requires a regular time axis at :meth:fit, since a cycle length in
observations is only meaningful if observations are evenly spaced. Later
series may have gaps: the phase of each timestamp is recovered arithmetically
from the training anchor.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
period
|
int | None
|
Length of a cycle in observations. When None it is detected from the autocorrelation of the training series. |
None
|
trend
|
bool
|
Estimate and remove a moving-average trend as well. Adds a NaN margin of half a period at each end, where the centred average has no window. |
False
|
Attributes:
| Name | Type | Description |
|---|---|---|
period_ |
int
|
Cycle length used, whether given or detected. |
seasonal_ |
ndarray
|
The learned profile, of length |
Examples:
>>> import numpy as np
>>> from hazure import TimeSeries
>>> values = np.tile([0.0, 1.0, 0.0, -1.0], 4)
>>> values[9] = 6.0
>>> time = np.arange("2024-01-01", "2024-01-17", dtype="datetime64[D]")
>>> ts = TimeSeries.from_arrays(time, values)
>>> scorer = SeasonalResidualScorer(period=4).fit(ts)
>>> scorer.period_
4
>>> int(np.argmax(scorer.score(ts).values))
9
Source code in src/hazure/scoring/seasonal_residual.py
hazure.thresholds¶
Where to draw the line, independently of what produced the score.
thresholds
¶
Thresholding: continuous scores in, binary labels out.
A threshold is the decision "is that unusual enough to report", kept as its own
object so that one policy can be reused across every scorer and swapped without
touching the scorer. Labels are 1.0 anomalous, 0.0 normal and NaN
unknown; a score of NaN yields a label of NaN, because a point nobody measured
cannot be declared normal.
:class:FixedThreshold draws the line where the caller says. The rest learn it
from history: a quantile of the training scores, a multiple of their
inter-quartile range or median absolute deviation, or a formal test of how
extreme a value can be before it stops looking like a sample from the same
distribution. :class:PotThreshold is parameterised the other way round: you
give it the false-alarm probability you are willing to accept, and it fits the
tail of the training scores well enough to place a fence there — including
beyond the largest score ever seen, which no quantile of a sample can reach.
EsdThreshold
¶
Bases: BaseThreshold
Flag scores by the generalised extreme Studentized deviate test.
The test [1]_ repeatedly removes the observation furthest from the mean and compares its Studentized deviate against a critical value derived from the t distribution, stopping at the first observation the test accepts. Fitting therefore splits the training scores into outliers and a normal set, of which only three sufficient statistics need keeping: count, sum and sum of squares. Prediction adds one candidate point back to that set and runs a single step of the test, which is pure arithmetic and so vectorises over the whole series at once.
The test assumes approximately normal scores. Use it only where that holds;
:class:IqrThreshold makes no distributional assumption.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
alpha
|
float
|
Significance level, in |
0.05
|
Attributes:
| Name | Type | Description |
|---|---|---|
count_ |
int
|
Number of training scores judged normal. |
sum_ |
float
|
Their sum. |
sum_squares_ |
float
|
Their sum of squares. |
critical_value_ |
float
|
Critical value for the one-step test used at prediction time. |
Raises:
| Type | Description |
|---|---|
ValueError
|
|
ImportError
|
SciPy is not installed. |
Notes
Fitting is a sequential loop, because each removal moves the mean that decides the next removal. Sorting the scores first makes every step O(1) — the point furthest from the mean is always one of the two extremes of what remains — so the whole fit costs O(n log n).
Fewer than two valid training scores leave the statistics unknown and every label NaN, since a spread cannot be estimated from a single point.
References
.. [1] B. Rosner, "Percentage Points for a Generalized ESD Many-Outlier Procedure", Technometrics 25(2), 1983, pp. 165-172.
Examples:
>>> import numpy as np
>>> from hazure import TimeSeries
>>> rng = np.random.default_rng(0)
>>> values = rng.normal(size=60)
>>> values[17] = 12.0
>>> time = np.arange("2024-01-01", "2024-03-01", dtype="datetime64[D]")
>>> ts = TimeSeries.from_arrays(time, values)
>>> labels = EsdThreshold().fit(ts).run(ts).values.ravel()
>>> np.flatnonzero(labels == 1.0)
array([17])
Source code in src/hazure/thresholds/esd.py
FixedThreshold
¶
Bases: BaseThreshold
Flag scores outside a range the caller supplies.
Nothing is learned, so this is usable without :meth:fit. It is the right
choice when the acceptable range comes from domain knowledge — a pressure
limit, a service-level objective — rather than from history.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
low
|
float | None
|
Scores below this are flagged. None leaves the lower side unbounded. |
None
|
high
|
float | None
|
Scores above this are flagged. None leaves the upper side unbounded. |
None
|
Raises:
| Type | Description |
|---|---|
ValueError
|
Both bounds are None, which would make the threshold inert. |
Examples:
>>> import numpy as np
>>> from hazure import TimeSeries
>>> time = np.arange("2024-01-01", "2024-01-06", dtype="datetime64[D]")
>>> ts = TimeSeries.from_arrays(time, [0.0, 9.0, 1.0, np.nan, -9.0])
>>> FixedThreshold(low=-5.0, high=5.0).run(ts).values.ravel()
array([ 0., 1., 0., nan, 1.])
Source code in src/hazure/thresholds/fixed.py
IqrThreshold
¶
Bases: BaseThreshold
Flag scores far outside the training inter-quartile range.
The cut-offs are Q1 - factor_low * IQR and Q3 + factor_high * IQR,
the rule behind a box plot's whiskers. Quartiles ignore the tails, so the
very outliers being looked for cannot drag the line out to meet them, which
is why this is the test the compound detectors end with.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
factor
|
FactorSpec
|
One factor for both tails, or |
3.0
|
Attributes:
| Name | Type | Description |
|---|---|---|
low_ |
float
|
Absolute lower cut-off learned from the training scores. |
high_ |
float
|
Absolute upper cut-off learned from the training scores. |
Raises:
| Type | Description |
|---|---|
ValueError
|
A factor is negative, or the pair is not of length two. |
Examples:
>>> import numpy as np
>>> from hazure import TimeSeries
>>> time = np.arange("2024-01-01", "2024-01-10", dtype="datetime64[D]")
>>> ts = TimeSeries.from_arrays(time, [1.0, 2, 3, 4, 5, 6, 7, 8, 99])
>>> threshold = IqrThreshold(factor=1.5).fit(ts)
>>> (threshold.low_, threshold.high_)
(-3.0, 13.0)
>>> threshold.run(ts).values.ravel()
array([0., 0., 0., 0., 0., 0., 0., 0., 1.])
Source code in src/hazure/thresholds/iqr.py
MadThreshold
¶
Bases: BaseThreshold
Flag scores far from the training median, in units of the MAD.
The median absolute deviation is the median of |x - median(x)|. Scaled by
:data:MAD_SCALE it estimates the standard deviation of a normal sample, so
factor reads on the familiar "number of sigmas" scale while staying
immune to the outliers a real standard deviation would absorb.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
factor
|
FactorSpec
|
One factor for both tails, or |
3.0
|
Attributes:
| Name | Type | Description |
|---|---|---|
low_ |
float
|
Absolute lower cut-off learned from the training scores. |
high_ |
float
|
Absolute upper cut-off learned from the training scores. |
Raises:
| Type | Description |
|---|---|
ValueError
|
A factor is negative, or the pair is not of length two. |
Examples:
>>> import numpy as np
>>> from hazure import TimeSeries
>>> time = np.arange("2024-01-01", "2024-01-08", dtype="datetime64[D]")
>>> ts = TimeSeries.from_arrays(time, [4.0, 5.0, 6.0, 5.0, 4.0, 6.0, 40.0])
>>> MadThreshold().fit(ts).run(ts).values.ravel()
array([0., 0., 0., 0., 0., 0., 1.])
Source code in src/hazure/thresholds/mad.py
PotThreshold
¶
Bases: BaseThreshold
Flag scores beyond a generalised Pareto fit to the tail of the training scores.
Peaks-over-threshold: the training scores above their own level quantile
are kept as excesses over it, a generalised Pareto distribution is fitted to
those excesses by maximum likelihood, and the fence is placed where that
distribution says an exceedance has probability high.
What this buys over :class:~hazure.QuantileThreshold is extrapolation. A
quantile of the training scores can never be placed beyond the largest one
observed, so asking for a one-in-a-million fence from ten thousand samples
quietly gives you the maximum instead. The fitted tail is a model, and can be
asked about probabilities no sample of that size could resolve.
What it costs is an assumption. The theorem behind it is asymptotic in
level, and the fit needs enough excesses to identify two parameters, so
the two parameters pull against each other: raise level and the model is
better justified but fitted from less data. The default keeps 2% of the
training scores, which wants a few thousand of them to work with.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
low
|
float | None
|
Target exceedance probability for the lower tail, in |
None
|
high
|
float | None
|
Target exceedance probability for the upper tail, in |
0.0001
|
level
|
float
|
Quantile of the training scores where the tail is taken to start, in
|
0.98
|
Attributes:
| Name | Type | Description |
|---|---|---|
low_ |
float
|
Absolute lower cut-off, |
high_ |
float
|
Absolute upper cut-off, |
tail_ |
dict of dict
|
The fitted tail for each bounded side that had enough excesses, keyed
|
Raises:
| Type | Description |
|---|---|
ValueError
|
Both probabilities are None, one falls outside |
See Also
hazure.QuantileThreshold : A quantile of the training scores, no model and no extrapolation. update : Drive the same fence online, letting it move as scores arrive.
Notes
Write t for the level quantile of the n training scores, N_t
for how many exceeded it, and gamma and sigma for the fitted shape and
scale. The fence for a target probability q is then::
p = q * n / N_t
z_q = t + (sigma / gamma) * (p ** -gamma - 1)
z_q = t - sigma * log(p) # the gamma -> 0 limit
That p is q rescaled from a probability over the whole distribution to
one conditional on having cleared t, and it has to be below 1 for the
question to be about the tail at all — which is why high must be smaller
than 1 - level. Fitting says so rather than silently answering a different
question. The second line is the exponential tail, and is used exactly when
the fit chose a shape of zero rather than approached one.
The maximum likelihood fit follows Grimshaw [2]_: substituting
theta = gamma / sigma reduces two parameters to one, because the
likelihood can then be maximised over gamma in closed form for any fixed
theta. What is left is a scalar equation, solved here by scanning for sign
changes and bisecting each. The exponential limit is always evaluated as a
candidate too, and whichever candidate has the highest likelihood wins — so a
tail that really is exponential is fitted as one rather than through a shape
parameter driven to zero.
The search covers shapes down to about -1 and no further, which is the
regularity condition rather than a limitation of the solver: below -1 the
likelihood is unbounded, and it is maximised by putting the tail's endpoint at
the largest excess observed and letting the density diverge there. There is no
estimate to find in that region, so a sample light-tailed enough to ask for one
is fitted as exponential instead — which places the fence higher than the
degenerate solution would, and so errs towards silence rather than towards
false alarms.
The lower tail is fitted by negating the scores and running the identical upper-tail machinery, so both sides come from the same code and the same tests.
References
.. [1] A. Siffer, P.-A. Fouque, A. Termier and C. Largouet, "Anomaly Detection in Streams with Extreme Value Theory", KDD 2017, pp. 1067-1075. .. [2] S. D. Grimshaw, "Computing Maximum Likelihood Estimates for the Generalized Pareto Distribution", Technometrics 35(2), 1993, pp. 185-191.
Examples:
Two thousand scores from a standard exponential, and a fence asked to sit where exceedance has probability one in ten thousand. It lands beyond the largest score ever observed, which is the extrapolation a quantile of the same sample cannot perform at all:
>>> import numpy as np
>>> from hazure import TimeSeries
>>> rng = np.random.default_rng(0)
>>> scores = rng.exponential(size=2000)
>>> time = np.arange(2000, dtype=np.int64) * 60_000_000_000
>>> ts = TimeSeries.from_arrays(time, scores)
>>> threshold = PotThreshold(high=1e-4).fit(ts)
>>> round(threshold.high_, 2), round(float(scores.max()), 2)
(8.79, 8.15)
The true one-in-ten-thousand point of a standard exponential is
-log(1e-4) = 9.21, so 40 excesses got within 5% of a quantile the sample
itself could never have located:
Nothing in the training scores is beyond the fence, which is what asking for one in ten thousand from two thousand samples should mean:
Asking for a probability that is not in the tail is refused rather than answered from the body of the distribution:
>>> PotThreshold(high=0.1, level=0.98).fit(ts)
Traceback (most recent call last):
...
ValueError: PotThreshold cannot place the high fence at high=0.1: the tail...
Source code in src/hazure/thresholds/pot.py
update
¶
Absorb one new score, moving the fence, and return its label.
This is the streaming half of the method — SPOT, in the terms of [1]_. The fence is not a fixed verdict on a series you already hold but a running one, refitted as the tail fills in, so a monitor can be started from a week of history and then left to run.
Three things can happen to a score, and which one is the whole algorithm. A score beyond the fence is flagged and then discarded — not only from the excesses but from the count of observations too: an anomaly is not evidence about how the normal tail behaves, and letting it into either would move the fence on the strength of a point just declared not to belong. A score inside the fence but above the tail start joins the excesses and the fence is refitted from them. Anything else only increases the count, which lowers the fence slightly, since the same excesses now represent a rarer event.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
value
|
float
|
One score, on the same scale as the scores this was fitted on.
|
required |
Returns:
| Type | Description |
|---|---|
float
|
|
Raises:
| Type | Description |
|---|---|
RuntimeError
|
The threshold has not been fitted, so there is no tail to extend; or it was fitted on a frame, so it holds one fence per column and there is no single tail to extend. |
See Also
update_many : The same, over a batch of scores.
Notes
The label is decided against the fence as it stood before the score was absorbed, which is the only causal way round: a monitor cannot use a point to move the line it is about to judge that point against.
Refitting happens on every new excess rather than on a schedule. That is
one generalised Pareto fit per excess, so for a default level of 0.98
roughly one fit per fifty observations — cheap next to the interval
between samples in anything this is meant for, and it keeps the fence a
function of the data rather than of when the last refit happened.
Examples:
A thousand scores from the distribution it was fitted on, streamed past a fence asking for one alert in a thousand:
>>> import numpy as np
>>> from hazure import TimeSeries
>>> rng = np.random.default_rng(1)
>>> time = np.arange(4000, dtype=np.int64) * 60_000_000_000
>>> history = TimeSeries.from_arrays(time, rng.exponential(size=4000))
>>> threshold = PotThreshold(high=1e-3).fit(history)
>>> labels = threshold.update_many(rng.exponential(size=1000))
>>> int(labels.sum())
2
A genuine excursion is caught, and does not drag the fence up after it:
Source code in src/hazure/thresholds/pot.py
331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 | |
update_many
¶
Absorb a batch of scores in order, returning one label each.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
values
|
Sequence[float] | NDArray[float64]
|
Scores, in the order they were observed. |
required |
Returns:
| Type | Description |
|---|---|
ndarray
|
One label per score, decided against the fence as it stood when that score arrived. |
Raises:
| Type | Description |
|---|---|
RuntimeError
|
The threshold has not been fitted, or was fitted on a frame. |
See Also
update : The same, one score at a time.
Source code in src/hazure/thresholds/pot.py
QuantileThreshold
¶
Bases: BaseThreshold
Flag scores beyond quantiles of the training scores.
The quantiles are turned into absolute cut-offs at :meth:fit time, so the
line is drawn by history and then held fixed: a later series is judged
against what used to be normal, not against itself.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
low
|
float | None
|
Lower quantile in |
None
|
high
|
float | None
|
Upper quantile in |
None
|
Attributes:
| Name | Type | Description |
|---|---|---|
low_ |
float
|
Absolute lower cut-off learned from the training scores. |
high_ |
float
|
Absolute upper cut-off learned from the training scores. |
Raises:
| Type | Description |
|---|---|
ValueError
|
Both quantiles are None, or one falls outside |
Examples:
>>> import numpy as np
>>> from hazure import TimeSeries
>>> time = np.arange("2024-01-01", "2024-01-06", dtype="datetime64[D]")
>>> ts = TimeSeries.from_arrays(time, [1.0, 2.0, 3.0, 4.0, 100.0])
>>> threshold = QuantileThreshold(high=0.9).fit(ts)
>>> round(threshold.high_, 1)
61.6
>>> threshold.run(ts).values.ravel()
array([0., 0., 0., 0., 1.])
Source code in src/hazure/thresholds/quantile.py
hazure.features¶
Transformers: series in, series out. These sit upstream of a scorer.
features
¶
Feature engineering: series in, series out.
A transformer turns a series into the series an anomaly detector should actually
look at — a rolling median, a lag matrix, a seasonal residual, a principal
component. Because the output is a TimeSeries like the input, transformers
chain with each other and sit upstream of scorers in a pipeline.
Univariate transformers handle one series at a time and fan out automatically over the columns of a frame. Multivariate ones need every column at once, which is what lets them find a point that is unremarkable in each column on its own yet impossible taken together.
CustomizedTransformer
¶
CustomizedTransformer(
transform_func: Callable[..., Any],
transform_func_params: dict[str, Any] | None = None,
fit_func: Callable[..., Any] | None = None,
fit_func_params: dict[str, Any] | None = None,
)
Bases: BaseTransformer
Wrap user functions into a transformer.
Lets a one-off calculation join a pipeline, be cloned, and be grid-searched, without subclassing anything.
The contract is numpy in, numpy out:
transform_func(values, **params)receives the value matrix as afloat64array of shape(n_rows, n_columns), with missing observations asNaNand columns in the order training saw them. It must return an array of shape(n_rows,)or(n_rows, m); the row count may not change, because the time axis is reused.paramsistransform_func_paramsmerged over whateverfit_funcreturned, so an explicit parameter wins over a learned one of the same name.fit_func(values, **fit_func_params)receives the same kind of array and must return adictof keyword arguments fortransform_func. When it is None there is nothing to learn and the transformer may be used without calling :meth:fit.
Output columns take the input's names when the width is unchanged, and are
named value or value_0, value_1, ... otherwise.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
transform_func
|
Callable[..., Any]
|
The transformation, as described above. |
required |
transform_func_params
|
dict[str, Any] | None
|
Extra keyword arguments for |
None
|
fit_func
|
Callable[..., Any] | None
|
Optional training step, as described above. |
None
|
fit_func_params
|
dict[str, Any] | None
|
Extra keyword arguments for |
None
|
Attributes:
| Name | Type | Description |
|---|---|---|
learned_params_ |
dict
|
What |
Examples:
>>> import numpy as np
>>> from hazure import TimeSeries
>>> ts = TimeSeries.from_arrays(
... np.arange("2024-01-01", "2024-01-04", dtype="datetime64[D]"),
... np.array([[1.0, 2.0], [3.0, 4.0], [5.0, 6.0]]),
... ["a", "b"],
... )
>>> spread = CustomizedTransformer(
... transform_func=lambda values: values.max(axis=1) - values.min(axis=1)
... )
>>> spread.transform(ts).values.ravel()
array([1., 1., 1.])
Source code in src/hazure/features/customized_transformer.py
DoubleRollingAggregate
¶
DoubleRollingAggregate(
window: Window | tuple[Window, Window],
agg: str | tuple[str, str] = "mean",
agg_params: dict[str, Any]
| tuple[Any, Any]
| None = None,
center: bool = True,
min_periods: int
| tuple[int | None, int | None]
| None = None,
diff: Diff = "l1",
)
Bases: BaseTransformer
Compare the window before each point with the window after it.
This is the primitive behind spike, level-shift and volatility-shift detection: summarise the recent past and the near future separately, then measure the gap. A large gap means the series changed character there.
window, agg, agg_params and min_periods each accept a
2-tuple to configure the two sides independently. Asymmetric settings are
what separate the use cases — a long left window characterises "normal",
a short right window catches a blip.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
window
|
Window | tuple[Window, Window]
|
One spec for both sides, or |
required |
agg
|
str | tuple[str, str]
|
One statistic for both sides, or |
'mean'
|
agg_params
|
dict[str, Any] | tuple[Any, Any] | None
|
Extra arguments for the aggregation, or |
None
|
center
|
bool
|
When True (the default) the row sits on the boundary between the two windows, so a change is reported at the observation where it happens. When False the result is reported at the last observation of the right window, which is the earliest point at which a trailing-only detector could have known about it. |
True
|
min_periods
|
int | tuple[int | None, int | None] | None
|
Minimum non-missing observations per side, or |
None
|
diff
|
Diff
|
How to compare the two sides: |
'l1'
|
Examples:
A step change shows up as a peak at the step:
>>> import numpy as np
>>> from hazure import TimeSeries
>>> ts = TimeSeries.from_arrays(
... np.arange("2024-01-01", "2024-01-07", dtype="datetime64[D]"),
... np.array([0.0, 0.0, 0.0, 5.0, 5.0, 5.0]),
... )
>>> DoubleRollingAggregate(window=2, diff="diff").run(ts).values.ravel()
array([nan, nan, 2.5, 5. , 2.5, nan])
Source code in src/hazure/features/double_rolling_aggregate.py
OrdinaryLeastSquares
¶
Bases: Configurable
Least-squares linear regression with an intercept.
The default regressor for :class:RegressionResidual. Solved with
:func:numpy.linalg.lstsq, which handles a rank-deficient design matrix by
returning the minimum-norm solution rather than failing.
It takes no parameters, and is a :class:~hazure._core.config.Configurable
only so that a fitted regression can be serialised: a component's fitted state
has to be representable for :meth:~hazure.Component.to_dict to describe it,
and the regressor is part of that state. A regressor of your own does not have
to be one, and a serialisable one is the only kind that can be stored rather
than refitted.
Attributes:
| Name | Type | Description |
|---|---|---|
intercept_ |
float
|
Fitted constant term. |
coefficients_ |
ndarray
|
One fitted slope per feature column. |
Examples:
>>> import numpy as np
>>> X = np.array([[0.0], [1.0], [2.0]])
>>> model = OrdinaryLeastSquares().fit(X, np.array([1.0, 3.0, 5.0]))
>>> round(model.intercept_, 12), model.coefficients_.round(12)
(1.0, array([2.]))
fit
¶
fit(
X: NDArray[float64], y: NDArray[float64]
) -> OrdinaryLeastSquares
Fit the model and return self, for chaining.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
X
|
NDArray[float64]
|
Design matrix of shape |
required |
y
|
NDArray[float64]
|
Target of shape |
required |
Returns:
| Type | Description |
|---|---|
OrdinaryLeastSquares
|
This model. |
Source code in src/hazure/features/ordinary_least_squares.py
predict
¶
Predict the target for each row of X.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
X
|
NDArray[float64]
|
Design matrix of shape |
required |
Returns:
| Type | Description |
|---|---|
ndarray
|
Predictions of shape |
Source code in src/hazure/features/ordinary_least_squares.py
PcaColumnError
¶
Bases: _PcaBase
Split the reconstruction error across the columns that make it up.
The output keeps the input's column names, one column per input column, each
holding that column's own squared residual (x[t, j] - x_hat[t, j]) ** 2.
A squared Euclidean distance is a sum over coordinates,
||x - x_hat||**2 == sum over j of (x[t, j] - x_hat[t, j])**2
so the row sums are :class:PcaReconstructionError exactly, to floating
point. That identity is the reason to have this: it is not an attribution
heuristic bolted onto the score, it is the score's own terms written out, so
the shares are exhaustive and a flag raised by :class:hazure.PcaDetector
can be taken apart without a second model that might disagree with the first.
What a large share does not establish is which column is at fault. The residual is the part of the point the fitted subspace had no room for, and the subspace places a point using every column at once. When two columns that normally move together disagree, the disagreement lands on both, divided by the geometry of the subspace rather than by blame; which of the two actually moved is not in the data. Deciding that needs a third measurement the pair can be checked against, and PCA was not given one.
Nor are the columns on a common footing. The SVD is of the covariance and not
the correlation — the training rows are centred but never scaled — so a column
that varies in the thousands carries more absolute squared error than one that
varies in the ones, whatever either is doing. Where the columns carry
different units, comparing their shares compares the units as much as the
behaviour: put :class:StandardScale upstream first, and the shares become
comparable.
Rows with any missing value are excluded from fitting and are NaN across every output column, as for the other PCA views. A point is either placed in the subspace or it is not; there is no partial residual to divide up.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
k
|
int
|
Number of principal components to keep. |
1
|
Examples:
Two columns that normally agree, one of which breaks for two rows, plus a
third column unrelated to either. Three columns on a plane, so k=2:
>>> import numpy as np
>>> from hazure import TimeSeries
>>> from hazure.features import PcaReconstructionError
>>> a = np.arange(12.0)
>>> b = a.copy()
>>> b[6:8] += 6.0
>>> c = np.tile([0.0, 9.0, 3.0, 12.0], 3)
>>> ts = TimeSeries.from_arrays(
... np.arange("2024-01-01", "2024-01-13", dtype="datetime64[D]"),
... np.column_stack([a, b, c]),
... ["a", "b", "c"],
... )
>>> parts = PcaColumnError(k=2).fit_transform(ts)
>>> parts.columns
('a', 'b', 'c')
>>> parts.values[6:8].round(2)
array([[5.62, 3.33, 0. ],
[4.6 , 2.73, 0. ]])
The unrelated column is clean and the broken pair holds everything — but note
where inside the pair it went: the larger share is on a, which never
moved. The decomposition localises the relationship that failed, not the
column that failed it.
And the rows add back up to the score they came from:
>>> total = PcaReconstructionError(k=2).fit_transform(ts)
>>> bool(np.allclose(parts.values.sum(axis=1), total.values.ravel()))
True
Source code in src/hazure/features/pca_base.py
PcaProjection
¶
Bases: _PcaBase
Project each point onto the leading principal components.
Emits pc0 .. pc{k-1}, the coordinates of each time point in the
subspace the training data occupies. Rows with any missing value are
missing throughout.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
k
|
int
|
Number of principal components to keep. |
1
|
Examples:
Two perfectly correlated columns vary along one direction only:
>>> import numpy as np
>>> from hazure import TimeSeries
>>> base = np.array([0.0, 1.0, 2.0, 3.0])
>>> ts = TimeSeries.from_arrays(
... np.arange("2024-01-01", "2024-01-05", dtype="datetime64[D]"),
... np.column_stack([base, 2.0 * base]),
... ["a", "b"],
... )
>>> PcaProjection(k=1).fit_transform(ts).values.ravel().round(6)
array([-3.354102, -1.118034, 1.118034, 3.354102])
Source code in src/hazure/features/pca_base.py
PcaReconstruction
¶
Bases: _PcaBase
Rebuild each point from the leading principal components only.
The output keeps the input's column names: it is the input as the model
believes it should have looked, with everything outside the k-dimensional
subspace discarded. Rows with any missing value are missing throughout.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
k
|
int
|
Number of principal components to keep. |
1
|
Source code in src/hazure/features/pca_base.py
PcaReconstructionError
¶
Bases: _PcaBase
Measure how far each point lies from the principal subspace.
The output, named error, is the squared Euclidean distance between a
point and its reconstruction — the sum over columns of the squared residual,
not its square root. Squaring keeps it a sum of per-column contributions and
avoids a square root that no threshold needs.
A point can be unremarkable in every column and still have a large error, because the error measures whether the columns agree with each other.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
k
|
int
|
Number of principal components to keep. |
1
|
Examples:
Data lying exactly on a line is reconstructed exactly:
>>> import numpy as np
>>> from hazure import TimeSeries
>>> base = np.array([0.0, 1.0, 2.0, 3.0])
>>> ts = TimeSeries.from_arrays(
... np.arange("2024-01-01", "2024-01-05", dtype="datetime64[D]"),
... np.column_stack([base, 2.0 * base + 1.0]),
... ["a", "b"],
... )
>>> PcaReconstructionError(k=1).fit_transform(ts).values.ravel().round(12)
array([0., 0., 0., 0.])
Source code in src/hazure/features/pca_base.py
RegressionResidual
¶
RegressionResidual(
target: str, regressor: Regressor | None = None
)
Bases: BaseTransformer
Predict one column from the others and return the residual.
Where the columns are physically linked — a flow and the pressure it causes, a control and its outcome — the regression captures the link and the residual is what the link fails to explain. A large residual means the relationship broke, which no single column would show.
Rows with any missing value are dropped from training and yield a missing
residual. The output column is named residual.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
target
|
str
|
Name of the column to predict. Every other column is a feature. |
required |
regressor
|
Regressor | None
|
Any object with |
None
|
Attributes:
| Name | Type | Description |
|---|---|---|
regressor_ |
Regressor
|
The fitted regressor, which is |
features_ |
tuple of str
|
Feature column names, in the order the regressor was fitted with. |
Examples:
>>> import numpy as np
>>> from hazure import TimeSeries
>>> drive = np.arange(6.0)
>>> follow = 2.0 * drive + 1.0
>>> follow[4] += 10.0
>>> ts = TimeSeries.from_arrays(
... np.arange("2024-01-01", "2024-01-07", dtype="datetime64[D]"),
... np.column_stack([drive, follow]),
... ["drive", "follow"],
... )
>>> residual = RegressionResidual(target="follow").fit_transform(ts)
>>> int(np.argmax(np.abs(residual.values)))
4
Source code in src/hazure/features/regression_residual.py
Regressor
¶
Bases: Protocol
The interface :class:RegressionResidual needs from a regressor.
Deliberately the same two methods scikit-learn estimators expose, so one can
be passed straight in, but structural: any object with a matching fit and
predict works and no import is implied.
fit
¶
Retrospect
¶
Bases: BaseTransformer
Emit lagged copies of the series as columns.
The result is the design matrix for autoregression: row t holds the
values at t - till, t - till - step_size, and so on, which is what a
model needs to learn how a control's effect is delayed and how long it
lasts. Columns are named t-0, t-1, ...; a negative lag looks ahead
and is named t+1, t+2, ...
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
n_steps
|
int
|
Number of lagged columns. |
1
|
step_size
|
int
|
Gap in observations between consecutive columns. |
1
|
till
|
int
|
Nearest lag, in observations. 0 is the current point. |
0
|
Examples:
>>> import numpy as np
>>> from hazure import TimeSeries
>>> ts = TimeSeries.from_arrays(
... np.arange("2024-01-01", "2024-01-05", dtype="datetime64[D]"),
... np.array([0.0, 1.0, 2.0, 3.0]),
... )
>>> lagged = Retrospect(n_steps=2, step_size=2, till=1).run(ts)
>>> lagged.columns
('t-1', 't-3')
>>> lagged.values
array([[nan, nan],
[ 0., nan],
[ 1., nan],
[ 2., 0.]])
Source code in src/hazure/features/retrospect.py
RollingAggregate
¶
RollingAggregate(
window: Window,
agg: str = "mean",
agg_params: dict[str, Any] | None = None,
center: bool = False,
min_periods: int | None = None,
closed: Closed | None = None,
)
Bases: BaseTransformer
Aggregate a sliding window into a new series.
A rolling statistic is the workhorse feature of time series anomaly detection: comparing a point with a summary of its neighbourhood is what turns "3.4 volts" into "3.4 volts, where the last hour averaged 1.1".
Most aggregations return one number per window and therefore one column.
Two return a vector and so widen the series: "quantile" with a list of
quantiles, and "hist".
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
window
|
Window
|
Observations ( |
required |
agg
|
str
|
A name from :data: |
'mean'
|
agg_params
|
dict[str, Any] | None
|
Extra arguments for the aggregation:
|
None
|
center
|
bool
|
Centre the window on each row instead of trailing it. |
False
|
min_periods
|
int | None
|
Minimum non-missing observations for a result. Defaults to the full window for integer windows and to 1 for duration windows. |
None
|
closed
|
Closed | None
|
Which window endpoints to include: |
None
|
Examples:
>>> import numpy as np
>>> from hazure import TimeSeries
>>> ts = TimeSeries.from_arrays(
... np.arange("2024-01-01", "2024-01-06", dtype="datetime64[D]"),
... np.array([1.0, 2.0, 3.0, 4.0, 5.0]),
... )
>>> RollingAggregate(window=2).run(ts).values.ravel()
array([nan, 1.5, 2.5, 3.5, 4.5])
A list of quantiles fans out into one column per quantile:
>>> RollingAggregate(
... window=3, agg="quantile", agg_params={"q": [0.0, 1.0]}
... ).run(ts).columns
('q0.0', 'q1.0')
Source code in src/hazure/features/rolling_aggregate.py
SeasonalDecomposition
¶
SeasonalDecomposition(
period: int | None = None,
trend: bool = False,
component: SeasonalComponent = "residual",
)
Bases: BaseTransformer
Split a series into trend, a repeating seasonal profile, and residual.
Additive classic decomposition, in numpy alone: the trend is a centred moving average one period wide, and the seasonal profile is the mean of each phase of the detrended series. The residual is what neither explains, which is usually the series an anomaly detector should look at.
:meth:fit learns the period (unless given) and the seasonal profile;
:meth:transform re-estimates the trend of the series it is given but
reuses the trained profile, which assumes seasonality is stable over time.
The profile is anchored to the first training timestamp, and a later series' phase is recovered arithmetically from its timestamps, so a test window arbitrarily far from training costs the same as an adjacent one.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
period
|
int | None
|
Length of a cycle in observations. When None it is detected from the autocorrelation of the training series. |
None
|
trend
|
bool
|
Estimate and remove a moving-average trend. When False the series is taken to be a seasonal profile plus residual, and the profile carries the series' level. |
False
|
component
|
SeasonalComponent
|
Which part to return: |
'residual'
|
Attributes:
| Name | Type | Description |
|---|---|---|
period_ |
int
|
Cycle length used, whether given or detected. |
seasonal_ |
ndarray
|
The seasonal profile, of length |
Raises:
| Type | Description |
|---|---|
ValueError
|
The time axis is irregular, the period is unusable, no seasonality
could be detected, |
Examples:
>>> import numpy as np
>>> from hazure import TimeSeries
>>> profile = np.array([0.0, 1.0, 0.0, -1.0])
>>> ts = TimeSeries.from_arrays(
... np.arange("2024-01-01", "2024-01-13", dtype="datetime64[D]"),
... np.tile(profile, 3),
... )
>>> model = SeasonalDecomposition(period=4).fit(ts)
>>> model.seasonal_
array([ 0., 1., 0., -1.])
>>> model.run(ts).values.ravel()
array([0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.])
Source code in src/hazure/features/seasonal_decomposition.py
StandardScale
¶
Bases: BaseTransformer
Centre and scale a series by its own mean and standard deviation.
Scaling is per-series and computed from the series being transformed, so a
frame of columns in different units becomes comparable without training.
The standard deviation is the sample one (ddof=1); a constant series has
none, and is centred but left unscaled rather than divided by zero.
Source code in src/hazure/features/standard_scale.py
SumAll
¶
Bases: BaseTransformer
Sum every column into one series, propagating missing values.
A row is only summable if every column contributed to it, so a single
missing observation makes the total missing rather than quietly smaller.
The output column is named sum.
Examples:
>>> import numpy as np
>>> from hazure import TimeSeries
>>> ts = TimeSeries.from_arrays(
... np.arange("2024-01-01", "2024-01-04", dtype="datetime64[D]"),
... np.array([[1.0, 2.0], [3.0, np.nan], [5.0, 6.0]]),
... ["a", "b"],
... )
>>> SumAll().run(ts).values.ravel()
array([ 3., nan, 11.])
Source code in src/hazure/features/sum_all.py
hazure.ensemble¶
Aggregators, for combining several detectors' verdicts.
ensemble
¶
Ensembling: several label series in, one out.
An aggregator reduces the outputs of several detectors to a single label series,
which is how multi-condition rules and detector ensembles are expressed. Every
aggregator emits one column named anomaly.
Three-valued logic
A label is 1.0 anomalous, 0.0 normal or NaN unknown, all in a float
column. NaN is a real third state, not a missing 0: a rolling detector cannot
label the first few points of a series, and saying "unknown" there is different
from saying "normal". Combining labels is therefore three-valued logic, and a
result is only known once the inputs settle it:
===== ===== ===== =====
a b OR AND
===== ===== ===== =====
1 1 1 1
1 0 1 0
1 NaN 1 NaN
0 1 1 0
0 0 0 0
0 NaN NaN 0
NaN 1 1 NaN
NaN 0 NaN 0
NaN NaN NaN NaN
===== ===== ===== =====
Read the columns as: OR is 1 as soon as one input says 1, whatever the others say; it is 0 only when every input is a definite 0; otherwise the unknown input could have gone either way, so the answer is unknown. AND is the mirror image — 0 as soon as one input says 0, 1 only when every input is a definite 1.
:class:VoteAggregator generalises both by counting rather than quantifying:
unknown inputs leave the vote entirely, contributing to neither the numerator nor
the denominator, and a row where nothing is known is unknown.
Any non-zero label counts as anomalous, so a detector that emits counts or confidences rather than a strict 0/1 still aggregates sensibly.
Combining scores instead
Labels are not the only thing worth combining. :class:ScoreAggregator takes the
scores themselves, before any threshold has turned them into verdicts, so the
ensemble keeps the ranking that binarising throws away — and, because raw scores
from different scorers share no yardstick, it normalises each input first.
Unknowns abstain there too.
AndAggregator
¶
Bases: BaseAggregator
Flag a point only where every input agrees.
The intersection of the inputs: use it to demand corroboration, so that a point counts only when several independent tests object to it. One definite "normal" settles the row; short of that, an unknown input leaves the answer unknown.
Examples:
>>> import numpy as np
>>> from hazure import TimeSeries
>>> ts = TimeSeries.from_arrays(
... np.arange("2024-01-01", "2024-01-05", dtype="datetime64[D]"),
... np.array([[1.0, 1.0], [1.0, 0.0], [0.0, np.nan], [np.nan, 1.0]]),
... ["a", "b"],
... )
>>> AndAggregator().aggregate(ts)["anomaly"].tolist()
[1.0, 0.0, 0.0, nan]
Source code in src/hazure/ensemble/and_aggregator.py
CustomizedAggregator
¶
CustomizedAggregator(
aggregate_func: Callable[..., Any],
aggregate_func_params: dict[str, Any] | None = None,
)
Bases: BaseAggregator
Wrap a user function into an aggregator.
The contract is numpy in, numpy out: aggregate_func(labels, **params)
receives a float64 array of shape (n_rows, n_inputs) whose columns
are the inputs in the order they were passed, with unknown labels as
NaN, and must return an array of shape (n_rows,) — or
(n_rows, 1), which is flattened. Returning NaN for a row means the
combination is unknown there.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
aggregate_func
|
Callable[..., Any]
|
The combining function, as described above. |
required |
aggregate_func_params
|
dict[str, Any] | None
|
Extra keyword arguments for |
None
|
Raises:
| Type | Description |
|---|---|
ValueError
|
The function returned something other than one value per row. |
Examples:
A weighted vote, trusting the first input twice as much as the second:
>>> import numpy as np
>>> from hazure import TimeSeries
>>> ts = TimeSeries.from_arrays(
... np.arange("2024-01-01", "2024-01-04", dtype="datetime64[D]"),
... np.array([[1.0, 0.0], [0.0, 1.0], [0.0, 0.0]]),
... ["a", "b"],
... )
>>> weighted = CustomizedAggregator(
... aggregate_func=lambda labels, w: (labels @ w >= 0.5).astype(float),
... aggregate_func_params={"w": np.array([0.67, 0.33])},
... )
>>> weighted.aggregate(ts)["anomaly"].tolist()
[1.0, 0.0, 0.0]
Source code in src/hazure/ensemble/customized_aggregator.py
OrAggregator
¶
Bases: BaseAggregator
Flag a point that any input flags.
The union of the inputs: use it to collect several kinds of anomaly — a spike detector, a level-shift detector, a range check — into one label series. Unknown inputs only matter when nothing else has already flagged the point.
Examples:
>>> import numpy as np
>>> from hazure import TimeSeries
>>> ts = TimeSeries.from_arrays(
... np.arange("2024-01-01", "2024-01-05", dtype="datetime64[D]"),
... np.array([[1.0, 1.0], [1.0, 0.0], [0.0, np.nan], [np.nan, 1.0]]),
... ["a", "b"],
... )
>>> OrAggregator().aggregate(ts)["anomaly"].tolist()
[1.0, 1.0, nan, 1.0]
Source code in src/hazure/ensemble/or_aggregator.py
ScoreAggregator
¶
Bases: BaseAggregator
Combine several anomaly scores into one score.
The label aggregators reconcile verdicts, which means each input has already been reduced to yes or no. That is a lossy step to take first: a point two scorers nearly flagged becomes indistinguishable from one neither came close to flagging, and the ranking — the thing an operator actually works down, worst first — is gone. Combining the scores keeps it, which is why score-level ensembling is the usual way anomaly-detection ensembles are built.
What makes it harder than combining labels is that scores are not
commensurable. A reconstruction error in squared units, a robust z-score and
a rolling-quantile exceedance have no common yardstick, so averaging them raw
is not an average of opinions but a vote weighted by whichever scorer happens
to emit the largest numbers. normalize is the answer to that, and is the
parameter worth thinking about.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
how
|
How
|
How one row of normalised scores is reduced across the inputs.
|
'mean'
|
normalize
|
Normalize
|
How each input column is put on a comparable scale before combining.
|
'rank'
|
Raises:
| Type | Description |
|---|---|
ValueError
|
|
Notes
Nothing is learned, so :attr:trainable is False and there is no fit.
The normalisation is recomputed from the very series being combined, as
:class:hazure.StandardScale does, rather than fitted on a training period.
For "rank" that is not a shortcut but the definition: a rank exists only
relative to a sample, and there is no sample here but the one in hand.
"robust" could honestly be fitted, and if a fixed yardstick is what you
want, put a :class:hazure.DeviationScorer in front of each input and
combine with normalize="none".
NaN means unknown and abstains rather than propagating, exactly as it
does in :class:VoteAggregator: a row with two known scores and one unknown
is the combination of the two, because a detector still inside its warm-up
window has no opinion to contribute and should not be read as a quiet one.
A row where every input is unknown is NaN.
A column whose median absolute deviation is zero has no observed spread to
divide by — a mostly constant score with a few excursions is enough to do
that. Under "robust" such a column is centred and left unscaled, which
is what :class:hazure.StandardScale does with a constant series: values on
the median contribute exactly 0.0, and the excursions stay finite and
ordered instead of becoming an infinity that would swallow every other
input's contribution to a mean.
Examples:
Two scorers whose scores differ by three orders of magnitude. Ranking makes the units irrelevant, so the expensive scorer's larger numbers do not decide the outcome on their own:
>>> import numpy as np
>>> from hazure import TimeSeries
>>> ts = TimeSeries.from_arrays(
... np.arange("2024-01-01", "2024-01-05", dtype="datetime64[D]"),
... np.array([[0.1, 300.0], [0.2, 100.0], [0.9, 200.0], [0.3, np.nan]]),
... ["cheap", "expensive"],
... )
>>> averaged = ScoreAggregator().aggregate(ts)["anomaly"]
>>> [round(float(value), 3) for value in averaged]
[0.5, 0.167, 0.75, 0.667]
The last row's second input is unknown, so it is the first input alone. Taking the maximum instead lets whichever input is loudest carry the row:
>>> loudest = ScoreAggregator(how="max").aggregate(ts)["anomaly"]
>>> [round(float(value), 3) for value in loudest]
[1.0, 0.333, 1.0, 0.667]
Source code in src/hazure/ensemble/score_aggregator.py
VoteAggregator
¶
Bases: BaseAggregator
Flag a point when enough of the inputs do.
The middle ground between :class:OrAggregator and :class:AndAggregator,
and the natural way to ensemble many detectors of similar quality: one
detector's false positive is outvoted, while a genuine anomaly that most of
them see survives.
Unknown inputs abstain rather than vote against, so a detector that cannot label the start of a series does not drag the fraction down there.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
threshold
|
float
|
Fraction of the known inputs that must flag a point, from 0 to 1 inclusive. The default of 0.5 is a simple majority, counting a tie as anomalous. At 1 every known input must agree; at 0 every row with any known label is flagged. |
0.5
|
Raises:
| Type | Description |
|---|---|
ValueError
|
|
Examples:
>>> import numpy as np
>>> from hazure import TimeSeries
>>> ts = TimeSeries.from_arrays(
... np.arange("2024-01-01", "2024-01-04", dtype="datetime64[D]"),
... np.array([[1.0, 1.0, 0.0], [1.0, 0.0, 0.0], [1.0, np.nan, np.nan]]),
... ["a", "b", "c"],
... )
>>> VoteAggregator(threshold=0.5).aggregate(ts)["anomaly"].tolist()
[1.0, 0.0, 1.0]
Source code in src/hazure/ensemble/vote_aggregator.py
hazure.compose¶
Chaining and wiring components into models.
compose
¶
Combining components into models.
A single detector rarely captures a real-world notion of "anomalous". More often the answer is a small graph: extract a feature, score it, threshold it, and require two independent signals to agree before reporting anything.
Two shapes cover almost everything:
- :class:
Pipelinefor a straight chain — the common case, and the one worth keeping legible. - :class:
Graphfor anything branching or converging, expressed as named nodes with named inputs.
Both are themselves components, so a pipeline can be a node of a graph, and either can be handed to anything that accepts a detector.
Wiring is checked once, when the structure is first used, and the resolved
execution order is cached until the structure changes. Intermediate results stay
as :class:~hazure.TimeSeries throughout, so a chain of ten steps performs one
conversion in and one out.
Pipeline
¶
Pipeline(steps: Sequence[tuple[str, Component]])
Bases: _Composite
A straight chain of components, each fed by the previous one.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
steps
|
Sequence[tuple[str, Component]]
|
|
required |
Raises:
| Type | Description |
|---|---|
ValueError
|
|
Examples:
Chain a feature, a score and a cut-off, then use it as one detector:
>>> from hazure import Pipeline
>>> model = Pipeline(
... [
... ("deseasonalise", SeasonalDecomposition(period=24)),
... ("score", DeviationScorer()),
... ("cut", IqrThreshold(factor=3.0)),
... ]
... )
>>> anomalies = model.fit_detect(series)
Source code in src/hazure/compose.py
named_steps
¶
named_steps() -> dict[str, Component | BaseAggregator]
clone
¶
clone() -> Pipeline
Return an unfitted copy whose steps are themselves fresh clones.
Returns:
| Type | Description |
|---|---|
Pipeline
|
A fresh, unfitted pipeline. |
Source code in src/hazure/compose.py
summary
¶
Return a plain-text description of the chain.
Returns:
| Type | Description |
|---|---|
str
|
One line per step, in execution order. |
Source code in src/hazure/compose.py
to_mermaid
¶
Return the chain as a Mermaid flowchart.
Text rather than a drawing, so it renders in a README, a pull request or a notebook without pulling in a plotting dependency.
Returns:
| Type | Description |
|---|---|
str
|
Mermaid |
Source code in src/hazure/compose.py
Node
dataclass
¶
Node(
name: str,
model: Component | BaseAggregator,
inputs: tuple[str, ...] = (SOURCE,),
columns: tuple[Sequence[str] | None, ...] | None = None,
)
One vertex of a :class:Graph.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
name
|
str
|
Unique label for this node. |
required |
model
|
Component | BaseAggregator
|
The component to run here. |
required |
inputs
|
tuple[str, ...]
|
Names of the nodes feeding this one, or |
(SOURCE,)
|
columns
|
tuple[Sequence[str] | None, ...] | None
|
Optional per-input column selection, same length as |
None
|
Graph
¶
Bases: _Composite
Components wired as a directed acyclic graph.
Use this when the model branches or converges — scoring one series two ways
and requiring both to agree, or feeding one transformed series to several
detectors. For a straight chain, :class:Pipeline reads better.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
nodes
|
Iterable[Node] | Mapping[str, Node]
|
The vertices. Order is irrelevant; execution order is derived from the
wiring. A mapping of |
required |
Raises:
| Type | Description |
|---|---|
ValueError
|
Names repeat, a named input does not exist, the graph has a cycle, or more than one node is left unconsumed so the output would be ambiguous. |
Examples:
Require two independent signals to agree:
>>> from hazure import Graph, Node
>>> model = Graph(
... [
... Node("spike", SpikeDetector(window=5)),
... Node("shift", LevelShiftDetector(window=10)),
... Node("both", AndAggregator(), inputs=("spike", "shift")),
... ]
... )
>>> anomalies = model.fit_detect(series)
Source code in src/hazure/compose.py
terminal
property
¶
terminal: Component | BaseAggregator
The component whose output is the graph's output.
named_nodes
¶
named_nodes() -> dict[str, Component | BaseAggregator]
Return the nodes as a mapping, for reaching a fitted sub-component.
clone
¶
clone() -> Graph
Return an unfitted copy whose nodes hold themselves fresh clones.
Returns:
| Type | Description |
|---|---|
Graph
|
A fresh, unfitted graph. |
Source code in src/hazure/compose.py
trace
¶
Run the graph and return every node's output, not just the last.
Useful for working out which branch flagged what, or for plotting the intermediate series while tuning.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
data
|
Any
|
Any supported dataframe, series, or :class: |
required |
Returns:
| Type | Description |
|---|---|
dict
|
Node name mapped to that node's output in native form, plus
|
Raises:
| Type | Description |
|---|---|
RuntimeError
|
The graph has not been fitted. |
Source code in src/hazure/compose.py
summary
¶
Return a plain-text description of the wiring, in execution order.
Returns:
| Type | Description |
|---|---|
str
|
One line per node. |
Source code in src/hazure/compose.py
to_mermaid
¶
Return the wiring as a Mermaid flowchart.
Text rather than a drawing, so it renders in a README, a pull request or a notebook without pulling in a plotting dependency.
Returns:
| Type | Description |
|---|---|
str
|
Mermaid |
Source code in src/hazure/compose.py
hazure.events¶
Moving between one-label-per-sample and the anomalous-interval view of the same thing.
events
¶
Working with labels and anomalous intervals.
Two jobs live here, both about the shape of the data rather than about detection:
- :class:
Eventsand the converters :func:to_events/ :func:to_labels, which move between the one-label-per-sample and the anomalous-interval view of the same thing, plus :func:expand_events, which widens intervals by a margin. - :func:
validate_series, which applies the normalisation every detector does internally and hands the result back, so it can be inspected rather than guessed at.
Events
dataclass
¶
A sorted, disjoint set of closed anomalous time intervals.
Construct with :meth:from_bounds, :meth:from_any or :meth:empty.
Calling Events(...) directly skips validation and is only for code that
already holds sorted, merged bounds.
Attributes:
| Name | Type | Description |
|---|---|---|
bounds |
NDArray[int64]
|
Shape |
origin |
Origin
|
Provenance borrowed from the series the events came from, so
:meth: |
Examples:
>>> events = Events.from_any([("2024-01-01T00:00", "2024-01-01T06:00")])
>>> events
Events([2024-01-01T00:00:00..2024-01-01T06:00:00])
>>> events.n_events
1
durations
property
¶
Length of each event in nanoseconds, end - start + 1.
Inclusive integer bounds are half-open in effect: [start, end] in
whole nanoseconds occupies [start, end + 1) of continuous time, so
the + 1 counts the end nanosecond rather than being an off-by-one.
Two things follow, and both are what makes the event-based metrics come
out to round numbers. An instantaneous event lasts 1 ns, not 0. An event
covering k consecutive samples of a series sampled every freq
lasts exactly k * freq, so a duration ratio and a count of samples
agree exactly.
from_bounds
classmethod
¶
from_bounds(
bounds: Any, *, origin: Origin | None = None
) -> Events
Validate, sort and merge bounds into an Events.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
bounds
|
Any
|
Anything array-like of shape |
required |
origin
|
Origin | None
|
Provenance for :meth: |
None
|
Returns:
| Type | Description |
|---|---|
Events
|
Sorted, disjoint events. Overlapping, nested and adjacent inputs are merged, so the result may be shorter than the input. |
Raises:
| Type | Description |
|---|---|
ValueError
|
The array is not 2-D with two columns, or some |
TypeError
|
The array's dtype is neither integer nor |
Examples:
Adjacent intervals collapse into one:
Source code in src/hazure/events/interval.py
from_any
classmethod
¶
from_any(
events: Any, *, origin: Origin | None = None
) -> Events
Build an Events from whatever the caller has to hand.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
events
|
Any
|
One of:
|
required |
origin
|
Origin | None
|
Provenance override. Inferred from a dataframe input, otherwise defaults to a pandas frame. |
None
|
Returns:
| Type | Description |
|---|---|
Events
|
Sorted, disjoint events. |
Raises:
| Type | Description |
|---|---|
TypeError
|
The object is not one of the shapes above, or an element is not a recognisable timestamp. |
ValueError
|
A dataframe input lacks a |
Examples:
Source code in src/hazure/events/interval.py
empty
classmethod
¶
empty(*, origin: Origin | None = None) -> Events
Return an Events holding nothing.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
origin
|
Origin | None
|
Provenance for :meth: |
None
|
Returns:
| Type | Description |
|---|---|
Events
|
An event set with |
Source code in src/hazure/events/interval.py
intersect
¶
Return the intervals covered by both this set and other.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
other
|
Events
|
Event set to intersect with. |
required |
Returns:
| Type | Description |
|---|---|
Events
|
Sorted, disjoint intersection, carrying this set's |
Raises:
| Type | Description |
|---|---|
TypeError
|
|
Examples:
>>> a = Events.from_bounds([[0, 100]])
>>> b = Events.from_bounds([[50, 200]])
>>> a.intersect(b).bounds.tolist()
[[50, 100]]
Source code in src/hazure/events/interval.py
union
¶
Return the intervals covered by either this set or other.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
other
|
Events
|
Event set to merge with. |
required |
Returns:
| Type | Description |
|---|---|
Events
|
Sorted, disjoint union, carrying this set's |
Raises:
| Type | Description |
|---|---|
TypeError
|
|
Examples:
>>> a = Events.from_bounds([[0, 100]])
>>> b = Events.from_bounds([[50, 200]])
>>> a.union(b).bounds.tolist()
[[0, 200]]
Source code in src/hazure/events/interval.py
to_list
¶
Return the events as native timestamps.
An interval becomes a (start, end) 2-tuple; an instantaneous event
becomes a bare timestamp, so a set of point anomalies reads as a plain
list of the times they happened.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
backend
|
str | None
|
Emit timestamps of this backend's flavour instead of the original
one. pandas yields |
None
|
Returns:
| Type | Description |
|---|---|
list
|
One entry per event, in order. |
Examples:
>>> Events.from_any([0, (10, 20)]).to_list(backend="pandas")
[Timestamp('1970-01-01 00:00:00'), (Timestamp('1970-01-01 00:00:00.000000010'), Timestamp('1970-01-01 00:00:00.000000020'))]
Source code in src/hazure/events/interval.py
to_frame
¶
Return the events as a two-column start/end dataframe.
Timestamps are emitted at nanosecond resolution regardless of the original series' unit: a merged period-based event ends one nanosecond short of the following sample, and a coarser unit would round that away.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
backend
|
str | None
|
Emit into this backend instead of the original one. Events built from raw arrays have no original backend, and default to pandas here: asking for a frame is asking for a dataframe library. |
None
|
Returns:
| Type | Description |
|---|---|
Any
|
A native dataframe with columns |
Source code in src/hazure/events/interval.py
expand_events
¶
expand_events(
events: Any,
*,
before: int | str | timedelta | timedelta64 = 0,
after: int | str | timedelta | timedelta64 = 0,
) -> Events | dict[str, Events]
Widen every event by a margin, then re-merge what now overlaps.
Useful for scoring tolerantly — an alert a minute early still counts — and for turning instantaneous detections into inspectable windows.
.. warning::
A bare int is a number of nanoseconds, not seconds and not
samples. Pass a string such as "1h" or a timedelta whenever you
mean a human-scale duration.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
events
|
Any
|
An |
required |
before
|
int | str | timedelta | timedelta64
|
Margin added before each |
0
|
after
|
int | str | timedelta | timedelta64
|
Margin added after each |
0
|
Returns:
| Type | Description |
|---|---|
Events or dict of Events
|
Expanded events, re-merged. A dict input gives a dict output with the same keys. |
Raises:
| Type | Description |
|---|---|
TypeError
|
A margin is neither a duration nor an int of nanoseconds. |
ValueError
|
A margin is negative, or a duration string is unparseable. |
Examples:
>>> events = Events.from_any([("2024-01-01T06:00", "2024-01-01T07:00")])
>>> expand_events(events, before="1h")
Events([2024-01-01T05:00:00..2024-01-01T07:00:00])
Ten nanoseconds, not ten seconds:
Source code in src/hazure/events/convert.py
to_events
¶
Convert binary labels into anomalous intervals.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
labels
|
Any
|
Anything :meth: |
required |
as_periods
|
bool | None
|
Whether a labelled timestamp stands for the period it opens rather than
an instant. |
None
|
Returns:
| Type | Description |
|---|---|
Events or dict of Events
|
One |
Raises:
| Type | Description |
|---|---|
ValueError
|
|
Examples:
Two adjacent labels on hourly data are one two-hour event:
>>> import pandas as pd
>>> index = pd.date_range("2024-01-01", periods=4, freq="h")
>>> to_events(pd.Series([0, 1, 1, 0], index=index))
Events([2024-01-01T01:00:00..2024-01-01T02:59:59.999999999])
Treat each labelled sample as an instant instead:
>>> to_events(pd.Series([0, 1, 1, 0], index=index), as_periods=False)
Events([2024-01-01T01:00:00, 2024-01-01T02:00:00])
Source code in src/hazure/events/convert.py
to_labels
¶
to_labels(
events: Any,
time: Any,
*,
as_periods: bool | None = None,
backend: str | None = None,
) -> Any
Convert anomalous intervals back into binary labels on a time axis.
Inverse of :func:to_events: with matching as_periods,
to_events(to_labels(events, time)) == events for any events that
:func:to_events could have produced on this axis. Bounds that fall between
samples are snapped outwards to the samples they touch, since a label series
can only say something about the samples it has.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
events
|
Any
|
An |
required |
time
|
Any
|
The time axis to label. May be a |
required |
as_periods
|
bool | None
|
Whether a sample stands for the period it opens. |
None
|
backend
|
str | None
|
Emit into this backend instead of the one the time axis came from. |
None
|
Returns:
| Type | Description |
|---|---|
Any
|
A native label series or frame of |
Raises:
| Type | Description |
|---|---|
ValueError
|
|
Examples:
>>> import pandas as pd
>>> index = pd.date_range("2024-01-01", periods=4, freq="h")
>>> labels = pd.Series([0, 1, 1, 0], index=index)
>>> to_labels(to_events(labels), labels).tolist()
[0.0, 1.0, 1.0, 0.0]
A dict of event sets becomes one column per key:
>>> frame = to_labels({"spike": [index[0]], "shift": [index[3]]}, labels)
>>> list(frame.columns)
['spike', 'shift']
Source code in src/hazure/events/convert.py
validate_series
¶
Normalise a time series and hand it back in its own flavour.
Every detector runs this normalisation internally. Calling it yourself makes the result inspectable, so a surprising detection can be traced back to a reordered axis or a dropped duplicate rather than guessed at.
The normalisation is: read the time axis, sort it, keep the first
observation at each timestamp, and cast the value columns to float64.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
data
|
Any
|
Anything :meth: |
required |
sort
|
bool
|
Sort by time when the input is not already ordered. |
True
|
drop_duplicates
|
bool
|
Keep the first observation at each timestamp. When False, duplicated timestamps raise instead of being collapsed. |
True
|
Returns:
| Type | Description |
|---|---|
Any
|
A native object of the same flavour as |
Raises:
| Type | Description |
|---|---|
TypeError
|
|
ValueError
|
Timestamps are missing, or duplicated with |
Examples:
>>> import pandas as pd
>>> index = pd.to_datetime(["2024-01-02", "2024-01-01", "2024-01-01"])
>>> validate_series(pd.Series([3.0, 1.0, 9.0], index=index)).tolist()
[1.0, 3.0]
Source code in src/hazure/events/convert.py
hazure.evaluation¶
Point- and event-based metrics, how late each alert was, threshold-free ranking quality, and time-ordered folds to compute any of them over.
evaluation
¶
Judging a model: metrics, and the folds to compute them over.
:func:precision, :func:recall, :func:f1_score and :func:iou accept either
label series (scored per sample) or :class:~hazure.events.Events (scored per
interval), and a dict of either to score several anomaly types at once. They say
whether an outage was caught. :func:detection_delay and
:func:detection_delays say how late, which is the other half of the question
and is not visible in any of the four.
:func:average_precision and :func:roc_auc need no threshold at all. They
score a continuous score directly, by how well it ranks the anomalous samples,
which is the only way to judge a :class:~hazure.BaseScorer without also judging
the fence you put around it.
:func:split_train_test builds time-ordered folds. Shuffling is not an option for
a time series, so the four schemes it offers differ in how the training window
grows and where the test block sits.
detection_delay
¶
One summary of :func:detection_delays, over the events that were caught.
Read this next to a recall, always. The delays of undetected events are not reported here — they do not exist — so a detector that catches one outage out of fifty, quickly, posts a better delay than one that catches all fifty a little more slowly. The pair of numbers is the honest description; either one alone can be made to look good.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
y_true
|
Any
|
Ground truth, in any form :func: |
required |
y_pred
|
Any
|
Predictions, in the same form as |
required |
statistic
|
str
|
How to reduce the per-event delays: |
'mean'
|
Returns:
| Type | Description |
|---|---|
float or dict of float
|
The reduction, in nanoseconds, or one per column / dict key. |
Raises:
| Type | Description |
|---|---|
TypeError
|
The two arguments are not both point-based or both event-based. |
ValueError
|
|
See Also
detection_delays : The per-event delays this reduces.
Notes
Undetected events are dropped before the reduction rather than counted as
zero or as infinite, for the reason given in :func:detection_delays.
Examples:
Two events caught 50 ns in, one missed entirely:
>>> truth = [(0, 100), (200, 300), (400, 500)]
>>> guess = [(50, 60), (250, 260)]
>>> detection_delay(truth, guess)
50.0
The missed event does not drag the average up, so quote the recall with it:
An hourly example, read back as a duration:
>>> import numpy as np, pandas as pd
>>> index = pd.date_range("2024-01-01", periods=6, freq="h")
>>> truth = pd.Series([0, 1, 1, 1, 0, 0], index=index)
>>> guess = pd.Series([0, 0, 1, 1, 0, 0], index=index)
>>> str(np.timedelta64(int(detection_delay(truth, guess)), "ns").astype("m8[h]"))
'1 hours'
Source code in src/hazure/evaluation/delay.py
109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 | |
detection_delays
¶
Time from each true event's start to the first prediction inside it.
Precision and recall say whether an outage was caught; they never say when. This does, one figure per true event, so that a detector which finds everything six hours late stops looking perfect.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
y_true
|
Any
|
Ground truth. An |
required |
y_pred
|
Any
|
Predictions, in the same form as |
required |
Returns:
| Type | Description |
|---|---|
numpy.ndarray or dict of numpy.ndarray
|
One delay in nanoseconds per true event, in event order, as |
Raises:
| Type | Description |
|---|---|
TypeError
|
The two arguments are not both point-based or both event-based. |
ValueError
|
The two label series sit on different time axes, or the two inputs carry different column names or keys. |
See Also
detection_delay : The same delays, reduced to one number. hazure.evaluation.recall : Whether each event was caught at all.
Notes
An event nobody ever overlapped is nan, not zero and not the event's own
duration. Both of those would be lies that average into a summary; a missed
event has no delay, and :func:~hazure.evaluation.recall is the metric that
counts it.
A prediction that opens before the true event scores 0, never a negative
number. The delay measures the first moment the prediction and the event
coincide, and that moment cannot precede the event: firing early is not being
early to an outage that had not started, it is a false positive that happens
to run into one.
Label series are converted with :func:~hazure.events.to_events, so a run
of consecutive flags on a regular axis is one event rather than several, and
the delay is measured from the run's first sample.
Examples:
A three-hour outage opening at 01:00, first flagged at 03:00, is two hours late:
>>> import numpy as np, pandas as pd
>>> index = pd.date_range("2024-01-01", periods=6, freq="h")
>>> truth = pd.Series([0, 1, 1, 1, 0, 0], index=index)
>>> guess = pd.Series([0, 0, 0, 1, 0, 0], index=index)
>>> delays = detection_delays(truth, guess)
>>> delays
array([7.2e+12])
>>> str(np.timedelta64(int(delays[0]), "ns").astype("m8[m]"))
'120 minutes'
The second event here is never touched, so its delay is unknown rather than large:
Source code in src/hazure/evaluation/delay.py
f1_score
¶
f1_score(
y_true: Any,
y_pred: Any,
recall_thresh: float = 0.5,
precision_thresh: float = 0.5,
) -> float | dict[str, float]
Harmonic mean of :func:precision and :func:recall.
The two thresholds are independent because they answer different questions: how much of a true event must be covered to count as caught, and how much of an alert must be real to count as justified. A monitoring team that tolerates broad alerts but wants outages caught early will set them differently.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
y_true
|
Any
|
Ground truth, in any form :func: |
required |
y_pred
|
Any
|
Predictions, in the same form as |
required |
recall_thresh
|
float
|
Coverage threshold passed to :func: |
0.5
|
precision_thresh
|
float
|
Coverage threshold passed to :func: |
0.5
|
Returns:
| Type | Description |
|---|---|
float or dict of float
|
One score, or one per column / dict key. |
Raises:
| Type | Description |
|---|---|
TypeError
|
The two arguments are not both point-based or both event-based. |
ValueError
|
A threshold is outside |
Examples:
The one true outage is caught and the one alert is justified:
>>> import pandas as pd
>>> from hazure.events import to_events
>>> index = pd.date_range("2024-01-01", periods=5, freq="h")
>>> truth = pd.Series([1, 1, 1, 1, 0], index=index)
>>> guess = pd.Series([1, 1, 1, 0, 0], index=index)
>>> f1_score(to_events(truth), to_events(guess))
1.0
Point-based, on the same labels: recall 0.75 against precision 1.0.
Source code in src/hazure/evaluation/metrics.py
iou
¶
Intersection over union of the anomalous regions.
Unlike the three metrics above this has no threshold: it measures agreement directly, as the size of the region both inputs call anomalous over the size of the region either of them does. Point-based inputs count samples; event-based inputs measure duration.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
y_true
|
Any
|
Ground truth, in any form :func: |
required |
y_pred
|
Any
|
Predictions, in the same form as |
required |
Returns:
| Type | Description |
|---|---|
float or dict of float
|
One score in |
Raises:
| Type | Description |
|---|---|
TypeError
|
The two arguments are not both point-based or both event-based. |
ValueError
|
The two label series sit on different time axes, or the two inputs carry different column names or keys. |
Examples:
Event-based: three of the four anomalous hours agree, and the union is all four.
>>> import pandas as pd
>>> from hazure.events import to_events
>>> index = pd.date_range("2024-01-01", periods=5, freq="h")
>>> truth = pd.Series([1, 1, 1, 1, 0], index=index)
>>> guess = pd.Series([1, 1, 1, 0, 0], index=index)
>>> iou(to_events(truth), to_events(guess))
0.75
Point-based on the same labels gives the same number, because a duration ratio over period events and a count of samples are the same measurement.
Source code in src/hazure/evaluation/metrics.py
precision
¶
Fraction of detections that were real.
This is :func:recall with the arguments swapped: a predicted event counts
as correct when enough of its duration is covered by the ground truth.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
y_true
|
Any
|
Ground truth, in any form :func: |
required |
y_pred
|
Any
|
Predictions, in the same form as |
required |
thresh
|
float
|
Event-based only: the fraction of a predicted event's duration that the
ground truth must cover for it to count as correct. In |
0.5
|
Returns:
| Type | Description |
|---|---|
float or dict of float
|
One score, or one per column / dict key. |
Raises:
| Type | Description |
|---|---|
TypeError
|
The two arguments are not both point-based or both event-based. |
ValueError
|
|
Examples:
The prediction lies entirely inside the true outage, so it is fully justified:
>>> import pandas as pd
>>> from hazure.events import to_events
>>> index = pd.date_range("2024-01-01", periods=5, freq="h")
>>> truth = pd.Series([1, 1, 1, 1, 0], index=index)
>>> guess = pd.Series([1, 1, 1, 0, 0], index=index)
>>> precision(to_events(truth), to_events(guess))
1.0
Predicting a second, spurious event halves it:
>>> noisy = pd.Series([1, 1, 0, 0, 1], index=index)
>>> precision(to_events(truth), to_events(noisy))
0.5
Point-based, on the same labels: every flagged point was a true one.
Source code in src/hazure/evaluation/metrics.py
recall
¶
Fraction of true anomalies that were detected.
Also known as sensitivity, hit rate, or the true positive rate.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
y_true
|
Any
|
Ground truth. A label series or frame, an |
required |
y_pred
|
Any
|
Predictions, in the same form as |
required |
thresh
|
float
|
Event-based only: the fraction of a true event's duration that the
prediction must cover for it to count as detected. In |
0.5
|
Returns:
| Type | Description |
|---|---|
float or dict of float
|
One score, or one per column / dict key. |
Raises:
| Type | Description |
|---|---|
TypeError
|
The two arguments are not both point-based or both event-based. |
ValueError
|
|
Examples:
Event-based: a four-hour outage with three of its hours flagged is covered well past the default threshold, so the outage counts as detected.
>>> import pandas as pd
>>> from hazure.events import to_events
>>> index = pd.date_range("2024-01-01", periods=5, freq="h")
>>> truth = pd.Series([1, 1, 1, 1, 0], index=index)
>>> guess = pd.Series([1, 1, 1, 0, 0], index=index)
>>> recall(to_events(truth), to_events(guess))
1.0
Coverage is exactly three of the four hours, so demanding more misses it:
Point-based, on the same labels: three of the four true points were found.
Source code in src/hazure/evaluation/metrics.py
average_precision
¶
Area under the precision-recall curve of a continuous score.
Every other metric in this module needs labels, so evaluating a scorer with them means picking a threshold first and then measuring the threshold as much as the score. This does not: it asks only whether the anomalous samples are ranked above the normal ones, over every threshold at once.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
y_true
|
Any
|
Ground truth labels: a series or frame that
:meth: |
required |
scores
|
Any
|
A continuous score on the same time axis, higher meaning more anomalous —
the output of any :class: |
required |
Returns:
| Type | Description |
|---|---|
float or dict of float
|
Average precision in |
Raises:
| Type | Description |
|---|---|
TypeError
|
Either argument is an |
ValueError
|
The two series sit on different time axes, or they carry different column names or keys. |
See Also
roc_auc : The other threshold-free summary, less sensitive to rarity.
Notes
Sample-based, never event-based. Take the distinct score values in
decreasing order as thresholds. Writing tp_k and fp_k for the
anomalous and normal samples scoring at least as high as the k-th of them,
n_P for the number of anomalous samples, and summing over the thresholds::
P_k = tp_k / (tp_k + fp_k)
R_k = tp_k / n_P, R_0 = 0
AP = sum_k (R_k - R_{k-1}) * P_k
This is what sklearn.metrics.average_precision_score computes, and the
tests check that it agrees. Two details in it are the ones that matter.
Distinct values, so a block of tied scores contributes exactly one point
and no arbitrary order within the block can flatter the result. And a
right-hand rectangle rather than a trapezoid, so the curve is never
interpolated: the segment between two achievable operating points is
generally not itself achievable, and integrating it would be optimistic.
There is no event-based counterpart, deliberately. An event-based score would
need one number per interval, and there is no defensible way to choose it:
the maximum over the interval rewards a single lucky sample, the mean punishes
a detector that is right for one minute of a six-hour outage, and either
choice changes the ranking. Reduce the score to labels and use
:func:~hazure.evaluation.recall and :func:~hazure.evaluation.precision if
events are what you care about.
Examples:
A score that separates the two classes cleanly:
>>> import pandas as pd
>>> index = pd.date_range("2024-01-01", periods=4, freq="h")
>>> truth = pd.Series([0, 0, 1, 1], index=index)
>>> average_precision(truth, pd.Series([0.1, 0.2, 0.8, 0.9], index=index))
1.0
A score that says nothing, because every sample ties. Half the samples are anomalous, and half is what a coin gets:
The exactly wrong ranking is not 0.0: the last threshold still has to include every sample, and by then precision is the base rate.
Source code in src/hazure/evaluation/ranking.py
23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 | |
roc_auc
¶
Area under the ROC curve of a continuous score.
The probability that a randomly chosen anomalous sample outranks a randomly
chosen normal one, with ties counting half. It is threshold-free like
:func:average_precision, and differs in what it is sensitive to: because
the false positive rate has the whole normal class in its denominator, a
detector can look excellent here while alerting far more often than it is
right. On the rare-anomaly data this library is aimed at, quote both.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
y_true
|
Any
|
Ground truth labels, in any form :func: |
required |
scores
|
Any
|
A continuous score on the same time axis, higher meaning more anomalous.
Rows whose score is |
required |
Returns:
| Type | Description |
|---|---|
float or dict of float
|
Area in |
Raises:
| Type | Description |
|---|---|
TypeError
|
Either argument is an |
ValueError
|
The two series sit on different time axes, or they carry different column names or keys. |
See Also
average_precision : The other threshold-free summary, harder to flatter.
Notes
Sample-based, never event-based, for the reason given in
:func:average_precision.
Computed exactly, through midranks rather than by walking a curve. Rank the
retained samples by score in increasing order from 1, giving every member of
a tied block the average r_i of the ranks that block occupies. With P
the set of anomalous samples, n_P its size and n_N the number of
normal samples, this is the Mann-Whitney statistic::
AUC = (sum_{i in P} r_i - n_P * (n_P + 1) / 2) / (n_P * n_N)
The midranks are what handle ties, and they handle them the way the curve
does: a tied block is one diagonal step of the ROC, so what it contributes is
the trapezoid under that diagonal, which is half of its tied pairs. Sorting
the block arbitrarily instead would turn that diagonal into a staircase, and
a score carrying no information at all could then come out anywhere between 0
and 1. The tests check the result against
sklearn.metrics.roc_auc_score, ties included.
Examples:
>>> import pandas as pd
>>> index = pd.date_range("2024-01-01", periods=4, freq="h")
>>> truth = pd.Series([0, 0, 1, 1], index=index)
>>> roc_auc(truth, pd.Series([0.1, 0.2, 0.8, 0.9], index=index))
1.0
>>> roc_auc(truth, pd.Series([0.9, 0.8, 0.2, 0.1], index=index))
0.0
Every sample tied is exactly 0.5, whatever order the sort happened to put them in:
One anomaly ranked top, the other tied with a normal sample halfway down:
Source code in src/hazure/evaluation/ranking.py
122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 | |
split_train_test
¶
split_train_test(
data: Any,
mode: int = 1,
n_splits: int = 1,
train_ratio: float = 0.7,
) -> list[tuple[Any, Any]]
Split a time series into n_splits (train, test) folds.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
data
|
Any
|
Anything :meth: |
required |
mode
|
int
|
Which of the four splitting schemes to use, 1 to 4. See Notes. |
1
|
n_splits
|
int
|
Number of folds to produce, at least 1. |
1
|
train_ratio
|
float
|
Fraction of each fold given to training, strictly between 0 and 1. Modes
3 and 4 derive their cut points from |
0.7
|
Returns:
| Type | Description |
|---|---|
list of tuple
|
|
Raises:
| Type | Description |
|---|---|
TypeError
|
|
ValueError
|
|
Notes
In the diagrams below, 1 marks a training position, 2 a testing
position and 0 a position that fold does not use. Each row is one fold.
All four show a 40-sample series with n_splits=4, train_ratio=0.7.
Mode 1 — n_splits equal disjoint folds, each cut at
train_ratio. Every observation is used exactly once, so the folds are
independent, but a later fold trains on no more history than an earlier
one::
1111111222000000000000000000000000000000
0000000000111111122200000000000000000000
0000000000000000000011111112220000000000
0000000000000000000000000000001111111222
Mode 2 — nested folds, all starting at the first observation, each cut
at train_ratio. Fold k covers the first k / n_splits of the
series, so training and testing both grow::
1111111222000000000000000000000000000000
1111111111111122222200000000000000000000
1111111111111111111112222222220000000000
1111111111111111111111111111222222222222
Mode 3 — an expanding training window with a fixed-size test block appended. Every fold tests on the same number of observations, which makes scores directly comparable across folds::
1111111122222222000000000000000000000000
1111111111111111222222220000000000000000
1111111111111111111111112222222200000000
1111111111111111111111111111111122222222
Mode 4 — an expanding training window tested against the whole remainder. This is the "what would I have known at time t" question, at the cost of test sets of unequal size::
1111111122222222222222222222222222222222
1111111111111111222222222222222222222222
1111111111111111111111112222222222222222
1111111111111111111111111111111122222222
Examples:
>>> import pandas as pd
>>> index = pd.date_range("2024-01-01", periods=10, freq="D")
>>> data = pd.DataFrame({"x": range(10)}, index=index)
>>> [(len(train), len(test)) for train, test in split_train_test(data)]
[(7, 3)]
>>> [
... (len(train), len(test))
... for train, test in split_train_test(data, mode=4, n_splits=2)
... ]
[(3, 7), (6, 4)]
Source code in src/hazure/evaluation/split.py
24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 | |
hazure.calibration¶
Choosing the cut-off: from labelled incidents, or from how many alerts a week anyone will read.
calibration
¶
Choosing where to draw the line.
A scorer says how unusual each point is. Turning that into an alert needs a
number, and picking that number is the question people actually get stuck on:
every threshold in :mod:hazure.thresholds is parameterised by something, and
none of those somethings is "the answer I want".
Two ways out of it, depending on what you have.
If you have labelled history — even a handful of incidents somebody wrote down —
:func:tune_threshold searches for the cut-off that scores best against them.
Event-based F1 by default, because a monitor is judged on outages caught and
pages sent, not on samples classified.
If you have no labels, which is the case this library is mostly about, you still
have an alert budget: whatever the metric does, nobody is going to look at more
than one page a week. :func:budget_threshold finds the most sensitive cut-off
that stays inside that budget, which turns an unanswerable question about
distributions into an answerable one about how much attention you have.
Both hand back a :class:Calibration, which carries the whole curve it searched
and not only the winner — a threshold chosen off a flat peak is worth knowing
about, and so is one chosen off a spike.
Hyperparameters other than the cut-off need no machinery from here: components
follow the scikit-learn parameter conventions closely enough for
sklearn.model_selection.GridSearchCV to drive them directly, and
:func:~hazure.split_train_test supplies folds that respect time order.
Calibration
dataclass
¶
Calibration(
cut_off: float,
score: float,
objective: str,
curve: tuple[tuple[float, float], ...],
)
Where the line was drawn, what that achieved, and what else was considered.
Attributes:
| Name | Type | Description |
|---|---|---|
cut_off |
float
|
The chosen cut-off. Scores strictly above it are anomalous. |
score |
float
|
What :attr: |
objective |
str
|
What was being optimised, as text — |
curve |
tuple of tuple
|
Every |
See Also
tune_threshold : Choose a cut-off against labelled history. budget_threshold : Choose a cut-off against an alert budget.
Examples:
>>> calibration = Calibration(2.5, 0.8, "event f1", ((1.0, 0.5), (2.5, 0.8)))
>>> calibration.threshold
FixedThreshold(high=2.5)
threshold
property
¶
threshold: FixedThreshold
The cut-off as a threshold, ready to pair with a scorer.
Returns:
| Type | Description |
|---|---|
FixedThreshold
|
A one-sided fence at :attr: |
tune_threshold
¶
tune_threshold(
y_true: Any,
scores: Any,
*,
metric: str | Callable[..., Any] = "f1",
candidates: int = 200,
events: bool = True,
**options: Any,
) -> Calibration | dict[str, Calibration]
Find the cut-off that scores best against labelled history.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
y_true
|
Any
|
Ground truth: a label series, an :class: |
required |
scores
|
Any
|
Continuous scores from any :class: |
required |
metric
|
str | Callable[..., Any]
|
What to maximise: |
'f1'
|
candidates
|
int
|
Roughly how many cut-offs to try. They are placed at quantiles of the scores rather than at evenly spaced values, half evenly and half packed towards the largest score, so the search resolves the extreme tail where a usable fence normally sits. |
200
|
events
|
bool
|
Score by interval rather than by sample. This is the default because it is the question a monitor is judged on — an outage caught late is caught, and a hundred alerts inside one incident are one page. Turn it off to weight every sample equally. |
True
|
**options
|
Any
|
Passed through to the metric, e.g. |
{}
|
Returns:
| Type | Description |
|---|---|
Calibration or dict of Calibration
|
The chosen cut-off and the curve behind it, or one per score column. |
Raises:
| Type | Description |
|---|---|
KeyError
|
|
TypeError
|
|
ValueError
|
|
See Also
budget_threshold : The same question without labels, answered from an alert budget instead.
Notes
Ties go to the higher cut-off. A flat top to the curve is common — the metric cannot tell apart two fences with no scores between them — and of two equally good fences the higher one raises fewer alerts, so it is the one that survives contact with people.
A metric that comes back undefined, as precision does when nothing at all is
flagged, is recorded in the curve as nan and never chosen.
None of the four metrics counts alerts, and event-based precision in particular
does not punish a fragmented one: two alerts inside the same true event are two
justified alerts. So a cut-off with a perfect F1 can still page twice for one
incident. Check :func:~hazure.to_events on the result, or debounce with
:func:~hazure.expand_events, or calibrate with :func:budget_threshold
instead, which counts alerts by construction.
The cut-off this finds is fitted to y_true, and quoting the same metric at
the same cut-off on the same data would be reporting a training score. Tune on
one fold and measure on the next; :func:~hazure.split_train_test exists for
that.
Examples:
A series with two planted outages, scored by deviation from the median, and a cut-off chosen against the intervals somebody wrote down:
>>> import numpy as np
>>> import pandas as pd
>>> from hazure import DeviationScorer
>>> index = pd.date_range("2024-01-01", periods=480, freq="h", name="time")
>>> rng = np.random.default_rng(0)
>>> values = pd.Series(rng.normal(0, 1, 480), index=index, name="x")
>>> values.iloc[100:106] += 6.0
>>> values.iloc[300:304] += 5.0
>>> truth = pd.Series(0.0, index=index)
>>> truth.iloc[100:106] = 1.0
>>> truth.iloc[300:304] = 1.0
>>> scores = DeviationScorer().fit_score(values)
>>> best = tune_threshold(truth, scores)
>>> best
Calibration(cut_off=3.96769, event f1=1, over 200 candidates)
Both outages are caught and nothing else is flagged — but the fence cuts one of them into two alerts, and an event-based F1 of 1.0 does not notice, because both fragments lie inside a true event and so both count as justified:
That is worth knowing before trusting the number. Read the alert count next to
it, or debounce with :func:~hazure.expand_events first.
Source code in src/hazure/calibration.py
124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 | |
budget_threshold
¶
budget_threshold(
scores: Any,
*,
alerts: float = 1.0,
per: str | timedelta | timedelta64 = "7d",
gap: int | str | timedelta | timedelta64 | None = None,
candidates: int = 200,
) -> Calibration | dict[str, Calibration]
Find the most sensitive cut-off that stays inside an alert budget.
No labels needed, which is the point. What is known instead is how much attention exists: one page a week, five a day. This lowers the fence as far as that allows and no further.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
scores
|
Any
|
Continuous scores from any :class: |
required |
alerts
|
float
|
How many alerts are affordable per |
1.0
|
per
|
str | timedelta | timedelta64
|
The period the budget is expressed over: |
'7d'
|
gap
|
int | str | timedelta | timedelta64 | None
|
Alerts closer together than this are one alert. Without it a flagged
stretch broken by a single quiet sample counts twice, which overstates the
rate a monitor would actually produce, since real ones debounce. A
duration, or an |
None
|
candidates
|
int
|
Roughly how many cut-offs to try, placed at quantiles of the scores and packed towards the largest of them — an alert budget of one a week lives in the top fraction of a percent, which an evenly spaced grid cannot resolve. |
200
|
Returns:
| Type | Description |
|---|---|
Calibration or dict of Calibration
|
The chosen cut-off, the alert rate it realises, and the whole rate curve,
or one per score column. :attr: |
Raises:
| Type | Description |
|---|---|
ValueError
|
|
See Also
tune_threshold : The same question when there is labelled history to aim at.
hazure.expand_events : What gap is doing, available on its own.
Notes
The search descends. It starts at the highest candidate — which flags nothing and so costs nothing — and lowers the fence one candidate at a time, stopping at the first cut-off that breaks the budget and taking the last one that did not.
Descending rather than scanning for the lowest cut-off that happens to fit is not a detail. Alert count is not monotone in the cut-off: raising a fence removes flagged samples, and removing one from the middle of a flagged stretch splits one alert into two. Taken far enough that reverses completely — a fence at the 1st percentile flags almost every sample, which merges into a single enormous alert and satisfies any budget you like. Descending until the budget breaks never reaches that region, because the count passes through the budget on the way down long before the alerts begin to merge.
The rate is measured over the span the scores cover, so it is an estimate from one sample of history and inherits everything that was unusual about that history. A budget met over a quiet fortnight will be missed in a busy one.
Examples:
Three weeks of a metric with a handful of excursions in it, and a budget of one alert a week:
>>> import numpy as np
>>> import pandas as pd
>>> from hazure import DeviationScorer
>>> index = pd.date_range("2024-01-01", periods=24 * 21, freq="h", name="time")
>>> rng = np.random.default_rng(0)
>>> values = pd.Series(rng.normal(0, 1, len(index)), index=index, name="rps")
>>> for start in (40, 150, 300, 380, 460):
... values.iloc[start : start + 3] += 5.0
>>> scores = DeviationScorer().fit_score(values)
>>> chosen = budget_threshold(scores, alerts=1, per="7d")
>>> chosen
Calibration(cut_off=2.88988, alerts per 7d=1, over 200 candidates)
Three alerts over three weeks is the budget, met exactly:
Ask for more attention and the fence comes down:
>>> generous = budget_threshold(scores, alerts=5, per="7d")
>>> generous.cut_off < chosen.cut_off
True
Source code in src/hazure/calibration.py
248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 | |
hazure.streaming¶
Driving a fitted component one observation at a time, for a series that is still being produced.
streaming
¶
Driving a fitted component one observation at a time.
Everything else in hazure takes a whole series and answers about all of it. That is the right shape for looking back over a month of history, and the wrong shape for the thing the library is aimed at: a metric that is still being produced, one sample at a time, where the question is asked again every minute and the answer has to arrive before the next sample does.
:class:Stream bridges the two without a second implementation of any algorithm.
It keeps the recent past in a buffer, and for each arriving observation runs the
component over that buffer and returns the verdict on the last row. So the online
answer is the batch answer, by construction — not a reimplementation that agrees
with it on the cases anyone thought to test.
What that costs is a buffer long enough for the component to see what it needs. Get it wrong and a rolling window is computed over a window that was never full, which is the kind of mistake that produces plausible numbers rather than an error. And not every component can be streamed at all: one that reads forward — a centred window, the right-hand window of a double rolling aggregate — needs observations that have not happened yet.
:meth:Stream.prime refuses both. It computes the component's answers from the
full history you hand it, recomputes several of them from the buffer alone, and
says which of the two problems it found when they disagree.
The other half of the story is that the fit does not move. A component is fitted
once, on a period you are willing to call normal, and can be stored with
:meth:~hazure.Component.to_dict and reloaded next week — so what "normal" means
is a decision you made deliberately and can point at, rather than a property of
whatever window the monitor happens to be looking at. Where you do want the
fence to move as scores arrive, :meth:hazure.PotThreshold.update is the piece
that does it, and the two compose: stream the scorer, and hand each score to the
threshold.
Stream
¶
Stream(
component: Component, history: int | str | timedelta
)
Bases: Configurable
A fitted component, fed one observation at a time.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
component
|
Component
|
Any fitted :class: |
required |
history
|
int | str | timedelta
|
How much of the past to keep. An |
required |
Attributes:
| Name | Type | Description |
|---|---|---|
n_seen |
int
|
Observations pushed in so far, counting those that arrived through
:meth: |
columns |
tuple of str or None
|
Column names being tracked, or None before the first observation. |
Raises:
| Type | Description |
|---|---|
TypeError
|
|
ValueError
|
|
See Also
prime : Fill the buffer from history before the first live observation. hazure.PotThreshold.update : A fence that moves as scores arrive, for the cases where holding it fixed is not what you want.
Notes
Each observation costs one run of the component over the whole buffer, so a buffer of h rows makes the per-observation cost O(h) rather than the O(1) an incremental reformulation of each algorithm could reach. That is a deliberate trade. The series this is for arrive at human timescales — a sample a minute, a sample every ten seconds — and at those rates the cost is irrelevant next to the interval between samples, while the guarantee it buys is not: there is exactly one implementation of every algorithm, so the online and batch answers cannot drift apart.
A stream carries its buffer, so :meth:~hazure.Component.to_dict stores a
running monitor whole — the fitted component and the recent past it was
judging against — and :meth:~hazure.Component.from_dict resumes it where it
left off. history has to be an int or a string for that, since a
:class:~datetime.timedelta has no JSON representation.
Sizing history is the one thing that needs care, and it is not simply the
window a detector was configured with. A detector built on
:class:~hazure.DoubleRollingAggregate looks back over two windows;
:class:~hazure.SeasonalDetector needs at least a period, and reads better
with several; a :class:~hazure.Pipeline needs the sum of what its steps
consume, because each step's output starts later than its input did. Rather
than reason about it, hand :meth:prime more history than you think you need
and let it tell you.
Examples:
Fit on a fortnight, then stream the next day past it. The detector is fitted once and never sees the live data as training material:
>>> import numpy as np
>>> import pandas as pd
>>> from hazure import SpikeDetector
>>> index = pd.date_range("2024-03-01", periods=24 * 14, freq="h", name="time")
>>> rng = np.random.default_rng(0)
>>> history = pd.Series(100 + rng.normal(0, 2, len(index)), index=index, name="rps")
>>> detector = SpikeDetector(window=24).fit(history)
>>> stream = Stream(detector, history=48).prime(history)
A sample in line with the fortnight is unremarkable, and one four times the normal level is not:
Streaming a batch answers the same question repeatedly, which is how a streaming setup gets backtested before it is deployed:
>>> later = pd.Series(
... 100 + rng.normal(0, 2, 12),
... index=pd.date_range("2024-03-15T02:00", periods=12, freq="h", name="time"),
... name="rps",
... )
>>> labels = stream.update_many(later)
>>> float(labels.sum())
0.0
>>> stream.n_seen
350
Source code in src/hazure/streaming.py
columns
property
¶
Column names being tracked, or None before the first observation.
buffer
property
¶
buffer: TimeSeries
The retained past, as the component sees it.
Returns:
| Type | Description |
|---|---|
TimeSeries
|
The buffered observations. This is the internal representation rather than a reconstruction of whatever backend the data arrived in, since the buffer is assembled from many separate observations and has no single origin to return to. |
prime
¶
prime(data: Any, *, check: bool = True) -> Stream
Fill the buffer from history, and check that it is long enough.
Without this the first observations are judged against an almost empty
buffer, and a component that needs a window to look back over has no
choice but to answer NaN until one has accumulated — for a detector
with a daily period, that is a day of blindness immediately after
deployment.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
data
|
Any
|
Historical observations, anything :meth: |
required |
check
|
bool
|
Verify that the retained buffer is long enough for the component, by
comparing the answer it gives for the final observation against the
answer the full |
True
|
Returns:
| Type | Description |
|---|---|
Stream
|
This stream, for chaining. |
Raises:
| Type | Description |
|---|---|
RuntimeError
|
The component has not been fitted. |
ValueError
|
The buffer is too short for what the component looks back over, or the
component reads forward and so cannot be streamed at any buffer
length. Also raised if |
Notes
The check compares the newest observation and a handful of interior ones, because the two ways streaming can fail show up in different places. A buffer that does not reach far enough back is visible at the newest observation. A component that reads forward is not — at the newest observation there is no future in either the batch pass or the buffer, so both are equally blind and agree. Interior rows are where the batch pass can see what came after and a stream could not.
It is exact rather than statistical, and still not a proof: a buffer one sample short of a rolling window can agree on a quiet stretch of history by luck. Prime on a stretch that contains something interesting where you can.
Examples:
A buffer far too short for a 24-sample window says so, rather than
quietly reporting a window's worth of NaN:
>>> import numpy as np
>>> import pandas as pd
>>> from hazure import SpikeDetector
>>> index = pd.date_range("2024-01-01", periods=200, freq="h", name="time")
>>> rng = np.random.default_rng(0)
>>> values = pd.Series(rng.normal(size=200), index=index, name="x")
>>> detector = SpikeDetector(window=24).fit(values)
>>> Stream(detector, history=5).prime(values)
Traceback (most recent call last):
...
ValueError: Stream(history=5) is too short for SpikeDetector: ...
Source code in src/hazure/streaming.py
202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 | |
update
¶
update(
time: Any,
values: float
| Mapping[str, float]
| Sequence[float]
| NDArray[float64],
) -> float | dict[str, float]
Push one observation and return the component's verdict on it.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
time
|
Any
|
Its timestamp: a :class: |
required |
values
|
float | Mapping[str, float] | Sequence[float] | NDArray[float64]
|
The observation. A single number for a one-column component, or a mapping from column name to number, or a sequence in the order the component was fitted on. |
required |
Returns:
| Type | Description |
|---|---|
float or dict of float
|
The component's output for this observation — a label for a detector,
a score for a scorer — as one number, or one per output column when
the component emits several. |
Raises:
| Type | Description |
|---|---|
RuntimeError
|
The component has not been fitted. |
TypeError
|
|
ValueError
|
|
See Also
update_many : The same, over a batch of observations.
Notes
The observation goes into the buffer before it is judged, which is what
makes the answer identical to the batch one: every detector here decides
about a point using that point, and a spike is only a spike relative to
the neighbours it sits among. Nothing after it is used, because nothing
after it exists yet — which is also why a detector whose batch answer
depends on the future, as a centred window does, will not agree with
itself here. :meth:prime catches that.
Source code in src/hazure/streaming.py
386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 | |
update_many
¶
Push a batch of observations in order, returning one verdict each.
Every row is judged as if it had just arrived, so this is not the same as
running the component over data directly: a row is placed against the
buffer as it stood at that moment and never against what came after it.
That makes this the way to backtest a streaming setup — the numbers it
produces are the numbers the monitor would have produced.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
data
|
Any
|
Observations in time order, anything
:meth: |
required |
Returns:
| Type | Description |
|---|---|
Any
|
One verdict per row of |
Raises:
| Type | Description |
|---|---|
RuntimeError
|
The component has not been fitted. |
ValueError
|
The batch is not entirely after the newest buffered observation, or it carries columns the component was not fitted on. |
See Also
update : The same, one observation at a time.
Source code in src/hazure/streaming.py
hazure.methods¶
Further method families: spectral residual, Hampel filtering and rolling quantile bands, change-point segmentation, matrix-profile discords, and STL / MSTL residuals.
methods
¶
Detection methods beyond the rolling-window rules.
Five families, each a scorer and a ready-made detector pairing it with a threshold:
Spectral residual Saliency from a Fourier transform with its smooth amplitude envelope removed. Assumes no period, no trend and no distribution. Local order statistics The Hampel filter and a rolling quantile band, for a normal range that drifts and a scale the outliers cannot inflate. Segmentation Asks when the series became a different series rather than which points do not belong. PELT in numpy, plus an adapter for other search strategies. Matrix profile The subsequences least like anything else in the series. Scores a shape rather than a point, so it finds anomalies made entirely of ordinary values. Seasonal-trend decomposition Single- or multi-period, scoring what the decomposition cannot explain.
The spectral residual, the Hampel filter and PELT are implemented here and need
nothing but numpy. :class:RupturesScorer, the matrix-profile scorers and the
STL scorers import their backend lazily and say how to install it if it is
missing.
DampDetector
¶
Bases: ScoreDetector
Flag the stretches of the series unlike anything that came before them.
:class:DampScorer paired with an inter-quartile-range rule, the same pairing
:class:MatrixProfileDetector uses and for the same reason: every series has
a least-matched subsequence, so a distance is only interesting when it is out
of proportion to the rest of the distances.
The difference from :class:MatrixProfileDetector is which distances those
are. Here a subsequence is compared only with subsequences that started
earlier, so a shape that recurs later in the series cannot explain away its
own first appearance. That makes this the one to reach for when an anomaly
might happen twice, and the one to reach for when the question is "was this
novel at the time" rather than "is this unique in the record".
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
window
|
int
|
Subsequence length, in observations. |
required |
factor
|
Factor
|
Inter-quartile-range factor deciding how far from its nearest earlier neighbour a subsequence has to be. One-sided: a shape with a close match in the past is never anomalous. |
3.0
|
normalize
|
bool
|
Compare shapes (z-normalised) rather than raw amplitudes. |
True
|
Raises:
| Type | Description |
|---|---|
TypeError
|
|
ValueError
|
|
ImportError
|
|
Notes
The first two windows score NaN — see :class:DampScorer — and the fence is
fitted on the distances that remain. A series barely longer than the warm-up
therefore learns its fence from very little, and the fence will move as more
data arrives.
Examples:
Source code in src/hazure/methods/damp_detector.py
DampScorer
¶
Bases: BaseScorer
Score each subsequence against its nearest neighbour in the past only.
The same idea as :class:MatrixProfileScorer with one restriction: a
subsequence is compared only against subsequences that started before it. The
score then answers "has anything like this happened yet", which is the
question a monitor asks, rather than "does anything like this happen anywhere
in the series", which needs the future to answer.
That restriction changes results in a way worth understanding. A two-off anomaly — an unusual shape that happens twice — is invisible to a full matrix profile, because each occurrence is the other's near neighbour and both score low. Scored against the past alone, the first occurrence has nothing to match and stands out.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
window
|
int
|
Subsequence length, in observations. |
required |
normalize
|
bool
|
Compare shapes (z-normalised) rather than raw amplitudes. |
True
|
Raises:
| Type | Description |
|---|---|
TypeError
|
|
ValueError
|
|
ImportError
|
|
Notes
stumpy reports the index of each subsequence's nearest left neighbour
but not the distance to it, so the distances are computed here from those
indices — one distance per subsequence, which costs a single pass and matches
the profile's own metric (z-normalised Euclidean, or plain Euclidean when
normalize is False). Should a future version expose a left profile
directly, this becomes a lookup.
The first two windows of the series are a warm-up and score NaN. They have to be: with almost no past to be compared against, the opening of any series is the least-matched part of it, and scoring it would report the shortage of history rather than an anomaly. Two windows is the minimum that leaves a subsequence something to match; a longer series is better judged from further in still. Subsequences whose neighbour is undefined — because one of the two is flat, and a flat shape does not z-normalise — also score NaN.
Nothing is learned; :meth:fit is optional.
Examples:
Source code in src/hazure/methods/damp_scorer.py
HampelDetector
¶
Bases: ScoreDetector
Flag points too far from the local median to be part of the local noise.
:class:HampelScorer with a fixed cut-off. The cut-off is fixed rather than
learned because the score is already expressed in standard deviations of the
local noise: factor=3.0 means "three sigma away from where this stretch
of the series sits", which is the Hampel filter's own rule and needs no
reference to the distribution of the scores.
Nothing is learned, so this detector can be used without :meth:fit.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
window
|
Window
|
Observations or duration making up each point's neighbourhood. |
7
|
factor
|
float
|
How many local standard deviations away is too far. |
3.0
|
center
|
bool
|
Centre the window on each point rather than trailing it. |
True
|
Raises:
| Type | Description |
|---|---|
ValueError
|
The window is not positive. |
Examples:
>>> import numpy as np
>>> from hazure import TimeSeries
>>> values = np.tile([10.0, 11.0, 12.0, 11.0], 8)
>>> values[17] = 40.0
>>> time = np.arange("2024-01-01", "2024-02-02", dtype="datetime64[D]")
>>> labels = HampelDetector().detect(TimeSeries.from_arrays(time, values))
>>> np.flatnonzero(labels.values.ravel() == 1.0)
array([17])
Source code in src/hazure/methods/hampel_detector.py
HampelScorer
¶
Bases: BaseScorer
Score each point by its distance from the local median, in local MADs.
The Hampel filter [1]_. For every point, take the median of the window around it as the local notion of normal and the median absolute deviation of that window as the local notion of scale, then report
|x - local median| / (1.4826 * local MAD).
Both estimates are order statistics, so a single wild value moves neither: that is what lets the filter measure an outlier against a scale the outlier did not inflate.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
window
|
Window
|
Observations ( |
7
|
center
|
bool
|
Centre the window on each point rather than trailing it. Centred is the default because a filter is normally applied retrospectively, and a two-sided neighbourhood judges a point without the level shift of the point itself dragging the centre. Set False for a causal score that uses only the past. |
True
|
Raises:
| Type | Description |
|---|---|
ValueError
|
The window is not positive, or a duration window is used on a series whose time axis cannot support it. |
Notes
The factor 1.4826 turns a median absolute deviation into an estimate of the
standard deviation of a normal sample, so the score reads on the familiar
"number of sigmas" scale: for X ~ N(mu, sigma) the median of |X - mu| is
0.6745 * sigma, and 1 / 0.6745 = 1.4826. Without it, a score of 3 would
mean three MADs, which is only two standard deviations, and every published
Hampel threshold would read wrong.
A window with no spread at all — a locally constant stretch — has no scale, so the distance cannot be expressed in units of it. Those points score NaN, "undefined", rather than infinity: a flat window is a statement about the absence of information, not about the size of a departure.
The spread is the rolling median of |x - local median|, which keeps both
passes to a single vectorised rolling call over bounded memory. Where the level
is locally stable — the case the filter is designed for — every point in a
window shares that window's median, and this agrees exactly with the classical
definition of taking the deviations from one window's own centre.
Points whose window holds fewer than a full complement of observations score NaN, which for a centred window is a margin of half a window at each end. Whether a point is scored is decided by the centre: the spread is taken from however many deviations its window contains, so the second pass does not widen that margin to a whole window.
Nothing is learned; :meth:fit is optional.
References
.. [1] F. R. Hampel, "The Influence Curve and its Role in Robust Estimation", Journal of the American Statistical Association 69(346), 1974, pp. 383-393.
Examples:
>>> import numpy as np
>>> from hazure import TimeSeries
>>> values = np.tile([10.0, 11.0, 12.0, 11.0], 8)
>>> values[17] = 40.0
>>> time = np.arange("2024-01-01", "2024-02-02", dtype="datetime64[D]")
>>> scores = HampelScorer().score(TimeSeries.from_arrays(time, values))
>>> int(np.nanargmax(scores.values.ravel()))
17
Source code in src/hazure/methods/hampel_scorer.py
MatrixProfileDetector
¶
Bases: ScoreDetector
Flag the stretches of the series least like anything else in it.
:class:MatrixProfileScorer paired with an inter-quartile-range rule. The
threshold is what turns "which shape is strangest" into "which shapes are
strange enough to report": the profile always has a maximum, even in a series
with nothing wrong in it, so the largest distance is only interesting when it
is out of proportion to the rest of the profile.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
window
|
int
|
Subsequence length, in observations. |
required |
factor
|
Factor
|
Inter-quartile-range factor deciding how far from its neighbours a subsequence has to be. One-sided: a shape that matches the series well is never anomalous. |
3.0
|
normalize
|
bool
|
Compare shapes (z-normalised) rather than raw amplitudes. Leave it on to match a pattern wherever it sits and however large it is; turn it off when the level and the amplitude are part of what makes the shape itself. |
True
|
Raises:
| Type | Description |
|---|---|
TypeError
|
|
ValueError
|
|
ImportError
|
|
Examples:
>>> from hazure.methods import MatrixProfileDetector
>>> MatrixProfileDetector(window=24)
MatrixProfileDetector(window=24)
Source code in src/hazure/methods/matrix_profile_detector.py
MatrixProfileScorer
¶
Bases: BaseScorer
Score each point by how unlike the rest of the series its shape is.
For every subsequence of window consecutive observations, the matrix
profile records the distance to the closest other subsequence in the series.
A subsequence with no close match is a discord, and its distance is the score.
Distances are z-normalised by default, which is what makes the comparison one
of shape: a pattern is judged the same whether it happened at a high level
or a low one, loudly or quietly. Set normalize=False to compare raw
amplitudes instead, which is right when the level itself is part of the
pattern.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
window
|
int
|
Subsequence length, in observations. This is the one parameter that matters: it declares how long the pattern of interest is. Roughly one period of the behaviour being described is the usual starting point. |
required |
normalize
|
bool
|
Compare shapes (z-normalised) rather than raw amplitudes. |
True
|
Raises:
| Type | Description |
|---|---|
TypeError
|
|
ValueError
|
|
ImportError
|
|
Notes
A distance belongs to a subsequence, not to a point, so it is broadcast back
over the window observations the subsequence covers, and each point takes
the largest distance of any subsequence containing it. The maximum rather than
the mean because a point that participates in one unmatched shape is
implicated by it, however many ordinary shapes it also belongs to; the cost is
that the flagged region is a window wide rather than a single point.
Nothing is learned. The matrix profile is a self-join — the series is compared
against itself — so it is a property of the series being scored, and
:meth:fit is optional.
Subsequences containing a missing observation have no comparable neighbour, and the points covered only by such subsequences score NaN.
References
.. [1] C.-C. M. Yeh et al., "Matrix Profile I: All Pairs Similarity Joins for Time Series", IEEE ICDM 2016, pp. 1317-1322.
Examples:
>>> from hazure.methods import MatrixProfileScorer
>>> MatrixProfileScorer(window=24)
MatrixProfileScorer(window=24)
Source code in src/hazure/methods/matrix_profile_scorer.py
MstlDetector
¶
MstlDetector(
periods: int | Sequence[int],
robust: bool = True,
windows: int | Sequence[int] | None = None,
factor: Factor = 3.0,
)
Bases: ScoreDetector
Flag points that break none of several rhythms but still do not fit.
:class:MstlResidualScorer paired with an inter-quartile-range rule. Use this
rather than :class:StlDetector whenever the series has more than one
rhythm: with a second cycle left in the residual, the residual's spread is set
by that cycle rather than by the noise, and the threshold ends up asking how
unusual a point is compared with a systematic pattern instead of compared with
chance.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
periods
|
int | Sequence[int]
|
Cycle lengths in observations: one integer, or several. |
required |
robust
|
bool
|
Reweight the loess fits to discount outliers. |
True
|
windows
|
int | Sequence[int] | None
|
Seasonal smoother length per period. |
None
|
factor
|
Factor
|
Inter-quartile-range factor deciding how large a residual is too large. |
3.0
|
Raises:
| Type | Description |
|---|---|
ValueError
|
The time axis is irregular, or a period is unusable. |
ImportError
|
|
Examples:
>>> import numpy as np
>>> from hazure import TimeSeries
>>> hours = np.arange(24 * 28)
>>> values = (
... 10.0
... + 3.0 * np.sin(hours * 2 * np.pi / 24)
... + 5.0 * np.sin(hours * 2 * np.pi / 168)
... )
>>> values[300] += 20.0
>>> time = hours * np.timedelta64(1, "h") + np.datetime64("2024-01-01")
>>> labels = MstlDetector(periods=(24, 168), factor=25.0).fit_detect(
... TimeSeries.from_arrays(time, values)
... )
>>> np.flatnonzero(labels.values.ravel() == 1.0)
array([300])
Source code in src/hazure/methods/mstl_detector.py
MstlResidualScorer
¶
MstlResidualScorer(
periods: int | Sequence[int],
robust: bool = True,
windows: int | Sequence[int] | None = None,
)
Bases: BaseScorer
Score each point by the size of its residual after removing several cycles.
MSTL applies STL once per period, longest cycle last, subtracting each seasonal component before fitting the next. A series sampled hourly usually has two rhythms — the hour of the day and the day of the week — and a single-period decomposition has to choose one and leave the other in the residual, where it dominates and drowns everything else. Removing both leaves a remainder that is genuinely just the remainder.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
periods
|
int | Sequence[int]
|
Cycle lengths in observations: one integer, or several. For hourly data,
|
required |
robust
|
bool
|
Reweight the loess fits to discount outliers, in every STL pass. |
True
|
windows
|
int | Sequence[int] | None
|
Seasonal smoother length per period, as an integer or one per period.
None leaves them to |
None
|
Raises:
| Type | Description |
|---|---|
ValueError
|
The time axis is irregular, |
ImportError
|
|
Notes
Requires a regular time axis. Missing observations are interpolated before the
decomposition and their scores set back to NaN, as for
:class:StlResidualScorer.
Each period needs two full cycles of data to be estimable, so the longest period governs how much history the scorer needs.
Nothing is learned; :meth:fit is optional.
References
.. [1] K. Bandara, R. J. Hyndman and C. Bergmeir, "MSTL: A Seasonal-Trend Decomposition Algorithm for Time Series with Multiple Seasonal Patterns", International Journal of Operational Research, 2021.
Examples:
A daily and a weekly rhythm together, with one point breaking both:
>>> import numpy as np
>>> from hazure import TimeSeries
>>> hours = np.arange(24 * 28)
>>> values = (
... 10.0
... + 3.0 * np.sin(hours * 2 * np.pi / 24)
... + 5.0 * np.sin(hours * 2 * np.pi / 168)
... )
>>> values[300] += 20.0
>>> time = hours * np.timedelta64(1, "h") + np.datetime64("2024-01-01")
>>> scorer = MstlResidualScorer(periods=(24, 168))
>>> scores = scorer.score(TimeSeries.from_arrays(time, values))
>>> int(np.nanargmax(scores.values.ravel()))
300
Source code in src/hazure/methods/mstl_residual_scorer.py
PeltDetector
¶
Bases: ScoreDetector
Flag the points at which the series changed regime.
:class:PeltScorer with a threshold that passes its non-zero scores through.
There is deliberately no factor to tune: the penalty has already decided which
changes are large enough to be worth a segment, and second-guessing that with
a rule on the score would be answering the same question twice with less
information. To report fewer changes, raise penalty.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
penalty
|
float | None
|
Cost of admitting one more segment. None derives a BIC-style value from the data. |
None
|
cost
|
Cost
|
|
'l2'
|
min_size
|
int
|
Shortest segment allowed, in observations. |
2
|
jump
|
int
|
Consider only breakpoints at multiples of this many observations. |
1
|
Raises:
| Type | Description |
|---|---|
ValueError
|
|
Examples:
>>> import numpy as np
>>> from hazure import TimeSeries
>>> rng = np.random.default_rng(0)
>>> values = np.concatenate([rng.normal(size=60), rng.normal(loc=10.0, size=60)])
>>> time = np.arange("2024-01-01", "2024-04-30", dtype="datetime64[D]")
>>> labels = PeltDetector().fit_detect(TimeSeries.from_arrays(time, values))
>>> np.flatnonzero(labels.values.ravel() == 1.0)
array([60])
Source code in src/hazure/methods/pelt_detector.py
PeltScorer
¶
Bases: _BreakpointScorer
Segment the series exactly, by pruned exact linear time search.
PELT [1]_ minimises the penalised segmentation cost by dynamic programming over the series::
F(t) = min over s < t of F(s) + cost(s..t) + penalty
where F(t) is the best total cost of segmenting the first t
observations. Evaluated naively that is quadratic. The pruning rule is what
makes it near-linear: once a candidate start s satisfies
F(s) + cost(s..t) > F(t)
it can never be part of an optimal segmentation of any longer prefix either,
because extending the segment only adds cost — so s is discarded for good
rather than reconsidered at every later step. The answer is nevertheless the
exact optimum, not an approximation of it: nothing is pruned that could have
won.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
penalty
|
float | None
|
Cost of admitting one more segment. None derives a value from the data; see Notes. Larger means fewer breakpoints. |
None
|
cost
|
Cost
|
|
'l2'
|
min_size
|
int
|
Shortest segment allowed, in observations. Also the resolution of the answer: no two breakpoints will be closer than this. |
2
|
jump
|
int
|
Consider only breakpoints at multiples of this many observations. 1 considers every position; larger values trade resolution for speed. |
1
|
Attributes:
| Name | Type | Description |
|---|---|---|
breakpoints_ |
ndarray
|
Positions at which a new segment starts. |
penalty_ |
float
|
The penalty used, whether given or derived. |
Raises:
| Type | Description |
|---|---|
ValueError
|
|
Notes
The default penalty. With no penalty supplied, a BIC-style value is used:
2 * sigma**2 * log(n) of squared error is what an extra segment has to
buy to be worth having. The information criterion charges log(n) per free
parameter, and an extra segment introduces two — where the change happened,
and the level after it — which is where the factor of two comes from. Charging
for only the level admits visibly spurious segments: on pure noise, that
weaker penalty found five breakpoints in 200 observations where this one finds
none.
sigma is estimated from the median absolute difference of consecutive
observations, scaled by 1.4826 / sqrt(2). Differencing is what makes that
estimate usable here: it removes any level, so the changes being searched for
do not inflate the noise estimate that decides how many of them are real,
which the standard deviation of the raw series certainly would. The median
absolute difference is in turn immune to the individual jumps at the
breakpoints themselves; 1.4826 puts a median absolute deviation on a
standard-deviation scale, and 1 / sqrt(2) undoes the doubling of variance
that differencing introduces. For cost="l1" the penalty is
2 * sigma * log(n) instead, because absolute deviations are measured in
units of the series and squared ones in units of its square: the same
expression would otherwise mean something different for each cost.
Missing observations are skipped. They contribute nothing to a segment's cost, so a gap neither creates nor hides a breakpoint.
Cost. The l2 cost of any segment is O(1) from cumulative sums of the
values and their squares, which is what makes the whole search fast. The
l1 cost has no such summary — a median cannot be assembled from prefix
statistics — so each step sorts its surviving candidates' segments, and a long
series with many breakpoints will notice.
The dynamic programme is a genuine sequential recurrence: each step's answer depends on the previous ones, so the loop over observations cannot be vectorised away. The work inside each step is vectorised across all surviving candidates.
References
.. [1] R. Killick, P. Fearnhead and I. A. Eckley, "Optimal Detection of Changepoints With a Linear Computational Cost", Journal of the American Statistical Association 107(500), 2012, pp. 1590-1598.
Examples:
A series that steps from 0 to 10 half way through:
>>> import numpy as np
>>> from hazure import TimeSeries
>>> rng = np.random.default_rng(0)
>>> values = np.concatenate([rng.normal(size=60), rng.normal(loc=10.0, size=60)])
>>> time = np.arange("2024-01-01", "2024-04-30", dtype="datetime64[D]")
>>> scorer = PeltScorer().fit(TimeSeries.from_arrays(time, values))
>>> scorer.breakpoints_
array([60])
>>> scores = scorer.score(TimeSeries.from_arrays(time, values)).values.ravel()
>>> round(float(scores[60]), 1)
10.0
Source code in src/hazure/methods/pelt_scorer.py
RollingQuantileDetector
¶
RollingQuantileDetector(
window: Window,
low: float = 0.05,
high: float = 0.95,
factor: Factor = 3.0,
)
Bases: ScoreDetector
Flag the points that leave a normal range which follows the series.
:class:RollingQuantileScorer paired with a one-sided inter-quartile-range
rule on the excursion. What this buys over a global quantile rule is a range
that moves: a series drifting upwards over a month spends the second half of
it above any fixed upper quantile, so a global rule reports the drift, while a
rolling band follows the level and reports only departures from it.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
window
|
Window
|
Observations ( |
required |
low
|
float
|
Lower quantile of the band, in |
0.05
|
high
|
float
|
Upper quantile of the band, in |
0.95
|
factor
|
Factor
|
Inter-quartile-range factor deciding how large an excursion beyond the band is too large. One-sided: a point inside the band is never anomalous. |
3.0
|
Raises:
| Type | Description |
|---|---|
ValueError
|
A quantile falls outside |
Notes
Why a fitted fence rather than "outside the band at all": the window trails and includes the point being scored, so on any series with a trend the newest observation is routinely the largest in its own window and sits at the band's own edge. Treating every non-zero excursion as an anomaly flags a sizeable fraction of a plainly ordinary series — 17 of the 40 points in the example below. The excursions are a distribution like any other, and what matters is an excursion out of proportion to the rest of them.
Consequently the band's quantiles and factor do different jobs. The band
decides what counts as an excursion at all; factor decides which
excursions get reported. Widening the band shrinks every excursion toward
zero, which eventually leaves the fence with nothing to separate — a window
spanning most of the series flags nothing, correctly.
An int window leaves the first window - 1 observations NaN, since the
band is not yet estimable. A duration window does not: it reports from the
first observation, on however few observations the duration covers.
Examples:
A series drifting upwards, with one point that breaks the drift:
>>> import numpy as np
>>> from hazure import TimeSeries
>>> from hazure.methods import RollingQuantileDetector
>>> values = np.arange(40.0) + np.tile([0.0, 0.5, -0.5, 0.2], 10)
>>> values[26] += 12.0
>>> time = np.arange("2024-01-01", "2024-02-10", dtype="datetime64[D]")
>>> labels = RollingQuantileDetector(window=10).fit_detect(
... TimeSeries.from_arrays(time, values)
... )
>>> np.flatnonzero(labels.values.ravel() == 1.0)
array([26])
Source code in src/hazure/methods/rolling_quantile_detector.py
RollingQuantileScorer
¶
Bases: BaseScorer
Score each point by how far outside a rolling quantile band it sits.
The band is the low and high quantiles of the window ending at each
point; the score is the distance beyond whichever edge was crossed, and zero
inside the band. Two order statistics per point, nothing fitted.
What this buys over a quantile of the whole series is a normal range that moves. A series that drifts upwards over a month spends the second half of it above any global upper quantile, and a global rule then reports the drift rather than the anomalies; a rolling band follows the level and keeps reporting only departures from it.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
window
|
Window
|
Observations ( |
required |
low
|
float
|
Lower quantile of the band, in |
0.05
|
high
|
float
|
Upper quantile of the band, in |
0.95
|
Raises:
| Type | Description |
|---|---|
ValueError
|
A quantile falls outside |
Notes
The window trails, and it includes the point being scored. A point extreme enough to move the band's own edge therefore scores lower than it otherwise would, which is a conservative bias — the band is harder to leave, not easier. Widening the window weakens the effect.
The score is a magnitude in the units of the series, and it is zero, not NaN, inside the band: "inside the normal range" is a measurement, not a missing one.
Nothing is learned; :meth:fit is optional.
Examples:
A series drifting upwards with one point that breaks the drift:
>>> import numpy as np
>>> from hazure import TimeSeries
>>> values = np.arange(40.0) + np.tile([0.0, 0.5, -0.5, 0.2], 10)
>>> values[26] += 12.0
>>> time = np.arange("2024-01-01", "2024-02-10", dtype="datetime64[D]")
>>> scores = RollingQuantileScorer(window=10).score(
... TimeSeries.from_arrays(time, values)
... )
>>> int(np.nanargmax(scores.values.ravel()))
26
Source code in src/hazure/methods/rolling_quantile_scorer.py
RupturesDetector
¶
RupturesDetector(
model: str = "binseg",
cost: str = "l2",
penalty: float | None = None,
n_bkps: int | None = None,
)
Bases: ScoreDetector
Flag the points at which the series changed regime, via ruptures.
:class:RupturesScorer with a threshold that passes its non-zero scores
through, exactly as :class:PeltDetector does for the built-in search. There
is no factor to tune, because the search has already decided which changes are
worth a segment: raise penalty, or set n_bkps, to report fewer.
Reach for this over :class:PeltDetector for one of two reasons — a cost model
hazure does not implement ("rbf" and "normal" detect changes in
distribution rather than only in mean), or a known number of changes,
which n_bkps states directly instead of tuning a penalty backwards into it.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
model
|
str
|
Search strategy: |
'binseg'
|
cost
|
str
|
A cost model name |
'l2'
|
penalty
|
float | None
|
Cost of admitting one more segment. Ignored when |
None
|
n_bkps
|
int | None
|
Ask for exactly this many breakpoints instead of penalising their number. |
None
|
Raises:
| Type | Description |
|---|---|
ValueError
|
|
ImportError
|
|
Notes
Like every change-point detector here, this reports the moment of change and
then goes quiet — not the stretch that follows it. Evaluating it against
interval-shaped ground truth shows near-zero recall while it is working
perfectly; reduce the truth to change points and allow a tolerance with
:func:~hazure.events.expand_events.
ruptures requires Python below 3.14, so this is unavailable on newer
interpreters; :class:PeltDetector needs nothing beyond numpy.
Examples:
>>> from hazure.methods import RupturesDetector
>>> RupturesDetector(model="dynp", n_bkps=1)
RupturesDetector(model='dynp', n_bkps=1)
Source code in src/hazure/methods/ruptures_detector.py
RupturesScorer
¶
RupturesScorer(
model: str = "binseg",
cost: str = "l2",
penalty: float | None = None,
n_bkps: int | None = None,
)
Bases: _BreakpointScorer
Segment the series with one of the ruptures search strategies.
An adapter, for the searches the built-in :class:PeltScorer does not
provide. The one that earns its keep is n_bkps: binary segmentation and
dynamic programming can both be asked for exactly k breakpoints, which is
the natural way to state the problem when the number of regimes is known and
a penalty would have to be tuned backwards into it.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
model
|
str
|
Search strategy: |
'binseg'
|
cost
|
str
|
A cost model name |
'l2'
|
penalty
|
float | None
|
Cost of admitting one more segment. Ignored when |
None
|
n_bkps
|
int | None
|
Ask for exactly this many breakpoints instead of penalising their number. |
None
|
Attributes:
| Name | Type | Description |
|---|---|---|
breakpoints_ |
ndarray
|
Positions at which a new segment starts. |
penalty_ |
float
|
The penalty used, or NaN when |
Raises:
| Type | Description |
|---|---|
ValueError
|
|
ImportError
|
|
Notes
ruptures requires Python below 3.14, so this adapter is unavailable on
newer interpreters. :class:PeltScorer is the portable option: it needs
nothing beyond numpy, runs everywhere hazure runs, and solves the penalised
problem exactly.
Missing observations are filled by linear interpolation before the search,
because the cost models take a dense matrix. Where that is unacceptable, use
:class:PeltScorer, which skips them.
Examples:
>>> from hazure.methods import RupturesScorer
>>> RupturesScorer(model="dynp", n_bkps=1)
RupturesScorer(model='dynp', n_bkps=1)
Source code in src/hazure/methods/ruptures_scorer.py
SpectralResidualDetector
¶
SpectralResidualDetector(
window: int = 3,
series_window: int = 21,
score_window: int = 21,
factor: Factor = 3.0,
)
Bases: ScoreDetector
Flag points whose saliency stands far above its neighbourhood.
:class:SpectralResidualScorer paired with an inter-quartile-range rule on
the score. Useful when the series has structure nobody has characterised: no
period, trend or distribution has to be named, and a point is reported when
its saliency is out of proportion to the saliency around it.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
window
|
int
|
Width, in frequency bins, of the moving average over the log amplitude spectrum. |
3
|
series_window
|
int
|
Trailing observations the right-edge extrapolation is estimated from. |
21
|
score_window
|
int
|
Trailing saliency points each point is compared against. |
21
|
factor
|
Factor
|
Inter-quartile-range factor deciding how high a relative saliency is too high. One-sided: a low saliency is never interesting. |
3.0
|
Raises:
| Type | Description |
|---|---|
ValueError
|
A window is out of range, |
Examples:
>>> import numpy as np
>>> from hazure import TimeSeries
>>> values = 10.0 + np.sin(np.arange(200) * np.pi / 12)
>>> values[137] = 30.0
>>> time = np.arange(200) * np.timedelta64(1, "h") + np.datetime64("2024-01-01")
>>> labels = SpectralResidualDetector(factor=12.0).fit_detect(
... TimeSeries.from_arrays(time, values)
... )
>>> np.flatnonzero(labels.values.ravel() == 1.0)
array([137])
Source code in src/hazure/methods/spectral_residual_detector.py
SpectralResidualScorer
¶
Bases: BaseScorer
Score each point by how far its saliency exceeds its neighbourhood.
Three steps, all in the frequency domain:
- Take the discrete Fourier transform of the series and split it into a log amplitude spectrum and a phase.
- Subtract a moving average (
windowwide) from the log amplitude. What an average of neighbouring frequencies predicts is the expected spectrum of a well-behaved signal; the difference — the spectral residual — is what the signal does that its own smooth spectral envelope does not explain. - Invert the transform from the residual amplitude and the original phase. The magnitude of the result is the saliency map: it is near zero wherever the series is either flat or regularly structured, and rises where the series does something unaccounted for.
Saliency is then judged locally, by dividing each point by the average
saliency of the score_window points up to and including it, so the score
is dimensionless and a busy series does not read as anomalous throughout.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
window
|
int
|
Width, in frequency bins, of the moving average subtracted from the log amplitude spectrum. Small: a wide average would smooth away the very structure it is supposed to predict. |
3
|
series_window
|
int
|
How many trailing observations the extrapolation of the right edge is estimated from. See Notes. |
21
|
score_window
|
int
|
How many trailing points of the saliency map each point is compared against. |
21
|
Raises:
| Type | Description |
|---|---|
ValueError
|
A window is less than 1, |
Notes
A discrete transform assumes the series repeats, so it behaves badly at the
right edge: the last few points are the ones whose saliency is most distorted
by the jump between the end of the series and its wrapped-around beginning.
That edge is exactly the part anyone monitoring a live series cares about —
the most recent point is usually the point in question. So before the
transform, _EXTENSION points are appended, each one step further along the
average gradient of the last series_window observations, and after the
saliency map is built the extrapolated tail is discarded. The edge artefact
then falls on the invented points rather than on the real ones, which is what
makes the score usable on the most recent observation.
Missing observations are filled by linear interpolation before the transform, since a transform has no notion of a gap, and their scores are set back to NaN afterwards.
Because the score is a ratio, it is never zero: the first point's neighbourhood is itself, so its score is exactly 1 by construction, and a series with no anomalies scores around 1 throughout rather than around 0. It is the relative height that carries the meaning, which is what makes one threshold factor usable across series of wildly different scales.
Nothing is learned: the saliency map is a property of the series being
scored, so :meth:fit is optional and each call recomputes it.
Examples:
A single spike in an otherwise smooth series is the most salient point:
>>> import numpy as np
>>> from hazure import TimeSeries
>>> values = 10.0 + np.sin(np.arange(200) * np.pi / 12)
>>> values[137] = 30.0
>>> time = np.arange(200) * np.timedelta64(1, "h") + np.datetime64("2024-01-01")
>>> scores = SpectralResidualScorer().score(TimeSeries.from_arrays(time, values))
>>> int(np.nanargmax(scores.values.ravel()))
137
Source code in src/hazure/methods/spectral_residual_scorer.py
StlDetector
¶
StlDetector(
period: int | None = None,
robust: bool = True,
seasonal: int | None = None,
factor: Factor = 3.0,
)
Bases: ScoreDetector
Flag points an STL decomposition cannot account for.
:class:StlResidualScorer paired with an inter-quartile-range rule on the
residual magnitudes. The rule is learned from the residuals rather than fixed,
because how large a residual is large depends entirely on how well the
decomposition fits the series in the first place.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
period
|
int | None
|
Length of the cycle, in observations. None derives it from the sampling interval. |
None
|
robust
|
bool
|
Reweight the loess fits to discount outliers. |
True
|
seasonal
|
int | None
|
Length of the seasonal smoother, an odd number of at least 7. |
None
|
factor
|
Factor
|
Inter-quartile-range factor deciding how large a residual is too large. |
3.0
|
Raises:
| Type | Description |
|---|---|
ValueError
|
The time axis is irregular, or the period is unusable. |
ImportError
|
|
Examples:
>>> import numpy as np
>>> from hazure import TimeSeries
>>> rng = np.random.default_rng(0)
>>> hours = np.arange(240)
>>> values = 10.0 + 3.0 * np.sin(hours * 2 * np.pi / 24) + rng.normal(size=240)
>>> values[100] += 12.0
>>> time = hours * np.timedelta64(1, "h") + np.datetime64("2024-01-01")
>>> labels = StlDetector(factor=6.0).fit_detect(
... TimeSeries.from_arrays(time, values)
... )
>>> np.flatnonzero(labels.values.ravel() == 1.0)
array([100])
Source code in src/hazure/methods/stl_detector.py
StlResidualScorer
¶
Bases: BaseScorer
Score each point by the size of its STL residual.
The series is decomposed into a trend, a seasonal component of one period, and a remainder; the score is the magnitude of that remainder. A point scores high when it is far from what the rhythm and the drift of the series together predicted for its position — which is a different and usually sharper question than whether its value is unusual outright.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
period
|
int | None
|
Length of the cycle, in observations. None derives it from the sampling interval: one day's worth of observations for anything sampled more often than daily, and one week for daily data. Pass it explicitly whenever the cycle is neither. |
None
|
robust
|
bool
|
Reweight the loess fits to discount outliers. On by default, and the reason to keep it on is circular in a useful way: the anomalies are exactly what must not be allowed to shape the decomposition they are being measured against. Turning it off is faster and appropriate for clean data. |
True
|
seasonal
|
int | None
|
Length of the smoother applied to the seasonal component, in observations;
must be an odd number of at least 7. Larger means a seasonal shape that
changes more slowly over the series. None leaves it to |
None
|
Raises:
| Type | Description |
|---|---|
ValueError
|
The time axis is irregular, |
ImportError
|
|
Notes
Requires a regular time axis: a period expressed in observations only means something if observations are evenly spaced. Missing observations are filled by linear interpolation, because the loess fits need a dense series, and their scores are set back to NaN afterwards.
The score is a magnitude, so it is one-sided by construction. Where the direction of the departure matters, take the residual's sign from the decomposition itself.
Nothing is learned: STL is a smoother, not a model that predicts, so it has to
see the series it is decomposing and :meth:fit is optional. A consequence
worth knowing is that the decomposition is retrospective — every point's score
depends on the whole series, including what came after it.
References
.. [1] R. B. Cleveland, W. S. Cleveland, J. E. McRae and I. Terpenning, "STL: A Seasonal-Trend Decomposition Procedure Based on Loess", Journal of Official Statistics 6(1), 1990, pp. 3-73.
Examples:
>>> import numpy as np
>>> from hazure import TimeSeries
>>> hours = np.arange(240)
>>> values = 10.0 + 3.0 * np.sin(hours * 2 * np.pi / 24)
>>> values[100] += 9.0
>>> time = hours * np.timedelta64(1, "h") + np.datetime64("2024-01-01")
>>> scorer = StlResidualScorer()
>>> scores = scorer.score(TimeSeries.from_arrays(time, values))
>>> int(np.nanargmax(scores.values.ravel()))
100
Source code in src/hazure/methods/stl_residual_scorer.py
hazure.datasets¶
Series to try things on: generated with a known answer, or fetched from the Numenta Anomaly Benchmark.
datasets
¶
Series to try things on, and a loop for comparing detectors over them.
Two sources, for two different questions.
:func:make_series generates one, with anomalies of a chosen shape planted at a
chosen strength. Nothing is downloaded, the answer is exact, and the strength is a
dial — which makes it the right tool for "at what point does this detector stop
seeing a level shift", a question no fixed benchmark can answer.
:func:load_nab fetches a series from the Numenta Anomaly Benchmark, labelled by
hand from real systems. Real data misbehaves in ways nobody thinks to generate:
gaps, drift, seasonality that is nearly but not quite daily, and anomalies that
are arguable. The labels are fixed and public, so a number quoted against them
means something to somebody else.
:func:compare runs several detectors over either kind and lays the metrics side
by side, alert counts included — because recall alone can be bought.
Dataset
dataclass
¶
Dataset(
name: str, data: Any, events: Events, description: str
)
A series and the intervals somebody says are anomalous in it.
Ground truth is carried as :class:~hazure.Events rather than as a label
column, because that is the form it arrives in and the form it means. An
incident is a stretch of time; which samples fall inside it depends on the
sampling interval, and a benchmark whose labels were snapped onto one axis
cannot be resampled onto another without quietly changing what it claims.
:attr:labels does the conversion when a metric wants it.
Attributes:
| Name | Type | Description |
|---|---|---|
name |
str
|
What this series is, short enough to use as a row label in a table. |
data |
Any
|
The series itself, in whatever backend the loader was asked for. |
events |
Events
|
The anomalous intervals. |
description |
str
|
Where it came from, or what was planted in it and how strongly. |
See Also
hazure.datasets.make_series : Generate one of these with a known answer. hazure.datasets.load_nab : Load one from the Numenta Anomaly Benchmark.
Examples:
>>> from hazure.datasets import make_series
>>> dataset = make_series("spike", n=1000, n_anomalies=2)
>>> dataset.name, dataset.events.n_events
('spike', 2)
>>> print(dataset.description)
2 x spike (the level jumps up for a moment...), 1 sample(s) each at strength 8...
labels
property
¶
The ground truth as a label series on :attr:data's own time axis.
Returns:
| Type | Description |
|---|---|
Any
|
|
compare
¶
compare(
detectors: Mapping[str, BaseDetector],
dataset: Dataset,
*,
thresh: float = 0.5,
fit_on: Any = None,
) -> dict[str, dict[str, float]]
Score several detectors against one dataset's ground truth.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
detectors
|
Mapping[str, BaseDetector]
|
Detectors to compare, keyed by whatever name should appear in the result. Each is used as given, so anything already fitted stays fitted. |
required |
dataset
|
Dataset
|
What to run them on, from :func: |
required |
thresh
|
float
|
Coverage threshold for the event-based metrics: the fraction of an event
that must be covered for it to count. In |
0.5
|
fit_on
|
Any
|
Fit each detector on this before running it, instead of on |
None
|
Returns:
| Type | Description |
|---|---|
dict of dict
|
One entry per detector, holding |
Raises:
| Type | Description |
|---|---|
TypeError
|
|
ValueError
|
A detector returned labels for more than one column. |
See Also
hazure.evaluation : The metrics this is a loop over.
hazure.budget_threshold : Choosing a cut-off from what alerts costs.
Notes
alerts is there because the other three can all be bought with it. A
detector that flags a third of the series will catch everything, and the
event-based precision that ought to punish it often does not — a wide alert
overlapping a true event counts as justified. Read the four together.
Examples:
Three detectors against three planted level shifts. Only one of them is built to see a change in the mean at all:
>>> from hazure import IqrDetector, LevelShiftDetector, SpikeDetector
>>> from hazure.datasets import compare, make_series
>>> dataset = make_series("level_shift", n=2000, n_anomalies=3, strength=5.0)
>>> table = compare(
... {
... "iqr": IqrDetector(),
... "spike": SpikeDetector(window=24),
... "shift": LevelShiftDetector(window=24),
... },
... dataset,
... )
>>> table["shift"]["recall"], table["iqr"]["recall"]
(1.0, 0.0)
Source code in src/hazure/datasets/benchmark.py
load_nab
¶
load_nab(
name: str,
*,
cache_dir: str | Path | None = None,
download: bool = True,
backend: str | None = None,
) -> Dataset
Load one labelled series from the Numenta Anomaly Benchmark.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
name
|
str
|
Which series, as |
required |
cache_dir
|
str | Path | None
|
Where to keep downloaded files. Defaults to |
None
|
download
|
bool
|
Fetch what is not cached. With |
True
|
backend
|
str | None
|
Emit the series into this dataframe backend — |
None
|
Returns:
| Type | Description |
|---|---|
Dataset
|
The series, its labelled anomaly windows, and its name. |
Raises:
| Type | Description |
|---|---|
KeyError
|
The benchmark has no series by that name. |
OSError
|
A needed file is neither cached nor reachable. |
See Also
nab_names : What names this accepts. hazure.datasets.make_series : Synthetic series, no network and a known answer.
Notes
A NAB label window is a stretch of wall-clock time centred on the anomaly and
deliberately wider than it, because the benchmark scores early detection
favourably. So the windows are what the benchmark's own scoring uses, and they
are not a claim that every sample inside one is abnormal — an event-based
:func:~hazure.recall against them is the metric they support, and a
sample-based :func:~hazure.precision against them is not really answerable.
The series are irregular in places: several have gaps of hours where collection stopped. Detectors configured with sample counts are unaffected; ones configured with durations will find the gaps, which is a property of the data worth knowing about rather than one to paper over.
The files come from tag v1.1 of the benchmark's repository, so a cached
copy and a fresh download are the same bytes.
Examples:
Needs the network the first time, and nothing after that::
>>> from hazure.datasets import load_nab, nab_names
>>> nab_names()[:2] # doctest: +SKIP
['artificialNoAnomaly/art_daily_no_noise',
'artificialNoAnomaly/art_daily_perfect_square_wave']
>>> taxi = load_nab("realKnownCause/nyc_taxi") # doctest: +SKIP
>>> taxi # doctest: +SKIP
Dataset('realKnownCause/nyc_taxi', 10320 rows, 5 events)
Source code in src/hazure/datasets/nab.py
96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 | |
nab_names
¶
List the series the benchmark has labels for.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
cache_dir
|
str | Path | None
|
Where to keep downloaded files. Defaults to |
None
|
download
|
bool
|
Fetch the catalogue if it is not cached. With |
True
|
Returns:
| Type | Description |
|---|---|
list of str
|
Names accepted by :func: |
Raises:
| Type | Description |
|---|---|
OSError
|
The catalogue is neither cached nor reachable. |
See Also
load_nab : Load one of these.
Source code in src/hazure/datasets/nab.py
make_series
¶
make_series(
kind: str = "spike",
*,
n: int = 2000,
n_anomalies: int = 3,
width: int | None = None,
strength: float = 8.0,
noise: float = 1.0,
level: float = 100.0,
period: int | None = None,
amplitude: float = 20.0,
freq: str | timedelta | timedelta64 = "5min",
start: str = "2024-01-01",
seed: int = 0,
backend: str | None = None,
) -> Dataset
Generate a series with anomalies of one shape planted in it.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
kind
|
str
|
Which shape to plant. One of the keys of :data: |
'spike'
|
n
|
int
|
Length of the series in samples. |
2000
|
n_anomalies
|
int
|
How many to plant. They are spaced evenly across the middle 70% of the series, so none of them straddles an end where a rolling window has nothing to compare against — a detector missing an anomaly in its first window is a fact about the window, not about the detector. |
3
|
width
|
int | None
|
Length of each anomaly in samples. Defaults to something sensible for the kind: 1 for a spike or dip, 24 for the three that are only visible over a stretch. |
None
|
strength
|
float
|
How obvious each anomaly is, in multiples of |
8.0
|
noise
|
float
|
Standard deviation of the Gaussian noise on the normal stretches. |
1.0
|
level
|
float
|
The baseline the series sits at. |
100.0
|
period
|
int | None
|
Samples per seasonal cycle, or None for no seasonality. Required for
|
None
|
amplitude
|
float
|
Peak deviation of the seasonal component, when |
20.0
|
freq
|
str | timedelta | timedelta64
|
Sampling interval: a duration string such as |
'5min'
|
start
|
str
|
Timestamp of the first sample, as an ISO 8601 string. |
'2024-01-01'
|
seed
|
int
|
Seed for the noise, so a given call always produces the same series. |
0
|
backend
|
str | None
|
Emit the series into this dataframe backend — |
None
|
Returns:
| Type | Description |
|---|---|
Dataset
|
The series, the intervals the anomalies occupy, and a description of what was planted. |
Raises:
| Type | Description |
|---|---|
KeyError
|
|
ValueError
|
An argument is non-positive, |
See Also
hazure.datasets.load_nab : Real series, with labels somebody else argued over.
Examples:
Three spikes in a fortnight of five-minute samples:
>>> from hazure.datasets import make_series
>>> dataset = make_series("spike", n=4032, n_anomalies=3)
>>> dataset.events.n_events
3
>>> dataset
Dataset('spike', 4032 rows, 3 events)
The ground truth is intervals, and comes back as labels on the series' own time axis when that is what a metric wants:
A level shift, which is invisible in the value distribution — the shifted samples are all well inside the normal range — and obvious in the rolling mean:
>>> shifted = make_series("level_shift", n=2000, strength=4.0)
>>> from hazure import IqrDetector, LevelShiftDetector
>>> from hazure.evaluation import recall
>>> recall(shifted.events, to_events(IqrDetector().fit_detect(shifted.data)))
0.0
>>> detector = LevelShiftDetector(window=24)
>>> recall(shifted.events, to_events(detector.fit_detect(shifted.data)))
1.0
Source code in src/hazure/datasets/synthetic.py
65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 | |
hazure.plotting¶
Drawing a series, its verdicts and its scores. Needs the viz extra.
plotting
¶
Looking at a series, its anomalies and its scores.
Detection output is easiest to judge by eye, so this module offers one function,
:func:plot, which draws a series together with any number of verdicts about it.
Three things shaped it:
- Verdicts have to be comparable.
anomalyaccepts a mapping of names to label series or :class:~hazure.events.Events, and each one gets its own colour on the same time axis, so "did these two detectors agree" is a glance rather than a join. - Scores and data share no units. A score is drawn on its own panel below the data rather than on top of it, so a score in the thousands cannot flatten the series it came from.
- Nothing global changes. matplotlib is imported inside the function, no
style sheet is installed and
rcParamsis only ever read, so a caller's theme survives the call. Wrap the call inmatplotlib.style.context(...)to draw under a different style.
matplotlib is an optional extra: pip install hazure[viz].
plot
¶
plot(
series: Any = None,
*,
anomaly: Any = None,
score: Any = None,
layout: str | Sequence[str | Sequence[str]] = "each",
axes: Sequence[Axes] | None = None,
figsize: tuple[float, float] | None = None,
style: str = "span",
palette: Sequence[Any]
| Mapping[str, Any]
| None = None,
alpha: float = 0.3,
legend: bool = True,
title: str | None = None,
) -> tuple[Figure, list[Axes]]
Draw a time series with anomalies marked and scores panelled below it.
Every argument is optional, and at least one of series, anomaly and
score must be given. Passing only anomaly draws the flagged intervals
on a bare time axis, which is a useful way to compare several detectors'
verdicts without the data getting in the way.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
series
|
Any
|
The data to draw. Any pandas / polars / pyarrow object with a time axis,
or a :class: |
None
|
anomaly
|
Any
|
Anomalies to mark. One of:
|
None
|
score
|
Any
|
Continuous scores, as a :class: |
None
|
layout
|
str | Sequence[str | Sequence[str]]
|
How to distribute |
'each'
|
axes
|
Sequence[Axes] | None
|
Draw into these axes instead of creating a figure. Must hold at least as many axes as the layout needs; the surplus is left untouched. |
None
|
figsize
|
tuple[float, float] | None
|
Figure size in inches. Defaults to a width of 10 and 2.2 per panel.
Ignored when |
None
|
style
|
str
|
|
'span'
|
palette
|
Sequence[Any] | Mapping[str, Any] | None
|
Colours for the anomaly overlays, overriding the property cycle. Either a sequence, used in order and cycled, or a mapping from anomaly name to colour, which must cover every name. |
None
|
alpha
|
float
|
Opacity of the shaded spans, in |
0.3
|
legend
|
bool
|
Add a legend to each panel that has something to name. |
True
|
title
|
str | None
|
Figure title. A single-panel figure gets it as an axes title instead, so it sits closer to the data. |
None
|
Returns:
| Type | Description |
|---|---|
Figure
|
The figure drawn into, so it can be saved and closed by the caller. |
list of matplotlib.axes.Axes
|
The panels used, data first and scores after, in drawing order. |
Raises:
| Type | Description |
|---|---|
ImportError
|
matplotlib is not installed. |
ValueError
|
Nothing was passed to draw, |
Examples:
>>> import numpy as np, pandas as pd
>>> from hazure.detection import SpikeDetector
>>> from hazure.plotting import plot
>>> index = pd.date_range("2024-01-01", periods=500, freq="h")
...
>>> values = pd.Series(np.sin(np.arange(500) / 12), index=index)
...
>>> labels = SpikeDetector(window=24).fit_detect(values)
>>> fig, axs = plot(values, anomaly=labels)
Compare two detectors on one chart, and read the score that drove them:
>>> from hazure.scoring import DeviationScorer
>>> fig, axs = plot(
... values,
... anomaly={"spike": labels, "level": other_labels},
... score=DeviationScorer().fit_score(values),
... style="marker",
... )
>>> fig.savefig("anomalies.png")
Source code in src/hazure/plotting.py
53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 | |