Ionosphere learning from repetitive patterns

By default learning from repetitive patterns is not enabled because you should define the metrics to exclude from this learning before enabling it.

Exclude and include

You need to decide what metrics you want to exclude from learning from repetitive patterns. You do not want Skyline learning from repetitive patterns on bad metrics, like metrics that are related to errors, 50x status codes, access_denied, etc. These types of metrics can be excluded from the learning by defining them in the settings.IONOSPHERE_REPETITIVE_PATTERNS_EXCLUDE.

For convenience sake there is a settings.IONOSPHERE_REPETITIVE_PATTERNS_INCLUDE setting which allows you to define only certain metrics to be learnt from repetitive patterns. If it is defined it is evaluated before the EXCLUDE to filter only the metrics that match definitions. The EXCLUDE (if defined) is then applied to the INCLUDE metrics list. This setting is to allow for testing repetitive learning with limited set of metrics before implementing it on the entire metric population.

For a detailed description of the data structures both of these dictionaries see the annotated example under settings.IONOSPHERE_REPETITIVE_PATTERNS_EXCLUDE in


Two methods of learning from repetitive patterns can be implemented in Ionosphere. Each method has slight differences and running both achieves the best results.

Often metrics exhibit repetitive patterns where there are large fluctuations periodically. A number of these are caused by normal operations of a process, for example something like prometheus_tsdb_compactions_total or VictoriaMetrics process_io_write_syscalls_total are processes that occur periodically and tend to generate spikes of similar and varying magnitude. There are many other types of metrics that fit into this category, things like backups which generally cause spikes in disk I/O, network bandwidth, etc, and other things like cron jobs, log rotation, etc, that run on a scheduled basis can result in these types of patterns in metrics.

The learn_repetitive_patterns processes evaluates current and previous anomalies on metrics and calculate the features profile for each of these, then comparisons are made between all the calculated features profiles and if 3 are found to be similar then these patterns are learnt as being normal behaviour.

The two methods, namely learn_repetitive_patterns and find_repetitive_patterns are described here, both are similar but each searches for a different kind of pattern.


This method learns daily periodic patterns. In new metrics these patterns can be found and learnt after 10 days. Thereafter if the pattern changes the method can learn the new daily periodic pattern after 7 days.

This method is run periodically against metrics that have training data. It is used to find metrics that exhibit daily patterns in terms of anomalies which occur during similar periods, e.g. between 00h00 and 00h15 or 03h15 and 03h30. Processes such as cron jobs, compaction, log rotation, storage bucket rolling, etc, all tend to cause this type of behaviour in metrics. The training data for each metric is evaluated and any training data that is discovered with anomalies aligned in similar periods/windows are evaluated. The evaluation compares the features profile sums of each training data set and if 3 training data sets are found to be significantly similar, these are classified as normal and they are automatically trained on.


This method can learn patterns in metrics after 7 days.

This method is run periodically against all anomalies. Unlike learn_repetitive_patterns this method does not use training data or period alignment, it uses anomalies from the metric in the last 30 days. This results in finding patterns of normal behaviour that are not necessarily periodically aligned but that can occur sporadically but are normal over a longer period and happen frequently enough to be considered as normal.

After the normal Ionosphere learn window has passed, the find_repetitive_patterns process determines all anomalies which have not been trained on since the last evaluation (the first run considers only the previous 24 hours).

  • The process surfaces every anomaly that occurred on the metric in the past 30 days.

  • The time series data for each of these anomalies is then fetched and the features profile sum is calculated for each anomalous time series.

  • These feature profile sums are then compared to the features profile sums of all the other anomalies, which do not fall in same weekly period as the feature profile sum being evaluated. This means that only anomalies that are in different weeks are evaluated against each other and anomalies in the same weekly window are not considered. Ensuring that the patterns found occur frequently and are not just a representation of recent behaviour.

  • A confusion matrix of similarity is generated and if any 3 or more are found to be similar they are classified as normal and trained upon.

An implementation factor to be aware of is that due to the fact that normal MinMax scaling checks and comparisons that occur in the normal Ionosphere process based on similar ranges is not implemented here. MinMax scaling is implemented via threshold on the average value of the time series defined in settings.IONOSPHERE_REPETITIVE_PATTERNS_MINMAX_AVG_VALUE. Due to the fact that patterns and not absolutes are being looked for here and that the features profiles being compared are separated by at least 7 days, the features profile sums should be sufficient to differentiate dissimilar patterns. There is a very small possibility that the MinMax scaling will result in false positive matches, however the metric would have to change proportionally in average, peak and trough values for this to occur which is probably unlikely and even if it did, unless those changes were significant in magnitude, it is desirable that they do match as similar.

settings.IONOSPHERE_REPETITIVE_PATTERNS_MINMAX_AVG_VALUE can be set to 0 to disable MinMax scaling comparisons.