skyline.mirage package
Submodules
skyline.mirage.agent module
skyline.mirage.mirage module
mirage.py
- class Mirage(parent_pid)[source]
Bases:
Thread
The Mirage thread
- spawn_alerter_process(alert, metric, second_order_resolution_seconds, context, triggered_algorithms)[source]
Spawn a process to trigger an alert. This is used by smtp alerters so that matplotlib objects are cleared down and the alerter cannot create a memory leak in this manner and plt.savefig keeps the object in memory until the process terminates. Seeing as data is being surfaced and processed in the alert_smtp context, multiprocessing the alert creation and handling prevents any memory leaks in the parent. # @added 20160814 - Bug #1558: Memory leak in Analyzer # Issue #21 Memory leak in Analyzer # https://github.com/earthgecko/skyline/issues/21
- mirage_load_metric_vars(metric_vars_file)[source]
Load the metric variables for a check from a metric check variables file
- Parameters:
metric_vars_file (str) – the path and filename to the metric variables files
- Returns:
the metric_vars list or
False
- Return type:
list
- dump_garbage()[source]
DEVELOPMENT ONLY
# @added 20160806 - Bug #1558: Memory leak in Analyzer # Debug with garbage collection - http://code.activestate.com/recipes/65333/
show us what’s the garbage about
- populate_redis(i, metric)[source]
Get FULL_DURATION data from Graphite for a metric and populate Redis
skyline.mirage.mirage_alerters module
- LOCAL_DEBUG = False
Create any alerter you want here. The function will be invoked from trigger_alert. Four arguments will be passed, two of them tuples: alert and metric.
alert: the tuple specified in your settings:
alert[0]: The matched substring of the anomalous metric
alert[1]: the name of the strategy being used to alert
alert[2]: The timeout of the alert that was triggered
alert[3]: The SECOND_ORDER_RESOLUTION_HOURS
metric: information about the anomaly itself
metric[0]: the anomalous value
metric[1]: The full name of the anomalous metric
metric[2]: anomaly timestamp
second_order_resolution_seconds: int context: app name
- alert_smtp(alert, metric, second_order_resolution_seconds, context, triggered_algorithms)[source]
Called by
trigger_alert()
and sends an alert via smtp to the recipients that are configured for the metric.
- alert_pagerduty(alert, metric, second_order_resolution_seconds, context, triggered_algorithms)[source]
Called by
trigger_alert()
and sends an alert via PagerDuty
- alert_hipchat(alert, metric, second_order_resolution_seconds, context, triggered_algorithms)[source]
Called by
trigger_alert()
and sends an alert the hipchat room that is configured in settings.py.
- alert_syslog(alert, metric, second_order_resolution_seconds, context, triggered_algorithms)[source]
Called by
trigger_alert()
and logs anomalies to syslog.
- alert_http(alert, metric, second_order_resolution_seconds, context, triggered_algorithms)[source]
Called by
trigger_alert()
and sends anomalies to a http endpoint.
- alert_sms(alert, metric, second_order_resolution_seconds, context, triggered_algorithms)[source]
Called by
trigger_alert()
and sends anomalies to a SMS endpoint.
- trigger_alert(alert, metric, second_order_resolution_seconds, context, triggered_algorithms)[source]
Called by
run
to trigger an alert, Mirage passes two arguments, both of them tuples. The alerting strategy is determined and the approriate alert def is then called and passed the tuples.- Parameters:
alert –
The alert tuple specified in settings.py e.g. (‘stats.*’, ‘smtp’, 3600, 168)
alert[0]: The matched substring of the anomalous metric (str)
alert[1]: the name of the strategy being used to alert (str)
alert[2]: The timeout of the alert that was triggered (int)
alert[3]: The second order resolution hours [optional for Mirage] (int)
alert[4]: The type [optional for http_alerter only] (dict)
The snab_details [optional for SNAB and slack only] (list)
alert[5]: The snab_details [optional for SNAB and slack only] (list)
The anomaly_id [optional for http_alerter only] (list)
metric –
The metric tuple e.g. (2.345, ‘server-1.cpu.user’, 1462172400)
metric[0]: the anomalous value (float)
metric[1]: The base_name of the anomalous metric (str)
metric[2]: anomaly timestamp (float or int)
context (str) – app name
skyline.mirage.mirage_algorithms module
mirage_algorithms.py
- tail_avg(timeseries, second_order_resolution_seconds)[source]
This is a utility function used to calculate the average of the last three datapoints in the series as a measure, instead of just the last datapoint. It reduces noise, but it also reduces sensitivity and increases the delay to detection.
- median_absolute_deviation(timeseries, second_order_resolution_seconds)[source]
A timeseries is anomalous if the deviation of its latest datapoint with respect to the median is X times larger than the median of deviations.
- grubbs(timeseries, second_order_resolution_seconds)[source]
A timeseries is anomalous if the Z score is greater than the Grubb’s score.
- first_hour_average(timeseries, second_order_resolution_seconds)[source]
Calcuate the simple average over one hour, second order resolution seconds ago. A timeseries is anomalous if the average of the last three datapoints are outside of three standard deviations of this value.
- stddev_from_average(timeseries, second_order_resolution_seconds)[source]
A timeseries is anomalous if the absolute value of the average of the latest three datapoint minus the moving average is greater than three standard deviations of the average. This does not exponentially weight the MA and so is better for detecting anomalies with respect to the entire series.
- stddev_from_moving_average(timeseries, second_order_resolution_seconds)[source]
A timeseries is anomalous if the absolute value of the average of the latest three datapoint minus the moving average is greater than three standard deviations of the moving average. This is better for finding anomalies with respect to the short term trends.
- mean_subtraction_cumulation(timeseries, second_order_resolution_seconds)[source]
A timeseries is anomalous if the value of the next datapoint in the series is farther than three standard deviations out in cumulative terms after subtracting the mean from each data point.
- least_squares(timeseries, second_order_resolution_seconds)[source]
A timeseries is anomalous if the average of the last three datapoints on a projected least squares model is greater than three sigma.
- histogram_bins(timeseries, second_order_resolution_seconds)[source]
A timeseries is anomalous if the average of the last three datapoints falls into a histogram bin with less than 20 other datapoints (you’ll need to tweak that number depending on your data)
Returns: the size of the bin which contains the tail_avg. Smaller bin size means more anomalous.
- ks_test(timeseries, second_order_resolution_seconds)[source]
A timeseries is anomalous if 2 sample Kolmogorov-Smirnov test indicates that data distribution for last 10 minutes is different from last hour. It produces false positives on non-stationary series so Augmented Dickey-Fuller test applied to check for stationarity.
- get_function_name()[source]
This is a utility function is used to determine what algorithm is reporting an algorithm error when the record_algorithm_error is used.
- record_algorithm_error(algorithm_name, traceback_format_exc_string)[source]
This utility function is used to facilitate the traceback from any algorithm errors. The algorithm functions themselves we want to run super fast and without fail in terms of stopping the function returning and not reporting anything to the log, so the pythonic except is used to “sample” any algorithm errors to a tmp file and report once per run rather than spewing tons of errors into the log.
Note
- algorithm errors tmp file clean up
the algorithm error tmp files are handled and cleaned up in
Analyzer
after all the spawned processes are completed.
- Parameters:
algorithm_name (str) – the algoritm function name
traceback_format_exc_string (str) – the traceback_format_exc string
- Returns:
True
the error string was written to the algorithm_error_fileFalse
the error string was not written to the algorithm_error_file
- Return type:
boolean
- negatives_present(timeseries)[source]
Determine if there are negative number present in a time series
skyline.mirage.mirage_labelled_metrics module
mirage_labelled_metrics.py
- class MirageLabelledMetrics(parent_pid)[source]
Bases:
Thread
The MirageLabelledMetrics thread
- mirage_load_metric_vars(metric_vars_file)[source]
Load the metric variables for a check from a metric check variables file
- Parameters:
metric_vars_file (str) – the path and filename to the metric variables files
- Returns:
the metric_vars list or
False
- Return type:
list
skyline.mirage.mirage_vortex module
mirage_vortex.py
- class MirageVortex(parent_pid)[source]
Bases:
Thread
The MirageVortex thread
- check_consensus(consensus, triggered_algorithms, algorithms_run)[source]
Check is consensus is achieved and still possible
- add_training_data(request_id, vortex_metric_data, timeseries, ionosphere_enabled)[source]
Add training_data
- add_results(request_id, metric_data, analysis_start_time)[source]
Add result to the mirage.vortex Redis hash
- process_ionosphere_results(run_timestamp, ionosphere_results)[source]
Process results from Ionosphere.
- return_request_results(run_timestamp, return_results_for)[source]
Assign a metrics for a process to analyze.
- add_to_illuminance(run_timestamp, labelled_metric_name, metric_timestamp, value, triggered_algorithms)[source]
- send_anomaly_to_panorama(labelled_metric_name, value, from_timestamp, metric_timestamp, algorithms_run, triggered_algorithms)[source]
skyline.mirage.mirage_vortex_algorithms module
mirage_vortex_algorithms.py
- tail_avg(timeseries, second_order_resolution_seconds)[source]
This is a utility function used to calculate the average of the last three datapoints in the series as a measure, instead of just the last datapoint. It reduces noise, but it also reduces sensitivity and increases the delay to detection.
- median_absolute_deviation(timeseries, second_order_resolution_seconds)[source]
A timeseries is anomalous if the deviation of its latest datapoint with respect to the median is X times larger than the median of deviations.
- grubbs(timeseries, second_order_resolution_seconds)[source]
A timeseries is anomalous if the Z score is greater than the Grubb’s score.
- first_hour_average(timeseries, second_order_resolution_seconds)[source]
Calcuate the simple average over one hour, second order resolution seconds ago. A timeseries is anomalous if the average of the last three datapoints are outside of three standard deviations of this value.
- stddev_from_average(timeseries, second_order_resolution_seconds)[source]
A timeseries is anomalous if the absolute value of the average of the latest three datapoint minus the moving average is greater than three standard deviations of the average. This does not exponentially weight the MA and so is better for detecting anomalies with respect to the entire series.
- stddev_from_moving_average(timeseries, second_order_resolution_seconds)[source]
A timeseries is anomalous if the absolute value of the average of the latest three datapoint minus the moving average is greater than three standard deviations of the moving average. This is better for finding anomalies with respect to the short term trends.
- mean_subtraction_cumulation(timeseries, second_order_resolution_seconds)[source]
A timeseries is anomalous if the value of the next datapoint in the series is farther than three standard deviations out in cumulative terms after subtracting the mean from each data point.
- least_squares(timeseries, second_order_resolution_seconds)[source]
A timeseries is anomalous if the average of the last three datapoints on a projected least squares model is greater than three sigma.
- histogram_bins(timeseries, second_order_resolution_seconds)[source]
A timeseries is anomalous if the average of the last three datapoints falls into a histogram bin with less than 20 other datapoints (you’ll need to tweak that number depending on your data)
Returns: the size of the bin which contains the tail_avg. Smaller bin size means more anomalous.
- ks_test(timeseries, second_order_resolution_seconds)[source]
A timeseries is anomalous if 2 sample Kolmogorov-Smirnov test indicates that data distribution for last 10 minutes is different from last hour. It produces false positives on non-stationary series so Augmented Dickey-Fuller test applied to check for stationarity.
- get_function_name()[source]
This is a utility function is used to determine what algorithm is reporting an algorithm error when the record_algorithm_error is used.
- record_algorithm_error(algorithm_name, traceback_format_exc_string)[source]
This utility function is used to facilitate the traceback from any algorithm errors. The algorithm functions themselves we want to run super fast and without fail in terms of stopping the function returning and not reporting anything to the log, so the pythonic except is used to “sample” any algorithm errors to a tmp file and report once per run rather than spewing tons of errors into the log.
Note
- algorithm errors tmp file clean up
the algorithm error tmp files are handled and cleaned up in
Analyzer
after all the spawned processes are completed.
- Parameters:
algorithm_name (str) – the algoritm function name
traceback_format_exc_string (str) – the traceback_format_exc string
- Returns:
True
the error string was written to the algorithm_error_fileFalse
the error string was not written to the algorithm_error_file
- Return type:
boolean