skyline.webapp package
Submodules
skyline.webapp.api_get_analysed_events module
skyline.webapp.api_get_bulk_training_data module
api_get_bulk_training_data.py
skyline.webapp.api_get_flux_test_metrics module
skyline.webapp.api_get_fp_timeseries module
api_get_fp_timeseries.py
skyline.webapp.api_get_metric_analysed_events module
skyline.webapp.api_get_namespace_metric_count module
api_get_namespace_metric_count.py
- api_get_namespace_metric_count(current_skyline_app)[source]
Return a dict with the number of metrics in the namespace with totals for total_metrics, inactive_metrics and active_metrics.
- Parameters:
current_skyline_app (str) – the app calling the function
- Returns:
namespace_metric_count
- Return type:
dict
skyline.webapp.api_get_panorama_all_activity module
api_get_panorama_all_activity.py
- get_panorama_all_activity(current_skyline_app)[source]
Return a dict with the number of analysed events, estimated the first day based on dividing the total of the first day by the elapsed number of minutes that have past in the until_timestamp day.
- Parameters:
current_skyline_app (str) – the app calling the function
cluster_data (bool) – the cluster_data parameter from the request
- Returns:
analysed_events
- Return type:
dict
skyline.webapp.api_mute_alerts_on module
skyline.webapp.backend module
- panorama_request()[source]
Gets the details of anomalies from the database, using the URL arguments that are passed in by the
request.argsto build the MySQL select query string and queries the database, parse the results and creates an array of the anomalies that matched the query and creates thepanaroma.jsonfile, then returns the array. The Webapp needs both the array and the JSONP file to serve to the browser for the client sidepanaroma.js.- Parameters:
None – determined from
request.args- Returns:
array
- Return type:
array
Note
And creates
panaroma.jsfor client side javascript
- luminosity_remote_data(anomaly_timestamp, resolution, namespace=None)[source]
Gets all the unique_metrics from Redis and then mgets Redis data for all metrics. The data is then preprocessed for the remote Skyline luminosity instance and only the relevant fragments of the time series are returned. This return is then gzipped by the Flask Webapp response to ensure the minimum about of bandwidth is used.
- Parameters:
anomaly_timestamp (int) – the anomaly timestamp
- Returns:
list
- Return type:
list
- get_cluster_data(api_endpoint, data_required, only_host='all', endpoint_params={})[source]
Gets data from the /api of REMOTE_SKYLINE_INSTANCES. This allows the user to query a single Skyline webapp node in a cluster and the Skyline instance will respond with the concentated responses of all the REMOTE_SKYLINE_INSTANCES in one a single response.
- Parameters:
api_endpoint (str) – the api endpoint to request data from the remote Skyline instances
data_required (str) – the element from the api json response that is required
only_host (str) – The remote Skyline host to query, if not passed all are queried.
endpoint_params (dict) – A dictionary of any additional parameters that may be required
- Returns:
list
- Return type:
list
- no_numba_yhat_dict(x, y, means, lowers, uppers, anomaly_timestamps_indices, include_yhat_real_lower)[source]
- numba_yhat_dict(x, y, means, lowers, uppers, anomaly_timestamps_indices, include_yhat_real_lower)[source]
- get_yhat_values(metric, from_timestamp, until_timestamp, include_value, include_mean, include_yhat_real_lower, include_anomalous_periods)[source]
- get_mirage_not_anomalous_metrics(metric=None, from_timestamp=None, until_timestamp=None, anomalies=False)[source]
Determine mirage not anomalous metrics from mirage.panorama.not_anomalous_metrics and ionosphere.panorama.not_anomalous_metrics
- Parameters:
metric (str) – base_name
from_timestamp (int) – the from_timestamp
until_timestamp (int) – the until_timestamp
anomalies (boolean) – whether to report anomalies as well
- Returns:
(dict, dict)
- Return type:
tuple
- plot_not_anomalous_metric(not_anomalous_dict, anomalies_dict, plot_type)[source]
Plot the metric not anomalous or anomalies graph and return the file path
- Parameters:
not_anomalous_dict (dict) – the dictionary of not anomalous events for the metric
anomalies_dict (dict) – the dictionary of anomalous events for the metric
- Returns:
path and filename
- Return type:
str
- namespace_stale_metrics(namespace, cluster_data, exclude_sparsely_populated)[source]
Plot the metric not anomalous or anomalies graph and return the file path
- Parameters:
not_anomalous_dict (dict) – the dictionary of not anomalous events for the metric
anomalies_dict (dict) – the dictionary of anomalous events for the metric
- Returns:
path and filename
- Return type:
str
skyline.webapp.crucible_backend module
- submit_crucible_job(from_timestamp, until_timestamp, metrics_list, namespaces_list, source, alert_interval, user_id, user, add_to_panorama, pad_timeseries, training_data_json, run_algorithms, summarise)[source]
Get a list of all the metrics passed and generate Crucible check files for each
- Parameters:
from_timestamp (int) – the timestamp at which to start the time series
until_timestamp (int) – the timestamp at which to end the time series
metrics_list (list) – a list of metric names to analyse
namespaces_list (list) – a list of metric namespaces to analyse
source (str) – the source webapp making the request
alert_interval (int) – the alert_interval at which Crucible should trigger anomalies
user_id (int) – the user id of the user making the request
user (str) – the username making the request
add_to_panorama (boolean) – whether Crucible should add Skyline CONSENSUS anomalies to Panorama
pad_timeseries (str) – the amount of data to pad the time series with
training_data_json (str) – the full path to the training_data json file if source is training_data
run_algorithms (list) – list of algorithms to run
summarise (int) – the number of seconds to summarise data at
- Returns:
tuple of lists
- Return type:
(list, list, list, list)
Returns (crucible_job_id, metrics_submitted_to_process, fail_msg, trace)
- get_crucible_jobs()[source]
Get a list of all the metrics passed and generate Crucible check files for each
- Parameters:
requested_timestamp (str) – the training data timestamp
context (str) – the request context, training_data or features_profiles
- Returns:
tuple of lists
- Return type:
(list, list, list, list)
- get_crucible_job(crucible_job_id, metric)[source]
Get the crucible data for a Crucible analysis
- Parameters:
crucible_job_id (str) – the crucible_job_id
metric (str) – the metric name
- Returns:
tuple of lists
- Return type:
(list, boolean, boolean, list, list, list, list, str, str, str)
Returns (crucible_job_details, completed_job, has_anomalies, skyline_anomalies, skyline_consensus_anomalies, panorama_done, panorama_done_timestamp, panorama_done_user_id, image_files, image_file_names, graph_image_file, fail_msg, trace)
- send_crucible_job_metric_to_panorama(crucible_job_id, base_name, user_id, user, skyline_consensus_anomalies)[source]
Send the Crucible Skyline CONSENSUS anomalies for a crucible_job and metric to Panorama to insert into the anomalies database.
- Parameters:
crucible_job_id (str) – the crucible_job_id
base_name (str) – the metric name
user_id (int) – the user_id
user (str) – the username
skyline_consensus_anomalies (list) – the Crucible Skyline CONSENSUS anomalies
- Returns:
tuple of lists
- Return type:
(int, str, str)
Returns (len(skyline_consensus_anomalies), fail_msg, trace)
skyline.webapp.expose_metrics module
skyline.webapp.get_analysed_events module
get_analysed_events.py
- get_analysed_events(current_skyline_app, period, from_timestamp, until_timestamp, namespace=None)[source]
Return a dict with the number of analysed events, estimate the first day count based on dividing the total of the first day by the elapsed number of minutes that have past in the until_timestamp day.
- Parameters:
current_skyline_app (str) – the app calling the function
period (str) – get analysed events for
from_timestamp (int) – get analysed events from
until_timestamp (int) – get analysed events until
namespace (str) – get analysed events for namespace
- Returns:
analysed_events
- Return type:
dict
skyline.webapp.get_boundary_metrics module
skyline.webapp.get_saved_training_data module
skyline.webapp.get_snab_results module
skyline.webapp.gunicorn module
skyline.webapp.gunicorn_features_profile module
skyline.webapp.ionosphere_backend module
ionosphere_backend.py
- ionosphere_get_metrics_dir(requested_timestamp, context)[source]
Get a list of all the metrics in timestamp training data or features profile folder
- Parameters:
requested_timestamp (str) – the training data timestamp
context (str) – the request context, training_data or features_profiles
- Returns:
tuple of lists
- Return type:
(list, list, list, list)
- ionosphere_data(requested_timestamp, data_for_metric, context)[source]
Get a list of all training data or profiles folders and metrics
- Parameters:
requested_timestamp (str) – the training data or profile timestamp
data_for_metric (str) – the metric base_name
context (str) – the request context, training_data or features_profiles
- Returns:
tuple of lists
- Return type:
(list, list, list, list)
- ionosphere_metric_data(requested_timestamp, data_for_metric, context, fp_id)[source]
Get a list of all training data folders and metrics
- features_profile_details(fp_id)[source]
Get the Ionosphere details of a fetures profile
- Parameters:
fp_id (str) – the features profile id
- Returns:
tuple
- Return type:
(str, boolean, str, str)
- ionosphere_search(default_query, search_query)[source]
Gets the details features profiles from the database, using the URL arguments that are passed in by the
request.argsto build the MySQL select query string and queries the database, parse the results and creates an array of the features profiles that matched the query.- Parameters:
None – determined from
request.args- Returns:
array
- Return type:
array
- create_ionosphere_layers(base_name, fp_id, requested_timestamp)[source]
Create a layers profile.
- Parameters:
None – determined from
request.args- Returns:
array
- Return type:
array
- feature_profile_layers_detail(fp_layers_id)[source]
Get the Ionosphere layers details of a fetures profile
- Parameters:
fp_layers_id – the features profile layers_id
- Returns:
tuple
- Return type:
(str, boolean, str, str, object)
- feature_profile_layer_alogrithms(fp_layers_id)[source]
Get the Ionosphere layer algorithm details of a layer
- Parameters:
fp_layers_id – the features profile layers_id
- Returns:
tuple
- Return type:
(str, boolean, str, str)
- metric_layers_alogrithms(base_name)[source]
Get the Ionosphere layer algorithm details of a metric
- Parameters:
base_name (str) – the metric base_name
- Returns:
tuple
- Return type:
(str, boolean, str, str)
- edit_ionosphere_layers(layers_id)[source]
Edit a layers profile.
- Parameters:
layers_id – the layer id to edit
- Returns:
array
- Return type:
array
- validate_fp(update_id, id_column_name, user_id)[source]
Validate a single features profile or validate all enabled, unvalidated features profiles for a metric_id.
- Parameters:
update_id (int) – the features profile id or metric_id to validate
id_column_name (str) – the column name to select where on, e.g. id or metric_id
user_id (int) – the user id of the user that is validating
- Returns:
tuple
- Return type:
(boolean, str, str)
- save_training_data_dir(timestamp, base_name, label, hdate)[source]
Save training_data and return details or just return details if exists
- Parameters:
timestamp (str) – the Ionosphere training_data metric timestamp
base_name (str) – metric base_name
label (str) – the saved training_data label
hdate (str) – human date for the saved training_data
- Returns:
saved_successful, details, fail_msg, trace
- Return type:
boolean, list, str, str
- features_profile_family_tree(fp_id)[source]
Returns the all features profile ids of the related progeny features profiles, the whole family tree.
- Parameters:
fp_id – the features profile id
- Returns:
array
- Return type:
array
- disable_features_profile_family_tree(fp_ids)[source]
Disable a features profile and all related progeny features profiles
- Parameters:
fp_ids – a list of the the features profile ids to disable
- Returns:
array
- Return type:
array
- get_fp_matches(metric, metric_like, get_fp_id, get_layer_id, from_timestamp, until_timestamp, limit, sort)[source]
Get all the matches.
- Parameters:
metric – all or the metric name
metric_like – False or the metric MySQL like string e.g statsd.%
get_fp_id – None or int
get_layer_id – None or int
from_timestamp – timestamp or None
until_timestamp – timestamp or None
limit – None or number to limit to
sort – DESC or ASC
- Returns:
list
- Return type:
list
- get_matched_id_resources(matched_id, matched_by, metric, requested_timestamp)[source]
Get the Ionosphere matched details of a features profile or layer
- Parameters:
matched_id – the matched id
matched_by – either features_profile, layers or motif
metric – metric base_name
requested_timestamp – the timestamp of the features profile
- Returns:
tuple
- Return type:
(str, boolean, str, str)
- get_features_profiles_to_validate(base_name)[source]
Get the details for Ionosphere features profiles that need to be validated for a metric and returns a list of the details for each of the features profile including the ionosphere_image API URIs for all the relevant graph images for the weabpp Ionosphere validate_features_profiles page. For example:
[[ fp_id, metric_id, metric, fp_full_duration, anomaly_timestamp, fp_parent_id, parent_full_duration, parent_anomaly_timestamp, fp_date, fp_graph_uri, parent_fp_date, parent_fp_graph_uri, parent_parent_fp_id, fp_learn_graph_uri, parent_fp_learn_graph_uri, minimum_full_duration, maximum_full_duration, generation]]
- Parameters:
base_name (str) – metric base_name
- Returns:
list of lists
- Return type:
[[int, int, str, int, int, int, int, int, str, str, str, str, int, str, str, int, int]]
- get_metrics_with_features_profiles_to_validate()[source]
Get the metrics with Ionosphere features profiles that need to be validated and return a list of the details for each metric. [[metric_id, metric, fps_to_validate_count]]
- Returns:
list of lists
- Return type:
[[int, str, int]]
- ionosphere_show_graphs(requested_timestamp, data_for_metric, fp_id)[source]
Get a list of all graphs
- webapp_update_slack_thread(base_name, metric_timestamp, value, message_context)[source]
Update slack threads with enabled events.
- Parameters:
base_name (str) – metric base_name
metric_timestamp (str or int) – the anomaly_timestamp
value (int or None) – the features profile id, the validated_count or None
message_context (str) – training_data_viewed or layers_created
- Returns:
True or False
- Return type:
boolean
- validate_ionosphere_match(match_id, validate_context, match_validated, user_id)[source]
Update the validated value in the DB for the match.
- Parameters:
match_id (str) – the match id
validate_context (str) – the context to validate either ionosphere_matched or ionosphere_layers_matched
match_validated (int) – 1 for valid or 2 for invalid
user_id (int) – the user id of the user validating
- Returns:
True or False
- Return type:
boolean
- label_anomalies(start_timestamp, end_timestamp, metrics, namespaces, label)[source]
Label all anomalies from start_timestamp to end_timestamp for metrics and and metric namespaces with the given label.
- Parameters:
start_timestamp (int) – the start timestamp
end_timestamp (int) – the end timestamp
metrics (list) – a list of metric names
namespaces – a list of namespaces
- Returns:
boolean
- Return type:
(list, list, list, list)
- expected_features_profiles_dirs()[source]
Generate a dict of all features_profiles ids and directories that SHOULD exist.
- Return type:
dict
- get_matched_motifs(metric, metric_like, from_timestamp, until_timestamp, sort_by)[source]
Get all motif matches.
- Parameters:
metric – all or the metric name
metric_like – False or the metric MySQL like string e.g statsd.%
from_timestamp – timestamp or None
until_timestamp – timestamp or None
limit – None or number to limit to
- Returns:
list
- Return type:
list
skyline.webapp.ionosphere_performance module
- get_ionosphere_performance(metric, metric_like, from_timestamp, until_timestamp, output_format, anomalies, new_fps, fps_matched_count, layers_matched_count, sum_matches, title, period, height, width, fp_type, timezone_str, min_full_duration)[source]
Analyse the performance of Ionosphere on a metric or metric namespace and create the graph resources or json data as required.
- Return type:
dict
skyline.webapp.luminosity_backend module
- get_classify_metrics(base_name, timestamp, algorithm, significant)[source]
Get a list of all the files for metrics in classify_metrics
- Parameters:
base_name (str) – the name of the metric
timestamp (str) – the timestamp
algorithm (str) – the algorithm
significant – whether to return only significant results
- Returns:
list of dicts
- Return type:
{}
Returns a dict of algorithm, metrics, timestamps and significance {
- “classify_metrics”: {
- “level_shift”: {
- ‘metric1’: {
1604596200: {‘siginifcant’: True}, 1606410600: {‘siginifcant’: True}, 1607015400: {‘siginifcant’: True}, 1602177000: {‘siginifcant’: False}
}, ‘metric2’: {
1604596200: {‘siginifcant’: True}, 1606410600: {‘siginifcant’: True}, 1607015400: {‘siginifcant’: True}, 1602177000: {‘siginifcant’: False}
},
}, “volatility_shift”: {
- ‘metric1’: {
1604596200: {‘siginifcant’: True}, 1606410600: {‘siginifcant’: True}, 1607015400: {‘siginifcant’: True}, 1602177000: {‘siginifcant’: False}
}, ‘metric2’: {
1604596200: {‘siginifcant’: True}, 1606410600: {‘siginifcant’: True}, 1607015400: {‘siginifcant’: True}, 1602177000: {‘siginifcant’: False}
},
}
}
}
skyline.webapp.luminosity_cloudbursts module
- get_filtered_metrics(redis_conn_decoded, namespaces)[source]
Get create a list of filter_by_metrics.
- Parameters:
redis_conn_decoded (str) – the redis_conn_decoded object
namespaces – the namespaces to match
from_timestamp – the from_timestamp
until_timestamp – the until_timestamp
- Returns:
list of metrics
- Return type:
list
- get_metric_ids(redis_conn_decoded, filter_by_metrics)[source]
Get create a list of metric ids and dict of metric_names_with_ids.
- Parameters:
redis_conn_decoded (str) – the redis_conn_decoded object
namespaces – the namespaces to match
from_timestamp – the from_timestamp
until_timestamp – the until_timestamp
- Returns:
(list of metrics, dict of metric_names_with_ids)
- Return type:
(list, dict)
- get_cloudbursts(metric, namespaces, from_timestamp, until_timestamp)[source]
Get create a dict of all the cloudbursts.
- Parameters:
metric (str) – the name of the metric
namespaces (list) – the namespaces to match
from_timestamp (int) – the from_timestamp
until_timestamp (int) – the until_timestamp
- Returns:
dict of cloudbursts
- Return type:
{}
Returns a dict of cloudbursts {
- “cloudbursts”: {
- <id>: {
‘metric_id’: <int>, ‘metric’: <str>, ‘timestamp’: <int>, ‘end’: <int>, ‘duration’: <int>, ‘duration’: <int>, ‘from_timestamp’: <int>, ‘resolution’: <int>, ‘full_duration’: <int>, ‘anomaly_id’: <int>, ‘match_id’: <int>, ‘fp_id’: <int>, ‘layer_id’: <int>, ‘added_at’: <int>,
},
}
}
skyline.webapp.luminosity_plot_cloudburst module
- get_cloudburst_plot(cloudburst_id, base_name, shift, all_in_period=False)[source]
Create a plot of the cloudburst and return the path and filename
- Parameters:
cloudburst_id (int) – the cloudburt id
base_name (str) – the name of the metric
shift (int) – the number of indice to shift the plot
- Returns:
path and file
- Return type:
str
skyline.webapp.on_demand_motif_analysis module
skyline.webapp.panorama_plot_anomalies module
Plot anomalies for a metric
- panorama_plot_anomalies(base_name, from_timestamp=None, until_timestamp=None, matches=False)[source]
Create a plot of the metric with its anomalies and return the anomalies dict, the path and filename and the matches_dict
- Parameters:
base_name (str) – the name of the metric
from_timestamp (int) – the from timestamp
until_timestamp (int) – the until timestamp
matches (bool) – whether to plot matches as well
- Returns:
(anomalies_dict, path and file, matches_dict)
- Return type:
tuple
skyline.webapp.snab_backend module
skyline.webapp.timeseries_graph module
Plot anomalies for a metric
- timeseries_graph()[source]
Create a plot of the metric and return the path and filename
- Parameters:
metric (str) – the name of the metric
from_timestamp (int) – the from timestamp
until_timestamp (int) – the until timestamp
data_source (str) – what data_source to use
- Returns:
path and file
- Return type:
str
skyline.webapp.utilities module
- alerts_matcher(base_name, pattern, alerter, second_order_resolution_hours)[source]
Get a list of all the metrics that would match an ALERTS pattern
- Parameters:
base_name (str) – The metric name
pattern (str) – the ALERT pattern
alerter (str) – the alerter name e.g. smtp, syslog, hipchat, pagerdaty
second_order_resolution_hours (int) – (optional) The number of hours that Mirage should surface the metric timeseries for
- Returns:
matched_by
- Return type:
str
(‘metric3’, ‘hipchat’, 600), # Log all anomalies to syslog (‘stats.’, ‘syslog’, 1), # Wildcard namespaces can be used as well (‘metric4.thing.*.requests’, ‘stmp’, 900), # However beware of wildcards as the above wildcard should really be (‘metric4.thing..*..requests’, ‘stmp’, 900),
skyline.webapp.vortex module
skyline.webapp.webapp module
webapp.py
- limit_remote_addr()[source]
This function is called to check if the requesting IP address is in the settings.WEBAPP_ALLOWED_IPS array, if not 403.
- check_auth(username, password)[source]
This function is called to check if a username / password combination is valid.
- internal_error(message, traceback_format_exc)[source]
Show traceback in the browser when running a flask app on a production server. By default, flask does not show any useful information when running on a production server. By adding this view, we output the Python traceback to the error 500 page and log.
As per: Show flask traceback when running on production server https://gist.github.com/probonopd/8616a8ff05c8a75e4601 - Python traceback rendered nicely by Jinja2
This can be tested by hitting SKYLINE_URL/a_500
- get_client_details()[source]
Gets the first X-Forwarded-For address and sets as the IP address. Gets the client_id by simply using a md5 hash of the client IP address and user agent. Determines whether the request was proxied. Determines the client protocol.
- Returns:
client_id, protocol, proxied, salt
- Return type:
str, str, boolean, str
- decode_token(client_id)[source]
Use the app.secret, client_id and salt to decode the token JWT encoded payload and determine the Redis password.
- Parameters:
client_id (str) – the client_id string
- Returns:
token, decoded_redis_password, fail_msg, trace
- Return type:
str, str, str, str
skyline.webapp.webapp_features_profile module
webapp_features_profile.py