skyline.webapp package¶
Submodules¶
skyline.webapp.backend module¶
-
panorama_request
()[source]¶ Gets the details of anomalies from the database, using the URL arguments that are passed in by the
request.args
to build the MySQL select query string and queries the database, parse the results and creates an array of the anomalies that matched the query and creates thepanaroma.json
file, then returns the array. The Webapp needs both the array and the JSONP file to serve to the browser for the client sidepanaroma.js
.Parameters: None – determined from request.args
Returns: array Return type: array Note
And creates
panaroma.js
for client side javascript
-
get_list
(thing)[source]¶ Get a list of names for things in a database table.
Parameters: thing (str) – the thing, e.g. ‘algorithm’ Returns: list Return type: list
-
luminosity_remote_data
(anomaly_timestamp, resolution)[source]¶ Gets all the unique_metrics from Redis and then mgets Redis data for all metrics. The data is then preprocessed for the remote Skyline luminosity instance and only the relevant fragments of the time series are returned. This return is then gzipped by the Flask Webapp response to ensure the minimum about of bandwidth is used.
Parameters: anomaly_timestamp (int) – the anomaly timestamp Returns: list Return type: list
-
get_cluster_data
(api_endpoint, data_required, only_host='all', endpoint_params={})[source]¶ Gets data from the /api of REMOTE_SKYLINE_INSTANCES. This allows the user to query a single Skyline webapp node in a cluster and the Skyline instance will respond with the concentated responses of all the REMOTE_SKYLINE_INSTANCES in one a single response.
Parameters: - api_endpoint (str) – the api endpoint to request data from the remote Skyline instances
- data_required (str) – the element from the api json response that is required
- only_host (str) – The remote Skyline host to query, if not passed all are queried.
- endpoint_params (dict) – A dictionary of any additional parameters that may be required
Returns: list
Return type: list
-
get_yhat_values
(metric, from_timestamp, until_timestamp, include_value, include_mean, include_yhat_real_lower, include_anomalous_periods)[source]¶
-
get_mirage_not_anomalous_metrics
(metric=None, from_timestamp=None, until_timestamp=None, anomalies=False)[source]¶ Determine mirage not anomalous metrics from mirage.panorama.not_anomalous_metrics and ionosphere.panorama.not_anomalous_metrics
Parameters: - metric (str) – base_name
- from_timestamp (int) – the from_timestamp
- until_timestamp (int) – the until_timestamp
- anomalies (boolean) – whether to report anomalies as well
Returns: (dict, dict)
Return type: tuple
-
plot_not_anomalous_metric
(not_anomalous_dict, anomalies_dict, plot_type)[source]¶ Plot the metric not anomalous or anomalies graph and return the file path
Parameters: - not_anomalous_dict (dict) – the dictionary of not anomalous events for the metric
- anomalies_dict (dict) – the dictionary of anomalous events for the metric
Returns: path and filename
Return type: str
-
namespace_stale_metrics
(namespace, cluster_data, exclude_sparsely_populated)[source]¶ Plot the metric not anomalous or anomalies graph and return the file path
Parameters: - not_anomalous_dict (dict) – the dictionary of not anomalous events for the metric
- anomalies_dict (dict) – the dictionary of anomalous events for the metric
Returns: path and filename
Return type: str
skyline.webapp.crucible_backend module¶
-
submit_crucible_job
(from_timestamp, until_timestamp, metrics_list, namespaces_list, source, alert_interval, user_id, user, add_to_panorama, pad_timeseries, training_data_json, run_algorithms)[source]¶ Get a list of all the metrics passed and generate Crucible check files for each
Parameters: - from_timestamp (int) – the timestamp at which to start the time series
- until_timestamp (int) – the timestamp at which to end the time series
- metrics_list (list) – a list of metric names to analyse
- namespaces_list (list) – a list of metric namespaces to analyse
- source (str) – the source webapp making the request
- alert_interval (int) – the alert_interval at which Crucible should trigger anomalies
- user_id (int) – the user id of the user making the request
- user (str) – the username making the request
- add_to_panorama (boolean) – whether Crucible should add Skyline CONSENSUS anomalies to Panorama
- pad_timeseries (str) – the amount of data to pad the time series with
- training_data_json (str) – the full path to the training_data json file if source is training_data
- run_algorithms (list) – list of algorithms to run
Returns: tuple of lists
Return type: (list, list, list, list)
Returns (crucible_job_id, metrics_submitted_to_process, fail_msg, trace)
-
get_crucible_jobs
()[source]¶ Get a list of all the metrics passed and generate Crucible check files for each
Parameters: - requested_timestamp (str) – the training data timestamp
- context (str) – the request context, training_data or features_profiles
Returns: tuple of lists
Return type: (list, list, list, list)
-
get_crucible_job
(crucible_job_id, metric)[source]¶ Get the crucible data for a Crucible analysis
Parameters: - crucible_job_id (str) – the crucible_job_id
- metric (str) – the metric name
Returns: tuple of lists
Return type: (list, boolean, boolean, list, list, list, list, str, str, str)
Returns (crucible_job_details, completed_job, has_anomalies, skyline_anomalies, skyline_consensus_anomalies, panorama_done, panorama_done_timestamp, panorama_done_user_id, image_files, image_file_names, graph_image_file, fail_msg, trace)
-
send_crucible_job_metric_to_panorama
(crucible_job_id, base_name, user_id, user, skyline_consensus_anomalies)[source]¶ Send the Crucible Skyline CONSENSUS anomalies for a crucible_job and metric to Panorama to insert into the anomalies database.
Parameters: - crucible_job_id (str) – the crucible_job_id
- base_name (str) – the metric name
- user_id (int) – the user_id
- user (str) – the username
- skyline_consensus_anomalies (list) – the Crucible Skyline CONSENSUS anomalies
Returns: tuple of lists
Return type: (int, str, str)
Returns (len(skyline_consensus_anomalies), fail_msg, trace)
skyline.webapp.gunicorn module¶
skyline.webapp.ionosphere_backend module¶
-
ionosphere_get_metrics_dir
(requested_timestamp, context)[source]¶ Get a list of all the metrics in timestamp training data or features profile folder
Parameters: - requested_timestamp (str) – the training data timestamp
- context (str) – the request context, training_data or features_profiles
Returns: tuple of lists
Return type: (list, list, list, list)
-
ionosphere_data
(requested_timestamp, data_for_metric, context)[source]¶ Get a list of all training data or profiles folders and metrics
Parameters: - requested_timestamp (str) – the training data or profile timestamp
- data_for_metric (str) – the metric base_name
- context (str) – the request context, training_data or features_profiles
Returns: tuple of lists
Return type: (list, list, list, list)
-
ionosphere_metric_data
(requested_timestamp, data_for_metric, context, fp_id)[source]¶ Get a list of all training data folders and metrics
-
features_profile_details
(fp_id)[source]¶ Get the Ionosphere details of a fetures profile
Parameters: fp_id (str) – the features profile id Returns: tuple Return type: (str, boolean, str, str)
-
ionosphere_search
(default_query, search_query)[source]¶ Gets the details features profiles from the database, using the URL arguments that are passed in by the
request.args
to build the MySQL select query string and queries the database, parse the results and creates an array of the features profiles that matched the query.Parameters: None – determined from request.args
Returns: array Return type: array
-
create_ionosphere_layers
(base_name, fp_id, requested_timestamp)[source]¶ Create a layers profile.
Parameters: None – determined from request.args
Returns: array Return type: array
-
feature_profile_layers_detail
(fp_layers_id)[source]¶ Get the Ionosphere layers details of a fetures profile
Parameters: fp_layers_id – the features profile layers_id Returns: tuple Return type: (str, boolean, str, str, object)
-
feature_profile_layer_alogrithms
(fp_layers_id)[source]¶ Get the Ionosphere layer algorithm details of a layer
Parameters: fp_layers_id – the features profile layers_id Returns: tuple Return type: (str, boolean, str, str)
-
metric_layers_alogrithms
(base_name)[source]¶ Get the Ionosphere layer algorithm details of a metric
Parameters: base_name (str) – the metric base_name Returns: tuple Return type: (str, boolean, str, str)
-
edit_ionosphere_layers
(layers_id)[source]¶ Edit a layers profile.
Parameters: layers_id – the layer id to edit Returns: array Return type: array
-
validate_fp
(update_id, id_column_name, user_id)[source]¶ Validate a single features profile or validate all enabled, unvalidated features profiles for a metric_id.
Parameters: - update_id (int) – the features profile id or metric_id to validate
- id_column_name (str) – the column name to select where on, e.g. id or metric_id
- user_id (int) – the user id of the user that is validating
Returns: tuple
Return type: (boolean, str, str)
-
save_training_data_dir
(timestamp, base_name, label, hdate)[source]¶ Save training_data and return details or just return details if exists
Parameters: - timestamp (str) – the Ionosphere training_data metric timestamp
- base_name (str) – metric base_name
- label (str) – the saved training_data label
- hdate (str) – human date for the saved training_data
Returns: saved_successful, details, fail_msg, trace
Return type: boolean, list, str, str
-
features_profile_family_tree
(fp_id)[source]¶ Returns the all features profile ids of the related progeny features profiles, the whole family tree.
Parameters: fp_id – the features profile id Returns: array Return type: array
-
disable_features_profile_family_tree
(fp_ids)[source]¶ Disable a features profile and all related progeny features profiles
Parameters: fp_ids – a list of the the features profile ids to disable Returns: array Return type: array
-
get_fp_matches
(metric, metric_like, get_fp_id, get_layer_id, from_timestamp, until_timestamp, limit, sort)[source]¶ Get all the matches.
Parameters: - metric – all or the metric name
- metric_like – False or the metric MySQL like string e.g statsd.%
- get_fp_id – None or int
- get_layer_id – None or int
- from_timestamp – timestamp or None
- until_timestamp – timestamp or None
- limit – None or number to limit to
- sort – DESC or ASC
Returns: list
Return type: list
-
get_matched_id_resources
(matched_id, matched_by, metric, requested_timestamp)[source]¶ Get the Ionosphere matched details of a features profile or layer
Parameters: - matched_id – the matched id
- matched_by – either features_profile, layers or motif
- metric – metric base_name
- requested_timestamp – the timestamp of the features profile
Returns: tuple
Return type: (str, boolean, str, str)
-
get_features_profiles_to_validate
(base_name)[source]¶ Get the details for Ionosphere features profiles that need to be validated for a metric and returns a list of the details for each of the features profile including the ionosphere_image API URIs for all the relevant graph images for the weabpp Ionosphere validate_features_profiles page. For example:
[[ fp_id, metric_id, metric, fp_full_duration, anomaly_timestamp, fp_parent_id, parent_full_duration, parent_anomaly_timestamp, fp_date, fp_graph_uri, parent_fp_date, parent_fp_graph_uri, parent_parent_fp_id, fp_learn_graph_uri, parent_fp_learn_graph_uri, minimum_full_duration, maximum_full_duration, generation]]
Parameters: base_name (str) – metric base_name Returns: list of lists Return type: [[int, int, str, int, int, int, int, int, str, str, str, str, int, str, str, int, int]]
-
get_metrics_with_features_profiles_to_validate
()[source]¶ Get the metrics with Ionosphere features profiles that need to be validated and return a list of the details for each metric. [[metric_id, metric, fps_to_validate_count]]
Returns: list of lists Return type: [[int, str, int]]
-
ionosphere_show_graphs
(requested_timestamp, data_for_metric, fp_id)[source]¶ Get a list of all graphs
-
webapp_update_slack_thread
(base_name, metric_timestamp, value, message_context)[source]¶ Update slack threads with enabled events.
Parameters: - base_name (str) – metric base_name
- metric_timestamp (str or int) – the anomaly_timestamp
- value (int or None) – the features profile id, the validated_count or None
- message_context (str) – training_data_viewed or layers_created
Returns: True or False
Return type: boolean
-
validate_ionosphere_match
(match_id, validate_context, match_validated, user_id)[source]¶ Update the validated value in the DB for the match.
Parameters: - match_id (str) – the match id
- validate_context (str) – the context to validate either ionosphere_matched or ionosphere_layers_matched
- match_validated (int) – 1 for valid or 2 for invalid
- user_id (int) – the user id of the user validating
Returns: True or False
Return type: boolean
-
label_anomalies
(start_timestamp, end_timestamp, metrics, namespaces, label)[source]¶ Label all anomalies from start_timestamp to end_timestamp for metrics and and metric namespaces with the given label.
Parameters: - start_timestamp (int) – the start timestamp
- end_timestamp (int) – the end timestamp
- metrics (list) – a list of metric names
- namespaces – a list of namespaces
Returns: boolean
Return type: (list, list, list, list)
-
expected_features_profiles_dirs
()[source]¶ Generate a dict of all features_profiles ids and directories that SHOULD exist.
Return type: dict
-
get_matched_motifs
(metric, metric_like, from_timestamp, until_timestamp, sort_by)[source]¶ Get all motif matches.
Parameters: - metric – all or the metric name
- metric_like – False or the metric MySQL like string e.g statsd.%
- from_timestamp – timestamp or None
- until_timestamp – timestamp or None
- limit – None or number to limit to
Returns: list
Return type: list
skyline.webapp.ionosphere_performance module¶
-
get_ionosphere_performance
(metric, metric_like, from_timestamp, until_timestamp, format, anomalies, new_fps, fps_matched_count, layers_matched_count, sum_matches, title, period, height, width, fp_type, timezone_str)[source]¶ Analyse the performance of Ionosphere on a metric or metric namespace and create the graph resources or json data as required.
Return type: dict
skyline.webapp.luminosity_backend module¶
-
get_classify_metrics
(base_name, timestamp, algorithm, significant)[source]¶ Get a list of all the files for metrics in classify_metrics
Parameters: - base_name (str) – the name of the metric
- timestamp (str) – the timestamp
- algorithm (str) – the algorithm
- significant – whether to return only significant results
Returns: list of dicts
Return type: {}
Returns a dict of algorithm, metrics, timestamps and significance {
- “classify_metrics”: {
- “level_shift”: {
- ‘metric1’: {
- 1604596200: {‘siginifcant’: True}, 1606410600: {‘siginifcant’: True}, 1607015400: {‘siginifcant’: True}, 1602177000: {‘siginifcant’: False}
}, ‘metric2’: {
1604596200: {‘siginifcant’: True}, 1606410600: {‘siginifcant’: True}, 1607015400: {‘siginifcant’: True}, 1602177000: {‘siginifcant’: False}},
}, “volatility_shift”: {
- ‘metric1’: {
- 1604596200: {‘siginifcant’: True}, 1606410600: {‘siginifcant’: True}, 1607015400: {‘siginifcant’: True}, 1602177000: {‘siginifcant’: False}
}, ‘metric2’: {
1604596200: {‘siginifcant’: True}, 1606410600: {‘siginifcant’: True}, 1607015400: {‘siginifcant’: True}, 1602177000: {‘siginifcant’: False}},
}
}
}
skyline.webapp.on_demand_motif_analysis module¶
skyline.webapp.snab_backend module¶
skyline.webapp.utilities module¶
-
alerts_matcher
(base_name, pattern, alerter, second_order_resolution_hours)[source]¶ Get a list of all the metrics that would match an ALERTS pattern
Parameters: - base_name (str) – The metric name
- pattern (str) – the ALERT pattern
- alerter (str) – the alerter name e.g. smtp, syslog, hipchat, pagerdaty
- second_order_resolution_hours (int) – (optional) The number of hours that Mirage should surface the metric timeseries for
Returns: matched_by
Return type: str
(‘metric3’, ‘hipchat’, 600), # Log all anomalies to syslog (‘stats.’, ‘syslog’, 1), # Wildcard namespaces can be used as well (‘metric4.thing.*.requests’, ‘stmp’, 900), # However beware of wildcards as the above wildcard should really be (‘metric4.thing..*..requests’, ‘stmp’, 900),
skyline.webapp.webapp module¶
webapp.py
-
limit_remote_addr
()[source]¶ This function is called to check if the requesting IP address is in the settings.WEBAPP_ALLOWED_IPS array, if not 403.
-
check_auth
(username, password)[source]¶ This function is called to check if a username / password combination is valid.
-
internal_error
(message, traceback_format_exc)[source]¶ Show traceback in the browser when running a flask app on a production server. By default, flask does not show any useful information when running on a production server. By adding this view, we output the Python traceback to the error 500 page and log.
As per: Show flask traceback when running on production server https://gist.github.com/probonopd/8616a8ff05c8a75e4601 - Python traceback rendered nicely by Jinja2
This can be tested by hitting SKYLINE_URL/a_500
-
get_client_details
()[source]¶ Gets the first X-Forwarded-For address and sets as the IP address. Gets the client_id by simply using a md5 hash of the client IP address and user agent. Determines whether the request was proxied. Determines the client protocol.
Returns: client_id, protocol, proxied, salt Return type: str, str, boolean, str
-
decode_token
(client_id)[source]¶ Use the app.secret, client_id and salt to decode the token JWT encoded payload and determine the Redis password.
Parameters: client_id (str) – the client_id string Returns: token, decoded_redis_password, fail_msg, trace Return type: str, str, str, str