skyline.webapp package

Submodules

skyline.webapp.backend module

panorama_request()[source]

Gets the details of anomalies from the database, using the URL arguments that are passed in by the request.args to build the MySQL select query string and queries the database, parse the results and creates an array of the anomalies that matched the query and creates the panaroma.json file, then returns the array. The Webapp needs both the array and the JSONP file to serve to the browser for the client side panaroma.js.

Parameters:None – determined from request.args
Returns:array
Return type:array

Note

And creates panaroma.js for client side javascript

get_list(thing)[source]

Get a list of names for things in a database table.

Parameters:thing (str) – the thing, e.g. ‘algorithm’
Returns:list
Return type:list
luminosity_remote_data(anomaly_timestamp, resolution)[source]

Gets all the unique_metrics from Redis and then mgets Redis data for all metrics. The data is then preprocessed for the remote Skyline luminosity instance and only the relevant fragments of the time series are returned. This return is then gzipped by the Flask Webapp response to ensure the minimum about of bandwidth is used.

Parameters:anomaly_timestamp (int) – the anomaly timestamp
Returns:list
Return type:list
panorama_anomaly_details(anomaly_id)[source]

Gets the details for an anomaly from the database.

get_cluster_data(api_endpoint, data_required, only_host='all', endpoint_params={})[source]

Gets data from the /api of REMOTE_SKYLINE_INSTANCES. This allows the user to query a single Skyline webapp node in a cluster and the Skyline instance will respond with the concentated responses of all the REMOTE_SKYLINE_INSTANCES in one a single response.

Parameters:
  • api_endpoint (str) – the api endpoint to request data from the remote Skyline instances
  • data_required (str) – the element from the api json response that is required
  • only_host (str) – The remote Skyline host to query, if not passed all are queried.
  • endpoint_params (dict) – A dictionary of any additional parameters that may be required
Returns:

list

Return type:

list

get_yhat_values(metric, from_timestamp, until_timestamp, include_value, include_mean, include_yhat_real_lower, include_anomalous_periods)[source]
get_mirage_not_anomalous_metrics(metric=None, from_timestamp=None, until_timestamp=None, anomalies=False)[source]

Determine mirage not anomalous metrics from mirage.panorama.not_anomalous_metrics and ionosphere.panorama.not_anomalous_metrics

Parameters:
  • metric (str) – base_name
  • from_timestamp (int) – the from_timestamp
  • until_timestamp (int) – the until_timestamp
  • anomalies (boolean) – whether to report anomalies as well
Returns:

(dict, dict)

Return type:

tuple

plot_not_anomalous_metric(not_anomalous_dict, anomalies_dict, plot_type)[source]

Plot the metric not anomalous or anomalies graph and return the file path

Parameters:
  • not_anomalous_dict (dict) – the dictionary of not anomalous events for the metric
  • anomalies_dict (dict) – the dictionary of anomalous events for the metric
Returns:

path and filename

Return type:

str

namespace_stale_metrics(namespace, cluster_data, exclude_sparsely_populated)[source]

Plot the metric not anomalous or anomalies graph and return the file path

Parameters:
  • not_anomalous_dict (dict) – the dictionary of not anomalous events for the metric
  • anomalies_dict (dict) – the dictionary of anomalous events for the metric
Returns:

path and filename

Return type:

str

skyline.webapp.crucible_backend module

get_an_engine()[source]
engine_disposal(engine)[source]
submit_crucible_job(from_timestamp, until_timestamp, metrics_list, namespaces_list, source, alert_interval, user_id, user, add_to_panorama, pad_timeseries, training_data_json, run_algorithms)[source]

Get a list of all the metrics passed and generate Crucible check files for each

Parameters:
  • from_timestamp (int) – the timestamp at which to start the time series
  • until_timestamp (int) – the timestamp at which to end the time series
  • metrics_list (list) – a list of metric names to analyse
  • namespaces_list (list) – a list of metric namespaces to analyse
  • source (str) – the source webapp making the request
  • alert_interval (int) – the alert_interval at which Crucible should trigger anomalies
  • user_id (int) – the user id of the user making the request
  • user (str) – the username making the request
  • add_to_panorama (boolean) – whether Crucible should add Skyline CONSENSUS anomalies to Panorama
  • pad_timeseries (str) – the amount of data to pad the time series with
  • training_data_json (str) – the full path to the training_data json file if source is training_data
  • run_algorithms (list) – list of algorithms to run
Returns:

tuple of lists

Return type:

(list, list, list, list)

Returns (crucible_job_id, metrics_submitted_to_process, fail_msg, trace)

get_crucible_jobs()[source]

Get a list of all the metrics passed and generate Crucible check files for each

Parameters:
  • requested_timestamp (str) – the training data timestamp
  • context (str) – the request context, training_data or features_profiles
Returns:

tuple of lists

Return type:

(list, list, list, list)

get_crucible_job(crucible_job_id, metric)[source]

Get the crucible data for a Crucible analysis

Parameters:
  • crucible_job_id (str) – the crucible_job_id
  • metric (str) – the metric name
Returns:

tuple of lists

Return type:

(list, boolean, boolean, list, list, list, list, str, str, str)

Returns (crucible_job_details, completed_job, has_anomalies, skyline_anomalies, skyline_consensus_anomalies, panorama_done, panorama_done_timestamp, panorama_done_user_id, image_files, image_file_names, graph_image_file, fail_msg, trace)

send_crucible_job_metric_to_panorama(crucible_job_id, base_name, user_id, user, skyline_consensus_anomalies)[source]

Send the Crucible Skyline CONSENSUS anomalies for a crucible_job and metric to Panorama to insert into the anomalies database.

Parameters:
  • crucible_job_id (str) – the crucible_job_id
  • base_name (str) – the metric name
  • user_id (int) – the user_id
  • user (str) – the username
  • skyline_consensus_anomalies (list) – the Crucible Skyline CONSENSUS anomalies
Returns:

tuple of lists

Return type:

(int, str, str)

Returns (len(skyline_consensus_anomalies), fail_msg, trace)

skyline.webapp.gunicorn module

skyline.webapp.ionosphere_backend module

ionosphere_get_metrics_dir(requested_timestamp, context)[source]

Get a list of all the metrics in timestamp training data or features profile folder

Parameters:
  • requested_timestamp (str) – the training data timestamp
  • context (str) – the request context, training_data or features_profiles
Returns:

tuple of lists

Return type:

(list, list, list, list)

ionosphere_data(requested_timestamp, data_for_metric, context)[source]

Get a list of all training data or profiles folders and metrics

Parameters:
  • requested_timestamp (str) – the training data or profile timestamp
  • data_for_metric (str) – the metric base_name
  • context (str) – the request context, training_data or features_profiles
Returns:

tuple of lists

Return type:

(list, list, list, list)

get_an_engine()[source]
engine_disposal(engine)[source]
ionosphere_metric_data(requested_timestamp, data_for_metric, context, fp_id)[source]

Get a list of all training data folders and metrics

features_profile_details(fp_id)[source]

Get the Ionosphere details of a fetures profile

Parameters:fp_id (str) – the features profile id
Returns:tuple
Return type:(str, boolean, str, str)

Gets the details features profiles from the database, using the URL arguments that are passed in by the request.args to build the MySQL select query string and queries the database, parse the results and creates an array of the features profiles that matched the query.

Parameters:None – determined from request.args
Returns:array
Return type:array
create_ionosphere_layers(base_name, fp_id, requested_timestamp)[source]

Create a layers profile.

Parameters:None – determined from request.args
Returns:array
Return type:array
feature_profile_layers_detail(fp_layers_id)[source]

Get the Ionosphere layers details of a fetures profile

Parameters:fp_layers_id – the features profile layers_id
Returns:tuple
Return type:(str, boolean, str, str, object)
feature_profile_layer_alogrithms(fp_layers_id)[source]

Get the Ionosphere layer algorithm details of a layer

Parameters:fp_layers_id – the features profile layers_id
Returns:tuple
Return type:(str, boolean, str, str)
metric_layers_alogrithms(base_name)[source]

Get the Ionosphere layer algorithm details of a metric

Parameters:base_name (str) – the metric base_name
Returns:tuple
Return type:(str, boolean, str, str)
edit_ionosphere_layers(layers_id)[source]

Edit a layers profile.

Parameters:layers_id – the layer id to edit
Returns:array
Return type:array
validate_fp(update_id, id_column_name, user_id)[source]

Validate a single features profile or validate all enabled, unvalidated features profiles for a metric_id.

Parameters:
  • update_id (int) – the features profile id or metric_id to validate
  • id_column_name (str) – the column name to select where on, e.g. id or metric_id
  • user_id (int) – the user id of the user that is validating
Returns:

tuple

Return type:

(boolean, str, str)

save_training_data_dir(timestamp, base_name, label, hdate)[source]

Save training_data and return details or just return details if exists

Parameters:
  • timestamp (str) – the Ionosphere training_data metric timestamp
  • base_name (str) – metric base_name
  • label (str) – the saved training_data label
  • hdate (str) – human date for the saved training_data
Returns:

saved_successful, details, fail_msg, trace

Return type:

boolean, list, str, str

features_profile_family_tree(fp_id)[source]

Returns the all features profile ids of the related progeny features profiles, the whole family tree.

Parameters:fp_id – the features profile id
Returns:array
Return type:array
disable_features_profile_family_tree(fp_ids)[source]

Disable a features profile and all related progeny features profiles

Parameters:fp_ids – a list of the the features profile ids to disable
Returns:array
Return type:array
get_fp_matches(metric, metric_like, get_fp_id, get_layer_id, from_timestamp, until_timestamp, limit, sort)[source]

Get all the matches.

Parameters:
  • metric – all or the metric name
  • metric_like – False or the metric MySQL like string e.g statsd.%
  • get_fp_id – None or int
  • get_layer_id – None or int
  • from_timestamp – timestamp or None
  • until_timestamp – timestamp or None
  • limit – None or number to limit to
  • sort – DESC or ASC
Returns:

list

Return type:

list

get_matched_id_resources(matched_id, matched_by, metric, requested_timestamp)[source]

Get the Ionosphere matched details of a features profile or layer

Parameters:
  • matched_id – the matched id
  • matched_by – either features_profile, layers or motif
  • metric – metric base_name
  • requested_timestamp – the timestamp of the features profile
Returns:

tuple

Return type:

(str, boolean, str, str)

get_features_profiles_to_validate(base_name)[source]

Get the details for Ionosphere features profiles that need to be validated for a metric and returns a list of the details for each of the features profile including the ionosphere_image API URIs for all the relevant graph images for the weabpp Ionosphere validate_features_profiles page. For example:

[[  fp_id, metric_id, metric, fp_full_duration, anomaly_timestamp,
    fp_parent_id, parent_full_duration, parent_anomaly_timestamp, fp_date,
    fp_graph_uri, parent_fp_date, parent_fp_graph_uri, parent_parent_fp_id,
    fp_learn_graph_uri, parent_fp_learn_graph_uri, minimum_full_duration,
    maximum_full_duration, generation]]
Parameters:base_name (str) – metric base_name
Returns:list of lists
Return type:[[int, int, str, int, int, int, int, int, str, str, str, str, int, str, str, int, int]]
get_metrics_with_features_profiles_to_validate()[source]

Get the metrics with Ionosphere features profiles that need to be validated and return a list of the details for each metric. [[metric_id, metric, fps_to_validate_count]]

Returns:list of lists
Return type:[[int, str, int]]
ionosphere_show_graphs(requested_timestamp, data_for_metric, fp_id)[source]

Get a list of all graphs

webapp_update_slack_thread(base_name, metric_timestamp, value, message_context)[source]

Update slack threads with enabled events.

Parameters:
  • base_name (str) – metric base_name
  • metric_timestamp (str or int) – the anomaly_timestamp
  • value (int or None) – the features profile id, the validated_count or None
  • message_context (str) – training_data_viewed or layers_created
Returns:

True or False

Return type:

boolean

validate_ionosphere_match(match_id, validate_context, match_validated, user_id)[source]

Update the validated value in the DB for the match.

Parameters:
  • match_id (str) – the match id
  • validate_context (str) – the context to validate either ionosphere_matched or ionosphere_layers_matched
  • match_validated (int) – 1 for valid or 2 for invalid
  • user_id (int) – the user id of the user validating
Returns:

True or False

Return type:

boolean

label_anomalies(start_timestamp, end_timestamp, metrics, namespaces, label)[source]

Label all anomalies from start_timestamp to end_timestamp for metrics and and metric namespaces with the given label.

Parameters:
  • start_timestamp (int) – the start timestamp
  • end_timestamp (int) – the end timestamp
  • metrics (list) – a list of metric names
  • namespaces – a list of namespaces
Returns:

boolean

Return type:

(list, list, list, list)

expected_features_profiles_dirs()[source]

Generate a dict of all features_profiles ids and directories that SHOULD exist.

Return type:dict
get_matched_motifs(metric, metric_like, from_timestamp, until_timestamp, sort_by)[source]

Get all motif matches.

Parameters:
  • metric – all or the metric name
  • metric_like – False or the metric MySQL like string e.g statsd.%
  • from_timestamp – timestamp or None
  • until_timestamp – timestamp or None
  • limit – None or number to limit to
Returns:

list

Return type:

list

get_matched_motif_id(fp_id, timestamp, index, size)[source]

Return the matched_motif_id, motif_validated, ionosphere_matched_id for a motif

skyline.webapp.ionosphere_performance module

get_ionosphere_performance(metric, metric_like, from_timestamp, until_timestamp, format, anomalies, new_fps, fps_matched_count, layers_matched_count, sum_matches, title, period, height, width, fp_type, timezone_str)[source]

Analyse the performance of Ionosphere on a metric or metric namespace and create the graph resources or json data as required.

Return type:dict

skyline.webapp.luminosity_backend module

get_classify_metrics(base_name, timestamp, algorithm, significant)[source]

Get a list of all the files for metrics in classify_metrics

Parameters:
  • base_name (str) – the name of the metric
  • timestamp (str) – the timestamp
  • algorithm (str) – the algorithm
  • significant – whether to return only significant results
Returns:

list of dicts

Return type:

{}

Returns a dict of algorithm, metrics, timestamps and significance {

“classify_metrics”: {
“level_shift”: {
‘metric1’: {
1604596200: {‘siginifcant’: True}, 1606410600: {‘siginifcant’: True}, 1607015400: {‘siginifcant’: True}, 1602177000: {‘siginifcant’: False}

}, ‘metric2’: {

1604596200: {‘siginifcant’: True}, 1606410600: {‘siginifcant’: True}, 1607015400: {‘siginifcant’: True}, 1602177000: {‘siginifcant’: False}

},

}, “volatility_shift”: {

‘metric1’: {
1604596200: {‘siginifcant’: True}, 1606410600: {‘siginifcant’: True}, 1607015400: {‘siginifcant’: True}, 1602177000: {‘siginifcant’: False}

}, ‘metric2’: {

1604596200: {‘siginifcant’: True}, 1606410600: {‘siginifcant’: True}, 1607015400: {‘siginifcant’: True}, 1602177000: {‘siginifcant’: False}

},

}

}

}

skyline.webapp.on_demand_motif_analysis module

on_demand_motif_analysis(metric, timestamp, similarity, batch_size, top_matches, max_distance, range_padding, max_area_percent_diff)[source]

Process a motif similarity search on demand

skyline.webapp.snab_backend module

get_snab_engine()[source]
snab_engine_disposal(engine)[source]
update_snab_result(snab_id, anomaly_id, snab_result)[source]

Update the relevant field in the snab table.

Parameters:
  • snab_id (int) – the snab table id
  • anomaly_id (int) – the anomaly id
  • snab_result – a selected result
Returns:

snab_result_updated, base_name, anomaly_timestamp

Return type:

tuple

skyline.webapp.utilities module

alerts_matcher(base_name, pattern, alerter, second_order_resolution_hours)[source]

Get a list of all the metrics that would match an ALERTS pattern

Parameters:
  • base_name (str) – The metric name
  • pattern (str) – the ALERT pattern
  • alerter (str) – the alerter name e.g. smtp, syslog, hipchat, pagerdaty
  • second_order_resolution_hours (int) – (optional) The number of hours that Mirage should surface the metric timeseries for
Returns:

matched_by

Return type:

str

(‘metric3’, ‘hipchat’, 600), # Log all anomalies to syslog (‘stats.’, ‘syslog’, 1), # Wildcard namespaces can be used as well (‘metric4.thing.*.requests’, ‘stmp’, 900), # However beware of wildcards as the above wildcard should really be (‘metric4.thing..*..requests’, ‘stmp’, 900),

skyline.webapp.webapp module

webapp.py

limit_remote_addr()[source]

This function is called to check if the requesting IP address is in the settings.WEBAPP_ALLOWED_IPS array, if not 403.

allowed_file(filename)[source]
check_auth(username, password)[source]

This function is called to check if a username / password combination is valid.

authenticate()[source]

Sends a 401 response that enables basic auth

requires_auth(f)[source]
gzipped(f)[source]
url_encode_metric_name(metric_name)[source]

URL Encode a metric name

internal_error(message, traceback_format_exc)[source]

Show traceback in the browser when running a flask app on a production server. By default, flask does not show any useful information when running on a production server. By adding this view, we output the Python traceback to the error 500 page and log.

As per: Show flask traceback when running on production server https://gist.github.com/probonopd/8616a8ff05c8a75e4601 - Python traceback rendered nicely by Jinja2

This can be tested by hitting SKYLINE_URL/a_500

index()[source]
a_500()[source]
now()[source]
then()[source]
anomalies()[source]
panorama_anomalies()[source]
panorama_not_anomalous()[source]
app_settings()[source]
version()[source]
api()[source]
mock_api()[source]
luminosity_remote_data_endpoint()[source]
docs()[source]
panorama()[source]
crucible()[source]
ionosphere()[source]
ionosphere_images()[source]
ionosphere_files()[source]
ionosphere_file()[source]
utilities()[source]
flux_frontend()[source]
upload_data()[source]
otel_trace()[source]
logstash_test()[source]
match_metric()[source]
thunder_test()[source]
update_external_settings()[source]
remove_namespace_quota()[source]
luminosity()[source]
snab()[source]
webapp_up()[source]
get_redis(host, port, db, password, decode)[source]
get_client_details()[source]

Gets the first X-Forwarded-For address and sets as the IP address. Gets the client_id by simply using a md5 hash of the client IP address and user agent. Determines whether the request was proxied. Determines the client protocol.

Returns:client_id, protocol, proxied, salt
Return type:str, str, boolean, str
decode_token(client_id)[source]

Use the app.secret, client_id and salt to decode the token JWT encoded payload and determine the Redis password.

Parameters:client_id (str) – the client_id string
Returns:token, decoded_redis_password, fail_msg, trace
Return type:str, str, str, str
rebrow()[source]

Start page

rebrow_server_db(host, port, db)[source]

List all databases and show info on server

rebrow_keys(host, port, db)[source]

List keys for one database

rebrow_key(host, port, db, key)[source]

Show a specific key. key is expected to be URL-safe base64 encoded

urlsafe_base64_encode(s)[source]
class App[source]

Bases: object

run()[source]
run()[source]

Start the Webapp server

Module contents