skyline.webapp package

Submodules

skyline.webapp.add_bq_backfill_job module

add_bq_backfill_job.py

add_bq_backfill_job(current_skyline_app)[source]

Return a dict with the backfill job details.

Parameters:: current_skyline_app (str) – the app calling the function
Returns:: bq_backfill_job_dict
Return type:: dict

skyline.webapp.add_bq_update_job module

add_bq_update_job.py

add_bq_update_job(current_skyline_app)[source]

Return a dict with the backfill job details.

Parameters:: current_skyline_app (str) – the app calling the function
Returns:: bq_update_job_dict
Return type:: dict

skyline.webapp.api_alias_features_profile module

api_alias_features_profile.py

api_alias_features_profile(current_skyline_app, user=None, user_id=0)[source]

Return a dict with the aliased fps for a metric pattern.

Parameters:: current_skyline_app (str) – the app calling the function
Returns:: alias_features_profile_response_dict
Return type:: dict

skyline.webapp.api_bq_query module

api_bq_query.py

api_bq_query(current_skyline_app)[source]

Return a dict with the backfill job details.

Parameters:: current_skyline_app (str) – the app calling the function
Returns:: bq_query_results
Return type:: dict

skyline.webapp.api_create_training_data module

api_create_training_data.py

api_create_training_data(current_skyline_app, cluster_data=False)[source]

Create training data for an anomaly if it does not exist. No echo features profile will be created.

Parameters:

current_skyline_app (str) – the app calling the function
cluster_data (bool) – the cluster_data parameter from the request

Returns:

training_data_url

Return type:

str

skyline.webapp.api_get_analysed_events module

skyline.webapp.api_get_analysed_timeseries module

skyline.webapp.api_get_anomalies_updated module

api_get_anomalies_updated.py

api_get_anomalies_updated(current_skyline_app)[source]

Return a dict with the anomaly DB objects for a metric in the given period.

Parameters:

current_skyline_app (str) – the app calling the function
anomaly_id (int) – the anomaly id

Returns:

anomaly_dict

Return type:

dict

skyline.webapp.api_get_anomaly module

api_get_anomaly.py

api_get_anomaly(current_skyline_app, anomaly_id)[source]

Return a dict with the anomaly DB object and the addtion of ids interpolated to apps, etc where relevant. Although there are multiple methods to get anomaly details via the webapp, none simply return the actually DB anomaly object, they return less or more, more being correlations, matches, etc which are expensive. This is to allow external apps to look up an anomaly quickly and determine if it is a boundary or mirage anomaly, etc.

Parameters:

current_skyline_app (str) – the app calling the function
anomaly_id (int) – the anomaly id

Returns:

anomaly_dict

Return type:

dict

skyline.webapp.api_get_bulk_training_data module

api_get_bulk_training_data.py

api_get_bulk_training_data(current_skyline_app)[source]

Return a dict with the available bulk training data.

Parameters:: current_skyline_app (str) – the app calling the function
Returns:: training_data
Return type:: dict

skyline.webapp.api_get_flux_test_metrics module

skyline.webapp.api_get_fp_timeseries module

api_get_fp_timeseries.py

api_get_fp_timeseries(current_skyline_app)[source]

Return a dict with the metric and fp timeseries.

Parameters:: current_skyline_app (str) – the app calling the function
Returns:: fp_timeseries_dict
Return type:: dict

skyline.webapp.api_get_metric_analysed_events module

skyline.webapp.api_get_metric_anomalies module

api_get_metric_anomalies.py

api_get_metric_anomalies(current_skyline_app)[source]

Return a dict with the anomaly DB objects for a metric in the given period.

Parameters:

current_skyline_app (str) – the app calling the function
anomaly_id (int) – the anomaly id

Returns:

anomaly_dict

Return type:

dict

skyline.webapp.api_get_namespace_metric_count module

api_get_namespace_metric_count.py

api_get_namespace_metric_count(current_skyline_app)[source]

Return a dict with the number of metrics in the namespace with totals for total_metrics, inactive_metrics and active_metrics.

Parameters:: current_skyline_app (str) – the app calling the function
Returns:: namespace_metric_count
Return type:: dict

skyline.webapp.api_get_panorama_all_activity module

api_get_panorama_all_activity.py

get_panorama_all_activity(current_skyline_app)[source]

Return a dict with the number of analysed events, estimated the first day based on dividing the total of the first day by the elapsed number of minutes that have past in the until_timestamp day.

Parameters:

current_skyline_app (str) – the app calling the function
cluster_data (bool) – the cluster_data parameter from the request

Returns:

analysed_events

Return type:

dict

skyline.webapp.api_mute_alerts_on module

skyline.webapp.api_populate_redis_metric module

api_populate_redis_metric.py

api_populate_redis_metric(current_skyline_app)[source]

Return a dict with the timeseries.

Parameters:

current_skyline_app (str) – the app calling the function
cluster_data (bool) – the cluster_data parameter from the request

Returns:

bq_backfill_job_dict

Return type:

dict

skyline.webapp.api_reprocess_batch_metrics module

api_reprocess_batch_metrics.py

api_reprocess_batch_metrics(current_skyline_app)[source]

Return a dict with the timeseries.

Parameters:: current_skyline_app (str) – the app calling the function
Returns:: bq_backfill_job_dict
Return type:: dict

skyline.webapp.api_skip_analysis module

skyline.webapp.api_smoke module

skyline.webapp.backend module

panorama_request()[source]

Gets the details of anomalies from the database, using the URL arguments that are passed in by the request.args to build the MySQL select query string and queries the database, parse the results and creates an array of the anomalies that matched the query and creates the panaroma.json file, then returns the array. The Webapp needs both the array and the JSONP file to serve to the browser for the client side panaroma.js.

Parameters:: None – determined from request.args
Returns:: array
Return type:: array

Note

And creates panaroma.js for client side javascript

luminosity_remote_data(anomaly_timestamp, resolution, namespace=None)[source]

Gets all the unique_metrics from Redis and then mgets Redis data for all metrics. The data is then preprocessed for the remote Skyline luminosity instance and only the relevant fragments of the time series are returned. This return is then gzipped by the Flask Webapp response to ensure the minimum about of bandwidth is used.

Parameters:: anomaly_timestamp (int) – the anomaly timestamp
Returns:: list
Return type:: list

panorama_anomaly_details(anomaly_id)[source]: Gets the details for an anomaly from the database.

get_cluster_data(api_endpoint, data_required, only_host='all', endpoint_params={})[source]

Gets data from the /api of REMOTE_SKYLINE_INSTANCES. This allows the user to query a single Skyline webapp node in a cluster and the Skyline instance will respond with the concentated responses of all the REMOTE_SKYLINE_INSTANCES in one a single response.

Parameters:

api_endpoint (str) – the api endpoint to request data from the remote Skyline instances
data_required (str) – the element from the api json response that is required
only_host (str) – The remote Skyline host to query, if not passed all are queried.
endpoint_params (dict) – A dictionary of any additional parameters that may be required

Returns:

list

Return type:

list

numba_sigma3_array(x, y, first_index_with_value, first_value)[source]

no_numba_yhat_dict(x, y, means, lowers, uppers, anomaly_timestamps_indices, include_yhat_real_lower)[source]

numba_yhat_dict(x, y, means, lowers, uppers, anomaly_timestamps_indices, include_yhat_real_lower)[source]

get_yhat_values(metric, from_timestamp, until_timestamp, include_value, include_mean, include_yhat_real_lower, include_anomalous_periods)[source]

get_mirage_not_anomalous_metrics(metric=None, from_timestamp=None, until_timestamp=None, anomalies=False)[source]

Determine mirage not anomalous metrics from mirage.panorama.not_anomalous_metrics and ionosphere.panorama.not_anomalous_metrics

Parameters:

metric (str) – base_name
from_timestamp (int) – the from_timestamp
until_timestamp (int) – the until_timestamp
anomalies (boolean) – whether to report anomalies as well

Returns:

(dict, dict)

Return type:

tuple

plot_not_anomalous_metric(not_anomalous_dict, anomalies_dict, plot_type)[source]

Plot the metric not anomalous or anomalies graph and return the file path

Parameters:

not_anomalous_dict (dict) – the dictionary of not anomalous events for the metric
anomalies_dict (dict) – the dictionary of anomalous events for the metric

Returns:

path and filename

Return type:

str

namespace_stale_metrics(namespace, cluster_data, exclude_sparsely_populated)[source]

Plot the metric not anomalous or anomalies graph and return the file path

Parameters:

not_anomalous_dict (dict) – the dictionary of not anomalous events for the metric
anomalies_dict (dict) – the dictionary of anomalous events for the metric

Returns:

path and filename

Return type:

str

skyline.webapp.crucible_backend module

get_an_engine()[source]

engine_disposal(engine)[source]

submit_crucible_job(from_timestamp, until_timestamp, metrics_list, namespaces_list, source, alert_interval, user_id, user, add_to_panorama, pad_timeseries, training_data_json, run_algorithms, summarise)[source]

Get a list of all the metrics passed and generate Crucible check files for each

Parameters:

from_timestamp (int) – the timestamp at which to start the time series
until_timestamp (int) – the timestamp at which to end the time series
metrics_list (list) – a list of metric names to analyse
namespaces_list (list) – a list of metric namespaces to analyse
source (str) – the source webapp making the request
alert_interval (int) – the alert_interval at which Crucible should trigger anomalies
user_id (int) – the user id of the user making the request
user (str) – the username making the request
add_to_panorama (boolean) – whether Crucible should add Skyline CONSENSUS anomalies to Panorama
pad_timeseries (str) – the amount of data to pad the time series with
training_data_json (str) – the full path to the training_data json file if source is training_data
run_algorithms (list) – list of algorithms to run
summarise (int) – the number of seconds to summarise data at

Returns:

tuple of lists

Return type:

(list, list, list, list)

Returns (crucible_job_id, metrics_submitted_to_process, fail_msg, trace)

get_crucible_jobs()[source]

Get a list of all the metrics passed and generate Crucible check files for each

Parameters:

requested_timestamp (str) – the training data timestamp
context (str) – the request context, training_data or features_profiles

Returns:

tuple of lists

Return type:

(list, list, list, list)

get_crucible_job(crucible_job_id, metric)[source]

Get the crucible data for a Crucible analysis

Parameters:

crucible_job_id (str) – the crucible_job_id
metric (str) – the metric name

Returns:

tuple of lists

Return type:

(list, boolean, boolean, list, list, list, list, str, str, str)

Returns (crucible_job_details, completed_job, has_anomalies, skyline_anomalies, skyline_consensus_anomalies, panorama_done, panorama_done_timestamp, panorama_done_user_id, image_files, image_file_names, graph_image_file, fail_msg, trace)

send_crucible_job_metric_to_panorama(crucible_job_id, base_name, user_id, user, skyline_consensus_anomalies)[source]

Send the Crucible Skyline CONSENSUS anomalies for a crucible_job and metric to Panorama to insert into the anomalies database.

Parameters:

crucible_job_id (str) – the crucible_job_id
base_name (str) – the metric name
user_id (int) – the user_id
user (str) – the username
skyline_consensus_anomalies (list) – the Crucible Skyline CONSENSUS anomalies

Returns:

tuple of lists

Return type:

(int, str, str)

Returns (len(skyline_consensus_anomalies), fail_msg, trace)

skyline.webapp.expose_metrics module

expose_metrics()[source]

Generate Prometheus metrics and expose them.

Returns:: metrics
Return type:: str

skyline.webapp.get_analysed_events module

get_analysed_events.py

get_analysed_events(current_skyline_app, period, from_timestamp, until_timestamp, namespace=None)[source]

Return a dict with the number of analysed events, estimate the first day count based on dividing the total of the first day by the elapsed number of minutes that have past in the until_timestamp day.

Parameters:

current_skyline_app (str) – the app calling the function
period (str) – get analysed events for
from_timestamp (int) – get analysed events from
until_timestamp (int) – get analysed events until
namespace (str) – get analysed events for namespace

Returns:

analysed_events

Return type:

dict

skyline.webapp.get_boundary_metrics module

skyline.webapp.get_bq_backfill_job module

skyline.webapp.get_saved_training_data module

skyline.webapp.get_snab_report module

get_snab_engine()[source]

snab_engine_disposal(engine)[source]

get_snab_report(filter_on)[source]

Get the relevant data from the snab table.

Parameters:

filter_on (dict) – the filter_on dictionary
algorithms (dict) – the algorithms dictionary
algorithm_groups (dict) – the algorithm_groups dictionary

Returns:

report_data

Return type:

dict

skyline.webapp.get_snab_results module

get_snab_engine()[source]

snab_engine_disposal(engine)[source]

get_snab_results(filter_on)[source]

Update the relevant field in the snab table.

Parameters:

filter_on (dict) – the filter_on dictionary
algorithms (dict) – the algorithms dictionary
algorithm_groups (dict) – the algorithm_groups dictionary

Returns:

results_data

Return type:

dict

skyline.webapp.gunicorn module

skyline.webapp.gunicorn_features_profile module

skyline.webapp.ionosphere_backend module

ionosphere_backend.py

ionosphere_get_metrics_dir(requested_timestamp, context)[source]

Get a list of all the metrics in timestamp training data or features profile folder

Parameters:

requested_timestamp (str) – the training data timestamp
context (str) – the request context, training_data or features_profiles

Returns:

tuple of lists

Return type:

(list, list, list, list)

ionosphere_data(requested_timestamp, data_for_metric, context, offset=None, perpage=None)[source]

Get a list of all training data or profiles folders and metrics

Parameters:

requested_timestamp (str) – the training data or profile timestamp
data_for_metric (str) – the metric base_name
context (str) – the request context, training_data or features_profiles
offset (int) – the offset if pagination parameters are passed
perpage (int) – the number of items to return from offset if pagination parameters are passed

Returns:

tuple of lists

Return type:

(list, list, list, list)

get_an_engine()[source]

engine_disposal(engine)[source]

ionosphere_metric_data(requested_timestamp, data_for_metric, context, fp_id, response_format='html')[source]: Get a list of all training data folders and metrics

features_profile_details(fp_id)[source]

Get the Ionosphere details of a fetures profile

Parameters:: fp_id (str) – the features profile id
Returns:: tuple
Return type:: (str, boolean, str, str)

ionosphere_search(default_query, search_query)[source]

Gets the details features profiles from the database, using the URL arguments that are passed in by the request.args to build the MySQL select query string and queries the database, parse the results and creates an array of the features profiles that matched the query.

Parameters:: None – determined from request.args
Returns:: array
Return type:: array

create_ionosphere_layers(base_name, fp_id, requested_timestamp)[source]

Create a layers profile.

Parameters:: None – determined from request.args
Returns:: array
Return type:: array

feature_profile_layers_detail(fp_layers_id)[source]

Get the Ionosphere layers details of a fetures profile

Parameters:: fp_layers_id – the features profile layers_id
Returns:: tuple
Return type:: (str, boolean, str, str, object)

feature_profile_layer_alogrithms(fp_layers_id)[source]

Get the Ionosphere layer algorithm details of a layer

Parameters:: fp_layers_id – the features profile layers_id
Returns:: tuple
Return type:: (str, boolean, str, str)

metric_layers_alogrithms(base_name)[source]

Get the Ionosphere layer algorithm details of a metric

Parameters:: base_name (str) – the metric base_name
Returns:: tuple
Return type:: (str, boolean, str, str)

edit_ionosphere_layers(layers_id)[source]

Edit a layers profile.

Parameters:: layers_id – the layer id to edit
Returns:: array
Return type:: array

validate_fp(update_id, id_column_name, user_id)[source]

Validate a single features profile or validate all enabled, unvalidated features profiles for a metric_id.

Parameters:

update_id (int) – the features profile id or metric_id to validate
id_column_name (str) – the column name to select where on, e.g. id or metric_id
user_id (int) – the user id of the user that is validating

Returns:

tuple

Return type:

(boolean, str, str)

save_training_data_dir(timestamp, base_name, label, hdate)[source]

Save training_data and return details or just return details if exists

Parameters:

timestamp (str) – the Ionosphere training_data metric timestamp
base_name (str) – metric base_name
label (str) – the saved training_data label
hdate (str) – human date for the saved training_data

Returns:

saved_successful, details, fail_msg, trace

Return type:

boolean, list, str, str

features_profile_family_tree(fp_id)[source]

Returns the all features profile ids of the related progeny features profiles, the whole family tree.

Parameters:: fp_id – the features profile id
Returns:: array
Return type:: array

disable_features_profile_family_tree(fp_ids)[source]

Disable a features profile and all related progeny features profiles

Parameters:: fp_ids – a list of the the features profile ids to disable
Returns:: array
Return type:: array

get_fp_matches(metric, metric_like, get_fp_id, get_layer_id, from_timestamp, until_timestamp, limit, sort)[source]

Get all the matches.

Parameters:

metric – all or the metric name
metric_like – False or the metric MySQL like string e.g statsd.%
get_fp_id – None or int
get_layer_id – None or int
from_timestamp – timestamp or None
until_timestamp – timestamp or None
limit – None or number to limit to
sort – DESC or ASC

Returns:

list

Return type:

list

get_matched_id_resources(matched_id, matched_by, metric, requested_timestamp)[source]

Get the Ionosphere matched details of a features profile or layer

Parameters:

matched_id – the matched id
matched_by – either features_profile, layers or motif
metric – metric base_name
requested_timestamp – the timestamp of the features profile

Returns:

tuple

Return type:

(str, boolean, str, str)

get_features_profiles_to_validate(base_name)[source]

Get the details for Ionosphere features profiles that need to be validated for a metric and returns a list of the details for each of the features profile including the ionosphere_image API URIs for all the relevant graph images for the weabpp Ionosphere validate_features_profiles page. For example:

[[  fp_id, metric_id, metric, fp_full_duration, anomaly_timestamp,
    fp_parent_id, parent_full_duration, parent_anomaly_timestamp, fp_date,
    fp_graph_uri, parent_fp_date, parent_fp_graph_uri, parent_parent_fp_id,
    fp_learn_graph_uri, parent_fp_learn_graph_uri, minimum_full_duration,
    maximum_full_duration, generation]]

Parameters:: base_name (str) – metric base_name
Returns:: list of lists
Return type:: [[int, int, str, int, int, int, int, int, str, str, str, str, int, str, str, int, int]]

get_metrics_with_features_profiles_to_validate()[source]

Get the metrics with Ionosphere features profiles that need to be validated and return a list of the details for each metric. [[metric_id, metric, fps_to_validate_count]]

Returns:: list of lists
Return type:: [[int, str, int]]

ionosphere_show_graphs(requested_timestamp, data_for_metric, fp_id)[source]: Get a list of all graphs

webapp_update_slack_thread(base_name, metric_timestamp, value, message_context, snab_dict={})[source]

Update slack threads with enabled events.

Parameters:

base_name (str) – metric base_name
metric_timestamp (str or int) – the anomaly_timestamp
value (int or None) – the features profile id, the validated_count or None
message_context (str) – training_data_viewed or layers_created
snab_dict (dict) – the snab_dict with algorithms_results and consensus_achieved

Returns:

True or False

Return type:

boolean

validate_ionosphere_match(match_id, validate_context, match_validated, user_id)[source]

Update the validated value in the DB for the match.

Parameters:

match_id (str) – the match id
validate_context (str) – the context to validate either ionosphere_matched or ionosphere_layers_matched
match_validated (int) – 1 for valid or 2 for invalid
user_id (int) – the user id of the user validating

Returns:

True or False

Return type:

boolean

label_anomalies(start_timestamp, end_timestamp, metrics, namespaces, label)[source]

Label all anomalies from start_timestamp to end_timestamp for metrics and and metric namespaces with the given label.

Parameters:

start_timestamp (int) – the start timestamp
end_timestamp (int) – the end timestamp
metrics (list) – a list of metric names
namespaces – a list of namespaces

Returns:

boolean

Return type:

(list, list, list, list)

expected_features_profiles_dirs()[source]

Generate a dict of all features_profiles ids and directories that SHOULD exist.

Return type:: dict

get_matched_motifs(metric, metric_like, from_timestamp, until_timestamp, sort_by)[source]

Get all motif matches.

Parameters:

metric – all or the metric name
metric_like – False or the metric MySQL like string e.g statsd.%
from_timestamp – timestamp or None
until_timestamp – timestamp or None
limit – None or number to limit to

Returns:

list

Return type:

list

get_matched_motif_id(fp_id, timestamp, index, size)[source]: Return the matched_motif_id, motif_validated, ionosphere_matched_id for a motif

skyline.webapp.ionosphere_performance module

get_ionosphere_performance(metric, metric_like, from_timestamp, until_timestamp, output_format, anomalies, new_fps, fps_matched_count, layers_matched_count, sum_matches, title, period, height, width, fp_type, timezone_str, min_full_duration)[source]

Analyse the performance of Ionosphere on a metric or metric namespace and create the graph resources or json data as required.

Return type:: dict

skyline.webapp.luminosity_backend module

get_classify_metrics(base_name, timestamp, algorithm, significant)[source]

Get a list of all the files for metrics in classify_metrics

Parameters:

base_name (str) – the name of the metric
timestamp (str) – the timestamp
algorithm (str) – the algorithm
significant – whether to return only significant results

Returns:

list of dicts

Return type:

{}

Returns a dict of algorithm, metrics, timestamps and significance {

“classify_metrics”: {

“level_shift”: {

‘metric1’: {
1604596200: {‘siginifcant’: True}, 1606410600: {‘siginifcant’: True}, 1607015400: {‘siginifcant’: True}, 1602177000: {‘siginifcant’: False}

}, ‘metric2’: {

1604596200: {‘siginifcant’: True}, 1606410600: {‘siginifcant’: True}, 1607015400: {‘siginifcant’: True}, 1602177000: {‘siginifcant’: False}

},

}, “volatility_shift”: {

‘metric1’: {
1604596200: {‘siginifcant’: True}, 1606410600: {‘siginifcant’: True}, 1607015400: {‘siginifcant’: True}, 1602177000: {‘siginifcant’: False}

}, ‘metric2’: {

1604596200: {‘siginifcant’: True}, 1606410600: {‘siginifcant’: True}, 1607015400: {‘siginifcant’: True}, 1602177000: {‘siginifcant’: False}

},

}

}

}

skyline.webapp.luminosity_cloudbursts module

get_filtered_metrics(redis_conn_decoded, namespaces)[source]

Get create a list of filter_by_metrics.

Parameters:

redis_conn_decoded (str) – the redis_conn_decoded object
namespaces – the namespaces to match
from_timestamp – the from_timestamp
until_timestamp – the until_timestamp

Returns:

list of metrics

Return type:

list

get_metric_ids(redis_conn_decoded, filter_by_metrics)[source]

Get create a list of metric ids and dict of metric_names_with_ids.

Parameters:

redis_conn_decoded (str) – the redis_conn_decoded object
namespaces – the namespaces to match
from_timestamp – the from_timestamp
until_timestamp – the until_timestamp

Returns:

(list of metrics, dict of metric_names_with_ids)

Return type:

(list, dict)

get_cloudbursts(metric, namespaces, from_timestamp, until_timestamp)[source]

Get create a dict of all the cloudbursts.

Parameters:

metric (str) – the name of the metric
namespaces (list) – the namespaces to match
from_timestamp (int) – the from_timestamp
until_timestamp (int) – the until_timestamp

Returns:

dict of cloudbursts

Return type:

{}

Returns a dict of cloudbursts {

“cloudbursts”: {

<id>: {
‘metric_id’: <int>, ‘metric’: <str>, ‘timestamp’: <int>, ‘end’: <int>, ‘duration’: <int>, ‘duration’: <int>, ‘from_timestamp’: <int>, ‘resolution’: <int>, ‘full_duration’: <int>, ‘anomaly_id’: <int>, ‘match_id’: <int>, ‘fp_id’: <int>, ‘layer_id’: <int>, ‘added_at’: <int>,

},

}

}

skyline.webapp.luminosity_pearson_closest module

luminosity_pearson_closest.py

determine_pearson_closest(current_skyline_app)[source]

Get pearson correlations

Parameters:: current_skyline_app (str) – The skyline app
Returns:: pearson_correlations
Return type:: dict

skyline.webapp.luminosity_plot_cloudburst module

get_cloudburst_plot(cloudburst_id, base_name, shift, all_in_period=False)[source]

Create a plot of the cloudburst and return the path and filename

Parameters:

cloudburst_id (int) – the cloudburt id
base_name (str) – the name of the metric
shift (int) – the number of indice to shift the plot

Returns:

path and file

Return type:

str

skyline.webapp.motif_annihilations_to_validate module

motif_annihilations_to_validate.py

get_motif_annihilations_to_validate(from_timestamp, until_timestamp, state='unvalidated', namespace=None)[source]

skyline.webapp.on_demand_motif_analysis module

on_demand_motif_analysis(metric, timestamp, similarity, batch_size, top_matches, max_distance, range_padding, max_area_percent_diff)[source]: Process a motif similarity search on demand

skyline.webapp.panorama_plot_anomalies module

Plot anomalies for a metric

panorama_plot_anomalies(base_name, from_timestamp=None, until_timestamp=None, matches=False)[source]

Create a plot of the metric with its anomalies and return the anomalies dict, the path and filename and the matches_dict

Parameters:

base_name (str) – the name of the metric
from_timestamp (int) – the from timestamp
until_timestamp (int) – the until timestamp
matches (bool) – whether to plot matches as well

Returns:

(anomalies_dict, path and file, matches_dict)

Return type:

tuple

skyline.webapp.save_training_data_comment module

save_training_data_comment(timestamp, metric, metric_id, user_id, comment_data, anomaly_id=None, fp_id=None, match_id=None, motif_match_id=None, snab_id=None, ionosphere_match_url=None, motif_match_url=None)[source]

Insert a comment into the comments table.

Parameters:

timestamp (int) – the timestamp of the event
metric (str) – the metric
metric_id (int) – the metric_id
user_id (int) – the user_id
comment_data (str) – the comment_data
anomaly_id (int) – the anomaly_id
fp_id (int) – the fp_id
match_id (int) – the match_id
motif_match_id (int) – the motif_match_id
snab_id (int) – the snab_id
ionosphere_match_url (str) – the ionosphere_match_url
motif_match_url (str) – the motif_match_url

Returns:

new_comment_id

Return type:

int

skyline.webapp.snab_backend module

get_snab_engine()[source]

snab_engine_disposal(engine)[source]

update_snab_result(snab_id, anomaly_id, snab_result, save_training_data=False)[source]

Update the relevant field in the snab table.

Parameters:

snab_id (int) – the snab table id
anomaly_id (int) – the anomaly id
snab_result – a selected result

Returns:

snab_result_updated, base_name, anomaly_timestamp

Return type:

tuple

get_snab_algorithms(algorithms)[source]

skyline.webapp.timeseries_graph module

Plot anomalies for a metric

timeseries_graph()[source]

Create a plot of the metric and return the path and filename

Parameters:

metric (str) – the name of the metric
from_timestamp (int) – the from timestamp
until_timestamp (int) – the until timestamp
data_source (str) – what data_source to use

Returns:

path and file

Return type:

str

skyline.webapp.utilities module

alerts_matcher(base_name, pattern, alerter, second_order_resolution_hours)[source]

Get a list of all the metrics that would match an ALERTS pattern

Parameters:

base_name (str) – The metric name
pattern (str) – the ALERT pattern
alerter (str) – the alerter name e.g. smtp, syslog, hipchat, pagerdaty
second_order_resolution_hours (int) – (optional) The number of hours that Mirage should surface the metric timeseries for

Returns:

matched_by

Return type:

str

(‘metric3’, ‘hipchat’, 600), # Log all anomalies to syslog (‘stats.’, ‘syslog’, 1), # Wildcard namespaces can be used as well (‘metric4.thing.*.requests’, ‘stmp’, 900), # However beware of wildcards as the above wildcard should really be (‘metric4.thing..*..requests’, ‘stmp’, 900),

skyline.webapp.vortex module

get_run_algorithms()[source]

save_local_csv()[source]

get_data_source()[source]

get_consensus()[source]

get_timeseries_from_training_data()[source]

get_timeseries_from_data_source(data_source, metric, from_timestamp, until_timestamp)[source]

get_vortex_results(vortex_post_data)[source]

vortex_request()[source]: Process a vortex request.

skyline.webapp.webapp module

webapp.py

override_skyline_version()[source]: To allow for development testing allow the skyline_version be passed as an request argument to allow it to be overridden on all requests passed. Also allow to revert to normal with default_version=true argument

limit_remote_addr()[source]: This function is called to check if the requesting IP address is in the settings.WEBAPP_ALLOWED_IPS array, if not 403.

allowed_file(filename)[source]

check_auth(username, password)[source]: This function is called to check if a username / password combination is valid.

authenticate()[source]: Sends a 401 response that enables basic auth

requires_auth(f)[source]

gzipped(f)[source]

url_encode_metric_name(metric_name)[source]: URL Encode a metric name

add_app_version_header(response)[source]

internal_error(message, traceback_format_exc)[source]

Show traceback in the browser when running a flask app on a production server. By default, flask does not show any useful information when running on a production server. By adding this view, we output the Python traceback to the error 500 page and log.

As per: Show flask traceback when running on production server https://gist.github.com/probonopd/8616a8ff05c8a75e4601 - Python traceback rendered nicely by Jinja2

This can be tested by hitting SKYLINE_URL/a_500

index()[source]

a_500()[source]

now()[source]

then()[source]

anomalies()[source]

panorama_anomalies()[source]

panorama_not_anomalous()[source]

app_settings()[source]

version()[source]

api()[source]

mock_api()[source]

luminosity_remote_data_endpoint()[source]

docs()[source]

panorama()[source]

crucible()[source]

ionosphere()[source]

ionosphere_images()[source]

ionosphere_files()[source]

ionosphere_file()[source]

utilities()[source]

flux_frontend()[source]

upload_data()[source]

otel_trace()[source]

logstash_test()[source]

match_metric()[source]

thunder_test()[source]

update_external_settings()[source]

remove_namespace_quota()[source]

mute_alerts_on()[source]

bq_backfill()[source]

bq_update()[source]

reprocess_batch_metrics()[source]

bq_query()[source]

luminosity()[source]

snab()[source]

wind_request()[source]

webapp_up()[source]

get_prometheus_scrape_metrics()[source]: Expose Prometheus-like metrics

csv_to_timeseries_data()[source]

vortex()[source]

accept_test_request()[source]

tsdbs()[source]

get_redis(host, port, db, password, decode)[source]

get_client_details()[source]

Gets the first X-Forwarded-For address and sets as the IP address. Gets the client_id by simply using a md5 hash of the client IP address and user agent. Determines whether the request was proxied. Determines the client protocol.

Returns:: client_id, protocol, proxied, salt
Return type:: str, str, boolean, str

decode_token(client_id)[source]

Use the app.secret, client_id and salt to decode the token JWT encoded payload and determine the Redis password.

Parameters:: client_id (str) – the client_id string
Returns:: token, decoded_redis_password, fail_msg, trace
Return type:: str, str, str, str

rebrow()[source]: Start page

rebrow_server_db(host, port, db)[source]: List all databases and show info on server

rebrow_keys(host, port, db)[source]: List keys for one database

rebrow_key(host, port, db, key)[source]: Show a specific key. key is expected to be URL-safe base64 encoded

urlsafe_base64_encode(s)[source]

class App[source]

Bases: object

run()[source]

run()[source]: Start the Webapp server

skyline.webapp.webapp_features_profile module

webapp_features_profile.py

ionosphere_features_profile()[source]: Calculate features profile and return the details in a json response.

class App[source]

Bases: object

run()[source]

run()[source]: Start the webapp_features_profile server

skyline.webapp.wind module

wind.py

wind(results)[source]

Create a job or accept the results of a job and submit them to the appropriate work Redis hash.

wind expects a json payload with the required_keys and will either post to results of the job to the originating source node or will accept the results from a job and post them to the appropriate work Redis hash. If results are present, results_url must be set to None, if this is a job being submitted results_url must be set and results must be an empty dict or None/null.

post_data = {

‘source_host’: skyline_node, ‘app’: ‘ionosphere’, ‘job’: ‘motif_annihilation’, ‘redis_work_hash’: ‘ionosphere.find_repetitive_patterns.motif_annihilation.work’, ‘redis_work_hash_key’: ‘1712230013.1507778’, ‘data’: {‘metric_id’: metric_id, ‘metric’: metric,

‘anomaly_id’: anomaly_id, ‘anomaly_timestamp’: anomaly_timestamp

}, ‘results_url’: ‘url_to_post_results_to or None/null’, ‘results’: results_dict_or_None/null,

}

Parameters:: results (dict) – a dict with status_code, etc to populate and return
Returns:: results
Return type:: dict

skyline.webapp package

Submodules

skyline.webapp.add_bq_backfill_job module

skyline.webapp.add_bq_update_job module

skyline.webapp.api_alias_features_profile module

skyline.webapp.api_bq_query module

skyline.webapp.api_create_training_data module

skyline.webapp.api_get_analysed_events module

skyline.webapp.api_get_analysed_timeseries module

skyline.webapp.api_get_anomalies_updated module

skyline.webapp.api_get_anomaly module

skyline.webapp.api_get_bulk_training_data module

skyline.webapp.api_get_flux_test_metrics module

skyline.webapp.api_get_fp_timeseries module

skyline.webapp.api_get_metric_analysed_events module

skyline.webapp.api_get_metric_anomalies module

skyline.webapp.api_get_namespace_metric_count module

skyline.webapp.api_get_panorama_all_activity module

skyline.webapp.api_mute_alerts_on module

skyline.webapp.api_populate_redis_metric module

skyline.webapp.api_reprocess_batch_metrics module

skyline.webapp.api_skip_analysis module

skyline.webapp.api_smoke module

skyline.webapp.backend module

skyline.webapp.crucible_backend module

skyline.webapp.expose_metrics module

skyline.webapp.get_analysed_events module

skyline.webapp.get_boundary_metrics module

skyline.webapp.get_bq_backfill_job module

skyline.webapp.get_saved_training_data module

skyline.webapp.get_snab_report module

skyline.webapp.get_snab_results module

skyline.webapp.gunicorn module

skyline.webapp.gunicorn_features_profile module

skyline.webapp.ionosphere_backend module

skyline.webapp.ionosphere_performance module

skyline.webapp.luminosity_backend module

skyline.webapp.luminosity_cloudbursts module

skyline.webapp.luminosity_pearson_closest module

skyline.webapp.luminosity_plot_cloudburst module

skyline.webapp.motif_annihilations_to_validate module

skyline.webapp.on_demand_motif_analysis module

skyline.webapp.panorama_plot_anomalies module

skyline.webapp.save_training_data_comment module

skyline.webapp.snab_backend module

skyline.webapp.timeseries_graph module

skyline.webapp.utilities module

skyline.webapp.vortex module

skyline.webapp.webapp module

skyline.webapp.webapp_features_profile module

skyline.webapp.wind module

Module contents