skyline.ionosphere package

Submodules

skyline.ionosphere.agent module

skyline.ionosphere.common_functions module

get_metrics_db_object(base_name)[source]

Returns the data of a metric from metrics table as an object and populates memcached with a dict of the object

Parameters:

base_name (str) – the metric base_name

Returns:

metrics_db_object

Return type:

object

get_calculated_features(calculated_feature_file)[source]

Called by spin_process to create a list of feature_names and their calc_values.

Parameters:

calculated_feature_file (str) – the full path and file name of the features.transposed.csv file

Returns:

calculated_features

Return type:

list

numba_minmax(x, y)[source]

This a numba minmax function, the normal minmax function for 1000 loops on a timeseries of length 1448 took 6.046057 seconds, the numba_minmax function took 3.808568 seconds. On a single run normal minmax took 0.016309 seconds and numba_minmax took 0.006180 seconds.

minmax_scale_check(fp_id_metric_ts, anomalous_timeseries, range_tolerance, range_tolerance_percentage, fp_id, base_name, metric_timestamp, features_percentage_diff)[source]

Called by nothing yet. Used to run a minmax scaling check and determine if the features_sum of the 2 minmax scaled timeseries match.

Parameters:

  • fp_id_metric_ts

  • anomalous_timeseries

  • range_tolerance

:param range_tolerance_percentage :param fp_id: :param base_name: :param metric_timestamp: :param features_percentage_diff: :type fp_id_metric_ts: int :return: (minmax_not_anomalous, minmax_fp_features_sum, minmax_fp_features_count, minmax_anomalous_features_sum, minmax_anomalous_features_count) :rtype: tuple

skyline.ionosphere.echo module

echo_get_metric_from_metrics(base_name, engine)[source]

Called by echo() and returns the metric id and metric db object

Parameters:

timestamp (int) – timestamp at which learn was called

Returns:

tuple

Return type:

(int, object)

ionosphere_echo(base_name, mirage_full_duration)[source]

skyline.ionosphere.find_repetitive_patterns module

find_repetitive_patterns.py

prune_find_repetitive_patterns_dirs()[source]
update_last_processed_anomaly_timestamp(anomaly_timestamp)[source]
find_repetitive_patterns(metric, anomaly_timestamp)[source]
ionosphere_find_repetitive_patterns(timestamp)[source]

Called by spawn_learn_process to re-evaluate anomalies for a metric and determine if the current anomaly or previous anomalies in the last month are similar to other anomalies in last month, if there are more than 2, LEARNT features profiles are created for them. To ensure that only the minimium number of features profiles are learnt a number of conditions are set: 1. any anomalies that occur in the period of any existing features profile are excluded from consideration. 2. any similar training data sets that are are found that occur less than 12 hours apart are deduplicated and the latest of those is used only 3. , the and any that are up to 12 hours older are discarded. Without these conditionals, there tends to be excessive features profile found to learn.

Although it is possible to iterate over all the anomalies and check them all at the same time, here the function is doing one metric per run. repetitive learning

Parameters:

timestamp (int) – timestamp at which learn was called

Returns:

True or False

Return type:

boolean

skyline.ionosphere.inference module

inference.py

ionosphere_motif_inference(metric, timestamp)[source]

skyline.ionosphere.ionosphere module

ionosphere.py

class Ionosphere(parent_pid)[source]

Bases: Thread

The Ionosphere class which controls the ionosphere thread and spawned processes.

check_if_parent_is_alive()[source]

Self explanatory

mysql_insert(insert)[source]

Insert data into mysql table

Parameters:

insert (str) – the insert string

Returns:

int

Return type:

int or boolean

  • Example usage:

    query = 'insert into host (host) VALUES ('this_host')'
result = self.mysql_insert(query)

Note

  • If the MySQL query fails a boolean will be returned not a tuple

    • False

    • None

purge_old_data_dirs(dir_path, older_than)[source]
remove_metric_check_file(metric_check_file)[source]
manage_ionosphere_unique_metrics()[source]

  • Create a Redis set of all Ionosphere enabled metrics.

  • Manage the ionosphere.untrainable_metrics set, removing items when they ‘expire’

Parameters:

i – python process id

Returns:

returns True

new_load_metric_vars(metric_vars_file)[source]

Load the metric variables for a check from a metric check variables file

Parameters:

metric_vars_file (str) – the path and filename to the metric variables files

Returns:

the metric_vars list or False

Return type:

list

spawn_learn_process(i, timestamp, learn_job_context)[source]

Spawn a process to learn.

This is used for Ionosphere to learn if anomalous metrics remain anomalous over time, as the resolution decreases. It follows the multiprocessing methodology the was introduced in Analyzer and Mirage in the context of the process objects being cleared down and the learn processes cannot create memory leaks as the process always terminates or is terminated this prevents any memory leaks in the parent.

process_ionosphere_echo(i, metric_check_file)[source]

Spawn a process_ionosphere_echo check to create features profiles at settings.FULL_DURATION for Mirage metrics

Parameters:

  • i (object) – python process id

  • metric_check_file (str) – full path to the metric check file

Returns:

boolean

Return type:

boolean

spin_process(i, metric_check_file, ionosphere_busy)[source]

Assign an anomalous metric to check against features profiles.

Parameters:

  • i (object) – python process id

  • metric_check_file (str) – full path to the metric check file

  • ionosphere_busy (boolen) – whether to Ionosphere manage and alternate between normal Ionosphere and echo analysis

Returns:

int

Return type:

int or boolean

run()[source]

Called when the process intializes.

skyline.ionosphere.layers module

run_layer_algorithms(base_name, layers_id, timeseries, layers_count, layers_checked)[source]

Called by spin_process to evaluate anomalies against a custom layers boundary algorithm.

Parameters:

  • metric (str) – the metric base_name

  • layers_id – the layer id

  • timeseries (list) – the time series list

  • layers_count (int) – the number of layers for the metric

  • layers_checked (int) – the number of layers that have been checked

Returns:

True or False

Return type:

boolean

skyline.ionosphere.learn module

learn.py

learn_load_metric_vars(metric_vars_file)[source]

Load the metric variables for a check from a metric check variables file

Parameters:

metric_vars_file (str) – the path and filename to the metric variables files

Returns:

the metric_vars list or False

Return type:

list

get_learn_json(learn_json_file, base_name, use_full_duration, metric_timestamp, learn_full_duration_days)[source]

Called by learn() to surface a use_full_duration timeseries for the metric from Graphite and save as json.

get_metric_from_metrics(base_name, engine)[source]

Called by learn() and returns the metric id and metric db object

Parameters:

timestamp (int) – timestamp at which learn was called

Returns:

tuple

Return type:

(int, object)

get_ionosphere_fp_ids(base_name, metrics_id, engine)[source]

Called by learn() and returns the fp_ids list

get_ionosphere_record(fp_id, engine)[source]

Called by learn() and returns the fp row object

remove_work_list_from_redis_set(learn_metric_list)[source]

Called by learn() to remove a work item from the Redis set

ionosphere_learn(timestamp)[source]

Called by spawn_learn_process to re-evaluate anomalies and such, like creating learning features profiles, when a human makes a features profile and the automated creation of learnt features profiles and learn features profiles.

Parameters:

timestamp (int) – timestamp at which learn was called

Returns:

True or False

Return type:

boolean

learn uses a Redis set as a work queue. This set is populated with the learn jobs, jobs being pieces of work that learn needs to do. The learn Redis work set list item has the following elements:

[str(deadline_type), str(job_type), int(metric_timestamp), str(base_name), int(parent_id), int(generation)]

Each job_type has a deadline_time that learn calculates from the job_type

ionosphere.learn.work deadlines - deadline_types

  • Hard - missing a deadline is a total system failure.

  • Firm - infrequent deadline misses are tolerable, but may degrade the system’s quality of service. The usefulness of a result is zero after its deadline.

  • Soft - the usefulness of a result degrades after its deadline, thereby degrading the system’s quality of service.

references:

learn work types:

  • learn_fp_human - Create a features profile for the human created features profile after LEARN_VALID_TIMESERIES_OLDER_THAN_SECONDS at LEARN_FULL_DURATION_DAYS whatever those maybe it, will find out. Copy the training_data dir to settings.IONOSPHERE_LEARN_FOLDER and the metric_check_file is rewritten with the determined (LEARN_FULL_DURATION_DAYS * 86400) as the new full_duration. Then this features profile is created as a generation 0 features profile at the learn use_full_duration seconds. This jobs is only ever add via the Ionosphere UI ~skyline.ionosphere_functions.create_features_profile TODO: now_timestamp/ where the now_timestamp relaces the metric_timestamp context and as the metric_check_file has been replaced and this features profile was created later. This prevents the pollution of metric_timestamp training data features profile. This is a new features profile. Is it generation 0 or generation 1? The more I think I am edging to generation 1, however as long as the first automated full_duration profiles that they create are generation 1 as well… hmm the generation game… TDB - deadline: ‘Soft’

  • learn_fp_automatic - Create a features profile for the automatic created features profile as per the learn_fp_human above. - deadline: ‘Soft’

  • learn_fp_generation - Create a features profile for the Ionosphere training_data metric after LEARN_VALID_TIMESERIES_OLDER_THAN_SECONDS at LEARN_FULL_DURATION_DAYS have passed and compare to other known LEARN_FULL_DURATION_DAYS features profiles. If it matches any, then the metric training_data is added as features profile incremented by 1 generation if MAX_GENERATIONS is not breached. - deadline: ‘Soft’

  • learn_fp_learnt - Create a features profile for the Ionosphere learnt features profile after LEARN_VALID_TIMESERIES_OLDER_THAN_SECONDS at LEARN_FULL_DURATION_DAYS whatever those maybe it, will find out. Copies the training_data dir to settings.IONOSPHERE_LEARN_FOLDER/now_timestamp/ where the now_timestamp relaces the metric_timestamp context and the metric_check_file has the full_duration replace by the relevant use_full_duration. Then this features profile is created as an incremented generation features profile at use_full_duration via learn. - deadline: ‘Soft’

skyline.ionosphere.learn_repetitive_patterns module

learn_repetitive_patterns.py

get_training_to_evaluate(metrics_to_evaluate)[source]

Return a dictionary of training data to evaluate.

Parameters:

metrics_to_evaluate (dict) – the dictionary of metrics_to_evaluate

Returns:

training_to_evaluate

Return type:

dict

get_metrics_to_evaluate(training_data_dict, last_evaluation_timestamp)[source]

Return a dictionary of metrics_to_evaluate.

Parameters:

  • training_data_dict (dict) – the dictionary of training data

  • last_evaluation_timestamp (int) – the last timestamp that training was evaluated

Returns:

metrics_to_evaluate

Return type:

dict

get_features_to_compare(training_to_evaluate)[source]

Return a dictionary offeatures_to_compare.

Parameters:

training_to_evaluate (dict) – the dictionary of training data to evaluate

Returns:

features_to_compare

Return type:

dict

get_metrics_to_train(features_to_compare, timestamp)[source]

Return a dictionary of metrics_to_train.

Parameters:

features_to_compare (dict) – the dictionary of features_to_compare

Returns:

metrics_to_train

Return type:

dict

ionosphere_learn_repetitive_patterns(timestamp)[source]

Called by ionosphere_learn_repetitive_patterns to re-evaluate anomalies and such, like creating learning features profiles, when a human makes a features profile and the automated creation of learnt features profiles and learn features profiles.

Parameters:

timestamp (int) – timestamp at which learn was called

Returns:

True or False

Return type:

boolean

skyline.ionosphere.motif_annihilation module

motif_annihilation.py

class MotifAnnihilation(parent_pid)[source]

Bases: Thread

The MotifAnnihilation class which controls the snab thread and spawned threads.

check_if_parent_is_alive()[source]

Self explanatory

spin_motif_annihilation(work_dict)[source]

Assign motif annihilation work item to process.

Parameters:

  • self (object) – the self object

  • work_dict (dict) – a work dict from the ionosphere.find_repetitive_patterns.motif_annihilation.work Redis hash

Returns:

None

Return type:

None

run()[source]

  • Called when the process intializes.

  • Determine if Redis is up and discover checks to run.

  • Assign check to run.

  • Wait for the processes to finish.

Module contents