skyline.luminosity package

Submodules

skyline.luminosity.agent module

skyline.luminosity.classify_anomalies module

classify_anomalies.py

classify_anomalies(i, classify_anomalies_set, start_timestamp, classify_for)[source]

skyline.luminosity.classify_metrics module

classify_metrics(i, start_timestamp, classify_for)[source]

skyline.luminosity.cloudburst module

class Cloudburst(parent_pid)[source]

Bases: Thread

The Cloudbursts class which controls the luminosity/cloudburst thread and spawned processes. luminosity/cloudburst analyses metrics to identify significant changepoints using the m66 algorithm.

check_if_parent_is_alive()[source]

Self explanatory

find_cloudbursts(i, unique_metrics)[source]

Create and manage the required lists and Redis sets

run()[source]
  • Called when the process intializes.

  • Determine if Redis is up

  • Spawn a find_cloudbursts process to do analysis

  • Wait for the process to finish.

  • run_every 300 seconds

skyline.luminosity.cloudbursts module

class Cloudbursts(parent_pid)[source]

Bases: Thread

The Cloudbursts class which controls the luminosity/cloudbursts thread and spawned processes. luminosity/cloudbursts finds metrics related to each cloudburst that have been identified by significant changepoints using the m66 algorithm.

check_if_parent_is_alive()[source]

Self explanatory

Find metrics that are related to cloudbursts using the ppscore correlations.

run()[source]
  • Called when the process intializes.

  • Determine if Redis is up

  • Determine new cloudbursts to find related metrics on

  • Wait for the process to finish.

  • run_every 30 seconds

skyline.luminosity.luminosity module

luminosity.py

class PaulBourke[source]

Bases: object

# @added 20180526 - Branch #2270: luminosity The PaulBourke class pays homage to Paul Bourke who originally described the cross correlation method on which Linkedin’s luminol.correlate library is based upon, he described this in August 1996 and it is upon this which Skyline Luminosity is also based.

So Skyline has moved from: Shewhart’s statistical process control circa 1924 tech To tsfresh - circa 20161029 when it entered Skyline Now to 1996(2015), although 1996 ideas can still be very, very useful and work well.

Please do visit these following two URLs in order: http://paulbourke.net/miscellaneous/correlate/thanks_mr_bourke_from_skyline_for # this will 404 but hopefully he has something watching his 404 rate :) http://paulbourke.net/miscellaneous/correlate/ # this will 200

This is all part of the adventures in Skyline. If you enjoy this sort of thing, then I posit to you that Sir Walter Munk is one of the least known remarkable scientist of tthe current and previous millienia. Born: October 19, 1917 (age 100 years) His accomplishments abound, he is like Turing, Marconi, Berners-Lee and Einstein rolled into Oceangraphy, but few have ever heard of him.

If we are giving kudos to Shewhart, tsfresh and Paul Bourke, we can slip some Walther Munk in here too, although currently he nor his work has anything to do with Skyline, however there are some ideas, not necessarily desrcibed in the roadmap, that revolve a byproduct of monitoring surf conditions via a webcams, which could be used for monitoring tide levels too, which would be right up Dr Munk’s alley.

class Luminosity(parent_pid)[source]

Bases: Thread

The Luminosity class which controls the luminosity thread and spawned processes.

check_if_parent_is_alive()[source]

Self explanatory

classify_anomalies(i, classify_anomalies_set, start_timestamp, max_run_seconds)[source]

Classify anomalies

Parameters:
  • i (object) – python process id

  • classify_anomalies_set (set) – set from luminosity.classify_anomalies

  • start_timestamp (int) – the process star timestamp

  • max_run_seconds (int) – the max number of seconds to run for

Returns:

boolean

Return type:

boolean

classify_metrics(i, start_timestamp, max_run_seconds)[source]

Classify metrics

Parameters:
  • i (object) – python process id

  • start_timestamp (int) – the process star timestamp

  • max_run_seconds (int) – the max number of seconds to run for

Returns:

boolean

Return type:

boolean

spin_process(i, anomaly_id)[source]

Assign an anomalous metric and determine correlated metrics

Parameters:
  • i (object) – python process id

  • anomaly_id (int) – the anomaly_id

Returns:

boolean

Return type:

boolean

run()[source]

Called when the process intializes.

skyline.luminosity.process_correlations module

skyline.luminosity.related_metrics module

class RelatedMetrics(parent_pid)[source]

Bases: Thread

The RelatedMetrics class controls the luminosity/related_metrics thread and spawned processes. luminosity/related_metrics analyses the results of luminosity cross_correlations and related_metricss to create and maintain metric groups.

check_if_parent_is_alive()[source]

Self explanatory

  • Determine when a metric group was last updated.

  • Determine if any new anomalies have occurred on the metric.

  • If there are any new anomalies on the metric, determine if new cross_correlations or related_metrics have occured on a metric

  • If there are new cross_correlations or related_metricss calculate the metric group with the new data

  • Determine if the new data modifies the metric group, if so update.

run()[source]
  • Called when the process intializes.

  • Determine if Redis is up

  • Spawn a process_metric process to do analysis

  • Wait for the process to finish.

  • run_every 300 seconds

Module contents

Luminosity

  • Determine correlations

  • Insert in correlations table (slow stream of inserts, not gluts, queue them)
    • anomaly_id, mertic_id, coefficient, shift, shifted_coefficient

    • a table per metric a la z_ts or;

    • a table per month and a daily aggregated table?

  • In Ionosphere:
    • Display Luminosity sorted_correlated_metrics in a block with a link to the anomaly page

  • In Panorama:
    • Display a Luminosity link the anomaly page