skyline.horizon package

Submodules

skyline.horizon.agent module

class Horizon

Bases: object

Initializes Horizon

run()

Determine the MAX_QUEUE_SIZE for the listen process.

Determine if horizon should populate the mini redis store for Oculus.

Starts the defined number of WORKER_PROCESSES, with the first worker populating the canary metric.

Start the pickle (and UDP) listen processes.

Start roomba.

run()

Start the Horizon agent and logger.

skyline.horizon.listen module

listen.py

class SafeUnpickler(file, *, fix_imports=True, encoding='ASCII', errors='strict', buffers=())

Bases: Unpickler

PICKLE_SAFE = {'__builtin__': {'object'}, 'copy_reg': {'_reconstructor'}}

find_class(module, name)

Return an object from a specified module.

If necessary, the module will be imported. Subclasses may override this method (e.g. to restrict unpickling of arbitrary classes and functions).

This method is called whenever a class or a function object is needed. Both arguments passed are str objects.

classmethod loads(pickle_string)

class Listen(port, queue, parent_pid, d_type='pickle')

Bases: Process

The listener is responsible for listening on a port.

gen_unpickle(infile)

Generate a pickle from a stream

read_all(sock, n)

Read n bytes from a stream

check_if_parent_is_alive()

Self explanatory

listen_pickle()

Listen for pickles over tcp

listen_udp()

Listen over udp for MessagePack strings

run()

Called when process intializes.

skyline.horizon.prometheus module

prometheus.py

create_global_reload_skip_dict()

skip_dict in shared memory

get_global_reload_skip_dict()

Reload shared memory skip_dict

update_global_reload_skip_dict(value)

Update shared memory skip_dict

create_global_skip_dict()

get_global_skip_dict()

update_global_skip_dict(skip_dict)

create_global_prom_to_vm_metrics_dict()

get_global_prom_to_vm_metrics_dict()

update_global_prom_to_vm_metrics_dict(prom_to_vm_metrics_dict)

create_global_shard_metrics_dict()

get_global_shard_metrics_dict()

update_global_shard_metrics_dict(prom_to_vm_metrics_dict)

create_global_metrics_type_dict()

get_global_metrics_type_dict()

update_global_metrics_type_dict(metrics_type_dict)

create_global_metrics_with_id_dict()

get_global_metrics_with_id_dict()

update_global_metrics_with_id_dict(metrics_with_id_dict)

class PrometheusMetrics(parent_pid)

Bases: Process

The PrometheusMetrics process is responsible for getting metrics submitted to flux from the flux.prometheus_metrics Redis hash set and adding them to the Horizon worker queue to be submitted to Redis.

check_if_parent_is_alive()

Self explanatory

metric_in_shard(metric_name)

Check if the metric belongs to the Horizon instance shard.

submit_PrometheusMetrics()

Get Prometheus metrics from the Redis flux.prometheus_metrics set every 60 seconds, aggregated them to 60 seconds and submit them to Redis.

run()

Called when process intializes.

skyline.horizon.roomba module

class Roomba(parent_pid, skip_mini)

Bases: Thread

The Roomba is responsible for deleting keys older than DURATION.

check_if_parent_is_alive()

Self explanatory.

vacuum(i, namespace, duration)

Trim metrics that are older than settings.FULL_DURATION and purge old metrics.

run()

Called when process initializes.

skyline.horizon.worker module

worker.py

class Worker(queue, parent_pid, skip_mini, worker_number, canary=False)

Bases: Process

The worker processes chunks from the queue and appends the latest datapoints to their respective timesteps in Redis.

check_if_parent_is_alive()

Self explanatory.

in_skip_list(metric_name)

Check if the metric is in SKIP_LIST.

# @added 20170319 - Feature #1978: worker - DO_NOT_SKIP_LIST The SKIP_LIST allows for a string match or a match on dotted elements within the metric namespace.

in_shard(metric_name)

Check if the metric belongs to the Horizon instance shard.

run()

Called when the process intializes.

Module contents