4.0.0

v4.0.0 - xxx xx, 2023

This is a MAJOR release with changes and enhancements across the board, including changes and additions to backend components.

Notable new features and functionality

There are a plethera of changes and additions in v4.0.0

Added support for Prometheus/VictoriaMetrics - ingesting, storing and analysing labelled metrics or tagged metrics from Prometheus, VictoriaMetrics and via plain HTTP JSON.

This version also starts making use of the very, very long awaited RedisTimeseries for storing labelled metric timeseries data.

Mirage has been significantly improved in its processing methodology to process many metrics per run rather than a single metric per run. This reduces the occurrences of WATERFALL ALERTS to practically zero. Mirage has also been enhanced with additional algorithms to identify normal daily peaks and to identify irregular, unstable timeseries and run an additional analysis on these types of metrics at a much longer time period due to the sparsity of normal events in a weekly period. These two additional analysis methodologies vastly decrease false positives on metrics which display these behaviours.

There is a plethera of additions to custom algorithms and the addition of a new adhoc analysis service called Skyline/Vortex which does adhoc analysis on any 24hr or 7day (overrrideable) uploaded timeseries or timeseries in the system and uses any combination/s of custom algorithms that the user chooses to analyse the timeseries and return results. Vortex can be used directly via a HTTP API and there is also a webapp UI for it as well.

There is also the addition of numerous mainstream custom algorithms to choose from that can now also be used in any part of the analysis pipeline or at least be used with Mirage and Vortex (for which these are intended to be used with, they are not intended to be used as custom_algorithms in Analyzer):

  • sigma

  • matrixprofile

  • spectral_residual

  • lof

  • isolation_forest

  • dbscan

  • pca

  • prophet

  • one_class_svm

  • anomalous_daily_peak

  • irregular_unstable

  • mstl

  • autoarima

Learning repetitive patterns has been added.

An adhoc analysis service has been added called Vortex.

There is a new internal monitoring service named Thunder that handles and routes internal alerts relating to metric namespaces and monitors the internal availability and functioning of the Skyline apps.

There has been a lot of computationally performance improvements by refactoring all/most viable methods in algorithms and data processing into numba functions where appropriate.

The webapp has a number of new pages and features.

There are significant changes to how both webapp and flux are run by gunicorn, both apps have changed to using gevent for the unicorn worker_class.

There are also a number of new dependencies and dependency updates as well.

Noteable Backend changes

Flux and Webapp changed to gevent

There are significant changes to how both webapp and flux are run by gunicorn, both apps have changed to using gevent for the unicorn worker_class. This is a significant operational change. Further there is also one remaining webapp process that needs to be started as the default sync worker_class due to not being able to get flask, gunicorn, gevent and the use of multiprocessing in tsfresh webapp functions to work as desired. Although there are ways to run gevent and multiprocessing together, none of the recommended methods or other attempted methods worked with flask, gunicorn, gevent and tsfresh. The simplest solution therefore is to just have an additional gunicorn process start using sync to serve the few tsfresh related webapp requests, which have always worked and with this method still do. It is a tradeoff incurring a new process as opposed to determining a convoluted method for flask, gunicorn, gevent and tsfresh to work together and maintaining that.

numba optimisations

The starting of apps is a little slower in some cases and the first analysis run on some apps can take longer than normal. This is because gaining performance on every analysis thereafter has the tradeoff in that all the numba algorithms and numba data processing functions must be compiled on their initialisation. Thereafter the performance improvements are gained on every function call.

RedisTimeseries

There are 2 available of methods to deploy and use RedisTimeseries, one being use the normal OSS Redis and compile RedisTimeseries and load the module via the include or use redis-stack-server. Upgrading Redis or changing to redis-stack-server is beyond the scope of this document as it will be vary OS/version (and there are lots of options). There is a guide at https://earthgecko-skyline.readthedocs.io/en/latest/upgrading/redis-to-redis-stack-server.html that explains the upgrade process, but if you run your own Redis build/config/pattern then you may have your own update/upgrade strategy.

That said, if your want to run Skyline with labelled metrics you will need a version of Redis that has the RedisTimeseries module. Consider not just using the redis.io packages repos for yum/apt as they provide redis-stack-server RC (release candidate) packages at times and in terms of Skyline Redis needs to be stable, using the download binaries for your OS is a more stable method.

matrixprofile replaced by stumpy

The matrix-profile-foundation/matrixprofile library is no longer maintained and this has been replaced with the stumpy matrixprofile implementation. This affects the matrixprofile custom_algorithm which will automatically use the stumpy version if available. matrixprofile has been removed as a dependency.

Prometheus/VictoriaMetrics metrics

  • Accept labelled metrics data from Prometheus, InfluxDB (telegraf) and VictoriaMetrics.

  • Use VictoriaMetrics as Skyline’s longer term TSDB for labelled metrics.

Changes

The changes list is LONG, so listed at the end of this page.

Upgrade notes

  • These update instructions apply to upgrading from v3.0.4

  • There are are changes to the DB in v4.0.0

  • There are changes to settings.py in v4.0.0

  • There are changes to dependencies in v4.0.0

  • There is a new service called thunder in v4.0.0

How to update from v3.0.4

  • Build a new Python version and virtualenv, due to this being a major version release we use a new virtualenv which all the required libraries and do not affect your running install so you can rollback if necessarily.

  • Download the new release tag or clone/update to get it to a temp location, ready to be deployed.

  • Copy your settings.py from v3.0.4 and update/replace the v3.0.4 settings.py with your new v4.0.0 settings.py file

  • Stop the Skyline apps

  • Backup the database and Redis

  • OPTIONALLY upgrade MariaDB and/or Redis (redis-stack-server)

  • Update the Python dependencies (requirements.txt)

  • Move to the new version,

  • Start the Skyline apps.

Ensure you read through all the steps below first and assess the steps you wish to do and if any paths, etc need to be changed for your set up. Also you need to make a decision and assess whether you want to upgrade to use redis-stack-server.

Here are the steps….

  • Get the new version.

# Get the new version
NEW_SKYLINE_VERSION="v4.0.0"    # Your new Skyline version
OLD_SKYLINE_VERSION="v3.0.4"    # Your old Skyline version

CURRENT_SKYLINE_PATH="/opt/skyline/github/skyline"                 # Your Skyline path
NEW_SKYLINE_PATH="${CURRENT_SKYLINE_PATH}.${NEW_SKYLINE_VERSION}"  # Your new Skyline path

mkdir -p "${CURRENT_SKYLINE_PATH}.${NEW_SKYLINE_VERSION}"
cd "${CURRENT_SKYLINE_PATH}.${NEW_SKYLINE_VERSION}"
git clone https://github.com/earthgecko/skyline .
git checkout "$NEW_SKYLINE_VERSION"

# settings.py
cp "$NEW_SKYLINE_PATH/skyline/settings.py" "$NEW_SKYLINE_PATH/skyline/settings.py.${NEW_SKYLINE_VERSION}.bak"

# Create a new settings.py file in the new version based on your existing
# settings.py file
cat "${CURRENT_SKYLINE_PATH}/skyline/settings.py" > "$NEW_SKYLINE_PATH/skyline/settings.py"

# You can diff the new settings with your settings that you just copied
diff "$NEW_SKYLINE_PATH/skyline/settings.py.${NEW_SKYLINE_VERSION}.bak" "$NEW_SKYLINE_PATH/skyline/settings.py"

  • You are probably going to want to diff those settings in a visual diff like Meld.

  • Deploy the new Python version and a new virtualenv and install the dependencies.

# DEPLOY A NEW Python 3.8.17 virtualenv
PYTHON_VERSION="3.8.17"
PYTHON_MAJOR_VERSION="3.8"
PYTHON_VIRTUALENV_DIR="/opt/python_virtualenv"
mkdir -p "${PYTHON_VIRTUALENV_DIR}/versions/${PYTHON_VERSION}"
mkdir -p "${PYTHON_VIRTUALENV_DIR}/projects"
cd "${PYTHON_VIRTUALENV_DIR}/versions/${PYTHON_VERSION}"
wget -q "https://www.python.org/ftp/python/${PYTHON_VERSION}/Python-${PYTHON_VERSION}.tgz"
tar -zxvf "Python-${PYTHON_VERSION}.tgz"
cd ${PYTHON_VIRTUALENV_DIR}/versions/${PYTHON_VERSION}/Python-${PYTHON_VERSION}
./configure --prefix=${PYTHON_VIRTUALENV_DIR}/versions/${PYTHON_VERSION}
make -j4
make altinstall

# Create a new skyline-py3817 virtualenv
PROJECT="skyline-py3817"
cd "${PYTHON_VIRTUALENV_DIR}/projects"
virtualenv --python="${PYTHON_VIRTUALENV_DIR}/versions/${PYTHON_VERSION}/bin/python${PYTHON_MAJOR_VERSION}" "$PROJECT"
ln -sf "${PYTHON_VIRTUALENV_DIR}/projects/skyline" "${PYTHON_VIRTUALENV_DIR}/projects/${PROJECT}"

# Ensure the requirements are installed in your new Python 3.8.17 virtualenv
cd "${PYTHON_VIRTUALENV_DIR}/projects/${PROJECT}"
source bin/activate

# Deploy the new requirements
"bin/pip${PYTHON_MAJOR_VERSION}" install -r "${CURRENT_SKYLINE_PATH}.${NEW_SKYLINE_VERSION}"/requirements.txt

deactivate
cd

# Fix python-daemon
cp "/opt/python_virtualenv/projects/${PROJECT}/lib/python${PYTHON_MAJOR_VERSION}/site-packages/daemon/runner.py" "/opt/python_virtualenv/projects/${PROJECT}/lib/python${PYTHON_MAJOR_VERSION}/site-packages/daemon/runner.py.original.bak"
cat "$NEW_SKYLINE_PATH/utils/python-daemon/runner.2.3.0.py" > "/opt/python_virtualenv/projects/${PROJECT}/lib/python${PYTHON_MAJOR_VERSION}/site-packages/daemon/runner.py"
# minor change related to unbuffered bytes I/O
diff "/opt/python_virtualenv/projects/${PROJECT}/lib/python${PYTHON_MAJOR_VERSION}/site-packages/daemon/runner.py.original.bak" "/opt/python_virtualenv/projects/${PROJECT}/lib/python${PYTHON_MAJOR_VERSION}/site-packages/daemon/runner.py"

  • Ensure you have added all the new (and changed) settings to $NEW_SKYLINE_PATH/skyline/settings.py and test them

# Get the new version
NEW_SKYLINE_VERSION="v4.0.0"    # Your new Skyline version
CURRENT_SKYLINE_PATH="/opt/skyline/github/skyline"                 # Your Skyline path
NEW_SKYLINE_PATH="${CURRENT_SKYLINE_PATH}.${NEW_SKYLINE_VERSION}"  # Your new Skyline path

# You can diff the new settings with your settings that you just copied
diff "$NEW_SKYLINE_PATH/skyline/settings.py.${NEW_SKYLINE_VERSION}.bak" "$NEW_SKYLINE_PATH/skyline/settings.py"

# UPDATE YOUR $NEW_SKYLINE_PATH/skyline/settings.py as appropriate

# **TEST**
# Test your new settings.py BEFORE continuing with the upgrade,
# test_settings.sh runs the validate_settings.py that the Skyline apps run
# when they start
$NEW_SKYLINE_PATH/bin/test_settings.sh

  • At this point we are going to stop Skyline services, update the DB and you can also upgrade MariaDB and/or Redis at this point if you choose to.

  • Stop/disable any/all service controls like monit, etc that are controlling Skyline services.

# Stop/disable any/all service controls like monit, etc that are controlling
# Skyline services.

# NOTE ALL SKYLINE SERVICES ARE LISTED HERE, REMOVE TO ONES YOU DO NOT RUN
# or do not wish to run.
SKYLINE_SERVICES="analyzer
analyzer_batch
mirage
crucible
boundary
ionosphere
luminosity
panorama
webapp
vista
snab
flux
horizon"
for i in $SKYLINE_SERVICES
do
  systemctl stop "$i"
done

  • Remove the numba cache directory. This needs to be done to ensure that new jit cache files are compiled for numba jit functions with the updated Python version and required dependencies like numpy (even though they do not exist yet, this is for next time so it does not get forgetten).

rm -rf /opt/skyline/.cache/numba/

  • Backup the DB and apply the update

# BACKUP THE DB AND APPLY THE NEW SQL
BACKUP_DIR="/tmp"  # Where you want to backup the DB to
MYSQL_USER="skyline"
MYSQL_HOST="127.0.0.1"  # Your MySQL IP
MYSQL_DB="skyline"  # Your MySQL Skyline DB name

# Backup DB
mkdir -p $BACKUP_DIR
mysqldump -u$MYSQL_USER -p $MYSQL_DB > $BACKUP_DIR/pre.$NEW_SKYLINE_VERSION.$MYSQL_DB.sql

# Check you dump exists and has data
ls -al $BACKUP_DIR/pre.$NEW_SKYLINE_VERSION.$MYSQL_DB.sql

# Update DB
mysql -u$MYSQL_USER -p $MYSQL_DB < "${NEW_SKYLINE_PATH}/updates/sql/v4.0.0.sql"
echo $?

  • If you are wanting to upgrade MariaDB now is the time to do that.

  • If you want to upgrade Redis or deploy redis-stack-server, now is the time to that.

  • Move your current Skyline directory to a backup directory and move the new Skyline v4.0.0 with your new settings.py from the temp location to your working Skyline directory, (change your paths as appropriate) e.g.

mv "$CURRENT_SKYLINE_PATH" "${CURRENT_SKYLINE_PATH}.${OLD_SKYLINE_VERSION}"
mv "$NEW_SKYLINE_PATH" "$CURRENT_SKYLINE_PATH"

# Set permission on the dump dir
chown skyline:skyline "$CURRENT_SKYLINE_PATH"/skyline/webapp/static/dump

  • Replace your /etc/skyline/skyline.conf to reflect the new Python version and PYTHON_VIRTUALENV_DIR

mv /etc/skyline/skyline.conf /etc/skyline/skyline.conf.pre.v4.0.0.bak
cp "$CURRENT_SKYLINE_PATH"/etc/skyline.conf /etc/skyline/
diff /etc/skyline/skyline.conf.pre.v4.0.0.bak /etc/skyline/skyline.conf

  • Backup and replace your systemd unit files to to reflect the new Python version and PYTHON_VIRTUALENV_DIR, unless you control systemd units via configuration management or other, then assess how to update your systemd unit files. Note there is a new thunder service.

mkdir /opt/skyline/github/skyline/etc/systemd/system.pre.v4.0.0.bak
for i in $(find /opt/skyline/github/skyline/etc/systemd/system -type f)
do
  unitfile=$(basename "$i")
  cp "/etc/systemd/system/${unitfile}" /opt/skyline/github/skyline/etc/systemd/system.pre.v4.0.0.bak/
  /bin/cp -f $i /etc/systemd/system/
done
systemctl daemon-reload

  • Start the all Skyline services (change as appropriate for your set up) e.g.

# NOTE ALL SKYLINE SERVICES ARE LISTED HERE, REMOVE TO ONES YOU DO NOT RUN
# apart from the new thunder Skyline app

# Start all other Skyline services
SKYLINE_SERVICES="horizon
flux
panorama
webapp
vista
analyzer
analyzer_batch
mirage
crucible
boundary
ionosphere
luminosity
snab
thunder"
for i in $SKYLINE_SERVICES
do
  systemctl start "$i"
  if [ $? -ne 0 ]; then
    echo "failed to start $i"
  else
    echo "started $i"
  fi
  systemctl enable "$i"
done
# Restart any/all service controls like monit, etc that are controlling
# Skyline services.

  • Check the logs. You may see a few errors related to some missing Redis keys/hashes this is expected as they do not exist on starting the new version.

# How are they running
tail -n 20 /var/log/skyline/*.log

# Any errors - each app
find /var/log/skyline -type f -name "*.log" | while read skyline_logfile
do
  echo "#####
# Checking for errors in $skyline_logfile"
  cat "$skyline_logfile" | grep -B2 -A10 -i "error ::\|traceback" | tail -n 60
  echo ""
  echo ""
done

Congratulations, you are now running the best OSS anomaly detection stack in the world (probably).

New slack messaging does not handle public channel (111) thunder (1444) panorama check file fails (1460) Memory leak in Analyzer (1558) Patterning Skyline Ionosphere - updated to tsfresh-0.3.0 (1658) Update deps in Ionosphere (1758) Update to tsfresh-0.3.0 (1778) Shapelet extraction (1792) tsfresh extract_features process stalling (1822) ionosphere - mirage Redis data features (1828) Ionosphere alerts (1830) Ionosphere - Graphite now graphs (1842) Ionosphere - features_profile matched graphite graphs (1852) Ionosphere learn (1886) Ionosphere learn - child like parent with evolutionary maturity (1886) Handle non filesystem friendly metric names in check files (1904) ionosphere_layers (1960) Ionosphere - matches page (1996) analyse_derivatives (2034) analyse_derivatives - change in monotonicity (2050) MySQL - Aborted_clients (2130) Optimise Ionosphere DB usage (2132) Update deps (2138) Test skyline with bandit (2160) luminosity (2270) update deps for luminosity branch (2272) pandas-0.22.0 and numpy (2274) CORRELATE_ALERTS_ONLY (2360) Add redis auth to Skyline and rebrow (2378) Ionosphere validate learnt features profiles page (2430) Ionosphere validate learnt features profiles page –> (2430) luminosity_remote_data (2464) FULL_DURATION feature profiles (2484) illuminance (2580) Update dependencies (2588) mysql-connector-python - use_pure (2590) Build Skyline on nodes at v1.2.8 (2596) statsmodels-0.9.0 - cannot import name _representation (2600) alert_slack (2618) slack (2646) CVE-2018-18074 - Skyline requests (2652) Reduce mirage ionosphere alert loop (2682) python-daemon does not support Python 3 buffer method (2692) Failing to load metric vars (2708) Prometheus to Skyline (2732) Mutliple SQL Injection Security Vulnerabilities (2818) Test redis-py upgrade (2824) Skyline - Python 3.7.2 (2828) Mirage - periodic_check (2882) Initial Ionosphere echo load and Ionosphere feedback (2904) ANALYZER_ENABLED setting (2916) Update dependencies (2926) Update dependencies (2964) New slack messaging does not handle public channel (2986) docker (3002) Reduce multiprocessing Manager list usage (3034) Reduce multiprocessing Manager list usage, this was (3034) Update dependencies (3060) SNAB (3068) Add CUSTOM_ALERT_OPTS to settings (3194) Allow Skyline to send metrics to another Carbon host (3250) webapp api - unique_metrics (3252) Review and test PR 124 (3256) py3 (3262) py3 (3263) py3 Redis binary objects not strings (3266) Deprecate string.replace for py3 (3270) Always send anomaly_breakdown and exception metrics (3288) Handle urllib2 in py3 (3290) Record anomaly_end_timestamp (3306) webapp api - derivative_metrics (3336) http_alerter (3396) Panorama - add metric to metrics table immediately (3442) Allow flux to backfill (3444) Add IONOSPHERE_MANAGE_PURGE (3462) ionosphere.training_data Redis set (3472) webapp api - training_data (3474) batch_processing (3480) analyzer_batch (3486) webapp - crucible_process_metrics (3500) ionosphere.untrainable_metrics (3508) matched_or_regexed_in_list function (3512) Sort all time series (3532) webapp - upload_data endpoint (3538) flux.uploaded_data_worker (3550) Update deps (3556) External alert config (3560) custom_algorithms (3566) Apply list to settings import (3572) inference (3590) Add timestamp to ENABLE_BOUNDARY_DEBUG output (3594) Update Skyline to Python 3.8.3 and deps (3608) Upgrade to slack v2 (3612) Anomaly type classification (3642) ROOMBA_DO_NOT_PROCESS_BATCH_METRICS (3650) Handle multiple metrics in base_name conversion (3652) IONOSPHERE_GRAPHITE_NOW_GRAPHS_OVERRIDE (3654) Change mirage.last_check keys to timestamp value (3662) flux - POST multiple metrics (3694) FLUX_ZERO_FILL_NAMESPACES (3708) Update sqlalchemy - SNYK-PYTHON-SQLALCHEMY-590109 (3720) Validate Mirage running multiple processes (3730) waterfall alerts (3734) POC matrixprofile (3744) POC SNAB (3748) Add the anomaly_id to the http_alerter json (3772) waterfall_alert - no analyzer triggered_algorithms in waterfall_panorama_data on MIRAGE_ALWAYS_METRICS (3776) Handle single encoded forward slash requests to Graphite (3778) skyline_functions - sanitise_graphite_url (3780) snab_flux_load_test (3788) flux - pickle to Graphite (3790) Handle feedback metrics in Mirage and waterfall alerts (3800) ANALYZER_DYNAMICALLY_ANALYZE_LOW_PRIORITY_METRICS (3808) HORIZON_SHARDS (3820) get_cluster_data (3824) Add inactive columns to the metrics DB table (3828) metrics_manager (3830) custom_algorithms - run_before_3sigma parameter (3848) webapp - yhat_values API endoint (3850) Handle boring sparsely populated metrics in derivative_metrics (3856) flux - try except everything (3864) MIRAGE_ENABLE_HIGH_RESOLUTION_ANALYSIS (3866) POC MIRAGE_ENABLE_HIGH_RESOLUTION_ANALYSIS (3868) metrics_manager - check_data_sparsity (3870) SNYK-PYTHON-PY-1049546 (3874) ANALYZER_CHECK_LAST_TIMESTAMP (3884) Test Python 3.8.6 and deps (3898) Test Skyline with dependencies updates - tsfresh v0.17.9 (3926) Update Skyline to use new tsfresh feature extraction method (3928) ionosphere_performance (3934) SNYK-PYTHON-PILLOW-1055461 and SNYK-PYTHON-PILLOW-1055462 (3940) xlrd-2.0.1 removed support for xlsx format (3944) luminosity - motifs (3956) Handle secondary algorithms in yhat_values (3958) SNYK-PYTHON-JINJA2-1012994 (3960) custom_algorithm - adtk_level_shift (3970) SNYK-PYTHON-PILLOW-1080635 (3972) SNYK-PYTHON-AIOHTTP-1079232 (3974) luminosity - classify_metrics (3978) SNYK-PYTHON-URLLIB3-1085966 (3988) Panorama - mirage not anomalous (3994) EXTERNAL_SETTINGS (4000) flux - aggregator.py and FLUX_AGGREGATE_NAMESPACES (4004) SNYK-PYTHON-PILLOW-1090584 (4008) Ionosphere - inference (4014) Move mysql_select calls to SQLAlchemy (4022) Change from scipy array to numpy array (4026) refactoring (4030) Mirage - removing feedback metrics to be processed (4048) skyline.flux.worker.discarded metrics (4060) ANALYZER_SKIP (4068) CUSTOM_STALE_PERIOD (4076) crucible - custom_algorithms (4118) deps updates (4140) webapp - stale_metrics API endpoint (4144) check_data_sparsity - incorrect on low fidelity and inconsistent metrics (4146) analyzer.metrics_manager.resolutions (4148) metrics_manager - roomba batch processing metrics (4150) DO_NOT_SKIP_SKYLINE_FEEDBACK_NAMESPACES (4152) luminosity - cloudbursts (4164) custom_algorithm - m66 (4180) metrics_manager.boundary_metrics (4188) functions.aws.send_sms (4196) luminosity - cross_correlation_relationships (4264) luminosity - related_namespaces (4278) aet.metrics_manager.derivative_metrics Redis hash (4280) Update dependencies (4282) flux - telegraf (4284) prometheus (4300) matrixprofile - fN on big drops (4308) flux - reload external_settings (4324) webapp - panorama_plot_anomalies (4326) BATCH_METRICS_CUSTOM_FULL_DURATIONS (4328) SNYK-PYTHON-AIOHTTP-1584144 (4340) Update dependencies (4344) statsmodel 0.13.1 error (4346) matplotlib 3.5.1 error (4348) snyk - numpy and pillow updates (4362) Prune old thunder.events (4364) Switch git clone to https (4368) webapp - handle url encoded chars (4374) webapp - update_external_settings (4376) flux - return reason with 400 (4380) flux - quota (4400) flux - external_settings - aggregation (4404) settings - LOCAL_EXTERNAL_SETTINGS (4442) webapp - inactive_metrics (4444) Optimise horizon worker in_skip_list (4446) Update dependencies (4452) v0.19.1 (4456) v0.19.1 (earthgecko-tsfresh) (4456) flux - quota - cluster_sync (4464) flux - remove_namespace_quota_metrics (4468) Stationarity (4480) Test alerts (4482) settings - MONOTONIC_METRIC_NAMESPACES (4502) Integrate opentelemetry (4514) flux - opentelemetry traces (4516) settings - LAST_KNOWN_VALUE_NAMESPACES (4518) settings - ZERO_FILL_NAMESPACES (4520) metrics_manager - derivative_metric_check (4528) namespace.analysed_events (4530) opentelemetry - rename InstrumentationLibrary to InstrumentationScope (4532) Handle Redis failure (4536) Plot matched timeseries (4540) authoritative_node (4564) mirage - process multiple metrics (4576) webapp - api_get_fp_timeseries (4578) custom_algorithm - skyline_ARTime (4584) Ionosphere - inference - further validate all_in_range (4588) flux - prometheus_alerts (4596) functions.settings.manage_external_settings - use cache data on bad response (4608) Support labelled metrics (4614) memray (4620) Change all dict copy to deepcopy (4624) ionosphere.bulk.training (4650) http_alerter - dotted_representation (4652) ionosphere.learn_repetitive_patterns (4658) settings.LUMINOSITY_CLOUDBURST_SKIP_METRICS (4662) cloudburst active events only (4674) analyzer - illuminance.all key (4676) analyzer - anomalous metrics sets not flushing (4696) Optimise Ionosphere Redis sets refresh when busy (4698) algorithms - single series (4700) numba optimisations (4702) get_ionosphere_performance - allow multiple metric_like string (4704) Add request_id and timing to ionosphere requests (4706) ionosphere - store and cache fp minmax data (4708) Reset mirage trigger_history on Ionosphere match (4712) opentelemetry check is_instrumented_by_opentelemetry (4714) dotted_representation breaking alert resend_queue (4720) Handle alert and waterfall alert overlap (4722) custom_algorithms - anomalous_daily_peak (4724) vortex (4728) flux vortex (4732) mirage_vortex (4734) custom_algorithms - sigma (4736) Allow first_hour_average to handle different resolution (4738) custom_algorithms - lof (4740) custom_algorithms - skyline_prophet (4742) custom_algorithms - spectral_residual (4744) custom_algorithm - dbscan (4746) custom_algorithm - isolation_forest (4748) custom_algorithm - one_class_svm (4750) custom_algorithm - pca (4750) csv_to_timeseries (4754) Use gevent gunicorn worker_class (4756) Group metrics based on names and labels (4762) analyzer_labelled_metrics - monotonicity checked incorrect classification (4766) webapp - link to current VictoriaMetrics graph (4772) Handle multiple metrics in victoriametrics response (4776) v4.0.0 - update dependencies (4778) Switch from matrixprofile to stumpy (4786) functions.metrics_manager.manage_inactive_metrics (4792) Deprecate run_script from crucible (4798) Deprecate unittest2 (4800) Manage NUMBA_CACHE_DIR (4806) custom_algorithms - numba_cache_dirs (4808) Handle error in sqlalchemy change (4816) Deprecation of numpy asscalar function in luminol (4820) vista - warn on remote_error (4824) EXTERNAL_SETTINGS - handle key value in external_settings (4828) webapp - panorama_plot_anomalies - all_events (4830) webapp - api - get_all_activity (4834) functions.metrics.get_namespace_metric.count (4838) flux - prometheus - drop bucket metrics (4840) flux - prometheus - x-test-only header (4840) webapp - api - get_flux_test_metrics (4842) Replace Redis getset with set with get (4844) mirage - analyse.irregular.unstable.timeseries.at.30days (4848) Allow for level to be passed in SYSLOG_OPTS (4852) boundary - labelled_metrics (4854) flux - prometheus - no 0s (4856) analyzer_labelled_metrics - exceptions metrics set (4864) Optimise luminosity for labelled_metrics (4872) luminosity - related_metrics - labelled_metrics (4874) labelled_metrics - resolution and data sparsity (4882) Update dependencies - redis-py - CVE-2023-28858 (4884) analyzer - operation_timings (4886) analyzer - load_shedding (4888) analyzer_labelled_metrics - use mrange (4890) SNAB - labelled_metrics (4892) labelled_metrics - SKYLINE_FEEDBACK_NAMESPACES (4894) custom_algorithms - mstl (4896) custom_algorithms - autoarima (4898) Prevent training on metrics newer than 7 days (4902) Handle send_graphite_metric failure (4904) mute_alerts_on (4932) Inefficient full_duration_timeseries_fill - last_known_value (4940) vortex - m66 (4946) vortex - adtk algorithms (4948) custom_algorithm - spectral_entropy (4950) vortex - consensus_count (4952) webapp_features_profile - status (4958) custom_algorithms - macd (4960) Build and test skyline v4.0.0 (4962) dawn - redis-stack-server (4964) horizon.prometheus - handle TSDB not being able to parse LABELS (4968) Ionosphere (922) flux (956)