1.3.0

v1.3.0 - September 25, 2019

This is a features functionality release with minor bug fixes. This release introduces a few new features and new functionality:

  • A new Ionosphere feature related to validating matches.
  • The ability to run the Skyline stack on docker
  • Introduces Skyline Flux for receiving metrics via HTTP and submitting them to Graphite.
  • Introduces Skyline Vista for fetching metrics from remote sources.

Please see the new documentation pages for full details about each new addition.

Changes in v1.3.0

Lots of changes have been introduced to accommodate docker and the addition of Flux and Vista.

  • Use GRAPHITE_RENDER_URI and do not try to alert if the settings are default in analyzer/alerters.py (3002)
  • Wrapped new get_graphite_render_uri and get_graphite_custom_headers functions in try and except in analyzer/alerters.py (3002)
  • The management of the analyzer Redis algorithm exception sets was previously handled a if settings.ENABLE_MIRAGE block. This meant that if there are no mirage metrics, the sets were not managed. Added manage_analyzer_algorithm_exception_sets in analyzer/analyzer.py to manage these sets in their own context. (3002, 3020)
  • Re-enabled the illuminance metric in analyzer (2580)
  • Declare and test if the derivative_metrics Redis set exists in analyzer so that errors are not thrown if there are actually no derivative metrics (3002)
  • Use GRAPHITE_RENDER_URI and use new CARBON_HOST variable instead of GRAPHITE_HOST in send_graphite_metric (3002)
  • Handle self signed certificate on Docker with requests (3002)
  • Handle when Skyline is first installed, if luminosity is enabled it reports errors as there are no anomaly ids (3002, 3016)
  • Log appropriate to whether memcache is enabled or not (3002, 3016)
  • Handle self signed certificate on Docker with requests (3002)
  • Do not analyse short time series, only return a time series sample if the sample has sufficient data points otherwise get_anomalies() will throw an error (3002, 3008)
  • Added docker settings (3002)
  • Allow rebrow to connect to Redis on the socket too and if docker dsplay the REDIS_PASSWORD in rebrow_login (3002)
  • Use GRAPHITE_RENDER_URI (3002)
  • Revert to original skyline.httpd.conf.d.example and added skyline.docker.apache2.conf.d.example (3002)
  • Reformat print in analyzer/agent.py (3002)
  • When there are multiple channels declared Skyline only wants to update the panorama.slack_thread_ts Redis key once for the primary Skyline channel, so if the cache key exists do not update, analyzer/alerters.py (3110)
  • Added a hashlib hexdigest method which has not been verified so using the original method useless the base_name is longer than 254 chars, analyzer/alerters.py (3002)
  • Allow for DOCKER_FAKE_EMAIL_ALERTS and GRAPHITE_CUSTOM_HEADERS, analyzer/alerters.py (3002)
  • Added get_graphite_port, get_graphite_render_uri and get_graphite_custom_headers functions, analyzer/alerters.py (3002)
  • Reduce amount of Manager instances that are used as each requires a copy of entire memory to be copied into each subprocess so this results in a python process per Manager instance, using as much memory as the parent. OK on a server, not so much in a container. Disabled all the Manager() lists and replaced with Redis sets, analyzer/analyzer.py (3002, 3032, 3034)
  • Moved the interpolation of the MIRAGE_PERIODIC variables to the top of the file out of spawn_process so they can be accessed in run too and check the added mirage_periodic_check_keys list rather than checking each and every metric name key, analyzer/analyzer.py (2882)
  • Disabled for now as in concept phase, analyzer/analyzer.py (2580)
  • The management of the analyzer Redis algorithm exception sets was previously handled in an if settings.ENABLE_MIRAGE block This meant that if there are no mirage metrics, the sets were not managed, analyzer/analyzer.py (3020)
  • Changes for docker, not complete or fully tested.
  • Reduce amount of Manager instances that are used as each requires a copy of entire memory to be copied into each subprocess so this results in a python process per Manager instance, using as much memory as the parent. OK on a server, not so much in a container. Disabled all the Manager() lists and replaced with Redis sets, boundary/boundary.py (3002, 3032, 3034)
  • Use GRAPHITE_RENDER_URI, boundary/boundary.py (3002)
  • Reduce amount of Manager instances that are used as each requires a copy of entire memory to be copied into each subprocess so this results in a python process per Manager instance, using as much memory as the parent. OK on a server, not so much in a container. Disabled all the Manager() lists and replaced with Redis sets, crucible/crucible.py (3002, 3032)
  • Add a crucible - skyline.consensus.anomalies.png to plot Skyline anomalies if CONSENSUS is achieved and create file resources skyline.anomalies_score.txt and skyline.anomalies.csv, crucible/crucible_algorithms.py (3106)
  • Reduce amount of Manager instances that are used as each requires a copy of entire memory to be copied into each subprocess so this results in a python process per Manager instance, using as much memory as the parent. OK on a server, not so much in a container. Disabled all the Manager() lists and replaced with Redis sets, skyline/ionosphere/ionosphere.py (3002, 3032)
  • Do not run checks if namespace has matched multiple times in the last 10 minutes. However determining which Skyline related metrics are feeding back are quite difficult to ascertain. So use the ionosphere_busy logic again and use or find the skyline host namespace and if busy do not analyse the Skyline host namespace while ionosphere is busy related to SKYLINE_FEEDBACK_NAMESPACES, skyline/ionosphere/ionosphere.py (3050)
  • Only process FULL_DURATION feature profiles if there is a ionosphere.unique_metrics Redis set (2484)
  • Handle self signed certificate on Docker, skyline/ionosphere/learn.py (3002)
  • Reduce amount of Manager instances that are used as each requires a copy of entire memory to be copied into each subprocess so this results in a python process per Manager instance, using as much memory as the parent. OK on a server, not so much in a container. Disabled all the Manager() lists and replaced with Redis sets, skyline/luminosity/luminosity.py (3002, 3032)
  • When Skyline is first installed, if luminosity is enabled it reports errors as there are no anomaly ids, skyline/luminosity/luminosity.py (3016)
  • Handle self signed certificate on Docker, skyline/luminosity/process_correlations.py (3002)
  • Reduce amount of Manager instances that are used as each requires a copy of entire memory to be copied into each subprocess so this results in a python process per Manager instance, using as much memory as the parent. OK on a server, not so much in a container. Disabled all the Manager() lists and replaced with Redis sets, skyline/mirage/mirage.py (3002, 3032)
  • When there are multiple channels declared Skyline only wants to update the panorama.slack_thread_ts Redis key once for the primary Skyline channel, so if the cache key exists do not update, skyline/mirage/mirage_alerters.py (3110)
  • Reduce amount of Manager instances that are used as each requires a copy of entire memory to be copied into each subprocess so this results in a python process per Manager instance, using as much memory as the parent. OK on a server, not so much in a container. Disabled all the Manager() lists and replaced with Redis sets, skyline/mirage/mirage.py (3002, 3032)
  • Allow webapp to connect on reborw via socket if running on docker, skyline/webapp/templates/rebrow_login.html (3002)
  • Handle self signed certificate on Docker, skyline/webapp/ionosphere_backend.py (3190)
  • Handle if all features profiles have been disabled in the ionosphere_search function, skyline/webapp/ionosphere_backend.py (3190)
  • Added Ionosphere - validated matches (3082, 3084)
  • Added metrics id to relevant web pages, skyline/webapp/webapp.py (2990)
  • Added docker related settings to settings.py (3002)
  • Added Ionosphere - validated matches DB changes to skyline.sql (3084)
  • Added docker changes and some Pyton 3.7 testing changes to skyline/skyline_functions.py (3002, 2828)
  • Updated version and branch in skyline/skyline_version.py (3002)
  • Updated to 1.3.0 - skyline/utils/dawn/skyline.dawn.sh (3002)
  • Uppdated dependencies
  • skyline.dawn.sh changes related to docker (3002)
  • Update version and dependencies
  • Added a users tables to the DB to allow to map user to actions, labels, etc (3230, 2476, 2516)

Update notes

  • These update instruction apply to upgrading from v1.2.17 to v1.3.0 only. However as with all Skyline updates it is possible to go through the update notes for each version and make your own update notes/process to take you from version x to version y.
  • There are changes to the DB in v1.3.0

Warning

It is assumed that the database is not running with an offset and the database increments ids by 1. If you are running an offset please review all SQL updates to determine if you need to change anything in the SQL when you apply Skyline SQL updates.

  • There are changes to settings.py in v1.3.0, please ensure you diff your current and the new settings.py as there are appended additions but there are also additional settings that have been added to existing settings blocks. The additions and changes described below.

settings.py Changes

How to update from v1.2.17

  • Download the new release tag or clone/update to get it to a temp location, ready to be deployed.
NEW_SKYLINE_VERSION="v1.3.0"    # Your new Skyline version
OLD_SKYLINE_VERSION="v1.2.17"    # Your old Skyline version

CURRENT_SKYLINE_PATH="/opt/skyline/github/skyline"                 # Your Skyline path
NEW_SKYLINE_PATH="${CURRENT_SKYLINE_PATH}.${NEW_SKYLINE_VERSION}"  # Your new Skyline path

mkdir -p "${CURRENT_SKYLINE_PATH}.${NEW_SKYLINE_VERSION}"
cd "${CURRENT_SKYLINE_PATH}.${NEW_SKYLINE_VERSION}"
git clone https://github.com/earthgecko/skyline .
git checkout "$NEW_SKYLINE_VERSION"

cp "$NEW_SKYLINE_PATH/skyline/settings.py" "$NEW_SKYLINE_PATH/skyline/settings.py.${NEW_SKYLINE_VERSION}.bak"

# DIFF your settings.py with the new settings.py
# diff "${CURRENT_SKYLINE_PATH}/skyline/settings.py" "$NEW_SKYLINE_PATH/skyline/settings.py.${NEW_SKYLINE_VERSION}.bak"

#
cat "${CURRENT_SKYLINE_PATH}/skyline/settings.py" > "$NEW_SKYLINE_PATH/skyline/settings.py"

# ADD the appropriate new settings to your settings file.
vi "$NEW_SKYLINE_PATH/skyline/settings.py"

# Stop Skyline DB related services
SKYLINE_SERVICES="webapp
ionosphere
luminosity
panorama"
for i in $SKYLINE_SERVICES
do
  /etc/init.d/$i stop
done
# And stop any service controls like monit or systemd

# BACKUP THE DB AND APPLY THE NEW SQL
BACKUP_DIR="/tmp"  # Where you want to backup the DB to
MYSQL_USER="<YOUR_MYSQL_USER>"
MYSQL_HOST="127.0.0.1"  # Your MySQL IP
MYSQL_DB="skyline"  # Your MySQL Skyline DB name

# Backup DB
mkdir -p $BACKUP_DIR
mysqldump -u$MYSQL_USER -p $MYSQL_DB > $BACKUP_DIR/pre.$NEW_SKYLINE_VERSION.$MYSQL_DB.sql

# Update DB
mysql -u$MYSQL_USER -p $MYSQL_DB < "${NEW_SKYLINE_PATH}/updates/sql/${NEW_SKYLINE_VERSION}.sql"

# Stop all other Skyline services
SKYLINE_SERVICES="horizon
analyzer
mirage
crucible
boundary
ionosphere
luminosity
panorama
webapp"
for i in $SKYLINE_SERVICES
do
  /etc/init.d/$i stop
done
  • Move your current Skyline directory to a backup directory and move the new Skyline v1.3.0 with your new settings.py from the temp location to your working Skyline directory, (change your paths as appropriate) e.g.
mv "$CURRENT_SKYLINE_PATH" "${CURRENT_SKYLINE_PATH}.${OLD_SKYLINE_VERSION}"
mv "$NEW_SKYLINE_PATH" "$CURRENT_SKYLINE_PATH"
  • Update the dependencies
PYTHON_MAJOR_VERSION="2.7"
PYTHON_VIRTUALENV_DIR="/opt/python_virtualenv"
# Handle 2.7.14 and 2.7.16
PROJECT="skyline-py2716"
if [ ! -d "${PYTHON_VIRTUALENV_DIR}/projects/${PROJECT}" ]; then
  PROJECT="skyline-py2714"
fi
cd "${PYTHON_VIRTUALENV_DIR}/projects/${PROJECT}"
source bin/activate
# This can take lots of minutes with the new pandas
bin/"pip${PYTHON_MAJOR_VERSION}" install $(cat "${CURRENT_SKYLINE_PATH}/requirements.txt" | grep "^numpy\|^scipy\|^patsy\|^pandas" | tr '\n' ' ')
# Remove the luminol egg as it will already be installed and pip will moan
cat "${CURRENT_SKYLINE_PATH}/requirements.txt" | grep -v "luminol" > /tmp/requirements.txt

# IF YOU ARE ON CENTOS 6 and running mainstream mysql-server-5.1 uncomment
# the following line and run it to fix to mysql-python-connector 8.0.6
#cat "${CURRENT_SKYLINE_PATH}/requirements.txt" | grep -v "luminol\|mysql-connector-python" > /tmp/requirements.txt

bin/"pip${PYTHON_MAJOR_VERSION}" install -r /tmp/requirements.txt
  • Start the all Skyline services (change as appropriate for your set up) e.g.
# Start all other Skyline services
SKYLINE_SERVICES="panorama
luminosity
horizon
analyzer
mirage
crucible
boundary
ionosphere
webapp"
for i in $SKYLINE_SERVICES
do
  /etc/init.d/$i start
done
  • Check the logs
# How are they running
tail -n 20 /var/log/skyline/*.log

# Any errors - each app
find /var/log/skyline -type f -name "*.log" | while read skyline_logfile
do
  echo "#####
# Checking for errors in $skyline_logfile"
  cat "$skyline_logfile" | grep -B2 -A10 -i "error ::\|traceback" | tail -n 60
  echo ""
  echo ""
done