2.0.0

-v2.0.0 - December 31, 2019 (not released)- v2.0.0 - August 11, 2020

This is a major release with additional features functionality and bug fixes. More importantly this release moves Skyline fully to Python-3.8

IMPORTANT NOTICE

  • In order to upgrade to Python 3.8.x see Upgrading from py2 to py3
  • Skyline requires a modification to the runner.py of the python-daemon 2.2.x package to allow it to run on Python 3.

Summary of changes in v2.0.0

Lots of changes have been introduced to accommodate Python 3.x and 2.7, not all changes are documented here related to the py3 changes just the main ones and additional functionality. For a full list of the changes see https://github.com/earthgecko/skyline/compare/v1.3.0…v2.0.0 (v1.3.1 was a very minor change).

  • py3 (3262)
  • Update least_squares & grubbs algorithms by using sample standard deviation as per PR #124 (3256), thanks to @51wenyu
  • py3 Redis binary objects not strings (3266)
  • Always send anomaly_breakdown and exception metrics (3288)
  • Record the anomaly_end_timestamp (3306)
  • Gracefully handle db failure (3310)
  • Added multiple webapp api methods (3336, 3350)
  • Add CUSTOM_ALERT_OPTS to settings (3194) ported to Boundary
  • py3 - handle system parameter in Graphite cactiStyle (3294)
  • Handle urllib2 in py3 (3290)
  • flux.populateMetricQueue Full (3254)
  • flux - populate_metric_worker - handle None in datapoints (3312)
  • Allow Skyline to send metrics to another Carbon host (3250) in flux
  • py3 handle bytes and not str in pickles (3278)
  • horizon - listen - py3 handle msgpack bytes (3272)
  • Handle py2 xange and py3 range (3280)
  • py3 - handle pymemcache bytes not str (3304)
  • Handle csv.reader in py3 (3302)
  • Add label to features profile (2516) - Added missing echo label
  • Ionosphere spawn_learn_process hanging on docker (3296)
  • Vista - batch Graphite requests (3338)
  • Reduce vista logging (3258)
  • vista - fetcher add time_to_fetch metric (3260)
  • Deprecate string.replace for py3 (3270)
  • ionosphere_backend.py fallback to available ts_json (3286)
  • users DB table (3230)
  • Label and relate anomalies (2476)
  • Add label to features profile (2516)
  • fp creation json response (3348)
  • webapp - rebrow - jwt.encode generating bytes instead of a string in py3 (3268)
  • docker (3002) - functions added
  • Make Skyline docker handle OOMkiller (3318)
  • And many, many more …

Update notes

  • These update instruction apply to upgrading from v1.3.0 (or v1.3.1) to v2.0.0 only. However as with all Skyline updates it is possible to go through the update notes for each version and make your own update notes/process to take you from version x to version y.
  • Due to the nature of the upgrade ensure that you have v1.3.0 (or v1.3.1) running stable for a period on Python 2.7 before attempting to update to v2.0.0
  • There are changes to the DB in v2.0.0
  • There are changes to settings.py in v2.0.0, please ensure you diff your current and the new settings.py as there are appended additions but there are also additional settings that have been added to existing settings blocks. The additions and changes described below.

settings.py Changes

Modified settings:

New settings:

How to update from v1.3.0 (and v1.3.1)

PYTHON_MAJOR_VERSION="3.8"
PYTHON_VIRTUALENV_DIR="/opt/python_virtualenv"
PROJECT="skyline-py383"
cd "${PYTHON_VIRTUALENV_DIR}/projects/${PROJECT}"
source bin/activate
# This can take lots of minutes with the new pandas
bin/"pip${PYTHON_MAJOR_VERSION}" install $(cat "${CURRENT_SKYLINE_PATH}/requirements.txt" | grep "^numpy\|^scipy\|^patsy\|^pandas" | tr '\n' ' ')
# Remove the luminol egg as it will already be installed and pip will moan
cat "${CURRENT_SKYLINE_PATH}/requirements.txt" | grep -v "luminol" > /tmp/requirements.txt

# IF YOU ARE ON CENTOS 6 and running mainstream mysql-server-5.1 uncomment
# the following line and run it to fix to mysql-python-connector 8.0.6
#cat "${CURRENT_SKYLINE_PATH}/requirements.txt" | grep -v "luminol\|mysql-connector-python" > /tmp/requirements.txt

bin/"pip${PYTHON_MAJOR_VERSION}" install -r /tmp/requirements.txt
  • Download the new release tag or clone/update to get it to a temp location, ready to be deployed.
NEW_SKYLINE_VERSION="v2.0.0"    # Your new Skyline version
OLD_SKYLINE_VERSION="v1.3.0"    # Your old Skyline version
# OR v1.3.1 if you updated to that version
# OLD_SKYLINE_VERSION="v1.3.1"    # Your old Skyline version

CURRENT_SKYLINE_PATH="/opt/skyline/github/skyline"                 # Your Skyline path
NEW_SKYLINE_PATH="${CURRENT_SKYLINE_PATH}.${NEW_SKYLINE_VERSION}"  # Your new Skyline path

mkdir -p "${CURRENT_SKYLINE_PATH}.${NEW_SKYLINE_VERSION}"
cd "${CURRENT_SKYLINE_PATH}.${NEW_SKYLINE_VERSION}"
git clone https://github.com/earthgecko/skyline .
git checkout "$NEW_SKYLINE_VERSION"

# Fix python-daemon - which fails to run on Python 3 (numerous PRs are waiting
# to fix it https://pagure.io/python-daemon/pull-requests)
cp /opt/python_virtualenv/projects/skyline-py383/lib/python3.8/site-packages/daemon/runner.py /opt/python_virtualenv/projects/skyline-py383/lib/python3.8/site-packages/daemon/runner.py.bak
cat "$NEW_SKYLINE_PATH/utils/python-daemon/runner.2.2.4.py" > /opt/python_virtualenv/projects/skyline-py383/lib/python3.8/site-packages/daemon/runner.py
# minor change related to unbuffered bytes I/O
diff /opt/python_virtualenv/projects/skyline-py383/lib/python3.8/site-packages/daemon/runner.py.bak /opt/python_virtualenv/projects/skyline-py383/lib/python3.8/site-packages/daemon/runner.py

# settings.py
cp "$NEW_SKYLINE_PATH/skyline/settings.py" "$NEW_SKYLINE_PATH/skyline/settings.py.${NEW_SKYLINE_VERSION}.bak"
# DIFF your settings.py with the new settings.py
diff "${CURRENT_SKYLINE_PATH}/skyline/settings.py" "$NEW_SKYLINE_PATH/skyline/settings.py.${NEW_SKYLINE_VERSION}.bak"

# Create a new settings.py file in the new version based on your existing
# settings.py file
cat "${CURRENT_SKYLINE_PATH}/skyline/settings.py" > "$NEW_SKYLINE_PATH/skyline/settings.py"

# ADD the appropriate new settings to your settings file and modify any
# changed settings as appropriate for your set up.
vi "$NEW_SKYLINE_PATH/skyline/settings.py"

# Stop any/all service controls like monit, etc that are controlling
# Skyline services.

# Stop Skyline DB related services
SKYLINE_SERVICES="webapp
ionosphere
luminosity
panorama"
for i in $SKYLINE_SERVICES
do
  # /etc/init.d/$i stop
  # or
  systemctl stop $i
done

# BACKUP THE DB AND APPLY THE NEW SQL
BACKUP_DIR="/tmp"  # Where you want to backup the DB to
MYSQL_USER="<YOUR_MYSQL_USER>"
MYSQL_HOST="127.0.0.1"  # Your MySQL IP
MYSQL_DB="skyline"  # Your MySQL Skyline DB name

# Backup DB
mkdir -p $BACKUP_DIR
mysqldump -u$MYSQL_USER -p $MYSQL_DB > $BACKUP_DIR/pre.$NEW_SKYLINE_VERSION.$MYSQL_DB.sql

# Update DB
mysql -u$MYSQL_USER -p $MYSQL_DB < "${NEW_SKYLINE_PATH}/updates/sql/${NEW_SKYLINE_VERSION}.sql"

# Stop all other Skyline services
SKYLINE_SERVICES="horizon
analyzer
mirage
crucible
boundary
ionosphere
luminosity
panorama
webapp
vista
flux"
for i in $SKYLINE_SERVICES
do
  # /etc/init.d/$i stop
  # or
  systemctl stop $i
done
  • Move your current Skyline directory to a backup directory and move the new Skyline v2.0.0 with your new settings.py from the temp location to your working Skyline directory, (change your paths as appropriate) e.g.
mv "$CURRENT_SKYLINE_PATH" "${CURRENT_SKYLINE_PATH}.${OLD_SKYLINE_VERSION}"
mv "$NEW_SKYLINE_PATH" "$CURRENT_SKYLINE_PATH"
  • Start the all Skyline services (change as appropriate for your set up) e.g.
# Start all other Skyline services
SKYLINE_SERVICES="panorama
horizon
flux
vista
luminosity
analyzer
mirage
crucible
boundary
ionosphere
webapp"
for i in $SKYLINE_SERVICES
do
  # /etc/init.d/$i start
  # or
  systemctl start $i
done
# Restart any/all service controls like monit, etc that are controlling
# Skyline services.
  • Check the logs
# How are they running
tail -n 20 /var/log/skyline/*.log

# Any errors - each app
find /var/log/skyline -type f -name "*.log" | while read skyline_logfile
do
  echo "#####
# Checking for errors in $skyline_logfile"
  cat "$skyline_logfile" | grep -B2 -A10 -i "error ::\|traceback" | tail -n 60
  echo ""
  echo ""
done