2.0.0¶
-v2.0.0 - December 31, 2019 (not released)- v2.0.0 - August 11, 2020
This is a major release with additional features functionality and bug fixes. More importantly this release moves Skyline fully to Python-3.8
IMPORTANT NOTICE¶
- In order to upgrade to Python 3.8.x see Upgrading from py2 to py3
- Skyline requires a modification to the runner.py of the python-daemon 2.2.x package to allow it to run on Python 3.
Summary of changes in v2.0.0¶
Lots of changes have been introduced to accommodate Python 3.x and 2.7, not all changes are documented here related to the py3 changes just the main ones and additional functionality. For a full list of the changes see https://github.com/earthgecko/skyline/compare/v1.3.0…v2.0.0 (v1.3.1 was a very minor change).
- py3 (3262)
- Update least_squares & grubbs algorithms by using sample standard deviation as per PR #124 (3256), thanks to @51wenyu
- py3 Redis binary objects not strings (3266)
- Always send anomaly_breakdown and exception metrics (3288)
- Record the anomaly_end_timestamp (3306)
- Gracefully handle db failure (3310)
- Added multiple webapp api methods (3336, 3350)
- Add CUSTOM_ALERT_OPTS to settings (3194) ported to Boundary
- py3 - handle system parameter in Graphite cactiStyle (3294)
- Handle urllib2 in py3 (3290)
- flux.populateMetricQueue Full (3254)
- flux - populate_metric_worker - handle None in datapoints (3312)
- Allow Skyline to send metrics to another Carbon host (3250) in flux
- py3 handle bytes and not str in pickles (3278)
- horizon - listen - py3 handle msgpack bytes (3272)
- Handle py2 xange and py3 range (3280)
- py3 - handle pymemcache bytes not str (3304)
- Handle csv.reader in py3 (3302)
- Add label to features profile (2516) - Added missing echo label
- Ionosphere spawn_learn_process hanging on docker (3296)
- Vista - batch Graphite requests (3338)
- Reduce vista logging (3258)
- vista - fetcher add time_to_fetch metric (3260)
- Deprecate string.replace for py3 (3270)
- ionosphere_backend.py fallback to available ts_json (3286)
- users DB table (3230)
- Label and relate anomalies (2476)
- Add label to features profile (2516)
- fp creation json response (3348)
- webapp - rebrow - jwt.encode generating bytes instead of a string in py3 (3268)
- docker (3002) - functions added
- Make Skyline docker handle OOMkiller (3318)
- And many, many more …
Update notes¶
- These update instruction apply to upgrading from v1.3.0 (or v1.3.1) to v2.0.0 only. However as with all Skyline updates it is possible to go through the update notes for each version and make your own update notes/process to take you from version x to version y.
- Due to the nature of the upgrade ensure that you have v1.3.0 (or v1.3.1) running stable for a period on Python 2.7 before attempting to update to v2.0.0
- There are changes to the DB in v2.0.0
- There are changes to settings.py in v2.0.0, please ensure you diff your current and the new settings.py as there are appended additions but there are also additional settings that have been added to existing settings blocks. The additions and changes described below.
settings.py Changes¶
Modified settings:
settings.ALERTS
modified to include http_alerter typesettings.CUSTOM_ALERT_OPTS
modified to include ionosphere_link_pathsettings.DO_NOT_SKIP_LIST
modified to include skyline.exceptionssettings.BOUNDARY_METRICS
modified to include http_alerter typesettings.BOUNDARY_ALERTER_OPTS
modified to include http_alerter type examplesettings.DO_NOT_SKIP_LIST
settings.IONOSPHERE_LEARN_NAMESPACE_CONFIG
settings.VISTA_FETCH_GRAPHITE_METRICS
was changed to andsettings.VISTA_FETCH_METRICS
the example Prometheus metric using a custom URI was corrected to not pass a pre-population tuple
New settings:
settings.VERIFY_SSL
settings.SKYLINE_METRICS_CARBON_HOST
settings.SKYLINE_METRICS_CARBON_PORT
settings.DATA_UPLOADS_PATH
settings.HTTP_ALERTERS_ENABLED
settings.START_IF_NO_DB
settings.IDENTIFY_AIRGAPS
settings.MAX_AIRGAP_PERIOD
CHECK_AIRGAPS
SKIP_AIRGAPS
IDENTIFY_UNORDERED_TIMESERIES
BATCH_PROCESSING
BATCH_PROCESSING_STALE_PERIOD
BATCH_PROCESSING_DEBUG
BATCH_PROCESSING_NAMESPACES
METRICS_INACTIVE_AFTER
settings.CUSTOM_ALGORITHMS
settings.DEBUG_CUSTOM_ALGORITHMS
settings.EXTERNAL_ALERTS
settings.HTTP_ALERTERS_OPTS
settings.ROOMBA_DO_NOT_PROCESS_BATCH_METRICS
settings.PANORAMA_CHECK_INTERVAL
settings.PANORAMA_INSERT_METRICS_IMMEDIATELY
settings.MIRAGE_ALWAYS_METRICS
settings.MIRAGE_AUTOFILL_TOOSHORT
settings.WEBAPP_ACCEPT_DATA_UPLOADS
settings.IONOSPHERE_MANAGE_PURGE
settings.IONOSPHERE_GRAPHITE_NOW_GRAPHS_OVERRIDE
settings.IONOSPHERE_UNTRAINABLES
settings.LUMINOSITY_RELATED_TIME_PERIOD
settings.FLUX_MAX_AGE
settings.FLUX_PROCESS_UPLOADS
settings.FLUX_SAVE_UPLOADS
settings.FLUX_SAVE_UPLOADS_PATH
settings.FLUX_UPLOADS_KEYS
settings.VISTA_FETCH_METRICS
settings.VISTA_GRAPHITE_BATCH_SIZE
How to update from v1.3.0 (and v1.3.1)¶
- Deploy a new Python-3.8.3 virtualenv, see Running Skyline in a Python virtualenv
- Ensure all the dependencies are at the correct versions in the new Python-3.8.3 virtualenv
PYTHON_MAJOR_VERSION="3.8"
PYTHON_VIRTUALENV_DIR="/opt/python_virtualenv"
PROJECT="skyline-py383"
cd "${PYTHON_VIRTUALENV_DIR}/projects/${PROJECT}"
source bin/activate
# This can take lots of minutes with the new pandas
bin/"pip${PYTHON_MAJOR_VERSION}" install $(cat "${CURRENT_SKYLINE_PATH}/requirements.txt" | grep "^numpy\|^scipy\|^patsy\|^pandas" | tr '\n' ' ')
# Remove the luminol egg as it will already be installed and pip will moan
cat "${CURRENT_SKYLINE_PATH}/requirements.txt" | grep -v "luminol" > /tmp/requirements.txt
# IF YOU ARE ON CENTOS 6 and running mainstream mysql-server-5.1 uncomment
# the following line and run it to fix to mysql-python-connector 8.0.6
#cat "${CURRENT_SKYLINE_PATH}/requirements.txt" | grep -v "luminol\|mysql-connector-python" > /tmp/requirements.txt
bin/"pip${PYTHON_MAJOR_VERSION}" install -r /tmp/requirements.txt
- Download the new release tag or clone/update to get it to a temp location, ready to be deployed.
NEW_SKYLINE_VERSION="v2.0.0" # Your new Skyline version
OLD_SKYLINE_VERSION="v1.3.0" # Your old Skyline version
# OR v1.3.1 if you updated to that version
# OLD_SKYLINE_VERSION="v1.3.1" # Your old Skyline version
CURRENT_SKYLINE_PATH="/opt/skyline/github/skyline" # Your Skyline path
NEW_SKYLINE_PATH="${CURRENT_SKYLINE_PATH}.${NEW_SKYLINE_VERSION}" # Your new Skyline path
mkdir -p "${CURRENT_SKYLINE_PATH}.${NEW_SKYLINE_VERSION}"
cd "${CURRENT_SKYLINE_PATH}.${NEW_SKYLINE_VERSION}"
git clone https://github.com/earthgecko/skyline .
git checkout "$NEW_SKYLINE_VERSION"
# Fix python-daemon - which fails to run on Python 3 (numerous PRs are waiting
# to fix it https://pagure.io/python-daemon/pull-requests)
cp /opt/python_virtualenv/projects/skyline-py383/lib/python3.8/site-packages/daemon/runner.py /opt/python_virtualenv/projects/skyline-py383/lib/python3.8/site-packages/daemon/runner.py.bak
cat "$NEW_SKYLINE_PATH/utils/python-daemon/runner.2.2.4.py" > /opt/python_virtualenv/projects/skyline-py383/lib/python3.8/site-packages/daemon/runner.py
# minor change related to unbuffered bytes I/O
diff /opt/python_virtualenv/projects/skyline-py383/lib/python3.8/site-packages/daemon/runner.py.bak /opt/python_virtualenv/projects/skyline-py383/lib/python3.8/site-packages/daemon/runner.py
# settings.py
cp "$NEW_SKYLINE_PATH/skyline/settings.py" "$NEW_SKYLINE_PATH/skyline/settings.py.${NEW_SKYLINE_VERSION}.bak"
# DIFF your settings.py with the new settings.py
diff "${CURRENT_SKYLINE_PATH}/skyline/settings.py" "$NEW_SKYLINE_PATH/skyline/settings.py.${NEW_SKYLINE_VERSION}.bak"
# Create a new settings.py file in the new version based on your existing
# settings.py file
cat "${CURRENT_SKYLINE_PATH}/skyline/settings.py" > "$NEW_SKYLINE_PATH/skyline/settings.py"
# ADD the appropriate new settings to your settings file and modify any
# changed settings as appropriate for your set up.
vi "$NEW_SKYLINE_PATH/skyline/settings.py"
# Stop any/all service controls like monit, etc that are controlling
# Skyline services.
# Stop Skyline DB related services
SKYLINE_SERVICES="webapp
ionosphere
luminosity
panorama"
for i in $SKYLINE_SERVICES
do
# /etc/init.d/$i stop
# or
systemctl stop $i
done
# BACKUP THE DB AND APPLY THE NEW SQL
BACKUP_DIR="/tmp" # Where you want to backup the DB to
MYSQL_USER="<YOUR_MYSQL_USER>"
MYSQL_HOST="127.0.0.1" # Your MySQL IP
MYSQL_DB="skyline" # Your MySQL Skyline DB name
# Backup DB
mkdir -p $BACKUP_DIR
mysqldump -u$MYSQL_USER -p $MYSQL_DB > $BACKUP_DIR/pre.$NEW_SKYLINE_VERSION.$MYSQL_DB.sql
# Update DB
mysql -u$MYSQL_USER -p $MYSQL_DB < "${NEW_SKYLINE_PATH}/updates/sql/${NEW_SKYLINE_VERSION}.sql"
# Stop all other Skyline services
SKYLINE_SERVICES="horizon
analyzer
mirage
crucible
boundary
ionosphere
luminosity
panorama
webapp
vista
flux"
for i in $SKYLINE_SERVICES
do
# /etc/init.d/$i stop
# or
systemctl stop $i
done
- Move your current Skyline directory to a backup directory and move the new Skyline v2.0.0 with your new settings.py from the temp location to your working Skyline directory, (change your paths as appropriate) e.g.
mv "$CURRENT_SKYLINE_PATH" "${CURRENT_SKYLINE_PATH}.${OLD_SKYLINE_VERSION}"
mv "$NEW_SKYLINE_PATH" "$CURRENT_SKYLINE_PATH"
- Start the all Skyline services (change as appropriate for your set up) e.g.
# Start all other Skyline services
SKYLINE_SERVICES="panorama
horizon
flux
vista
luminosity
analyzer
mirage
crucible
boundary
ionosphere
webapp"
for i in $SKYLINE_SERVICES
do
# /etc/init.d/$i start
# or
systemctl start $i
done
# Restart any/all service controls like monit, etc that are controlling
# Skyline services.
- Check the logs
# How are they running
tail -n 20 /var/log/skyline/*.log
# Any errors - each app
find /var/log/skyline -type f -name "*.log" | while read skyline_logfile
do
echo "#####
# Checking for errors in $skyline_logfile"
cat "$skyline_logfile" | grep -B2 -A10 -i "error ::\|traceback" | tail -n 60
echo ""
echo ""
done