1.3.0¶
v1.3.0 - September 25, 2019
This is a features functionality release with minor bug fixes. This release introduces a few new features and new functionality:
- A new Ionosphere feature related to validating matches.
- The ability to run the Skyline stack on docker
- Introduces Skyline Flux for receiving metrics via HTTP and submitting them to Graphite.
- Introduces Skyline Vista for fetching metrics from remote sources.
Please see the new documentation pages for full details about each new addition.
Changes in v1.3.0¶
Lots of changes have been introduced to accommodate docker and the addition of Flux and Vista.
- Use GRAPHITE_RENDER_URI and do not try to alert if the settings are default in analyzer/alerters.py (3002)
- Wrapped new get_graphite_render_uri and get_graphite_custom_headers functions in try and except in analyzer/alerters.py (3002)
- The management of the analyzer Redis algorithm exception sets was previously handled a if settings.ENABLE_MIRAGE block. This meant that if there are no mirage metrics, the sets were not managed. Added manage_analyzer_algorithm_exception_sets in analyzer/analyzer.py to manage these sets in their own context. (3002, 3020)
- Re-enabled the illuminance metric in analyzer (2580)
- Declare and test if the derivative_metrics Redis set exists in analyzer so that errors are not thrown if there are actually no derivative metrics (3002)
- Use GRAPHITE_RENDER_URI and use new CARBON_HOST variable instead of GRAPHITE_HOST in send_graphite_metric (3002)
- Handle self signed certificate on Docker with requests (3002)
- Handle when Skyline is first installed, if luminosity is enabled it reports errors as there are no anomaly ids (3002, 3016)
- Log appropriate to whether memcache is enabled or not (3002, 3016)
- Handle self signed certificate on Docker with requests (3002)
- Do not analyse short time series, only return a time series sample if the sample has sufficient data points otherwise get_anomalies() will throw an error (3002, 3008)
- Added docker settings (3002)
- Allow rebrow to connect to Redis on the socket too and if docker dsplay the REDIS_PASSWORD in rebrow_login (3002)
- Use GRAPHITE_RENDER_URI (3002)
- Revert to original skyline.httpd.conf.d.example and added skyline.docker.apache2.conf.d.example (3002)
- Reformat print in analyzer/agent.py (3002)
- When there are multiple channels declared Skyline only wants to update the panorama.slack_thread_ts Redis key once for the primary Skyline channel, so if the cache key exists do not update, analyzer/alerters.py (3110)
- Added a hashlib hexdigest method which has not been verified so using the original method useless the base_name is longer than 254 chars, analyzer/alerters.py (3002)
- Allow for DOCKER_FAKE_EMAIL_ALERTS and GRAPHITE_CUSTOM_HEADERS, analyzer/alerters.py (3002)
- Added get_graphite_port, get_graphite_render_uri and get_graphite_custom_headers functions, analyzer/alerters.py (3002)
- Reduce amount of Manager instances that are used as each requires a copy of entire memory to be copied into each subprocess so this results in a python process per Manager instance, using as much memory as the parent. OK on a server, not so much in a container. Disabled all the Manager() lists and replaced with Redis sets, analyzer/analyzer.py (3002, 3032, 3034)
- Moved the interpolation of the MIRAGE_PERIODIC variables to the top of the file out of spawn_process so they can be accessed in run too and check the added mirage_periodic_check_keys list rather than checking each and every metric name key, analyzer/analyzer.py (2882)
- Disabled for now as in concept phase, analyzer/analyzer.py (2580)
- The management of the analyzer Redis algorithm exception sets was previously handled in an if settings.ENABLE_MIRAGE block This meant that if there are no mirage metrics, the sets were not managed, analyzer/analyzer.py (3020)
- Changes for docker, not complete or fully tested.
- Reduce amount of Manager instances that are used as each requires a copy of entire memory to be copied into each subprocess so this results in a python process per Manager instance, using as much memory as the parent. OK on a server, not so much in a container. Disabled all the Manager() lists and replaced with Redis sets, boundary/boundary.py (3002, 3032, 3034)
- Use GRAPHITE_RENDER_URI, boundary/boundary.py (3002)
- Reduce amount of Manager instances that are used as each requires a copy of entire memory to be copied into each subprocess so this results in a python process per Manager instance, using as much memory as the parent. OK on a server, not so much in a container. Disabled all the Manager() lists and replaced with Redis sets, crucible/crucible.py (3002, 3032)
- Add a crucible - skyline.consensus.anomalies.png to plot Skyline anomalies if CONSENSUS is achieved and create file resources skyline.anomalies_score.txt and skyline.anomalies.csv, crucible/crucible_algorithms.py (3106)
- Reduce amount of Manager instances that are used as each requires a copy of entire memory to be copied into each subprocess so this results in a python process per Manager instance, using as much memory as the parent. OK on a server, not so much in a container. Disabled all the Manager() lists and replaced with Redis sets, skyline/ionosphere/ionosphere.py (3002, 3032)
- Do not run checks if namespace has matched multiple times in the last 10 minutes. However determining which Skyline related metrics are feeding back are quite difficult to ascertain. So use the ionosphere_busy logic again and use or find the skyline host namespace and if busy do not analyse the Skyline host namespace while ionosphere is busy related to SKYLINE_FEEDBACK_NAMESPACES, skyline/ionosphere/ionosphere.py (3050)
- Only process FULL_DURATION feature profiles if there is a ionosphere.unique_metrics Redis set (2484)
- Handle self signed certificate on Docker, skyline/ionosphere/learn.py (3002)
- Reduce amount of Manager instances that are used as each requires a copy of entire memory to be copied into each subprocess so this results in a python process per Manager instance, using as much memory as the parent. OK on a server, not so much in a container. Disabled all the Manager() lists and replaced with Redis sets, skyline/luminosity/luminosity.py (3002, 3032)
- When Skyline is first installed, if luminosity is enabled it reports errors as there are no anomaly ids, skyline/luminosity/luminosity.py (3016)
- Handle self signed certificate on Docker, skyline/luminosity/process_correlations.py (3002)
- Reduce amount of Manager instances that are used as each requires a copy of entire memory to be copied into each subprocess so this results in a python process per Manager instance, using as much memory as the parent. OK on a server, not so much in a container. Disabled all the Manager() lists and replaced with Redis sets, skyline/mirage/mirage.py (3002, 3032)
- When there are multiple channels declared Skyline only wants to update the panorama.slack_thread_ts Redis key once for the primary Skyline channel, so if the cache key exists do not update, skyline/mirage/mirage_alerters.py (3110)
- Reduce amount of Manager instances that are used as each requires a copy of entire memory to be copied into each subprocess so this results in a python process per Manager instance, using as much memory as the parent. OK on a server, not so much in a container. Disabled all the Manager() lists and replaced with Redis sets, skyline/mirage/mirage.py (3002, 3032)
- Allow webapp to connect on reborw via socket if running on docker, skyline/webapp/templates/rebrow_login.html (3002)
- Handle self signed certificate on Docker, skyline/webapp/ionosphere_backend.py (3190)
- Handle if all features profiles have been disabled in the ionosphere_search function, skyline/webapp/ionosphere_backend.py (3190)
- Added Ionosphere - validated matches (3082, 3084)
- Added metrics id to relevant web pages, skyline/webapp/webapp.py (2990)
- Added docker related settings to settings.py (3002)
- Added Ionosphere - validated matches DB changes to skyline.sql (3084)
- Added docker changes and some Pyton 3.7 testing changes to skyline/skyline_functions.py (3002, 2828)
- Updated version and branch in skyline/skyline_version.py (3002)
- Updated to 1.3.0 - skyline/utils/dawn/skyline.dawn.sh (3002)
- Uppdated dependencies
- skyline.dawn.sh changes related to docker (3002)
- Update version and dependencies
- Added a users tables to the DB to allow to map user to actions, labels, etc (3230, 2476, 2516)
Update notes¶
- These update instruction apply to upgrading from v1.2.17 to v1.3.0 only. However as with all Skyline updates it is possible to go through the update notes for each version and make your own update notes/process to take you from version x to version y.
- There are changes to the DB in v1.3.0
Warning
It is assumed that the database is not running with an offset and the database increments ids by 1. If you are running an offset please review all SQL updates to determine if you need to change anything in the SQL when you apply Skyline SQL updates.
- There are changes to settings.py in v1.3.0, please ensure you diff your current and the new settings.py as there are appended additions but there are also additional settings that have been added to existing settings blocks. The additions and changes described below.
settings.py Changes¶
- New settings:
settings.GRAPHITE_AUTH_HEADER
settings.GRAPHITE_CUSTOM_HEADERS
settings.GRAPHITE_RENDER_URI
settings.CARBON_HOST
settings.SKYLINE_FEEDBACK_NAMESPACES
settings.CUSTOM_ALERT_OPTS
settings.DOCKER
settings.DOCKER_DISPLAY_REDIS_PASSWORD_IN_REBROW
settings.DOCKER_FAKE_EMAIL_ALERTS
settings.FLUX_IP = '127.0.0.1
settings.FLUX_PORT
settings.FLUX_WORKERS
settings.FLUX_SELF_API_KEY
settings.FLUX_BACKLOG
settings.FLUX_SEND_TO_CARBON
settings.FLUX_CARBON_HOST
settings.FLUX_CARBON_PORT
settings.FLUX_CARBON_PICKLE_PORT
settings.FLUX_SEND_TO_STATSD
settings.FLUX_STATSD_HOST
settings.FLUX_STATSD_PORT
settings.VISTA_ENABLED
settings.VISTA_FETCHER_PROCESSES
settings.VISTA_FETCHER_PROCESS_MAX_RUNTIME
settings.VISTA_WORKER_PROCESSES
settings.VISTA_DO_NOT_SUBMIT_CURRENT_MINUTE
settings.VISTA_FETCH_GRAPHITE_METRICS
- New Graphite related settings have been added to the main settings block,
settings.GRAPHITE_AUTH_HEADER
andsettings.GRAPHITE_CUSTOM_HEADERS
neither are enabled by default. Additionally asettings.GRAPHITE_RENDER_URI
setting has been added that is now used in the existing and modifiedsettings.GRAPH_URL
setting. settings.CARBON_HOST
setting has been added that defaults tosettings.GRAPHITE_HOST
to enable the use of a different CARBON_HOST.- A new settings
settings.SKYLINE_FEEDBACK_NAMESPACES
has been added to to reduce analysis load during busy times, read the docstring for a full description. - A new setting
settings.CUSTOM_ALERT_OPTS
allows to add custom labels, graph titles and subjects to alerts. - A new Docker settings block has been added with all the Docker related settings, the defaults these settings are for Docker not being enabled.
- A new Flux settings block has been added for all the Flux settings
- A new Vista settings block has been added for all the Vista settings.
How to update from v1.2.17¶
- Download the new release tag or clone/update to get it to a temp location, ready to be deployed.
NEW_SKYLINE_VERSION="v1.3.0" # Your new Skyline version
OLD_SKYLINE_VERSION="v1.2.17" # Your old Skyline version
CURRENT_SKYLINE_PATH="/opt/skyline/github/skyline" # Your Skyline path
NEW_SKYLINE_PATH="${CURRENT_SKYLINE_PATH}.${NEW_SKYLINE_VERSION}" # Your new Skyline path
mkdir -p "${CURRENT_SKYLINE_PATH}.${NEW_SKYLINE_VERSION}"
cd "${CURRENT_SKYLINE_PATH}.${NEW_SKYLINE_VERSION}"
git clone https://github.com/earthgecko/skyline .
git checkout "$NEW_SKYLINE_VERSION"
cp "$NEW_SKYLINE_PATH/skyline/settings.py" "$NEW_SKYLINE_PATH/skyline/settings.py.${NEW_SKYLINE_VERSION}.bak"
# DIFF your settings.py with the new settings.py
# diff "${CURRENT_SKYLINE_PATH}/skyline/settings.py" "$NEW_SKYLINE_PATH/skyline/settings.py.${NEW_SKYLINE_VERSION}.bak"
#
cat "${CURRENT_SKYLINE_PATH}/skyline/settings.py" > "$NEW_SKYLINE_PATH/skyline/settings.py"
# ADD the appropriate new settings to your settings file.
vi "$NEW_SKYLINE_PATH/skyline/settings.py"
# Stop Skyline DB related services
SKYLINE_SERVICES="webapp
ionosphere
luminosity
panorama"
for i in $SKYLINE_SERVICES
do
/etc/init.d/$i stop
done
# And stop any service controls like monit or systemd
# BACKUP THE DB AND APPLY THE NEW SQL
BACKUP_DIR="/tmp" # Where you want to backup the DB to
MYSQL_USER="<YOUR_MYSQL_USER>"
MYSQL_HOST="127.0.0.1" # Your MySQL IP
MYSQL_DB="skyline" # Your MySQL Skyline DB name
# Backup DB
mkdir -p $BACKUP_DIR
mysqldump -u$MYSQL_USER -p $MYSQL_DB > $BACKUP_DIR/pre.$NEW_SKYLINE_VERSION.$MYSQL_DB.sql
# Update DB
mysql -u$MYSQL_USER -p $MYSQL_DB < "${NEW_SKYLINE_PATH}/updates/sql/${NEW_SKYLINE_VERSION}.sql"
# Stop all other Skyline services
SKYLINE_SERVICES="horizon
analyzer
mirage
crucible
boundary
ionosphere
luminosity
panorama
webapp"
for i in $SKYLINE_SERVICES
do
/etc/init.d/$i stop
done
- Move your current Skyline directory to a backup directory and move the new Skyline v1.3.0 with your new settings.py from the temp location to your working Skyline directory, (change your paths as appropriate) e.g.
mv "$CURRENT_SKYLINE_PATH" "${CURRENT_SKYLINE_PATH}.${OLD_SKYLINE_VERSION}"
mv "$NEW_SKYLINE_PATH" "$CURRENT_SKYLINE_PATH"
- Update the dependencies
PYTHON_MAJOR_VERSION="2.7"
PYTHON_VIRTUALENV_DIR="/opt/python_virtualenv"
# Handle 2.7.14 and 2.7.16
PROJECT="skyline-py2716"
if [ ! -d "${PYTHON_VIRTUALENV_DIR}/projects/${PROJECT}" ]; then
PROJECT="skyline-py2714"
fi
cd "${PYTHON_VIRTUALENV_DIR}/projects/${PROJECT}"
source bin/activate
# This can take lots of minutes with the new pandas
bin/"pip${PYTHON_MAJOR_VERSION}" install $(cat "${CURRENT_SKYLINE_PATH}/requirements.txt" | grep "^numpy\|^scipy\|^patsy\|^pandas" | tr '\n' ' ')
# Remove the luminol egg as it will already be installed and pip will moan
cat "${CURRENT_SKYLINE_PATH}/requirements.txt" | grep -v "luminol" > /tmp/requirements.txt
# IF YOU ARE ON CENTOS 6 and running mainstream mysql-server-5.1 uncomment
# the following line and run it to fix to mysql-python-connector 8.0.6
#cat "${CURRENT_SKYLINE_PATH}/requirements.txt" | grep -v "luminol\|mysql-connector-python" > /tmp/requirements.txt
bin/"pip${PYTHON_MAJOR_VERSION}" install -r /tmp/requirements.txt
- Start the all Skyline services (change as appropriate for your set up) e.g.
# Start all other Skyline services
SKYLINE_SERVICES="panorama
luminosity
horizon
analyzer
mirage
crucible
boundary
ionosphere
webapp"
for i in $SKYLINE_SERVICES
do
/etc/init.d/$i start
done
- Check the logs
# How are they running
tail -n 20 /var/log/skyline/*.log
# Any errors - each app
find /var/log/skyline -type f -name "*.log" | while read skyline_logfile
do
echo "#####
# Checking for errors in $skyline_logfile"
cat "$skyline_logfile" | grep -B2 -A10 -i "error ::\|traceback" | tail -n 60
echo ""
echo ""
done