1.2.12

v1.2.12 - April 3, 2019

Stable for new installs. Beta if upgrading.

DOWNGRADED to a pre-release as an update

Downgrade to a pre-release - April 4, 2019

To add a mechanism that reduces to initial load on Ionosphere when Ionosphere echo is enabled if there are a lot of existing Mirage feature profiles and to add some self load management to Ionosphere in general under the new Ionosphere echo paradigm.

New functionality

• Adds the new Ionosphere echo functionality related to improving the matching of Mirage based features profiles by between 30-50% depending on the metric. See https://earthgecko-skyline.readthedocs.io/en/stable/ionosphere_echo.html

Fixes

• Adds the ability for Skyline Boundary to handle very short time series.
• Updated tsfresh-0.4.0.py2.data.json.features.transposed.csv with new longer decimal values introduced by numpy==1.15.1 pandas==0.24.3

Changes from v1.2.11

• Allow Boundary to analyse short time series (#88, #2862)
• Bifurcate dependencies for py3 testing (#2588, #2828)
• Added functionality and changes to implement Ionosphere echo (#2484)
• Updated tsfresh-0.4.0.py2.data.json.features.transposed.csv with new longer decimal values introduced by numpy==1.15.1 pandas==0.24.3 (2902)

Update notes

• You can only upgrade to v1.2.12 from v1.2.11
• There are changes to the DB
• There are some changes to settings.py

How to update from v1.2.11

• Download the new release tag or clone/update to get it to a temp location, ready to be deployed.
• Diff your current running version skyline/settings.py with the new version of skyline/settings.py and create a new settings.py in the temp location appropriate for Skyline v1.2.12 and your set up with the new IONOSPHERE_ECHO settings, ready to be deployed. New settings to be added are:

IONOSPHERE_MAX_RUNTIME = 120
"""
:var IONOSPHERE_MAX_RUNTIME: The maximum number of seconds an Ionosphere check
    should run for.
:vartype IONOSPHERE_MAX_RUNTIME: int
"""

IONOSPHERE_ECHO_ENABLED = True
"""
:var IONOSPHERE_ECHO_ENABLED: This enables Ionosphere to create and test
    features profiles for Mirage metrics but at settings.FULL_DURATION as well.
    Features profiles will be made on the fly for any existing, validated Mirage
    metric features profiles.  Ionosphere's matching performance is increased
    between 30 to 50 percent when Ionosphere echo is run.
:vartype IONOSPHERE_ECHO_ENABLED: boolean
"""

IONOSPHERE_ECHO_MAX_FP_CREATE_TIME = 55
"""
:var IONOSPHERE_ECHO_MAX_FP_CREATE_TIME: The maximum number of seconds an
    Ionosphere echo process should run creating FULL_DURATION features profiles
    for created Mirage features profiles.  This setting is specifically relevant
    for Skyline implematations pre Ionosphere echo (v1.2.12) to prevent timeouts
    if Ionosphere echo needs to make > 30 echo features profiles for Mirage
    metrics with lots of existing features profiles.
:vartype IONOSPHERE_ECHO_MAX_FP_CREATE_TIME: int
"""

IONOSPHERE_ECHO_FEATURES_PERCENT_SIMILAR = 2.5
"""
:var IONOSPHERE_ECHO_FEATURES_PERCENT_SIMILAR: In terms of Ionosphere echo a
    value of 2.0 is the default.  This default is above the normal
    IONOSPHERE_FEATURES_PERCENT_SIMILAR due to that fact that the the resolution
    of Ionosphere echo is at FULL_DURATION.  During testing this value was
    tested at 1.0, 2 and 2.5, with 2.5 resulting in the most desirable results
    in terms of matching time series that are similarly not anomalous.
:vartype IONOSPHERE_ECHO_FEATURES_PERCENT_SIMILAR: float
"""

IONOSPHERE_ECHO_MINMAX_SCALING_FEATURES_PERCENT_SIMILAR = 3.5
"""
:var IONOSPHERE_ECHO_MINMAX_SCALING_FEATURES_PERCENT_SIMILAR: In terms of
    Ionosphere echo Min-Max scaling percentage similar, a value of 3.5 is the
    default.  This default is above the normal IONOSPHERE_FEATURES_PERCENT_SIMILAR
    due to that fact that the the resolution of Ionosphere echo is at FULL_DURATION
    and echo is using the normal IONOSPHERE_MINMAX_SCALING_RANGE_TOLERANCE to
    determine if Min-Max scaling should be run.  During testing this value was
    tested at 1, 2 and 3.5, with 3.5 resulting in the most desirable results in
    terms of matching time series that are similarly not anomalous.
:vartype IONOSPHERE_ECHO_MINMAX_SCALING_FEATURES_PERCENT_SIMILAR: float
"""

• Stop Skyline Ionosphere and Panorama services.
• Backup your Skyline database.
• Apply the SQL update to your Skyline database as per described below, change the below variables as appropriate to your set up.
• Stop all the Skyline services and ensure any service controls such as monit, supervisor or systemd do not start them again, (change as appropriate for your set up)

NEW_SKYLINE_VERSION="v1.2.12"    # Your new Skyline version
OLD_SKYLINE_VERSION="v1.2.11"    # Your old Skyline version

CURRENT_SKYLINE_PATH="/opt/skyline/github/skyline"                 # Your Skyline path
NEW_SKYLINE_PATH="${CURRENT_SKYLINE_PATH}.${NEW_SKYLINE_VERSION}"  # Your new Skyline path

mkdir -p "${CURRENT_SKYLINE_PATH}.${NEW_SKYLINE_VERSION}"
cd "${CURRENT_SKYLINE_PATH}.${NEW_SKYLINE_VERSION}"
git clone https://github.com/earthgecko/skyline .
git checkout "$NEW_SKYLINE_VERSION"

cp "$NEW_SKYLINE_PATH/skyline/settings.py" "$NEW_SKYLINE_PATH/skyline/settings.py.v1.2.12.bak"
cat "${CURRENT_SKYLINE_PATH}/skyline/settings.py" > "$NEW_SKYLINE_PATH/skyline/settings.py"

# ADD the new IONOSPHERE_ECHO to settings
vi "$NEW_SKYLINE_PATH/skyline/settings.py"

# Stop Skyline DB related services
SKYLINE_SERVICES="ionosphere
luminosity
panorama"
for i in $SKYLINE_SERVICES
do
  /etc/init.d/$i stop
done
# And stop any service controls like monit or systemd

# BACKUP THE DB AND APPLY THE NEW SQL
BACKUP_DIR="/tmp"  # Where you want to backup the DB to
MYSQL_USER="<YOUR_MYSQL_USER>"
MYSQL_HOST="127.0.0.1"  # Your MySQL IP
MYSQL_DB="skyline"  # Your MySQL Skyline DB name

# Backup DB
mkdir -p $BACKUP_DIR
mysqldump -u$MYSQL_USER -p $MYSQL_DB > $BACKUP_DIR/pre.v1.2.12.$MYSQL_DB.sql

# Update DB
mysql -u$MYSQL_USER -p $MYSQL_DB < "${NEW_SKYLINE_PATH}/updates/sql/v1.2.12-stable.sql"

# Stop all other Skyline services
SKYLINE_SERVICES="horizon
analyzer
mirage
crucible
boundary
ionosphere
luminosity
panorama
webapp"
for i in $SKYLINE_SERVICES
do
  /etc/init.d/$i stop
done

• Move your current Skyline directory to a backup directory and move the new Skyline v1.2.12 with your new settings.py from the temp location to your working Skyline directory, (change your paths as appropriate) e.g.

mv "$CURRENT_SKYLINE_PATH" "${CURRENT_SKYLINE_PATH}.${OLD_SKYLINE_VERSION}"
mv "$NEW_SKYLINE_PATH" "$CURRENT_SKYLINE_PATH"

• Start the all Skyline services (change as appropriate for your set up) e.g.

# Start all other Skyline services
SKYLINE_SERVICES="panorama
luminosity
horizon
analyzer
mirage
crucible
boundary
ionosphere
webapp"
for i in $SKYLINE_SERVICES
do
  /etc/init.d/$i start
done

• Check the logs

# How are they running
tail -n 20 /var/log/skyline/*.log

# Any errors - each app
find /var/log/skyline -type f -name "*.log" | while read skyline_logfile
do
  echo "#####
# Checking for errors in $skyline_logfile"
  cat "$skyline_logfile" | grep -B2 -A10 -i "error ::\|traceback" | tail -n 60
  echo ""
  echo ""
done