====== 1.2.12 ====== v1.2.12 - April 3, 2019 Stable for new installs. Beta if upgrading. DOWNGRADED to a pre-release as an update ---------------------------------------- Downgrade to a pre-release - April 4, 2019 To add a mechanism that reduces to initial load on Ionosphere when Ionosphere echo is enabled if there are a lot of existing Mirage feature profiles and to add some self load management to Ionosphere in general under the new Ionosphere echo paradigm. New functionality ----------------- - Adds the new Ionosphere echo functionality related to improving the matching of Mirage based features profiles by between 30-50% depending on the metric. See https://earthgecko-skyline.readthedocs.io/en/stable/ionosphere_echo.html Fixes ----- - Adds the ability for Skyline Boundary to handle very short time series. - Updated tsfresh-0.4.0.py2.data.json.features.transposed.csv with new longer decimal values introduced by numpy==1.15.1 pandas==0.24.3 Changes from v1.2.11 -------------------- - Allow Boundary to analyse short time series (#88, #2862) - Bifurcate dependencies for py3 testing (#2588, #2828) - Added functionality and changes to implement Ionosphere echo (#2484) - Updated tsfresh-0.4.0.py2.data.json.features.transposed.csv with new longer decimal values introduced by numpy==1.15.1 pandas==0.24.3 (2902) Update notes ------------ - You can only upgrade to v1.2.12 from v1.2.11 - There are changes to the DB - There are some changes to settings.py How to update from v1.2.11 -------------------------- - Download the new release tag or clone/update to get it to a temp location, ready to be deployed. - Diff your current running version skyline/settings.py with the new version of skyline/settings.py and create a new settings.py in the temp location appropriate for Skyline v1.2.12 and your set up with the new IONOSPHERE_ECHO settings, ready to be deployed. New settings to be added are: .. code-block:: python IONOSPHERE_MAX_RUNTIME = 120 """ :var IONOSPHERE_MAX_RUNTIME: The maximum number of seconds an Ionosphere check should run for. :vartype IONOSPHERE_MAX_RUNTIME: int """ IONOSPHERE_ECHO_ENABLED = True """ :var IONOSPHERE_ECHO_ENABLED: This enables Ionosphere to create and test features profiles for Mirage metrics but at settings.FULL_DURATION as well. Features profiles will be made on the fly for any existing, validated Mirage metric features profiles. Ionosphere's matching performance is increased between 30 to 50 percent when Ionosphere echo is run. :vartype IONOSPHERE_ECHO_ENABLED: boolean """ IONOSPHERE_ECHO_MAX_FP_CREATE_TIME = 55 """ :var IONOSPHERE_ECHO_MAX_FP_CREATE_TIME: The maximum number of seconds an Ionosphere echo process should run creating FULL_DURATION features profiles for created Mirage features profiles. This setting is specifically relevant for Skyline implematations pre Ionosphere echo (v1.2.12) to prevent timeouts if Ionosphere echo needs to make > 30 echo features profiles for Mirage metrics with lots of existing features profiles. :vartype IONOSPHERE_ECHO_MAX_FP_CREATE_TIME: int """ IONOSPHERE_ECHO_FEATURES_PERCENT_SIMILAR = 2.5 """ :var IONOSPHERE_ECHO_FEATURES_PERCENT_SIMILAR: In terms of Ionosphere echo a value of 2.0 is the default. This default is above the normal IONOSPHERE_FEATURES_PERCENT_SIMILAR due to that fact that the the resolution of Ionosphere echo is at FULL_DURATION. During testing this value was tested at 1.0, 2 and 2.5, with 2.5 resulting in the most desirable results in terms of matching time series that are similarly not anomalous. :vartype IONOSPHERE_ECHO_FEATURES_PERCENT_SIMILAR: float """ IONOSPHERE_ECHO_MINMAX_SCALING_FEATURES_PERCENT_SIMILAR = 3.5 """ :var IONOSPHERE_ECHO_MINMAX_SCALING_FEATURES_PERCENT_SIMILAR: In terms of Ionosphere echo Min-Max scaling percentage similar, a value of 3.5 is the default. This default is above the normal IONOSPHERE_FEATURES_PERCENT_SIMILAR due to that fact that the the resolution of Ionosphere echo is at FULL_DURATION and echo is using the normal IONOSPHERE_MINMAX_SCALING_RANGE_TOLERANCE to determine if Min-Max scaling should be run. During testing this value was tested at 1, 2 and 3.5, with 3.5 resulting in the most desirable results in terms of matching time series that are similarly not anomalous. :vartype IONOSPHERE_ECHO_MINMAX_SCALING_FEATURES_PERCENT_SIMILAR: float """ - Stop Skyline Ionosphere and Panorama services. - Backup your Skyline database. - Apply the SQL update to your Skyline database as per described below, change the below variables as appropriate to your set up. - Stop all the Skyline services and ensure any service controls such as monit, supervisor or systemd do not start them again, (change as appropriate for your set up) .. code-block:: bash NEW_SKYLINE_VERSION="v1.2.12" # Your new Skyline version OLD_SKYLINE_VERSION="v1.2.11" # Your old Skyline version CURRENT_SKYLINE_PATH="/opt/skyline/github/skyline" # Your Skyline path NEW_SKYLINE_PATH="${CURRENT_SKYLINE_PATH}.${NEW_SKYLINE_VERSION}" # Your new Skyline path mkdir -p "${CURRENT_SKYLINE_PATH}.${NEW_SKYLINE_VERSION}" cd "${CURRENT_SKYLINE_PATH}.${NEW_SKYLINE_VERSION}" git clone https://github.com/earthgecko/skyline . git checkout "$NEW_SKYLINE_VERSION" cp "$NEW_SKYLINE_PATH/skyline/settings.py" "$NEW_SKYLINE_PATH/skyline/settings.py.v1.2.12.bak" cat "${CURRENT_SKYLINE_PATH}/skyline/settings.py" > "$NEW_SKYLINE_PATH/skyline/settings.py" # ADD the new IONOSPHERE_ECHO to settings vi "$NEW_SKYLINE_PATH/skyline/settings.py" # Stop Skyline DB related services SKYLINE_SERVICES="ionosphere luminosity panorama" for i in $SKYLINE_SERVICES do /etc/init.d/$i stop done # And stop any service controls like monit or systemd # BACKUP THE DB AND APPLY THE NEW SQL BACKUP_DIR="/tmp" # Where you want to backup the DB to MYSQL_USER="" MYSQL_HOST="127.0.0.1" # Your MySQL IP MYSQL_DB="skyline" # Your MySQL Skyline DB name # Backup DB mkdir -p $BACKUP_DIR mysqldump -u$MYSQL_USER -p $MYSQL_DB > $BACKUP_DIR/pre.v1.2.12.$MYSQL_DB.sql # Update DB mysql -u$MYSQL_USER -p $MYSQL_DB < "${NEW_SKYLINE_PATH}/updates/sql/v1.2.12-stable.sql" # Stop all other Skyline services SKYLINE_SERVICES="horizon analyzer mirage crucible boundary ionosphere luminosity panorama webapp" for i in $SKYLINE_SERVICES do /etc/init.d/$i stop done - Move your current Skyline directory to a backup directory and move the new Skyline v1.2.12 with your new settings.py from the temp location to your working Skyline directory, (change your paths as appropriate) e.g. .. code-block:: bash mv "$CURRENT_SKYLINE_PATH" "${CURRENT_SKYLINE_PATH}.${OLD_SKYLINE_VERSION}" mv "$NEW_SKYLINE_PATH" "$CURRENT_SKYLINE_PATH" - Start the all Skyline services (change as appropriate for your set up) e.g. .. code-block:: bash # Start all other Skyline services SKYLINE_SERVICES="panorama luminosity horizon analyzer mirage crucible boundary ionosphere webapp" for i in $SKYLINE_SERVICES do /etc/init.d/$i start done - Check the logs .. code-block:: bash # How are they running tail -n 20 /var/log/skyline/*.log # Any errors - each app find /var/log/skyline -type f -name "*.log" | while read skyline_logfile do echo "##### # Checking for errors in $skyline_logfile" cat "$skyline_logfile" | grep -B2 -A10 -i "error ::\|traceback" | tail -n 60 echo "" echo "" done