====== 1.2.14 ====== v1.2.14 - April 15, 2019 This is release includes: - Dependencies updates - Code changes for redis-py package update - mysql-connector-python notes - Minor bug fixes - Updates on Python and Redis versions in docs and the test build script A note on version changes ------------------------- The documentation has been updated to use Python 2.7.16 and Redis 4.0.14, these changes do not require any immediate updates of Skyline installations running Python 2.7.14 or an earlier version of Redis, v1.2.14 will still work on Python-2.7.14 and Redis 3.x. Changes in v1.2.14 ------------------ - Update dependencies (2926) - Note requirement of old version of mysql-connector-python if mainstream MySQL 5.1 is used (2590) - Updated skyline.conf USE_PYTHON to py2716 (2926) - Update docs to Python-2.7.16 and redis-4.0.14 (2926) - Corrected adding the metric list to the self.real_anomalous_metrics list instead of just the basename for Mirage periodic checks (2932, 2882) - calc_value in get_calculated_features sometimes generates an error so wrapped in try with logging relating to Ionosphere echo (2484) to determine what value is causing: `ValueError: could not convert string to float` - redis-py 3.x only accepts user data as bytes, strings or numbers (ints, longs and floats). All 2.X users should make sure that the keys and values they pass into redis-py are either bytes, strings or numbers. Changed all key values that were passed as a list to a str, they are all evaluted upstream with literal_eval already (2824, 2926) - calc_value in get_calculated_features is sometimes empty for autocorrelation_lag values so disabled traceback and log as info not error in the Ionosphere echo context (2484) - Updated roadmap doc regarding NAB scoreboard update (1884, 2802) - Any features profiles that were created before the introduction of (1960) ionosphere_layers will not have a mirage.redis.24h.json file as the creation of these resources in the training_data dir was only added at that point, so they are excluded. Less than 20170307 1488844800 excl. (2934) - Only process metric with echo if it is an ionosphere enabled metric (2484) - Corrected adding the metric list to the self.real_anomalous_metrics list instead of just the basename (2932, 2882) Update notes ------------ - These update instruction apply to upgrading from v1.2.13 to v1.2.14 only. However as with all Skyline updates it is possible to go through the update notes for each version and make your own update notes/process to take you from version x to version y. - There are no changes to the DB in v1.2.14 - There are no changes to settings.py in v1.2.14 How to update from v1.2.13 -------------------------- - Download the new release tag or clone/update to get it to a temp location, ready to be deployed. - Stop all the Skyline services and ensure any service controls such as monit, supervisor or systemd do not start them again, (change as appropriate for your set up) .. code-block:: bash NEW_SKYLINE_VERSION="v1.2.14" # Your new Skyline version OLD_SKYLINE_VERSION="v1.2.13" # Your old Skyline version CURRENT_SKYLINE_PATH="/opt/skyline/github/skyline" # Your Skyline path NEW_SKYLINE_PATH="${CURRENT_SKYLINE_PATH}.${NEW_SKYLINE_VERSION}" # Your new Skyline path mkdir -p "${CURRENT_SKYLINE_PATH}.${NEW_SKYLINE_VERSION}" cd "${CURRENT_SKYLINE_PATH}.${NEW_SKYLINE_VERSION}" git clone https://github.com/earthgecko/skyline . git checkout "$NEW_SKYLINE_VERSION" cp "$NEW_SKYLINE_PATH/skyline/settings.py" "$NEW_SKYLINE_PATH/skyline/settings.py.$OLD_SKYLINE_VERSION.bak" cat "${CURRENT_SKYLINE_PATH}/skyline/settings.py" > "$NEW_SKYLINE_PATH/skyline/settings.py" - Update the dependencies in you Python virtualenv (this takes a long time). The redis-py update is left till the very end otherwise Skyline will start to error. - Update the pip dependencies PLEASE USE THE APPROPRIATE Python verison for your install, e.g. .. warning:: If you are running on CentOS 6 and are running mainstream mysql-server-5.1.x, in the example below take note of the line you need to uncomment as you need to stay on mysql-python-connector==8.0.6 and NOT upgrade to mysql-python-connector==8.0.15 .. code-block:: bash PYTHON_MAJOR_VERSION="2.7" PYTHON_VIRTUALENV_DIR="/opt/python_virtualenv" # Handle 2.7.14 and 2.7.16 PROJECT="skyline-py2716" if [ ! -d "${PYTHON_VIRTUALENV_DIR}/projects/${PROJECT}" ]; then PROJECT="skyline-py2714" fi cd "${PYTHON_VIRTUALENV_DIR}/projects/${PROJECT}" source bin/activate # This can take lots of minutes with the new pandas bin/"pip${PYTHON_MAJOR_VERSION}" install $(cat "${CURRENT_SKYLINE_PATH}.${NEW_SKYLINE_VERSION}/requirements.txt" | grep "^numpy\|^scipy\|^patsy\|^pandas" | tr '\n' ' ') # Remove the luminol egg as it will already be installed and pip will moan # Remove Redis to be updated last cat "${CURRENT_SKYLINE_PATH}.${NEW_SKYLINE_VERSION}/requirements.txt" | grep -v "luminol\|redis" > /tmp/requirements.txt # IF YOU ARE ON CENTOS 6 and running mainstream mysql-server-5.1 uncomment # the following line and run it to fix to mysql-python-connector 8.0.6 #cat "${CURRENT_SKYLINE_PATH}.${NEW_SKYLINE_VERSION}/requirements.txt" | grep -v "luminol\|redis\|mysql-connector-python" > /tmp/requirements.txt bin/"pip${PYTHON_MAJOR_VERSION}" install -r /tmp/requirements.txt # Stop all other Skyline services and any sevice controls you may have SKYLINE_SERVICES="horizon analyzer mirage crucible boundary ionosphere luminosity panorama webapp" for i in $SKYLINE_SERVICES do /etc/init.d/$i stop done # Update redis-py cat "${CURRENT_SKYLINE_PATH}.${NEW_SKYLINE_VERSION}/requirements.txt" | grep "redis" > /tmp/redis.requirements.txt bin/"pip${PYTHON_MAJOR_VERSION}" install -r /tmp/redis.requirements.txt deactivate - Move your current Skyline directory to a backup directory and move the new Skyline v1.2.14 with your new settings.py from the temp location to your working Skyline directory, (change your paths as appropriate) e.g. .. code-block:: bash mv "$CURRENT_SKYLINE_PATH" "${CURRENT_SKYLINE_PATH}.${OLD_SKYLINE_VERSION}" mv "$NEW_SKYLINE_PATH" "$CURRENT_SKYLINE_PATH" - Start the all Skyline services (change as appropriate for your set up) e.g. .. code-block:: bash # Start all other Skyline services SKYLINE_SERVICES="panorama luminosity horizon analyzer mirage crucible boundary ionosphere webapp" for i in $SKYLINE_SERVICES do /etc/init.d/$i start done - Check the logs .. code-block:: bash # How are they running tail -n 20 /var/log/skyline/*.log # Any errors - each app find /var/log/skyline -type f -name "*.log" | while read skyline_logfile do echo "##### # Checking for errors in $skyline_logfile" cat "$skyline_logfile" | grep -B2 -A10 -i "error ::\|traceback" | tail -n 60 echo "" echo "" done