1.2.14

v1.2.14 - April 15, 2019

This is release includes:

  • Dependencies updates
  • Code changes for redis-py package update
  • mysql-connector-python notes
  • Minor bug fixes
  • Updates on Python and Redis versions in docs and the test build script

A note on version changes

The documentation has been updated to use Python 2.7.16 and Redis 4.0.14, these changes do not require any immediate updates of Skyline installations running Python 2.7.14 or an earlier version of Redis, v1.2.14 will still work on Python-2.7.14 and Redis 3.x.

Changes in v1.2.14

  • Update dependencies (2926)
  • Note requirement of old version of mysql-connector-python if mainstream MySQL 5.1 is used (2590)
  • Updated skyline.conf USE_PYTHON to py2716 (2926)
  • Update docs to Python-2.7.16 and redis-4.0.14 (2926)
  • Corrected adding the metric list to the self.real_anomalous_metrics list instead of just the basename for Mirage periodic checks (2932, 2882)
  • calc_value in get_calculated_features sometimes generates an error so wrapped in try with logging relating to Ionosphere echo (2484) to determine what value is causing: ValueError: could not convert string to float
  • redis-py 3.x only accepts user data as bytes, strings or numbers (ints, longs and floats). All 2.X users should make sure that the keys and values they pass into redis-py are either bytes, strings or numbers. Changed all key values that were passed as a list to a str, they are all evaluted upstream with literal_eval already (2824, 2926)
  • calc_value in get_calculated_features is sometimes empty for autocorrelation_lag values so disabled traceback and log as info not error in the Ionosphere echo context (2484)
  • Updated roadmap doc regarding NAB scoreboard update (1884, 2802)
  • Any features profiles that were created before the introduction of (1960) ionosphere_layers will not have a mirage.redis.24h.json file as the creation of these resources in the training_data dir was only added at that point, so they are excluded. Less than 20170307 1488844800 excl. (2934)
  • Only process metric with echo if it is an ionosphere enabled metric (2484)
  • Corrected adding the metric list to the self.real_anomalous_metrics list instead of just the basename (2932, 2882)

Update notes

  • These update instruction apply to upgrading from v1.2.13 to v1.2.14 only. However as with all Skyline updates it is possible to go through the update notes for each version and make your own update notes/process to take you from version x to version y.
  • There are no changes to the DB in v1.2.14
  • There are no changes to settings.py in v1.2.14

How to update from v1.2.13

  • Download the new release tag or clone/update to get it to a temp location, ready to be deployed.
  • Stop all the Skyline services and ensure any service controls such as monit, supervisor or systemd do not start them again, (change as appropriate for your set up)
NEW_SKYLINE_VERSION="v1.2.14"    # Your new Skyline version
OLD_SKYLINE_VERSION="v1.2.13"    # Your old Skyline version

CURRENT_SKYLINE_PATH="/opt/skyline/github/skyline"                 # Your Skyline path
NEW_SKYLINE_PATH="${CURRENT_SKYLINE_PATH}.${NEW_SKYLINE_VERSION}"  # Your new Skyline path

mkdir -p "${CURRENT_SKYLINE_PATH}.${NEW_SKYLINE_VERSION}"
cd "${CURRENT_SKYLINE_PATH}.${NEW_SKYLINE_VERSION}"
git clone https://github.com/earthgecko/skyline .
git checkout "$NEW_SKYLINE_VERSION"

cp "$NEW_SKYLINE_PATH/skyline/settings.py" "$NEW_SKYLINE_PATH/skyline/settings.py.$OLD_SKYLINE_VERSION.bak"
cat "${CURRENT_SKYLINE_PATH}/skyline/settings.py" > "$NEW_SKYLINE_PATH/skyline/settings.py"
  • Update the dependencies in you Python virtualenv (this takes a long time). The redis-py update is left till the very end otherwise Skyline will start to error.
  • Update the pip dependencies PLEASE USE THE APPROPRIATE Python verison for your install, e.g.

Warning

If you are running on CentOS 6 and are running mainstream mysql-server-5.1.x, in the example below take note of the line you need to uncomment as you need to stay on mysql-python-connector==8.0.6 and NOT upgrade to mysql-python-connector==8.0.15

PYTHON_MAJOR_VERSION="2.7"
PYTHON_VIRTUALENV_DIR="/opt/python_virtualenv"
# Handle 2.7.14 and 2.7.16
PROJECT="skyline-py2716"
if [ ! -d "${PYTHON_VIRTUALENV_DIR}/projects/${PROJECT}" ]; then
  PROJECT="skyline-py2714"
fi
cd "${PYTHON_VIRTUALENV_DIR}/projects/${PROJECT}"
source bin/activate
# This can take lots of minutes with the new pandas
bin/"pip${PYTHON_MAJOR_VERSION}" install $(cat "${CURRENT_SKYLINE_PATH}.${NEW_SKYLINE_VERSION}/requirements.txt" | grep "^numpy\|^scipy\|^patsy\|^pandas" | tr '\n' ' ')
# Remove the luminol egg as it will already be installed and pip will moan
# Remove Redis to be updated last
cat "${CURRENT_SKYLINE_PATH}.${NEW_SKYLINE_VERSION}/requirements.txt" | grep -v "luminol\|redis" > /tmp/requirements.txt

# IF YOU ARE ON CENTOS 6 and running mainstream mysql-server-5.1 uncomment
# the following line and run it to fix to mysql-python-connector 8.0.6
#cat "${CURRENT_SKYLINE_PATH}.${NEW_SKYLINE_VERSION}/requirements.txt" | grep -v "luminol\|redis\|mysql-connector-python" > /tmp/requirements.txt

bin/"pip${PYTHON_MAJOR_VERSION}" install -r /tmp/requirements.txt

# Stop all other Skyline services and any sevice controls you may have
SKYLINE_SERVICES="horizon
analyzer
mirage
crucible
boundary
ionosphere
luminosity
panorama
webapp"
for i in $SKYLINE_SERVICES
do
  /etc/init.d/$i stop
done

# Update redis-py
cat "${CURRENT_SKYLINE_PATH}.${NEW_SKYLINE_VERSION}/requirements.txt" | grep "redis" > /tmp/redis.requirements.txt
bin/"pip${PYTHON_MAJOR_VERSION}" install -r /tmp/redis.requirements.txt
deactivate
  • Move your current Skyline directory to a backup directory and move the new Skyline v1.2.14 with your new settings.py from the temp location to your working Skyline directory, (change your paths as appropriate) e.g.
mv "$CURRENT_SKYLINE_PATH" "${CURRENT_SKYLINE_PATH}.${OLD_SKYLINE_VERSION}"
mv "$NEW_SKYLINE_PATH" "$CURRENT_SKYLINE_PATH"
  • Start the all Skyline services (change as appropriate for your set up) e.g.
# Start all other Skyline services
SKYLINE_SERVICES="panorama
luminosity
horizon
analyzer
mirage
crucible
boundary
ionosphere
webapp"
for i in $SKYLINE_SERVICES
do
  /etc/init.d/$i start
done
  • Check the logs
# How are they running
tail -n 20 /var/log/skyline/*.log

# Any errors - each app
find /var/log/skyline -type f -name "*.log" | while read skyline_logfile
do
  echo "#####
# Checking for errors in $skyline_logfile"
  cat "$skyline_logfile" | grep -B2 -A10 -i "error ::\|traceback" | tail -n 60
  echo ""
  echo ""
done