1.2.16

v1.2.16 - May 4, 2019

This is a features and dependencies update release.

Changes in v1.2.16

  • Dependencies updated (2964)
  • Report defaulted settings to log. Added all the globally declared settings to enable reporting in the log the state of each setting (2950)
  • Added Skyline slack thread updates on alerts (2646)

Update notes

  • These update instruction apply to upgrading from v1.2.15 to v1.2.16 only. However as with all Skyline updates it is possible to go through the update notes for each version and make your own update notes/process to take you from version x to version y.
  • There are changes to the DB in v1.2.16
  • There are changes to settings.py in v1.2.16

How to update from v1.2.15

  • Download the new release tag or clone/update to get it to a temp location, ready to be deployed.
  • Diff your current running version skyline/settings.py with the new version of skyline/settings.py and create a new settings.py in the temp location appropriate for Skyline v1.2.16 and your instance with the new settings.SLACK_OPTS dictionary options, ready to be deployed. The new options that have been added to the SLACK_OPTS dictionary are shown below. Replace the YOUR_ variables with your channel and channel_id strings. You can set the message_on options for each event type. If you do not want Skyline to update slack threads, set thread_updates: False:
# Your default slack Skyline channel name e.g. '#skyline'
'default_channel': 'YOUR_default_slack_channel',
# Your default slack Skyline channel id e.g. 'C0XXXXXX'
'default_channel_id': 'YOUR_default_slack_channel_id',
# Whether to update slack message threads on any of the below events
'thread_updates': True,
# You can disable or enable each message_on event
'message_on_training_data_viewed': True,
'message_on_training_data_viewed_reaction_emoji': 'eyes',
'message_on_features_profile_created': True,
'message_on_features_profile_created_reaction_emoji': 'thumbsup',
'message_on_features_profile_learnt': True,
'message_on_features_profile_learnt_reaction_emoji': 'heavy_check_mark',
'message_on_features_profile_disabled': True,
'message_on_features_profile_disabled_reaction_emoji': 'x',
'message_on_validated_features_profiles': True,
  • Stop Skyline Ionosphere and Panorama services.
  • Backup your Skyline database.
  • Apply the SQL update to your Skyline database as per described below, change the below variables as appropriate to your set up.
  • Stop all the Skyline services and ensure any service controls such as monit, supervisor or systemd do not start them again, (change as appropriate for your set up)
NEW_SKYLINE_VERSION="v1.2.16"    # Your new Skyline version
OLD_SKYLINE_VERSION="v1.2.15"    # Your old Skyline version

CURRENT_SKYLINE_PATH="/opt/skyline/github/skyline"                 # Your Skyline path
NEW_SKYLINE_PATH="${CURRENT_SKYLINE_PATH}.${NEW_SKYLINE_VERSION}"  # Your new Skyline path

mkdir -p "${CURRENT_SKYLINE_PATH}.${NEW_SKYLINE_VERSION}"
cd "${CURRENT_SKYLINE_PATH}.${NEW_SKYLINE_VERSION}"
git clone https://github.com/earthgecko/skyline .
git checkout "$NEW_SKYLINE_VERSION"

cp "$NEW_SKYLINE_PATH/skyline/settings.py" "$NEW_SKYLINE_PATH/skyline/settings.py.${NEW_SKYLINE_VERSION}.bak"
cat "${CURRENT_SKYLINE_PATH}/skyline/settings.py" > "$NEW_SKYLINE_PATH/skyline/settings.py"

# ADD the new SLACK_OPTS to settings
vi "$NEW_SKYLINE_PATH/skyline/settings.py"

# Stop Skyline DB related services
SKYLINE_SERVICES="ionosphere
luminosity
panorama"
for i in $SKYLINE_SERVICES
do
  /etc/init.d/$i stop
done
# And stop any service controls like monit or systemd

# BACKUP THE DB AND APPLY THE NEW SQL
BACKUP_DIR="/tmp"  # Where you want to backup the DB to
MYSQL_USER="<YOUR_MYSQL_USER>"
MYSQL_HOST="127.0.0.1"  # Your MySQL IP
MYSQL_DB="skyline"  # Your MySQL Skyline DB name

# Backup DB
mkdir -p $BACKUP_DIR
mysqldump -u$MYSQL_USER -p $MYSQL_DB > $BACKUP_DIR/pre.$NEW_SKYLINE_VERSION.$MYSQL_DB.sql

# Update DB
mysql -u$MYSQL_USER -p $MYSQL_DB < "${NEW_SKYLINE_PATH}/updates/sql/v1.2.16-stable.sql"

# Stop all other Skyline services
SKYLINE_SERVICES="horizon
analyzer
mirage
crucible
boundary
ionosphere
luminosity
panorama
webapp"
for i in $SKYLINE_SERVICES
do
  /etc/init.d/$i stop
done
  • Move your current Skyline directory to a backup directory and move the new Skyline v1.2.16 with your new settings.py from the temp location to your working Skyline directory, (change your paths as appropriate) e.g.
mv "$CURRENT_SKYLINE_PATH" "${CURRENT_SKYLINE_PATH}.${OLD_SKYLINE_VERSION}"
mv "$NEW_SKYLINE_PATH" "$CURRENT_SKYLINE_PATH"
  • Update the dependencies
PYTHON_MAJOR_VERSION="2.7"
PYTHON_VIRTUALENV_DIR="/opt/python_virtualenv"
# Handle 2.7.14 and 2.7.16
PROJECT="skyline-py2716"
if [ ! -d "${PYTHON_VIRTUALENV_DIR}/projects/${PROJECT}" ]; then
  PROJECT="skyline-py2714"
fi
cd "${PYTHON_VIRTUALENV_DIR}/projects/${PROJECT}"
source bin/activate
# This can take lots of minutes with the new pandas
bin/"pip${PYTHON_MAJOR_VERSION}" install $(cat "${CURRENT_SKYLINE_PATH}/requirements.txt" | grep "^numpy\|^scipy\|^patsy\|^pandas" | tr '\n' ' ')
# Remove the luminol egg as it will already be installed and pip will moan
cat "${CURRENT_SKYLINE_PATH}/requirements.txt" | grep -v "luminol" > /tmp/requirements.txt

# IF YOU ARE ON CENTOS 6 and running mainstream mysql-server-5.1 uncomment
# the following line and run it to fix to mysql-python-connector 8.0.6
#cat "${CURRENT_SKYLINE_PATH}/requirements.txt" | grep -v "luminol\|mysql-connector-python" > /tmp/requirements.txt

# With v1.2.16 a urllib version of 1.25.2 was available but it does not work
# with requests
cat /tmp/requirements.txt | sed -e 's/urllib3.*/urllib3==1.24.3/g' > /tmp/requirements.2.1.16.txt
cat /tmp/requirements.2.1.16.txt > /tmp/requirements.txt

bin/"pip${PYTHON_MAJOR_VERSION}" install -r /tmp/requirements.txt
  • Start the all Skyline services (change as appropriate for your set up) e.g.
# Start all other Skyline services
SKYLINE_SERVICES="panorama
luminosity
horizon
analyzer
mirage
crucible
boundary
ionosphere
webapp"
for i in $SKYLINE_SERVICES
do
  /etc/init.d/$i start
done
  • Check the logs
# How are they running
tail -n 20 /var/log/skyline/*.log

# Any errors - each app
find /var/log/skyline -type f -name "*.log" | while read skyline_logfile
do
  echo "#####
# Checking for errors in $skyline_logfile"
  cat "$skyline_logfile" | grep -B2 -A10 -i "error ::\|traceback" | tail -n 60
  echo ""
  echo ""
done