1.2.8 - the luminosity and dawn branches

v1.2.8-luminosity - October 17, 2018

New functionality

  • This use of approximately close with Ionosphere layers
  • A testing build script for CentOS 6 & 7 and Ubuntu 16.06
  • The ability to validate all learnt features profiles for a metric
  • Slack alerting
  • Updated dependencies

Minor bug fixes and maintenance

  • Updated dependencies
  • A fix to Analyzer Ionosphere alerts
  • A fix to alert graph timestamp parameters order

Changes from v1.2.7

  • Bumped version to v1.2.8
  • Added the skyline.dawn.sh build script for testing and dawn documentation (2548, 2550)
  • Updated dependencies (2588)
  • Updated algorithms adding the rcond=-1 parameter to the np.linalg.lstsq functions (2588)
  • Updated documentation for v1.2.8 and log strings (2598)
  • Fixed Analyzer/Ionosphere alerting. Due to ionosphere_alerts being added to the self.all_anomalous_metrics after Analyzer metrics here we need to ensure that we only alert on the last item for the metric in the list so that the alert is not sent out with any current anomaly data from the current Analyzer run, but with the data from the ionosphere_alerts item (2594)
  • Added a concept test for illuminance/flux to analyzer.py (2580, 1986)
  • Extended the validate_fp function to validate a single fp id or all the unvalidated, enabled features profiles for a metric_id in ionosphere_backend.py (2430)
  • Added value approx_close in ionosphere_backend.py (2558)
  • SQL update for approx_close (2558)
  • Added alert_slack to alerters (2618)
  • Added SLACK_OPTS to settings.py (2618)
  • Corrected the from_timestamp and until_timestamp as they were incorrectly order, however Graphite still rendered the correct graph as it plotted reverse, which is the same. Also using the metric[0] value instead of time() in alerters (2498)

Update notes

  • NOTE: If you are running v1.x you CANNOT upgrade from v1.x directly to v1.2.8
  • You can update directly to v1.2.8 from any version between v1.2.2 to v1.2.7
  • There is a change to the DB and SQL
  • There are changes to settings.py

How to update from v1.2.2 - v1.2.7

  • Download the new release tag or clone/update to get it to a temp location, ready to be deployed.
NEW_SKYLINE_VERSION="v1.2.8-stable-luminosity"    # Your new Skyline version
OLD_SKYLINE_VERSION="v1.2.7-stable-luminosity"    # Your old Skyline version
# OR
# OLD_SKYLINE_VERSION="v1.2.2-stable-luminosity"
# OLD_SKYLINE_VERSION="v1.2.5-stable-luminosity"
# etc

CURRENT_SKYLINE_PATH="/opt/skyline/github/skyline"                 # Your Skyline path
NEW_SKYLINE_PATH="${CURRENT_SKYLINE_PATH}.${NEW_SKYLINE_VERSION}"  # Your new Skyline path

mkdir -p "${CURRENT_SKYLINE_PATH}.${NEW_SKYLINE_VERSION}"
cd "${CURRENT_SKYLINE_PATH}.${NEW_SKYLINE_VERSION}"
git clone https://github.com/earthgecko/skyline .
git checkout "$NEW_SKYLINE_VERSION"
  • Diff your current running version skyline/settings.py with the new version of skyline/settings.py and create a new settings.py in the temp location appropriate for Skyline v1.2.8 and your set up, ready to be deployed.
diff "${CURRENT_SKYLINE_PATH}/skyline/settings.py" "${NEW_SKYLINE_PATH}/skyline/settings.py"
# Create the new settings.py with your variables and the newly required
# variables and save as ${NEW_SKYLINE_PATH}/skyline/settings.py
  • Stop Skyline Ionosphere and Panorama services.
  • Backup your Skyline database.
  • Apply the SQL update to your Skyline database as per described below, change the below variables as appropriate to your set up.
BACKUP_DIR="/tmp"  # Where you want to backup the DB to
MYSQL_USER="<YOUR_MYSQL_USER>"
MYSQL_HOST="127.0.0.1"  # Your MySQL IP
MYSQL_DB="skyline"  # Your MySQL Skyline DB name
SKYLINE_PATH="/opt/skyline/github/skyline"  # Your Skyline path

# Backup DB
mkdir -p $BACKUP_DIR
mysqldump -u$MYSQL_USER -p $MYSQL_DB > $BACKUP_DIR/pre.v1.2.8.$MYSQL_DB.sql

# Update DB
mysql -u$MYSQL_USER -p $MYSQL_DB < $SKYLINE_PATH/updates/sql/luminosity-v1.2.8-stable.sql
  • Stop all the Skyline services and ensure any service controls such as monit, supervisor or systemd do not start them again, (change as appropriate for your set up) e.g.
# Stop all other Skyline services
SKYLINE_SERVICES="horizon
analyzer
mirage
crucible
boundary
ionosphere
luminosity
panorama
webapp"
for i in $SKYLINE_SERVICES
do
  /etc/init.d/$i stop
done
  • Move your current Skyline directory to a backup directory and move the new Skyline v1.2.8 with your new settings.py from the temp location to your working Skyline directory, (change your paths as appropriate) e.g.
mv "$CURRENT_SKYLINE_PATH" "/opt/skyline/github/skyline.$OLD_SKYLINE_VERSION"
mv "$NEW_SKYLINE_PATH" "$CURRENT_SKYLINE_PATH"
  • Update the pip dependencies, e.g.
PYTHON_MAJOR_VERSION="2.7"
PYTHON_VIRTUALENV_DIR="/opt/python_virtualenv"
PROJECT="skyline-py2714"
cd "${PYTHON_VIRTUALENV_DIR}/projects/${PROJECT}"
source bin/activate
# As of statsmodels 0.9.0 numpy, et al need to be installed before
# statsmodels in requirements
# https://github.com/statsmodels/statsmodels/issues/4654
# This can take lots of minutes with the new pandas
bin/"pip${PYTHON_MAJOR_VERSION}" install $(cat /opt/skyline/github/skyline/requirements.txt | grep "^numpy\|^scipy\|^patsy\|^pandas" | tr '\n' ' ')
# Remove the luminol egg as it will already be installed and pip will moan
cat /opt/skyline/github/skyline/requirements.txt | grep -v luminol > /tmp/requirements.txt
bin/"pip${PYTHON_MAJOR_VERSION}" install -r /tmp/requirements.txt
deactivate
  • Start the all Skyline services (change as appropriate for your set up) e.g.
# Start all other Skyline services
SKYLINE_SERVICES="panorama
luminosity
horizon
analyzer
mirage
crucible
boundary
ionosphere
webapp"
for i in $SKYLINE_SERVICES
do
  /etc/init.d/$i start
done