======
1.3.0
======

v1.3.0 - September 25, 2019

This is a features functionality release with minor bug fixes.  This release
introduces a few new features and new functionality:

- A new Ionosphere feature related to validating matches.
- The ability to run the Skyline stack on docker
- Introduces Skyline Flux for receiving metrics via HTTP and submitting them to
  Graphite.
- Introduces Skyline Vista for fetching metrics from remote sources.

Please see the new documentation pages for full details about each new addition.

Changes in v1.3.0
------------------

Lots of changes have been introduced to accommodate docker and the addition of
Flux and Vista.

- Use GRAPHITE_RENDER_URI and do not try to alert if the settings are default in
  analyzer/alerters.py (3002)
- Wrapped new get_graphite_render_uri and get_graphite_custom_headers functions
  in try and except in analyzer/alerters.py (3002)
- The management of the analyzer Redis algorithm exception sets was previously
  handled a if settings.ENABLE_MIRAGE block.  This meant that if there are no
  mirage metrics, the sets were not managed.  Added manage_analyzer_algorithm_exception_sets
  in analyzer/analyzer.py to manage these sets in their own context. (3002, 3020)
- Re-enabled the illuminance metric in analyzer (2580)
- Declare and test if the derivative_metrics Redis set exists in analyzer so
  that errors are not thrown if there are actually no derivative metrics (3002)
- Use GRAPHITE_RENDER_URI and use new CARBON_HOST variable instead of
  GRAPHITE_HOST in send_graphite_metric (3002)
- Handle self signed certificate on Docker with requests (3002)
- Handle when Skyline is first installed, if luminosity is enabled it reports
  errors as there are no anomaly ids (3002, 3016)
- Log appropriate to whether memcache is enabled or not (3002, 3016)
- Handle self signed certificate on Docker with requests (3002)
- Do not analyse short time series, only return a time series sample if the
  sample has sufficient data points otherwise get_anomalies() will throw an
  error (3002, 3008)
- Added docker settings (3002)
- Allow rebrow to connect to Redis on the socket too and if docker dsplay the
  REDIS_PASSWORD in rebrow_login (3002)
- Use GRAPHITE_RENDER_URI (3002)
- Revert to original skyline.httpd.conf.d.example and added
  skyline.docker.apache2.conf.d.example (3002)
- Reformat print in analyzer/agent.py (3002)
- When there are multiple channels declared Skyline only wants to update the
  panorama.slack_thread_ts Redis key once for the primary Skyline channel, so if
  the cache key exists do not update, analyzer/alerters.py (3110)
- Added a hashlib hexdigest method which has not been verified so using the
  original method useless the base_name is longer than 254 chars,
  analyzer/alerters.py (3002)
- Allow for DOCKER_FAKE_EMAIL_ALERTS and GRAPHITE_CUSTOM_HEADERS,
  analyzer/alerters.py (3002)
- Added get_graphite_port, get_graphite_render_uri and
  get_graphite_custom_headers functions, analyzer/alerters.py (3002)
- Reduce amount of Manager instances that are used as each requires a copy of
  entire memory to be copied into each subprocess so this results in a python
  process per Manager instance, using as much memory as the parent.  OK on a
  server, not so much in a container.  Disabled all the Manager() lists and
  replaced with Redis sets, analyzer/analyzer.py (3002, 3032, 3034)
- Moved the interpolation of the MIRAGE_PERIODIC variables to the top of the
  file out of spawn_process so they can be accessed in run too and check the
  added mirage_periodic_check_keys list rather than checking each and every
  metric name key, analyzer/analyzer.py (2882)
- Disabled for now as in concept phase, analyzer/analyzer.py (2580)
- The management of the analyzer Redis algorithm exception sets was previously
  handled in an if settings.ENABLE_MIRAGE block This meant that if there are no
  mirage metrics, the sets were not managed, analyzer/analyzer.py (3020)
- Changes for docker, not complete or fully tested.
- Reduce amount of Manager instances that are used as each requires a copy of
  entire memory to be copied into each subprocess so this results in a python
  process per Manager instance, using as much memory as the parent.  OK on a
  server, not so much in a container.  Disabled all the Manager() lists and
  replaced with Redis sets, boundary/boundary.py (3002, 3032, 3034)
- Use GRAPHITE_RENDER_URI, boundary/boundary.py (3002)
- Reduce amount of Manager instances that are used as each requires a copy of
  entire memory to be copied into each subprocess so this results in a python
  process per Manager instance, using as much memory as the parent.  OK on a
  server, not so much in a container.  Disabled all the Manager() lists and
  replaced with Redis sets, crucible/crucible.py (3002, 3032)
- Add a crucible - skyline.consensus.anomalies.png to plot Skyline anomalies if
  CONSENSUS is achieved and create file resources skyline.anomalies_score.txt
  and skyline.anomalies.csv, crucible/crucible_algorithms.py (3106)
- Reduce amount of Manager instances that are used as each requires a copy of
  entire memory to be copied into each subprocess so this results in a python
  process per Manager instance, using as much memory as the parent.  OK on a
  server, not so much in a container.  Disabled all the Manager() lists and
  replaced with Redis sets, skyline/ionosphere/ionosphere.py (3002, 3032)
- Do not run checks if namespace has matched multiple times in the last 10
  minutes.  However determining which Skyline related metrics are feeding back
  are quite difficult to ascertain.  So use the ionosphere_busy logic again and
  use or find the skyline host namespace and if busy do not analyse the Skyline
  host namespace while ionosphere is busy related to SKYLINE_FEEDBACK_NAMESPACES,
  skyline/ionosphere/ionosphere.py (3050)
- Only process FULL_DURATION feature profiles if there is a
  ionosphere.unique_metrics Redis set (2484)
- Handle self signed certificate on Docker, skyline/ionosphere/learn.py (3002)
- Reduce amount of Manager instances that are used as each requires a copy of
  entire memory to be copied into each subprocess so this results in a python
  process per Manager instance, using as much memory as the parent.  OK on a
  server, not so much in a container.  Disabled all the Manager() lists and
  replaced with Redis sets, skyline/luminosity/luminosity.py (3002, 3032)
- When Skyline is first installed, if luminosity is enabled it reports errors as
  there are no anomaly ids, skyline/luminosity/luminosity.py (3016)
- Handle self signed certificate on Docker,
  skyline/luminosity/process_correlations.py (3002)
- Reduce amount of Manager instances that are used as each requires a copy of
  entire memory to be copied into each subprocess so this results in a python
  process per Manager instance, using as much memory as the parent.  OK on a
  server, not so much in a container.  Disabled all the Manager() lists and
  replaced with Redis sets, skyline/mirage/mirage.py (3002, 3032)
- When there are multiple channels declared Skyline only wants to update the
  panorama.slack_thread_ts Redis key once for the primary Skyline channel, so if
  the cache key exists do not update, skyline/mirage/mirage_alerters.py (3110)
- Reduce amount of Manager instances that are used as each requires a copy of
  entire memory to be copied into each subprocess so this results in a python
  process per Manager instance, using as much memory as the parent.  OK on a
  server, not so much in a container.  Disabled all the Manager() lists and
  replaced with Redis sets, skyline/mirage/mirage.py (3002, 3032)
- Allow webapp to connect on reborw via socket if running on docker,
  skyline/webapp/templates/rebrow_login.html (3002)
- Handle self signed certificate on Docker, skyline/webapp/ionosphere_backend.py
  (3190)
- Handle if all features profiles have been disabled in the ionosphere_search
  function, skyline/webapp/ionosphere_backend.py (3190)
- Added Ionosphere - validated matches (3082, 3084)
- Added metrics id to relevant web pages, skyline/webapp/webapp.py (2990)
- Added docker related settings to settings.py (3002)
- Added Ionosphere - validated matches DB changes to skyline.sql (3084)
- Added docker changes and some Pyton 3.7 testing changes to
  skyline/skyline_functions.py (3002, 2828)
- Updated version and branch in skyline/skyline_version.py (3002)
- Updated to 1.3.0 - skyline/utils/dawn/skyline.dawn.sh (3002)
- Uppdated dependencies
- skyline.dawn.sh changes related to docker (3002)
- Update version and dependencies
- Added a users tables to the DB to allow to map user to actions, labels, etc
  (3230, 2476, 2516)

Update notes
------------

- These update instruction apply to upgrading from v1.2.17 to v1.3.0 only.
  However as with all Skyline updates it is possible to go through the update
  notes for each version and make your own update notes/process to take you from
  version x to version y.
- There are changes to the DB in v1.3.0

.. warning:: It is assumed that the database is not running with an offset and
  the database increments ids by 1.  If you are running an offset please review
  all SQL updates to determine if you need to change anything in the SQL when
  you apply Skyline SQL updates.

- There are changes to settings.py in v1.3.0, please ensure you diff your
  current and the new settings.py as there are appended additions but there are
  also additional settings that have been added to existing settings blocks.
  The additions and changes described below.

settings.py Changes
~~~~~~~~~~~~~~~~~~~

* New settings:

  - :mod:`settings.GRAPHITE_AUTH_HEADER`
  - :mod:`settings.GRAPHITE_CUSTOM_HEADERS`
  - :mod:`settings.GRAPHITE_RENDER_URI`
  - :mod:`settings.CARBON_HOST`
  - :mod:`settings.SKYLINE_FEEDBACK_NAMESPACES`
  - :mod:`settings.CUSTOM_ALERT_OPTS`
  - :mod:`settings.DOCKER`
  - :mod:`settings.DOCKER_DISPLAY_REDIS_PASSWORD_IN_REBROW`
  - :mod:`settings.DOCKER_FAKE_EMAIL_ALERTS`
  - :mod:`settings.FLUX_IP = '127.0.0.1`
  - :mod:`settings.FLUX_PORT`
  - :mod:`settings.FLUX_WORKERS`
  - :mod:`settings.FLUX_SELF_API_KEY`
  - :mod:`settings.FLUX_BACKLOG`
  - :mod:`settings.FLUX_SEND_TO_CARBON`
  - :mod:`settings.FLUX_CARBON_HOST`
  - :mod:`settings.FLUX_CARBON_PORT`
  - :mod:`settings.FLUX_CARBON_PICKLE_PORT`
  - :mod:`settings.FLUX_SEND_TO_STATSD`
  - :mod:`settings.FLUX_STATSD_HOST`
  - :mod:`settings.FLUX_STATSD_PORT`
  - :mod:`settings.VISTA_ENABLED`
  - :mod:`settings.VISTA_FETCHER_PROCESSES`
  - :mod:`settings.VISTA_FETCHER_PROCESS_MAX_RUNTIME`
  - :mod:`settings.VISTA_WORKER_PROCESSES`
  - :mod:`settings.VISTA_DO_NOT_SUBMIT_CURRENT_MINUTE`
  - :mod:`settings.VISTA_FETCH_GRAPHITE_METRICS`

* New Graphite related settings have been added to the main settings block,
  :mod:`settings.GRAPHITE_AUTH_HEADER` and
  :mod:`settings.GRAPHITE_CUSTOM_HEADERS` neither are enabled by default.
  Additionally a :mod:`settings.GRAPHITE_RENDER_URI` setting has been added that
  is now used in the existing and modified :mod:`settings.GRAPH_URL` setting.
* :mod:`settings.CARBON_HOST` setting has been added that defaults to
  :mod:`settings.GRAPHITE_HOST` to enable the use of a different CARBON_HOST.
* A new settings :mod:`settings.SKYLINE_FEEDBACK_NAMESPACES` has been added to
  to reduce analysis load during busy times, read the docstring for a full
  description.
* A new setting :mod:`settings.CUSTOM_ALERT_OPTS` allows to add custom labels,
  graph titles and subjects to alerts.
* A new Docker settings block has been added with all the Docker related
  settings, the defaults these settings are for Docker not being enabled.
* A new Flux settings block has been added for all the Flux settings
* A new Vista settings block has been added for all the Vista settings.

How to update from v1.2.17
--------------------------

- Download the new release tag or clone/update to get it to a temp location,
  ready to be deployed.

.. code-block:: bash

    NEW_SKYLINE_VERSION="v1.3.0"    # Your new Skyline version
    OLD_SKYLINE_VERSION="v1.2.17"    # Your old Skyline version

    CURRENT_SKYLINE_PATH="/opt/skyline/github/skyline"                 # Your Skyline path
    NEW_SKYLINE_PATH="${CURRENT_SKYLINE_PATH}.${NEW_SKYLINE_VERSION}"  # Your new Skyline path

    mkdir -p "${CURRENT_SKYLINE_PATH}.${NEW_SKYLINE_VERSION}"
    cd "${CURRENT_SKYLINE_PATH}.${NEW_SKYLINE_VERSION}"
    git clone https://github.com/earthgecko/skyline .
    git checkout "$NEW_SKYLINE_VERSION"

    cp "$NEW_SKYLINE_PATH/skyline/settings.py" "$NEW_SKYLINE_PATH/skyline/settings.py.${NEW_SKYLINE_VERSION}.bak"

    # DIFF your settings.py with the new settings.py
    # diff "${CURRENT_SKYLINE_PATH}/skyline/settings.py" "$NEW_SKYLINE_PATH/skyline/settings.py.${NEW_SKYLINE_VERSION}.bak"

    #
    cat "${CURRENT_SKYLINE_PATH}/skyline/settings.py" > "$NEW_SKYLINE_PATH/skyline/settings.py"

    # ADD the appropriate new settings to your settings file.
    vi "$NEW_SKYLINE_PATH/skyline/settings.py"

    # Stop Skyline DB related services
    SKYLINE_SERVICES="webapp
    ionosphere
    luminosity
    panorama"
    for i in $SKYLINE_SERVICES
    do
      /etc/init.d/$i stop
    done
    # And stop any service controls like monit or systemd

    # BACKUP THE DB AND APPLY THE NEW SQL
    BACKUP_DIR="/tmp"  # Where you want to backup the DB to
    MYSQL_USER="<YOUR_MYSQL_USER>"
    MYSQL_HOST="127.0.0.1"  # Your MySQL IP
    MYSQL_DB="skyline"  # Your MySQL Skyline DB name

    # Backup DB
    mkdir -p $BACKUP_DIR
    mysqldump -u$MYSQL_USER -p $MYSQL_DB > $BACKUP_DIR/pre.$NEW_SKYLINE_VERSION.$MYSQL_DB.sql

    # Update DB
    mysql -u$MYSQL_USER -p $MYSQL_DB < "${NEW_SKYLINE_PATH}/updates/sql/${NEW_SKYLINE_VERSION}.sql"

    # Stop all other Skyline services
    SKYLINE_SERVICES="horizon
    analyzer
    mirage
    crucible
    boundary
    ionosphere
    luminosity
    panorama
    webapp"
    for i in $SKYLINE_SERVICES
    do
      /etc/init.d/$i stop
    done

- Move your current Skyline directory to a backup directory and move the new
  Skyline v1.3.0 with your new settings.py from the temp location to your
  working Skyline directory, (change your paths as appropriate) e.g.

.. code-block:: bash

    mv "$CURRENT_SKYLINE_PATH" "${CURRENT_SKYLINE_PATH}.${OLD_SKYLINE_VERSION}"
    mv "$NEW_SKYLINE_PATH" "$CURRENT_SKYLINE_PATH"

- Update the dependencies

.. code-block:: bash

    PYTHON_MAJOR_VERSION="2.7"
    PYTHON_VIRTUALENV_DIR="/opt/python_virtualenv"
    # Handle 2.7.14 and 2.7.16
    PROJECT="skyline-py2716"
    if [ ! -d "${PYTHON_VIRTUALENV_DIR}/projects/${PROJECT}" ]; then
      PROJECT="skyline-py2714"
    fi
    cd "${PYTHON_VIRTUALENV_DIR}/projects/${PROJECT}"
    source bin/activate
    # This can take lots of minutes with the new pandas
    bin/"pip${PYTHON_MAJOR_VERSION}" install $(cat "${CURRENT_SKYLINE_PATH}/requirements.txt" | grep "^numpy\|^scipy\|^patsy\|^pandas" | tr '\n' ' ')
    # Remove the luminol egg as it will already be installed and pip will moan
    cat "${CURRENT_SKYLINE_PATH}/requirements.txt" | grep -v "luminol" > /tmp/requirements.txt

    # IF YOU ARE ON CENTOS 6 and running mainstream mysql-server-5.1 uncomment
    # the following line and run it to fix to mysql-python-connector 8.0.6
    #cat "${CURRENT_SKYLINE_PATH}/requirements.txt" | grep -v "luminol\|mysql-connector-python" > /tmp/requirements.txt

    bin/"pip${PYTHON_MAJOR_VERSION}" install -r /tmp/requirements.txt

- Start the all Skyline services (change as appropriate for your set up) e.g.

.. code-block:: bash

    # Start all other Skyline services
    SKYLINE_SERVICES="panorama
    luminosity
    horizon
    analyzer
    mirage
    crucible
    boundary
    ionosphere
    webapp"
    for i in $SKYLINE_SERVICES
    do
      /etc/init.d/$i start
    done

- Check the logs

.. code-block:: bash

    # How are they running
    tail -n 20 /var/log/skyline/*.log

    # Any errors - each app
    find /var/log/skyline -type f -name "*.log" | while read skyline_logfile
    do
      echo "#####
    # Checking for errors in $skyline_logfile"
      cat "$skyline_logfile" | grep -B2 -A10 -i "error ::\|traceback" | tail -n 60
      echo ""
      echo ""
    done