=====
3.0.2
=====

v3.0.2 - May 6, 2022

This is a minor release with minor bug fixes for v3.0.0

Changes
~~~~~~~

- Change error to warning for Graphite timeseries is unavailable (4164: luminosity - cloudbursts)
- Wrapped opentelemetry instrument functions in try. Fail gracefully if breaks
  it breaks (4514: Integrate opentelemetry)
- Handle metrics with url encoded names try fetch with %25 if % fails (4374: webapp - handle url encoded chars)
- If there is only 1 data point in the timeseries determine_data_frequency
  will not return a resolution so set a sensible default as it does not
  matter, it will be boring (4518: settings - LAST_KNOWN_VALUE_NAMESPACES)
- Wrapped aligned_timeseries.append in try (4518)
- Remove all nan from downsampled data because they break histogram_bins
  (3866: MIRAGE_ENABLE_HIGH_RESOLUTION_ANALYSIS)
- If the downsampled data is not anomalous, remove the entry from (3866)
- Prune entries older than 1 day from the trigger_history (3866)
- Change to correct setex method so the keys expire (4536: Handle Redis failure)
- Declare use_old_timestamp_keys before the ignore_submitted_timestamps (4536)
- Clean up tmp timings and count files (3486: analyzer_batch)
- waterfall alerts - handle single value timeseries (3734: waterfall alerts)
- Added functions and API methods and documentation (4530: namespace.analysed_events)
- Change from using skyline_functions correlate_or_relate_with to (4544: Handle EXTERNAL_SETTINGS in correlate_or_relate_with)
- Deprecated correlate_or_relate_with in skyline_functions (4544, 3858: skyline_functions - correlate_or_relate_with)
- Added as functions.metrics.correlate_or_relate_with with the ability to handle
  external settings (4544)
- Added illuminance Redis hashes to track all anomalous events for future R&D (2580: illuminance)
- Correct misspelt missing function in v3.0.0 (4530: namespace.analysed_events)
- Handle cluster data in api metric endpoint (4530)
- Use str for fp_id in cluster data (3890: metrics_manager - sync_cluster_files)

Upgrade notes
-------------

- These update instructions apply to upgrading from v3.0.0 to v3.0.2 ONLY
- There are no changes to the DB in v3.0.2
- There are no changes to settings.py in v3.0.2

How to update from v3.0.0
-------------------------

- Download the new release tag or clone/update to get it to a temp location,
  ready to be deployed.
- Copy your settings.py from v3.0.0 and replace the v3.0.2 settings.py
- Stop the Skyline apps, backup the database and Redis, move to the new version,
  start the Skyline apps.


.. code-block:: bash

    # Get the new version
    NEW_SKYLINE_VERSION="v3.0.2"    # Your new Skyline version
    OLD_SKYLINE_VERSION="v3.0.0"    # Your old Skyline version

    CURRENT_SKYLINE_PATH="/opt/skyline/github/skyline"                 # Your Skyline path
    NEW_SKYLINE_PATH="${CURRENT_SKYLINE_PATH}.${NEW_SKYLINE_VERSION}"  # Your new Skyline path

    mkdir -p "${CURRENT_SKYLINE_PATH}.${NEW_SKYLINE_VERSION}"
    cd "${CURRENT_SKYLINE_PATH}.${NEW_SKYLINE_VERSION}"
    git clone https://github.com/earthgecko/skyline .
    git checkout "$NEW_SKYLINE_VERSION"

    # settings.py
    cp "$NEW_SKYLINE_PATH/skyline/settings.py" "$NEW_SKYLINE_PATH/skyline/settings.py.${NEW_SKYLINE_VERSION}.bak"

    # Create a new settings.py file in the new version based on your existing
    # settings.py file
    cat "${CURRENT_SKYLINE_PATH}/skyline/settings.py" > "$NEW_SKYLINE_PATH/skyline/settings.py"

    # **TEST**
    # Test your new settings.py BEFORE continuing with the upgrade,
    # test_settings.sh runs the validate_settings.py that the Skyline apps run
    # when they start
    $NEW_SKYLINE_PATH/bin/test_settings.sh

    # Stop/disable any/all service controls like monit, etc that are controlling
    # Skyline services.

    # NOTE ALL SKYLINE SERVICES ARE LISTED HERE, REMOVE TO ONES YOU DO NOT RUN
    # or do not wish to run.
    SKYLINE_SERVICES="analyzer
    analyzer_batch
    mirage
    crucible
    boundary
    ionosphere
    luminosity
    panorama
    webapp
    vista
    snab
    flux
    horizon"
    for i in $SKYLINE_SERVICES
    do
      systemctl stop "$i"
    done

- Move your current Skyline directory to a backup directory and move the new
  Skyline v3.0.2 with your new settings.py from the temp location to your
  working Skyline directory, (change your paths as appropriate) e.g.

.. code-block:: bash

    mv "$CURRENT_SKYLINE_PATH" "${CURRENT_SKYLINE_PATH}.${OLD_SKYLINE_VERSION}"
    mv "$NEW_SKYLINE_PATH" "$CURRENT_SKYLINE_PATH"

    # Set permission on the dump dir
    chown skyline:skyline "$CURRENT_SKYLINE_PATH"/skyline/webapp/static/dump


- Start the all Skyline services (change as appropriate for your set up) e.g.

.. code-block:: bash

    # NOTE ALL SKYLINE SERVICES ARE LISTED HERE, REMOVE TO ONES YOU DO NOT RUN
    # apart from the new thunder Skyline app

    # Start all other Skyline services
    SKYLINE_SERVICES="horizon
    flux
    panorama
    webapp
    vista
    analyzer
    analyzer_batch
    mirage
    crucible
    boundary
    ionosphere
    luminosity
    snab
    thunder"
    for i in $SKYLINE_SERVICES
    do
      systemctl start "$i"
      if [ $? -ne 0 ]; then
        echo "failed to start $i"
      else
        echo "started $i"
      fi
    done
    # Restart any/all service controls like monit, etc that are controlling
    # Skyline services.

- Check the logs

.. code-block:: bash

    # How are they running
    tail -n 20 /var/log/skyline/*.log

    # Any errors - each app
    find /var/log/skyline -type f -name "*.log" | while read skyline_logfile
    do
      echo "#####
    # Checking for errors in $skyline_logfile"
      cat "$skyline_logfile" | grep -B2 -A10 -i "error ::\|traceback" | tail -n 60
      echo ""
      echo ""
    done

Congratulations, you are now running the best open source anomaly detection
stack in the world (probably).