Building documentation

Currently Sphinx and sphinx-apidoc is being used to documented the code and project, however to get working docs, a small modification is required to the files outputted by sphinx-apidoc.

This is related to the package restructure undertaken to make the Skyline code and layout more in line with a normal python package. Although this seems to have been achieved, the small hack to the sphinx-apidoc output suggests that this is not 100% correct. Further evidence of this is in terms of importing from settings.py, the path needs to be appended in the code, which really should not be required. However, it is working and in the future this should be figured and fixed. Perhaps the below edit of the auto generated .rst files could be achieved with a sphinx conf.py setting, if anyone knows please do let us know :)

Part of the build

The below documentation script not only builds the documentation, it also auto generates some documentation and it also auto generates some Python code when updates are required, such as the compilation of the tsfresh_features.py

For now…

Install docs-requirements.txt

In your Python virtualenv first pip install the required modules in docs-requirements.txt (as per documented Running in Python virtualenv)

PYTHON_MAJOR_VERSION="3.8"
PYTHON_VIRTUALENV_DIR="/opt/python_virtualenv"
PROJECT="skyline-py383"

cd "${PYTHON_VIRTUALENV_DIR}/projects/${PROJECT}"
source bin/activate

bin/"pip${PYTHON_MAJOR_VERSION}" install -r /opt/skyline/github/skyline/docs-requirements.txt

deactivate

Your Python interpretor

The docs/conf.py self interpolates the Python path if you are running in a virtualenv in the documented manner. If you are not you may need to change the following to your python interpretor site-packages path by setting python_site_packages_path in docs/conf.py, e.g.

#    sys.path.insert(0, os.path.abspath('/opt/python_virtualenv/projects/skyline-py2716/lib/python2.7/site-packages'))
    sys.path.insert(0, os.path.abspath(python_site_packages_path))

.rst and .md wtf?

The documentation is written in both .md and .rst format, because it can be thanks to the awesome of Sphinx. The original Skyline documentation was written in .md for github. The documentation is being ported over to .rst to allow for the full functionality of Sphinx documentation.

bandit

The below build script tests the code with bandit (https://github.com/PyCQA/bandit). A number of places in the code are flagged with # nosec to skip bandit checks all with valid reasons. The build below also –skip B110,B112 to exclude checks relating to except, pass patterns. The except, pass patterns in the code are not flagged with # nosec. Except, pass is used a lot as certain functions in Skyline could potentially log millions of lines. The except, pass conditions are used where appropriate.

Build

PYTHON_MAJOR_VERSION="3.8"
PYTHON_VIRTUALENV_DIR="/opt/python_virtualenv"
PROJECT="skyline-py386"
CURRENT_BRANCH="SNAB"

cd "${PYTHON_VIRTUALENV_DIR}/projects/${PROJECT}"
source bin/activate

function build_docs() {

  # Arguments:
  # APP_DIR - path to your Skyline dir, e.g.
  # build_docs  # e.g. ~/github/earthgecko/skyline/develop/skyline
  # pyflakes    # run pyflakes if passed

  if [ -n "$1" ]; then
    APPDIR=$1
  fi

  if [ -z "$APPDIR" ]; then
    echo "error: could not determine APPDIR - $APPDIR"
    return 1
  fi

  if [ ! -d "$APPDIR/docs" ]; then
    echo "error: directory not found - $APPDIR/docs"
    return 1
  fi

  # @added 20161119 - Branch #922: ionosphere
  #                   Task #1718: review.tsfresh
  # Build the pytz.rst page to generate the pytz timezone list for Skyline
  # Ionosphere and tsfresh, creates "$APPDIR/docs/development/pytz.rst"

  python${PYTHON_MAJOR_VERSION} "$APPDIR/skyline/tsfresh_features/scripts/make-pytz.all_timezones-rst.py"

  # Run tests
  ORIGINAL_DIR=$(pwd)
  cd "$APPDIR"
  python${PYTHON_MAJOR_VERSION} -m pytest tests/
  if [ $? -ne 0 ]; then
    echo "Tests failed not building documentation"
    return 1
  fi

  # @added 20170308 - Task #1966: Add pyflakes tests to build_docs
  #                   Feature #1960: ionosphere_layers
  if [ -n "$2" ]; then
    find "$APPDIR" -type f -name "*.py" | while read i_file
    do
      pyflakes "$i_file"
    done
  fi

  # @added 20170913 - Task #2160: Test skyline with bandit
  # For static analysis - https://github.com/openstack/bandit
  # @modified 20200808 - Task #3608: Update Skyline to Python 3.8.3 and deps
  # Skip bandit except, pass checks
  # [B110:try_except_pass] Try, Except, Pass detected.
  # [B112:try_except_continue] Try, Except, Continue detected.
  # bandit -r "$APPDIR" -x "${APPDIR}/skyline/settings.py"
  bandit -r "$APPDIR" -x "${APPDIR}/skyline/settings.py" --skip B110,B112

  cd "$APPDIR/docs"
  echo "Building Skyline documentation - in $APPDIR/docs"
  sphinx-apidoc --force -o "${APPDIR}/docs" "${APPDIR}/skyline" skyline

  # Inline edit all apidoc generated .rst files in docs/skyline.*rst
  for i in $(find "${APPDIR}/docs" -type f -name "skyline.*rst")
  do
    cat "$i" > "${i}.org"
    cat "${i}.org" | sed -e '/package/!s/automodule:: skyline\./automodule:: /g' > "$i"
    rm -f "${i}.org"
  done

  cd "$APPDIR/docs"
  make clean
  rm -rf _build/*
  make html
  for i in $(find "$APPDIR" -type f -name "*.pyc")
  do
    rm -f "$i"
  done
  for i in $(find "$APPDIR" -type d -name "__pycache__")
  do
    rm -rf "$i"
  done
  cd "$ORIGINAL_DIR"
}

# Usage: build_docs <app_dir>
# e.g.
# cd /opt/python_virtualenv/projects/skyline-py386/
# build_docs /home/gary/sandbox/of/github/earthgecko/skyline/$CURRENT_BRANCH/skyline

Auto generating .rst files

This may be a little unconventional but it probably beats trying to do it via Sphinx support custom extensions, without using generates or includes or Jinga templating, which may or may not work with readthedocs.

The script skyline/tsfresh_features/scripts/make-pytz.all_timezones-rst.py introduces a novel way to automatically generate the docs/development/pytz.rst during the local build process to provide a list of all pytz timezones at the current version.

This pattern could be reused fairly easier.

Building workflow diagrams with UML

This can be quite handy to make simple diagrams, if not finicky. A good resource is the PlantUML.com server is handy for making workflow diagrams, without having to create and edit SVGs.

The docs/skyline.simplified.workflow.uml rendered by the PlantUML server: Simplified Skyline workflow with PlantUML server

The UML source to the above image is:

@startuml
title <font color=#6698FF>Sky</font><font color=#dd3023>line</font> Webapp - basic overview

actor You << Human >>

node "Graphite"

node "Redis"

node "MySQL"

node "webapp" {
  package "now" as now
  package "Panorama" as Panorama
  package "rebrow" as rebrow
  now <.. Redis : timeseries data
  Panorama <.. MySQL : anomaly details
  Panorama <.. Graphite : timeseries data for anomaly
  rebrow <.. Redis : keys
}

You <.. webapp : View UI via browser

right footer \nSource https://github.com/earthgecko/skyline/tree/v1.1.0-beta-ionosphere/docs/building-documentation.html\nGenerated by http://plantuml.com/plantuml
@enduml