Developer Guide

Warning

To create new processes look at examples in Emu.

Re-create a fresh environment

$ make stop  # in case you previously did 'make start'
$ conda deactivate  # exit the current 'raven' conda env so we can destroy it
$ conda env remove -n raven  # destroy the current conda env to recreate one from scratch
$ conda env create -f environment.yml
$ conda activate raven
$ make develop  # install raven-wps and additional dev tools

Building the docs

First install dependencies for the documentation:

$ make develop

Run the Sphinx docs generator:

$ make docs

Running tests

Run tests using pytest.

First activate the raven Conda environment and install pytest.

$ source activate raven
$ pip install -r requirements_dev.txt  # if not already installed
OR
$ make develop

Run quick tests (skip slow and online):

$ pytest -m 'not slow and not online'"

Run all tests:

$ pytest

Check PEP8:

$ flake8

Run tests the lazy way

Do the same as above using the Makefile.

$ make test
$ make test-all
$ make lint

Running notebooks tests

Assuming that the raven conda env has already been created and is up-to-date and raven-wps has been installed with make develop:

   # start local raven-wps server to test against
   $ make start  # remember to make stop once done

   # to test all notebooks
   $ make test-notebooks

Or::

   # to test a single notebook (note the .run at the end of the notebook path)
   $ make docs/source/notebooks/Subset_climate_data_over_watershed.ipynb.run

The notebooks may also require other WPS services (Finch and Flyingpigeon). By default these are from the production server but we can point the notebooks to local servers if needed for development purposes:

# to test all notebooks
$ make FLYINGPIGEON_WPS_URL=http://localhost:8093 FINCH_WPS_URL=http://localhost:5000 test-notebooks

Or:

# to test a single notebook (note the .run at the end of the notebook path)
$ make FLYINGPIGEON_WPS_URL=http://localhost:8093 FINCH_WPS_URL=http://localhost:5000 docs/source/notebooks/Subset_climate_data_over_watershed.ipynb.run

If instead we want to run the notebooks against the production raven-wps server or any other raven-wps servers:

# to test all notebooks
$ make WPS_URL=https://pavics.ouranos.ca/twitcher/ows/proxy/raven/wps test-notebooks

Or:

# to test juste 1 notebook (note the .run at the end of the notebook path)
$ make WPS_URL=https://pavics.ouranos.ca/twitcher/ows/proxy/raven/wps docs/source/notebooks/Subset_climate_data_over_watershed.ipynb.run

We can also override all three of the server variables (WPS_URL, FINCH_WPS_URL, FLYINGPIGEON_WPS_URL) to pick and choose any servers/services from anywhere we want.

Starting local Jupyter server to edit/develop notebooks

Assuming that the raven conda env has already been created and is up-to-date and raven-wps has been installed with make develop:

# start local raven-wps server to test against
$ make start  # remember to make stop once done

# to start local jupyter notebook server listing all current notebooks
$ make notebook  # Control-C to terminate once done

# Can also override all three WPS_URL, FINCH_WPS_URL and FLYINGPIGEON_WPS_URL here as well,
# just like 'make test-notebooks' to be able to pick and choose any servers anywhere we want.

# By overriding these variables at the 'make notebook' step, we will not need to
# override them one by one in each notebook as each notebook will also look
# for those variables as environment variables.

Bulk refresh all notebooks output

This automated refresh only works for notebooks that passed make test-notebooks above. For those that failed, manually starting a local Jupyter server and refresh them manually.

Assuming that the raven conda env has already been created and is up-to-date and raven-wps has been installed with make develop:

# start local raven-wps server to test against
$ make start  # remember to make stop once done

# to refresh all notebooks
$ make refresh-notebooks

Or:

# to refresh a single notebook (note the .refresh at the end of the notebook path)
$ make docs/source/notebooks/Assess_probabilistic_flood_risk.ipynb.refresh

# Can also override all three of the server variables (WPS_URL, FINCH_WPS_URL and FLYINGPIGEON_WPS_URL) here as well,
# just like 'make test-notebooks' to be able to pick and choose any servers/services from anywhere we want.

Prepare a release

Update the Conda specification file to build identical environments on a specific OS.

Note

You should run this on your target OS, in our case Linux.

$ conda env create -f environment.yml
$ source activate raven
$ make clean
$ make install
$ conda list -n raven --explicit > spec-file.txt

Bump a new version

Make a new version of Raven in the following steps:

  • Make sure everything is commit to GitHub.

  • Update CHANGES.rst with the next version.

  • Dry Run: bumpversion --dry-run --verbose --new-version 0.8.1 patch

  • Do it: bumpversion --new-version 0.8.1 patch

  • … or: bumpversion --new-version 0.9.0 minor

  • Push it: git push

  • Push tag: git push --tags

See the bumpversion documentation for details.