Accessing PAVICS THREDDS Server¶

The THREDDS data storing netCDF file on PAVICS has some public and private directories. Data from public directories can be accessed anonymously, while data from private directories require authentication. This notebook shows how to access public and private data on the THREDDS server.

The PAVICS THREDDS server has a testdata/ folder, in which we store test datasets to validate process requests. Within that directory is a secure/ folder whose file access requires authentication.

# define some useful variables for following steps
import os

PAVICS_HOST = os.getenv("PAVICS_HOST", "pavics.ouranos.ca")
THREDDS_URL = f"https://{PAVICS_HOST}/twitcher/ows/proxy/thredds"

assert PAVICS_HOST != "", "Invalid PAVICS HOST value."
print("THREDDS URL:", THREDDS_URL)

THREDDS URL: https://pavics.ouranos.ca/twitcher/ows/proxy/thredds

First let’s just open an unsecured link.

Now let’s do the same with a secured link.

from webob.exc import HTTPError

SECURED_URL = f"{THREDDS_URL}/dodsC/birdhouse/testdata/secure/tasmax_Amon_MPI-ESM-MR_rcp45_r2i1p1_200601-200612.nc"
try:
    ds = xr.open_dataset(SECURED_URL, decode_cf=False)
# depending on 'xarray' version, different errors are raised when failing authentication according to how they handle it
except OSError as exc:
    # "NetCDF: Access failure" xarray >= 0.20
    # "Authorization failure" xarray < 0.17
    assert "NetCDF: Access failure" in str(exc) or "Authorization failure" in str(exc)
except HTTPError as exc:  # xarray >= 0.17
    # note: raised error is 500 with 'message' Unauthorized instead of directly raising HTTPUnauthorized
    assert "401 Unauthorized" in str(exc)
else:
    raise RuntimeError(
        "Expected unauthorized response, but dataset open operation did not raise!"
    )
print("Unauthorized was raised as expected.")

Unauthorized was raised as expected.

syntax error, unexpected WORD_WORD, expecting SCAN_ATTR or SCAN_DATASET or SCAN_ERROR
context: <?xml^ version="1.0" encoding="utf-8"?><ExceptionReport version="1.0.0" xmlns="http://www.opengis.net/ows/1.1" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.opengis.net/ows/1.1 http://schemas.opengis.net/ows/1.1.0/owsExceptionReport.xsd"> <Exception exceptionCode="NoApplicableCode" locator="AccessForbidden"> <ExceptionText>Access to service is forbidden.</ExceptionText> </Exception></ExceptionReport>

To open a secured link, we need to open a session with Authentication. Using wrong Authentication credentials will not work. They will raise immediately when failing login procedure. Using valid credentials will instead succeed login, but will raise a forbidden response when attempting to retrieve the data. Either way, user must be logged in and have appropriate access to fulfill Authorization requirements of the resource.

Let’s see the result when credentials are invalid.

import requests
from requests_magpie import MagpieAuth, MagpieAuthenticationError

BAD_USR = "an-invalid-user"
BAD_PWD = "or-bad-password"

try:
    with requests.session() as session:
        session.auth = MagpieAuth(f"https://{PAVICS_HOST}/magpie", BAD_USR, BAD_PWD)
        xr.open_dataset(
            SECURED_URL, decode_cf=False
        )  # Attributes are problematic with this file.
# specific error depends on what raises (unauthorized, forbidden, login failure) and 'xarray' version
except (OSError, HTTPError, MagpieAuthenticationError) as exc:
    print("Access with invalid credentials was not permitted as expected.")
else:
    raise RuntimeError(
        "Expected authentication failure response, but login operation did not raise!"
    )

Access with invalid credentials was not permitted as expected.

syntax error, unexpected WORD_WORD, expecting SCAN_ATTR or SCAN_DATASET or SCAN_ERROR
context: <?xml^ version="1.0" encoding="utf-8"?><ExceptionReport version="1.0.0" xmlns="http://www.opengis.net/ows/1.1" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.opengis.net/ows/1.1 http://schemas.opengis.net/ows/1.1.0/owsExceptionReport.xsd"> <Exception exceptionCode="NoApplicableCode" locator="AccessForbidden"> <ExceptionText>Access to service is forbidden.</ExceptionText> </Exception></ExceptionReport>

As we can see, the server identified that credentials were provided, but they were incorrect and could not log in. Similar result would happen if login succeeded, but user was forbidden access due to insufficient permissions.

We’ve created an authtest user in advance that has access to the secure contents to facilitate testing.

Let’s use it now to obtain the secured resource.

# NBVAL_IGNORE_OUTPUT

AUTH_USR = os.getenv("TEST_MAGPIE_AUTHTEST_USERNAME", "authtest")
AUTH_PWD = os.getenv("TEST_MAGPIE_AUTHTEST_PASSWORD", "authtest1234")

# Open session
with requests.Session() as session:
    session.auth = MagpieAuth(f"https://{PAVICS_HOST}/magpie", AUTH_USR, AUTH_PWD)
    # Open a PyDAP data store and pass it to xarray
    store = xr.backends.PydapDataStore.open(SECURED_URL, session=session)
    ds = xr.open_dataset(
        store, decode_cf=False
    )  # Attributes are problematic with this file.
ds

---------------------------------------------------------------------------
ModuleNotFoundError                       Traceback (most recent call last)
Cell In[5], line 10
      8     session.auth = MagpieAuth(f"https://{PAVICS_HOST}/magpie", AUTH_USR, AUTH_PWD)
      9     # Open a PyDAP data store and pass it to xarray
---> 10     store = xr.backends.PydapDataStore.open(SECURED_URL, session=session)
     11     ds = xr.open_dataset(
     12         store, decode_cf=False
     13     )  # Attributes are problematic with this file.
     14 ds

File ~/checkouts/readthedocs.org/user_builds/pavics-sdi/conda/latest/lib/python3.10/site-packages/xarray/backends/pydap_.py:110, in PydapDataStore.open(cls, url, application, session, output_grid, timeout, verify, user_charset)
     99 @classmethod
    100 def open(
    101     cls,
   (...)
    108     user_charset=None,
    109 ):
--> 110     import pydap.client
    111     import pydap.lib
    113     if timeout is None:

ModuleNotFoundError: No module named 'pydap'

Successful listing of the above data means the user was granted access for this reference.

Accessing PAVICS THREDDS Server¶

Previous topic

Next topic

This Page