{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Accessing PAVICS THREDDS Server\n", "\n", "\n", "The THREDDS data storing netCDF file on PAVICS has some public and private directories. Data from public directories can be accessed anonymously, while data from private directories require authentication. This notebook shows how to access public and private data on the THREDDS server.\n", "\n", "The PAVICS THREDDS server has a `testdata/` folder, in which we store test datasets to validate process requests. Within that directory is a `secure/` folder whose file access requires authentication." ] }, { "cell_type": "code", "execution_count": 1, "metadata": { "pycharm": { "name": "#%%\n" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "THREDDS URL: https://pavics.ouranos.ca/twitcher/ows/proxy/thredds\n" ] } ], "source": [ "# define some useful variables for following steps\n", "import os\n", "\n", "PAVICS_HOST = os.getenv(\"PAVICS_HOST\", \"pavics.ouranos.ca\")\n", "THREDDS_URL = f\"https://{PAVICS_HOST}/twitcher/ows/proxy/thredds\"\n", "\n", "assert PAVICS_HOST != \"\", \"Invalid PAVICS HOST value.\"\n", "print(\"THREDDS URL:\", THREDDS_URL)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "First let's just open an unsecured link." ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "
<xarray.Dataset>\n",
       "Dimensions:    (bnds: 2, lat: 160, lon: 320, plev: 23, time: 120)\n",
       "Coordinates:\n",
       "  * time       (time) datetime64[ns] 1991-01-16T12:00:00 ... 2000-12-16T12:00:00\n",
       "  * plev       (plev) float64 1e+05 9.25e+04 8.5e+04 7e+04 ... 200.0 100.0 40.0\n",
       "  * lat        (lat) float64 -89.14 -88.03 -86.91 -85.79 ... 86.91 88.03 89.14\n",
       "  * lon        (lon) float64 0.0 1.125 2.25 3.375 ... 355.5 356.6 357.8 358.9\n",
       "Dimensions without coordinates: bnds\n",
       "Data variables:\n",
       "    time_bnds  (time, bnds) datetime64[ns] 1991-01-01 1991-02-01 ... 2001-01-01\n",
       "    lat_bnds   (lat, bnds) float64 -90.0 -88.59 -88.59 ... 88.59 88.59 90.0\n",
       "    lon_bnds   (lon, bnds) float64 -0.5625 0.5625 0.5625 ... 358.3 358.3 359.4\n",
       "    ta         (time, plev, lat, lon) float32 ...\n",
       "Attributes: (12/28)\n",
       "    institution:                     MRI (Meteorological Research Institute, ...\n",
       "    institute_id:                    MRI\n",
       "    experiment_id:                   decadal1980\n",
       "    source:                          MRI-CGCM3 2011 atmosphere: GSMUV (gsmuv-...\n",
       "    model_id:                        MRI-CGCM3\n",
       "    forcing:                         GHG, SA, Oz, LU, Sl, Vl, BC, OC (GHG inc...\n",
       "    ...                              ...\n",
       "    title:                           MRI-CGCM3 model output prepared for CMIP...\n",
       "    parent_experiment:               N/A\n",
       "    modeling_realm:                  atmos\n",
       "    realization:                     1\n",
       "    cmor_version:                    2.7.1\n",
       "    DODS_EXTRA.Unlimited_Dimension:  time
" ], "text/plain": [ "\n", "Dimensions: (bnds: 2, lat: 160, lon: 320, plev: 23, time: 120)\n", "Coordinates:\n", " * time (time) datetime64[ns] 1991-01-16T12:00:00 ... 2000-12-16T12:00:00\n", " * plev (plev) float64 1e+05 9.25e+04 8.5e+04 7e+04 ... 200.0 100.0 40.0\n", " * lat (lat) float64 -89.14 -88.03 -86.91 -85.79 ... 86.91 88.03 89.14\n", " * lon (lon) float64 0.0 1.125 2.25 3.375 ... 355.5 356.6 357.8 358.9\n", "Dimensions without coordinates: bnds\n", "Data variables:\n", " time_bnds (time, bnds) datetime64[ns] ...\n", " lat_bnds (lat, bnds) float64 ...\n", " lon_bnds (lon, bnds) float64 ...\n", " ta (time, plev, lat, lon) float32 ...\n", "Attributes: (12/28)\n", " institution: MRI (Meteorological Research Institute, ...\n", " institute_id: MRI\n", " experiment_id: decadal1980\n", " source: MRI-CGCM3 2011 atmosphere: GSMUV (gsmuv-...\n", " model_id: MRI-CGCM3\n", " forcing: GHG, SA, Oz, LU, Sl, Vl, BC, OC (GHG inc...\n", " ... ...\n", " title: MRI-CGCM3 model output prepared for CMIP...\n", " parent_experiment: N/A\n", " modeling_realm: atmos\n", " realization: 1\n", " cmor_version: 2.7.1\n", " DODS_EXTRA.Unlimited_Dimension: time" ] }, "execution_count": 2, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# NBVAL_IGNORE_OUTPUT\n", "\n", "import xarray as xr\n", "\n", "PUBLIC_URL = f\"{THREDDS_URL}/dodsC/birdhouse/testdata/ta_Amon_MRI-CGCM3_decadal1980_r1i1p1_199101-200012.nc\"\n", "ds = xr.open_dataset(PUBLIC_URL)\n", "ds" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now let's do the same with a secured link." ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Unauthorized was raised as expected.\n" ] } ], "source": [ "from webob.exc import HTTPError\n", "\n", "SECURED_URL = f\"{THREDDS_URL}/dodsC/birdhouse/testdata/secure/tasmax_Amon_MPI-ESM-MR_rcp45_r2i1p1_200601-200612.nc\"\n", "try:\n", " ds = xr.open_dataset(SECURED_URL, decode_cf=False)\n", "# depending on 'xarray' version, different errors are raised when failing authentication according to how they handle it\n", "except OSError as exc:\n", " # \"NetCDF: Access failure\" xarray >= 0.20\n", " # \"Authorization failure\" xarray < 0.17\n", " assert \"NetCDF: Access failure\" in str(exc) or \"Authorization failure\" in str(exc)\n", "except HTTPError as exc: # xarray >= 0.17\n", " # note: raised error is 500 with 'message' Unauthorized instead of directly raising HTTPUnauthorized\n", " assert \"401 Unauthorized\" in str(exc)\n", "else:\n", " raise RuntimeError(\n", " \"Expected unauthorized response, but dataset open operation did not raise!\"\n", " )\n", "print(\"Unauthorized was raised as expected.\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "To open a secured link, we need to open a session with `Authentication`.\n", "Using wrong `Authentication` credentials will not work. They will raise immediately when failing login procedure.\n", "Using valid credentials will instead succeed login, but will raise a forbidden response when attempting to retrieve\n", "the data. Either way, user must be logged in and have appropriate access to fulfill `Authorization` requirements\n", "of the resource.\n", "\n", "Let's see the result when credentials are invalid." ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Access with invalid credentials was not permitted as expected.\n" ] } ], "source": [ "import requests\n", "from requests_magpie import MagpieAuth, MagpieAuthenticationError\n", "\n", "BAD_USR = \"an-invalid-user\"\n", "BAD_PWD = \"or-bad-password\"\n", "\n", "try:\n", " with requests.session() as session:\n", " session.auth = MagpieAuth(f\"https://{PAVICS_HOST}/magpie\", BAD_USR, BAD_PWD)\n", " xr.open_dataset(\n", " SECURED_URL, decode_cf=False\n", " ) # Attributes are problematic with this file.\n", "# specific error depends on what raises (unauthorized, forbidden, login failure) and 'xarray' version\n", "except (OSError, HTTPError, MagpieAuthenticationError) as exc:\n", " print(\"Access with invalid credentials was not permitted as expected.\")\n", "else:\n", " raise RuntimeError(\n", " \"Expected authentication failure response, but login operation did not raise!\"\n", " )" ] }, { "cell_type": "markdown", "metadata": { "pycharm": { "name": "#%% md\n" } }, "source": [ "As we can see, the server identified that credentials were provided, but they were incorrect and could not log in.\n", "Similar result would happen if login succeeded, but user was forbidden access due to insufficient permissions.\n", "\n", "We've created an `authtest` user in advance that has access to the `secure` contents to facilitate testing.\n", "\n", "Let's use it now to obtain the secured resource.\n" ] }, { "cell_type": "code", "execution_count": 5, "metadata": { "pycharm": { "name": "#%%\n" } }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "
<xarray.Dataset>\n",
       "Dimensions:             (bnds: 2, lat: 96, lon: 192, time: 12)\n",
       "Coordinates:\n",
       "  * time                (time) float64 5.699e+04 5.702e+04 ... 5.733e+04\n",
       "  * lat                 (lat) float64 -88.57 -86.72 -84.86 ... 84.86 86.72 88.57\n",
       "  * lon                 (lon) float64 0.0 1.875 3.75 5.625 ... 354.4 356.2 358.1\n",
       "Dimensions without coordinates: bnds\n",
       "Data variables:\n",
       "    time_bnds           (time, bnds) float64 5.698e+04 5.701e+04 ... 5.734e+04\n",
       "    lat_bnds            (lat, bnds) float64 -90.0 -87.65 -87.65 ... 87.65 90.0\n",
       "    lon_bnds            (lon, bnds) float64 -0.9375 0.9375 ... 357.2 359.1\n",
       "    latitude_longitude  <U1 ''\n",
       "    tasmax              (time, lat, lon) float32 ...\n",
       "Attributes: (12/28)\n",
       "    institution:                     Max Planck Institute for Meteorology\n",
       "    institute_id:                    MPI-M\n",
       "    experiment_id:                   rcp45\n",
       "    source:                          MPI-ESM-MR 2011; URL: http://svn.zmaw.de...\n",
       "    model_id:                        MPI-ESM-MR\n",
       "    forcing:                         GHG,Oz,SD,Sl,Vl,LU\n",
       "    ...                              ...\n",
       "    title:                           MPI-ESM-MR model output prepared for CMI...\n",
       "    parent_experiment:               historical\n",
       "    modeling_realm:                  atmos\n",
       "    realization:                     1\n",
       "    cmor_version:                    2.6.0\n",
       "    DODS_EXTRA.Unlimited_Dimension:  time
" ], "text/plain": [ "\n", "Dimensions: (bnds: 2, lat: 96, lon: 192, time: 12)\n", "Coordinates:\n", " * time (time) float64 5.699e+04 5.702e+04 ... 5.733e+04\n", " * lat (lat) float64 -88.57 -86.72 -84.86 ... 84.86 86.72 88.57\n", " * lon (lon) float64 0.0 1.875 3.75 5.625 ... 354.4 356.2 358.1\n", "Dimensions without coordinates: bnds\n", "Data variables:\n", " time_bnds (time, bnds) float64 ...\n", " lat_bnds (lat, bnds) float64 ...\n", " lon_bnds (lon, bnds) float64 ...\n", " latitude_longitude |S128 ...\n", " tasmax (time, lat, lon) float32 ...\n", "Attributes: (12/28)\n", " institution: Max Planck Institute for Meteorology\n", " institute_id: MPI-M\n", " experiment_id: rcp45\n", " source: MPI-ESM-MR 2011; URL: http://svn.zmaw.de...\n", " model_id: MPI-ESM-MR\n", " forcing: GHG,Oz,SD,Sl,Vl,LU\n", " ... ...\n", " title: MPI-ESM-MR model output prepared for CMI...\n", " parent_experiment: historical\n", " modeling_realm: atmos\n", " realization: 1\n", " cmor_version: 2.6.0\n", " DODS_EXTRA.Unlimited_Dimension: time" ] }, "execution_count": 5, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# NBVAL_IGNORE_OUTPUT\n", "\n", "AUTH_USR = os.getenv(\"TEST_MAGPIE_AUTHTEST_USERNAME\", \"authtest\")\n", "AUTH_PWD = os.getenv(\"TEST_MAGPIE_AUTHTEST_PASSWORD\", \"authtest1234\")\n", "\n", "# Open session\n", "with requests.Session() as session:\n", " session.auth = MagpieAuth(f\"https://{PAVICS_HOST}/magpie\", AUTH_USR, AUTH_PWD)\n", " # Open a PyDAP data store and pass it to xarray\n", " store = xr.backends.PydapDataStore.open(SECURED_URL, session=session)\n", " ds = xr.open_dataset(\n", " store, decode_cf=False\n", " ) # Attributes are problematic with this file.\n", "ds" ] }, { "cell_type": "markdown", "metadata": { "pycharm": { "name": "#%% md\n" } }, "source": [ "Successful listing of the above data means the user was granted access for this reference.\n" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.9.13" } }, "nbformat": 4, "nbformat_minor": 4 }