Earth System Grid Federation Data Access

The Earth System Grid Federation (ESGF) has a search API that can be used by clients to query catalog content matching constraints (see API documentation). It’s possible to send requests directly to the API using a simple function (see example), but here we’ll use a python client named pyesgf to interact with the search API and get data from the ESGF THREDDS servers. The following shows examples of typical queries for data.

If a login username and credentials are required, follow these instructions.

# NBVAL_IGNORE_OUTPUT

from pyesgf.search import SearchConnection

# Create a connection for search on ESGF nodes. Note that setting `distrib=True` can lead to weird failures.
conn = SearchConnection("https://esgf-node.llnl.gov/esg-search/", distrib=False)

# Launch a search query.
# Here we're looking for any variable related to humidity within the CMIP6 SSP2-4.5 experiment.
# Results will be stored in a dictionary with keys defined by the `facets` argument.
ctx = conn.new_context(
    project="CMIP6",
    experiment_id="ssp245",
    query="humidity",
    facets="variable_id,source_id",
)

print("Number of results: ", ctx.hit_count)
print("Variables related to humidity: ")
ctx.facet_counts["variable_id"]

Number of results:  10647
Variables related to humidity: 

{'tnhusscpbl': 163,
 'tnhusscp': 70,
 'tnhuspbl': 70,
 'tnhusmp': 172,
 'tnhusd': 34,
 'tnhusc': 224,
 'tnhusa': 180,
 'tnhus': 76,
 'hussLut': 36,
 'huss': 1993,
 'hus850': 190,
 'hus': 2420,
 'hursmin': 699,
 'hursmax': 682,
 'hurs': 1986,
 'hur': 1652}

# NBVAL_IGNORE_OUTPUT

# Now let's look for simulations that have the `hurs` variable and pick the first member.
ctx.constrain(variable_id="hurs", ensemble="r1i1p1f1")
ctx.facet_counts["source_id"]

{'UKESM1-0-LL': 126,
 'TaiESM1': 20,
 'NorESM2-MM': 36,
 'NorESM2-LM': 188,
 'NESM3': 21,
 'MRI-ESM2-0': 663,
 'MPI-ESM1-2-LR': 999,
 'MPI-ESM1-2-HR': 51,
 'MIROC6': 1989,
 'MIROC-ES2L': 1861,
 'MIROC-ES2H': 84,
 'MCM-UA-1-0': 8,
 'KIOST-ESM': 30,
 'KACE-1-0-G': 68,
 'IPSL-CM6A-LR': 236,
 'INM-CM5-0': 26,
 'INM-CM4-8': 26,
 'IITM-ESM': 17,
 'HadGEM3-GC31-LL': 110,
 'GISS-E2-2-G': 40,
 'GISS-E2-1-H': 84,
 'GISS-E2-1-G-CC': 8,
 'GISS-E2-1-G': 325,
 'GFDL-ESM4': 20,
 'GFDL-CM4': 34,
 'FIO-ESM-2-0': 16,
 'FGOALS-g3': 88,
 'FGOALS-f3-L': 12,
 'EC-Earth3-Veg-LR': 37,
 'EC-Earth3-Veg': 100,
 'EC-Earth3-CC': 64,
 'EC-Earth3-AerChem': 2,
 'EC-Earth3': 817,
 'E3SM-1-1': 22,
 'CanESM5-CanOE': 24,
 'CanESM5-1': 208,
 'CanESM5': 1033,
 'CNRM-ESM2-1': 108,
 'CNRM-CM6-1-HR': 19,
 'CNRM-CM6-1': 78,
 'CMCC-ESM2': 20,
 'CMCC-CM2-SR5': 18,
 'CIESM': 9,
 'CESM2-WACCM': 168,
 'CESM2': 184,
 'CAS-ESM2-0': 20,
 'CAMS-CSM1-0': 6,
 'BCC-CSM2-MR': 20,
 'AWI-CM-1-1-MR': 19,
 'ACCESS-ESM1-5': 408,
 'ACCESS-CM2': 77}

# We can now refine the search and get datasets corresponding within our search context
results = ctx.constrain(source_id="CanESM5").search()
r = results[0]
r.dataset_id

'CMIP6.ScenarioMIP.CCCma.CanESM5.ssp245.r14i1p2f1.Amon.hus.gn.v20190429|crd-esgf-drc.ec.gc.ca'

# To get file download links, there's an extra step
file_ctx = r.file_context()
file_ctx.facets = "*"
files = file_ctx.search()
[f.download_url for f in files]

['http://crd-esgf-drc.ec.gc.ca/thredds/fileServer/esgD_dataroot/AR6/CMIP6/ScenarioMIP/CCCma/CanESM5/ssp245/r14i1p2f1/Amon/hus/gn/v20190429/hus_Amon_CanESM5_ssp245_r14i1p2f1_gn_201501-210012.nc']

# Instead of a download URL, we can also get OPeNDAP links.
urls = [f.opendap_url for f in files]
print(urls)

# It's sometimes possible to request aggregations of multiple netCDF into one OPeNDAP link,
# but this option is often unavailable.
agg_ctx = r.aggregation_context()
agg_ctx.facets = "*"
agg = agg_ctx.search()[0]
print(agg.opendap_url)

['http://crd-esgf-drc.ec.gc.ca/thredds/dodsC/esgD_dataroot/AR6/CMIP6/ScenarioMIP/CCCma/CanESM5/ssp245/r14i1p2f1/Amon/hus/gn/v20190429/hus_Amon_CanESM5_ssp245_r14i1p2f1_gn_201501-210012.nc']

None

# Open the opendap link with xarray
import xarray as xr

ds = xr.open_mfdataset(urls)
ds

<xarray.Dataset> Size: 643MB
Dimensions:    (time: 1032, bnds: 2, plev: 19, lat: 64, lon: 128)
Coordinates:
  * time       (time) object 8kB 2015-01-16 12:00:00 ... 2100-12-16 12:00:00
  * plev       (plev) float64 152B 1e+05 9.25e+04 8.5e+04 ... 1e+03 500.0 100.0
  * lat        (lat) float64 512B -87.86 -85.1 -82.31 ... 82.31 85.1 87.86
  * lon        (lon) float64 1kB 0.0 2.812 5.625 8.438 ... 351.6 354.4 357.2
Dimensions without coordinates: bnds
Data variables:
    time_bnds  (time, bnds) object 17kB dask.array<chunksize=(1032, 2), meta=np.ndarray>
    lat_bnds   (lat, bnds) float64 1kB dask.array<chunksize=(64, 2), meta=np.ndarray>
    lon_bnds   (lon, bnds) float64 2kB dask.array<chunksize=(128, 2), meta=np.ndarray>
    hus        (time, plev, lat, lon) float32 643MB dask.array<chunksize=(1032, 19, 64, 128), meta=np.ndarray>
Attributes: (12/54)
    CCCma_model_hash:                fc4bb7db954c862d023b546e19aec6c588bc0552
    CCCma_parent_runid:              p2-his14
    CCCma_pycmor_hash:               26c970628162d607fffd14254956ebc6dd3b6f49
    CCCma_runid:                     p2-s4514
    Conventions:                     CF-1.7 CMIP-6.2
    YMDH_branch_time_in_child:       2015:01:01:00
    ...                              ...
    variable_id:                     hus
    variant_label:                   r14i1p2f1
    version:                         v20190429
    license:                         CMIP6 model data produced by The Governm...
    cmor_version:                    3.5.0
    DODS_EXTRA.Unlimited_Dimension:  time

xarray.Dataset

• Dimensions:
  • time: 1032
  • bnds: 2
  • plev: 19
  • lat: 64
  • lon: 128
• Coordinates: (4)
  • time
    (time)
    object
    2015-01-16 12:00:00 ... 2100-12-...

    bounds :

    time_bnds

    axis :

    T

    long_name :

    time

    standard_name :

    time

    _ChunkSizes :

    1

    array([cftime.DatetimeNoLeap(2015, 1, 16, 12, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(2015, 2, 15, 0, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(2015, 3, 16, 12, 0, 0, 0, has_year_zero=True),
           ...,
           cftime.DatetimeNoLeap(2100, 10, 16, 12, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(2100, 11, 16, 0, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(2100, 12, 16, 12, 0, 0, 0, has_year_zero=True)],
          dtype=object)

  • plev
    (plev)
    float64
    1e+05 9.25e+04 ... 500.0 100.0

    units :

    Pa

    axis :

    Z

    positive :

    down

    long_name :

    pressure

    standard_name :

    air_pressure

    array([100000.,  92500.,  85000.,  70000.,  60000.,  50000.,  40000.,  30000.,
            25000.,  20000.,  15000.,  10000.,   7000.,   5000.,   3000.,   2000.,
             1000.,    500.,    100.])

  • lat
    (lat)
    float64
    -87.86 -85.1 -82.31 ... 85.1 87.86

    bounds :

    lat_bnds

    units :

    degrees_north

    axis :

    Y

    long_name :

    Latitude

    standard_name :

    latitude

    array([-87.863799, -85.096527, -82.312913, -79.525607, -76.7369  , -73.947515,
           -71.157752, -68.367756, -65.577607, -62.787352, -59.99702 , -57.206632,
           -54.4162  , -51.625734, -48.835241, -46.044727, -43.254195, -40.463648,
           -37.67309 , -34.882521, -32.091944, -29.30136 , -26.510769, -23.720174,
           -20.929574, -18.138971, -15.348365, -12.557756,  -9.767146,  -6.976534,
            -4.185921,  -1.395307,   1.395307,   4.185921,   6.976534,   9.767146,
            12.557756,  15.348365,  18.138971,  20.929574,  23.720174,  26.510769,
            29.30136 ,  32.091944,  34.882521,  37.67309 ,  40.463648,  43.254195,
            46.044727,  48.835241,  51.625734,  54.4162  ,  57.206632,  59.99702 ,
            62.787352,  65.577607,  68.367756,  71.157752,  73.947515,  76.7369  ,
            79.525607,  82.312913,  85.096527,  87.863799])

  • lon
    (lon)
    float64
    0.0 2.812 5.625 ... 354.4 357.2

    bounds :

    lon_bnds

    units :

    degrees_east

    axis :

    X

    long_name :

    Longitude

    standard_name :

    longitude

    array([  0.    ,   2.8125,   5.625 ,   8.4375,  11.25  ,  14.0625,  16.875 ,
            19.6875,  22.5   ,  25.3125,  28.125 ,  30.9375,  33.75  ,  36.5625,
            39.375 ,  42.1875,  45.    ,  47.8125,  50.625 ,  53.4375,  56.25  ,
            59.0625,  61.875 ,  64.6875,  67.5   ,  70.3125,  73.125 ,  75.9375,
            78.75  ,  81.5625,  84.375 ,  87.1875,  90.    ,  92.8125,  95.625 ,
            98.4375, 101.25  , 104.0625, 106.875 , 109.6875, 112.5   , 115.3125,
           118.125 , 120.9375, 123.75  , 126.5625, 129.375 , 132.1875, 135.    ,
           137.8125, 140.625 , 143.4375, 146.25  , 149.0625, 151.875 , 154.6875,
           157.5   , 160.3125, 163.125 , 165.9375, 168.75  , 171.5625, 174.375 ,
           177.1875, 180.    , 182.8125, 185.625 , 188.4375, 191.25  , 194.0625,
           196.875 , 199.6875, 202.5   , 205.3125, 208.125 , 210.9375, 213.75  ,
           216.5625, 219.375 , 222.1875, 225.    , 227.8125, 230.625 , 233.4375,
           236.25  , 239.0625, 241.875 , 244.6875, 247.5   , 250.3125, 253.125 ,
           255.9375, 258.75  , 261.5625, 264.375 , 267.1875, 270.    , 272.8125,
           275.625 , 278.4375, 281.25  , 284.0625, 286.875 , 289.6875, 292.5   ,
           295.3125, 298.125 , 300.9375, 303.75  , 306.5625, 309.375 , 312.1875,
           315.    , 317.8125, 320.625 , 323.4375, 326.25  , 329.0625, 331.875 ,
           334.6875, 337.5   , 340.3125, 343.125 , 345.9375, 348.75  , 351.5625,
           354.375 , 357.1875])

• Data variables: (4)
  • time_bnds
    (time, bnds)
    object
    dask.array<chunksize=(1032, 2), meta=np.ndarray>

    _ChunkSizes :

    [1 2]

    Array Chunk Bytes 16.12 kiB 16.12 kiB Shape (1032, 2) (1032, 2) Dask graph 1 chunks in 2 graph layers Data type object numpy.ndarray

  • lat_bnds
    (lat, bnds)
    float64
    dask.array<chunksize=(64, 2), meta=np.ndarray>

    _ChunkSizes :

    [64 2]

    Array Chunk Bytes 1.00 kiB 1.00 kiB Shape (64, 2) (64, 2) Dask graph 1 chunks in 2 graph layers Data type float64 numpy.ndarray

  • lon_bnds
    (lon, bnds)
    float64
    dask.array<chunksize=(128, 2), meta=np.ndarray>

    _ChunkSizes :

    [128 2]

    Array Chunk Bytes 2.00 kiB 2.00 kiB Shape (128, 2) (128, 2) Dask graph 1 chunks in 2 graph layers Data type float64 numpy.ndarray

  • hus
    (time, plev, lat, lon)
    float32
    dask.array<chunksize=(1032, 19, 64, 128), meta=np.ndarray>

    standard_name :

    specific_humidity

    long_name :

    Specific Humidity

    comment :

    Specific humidity is the mass fraction of water vapor in (moist) air.

    units :

    1

    original_name :

    SHUM

    cell_methods :

    time: mean

    cell_measures :

    area: areacella

    history :

    2019-09-25T23:36:52Z altered by CMOR: Reordered dimensions, original order: lat lon plev time. 2019-09-25T23:36:52Z altered by CMOR: replaced missing value flag (1e+38) and corresponding data with standard missing value (1e+20).

    _ChunkSizes :

    [ 1 19 64 128]

    Array Chunk Bytes 612.75 MiB 612.75 MiB Shape (1032, 19, 64, 128) (1032, 19, 64, 128) Dask graph 1 chunks in 2 graph layers Data type float32 numpy.ndarray

• Indexes: (4)
  • time
    PandasIndex

    PandasIndex(CFTimeIndex([2015-01-16 12:00:00, 2015-02-15 00:00:00, 2015-03-16 12:00:00,
                 2015-04-16 00:00:00, 2015-05-16 12:00:00, 2015-06-16 00:00:00,
                 2015-07-16 12:00:00, 2015-08-16 12:00:00, 2015-09-16 00:00:00,
                 2015-10-16 12:00:00,
                 ...
                 2100-03-16 12:00:00, 2100-04-16 00:00:00, 2100-05-16 12:00:00,
                 2100-06-16 00:00:00, 2100-07-16 12:00:00, 2100-08-16 12:00:00,
                 2100-09-16 00:00:00, 2100-10-16 12:00:00, 2100-11-16 00:00:00,
                 2100-12-16 12:00:00],
                dtype='object', length=1032, calendar='noleap', freq=None))

  • plev
    PandasIndex

    PandasIndex(Index([100000.0,  92500.0,  85000.0,  70000.0,  60000.0,  50000.0,  40000.0,
            30000.0,  25000.0,  20000.0,  15000.0,  10000.0,   7000.0,   5000.0,
             3000.0,   2000.0,   1000.0,    500.0,    100.0],
          dtype='float64', name='plev'))

  • lat
    PandasIndex

    PandasIndex(Index([ -87.86379883923273,  -85.09652698831745,  -82.31291294788636,
            -79.52560657265951,   -76.7368996803684,  -73.94751515398974,
            -71.15775201158739,  -68.36775610831324,  -65.57760701082788,
            -62.78735179896313, -59.997020108491355,   -57.2066315276433,
            -54.41619952608627, -51.625733674938296,  -48.83524096625064,
           -46.044726631101724,  -43.25419466535099,  -40.46364817811508,
            -37.67308962904537,   -34.8825209937735, -32.091943881744044,
           -29.301359621762764, -26.510769325211022,  -23.72017393353477,
            -20.92957425448953, -18.138970990239372, -15.348364759491508,
            -12.55775611523069,  -9.767145559195578,  -6.976533553948642,
            -4.185920533189158, -1.3953069108194975,  1.3953069108194975,
             4.185920533189158,   6.976533553948642,   9.767145559195578,
             12.55775611523069,  15.348364759491508,  18.138970990239372,
             20.92957425448953,   23.72017393353477,  26.510769325211022,
            29.301359621762764,  32.091943881744044,    34.8825209937735,
             37.67308962904537,   40.46364817811508,   43.25419466535099,
            46.044726631101724,   48.83524096625064,  51.625733674938296,
             54.41619952608627,    57.2066315276433,  59.997020108491355,
             62.78735179896313,   65.57760701082788,   68.36775610831324,
             71.15775201158739,   73.94751515398974,    76.7368996803684,
             79.52560657265951,   82.31291294788636,   85.09652698831745,
             87.86379883923273],
          dtype='float64', name='lat'))

  • lon
    PandasIndex

    PandasIndex(Index([     0.0,   2.8125,    5.625,   8.4375,    11.25,  14.0625,   16.875,
            19.6875,     22.5,  25.3125,
           ...
            331.875, 334.6875,    337.5, 340.3125,  343.125, 345.9375,   348.75,
           351.5625,  354.375, 357.1875],
          dtype='float64', name='lon', length=128))

• Attributes: (54)

  CCCma_model_hash :

  fc4bb7db954c862d023b546e19aec6c588bc0552

  CCCma_parent_runid :

  p2-his14

  CCCma_pycmor_hash :

  26c970628162d607fffd14254956ebc6dd3b6f49

  CCCma_runid :

  p2-s4514

  Conventions :

  CF-1.7 CMIP-6.2

  YMDH_branch_time_in_child :

  2015:01:01:00

  YMDH_branch_time_in_parent :

  2015:01:01:00

  activity_id :

  ScenarioMIP

  branch_method :

  Spin-up documentation

  branch_time_in_child :

  60225.0

  branch_time_in_parent :

  60225.0

  contact :

  ec.cccma.info-info.ccmac.ec@canada.ca

  creation_date :

  2019-09-25T23:36:52Z

  data_specs_version :

  01.00.30

  experiment :

  update of RCP4.5 based on SSP2

  experiment_id :

  ssp245

  external_variables :

  areacella

  forcing_index :

  1

  frequency :

  mon

  further_info_url :

  https://furtherinfo.es-doc.org/CMIP6.CCCma.CanESM5.ssp245.none.r14i1p2f1

  grid :

  T63L49 native atmosphere, T63 Linear Gaussian Grid; 128 x 64 longitude/latitude; 49 levels; top level 1 hPa

  grid_label :

  gn

  history :

  2019-09-25T23:36:52Z ;rewrote data to be consistent with ScenarioMIP for variable hus found in table Amon.

  initialization_index :

  1

  institution :

  Canadian Centre for Climate Modelling and Analysis, Environment and Climate Change Canada, Victoria, BC V8P 5C2, Canada

  institution_id :

  CCCma

  mip_era :

  CMIP6

  nominal_resolution :

  500 km

  parent_activity_id :

  CMIP

  parent_experiment_id :

  historical

  parent_mip_era :

  CMIP6

  parent_source_id :

  CanESM5

  parent_time_units :

  days since 1850-01-01 0:0:0.0

  parent_variant_label :

  r14i1p2f1

  physics_index :

  2

  product :

  model-output

  realization_index :

  14

  realm :

  atmos

  references :

  Geophysical Model Development Special issue on CanESM5 (https://www.geosci-model-dev.net/special_issue989.html)

  source :

  CanESM5 (2019): aerosol: interactive atmos: CanAM5 (T63L49 native atmosphere, T63 Linear Gaussian Grid; 128 x 64 longitude/latitude; 49 levels; top level 1 hPa) atmosChem: specified oxidants for aerosols land: CLASS3.6/CTEM1.2 landIce: specified ice sheets ocean: NEMO3.4.1 (ORCA1 tripolar grid, 1 deg with refinement to 1/3 deg within 20 degrees of the equator; 361 x 290 longitude/latitude; 45 vertical levels; top grid cell 0-6.19 m) ocnBgchem: Canadian Model of Ocean Carbon (CMOC); NPZD ecosystem with OMIP prescribed carbonate chemistry seaIce: LIM2

  source_id :

  CanESM5

  source_type :

  AOGCM

  sub_experiment :

  none

  sub_experiment_id :

  none

  table_id :

  Amon

  table_info :

  Creation Date:(09 May 2019) MD5:38ea1fbdf3f31659e83c719fb4c48428

  title :

  CanESM5 output prepared for CMIP6

  tracking_id :

  hdl:21.14100/b99cff78-c3d9-44b8-8f22-65e519bf820d

  variable_id :

  hus

  variant_label :

  r14i1p2f1

  version :

  v20190429

  license :

  CMIP6 model data produced by The Government of Canada (Canadian Centre for Climate Modelling and Analysis, Environment and Climate Change Canada) is licensed under a Creative Commons Attribution ShareAlike 4.0 International License (https://creativecommons.org/licenses). Consult https://pcmdi.llnl.gov/CMIP6/TermsOfUse for terms of use governing CMIP6 output, including citation requirements and proper acknowledgment. Further information about this data, including some limitations, can be found via the further_info_url (recorded as a global attribute in this file) and at https:///pcmdi.llnl.gov/. The data producers and data providers make no warranty, either express or implied, including, but not limited to, warranties of merchantability and fitness for a particular purpose. All liabilities arising from the supply of the information (including any liability arising in negligence) are excluded to the fullest extent permitted by law.

  cmor_version :

  3.5.0

  DODS_EXTRA.Unlimited_Dimension :

  time