The DKRZ CMIP Data Pool

This is a beginners-level demonstration notebook and introduces you to the Data Pool at DKRZ. Based on the example of the recent phase 6 of the Coupled Model Intercomparison Project (CMIP6), you will learn - how you benefit from the CMIP Data Pool (CDP) - how to approach CMIP data - how to use the python packages intake-esm, xarray and pandas to investigate the CMIP Data Pool

This notebook can be executed on DKRZ’s jupyterhub platform. For a detailled introduction into jupyterhub and intake, we recommend the DKRZ tech talks

Customizing the code inside, however, only requires basic python knowledge.

Introduction

The Scientific Steering Commitee has thankfully granted a disk space on mistral lustre file system of 5PB for the CMIP Data Pool for 2021. Started in 2016, DKRZ runs and maintains this common storage place.

📢 The DKRZ CMIP data pool contains often needed flagship collections of climate model data, is hosted as part of the DKRZ data infrastructure and supports scientists in high volume climate data collection, access and processing.

The notebook sources for the doc pages are available in this gitlab-repo

Important news and updates will be announced - on the new DKRZ user portal where you can also find this demonstration notebook. - via a mailing list. Subscribe for ✉ cmip-data-poolATlists.dkrz.de

⭐ Highlight CDP climate model data collections are: - CMIP6: In May 2021, DKRZ provides Europe’s largest data pool with an amount of 4 PB for the recent phase of the Coupled Model Intercomparison Project - CORDEX: The size of data for the Coordinated Regional Downscaling Experiment is about 600TB over different projects. - CMIP5: The fifth phase of CMIP.

An example of a project which is also in the data pool, but not included in the term CMIP6⁺: - ERA5: Weather data from the European Centre for Medium-Range Weather Forecasts by re-analysed and homogenised observation data.

from IPython.display import HTML, display, Markdown, IFrame
display(Markdown("Time series of three different data pool disk space measures. DKRZ has published about 1.5 PB, 2.5 PB are replicated data from other data nodes. An average CMIP6 dataset contains about 5 files and covers 4GB."))
IFrame(src="https://swift.dkrz.de/v1/dkrz_a44962e3ba914c309a7421573a6949a6/Pool-Statistics/pool-timeseries-hvplot.html",width="900",height="550",frameborder="0")

Time series of three different data pool disk space measures. DKRZ has published about 1.5 PB, 2.5 PB are replicated data from other data nodes. An average CMIP6 dataset contains about 5 files and covers 4GB.

Ongoing Activities - Analysis support

One part of the data pool support is to make computational resources available to a broader and EU-wide user community. Two IS-ENES services both free of charge but limited to the available resources enable users to join computational projects equipped with sufficient resources for CMIP analysis and for a temporary amount of time.

  1. The phase 3 of IS-ENES provides the Analysis Platforms service

    • Regular proposal mechanism with a review procedure.

    • Successful proposals are granted exclusively resources for server-side data analyses.

  2. The ENES Climate Analytics Service ECAS

    • Minimal application procedure most likely with a positive outcome

    • One month limited and shared resources

display(Markdown("We develop, prepare and provide [jupyter notebook demonstrations](https://gitlab.dkrz.de/mipdata/tutorials-and-use-cases) <br> "
                 "- as tutorials for software packages and applications *starting from scratch* </br>"
                 "- for more frequent use cases like the plot of `tas` of one member of two experiments and simulated by the German ESMs."))
IFrame(src="https://swift.dkrz.de/v1/dkrz_a44962e3ba914c309a7421573a6949a6/plots/globalmean-yearlymean-tas.html",width="1000",height="650",frameborder="0")

We develop, prepare and provide jupyter notebook demonstrations - as tutorials for software packages and applications starting from scratch - for more frequent use cases like the plot of tas of one member of two experiments and simulated by the German ESMs.

Why do we host the CDP? 🤔

👉 The key benefit of the data pool is that the data is available on lustre (/work) so that All DKRZ users with a current account have access. There is less need for local copies or data downloads. 👈

Where can I find the data pool? 🐕

The Data pool can be accessed from different portals.

  • Server-side on the file system e.g. under /pool/data/CMIP6/data

    • All mistral/levante users with a current account have permission to do that.

    • Fastest way to work with the data

%%bash
#Browsing with linux commands
ls /pool/data/CMIP6/data/ -x
echo ""
#For which MIPs did MPI-ESM1-2-XR produce data for?
find /pool/data/CMIP6/data/ -maxdepth 3 -name MPI-ESM1-2-XR -type d
AerChemMIP   C4MIP   CDRMIP  CFMIP   CMIP   DAMIP  DCPP   FAFMIP  GeoMIP
GMMIP            HighResMIP  ISMIP6  LS3MIP  LUMIP  OMIP   PAMIP  PMIP    RFMIP
ScenarioMIP

/pool/data/CMIP6/data/HighResMIP/MPI-M/MPI-ESM1-2-XR
%%bash
#Using the FreVA CMIP-ESMVal tool
module load cmip6-dicad/1.0
#freva --databrowser --help
bash 4.x auto-completion script successfully loaded
cmip6 Evaluation System by Freva successfully loaded.
If you are using bash, try the auto complete feature for freva and freva --databrowser byhitting tab as usual.
For more help/information check: cmip-esmvaltool.dkrz.de

cmip6-prod by Freva
Available commands:
  --plugin       : Applies some analysis to the given data.
  --history      : provides access to the configuration history
  --databrowser  : Find data in the system
  --crawl_my_data: Use this command to update your projectdata.
  --esgf         : Browse ESGF data and create wget script

Usage: freva --COMMAND [OPTIONS]
To get help for the individual commands use
  freva --COMMAND --help

Understanding CMIP6 data

🧑‍🏫 The goal of CMIP6

In order to evaluate and compare climate models, a globally organized intercomparison project is periodically conducted. CMIP6 tackles three major questions:

  • How does the Earth system respond to forcing? 🚂

  • What are the origins and consequences of systematic model biases? 🐞

  • How can we assess future climate changes given internal climate variability, predictability, and uncertainties and scenarios? 🌡

From Eyring et al., 2016. Schematic of the CMIP/CMIP6 experiment design

The CMIP6 framework allows smaller model intercomparison projects (MIPs) with a specific focus to be endorsed to CMIP6. That means, each model that runs the standard CMIP experiments can participate in CMIP6 and further MIPs.

Metadata: Required Attributes and Controlled Vocabularies

CDP data is self-descriptive as it contains extensive and controlled metadata. This metadata is prepared in the search facets of the data portals and catalogs.

📜

Besides the technical requirements, the CMIP data standard defines required attributes in so called Controlled Vocabularies (CV). While some values are predefined, models and institutions have to be registered to become a valid value of corresponding attributes. For many attributes, both a short form with _id and a longer description exist.

Important required attributes:

  • activity_id: A CMIP6-endorsed MIP that investigates a specific research question. It defines experiments and requests data for it.

  • source_id : An ID for the Earth System Model used to produce the data.

  • experiment_id: The experiment which was conducted by the source_id.

  • member_id : The ensemble simulation member of the experiment_id. All members should be statistically equal.

Investigating the CMIP6 data pool with intake-esm

Features

  • display catalogs as clearly structured tables inside jupyter notebooks for easy investigation

import intake
cloudpath="https://swift.dkrz.de/v1/dkrz_a44962e3ba914c309a7421573a6949a6/intake-esm/mistral-cmip6.json"
poolpath="/pool/data/Catalogs/mistral-cmip6.json"
col = intake.open_esm_datastore(cloudpath)
col.df.head()
activity_id institution_id source_id experiment_id member_id table_id variable_id grid_label dcpp_init_year version time_range path opendap_url
0 AerChemMIP BCC BCC-ESM1 hist-piAer r1i1p1f1 AERmon c2h6 gn NaN v20200511 185001-201412 /mnt/lustre02/work/ik1017/CMIP6/data/CMIP6/Aer... NaN
1 AerChemMIP BCC BCC-ESM1 hist-piAer r1i1p1f1 AERmon c3h6 gn NaN v20200511 185001-201412 /mnt/lustre02/work/ik1017/CMIP6/data/CMIP6/Aer... NaN
2 AerChemMIP BCC BCC-ESM1 hist-piAer r1i1p1f1 AERmon c3h8 gn NaN v20200511 185001-201412 /mnt/lustre02/work/ik1017/CMIP6/data/CMIP6/Aer... NaN
3 AerChemMIP BCC BCC-ESM1 hist-piAer r1i1p1f1 AERmon cdnc gn NaN v20200522 185001-201412 /mnt/lustre02/work/ik1017/CMIP6/data/CMIP6/Aer... NaN
4 AerChemMIP BCC BCC-ESM1 hist-piAer r1i1p1f1 AERmon ch3coch3 gn NaN v20200511 185001-201412 /mnt/lustre02/work/ik1017/CMIP6/data/CMIP6/Aer... NaN

Features

  • browse through the catalog and select your data without being on the pool file system

⇨ A pythonic reproducable alternative compared to complex find commands or GUI searches. No need for Filesystems and filenames.

tas = col.search(experiment_id="historical", source_id="MPI-ESM1-2-HR", variable_id="tas", table_id="Amon", member_id="r1i1p1f1")
tas

mistral-cmip6 catalog with 1 dataset(s) from 33 asset(s):

unique
activity_id 1
institution_id 1
source_id 1
experiment_id 1
member_id 1
table_id 1
variable_id 1
grid_label 1
dcpp_init_year 0
version 1
time_range 33
path 33
opendap_url 0

Features

  • open climate data in an analysis ready dictionary of xarray datasets

Forget about temporary merging and reformatting steps!

tas.to_dataset_dict()
--> The keys in the returned dictionary of datasets are constructed as follows:
    'activity_id.institution_id.source_id.experiment_id.table_id.grid_label'
100.00% [1/1 00:00<00:00]
{'CMIP.MPI-M.MPI-ESM1-2-HR.historical.Amon.gn': <xarray.Dataset>
 Dimensions:    (bnds: 2, lat: 192, lon: 384, member_id: 1, time: 1980)
 Coordinates:
   * time       (time) datetime64[ns] 1850-01-16T12:00:00 ... 2014-12-16T12:00:00
   * lat        (lat) float64 -89.28 -88.36 -87.42 -86.49 ... 87.42 88.36 89.28
   * lon        (lon) float64 0.0 0.9375 1.875 2.812 ... 356.2 357.2 358.1 359.1
     height     float64 ...
   * member_id  (member_id) <U8 'r1i1p1f1'
 Dimensions without coordinates: bnds
 Data variables:
     time_bnds  (time, bnds) datetime64[ns] dask.array<chunksize=(60, 2), meta=np.ndarray>
     lat_bnds   (time, lat, bnds) float64 dask.array<chunksize=(60, 192, 2), meta=np.ndarray>
     lon_bnds   (time, lon, bnds) float64 dask.array<chunksize=(60, 384, 2), meta=np.ndarray>
     tas        (member_id, time, lat, lon) float32 dask.array<chunksize=(1, 60, 192, 384), meta=np.ndarray>
 Attributes: (12/49)
     source:                  MPI-ESM1.2-HR (2017): naerosol: none, prescribe...
     mip_era:                 CMIP6
     initialization_index:    1
     nominal_resolution:      100 km
     source_type:             AOGCM
     sub_experiment:          none
     ...                      ...
     forcing_index:           1
     parent_activity_id:      CMIP
     grid_label:              gn
     project_id:              CMIP6
     physics_index:           1
     intake_esm_dataset_key:  CMIP.MPI-M.MPI-ESM1-2-HR.historical.Amon.gn}

Intake best practises:

  • Intake can make your scripts reusable.

    • Instead of working with local copy or editions of files, always start from a globally defined catalog which everyone can access

    • Save the subset of the catalog which you work on as a new catalog instead of a subset of files

  • Check for new ingests by just repeating your script - it will open the most recent catalog.

  • Only load datasets with to_dataset_dict into xarrray which do not exceed your memory limits

Let’s get an overview over the CMIP6 Data pool by - finding the number of unique values of attributes - group and plot the names and sizes of different entries

The resulting statistics is about the percentage of File numbers.

unique_activites=col.unique("activity_id")
print(list(unique_activites["activity_id"].values()))
[19, ['AerChemMIP', 'C4MIP', 'CDRMIP', 'CFMIP', 'CMIP', 'DAMIP', 'DCPP', 'FAFMIP', 'GMMIP', 'GeoMIP', 'HighResMIP', 'ISMIP6', 'LS3MIP', 'LUMIP', 'OMIP', 'PAMIP', 'PMIP', 'RFMIP', 'ScenarioMIP']]
def pieplot(gbyelem) :
    #groupby, sort and select the ten largest
    size = col.df.groupby([gbyelem]).size().sort_values(ascending=False)
    size10 = size.nlargest(10)
    #Sum all others as 10th entry
    size10[9] = sum(size[9:])
    size10.rename(index={size10.index.values[9]:'all other'},inplace=True)
    #return a pie plot
    return size10.plot.pie(figsize=(18,8),ylabel='',autopct='%.2f', fontsize=16)
pieplot("activity_id")
<matplotlib.axes._subplots.AxesSubplot at 0x7ffa846f2630>
unique_sources=col.unique("source_id")
print("Number of unique earth system models in the cmip6 data pool: "+str(list(unique_sources["source_id"].values())[0]))
Number of unique earth system models in the cmip6 data pool: 101
pieplot("source_id")
<matplotlib.axes._subplots.AxesSubplot at 0x7ffa85266a90>
unique_members=col.unique("member_id")
list(unique_members["member_id"].values())[1][0:3]
['r100i1p1f1', 'r100i1p2f1', 'r101i1p1f1']

Data Reference Syntax

An atomic Dataset contains all files which cover the entire time span of a single variable of a single simulation. This can be multiple files in one.

The Data Reference Syntax (DRS) is a set of required attributes which uniquely identify and describe a dataset. The DRS usually includes all attributes used in the path templates so that both words are used synonymously. The DRS elements are arranged to a hierarchical path template for CMIP6:

CMIP6: mip_era/activity_id/institution_id/source_id/experiment_id/member_id/table_id/variable_id/grid_label/version

Be careful when browsing through the CMIP6 data tree!

Unique in CMIP6 data hierarchy: - experiment_id (only in one activity_id) - variable_id in table_id : Both combined represent the CMIP Variable - Only one version for one dataset should be published

# Searching for the MIP which defines the experiment 'historical':

cat = col.search(experiment_id="historical")
cat.unique("activity_id")
{'activity_id': {'count': 1, 'values': ['CMIP']}}
# Searching for all tables which contain the variable 'tas':

cat = col.search(variable_id="tas")
cat.unique("table_id")
{'table_id': {'count': 9,
  'values': ['3hr',
   '6hrPlev',
   '6hrPlevPt',
   'AERhr',
   'Amon',
   'CFsubhr',
   'ImonAnt',
   'ImonGre',
   'day']}}

Not Unique in CMIP6 data hierarchy: - institution_id for both source_id + experiment_id ( + member_id )

No requirements for member_id

# Searching for all institution_ids which uses the model 'MPI-ESM1-2-HR' to produce 'ssp585' results:

cat = col.search(source_id="MPI-ESM1-2-HR", experiment_id="ssp585")
cat.unique("institution_id")
{'institution_id': {'count': 2, 'values': ['DKRZ', 'DWD']}}
# Searching for all experiment_ids produced with ESM 'EC-Earth3' and as ensemble member 'r1i1p1f1':

cat = col.search(source_id="EC-Earth3", member_id="r1i1p1f1")
cat.unique("experiment_id")
{'experiment_id': {'count': 12,
  'values': ['amip',
   'dcppA-hindcast',
   'dcppC-amv-neg',
   'dcppC-amv-pos',
   'historical',
   'piClim-aer',
   'piClim-control',
   'piControl',
   'ssp126',
   'ssp245',
   'ssp370',
   'ssp585']}}
# Searching for all valid ensemble member_ids produced with ESM 'EC-Earth3' for experiment 'abrupt-4xCO2'

cat = col.search(source_id="EC-Earth3", experiment_id="abrupt-4xCO2")
cat.unique("member_id")
{'member_id': {'count': 2, 'values': ['r3i1p1f1', 'r8i1p1f1']}}

Do not search for institution_id, table_id and member_id unless you are sure about what you are doing. Instead, begin to search for experiment_id, source_id, variable_id.

How can I find the variables I need? 🔎

  1. Searchfor the matching ``standard_name``

Most of the data in the data pool is compliant to the Climate and Forecast Convention. This defines standard_names, which need to be assigned to variables as a variable attribute inside the data. As a reliable description of the variable, the standard_name is a bridge to the shorter variable identifier name in the data, the so-called short_name. This short name is saved in the data catalogs which can be searched.

  1. Searchfor corresponding ``short_name``s in the CMIP6 data request

E.g., you get many results for air_temperature. Multiple definitions for one ‘physical’ variable like air_temperature exist in CMIP which are mostly specific diagnostics of that variable like tasmin and tasmax. Sometimes, there is output for a specific level given as a variable, e.g. ta500. This can be the case if not all levels are requested for a specific frequency.

Best practice in ESGF

  1. Search for the fitting ``mip_table``

Each mip_table is a combination of requirements for an output variable_id including - frequency - time cell methods (average or instantaneous) - vertical level (e.g. interpolated on pressure levels) - grid - realm (e.g. atmosphere model output or ocean model output)

This requirements are set according to the interest of the MIPs. Variables with the similar requirements are collected in one MIP-table which can be identified by table_id.

The data infrastructure for the DKRZ CDP

In order to tackle the challenges of data provision and dissemination for a 4 PB repository, a state-of-the-art data infrastructure has been developed around that pool. In the following, we highlight three aspects of the data workflow.

You benefit from the DKRZ CDP because

  • its data is standardized and quality controlled 🛂

  • it is a curated, updated, published and catalogized data repository 👩‍🏭

  • it prevents data duplication and downloading into local workspaces which is inefficient, expensive and just a waste of storage resources 🗑

Data quality

CMIP6 data is only available in a common and reliable Data format - No adaptions needed for output of specific models - Makes data interoperable 📠 enabling evaluation software products as, for example, ESMValTool

🏅 CMIP6 data was quality controlled before published with PrePARE

CMIP6 data is transparent about occuring errors - Search the errata data base for origins of suspicious analysis results ⚠

If you find an error, please inform the modeling group. Either via the contact in the citation or, if available, via the contact attriubte in the file.

Data publication

  • Exentended documentation for simulation conducts provided in the ES-Doc data base

  • Persistent Identfier (PIDs) ensure long-term webaccess to dataset information

  • Citation information and DOIs for all published datasets easily retrievable

One method to retrieve a citation from the data is via the attribute further_info_url

import xarray
random_file=xarray.open_dataset(cat.df["path"][0])
random_file.attrs["further_info_url"]
'https://furtherinfo.es-doc.org/CMIP6.EC-Earth-Consortium.EC-Earth3.abrupt-4xCO2.none.r3i1p1f1'

When using data provided in the framework of the DKRZ CMIP Data Pool as basis for a publication, we ask you to add the following text to the Acknowledgements-Section:

“We acknowledge the CMIP6 community for providing the climate model data, retained and globally distributed in the framework of the ESGF. The CMIP6 data of this study were replicated and made available for this study by the DKRZ.”

Upcoming primary publications

⭐ In May 2021, we joyfully expect to fill the remaining 600TB of the 5 PB CDP with primary publications of

  • data for the activity_ids DCPP (hindcasts), DAMIP (Detection and Attribution), VolMIP, FAFMIP and PMIP

  • Ensembles for specific experiments and settings (emission driven simulations)

  • ICON-ESM data

Contacts

This notebook is a collaboration effort by the DM Data Infrastructure team.

🙂 Thank you for your attention!