All Posts

DKRZ CDP Updates July 21

We proudly 🥳 announce that the CDP is extended by new sets of CMIP6 data primarily published at DKRZ. We also published new versions of corrected variables for the MPI-ESM1-2 Earth System Models.

The ensemble set of simulations from the ESM MPI-ESM1-2-HR for the dcppA-hindcast experiment is completed by another 5 realizations (8.5TB). In total, this set consists of about 10 realizations for 60 initialization years in the interval from 1960-2019 resulting in 595 realizations and 31 TB. For each realization, about 100 variables are available for a simulation time of about 10 years.

Read more ...


DKRZ CMIP Data Pool

We proudly announce new publications of model simulations when we publish them at our DKRZ ESGF node. We also keep you updated about the status and the services around the CMIP Data Pool. Find extensive documentions under this link.

Read more ...


Webpack and Django

I recently started to modernize the JavaScript part of a medium sized Django site we run at DKRZ to manage our projects. We have used a version of this site since 2002 and the current Django implementation was initially developed in 2011.

Back then JavaScript was in the form of small scripts embedded into the Django templates. jQuery was used abundantly. All in all, JavaScript was handled very haphazardly because we wanted to get back to working with Python as soon as possible.

Read more ...


Transition from Mistral to Levante for projects

In this post we want to answer a few questions which may arise for project administrators and principal investigators at DKRZ. Some of the dates for requesting new resource allocations will be different in 2021. From 2022 on we will return to the usual schedule.

Your project will be automatically extended with the same resources as for the current allocation period. After July 1, 2021, you can continue working on Mistral as you did before.

Read more ...


SLURM update / Memory use

Slurm config on Mistral has been updated to fix an issue related to memory use.

Prior the update, some Slurm jobs continue consuming the available memory (and even swap) of the allocated node and exceed the allocated memory (set in sbatch or srun). If this occurs, it also affect other jobs/users.

error message

Read more ...


Dask jobqueue on Mistral

According to the official Web site, Dask jobqueue can be used to deploy deploy Dask on job queuing systems like PBS, Slurm, MOAB, SGE, LSF, and HTCondor. Since the queuing system on Mistral is Slurm, we are going to show how to start a Dask cluster there. The idea is simple as described here. The difference is that the workers can be distributed through multiple nodes from the same partition. Using Dask jobqueue will Dask cluster as a Slurm jobs.

In this case, Jupyterhub will often play an interface role and the Dask can use more than the allocated resources to your jupyterhub session (profiles).

Dask jobqueue

Read more ...


Jupyter notebook/lab extensions

Extensions bring additional interesting features to Jupyter*. Depending on the workflow in the notebook, users can install/enable extensions when required. Although is easy to add extensions to both Jupyter notebook an lab, the process can be sometimes annoying based on where jupyter is served from.

In general, installing and enabling extensions in your laptop or using the start-jupyter script is straightforward, especially when the developers well describe their extensions. There should be no restrictions or permissions issues, just follow the instructions.

Extensions configurator

Read more ...


Enable NCL Kernel in Jupyterhub

can’t use NCL (Python) as kernel in Jupyter

This tutorial won’t work

Read more ...


Single jupyter notebooks in containers

you are using singularity containers

you need jupyter notebooks

Read more ...


Spawner options now savable

We introduced a new feature to the preset and advanced options form. This is a nice feature especially for the advanced options form, which contain many fields. You can also reset the options to their initial values by clicking on reset. The form options are saved in the client’s browser every 10 seconds and are not lost if:

the browser crashes

_images/options_saved.gif

Read more ...


New Singularity module deployed

Recently, we deployed a new version of Singularity: 3.6.1. The old version is not available anymore due to many bugs reported by some users.

Errors like these are now fixed:

Read more ...


Connection error in your code?

Connection Error

DownloadWarning

Read more ...


VS Code Remote on Mistral

vs code is your favorite IDE

interested to use the remote extension

https://code.visualstudio.com/assets/docs/remote/remote-overview/architecture.png

Read more ...


CF Python package added to the software tree

According to this link:

The Python cf package is an Earth Science data analysis library that is built on a complete implementation of the CF data model. The cf package implements the CF data model 1 for its internal data structures and so is able to process any CF-compliant dataset. It is not strict about CF-compliance, however, so that partially conformant datasets may be ingested from existing datasets and written to new datasets. This is so that datasets that are partially conformant may nonetheless be modified in memory.

Read more ...


Jupyterhub log file

Each Jupyter notebook is running as a SLUM job on MIstral. By default, stdout and stderr of the SLURM batch job that is spawned by Jupyterhub is written to your HOME directory on the HPC system. In order to make it simple to locate the log file:

if you use the preset options form: the log file is named jupyterhub_slurmspawner_preset_<id>.log.

Read more ...


GLIBC and the container-based workaround

Have you ever tried to install/use a software on Mistral and seen a message like this?

This is for example one of the reasons why PyTorch is not available in our python3 module. Those software packages require a newer version of glibc. Unfortunately, most of Mistral nodes are based on CentOS 6 kernel. To check the version of glibc:

Read more ...


Simple Dask clusters in Jupyterhub

There are multiple ways to create a dask cluster, the following is only an example. Please consult the official documentation. The Dask library is installed and can be found in any of the python3 kernels in jupyterhub. Of course, you can use your own python environment.

The simplest way to create a Dask cluster is to use the distributed module:

Dask Labextension

Read more ...


DKRZ Tech Talks

It is our great pleasure to introduce the DKRZ Tech Talks. In this series of virtual talks we will present services of DKRZ and provide a forum for questions and answers. They will cover technical aspects of the use of our compute systems as well as procedures such as compute time applications and different teams relevant to DKRZ such as our machine learning specialists. The talks will be recorded and uploaded afterwards for further reference.

Go here for more information.

Read more ...


New Jupyterhub server at DKRZ

  • 03 September 2020
  • news

On August 20th, 2020 we deployed a new Jupyterhub server at the DKRZ. The new release has various new features that enhance the user experience.

Link to Jupyterhub server

Read more ...