file version: 06 May 2021
The DKRZ operates a hierarchal storage management system (HSM) used for the storage of all relevant data created and post processed on DKRZ systems. The hardware of the HSM consists of a disc cache and two tape libraries. The primary tape archive is located in the DKRZ building in Hamburg. Selected files are mirrored to the secondary tape archive located at the Max Planck Computing and Data Facility (MPCDF) in Garching. The software installed to operate the HSM is StrongLink. All command-line based user-interaction with the tape archive goes through StrongLink and its command line tool
If you have questions, which are not answered on this page or on the page linked in Further reading, please have a look into our FAQ. If you do not find an answer there, please contact us via firstname.lastname@example.org .
Storage options and quota¶
The amount of data that can be stored in the tape archive per project is limited by the available storage quota of that project. Individual users do not have a quota. Storage space on the HSM is applied for in conjunction with the (bi-)annual application for DKRZ compute and storage resources. There is normal tape archive quota denoted as
arch and quota for long term archival denoted as
docu. Additionally, users might select very important files to be stored twice, i.e. one copy in Hamburg and one copy in Garching. The following table provides an overview:
How to achieve this
Past location (HPSS)
New location (StrongLink)
single copy on tape
1 year after expiration of DKRZ project
default storage type
second copy on separate tape
1 year after expiration of project
store data in specific root namespace (see right column)
long-term storage for reference purpose
10 years after expiration of project
Please contact email@example.com
The output of
slk list contains an extra column which indicates whether a file meant for duplication has already been copied to Garching.
slk: Command Line Tool for HSM access within the DKRZ network¶
slk stores a login token in the home directory of each user (
~/.slk/config.json). The login token is valid for 30 days. By default, this file can only accessed by the respective user (permissions:
600). However, users should be careful when doing things like
chmod 755 * in their home directory. If you assume that your slk login token has been compromized please contact firstname.lastname@example.org .
The StrongLink software comes with a command line tool suite
slk is the user interface to the StrongLink software and allows the user to interact with the HSM. The available commands are:
help: displays the
login: log in to the system with LDAP credentials
archive: copy files to the HSM
chmod: modify permissions of archived files (same as
chmodon the Linux shell)
delete: delete a namespace (and all child objects for the namespace) or a specific file
group: change group ownership of archived files
owner: change ownership of archived files; for admins only
tag: modify metadata of archived files
search: search archived files based on metadata
list: list searched files and some of their metadata (similar to
lson the Linux shell)
retrieve: retrieve files based on search result or based on absolute path
rename: rename a file or a namespace (might be merged with
slk movein future)
StrongLink uses the term “namespace” or “global namespace” (gns). A “namespace” is comparable to a “directory” or “path” on a common file system.
After logging on to the system,
slk does not provide its own shell, but the user still navigates through the local file system, i.e. the parallel file system of mistral.
slk therefore behaves more like a
cp command on the Linux shell. It is also not possible to navigate through the emulated directory structure of the HSM using
Please have a look into StrongLink Command Line Interface (slk) (on doc.dkrz.de) or into the
StrongLink Command Line Interface Guide v3.0 for a detailed description of the individual commands. Alternatively, the sections Switching from pftp to slk and slk Use Cases contain several usage examples.
Metadata: harvesting, manual manipulation and search¶
The StrongLink software reads and extracts extended file metadata from the headers of archived netCDF files (this feature will be available in the production environment (and does work in the test environment)). Users may edit some of these metadata and add further metadata via
slk tag. The metadata which are saved are described in detail on our metadata manual page (Metadata in StrongLink). Additionally, extended metadata for some common file formats such as jpeg are extracted. Harvesting extended metadata from additional file formats used in the context of Earth System modeling is planned to be implemented.
Files can be searched and found based on their metadata. The StrongLink software provides the command line tools
slk list and
slk retrieve to search, list and retrieve files based on their metadata, respectively. Retrieval of files based on their absolute path in the HSM is also possible. Please see the sections Metadata in StrongLink, slk Use Cases and StrongLink Command Line Interface (slk) for details on the usage of
slk in this context.
Packing of data¶
The tape archive delivers its best performance if the files to be archived are sufficiently large. The recommendations on packing developed on the basis of HPSS therefore remain effective for the time being. Like with the HPSS system, the accounting of used quota is done in increments of 1GB per archived file.
packems, which was developed by MPI-M and DKRZ for the HPSS, is planned to be adapted to the new HSM system. The process of packing & archiving of multiple data files to tape and their retrieval is simplified by this package. It consists of three command line programs:
a pack-&-archive tools
a list-archived-content tool
a retrieve-&-unpack tool
module load packems on mistral to load the packems package. Currently, the HPSS-compatible packems version is loaded. It will be switched after the migration from HPSS to StrongLink has been performed successfully. For details on the usage of
packems please have a look into the packems manual.
Backend data handling¶
Just like with the previous HSM system HPSS, the fast disc cache is installed upstream of the tape system. Files selected for archival are first copied to the disc cache and then successively written onto tape. Files selected for retrieval are first copied from tape to the cache and then copied to the specified target locations. The retrieval of files that are still/already stored in the disc cache is considerably faster then the retrieval of files that are located on tape only.
The distribution of the files in the disc cache, primary tape archive and secondary tape archive is automatically controlled by the software StrongLink. The users have no control regarding the storage location of their data.
t is appended to the permissions string of each file in the output of
slk list. The
- indicates that the file is stored in the cache.
- StrongLink Command Line Interface (slk)
- slk Use Cases
- Obtain Access Token
- Search files
- search files by owner / user
- search files larger than 1 MB
- search file with specific value in field of optional metadata schema
- search a file by name
- search files by name using regular expressions
- search files by one of two owners (logical OR)
- search files based on two metadata fields (logical AND)
- save search ID into shell variable
- using shell variables in searches
- List files
- Retrieve files
- tag files (set metadata)
- Change permissions and ownership of files and directories
- Get user/group IDs and names
- Switching from pftp to slk
- External Access
- Metadata in StrongLink
- StrongLink query language