Getting Started with slk

file version: 15 October 2021

current slk version: 3.3.6

Overview

The slk comands behave very similar to known commands on the Linux terminal (POSIX compliant):

  • slk archive and slk retrieve => cp

  • slk delete => rm

  • slk move and slk rename => mv

  • slk chmod => chmod

  • slk group => chgrp

  • slk owner => chown

  • slk search => find

However, there are a few differences which might cause issues if users expect a fully POSIX-compliant slk. Please read slk pitfalls for details. What we know as a folder or directory is called namespace in StrongLink.

Before you can use the slk commands the first time, you must call slk login and login with your DKRZ/mistral/luv login credentials.

Load slk module

The slk is available as module on all mistral nodes. Just do

$ module load slk

to load the most recent slk version. If you wish to use a specific old release (e.g. slk version 3.1.47), please do

$ module load slk/3.1.47

On which nodes to run slk

We suggest running slk archive and slk retrieve on the mistralpp and compute/compute2 nodes. The run time on the mistralpp nodes considerably depends on the activity of other users on these nodes.

Please do not run slk archive and retrieve on the mistral login nodes (mlogin10X) when you archive large amounts of data because slk causes high CPU load and uses much memory.

The available memory per job on the shared nodes is very low. Therefore, slk archive and slk retrieve are slower than on other nodes. The run time can be expected to be two to four times as long as on the mistralpp and compute/compute2 nodes.

Login

Call slk login and use your common DKRZ credentials to log in. You do not login to another shell with slk login (like pftp) but it creates a login token. This login token allows the usage of the other slk commands. It is located in ~/.slk/config.json and is valid for 30 days.

$ slk login

All mistral nodes share the same home directory. Therefore, you need to login only once and you can run it on any mistral node.

Archive a file or directory

slk archive can be used to archive files or directories. A progress bar is printed when slk archive is used in interactive mode (session on mlogin, on mistralpp or in an interactive job via salloc). When slk archive is used in a batch script, no output will be written into the job log. An example slk archive call would be:

$ slk archive /work/bm0146/k204221/some_file.nc /arch/bm0146/k204221/my_data/

slk archive allows the usage of * as wildcard (but not ? or [...]). Add -R to do recursive archival of directories. slk archive works semi-recursive without -R (see slk pitfalls for details).

List content of a namespace

slk list automatically prints its findings in a pagination mode with 25 items per page. That means that you see the first 25 results and have to type Return/Enter to show the next 25 results. The pagination mode is deactivated when the output of slk list is piped (|) into another command – e.g. cat, less or more.

$ slk list /arch/bm0146/k204221/my_data | cat
-rw-rw-rw-- k204221     bm0146          1.2K   27 Mar 2020  borehole_01.nc
-rw-rw-rw-- k204221     bm0146          1.2K   04 Mar 2021  nc3.nc
-rw-rw-rw-- k204221     bm0146          1.2K   04 Mar 2021  nc_k_2.nc
-rw-rw-rw-- k204221     bm0146          4.0M   04 Mar 2021  nc_k_3.nc
-rw-rw-rw-- k204221     bm0146          4.0M   04 Mar 2021  nc_k_4.nc
-rw-rw-rw-- k204221     bm0146         13.1K   08 Dec 2020  small.nc
-rwxrwx-w-- k204221     bm0146        105.5M   08 Nov 2019  small_BPb4-Sl-mT_00062104_00040000000_01040000000.AGM07807972.freeze.nc
-rwxrwx-w-- k204221     bm0146        105.5M   14 Nov 2019  small_BPb4-Sl-mT_00062104_00040000000_01040000000.AGM07807972.nc
-rw-rw-rw-- k204221     bm0146          1.2K   23 Mar 2021  test.nc
-rw-rw-rw-- k204221     bm0146          1.2K   23 Mar 2021  zonk.nc
-rw-rw-rw-- k204221     bm0146          1.2K   28 Jun 2021  some_file.nc
Files: 11

Retrieve a file or namespace

slk retrieve is the counterpart of slk archive. It behaves quite the same as slk archive (progress bar; -R) but the * works only for files and not for namesspaces.

slk retrieve /arch/bm0146/k204221/my_data/nc3.nc /work/bm0146/k204221/results

Move, rename and delete files or namespaces

Move a file from one namespace to another. The file’s name cannot be changed by this command.

$ slk move /arch/bm0146/k204221/my_data/nc_k_2.nc /arch/bm0146/k204221/old_data

Rename a file. The file’s location cannot be changed by this command.

$ slk rename /arch/bm0146/k204221/my_data/nc_k_3.nc a_netcdf_file.nc

Delete a file. If we apply slk delete onto a namespace then it deletes all files in this namespace without confirmation. Files in sub-namespaces are not deleted. To delete the whole namespace with all sub-, subsub-, …-namespaces and their content, please append -R.

$ slk delete /arch/bm0146/k204221/my_data/zonk.nc

Search files

Please see the page Metadata in StrongLink.

Debugging

Check the exit code of slk by reading out $? which prints the exit code of the previously called command.

See the log file in ~/.slk/slk-cli.log for detailed error messages