How to use PROTEUS

Any member of the Carlos I Interuniversity Institute of Theoretical and Computational Physics is entitled to use this computing service. In addition, the iC1 has collaboration and development agreements with other institutions and research centres.

To use PROTEUS, it is necessary to be a member of the Carlos I Interuniversity Institute of Theoretical and Computational Physics.

ACCESS TO THE CLUSTER

_

To work in PROTEUS it is necessary to connect via SSH to its gateway:

# ssh username@proteus.ugr.es

There are no access restrictions (such as a specific IP). However, it has a security filter: if we make a mistake 3 times in a row when typing our user/password, the system perceives it as an attack and blocks the IP. In this case, you will have to contact the administrator to unblock it.

IMPORTANT: PROTEUS is the input machine but NO jobs are executed on it. We have to send our jobs to the queue manager and it will execute them in one of the machines for us.

TRANSFER OF FILES AND DATA

_

You can transfer data from your computers to PROTEUS using any SFTP client. Filezilla is recommended.

Storage

_

We are preparing this information. Please check back in a few days.

Disc cuotes

_

Although in principle there are no disk quotas, directories that have not been modified for more than 3 years will be compressed to save space.

WORKING WITH PROTEUS

_

PROTEUS is geared towards massive computation of applications in a reliable, fault-tolerant, robust, secure and user-friendly way.

A queue manager is in charge of scheduling and executing the jobs requested by users. The gateway to the cluster is through the proteus.ugr.es node, from which jobs are sent to the execution queue.

The queue manager is in charge of choosing which jobs are to be executed (priority order), the machine on which they will be executed (availability of the machine), supervising the execution and collecting results and possible errors.

To compile programs, the latest versions of the GNU (gcc, gfortran) and Intel (icc, ifort) compilers for C, C++ and Fortran are available.

Also available are the openMP and MPI parallel and distributed programming libraries and the BLAS mathematical libraries optimised for the cluster architecture. The software is managed by a module system. Full details can be found on this page.

Any free software can be installed on request. If the software is paid software and the interested party provides the licence, there is no problem to install it, as long as all cluster members can make use of it.

Compile programs

_

To avoid problems with different library versions, it is recommended to compile the source code in PROTEUS.

All PROTEUS processors are Intel x86_64, from 3 families. The environment is configured to generate optimised code for these, as long as the Intel compiler is used. In addition, it is possible to add optimisations by using the general optimisation flags (-O[1,2,3]).

The Intel and GNU compilers for the C/C++ and Fortran languages (icc – ifort | gcc – gfortran) are available.

The use of the Intel compilers is recommended because we have empirically observed that programs obtained with these compilers run faster.

Use of the queuing system

_

The queue manager used in the cluster is SLURM. Further information can be found in the guides and tutorials available at:

Job submissions

_

In order to execute a program, it is necessary to indicate to the queue manager a series of information such as the name of the executable, the resources it needs (CPU, RAM,…), its execution time, etc.

The way to do it is by means of a script where we have to indicate these data. Although its syntax is not too complex, it is necessary to know it and write a script for each program that is sent.

In order to alleviate this extra effort, a small program has been created to create the script and send the work to SLURM.

Scripts for job submission

_
slanzarv

Script that facilitates the submission of jobs to SLURM.

How to use:

$ slanzarv [opciones] programa [argumentos]

Options:

  • –cpus-per-task | -c : numbers of CPUs
  • –mem | -m : memory size in MB
  • –nomail : deactivate the sending of e-mails in case of failure or when the work is finished.
  • –short : send job to short queue
  • –time | -t: maximum execution time (must be less than the partition timeout). Defaults to the partition’s default time. Formats: “minutes”, “minutes:seconds”, “hours:minutes:seconds”, “days-hours”, “days-hours:minutes” and “days-hours:minutes:seconds”.
  • –use-only-faster-nodes : the job can only be executed on the most powerful nodes (metis, hermes or kratos group). It has no effect on the short queue, since this queue only uses the fastest nodes.
  • –family : allows to select the family of CPUs on which the job will be executed. The available options are (ordered from newest to oldest)
    • by cluster: metis, hermes, kratos, calypso
    • by microarchitecture: cascadelake, skylake, broadwell, haswell, westmere-ep, harpertown
  • –min-family : allows to select the oldest family of CPUs to use. Example: -min-family=kratos will be able to use metis, hermes and kratos, but not calypso.
  • –with-modules : comma separated list without spaces of modules needed for execution.

Example of use:

Request 8 CPUs and 2400 MB of RAM for miprogram called with the arguments ‘230 asx’.

slanzarv -c 8 -m 24000 ./myprogram 230 asx

Advanced use:

Additional options can be passed to sbatch. To do so, it is necessary to indicate the end of the options with —

slanzarv -c 4 -m 3400 --contiguous --hold -- ./myprogram 230 asx
sbatch y srun

If slanzarv is not suitable for the job, or is simply preferred, the SLURM commands sbatch and srun can be used directly.

Short summary with their main options from this link.

Status of jobs

_

After a job is submitted, it passes a queue waiting to be executed. The queue manager assigns it a slot with the requested resources according to the available resources and the user’s priority.

The main states a job can be in are:

  • PD (pending)
  • R (running)
  • CA (cancelled)
  • CF(configuring)
  • CG (completing)
  • CD (completed)
  • F (failed)
  • TO (timeout)
  • NF (node failure)
  • RV (revoked)
  • SE (special exit state)

squeue

Displays information on the status of the work.

$ squeue

Display information for a specific job.

$ squeue -j <jobid>

Know the estimated start time for the execution of the pending works.

$ squeue --start

Show jobs in progress.

$ squeue -t RUNNING

Jobs to be done

$ squeue -t PENDING

List the jobs in a given partition.

$ squeue -p <partition name>

Know the status of the user’s jobs:

$ mi_squeue

Partitions and node status

_

Partitions are groups of nodes that have the same characteristics and permissions for use. More info.

sinfo allows to obtain partition and node status information.

$ sinfo

which returns the following columns:

  • PARTITION: partition name
  • AVAIL: whether available (up) or not (down)
  • TIMELIMIT: maximum execution time
  • NODES: number of nodes
  • STATE: status of these nodes. This can be:
    • idle: available
    • alloc: in use
    • mix: part available, part in use
    • resv: reserved
    • drain/drng: not available for technical reasons

Other usage examples:

Display the information of a specific partition:

$ sinfo -p <partitionname>

Show reasons why nodes are: down, have failed or are failing now:

$ sinfo -R

Cancel jobs

_

scancel

Allows a job to be cancelled. The job stops running and disappears from the queue. The files it has generated are kept.

$ scancel <jobid>

Cancel all pending jobs

$ scancel -t PENDING

Other SLURM commands

_

sacct

Consultation of the job history

Standard and error output

_

Slurm redirects standard and error output to files. If the slanzarv script has been used, these files are respectively:

<programa>.<jobid>.out

<programa>.<jobid>.err

By default these files are saved in the send directory. You can modify this behaviour with the following options of the sbatch command​ :

  • –error /path/to/dir/filename
  • –output /path/to/dir/filename

Email notifications

_

It is possible to receive notifications of events and statuses that a program goes through, such as when it starts, when it finishes, if it has failed, etc.

slanzarv by default sends an email when the program has finished or if it has failed. This behaviour can be disabled with the flag –no-mail

$ slanzarv --no-mail miprograma

Slurm provides the following options to control the behaviour of email notifications:

–mail-user: e-mail address to send to

–mail-type: event for which to send the email. These events can be any of the following:

  • NONE
  • BEGIN
  • END
  • FAIL
  • REQUEUE
  • STAGE_OUT
  • ALL (= all of the above)
  • TIME_LIMIT_50 (when 50% of the maximum execution time of the programme has been reached)
  • TIME_LIMIT_80 (ídem 80%)
  • TIME_LIMIT_90 (ídem 90%)
  • TIME_LIMIT (ídem total)

The e-mail address you have used for registration to this service will also be added to a news list, to which changes, news, notices and bugs are sent.

Requirements for software

_

Programs may have different hardware requirements for their execution, such as the processor architecture, the number of processors, the amount of main memory or disk storage, and so on.

This is specified through slanzarv or, if this is insufficient to express all the requirements, through the description file.

The program will wait in queue until the resources it has requested are available. Obviously, the lower the requirements, the easier it will be to make them available at a given time. If they cannot be satisfied, the program will remain waiting in queue indefinitely.

PRIORITIES IN PROTEUS

_

HTCondor

implements an equitable use of cluster resources for all its users. It keeps a history of the resources consumed by each user and, based on this, it calculates the priority of each user so that the users that have been or are using the cluster the most receive a lower priority in favour of those that are using it the least, thus ensuring that in the long run all users can use the same amount of cluster resources.

In case a new job enters the queue and the resources it has requested are in use by another job (either because the cluster is fully in use or because they are specific resources that only have a few nodes), the priorities of the users that are currently running jobs on those resources and that of the owner of the new job are compared. If the latter is higher, the old job goes to standby status (stops executing) and its place is taken by the new one.

It is also possible to distinguish between the priorities of a user’s jobs, giving preference to the ones we want. That is, if we have several jobs and we want some of them to be executed before others. To do this, we can use the command condor_prio

_

Do you need more info?

_