slurm.conf(5) Slurm configuration file slurm.conf(5)
NAME
slurm.conf - Slurm configuration file
DESCRIPTION
slurm.conf is an ASCII file which describes general SLURM configuration information, the nodes to be managed, information about how those
nodes are grouped into partitions, and various scheduling parameters associated with those partitions. This file should be consistent
across all nodes in the cluster.
The file location can be modified at system build time using the DEFAULT_SLURM_CONF parameter or at execution time by setting the
SLURM_CONF environment variable. The SLURM daemons also allow you to override both the built-in and environment-provided location using the
"-f" option on the command line.
The contents of the file are case insensitive except for the names of nodes and partitions. Any text following a "#" in the configuration
file is treated as a comment through the end of that line. The size of each line in the file is limited to 1024 characters. Changes to
the configuration file take effect upon restart of SLURM daemons, daemon receipt of the SIGHUP signal, or execution of the command "scon-
trol reconfigure" unless otherwise noted.
If a line begins with the word "Include" followed by whitespace and then a file name, that file will be included inline with the current
configuration file.
Note on file permissions:
The slurm.conf file must be readable by all users of SLURM, since it is used by many of the SLURM commands. Other files that are defined
in the slurm.conf file, such as log files and job accounting files, may need to be created/owned by the user "SlurmUser" to be successfully
accessed. Use the "chown" and "chmod" commands to set the ownership and permissions appropriately. See the section FILE AND DIRECTORY
PERMISSIONS for information about the various files and directories used by SLURM.
PARAMETERS
The overall configuration parameters available include:
AccountingStorageBackupHost
The name of the backup machine hosting the accounting storage database. If used with the accounting_storage/slurmdbd plugin, this
is where the backup slurmdbd would be running. Only used for database type storage plugins, ignored otherwise.
AccountingStorageEnforce
This controls what level of association-based enforcement to impose on job submissions. Valid options are any combination of asso-
ciations, limits, qos, and wckeys, or all for all things. If limits, qos, or wckeys are set, associations will automatically be
set. In addition, if wckeys is set, TrackWCKey will automatically be set. By enforcing Associations no new job is allowed to run
unless a corresponding association exists in the system. If limits are enforced users can be limited by association to whatever job
size or run time limits are defined. With qos and/or wckeys enforced jobs will not be scheduled unless a valid qos and/or workload
characterization key is specified. When AccountingStorageEnforce is changed, a restart of the slurmctld daemon is required (not
just a "scontrol reconfig").
AccountingStorageHost
The name of the machine hosting the accounting storage database. Only used for database type storage plugins, ignored otherwise.
Also see DefaultStorageHost.
AccountingStorageLoc
The fully qualified file name where accounting records are written when the AccountingStorageType is "accounting_storage/filetxt" or
else the name of the database where accounting records are stored when the AccountingStorageType is a database. Also see Default-
StorageLoc.
AccountingStoragePass
The password used to gain access to the database to store the accounting data. Only used for database type storage plugins, ignored
otherwise. In the case of SLURM DBD (Database Daemon) with MUNGE authentication this can be configured to use a MUNGE daemon
specifically configured to provide authentication between clusters while the default MUNGE daemon provides authentication within a
cluster. In that case, AccountingStoragePass should specify the named port to be used for communications with the alternate MUNGE
daemon (e.g. "/var/run/munge/global.socket.2"). The default value is NULL. Also see DefaultStoragePass.
AccountingStoragePort
The listening port of the accounting storage database server. Only used for database type storage plugins, ignored otherwise. Also
see DefaultStoragePort.
AccountingStorageType
The accounting storage mechanism type. Acceptable values at present include "accounting_storage/filetxt", "accounting_stor-
age/mysql", "accounting_storage/none", "accounting_storage/pgsql", and "accounting_storage/slurmdbd". The "accounting_stor-
age/filetxt" value indicates that accounting records will be written to the file specified by the AccountingStorageLoc parameter.
The "accounting_storage/mysql" value indicates that accounting records will be written to a MySQL database specified by the Account-
ingStorageLoc parameter. The "accounting_storage/pgsql" value indicates that accounting records will be written to a PostgreSQL
database specified by the AccountingStorageLoc parameter. The "accounting_storage/slurmdbd" value indicates that accounting records
will be written to the SLURM DBD, which manages an underlying MySQL or PostgreSQL database. See "man slurmdbd" for more information.
The default value is "accounting_storage/none" and indicates that account records are not maintained. Note: the PostgreSQL plugin
is not complete and should not be used if wanting to use associations. It will however work with basic accounting of jobs and job
steps. If interested in completing, please email slurm-dev@lists.llnl.gov. Also see DefaultStorageType.
AccountingStorageUser
The user account for accessing the accounting storage database. Only used for database type storage plugins, ignored otherwise.
Also see DefaultStorageUser.
AccountingStoreJobComment
If set to "YES" then include the job's comment field in the job complete message sent to the Accounting Storage database. The
default is "YES".
AuthType
The authentication method for communications between SLURM components. Acceptable values at present include "auth/none",
"auth/authd", and "auth/munge". The default value is "auth/munge". "auth/none" includes the UID in each communication, but it is
not verified. This may be fine for testing purposes, but do not use "auth/none" if you desire any security. "auth/authd" indicates
that Brett Chun's authd is to be used (see "http://www.theether.org/authd/" for more information. Note that authd is no longer
actively supported). "auth/munge" indicates that LLNL's MUNGE is to be used (this is the best supported authentication mechanism
for SLURM, see "http://munge.googlecode.com/" for more information). All SLURM daemons and commands must be terminated prior to
changing the value of AuthType and later restarted (SLURM jobs can be preserved).
BackupAddr
The name that BackupController should be referred to in establishing a communications path. This name will be used as an argument to
the gethostbyname() function for identification. For example, "elx0000" might be used to designate the Ethernet address for node
"lx0000". By default the BackupAddr will be identical in value to BackupController.
BackupController
The name of the machine where SLURM control functions are to be executed in the event that ControlMachine fails. This node may also
be used as a compute server if so desired. It will come into service as a controller only upon the failure of ControlMachine and
will revert to a "standby" mode when the ControlMachine becomes available once again. This should be a node name without the full
domain name. I.e., the hostname returned by the gethostname() function cut at the first dot (e.g. use "tux001" rather than
"tux001.my.com"). While not essential, it is recommended that you specify a backup controller. See the RELOCATING CONTROLLERS
section if you change this.
BatchStartTimeout
The maximum time (in seconds) that a batch job is permitted for launching before being considered missing and releasing the alloca-
tion. The default value is 10 (seconds). Larger values may be required if more time is required to execute the Prolog, load user
environment variables (for Moab spawned jobs), or if the slurmd daemon gets paged from memory.
CacheGroups
If set to 1, the slurmd daemon will cache /etc/groups entries. This can improve performance for highly parallel jobs if NIS servers
are used and unable to respond very quickly. The default value is 0 to disable caching group data.
CheckpointType
The system-initiated checkpoint method to be used for user jobs. The slurmctld daemon must be restarted for a change in Checkpoint-
Type to take effect. Supported values presently include:
checkpoint/aix for AIX systems only
checkpoint/blcr Berkeley Lab Checkpoint Restart (BLCR). NOTE: If a file is found at sbin/scch (relative to the SLURM installation
location), it will be executed upon completion of the checkpoint. This can be a script used for managing the
checkpoint files.
checkpoint/none no checkpoint support (default)
checkpoint/ompi OpenMPI (version 1.3 or higher)
ClusterName
The name by which this SLURM managed cluster is known in the accounting database. This is needed distinguish accounting records
when multiple clusters report to the same database.
CompleteWait
The time, in seconds, given for a job to remain in COMPLETING state before any additional jobs are scheduled. If set to zero, pend-
ing jobs will be started as soon as possible. Since a COMPLETING job's resources are released for use by other jobs as soon as the
Epilog completes on each individual node, this can result in very fragmented resource allocations. To provide jobs with the minimum
response time, a value of zero is recommended (no waiting). To minimize fragmentation of resources, a value equal to KillWait plus
two is recommended. In that case, setting KillWait to a small value may be beneficial. The default value of CompleteWait is zero
seconds. The value may not exceed 65533.
ControlAddr
Name that ControlMachine should be referred to in establishing a communications path. This name will be used as an argument to the
gethostbyname() function for identification. For example, "elx0000" might be used to designate the Ethernet address for node
"lx0000". By default the ControlAddr will be identical in value to ControlMachine.
ControlMachine
The short hostname of the machine where SLURM control functions are executed (i.e. the name returned by the command "hostname -s",
use "tux001" rather than "tux001.my.com"). This value must be specified. In order to support some high availability architectures,
multiple hostnames may be listed with comma separators and one ControlAddr must be specified. The high availability system must
insure that the slurmctld daemon is running on only one of these hosts at a time. See the RELOCATING CONTROLLERS section if you
change this.
CryptoType
The cryptographic signature tool to be used in the creation of job step credentials. The slurmctld daemon must be restarted for a
change in CryptoType to take effect. Acceptable values at present include "crypto/munge" and "crypto/openssl". The default value
is "crypto/munge".
DebugFlags
Defines specific subsystems which should provide more detailed event logging. Multiple subsystems can be specified with comma sepa-
rators. Valid subsystems available today (with more to come) include:
Backfill Backfill scheduler details
BGBlockAlgo BlueGene block selection details
BGBlockAlgoDeep BlueGene block selection, more details
BGBlockPick BlueGene block selection for jobs
BGBlockWires BlueGene block wiring (switch state details)
CPU_Bind CPU binding details for jobs and steps
FrontEnd Front end node details
Gres Generic resource details
Gang Gang scheduling details
NO_CONF_HASH Do not log when the slurm.conf files differs between SLURM daemons
Priority Job prioritization
Reservation Advanced reservations
SelectType Resource selection plugin
Steps Slurmctld resource allocation for job steps
Triggers Slurmctld triggers
Wiki Sched/wiki and wiki2 communications
DefMemPerCPU
Default real memory size available per allocated CPU in MegaBytes. Used to avoid over-subscribing memory and causing paging.
DefMemPerCPU would generally be used if individual processors are allocated to jobs (SelectType=select/cons_res). The default value
is 0 (unlimited). Also see DefMemPerNode and MaxMemPerCPU. DefMemPerCPU and DefMemPerNode are mutually exclusive. NOTE: Enforce-
ment of memory limits currently requires enabling of accounting, which samples memory use on a periodic basis (data need not be
stored, just collected).
DefMemPerNode
Default real memory size available per allocated node in MegaBytes. Used to avoid over-subscribing memory and causing paging.
DefMemPerNode would generally be used if whole nodes are allocated to jobs (SelectType=select/linear) and resources are shared
(Shared=yes or Shared=force). The default value is 0 (unlimited). Also see DefMemPerCPU and MaxMemPerNode. DefMemPerCPU and
DefMemPerNode are mutually exclusive. NOTE: Enforcement of memory limits currently requires enabling of accounting, which samples
memory use on a periodic basis (data need not be stored, just collected).
DefaultStorageHost
The default name of the machine hosting the accounting storage and job completion databases. Only used for database type storage
plugins and when the AccountingStorageHost and JobCompHost have not been defined.
DefaultStorageLoc
The fully qualified file name where accounting records and/or job completion records are written when the DefaultStorageType is
"filetxt" or the name of the database where accounting records and/or job completion records are stored when the DefaultStorageType
is a database. Also see AccountingStorageLoc and JobCompLoc.
DefaultStoragePass
The password used to gain access to the database to store the accounting and job completion data. Only used for database type stor-
age plugins, ignored otherwise. Also see AccountingStoragePass and JobCompPass.
DefaultStoragePort
The listening port of the accounting storage and/or job completion database server. Only used for database type storage plugins,
ignored otherwise. Also see AccountingStoragePort and JobCompPort.
DefaultStorageType
The accounting and job completion storage mechanism type. Acceptable values at present include "filetxt", "mysql", "none", "pgsql",
and "slurmdbd". The value "filetxt" indicates that records will be written to a file. The value "mysql" indicates that accounting
records will be written to a mysql database. The default value is "none", which means that records are not maintained. The value
"pgsql" indicates that records will be written to a PostgreSQL database. The value "slurmdbd" indicates that records will be writ-
ten to the SLURM DBD, which maintains its own database. See "man slurmdbd" for more information. Also see AccountingStorageType and
JobCompType.
DefaultStorageUser
The user account for accessing the accounting storage and/or job completion database. Only used for database type storage plugins,
ignored otherwise. Also see AccountingStorageUser and JobCompUser.
DisableRootJobs
If set to "YES" then user root will be prevented from running any jobs. The default value is "NO", meaning user root will be able
to execute jobs. DisableRootJobs may also be set by partition.
EnforcePartLimits
If set to "YES" then jobs which exceed a partition's size and/or time limits will be rejected at submission time. If set to "NO"
then the job will be accepted and remain queued until the partition limits are altered. The default value is "NO". NOTE: If set,
then a job's QOS can not be used to exceed partition limits.
Epilog Fully qualified pathname of a script to execute as user root on every node when a user's job completes (e.g. "/usr/local/slurm/epi-
log"). This may be used to purge files, disable user login, etc. By default there is no epilog. See Prolog and Epilog Scripts for
more information.
EpilogMsgTime
The number of microseconds that the slurmctld daemon requires to process an epilog completion message from the slurmd dameons. This
parameter can be used to prevent a burst of epilog completion messages from being sent at the same time which should help prevent
lost messages and improve throughput for large jobs. The default value is 2000 microseconds. For a 1000 node job, this spreads the
epilog completion messages out over two seconds.
EpilogSlurmctld
Fully qualified pathname of a program for the slurmctld to execute upon termination of a job allocation (e.g.
"/usr/local/slurm/epilog_controller"). The program executes as SlurmUser, which gives it permission to drain nodes and requeue the
job if a failure occurs or cancel the job if appropriate. The program can be used to reboot nodes or perform other work to prepare
resources for use. See Prolog and Epilog Scripts for more information.
FastSchedule
Controls how a node's configuration specifications in slurm.conf are used. If the number of node configuration entries in the con-
figuration file is significantly lower than the number of nodes, setting FastSchedule to 1 will permit much faster scheduling deci-
sions to be made. (The scheduler can just check the values in a few configuration records instead of possibly thousands of node
records.) Note that on systems with hyper-threading, the processor count reported by the node will be twice the actual processor
count. Consider which value you want to be used for scheduling purposes.
1 (default)
Consider the configuration of each node to be that specified in the slurm.conf configuration file and any node with less than
the configured resources will be set DOWN.
0 Base scheduling decisions upon the actual configuration of each individual node except that the node's processor count in
SLURM's configuration must match the actual hardware configuration if SchedulerType=sched/gang or SelectType=select/cons_res
are configured (both of those plugins maintain resource allocation information using bitmaps for the cores in the system and
must remain static, while the node's memory and disk space can be established later).
2 Consider the configuration of each node to be that specified in the slurm.conf configuration file and any node with less than
the configured resources will not be set DOWN. This can be useful for testing purposes.
FirstJobId
The job id to be used for the first submitted to SLURM without a specific requested value. Job id values generated will incremented
by 1 for each subsequent job. This may be used to provide a meta-scheduler with a job id space which is disjoint from the interac-
tive jobs. The default value is 1. Also see MaxJobId
GetEnvTimeout
Used for Moab scheduled jobs only. Controls how long job should wait in seconds for loading the user's environment before attempting
to load it from a cache file. Applies when the srun or sbatch --get-user-env option is used. If set to 0 then always load the user's
environment from the cache file. The default value is 2 seconds.
GresTypes
A comma delimited list of generic resources to be managed. These generic resources may have an associated plugin available to pro-
vide additional functionality. No generic resources are managed by default. Insure this parameter is consistent across all nodes
in the cluster for proper operation. The slurmctld daemon must be restarted for changes to this parameter to become effective.
GroupUpdateForce
If set to a non-zero value, then information about which users are members of groups allowed to use a partition will be updated
periodically, even when there have been no changes to the /etc/group file. Otherwise group member information will be updated peri-
odically only after the /etc/group file is updated The default vaue is 0. Also see the GroupUpdateTime parameter.
GroupUpdateTime
Controls how frequently information about which users are members of groups allowed to use a partition will be updated. The time
interval is given in seconds with a default value of 600 seconds and a maximum value of 4095 seconds. A value of zero will prevent
periodic updating of group membership information. Also see the GroupUpdateForce parameter.
HealthCheckInterval
The interval in seconds between executions of HealthCheckProgram. The default value is zero, which disables execution.
HealthCheckProgram
Fully qualified pathname of a script to execute as user root periodically on all compute nodes that are not in the NOT_RESPONDING
state. This may be used to verify the node is fully operational and DRAIN the node or send email if a problem is detected. Any
action to be taken must be explicitly performed by the program (e.g. execute "scontrol update NodeName=foo State=drain Rea-
son=tmp_file_system_full" to drain a node). The interval is controlled using the HealthCheckInterval parameter. Note that the
HealthCheckProgram will be executed at the same time on all nodes to minimize its impact upon parallel programs. This program is
will be killed if it does not terminate normally within 60 seconds. By default, no program will be executed.
InactiveLimit
The interval, in seconds, after which a non-responsive job allocation command (e.g. srun or salloc) will result in the job being
terminated. If the node on which the command is executed fails or the command abnormally terminates, this will terminate its job
allocation. This option has no effect upon batch jobs. When setting a value, take into consideration that a debugger using srun to
launch an application may leave the srun command in a stopped state for extended periods of time. This limit is ignored for jobs
running in partitions with the RootOnly flag set (the scheduler running as root will be responsible for the job). The default value
is unlimited (zero) and may not exceed 65533 seconds.
JobAcctGatherType
The job accounting mechanism type. Acceptable values at present include "jobacct_gather/aix" (for AIX operating system),
"jobacct_gather/linux" (for Linux operating system) and "jobacct_gather/none" (no accounting data collected). The default value is
"jobacct_gather/none". In order to use the sstat tool, "jobacct_gather/aix" or "jobacct_gather/linux" must be configured.
JobAcctGatherFrequency
The job accounting sampling interval. For jobacct_gather/none this parameter is ignored. For jobacct_gather/aix and
jobacct_gather/linux the parameter is a number is seconds between sampling job state. The default value is 30 seconds. A value of
zero disables real the periodic job sampling and provides accounting information only on job termination (reducing SLURM interfer-
ence with the job). Smaller (non-zero) values have a greater impact upon job performance, but a value of 30 seconds is not likely
to be noticeable for applications having less than 10,000 tasks. Users can override this value on a per job basis using the
--acctg-freq option when submitting the job.
JobCheckpointDir
Specifies the default directory for storing or reading job checkpoint information. The data stored here is only a few thousand bytes
per job and includes information needed to resubmit the job request, not job's memory image. The directory must be readable and
writable by SlurmUser, but not writable by regular users. The job memory images may be in a different location as specified by
--checkpoint-dir option at job submit time or scontrol's ImageDir option.
JobCompHost
The name of the machine hosting the job completion database. Only used for database type storage plugins, ignored otherwise. Also
see DefaultStorageHost.
JobCompLoc
The fully qualified file name where job completion records are written when the JobCompType is "jobcomp/filetxt" or the database
where job completion records are stored when the JobCompType is a database. Also see DefaultStorageLoc.
JobCompPass
The password used to gain access to the database to store the job completion data. Only used for database type storage plugins,
ignored otherwise. Also see DefaultStoragePass.
JobCompPort
The listening port of the job completion database server. Only used for database type storage plugins, ignored otherwise. Also see
DefaultStoragePort.
JobCompType
The job completion logging mechanism type. Acceptable values at present include "jobcomp/none", "jobcomp/filetxt", "jobcomp/mysql",
"jobcomp/pgsql", and "jobcomp/script"". The default value is "jobcomp/none", which means that upon job completion the record of the
job is purged from the system. If using the accounting infrastructure this plugin may not be of interest since the information here
is redundant. The value "jobcomp/filetxt" indicates that a record of the job should be written to a text file specified by the Job-
CompLoc parameter. The value "jobcomp/mysql" indicates that a record of the job should be written to a mysql database specified by
the JobCompLoc parameter. The value "jobcomp/pgsql" indicates that a record of the job should be written to a PostgreSQL database
specified by the JobCompLoc parameter. The value "jobcomp/script" indicates that a script specified by the JobCompLoc parameter is
to be executed with environment variables indicating the job information.
JobCompUser
The user account for accessing the job completion database. Only used for database type storage plugins, ignored otherwise. Also
see DefaultStorageUser.
JobCredentialPrivateKey
Fully qualified pathname of a file containing a private key used for authentication by SLURM daemons. This parameter is ignored if
CryptoType=crypto/munge.
JobCredentialPublicCertificate
Fully qualified pathname of a file containing a public key used for authentication by SLURM daemons. This parameter is ignored if
CryptoType=crypto/munge.
JobFileAppend
This option controls what to do if a job's output or error file exist when the job is started. If JobFileAppend is set to a value
of 1, then append to the existing file. By default, any existing file is truncated.
JobRequeue
This option controls what to do by default after a node failure. If JobRequeue is set to a value of 1, then any batch job running
on the failed node will be requeued for execution on different nodes. If JobRequeue is set to a value of 0, then any job running on
the failed node will be terminated. Use the sbatch --no-requeue or --requeue option to change the default behavior for individual
jobs. The default value is 1.
JobSubmitPlugins
A comma delimited list of job submission plugins to be used. The specified plugins will be executed in the order listed. These are
intended to be site-specific plugins which can be used to set default job parameters and/or logging events. Sample plugins avail-
able in the distribution include "cnode", "defaults", "logging", "lua", and "partition". For examples of use, see the SLURM code in
"src/plugins/job_submit" and "contribs/lua/job_submit*.lua" then modify the code to satisfy your needs. No job submission plugins
are used by default.
KillOnBadExit
If set to 1, the job will be terminated immediately when one of the processes is crashed or aborted. With the default value of 0, if
one of the processes is crashed or aborted the other processes will continue to run. The user can override this configuration param-
eter by using srun's -K, --kill-on-bad-exit.
KillWait
The interval, in seconds, given to a job's processes between the SIGTERM and SIGKILL signals upon reaching its time limit. If the
job fails to terminate gracefully in the interval specified, it will be forcibly terminated. The default value is 30 seconds. The
value may not exceed 65533.
Licenses
Specification of licenses (or other resources available on all nodes of the cluster) which can be allocated to jobs. License names
can optionally be followed by an asterisk and count with a default count of one. Multiple license names should be comma separated
(e.g. "Licenses=foo*4,bar"). Note that SLURM prevents jobs from being scheduled if their required license specification is not
available. SLURM does not prevent jobs from using licenses that are not explicitly listed in the job submission specification.
MailProg
Fully qualified pathname to the program used to send email per user request. The default value is "/usr/bin/mail".
MaxJobCount
The maximum number of jobs SLURM can have in its active database at one time. Set the values of MaxJobCount and MinJobAge to insure
the slurmctld daemon does not exhaust its memory or other resources. Once this limit is reached, requests to submit additional jobs
will fail. The default value is 10000 jobs. This value may not be reset via "scontrol reconfig". It only takes effect upon restart
of the slurmctld daemon.
MaxJobId
The maximum job id to be used for jobs submitted to SLURM without a specific requested value. Job id values generated will incre-
mented by 1 for each subsequent job. This may be used to provide a meta-scheduler with a job id space which is disjoint from the
interactive jobs. Once MaxJobId is reached, the next job will be assigned FirstJobId. The default value is 4294901760
(0xffff0000). Also see FirstJobId.
MaxMemPerCPU
Maximum real memory size available per allocated CPU in MegaBytes. Used to avoid over-subscribing memory and causing paging.
MaxMemPerCPU would generally be used if individual processors are allocated to jobs (SelectType=select/cons_res). The default value
is 0 (unlimited). Also see DefMemPerCPU and MaxMemPerNode. MaxMemPerCPU and MaxMemPerNode are mutually exclusive. NOTE: Enforce-
ment of memory limits currently requires enabling of accounting, which samples memory use on a periodic basis (data need not be
stored, just collected).
MaxMemPerNode
Maximum real memory size available per allocated node in MegaBytes. Used to avoid over-subscribing memory and causing paging.
MaxMemPerNode would generally be used if whole nodes are allocated to jobs (SelectType=select/linear) and resources are shared
(Shared=yes or Shared=force). The default value is 0 (unlimited). Also see DefMemPerNode and MaxMemPerCPU. MaxMemPerCPU and
MaxMemPerNode are mutually exclusive. NOTE: Enforcement of memory limits currently requires enabling of accounting, which samples
memory use on a periodic basis (data need not be stored, just collected).
MaxStepCount
The maximum number of steps that any job can initiate. This parameter is intended to limit the effect of bad batch scripts. The
default value is 40000 steps.
MaxTasksPerNode
Maximum number of tasks SLURM will allow a job step to spawn on a single node. The default MaxTasksPerNode is 128.
MessageTimeout
Time permitted for a round-trip communication to complete in seconds. Default value is 10 seconds. For systems with shared nodes,
the slurmd daemon could be paged out and necessitate higher values.
MinJobAge
The minimum age of a completed job before its record is purged from SLURM's active database. Set the values of MaxJobCount and Min-
JobAge to insure the slurmctld daemon does not exhaust its memory or other resources. The default value is 300 seconds. A value of
zero prevents any job record purging. May not exceed 65533.
MpiDefault
Identifies the default type of MPI to be used. Srun may override this configuration parameter in any case. Currently supported
versions include: lam, mpich1_p4, mpich1_shmem, mpichgm, mpichmx, mvapich, none (default, which works for many other versions of
MPI) and openmpi. More information about MPI use is available here <http://www.schedmd.com/slurmdocs/mpi_guide.html>.
MpiParams
MPI parameters. Used to identify ports used by OpenMPI only and the input format is "ports=12000-12999" to identify a range of com-
munication ports to be used.
OverTimeLimit
Number of minutes by which a job can exceed its time limit before being canceled. The configured job time limit is treated as a
soft limit. Adding OverTimeLimit to the soft limit provides a hard limit, at which point the job is canceled. This is particularly
useful for backfill scheduling, which bases upon each job's soft time limit. The default value is zero. May not exceed exceed
65533 minutes. A value of "UNLIMITED" is also supported.
PluginDir
Identifies the places in which to look for SLURM plugins. This is a colon-separated list of directories, like the PATH environment
variable. The default value is "/usr/local/lib/slurm".
PlugStackConfig
Location of the config file for SLURM stackable plugins that use the Stackable Plugin Architecture for Node job (K)control (SPANK).
This provides support for a highly configurable set of plugins to be called before and/or after execution of each task spawned as
part of a user's job step. Default location is "plugstack.conf" in the same directory as the system slurm.conf. For more informa-
tion on SPANK plugins, see the spank(8) manual.
PreemptMode
Enables gang scheduling and/or controls the mechanism used to preempt jobs. When the PreemptType parameter is set to enable preemp-
tion, the PreemptMode selects the mechanism used to preempt the lower priority jobs. The GANG option is used to enable gang sched-
uling independent of whether preemption is enabled (the PreemptType setting). The GANG option can be specified in addition to a
PreemptMode setting with the two options comma separated. The SUSPEND option requires that gang scheduling be enabled (i.e, "Pre-
emptMode=SUSPEND,GANG").
OFF is the default value and disables job preemption and gang scheduling. This is the only option compatible with Sched-
ulerType=sched/wiki or SchedulerType=sched/wiki2 (used by Maui and Moab respectively, which provide their own job pre-
emption functionality).
CANCEL always cancel the job.
CHECKPOINT preempts jobs by checkpointing them (if possible) or canceling them.
GANG enables gang scheduling (time slicing) of jobs in the same partition. NOTE: Gang scheduling is performed independently
for each partition, so configuring partitions with overlapping nodes and gang scheduling is generally not recommended.
REQUEUE preempts jobs by requeuing them (if possible) or canceling them.
SUSPEND preempts jobs by suspending them. A suspended job will resume execution once the high priority job preempting it com-
pletes. The SUSPEND may only be used with the GANG option (the gang scheduler module performs the job resume operation)
and with PreemptType=preempt/partition_prio (the logic to suspend and resume jobs current only has the data structures
to support partitions).
PreemptType
This specifies the plugin used to identify which jobs can be preempted in order to start a pending job.
preempt/none
Job preemption is disabled. This is the default.
preempt/partition_prio
Job preemption is based upon partition priority. Jobs in higher priority partitions (queues) may preempt jobs from lower
priority partitions.
preempt/qos
Job preemption rules are specified by Quality Of Service (QOS) specifications in the SLURM database a database. This is not
compatible with PreemptMode=OFF or PreemptMode=SUSPEND (i.e. preempted jobs must be removed from the resources).
PriorityDecayHalfLife
This controls how long prior resource use is considered in determining how over- or under-serviced an association is (user, bank
account and cluster) in determining job priority. If set to 0 no decay will be applied. This is helpful if you want to enforce
hard time limits per association. If set to 0 PriorityUsageResetPeriod must be set to some interval. Applicable only if Priority-
Type=priority/multifactor. The unit is a time string (i.e. min, hr:min:00, days-hr:min:00, or days-hr). The default value is 7-0
(7 days).
PriorityCalcPeriod
The period of time in minutes in which the half-life decay will be re-calculated. Applicable only if PriorityType=priority/multi-
factor. The default value is 5 (minutes).
PriorityFavorSmall
Specifies that small jobs should be given preferential scheduling priority. Applicable only if PriorityType=priority/multifactor.
Supported values are "YES" and "NO". The default value is "NO".
PriorityMaxAge
Specifies the job age which will be given the maximum age factor in computing priority. For example, a value of 30 minutes would
result in all jobs over 30 minutes old would get the same age-based priority. Applicable only if PriorityType=priority/multifactor.
The unit is a time string (i.e. min, hr:min:00, days-hr:min:00, or days-hr). The default value is 7-0 (7 days).
PriorityUsageResetPeriod
At this interval the usage of associations will be reset to 0. This is used if you want to enforce hard limits of time usage per
association. If PriorityDecayHalfLife is set to be 0 no decay will happen and this is the only way to reset the usage accumulated
by running jobs. By default this is turned off and it is advised to use the PriorityDecayHalfLife option to avoid not having any-
thing running on your cluster, but if your schema is set up to only allow certain amounts of time on your system this is the way to
do it. Applicable only if PriorityType=priority/multifactor.
NONE Never clear historic usage. The default value.
NOW Clear the historic usage now. Executed at startup and reconfiguration time.
DAILY Cleared every day at midnight.
WEEKLY Cleared every week on Sunday at time 00:00.
MONTHLY Cleared on the first day of each month at time 00:00.
QUARTERLY Cleared on the first day of each quarter at time 00:00.
YEARLY Cleared on the first day of each year at time 00:00.
PriorityType
This specifies the plugin to be used in establishing a job's scheduling priority. Supported values are "priority/basic" (jobs are
prioritized by order of arrival, also suitable for sched/wiki and sched/wiki2) and "priority/multifactor" (jobs are prioritized
based upon size, age, fair-share of allocation, etc). The default value is "priority/basic".
PriorityWeightAge
An integer value that sets the degree to which the queue wait time component contributes to the job's priority. Applicable only if
PriorityType=priority/multifactor. The default value is 0.
PriorityWeightFairshare
An integer value that sets the degree to which the fair-share component contributes to the job's priority. Applicable only if Pri-
orityType=priority/multifactor. The default value is 0.
PriorityWeightJobSize
An integer value that sets the degree to which the job size component contributes to the job's priority. Applicable only if Priori-
tyType=priority/multifactor. The default value is 0.
PriorityWeightPartition
An integer value that sets the degree to which the node partition component contributes to the job's priority. Applicable only if
PriorityType=priority/multifactor. The default value is 0.
PriorityWeightQOS
An integer value that sets the degree to which the Quality Of Service component contributes to the job's priority. Applicable only
if PriorityType=priority/multifactor. The default value is 0.
PrivateData
This controls what type of information is hidden from regular users. By default, all information is visible to all users. User
SlurmUser and root can always view all information. Multiple values may be specified with a comma separator. Acceptable values
include:
accounts
(NON-SLURMDBD ACCOUNTING ONLY) prevents users from viewing any account definitions unless they are coordinators of them.
jobs prevents users from viewing jobs or job steps belonging to other users. (NON-SLURMDBD ACCOUNTING ONLY) prevents users from
viewing job records belonging to other users unless they are coordinators of the association running the job when using
sacct.
nodes prevents users from viewing node state information.
partitions
prevents users from viewing partition state information.
reservations
prevents regular users from viewing reservations.
usage (NON-SLURMDBD ACCOUNTING ONLY) prevents users from viewing usage of any other user. This applies to sreport.
users (NON-SLURMDBD ACCOUNTING ONLY) prevents users from viewing information of any user other than themselves, this also makes it
so users can only see associations they deal with. Coordinators can see associations of all users they are coordinator of,
but can only see themselves when listing users.
ProctrackType
Identifies the plugin to be used for process tracking. The slurmd daemon uses this mechanism to identify all processes which are
children of processes it spawns for a user job. The slurmd daemon must be restarted for a change in ProctrackType to take effect.
NOTE: "proctrack/linuxproc" and "proctrack/pgid" can fail to identify all processes associated with a job since processes can become
a child of the init process (when the parent process terminates) or change their process group. To reliably track all processes,
one of the other mechanisms utilizing kernel modifications is preferable. NOTE: "proctrack/linuxproc" is not compatible with
"switch/elan." Acceptable values at present include:
proctrack/aix which uses an AIX kernel extension and is the default for AIX systems
proctrack/cgroup which uses linux cgroups to constrain and track processes. NOTE: see "man cgroup.conf" for configuration
details
proctrack/linuxproc which uses linux process tree using parent process IDs
proctrack/lua which uses a site-specific LUA script to track processes
proctrack/rms which uses Quadrics kernel patch and is the default if "SwitchType=switch/elan"
proctrack/sgi_job which uses SGI's Process Aggregates (PAGG) kernel module, see http://oss.sgi.com/projects/pagg/ for more infor-
mation
proctrack/pgid which uses process group IDs and is the default for all other systems
Prolog Fully qualified pathname of a program for the slurmd to execute whenever it is asked to run a job step from a new job allocation
(e.g. "/usr/local/slurm/prolog"). The slurmd executes the script before starting the first job step. This may be used to purge
files, enable user login, etc. By default there is no prolog. Any configured script is expected to complete execution quickly (in
less time than MessageTimeout). See Prolog and Epilog Scripts for more information.
PrologSlurmctld
Fully qualified pathname of a program for the slurmctld to execute before granting a new job allocation (e.g.
"/usr/local/slurm/prolog_controller"). The program executes as SlurmUser, which gives it permission to drain nodes and requeue the
job if a failure occurs or cancel the job if appropriate. The program can be used to reboot nodes or perform other work to prepare
resources for use. While this program is running, the nodes associated with the job will be have a POWER_UP/CONFIGURING flag set in
their state, which can be readily viewed. A non-zero exit code will result in the job being requeued (where possible) or killed.
See Prolog and Epilog Scripts for more information.
PropagatePrioProcess
Controls the scheduling priority (nice value) of user spawned tasks.
0 The tasks will inherit the scheduling priority from the slurm daemon. This is the default value.
1 The tasks will inherit the scheduling priority of the command used to submit them (e.g. srun or sbatch). Unless the job is
submitted by user root, the tasks will have a scheduling priority no higher than the slurm daemon spawning them.
2 The tasks will inherit the scheduling priority of the command used to submit them (e.g. srun or sbatch) with the restriction
that their nice value will always be one higher than the slurm daemon (i.e. the tasks scheduling priority will be lower than
the slurm daemon).
PropagateResourceLimits
A list of comma separated resource limit names. The slurmd daemon uses these names to obtain the associated (soft) limit values
from the users process environment on the submit node. These limits are then propagated and applied to the jobs that will run on
the compute nodes. This parameter can be useful when system limits vary among nodes. Any resource limits that do not appear in the
list are not propagated. However, the user can override this by specifying which resource limits to propagate with the srun com-
mands "--propagate" option. If neither of the 'propagate resource limit' parameters are specified, then the default action is to
propagate all limits. Only one of the parameters, either PropagateResourceLimits or PropagateResourceLimitsExcept, may be speci-
fied. The following limit names are supported by SLURM (although some options may not be supported on some systems):
ALL All limits listed below
NONE No limits listed below
AS The maximum address space for a process
CORE The maximum size of core file
CPU The maximum amount of CPU time
DATA The maximum size of a process's data segment
FSIZE The maximum size of files created. Note that if the user sets FSIZE to less than the current size of the slurmd.log, job
launches will fail with a 'File size limit exceeded' error.
MEMLOCK The maximum size that may be locked into memory
NOFILE The maximum number of open files
NPROC The maximum number of processes available
RSS The maximum resident set size
STACK The maximum stack size
PropagateResourceLimitsExcept
A list of comma separated resource limit names. By default, all resource limits will be propagated, (as described by the Propa-
gateResourceLimits parameter), except for the limits appearing in this list. The user can override this by specifying which
resource limits to propagate with the srun commands "--propagate" option. See PropagateResourceLimits above for a list of valid
limit names.
ResumeProgram
SLURM supports a mechanism to reduce power consumption on nodes that remain idle for an extended period of time. This is typically
accomplished by reducing voltage and frequency or powering the node down. ResumeProgram is the program that will be executed when a
node in power save mode is assigned work to perform. For reasons of reliability, ResumeProgram may execute more than once for a
node when the slurmctld daemon crashes and is restarted. If ResumeProgram is unable to restore a node to service, it should requeue
any node associated with the node and set the node state to DRAIN. The program executes as SlurmUser. The argument to the program
will be the names of nodes to be removed from power savings mode (using SLURM's hostlist expression format). By default no program
is run. Related configuration options include ResumeTimeout, ResumeRate, SuspendRate, SuspendTime, SuspendTimeout, SuspendProgram,
SuspendExcNodes, and SuspendExcParts. More information is available at the SLURM web site ( http://www.schedmd.com/slurm-
docs/power_save.html ).
ResumeRate
The rate at which nodes in power save mode are returned to normal operation by ResumeProgram. The value is number of nodes per
minute and it can be used to prevent power surges if a large number of nodes in power save mode are assigned work at the same time
(e.g. a large job starts). A value of zero results in no limits being imposed. The default value is 300 nodes per minute. Related
configuration options include ResumeTimeout, ResumeProgram, SuspendRate, SuspendTime, SuspendTimeout, SuspendProgram, SuspendExcN-
odes, and SuspendExcParts.
ResumeTimeout
Maximum time permitted (in second) between when a node is resume request is issued and when the node is actually available for use.
Nodes which fail to respond in this time frame may be marked DOWN and the jobs scheduled on the node requeued. The default value is
60 seconds. Related configuration options include ResumeProgram, ResumeRate, SuspendRate, SuspendTime, SuspendTimeout, SuspendPro-
gram, SuspendExcNodes and SuspendExcParts. More information is available at the SLURM web site ( http://www.schedmd.com/slurm-
docs/power_save.html ).
ResvOverRun
Describes how long a job already running in a reservation should be permitted to execute after the end time of the reservation has
been reached. The time period is specified in minutes and the default value is 0 (kill the job immediately). The value may not
exceed 65533 minutes, although a value of "UNLIMITED" is supported to permit a job to run indefinitely after its reservation is ter-
minated.
ReturnToService
Controls when a DOWN node will be returned to service. The default value is 0. Supported values include
0 A node will remain in the DOWN state until a system administrator explicitly changes its state (even if the slurmd daemon regis-
ters and resumes communications).
1 A DOWN node will become available for use upon registration with a valid configuration only if it was set DOWN due to being
non-responsive. If the node was set DOWN for any other reason (low memory, prolog failure, epilog failure, unexpected reboot,
etc.), its state will not automatically be changed.
2 A DOWN node will become available for use upon registration with a valid configuration. The node could have been set DOWN for
any reason. (Disabled on Cray systems.)
SallocDefaultCommand
Normally, salloc(1) will run the user's default shell when a command to execute is not specified on the salloc command line. If
SallocDefaultCommand is specified, salloc will instead run the configured command. The command is passed to '/bin/sh -c', so shell
metacharacters are allowed, and commands with multiple arguments should be quoted. For instance:
SallocDefaultCommand = "$SHELL"
would run the shell in the user's $SHELL environment variable. and
SallocDefaultCommand = "xterm -T Job_$SLURM_JOB_ID"
would run xterm with the title set to the SLURM jobid.
SchedulerParameters
The interpretation of this parameter varies by SchedulerType. Multiple options may be comma separated.
default_queue_depth=#
The default number of jobs to attempt scheduling (i.e. the queue depth) when a running job completes or other routine actions
occur. The full queue will be tested on a less frequent basis. The default value is 100. In the case of large clusters (more
than 1000 nodes), configuring a relatively small value may be desirable.
defer Setting this option will avoid attempting to schedule each job individually at job submit time, but defer it until a later
time when scheduling multiple jobs simultaneously may be possible. This option may improve system responsiveness when large
numbers of jobs (many hundreds) are submitted at the same time, but it will delay the initiation time of individual jobs.
Also see default_queue_depth above.
bf_interval=#
The number of seconds between iterations. Higher values result in less overhead and better responsiveness. The default
value is 30 seconds. This option applies only to SchedulerType=sched/backfill.
bf_resolution=#
The number of seconds in the resolution of data maintained about when jobs begin and end. Higher values result in less over-
head and better responsiveness. The default value is 60 seconds. This option applies only to SchedulerType=sched/backfill.
bf_window=#
The number of minutes into the future to look when considering jobs to schedule. Higher values result in more overhead and
less responsiveness. The default value is 1440 minutes (one day). This option applies only to SchedulerType=sched/backfill.
max_job_bf=#
The maximum number of jobs to attempt backfill scheduling for (i.e. the queue depth). Higher values result in more overhead
and less responsiveness. Until an attempt is made to backfill schedule a job, its expected initiation time value will not be
set. The default value is 50. In the case of large clusters (more than 1000 nodes) configured with Select-
Type=select/cons_res, configuring a relatively small value may be desirable. This option applies only to Scheduler-
Type=sched/backfill.
max_switch_wait=#
Maximum number of seconds that a job can delay execution waiting for the specified desired switch count. The default value is
60 seconds.
SchedulerPort
The port number on which slurmctld should listen for connection requests. This value is only used by the Maui Scheduler (see Sched-
ulerType). The default value is 7321.
SchedulerRootFilter
Identifies whether or not RootOnly partitions should be filtered from any external scheduling activities. If set to 0, then RootOnly
partitions are treated like any other partition. If set to 1, then RootOnly partitions are exempt from any external scheduling
activities. The default value is 1. Currently only used by the built-in backfill scheduling module "sched/backfill" (see Scheduler-
Type).
SchedulerTimeSlice
Number of seconds in each time slice when gang scheduling is enabled (PreemptMode=GANG). The value must be between 5 seconds and
65533 seconds. The default value is 30 seconds.
SchedulerType
Identifies the type of scheduler to be used. Note the slurmctld daemon must be restarted for a change in scheduler type to become
effective (reconfiguring a running daemon has no effect for this parameter). The scontrol command can be used to manually change
job priorities if desired. Acceptable values include:
sched/builtin
for the built-in FIFO (First In First Out) scheduler. This is the default.
sched/backfill
for a backfill scheduling module to augment the default FIFO scheduling. Backfill scheduling will initiate lower-priority
jobs if doing so does not delay the expected initiation time of any higher priority job. Effectiveness of backfill schedul-
ing is dependent upon users specifying job time limits, otherwise all jobs will have the same time limit and backfilling is
impossible. Note documentation for the SchedulerParameters option above.
sched/gang
Defunct option. See PreemptType and PreemptMode options.
sched/hold
to hold all newly arriving jobs if a file "/etc/slurm.hold" exists otherwise use the built-in FIFO scheduler
sched/wiki
for the Wiki interface to the Maui Scheduler
sched/wiki2
for the Wiki interface to the Moab Cluster Suite
SelectType
Identifies the type of resource selection algorithm to be used. Changing this value can only be done by restarting the slurmctld
daemon and will result in the loss of all job information (running and pending) since the job state save format used by each plugin
is different. Acceptable values include
select/linear
for allocation of entire nodes assuming a one-dimensional array of nodes in which sequentially ordered nodes are preferable.
This is the default value for non-BlueGene systems.
select/cons_res
The resources within a node are individually allocated as consumable resources. Note that whole nodes can be allocated to
jobs for selected partitions by using the Shared=Exclusive option. See the partition Shared parameter for more information.
select/bluegene
for a three-dimensional BlueGene system. The default value is "select/bluegene" for BlueGene systems.
select/cray
for a Cray system. The default value is "select/cray" for all Cray systems.
SelectTypeParameters
The permitted values of SelectTypeParameters depend upon the configured value of SelectType. SelectType=select/bluegene supports no
SelectTypeParameters. The only supported option for SelectType=select/linear are CR_ONE_TASK_PER_CORE and CR_Memory, which treats
memory as a consumable resource and prevents memory over subscription with job preemption or gang scheduling. The following values
are supported for SelectType=select/cons_res:
CR_CPU CPUs are consumable resources. There is no notion of sockets, cores or threads; do not define those values in the node spec-
ification. If these are defined, unexpected results will happen when hyper-threading is enabled CPUs= should be used
instead. On a multi-core system, each core will be considered a CPU. On a multi-core and hyper-threaded system, each thread
will be considered a CPU. On single-core systems, each CPUs will be considered a CPU.
CR_CPU_Memory
CPUs and memory are consumable resources. There is no notion of sockets, cores or threads; do not define those values in the
node specification. If these are defined, unexpected results will happen when hyper-threading is enabled CPUs= should be
used instead. Setting a value for DefMemPerCPU is strongly recommended.
CR_Core
Cores are consumable resources. On nodes with hyper-threads, each thread is counted as a CPU to satisfy a job's resource
requirement, but multiple jobs are not allocated threads on the same core.
CR_Core_Memory
Cores and memory are consumable resources. On nodes with hyper-threads, each thread is counted as a CPU to satisfy a job's
resource requirement, but multiple jobs are not allocated threads on the same core. Setting a value for DefMemPerCPU is
strongly recommended.
CR_ONE_TASK_PER_CORE
Allocate one task per core by default. Without this option, by default one task will be allocated per thread on nodes with
more than one ThreadsPerCore configured.
CR_CORE_DEFAULT_DIST_BLOCK
Allocate cores using block distribution by default. This default behavior can be overridden specifying a particular "-m"
parameter with srun/salloc/sbatch. Without this option, cores will be allocated cyclicly across the sockets.
CR_Socket
Sockets are consumable resources. On nodes with multiple cores, each core or thread is counted as a CPU to satisfy a job's
resource requirement, but multiple jobs are not allocated resources on the same socket. Note that jobs requesting one CPU
will only be allocated that one CPU, but no other job will share the socket.
CR_Socket_Memory
Memory and sockets are consumable resources. On nodes with multiple cores, each core or thread is counted as a CPU to sat-
isfy a job's resource requirement, but multiple jobs are not allocated resources on the same socket. Note that jobs request-
ing one CPU will only be allocated that one CPU, but no other job will share the socket. Setting a value for DefMemPerCPU is
strongly recommended.
CR_Memory
Memory is a consumable resource. NOTE: This implies Shared=YES or Shared=FORCE for all partitions. Setting a value for
DefMemPerCPU is strongly recommended.
SlurmUser
The name of the user that the slurmctld daemon executes as. For security purposes, a user other than "root" is recommended. This
user must exist on all nodes of the cluster for authentication of communications between SLURM components. The default value is
"root".
SlurmdUser
The name of the user that the slurmd daemon executes as. This user must exist on all nodes of the cluster for authentication of
communications between SLURM components. The default value is "root".
SlurmctldDebug
The level of detail to provide slurmctld daemon's logs. Values from 0 to 9 are legal, with `0' being "quiet" operation and `9'
being insanely verbose. The default value is 3.
SlurmctldLogFile
Fully qualified pathname of a file into which the slurmctld daemon's logs are written. The default value is none (performs logging
via syslog).
See the section LOGGING if a pathname is specified.
SlurmctldPidFile
Fully qualified pathname of a file into which the slurmctld daemon may write its process id. This may be used for automated signal
processing. The default value is "/var/run/slurmctld.pid".
SlurmctldPort
The port number that the SLURM controller, slurmctld, listens to for work. The default value is SLURMCTLD_PORT as established at
system build time. If none is explicitly specified, it will be set to 6817. SlurmctldPort may also be configured to support a range
of port numbers in order to accept larger bursts of incoming messages by specifying two numbers separated by a dash (e.g. Slurmctld-
Port=6817-6818). NOTE: Either slurmctld and slurmd daemons must not execute on the same nodes or the values of SlurmctldPort and
SlurmdPort must be different.
SlurmctldTimeout
The interval, in seconds, that the backup controller waits for the primary controller to respond before assuming control. The
default value is 120 seconds. May not exceed 65533.
SlurmdDebug
The level of detail to provide slurmd daemon's logs. Values from 0 to 9 are legal, with `0' being "quiet" operation and `9' being
insanely verbose. The default value is 3.
SlurmdLogFile
Fully qualified pathname of a file into which the slurmd daemon's logs are written. The default value is none (performs logging
via syslog). Any "%h" within the name is replaced with the hostname on which the slurmd is running.
See the section LOGGING if a pathname is specified.
SlurmdPidFile
Fully qualified pathname of a file into which the slurmd daemon may write its process id. This may be used for automated signal
processing. The default value is "/var/run/slurmd.pid".
SlurmdPort
The port number that the SLURM compute node daemon, slurmd, listens to for work. The default value is SLURMD_PORT as established at
system build time. If none is explicitly specified, its value will be 6818. NOTE: Either slurmctld and slurmd daemons must not exe-
cute on the same nodes or the values of SlurmctldPort and SlurmdPort must be different.
SlurmdSpoolDir
Fully qualified pathname of a directory into which the slurmd daemon's state information and batch job script information are writ-
ten. This must be a common pathname for all nodes, but should represent a directory which is local to each node (reference a local
file system). The default value is "/var/spool/slurmd." NOTE: This directory is also used to store slurmd's shared memory lockfile,
and should not be changed unless the system is being cleanly restarted. If the location of SlurmdSpoolDir is changed and slurmd is
restarted, the new daemon will attach to a different shared memory region and lose track of any running jobs.
SlurmdTimeout
The interval, in seconds, that the SLURM controller waits for slurmd to respond before configuring that node's state to DOWN. A
value of zero indicates the node will not be tested by slurmctld to confirm the state of slurmd, the node will not be automatically
set to a DOWN state indicating a non-responsive slurmd, and some other tool will take responsibility for monitoring the state of
each compute node and its slurmd daemon. SLURM's hierarchical communication mechanism is used to ping the slurmd daemons in order
to minimize system noise and overhead. The default value is 300 seconds. The value may not exceed 65533 seconds.
SlurmSchedLogFile
Fully qualified pathname of the scheduling event logging file. The syntax of this parameter is the same as for SlurmctldLogFile.
In order to configure scheduler logging, set both the SlurmSchedLogFile and SlurmSchedLogLevel parameters.
SlurmSchedLogLevel
The initial level of scheduling event logging, similar to the SlurmctlDebug parameter used to control the initial level of slurmctld
logging. Valid values for SlurmSchedLogLevel are "0" (scheduler logging disabled) and "1" (scheduler logging enabled). If this
parameter is omitted, the value defaults to "0" (disabled). In order to configure scheduler logging, set both the SlurmSchedLogFile
and SlurmSchedLogLevel parameters. The scheduler logging level can be changed dynamically using scontrol.
SrunEpilog
Fully qualified pathname of an executable to be run by srun following the completion of a job step. The command line arguments for
the executable will be the command and arguments of the job step. This configuration parameter may be overridden by srun's --epilog
parameter. Note that while the other "Epilog" executables (e.g., TaskEpilog) are run by slurmd on the compute nodes where the tasks
are executed, the SrunEpilog runs on the node where the "srun" is executing.
SrunProlog
Fully qualified pathname of an executable to be run by srun prior to the launch of a job step. The command line arguments for the
executable will be the command and arguments of the job step. This configuration parameter may be overridden by srun's --prolog
parameter. Note that while the other "Prolog" executables (e.g., TaskProlog) are run by slurmd on the compute nodes where the tasks
are executed, the SrunProlog runs on the node where the "srun" is executing.
StateSaveLocation
Fully qualified pathname of a directory into which the SLURM controller, slurmctld, saves its state (e.g. "/usr/local/slurm/check-
point"). SLURM state will saved here to recover from system failures. SlurmUser must be able to create files in this directory.
If you have a BackupController configured, this location should be readable and writable by both systems. Since all running and
pending job information is stored here, the use of a reliable file system (e.g. RAID) is recommended. The default value is "/tmp".
If any slurm daemons terminate abnormally, their core files will also be written into this directory.
SuspendExcNodes
Specifies the nodes which are to not be placed in power save mode, even if the node remains idle for an extended period of time.
Use SLURM's hostlist expression to identify nodes. By default no nodes are excluded. Related configuration options include Resume-
Timeout, ResumeProgram, ResumeRate, SuspendProgram, SuspendRate, SuspendTime, SuspendTimeout, and SuspendExcParts.
SuspendExcParts
Specifies the partitions whose nodes are to not be placed in power save mode, even if the node remains idle for an extended period
of time. Multiple partitions can be identified and separated by commas. By default no nodes are excluded. Related configuration
options include ResumeTimeout, ResumeProgram, ResumeRate, SuspendProgram, SuspendRate, SuspendTime SuspendTimeout, and SuspendExcN-
odes.
SuspendProgram
SuspendProgram is the program that will be executed when a node remains idle for an extended period of time. This program is
expected to place the node into some power save mode. This can be used to reduce the frequency and voltage of a node or completely
power the node off. The program executes as SlurmUser. The argument to the program will be the names of nodes to be placed into
power savings mode (using SLURM's hostlist expression format). By default, no program is run. Related configuration options
include ResumeTimeout, ResumeProgram, ResumeRate, SuspendRate, SuspendTime, SuspendTimeout, SuspendExcNodes, and SuspendExcParts.
SuspendRate
The rate at which nodes are place into power save mode by SuspendProgram. The value is number of nodes per minute and it can be
used to prevent a large drop in power power consumption (e.g. after a large job completes). A value of zero results in no limits
being imposed. The default value is 60 nodes per minute. Related configuration options include ResumeTimeout, ResumeProgram,
ResumeRate, SuspendProgram, SuspendTime, SuspendTimeout, SuspendExcNodes, and SuspendExcParts.
SuspendTime
Nodes which remain idle for this number of seconds will be placed into power save mode by SuspendProgram. A value of -1 disables
power save mode and is the default. Related configuration options include ResumeTimeout, ResumeProgram, ResumeRate, SuspendProgram,
SuspendRate, SuspendTimeout, SuspendExcNodes, and SuspendExcParts.
SuspendTimeout
Maximum time permitted (in second) between when a node suspend request is issued and when the node shutdown. At that time the node
must ready for a resume request to be issued as needed for new work. The default value is 30 seconds. Related configuration
options include ResumeProgram, ResumeRate, ResumeTimeout, SuspendRate, SuspendTime, SuspendProgram, SuspendExcNodes and SuspendEx-
cParts. More information is available at the SLURM web site ( http://www.schedmd.com/slurmdocs/power_save.html ).
SwitchType
Identifies the type of switch or interconnect used for application communications. Acceptable values include "switch/none" for
switches not requiring special processing for job launch or termination (Myrinet, Ethernet, and InfiniBand), "switch/elan" for
Quadrics Elan 3 or Elan 4 interconnect. The default value is "switch/none". All SLURM daemons, commands and running jobs must be
restarted for a change in SwitchType to take effect. If running jobs exist at the time slurmctld is restarted with a new value of
SwitchType, records of all jobs in any state may be lost.
TaskEpilog
Fully qualified pathname of a program to be execute as the slurm job's owner after termination of each task. See TaskProlog for
execution order details.
TaskPlugin
Identifies the type of task launch plugin, typically used to provide resource management within a node (e.g. pinning tasks to spe-
cific processors). More than one task plugin can be specified in a comma separated list. The prefix of "task/" is optional. Accept-
able values include:
task/affinity enables resource containment using CPUSETs. This enables the --cpu_bind and/or --mem_bind srun options. If you use
"task/affinity" and encounter problems, it may be due to the variety of system calls used to implement task affinity
on different operating systems. If that is the case, you may want to install Portable Linux Process Affinity (PLPA,
see http://www.open-mpi.org/software/plpa), which is supported by SLURM.
task/cgroup enables resource containment using Linux control cgroups. This enables the --cpu_bind and/or --mem_bind srun
options. NOTE: see "man cgroup.conf" for configuration details.
task/none for systems requiring no special handling of user tasks. Lacks support for the --cpu_bind and/or --mem_bind srun
options. The default value is "task/none".
TaskPluginParam
Optional parameters for the task plugin. Multiple options should be comma separated If None, Sockets, Cores, Threads, and/or Ver-
bose are specified, they will override the --cpu_bind option specified by the user in the srun command. None, Sockets, Cores and
Threads are mutually exclusive and since they decrease scheduling flexibility are not generally recommended (select no more than one
of them). Cpusets and Sched are mutually exclusive (select only one of them).
Cores Always bind to cores. Overrides user options or automatic binding.
Cpusets Use cpusets to perform task affinity functions. By default, Sched task binding is performed.
None Perform no task binding. Overrides user options or automatic binding.
Sched Use sched_setaffinity or plpa_sched_setaffinity (if available) to bind tasks to processors.
Sockets Always bind to sockets. Overrides user options or automatic binding.
Threads Always bind to threads. Overrides user options or automatic binding.
Verbose Verbosely report binding before tasks run. Overrides user options.
TaskProlog
Fully qualified pathname of a program to be execute as the slurm job's owner prior to initiation of each task. Besides the normal
environment variables, this has SLURM_TASK_PID available to identify the process ID of the task being started. Standard output from
this program can be used to control the environment variables and output for the user program.
export NAME=value Will set environment variables for the task being spawned. Everything after the equal sign to the end of the
line will be used as the value for the environment variable. Exporting of functions is not currently supported.
print ... Will cause that line (without the leading "print ") to be printed to the job's standard output.
unset NAME Will clear environment variables for the task being spawned.
The order of task prolog/epilog execution is as follows:
1. pre_launch() Function in TaskPlugin
2. TaskProlog System-wide per task program defined in slurm.conf
3. user prolog Job step specific task program defined using srun's --task-prolog option or SLURM_TASK_PROLOG environment vari-
able
4. Execute the job step's task
5. user epilog Job step specific task program defined using srun's --task-epilog option or SLURM_TASK_EPILOG environment vari-
able
6. TaskEpilog System-wide per task program defined in slurm.conf
7. post_term() Function in TaskPlugin
TmpFS Fully qualified pathname of the file system available to user jobs for temporary storage. This parameter is used in establishing a
node's TmpDisk space. The default value is "/tmp".
TopologyPlugin
Identifies the plugin to be used for determining the network topology and optimizing job allocations to minimize network contention.
See NETWORK TOPOLOGY below for details. Additional plugins may be provided in the future which gather topology information directly
from the network. Acceptable values include:
topology/3d_torus default for Sun Constellation systems, best-fit logic over three-dimensional topology
topology/node_rank orders nodes based upon information a node_rank field in the node record as generated by a select plugin. SLURM
performs a best-fit algorithm over those ordered nodes
topology/none default for other systems, best-fit logic over one-dimensional topology
topology/tree used for a hierarchical network as described in a topology.conf file
TrackWCKey
Boolean yes or no. Used to set display and track of the Workload Characterization Key. Must be set to track wckey usage.
TreeWidth
Slurmd daemons use a virtual tree network for communications. TreeWidth specifies the width of the tree (i.e. the fanout). On
architectures with a front end node running the slurmd daemon, the value must always be equal to or greater than the number of front
end nodes which eliminates the need for message forwarding between the slurmd daemons. On other architectures the default value is
50, meaning each slurmd daemon can communicate with up to 50 other slurmd daemons and over 2500 nodes can be contacted with two mes-
sage hops. The default value will work well for most clusters. Optimal system performance can typically be achieved if TreeWidth
is set to the square root of the number of nodes in the cluster for systems having no more than 2500 nodes or the cube root for
larger systems.
UnkillableStepProgram
If the processes in a job step are determined to be unkillable for a period of time specified by the UnkillableStepTimeout variable,
the program specified by UnkillableStepProgram will be executed. This program can be used to take special actions to clean up the
unkillable processes and/or notify computer administrators. The program will be run SlurmdUser (usually "root"). By default no
program is run.
UnkillableStepTimeout
The length of time, in seconds, that SLURM will wait before deciding that processes in a job step are unkillable (after they have
been signaled with SIGKILL) and execute UnkillableStepProgram as described above. The default timeout value is 60 seconds.
UsePAM If set to 1, PAM (Pluggable Authentication Modules for Linux) will be enabled. PAM is used to establish the upper bounds for
resource limits. With PAM support enabled, local system administrators can dynamically configure system resource limits. Changing
the upper bound of a resource limit will not alter the limits of running jobs, only jobs started after a change has been made will
pick up the new limits. The default value is 0 (not to enable PAM support). Remember that PAM also needs to be configured to sup-
port SLURM as a service. For sites using PAM's directory based configuration option, a configuration file named slurm should be
created. The module-type, control-flags, and module-path names that should be included in the file are:
auth required pam_localuser.so
auth required pam_shells.so
account required pam_unix.so
account required pam_access.so
session required pam_unix.so
For sites configuring PAM with a general configuration file, the appropriate lines (see above), where slurm is the service-name,
should be added.
VSizeFactor
Memory specifications in job requests apply to real memory size (also known as resident set size). It is possible to enforce virtual
memory limits for both jobs and job steps by limiting their virtual memory to some percentage of their real memory allocation. The
VSizeFactor parameter specifies the job's or job step's virtual memory limit as a percentage of its real memory limit. For example,
if a job's real memory limit is 500MB and VSizeFactor is set to 101 then the job will be killed if its real memory exceeds 500MB or
its virtual memory exceeds 505MB (101 percent of the real memory limit). The default valus is 0, which disables enforcement of vir-
tual memory limits. The value may not exceed 65533 percent.
WaitTime
Specifies how many seconds the srun command should by default wait after the first task terminates before terminating all remaining
tasks. The "--wait" option on the srun command line overrides this value. If set to 0, this feature is disabled. May not exceed
65533 seconds.
The configuration of nodes (or machines) to be managed by SLURM is also specified in /etc/slurm.conf. Changes in node configuration (e.g.
adding nodes, changing their processor count, etc.) require restarting the slurmctld daemon. Only the NodeName must be supplied in the
configuration file. All other node configuration information is optional. It is advisable to establish baseline node configurations,
especially if the cluster is heterogeneous. Nodes which register to the system with less than the configured resources (e.g. too little
memory), will be placed in the "DOWN" state to avoid scheduling jobs on them. Establishing baseline configurations will also speed SLURM's
scheduling process by permitting it to compare job requirements against these (relatively few) configuration parameters and possibly avoid
having to check job requirements against every individual node's configuration. The resources checked at node registration time are: CPUs,
RealMemory and TmpDisk. While baseline values for each of these can be established in the configuration file, the actual values upon node
registration are recorded and these actual values may be used for scheduling purposes (depending upon the value of FastSchedule in the con-
figuration file.
Default values can be specified with a record in which "NodeName" is "DEFAULT". The default entry values will apply only to lines follow-
ing it in the configuration file and the default values can be reset multiple times in the configuration file with multiple entries where
"NodeName=DEFAULT". The "NodeName=" specification must be placed on every line describing the configuration of nodes. In fact, it is gen-
erally possible and desirable to define the configurations of all nodes in only a few lines. This convention permits significant optimiza-
tion in the scheduling of larger clusters. In order to support the concept of jobs requiring consecutive nodes on some architectures, node
specifications should be place in this file in consecutive order. No single node name may be listed more than once in the configuration
file. Use "DownNodes=" to record the state of nodes which are temporarily in a DOWN, DRAIN or FAILING state without altering permanent
configuration information. A job step's tasks are allocated to nodes in order the nodes appear in the configuration file. There is
presently no capability within SLURM to arbitrarily order a job step's tasks.
Multiple node names may be comma separated (e.g. "alpha,beta,gamma") and/or a simple node range expression may optionally be used to spec-
ify numeric ranges of nodes to avoid building a configuration file with large numbers of entries. The node range expression can contain
one pair of square brackets with a sequence of comma separated numbers and/or ranges of numbers separated by a "-" (e.g.
"linux[0-64,128]", or "lx[15,18,32-33]"). Note that the numeric ranges can include one or more leading zeros to indicate the numeric por-
tion has a fixed number of digits (e.g. "linux[0000-1023]"). Up to two numeric ranges can be included in the expression (e.g.
"rack[0-63]_blade[0-41]"). If one or more numeric expressions are included, one of them must be at the end of the name (e.g.
"unit[0-31]rack" is invalid), but arbitrary names can always be used in a comma separated list.
On BlueGene systems only, the square brackets should contain pairs of three digit numbers separated by a "x". These numbers indicate the
boundaries of a rectangular prism (e.g. "bgl[000x144,400x544]"). See BlueGene documentation for more details. The node configuration
specified the following information:
NodeName
Name that SLURM uses to refer to a node (or base partition for BlueGene systems). Typically this would be the string that
"/bin/hostname -s" returns. It may also be the fully qualified domain name as returned by "/bin/hostname -f" (e.g. "foo1.bar.com"),
or any valid domain name associated with the host through the host database (/etc/hosts) or DNS, depending on the resolver settings.
Note that if the short form of the hostname is not used, it may prevent use of hostlist expressions (the numeric portion in brackets
must be at the end of the string). Only short hostname forms are compatible with the switch/elan and switch/federation plugins at
this time. It may also be an arbitrary string if NodeHostname is specified. If the NodeName is "DEFAULT", the values specified
with that record will apply to subsequent node specifications unless explicitly set to other values in that node record or replaced
with a different set of default values. For architectures in which the node order is significant, nodes will be considered consecu-
tive in the order defined. For example, if the configuration for "NodeName=charlie" immediately follows the configuration for
"NodeName=baker" they will be considered adjacent in the computer.
NodeHostname
Typically this would be the string that "/bin/hostname -s" returns. It may also be the fully qualified domain name as returned by
"/bin/hostname -f" (e.g. "foo1.bar.com"), or any valid domain name associated with the host through the host database (/etc/hosts)
or DNS, depending on the resolver settings. Note that if the short form of the hostname is not used, it may prevent use of hostlist
expressions (the numeric portion in brackets must be at the end of the string). Only short hostname forms are compatible with the
switch/elan and switch/federation plugins at this time. A node range expression can be used to specify a set of nodes. If an
expression is used, the number of nodes identified by NodeHostname on a line in the configuration file must be identical to the num-
ber of nodes identified by NodeName. By default, the NodeHostname will be identical in value to NodeName.
NodeAddr
Name that a node should be referred to in establishing a communications path. This name will be used as an argument to the gethost-
byname() function for identification. If a node range expression is used to designate multiple nodes, they must exactly match the
entries in the NodeName (e.g. "NodeName=lx[0-7] NodeAddr="elx[0-7]"). NodeAddr may also contain IP addresses. By default, the
NodeAddr will be identical in value to NodeName.
CoresPerSocket
Number of cores in a single physical processor socket (e.g. "2"). The CoresPerSocket value describes physical cores, not the logi-
cal number of processors per socket. NOTE: If you have multi-core processors, you will likely need to specify this parameter in
order to optimize scheduling. The default value is 1.
CPUs Number of logical processors on the node (e.g. "2"). If CPUs is omitted, it will set equal to the product of Sockets, CoresPer-
Socket, and ThreadsPerCore. The default value is 1.
Feature
A comma delimited list of arbitrary strings indicative of some characteristic associated with the node. There is no value associ-
ated with a feature at this time, a node either has a feature or it does not. If desired a feature may contain a numeric component
indicating, for example, processor speed. By default a node has no features. Also see Gres.
Gres A comma delimited list of generic resources specifications for a node. Each resource specification consists of a name followed by
an optional colon with a numeric value (default value is one) (e.g. "Gres=bandwidth:10000,gpu:2"). A suffix of "K", "M" or "G" may
be used to mulitply the number by 1024, 1048576 or 1073741824 respectively (e.g. "Gres=bandwidth:4G,gpu:4").. By default a node has
no generic resources. Also see Feature.
Port The port number that the SLURM compute node daemon, slurmd, listens to for work on this particular node. By default there is a sin-
gle port number for all slurmd daemons on all compute nodes as defined by the SlurmdPort configuration parameter. Use of this option
is not generally recommended except for development or testing purposes.
Procs See CPUs.
RealMemory
Size of real memory on the node in MegaBytes (e.g. "2048"). The default value is 1.
Reason Identifies the reason for a node being in state "DOWN", "DRAINED" "DRAINING", "FAIL" or "FAILING". Use quotes to enclose a reason
having more than one word.
Sockets
Number of physical processor sockets/chips on the node (e.g. "2"). If Sockets is omitted, it will be inferred from CPUs, CoresPer-
Socket, and ThreadsPerCore. NOTE: If you have multi-core processors, you will likely need to specify these parameters. The default
value is 1.
State State of the node with respect to the initiation of user jobs. Acceptable values are "DOWN", "DRAIN", "FAIL", "FAILING" and
"UNKNOWN". "DOWN" indicates the node failed and is unavailable to be allocated work. "DRAIN" indicates the node is unavailable to
be allocated work. "FAIL" indicates the node is expected to fail soon, has no jobs allocated to it, and will not be allocated to
any new jobs. "FAILING" indicates the node is expected to fail soon, has one or more jobs allocated to it, but will not be allo-
cated to any new jobs. "UNKNOWN" indicates the node's state is undefined (BUSY or IDLE), but will be established when the slurmd
daemon on that node registers. The default value is "UNKNOWN". Also see the DownNodes parameter below.
ThreadsPerCore
Number of logical threads in a single physical core (e.g. "2"). Note that the SLURM can allocate resources to jobs down to the res-
olution of a core. If your system is configured with more than one thread per core, execution of a different job on each thread is
not supported unless you configure SelectTypeParameters=CR_CPU plus CPUs; do not configure Sockets, CoresPerSocket or ThreadsPer-
Core. A job can execute a one task per thread from within one job step or execute a distinct job step on each of the threads. Note
also if you are running with more than 1 thread per core and running the select/cons_res plugin you will want to set the SelectType-
Parameters variable to something other than CR_CPU to avoid unexpected results. The default value is 1.
TmpDisk
Total size of temporary disk storage in TmpFS in MegaBytes (e.g. "16384"). TmpFS (for "Temporary File System") identifies the loca-
tion which jobs should use for temporary storage. Note this does not indicate the amount of free space available to the user on the
node, only the total file system size. The system administration should insure this file system is purged as needed so that user
jobs have access to most of this space. The Prolog and/or Epilog programs (specified in the configuration file) might be used to
insure the file system is kept clean. The default value is 0.
Weight The priority of the node for scheduling purposes. All things being equal, jobs will be allocated the nodes with the lowest weight
which satisfies their requirements. For example, a heterogeneous collection of nodes might be placed into a single partition for
greater system utilization, responsiveness and capability. It would be preferable to allocate smaller memory nodes rather than
larger memory nodes if either will satisfy a job's requirements. The units of weight are arbitrary, but larger weights should be
assigned to nodes with more processors, memory, disk space, higher processor speed, etc. Note that if a job allocation request can
not be satisfied using the nodes with the lowest weight, the set of nodes with the next lowest weight is added to the set of nodes
under consideration for use (repeat as needed for higher weight values). If you absolutely want to minimize the number of higher
weight nodes allocated to a job (at a cost of higher scheduling overhead), give each node a distinct Weight value and they will be
added to the pool of nodes being considered for scheduling individually. The default value is 1.
The "DownNodes=" configuration permits you to mark certain nodes as in a DOWN, DRAIN, FAIL, or FAILING state without altering the permanent
configuration information listed under a "NodeName=" specification.
DownNodes
Any node name, or list of node names, from the "NodeName=" specifications.
Reason Identifies the reason for a node being in state "DOWN", "DRAIN", "FAIL" or "FAILING. Use quotes to enclose a reason having more
than one word.
State State of the node with respect to the initiation of user jobs. Acceptable values are "BUSY", "DOWN", "DRAIN", "FAIL", "FAILING,
"IDLE", and "UNKNOWN".
DOWN Indicates the node failed and is unavailable to be allocated work.
DRAIN Indicates the node is unavailable to be allocated work.on.
FAIL Indicates the node is expected to fail soon, has no jobs allocated to it, and will not be allocated to any new jobs.
FAILING Indicates the node is expected to fail soon, has one or more jobs allocated to it, but will not be allocated to any new
jobs.
FUTURE Indicates the node is defined for future use and need not exist when the SLURM daemons are started. These nodes can be
made available for use simply by updating the node state using the scontrol command rather than restarting the slurmctld
daemon. After these nodes are made available, change their State in the slurm.conf file. Until these nodes are made avail-
able, they will not be seen using any SLURM commands or Is nor will any attempt be made to contact them.
UNKNOWN Indicates the node's state is undefined (BUSY or IDLE), but will be established when the slurmd daemon on that node regis-
ters. The default value is "UNKNOWN".
On computers where frontend nodes are used to execute batch scripts rather than compute nodes (BlueGene or Cray systems), one may configure
one or more frontend nodes using the configuration parameters defined below. These options are very similar to those used in configuring
compute nodes. These options may only be used on systems configured and built with the appropriate parameters (--have-front-end,
--enable-bluegene-emulation) or a system determined to have the appropriate architecture by the configure script (BlueGene or Cray sys-
tems). The front end configuration specifies the following information:
FrontendName
Name that SLURM uses to refer to a frontend node. Typically this would be the string that "/bin/hostname -s" returns. It may also
be the fully qualified domain name as returned by "/bin/hostname -f" (e.g. "foo1.bar.com"), or any valid domain name associated with
the host through the host database (/etc/hosts) or DNS, depending on the resolver settings. Note that if the short form of the
hostname is not used, it may prevent use of hostlist expressions (the numeric portion in brackets must be at the end of the string).
If the FrontendName is "DEFAULT", the values specified with that record will apply to subsequent node specifications unless explic-
itly set to other values in that frontend node record or replaced with a different set of default values. Note that since the nam-
ing of front end nodes would typically not follow that of the compute nodes (e.g. lacking X, Y and Z coordinates found in the com-
pute node naming scheme), each front end node name should be listed separately and without a hostlist expression (i.e. fron-
tend00,frontend01" rather than "frontend[00-01]").</p>
FrontendAddr
Name that a frontend node should be referred to in establishing a communications path. This name will be used as an argument to the
gethostbyname() function for identification. As with FrontendName, list the individual node addresses rather than using a hostlist
expression. The number of FrontendAddr records per line must equal the number of FrontendName records per line (i.e. you can't map
to node names to one address). FrontendAddr may also contain IP addresses. By default, the FrontendAddr will be identical in value
to FrontendName.
Port The port number that the SLURM compute node daemon, slurmd, listens to for work on this particular frontend node. By default there
is a single port number for all slurmd daemons on all frontend nodes as defined by the SlurmdPort configuration parameter. Use of
this option is not generally recommended except for development or testing purposes.
Reason Identifies the reason for a frontend node being in state "DOWN", "DRAINED" "DRAINING", "FAIL" or "FAILING". Use quotes to enclose a
reason having more than one word.
State State of the frontend node with respect to the initiation of user jobs. Acceptable values are "DOWN", "DRAIN", "FAIL", "FAILING"
and "UNKNOWN". "DOWN" indicates the frontend node has failed and is unavailable to be allocated work. "DRAIN" indicates the fron-
tend node is unavailable to be allocated work. "FAIL" indicates the frontend node is expected to fail soon, has no jobs allocated
to it, and will not be allocated to any new jobs. "FAILING" indicates the frontend node is expected to fail soon, has one or more
jobs allocated to it, but will not be allocated to any new jobs. "UNKNOWN" indicates the frontend node's state is undefined (BUSY
or IDLE), but will be established when the slurmd daemon on that node registers. The default value is "UNKNOWN". Also see the
DownNodes parameter below.
The partition configuration permits you to establish different job limits or access controls for various groups (or partitions) of nodes.
Nodes may be in more than one partition, making partitions serve as general purpose queues. For example one may put the same set of nodes
into two different partitions, each with different constraints (time limit, job sizes, groups allowed to use the partition, etc.). Jobs
are allocated resources within a single partition. Default values can be specified with a record in which "PartitionName" is "DEFAULT".
The default entry values will apply only to lines following it in the configuration file and the default values can be reset multiple times
in the configuration file with multiple entries where "PartitionName=DEFAULT". The "PartitionName=" specification must be placed on every
line describing the configuration of partitions. If a partition that is in use is deleted from the configuration and slurm is restarted or
reconfigured (scontrol reconfigure), jobs using the partition are canceled. NOTE: Put all parameters for each partition on a single line.
Each line of partition configuration information should represent a different partition. The partition configuration file contains the
following information:
AllocNodes
Comma separated list of nodes from which users can execute jobs in the partition. Node names may be specified using the node range
expression syntax described above. The default value is "ALL".
AllowGroups
Comma separated list of group names which may execute jobs in the partition. If at least one group associated with the user
attempting to execute the job is in AllowGroups, he will be permitted to use this partition. Jobs executed as user root can use any
partition without regard to the value of AllowGroups. If user root attempts to execute a job as another user (e.g. using srun's
--uid option), this other user must be in one of groups identified by AllowGroups for the job to successfully execute. The default
value is "ALL". NOTE: For performance reasons, SLURM maintains a list of user IDs allowed to use each partition and this is checked
at job submission time. This list of user IDs is updated when the slurmctld daemon is restarted, reconfigured (e.g. "scontrol
reconfig") or the partition's AllowGroups value is reset, even if is value is unchanged (e.g. "scontrol update PartitionName=name
AllowGroups=group"). For a user's access to a partition to change, both his group membership must change and SLURM's internal user
ID list must change using one of the methods described above.
Alternate
Partition name of alternate partition to be used if the state of this partition is "DRAIN" or "INACTIVE."
Default
If this keyword is set, jobs submitted without a partition specification will utilize this partition. Possible values are "YES" and
"NO". The default value is "NO".
DefMemPerCPU
Default real memory size available per allocated CPU in MegaBytes. Used to avoid over-subscribing memory and causing paging.
DefMemPerCPU would generally be used if individual processors are allocated to jobs (SelectType=select/cons_res). If not set, the
DefMemPerCPU value for the entire cluster will be used. Also see DefMemPerNode and MaxMemPerCPU. DefMemPerCPU and DefMemPerNode
are mutually exclusive. NOTE: Enforcement of memory limits currently requires enabling of accounting, which samples memory use on a
periodic basis (data need not be stored, just collected).
DefMemPerNode
Default real memory size available per allocated node in MegaBytes. Used to avoid over-subscribing memory and causing paging.
DefMemPerNode would generally be used if whole nodes are allocated to jobs (SelectType=select/linear) and resources are shared
(Shared=yes or Shared=force). If not set, the DefMemPerNode value for the entire cluster will be used. Also see DefMemPerCPU and
MaxMemPerNode. DefMemPerCPU and DefMemPerNode are mutually exclusive. NOTE: Enforcement of memory limits currently requires
enabling of accounting, which samples memory use on a periodic basis (data need not be stored, just collected).
DefaultTime
Run time limit used for jobs that don't specify a value. If not set then MaxTime will be used. Format is the same as for MaxTime.
DisableRootJobs
If set to "YES" then user root will be prevented from running any jobs on this partition. The default value will be the value of
DisableRootJobs set outside of a partition specification (which is "NO", allowing user root to execute jobs).
GraceTime
Specifies, in units of seconds, the preemption grace time to be extended to a job which has been selected for preemption. The
default value is zero, no preemption grace time is allowed on this partition. (Meaningful only for PreemptMode=CANCEL)
Hidden Specifies if the partition and its jobs are to be hidden by default. Hidden partitions will by default not be reported by the SLURM
APIs or commands. Possible values are "YES" and "NO". The default value is "NO". Note that partitions that a user lacks access to
by virtue of the AllowGroups parameter will also be hidden by default.
MaxMemPerCPU
Maximum real memory size available per allocated CPU in MegaBytes. Used to avoid over-subscribing memory and causing paging.
MaxMemPerCPU would generally be used if individual processors are allocated to jobs (SelectType=select/cons_res). If not set, the
MaxMemPerCPU value for the entire cluster will be used. Also see DefMemPerCPU and MaxMemPerNode. MaxMemPerCPU and MaxMemPerNode
are mutually exclusive. NOTE: Enforcement of memory limits currently requires enabling of accounting, which samples memory use on a
periodic basis (data need not be stored, just collected).
MaxMemPerNode
Maximum real memory size available per allocated node in MegaBytes. Used to avoid over-subscribing memory and causing paging.
MaxMemPerNode would generally be used if whole nodes are allocated to jobs (SelectType=select/linear) and resources are shared
(Shared=yes or Shared=force). If not set, the MaxMemPerNode value for the entire cluster will be used. Also see DefMemPerNode and
MaxMemPerCPU. MaxMemPerCPU and MaxMemPerNode are mutually exclusive. NOTE: Enforcement of memory limits currently requires
enabling of accounting, which samples memory use on a periodic basis (data need not be stored, just collected).
MaxNodes
Maximum count of nodes which may be allocated to any single job. For BlueGene systems this will be a c-nodes count and will be
converted to a midplane count with a reduction in resolution. The default value is "UNLIMITED", which is represented internally as
-1. This limit does not apply to jobs executed by SlurmUser or user root.
MaxTime
Maximum run time limit for jobs. Format is minutes, minutes:seconds, hours:minutes:seconds, days-hours, days-hours:minutes,
days-hours:minutes:seconds or "UNLIMITED". Time resolution is one minute and second values are rounded up to the next minute. This
limit does not apply to jobs executed by SlurmUser or user root.
MinNodes
Minimum count of nodes which may be allocated to any single job. For BlueGene systems this will be a c-nodes count and will be
converted to a midplane count with a reduction in resolution. The default value is 1. This limit does not apply to jobs executed
by SlurmUser or user root.
Nodes Comma separated list of nodes (or base partitions for BlueGene systems) which are associated with this partition. Node names may be
specified using the node range expression syntax described above. A blank list of nodes (i.e. "Nodes= ") can be used if one wants a
partition to exist, but have no resources (possibly on a temporary basis).
PartitionName
Name by which the partition may be referenced (e.g. "Interactive"). This name can be specified by users when submitting jobs. If
the PartitionName is "DEFAULT", the values specified with that record will apply to subsequent partition specifications unless
explicitly set to other values in that partition record or replaced with a different set of default values.
PreemptMode
Mechanism used to preempt jobs from this partition when PreemptType=preempt/partition_prio is configured. This partition specific
PreemptMode configuration parameter will override the PreemptMode configuration parameter set for the cluster as a whole. The clus-
ter-level PreemptMode must include the GANG option if PreemptMode is configured to SUSPEND for any partition. The cluster-level
PreemptMode must not be OFF if PreemptMode is enabled for any partition. See the description of the cluster-level PreemptMode con-
figuration parameter above for further information.
Priority
Jobs submitted to a higher priority partition will be dispatched before pending jobs in lower priority partitions and if possible
they will preempt running jobs from lower priority partitions. Note that a partition's priority takes precedence over a job's pri-
ority. The value may not exceed 65533.
RootOnly
Specifies if only user ID zero (i.e. user root) may allocate resources in this partition. User root may allocate resources for any
other user, but the request must be initiated by user root. This option can be useful for a partition to be managed by some exter-
nal entity (e.g. a higher-level job manager) and prevents users from directly using those resources. Possible values are "YES" and
"NO". The default value is "NO".
Shared Controls the ability of the partition to execute more than one job at a time on each resource (node, socket or core depending upon
the value of SelectTypeParameters). If resources are to be shared, avoiding memory over-subscription is very important. Select-
TypeParameters should be configured to treat memory as a consumable resource and the --mem option should be used for job alloca-
tions. Sharing of resources is typically useful only when using gang scheduling (PreemptMode=suspend or PreemptMode=kill). Possi-
ble values for Shared are "EXCLUSIVE", "FORCE", "YES", and "NO". The default value is "NO". For more information see the following
web pages:
http://www.schedmd.com/slurmdocs/cons_res.html,
http://www.schedmd.com/slurmdocs/cons_res_share.html,
http://www.schedmd.com/slurmdocs/gang_scheduling.html, and
http://www.schedmd.com/slurmdocs/preempt.html.
EXCLUSIVE Allocates entire nodes to jobs even with select/cons_res configured. Jobs that run in partitions with "Shared=EXCLU-
SIVE" will have exclusive access to all allocated nodes.
FORCE Makes all resources in the partition available for sharing without any means for users to disable it. May be followed
with a colon and maximum number of jobs in running or suspended state. For example "Shared=FORCE:4" enables each node,
socket or core to execute up to four jobs at once. Recommended only for BlueGene systems configured with small blocks
or for systems running with gang scheduling (SchedulerType=sched/gang).
YES Makes all resources in the partition available for sharing, but honors a user's request for dedicated resources. If
SelectType=select/cons_res, then resources will be over-subscribed unless explicitly disabled in the job submit request
using the "--exclusive" option. With SelectType=select/bluegene or SelectType=select/linear, resources will only be
over-subscribed when explicitly requested by the user using the "--share" option on job submission. May be followed
with a colon and maximum number of jobs in running or suspended state. For example "Shared=YES:4" enables each node,
socket or core to execute up to four jobs at once. Recommended only for systems running with gang scheduling (Sched-
ulerType=sched/gang).
NO Selected resources are allocated to a single job. No resource will be allocated to more than one job.
State State of partition or availability for use. Possible values are "UP", "DOWN", "DRAIN" and "INACTIVE". The default value is "UP".
See also the related "Alternate" keyword.
UP Designates that new jobs may queued on the partition, and that jobs may be allocated nodes and run from the partition.
DOWN Designates that new jobs may be queued on the partition, but queued jobs may not be allocated nodes and run from the par-
tition. Jobs already running on the partition continue to run. The jobs must be explicitly canceled to force their termi-
nation.
DRAIN Designates that no new jobs may be queued on the partition (job submission requests will be denied with an error message),
but jobs already queued on the partition may be allocated nodes and run. See also the "Alternate" partition specifica-
tion.
INACTIVE Designates that no new jobs may be queued on the partition, and jobs already queued may not be allocated nodes and run.
See also the "Alternate" partition specification.
Prolog and Epilog Scripts
There are a variety of prolog and epilog program options that execute with various permissions and at various times. The four options most
likely to be used are: Prolog and Epilog (executed once on each compute node for each job) plus PrologSlurmctld and EpilogSlurmctld (exe-
cuted once on the ControlMachine for each job).
NOTE: Standard output and error messages are normally not preserved. Explicitly write output and error messages to an appropriate loca-
tion if you wish to preserve that information.
NOTE: The Prolog script is ONLY run on any individual node when it first sees a job step from a new allocation; it does not run the Prolog
immediately when an allocation is granted. If no job steps from an allocation are run on a node, it will never run the Prolog for that
allocation. The Epilog, on the other hand, always runs on every node of an allocation when the allocation is released.
Information about the job is passed to the script using environment variables. Unless otherwise specified, these environment variables are
available to all of the programs.
BASIL_RESERVATION_ID
Basil reservation ID. Available on Cray XT/XE systems only.
MPIRUN_PARTITION
BlueGene partition name. Available on BlueGene systems only.
SLURM_JOB_ACCOUNT
Account name used for the job. Available in PrologSlurmctld and EpilogSlurmctld only.
SLURM_JOB_CONSTRAINTS
Features required to run the job. Available in PrologSlurmctld and EpilogSlurmctld only.
SLURM_JOB_DERIVED_EC
The highest exit code of all of the job steps. Available in EpilogSlurmctld only.
SLURM_JOB_EXIT_CODE
The exit code of the job script (or salloc). Available in EpilogSlurmctld only.
SLURM_JOB_GID
Group ID of the job's owner. Available in PrologSlurmctld and EpilogSlurmctld only.
SLURM_JOB_GROUP
Group name of the job's owner. Available in PrologSlurmctld and EpilogSlurmctld only.
SLURM_JOB_ID
Job ID.
SLURM_JOB_NAME
Name of the job. Available in PrologSlurmctld and EpilogSlurmctld only.
SLURM_JOB_NODELIST
Nodes assigned to job. A SLURM hostlist expression. "scontrol show hostnames" can be used to convert this to a list of individual
host names. Available in PrologSlurmctld and EpilogSlurmctld only.
SLURM_JOB_PARTITION
Partition that job runs in. Available in PrologSlurmctld and EpilogSlurmctld only.
SLURM_JOB_UID
User ID of the job's owner.
SLURM_JOB_USER
User name of the job's owner.
NETWORK TOPOLOGY
SLURM is able to optimize job allocations to minimize network contention. Special SLURM logic is used to optimize allocations on systems
with a three-dimensional interconnect (BlueGene, Sun Constellation, etc.) and information about configuring those systems are available on
web pages available here: <http://www.schedmd.com/slurmdocs/>. For a hierarchical network, SLURM needs to have detailed information about
how nodes are configured on the network switches.
Given network topology information, SLURM allocates all of a job's resources onto a single leaf of the network (if possible) using a
best-fit algorithm. Otherwise it will allocate a job's resources onto multiple leaf switches so as to minimize the use of higher-level
switches. The TopologyPlugin parameter controls which plugin is used to collect network topology information. The only values presently
supported are "topology/3d_torus" (default for IBM BlueGene, Sun Constellation and Cray XT/XE systems, performs best-fit logic over
three-dimensional topology), "topology/none" (default for other systems, best-fit logic over one-dimensional topology), "topology/tree"
(determine the network topology based upon information contained in a topology.conf file, see "man topology.conf" for more information).
Future plugins may gather topology information directly from the network. The topology information is optional. If not provided, SLURM
will perform a best-fit algorithm assuming the nodes are in a one-dimensional array as configured and the communications cost is related to
the node distance in this array.
RELOCATING CONTROLLERS
If the cluster's computers used for the primary or backup controller will be out of service for an extended period of time, it may be
desirable to relocate them. In order to do so, follow this procedure:
1. Stop the SLURM daemons
2. Modify the slurm.conf file appropriately
3. Distribute the updated slurm.conf file to all nodes
4. Restart the SLURM daemons
There should be no loss of any running or pending jobs. Insure that any nodes added to the cluster have the current slurm.conf file
installed.
CAUTION: If two nodes are simultaneously configured as the primary controller (two nodes on which ControlMachine specify the local host and
the slurmctld daemon is executing on each), system behavior will be destructive. If a compute node has an incorrect ControlMachine or
BackupController parameter, that node may be rendered unusable, but no other harm will result.
EXAMPLE
#
# Sample /etc/slurm.conf for dev[0-25].llnl.gov
# Author: John Doe
# Date: 11/06/2001
#
ControlMachine=dev0
ControlAddr=edev0
BackupController=dev1
BackupAddr=edev1
#
AuthType=auth/munge
Epilog=/usr/local/slurm/epilog
Prolog=/usr/local/slurm/prolog
FastSchedule=1
FirstJobId=65536
InactiveLimit=120
JobCompType=jobcomp/filetxt
JobCompLoc=/var/log/slurm/jobcomp
KillWait=30
MaxJobCount=10000
MinJobAge=3600
PluginDir=/usr/local/lib:/usr/local/slurm/lib
ReturnToService=0
SchedulerType=sched/backfill
SlurmctldLogFile=/var/log/slurm/slurmctld.log
SlurmdLogFile=/var/log/slurm/slurmd.log
SlurmctldPort=7002
SlurmdPort=7003
SlurmdSpoolDir=/usr/local/slurm/slurmd.spool
StateSaveLocation=/usr/local/slurm/slurm.state
SwitchType=switch/elan
TmpFS=/tmp
WaitTime=30
JobCredentialPrivateKey=/usr/local/slurm/private.key
JobCredentialPublicCertificate=/usr/local/slurm/public.cert
#
# Node Configurations
#
NodeName=DEFAULT CPUs=2 RealMemory=2000 TmpDisk=64000
NodeName=DEFAULT State=UNKNOWN
NodeName=dev[0-25] NodeAddr=edev[0-25] Weight=16
# Update records for specific DOWN nodes
DownNodes=dev20 State=DOWN Reason="power,ETA=Dec25"
#
# Partition Configurations
#
PartitionName=DEFAULT MaxTime=30 MaxNodes=10 State=UP
PartitionName=debug Nodes=dev[0-8,18-25] Default=YES
PartitionName=batch Nodes=dev[9-17] MinNodes=4
PartitionName=long Nodes=dev[9-17] MaxTime=120 AllowGroups=admin
FILE AND DIRECTORY PERMISSIONS
There are three classes of files: Files used by slurmctld must be accessible by user SlurmUser and accessible by the primary and backup
control machines. Files used by slurmd must be accessible by user root and accessible from every compute node. A few files need to be
accessible by normal users on all login and compute nodes. While many files and directories are listed below, most of them will not be
used with most configurations.
AccountingStorageLoc
If this specifies a file, it must be writable by user SlurmUser. The file must be accessible by the primary and backup control
machines. It is recommended that the file be readable by all users from login and compute nodes.
Epilog Must be executable by user root. It is recommended that the file be readable by all users. The file must exist on every compute
node.
EpilogSlurmctld
Must be executable by user SlurmUser. It is recommended that the file be readable by all users. The file must be accessible by the
primary and backup control machines.
HealthCheckProgram
Must be executable by user root. It is recommended that the file be readable by all users. The file must exist on every compute
node.
JobCheckpointDir
Must be writable by user SlurmUser and no other users. The file must be accessible by the primary and backup control machines.
JobCompLoc
If this specifies a file, it must be writable by user SlurmUser. The file must be accessible by the primary and backup control
machines.
JobCredentialPrivateKey
Must be readable only by user SlurmUser and writable by no other users. The file must be accessible by the primary and backup con-
trol machines.
JobCredentialPublicCertificate
Readable to all users on all nodes. Must not be writable by regular users.
MailProg
Must be executable by user SlurmUser. Must not be writable by regular users. The file must be accessible by the primary and backup
control machines.
Prolog Must be executable by user root. It is recommended that the file be readable by all users. The file must exist on every compute
node.
PrologSlurmctld
Must be executable by user SlurmUser. It is recommended that the file be readable by all users. The file must be accessible by the
primary and backup control machines.
ResumeProgram
Must be executable by user SlurmUser. The file must be accessible by the primary and backup control machines.
SallocDefaultCommand
Must be executable by all users. The file must exist on every login and compute node.
slurm.conf
Readable to all users on all nodes. Must not be writable by regular users.
SlurmctldLogFile
Must be writable by user SlurmUser. The file must be accessible by the primary and backup control machines.
SlurmctldPidFile
Must be writable by user root. Preferably writable and removable by SlurmUser. The file must be accessible by the primary and
backup control machines.
SlurmdLogFile
Must be writable by user root. A distinct file must exist on each compute node.
SlurmdPidFile
Must be writable by user root. A distinct file must exist on each compute node.
SlurmdSpoolDir
Must be writable by user root. A distinct file must exist on each compute node.
SrunEpilog
Must be executable by all users. The file must exist on every login and compute node.
SrunProlog
Must be executable by all users. The file must exist on every login and compute node.
StateSaveLocation
Must be writable by user SlurmUser. The file must be accessible by the primary and backup control machines.
SuspendProgram
Must be executable by user SlurmUser. The file must be accessible by the primary and backup control machines.
TaskEpilog
Must be executable by all users. The file must exist on every compute node.
TaskProlog
Must be executable by all users. The file must exist on every compute node.
UnkillableStepProgram
Must be executable by user SlurmUser. The file must be accessible by the primary and backup control machines.
LOGGING
Note that while SLURM daemons create log files and other files as needed, it treats the lack of parent directories as a fatal error. This
prevents the daemons from running if critical file systems are not mounted and will minimize the risk of cold-starting (starting without
preserving jobs).
Log files and job accounting files, may need to be created/owned by the "SlurmUser" uid to be successfully accessed. Use the "chown" and
"chmod" commands to set the ownership and permissions appropriately. See the section FILE AND DIRECTORY PERMISSIONS for information about
the various files and directories used by SLURM.
It is recommended that the logrotate utility be used to insure that various log files do not become too large. This also applies to text
files used for accounting, process tracking, and the slurmdbd log if they are used.
Here is a sample logrotate configuration. Make appropriate site modifications and save as /etc/logrotate.d/slurm on all nodes. See the
logrotate man page for more details.
##
# SLURM Logrotate Configuration
##
/var/log/slurm/*log {
compress
missingok
nocopytruncate
nocreate
nodelaycompress
nomail
notifempty
noolddir
rotate 5
sharedscripts
size=5M
create 640 slurm root
postrotate
/etc/init.d/slurm reconfig
endscript
}
COPYING
Copyright (C) 2002-2007 The Regents of the University of California. Copyright (C) 2008-2010 Lawrence Livermore National Security. Por-
tions Copyright (C) 2010 SchedMD <http://www.sched-md.com>. Produced at Lawrence Livermore National Laboratory (cf, DISCLAIMER).
CODE-OCEC-09-009. All rights reserved.
This file is part of SLURM, a resource management program. For details, see <http://www.schedmd.com/slurmdocs/>.
SLURM is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free
Software Foundation; either version 2 of the License, or (at your option) any later version.
SLURM is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.
FILES
/etc/slurm.conf
SEE ALSO
bluegene.conf(5), cgroup.conf(5), gethostbyname (3), getrlimit (2), gres.conf(5), group (5), hostname (1), scontrol(1), slurmctld(8),
slurmd(8), slurmdbd(8), slurmdbd.conf(5), srun(1), spank(8), syslog (2), topology.conf(5), wiki.conf(5)
slurm.conf 2.3 September 2011 slurm.conf(5)