Submitting a Job
Jobs are submitted with the
The options tell Torque information about the job, such as what resources will be needed. These can be specified in the job-script as PBS directives, or on the command line as options, or both (in which case the command line options take precedence should the two contradict each other). For each option there is a corresponding PBS directive with the syntax:
For example, you can specify that a job needs 2 nodes and 8 cores on each node by adding to the script the directive:
or as a command-line option to
qsub when you submit the job:
Options to manage job output:
Give the job a name. The default is the filename of the job script. Within the job,
$PBS_JOBNAMEexpands to the job name
path/for/stdout. Can be a filename or an existing directory. The default filename is
myjob.o12345, in the directory from which the job was submitted
path/for/stderr. Same usage as for
Send email to firstname.lastname@example.org when certain events occur. By default an email is sent only if the job is killed by the batch system.
-m b -m e -m a -m abe
Send email when the job begins (
b), ends (
e) and/or is aborted (
Options to set the job environment:
Use the shell at
/path/to/shellto interpret the script. Default is your login shell, which at NYU HPC is normally
-v VAR1,VAR2="some value",VAR3
Pass variables to the job, either with a specific value (the
VAR=form) or from the submitting environment (without "
Pass the full environment the job was submitted from
Options to request compute resources:
Maximum wallclock time the job will need. Default depends on queue, mostly 1 hour. Walltime is specified in seconds or as
Maximum memory per node the job will need. Default depends on queue, normally 2GB for serial jobs and the full node for parallel jobs. Memory should be specified with units, eg
8GB. The available memory per node for different nodes of Mercer is described here.
Total number of CPUs required. Use this if it does not matter how CPUs are grouped onto nodes - eg, for a purely-MPI job. Don't combine this with
-l nodes=numor odd behavior will ensue.
Number of nodes and number of processors per node required. Use this if you need processes to be grouped onto nodes - eg, for an MPI/OpenMP hybrid job with 4 MPI processes and 8 OpenMP threads each, use -l nodes=4:ppn=8. Don't combine this with
-l procs=numor odd behavior will ensue. Default is 1 node and 1 processor per node. When using multiple nodes the job script will be executed on the first allocated node.
Torque will set the environment variables
PBS_NUM_NODESto the number of nodes requested,
PBS_NUM_PPNto the value of
PBS_NPto the total number of processes available to the job.
If your job needs GPUs, you should specify the number of GPUs per node in the
We have nodes with older Tesla GPUs and newer Titan GPUs - if you specifically need the newer GPUs you should add the requirement "
:titan" to the gpus specification.
This is an NYU HPC extension: if you need whole nodes but do not mind how many cores, you can request full nodes this way. Specifying -n will have the same effect. The environment variable $PSB_PPN will be set in the job to the number of cores on each node the job is running on. Jobs are always allocated to sets of nodes with the same number of cores, so you will not get one node with 12 cores and another with 20.
Note that this currently only works when used in directives, not on the command line.
Request exclusive use of nodes. If this is specified, no other jobs will share a node with this job, and if you did not specify a memory limit, no memory limit will be enforced (Note however that if you do not specify a memory limit, you may land on a node with only 24GB of memory)
Submit to a specific queue. If not specified, Torque will choose a queue based on the resources requested.
In almost all cases, it is best not to specify a queue - the system will choose the best queue according to the resources you request
Options for running interactively on the compute nodes:
Don't just submit the job, but also wait for it to start and connect
stdinto the current terminal.
Enable X forwarding, so programs using a GUI can be used during the session (provided you have X forwarding to your workstation set up)
Run specifically in the interactive queue. At NYU HPC, this queue has smaller job limits (maximum of two nodes and 4 hours walltime) but very high priority.
Pass the current environment to the interactive batch job
- To leave an interactive batch session, type
exitat the command prompt.
Options for delaying starting a job:
Delay starting this job until
jobidhas completed successfully.
Delay starting this job until after the specified date and time. Month (
MM) and day-of-month (
DD) are optional, hour and minute are required.
Options for many similar jobs (array jobs and pbsdsh):
Submit an array of jobs with array ids as specified. Array ids can be specified as a numerical range, a comma-separated list of numbers, or as some combination of the two. Each job instance will have an environment variable
As above, but the appended '
%n' specifies the maximum number of array items (in this case, 5) which should be running at one time
- Submit a single "shepherd" job requesting multiple processes and from it start individual jobs with