This page is retained from an earlier version of the HPC wiki only for reference, and the equivalent up-to-date page is at Running jobs on the NYU HPC clusters.
Using fast local disk
Each node has some local disk. When you start a job, the environment variable $PBS_JOBTMP points to a directory on the local disk which your job can use. This is particularly suitable for jobs which write many small or temporary files, as these typically do not perform well on shared parallel filesystems.
All files in $PBS_JOBTMP are deleted at the end of the job.
Any files you need to keep should be copied to $HOME, $WORK or $SCRATCH before the end of the job
There is one local disk per physical node, so if you use "
-l nodes=3:ppn=4", and your job is run on:
- 4 cores of one node (say, compute-1-1)
- another 4 cores on that same physical node, and
- 4 cores on a different node (say, compute-2-10)
then $PBS_JOBTMP will be the same for the first two "nodes" and different for the third one. (Because the first two "nodes" share a physical node)
You can also request a part of the RAM on a node to be presented as a disk. This is like using local disk, but faster still - however, it occupies memory on the node.
To do this, specify:
This will reserve some memory and present it is a local disk, which you can access with the environment variable $PBS_MEMDISK.
In the second line, a memory disk of 20GB is created. In the first line, a memory disk of "some nominal amount" is created - normally 8gb. If you have exclusive use of a node and request memdisk with no specific size, the memdisk will grow as you write to it, until the job plus the memdisk have entirely filled the node's memory. This gives flexibility for when you don't know how much memdisk you need, but may result in the job (or the node) crashing if you use too much memory plus memdisk.
If you did not request memdisk, $PBS_MEMDISK will point to the local disk space at $PBS_JOBTMP.
See Managing Data for information about using rsync to synchronize directories.
Working with NYU HPC filesystems
This page is retained from an earlier version of the HPC wiki only for reference.