In a Linux cluster there are hundreds of computing nodes inter-connected by high speed networks. Linux operating system runs on each of the nodes individually. The resources are shared among many users for their technical or scientific computing purposes. Slurm is a cluster software layer built on top of the inter-connected nodes, aiming at orchestrating the nodes' computing activities, so that the cluster could be viewed as a unified, enhanced and scalable computing system by its users. In NYU HPC clusters the users coming from many departments with various disciplines and subjects, with their own computing projects, impose on us very diverse requirements regarding hardware, software resources and processing parallelism. Users submit jobs, which compete for computing resources. The Slurm software system is a resource manager and a job scheduler, which is designed to allocate resources and schedule jobs. Slurm is an open source software, with a large user community, and has been installed on many top 500 supercomputers.
For more details of nodes, file system's hardware configuration, please click the link "Cluster - Prince".
Return to Top
|Connecting to |
The Prince picture
NOTE: The cluster nodes can still access the internet directly. This may be useful when copying data from servers outside the NYU Network
NOTE: Alternatively, instead of login to the bastion hosts, you can use VPN to get inside NYU's network and access the HPC clusters directly. Instructions on how to install and use the VPN client are available here
NOTE: You can't do anything on the bastion hosts, except
ssh to the HPC clusters (Prince, Dumbo).
Return to Top
Connecting to Prince
Logging onto the Prince cluster and submitting jobs is analogous to triple jump the Olympic game which was originated in ancient Greece. First, open a terminal on your Mac workstation. If your workstation is outside of NYU network, follow these three steps:
Hop - from your workstation, ssh onto one bastion host, which can be either hpc.nyu.edu or hpc2.nyu.edu
Step - from any bastion host, ssh to the Prince cluster login node prince.hpc.nyu.edu
Jump - from any login node, run command "sbatch" or "srun" to submit jobs which will land on the computing node(s)
If you are inside NYU network, the first step 'hop' could be omitted.
See for instance a complete HPC session:
ITSs-Air-3:~ johd$ ssh firstname.lastname@example.org
WARNING: UNAUTHORIZED PERSONS ........ DO NOT PROCEED
~~~~~~~ ~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~
This computer system is operated by New York University (NYU) and may be
accessed only by authorized users. Authorized users are granted specific,
limited privileges in their use of the system. The data and programs
in this system may not be accessed, copied, modified, or disclosed without
prior approval of NYU. Access and use, or causing access and use, of this
computer system by anyone other than as permitted by NYU are strictly pro-
hibited by NYU and by law and may subject an unauthorized user, including
unauthorized employees, to criminal and civil penalties as well as NYU-
initiated disciplinary proceedings. The use of this system is routinely
monitored and recorded, and anyone accessing this system consents to such
monitoring and recording. Questions regarding this access policy should be
directed (by e-mail) to email@example.com or (by phone) to 212-998-3333.
Questions on other topics should be directed to COMMENT (by email) or to
212-998-3333 by phone.
Last login: Wed Jan 11 09:59:54 2017 from ool-18befec8.dyn.optonline.net
[johd@hpc ~]$ ssh prince.hpc.nyu.edu
Last login: Sat Jan 14 11:45:42 2017 from hpc2.es.its.nyu.edu
[johd@log-1 ~]$ sbatch --wrap "hostname; echo 'hello, this is a test'"
Submitted batch job 9870
[johd@log-1 ~]$ exit
For access from Windows station using PuTTY, please read below.
Enter "hpc2.nyu.edu" for host name, and leave "22" the default for port. If you want, you may enter
a name for saved session e.g. "hpc2", click "Save" for use next time. Hit "Open".
Click "Yes" when a Window as below showing up.
Enter your NetID username, and password. This will get you on the bastion host hpc2.
On hpc2, enter command "ssh prince.hpc.nyu.edu" or "ssh prince" for short hostname, answer "yes" to the question
and type your NetID password. Suppose everything goes on smoothly, you will land on one prince login node!