Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

In a Linux cluster there are hundreds of computing nodes inter-connected by high speed networks. Linux operating system runs on each of the nodes individually. The resources are shared among many users for their technical or scientific computing purposes. Slurm is a cluster software layer built on top of the inter-connected nodes, aiming at orchestrating the nodes' computing activities, so that the cluster could be viewed as a unified, enhanced and scalable computing system by its users. In NYU HPC clusters the users coming from many departments with various disciplines and subjects, with their own computing projects, impose on us very diverse requirements regarding hardware, software resources and processing parallelism. Users submit jobs, which compete for computing resources. The Slurm software system is a resource manager and a job scheduler, which is designed to allocate resources and schedule jobs. Slurm is an open source software, with a large user community, and has been installed on many top 500 supercomputers.

...

For more details of nodes, file system's hardware configuration, please click the link "Cluster - Prince".

Return to Top

Section
Connecting to

The Prince picture

NOTE: The cluster nodes can still access the internet directly. This may be useful when copying data from servers outside the NYU Network

NOTE: Alternatively, instead of login to the bastion hosts, you can use VPN to get inside NYU's network and access the HPC clusters directly. Instructions on how to install and use the VPN client are available here

NOTE: You can't do anything on the bastion hosts, except ssh to the HPC clusters (Prince, Dumbo).

Return to Top 

Section
Column

Connecting to Prince

Logging onto the Prince cluster and submitting jobs is analogous to triple jump the Olympic game which was originated in ancient Greece. First, open a terminal on your Mac workstation. If your workstation is outside of NYU network, follow these three steps:

  1. Hop  - from your workstation, ssh onto one bastion host, which can be either hpc.nyu.edu or hpc2.nyu.edu
  2. Step - from any bastion host, ssh to the Prince cluster login node prince.hpc.nyu.edu
  3. Jump - from any login node, run command "sbatch" or "srun" to submit jobs which will land on the computing node(s)

If you are inside NYU network, the first step 'hop' could be omitted.
See for instance a complete HPC session:

Panel
bgColorwhitesmoke

ITSs-Air-3:~ johd$ ssh johd@hpc.nyu.edu
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
           WARNING:  UNAUTHORIZED PERSONS ........ DO NOT PROCEED
           ~~~~~~~   ~~~~~~~~~~~~~~~~~~~~          ~~~~~~~~~~~~~~
 This computer system is operated by New York University (NYU) and may be
 accessed only by authorized users.  Authorized users are granted specific,
 limited privileges in their use of the system.  The data and programs
 in this system may not be accessed, copied, modified, or disclosed without
 prior approval of NYU.  Access and use, or causing access and use, of this
 computer system by anyone other than as permitted by NYU are strictly pro-
 hibited by NYU and by law and may subject an unauthorized user, including
 unauthorized employees, to criminal and civil penalties as well as NYU-
 initiated disciplinary proceedings.  The use of this system is routinely
 monitored and recorded, and anyone accessing this system consents to such
 monitoring and recording.  Questions regarding this access policy should be
 directed (by e-mail) to askits@nyu.edu or (by phone) to 212-998-3333.
 Questions on other topics should be directed to COMMENT (by email) or to
 212-998-3333 by phone.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
johd@hpc.nyu.edu's password: 
Last login: Wed Jan 11 09:59:54 2017 from ool-18befec8.dyn.optonline.net


[johd@hpc ~]$ ssh prince.hpc.nyu.edu
johd@prince.hpc.nyu.edu's password: 
Last login: Sat Jan 14 11:45:42 2017 from hpc2.es.its.nyu.edu


[johd@log-1 ~]$ sbatch --wrap "hostname; echo 'hello, this is a test'"
Submitted batch job 9870


[johd@log-1 ~]$ exit

For access from Windows station using PuTTY, please read below.

Expand
Column

Step 1.

Enter "hpc2.nyu.edu" for host name, and leave "22" the default for port. If you want, you may enter
a name for saved session e.g. "hpc2", click "Save" for use next time. Hit "Open".

Click "Yes" when a Window as below showing up.

 

Step 2.

Enter your NetID username, and password. This will get you on the bastion host hpc2.

 

Step 3.

On hpc2, enter command "ssh prince.hpc.nyu.edu" or "ssh prince" for short hostname, answer "yes" to the question
and type your NetID password. Suppose everything goes on smoothly, you will land on one prince login node!

 

 

Column
width40%

 

...