The HPC clusters cannot be accessed directly from the Internet - access is a two-step process:
- Log in to
This is a "bastion host", providing a secure gateway between the NYU HPC clusters and the Internet. You will find that on
hpc.nyu.eduitself you can do very little save but to log in to an NYU HPC cluster
- Log in to the cluster you wish to use
This will put you on a "login" node for that cluster, from where you can manipulate files and initiate jobs. This is not the place for heavy compute work: you submit that as a job to the compute nodes via the batch queuing system. Instructions for using the batch system can be found here (but you don't need to worry about that yet)
To log in you will use the program
ssh from a terminal window. If you are unfamiliar with the command line interface this may seem daunting - relax, it's easy and vastly more powerful than point-and-click. We have a basic tutorial here.
If you wish to use any software with a graphical interface, your Mac or Linux workstation must be configured to handle remote windows. This is a standard feature of Linux, but Mac users will first need to download and install an X server such as XQuartz.
For easier access and to transfer files between your workstation and the clusters, you will eventually want to set up SSH tunneling
Logging In - the easy, primitive way
Now you are ready to begin the two-step process.
In the boxes below and elsewhere in this wiki, the symbol "
$" at the beginning of a line represents the command prompt - don't type the "
$", type only the remainder of the line following the "
$". Also wherever
NetID appears, replace it with your NYU NetID.
- Log in to
-Yoption allows X forwarding, that is, GUI applications running on the cluster can draw windows on your screen (as long as you have an X server such as XQuartz installed)
From there, log in to the cluster you wish to use
The clusters are named as follows:
Cluster Host name Mercer
Therefore, log in using one of the following commands, according to which cluster you wish to use:
On Mercer you may notice that you are now on a host named "
login-0-0" or "
login-0-3" or something similar. The cluster uses multiple login nodes and which one you get depends how busy each is at the time. The login nodes are configured identically and see the same filesystems, so the specific node you are logged in to is not important.
Setting up SSH Tunneling
In computer networking, a computer decides what to do with an incoming network packet according to the "port" it arrived on. The port is simply a number attached to the packet. Certain ports are reserved for specific functions, for example packets arriving on port 22 are assumed to be intended for the SSH handler, so the computer passes those packets to SSH to interpret. Other port numbers are available to use for whatever you like, and as long as the same port is not used for different things on the same computer, everything works.
With SSH Tunneling, you will start an SSH session between your workstation and the bastion host
hpc.nyu.edu, and instruct that session to create a tunnel. Your workstation will make one end of the tunnel, at "
localhost, port 8023" ("
localhost" is the computer's name for itself, so packets arriving at your workstation port 8023 will be sent into the tunnel). The bastion host will make the other end of the tunnel, at "
mercer.es.its.nyu.edu, port 22", so anything coming through the tunnel will be forwarded to the normal SSH port (22) of Mercer. The fact that your workstation cannot see Mercer does not matter, it only needs to see its end of the tunnel.
The following diagram illustrates the process. It looks complex, but only requires 2 steps: the blue text shows what happens when you create the tunnel (step 1) and the green arrows indicate using the tunnel (step 2).
You only need to do step 1 once, and then you can use the tunnel (step 2) as many times as you like - for example, you might have two terminal sessions and a WinSCP session all using the same tunnel created with step 1.
In these instructions we are using port 8023. If it happens that another program on your computer is watch this port (which is fairly unlikely) then it won't work, and you'll need to choose a different port number, eg 9020, and substitute that throughout these instructions. 4-digit numbers starting with an 8 or a 9 are usually good ones to choose.
(Mac only) Preparing your Mac for SSH Tunneling
Recent versions of OSX do not require changes to System Preferences, so first try skipping to Setting up a tunnel you can reuse first. If you are then unable to connect through your tunnel, try the 4 steps below.
We will be instructing your Mac to forward certain incoming packets to a tunnel, so first the Mac must be willing to accept the incoming packets at all. To enable this:
- Open System Preferences and click Sharing.
- Select the Remote Login checkbox.
- Return to system preferences and click "Security" and then "Firewall options" (if you are using Mavericks) or "Advanced" (for older versions of OSX)
- Uncheck "Block all incoming connections".
We have a video guide of this process.
Also, make sure you have prepared your Mac for X in accordance with the instructions above.
Note that Linux users do not need to do this: remote logins are enabled by default under Linux.
Setting up a tunnel you can reuse (the best approach)
To avoid repeatedly setting up a tunnel, we write the details of the tunnel into your SSH configuration file. This is found in the hidden ".ssh/" directory under your home directory. To access it, first open a Terminal window.
The process is the same for Linux and Mac, and is demonstrated in the first minute and a half of this video guide (the remainder of the video demonstrates transferring files over the tunnel, which is covered here).
To see the contents of a directory, enter "
ls -la" at the command prompt. The "
a" is important, without it files and directories starting with "
." will be hidden. We are looking for a directory called "
If you do not have a "
.ssh" directory, create one as follows. The permissions of this directory are important, hence the
Using your favorite editor, open the file "
.ssh/config". If you are not familiar with Unix-ish editors, we have some help here.
Add the following lines to
Starting the tunnel
To create the tunnel, ssh to it with the following command:
Important: you must leave this window open for the tunnel to remain open. It is best to start a new terminal window for subsequent logins.
Logging in via the tunnel
- Open a new terminal window
Use ssh to log in to the cluster, as shown below. Note that you must use the short name defined above in your
.ssh/configfile, not the fully qualified domain name
In the command above we have not used the
-X flag to
ssh. This is no longer needed because we already specified "
ForwardX11 yes" in the
Creating a once-off tunnel (not the best approach)
You can set up a once-off tunnel without editing
.ssh/config by running the following command:
This is the equivalent to running "
ssh hpctunnel" in the reusable tunnel instructions, but the port forwarding is specified on the command line. You can setup tunneling to a second cluster (eg Babar) by adding a second -L option to the command line.
Logging in via a once-off tunnel
However this does not create the convenient alias, so when connecting with ssh, or scp or rsync, you must explicitly connect to that port on your workstation (localhost):
- Open a new terminal window
Use ssh to log in to the cluster by logging into to "localhost" on the appropriate port, as shown below.
With a once-off tunnel we do need the
-X flag to
ssh, and also the
-p flag (lowercase) to specify the port number.
At the end of your session, you can log out of a cluster, and of
hpc.nyu.edu, with the command:
Don't forget to also log out from the session on hpc.nyu.edu, which you opened to start the tunnel!