Skip to end of metadata
Go to start of metadata

 

Overview

The HPC clusters cannot be accessed directly from the Internet - access is a two-step process:

  1. Log in to hpc.nyu.edu
    This is a "bastion host", providing a secure gateway between the NYU HPC clusters and the Internet. You will find that on hpc.nyu.edu itself you can do very little save but to log in to an NYU HPC cluster  
  2. Log in to the cluster you wish to use
    This will put you on a "login" node for that cluster, from where you can manipulate files and initiate jobs. This is not the place for heavy compute work: you submit that as a job to the compute nodes via the batch queuing system. Instructions for using the batch system can be found here (but you don't need to worry about that yet)

To log in you will need an SSH client. We recommend PuTTY SSH but others are also available, see ITS Software & Licensing for links.

After using PuTTY to get to hpc.nyu.edu you will use the program ssh from a terminal window. If you are unfamiliar with the command line interface this may seem daunting - relax, it's easy and vastly more powerful than point-and-click. We have a basic tutorial here.

If you wish to use any software with a graphical interface, your Windows workstation must have an X server installed and running. This is described further in Preparing your Windows workstation below

For easier access and to transfer files between your workstation and the clusters, you will eventually want to set up SSH tunneling

Preparing your Windows workstation for X

If you wish to use any software with a graphical interface, you will need an X server. This is a software package that draws on your local screen windows created on a remote computer (such as an NYU HPC cluster). There are a couple of options out there:

  • We recommend Cygwin/X. Instructions for downloading and installing it can be found here.
    Before starting PuTTY you will need to have the X server running, by double-clicking the "XWin Server" shortcut under Cygwin-X on the Start Menu. You may wish to add this to your Windows Startup folder so it runs automatically after starting Windows 
  • Another good option is Xming. Installation instructions can be found on its web site.
    As per Cygwin/X, you will need to launch Xming before starting PuTTY.  

You will also need to download and install PuTTY SSH if you have not already.

Logging In - the easy, primitive way

This method is useful as a first-time login, to ensure you are able to connect. For ongoing use however, we recommend setting up and using SSH tunneling instead.

 

When you installed PuTTY you will have added to your Desktop and also the the Start Menu and icon labeled "putty". Double-clicking this icon will launch PuTTY, with the dialog initially in the "Session" category as in the screenshot below:

If this is not your first time through here, and you already saved a setup previously, you can select the saved session and hit "Load". This will do the following steps for you so you can just hit "Open". Otherwise, read on...

Enter the as Host Name "hpc.nyu.edu", leave the Port at its default value of 22 and select the "SSH" Protocol. Don't hit "Open" just yet!

Eventually, if not now, you will probably want to use some graphical applications. Enable this by selecting Connection -> SSH -> X11 from the Category menu on the left:

Check the "Enable X11 forwarding" box. Again, don't hit "Open" just yet...

You can save these settings to use again next time. Return to the "Session" category and give this setup a name in the "Saved Sessions" box (perhaps NYU HPC"). Now click "Save".

Finally, hit "Open". You will get a command terminal for hpc.nyu.edu. This is a deliberately restricted environment from which you can do little other than to log in to one of the NYU clusters, which are named as follows:

ClusterHost name
Dumbodumbo.es.its.nyu.edu

 

Therefore, log in using one of the following commands, according to which cluster you wish to use:

$ ssh -X NetID@dumbo.es.its.nyu.edu

In the boxes below and elsewhere in this wiki, the symbol "$" at the beginning of a line represents the command prompt - don't type the "$", type only the remainder of the line following the "$". Also wherever NetID appears, replace it with your NYU NetID.

Setting up SSH Tunneling

In computer networking, a computer decides what to do with an incoming network packet according to the "port" it arrived on. The port is simply a number attached to the packet. Certain ports are reserved for specific functions, for example packets arriving on port 22 are assumed to be intended for the SSH handler, so the computer passes those packets to SSH to interpret. Other port numbers are available to use for whatever you like, and as long as the same port is not used for different things on the same computer, everything works.

 

With SSH Tunneling, you will start an SSH session between your workstation and the bastion host hpc.nyu.edu, and instruct that session to create a tunnel. Your workstation will make one end of the tunnel, at "localhost, port 8023" ("localhost" is the computer's name for itself, so packets arriving at your workstation port 8023 will be sent into the tunnel). The bastion host will make the other end of the tunnel, at "mercer.es.its.nyu.edu, port 22", so anything coming through the tunnel will be forwarded to the normal SSH port (22) of Mercer. The fact that your workstation cannot see Mercer does not matter, it only needs to see its end of the tunnel.

The following diagram illustrates the process. It looks complex, but only requires 2 steps: the blue text shows what happens when you create the tunnel (step 1) and the green arrows indicate using the tunnel (step 2). 

You only need to do step 1 once, and then you can use the tunnel (step 2) as many times as you like - for example, you might have two terminal sessions and a WinSCP session all using the same tunnel created with step 1.

In these instructions we are using port 8023. If it happens that another program on your computer is watch this port (which is fairly unlikely) then it won't work, and you'll need to choose a different port number, eg 9020, and substitute that throughout these instructions. 4-digit numbers starting with an 8 or a 9 are usually good ones to choose.

Step 1: Creating the tunnel

  1. First open Putty and prepare to log in to hpc.nyu.edu, as you did in Logging in from Windows - primitive way. If you saved your session during that process, you can load it by selecting from the "Saved Sessions" box and hitting "Load". Don't hit "Open" yet!
  2. Under "Connection" -> "SSH", just below "X11", select "Tunnels"
  3. Select "Local ports accept connections from other hosts"
  4. Write "8023" (the port number) in the "Source port" box, and "mercer.es.its.nyu.edu:22" (the machine you wish to tunnel to - 22 is the port that ssh listens on) in the "Destination" box. 
  5. Click "Add". You can repeat step 4 with a different port number and a different destination, if you like (for instance, Babar). If you do this you will create multiple tunnels, one to each destination.
  6. Before hitting "Open", go back to the "Sessions" page, give the session a name ("hpctunnel") and hit "Save". Then next time you need not do all this again, just load the saved session.
  7. Hit "Open" to login in to hpc.nyu.edu and create the tunnel. A terminal window will appear, asking for your login name (NYU NetID) and password (NYU password). Windows may also ask you to allow certain connections through its firewall - this is so you can ssh to port 8023 on your workstation - the entrance to the tunnel.

You can add other NYU hosts to the tunnel by adding a new source port and destination and clicking "Add". For example, Mercer has 4 login hosts, if you specifically need to use number 1, you could add "Source port = 8024" and "Destination = mercer1.es.its.nyu.edu:22", then press "Add". You would then perform Step 2 (below) twice - once for mercer on port 8023 and once for mercer1 on port 8024.

 

 

Using your SSH tunnel

To log in via the tunnel, first the tunnel must be open. If you've just completed Step 1, it will be open and you can jump down to "Step 2: Logging in via your SSH tunnel". If you completed Step 1 yesterday, and now want to re-use the tunnel you created, first start the tunnel:

Starting the tunnel

During a session, you need only do this once - as long as the tunnel is open, new connections will go over it.

  1. Start Putty.exe (again, if necessary), and load the session you saved in Setting up SSH Tunneling
  2. Hit "Open", and log in to the bastion host with your NYU NetID and password. This will create the tunnel. 

 

Step 2: Logging in via your SSH tunnel

  1. Start Putty.exe. In the "Host Name" box, write "localhost" and in the "Port" box, write "8023" (or whichever port number you specified when you set up the tunnel in Setting up SSH Tunneling)
    We use "localhost" because the entrance of the tunnel is actually on this workstation, at port 8023.


  2. Go to "Connections" -> "SSH" -> "X11" and check "Enable X11 forwarding".



  3. Optionally, give this session a name (in "Saved Sessions") and hit "Save" to save it. Then next time instead of steps 1 and 2 you can simply load this saved session.

  4. Hit "Open". You will again get a terminal window asking for your login (NYU NetID) and password (NYU password). You are now logged in to the HPC cluster!

Logging Out

At the end of your session, you can log out of a cluster, and of hpc.nyu.edu, with the command:

$ exit

Don't forget to also log out from the session on hpc.nyu.edu, which you opened to start the tunnel!

 

 

  • No labels