Child pages
  • Programming for Biologists
Skip to end of metadata
Go to start of metadata

 

Getting Started on HPC Prince Cluster

Logging in to Prince is the same 2-stage process as logging into to any other NYU HPC cluster. The host name is 'prince.hpc.nyu.edu'.

The HPC clusters (Prince and Dumbo) are not directly visible to the internet (outside the NYU Network). If you are outside NYU's Network (off-campus) you must first login to a bastion host named gw.hpc.nyu.edu.

 The diagram below illustrates the login path.

The clusters can still access the internet directly. This may be useful when copying data from servers outside the NYU Network - see: How to copy files to and from the HPC clusters.

Alternatively, instead of login to the bastion host, you can use VPN to get inside NYU's network and access the HPC clusters directly. Instructions on how to install and use the VPN client are available here.

You can't do anything on the bastion host, except ssh to the HPC clusters.

  • From an off-campus location (outside NYU-NET), logging in to the HPC clusters is a two-step process:
    1. First login to the bastion host, gw.hpc.nyu.edu. From a Mac or Linux workstation, this is a simple terminal command (replace my_netid with your NetId). Your password is the same password you use for NYU Home:
      ssh -X NYUNetID@gw.hpc.nyu.edu
      Windows users will need to use Putty, see here for instructions.

    2. Next login to the cluster. For Prince, this is done with: 
      ssh -X prince
  • From within the NYU network, that is, from an on-campus location, or after you VPN inside NYU's network, you can login to the HPC clusters directly.  You do not need to login to the bastion host.

To login in to the HPC cluster (Prince), simply use:
ssh -X NYUNetID@prince.hpc.nyu.edu

Compute and Storage Facilities

Prince has 2 login and 385 compute nodes:

 

System Name

HPC Cluster Prince

Vendor

Dell

Network

  • Infiniband by Mellanox for MPI and access to file systems  (home (ZFS), scratch (Luster and BeeGFS), and archive (ZFS) )
  • 10Gbit Management Network (node provisioning and configuration)
  • 1 Gb Ethernet Service Network for IPMI/iDRAC access
  • 10Gbit access to the public NYU Network (only available on the Prince login nodes and selected management nodes)

Operating System

CentOS 7.3

Login Nodes

2 login nodes: prince0 and prince1.hpc.nyu.edu

Each login nodes has 2 Intel Xeon E5-2680v4 2.4GHz CPUs ("Broadwell", 14 cores/socket, a total of 28 cores/login node) and 128 GB memory

Compute Nodes

Standard Compute Nodes

  • 68 nodes each with 2 Intel Xeon E5-2690v4 2.6GHz CPUs ("Broadwell", 14 cores/socket, 28 cores/node) and 125GB memory, EDR interconnects
  • 32 nodes each with 2 Intel Xeon E5-2690v4 2.6GHz CPUs ("Broadwell", 14 cores/socket, 28 cores/node) and 250GB memory, EDR interconnects 
  • 32 nodes each with 2 Intel Xeon E5-2660v3 2.6GHz CPUs ("Haswell", 10 cores/socket, 20 cores/node) and 62 GB memory. The 32 nodes are M630 Blade servers on 2 M1000e chassis and are interconnected via FDR Infiniband
  • 64 nodes each with 2 Intel Xeon E5-2690v2 3.0GHz CPUs ("Ivy Bridge", 10 cores/socket, 20 cores/node) and 62 GB memory. The 64 nodes are M620 Blade servers on 4 M1000e chassis and are interconnected via FDR Infiniband (used to be Mercer chassis 0, 1, 2, 3)
  •  112 nodes each with 2 Intel Xeon E-2690v2 3.0GHz CPUs ("Ivy Bridge", 10 cores/socket, 20 cores/node) and 62 GB memory. The 112 nodes are M620 Blade servers on 7 M1000e chassis and are interconnected via QDR Infiniband (Mercer chassis 14-20)
  •  48 nodes each with 2 Intel Xeon E-2690v2 3.0GHz CPUs ("Ivy Bridge", 10 cores/socket, 20 cores/node) and 189 GB memory. The 48 nodes are M620 Blade servers on 3 M1000e chassis and are interconnected via QDR Infiniband (Mercer chassis 21-23)

Nodes equipped with NVIDIA GPUs

  • 9 nodes each with 2 Intel Xeon E5-2690v4 2.6GHz CPUs ("Broadwell", 14 cores/socket, 28 cores/node) and 256GB memory, EDR interconnects, each node equipped with 2 NVIDIA K80 GPUs (24GB, split between 2 GPU cards)
  • 4 nodes each with 2 Intel Xeon E5-2690v4 2.6GHz CPUs ("Broadwell", 14 cores/socket, 28 cores/node) and 128GB memory, EDR interconnects, each node equipped with 4 NVIDIA GTX 1080 GPUs (8 GB)
  • 8 nodes each with 2 Intel Xeon E5-2670v2 2.5GHz CPUs ("Ivy Bridge", 10 cores/socket, 20 cores/node) and 128 GB memory, FDR interconnects, each node equipped with 4 NVIDIA K80 GPUs

Medium Memory Node

  • 4 nodes each with 2 Intel Xeon E5-2687Wv3 3.1GHz ("Haswell", 10 cores/socket, 20 cores/node), 512GB memory, FDR interconnects.

High Memory Nodes

  • 2 nodes each with 4 Intel Xeon E7-8857v2 3.0GHz ("Ivy Bridge", 12 cores/socket, 48 cores/node), 1.5TB of memory, FDR interconnects.

 


Total Nodes
387 (385 Compute Nodes + 2 Login Nodes)
CPU cores
8928 cores on compute nodes + 56 cores on login nodes
GPUs

50 NVIDIA K80 (24GB)

16 NVIDIA GTX 1080 (8GB)

Total memory46TB for compute + 256GB for login nodes

 

Basic Linux Commands and available file systems

Files Systems for usage:

The NYU HPC clusters have multiple filesystems for users files. Each filesystem is configured differently to serve a different purpose.

Mountpoint

Storage Capacity

(User Quota)

FS Type

Backed up?

Flushed?

Availability

Variable

Value

/home

43 TB

(20 GB / user)

ZFS

Yes

No

All Prince nodes (login, compute)

$HOME

/home/$USER

/scratch

1.1 PB

(5 TB / user)

Lustre

NO

 

YES

Files unused for 60 days are deleted

All Prince nodes (login, compute)

$SCRATCH

/scratch/$USER

/beegfs

500TB

(2 TB / user)

BeeGFSNO

YES

Files unused for 60 days are deleted

All nodes (login, compute)
$BEEGFS/beegfs/$USER

/archive

700 TB

(2 TB / user)

ZFS

Yes

No

Only on login nodes

$ARCHIVE

/archive/$USER

/state/partition1Varies, mostly >100GBext3NO

YES

at the end of each job

Separate local filesystem on each compute node$SLURM_JOBTMP/state/partition1/$SLURM_JOBID

 


Navigating the directory structure on Linux and Basic Commands:

We've already seen ssh, which takes us from the host we are on to a different host, and hostname, which tells us which host we are on now. Mostly you'll move around filesystems and directories, which resemble inverted tree structures as shown below schematically:

  • "pwd" - "print working directory", or "where am I now?".
    In Unix, filesystems and directories are arranged in a hierarchy. A forward slash "/" is the directory separator, and the topmost directory visible to a host is called "/". Filesystems are also mounted into this directory structure, so you can access everything that is visible on this host by moving around in the directory hierarchy.

    You should see something like /home/NetID

  • "cd" - To change to a different directory, use "cd" ("change directory"). You'll need to give it the path to the directory you wish to change into, eg "cd /scratch/NetID". You can go up one directory with "cd ..".
    If you run "cd" with no arguments, you will be returned to your home directory.
    And if you run "cd -" you will be returned to the directory you were in most recently. 
  • "mkdir" - To create a new location, use "mkdir new_location_name".

  • "ls" - To see what files are in the current directory, use "ls" ("list").
    If this is your first time using the HPC cluster, "ls" probably won't return anything, because you have no files to list.
    There are a couple of useful options for ls:

    • "ls -l" lists the directory contents in long format, one file or directory per line, with extra information about who owns the file, how big it is, and what permissions are set. 

    • "ls -a" lists hidden files. In Unix, files whose names begins with "." are hidden. This does not stop anything from using those files, it simply instructs ls not to show the files unless the -a option is used.

  • "rmdir" - To remove a directory, use "rmdir new_location_name".


Hands-on Exercise 1:

Exercise
$ cd $SCRATCH
 
$ pwd
 
$ mkdir project1
 
$ cd project1
 
$ pwd


Copying, moving or deleting files locally:

Copying: The "cp" command makes a duplicate copy of files and directories within a cluster or machine. The general usage is "cp source destination":

Command

Explanation

cp test_file.txt test_file2.txt

Copies a duplicate copy of test_file1.txt with new name test_file2.txt

cp -r subdir subdir2

Recursively copy the directory "subdir" to a new directory "subdir2".

That is, a new directory "subdir2" is created and each file and

directory under subdir is replicated in subdir2.

Moving: The "mv" command renames files and directories within a cluster or machine. The general usage is "mv source destination":

Command

Explanation

mv dummy_file.txt test_file.txt

Renames dummy_file.txt as test_file.txt.

mv subdir new_subdir

Renames the directory "subdir" to a new directory "new_subdir".

Deleting files:

The "rm" (remove) command deletes files and optionally directories within a cluster or machine. There is no undelete in Unix. Once it is gone, it is gone.

Command

Explanation

rm dummy_file.txt

rm -i dummy_file.txt

Remove a file.

If you use -i you will be prompted for confirmation before each file is deleted.

rm -f serious_file.txt

Forcibly remove a file, without asking, regardless of its permissions (provided you own the file)

rmdir subdir/Remove subdir if it is already empty. Otherwise, command fails.

rm -r subdir/

Recursively delete the directory subdir and everything in it. Use with care!

 

The head and tail commands:

The command "head filename" displays the first few lines of a text file filename.

The command "tail filename" displays the last few lines of a text file filename.

Hands-on Exercise 2:

Exercise
$pwd

---- copy command --------  
$ cp /share/apps/examples/bio-workflow/ChIPseq_chr19.fastq .
 
---- head command --------  
$ head ChIPseq_chr19.fastq
@HWUSI-EAS610_0001:3:1:4:1405#0/1
GATAGTTCAATTCCAGAGATCAGAGAGAGGTGAGTG
+
B;30;<4@7/5@=?5?7?1>A2?0<6?<<80>79##
@HWUSI-EAS610_0001:3:1:5:1490#0/1
GGGCTGGTGGAGTGATCCCAAGGGGTGGGGATGGGG
+
B@A?AAA1BB;A5B44>AA3'@AB>+>@AB94A?A?
@HWUSI-EAS610_0001:3:1:6:388#0/1
CAGAGTTCATGAAATAGGCCTCTAGTCTTCCTAGAC
 
---- tail command --------
$ tail ChIPseq_chr19.fastq

+
7B68BBBA63;A@A@B@?6=75(4>B@A:?:@BA@A
@HWUSI-EAS610_0001:3:120:1789:543#0/1
TATCCCATTGCCTTGGCATCCAAAAGAGATAACAGG
+
BCC>ABBBAAAABAABBBB@ABBBABABA?BA?BAA
@HWUSI-EAS610_0001:3:120:1791:1598#0/1
TAATAACAGACAATTCAAATAAAAACCTGACGGGAC
+
BB@9@A1<@924=@@/@9<;9=;77'/7:98>;?96

 

---- word count --------  
$ wc -l ChIPseq_chr19.fastq

 

How to map DNA Reads

 

Text Editor (NANO):

"nano" is a friendly text editor which can be used to edit content of existing file or create a new file. Here are some options used in nano editor.

OptionExplanation
Ctrl + OSave the changes
Ctrl + XExit nano
Ctrl + KCut single line
Ctrl + UPaste the text


Softwares and Environment Module

Environment Modules is a tool for managing multiple versions and configurations of software packages, and is used by many HPC centers around the world.

To use a given software package, you load the corresponding module. Unloading the module afterwards cleanly undoes the changes that loading the module made to your environment, thus freeing you to use other software packages that might have conflicted with the first one.

Working with software packages on the NYU HPC clusters.

The command for seeing what software packages are available is:

$ module avail

Finding out more about a software package

$ module whatis module-name

A module file may include more detailed help for the software package, which can be seen with:

$ module help module-name

You can also see exactly what effect loading the module will have with:

$ module show module-name

You can check which modules are currently loaded in your environment with:

 $ module list

To load a module:

$ module load module-name

To unload the module:

 $ module unload module-name

Unloading all modules - You can remove all loaded modules from your environment with:

NOTE: It's a good idea to use "module purge" before loading modules to ensure you have a consistent environment each time you run.

 $ module purge

sbatch Job Submission

In a Linux cluster there are hundreds of computing nodes inter-connected by high speed networks. Linux operating system runs on each of the nodes individually. The resources are shared among many users for their technical or scientific computing purposes. Slurm is a cluster software layer built on top of the inter-connected nodes, aiming at orchestrating the nodes' computing activities, so that the cluster could be viewed as a unified, enhanced and scalable computing system by its users. In NYU HPC clusters the users coming from many departments with various disciplines and subjects, with their own computing projects, impose on us very diverse requirements regarding hardware, software resources and processing parallelism. Users submit jobs, which compete for computing resources. The Slurm software system is a resource manager and a job scheduler, which is designed to allocate resources and schedule jobs. Slurm is an open source software, with a large user community, and has been installed on many top 500 supercomputers.

As an user we interact mostly with sbatch. This hands-on tutorial introduces the usage of its three commands:

  1. sbatch: submit jobs
  2. squeue: monitor jobs
  3. scancel: terminate jobs before completion

sbatch

Batch job submission can be accomplished with the command sbatch. Like in Torque qsub, we create a bash script to describe our job requirement: what resources we need, what softwares and processing we want to run, memory and CPU requested, and where to send job standard output and error etc. After a job is submitted, Slurm will find the suitable resources, schedule and drive the job execution, and report outcome back to the user. The user can then return to look at the output files.

Example-1:
In the first example, we create a small bash script, run it locally, then submit it as a job to Slurm using sbatch, and compare the results.  

 

Example-2:

Follow the recipe below to submit a job. The job can be used later as an example for practicing how to check job status. In my test its running duration is about 7 minutes.

Below is the content of the bash script "run-matlab.s" just used in the job submission:


#!/bin/bash
#
##SBATCH --nodes=1
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=1
#SBATCH --cpus-per-task=2
#SBATCH --time=1:00:00
#SBATCH --mem=10GB
#SBATCH --job-name=myMatlabTest
#SBATCH --mail-type=END
##SBATCH --mail-user=bob.smith@nyu.edu
#SBATCH --output=slurm_%j.out

module purge
module load matlab/2016b

cd /scratch/$USER/mytest1
cat thtest.m | srun matlab -nodisplay


For reference - Explanation of script

squeue

The squeue command lists jobs which are in a state of either running, or waiting or completing etc. It can also display jobs owned by a specific user or with specific job ID.

Run 'man sinfo' or 'man squeue' to see the explanations for the results.

 

To remove a queued job from the batch queuing system, or kill a running job, use scancel. To delete all jobs emitted by you, turn on '-u' argument.

 

A R Job Example

 How do I run a R job?

Basic R jobs

Multiple R versions exist in HPC environment. To check what are available, on Prince:

Suppose we want to use 3.3.2, run these commands:

We first clean up the environment by doing 'module purge'. Then we load the R version selected, check what are available in current environment. We can see that R 3.3.2 is indeed loaded along with its dependency modules. Let's try this basic R example. We name it "example.R":

Below is the screen scene while running it on Prince:

What is shown above is a simple demo case on login nodes. For real interactive analysis scenario, users are encouraged to run on compute nodes using the 'srun' command to request dedicated resources, e.g.:

Besides running our analysis interactively, long running and big data crunching jobs ought to be submitted to the batch system slurm. The "example.R" can be submitted to slurm to run in batch mode.

Copy example files to your newly created directory.

example

Below is how the example looks like:

example

Then create a sbatch job script as:

Once the sbatch script file is ready, it can be submitted to the job scheduler using sbatch. After successful completion of job, verify output log file for detail output information.

sbatch

 

 

(more...

 

 

 

 


  • No labels