Child pages
  • HPC Storage Policies
Skip to end of metadata
Go to start of metadata

Storage Allocations and Policies

Each individual researcher is assigned a standard storage allocation or quota on /home, /scratch and /archive. Researchers who use more than their allocated space will be blocked from submitting jobs until they clean their space and reduce their usage, or in the case of /archive, purchase additional storage. The chart below shows the storage allocations for individual accounts and the cost of additional /archive space.

Space

Space Purpose

Backed up?

Allocation

Additional Storage Cost

Total Size

File System

/home

Program development space; storing small files you
want to keep long term , e.g. source code, scripts

Yes

5GB

N/A

~1TB*

NFS

/archive

Long-term storage

Yes

2TB**

$500/year for 1TB

200TB

ZFS

/scratch

Computational work space

No

5TB;
inode quota: 1 million
Policy

N/A

301TB

Lustre

Important: Of all the space, only /scratch should be used for computational purposes. Please do not write to /home when running jobs as it can easily be filled up.

*Note:  Capacity of the /home file system varies from cluster to cluster. Unlike /scratch and /archive, the /home file system is not mounted across clusters. Each cluster has its own /home, its own user base and /home allocation policy.   

To purchase additional storage, send email to hpc@nyu.edu.

/scratch Policy

The /scratch storage system is a shared resource that needs to run as efficiently as possible for the benefit of all.  All HPC account holders have a /scratch disk space quota of 5TB and inode quota of 1 million. There is no system backup for data in /scratch, it is the user's responsibility to back up data. We cannot recover any data in /scratch, including files lost to system crashes or hardware failure so it is important to make copies of your important data regularly.

  • All inactive files older than 30 days will be removed.  It is a policy violation to use scripts to change the file access time. Any user found to be violating this policy will have their user's HPC account locked. A second violation may result in your HPC account being turned off.  
  • We strongly urge users to regularly clean up their data in /scratch to decrease /scratch usage by backing up files you need to retain either on /archive or elsewhere. 
  • All users will be asked to do cleanup if total /scratch usage is above 75%, which will decrease the scratch file system performance.
  • We retain the right to clean up files on /scratch at any time if it is needed to improve system performance.

There are some recommendations:

  • Do not put important source code, scripts, libraries, executables in /scratch. These important files should be stored in /home.
  • Do not make soft link for the folders in /scratch to /home for /scratch access
  • We strongly suggest users work with big size files, instead of many small size files.
  • For frequently accessed temporary files during job running process, please use local disk in the compute node or even RAM file system on the compute node to decrease IO load to /scratch file system.

Group Quotas on /archive

HPC accounts include an allocation of 2 TB of storage in /archive. An HPC sponsor may request that his/her quota and the quota of his/her research group be combined to create a single larger space. Some conditions:

  • Requests must be made by the sponsor
  • All of the members of the group must share the same sponsor
  • All group members must be active users of the HPC system

The sponsor's account will hold the full quota and each individual's quota will be set to 0.

Requests will be considered by HPC management and assessed by evaluating the need for it and availability.

Maximum size of group quota is 10 TB. Additional storage can be added for $500/TB/year (based on availability)

To apply for a group quota please use the form at this link.   You will receive a response to your request within 1 week.

Automatic File Deletion Policy

The table below describes the policy concerning the automatic deletion of files.

Space

Automatic File Deletion Policy

/home

none

/archive

none

/scratch

Files may be deleted as needed without warning if required for system productivity.

ALL

ALL /home and /archive files associated with expired accounts will be automatically deleted 90 days after account expiration.
/scratch files will automatically be deleted no later than 30 days after account expiration. 

  • No labels