This is a Core Concept page which offers an overview of key topics essential for understanding Storage. Start here for foundational knowledge, then explore our guides and tutorials for more in-depth learning.

Introduction

Storage is a key element in high performance computing, as.In order to ensure optimal performance we use a combination of efficient cold storage aimed at storing source and results files, as well as shared storage attached to your computing cluster and used during computation. In order to ensure optimal performance we use a combination of efficient cold storage aimed at storing source and results files, as well as shared storage attached to your computing cluster and used during computation.

Glossary

Cold storage - Storage used to store data permanently with infrequent access (read / write).
Hot Storage - Temporary storage used while the task is computing

Fundamental Principles

Transfer from/to your task execution environment

Just like on your local machine, on Qarnot HPC, your data is stored in folders which are physically located in one of our long term storage server. When you launch a task, this data is transferred to an on-site temporary storage system and then mounted as a shared folder to your task execution environment. All files are accessible on /job unless you specified otherwise in your task setup.

During the file transfer your task has the status Partially Dispatched or Fully Dispatched.

When your task has completed, the data in /job that has been edited is transferred to your destination folder, the status is then Uploading Results.

Billing

10GiB of Storage is included with any Qarnot Account. This means that you can save 10GiB of data . Please contact us to expand your storage capacity.

The ephemeral storage of data generated while computing (3D rendering, CFD solving, Data processing ….) and not saved to your folder is included in the computing prices.

Data transfers fees (ingress, egress) are not billed.

Core Components

Folder

To store your data while it is not being processed we use S3-compatible object storage, which means you can use any S3-compatible tool to interact with your cold data. As this data is meant to be processed on Qarnot or consists in computation results it is not meant to be stored for months on the folders. As a consequence, we do not ensure redundancy nor high availability.

Local Cluster Storage

Once your data is transferred to your cluster to be processed it is stored on a local storage server and attached to your simulation, you can access it using the /job folder.

Temporary data is stored on this server under the /cache directory for local data or /share if the data is to be shared between task instances

White Listing / Black Listing

To precisely define which files you want to compute with, snapshot or download at the end of your computation, Qarnot offers you the possibility to whitelist (authorize) or blacklist (exclude) files or folders from the file upload, snapshot or download operations.

File selection (or exclusion) is done using a Regex. For example to select all image files you can use the following regex [^\\s]+(.*?)\\.(jpg|jpeg|png|gif|JPG|JPEG|PNG|GIF)$.

For more information please read our whitelist/black list documentation.