Using available storage
On a Linux cluster, various types of storage are commonly available. The list of choices varies from one system to the next. Hence it is important to view what the system you are using offers you. Please find server-specific documentation at the bottom of the page. The appropriate choice for your needs depends on multiple parameters:
- What is the size of the files in question.
- How many files are needed?
- Are your file temporary or do they need to be preserved?
- What is the files' format?
- Is the file accessed sequentially?
Once you have answered these questions, you can choose what kind of storage is most appropriate for your needs.
- Only use text format for files that are smaller than a few MB.
- As far as possible, use local storage for temporary files.
- If your program must search within a file, it is fastest to do it by reading it in completely before searching, or to use a RAM disk ($RAMDISK or /dev/shm).
- Regularly clean up parallel file systems, because those systems are used for huge data collections.
- If you no longer use certain files, compress them (you should group them before) and back them up (if possible).
- If your needs are not well served by the available storage options please contact Calcul Québec's user support team
Storage options are distinguished by the available hardware, access mode and write system. Typically, the most systems offer the following storage types :
- Network file system (NFS)
- This type of storage is generally equally visible on both login and compute nodes. This is the appropriate place to put files that are regularly used: source code, programs and configuration files. This type of storage offers performance comparable to a conventional hard disk.
- Parallel file system (Lustre or GPFS)
- This type of storage is generally equally visible on both login and compute nodes. Combining multiple disk arrays and fast servers, it offers excellent performance for large files and large input/output operations. Often two types of storage are distinguished on such systems: long term storage and temporary storage (scratch). Performance is subject to variations caused by other users.
- Local file system
- This type of storage consists of a local hard drive at every compute node. Its advantage is that its write performance is stable, because only one user can use it. However, you should not forget to repatriate your files before closing your session because everything will be cleaned after each job.
- RAM (memory) file system
- This is a file system that exists within a compute node's RAM. So its use reduces available memory for computations. Such file systems are very fast for small files and particularly faster than other systems when file access is random. A RAM disk is always cleaned at the end of a session.
The following table summarizes the properties of these storage types.
|Storage type||Typical directory name||Accessibility||Throughput (large operations, > 1 MB per operation)||Latency (small operations)||Life span|
|Network File System (NFS)||$HOME||All nodes||100 MB/s shared||High||Long term|
|Long term parallel file system||$HOME, $RAP, /home, /sb/project/, /gs/project||All nodes||1-10 GB/s shared||High||Long term|
|Short term parallel file system||$SCRATCH||All nodes||1-10 GB/s shared||High||Short term (periodically cleaned)|
|Local file system||$LSCRATCH||Local on the node||100 MB/s||Medium||Very short term|
|Memory (RAM) file system||$RAMDISK, /dev/shm||Local on the node||1-10 GB/s||Very short||Very short term, cleaned after every job|
Storage units are distinguished by the hardware, access and available write system. Typically, most systems offer the following storage types (note that the exact names may change from one server to the the other):
- This is the directory where you arrive when you start a session. You can see it on all login nodes and all worker nodes. This is the appropriate place to put files that you regularly use: source code, programs, and configuration files. Often this data is backed up and can be retrieved in case of loss. For Cottos, Mp2, Ms2, and Psi this is a network file system, where as it is a parallel file system for Briarée, Colosse, Guillimin and Hadès.
- This directory is placed on a parallel filesystem, Lustre (for Colosse, Mp2, Ms2 and Cottos) or GPFS (for Briarée and Guillimin). It is generally visible from all nodes. Using it is very fast for large files, but not very efficient for many small files. This is the appropriate place to store large files that you use for a few days or weeks only. Periodically, it may be automatically cleaned (files being deleted).
- If available, this storage unit is local, since it is located on every worker node's hard drive. Its advantage is that its write performance is stable, because only one user can use is. However, you should not forget to repatriate your files before closing the session because this space will be cleaned after every job.
- This unit is located on a virtual drive within the node's RAM. So its use reduces available memory for computations. Such file systems are very fast for small files and particularly faster than other systems when file access is random. A RAM disk is always cleaned at the end of a session.
Certain systems contain a longer list of storage types. To get to know the specifics for the system you use, look at the tabs at the bottom of this page.
As for all units, various choices exist for your files' format. The number of choices depends on the application's type (serial or parallel), of the language it is written in (C, C++, Fortran, Python, etc.), of the size of the data you need to write, etc. Below we describe the most commonly used formats. The first two are the base formats on top of which the others are constructed.
- Also called ASCII (but other character encodings are possible), this format is usually human readable. It may be edited and modified in any text editor. However, reading and writing using this format is slow, and files use more space. Although portable, this format may require some changes, especially regarding carriage return and line feeds. This format is practical for configuration and parameter files. Structured files like XML files use this format. This file format can be read and written by any programming language.
- The main inconvenience with this file format is that it is not human readable even though some editors can edit those files. It however provides much faster reading and writing speed and requires less space than the text format. Portability is somewhat limited if you change platform, due to endianness issues. This format can be read and written by any language, but portability is limited between Fortran and other languages.
- Only for parallel codes using MPI. This format is a subcategory of binary format. The difference with respect to binary is rather in the process of reading and writing than in the file itself. Using MPI-IO may help making a code independent of the number of MPI ranks (compared to writing in binary with only one rank). Reading/writing speed is similar to binary, but can be faster on a parallel filesystem such as Lustre or GPFS depending on the network. Portability issues related to Fortran are partially avoided.
- This library makes structuring complex data easier. Since it is a self-describing format, it helps maintaining a code with changing data schemes. Endianness issues are managed by the library, making the files more portable. Reading/writing speeds are similar to what can be achieved with MPI-IO. HDF5 also supports compression which may reduce the file sizes, and contains optimizations for Lustre and GPFS.
- This library is also a library for structuring complex data. Since version 4, it is built on top of HDF5. The main advantage is its interface which is simpler than that of HDF5. However, it only offers a subset of the capabilities of HDF5.
|Text||Larger than required||Slow||Very good|