Books and tutorials
High Performance Computing, 2nd edition, Kevin Dowd & Charles Severance, O'Reilly Publishers, 1998 ISBN 1-56592-312-X
- Excellent general introduction to all concepts within the world of parallel computers and modern supercomputers. Processor architecture, memory. Programming, optimization, performance evaluation. Comparison between shared memory parallelization (threads, OpenMP), message passing (MPI, PVM), and the HPF standard. It will be a lot easier to understand technical documentation of manufacturers, for any machine type, after having read this book (particularly Chapter 3, covering memory).
Cache Advantage by David F. Bacon, in BYTE magazine, august 1994, page 79-86
- Covers a little more details and clarity on the subject of cache memory. Excellent complement to Chapter 3 of the above book.
Performance Optimization of Numerically Intensive Codes, by Stefan Goedecker & Adolfy Hoisie, SIAM Publishers, 2001 ISBN 0-89871-484-2
- Architecture of superscalar processors and memory. Optimization of floating point operations, memory access, and parallelism. Coarse and fine grain parallelism. Important examples. Diagnostic programs are given in the annex and on the internet. This books put a certain accent on IBM machines (SP and workstations); it gives various comparisons including also the Origin 2000 machines.
If you must choose (because of time or money constraint), between the book by Dowd & Severance and the book by Goedecker & Hoisie, and if you are not an experienced programmer, I (J.R.) would recommend, without hesitation, the first one, even if it is a bit older and bigger. It gives fewer examples of programs, but explains concepts and problems better and in more detail, and also explains how compilers work and their limitations.
Numerical methods and simulations
Lecture notes for PHQ 405 : Méthodes numériques et simulations", prepared by David Sénéchal of the University of Sherbrooke, which presents the main numerical methods encountered in physics. Available in letter format and in iPad format.
On parallel programming
Introduction to Parallel Computing Tutorial, Lawrence Livermore National Laboratory
- Generic tutorial on parallel programming.
Programming on Parallel Machines; GPU, Multicore, Clusters and More, Professor Norm Matloff , University of California, Davis
- Free online book that summarizes a vast range of subjects: MPI, OpenMP, Cuda, and Thrust programming, cloud computing, everything is there, including a wide variety of algorithms such as matrix multiplication, sorting data, statistics, and image processing.
Parallel Programming in OpenMP, by Rohit Chandra et al., Academic Press, 2001 ISBN 1-55860-671-8
A good book that concentrates on OpenMP, in particular for Fortran 90 codes. It discusses most OpenMP directives which are of normally of interest to scientific programmers, and there are also exercises at the end of each chapter. Most examples in the book use Fortran 90 but there is always a reference to using C/C++.
Practical Parallel Programming, by Barr E. Bauer, Academic Press, 1992 ISBN 0-12-082810-3
- Excellent general introduction to shared memory parallel programming (concepts, difficulties, solutions, automatic parallelization tools). This book does not describe the OpenMP standard, which was established more recently, but one can find the essentials of OpenMP there.
It describes "MP" directives, which precede OpenMP on SGI computers, which a set of directives which are still supported today (but on the way out), and which served as an inspiration to OpenMP's founders. This book is illustrated with examples in Fortran and C.
A “Hands-on” Introduction to OpenMP , openmp.org
OpenMP Tutorial, Lawrence Livermore National Laboratory
Using MPI, by William Gropp, Ewing Lusk, and Anthony Skjellum, MIT Press, 1999 ISBN 0-262-57132-3
- Without doubt one of the best books on the subject. Describes MPI exhaustively and introduces the details and tools (procedures and functions) available in an incremental fashion, with the help of concrete examples (C, C++ et FORTRAN 90 and 77), where the source codes are available on the Internet. Note however that one of the chapter describes the MPE library, containing graphical and diagnostic utilities, is not part of the standard MPI distribution.
Using MPI-2, by William Gropp, Ewing Lusk, et Rajeev Thakur, MIT Press, 1999 ISBN 0-262-57133-1
- Complementing the previous book, covering new features introduced by the new MPI-2 standard. Covers mainly parallel input/output and remote memory access. Explains also how to combine MPI and threads for better utilization of shared memory machines.
Tutorial on MPI de William Gropp, Mathematics and Computer Science Division, Argonne National Laboratory
Message Passing Interface (MPI) Tutorial Lawrence Livermore National Laboratory
Advanced MPI Lawrence Livermore National Laboratory