From 0a26112a93a66c5b4832ac45f19886f605506416 Mon Sep 17 00:00:00 2001
From: jpekkila <johannes.pekkila@protonmail.com>
Date: Wed, 19 Aug 2020 09:48:39 +0000
Subject: [PATCH] Updated README.md that MPI runs are production-ready.

---
 README.md | 172 +++++++++++++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 171 insertions(+), 1 deletion(-)

diff --git a/README.md b/README.md
index 8cf4b52..fa350e1 100644
--- a/README.md
+++ b/README.md
@@ -50,7 +50,177 @@ In the base directory, run
 
 
 ## Standalone Module
+1
 
+# Astaroth - A Multi-GPU Library for Generic Stencil Computations {#mainpage}
+
+2
+
+​
+
+3
+
+[Specification](doc/Astaroth_API_specification_and_user_manual/API_specification_and_user_manual.md) | [Contributing](CONTRIBUTING.md) | [Licence](LICENCE.md) | [Repository](https://bitbucket.org/jpekkila/astaroth) | [Issue Tracker](https://bitbucket.org/jpekkila/astaroth/issues?status=new&status=open) | [Wiki](https://bitbucket.org/jpekkila/astaroth/wiki/Home)
+
+4
+
+​
+
+5
+
+Astaroth is a multi-GPU library for three-dimensional stencil computations. It is designed especially for performing high-order stencil
+
+6
+
+computations in structured grids, where several coupled fields are updated each time step. Astaroth consists of a multi-GPU and single-GPU
+
+7
+
+APIs and provides a domain-specific language for translating high-level descriptions of stencil computations into efficient GPU code. This
+
+8
+
+makes Astaroth especially suitable for multiphysics simulations.
+
+9
+
+​
+
+10
+
+Astaroth is licenced under the terms of the GNU General Public Licence, version 3, or later
+
+11
+
+(see [LICENCE.txt](LICENCE.md)). For contributing guidelines,
+
+12
+
+see [Contributing](CONTRIBUTING.md).
+
+13
+
+​
+
+14
+
+​
+
+15
+
+## System Requirements
+
+16
+
+* An NVIDIA GPU with support for compute capability 3.0 or higher (Kepler architecture or newer)
+
+17
+
+​
+
+18
+
+## Dependencies
+
+19
+
+Relative recent versions of
+
+20
+
+​
+
+21
+
+`gcc cmake cuda flex bison`.
+
+22
+
+​
+
+23
+
+## Building
+
+24
+
+​
+
+25
+
+In the base directory, run
+
+26
+
+​
+
+27
+
+1. `mkdir build`
+
+28
+
+2. `cd build`
+
+29
+
+3. `cmake ..`
+
+30
+
+4. `make -j`
+
+31
+
+​
+
+32
+
+> **Optional:** Documentation can be generated by running `doxygen` in the base directory. Generated documentation can be found in `doc/doxygen`.
+
+33
+
+​
+
+34
+
+> **Tip:**  The library is configured by passing [options](#markdown-header-cmake-options) to CMake with `-D[option]=[ON|OFF]`. For example, double precision can be enabled by calling `cmake -DBUILD_DOUBLE_PRECISION=ON ..`. See [CMakeLists.txt](https://bitbucket.org/jpekkila/astaroth/src/master/CMakeLists.txt) for an up-to-date list of options.
+
+35
+
+​
+
+36
+
+> **Note:** CMake will inform you if there are missing dependencies.
+
+37
+
+​
+
+38
+
+## CMake Options
+
+39
+
+​
+
+40
+
+| Option | Description | Default |
+
+41
+
+|--------|-------------|---------|
+
+42
+
+| CMAKE_BUILD_TYPE | Selects the build type. Possible values: Debug, Release, RelWithDebInfo, MinSizeRel. See (CMake documentation)[https://cmake.org/cmake/help/latest/variable/CMAKE_BUILD_TYPE.html] for more details. | Release |
+
+43
+
+| CUDA_ARCHITECTURES | Selects CUDA architecture support. Multiple architectures delimited by `;`. See (CMake documentation)[https://cmake.org/cmake/help/latest/prop_tgt/CUDA_ARCHITECTURES.html] for more details. | "60;70" |
 
 ```Bash
 Usage: ./ac_run [options]
@@ -78,7 +248,7 @@ Can I use the code even if I don't make my changes public?
 
 How do I compile with MPI support?
 
-> MPI implementation for Astaroth is still work in progress, these commands are for testing only. Invoke CMake with `cmake -DMPI_ENABLED=ON -DBUILD_SAMPLES=ON ..`. Otherwise the build steps are the same. Run with `mpirun -np 4 ./mpitest`.
+> Ensure that your MPI implementation has been built with CUDA support and invoke CMake with `cmake -DMPI_ENABLED=ON -DBUILD_SAMPLES=ON ..`. Otherwise the build steps are the same. Assign exactly one process per GPU and run with, for example, `srun --gres=gpu:v100:<ngpus per node> --ntasks-per-socket=<ngpus per node / NICs per node> -n <total number of gpus> -N <number of nodes> ./mpitest` or equivalent `mpirun` command.
 
 How do I contribute?