MUST detects usage errors of the Message Passing Interface (MPI) and reports them to the user. As MPI calls are complex and usage errors common, this functionality is extremely helpful for application developers that want to develop correct MPI applications. This includes errors that already manifest – segmentation faults or incorrect results – as well as many errors that are not visible to the application developer or do not manifest on a certain system or MPI implementation.
MPI Runtime Correctness Analysis
To detect errors, MUST intercepts the MPI calls that are issued by the target application and evaluates their arguments. The two main usage scenarios for MUST arise during application development and when porting an existing application to a new system. When a developer adds new MPI communication calls, MUST can detect newly introduced errors, especially also some that may not manifest in an application crash. Further, before porting an application to a new system, MUST can detect violations to the MPI standard that might manifest on the target system. MUST reports errors in a log file that can be investigated once the execution of the target executable finishes.
MUST provides correctness checks for the following classes of errors:
- Constants and integer values
- Communicator usage
- Datatype usage
- Group usage
- Operation usage
- Request usage
- Leak checks (MPI resources not freed before calling MPI Finalize)
- Type mis-matches
- Overlapping buffers passed to MPI
- Deadlocks resulting from MPI calls
- Basic checks for thread level usage (MPI_Init_thread)
Current Release (Version 1.7.2)
Alternative stack trace implementation based on backward-cpp.
See below for older releases.
Must Feature-Preview (Version 1.8.0-preview)
This preview version introduces two new kinds of analysis, both depending on LLVM support.
TypeART support introduces checks for type-matching with application buffers.
ThreadSanitizer support introduces data race detection for MPI communication buffers.
The preview version depends on LLVM 10.
For more details, check out the README.md file in the tarball.
Scalability of MUST depends strongly on the scalability of the attached application. For high scaling applications we successfully analyzed up to 16'000 parallel processes.
- MUST depends on the infrastructure libraries PnMPI and GTI.
- For installation you need a build environment including CMake 3.9 or higher and python 2.6 or 2.7
- GTI depends on libxml2. The headers can be found in the libxml2-dev(el) package of your distro.
- For more detailed error output you optionally may install dyninst from the dyninst website.
- Another optional dependency is graphviz for graphical representations of deadlock and datatype error situations.
For details on installation visit the Documentation (pdf).
Older MUST Releases
Version 1.7.1 (June 2021)
Fixes compatibility for applications using petsc, fixes some false alerts, fixes build system issues
Version 1.7.0 (March 2021)
Enables building with OpenMPI 4. It also fixes several reported issues.
The support for hybrid OpenMP + MPI applications is more stable.
Version 1.6.0 (December 2019)
Includes fixes for building with certain MPI versions. Fixes some issues in Fortran codes. Starting with version 1.6.0, MUST, GTI, and PnMPI are delivered in a single archive file.
Version 1.6.0-rc3 (January 2019)
This version of MUST adds missing MPI-3 base datatypes; fixes some installation issues; speeds up tool startup with prebuilt configurations (see updated documentation); and adds support for multi-threaded applications. Starting with version 1.6.0, MUST, GTI, and PnMPI are delivered in a single archive file.
Version 1.6.0-rc1 (March 2018)
This version of MUST fixes some compatibility issues with newer versions of system libraries. Adds analysis for MPI-3 non-blocking collectives. And includes the build of GTI and PnMPI into a single build. Starting with version 1.6.0, MUST, GTI, and PnMPI are delivered in a single archive file.
- see also Known issues
Version 1.5.0 (July 2016)
- MUST Tool: must-1.5.0.tgz
- GTI Infrastructure: gti-for-must-1.5.0.tgz
- PnMPI Infrastructure: pnmpi-for-must-1.5.0.tgz
Version 1.5.0-rc1 (February 2016)
- MUST tool: must-1.5.0rc1.tgz
- GTI infrastructure: gti-for-must-1.5.0rc1.tgz
- PnMPI infrastructure: pnmpi-for-must-1.5.0rc1.tgz
Version 1.4.0 (March 2015)
This version of the tool is supposed to support MPI-3.0 compatible libraries
- MUST tool: must-1.4.0.tgz
- GTI infrastructure: gti-for-must-1.4.0.tgz
- PnMPI infrastructure: pnmpi-for-must-1.4.0.tgz
- Documentation: Documentation-1.4.0.pdf
Version 1.4.0-rc1 (November 2014)
- MUST tool: must-1.4.0rc1.tgz
- GTI infrastructure: gti-for-must-1.4.0rc1.tgz
- PnMPI infrastructure: pnmpi-for-must-1.4.0rc1.tgz
Version 1.3.0 (March 2014):
- MUST tool: must-1.3.0.tgz
- GTI infrastructure: gti-for-must-1.3.0.tgz
- PnMPI infrastructure: pnmpi-for-must-1.3.0.tgz
- Documentation: Documentation-1.3.0.pdf
MUST Known issues
- Version 1.6-rc1: during execution pnmpif.so cannot be found. Workaround:
- export LD_LIBRARY_PATH=<must-install-dir>/lib:$LD_LIBRARY_PATH
this is fixed in 1.6-rc3
- Version 1.6-rc3: Some issues with Fortran file-IO and datatype functions are still under investigation
Please use the command mustrun --help or have a look at the documentation to get a mail address for bug reports, feedback and feature request.