MUST

MUST detects usage errors of the Message Passing Interface (MPI) and reports them to the user. As MPI calls are complex and usage errors common, this functionality is extremely helpful for application developers that want to develop correct MPI applications. This includes errors that already manifest – segmentation faults or incorrect results – as well as many errors that are not visible to the application developer or do not manifest on a certain system or MPI implementation.

Permalink: https://itc.rwth-aachen.de/must/

MPI Runtime Correctness Analysis

To detect errors, MUST intercepts the MPI calls that are issued by the target application and evaluates their arguments. The two main usage scenarios for MUST arise during application development and when porting an existing application to a new system. When a developer adds new MPI communication calls, MUST can detect newly introduced errors, especially also some that may not manifest in an application crash. Further, before porting an application to a new system, MUST can detect violations to the MPI standard that might manifest on the target system. MUST reports errors in a log file that can be investigated once the execution of the target executable finishes.

MUST provides correctness checks for the following classes of errors:

  • Constants and integer values
  • Communicator usage
  • Datatype usage
  • Group usage
  • Operation usage
  • Request usage
  • Leak checks (MPI resources not freed before calling MPI Finalize)
  • Type mis-matches
  • Overlapping buffers passed to MPI
  • Deadlocks resulting from MPI calls
  • Basic checks for thread level usage, especially with OpenMP
  • Data races involving MPI function calls
Nicht verfügbar

Current Release (Version 1.10.0)

- Use TSan Fibers to analyze local buffer races with non-blocking communication
- Integrate MPI-handle-shim to solve issues with request and group handles
- Experimental support for MPI Sessions in GTI / PnMPI
- Allow moving html output
- Fix DL detection to support unknown requests
- Fix internal data races in multi-threaded analysis

MUST Scalability

Scalability of MUST depends strongly on the scalability of the attached application. For high scaling applications we successfully analyzed up to 16'000 parallel processes.

MUST Dependencies

  • MUST depends on the infrastructure libraries PnMPI and GTI (shipped with the tarball).
  • For installation you need a build environment including CMake 3.13.4 or higher and Python 3
  • GTI depends on libxml2. The headers can be found in the libxml2-dev(el) package of your distro.
  • For more detailed error reporting with stacktraces, MUST uses Backward which requires a library to unwind the stack (libdwarf) and a library to read debug infos (libbfd, libdw, libdwarf).
  • Optionally, dyninst can be used to generate stacktraces.
  • Another optional dependency is graphviz for graphical representations of deadlock and datatype error situations.

MUST Installation

For details on installation visit the Documentation (pdf).

Older MUST Releases

Version 1.9.2 (November 2023)

- Add option to send error reports via mail
- Add option to enable / disable TSan report functionality
- Fix several bugs in mustrun when using prebuilt infrastructure
- Fix build error in PnMPI due to wrong install locations in CMake
- Add missing predefined MPI_Ops (MPI_REPLACE and MPI_NO_OP)
- Change minimum required CMake version from 3.9 to 3.13.4 and fix compatibility with latest CMake

Version 1.9.1 (July 2023)

- Fix compatibility with older versions of CMake
- Fix some issues with backward, allow to disable specific debugging libraries in backward
- Mustrun returns an error code for all output formats
- Fix several bugs in analyses

Version 1.9.0 (May 2023)

- Added support for partitioned P2P communication
- Added support for MPI 4 functions
- Added JSON as new output format
- Reports use of MPI functions where MUST does not apply any analysis
- Added OMPT-based OpenMP analyses
- Integrated LLVM testing infrastructure to allow stand-alone testing

Version 1.8.0 (November 2022)

- Integrated ThreadSanitizer to detect data races involving MPI function calls in hybrid applications
- Employ TypeART to check for type inconsistencies in MPI function calls
- Allow filtering of MUST's output, splitting of HTML output across multiple files, and specification of the output path for HTML output
- Move from CTests to LLVM's lit for the test suite

Version 1.7.2 (October 2021)

Alternative stack trace implementation based on backward-cpp.

Version 1.8.0-PREVIEW (March 2021)

Alternative stack trace implementation based on backward-cpp.

Version 1.7.1 (June 2021)

Fixes compatibility for applications using petsc, fixes some false alerts, fixes build system issues

Version 1.7.0 (March 2021)

Enables building with OpenMPI 4. It also fixes several reported issues.
The support for hybrid OpenMP + MPI applications is more stable.

Version 1.6.0 (December 2019)

Includes fixes for building with certain MPI versions. Fixes some issues in Fortran codes. Starting with version 1.6.0, MUST, GTI, and PnMPI are delivered in a single archive file.

Version 1.6.0-rc3 (January 2019)

This version of MUST adds missing MPI-3 base datatypes; fixes some installation issues; speeds up tool startup with prebuilt configurations (see updated documentation); and adds support for multi-threaded applications. Starting with version 1.6.0, MUST, GTI, and PnMPI are delivered in a single archive file.

Version 1.6.0-rc1 (March 2018)

This version of MUST fixes some compatibility issues with newer versions of system libraries. Adds analysis for MPI-3 non-blocking collectives. And includes the build of GTI and PnMPI into a single build. Starting with version 1.6.0, MUST, GTI, and PnMPI are delivered in a single archive file.

Version 1.5.0 (July 2016)

Version 1.5.0-rc1 (February 2016)

Version 1.4.0 (March 2015)

This version of the tool is supposed to support MPI-3.0 compatible libraries

Version 1.4.0-rc1 (November 2014)

Version 1.3.0 (March 2014):

MUST Known issues

  • Version 1.6-rc1: during execution pnmpif.so cannot be found. Workaround:
    • export LD_LIBRARY_PATH=<must-install-dir>/lib:$LD_LIBRARY_PATH

    • this is fixed in 1.6-rc3

  • Version 1.6-rc3: Some issues with Fortran file-IO and datatype functions are still under investigation

Feedback

Please use the command mustrun --help or have a look at the documentation to get a mail address for bug reports, feedback and feature request.