5.0.x FeatureList

Pre-emptively creating the 5.0.x feature list page just to record thoughts and rationale here.

Features deferred from 4.0.x

• Remove TKR version of use mpi module.

  • This was deferred from 4.0.x because in April/May 2018, it was discovered that:
    1. The RHEL 7.x default gcc (4.8.5) still uses the TKR mpi module
    2. The NAG compiler still uses the TKR mpi module.
  • Is it time to re-evaluate? (RHEL 7 is going to be around for a while -- it may take a while before we can delete the the TKR mpi module, but keeping this item here for completeness)

• Remove all deleted MPI-1 and MPI-2 functionality

  • E.g., MPI_ATTR_DELETE, C++ bindings, etc.
  • These were all been marked as "deprecated" (possibly in 2.0.x series? definitely by the 3.0.x series).
  • None of them build by default in the v4.0.x series, unless you use configure CLI options to enable them.
  • The idea is that the deprecated/don't-build-by-default warnings in 2.0.x/3.0.x/4.0.x are enough to allow us to actually delete this stuff in v5.0.x.

• Need to revisit what we want to do w.r.t. RoCE and iWARP support in 5.0.0. History:

  • The openib BTL was almost deleted in v4.0 because Mellanox has long made it clear that their path forward is UCX.
  • That basically left IB / RoCE support under openib as best effort/community.
  • Chelsio, however, supported iWARP in openib. But still, the openib BTL was pretty much abandoned... but still used by default because libibverbs is inbox in most distros while UCX isn't (yet).
  • As such, there was discussion a) forking the openib btl into a new iwarp BTL (and stripping out lots of code that iWARP didn't use), and b) deleting the openib btl. The assumption was that RoCE devices would move to UCX.
  • However, it wasn't 100% clear if non-Mellanox RoCE devices worked in UCX. Broadcomm agreed to test, but not in time for v4.0.0.
  • Additionally, we later discovered that Libfabric supports both RoCE and iWARP, and its support for these 2 will get only better in the upcoming Fall 2018 Libfabric v1.7 release (which is also too late for Open MPI v4.0.0).
  • Hence, it seems like the future of RoCE and iWARP is either or both of Libfabric and UCX.
    • ...but neither of those will be 100% ready for Open MPI v4.0.0.
    • It didn't seem to make sense to make iWARP users move from openib to iwarp in v4.0.0 (and potentially something similar for non-Mellanox RoCE users), and then move them again to something else in v5.0.0 ("How to annoy your users, 101").
    • The lowest cost solution for v4.0.0 was to disable IB support by default in openib (i.e., only iWARP and RoCE will use it by default), and punt the ultimate decision about potentially deleting the openib BTL to v5.0.0. Note that v4.0.0 will also have a "back-door" MCA parameter to enable IB devices, for the "just in case" scenarios (where users, for whatever reason, who don't want to upgrade to UCX).
  • With all that, need to investigate and see what the Right course of action is for v5.0.0 (i.e., re-evaluate where Libfabric and/or UCX are w.r.t. RoCE support for non-Mellanox devices and iWARP support), and how to plumb that support into Open MPI / expose it to the user.

• (UTK) Better multithreading. - George

  • In OB1 PML, normal OMP parallel Sections. Improved for injection and extraction rates.
  • Implications for other PMLs. Very OB1 specific Maybe a little bit in progress.

• (UTK) ULFM support via new MPIX functions. Most is in MPIX, but some in PML.

  • Depends on PMIx v3.x

• Want Nathan's fix for Vader and other BTL to allow us to have SOMETHING for OSC_RDMA for one-sided + MT runs.

  • something similar coming into BTL-TCP

• Change defaults for embedding libevent / hwloc (see this issue) - HELP NEEDED see PR 5395