List the available MPI variables in the documentation

[cponder]

Your page here

https://www.open-mpi.org/doc/current/

lists all the subroutines but should also list the environment-variables that are available to use.
I always have a hard time finding this info in the documentation.

[jsquyres]

Thanks @cponder.

We are switching to an all-new documentation system for the upcoming Open MPI v5.0.x series -- see https://docs.open-mpi.org/en/v5.0.x/. Do you have a suggestion for where the relevant environment variables should be in that new tree of documentation?

[cponder]

One thing I'd want is, where each mpirun command-line flag is described, also list the equivalent environment-variable if there is one.
I'd add a chapter in section 9 on Environment Settings.

[jsquyres]

FWIW, there's https://docs.open-mpi.org/en/v5.0.x/man-openmpi/man1/mpirun.1.html, but I think you're more asking about MCA parameters -- not just strictly mpirun(1) CLI options.

For listing what MCA parameters are available, this very much depends on how Open MPI was configured on your system. As such, ompi_info(1) exists to list all the configuration options for your setup (https://docs.open-mpi.org/en/v5.0.x/man-openmpi/man1/ompi_info.1.html).

That being said, MCA parameters can be set multiple ways -- on the command line, as environment variables, or in files. See https://docs.open-mpi.org/en/v5.0.x/running-apps/tuning.html#setting-mca-parameter-values.

[cponder]

I see an explanation of how the environment-variables would be set, but only a few ever being mentioned by name and only in random locations.
WIth OMPI_LDFLAGS, for example, I see it in the FAQ but not in the ompi_info page.
I want to see the complete list of what environment-variables are available.
MVAPICH2 has a whole section on them:

http://mvapich.cse.ohio-state.edu/static/media/mvapich/mvapich2-userguide.html#x1-17600010

[rhc54]

These are the OMPI vars that are provided regardless of configuration:

OMPI_UNIVERSE_SIZE=10
OMPI_COMM_WORLD_SIZE=1
OMPI_WORLD_SIZE=1
OMPI_MCA_num_procs=1
OMPI_COMM_WORLD_LOCAL_SIZE=1
OMPI_WORLD_LOCAL_SIZE=1
OMPI_NUM_APP_CTX=1
OMPI_FILE_LOCATION=/Users/rhc/tmp/prte.Ralphs-iMac-2.1000/dvm.33899/1/0
OMPI_MCA_initial_wdir=/Users/rhc/pmix/dynasched
OMPI_COMMAND=env
OMPI_ARGV=
OMPI_MCA_cpu_type=x86_64
OMPI_COMM_WORLD_RANK=0
OMPI_COMM_WORLD_LOCAL_RANK=0
OMPI_COMM_WORLD_NODE_RANK=0

Is that what you are seeking?

[cponder]

Yes, those in particular. Are they listed somewhere?
But also debug settings and the ones affecting the compilation.
I'd be inclined to list them in groups.

[rhc54]

I'm unaware of them being listed anywhere, though I imagine someone could add them to the docs if not already present. Debug settings are governed by the MCA params, as Jeff indicated, and probably harder to put into docs. Compilation envars are just the normal CC, LDFLAGS, and friends - nothing specific to OMPI that I am aware of.

[cponder]

If the OMPI_LDFLAGS is valid, it would be fed specifically to the mpicc etc. compilers but ignored by gcc etc.
If it really does get used, I would want a sub-section for these kinds of parameters.

I get that the MCA parameters would derive from other parts of the system, and you can generate the list and use the rule to compose them as OMPI_MCA_ variable-names (which I use occasionally). My preference would still be to have the complete list in the doc in its own sub-section if it's possible.

[rhc54]

I've never heard of OMPI_LDFLAGS. A quick search of the code base shows it appears in some doc, but nowhere in the code itself is it used. My guess is that someone either incorrectly included it in the docs, or did so in anticipation of someone implementing it someday.

I defer to the OMPI people about the MCA params being in the doc. They could, I suppose, use the respective "info" commands and a tool to scrape them from the two submodules (PMIx and PRRTE) plus the OMPI level and combine them into a doc. Would require someone to write that code and I'm not sure who has the time.

[jjhursey]

Looking at Open MPI main, I see the following envars that are not OMPI_MCA prefixed (all MCA parameters have a recipe for turning them into envars). Note that not all of these should be set by the end user - so they need some pruning and double checking.

OMPI_ARGV
OMPI_BREAKPOINT
OMPI_COMMAND
OMPI_MPI_THREAD_LEVEL
OMPI_PARENT_PORT
OMPI_PRTERUN
OMPI_SHOW_MODEL_CALLBACK
OMPI_TIMING_ENABLE
OMPI_UNIVERSE_SIZE
--
OPAL_DESTDIR
OPAL_OUTPUT_INTERNAL_TO_STDOUT
OPAL_OUTPUT_REDIRECT
OPAL_OUTPUT_STDERR_FD
OPAL_OUTPUT_SUFFIX
OPAL_OUTPUT_SYSLOG_IDENT
OPAL_OUTPUT_SYSLOG_PRI
OPAL_PREFIX
OPAL_SYS_PARAMS_GIVEN
OPAL_TIMING_ENABLE
OPAL_USER_PARAMS_GIVEN
--
OMPI_MPICC (alias for OMPI_CC)
OMPI_MPICFLAGS (alias for OMPI_CFLAGS)
--
OMPI_MPICXX (alias for OMPI_CXX)
OMPI_MPICXXFLAGS (alias for OMPI_CXXFLAGS)
--
OMPI_MPIFC (alias for OMPI_FC)
OMPI_MPIFCFLAGS (alias for OMPI_FCFLAGS)
--
OMPI_MPICPPFLAGS (alias for OMPI_CPPFLAGS)
OMPI_MPILDFLAGS (alias for OMPI_LDFLAGS)
OMPI_MPILIBS (alias for OMPI_LIBS)
--
PRRTE_LAUNCHED
PRTE_PREFIX
--
PMIX_TEST_DEBUGGER_ATTACH
--
TMPDIR
TEMP
TMP
--
IPATH_NO_BACKTRACE
HFI_NO_BACKTRACE
PSM2_CUDA
PVFS2_NCACHE_TIMEOUT
PMI_GNI_PTAG
PMI_GNI_COOKIE

Per conversation, the OMPI_LDFLAGS is part of the wrapper compiler code generation see here - it's a bit difficult to see, but this is where it is checked for.

[rhc54]

I'm not familiar with many of those. The ones I listed are all set by the runtime and not the user. Note that for historical reasons, there are a few OMPI_MCA_xxx envars that are in fact set by the runtime and are not user-settable (they are in the list I provided). Perhaps you folks might want to consider deprecating and replacing them someday with non-MCA versions to avoid confusion?

[vsoch]

Hiya! I'm looking for the environment variable equivalents for:

• --map-by numa
• --rank-by core
• --bind-to none

We are interacting with openmpi through Flux so we aren't using mpirun, but the bindings in flux. I also did some searching for a page with a listing but couldn't find anything (and found this thread instead). Can we just use OMPI_ARGV?

ompi/ompi/runtime/ompi_mpi_init.c

Line 388 in 5dc2398

if (NULL == getenv("OMPI_ARGV") && 1 < argc) {

[vsoch]

I think maybe these?

OMPI_MCA_rmaps_base_ranking_policy=core
OMPI_MCA_rmaps_base_mapping_policy=numa
OMPI_MCA_hwloc_base_binding_policy=none

The question is whether they would get picked up by bindings without using mpirun directly?