Variable forwarding in case of spawn not working for children

[hamme74]

Running an MPI simulation with MPI_Comm_spawn and setting an MPI simulation does not work as expected.
The parent process can see the variable set by "-x", but the children cannot.

Open MPI v5.0.9 from tar.gz
hwloc v2.12.2 from tar.gz
OpenPMIx master branch from 2025-09-08 from git (also tested: v6.0.0 from tar.gz)
prrte: master branch from 2025-09-08 from git (also tested: v4.0.0 from tar.gz)
libfabric v2.2.0 from tar.gz

Tested on RHEL 8.10.

Code parent

#include <filesystem>
#include <iostream>
#include <mpi.h>

std::filesystem::path GetInstallPath(char *argv[]) {
    std::filesystem::path myInstallPath = std::filesystem::absolute(argv[0]);
    myInstallPath = myInstallPath.parent_path();
    myInstallPath = std::filesystem::canonical(myInstallPath);
    return myInstallPath;
}

int main(int argc, char *argv[]) {
    MPI_Init(&argc, &argv);

    char const* pUhu =std::getenv("uhu");
    if (pUhu) {
        std::cout << "parent  uhu: " << pUhu << std::endl;
    }

    std::filesystem::path sInstallPath = GetInstallPath(argv);
    std::string const sChildPath = (sInstallPath / "child.out").string();

    int iRank = -1;
    MPI_Comm_rank(MPI_COMM_WORLD, &iRank);

    int iSize = -1;
    MPI_Comm_size(MPI_COMM_WORLD, &iSize);

    int iName = -1;
    char sName[MPI_MAX_PROCESSOR_NAME];
    MPI_Get_processor_name(sName, &iName);

    std::cout << "I am a parent with rank: " << iRank << ", " << sName << std::endl;

    MPI_Comm parentcomm;
    MPI_Comm_get_parent(&parentcomm);

    MPI_Comm intercomm;

    MPI_Info info;
    MPI_Info_create(&info);
    MPI_Info_set(info, "map_by", "node");

    MPI_Comm_spawn(sChildPath.c_str(), MPI_ARGV_NULL, iSize, info, 0, MPI_COMM_WORLD, &intercomm, MPI_ERRCODES_IGNORE);
    MPI_Info_free(&info);

    MPI_Finalize();
    return 0;
}

Code child

#include <cstdlib>
#include <iostream>
#include <mpi.h>

int main(int argc, char *argv[]) {
    MPI_Init(&argc, &argv);

    char const* pUhu =std::getenv("uhu");
    if (pUhu) {
        std::cout << "child uhu: " << pUhu << std::endl;
    }

    int iRank = -1;
    MPI_Comm_rank(MPI_COMM_WORLD, &iRank);

    int iName = -1;
    char sName[MPI_MAX_PROCESSOR_NAME];
    MPI_Get_processor_name(sName, &iName);

    std::cout << "I am a child with rank: " << iRank << ", " << sName << std::endl;
    system("taskset -cp $$");

    MPI_Finalize();
    return 0;
}

$ export uhu=666
$ /opt/openmpi/bin/mpiexec --runtime-options fwd-environment --map-by node:oversubscribe -x uhu=777 --host node1,node2 -n 2 ./parent.out

The call above results in:
parent uhu: 777
parent uhu: 777
I am a parent with rank: 0, node1
I am a parent with rank: 1, node2
child uhu: 666
child uhu: 666
I am a child with rank: 1, node2
I am a child with rank: 0, node1
pid 667310's current affinity list: 0-7
pid 668545's current affinity list: 0-7

Even if --fwd-env is not set, only the parent process sees the variable "uhu".
However, the children should see the variable, too.
This would make the behavior consistent with --fwd-env.

Just to mention it:
In the latest version of pmix and prrte, fwd-environment works as expected.

Also --unset-env, --prepend-env and --append-env should be checked. I did not test these options, but I expect, that they will only modify the parent.

Many thanks in advance for looking into this issue.

[rhc54]

Hmmm...seems like a never-ending debate over what should automatically be inherited by child jobs. I think in this case we'll err on the side of consistency with prior behavior and not automatically inherit environment directives, but inherit if directed to do so.

You'll find the fix in openpmix/prrte#2266. The forward environment flag isn't required (unless you want to forward the entire local environment), but you'll need to add :inherit to the map-by option - e.g., --map-by node:oversubscribe:inherit

[rhc54]

Had to update this a bit as the initial fix wasn't quite complete. See openpmix/prrte#2268.

I think I won't bring this back to the PRRTE v3.0 release branch as we are getting ready to release today and I don't want to destabilize things. This isn't a regression as we have never (a) supported envar directives before, nor (b) inherited them. So we'll hold off and see how this behaves.

[hamme74]

This sounds pretty good to me. Using inherit to support variable forwarding seems like the perfect solution.
However, one thing is a bit confusing:
For option --runtime-options fwd-environment, INHERIT is the default. But for -x, NOINHERIT is the default.
Maybe I'm being too nitpicky. :-D

Many thanks for implementing my feature wish so quickly. I will test it today.

[hamme74]

Testing the option -x with INHERIT/NOINHERIT on the master branch did not show any issues.
Perfect.

For option --unset-env, INHERIT/NOINHERIT has no effect.
My suggestion: Supporting INHERIT/NOINHERIT for the options -x and --unset-env, --prepend-env, --append-env would be ideal and intuitive.

[rhc54]

My suggestion: Supporting INHERIT/NOINHERIT for the options -x and --unset-env, --prepend-env, --append-env would be ideal and intuitive.

That should be the case now. When you say "--unset-env, INHERIT/NOINHERIT has no effect.", what exactly did you pass for that option?

[hamme74]

$ export uhu=666
$ /opt/ompi/openmpi/bin/mpiexec -n 2 --runtime-options fwd-environment -x uhu=777 --map-by node:oversubscribe:inherit --host node1,node2 ./parent.out

results in:

parent uhu: 777
parent uhu: 777
I am a parent with rank: 1, node2
I am a parent with rank: 0, node1
child uhu: 777
I am a child with rank: 1, node2
child uhu: 777
I am a child with rank: 0, node1
pid 3093580's current affinity list: 0-7
pid 3093581's current affinity list: 0-7

Option -x overwrites the variable for parents and children.

$ export uhu=666
$ /opt/ompi/openmpi/bin/mpiexec -n 2 --runtime-options fwd-environment --unset-env uhu --map-by node:oversubscribe:inherit --host node1,node2 ./parent.out

results in:

I am a parent with rank: 1, node2
I am a parent with rank: 0, node1
child uhu: 666
I am a child with rank: 0, node1
child uhu: 666
I am a child with rank: 1, node2
pid 3093601's current affinity list: 0-7
pid 3093600's current affinity list: 0-7

Option --unset-env uhu does not remove the variable for the children.

[rhc54]

Probably a question of ordering in the code - I'll take a look at it, but it may be a bit before I can get to it.

[hamme74]

Great. Thank you.

[rhc54]

Okay, it did indeed prove to be just an ordering issue and easily fixed. While doing so, I noted that we had "prepend-env", "append-env", and "unset-env" options - but no "set-env" option. It's true that the -x foo=val option would do that for you, but it isn't an obvious correlation. So I went ahead and added "set-env".

You'll find the fix in PRRTE master (openpmix/prrte#2279) - if you want the "set-env" option, then you'll also have to update PMIx to get openpmix/openpmix#3681.