fix rank in init

Summary:
Currently, it requires passing a rank at creation time. However, rank can only be determined after the communicator is created, so semantically, it is not possible to provide a rank during creation. Even if we ignore semantics and pass a rank—as users often do—this approach only works for non-fault-tolerant scenarios, where the scheduler creates processes and sets the environment variable `RANK`.

In fault-tolerant scenarios, we have elastic process creation: processes can be added or removed at any time. This means we encounter the same issue—at creation time, we cannot specify ranks, as ranks can only be determined at initialization, once all participants are known.

To address this, we are moving MCCL to a new API where ranks are not provided at creation time.
At initialization, we support two options:

1. **User does not care about rank order:**
   MCCL `init` accepts a `std::unordered_set` of URLs, and MCCL is free to assign ranks internally based on its own considerations.

2. **User wants to specify rank order:**
   For non-fault-tolerant cases, where ranks are defined by the environment variable `RANK`, the user passes a `std::vector` of URLs to MCCL. MCCL will respect the order of URLs and assign ranks accordingly. Using a vector also ensures rank properties are satisfied: ranks go sequentially from 0 to `nRanks - 1`, with no repetitions or missing ranks.

Reviewed By: saifhhasan

Differential Revision: D84839780

fbshipit-source-id: c482b47976cf8711fca737c0f107896bc3bfe558

Author: dboyda

comms/ctran/tests/CtranXPlatUtUtils.cc

@@ -248,7 +248,6 @@ TestCtranCommRAII::TestCtranCommRAII(std::unique_ptr<mccl::McclComm> mcclComm)
 std::unique_ptr<TestCtranCommRAII> createDummyCtranComm() {
   CHECK_EQ(ctran::utils::commCudaLibraryInit(), commSuccess);
   mccl::McclCommCreateOpts mcclCreateOpts{
-      .rank = 0,
       .cudaDeviceId = 0,
       .enableFaultTolerance = false,
   };
@@ -259,7 +258,7 @@ std::unique_ptr<TestCtranCommRAII> createDummyCtranComm() {
   auto initWorkHandle = mcclComm->init(
       mccl::InitOpts{
           .uuid = uuid,
-          .urls = {initURL},
+          .urls = std::unordered_set<mccl::InitURL>{initURL},
       });
   initWorkHandle->waitCpu();
   auto initResult = initWorkHandle->getResult();

comms/ctran/tests/CtranXPlatUtUtils.h

@@ -288,9 +288,9 @@ class CtranDistTest : public ::testing::Test {
     return globalRank == 0;
   }
 
-  std::unordered_set<std::string>
+  std::vector<std::string>
   exchangeInitUrls(const std::string& selfUrl, int numRanks, int selfRank) {
-    std::unordered_set<std::string> res;
+    std::vector<std::string> res(numRanks);
     if (getInitEnvType() == InitEnvType::TCP_STORE) {
       std::vector<std::string> rankKeys(numRanks);
       const auto keyUid = getTcpStoreKey(TcpStorePhase::INIT);
@@ -305,8 +305,9 @@ class CtranDistTest : public ::testing::Test {
       tcpStore_->wait(rankKeys);
       if (tcpStore_->check(rankKeys)) {
         auto rankUrls = tcpStore_->multiGet(rankKeys);
-        for (const auto& url : rankUrls) {
-          res.emplace(std::string(url.begin(), url.end()));
+        for (int i = 0; i < numRanks; ++i) {
+          const auto& url = rankUrls.at(i);
+          res[i] = std::string(url.begin(), url.end());
         }
       } else {
         LOG(FATAL) << "TCPStore key check returned false";
@@ -326,7 +327,9 @@ class CtranDistTest : public ::testing::Test {
           MPI_CHAR,
           MPI_COMM_WORLD);
       for (int i = 0; i < numRanks; ++i) {
-        res.emplace(std::string(urls.data() + kMaxUrlLen * i));
+        const char* start = urls.data() + kMaxUrlLen * i;
+        size_t len = strnlen(start, kMaxUrlLen);
+        res[i] = std::string(start, len);
       }
     }
     return res;
@@ -339,7 +342,6 @@ class CtranDistTest : public ::testing::Test {
     // TODO: refactor mccl comm creation to generic ctran comm creation
     COMMCHECK_TEST(ctran::utils::commCudaLibraryInit());
     mccl::McclCommCreateOpts opts{
-        .rank = globalRank,
         .cudaDeviceId = cudaDev,
         .timeout = std::chrono::seconds(5),
     };