Merge pull request STEllAR-GROUP#6687 from uiuc-hpc/update-option

lci pp: change default ndevices=2 and progress_type=worker; improve document

Author: hkaiser

docs/sphinx/manual/using_the_lci_parcelport.rst

@@ -120,3 +120,22 @@ when using the LCI parcelport to get better performance.
 
 This setting needs roughly 800MB memory per process. The memory consumption mainly
 comes from the packets, which can be calculated using `LCI_SERVER_NUM_PKTS x LCI_PACKET_SIZE`.
+
+In addition, users can tune the following command-line options
+when using the LCI parcelport to get better performance.
+
+.. option:: --hpx:ini=hpx.parcel.lci.ndevices=<int>
+
+   The number of LCI devices to use. The default value is 2.
+   An LCI device represents a collection of network resources.
+   More devices lead to lower thread contention, but too many devices
+   may lead to load imbalance or hardware overhead.
+
+.. option:: --hpx:ini=hpx.parcel.lci.progress_type=<worker|rp>
+
+   The way to progress the LCI device. The default value is ``worker``.
+   The ``worker`` option uses all worker threads to progress the LCI devices.
+   The ``rp`` option uses dedicated pinned threads to progress the LCI devices.
+   Normally, the ``worker`` option gives better performance, but the ``rp`` option
+   has been observed with better performance on some clusters with 
+   prior generation of InfiniBand hardware.

libs/full/parcelport_lci/include/hpx/parcelport_lci/parcelport_lci.hpp

@@ -264,10 +264,10 @@ namespace hpx::traits {
                 "protocol = putsendrecv\n"
                 "comp_type_header = queue\n"
                 "comp_type_followup = queue\n"
-                "progress_type = rp\n"
+                "progress_type = worker\n"
                 "prepost_recv_num = 1\n"
                 "reg_mem = 1\n"
-                "ndevices = 1\n"
+                "ndevices = 2\n"
                 "ncomps = 1\n"
                 "enable_in_buffer_assembly = 1\n"
                 "send_nb_max_retry = 32\n"