[main] Can subhourly UPP jobs run on fewer than 4 nodes?

[MatthewPyle-NOAA]

While monitoring the real-time system since we switched to production-like resources, I've noticed the large number (20+) of UPP jobs running at a given time early in the forecast when 15 minute output is generated. Probably doesn't need to be changed as our peak usage doesn't seem to get too big, but worth exploring to see if these could run on 2 or 3 nodes for a bit longer?

[BenjaminBlake-NOAA]

@MatthewPyle-NOAA After thinking more about this, I think running the subhourly post jobs with fewer nodes is definitely worth trying. Each 15-minute file only contains 39 records, far fewer than the full hour files. Let's try 3 nodes first and see how that works.

Regarding testing, I imagine it would be best to test this change with the dev system on Cactus using ecflow. I believe @lgannoaa is currently testing some NCO changes there so perhaps once that test completes we can try this there?

[MatthewPyle-NOAA]

That sounds good to me - I haven't looked at the ecflow scripting for UPP to know if shifting the subh to 3 nodes will require ecf and UPP script generalization or modification.

[BenjaminBlake-NOAA]

Actually before testing in the real-time system, I will run a couple of standalone UPP tests to generate the subhourly files. Will try using 2 and 3 nodes, and compare to the runtime with 4 nodes.

[BenjaminBlake-NOAA]

@MatthewPyle-NOAA I didn't realize all of the RRFS post jobs were running with threading enabled. In the UPP RRFS regression test we don't use threading. Here are the results from my timing tests with the subhourly post:

4 nodes, 128 tasks per node, 4 threads: 3:27 (current RRFS configuration)
4 nodes, 128 tasks per node, 0 threads: 4:01
2 nodes, 128 tasks per node, 2 threads: 4:08
2 nodes, 128 tasks per node, 0 threads: 4:18

4 nodes, 48 tasks per node, 0 threads: 4:57
3 nodes, 48 tasks per node, 0 threads: 5:34
2 nodes, 96 tasks per node, 0 threads: 4:51
2 nodes, 72 tasks per node, 0 threads: 5:38
2 nodes, 64 tasks per node, 0 threads: 6:18
2 nodes, 48 tasks per node, 0 threads: 9:50

My recommendation would be to go with either of the bolded options. 2 nodes, 128 tasks per node, and either 0 threads or 2 threads. Both options would use select=2:ncpus=128 and the mpiexec lines would be as follows:
2 nodes, 0 threads: mpiexec -n 256 -ppn 128
2 nodes, 2 threads: mpiexec -n 128 -ppn 64 --cpu-bind core --depth 2

[MatthewPyle-NOAA]

@BenjaminBlake-NOAA Can you work this into testing and make a PR for it if it hasn't been on your radar? I'm fine with either of the options you bolded. Feel like this one fell out of view the new NBM product, and the AWIPS work.

[BenjaminBlake-NOAA]

Sure, and let's go with the 2 threads option since it was slightly faster.

I am meeting with @lgannoaa tomorrow and can discuss with him the best way to test this. The resources for the subhourly post jobs would need to be modified as follows:
#PBS -l place=vscatter:excl,select=4:ncpus=128:ompthreads=4:mem=500G
NNODES_POST='4' , PPN_POST='32' , TPP_POST='4'

#PBS -l place=vscatter:excl,select=2:ncpus=128:ompthreads=2:mem=500G
NNODES_POST='2' , PPN_POST='64' , TPP_POST='2'

[BenjaminBlake-NOAA]

These changes have been tested on Cactus and added to PR #1039