Fix Hera GNU out of memory error

[mingchen-NOAA]

Pull Request Summary

This PR addresses the “out of memory” error encountered when running matrix regression tests on Hera (GNU) after upgrading to Spack‑Stack 1.9.2.

Description

This PR fixes the “out of memory” error on Hera (GNU) with Spack‑Stack 1.9.2.

• The file regtests/bin/matrix_cmake_ncep was updated to reduce the sbatch configuration ntasks from 24 to 12 in order to lower memory usage.

• To ensure sufficient computational resources for large cases, regtests/bin/matrix_divider_cmake.sh was modified to increase ntasks to 140 for large cases.

Regression tests were run successfully without errors. A matrix comparison was performed, and a number of unexpected differences were observed.

Issue(s) addressed

Addressing issue #1466

Commit Message

Fix 'out of memory' error on Hera (GNU) with Spack-Stack 1.9.2

Check list

• Branch is up to date with the authoritative repository (NOAA-EMC) develop branch.
• Checked the checklist for a developer submitting to develop.
• If a version number update is required, checked the updating version number checklist.
• If a new feature was added, a regression test for testing the new feature is added.

Testing

• How were these changes tested? Matrix
• Are the changes covered by regression tests? (If not, why? Do new tests need to be added?)
• Have the matrix regression tests been run (if yes, please note HPC and compiler)? Hera using GNU
• Please indicate the expected changes in the regression test output, (Note the list of known non-identical tests.)
• Please provide the summary output of matrix.comp (matrix.Diff.txt, matrixCompFull.txt and matrixCompSummary.txt):

**********************************************************************
********************* non-identical cases ****************************
**********************************************************************
mww3_test_01/./work_PR2_UQ_MPI                     (2 files differ)
mww3_test_01/./work_PR1                     (2 files differ)
mww3_test_01/./work_PR3_UQ                     (2 files differ)
mww3_test_01/./work_PR3_UNO_MPI                     (2 files differ)
mww3_test_01/./work_PR3_UQ_MPI                     (2 files differ)
mww3_test_02/./work_PR2_UQ_b                     (4 files differ)
mww3_test_02/./work_PR3_UNO_MPI_d_c                     (2 files differ)
mww3_test_02/./work_PR3_UQ_c                     (2 files differ)
mww3_test_02/./work_PR3_UQ_MPI_b_c                     (2 files differ)
mww3_test_02/./work_PR1_b                     (2 files differ)
mww3_test_02/./work_PR1_MPI_c                     (4 files differ)
mww3_test_02/./work_PR2_UQ_c                     (2 files differ)
mww3_test_02/./work_PR2_UNO_c                     (2 files differ)
mww3_test_02/./work_PR3_UQ_b_c                     (4 files differ)
mww3_test_02/./work_PR3_UNO_MPI_a                     (2 files differ)
mww3_test_02/./work_PR1_a                     (2 files differ)
mww3_test_02/./work_PR3_UQ_MPI_d                     (4 files differ)
mww3_test_02/./work_PR2_UNO_MPI_c                     (2 files differ)
mww3_test_02/./work_PR3_UQ_a_c                     (2 files differ)
mww3_test_02/./work_PR3_UQ_b                     (2 files differ)
mww3_test_02/./work_PR3_UQ_d                     (4 files differ)
mww3_test_02/./work_PR3_UNO_MPI_b_c                     (2 files differ)
mww3_test_02/./work_PR1_d                     (2 files differ)
mww3_test_02/./work_PR2_UQ_MPI_c                     (2 files differ)
mww3_test_02/./work_PR2_UNO_a                     (2 files differ)
mww3_test_02/./work_PR1_c                     (2 files differ)
mww3_test_02/./work_PR3_UNO_c                     (4 files differ)
mww3_test_02/./work_PR3_UQ_MPI_a_c                     (2 files differ)
mww3_test_02/./work_PR2_UNO_MPI_b                     (2 files differ)
mww3_test_02/./work_PR3_UNO_d                     (2 files differ)
mww3_test_02/./work_PR3_UNO_b                     (2 files differ)
mww3_test_02/./work_PR3_UNO_b_c                     (2 files differ)
mww3_test_02/./work_PR3_UQ_MPI_a                     (2 files differ)
mww3_test_02/./work_PR2_UNO_d                     (2 files differ)
mww3_test_02/./work_PR1_MPI_b                     (2 files differ)
mww3_test_02/./work_PR3_UNO_MPI_c                     (2 files differ)
mww3_test_02/./work_PR3_UQ_d_c                     (2 files differ)
mww3_test_02/./work_PR1_MPI_a                     (2 files differ)
mww3_test_03/./work_PR2_UQ_b                     (2 files differ)
mww3_test_03/./work_PR3_UNO_MPI_d_c                     (2 files differ)
mww3_test_03/./work_PR3_UQ_c                     (2 files differ)
mww3_test_03/./work_PR1_MPI_e                     (2 files differ)
mww3_test_03/./work_PR3_UQ_a                     (2 files differ)
mww3_test_03/./work_PR3_UQ_MPI_b_c                     (8 files differ)
mww3_test_03/./work_PR1_b                     (4 files differ)
mww3_test_03/./work_PR1_MPI_c                     (6 files differ)
mww3_test_03/./work_PR3_UNO_MPI_a_c                     (2 files differ)
mww3_test_03/./work_PR1_MPI_d                     (4 files differ)
mww3_test_03/./work_PR2_UQ_c                     (4 files differ)
mww3_test_03/./work_PR2_UQ_MPI_b                     (4 files differ)
mww3_test_03/./work_PR3_UQ_MPI_c                     (6 files differ)
mww3_test_03/./work_PR2_UNO_c                     (2 files differ)
mww3_test_03/./work_PR3_UQ_b_c                     (4 files differ)
mww3_test_03/./work_PR3_UQ_MPI_d2_c                     (17 files differ)
mww3_test_03/./work_PR3_UNO_MPI_a                     (2 files differ)
mww3_test_03/./work_PR3_UNO_MPI_c_c                     (6 files differ)
mww3_test_03/./work_PR3_UQ_MPI_e_c                     (4 files differ)
mww3_test_03/./work_PR3_UQ_MPI_d                     (10 files differ)
mww3_test_03/./work_PR2_UNO_MPI_c                     (8 files differ)
mww3_test_03/./work_PR3_UQ_b                     (4 files differ)
mww3_test_03/./work_PR3_UQ_MPI_b                     (4 files differ)
mww3_test_03/./work_PR3_UQ_d                     (6 files differ)
mww3_test_03/./work_PR3_UNO_MPI_b_c                     (6 files differ)
mww3_test_03/./work_PR3_UNO_MPI_d                     (2 files differ)
mww3_test_03/./work_PR2_UNO_MPI_d                     (6 files differ)
mww3_test_03/./work_PR1_d                     (10 files differ)
mww3_test_03/./work_PR2_UQ_MPI_c                     (6 files differ)
mww3_test_03/./work_PR2_UNO_MPI_d2                     (15 files differ)
mww3_test_03/./work_PR2_UQ_MPI_d                     (4 files differ)
mww3_test_03/./work_PR3_UNO_MPI_b                     (6 files differ)
mww3_test_03/./work_PR2_UNO_a                     (2 files differ)
mww3_test_03/./work_PR2_UQ_MPI_d2                     (17 files differ)
mww3_test_03/./work_PR2_UQ_a                     (4 files differ)
mww3_test_03/./work_PR3_UQ_MPI_d2                     (17 files differ)
mww3_test_03/./work_PR2_UQ_d                     (4 files differ)
mww3_test_03/./work_PR2_UNO_b                     (6 files differ)
mww3_test_03/./work_PR1_c                     (2 files differ)
mww3_test_03/./work_PR3_UQ_MPI_c_c                     (6 files differ)
mww3_test_03/./work_PR3_UNO_c                     (2 files differ)
mww3_test_03/./work_PR3_UNO_a                     (2 files differ)
mww3_test_03/./work_PR3_UQ_MPI_a_c                     (6 files differ)
mww3_test_03/./work_PR2_UNO_MPI_b                     (2 files differ)
mww3_test_03/./work_PR3_UNO_d                     (4 files differ)
mww3_test_03/./work_PR3_UQ_e_c                     (2 files differ)
mww3_test_03/./work_PR3_UNO_b                     (2 files differ)
mww3_test_03/./work_PR3_UNO_d_c                     (4 files differ)
mww3_test_03/./work_PR1_e                     (2 files differ)
mww3_test_03/./work_PR3_UNO_b_c                     (4 files differ)
mww3_test_03/./work_PR2_UNO_MPI_a                     (2 files differ)
mww3_test_03/./work_PR3_UQ_MPI_a                     (2 files differ)
mww3_test_03/./work_PR3_UQ_c_c                     (4 files differ)
mww3_test_03/./work_PR2_UNO_d                     (4 files differ)
mww3_test_03/./work_PR1_MPI_d2                     (17 files differ)
mww3_test_03/./work_PR3_UNO_MPI_e_c                     (2 files differ)
mww3_test_03/./work_PR3_UNO_MPI_d2_c                     (18 files differ)
mww3_test_03/./work_PR1_MPI_b                     (4 files differ)
mww3_test_03/./work_PR3_UQ_MPI_e                     (4 files differ)
mww3_test_03/./work_PR3_UNO_MPI_c                     (2 files differ)
mww3_test_03/./work_PR2_UQ_e                     (2 files differ)
mww3_test_03/./work_PR2_UQ_MPI_a                     (4 files differ)
mww3_test_03/./work_PR3_UQ_d_c                     (4 files differ)
mww3_test_03/./work_PR3_UNO_a_c                     (4 files differ)
mww3_test_03/./work_PR3_UNO_e                     (2 files differ)
mww3_test_03/./work_PR3_UNO_MPI_d2                     (14 files differ)
mww3_test_03/./work_PR3_UNO_c_c                     (4 files differ)
mww3_test_03/./work_PR2_UNO_MPI_e                     (1 files differ)
mww3_test_03/./work_PR1_MPI_a                     (4 files differ)
mww3_test_04/./work_PR2_UQ_b                     (4 files differ)
mww3_test_04/./work_PR3_UQ_g_c                     (2 files differ)
mww3_test_04/./work_PR1_f                     (4 files differ)
mww3_test_04/./work_PR1_MPI_e                     (2 files differ)
mww3_test_04/./work_PR1_b                     (2 files differ)
mww3_test_04/./work_PR3_UQ_f                     (4 files differ)
mww3_test_04/./work_PR1_MPI_c                     (2 files differ)
mww3_test_04/./work_PR1_MPI_d                     (2 files differ)
mww3_test_04/./work_PR3_UNO_g                     (4 files differ)
mww3_test_04/./work_PR2_UQ_MPI_b                     (4 files differ)
mww3_test_04/./work_PR2_UQ_f                     (4 files differ)
mww3_test_04/./work_PR3_UQ_MPI_c                     (4 files differ)
mww3_test_04/./work_PR3_UQ_b_c                     (6 files differ)
mww3_test_04/./work_PR3_UNO_MPI_c_c                     (4 files differ)
mww3_test_04/./work_PR3_UQ_MPI_NC_b_c                     (2 files differ)
mww3_test_04/./work_PR2_UQ_MPI_e                     (4 files differ)
mww3_test_04/./work_PR3_UQ_MPI_d                     (2 files differ)
mww3_test_04/./work_PR2_UNO_MPI_c                     (2 files differ)
mww3_test_04/./work_PR3_UQ_a_c                     (4 files differ)
mww3_test_04/./work_PR3_UQ_b                     (2 files differ)
mww3_test_04/./work_PR3_UNO_MPI_b_c                     (2 files differ)
mww3_test_04/./work_PR3_UNO_MPI_d                     (2 files differ)
mww3_test_04/./work_PR2_UNO_MPI_d                     (4 files differ)
mww3_test_04/./work_PR2_UNO_f                     (2 files differ)
mww3_test_04/./work_PR2_UQ_MPI_c                     (4 files differ)
mww3_test_04/./work_PR2_UQ_g                     (4 files differ)
mww3_test_04/./work_PR2_UQ_MPI_d                     (4 files differ)
mww3_test_04/./work_PR3_UNO_f                     (2 files differ)
mww3_test_04/./work_PR3_UNO_MPI_b                     (2 files differ)
mww3_test_04/./work_PR2_UNO_a                     (2 files differ)
mww3_test_04/./work_PR2_UNO_b                     (2 files differ)
mww3_test_04/./work_PR3_UNO_g_c                     (4 files differ)
mww3_test_04/./work_PR3_UNO_MPI_e                     (4 files differ)
mww3_test_04/./work_PR3_UQ_MPI_c_c                     (2 files differ)
mww3_test_04/./work_PR3_UNO_a                     (2 files differ)
mww3_test_04/./work_PR1_g                     (2 files differ)
mww3_test_04/./work_PR2_UNO_MPI_b                     (2 files differ)
mww3_test_04/./work_PR3_UNO_b                     (4 files differ)
mww3_test_04/./work_PR3_UQ_MPI_d_c                     (2 files differ)
mww3_test_04/./work_PR3_UNO_b_c                     (4 files differ)
mww3_test_04/./work_PR3_UNO_f_c                     (6 files differ)
mww3_test_04/./work_PR3_UNO_MPI_e_c                     (4 files differ)
mww3_test_04/./work_PR3_UQ_f_c                     (4 files differ)
mww3_test_04/./work_PR3_UQ_MPI_e                     (4 files differ)
mww3_test_04/./work_PR3_UNO_MPI_c                     (2 files differ)
mww3_test_04/./work_PR3_UNO_a_c                     (6 files differ)
mww3_test_04/./work_PR2_UNO_MPI_e                     (2 files differ)
mww3_test_04/./work_PR3_UQ_g                     (2 files differ)
mww3_test_05/./work_ST3_PR2_UNO_MPI                     (2 files differ)
mww3_test_05/./work_ST1_PR1_MPI_OMPH                     (2 files differ)
mww3_test_05/./work_ST3_PR1_MPI                     (4 files differ)
mww3_test_05/./work_ST3_PR3_UNO_OMP                     (2 files differ)
mww3_test_05/./work_ST4_PR3_UQ_MPI                     (2 files differ)
mww3_test_05/./work_ST2_PR3_UQ_OMP                     (4 files differ)
mww3_test_05/./work_ST2_PR1_MPI_OMPH                     (4 files differ)
mww3_test_05/./work_ST4_PR1_OMP                     (4 files differ)
mww3_test_05/./work_ST6_PR1_MPI_OMPH                     (4 files differ)
mww3_test_05/./work_ST1_PR2_UNO_MPI_OMPH                     (2 files differ)
mww3_test_05/./work_ST6_PR2_UQ_MPI_OMPH                     (4 files differ)
mww3_test_05/./work_ST6_PR3_UQ_MPI                     (4 files differ)
mww3_test_05/./work_ST2_PR3_UQ_MPI                     (2 files differ)
mww3_test_05/./work_ST3_PR3_UQ_OMP                     (4 files differ)
mww3_test_05/./work_ST1_PR2_UNO_MPI                     (2 files differ)
mww3_test_05/./work_ST6_PR2_UQ_MPI                     (4 files differ)
mww3_test_05/./work_ST1_PR3_UNO_MPI_OMPH                     (4 files differ)
mww3_test_05/./work_ST2_PR2_UQ_OMP                     (2 files differ)
mww3_test_05/./work_ST3_PR2_UQ_MPI                     (2 files differ)
mww3_test_05/./work_ST1_PR3_UQ_MPI_OMPH                     (2 files differ)
mww3_test_05/./work_ST4_PR3_UNO_OMP                     (2 files differ)
mww3_test_05/./work_ST2_PR2_UNO_MPI_OMPH                     (4 files differ)
mww3_test_05/./work_ST4_PR2_UQ_MPI_OMPH                     (2 files differ)
mww3_test_05/./work_ST2_PR2_UNO_OMP                     (6 files differ)
mww3_test_05/./work_ST4_PR3_UQ_OMP                     (4 files differ)
mww3_test_05/./work_ST4_PR3_UNO_MPI                     (2 files differ)
mww3_test_05/./work_ST6_PR1_OMP                     (6 files differ)
mww3_test_05/./work_ST2_PR3_UNO_OMP                     (2 files differ)
mww3_test_05/./work_ST1_PR1_OMP                     (4 files differ)
mww3_test_05/./work_ST3_PR3_UQ_MPI_OMPH                     (2 files differ)
mww3_test_05/./work_ST6_PR3_UNO_MPI_OMPH                     (6 files differ)
mww3_test_05/./work_ST2_PR1_MPI                     (4 files differ)
mww3_test_05/./work_ST3_PR3_UNO_MPI                     (2 files differ)
mww3_test_05/./work_ST6_PR2_UNO_MPI_OMPH                     (2 files differ)
mww3_test_05/./work_ST1_PR2_UQ_MPI_OMPH                     (4 files differ)
mww3_test_05/./work_ST1_PR3_UQ_OMP                     (4 files differ)
mww3_test_05/./work_ST1_PR2_UQ_MPI                     (2 files differ)
mww3_test_05/./work_ST2_PR2_UQ_MPI                     (2 files differ)
mww3_test_05/./work_ST2_PR2_UNO_MPI                     (2 files differ)
mww3_test_05/./work_ST6_PR3_UQ_MPI_OMPH                     (2 files differ)
mww3_test_05/./work_ST2_PR3_UNO_MPI_OMPH                     (2 files differ)
mww3_test_05/./work_ST2_PR3_UQ_MPI_OMPH                     (2 files differ)
mww3_test_05/./work_ST4_PR2_UQ_OMP                     (2 files differ)
mww3_test_05/./work_ST2_PR3_UNO_MPI                     (4 files differ)
mww3_test_05/./work_ST6_PR2_UNO_OMP                     (2 files differ)
mww3_test_05/./work_ST4_PR2_UQ_MPI                     (2 files differ)
mww3_test_05/./work_ST1_PR3_UQ_MPI                     (2 files differ)
mww3_test_05/./work_ST6_PR2_UNO_MPI                     (6 files differ)
mww3_test_05/./work_ST3_PR2_UNO_OMP                     (4 files differ)
mww3_test_05/./work_ST3_PR2_UQ_OMP                     (2 files differ)
mww3_test_05/./work_ST3_PR1_MPI_OMPH                     (2 files differ)
mww3_test_05/./work_ST3_PR2_UNO_MPI_OMPH                     (2 files differ)
mww3_test_05/./work_ST1_PR3_UNO_OMP                     (2 files differ)
mww3_test_05/./work_ST6_PR3_UQ_OMP                     (2 files differ)
mww3_test_05/./work_ST4_PR3_UNO_MPI_OMPH                     (4 files differ)
mww3_test_05/./work_ST1_PR2_UNO_OMP                     (2 files differ)
mww3_test_06/./work_PR3_UQ_d                     (2 files differ)
mww3_test_06/./work_PR3_UNO_d                     (4 files differ)
mww3_test_07/./work_PR3_UQ                     (85 files differ)
mww3_test_09/./work_MPI                     (6 files differ)
mww3_test_09/./work_MPI_ASCII                     (4 files differ)
ww3_ta1/./work_UPD3_U                     (2 files differ)
ww3_ta1/./work_UPD2_O                     (2 files differ)
ww3_ta1/./work_UPD3_O                     (2 files differ)
ww3_ta1/./work_UPD5_U                     (2 files differ)
ww3_ta1/./work_UPD5_O                     (2 files differ)
ww3_ta1/./work_UPD6_U_cap                     (2 files differ)
ww3_tbt1.1/./work_BT9                     (2 files differ)
ww3_tbt1.1/./work_BT8_MPI                     (2 files differ)
ww3_tbt2.1/./work_BT9_MPI                     (2 files differ)
ww3_tic1.1/./work_IC4_M3                     (4 files differ)
ww3_tic1.1/./work_20km_IC1                     (2 files differ)
ww3_tic1.1/./work_1000m_IC1                     (2 files differ)
ww3_tic1.1/./work_IC4_M6H                     (5 files differ)
ww3_tic1.1/./work_1000m_nondisp_IC2_nrl                     (2 files differ)
ww3_tic1.1/./work_IC4_M8                     (4 files differ)
ww3_tic1.1/./work_2500m_IC3                     (2 files differ)
ww3_tic1.1/./work_2500m_IC1                     (2 files differ)
ww3_tic1.1/./work_IC4_M6                     (5 files differ)
ww3_tic1.1/./work_IC4_M4                     (5 files differ)
ww3_tic1.1/./work_IC4_M9                     (4 files differ)
ww3_tic1.1/./work_IC4_M6L                     (5 files differ)
ww3_tic1.1/./work_IC5_M1                     (2 files differ)
ww3_tic1.1/./work_IC4_M7                     (4 files differ)
ww3_tic1.1/./work_100m_IC1                     (2 files differ)
ww3_tic1.1/./work_IC4_M5                     (5 files differ)
ww3_tic1.2/./work_IC3_B0.5k_MPI                     (2 files differ)
ww3_tic1.2/./work_IC3_A1.0k                     (2 files differ)
ww3_tic1.2/./work_IC3_V1_G                     (2 files differ)
ww3_tic1.2/./work_IC3_A2.5k                     (2 files differ)
ww3_tic1.2/./work_IC3_A0.5k_MPI                     (2 files differ)
ww3_tic1.2/./work_IC3_B2.5k                     (2 files differ)
ww3_tic1.2/./work_IC3_V1_h                     (2 files differ)
ww3_tic1.2/./work_IC3_A0.5k                     (2 files differ)
ww3_tic1.2/./work_IC3_A1.0k_MPI                     (2 files differ)
ww3_tic1.2/./work_IC3_B1.0k                     (2 files differ)
ww3_tic1.3/./work_IC3_2.5k_PR1                     (2 files differ)
ww3_tic1.3/./work_IC3_V1_G                     (2 files differ)
ww3_tic1.3/./work_IC3_2.5k_PR3_UNO                     (2 files differ)
ww3_tic1.3/./work_IC3_0.5k_PR3_UNO                     (2 files differ)
ww3_tic1.3/./work_IC3_0.5k_PR2_UNO                     (2 files differ)
ww3_tic2.2/./work_IC2IS2                     (2 files differ)
ww3_tp1.1/./work_PR2_UQ                     (2 files differ)
ww3_tp1.2/./work_PR1                     (2 files differ)
ww3_tp1.2/./work_PR2_UQ                     (2 files differ)
ww3_tp1.4/./work_PR3_UNO                     (2 files differ)
ww3_tp1.4/./work_PR1                     (2 files differ)
ww3_tp1.4/./work_PR3_UQ                     (2 files differ)
ww3_tp1.5/./work_PR3_UNO                     (2 files differ)
ww3_tp1.5/./work_PR3_UQ                     (2 files differ)
ww3_tp1.5/./work_PR2_UQ                     (2 files differ)
ww3_tp1.6/./work_PR1                     (2 files differ)
ww3_tp1.8/./work_BJ_PR3_UQ                     (3 files differ)
ww3_tp2.1/./work_PR2_UQ_MPI                     (3 files differ)
ww3_tp2.1/./work_PR3_UNO                     (1 files differ)
ww3_tp2.1/./work_PR2_UNO                     (1 files differ)
ww3_tp2.1/./work_PR3_UQ                     (2 files differ)
ww3_tp2.1/./work_PR1_MPI                     (1 files differ)
ww3_tp2.1/./work_PR3_UNO_MPI                     (1 files differ)
ww3_tp2.1/./work_PR3_UQ_MPI                     (1 files differ)
ww3_tp2.10/./work_MPI_OMPH                     (6 files differ)
ww3_tp2.13/./work_PR1_TRPL                     (2 files differ)
ww3_tp2.13/./work_PR1_TRPL_MPI                     (2 files differ)
ww3_tp2.13/./work_PR1_MPI                     (2 files differ)
ww3_tp2.14/./work_OASICM                     (2 files differ)
ww3_tp2.14/./work_OASACM3                     (2 files differ)
ww3_tp2.15/./work_PR3_UQ_CONST                     (2 files differ)
ww3_tp2.15/./work_PR3_UQ                     (2 files differ)
ww3_tp2.15/./work_MPI_5km                     (2 files differ)
ww3_tp2.15/./work_ST6FLX5                     (2 files differ)
ww3_tp2.15/./work_PR3_UQ_MPI                     (2 files differ)
ww3_tp2.16/./work_MPI_OMPH                     (5 files differ)
ww3_tp2.16/./work_MPI                     (3 files differ)
ww3_tp2.17/./work_mc1                     (2 files differ)
ww3_tp2.18/./work_TIDE_MPI                     (2 files differ)
ww3_tp2.19/./work_1B_a                     (10 files differ)
ww3_tp2.19/./work_1C_a                     (10 files differ)
ww3_tp2.2/./work_PR2_UNO                     (2 files differ)
ww3_tp2.2/./work_PR3_UQ                     (2 files differ)
ww3_tp2.2/./work_PR2_UQ                     (2 files differ)
ww3_tp2.2/./work_PR1_MPI_BIN2NC                     (2 files differ)
ww3_tp2.2/./work_PR3_UNO_MPI                     (2 files differ)
ww3_tp2.21/./work_a                     (5 files differ)
ww3_tp2.21/./work_mb                     (5 files differ)
ww3_tp2.3/./work_PR3_UNO                     (5 files differ)
ww3_tp2.3/./work_PR1                     (5 files differ)
ww3_tp2.3/./work_PR2_UNO_MPI                     (5 files differ)
ww3_tp2.3/./work_PR2_UQ                     (5 files differ)
ww3_tp2.3/./work_PR1_MPI                     (5 files differ)
ww3_tp2.3/./work_PR3_UNO_MPI                     (5 files differ)
ww3_tp2.4/./work_PR2_UQ_MPI                     (2 files differ)
ww3_tp2.4/./work_PR1                     (2 files differ)
ww3_tp2.4/./work_PR2_UNO_MPI                     (2 files differ)
ww3_tp2.4/./work_PR3_UQ_curv                     (2 files differ)
ww3_tp2.4/./work_PR1_curv_MPI                     (2 files differ)
ww3_tp2.4/./work_PR3_UNO_MPI                     (2 files differ)
ww3_tp2.5/./work_PR2_UQ_MPI                     (5 files differ)
ww3_tp2.5/./work_REF_PR1_MPI                     (6 files differ)
ww3_tp2.5/./work_REF_PR3_UNO                     (6 files differ)
ww3_tp2.5/./work_PR2_UNO_MPI                     (5 files differ)
ww3_tp2.5/./work_REF_PR2_UNO                     (6 files differ)
ww3_tp2.5/./work_PR2_UQ                     (5 files differ)
ww3_tp2.5/./work_REF_PR3_UQ                     (6 files differ)
ww3_tp2.5/./work_PR3_UQ_MPI                     (5 files differ)
ww3_tp2.6/./work_ST4                     (15 files differ)
ww3_tp2.6/./work_ST0                     (15 files differ)
ww3_tp2.6/./work_ST4_ASCII                     (0 files differ)
ww3_tp2.6/./work_pdlib                     (15 files differ)
ww3_tp2.9/./work_PR1_a                     (2 files differ)
ww3_tp2.9/./work_PR2_UNO_b_MPI                     (1 files differ)
ww3_tp2.9/./work_PR1_a_MPI                     (3 files differ)
ww3_tp2.9/./work_PR3_UQ_b                     (3 files differ)
ww3_tp2.9/./work_PR2_UQ_b_MPI                     (1 files differ)
ww3_tp2.9/./work_PR2_UNO_a                     (2 files differ)
ww3_tp2.9/./work_PR2_UQ_a                     (2 files differ)
ww3_tp2.9/./work_PR2_UNO_b                     (2 files differ)
ww3_tp2.9/./work_PR3_UNO_a                     (2 files differ)
ww3_tp2.9/./work_PR1_b_MPI                     (1 files differ)
ww3_tp2.9/./work_PR3_UNO_b                     (2 files differ)
ww3_ts1/./work_NL5                     (2 files differ)
ww3_ts1/./work_ST1_RWND                     (2 files differ)
ww3_ts1/./work_ST4_WRT                     (2 files differ)
ww3_ts1/./work_ST4                     (2 files differ)
ww3_ts3/./work_ST1_PR1_MPI_OMPH                     (2 files differ)
ww3_ts3/./work_ST4_PR3_UQ_MPI                     (2 files differ)
ww3_ts3/./work_ST6_PR2_UQ_MPI_OMPH                     (2 files differ)
ww3_ts3/./work_ST3_PR3_UQ_MPI                     (2 files differ)
ww3_ts3/./work_ST6_PR3_UQ_MPI                     (2 files differ)
ww3_ts3/./work_ST1_PR2_UNO_MPI                     (2 files differ)
ww3_ts3/./work_ST6_PR2_UQ_MPI                     (2 files differ)
ww3_ts3/./work_ST1_PR3_UQ_MPI_OMPH                     (2 files differ)
ww3_ts3/./work_ST3_PR3_UNO_MPI_OMPH                     (2 files differ)
ww3_ts3/./work_ST2_PR2_UNO_OMP                     (2 files differ)
ww3_ts3/./work_ST4_PR3_UNO_MPI                     (2 files differ)
ww3_ts3/./work_ST4_PR1_MPI_OMPH                     (2 files differ)
ww3_ts3/./work_ST1_PR1_OMP                     (2 files differ)
ww3_ts3/./work_ST3_PR3_UQ_MPI_OMPH                     (2 files differ)
ww3_ts3/./work_ST1_PR2_UQ_OMP                     (2 files differ)
ww3_ts3/./work_ST4_PR2_UNO_MPI_OMPH                     (2 files differ)
ww3_ts3/./work_ST2_PR1_MPI                     (2 files differ)
ww3_ts3/./work_ST3_PR3_UNO_MPI                     (2 files differ)
ww3_ts3/./work_ST3_PR1_OMP                     (2 files differ)
ww3_ts3/./work_ST6_PR2_UNO_MPI_OMPH                     (2 files differ)
ww3_ts3/./work_ST1_PR3_UQ_OMP                     (2 files differ)
ww3_ts3/./work_ST6_PR3_UQ_MPI_OMPH                     (2 files differ)
ww3_ts3/./work_ST2_PR3_UNO_MPI                     (2 files differ)
ww3_ts3/./work_ST6_PR2_UNO_OMP                     (2 files differ)
ww3_ts3/./work_ST6_PR1_MPI                     (2 files differ)
ww3_ts3/./work_ST1_PR3_UQ_MPI                     (2 files differ)
ww3_ts3/./work_ST6_PR2_UNO_MPI                     (2 files differ)
ww3_ts3/./work_ST3_PR1_MPI_OMPH                     (2 files differ)
ww3_ts3/./work_ST1_PR1_MPI                     (2 files differ)
ww3_ts3/./work_ST3_PR2_UNO_MPI_OMPH                     (2 files differ)
ww3_ts3/./work_ST1_PR3_UNO_OMP                     (2 files differ)
ww3_ts3/./work_ST4_PR3_UQ_MPI_OMPH                     (2 files differ)
ww3_ts4/./work_rg_multi_MPI                     (2 files differ)
ww3_ts4/./work_rg_shel_MPI                     (2 files differ)
ww3_ufs1.1/./work_c_npl                     (2 files differ)
ww3_ufs1.1/./work_unstr_c                     (2 files differ)
ww3_ufs1.2/./work_a                     (6 files differ)
ww3_ufs1.3/./work_a                     (27 files differ)

matrixCompFull.txt
matrixCompSummary.txt
matrixDiff.txt