Correctness Checking

[MrBurmark]

Summary

Add more support for correctness checking.
Each kernel now can set its own tolerance and we print if the checksum met that tolerance in the show-progress screen output and the checksum output file.
This mechanism is now also used by the test executable.

How concerned should we be that checksums grow over multiple passes?

• This PR is a feature
• It does the following (modify list as needed):
  • Adds correctness checking at the request of Correctness/Robustness Checking #604
  • This needs to update RAJA once Add kahan sum reduce helper class. RAJA#1969 is merged

Example Output

Here is an example of -sp screen output.

Run kernel -- Basic_MULTI_REDUCE
        Running Base_Seq variant
                Running      default tuning -- 2.71738e-05 sec. x 50 rep. PASSED checksum
        Running Lambda_Seq variant
                Running      default tuning -- 2.71464e-05 sec. x 50 rep. PASSED checksum
        Running RAJA_Seq variant
                Running      default tuning -- 2.71234e-05 sec. x 50 rep. PASSED checksum
        Running Base_OpenMP variant
                Running      default tuning -- 0.000128336 sec. x 50 rep. PASSED checksum
        Running Lambda_OpenMP variant
                Running      default tuning -- 0.000129231 sec. x 50 rep. PASSED checksum
        Running RAJA_OpenMP variant
                Running      default tuning -- 1.70671e-05 sec. x 50 rep. FAILED checksum

Here is an example of checksum.txt file output.

Kernel                     
........................................................
Variants                   Result  Tolerance                   Average Checksum            Max Checksum Diff           Checksum Diff StdDev        
                                                                                           (vs. first variant listed)                              
----------------------------------------------------------------------------------------
Basic_MULTI_REDUCE         
........................................................
Base_Seq-default           PASSED  9.9999999999999995475e-08   5865.2700279413772306       0.0000000000000000000       0.0000000000000000000       
Lambda_Seq-default         PASSED  9.9999999999999995475e-08   5865.2700279413772306       0.0000000000000000000       0.0000000000000000000       
RAJA_Seq-default           PASSED  9.9999999999999995475e-08   5865.2700279413772306       0.0000000000000000000       0.0000000000000000000       
Base_OpenMP-default        PASSED  9.9999999999999995475e-08   5865.2700279413772306       0.0000000000000000000       0.0000000000000000000       
Lambda_OpenMP-default      PASSED  9.9999999999999995475e-08   5865.2700279413772306       0.0000000000000000000       0.0000000000000000000       
RAJA_OpenMP-default        FAILED  9.9999999999999995475e-08   2918.7998532503377369       2946.4701746910394893       2.2204460492503130808e-16 

-------------------------------------------------------

[rhornung67]

Either here or in another PR, it may be a good idea to note in the Dev Guide some things to think about when setting the tolerance for a kernel in the code. Maybe a brief explanation of how you determined the tolerance to set for a couple of representative kernels with different tolerances.

[rhornung67]

We've tried to deal with growing checksums by adding a multiplier to keep their magnitude reasonable. We also a Kahan sum approach for summing checksum values. We haven't looked at this in a while, at least I haven't. Maybe re-investigate since problem sizes are getting larger.

[artv3]

Do you mean multiple passes over the suite? or within the number of times the kernel is invoked?

[MrBurmark]

The checksums are added up over passes as well and the checksum scaling factor does not account for that effect.

[MrBurmark]

I updated the documentation for checksum consistency and added some documentation for the checksum tolerance and scaling factor. @rhornung67

[rhornung67]

@MrBurmark what do you think about dividing the checksum increment by the # passes (or multiplying by floating pt value 1.0/npasses) each time the checksum is updated for each kernel? This could be hidden in the KernelBase class method, doesn't add much additional arithmetic, and would guarantee that the checksums for a run would always be the same magnitude regardless of # passes.

[MrBurmark]

@rhornung67 Ya we could do something like that. How about dividing by the numbers of times the kernel was run in the getChecksum method?

[MrBurmark]

@rhornung67 I added the division.

[MrBurmark]

Thinking about the checksum a bit more and how to make it independent of the number of passes. How about we keep a sum of all the checksums, the min, and max? Then we could get the largest difference from the mean reference checksum at the end. That would avoid downplaying errors that only occurred in one pass.

[MrBurmark]

Regarding checksums and npasses when deciding on a reference checksum I chose to use the min checksum of the reference variant tuning for consistent kernels as the checksums should all be the same anyway and I chose to use the average checksum of the reference variant tuning for other kernels. Would it be simpler if I just used the first checksum of the reference variant tuning?

[MrBurmark]

I went ahead and switched to use the first pass of the kernel as the reference checksum.