Microbenchmark generation process

To ilustrate the generation process, let's play with the following hypothetical code:

private static int gnomesort(List<Integer> ar) {
    int i = 0; int k = 0;
    while (i < ar.size()) {
        if (i == 0 || ar.get(i-1) <= ar.get(i)) i++;
        else { int tmp = ar.get(i); ar.set(i, ar.get(i-1)); ar.set(--i, tmp); }
        k++;
    }
    return k;
}

//Sorts the list, add 'n' and returns the number 
//of steps needed to sort the list and do some math with it
public int addAndSorted(List<Integer> list, int n) {
	n = n < 1 ? 1 : n;
    list.add(n);

    /** @bench-this */
    int a = (int)Math.log(n * n / 2) * myBubbleSort(list);
    
    return a;  
}

Motivation example. Admittedly, is not the most realistic production code, but it will help us describe the functionalities of AutoJMH in one shot

We mentioned that AutoJMH takes as an input a code segment. We tell the tool the exact segment we want to benchmark using the /** @bench-this */ javadoc-like comment. The statement following the comment will be benchmarked. In this case is the statement int a = myCustomSortMethod(list);. We did not use regular annotations, because you can't just anotate any statement in Java and we wanted to create microbenchmarks for any statement.

Extract the segment

The first step is to create an empty JMH microbenchmark and extract the selected segment into it. At this point, AutoJMH will create a code lookig like the following (The ugly name given to the microbenchmark is composed of the fully qualified name of the class and the line containing the code):

@State(Scope.Thread)
public class org_packagename_YourClassName_348 {

    @Benchmark
    public void doBenchmark() {
        int a = (int)Math.log(n * n / 2) * myBubbleSort(list);
    }

}

STEP 1. Initial microbenchmark generated in by AutoJMH . This code won't compile

This code won't compile. The declaration of both the variables list, n and the method myCustomSortMethod are missing, meaning that AutoJMH still needs to perform another pass, where the tool declares all variables the segment needs as public fields in the generated microbenchmark. It also copy into the microbenchmark all the private static methods the segment uses. After the second pass, the microbenchmark can already compile:

@State(Scope.Thread)
public class org_packagename_YourClassName_348 {
    int n;
    List<Integer> list;

    private static int gnomesort(List<Integer> ar) {
        int i = 0; int k = 0;
        while (i < ar.size()) {
            if (i == 0 || ar.get(i-1) <= ar.get(i)) i++;
            else { int tmp = ar.get(i); ar.set(i, ar.get(i-1)); ar.set(--i, tmp); }
            k++;
        }
        return k;
    }

    @Benchmark
    public void doBenchmark() {
      int a = (int)Math.log(n * n / 2) * gnomesort(list);
    }
}

STEP 2. AutoJMH refines the microbenchmark to include variable and method declarations. The code can compile now, but still have deficiencies.

Initialize variables

The microbenchmark now compiles, but it has many problems. If we run this microbenchmark we will get nothing but a big fat NullPointerException since no variable is initialized. Here is where the training execution is used. AutoJMH instrument the original program and then runs the training execution to record the values of the variables entering and leaving the segment. Finally, this data is stored to disk and code to retrieve it is added to the microbenchmark:

@State(Scope.Thread)
public class org_packagename_YourClassName_348 {
    int n;
    List<Integer> list;

    private static int gnomesort(List<Integer> ar) {
        int i = 0; int k = 0;
        while (i < ar.size()) {
            if (i == 0 || ar.get(i-1) <= ar.get(i)) i++;
            else { int tmp = ar.get(i); ar.set(i, ar.get(i-1)); ar.set(--i, tmp); }
            k++;
        }
        return k;
    }

    @Setup
    public void setup() {
        Loader loader_348 = new Loader("usr/data/YourClassName_348");
		list = loader_348.readIntegerList();
		n = loader_348.readInt();
    }

    @Benchmark
    public int doBenchmark() {
        int a = (int)Math.log(n * n / 2) * gnomesort(list);
    }
}

STEP 3. AutoJMH has added code to initialize variables. Still, some more tweaks are needed

Now we have a microbenchmark that will execute. However, more work has to be done to avoid DCE and keep the microbenchmark performing the same computations in every run.

Refine the microbenchmark

In STEP 3 AutoJMH achieved a code that compiles and runs without exceptions. However, the generated code is affected by DCE. Also, after the first execution, the list gets sorted, changing the performance of the code. More tweaks are needed in order to achieve a good microbenchmark.

Avoid DCE

The code in STEP 2 still have some issues. The statement being benchmarked is going to be partially removed as dead code. Since no one is using the value of variable a, only the call to gnomesort will be made and the computations Math.log(n * n / 2) will be removed. AutoJMH is able to detect this by performing a liveness analysis in both the original code and the resulting microbenchmark so far. Then it ensures that all variables live in the original code, are still live in the microbenchmark by adding a final return to the microbenchmark or consuming it with a JMH black hole:

@State(Scope.Thread)
public class org_packagename_YourClassName_348 {
    int n;
    List<Integer> list;

    private static int gnomesort(List<Integer> ar) {
        int i = 0; int k = 0;
        while (i < ar.size()) {
            if (i == 0 || ar.get(i-1) <= ar.get(i)) i++;
            else { int tmp = ar.get(i); ar.set(i, ar.get(i-1)); ar.set(--i, tmp); }
            k++;
        }
        return k;
    }

    @Setup
    public void setup() {
        Loader loader_348 = new Loader("usr/data/YourClassName_348");
		list = loader_348.readIntegerList();
		n = loader_348.readInt();
    }

    @Benchmark
    public int doBenchmark() {
        int a = (int)Math.log(n * n / 2) * gnomesort(list);
        return a;
    }
}

The resulting microbenchmark after the DCE analysis. A final return is added and now the measured segment is not affected by DCE.

Avoid Transient-State

After the first execution of the microbenchmark, the list gets sorted. This changes the performance of the following executions and forbid us to asses the performance of the case in which the list is not sorted. To ensure that all executions of a microbenchmark performs the same computations, AutoJMH performs an analysis to determine which variables are being changed and adds code to reset them. Currently, there are some limiatations in this analysis. Variables being reset are those being assigned inside the microbenchmark also present a branching statement's expression (if, for, etc.). Mutable object passed as parameters and object whose methods are called are reset as well. In the current case, the list variable is reset by AutoJMH. To do so, values are readed from a variable specially created for it:

@State(Scope.Thread)
public class org_packagename_YourClassName_348 {
    int n;
    List<Integer> list;
    List<Integer> list___reset;

    private static int gnomesort(List<Integer> ar) {
        int i = 0; int k = 0;
        while (i < ar.size()) {
            if (i == 0 || ar.get(i-1) <= ar.get(i)) i++;
            else { int tmp = ar.get(i); ar.set(i, ar.get(i-1)); ar.set(--i, tmp); }
            k++;
        }
        return k;
    }

    @Setup
    public void setup() {
        Loader loader_348 = new Loader("usr/data/YourClassName_348");
		list___reset = loader_348.readIntegerList();
        list = loader_348.readEmptyList(); 
		n = loader_348.readInt();
    }

    @Setup(Level.Invocation)
    public void resetCode() {
        list.clear(); list.addAll(list___reset);
    }

    @Benchmark
    public int doBenchmark() {
        int a = (int)Math.log(n * n / 2) * gnomesort(list);
        return a;
    }
}

FINAL STEP. The microbenchmark is complete. Note the new @Setup method added to reset the variable list