8370562: Whitebox Fuzzer for C2

[oliviermattmann]

Issue: JDK-8370562

Summary

Request for feedback on instrumentation, tracking optimizations in the context of a whitebox fuzzer for C2.

Details

Hello everyone,

My name is Olivier and I am a Master's student at ETH Zurich. For my Master's thesis I am writing a whitebox fuzzer for the C2 JIT compiler of the JVM.
Three bugs were already discovered during the development of the fuzzer.

• JDK-8370405
• JDK-8370416
• JDK-8370502

We believe that single optimizations in isolation are well-tested by the jtreg test suite. Our intuition is that interactions between optimizations exhibit subtle quirks that possibly lead to bugs. As such we chose the approach to maximize and explore optimization interactions in C2 compilations.

In order to guide the fuzzer to these optimization interactions, we created a first draft of an instrumentation to count specific optimizations during a method compilation.

The purpose of this Draft PR is to collect feedback and comments on the instrumentation. Especially comments or suggestions on which optimizations would be interesting to track and where such counters could be placed. Comments on which currently tracked optimizations are believed irrelevant are appreciated as well.

Having a more diverse set of tracked optimizations leads to more diverse test cases and better coverage optimization interactions.

How the current instrumentation works

The current patch counts selected c2 optimizations every time a method is compiled and emits a compact summary that the white‑box fuzzer uses as its fitness vector. The counter array lives in Compile (src/hotspot/share/opto/compile.hpp:409) and is zeroed at the start of each compilation (src/hotspot/share/opto/compile.cpp:739). Any phase can add a new data point simply by calling Compile::record_optimization_event(...) (src/hotspot/share/opto/compile.cpp:1134).

The events are enumerated in OptimizationEvent (src/hotspot/share/opto/compile.hpp:109) and currently include:

• OptEvent_LoopUnrolling – loop unroll decisions (src/hotspot/share/opto/loopTransform.cpp:2179)
• OptEvent_LoopPeeling – loop peeling (src/hotspot/share/opto/loopTransform.cpp:780)
• OptEvent_ParallelInductionVars – creation of parallel IVs (src/hotspot/share/opto/loopnode.cpp:4444)
• OptEvent_SplitIf – Split-If in loop opts (src/hotspot/share/opto/loopopts.cpp:1475)
• OptEvent_LoopUnswitching – loop unswitch (src/hotspot/share/opto/loopUnswitch.cpp:408)
• OptEvent_ConditionalExpressionElimination – CEE in IfNode processing (src/hotspot/share/opto/ifnode.cpp:1811)
• OptEvent_FunctionInlining – inlining decisions (src/hotspot/share/opto/doCall.cpp:703)
• OptEvent_Deoptimization – scheduled uncommon traps (src/hotspot/share/opto/graphKit.cpp:2092)
• OptEvent_EscapeAnalysis – EA successes (src/hotspot/share/opto/escape.cpp:121)
• OptEvent_EliminateLocks – macro-phase lock elimination (src/hotspot/share/opto/macro.cpp:2132)
• OptEvent_LockCoarsening – lock coarsening (src/hotspot/share/opto/compile.cpp:4967)
• OptEvent_ConditionalConstantPropagation – CCP (src/hotspot/share/opto/phaseX.cpp:3185)
• OptEvent_EliminateAutobox – autobox elimination (src/hotspot/share/opto/macro.cpp:1103)
• OptEvent_BlockElimination – dead block removal (src/hotspot/share/opto/block.cpp:1021)
• OptEvent_NullCheckElimination – null-check folding (src/hotspot/share/opto/graphKit.cpp:1312)
• OptEvent_RangeCheckElimination – RCE in loops (src/hotspot/share/opto/loopTransform.cpp:2978)
• OptEvent_OptimizePtrCompare – pointer-compare opts (src/hotspot/share/opto/escape.cpp:3260)
• OptEvent_MergeStores – merge-stores IGVN pass (src/hotspot/share/opto/compile.cpp:2065)
• OptEvent_LoopPredication – loop predication (src/hotspot/share/opto/loopPredicate.cpp:1321)
• OptEvent_AutoVectorization – SuperWord/vectorization wins (src/hotspot/share/opto/loopnode.cpp:5278)
• OptEvent_PartialPeeling – partial loop peeling (src/hotspot/share/opto/loopopts.cpp:4169)

Adding new metrics is usually a one-line call next to the relevant optimization decision. If the optimization event is new, it has to be added to the enum in compile.hpp and a suitable name for the event in the optimization_event_names array located in src/hotspot/share/opto/compile.cpp:96.

How to try it out locally

Build a fastdebug image such that the TraceC2Optimizations flag is available.

Example:

build/linux-x86_64-fastdebug/images/jdk/bin/java \
-XX:+TraceC2Optimizations \
-XX:CompileCommand=compileonly,<SomeClass>::<Method> \
<YourMainClass>

How this optimization feedback is used

The records of optimizations is fed into our fuzzer. The fuzzer is mutation based and has various mutators which are designed to evoke optimization such as loop unrolling, loop unswitching, escape analysis etc. The records of optimizations are used in two ways.

1. Adapt the weights of the mutators such that the the scheduling of which mutator to use is guided.

2. Prioritize test cases for mutation which exhibit rare or novel optimization interactions. Test cases with new/different optimization behaviors are deemed interesting and are explored further. As a consequence a more diverse and fine grained feedback broadens the search space and allows for better coverage.

Any comment or feedback is highly appreciated!

---

Progress

• Change must be properly reviewed (1 review required, with at least 1 Reviewer)
• Change must not contain extraneous whitespace
• Commit message must refer to an issue

Issue

• JDK-8370562: Whitebox Fuzzer for C2 (Enhancement - P4)

Reviewing

Using git

Checkout this PR locally:
$ git fetch https://git.openjdk.org/jdk.git pull/27991/head:pull/27991
$ git checkout pull/27991

Update a local copy of the PR:
$ git checkout pull/27991
$ git pull https://git.openjdk.org/jdk.git pull/27991/head

Using Skara CLI tools

Checkout this PR locally:
$ git pr checkout 27991

View PR using the GUI difftool:
$ git pr show -t 27991

Using diff file

Download this PR as a diff file:
https://git.openjdk.org/jdk/pull/27991.diff

[bridgekeeper]

Hi @oliviermattmann, welcome to this OpenJDK project and thanks for contributing!

We do not recognize you as Contributor and need to ensure you have signed the Oracle Contributor Agreement (OCA). If you have not signed the OCA, please follow the instructions. Please fill in your GitHub username in the "Username" field of the application. Once you have signed the OCA, please let us know by writing /signed in a comment in this pull request.

If you already are an OpenJDK Author, Committer or Reviewer, please click here to open a new issue so that we can record that fact. Please use "Add GitHub user oliviermattmann" as summary for the issue.

If you are contributing this work on behalf of your employer and your employer has signed the OCA, please let us know by writing /covered in a comment in this pull request.

[openjdk]

❗ This change is not yet ready to be integrated.
See the Progress checklist in the description for automated requirements.

[openjdk]

⚠️ @oliviermattmann This pull request contains merges that bring in commits not present in the target repository. Since this is not a "merge style" pull request, these changes will be squashed when this pull request in integrated. If this is your intention, then please ignore this message. If you want to preserve the commit structure, you must change the title of this pull request to Merge <project>:<branch> where <project> is the name of another project in the OpenJDK organization (for example Merge jdk:master).

[openjdk]

@oliviermattmann The following label will be automatically applied to this pull request:

• hotspot-compiler

When this pull request is ready to be reviewed, an "RFR" email will be sent to the corresponding mailing list. If you would like to change these labels, use the /label pull request command.

[sendaoYan]

Do there is a another tools which can mutate the input tests. Only the input tests change, the records of optimization has meanings.

[oliviermattmann]

@sendaoYan I should have provided more context in the PR, I'll update it in a minute.
The records of optimizations are fed into my fuzzer. The fuzzer is mutation based and has various mutators which are designed to evoke optimization such as loop unrolling, loop unswitching, escape analysis etc.
The records of optimizations are used for two things. 1) The scheduling of which mutator to use and 2) the prioritization of test cases (i.e. which to mutate and test next). Test cases which exhibit rare or novel optimization interactions are prioritized. A more diverse record of optimization feedback leads to a more fine grained search space for the fuzzer, which would definitely help.

[sendaoYan]

The fuzzer is mutation based and has various mutators.

I'm very interested on this mutation tools. Do this/these mutation tools have a plan to open source.

[oliviermattmann]

I'm very interested on this mutation tools. Do this/these mutation tools have a plan to open source.

I do plan to make the fuzzer public, in it's current state it's not yet ready though. In the current state the fuzzer uses the mutators proposed by a paper called "Validating JVM Compilers via Maximizing Optimization Interactions".
You can find the description of the mutators in the paper. The ones not described in the paper can be found on the github repository in the supplementary material. (https://github.com/CGCL-codes/MopFuzzer/tree/main)

I do plan to add more powerful/different mutators though as it proved challenging to get past global value numbering and actually evoke the desired optimizations.

[connglli]

No idea on how to select optimizations, but the following are missed IMO:

• iterative gvns: it's better to mark on which iteration we're now on. More gvns performed means more reaching more deeper

  • Compile::Optimize()

• split loop iterations

  • https://github.com/openjdk/jdk/blob/master/src/hotspot/share/opto/loopnode.cpp#L5234

• reassociation of invariants

  • https://github.com/openjdk/jdk/blob/master/src/hotspot/share/opto/loopnode.cpp#L5171
  • https://github.com/openjdk/jdk/blob/master/src/hotspot/share/opto/loopnode.cpp#L5186

• intrinstics

  • https://github.com/openjdk/jdk/blob/master/src/hotspot/share/opto/loopnode.cpp#L5221
  • This one is not that interesting -- unless we generate a lot of intrinsics -- as we may not generate such code

It also looks like the codegen part is totally missed (perhaps you have your reasons), but those peephole opportunities might also be interesting?