8366787: Test runtime/StackGuardPages/TestStackGuardPagesNative.java hangs on some platforms

[mz1999]

Summary

This PR fixes a hang in the TestStackGuardPagesNative.java test that occurs on certain Linux distributions (e.g., CentOS 7). The fix replaces an unbounded for(;;) loop in the native test code (exeinvoke.c) with a bounded while loop, making the test's behavior deterministic and robust across all platforms.

Problem

The test would hang and eventually time out on some platforms. This was caused by an unbounded for(;;) loop in the do_overflow function, which was introduced as part of the "hardening" fix in JDK-8295344.

• On platforms like CentOS 7, this unbounded loop would not encounter a terminating signal in a timely manner, causing the native process to hang indefinitely until killed by the test harness.
• In contrast, on platforms like Ubuntu 24, the test would coincidentally pass because a SEGV_MAPERR would happen to terminate the loop. This highlighted that the test's success was reliant on platform-specific side effects, masking the underlying issue.

Solution

The solution is to replace the unbounded for(;;) loop with a bounded while loop. The condition while (_kp_rec_count == 0 || _rec_count < _kp_rec_count) ensures that the loop terminates deterministically after a specific number of allocations, corresponding to the overflow depth detected in the test's first phase.

This change makes the test's logic robust and its behavior consistent across different environments.

---

Progress

• Change must be properly reviewed (1 review required, with at least 1 Reviewer)
• Change must not contain extraneous whitespace
• Commit message must refer to an issue

Issue

• JDK-8366787: Test runtime/StackGuardPages/TestStackGuardPagesNative.java hangs on some platforms (Bug - P4)

Reviewers

• David Holmes (@dholmes-ora - Reviewer)

Reviewing

Using git

Checkout this PR locally:
$ git fetch https://git.openjdk.org/jdk.git pull/27114/head:pull/27114
$ git checkout pull/27114

Update a local copy of the PR:
$ git checkout pull/27114
$ git pull https://git.openjdk.org/jdk.git pull/27114/head

Using Skara CLI tools

Checkout this PR locally:
$ git pr checkout 27114

View PR using the GUI difftool:
$ git pr show -t 27114

Using diff file

Download this PR as a diff file:
https://git.openjdk.org/jdk/pull/27114.diff

Using Webrev

Link to Webrev Comment

[bridgekeeper]

👋 Welcome back mz1999! A progress list of the required criteria for merging this PR into master will be added to the body of your pull request. There are additional pull request commands available for use with this pull request.

[openjdk]

@mz1999 This change now passes all automated pre-integration checks.

ℹ️ This project also has non-automated pre-integration requirements. Please see the file CONTRIBUTING.md for details.

After integration, the commit message for the final commit will be:

8366787: Test runtime/StackGuardPages/TestStackGuardPagesNative.java hangs on some platforms

Reviewed-by: dholmes

You can use pull request commands such as /summary, /contributor and /issue to adjust it as needed.

At the time when this comment was updated there had been 67 new commits pushed to the master branch:

• 9e3fa32: 8349288: runtime/os/windows/TestAvailableProcessors.java fails on localized Windows platform
• b7b01d6: 8366984: Remove delay slot support
• 2705e88: 8366764: Deproblemlist java/awt/ScrollPane/ScrollPositionTest.java
• ... and 64 more: https://git.openjdk.org/jdk/compare/a2f8d3c4c25fdadf378313ef52185dceee98773d...master

As there are no conflicts, your changes will automatically be rebased on top of these commits when integrating. If you prefer to avoid this automatic rebasing, please check the documentation for the /integrate command for further details.

As you do not have Committer status in this project an existing Committer must agree to sponsor your change. Possible candidates are the reviewers of this PR (@dholmes-ora) but any other Committer may sponsor as well.

➡️ To flag this PR as ready for integration with the above commit message, type /integrate in a new comment. (Afterwards, your sponsor types /sponsor in a new comment to perform the integration).

[openjdk]

@mz1999 The following label will be automatically applied to this pull request:

• hotspot-runtime

When this pull request is ready to be reviewed, an "RFR" email will be sent to the corresponding mailing list. If you would like to change these labels, use the /label pull request command.

[mlbridge]

Webrevs

• 01: Full - Incremental (18ac75c7)
• 00: Full (b688eaca)

[dholmes-ora]

Actual fix looks good, but the test has to remain in the PL.

Thanks

[dholmes-ora]

Looks good. Thanks

[sendaoYan]

GHA report test java/lang/Thread/virtual/stress/GetStackTraceALotWhenPinned.java#id0 timed out on macos-aarch64. I think it's unrelated to this PR.

I have create a issue to record it.