8310233: Fix THP detection on Linux

[tabata-d]

JDK-8310233 is a bug where the JDK 11 Hotspot VM on Linux incorrectly detects Transparent Huge Pages (THPs) status and page size. This leads to misconfigurations and inefficient memory usage.

The fix was originally implemented in JDK 22. Now we are backporting this into JDK 11.
I confirmed that it was reproduced on Linux x86_64 using JDK 11.

Unclean Backport

• The backport was not entirely clean due to a fundamental dependency on JDK-8256155, an Enhancement with CSR present in newer JDK versions (e.g., JDK 17+, JDK 22) but absent in JDK 11. A full backport of JDK-8256155 was avoided as it would introduce significant behavioral changes and compatibility risks unsuitable for JDK 11 in maintenance mode.
• Instead, a targeted manual adaptation was implemented for JDK 11's static huge page mode.
  • JDK 22's os::large_page_init() (with JDK-8256155): The JDK 22 code (in the else block for static huge pages) explicitly retrieves os::PageSizes all_large_pages = HugePages::static_info().pagesizes(); . This all_large_pages (No longer used in JDK11) is a variable introduced in JDK 22 as part of JDK-8256155 , holds a comprehensive list of all large_page_sizes supported by the kernel not just the default. This allowed JDK 22 to intelligently select the most appropriate large page size (the largest available that is smaller than the bytes being reserved) based on LargePageSizeInBytes acting as a maximum allowed size. It then uses a complex conditional block to interpret LargePageSizeInBytes against this list and populates _page_sizes with multiple large page sizes less than or equal to _large_page_size.
  • JDK 11's os::large_page_init() (without JDK-8256155): The JDK 11 backport's else block for static huge pages is significantly simpler. It directly sets _large_page_size to HugePages::default_static_hugepage_size(). The all_large_pages variable and the complex logic used in JDK22 that leverages it for flexible page size selection are not present and not used in the JDK 11 backport. Instead, it populates _page_sizes only with the single configured static huge page size and the VM page size.

Other Code Differences and Adaptations:

• os_linux.cpp:
  • The removal of size_t os::Linux::_default_large_page_size = 0; (present in JDK 22) was not performed in JDK 11 as this member was absent.
  • The modification to int os::Linux::hugetlbfs_page_size_flag(size_t page_size) (done in JDK 22) was not performed in JDK 11 as the relevant code structure was different.
  • Instead of removing JDK 22's static size_t scan_default_large_page_size(), static os::PageSizes scan_multiple_page_support(), and size_t os::Linux::default_large_page_size(), the backport removed JDK 11's equivalent functions: size_t os::Linux::find_large_page_size() and size_t os::Linux::setup_large_page_size().
  • Use NULL instead of nullptr.
• os_linux.hpp:
  • The removal of static size_t _default_large_page_size; (present in JDK 22) was not performed in JDK 11 as this declaration was absent.
  • Instead of removing JDK 22's static size_t default_large_page_size();, static size_t scan_default_large_page_size();, and static os::PageSizes scan_multiple_page_support();, the backport removed JDK 11's equivalent declarations: static size_t find_large_page_size(); and static size_t setup_large_page_size();.
  • Use enum instead of enum class.
• HugePageConfiguration.java:
  • Use old switch statement.

Testing

System: Red Hat Enterprise Linux 9.4 (x86_64).
jtreg: A comprehensive jtreg run on the entire hotspot/jtreg test suite confirmed that all HotSpot tests passed.

---

• I confirm that I make this contribution in accordance with the OpenJDK Interim AI Policy.

---

Progress

• Change must be properly reviewed (1 review required, with at least 1 Reviewer)
• Change must not contain extraneous whitespace
• Commit message must refer to an issue
• JDK-8310233 needs maintainer approval

Integration blocker

 ⚠️ Dependency #3160 must be integrated first

Issue

• JDK-8310233: Fix THP detection on Linux (Bug - P4)

Reviewers

• Paul Hohensee (@phohensee - Reviewer)

Reviewing

Using git

Checkout this PR locally:
$ git fetch https://git.openjdk.org/jdk11u-dev.git pull/3171/head:pull/3171
$ git checkout pull/3171

Update a local copy of the PR:
$ git checkout pull/3171
$ git pull https://git.openjdk.org/jdk11u-dev.git pull/3171/head

Using Skara CLI tools

Checkout this PR locally:
$ git pr checkout 3171

View PR using the GUI difftool:
$ git pr show -t 3171

Using diff file

Download this PR as a diff file:
https://git.openjdk.org/jdk11u-dev/pull/3171.diff

Using Webrev

Link to Webrev Comment

[bridgekeeper]

👋 Welcome back dtabata! A progress list of the required criteria for merging this PR into pr/3160 will be added to the body of your pull request. There are additional pull request commands available for use with this pull request.

[openjdk]

❗ This change is not yet ready to be integrated.
See the Progress checklist in the description for automated requirements.

[openjdk]

This backport pull request has now been updated with issue from the original commit.

[mlbridge]

Webrevs

• 00: Full (59b53def)

[phohensee]

This looks ok standalone, but I doubt the maintainers will approve it. First, it will need follow-up backports, which increases risk. Second, this is a P4 bug, and arguably an enhancement. Third, the 17u backport is 2.5 years old, so an unclean 11u backport will be deemed high risk for a release in maintenance mode.

[bridgekeeper]

@tabata-d This pull request has been inactive for more than 4 weeks and will be automatically closed if another 4 weeks passes without any activity. To avoid this, simply issue a /touch or /keepalive command to the pull request. Feel free to ask for assistance if you need help with progressing this pull request towards integration!

[tabata-d]

/touch

[openjdk]

@tabata-d The pull request is being re-evaluated and the inactivity timeout has been reset.

[tabata-d]

/template append

[openjdk]

@tabata-d The pull request template has been appended to the pull request body