HADOOP-19638. [JDK17] Set Up CI Support JDK17.

[slfan1989]

Description of PR

JIRA: HADOOP-19638. [JDK17] Set Up CI Support JDK17.

Plan to establish a new build pipeline using JDK 17 for Apache Hadoop, as part of the ongoing effort to upgrade the project’s Java compatibility. This pipeline will serve as the foundation for testing and validating future JDK 17 support and ensuring smooth integration with existing CI/CD processes.

How was this patch tested?

yetus & jenkins

For code changes:

• Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
• Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
• If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
• If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

[slfan1989]

@GauthamBanasandra I’m working on adding a new pipeline in Trunk to support JDK17. While reviewing the history, I noticed you've done in-depth work on this area, particularly around the upgrade to jenkins.sh

As part of this change, I reviewed the implementation in HADOOP-16888(#2012) to better understand how JDK11 support was introduced. From what I can tell, it seems that enabling JDK17 unit test support only requires configuring the appropriate JDK17-related variables in jenkins.sh, similar to how JDK11 was handled. Could you please confirm if that's sufficient? Let me know if there are any additional compatibility steps or considerations I should be aware of.

HADOOP-19366. Install OpenJDK 17 in default ubuntu build container (#7226)
The download of jdk17 has been introduced.

Thank you very much!

[slfan1989]

@ayushtkn @aajisaka Could you kindly provide some help and guidance? I would really appreciate it.

[hadoop-yetus]

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	1m 0s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+0 🆗	shelldocs	0m 0s		Shelldocs was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s		The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
			_ trunk Compile Tests _
+0 🆗	mvndep	9m 46s		Maven dependency ordering for branch
+1 💚	mvninstall	30m 21s		trunk passed
+1 💚	mvnsite	0m 0s		trunk passed
+1 💚	shadedclient	30m 37s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 29s		Maven dependency ordering for patch
+1 💚	mvninstall	0m 0s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	mvnsite	0m 0s		the patch passed
+1 💚	shellcheck	0m 0s		No new issues.
+1 💚	shadedclient	30m 30s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	asflicense	0m 33s		The patch does not generate ASF License warnings.
		105m 45s

Subsystem	Report/Notes
Docker	ClientAPI=1.51 ServerAPI=1.51 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7831/1/artifact/out/Dockerfile
GITHUB PR	#7831
Optional Tests	dupname asflicense mvnsite unit codespell detsecrets shellcheck shelldocs
uname	Linux b2225bbcc657 5.15.0-143-generic #153-Ubuntu SMP Fri Jun 13 19:10:45 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / 0024f51
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7831/1/testReport/
Max. process+thread count	548 (vs. ulimit of 5500)
modules	C: U:
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7831/1/console
versions	git=2.25.1 maven=3.6.3 shellcheck=0.7.0
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

[GauthamBanasandra]

@slfan1989 yes, what you've done in this PR should be sufficient to get the multi-JDK tests running. The multi-JDK unit tests are only run as part of the nightly CI and not for PRs. Could you please trigger on nightly CI run on ci-hadoop.apache.org?

Let me know if you need me to do this.

[slfan1989]

@slfan1989 yes, what you've done in this PR should be sufficient to get the multi-JDK tests running. The multi-JDK unit tests are only run as part of the nightly CI and not for PRs. Could you please trigger on nightly CI run on ci-hadoop.apache.org?

Let me know if you need me to do this.

Thank you very much for your reply! I'm not very familiar with how to trigger a JDK17 build. If you happen to know how, would you mind helping trigger a JDK17 build for this PR? Also, if we want the PR builds to run under the JDK17 environment as well, do you know how we can configure that?

[slfan1989]

@steveloughran @ayushtkn @Hexiaoqiao @GauthamBanasandra Is it possible to remove JDK 8 compilation support from the trunk? We’d appreciate your input.

[GauthamBanasandra]

@slfan1989 I just checked the nightly pipeline for Linux and it looks like it's isn't using jenkins.sh anymore. The jenkins.sh is currently used by the nightly pipeline for Windows and the pre-commit pipeline for Linux.

Anyway, I just created a new pipeline that runs on JDK 17 - https://ci-hadoop.apache.org/view/Hadoop/job/hadoop-qbt-trunk-java17-linux-x86_64/.

You should be having the permission to trigger the builds. I've currently configured it to checkout your repo (I've mentioned the same in the pipeline's description page). I'll reconfigure it to github.com/apache/hadoop once you're done with the testing.

[GauthamBanasandra]

I've started a run with JDK 17 on that pipeline - https://ci-hadoop.apache.org/view/Hadoop/job/hadoop-qbt-trunk-java17-linux-x86_64/1/console.

[slfan1989]

@GauthamBanasandra Thank you very much for your kind help in configuring this pipeline!

[GauthamBanasandra]

Looks like the Jenkins CI had some trouble with the docker container when it was running the unit tests -

cd /home/jenkins/jenkins-home/workspace/hadoop-qbt-trunk-java17-linux-x86_64/sourcedir/hadoop-common-project/hadoop-auth
/usr/bin/mvn --batch-mode -Dmaven.repo.local=/home/jenkins/jenkins-home/workspace/hadoop-qbt-trunk-java17-linux-x86_64/yetus-m2/hadoop-HADOOP-19638-full-2 -DskipTests test-compile spotbugs:spotbugs -DskipTests=true > /home/jenkins/jenkins-home/workspace/hadoop-qbt-trunk-java17-linux-x86_64/out/branch-spotbugs-hadoop-common-project_hadoop-auth.txt 2>&1


Cleaning up docker image used for testing.
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?


============================================================================
============================================================================
                              Finished build.
============================================================================
============================================================================

I've started another run - https://ci-hadoop.apache.org/view/Hadoop/job/hadoop-qbt-trunk-java17-linux-x86_64/2/console

[GauthamBanasandra]

@steveloughran @ayushtkn @Hexiaoqiao @GauthamBanasandra Is it possible to remove JDK 8 compilation support from the trunk? We’d appreciate your input.

@slfan1989 I don't think we should remove JDK 8 support yet.

[ayushtkn]

I don't think we can maintain so many JDK versions. There would be some thirdparty dependencies which we might upgrade to & they would be compiled on higher JDK versions. So, the moment that happens, JDK-8 compilation will break. & Not upgrading to those dependencies or not using higher JDK features doesn't make sense.

Hadoop in branch-3 dropped JDK-7 support as well, branch-2 only has JDK-7 IIRC. We haven't marked the lower branches which are JDK-8 compliant as EOL, so, it isn't a deal breaker for someone who wants to stick to JDK-8, Moreover we can't run tests & all on all JDK versions, We need to pick one only.

I think most of the projects are dropping the JDK-8 support and chasing for higher versions. I know in Hive it is like 4.0 line -JDK-8, 4.1.0 line - JDK-17 & now master branch min JDK-21 compile time, same with Tez the master is on JDK-21 compile time support.

I don't think sticking to any legacy stuff is worth it if it adds any technical debt for us, Should chase for a higher version. Just my opinion, other might feel differently

[steveloughran]

@GauthamBanasandra 3.5 is java17+ only.

there are things we depend on which are java11; there are places where we can't make use of the JDK features, let alone language changes.

we have stayed on java8 for longer than we've ever been on any older java version. Time to move on.

If you want to keep java8 support alive, then backport new features to branch-3.4. Me? I'm looking at bug fixes only there.

[steveloughran]

...assuming we switch now to java17 on trunk, all backports to branch-3.4 must be retested on that branch in case something in the JDK itself changed. With the JUnit move that's needed anyway, IMO

[slfan1989]

@GauthamBanasandra @ayushtkn @steveloughran Thank you all for the feedback! From my perspective, it makes sense to drop support for JDK 8, and I also agree with using JDK 17 for the trunk branch. However, there are still some issues with the JDK 17 pipeline compilation, and I will work on fixing the related unit tests.

[slfan1989]

I have adjusted the trunk branch's build to default to JDK 17 support.

[hadoop-yetus]

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 53s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+0 🆗	shelldocs	0m 0s		Shelldocs was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s		The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
			_ trunk Compile Tests _
+0 🆗	mvndep	11m 29s		Maven dependency ordering for branch
+1 💚	mvninstall	29m 12s		trunk passed
+1 💚	mvnsite	0m 0s		trunk passed
+1 💚	shadedclient	30m 4s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 27s		Maven dependency ordering for patch
+1 💚	mvninstall	0m 0s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	mvnsite	0m 0s		the patch passed
+1 💚	shellcheck	0m 1s		No new issues.
+1 💚	shadedclient	30m 14s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	asflicense	0m 34s		The patch does not generate ASF License warnings.
		105m 11s

Subsystem	Report/Notes
Docker	ClientAPI=1.51 ServerAPI=1.51 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7831/2/artifact/out/Dockerfile
GITHUB PR	#7831
Optional Tests	dupname asflicense mvnsite unit codespell detsecrets shellcheck shelldocs
uname	Linux 459b670a40fa 5.15.0-143-generic #153-Ubuntu SMP Fri Jun 13 19:10:45 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / 577479b
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7831/2/testReport/
Max. process+thread count	547 (vs. ulimit of 5500)
modules	C: U:
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7831/2/console
versions	git=2.25.1 maven=3.6.3 shellcheck=0.7.0
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

[steveloughran]

+1