YARN-11843. Fix potential deadlock when auto-correction of container allocation is enabled #7855

TaoYang526 · 2025-08-05T02:25:16Z

Details please refer to YARN-11843.

Description of PR

How was this patch tested?

For code changes:

Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

…allocation is enabled

TaoYang526 · 2025-08-05T02:28:23Z

@shameersss1 @slfan1989 Could you please help to review this PR? Thanks!

shameersss1

LGTM +1

...main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/AbstractYarnScheduler.java

shameersss1 · 2025-08-05T02:56:00Z

@TaoYang526 - It is good catch indeed.
Just curious to know - May i know the use case behind using auto-correction feature ?

TaoYang526 · 2025-08-05T03:23:50Z

@TaoYang526 - It is good catch indeed. Just curious to know - May i know the use case behind using auto-correction feature ?

@shameersss1 Currently, we don't have a specific use case for the auto-correction feature. We have always been aware that YARN has an over-allocation issue, but considered it a minor concern, so no action was taken. When I discovered YARN-11702, I wanted to take a look and found that it could resolve most over-allocation issues, except for some in-scheduling cases. It's sufficient to reduce unnecessary consumption in the scheduling cycle and we may enable this feature in the future.

shameersss1 · 2025-08-05T05:15:41Z

@TaoYang526 - It is good catch indeed. Just curious to know - May i know the use case behind using auto-correction feature ?

@shameersss1 Currently, we don't have a specific use case for the auto-correction feature. We have always been aware that YARN has an over-allocation issue, but considered it a minor concern, so no action was taken. When I discovered YARN-11702, I wanted to take a look and found that it could resolve most over-allocation issues, except for some in-scheduling cases. It's sufficient to reduce unnecessary consumption in the scheduling cycle and we may enable this feature in the future.

Thanks for the details.

hadoop-yetus · 2025-08-05T12:23:44Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 49s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s		The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
			_ trunk Compile Tests _
+1 💚	mvninstall	52m 19s		trunk passed
+1 💚	compile	1m 8s		trunk passed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04
+1 💚	compile	0m 54s		trunk passed with JDK Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09
+1 💚	checkstyle	0m 57s		trunk passed
+1 💚	mvnsite	0m 59s		trunk passed
+1 💚	javadoc	0m 59s		trunk passed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	0m 49s		trunk passed with JDK Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09
+1 💚	spotbugs	1m 59s		trunk passed
+1 💚	shadedclient	41m 49s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 48s		the patch passed
+1 💚	compile	0m 57s		the patch passed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04
+1 💚	javac	0m 57s		the patch passed
+1 💚	compile	0m 47s		the patch passed with JDK Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09
+1 💚	javac	0m 47s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 44s		the patch passed
+1 💚	mvnsite	0m 50s		the patch passed
+1 💚	javadoc	0m 46s		the patch passed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	0m 41s		the patch passed with JDK Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09
+1 💚	spotbugs	1m 54s		the patch passed
+1 💚	shadedclient	42m 38s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	117m 4s		hadoop-yarn-server-resourcemanager in the patch passed.
+1 💚	asflicense	0m 39s		The patch does not generate ASF License warnings.
		270m 30s

Subsystem	Report/Notes
Docker	ClientAPI=1.51 ServerAPI=1.51 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7855/3/artifact/out/Dockerfile
GITHUB PR	#7855
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux 8e43c4d936c4 5.15.0-144-generic #157-Ubuntu SMP Mon Jun 16 07:33:10 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `29cd1d7`
Default Java	Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7855/3/testReport/
Max. process+thread count	939 (vs. ulimit of 5500)
modules	C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7855/3/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

zeekling · 2025-08-10T15:51:16Z

...main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/AbstractYarnScheduler.java

-            completedContainer(rmContainer, SchedulerUtils.createAbnormalContainerStatus(
-                rmContainer.getContainerId(), SchedulerUtils.EXPIRED_CONTAINER),
-                RMContainerEventType.EXPIRE);
+            asyncContainerRelease(rmContainer);


Will it cause the message at the end of the container to be different?

Yes, the message will be updated from "Container expired since it was unused" to "Container released by application", which I believe is more appropriate.

TaoYang526 · 2025-08-14T06:04:55Z

@slfan1989 @aajisaka Could you please help review this PR when you have time?

YARN-11843. Fix potential deadlock when auto-correction of container …

b8c0647

…allocation is enabled

github-actions bot added YARN trunk labels Aug 5, 2025

shameersss1 approved these changes Aug 5, 2025

View reviewed changes

shameersss1 reviewed Aug 5, 2025

View reviewed changes

...main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/AbstractYarnScheduler.java Outdated Show resolved Hide resolved

Update comment.

cc6bc5f

This comment was marked as outdated.

Sign in to view

Fix checkstyle warning.

29cd1d7

TaoYang526 self-assigned this Aug 8, 2025

zeekling reviewed Aug 10, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

YARN-11843. Fix potential deadlock when auto-correction of container allocation is enabled #7855

YARN-11843. Fix potential deadlock when auto-correction of container allocation is enabled #7855

TaoYang526 commented Aug 5, 2025

Uh oh!

TaoYang526 commented Aug 5, 2025

Uh oh!

shameersss1 left a comment

Uh oh!

Uh oh!

shameersss1 commented Aug 5, 2025

Uh oh!

TaoYang526 commented Aug 5, 2025 •

edited

Loading

Uh oh!

shameersss1 commented Aug 5, 2025

Uh oh!

This comment was marked as outdated.

This comment was marked as outdated.

hadoop-yetus commented Aug 5, 2025

Uh oh!

zeekling Aug 10, 2025

Uh oh!

TaoYang526 Aug 11, 2025

Uh oh!

TaoYang526 commented Aug 14, 2025

Uh oh!

Uh oh!

YARN-11843. Fix potential deadlock when auto-correction of container allocation is enabled #7855

Are you sure you want to change the base?

YARN-11843. Fix potential deadlock when auto-correction of container allocation is enabled #7855

Conversation

TaoYang526 commented Aug 5, 2025

Description of PR

How was this patch tested?

For code changes:

Uh oh!

TaoYang526 commented Aug 5, 2025

Uh oh!

shameersss1 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

shameersss1 commented Aug 5, 2025

Uh oh!

TaoYang526 commented Aug 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

shameersss1 commented Aug 5, 2025

Uh oh!

This comment was marked as outdated.

This comment was marked as outdated.

hadoop-yetus commented Aug 5, 2025

Uh oh!

zeekling Aug 10, 2025

Choose a reason for hiding this comment

Uh oh!

TaoYang526 Aug 11, 2025

Choose a reason for hiding this comment

Uh oh!

TaoYang526 commented Aug 14, 2025

Uh oh!

Uh oh!

TaoYang526 commented Aug 5, 2025 •

edited

Loading