HBASE-29574 Resolve the issue that splitting HFiles results in write failures due to storage policy during bulkload #7283

Huginn-kio · 2025-09-07T09:22:25Z

HBASE-29574 Resolve the issue that splitting HFiles results in write failures due to storage policy during bulkload

Huginn-kio · 2025-09-07T09:24:14Z

When performing bulk load in a cluster equipped only with SSD storage, if HFile splitting occurs, writing the split files (.top and .bottom) may fail due to the storage policy with this exception message: User class threw exception: org.apache.hadoop.ipc.RemoteException(java.io.IOException): File /bulkload/input/bulkload_job/c/.tmp/xxxxxxxxxxxxxxxx.top could only be written to 0 of the 1 minReplication nodes. There are 10 datanode(s) running and 0 node(s) are excluded in this operation.

We found that the storage policy of /bulkload/input/bulkload_job/c/.tmp/ is set to HOT, which ultimately causes the write failure.

…failures due to storage policy during bulkload

Apache-HBase · 2025-09-08T04:06:41Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 26s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	hbaseanti	0m 0s		Patch does not have any anti-patterns.
			_ master Compile Tests _
+1 💚	mvninstall	3m 32s		master passed
+1 💚	compile	3m 25s		master passed
+1 💚	checkstyle	0m 38s		master passed
+1 💚	spotbugs	1m 37s		master passed
+1 💚	spotless	0m 51s		branch has no errors when running spotless:check.
			_ Patch Compile Tests _
+1 💚	mvninstall	3m 8s		the patch passed
+1 💚	compile	3m 26s		the patch passed
+1 💚	javac	3m 26s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 37s		the patch passed
+1 💚	spotbugs	1m 43s		the patch passed
+1 💚	hadoopcheck	12m 28s		Patch does not cause any errors with Hadoop 3.3.6 3.4.0.
+1 💚	spotless	0m 46s		patch has no errors when running spotless:check.
			_ Other Tests _
+1 💚	asflicense	0m 10s		The patch does not generate ASF License warnings.
		40m 47s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7283/3/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#7283
Optional Tests	dupname asflicense javac spotbugs checkstyle codespell detsecrets compile hadoopcheck hbaseanti spotless
uname	Linux a7170548bf28 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `4b89973`
Default Java	Eclipse Adoptium-17.0.11+9
Max. process+thread count	85 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7283/3/console
versions	git=2.34.1 maven=3.9.8 spotbugs=4.7.3
Powered by	Apache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2025-09-08T07:33:26Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 28s		Docker mode activated.
-0 ⚠️	yetus	0m 3s		Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --author-ignore-list --blanks-eol-ignore-file --blanks-tabs-ignore-file --quick-hadoopcheck
			_ Prechecks _
			_ master Compile Tests _
+1 💚	mvninstall	3m 21s		master passed
+1 💚	compile	1m 5s		master passed
+1 💚	javadoc	0m 29s		master passed
+1 💚	shadedjars	6m 18s		branch has no errors when building our shaded downstream artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	3m 12s		the patch passed
+1 💚	compile	0m 59s		the patch passed
+1 💚	javac	0m 59s		the patch passed
+1 💚	javadoc	0m 28s		the patch passed
+1 💚	shadedjars	6m 11s		patch has no errors when building our shaded downstream artifacts.
			_ Other Tests _
+1 💚	unit	220m 15s		hbase-server in the patch passed.
		247m 33s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7283/3/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile
GITHUB PR	#7283
Optional Tests	javac javadoc unit compile shadedjars
uname	Linux 5e519300101a 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `4b89973`
Default Java	Eclipse Adoptium-17.0.11+9
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7283/3/testReport/
Max. process+thread count	5067 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7283/3/console
versions	git=2.34.1 maven=3.9.8
Powered by	Apache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

wchevreuil

Can we add UT to reproduce the issue, and then validate these changes solve it?

Huginn-kio · 2025-09-21T14:32:52Z

@wchevreuil Thank you for helping with the review. I have added the ut

Apache-HBase · 2025-09-23T13:40:48Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 27s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	hbaseanti	0m 0s		Patch does not have any anti-patterns.
			_ master Compile Tests _
+1 💚	mvninstall	3m 10s		master passed
+1 💚	compile	3m 24s		master passed
+1 💚	checkstyle	0m 37s		master passed
+1 💚	spotbugs	1m 33s		master passed
+1 💚	spotless	0m 45s		branch has no errors when running spotless:check.
			_ Patch Compile Tests _
+1 💚	mvninstall	3m 17s		the patch passed
+1 💚	compile	3m 17s		the patch passed
+1 💚	javac	3m 17s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 36s		the patch passed
+1 💚	spotbugs	1m 41s		the patch passed
+1 💚	hadoopcheck	12m 2s		Patch does not cause any errors with Hadoop 3.3.6 3.4.1.
+1 💚	spotless	0m 45s		patch has no errors when running spotless:check.
			_ Other Tests _
+1 💚	asflicense	0m 9s		The patch does not generate ASF License warnings.
		39m 32s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7283/3/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#7283
Optional Tests	dupname asflicense javac spotbugs checkstyle codespell detsecrets compile hadoopcheck hbaseanti spotless
uname	Linux c983bb00b060 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `c183bb5`
Default Java	Eclipse Adoptium-17.0.11+9
Max. process+thread count	85 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7283/3/console
versions	git=2.34.1 maven=3.9.8 spotbugs=4.7.3
Powered by	Apache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2025-09-23T17:02:53Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 30s		Docker mode activated.
-0 ⚠️	yetus	0m 3s		Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --author-ignore-list --blanks-eol-ignore-file --blanks-tabs-ignore-file --quick-hadoopcheck
			_ Prechecks _
			_ master Compile Tests _
+1 💚	mvninstall	4m 7s		master passed
+1 💚	compile	1m 38s		master passed
+1 💚	javadoc	1m 5s		master passed
+1 💚	shadedjars	6m 53s		branch has no errors when building our shaded downstream artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	3m 32s		the patch passed
+1 💚	compile	1m 6s		the patch passed
+1 💚	javac	1m 6s		the patch passed
+1 💚	javadoc	0m 27s		the patch passed
+1 💚	shadedjars	6m 13s		patch has no errors when building our shaded downstream artifacts.
			_ Other Tests _
+1 💚	unit	211m 56s		hbase-server in the patch passed.
		241m 40s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7283/3/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile
GITHUB PR	#7283
Optional Tests	javac javadoc unit compile shadedjars
uname	Linux 05ea275722d6 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `c183bb5`
Default Java	Eclipse Adoptium-17.0.11+9
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7283/3/testReport/
Max. process+thread count	4187 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7283/3/console
versions	git=2.34.1 maven=3.9.8
Powered by	Apache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

Huginn-kio force-pushed the HBASE-29574 branch from 2b7ea3c to 30659c3 Compare September 7, 2025 09:28

This comment has been minimized.

Sign in to view

HBASE-29574 Resolve the issue that splitting HFiles results in write …

4b89973

…failures due to storage policy during bulkload

Huginn-kio force-pushed the HBASE-29574 branch from 30659c3 to 4b89973 Compare September 8, 2025 03:24

wchevreuil reviewed Sep 8, 2025

View reviewed changes

Huginn-kio force-pushed the HBASE-29574 branch from 4b89973 to cf2c9b2 Compare September 21, 2025 10:15

Huginn-kio changed the base branch from master to branch-2.4 September 21, 2025 10:16

Huginn-kio force-pushed the HBASE-29574 branch from cf2c9b2 to fccf3c6 Compare September 21, 2025 14:26

Huginn-kio changed the base branch from branch-2.4 to master September 21, 2025 14:27

This comment has been minimized.

Sign in to view

add ut

7a7fc49

Huginn-kio force-pushed the HBASE-29574 branch from fccf3c6 to 7a7fc49 Compare September 23, 2025 08:43

This comment has been minimized.

Sign in to view

spotless check

c183bb5

This comment has been minimized.

Sign in to view

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

HBASE-29574 Resolve the issue that splitting HFiles results in write failures due to storage policy during bulkload #7283

HBASE-29574 Resolve the issue that splitting HFiles results in write failures due to storage policy during bulkload #7283

Huginn-kio commented Sep 7, 2025

Uh oh!

Huginn-kio commented Sep 7, 2025

Uh oh!

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

Apache-HBase commented Sep 8, 2025

Uh oh!

Apache-HBase commented Sep 8, 2025

Uh oh!

wchevreuil left a comment

Uh oh!

Huginn-kio commented Sep 21, 2025

Uh oh!

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

Apache-HBase commented Sep 23, 2025

Uh oh!

Apache-HBase commented Sep 23, 2025

Uh oh!

Uh oh!

HBASE-29574 Resolve the issue that splitting HFiles results in write failures due to storage policy during bulkload #7283

Are you sure you want to change the base?

HBASE-29574 Resolve the issue that splitting HFiles results in write failures due to storage policy during bulkload #7283

Conversation

Huginn-kio commented Sep 7, 2025

Uh oh!

Huginn-kio commented Sep 7, 2025

Uh oh!

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

Apache-HBase commented Sep 8, 2025

Uh oh!

Apache-HBase commented Sep 8, 2025

Uh oh!

wchevreuil left a comment

Choose a reason for hiding this comment

Uh oh!

Huginn-kio commented Sep 21, 2025

Uh oh!

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

Apache-HBase commented Sep 23, 2025

Uh oh!

Apache-HBase commented Sep 23, 2025

Uh oh!

Uh oh!