HADOOP-19622: [ABFS][ReadAheadV2] Implement Read Buffer Manager V2 with improved aggressiveness #7832

anujmodi2021 · 2025-07-29T07:44:46Z

Description of PR

JIRA: https://issues.apache.org/jira/browse/HADOOP-19622

Implementing ReadBufferManagerV2 as per the new design document.
Following capabilities are added to ReadBufferManager:

Configurable minimum and maximum number of prefetch threads.
Configurable minimum and maximum size of cached buffer pool
Dynamically adjusting thread pool size and buffer pool size based on workload requirement and resource utilization but within the limits defined by user.
Mapping prefetched data to file ETag so that multiple streams reading same file can share the cache and save TPS.

For more details on design doc please refer to the design doc attached to parent JIRA: https://issues.apache.org/jira/browse/HADOOP-19596

How was this patch tested?

TBA

hadoop-yetus · 2025-08-07T13:17:08Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 22s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+0 🆗	xmllint	0m 0s		xmllint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 9 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	28m 35s		trunk passed
+1 💚	compile	0m 25s		trunk passed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04
+1 💚	compile	0m 23s		trunk passed with JDK Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09
+1 💚	checkstyle	0m 24s		trunk passed
+1 💚	mvnsite	0m 28s		trunk passed
+1 💚	javadoc	0m 30s		trunk passed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	0m 25s		trunk passed with JDK Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09
+1 💚	spotbugs	0m 49s		trunk passed
+1 💚	shadedclient	25m 59s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 17s		the patch passed
+1 💚	compile	0m 17s		the patch passed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04
+1 💚	javac	0m 17s		the patch passed
+1 💚	compile	0m 15s		the patch passed with JDK Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09
+1 💚	javac	0m 15s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	0m 12s	/results-checkstyle-hadoop-tools_hadoop-azure.txt	hadoop-tools/hadoop-azure: The patch generated 8 new + 5 unchanged - 9 fixed = 13 total (was 14)
+1 💚	mvnsite	0m 20s		the patch passed
+1 💚	javadoc	0m 15s		the patch passed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	0m 16s		the patch passed with JDK Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09
-1 ❌	spotbugs	0m 42s	/new-spotbugs-hadoop-tools_hadoop-azure.html	hadoop-tools/hadoop-azure generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0)
-1 ❌	shadedclient	25m 8s		patch has errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	unit	0m 24s	/patch-unit-hadoop-tools_hadoop-azure.txt	hadoop-azure in the patch failed.
+0 🆗	asflicense	0m 25s		ASF License check generated no output?
		87m 38s

Reason	Tests
SpotBugs	module:hadoop-tools/hadoop-azure
	org.apache.hadoop.fs.azurebfs.services.ReadBuffer.getBuffer() may expose internal representation by returning ReadBuffer.buffer At ReadBuffer.java:by returning ReadBuffer.buffer At ReadBuffer.java:[line 111]
	org.apache.hadoop.fs.azurebfs.services.ReadBuffer.setBuffer(byte[]) may expose internal representation by storing an externally mutable object into ReadBuffer.buffer At ReadBuffer.java:by storing an externally mutable object into ReadBuffer.buffer At ReadBuffer.java:[line 115]

Subsystem	Report/Notes
Docker	ClientAPI=1.51 ServerAPI=1.51 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7832/9/artifact/out/Dockerfile
GITHUB PR	#7832
JIRA Issue	HADOOP-19622
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell detsecrets xmllint spotbugs checkstyle
uname	Linux ae06a4383cc7 5.15.0-142-generic #152-Ubuntu SMP Mon May 19 10:54:31 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `d258d90`
Default Java	Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7832/9/testReport/
Max. process+thread count	546 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7832/9/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2025-08-08T14:24:47Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 35s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+0 🆗	xmllint	0m 0s		xmllint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 10 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	39m 22s		trunk passed
+1 💚	compile	0m 43s		trunk passed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04
+1 💚	compile	0m 39s		trunk passed with JDK Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09
+1 💚	checkstyle	0m 35s		trunk passed
+1 💚	mvnsite	0m 43s		trunk passed
+1 💚	javadoc	0m 40s		trunk passed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	0m 35s		trunk passed with JDK Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09
+1 💚	spotbugs	1m 9s		trunk passed
+1 💚	shadedclient	34m 59s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 31s		the patch passed
+1 💚	compile	0m 32s		the patch passed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04
+1 💚	javac	0m 32s		the patch passed
+1 💚	compile	0m 29s		the patch passed with JDK Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09
+1 💚	javac	0m 29s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	0m 21s	/results-checkstyle-hadoop-tools_hadoop-azure.txt	hadoop-tools/hadoop-azure: The patch generated 9 new + 5 unchanged - 9 fixed = 14 total (was 14)
+1 💚	mvnsite	0m 32s		the patch passed
+1 💚	javadoc	0m 30s		the patch passed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	0m 27s		the patch passed with JDK Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09
-1 ❌	spotbugs	1m 10s	/new-spotbugs-hadoop-tools_hadoop-azure.html	hadoop-tools/hadoop-azure generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0)
+1 💚	shadedclient	35m 35s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	2m 51s		hadoop-azure in the patch passed.
+1 💚	asflicense	0m 37s		The patch does not generate ASF License warnings.
		125m 1s

Reason	Tests
SpotBugs	module:hadoop-tools/hadoop-azure
	org.apache.hadoop.fs.azurebfs.services.ReadBuffer.getBuffer() may expose internal representation by returning ReadBuffer.buffer At ReadBuffer.java:by returning ReadBuffer.buffer At ReadBuffer.java:[line 111]
	org.apache.hadoop.fs.azurebfs.services.ReadBuffer.setBuffer(byte[]) may expose internal representation by storing an externally mutable object into ReadBuffer.buffer At ReadBuffer.java:by storing an externally mutable object into ReadBuffer.buffer At ReadBuffer.java:[line 115]

Subsystem	Report/Notes
Docker	ClientAPI=1.51 ServerAPI=1.51 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7832/10/artifact/out/Dockerfile
GITHUB PR	#7832
JIRA Issue	HADOOP-19622
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell detsecrets xmllint spotbugs checkstyle
uname	Linux f6be6af2509d 5.15.0-143-generic #153-Ubuntu SMP Fri Jun 13 19:10:45 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `2444f92`
Default Java	Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7832/10/testReport/
Max. process+thread count	708 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7832/10/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2025-08-18T06:42:02Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	8m 59s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 1s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 10 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	25m 36s		trunk passed
+1 💚	compile	0m 27s		trunk passed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04
+1 💚	compile	0m 24s		trunk passed with JDK Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09
+1 💚	checkstyle	0m 23s		trunk passed
+1 💚	mvnsite	0m 28s		trunk passed
+1 💚	javadoc	0m 27s		trunk passed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	0m 23s		trunk passed with JDK Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09
+1 💚	spotbugs	0m 46s		trunk passed
+1 💚	shadedclient	20m 48s		branch has no errors when building and testing our client artifacts.
-0 ⚠️	patch	21m 0s		Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 18s		the patch passed
+1 💚	compile	0m 22s		the patch passed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04
+1 💚	javac	0m 22s		the patch passed
+1 💚	compile	0m 19s		the patch passed with JDK Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09
+1 💚	javac	0m 19s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	0m 14s	/results-checkstyle-hadoop-tools_hadoop-azure.txt	hadoop-tools/hadoop-azure: The patch generated 10 new + 5 unchanged - 9 fixed = 15 total (was 14)
+1 💚	mvnsite	0m 18s		the patch passed
+1 💚	javadoc	0m 19s		the patch passed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	0m 19s		the patch passed with JDK Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09
-1 ❌	spotbugs	0m 43s	/new-spotbugs-hadoop-tools_hadoop-azure.html	hadoop-tools/hadoop-azure generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0)
+1 💚	shadedclient	20m 54s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	2m 33s		hadoop-azure in the patch passed.
+1 💚	asflicense	0m 24s		The patch does not generate ASF License warnings.
		86m 15s

Reason	Tests
SpotBugs	module:hadoop-tools/hadoop-azure
	org.apache.hadoop.fs.azurebfs.services.ReadBuffer.getBuffer() may expose internal representation by returning ReadBuffer.buffer At ReadBuffer.java:by returning ReadBuffer.buffer At ReadBuffer.java:[line 111]
	org.apache.hadoop.fs.azurebfs.services.ReadBuffer.setBuffer(byte[]) may expose internal representation by storing an externally mutable object into ReadBuffer.buffer At ReadBuffer.java:by storing an externally mutable object into ReadBuffer.buffer At ReadBuffer.java:[line 115]

Subsystem	Report/Notes
Docker	ClientAPI=1.51 ServerAPI=1.51 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7832/11/artifact/out/Dockerfile
GITHUB PR	#7832
JIRA Issue	HADOOP-19622
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux 5e04ca0b1e01 5.15.0-142-generic #152-Ubuntu SMP Mon May 19 10:54:31 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `a6064ef`
Default Java	Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7832/11/testReport/
Max. process+thread count	567 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7832/11/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2025-08-19T15:30:18Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 22s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+0 🆗	detsecrets	0m 1s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 10 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	27m 5s		trunk passed
+1 💚	compile	0m 25s		trunk passed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04
+1 💚	compile	0m 24s		trunk passed with JDK Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09
+1 💚	checkstyle	0m 24s		trunk passed
+1 💚	mvnsite	0m 30s		trunk passed
+1 💚	javadoc	0m 30s		trunk passed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	0m 23s		trunk passed with JDK Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09
+1 💚	spotbugs	0m 46s		trunk passed
+1 💚	shadedclient	20m 55s		branch has no errors when building and testing our client artifacts.
-0 ⚠️	patch	21m 8s		Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 21s		the patch passed
+1 💚	compile	0m 20s		the patch passed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04
+1 💚	javac	0m 20s		the patch passed
+1 💚	compile	0m 18s		the patch passed with JDK Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09
+1 💚	javac	0m 18s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	0m 14s	/results-checkstyle-hadoop-tools_hadoop-azure.txt	hadoop-tools/hadoop-azure: The patch generated 10 new + 5 unchanged - 9 fixed = 15 total (was 14)
+1 💚	mvnsite	0m 18s		the patch passed
+1 💚	javadoc	0m 19s		the patch passed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	0m 19s		the patch passed with JDK Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09
-1 ❌	spotbugs	0m 48s	/new-spotbugs-hadoop-tools_hadoop-azure.html	hadoop-tools/hadoop-azure generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0)
+1 💚	shadedclient	21m 21s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	2m 30s		hadoop-azure in the patch passed.
+1 💚	asflicense	0m 26s		The patch does not generate ASF License warnings.
		79m 57s

Reason	Tests
SpotBugs	module:hadoop-tools/hadoop-azure
	org.apache.hadoop.fs.azurebfs.services.ReadBuffer.getBuffer() may expose internal representation by returning ReadBuffer.buffer At ReadBuffer.java:by returning ReadBuffer.buffer At ReadBuffer.java:[line 111]
	org.apache.hadoop.fs.azurebfs.services.ReadBuffer.setBuffer(byte[]) may expose internal representation by storing an externally mutable object into ReadBuffer.buffer At ReadBuffer.java:by storing an externally mutable object into ReadBuffer.buffer At ReadBuffer.java:[line 115]

Subsystem	Report/Notes
Docker	ClientAPI=1.51 ServerAPI=1.51 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7832/12/artifact/out/Dockerfile
GITHUB PR	#7832
JIRA Issue	HADOOP-19622
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux ed73b0a40bbd 5.15.0-142-generic #152-Ubuntu SMP Mon May 19 10:54:31 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `12d56c4`
Default Java	Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7832/12/testReport/
Max. process+thread count	555 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7832/12/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

anujmodi2021 added 2 commits July 28, 2025 21:44

RBMV2 Implementation

37898b4

Test Improvements

28cb97f

github-actions bot added trunk TOOLS ABFS labels Jul 29, 2025