[SPARK-54026][BUILD] Use BOM for AWS Java SDK V2 dependency management #52728
Conversation
| <dependency> | ||
| <groupId>software.amazon.awssdk</groupId> | ||
| <artifactId>bom</artifactId> | ||
| <version>${aws.java.sdk.v2.version}</version> |
There was a problem hiding this comment.
Since you have used BOM to manage versions, the explicit version definition at the sub-module's pom.xml is not required, right?
hadoop-cloud/pom.xml
resource-managers/kubernetes/integration-tests/pom.xml
There was a problem hiding this comment.
@pan3793 Both hadoop-cloud/pom.xml and resource-managers/kubernetes/integration-tests/pom.xml use bundle instead of individual jar dependencies while kinesis-asl relies on transitive dependencies of the producer library that use sts, auth and etc. BOM only defines individual dependencies and bundle still requires version to be specified in the dependency. Otherwise maven build fails:
./build/mvn -Pkubernetes-integration-tests -pl resource-managers/kubernetes/integration-tests
Using `mvn` from path: /local/home/vrozov/ws/spark-sbt/build/apache-maven-3.9.11/bin/mvn
[INFO] Maveniverse Nisse 0.4.6 loaded
[INFO] Nisse injecting 39 properties into User Properties
[INFO] Scanning for projects...
[ERROR] [ERROR] Some problems were encountered while processing the POMs:
[ERROR] 'dependencies.dependency.version' for software.amazon.awssdk:bundle:jar is missing. @ line 82, column 17
@
[ERROR] The build could not read 1 project -> [Help 1]
[ERROR]
[ERROR] The project org.apache.spark:spark-kubernetes-integration-tests_2.13:4.1.0-SNAPSHOT (/local/home/vrozov/ws/spark-sbt/resource-managers/kubernetes/integration-tests/pom.xml) has 1 error
[ERROR] 'dependencies.dependency.version' for software.amazon.awssdk:bundle:jar is missing. @ line 82, column 17
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/ProjectBuildingException
It may be necessary to see why both explicitly define dependency on bundle instead of relying on transitive dependencies. Though, I think, it is outside of the scope for this PR, WDYT?
| lazy val jacksonVersion = sys.props.get("fasterxml.jackson.version").getOrElse("2.20.0") | ||
| lazy val jacksonDeps = Bom.dependencies("com.fasterxml.jackson" % "jackson-bom" % jacksonVersion) | ||
| lazy val settings = jacksonDeps ++ Seq( | ||
| lazy val awsJavaSdkV2Version = sys.props.get("aws.java.sdk.v2.version").getOrElse("2.29.52") |
There was a problem hiding this comment.
the drawback is, we need to duplicate the version definition here again, would be better to investigate if we can extract the version from pom.xml directly
There was a problem hiding this comment.
Yes, I agree, it would be better to see if the property can be read from the pom.xml. As it needs to be done not only for AWS Java SDK v2, I'd suggest to do this on a separate PR.
2 participants
What changes were proposed in this pull request?
Use AWS Java SDK V2 BOM for consistent dependency management.
Why are the changes needed?
AWS Java SDK V2 expects consistent dependencies (https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/setup-project-maven.html)
Does this PR introduce any user-facing change?
No
How was this patch tested?
Checked Kinesis ASL works both with maven and SBT
Was this patch authored or co-authored using generative AI tooling?
No