[SPARK-54006][TESTS][DSTREAM][3.5] Fix for NoClassDefFoundError in WithAggregationKinesisBackedBlockRDDSuite and KinesisTestUitls

[vrozov]

What changes were proposed in this pull request?

• Add explicit dependency on javax.xml.bind:jaxb-api to the test scope of Kinesis ASL.
• Use streamExists waiter to wait for newly created Kinesis stream to become active

Why are the changes needed?

• JDK 9 and above removed javax.xml.bind package from JDK, so when test runs on JDK 9 and above it fails.
• Avoid ResourceNotFoundException while sending data

Does this PR introduce any user-facing change?

No

How was this patch tested?

with JDK 17
ENABLE_KINESIS_TESTS=1 build/mvn test -Pkinesis-asl -pl connector/kinesis-asl
and
ENABLE_KINESIS_TESTS=1 build/sbt -Pkinesis-asl

Was this patch authored or co-authored using generative AI tooling?

No

[dongjoon-hyun]

Hi, @vrozov . Currently, this PR has two mixed themes unlike the PR title and description.

Add explicit dependency on javax.xml.bind:jaxb-api to the test scope of Kinesis ASL.

JDK 9 and above removed javax.xml.bind package from JDK, so when test runs on JDK 9 and above it fails.

In order to help your PR get merged, I'd like to recommend to split this PR into two.

1. One is only chasing javax.xml.bind:jaxb-api (the scope of PR title and description)
2. The other is refactoring the code via the following.

private lazy val streamExistsWaiter = kinesisClient.waiters().streamExists()

Also, cc @sarutak , too

[vrozov]

@dongjoon-hyun @sarutak The primary focus of the PR is to fix NoClassDefFoundError. The problem is that without the second change test still fails as Kinesis throws ResourceNotFoundException even after describe stream returns ACTIVE. Please see #52630 (comment). I can split this PR into 2, but one will continue to fail till the second one is merged.

[dongjoon-hyun]

@dongjoon-hyun @sarutak The primary focus of the PR is to fix NoClassDefFoundError. The problem is that without the second change test still fails as Kinesis throws ResourceNotFoundException even after describe stream returns ACTIVE. Please see #52630 (comment). I can split this PR into 2, but one will continue to fail till the second one is merged.

Sorry but it literally means your PR description is insufficient because it doesn't include the code change.

[vrozov]

@dongjoon-hyun Updated title and description, please check.

[pan3793]

root pom.xml defines this dep with <version>2.2.11</version> in dependencyManagement, does that version work? if so, please remove the version overriding here.