Switch spark connector to use Weaviate Java client6 #130

antas-marcin · 2025-11-17T15:44:49Z

This PR adds switches Weaviate Java v5 client to latest client6

orca-security-eu

Orca Security Scan Summary

Status	Check	Issues by priority
Passed	Infrastructure as Code	0 0 0 0	View in Orca
Passed	SAST	0 0 0 0	View in Orca
Passed	Secrets	0 0 0 0	View in Orca
Passed	Vulnerabilities	0 0 0 0	View in Orca

g-despot · 2025-11-27T12:00:51Z

src/test/scala/TestWeaviateDataWriter.scala

-    assert(weaviateObject.getProperties.get("content") == "Sam")
-    assert(weaviateObject.getProperties.get("wordCount") == 5)
-    assert(weaviateObject.getTenant == "TenantA")
+    assert(weaviateObject.properties().get("title").equals("Sam"))


Suggested change

assert(weaviateObject.properties().get("title").equals("Sam"))

assert(weaviateObject.properties().get("title") == ("Sam"))

Not sure about Scala, but in Java you actually want to compare strings with .equals, because it compares the contents themselves, whereas == compares object references (which will always differ).

nvmnd, seems Scala doesn't have that catch 👍

g-despot

Looks good

bevzzz · 2025-11-27T13:26:06Z

build.sbt

-lazy val scalaCollectionCompatVersion = "2.13.0"
+lazy val sparkVersion = "4.0.1"
+lazy val grpcNettyShadedVersion = "1.76.0"
+lazy val weaviateClient6Version = "6.0.0-RC2"


We should be able to use 6.0.0 now ✨

bevzzz · 2025-11-27T13:30:53Z

src/main/scala/Weaviate.scala

+    if (result.isEmpty) throw WeaviateClassNotFoundError(s"Collection ${className} was not found.")
+    val properties = result.get().properties().asScala


suggestion: if you want to lean into Optional syntax hard

Suggested change

if (result.isEmpty) throw WeaviateClassNotFoundError(s"Collection ${className} was not found.")

val properties = result.get().properties().asScala

if (result.isEmpty) throw WeaviateClassNotFoundError(s"Collection ${className} was not found.")

val properties = result

.map(c -> c.properties().asScala)

.orElseThrow(() -> WeaviateClassNotFoundError(s"Collection ${className} was not found."))

bevzzz · 2025-11-27T13:35:24Z

src/main/scala/WeaviateDataWriter.scala

    if (weaviateOptions.id == null) {
-      builder.id(java.util.UUID.randomUUID.toString)
+      builder.uuid(java.util.UUID.randomUUID.toString)
    }


suggestion: this if-block is not strictly necessary

I believe this was originally added because BatchInsert over gRPC does not allow the client to leave out the UUID (unlike REST-based inserts).

It's something I also stumbled upon while working on client6, so in the new version every WeaviateObject gets a random UUID by default.

bevzzz · 2025-11-27T13:38:23Z

src/main/scala/WeaviateDataWriter.scala

+    val allVectors = ListBuffer.empty[Vectors]
+    if (vector != null) {
+      allVectors += Vectors.of(vector)
+    }
    if (vectors.nonEmpty) {
-      builder.vectors(vectors.map { case (key, arr) => key -> arr.map(Float.box) }.asJava)
+      val arr = vectors.map { case (key, arr) => Vectors.of(key, arr) }.toArray
+      allVectors ++= arr
    }
    if (multiVectors.nonEmpty) {
-      builder.multiVectors(multiVectors.map { case (key, multiVector) => key -> multiVector.map { vec => { vec.map(Float.box) }} }.toMap.asJava)
+      val arr = multiVectors.map { case (key, multiVector) => Vectors.of(key, multiVector) }.toArray
+      allVectors ++= arr
    }


note: there's now Vectors::withVectors method, that should let you do sth like:

val vectors = new Vectors(); if (vector != null) { vectors = vectors.withVectors(Vectors.of(vector)) } if (vectors.nonEmpty) { // ... so on ... }

But if the list-syntax works well for you there's also no real need to change it, bc withVectors creates more intermediate objects (more GC work).

bevzzz · 2025-11-27T13:39:56Z

src/main/scala/WeaviateDataWriter.scala

+          dt = p.dataTypes().get(0)
        }
      })
      if ((dt == "geoCoordinates" || dt == "phoneNumber") && valueFromField.isInstanceOf[String]) {


suggestion: you can also use data type names from the library itself, e.g. DataType.GEO_COORDINATES or DataType.PHONE_NUMBER

bevzzz · 2025-11-27T13:40:32Z

src/main/scala/WeaviateDataWriter.scala


  override def abort(): Unit = {
-    // TODO rollback previously written batch results if issue occured
+    // TODO rollback previously written batch results if issue occurred


question: Is that something Weaviate supports? Rolling things back?

bevzzz · 2025-11-27T13:42:10Z

src/test/scala/TestWeaviateDataWriter.scala

-    assert(weaviateObject.getProperties.get("content") == "Sam")
-    assert(weaviateObject.getProperties.get("wordCount") == 5)
-    assert(weaviateObject.getTenant == "TenantA")
+    assert(weaviateObject.properties().get("title").equals("Sam"))


nvmnd, seems Scala doesn't have that catch 👍

bevzzz · 2025-11-27T13:46:19Z

src/test/scala/UtilsTest.scala

+    val moreNested = new Property.Builder("moreNested", DataType.OBJECT).
+      nestedProperties(Property.text("a"), Property.number("b")).build()
+
+    val nestedObjects = new Property.Builder("nestedObjects", DataType.OBJECT_ARRAY)
+      .nestedProperties(
+        Property.bool("nestedBoolLvl2"),
+        Property.date("nestedDateLvl2"),
+        Property.numberArray("nestedNumbersLvl2"),
+        moreNested
+      ).build()
+
+    new Property.Builder("objectProperty", dataType)
+      .nestedProperties(
+        Property.integer("nestedInt"),
+        Property.number("nestedNumber"),
+        Property.text("nestedText"),
+        nestedObjects
+      ).build()


suggestion: you should be able to do

Property.object("objectProperty", obj -> obj .nestedProperties( ..., Property.numberArray("nestedNumbersLvl2"), Property.object("moreNested", more -> more .nestedProperties( Property.text("a"), Property.number("b") ) ) ) )

antas-marcin requested a review from a team as a code owner November 17, 2025 15:44

orca-security-eu bot reviewed Nov 17, 2025

View reviewed changes

antas-marcin force-pushed the client6-java-switch branch 3 times, most recently from e43d19b to fbbff3e Compare November 17, 2025 16:46

Switch spark connector to use Java client6

7c70d30

antas-marcin force-pushed the client6-java-switch branch from fbbff3e to 7c70d30 Compare November 17, 2025 16:46

antas-marcin changed the title ~~Switch to Weaviate Java Client6~~ Switch spark connector to use Java client6 Nov 17, 2025

antas-marcin changed the title ~~Switch spark connector to use Java client6~~ Switch spark connector to use Weaviate Java client6 Nov 17, 2025

antas-marcin added 2 commits November 17, 2025 18:00

update project dependencies to latest versions

0100b27

implementation chores, removing comments

e275d56

antas-marcin force-pushed the client6-java-switch branch 2 times, most recently from 54875c1 to 694010a Compare November 26, 2025 12:02

antas-marcin added 2 commits November 26, 2025 13:06

switch to Client6 RC2 version

76e405b

fix pyspark tests

d0d0a72

antas-marcin force-pushed the client6-java-switch branch from 694010a to d0d0a72 Compare November 26, 2025 12:43

g-despot reviewed Nov 27, 2025

View reviewed changes

g-despot approved these changes Nov 27, 2025

View reviewed changes

bevzzz approved these changes Nov 27, 2025

View reviewed changes

use client6 version 6.0.0

a0a4ce8

	assert(weaviateObject.properties().get("title").equals("Sam"))
	assert(weaviateObject.properties().get("title") == ("Sam"))

		if (result.isEmpty) throw WeaviateClassNotFoundError(s"Collection ${className} was not found.")
		val properties = result.get().properties().asScala

Switch spark connector to use Weaviate Java client6 #130

Are you sure you want to change the base?

Switch spark connector to use Weaviate Java client6 #130

Uh oh!

Conversation

antas-marcin commented Nov 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

orca-security-eu bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Orca Security Scan Summary

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

g-despot left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

antas-marcin commented Nov 17, 2025 •

edited

Loading

orca-security-eu bot left a comment •

edited

Loading