[SPARK-53329] Improve exception handling when adding artifacts

[HendrikHuebner]

What changes were proposed in this pull request?

When a user sends multiple artifacts with the addArtifacts API, we process each artifact one at a time on the server-side.

If the server detects the user attempting to modify an artifact (by overwriting an existing artifact of the same path with a different byte sequence), an exception is immediately thrown and artifact addition process is terminated.

Instead, the operation should be idempotent and the server should try to add as many artifacts as possible instead of returning early.

Why are the changes needed?

As explained, if the server encounters an error while adding artifacts it will return immediately. This can be a bit wasteful as the server discards all other artifacts sent over the wire regardless of their own status. Thus, an improvement can be made to process all artifacts, catch any exceptions and rethrow them at the end.

Does this PR introduce any user-facing change?

This PR does not modify the existing API or the return codes. If the above scenario is triggered, the only user facing change is that the server adds as many artifacts as possible. Therefore it should be fully backwards compatible. Additionally, if more than one artifact already existed, its exception is added as a suppressed exception. Currently, these suppressed exceptions are not serialized into the grpc object and sent over the wire, however.

How was this patch tested?

Unit tests and local testing.

Was this patch authored or co-authored using generative AI tooling?

No

[vicennial]

Thanks for the improvement, LGTM!

One important behaviour to note: Currently, the AddArtifact RPC is retry-friendly as overwriting a file with the same byte sequence is NOT considered as a "true overwrite" and thus, is simply ignored.
This PR does not change the above semantics and AddArtifact will remain retry-friendly.

Currently, these suppressed exceptions are not serialized into the grpc object and sent over the wire, however.

Thanks for the callout! I've filed https://issues.apache.org/jira/browse/SPARK-53350 to track this as a potential improvement/follow-up.

[vicennial]

Note to other reviewers: Hide whitespace changes

[HendrikHuebner]

@vicennial what do you mean? I saw there was a superfluous whitespace somewhere, I'll remove it before merging.

[vicennial]

@HendrikHuebner Not related to the code per-se, just the way that github displays the diff.

By default it does not hide whitespace modifications:

But if its enabled explicitly, the diff is cleaner and easier to understand what changed: