Add DeltaSnapshotManagerFactory with capability-based detection for catalog-managed commits #5562
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…nt interface This introduces the core contracts for catalog-managed commits: - CatalogWithManagedCommits: Trait for catalogs to declare managed commit support - ManagedCommitClient: Generic interface for snapshot operations These contracts enable catalog-agnostic architecture where catalogs (Unity Catalog, Glue, Polaris) can provide their own implementations without kernel-spark knowing specific catalog details.
Provides in-memory implementation that proves interface is complete and usable without actual catalog dependencies.
Move the trait from spark module to kernel-spark module to fix compilation error. The spark module doesn't depend on kernel-spark, so the trait needs to live in kernel-spark where it can directly reference ManagedCommitClient. The trait can still be implemented by catalog classes in the spark module or other modules.
Apply linter/formatter changes to Java files.
- Replace JUnit 4 with JUnit 5 (Jupiter) annotations - Remove Mockito dependency, use simple FakeSnapshot instead - Simplify tests to 3 focused test cases - Tests now compile with kernel-spark dependencies
…esign Use minimal SnapshotStub that only implements getVersion() - the only method actually used by MockManagedCommitClient. Test coverage: - Add and retrieve snapshots - Latest version calculation - Version existence checking - Empty table handling - Error cases (table not found, version not found) - Close clearing state All other Snapshot methods throw UnsupportedOperationException since they're not needed for testing the mock's storage/retrieval logic.
This was referenced Nov 23, 2025
Add adapter that wraps UC-specific UCCatalogManagedClient to implement the generic ManagedCommitClient interface. This adapter: - Lives in kernel-spark module to access both generic and UC interfaces - Delegates getSnapshot() to UCCatalogManagedClient.loadSnapshot() - Implements versionExists() and getLatestVersion() using UC client - Isolates UC-specific logic while providing generic API Also add sparkV2 dependency on kernelUnityCatalog to access UC classes.
… with test doubles
…atalog-managed commits
…mmitClient interface
TimothyW553
force-pushed
the
stack/factory-manager
branch
from
d9af874 to
3c4d13f
Compare
5 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
🥞 Stacked PR
Use this link to review incremental changes.
Which Delta project/connector is this regarding?
Description
How was this patch tested?
Does this PR introduce any user-facing changes?