feat: Support Keyspace #2454

PeterChen13579 · 2025-08-18T20:28:55Z

Support AWS Keyspace queries

codecov · 2025-08-19T02:48:14Z

Codecov Report

❌ Patch coverage is 1.23457% with 80 lines in your changes missing coverage. Please review.
✅ Project coverage is 79.55%. Comparing base (2b937bf) to head (6bf8b14).
⚠️ Report is 1 commits behind head on develop.

Files with missing lines	Patch %	Lines
src/data/KeyspaceBackend.hpp	0.00%	80 Missing ⚠️

❌ Your patch status has failed because the patch coverage (1.23%) is below the target coverage (20.00%). You can increase the patch coverage or adjust the target coverage.

Additional details and impacted files

@@             Coverage Diff             @@
##           develop    #2454      +/-   ##
===========================================
- Coverage    79.95%   79.55%   -0.41%     
===========================================
  Files          384      385       +1     
  Lines        16037    16118      +81     
  Branches      8387     8438      +51     
===========================================
- Hits         12823    12822       -1     
- Misses        1925     2006      +81     
- Partials      1289     1290       +1

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

docs/config-description.md

src/data/CassandraBackend.hpp

src/data/cassandra/impl/Batch.cpp

src/data/cassandra/impl/Cluster.cpp

kuznetsss

LGTM, but let's wait for an approval from @godexsoft

godexsoft

I have not thoroughly reviewed the code because i don't yet understand why we need so many changes if keyspaces is supposed to be compatible with cassandra?
In general i'm not a fan of adding custom stuff to the cassandra backend.. if we need to have custom behaviour maybe we should create a separate backend and inherit where we can, modify where we need.

src/data/CassandraBackend.hpp

docs/config-description.md

src/data/cassandra/Schema.hpp

PeterChen13579 · 2025-08-27T19:32:13Z

@godexsoft For some context, Keyspace is compatible with Cassandra but there's a few limitations. Specifically the places where we had:

IF ... IN clause 
PER PARTITION LIMIT 1 
Tuple comparisons, e.g.  (taxon, token_id) > ?

To tackle this, I split the queries that contains the above into 2 queries. ie, 2 queries will be equivalent to the 1 from before.

The goal of this PR is to get Clio to run with Keyspace while those running cassandra/scylladb would work the exact same.
Also, keyspace is trying to support the above statements that they don't support right now. Hopefully once they get it supported, I'm happy to revert this PR.

mathbunnyru

Left some comments to mostly make the code more readable and less error-prone

docs/config-description.md

src/data/CassandraBackend.hpp

src/data/cassandra/Schema.hpp

src/data/cassandra/impl/Cluster.cpp

src/data/cassandra/impl/Cluster.hpp

tests/integration/data/BackendFactoryTests.cpp

src/data/BackendFactory.hpp

src/data/CassandraBackend.hpp

godexsoft

I left a few comments, nothing major, should be easy enough to fix.
For the Schema inheritance idea let's discuss. I think we can still do slightly better than having two copies of the schema with modifications

src/data/CassandraBackend.hpp

src/data/KeyspaceBackend.hpp

src/data/cassandra/CassandraBackendFamily.hpp

src/data/cassandra/KeyspaceSchema.hpp

src/data/cassandra/impl/Cluster.hpp

src/data/CassandraBackend.hpp

godexsoft

Looks good 👍

Need to RC

mathbunnyru · 2025-10-06T12:56:25Z

src/data/BackendFactory.hpp

-        backend = std::make_shared<data::cassandra::CassandraBackend>(
-            data::cassandra::SettingsProvider{cfg}, cache, readOnly
-        );
+        if (cfg.getValueView("provider").asString() == toString(cassandra::impl::Provider::Keyspace)) {


It's better to do it vice-versa, have a function to convert string to a enum, and then compare enum values

mathbunnyru · 2025-10-06T12:57:31Z

src/data/KeyspaceBackend.hpp

+ * @tparam SettingsProviderType The settings provider type to use
+ * @tparam ExecutionStrategyType The execution strategy type to use
+ * @tparam FetchLedgerCacheType The ledger header cache type to use


Let's remove to use in the end, it's not helpful

mathbunnyru · 2025-10-06T12:59:03Z

src/data/KeyspaceBackend.hpp

+        // !range_.has_value() means the table 'ledger_range' is not populated;
+        // This would be the first write to the table.
+        // In this case, insert both min_sequence/max_sequence range into the table.
+        if (not(range_.has_value())) {


Well, using not this way ( not(x) instead of not x ) is something new, let's not do it 😆

mathbunnyru · 2025-10-06T13:00:02Z

src/data/KeyspaceBackend.hpp

+            executor_.writeSync(schema_->insertLedgerRange, false, ledgerSequence_);
+            executor_.writeSync(schema_->insertLedgerRange, true, ledgerSequence_);
+        }
+
+        if (not this->executeSyncUpdate(schema_->updateLedgerRange.bind(ledgerSequence_, true, ledgerSequence_ - 1))) {


I would probably add inline comments to a boolean variable like executor_.writeSync(schema_->insertLedgerRange, /* some_opt_name = */ false, ledgerSequence_);

mathbunnyru · 2025-10-06T13:01:04Z

src/data/KeyspaceBackend.hpp

+            // Keyspace and ScyllaDB uses the same logic for taxon-filtered queries
+            nftIDs = fetchNFTIDsByTaxon(issuer, *taxon, limit, cursorIn, yield);
+        } else {
+            // --- Amazon Keyspaces Workflow for non-taxon queries ---


No need for --- in the comment, I don't think we do that

mathbunnyru · 2025-10-06T13:06:17Z

src/data/cassandra/CassandraBackendFamily.hpp

+        if (auto const res = handle_.connect(); not res)
+            throw std::runtime_error("Could not connect to database: " + res.error());
+
+        if (not readOnly) {
+            if (auto const res = handle_.execute(schema_.createKeyspace); not res) {


same as above

mathbunnyru · 2025-10-06T13:06:49Z

src/data/cassandra/CassandraBackendFamily.hpp

+                    throw std::runtime_error("Could not create keyspace: " + res.error());
+            }
+
+            if (auto const res = handle_.executeEach(schema_.createSchema); not res)


same as above

mathbunnyru · 2025-10-06T13:07:30Z

src/data/cassandra/CassandraBackendFamily.hpp

+        if (auto const res = executor_.read(yield, schema_->selectLatestLedger); res) {
+            if (auto const& result = res.value(); result) {
+                if (auto const maybeValue = result.template get<uint32_t>(); maybeValue)
+                    return maybeValue;


same as above

mathbunnyru · 2025-10-06T13:26:45Z

tests/integration/data/cassandra/BackendTests.cpp

        {"database.cassandra.port", ConfigValue{ConfigType::Integer}.optional()},
        {"database.cassandra.keyspace",
         ConfigValue{ConfigType::String}.defaultValue(TestGlobals::instance().backendKeyspace)},
+        {"database.cassandra.provider", ConfigValue{ConfigType::String}.defaultValue("cassandra")},


Let's move it to a variable (and in places below as well)

mathbunnyru · 2025-10-06T13:28:05Z

src/data/cassandra/impl/Cluster.hpp

+enum class Provider { Cassandra, Keyspace };
+
+inline std::string
+toString(Provider provider)


You will have a function providerFromString instead, and this function could be removed

mathbunnyru · 2025-10-06T13:31:00Z

Left some comments, mostly about style

When you use optional/expected, I suggest using this pattern (has_value() + *):

if (my_opt.has_value()) {
  use_it(*my_opt);
}

PeterChen13579 added 2 commits August 18, 2025 15:38

feat: Support Keyspace

bbd2884

remove unnecessary comment

d598396

kuznetsss reviewed Aug 19, 2025

View reviewed changes

docs/config-description.md Outdated Show resolved Hide resolved

src/data/CassandraBackend.hpp Outdated Show resolved Hide resolved

src/data/cassandra/impl/Batch.cpp Show resolved Hide resolved

src/data/cassandra/impl/Cluster.cpp Outdated Show resolved Hide resolved

PeterChen13579 and others added 6 commits August 19, 2025 16:19

fix bug and comments

c3e0442

Merge branch 'develop' into SupportKeyspace

47c2af0

fix comments

0d9a83f

merge develop

8f6bec2

fix val

30da8d8

Merge branch 'develop' into SupportKeyspace

e03f5e4

PeterChen13579 requested a review from kuznetsss August 27, 2025 14:56

kuznetsss requested a review from godexsoft August 27, 2025 15:08

kuznetsss reviewed Aug 27, 2025

View reviewed changes

godexsoft reviewed Aug 27, 2025

View reviewed changes

src/data/CassandraBackend.hpp Outdated Show resolved Hide resolved

docs/config-description.md Show resolved Hide resolved

src/data/cassandra/Schema.hpp Outdated Show resolved Hide resolved

src/data/cassandra/Schema.hpp Show resolved Hide resolved

let scylladb use same schema to update ledger range as before

f88ce31

mathbunnyru previously requested changes Aug 28, 2025

View reviewed changes

mathbunnyru and others added 12 commits August 28, 2025 15:11

Merge branch 'develop' into SupportKeyspace

84db880

Merge branch 'develop' into SupportKeyspace

8bc36c2

Merge branch 'develop' into SupportKeyspace

fc88abd

fix comments

2653010

change connections per host

497721e

draft save for now

e0fbb27

fix build and hierachy

04fa4dd

Merge branch 'develop' into SupportKeyspace

c48cb3e

fix comments and hierchy

ee42d73

fix doxygen

4ad291d

fix doxygen

8e4ea39

fix a few comments

85f6133

godexsoft reviewed Oct 2, 2025

View reviewed changes

src/data/BackendFactory.hpp Outdated Show resolved Hide resolved

godexsoft reviewed Oct 2, 2025

View reviewed changes

src/data/BackendFactory.hpp Show resolved Hide resolved

godexsoft reviewed Oct 2, 2025

View reviewed changes

src/data/CassandraBackend.hpp Show resolved Hide resolved

godexsoft requested changes Oct 2, 2025

View reviewed changes

PeterChen13579 and others added 4 commits October 2, 2025 11:08

fix a few comments

1222f67

Use shared schema as base

e8222b7

isolate for keyspace schema

91e0077

Merge branch 'develop' into SupportKeyspace

1c4ff26

godexsoft previously approved these changes Oct 3, 2025

View reviewed changes

seperate quorum

d5c30b8

PeterChen13579 dismissed godexsoft’s stale review via d5c30b8 October 3, 2025 15:00

add back idempotent

6bf8b14

godexsoft approved these changes Oct 3, 2025

View reviewed changes

PeterChen13579 enabled auto-merge (squash) October 3, 2025 15:28

PeterChen13579 merged commit 1cfa06c into XRPLF:develop Oct 3, 2025
13 checks passed

mathbunnyru reviewed Oct 6, 2025

View reviewed changes

feat: Support Keyspace #2454

feat: Support Keyspace #2454

Uh oh!

Conversation

PeterChen13579 commented Aug 18, 2025

Uh oh!

codecov bot commented Aug 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kuznetsss left a comment

Choose a reason for hiding this comment

Uh oh!

godexsoft left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

PeterChen13579 commented Aug 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mathbunnyru left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

godexsoft left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

godexsoft left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mathbunnyru commented Oct 6, 2025

Uh oh!

Reviewers

codecov bot commented Aug 19, 2025 •

edited

Loading

PeterChen13579 commented Aug 27, 2025 •

edited

Loading