[fix][broker] Avoid split non-existent bundle #25031

coderzc · 2025-12-02T14:10:13Z

Motivation

2025-10-29T12:55:33,009+0000 [pulsar-modular-load-manager-37-1] INFO  org.apache.pulsar.broker.loadbalance.impl.ModularLoadManagerImpl - Successfully split namespace bundle cc/billing/0x80000000_0xc0000000
2025-10-29T12:55:33,009+0000 [pulsar-ordered-OrderedExecutor-0-0] INFO  org.apache.pulsar.broker.namespace.OwnedBundle - Disabling ownership: xx/billing/0x80000000_0x93c58e36
2025-10-29T12:55:33,009+0000 [pulsar-ordered-OrderedExecutor-0-0] INFO  org.eclipse.jetty.server.RequestLog - 10.247.8.118 - - [29/Oct/2025:12:55:32 +0000] ""PUT /admin/v2/namespaces/xx/billing/0x80000000_0xc0000000/split?unload=true HTTP/1.1"" 204 0 ""-"" ""Pulsar-Java-v4.0.4.1"" 20
2025-10-29T12:55:33,009+0000 [pulsar-ordered-OrderedExecutor-0-0] INFO  org.apache.pulsar.broker.admin.v2.Namespaces - [admin] Successfully split namespace bundle 0x80000000_0xc0000000


2025-10-29T12:55:33,008+0000 [pulsar-2-1] INFO  org.apache.pulsar.broker.service.BrokerService - [xx/billing] updating with Policies(auth_policies=AuthPoliciesImpl(namespaceAuthentication={ccuser=[produce, consume]}, topicAuthentication={persistent://xx/billing/sim-aggr-data={cdr=[produce]}}, subscriptionAuthentication={}), replication_clusters=[pulsar-sn-platform], allowed_clusters=[], bundles=BundlesDataImpl(boundaries=[0x00000000, 0x40000000, 0x80000000, 0xc0000000, 0xcd19337b, 0xd90f1766, 0xddc47076, 0xec49f116, 0xffffffff], numBundles=8), backlog_quota_map={}, clusterDispatchRate={}, topicDispatchRate={}, subscriptionDispatchRate={}, replicatorDispatchRate={}, clusterSubscribeRate={}, persistence=null, deduplicationEnabled=null, autoTopicCreationOverride=null, autoSubscriptionCreationOverride=null, publishMaxMessageRate={}, latency_stats_sample_rate={}, message_ttl_in_seconds=null, subscription_expiration_time_minutes=null, retention_policies=RetentionPolicies{retentionTimeInMinutes=0, retentionSizeInMB=0}, deleted=false, encryption_required=false, delayed_delivery_policies=null, inactive_topic_policies=null, subscription_auth_mode=None, max_producers_per_topic=null, max_consumers_per_topic=null, max_consumers_per_subscription=null, max_unacked_messages_per_consumer=null, max_unacked_messages_per_subscription=null, max_subscriptions_per_topic=null, compaction_threshold=null, offload_threshold=107374182400, offload_threshold_in_seconds=432000, offload_deletion_lag_ms=null, max_topics_per_namespace=null, schema_auto_update_compatibility_strategy=AlwaysCompatible, schema_compatibility_strategy=UNDEFINED, is_allow_auto_update_schema=true, schema_validation_enforced=false, offload_policies=OffloadPoliciesImpl(offloadersDirectory=./offloaders, managedLedgerOffloadDriver=null, managedLedgerOffloadMaxThreads=2, managedLedgerOffloadPrefetchRounds=1, managedLedgerOffloadThresholdInSeconds=432000, managedLedgerOffloadThresholdInBytes=107374182400, managedLedgerOffloadDeletionLagInMillis=null, managedLedgerOffloadedReadPriority=tiered-storage-first, managedLedgerExtraConfigurations={}, s3ManagedLedgerOffloadRegion=null, s3ManagedLedgerOffloadBucket=null, s3ManagedLedgerOffloadServiceEndpoint=null, s3ManagedLedgerOffloadMaxBlockSizeInBytes=67108864, s3ManagedLedgerOffloadReadBufferSizeInBytes=1048576, s3ManagedLedgerOffloadCredentialId=null, s3ManagedLedgerOffloadCredentialSecret=******** s3ManagedLedgerOffloadRole=null, s3ManagedLedgerOffloadRoleSessionName=pulsar-s3-offload, gcsManagedLedgerOffloadRegion=null, gcsManagedLedgerOffloadBucket=null, gcsManagedLedgerOffloadMaxBlockSizeInBytes=134217728, gcsManagedLedgerOffloadReadBufferSizeInBytes=1048576, gcsManagedLedgerOffloadServiceAccountKeyFile=null, fileSystemProfilePath=null, fileSystemURI=null, managedLedgerOffloadBucket=null, managedLedgerOffloadRegion=null, managedLedgerOffloadServiceEndpoint=null, managedLedgerOffloadMaxBlockSizeInBytes=null, managedLedgerOffloadReadBufferSizeInBytes=null), deduplicationSnapshotIntervalSeconds=null, subscription_types_enabled=[], properties={}, resource_group_name=null, migrated=false, dispatcherPauseOnAckStatePersistentEnabled=null, entryFilters=null)
2025-10-29T12:55:33,008+0000 [configuration-metadata-store-13-1] INFO  org.apache.pulsar.broker.resourcegroup.ResourceGroupNamespaceConfigListener - Metadata store notification: Path /admin/policies/xx/billing, Type Modified
2025-10-29T12:55:33,008+0000 [pulsar-2-4] INFO  org.apache.pulsar.broker.service.BrokerService - [xx/billing] updating with LocalPolicies(bundles=BundlesDataImpl(boundaries=[0x00000000, 0x40000000, 0x80000000, 0x93c58e36, 0xc0000000, 0xcd19337b, 0xd90f1766, 0xddc47076, 0xec49f116, 0xffffffff], numBundles=9), bookieAffinityGroup=null, namespaceAntiAffinityGroup=null, migrated=false)
2025-10-29T12:55:33,008+0000 [pulsar-2-3] INFO  org.apache.pulsar.broker.service.BrokerService - [xx/billing] updating with LocalPolicies(bundles=BundlesDataImpl(boundaries=[0x00000000, 0x40000000, 0x80000000, 0x93c58e36, 0xc0000000, 0xcd19337b, 0xd90f1766, 0xddc47076, 0xec49f116, 0xffffffff], numBundles=9), bookieAffinityGroup=null, namespaceAntiAffinityGroup=null, migrated=false)

...

2025-10-29T12:55:50,341+0000 [pulsar-web-46-6] INFO  org.apache.pulsar.broker.admin.impl.NamespacesBase - [admin] Split namespace bundle xx/billing/0x80000000_0xc0000000
2025-10-29T12:55:50,339+0000 [pulsar-modular-load-manager-37-1] INFO  org.apache.pulsar.broker.loadbalance.impl.ModularLoadManagerImpl - Load-manager splitting bundle xx/billing/0x80000000_0xc0000000 and unloading true
2025-10-29T12:55:50,334+0000 [pulsar-modular-load-manager-37-1] INFO  org.apache.pulsar.metadata.impl.AbstractMetadataStore - Deleting path: /loadbalance/bundle-data/xx/billing/0x80000000_0xc0000000 (v. Optional.empty)
...
2025-10-29T12:56:32,969+0000 [pulsar-load-manager-1-1] INFO  org.apache.pulsar.broker.loadbalance.impl.ModularLoadManagerImpl - Writing local data to metadata store because maximum change 1788.8082864051726% exceeded threshold 10%; time since last report written is 59.999 seconds
Caused by: javax.ws.rs.ClientErrorException: HTTP 412 {""reason"":""Invalid upper boundary for bundle xx/billing/0x80000000_0xc0000000. Expected upper boundary of xx/billing/0x80000000_0x93c58e36""}
...
org.apache.pulsar.client.admin.PulsarAdminException$PreconditionFailedException: Invalid upper boundary for bundle xx/billing/0x93c58e36_0xc0000000. Expected upper boundary of xx/billing/0x93c58e36_0xab7838eb
	at org.apache.pulsar.client.admin.PulsarAdminException.wrap(PulsarAdminException.java:252) ~[io.streamnative-pulsar-client-admin-api-4.0.4.1.jar:4.0.4.1]
	at org.apache.pulsar.client.admin.internal.BaseResource.sync(BaseResource.java:352) ~[io.streamnative-pulsar-client-admin-original-4.0.4.1.jar:4.0.4.1]
	at org.apache.pulsar.client.admin.internal.NamespacesImpl.splitNamespaceBundle(NamespacesImpl.java:877) ~[io.streamnative-pulsar-client-admin-original-4.0.4.1.jar:4.0.4.1]
	at org.apache.pulsar.client.admin.internal.NamespacesImpl.splitNamespaceBundle(NamespacesImpl.java:864) ~[io.streamnative-pulsar-client-admin-original-4.0.4.1.jar:4.0.4.1]
	at org.apache.pulsar.broker.loadbalance.impl.ModularLoadManagerImpl.checkNamespaceBundleSplit(ModularLoadManagerImpl.java:776) ~[io.streamnative-pulsar-broker-4.0.4.1.jar:4.0.4.1]
	at org.apache.pulsar.broker.loadbalance.impl.ModularLoadManagerImpl.updateAll(ModularLoadManagerImpl.java:476) ~[io.streamnative-pulsar-broker-4.0.4.1.jar:4.0.4.1]
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ~[?:?]
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[?:?]
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[io.netty-netty-common-4.1.119.Final.jar:4.1.119.Final]
	at java.base/java.lang.Thread.run(Unknown Source) [?:?]

As you can see from the log, after 0x80000000_0xc0000000 was successfully split, it was split again for the second time calling the updateAll() method and failed to split old bundle since bundle been deleted.

Modifications

Skip split the bundles that do not exist in NamespaceBundles

Verifying this change

Make sure that the change passes the CI checks.

(Please pick either of the following options)

This change is a trivial rework / code cleanup without any test coverage.

(or)

This change is already covered by existing tests, such as (please describe tests).

(or)

This change added tests and can be verified as follows:

(example:)

Added integration tests for end-to-end deployment with large payloads (10MB)
Extended integration test for recovery after broker failure

Does this pull request potentially affect one of the following parts:

If the box was checked, please highlight the changes

Documentation

doc
doc-required
doc-not-needed
doc-complete

Matching PR in forked repository

PR in forked repository:

github-actions · 2025-12-02T14:11:02Z

@coderzc Please add the following content to your PR description and select a checkbox:

- [ ] `doc` <!-- Your PR contains doc changes -->
- [ ] `doc-required` <!-- Your PR changes impact docs and you will update later -->
- [ ] `doc-not-needed` <!-- Your PR changes do not impact docs -->
- [ ] `doc-complete` <!-- Docs have been already added -->

lhotari · 2025-12-04T06:55:56Z

...r-broker/src/main/java/org/apache/pulsar/broker/loadbalance/impl/ModularLoadManagerImpl.java

                }
            }

+            writeBrokerDataOnZooKeeper(true);


This wouldn't prevent race conditions. It seems that the method body of org.apache.pulsar.broker.loadbalance.impl.ModularLoadManagerImpl#updateAll method should be wrapped with synchronized (bundleSplitStrategy) { to prevent race conditions.

Manually updating BrokerData will lead to severe lock contention. I checked again whether the bundle exists before the split the bundle.

What is the reason for the severe lock contention? How could it be "severe"?

…data store

codecov-commenter · 2025-12-09T14:14:02Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 74.29%. Comparing base (a8b41b9) to head (eeaaf73).
⚠️ Report is 11 commits behind head on master.

Additional details and impacted files

@@             Coverage Diff              @@
##             master   #25031      +/-   ##
============================================
- Coverage     74.35%   74.29%   -0.07%     
+ Complexity    34102    34083      -19     
============================================
  Files          1920     1921       +1     
  Lines        150313   150382      +69     
  Branches      17459    17470      +11     
============================================
- Hits         111771   111726      -45     
- Misses        29635    29723      +88     
- Partials       8907     8933      +26

Flag	Coverage Δ
inttests	`26.32% <0.00%> (+0.13%)`	⬆️
systests	`22.87% <0.00%> (+0.02%)`	⬆️
unittests	`73.82% <100.00%> (-0.08%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
...roker/loadbalance/impl/ModularLoadManagerImpl.java	`83.41% <100.00%> (-1.12%)`	⬇️

... and 87 files with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[fix][load] Avoid get old loadData during split bundle

c08ff26

coderzc changed the title ~~[fix][load] Avoid get old loadData during split bundle~~ [fix][broker] Avoid get old loadData during split bundle Dec 2, 2025

github-actions bot added the doc-label-missing label Dec 2, 2025

github-actions bot added doc Your PR contains doc changes, no matter whether the changes are in markdown or code files. and removed doc-label-missing labels Dec 2, 2025

coderzc added the ready-to-test label Dec 2, 2025

coderzc closed this Dec 2, 2025

coderzc reopened this Dec 2, 2025

lhotari reviewed Dec 4, 2025

View reviewed changes

github-actions bot added doc-not-needed Your PR changes do not impact docs and removed doc Your PR contains doc changes, no matter whether the changes are in markdown or code files. labels Dec 5, 2025

coderzc force-pushed the fix_split_old_bundle branch 3 times, most recently from e2b577f to 2da2096 Compare December 9, 2025 11:30

Only skip split the bundles that do not exist in Metadata

d582d6b

coderzc force-pushed the fix_split_old_bundle branch from 2da2096 to d582d6b Compare December 9, 2025 11:35

Update bundle existence check to use NamespaceBundles instead of meta…

eeaaf73

…data store

coderzc changed the title ~~[fix][broker] Avoid get old loadData during split bundle~~ [fix][broker] Avoid split non-existent bundle Dec 10, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[fix][broker] Avoid split non-existent bundle #25031

[fix][broker] Avoid split non-existent bundle #25031

coderzc commented Dec 2, 2025 •

edited

Loading

Uh oh!

github-actions bot commented Dec 2, 2025

Uh oh!

lhotari Dec 4, 2025

Uh oh!

coderzc Dec 10, 2025

Uh oh!

lhotari Dec 10, 2025

Uh oh!

codecov-commenter commented Dec 9, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

[fix][broker] Avoid split non-existent bundle #25031

Are you sure you want to change the base?

[fix][broker] Avoid split non-existent bundle #25031

Conversation

coderzc commented Dec 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Modifications

Verifying this change

Does this pull request potentially affect one of the following parts:

Documentation

Matching PR in forked repository

Uh oh!

github-actions bot commented Dec 2, 2025

Uh oh!

lhotari Dec 4, 2025

Choose a reason for hiding this comment

Uh oh!

coderzc Dec 10, 2025

Choose a reason for hiding this comment

Uh oh!

lhotari Dec 10, 2025

Choose a reason for hiding this comment

Uh oh!

codecov-commenter commented Dec 9, 2025

Codecov Report

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

coderzc commented Dec 2, 2025 •

edited

Loading