Forward sync columns by root #7946

pawanjay176 · 2025-08-27T02:09:19Z

Issue Addressed

N/A

The Problem

Our current strategy of syncing blocks + columns by range works roughly as follows for each batch:

Find a peer from the current SyncingChain to fetch blocks from and send a BlocksByRange request
Find peer(s) from the global peer pool that should have columns for the same batch based on their status message and the columns they are supposed to custody , and send them a by DataColumnsByRange request at the same time
Once we get responses for all blocks + columns components, try to couple them by checking that the block_root and the kzg_commitment matches. If the coupling failed, try to re-request the failed columns.
Send them for processing and try to progress the chain

This strategy works decently well when the chain is finalizing as most of our peers are on the same chain. However, in times of non-finality we need to potentially sync multiple head chains.
This leads to issues with our current approach because the block peer and the data column peers might have a different view of the canonical chain due to multiple heads. So when we use the above approach, it is possible that the block peer returns us a batch of blocks for chain A while some or all data column peers send us the batch of data columns for a different chain B. Different data column peers might also be following different chains.

We initially tried to get around this problem by selecting column peers only from within the current SyncingChain. Each SyncingChain represents a head_root that we are trying to sync to and we group peers based on same head_root. That way, we know for sure that the block and column peers are on the same chain. This works in theory, but in practice, during long periods of non-finality, we tend to create multiple head chains based on the head_root and split the global peerset. Pre-fulu, this isn't a big deal since all peers are supposed to have all the blob data.
But splitting peers with peerdas is a big challenge due to not all peers having the full data available. There are supernodes, but during bad network conditions, supernodes would be getting way too many requests and not even have any incoming peer slots. As we saw on fusaka devnets, this strategy leads to sync getting stalled and not progressing.

Proposed Changes

1. Use `DataColumnsByRoot` instead of `DataColumnsByRange` to fetch columns for forward sync

This is the main change. The new strategy would go as follows:

Find a peer from the current SyncingChain to fetch blocks from and send a BlocksByRange request
Hold off on requesting for columns until we receive the response for the above blocks request
Once we get the blocks response, extract all the block_roots and trigger a DataColumnsByRoot request for every block in the response that has any blobs based on the expected_kzg_commitments field.
Since we request by root, we know what we are expecting in the response. The column peer's view of the canonical chain might be chain A, but if we ask for chain B and they have chain B in their fork choice, they can still serve us what we need.
We couple the block + column responses and send them for processing as before.

(4) kinda assumes that most synced/advanced peers would have different chains in their fork choice to be able to serve specific by root requests. My hunch is that this is true, but we should validate this in a devnet 4 like chain split scenario.

Note: that we currently use this by root strategy only for forward sync, not for backfill. Backfill has to deal with only a single canonical chain so byrange requests should work well there.

2. ResponsiblePeers to attribute peer fault correctly

Adds the ResponsiblePeers struct which stores the block and the column peers that we made the download requests to.
For most of our peer attributable errors, the processing error indicates whether the block peer was at fault or if the column peer was at fault.

We now communicate this information back to sync and downscore specific peers based on the fault type. This imo, is an improvement over current unstable where most of the time, we attribute fault to the peer that "completed" the request by being the last peer to respond.
Due to this ambiguity in fault attribution, we weren't downscoring pretty serious processing errors like InvalidKzgProofs, InvalidExecutionPayload etc. I think this PR attributes the errors to the right peers. Reviewers please check that this claim is actually true.

3. Make `AwaitingDownload` an allowable in-between state

Note: This has been extracted to its own PR here and merged #7984
Prior to peerdas, a batch should never have been in AwaitingDownload state because we immediataly try to move from AwaitingDownload to Downloading state by sending batches. This was always possible as long as we had peers in the SyncingChain in the pre-peerdas world.

However, this is no longer the case as a batch can be stuck waiting in AwaitingDownload state if we have no peers to request the columns from. This PR makes AwaitingDownload to be an allowable in between state. If a batch is found to be in this state, then we attempt to send the batch instead of erroring like before.
Note to reviewer: We need to make sure that this doesn't lead to a bunch of batches stuck in AwaitingDownload state if the chain can be progressed.

beacon_node/network/src/sync/range_sync/chain.rs

beacon_node/network/src/sync/block_sidecar_coupling.rs

jimmygchen

Hi @pawanjay176
I haven't managed to go through all the logic, but I've added comments for the findings I have so far. Let me know what you think. I'll continue next week.

beacon_node/network/src/sync/network_context.rs

beacon_node/network/src/sync/block_sidecar_coupling.rs

jimmygchen · 2025-09-05T06:31:11Z

beacon_node/network/src/sync/block_sidecar_coupling.rs

+                Some(resp)
+            }
+            // Reuse same logic that we use for coupling data columns for now.
+            // todo(pawan): we should never get a coupling error here, so simplify this


What do you mean here?

I meant that since we are requesting by root and also verifying the inclusion proof when adding a data column, by the time we reach here we should have caught all errors.
If we get a valid response for the data columns (no VerifyError), then there shouldn't be any issues with coupling

beacon_node/network/src/sync/range_sync/batch.rs

beacon_node/network/src/sync/network_context.rs

dapplion

Why not use the same strategy for backfill sync too? Would simplify the code. Do you believe it would make backfill sync much slower?

beacon_node/network/src/sync/manager.rs

dapplion · 2025-09-09T22:01:17Z

beacon_node/network/src/sync/network_context/requests/data_columns_by_root.rs

+    }
+}
+
+impl<E: EthSpec> ActiveRequestItems for DataColumnsByRootRangeRequestItems<E> {


You can de-duplicate this code by making DataColumnsByRootRequestItems take a vec of block roots

I want to keep them different. I understand its more code but i feel its clearer to keep different components separate

beacon_node/network/src/sync/range_sync/batch.rs

beacon_node/network/src/sync/range_sync/chain.rs

beacon_node/network/src/sync/network_context.rs

dapplion · 2025-09-09T22:29:48Z

beacon_node/network/src/sync/network_context.rs

+    ///
+    /// This function must be manually invoked at regular intervals or when a new peer
+    /// gets added.
+    pub fn retry_pending_root_range_requests(&mut self) -> Result<(), String> {


The logic here is good. To sum up, the new functionality added in this PR is:

New by_range request type to:

Request blocks first

Then with those roots request data columns by root

If any of those requests fail or mismatch retry

All this logic can be grouped and moved into its own file (the coupling service) and make network_context.rs less chaotic. I did so in my tree-sync WIP, and subjectively feels nicer https://github.com/dapplion/lighthouse/blob/47c93578c418bdac5c3beb3064ab5f675c3c177d/beacon_node/network/src/sync/network_context/block_components_by_range.rs

Can we do this in a future PR, this one is already big imo

dapplion · 2025-09-09T22:31:24Z

beacon_node/network/src/sync/network_context.rs

+
+        // Re-insert entries that still need to be retried
+        self.pending_column_by_root_range_requests
+            .extend(entries_to_keep);


This retries should have either a retry counter or an expiry time. We should have metrics also to track the count of retries

Good idea re the metrics. Not sure I understand the need for adding a retry counter. If we remove/expire an entry, that might stall sync which was expecting a value.

dapplion · 2025-09-09T22:39:38Z

beacon_node/network/src/sync/block_sidecar_coupling.rs

+        column_requests: Vec<(DataColumnsByRootRequestId, Vec<ColumnIndex>)>,
+    ) -> Result<(), String> {
+        // Nothing to insert, do not initialize
+        if column_requests.is_empty() {


Should this ever happen?

It shouldn't, but it took me a long time to find a bug where i was calling this function with an empty list, so just kept this as a safety condition

beacon_node/network/src/sync/block_sidecar_coupling.rs

dapplion · 2025-09-09T22:42:11Z

beacon_node/network/src/sync/block_sidecar_coupling.rs

+    /// Note: this variant starts out in an uninitialized state because we typically make
+    /// the column requests by root only **after** we have fetched the corresponding blocks.
+    /// We can initialize this variant only after the columns requests have been made.
+    DataColumnsFromRoot {


Do we really need a new variant for it?

I think we do. This allows us to switch between the roots and range approach easily and work with them differently. I would prefer to merge them into one variant if required in a future PR.

N/A Extracts (3) from sigp#7946. Prior to peerdas, a batch should never have been in `AwaitingDownload` state because we immediataly try to move from `AwaitingDownload` to `Downloading` state by sending batches. This was always possible as long as we had peers in the `SyncingChain` in the pre-peerdas world. However, this is no longer the case as a batch can be stuck waiting in `AwaitingDownload` state if we have no peers to request the columns from. This PR makes `AwaitingDownload` to be an allowable in between state. If a batch is found to be in this state, then we attempt to send the batch instead of erroring like before. Note to reviewer: We need to make sure that this doesn't lead to a bunch of batches stuck in `AwaitingDownload` state if the chain can be progressed. Backfill already retries all batches in AwaitingDownload state so we just need to make `AwaitingDownload` a valid state during processing and validation. This PR explicitly adds the same logic for forward sync to download batches stuck in `AwaitingDownload`. Apart from that, we also force download of the `processing_target` when sync stops progressing. This is required in cases where `self.batches` has > `BATCH_BUFFER_SIZE` batches that are waiting to get processed but the `processing_batch` has repeatedly failed at download/processing stage. This leads to sync getting stuck and never recovering.

pawanjay176 · 2025-09-17T00:53:51Z

Why not use the same strategy for backfill sync too?

I don't want to nuke the existing coupled strategy we have got just yet. Also, for backfill, since its one chain, the coupled approach makes more logical sense imo. Want to test this out in the wild sufficiently and then we can use this uncoupled strategy once we are confident it works.

pawanjay176 · 2025-09-17T22:17:38Z

This one should be ready for another round of review. Used claude to fix up the sync tests, lmk if you wanted a different approach. I don't have strong opinions on the test suite.

mergify · 2025-09-22T01:04:53Z

This pull request has merge conflicts. Could you please resolve them @pawanjay176? 🙏

pawanjay176 added 20 commits August 14, 2025 09:07

Penalize if invalid EL block

490b627

Priorotize status v2

836f9c6

Increase columns_by_root quota

156449c

Reduce backfill buffer size

6bd8944

Without retries

9455153

Add a function to retry column requests that could not be made

5337e46

Small fixes

ca9cfd5

Try to avoid chains failing for rpc errors

68cce37

Fix bug in initialization code

6da924b

Also penalize all batch peers for availability check errors

1a0df30

Avoid root requests for backfill sync

17c4e34

Implement responsible peer tracking

fdce537

Request columns from global peer pool

4540195

Random logs

521778b

Merge branch 'unstable' into blocks-then-columns

da27441

Handle 0 blobs per epoch case

52762b9

Merge branch 'unstable' into blocks-then-columns

7c214f5

Merge branch 'unstable' into blocks-then-columns

90d319f

Remove debug statements

27d0b36

Add docs

a97cf88

jimmygchen added syncing v8.0.0-rc.0 Q3 2025 release for Fusaka on Holesky labels Aug 27, 2025

pawanjay176 added 7 commits August 27, 2025 14:26

Fix bug with partial column responses before all column requests sent

05adb71

Remove more debug logs

b4bc7fe

Merge branch 'unstable' into blocks-then-columns

8386bd9

AwaitingValidation state only needs block peer

7331323

Revise error tolerance

da1aaba

Merge branch 'unstable' into blocks-then-columns

8e1337d

Merge branch 'unstable' into blocks-then-columns

19b0a5c

jimmygchen self-requested a review August 29, 2025 08:35

jimmygchen reviewed Sep 5, 2025

View reviewed changes

beacon_node/network/src/sync/range_sync/chain.rs Outdated Show resolved Hide resolved

jimmygchen reviewed Sep 5, 2025

View reviewed changes

beacon_node/network/src/sync/block_sidecar_coupling.rs Outdated Show resolved Hide resolved

jimmygchen reviewed Sep 5, 2025

View reviewed changes

beacon_node/network/src/sync/block_sidecar_coupling.rs Show resolved Hide resolved

jimmygchen reviewed Sep 5, 2025

View reviewed changes

beacon_node/network/src/sync/block_sidecar_coupling.rs Outdated Show resolved Hide resolved

jimmygchen requested changes Sep 5, 2025

View reviewed changes

pawanjay176 added 4 commits September 5, 2025 12:48

Fix some issues from review

6a2a33d

More renamings

e259ecd

Merge branch 'unstable' into blocks-then-columns

4f62a9c

Fix some more issues from review

04398ad

dapplion reviewed Sep 9, 2025

View reviewed changes

jimmygchen mentioned this pull request Sep 10, 2025

Error "Not all columns consumed for block" on fusaka-devnet-2 #7747

Open

pawanjay176 added 3 commits September 11, 2025 15:14

Fix some issues from lion's review

bf09d57

Reduce code duplication

cffbd34

fmt

08bba3f

pawanjay176 added 3 commits September 16, 2025 16:38

Fix small bug

9db4c30

Merge branch 'unstable' into blocks-then-columns

baee27a

Remove retry test that we do not use anymore

e3aed89

pawanjay176 added 3 commits September 17, 2025 15:01

Fix tests

b3b3756

Add some metrics

2f35c36

Merge branch 'unstable' into blocks-then-columns

4a59d35

pawanjay176 added ready-for-review The code is ready for review and removed waiting-on-author The reviewer has suggested changes and awaits thier implementation. labels Sep 17, 2025

mergify bot added waiting-on-author The reviewer has suggested changes and awaits thier implementation. and removed ready-for-review The code is ready for review labels Sep 22, 2025

Forward sync columns by root #7946

Are you sure you want to change the base?

Forward sync columns by root #7946

Conversation

pawanjay176 commented Aug 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Issue Addressed

The Problem

Proposed Changes

1. Use DataColumnsByRoot instead of DataColumnsByRange to fetch columns for forward sync

2. ResponsiblePeers to attribute peer fault correctly

3. Make AwaitingDownload an allowable in-between state

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jimmygchen left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jimmygchen Sep 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dapplion left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pawanjay176 commented Sep 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pawanjay176 commented Sep 17, 2025

Uh oh!

mergify bot commented Sep 22, 2025

Uh oh!

Uh oh!

pawanjay176 commented Aug 27, 2025 •

edited

Loading

1. Use `DataColumnsByRoot` instead of `DataColumnsByRange` to fetch columns for forward sync

3. Make `AwaitingDownload` an allowable in-between state

jimmygchen Sep 5, 2025 •

edited

Loading

pawanjay176 commented Sep 17, 2025 •

edited

Loading