[Storage] Refactor stored chunk data pack #7983

zhangchiqing · 2025-09-30T03:08:41Z

This PR:

Introduces a new StoredChunkDataPacks store and refactors NewChunkDataPacks to depend on it, wiring it through node startup and CLI tools (execution builder, read-badger, rollback cmd). This splits storage of chunk packs from the protocol DB.
Changes the write path: chunkDataPacks.Store(...) now returns a closure that writes the ChunkID→StoredChunkDataPack.ID mapping inside the protocol DB batch; only LockInsertOwnReceipt is held. Improves atomicity and clarifies failure modes.
Updates rollback to batch-remove multiple chunk data packs at once (BatchRemove(chunkIDs, writeBatch, chunkBatch)), simplifying error handling.
Verification requester API becomes more informative: RequestQualifierFunc now returns (bool, string) and MaxAttemptQualifier includes a reason when unqualified.

…stored-chunk-data-pack

codecov-commenter · 2025-10-01T00:26:27Z

Codecov Report

❌ Patch coverage is 43.30544% with 271 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
storage/mock/stored_chunk_data_packs.go	0.00%	74 Missing ⚠️
storage/chunk_data_packs_stored.go	0.00%	47 Missing ⚠️
storage/mock/chunk_data_packs.go	0.00%	41 Missing ⚠️
...ck-executed-height/cmd/rollback_executed_height.go	23.91%	35 Missing ⚠️
storage/store/chunk_data_packs.go	75.49%	19 Missing and 6 partials ⚠️
storage/store/chunk_data_packs_stored.go	75.00%	10 Missing and 5 partials ⚠️
engine/execution/state/state.go	78.78%	5 Missing and 2 partials ⚠️
engine/verification/requester/qualifier.go	53.84%	6 Missing ⚠️
module/pruner/pruners/chunk_data_pack.go	0.00%	6 Missing ⚠️
cmd/execution_builder.go	0.00%	4 Missing ⚠️
... and 5 more

📢 Thoughts on this report? Let us know!

…gestions2_stored-chunk-data-pack

AlexHentschel

Thanks for the iterations. PR looks great: very clear and well documented.

The only aspect that I am worried about being merged to master is the overlapping batch-writes for the chunk data pack removal (see my comment here). Any hotfix would do from my perspective that prevents accidental data corruption.

The remaining comments are largely just very minor suggestions to improve code clarity further.

storage/operation/chunk_data_packs_test.go

AlexHentschel · 2025-10-14T20:28:11Z

storage/operation/prefix.go

+	// Compared to the deprecated `codeChunkDataPack`, which stored chunkID -> storedChunkDataPack relationship:
+	//  - `codeIndexChunkDataPackByChunkID` stores the chunkID->chunkDataPackID index, and
+	//  - `codeChunkDataPack` stores chunkDataPackID -> storedChunkDataPack relationship.
+	// This breakup allows us to store chunk data packs in a different database in a concurrent safe way
+	codeIndexChunkDataPackByChunkID  = 112


[Leo] I decided to keep using the existing codeChunkDataPack prefix for storing the new chunk data pack. That’s fine, since during the rollout I’ll be removing all existing chunk data pack entries from the database.

sounds good.

Some suggestions:

what do you think about using the prefix code 99 for codeIndexChunkDataPackByChunkID? That way, it would be listed right before codeChunkDataPack

I think that would be beneficial for ease of documentation and understanding the code, if codeChunkDataPack and codeIndexChunkDataPackByChunkID as well as their combined documentation would be all together.

to documentation still talks about "deprecated codeChunkDataPack" which no longer applies.

The resulting code could look something like the following (already updated documentation):

// EXECUTION RESULTS: // // The storage prefixes `codeChunkDataPack` and `codeIndexChunkDataPackByChunkID` are used primarily by execution nodes // to persist their own results for chunks they executed. // - `codeIndexChunkDataPackByChunkID` stores the chunkID → chunkDataPackID index, and // - `codeChunkDataPack` stores the chunk data pack by its own ID. // This breakup allows us to store chunk data packs in a different database in a concurrent safe way. codeIndexChunkDataPackByChunkID = 99 codeChunkDataPack = 100 // legacy codes (should be cleaned up) codeCommit = 101 ⋮

AlexHentschel · 2025-10-14T20:32:53Z

storage/operation/chunk_data_packs.go

-	return RetrieveByKey(r, MakePrefix(codeChunkDataPack, chunkID), c)
+// RetrieveStoredChunkDataPack retrieves a chunk data pack by stored chunk data pack ID.
+// It returns [storage.ErrNotFound] if the chunk data pack is not found
+func RetrieveStoredChunkDataPack(r storage.Reader, storeChunkDataPackID flow.Identifier, c *storage.StoredChunkDataPack) error {


for simplicity, maybe we could rename those methods to InsertChunkDataPack and RetrieveChunkDataPack. The fact that we are dealing with the reduced data type StoredChunkDataPack for storage is in my opinion very well reflected by the method signature.

AlexHentschel · 2025-10-14T21:52:17Z

storage/store/chunk_data_packs.go

+	// the actual chunk data pack is stored here, which is a separate storage from protocol DB
+	stored storage.StoredChunkDataPacks


Couple suggestions:

I would prefer a more descriptive name. How about: cdpStorage

I think it would be helpful to document that we assume that cdpStorage has its own caching built in.

Suggested change

// the actual chunk data pack is stored here, which is a separate storage from protocol DB

stored storage.StoredChunkDataPacks

// cdpStorage persists the actual chunk data packs, which is a separate storage from protocol DB.

// We assume that `cdpStorage` has its own caching already built in.

cdpStorage storage.StoredChunkDataPacks

storage/store/chunk_data_packs.go

AlexHentschel · 2025-10-15T03:54:43Z

engine/execution/state/state.go

I really like how this has turned out. I think from the business logic's perspective, it is really quite clear what happens at which state (with the help of some documentation). Well done, tanks for your iterations and patience. 👏

cmd/util/cmd/rollback-executed-height/cmd/rollback_executed_height.go

AlexHentschel · 2025-10-15T04:04:51Z

cmd/util/cmd/rollback-executed-height/cmd/rollback_executed_height.go

 // use badger instances directly instead of stroage interfaces so that the interface don't
 // need to include the Remove methods


this doc could use an update, please.

AlexHentschel · 2025-10-15T04:09:27Z

cmd/util/cmd/rollback-executed-height/cmd/rollback_executed_height.go

+			chunkDataPackIDs, err := chunkDataPacks.BatchRemove(chunkIDs, protocolDBBatch)
+			if err != nil {
+				return fmt.Errorf("could not remove chunk data packs at %v: %w", flagHeight, err)
+			}
+
+			err = storedChunkDataPacks.Remove(chunkDataPackIDs)
+			if err != nil {
+				return fmt.Errorf("could not commit chunk batch at %v: %w", flagHeight, err)
+			}


⚠️ repeated removal (?)

chunkDataPacks.BatchRemove internally also calls storedChunkDataPacks.Remove

cmd/util/cmd/rollback-executed-height/cmd/rollback_executed_height.go

tim-barry

Overall looks good; mostly focused on documentation.
I believe since all usages of it have been removed, we can completely remove storage.LockInsertChunkDataPack.

tim-barry · 2025-10-14T21:57:59Z

model/flow/chunk.go


+type ChunkDataPackHeader struct {


Even if we are only using this type to generate the ID for ChunkDataPack, I think we can still mark this type as structwrite:immutable as well, and add a short comment about its current use.

This has already been covered in Alex's documentation PR.

tim-barry · 2025-10-14T22:48:24Z

storage/chunk_data_packs.go

+	//     to chunk data pack ID in the protocol database. This mapping persists that the Execution Node committed to the result
+	//     represented by this chunk data pack. This function returns [storage.ErrDataMismatch] when a _different_ chunk data pack
+	//     ID for the same chunk ID has already been stored (changing which result an execution Node committed to would be a
+	//     slashable protocol violation). The caller must acquire [storage.LockInsertChunkDataPack] and hold it until the database


As far as I can tell, all usages of storage.LockInsertChunkDataPack were removed - should this be storage.LockInsertOwnReceipt?

tim-barry · 2025-10-15T08:44:33Z

storage/store/chunk_data_packs.go

+//     to chunk data pack ID in the protocol database. This mapping persists that the Execution Node committed to the result
+//     represented by this chunk data pack. This function returns [storage.ErrDataMismatch] when a _different_ chunk data pack
+//     ID for the same chunk ID has already been stored (changing which result an execution Node committed to would be a
+//     slashable protocol violation). The caller must acquire [storage.LockInsertChunkDataPack] and hold it until the database


Again here I believe storage.LockInsertChunkDataPack should instead be storage.LockInsertOwnReceipt

Good catch, I think we should still use that lock, and I renamed it to storage.LockIndexChunkDataPackByID

tim-barry · 2025-10-15T09:12:48Z

storage/store/chunk_data_packs_test.go

+			// Verify chunk data packs are removed from both protocol and chunk data pack DBs
+			for _, chunkID := range chunkIDs {
+				_, err := chunkDataPackStore.ByChunkID(chunkID)


I believe this only tests that they are removed from the protocol DB - the stored chunk data pack may still be present in stored. To verify both "protocol DB mappings and chunk data pack DB content" are removed as documented, we probably want to record the chunkDataPack IDs and directly query the stored DB for them after the removal.

…ata-pack Suggested documentation extensions for Chunk Data Pack PR #7983

Co-authored-by: Alexander Hentschel <[email protected]>

…ight.go Co-authored-by: Alexander Hentschel <[email protected]>

Co-authored-by: Alexander Hentschel <[email protected]>

zhangchiqing added 11 commits September 29, 2025 10:44

add StoreChunkDataPacks

05faf10

update store chunk data pack

a75653e

fix test case

82f14f0

fix for execution state

e690d56

fix test

70f3cc6

fix builder

c998e28

update testutil engine

9ab5e0a

update chunk data packs and tests

6a4b59e

fix executor tests

17821a5

fix lint

848c9e7

update comments

fe584bd

zhangchiqing mentioned this pull request Sep 30, 2025

[Storage] Refactor insert chunk data pack #7939

Merged

zhangchiqing added 12 commits September 30, 2025 11:30

update comments in execution state

cbc5873

refactor BatchRemove

936f38b

add comments for BatchRemove

7ad596e

Merge branch 'leo/refactor-insert-chunk-data-pack' into leo/refactor-…

e875957

…stored-chunk-data-pack

use two databases for chunk data pack tests

044325d

add test cases for BatchRemove

d8e884d

update StoreChunkDataPack.Equals

24744c5

fix pruner tests

9d287d6

update mocks

be6f1ce

fix tests

5e0e6a1

update comments

c37af9c

update tests

7fa1ef6

Base automatically changed from leo/refactor-insert-chunk-data-pack to master October 1, 2025 00:14

Merge branch 'master' into leo/refactor-stored-chunk-data-pack

d22661a

zhangchiqing force-pushed the leo/refactor-stored-chunk-data-pack branch from e3a3b6b to d22661a Compare October 1, 2025 00:22

zhangchiqing added 2 commits October 1, 2025 21:40

Merge branch 'master' into leo/refactor-stored-chunk-data-pack

58cd332

fix integration tests

8dbfdc1

AlexHentschel and others added 4 commits October 14, 2025 17:48

storrage.ChunkDataPacks API update

629da34

skip the chunk data pack roll back if storing failed

52b05be

Merge commit '52b05be05d5439239eb6d4386922f669336b7774' into alex/sug…

5a83763

…gestions2_stored-chunk-data-pack

minor struct goDocs

c59b4de

AlexHentschel approved these changes Oct 15, 2025

View reviewed changes

AlexHentschel mentioned this pull request Oct 15, 2025

Suggested documentation extensions for Chunk Data Pack PR #7983 #8038

Merged

tim-barry approved these changes Oct 15, 2025

View reviewed changes

j1010001 and others added 18 commits October 15, 2025 08:00

Merge branch 'master' into leo/refactor-stored-chunk-data-pack

28a3fae

Merge pull request #8038 from onflow/alex/suggestions2_stored-chunk-d…

7979348

…ata-pack Suggested documentation extensions for Chunk Data Pack PR #7983

update storage/operation/prefix.go and chunk data packs

7332487

add review comments

2c53081

Update storage/operation/chunk_data_packs_test.go

10dbe5b

Co-authored-by: Alexander Hentschel <[email protected]>

Update cmd/util/cmd/rollback-executed-height/cmd/rollback_executed_he…

b4ee3b0

…ight.go Co-authored-by: Alexander Hentschel <[email protected]>

Update storage/store/chunk_data_packs.go

701f9cf

Co-authored-by: Alexander Hentschel <[email protected]>

rename to cdpStorage

1b7b074

use LockIndexChunkDataPackByChunkID

bbbad00

fix chunk data packs tests

1ed6227

fix tests

4555afa

Update storage/store/chunk_data_packs.go

e7348f5

Co-authored-by: Alexander Hentschel <[email protected]>

update chunk data pack remove

59b5ae4

refactor chunk data pack constructor

41d9460

fix locks in tests

3a664ce

Merge branch 'master' into leo/refactor-stored-chunk-data-pack

643c27f

fix lint

f6629e7

Merge branch 'master' into leo/refactor-stored-chunk-data-pack

b651d90

zhangchiqing enabled auto-merge October 15, 2025 17:34

zhangchiqing added this pull request to the merge queue Oct 15, 2025

Merged via the queue into master with commit fc546a5 Oct 15, 2025
57 checks passed

zhangchiqing deleted the leo/refactor-stored-chunk-data-pack branch October 15, 2025 18:27

		// the actual chunk data pack is stored here, which is a separate storage from protocol DB
		stored storage.StoredChunkDataPacks

-	// the actual chunk data pack is stored here, which is a separate storage from protocol DB
-	stored storage.StoredChunkDataPacks
+	// cdpStorage persists the actual chunk data packs, which is a separate storage from protocol DB.
+	// We assume that `cdpStorage` has its own caching already built in.
+	cdpStorage storage.StoredChunkDataPacks

		// use badger instances directly instead of stroage interfaces so that the interface don't
		// need to include the Remove methods

[Storage] Refactor stored chunk data pack #7983

[Storage] Refactor stored chunk data pack #7983

Conversation

zhangchiqing commented Sep 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov-commenter commented Oct 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

AlexHentschel left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

tim-barry left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

zhangchiqing commented Sep 30, 2025 •

edited

Loading

codecov-commenter commented Oct 1, 2025 •

edited

Loading

AlexHentschel left a comment •

edited

Loading