feat: Populate state proofs with real Merkle paths #22253

mhess-swl · 2025-11-20T00:21:44Z

This PR populates the merkle paths inside of a block's StateProof, which must be generated any time a block is produced without an immediate signature.

There are three merkle paths to generate for each block:

Merkle Path 1: a leaf containing the block's starting timestamp
Merkle Path 2: a path containing the sibling hashes for the rest of the block's contents
Merkle Path 3: the parent path of 1 and 2

A full state proof of an unsigned block must also contain the merkle paths of each succeeding block that precedes the next signed block. That is, if a sequence of blocks A to N are unsigned, and block N+1 is signed, then a full proof for any single block X between [A, N] includes the merkle paths for X, and additionally must contain all the merkle paths for blocks X+1, X+2, ... X+N. The point is that the additional paths are necessary in order for any single state proof to provide all the hashes required to produce the hash of the signed block.

The state proof design requires the merkle paths to follow a depth-first ordering. Consequently, this code produces each set of paths using a pseudo pre-order traversal, resulting in the following array structure:

Index	Path
0	Merkle path 1 of N
1	Merkle path 1 of N-1
...	...
N-1	Merkle path 1 of A
N	Merkle path 2 of A
N+1	Merkle path 3 of A
N+2	Merkle path 2 of B
N+3	Merkle path 3 of B
...	...

The first N-1 elements contain the paths representing the ordered timestamps of each block. The remainder of the array contains pairings of paths 2 and 3, with the final merkle path–representing the block's root hash–at the end of the array.

Testing for this feature has been done in three parts:

A test of a known block sequence from 2 to 7, where only block 7 is signed;
A modified repeatable test that verifies indirect proofs are created;
Additions to the StateChangesValidator class to verify indirect proofs when encountered in the block stream

Due to the relative (perceived?) rarity of blocks without a signature in our hapi test suites, it may be prudent to write a mechanism that forces the production of more unsigned blocks.

Closes #21209

EDIT: Design Changes

After discussion of which Merkle paths are necessary, we settled on three total paths for each state proof:

Merkle path 1 only contains the timestamp of the signed block
Merkle path 3 is still defined as the parent of the timestamp leaf and the remainder of the path
Merkle path 2, instead of having multiple instances inside a single proof, will be expanded to include the entire path from block X all the way to block N.

The production code and test examples have been updated accordingly.

Signed-off-by: Matt Hess <[email protected]>

lfdt-bot · 2025-11-20T00:22:01Z

✅ Snyk checks have passed. No issues have been found so far.

Status	Scanner	Critical	High	Medium	Low	Total (0)
✅	Open Source Security	0	0	0	0	0 issues

💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse.

Signed-off-by: Matt Hess <[email protected]>

codecov · 2025-11-20T04:57:28Z

Codecov Report

❌ Patch coverage is 57.85124% with 51 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
...a/node/app/blocks/impl/BlockStreamManagerImpl.java	37.03%	30 Missing and 4 partials ⚠️
.../com/hedera/node/app/blocks/impl/PendingBlock.java	12.50%	7 Missing ⚠️
...app/blocks/impl/streaming/FileBlockItemWriter.java	0.00%	7 Missing ⚠️
...node/app/blocks/impl/BlockStateProofGenerator.java	94.23%	1 Missing and 2 partials ⚠️

@@            Coverage Diff            @@
##               main   #22253   +/-   ##
=========================================
  Coverage     74.67%   74.67%           
- Complexity    23559    23573   +14     
=========================================
  Files          2520     2522    +2     
  Lines         95687    95768   +81     
  Branches      10173    10180    +7     
=========================================
+ Hits          71451    71518   +67     
- Misses        20445    20453    +8     
- Partials       3791     3797    +6

Files with missing lines	Coverage Δ	Complexity Δ
...com/hedera/node/app/blocks/BlockStreamManager.java	`100.00% <ø> (ø)`	`1.00 <0.00> (ø)`
...com/hedera/node/config/data/BlockStreamConfig.java	`100.00% <ø> (ø)`	`3.00 <0.00> (ø)`
...node/app/blocks/impl/BlockStateProofGenerator.java	`94.23% <94.23%> (ø)`	`8.00 <8.00> (?)`
.../com/hedera/node/app/blocks/impl/PendingBlock.java	`12.50% <12.50%> (ø)`	`1.00 <1.00> (?)`
...app/blocks/impl/streaming/FileBlockItemWriter.java	`40.42% <0.00%> (-1.57%)`	`21.00 <0.00> (ø)`
...a/node/app/blocks/impl/BlockStreamManagerImpl.java	`72.89% <37.03%> (-2.81%)`	`67.00 <8.00> (-2.00)`

... and 19 files with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

codacy-production · 2025-11-20T04:57:33Z

Coverage summary from Codacy

See diff coverage on Codacy

Coverage variation	Diff coverage
✅ +0.01% (target: -1.00%)	✅ 62.81%

Coverage variation details

	Coverable lines	Covered lines	Coverage
Common ancestor commit (`976bbea`)	95590	75194	78.66%
Head commit (`d3aacf8`)	95671 (+81)	75267 (+73)	78.67% (+0.01%)

Coverage variation is the difference between the coverage for the head and common ancestor commits of the pull request branch: <coverage of head commit> - <coverage of common ancestor commit>

Diff coverage details

	Coverable lines	Covered lines	Diff coverage
Pull request (#22253)	121	76	62.81%

Diff coverage is the percentage of lines that are covered by tests out of the coverable lines that the pull request added or modified: <covered lines added or modified>/<coverable lines added or modified> * 100%

See your quality gate settings Change summary preferences

Signed-off-by: Matt Hess <[email protected]>

timo0

pending_proof.proto LGTM, thanks @mhess-swl

Signed-off-by: Matt Hess <[email protected]>

Neeharika-Sompalli · 2025-11-26T02:47:03Z

.../hedera-app/src/main/java/com/hedera/node/app/blocks/impl/streaming/FileBlockItemWriter.java

+                    || Objects.equals(pendingProof.blockTimestamp(), Timestamp.DEFAULT)
+                    || pendingProof.siblingHashesFromPrevBlockRoot().size() != 4) {
+                logger.warn(
+                        "Pending proof metadata from {} is missing required fields (not considering remaining - {})",


Why do we need to log warning here ?

Logged this since the code doesn't try to reload any more blocks. And also because it logs a similar message just above.

Neeharika-Sompalli · 2025-11-26T04:25:52Z

...a/com/hedera/services/bdd/junit/support/validators/block/IndirectProofSequenceValidator.java

+                    registered,
+                    ("Unsigned block should be registered for every block between [%s, %s] –– block"
+                                    + " %s not registered")
+                            .formatted(firstUnsignedBlockNum, signedBlockNum - 1, i));


Have we validated these proofs work with the Verifiers given in Notion page ?

ArrayStateProofVerifier or Option1DStateProofVerifier

I talked with @edward-swirldslabs and @jsync-swirlds before the break–the verifiers actually need to be fixed. At the very least they need to take into account the single child prefix.

Also, the Option1DStateProofVerifier incorrectly expects the timestamp leaf to point to the siblings node instead of the parent node. I'll need to coordinate with @edward-swirldslabs to get the right algorithm.

Neeharika-Sompalli · 2025-11-26T04:33:35Z

...ra-node/hedera-app/src/main/java/com/hedera/node/app/blocks/impl/BlockStreamManagerImpl.java

                attempt.signatureFuture().thenAcceptAsync(signature -> {
-                    finishProofWithSignature(
-                            finalBlockRootHash, signature, attempt.verificationKey(), attempt.chainOfTrustProof());
+                    if (signature == null || Objects.equals(signature, Bytes.EMPTY)) {


Is it possible to have Bytes.EMPTY signature?

I hope not. I did see one instance in local testing where the sig appeared to be empty, but wasn't able to confirm.

Neeharika-Sompalli · 2025-11-26T04:34:58Z

...ra-node/hedera-app/src/main/java/com/hedera/node/app/blocks/impl/BlockStreamManagerImpl.java

+                // Block signatures on the current block will always produce a TssSignedBlockProof
+                proof = currentPendingBlock.proofBuilder().signedBlockProof(latestSignedBlockProof);
+                // Explicitly set empty per the protobuf spec
+                proof.siblingHashes(Collections.emptyList());


As discussed these wont be used anymore right?

I believe that's correct, yes. Will confirm today

Removed in the latest commit

Neeharika-Sompalli · 2025-11-26T04:35:43Z

...ra-node/hedera-app/src/main/java/com/hedera/node/app/blocks/impl/BlockStreamManagerImpl.java

+                proof = currentPendingBlock.proofBuilder().blockStateProof(stateProof);
+
+                // The first mp2 instance from the state proof has this block's sibling hashes
+                final var firstMp2Index = stateProof.paths().size() / 3;


What is 3 here. Defining a constant with a descriptive name will be helpful.

I've defined a number of constants in the new code for any magic numbers :)

Neeharika-Sompalli · 2025-11-26T04:42:08Z

.../main/java/com/hedera/services/bdd/junit/support/validators/block/StateChangesValidator.java

+
+            // We can't verify the indirect proof until we have a signed block proof, so store the indirect proof for
+            // later verification and short-circuit the hints, history proofs (which require a signature)
+            indirectProofSeq.registerStateProof(


Does this actually run in CI or locally, because there are several indirect proofs? May be just Crypto check generates indirect proofs because TSS is enabled?

Yes, it often runs on hapiTestRestart. See example failure here.

Signed-off-by: Matt Hess <[email protected]>

timo0

pending_proof.proto LGTM :-)

Signed-off-by: Matt Hess <[email protected]>

feat: Populate state proofs with real Merkle paths

73641ca

Signed-off-by: Matt Hess <[email protected]>

mhess-swl added this to the v0.69 milestone Nov 20, 2025

mhess-swl self-assigned this Nov 20, 2025

mhess-swl added 2 commits November 19, 2025 17:37

Visibility

080f5fa

Signed-off-by: Matt Hess <[email protected]>

Fix javadoc

a7ba4ea

Signed-off-by: Matt Hess <[email protected]>

mhess-swl added 4 commits November 19, 2025 22:09

Tweaks

1c35d5a

Signed-off-by: Matt Hess <[email protected]>

Fixes

7bcfd87

Signed-off-by: Matt Hess <[email protected]>

Fixes

5ed7f4e

Signed-off-by: Matt Hess <[email protected]>

Fixes

d87880b

Signed-off-by: Matt Hess <[email protected]>

mhess-swl marked this pull request as ready for review November 20, 2025 15:35

mhess-swl requested review from a team as code owners November 20, 2025 15:35

mhess-swl requested a review from timo0 November 20, 2025 15:35

timo0 previously approved these changes Nov 20, 2025

View reviewed changes

edward-swirldslabs self-requested a review November 20, 2025 19:36

edward-swirldslabs previously approved these changes Nov 21, 2025

View reviewed changes

Address comments

d0e40c4

Signed-off-by: Matt Hess <[email protected]>

mhess-swl dismissed stale reviews from edward-swirldslabs and timo0 via d0e40c4 November 21, 2025 16:50

mhess-swl added 7 commits November 21, 2025 09:56

Handle missing timestamp case

deb5091

Signed-off-by: Matt Hess <[email protected]>

Handle missing timestamp case

242a8c1

Signed-off-by: Matt Hess <[email protected]>

Tweaks to validator

6ab47f5

Signed-off-by: Matt Hess <[email protected]>

Fixes

c1e2603

Signed-off-by: Matt Hess <[email protected]>

Fix

50b0a32

Signed-off-by: Matt Hess <[email protected]>

Sanity check to avoid parsing pending blocks without siblings

88c0658

Signed-off-by: Matt Hess <[email protected]>

Minor cleanup

a04cf2f

Signed-off-by: Matt Hess <[email protected]>

Neeharika-Sompalli reviewed Nov 26, 2025

View reviewed changes

mhess-swl added 2 commits December 2, 2025 14:09

State proofs: implement three total merkle paths

b4063e3

Signed-off-by: Matt Hess <[email protected]>

Merge remote-tracking branch 'origin/main' into 21209-block-state-proofs

b30caf3

mhess-swl dismissed timo0’s stale review via b30caf3 December 2, 2025 21:10

mhess-swl added 3 commits December 2, 2025 14:17

Fix compile error

b120995

Signed-off-by: Matt Hess <[email protected]>

Fix indirect seq validator

fb74d77

Signed-off-by: Matt Hess <[email protected]>

Merge remote-tracking branch 'origin/main' into 21209-block-state-proofs

2ca3f55

mhess-swl requested review from Neeharika-Sompalli and derektriley December 3, 2025 08:09

timo0 previously approved these changes Dec 3, 2025

View reviewed changes

Remove deprecated sibling_hashes

277e6c1

Signed-off-by: Matt Hess <[email protected]>

mhess-swl dismissed timo0’s stale review via 277e6c1 December 3, 2025 18:06

mhess-swl added 2 commits December 4, 2025 09:47

Javadoc and static method

c899979

Signed-off-by: Matt Hess <[email protected]>

Merge remote-tracking branch 'origin/main' into 21209-block-state-proofs

82e594c

mhess-swl mentioned this pull request Dec 4, 2025

feat: (Cherry-Pick): Populate state proofs with real Merkle paths #22434

Closed

mhess-swl modified the milestones: v0.69, v0.70 Dec 4, 2025

mhess-swl added 4 commits December 8, 2025 14:50

Merge remote-tracking branch 'origin/main' into 21209-block-state-proofs

b940f8f

Extract constant for number of siblings per block

bc58bd4

Signed-off-by: Matt Hess <[email protected]>

Merge remote-tracking branch 'origin/main' into 21209-block-state-proofs

2b03740

Put real block state proofs behind feature flag

374dfa1

Signed-off-by: Matt Hess <[email protected]>

mhess-swl requested a review from a team as a code owner December 9, 2025 18:49

mhess-swl requested a review from andrewb1269 December 9, 2025 18:49

mhess-swl added 2 commits December 10, 2025 10:52

Merge remote-tracking branch 'origin/main' into 21209-block-state-proofs

e707214

Add missing hash of timestamp

d3aacf8

Signed-off-by: Matt Hess <[email protected]>

feat: Populate state proofs with real Merkle paths #22253

Are you sure you want to change the base?

feat: Populate state proofs with real Merkle paths #22253

Conversation

mhess-swl commented Nov 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

EDIT: Design Changes

Uh oh!

lfdt-bot commented Nov 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Snyk checks have passed. No issues have been found so far.

Uh oh!

codecov bot commented Nov 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

codacy-production bot commented Nov 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Coverage summary from Codacy

Uh oh!

timo0 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mhess-swl Nov 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mhess-swl Nov 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mhess-swl Dec 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Neeharika-Sompalli Nov 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mhess-swl Nov 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

timo0 left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

mhess-swl commented Nov 20, 2025 •

edited

Loading

lfdt-bot commented Nov 20, 2025 •

edited

Loading

codecov bot commented Nov 20, 2025 •

edited

Loading

codacy-production bot commented Nov 20, 2025 •

edited

Loading

mhess-swl Nov 26, 2025 •

edited

Loading

mhess-swl Nov 29, 2025 •

edited

Loading

mhess-swl Dec 3, 2025 •

edited

Loading

Neeharika-Sompalli Nov 26, 2025 •

edited

Loading

mhess-swl Nov 26, 2025 •

edited

Loading