Testnet: unable to sync from genesis

[CharlieC3]

When syncing from genesis on testnet, the node stalls at burn block 2700, and stacks block 255. It prints the following message repeatedly:

{"msg":"Process burn block 2701 reward cycle 3 in /hirosystems/data/krypton/burnchain","level":"INFO","ts":"2025-01-09T18:55:40.201743228Z","thread":"chains-coordinator-0.0.0.0:20443","line":1085,"file":"stackslib/src/chainstate/nakamoto/coordinator/mod.rs","burn_block_hash":"35616f7bc3039a6f5c33caf089031824f530f2ef75a77a04459750ee2382ff5e","is_prior_in_prepare_phase":true,"is_rc_start":true,"in_prepare_phase":false}
{"msg":"No PoX anchor block known yet for cycle 3","level":"INFO","ts":"2025-01-09T18:55:40.320665578Z","thread":"chains-coordinator-0.0.0.0:20443","line":527,"file":"stackslib/src/chainstate/nakamoto/coordinator/mod.rs"}
{"msg":"Missing canonical anchor block","level":"ERRO","ts":"2025-01-09T18:55:40.320710088Z","thread":"chains-coordinator-0.0.0.0:20443","line":752,"file":"stackslib/src/chainstate/nakamoto/coordinator/mod.rs"}

This should be easy to replicate locally, as it just requires running a stacks node on version 3.1.0.0.2.

[CharlieC3]

cc @kantai @hstove

[hstove]

I believe the current theory is that this is fixed by #5675, though I don't know if we've proven that yet

[kantai]

It isn't fixed by #5675 -- it should be fixed by #5655 (this is the behavior I believe its supposed to fix), but it isn't fixed by any of these PRs yet.

[jcnelson]

This wouldn't be fixed by #5655. The first testnet block is treated as invalid for some reason.

[kantai]

This wouldn't be fixed by #5655. The first testnet block is treated as invalid for some reason.

That sounds like there's a config.toml mismatch when you ran it.

This is my toml:

[node]
rpc_bind = "127.0.0.1:20443"
p2p_bind = "127.0.0.1:20444"
bootstrap_node = "029266faff4c8e0ca4f934f34996a96af481df94a89b0c9bd515f3536a95682ddc@seed.testnet.hiro.so:30444"

[burnchain]
mode = "krypton"
peer_host = "bitcoin.regtest.hiro.so"
peer_port = 18444
pox_prepare_length = 100
pox_reward_length = 900

[[ustx_balance]]
address = "ST2QKZ4FKHAH1NQKYKYAYZPY440FEPK7GZ1R5HBP2"
amount = 10000000000000000

[[ustx_balance]]
address = "ST319CF5WV77KYR1H3GT0GZ7B8Q4AQPY42ETP1VPF"
amount = 10000000000000000

[[ustx_balance]]
address = "ST221Z6TDTC5E0BYR2V624Q2ST6R0Q71T78WTAX6H"
amount = 10000000000000000

[[ustx_balance]]
address = "ST2TFVBMRPS5SSNP98DQKQ5JNB2B6NZM91C4K3P7B"
amount = 10000000000000000

[[burnchain.epochs]]
epoch_name = "1.0"
start_height = 0

[[burnchain.epochs]]
epoch_name = "2.0"
start_height = 0

[[burnchain.epochs]]
epoch_name = "2.05"
start_height = 1

[[burnchain.epochs]]
epoch_name = "2.1"
start_height = 2

[[burnchain.epochs]]
epoch_name = "2.2"
start_height = 3

[[burnchain.epochs]]
epoch_name = "2.3"
start_height = 4

[[burnchain.epochs]]
epoch_name = "2.4"
start_height = 5

[[burnchain.epochs]]
epoch_name = "2.5"
start_height = 6

[[burnchain.epochs]]
epoch_name = "3.0"
start_height = 1900

[[burnchain.epochs]]
epoch_name = "3.1"
start_height = 2000

The node syncs just fine until it reaches the third reward cycle. Then it stops because it hasn't processed the reward set yet.

[kantai]

With that config.toml, I sync to stacks block height ~113 within a minute or two.

[jcnelson]

Okay, I think I found the problem. The p2p stack cuts over to Nakamoto-only mode too early, and fails to download the last of the epoch 2.5 blocks. This, in turn, prevents processing of the subsequent Nakamoto blocks.

[wileyj]

Confirmed that the branch fix/5676 appears to resolve this issue - i've built and synced past btc block 900 using this branch. nicely done!