MIssing Blocks and Transactions

[bh2smith]

There are three gaps in our blocks and transactions table identified with:

WITH all_numbers AS (
    SELECT generate_series(MIN(18562218), MAX(19316649)) AS num
    FROM blocks
),

missing_blocks as (SELECT an.num
                   FROM all_numbers an
                            LEFT JOIN blocks yt ON an.num = yt.number
                   WHERE yt.number IS NULL)

-- select count(*) from transactions where block_number in (select num from missing_blocks);

-- select * from erc721_transfer where block_number in (select num from missing_blocks);

The results are in three isolated chunks:

1. (200 blocks) from 18562220 to 18562419
2. (200 blocks) from 18562520 to 18562719
3. (50 blocks) from 18844270 to 18844319

The logs for this are too old to recover.

I propose clear the DB from 18562220 and let it repopulate. This would give us the opportunity to see whats went wrong and shouldn't take too long to recover from (i.e. November 13 2023)

[bh2smith]

Temporarily resolved by manual truncation (for now). It is suspected that the issue happens when indexing non-finalized blocks.

[bh2smith]

This is still happening and it seems this block of code:

            let block_queries: Vec<(_, _)> = (earliest..to + 1)
                .map(|block: u64| {
                    (
                        eth::GetBlockByNumber,
                        (BlockSpec::Number(U256::from(block)), Hydrated::Yes),
                    )
                })
                .collect();
            let block_tx_data = self.eth.batch(block_queries).await?;

Is sometimes returning a vector of None.

I picked a block number for which we got None responses and got a non-trivial result:

curl -X POST $NODE_URL --data '{"jsonrpc":"2.0","method":"eth_getBlockByNumber","params":["0x11b3f66", true],"id":1}' | jq