Just wanted to put this out there before I hit up the mailing list.
I've run into this problem a bunch of times. The testnet4 node won't boot up because in the gap between shutting down and starting back up, the chain gets reorged, leading to the Error: Cannot read last block filter index. The only fix I’ve got is to reindex using bitcoind reindex. Haven't found any other workaround for this.
I can’t be the only one testing bitcoin who’s seen this. This just adds to the case for ditching that 20-min difficulty rule.
Does this happen every time, or only once every few restarts? I just stopped and started mine (running Bitcoin Core 28.0) without problems. See tmp.loyce.club/testnet4_debug.log for my last 10,000 debug.log entries.
You need to start it after a long period of inactivity, so that CPU miner Bob has reorged another CPU miner Charlie (the chain of which you were following) and an ASIC miner mined on top of Bob instead of Charlie.
Try stopping it, and starting it tomorrow at the current time. It has to be definitely this, because the error clearly says that another block header was expected, which signifies the chain was reorged, and now my chain, and therefore my index is questioned.
BHC might be using compact block filters with -blockfilterindex=<type> since there's a ".bitcoin/testnet4/indexes/blockfilter/basic/db" indicated in his log.
I haven't experience this either and like yours, I don't use compact block filters on my testnet4 node.
I've applied -blockfilterindex=1 to my testnet4 node, closed it and I'll report tomorrow if it'll result with the same error.
My testnet4 node started this morning without problems. See logs.
Did mempool.space get an upgrade? They now show stale blocks (and that's much more visible for testnet4 than anything else):
I do, but I feel like I've seen this error before introducing block filters.
This is normal. You just received the chain which got accepted and not reorged. Try doing this a couple of times.
Yep. And it even misses some in testnet4. Fork-observer is the most reliable of all, and it misses some branches as well!
I entered this on my console:
Feel free to keep an eye on my logs the coming week If I forget it, I hope it keeps running
Update a day later: weird, all my waiting "sleep" commands were gone. Let's try again, I don't know what killed it.
Some Blockchain software faces challenges removing stale blocks. I think it mostly got better with bitcoin core specifically but never fully improved so it may be worth renaming the block with the highest height before it starts a rescan/reindex (don't run it with that flag, it should do what it needs to do).
I've added it anyways in case it's related to the issue since it's mentioned in the other error line.
Anything else in your bitcoin.conf file that may help with reproducing the errors?
BTW, mine opened normally as well after approximately 24hours.
Current tip:
Hey Loyce, since your server is still running without any error messages, could you add blockfilterindex=1 in your bitcoin.conf and restart it? It is weird how you've never got the same error as I did. I've had the same issue countless times on my virtual machines since then, but I always had blockfilterindex=1.
I think the problem is that when the node restarts, the chainstate gets reconciled with the new best chain, but the block filter index still holds a reference to a block header that's no longer on the active chain, but I am absolutely not sure.
Edit: you might need to restart with bitcoind --reindex
That may be potential culprit.
VM may be suspended commonly happens if you click X before full shutdown at the moment when bitcoind was in the middle of writing data to disk which may result in inconsistent index state.
Try run node on host machine.
Good, but what about snapshots? They can cause the same issue regardless of whether theyre triggered manually or automatically, so if I were running node on VM, Id avoid using them altogether. Anyway to identify the potential culprit Id try running the node on the host machine first and then move it back to the VM to compare its behavior.
Before I test this: have you had this problem again in recent days? Fork Observer shows the Crazy Fork Days on Testnet4 are over. So I expect I'd need many attempts before I manage to shut it down right before a Fork.
Of course. Because blocks are propagated in the first-seen model: if you hear block A, and later you receive block B, at the same height, then it is accepted, but not propagated further.
Also note, that anyone can make a CPU stale block, at any point in time. And then, by default, the client will get only your block header of the stale block, and not even ask about its content. And of course, it won't pass it to the rest of the network.
Which means, that even if you will have perfect communication with a lot of nodes, you will still miss some stale blocks, just because the default rules are designed to not share them, unless necessary, or explicitly asked for.
By the way: you can start a few regtest nodes, and start producing stale blocks. Everything on 127.0.0.1, no communication problems. And then, you will see, how stale blocks are not reaching everyone, if you don't have N(N-1)/2 total connections for N nodes.
Edit: Try to reproduce that in regtest. If you succeed, then you can submit it as an issue into Bitcoin Core.
If not, then it may be related to using an unofficial version: if you run the official client, and your modified version, by using the same data directory, then changes from one version may be incompatible with the other, and it may lead to some crashes, or weird bugs.
Update: So I've been doing this a couple of times and it never experienced corruption even with --blockfilterindex on.
IDK if it's just a bad (good) timing or due to my testnet4 node constantly connecting to peers that follow the same chain.
I'm thinking that "addnode" could play a big role on successful reproduction of this in testnet4.
Anyways, the linked issue above already has an open PR that could fix this.