Is my blk00000.dat file corrupted?

3 replies 226 views
Posts: 20 · Reputation: 127
#1Mar 8, 2021, 02:02 PM
So I've had my full node running for a while now without any hiccups. It's been syncing just fine and keeping up with blocks. But I tried to dig through my local data history faster without using RPC, relying on blockchain_parser.blockchain to create a new utxos table for my analysis. The issue? Every time I tried to index the raw data, my CPU went through the roof, almost hitting 100% for hours and making no progress. I think I figured out that it's getting stuck on the very first block. The file at ~/.bitcoin/blocks/blk00000.dat starts with: that's not normal, right? Is there a way to force an update for just that block? I'm curious if anyone has better ideas for processing this data. Also, how did my initial block end up like this? Edit: I did more digging and noticed that all the blocks (at least the first 5) start with those same bytes. I checked the entire folder and every single blk*.dat file kicks off with the magic string 3d112a31. Does this indicate I need to redownload or resync my whole history? I'm puzzled about how this even happened... has anyone dealt with this issue before? I'd really appreciate any tips to help me avoid a lengthy resync down the line. By the way, I'm using Bitcoin Core v28.99.-0 on Ubuntu 24.04.1 LTS.
2 Reply Quote Share
gr3g.0rbitHero Member
Posts: 1025 · Reputation: 2646
#2Mar 8, 2021, 03:33 PM
blockchain_parser.blockchain may not be designed to correctly parse the new XOR'd blocks of Bitcoin v28.0+ Check its repo if there are reported issues related to it. Once you confirm that it's the issue, you may restart with a fresh data directory but make sure that blocksxor=0 is set in your config file or use it as a command line arg.
2 Reply Quote Share
Posts: 20 · Reputation: 127
#3Mar 8, 2021, 05:04 PM
Appreciate that tip (I've added blocksxor=0), I also discovered I wasn't running a stable release of 28, so I'm updating to 29.0 Downloading and syncing again. Should fix it.
4 Reply Quote Share
im_lynxHero Member
Posts: 515 · Reputation: 2161
#4Mar 8, 2021, 06:06 PM
Wouldn't it be faster to "un-XOR" existing blockchain data files if some of your tools can't cope with non-zero XORed block files, instead of repeating an IBD from scratch with blocksxor=0? The file xor.dat in Core's blocks folder stores the used obfuscation 8-byte XOR value. It shouldn't be very difficult to write a tool to do the un-XORing. I haven't tried it myself...
4 Reply Quote Share
?Reply
Sign in to reply to this topic

Related topics