Blockchain Scalpel v2.0.0 is here with XOR-key support and incremental parsing

Hey everyone, I just dropped version 2.0.0 of my lightweight blockchain parser script, now called Blockchain Scalpel. This tool lets you parse raw Bitcoin blk*.dat files straight from your disk, and it's designed for research, analytics, and working with blockchain data at a low level. Here’s what’s new in this version: - Added support for XOR-key handling for blocksdir *.dat files. If there's no key or it’s set to zero, it’ll parse the files directly. - Incremental parsing is now a thing! It checks the output directory for files that have already been processed and only deals with the new block files. - If it finds an incomplete .dat file (usually the last one), the script will throw an error and stop to prevent any messed up output. - Updated documentation and release notes. Check out the project repository if you want to dive deeper. The script is still lightweight, has no dependencies, and is easy to tweak. Let me know what you think. - Denis

How feasible would it be for someone to use this to power an SPV client instead of using the built-in RPC calls of bitcoind, which are somewhat incomplete and inefficient for such a purpose?

It's doable, but it's not a straight swap for RPC unless you're happy to basically re-build half a node. Reading blk*.dat gives you raw blocks, sure, but an SPV client usually wants a clean header chain, reorg handling, and then either merkle proofs for specific txs or some way to discover relevant txs without scanning the entire chain every time. If your backend is just parsing blocks from disk, you still need indexes (at minimum headers + tx lookup + proof building), plus something equivalent to chainstate awareness unless you want to get weird during reorgs and rescans.

Hmmm.... I see. Let's say I purchase one of those huge HPC servers that ship with large amounts of RAM (up to 2TB in some cases), meaning I could fit the entire blockchain in memory. Theoretically, I could calculate the chainstate (i.e. utxo set + timestamps) from this and also store it in RAM. I guess performance would not be a major issue anymore since there is no disk lookup unless a new block is created. So I could probably store chainstate like this: That way, if block 900010 is somehow invalidated by a re-org I would have to do a costly recalculation of the entire utxo set. I used 10 blocks in this example which should be good enough. How long would Merkle proof building take for a subset of transactions? Is it fast? Or will I experience a noticeable slowdown relative to the rest of my hypothetical backend?