Complete Bitcoin Core Backup Plan

Hey everyone, I've been running a few full nodes, but I’m gonna have to stick to just one for now. I’m considering keeping a full backup of the whole blockchain on an HDD, just in case I need to set up another node later on. This way, I can skip the whole Initial Block Download (IBD) process. Here’s what I’m thinking: 1. Backing up the entire ~/.bitcoin/ folder, including everything in it (blocks, chainstate, wallets, etc). 2. Ensuring the HDD is fully encrypted. 3. Doing regular updates for the backups (not sure how often that should be). Got a couple of questions: (a) Is this a solid idea? If not, what’s the downside? (b) What’s the best way to go about this? Just a heads up: I’ve got my wallets backed up separately on other encrypted drives, so this won’t be my only safety net.

But why? Taken slightly out of context, it reminds me of this: (note: he wasn't talking about wallets) Unless you're restricted on bandwidth or RAM, I wouldn't worry about backing up blockchain data as long as thousands of nodes have a backup online! I would use this: This way, you don't need to overwrite old backups, while keeping multiple "snapshots" of older backups barely takes more disk space than your latest backup.

If you want to be more ideal, i have these suggestion. 1. Write down details of the backup, such as latest version of Bitcoin Core used to create/load the ~/.bitcoin/ directory. 2. Compute and store checksum for each files. It should be much faster than asking Bitcoin Core to perform reindex if you also worry about file corruption/integrity. 3. If you wish to reduce storage usage on your HDD, compress entire ~/.bitcoin/ directory. I remember old thread that discuss it could reduce the size by about 20%, although i can't find the thread.

I tested this with 7 blocks-files and pigz: 135 MB was reduced to 94 MB.

It's not only the bandwidth restrictions, but, isn't it also a pain when you need to wait 3 days to have a fully functional node up & running again? Can you elaborate on this please? Computing and storing checksums will certainly help with integrity, but how will they help in speed? Yeah, that's cool. I could also add an encryption password on the compressed file, instead of messing with encrypting the entire disk which takes time. So, one layer of encryption on the .tar file, another on the wallets themselves.

I know I'm just 16 GB of RAM short from making this 2 days less Other than that, it's up to you to decide whether or not that wait is a pain. I'm not using Bitcoin Core for anything that couldn't wait for 3 days. Compressing files makes incremental backups a lot more difficult. I used pigz for my test, which compressed better than bzip, but there's probably compression software that offers better results at the expense of slower compression. If you'd be running a pruned node, I'd say a backup (~15 GB) is worth saving a ~700 GB download. All this is coming from someone who had to restart a new IBD a few times over the years after I did something that fucked up my data directory.... I still don't have a backup

To be absolutely honest with you, I'm lucky enough not to have very urgent payments that I can't process otherwise, in a scenario where my Core nodes are totally destroyed, suddenly. I'm just lazy to wait for 3 days  I am running a pruned node, just for hot wallet purposes with small amounts. Haha, yeah I feel you. That's what I'd like to avoid.

The main problem isn't even the big size of the blockchain, downloading or storing it (which is a problem), but the actual validation of increasingly complex blocks due spammers. Check the charts: https://statoshi.info/d/000000009/unspent-transaction-output-set?orgId=1&from=now-5y&to=now&timezone=browser&refresh=10m If you go there you'll see that when the Ordinals protocol started and all this nonsense became popular (Ordinal Punks, Taproot Wizards, Bitcoin Rocks, etc.) followed throughout Jan–Feb 2023 and into 2023+ you'll notice it becomes increasingly difficult for the node to validate blocks. "Size of Serialized UTXO Set" is the total disk size of the serialized Unspent Transaction Output set (UTXO set) and so when you are syncing the blockchain, specially if you are using a non-infected computer (a computer from 2008 or earlier, without an NSA embedded OS such as Intel ME) then it will become a nightmare to sync it as it will struggle to process those blocks. Bitcoin Core v30 will only make this worse, which is why the logical conclusion is that Bitcoin Core v30 is a de-facto attack on the network and Knots 29 is the only thing we've got to corner spammers into irrelevance (and not to mention the whole CSAM-on-the-blockchain attack vector as well) As far as the backup, I would backup everything, but for a sanity check, I like to generate fresh chainstate files when running a brand new backup. The problem once again is, it's becoming a nightmare to do so on any useful laptops you may have. You pretty much need a gaming computer these days to verify the full blockchain. This is a disaster when it comes to running Bitcoin with maximum security.

That's not true. The thing that separates a "gaming computer" from a "normal" PC is a GPU, and Bitcoin Core doesn't use that. If you're limiting yourself to 17 years old hardware, then indeed Bitcoin Core becomes more and more challenging to run.

Although I get your point, I kinda disagree. I think what makes a difference is: 1. the RAM. If you've got enough RAM you can fit in more data during initial blockchain download. This will ensure that you're going faster than someone else who is limited with half of your RAM. 2. the bandwidth. Obviously you need a good internet connection. 3. the type of storage medium. SSDs are faster than HDDs and NVMEs are faster than SSDs. So, these 3 things exist on most modern (non-gaming) computers. At least I've got a decent laptop that fits most of these criteria. If I had an NVME, I believe I could decrease the IBD to 1/2 of the time.

I think a system with 32 GB RAM and HDD could be faster than the same system with 8 GB RAM and NVMe

yeah a wd gold 8tb hdd is fast enough with a good cpu and ram of 32gb or more. https://www.amazon.com/gp/product/B0CVMZTBJ4/ref=ox_sc_act_title_1?smid=ATVPDKIKX0DER&th=1 I have done a lot of pcs running  nodes.  Ram is a big deal

I guess both of you assume that the dbcache will be adjusted accordingly, otherwise I suppose 8GB vs 32GB doesn't make a difference, if you only use 600 MB of it.

I've never tested it, and I don't have spare systems with 32 GB, but I assume Linux file system cache can handle most of the work dbcache does. Other than that, I did indeed assume the correct settings to use the available RAM.

I mean computing checksum/hash than using one of Bitcoin Core features (such as verifychain[1] or reindex). [1] https://bitcoincore.org/en/doc/29.0.0/rpc/blockchain/verifychain/ As reminder, 1. OP_RETURN doesn't enter UTXO set/chainstate. 2. It's possible to add malicious arbitrary data on Bitcoin blockchain with other approach and it already happened long time ago.

I've always done this and it's a great idea, of course! However, since I'm running out of storage space and don't have any backups of the Bitcoin folder/data, I need to resolve this soon. Nowadays, if I need to redownload the database and blockchain again, I don't think I'll have many problems. However, with each passing year (every day, actually), the size increases, and it will take longer to synchronize a full node. I just want a way to make these backups, and when restoring, for example, from one laptop to another, move the (updated) backup to the laptop with the outdated Bitcoin folder, without having to delete the blockchain/blocks/database, allowing the updated backup to complement the outdated laptop's database. I don't know if that was clear, but that would be amazing.

If your node isn't running, can't you simply drag & drop the updated blockchain to the other node's database directory? Won't this simply replace the missing files?

On the other hand: with every hardware upgrade, it's a lot faster again For Linux, that's what rsync is for. But note that different Bitcoin Core downloads don't have the exact same blocks files, so if you're trying to "complement" one node with data from another note, it could still require copying almost everything.

Well lately I have been using a Mac Studio with a 2tb internal ssd. And 36gb ram with a 10000 speed eth cable. Although I use 1000 speed internet. And I cloned that to an external thunderbolt 5 ssd 4tb nvme . So I have a backup and the original. But all of above = money. No one needs this storage or speed for a full node.

Honestly, I've never tried this for fear corrupting the data, so I copy the entire current blockchain to an external HD or SSD (whichever I have available), but first I delete the outdated backup because it's less than 2tb available. So there's nothing I can do to optimize how backups are handled, right? I've already considered that possibility.