So, I didn't wanna clutter the challenge thread with this, so I'm starting a new one for my own question. I'm getting really curious about the tech side of running a node.
I noticed that while my Bitcoin Core wallet is syncing, blocks are being downloaded as .dat files, which is the same type my Electrum wallet uses. Can someone break it down for me like I'm five? What’s the deal with this format, and how do block files differ from wallet files even though they look the same? If I’ve got all the block files, why is the syncing still taking so long, like it’s reading everything again? I did a little test where I saved my downloaded blocks, uninstalled Bitcoin Core, and then put the block files back in the folder after reinstalling it. Is there a way to skip certain blocks, like the ones from 2009 to 2012, and just grab the blocks from 2013 onward? It seems kinda pointless to download everything from the beginning since pruning takes away the old blocks and keeps syncing with the newer ones.
And what’s the point of running a node, anyway? Am I contributing to making the blockchain more decentralized? I’m a total noob, so please take it easy on me and keep it simple so I can understand.
They just happen to have same extension, the format of those files are different. They have different format and usually Electrum create wallet without extension. If you actually refer to Bitcoin Core's wallet.dat, they just happen to have same extensions.
I'm not fully understand your question, but you need to copy whole folder (which also contain other stuff such as "chainstate" folder) rather than only block files.
Theoretically it's possible, assuming you have programming skill to modify Bitcoin Core (or other full node) software. But that means you're skipping some verification process which is point of full node.
You get to improve your own privacy and the decentralization of the network. Just as with mining, the more divers nodes we have keeping a record of their own mempool, the more decentralized blockchain information is, making the information immutable.
- Jay -
I back up that too including all the files on the file directory for Bitcoin core. I manually pasted all the files that I previously downloaded to the new file directory of Bitcoin Core after I reinstalled the apps. Still, the synchronization of "headers(not sure of the exact word)" is so slow that it almost didn't progress anymore so I just forgot my previous progress and started fresh again.
If it stuck during syncing block header (NOT block), it's probably due to internet connection. There's no need to start fresh again (as in download blockchain from beginning).
Bitcoin nodes need to assemble the global UTXO set while they are syncing. This is basically the outputs of every bitcoin address that have not been spent yet. In order to make such a list, it needs to read the blocks in order, without skipping anything so that it can verify that there are no invalid transactions inside a block (which should never happen in theory but you can always edit the .dat file to make such a corruption). That is also why it is reading the blocks even after you save them.
In case you still find it strange, it is verification which takes most of the time. Download speed is pretty neat in almost every part of the world. With an average 10 MB/s, it would take you about just a day to download the blockchain. However, verification might even take a week, depending on your computer resources.
You know the truth. You need no third parties to figure out how much money you have. It also a privacy upgrade, since you let no third party scrutinize your Bitcoin activity.
The file extension doesn't matter much, it could be .txt or anything else. What matters is the data stored on those files. blk.dat files contain data related to the blocks and transactions that have been mined while wallet.dat contains data related to your wallet.
The blockchain, as the name suggests, is a chain of block where each block is connected to the previous one. To verify the validity of the blocks and transactions you have downloaded, your node needs to verify the validity of each of them going back to the genesis block (first block ever mined).
I believe not for the reason explained above.
Yes, by running a full node you contribute to the decentralization of the network and by using bitcoin core as your wallet or connecting your wallet to your own full node you improve your anonymity.
I appreciate all the replies and learnings to this uncharted territory for me.
@NotATether I will use this thread to inquire about my problem with your running a node challenge.
Bitcoin core syncs very slowly when it reaches 15% progress while Bitcoind performs at a usual speed. I'm not sure what is the problem because my RAM and CPU usage were still the same when I started this challenge including the storage on the hard disk since I used the prune command that you suggested on the challenge thread.
the screenshot below is my current stats
it doesn't have any good progress even though I left it overnight running.
There is no problem and syncing becoming slower is completely normal.
As you know, your node verifies the validity of each block it downloads and the transactions in it. Currently, your node is synced up to 2016 (according to the screenshot you posted) and prior to that date bitcoin wasn't so popular so blocks back then we're either empty or had few transactions on them therefore verifying their validity (syncing) doesn't take too long.
Gotcha. This makes sense a lot. Thank you, I'm just a little bit carried away by how fast it was synced on the first day it almost completed 15% so I thought that it would be inconsistent speed. So this means that the 2016 onwards block content is much heavier in file size due to the volume of transactions.
Is there a way to speed this up a little bit using the remaining unused RAM and CPU?
Follow-up question, What is the data shown by the Bitcoind window? Is it related to the syncing process on Bitcoin core because my Bitcoin core is progressing even though Bitcoind is closed?
Last I checked Electrum wallet file didn't have any extension and it was not a dat file either. It's plain text using JSON format.
You should be able to increase your cache size and speed up the verification process. Set the dbcache in your bitcoin.conf file to 8 GB (since you have 16 GB memory, I believe half is good).
Set blocksonly=1 in bitcoin.config file. As it was mentioned by gmaxwell it may result in "reduced the node's bandwidth usage by 88%." After completing IBD you may set default , i.e. blocksonly=0
88% was obtained from Bitcoin Core 0.12 though. Since then, Bitcoin Core implement compact block which allow faster block propagation while reducing node's bandwidth usage (since node doesn't have to send transaction data (standalone and in block). And i recall node which in sync progress doesn't obtain TX mempool, although i can't find source of that statement.
You are correct, even though it is not 88% nowadays but "absolute minimum" matters. In fact , I had a chance to appreciate blocksonly=1 setting a few months ago at my node's IBD.
How can I know if the "dbcache=8000" setup is already working? The Bitcoind windows process too much data quickly which makes it hard to read the initial summary when I open it.
why this code doesn't have "#" in the beginning compared to other commands listed on my bitcoin.conf. Lastly, what's the difference between Bitcoin.conf file located on the file path in which I installed my Bitcoin core and Bitcoin.conf that I created on C:\Users\YourUserName\AppData\Roaming\Bitcoin path when I input the pruned command? Thanks for your guidance sir.
# is a commenting character. Putting it before the command disables this command.
Regarding the value of dbcache. As a rule of thumb dbcache=1/4 RAM. Thus in 16 GB case the optimal would be dbcache=4096
Read more on dbcache which essentially acts as write buffer rather than a cache.
You should create your bitcoin.conf not .txt and once it's setted you should see changes
The # makes it a comment jut like writing comments for codes
Comments don't get to be read in a code just for readability and reference mssg
On
I have a follow-up question to this, why did blocks during 2016 give me a slower sync while Bitcoin transactions during 2017 onwards were much higher due to the demand? I sync faster now after I finished syncing blocks in 2016. I'm already at 85% progress compared to when I posted this which is just 15% sync stuck for a long period.
I'm curious what's with 2016 blocks that cause slow sync?
You have been given a few meaningful optimization tweaks. Did you apply those and now wonder why sync speed improved?
We can't know how your computer is utilized and thus what all affects the sync process of your Core node.
I once made a test with my Raspi 4B 8GB RAM with a 1TB SATA-USB3 SSD (UASP active) running a Umbrel node: I had to uninstall any app in that Umbrel and thus wiped all Bitcoin Core blockchain data on purpose. I then installed only the Bitcoin Core app and tweaked dbcache to use about half of RAM (4883MB were accepted as dbcache value). The initial blockchain download and sync took (Tor-only for outgoing connections) to my surprise only about 95h on my Raspi 4B (I have stable ~100MBit downstream for my internet connection, you hardly get this speed sustained over Tor all the time; but it's possible this Umbrel node talked to any of my other three local Core nodes that are running in my local network, I haven't excluded this possibility to keep things simple).
All those who complain a Raspi 4B takes weeks to sync, I can reply that with optimized parameters a Raspi 4B is capable of executing an IBD in less than one week with a normal 1TB SATA-USB3 SSD enclosure for data storage.
P.S.
I still have the debug.log of this IBD experiment. I started it on approx. 2023-06-22T18:00:00Z and finished sync to chaintip at block 796033 on 2023-06-26T16:46:31Z. Later bloating of the UTXO set won't make the sync process any easier for low RAM devices.