New node indexing is really slow now

13 replies 163 views
max42Member
Posts: 2 · Reputation: 87
#1Dec 8, 2020, 11:17 PM
So, I'm setting up a bitcoin node, using Bitcoin Core Ver 26.0. It started indexing pretty quickly, but now it's barely moving at all. Bitcoin Core crashed twice on me, once because my hard drive went to sleep after I forgot to change the setting, and I'm not sure what happened the second time. Could that have messed up the indexing? When I restarted it, it booted up fine, but the progress is just crawling. I ended the program using control q to shut it down cleanly and rebooted my PC, but no luck. I'm stuck with 361,355 blocks to go, and it says it’ll take 10 days to finish. It's been running for 30 hours already. It was down to 47 hours last night but now it’s just going down really slowly. Any ideas? Thanks!
4 Reply Quote Share
SwiftMinerSenior Member
Posts: 259 · Reputation: 1036
#2Dec 9, 2020, 12:47 AM
What you are experiencing is normal. Firstly you don't download the node rather the computer you are using runs as the node. You are actually downloading all of Bitcoin's block chain data and it's actually over 500Gb except you are trying to run a prune node by download some data and omitting some from a particular time frame. Depending on the computation power of your computer and network speed, downloading the Blockchain data from the Genesis block till date and also syncing, could take a couple of days. The faster you computational power and network strength, the faster the syncing process. Make sure you have a storage of up to 1tb preferably SSD to prevent issues due to space as it syncs and also when you attempt running it.
2 Reply Quote Share
max42Member
Posts: 2 · Reputation: 87
#3Dec 9, 2020, 04:50 AM
Thank you for the replies.  I am running a PC with 8gb ram and and an i7 processor and 400 Mbps internet connection.  I have an attached Samsung T7 2tb external drive to run a full node, not pruned.  It just seemed like it was indexing faster and faster than all of a sudden barely moving.  If it's still downloading the blockchain regardless of speed, does that mean there was no corruption in the blockchain that is downloaded so far? When you say start resyncing, you mean delete everything and start over?
4 Reply Quote Share
im_lynxHero Member
Posts: 515 · Reputation: 2161
#4Dec 9, 2020, 08:48 AM
Increasing dbcache=<about half of RAM in MB> might help a bit (add in your bitcoin.conf file). blocksonly=1 during initial blockchain download (IBD) lets your node only deal with block data over the network, don't bother at that time with unconfirmed transations. External storage via USB isn't ideal, USB protocol eats some performance and with an UTXO set of around 16GB size your storage has to shuffle quite some data around when it doesn't fit all in your RAM during blockchain sync. With optimized settings for a Raspi 4B with 8GB RAM and a 1TB SSD attached via USB3 a full IBD took about 4 days some months ago (my internet speed is 100MBit/s downstream). I was kind of surprised, I expected more time as a Raspi 4B isn't really fast compared to normal PC CPUs. I did this with an Umbrel node as an experiment and I hope Umbrel doesn't use some sort of shortcuts for IBD. I might still have the debug.log of this experimental IBD.
3 Reply Quote Share
byte_protoFull Member
Posts: 84 · Reputation: 625
#5Dec 9, 2020, 12:23 PM
I don't think you have to restart syncing, try Increasing the database cache size to since you are using an external drive this would increase the syncing process yes but you don't have to use this
1 Reply Quote Share
bull_2019Senior Member
Posts: 296 · Reputation: 1992
#6Dec 11, 2020, 01:20 AM
Firstly, check the usage of your hardisk if it's  getting  to 100% it will definitely  be slow no matter what . I've been unable to download mine due to low bandwidth  in my country  but I found a great trick that makes is fast . Get an external SSD drive Create a copy of your bitcoin data in the new SSD drive and rename the initial to another name to serve as backup incase of any  error Now follow this step as directed  from @NotATether if you're using the GUI mode In addition , the file is the bitcoin and you should save it as a conf file not as a text (note this)
4 Reply Quote Share
humbleledgerLegendary
Posts: 1027 · Reputation: 6554
#7Dec 11, 2020, 06:32 AM
There are 2 things I would improve: get more RAM, and don't use an external drive. It's very easy to accidentally disconnect the USB cable, and that will lead to file corruption. Chainstate when fully synced is 11 GB, that doesn't fit your RAM which will lead to terabytes of data written to your disk. Adding RAM will improve your overall computing experience, and is not very expensive anyway.
4 Reply Quote Share
bull_2019Senior Member
Posts: 296 · Reputation: 1992
#8Dec 11, 2020, 06:49 AM
What If I use a  no sdd drive?? Just  asking  because I got an external   ssd drive because my disk functionality  was been  used up entirely to 100% while synchronising. Does the RAM provides a way of not getting  the disk  used up ??
4 Reply Quote Share
im_lynxHero Member
Posts: 515 · Reputation: 2161
#9Dec 11, 2020, 07:36 AM
What is a "no SSD drive", a spinning magnetic platter aka HDD drive? What kind of an interface does your external drive use? Most common would be USB3 or up and USB protocol, especially when UASP is not used (your USB adapter needs to support it and the driver must have it enabled, that's not always on by default, depending what OS you use), slows down the performance of a SSD quite a bit. That's what LoyceV summerized with "internal drives are preferable", especially for SSDs. More RAM means, more data can be kept in RAM where access speed is still orders of magnitude faster than for internal and to greater extend for external drives. Less often data needs to be flushed out to storage. The OS has more space for file I/O buffers, can better optimize how data is written out to storage (moving R/W heads for HDDs is snail slow compared to flash storage with no moving parts).
4 Reply Quote Share
humbleledgerLegendary
Posts: 1027 · Reputation: 6554
#10Dec 11, 2020, 11:14 AM
Without enough RAM, that sucks If the 11 GB from chainstate doesn't fit in your file cache, you're going to be scraping that HDD. I have to ask: if your system still runs on an old HDD, why don't you replace it for an internal SSD instead of external? I made that switch years ago, and I'll never go back! Buy a decent one: some (budget) SSDs have low performance on sustained writes. I did another IBD test recently, with 8 GB RAM, Linux, and SSD, and the SSD was the bottleneck. It's was reading/writing at maximum capacity for hours in a row. Yes.
1 Reply Quote Share
bull_2019Senior Member
Posts: 296 · Reputation: 1992
#11Dec 12, 2020, 11:49 AM
Explains why my OS  crashed without  giving  any sign Almost lost my important  files though Or does Windows sucks entirely?? Yes Never cared about HDD until I was running  a bitcoin Core on the PC
2 Reply Quote Share
humbleledgerLegendary
Posts: 1027 · Reputation: 6554
#12Dec 12, 2020, 02:44 PM
Not really: the OS should kill the program that runs out of memory instead of crashing.
4 Reply Quote Share
bull_2019Senior Member
Posts: 296 · Reputation: 1992
#13Dec 14, 2020, 04:18 PM
But I got a crashed os luckily  I know how I could use the cmd-pmt for the retrieval of my file before resetting because i do keep my files hidden in the program data directory, what could have triggered it  because  It's  was my first  time experiencing it and that was when I started running  the Bitcoin core
0 Reply Quote Share
hash_bossLegendary
Posts: 1166 · Reputation: 5261
#14Dec 14, 2020, 09:43 PM
This is extremely slow. Last time i had to perform reindex, it took about 1-2 days where all files stored on HDD. And if you don't use your PC for other task, you can close most application and background application so you can let Bitcoin Core use a bit more of RAM. I recommend you to create new thread about your problem, rather than continue discussing on this thread.
4 Reply Quote Share
?Reply
Sign in to reply to this topic

Related topics