Hey everyone,
I’m dealing with a nagging issue where my RAM keeps getting maxed out while using Bitcoin Core Desktop from Flathub.
Here’s what’s going on:
I set up a Bitcoin Core node on my machine and hit it up every minute with a Python script to check for new blocks.
When I use the API /rest/block/<BLOCK-HASH>.json, I get an almost instant reply, but the memory usage of the daemon skyrockets with each block request, and it doesn’t seem to free up afterwards.
To fix this, I’ve got to shut down Bitcoin Core and restart it now and then.
On the flip side, if I use /rest/block/notxdetails/<BLOCK-HASH>.json and handle the JSON response by querying each transaction, my memory usage stays pretty steady, but it’s way slower.
Any ideas on how to tackle this issue? Thanks a bunch!
MEDIUM SOLUTION:
So, I ended up downloading bitcoind and got it installed. I created symbolic links to where my blocks are stored, ran bitcoind -daemon, and let my scripts loose. To my surprise, the memory barely budged. I queried over a thousand blocks, and my RAM only went up by about 150MB, which isn’t much at all when you think about those last 1000 blocks.
Meanwhile, with Bitcoin Core Desktop from Flathub, just querying a single block at full verbosity cranked my memory up by around 100MB.
Bitcoin core uses the RAM alot and to reduce this your hardisk function usually gets involved and if it reads at a lower rate then you could end up getting a slow response because it will get used up faster while running your node ....
There are some couple of things you could do to speed it up
Get a new SSD Drive with a larger storage since hardisk needs more space to peform faster If your PC has an additional slot for an extra ram then you could get an additional RAM for your PC else if you have just a single slot and your ram is low, then replace it with a higher Ram .
You could take down some background processing applications to free up usage too although it's less effective but it contributes.
With these you should have it speed up
Yes! reducing the query time interval can also help in speeding things up
How much memory do you have?
12GB RAM availability
Also can you try a 5 minute query vs a 1 minute query?
Its the same, in each query begins to increase. If you want you can try it and see how it increases:
Thanks for your time.
It is not a scalable solution for me, the day will come when even if I have 500GB ram I will reach the RAM limit.
It also doesn't seem very smart to me to be closing and opening Bitcoin Core every 2 days.
It is interesting to note that using the API:
And then processing each tx query with my python script:
The memory always remains the same! but it takes between 30 to 45 seconds for my script to query each of the tx inside each block (blocks with more than 3K tx inside)
I made a Benchmark python script:
With API /rest/block/{blockhash}
With API /rest/block/notxdetails/{blockhash}
Okay what's your pc processor if I may ask ??
For the hardware component try getting an additional SSD if you have an extra slot for hardisk on your pc . The exFAT format is for windows actually move your node_block directory to this new SSD drive and create a partition from your bitcoin.conf file. Note don't delete your old node_block directory...you can just remain it .
Now you can query from the new directory.... i don't know if I'm on point that align with yours
For the software component you could give the response give by @LoyceV a trial
Seems like there's a memory leak somewhere. Do you see the same problem if you use the getblock RPC with verbosity 2?
e.g.
Both that rest api and the rpc use almost the same code so this would help narrow down what's causing the problem.
First of all, thanks to everyone who has taken the time to help me solve or detect the bug...
I ran the command
and the memory leak problems continue, every query increases the memory allocated to bitcoin process by approximately 100MB.
What is extremely strange is that it happens when the full verbosity block is consulted. Because when the block is consulted without verbosity and I consult all the tx of the block separately (my python script), I do not have memory leaks, in fact my memory remains stable.
I'm curious, can you consult that block and see your memory? Is it just me with this problem?
First of all, thanks to everyone who has taken the time to help me solve or detect the bug...
My Fedora Distro is installed in my laptop SSD and my org.bitcoincore.bitcoin-qt directory is in my external SSD. Im using a Bitcoin Core Flathub installation.
Im going to try the approach of install bitcoind and test to see if the problem remains.
I seem to have reproduced this on my machine, at least calling getblock with verbosity 2 has increased the memory usage for most of the calls.
I've opened an issue on github so other developers can take a look and debug this: https://github.com/bitcoin/bitcoin/issues/30052
The behavior that I've observed is that it doesn't always increase the memory usage. This suggests to me that it's probably loading something into an unbounded cache, and the cache is growing too large causing the crash.
I think the error is reproduced with all current blocks, blocks with a lot of data.
Thank you for the actions taken, how do I follow up on these issues?
How long does this whole process take?
Thank you
If you have a github account, you can subscribe to the issue to get notifications when comments are posted to the issue. If you have anything else to add, you can also comment on it yourself.
It depends entirely on if anyone is interested in debugging this, and how hard it actually is to debug. Either way, someone will have to do the work to figure out what is wrong, then open a PR to fix it. Then the PR will need to be reviewed, and that can take anywhere from a few days to several years, depending on how complicated it is. You won't get the benefits of it until it hits a release, and the next major release is scheduled for October. There may be a backport release containing this fix before then, but these are not guaranteed nor are they scheduled, so don't count on it.
Basically, it'll be at least several months before a release is made with this fixed.
MEDIUM SOLUTION:
It turns out that I downloaded bitcoind and installed it, created symbolic links to my folders where my blocks are located, ran bitcoind -daemon and ran my scripts, the surprise I had was that my memory barely changed, I consulted more than a thousand blocks and my Memory barely increased 150MB, something quite insignificant considering the last 1000 blocks.
With Bitcoin Core Desktop version (Flathub) querying only one block and full verbosity increased my memory by approximately 100 MB.
I want to note here since you use python that the Python interpreter does not garbage collect freed memory in many instances but instead holds onto it to reuse it for some other variable, even if you explicitly delete the variable or call the garbage collection functions in Python. It's not directly related to your problem, but if your script sees high memory usage that is not going down, this is why. The only way to fix that is to run your code in a subprocess that you create in your script.
But anyway, I have also previously experienced this problem in Bitcoin Core that you are referring to. The large memory usage happens when you try to fetch many transactions at once (as what happens when you fetch a block) because the node has to get more information for each of these transactions from other sources i.e. other blocks. The higher the verbosity, the worse the memory usage becomes. It is also quite independent of the number of RPC threads you are running, but the memory usage becomes noticeably higher if you batch many RPC calls together.
Neither I nor any of the other developers are observing unbounded memory usage. It does go up a bit, but it always plateaus pretty quickly and doesn't seem to get into memory exhaustion.
Seeing some memory usage increase with this level of block verbosity is expected since the block has to be read into memory, decoded, and turned into JSON text with a ton of extra details that are also not explicitly in the block file itself, and text is not very efficient.