Bitcoin Knots Update SSD Daily TBW Soars

Check out this chart I've attached. It shows the software my node's been running along with the total gigabytes/terabytes written on the SSD. I'm using Xubuntu on an 8th gen Intel Celeron with a SATA SSD. So, every time I update Bitcoin Knots to a version higher than 28.1, my disk writes go through the roof. Before, I was seeing just below 20 GB per day, but now it’s shot up to 230 GB per day. That's a crazy increase! I suspect that something in the newer node version is making the indexer (I'm using Fulcrum) work way harder or something like that. I’m kinda stumped here. Can anyone help me out? Why is the newer Bitcoin Knots version causing all these extra disk writes? I’m on an SSD, and I really don’t want to wear it out too soon. Is there a setting I can change to help with this? What’s the difference between versions 28.1 and 29.3 that’s making the indexer write so much more? I asked something similar not too long ago, but it was regarding the ElectrumX indexer instead of the node version. Initially, I tried that and saw high disk writes, which is why I switched to Fulcrum. Here's some extra info about my setup: - Fulcrum 1.11.0. - Xubuntu 24.04 LTS - 8 GB of RAM - Intel 8th gen Celeron CPU (2 cores, 2 threads). - With Fulcrum and Knots 29.3 running, my task manager shows only 2.0 out of 8 GB of RAM in use, so I doubt it's a SWAP issue.

Your problem is the same as it was 2 years ago: The problem isn't swap, it's chainstate. If it doesn't fit in RAM, it needs to be largely rewritten for every new block. Two years ago it was 12 GB, now it's 13 GB. I can't tell you this, I've never used Bitcoin Knots. But RAM would be the first thing to upgrade.

For additional information, UTXO/chainstate size passed 8GB in middle of 2023[1]. That's why the total write suddenly skyrocket. It's when Ordinal gaining popularity, so Fulcrrum also works harder to index more address and UTXO. [1] https://statoshi.info/d/000000009/unspent-transaction-output-set?orgId=1&from=now-5y&to=now&timezone=browser&refresh=10m

I'm not sure if it's actually his ram. Because in the graph that he provided, the average disk usage reduced again when he downgraded his client to the unaffected version and then spiked again after another upgrade. If it's hardware, any of those versions should exhibit similar performance issues. @OP how accurate are those client version labels in the graph? I'm not using Knots to help you accurately but those versions have BIP110 activated, perhaps it has something to do with its new consensus.

Even the lower value of 20 writes per day is high, with sufficient RAM I'd expect it to be a lot less. Could it be they flush dbcache to disk a lot more frequently in the newer version?

Those labels are perfectly accurate, down to the day. I know because I remember when I switched versions. Your observation is correct, as soon as I downgrade back to 28.1, the writes are reduced significantly, back to their prior trend (20 GB/day). Perhaps it is related to the new consensus rules, but I don't really understand why that would be related to this. I'd assume it would be some other change that is causing the indexer to work harder. But I don't know what. Also, regarding this comment by LoyceV: "Even the lower value of 20 writes per day is high, with sufficient RAM I'd expect it to be a lot less." Keep in mind that this is total system disk writes, not only the node, but also the indexer (Fulcrum). The indexer is responsible for most of those. Would the indexer be writing less if the system had more RAM? I can test this with 16 GB of RAM next weekend, but I doubt this is the problem, considering the task manager isn't showing high RAM utilization and I'm not aware of what the newer versions of Knots would be doing that would require so much more.

Even with the issue, this might help to reduce your disk usage since you can keep the whole or a large chunk of the UTXO-set in your memory. Consider bumping your dbcache value after you upgrade your RAM. IICRC, they also updated how their dbcache is set by default. If it backported updates from Bitcoin Core, its default should have been increased to 1GB and I've also read that Knots auto-sets it depending on how much RAM the system has. (CMIIAM) However, the first v29.x version that you've used which also exhibit the issue doesn't have those updates yet so it shouldn't be related to the new dbcache defaults.

@OP Have you tested how current Bitcoin Core versions behave in your environment? Would be interesting if there's a similar change in daily disk writes. I'm not a fan of Knots and the nonsense in my opinion it shills (YMMV!), so take my words as someone who wouldn't run Knots at all. If you have reasons to use Knots, not my business. Well, because I'm pretty much Knots agnostic, I've no idea what causes the change of disk write pressure you observe. I don't want to endorse to change too many variables, but I have this question: why do you run outdated version of Fulcrum? In the v1.xx branch your version isn't the last, IIRC Fulcrum 1.12 was latest. Version 2.x of Fulcrum brought more database reliability where database corruption should be a thing of the past. Upgrading to the v2 branch requires either a database upgrade or starting from scratch by cleaning the datadir of Fulcrum (slow multiday rebuild from block 0). At the moment I run Bitcoin Core v30.2 and Fulcrum v2.1.0 in a virtual machine with allocated 24GiB RAM (will reduce it to 20 or even 16GiB soon as the VM host would benefit from a bit more RAM back). I don't look at the daily disk write stats. Maybe I should?

Good idea to test this, I just switched to Core 31.0 tonight (still 8 GB RAM). I'll let it go for a few days and see if the writes are still high and report back here. I don't have a good reason. I've just never bothered to update it. I'm assuming the database migration will take a while to complete, so I think it makes sense to wait to try this until I've explored other possibilities first. Yeah, I'd recommend monitoring this. I have a python script that runs nightly which uses smartmontools to pull the disk writes and append it to a csv file.

Most users don't need to worry about this. Even at 230 GBW/day, that's still only 84 TBW/year, which means it takes much longer to wear out a modern SSD than it's economic lifespan. Simply put: you'd want a bigger one long before it reaches it's limit. I just checked mine, and SMART shows this after writing 50 TB to the disk: Wasn't the whole part of "SMART" that it gives you a warning when you need to start worrying? I've had old disks with much lower TBW-numbers, I think I've seen 60 or 120 TBW limits, but those are far too small to run full nodes with Fulcrum anyway.

Yeah, I'm not worried at all, my VM host's NVMe SSD will likely last longer than this mad RAM and flash memory shortage crisis will last. Here are my current S.M.A.R.T. stats of it, if anyone is interested. This SSD can very likely endure quite a bit more data written to it. I didn't see much of a need to monitor this before and am not convinced to need it in the future. I'm fine to look every few months or so at the data and be good with it. The main action hogs on this machine are docker containers with Bitcoin Core and Fulcrum which run 24/7/365. There are a few other docker containers with local mempool.space and bitcoinexplorer.com instances that I occasionally use for privacy when I browse the blockchain.

68.8 TB in 14.086 hours is 0.00488 TB/hour. At this rate, you'll reach the limit in 68 years. Long before that, you'll need to upgrade because the blockchain doesn't fit anymore

It looks like Core 31.0 has this problem too. https://i.imgur.com/LMKeNcQ.png So that eliminates it being Knots-specific or related to the new consensus rules. Any ideas now? Has anything in Core changed recently that would cause this? I will still try putting more RAM in it this weekend, just haven't had time to do that yet. I can also try updating Fulcrum, but I will try that last. If there are any settings I could configure that might address this, I think that would also be worth trying first.

If I have to guess: they may have changed something to make it faster if you have enough RAM. Last time I did a full IBD with Bitcoin Core 26, it barely used any RAM despite setting a high dbcache value. I've seen performance improvements from upgrades before, and I assume this is the same. Unless you're low on RAM, then it's counter productive, apparently.

Hm, interesting. I would then assume that a similar version of Core to a Knots version that has lower daily write pressure should perform likely the same. I'm not sure if it makes sense to determine which exact version of Core has lower write pressure under otherwise constant conditions. I can't quite recall changes of the versions of Core that would point to significantly more write pressure (maybe limited to lower RAM environments as LoyceV hints at). Don't change too many variables, so keeping your current Fulcrum version makes sense at first.

OK, I just put in another 8GB DIMM so now the system has 16 GB total. Unfortunately the system only has two RAM slots and I don't have any 16 GB DIMMs, so if 16 isn't enough, I won't have a way to test more than this at the moment. Still running Core 31.0 for testing purposes. From looking at task manager, I'm not seeing a meaningful difference in the amount of RAM used (still hovering around 2GB) but maybe I'll see a reduction in writes. I'll give it another day or two and we'll see.

Have you increased your node's database cache settings? If not, it wont fully utilize those added RAM. Try to set your dbcache to 8GiB or higher if your system doesn't have much background/other processes. Then restart your node.

Out of curiosity, is the sync speed noticeably slower (as in total downloaded and verified block) compared with Knots 29.3 or Core 31? You can use debug.log to know the overall sync speed. I skimmed release note from 1.12 to latest version, but it's mostly about performance improvement, preventing corruption and other general fix/improvement. There's no mention it would reduce disk write. But I still recommend you to update if you have spare time.

Thanks for the reminder. This morning I set dbcache=12288 and restarted the machine. In all my prior tests so far with 8GB RAM (I was just switching node versions), this had been left as unconfigured. We'll see if that makes a difference. I'm not quite sure what I'm looking for in debug.log to determine this. Nothing was jumping out at me when I looked through it, but I don't really know what I'm looking for. I also started looking at the fulcrum logs to see if I could notice any difference between when I was running Knots 28.1 (low writes) and Knots 29.3 or Core 31.0, but unfortunately nothing is jumping out at me as meaningfully different. Also, I read some of the Fulcrum documentation  (https://raw.githubusercontent.com/cculianu/Fulcrum/4b38a7590e2f1d28d5f3071b799c2bb29ee6fc81/doc/fulcrum-example-config.conf) and I noticed there are some options I might want to tweak there: db_memdb_max_open_filestxhash_cachebitcoind_throttle If increasing system RAM and the bitcoin dbcache value doesn't fix this, perhaps tweaking some of those options will. Let me know if you have any thoughts on that.

This is quite a large value of 12GiB for dbcache compared to available 16GiB for your whole system. Don't forget that your OS needs some RAM and Fulcrum also. You should carefully observe RAM consumption and not forcing something to use swap heavily. (I assume you have some default settings for swap space?) If you tweak settings of Fulcrum, in whatever direction, keep an eye on its RAM usage. If you give Fulcrum more, you have to take it away from your node and vice-versa.