Expecting High Disk Writes?

So I've been running Bitcoin Core 27.0 with txindex=1 and ElectrumX 1.16.0 on a Samsung 860 Evo SSD using Xubuntu. This setup is totally dedicated to just those two apps all day long. I’ve been using smartmontools to keep an eye on the overall disk writes each day, and honestly, the numbers are pretty alarming. Since SSDs have a limited lifespan based on write cycles, this has me worried. Right now, I've only got a few data points, but these were collected AFTER I finished the initial block download and after the transaction index was built: 30 total gigabytes written to the SSD from June 24th to June 25th 108 total gigabytes written to the SSD from June 25th to June 26th 158 total gigabytes written to the SSD from June 26th to June 27th Just to clarify, these numbers show the increase in total GiB (512 bytes * Total LBAs Written / 1024**3) written to the drive from one day to the next, and not the actual storage capacity used. The disk space used is pretty much what I expected, only going up a few hundred megabytes daily. I'm getting the LBAs written via smartmontools by running: sudo smartctl -a /dev/sda and checking the Total_LBAs_Written field, in case anyone else wants to monitor this on their setup and see how it compares. I know I’ve got more tests to run before I can pinpoint what’s causing this for sure, but my gut tells me it’s got to be somehow linked to Bitcoin Core or ElectrumX since it’s a dedicated machine for those.

Can you share how much RAM you have, how much swap is in use, and Bitcoin Core's dbcache setting? It looks like something changed in between those days. I'd expect Bitcoin Core to write approximately the same amount each day (at least in days so close together). Correct. I've used this script (use at your own risk) to check it in the past. Here's my findings: on a standard pruned Bitcoin Core (so no txindex I guess, and no ElectrumX) with only 8 GB of RAM, my SSD wrote just under 5 TB for the entire IBD. It looks like you're writing a lot more than that. The 500 GB version can handle 300 TBW. Even at 150 GB per day, you're looking at 5 years.

The system has 8GB of RAM on an Intel 8th gen Celeron CPU (2 cores, 2 threads). The node isn't pruned. Xubuntu's built-in task manager is showing Memory at 25% (1.9 GiB / 7.6 GiB) and Swap at 34% (1.4 GiB / 4 GiB) at this exact moment, having had both Bitcoin Core and ElectrumX running 24/7 for about 3 or 4 days. I guess I don't really understand why the Swap is being used significantly given how much memory appears to be available, but maybe this is normal. If not, let me know. I don't have anything in the Bitcoin.conf file specifically changing the database cache. However, from what I can see in the GUI using bitcoin-qt, under Settings -> Options -> Size of database cache, I see it is set to 450 MiB. Also, I checked smartmontools again and confirmed the sector size is 512 bytes, so my math should be correct.

At a glance, i find it's reasonable since you only have 8GB of RAM when current UTXO size is about 11.6GB according to https://statoshi.info/d/000000009/unspent-transaction-output-set and you need to share with both OS and ElectrumX. But for starter, you probably want to run iotop --accumulated and wait for few days to know which process perform most write. In addition, according to Jameson Lopp benchmark, the write is much lower if you have much bigger RAM and configure Bitcoin Core to use lots of RAM. Although his benchmark doesn't enable txindex either.

Adding more RAM should largely reduce your disk writes, but that may not be possible considering it's a Celeron. That's normal. The other memory is not empty, it's probably in use as file cache. Writing to your swap file is meant to improve overall system performance. You can adjust this behaviour by changing "swappiness", but be careful not to make it worse. That's the default, and usually a larger value is recommended. Ideally, it should be enough to fit chainstate, but since that's 12 GB, most people don't have that luxory. I've never tested how increasing dbcache affects performance. I assume Linux file cache is quite good at it by default. That makes sense: with enough RAM, chainstate can stay in memory until you shut down Bitcoin Core. So all that's written is the full blockchain, which must have been 561 GB back then.

I'm getting the impression that adding more RAM mostly reduces disk writes before the node is fully synced, not after. But I have extra RAM and I can bump it up to 16GB so I can give it a shot. The Celeron supports up to 64 GB of RAM according to Intel's website. I could also put in 32 GB later (I can swap it out from another old computer), but I'll start with 16 for now. From what I can tell, I should set this line in bitcoin.conf (12288 = 12 * 1024): Is that the only thing I need to change? Or are there other settings I should try too?

Each new block requires verification, which includes reading and writing to chainstate. Usually the mainboard is the limiting factor. With 16 GB RAM, that doesn't leave much for the rest of your OS. I'd go for no more than half your RAM, so try 8 GB dbcache if you install 16 GB RAM. That should be enough. For what it's worth: my system has 16 GB RAM (and the mainboard can't handle more), and during IDB I increase dbcache, but when normally running I reduce it to 1024 again. I don't want Bitcoin Core to occupy the majority of my RAM, I need it for other things. I just checked: I've written 7.5 TB in the past year (I didn't do a new IBD on it). That's 21 GB written per day, for Bitcoin Core, swap and everything else running on my system. I keep chainstate on this SSD, and blocks on my HDD. At this rate, I'll reach my SSD's rated writes after 80 years.

Thanks for suggesting iotop. I've been running it for about 10 minutes and I already get the impression electrumx is doing all the disk writes, not even Bitcoin core. After just 10 minutes I'm seeing: Disk Writes          Command 3.36 G                 python3 /home/user/Desktop/electrumx/electrumx_server 17.97 M               [jdb2/sda2-8] 6.52 M                 systemd-journald 3.84 M                 bitcoin-qt -min -chain=main [b-scheduler] 3.75 M                 bitcoin-qt -min -chain=main [b-msghand] 3.36 M                 tor --defaults-torrc /usr/share/tor/tor-service-defaults-torrc -f /etc/tor/torrc --RunAsDaemon 0 So it's almost certainly my elecrumx configuration I'll need to scrutinize, not Bitcoin core. Edit: Here's what I'm seeing after one hour: Disk Writes          Command 8.49 G                 python3 /home/user/Desktop/electrumx/electrumx_server 56.66 M               [jdb2/sda2-8] 23.80 M               systemd-journald 20.05 M               bitcoin-qt -min -chain=main [b-msghand] 16.27 M               bitcoin-qt -min -chain=main [b-scheduler] 13.89 M               tor --defaults-torrc /usr/share/tor/tor-service-defaults-torrc -f /etc/tor/torrc --RunAsDaemon 0

say monero clearly stated that their node will ruin ssd. i'll rather go with sas hba and sas hdd. used sas1 are cheap now as  are sas2 hdd's, it's enterprise-grade things. by hba i mean raid card flashed to IT mode (it'll be LSI one most probably). Because if it's not flashed already you'll need to do this https://www.reddit.com/r/DataHoarder/comments/lck2ou/comment/gm93zpp/      on your own.

Funny you say that because the electrumx devs specifically recommended using an SSD. (As I pointed out above, it's electrumx that is doing all the disk writes, not Bitcoin Core.)

and that's after he'd checked out factory specs. hopeless case

I've never used Electrum server, but I'm curious: what could it possibly be doing that requires so much disk writes? This has nothing to do with Monero.

Assuming the output comes after you use 16GB RAM and set higher dbcache value, it's not surprising since Bitcoin Core would use disk less frequently. I saw you asked on GitHub whether changing CACHE_MB value may solve the problem. But description under CACHE_MB also mention ElectrumX write something called "unflushed history" frequently which doesn't seem could be changed. I recall their node software also recommends using SSD if you store the data on HDD. Although i don't know if they still show that recommendation today.

At the moment I cannot find any settings to change for electrumx that would lower the disk writes. As a temporary solution for now I'm thinking I might use an extra 2.5 inch HDD from one of my old laptops and dedicate it to just the electrumx db folder (this is where all the writes are happening) so I can keep it separate from the SSD the OS and Bitcoin Core are installed on. Obviously performance will be worse but I'm not sure it will be bad enough to really matter for me. Either that or I will use a much cheaper extra 256 GB SSD instead.

I wouldn't move the writes to a HDD. That's terrible for performance. Note that cheap SSDs are usually lower quality, both in endurance as well as performance. I'd just let it be, keep backups of the data, and by the time your SSD is "worn" in a few years, replace it with a bigger one.

Using HDD seems to be poor choice. If i were you, i would try different Electrum server implementation such as Fulcrum or Electrs. Sparrow Wallet developer even write a benchmark about it on https://www.sparrowwallet.com/docs/server-performance.html.

I'm giving Fulcrum a try right now. The database is currently building. It uses rocksdb while electrumx (by default) used leveldb, so maybe the write activity will be different. I will report back once it's all finished.

1tb ssd 32gb ram I5 8500  or newer using the Celeron and 8gb ram is simply a shit show. I have run a ton of different  wallets in fact I will be doing a new one as I now have 1 gb  vs my old internet

I'm not sure about this. As I mentioned, not very much RAM was being used according to Xubuntu's built-in task manager, and when I put in 16 GB instead of 8 GB of RAM, I noticed absolutely no difference. I highly doubt the processor would impact the amount of disk writes. Once again, it's not like the performance is bad; it's actually been entirely sufficient for me. I just can't figure out why ElectrumX is writing so much to disk. This very well could simply be how it was designed, and everybody, regardless of hardware, would experience this.

Did you miss this part? No difference in memory usage, or no difference in disk writes? I guess it depends on what your Electrum server is doing: Bitcoin Core would benefit from larger dbcache, I don't know anything about Electrum server. Agreed.