How to minimize disk usage for a node

Hey everyone, I’m building a web app aimed at bitcoin enthusiasts. The app needs to make RPC calls to a node to pull some data. The only thing I care about is the mempool, nothing else. So, I want to set up a lightweight node on my web server. I tried tweaking some settings on my local setup, but even with the minimal pruning, my .bitcoin folder is still around 12 GB because the chainstate is taking up about 11 GB. Is there a way to make that smaller? I'm trying to keep the hard drive usage really low since my cloud droplet has only 25 GB, and 12 GB is just too much. Is there any option in bitcoin.conf that can eliminate chainstate altogether? Here's the config that gives me that 12 GB: server=1 daemon=1 prune=550 maxmempool=100 rpcuser=redacted rpcpassword=alsoredacted rpcallowip=127.0.0.1 rpcport=8332 Also, is there any public RPC server out there that lets you make calls every 5 minutes or so?

I don't think that's possible. Without chainstate, Bitcoin Core can't know which transactions in your mempool can't exist.

I believe the maxuploadtarget is what you need for this

Wrong. Bandwidth has nothing to do with disk usage.

thanks. If you're right, it boils down to mounting an extra drive or finding a public bitcoin node that allows external RPC calls. I need one call every 5 minutes, that's 288 per day

Impossible, chainstate (which contain UTXO) is necessary to verify whether transaction violate Bitcoin protocol or not. There are few third party services which let you perform RPC operation which supported by full node software they run. I'm only aware of getblock.io which offer such thing, but take note i never use that service and i don't know whether it's legit or fake service. So i strongly recommend you to do your own research.

Most blockchain explorers offer a public API for users to query but that'll mean that you would have to trust their information to be accurate. If that fits your purpose, then you can try to use those: BlockCypher, SoChain for example. Caveat is that since you're not able to run your own node and verify the data, there is no way of knowing if they are accurate.

I *think* that if you set the dbcache to a high enough value then it's just going to keep the entire chainstate in RAM and then only flush it to disk on program exit (which I know is not very helpful but maybe you can delete the folder when the program is finished. Caveat: I'm don't know if this will make Bitcoin Core complain.) Getblock is not a fake service, I have used it before.

It'd require OP to rent VPS which have at least 12GB though. My Bitcoin Core's chainstate currently has size about 10.8GB, you also need 300MB of RAM for mempool and some more for OS. That's good to know, i said that since i recall they used to perform somewhat aggressive advertising here (where some of them deleted by moderator).

And you can add 1 or 2 GB RAM per year to keep up with the growth

Storage is far cheaper than RAM, and note that since your entire chainstate is in RAM, any corruption to the RAM would result in a catastrophic failure and thus another reindex. I doubt it wouldn't try to reserve the space for chainstate on the disk at all times. So it'll be far easier to just get more storage.

That's true, but if you're low on RAM, more storage isn't going to help you. Any failure in your RAM pretty much fucks up your computer. Luckily, it's rare for RAM to fail after it's properly installed. Each time Bitcoin Core verifies a block, it reads/writes a lot to chainstate. If you're low on RAM, the disk will be the bottleneck.

Yeah, that's why mission critical configurations have ECC and not for normal consumers. Even a slight corruption in the chainstate would be bad. Bitcoin Core only stores as much chainstate on the ram as you allow it. Keeping the entire chainstate on the RAM, permanently and not as a cache, would probably cause more problems than it solves. Regardless, the behavior for the chainstate flushing during IBD of a pruned node should be that the flushing occurs everytime that the blocks are removed from disk. Even if RAM is extremely reliable and that the server is never shutdown, Core would still attempt to flush chainstate to disk as the IBD progresses. Hence, it is necessary for the disk to have sufficient space to accomodate the chainstate, no way around that.

I've never tested it. It would be interesting to try: compare disk writes during IBD on a system with little RAM, to a system with 32 GB dbcache. I don't have the RAM to test this though. I've synced a pruned node in /dev/shm in the past. Works fine

That's including swap I suppose. If you're trying to keep your chainstate in your RAM, then that's the expected behavior (flushing) which is quite different from using your ram as a tmpfs or ramdisk. If you're using /dev/shm as a ram disk, then you can only use half of the RAM, and you'll have to consider the different overheads as well. Likely would require over 32GB of ram, which would probably mean that it's going to be a couple of times more than just adding half a terabyte to your server.

Did you TEST your app with testnet 1st? Why play with getting it working on the live chain and dealing with these issues before you know it is going to work. If you have issues now you will not know if it's a programming / app problem or a configuration issues because you tried to squeeze it into something too small. Also, you really should be using a hosted solution with a lot more storage & power. If it does work and you release it if you are that limited on storage and RAM it's going to get clobbered once it's in the real world. -Dave

There's a reason why you can't prune the chainstate. It's this one thing you absolutely need to verify that the transactions do not go against the protocol. The most you could do to save up space is enable prune=550 (doesn't go less than that). It is theoretically possible to run a full node from home (where you probably have sufficient storage), and configure your private server to communicate with your home node. But, that's just too much effort. Just buy more storage on your private server.