Looking for Tips on Setting Up a Full Node for Deep Analytics

So, here’s the deal: I want to set up a full Bitcoin node to do some deep analytics and tackle a few research questions I have about wallet behaviors, scarcity, and how price changes impact the whole ecosystem. I could really use some input from you guys on the hardware setup, any possible bottlenecks, or anything I might be missing at this point. A bit about me: I’m a data engineer with a background in computer science, network engineering, and physics. I’ve got a solid track record working with big data, complex models, and analytics. For the last ten years, I’ve used these skills in scientific research, stock market analysis, and behavioral studies. Now, I’m jumping into blockchain data and looking to create models to answer some of the lingering questions out there. Here are some of the questions I’m digging into: - How do active wallets behave compared to inactive ones? - What’s the impact of price swings on ecosystem behavior? - How can we track the flow of scarcity between known and unknown entities over time? - Who are the "gatherers" the addresses that keep stacking BTC no matter what the price does? What’s their impact on scarcity? - How can we project Bitcoin’s scarcity under different price scenarios all the way to 2050 and beyond? I’ve seen some opinions flying around on these topics, but I’m having a tough time finding solid research that’s backed by real-world data models. If anyone has come across relevant work in these areas, I’d really appreciate it!

The system is fine.  If anything, it's overkill.  One of my full nodes runs a Dell mini-pc, i5, 32gb of ram, and a 2TB nvme. Starlink is the best option for one of the properties I own (and lived in until recently.)  Again, no issues downloading the blockchain or running the node once it's synchronized.  Your hard drive choice will have more impact on the sync speed than your bandwidth. There's nothing wrong with Ubuntu, but I prefer Debian.  Debian is lighter, and unlike Ubuntu it's 100% open source.  Ubuntu is built on Debian, so unless it has specific features you need (which it won't for your purposes) there's no reason to choose the more bulky OS.  YMMV.  ETA: On this note; there are many users who are more technical capable than I who insist that OpenBSD is the best OS for a node.  I've played with it, and it's very fast and light.  It does require a small learning curve.  If you're already an advanced Linux user it should be easy to switch. Build a different system for that.  And yes, you'll want a more advanced GPU (or 8) for that purpose.  Or, use the system your proposing and buy a cheap mini-dell off of Ebay to run the node. If you don't have a lot of experience with Linux or Bitcoin, it may be best for you to start with an Umbrel sysetem.  They are easy to set up, can be run on a Raspberry Pi, and one click installation of an SPV server and blockchain explorer.  Once you understand what you need then you can research how to build a node the hard way.

Based from your plan, overriding these settings' default may be needed: "txindex" will enable some features of RPC commands like getrawtransaction which aren't available to nodes without a full transaction index. Enabling it before the "Initial Block Download" will make sure that the database is built while syncing, that could take hours if enabled later."maxmempool" will set your node's mempool size from the default 300MB to your preferred size. This is useful to your user-case to be able to keep most of the unconfirmed transactions since the default isn't enough during "peak" seasons. You may also increase your node's database cache with dbcache=<size in megabytes> for it to utilize more of your memory.

For a node, I agree. For blockchain analytics, I'd say the more RAM the better. @OP: before you do a lot of extra work, have you seen Bitcoin block data? It includes daily dollar prices.

If your software (which perform analytic) could utilize GPU, you should know GT 1030 is old and low-end GPU. So you probably want to get something faster. More RAM could allow faster analytic since you could store more data on RAM rather than accessing it from the SSD. By default, Ubuntu server doesn't include GUI. You probably want to use Ubuntu 24.04 LTS instead. I don't know about machine learning or AI field, but your build should support 4 RAM and 2 GPU. And FWIW, marketplace which rent GPU or high-end computer exist.

Appreciate your insights. I just put the hardware together today, installed the OS, cloned bitcoin, compiled/ran the core and it's now downloading the blocks and verifying. Decided to stick to what I know (Ubuntu Server LTS)... hopefully I don't regret it later, but I guess I could always just rebuild if it comes to that. Thanks so much for this, looks like those will really help me, I put them into the config before I ran the core for the first time and started downloading/verifying I'll look for deals in the coming months, and see if I can upgrade to 64 of 128. Also thanks for the link, I'll check it out. I'm happy to stand up a node to help broaden the security of the network...and I'm sure having it all local will make processing easier, as I'm planning to do some complex indexing and datasets. At least I have ideas in mind... will need to see if any of them pan out. For now, whie I'm waiting for the sync this gives me something to look at I guess I'll find out when I get into the heavy processing. I'm developing from scratch, so we'll see if I get frustrated with the limitations of the system and incorporate a GPU. If nothing else this is my attempt at learning something new and stretching my development skills. I am energized by projects where I can ultilize pervious experience into new problems. Not exactly sure what I would call a problem here, I'm just tinkering with the modeling for now... looking for something that might be of use to everyone as I go. I've developed full stack, and really enjoy the visualizations on the front end. Or making useful interactions with data. Cheers. Appreciate all the help so far. I'm sure I'll have questions soon

The sync was faster than I expected... took about 22 hours. Created a schema tonight for the 10 structures I'm focused on along with an ingest script... sent it to work on the dataset -- looks like I should have this all organized in about 12 more hours... maybe a little longer than I expect, hard to tell because the fist 10% of the chain processed so incredibly fast (a lot less activity in the early years I assume)... I think I'm going to need more innternal storage (2TB)... if my initial schema doesn't grow in scope (it will), I'll run out of psace in about 2.75-3 years - easy problem to solve SSDs are relatively cheap. At least I have some time to figure that out. Getting an ideal GPU is probably next... Cheers! Edit... wow, really slowed down around 25% (I guess that would be 2013ish). Might take a couple days longer than expected

Hmm, I expected that it would only take hours with your build. Have you set a higher dbcache as I recommended in my previous reply? Because its default setting of 450MiB isn't ideal to your machine's specs. If you're currently using the GUI, you can set it in Settings->Options... "Size of database cache". Or in the config file as dbcache=15259 to set to about 16GB of RAM. (requires restart)

I guess the IBD was mostly limited by internet speed. Google's first link is their legal disclaimer: The year is a useless indicator for progress. It's better to look at "GB processed" if you can add that to your data processing.

It's unfair how it's advertised to OP with "150-200Mbps download" while their legal document indicates that it's typically within that range. Anyways, 25Mbps (about 3MBps) isn't too bad for IBD, unless its bandwidth is used by other processes and devices.

You might want to look into upgrading the processor to a 7850X3D, if your motherboard supports it. Currently, it is the best the market has to offer for now. If you're going to be installing a lot of packages from package managers and you don't mind the slightly longer setup time, then I say you should change your setup to Arch Linux. They have the latest versions of everything already in the package manager, so you don't have to fiddle with Pip/Poetry/Npm/Cargo. And then there's the AUR which should cover all the software you need to get including Bitcoin Core. PS. How many disks does your motherboard support? It might be easier to back everything up and then create an LVM with one giant volume after you add those extra disks.

Oh no, I missed that one. Just turned it up to 16GB and it didn't seem to change the speed. I'm on about block 400,000 after clearing the tables and starting over 36 hours ago (I noticed something in my code was failing to capture the addresses properly and everything was being labeled as 'unknown'. I've fixed it, and verified a few other issues with the data gathering. Started everything over. Checking the system, it appears that RAM isn't being heavily used (only 3GB), and the real culprit is Postgres (taking up 50% of CPU - while processing each block in about 2-3 seconds - sometimes it will do 3-5 very quickly). This will eventually catch up, but that's not ideal. Probably not the optimal dB choice due to indexing (I turned them off, but plan to turn them back on after I'm caught up)... I should probably add records at speed and deal with deconficting and adding in the indexes after? Or possibly move to a time series database? I guess this gets into what exactly I want to do with the data... Haven't quite sorted that out yet, I wanted to see it first... I tried removing all my ON CONFLICT statements, but that didn't seem to imrove things. I tried batching, and it didn't change the speed much either. I think this is just a Postgres insert issue. I should find a faster way to dump the data in, probably from a flat file? I don't have much experience with datasets this large, I've usually gotten away with inserts as I go... Currently, I'm at 93m unique addresses, 210m input transactions, and 250m utxos On block 374,000 I'm at 305GB for the postgres table size. At this rate, I may run out of storage, since I believe the first 150k blocks barely had any data to them. Yep, it's a marketing thing. I can say though it worked great for the initial sync. I pulled it all down relatively fast... it did get me to about the 1TB soft 'limit' so I'm curious to see what the throttling will be for the next 3 weeks... so far it hasn't impacted streaming. Indeed, same for block height... which is what I'm printing. Good idea, I'll put that in the log. Thanks for the input, I'll look into those! I can add another (I would probably get a 4TB so I'm not worried, and possibly run a time series and relational database - each for a different purpose). I think I'm all in on this project now, I'm having fun so far.

I count 1,365,198,853 unique addresses (based on last week's data). If that's any indication, you're looking at about 15 times more data.

Maybe significantly more... the last 11k blocks (Fall 2015) added 40GB - at that rate, I'm looking at about 2.5T total, just for this database. This is where I'm at currently...      table_name         | total_size ------------------------+------------  utxos                      | 131 GB  transaction_inputs   | 81 GB  transaction_outputs | 77 GB  transactions            | 40 GB  addresses               | 19 GB  blocks                    | 100 MB  wallets                   | 16 kB

I would not use a Ryzen CPU for this if you want to me dealing with large datasets / databases and searches an EPYC is the better choice if you want to stick with AMD and if you want to go Intel use a good Xeon. Same with RAM, if you are manipulating large data sets to analyze you start with the largest one that you may want to look at, at this point from your last post it's the utxos and double it so you would want about 262GB of RAM. You could probably get away with 256GB at that point it's still not ideal but you would be able to load everything into RAM instead of pulling from the drive and look at it there. If you are going to do it, do it right. I spend a lot of time telling customers 'I told you so' when they try to do things with lower spec hardware for things and they complain it's slow. For a desktop PC having to wait a few extra seconds here and there for some things because you got an i3 instead of an i5 or i7 is one thing. Depending on what you are doing in terms of analyzing this becomes hours instead of minutes. -Dave

Good points. (Un)fortunately this is just a hobby/interest for now, if I get past the exploration phase and want to do real things with the data I will need to scale up. It's definitely not the CPU (23% capacity) that is the bottleneck, or the RPC commands. It's the writing on disk (running close to 80% everytime a block gets to the insert part -- which surprises me with write speeds around 7500MB/s)... I suspect the way I am batching everything for each block all at once is what is causing this speed issue (there can be quite a few utxos per block) combined with the indexing I'm using to ensure data integrity (maybe not necessary now that it's running really stable... I just didn't want partially processed blocks re-writing on restarts). I'm ingesting about 15,000-20,000 blocks a day currently... I may attempt to change this so I add sets of 1000 at a time, instead of inserting the entire block all at once after being read. That would require some logic, as some of what's being added to the transactions table needs to be referenced and linked in other tables - so order of entry matters across the entire block. But at this pace, it'll get done one way or another within a couple weeks. I'm up to block 481,000... and I'm just past 800GB for the database - but on average it's growing about 80GB per 12,000 blocks now (2017 things really picked up). I estimate, based on some assumptions running a test scripts on sections ahead of me, that this will end up being approximately 3.4TB in size when I'm done, so I'm about a third of the way there. I may move some of the tables onto an external once I index the things I'm really interested in. Slow and steady, I'll get there eventually.

Shouldn't disk writes be handled by file system cache? To compare, even on an old HDD, writing 10 million small files (yes I do crazy things like that) is almost as fast as sustained writes. Reading them is very slow, because then the disk head needs to search for each file. Writing is fast, straight from file system cache onto the disk. I'm not sure how this would work with a database, but if writing 3500 transactions takes 4 seconds, that seems slow to me.

Completely depends on the filesystem. Most of them like ext3, ext4 and such use journalling, so when you batch all that data to write into the disk, it actually goes inside the journal first. Usually the default settings of the journal is to write deltas of the changed bytes on to the disk. This is more reliable than just doing a write-back to the disk, but it's slightly slower. You can actually change the filesystem settings to more aggressively utilize the disk cache, but it will only take effect on the next reboot.

I use ext4 with default journaling.

Then writeback mode should make your write performance much better: And you can turn it on like this, on an unmounted filesystem: Run fsck after that before remounting, just in case.