Building a Bitcoin node for data analytics

Hey everyone, I’m looking to actively participate in the Bitcoin network by running my own node. Not only do I want to contribute, but I also want to tap into the blockchain data for various analyses. I’m planning to set up my wallet on this node too. Right now, I’m on the hunt for the right hardware. Is a Raspberry Pi (4 or 5) a decent choice, or should I invest in something beefier if I plan to pull data from the blockchain and perform some analysis? - Should I go for 16GB of RAM or just jump to 32GB? - Is a 4TB SSD enough, or do I need a bigger drive considering Ordinals are making data grow faster on the Bitcoin blockchain? - What other essential hardware should I consider? I’d appreciate any extra tips on this. I’m ready to spend a bit of cash on the hardware if needed, so I’m not just looking for the cheapest option here. Cheers, Schabernack

In that case, you might want to consider to "max-out" the motherboard of your choice. (for futureproofing purposes) Ordinals haven't increased the pace since the maximum block weight is still at 4WU (1vMB) even after TapRoot upgrade. Maybe a bit because of the more consistent full blocks that's caused by the large transaction volume that we're experiencing since then. Refer to this chart: https://ycharts.com/indicators/bitcoin_blockchain_size At that pace, your 4TB SSD should be sufficient enough.

May I ask what exactly do you want to do? For personal reasons? Prefferably aim for: 8GB RAM 2TB SSD Anything higher than that is good but not necessary / mandatory.

Honestly it comes down to which data analysis software you use and additional software you run (for example, self-hosted block explorer). Current blockchain is size is about 533GB[1] and maximum possible growth per month is 17280MB (4MB * 144 blocks/day * 30 day). So 4TB should be sufficient. I'd say CPU assuming data analysis software you use support using multiple core/thread. [1] https://blockchair.com/bitcoin

Hi guys, first of all thank you all for your feedback, including details and links with further information! To answer your questions, I will break down my answer into several points: 1.) Data Analysis -I want to run analyses to "explore" and "visualize" the Bitcoin ecosystem (some data providers do not provide the raw data that I intend to use). -There are obviously data providers that I could use but I want to make sure that 1. I always have access to the data 2. I familiarize myself with interacting with Bitcoin on a "data-level" 3. I have access to data I need for my analysis and on a high level of granularity -Initially, I only intend to download data, store it in tables, and use Python to run further analysis. This might change as time goes by. 2.) I will run the node and use the data for personal reasons (being part of the Bitcoin network, research interest) 3.) I was actually considering to use a Raspberry Pi as it is very compact in size and easy to store somewhere in a room. However, I am unsure about how well it performs when I want to do some data analysis. That is why I want to use a device with more performance so that I can host the node and conduct further activities on the device simultaneously. I will try to get a decent CPU and sufficient RAM. As for now, I will use a 4TB SSD for storage. Regarding the computer, I have one more question: which OS is the "best" to use? I am a Microsoft user and I am aware of its vulnerability compared to other OS. Is it still a valid option to use, even if I also want to use my node to host my own wallet, or are the risks to high if I use Microsoft?

I have multiple headless nodes on a few Raspberry Pi 4B with 8GB RAM (RaspiBlitz, RaspiBolt, Umbrel). All work fine and are nicely power efficient, especially when you run them 24/7/365. While such nodes like RaspiBlitz or especially Umbrel abstract the underlying Linux from the user, it's still beneficial to have or learn some Linux knowledge. You can do it all on Windows, too, but frankly I would rather choose a Linux base. Another suggestion is to get a used business laptop with 16GB RAM (or option to upgrade to it) which are usually not very expensive. On my ThinkPad T540 I run Ubuntu and another Bitcoin node (no, I'm no Bitcoin node addict, I think). While idle (Bitcoin Core runs still in the background) and no active display this laptop consumes on average somewhere in the ballpark of 11-12W (about double of a Raspi Pi 4B).

Since you only mention Python, Linux probably is better option since Python comes with many Linux distro and you could allocate more RAM and CPU for your Python script since Linux use less RAM/CPU.

Thank you again very much for the information! Right now, I am thinking about running two nodes on two devices: 1. Windows laptop (as of now, easier for me) Is a setup including a 16GB RAM and Intel Core i5 (12th or 13th generation) sufficient to run a node and do some data analysis (Python, SQL) simultaneously? I know it is not easy to answer such a question, but I would appreciate your opinion. Can someone please tell me if the following idea makes sense: I will set up a node that I will only use for actions related to interacting with Bitcoin. Ocassionally, I will download some data from the blockchain and store the data on another external harddrive. Then I will use this harddrive to transfer the data to my current laptop (which has a good performance) and do the data analysis on my current laptop. Or can I simply disconnect my SDD (where the blockchain data is stored) from my node for a few hours and download the blockchain data directly to my other laptop? Will it cause problems if my node is offline for several hours or does it just need some time to synch once it is online again? 2. Rasperry Pi I am eager to run one node on the Rasperry Pi for the previously mentioned reasons and to get more familiar with Linux. Accumulating knowledge and skills will take time but I am willing to invest the time.

Bitcoin nodes can be shut down and then they will continue exactly from where they stopped. But of course they need to be terminated gracefully! Why do you need to transfer the SSD from one computer to the other? You want one computer for the node and one computer for the "analysis"?

Before doing all this for data analysis, have a look at these topics:

In my view you should try both options and select the most optimal one which depends on that how resource-avid  is your your data analysis. If you current laptop   has a juice to do both jobs simultaneously, i.e. to support the copy of  blockchain on your drive in sync state and at same time perform data analysis then there is  no need in the second  laptop. In any case don't forget   to have txindex=1 in your conf file so that you could analyse all transactions rather than your own ones.

Probably yes. You can configure how much RAM used by your SQL database. And after you download all blockchain, you can setup Bitcoin Core to use less RAM as well without much problem. Just don't try to load all data to RAM at once. Yes, assuming you use Bitcoin Core to download blockchain data. It's the latter. I expect Raspberry Pi will struggle for data analysis.

I second what is said above: Don't use Raspberry Pi. My Pi 4 was overheating constantly when running a node and I had to get active cooling on it. It is okay to remove your data directory, provided that your Core has shutdown properly and is not writing any data on it, or it'll get corrupted and you'll have to do a reindex again. You might just want to run everything on a single computer instead; indexing the transaction data and parsing them to another format can take quite a while and external harddrive are slow, and are prone to corruption.

Out of curiousity, what where you running on the Raspberry and how much RAM did it have? I run 2 Raspberry Pi 4b 8GB RAM and I never had overheating issues. I use this case on both of them and the temperature never exceeds 58-60 celcius. It has passive cooling and there is no noise at all.

I think I ran it with my RPi 4, 4GB. How were your temps looking during the synchronization? Mine was fairly hot throughout, granted that it was just a heatsink and not a fancy heat dispersing case. Not to mention, it took weeks.

What OS did you run? Mine is downloading the monero blockchain now and the temp is currently 43.8 celcius. I don't remember exactly with Bitcoin but I can tell you for sure that it never went over 58-60 celcius. But RAM is very important for this. So 8GB work much better than 4GB.

Raspbian. Mine was above 70 for most of the time, but again your cooling method makes a huge difference. Having the entire casing as a heatsink is undoubtedly better with what I've had on hand. RPi4s were notorious for overheating actually, might've been the issue with the earlier versions. RAM is important, but RPis are pretty expensive so I had to skimp on it back then. RAM shouldn't affect temperature anyways.

To be honest I have created a node as a series of tutorials for Bitcointalk here. This node runs Raspbian Lite and to be honest the temperature went close to 65-67 degrees during the IBD. The other node that I have runs Start9 and the IBD went very smoothly and much faster than the other one's. I used to have Umbrel too and I don't remember any issues too. I just switched to Start9 because I loved its simplicity and the fact that it is much easier  to customize.

The thing burns 6.4W max. That shouldn't be too difficult to dissipate with passive cooling.

On a fairly compact surface area and my aluminium heat sink was apparently not enough, the earlier versions were at 7-8Ws and were known to overheat. Might be better to get a bigger heatsink and a better thermal pad for it.