Is it necessary for -dbcache to be a multiple of 1024?

6 replies 406 views
alt21Senior Member
Posts: 398 · Reputation: 1732
#1Nov 4, 2023, 08:24 PM
I came across this topic in another discussion and thought I'd clarify things a bit. So, should -dbcache be set to a number that's a multiple of 1024? This is a pretty common misunderstanding, and I figured it needed some clarification. Trying to set it to a value that’s a multiple of 1024 isn’t really necessary. You can actually set -dbcache to any integer value without any issue. Whether you go with -dbcache=8000 or -dbcache=8192, it doesn't really matter. Memory is divided into pages, usually 4 KiB each. When a program requests memory, it typically gets smaller 4 KiB chunks instead of one big continuous block. So, when you tell Bitcoin Core you need 8000 MiB, here’s what happens: 8000 MiB equals 8,388,608,000 bytes. 4 KiB equals 4096 bytes. That means you’ll have 8,388,608,000 divided by 4096, which equals 2,048,000 memory pages allocated for Bitcoin Core. Now, let’s break down why any number you choose for dbcache won't cause any issues. dbcache is just an integer measured in MiB (mebibytes). 1 MiB equals 1,048,576 bytes, which is 2^20 bytes. Each memory page is 4 KiB, or 4096 bytes, which is 2^12 bytes. So, when you divide 2^20 by 2^12, you get 2^8, which is 256. This means 1 MiB equals 256 pages of 4 KiB each. So feel free to pick any integer for -dbcache. Let’s say you choose n. Then, the number of pages will be n MiB, which is n times 1,048,576 bytes. Since 1,048,576 is divisible by 4096, any n you choose will keep that divisibility intact. Bottom line, you can use any number for -dbcache without any concerns. If you're curious about the right size for dbcache, there’s a solid article here: https://blog.lopp.net/effects-dbcache-size-bitcoin-node-sync-speed/, but in general...
3 Reply Quote Share
im_lynxHero Member
Posts: 515 · Reputation: 2161
#2Nov 4, 2023, 11:51 PM
I agree that it doesn't matter what value of n you specify for dbcache=n, even when I don't exactly know how Bitcoin Core actually allocates memory for such caches. But 4 KiB pages aren't the only size that most CPU and OS architectures support. There's usually also something called large or huge pages, because having to manage only a fixed small size like 4 KiB isn't efficient in certain cases. Maybe have a look here to dig deeper, for those who are interested: https://en.wikipedia.org/wiki/Page_(computer_memory)
2 Reply Quote Share
alt21Senior Member
Posts: 398 · Reputation: 1732
#3Nov 5, 2023, 02:44 AM
Absolutely, and it's good that you mention it. In fact, I wanted to write about it myself, but I thought that since typically 4 KiB is the typical situation, I should expand my thesis based on this assumption. As far as Bitcoin Core is concerned, I 'd love to hear some input from someone who knows, but I know that in general the amount of dbcache that you put in the argument is the total cache that you want Core to use and then the program splits it in smaller batches and uses it for the chainstate, the UTXO etc.
0 Reply Quote Share
Posts: 11 · Reputation: 151
#4Nov 5, 2023, 07:12 AM
we know that most systems use 4 KiB as the default memory page size, but typically of 2 MiB or even of 1 GiB in size, they also support large pages (also called huge pages) large pages reduce overhead of managing many small pages and improve CPU cache efficiency(as u said )  Bitcoin Core does not request explicitly for the use of large pages when allocating memory for dbcache it uses normal memory allocation methods such as malloc and new and it relies on the operating system’s default behavior that uses standard 4 KiB pages bitcoin Core would need to perform special operations in the event that it wanted to use large pages (like mmap with MAP HUGETLB on Linux, or VirtualAlloc with MEM_LARGE_PAGES on Windows) and mainly due to the fact that it does not voluminous pages require additional system configuration or admins configure Certain fragmentation or failure issues can occur when managing a number of large pages this can be caused by managing of large pages in a dynamic way Bitcoin Core needs to be simple,stable and portable across many of the OSes Bitcoin Core will allocate memory in a safe and efficient manner, regardless of the value you set for dbcache=n as a result each integer n that is for dbcache still will align in a neat way to the page system, without any waste or inefficiency, due to 1 MiB = 1,048,576 bytes = 256 × 4 KiB pages, setting dbcache to a value that is divisible by one thousand twentyfour is not at all necessary each integer is perfectly fine
0 Reply Quote Share
silentchainHero Member
Posts: 473 · Reputation: 2317
#5Nov 5, 2023, 01:28 PM
In fact 450 MiB is obviously divisible by 1024. 450 MiB = 450 x 1024 KiB. 450 MiB/ 1024 = (450 x 1024) KiB/1024 = 450 KiB Or 450 MiB/ 1024 = 450 (1024)2 Byte/1024 = 450x1024 Byte = 460800 Byte Core developers has highlighted this in their doc saying that dbcache parameter is specified as MiB(1024) indicating that memory must be measured in binary units rather than in decimal ones.
0 Reply Quote Share
alt21Senior Member
Posts: 398 · Reputation: 1732
#6Nov 5, 2023, 03:14 PM
You are right, I will change it in my OP. Because, actually, any number can be put in the dbcache option. Any number is divisible with 1024 since it's measured in MiB.
2 Reply Quote Share
hash_bossLegendary
Posts: 1166 · Reputation: 5261
#7Nov 6, 2023, 06:51 PM
I remember those stuff sometimes mentioned on discussion about docker and virtual machine. But AFAIK it's not enabled by default on some OS and linux distro. Although when it's enabled on Linux, some application can take advantage of it without additional programming.
4 Reply Quote Share
?Reply
Sign in to reply to this topic

Related topics