Lately, I've been trying to get past just the "Bitcoin price is going up" mindset and really dive into the mechanics of how it all works.
One topic that's been a bit of a puzzle for me is the mempool and the whole transaction prioritization thing.
Initially, I thought it was straightforward.
But the deeper I go, the more complex it gets.
I’ve got a few questions that I’m genuinely puzzled about, and I'd appreciate any insights from those who’ve run nodes or really studied the protocol up close.
1. Is the selection of transactions purely based on fee rate?
People always say, "higher sat/vByte gets priority," but is that the complete picture?
Do miners just line up transactions by fee rate, or are there other factors in play, like:
[]size of the transaction
[]how old the transaction is
[]dependencies between transactions
[]mempool policies
or maybe even private agreements?
For instance, can a miner choose to add a lower-fee transaction first for some specific reason?
2. Do all nodes share the same mempool?
I was surprised when I discovered this part.
I used to think Bitcoin operated with one massive global mempool that everyone accessed equally, but that doesn’t seem to be the case.
Now I’m curious:
[]can two nodes have totally different contents in their mempools?
[]can one node notice a transaction while another one hasn’t received it yet?
When there's congestion, do nodes begin to drop different transactions independently?
If that’s the case, then the mempool is more of a local perspective each node keeps rather than a completely synchronized global list.
3. What really happens to low-fee transactions when there's congestion?
Let’s say I send out a transaction with a low fee...
Your post would probably have been the best decision for the "Bitcoin Technical Support" board. Because every question in your post is directly related to the discussion on the "Bitcoin Technical Support" board.
Feerate already accounts for the size, as it's the amount of sats per vbyte and this is the most important metric to miners, they are there to maximize profits at the end of the day. They can independently include or exclude a transaction from their block but the common occurrence is the tx with the highest feerate getting in first.
Mempools arr fairly standard from node to node. Each node can have different settings so the transactions they hold may not always be exact, but several settings/policies are standard so it's unlikely for 2 nodes to have completely different transactions.
How txs are dropped too depends on the nodes settings, it's usually set at 2 weeks but it can vary with some nodes.
Mempool is very much synchronized, if it weren't txs will not have any order in how they are confirmed.
This depends on the general setting, it'll usually sit in for about 2 weeks with most nodes after which it will be dropped but it could also be dropped if most nodes go above their size limit and have to include priority txs. It can also be rebroadcasted at anytime and remain in the pool indefinitely.
I'll return to the other questions in a bit if they've not already been answered by that time.
Of course. For example: https://mempool.space/testnet/tx/2703e6d0215261d4916e0436438a366f4198fe473ff856036e01306ddce20315
This miner included his zero fee transaction first, before other transactions.
No. Each node has its own mempool.
Of course.
By default, it is dropped after two weeks. However, any node can decide to store it forever, and broadcast it later.
It means that a given node can refuse to drop it for whatever reason (for example because of using the old software).
If you use the latest version, then transactions with lower fees can be replaced by bigger fees. Which makes it possible to "unstuck" it.
Only by those, who upgraded their code.
Yes.
Of course.
Nothing, because after reaching a single confirmation, everyone will know, which version was placed in a block. And all new blocks on top of it simply couldn't include a second version, without invalidating that block.
This is the whole reason, why miners are needed in general. Because Alice could create two transactions: one sending coins to Bob, and one sending coins to Charlie. She could broadcast different transactions to different nodes. And then, whatever miners include first, will be enforced in practice.
Only if transactions are independent. If you have Alice to Bob transaction, and Bob to Charlie transaction, using Bob's coins, received from the previous transaction, then they have to happen in order. But if they are completely disconnected, then miners can swap them as they wish.
Of course. You can send a transaction only to a given miner, and nowhere else. If that miner will include it directly into the block, then the rest of the network will get it, only when that block will be mined. This is how people get non-standard transactions confirmed, or solve transaction puzzles, where revealing the public key would allow moving the coins by someone else (because of weak N-bit private key).
Yes. They just create a template, where they pick some transactions, and then they constantly replace some transactions with others, to get more fees out of it. But in general, mining is NP-hard, and they solve the knapsack problem here.
It was the case in the older versions: https://en.bitcoin.it/wiki/Miner_fees#Priority_transactions
Now, it is counted in satoshis per virtual kilobyte.
once the miner includes your transaction in the block and broadcast that block, how does the rest of the network react if the transaction was never in the public mempool? Do nodes accept it cleanly as part of the block, or do you miners see it sometimes it as trn- mempool -conflict or other validation issues ?
Once a node receives a new block, it check whether the block is valid or not. If the block is valid, the nodes adds it to their copy of the blockchain.
It doesn't matter whether the node had the transactions in their mempool or not.
Also note that each node has its own mempool and therefore there are always transactions that nodes adds to their copy of the blockchain even though they never had them in their mempool.
By default, they accept the new block as valid, and reject all conflicting transactions, which they had in their mempools. Note that the coinbase transaction is also never in the public mempool, but it is normally accepted as valid.
Of course, some miners can try to reject a given block, but then, they usually need two blocks to do that. Because blocks are accepted, based on first-seen rule: if you produce a different block on the same height, then you need a second block on top of it, to trigger chain reorganization.
And also, sometimes two blocks can be broadcasted at almost the same time (and they can contain different transactions), and then, the next block will show everyone, which chain is the correct one.
I feel like you already know the answers to some of your questions considering some of the questions are related to the answers to the previous ones.
Confirming already researched subjects I guess.
Anyways, I'll just add some notes to these parts with keywords to DYOR:
It doesn't have to be the same transaction, the replacement can use only one of the input of the original transaction, it can also replace all of the outputs set by the original transaction.
RBF is taking advantage of the "UTXO model" that Bitcoin uses:
Since it can only be spent once, a new transaction that spent any of the UTXO used as input by an unconfirmed transaction will make it its "replacement" whether there are originally 10 other inputs there.
Of course, it still depends on how both transactions will propagate and which one will be included to a block first.
And yeah, it's a decentralized system so you can't just force everyone to follow a certain policy.
Most of the standards are set by default, but node operators can still change it through either their config, using different client or changes in the code.
This is exactly what those "Transaction Accelerators" are.
You may not heard of it since it's not famous today with the implementation of "full-RBF" and overall lower average fee rate;
But in the past, people are getting in line to just use such free service if they can't bump their transaction.
(not to be mistaken with those rebroadcast services that advertised themselves as transaction accelerators)
That's a good interpretation, this is why we have full-RBF/opt-in RBF and it's enforced by the majority of the nodes.
I do hope this my simple way of explaining it to you makes your understanding better. Its when your transaction get stocked probably due to a very small that u decide to pay during the process of initiating the the transaction. obviously miners will not pick your transaction to broadcast it and add it to be next block there for and if you dont want your transaction stocked for long you have the option of Replace-by-fee RBF and also Child pay for parent CPFP
Its just about how much fee youre paying to attract miners actually, because Miners always priorities transactions with higher fees. Its just a simple logic, I want the work i start which got stocked because I didnt pay enough fee for the labor done now, Ill need to pay more fee to get it finish.
Yes
Every node is an individual entity even though there is a consensus in it protocol when it comes to transactions fee each node decides what fee is okay for them.
What origin way? you mean paying a higher fee from the initial transaction? If that was done there wont be stocked transactions and there will be no need for RBF
AFAIK, in early‑age Bitcoin the reason miners included low‑fee transactions first was the coin‑age priority system. There was even a priority parameter that calculated UTXO age, but it stopped mattering for miners a long time ago. I dont remember exactly when, but its no longer used. The reasons miners may include the lower‑fee transaction today might be 1)transaction is their own, or, 2) someone made an agreement with them.
I think the question is actually about if someone does an RBF if the miners will actually drop the first transaction (original transaction) and replace it the new one, or they will decide to keep the original transaction. Actually the nodes have their own right of replacing or not replacing new transactions in their mempool, but one thing is most nodes are now full RBF enabled which means they accept transaction replacement. This is even logical since they most prioritize transactions base on fees
But lets say for instance a node decides to keep the original transaction and not replaced it with the new then and then this node actually gets to mine a valid block with this transaction in it, other nodes will just verify if the block is valid and then add it to their blockchain and consequently remove that RBFed transaction from their mempool because its has already been invalidated since its input has already been spent by the original transaction.
So there is nothing actually wrong in nodes keeping hold of the old transaction but in reality the new transaction with higher fee is definitely going to be confirmed first due up fee priority and it will render that old transaction invalid
Yes, but if they got to know the newer RBF replacement transaction(s), then it doesn't make economical sense to not mine the replacement!
IIRC, the RBF replacement has to pay absolutely more transaction fee than the replaced transaction (and if I think about it, it's very likely also true that the fee rate of the replacement transaction has to be higher than the replaced one). I find the fee adjustment rules of RBF not quite straightforward. It's complicated from memory and I admit, that I don't have every bloody detail of RBF in my head. Please, correct me if I'm wrong!
Therefore a miner has an economical incentive to mine the newest replacement transaction instead of the original transaction or any older replacement (a transaction can be RBF replaced multiple times as long as it's not yet confirmed).
This is true, but it's straightforward.
It has to pay for its own minimum relay fee and all (up to 100) of the would-be-evicted transactions' absolute fee including the fee of the one that it's directly trying to replace.
That "up to 100" are the will-be-replaced transaction's children if it has any.
The required fee goes way higher than it seems in this case, probably why you think it's not straightforward.
Not always. They do it by default, but a miner can accept 0 sat/vByte transactions if they wanted to, and there are such cases in the blockchain. Even if fees were to rise and you needed to pay 1000 sats/vByte for your transaction to be confirmed, a miner could include their 0 sats/vByte transaction into its block instead of yours and any other transaction paying less than yours. It doesn't make sense economically but it can still happen.
Minimum fee rates aren't part of the consensus rule I remember reading once. The "Laws of Bitcoin" don't force nodes to only accept transactions above a certain fee rate. Theoretically, nodes can decide to reject all transactions paying less than 10/100/1000 sats/vByte, but the question again is where is the economic sense in that?
He didn't say "origin way." He said "original anyway." The question was about if miners can ignore replacement transactions and still mine the first (original ones) their nodes picked up. They can. Maybe the replacement transaction didn't propagate to their node by the time they mined their block or they reject RBF transactions. Either way, if the original transaction is valid and gets included in a valid and confirmed block, other nodes will accept it as such, even if their mempools added the replacement transaction.
The over-arching principle is that a miner has free reign on which transactions he/she wishes to include or not.
You will occasionally see a zero-fee transaction as one of the first transactions in a block. This is, most certainly, a TX that was manually added to their block template construction, as 1. it would not have made its way into their mempool via standard relay policy, and 2. a miner has no economic reason to include a zero-fee transaction in a block.
Also, if you take this to the extreme, a miner can include any transaction that is valid according to Bitcoin's consensus rules.. Consensus is much more permissive than policy. There are only a few output script types that are considered "standard" by default policy, as well as plenty of other policy-side impositions, such as transaction sizes, scriptSig size, number of sig-ops, permitted opcodes in the scriptSig, witness stack size, etc...
Part of policy is having a minimum relay transaction fee, i.e., the minimum fee rate a transaction must pay in order for your node to relay it. You can tune this in your node's bitcoin.conf configuration file via the minrelaytxfee=... parameter, and if you're a miner, you can tune the min. TX fee rate of TXs you are willing to include in your blocks via blockmintxfee=....
Something else that is related to what you're asking is that some pools have private mempools. I.e., they have mempools that are purposefully isolated from the rest of the network. This is probably the case of MARA Slipstream, for example.
You can easily do something like this yourself on a single machine. Set up node A that is the node you mine to with your desired relay and mempool settings. Then set up node B and set blocksonly=1 in your configuration file - this makes node B unable to relay transactions: only blocks. Then, use connect=localhost:NODE_B_P2P_PORT in node A's settings to set it to exclusively connect to node B, and not the rest of the network. Now, you can submit transactions to node A without them being relayed to the rest of the network, and have them included in any blocks you mine.
The full story is that starting from v31.0, Bitcoin Core replaced the old mining paradigm which was looking at every single transaction and grabbing those with the highest feerate first with a new cluster model in which every transaction in the mempool gets assigned to cluster. Every cluster gets the fee-per-byte score (total fees of the group divided by total cluster size), and the miner picks clusters in score order, highest first, until the block is full.
v31.0 established hard caps for a single cluster: 64 transactions with a maximum size of 101 kB. A cluster may contain just a single transaction if the latter has no parent/child relationships in the mempool.
You've got that jumbled.
Transactions hadn't been considered one at time for more than a decade. Making a one at a time decision doesn't let you do a child-pays-for-parent.
The prior algorithm scored every transaction based on on the total fee per weight of the transaction itself and its unconfirmed parents, and then selected the best score to include. If a transaction is picked that has parents its parents come with it (obviously) even though they would have been lower in the ranking. In the case of simple linear parent-child chains this results in the correct fee optimizing behavior.
But in the case of there being multiple children that share parents or other more complicated graphs of transaction this algorithm would make poor decisions that failed to maximize income. The algorithms performance also meant limiting the number of dependent transactions to chains of at most 25.
The new approach first groups transactions into clusters of parent child interrelated transactions. All the transactions in each cluster cluster are put into a topology obeying order (parents before children) where the income per weight is maximized if you take an random portion from the beginning. The the block is constructed by going iteratively most profitable beginning of cluster and cutting to the next most profitable cluster when it becomes more profitable than what remains. It doesn't just take a whole cluster at a time-- that would result in very poor selections when e.g. there was a very profitable transaction with a lot of worthless children.
Interestingly, the same "put interdependent things in the most profitable order" occurs in the mining business. I don't mean bitcoin mining but mineral extraction: it's the same question you have to answer on whats the most profitable digging order in an open pit mine, given you can't extract a unit of earth without extracting the units above/around it. Bitcoin core devs developed a new algorithm for this problem that in practice exceeds the performance of the ones described in the literature.
Beyond better maximizing miner income and avoiding cases where users are surprised when less profitable transactions get mined over their more profitable ones this new approach also makes transaction eviction and replacement work in a more sensible way. Eviction is just the reverse of mining-- you evict the last transactions mining would select. And replacement works by considering the cluster with and without the replacement and asking which one has a better 'area under the fee vs weight curve'.
The new approach also makes block construction much faster because computationally expensive part of putting clusters in a good order happens in advance as transactions come in or are confirmed.
In any case the sizes of clusters are indeed capped, but the new caps are I believe unconditionally higher than the old dependent-transaction limits owing to the new approach's greater efficiency.
Knowingly and willfully I used the fast-track mechanism when referring to the old model because my post focused on new scheme, which is correct by all measures, though unlike yours presented in laconic manner since it was supported by direct citation from the 31.0 release.
Anyway, thanks for your lengthy commentary which explains in details that citation from 31.0 release. Sorry if I misled you in any way.
I didn't intend to offend-- but I responded at all primary because you said "and the miner picks clusters in score order," which made it sound to me like whole clusters go in, which would be a significant misunderstanding to anyone else who read it that way and didn't already know otherwise!
In the new Bitcoin core cluster-based block template construction algorithm, i will like to know how are the clusters of interdependent parent-child transaction precisely defined and also identified, i mean like what constitute the boundaries of a connected component in the transaction dependency graph? also i really dont know what specific algorithm is used to compute the topology- respecting linear order (parent for children ) that actually maximize the prefix-per-weight and how it compare in efficiency to standard topological sort?