Hey folks,
I got curious after reading some papers about how Bitcoin blocks get propagated, so I set up a little network to measure live block propagation on the mainnet (Atlanta / Frankfurt / Singapore).
I've checked out over 2000 blocks so far, and I've noticed some pretty interesting and consistent patterns in how they spread (still going through the data). But there's one fork that really stands out, and I'm trying to wrap my head around what’s going on.
This fork happened at height 928484, involving GDPool and AntPool.
From what my probes indicate, GDPool’s block was visible at all locations about a minute and a half before AntPool’s block showed up. At first, I thought I was missing something, so I spent a few days digging deeper.
I started to think maybe AntPool’s block had something about it that made it more favorable to the network, like a higher weight or that it reached more nodes through a different path before my probes picked it up, even with that big timing gap. But then I looked at the next block, 928485, and it got even weirder.
When I checked my database for 928485, I found that AntPool’s block from 928484 and its next block at 928485 appeared to my probes almost simultaneously (within ~10ms across all locations). As I understand it, when a block is mined on top of a competing block, the longer chain is picked, no matter which block showed up first, since there’s no universal time authority.
So, to sum it up, GDPool’s block came first, but AntPool’s block...
What matters for ultimate selection is what hashpower sees, not what 'nodes' see. So monitoring many nodes may not give you a fair assessment of what actually matters. This is made more likely by the fact that good operational practice is to not accept incoming connections directly on the node controlling mining in order to protect it against inbound DOS attack, so the nodes that matter most for your measure are ones you literally can't connect to.
Also,
Consider a race between A and A'-- if the nodes you are connected to see A first then they suppress transmission of A' by virtue having already accepted A. However, in realty A' might actually be preferred by significant hashpower while the dominance of A exists mostly just in what you are monitoring. In any case, even if your view is fair A' will still probably be the first block seen by at least the author of A' and so the probability of it being extended first is non-zero.
Later block A'B is mined, at that point it becomes the preferred tip and all the nodes that previously preferred A will now switch to A'B and relay A' (parent of A'B) to you first. So what you see is A' then A'B at once, but in reality A' could have existed in public for a long time, it just wasn't making it to you.
So the success of A has suppressed your visibility of A' until the chain with A' overtook. It's also possible that the author of A' withheld it, but that is a needlessly complicated explanation for an otherwise expected observation.
Unfortunately ever since compact blocks (particularly the use of cut-through announcements) well relayed blocks tend to block the knowledge of competitors pretty effectively. Prior to it, validation time was in the critical path at every hop and network latency meant that close announced competing tips tended to be more visible.
I've previously suggested a protocol addition where each node would announce just the header of any POW valid block that connects 'near' the current best tip, even if they haven't accepted the block, don't have the block, and/or don't know where to get the block-- just to improve the visibility of races/forks/stale-blocks. This needs a new P2P message, because the existing blocks/headers messages mean that you have the block and are able to share it. Fortunately it would be a very trivial change needing only changes in the node software and not the rest of the ecosystem.
I think your kind of monitoring will inherently be very limited and somewhat misleading without that kind of protocol addition-- particularly because I don't think you can distinguish a withholding attack (which would be very interesting!) from the natural suppression of near-ties without something like that.
Going a step further it would be useful if miners published and the network shared header messages for attempted blocks meeting e.g. 10% or even 1% of the difficulty target. This would give realtime visibility in to what forks the network was attempting to extent. Such an addition though would require changes to mining software in addition to nodes, so it would be inherently much harder to deploy and people may want to extend the mechanism to help improve block propagation (by using it to pre-announce likely to be mined transactions)-- so unlike the alternative-chain-headers proposal would be likely to suffer scope creep.
Unfortunately the bitcoin community seems to have lost sight of the goal of maintaining a healthy network and instead prefers to navel gaze and debate paid provocateurs about the merits of altering the system to steal coins, censor transactions, or other such nonsense. So unless you feel like doing that work yourself, I wouldn't suggest holding your breath.
I am going to take that comment a step past there as I have before.
For you or me or just about anyone to run a node is not a big deal.
For people running mining pools there is a lot of extra security involved. Or at least there should be....
Including, checking across multiple nodes in different locations.
AND
making sure they are well secured and protected.
Nobody wants to talk about security. But quoting myself from years past on a different issue with some odd mining things.
So Antpool may have not even fully seen the block that they orphaned.
People think because their home 'firewall' can move data at a certain speed anything can.
But proper security and filtering takes real time.
Edit the next AM to add: When I say "takes time" I am not talking minutes but multiples of seconds. That can stack up quickly leading to longer times approaching minutes. I can at home download and open a 5 meg PDF somewhat quickly sa in under 10 seconds. I have some corporate clients that between the click to download and the full open on the desktop is 45+ seconds.
The flip side of that is when someone deliberately tried to send them some very very sophisticated malware. 2 of the 5 things it had to pass though stopped it. Never would have opened after the 1st one, but it still passed through the rest of the scanners anyway to keep checking. But doing things like this and then cross verifying will take time.
-Dave
Wow I kind of wish I didn't post this on the holidays so I could have gotten back to it a bit quicker.
I needed some time to digest all of that but that's why I'm here so let's get into it.
Regarding The Fork
This is an interesting insight. It's also actually supported by the data I've collected so far because I've noted that, from the vantage point of my probes, GDPool has a tighter distribution profile than AntPool. Which would imply that if, as you say, public nodes can only request peers on what we might say is the "public boundary" implying that miners inject block broadcasts at layers beneath these and propagate upwards, the probability of a partition happening where A (say from GDPool) and A' (from AntPool) where injected at nearly the same time (say within ~3-20ms apart) then it is very possible that all of my probes would see A first.
However, I should mention, I'm timestamping blocks in and classifying them based on the msg_type chosen. Meaning that I also capture all of the broadcasts from each peer as they come in.
For each block from my 3 probes I typically get 24 entries in my database and for this specific incident all 24 were partial to A before A'B was captured ~90s later.
Which I just mention to say, that if that's true, it means that A' was stuck at a deeper layer because A blocked all of it's exit points to the public nodes layer before it could escape. Alternatively, it may have actually made it to a few public nodes, but it must have been a limited segment of public nodes that I don't currently monitor. If the latter, then I could potentially have captured that if I added more probes to different regions to cast a wider net to get the required resolution to witness an event like this. Otherwise, as you said, it is invisible unless the protocol is modified because it has not surfaced to the public layer (and that is unfortunate).
I would wager though that odds are, both scenarios happen. So, I might be able to capture some fork partitions some of the time if I cast a wide enough net. But unfortunately, I also don't see how this could reliably tell us anything about intentional withholding.
In my defense, I wasn't originally really trying to specifically do this anyway. I just wanted to look for miner propagation invariants and evidence is mounting that I might have found them. Although, ultimately I was hoping the insights would be useful outside of my own curiosity. Of that I'm not so sure yet.
This is actually relevant to another assumption I made about a different stage of my analysis which was based on this quote from a paper:
I had interpreted their results as that outbound propagation and inbound propagation was symmetric. But, if inbound propagation regularly is required to pass through security steps that outbound doesn't. Then obviously that can't be true. Now I'm realizing it could also have been implied from the partitioning that gmaxwell was talking about.
At this point I have to wonder how it's even possible to implement intentional withholding, especially at the typical global hashrate that exists in present day.
The P2P Protocol Addition
Well, frankly, this work is my attempt at doing something useful that also plays to my strengths to enter Bitcoin. I'd be willing to attempt a PR if you would be willing to share the details of the change in more depth.
Quick question though, even if a patch was written - if there were miners intentionally withholding - wouldn't they just skip this version to remain undetected?