What's up with pools forking their own blocks?

I noticed something odd recently on a site called fork.observer. There are a few instances there showing when a mining pool has forked its own blocks: My node didn’t catch the earlier fork, so I can’t give much more info. Also, this particular fork didn’t show up on my node either. But the last fork did get picked up by my node. I checked the debug.log and saw something interesting: The two announcements for block 912722 came just a second apart on my node. What threw me off was seeing a second UpdateTip event for block 912721, quite a while after the first one. I’m curious why this happens to pools. So far, I’ve only seen it with Foundry USA and ViaBTC. Any thoughts or insights? From my simple perspective, I figured that once a pool finds a block, they’re the first to know and start mining on top of that block right away. If they happen to find another block quickly, they could have two blocks secured. But forking their own block doesn’t really click for me. I’m not really an expert on mining pool mechanics, so I’d love to understand why this occurs and why pools don’t try to stop it.

My speculation is those pool run multiple servers either for redundancy or allowing miner on different region to have lower latency/higher bandwidth. Node on those server immediately broadcast mined block, without checking whether other of their server have also mined block with same height first.

When 2 users of a mining pool find the same block number at the same time, one has to fork. The second block 912721 minutes later was not a new block, it was the new best known to your node at that moment after block 912723 removed your block 912722 as best block.

Short answer: it isn't intentional and there's nothing "mystical" going on – you're seeing race conditions inside a very large pool. A pool is not one machine. Foundry/ViaBTC have many stratum servers, proxies and miners spread across the world. When one of their miners finds a block at height H, the pool has to build a new template whose prev-hash is that block and push a clean job to all connected miners so they stop working on the old tip. Neither step is instantaneous. It takes tens–hundreds of milliseconds to build/distribute a new job, and some miners/proxies are a few hundred ms or even seconds away over the network. During that window a different miner from the same pool can still be hashing on the old job and may find another block. Now the pool has two blocks at the same height. Whichever propagates better wins; the other becomes an orphan. From the outside this looks like "the pool forked itself". Why you didn't always see it on your node? Probably because very short-lived blocks get eclipsed before they reach all peers. You only see the block that reached you first; fork.observer aggregates data from many nodes and can show blocks that your node never received. In your log you have two UpdateTip lines for the same height with different hashes – that's exactly a one-block reorg caused by two competing blocks (both tagged to the same pool). Pools can't really totally prevent this, they can only reduce the probability by doing things like: Push "clean_jobs=true" immediately after a new tip, keep very fast block propagation (FIBRE, compact blocks, good peering), lower template refresh latency and avoid slow stratum/proxy setups, Stratum V2 (job negotiation) helps, but still can't make propagation instantaneous. As long as pools are globally distributed and block propagation isn't instantaneous, occasional self-forks are unavoidable. Your observations for Foundry/ViaBTC are consistent with that.

Some sort of decentralization (redundancy and/or regional) was also something I had in mind. Sounds plausible to me. I know that block 912721 was in both UpdateTip events the same, clearly visible by the same hash. I was just surprised why it would need a second UpdateTip when this block 912721 wasn't realy affected by the fork of block 912722. It was block 912723 which decided what branch of the former tip of 912722 should be followed as the "true" blockchain (most accumulated hash work). Thanks for the more in-depth explanations. I had only vague and more incomplete thoughts what probably was going on. With the immense hash power of those pools, many many devices are crunching hashes, timely coordination becomes a real problem and the competitive mining space doesn't make it easier. I assume it's not trivial to manage such a large amount of mining gear and participating miners. As you say, I can follow that occasional race conditions happen. Trying to cope and avoid them completely likely has more or other disadvantages than letting them happen. I didn't expect that my node see most or a lot of the forks. It totally depends on my node's peers and I didn't specifically tune it to be any optimized for such a task. RPC getchaintips gives me a list of branch forks where block 723102 is the first in my list and remarkable entries are the invalid blocks 783426, 784121 and 809478.

It updates one block at a time. To go from old 912722 to new 912723 it has to count down making 912721 the new best for only a fraction of a second before continuing with the new 912722 and 912723.

I've been thinking a bit more about this situation. From my limited perspective I still consider it a serious disadvantage to not avoid such an own fork. If a pool happens to produce such a fork, now how does the pool decide on which branch they should concentrate their hash power? I know that mining is a random process, the more lottery tickets (aka blockheader hashes) you draw, the more likely it is to find one that matches the current difficulty threshold for a valid block. In such a fork situation the pool would have to decide and possibly gamble on which branch of the fork they continue to build upon. Can't see how it could be beneficial to distribute hash power on both branches. Then I was thinking: do some or all serious pools sample what kind of Stratum work items other pools distribute? I think this is what https://mempool.space/stratum does, while not being a mining pool but rather a very nice block explorer. Wouldn't a pool be able to sample by this, what other public pools are working on? A fork situation is a gamble in any case, regardless if the fork tips have been produced by the same or different mining pools. So, having a glimpse on which fork branch other pools crunch their new template block seems beneficial and smart to me.

It doesn't matter.