Seeking specific examples for BTC mining calculations

I've been searching for two weeks for a detailed breakdown of how to hash the block header, but nothing covers the entire process. So I figured I'd reach out here for some help. I'm coding for BTC mining using a CPU. I get that it’s not the same as mining with ASICs like my S21 fleet, but this is more of an academic project for me. I’ve got my setup to connect to the pool and can submit fake or stale shares just to confirm that I can log in and start mining. Now, I want to mess around with the nonce and other parameters to actually find some legitimate shares. I’m using a Ryzen CPU, and who knows, maybe I’ll get lucky and find a share before my block times out. But my main goal is to really grasp the whole process by writing my own code. I found an almost complete example online and got my code to generate the same good hash the original author did. The issue is I had to manually input the Merkle root since there wasn’t enough info in the resource to understand how it was derived. I know the theory behind its calculation, but without that info, I can't be sure I get it. I also have traffic data from my S21s to work with, but I haven't managed to produce any hash that fits. My hashes always start and end with a non-zero hex digit, which doesn't make sense to me. So I’ve set that aside for now and kept digging.

You could inspect the code of mining tools like cgminer or bfgminer. Or try to figure out stuff at https://learnmeabitcoin.com/technical/mining/. On this site you can go pretty deep into the details, you can expand interactive tools there and learn a lot. Look at stuff like candidate block and how the block header is assembled for it, also how Merkle root is computed exactly. Look for early block examples with few transactions only and see if your code does it right. I don't quite remember if the Bitcoinbook explain it detailed enough. I guess, it's time to read another pass through it...

Yes, thanks, Cricktor. Of all the source information I was able to find, this one (which I already knew about)  came the closest to solving my troubles. It filled in a lot of holes, but still lacks a 100% completely worked example start to finish. I'll try the other source you mentioned since I haven't seen it yet. Thanks. -gt- Okay, I've been able to get a little further with this now. I found an example that had the coin base hash. Still missing this:   coinbase = coinbase1 + extranonce1 + extranonce2 + coinbase2 However this SHOULD be one of the easiest parts to deal with. If I had the pieces to make coinbase, I could hash it twice and I'd get the double hash of coinbase. But I'm pretty close now, because the example I now found misses these pieces, but DOES include the double hash of coinbase. I was able to use that to calculate the correct Merkle Root.    So even though I STILL haven't managed to find a complete worked example end-to-end, the only missing link is extremely trivial - just concatenating four things and double hashing them. Therefore, I'm assuming that my captured mining traffic will now work and generate a reasonable hash. Although it's not over yet, its pretty close. I found so much misleading information. One source claimed that regardless of what kind of BTC operation it is (transmitting funds, or submitting shares) that you have to hash pairs linearly, not in a tree. Another source insisted that the concatenation of pieces from the merkle branches had to be paired in lexographic order, not in the order they appeared. Finally, an (apparently) correct source indicated NOT to lexographically order, but to build a binary tree hash the nodes as pairwise leaves moving them up a level after each completion until only the hashed root remains. THAT strategy allowed me to reproduce the example's final value from the coinbase and merkel branches. I even tried using ChatGPT, but he kept changing his mind about which of the three strategies to use to hash the merkel tree. I can't believe there doesn't seem to be a complete, proper, end-to-end example out there on the net, and what's more surprising is all the posted code I've found which clearly uses completely different strategies to do the hashing. Every posting claims to be tested and correct and for BTC mining, but each one gives a different final hash for the same inputs. -gt- Well, I'm not out of the woods yet. Having taken data from https://learnmeabitcoin.com/ I now realize that I can't tell how it relates to my captured mining data. For example, when I see the table of TX hashes in the table on the website example, does the endian-ness match how they look in the JSON strings from the miner/pool? There are still too many pieces and unknown details, and too many possible things to reverse or not to reverse, so without more details I STILL can't resolve this. I've decided to just post the pool/miner exchange that led to a accepted packet and hope that SOMEONE who sees this feels like seeing if they can put it all together in a way that gives a block header that hashes to a reasonable hash. I've removed all traffic except that relevant to the successfully mined share. However, the exchanges are probably not in order. Here's what I got: The miner subscribes: {"id": 3, "method": "mining.subscribe", "params": ["Antminer S19j Pro+/Fri Apr 14 21:39:16 CST 2023"]} The pool responds: {"id":3,"result":[[["mining.notify","00"],["mining.set_difficulty","00"]],"00",7],"error":null} ExtraNonce1: '00',ExtraNonce2_size: 7, The miner asks to authorize: {"id": 5, "method": "mining.authorize", "params": ["1AGf9BSutC5ZAnMXi8rvE6nxoaVv4rYfNa.018", "x"]} The pool responds: {"id":5,"result":true,"error":null} Pool offers job to miner: {"id":null,"method":"mining.notify","params":["3782626","f29143edc62458e69f1a5661f85d2cdff4aeff9000008ac30000000000000000","01000000010000000000000000000000000000000000000000000000000000000000000000fffff fff5a032f840d1d506f7765726564206279204c75786f72205465636868005f002a971a59fabe6d 6d648fad0e94f875d4783a1d2b146e0fa869052d181b79a88ebc54783aeda6018e1000000000000 00000010666","ffffffff05220200000000000017a914bf73ad4cf3a107812bad3deb310611bee49a3c79871a14c 1120000000017a914056adde53ebc396a1b3b678bb0d3a5c116ff430c870000000000000000266a 24aa21a9ed23ed3159409e966bff9642d36a658a385d5b655b0ba0bbba8d8b9317ce9a841500000 000000000002f6a2d434f524501a21cbd3caa4fe89bccd1d716c92ce4533e4d4733f459cc4ca322 d298304ff163b2a360d756c5db8400000000000000002b6a2952534b424c4f434b3a5033e57db45 8230515b003ad6b37755e76755337c12a9d8efd024916006f4f1500000000",["298563959d824c769c90df6880153ffe37ce692f40c86030f8da46072883ca98","713f3880adf3ec99241e10773c5994e0173631b23013be3a406423744523b5cb","d2cedcb2598099f5155c70f3964c8b9983d6bacdeae094a0c125eee2826332af","cd0308e9611464d808fc0cc7cd7a5220e7db837d27f8753ddf7ba1cf0e1004c7","43d98cfae83c979cb52376012ac6087c566b48e5b13c1e1fffbad82ab1ef429a","86c27c93418feac302f2fa49165314470c1c98fb989b7dd0712cba3c7ae515c9","fa5d9bd36f07cd05d7c38b37e5dcb18a4ff5a1c5f137c6c03f97acce091b4770","05b0ee676903b07295f97c2833348101368d5f0f0039fde81e35f3067df03278","4fe266926aab2d9e481aa79ab081b2953fb043c873f8111ec171673aafeb3981","96511215a92f1011bdfae46a4e2b86359571a72f1beaa5f795abb7493fbe4206"],"20000000","17028bb1","67c28154",false]} Miner submits a share: {"params": ["1AGf9BSutC5ZAnMXi8rvE6nxoaVv4rYfNa.018", "3782626", "ec100000000000", "67c28154", "16384721", "09b66000"], "id": 19, "method": "mining.submit"} The pool responds: {"id":19,"result":true,"error":null} ========================================== {   "params": [     "1AGf9BSutC5ZAnMXi8rvE6nxoaVv4rYfNa.018",  // Username (Bitcoin address + worker name)     "3782626",                                // Job ID     "ec100000000000",                         // Extranonce2     "67c28154",                               // Time     "16384721",                               // Nonce     "09b66000"                                // Extra nonce (optional, rarely used)   ],   "id": 19,                                   // Request ID   "method": "mining.submit"                   // Method name } pool offers job to miner {   "id": null,                          // Identifier for the request (null means no specific ID)   "method": "mining.notify",           // Method being called (notifies miners of a new block)   "params": [                          // Parameters for the mining job     "3782626",                         // Job ID (unique identifier for this mining job)     "f29143edc62458e69f1a5661f85d2cdff4aeff9000008ac30000000000000000", // Previous block hash     "01000000010000000000000000000000000000000000000000000000000000000000000000fffff fff5a032f840d1d506f7765726564206279204c75786f72205465636868005f002a971a59fabe6d 6d648fad0e94f875d4783a1d2b146e0fa869052d181b79a88ebc54783aeda6018e1000000000000 00000010666", // Coinbase transaction (block reward + fees)     "ffffffff05220200000000000017a914bf73ad4cf3a107812bad3deb310611bee49a3c79871a14c 1120000000017a914056adde53ebc396a1b3b678bb0d3a5c116ff430c870000000000000000266a 24aa21a9ed23ed3159409e966bff9642d36a658a385d5b655b0ba0bbba8d8b9317ce9a841500000 000000000002f6a2d434f524501a21cbd3caa4fe89bccd1d716c92ce4533e4d4733f459cc4ca322 d298304ff163b2a360d756c5db8400000000000000002b6a2952534b424c4f434b3a5033e57db45 8230515b003ad6b37755e76755337c12a9d8efd024916006f4f1500000000", // Merkle root (hash of all transactions)     [       "298563959d824c769c90df6880153ffe37ce692f40c86030f8da46072883ca98", // Transaction hash 1       "713f3880adf3ec99241e10773c5994e0173631b23013be3a406423744523b5cb", // Transaction hash 2       "d2cedcb2598099f5155c70f3964c8b9983d6bacdeae094a0c125eee2826332af", // Transaction hash 3       "cd0308e9611464d808fc0cc7cd7a5220e7db837d27f8753ddf7ba1cf0e1004c7", // Transaction hash 4       "43d98cfae83c979cb52376012ac6087c566b48e5b13c1e1fffbad82ab1ef429a", // Transaction hash 5       "86c27c93418feac302f2fa49165314470c1c98fb989b7dd0712cba3c7ae515c9", // Tr

"I am also a beginner in writing Bitcoin mining code in Python. I know Python is slow, but I need to write a full Bitcoin mining program in Python. Can you share your code? I need to learn." "How do I get the correct block template to start mining, and if I successfully find a block, how do I send and receive my reward?" i have 8 core amd cpu                  # i need understand & write python code i have Bitmain Antminer S19 Pro # after convert c++ https://github.com/iceland2k14/solominer/blob/main/solo_miner.py  # i know this git . but i cant understand. @gt.townsend Can you share your code? I need to learn.

Hello, sdfasdf I had a look at the URL you sent. I note that the author includes a commented out test "blockheader" and he is hashing it correctly. I know this for two reasons, i), the hash value obtained makes sense, and ii) I can take that blockheader and run it through any online tool and obtain the same hash. If you "uncomment" that line of code, you can confirm that his code really does generate the correct hash for it. However, if you examine the test blockheader, you'll note that it is clear that the version has been endian reversed. If you add a statement to print the blockheader generated from his program from the mining pool, you can see that the version is embeded with the OPPOSITE endian. This is the problem I'm running into - all sorts of code that claims to do one thing but does something else. I'll show you what I added to thecode to print the blockheaders that the program assembles from the pool: Find this part of the code:         blockheader = ctx.version + ctx.prevhash + merkle_root + ctx.ntime + ctx.nbits + nonce         # Test         #blockheader = "00000034a701be3e9898775007c26f5f956db38ad858465828a500000000000000000000482eb3d 98241ac8bb925d6419a20d67d1a79e3f217b21805776f9ca537fad3809fe96667fa970217bf573c ce" and modify it like this:         blockheader = ctx.version + ctx.prevhash + merkle_root + ctx.ntime + ctx.nbits + nonce         print(blockheader);         # Test         #blockheader = "00000034a701be3e9898775007c26f5f956db38ad858465828a500000000000000000000482eb3d 98241ac8bb925d6419a20d67d1a79e3f217b21805776f9ca537fad3809fe96667fa970217bf573c ce" compare the test blockheader to what the printout produces, in my case the print statement printed this: 20000000c26520e21ad7f8a9f277dc76355d111ab8f107cd00008d7d0000000000000000048b5d6 f84057432be23908430f4c7325f1266cc899a3656be49dbbdc33cbaef67c9bbf817028bb1c1778f dc comparing these together (any of course they won't be the same, but the layout/organization should match): 00000034a701be3e9898775007c26f5f956db38ad858465828a500000000000000000000482eb3d 98241ac8bb925d6419a20d67d1a79e3f217b21805776f9ca537fad3809fe96667fa970217bf573c ce 20000000c26520e21ad7f8a9f277dc76355d111ab8f107cd00008d7d0000000000000000048b5d6 f84057432be23908430f4c7325f1266cc899a3656be49dbbdc33cbaef67c9bbf817028bb1c1778f dc The font fools you here - these are indeed the SAME length, even though it might not seem to be so. Anyway, clearly the version in the test block is: 34000000 and has been switched to little endian in the block, while the version printed from the pool is: 20000000 and appears in BIG endian order in the block That should immediately raise a red flag. As far as I understand, the pool would reconstruct the block with the version flipped as 00000020 ... we can't get the same has at the miner that the pool gets if one flips it and the other doesn't!!!!

Well, here's an update for anyone who cares.   I now FINALLY have all the bits and pieces I need to sort this out. From greg's  "learn me a bit coin" website,  I found an example of everything EXCEPT the coinbase value. What WAS included was everything else, plus the 32 bit double SHA256 hash of coinbase. Tonight I just found out that you can use this URL is you have a coinbasehash, and it will tell you the original coinbase value. [Obviously it cannot hash backwards, but it searches the block chain for the hash, and finds coinbase elsewhere in the block that it came from: https://blockstream.info/api/tx/<inset coinbase hash here>/hex for example: https://blockstream.info/api/tx/18a16d322b235f636ab90e62e79a9f20a0b9c14e8da51e9dc0974f99f82ee444/hex This gives coinbase=01000000010000000000000000000000000000000000000000000000000000000000000000ffffffff0804233fa04e028b12ffffffff0130490b2a010000004341047eda6bd04fb27cab6e7c28c99b94977f073e912f25d1ff7165d9c95cd9bbe6da7e7ad7f2acb09e0ced91705f7616af53bee51a238b7dc527f2be0aa60469d140ac00000000 Now you can break this apart as follows ... ask ChatGPT for the details: 01000000 010000000000000000000000000000000000000000000000000000000000000000000000 ffffffff 0804233fa04e028b12 ffffffff 01 30490b2a01000000 43 41047eda6bd04fb27cab6e7c28c99b94977f073e912f25d1ff7165d9c95cd9bbe6da7e7ad7f2acb 09e0ced91705f7616af53bee51a238b7dc527f2be0aa60469d140ac 00000000 and then we get coinbase1 = 01000000010000000000000000000000000000000000000000000000000000000000000000fffff fff0804233fa04e028b12 extranonce1+extranonce2=ffffffff coinbase2 = 0130490b2a010000004341047eda6bd04fb27cab6e7c28c99b94977f073e912f25d1ff7165d9c95 cd9bbe6da7e7ad7f2acb09e0ced91705f7616af53bee51a238b7dc527f2be0aa60469d140ac0000 0000 An now, FINALLY we have ALL the bits and pieces required to compute the entire block header. Everything else required is packed into the example blocks and is easy to find on greg's website. The above was the only missing link. -gt-

Well, apparently it is not that easy. I STILL cannot produce a decent hash. I've been working with an older version of cgminer that supports BTC mining, and I've written a program to act as a fake pool which serves up the exact same block every time. I've hacked the cgminer code to freeze the timestamp and extranonce2 and nonce to the values in my mined share. I'm having trouble figuring our where to set the nonce, so I'm not done yet, but in the meantime, I've discovered that cgminer uses a liner approach to create the merkle root, NOT a binary tree hash. I've also discovered that there is an endian change, not byte-wise, but a 32-bit endian change that I had no idea existed. So many more pieces of the puzzle are now solved, but the puzzle is not solved yet ...

In case anyone still cares about this post, I should mention that I FINALLY go the thing 100% solved. After wondering about the extra "sixth field" that the miner was submitting after the usual five fields that are clearly explained in the documentation, I looked more carefully and realized that during the JSON exchange, early after authenticating, the miner negotiated something with the pool called "version rolling" and the pool gave the miner a "version rolling mask". It turns out I was doing everything correct EXCEPT I did not take this into account. It seems that the miner can negotiate with the pool to alter the "version number" in the block header, limited by the constraints specified in the "version rolling bitmask". There's a little Boolean equation that you use to combine your suggested version, with the original version, and with this bitmask to produce a "new_version" that you use in place of the original. Put in a simplified way, you are allowed to alter the bits in the original version as dictated by the bitmask. The sixth field contains your suggested modification to the bitmask. This acts like an "extra" extranonce that you can play with, but the advantage is that you don't need to recalculate the coinbase, its hash, and the merkle root. You still have to hash the new blockheader, but that's quite a bit less work! Anyway, I made that small adjustment and suddenly I calculated a share with a nice long string of leading zeros.