BITCOIN MYSTERY: THE PREFIX CHALLENGE A follow-up

Just wanted to follow up on this discussion: BITCOIN MYSTERY: THE PREFIX CHALLENGE Since @mcdouglasx closed that thread and replies are not possible anymore. I’ve got a couple of questions: So, does your test work the same way as the previous ones? 1a. Define a specific search space size. 1b. Generate a certain number of search blocks within that space. 2. Randomly create a target to search for; pick a few leading bits as a prefix. 3. Take those leading bits from the randomly generated prefix. 4. Test two methods: a. find the prefix and jump to the next block if it’s not found, or b. find the prefix(es) but go through the whole block before moving on. 5. Both methods will go to the next block afterwards. 5. Keep going until you find the target. Is it something like that? Let me know if I’ve got it wrong. And what are you measuring for the results? Are you looking at how fast you find a prefix and skip to the next search block compared to checking each block completely? If that’s the idea, I’ve got some actual data from runs. It seems pretty clear this method will be quicker since it wraps up the searches faster, right? The only downside might be missing the actual target when using the prefix method. Does that sound about right to you?

Run 1 = 32 x 2^40 search space size and searching for leading 40 bits (prefix but with bits) Prefix Run 1: Normal 100% Check Run 1:

(Saved for Op data / tests)

Run 2 = 32 x 2^40 search space size and searching for leading 40 bits (prefix but with bits) Prefix Run 2: Normal 100% Check Run 2:

Run 3 = 107 x 2^40 search space size and searching for leading 40 bits (prefix but with bits) Prefix Run 3: Normal 100% Check Run 3:

Totally legit. That topic was closed because the method was demolished (for the Nth time) by its own basic principles, and in the very end, the exact same anti-arguments that were presented for the anti-prefix methods (e.g. always simply stopping after exactly N checks, no prefix check ever needed) were actually also valid (identical reasoning) for the prefix method itself. But I guess closing the topic was a better idea since this conclusion was imminent. Anyway, there's an ultimate search method: scan one key, then stop. Fast. Optimal. Rational choice FTW, here's why: If it WINS, then only in that case ever compare it to a straight up full scan. If it LOSES - ignore such a case completely. We only care about results when BOTH methods find a solution, if someone doesn't agree on this, then they don't understand probabilities. Efficiency ratio: infinity to 1. Can't beat this. Common sense score: NaN, just as with the original "dilemma".

I guess that is the part I was trying to understand in that older/closed topic. Was he saying prefix method was better and would find the key 100% of the time, or just that using prefix allows you to skip some part of ranges where the target key is less than some % to be in that skipped range? I do not see how the 100% scanned range lost every single time and only tied some of the time. One can look at my data and obviously see that depending on the random target key selected (if this were a test like his) that x percent of the time, the prefix method would skip it and so the 100% sequential would win, some of the time. Maybe I am just confused as to what his test/words were saying. Prefix/bit method will obviously be faster as in scanning ranges/keys because it skips some keys after a prefix is found, so it can move onto the next block/keys. BUT, if the target key is in the same block as another similar prefix, it may not be found by prefix method, but would 100% be found by the sequential method. But I feel like everyone already knew this. Maybe he is saying prefix would just be faster to scan the entire range, but the downside is, it may skip the target key...but it looked like he was saying the sequential never won, which can't be true. If you look at my real world data, you can see, many more keys were found with 100% sequential versus prefix so in some blocks, prefixes were skipped. That's the part that has me stumped.

I think everyone's even more confused now.   You're suggesting somehow that, by some way, there are more chances of finding an exact match, or "faster"? If so, compared to what exactly? If it's about checking the exact same sequence of keys in the same order in both cases... the topic was dissected way too many times. Once you answer yourself that, repeat the test, but don't do any prefix checks at all, just stop after 63.2% of a block was scanned. This time, one "prefix" means that you stopped and skipped. If you get different results this way: congrats, you broke crypto. If you didn't: perhaps it shows that the results have nothing to do with any prefixes at all, right? And that the prefixes are simply natural markers that occur with the expected chance they're supposed to occur with.

Lol. yeah, maybe you got lost as well, like I did in y'alls 3 page topic. I am trying to figure out what mcd was really trying to say. I am saying that if you are searching for x bit prefixes, and then if found, skipping to the next range, this way is obviously going to finish ALL ranges faster than the 100% sequential method, 100% no brainer. Anyone can argue with a wall, I don't care, my data proves it, but really, a test was not needed to know that would always be the case. So was mcd saying that in every instance, the prefix method found the target faster or at least tied the sequential method: But how?!?! Because there will be more than one prefixes in some of those ranges. If the prefix method found the first and skipped to next range, and the target was in that skipped range, the sequential method would have found it and thus, it would have been a win for the sequential method. But he said sequential had 0 wins. He also stated: But to me, it's a play on words. "optimized", "efficiency", "higher probability of success"... It's not optimized nor more efficient, it's using the same program (hashing); and the higher probability of success, is not true, since prefix method can skip some prefixes, to me, it has a less probability of success rate. It is more efficient in finishing x amount of blocks, but could skip target, but not more efficient in checking actual keys. But again, I was trying to get mcd to retort, and tell me how sequential method never had a win versus prefix method?!?! Or was I not understanding his results/what he was saying.

Test Number 4 Prefix only (625 ranges x 40 bit in size, searching for 40+ bit prefix matches; all in puzzle 71's range and its address): I forgot I had the program running so it ran 625 ranges (each range was 40 bit in size like the other tests) via prefix method. Total prefixes found = 401 (all 40+ bit) Total time = 1,114.30 minutes with an average scanned range time of 106 seconds. The average time for checking an entire 40 bit range is 167 seconds. If all ranges were checked 100%, that would come out to around 1,739 minutes. So during this prefix method run, an average of 61 seconds was saved by jumping to next range once a prefix was found. That's an overall time savings of roughly 625 minutes. So as obvious, the prefix method will get through x amount of ranges quicker, but it will also obviously skip a lot of other prefixes, and COULD skip the actual target address. In previous runs, the prefix method found between 62-67% of all prefixes. If we use a 65% rate, that would mean if we ran the 625 ranges and checked every key, we would likely find 620 prefixes. The overall time is also roughly 65%. To summarize: Prefix method 1,114.30 minutes and 401 prefixes found 100% method 1,739 minutes and roughly 620 prefixes found The more ranges ran, the more time will be saved via the prefix method, but also, the more prefixes will be skipped. So you will gain 35% speed up but at the cost of 35% less found prefixes.

I still don't understand what you are trying to say. What is the difference between scanning those 65% by stopping at prefixes, and simply scanning 65% by stopping after, well, 65% was scanned, in whatever way you wanted it to scan? In my view, none. So I don't really get what kind of speed up you are talking about, since it's obvious that, if you scan less, then you finish faster. But what does this have to do with prefixes whatsoever? If the goal is to collect prefixes, then again, by the same logic, why does it matter that you stopped and skipped after a prefix was found, instead of, for example, scanning the very first 65% of the entire range (or the last 65%, or the middle 65%, or whatever randomly chosen keys that make up a total of 65%) and collecting your prefixes along the way? Yeah, it will finish faster, (than a 100%) but it will finish after the same amount of time and same amount of steps as with "match and skip" method, and it WILL find the same amount of prefixes, on average. That's why I said everyone is confused again, since you didn't really explain what is "faster" here, I don't see any actual over-proportional benefits or anything like that, your results look totally normal. Scanning one key and stopping is also faster. Not scanning anything is also much faster.

Any method that: measures speed only when it succeeds ignores probability of success skips failure cases is not an optimization.... It’s cherry-picking dressed up as engineering. Bottom line: skipping work doesn’t beat math, it just dodges it.

What I meant by the prefix method is that it's better for searchers with limited resources, since the probability of finding two identical prefixes in the range is theoretically reduced. Therefore, once you find the prefix, you omit the theoretical space with the lowest probability, giving you a search advantage by using a probabilistic strategy instead of traversing the entire range, which is impossible in this context. In other words, for those who try their luck... you won't find the solution every time, but you'll be searching strategically, guided by the mathematical truths of hashes and their probability. The 65% Blind Cut is a guillotine (an absurd one, by the way). Because no matter what's happening in the data, the search stops. If the key was in the 66%, you always lose it. You create a systematic blind spot. The Prefix Jump is a sensor. The program stops because it has detected a match, which tells us that the block has already met its statistical quota. If the key was in the 66%, the Prefix method has a chance of finding it if there were no false positives before your attempt.

The exact same "argument" applies to whatever Prefix-related search. Because, no matter what's happening in the data, the search stops. So, the prefix% cutoff, is, by your own definition, a Blind Cut guideline, which, by your own words, is of course, an absurd one. By the way. Unless of course, we start adding the "proximity bias" and other non-sense, in which case, pardon me: you are right!

Considering that in the prefix method, the search window is elastic. If a block has no prefixes, you scan it 100%. If it has one at the beginning, you jump to 5%, for example. This variability is what eliminates the systematic blind spot you propose, and a probabilistic failure is always better than a design failure. I don't cut the block because I'm tired or because the clock says so; I cut it because the data tells me that the block has already yielded its probable result. You fail due to design arrogance; I fail, if at all, due to statistical chance, but you don't seem to see that.

What data?! What is a "block"? It's impossible to have any sort of rational discussion with someone who acknowledges two contradictory things at the same time: a) that any H160 is independent of any other H160 b) that says that some H160(inputA) has any sort of correlation to some other H160(inputB) If "blocks" contain independent H160 stuff, then it's exactly the same thing whether you view that block as contigous or whether it was formed by some subset of keys from whatever positions. If they are independent H160, then scanning the first 65% or the last 65% or whatever 65% will yield, on average, the exact same results, the same stats, the same "wins", the same whatever statistics you throw at them. Making them equivalent. Make up your mind once and for all.

You're confusing (intentionally, or trying to) statistical independence with algorithmic design. Nobody is saying that one H160 affects the next. What we're saying is that using the prefix as a jump signal eliminates location bias. Your 65% cutoff is a human-imposed bias; my prefix jump is a response to data entropy. My method is dynamic, yours is a blind guillotine. That's why mine is a search strategy and yours is just a poorly applied hardware limitation. For example, we all know that for every 4096 hashes, at least one 3-character prefix is ​​expected (there could be more, or there could be none, but this doesn't change the strategy). I don't know why you want to equate your theories with the search method; it's absurd. They're not the same. Whether you stop at 65%, stop randomly, or use prefixes for probabilistic searches, prefixes are the only method that guarantees a solid and justified mathematical basis when jumping to the next block.

I was merely trying to understand what mcd was doing/saying in his analysis. But, with actual ranges and prefixes that are in the curve. To see if I understood his "method"/data first, but also see if it held up under real world tests. I agree with what you are saying and I don't think mcd would disagree here (but it could be seen as a "blind" test versus mcd's test), but he is merely trying to: Yes, he acknowledges that the range may have more than 1 prefix in it and it's possible to skip the target (I think he believes/knows that). It seems he is trying to say, "the odds are less"...to find more than 1 in a range, but you could, but the odds/theory says otherwise. So he is trying to give the search some teeth versus doing it blindly. I may indeed do a test with what you say: That will be interesting to see how close the number of prefixes are found, if we only run the first 65% of previously ran ranges. Data to follow...

Independent events don't care about "algorithmic design", whatever that means to you. Independent events don't care about "position bias", whatever that means to you. Who's "we"? How can you contradict yourself from one sentence to the very next? It's amazing. Independent events don't care about "data entropy", whatever that means to you. Independent events do not have "search strategies", because they are all independent. Independent events only have a single attribute: their probability. So what you are naming a "blind hardware guillotine" is identically equivalent to your prefix search, because they are both doing the same exact thing: hashing some whatever X% amount of some big set of inputs, that produce independent events. No, it is false that "at least one 3-character prefix is expected". What is true is that "one 3-character prefix is expected (on average)" followed by the entire distribution of probabilities for all the valid possible cases (zero to 4096 prefix encounters). I believe you should take those dozens of books on probability, read them up, and check out the chapters about independent events. Independent events don't have search methods. It's absurd. I don't have any theories at all, other than, if you scan X% of some big set of independent events, it doesn't matter in what order the independent events run, because they are... independent events!!!! They'r exactly one and the same thing. Identical. You compared them yourself. They had the same results. Because they are independent events, and the order does not matter, and because there are no blocks, no jumps, no skipping, no memory, no position bias, no location bias, and so on, with independent events. The only solid and justified mathematical basis is that independent events are not groups of buddies that gather up in wolf packs. Sorry I meant "blocks with data entropy and location bias, let's jump around". Now let's talk a bit in the prefix search terminology. Look, you start off your prefix search of the range blocks. You get a prefix in the first block. You want to jump. Answer this: Why, instead of jumping, you can't just continue? Why is it that the rest of the "block" is, in your view, not identically equivalent to whatever same amount of keys at the end of the range?. Why can't you simply scan the rest of the block (and trim down from the end of the last block), and continue this way, ending up with a perfectly continuous X% scan? // End of talking in prefix method terminology. All the H160s in some range are statistically speaking, extractions out of an urn of balls. You take one ball out, that's a H160. You put the ball back in. Repeat N times (for example 2**70 times). See? There is no concept of "skips". Skip what? Extracting balls? How do you even count skips? If someone can still believe in "location bias" and "jumps"  "blocks with entropy" when the problem is as simple as extracting balls and putting them back in, then they have some much bigger problem. You should paint on your wall: "Independent events!" On the palm of your hand you should tattoo: "Independent events!" On your forehead, when you look in the mirror, it should say, mirrored: "Independent events" I'm pretty much way too tired of the subject. I wish you nice holiday, and don't forget: INDEPENDENT EVENTS. INDEPENDENT Yes, independent events.

kTimesG, get "Independent Events" tattooed on yourself if you want, but the probability is 1/N, and my skipping strategy is the only one that's engineering optimal. If I find my prefix in any portion of N, it's better to sacrifice the rest of that range because it's statistically less profitable to stay there. I prefer to move to a block where the 1/N probability is "fresh" and not contaminated by a previous find. Any self-respecting programmer chooses the dynamic prefix option. We're not looking for the solution 100% of the time; we're looking to scan statistically consistent parts. Your 65% method is a blind guillotine that forbids you from looking at 35% of the space by design. Mine is an intelligent search. I'd rather fail by chance than fail because my own code prevents me from finding the solution. @WanderingPhilospher, now do you understand why I closed the old thread? This person wants to refute an idea based on generalized concepts and fallacies. kTimesG confuses the statistics of the object (the hash) with the statistics of the process (the search). Independent events are independent; nobody is saying there can't be 10 consecutive prefixes in a block N (it's just unlikely). Hashes are uniformly distributed, and that's why the strategy works. If hashes weren't uniformly distributed, the prefix method wouldn't have a solid foundation. But since they are, finding a prefix is ​​the statistical signal that that segment has already "met its quota" of expected probability. Mathematics aren't being broken here; it's just a statistical strategy that takes advantage of them.