lightweight database for brute-forcing public keys-32Mk =3.81MB(secp256k1)

Building a lightweight database with billions of keys. Last updated: 12/06/2023 1- We create a binary file with public keys in 1s and 0s. 0 = even, 1 = odd. Single public key. If you need to make a database bigger than your memory can handle, like 1000 million keys when your memory limit is 100 million, just divide it by 10 and adjust this variable: Avoid messing with this, or you’ll end up finding private keys with altered last digits. We did it! Now we have a massive database that takes up minimal disk space. Since there’s no pattern between the even and odd public keys, we can set a collision margin of 64, 128, 256... As you increase this margin, the chance of getting the same binary sequence goes down. What’s the point of this? We just need to randomly generate binary sequences and check if they’re in the database. Searching for a single public key For multiple public keys: First, we build the database by doing random subtractions within a range that we choose; you can make your own list. Db for multiple public keys: We create our database to handle several keys. Scanning multiple public keys with bit version. Scanning multiple public keys with byte version (this is faster). The key difference is that the byte version masks some bits within a byte, potentially missing some collisions. The bit version doesn’t have this issue. This code is just a sample and might not be fully optimized; it’s better to write your own version in C.

What's the advantage of this over storing full public keys at specific intervals? Also as you target a larger number of keys, your time spent scanning for the bit sequence increases, unless you use some smarter data structures (which would not be as small as 3.81mb anymore)

generating sequences 01001.... The possibility of finding an identical sequence is very low. The traditional way limits you in space, and it works the same. If you choose 64 as the collision margin you will divide your computing power/64, but with a huge database, you would still find the key faster than using xpoint.

If you sort and store partial keys you can search by using binary search, cutting search time from linear to log. For larger number of keys, this becomes much faster than your single bit storage. To reduce space requirements, only store public keys that fit a criteria (like ending with the six lower bits zero) when building. When searching, subtract until you hit the bit criteria, then look up with binary search. Also, the fact that the public keys you "store" have to be sequential makes these methods much less efficient than kangaro/BSGS etc.

You only store binary sequences in large spaces, if you want to spread your database over the entire range, and not in the same consecutive range. Using jump addition and subtraction to your target.

Tested for 35 bit range, 50 million keys. You write that the probability of false collisions is small, but I see something else.

What is that? I don't understand, are they false positives? If so, you must increase the collision margin from 64 to 128 or higher and problem solved. It is still random+ scalar so you can increase the margin of precision thanks to scalar multiplication. Edit: Although I tested it with a collision margin of 64 and I did not get false positives. Maybe you are doing something wrong if you modified the code, share details and we will find a solution.

By the way, for the 40-bit range there are no collisions at all. What database size is required for higher ranges? How can one calculate the size for a specific range? Is it possible to parallelize work across physical processor cores?

You already deleted the message, but I saw it in the mail. Here are other options. But I don't see any difference. Although knowing the first digit and 50% of the second, this greatly facilitates further work) Thanks for the work done, I will continue testing.

Told you not to share anything public, no actually working script. ☠ So it only takes 1 bit to store a key? I can't imagine what would that be like with 100TB space to store, and with enough fire power + bsgs or keyhunt it should be easy to solve large keys. Can you also change n to secp256k1 n - leading f's? To check if you can figure something out of it. I'd say there will be no floats if you do that.

I'm sorry, I deleted the message because I'm not very sure of the solution, try this. inx = f.find(s) replace inx = f.find(s)*sustract Let me know if it gives you the correct pk at collision margin 64. my apologies. edit apparently it has to do with the jump of 10 in subtraction, not with the collision margin edit2: fixed To change N directly to secp256k1.dll you can do it with a hex editor. The script works, the problem lies when it makes subtraction jumps other than 1, I hope it is solved. This is a good option, the FUDs say that it is of no use, they do not see further, nor do they deign to try (blah blah), they only limit themselves to criticizing

I checked that the script is working correctly now. But in the high ranges of 40 and above, there are no coincidences yet. In principle, up to 40 bits and simple random can cope with the same success.

you mean this range? 549755813887 :1099511627775 take into account: pubkey is in range? if you make jumps in database subtract= 10, in scan set it the same. Use 64 collision margin as long as you do not receive false positives, otherwise increasing it too much decreases your search speed. At high ranges use multiple objectives to cover more space in the range.

I think you are writing the ones and zeros as bytes, when you should be writing them out as bits. Here's what you should do to make your program faster. set a counter like i to 0, and then each time you perform a subtraction, do byte_value |= 0 [or 1] << i; i = (i + 1) % 8. Then only do a write after every 8 iterations. Although, you can make the writing process even faster by waiting until you fill thousands of bytes like this, and then just write them all at once in one batch.

This will depend on the amount of RAM you have, of course. However, That's what I did, I wrote 2^26 keys at once. Ate up at highest point, 14GB of RAM. If limited RAM you could do the counter as you suggested. I have found a key in the 44 bit range in about 3-4 minutes, 5 or 6 times. It's an interesting script. It's interesting in the way it can store massive amounts of "keys" in such a little file. For 2^26 keys, it only uses a file size of 8,192 kb. That is impressive. The searching method is not great in terms of speed. Also, you do not/should not use a start range of 1 (unless you are subtracting from original key and shrinking the key). When I generated 2^26 keys with a num = 64 option, I set the start range to 8796093022208-(2^26-64) = 8796025913408. I do not know where the key is but I know it's max (17592186044415) and it's minimum (8796093022208) and since my subtraction was set at 1 (sequential with no steps such as using a subtraction as 7 or 10, etc) I know the target priv/pubkey will only be from Target down to - 2^26 (keys generated) - 64 (num option). start (min) =   8796025913408 end  (max) = 17592186044415 That's how I approached this script when running tests. You could also speed the script up using an upfront pub subtraction to cut the range in half. Example, if we know the key is in the 44 bit range, we can do an upfront subtraction of 0x80000000000, and then use that newly generated pub in this script and then search in a max 43 bit range. Now to ponder how to add this script's storage size function to an existing GPU script...

This script speeds up the creation of the database. but as @WanderingPhilospher says

How would you break it up as NotA was saying: "|= 0 [or 1] << i; i = (i + 1) % 8" for those with smaller RAM.

Hi! get this error num = 128000000 Traceback (most recent call last):   File "D:\BTC\lightweight-database\lightweight-database\binary_Db.py", line 20, in <module>     hc= int(h[2:], 16)         ^^^^^^^^^^^^^^ ValueError: invalid literal for int() with base 16: ''

I just tested and works, apparently that mistakes occurs when you try to put an additional line if you have an previous script, update. The script works like this: You have a database like this 1010100010110101000101101010001010101000101101000101101010101010001011010100010 1101010001010101000101101010001011011010100010000 target is = 1101010001011010100010101010001 The target is in: 10101000101101010001011010100010101010001011010001011010101010100010110101000101101010001010101000101101010001011011010100010000 If we separate in bits: 10101000 1011010 10001011 01010001 01010100 01011010 00101101 01010101 00010110 10100010 11010100 01010101 00010110 10100010 11011010 10001000 No coincidences were found. edit: I update. - Script aggregate that solves the memory limit ​