the quickest way to bulk-create addresses in Python

Hey everyone, Lately, I've been experimenting with different methods in Python to create random private keys (in hex) along with their corresponding Bitcoin addresses (whether uncompressed, compressed, bech32, or mixed) as efficiently as I can. There are a bunch of different ways to do this. So far, I'm pretty impressed with the "bit" library from Ofek since it seems to be the fastest I've tried. But if you're looking to generate bech32 addresses too, you'll have to put in some extra work manually, which kinda takes away the speed advantage of using bit. As a solid alternative, I've found that using fastecdsa and iceland2k14/secp256k1 is really quick. EDIT: Note1: I found out that the iceland2k14/secp256k1 Python library is actually written in C++ and just wrapped for Python. Plus, it's closed-source, so we don't know the source code. No security issues have been reported so far, but it's still something to keep an eye on. Some Bitcoin community tools are using this library for speedy calculations. Note2: As I kept working, I realized that the fastecdsa library is kinda bloated for generating random private keys and doesn't perform the best. On the flip side, the secure Python library "secrets" is quick when it comes to randomness and performs well too, giving over +32% better performance. In a few of my Python scripts that I'm testing, I've added a performance counter that shows the total number of addresses generated and the rate (addresses/sec) while the program runs.

I see that you're using gen_private_key. This generates arbitrary secure random private keys, using urandom. urandom might be slow depending on your system - do you really need it? I assume whatever you're doing (brute force search?) will generate the private keys in some other fashion, so you can just set prvkey_dec to an integer representing the private key directly. I also see that iceland2k14/secp256k1 does not provide the source for its libraries which is a bit concerning.

It has to have a purpose otherwise it is a complete waste of time even if someone could optimize the code to generate 1 billion addresses in 5 seconds.

I think I understand what you mean. There is no prize money, so I better rewrite my previous sentence: Let's make a contest-like game. Create a simple code like this and try to beat the high score of currently 4.7 seconds for 1million bech32 addresses without using GPU. I don't see it as a waste, if someone makes helpful and further posts in the forum because for this not only the current forum community benefits but also those who will read here in the future. I see it as an enrichment for the forum. We could change the rules to say that the use of GPU via CUDA should be allowed. I myself have tried various approaches so far using pycu and numba, but I was always unsuccessful. So if someone can do this simple task also within CUDA and can present the result, please feel free to post here. I think everyone will benefit from it. Honestly said, I don't care about if urandom is used or random, I just want to quickly generate random numbers for serving as a privkey. Can you suggest a faster approach which can replace this task? what concerns in detail you have, can you explain, please?

One of my teachers used to say: "Thinking is not knowing" why would the author iceland2k14 then hide the source code by not disclosing it? Especially if he still uses source code from AlbertoBSD and Jean-Luc-Ponds? Apart from the fact that this would be illegal as far as I know, since the tools mentioned use appropriate GNU licenses, why would iceland2k14 hide the code if there was nothing to hide? I think this is an interesting point that ymgve2 has made here. I hadn't given this library any serious thought before, but he's right in a way. To the extent that we as users of this closed-source library don't know if malicious functions are being run in the background or data is being exfiltrated, I see that as a danger as well. I will therefore consider not using this library until I am convinced otherwise. Thanks to both of you for this important advice. I understand that while Python code executed on the GPU using these libraries can be very fast, it may not be as fast as native C++ code running on the GPU. This is because the Python interpreter introduces some overhead, and the libraries themselves may have some overhead as well. However, for many applications, the speedup provided by running on the GPU can more than make up for any overhead introduced by Python, and can still result in significantly faster calculations than would be possible using only the CPU. That is why I think it is reasonable to get such calculations running on the GPU, it should be possible with Python. Python can be used for fast calculations with CUDA by using libraries that provide Python bindings for CUDA. These libraries allow you to write code in Python that can be executed on the GPU, which can provide significant speedups for certain types of calculations. Examples of such libraries that provide Python bindings for CUDA are -as already mentioned before- PyCUDA or Numba. PyCUDA allows writing code in Python that is executed on the GPU using CUDA. It provides a Pythonic interface to the CUDA programming model, making it easy to use for Python developers. Numba, which is a just-in-time (JIT) compiler for Python that can automatically optimize Python code for execution on the GPU. Numba can be used to write GPU-accelerated code in Python using familiar Python syntax, and it can provide significant performance improvements for certain types of calculations. Unfortunately I wasn't able to successful use those two libraries until now, hopefully anyone else can give some insight and assistance here. Sample code is provided on my first post.

Prize is incentive not purpose. A purpose could be something like a tool that would generate vanity addresses so you'd need the fastest possible algorithm and implementation that is capable of checking very large number of addresses per second. Or the purpose could be a recovery tool where you have 8-9 words out of a 12 word seed phrase so you need to check millions of addresses that is generated from it at different derivation paths to find the right combination. The importance of "purpose" is that it will also clarify HOW to implement it. Right now if I understood the code correctly, you are just aimlessly writing random addresses to disk so there is not much to work with. But if the example I used is the purpose (ie vanity address), there is no need for IO and its bottleneck hence the algorithm changes, there is also no need for Base58 or Bech32 encoding, there is also no need for ECMult each permutation. These very simple changes lead to significant increase in speed.

I think you misunderstood. The whole should not be a project that I commission here and want to have programmed free of charge by someone. On the contrary, it is about learning and understanding so that you can use it later in various projects. It's not about me, but in general about a kind of knowledge database, from which the whole community benefits. I read here in various posts the most diverse technical questions, often even one and the same the topic. Or it is tried to reinvent the wheel. There are a few who are very good and experiences in developing good code and the matter, but knowledge is shared sparsely, so at least my impression. I am honestly said a newbie in this matter who tries to get in and understand above all. I don't have a real intention, so it's not really about finding vanity addresses or cracking puzzles. There are already solutions for that and I can't develop on those if solid basic knowledge is missing. I want to learn and maybe some day I can find ways to optimize or help others to fulfill this task. The road is the goal. The example I showed is the basic building block of all conceivable tools that are used in this topic. If Ofek Lev had used an existing library and made do with it, there would have been no "bit" library. But he has thought about how to optimize and improve, or shortcut ways to make the whole thing more efficient. That's what it's all about. I see no disadvantage in sharing such knowledge here, I see clear advantages. If you would work together to optimize and extend such basic calculations like creating an address, a key or derivations of it, then everybody would benefit from it because it is shared publicly. This base can be used on every bitcoin tool out there. So far, unfortunately, I have not found any explanations or examples here that illustrate how such calculations would be implemented using CuPy, Numba or JIT, ergo also the origin of this thread. I would like to optimize Python code that make use of GPU and make the calculcations as fast as possible. If you or anyone else can contribute factually, I would greatly appreciate it on behalf of the forum community and all future readers here. But describing the whole thing as pointless from the ground up helps neither me, nor you, nor anyone else.

Since fastecdsa already utilize C for heavy operation, i doubt we can see major improvement without either 1. Modify some part of fastecdsa to use assembly. 2. Rewrite everything using C/C++/Rust and assembly.

I was curious to try out a Rust implementation and took the first thing I could find on the web when looking for 'Rust Bitcoin address generator'. So far, very disappointing. I added a loop of 1000 iterations to this, and some disk I/O instead of the print statements: https://github.com/mrgian/bitcoin-address-generator It takes around 5s for 1000 addresses, 538.78s for 100k addresses. Sure, single-threaded, but your initial Python implementation was single-threaded too and just took 82s for 1 million addresses. As ETFbitcoin mentioned, those Python libraries you use are probably heavily optimized already.

next step is to shift the load to GPU with pycuda, numba, jit or similar. To use the wrapper PyCUDA we need to modify the code to offload the address generation and private key conversion tasks to the GPU. This can be done by defining GPU kernels that perform these tasks and then using PyCUDA's DeviceAllocation and Kernel functions to run these kernels on the GPU. Unfortunately I was unsucessful with that task.

Long time ago I wrote a python script that computes 16.4 million of consecutive (non random) public keys (not addresses) in 12.3 s. No numpy, pure python, only 1 thread. ecc.py gen_batches.py Generating consecutive public keys is way faster than generate random public keys.

what was the reason therefore? lowest_key is just the number to start the consecutive run IMHO. And I doubt that  you generated 1,007,862 MILLION (!) in 1.6 seconds With number_of_batches = 82 you should have been generated 1,007,862 keys (which is about 1 million) which the result for of 1.6 is realistic. You could replace the very last line of arulsbero's code in gen_batches.py to get a more clear output: We should take into consideration that arulbero's way does not generate the corresponding addresses and not saving them into a file, we need to rewrite the code and see how it behaves. @arulbero: thanks for sharing, sounds interesting. I need to dig into it, unfortunately I have to go right now but I certainly will check your code later when back at home. I need to think about the consecutive vs. random part and make some comparisons each other. Currently when I run your code it says it generated 1 million keys in average of 1.22 seconds. I just need to test if this speed is realistisc. Can you modify your code so it will generate an address and output all the generated addresses to a file called address.out ?

I don't have a fast  'pub_to_addr' function from 12.3 s to 3m and 9s to generate 16.4 M of addresses,  16.4 M / 190 s = about 86k addresses/s. 12 seconds to generate 16.4 M public keys, almost 3m to compute 16.4 M addresses. new ecc.py new gen_batches.py

I had already thought that halfway. Now with your updated version which also generates addresses it's much slower, the result is clearly different than just calculating the pubkey as you showed in your initial version. I get about 70,000 addresses/sec with your updated version: I realized that you output tuples of addresses using list(map). I suggest to adjust your program so it outputs a single address on each line so we compares apples by apples. I am not sure if the list output affects performance in some way, that's why I am pointing to it. The example I originally showed using iceland2k14/libsecp256k1 was about half the speed. With it I can generate 1 million addresses using all cores (in my case 16 cores) and write to the file in 4.8 seconds, this is a rate of about 208,300 keys/sec. I have modified my initial program so that you can now also configure the cores under which the program is executed. So one can select specifically "1" as value, so that we also compare apples with apples. Running this under one single core, now I get 26.5 sec for 1 million generated addresses. This is a rate of 37,735 keys/second. I am curious to see your program using all available cores.

To generate 1 address for line you need to replace print (*addresses,sep="\n") print (*addresses2,sep="\n") print (*addresses3,sep="\n") with for i in addresses:       print (*i, sep="\n") for i in addresses2:       print (*i, sep="\n") for i in addresses3:       print (*i, sep="\n") The results are the same (slightly faster):

I would like to try another approach and using iceland's secp256k1 library for generating the address from the pubkey. Can you tell me in which variable I can find the public key in your program? For example. When I print the second element of the list "batch" I get The first element is the xkey (pubkey) of that key: the second element 66899334846368891744016007453456293662065123677302105461153252024609684696055 represents the public pair y what is third and fourth element representing? 54338240394213138931889225770364196918256374885639854771044914499200701239376 91288621757068410199936994202114003372515557459679014336095768812037691308008 As I recognize, they seem not to be the public keys of the consequent keys 36028797808045094, 36028797808045095, 36028797808045096

The keys are generated in this way: P (P + 1*G , P - 1*G)   -> 4 coordinates, x and y * 2 points (P + 2*G , P - 2*G) (P + 3*G , P - 3*G) (P + 4*G , P - 4*G) (P + 5*G , P - 5*G) (P + 6*G , P - 6*G) ..... ..... (P + 2048*G, P - 2048*G) each batch has all the consecutive keys from P - 2048*G to P+2048*G   (4097 public keys), but they are not in order

@arulbero Impressive how you're getting those speeds on Python, considering that multithreading is complicated to do efficiently in that language, and GPU/FPGA support is lackluster to nonexistent.

It is not programming related, computing P+G is much more easier than compute P=3512878756844563653653674365654352352*G

Ok, finally I managed to use your multithread program with my script. Results:   30s to generate 16.4 M of addresses, about 540k addresses / s, and store them in a file. Only 1.3 s to generate 16.4 M public keys (without writing) 30s to generate 16.4 M addresses and store them in a file (24s if each thread writes on the file without waiting for the others)