Yeah, except the points are in random order so you have no way of knowing how far from the median the point is - which would imply that the computation would be faster.
This is still going to be prohibitively expensive for large N. It is O(n). Any method with a runtime greater than O(log N) is not going to be practical in cryptography.