We propose and analyze a randomized zeroth-order optimization method based on approximating the exact gradient by finite differences computed in set of orthogonal random directions that changes with each iteration. A number previously proposed methods are recovered as special cases including spherical smoothing, coordinate descent, well discretized descent. Our main contribution is proving conv...
