Inverse Ising inference allows pairwise interactions of complex binary systems to be reconstructed from empirical correlations. Typical estimators used for this inference, such as Pseudo-likelihood maximization (PLM), are biased. Using the Sherrington-Kirkpatrick (SK) model a benchmark, we show that these biases large in critical regimes close phase boundaries, and may alter qualitative interpr...