An Improved Outlier Detection Algorithm and Robust Estimation

The requirements of model accuracy and robustness make the outlier detection and robust estimation become more and more important in the model construction. In this paper, we first use the high-dimensional influential measure (HIM) based on the marginal correlation and the high-dimensional discriminant method based on the distance correlation (HDC) to respectively detect the outliers in the data set. Then the points are divided into two parts: normal points and abnormal points. Based on the initial normal point set, we construct the method of recovery for the points that are misclassified to normal point set, by using a kind of robust coefficient estimation method and the concept of hyper ellipsoid contour in residual space. Thereafter the outlier probability of each point in the abnormal point set are calculated to further recover the normal points that are misspecified in the abnormal point set and thus detect the true outlier value. The accuracy rate of outlier detection has been further improved. The performance of the proposed method is illustrated through simulations of three types of anomaly data under two predictive data structures, as well as three real examples.