reddit

A robust method for identifying outliers in data that doesn’t conform to a standard bell curve is the focus. This approach adjusts the standard z-score calculation to be less sensitive to extreme values. Instead of using the mean and standard deviation, which are easily influenced by outliers, it utilizes the median and median absolute deviation (MAD). The formula involves subtracting the median from each data point, dividing by the MAD, and then multiplying by a constant factor, often 0.6745 (assuming an underlying normal distribution for the MAD constant). For example, a data point significantly deviating from the median, when subjected to this modified calculation, yields a higher score, potentially flagging it as an outlier.

Employing this alternative score offers several advantages when dealing with datasets that violate normality assumptions. Traditional z-scores can be misleading in skewed or heavy-tailed distributions, leading to either an excess or deficit of outlier detections. By relying on the median and MAD, which are resistant to extreme values, the resulting scores are more stable and provide a more accurate representation of the relative extremity of each data point. This approach provides a more reliable assessment of unusual observations in situations where standard parametric methods are inappropriate. Its practicality has spurred discussion and application in various fields analyzing complex and non-normally distributed datasets.

Read more