One of the principles for understanding data is that while some data contain signals, all data contain noise. Therefore, before you can detect the signals you’ll have to filter out the noise. This act of filtration is the essence of all data analysis techniques. It’s the foundation for our use of data and all the predictions we make based on those data. In this column, we’ll look at the mechanism used by all modern data analysis techniques to filter out the noise.

Given a collection of data, it’s common to begin with the computation of some summary statistics for location and dispersion. Averages and medians are used to characterize location, while either the range statistic or the standard deviation statistic is used to characterize dispersion. This much is taught in every introductory class. However, what’s usually not taught is that the structures within our data will often create alternate ways of computing these measures of dispersion. Understanding the roles of these different methods of computation is essential for anyone who wishes to analyze data.

 …