Bootstrap

The Bootstrap application was designed to perform statistical analysis of various codecs rated by multiple listeners. It uses a technique called bootstrap resampling, which means that it runs many simulations using randomized versions of the original data set to determine the probability that the original relationships occurred by chance.   Resampling techniques incorporate the correlations of the original data set, and thus tend to be more powerful than their classical counterparts on certain types of data sets (skewed).

The bootstrap program resamples independently per listener (blocked analysis) by default. This increases power by assuming that the ratings a listener produces are correlated with each other, although the range of values used by a particular listener don't necessarily correspond to the range used by another.

The program adjusts the p-values to take into account that multiple measurements increase the chances that a significant result in a single comparison between two codecs may not be significant in the context of the overall experiment. The program currently uses a free step-down p-value adjustment, but in the future will use a more powerful restricted p-value adjustment technique.