Statistical Significance and Effect Size
-the overused and the under-used
(last updated on 2011-02-19)
Statistical significance is one of the most widely used statistical term, and likely,
the most overused or abused statistical term. No matter how small α is for
a treatment, it means nothing more than that the treatment probably makes difference.
Statistical significance does not tell how big the difference is. In some sense,
"significance" is an unfortunate choice of words for this statistical
term because it converys different meaning than its use in daily speech. For example,
suppose a cholesterol lowering drug reduces cholesterol level by 1 mg/dL, or less
than 1%. Assuming the standard deviations of the treatment and control arms are
10 mg/dL, it can be shown with
sampling and t-test simulator that a significance level of α < 0.05 can
be achieved with only 500 samples in each arm. Apparently this
drug would be essentially useless despite the statistical significance it may
show in lowering cholesterol. If it had any side effects, it might do more
harm than good.
The effect size is usually our ultimate concern. For the above example,
the effect size is 1 mg/dL (or 1 mg/dL devided by the standard deviation).
Of course, an effect size is meaningful only if it is backed by an acceptable signficance
The scenario where the statistical significance is more likely used in lieu of the
effect size is when both the treatment of the effect are binary. Vaccines
are good examples in this category. The treatment variable of a vaccine is whether a subject
has received the vaccine;
the effect variable is whether the subject has contracted the corresponding disease
after the vaccination. In this case, the
best effect size is odds ratio
which shows how much difference the vaccine makes in terms of the likelyhood of
contracting the disease.