Hypothesis Testing

In statistics, hypothesis testing is a method for evaluating hypotheses. A hypothesis is a proposed explanation for an observed phenomenon.

The way hypothesis testing works is in a way analogous to how proof by contradiction works [1].

Unlike proof by contradiction though, the conclusions obtained via hypothesis testing do not underline some absolute truth. This is by nature of statistics, which is inherently inferential and approximate.

• we are interested in a candidate statement which we want to demonstrate to be false
• we start by assuming the statement to be true
• we then seek for a contradiction to this statement
• if we demonstrate a contradiction, then we will have proven the candidate statement to be false
• if we fail to find a contradiction, then we will have failed to prove the statement to be false.

Analogously, in hypothesis testing,

• we have a null hypothesis, which we seek to reject so to instead accept an opposing alternative hypothesis
• we start by assuming the null hypothesis to be true
• we then make a number of observations, collecting data on the outcomes so to estimate the probability of the observed outcomes, assuming that nothing observed is extraordinary
• if what we observe has a probability lower than some significance threshold which we define in advance, then we conclude that this is evidence that our assumptions (null hypothesis) are incorrect (false) and we must reject them, accepting the alternative hypothesis instead.1
• otherwise, we fail to reject our null hypothesis. This does not mean we accept the null hypothesis.
• after all, the null hypothesis is nothing but a “tool” for producing a “proof” by contradiction, or better, for coming to statistical conclusions.

Here is a table that may be useful in seeing the analogy

Candidate Statement (CS) Null Hypothesis (NH)
we want to prove CS to be false we want to reject NH
assume CS to be true assume NH to be true
seek for contradiction collect data on phenomenon
contradiction found -> prove CS false probability < $$\alpha$$ -> reject NH
otherwise -> fail to prove CS false otherwise -> fail to reject NH

References

[1] Goldman, D. S. (2018). The Basics of Hypothesis Tests and Their Interpretations. Center for Open Science.

1. Rejecting the null hypothesis does not necessarily mean accepting the alternative hypothesis.

• This is because the null hypothesis also assumes an underlying distribution, which is assumed in the alternative hypothesis too.
• As such, rejecting the null hypothesis either means that indeed there is something unusual that warrants an acceptance of a new (alternative hypothesis) or that the original distribution which we chose was actually wrong.

Notes mentioning this note

Here are all the zettels in this zettelkasten, along with their links, visualized as a graph. You may need to zoom and pan around to see something.