Math 132A

Errors in Hypothesis Testing

Procedure for Hypotheses Testing

• Choose your significance level \(\alpha\) (e.g. 5%).
• Choose whether you want two-sided (default) or one-sided test.
• Formulate your \(H_0: \mu = \mu_0\) and \(H_A\).
• Choose sample size \(n\).
• Collect your data.
• Verify assumptions!!!!
• Calculate the point estimate \(\bar{x}\) and the sample standard deviation \(s\).
• Calculate the standard error \(SE = \frac{s}{\sqrt{n}}\).
• Calculate the test statistic \(t = \frac{\bar{x} - \mu_0}{SE}\).
• Determine the \(p\)-value.
• Compare the \(p\)-value with the significance level \(\alpha\).
• Write down the conclusion.

The \(p\)-value…

The \(p\)-value measures how well does the null model explain the data.

The smaller the \(p\)-value, the stronger the evidence against the null model.

• If the \(p\)-value is as small or smaller than \(\alpha\), we reject the null model as not being useful. We say that the result is statistically significant at level \(\alpha\). That means we falsified our mathematical model.

• If the \(p\)-value is larger than \(\alpha\), we do not reject the null model. The result is not statistically significant at level \(\alpha\). In other words, the evidence does not contradict the null hypothesis.

What does small \(p\)-value really mean?

The \(p\) value is the conditional probability of a simple random sample of size \(n\) having the \(t\)-statistic equal to or more extreme than the sample we have, given the null model is correct.

Small \(p\)-value could mean:

• We have a really strange sample (unlikely if sample is simple random, but quite possible if it is not!)

• There is something wrong with the model:

  • Maybe we used a wrong distribution? (Assumptions not satisfied)
  • Maybe the null hypothesis is false.

Finding the \(p\)-value

• Suppose the \(t\)-statistic \(t = \pm 3.2\) and sample size \(n = 6\):

• Left-tailed test: the area to the left of \(t\) with \(n-1\) degrees of freedom:

  With \(t = -3.2\) and \(n = 6\):

  pt(-3.2, df = 5)

  [1] 0.01199759

• Right-tailed test: the area to the right of \(t\) with \(n-1\) degrees of freedom:

  With \(t = 3.2\) and \(n = 6\):

  1 - pt(3.2, df = 5)

  [1] 0.01199759

Two-tailed test

• twice the area to the right of \(t\) if \(t > 0\) or left of \(t\) if \(t < 0\), with \(n-1\) degrees of freedom:

  With \(t = 3.2\) and \(n = 6\):

  2*(1 - pt(3.2, df = 5))

  [1] 0.02399518

  With \(t = -3.2\) and \(n = 6\):

  2*(pt(-3.2, df = 5))

  [1] 0.02399518

• Another way: twice the area to the right of the \(\left|t\right|\)

  With \(t = -3.2\) or \(t = 3.2\) and \(n = 6\):

  t = -3.2
  2*(1 - pt(abs(t), df = 5))

  [1] 0.02399518

Types of Errors

	\(H_0\) true	\(H_0\) false
reject \(H_0\)	type I error	desired
don’t reject \(H_0\)	desired	type II error

\(H_0\) is never true !!!!

Types of Errors (improved)

	\(H_0\) “true”	\(H_0\) false
reject \(H_0\)	type I error	desired
don’t reject \(H_0\)	desired	type II error

• In theory, if the \(H_0\) is really true, the significance level \(\alpha\) is the probability of type I error.
• What is actually true:
  • Choosing smaller \(\alpha\) makes it harder to reject \(H_0\):
    • decreasing the probability of type I error
    • increasing the probability of type II error
  • Similarly, larger \(\alpha\) makes it easier to reject \(H_0\):
    • increasing the probability of type I error
    • decreasing the probability of type II error

An Experiment

Let’s experiment with different null hypotheses and different sample sizes.

Summary of the Experiment

• If \(H_0\) is really true, and samples are simple random, the probability of type I error is \(\alpha\) (except it’s really not).

• When \(H_0\) is false, the probability of Type II error decreases with increasing sample size.

  It also seems to depend on “how false” the null hypothesis is.

• If \(H_0\) is “almost true”, so that it can be considered true for practical purposes, the probability of type I error increases with sample size.

• With very large samples, the test becomes ridiculously sensitive, leading to rejection of perfectly reasonable models.

• If you need the precision that comes with large samples, you should not be doing hypothesis tests. It is better to use an interval estimate instead.

Garden of Forking Paths

Another problem is with what we call researcher degrees of freedom

• At any step of en experiment, researchers must make decisions:
  • Should this particular piece of data be included?
  • How do we handle outliers?
  • Which method do we use to find the p-value?
  • How do we round?
  • …

All of these will affect the p-value.

How do we make sure the researchers do not (consciously or not) select the choice that will decrease the p-value?

Probability of Type I Error

is always (possibly a lot) larger than \(\alpha\):

• \(H_0\) is never really true!
  • Do not use very large samples!
• Sample are almost never simple random!
• “Researcher degrees of freedom”
  • Problem especially with small samples
• “Publication bias”