Math 132A

Hypothesis Testing with Simulations

Another Study Example

A group of residents of a county is considering submitting a question on certain matter for a local referendum in the next election cycle. Submitting a question for a referendum requires a fair amount of work, as well as large financial commitment, so before they do all that, they want to find out whether there is sufficient support for the matter in the county.

They select a simple random sample of 16 eligible voters in the county and ask them if they would support the measure.

Out of the 16 voters, 11 responded that they would support the matter, while the remaining 5 would not.

Should the group go ahead with the submission?

Sources of Randomness

• In the sex discrimination study, randomness was created by the random assignment of the supervisors into two groups. It was done in order to eliminate confounding.

  The question the researchers were asking was whether there is a causal association between the sex variable and the decision variable.

• In the referendum study, the randomness was created by selecting a simple random sample of voters, in order to have a representative sample.

  The question the group was asking was whether the result they saw in the sample could be generalized to the whole population.

Possible Explanations

In each of the studies, we had two possible explanations of the study result.

• In the sex discrimination study, after eliminating confounding, the explanations were:
  • The supervisors discriminate based of sex of the applicant.
  • The result happened by chance, due to the random assignment of the supervisors to the two groups.
• In the referendum study, the explanations were:
  • There is a sufficient support for the measure in the population.
  • The result happened by chance, because of the random selection of the sample.

Analyzing Randomness

Each study came up with a number that suggested an effect:

• 29.2% observed difference in the promotion rates of males and females.
• 11 out of 16 voters expressing support for the measure.

The question to ask:

How likely would it be to observe numbers like that, if it was a result of chance?

---

We repeatedly simulate the process, assuming there was no actual effect, and see how often we get a result like the one we observed.

The Null Model

We made an assumption that there is no effect:

• There is no discrimination happening.
• There isn’t sufficient support for the measure.

We created a model of that situation.

Since it is a model of no effect, we call it a null model.

The model must be described well enough so we can simulate it.

We repeatedly simulate the study under the condition of the model, and see how often do we get a result like the one observed in the actual study, or even more extreme.

What Can Happen?

When we repeatedly simulate the study under the condition of the model,

• the actual observed result happens only very few times.
  • The observed result is very unlikely under the conditions of the model.
  • The model is not a good explanation for the observed result.
  • We reject the model, and accept some alternative explanation.
• the actual observed result happens frequently.
  • The observed result can easily happen under the condition of the model.
  • The model is perfectly plausible explanation for the observed result.
  • We cannot reject the model. The model and any other explanation that we did not eliminate yet are all plausible, and we cannot decide between them.
  • The model is useful, but not necessarily right.

The Null Hypothesis

The models we used:

• The promotion rate does not depend on the sex of the applicant.

  \(p_{\text{males}} - p_{\text{females}} = 0\).

• Exactly half of the votes support the measure, or voters don’t care, and vote randomly.

  \(p_{\text{support}} = \frac{1}{2}\).

The null model is always represented by an equality.

These equalities are called null hypotheses. We denote them \(H_0\).

• \(H_0: p_{\text{males}} - p_{\text{females}} = 0\)
• \(H_0: p_{\text{support}} = \frac{1}{2}\).

The Alternative Hypothesis

In each study, there was always one alternative explanation of the observed result:

• The female applicants are discriminated against by the supervisors.

  \(p_{\text{males}} - p_{\text{females}} > 0\).

• More than half of the votes support the measure.

  \(p_{\text{support}} > \frac{1}{2}\).

These are called alternative hypotheses, and are always sharp equalities.

We denote them \(H_A\) or \(H_1\).

• \(H_A: p_{\text{males}} - p_{\text{females}} > 0\)
• \(H_A: p_{\text{support}} > \frac{1}{2}\).

Common Terminology and Problems

• When we reject the null model, we often say that we reject the null hypothesis.
• In that case, we can also say that we support the alternative hypothesis.
  • That is somewhat problematic!
• When we cannot reject the null model, we say that we cannot reject the null hypothesis.
  • That does not mean that we support the null hypothesis!!!!!!!!!
  • What it means is that both the null hypothesis and the alternative hypothesis are plausible explanations of the observed result.
• People often say that we fail to reject the null hypothesis.
  • I advise strongly against that formulation!
  • Not rejecting the null model should not be considered a failure!
  • Rejecting the null hypothesis is not the goal of hypothesis testing!!!