Regression slope

We create a null distribution by shuffling (or “permuting” to use the official stats term) the values of x. This simulates a world where all the real, measured values of both x and y are still the same, but where the relationship between x and y doesn’t matter. This eliminates any association between x and y.

Think of this as being a world where there is no relationship between x and y. Importantly, this doesn’t mean that the slope is exactly 0. There is variation in the data, and that variation is reflected in the null world. What it means is that in the null world, the slope is 0 ± some amount.

Here’s what this null world looks like:

viewof n_reps = Inputs.range([100, 2000], {
  step: 100,
  value: 500,
  label: "Number of simulations:"
})

Here’s another way to see the null world slopes. Each thin line is a regression line for one of the simulated worlds where where x and y are unrelated.

Next we put δ inside that null world and see how comfortably it fits there.

Is it surprising to see the red line in this null world? Is the line way out to one of the sides, or is it near the middle with the rest of the null world?

Or alternatively, we can put the observed regression line from the actual data into the scatterplot with the null world regression lines. Is it surprising to see the red line in this null world?

We can actually quantify the probability of seeing that red line in a null world. This is a p-value—the probability of seeing a δ at least that big in a world where there’s no relationship between x and y.

Finally, we have to decide if the p-value meets an evidentiary standard or threshold that would provide us with enough evidence that we aren’t in the null world (or, in more statsy terms, enough evidence to reject the null hypothesis).

There are lots of possible thresholds. By convention, most people use a threshold (often shortened to α) of 0.05, or 5%. But that’s not required! You could have a lower standard with an α of 0.1 (10%), or a higher standard with an α of 0.01 (1%).

viewof alpha = Inputs.select([0.10, 0.05, 0.01], {
  label: "Significance threshold (α):",
  value: 0.05
})