Statistical testing in null worlds

There is only one test

All hypothesis tests follow the same workflow:

https://moderndive.com/v2/hypothesis-testing.html#ht-case-study - This workflow allows you to test any statistical hypothesis through simulation by following this process for the idea that there is only one statistical test:

• Step 1: Calculate a sample statistic, or \(\delta\). This is the main measure you care about: the difference in means, the average, the median, the proportion, the difference in proportions, the chi-squared value, etc.
• Step 2: Use simulation to invent a world where \(\delta\) is null. Simulate what the world would look like if there was no difference between two groups, or if there was no difference in proportions, or where the average value is a specific number.
• Step 3: Look at \(\delta\) in the null world. Put the sample statistic in the null world and see if it fits well.
• Step 4: Calculate the probability that \(\delta\) could exist in null world. This is the p-value, or the probability that you’d see a \(\delta\) at least that high in a world where there’s no difference.
• Step 5: Decide if \(\delta\) is statistically significant. Choose some evidentiary standard or threshold for deciding if there’s sufficient proof for rejecting the null world. Standard thresholds (from least to most rigorous) are 0.1, 0.05, and 0.01.

Examples

• Difference in Means
• Difference in Proportions
• Regression Slope
• One-Sample Mean
• Theoretical P-values