Significance Level ‘α’The significance level is the value at which we will decide whether or not to call the result of a hypothesis test “statistically significant” or “not statistically significant”.
We call this significance level ‘alpha’ (α)
Much like the confidence level ‘C’ for confidence intervals must be decided in advance, we must also decide the significance level (α) in advance.
Much like we commonly choose 95% for ‘C’, there is also a “typical” value for alpha: It is 0.05.
That is, if p <= 0.05 we call our result significant
If p>0.05, we call our result not-statistically significant
Tradeoff: Recall the ‘tradeoff” when choosing a C: The higher the C, we’ll be more confident, but at the price of a higher margin of error. Things work very similarly, for statistical significance. The main difference is that we want a lower value for α. As with C, it’s up to us to decide what value of α we are “comfortable” with. Typically, we choose 5%. Allowing a lower α is more forgiving, but just as with desiring a higher C, there is a cost. If we choose a very low significance level, we are setting the bar extremely high for rejecting the null hypothesis.