This is one of the most confusing terms in statistics. But really, it just comes down to whether or not we have found evidence that a treatment works for people with a particular condition.
Evidence from comparing groups of people
When researchers want to find out whether a treatment is effective, they usually get two groups of people. They give the treatment to one group of people. The other group of people don’t get the treatment.
After a period of time, the researchers come back and see how people in both groups are doing. They might count how many people got better in each group. If more people got better in the treatment group, than in the no treatment group, that could mean the treatment works.
Why are groups of people different?
But a difference between the groups could be caused by something else.
Let’s say you want to know if one of two people is better at flipping coins. You get them to flip a coin 50 times each, and one of them got more heads than the other. Does that mean one of them is better at getting heads? Or is the difference just down to chance or ‘luck’.
A difference between two outcomes might be completely down to random variation.
So if we’re testing whether a medication makes more people better than if we just left them alone, how do we check whether any difference between the groups of patients is because of the medication, rather than just down to chance?
This is where ‘statistical significance’ comes in.
A Test to work out chance vs cause and effect
‘Significance’ is the name of a test to find out how likely it is that a difference between two groups is just down to chance. The researchers pick a threshold for the test. If the result reaches that level, they will say that the difference is ‘real’ because it’s very unlikely that the difference is due to chance.
What if the test doesn’t reach the threshold that the researchers set? Then the researchers will say that there’s no ‘statistically significant’ difference between the groups.
That doesn’t necessarily mean there’s no difference. It also doesn’t mean the treatment is definitely not effective. It just means that from this research, scientists haven’t found evidence that the treatment is effective.
This is clearly stated on all of our resources on topics where research showed no statistically significant difference between the treatment group and the comparison group.