Cognitive Connie
Understanding data analysis & statistics
Collecting data is only half the work. The numbers and narratives researchers gather have no meaning until they are analysed, interpreted, and placed in context — and the conceptual tools used to do that analysis determine what conclusions can and cannot be drawn.
Key figures
Karl Pearson
1857–1936British statistician who developed the Pearson product-moment correlation coefficient (r) in 1896 — still the most widely used measure of linear association in psychology. Pearson also developed chi-square and principal components analysis, laying the groundwork for the quantitative statistical approach that dominates 20th and 21st century psychology. His Biometrika journal (founded 1901) became the leading outlet for statistical methodology applied to biological and social data.
Ronald Fisher
1890–1962Introduced null hypothesis significance testing (NHST) and the p < .05 threshold that became the de facto standard for psychology. Fisher developed ANOVA, the F-test, and the conceptual framework of the null hypothesis. His Statistical Methods for Research Workers (1925) and The Design of Experiments (1935) were the operating manuals of 20th century empirical science — and the source of the p-value framework that is now both ubiquitous and heavily criticised.
Jacob Cohen
1923–1998Statistician who demonstrated in a landmark 1962 survey that most published psychology studies were drastically underpowered — typically below 50% power for medium effects. Cohen's Statistical Power Analysis for the Behavioral Sciences (1969/1988) formalised effect size conventions (d, f, r, ω²) and provided the mathematical tools for power analysis. His 1994 American Psychologist paper "The earth is round (p < .05)" made the case for effect sizes and confidence intervals over p-values and became one of the most cited articles on statistical reform.
Key concepts
Quantitative data
Data consisting of numbers — frequencies, scores, measurements, ratings — that can be analysed statistically. Enables objective comparison across groups, hypothesis testing, and generalisation via inferential statistics. Generated by closed questionnaire items, reaction time tasks, physiological measures, structured observations with frequency counts, and standardised tests.
Qualitative data
Data consisting of non-numerical information — words, descriptions, themes, narratives — typically analysed through thematic analysis, content analysis, grounded theory, or interpretative phenomenological analysis (IPA). Provides depth, nuance, and insight into meaning and experience that quantitative data cannot capture. Generated by open-ended interviews, open survey questions, case studies, and naturalistic observations.
Correlation
A statistical relationship between two variables: as one changes, the other tends to change in a consistent direction. Positive correlation: both increase together. Negative correlation: as one increases, the other decreases. Zero correlation: no systematic linear relationship. Correlations describe associations but do not establish cause — the direction of influence is unknown, and both variables may be driven by a third.
Correlation coefficient (r)
A standardised measure of the strength and direction of a linear relationship between two variables, ranging from −1.00 to +1.00. The sign indicates direction; the absolute value indicates strength (closer to 1 = stronger, scores cluster tightly around the trend line; closer to 0 = weaker, scores scatter widely). Cohen's benchmarks: |r| ≈ 0.1 small, |r| ≈ 0.3 medium, |r| ≈ 0.5 large.
Correlation ≠ causation
Three reasons prevent a correlation from establishing causation. Reverse causation: the direction of influence is unknown (does A cause B, or B cause A?). Third-variable problem: a confounding variable C may cause both A and B, producing an association with no direct causal link between them. No manipulation: in a correlational study, the researcher does not change any variable — only true experiments (with IV manipulation and random allocation) establish causal direction.
Statistical significance and p-values
A p-value answers: given that the null hypothesis is true, how probable is it that we would observe results at least as extreme as these? A p < .05 threshold means researchers accept a 5% risk of falsely rejecting the null when it is actually true (Type I error rate). Statistical significance does not mean the effect is large, important, or replicable — only that it is unlikely to be due to chance at the chosen alpha level.
Type I error (false positive)
Rejecting the null hypothesis when it is actually true — concluding there is an effect when there is none. The probability of a Type I error is set by the alpha level (typically .05). Reducing alpha (e.g. to .01) decreases Type I errors but increases Type II errors. Multiple comparisons inflate Type I error rates — testing 20 hypotheses at p < .05 produces on average one false positive by chance.
Type II error (false negative)
Failing to reject the null hypothesis when it is actually false — missing a real effect. The probability of a Type II error is β; statistical power = 1 − β (the probability of detecting a real effect). Power increases with: larger sample sizes, larger true effect sizes, more sensitive measurement, and higher alpha (at the cost of Type I errors). Most psychology studies conducted before the 2010s were substantially underpowered (Cohen, 1962).
Effect size and Cohen's d
Effect size quantifies how large a difference or relationship is, independent of sample size. Cohen's d is the standardised difference between two group means: d = (M₁ − M₂) / SD_pooled. Benchmarks: d ≈ 0.2 small, d ≈ 0.5 medium, d ≈ 0.8 large. A study can be statistically significant (unlikely due to chance) but have a trivially small effect size — common in large-sample studies. Effect sizes are comparable across studies using different scales and are the basis of meta-analysis.
Test your knowledge
Frequently asked questions
What does a p-value of .03 actually mean?+
A p-value of .03 means: if the null hypothesis (no effect) were true, there would be a 3% probability of obtaining results at least as extreme as those observed by chance. Since 3% < 5% (the conventional alpha threshold), the null hypothesis is rejected and the result is declared statistically significant. Common misconceptions: the p-value is not the probability that the null hypothesis is true, not the probability that the result is a false positive, and says nothing about how large the effect is. A p-value of .03 is compatible with a negligibly small effect in a large sample.
Why is "correlation does not imply causation" so important?+
Because our intuition runs the other way — when we observe two things moving together, we instinctively infer that one is causing the other. This leads to systematic errors in everyday reasoning and in early-stage research. Classic examples: countries with more TVs per capita tend to have lower infant mortality — does TV prevent infant deaths? No: both are caused by economic development. Ice cream sales correlate with drowning rates — does ice cream cause drowning? No: both correlate with summer temperature. Establishing causation requires ruling out all plausible alternative explanations — reverse causation, third variables, confounds — which is why only well-designed experiments with random allocation can do it cleanly.
What is the difference between statistical significance and practical significance?+
Statistical significance answers: is this result unlikely to be due to chance? Practical (clinical) significance asks: is this effect large enough to matter in the real world? A drug that reduces blood pressure by 0.5 mmHg on average may be highly statistically significant (p < .0001) in a trial of 100,000 patients, but the effect is too small to have any clinical relevance. Conversely, a therapy that reduces depression scores by 40% may fail to reach significance in an underpowered study of 20 patients. Effect sizes — Cohen's d, r, η² — quantify practical significance independent of sample size and are essential companions to p-values in interpreting research.
Sources
Last reviewed July 2025- 1.
Cohen J. (1988). Statistical Power Analysis for the Behavioral Sciences (2nd ed.). Lawrence Erlbaum Associates.
+About this source
Foundational text introducing power analysis and effect size conventions (small, medium, large) that remain standard in psychology.
- 2.
Field A. (2018). Discovering Statistics Using IBM SPSS Statistics (5th ed.). Sage.
+About this source
Widely used statistics textbook covering statistical significance, effect sizes, and data analysis procedures.