Article Category: Analysis and Interpretation

By Dr Slava Vaisman, University of Queensland School of Mathematics and Physics    

In the natural sciences, we are constantly observing variables that move in tandem. For example, when temperatures rise, retail sales often surge, or when a new corporate policy is rolled out, employee absenteeism might spike. These patterns are tempting to simplify, but they pose a fundamental question: Is the relationship causal, or are both variables being driven by a separate, external force? Confusing correlation (or, more generally, association) with causation is not just a statistical slip; it is one of the most pervasive and expensive logical fallacies in scientific research, business, and public policy. Misinterpreting these relationships can lead to deeply flawed scientific conclusions, wasted resources, ineffective laws, and fundamentally unsound strategic decisions. 

Why is it relevant?

Mistaking correlation for causation leads to erroneous scientific conclusions, a breakdown of public trust, and wasted investment in strategies that target the wrong variables. For instance, a city might invest heavily in more police patrols in neighbourhoods with high ice cream sales, failing to realize that both are simply correlated with rising summer temperatures. By targeting the symptom rather than the driver, organizations often fail to address the real root cause of a problem.  

The stakes involved make the causality subject essential not only for researchers, journal editors, and reviewers who safeguard scientific rigor, but also for the policymakers and analysts who translate data into real-world action. 

Practical Steps

1. Before assuming A causes B, ask if there exists a hidden factor C that influences both A and B? A classic example is the correlation between ice cream sales and drowning incidents. The confounder here is hot weather: it causes people both to buy ice cream and to go swimming. More swimmers, in turn, lead to more drownings.
2. Ask if A causes B, or maybe B causes A, or could the causation be circular? For example, consider the link between happiness and productivity: are happy employees more productive, or does being productive (receiving praise) cause happiness?
3. Finally, to verify that A causes B, perform a controlled test by randomly splitting your subjects into two groups. Change A for one group (the treatment) and leave it unchanged for the other (the control). Because the groups were assigned randomly, any difference in B must be caused by the change in A.

Case Study

Case Study 1: A city observes that neighbourhoods with the most public parks also have the highest household incomes and the lowest crime rates. This leads to the (false) conclusion that building a new park in a distressed area will directly create local prosperity. 

Applying the three rules helps to expose the flaw. 

1. Identify the confounder. The initial conclusion ignores the role of the municipal tax base. Wealthier neighbourhoods generate more tax revenue, which the city then uses to build and maintain parks. Prosperity causes the parks, not the other way around.
2. Examine the Direction. The assumed causal direction is also questionable. While parks may attract some residents, it is equally plausible that wealthy residents possess the political capital to demand more parks in their neighbourhoods, creating a circular relationship.
3. To properly justify the investment, the council would need to conduct a controlled experiment. This would involve introducing parks in a selected group of neighbourhoods while leaving a comparable control group unchanged. After several years, the council could then analyse the differences in household incomes and crime rates between the two sets of neighbourhoods. Without this rigorous comparison over time, the city cannot validate the expense. By treating a symptom of wealth (green space) rather than the root causes of poverty, it risks wasting public funds.

Case study 2: In orthopaedic research, observational studies may show that patients with very low BMI experience higher mortality rates after total joint arthroplasty (TJA). This suggests an observable association, and statistical analyses may simply indicate a clear correlation between low BMI and mortality. However, more comprehensive investigations often reveal a more complex picture than simply concluding that low BMI causes higher mortality. This example helps clarify the different possible relationships between these variables: 

• Association: Patients with very low BMI were observed to have higher mortality rates after TJA. 

• Correlation: Statistical analyses showed that lower BMI consistently coincided with increased postoperative mortality. 

• Causation: At first glance, the findings may suggest that low BMI causes higher mortality risk. However, many low-BMI patients also had serious underlying illnesses such as cancer, frailty, or heart failure, which both reduced BMI and increased mortality risk. In this case, the observed relationship was partly driven by confounding and reverse causality rather than a direct causal effect of BMI itself. 

This example shows why statistical relationships alone are insufficient to establish causation and why researchers must carefully account for confounding variables and reverse causality when interpreting observational findings. [1] 

By Dr Pedram Rashidi and Emma Cooney, University of Queensland Centre for Policy Futures   

P-hacking is generally defined as ‘choosing to report a subset of multiple dependent variables or adding observations until the effect of interest is significant’. [1] There are many ways one can engage in p-hacking. [2] Typically, we think of p-hacking as the practice of using trial and error to fish for statistically significant relationships (a low p-value) with no regard to the theoretical underpinnings of the research. Selective reporting is a broader concept and often involves failing to publish studies that do not produce statistically significant results. The following example illustrates the practice of p-hacking:

A researcher studies whether a new diet improves weight loss compared to a control group.

1. They collect data and initially find no statistically significant effect (p = 0.12).
2. Instead of reporting this null result, they:
   • Remove a few ‘outliers’ (participants who gained weight).
   • Re-run the analysis -> p = 0.07 (still not significant).
   • Then:
     • Try different statistical models
     • Test only a subgroup (e.g., participants under age 40)
     • Adjust which variables are controlled for
3. Eventually, one analysis shows p = 0.04.
4. The researcher reports only this significant result and omits all other analyses.

Why is it relevant?

P-hacking often arises from the academic ‘publish or perish’ culture. The problem is exacerbated by scientific journals’ tendency to favour statistically significant or novel results. In this environment, researchers may, often unintentionally, make biased analytical choices. P‑hacking is also sustained by a lack of clear education and sometimes even institutional mandates. [3]

However, it is a detrimental practice that contributes to the proliferation of false-positive results and the inability to replicate prior studies. This has undermined confidence in some scientific results. [4] It can also lead to devastating consequences when data that has been manipulated is relied upon, especially in sensitive fields such as medicine. [5]

Practical Steps

You may inadvertently engage in outcome fishing as there are many ways in which one can analyse a dataset, and each of these decisions may contain implicit bias. [3] There are some practical steps to avoid this:

1. Acknowledge degrees of freedom. Be aware, and explicitly report on, the degrees of freedom within the design, analysis, and reporting of statistical analysis which can be used to generate statistically significant results. [3]
2. Justify all choices in data preparation and analysis. Ensure that all choices to exclude outliers, transform variables, analyse a certain range or with certain dependent variables are well-justified.
3. Consider implementing preregistration. Preregistration requires you to provide a specific, precise and exhaustive plan of the study. Each step within the plan needs to be detailed, providing only one way in which it can be interpreted or implemented. [6] This avoids outcome fishing.
4. Minimise bias in analysis. Whenever possible, perform analyses without knowing which data points belong to which groups, making it difficult to intentionally manipulate results toward a specific outcome.
5. Report results transparently. Fully report all measures, conditions, and analyses including insignificant results.

Case Study

The head of Cornell University’s Food and Brand Lab, Brian Wansink, has become notorious for instructing his research students to dig through data sets to discover relationships that were statistically significant. In an email to a student, he described the processes of data analysis as “squeez[ing] some blood out of this rock.” Following these revelations, Cornell opened an investigation into research misconduct and many of these studies were retracted. [7]