Essential Evidence Category: Public Policies

Dr Adrian Davis

Anyone faced with making a decision about the effectiveness of a road transport intervention, such as traffic restraint or cycling promotion behaviour change programme, faces a formidable task. The research findings to help answer the question may exist, but locating the research,1 assessing its evidential “weight” and relevance, and incorporating it with other information is often difficult.2 More particularly, it is recognised that transport planning practitioners may have little background in evidence-based research.

In considering appropriate methodological approaches when setting up evaluations of interventions one commonly used tool is an evidence hierarchy. This lists a range of study designs ranked in order of increasing internal validity (relating to cause and effect), see Box 1. The higher up the hierarchy the more likely are ‘confounding variables’ to be accounted for (ie other factors which may have an effect on the intervention). Thus, low level evidence such as cross sectional surveys are of less value compared to high level systematic reviews and meta analysis,3 where they are available. Study designs of transport interventions are mostly cross sectional and case controlled.

However, there is increasing recognition that at least as important as the study design is methodological aptness, that different types of research question are best answered by different types of study design. So, in certain circumstances the most robust study design may not be the most appropriate study design. Also, there may be value in combining the two in order to capture both quantitative and qualitative research: eg robust evidence of outcomes from RCTs and qualitative evidence on the process by which those outcomes were achieved (such as from cyclist intercept surveys).

Top line: It is valuable to understand the evidence hierarchy but it may be most useful to think of how you can best use the wide range of evidence available – and particularly to consider what study designs are most suitable for answering particular types of question.

1 Wenz, R., Roberts, I., Bunn, F., Edwards, P., Kwan, I., Lefebvre, C. 2001 Identifying controlled evaluation studies of road safety interventions. Searching for needles in a haystack, Journal of Safety Research, 32: 267-276.
2 Petticrew, M, Roberts, H. 2003 Evidence, hierarchies, and typologies: horses for courses, Journal of Epidemiology and Community
Health, 57: 527-529.
3 A systematic review is a literature review focused on a single question which tries to identify, appraise, select and synthesize all high quality research evidence relevant to that question. A meta-analysis combines the results of several studies that address a set of related research hypotheses. This is normally done by identification of a common measure of effect size, which is modelled using a form of meta regression.

No. 3: The Evidence Hierarchy Download pdf PDF approximately 54.42 K