Systematic Reviews in the Social Sciences: are we being misled?

At the Centre for Evidence & Policy we examine a large number of systematic reviews in order to provide extended abstracts for the UK database Social Care Online. We are increasingly concerned that many of those we initially select as specifically relevant to the UK are not actually systematic at all even though this is claimed in the title.

The methodologies used in the analysis of the papers under consideration may be systematic and of a high quality but the selection of databases to find the research or practice evidence is anything but. A huge American bias is being demonstrated in the selection of databases in the majority of cases. In addition the search strategies used to identify papers are often at best vague and at worst relatively useless.

We are very concerned at these inadequacies as many of these systematic reviews have been prepared under contract from government departments wishing to address policy issues in social care; community development; crime and justice etc. Basing evidence on predominantly United States research is just not acceptable. Is the reason for ignoring UK produced services a lack of awareness by the researcher; or is it down to the researcher’s laziness in tracking down appropriate databases? We suspect that some researchers are handing the crucial job of finding the evidence to post graduates with time on their hands on the dubious premise that searching is ‘an under-labourer’s job. We also wonder if another reason is the failure of academic librarians to keep abreast of what the UK is producing in terms of quality bibliographic databases having already invested heavily in USA sources such as Web of Knowledge; Sociological Abstracts; PsycInfo etc which are selective in their coverage of UK and European papers and rarely cover UK reports and other valuable grey material.

Without a searching process which uses the full range of databases covering the chosen subject field these systematic reviews are little more than academic exercises offering virtually no value in helping prepare crucial social policies. A recent example taken from our file of hundreds of systematic reviews will illustrate the scale of the problem:

A report in 2010 for the DCSF entitled “Short breaks provision on disabled children and families: an international literature review” searched just four American databases. How could this be ‘international’? Why were Social Policy & Practice, ChildData, Social Care Online (SCO), Community Abstracts or Family & Society (Australia) not searched? These excellent, predominantly UK sources are subscription based (except for SCO which is free) but so are the USA sources. One suspects that there was no local subscription in existence and the authors could not be bothered to go to The British Library to make use of its excellent database resources.

This example is the tip of the iceberg! Similar reviews on crime and justice issues; health, sport etc specific to the United Kingdom are just as inadequate in their use of UK databases.

Whilst there is obviously no guarantee that searching a wider and more appropriate range of databases will provided an increased number of relevant papers and other documentation, the reader is still left with the very real concern that important evidence has been missed and that the systematic review is biased or inadequately researched.

Why is this happening?

It is clear that UK academic researchers, and, perhaps worse, their support services in many university libraries, are largely ignorant of the UK sources available to them. Also, it does appear that those who peer review the papers submitted for publication are equally ignorant of the databases, or have a touching belief in the comprehensiveness of those services cited in the review, even if only one or two are used. The USA produced sources are excellent but inevitably are biased towards USA produced papers; but in the social sciences especially such bias in the finished product is not acceptable.

References

Grayson, L. and Gomersall, A. (2003) A difficult business: Finding the evidence for social science reviews, Working Paper 19, London: ESRC UK Centre for Evidence Based Policy and Practice. http://evidencenetwork.org/Documents/wp19.pdf.

Alan Gomersall is a Senior Visiting Research Fellow Centre for Evidence & Policy King’s College London.

Our guest blogs are written by people from our network and the research community, who take an interest in our work and do not necessarily reflect the RIN’s policies. If you have a burning issue you’d like to air in our guest blog, please email kissley.leonor@rin.ac.uk