Short communication: Handsearching the EMHJ for reports of randomized controlled trials by UK Cochrane Centre (Bahrain)

PDF version

A. Al Hajeri,¹ J. Al Sayyad² and A. Eisinga³

ABSTRACT This study used handsearching to find reports of randomized controlled trials in the Eastern Mediterranean Health Journal (EMHJ). EMBASE and MEDLINE were also searched electronically to identify if the reports found by the handsearch were already included in either of these databases. Nine reports were identified: 7 randomized controlled trials and 2 controlled clinical trials. The added value of the handsearch over EMBASE was 6 additional reports and over MEDLINE was 4. Reports identified were sent to the UK Cochrane Centre for verification and publication in The Cochrane Central Register of Controlled Trials (CENTRAL).

Recherche manuelle de comptes rendus d’essais contrôlés randomisés dans La Revue de Santé de la Méditerranée orientale par la branche locale du Centre Cochrane du Royaume-Uni à Bahreïn

RÉSUMÉ Cette étude a utilisé la recherche manuelle pour trouver des comptes rendus d’essais contrôlés randomisés dans La Revue de Santé de la Méditerranée orientale. Une recherche électronique a également été effectuée dans EMBASE et MEDLINE pour déterminer si les comptes rendus trouvés manuellement étaient déjà inclus dans l’une de ces bases de données. Neuf comptes rendus ont été identifiés : 7 essais contrôlés randomisés et 2 essais cliniques contrôlés. La valeur ajoutée de la recherche manuelle par rapport à EMBASE était de 6 articles supplémentaires, et de 4 par rapport à MEDLINE. Les comptes rendus identifiés ont été envoyés au Centre Cochrane du Royaume-Uni pour vérification et publication dans le Registre central Cochrane des essais contrôlés (CENTRAL).

¹Bahrain Branch of UK Cochrane Centre, The Cochrane Collaboration, Bahrain (Correspondence to A. Al Hajeri: This email address is being protected from spambots. You need JavaScript enabled to view it.).
²Medical Review Office, Ministry of Health, Bahrain.
³UK Cochrane Centre, Oxford, United Kingdom.
The views expressed in this paper represent those of the authors and are not necessarily the views or the official policy of The Cochrane Collaboration.
EMHJ, 2006, 12(Supplement 2): 253-257 

Background

There is heightened interest in the scientific community in charting the publication of medical research by geographical region. A study of the number of MEDLINE-indexed publications in Arab countries indicated that these countries produce less than 1% of the biomedical citations in the world [1]. However, El Ansari [2] stressed that a count of publications indexed in MEDLINE does not accurately reflect the extent of biomedical output from Arab countries and suggested several explanations. Much of the research is published in the Arabic language (as of 20 July 2005, only 228 of 15 million citations in MEDLINE contain studies in Arabic), in formats not indexed (e.g. conference abstracts) and in journals that are not indexed in MEDLINE.

It is increasingly recognized that health care decision-making around the world needs to be informed by high quality and timely research evidence. The randomized controlled trial has long been considered the “gold standard” in the hierarchy of evidence and trials involving sufficient numbers of participants are essential to distinguish reliably between the effects of health care interventions and the effects of bias or chance. The synthesis of the results of these trials in systematic reviews can provide reliable evidence about the effects of these interventions. The Cochrane Collaboration is an international organization dedicated to improving health care for the world’s population by preparing, maintaining and promoting the accessibility of Cochrane systematic reviews of the evidence of the effects of health care interventions.

The validity of the results of a systematic review is highly dependent on the data included and one prerequisite is to ensure that the set of studies is as unbiased and complete as possible. The Cochrane Collaboration has focused on the systematic electronic searching of MEDLINE and EMBASE and the systematic handsearching of currently over 2000 general and specialized health care journals for reports of randomized controlled trials. This handsearching involves reading each document in a journal to decide, according to the eligibility criteria set by Cochrane [3], if it might be a report of a randomized trial. The efforts of the many volunteers working within The Cochrane Collaboration have added a substantial number of previously “buried” reports of randomized controlled trials to the Cochrane Central Register of Controlled Trials (CENTRAL) published in The Cochrane Library. Some of these reports of trials may have been overlooked as a result of inconsistencies in indexing (indexing bias) [4], a lack of cover-to-cover indexing or because they have been published in journals not indexed in the major health care databases such as MEDLINE and EMBASE (database bias) [5] or in journals published in languages other than English (language bias) [6,7].

The Bahrain Branch of the UK Cochrane Centre is actively seeking to minimize these effects of bias by addressing problems of study identification through a comprehensive handsearching programme of journals published in the Arab world. This study sought to identify reports of randomized controlled trials by handsearching the Eastern Mediterranean Health Journal (EMHJ) which is indexed in EMBASE (from 2002) and in MEDLINE (from 1999) and which includes studies published in the Arab world as well as from elsewhere. As part of our study we wished to check whether database indexers assigned to index the EMHJ index the studies consistently in MEDLINE and EMBASE so that all the reports of randomized controlled trials published in this Journal (some of which will be from the Arab world) can be retrieved effectively. The study therefore also aimed to determine the added value of the handsearch in minimizing the effects of indexing bias by assessing the precision and sensitivity of the EMBASE index term “randomized controlled trial” and the MEDLINE Publication Types “randomized controlled trial” and “controlled clinical trial” as a means of confirming the reports found by handsearching.

Methods

All issues of the EMHJ (1995 to 2003) were searched by hand from cover to cover for reports of trials. These were classified as randomized controlled trials or controlled clinical trials according to the Cochrane eligibility criteria for reports of randomized trials in which participants were

definitely or possibly assigned prospectively to 1 of 2 or more alternative forms of health care using random allocation or some quasi-random method of allocation such as alternation, date of birth or medical record number. The handsearcher classified reports of trials as “randomized controlled trials” if the groups compared in the trial were established by random allocation. If the author(s) did not state explicitly that the trial was randomized but randomization could not be ruled out, the report was classified as a “controlled clinical trial”. Controlled clinical trial was also applied to quasi-randomized studies where the method of allocation was known but not considered strictly random (e.g. date of birth), and for possibly quasi-randomized studies.

Photocopies of the bibliographic details and of the pages describing the study design of the reports identified were sent to the UK Cochrane Centre for verification and to be processed for submission to the US Cochrane Center for publication in CENTRAL in The Cochrane Library.

EMBASE and MEDLINE, 2 major biomedical bibliographic databases, were also searched electronically to identify if the reports found by the handsearch were already included in either of these databases.

Results

Nine (9) reports of trials were identified, 7 randomized controlled trials and 2 controlled clinical trials. Only 3 (33%) of these were indexed as randomized controlled trials in EMBASE and 5 (56%) in MEDLINE. The added value of the hand-search over EMBASE was therefore 6 additional reports (67%) and over MEDLINE was 4 (44%).

The distribution of the reports of trials found by the handsearch by country of principal investigator was highest for Egypt and Iraq (3 papers), followed by Jordan (2). One report from a non-Arab country was found: Pakistan.

Of the 9 reports of controlled trials found by the handsearch, 4 were in EMBASE and 3 (33%) of these had been given the index term “randomized controlled trial”. The overall added value of the handsearch in relation to EMBASE (defined as the total number of reports of trials published in this Journal but not indexed or not indexed as randomized controlled trials in EMBASE and therefore not easily identified except through the handsearch) was 6 of 9 (67%).

Of the 9 reports found by the hand-search, 5 were in MEDLINE and these were all indexed with the appropriate Publication Types for randomized or controlled clinical trials. The added value of the handsearch in relation to MEDLINE, defined as the total number of reports of trials published in EMHJ but not indexed in MEDLINE and therefore not easily identified except through the handsearch of this Journal was 4 of 9 (44%).

Discussion

To minimize bias due to the selective availability of data, systematic reviewers need to identify as many relevant studies as possible to provide reliable evidence on which to base health care decisions. It has been shown previously that the identification of trials from bibliographic databases can be problematic [5]. In an effort to minimize the effects of lack of availability of appropriate indexing terms and inconsistencies in indexing (indexing bias), The Cochrane Collaboration has carried out systematic electronic searches of MEDLINE and EMBASE using extensive search strategies designed to be sensitive, i.e. to avoid missing reports of trials. The reports of trials, which were identified by an assessment of the titles and abstracts only, using these sensitive search strategies for MEDLINE and EMBASE are included in CENTRAL.

However, despite sensitive searching of electronic databases, it has been found that handsearching still provides additional reports of trials missed by the electronic searches [8,9]. The present study, based on the EMHJ, confirms this. It also reveals that for the reports of trials found by the handsearch and also indexed in EMBASE and MEDLINE, the indexing was consistent: 3 out of 4 in EMBASE were indexed as randomized controlled trials and 5 out of 5 in MEDLINE were indexed as randomized or controlled clinical trials. This finding contrasts with 2 recent studies [10,11] which compared the handsearch of journals published in Arab countries with electronic searches of EMBASE, where the indexing was found to be inconsistent and led to a number of reports being missed by the electronic search.

The reports missed by the electronic searches in our study almost all came from issues of the EMHJ which were published before the Journal was indexed. This finding confirms the importance of hand-searching journals to make available the reports of trials from issues of journals published before a journal was indexed in electronic databases, thereby minimizing the effects of “database bias”.

The handsearching programme of the newly established Bahrain Branch of the UK Cochrane Centre is already providing a valuable and unique contribution from the Arab region and elsewhere to the global effort by The Cochrane Collaboration. This will help to close the gap between the number of reports of trials that exist and the number of reports of trials accessible to authors of Cochrane reviews and others needing to make informed decisions about the effects of health care interventions. It will also contribute to a more comprehensive assessment of the biomedical research output of Arab countries.

The handsearch of this Journal in the programme should help reviewers to minimize the effects of publication bias by providing reports of trials not previously identified. Although EMHJ is indexed in EMBASE and MEDLINE, the handsearch has ensured that reports of trials will not remain “buried” because of incomplete coverage of this Journal in these databases.

Further research is required to assess the quality of the trials identified and to assess how many trials were duplicated. Additionally, comparisons need to be made in the quality of trials and the treatment effects of trials reported in Arabic with those reported in English to determine whether there might be differences which could lead to bias being introduced into reviews based exclusively on English language reports [12].

Conclusions

Handsearching can identify reports of trials not found by electronic searches. If resources for handsearching are limited, it is beneficial to target issues of journals published before a journal was indexed in major electronic databases or journals that are not indexed.

References

1. Tadmouri GO, Bissar-Tadmouri N. Biomedical publications in an unstable region: the Arab world, 1988–2002. Lancet, 2003, 362:1766.
2. El Ansari W, Soweid RAA, Jabbour S. Geography of biomedical publications. Lancet, 2004, 363:489.
3. Higgins JPT, Green S, eds. Cochrane and National Library of Medicine randomized controlled trial and controlled clinical trial criteria. Cochrane Handbook of Systematic Reviews of Interventions 4.2.5 [updated May 2005]; Appendix 5a. In: The Cochrane Library, Issue 3, 2005. Chichester, UK, John Wiley and Sons, 2005.
4. Lefebvre C, Clarke M. Identifying randomised trials. In: Egger M, Davey G Smith, Altman DG, eds. Systematic reviews in health care: meta-analysis in context. 2nd ed. London, BMJ Books, 2001:69–86.
5. Egger M, Smith GD. Bias in location and selection of studies. British medical journal, 1998, 316:61–6.
6. Marti J et al. Identificacion y descripcion de los ensayos clinicos publicados en revistas espanolas de medicina general e interna durante el periodo 1971–1995. [The identification and description of clinical trials published in Spanish journals of general and internal medicine during the period of 1971–1995]. Medicina clinica (Barcelona), 1999, 112(Suppl. 1):28–34.
7. Turp JC, Schulte JM, Antes G. Nearly half of dental randomized controlled trials published in German are not included in MEDLINE. European journal of oral science, 2002, 110(6):405–11.
8. Hopewell S et al. A comparison of handsearching versus MEDLINE searching to identify reports of randomized controlled trials. Statistics in medicine, 2002, 21:1625–34.
9. Hopewell S et al. Handsearching versus electronic searching to identify reports of randomized trials. Cochrane database of methodology reviews, 2002, Issue 4, No. MR000001.DOI:10.1002/14651858.MR000001.
10. Al Hajeri AA, Fedorowicz Z. A comparison of handsearching versus EMBASE searching of the Bahrain Medical Bulletin to identify reports of randomized controlled trials. Bahrain medical bulletin, 2005, 27(4):160–3.
11. Cathey JT, Al Hajeri AA, Fedorowicz Z. A comparison of handsearching versus EMBASE searching of the Annals of Saudi Medicine to identify reports of randomized controlled trials. Annals of Saudi medicine, 2006, 26(1):49–51.
12. Jüni P et al. Direction and impact of language bias in meta-analyses of controlled trials: empirical study. International journal of epidemiology, 2002, 31(1):115–23.