Bahareh Yazdizadeh,1 Ahmed Mandil,2 Sima Nikooee1 and Arash Rashidian2
1Knowledge Utilization Research Center, Tehran University of Medical Sciences, Tehran, Islamic Republic of Iran. 2World Health Organization Office for the Eastern Mediterranean, Cairo, Egypt. (Correspondence to: Ahmed Mandil:
Abstract
Background: For decades, WHO has been providing targeted funding for health research on priority areas of public health in the Eastern Mediterranean Region through different grant schemes.
Aims: This paper investigated the impact of WHO/EMRO’s funding schemes and factors facilitating or hindering such impact.
Methods: We assessed the impact of health research funded by WHO/EMRO during 2010–2018 from the health, economic, decision-making, and knowledge translation perspectives, emphasizing accountability and analysis, using the Payback framework, mixed-method approach (quantitative, qualitative), and triangulation.
Results: Principal investigators of 45 (45.9%) out of the 98 funded projects responded to the questionnaire. Almost all (88.0%) the 45 projects reported developing at least one decision-making document. Less than half reported producing peer-reviewed documents and conducting target group empowerment, while 24.0% said they secured research funds from other organizations. For 23 projects (51.0%), research results could have had a direct impact on health and on economy, and 25 (56.0%) projects conducted at least one active knowledge translation activity. Using multiple logistic regression, there was no significant association between the country of research and impact on decision-making and implementation of result if health or economic impact was expected.
Conclusion: To strengthen the impact of research, WHO/EMRO should embark on a series of interventions to guide and empower countries in the use of research results. Discrepancies between health research systems in the Eastern Mediterranean Region and differences in individual and organizational capacities in the different countries require targeted interventions.
Keywords: Research, small grant, Eastern Mediterranean, WHO/EMRO, health systems
Citation: Yazdizadeh B; Mandil A; Nikooee S; Rashidian A. Assessing the impact of small-research grants supported by WHO in the Eastern Mediterranean Region in 2010–2018. East Mediterr Health J. 2022;28(10):751–757. https://doi.org/10.26719/emhj.22.072 Received: 02/12/21; accepted: 14/08/22
Copyright © Authors 2022; licensee World Health Organization. EMHJ is an open access journal. This paper is available under the Creative Commons Attribution Non-Commercial ShareAlike 3.0 IGO licence (CC BY-NC-SA 3.0 IGO; https://creativecommons.org/licenses/by-nc-sa/3.0/igo).
Introduction
Since the 1940s and the inception of WHO, 3 of its core functions have been dedicated to health research. These include shaping the research agenda and stimulating the generation, translation and dissemination of valuable knowledge; articulating ethical and evidence-based policy options; and providing technical support, catalysing change, and building sustainable institutional capacity. Over the decades, several publications related to health research were produced by WHO and these could be accessed at: http://www.emro.who.int/rpc/publications/. The World Health Assembly (WHA) in its resolution WHA63.21 (WHO’s role and responsibilities in health research) called on Member States to establish governance mechanisms for research for health; to ensure rigorous application of good research norms and standards, including protection for human subjects involved in research; and to promote open dialogue between policymakers and researchers on national health needs, capacities, and constraints.
One of the activities to achieve WHO objectives is the provision of funds for local evidence. In the Eastern Mediterranean Region (EMR), WHO has been providing targeted funding for health research on priority areas of public health for decades. Applications are being received through open calls for proposals via different grant schemes, including: Research in Priority Areas in Public Health (RPPH), Tropical Disease Research – Small Grant Scheme (TDR-SGS), and occasionally improving Programme Implementation through Embedded Research (iPIER). Since 2020, new small grant schemes have been initiated focusing on migration health research, International Health Regulations (IHRs), and COVID-19-related priority research (1). These grant schemes are very popular, and every biennium, WHO receives about 10 times more proposals than can be supported in response to each call. These grant schemes are very important for producing local evidence, which has been emphasized in recent reports (2-4). EMR has implemented some interventions to empower countries to use the local evidence, such as asking principal investigators to share a “manuscript for consideration for publication”, instead of the final grant report, thus encouraging principal investigators to publish and share outcomes of their research with the scientific community. To improve these funding schemes, it is important to investigate their impact in the region and understand factors that facilitate or hinder their performance.
Generally, the impact assessment is conducted to facilitate advocacy, accountability, fund allocation, and analysis (5). As funding agencies are often at the core of strategies to improve “implementation to impact”(6), this assessment is very important for EMR to tailor it’s research policymaking and processes. Hence, this study aimed to investigate the impact of WHO/EMRO’s funding schemes and understand factors facilitating or hindering such impact as a first step towards refining grant programmes and tailoring capacity-building activities to countries context.
While studies have been carried out to map health research institutions and assess evidence to policy and practices and health research outputs in the EMR (9–11,18–20), we are not aware of any regional studies assessing the impact of WHO-funded research.
Methods
Health research impact assessment can be both top-down (ecologic studies) and bottom-up (case studies) (6). In top-down studies, the variables are collected and analysed in aggregated levels, such as evaluating the relationship between the expenditure on research and the impact of that research on reducing mortality burden attributed to specific diseases. In bottom-up studies, a set of research studies is selected, examining their impact over time via intense data collection (desk review and interview), where attribution of impact to specific research is more possible (although a big challenge in the top-down method).
The payback framework for health research impact assessment was developed in 1996 for conducting case study approach (7–9). It introduces the impact of health research in 5 categories: knowledge advancement, capacity-building, informing decision-making, as well as impact on health, society and economy. In 2009, detailed indicators for each of the areas covered by the payback framework were suggested by the Canadian Academy of Health Sciences, helping in the measurement of impact (10). Based on this framework and related indicators, several studies have been conducted to evaluate the impact of research in different countries, including Australia, Hong-Kong SAR China, and United Kingdom (11–14).
In this study we chose the case study approach using the payback framework as a base and mixed-methods approach (triangulation design with validating quantitative data) for data collection (12). The data collection tool included quantitative and qualitative questions, where narrative responses were used to validate quantitative responses. For example, in one question we asked, “Have the research outcomes been utilized in policymaking outside the health system (directly or indirectly)?”, and we asked principal investigator, “If your answer is yes, please explain how”.
The study targeted principal investigators of health research projects funded by WHO/EMRO during the period 2010 to 2018, under 3 grant schemes, i.e. iPIER (14 projects), TDR-SGS” (36 projects), and “RPPH” (48 projects). This period was selected to allow for sufficient time from the completion of the studies, as most research impact requires time to materialize.
To assess the impact of funded health research in the EMR, an updated version of a data collection tool used in previous studies (7,8) and had been designed according to the payback framework, was used. The tool included sections enquiring about project objectives; expected impact; impact domains, including “knowledge advancement; capacity-building achieved by conducting research (empowering target group during research, using the outcome of research to define the following projects, to facilitate securing of research funds from other organizations); impact on decision-making (producing systematic reviews, clinical practice guidelines, public health guidelines, health technology assessment, policy briefs, legislation, health policymaking); health and economic impact (using the outcome of research to change some decisions)”; knowledge translation activities; and stakeholder engagement. For each question, we asked about the relevance of the research to making the impact, firstly, and if the response was “yes”, we asked principal investigators to continue. These two last variables were chosen because they increase the probability of changing practice (13). For each claimed impact, we requested attachment of related documents to the online questionnaire.
An online version of the questionnaire was prepared, using SurveyMonkey platform and the link to the online tool was sent to all the principal investigators. Two reminders were sent at regular intervals to researchers who did not respond.
To verify responses to the quantitative questions, one of the researchers examined the narrative responses and the attached documents, then “perceived invalid responses to quantitative questions” were discussed with another research team member, to reach consensus.
Descriptive and analytical statistical methods were used for analysis. For the field “advanced knowledge”, attached documents were peer-reviewed and bibliometric information (impact factor and number of citations) checked using Scopus database. The aggregation variables were defined as follows:
Active knowledge translation interventions: including sending a summary or full report to potential users; providing findings to media reporters or participating in interviews; organizing a meeting with potential users; preparing or submitting outcomes in plain language for suitable audience; and taking necessary measures to commercialize the findings.
Impact on decision-making through decision-making document (IDMD): if research outcomes were used in one of the decision-making documents.
Impact on health or the economy (IHE): if the response to one of the 8 questions related to the real use of research results and the impact on health or economy was “yes”.
The relationship between some predictor variables and the two variables “IDMD” and “IHE” was investigated. Predictor variables were, the type of grant, the year of proposal approval, active knowledge translation intervention, and order-oriented research subject.
First, bivariate analysis was performed by using chi-square test (and if necessary Fisher’s exact test), then, the variables that had significant relationship were entered into multivariable analysis using logistic regression method.
Results
The questionnaire was sent to the principal investigators of 98 funded studies, of whom 45 responded (response rate: 45.9%). To clarify the selection bias, we compared the response and non-response cases based on the three variables, i.e. type of grant, year of grant approval, and country. Only year of grant approval was found to be statistically different, the response rate was more in the new approved proposals.
Respondents reported 10 peer-reviewed publications, 2 factsheets, 1 national roadmap, 1 national strategy document, and 1 preprint paper. For the 10 peer-reviewed publications, the mean citation was 2.83 (minimum 0 and maximum 8) and the mean impact factor of journals that published the works was 2.87 (minimum 2 and maximum 3.29)
We observed more visible impact on capacity-building, as summarised in Table 1. Twenty-two research studies (out of 45) reported enhanced capacity (change in awareness, attitudes, skills) of the target groups as a result of the funded study.
Fifteen research studies (out of 45) were conducted in response to expressed needs of the policymakers. Tables 2 and 3 present the results of other questions in this section. Despite being relevant, the results of 10 studies were not used in any policymaking. In 17 cases, despite being relevant, the results were not used in national policies and in 3 cases they were used in the local organizational policy. Generally, after omitting missing values, 36 studies had at least been relevant enough to use their results for one of three levels of policymaking. In 20 cases, they were used for decision-making in at least 1 level.
Respondents for 23 projects (out of 45) reported that the results of their research could have had direct impact on health if the stakeholders used them, of which 13 research studies were reportedly used (56.5% of 23 projects). For 23 research studies (51%), research results could have had direct impact on economy if the stakeholders used them, of which 7 (30.4%) research results were reported to have been used.
Generally, 29 projects (64.4%) could have had direct impact on health or economy; for 14 projects (48%) the results were reported to have been used. For 25 research studies (out of 45), at least one active knowledge translation activity was reported to have been performed; but for 12 projects (27%) no knowledge translation activity was done. Regarding stakeholder engagement, for 26 studies (57.8%), the identified stakeholders did not play a role in research question formulation and proposal writing.
Using Chi-squared test, the relationship between IDMD and health or economic impact with predictor variables was examined. None of the variables was found to be associated with the impact on decision-making, however, in terms of the impact on health or economy, the variable related to conducting research in response to policymaker demand was found to be significantly linked (p-value: 0.002 (Table 4). All variables which their P-values were less than 0.1 were entered to the logistic regression (14) (Table 5). No significant association between the country of research and impact on decision-making and implementation of result if health or economic impact was expected.
Discussion
The objective of this study was to assess the impact of the research projects that were funded by WHO/EMRO from 2010 to 2018 to enable improvements in research policymaking and processes in this organization. There was only a modest potential impact in generating (and sharing) new knowledge, as only 10 research studies (about 1 in 5) reported publishing their results in peer-reviewed publications. The impact on capacity-building was more prominent, as half of research studies reported empowerment of their target groups. The main premise of the enquiry in our study was that condcuting participatory research and integrated collaboration (in which the target group is included in the research team) will empower target groups and increase the probability of using research results (15). A recent WHO study on EMR health research institutions shows that training on this kind of research is rare in the region (16). It is therefore necessary to implement special capacity-building on how to conduct participatory research.
One in 4 of the research teams reported that the WHO small grant helped them in receiving subsequent research fundings. This is important in several ways, as it helped gain the trust of relevant organizations (getting credit from providers or executive agencies), and the research results have led to new research questions (which are needed to solve health problems).
Regarding impact on decision-making, reported documentation of research outcomes to inform decision-making was very low, usually assessed by the production of decision-making documents in respective countries (11). Less than half of the respondents reported publishing a peer-reviewed article. Further efforts are required to link research to policy and enhance evidence-informed policymaking. A little above half of the respondents (55%) reported using the research outcomes for policymaking.
About two-thirds of respondents (64%) reported that their research outcomes had potential to make direct impact on health or economy, while about half (48%) reported that their research outcomes were used to support health and wellbeing.
Regarding impact on decision-making and impact on health and organization, the relevance of research proposals to the impact observed was related to the funding agency’s vision, mission and criteria to accept the proposals, and an impact is looked at as a consequence of contextual factors within each country of implementation. Based on the result of this study, it is essential to change some criteria for accepting proposals to promote the use of evidence in decision-making in countries.
On the factors influencing the probability of using research for decision-making, our findings demonstrate the importance of conducting research in response to decision-makers’ needs. One-third of the research studies included in this assessment were conducted in response to policymakers’ needs, and the results were more likely to be adopted for subsequent decisions. About two-thirds of respondents (58%) reported that the stakeholders did not play a role in the formulation of the research question(s) and the writing of the proposal. It therefore seems that it is necessary to improve the research priority-setting in countries and in EMR, and encourage stakeholders to get involved in shaping the research question.
One major limitation of our study was that more than half of the study population (those receiving WHO/EMRO small grants) did not respond to our online survey, in spite of repeated reminders. If non-response could be explained by recall difficulties, then the selection bias by the variable “time” would result in underestimating the impact. If non-response could be attributed to non-impact, then the selection bias has forced overestimation. Another limitation was that during the verification of responses, we found that respondents did not understand some questions properly. A third limitation is insufficient evidence required to compare the health research systems of countries in the region. To be able to assess research impact and its influencing factors and make appropriate recommendations, countries should be compared in terms of health system building blocks, relationship between the health research system and health system, and the situation of KTE components. The main strength of this study was the data collection tool, which included questions about the relevance of the research to making an impact, and the qualitative questions about claimed impact, which helped in verifying the quantitative responses.
To strengthen implementation and the impact of research outcomes, we recommend the following actions:
An integrated knowledge translation approach should be observed when approving proposals for funding, i.e. principal investigators should ensure that stakeholders will actively participate in the research cycle from the beginning to the end, and that its outcomes will indeed be used by relevant health policymakers.
Research questions should be based on national, provincial or public health priorities. Therefore, WHO/EMRO’s calls for proposals, including the small grant schemes, should relate to national priorities, taking into account outcomes of national research priority setting exercises, like the ones conducted in Jordan and Pakistan (17,18).
To strengthen the use of research-generated evidence by health systems, we recommend that countries should regularly conduct research impact assessment and interventions to increase the skills of researchers to identify challenges of the health systems and improve their skills regarding the importance of active participation of stakeholders.
To identify factors affecting the implementation of research outcomes, in-depth studies of the research cycle (from the time of planning to the implementation/non-implementation of outcomes) are needed (qualitatively and quantitatively).
It is important to review and compare studies that had impact on health and/or economy and/or decision-making (using the decision-making documents).
We need to conduct situation analysis of the national health research systems in each country, which would help identify gaps, challenges, and priorities and allow better use of resources and facilitate evidence-based decision-making.
Acknowledgement
We thank the principal investigators of WHO-funded grants in the EMR during 2010–2018, who responded to our data collection tool, and hence supported the outcomes and recommendations of this study. We also acknowledge the contributions of Mr Ahmed Elarousy, Mr Hosam Younes, and Ms Fatma Abdel-Megeed for their assistance in developing/testing the online data collection form.
Funding: This study was supported by the Division of Science, Information & Dissemination, WHO/EMRO, Cairo, Egypt.
Competing interests: This study was ethically cleared by the EM-Research Ethics Review Committee in June 2020.
Évaluation de l'impact des petites subventions à la recherche financées par l'OMS dans la Région de la Méditerranée orientale de 2010 à 2018
Résumé
Contexte : Depuis des décennies, l'OMS fournit un financement ciblé pour la recherche en santé dans des domaines prioritaires de la santé publique dans la Région de la Méditerranée orientale à travers différents programmes de subventions.
Objectifs : Le présent article examine l'impact des programmes de financement du Bureau régional de l'OMS pour la Méditerranée orientale et les facteurs qui facilitent ou entravent cet impact.
Méthodes : Nous avons évalué l'impact de la recherche en santé financée par le Bureau régional de l'OMS pour la Méditerranée orientale entre 2010 et 2018 du point de vue de la santé, de l'économie, de la prise de décision et de l'application des connaissances, en mettant l'accent sur la responsabilisation et l'analyse, au moyen du cadre de récupération, de l'approche mixte (quantitative, qualitative) et de la triangulation.
Résultats : Les chercheurs principaux de 45 (45,9 %) des 98 projets financés ont répondu au questionnaire.
La plupart de ces chercheurs (88,0 %) ont indiqué avoir élaboré au moins un document de prise de décision. Moins de la moitié d'entre eux ont déclaré avoir produit des documents évalués par des pairs et avoir mené des actions d'autonomisation des groupes cibles, tandis que 24,0 % ont signalé avoir obtenu des fonds de recherche par le biais d'autres organisations. Les résultats de recherche de 23 projets (51,0 %) ont pu avoir un impact direct sur la santé et l'économie, et 25 projets (56,0 %) ont permis de mener au moins une activité d'application des connaissances. À l'aide d'une régression logistique multiple, aucune association significative n'a été trouvée entre le pays sur lequel porte la recherche et l'impact sur la prise de décision ainsi que sur la mise en œuvre du résultat lorsqu'un impact sur la santé ou l'économie était attendu.
Conclusion : Pour renforcer l'impact de la recherche, le Bureau régional de l'OMS pour la Méditerranée orientale devrait entreprendre une série d'interventions visant à orienter et à autonomiser les pays dans l'utilisation des résultats des recherches. Les disparités entre les systèmes de recherche en santé de la Région de la Méditerranée orientale et les différences dans les capacités individuelles et organisationnelles des différents pays nécessitent des interventions ciblées.
تقييم أثر المِنَح البحثية الصغيرة التي تدعمها المنظمة في إقليم شرق المتوسط في المدة ما بين عامَيْ 2010 و2018
بهارة يزدزاده، أحمد منديل ، سيما نيكوي، أرش رشيديان
الخلاصة
الخلفية: قدَّمت منظمة الصحة العالمية، على مدى عقود، تمويلًا مُستهدَفًا للبحوث الصحية في المجالات ذات الأولوية للصحة العامة في إقليم شرق المتوسط من خلال خُطَط مختلفة للمِنَح.
الأهداف: هدفت هذه الدراسة الى تقصي أثر الخطط التمويلية للمكتب الإقليمي لشرق المتوسط، التابع لمنظمة الصحة العالمية، وعوامل الارتقاء بهذا الأثر أو ما يعوقه.
طرق البحث: أجرينا تقييمًا لتأثير البحوث الصحية التي موَّلتها منظمة الصحة العالمية/ المكتب الإقليمي لشرق المتوسط، خلال المدة ما بين عامَيْ 2010 و2018، من منظور الصحة والاقتصاد وصُنع القرار وترجمة المعلومات، مع التأكيد على المساءلة والتحليل، باستخدام إطار العائد ونهج متعدد الأساليب (كمي ونوعي)، والتثليث.
النتائج: لقد شارك في الاستبيان الباحثون الرئيسيون من 45 من أصل 98 مشروعًا ممولًا (45.9%). وأفادت جميع المشاريع الخمسة والأربعين تقريبًا (88.0%) أنها وضعت وثيقة صُنع قرار واحدة على الأقل. وأفاد أقل من نصف المشاركين بوضع مستندات خضعت لمراجعة الأقران، وإجراء تمكين المجموعة المُستهدَفة، في حين قال 24.0% إنهم حصلوا على تمويل من منظمات أخرى. وفي حالة 23 مشروعًا (51.0%)، كان من الممكن أن يكون لنتائج البحوث تأثير مباشر على الصحة والاقتصاد، بينما نفَّذ 25 مشروعًا (56.0%) نشاطًا واحدًا على الأقل من أنشطة ترجمة المعلومات الفعالة. وباستخدام الانحدار اللوجستي المتعدد، تبيَّن عدم وجود ارتباط يُعتدُّ به بين بلد البحث والأثر الواقع على عملية اتخاذ القرار وتنفيذ النتائج، إذا كان الأثر الصحي أو الاقتصادي مُتوقَّعًا.
الاستنتاجات: لتعزيز أثر البحوث، ينبغي للمكتب الإقليمي لمنظمة الصحة العالمية لشرق المتوسط بدء سلسلة من التدخلات لتوجيه البلدان وتمكينها من الاستفادة من نتائج البحوث. وعلاوةً على ذلك، فإن التفاوت بين نُظُم البحوث الصحية في إقليم شرق المتوسط، والتفاوت في القدرات الفردية والتنظيمية باختلاف البلدان، تتطلَّب تدخلات مُوجَّهة.
References
1. Information, Evidence and Research, Eastern Mediterranean Regional Office, World Health Organization , http://www.emro.who.int/about-who/regional-director/dr-arash-rashidian-director-information-evidence-and-research.html
2. The world health report 2013: research for universal health coverage.
3. Hanney SR, Kanya L, Pokhrel S, Jones TH, Boaz A. How to strengthen a health research system: WHO’s review, whose literature and who is providing leadership? Health Research Policy and Systems. 2020;18(1):72.
4. Alla K, Hall WD, Whiteford HA, Head BW, Meurk CS. How do we define the policy impact of public health research? A systematic review. Health Res Policy Syst. 2017;15(1):84.
5. Guthrie S, Wamae W, Diepeveen S, Wooding S, Grant J. Developing a research evaluation framework: RAND Corporation; 2013.
6. Hanney SR, Ovseiko PV, Graham KER, Chorzempa H, Miciak M. A systems approach for optimizing implementation to impact: meeting report and proceedings of the 2019 In the Trenches: Implementation to Impact International Summit BMC Proceedings. 2020;14(6):10.
7. Yazdizadeh B, Majdzadeh R, Janani L, Mohtasham F, Nikooee S, Mousavi A, et al. An assessment of health research impact in Iran. Health Res Policy Syst. 2016;14(1):56.
8. Yazdizadeh B, Mohtasham F, Velayati A. Impact assessment of Iran’s health technology assessment programme. Health Res Policy Syst. 2018;16(1):15.
9. Tadmouri GO, Mandil A, Rashidian A. Biomedical and health research geography in the Eastern Mediterranean Region. East Mediterr Health J. 2019;25(10):728-43.
10. Rashidian A, Mandil A, Mahjour J. Improving evidence informed policy-making for health in the Eastern Mediterranean Region. East Mediterr Health J. 2018;23(12):793-4.
11. El-Jardali F, Mandil A, Jamal D, BouKarroum L, El-Feky S, Nour M, et al. Engagement of health research institutions in knowledge translation in the Eastern Mediterranean Region. East Mediterr Health J. 2018;24(7):672-9.
12. JW. C, Plano Clark. VL. Designing and Conducting Mixed Methods Research Second Edition: Sage publication; 2007.
13. LaRocca R, Yost J, Dobbins M, Ciliska D, Butt M. The effectiveness of knowledge translation strategies used in public health: a systematic review. BMC Public Health. 2012;12(1):751.
14. Jewell N. Statistics for Epidemiology. 1st Edition ed: Chapman and Hall/CRC; 2003.
15. Graham ID, McCutcheon C, Kothari A. Exploring the frontiers of research co-production: the Integrated Knowledge Translation Research Network concept papers. Health Res Policy Syst. 2019;17(1):88.
16. Mandil A, El-Jardali F, El-Feky S, Nour M, Al-Abbar M, Bou-Karroum L. Health research institutional mapping: an Eastern Mediterranean Regional perspective. East Mediterr Health J. 2018;24(2):189-97.
17. Mandil A, Rashidian A, Hajjeh R. Health research prioritization: global and regional perspectives. East Mediterr Health J. 2020;26(3):254-6.
18. Mandil A, Motevalian SA, Altaf A, Obeidat R, Muthu N, Afzal S, et al. Research on health priorities: a regional agenda and its implementation in Jordan and Pakistan. East Mediterr Health J. 2021;27(4):319-20.
