Search Search

Framework for assessing stewardship of the oral health system in Islamic Republic of Iran

PDF version

B. Tahani,1 S. Yazdani,2 M.H. Khoshnevisan,1,6 P. Dugdale,4 S. Siddiqi 5 and A. Ebn Ahmady 1

إطار لتقييم القوامة في نظام صحة الفم في جمهورية إيران الإسلامية

باهاره تهاني، شهرام يازداني، محمد خوشنيفيسان، بول دوغدالي، ثمين صديقي، أريزو ابن أحمدي

الخلاصة: وضعت هذه الدراسة إطار عمل لتقييم وظيفة القوامة في نظام صحة الفم في جمهورية إيران الإسلامية. وتم استخدام الطريقة المعدلة لمؤسسة البحث والتقييم التابعة لجامعة كاليفورنيا في لوس أنجلوس على خطوتين للربط بين البيِّنات المنشورة والرأي الجماعي للخبراء. وبعد إجراء مراجعة شاملة للنّشريات، تم استخلاص النصوص الخاصة بالسياسات المتعلقة بعنصر القوامة كمقاييس مقترحة، وتم تصنيفها وفقاً للوظائف الفرعية الست للقوامة (المساءلة، والتوجهات الاستراتيجية المحددة؛ وترتيب أهداف السياسيات؛ والهيكل التنظيمي؛ والتنظيم، والقيادة المشتركة بين القطاعات، وإنتاج المعلومات). ثم قام خمسة مبلِّغين رئيسيين بتقييم مدى ملاءمة المعايير البالغ عددها 85 مقياساً وفقاً لمقياس ليكرت ذي النقاط الخمس. وقد شكلت المقاييس التي حصلت على أعلى ترتيب والبالغ عددها 38 مقياساً، بما فيها مقياسان على الأقل في كل من الوظائف الفرعية الست، مجموعة من المقاييس المقترحة لتقييم القوامة الحالية لنظام صحة الفم. وسيتم تقديم تقرير عن هذه الأداة بشكل منفصل.

ABSTRACT This study designed a framework for assessing the stewardship function of the oral health system in Islamic Republic of Iran. The modified RAND Corporation/University of California Los Angeles (RAND-UCLA) Appropriateness Method was used in a 2-step process that combined literature evidence and the collective judgement of experts. After a comprehensive literature review, policy instruments related to stewardship components were extracted as candidate standards and categorized according to the 6 sub-functions of stewardship (accountability; defining strategic direction; alignment of policy objectives and organizational structure; regulation; intersectoral leadership; and generation of intelligence). Five key informants then rated the appropriateness of the 85 standards on a 5-point Likert scale. The 38 highest ranked standards, including at least 2 standards in each of the 6 sub-functions, formed a set of proposed standards for evaluating the current stewardship of oral health system. Piloting of the instrument will be reported separately.

Cadre d’évaluation de la gouvernance du système de santé bucco-dentaire en République islamique d’Iran

RÉSUMÉ La présente étude a conçu un cadre d’évaluation de la fonction de gouvernance du système de santé bucco-dentaire en République islamique d’Iran. La Méthode modifiée de détermination de la pertinence des indications de la Rand Corporation et de l’Université de Californie Los Angeles (Rand/UCIA) a été utilisée, dans un processus en deux étapes, combinant les données probantes de la littérature et le point de vue collectif d’experts. Après un examen exhaustif de la littérature, des instruments de politiques liés aux composantes de gouvernance ont été extraits en tant que normes candidates puis classés en catégories selon les six sous-fonctions de gouvernance (responsabilisation, définition d’une orientation stratégique, harmonisation des objectifs politiques et de la structure organisationnelle, réglementation, direction intersectorielle et production de données). Cinq informateurs clés ont ensuite attribué une note à la pertinence de 85 normes sur l’échelle de Likert en cinq points. Les 38 normes les mieux notées, dont au moins deux normes dans chacune des six sous-fonctions, ont formé un ensemble proposé pour l’évaluation de la gouvernance actuelle du système de santé bucco-dentaire. Le pilotage de l’instrument fera l’objet d’un rapport distinct.

1Department of Community Oral Health, School of Dentistry; 2Educational Development Centre; 6Preventive Dentistry Research Centre, Research Institute of Dental Sciences, Shahid Beheshti University of Medical Science, Tehran, Islamic Republic of Iran (Correspondence to M.H. Khoshnevisan: This email address is being protected from spambots. You need JavaScript enabled to view it., This email address is being protected from spambots. You need JavaScript enabled to view it.).
4Centre for Health Stewardship, Australian National University, Canberra, Australia. 5Information Evidence and Research, World Health Organization Regional Office for the Eastern Mediterranean, Cairo, Egypt.
Received: 16/11/12; accepted: 23/04/13
EMHJ, 2014, 20(2):73-81

Introduction

Population health can be linked directly to the effectiveness of health-care systems in performing their functions [1]. This recognition of the importance of maintaining an efficient and effective health-care sector has driven efforts to develop suitable metrics to monitor the performance of health systems. Publication of the World health report 2000 on improving the performance of health systems [2] provoked a wide range of debate and criticism about approaches and methods for assessing health systems’ performance, both nationally and internationally [3]. Leadership and governance (also known as “stewardship”) [4] is possibly the most multifaceted and crucial function of any health system, but is usually neglected in performance assessment [5] Stewardship is basically “a function of governments responsible for the welfare of populations and concerned about the trust and legitimacy with which its activities are viewed by the general public” [2]. The need for pragmatic research into stewardship and governance in health through descriptive studies of the stewardship tasks, approaches and styles has been emphasized internationally. However, the methods for systematically assessing this key function of health systems are limited.

The oral health-care system is a small yet important component of the wider health-care system. In 10 priority action areas for global oral health identified by the World oral health report 2003, oral health systems were listed as one of the areas with high priority [6]. To the best of our knowledge, little research has addressed systematic measures of oral health systems. Often these systems have been described on the basis of only one or two characteristics [7,8], without any effort to value judge and to benchmark their structural and functional performance. Within these frameworks, the building block of stewardship in oral health care tends to be neglected.

Those responsible for stewardship often use consensus standards to oversee the system they are responsible for. In this paper, we turn this approach around, and will show how a set of standards can be designed for the purpose of evaluating and guiding the stewardship function itself. These standards are intended to represent those actions and capacities that will be necessary for an oral health system in a developing country to perform at the highest possible levels. In this study—part of a larger project on oral health system stewardship—we applied a modified version of the RAND Corporation/University of California Los Angeles (RAND-UCLA) Appropriateness Method [9] to design a framework to assess the stewardship of the oral health-care system in the Islamic Republic of Iran.

Methods

Study design and setting

This 2-step standards development study was conducted in 2012 in collaboration with the World Health Organization (WHO) Collaborating Centre for Training and Research in Dental Public Health at Shahid Beheshti University of Medical Sciences, Tehran, Islamic Republic of Iran.

Standard-setting is usually based on collective judgements and relies on experts’ definition of what constitutes effective practice [10]. Evidence about the past performance of similar programmes is also a sound basis for setting standards. Among the different qualitative methods seeking to achieve these objectives, we chose the RAND-UCLA Appropriateness Method as the best method for systematically combining evidence and the collective judgement of experts [9]. This method requires a comprehensive literature review to assess the existing evidence about a specific subject and to develop the indicators, followed by collating experts’ opinions about them.

Step 1: Identifying the candidate standards based on current evidence

In the first step we reviewed performance reports from health systems, policy documents, recommendations from international organizations and documents about national case studies, published mostly on the Internet. The purpose was to find evidence of effective policies that were associated with high levels of health systems’ performance, subsumed under the meta-function of stewardship. We compiled these documents by searching official international health-policy websites (e.g. WHO Library, European Observatory on Health Systems and Policies), as well as electronic databases such as PubMed and Google Scholar. Specific keywords relevant to stewardship and its 6 sub-functions, as defined by Travis et al. [11], in oral health systems or health systems as a whole were used. The publications were mostly in English language, but some international reports translated into Persian were also used. The compiled documents were critically reviewed by the 2 main authors of this study to identify policy statements in the scope of stewardship that could potentially improve the intermediate (access, efficiency, quality) and final goals of health systems [4]. Given that the private sector delivers most of the oral health care in the Islamic Republic of Iran, and that most common oral health diseases are largely preventable, we also considered these characteristics in choosing the relevant policies.

To manage the information, we used Microsoft Office Access software, version 2007. We entered all the policy statements elicited as standards into a pre-designed Access template. Likewise, the level of evidence (e.g. international recommendations, regional evidence or national case study), the mandated verb tense of the standard (e.g. “must” or “should”), and the final or intermediate goals of the health system affected by that standard were all imported into the template. The information was coded in order to facilitate grouping and sorting of the documents.

To establish the list of candidate standards from the database, we then identified those statements that were specific to oral health systems. The identification of candidate standards was based mainly on the evidence of their effectiveness in achieving oral health system goals (outputs and outcomes). Other generic standards that did not refer to oral health were then modified in order to describe the specific nature of the oral health system.

Step 2: Selecting proposed standards through expert consensus

In the second step, to achieve consensus about the standards, the candidate standards were categorized by a panel of experts based on sub-functions of stewardship according to the definitions of WHO [1] and Travis et al. [11]. Eventually a questionnaire was developed that included all candidate standards. The questionnaire was written in Persia language but some of the important words were written also in English as footnotes for clarification. The sub-functions comprised the following:

  • generating and disseminating intelligence;
  • defining strategic policy direction;
  • making use of regulation to steer health system performance;
  • exerting influence over all related sectors via intersectoral leadership;
  • ensuring alignment of policy objectives and organizational structure;
  • ensuring accountability.

In order to reveal how appropriate each proposed standard was, the following 2 main questions were designed for each item:

  • Does each candidate standard appropriately describe that sub-function of stewardship under which this standard is subsumed (relevance)?
  • Is each candidate standard important enough to be considered for measuring that sub-function (importance)?

The appropriateness of the standards were considered by a multidisciplinary group of experts, who had been involved in health-system policy development at the Iranian Ministry of Health and Medical Education (MOH), or had experience in oral health system decision-making as well as dental public health research activities at the national level, and were familiar with the organization of oral health systems. The experts were purposively recruited to this project [12], as for qualitative studies that select “the most knowledgeable individuals in the relevant field” [13]. We found 7 available qualified persons who had all the prerequisite characteristics (significant experience in policy development, and informed about the organization of the oral health system in the Islamic Republic of Iran). The electronic copies of the questionnaire were mailed with a covering letter explaining the project’s specifications, including information on a contact person in case more information was necessary. Five out of 7 invited experts agreed to participate in the study.

The key experts were asked to rate the importance and relevance of the candidate standards on a 5-point Likert-type scale (1 = “strongly agree” through to 5 = “strongly disagree”). Points for each question were summed up to give a simple sum of score for each standard, which could be in a range between 5 and 25. For ease of presentation we then subtracted 4 from each score sum, resulting in a score range of 1 (most appropriate) to 21 (least appropriate). We considered an appropriateness score of 4 as the cut-off point in acceptance of each standard. Therefore, any statement with score 4 or below for both importance and relevance were selected as a proposed standard. This cut-off score was based on the best judgement of the main investigators of this study and the available recommendations about defining cut-off scores. Although estimates of the cut-off score is an arbitrary decision that comes from a subjective judgement of experts, usually when 70% of the standards are of the “must” category (according to their mandate verb) then it may be appropriate to set the cut-off score marks at least 70% [14].

At the end of each sub-function section, experts were also asked to rate the completeness of standards in reference to their sub-function; this was scored on a Likert scale (as described above). Finally, in an open question, respondents were asked to suggest any additional standards, if any.

Results

There were 85 candidate standards: 4 for accountability; 13 for defining strategic direction; 12 for alignment of policy objectives and organizational structure; 41 for regulation; 11 for intersectoral leadership; and 4 for the generation of intelligence sub-functions. The final list of proposed standards (Table 1) was developed according to the scores received by the experts.

Importance of standards

Experts achieved general agreement about importance for 64 out of 85 proposed standards. In the sub-function of regulation, 6 out of 41 candidate standards were scored as unimportant. These were mostly about the role of the MOH in taking legal action in dealing with complaints against providers, and its role in managing corruption in the oral health system. Examples are, “MOH must determine the process of claiming and dealing with complaints for each of the regulations” or “MOH must clarify and establish financial and administrative corruption instances in an oral health system”. In the sub-function of “defining strategic policy direction”, 5 out of 13 candidate standards were scored as unimportant. These were generally about strengthening evidence-based policy-making through engagement of professional groups. Examples are, “MOH must support the formation of professional groups responsible for decision-making in order to strengthen the evidence-based policy-making in oral health system” or “MOH must establish and refresh national oral health policy priorities based on reliable intelligence, global improvements, governing values, goals and consensus of main stakeholders and experts”.

Relevance of standards

In total, 53 out of 85 candidate standards were rated as relevant. The most disagreement among experts about the relevance of candidate standards was observed in the sub-function of regulation; 16 out of 41 standards did not achieve the minimum level of appropriateness. These statements were mostly about the role of government in defining regulations and legislation for insurance activities, such as “MOH must clarify regulations for setting premiums in dental insurance plans” or “MOH must use regulation for specifying minimum oral health services that must be included in the benefit package”. Disagreements were also observed about the role of the MOH in controlling the private sector by regulating prices and devising antitrust regulations, for example “MOH must have regulations in place to control the prices of interventions” or “MOH must establish and enforce antitrust regulations to reduce private dental practice monopolies”.

Final list of standards

A total of 38 standards gained the minimum level of appropriateness in both relevance and importance. There were 2 standards for accountability, 8 for defining strategic direction, 5 for alignment of policy objectives and organizational structure, 18 for regulation, 3 for intersectoral leadership and 2 for generation of intelligence sub-functions as shown in Table 1.

Completeness of the standards

Asked to give their opinions about whether the standard encompassed all the performance measures in each subsection, we found full agreement among experts for almost all of them. Experts provided some recommendations for example to “consider strategies to improve the partnership between public and private sectors”.

Discussion

Many countries around the world are considering approaches for assessing the performance of their health systems to improve their efficiency and responsiveness to the needs of the population [15]. Despite the fundamental role of health system stewardship/governance in attaining national, regional and global goals in health care, little systematic effort has been made towards a proper assessment of this. The limited frameworks available—such as the 6 basic aspects of governance suggested by the World Bank or the United Nations Development Programme’s principles of good governance [16,17]—are mostly developed for the analysis of national governance and are not adequately designed for assessing the governance of health systems. There are other proposed frameworks for measuring the health system stewardship or governance—such as the framework applied in developing countries [18] or the framework proposed to assess national health ministries in the WHO European Region [19]. These frameworks basically include broad, qualitative questions subsumed under the various tasks of stewardship/governance, without any judgement about the content or quality of stewardship activities that would allow their success at the country level to be measured. Other instruments—such as the Essential Public Health Functions instrument established by the Pan American Health Organization [20]—are not designed to assess stewardship per se. Functional frameworks such as the one used by the WHO can more deeply analyse the major aspects of a health system and its functional components including stewardship. It should be noted that these frameworks mostly use outcome measures and do not determine the completeness and appropriateness of stewardship.

Accurate collection of health systems indicator data (including outcome and output data) relies on the existence of reliable and well-established health information systems [21]. Also, to undertake a “diagnostic journey” to identify the causes of unsatisfactory results in health systems, valid and reliable data should be not only available but also accessible [22]. So far, in many developing countries [23] including the Islamic Republic of Iran, adequate, valid basic data are not collected routinely. Evidence concerning outcomes such as financial protection and client satisfaction are especially limited. Even when data are available, many of these countries have a limited capacity to use and manage the data appropriately [24] and have difficulties in linking the functional performance of health systems with the outcomes [25].

We therefore hypothesized that using input measures describing functional components of a health system might be an alternative to using outcome measures. What is needed is to make sure that these link to the ultimate goals of health systems. In contrast to the “black box” misconception about health systems that claims that the fundamental mechanisms of health systems are too complicated to understand [26], we believe that an adequate body of knowledge about specific interventions to improve the performance of health system is available [26], which in turn could be helpful in generating processes of shared learning.

The frameworks that are currently used for assessing oral health systems are mostly descriptive [7,8], presenting the status of the workforce, costs, models of provision and some of the specifics of oral health programmes without a discussion about how the systems work or how to use policy to improve their functions. Stewardship is usually neglected. In this study we designed a set of proposed standards for measuring the performance of an oral health system based on stewardship. To our knowledge this is the first study to design a framework for assessing the stewardship of oral health systems. The method of developing these standards was a combination of evidence-based and expert consensus development. By merging the best knowledge and experience of experts with the available data, consensus studies are used to compensate for the lack of conclusive data. The expert judgement of those who participated in eliciting the candidate standards through comprehensive article review and their consensus might be considered as the first step in assuring the content validity. It was based on the available evidence that measures developed by consensus techniques have face validity and those based on rigorous evidence have content validity. Also, there was some evidence of predictive validity of indicators developed by the RAND-UCLA Appropriateness Method [27].

According to the findings of this study, by using “importance” and “relevance” as the main facets of appropriateness, nearly half of our initial standards attained the proposed level of appropriateness.

Most of the standards which garnered disagreement were in the sub-function of regulation. It seems that experts did not agree with the role of the Iranian MOH in devising regulations to deal with complaints against providers, nor for determining instances of administrative and fiscal corruption. Some of them thought it might be the responsibility of other organizations such as the judiciary body. In most countries, investigation of medical malpractice or misconduct cases is assigned to medical professional societies because it is more acceptable to the profession. To prevent the problem of so-called “regulatory capture” governments should consider regulating the regulators by, for example, increasing the transparency of the review process and defining ways of increasing the accountability of professionals to the public [28]. However, if the process for imposing penalties is enforced by police, prosecutors or judges, there is the potential for personal preferences to determine how the enforcement should occur [22].

Some of the disagreement among experts concerned the standards on the role of the MOH in regulating dental insurance plans. This might be due to recent government decisions in the Islamic Republic of Iran in which the responsibility for regulating health insurance (including dental insurance) has been devolved mostly to the Supreme Council for Health Insurance, which is a parastatal organization under the supervision of the Ministry of Welfare and Social Security (MOW). According to the 5th National Developmental Plan and Policy [29], the state is required to transfer the Supreme Council for Iranian Health Insurance from MOW and keep it under the supervision of the MOH with a combination of members from both organizations. Therefore, in view of the importance of the regulatory role of the MOH and the recent changes proposed for the regulatory structure of health insurance, it seems that new standards for regulating dental health insurance might be considered, even though they are not currently in the oral health system.

Other disagreements centred on the role of the MOH in controlling the private sector by for instance regulating prices and devising antitrust regulations. Despite the important role of the private sector in health systems, there are some problems with their performance. Undesirable characteristics of free health markets, such as asymmetric information, questionable moral behaviour and externality, lead to market failure and result in inequities in health care [28]. Therefore, regulation is considered to be a potential tool to control and overcome these problems, especially in mixed systems such as the oral health system of Islamic Republic of Iran, where most of the services are provided by the private sector [30].

Other disagreements among experts were about the standards of the decision-making cycle and the necessary role of MOH in enrolment of various stakeholders and sources of evidence. This might be due to the executive challenges that this Ministry is faced with in managing and steering the implementation of formulated policies. The MOH plays the main role in policy-making and governing the health sector at the macro level. Occasionally, other entities, such as the MOW, the Medical Council and insurance companies, have parallel but uncoordinated plans in cooperation or in opposition to the MOH, which restrict its supervisory capacities [29]. Nevertheless, for a health system to overcome traditional “top-down” governance, it might be necessary to engage the state in networked governance across different organizations [31]. This network should provide the grounds for participation of all relevant stakeholders from both the state and non-state agencies in designing policies [32].

We believe that the characteristics of the instrument developed in this study will allow us to determine the degree of fulfilment of each standard. This can be done by using an ordered response scale, which is in accordance with the WHO recommendations for development of instruments to assess stewardship, and by using additional questions to identify the sub-functions for each stewardship to be administered for selected key health system actors and to be answered using ordered, categorical response scales [11].

This study is the first to design a framework for assessing stewardship of an oral health system, and further work will be important in defining measures for assessing the level of attainment. Further on developing measurements for the defined standards studies have been considered by the authors. The main limitations of this study might be the limited number of experts in the consensus process and the absence of formative evaluation of the framework’s implementation over time to assess its construct validity. Analytical work is also required to explore the links between the organization and operation of the stewardship function and the different oral health system outcomes in different settings. Defining some intermediate or instrumental goals that are more directly related to this function could enable policy-makers to more rigorously and quantitatively evaluate the performance of the Iranian MOH in steering its stewardship role.

Conclusions

The present paper has identified a basic framework for assessing the stewardship of the oral health system in the Islamic Republic of Iran using 38 candidate standards. After piloting , we are optimistic that these standards could be appropriate for use as a benchmark instrument for the assessment and improvement of stewardship in oral health systems in developing and underdeveloped countries.

Acknowledgements

This paper was prepared as a partial requirement for the PhD degree in Dental Public Health, reporting the results of an investigation supported by Shahid Beheshti University of Medical Science. This research project was undertaken in collaboration with the WHO Collaborating Centre of Training and Research in Dental Public Health, Tehran, Islamic Republic of Iran. The authors wish to acknowledge and express appreciation to Dr Zafarmand of the Department of Community Oral Health and the Vice Dean for Academic Affairs at the School of Dentistry, Shahid Beheshti University of Medical Sciences, for providing valuable comments and contributions.

Funding: This project was conducted with the support of the School of Dentistry, Shahid Beheshti University of Medical Sciences.

Competing interests: None declared.

References

  1. Samb B et al. Prevention and management of chronic disease: a litmus test for health-systems strengthening in low-income and middle-income countries. Lancet, 2010, 376:1785–1797.
  2. The world health report 2000. Health systems: improving performance. Geneva, World Health Organization, 2000 (http://www.who.int/whr/2000/en/, accessed 20 October 2013).
  3. McKee M. The world health report 2000: 10 years on. Health Policy and Planning, 2010, 25:346–348.
  4. Everybody’s business: strengthening health systems to improve health outcomes. WHO’s framework for action. Geneva, World Health Organization, 2007 (http://www.who.int/healthsystems/strategy/everybodys_business.pdf, accessed 20 October 2013).
  5. Technical consultation on stewardship. In: Murray CJL, Evans DB, eds. Health system performance assessment: debated, methods and empiricism. Geneva, World Health Organization, 2003.
  6. Petersen PE. The world oral health report 2003. Continuous improvement of oral health in the 21st century—the approach of the WHO Global Oral Health Programme. Community Dentistry and Oral Epidemiology, 2003, 31(Suppl. 1);3–23.
  7. Chen M et al. Comparing oral health care systems. A second international collaborative study. Geneva, World Health Organization, 1997.
  8. Widström E, Eaton KA, Vanobbergen J. Oral healthcare systems in the extended European Union. Oral Health and Preventive Dentistry, 2004, 2:155–194.
  9. Fitch K et al. The RAND/UCLA Appropriateness Method user’s manual. Santa Monica, RAND, 2001.
  10. Heidemann EG. Contemporary use of standards in health care. Geneva, World Health Organization. Division of Strengthening of Health Services, District Health Systems, 1993.
  11. Travis P et al. Towards better stewardship: concepts and critical issues. Evidence and information for policy. Geneva, World Health Organization, 2002:289–300.
  12. Onwuegbuzie A, Leech NL. Sampling designs in qualitative research: making the sampling process more public. Qualitative Report, 2007, 12:238–254.
  13. Polkinghorne DE. Language and meaning: data collection in qualitative research. Journal of Counseling Psychology, 2005, 52:137–145.
  14. Barman A. Standard setting in student assessment: is a defensible method yet to come? Annals of the Academy of Medicine, Singapore, 2008, 37:957–963.
  15. Hsiao W, Saidat B. Health systems: concepts and deterministic models of performance. A background paper prepared for the workshop on research agendas on global health systems, held at Harvard University on December 3–5 2008.
  16. Van Olmen J et al. Health systems frameworks in their political context: framing divergent agendas. BMC Public Health, 2012, 12:774.
  17. Kaufman D et al. Measuring governance using cross-country perceptions data. Washington DC, World Bank, 2005 (http://www.worldbank.org/wbi/governance/govdata/, accessed 20 October 2013).
  18. Siddiqi S et al. Framework for assessing governance of the health system in developing countries: gateway to good governance. Health Policy, 2009, 90:13–25.
  19. Veillard JH et al. Health system stewardship of national health ministries in the WHO European region: concepts, functions and assessment framework. Health Policy, 2011, 103:191–199.
  20. Handler A, Issel M, Turnock B. A conceptual framework to measure performance of the public health system. American Journal of Public Health, 2001, 91:1235–1239.
  21. Nolte E, Wait S, McKee M. Investing in health: benchmarking health systems. London, Nuffield Trust, 2006.
  22. Roberts MJ et al. Getting health reform right. New York, Oxford University Press, 2003.
  23. Healy J. Analysing health care systems performance: the story behind the statistics. Australian and New Zealand Journal of Public Health, 2003, 27:642–644.
  24. Oxman AD et al. A framework for mandatory impact evaluation to ensure well informed public policy decisions. Lancet, 2010, 375:427–431.
  25. Hafner T, Shiffman J. The emergence of global attention to health systems strengthening. Health Policy and Planning, 2012, 28(1):41–50.
  26. Frenk J. The global health system: strengthening national health systems as the next step for global progress. PLoS Medicine, 2010, 7:e1000089.
  27. Campbell SM et al. Research methods used in developing and applying quality indicators in primary care. British Medical Journal, 2003, 326:816–819.
  28. Kumaranayake L. The role of regulation: influencing private sector activity within health sector reform. Journal of International Development, 1997, 9:641–649.
  29. [Health of Islamic Republic of Iran in 5th five year economic, social and cultural development plan, 8th ed]. Tehran, Islamic Republic of Iran, Ministry of Health and Medical Education, 2010 [in Farsi].
  30. Bayat F et al. Impact of insurance scheme on adults’ dental check-ups in a developing oral health care system. European Journal of Dentistry, 2008, 2:3–10.
  31. Smith PC et al. Leadership and governance in seven developed health systems. Health Policy, 2012, 106:37–49.
  32. Mikkelsen-Lopez I, Wyss K, de Savigny D. An approach to addressing governance from a health system framework perspective. BMC International Health and Human Rights, 2011, 11:13.

Privacy policy

© WHO 2024