Evaluating the quality of health technology assessment research reports until 2020: the experience of a developing country, Islamic Republic of Iran

PDF

Asma Sabermahani,1,3 Vahid Yazdi-Feyzabadi2,3 and Salman Bashzar1

1Student Research Committee, School of Health Management and Information Sciences; 2Health Services Management Research Centre, Institute for Futures Studies in Health; 3Department of Health Management, Policy and Economics, Faculty of Management and Medical Information Sciences, Kerman University of Medical Sciences, Kerman, Islamic Republic of Iran (Correspondence to Salman Bashzar:عنوان البريد الإلكتروني هذا محمي من روبوتات السبام. يجب عليك تفعيل الجافاسكربت لرؤيته.).

Abstract

Background: There is no single method for health technology assessment (HTA) which can be used in all countries to meet all the needs of the health care system policy- and decision-makers. Still, some minimum criteria for performing HTA should be in place in all the HTA structures worldwide, and many HTA agencies have reached a consensus in this regard.

Aim: This study aimed to assess the quality of Iranian HTA reports.

Method: Were examined all the HTA research reports published by the Iranian HTA office up to 2020. The International Network of Agencies for Health Technology Assessment checklist was employed for quality assessment.

Results: A total of 97 reports were examined: only 10.0% had presented complete and appropriate contact details for obtaining further information, and 5.6% clearly stated a conflict of interests. In 87.78% of the reports, the scope of assessment was clearly determined. The quality of the reports was relatively appropriate as well as the details of the sources of information and text search strategies. Legal aspects, economic analysis, ethical implications, social implications and other stakeholder perspectives were taken into account in 7.8%, 74.4%, 11.1%, 8.9%, and 4.4% of the reports, respectively.

Conclusion: As Iranian HTA reports are not of suitable quality, it is recommended that minimum standard criteria be revised and modified in the HTA process so that large-scale health care policy- and decision-makers can make reliable decisions on the basis of the results.

Keywords: health technology assessment, quality assessment, health policy, Iran

Citation: Sabermahani A, Yazdi-Feyzabadi V, Bashzar S. Evaluating the quality of health technology assessment research reports until 2020: the experience of a developing country, Islamic Republic of Iran. East Mediterr Health J. 2023;29(5):xxx–xxx. https://doi.org/10.26719/emhj.23.033

Received: 09/08/21; accepted: 24/10/21

Copyright: © Authors; licensee World Health Organization. EMHJ is an open access journal. All papers published in EMHJ are available under the Creative Commons Attribution Non-Commercial ShareAlike 3.0 IGO licence (CC BY-NC-SA 3.0 IGO; https://creativecommons.org/licenses/by-nc-sa/3.0/igo).

Introduction

Growing concerns about limiting the increase in health care costs while maintaining and strengthening access to high quality medical care have aroused interest in the better use of medical interventions (1). Discussions about the use of scientific evidence in decision-making have been revolutionized over time and, at present, evidence-based methods are the mainstream approach in many public sectors (2,3). In the health care domain, the evidence-based medical principles for clinical measures have expanded in the context of health care management and policy-making, and the number of experimental studies for raising the awareness of decision-makers is rising (4).

Moreover, advances in technology in recent years have brought about considerable change in medical care and treatment such that, annually, global medical equipment technology presents thousands of products to the market (5). Policy-makers cannot judge the values and consequences of technologies based merely on complex technical data, and for reasonable decision-making, they need to understand the vast economic, social, ethical and legal effects. Since issues relating to health technologies pose constant challenges to health care systems, it must be guaranteed that health technology is accurately evaluated and efficiently and effectively used in health care. For optimal exploitation of the existing resources, the most effective technologies should be propagated and used in light of organizational, social, ethical and economic issues (6).

Owing to the scarcity of resources in health care, decisions should be evidence-based, especially when selecting expensive technologies (16). This has made many countries develop mechanisms for the introduction and reasonable use of such technologies in order to control the costs and prevent them from increasing inordinately, optimally allocate these costs, and prevent the entry of technologies with low safety and effectiveness (6,8).

The most salient example of scientific research conducted to provide input in health care policy-making is, doubtless, found under the health technology assessment (HTA) model (4). This is a multi-disciplinary context of policy analysis research into economic, ethical, social and medical outcomes, as well as the development, propagation and use of health technologies (3). It emerged as a result of increasing concerns about the wide-ranging spread of medical equipment in the 1970s and the funding ability of insurance companies (9). The use of HTA has remarkably expanded in the last 2 decades; it is currently used for the evaluation and estimation of the value of medical technologies (10).

Historically, most HTA agencies have emphasized the development of high-quality evaluation reports which can be used by a wide range of decision-makers, e.g. the Canadian Agency for Drugs and Technologies in Health, the Swedish Council for Health Technology Assessment, the German Agency of Health Technology Assessment at the German Institute for Medical Documentation and Information and agencies in many other European countries (1). Still, organizations are increasingly performing or launching HTA for making certain decisions about resource allocation. For example, the National Institute for Health and Clinical Excellence in the United Kingdom utilizes HTAs for developing guidelines on the use of health technologies in the National Health Service in England and Wales. In Germany, the Institute for Quality and Efficiency in Health Care receives HTA requests from the Federal Joint Committee to make recommendations based on which the pricing and reimbursement for technologies are made (1).

In the Islamic Republic of Iran, HTA was launched in the form of an HTA secretariat at the Health Economy Department of the Network Development and Health Promotion Center in October 2007. The initial stages of its formation were performed with the cooperation and support of professors and researchers for receiving HTA orders and, eventually, receiving HTA reports. The overall project was approved in April 2008 at the Deputy for Coordination, Ministry of Health and Medical Education. In the next stage, the objectives, responsibilities, method of establishment and general structure of the Iranian HTA system were discussed and approved in the policy-making council at the Ministry of Health and Medical Education, supervised by the Deputy for Coordination. Joint expert teams were then formed, and with the consultation of foreign experts, 6 HTA projects were developed and their results were simultaneously presented at the executive meetings to facilitate decision-making. Since March 2010 and following the change in the structure of the Ministry of Health and Medical Education, the deputies for health and treatment were split, and the HTA department at the Technology Evaluation Office started its standard development and health care price-setting activities under the supervision of the Deputy for Treatment with a new structure. Since then, it has published many reports on health technologies.

There is no single method for performing HTA which can meet the needs of all decision-makers, stakeholders, and societies (1): HTA agencies have their own guidelines for the performance and presentation of reports, e.g. the guidelines by the International Network of Agencies for Health Technology Assessment (INAHTA). In the Islamic Republic of Iran, the Ministry of Health and Medical Education, which is the health care service provider and funder of HTA studies, is in charge of performing HTA. Therefore, the present study aimed to assess the quality of Iranian HTA reports from the foundation of HTA until 19 March 2020.

Methods

This descriptive cross-sectional study was conducted in 2020. All the reports from the HTA office in the Islamic Republic of Iran presented under the title of HTA projects, were retrieved from the website of the office of the Department of Health Technology Assessment in the Ministry of Health and Medical Education (http://ihta.behdasht.gov.ir). The inclusion criteria for the reports were: HTA reports, theses, and dissertations compatible with the priorities of the HTA office or available on this office’s list of reports. Then, these reports were evaluated based on a checklist developed by the INAHTA (11). This checklist encompasses 6 domains and has a total of 31 items, including preliminary information (5 items), why the assessment has been undertaken (4 items), how the assessment has been undertaken (10 items), information based on the evaluation and interpretation of the selected data and information (4 items), context (5 items), and post-evaluation events (3 items). This checklist assesses the HTA reports on three levels (yes, partly, no). The checklist was first translated into Farsi by 2 HTA researchers and health policy-makers, and then examined by 7 HTA experts. After expert approval, the checklist was back-translated into English to ensure its reliability and validity. The reports were evaluated by 2 researchers independently, and cases of disagreement were reported to the third researcher to reach a consensus. The data were extracted, input to a researcher-made form in Excel, and then described and analysed using descriptive statistics.

Ethics clearance was obtained from the Kerman University of Medical Sciences ethics board (ethics clearance certificate number IR.KMU.REC.1398.894).

Results

A total of 101 reports were found on the Iranian HTA office website. We eliminated 1 report due to being a duplicate, and 3 due to being non-evaluation reports. Finally, 97 reports were assessed in terms of general features, and 90 reports could be assessed based on the checklist. Of the 97 4eports examined, in terms of the type of technology investigated, the majority focused on therapeutic technologies (equipment) (47.4%), followed by diagnostic and pharmaceutical technologies (both 22.7%) (Table 1).

A number of technologies investigated in the 97 reports dealt with neoplasms (18.6%), followed by technologies dealing with health-related equipment and devices (13.4%), diseases of the nervous system (12.37%), factors affecting health status or contacting health services (10.3%); a full list of distribution according to condition is given in Table 2.

In 61 reports there was 1 first author, 6 reports had 2 first authors, and the authors of 30 reports (31.0%) were unknown. Meanwhile, 34.0% of the HTA studies were conducted by only 9 researchers, each working with his/her own team; in fact, 11 reports were written by a single researcher, 6 were written by a different researcher, and 6 authors conducted 2 studies each.

The greatest cooperation in performing HTA was exhibited by the National Institute for Health Research and the centres affiliated with Tehran University of Medical Sciences and the HTA office of the Ministry of Health and Medical Education (52.58%), the Evidence-Based Medical Research Center at Tabriz University of Medical Sciences (9.27%), and the Health Management and Economy Research Center of Isfahan University of Medical Sciences (2.1%). In 30.9% of cases, the researchers’ organizational affiliation was unknown.

For the first item on the checklist, preliminary information, only 10% of the reports provided complete and appropriate contact details for obtaining further information, while 42.2% of the reports lacked such information. The authors were identified in 8 reports (8.9%), and 5.6% transparently stated their conflict of interests. In 98.9% of the reports there was no statement on being externally reviewed. A short summary in a non-technical language was presented in only 46.7% (Table 3).

Concerning making reference to the policy question, in 57.8% of the reports this was adhered to completely, and partly stated in 31 reports (34.4%). In 74.4% of the reports, reference was made to the research question(s); in 87.8%, the scope of assessment is clearly determined; and in 82.2% there is a proper description of the health technology that has been assessed (Table 3). For the sources of information and text search strategy, the Iranian HTA reports presented precise details about a complete reference list of the included studies (97.8%), databases (86.7%), search strategy (85.6%), and years covered (84.4%). A list of excluded studies was missing in 78.9% of reports.

The findings show that the data extraction method was clearly stated in 68.8% of the reports, and a critical appraisal method was presented in 61.1%. Also, the reports presented appropriate and sufficient information in terms of the description of the method of data synthesis (61.1%) and clear presentation of assessment results (78.9%). Furthermore, in terms of the context of the reports, 74.4% considered the economic analysis; only 11.1% considered the ethical implication and only 7.8% the legal implications. In terms of discussing the findings of the assessment, 84.4% did this properly, 67.8% clearly stated the conclusions from the assessment and only 16.7% made suggestions for further action (Table 3).

Discussion

The majority of technologies evaluated in the Islamic Republic of Iran are therapeutic, diagnostic and medical; most of them deal with noncommunicable diseases or their risk factors. This shows that the epidemiological movement of diseases from communicable to noncommunicable has greatly affected the technologies required by these diseases, which constitute > 60% of the disability-adjusted life years (DALYs) and 70% of global deaths (12). In this regard, the ever-increasing growth of technologies related to these diseases should be taken into account (5).

Our findings indicate that a limited number of researchers conduct the HTA studies: 34.0% had been conducted by only 9 researchers. The majority of these researchers possessed the experience and skills of performing HTA in the Islamic Republic of Iran. Therefore, to properly conduct HTA projects, a sufficient number of HTA experts possessing the required skills should be trained and involved in conducting such projects, and this is an important measure to be taken before establishing official HTA agencies (13). The strong point of Iranian HTAs is the good organizational relationship between most of these researchers and the health care legislator.

Having proper contact details, stating the conflict of interests, and stating whether the HTA report has been reviewed are essential items for ensuring transparency (11). However, our findings revealed that only 10% of the Iranian HTA reports presented complete and proper contact details for obtaining further information, and 42.2% of the reports lacked any such information. Only 5.6% clearly stated the conflict of interests, and 1.1% had any statement about being externally reviewed. The presentation of a short non-technical summary to be understood by a wider audience is optimal (11) and enhances the impartiality and transparency of HTA activity. This summary was included in less than half the reports.

In this study, the scope of assessment was clearly determined in 87.8% of the HTA reports. Drummond et al. explain 15 key principles for improving HTAs (1). The first states that the HTA objectives and scope should be explicit and compatible with its use. Based on this principle, questions which are to be answered should be stated with maximum precision in the form of specific objectives, and, if possible, testable hypotheses should be formed. In HTA, the answers to the main questions should be presented so that the outcome of the assessment can be stated with a shared understanding of the objective and all the evidence required for answering the questions (1). In terms of answering the policy question, > 70% of the Iranian reports made reference to the questions that were to be addressed. Nevertheless, in terms of the policy question, only 57.8% of the reports completely adhered to this principle.

Since HTA aims to provide information for decision-making for policy and action (14), it should adopt appropriate methods for cost–benefit analysis (1,15) and take into account a wide range of evidence and outcomes (1). As for the sources of information and text search strategy, More than 50% of of the Iranian HTA reports presented precise details; the exception was listing excluded studies (22.2% of reports). Evidently, those who perform HTA in the Islamic Republic of Iran have actively searched maximum data based on HTA guidelines.

The HTA process is multi-disciplinary; it examines legal aspects, economic analysis, ethical implications, social implications, and other stakeholder perspectives (16–18). However, the Iranian HTA reports were not outstanding, and most of them failed to consider these factors; most of the reports only discussed economic aspects. It should be kept in mind that the economic assessment of health care interventions, especially new medications and technologies, is often performed to identify the best purchases. Eventually, policy-makers and state institutions may fund a package of general benefits (19); thus, the other aspects related to technologies should also be examined. It is important that HTAs should meet the national, regional and local needs (1). Nevertheless, many Iranian HTAs were developed in the form of safety assessment or cost–effectiveness assessment studies that failed to attend to other aspects of an HTA study.

An important principle proposed by Drummond et al. is the active cooperation of all the key stakeholders with HTA performers (1), but no trace of this principle is found in Iranian HTA reports. Although the HTA structure in the Islamic Republic of Iran is similar to the European HTA core model, there are clear differences between the Iranian HTA structure and that of other organizations, such as those in the United Kingdom (20–22) and Germany (23,24).

Thus, to create an appropriate input for determining the priorities, resource allocation, and decision- and policy-making in technology-related spheres, HTA reports should accurately evaluate their findings, clearly report their conclusions, and make suggestions about further action. Moreover, to comply with a major principle of HTA, a clear distinction should be made between assessment and decision-making (10,25); in other words, since HTA results may not be precisely adopted in decision-making, clear conclusions should be stated in the report. We found that 84.4% of the reports properly discussed the assessment findings, but only 67.8% clearly expressed conclusions from the assessment. Furthermore, suggestions for further action were made only in 16.7% of the reports. No similar study has examined the HTA reports of a country; consequently, no comparison can be made between the status of HTA studies in the Islamic Republic of Iran and other countries. However, on comparison with one available study (26), we can conclude that the Iranian studies have from major problems.

According to a study by Newman et al. on 14 selected HTA organizations around the world, there is widespread support for some principles, such as determining the objectives and scope of HTA, using a wide range of evidence, and HTA impartiality and transparency (27), out of the 15 principles proposed by Drummond et al. for developing an ideal HTA (1). Less support has been provided for some other principles, e.g. generalizability and transferability, transparency in connecting HTA results to decision-making processes, adopting a comprehensive social perspective and monitoring the implementation of HTA results.

This study is limited in that many HTA performers were not identified; thus, lack of access to many researchers led to loss of information on HTA reports, including the method of implementation, and the factors that motivated the researchers to develop such reports.

Conclusion

No single recommendation can be made for HTA studies around the world. Still, in its simplest form, an HTA should possess certain components so that an appropriate input can be offered for policy- and decision-making at the desired levels. Our study discussed the strong and weak points of Iranian HTA reports and showed that there is a considerable space for the advancement of the HTA system in HTA project implementation at the level of international standards. Despite this, the Iranian HTA system has greatly progressed and can have a promising future if an appropriate structure is created and local guidelines for HTA are developed and presented to decision- and policy-makers.

Acknowledgments

This study is based on a health policy-making PhD thesis entitled “The Health Technology Assessment System analysis in the Islamic Republic of Iran and the Presentation of Policy Option”, approved on 10 February 2020 at the Deputy for Research and Technology of Kerman University of Medical Sciences (code: 980000894) and registered at the Ethics Committee (IR.KMU.REC.1398.644). The researchers are grateful to the Deputy for Research and Technology at Kerman University of Medical Sciences.

Funding: This research was funded by the Deputy for Research and Technology at Kerman University of Medical Sciences

Competing interests: The authors declare that they have no competing interests.

References

  1. Drummond MF, Schwartz JS, Jönsson B, Luce BR, Neumann PJ, Siebert U, et al. Key principles for the improved conduct of health technology assessments for resource allocation decisions. Int J Technol Assess Health Care. 2008;24(3):244–58. doi:10.1017/S0266462308080343
  2. Learmonth M, Harding N. Evidence-based management: the very idea. Public Administration. 2006;84(2):245–66. doi:10.1111/j.1467-9299.2006.00001.x
  3. Ciani O, Tarricone R, Torbica A. Diffusion and use of health technology assessment in policy making: what lessons for decentralised healthcare systems? Health Policy. 2012 Dec;108(2–3):194–202. doi:10.1016/j.healthpol.2012.09.017
  4. Velasco-Garrido M, Gerhardus A, Røttingen J-A, Busse R. Developing Health Technology Assessment to address health care system needs. Health Policy. 2010 Mar;94(3):196–202. doi:10.1016/j.healthpol.2009.10.002
  5. Payam MM, Sari AA, Ay MR, Mobinizadeh MR, Maanavi S. Safety and diagnostic performance of dual-source ct scan in comparison with single source ct scan and conventional angiography in coronary heart diseases. J Hospital. 2010;9(1–2):25–32.
  6. Garrido MV, Kristensen FB, Busse R, Nielsen C. Health technology assessment and health policy-making in Europe: current status, challenges and potential: Copenhagen: World Health Organization Regional Office for Europe, European Observatory on Health Systems and Policies, EuropeanNetwork for HTA; 2008 (Observatory Studies Series No 14; https://apps.who.int/iris/handle/10665/107911, accessed 27 October 2021).
  7. 7Philips Z, Ginnelly L, Sculpher M, Claxton K, Golder S, Riemsma R, et al. Review of guidelines for good practice in decision-analytic modelling in health technology assessment. Health Technol Assess. 2004 Sep;8(36):iii–iv, ix–xi, 1–158. doi:10.3310/hta8360
  8. Sacchini D, Virdis A, Refolo P, Pennacchini M, De Paula IC. Health technology assessment (HTA): ethical aspects. Med Health Care Philos. 2009 Nov;12(4):453–7. doi:10.1007/s11019-009-9206-y
  9. Banta D, Jonsson E. History of HTA: Introduction. Int J Technol Assess Health Care. 2009 Jul;25 Suppl 1:1–6. doi:10.1017/S0266462309090321
  10. Angelis A, Kanavos P. Multiple criteria decision analysis (MCDA) for evaluating new medicines in health technology assessment and beyond: the advance value framework. Soc Sci Med. 2017 Sep;188:137–156. doi:10.1016/j.socscimed.2017.06.024
  11. A checklist for health technology assessment reports. Southampton: INAHTA Secretariat; 2007 (https://www.inahta.org/wp-content/uploads/2014/04/INAHTA_HTA_Checklist_English.pdf, accessed 27 October 2021).
  12. Allen L. Non-communicable disease funding. Lancet Diabetes Endocrinol. 2017 Feb;5(2):92. doi:10.1016/S2213-8587(16)30420-X
  13. Vo TQ. Health technology assessment in developing countries: a brief introduction for vietnamese health-care policymakers. Asian J Pharm. 2018;12(01). https://dx.doi.org/10.22377/ajp.v12i01.2340
  14. Hailey D, Babidge W, Cameron A, Davignon L-A. HTA agencies and decision makers. An INAHTA guidance document. Stockholm: INAHTA; 2010 (http://pds20.egloos.com/pds/201009/07/46/HTA__Decision_Makers.pdf, accessed 27 October 2021).
  15. Tarn T, Smith MD. Pharmacoeconomic guidelines around the world. ISPOR Connections. 2004;10(4):5–15.
  16. Banta D. The development of health technology assessment. Health Policy. 2003 Feb;63(2):121–32. doi:10.1016/s0168-8510(02)00059-3
  17. Thokala P, Duenas A. Multiple criteria decision analysis for health technology assessment. Value in Health. 2012;15(8):1172–81. doi:10.1016/j.jval.2012.06.015
  18. Facey K, Boivin A, Gracia J, Hansen HP, Scalzo AL, Mossman J, et al. Patients' perspectives in health technology assessment: a route to robust evidence and fair deliberation. Int J Technol Assess Health Care. 2010;26(3):334–40. doi:10.1017/S0266462310000395
  19. Verguet S, Kim JJ, Jamison DT. Extended cost-effectiveness analysis for health policy assessment: a tutorial. Pharmacoeconomics. 2016 Sep;34(9):913-23. doi:10.1007/s40273-016-0414-z
  20. Raftery J. Review of NICE's recommendations, 1999-2005. Bmj. BMJ. 2006 May 27;332(7552):1266–8. doi:10.1136/bmj.332.7552.1266
  21. Tosh JC, Longworth LJ, George E. Utility values in National Institute for Health and Clinical Excellence (NICE) Technology Appraisals. Value in Health. 2011;14(1):102–9. doi:10.1016/j.jval.2010.10.015
  22. Wonderling D, Sawyer L, Fenu E, Lovibond K, Laramée P. National clinical guideline centre cost-effectiveness assessment for the national institute for health and clinical excellence. Ann Intern Med. 2011 Jun 7;154(11):758–65. doi:10.7326/0003-4819-154-11-201106070-0000
  23. Koch K, Lange S. Response to the expertise: procedures and methods of benefit assessments for medicines in Germany, by Geertruida E. Bekkering and Jos Kleijnen. Eur J Health Econ. 2009 May;10(2):233–6. doi:10.1007/s10198-008-0142-1
  24. Herpers M, Dintsios C-M. Methodological problems in the method used by IQWiG within early benefit assessment of new pharmaceuticals in Germany. Eur J Health Econ. 2019 Feb;20(1):45–57. doi:10.1007/s10198-018-0981-3
  25. Mitton C, Seixas BV, Peacock S, Burgess M, Bryan S. Health technology assessment as part of a broader process for priority setting and resource allocation. Appl Health Econ Health Policy. 2019 Oct;17(5):573–6. doi:10.1007/s40258-019-00488-1
  26. Bullement A, Podkonjak T, Robinson MJ, Benson E, Selby R, Hatswell AJ, et al. Real-world evidence use in assessments of cancer drugs by NICE. Int J Technol Assess Health Care. 2020 Jul 10:1–7. doi:10.1017/S026646232000043
  27. International Working Group for HTA Advancement, Neumann PJ, Drummond MF, Jönsson B, Luce BR, Schwartz JS, Siebert U, et al. Are Key Principles for improved health technology assessment supported and used by health technology assessment organizations? Int J Technol Assess Health Care. 2010;26(1): 71–8. doi:10.1017/S0266462309990833

سياسة الخصوصية

© منظمة الصحة العالمية 2024. جميع الحقوق محفوظة