Evaluating the quality of health technology assessment reports in a developing country

PDF version

Asma Sabermahani,¹ Vahid Yazdi-Feyzabadi ² and Salman Bashzar ¹

¹Student Research Committee, Faculty of Management and Medical Information Sciences, Kerman University of Medical Sciences, Kerman, Islamic Republic of Iran. ²Health Services Management Research Center, Institute for Futures Studies in Health, Kerman University of Medical Sciences, Kerman, Islamic Republic of Iran. (Correspondence to Salman Bashzar: This email address is being protected from spambots. You need JavaScript enabled to view it.).

Abstract

Background: No single method of health technology assessment (HTA) can meet all the policy- and decision-making needs in a country. However, there should be minimum criteria for performing HTA worldwide, and many HTA agencies have reached a consensus on this.

Aim: This study aimed to assess the quality of HTA reports in the Islamic Republic of Iran.

Method: We examined all the HTA research reports published by the Iranian HTA office up to 2020, using the International Network of Agencies for Health Technology Assessment checklist for quality assessment.

Results: A total of 97 reports were examined, of which only 10.0% provided complete and appropriate contact details for further information and 5.6% clearly stated a conflict of interest. In 87.78% of the reports, the scope of assessment was clearly determined. The quality of the reports was relatively appropriate as well as details of the sources of information and text search strategies. Some 7.8%, 74.4%, 11.1%, 8.9%, and 4.4%, respectively, of the reports considered legal aspects, economic analysis, ethical implications, social implications, and other stakeholder perspectives.

Conclusion: We recommend that minimum standards be established for the HTA process so that healthcare policy- and decision-makers can make reliable decisions on the basis of the HTA reports.

Keywords: health technology assessment, quality assessment, health policy, Iran

Citation: Sabermahani A, Yazdi-Feyzabadi V, Bashzar S. Evaluating the quality of health technology assessment reports in a developing country. East Mediterr Health J. 2023;29(7):554–561. https://doi.org/10.26719/emhj.22.017 Received: 09/08/21; accepted: 24/10/21

Copyright © Authors 2023; Licensee: World Health Organization. EMHJ is an open access journal. This paper is available under the Creative Commons Attribution Non-Commercial ShareAlike 3.0 IGO licence (CC BY-NC-SA 3.0 IGO; https://creativecommons.org/licenses/by-nc-sa/3.0/igo). 

Introduction

Growing concerns about reducing health care costs while maintaining and strengthening access to high quality medical care have aroused interest in the better use of medical interventions (1). Discussions about the use of scientific evidence in decision-making have been revolutionized over time and, at present, evidence-based methods are the mainstream approach in many public sectors (2,3). In the healthcare domain, the evidence-based medical principles for clinical measures have expanded in the context of healthcare management and policymaking, and the number of experimental studies for raising the awareness of decision-makers is rising (4).

Moreover, advances in technology in recent years have brought about considerable change in medical care and treatment such that, annually, global medical equipment technology presents thousands of products to the market (5). Policymakers cannot estimate the values and consequences of technologies based merely on complex technical data, and for reasonable decision-making, they need to understand the vast economic, social, ethical and legal effects. Since issues relating to health technologies pose constant challenges to healthcare systems, health technology must be accurately evaluated and efficiently and effectively used in health care. For optimal use of the existing resources, the most effective technologies should be promoted and used in light of organizational, social, ethical and economic issues (6).

Owing to the scarcity of resources for health care, decisions should be evidence-based, especially when selecting expensive technologies (16). This has made many countries develop mechanisms for the introduction and reasonable use of such technologies. This will help control and prevent costs from increasing inordinately, optimally allocate these costs, and prevent the entry of technologies with poor safety and effectiveness records (6,8).

The most salient example of scientific research conducted to provide input to healthcare policymaking is, doubtless, found under the health technology assessment (HTA) model (4). This is a multi-disciplinary context of policy analysis research into the economic, ethical, social and medical outcomes, as well as the development, propagation and use of health technologies (3). It emerged as a result of increasing concerns about the wide-ranging spread of medical equipment in the 1970s and the funding ability of insurance companies (9). The use of HTA for the evaluation and estimation of the value of medical technologies has remarkably increased in the last 2 decades (10).

Historically, most HTA agencies have emphasized the development of high-quality evaluation reports which can be used by a wide range of decision-makers, e.g. the Canadian Agency for Drugs and Technologies in Health, the Swedish Council for Health Technology Assessment, the German Agency of Health Technology Assessment at the German Institute for Medical Documentation and Information and agencies in many other European countries (1). Organizations are increasingly launching HTA for making certain decisions about resource allocation. For example, the National Institute for Health and Clinical Excellence in the United Kingdom uses HTAs for developing guidelines on the use of health technologies in the National Health Service in England and Wales. In Germany, the Institute for Quality and Efficiency in Health Care receives HTA requests from the Federal Joint Committee to make recommendations based on which the pricing and reimbursement for technologies are made (1).

In the Islamic Republic of Iran, HTA was launched in the form of an HTA secretariat at the Health Economy Department of the Network Development and Health Promotion Center in October 2007. The initial stages of its formation were performed with the cooperation and support of professors and researchers for receiving HTA orders and, eventually, receiving HTA reports. The overall project was approved in April 2008 at the Deputy for Coordination, Ministry of Health and Medical Education. During the next stage, the objectives, responsibilities, method of establishment and general structure of the Iranian HTA system were discussed and approved in the policymaking council at the Ministry of Health and Medical Education, supervised by the Deputy for Coordination. Joint expert teams were then formed, and with the consultation of foreign experts, 6 HTA projects were developed and their results were simultaneously presented at the executive meetings to facilitate decision-making. Since March 2010 and following the change in the structure of the Ministry of Health and Medical Education, the deputies for health and treatment were split, and the HTA department at the Technology Evaluation Office started its standard development and healthcare price-setting activities under the supervision of the Deputy for Treatment with a new structure. Since then, it has published many reports on health technologies.

There is no single method for performing HTA which can meet the needs of all decision-makers, stakeholders, and societies (1): HTA agencies have their own guidelines for the performance and presentation of reports, e.g. the guidelines by the International Network of Agencies for Health Technology Assessment (INAHTA). In the Islamic Republic of Iran, the Ministry of Health and Medical Education, which is the health care service provider and funder of HTA studies, is in charge of performing HTA. Therefore, this study aimed to assess the quality of Iranian HTA reports from the foundation of HTA until 19 March 2020.

Methods

This descriptive cross-sectional study was conducted in 2020. All the reports from the HTA office in the Islamic Republic of Iran presented under the title of HTA projects, were retrieved from the website of the Department of Health Technology Assessment in the Ministry of Health and Medical Education (http://ihta.behdasht.gov.ir). The inclusion criteria for the reports were: HTA reports, theses, and dissertations compatible with the priorities of the HTA office or available in the office’s list of reports. Then, these reports were evaluated based on a checklist developed by the INAHTA (11). The checklist has 6 domains and a total of 31 items, including preliminary information (5 items), why the assessment was undertaken (4 items), how the assessment was undertaken (10 items), results of the evaluation and interpretation of the selected data and information (4 items), context (5 items), and post-evaluation events (3 items). It assesses the HTA reports on three levels (yes, partly, no) and was first translated into Farsi by 2 HTA researchers and health policymakers, and then examined by 7 HTA experts. After expert approval, the checklist was back-translated into English to ensure its reliability and validity. The reports were evaluated by 2 researchers independently, and cases of disagreement were discussed with a third researcher to reach a consensus. The data were extracted and entered into a researcher-made form in Excel, and then described and analysed using descriptive statistics.

Ethics clearance was obtained from the Kerman University of Medical Sciences ethics board (ethics clearance certificate number IR.KMU.REC.1398.894).

Results

A total of 101 reports were found on the Iranian HTA office website. We excluded 1 duplicate report and 3 non-evaluation reports. We then assessed 97 reports for general features and 90 reports based on the checklist. Of the 97 reports examined, in terms of the type of technology investigated, the majority focused on therapeutic technologies (equipment) (47.4%), followed by diagnostic and pharmaceutical technologies (both 22.7%) (Table 1).

A number of technologies investigated in the 97 reports dealt with neoplasms (18.6%), followed by health-related equipment and devices (13.4%), diseases of the nervous system (12.37%) and factors affecting health status or contacting health services (10.3%)(Table 2).

In 61 reports there was 1 first author, 6 reports had 2 first authors, and the authors of 30 reports (31.0%) were unknown. Some 34.0% of the HTA studies were conducted by only 9 researchers, each working with their own team; in fact, 11 reports were written by a single researcher, 6 were written by a different researcher, and 6 authors conducted 2 studies each.

The greatest cooperation in performing HTA was reported by the National Institute for Health Research and the centres affiliated with Tehran University of Medical Sciences and the HTA office of the Ministry of Health and Medical Education (52.58%), the Evidence-Based Medical Research Center at Tabriz University of Medical Sciences (9.27%), and the Health Management and Economy Research Center of Isfahan University of Medical Sciences (2.1%). In 30.9% of cases, the researchers’ organizational affiliations were unknown.

For the first item on the checklist, preliminary information, only 10% of the reports provided complete and appropriate contact details for obtaining further information, while 42.2% of the reports lacked such information. The authors were identified in 8 reports (8.9%), and 5.6% transparently stated their conflicts of interest. In 98.9% of the reports there was no statement about external review. A short summary in a non-technical language was presented in only 46.7% (Table 3).

Concerning making reference to the policy question, in 57.8% of the reports this was adhered to completely, and partly stated in 31 reports (34.4%). In 74.4% of the reports, reference was made to the research question(s); in 87.8%, the scope of assessment was clearly determined; and in 82.2% there was a proper description of the health technology that was assessed (Table 3). For the sources of information and text search strategy, the Iranian HTA reports presented precise details about a complete reference list of the included studies (97.8%), databases (86.7%), search strategy (85.6%), and years covered (84.4%). A list of excluded studies was missing in 78.9% of reports.

The findings show that the data extraction method was clearly stated in 68.8% of the reports, and a critical appraisal method was presented in 61.1%. Also, the reports presented appropriate and sufficient information about the data analysis method (61.1%) and the assessment results (78.9%). In terms of the context of the reports, 74.4% considered the economic analysis; only 11.1% considered the ethical implication and only 7.8% the legal implications. In terms of discussing the findings of the assessment, 84.4% did this properly, 67.8% clearly stated the conclusions from the assessment and only 16.7% made recommendations for further action (Table 3).

Discussion

The majority of technologies evaluated in the Islamic Republic of Iran were therapeutic, diagnostic and medical; most of them dealt with noncommunicable diseases or their risk factors. This shows that the epidemiological change from communicable to noncommunicable diseases greatly affected the technologies required and constituted > 60% of the disability-adjusted life years (DALYs) and 70% of global deaths (12). In this regard, the continuous growth of technologies related to these diseases should be taken into account (5).

Our findings indicate that a limited number of researchers conducted the HTA studies: 34.0% were conducted by only 9 researchers. The majority of these researchers possessed the experience and skills to conduct HTA. Therefore, to properly conduct HTA projects, a sufficient number of HTA experts possessing the required skills should be trained and involved, and this is an important measure to be taken before establishing official HTA agencies (13). The strong point of Iranian HTAs is the good organizational relationship between most of the researchers and the healthcare legislator.

Having proper contact details, stating the conflict of interest, and stating whether the HTA report was reviewed are essential for assuring transparency (11). However, our findings revealed that only 10% of the Iranian HTA reports presented complete and proper contact details, and 42.2% of the reports lacked any such information. Only 5.6% clearly stated the conflict of interest, and 1.1% had statement about external review. The presentation of a short non-technical summary enhances understanding of HTA reports and less than half of the reports included this (11).

In this study, the scope of assessment was clearly determined in 87.8% of the HTA reports. Drummond et al. explained 15 key principles for improving HTAs (1). The first states that the HTA objectives and scope should be explicit and compatible with its use. Based on this principle, questions which are to be answered should be stated with maximum precision in the form of specific objectives, and, if possible, testable hypotheses should be formed. In HTA, the answers to the main questions should be presented so that the outcome of the assessment can be stated with a shared understanding of the objective and all the evidence required for answering the questions (1). In terms of answering the policy question, > 70% of the Iranian reports made reference to the questions that were addressed. However, in terms of the policy question, only 57.8% of the reports completely adhered to this principle.

Since HTA aims to provide information for decision-making for policy and action (14), it should adopt appropriate methods for cost–benefit analysis (1,15) and take into account a wide range of evidence and outcomes (1). As for the sources of information and text search strategy, more than 50% of the Iranian HTA reports presented precise details; 22.2% listed the exclusion criteria. Evidently, the researchers who conducted HTA in the Islamic Republic of Iran actively searched maximum data as recommended by the HTA guidelines.

The HTA process is multi-disciplinary; it examines legal aspects, economic analysis, ethical and social implications, and other stakeholder perspectives (16–18). However, most of the Iranian HTA reports failed to consider these factors; most of them only discussed economic aspects. Economic assessment of healthcare interventions, especially new medications and technologies, is often performed to identify the best purchases. Eventually, policymakers and state institutions may fund a package that offers general benefits (19); thus, the other aspects related to technologies should also be examined.

HTAs should meet the national, regional and local needs (1). However, many Iranian HTAs were developed in the form of safety assessment or cost–effectiveness assessment studies that failed to attend to other aspects of an HTA study.

An important principle proposed by Drummond et al. is the active consultation with all the key stakeholders by HTA performers (1), but no record of this was found in the Iranian HTA reports. Although the HTA structure in the Islamic Republic of Iran is similar to the European HTA core model, there were clear differences between the Iranian HTA structure and those of other countries, such as the United Kingdom (20–22) and Germany (23,24).

To create an appropriate input for determining the priorities, resource allocation, and decision- and policy-making in technology-related spheres, HTA reports should accurately evaluate their findings, clearly report their conclusions, and make recommendations for further action. And to comply with a major principle of HTA, a clear distinction should be made between assessment and decision-making (10,25); in other words, since HTA results may not be precisely adopted by decision-makers, clear conclusions should be stated in the report. We found that 84.4% of the reports properly discussed the assessment findings, but only 67.8% clearly expressed conclusions from the assessment. Recommendations for further action were made only in 16.7% of the reports. No similar study has examined the HTA reports of a country; consequently, no comparison can be made between the status of HTA studies in the Islamic Republic of Iran and other countries. However, in comparison with one available study (26), we can conclude that the Iranian studies had major problems.

According to a study by Newman et al. of 14 selected HTA organizations around the world, there is widespread support for some principles, such as determining the objectives and scope of HTA, using a wide range of evidence, as well as impartiality and transparency (27), out of the 15 principles proposed by Drummond et al. for developing an ideal HTA (1). Less support has been provided for some other principles, e.g. generalizability and transferability, transparency in linking HTA results to decision-making processes, adopting a comprehensive social perspective and monitoring the implementation of HTA results.

This study has limitations in that many HTA performers were not identified; thus, lack of access to many researchers led to loss of information about HTA reports, including the method of implementation, and the factors that motivated the researchers to develop such reports.

Conclusion

No single recommendation can be made for HTA studies around the world. However, in its simplest form, an HTA report should possess certain components so that it can provide adequate information for policy- and decision-making. Our study discussed the strong and weak points of Iranian HTA reports and showed that there is a need for the advancement of HTA implementation and the establishment of international standards. The Iranian HTA system has greatly progressed and has great potential for improvements if an appropriate structure is created and local guidelines for HTA are developed and adopted.

Acknowledgment

This study is based on a health policymaking PhD thesis entitled “The Health Technology Assessment System analysis in the Islamic Republic of Iran and the Presentation of Policy Option”, approved on 10 February 2020 at the Deputy for Research and Technology of Kerman University of Medical Sciences (code: 98000894) and registered at the Ethics Committee (IR.KMU.REC.1398.644). The researchers are grateful to the Deputy for Research and Technology at Kerman University of Medical Sciences.

Funding: This research was funded by the Deputy for Research and Technology at Kerman University of Medical Sciences.

Competing interests: None declared.

Examen de la qualité des rapports d'évaluation des technologies de la santé dans un pays en développement

Résumé

Contexte : Aucune méthode d'évaluation des technologies de la santé (ETS) ne peut à elle seule répondre à tous les besoins en matière d'élaboration de politiques et de prise de décisions dans un pays. Cependant, il devrait y avoir des critères minimaux pour encadrer l'utilisation de l'ETS à l'échelle mondiale. De nombreux organismes œuvrant dans ce domaine sont parvenus à un consensus à ce sujet.

Objectif : La présente étude visait à évaluer la qualité des rapports ETS en République islamique d'Iran.

Méthode : Nous avons examiné tous les rapports de recherche ETS publiés jusqu'en 2020 par le bureau iranien chargé de ces évaluations. Nous avons utilisé la liste de contrôle du réseau INAHTA (International Network of Agencies for Health Technology Assessment – Réseau international des organismes d'évaluation des technologies de la santé) pour l'évaluation de la qualité.

Résultats : Au total, 97 rapports ont été examinés, dont 10,0 % seulement fournissaient des coordonnées complètes et appropriées permettant d'obtenir des informations complémentaires, et 5,6 % indiquaient explicitement un conflit d'intérêts. Dans 87,78 % des rapports, la portée de l'évaluation était clairement déterminée. La qualité des rapports était relativement adéquate ainsi que les détails des sources d'information et des stratégies de recherche de textes. Parmi les rapports, 7,8 %, 74,4 %, 11,1 %, 8,9 % et 4,4 %, respectivement, prenaient en compte les aspects juridiques, les analyses économiques, les implications éthiques et sociales ainsi que les autres perspectives des parties prenantes.

Conclusion : Nous recommandons la définition de normes minimales pour encadrer le processus ETS, afin que les responsables de l'élaboration des politiques de santé et les décideurs du domaine puissent s'appuyer sur les rapports ETS pour prendre des décisions fiables.

تقييم جودة تقارير تقييم التكنولوجيات الصحية في أحد البلدان النامية

اسماء صابر ماهانی، وحید یزدی-فیض آبادی، سلمان باش زر

الخلاصة

الخلفية: لا يمكن لطريقة واحدة فقط لتقييم التكنولوجيات الصحية أن تلبي جميع الاحتياجات اللازمة لوضع السياسات واتخاذ القرارات في أي بلد. ومع ذلك، يجب أن يكون هناك حد أدنى من المعايير لتنفيذ تقييم التكنولوجيات الصحية على مستوى العالم، وقد توصلت وكالات عديدة معنية بتقييم التكنولوجيات الصحية إلى توافق في الآراء في هذا الشأن.

الأهداف: هدفت هذه الدراسة إلى تقييم جودة تقارير تقييم التكنولوجيات الصحية في جمهورية إيران الإسلامية.

طرق البحث: فحصنا جميع تقارير بحوث تقييم التكنولوجيات الصحية التي نشرها المكتب الإيراني المعني بتقييم التكنولوجيات الصحية حتى عام 2020. واستخدمنا القائمة المرجعية للشبكة الدولية للوكالات المعنية بتقييم التكنولوجيات الصحية بقصد تقييم الجودة.

النتائج: تم فحص ما مجموعه 97 تقريرًا: قدم 10.0% منها فقط بيانات اتصال وافية ومناسبة للحصول على مزيد من المعلومات، بينما أظهر 5.6% منها تضاربًا واضحًا في المصالح. وفي 87.78% من التقارير، حُدد نطاق التقييم بوضوح. وكانت جودة التقارير ملائمة نسبيًّا وكذلك التفاصيل عن مصادر المعلومات واستراتيجيات البحث عن النصوص. وروعيت الجوانب القانونية، والتحليل الاقتصادي، والآثار الأخلاقية، والآثار الاجتماعية وغيرها من وجهات نظر أصحاب المصلحة في 7.8%، و74.4%، و11.1%، و8.9%، و4.4% من التقارير، على التوالي.

الاستنتاجات: أوصينا بوضع الحد الأدنى من المعايير الخاصة بعملية تقييم التكنولوجيات الصحية، حتى يتمكن واضعو السياسات ومتخذو القرارات في مجال الرعاية الصحية من اتخاذ قرارات موثوق بها، استنادًا إلى تقارير تقييم التكنولوجيات الصحية.

References

1. Drummond MF, Schwartz JS, Jönsson B, Luce BR, Neumann PJ, Siebert U, et al. Key principles for the improved conduct of health technology assessments for resource allocation decisions. Int J Technol Assess Health Care. 2008;24(3):244–58. doi:10.1017/S0266462308080343
2. Learmonth M, Harding N. Evidence-based management: the very idea. Public Administration. 2006;84(2):245–66. doi:10.1111/j.1467-9299.2006.00001.x
3. Ciani O, Tarricone R, Torbica A. Diffusion and use of health technology assessment in policy making: what lessons for decentralised healthcare systems? Health Policy. 2012 Dec;108(2–3):194–202. doi:10.1016/j.healthpol.2012.09.017
4. Velasco-Garrido M, Gerhardus A, Røttingen J-A, Busse R. Developing Health Technology Assessment to address health care system needs. Health Policy. 2010 Mar;94(3):196–202. doi:10.1016/j.healthpol.2009.10.002
5. Payam MM, Sari AA, Ay MR, Mobinizadeh MR, Maanavi S. Safety and diagnostic performance of dual-source ct scan in comparison with single source ct scan and conventional angiography in coronary heart diseases. J Hospital. 2010;9(1–2):25–32.
6. Garrido MV, Kristensen FB, Busse R, Nielsen C. Health technology assessment and health policy-making in Europe: current status, challenges and potential: Copenhagen: World Health Organization Regional Office for Europe, European Observatory on Health Systems and Policies, EuropeanNetwork for HTA; 2008 (Observatory Studies Series No 14; https://apps.who.int/iris/handle/10665/107911, accessed 27 October 2021).
7. 7Philips Z, Ginnelly L, Sculpher M, Claxton K, Golder S, Riemsma R, et al. Review of guidelines for good practice in decision-analytic modelling in health technology assessment. Health Technol Assess. 2004 Sep;8(36):iii–iv, ix–xi, 1–158. doi:10.3310/hta8360
8. Sacchini D, Virdis A, Refolo P, Pennacchini M, De Paula IC. Health technology assessment (HTA): ethical aspects. Med Health Care Philos. 2009 Nov;12(4):453–7. doi:10.1007/s11019-009-9206-y
9. Banta D, Jonsson E. History of HTA: Introduction. Int J Technol Assess Health Care. 2009 Jul;25 Suppl 1:1–6. doi:10.1017/S0266462309090321
10. Angelis A, Kanavos P. Multiple criteria decision analysis (MCDA) for evaluating new medicines in health technology assessment and beyond: the advance value framework. Soc Sci Med. 2017 Sep;188:137–156. doi:10.1016/j.socscimed.2017.06.024
11. A checklist for health technology assessment reports. Southampton: INAHTA Secretariat; 2007 (https://www.inahta.org/wp-content/uploads/2014/04/INAHTA_HTA_Checklist_English.pdf, accessed 27 October 2021).
12. Allen L. Non-communicable disease funding. Lancet Diabetes Endocrinol. 2017 Feb;5(2):92. doi:10.1016/S2213-8587(16)30420-X
13. Vo TQ. Health technology assessment in developing countries: a brief introduction for vietnamese health-care policymakers. Asian J Pharm. 2018;12(01). https://dx.doi.org/10.22377/ajp.v12i01.2340
14. Hailey D, Babidge W, Cameron A, Davignon L-A. HTA agencies and decision makers. An INAHTA guidance document. Stockholm: INAHTA; 2010 (http://pds20.egloos.com/pds/201009/07/46/HTA__Decision_Makers.pdf, accessed 27 October 2021).
15. Tarn T, Smith MD. Pharmacoeconomic guidelines around the world. ISPOR Connections. 2004;10(4):5–15.
16. Banta D. The development of health technology assessment. Health Policy. 2003 Feb;63(2):121–32. doi:10.1016/s0168-8510(02)00059-3
17. Thokala P, Duenas A. Multiple criteria decision analysis for health technology assessment. Value in Health. 2012;15(8):1172–81. doi:10.1016/j.jval.2012.06.015
18. Facey K, Boivin A, Gracia J, Hansen HP, Scalzo AL, Mossman J, et al. Patients' perspectives in health technology assessment: a route to robust evidence and fair deliberation. Int J Technol Assess Health Care. 2010;26(3):334–40. doi:10.1017/S0266462310000395
19. Verguet S, Kim JJ, Jamison DT. Extended cost-effectiveness analysis for health policy assessment: a tutorial. Pharmacoeconomics. 2016 Sep;34(9):913-23. doi:10.1007/s40273-016-0414-z
20. Raftery J. Review of NICE's recommendations, 1999-2005. Bmj. BMJ. 2006 May 27;332(7552):1266–8. doi:10.1136/bmj.332.7552.1266
21. Tosh JC, Longworth LJ, George E. Utility values in National Institute for Health and Clinical Excellence (NICE) Technology Appraisals. Value in Health. 2011;14(1):102–9. doi:10.1016/j.jval.2010.10.015
22. Wonderling D, Sawyer L, Fenu E, Lovibond K, Laramée P. National clinical guideline centre cost-effectiveness assessment for the national institute for health and clinical excellence. Ann Intern Med. 2011 Jun 7;154(11):758–65. doi:10.7326/0003-4819-154-11-201106070-0000
23. Koch K, Lange S. Response to the expertise: procedures and methods of benefit assessments for medicines in Germany, by Geertruida E. Bekkering and Jos Kleijnen. Eur J Health Econ. 2009 May;10(2):233–6. doi:10.1007/s10198-008-0142-1
24. Herpers M, Dintsios C-M. Methodological problems in the method used by IQWiG within early benefit assessment of new pharmaceuticals in Germany. Eur J Health Econ. 2019 Feb;20(1):45–57. doi:10.1007/s10198-018-0981-3
25. Mitton C, Seixas BV, Peacock S, Burgess M, Bryan S. Health technology assessment as part of a broader process for priority setting and resource allocation. Appl Health Econ Health Policy. 2019 Oct;17(5):573–6. doi:10.1007/s40258-019-00488-1
26. Bullement A, Podkonjak T, Robinson MJ, Benson E, Selby R, Hatswell AJ, et al. Real-world evidence use in assessments of cancer drugs by NICE. Int J Technol Assess Health Care. 2020 Jul 10:1–7. doi:10.1017/S026646232000043
27. International Working Group for HTA Advancement, Neumann PJ, Drummond MF, Jönsson B, Luce BR, Schwartz JS, Siebert U, et al. Are Key Principles for improved health technology assessment supported and used by health technology assessment organizations? Int J Technol Assess Health Care. 2010;26(1): 71–8. doi:10.1017/S0266462309990833