Business Brief

Streamlining Health Technology Assessments by Automating Literature Reviews
In the past two decades, rapid technology advances and escalating healthcare delivery costs have prompted governments, hospital administrators, and other decision-makers to put in place stronger frameworks for assessing new health interventions. As demand for health technology assessment (HTA) grows, HTA bodies are under increased pressure to improve the efficiency of their processes. This business brief reviews how automating aspects of the systematic literature review (SLR) — as one of the primary information streams feeding into an HTA — can streamline HTA activities and address some of the common challenges of HTA development.
Defining Health Technology Assessment
HTA is a process of evaluating the value of healthcare interventions, generally to inform the government or other organizations making decisions about market entry, pricing, or reimbursement.1,2 At the broadest level, it is intended to promote “an equitable, efficient, and high-quality health system.”3 Some countries include HTA as a step before market access for pharmaceuticals and other healthcare interventions, and some require it to be performed following market authorization to inform decisions about coverage for a new product.

A total of 102 countries and regions use a systematic, formal decision-making process to evaluate health interventions, according to a 2020-21 survey by the World Health Organization (WHO).4 Most European nations apply some aspects of an HTA approach to decisions about pharmaceuticals, and more than two-thirds use an HTA to support decisions about other health technologies (e.g., medical devices).5 Major public and individual health and economic consequences flow from HTA findings.

An HTA can be conducted at any point in a health intervention’s life cycle (see figure 1). It can be done before or after market introduction, or when a new indication for a marketed pharmaceutical is under consideration. As the focus on evidence-based decision-making in health care has increased, the role and scope of the HTA has evolved and demand for assessments grown. These factors create pressure to improve the efficiency of HTA processes.

HTAs and the Systematic Literature Review

The SLR is one of the primary information streams feeding into an HTA. Systematic reviews, by definition, attempt to offer a complete synthesis of all relevant studies in a transparent, verifiable manner. Comprehensiveness and transparency reduce the risk of bias in the selection and appraisal of studies. Yet these goals can be challenging to achieve given the high volume of information to evaluate under deadline pressure with limited resources while also reducing error during the process.

Typical Steps in Systematic Reviews for HTA6

  • Topic identification
  • Search design (terms, time frame, etc.)
  • Literature searching, screening, and retrieval
  • Data extraction
  • Assessment of evidence quality and bias risk
  • New evidence collection or generation, if appropriate
  • Data synthesis/meta-analysis
  • Formulation and dissemination of findings and recommendations
Role of Systematic Reviews in Cost-based Studies.

Figure 1: Health Intervention’s Life Cycle

An HTA can assess the value of a health intervention in multiple ways — including its clinical efficacy and effectiveness, safety, and cost-effectiveness — and from the viewpoint of various populations, such as patients, their families, the healthcare system, the general population of a country, or the members of a health plan.7 Clinical efficacy and effectiveness analyses integrate and summarize data about the effect of an intervention in controlled or ideal (e.g., clinical trial) and real-world conditions. A cost-effectiveness analysis compares the cost of various intervention options to their outcomes, measuring cost in monetary units and outcomes in terms of non-monetary factors such as lives saved, adverse outcomes prevented (e.g., stroke, myocardial infarction, disability), or quality-adjusted life-year gained. A budget impact analysis assesses the financial impact of an intervention on a government or payer’s budget.

Literature gathered for an HTA can therefore cover a variety of topics, including the epidemiology and clinical burden of the disease or condition impacted by the health technology evaluated; the clinical efficacy, safety, and cost of available interventions and the intervention under evaluation; and organizational, social, and legal considerations.

Systematic reviews therefore may search not only peer-reviewed biomedical literature but also gray literature (e.g., non-peer-reviewed publications, unpublished material, government or association documents) and de-identified real-world evidence from government, nation-based, or payer claims databases describing the intervention’s clinical and cost impact on the target population.

There are acute challenges associated with HTA SLRs, which organizations are confronted with in their work.

Challenges to Developing an HTA

  • Information overload
  • Time constraints
  • The need for transparency and accuracy
Information Overload
“Support for the early stages of the systematic review process — searching and screening studies for eligibility — is necessary because it is currently impossible to search for relevant research with precision.”8

A literature search for a systematic review can yield thousands of records, most of which may be deemed irrelevant upon review. On average, only 2.9% of retrieved records are included in the final report for which they were obtained.9 The volume of studies to search has risen sharply in the last two decades or so. Investigations listed on ClinicalTrials.gov rose more than 250-fold from 2000 to 2019 (from 1,255 in 2000 to 320,210 in October 2019), for example.10 Literature indexed in biomedical bibliographic databases is only one stream of information feeding into an HTA. Other potential sources include gray literature, real-world data, and information about the patient perspective.
Time Constraints
“Agencies may be requested to carry out assessments at short notice and with short timeframes for completion.”11

European agencies typically have two to three months to complete an HTA for a pharmaceutical product.12 The National Institute for Health and Care Excellence (NICE), the HTA agency for England and Wales, aims to publish assessments within 90 days of market authorization.13 Respondents to a WHO survey about HTA processes said that assessments took from one to 12 months.14 Rapid assessments also may be required.15 Systematic reviews can quickly become outdated as new studies are published, so HTAs may require updating.

The rising number of new pharmaceuticals, medical devices, and other health innovations, along with their novelty and increasing complexity, have led to an increased workload for HTA agencies.

Demands on the NICE technology appraisal program rose by 83% over three years, for example.
Transparency and Accuracy
“The search process should be documented in real time and reported in a transparent manner.”16

Steps in HTA development typically include formulating the question(s), designing the search parameters (i.e., terms, time frame, population, etc.); assessing, retrieving, and extracting data from the returned literature; evaluating evidence quality and risk of bias; synthesizing the data; and writing the report. It is important to standardize the processes at each phase to avoid inconsistency and the appearance of bias. Additionally, transparency about how these processes are conducted is crucial for the government, patients, and other constituencies to trust the report.

Human reviewers may not apply criteria for literature assessment, retrieval, data extraction, and bias evaluation consistently to each study they screen. Systematic reviews are often conducted by teams organized by subject matter expertise. Multiple professionals may apply search and assessment criteria in slightly different ways, introducing inconsistency, error, and inadvertent bias.

Only 51% of 152 reviews evaluated in a recent study reported the use of a standardized extraction form. Only 20% reported using software for study selection, and just 12% for extraction.17 Data extraction errors are common — a recent analysis of 201 systematic reviews found an error rate of 85%.18 Spreadsheets, a common tool in developing an SLR, are prone to manual data entry errors and duplicate references that can be difficult to detect.

How can automation improve HTAs?
Automation can facilitate literature screening and retrieval, data extraction, and risk of bias assessment, according to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) 2020 guideline for reporting systematic reviews.19 A survey of professionals who develop systematic reviews, HTAs, or guidelines, conducted in October 2020, indicated that:20
Role of Systematic Reviews in Cost-based Studies.
Figure 2: The results from a survey of professionals who develop systematic reviews, HTAs, or guidelines, conducted in October 2020
Roughly half of the respondents (51%) to the survey reported a lack of knowledge as the biggest barrier to using automation tools. Respondents suggested developing tools for literature searching and data extraction – processes for which multiple automation tools exist.21

DistillerSR, for example, automates the entire literature review life cycle, from search and screening to full-text retrieval, data extraction, and reporting. The software automates the management of literature collection, triage, and assessment using intelligent workflows. The workflow is configured based on the literature review methods of any study protocol, and the user can modify the workflow to suit their needs.

Specifically, DistillerSR automates the conduct of HTA by:

Literature searching and screening

  • Imports references from native search platforms and from a variety of sources including gray literature
  • Integrates with PubMed, Ovid, and EBSCO
  • Automates literature searching, de-duping, and updates
  • Applies AI to continuously reorder references based on relevance, so that team members are presented with those deemed most pertinent first

An example is a customer who uses the LitConnect feature on DistillerSR to reduce time spent updating searches by 50%. This allows for newly published references to be imported automatically. DistillerSR applies AI to determine inclusion and exclusion patterns, reordering references based on relevance and thereby producing a more efficient overall review process and faster completion rates.

Full-text retrieval

  • Connects directly to full-text open-access and paid subscription sources

DistillerSR allows customers to focus on key tasks, such as capturing high-quality data from the literature, while the administrative tasks, such as copyright management, are conducted by integrated software partners in the platform. DistillerSR’s integration with these partners allows access to free full-text or purchased subscriptions, further reducing time spent searching for articles while lowering overall literature costs.

Data extraction and bias assessment

  • Validates data before submission to avoid errors
  • Converts data with built-in calculations
  • Automates compilation of tables

An example of DistillerSR’s automation of data extraction is its ability to capture and analyze complex data such as time points across multiple studies. All the collected data can be analyzed, shaped, and exported using the data reporting tool, either in real time or as a scheduled job.

Advantages of Automation for Systematic Reviews in HTAs

  • Speed
    • DistillerSR reduced the title/abstract screening burden by a median of 40.6% and saved a median of 29.8 hours, in one study.22
  • Accuracy
    • A study examined DistillerSR for accuracy in auditing excluded references, previewing the predictions of unscreened references, and screening references based on the predictions. Software choices were compared with those made by two independent human reviewers. Most (92% to 99%) AI decisions were correct. Variation in cost-effectiveness models was 1% to 3%; in randomized controlled trials, variation was between 1% and 5%. These ranges are similar to the human reviewer’s margin of error,23 but on average 50-60% faster. Automation also avoids manual entry errors.
  • Transparency
    • By recording inclusion/exclusion decisions, DistillerSR enables the provision of data for audits, transparency, and reproducibility.
    • Results from each stage of the review process are displayed in a PRISMA 2020 flow chart that reports sources searched, references screened, and inclusion/exclusion decisions.
    • By tracking all review activity, DistillerSR makes it easy to view the provenance of each data cell.
Facilitating Collaboration

Multiple teams within HTA agencies work together to develop assessments, and many HTA agencies outsource to external groups.24 As a web-based platform, DistillerSR acts as the nerve center for systematic review for HTA projects, making data accessible to all team members across different, often global, locations for analysis. It tracks which references have been screened and what data has been extracted to avoid duplication of effort.

It can classify references and assign them to the relevant subject matter expert for screening via a standard workflow configured by the project’s administrator. It also notifies relevant team members when new references are available for screening.

DistillerSR also has a module called CuratorCR. Integrated seamlessly with DistillerSR, CuratorCR is a research knowledge center that centrally and dynamically manages an organization’s evidence-based research, allowing you to continuously collect, update, share, and reuse its data. As a result, CuratorCR eliminates re-analyzing and extracting the same data from already screened and processed references throughout an organization — speeding screening and data extraction times while reducing overall subscription costs for references. Moreover, teams can create segmented project or subject-based databases consisting of previously collected data for reuse in other reviews, or for joint research consortia between private and public sector organizations.

The data reuse approach is timely. A movement in Europe toward joint clinical assessments and scientific consultations for the most technical and demanding innovations places a premium on technology or structures that facilitate collaboration across HTA bodies. National HTA agencies often evaluate the same technologies within the same time frame, resulting in duplicate effort. The Council of the EU and the European Parliament have adopted the EU HTA Regulation, which “aims to harmonize methodological standards and to foster collaboration among European HTA bodies.”25

“A sustainable mechanism of HTA cooperation within Europe that meets the information needs of decision-makers would decrease the duplication of efforts and result in increased efficiency within national HTA agencies and across MS (member states).”26

The HTA Regulation creates the EUnetHTA 21 consortium and allows for “a very limited number of joint clinical assessments and joint scientific consultations” from 2022 to 2025 as the EU gradually moves toward integrated HTA development.

New oncology medications and advanced therapy medicinal products will be assessed jointly as of 2025, though each country makes the final HTA appraisals and reimbursement decisions. Orphan medical products will be jointly assessed as of 2028.

Software such as DistillerSR and its add-on module CuratorCR, which enable the integration of assessments by the EUnetHTA 21 consortium with the systems of member states, could only be beneficial.

Automating some of the most time-consuming and error-prone functions of a systematic review can increase the efficiency and accuracy of HTA development at a time when HTA agencies are under pressure to produce more and better-quality assessments. Literature review automation software that facilitates collaboration is especially desirable in this team-based industry where outsourcing is common.

  1. O’Rourke B, Oortwijn W, Schuller T. Announcing the new definition of health technology assessment. Value Health. 2020;23(6):824-825. DOI: 10.1016/j.jval.2020.05.001
  2. Kristensen FB, Husereau D, Huić M, et al. Identifying the need for good practices in health technology assessment: Summary of the ISPOR HTA council working group report on good practices in HTA. Value Health. 2019;22(1):13-20. DOI: 10.1016/j.jval.2018.08.010
  3. O’Rourke B, Oortwijn W, Schuller T. Announcing the new definition of health technology assessment. Value Health. 2020;23(6):824-825. DOI: 10.1016/j.jval.2020.05.001
  4. World Health Organization. Health Technology Assessment Survey 2020/21. Accessed October 13, 2022. https://www.who.int/data/stories/health-technology-assessment-a-visual-summary
  5. European Network for Health Technology Assessment (EUnetHTA). An Analysis of HTA and Reimbursement Procedures in EUnetHTA Partner Countries: Final Report. Accessed October 13, 2022. https://www.eunethta.eu/wp-content/uploads/2018/02/WP7-Activity-1-Report.pdf
  6. European Patients’ Academy on Therapeutic Innovation. Health Technology Assessment Process: Fundamentals. Accessed October 13, 2022. https://toolbox.eupati.eu/resources/health-technology-assessment-process-fundamentals/
  7. O’Rourke B, Oortwijn W, Schuller T. Announcing the new definition of health technology assessment. Value Health. 2020;23(6):824-825. DOI: 10.1016/j.jval.2020.05.001
  8. Schmidt L, Olorisade BK, McGuinness LA, Thomas J, Higgins JPT. Data extraction methods for systematic review (semi)automation: A living review protocol. F1000Res. 2020;9:210. DOI: 10.12688/f1000research.22781
  9. Zhang Y, Liang S, Feng Y, et al. Automation of literature screening using machine learning in medical evidence synthesis: a diagnostic test accuracy systematic review protocol. Systematic Reviews. 2022;11(1):11. DOI: 10.1186/s13643-021-01881-5
  10. Schmidt L, Olorisade BK, McGuinness LA, Thomas J, Higgins JPT. Data extraction methods for systematic review (semi)automation: A living review protocol. F1000Res. 2020;9:210. DOI: 10.12688/f1000research.22781
  11. European Network for Health Technology Assessment (EUnetHTA). An Analysis of HTA and Reimbursement Procedures in EUnetHTA Partner Countries: Final Report. Accessed October 13, 2022. https://www.eunethta.eu/wp-content/uploads/2018/02/WP7-Activity-1-Report.pdf
  12. European Network for Health Technology Assessment (EUnetHTA). An Analysis of HTA and Reimbursement Procedures in EUnetHTA Partner Countries: Final Report. Accessed October 13, 2022. https://www.eunethta.eu/wp-content/uploads/2018/02/WP7-Activity-1-Report.pdf
  13. National Institute for Heath and Care Excellence. Health Technology Evaluation at NICE: What Happens After the Transition Period? Accessed October 13, 2022. https://www.nice.org.uk/about/who-we-are/corporate-publications/health-technology-evaluation-at-nice–what-happens-after-the-transition-period
  14. World Health Organization. Health Technology Assessment Survey 2020/21. Accessed October 13, 2022. https://www.who.int/data/stories/health-technology-assessment-a-visual-summary
  15. European Network for Health Technology Assessment (EUnetHTA). An Analysis of HTA and Reimbursement Procedures in EUnetHTA Partner Countries: Final Report. Accessed October 13, 2022.https://www.eunethta.eu/wp-content/uploads/2018/02/WP7-Activity-1-Report.pdf
  16. European Network for Health Technology Assessment. Process of Information Retrieval for Systematic Reviews and Health Technology Assessments on Clinical Effectiveness. Version 2.0. December 2019. Accessed October 13, 2022. https://www.eunethta.eu/wp-content/uploads/2020/01/EUnetHTA_Guideline_Information_Retrieval_v2-0.pdf
  17. Büchter RB, Weise A, Pieper D. Reporting of methods to prepare, pilot and perform data extraction in systematic reviews: analysis of a sample of 152 Cochrane and non-Cochrane reviews. BMC Med Res Methodol. 2021;21(1):240. DOI: 10.1186/s12874-021-01438-z
  18. Xu C, Yu T, Furuya-Kanamori L, et al. Validity of data extraction in evidence synthesis practice of adverse events: reproducibility study. BMJ. 2022;377:e069155. DOI: 10.1136/bmj-2021-069155
  19. Page MJ, McKenzie JE, Bossuyt PM, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ. 2021;372:n71. DOI: 10.1136/bmj.n71
  20. Scott AM, Forbes C, Clark J, Carter M, Glasziou P, Munn Z. Systematic review automation tools improve efficiency but lack of knowledge impedes their adoption: a survey. J Clin Epidemiol. 2021;138:80-94. DOI: 10.1016/j.jclinepi.2021.06.030
  21. Scott AM, Forbes C, Clark J, Carter M, Glasziou P, Munn Z. Systematic review automation tools improve efficiency but lack of knowledge impedes their adoption: a survey. J Clin Epidemiol. 2021;138:80-94. DOI: 10.1016/j.jclinepi.2021.06.030
  22. Hamel C, Kelly SE, Thavorn K, Rice DB, Wells GA, Hutton B. An evaluation of DistillerSR’s machine learning-based prioritization tool for title/abstract screening – impact on reviewer-relevant outcomes. BMC Med Res Methodol. 2020;20(1):256. DOI: 10.1186/s12874-020-01129-1
  23. Smela B, Myjak I, O’Blenis P, Millier A. PNS60 Use of Artificial Intelligence with DistillerSR Software in Selected Systematic Literature Reviews. Value Health. 2020;22:suppl:S92. DOI: https://doi.org/10.1016/j.vhri.2020.07.479
  24. Wang T, Lipska I, McAuslane N, et al. Benchmarking health technology assessment agencies-methodological challenges and recommendations. Int J Techol Assess Health Care. 2020:1-17. DOI: 10.1017/s0266462320000598
  25. Julian E, Gianfrate F, Sola-Morales O, et al. How can a joint European health technology assessment provide an ‘additional benefit’ over the current standard of national assessments? Insights generated from a multi-stakeholder survey in hematology/oncology. Health Econ Rev. 2022;12(1):30. DOI: 10.1186/s13561-022-00379-7
  26. European Network for Health Technology Assessment (EUnetHTA). An Analysis of HTA and Reimbursement Procedures in EUnetHTA Partner Countries: Final Report. Accessed October 13, 2022. https://www.eunethta.eu/wp-content/uploads/2018/02/WP7-Activity-1-Report.pdf

Download the PDF version of this business brief.

Related Resources

Medlior Case Study, DistillerSR

Case Study

With as many as 40,000 references per project, Medlior, a HEOR consultancy, used DistillerSR to more accurately and efficiently manage their literature reviews.

HEOR Business Brief, Information Overload, DistillerSR

Business Brief

Information Overload Drives New Approaches to Managing HEOR Literature Reviews.

Information Overload Webinar, DistillerSR

Webinar

How Information Overload is Driving New Approaches for HEOR Systematic Literature Reviews.

Learn More About DistillerSR