Business Brief

Improve Cost-Effectiveness Analysis and Budget Impact Modeling Using Literature Review Automation Software
Automating literature reviews for HEOR researchers

There is a substantial proliferation of published cost-effectiveness analyses and budget impact assessments, resulting in an increase in systematic literature reviews (SLRs). Concurrently, advances in automated literature screening, classification and data management are enabling researchers and decision makers to interpret these studies faster and more accurately and subsequently apply the results to clinical practices.

Increasingly, healthcare decisions are based on intersecting data of clinical evidence and value, including cost-effectiveness analysis (CEA) and budget impact modeling (BIM). Many CEA and BIM assessments utilize systematic literature reviews and meta-analysis to estimate economic, patient-reported, and clinical consequences. SLRs are used to synthesize the multitude of cost studies for use of in-value assessments. These evaluations guide health economics and outcomes research (HEOR) professionals and other decisionmakers to deliver better clinical care, improve healthcare policy, and manage costs.

Cost-based studies are noted for their heterogeneity of methods, outcomes, and perspectives. This makes them particularly challenging to interpret. HEOR researchers are seeking automated SLR tools that deliver higher quality reviews, configurable to the complexities of economic evaluations, and are scalable to handle the high volume of references. This business brief offers a SLR solution that is fit-for-purpose for the unique needs of health economics and outcomes research.

Why Are Cost Studies Gaining Momentum?

  • Substantiate drug formulary inclusion for a private health care system
  • Compare a new intervention against a threshold value for a country’s health system
  • Determine efficient allocation of limited resources by a health technology assessment body
  • Determine reimbursement level for a new drug by a payer
  • Negotiate metrics in value-based pricing agreements
  • Inform clinical care pathways to ensure optimized treatment, patterns of care, and resource allocation

Role of Systematic Reviews in Economic Models

In modeled CEA and BIM studies, the efficacy or effectiveness measures (e.g., outcomes) are derived from clinical trials, real world evidence, systematic reviews or meta-analysis. Systematic literature reviews are also used to synthesize the vast amount of cost-based studies, to enable clinical and policy-based decision-making.

Systematic literature reviews of economic analysis such as CEA and BIM must consider quality and bias risk to ensure valid output that is comprehensive, robust, transparent and reproducible.

Role of Systematic Reviews in Cost-based Studies.

Figure 1: The diagram presents an example of how AI may be integrated into the review process.

In Figure 1, SLRs establish a standard evidence-based process that imbues greater levels of research rigor and transparency, all of which engenders a high degree of confidence in decision-making.

To achieve the highest quality SLR for cost-based studies, HEOR researchers may rely upon several approaches to inform their process:

  1. ISPOR Good Practices Report for Systematic Reviews with Costs and Cost-effectiveness Outcomes (Mandrik,2021)
  2. Checklists for quality and quality reporting of cost and cost-effectiveness studies (Wijnen, 2016)
  3. ISPOR CiCERO Checklist (Mandrik, 2021)
  4. Transparent reporting of methods and results by adherence to evidence-based reporting standards, such as PRISMA (Page 2021) for systematic review and CHEERS (Husereau, 2013) for health economic evaluation
  5. Automated tools that accelerate the delivery of high-quality literature reviews, such as DistillerSR

Cost Study Systematic Reviews and Critical Appraisal of Included Studies

HEOR researchers conducting reviews of cost studies should refer to the recently published Good Practices Report, “Critical Appraisal of Systematic Reviews with Costs and Cost-Effectiveness Outcomes,” which is based on a multidisciplinary task force convened by ISPOR (The Professional Society for Health Economics and Outcomes Research) (Mandrik, 2021). The report also includes the ISPOR Criteria for Cost-Effectiveness Review Outcomes (CiCERO) Checklist.

The guidance is intended to assist researchers, drug and health technology developers, and evidence users with guidance to interpret cost study quality and bias risk.

Value-based Decision-Making Powered by Systematic Literature Reviews

“HEOR teams are challenged to keep up with both the volume and diversity of evidence that they are required to bring together for their stakeholders. This has resulted in an imperative to consider new technology and approaches to meet the needs of their clients.”
- Jennifer Tetzlaff, Epidemiologist-in-Residence, DistillerSR

Assessing the methodologic quality of cost-based studies is dependent upon the study design, and Wijnen et al (Wijnen 2016) provides an overview of various checklists that are commonly used.

In addition to the above tools, HEOR researchers may use the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA), an evidence-based minimum set of items for reporting in systematic reviews and meta-analyses.

The PRISMA 2020 guidance was recently published (Page 2021). While PRISMA was originally developed for reviews evaluating randomized trials, the guidance is also used as a basis for reporting systematic reviews of other types of research, particularly evaluations of interventions.

In addition to PRISMA, HEOR researchers may utilize the Consolidated Health Economic Evaluation Reporting Standards (CHEERS) (Husereau, 2013), a reporting guideline to optimize the reporting of health economic evaluations, including CEA and BIM. A revision of CHEERS, called CHEERS II is under review and is expected to be released in early 2022.

Addressing the Challenges with Systematic Review and Meta-analyses for CEA and BIA
Systematic reviews of health economic literature are typically quite complex because of the numerous and variable outcomes to be assessed, including diversity in study design (Mandrik, 2021).

For example, cost and cost-effectiveness studies may include several outcomes ranging from efficacy, effectiveness, safety, patient-reported outcomes, quality metrics, as well as resource utilization or costs. Two or more comparators may be included, and the assessments may span across multiple countries, therapeutic areas, variable time horizons, a range of scenarios or sub-populations, diseases and stages, as well as perspectives. Study selection and eligibility is complex, and grey literature may be included.

Study design can include model-based and/or empirical studies. Data volumes are typically sizable and data extraction is complicated. These realities often require multiple reviewers to accomplish the task, leading to valid concerns about errors and omissions. Organizations are mindful of the need for an audit trail for regulatory bodies and desire to publish the systematic review. Figure 2 provides a summary of key SLR challenges for economic evaluations. These challenges are addressed by the efficient, repeatable, scalable, and automated literature review process offered by DistillerSR.

Figure 2: Key challenges associated with SLRs for CEA and BIM.

Figure 2: Key challenges associated with SLRs for CEA and BIM.

DistillerSR Advantages for HEOR Experts Conducting Systematic Reviews
DistillerSR automates the management of literature collection, triage, and assessment using AI and intelligent workflows including tracking incoming references, de-duplication, reprioritization, and classification, acquiring full-text documents, automating search updates, and coordinating workload assignments.

The cloud-based platform also has a highly configurable and intelligent workflow, which allows HEOR professional to track all activity and coordinate work assignments.

Taking Control of Your Evidence with DistillerSR
DistillerSR was designed with input from SLR teams around the world to automate and accelerate literature reviews. More than 300 of the world’s leading pharmaceutical and medical device companies, contract research organizations, governments, NGOs, and academic institutions trust DistillerSR to produce transparent, audit-ready, and compliant reviews. DistillerSR increases the quality of and confidence in systematic review outputs when building literature-based economic models for treatment interventions.
1. Automation
Many researchers may be hesitant to implement automation into their systematic review process. That said, human reviewers can be imperfect. The key to utilize low-risk, big reward ways is to implement automation and AI into the HEOR literature review process, providing a greater degree of review efficiency and thoroughness.

For example, DistillerSR’s continuous and automatic AI re-rank uses machine learning to learn from the references you are including and excluding to automatically reorder the ones left to screen, putting the references most likely to be relevant in front of you. This means you find and include pertinent references much more quickly. AI classifiers can be trained to automatically answer questions, such as: “Is this a randomized controlled trial” or ”Is this an adverse event?” These classifiers can be applied to screening or assessment forms to answer targeted questions automatically. References are automatically imported to update a review using DistillerSR. Review tasks, meanwhile, are automatically assigned to the appropriate reviewer with intelligent workflows configured to the way your team works.

Automating elements of your CEA and BIM systematic literature reviews with DistillerSR leads to:

  • Efficient identification of relevant evidence
  • Continuously updated literature
  • Faster time to completion
  • Better team collaboration
  • Fewer errors
2. Transparency and Reproducibility
Critical components for successful cost-effectiveness analyses and budget impact models are transparency and reproducibility, at both the data input stage and in the presentation of the results.

Stakeholders, whether regulators, payers and health technology assessment bodies, clinicians, or journal reviewers, need to understand the systematic review protocol used to populate the model, if it is based on literature. Organizations that provide research for regulatory review must follow specific standards when it comes to reporting systematic literature review and economic models; audit trails are, therefore, essential for all systematic reviews.

With DistillerSR, everything is automatically tracked and documented. It is easy for an auditor or decision-making body to confirm that the literature review was conducted according to required standards and best practices. It allows stakeholders to trust the data and findings being presented.

An audit trail that utilizes user, date, and timestamps is as transparent as it gets. Also, the act of automatically recording everything makes sure everyone on the team is following the proper protocol. The major purpose of an audit trail, in this instance, is having the ability to answer questions efficiently and accurately in conformity assessments. For instance, ‘Who reviewed this reference?’ and ‘When was this work completed?’ are both questions that can be quickly answered thanks to an audit trail.

In addition to assisting with compliance requirements, an audit trail helps teams collaborate better and faster. Timestamping and user stamping ensures that every change made to the data is noted. That way, if there is an error, it can be found and corrected efficiently, with a full record of any corrections made.

In summary, audit trails help preserve data provenance, tracking all changes and preserving every version of the data throughout the process. These types of “digital forensics” are important for organizations that deal with data that informs policy and guidelines or organizations that could be the subject of legal proceedings.

3. Data Extraction and Conflict Management

Data extraction may be described as picking and pulling specific, relevant, and important information, or data, from literature. It is, essentially, the process of converting unstructured reference text to codified data that can be used in analysis. To ensure high-quality data, best practices dictate that two reviewers extract data in parallel and then compare results. This can be a highly labor-intensive and error-prone process.

DistillerSR's Audit-ready Digital Forensics at your Fingertips

Figure 3: Auditable systematic reviews provide HEOR professionals with a higher degree of confidence in their data.

HEOR researchers are tasked with managing this process with an eye on efficiency and rigor, achieving both without sacrificing either.

In most SLRs, groups commonly use one of three approaches: single reviewer, dual reviewers, or reviewer plus quality assurance (QA) in combination.

A single reviewer data extraction is efficient and less resourceintensive but does not have a second reviewer to catch errors or address initial review bias. This approach may be used for non-regulatory projects or scoping reviews.

A dual reviewer extraction process – the “gold standard” of SLR is commonly used for regulatory submissions, guideline development or other high-rigor projects. However, adding the second reviewer doubles the resource inputs. Dual reviews are challenging when they are complex. For example, this could include repeating data sets, such as capturing multiple time points for multiple outcomes within a single study. Data, for example, can differ between studies, or open-ended questions require consensus among reviewers.

The third approach, reviewer plus QA, is a hybrid comprised of data extraction initially done by a reviewer, followed by a second reviewer’s validation.

If discrepancies exist, the reference stays in the data extraction stage until resolved. This approach offers lower error and bias rates compared with the single reviewer approach and it is more efficient than the dual reviewer stage. False conflicts (slightly different information entered by two reviewers), and data syncing from complex repeating datasets are also less frequent.

However, proving true agreements is still challenging and can often involve bias, which may still exist from the first reviewer.

DistillerSR offers the overall benefit that all extracted and coded data for the entire review resides in one place, making validation and updates easier. Any changes to the data are version-controlled and extracted data can be exported using a variety of formatting and filtering options.

DistillerSR has built-in Automated Conflict Checking which enables a user to specify what constitutes a disagreement between data extractors and reviewers. This capability allows reviewers to identify and manage conflicts more accurately and efficiently. The approach is flexible enough to work with closed-ended questions, numeric fields, dates, and/or free-form text.

DistillerSR’s Quality Control (QC) function enables reviewers to have one person do the initial screening or data extraction, while a second person reviews their work for accuracy. If the QC reviewer does not agree with the initial reviewer, the reference will be flagged by DistillerSR as a conflict for resolution. Even when handling complex, repeating or hierarchical data, (such as follow-up data captured at multiple time points), DistillerSR easily identifies conflicts for efficient conflict resolution.

DistillerSR’s conflict management and quality control functionalities lead to:

  • Faster, easier and configurable data extraction
  • Better workflow
  • Improved project management
  • Greater efficiency
  • Happier reviewers!
DistillerSR’s QC functionality facilitates validation of initial reviewers work by a secondary reviewer for a more time-efficient alternative to dual reviewing each citation.

Figure 4: DistillerSR’s QC functionality facilitates validation of initial reviewers work by a secondary reviewer for a more time-efficient alternative to dual reviewing each citation.

DistillerSR automatically identifies disagreements specified by your review criteria, making conflicts easy to address and manage.

Figure 5: DistillerSR automatically identifies disagreements specified by your review criteria, making conflicts easy to address and manage.

In today’s value-based healthcare environment, there is an increased demand for cost-effectiveness and budget impact assessments that are powered by robust systematic literature reviews. HEOR professionals face significant challenges when conducting literature reviews, including growing data volumes with multiple outcomes, the need for better transparency and reproducibility, and error-prone, slow data extraction. DistillerSR offers an automated, industry-proven platform that provides efficiency, accuracy, and transparency, leading to better decision-making and improved healthcare.
  1. Khan ZM, Pizzo LT. Choosing the Right Path in HEOR Publishing. Value & Outcomes Spotlight. 2021(March/April):3. Accessed on May 13, 2021 from
  2. Mandrik O, Severens JL, Bardach A, et al. Critical Appraisal of Systematic Reviews With Costs and Cost-Effectiveness Outcomes (ISPOR Report). Value in Health. 2021 April 1;24(4):463-472. DOI:
  3. Wijnen B, Van Mastrigt G, Redekop WK, et al. How to prepare a systematic review of economic evaluations for informing evidence-based healthcare decisions: data extraction, risk of bias, and transferability (part 3/3).Expert Rev Pharmacoecon Outcomes Res. 2016; 16: 723-732
  4. Page MJ, McKenzie JE, Bossuyt PM, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ 2021; 372 doi: (Published 29 March 2021)
  5. Husereau D, Drummond M, Petrou S, et al. Consolidated health economic evaluation reporting standards (CHEERS)—explanation and elaboration: a report of the ISPOR Health Economic Evaluations Publication Guidelines Good Reporting Practices Task Force. Value Health. 2013;16(2):231-250.

Download the PDF version of this business brief.

Related Resources

Medlior Case Study, DistillerSR

Case Study

With as many as 40,000 references per project, Medlior, a HEOR consultancy, used DistillerSR to more accurately and efficiently manage their literature reviews.

HEOR Business Brief, Information Overload, DistillerSR

Business Brief

Information Overload Drives New Approaches to Managing HEOR Literature Reviews.

Information Overload Webinar, DistillerSR


How Information Overload is Driving New Approaches for HEOR Systematic Literature Reviews.

Learn More About DistillerSR