What is a Systematic
Review? Ultimate Guide to Systematic Reviews

What is a systematic review?

systematic review is a rigorous research method for collecting, reviewing, and reporting on large sets of published data.  Many consider systematic literature reviews the gold standard in evidence-based research and studies.

That is the short answer to “what is a systematic review?” But systematic reviews go much deeper and play an important role in mitigating bias, establishing the credibility of authors, and producing evidence-based quality studies.

Systematic reviews involve taking a robust, systematic approach to collecting all the relevant literature about a specific subject and synthesizing it for use in answering a well-defined research question or questions. The literature used in the systematic reviews includes peer-reviewed journal articles, grey literature, and other sources. Understand the differences between a peer review and a systematic review.

Systematic reviews are used throughout evidence-based healthcare, including in the creation of medical practice guidelines, to inform social and health-related policy decisions, and to assess the current state of knowledge in specific health science areas. Of course, there are many other uses for the systematic review methodology, but health sciences are where the majority of work is being done today.  

One of the first examples of what we now know as a “systematic review” was conducted in 1753 by James Lind — he published a report that looked at all the unbiased evidence on scurvy. However, systematic reviews truly came into prominence in the 1970s and 1980s after Archie Cochrane’s influential Effectiveness and Efficiency text was published, urging the practice of evidence-based medicine. This led to the inception of the eponymous Cochrane Collaboration, a collection of healthcare researchers around the world who share the commitment to using high-quality evidence to make healthcare decisions, and the belief that evidence should be easily accessible, quality-assured, and cumulative. Systematic reviews address these commitments and ensure that decisions are made with proper evidence. Eventually, it became clear that evidence synthesis was valuable in more than just healthcare, resulting in the creation of the Campbell Collaboration, an international scientific research organization that promotes evidence-based research beyond health sciences.

Why are systematic reviews valuable?

Simply speaking, better research benefits the world. As a top method for gathering and reporting on data, systematic reviews are essential to the global population in terms of quality scientific research and innovation. Systematic reviews aim to provide credible, reproducible studies, and reduce bias.

Policymakers and guideline developers must have the full picture on any given topic before making recommendations or informing new policies and guidelines. They rely on systematic reviews as one of the main tools to inform their decisions which impact the general population, such as when setting regulatory standards.

Illustration of person holding a coin in front of a money tree
Survey Ultimate Guide to Systematic Literature Review, DistillerSR

Systematic Reviews vs Primary Studies

primary study is research that collects data from an original source rather than from research that has already been done. Examples of primary research include surveys, interviews, cohort studies, randomized clinical trials (RCTs), case studies, lab notebooks, and dissertations to name a few primary research methods.

Unlike primary studies, systematic reviews gather the entire catalog of data on a specific topic, which makes them statistically more powerful. In some cases, systematic reviews can be done before the primary research. Systematic review findings enable research teams to “tune” their primary studies to only answer the questions not answered by the systematic review. This approach is a good idea because it tends to create safer studies with fewer patient interventions. Systematic reviews are also often less costly and time-consuming than primary studies, which makes them vital to keep up with the high demand for evidence-based research today.

Systematic Review or Literature Review?

In certain industries or academic streams, “literature review” is synonymous with “systematic review” (for example, in preparation of clinical evaluation reports), while in others, the defining aspect of a systematic review is that it is systematic and follows a set of specific guidelines, whereas a general literature review may not adhere to the same kinds of standards and expectations.

However, the name and definition depends entirely on specific terminology and context. Depending on the geographic location, industry, or even the organization, you could call it a “systematic review,” “literature review,” or even “systematic literature review.” Read about the types of Systematic Reviews.

Illustration of person holding a clipboard and pointing to a giant question mark
systematic review lifecycle

 What is the systematic review protocol?

The Systematic Review Lifecycle can be broken down into five steps. Each of the following steps is vital to the systematic review protocol:

  1. Search: The research question is the most crucial component of the systematic review. A well-defined systematic review question will inform the search strategy, screening criteria, data extraction protocol, and the final report. Using a framework like PICO is a great way to define the research question and begin your search.
  2. Screen: This is often the most time-consuming part of the systematic review process. Related studies are gathered and depending on the methods of the research team, they might start by screening title and abstract first then moving on to full-text screening. At each screening level, researchers will decide whether the reference is relevant (and should be included) or if it is not relevant (and should be excluded.)
  3. Full-Text Retrieval: In this stage of the systematic review protocol, researchers will gather the full-text documents identified in their title and abstract screening.
  4. Data Extraction/Appraisal: In this step, researchers will capture repeating data sets in a reportable format. This is also the step where the work is double-checked with some sort of quality assessment tool.
  5. Reporting: Researchers will produce the reference lists, outcomes, and summary tables, PRISMA reports, and meta-analyses and forest plots where appropriate. They will also file for or submit articles for peer review. Learn more about the difference between a systematic review vs meta-analysis.

These are the five main steps involved in completing a systematic review. Still, you should also know about a sixth step:

  1. Update: This involves monitoring for new references on the topic and adjusting your research question if needed per the new information available.

This is what makes a systematic review more like a cycle rather than a linear process. We refer to this process as maintaining a “living” or “evergreen” review process. It’s essential for research groups who are working towards regulatory compliance to keep their systematic reviews up to date and relevant.

Systematic Review Best Practices

Over the years, systematic literature review methodology has changed, improved, and innovated with technological advances, and the need for fast, detailed, and evidence-based research has reached new levels. 

As early as 20 years ago, systematic reviews were done with pen and paper. Researchers and staff were physically printing off references, screening, highlighting, and keeping references in separate stacks or filing cabinets. As you can imagine, this manual process was extraordinarily time-consuming and resource-intensive.

Fifteen years ago, systematic reviews moved into the digital sphere and were completed using spreadsheets. Researchers would copy and paste data into cells. This saved a lot of paper, but as the demand for evidence grew, the spreadsheet method began to severely lacked the efficiency and accuracy needed by modern researchers. With the help of technology, systematic review tools have become more advanced and new methods have been implemented.

Today, systematic review software is the status quo for completing fast and accurate systematic reviews–more on that later. 

Regardless of the purpose of your review, there are several universally accepted systematic review best practices. These best practices improve efficiency, transparency, and reproducibility in systematic reviews, which are goals that all reviews strive to achieve. 

  • Dual independent screening is the gold standard best practice for systematic reviews. Humans make errors, so it’s critical to have methods in place that will catch the errors before they become catastrophic to your research. 
  • Establish an experienced team to carry out the systematic review tasks. If there are certain elements of the review that your organization is not experienced with, it would be advisable to hire an information specialist or other subject matter experts to help including people with experience in systematic literature reviews. 
  • Search all relevant databases and alternative information sources for grey literature. An incomplete search is one of the foremost reasons that a systematic literature review could be invalidated.
  • Determine inclusion and exclusion criteria and develop a conflict resolution protocol beforehand. Stay away from ad hoc processes.
  • Capture the reason for exclusion. Researchers can easily reference the article exclusion list during the peer review or audit stage. The list of excluded articles could also help expedite the search and screening process in different but similar reviews that look at some of the same references. 
  • Systematic reviews hinge on the idea that evidence should be transparent. A brief look at the PRISMA checklist shows that transparency in every step of the review from developing the search strategy to reporting the results ought to be detailed and as transparent as possible.
Best Practices Ultimate Guide to Systematic Literature Review, DistillerSR
Challenges Ultimate Guide to Systematic Literature Review, DistillerSR

Systematic Review Challenges

The research community faces numerous challenges in their work every day. Chief among those challenges is the fact that standards are always changing, science is always innovating, and best practices are ever-shifting, all while resources like time and funding for studies are becoming more scarce. Keeping up with the demand for evidence-based research is challenging. But there are a few specific factors that make systematic reviews particularly challenging in today’s age:

  • Often, researchers and reviewers are dealing with massive datasets that take considerable time and effort to screen. Even one article alone can take a lot of time and effort
  • Research teams are often strapped for resources; time and funding can be scarce, so all resources must be optimized for efficiency and results
  • Many tools are not designed to support effective collaboration between reviewers, writers, researchers and project managers.
  • New, more stringent regulatory requirements put more pressure on organizations to show transparency and reproducible results

The use of comprehensive systematic literature review software is one of the best methods to solve numerous challenges and completely streamline the systematic literature review process.

Literature Review for EU MDR

EU MDR is a new medical device regulation directive in Europe. It impacts all new and legacy (existing) medical devices sold in the European Union. The new directive was released in 2017, and since then, medical device manufacturers have been working towards ensuring their products are MDR-compliant

Under EU MDR, medical devices are reviewed by notified bodies who are designated by the European Commission. These notified bodies have a duty to perform the conformity assessments and audits to confirm if medical devices meet the current standards. They are for-profit organizations hired by medical device manufacturers to perform “unannounced audits,” so it’s critical for manufacturers always to be prepared. 

So what constitutes compliance? For medical device manufacturers, the literature review is just one (albeit important) part of their conformity assessment. Notified bodies look for a few things when assessing a CER literature review. Some of the biggest challenges medical device manufacturers face with their literature reviews include:

  • Incomplete audit trail: NB’s want to see transparency in the literature review. They want to know the path researchers take to reach their conclusions. An audit trail is essential for showing work
  • Ad hoc processes: A proper literature review must follow a systematic and reproducible process. 
  • Incomplete search coverage: Missing studies is a critical error in a CER literature review, and missing relevant information could invalidate your work and lead to time to market delays. 
  • Data integrity: CER literature reviews are used to inform important healthcare decisions, so accuracy is essential. 
  • Efficiency: When dealing with massive amounts of data to make informed decisions for evidence-based healthcare, it’s important to work as efficiently as possible.

Although these are common challenges for organizations seeking EU MDR-compliance, there are several ways to mitigate the challenges and make it easier to comply with the new medical device standards, so your device doesn’t fail its audit. Medical device manufacturers must establish “State of the Art” in their CER literature review, which is essentially a section of the report that >“describes what is currently and generally considered standard of care, or best practice, for the medical condition or treatment for which the device is used.” However, the State of the Art section impacts the entirety of the review, and if you fail to establish it, your review could be considered incomplete.

EU MDR Ultimate Guide to Systematic Literature Review, DistillerSR
Software-Ultimate Guide to Systematic Literature Review, DistillerSR

How does systematic literature review software work?

Systematic review software has become more common in the research community in the past 5-10 years and continues to improve the speed and quality of systematic reviews. The demand for a faster and better way to perform systematic reviews has driven the need for better tools for researchers to use. 

DistillerSR is one of the most widely used systematic review software on the market. But how does it work? Where does DistillerSR fit into your workflow? 

Here’s a basic overview of how this type of systematic review software works: 

  1. Upload your references: You can drag and drop references from any reference management software, upload references and their full-text documents. You can search and import references directly from PubMed as well as directly access full text documents from Article GalaxyCopyright Clearance Center, and PubMed Central. You can then use DistillerSR’s powerful duplicate detection engine to find and quarantine duplicate references.,
  2. Build forms: DistillerSR enables users to build custom screening and data  extraction forms which are used to quickly and efficiently screen, extract data from, and assess the quality of references for the review. The forms are 100% customizable and can be reused across reviews.
  3. Configure your workflow and assign reviewers: This is what truly sets DistillerSR apart — enables users to customize their own workflow and process based on their review requirements. Users can set up any number of forms, create custom inclusion/exclusion protocols, configure keyword highlighting, and use boolean filters to branch the studies further. 
  4. Monitor and tune your review: You can see real-time results from your reviewers, including reference progress, reviewer conflicts, and included/excluded reference counts. You can also track time spent on the review, get reviewer participation reports. DistillerSR automatically calculates kappa interrater reliability scores for the systematic review.
  5. Export your results: Create reference tables, PRISMA reports, and included/excluded study lists with the click of a button. The reports can also be fully customized for your needs. 

Did you notice that the Systematic Review Lifecycle lines up almost identically with this workflow? Coincidence? We think not! When choosing systematic literature review software, it’s important to think about your unique challenges and review workflow. However, be warned! In this case, relying on a feature matrix can be misleading, so it’s best to do extensive research about what review software best meets your needs before buying.

How systematic review software can save costs and time

If you consider the “old way” of doing a manual systematic review, with pen and paper or with clunky spreadsheet programs, it makes sense to use specialized systematic review software instead. After all, the research is of critical importance; doesn’t it make sense to do everything you can to make sure it’s done correctly? 

Systematic review software (SRS) benefits research teams in many different ways, but the two most important things to know are that it saves time and money. 

With SRS, researchers can work more efficiently, reduce errors, and, therefore, spend less time fixing mistakes when completing reviews. This efficiency results in lower costs and faster access to better evidence-based research. 

Other significant benefits of Systematic Review Software include:

  • Real-time collaboration: Specialized systematic review software like DistillerSR enables users to work on their review from any browser, which means you can work on projects regardless of your location. 
  • Audit-trail: The software also has a clear audit trail with timestamping and user stamping, which promotes transparency, ensures the team follows proper protocol and helps pinpoint errors and changes to the project. 
  • Simple reporting: Access to reports including Kappa, PRISMA, User Metrics, and Reference Progress with the simple click of a button. 
  • Automation: Other features such as deduplication and reference prioritization are automations that save even more time when completing the review.
  • Simplified data extraction: Custom data extraction forms enable users to manage complex datasets including capturing related, repeating blocks of information (also known as hierarchical data.)
Save Time Ultimate Guide to Systematic Literature Review, DistillerSR
AI Ultimate Guide to Systematic Literature Review, DistillerSR

How artificial intelligence can help the systematic review protocol

One of the most exciting new developments in the world of systematic reviews is the adoption of artificial intelligence (AI) to automate some of the more logistic-heavy tasks involved in conducting a systematic review. 

We talk a lot about the “demand for evidence-based research” reaching new heights and it’s getting ridiculous —   the demand is so high that many research teams can’t keep up with their workload. It’s a different world we live in today, and therefore, we need better tools to handle the onslaught of new information that continually overwhelms researchers. 

The answer lies in artificial intelligence. 

Today’s systematic review is still mostly done by human reviewers, but with the adoption of AI into workflows, researchers can save even more time and effort on their reviews.

So how does it work?

AI in systematic reviews is powered by a subfield of AI known as “Natural Language Processing” (NLP). This technology uses algorithms to break down sentences and words to their absolute base. By doing this and assigning mathematical properties to sentences, NLP can take large amounts of unformatted text (e.g., articles, blog posts, books, etc.) and turn it into data that can be understood by a computer. 

Natural language processing is the subfield of AI that powers many tools we use every day, such as spam filters, spelling and grammar checkers, and search engines. 

Natural language processing is the AI that powers the real heavy hitter when it comes to automating systematic review tasks. Classifiers are a statistical model that uses NLP to classify information and process it accordingly. 

In a systematic review, a classifier can answer a specific question about the text you are reviewing. “Is this an RCT?” “Is the study about humans?” are just a couple of questions that can be answered by natural language processing.

As you can imagine, this type of technology holds limitless possibilities for systematic reviewers who spend most of their time deciding whether or not to include or exclude references based on the presence or absence of specific data. 

However, before getting too excited, it’s important to remember to take a pragmatic approach to AI in systematic reviews. While the technology is fascinating and holds many opportunities to save time and automate many time-consuming SR tasks, we need to remember that the AI of today is not perfect. 

The current gold standard for accuracy in a systematic review is using dual independent screening, so we should expect AI to at least meet the accuracy rating of humans instead of expecting it to be perfect. 

It’s important to use AI in a way that it doesn’t need to perfect — use safeguards that would prevent any catastrophic problems if the AI makes a mistake, and make sure to double-check and test the AI frequently to make sure it’s working the way it should. Consider it “training” your robot to do the tasks you need it to. 

In research communities where accuracy is the difference between vital resources getting allocated to specific populations, and potential harm done to humans, we must take a pragmatic, cautious approach to automating tasks. By taking smaller steps towards integrating AI into systematic review workflows, we can build confidence in the tools, so they are used in a way that both saves time and reduces errors.

AI & NLP Ultimate Guide to Systematic Literature Review, DistillerSR