How is a systematic review defined, and what’s the process?
Definition
Cochrane defines a systematic review well in their article about Cochrane reviews.
Quote: “A systematic review attempts to identify, appraise and synthesize all the empirical evidence that meets pre-specified eligibility criteria to answer a specific research question. Researchers conducting systematic reviews use explicit, systematic methods that are selected with a view aimed at minimizing bias, to produce more reliable findings to inform decision making.”
In my words, a systematic review combines existing studies to provide an answer to a new research question. The process followed aims to minimise bias, and ensure replicability by others.
Not to be confused with…
There are lots of types of “reviews”, and Systematic Reviews are one of the biggest / most rigorous / daunting in the stack. Others you might hear of include:
- Meta-analysis - a common part of a systematic review process, combining and analysing data across many studies.
- Rapid review - a specific approach to running a systematic review, aimed at speeding up the process.
- Scientometric analysis - the science of measuring science o_O which can mean lots of things, like citations, funding, demography, or geography.
- Literature review - a light-weight, very loosely defined process for reviewing literature on a topic.
- Scoping review - like Literature reviews, but often at the start of a systematic review, or when looking for a gap that needs filling in the science.
- Evidence mapping - the building of a traversable base of evidence on a given topic. Useful when the question is as broad as "What is there to know about X?".
The systematic review process
I won’t try to provide the worlds most accurate description of the process. Instead, I’ll be trying to provide one that’s simple to follow, and easy to digest. I was a n00b at this for a while, so I'm writing this for all my fellow n00bs.
Also, this article is written to YOU - the hero at the centre of a systematic review that is actually making things happen. The chief cat-herder, and paper-shuffler. We have tips and tricks for making your life easier, as well as your collaborators.
A systematic review has 8 steps, under 3 high level phases.
- Define your review
- Step 1 - Decide on your research question
- Step 2 (optional) - Publish your protocol
- Identify research to include in your review
- Step 3 - Search for all potentially relevant articles
- Step 4 - Title and abstract screening
- Step 5 - Full text screening
- Complete your review, using included research
- Step 6 - Data extraction
- Step 7 - Synthesis
- Step 8 - Write your paper
There are SO many places within those 8 steps that can create frustrating gotcha moments, that we'll probably write a series about pitfalls to avoid in each stage, as a first-timer at SR's.
Step 1 - Decide on your research question
Deciding on your research question involves scoping the literature, including any systematic reviews that have been done on similar topics.
This step, unsurprisingly, is more about content knowledge, cursory searches, and an informal scanning of content. Once you have your question, know it’s important, and know it hasn’t been answered recently, you can write your protocol.
Step 2 (optional) - Publishing a protocol
Here you are publishing a plan for your review. This is public so that people know what you’re going to do, and can follow along.
People typically publish in one of 3 places:
- For non-Cochrane reviews, a word doc uploaded to something like PROSPERO or Open Science Framework is OK.
- Publish your intent to do a systematic review in a journal
- For Cochrane reviews, they publish through a journal with a strict template.
Step 3 - Search for all potentially relevant articles
Or, put better, find the minimal set of articles that reasonably covers the scope of your domain. This is a trade-off. Wade through lots of abstracts, or risk missing key research you should be including.
It's also where the real work starts. Typically one person will embark on a journey to find the narrowest scope of articles that will still include a handful of known “seminal” articles that were identified in the earlier searches in Step 1.
🏃 What you might do
🤔 What you might be thinking
Search query 1 = 5,000 results, all 5 seminal articles included.
Try to narrow the results.
Search query 2 - 2,000 results, 3 of 5 seminal articles included.
Too narrow. Need to expand out to include all of my seminal articles.
Search query 3 - 2,800 results, all 5 seminal articles included.
Great! They're all there. Let's see if we can get it any narrower…
Search query 4 - 2,500 results, 4 of 5 seminal articles included.
Nope, too far… Looks like my last search was the best I can do.
Final query = Search query 3
In reality your searches might bring up a lot more or a lot less articles than that. It can really depend on your discipline.
As you can see, this is an iterative push-pull between too much and not enough, which can also be a bit of a spider-sense that comes with experience.
Librarians can help a ton with this step.
Step 4 - Title and abstract screening
This step is in fact a two-in-one (or a five in one…). First, you need to remove any duplicates from your corpus of articles - if you’ve searched across multiple databases, it’s almost certain you’ll have the same article appearing twice or more in your set.
Part 1 - Deduplication
There are a number of software products that will help you find duplicates within your corpus. Syras is one of these products.
Part 2 - Screening
Once you’ve removed duplicates, at least two people need to assess each article for relevance in your systematic review. This is a “double screening.” Each title and abstract is screened against a set of eligibility criteria, typically written by the person running the review (the PhD Hero).
There are a number of software products that can help you screen articles - even Excel can be set up to do it. Syras is one of these products too; easy to use, and low cost.
Part 3 - Disagreements
Once both people have completed screening, their include/exclude assessments are compared. In many cases, they will agree to include, or agree to exclude an article, based on the eligibility criteria. But they don’t always agree.
When one person thinks an article should be included, and another thinks it should be excluded, this disagreement needs to be resolved. Typically, a resolution will err on the side of inclusion, to avoid missing potentially relevant research. Most software that helps with screening also helps you deal with this, including Syras.
Once all articles have been double screened, and disagreements completed, the included set of articles proceeds into the next step.
Aside: There are risks in this step
This step can be fraught with trial and error, as well as plain human error.
It’s easy to get half way through screening, and realise your eligibility criteria aren’t quite right, and need refining. But changing them half way through would result in an inaccurate outcome, so you typically have to start screening again!
Also, since people are human, it’s easy to miss a file that has a bunch of articles when completing an initial import, and only realise part way through screening…
That’s why Syras is focussed on this part of Systematic Reviews. We help make this less error prone, and faster for your systematic review.
Step 5 - Full text screening
The full text screening is used to assess whether an article should be included or excluded, based on information that wasn’t in the abstract.
This is again a double screening process, where full texts are viewed, and articles are assessed. Sourcing all your articles can be a heroic mission at this stage. Some are easily available, but others less so.
When you have all the articles, you need to distribute them to the other people completing your full text screening.
It’s common here to use a handful of keywords or phrases with the “Ctrl + F” shortcut to find that exact part of the text that will clearly indicate include or exclude. At this stage, it's not a full reading, it's again just a screening - so you're mostly looking for reasons to exclude it from your research.
Step 6 - Data extraction
This is about extracting any relevant data for your research, from all the articles that have been included in your systematic review.
It’s common to use a shared spreadsheet, like a Google Doc, or a specific tool for data extraction. This creates a giant table with all the data-points that you can use to synthesise your new research.
Setting this spreadsheet up can be diabolical drama, and it’s tricky having a spreadsheet that isn’t in the cloud, because people can double-up their work easily, wasting a ton of time.
TODO Here’s a Google Docs data extraction spreadsheet template we’ve seen used for Data Extraction. You can duplicate it and use it as a launchpad for yourself.
Step 7 - Synthesis
This step is where your research happens. With access to all the data from relevant articles, you’re able to draw a conclusion around your specific question that you created in Step 1.
It's in this step that you complete any meta-analysis. Meta-analysis refers to the combining of data across multiple studies to create a more precise estimate of the effects of an intervention.
A meta-analysis can form part of a systematic review, but not all systematic reviews include it. When you’ve got evidence from a diverse set of studies an approach such as narrative synthesis is commonly used and well recognised.
Step 8 - Write your paper
You should now have everything you need to write your paper, and get it submitted. A meta-analysis driven paper would have hard-data with a finite conclusion, whilst a narrative paper would paint a compelling picture with a subjective conclusion.
In some cases, both narrative and meta-analysis style conclusions are relevant, where a subset of included articles didn’t have appropriate data crossover to be included in the meta-analysis.
Upon concluding a Systematic Review, one must always put one's feet up.
Wrapping up
Systematic reviews are a big investment of your time and handling large amounts of articles can be a daunting task. At Syras, our years of tech experience have taught us a lot about making life easier and processes faster for customers. We hope to bring this experience to bear on systematic reviews, making the process more efficient and less prone to human error.
You can sign up for Syras to get started with that middle phase of your systematic review. We hope that it not only helps you and your review, but that you actually enjoy using it!