Covidence vs. Syras vs. Rayyan

This article steps through a few of the popular cloud software solutions for Systematic reviews. All three of these products provide a cloud solution – no installation or software management on your machine – and all three have some notable pros and cons.

Comparison table

CovidenceSyrasRayyan
Pricing$240USD for 1 review, included in 1 year subscription.
$211 per review, if you buy 3x reviews.
$50 per review (double)$150 per review (team)Free option
$4+ per month paid
Workflow steps coveredScreening
Full text review
Data extraction (for Cochrane)
ScreeningScreening
Notable Pros / Cons Fully featured solution (for medical reviews)
Institutional licenses
Rigid format, limited configurability
Expensive + subscription model
Reasonable, per project price
High flexibility
Screening process only
Free option
Risky data policy: they keep + can reuse your screening data
Poor design
Screening process only

Workflow

Systematic reviews have three high level stages, with steps inside each stage. Here’s our summary from a blog post we wrote about the definition of a systematic review.

Let’s look at how these three products differ in what they offer, within this workflow.

Systematic review stepCovidenceSyrasRayyan
Step 1 & 2 – Decide on your research questionThis is all you!This is all you!This is all you!
Step 3 – Search for all potentially relevant articles No support. No support No support
Step 4 – Title and abstract screening Great support. Great support. Great support.
Step 5 – Full text screening Possible, but tedious compared to Endnote. No support. Possible, but tedious compared to Endnote.
Step 6 – Data extraction Yes, specific to Cochrane studies only No support. No support
Step 7 – SynthesisThis is all you!This is all you!This is all you!
Step 8 – Write your paperThis is all you!This is all you!This is all you!

That looks pretty bleak, but there are a couple of things worth noting:

  • 80% of the repetitive, tedious, painful, error-prone effort is within steps 4, 5, and 6.
  • There are lots of other software solutions that support the rest of the workflow. Products like Endnote, and even Excel or Google Docs can be used successfully for the other workflow steps.

Breaking down the Title and Abstract screening step

There are actually a number of steps within the step here, where we can really compare the products.

Title and Abstract screening stepCovidenceSyrasRayyan
Import articlesEndNote XML PubMed text format Refman RIS (.ris)Refman RIS (.ris)Medline / Pubmed (.nbib)Endnote XMLEndnote JSONEndNote Export (.enw) Refman RIS (.ris)CSVPubMed XML
Remove duplicates from multiple importsAutomatically checked, strict rules, record-by-record resolution.

They recommend doing deduplication in Endnote
Multiple manual scans allowed, using algorithmic approaches, with bulk resolution.

Less likely to leave some duplicates than both
Automatically checked, opaque rules, record-by-record resolution.

They recommend doing deduplication in Endnote
Screening of abstracts – interfaceWell structured interfaceErgonomically designed, high-efficiency interface with shortcut keys.Cluttered and complex screening interface.
Screening of abstracts – keyboard shortcutsNo supportConfigurable shortcut keysSpecified shortcut keys
Screening of abstracts – Highlight keywordsSupportedSupportedNo support
Screening of abstracts – Screening criteriaSupported, as hintsSupported, as hints, or an interactive check-listSupport, as “exclusion reasons”
Screening of abstracts with a team of researchers.Unstructured approach – any user can screen abstracts that need screeningStructured + tracked allocation of screening work between sub-teamsUnstructured approach – requires effort to coordinate
Resolving disagreementsSpecific workflow across entire corpus.Specific workflow within assigned sets, or across entire corpus.Completed by unblinding, and changing original votes.
ExportingRefman RIS (.ris)Medline / Pubmed (.nbib)Endnote ExcelRefman RIS (.ris)Medline / Pubmed (.nbib)Endnote CSV (with comments, ratings, etc)Refman RIS (.ris)Medline / Pubmed (.nbib)Endnote CSV
Knowledge-base / Help docsSupport, training, knowledge base with Youtube videosSupport, knowledge base with Youtube videosCommunity support, knowledge base, Youtube videos
Non-Cochrane screening approachesNoneSuper-fast title-only screening (coming soon)Super-fast title-only screening

In a nutshell

Syras

The product: Syras is a flexible screening tool, useful in a variety of scenarios, and powerful enough to work well with a team. Syras can help in a wide range of uses, including Cochrane reviews. Payment is per-project, and not subscription based, which aligns with the expectations of academics.

Syras is reasonably priced, and is sold to the academics doing the reviews. To thrive, Syras will need to be a pleasure to use for academics. Expect this aspiration to be a constant into the future.

The company: Scipilot (the makers of Syras) mission is about accelerating the scientific process.

Accelerate the scientific process, for the betterment of humanity.

https://syras.org/2021/10/10/we-want-to-accelerate-science/

While both Covidence and Rayyan are born from the world of academia, Syras is delivered by two tech-sector veterans, wanting to apply their experience to academic software. With loose ties to academia, they’re approaching things with a different perspective, looking for new and innovative ways to improve the scientific process.

Scipilot is based in Australia.

Covidence

The product: Covidence provides a fully featured Cochrane-focused review tool, which fits hand-in-glove with that process. If you’re deviating from that, and looking for flexibility, it might not tick all your boxes.

Covidence is priced as an annual subscription, and is quite steep for an individual to fork out for. Cochrane tends to rely on it’s network of high-up relationships (and endorsement by Cochrane as “the solution”) to sell licenses at scale, to institutions. With the institution as the buyer, expect the experience of the end-user (that’s you) to erode over time. They’re motivated to keep institutions happy, not really academics. This is a common software trap, where buyer ≠ user.

The company: If the mission of a company means a lot to you, you’ll be glad to hear that Covidence has a noble mission:

Covidence is a social enterprise platform, used around the world, to help turn the flood of new scientific research into high-quality knowledge by accelerating the systematic review workflow.

We are on a mission to help the world create trustworthy knowledge.

https://www.covidence.org/careers-covidence/

They’re Australian based, and have a strong history with Cochrane, partnering closely with them for training and much more.

Rayyan

The product: Rayyan provides a free tool to get through the screening process of your systematic review. It’s great for small scale reviews, but has limited configurability, and a fairly rough interface. If good looks, and the privacy of your review data aren’t important to you, it’s a good solution for you.

The company: Rayyan’s mission is about accelerating scientific discovery.

Rayyan Systems Inc. is a leading provider of collaboration and research tools that power the global research community in pursuit of accelerated scientific discovery.

https://www.rayyan.ai/about-us

It’s worth noting that “Rayyan was developed through the Qatar Computing Research Institute, funded by the Qatar Foundation, a nonprofit that supports education, science, research, and community development initiatives in Qatar.” (source), and as part of the Rayyan terms, there is a fairly sweeping clause around “ownership and content, which gives them ownership rights on content you upload, including content such as comments in a review.

When products are free, you could be the product. This idiom holds true for almost all products. While it’s difficult to be certain, it’s hard to not wonder, when reading the Rayyan fine-print.

Leave a Reply

Your email address will not be published. Required fields are marked *