Systematic review screening tools

NOTE: This article concentrates on the systematic review screening tool Rayyan, as it is web based and free to use. Other tools are available and there is more information about these at the end of the article.

During the search process for your systematic review you will likely gather together a large number of references that may potentially be relevant to your final review.

Because it's so important to identify all relevant studies, the search process should be very sensitive and this will inevitably mean that a significant proportion of your results will not be relevant.

The screening process

The process of deciding which references from your search to include or exclude is called screening and is usually carried out in two stages:

• Title/abstract screening - deciding which results are likely to be relevant based on information from the title/abstract only
• Full text screening - Final decision on whether an article should be included in your review by reading the full text of the articles which made it through from the title/abstract screening stage

Inclusion/exclusion criteria

You should have decided what criteria you will use to include or exclude a reference in the protocol for your systematic review - i.e in advance of carrying out your searches. For example, a systematic review looking to answer a therapy question might only include randomised controlled trials and exclude other types of study, or you might only be interested in studies in a particular population, intervention or outcome(s).

Multiple reviewers

Ideally more than one person should screen all the references at the title/abstract stage so that you can minimise the risk of bias.

• Reviewers should not know what inclusion/exclusion decisions the other screeners have made in advance of doing their own screening (blinding) so that their decisions are made using their own judgement only
• Disagreements in what to include/exclude should be resolved, either by majority opinion (if an odd number of reviewers) or by discussion and consensus. 

How can Rayyan help?

Rayyan is a free online reference management tool which can help with the screening phase of your systematic review or other large literature search project.

Rayyan lets you:

• Store references from multiple search sources in one place
• Assign inclusion and exclusion criteria to your references so you can track which references should be taken forward for inclusion in your systematic review
• Work with multiple reviewers on including/excluding references from your review to either:
  • Share out the work of screening large numbers of references
  • Have references screened "blind" by more than one reviewer to reduce bias

How to use Rayyan

1. Create a free account by clicking on the Sign Up link on the home page. If you already have an account you can also log in by clicking on Sign In.
2. Create a review by clicking on the New Review link and giving your review a title and description
3. Import your references - you will need to have your references saved in a format that Rayyan can read. RIS format is a commonly available download format that most databases and other sources can export in. Alternatively, use a Bibliographic Reference Management tool such as Refworks or Endnote to gather your search results together and then export the full set of references as an RIS file.
4. Invite reviewers - if you want more than one person to screen the references you can invite others to join your review. From the My Reviews list, click on the title of your review and then Invite. Reviews that you've been invited to join as a screener will appear under the Collaboration Reviews heading. The default is for reviews to be screened "blind" - ie you can't see what decisions other reviewers are making while screening is going on.
5. Screen articles - Each reviewer should go through the list of articles selecting Include, Exclude or Maybe. You can also give a reason for excluding and use the label and notes features if required. The owner of the review can keep track of the screening progress from the My Reviews area, but cannot see the decisions made by other reviewers while the "Blind" option is selected. This useful article from McGill University Library has more information on screening including using keyboard shortcuts to speed up the process.
6. Resolving conflicts - Once the screening process is complete, the review owner can switch off blinding and the screeners can review and discuss any conflicts by using the Conflicts filter in the left-hand toolbar.
7. Export your results - Once blinding has been turned off you can use the Export button to export your references including the inclusion/exclusion decisions, reasons, labels and any notes. Export in CSV format to produce a spreadsheet of your results, or in RIS format for uploading to a Bibliographic Reference Management tool such as Refworks.

More information on using Rayyan

• Rayyan user community support forum
• McGill University Library Rayyan help pages
• University of Hawaii Health Sciences Library Rayyan help pages including video tutorial

Other screening tools

• Excel - for small numbers of references, you may find that downloading your references as a CSV file and then saving as an Excel workbook is adequate.
• Covidence - Covidence is a subscription based systematic review tool which was originally created to support Cochrane reviews. It is a more sophisticated tool than Rayyan and has more options for keeping track of title/abstract and full text screening, including automatically populating a PRISMA diagram for download at the end of the process. Small reviews may be able to take advantage of the trial version (max 500 records), or you may decide that it's worth including the cost of a Covidence single-review subscription into your systematic review proposal. This article compares Covidence and Rayyan functionality.
• Systematic Review Toolbox - this site allows you to see what other tools are available to support your systematic review.