A quick trial of JBI SUMARI systematic review of the literature software
SUMARI (System for the Unified Management, Assessment and Review of Information) is software developed by the Joanna Brigg’s Institute to support systematic reviews. The software aims to facilitate the entire review process from protocol development to writing a systematic review report. Recently new and improved, at ScHARR we were offered a free trial via Ovid so I was keen to give it a go and report back to the SRToolbox community.
If you are accessing SUMARI via Ovid, it appears in the “EBP Tools” drop-down menu at the top of the screen when you are using a database in Ovid such as MEDLINE. Alternatively you can login direct from the SUMARI website and select the “log in via EBP Network/Ovid” option. You do need to be logged into an Ovid personal account to access it, if you don’t have one already see how to create one here.
Create a Review
Once you have accessed SUMARI, your first option is to create a review project. SUMARI now supports ten review types (see Table 1). There is also the ability to create a custom review if none of the ten review types meet your requirements.
If you create the review project, you will be designated as project owner. It is simple to invite other “participants” to the review by entering names and email addresses. Once you have multiple participants, it is possible to assign and/or change roles. For a full list of the available roles and permissions, see here.
The Review Protocol
Once a review project has been created, there are a number of tabs across the top of the screen. The first is “Protocol”. This tab contains an extensive protocol template that can be edited. This seems very useful, particular for those new to reviewing. The template itself can be edited by adding or removing sections, again allowing for customisation of your own review. There is an option to import citations to use in your protocol (via an XML file) or manually enter citations. Once you have citations available, you can simply drag and drop them where you want them to appear and the reference list is automatically updated. This took me a while to figure out, as there is no instruction on the protocol tab to do this. However, once I’d watched the tutorial on Developing your protocol it was easy. Your protocol can be exported as a DOCX which is useful for sharing with anyone beyond those collaborating on SUMARI, including registering your protocol.
As an Information Specialist, I was really interested in trying out the importing studies function to see how that worked. Currently, the only options are manual entry, or importing as an XML file (although “Importing other reference management files into the system” is listed as a priority feature to be developed in the future). I initially tried exporting approximately 450 results direct from Medline as XML and importing that. This didn’t work and I got an error asking me to check it was a valid XML file. After a bit of trial and error, I discovered it only works via a reference management tool such as Endnote. I imported from MEDLINE into Endnote and then exported an XML file from there, and imported this into SUMARI. Despite there being multiple steps, once I imported into SUMARI, the 450 studies were imported quickly, in around 20 seconds. Using SUMARI in conjunction with a reference management tool such as Endnote doesn’t particularly concern me and in my experience is often the case with other systematic review software tools too.
Once the studies are imported, you can clearly see how many need to screen, with a handy “decision pending” at the top of screen.
The list of studies appear in alphabetical order (by first author). There does not seem to be an option to sort the list alternatively, but there is a filter option. However, this appears to search for the term(s) entered in the citation only (effectively the title of the study) so can not be used as definitive. For example a search for “randomised” retrieved one study from my 450 as it appeared in the title as “a randomised study”. Searching additional citation fields (such as the abstract) would be useful here I think.
Which leads me on to my discovery that abstracts hadn’t been imported from Endnote. I initially thought it was because of the Output style in Endnote I’d used. However, I re-imported the same studies using an output style which includes abstracts and still no abstracts. This means in screening studies, you would need to do abstract screening elsewhere (SUMARI offers title screening only currently) or have a list of the abstracts to view if you prefer to title and abstract screen at same time. This is also a factor in using SUMARI in tandem with a reference management tool. During this experiment with importing, I also discovered that SUMARI allows the importing of duplicates and there is no alert to warn you of duplicates nor a duplicate check. You could manually check the list of studies and delete any duplicates. However, you would need to keep a note of how many were duplicates, as I can’t see that SUMARI has a “bin” or “trash” area where they remain. My preference would be to use Endnote for this, import all search results into Endnote, delete any duplicates in Endnote by running a duplicate check, and then export the remaining unique references as an XML file to import into SUMARI.
For each study citation, there is an option to edit if something is incorrect. There is no abstract field, so as well as not being able to import them you could not enter them manually if you were so inclined.
The screening options are “include”, “exclude” or “delete”, there is no query or unsure option. Again this is a matter of preference, studies (even those you are unsure of) could be included until you decide otherwise, and you could use the comments field to note that you are unsure at this point. I was pleased to see that if you delete a citation, you do get a warning and have to confirm that you really want to delete it.
The screening function took me a little while to get used to. I found that when you click on your decision, the box doesn’t shade or change colour so it wasn’t immediately obvious the mouse click had worked. But I was reassured when the citation promptly disappeared and saw the number of decisions pending decrease.
If you click “exclude” you are prompted to enter a reason. I was pleased to see as you type you get a drop down menu of reasons you’ve previously used in this review, so this is good for consistency in terms of reporting.
Once you have included a study it goes through to appraisal stage. I couldn’t see an option within the appraisal section to change your mind and exclude, but you can go back to the list of included and excluded studies, which has options to “revert” any study back to the decision pending list.
There are various tools for appraisal integrated into SUMARI including the JBI Critical Appraisal Tools and the Cochrane Risk of Bias Tool. Currently there does not seem to be an option to import other appraisal tools, but creating your own critical appraisal tools is listed as a priority feature for development. You are required to select which appraisal tool you wish to use for each individual study, and it seems to default to RCT as this is the first on list. Some mapping might be useful here, to suggest an appropriate tool based on the citation, or to remember which tool you used last as it is likely you will be using the same tool across multiple studies. I also noted there is no option for the PDF of the study to annotate as you do the appraisal, you would need to do this elsewhere if this is a function you would normally use.
You need to remember to save your appraisal once you have completed it, for it to appear in the list of studies as “completed”. I found if you don’t save it and go back to the list it remains “In Progress”.
Data Extraction & Synthesis
Despite completing an appraisal, the study did not go into the extraction section. I checked the help pages and found this is because it needs two reviewers to complete critical appraisals before extraction can take place. I can not see that there is an option to set number of reviewer preferences when you create a review. Without completing the extraction, I could not trial the synthesis options either, but I can see that there are functions for both qualitative synthesis or meta-analysis.
The review (reporting) option is not yet available but will be in the future. Currently there is a link to author guidelines for JBI Database of Systematic Reviews and Implementation Reports as an alternative.
For a quick glance of your review in progress, the overview section gives you a quick summary of how many screened, appraised, extracted, from the total number of studies.
What do I think?
I’ve listed what I think are the current strengths of using SUMARI from this quick trial below. I’ve also listed suggested functions for development I would like to see in future versions (see Table 2).
Less of a priority, but some other features I would find useful are:
- More brief instructions as you hover your mouse over functions – I would have found this particularly useful in inserting citations into the protocol, and importing studies from an XML file.
- A sample review available to practise on as you learn how to use the software
- The functionality to populate a PRISMA Flow Diagram (this may be planned as part of the report builder function which is forthcoming)
If you have any comments about using SUMARI or have any suggestions for features to be developed, you can contact the team at the Joanna Briggs Institute at: firstname.lastname@example.org.
ScHARR, The University of Sheffield