Author Archives: Anthea Sutton

How we find the tools…

You might have noticed we’ve added a few new things to the toolbox site recently to give our users a bit more information on how the toolbox works.  We’ve recently published our eligibility criteria for tools here, but how do we find potential candidates for inclusion in the first place?

Regular visitors, particularly those who follow @SRToolbox on Twitter, may know that we run a search on MEDLINE every month, with the aim to find any new tools, or new papers about existing tools.  The search strategy was developed from a mapping exercise we conducted in 2017.  We analysed the existing papers catalogued in the toolbox using the Yale MeSH Analyzer tool.  This identified 77 MeSH headings which were ranked in order of frequency.  The bibliographic data (including abstracts) of the analysed publications were then uploaded to VOSviewer and a density visualization map (See Figure 1) was generated to identify a network of frequently occurring and relevant free-text terms.

The most frequently occuring terms include: data, validity, tool, systematic review, review, risk, bias, user

Figure 1: Part of the Density Visualization Map generated via VOSviewer showing the most frequently occurring terms in red.

The MEDLINE search strategy was then developed based on the findings of the mapping exercise.

But this is only part of the story.  We have found very little duplication across the different retrieval methods we use, therefore using a range of approaches seems to be best.  An analysis of the tools identified between January and September 2018 revealed that while the MEDLINE search is successful at identifying unique tools, engaging with the systematic review community via the toolbox website and social media found the most tools.  So, as we have always said, the toolbox needs you! If you see, hear about, or develop a tool, please add it to the site here to help us ensure the toolbox remains current, relevant and useful. And do feel free to contact us with any queries or suggestions.


SRToolbox at the Cochrane Colloquium

If you follow us on Twitter, you may have spotted that the SRToolbox was at the Cochrane Colloquium the past few days. We had a poster presentation on “Better, faster, stronger: how to find tools to expedite the systematic review process”.  This is based on some work we’ve been doing behind the scenes over the past year or so.  We’re keen for the toolbox to be as comprehensive and up-to-date as it can be, but with an ever-increasing number of tools being developed, how can we do this efficiently?  In 2017, we mapped the tools literature and used this to develop a MEDLINE strategy that could be automatically run but this is only part of the picture. We want to know about tools as soon as possible, and often we find out about them first from systematic reviewers and developers, either submitting to the toolbox or via social media.

Our Poster at Cochrane

We’ve been monitoring where new and updated tools came from in 2018, and presented the results at the Cochrane Colloquium. The findings are very much as we expected – we need to continue to use a variety of methods (databases, hand-searching, monitoring Twitter) and engaging with the systematic review community is essential! So if you’ve submitted any tools to the toolbox, we thank you and hope you will continue to use and contribute to this valuable resource.  Want to know more? Please feel free to contact us via Twitter or direct.

Posted by Anthea

A quick trial of JBI SUMARI systematic review of the literature software

A quick trial of JBI SUMARI systematic review of the literature software

SUMARI (System for the Unified Management, Assessment and Review of Information) is software developed by the Joanna Brigg’s Institute to support systematic reviews.  The software aims to facilitate the entire review process from protocol development to writing a systematic review report.  Recently new and improved, at ScHARR we were offered a free trial via Ovid so I was keen to give it a go and report back to the SRToolbox community.

Access     Create a Review     Collaboration     The Review Protocol     Importing Studies

Screening   Appraisal    Data Extraction & Synthesis    Reporting    Overview   What do I think?


If you are accessing SUMARI via Ovid, it appears in the “EBP Tools” drop-down menu at the top of the screen when you are using a database in Ovid such as MEDLINE.  Alternatively you can login direct from the SUMARI website and select the “log in via EBP Network/Ovid” option.   You do need to be logged into an Ovid personal account to access it, if you don’t have one already see how to create one here.

Create a Review

Once you have accessed SUMARI, your first option is to create a review project.  SUMARI now supports ten review types (see Table 1).  There is also the ability to create a custom review if none of the ten review types meet your requirements.

Table 1 Review Types supported by Sumari


If you create the review project, you will be designated as project owner.  It is simple to invite other “participants” to the review by entering names and email addresses.  Once you have multiple participants, it is possible to assign and/or change roles. For a full list of the available roles and permissions, see here.

The Review Protocol

Once a review project has been created, there are a number of tabs across the top of the screen.  The first is “Protocol”. This tab contains an extensive protocol template that can be edited. This seems very useful, particular for those new to reviewing. The template itself can be edited by adding or removing sections, again allowing for customisation of your own review. There is an option to import citations to use in your protocol (via an XML file) or manually enter citations.  Once you have citations available, you can simply drag and drop them where you want them to appear and the reference list is automatically updated. This took me a while to figure out, as there is no instruction on the protocol tab to do this. However, once I’d watched the tutorial on Developing your protocol it was easy.  Your protocol can be exported as a DOCX which is useful for sharing with anyone beyond those collaborating on SUMARI, including registering your protocol.

Importing Studies

As an Information Specialist, I was really interested in trying out the importing studies function to see how that worked.  Currently, the only options are manual entry, or importing as an XML file (although “Importing other reference management files into the system” is listed as a priority feature to be developed in the future). I initially tried exporting approximately 450 results direct from Medline as XML and importing that.  This didn’t work and I got an error asking me to check it was a valid XML file. After a bit of trial and error, I discovered it only works via a reference management tool such as Endnote.  I imported from MEDLINE into Endnote and then exported an XML file from there, and imported this into SUMARI. Despite there being multiple steps, once I imported into SUMARI, the 450 studies were imported quickly, in around 20 seconds.  Using SUMARI in conjunction with a reference management tool such as Endnote doesn’t particularly concern me and in my experience is often the case with other systematic review software tools too.

Once the studies are imported, you can clearly see how many need to screen, with a handy “decision pending” at the top of screen.

The list of studies appear in alphabetical order (by first author).  There does not seem to be an option to sort the list alternatively, but there is a filter option.  However, this appears to search for the term(s) entered in the citation only (effectively the title of the study) so can not be used as definitive.  For example a search for “randomised” retrieved one study from my 450 as it appeared in the title as “a randomised study”. Searching additional citation fields (such as the abstract) would be useful here I think.

Which leads me on to my discovery that abstracts hadn’t been imported from Endnote.  I initially thought it was because of the Output style in Endnote I’d used. However, I re-imported the same studies using an output style which includes abstracts and still no abstracts.  This means in screening studies, you would need to do abstract screening elsewhere (SUMARI offers title screening only currently) or have a list of the abstracts to view if you prefer to title and abstract screen at same time.  This is also a factor in using SUMARI in tandem with a reference management tool. During this experiment with importing, I also discovered that SUMARI allows the importing of duplicates and there is no alert to warn you of duplicates nor a duplicate check. You could manually check the list of studies and delete any duplicates.  However, you would need to keep a note of how many were duplicates, as I can’t see that SUMARI has a “bin” or “trash” area where they remain. My preference would be to use Endnote for this, import all search results into Endnote, delete any duplicates in Endnote by running a duplicate check, and then export the remaining unique references as an XML file to import into SUMARI.

For each study citation, there is an option to edit if something is incorrect.  There is no abstract field, so as well as not being able to import them you could not enter them manually if you were so inclined.  


The screening options are “include”, “exclude” or “delete”, there is no query or unsure option.  Again this is a matter of preference, studies (even those you are unsure of) could be included until you decide otherwise, and you could use the comments field to note that you are unsure at this point.  I was pleased to see that if you delete a citation, you do get a warning and have to confirm that you really want to delete it.

The screening function took me a little while to get used to.  I found that when you click on your decision, the box doesn’t shade or change colour so it wasn’t immediately obvious the mouse click had worked.  But I was reassured when the citation promptly disappeared and saw the number of decisions pending decrease.

If you click “exclude” you are prompted to enter a reason.  I was pleased to see as you type you get a drop down menu of reasons you’ve previously used in this review, so this is good for consistency in terms of reporting.


Once you have included a study it goes through to appraisal stage.  I couldn’t see an option within the appraisal section to change your mind and exclude, but you can go back to the list of included and excluded studies, which has options to “revert” any study back to the decision pending list.

There are various tools for appraisal integrated into SUMARI including the JBI Critical Appraisal Tools and the Cochrane Risk of Bias Tool.  Currently there does not seem to be an option to import other appraisal tools, but creating your own critical appraisal tools is listed as a priority feature for development.  You are required to select which appraisal tool you wish to use for each individual study, and it seems to default to RCT as this is the first on list.  Some mapping might be useful here, to suggest an appropriate tool based on the citation, or to remember which tool you used last as it is likely you will be using the same tool across multiple studies.  I also noted there is no option for the PDF of the study to annotate as you do the appraisal, you would need to do this elsewhere if this is a function you would normally use.

You need to remember to save your appraisal once you have completed it, for it to appear in the list of studies as “completed”. I found if you don’t save it and go back to the list it remains “In Progress”.

Data Extraction & Synthesis

Despite completing an appraisal, the study did not go into the extraction section.  I checked the help pages and found this is because it needs two reviewers to complete critical appraisals before extraction can take place.  I can not see that there is an option to set number of reviewer preferences when you create a review. Without completing the extraction, I could not trial the synthesis options either, but I can see that there are functions for both qualitative synthesis or meta-analysis.


The review (reporting) option is not yet available but will be in the future.  Currently there is a link to author guidelines for JBI Database of Systematic Reviews and Implementation Reports as an alternative.


For a quick glance of your review in progress, the overview section gives you a quick summary of how many screened, appraised, extracted, from the total number of studies.

What do I think?

I’ve listed what I think are the current strengths of using SUMARI from this quick trial below.  I’ve also listed suggested functions for development I would like to see in future versions (see Table 2).

Table 2: Strengths & Potential Areas of Development

Less of a priority, but some other features I would find useful are:

  • More brief instructions as you hover your mouse over functions – I would have found this particularly useful in inserting citations into the protocol, and importing studies from an XML file.
  • A sample review available to practise on as you learn how to use the software
  • The functionality to populate a PRISMA Flow Diagram (this may be planned as part of the report builder function which is forthcoming)

If you have any comments about using SUMARI or have any suggestions for features to be developed, you can contact the team at the Joanna Briggs Institute at:

Anthea Sutton

Information Specialist

ScHARR, The University of Sheffield

The Systematic Review Toolbox at ISPOR 20th Annual European Congress

We’ve been working away behind the scenes at the Systematic Review Toolbox with some exciting stuff on horizon scanning for new tools. We conducted a mapping exercise of the software tool research publications currently indexed in the toolbox using Yale MeSH Analyser and VOS Viewer text mining software.  Our findings are informing the development of a search strategy to ensure new tools are identified and indexed in the toolbox efficiently.

We’re pleased to be presenting this work via a research methods poster at the ISPOR 20th Annual European Congress in Glasgow this week.  The poster (PRM246 Location I12) will be on display on Wednesday 8th November from 08.45 until 14.00.  The author discussion hour with Dr Chris Marshall, developer and editor of the SR toolbox, will be between 13.00-14.00, so if you are attending ISPOR and want to know more, please come and say hello then.

Density Visualization Map of potential search terms generated by VOSviewer

Density Visualization Map of potential search terms generated by VOSviewer



Demonstrating the SR Toolbox

Last week I was invited to demonstrate the Systematic Review Toolbox (SR Toolbox) at our in-house Systematic Reviews Issues and Updates Symposium (SYRIUS) at ScHARR (School of Health and Related Research) at The University of Sheffield.  The symposium provides an opportunity for researchers to get together and share updates of systematic review methodological work being undertaken in ScHARR, so it was a great opportunity to promote the SR Toolbox and the resources it contains.  The SR Toolbox slot on the symposium programme consisted of a short presentation introducing the toolbox with a potted history of its creation and development.  This was followed by a live demonstration, which included some tips on using the toolbox.  Here are a few of those tips:

1.  You can use “Quick Search” to search for more than just the tool name

You can use “Quick Search” to search for tools by name, but you can also use it to search the titles and descriptions of tools. For example, if you’re interested in finding tools for critical appraisal, type it into the “Quick Search” box and you will find tools that mention “critical appraisal” in either the title or description of the tool. However, be aware this is exactly what it says it is, a “Quick Search”, so if you want a comprehensive list of all the critical appraisal tools in the toolbox, be sure to use “Advanced Search”.


2.  In “Advanced Search” selecting more than one feature will search for the features using “AND”

When using “Advanced Search”, it is important to note that if you select more than one feature, the toolbox uses the Boolean Operator “AND” and will return tools that meet all the features you selected.


3.  If you want to browse all the tools in the toolbox…

For Software Tools, tick the “Any” box underneath where it says “Check ‘Any’ if not concerned about any specific features”.


For “Other Tools”, check all 4 “Find me” boxes.


4.  The toolbox provides references to tool-related journal articles where available

The tool records in the toolbox link to/reference journal articles where they are available.  This might be an article about the development of the tool or a review of using the tool by a systematic reviewer who’s tried it out. If you know of any articles relating to tools in the toolbox, please get in touch and we will update the tool record accordingly.

I concluded the session by discussing the “community-driven” aspect of the toolbox.  Systematic reviewers and tool developers are encouraged to submit tools to the toolbox via the “Add a New Tool” feature.  The remit of the toolbox is to catalogue both software and other types of tools/supporting mechanisms (such as checklists, guidelines and reporting standards).  So if you discover a new tool that meets these criteria, please share it with your systematic review peers and colleagues by submitting it to the toolbox, which will help to continue the development of this really useful resource.

Anthea Sutton

Senior Information Specialist

ScHARR, The University of Sheffield