Systematic Review Toolbox in 2017

2017 was an exciting year for the toolbox! Let’s begin with the tools that were added to the database throughout the year…

24 tools were added to the toolbox in 2017.  Of these, 18 were some form of software tool and 6 were a type of checklist, guideline or reporting standard.


Below is a monthly breakdown of what was added throughout the year.













There were also some updates to existing tools in the toolbox:

This year, we ran our third workshop on software tools and packages to support systematic reviews at the University of York.  We had a packed agenda with plenty of hands-on with various freely and commercially available systematic review tools.  We’ll be running the workshop again this year.  If anyone is interested, please visit the YHEC training page for more information.


In October, I packed up the toolbox and went along to the 3rd annual International Collaboration for the Automation of Systematic Reviews.  It was a really interesting meeting and it was great to see so many people in one room enthusiastic about review tool development and evaluation.  The future of systematic review automation is very bright!

In November, the toolbox made it’s second appearance at ISPOR to show off it’s new search strategy!  Thanks to Anthea Sutton, we now have a dedicated search strategy to identify new tools to add to the toolbox, which we plan to run once a month.  This month (January), there have already been a handful of new tools identified by the search and added to the toolbox, which is great!  Check out the abstract here.


A big thank you to all those who submitted tools (and updates to tools) in 2017 and to those who attended the workshop!  There are some exciting plans for the toolbox in 2018 so watch this space!



The Systematic Review Toolbox at ISPOR 20th Annual European Congress

We’ve been working away behind the scenes at the Systematic Review Toolbox with some exciting stuff on horizon scanning for new tools. We conducted a mapping exercise of the software tool research publications currently indexed in the toolbox using Yale MeSH Analyser and VOS Viewer text mining software.  Our findings are informing the development of a search strategy to ensure new tools are identified and indexed in the toolbox efficiently.

We’re pleased to be presenting this work via a research methods poster at the ISPOR 20th Annual European Congress in Glasgow this week.  The poster (PRM246 Location I12) will be on display on Wednesday 8th November from 08.45 until 14.00.  The author discussion hour with Dr Chris Marshall, developer and editor of the SR toolbox, will be between 13.00-14.00, so if you are attending ISPOR and want to know more, please come and say hello then.

Density Visualization Map of potential search terms generated by VOSviewer

Density Visualization Map of potential search terms generated by VOSviewer



Tool round-up: January – April 2017

Here is a round-up of the tools added to the toolbox for 2017 so far (January – April):

DECiMAL – Data extraction guide for complex meta-analysis, primarily aimed at systematic reviewers preparing data for meta-analysis: – tool to extract data from web pages. In the context of systematic review it has been used to download citations from websites to increase transparency and reproducibility of grey literature searching: 

VOSviewer – software tool for constructing and visualising bibliometric networks. These networks may for instance include journals, researchers, or individual publications, and they can be constructed based on co-citation, bibliographic coupling, or co-authorship relations. VOSviewer also offers text mining functionality that can be used to construct and visualise co-occurrence networks of important terms extracted from a body of scientific literature: 

Gephi – Open source visualization and exploration software for all kinds of graphs and networks: 

RIS Export – Tool that converts search results files into RIS files for ease of importing into reference management software:

CADIMA – supports the conduct of systematic reviews and evidence/systematic maps by the provision of a freely available online tool that: 1. guides review authors through the evidence synthesis process, 2. facilitates the coordination of cooperating team members, 3. eases steps with considerable workload and 4. guarantees for its thorough documentation. The evidence synthesis tool was established and is further developed in a close collaboration between the Julius Kühn-Institut and the Collaboration for Environmental Evidence is free and supports the conduct of systematic reviews and evidence/systematic maps:

3D Covariate Visualiser – 3-D evidence network plot system: a new software tool to aid the exploration of covariate imbalances and effects in network meta-analysis. Users are required to register to use the system. This is an innovative tool which was developed by DRG Abacus in collaboration with Leicester University: 

Health Assessment Workspace Collaborative (HAWC) – HAWC is a modular, content management system designed to store, display, and synthesize multiple data sources for the purpose of producing human health assessments of chemicals. This online application documents the overall workflow of developing an assessment, from literature search and systematic review, to data extraction (human epidemiology, animal bioassay, and in vitro assay), dose-response analysis, and finally evidence synthesis and visualization: 

JSTOR Text Analyser – JSTOR Text Analyser allows you to upload a document and will analyse the text to find the key topics and terms used. JSTOR Text Analyser then priortises these terms and uses the ones it deems most important to find similar content (articles, book chapters etc) in JSTOR. You can refine the search results, by adding, removing or adjusting the priority of terms:

COMPASS – The tool aims to give an outline of the key elements with respect to quality of clinical evidence of orphan medicinal products (OMPs). The tool can be applied to assess the quality of evidence of an OMP based on information in the registration dossier, for example by local reimbursement agencies, pharmacists or clinicians –

Thanks to all those who submitted tools.


Demonstrating the SR Toolbox

Last week I was invited to demonstrate the Systematic Review Toolbox (SR Toolbox) at our in-house Systematic Reviews Issues and Updates Symposium (SYRIUS) at ScHARR (School of Health and Related Research) at The University of Sheffield.  The symposium provides an opportunity for researchers to get together and share updates of systematic review methodological work being undertaken in ScHARR, so it was a great opportunity to promote the SR Toolbox and the resources it contains.  The SR Toolbox slot on the symposium programme consisted of a short presentation introducing the toolbox with a potted history of its creation and development.  This was followed by a live demonstration, which included some tips on using the toolbox.  Here are a few of those tips:

1.  You can use “Quick Search” to search for more than just the tool name

You can use “Quick Search” to search for tools by name, but you can also use it to search the titles and descriptions of tools. For example, if you’re interested in finding tools for critical appraisal, type it into the “Quick Search” box and you will find tools that mention “critical appraisal” in either the title or description of the tool. However, be aware this is exactly what it says it is, a “Quick Search”, so if you want a comprehensive list of all the critical appraisal tools in the toolbox, be sure to use “Advanced Search”.


2.  In “Advanced Search” selecting more than one feature will search for the features using “AND”

When using “Advanced Search”, it is important to note that if you select more than one feature, the toolbox uses the Boolean Operator “AND” and will return tools that meet all the features you selected.


3.  If you want to browse all the tools in the toolbox…

For Software Tools, tick the “Any” box underneath where it says “Check ‘Any’ if not concerned about any specific features”.


For “Other Tools”, check all 4 “Find me” boxes.


4.  The toolbox provides references to tool-related journal articles where available

The tool records in the toolbox link to/reference journal articles where they are available.  This might be an article about the development of the tool or a review of using the tool by a systematic reviewer who’s tried it out. If you know of any articles relating to tools in the toolbox, please get in touch and we will update the tool record accordingly.

I concluded the session by discussing the “community-driven” aspect of the toolbox.  Systematic reviewers and tool developers are encouraged to submit tools to the toolbox via the “Add a New Tool” feature.  The remit of the toolbox is to catalogue both software and other types of tools/supporting mechanisms (such as checklists, guidelines and reporting standards).  So if you discover a new tool that meets these criteria, please share it with your systematic review peers and colleagues by submitting it to the toolbox, which will help to continue the development of this really useful resource.

Anthea Sutton

Senior Information Specialist

ScHARR, The University of Sheffield

Tools of the Year 2016

Let’s kick-off the new ‘News’ section with a look back over the tools that were added to the Toolbox in 2016.

This year, 38 tools were submitted and added to the Systematic Review Toolbox. This included 23 software tools and 15 other (i.e. checklists, guidelines and reporting standards etc.) tools.


Below is a month-by-month recap (with links to the tool pages in the toolbox!)…














A big thanks to everyone who submitted tools to the Toolbox throughout the year!