Cureus: Peer Review Dashboard
Enabling author autonomy and reducing frustration by improving the peer review dashboard
Responsibilities
UX/UI Design
Product Management
Background
Powered by streamlined tech and a well-oiled team of gig-based editors, Cureus publishes medical journal articles at low cost, and with an average publication time of just 3 weeks. When I joined, the author reviews process was causing kinks in the system and author frustration.
Spreading Medical Knowledge
The Cureus Journal of Medical Science is an innovative, more efficient platform for medical clinicians and researchers to publish and share research. With Open Access and free or low cost fees, Cureus is in an unique position to eliminate the barriers to medical publication.
How the Cureus machine works
Streamlining publication with the Cureus machine
The Cureus Journal of Medical Science utilizes technology to cut out the paperwork and to effortlessly route articles to the right people at the right time. Upon joining the team, I built the following flowchart to visualize the submission process and to aid in internal conversations.
%20Product%20Flow%20Chart%20(1).png)
The Cureus publication process is much more complex than it looks like from the surface.
Discovering the people who power the machine
Cureus has a wide range of users that all depend on the site. I categorized the users into 4 groups. Throughout my time at Cureus, I built empathy for each type of user.
Medical Professionals:
The primary audience includes authors, peer reviewers and readers.
Client Members:
Cureus supports institution channels and hosts competitions.
Gig-based Editors:
About 7 different editor roles with varying ownership and responsibilities.
Internal Users:
Team members that use the platform to perform job tasks and more.
Innovating with a skeleton crew
The Cureus internal team consists of about 5-6 people who are responsible for developing the platform and operating the journal. My responsibilities spanned both Product Design and Product management. My teammates included 2-3 engineers/CTO, the CMO and the Director of Editorial Operations. We held remote daily stand ups with the CEO.
Leading the UX process
Project Challenge:
Improve the author dashboard to reduce frustration with the peer reviews process.
Defining the business need
When I first joined Cureus, the team had been receiving an overwhelming amount of support emails from frustrated authors asking about the status of their article undergoing peer review. Our Director of Editorial Operations handled all the support emails and this was preventing him from getting to his other important tasks. The goal of the project was to improve the author dashboard so authors could get the answers they need on their own and without frustration.
Peer Review Process Overview
When an article is approved for peer review, 8-10 reviewers are invited. The reviewers that accept the invitation provide feedback as in-line comments. Once two reviews are completed, the author updates the article based on the feedback and submits it back to the editors for their approval.
Discovering user needs
My approach was to gather data by interviewing my team members about the peer reviews problem and other dashboard issues. Then I reviewed the customer support emails and ran a user survey. I synthesized the data into 5 user needs.
Authors need the ability to remind reviewers to complete their review to speed up peer review.
Authors need to the ability to invite new reviewers during the review period so they can speed up the reviews process.
Authors need to see all progress of their article's peer reviews on their dashboard so they don't have to email support.
Authors need an intuitive way to see open tasks so they can know what needs to be done.
Authors need a better understanding of the publication process so they can know what to expect.
Mapping out the detours
As I familiarized myself with the platform, I discovered the range of detours that the article can take along its journey. I mapped out a diagram for each state that included the existing UI and flowcharts to indicate how the article can flow from there. I used this as a discussion guide to confirm the process with the Director of Editorial and to ensure a thorough design and testing process.
Each color-coded box represents a state the article can be in during the peer review process.
Zooming into the existing user experience
Once I mapped out the full peer review process, I evaluated the existing state of the UI. I found it to be quite lacking and uninformative. This meant there was much room for improvement and a big opportunity to delight our users.
The existing author dashboard was barebones. There was much opportunity for improvement.
Beginning the design phase
V1: Exploring the style guide and responsive UI
To begin the design phase, I started by creating a few mockups to explore our style guide and to learn responsive web design fundamentals. I collaborated with our CTO to understand what was feasible for the team to develop. Based on our collaboration, we decided to develop a new table style since the existing internal tools table was too clunky for our medical professionals.
I explored how to work within a responsive grid system and saw the need to create a new table style.
V2: Bringing feature ideas to life
Next, I worked on mockups that incorporated all features from the user needs list. My goal was to present the mockups to the full team to get their feedback on the variety of features and layout. I created a total of 5 mockups and am sharing 3 here.
V2 Mockup 1: I added an aggregated data module with easy to access invite and remind buttons. I experimented with the highlight color of the reviewer's inline comments in the reviewer activity table.
V2 Mockup 2: Since the reviewer activity table was too tight in the right column, I moved the reviewer table into the center column. I explored adding a task list to provide a way for authors to see open tasks.
V2 Mockup 3: I moved the yellow info box to the left column since it's not a front and center feature and I explored adding an invited reviewers list to bring a more human element into the experience.
V3: Honing in on the best ideas
During the feedback session with the team, we decided the task list was unnecessary and that the yellow info box would suffice. We liked the prominent buttons in the aggregated data box since the medical professional audience is not the most tech savvy. While the invited users list was not a listed user need, we were curious to learn what they thought about the idea.
V3 Mockup: Since we decided to cut the task list, I was able to stack two boxes in the center column instead of 3 or 4.
Prototyping dashboard interactions
I chose Figma to prototype the reviews dashboard interactions. In the video, you will also notice the various modals that I designed as a part of this project.
This video demonstrates the peer review dashboard user experience post usability study updates.
Refining with usability studies
Moderating the usability studies
Medical professionals are difficult to get ahold of and it's important we use their time wisely. Since they are not considered the most tech savvy and I would need to conduct the studies remotely, I chose moderated studies in which I drove the prototype. I did not want to risk wasting time to get prototypes to load or having them fall off the rails.
 |  |
|---|---|
 |  |
Following an identical script with each research participant was important to get valid and consistent results.
Transcribing the feedback
I conducted a total of 10 usability studies and received valuable feedback from each one. I transcribed the conversations to ensure I did not miss any data and to share with the rest of the team.
Transcribing the user research was helpful to communicate user needs to the team.
Reviewer beta has viewed the article a bunch of times but hasn't completed the review. That's frustrating, why are they viewing it and not completing it?
- Research Participant
Synthesizing user data
Overall, participants were very pleased and excited about the new features that I presented. However, we identified some important patterns that needed to be addressed.
Participants felt surfacing the invited users list would impact the single blind philosophy. Therefore we will remove the invited users list.
However, participants liked the ability to add an email directly from the dashboard. Therefore we will adjust the box instead of removing it entirely.
Participants liked seeing the highlight color of the reviewer, however the full row background was confusing. Therefore we will add a small box for the highlight color.
Participants felt concerned that sending too many reminder emails would annoy reviewers. Therefore we will rate limit reminder emails.
Participants felt the 'views' column was unnecessary and it could be frustrating if a reviewer kept viewing the article without submitting it. Therefore we will remove the column.
Participants were confused by the "Pending" terminology in the reviewer activity box. Therefore we will change it to, "in Review."
Updating the UI based on user feedback
Since the feedback was positive in general, there were no sweeping changes. We felt confident to release the dashboard update with the changes identified during the usability studies.
Usability updates legend:
-
The new activity indicator was a nice-to-have feature and was high engineering effort. We cut this feature.
-
Because of participant confusion with the terminology, we changed the term 'Pending' to 'In Review.'
-
Our style guide did not support buttons that were that small. I created a new type of button in a less prominent color.
-
Some participants felt surfacing the user profiles was against the single blind philosophy. However, most users liked the ability to input an email address on the dashboard. Therefore we removed the user list but kept the invite box.
-
Participants felt seeing the number of times a reviewer viewed an article without submitting it would be frustrating. Therefore we removed it.
-
Participants felt the number of messages sent was not a necessary detail to know. Therefore we removed it.
-
From an engineering standpoint, tracking read vs. unread was not an easy task and was a nice-to-have. We cut this feature.
-
To provide proper emphasis on the button, we changed the color to grey. Since the button popped up a search modal, we changed the title to 'Search' instead of 'Invite.'
-
Participants felt seeing the reviewer's highlight color was useful, however the full row background was confusing. Therefore we moved it into a small box.
-
Since we removed columns table, I adjusted the columns by removing the magnifying glass icon from the comments column and adding a comments indicator to make it more clear that the user could click to see the modal.
Managing development
Streamlining cross-disciplinary communication
Our Director of Editorial Operations owned all copy throughout the website. Since I handled communication with the developers, I needed to create a workflow for efficient communication. In this example, I created a document with numbered screenshots so that Director of Editorial could provide the copy, and I could hand it off to the developers when he was ready.
Streamlining cross-disciplinary communication with the tooltips document.
Implementing a behavioral analytics provider
Cureus did not have a behavioral analytics provider implemented and was missing out on key data. With this project we chose to implement Amplitude. I defined the data taxonomy and prioritized the events to implement. In addition to monitoring for bottlenecks in the process, we were now able to drill down into peer review completion time over time.
The data taxonomy specs for tracking user behavior.
Managing an agile backlog of stories
We used Pivotal Tracker as our project management tool. In addition to my product design responsibilities, I wrote, tested and managed the backlog of stories.
Pivotal Tracker is an agile development tool that enabled us to track development.
Final outcome
Comparing dashboards side-by-side
The peer review dashboard was transformed from a barebones experience into a feature rich experience that provides authors the tools they need to perform tasks and get their questions answered without needing to email support.
Original Peer Review Dashboard
Updated Peer Review Dashboard
Detailing how we met the needs of users
We provided solutions for all 5 user needs identified in the user research.
%202021-10-13%2023-48-.png)
Authors need the ability to remind reviewers to complete their review to speed up peer review.
We met this need by enabling authors to send reminders directly from their dashboard. Based on the usability study, we implemented a frequency limit so authors can't spam reviewers.
My favorite part of the peer review process was the “remind” button, which sent a reminder email to each peer reviewer. I clicked it every few days, which ultimately meant that the entire peer review process lasted just a couple weeks.
- Chris Anama-Green
%202021-10-13%2023-54-.png)
Authors need to the ability to invite new reviewers during the review period so they can speed up the reviews process.
We met this need by providing two ways for authors to trigger the invitation modal directly from the dashboard.
%202021-10-13%2023-59-.png)
Authors need to see all progress of their article's peer reviews on their dashboard so they don't have to email support.
We met this need by implementing a comprehensive Peer Review Progress module and Reviewer Activity table.
Besides my paper submitted to Cureus I have no idea what is going on with the papers that are under review. However, Cureus provides clear data on how many reviewers have been asked and how many have accepted. I can see the reviews as soon as they are done.
- Rajkumar Rajendram
%202021-10-14%2000-06-.png)
Authors need an intuitive way to see open tasks so they can know what needs to be done.
Authors need a better understanding of the publication process so they can know what to expect.
We met both of these needs by implementing a dynamically changing yellow info box that includes links to the available tasks.
Further Iterations
Giving authors more control over the reviews process
As the Cureus journal evolved, so did the dashboard. In this example, authors needed the ability to end peer reviews and editors needed the ability to reject peer reviews. Therefore I designed status icons to indicate these new states in the reviewer activity table.
Since we now supported the ability to end reviews early and reject reviews, we added icons to clearly indicate the reviewer's status.
Managing coauthors from the dashboard
In response to support emails and the fact that editors were spending too much time editing author details, we decided to add the ability for the author to edit their coauthor's information at any time during the publication process. Therefore we added an authors box to the dashboard with the ability to edit and adjust the order of the coauthors.
%20_%2020.png)
A static list of authors was replaced with a fully functional Authors box so the author could easily manage their coauthors at any time.
Adjusting peer review requirements
To increase the rigor of the peer reviews process, we added the requirement to have one reviewer be invited by a Cureus peer review panel member. Therefore, I removed the completion meter in the peer review process box and added a checkmark with descriptive copy.
