Usability study conducted for a client at Providence Health & Services for their Swedish Express Care site as part of a Usability Studies course
Jan 2018 - Mar 2018
Company and Website Overview
Providence Health & Services is a health care system headquartered in Washington state. Swedish Medical Group is one of Providence's subsidiaries. Swedish Express Care is a collection of medical services that provides express medical services for low-acuity health problems, such as a colds, skin irritation, and sprains, as opposed to high acuity, emergency problems like breaks and heart attacks.
There are 4 services, as shown above, and the Express Care website allows patients to learn more about each service and schedule appointments.
Our client at Providence was a Senior UX Researcher on the UX team and asked us to conduct a usability test on Swedish Express Care's website. Specifically, he wanted to find out if users could use the site to determine which Express Care service they wanted to use.
In order to fulfill the client's research goal, we identified 3 research questions to guide our usability study.
01 Do users comprehend what the different Express Care Options are?
02 How easily and successfully do users discover which conditions are treatable by Express Care services?
03 How easily and successfully do users discover what insurance Express Care services take?
All participants were recruited through User Research International based on the client's criteria. Participants were to have never interacted with the Express Care site before, have booked a medical appointment in the last 6 months, and not be afflicted with chronic high-severity conditions. In addition, our client wanted to look specifically at 3 segments of people, Millennials, Gen Xers and Parents with children under 17 years old. Full screener can be found here.
Each session was one-on-one and involved a participant, moderator, and notetaker. The sessions consisted of a Task Observation where participants undertook 3 tasks on the Express Care website, and a Semi-Structured Interview.
The first part of our study had participants completing 3 tasks using the Express Care website. We chose to do a task observation to see exactly where the breakdown (if any) of using the site was. These tasks were designed to address our research questions 2 and 3, as they let us look at how users went about looking for information like insurance accepted or problems the different services could treat which might inform their decisions in choosing a specific service.
The participants were given a scenario that asked them to complete the following tasks (pictures are of the target sections of the website for each task):
Task 1 - Treatable Condition
Determine which of the services, if any, could treat pink eye.
Task 2 - Untreatable Condition
Determine which of the services, if any, could treat a broken leg.
Task 3 - Insurance
Determine if the services take "Premera" insurance.
Participants were instructed to think aloud during the tasks. We observed participants' navigation of the site in completing their tasks and also recorded the screens and participants using Morae. The specific task protocol can be found here.
After the task observation portion of the sessions, we asked 4 questions, some of which addressed the overall usability of the site. The 3rd question aimed to uncover if users comprehended the differences between each Express Care service (research question 1). We thought a participant-provided explanation of each of the services would best show that particular user's comprehension of the services rather than a task, as tasks often do not show the user's mental models even if thinkaloud is encouraged.
We conducted data analysis by reading over our notes from sessions and rewatching the videos from Morae. We used RealTimeBoard to create affinity diagrams and userflows for each task.
- Do users comprehend what the different Express Care Options are?
- Yes, somewhat. A lot of it is through pre-existing assumptions.
- How easily and successfully do users discover which conditions are treatable by Express Care services?
- Most could discover the list of treatable conditions, but with difficulty (extra steps taken, thrown into loops).
- How easily and successfully do users discover what insurance Express Care services take?
- Most participants discovered Premera was taken, but with difficulty (extra steps taken, thrown into loops).
We found 3 main reasons for the usability issues our participants experienced, explained below. A comprehensive detailing of our findings can be found in the full usability research report.
1. Information Overload
Participants felt that many of the pages, especially the home page, were too busy and led to confusion of where to seek wanted information.
2. Confusion Over Site Structure
Participants had trouble completing each task because at many points, a link on the homepage would lead to anchor points at the bottom of a new page. It was also difficult to find information about insurance and took many different types of steps between each service. Information about insurance for Virtual Care, for example, was in the FAQ section, while it was on a separate page for At Home Care.
3. Page Format Inconsistencies
Another point of confusion was the inconsistent way the same information was presented across the different pages for the different services.
Impact + Feedback
We presented our high-level findings to the UX, finance, marketing, and engineering teams at Providence. The presentation deck can be seen here.
One of the UX team's goals for the year is to redesign the Express Care website for the rest of the year and indicated they will be considering our findings when undergoing redesign efforts.
The study also started conversation among the different teams, especially the ones more removed from design, about the importance of the website's layout in customer experience and satisfaction. For example, the finance, marketing, and UX teams agreed to set up a future meeting to discuss the findings further and what it meant for the workflow for the next few months.
- Spend time thinking through how data is going to be used before undertaking data analysis: We first thought to do a quantitative analysis comparing the number of clicks and scrolls each person completed while doing each task. However, many of the tasks actually fed into each other and a mere quantitative comparison would not be meaningful. The process to count each action by rewatching each session wasted time, and in the end, the data was not valuable for our client.
- Standardize protocols and have each moderator practice multiple, or just have one moderator to ensure consistency: We had very inconsistent moderation due to wording and different styles of encouraging participants to think aloud. We feel many results and different ways of approaching the tasks resulted because of the moderation difference. As this was a class, and every team member was required to moderate, we could not stick to one moderator, but in an industry situation, I would opt to have a single moderator, or standardize protocol down to the prompts to get participants back on track.
- Do pilot runs of tasks before testing on real participants: We found that one of the tasks (finding that a broken leg was not treatable at any of the Express Care services) was a sort of "trick" task, as this information was never stated. However, the wording of the task, "where can you treat a broken leg" made participants assume it could be treated at one of the services. This confusion over the task was only discovered after we had already tested 4 out of 10 participants as we scheduled back-to-back sessions and did not have time to reflect between. Doing a pilot would have easily allowed us to discover this high-level problem in our protocol and may have led to different results for the task.
- Many times design decisions are not autonomous to the UX and design teams: Our team was surprised when one of the Providence team members interrupted the presentation to say that they were aware of the inconsistencies with the Virtual Care website, but could not do anything about it because the page was standardized to the video service they used for virtual appointments. Despite obvious usability issues in some sites, there are often bureacratic matters design teams must face in redesigning a website.