Laptop with screenshot of AUS score displaying

Building the Accessible Usability Scale

In the early days of Fable, I was attending an #A11yTO meetup at the Firkin on Yonge in Toronto. I remember sharing the idea of what we were building and being asked a question that I didn’t know the answer to, “Are you helping with accessibility or usability for people with disabilities?”

Low-hanging fruit

It was early 2018 and Fable had only just started to engage screen reader users in digital product testing. If you’re unfamiliar with screen readers, they are a common assistive technology (AT) for people with blindness to access and navigate computers. Almost everything that we were finding on websites had to do with poorly labelled form elements and a lack of structured content, common issues that are often referred to as low-hanging fruit when it comes to accessibility.

“Low-hanging fruit doesn’t mean low impact.” – Samuel Proulx, Community Manager, Fable

Accessibility or usability?

The W3C says that web accessibility relates to people with disabilities being able to equally perceive, understand, navigate, and interact with web content. They then define usability as relating to a user experience being effective, efficient, and satisfying. These ideas are different, but not mutually exclusive.

At Fable, we help advance both accessibility and usability. Through thousands of hours of customers engaging people with disabilities on Fable’s crowdtesting platform, our customers are making products easier to understand and navigate. Accessibility is often measured by conformance to the Web Content and Accessibility Guidelines (WCAG). But how do we measure usability for people with disabilities?

Measuring usability for assistive technology users

As companies build and mature their accessibility programs, we often see common themes materialize. After a team builds accessibility into their development cycle and addresses their backlog of WCAG issues, the focus shifts to user experience. Companies consider “do AT users like our product” and how “does the UX compare to the rest of our users”?

In hindsight, “time to complete a task” was the least meaningful metric that we found in our efforts to measure user experience at Fable. The time it takes for an AT user to complete a task can be a valuable datapoint at the individual level but loses utility in aggregate. The problem is with averaging and comparing. Comparing an expert JAWS user to a novice Dragon NaturallySpeaking user is not meaningful. When looking at data related to the time to complete a task, the likelihood of misinterpretation increases. These experiences are fundamentally different.

Build on something that works

The System Usability Scale (SUS) was developed in 1986 by John Brooke. It’s one of many psychometrically designed surveys, though it is unique both in that it remains popular and has been consistently used by organizations for decades. John Brooke’s objective with the System Usability Scale was to take a quick snapshot of people’s satisfaction when using systems, or what we’d commonly refer to today as digital products like apps and websites.

The context is different for an AT user

The SUS consists of ten statements to be responded to on a Likert scale, which is multiple choice with options ranging from ‘Strongly Disagree’ to ‘Strongly Agree’.

One of the questions in the SUS survey is, “I think that I would need the support of a technical person to be able to use this system.” Imagine a screen reader user was going through a sign-up form that required them to draw their signature. In this case, the user likely requires assistance not from a technical person, but another person who doesn’t rely on a screen reader themselves.

Another question is, “I would imagine that most people would learn to use this system very quickly.” In this case, assistive technology users are highly aware that their experiences are unlike most users.

By adapting the SUS questions to relate to the context of an assistive technology user more meaningfully, we can similarly quantify and measure the perceived usability of digital products for assistive technology users. Now, the framework can still measure usability issues, but can do so for those who are impacted by accessibility issues.

Presenting the Accessible Usability Scale 1.0

In the first iteration of the Accessible Usability Scale (AUS), we decided to specify the system we are asking for feedback on. It’s easy to replace the word “system” with one that better aligns to how digital products are referred to today such as “website” or “app”.

We also updated some of the language to make it easier to understand and less vague.

Read how this compares below:

Accessible Usability Scale System Usability Scale
I would use this website frequently, if I had a reason to. I think that I would like to use this system frequently.
I found the website unnecessarily complex. I found the system unnecessarily complex.
I thought the website was easy to use. I thought the system was easy to use.
I think that I would need the support of another person to use all the features of this website. I think that I would need the support of a technical person to be able to use this system.
I found the various functions of the website made sense and were compatible with my  technology. I found th