Benchmarking accessibility using Fable Engage

Benchmarking approaches

Measure usability with AUS

The Accessible Usability Scale (AUS) is a free tool to measure the usability of a digital product for assistive technology users. Inspired by the System Usability Scale (SUS), the Accessible Usability Scale consists of ten questions administered at the end of a user experience to generate a score.

Using Fable Engage, you can get an AUS score after a User Interview and from a Self-Guided Task. You can average a minimum of three user’s AUS scores for each task flow, but we recommend running a study with five to seven users of different types of assistive technologies to get a more robust score.

You can track the AUS score for all requests in a project using the project analytics feature in Fable Engage as shown below.

An Average AUS score over time report for requests in all projects is available to company managers on the company dashboard in Fable Engage:

If you’re using the System Usability Scale (SUS) as a metric already, you can evaluate it against your AUS scores to see how the experiences of assistive technology users compare with all product users.

Measure task completion with Compatibility Tests

Compatibility tests will give you the following quantitative data points:

• Completion rate (how many testers completed the task)
• Accessibility (how accessible did testers perceive the task to be)
• Recommendation (whether they would recommend the task to another user of their assistive technology)
• Ease of use (how easy/difficult the task was to complete)
• How many issues were identified

You can also review the tester feedback and assign a severity rating. I like to use the following approach to evaluate severity:

High: Issue blocks completion of a task​

Medium: User can’t complete task as expected ​
(for example, it’s hard to do, takes a long time or requires a work around)​

Low: ​

• User suggests a fix based on preference/ease of use​
• User has to think about how to complete task ​
• User makes more than one attempt before succeeding​

You can track the task completion score for all requests in a project using the project analytics feature in Fable Engage, as shown below.

A Task completion rate over time report for requests in all projects is available to company managers on the company dashboard in Fable Engage:

You can manually combine task completion rates for different products to compare accessibility progress across your organization: 

Creating a benchmarking project

Here’s how you can create a Fable project specifically for benchmarking.

1. Determine what product or task flow you want to benchmark. Benchmarking works best with a product that is actively being worked on and has accessibility improvements as part of that workflow. Create a new benchmarking project for that product.
2. Determine what the most important tasks are within the product or choose a single task flow to benchmark. For example, here are some common key task flows:
   • Create an account and log in
   • Search
   • Add to cart
   • Checkout
   • Contact support
3. Decide which request types to use. Use Compatibility Tests for more quantitative data points and to get feedback from more users with fewer requests. Use User Interviews or Self-Guided Tasks to get AUS scores and more qualitative usability feedback.
4. Decide how many requests to run for each key task flow. The more data points you collect, the more accurate your benchmarking will be. A comprehensive benchmarking project that presents a snapshot of your current level of accessibility could involve:
   • Compatibility Tests to get completion rates for each key task.
   • Five to seven User Interviews or Self-Guided Tasks to get an AUS score for each key task.
5. Create requests for each task flow within the project. If using the example above, you’d create one Compatibility Test and five to seven User Interviews or Self-Guided Tasks for each key task flow.
6. Once you’ve set up the project and all the requests within it, you can re-run the requests on a regular cadence using the recurring request feature. This works best if you make changes to the product based on findings before re-running benchmarking.

Example benchmarking approaches

Benchmarking usability for people with disabilities versus usability for all users

This approach complements a general usability metric such as SUS. Use User Interviews or Self-Guided Tasks to collect AUS scores to compare against the SUS scores.

This is a good approach for evaluating if you are truly being inclusive as an organization. If you have significantly worse usability for users of assistive technology compared to all users, that’s a sign that you need to make sure that accessibility is prioritized as part of user experience within your organization.

Analyze your data

Compatibility Tests

The completion rate of a Compatibility Test can be a good indicator of severe accessibility barriers. Anything less than 100% of testers complete the task, it means there was a barrier that prevented them from completing the task and those barriers should be high priority to fix. You can easily find the blockers identified by reviewing the tester feedback on Fable Engage.

The number of issues identified is less important than the feedback you get from testers. You need to put the issues in context of severity (using the guidance I gave in Measure task completion with Compatibility Tests or your own rating system). A compatibility test with 5 issues that are all high severity is more concerning than one with 25 issues that are all low severity.

Recommendation can be a third metric that you use along with completion rate and number of high severity issues to help contextualize the data.

Accessible Usability Scale

For the AUS score, it’s critical to understand not just the score itself, but to have the notes from the researcher on what happened during the User Interview that caused a low or high score. For Self-Guided Tasks, you can take notes as you watch the video clip to determine the parts of the experience that were good and bad.

It’s the notes that will help you understand what needs to be fixed so that your next benchmarking exercise can lead to a higher score. This isn’t always the case because sometimes fixing one issue that was a major barrier, just means users will move further along in the task flow and discover other major barriers that they couldn’t previously identify. This is still good learning for the team. Using prototype reviews and inclusive design approaches can help avoid these major barriers in future product development and redesigns.

Project Analytics

You can also use the Project Analytics feature to visualize the data generated by your requests. For any projects where you have more than one request, when you open the Project Analytics tab you will get statistics that include:

• Average AUS score for all User Interviews and Self-Guided Tasks in the project (and how the results compare to your AUS target if you’ve set one).
• Average task flow completion for all Compatibility Tests in the project.
• Number and type of requests completed.
• Number of tester engagements by assistive tech type.

You can also create reports here that will give you charts with the following:

• Average task flow completion by assistive technology type
• Average issues logged per user by assistive technology type
• Average AUS score by assistive technology type

Comparison charts let you compare up to 10 requests within a project:

• AUS score comparison
• Task completion comparison
• Compatibility Test survey result comparison

These charts can give you a more high-level overview of a series of requests and help you to quickly identify areas that need to be improved.