Tracking progress: KPIs to measure progress in digital accessibility
A key performance indicator (KPI) is a quantitative measure of progress towards a goal. Fable’s KPI framework offers a practical way for teams to report on accessibility progress. They are easily communicated to senior leadership, sidestepping the complexity and resource demands of measuring accessibility using the Web Content Accessibility Guidelines (WCAG).
Measuring beyond compliance
Many leaders struggle with evaluating the success of their accessibility programs and understanding if progress is being made.
Accessibility awareness grew through the development of international standards and legislation such as the Americans with Disabilities Act (ADA). Today, the most common way to measure success or progress is through a legal compliance lens. Compliance is measured by auditing digital products against the Web Content Accessibility Guidelines (WCAG). Organizations may find this approach challenging because:
- The work is often outsourced to a vendor and budgets may only accommodate an annual accessibility audit.
- Digital products are not static, and a formal accessibility audit only gives you a snapshot of a particular point in time.
- Audits come too late in the product development lifecycle, typically after a product is released and it’s more difficult to fix accessibility issues.
- It’s hard to action the findings and connect them to progress if you have hundreds of issues across multiple different products.
- Accessibility audits usually don’t tell you what specific challenges people with disabilities have using your digital products in real life.
- Auditing doesn’t uncover opportunities to make improvements to overall usability and user experience.
Compliance audits can be the right tool for certain scenarios. For example, when you have a regulatory obligation to report on accessibility, or a customer requires an Accessibility Conformance Report as part of the procurement process. For regularly measuring and reporting on progress, consider using KPIs instead.
The opportunities with KPIs
- Practical
Auditing for conformance with the Web Content Accessibility Guidelines (WCAG) can be time consuming and requires expertise that not all teams have. KPIs can be used by anyone and tracked on a more frequent cadence. - Understandable
Executives need accessibility progress reports with metrics that are relevant to the business and easy to understand. KPIs provide an internal reporting method that complements external compliance reporting. - Achievable
A large volume of accessibility issues can overwhelm teams. With clear KPIs teams can focus on what matters most and track their progress over time. - Diagnostic
Accessibility KPIs can help identify gaps in your product development
Accessibility KPI categories
Fable’s three overarching categories for accessibility KPIs are:
Each of these three categories will have dependencies and can tell you a lot about the state of accessibility within your organization. For example, people-related KPIs will tell you if teams have the skills needed to build accessible products. Without implementing inclusive development processes, teams will remain reactive rather than proactive, slowing down progress. These two KPI categories (people and process) are the foundational metrics you need to create accessible products.
1. People accessibility KPIs
Organizations need to have researchers, designers, engineers, and product managers with accessibility knowledge. Monitoring who has completed training and what level their skills are at helps you know if you have the internal capabilities to create accessible products.
Framework for people KPIs
The dimensions you want to consider are skills, time, user needs, and work deliverables. Questions that your KPIs should answer include:
- Do teams have accessibility skills?
- Do teams have time to apply accessibility in the work that they do?
- Do teams understand user needs of people with disabilities?
- Are teams applying accessibility to their work?
Here are some examples of KPIs that can help answer these questions.
Tie people KPIs to maturity
If your organization uses an accessibility maturity model such as the Digital Accessibility Maturity Model, you can assign maturity levels to your KPIs for executive reporting. For example, if a model has the four dimensions Reactive, Defined, Managed, and Optimized, you can evaluate each KPI against those dimensions.
Here’s how you might assign maturity levels to time spent on accessibility:
- Reactive: no proactive accessibility, but team will fix bugs assigned to them
- Defined: specific time periods or percentage of hours spent on accessibility
- Managed: time spent on accessibility is tracked and tied to product quality
- Optimized: accessibility issues are traced to process gaps which are fixed
2. Process accessibility KPIs
The processes that teams follow and the tools that they use will either support their accessibility efforts or hinder them. Choosing tools that support and automate parts of your accessibility efforts, including user recruitment and testing, can make inclusive product development more achievable for teams.
Framework for process KPIs
The dimensions to consider are accessibility issues, automated testing, design system coverage, and inclusive research frequency. Questions that your KPIs should answer include:
- Are your processes reducing accessibility issues created over time (or the severity of issues or the time needed to fix issues)?
- How much automated testing coverage do you have across your products?
- What portion of design system components have accessibility baked in?
- How frequently are teams user testing with people with disabilities?
To answer these questions, consider the following KPI examples.
Tie process KPIs to maturity
Here’s an example of how you could assign accessibility maturity levels to your design system.
- Reactive: no design system and only ad-hoc component creation
- Defined: design system adopted with accessibility currently being added
- Managed: accessibility embedded in design system components and documented
- Optimized: design system is used to innovate and collaborate on accessibility
3. Product accessibility KPIs
Rather than evaluating digital products from a compliance lens, you can consider the user experience for people with disabilities and measure it using a variety of common metrics.
Framework for product KPIs
Product KPI dimensions to consider are compatibility with assistive technologies, usability for people with disabilities, customer satisfaction, and specific accessibility areas of focus. Questions that your KPIs should answer include:
- Can users complete critical tasks (i.e. is it compatible with assistive technology)?
- How usable is a product for people with disabilities?
- How satisfied are people with disabilities with your product?
- Are you succeeding in accessibility efforts for a specific area of focus?
A few KPIs for products that could help answer these questions include:
Tie product KPIs to maturity
If you were to assign accessibility maturity levels to features, here’s how you could evaluate them.
- Reactive: no compatibility or user testing, bug fixes only
- Defined: regularly testing feature accessibility prior to launch
- Managed: accessibility testing is part of all stages of product development
- Optimized: research on accessibility needs drives new features and innovation
Choosing accessibility KPIs
Within each category of people, process, and product there are many KPIs you can monitor. You don’t need to track all of them. Consider the culture of your organization, existing KPIs you track, and choose KPIs that align with both.
Start with one to three KPIs, build the muscle of tracking accessibility and making improvements, and then add more KPIs later. Having too many metrics is like having too many priorities – it can slow down work rather than help you make progress.
Choosing accessibility KPIs
Within each category of people, process, and product there are many KPIs you can monitor. You don’t need to track all of them. Consider the culture of your organization, existing KPIs you track, and choose KPIs that align with both.
“There’s a lot of data available, but that doesn’t mean that it’s all meaningful data. Choose metrics that clearly indicate where you are in relation to your goals.” — Melissa Sape, Global Digital Accessibility Service Manager at Dow
Start with one to three KPIs, build the muscle of tracking accessibility and making improvements, and then add more KPIs later. Having too many metrics is like having too many priorities – it can slow down work rather than help you make progress.
“There’s a lot of data available, but that doesn’t mean that it’s all meaningful data. Choose metrics that clearly indicate where you are in relation to your goals.”
Melissa Sape
Global Digital Accessibility Service Manager at Dow
Next steps
This isn’t a complete list of all possible KPIs for accessibility. It’s a starting point to help you think beyond compliance when it comes to measuring your progress, in a way that is meaningful and achievable for your organization.
The best KPIs can be measured frequently, are easy to understand, and align with your organization’s existing approaches to monitoring digital products.
By shifting to metrics that consider people, process, and product, you can unlock an outcome-focused approach to accessibility.
We’re looking for feedback on these KPIs and case studies from other organizations.
Get in touch with Kate Kalcevich, Head of Accessibility Innovation if you’d like to share.