Fable webinar recap
How to scope your accessibility research study
How much research is enough? How can you scope your accessibility research? In a recent webinar, Elana Chapman, Accessibility Research Manager at Fable, presented some findings from Fable’s investigation into what saturation might look like in the context of accessibility research.
This article gives a brief overview of the content; for more detail, you can watch the 10-minute video summary, or view the entire webinar recording if you’d really like to dive in.
What is saturation?
“Saturation in a qualitative study is a point where themes emerging from the research are fleshed out enough such that conducting more interviews won’t provide new insights that would alter those themes.” – Neilson Norman Group
The basic idea of saturation is that as you conduct user research, you are less likely to uncover unique findings with each additional user engagement. Eventually, you will hit the point of saturation: when the value you could gain from conducting more research is less than the value you could get from acting on your findings or conducting research on a different topic.
However, when approaching research from the perspective of accessibility and inclusion, achieving this saturation can be difficult, if not impossible. There’s always a different technology or disability type you could include. How can the framework of saturation be redefined to help scope accessibility research studies?
Hypothesis
In order to investigate these questions, the Fable research team created two hypotheses:
- Assistive Technology (AT) users uncover more: one unmoderated session with an AT user produces more insights than five unmoderated sessions with a non-AT user from the general population.
- AT users find more issues than just accessibility: even after removing technical accessibility findings, there will still be more insights from one AT user than from five non-AT, general population users.
Proving these hypotheses would reframe the idea of saturation in accessibility research from value lost, to value gained, by helping to understand how much value can be gained from a small user group.
The research
In order to test these hypotheses, Fable conducted a study leveraging the Canada Helps website. Three tasks were explored in unmoderated usability sessions:
- Search for and learn about a charity
- Create an account
- Purchase a gift card for $100
A total of 24 users were engaged, with each task explored by five general population users and three AT users, a screen reader user, a magnification user, and an alternative navigation user. Analysis for each of the four user groups was done by a different researcher to reduce influence between different user groups.
The findings
These results show us several interesting things. One being, the impacts that having a targeted user can have on research. One of the general population users for the first task happened to have a lot on knowledge on charities and had many preferences and opinions on the task. They alone uncovered 13 of the 18 general population findings. Another finding is that we can see the importance of performing more than one exploration in our study. As each task had different results for our hypotheses, we know that we haven’t reached saturation yet and there is more to learn.
If you would like more information and background on these results, we highly recommend watching the full webinar recording, available on this page.
Takeaways and next steps
This research shows that assistive technology users can, at times, provide more value than a larger group of general population users. However, that value is best obtained by working with multiple types of assistive technologies and triangulating the findings. You can also clearly see the value of targeted participant pools for providing high value results.
Going forward, Fable intends to conduct further research into these hypotheses to further refine our conclusions and findings. We also plan on publishing a deeper dive into this research in the future. In the meantime, we welcome and encourage on-going dialogue on these topics.
