How ACT is making college entrance exams more accessible

In 2019, ACT started looking into whether braille paper tests (and even screen reader accessible PDFs), offered by most assessment companies including ACT, were actually the most equitable way for blind and visually impaired examinees to experience the test. The issue was one of independence—blind examinees using these accommodations usually require a scribe to enter answer choices for them. After some investigation, we determined that students should be able to use a screen reader on our test, to provide the best opportunity for them to demonstrate their knowledge and skills.

I knew that we would need users to inform our interaction design decisions but connecting with students with disabilities wasn’t easy. I reached out to school counselors hoping they could help us recruit students with disabilities who could come to the ACT campus in Iowa City to participate in UX testing. We wanted to facilitate traditional, in-person, observational studies. But travel and wayfinding aren’t necessarily easy for high school kids with disabilities. The whole process was beginning to seem too time intensive and complex for everyone.

So, in early March of 2020, I started looking for a remote solution for usability studies. I tested a few standard UX tools and couldn’t find one that was itself accessible. COVID-19 hit and then, suddenly, in person testing was impossible. Remote testing was no longer just convenient; it was our only way forward. Luckily, in April, Fable and I connected. I saw one demo of their remote user testing platform with assistive technology users and I was sold. The price was reasonable, which made getting budget approval easy.

Development was in early stages, and we were all excited to have this great opportunity to shift left on a normally unalterable exam. But we didn’t always agree internally or with our external partners about the best interaction design for certain test items. The testing we were able to do with Fable laid the foundation for the more challenging accessibility decisions we made along the way. That direct feedback based on lived experience, from around 20 skilled assistive technology users, empowered ACT to challenge assumptions in the assessment industry about what is and is not best for accessibility.

For example, in an English test, we have questions that refer to a phrase from a passage. How should a blind student have access to the context they need within the passage? The prevailing opinion internally and from other assessment companies was that part of the passage should be brought into the question stem. Through testing and discussion with the Fable community, we found that this may give an examinee too much or too little information, compared to the context a sighted student can gather by scanning the passage.

We decided instead to put an anchor link from the question back to the point where the phrase occurs in the passage. This gives a screen reader user the freedom to explore and navigate from that specific point. Through further compatibility testing we found that Firefox and Chrome treat focus differently after a user follows an anchor link. The results from that testing enabled us to make informed decisions about which browser and assistive tech combinations we could support, while maintaining a level of consistency through content delivery.

There is a lot of guidance out there about how to make websites accessible, but very little is precise enough for standardized testing. What made the difference for ACT was listening to the users, and bringing the experience of people with disabilities into our processes.

There are differences between assistive technologies that present challenges on a standardized test. WCAG is an excellent technical resource but is not a sufficient guide for exactly what we were trying to do. It leaves a lot of content and UX decisions up to the creators, intentionally. I needed everyone on our content writing and development teams to see what was really happening with users so they could bring their expertise to the problems. Being able to work with the Fable community helped bridge that gap.

One issue that stands out is how assistive technology, particularly screen readers, interacts with math questions.

Initially, nothing in our questions would be read by the screen readers. Then, everything except some numbers would be read, and it seemed completely random. We tested more and found that plain text numbers were being read but MathML was not. We worked with Fable and found that no matter what we tried, if we had a label on MathML content with “role=math,” NVDA would not read the MathML, and JAWS would not read the label. So, an NVDA user wouldn’t be able to get to the MathML and would therefore have to do math by simply hearing it in sentence form through the label. Of course, this would be incredibly difficult, and would require a lot of working memory. While this might be okay for everyday schoolwork, the differences in the way the math was read by NVDA and JAWS provided too inconsistent an experience for a standardized test. We consulted Fable users and rapidly iterated for about a month, looking for a solution that would work for everyone.

Finally, we made a tough call to remove the labels for now, and rely on the MathML since it is navigable character by character. This was a controversial idea, particularly for standardized test designers who prefer labels for the perfect consistency they provide. But for users, it turns out that labels don’t provide the best experience when trying to work through math.

Remote testing with Fable crushed any barriers we had. Nobody had to deal with travel and wayfinding through buildings. There was no equipment setup, no contact information in spreadsheets, or tracking of consent in emails all over my inbox. Fable cut out a lot of the management and administrative tasks typical of this kind of engagement.

One of the challenges for me with accessibility-in-use testing is finding ways to share the results with team members in a clear, quick way that fosters empathy and understanding. Remote testing allowed us to better demonstrate user stories, communicate our message more clearly, and enable a deeper understanding of the issues internally, which in turn enabled developers to take ownership of accessibility themselves. Then they can bring to bear all their personal expertise to collaborate on novel solutions.

Ultimately, Fable eliminated barriers, cut down on recruitment time and costs, and ensured easy documentation, sharing, and collaboration, all of which accelerated ACT’s accessibility efforts.

If it wasn’t clear already, the Fable community is completely essential for us and we’re excited to keep working together.