https://hackmd.io/@gabalafou/waaat

WAAAT!

JupyterLab as a Lab for
Web App Automated Accessibility Testing

15:30 - 16:00
📍 Louis Armand 2

Note:

• ➡️ Click here for these slides in presentation mode ⬅️

Speaker: Hello, hello, welcome, welcome. Good afternoon. You made it! You made it to the end of JupyterCon. Give yourself a little stretch if it feels good. This, by the way, this whole thing here, is proof that you can't trust clichés like "they save the best for last." But, actually, this isn't quite the last part of JupyterCon. After this, there are lightning talks, and there are sprints this weekend, but as far as I'm aware, that's it. So we're going to get started in a bit. The title of this talk is Waaat! JupyterLab as a Lab for Web App Automated Accessibility Testing. waaat does that even mean? I'll explain later. Today is Friday, May 12. Gorgeous weather in Paris. We're scheduled from 3:30 to 4 in Louis Armand 2. If it helps you to follow along you can load my presentation in your browser at hackmd.io/@gabalafou/waaat. This will show you all of the content in one page, including my speaker notes. If you prefer to see the content in slide mode, just like it's shown up here, there is a link to the slide mode in the notes of the document view.

Links:

• This same document in slide mode
• Wondering why this presentation is in Comic Sans font?

---

hackmd.io/@gabalafou/waaat

Intro

Note:

Speaker: Before we get started, you should know that I'm not going to be doing an intro to accessibility, so if you're not familiar with accessibility, there will probably be some parts of the presentation that might not make sense, but hopefully most of it should be understandable.

---

hackmd.io/@gabalafou/waaat

Talk outline

1. Story: Focus visibility
2. Conclusion
3. Q&A

Note:

Speaker: Here is a slide of our outline today. First, I am going to tell a story and walk you through one accessibility issue that I worked on. My hope is that by using a specific issue, it will help to make things concrete that might otherwise be a bit abstract. After that, I will do a little wrap up. And finally, I hope to leave some time for questions. Oh and there will be floofs along the way. A floof, in case you don't know, is just a cute word for pet. And on this slide we have one of my coworker's floofs. She's a black and white cat and in this photo she's sitting with a very serious face on the purple roofline of what I imagine is some kind of pet house? Stephannie assures me that Frachesca always has a very serious face.

---

hackmd.io/@gabalafou/waaat

Who is Gabriel?

GitHub @gabalafou

Front-end / Accessibility 🦙
Quansight Labs

Note:

Speaker: But first, who am I? My name is Gabriel Fouasnon. I work at Quansight Labs. My background is as a front-end engineer and my focus these days is in accessibility. One fun fact about me: I once had a 15 minute conversation with Marvin Minsky, sometimes called the father of artificial intelligence, about long division, and I didn't realize until weeks later who I had been talking with.

---

hackmd.io/@gabalafou/waaat

Note:

Speaker: Okay another floof. This is Moose. Nope he's not a moose. He's just a golden doodle who's sometimes a little confused about his identity, which, like, I can relate.

---

hackmd.io/@gabalafou/waaat

Story: Focus Visible

Note:

Speaker: So now I'm going to tell you a story about solving an accessibility issue in JupyterLab.

---

hackmd.io/@gabalafou/waaat

Backstory

#9399 and WCAG 2.4.7

Note:

Speaker: But first, a little backstory. So, working in open source is interesting. When I joined the folks working on accessibility in JupyterLab, I quickly learned about an issue on GitHub, number 9399. It was posted in 2020 by the user "manfromjupyter". I asked a coworker more about this issue, and they said they actually don't know who "manfromjupyter" is. Like the man in the iron mask, which, I felt obliged to make that reference since we are in France. So #9399 is a bit infamous, not just for the air of mystery, but also because it's a massive, extensive accessibility audit of JupyterLab. And I would like to draw your attention to just one small part of #9399, just one accessibility issue out of many reported in that audit. In this one area, manfromjupyter points out that the JupyterLab version 2.2.6 was not following the web content accessibility guideline number 2.4.7, which is about making focus visible.

Links:

• jupyterlab #9399
• WCAG 2.4.7

---

hackmd.io/@gabalafou/waaat

Focus indicator

on first	on last

Note:

Speaker: what is focus visible? So focus visible, without getting too technical about it, is about putting a visual indicator around whichever part of the web page the user is interacting with. The typical example for this is, if you're a sigted user, you're on a web form and you move from one form field to another and you see an outline appear around the form field that you just entered, like in the example screenshot on this slide. On the left, the first name field has focus. On the right, the last name field has focus. But focus is not just for form fields. If you rely on the keyboard, and you press the tab key to move around web pages, then you also want focus indicators around links and buttons and such. In terms of accessibility, having a visible focus indicator is an important feature for anyone who relies on a keyboard to operate a page. It's also important for folks with attention limitations.

---

hackmd.io/@gabalafou/waaat

Fixme: JupyterLab focus

Note:

Speaker: So how does this show up in JupyterLab? Here we have a screenshot of the first screen of the JupyterLab UI, with the file browswer and the launcher tab open. Which part of the UI has focus right now? You might want to guess the new file button, this one here in the top left labeled with a "+" symbol. But you would be wrong. It's actually just impossible to look at this screenshot and see which element has focus. And that's the whole problem.

---

hackmd.io/@gabalafou/waaat

Culprit: bad css

😭 Don't do this ❗️

/* Disable default focus outline */
:focus {
  outline: unset;
}

Note:

Speaker: Digging into this, it was pretty easy to find the culprit. There was a rule in the CSS that I've put up here on the slide. It says: for any element on the page that has focus, unset the outline, which is also known as a focus ring. So this just turns off the default browser focus indicator. Which is bad for accessibility. Please don't do this. Some elements in JupyterLab created their own focus indicators, using things like the element border, so they were unaffected by this rule. But for everything else, this created an accessibility issue.

---

hackmd.io/@gabalafou/waaat

Fixed in JupyterLab

Note:

Speaker: So the fix was pretty easy. I just had to remove the CSS and then fix a few things that got broken in the UI. And here's the result. Now you know which element had focus in the previous JupyterLab screenshot: it was the first file in the file browser. So why is this important? There are lots of reasons, but let's just take one. Imagine you have a disability that makes it hard for you to use a mouse but you can use a keyboard. If you rely on the keyboard, you probably heavily rely on keyboard shortcuts, so you want to know if you press a keyboard shortcut, for example to delete a file or rename a file, then you want to know which file will be affected by that keyboard shortcut. And if you can see which file has focus, that can be difficult if not impossible.

---

hackmd.io/@gabalafou/waaat

Note:

Speaker: Okay. It's floof time again. This is Maysoon Zayid with her late cat Beyoncé. Maysoon is an actress, comedian, writer, and disability advocate. She had the number one TED Talk of 2014, titled "I got 99 problems … palsy is just one." It's really funny, I recommend it. This photo, by the way, is from the Disability Visability project, which you can visit by going to disabilityvisabilityproject.com. You can find links to all of these in the presentation document.

Links:

• I got 99 problems … palsy is just one
• Disability Visability Project
• Licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.

---

hackmd.io/@gabalafou/waaat

Testing

Note:

Speaker: Okay? We fixed the accessibility bug, yay! We made the world a better place, Frodo, can we go back to the Shire now? This is where I would like to talk about the security analogy. By which I mean, a lot of times in the web accessibility space, an analogy is made to security. They're both cross-cutting concerns. They're both kind of asymptotic in the sense that you can never really say your app is 100% secure; similarly you can't ever really say your app is 100% accessibile. Such statements don't really mean anything. If this were a security bug, you probably wouldn't call the job done until you'd written a test to make sure that the bug doesn't reoccur. Because that would be embarrassing, if somebody found reported the same security bug some time down the road, after you'd already declared it fixed. It's the same with accessibility. Once you fix it, don't you want to make sure it stays fixed?

---

hackmd.io/@gabalafou/waaat

Accessibility checkers?

Pa11y? Lighthouse? Axe-core?

Note:

Speaker: If you ask DuckDuckGo or some other more popular but less privacy-protecting search engines about accessibility testing, there are a number of different things out there that you will probably find, but three of the most popular accessibility checkers that you will find are Pa11y, Lighthouse and Axe-core. But in reality both Pa11y and Lighthouse are built on Axe-core, so it's really just Axe-core. Pa11y actually support one other engine called HTML CodeSniffer. But my point is that if you do web search it will look like there are lots of different checkers to choose from but there's really only a few, and the space is really dominated by Axe-score. So what happens if you run these checkers against JupyterLab?

Links:

• Pa11y GitHub
• Lighthouse GitHub
• Axe-core GitHub

---

Note:

Speaker: In this screenshot, you have JupyterLab on the left and Lighthouse on the right. Specifically, you have a Lighthouse accessibility report on the JupyterLab UI that you see on the left. And guess what? There are a few issues it identifies but none of them are about WCAG 2.4.7, focus visible.

---

hackmd.io/@gabalafou/waaat

100/100 does not mean accessible

Note:

Speaker: I bring this up because I think it's really important for developers to understand what these checkers say when they have scanned your web site or app and found no accessibility issues. It's a little bit like using a spell checker. Just because you use a spell checker doesn't mean that your paper is free of writing errors. But I think that's also where these tools create confusion. Because if you run a spell checker against a document and you get no errors, then you probably don't have any spelling errors. You may have some malapropisms or something like that but you can be fairly confident that every word in your document is also in a dictionary. But if you run an accessibility checker against your web site or app, it does NOT mean that your site is free of accessibility bugs. And, at this point I should mention that I am really indebted to a developer named Manuel Matuzovic for my thinking on this subject. I saw a talk of his in which he went wild on a site and made it diabolically inaccessible, like down to the source code. This slide shows on the left the highly accessible website that he started with and on the right—no this isn't a bug in my slides—he actually made the site invisible, but still got a "perfect" score on Lighthouse. Okay, so, where does that leave us? For this particular acccessibility bug about a missing focus indicator, we can't use the off-the-shelf checkers. They don't catch the bug. Maybe we could write a manual test script. What would that look like? If I imagine myself, I would load JupyterLab in the browser, I would press the tab key, and each time I press the tab key, I would check the screen for changes. If at any moment, there's no focus indicator, then the test fails. If only we had a tool that would let us write a test like that …

Links:

• Manuel Matuzović's talk: Building the most inaccessible site possible with a perfect Lighthouse score

---

hackmd.io/@gabalafou/waaat

Playwright

Note:

Speaker: This is an image I took from the Wikipedia page for Deux ex machina. It's of Euripedes' play Medea, performed in 2009 in Syracuse, Italy. The sun god sends a golden chariot to rescue Medea. Deux ex machina: because we've reached that point in our story. Turns out, there is a tool that does exactly what we want. It's called Playwright, it's a free and open source tool from Microsoft.

Links:

• Playwright GitHub
• Image license CC-BY-2.5, as seen on the Deus ex machina Wikipedia image page

---

hackmd.io/@gabalafou/waaat

Go-between

↔️ ↔️

Note:

Speaker: So what does Playwright do? Well, it acts as a kind of go-between. You can imagine your test code on one side, Playwright in the middle, and on the other side, a browser engine, such as WebKit (the engine behind Safari), or Gecko (the engine behind Firefox), or Chromium (the engine behind Chrome and just about every other browser on the market now). Your test code can tell Playwright to interact with the page in ways that are closer to how a user would interact with the page. So that means you can ask Playwright to click buttons or links on the page, to fill out form fields, to take screenshots, to navigate backwards or forwards, and so on.

Links:

• Clip art source

---

hackmd.io/@gabalafou/waaat

Contrast: A11y checkers vs. Playwright

Web page ➡️ Axe-core
Test script ➡️ Playwright

Note:

Speaker: I think probably the easiest way to contrast Playwright with an accessibility checker like Axe-core is to look at what each of them take as input. Axe-core takes a web page as input. It's almost like a Xerox. You give it one page at a time, it scans the page, and it tells you what it found. Playwright, on the other hand, takes as input a test script, like find this button, click it, go to this other page, check a value at the bottom, etc.

---

hackmd.io/@gabalafou/waaat

Test: is focus visible?

test("should have visible focus indicator", async (
  { page }
) => {
  while (true) {
    const node = await page.evaluate(() =>/* browser */
                                document.activeElement);
    const focus = await node.screenshot();
    await page.keyboard.press("Tab");
    const noFocus = await node.screenshot();
    expect(
      focus.equals(noFocus)
    ).toBe(false);
  } // not shown: breaking out of the loop
});

Note:

Speaker: And here is an example of test script. Don't worry: I'm going to explain this code. This is basically the code version of the manual test script that I described earlier. Let's break it down. On the first line, we call a function called test. We pass it the name of the test as the first argument: "should have visible focus indicator". And we pass the test function as the second argument. On line 2, Playwright passes a variable called "page" to our test function, which allows us to manipulate and interact with the web page that Playwright has opened for our test. (Remember Playwright is a go-between.) The body of the test function is a loop that, well, loops over all of the tab-focusable elements on the page. Let's take a look at what that does. On lines 5 and 6 we call a function page.evaluate. This is a key functionality in Playwright and one that tripped me up a little bit at the beginning. We pass a function to page.evaluate. This function return document.activeElement, which is basically whatever element in the browser currently has focus. But here's the thing, what you need to understand is that Playwright takes this function that you pass to page.evaluate, and it actually runs the function in the browser's execution environment, waits for the return value, and then relays that return value from browser back into your test code. So that's how lines 5 and 6 get the currently focussed element and we store a reference to that browser element in a variable called node. Next we take a screenshot of the currently focussed element by calling node.screenshot(), which is another useful functionality that Playwright provides. It allows you to actually take screenshots of the page, and yes these are just normal screenshots like you would take. On line 8, we ask Playwright to press the tab key by passing the word "Tab" to the page.keyboard.press() method. And then on the next line we take a screenshot of the node while it doesn't have focus. And in the three lines 10-12, we compare the screenshots. And this is just a simple pixel-by-pixel comparison. If all of the pixels are equal, then we know the test has failed, because if there had been a visible focus indicator, then the two screenshots should be different because one was taken while the element had focus and one was taken while the element did not have focus.

---

hackmd.io/@gabalafou/waaat

Fail 1: node.screenshot()

Note:

Speaker: And of course, that didn't work. The problem, as I discovered after some debugging was that when you call node.screenshot(), Playwright takes a screenshot of what's called the bounding box, and unfortunately, the bounding box does not include the focus ring, which is what this image shows. This is a screenshot from a section of a Playwright test report onto which I attached the node screenshots. Sorry this can be maybe a little confusing, because I'm showing you a screenshot I took of screenshots that Playwright took. And what this image shows is that both screenshots are identical, but in reality when I tested the app manually, I could see that there was a visible focus indicator. So I was a little disappointed. I thought I had found a clever little way to write a test for the focus visible issue, but it didn't work. By the way, I should just point out that the screenshots that Playwright took are not actually blurry, as they appear in this image. It's just that these are screenshots of two small elements on the page, so the PNG files are actually quite small themselves, but then they get blown up for the report page, and also for these slides.

---

hackmd.io/@gabalafou/waaat

Fix: code

const { x, y, width, height } = await node.boundingBox();
await page.screenshot({
  clip: {
    x: x - 4,
    y: y - 4,
    width: width + 8,
    height: height + 8
  } 
});

Note:

Speaker: But do we give up? No we do not. We read the Playwright API docs and we discover—aha!—Playwright provides functionality for you to take a screenshot of a section of the page. Let's take a quick look at this code. On the first line, we call node.boundingBox(), and this gets us the x, y coordinates of the top-left pixel of the node, as well as its width and height. Next instead of calling node.screenshot() we call page.screenshot() and we pass an option called "clip". And this is basically what it sounds like. It tells Playwright to take a screenshot of the whole page, but then clip away all but this one area. So it's like we add a little bit of cushion to what Playwright was screenshotting, we expand that area just a bit, so that we can capture the focus ring in the screenshot. Alright, so, drum roll please…

---

hackmd.io/@gabalafou/waaat

Fix: screenshot

Note:

Speaker: Yes… yes! On this slide the two screenshots are different, the focussed state did indeed have a focus indicator and the unfocussed state did not. Okay so now does our test work?

---

hackmd.io/@gabalafou/waaat

More fail?

Note:

Speaker: Well, if computers didn't suck and the world weren't a giant a dumpster fire, maybe. So of course, I did find some more issues, but this was actually good, and really good reason to write tests. I discovered a fairly subtle focus issue with the menubar in Jupyter and I also discovered that the skip link was broken. So the failing test was actually working, as in, it was detecting more focus visible bugs, which is great. I've also read that there can be issues with comparing screenshots in this way, so I might have to tweak that a bit down the road as well.

---

hackmd.io/@gabalafou/waaat

Docs

PRs
Issues
Accessibility Guidelines
Limitations

Note:

Speaker: Okay, so we've written the test. We opened one pull request to fix the menubar focus issue and another to fix the skip link. Now what? It's documentation time. So, why documentation? I can think of at least two good reasons. Have any of you ever been worked in a library, you get an error, and then the error message has a link to a page that goes into a lot more detail about the error? Has that ever saved you a boat load of time? I know it has for me. And so I wanted to make sure that if this accessibility test fails, then whoever encounters the failure will have some clues to help pinpoint the problem. The other reason I can think of doing this kind of documentation is that it can also help others make accessibility assessments; it could be an input into an accessibility statement. Another reason: accessibility, as a knowledge domain, is currently a bit niche in the world of programming, a bit like security. As a field we want to increase accessibility literacy in the community, just as folks in security want to increase security literacy. But we're not there yet, and so I think that it can be helpful to other developers to have documentation about the accessibility test. So with those goals in mind, what do we want to put on our doc. Well, I can think of a few things right away. [NEXT SLIDE] If you were keeping count, it took me 3 separate PRs to fix the focus visible issue. I think the test doc should definitely link to those PRs. [NEXT SLIDE] I also think the test doc should link to all of the related, relevant GitHub issues. [NEXT SLIDE] It should also link to any accessibility guidelines that explain why this is truly an accessibility bug. [NEXT SLIDE] And I think there should also be a section that explains the limitations of the test. Can it give you a false negative? That kind of stuff.

---

hackmd.io/@gabalafou/waaat

Success! (almost)

Note:

Speaker: That's why on this screen you see a draft PR for my test because I still need to write those docs!

Links:

• Focus visible test PR

---

Note:

Speaker: We have another floof! That means we're getting close to the end. This is an image of Hannah Soyer and her cat, um, whose name I don't know. Hannah Soyer is a queer, disabled writer interested in perceptions and representations of ‘othered’ bodies. She is the founder of This Body is Worthy, a project designed to celebrate bodies outside of societal ideals. And I once again, got this photo from the Disability Visibility Project, so check them out.

Links:

• Photo is from the Disability Visibility project
• Used with permission via Creative Commons Attribution-NonCommercial 4.0 International License.

---

hackmd.io/@gabalafou/waaat

Wrap up

Note:

Speaker: Alright let's wrap up.

---

hackmd.io/@gabalafou/waaat

Scope and orientation

Note:

Speaker: Okay so now that I've told you my story, I want to situate it within a larger context. I'm hoping that this will also unpack the title of my talk, which, uh, just in case you forgot is: waaat! JupyterLab as a Lab for Web App Automated Accessibility Testing. If we zoom way, we start with accessibility. Accessibility is a big space. It encompasses both the built and tangible world, as well as the intangible, digital world.

---

hackmd.io/@gabalafou/waaat

Note:

Speaker: Next, the digital space of accessibility can be further subdivided into things like phone apps, desktop apps, and the web.

---

hackmd.io/@gabalafou/waaat

Note:

Speaker: We can then think of dividing the space of web accessibility into static web sites meant primarily for consumption versus web apps, such as JupyterLab which are highly interactive.

---

hackmd.io/@gabalafou/waaat

Note:

Speaker: Because of their greater complexity, web apps have different needs than static websites when it comes to accessibility tooling, which I hope the focus visibility story made clear. In order to write a test, we needed to use Playwright; we couldn't rely on Axe-core or other checkers. So while web apps and static websites have similar accessibility needs, they are also different enough that I treat them separately. Taking that as a given, we can talk about web app accessibility as a distinct thing. And beneath web app accessibility, we have things like testing, ARIA, the accessibility of underlying frameworks, and so forth.

---

hackmd.io/@gabalafou/waaat

Note:

Speaker: Finally, we divide the space of web app accessibility testing: automated versus manual. And now you understand the title of my talk: web app, automated, accessibility testing. Or automated accessibility testing for web apps.

---

hackmd.io/@gabalafou/waaat

Note:

Speaker: My point here is this is a niche of a niche a niche, which gets to another keyword in my talk title: "lab." Because this stuff is so niche, I often don't know what the tried and true solution is. Hence the lab analogy: we're really learning by doing, trying things out, seeing what works, what doesn't. It's all a work in progress.

Links:

• Image source and license info

---

hackmd.io/@gabalafou/waaat

Playwright as compromise

Note:

Speaker: But here's what I've learned so far. Accessibility checkers like Axe-core can be great as a first-pass. It's low cost but also low coverage. The best of all worlds is to have disabled people using and testing your app, but hiring human testers is more costly. Playwright sits somewhere between these two. It's more costly than Axe-core but also has more coverage, but is definitely not a substitute for disabled testers.

---

hackmd.io/@gabalafou/waaat

Zoom Out, Shift Left

Note:

Speaker: A lot of this talk has been about how to write regression tests so that when you fix an accessibility issue it doesn't break again. This however is symptomatic of a broken model. Again, let's turn to the analogy with security. A lot of software gets developed without too much thought put into security. Then security bugs are found. Then everybody scrambles to fix the security bug. Repeat, repeat, repeat. Similarly, a lot of software gets developed without too much thought put into accessibility. Then an accessibility audit is done, maybe leading to a lawsuit. Then fixes. In this talk, I focussed on adding accessibility to testing. But you can think about adding accessibility to earlier phases. Think about linting, think about documentation for contributors, think about education and workshops. If you can pay disabled people to test your app before major releases, that's great, but even better if you can get disabled people in the early planning and design stages doing UX research. And even better if you can hire disabled designers and coders to co-create and co-design. There's a gradient from access to participation, and I think all of us in open source don't want to put up barriers to participation, but we do it all the time.

Links:

• Shift left image source and credit

---

hackmd.io/@gabalafou/waaat

Goals and the Future

Note:

Speaker: Removing barriers to access is the long term mission, but for this testing work, we also have more near-term goals. The main thing that we're hoping to get out of this is a proof of concept that other web application in open source can use. That's why we have such a heavy emphasis on documentation. Eventually, once we've proved out this concept a bit more, we would like to integrate these tests into the default JupyterLab testing infrastructure and continuous integration.

Links:

• "Yet another type of Escher cookie cutter" by fdecomite is licensed under CC BY 2.0.

---

hackmd.io/@gabalafou/waaat

Acknowledgements

Note:

Speaker: In my last slide, I used the word "we" a few times and while I don't actually speak for the people I work with or anyone else, these are some of the fantastic people I had in mind: Tania, Isabela, Stephannie, and Tony. These are their avatars on GitHub. I literally wouldn't be here doing this work right now if it weren't for Tony, Isabela, and Tania because they did the work to write a grant proposal for the Chan Zuckberg Iniative, which allowed them to bring me on. There are so many trailblazers who came before me who have made my day-to-day possible, and I want to acknowledge that. You all also have Tony to thank for the floof pics. I once heard him tell another colleague that their slides needed more dog pics, and well, I took that advice to heart.

---

hackmd.io/@gabalafou/waaat

Let's go!

@gabalafou
@[e-mail]

Note:

Speaker: Speaking of which… This is Tania's floof Maisey, looking like the Belgian Malinois runway model that she is. Thank you all for coming. You made it to the end of JupyterCon, or at least to the end of the scheduled speakers.

Links:

• Our GitHub repo for this testing work