Your Site Passed the Audit. It’s Still Broken.
How I used a vision-language model to catch accessibility failures that code can’t see.
I’m a product manager for a transit agency in Washington State. My customers are people with disabilities. They depend on being able to navigate websites to live their lives like everyone else: shop, learn, explore. And to plan their transit trips. Real people, real stakes.
When WCAG compliance started carrying more legal weight, I noticed a problem. Teams were racing to pass automated audits and collect green checkmarks. But passing a scan and actually being usable are two different things. Tools that give false confidence make things worse, not better, for the people who need them most.
Most accessibility tools only read the code. They can’t see the page the way a human does. A focus ring that’s technically present but invisible against the background passes the audit and fails a keyboard user with low vision. That gap bothered me enough to build something different. It’s called PointCheck.
The Problem With DOM Inspection
Tools like Axe and Lighthouse check the DOM for focus styles, looking for outline rules, making sure it isn’t set to none. What they can’t do is render the page as pixels and answer a simple question: can a person actually see the focus indicator right now?
A white ring on a light background. A blue ring on a blue navbar. A focus style hidden behind a sticky header. These all pass code checks. They all fail in real life.
This is the class of failure PointCheck was built to catch. Here’s how the coverage compares to the two most widely used tools:
How PointCheck Works
PointCheck adds a second layer that code-only tools miss. It uses Playwright to load the page and tab through interactive elements, taking a screenshot at each focused state. Those screenshots go to Allen AI’s Molmo2, a vision-language model with an unusual capability: it can point.
Given a screenshot and a description, Molmo2 outputs pixel coordinates of what you’re looking for. Not a bounding box, not a classification — an actual <point x="42.3" y="67.1"> in the rendered image.
The question it asks is simple: “Find the element that currently has keyboard focus.” If Molmo2 can clearly locate it, the focus indicator is visible enough to pass. If it can’t, the element fails, regardless of what the CSS says.
It’s a two-gate system: traditional DOM inspection plus visual confirmation on the rendered page. An element has to pass both, and PointCheck has both test gates.
This approach catches things like:
Focus rings that exist in CSS but disappear due to a background compositing problem. When outline color is correctly specified but the element sits on an inherited transparent background, CSS inspection passes it clean. The actual rendered background, composited up through the DOM tree, tells a different story.
Focus rings that fail at the pixel level because the ring color has insufficient contrast against the element’s actual background in context. Code inspection would pass both. A keyboard user would be lost.
Beyond focus visibility, PointCheck also runs a static JS pre-scan before tab traversal to catch keyboard failures that DOM inspection misses entirely: javascript: href links, onclick handlers on non-interactive elements, and positive tabindex values that silently override natural tab order. Any tabindex > 0 is a WCAG 2.4.3 violation. The scan flags the element, its value, and its label so it’s immediately actionable.
PointCheck runs six tests covering keyboard-only navigation (WCAG 2.1.1, 2.1.2, 2.4.3), 200% zoom and reflow (1.4.4, 1.4.10), color blindness simulation (1.4.1, 1.4.3), focus visibility (2.4.7), form error handling (3.3.1–3.3.3), and page structure and semantics (1.1.1, 1.3.1, 2.4.2, 2.4.4, 2.5.5, 3.1.1, 4.1.2).
Together these cover approximately 85-90% of WCAG 2.1 Level AA success criteria programmatically. The criteria that remain out of scope (captions on live video, cognitive load, certain language-of-parts judgments) genuinely require human review.
What the Tests Show
I ran PointCheck against five sites, ranging from intentionally broken demos to a purpose-built benchmark with 142 documented failures.
The GDS accessibility tool audit page was built by the UK Government Digital Service specifically to contain every common accessibility failure. Lighthouse scored it 56/100 and Axe found 22 violations. PointCheck flagged it in all six of its test dimensions.
What Lighthouse and Axe caught were the static markup failures: missing alt text, table headers without scope, unlabeled form fields. What they couldn’t do was run the page. PointCheck did: text clipping at 200% zoom, contrast failures under color blindness simulation, and a focus indicator that Molmo2 couldn’t locate in the rendered screenshot despite it passing CSS inspection.
On the Mars Commuter site, Lighthouse flagged 10 violations and Axe found 8. Neither caught the 10 positive tabindex values that override natural tab order. That’s a direct WCAG 2.4.3 failure that affects every keyboard user on the page. PointCheck nailed it.
The adversarial test matters as much as the failure cases. I also ran PointCheck against Tenon UI, a React component library built to be genuinely accessible. It passed cleanly. A tool that cries wolf on good code is worse than no tool at all.
The AI Future of Accessibility Testing
WCAG 2.2 introduced new criteria that are genuinely hard to evaluate with code alone, particularly 2.4.11 (Focus Appearance), which specifies minimum area and contrast requirements for the focus indicator itself. Not just “does a focus ring exist” but “is it big enough and contrasty enough to meet a specific perceptual threshold.”
That’s a job for a vision-language model, not a CSS inspector. Molmo2’s point-coordinate output is the right tool for it: locate the indicator in the screenshot, reason about its visual properties, compare to the spec. It’s the natural next extension.
Try It
For people with disabilities, the difference between “technically compliant” and “actually usable” isn’t academic. It’s the difference between being able to participate and being shut out. PointCheck is my attempt to shrink that gap.
PointCheck is live at pointcheck.org. Paste any public URL and run the full suite. It’s free to try. The backend runs on Modal’s serverless infrastructure, so there may be a brief cold start on the first run.
Full source code is on GitHub at github.com/BrendanWorks/wcag-molmoweb-tester.
Built with Allen AI OLMo2, Allen AI Molmo2, Playwright, FastAPI, Modal, Next.js, and Vercel.