Scope of Automated Testing

About Octopus Scanner

What an automated accessibility scan can establish, what it can't and how to use Octopus Scanner reports as part of a complete audit.

For the practical checks to carry out after a scan, see After Your Scan.

1. The principle

Automated testing plays a specific role in an audit. Understanding it keeps a report from being mistaken for something it isn't.

  • Automation finds what is technically detectable: Octopus Scanner uses the axe-core engine to test rendered pages against rules that code can evaluate, such as whether an image has a text alternative, whether a form field has a label, or whether text meets a contrast ratio.
  • Human review finds what is meaningful: Many WCAG success criteria depend on judgement, context or how a page behaves in use. No automated tool can confirm these.
  • A report supports an audit. It does not replace one: A report with no violations means the rules that ran found no issues, not that the page is accessible or compliant. The W3C puts it directly: evaluation tools "can not determine accessibility, they can only assist in doing so."
  • How much automation finds: Deque, the developers of axe-core, state that axe-core finds on average 57% of WCAG issues automatically.
    • The figure comes from a 2021 study of over 2,000 first-time audits covering more than 13,000 pages, and measures the volume of issues found, not the number of WCAG success criteria covered.
    • Many success criteria can't be tested automatically at all, and the remaining issues can only be found by manual testing.

2. What an automated scan can establish

Automated rules reliably detect these kinds of issue.

  • Missing or empty attributes: Images without alternative text, form fields without labels, buttons and links without an accessible name and frames without a title.
  • Invalid structure and code: Incorrect ARIA roles, states and properties, improperly nested lists and tables with missing header associations.
  • Measurable visual properties: Text colour contrast against a solid background, and zoom or text spacing blocked by viewport settings or inline styles.
  • Document-level requirements: A missing page title, a missing or invalid page language and viewport settings that prevent zooming.

3. Which standards can be selected

A scan runs against the rule sets you select under Axe Scan Rules. Selecting a set has a specific, limited meaning.

  • The available sets: WCAG 2.0, 2.1 and 2.2 at Level A, Level AA and Level AAA, Best Practice, Section 508, EN 301 549, Trusted Tester and RGAA 4. Experimental rules, and rules that always need human confirmation, can also be included.
  • What selecting a standard does: It runs the automated rules that the engine maps to that standard. It doesn't test the standard in full, and no automated tool can.
  • Frameworks that are mostly manual: Trusted Tester and RGAA describe test procedures carried out by a person. Only the parts that code can evaluate are covered, so a clean scan against either is a starting point for that process, not a substitute for it.
  • Overlapping rules: The sets overlap heavily, because most derive from WCAG. Selecting several doesn't multiply coverage; it widens the net slightly at the edges.
Axe Tags under Axe Scan Rules, with the default selection.
Text description

Example of the Axe Scan Rules section showing Axe Tags: WCAG 2 A, WCAG 2 AA, WCAG 2.1 A, WCAG 2.1 AA, WCAG 2.2 A, WCAG 2.2 AA and Best Practice ticked; WCAG 2 AAA, WCAG 2.1 AAA, WCAG 2.2 AAA, Section 508, EN 301 549, Trusted Tester, RGAA 4, Review Items and Experimental unticked.

4. What an automated scan can't establish

These areas always need manual testing, whatever a report shows. After Your Scan turns them into a checklist.

  • Whether content is meaningful: A scan can confirm that alternative text exists, not that it describes the image accurately. The same applies to link text, headings, labels and page titles.
  • Keyboard operation: Whether every function can be used by keyboard, whether the focus order is logical, whether focus is visible at every step, and whether focus is trapped or lost in menus, dialogs and custom widgets.
  • Screen reader experience: Whether content is announced in a sensible order, whether dynamic updates are announced, and whether custom components behave as their roles describe.
  • Media: Whether captions, transcripts and audio descriptions exist and are accurate.
  • Reading order and meaning: Whether the order content is read in matches its visual order, and whether information conveyed by colour, shape, position or sound is also available in another form.
  • Behaviour over time: Time limits, moving or flashing content, unexpected changes of context and whether errors are explained in a way users can act on.
  • Clarity and consistency: Whether language is clear, and whether navigation and identification are consistent across pages.
  • Anything the scan did not reach: Pages outside the crawl, content behind sign-ins or interactions that weren't recorded, and page states that were never displayed during the scan.

5. How Octopus Scanner extends coverage

Several features reach further than a single scan of a page's initial state. They increase what automation can detect, but they don't remove the need for manual review. Each is described in full in the User Guide.

  • Needs Review: Where axe-core can't be certain of a result, it returns it as "incomplete" rather than guessing.
    • With Include Needs Review Section turned on (off by default), Octopus Scanner keeps these findings, such as contrast over an image or gradient, in a separate Needs Review section for a person to assess, rather than discarding them or reporting them as failures.
    • An element is dropped from Needs Review if the same rule already failed on that element.
  • Page states: When Force-Reveal Hidden Content expands something, the page is audited in both its expanded and collapsed states, and Scan Dark Mode runs a second audit when the page responds to the system dark mode preference. A dark theme switched on only by the site's own toggle isn't tested. These options are unavailable in Interactive Mode.
  • Screen sizes: Multi-viewport scanning tests the same page at desktop and mobile sizes by default, or at two desktop sizes, tablet and mobile with Scan All Viewports.
  • Signed-in areas and journeys: Pages behind a sign-in, and recorded user journeys, can be scanned in the same session.
  • PDF checks: Linked PDFs are checked against core accessibility requirements, including tags, a text layer, a document title and language, alternative text for images, headings, table headers, form field names, links, tab order and bookmarks. These checks read the file's structure and properties only. They don't confirm that its tags, reading order, alternative text or language are correct, or check colour contrast.
  • Triage: In the report, you can approve or remove Needs Review findings, move violations to another severity, and, when the report is opened from Past Reports, hide them, so the version you hand over reflects your own judgement. Report Anatomy explains each control.

6. Using reports responsibly

  • Treat a report as a starting point: Use it to find and fix detectable issues early, then carry out manual testing with a keyboard and assistive technology.
  • Do not present a report as proof of compliance: Conformance with WCAG, the Equality Act 2010, the Public Sector Bodies Accessibility Regulations, the European Accessibility Act, Section 508 or any other standard or law can't be established by automated testing alone. Accessibility Law: EU and UK summarises what the main laws ask for.
  • Review before sharing: axe-core is designed to return zero false positives, bugs notwithstanding, so treat a reported violation as real, and confirm by hand any Needs Review item, whose ticket shows a Review Priority. A finding can still be out of scope for your audit, or already addressed in another way. Check findings before sending a report on, and use the report's triage controls to record your decisions.
  • Involve disabled people: Testing with people who use assistive technology every day finds issues that neither automated tools nor expert review will.

7. Further reading

Send questions about this page to support@octopusscanner.com, or use Report Bug in the scanning interface.

Was this page helpful? Email your feedback.