Local LLM Review

Using Octopus Scanner

How to set up the optional Local LLM Review with a model you run on your own computer or local network, and how to read what it suggests.

For advanced users. Local LLM Review is meant for people comfortable running and configuring their own LLM server, including its address, API key and HTTPS certificate.

Optional and off by default. Octopus Scanner uses no AI unless you turn on Local LLM Review, and your scan data stays on your computer.

Local LLM Review works only with a model you run on your computer or your local network, and cloud AI providers aren't accepted.

LLMs can be wrong or inconsistent, and results depend on the model you choose, so treat everything it says as a suggestion, not a confirmed issue. You choose and run the model, so Octopus Scanner isn't responsible for the model, its output or any charges you receive if you connect a paid or metered service.

1. What Local LLM Review does

When you turn it on, Octopus Scanner sends details of the scanned pages to a large language model (LLM) on a server you choose, after crawling and scanning have finished and before the report is built. It can do two things, and you can turn on either or both.

  • Check axe-core Findings: The model gives a second opinion on a few instances of each issue type axe-core found. Its verdict appears in the ticket for that issue, next to the existing confidence. axe-core's findings and counts aren't changed.
  • Find Issues axe-core Can't Detect: The model looks at each reviewed page for problems that rules can't judge, such as alt text that doesn't describe the image or link text that makes no sense out of context. You choose what it looks for under Categories. Its suggestions go into a separate LLM Review section of the report, where you approve or remove each one.

If the review can't finish, for example because the server stops responding, the scan still produces its report, with any LLM results gathered before the review stopped.

If you select Finish Early & Scan while pages are still being discovered, the review still runs on the pages already scanned. Finish Early & Scan isn't available once the review has started. Cancel Scan stops the scan and the review without a report.

2. Checking its results

Local LLM Review's results depend heavily on the model and server you use, so check them before you rely on them:

  • Answers can be wrong: A model can call a real issue a false alarm, or suggest a problem that isn't there.
  • Answers can vary: The same content can get a different answer on another run or with another model. Consistent Answers reduces this but can't remove it.
  • It may change: How the review works, and what it shows in reports, may change as the feature develops.

Every LLM finding is a suggestion for you to check, in the same way as a Needs Review item. Nothing the model says is counted as a confirmed issue unless you approve it.

3. What you need

  • An LLM server you run yourself: Ollama, LM Studio or another server with an OpenAI-compatible API, running on your computer or on another computer on your local network.
  • A model you choose: Octopus Scanner doesn't include, download or recommend a model. You install one on your server and pick it in the scanning interface. Larger models usually give better answers but take longer.
  • A vision model for screenshots: Include Screenshots and the Visual categories, all on by default, need a model that accepts images. With other models, screenshots are left out automatically and the review checks text and HTML only.
  • Enough memory: Useful models need roughly 16 GB of RAM or a capable graphics card (GPU) on the computer that runs the model. Running the model on the same computer as a large scan uses that memory at the same time.

4. Setting up your server

Each server speaks the same OpenAI-compatible API, so the steps in Octopus Scanner are the same whichever you choose. Start the server and load or download a model before you select Test Connection.

Ollama

  • Base URL: http://localhost:11434/v1, filled in when you choose Ollama as the server.
  • Starting it: Open the Ollama app, or run ollama serve, and download a model with ollama pull followed by the model's name.
  • From another computer: Ollama only accepts connections from its own computer by default. To reach it from the computer running Octopus Scanner, set the OLLAMA_HOST environment variable to 0.0.0.0 on the computer running Ollama, restart Ollama and use that computer's local address in the Base URL, such as http://192.168.1.20:11434/v1.

LM Studio

  • Base URL: http://localhost:1234/v1, filled in when you choose LM Studio as the server.
  • Starting it: Download a model in LM Studio, then start its local server from the Developer section.
  • From another computer: Turn on Serve on Local Network in LM Studio's server settings, and use that computer's local address in the Base URL, such as http://192.168.1.20:1234/v1.

Another OpenAI-compatible server

  • Base URL: Choose Another OpenAI-Compatible Server and enter the server's address in Base URL, ending in /v1, such as http://localhost:8080/v1. The field starts blank for this option, so if Show All Settings is off, Base URL is shown on its own.
  • Starting it: Follow your server's own instructions, and make sure it listens on an address the computer running Octopus Scanner can reach.

Finding your server

If you don't know your server's address, Octopus Scanner can look for it. Both buttons are under Find a Server, below Server, and neither runs unless you select it.

  • Find Servers: Looks for LLM servers on this computer, on the ports Ollama, LM Studio and other local servers commonly use.
  • Search Local Network: Looks for LLM servers on other devices on your local network. It asks first, in a "Search your local network?" message: select Search Network to go ahead, or Cancel. Some workplace networks treat this kind of search as network scanning, so check it's allowed before you continue. The search takes up to about 20 seconds.

Each server found is listed with where it is (on this computer or on your local network), its Base URL and the number of models it has. Confirmed servers are listed first.

  • Needs an API key: Shown instead of the model count for a server that needs a key.
  • HTTPS service: An address that uses HTTPS with a certificate Octopus Scanner can't check, marked Not confirmed as an LLM server, because it might be something else, such as your router. If it's your server, add its certificate under Trusted Certificate (Optional), then search again.
  • Use: Fills in Server, Base URL and the Model list, even while Base URL is hidden.
    • A message under the results confirms what was filled in, and that server is marked In Use.
    • If the server needs a key or a certificate, only API Key (Optional) or Trusted Certificate (Optional) is shown, and focus moves to it. Add it, then select Test Connection.
    • Base URL is also shown when it isn't the server's usual address.

The search contacts only this computer, or the devices on your computer's own private local network, on a few common LLM ports. It never contacts internet addresses, it only asks each server for its list of models, and your API key is never sent while searching. If you have added a Trusted Certificate, it is used to check HTTPS servers. If you add or change it after a search, a message asks you to search again.

Connecting Octopus Scanner

  1. Select Customise Scan, then open Local LLM Review under Local AI (Optional).
  2. Turn on Enable Local LLM Review.
  3. Select Find Servers or Search Local Network, then Use next to your server, as described under Finding your server. Or choose your Server yourself, which fills in its usual Base URL. To check or change the Base URL, turn on Show All Settings.
  4. If your server needs a key, enter it in API Key (Optional). It is shown when a server needs it, or you can turn on Show All Settings.
  5. If your server uses HTTPS with its own certificate, paste the certificate into Trusted Certificate (Optional), or select Load File… to choose it. It is shown when a server needs it, or you can turn on Show All Settings.
  6. Select Test Connection. If it succeeds, the Model list fills with the models on your server. Until then, it shows Select Test Connection to list models.
  7. Choose your model from the list. If it isn't listed, choose Other model and enter its name in Model Name.
Local LLM Review after a successful Test Connection to Ollama on the same computer. The model name is an example. Simplified.
Text description

Example of the Local LLM Review section. Enable Local LLM Review is ticked. Server is set to Ollama, with the hint Fills in the server's usual Base URL. Under Find a Server are the Find Servers and Search Local Network buttons. Below the Test Connection button, a success message reads: Connected to the server on your computer. Found 2 models. Choose one under Model. Remember: results are experimental and depend on your model. Model is set to llama3.2-vision.

Only servers on your computer or your local network are allowed. Octopus Scanner refuses internet addresses, and a server name must point only to local or private addresses. If the server is on another computer on your network, a note appears after Test Connection: "This server is on your local network, not your computer. Page details from your scans will be sent to it." Make sure you trust that computer and the people who manage it.

If Test Connection fails, the message says why:

  • Only local models on your computer or your local network are supported: The Base URL points to an internet address. Use a server on your computer or your network.
  • Could not connect: The server isn't running, the address or port is wrong or the server isn't set up to accept connections from other computers. On a Mac, also check that Octopus Scanner is turned on in System Settings > Privacy & Security > Local Network. Without it, macOS blocks connections to other devices on your network, and a network search finds only your router.
  • The server rejected the key: Check API Key (Optional).
  • Certificate not trusted: add the server's certificate: Add the server's certificate in Trusted Certificate (Optional).
  • Certificate not trusted: the certificate under Trusted Certificate isn't the one this server uses: The Base URL is probably the wrong address. Check it's your LLM server's address, or add this server's certificate.

If the problem is with the key, the address or the certificate, or the Base URL isn't valid, that setting is shown if it was hidden, with an announcement, and focus moves to it.

The setup walkthrough also asks "Do you have a local LLM?". Answering Yes, I'll set it up turns on Enable Local LLM Review and opens these settings when you finish, so you can enter your server and test the connection. See The setup walkthrough.

5. The settings

Found under Customise Scan > Local AI (Optional) > Local LLM Review. The other settings appear once Enable Local LLM Review is on. Base URL, API Key (Optional), Trusted Certificate (Optional), What To Review, Include Screenshots, Consistent Answers, the limits and Custom Instructions (Optional) need Show All Settings; until you change them, they use the defaults below. Categories are always shown.

A hidden setting is shown on its own, with an announcement, when it needs attention or has a value. Show All Settings stays as you set it. This happens when:

  • you choose Another OpenAI-Compatible Server with no Base URL
  • you select Use on a server that needs a key or a certificate, or whose Base URL isn't the server's usual address
  • Test Connection fails because of the key, address or certificate
  • the Base URL isn't valid
  • you load a configuration with its own Base URL or a certificate
  • a key is saved for the server in Base URL.
  • Enable Local LLM Review: Turns the review on for your scans. Off by default.
  • Server: Ollama, LM Studio or Another OpenAI-Compatible Server. Choosing one fills in its usual Base URL.
  • Find Servers and Search Local Network: Under Find a Server. Look for LLM servers on this computer, or on other devices on your local network, and list them with a Use button. See Finding your server.
  • Base URL: The address of your server's OpenAI-compatible API, starting with http:// or https://. Only addresses on your computer or your local network are allowed. Server and Use fill it in, even while it is hidden.
  • API Key (Optional): A key, if your server needs one. The empty field shows an example, sk-local-1234. When you enter a key, Octopus Scanner saves it for that server and clears the field, which then reads Saved. Leave blank to keep using it. The key is used only with the server whose address it was saved for. Forget Saved Key removes it.
  • Trusted Certificate (Optional): A PEM certificate to trust, only needed if your server uses its own HTTPS certificate. It is added to the certificates your computer already trusts and doesn't replace them.
  • Test Connection: Checks that the server can be reached and lists its models.
  • Model: The model to use. Until models are listed, it shows Select Test Connection to list models and Other model. Test Connection, or Use after Find Servers, lists the models on your server under Choose a model. Choose Other model to type a name in Model Name instead. A model is required before a scan can start.
  • What To Review: Check axe-core Findings adds a second opinion to each issue type in the report. Find Issues axe-core Can't Detect adds suggestions to the LLM Review section. Both are on by default, and at least one must be on.
  • Include Screenshots: Needs a vision model. Sends a screenshot of each reviewed page, and of each issue where one exists. Requests take longer. On by default. If your model doesn't accept images, the review leaves them out and carries on without them. A page too large for the model to take in is left out of the review on its own, without affecting the rest.
  • Consistent Answers: Asks the model for the same answer each time it sees the same content. On by default. Some servers or models may not follow it fully.
  • Categories: What to look for when finding issues axe-core can't detect, in three groups, all on by default: Page Content, Visual and Interaction.
    • Visual is available when Include Screenshots is on, and Interaction makes scans take longer.
    • Categories are shown whenever Local LLM Review is on, but are unavailable while Find Issues axe-core Can't Detect is off. When it is on, at least one must be on.
    • See Categories.
  • Max Pages To Review: How many scanned pages the model looks at for new issues. Default 20, from 1 to 200. The model can suggest up to 25 issues on each page.
  • Findings Checked Per Issue Type: How many instances of each axe-core issue type the model checks. Default 3, from 1 to 10.
  • Timeout Per Request: How long to wait for each answer, in seconds.
    • Default 90, from 10 to 600.
    • The first request is allowed twice as long, because servers such as Ollama load the model when it's first used.
    • A finding or page whose request times out is skipped, and the review moves on. If 2 requests in a row time out, the review stops and the report keeps everything checked so far.
    • A request that meets a busy server (HTTP 500, 502, 503 or 504) is tried up to twice more.
  • Minimum Confidence: Suggestions the model is less sure of than this percentage are left out. Default 50%.
  • Custom Instructions (Optional): Your audit preferences, added to every request, up to 2,000 characters, for example "Our audits follow WCAG 2.2 AA." They can't override the scanner's safety rules.

Local LLM Review settings are saved with your configurations. When Local LLM Review is on, the Before you scan message reminds you that page details will be sent to your LLM server.

6. Categories

Found under Customise Scan > Local AI (Optional) > Local LLM Review > Categories once Enable Local LLM Review is on, and used when Find Issues axe-core Can't Detect is on. Each category looks for one kind of issue, and the WCAG success criteria it relates to are shown in brackets. Each suggestion in the report names its own criterion.

Page Content

Checked from each page's text, structure and links, and compared across pages. All are on by default.

  • Alt Text Quality (1.1.1): Alt text that is present but unhelpful, inaccurate, redundant or just a file name.
  • Link & Button Text (2.4.4): Link and button names that don't make their purpose clear, alone or with the text around them.
  • Headings (1.3.1): A heading outline that skips or misuses levels, such as a level 4 heading straight after a level 1.
  • Descriptive Headings & Labels (2.4.6): Headings that don't describe the content that follows them, and field labels that don't say what to enter, such as "Section 1" or "Field 1".
  • Forms (3.3.2): Missing instructions or cues, such as a required field with no visible required marker, or a field that needs a set format with no hint.
  • Label in Name (2.5.3): Controls whose accessible name doesn't contain the words shown on screen, so people using speech input can't say what they see.
  • Page Title (2.4.2): A page title that is missing, generic or doesn't describe the page.
  • Language (3.1.1, 3.1.2): A page language setting that is missing, invalid or doesn't match the page's text, and passages in another language that aren't marked as such.
  • Sensory Instructions (1.3.3): Instructions that rely only on shape, size, position or sound, such as "press the round button on the right".
  • Reading Order (1.3.2, 2.4.3): Content whose order in the code differs from its order on screen in a way that changes the meaning, such as numbered steps read out of order. Only checked on pages where the two orders differ.
  • Consistency (3.2.3, 3.2.4): A navigation menu whose links are in a different order on different pages, or a repeated link with the same destination but a different name. Compared across the reviewed pages.
  • Captions & Transcripts (1.2.1 to 1.2.5): Video and audio that likely lack captions, a transcript or audio description. Only checked on pages with video, audio or an embedded player. The model can't play media, so it judges from the page, such as caption tracks and nearby mentions of a transcript.

Visual

Checked from screenshots. Available when Include Screenshots is on, and all on by default. They need a vision model.

  • Text Over Images (1.4.3): Text over an image, gradient or pattern that is hard to read.
  • Use of Colour (1.4.1): Colour used as the only way to show information, status, required fields or links.
  • Images of Text (1.4.5): Images that are mainly words that could be real text, such as a banner or heading made from an image. Logos, photos and charts are fine.
  • Non-Text Contrast (1.4.11): Field borders, and icons that are the only way to identify a control, with contrast below 3:1. Octopus Scanner measures the contrast and sends the ratio with the screenshot.

Interaction

These check how the page responds, so they do a little more on each reviewed page and scans take longer with them on. Both are on by default. Turn them off for a quicker review.

  • Focus Visible (2.4.7): Keyboard focus stops with no visible focus indicator, or one too faint to notice.
    • Octopus Scanner presses Tab through the page, up to 30 stops, and records how each one's styling changes when it has focus.
    • It presses only Tab, never Enter or Space, so it doesn't follow links or press buttons. It stops early if focus cycles back to the start or stays in one place, then returns the page to where it was scrolled.
    • With Include Screenshots on, it also sends small crops of up to six stops where no style change was found.
  • Form Errors (3.3.1, 3.3.3): Fields whose errors wouldn't be identified in text, or wouldn't suggest how to fix them.
    • Octopus Scanner never submits a form, clicks, types into a field or fires page events, and it leaves password, file, payment card and one-time code fields alone.
    • For fields that must be filled in or need a set format, it briefly sets an empty or invalid value inside the page to read the browser's built-in validation message, then restores the original value straight away. It also checks how error messages are linked to each field.
    • Errors a site shows only after a real submit aren't seen.

7. Reading the results

An item in the LLM Review section. The suggestion, model name and values are examples.
Text description

Example of the LLM Review section: an LLM Review group with 2 issue types and 2 pages affected, and the note Experimental: Suggested by a local LLM. These are not confirmed issues: LLMs can be wrong or inconsistent, and results depend on the model used. Review each one before relying on it. Model: octo-chat on local network. The item Vague Link Text has View Ticket, Approve 82 percent, Remove and Next Issue buttons. Its Details list the description, the source Local LLM review (experimental), not detected by axe-core, LLM Review 82 percent, the suggested impact Serious with WCAG 244 and 2.4.4 Link Purpose (In Context), and 3 instances across 2 pages.

  • LLM Review on axe-core issues: When Check axe-core Findings is on, each issue type the model checked shows an LLM Review percentage: how likely the model thinks the issue is genuine. View Ticket shows the same percentage with the model's short reason. A low score is a prompt to look again, not proof that the issue is false.
  • The LLM Review section: Appears after Needs Review when the model suggested new issues, and has its own LLM Review tab.
    • It opens with "Experimental: Suggested by a local LLM. These are not confirmed issues".
    • Suggestions of the same type are combined into one item, with each affected element listed as an instance, just as axe-core combines its findings.
    • Each item is laid out like an axe-core issue: a short name, then Details with the description, the source ("Local LLM review (experimental), not detected by axe-core"), the model's confidence, a suggested impact with its WCAG success criterion, and the number of instances. When the report includes screenshots, each item has a screenshot of the page with the element outlined, stamped like other evidence when Stamp Screenshot is on.
    • Affected Elements & Instances lists each page with the element's HTML, its selector and the suggested fix, and View Ticket opens a full ticket, like any other issue.
  • Approve and Remove: Work as they do for Needs Review. Approve moves the item into its suggested severity group, and its details still say it was suggested by a local LLM. Remove removes it from the report.
  • Counts: axe-core's counts aren't changed. LLM suggestions are counted only in the LLM Review section.
  • Environment: The report's scan details name the model and whether it ran on a local computer or the local network, for example "octo-chat on local network". They never include the server's address or the key.
  • CSV and JSON: The CSV's last two columns are Source, which is "axe-core" or "Local LLM", and LLM Review, such as "valid 82% (2 of 3 agree)" for a checked finding or "suggested 75% (Alt Text Quality, LLM review)" for a suggestion. The JSON file adds an llmReview object with the model, counts and findings, and an llm field on each issue the model checked. See Report File Formats.

For how tickets, Approve and Remove work in general, see Report Anatomy.

8. Privacy and safety

  • What is sent: Only while Local LLM Review is on, and only to the server you chose.
    • For every reviewed page: its title, language and path, its heading outline, its images (accessible name or alt text, role, file name and nearby text), its links and buttons (accessible name, link path and surrounding text) and its form fields (type, label, whether required and description).
    • Depending on the categories you turn on: the text that follows each heading, short passages of the page's text (each up to 200 characters) with any language they are marked with, the words shown on controls where they differ from the accessible name, video and audio details (caption tracks and nearby text), the order of content in the code and on screen, navigation links to compare across pages and measured contrast ratios.
    • Focus Visible and Form Errors: Focus Visible sends each focus stop's name and how its styling changed. Form Errors sends each checked field's rules, the browser's validation message and how error messages are linked to it.
    • When checking axe-core findings: the rule with its description, WCAG tags and impact, the page title, the failing element's HTML and the failure summary.
    • With Include Screenshots on: page and issue screenshots, crops of large images for Images of Text and crops of focus stops for Focus Visible.
    • Custom Instructions: added to each request.
  • What is never sent: Values typed into form fields, passwords and Sensitive Values are never sent. Form Errors sends no field values, and leaves password, file and payment card fields out.
  • Finding servers: Find Servers and Search Local Network run only when you select them, and contact only this computer or devices on your local network. They only ask for a list of models, and never send your API key or page details.
  • Where it goes: Requests go only to the server on your computer or your local network that you entered. Nothing is sent to an AI provider or to the developer, and LLM requests never go through a proxy set for the scan.
  • Your key: The API key is saved in a file that only your user account can read.
    • It isn't encrypted, so software running as you could still read it.
    • It is never included in saved or exported configurations, is only sent to the server it was saved for, and the interface is never sent it back.
    • Keys in configurations saved by earlier versions are moved into this file the first time the interface opens. Only one key is kept, so if your configurations had keys for different servers, a message lists the configurations where you need to enter the key again.
    • Use a key made only for your LLM server, not one you use elsewhere.
  • Model output is treated as untrusted: Pages can contain text that tries to instruct a model, so Octopus Scanner treats every answer as data. Answers must match a fixed structure, are shortened to set lengths and are shown as plain text in the report. Suggestions about elements that weren't sent are discarded. The model is never given tools or any way to act on your computer.
  • Nothing sensitive is logged: Prompts, the model's full answers, page content and the key are never written to the Activity Log or log files. Only short progress lines are, such as the issue type and page checked, whether the model confirmed it and how sure it was, the categories of any suggestions, skipped requests, counts, timings and errors.
  • Secure connections stay secure: HTTPS certificates are always checked. Trusted Certificate adds your server's certificate without turning checks off.
  • Screenshots are temporary: Screenshots taken for the review are kept only while it runs and are then deleted.

For everything else Octopus Scanner stores and sends, see Data, Privacy & Security.

9. Limits and responsibility

An LLM doesn't test a page the way a person does. It sees only the details sent to it, can misread them and can give confident answers that are wrong. Results depend on the model you choose, and two models can disagree about the same element.

LLM findings are suggestions, not confirmed issues. Check each one yourself before you rely on it or pass it on. Local LLM Review doesn't replace manual testing, and a report with no LLM suggestions doesn't mean a page is accessible. See Scope of Automated Testing.

You choose and run the model. Octopus Scanner isn't responsible for the model, its output or any charges you receive if you connect a paid or metered service.

Local LLM Review runs on crawled, sitemap, listed and journey scans. It doesn't run in Interactive Mode.

10. Tips

  • Start with a few pages: Set a low Max Pages To Review, or scan a small part of the site, to see how your model performs and how long it takes before you run it on a large scan.
  • Keep Consistent Answers on: It makes results easier to compare between scans.
  • Set Minimum Confidence: Raise it if the LLM Review section fills with weak suggestions, and lower it if you want to see more.
  • Use screenshots only with a vision model: A model that can't read images can't use them, and screenshots make every request take longer. Turn off Include Screenshots if your model is text-only.
  • Prefer a model without reasoning: Reasoning (thinking) models can spend their whole reply on thinking and give no answer. Octopus Scanner gives them more room once, then stops the review with a message saying so. Choose a standard chat model, or turn thinking off on your server.
  • Allow for extra time: Requests are sent one at a time after scanning finishes, so a slow model or many pages adds to the time before the report is ready.
    • Each reviewed page gets one request for its text and, with Include Screenshots on, one for its screenshot.
    • Focus Visible and Form Errors, both on by default, each add another request per page, and Focus Visible also adds time while each page is scanned, so they make the review noticeably longer. Turn them off under Categories for a quicker review.

Was this page helpful? Email your feedback.