bug: the visual service fails on Android Chrome through Appium 3 in the default WebDriver BiDi session (workaround: wdio:enforceWebDriverClassic) #1232

Environment (please complete the following information):

• Node.js version: 24.20.0
• NPM version: 11.19.0
• Browser name and version: Chrome 133.0.6943.137 on an Android 16 emulator (Pixel 9 Pro, API 36)
• Platform name and version: macOS 27; Appium 3.8.0, UiAutomator2 driver 8.7.0 (appium-android-driver 14.2.0), Chromedriver 133.0.6943.141
• WebdriverIO version: 9.32.0 (also 10.0.0-alpha.163)
• @wdio/visual-service version: 10.1.0 (also the wdio0-v10-compliance branch of feat: support WebdriverIO v10 (keep v9) and fix the visual matchers with Jasmine #1213)

Config of WebdriverIO + @wdio/visual-service

// MODE=bidi (default), classic or forward
const mode = process.env.MODE ?? 'bidi'

export const capabilities = {
    platformName: 'Android',
    browserName: 'Chrome',
    'appium:automationName': 'UiAutomator2',
    'appium:deviceName': process.env.DEVICE ?? 'emulator-5554',
    'appium:newCommandTimeout': 240,
    'appium:chromeOptions': { args: ['--no-first-run', '--disable-fre'] },
    ...(mode === 'classic' ? { 'wdio:enforceWebDriverClassic': true } : {}),
    ...(mode === 'forward' ? { 'appium:chromedriverForwardBiDi': true } : {}),
}

// The Android emulator reaches the host at 10.0.2.2
export const pageUrl = 'http://10.0.2.2:8765/mobile.html'
export { mode }

import { capabilities, mode } from './capabilities.js'

export const config = {
    runner: 'local',
    hostname: '127.0.0.1',
    port: 4723,
    specs: ['./test/visual.spec.js'],
    capabilities: [capabilities],
    framework: 'mocha',
    reporters: ['spec'],
    services: [['visual', {
        baselineFolder: `./.tmp/${mode}/baseline`,
        screenshotPath: `./.tmp/${mode}/actual`,
    }]],
    logLevel: 'warn',
    connectionRetryCount: 0,
    mochaOpts: { timeout: 240000 },
}

Describe the bug

On Android Chrome through Appium 3, the visual service does not work in the default session. WebdriverIO opens a WebDriver BiDi session (browser.isBidi is true), but Appium does not support the browser BiDi commands in this session:

• browser.url() fails: WebDriver Bidi command "browsingContext.navigate" failed with error: unknown method - Method has not yet been implemented
• every browser.execute() fails: WebDriver Bidi command "script.callFunction" failed with error: unknown command

The visual service runs all its client-side scripts with browser.execute(), so all the visual commands and matchers fail. With WebdriverIO 9 and the published 10.1.0, the service before hook fails (Error getting mobile screen size), the matchers are not added, and the user only sees expect(...).toMatchScreenSnapshot is not a function.

This is not a bug in the service code. The workaround is 'wdio:enforceWebDriverClassic': true in the capabilities: then all the tests pass. This repository uses it in its Android web configs since #417, but the README does not tell users.

WebdriverIO v10 requires Appium 3, so all the users who upgrade will get this problem on Android Chrome.

To Reproduce

A small repro: a WebdriverIO-only script (probe.mjs, no service) and a visual test (test/visual.spec.js). Steps:

1. Start an Android emulator with Chrome (emulator-5554).
2. npm i -g appium@3, appium driver install uiautomator2, then appium --port 4723 --allow-insecure "uiautomator2:chromedriver_autodownload".
3. npm install, then npm run page (serves www/ on port 8765; the emulator opens http://10.0.2.2:8765/mobile.html).
4. MODE=bidi npm run probe, MODE=bidi npm test, and the same with MODE=classic and MODE=forward. Run MODE=classic npm test two times: the first run saves the baselines.

{
  "name": "wdio-appium3-android-bidi-repro",
  "private": true,
  "type": "module",
  "scripts": {
    "page": "python3 -m http.server 8765 --bind 127.0.0.1 --directory www",
    "test": "wdio run wdio.conf.js",
    "probe": "node probe.mjs"
  },
  "devDependencies": {
    "@wdio/cli": "^9.32.0",
    "@wdio/globals": "^9.31.3",
    "@wdio/local-runner": "^9.32.0",
    "@wdio/mocha-framework": "^9.32.0",
    "@wdio/spec-reporter": "^9.32.0",
    "@wdio/visual-service": "^10.1.0",
    "webdriverio": "^9.32.0"
  }
}

// No visual service, no test runner: shows the WebdriverIO + Appium result only.
// Usage: MODE=bidi|classic|forward node probe.mjs
import { remote } from 'webdriverio'
import { capabilities, mode, pageUrl } from './capabilities.js'

const browser = await remote({ hostname: '127.0.0.1', port: 4723, logLevel: 'error', capabilities })
const run = async (name, fn) => {
    try {
        console.log(`${name}: OK ${JSON.stringify(await fn())?.slice(0, 60) ?? ''}`)
    } catch (err) {
        console.log(`${name}: ERROR ${err.message.split('\n')[0].slice(0, 150)}`)
    }
}

try {
    console.log(`mode ${mode}, isBidi: ${browser.isBidi}`)
    await run('url()', async () => { await browser.url(pageUrl) })
    await run('getUrl()', () => browser.getUrl())
    await run('execute(() => document.title)', () => browser.execute(() => document.title))
    await run('executeScript("return document.title", [])', () => browser.executeScript('return document.title', []))
} finally {
    await browser.deleteSession()
}

import { browser, $, expect } from '@wdio/globals'
import { pageUrl } from '../capabilities.js'

describe('Android Chrome through Appium 3', () => {
    it('opens the page', async () => {
        console.log(`isBidi: ${browser.isBidi}`)
        await browser.url(pageUrl)
        await expect($('h1')).toHaveText('Visual test')
    })

    it('matches the screen', async () => {
        await expect(browser).toMatchScreenSnapshot('screen')
    })

    it('matches an element', async () => {
        await expect($('#target')).toMatchElementSnapshot('target')
    })
})

<!doctype html>
<html>
<head><meta charset="utf-8"><meta name="viewport" content="width=device-width, initial-scale=1"><title>Mobile</title>
<style>
    body { margin: 0; background: #fff; font-family: sans-serif; }
    h1 { margin: 16px; font-size: 24px; color: #1b5e20; }
    #target { margin: 16px; width: 200px; height: 100px; background: #388e3c; border: 4px solid #1b5e20; }
</style>
</head>
<body><h1>Visual test</h1><div id="target"></div></body>
</html>

Results (the same with WebdriverIO 9.32.0 and 10.0.0-alpha.163):

Expected behavior

The visual service works on Android Chrome through Appium 3 in the default WebdriverIO session, or the docs tell users to use 'wdio:enforceWebDriverClassic': true, and the service gives a clear error.

Log

WebdriverIO 9.32.0 with the published 10.1.0, MODE=bidi:

WARN webdriverio:ShadowRootManager: Could not register shadow root preload script, shadow DOM will not be pierced in this session: WebDriver Bidi command "script.addPreloadScript" failed with error: unknown command
WARN @wdio/visual-service:@wdio/image-comparison-core:utils: Error getting mobile screen size:
 Error: WebDriver Bidi command "script.callFunction" failed with error: unknown command - The requested resource could not be found, or a request was received using an HTTP method that is not supported by the mapped resource.
ERROR @wdio/utils:shim: Error: WebDriver Bidi command "script.callFunction" failed with error: unknown command - ...
1) Android Chrome through Appium 3 opens the page
WebDriver Bidi command "browsingContext.navigate" failed with error: unknown method - Method has not yet been implemented
2) Android Chrome through Appium 3 matches the screen
TypeError: expect(...).toMatchScreenSnapshot is not a function
3) Android Chrome through Appium 3 matches an element
TypeError: expect(...).toMatchElementSnapshot is not a function

Additional context

What I found about the cause (local tests on the emulator):

1. WebdriverIO adds webSocketUrl: true to every session that has a browserName, except Safari and 'wdio:enforceWebDriverClassic': true (packages/webdriver/src/utils.ts). There is no exception for Appium.
2. Since Appium 2.5.0, the Appium server sends back its own BiDi URL when the client asks for one. UiAutomator2 does not forward the BiDi commands to Chromedriver by default, so browsingContext.navigate and script.callFunction are not supported. An Appium maintainer said the same in No connection to WebDriver Bidi was established webdriverio#14600.
3. With Appium 2.19.0 (UiAutomator2 4.2.9), execute() also fails in this BiDi session, but url() works: WebdriverIO uses the Classic navigateTo() when browsingContext.navigate fails with unknown error (url.ts). Appium 3 sends unknown method, so this fallback does not apply. Thus the visual service already fails with Appium 2.5+ in BiDi; Appium 3 also breaks url().
4. The option 'appium:chromedriverForwardBiDi': true must forward BiDi to Chromedriver, but it has no effect in a browser session: startChromeSession() does not give the context name to setupNewChromedriver(), so cacheChromedriverCaps() returns before it saves the Chromedriver BiDi URL. (The webview path, line 292, gives it.) Chromedriver did return a webSocketUrl.
5. With that line patched locally, Appium forwards BiDi to Chromedriver and url() works, but execute() fails with no such frame - Context CHROMIUM not found: on mobile, the WebdriverIO context manager uses the Appium context name from getContext() as the current context (context.ts), and BiDi uses it as the browsing context ID. After browser.switchToWindow(await browser.getWindowHandle()), execute() works, and the visual service passes like in Classic. In this real BiDi session, execute('mobile: deviceInfo') fails with ReferenceError: deviceInfo is not defined, and executeScript('mobile: deviceInfo', []) works (this confirms the change of fix: run Appium mobile: commands with executeScript #1220).

iOS Safari through Appium 3 (XCUITest) is not affected: WebdriverIO does not ask for BiDi for Safari.

Proposed work in this repository:

• README: for Android Chrome through Appium, set 'wdio:enforceWebDriverClassic': true (WebdriverIO v9 and v10).
• When the matchers or the instance data cannot be set up because of this, give a clear warning with the workaround, not toMatchScreenSnapshot is not a function.

The root causes are in WebdriverIO (points 1, 3, 5) and in appium-android-driver (point 4). Issues there are not opened yet.

Follow-up

The default BiDi session works only when all of the fixes below are done. A partial fix does not help: for example, with only the Appium fix, url() works but every execute() still fails.

This repository (closes this issue)

• README: for Android Chrome through Appium, set 'wdio:enforceWebDriverClassic': true (WebdriverIO v9 and v10).
• A clear warning with this workaround when the service cannot set up in an Android BiDi session (not toMatchScreenSnapshot is not a function).

appium/appium-android-driver (issue or PR to open)

• startChromeSession() must give the context name to setupNewChromedriver(), so that appium:chromedriverForwardBiDi works in a browser session (point 4).
• Question: can the BiDi forward be the default for browser sessions?

webdriverio/webdriverio (issue to open)

• Mobile web BiDi: use the window handle, not the Appium context name (CHROMIUM), as the browsing context ID (point 5). This bug is independent of Appium.
• For Android Chrome through Appium: add appium:chromedriverForwardBiDi: true when WebdriverIO asks for BiDi (after the Appium fix is released), or do not ask for BiDi (point 1). Keep BiDi when the user sets webSocketUrl: true, and do not change appium-geckodriver and appium-chromium-driver, because they support BiDi.
• Optional: in url(), also use the Classic fallback for the Appium 3 error unknown method (point 3).
• v10 migration guide, Appium section: Android Chrome needs 'wdio:enforceWebDriverClassic': true until these fixes are released.

When the upstream fixes are released, we can test the default session again with the repro of this issue, and then update the README.