[Medformatik]

What changed and why

The installed app hid the voice search button the website shows, because the WebView has no microphone and browser recognition would send audio to a cloud service. The app now shows the same button on Android and iPhone and recognises speech on the phone. The button, its listening state and its warnings are the website's; only the engine underneath changes. Interim text fills the search field and the final transcript searches, as on the website.

Recognition uses expo-speech-recognition 57.1.1 (MIT), a wrapper around Android's SpeechRecognizer and Apple's SFSpeechRecognizer. Recording never goes to a server recogniser on either platform:

• Android 13 and later: requiresOnDeviceRecognition: true makes the package call createOnDeviceSpeechRecognizer, which has no network fallback. On Android 11–12 the package would only prefer offline recognition, so the button stays hidden there.
• iPhone: the package set requiresOnDeviceRecognition only when the recogniser supported it and otherwise fell back to Apple's servers (jamsch/expo-speech-recognition#169). Our patch checks the requested language before the microphone is touched and again right before recording, always sets the flag, and reports "language not supported" instead of falling back. The availability check takes a language, so the button reflects German and English rather than the phone's default language.

The decision record, including the packages that were rejected, is in docs/docs/developer/voice-input.md.

Android

The first tap shows Android's own microphone prompt. A refusal shows the existing warning, now pointing to Android settings:

If the requested language isn't installed for offline recognition, the existing warning says so and Android's own language download is triggered (here German on an emulator that only has English):

iPhone

The first tap asks for the microphone, then for speech recognition, each with our explanation (iOS 26.4 simulator):

The second alert carries Apple's fixed text, "Speech data from this app will be sent to Apple to process your requests", followed by ours. Apple shows that text for every speech-recognition request, regardless of requiresOnDeviceRecognition, so here it contradicts what the app does. The package itself skips this permission for on-device recognition, but Apple's guidance doesn't exempt that path, so the app requests it for now. If a real iPhone shows that on-device recognition works without it, the follow-up is to stop requesting it so the alert never appears.

A refusal of either permission shows the existing warning pointing to iPhone Settings > Privacy & Security. A language the iPhone can't recognise offline shows a warning pointing to the keyboard and Dictation settings, without a prompt or recording. iOS has no download trigger like Android's. After listening, the audio session is handed back (deactivated with notifyOthersOnDeactivation unless guidance is speaking) before a held guidance cue plays, so other audio resumes and guidance configures its own session.

Both platforms

• Bridge: the page sends voice.start, voice.stop and voice.cancel and receives voice.event (listening, partial, final, ended, error). It learns availability with a voice.probe request after the handshake. native.hello is unchanged, so a new APK still negotiates with the deployed website, and an older APK leaves the probe unanswered and the button stays hidden. No protocol version change. The handshake's existing platform picks the Android or iPhone wording.
• Lifecycle: one utterance, at most 15 seconds. Listening is cancelled when the app leaves the foreground, on page reload or load failure, on Cancel, and when the app closes. Permission prompts pause the app (an activity on Android, an alert on iOS), so they are exempted. Late results are dropped on both sides. A recogniser that never reports its end is released after 3 seconds, and the next start waits for it to confirm before listening again.
• Guidance: spoken directions are held while listening, and the latest held cue is spoken afterwards only if it is at most 10 seconds old. Listening waits for a cue being spoken (up to 6 seconds) rather than record it.
• User intent: the page cannot start the microphone on its own. The shell accepts voice.start only within 3 seconds of a touch it observed on its own view, one start per touch, so a script injected into the page cannot listen in a loop. With a screen reader on, activation arrives as an accessibility action rather than a touch, so the requirement is waived there. The operating system's own microphone indicator, which the page cannot hide, shows whenever the recogniser holds the microphone.
• Privacy: the WebView never gets the microphone. On Android react-native-webview is patched to deny audio and video capture; without that patch it would grant the page the microphone once the app holds RECORD_AUDIO. iOS keeps mediaCapturePermissionGrantType="deny". The speech package patch also removes three log lines that wrote transcripts to Logcat. Recordings are not persisted. The privacy pages, the compliance registry (voice-search-audio) and the store worksheets describe the practice for both platforms.
• Docker: pnpm verifies every patched dependency's file hash during install, so the web, API, ops-agent, transitous-runner and data-manager images now copy patches/ before pnpm install. Without it they fail with "Patch file not found" (reproduced locally).

The issue proposed a rationale screen and a settings button before the OS prompt. Both are left out because this change adds no UI; the OS prompt follows the tap that asks for it.

Related work

Refs #405. This does not close it: accuracy and latency on a consented recording set and device qualification on physical phones remain open.

Found while testing, not part of this change: the app does not start on the iOS 27 simulator ("Application failed to launch: UIScene life cycle is required"). Apps built with the iOS 27 SDK must adopt the scene-based life cycle, and the AppDelegate our Expo template generates doesn't yet, although Expo 57 already ships ExpoAppSceneDelegate. That affects the whole iPhone app and needs its own fix.

Verification

• Tests: Vitest 104 files / 2,130 tests (core protocol, web button and bridge, mobile config, compliance, Docker and supply-chain contracts); mobile Jest 66 suites / 1,319 tests. They cover the protocol messages, the probe in both compatibility directions, the button's stop/cancel/late-event behaviour and platform wording, the controller on both platforms (permissions and prompt exemption, refusal, iOS language selection and fail-closed start, Android model download, stop and settle deadlines, unconfirmed recogniser, background cancel, iOS audio hand-back order), the touch gate and the guidance hold.
• tsc --noEmit for core, web and mobile; Biome on changed files; Prettier on docs; store worksheets --check; the full check:policy; assert-no-community-runtime (checks the WebView patch and both the Android log removals and the iOS fail-closed guards in the installed speech package); clean Android and iOS prebuilds (the Android manifest adds only RECORD_AUDIO and the RecognitionService query; the iOS Info.plist carries both localized usage strings).
• Android emulator (API 36.1 Google Play image, debug build against a local web build):
  • the button appears after the probe;
  • Android's microphone prompt appears on the first tap, and a refusal shows the warning;
  • with permission, Android System Intelligence's on-device recogniser ran ("Offline recognizer – start listening", locale en-US), opened the microphone, reported no speech (the emulator microphone is silent), closed the microphone and shut down;
  • pressing Home while listening cancelled the session and closed the microphone at once, and the button was back to normal on return;
  • in airplane mode the offline recogniser still started;
  • German, not installed on the emulator, showed the warning and triggered Android's model download without opening the microphone;
  • getUserMedia({ audio: true }) from the page was refused while the app held RECORD_AUDIO;
  • a voice.start posted from the page without a touch was answered with an error and recorded nothing, while a real tap on the button went through.
• iOS 26.4 simulator (debug build of this branch, which compiles with the patched Swift):
  • the simulator reports on-device support for en-US only, and the button appears;
  • the first tap shows the microphone alert and then the Speech Recognition alert, with our usage strings;
  • refusing Speech Recognition shows the iPhone wording of the warning;
  • with the page in German, the tap shows the iPhone "not available offline" warning, with no alert and no recording;
  • a real tap passes the touch gate on iOS too.
• An automated security review flagged that the page could start the microphone without a gesture; the touch gate above addresses it.
• An independent review found three issues, all fixed with tests: a timed-out session could let the old recogniser's output into the next session; the handshake capability would have broken the deployed website's strict parser; and a background cancellation left the button showing "listening".
• Not verified:
  • real speech in German and English on either platform (the emulator microphone is silent; the simulator's audio input failed without host microphone access and later hung);
  • whether iOS on-device recognition needs the Speech Recognition permission;
  • guidance held during an actual trip;
  • physical phones and release builds;
  • Docker image builds.

Release and upgrade impact

No package release: This adds bridge messages to the private @openmapx/core protocol used between the web app and the installed shell, and the lockfile and workspace changes add a mobile-only dependency and patches for two mobile-only packages; the published extension packages' files and contracts stay the same.

Deploy order does not matter: the deployed website ignores the new APK's capabilities, and older APKs never answer the probe.