Conversation
…om AI screenshots When a click fails because the element is disabled, the click tool hovers it once and returns the text the app showed as disabledReason. Pilot's tool summary includes it, and the disabled-click hint no longer lists guessed preconditions that sent the tester after the wrong cause. Screenshots taken for see()/visualClick hide Playwright screencast overlays, so the vision model no longer reads Explorbot's own chapter captions as page content. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Resolve the click-tool conflict with main's judge change (failedToolResult no longer takes judge/intent). disabledReason and its suggestion are now set only when hovering showed text; otherwise the plain disabled hint applies. The rule no longer names a field that may be absent. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
In session
AssociatedWillowyChocolate907the tester found that the plan's Launch button was disabled, but blamed the wrong cause. The click-failure hint listed guessed preconditions ("a required field is empty, nothing is selected…"). The model picked the nearby "Set milestone" control, spent about 90s on it, and stopped with a wrong diagnosis. The real reason ("Continuous Integration is required…") appears only as a tooltip on hover, and the tester never hovered.In the same session,
see()reported "a dark instructional overlay reading 'Open first milestone combobox'". That was Explorbot's own screencast chapter caption, and the tester then tried to click it.Changes
element is not enabled, the tool hovers that same element once (I.moveCursorTowith the same locator and container). Text that appears as a result is returned asdisabledReason. It tries each disabled locator in turn and stops at the first hover that works.clickFailureSuggestion) andrules.ts: point the model todisabledReasoninstead of a list of guessed causes. The model looks for an unmet precondition only whendisabledReasonis empty.shown on hover of disabled target: ….page.screencast.hideOverlays()/showOverlays()aroundpage.screenshot().Limits
titletooltip is drawn by the browser, not added to the page, so hovering can't capture it.Testing
tests/unit/click-disabled.test.ts: a tooltip that appears on hover is reported, and an empty result is returned when nothing appears.bun test tests/unit(1626 pass) andbun test tests/integrationpass. Format and lint are clean.🤖 Generated with Claude Code