WebKit Bugzilla
Attachment 369207 Details for
Bug 195696
: Web Automation: elements larger than the viewport have incorrect in-view center point
Home
|
New
|
Browse
|
Search
|
[?]
|
Reports
|
Requests
|
Help
|
New Account
|
Log In
Remember
[x]
|
Forgot Password
Login:
[x]
[patch]
Patch
bug-195696-20190506170136.patch (text/plain), 35.35 KB, created by
BJ Burg
on 2019-05-06 17:01:36 PDT
(
hide
)
Description:
Patch
Filename:
MIME Type:
Creator:
BJ Burg
Created:
2019-05-06 17:01:36 PDT
Size:
35.35 KB
patch
obsolete
>Subversion Revision: 244956 >diff --git a/Source/WebCore/ChangeLog b/Source/WebCore/ChangeLog >index 87ff5149b122fb90b7ece5da662314ff61195f56..f6f7ae1afc461918e6e0dc09f8b9f9a98cd5ae02 100644 >--- a/Source/WebCore/ChangeLog >+++ b/Source/WebCore/ChangeLog >@@ -1,3 +1,32 @@ >+2019-05-06 Brian Burg <bburg@apple.com> >+ >+ Web Automation: elements larger than the viewport have incorrect in-view center point >+ https://bugs.webkit.org/show_bug.cgi?id=195696 >+ <rdar://problem/48737122> >+ >+ Reviewed by NOBODY (OOPS!). >+ >+ Some conversion methods do not exist for FloatRect/FloatPoint. >+ Fill them in as needed, and export some symbols used by WebDriver >+ code to compute an element's in-view center point in various coordinate systems. >+ >+ * page/FrameView.cpp: >+ (WebCore::FrameView::absoluteToLayoutViewportPoint const): >+ (WebCore::FrameView::layoutViewportToAbsoluteRect const): >+ (WebCore::FrameView::absoluteToLayoutViewportRect const): >+ * page/FrameView.h: >+ * platform/ScrollView.cpp: >+ (WebCore::ScrollView::viewToContents const): >+ (WebCore::ScrollView::contentsToView const): >+ (WebCore::ScrollView::contentsToRootView const): >+ * platform/ScrollView.h: >+ * platform/Widget.cpp: >+ (WebCore::Widget::convertToRootView const): >+ (WebCore::Widget::convertFromRootView const): >+ (WebCore::Widget::convertToContainingView const): >+ (WebCore::Widget::convertFromContainingView const): >+ * platform/Widget.h: >+ > 2019-05-05 Wenson Hsieh <wenson_hsieh@apple.com> > > fast/attachment/attachment-folder-icon.html is an Image Only failure on recent macOS builds >diff --git a/Source/WebKit/ChangeLog b/Source/WebKit/ChangeLog >index c3aa76e21057b2b1bc2ae7c1a20dd1bb0e7bb5ef..beb4d186e4c27f3f90716fde655889e0faf13d81 100644 >--- a/Source/WebKit/ChangeLog >+++ b/Source/WebKit/ChangeLog >@@ -1,3 +1,48 @@ >+2019-05-06 Brian Burg <bburg@apple.com> >+ >+ Web Automation: elements larger than the viewport have incorrect in-view center point >+ https://bugs.webkit.org/show_bug.cgi?id=195696 >+ <rdar://problem/48737122> >+ >+ Reviewed by NOBODY (OOPS!). >+ >+ This seems to be an omission in the specification. While it does mention that the in-view >+ center point (IVCP) must be within the viewport, the algorithm never intersects the element >+ bounding box with the viewport rect. >+ >+ * Platform/Logging.h: Add logging channel to dump fully resolved interaction details. >+ * UIProcess/Automation/WebAutomationSession.cpp: >+ (WebKit::WebAutomationSession::simulateMouseInteraction): >+ Rename locationInView -> locationInViewport. >+ >+ (WebKit::WebAutomationSession::simulateTouchInteraction): >+ This code is incorrect. The unobscuredContentRect is in screen coordinates, but >+ we are trying to see if (x, y) is outside the size of the viewport assumed to be at (0, 0). >+ Grab the visual viewport rect and see if the location exceeds the viewport size. >+ >+ * UIProcess/Automation/ios/WebAutomationSessionIOS.mm: >+ (WebKit::operator<<): Add logging helper for TouchInteraction enum. >+ (WebKit::WebAutomationSession::platformSimulateTouchInteraction): Move local variable. >+ >+ * WebProcess/Automation/WebAutomationSessionProxy.cpp: >+ (WebKit::elementInViewClientCenterPoint): Deleted. >+ (WebKit::elementInViewCenterPoint): >+ Pass in the main FrameView as we need to intersect the element rect and the viewport rect >+ before finding the center point of the part of the element that's visible in the viewport. >+ >+ (WebKit::WebAutomationSessionProxy::computeElementLayout): >+ This code is incorrect. For CoordinateSystem::LayoutViewport, coordinates should be in root view coordinates >+ so that it can be later converted to screen and synthesized as a HID event in screen coordinates. >+ >+ (WebKit::convertRectFromFrameClientToRootView): >+ (WebKit::convertPointFromFrameClientToRootView): >+ Added helpers to properly account for scroll contents position on iOS. >+ >+ * UIProcess/Automation/SimulatedInputDispatcher.cpp: >+ (WebKit::SimulatedInputDispatcher::transitionInputSourceToState): Fix a typo in logging. >+ >+ * UIProcess/Automation/Automation.json: Simplify enum name. >+ > 2019-05-04 Alex Christensen <achristensen@webkit.org> > > Revert r244953 and r244954 because they broke internal builds. >diff --git a/Source/WebCore/page/FrameView.cpp b/Source/WebCore/page/FrameView.cpp >index 50813d83a39b93f67b3ccaf1b632f05412860ec5..3797f8f80818dc48e4a41ed2c95e86419b9998b1 100644 >--- a/Source/WebCore/page/FrameView.cpp >+++ b/Source/WebCore/page/FrameView.cpp >@@ -4791,6 +4791,30 @@ FloatPoint FrameView::clientToDocumentPoint(FloatPoint point) const > return point; > } > >+FloatPoint FrameView::absoluteToLayoutViewportPoint(FloatPoint p) const >+{ >+ ASSERT(frame().settings().visualViewportEnabled()); >+ p.scale(1 / frame().frameScaleFactor()); >+ p.moveBy(-layoutViewportRect().location()); >+ return p; >+} >+ >+FloatRect FrameView::layoutViewportToAbsoluteRect(FloatRect rect) const >+{ >+ ASSERT(frame().settings().visualViewportEnabled()); >+ rect.moveBy(layoutViewportRect().location()); >+ rect.scale(frame().frameScaleFactor()); >+ return rect; >+} >+ >+FloatRect FrameView::absoluteToLayoutViewportRect(FloatRect rect) const >+{ >+ ASSERT(frame().settings().visualViewportEnabled()); >+ rect.scale(1 / frame().frameScaleFactor()); >+ rect.moveBy(-layoutViewportRect().location()); >+ return rect; >+} >+ > FloatPoint FrameView::layoutViewportToAbsolutePoint(FloatPoint p) const > { > ASSERT(frame().settings().visualViewportEnabled()); >diff --git a/Source/WebCore/page/FrameView.h b/Source/WebCore/page/FrameView.h >index 81d1cc62db70706913433b76fb03c04f92b66dde..d36e565645c3794e6a8513178d6edbff0cfdb9e8 100644 >--- a/Source/WebCore/page/FrameView.h >+++ b/Source/WebCore/page/FrameView.h >@@ -475,19 +475,23 @@ public: > float documentToAbsoluteScaleFactor(Optional<float> effectiveZoom = WTF::nullopt) const; > float absoluteToDocumentScaleFactor(Optional<float> effectiveZoom = WTF::nullopt) const; > >- FloatRect absoluteToDocumentRect(FloatRect, Optional<float> effectiveZoom = WTF::nullopt) const; >- FloatPoint absoluteToDocumentPoint(FloatPoint, Optional<float> effectiveZoom = WTF::nullopt) const; >+ WEBCORE_EXPORT FloatRect absoluteToDocumentRect(FloatRect, Optional<float> effectiveZoom = WTF::nullopt) const; >+ WEBCORE_EXPORT FloatPoint absoluteToDocumentPoint(FloatPoint, Optional<float> effectiveZoom = WTF::nullopt) const; > > FloatRect absoluteToClientRect(FloatRect, Optional<float> effectiveZoom = WTF::nullopt) const; > > FloatSize documentToClientOffset() const; >- FloatRect documentToClientRect(FloatRect) const; >+ WEBCORE_EXPORT FloatRect documentToClientRect(FloatRect) const; > FloatPoint documentToClientPoint(FloatPoint) const; > WEBCORE_EXPORT FloatRect clientToDocumentRect(FloatRect) const; > WEBCORE_EXPORT FloatPoint clientToDocumentPoint(FloatPoint) const; > >+ WEBCORE_EXPORT FloatPoint absoluteToLayoutViewportPoint(FloatPoint) const; > FloatPoint layoutViewportToAbsolutePoint(FloatPoint) const; > >+ WEBCORE_EXPORT FloatRect absoluteToLayoutViewportRect(FloatRect) const; >+ FloatRect layoutViewportToAbsoluteRect(FloatRect) const; >+ > // Unlike client coordinates, layout viewport coordinates are affected by page zoom. > WEBCORE_EXPORT FloatRect clientToLayoutViewportRect(FloatRect) const; > WEBCORE_EXPORT FloatPoint clientToLayoutViewportPoint(FloatPoint) const; >diff --git a/Source/WebCore/platform/ScrollView.cpp b/Source/WebCore/platform/ScrollView.cpp >index 255ff67f45686e97db110a216385c7a48f487356..67202ca42ebf5fe781cf48bd3f7ab65a3d850bbd 100644 >--- a/Source/WebCore/platform/ScrollView.cpp >+++ b/Source/WebCore/platform/ScrollView.cpp >@@ -834,6 +834,14 @@ IntPoint ScrollView::viewToContents(const IntPoint& point) const > return point + toIntSize(documentScrollPositionRelativeToViewOrigin()); > } > >+FloatPoint ScrollView::viewToContents(const FloatPoint& point) const >+{ >+ if (delegatesScrolling()) >+ return point; >+ >+ return viewToContents(IntPoint(point)); >+} >+ > IntPoint ScrollView::contentsToView(const IntPoint& point) const > { > if (delegatesScrolling()) >@@ -842,6 +850,14 @@ IntPoint ScrollView::contentsToView(const IntPoint& point) const > return point - toIntSize(documentScrollPositionRelativeToViewOrigin()); > } > >+FloatPoint ScrollView::contentsToView(const FloatPoint& point) const >+{ >+ if (delegatesScrolling()) >+ return point; >+ >+ return contentsToView(IntPoint(point)); >+} >+ > IntRect ScrollView::viewToContents(IntRect rect) const > { > if (delegatesScrolling()) >@@ -908,6 +924,16 @@ IntPoint ScrollView::contentsToRootView(const IntPoint& contentsPoint) const > return convertToRootView(contentsToView(contentsPoint)); > } > >+FloatPoint ScrollView::contentsToRootView(const FloatPoint& contentsPoint) const >+{ >+ return convertToRootView(contentsToView(contentsPoint)); >+} >+ >+FloatRect ScrollView::contentsToRootView(const FloatRect& contentsRect) const >+{ >+ return convertToRootView(contentsToView(contentsRect)); >+} >+ > IntRect ScrollView::rootViewToContents(const IntRect& rootViewRect) const > { > return viewToContents(convertFromRootView(rootViewRect)); >diff --git a/Source/WebCore/platform/ScrollView.h b/Source/WebCore/platform/ScrollView.h >index 8eb46831cc1c85ee1962675334ec1e35fb01c68b..7d824c48aaa51ab218f7373aa9d9e36b675f45dc 100644 >--- a/Source/WebCore/platform/ScrollView.h >+++ b/Source/WebCore/platform/ScrollView.h >@@ -230,7 +230,7 @@ public: > int scrollY() const { return scrollPosition().y(); } > > // Scroll position used by web-exposed features (has legacy iOS behavior). >- IntPoint contentsScrollPosition() const; >+ WEBCORE_EXPORT IntPoint contentsScrollPosition() const; > void setContentsScrollPosition(const IntPoint&); > > #if PLATFORM(IOS_FAMILY) >@@ -279,13 +279,18 @@ public: > > WEBCORE_EXPORT IntPoint rootViewToContents(const IntPoint&) const; > WEBCORE_EXPORT IntPoint contentsToRootView(const IntPoint&) const; >+ WEBCORE_EXPORT FloatPoint contentsToRootView(const FloatPoint&) const; > WEBCORE_EXPORT IntRect rootViewToContents(const IntRect&) const; > WEBCORE_EXPORT IntRect contentsToRootView(const IntRect&) const; > WEBCORE_EXPORT FloatRect rootViewToContents(const FloatRect&) const; >+ WEBCORE_EXPORT FloatRect contentsToRootView(const FloatRect&) const; > > IntPoint viewToContents(const IntPoint&) const; > IntPoint contentsToView(const IntPoint&) const; > >+ FloatPoint viewToContents(const FloatPoint&) const; >+ FloatPoint contentsToView(const FloatPoint&) const; >+ > IntRect viewToContents(IntRect) const; > IntRect contentsToView(IntRect) const; > >diff --git a/Source/WebCore/platform/Widget.cpp b/Source/WebCore/platform/Widget.cpp >index fc941f470412e9b5363b9d1d57a61fbc7209192e..48570aa681c90c793ea3f27788764c9f56527cd5 100644 >--- a/Source/WebCore/platform/Widget.cpp >+++ b/Source/WebCore/platform/Widget.cpp >@@ -95,6 +95,15 @@ IntRect Widget::convertToRootView(const IntRect& localRect) const > return localRect; > } > >+FloatRect Widget::convertToRootView(const FloatRect& localRect) const >+{ >+ if (const ScrollView* parentScrollView = parent()) { >+ FloatRect parentRect = convertToContainingView(localRect); >+ return parentScrollView->convertToRootView(parentRect); >+ } >+ return localRect; >+} >+ > IntPoint Widget::convertFromRootView(const IntPoint& rootPoint) const > { > if (const ScrollView* parentScrollView = parent()) { >@@ -113,6 +122,25 @@ IntPoint Widget::convertToRootView(const IntPoint& localPoint) const > return localPoint; > } > >+ >+FloatPoint Widget::convertFromRootView(const FloatPoint& rootPoint) const >+{ >+ if (const ScrollView* parentScrollView = parent()) { >+ FloatPoint parentPoint = parentScrollView->convertFromRootView(rootPoint); >+ return convertFromContainingView(parentPoint); >+ } >+ return rootPoint; >+} >+ >+FloatPoint Widget::convertToRootView(const FloatPoint& localPoint) const >+{ >+ if (const ScrollView* parentScrollView = parent()) { >+ FloatPoint parentPoint = convertToContainingView(localPoint); >+ return parentScrollView->convertToRootView(parentPoint); >+ } >+ return localPoint; >+} >+ > IntRect Widget::convertFromContainingWindow(const IntRect& windowRect) const > { > if (const ScrollView* parentScrollView = parent()) { >@@ -204,6 +232,11 @@ IntRect Widget::convertFromContainingView(const IntRect& parentRect) const > return parentRect; > } > >+FloatRect Widget::convertToContainingView(const FloatRect& localRect) const >+{ >+ return convertToContainingView(IntRect(localRect)); >+} >+ > FloatRect Widget::convertFromContainingView(const FloatRect& parentRect) const > { > return convertFromContainingView(IntRect(parentRect)); >@@ -225,6 +258,16 @@ IntPoint Widget::convertFromContainingView(const IntPoint& parentPoint) const > return parentPoint; > } > >+FloatPoint Widget::convertToContainingView(const FloatPoint& localPoint) const >+{ >+ return convertToContainingView(IntPoint(localPoint)); >+} >+ >+FloatPoint Widget::convertFromContainingView(const FloatPoint& parentPoint) const >+{ >+ return convertFromContainingView(IntPoint(parentPoint)); >+} >+ > #if !PLATFORM(COCOA) && !PLATFORM(GTK) && !PLATFORM(WIN) > > Widget::~Widget() >diff --git a/Source/WebCore/platform/Widget.h b/Source/WebCore/platform/Widget.h >index 211126544079a71667aea3f87cc2e844eeeca8e9..2aeaeab9309fba91342fca59e799c7dd842104d4 100644 >--- a/Source/WebCore/platform/Widget.h >+++ b/Source/WebCore/platform/Widget.h >@@ -150,11 +150,15 @@ public: > WEBCORE_EXPORT IntRect convertToRootView(const IntRect&) const; > IntRect convertFromRootView(const IntRect&) const; > >+ FloatRect convertToRootView(const FloatRect&) const; > FloatRect convertFromRootView(const FloatRect&) const; > > IntPoint convertToRootView(const IntPoint&) const; > IntPoint convertFromRootView(const IntPoint&) const; > >+ FloatPoint convertToRootView(const FloatPoint&) const; >+ FloatPoint convertFromRootView(const FloatPoint&) const; >+ > // It is important for cross-platform code to realize that Mac has flipped coordinates. Therefore any code > // that tries to convert the location of a rect using the point-based convertFromContainingWindow will end > // up with an inaccurate rect. Always make sure to use the rect-based convertFromContainingWindow method >@@ -186,9 +190,12 @@ public: > // Virtual methods to convert points to/from the containing ScrollView > WEBCORE_EXPORT virtual IntRect convertToContainingView(const IntRect&) const; > WEBCORE_EXPORT virtual IntRect convertFromContainingView(const IntRect&) const; >+ WEBCORE_EXPORT virtual FloatRect convertToContainingView(const FloatRect&) const; > WEBCORE_EXPORT virtual FloatRect convertFromContainingView(const FloatRect&) const; > WEBCORE_EXPORT virtual IntPoint convertToContainingView(const IntPoint&) const; > WEBCORE_EXPORT virtual IntPoint convertFromContainingView(const IntPoint&) const; >+ WEBCORE_EXPORT virtual FloatPoint convertToContainingView(const FloatPoint&) const; >+ WEBCORE_EXPORT virtual FloatPoint convertFromContainingView(const FloatPoint&) const; > > private: > void init(PlatformWidget); // Must be called by all Widget constructors to initialize cross-platform data. >diff --git a/Source/WebKit/Platform/Logging.h b/Source/WebKit/Platform/Logging.h >index 1f773a711e616c885d131bf38a13d971601c12de..4b8a6d25e838b7cc6364c2dae8cac35b33d03b0e 100644 >--- a/Source/WebKit/Platform/Logging.h >+++ b/Source/WebKit/Platform/Logging.h >@@ -42,6 +42,7 @@ extern "C" { > #define WEBKIT2_LOG_CHANNELS(M) \ > M(AdClickAttribution) \ > M(Automation) \ >+ M(AutomationInteractions) \ > M(ActivityState) \ > M(BackForward) \ > M(CacheStorage) \ >diff --git a/Source/WebKit/UIProcess/Automation/Automation.json b/Source/WebKit/UIProcess/Automation/Automation.json >index cad53c7da1f450a0d605b6663d1ec1b8d0903090..9130f0ee67c43d1ee39e618861a276adf3a426d2 100644 >--- a/Source/WebKit/UIProcess/Automation/Automation.json >+++ b/Source/WebKit/UIProcess/Automation/Automation.json >@@ -32,7 +32,7 @@ > "description": "The coordinate system in which rects, points, and sizes are to be interpreted.", > "enum": [ > "Page", >- "LayoutViewport" >+ "Viewport" > ] > }, > { >diff --git a/Source/WebKit/UIProcess/Automation/SimulatedInputDispatcher.cpp b/Source/WebKit/UIProcess/Automation/SimulatedInputDispatcher.cpp >index 26a74e9c6541b652faeda6f3ad45e69df2bfdf3f..151c3336d6df3bf7c91d27eeb2d4deace27a4dd8 100644 >--- a/Source/WebKit/UIProcess/Automation/SimulatedInputDispatcher.cpp >+++ b/Source/WebKit/UIProcess/Automation/SimulatedInputDispatcher.cpp >@@ -324,7 +324,7 @@ void SimulatedInputDispatcher::transitionInputSourceToState(SimulatedInputSource > LOG(Automation, "SimulatedInputDispatcher[%p]: simulating TouchDown @ (%d, %d) for transition to %d.%d", this, b.location.value().x(), b.location.value().y(), m_keyframeIndex, m_inputSourceStateIndex); > m_client.simulateTouchInteraction(m_page, TouchInteraction::TouchDown, b.location.value(), WTF::nullopt, WTFMove(eventDispatchFinished)); > } else if (a.pressedMouseButton && !b.pressedMouseButton) { >- LOG(Automation, "SimulatedInputDispatcher[%p]: simulating LiftUp @ (%d, %d) for transition to %d.%d", this, a.location.value().x(), a.location.value().y(), m_keyframeIndex, m_inputSourceStateIndex); >+ LOG(Automation, "SimulatedInputDispatcher[%p]: simulating LiftUp @ (%d, %d) for transition to %d.%d", this, b.location.value().x(), b.location.value().y(), m_keyframeIndex, m_inputSourceStateIndex); > m_client.simulateTouchInteraction(m_page, TouchInteraction::LiftUp, b.location.value(), WTF::nullopt, WTFMove(eventDispatchFinished)); > } else if (a.location != b.location) { > LOG(Automation, "SimulatedInputDispatcher[%p]: simulating MoveTo from (%d, %d) to (%d, %d) for transition to %d.%d", this, a.location.value().x(), a.location.value().y(), b.location.value().x(), b.location.value().y(), m_keyframeIndex, m_inputSourceStateIndex); >diff --git a/Source/WebKit/UIProcess/Automation/WebAutomationSession.cpp b/Source/WebKit/UIProcess/Automation/WebAutomationSession.cpp >index 018bf91acffa0076cc488a58a26fd1cb781142d7..780ecb2f572467038eb4cf7b278153d9a4a98a3c 100644 >--- a/Source/WebKit/UIProcess/Automation/WebAutomationSession.cpp >+++ b/Source/WebKit/UIProcess/Automation/WebAutomationSession.cpp >@@ -1477,11 +1477,10 @@ void WebAutomationSession::viewportInViewCenterPointOfElement(WebPageProxy& page > #if ENABLE(WEBDRIVER_MOUSE_INTERACTIONS) > void WebAutomationSession::simulateMouseInteraction(WebPageProxy& page, MouseInteraction interaction, WebMouseEvent::Button mouseButton, const WebCore::IntPoint& locationInViewport, CompletionHandler<void(Optional<AutomationCommandError>)>&& completionHandler) > { >- WebCore::IntPoint locationInView = WebCore::IntPoint(locationInViewport.x(), locationInViewport.y() + page.topContentInset()); >- page.getWindowFrameWithCallback([this, protectedThis = makeRef(*this), completionHandler = WTFMove(completionHandler), page = makeRef(page), interaction, mouseButton, locationInView](WebCore::FloatRect windowFrame) mutable { >- auto clippedX = std::min(std::max(0.0f, (float)locationInView.x()), windowFrame.size().width()); >- auto clippedY = std::min(std::max(0.0f, (float)locationInView.y()), windowFrame.size().height()); >- if (clippedX != locationInView.x() || clippedY != locationInView.y()) { >+ page.getWindowFrameWithCallback([this, protectedThis = makeRef(*this), completionHandler = WTFMove(completionHandler), page = makeRef(page), interaction, mouseButton, locationInViewport](WebCore::FloatRect windowFrame) mutable { >+ auto clippedX = std::min(std::max(0.0f, (float)locationInViewport.x()), windowFrame.size().width()); >+ auto clippedY = std::min(std::max(0.0f, (float)locationInViewport.y()), windowFrame.size().height()); >+ if (clippedX != locationInViewport.x() || clippedY != locationInViewport.y()) { > completionHandler(AUTOMATION_COMMAND_ERROR_WITH_NAME(TargetOutOfBounds)); > return; > } >@@ -1496,7 +1495,7 @@ void WebAutomationSession::simulateMouseInteraction(WebPageProxy& page, MouseInt > callbackInMap(AUTOMATION_COMMAND_ERROR_WITH_NAME(Timeout)); > callbackInMap = WTFMove(mouseEventsFlushedCallback); > >- platformSimulateMouseInteraction(page, interaction, mouseButton, locationInView, OptionSet<WebEvent::Modifier>::fromRaw(m_currentModifiers)); >+ platformSimulateMouseInteraction(page, interaction, mouseButton, locationInViewport, OptionSet<WebEvent::Modifier>::fromRaw(m_currentModifiers)); > > // If the event does not hit test anything in the window, then it may not have been delivered. > if (callbackInMap && !page->isProcessingMouseEvents()) { >@@ -1513,7 +1512,8 @@ void WebAutomationSession::simulateMouseInteraction(WebPageProxy& page, MouseInt > void WebAutomationSession::simulateTouchInteraction(WebPageProxy& page, TouchInteraction interaction, const WebCore::IntPoint& locationInViewport, Optional<Seconds> duration, CompletionHandler<void(Optional<AutomationCommandError>)>&& completionHandler) > { > #if PLATFORM(IOS_FAMILY) >- if (!page.unobscuredContentRect().contains(locationInViewport)) { >+ WebCore::FloatRect visualViewportBounds = WebCore::FloatRect({ }, page.unobscuredContentRect().size()); >+ if (!visualViewportBounds.contains(locationInViewport)) { > completionHandler(AUTOMATION_COMMAND_ERROR_WITH_NAME(TargetOutOfBounds)); > return; > } >diff --git a/Source/WebKit/UIProcess/Automation/ios/WebAutomationSessionIOS.mm b/Source/WebKit/UIProcess/Automation/ios/WebAutomationSessionIOS.mm >index 48ba280e3a84c1d3a87c4358438b9cbe099bcc99..0b103ab99cf3a4346a3a97222703a0d4118e91ad 100644 >--- a/Source/WebKit/UIProcess/Automation/ios/WebAutomationSessionIOS.mm >+++ b/Source/WebKit/UIProcess/Automation/ios/WebAutomationSessionIOS.mm >@@ -28,6 +28,7 @@ > > #if PLATFORM(IOS_FAMILY) > >+#import "Logging.h" > #import "NativeWebKeyboardEvent.h" > #import "WebAutomationSessionMacros.h" > #import "WebPageProxy.h" >@@ -175,15 +176,34 @@ void WebAutomationSession::platformSimulateKeySequence(WebPageProxy& page, const > #endif // ENABLE(WEBDRIVER_KEYBOARD_INTERACTIONS) > > #if ENABLE(WEBDRIVER_TOUCH_INTERACTIONS) >+#if !LOG_DISABLED >+static TextStream& operator<<(TextStream& ts, TouchInteraction interaction) >+{ >+ switch (interaction) { >+ case TouchInteraction::TouchDown: >+ ts << "TouchDown"; >+ break; >+ case TouchInteraction::MoveTo: >+ ts << "MoveTo"; >+ break; >+ case TouchInteraction::LiftUp: >+ ts << "LiftUp"; >+ break; >+ } >+ return ts; >+} >+#endif // !LOG_DISABLED >+ > void WebAutomationSession::platformSimulateTouchInteraction(WebPageProxy& page, TouchInteraction interaction, const WebCore::IntPoint& locationInViewport, Optional<Seconds> duration, AutomationCompletionHandler&& completionHandler) > { > WebCore::IntPoint locationOnScreen = page.syncRootViewToScreen(IntRect(locationInViewport, IntSize())).location(); >- _WKTouchEventGenerator *generator = [_WKTouchEventGenerator sharedTouchEventGenerator]; >+ LOG_WITH_STREAM(AutomationInteractions, stream << "platformSimulateTouchInteraction: interaction=" << interaction << ", locationInViewport=" << locationInViewport << ", locationOnScreen=" << locationOnScreen << ", duration=" << duration.valueOr(0_s).seconds()); > > auto interactionFinished = makeBlockPtr([completionHandler = WTFMove(completionHandler)] () mutable { > completionHandler(WTF::nullopt); > }); >- >+ >+ _WKTouchEventGenerator *generator = [_WKTouchEventGenerator sharedTouchEventGenerator]; > switch (interaction) { > case TouchInteraction::TouchDown: > [generator touchDown:locationOnScreen completionBlock:interactionFinished.get()]; >diff --git a/Source/WebKit/WebProcess/Automation/WebAutomationSessionProxy.cpp b/Source/WebKit/WebProcess/Automation/WebAutomationSessionProxy.cpp >index 9c31b829bb5a454e000243f13de8c198713e3744..d8af265c323a8afae90041146268580a923b344d 100644 >--- a/Source/WebKit/WebProcess/Automation/WebAutomationSessionProxy.cpp >+++ b/Source/WebKit/WebProcess/Automation/WebAutomationSessionProxy.cpp >@@ -477,35 +477,50 @@ void WebAutomationSessionProxy::focusFrame(uint64_t pageID, uint64_t frameID) > coreDOMWindow->focus(true); > } > >-static Optional<WebCore::FloatPoint> elementInViewClientCenterPoint(WebCore::Element& element, bool& isObscured) >+using ErrorType = Inspector::Protocol::Automation::ErrorMessage; >+using Result = Expected<WebCore::FloatPoint, ErrorType>; >+using UnexpectedResult = Unexpected<ErrorType>; >+ >+// Returns the in-view center point of an element in frame client coordinates. >+static Result elementInViewCenterPoint(WebCore::FrameView& frameView, WebCore::Element& element, bool& isObscured) > { >+ // NOTE: All local variables for points and rects in this function use frame client coordinates. >+ > // §11.1 Element Interactability. > // https://www.w3.org/TR/webdriver/#dfn-in-view-center-point >- auto* clientRect = element.getClientRects()->item(0); >- if (!clientRect) >- return WTF::nullopt; >- >- auto clientCenterPoint = WebCore::FloatPoint::narrowPrecision(0.5 * (clientRect->left() + clientRect->right()), 0.5 * (clientRect->top() + clientRect->bottom())); >- auto elementList = element.treeScope().elementsFromPoint(clientCenterPoint.x(), clientCenterPoint.y()); >- if (elementList.isEmpty()) { >- // An element is obscured if the pointer-interactable paint tree at its center point is empty, >- // or the first element in this tree is not an inclusive descendant of itself. >- // https://w3c.github.io/webdriver/webdriver-spec.html#dfn-obscured >- isObscured = true; >- return clientCenterPoint; >- } >- >+ Ref<DOMRectList> elementRects = element.getClientRects(); >+ auto* firstElementRect = elementRects->item(0); >+ if (!firstElementRect) >+ return UnexpectedResult(ErrorType::ElementNotInteractable); >+ >+ // The W3C WebDriver specification does not explicitly intersect the element with the visual viewport. >+ // Do that here so that the IVCP for an element larger than the viewport is within the viewport. >+ // See spec bug here: https://github.com/w3c/webdriver/issues/1402 >+ FloatRect viewportRect = frameView.documentToClientRect(frameView.visualViewportRect()); >+ FloatRect elementRect = FloatRect(firstElementRect->x(), firstElementRect->y(), firstElementRect->width(), firstElementRect->height()); >+ FloatRect visiblePortionOfElementRect = intersection(viewportRect, elementRect); >+ >+ if (visiblePortionOfElementRect.isEmpty()) >+ return UnexpectedResult(ErrorType::TargetOutOfBounds); >+ >+ FloatPoint elementInViewCenterPoint = visiblePortionOfElementRect.center(); >+ auto elementList = element.treeScope().elementsFromPoint(elementInViewCenterPoint.x(), elementInViewCenterPoint.y()); > auto index = elementList.findMatching([&element] (auto& item) { return item.get() == &element; }); >- if (index == notFound) >- return WTF::nullopt; >- >- if (index) { >- // Element is not the first one in the list. >- auto firstElement = elementList[0]; >- isObscured = !firstElement->isDescendantOf(element); >- } >- >- return clientCenterPoint; >+ if (elementList.isEmpty() || index == notFound) { >+ // We hit this case if the element is visibility:hidden or opacity:0, in which case it will not hit test >+ // at the calculated IVCP. An element is technically not "in view" if it is not within its own paint/hit test tree, >+ // so it cannot have an in-view center point either. And without an IVCP, the definition of 'obscured' makes no sense. >+ // See <https://w3c.github.io/webdriver/webdriver-spec.html#dfn-in-view>. >+ return UnexpectedResult(ErrorType::ElementNotInteractable); >+ } >+ >+ // Check the case where a non-descendant element hit tests before the target element. For example, a child <option> >+ // of a <select> does not obscure the <select>, but two sibling <div> that overlap at the IVCP will obscure each other. >+ auto firstElement = elementList[0]; >+ >+ // Node::isDescendantOf() is not self-inclusive, so that is explicitly checked here. >+ isObscured = firstElement.get() != &element && !firstElement->isDescendantOf(element); >+ return elementInViewCenterPoint; > } > > static WebCore::Element* containerElementForElement(WebCore::Element& element) >@@ -534,6 +549,32 @@ static WebCore::Element* containerElementForElement(WebCore::Element& element) > return &element; > } > >+static WebCore::FloatRect convertRectFromFrameClientToRootView(FrameView* frameView, WebCore::FloatRect clientRect) >+{ >+ if (!frameView->delegatesScrolling()) >+ return frameView->contentsToRootView(frameView->clientToDocumentRect(clientRect)); >+ >+ auto& frame = frameView->frame(); >+ float scaleFactor = frame.pageZoomFactor() * frame.frameScaleFactor(); >+ WebCore::FloatRect absoluteRect = clientRect; >+ absoluteRect.scale(scaleFactor); >+ absoluteRect.moveBy(frameView->contentsScrollPosition()); >+ return absoluteRect; >+} >+ >+static WebCore::FloatPoint convertPointFromFrameClientToRootView(FrameView* frameView, WebCore::FloatPoint clientPoint) >+{ >+ if (!frameView->delegatesScrolling()) >+ return frameView->contentsToRootView(frameView->clientToDocumentPoint(clientPoint)); >+ >+ auto& frame = frameView->frame(); >+ float scaleFactor = frame.pageZoomFactor() * frame.frameScaleFactor(); >+ WebCore::FloatPoint absolutePoint = clientPoint; >+ absolutePoint.scale(scaleFactor); >+ absolutePoint.moveBy(frameView->contentsScrollPosition()); >+ return absolutePoint; >+} >+ > void WebAutomationSessionProxy::computeElementLayout(uint64_t pageID, uint64_t frameID, String nodeHandle, bool scrollIntoViewIfNeeded, CoordinateSystem coordinateSystem, CompletionHandler<void(Optional<String>, WebCore::IntRect, Optional<WebCore::IntPoint>, bool)>&& completionHandler) > { > WebPage* page = WebProcess::singleton().webPage(pageID); >@@ -546,6 +587,7 @@ void WebAutomationSessionProxy::computeElementLayout(uint64_t pageID, uint64_t f > String frameNotFoundErrorType = Inspector::Protocol::AutomationHelpers::getEnumConstantValue(Inspector::Protocol::Automation::ErrorMessage::FrameNotFound); > String nodeNotFoundErrorType = Inspector::Protocol::AutomationHelpers::getEnumConstantValue(Inspector::Protocol::Automation::ErrorMessage::NodeNotFound); > String notImplementedErrorType = Inspector::Protocol::AutomationHelpers::getEnumConstantValue(Inspector::Protocol::Automation::ErrorMessage::NotImplemented); >+ String elementNotInteractableErrorType = Inspector::Protocol::AutomationHelpers::getEnumConstantValue(Inspector::Protocol::Automation::ErrorMessage::ElementNotInteractable); > > WebFrame* frame = frameID ? WebProcess::singleton().webFrame(frameID) : page->mainWebFrame(); > if (!frame || !frame->coreFrame() || !frame->coreFrame()->view()) { >@@ -569,35 +611,49 @@ void WebAutomationSessionProxy::computeElementLayout(uint64_t pageID, uint64_t f > > WebCore::FrameView* frameView = frame->coreFrame()->view(); > WebCore::FrameView* mainView = frame->coreFrame()->mainFrame().view(); >- WebCore::IntRect frameElementBounds = roundedIntRect(coreElement->boundingClientRect()); >- WebCore::IntRect rootElementBounds = mainView->rootViewToContents(frameView->contentsToRootView(frameElementBounds)); >+ WebCore::FloatRect elementBoundsInFrameClientCoordinates = coreElement->boundingClientRect(); >+ WebCore::FloatRect elementBoundsInRootCoordinates = convertRectFromFrameClientToRootView(frameView, elementBoundsInFrameClientCoordinates); >+ WebCore::FloatRect elementBoundsInMainFrameDocumentCoordinates = mainView->absoluteToDocumentRect(mainView->rootViewToContents(elementBoundsInRootCoordinates)); >+ WebCore::FloatRect elementBoundsInMainFrameViewportCoordinates = mainView->absoluteToLayoutViewportRect(elementBoundsInRootCoordinates); >+ > WebCore::IntRect resultElementBounds; >+ Optional<WebCore::IntPoint> resultInViewCenterPoint = WTF::nullopt; > switch (coordinateSystem) { > case CoordinateSystem::Page: >- resultElementBounds = WebCore::IntRect(mainView->clientToDocumentRect(WebCore::FloatRect(rootElementBounds))); >+ resultElementBounds = enclosingIntRect(elementBoundsInMainFrameDocumentCoordinates); > break; > case CoordinateSystem::LayoutViewport: >- // The element bounds are already in client coordinates. >- resultElementBounds = WebCore::IntRect(mainView->clientToLayoutViewportRect(WebCore::FloatRect(rootElementBounds))); >+ resultElementBounds = enclosingIntRect(elementBoundsInMainFrameViewportCoordinates); > break; > } > >- Optional<WebCore::IntPoint> resultInViewCenterPoint; >+ // If an <option> or <optgroup> does not have an associated <select> or <datalist> element, then give up. >+ if (!containerElement) { >+ completionHandler(elementNotInteractableErrorType, resultElementBounds, resultInViewCenterPoint, false); >+ return; >+ } >+ > bool isObscured = false; >- if (containerElement) { >- Optional<WebCore::FloatPoint> frameInViewCenterPoint = elementInViewClientCenterPoint(*containerElement, isObscured); >- if (frameInViewCenterPoint.hasValue()) { >- WebCore::IntPoint rootInViewCenterPoint = mainView->rootViewToContents(frameView->contentsToRootView(WebCore::IntPoint(frameInViewCenterPoint.value()))); >- switch (coordinateSystem) { >- case CoordinateSystem::Page: >- resultInViewCenterPoint = WebCore::IntPoint(mainView->clientToDocumentPoint(rootInViewCenterPoint)); >- break; >- case CoordinateSystem::LayoutViewport: >- // The point is already in client coordinates. >- resultInViewCenterPoint = WebCore::IntPoint(mainView->clientToLayoutViewportPoint(rootInViewCenterPoint)); >- break; >- } >- } >+ auto inViewCenterPointInFrameClientCoordinates = elementInViewCenterPoint(*frameView, *containerElement, isObscured); >+ >+ // If the element is not within the bounds of the viewport, then there will be no IVCP and the element is not interactable. >+ if (!inViewCenterPointInFrameClientCoordinates.has_value()) { >+ completionHandler(Inspector::Protocol::AutomationHelpers::getEnumConstantValue(inViewCenterPointInFrameClientCoordinates.error()), resultElementBounds, resultInViewCenterPoint, false); >+ return; >+ } >+ >+ WebCore::FloatPoint inViewCenterPointInRootCoordinates = convertPointFromFrameClientToRootView(frameView, inViewCenterPointInFrameClientCoordinates.value()); >+ WebCore::FloatPoint inViewCenterPointInMainFrameDocumentCoordinates = mainView->absoluteToDocumentPoint(inViewCenterPointInRootCoordinates); >+ WebCore::FloatPoint inViewCenterPointInMainFrameViewportCoordinates = mainView->absoluteToLayoutViewportPoint(inViewCenterPointInRootCoordinates); >+ switch (coordinateSystem) { >+ case CoordinateSystem::Page: >+ resultInViewCenterPoint = roundedIntPoint(inViewCenterPointInMainFrameDocumentCoordinates); >+ break; >+ case CoordinateSystem::LayoutViewport: >+ resultInViewCenterPoint = roundedIntPoint(inViewCenterPointInMainFrameViewportCoordinates); >+ break; >+ default: >+ RELEASE_ASSERT_NOT_REACHED(); > } > > completionHandler(WTF::nullopt, resultElementBounds, resultInViewCenterPoint, isObscured);
<b>You cannot view the attachment while viewing its details because your browser does not support IFRAMEs. <a href="attachment.cgi?id=369207">View the attachment on a separate page</a>.</b>
View Attachment As Diff
View Attachment As Raw
Actions:
View
|
Formatted Diff
|
Diff
Attachments on
bug 195696
:
364682
|
367478
|
369207
|
369241
|
369584
|
369659
|
369676
|
369812
|
369913
|
369936