WebKit Bugzilla
Attachment 369584 Details for
Bug 195696
: Web Automation: elements larger than the viewport have incorrect in-view center point
Home
|
New
|
Browse
|
Search
|
[?]
|
Reports
|
Requests
|
Help
|
New Account
|
Log In
Remember
[x]
|
Forgot Password
Login:
[x]
[patch]
Patch
bug-195696-20190510130604.patch (text/plain), 37.90 KB, created by
Devin Rousso
on 2019-05-10 13:06:05 PDT
(
hide
)
Description:
Patch
Filename:
MIME Type:
Creator:
Devin Rousso
Created:
2019-05-10 13:06:05 PDT
Size:
37.90 KB
patch
obsolete
>diff --git a/Source/WebCore/ChangeLog b/Source/WebCore/ChangeLog >index 227dd90f3d2d7b9e43df6a822c56314aae81f7bd..0de2fe29d7195aab1ecdd134ba600a0f290b517c 100644 >--- a/Source/WebCore/ChangeLog >+++ b/Source/WebCore/ChangeLog >@@ -1,3 +1,39 @@ >+2019-05-10 Devin Rousso <drousso@apple.com> >+ >+ Web Automation: elements larger than the viewport have incorrect in-view center point >+ https://bugs.webkit.org/show_bug.cgi?id=195696 >+ <rdar://problem/48737122> >+ >+ Reviewed by NOBODY (OOPS!). >+ >+ Original patch by Brian Burg <bburg@apple.com>. >+ >+ Some conversion methods do not exist for FloatRect/FloatPoint. >+ Fill them in as needed, and export some symbols used by WebDriver >+ code to compute an element's in-view center point in various coordinate systems. >+ >+ * dom/DOMRectReadOnly.h: >+ (WebCore::DOMRectReadOnly::operator FloatRect const): Added. >+ * dom/TreeScope.h: >+ * dom/TreeScope.cpp: >+ (WebCore::TreeScope::elementsFromPoint): Added. >+ * page/FrameView.h: >+ * page/FrameView.cpp: >+ (WebCore::FrameView::absoluteToLayoutViewportPoint const): Added. >+ (WebCore::FrameView::layoutViewportToAbsoluteRect const): Added. >+ (WebCore::FrameView::absoluteToLayoutViewportRect const): Added. >+ * platform/ScrollView.h: >+ * platform/ScrollView.cpp: >+ (WebCore::ScrollView::viewToContents const): Added. >+ (WebCore::ScrollView::contentsToView const): Added. >+ (WebCore::ScrollView::contentsToRootView const): Added. >+ * platform/Widget.h: >+ * platform/Widget.cpp: >+ (WebCore::Widget::convertToRootView const): Added. >+ (WebCore::Widget::convertFromRootView const): Added. >+ (WebCore::Widget::convertToContainingView const): Added. >+ (WebCore::Widget::convertFromContainingView const): Added. >+ > 2019-05-10 Antti Koivisto <antti@apple.com> > > Event region generation needs to know about backing-sharing >diff --git a/Source/WebKit/ChangeLog b/Source/WebKit/ChangeLog >index 8382293787323c64c1fc98ca65cc07175107f05d..d52cfe899485133220dfc58abd9ea8cb3d5b7eba 100644 >--- a/Source/WebKit/ChangeLog >+++ b/Source/WebKit/ChangeLog >@@ -1,3 +1,50 @@ >+2019-05-10 Devin Rousso <drousso@apple.com> >+ >+ Web Automation: elements larger than the viewport have incorrect in-view center point >+ https://bugs.webkit.org/show_bug.cgi?id=195696 >+ <rdar://problem/48737122> >+ >+ Reviewed by NOBODY (OOPS!). >+ >+ Original patch by Brian Burg <bburg@apple.com>. >+ >+ This seems to be an omission in the specification. While it does mention that the in-view >+ center point (IVCP) must be within the viewport, the algorithm never intersects the element >+ bounding box with the viewport rect. >+ >+ * Platform/Logging.h: Add logging channel to dump fully resolved interaction details. >+ * UIProcess/Automation/WebAutomationSession.cpp: >+ (WebKit::WebAutomationSession::simulateMouseInteraction): >+ Rename locationInView -> locationInViewport. >+ >+ (WebKit::WebAutomationSession::simulateTouchInteraction): >+ This code is incorrect. The unobscuredContentRect is in screen coordinates, but >+ we are trying to see if (x, y) is outside the size of the viewport assumed to be at (0, 0). >+ Grab the visual viewport rect and see if the location exceeds the viewport size. >+ >+ * UIProcess/Automation/ios/WebAutomationSessionIOS.mm: >+ (WebKit::operator<<): Add logging helper for TouchInteraction enum. >+ (WebKit::WebAutomationSession::platformSimulateTouchInteraction): Move local variable. >+ >+ * WebProcess/Automation/WebAutomationSessionProxy.cpp: >+ (WebKit::elementInViewClientCenterPoint): Deleted. >+ (WebKit::elementInViewCenterPoint): Added. >+ Pass in the main FrameView as we need to intersect the element rect and the viewport rect >+ before finding the center point of the part of the element that's visible in the viewport. >+ >+ (WebKit::WebAutomationSessionProxy::computeElementLayout): >+ This code is incorrect. For CoordinateSystem::LayoutViewport, coordinates should be in root view coordinates >+ so that it can be later converted to screen and synthesized as a HID event in screen coordinates. >+ >+ (WebKit::convertRectFromFrameClientToRootView): Added. >+ (WebKit::convertPointFromFrameClientToRootView): Added. >+ Added helpers to properly account for scroll contents position on iOS. >+ >+ * UIProcess/Automation/SimulatedInputDispatcher.cpp: >+ (WebKit::SimulatedInputDispatcher::transitionInputSourceToState): Fix a typo in logging. >+ >+ * UIProcess/Automation/Automation.json: Simplify enum name. >+ > 2019-05-10 Chris Fleizach <cfleizach@apple.com> > > AX: Crash at WebKit: WebKit::WebSpeechSynthesisClient::speak >diff --git a/Source/WebCore/dom/DOMRectReadOnly.h b/Source/WebCore/dom/DOMRectReadOnly.h >index e3a10daa64739d64d53ac6d781cfd80ecc7fe28e..b31a38413023307dd9cd244951ac65a90c6b9cfe 100644 >--- a/Source/WebCore/dom/DOMRectReadOnly.h >+++ b/Source/WebCore/dom/DOMRectReadOnly.h >@@ -26,6 +26,7 @@ > #pragma once > > #include "DOMRectInit.h" >+#include "FloatRect.h" > #include "ScriptWrappable.h" > #include <wtf/IsoMalloc.h> > #include <wtf/MathExtras.h> >@@ -51,6 +52,8 @@ public: > double bottom() const { return WTF::nanPropagatingMax(m_y, m_y + m_height); } > double left() const { return WTF::nanPropagatingMin(m_x, m_x + m_width); } > >+ operator FloatRect() const { return FloatRect(m_x, m_y, m_width, m_height); } >+ > protected: > DOMRectReadOnly(double x, double y, double width, double height) > : m_x(x) >diff --git a/Source/WebCore/dom/TreeScope.cpp b/Source/WebCore/dom/TreeScope.cpp >index 7ba3578a4f74efa7429ad65983f1d15ea703c352..cf9276c60814e3091fa61d120786fde5fc2fbfbd 100644 >--- a/Source/WebCore/dom/TreeScope.cpp >+++ b/Source/WebCore/dom/TreeScope.cpp >@@ -441,6 +441,11 @@ Vector<RefPtr<Element>> TreeScope::elementsFromPoint(double clientX, double clie > return elements; > } > >+Vector<RefPtr<Element>> TreeScope::elementsFromPoint(const FloatPoint& p) >+{ >+ return elementsFromPoint(p.x(), p.y()); >+} >+ > Element* TreeScope::findAnchor(const String& name) > { > if (name.isEmpty()) >diff --git a/Source/WebCore/dom/TreeScope.h b/Source/WebCore/dom/TreeScope.h >index b164bd9ea83972751e718b77b1cf41ef94c7730e..efb4425fbee3e997b4747fb2047c7493a16013db 100644 >--- a/Source/WebCore/dom/TreeScope.h >+++ b/Source/WebCore/dom/TreeScope.h >@@ -37,6 +37,7 @@ namespace WebCore { > class ContainerNode; > class Document; > class Element; >+class FloatPoint; > class HTMLImageElement; > class HTMLLabelElement; > class HTMLMapElement; >@@ -95,6 +96,7 @@ public: > > WEBCORE_EXPORT RefPtr<Element> elementFromPoint(double clientX, double clientY); > WEBCORE_EXPORT Vector<RefPtr<Element>> elementsFromPoint(double clientX, double clientY); >+ WEBCORE_EXPORT Vector<RefPtr<Element>> elementsFromPoint(const FloatPoint&); > > // Find first anchor with the given name. > // First searches for an element with the given ID, but if that fails, then looks >diff --git a/Source/WebCore/page/FrameView.cpp b/Source/WebCore/page/FrameView.cpp >index 50813d83a39b93f67b3ccaf1b632f05412860ec5..10016c293407fcc737046e6aa67e5a9b10027cde 100644 >--- a/Source/WebCore/page/FrameView.cpp >+++ b/Source/WebCore/page/FrameView.cpp >@@ -4791,6 +4791,14 @@ FloatPoint FrameView::clientToDocumentPoint(FloatPoint point) const > return point; > } > >+FloatPoint FrameView::absoluteToLayoutViewportPoint(FloatPoint p) const >+{ >+ ASSERT(frame().settings().visualViewportEnabled()); >+ p.scale(1 / frame().frameScaleFactor()); >+ p.moveBy(-layoutViewportRect().location()); >+ return p; >+} >+ > FloatPoint FrameView::layoutViewportToAbsolutePoint(FloatPoint p) const > { > ASSERT(frame().settings().visualViewportEnabled()); >@@ -4798,6 +4806,22 @@ FloatPoint FrameView::layoutViewportToAbsolutePoint(FloatPoint p) const > return p.scaled(frame().frameScaleFactor()); > } > >+FloatRect FrameView::layoutViewportToAbsoluteRect(FloatRect rect) const >+{ >+ ASSERT(frame().settings().visualViewportEnabled()); >+ rect.moveBy(layoutViewportRect().location()); >+ rect.scale(frame().frameScaleFactor()); >+ return rect; >+} >+ >+FloatRect FrameView::absoluteToLayoutViewportRect(FloatRect rect) const >+{ >+ ASSERT(frame().settings().visualViewportEnabled()); >+ rect.scale(1 / frame().frameScaleFactor()); >+ rect.moveBy(-layoutViewportRect().location()); >+ return rect; >+} >+ > FloatRect FrameView::clientToLayoutViewportRect(FloatRect rect) const > { > ASSERT(frame().settings().visualViewportEnabled()); >diff --git a/Source/WebCore/page/FrameView.h b/Source/WebCore/page/FrameView.h >index 81d1cc62db70706913433b76fb03c04f92b66dde..d36e565645c3794e6a8513178d6edbff0cfdb9e8 100644 >--- a/Source/WebCore/page/FrameView.h >+++ b/Source/WebCore/page/FrameView.h >@@ -475,19 +475,23 @@ public: > float documentToAbsoluteScaleFactor(Optional<float> effectiveZoom = WTF::nullopt) const; > float absoluteToDocumentScaleFactor(Optional<float> effectiveZoom = WTF::nullopt) const; > >- FloatRect absoluteToDocumentRect(FloatRect, Optional<float> effectiveZoom = WTF::nullopt) const; >- FloatPoint absoluteToDocumentPoint(FloatPoint, Optional<float> effectiveZoom = WTF::nullopt) const; >+ WEBCORE_EXPORT FloatRect absoluteToDocumentRect(FloatRect, Optional<float> effectiveZoom = WTF::nullopt) const; >+ WEBCORE_EXPORT FloatPoint absoluteToDocumentPoint(FloatPoint, Optional<float> effectiveZoom = WTF::nullopt) const; > > FloatRect absoluteToClientRect(FloatRect, Optional<float> effectiveZoom = WTF::nullopt) const; > > FloatSize documentToClientOffset() const; >- FloatRect documentToClientRect(FloatRect) const; >+ WEBCORE_EXPORT FloatRect documentToClientRect(FloatRect) const; > FloatPoint documentToClientPoint(FloatPoint) const; > WEBCORE_EXPORT FloatRect clientToDocumentRect(FloatRect) const; > WEBCORE_EXPORT FloatPoint clientToDocumentPoint(FloatPoint) const; > >+ WEBCORE_EXPORT FloatPoint absoluteToLayoutViewportPoint(FloatPoint) const; > FloatPoint layoutViewportToAbsolutePoint(FloatPoint) const; > >+ WEBCORE_EXPORT FloatRect absoluteToLayoutViewportRect(FloatRect) const; >+ FloatRect layoutViewportToAbsoluteRect(FloatRect) const; >+ > // Unlike client coordinates, layout viewport coordinates are affected by page zoom. > WEBCORE_EXPORT FloatRect clientToLayoutViewportRect(FloatRect) const; > WEBCORE_EXPORT FloatPoint clientToLayoutViewportPoint(FloatPoint) const; >diff --git a/Source/WebCore/platform/ScrollView.cpp b/Source/WebCore/platform/ScrollView.cpp >index 255ff67f45686e97db110a216385c7a48f487356..61bb518bca1a95ea95bf978f89fd46114ecb5d40 100644 >--- a/Source/WebCore/platform/ScrollView.cpp >+++ b/Source/WebCore/platform/ScrollView.cpp >@@ -842,6 +842,22 @@ IntPoint ScrollView::contentsToView(const IntPoint& point) const > return point - toIntSize(documentScrollPositionRelativeToViewOrigin()); > } > >+FloatPoint ScrollView::viewToContents(const FloatPoint& point) const >+{ >+ if (delegatesScrolling()) >+ return point; >+ >+ return viewToContents(IntPoint(point)); >+} >+ >+FloatPoint ScrollView::contentsToView(const FloatPoint& point) const >+{ >+ if (delegatesScrolling()) >+ return point; >+ >+ return contentsToView(IntPoint(point)); >+} >+ > IntRect ScrollView::viewToContents(IntRect rect) const > { > if (delegatesScrolling()) >@@ -908,6 +924,11 @@ IntPoint ScrollView::contentsToRootView(const IntPoint& contentsPoint) const > return convertToRootView(contentsToView(contentsPoint)); > } > >+FloatPoint ScrollView::contentsToRootView(const FloatPoint& contentsPoint) const >+{ >+ return convertToRootView(contentsToView(contentsPoint)); >+} >+ > IntRect ScrollView::rootViewToContents(const IntRect& rootViewRect) const > { > return viewToContents(convertFromRootView(rootViewRect)); >@@ -918,6 +939,11 @@ FloatRect ScrollView::rootViewToContents(const FloatRect& rootViewRect) const > return viewToContents(convertFromRootView(rootViewRect)); > } > >+FloatRect ScrollView::contentsToRootView(const FloatRect& contentsRect) const >+{ >+ return convertToRootView(contentsToView(contentsRect)); >+} >+ > IntPoint ScrollView::rootViewToTotalContents(const IntPoint& rootViewPoint) const > { > if (delegatesScrolling()) >diff --git a/Source/WebCore/platform/ScrollView.h b/Source/WebCore/platform/ScrollView.h >index 8eb46831cc1c85ee1962675334ec1e35fb01c68b..7d824c48aaa51ab218f7373aa9d9e36b675f45dc 100644 >--- a/Source/WebCore/platform/ScrollView.h >+++ b/Source/WebCore/platform/ScrollView.h >@@ -230,7 +230,7 @@ public: > int scrollY() const { return scrollPosition().y(); } > > // Scroll position used by web-exposed features (has legacy iOS behavior). >- IntPoint contentsScrollPosition() const; >+ WEBCORE_EXPORT IntPoint contentsScrollPosition() const; > void setContentsScrollPosition(const IntPoint&); > > #if PLATFORM(IOS_FAMILY) >@@ -279,13 +279,18 @@ public: > > WEBCORE_EXPORT IntPoint rootViewToContents(const IntPoint&) const; > WEBCORE_EXPORT IntPoint contentsToRootView(const IntPoint&) const; >+ WEBCORE_EXPORT FloatPoint contentsToRootView(const FloatPoint&) const; > WEBCORE_EXPORT IntRect rootViewToContents(const IntRect&) const; > WEBCORE_EXPORT IntRect contentsToRootView(const IntRect&) const; > WEBCORE_EXPORT FloatRect rootViewToContents(const FloatRect&) const; >+ WEBCORE_EXPORT FloatRect contentsToRootView(const FloatRect&) const; > > IntPoint viewToContents(const IntPoint&) const; > IntPoint contentsToView(const IntPoint&) const; > >+ FloatPoint viewToContents(const FloatPoint&) const; >+ FloatPoint contentsToView(const FloatPoint&) const; >+ > IntRect viewToContents(IntRect) const; > IntRect contentsToView(IntRect) const; > >diff --git a/Source/WebCore/platform/Widget.cpp b/Source/WebCore/platform/Widget.cpp >index fc941f470412e9b5363b9d1d57a61fbc7209192e..48570aa681c90c793ea3f27788764c9f56527cd5 100644 >--- a/Source/WebCore/platform/Widget.cpp >+++ b/Source/WebCore/platform/Widget.cpp >@@ -95,6 +95,15 @@ IntRect Widget::convertToRootView(const IntRect& localRect) const > return localRect; > } > >+FloatRect Widget::convertToRootView(const FloatRect& localRect) const >+{ >+ if (const ScrollView* parentScrollView = parent()) { >+ FloatRect parentRect = convertToContainingView(localRect); >+ return parentScrollView->convertToRootView(parentRect); >+ } >+ return localRect; >+} >+ > IntPoint Widget::convertFromRootView(const IntPoint& rootPoint) const > { > if (const ScrollView* parentScrollView = parent()) { >@@ -113,6 +122,25 @@ IntPoint Widget::convertToRootView(const IntPoint& localPoint) const > return localPoint; > } > >+ >+FloatPoint Widget::convertFromRootView(const FloatPoint& rootPoint) const >+{ >+ if (const ScrollView* parentScrollView = parent()) { >+ FloatPoint parentPoint = parentScrollView->convertFromRootView(rootPoint); >+ return convertFromContainingView(parentPoint); >+ } >+ return rootPoint; >+} >+ >+FloatPoint Widget::convertToRootView(const FloatPoint& localPoint) const >+{ >+ if (const ScrollView* parentScrollView = parent()) { >+ FloatPoint parentPoint = convertToContainingView(localPoint); >+ return parentScrollView->convertToRootView(parentPoint); >+ } >+ return localPoint; >+} >+ > IntRect Widget::convertFromContainingWindow(const IntRect& windowRect) const > { > if (const ScrollView* parentScrollView = parent()) { >@@ -204,6 +232,11 @@ IntRect Widget::convertFromContainingView(const IntRect& parentRect) const > return parentRect; > } > >+FloatRect Widget::convertToContainingView(const FloatRect& localRect) const >+{ >+ return convertToContainingView(IntRect(localRect)); >+} >+ > FloatRect Widget::convertFromContainingView(const FloatRect& parentRect) const > { > return convertFromContainingView(IntRect(parentRect)); >@@ -225,6 +258,16 @@ IntPoint Widget::convertFromContainingView(const IntPoint& parentPoint) const > return parentPoint; > } > >+FloatPoint Widget::convertToContainingView(const FloatPoint& localPoint) const >+{ >+ return convertToContainingView(IntPoint(localPoint)); >+} >+ >+FloatPoint Widget::convertFromContainingView(const FloatPoint& parentPoint) const >+{ >+ return convertFromContainingView(IntPoint(parentPoint)); >+} >+ > #if !PLATFORM(COCOA) && !PLATFORM(GTK) && !PLATFORM(WIN) > > Widget::~Widget() >diff --git a/Source/WebCore/platform/Widget.h b/Source/WebCore/platform/Widget.h >index 211126544079a71667aea3f87cc2e844eeeca8e9..2aeaeab9309fba91342fca59e799c7dd842104d4 100644 >--- a/Source/WebCore/platform/Widget.h >+++ b/Source/WebCore/platform/Widget.h >@@ -150,11 +150,15 @@ public: > WEBCORE_EXPORT IntRect convertToRootView(const IntRect&) const; > IntRect convertFromRootView(const IntRect&) const; > >+ FloatRect convertToRootView(const FloatRect&) const; > FloatRect convertFromRootView(const FloatRect&) const; > > IntPoint convertToRootView(const IntPoint&) const; > IntPoint convertFromRootView(const IntPoint&) const; > >+ FloatPoint convertToRootView(const FloatPoint&) const; >+ FloatPoint convertFromRootView(const FloatPoint&) const; >+ > // It is important for cross-platform code to realize that Mac has flipped coordinates. Therefore any code > // that tries to convert the location of a rect using the point-based convertFromContainingWindow will end > // up with an inaccurate rect. Always make sure to use the rect-based convertFromContainingWindow method >@@ -186,9 +190,12 @@ public: > // Virtual methods to convert points to/from the containing ScrollView > WEBCORE_EXPORT virtual IntRect convertToContainingView(const IntRect&) const; > WEBCORE_EXPORT virtual IntRect convertFromContainingView(const IntRect&) const; >+ WEBCORE_EXPORT virtual FloatRect convertToContainingView(const FloatRect&) const; > WEBCORE_EXPORT virtual FloatRect convertFromContainingView(const FloatRect&) const; > WEBCORE_EXPORT virtual IntPoint convertToContainingView(const IntPoint&) const; > WEBCORE_EXPORT virtual IntPoint convertFromContainingView(const IntPoint&) const; >+ WEBCORE_EXPORT virtual FloatPoint convertToContainingView(const FloatPoint&) const; >+ WEBCORE_EXPORT virtual FloatPoint convertFromContainingView(const FloatPoint&) const; > > private: > void init(PlatformWidget); // Must be called by all Widget constructors to initialize cross-platform data. >diff --git a/Source/WebKit/Platform/Logging.h b/Source/WebKit/Platform/Logging.h >index 1f773a711e616c885d131bf38a13d971601c12de..4b8a6d25e838b7cc6364c2dae8cac35b33d03b0e 100644 >--- a/Source/WebKit/Platform/Logging.h >+++ b/Source/WebKit/Platform/Logging.h >@@ -42,6 +42,7 @@ extern "C" { > #define WEBKIT2_LOG_CHANNELS(M) \ > M(AdClickAttribution) \ > M(Automation) \ >+ M(AutomationInteractions) \ > M(ActivityState) \ > M(BackForward) \ > M(CacheStorage) \ >diff --git a/Source/WebKit/UIProcess/Automation/Automation.json b/Source/WebKit/UIProcess/Automation/Automation.json >index cad53c7da1f450a0d605b6663d1ec1b8d0903090..9130f0ee67c43d1ee39e618861a276adf3a426d2 100644 >--- a/Source/WebKit/UIProcess/Automation/Automation.json >+++ b/Source/WebKit/UIProcess/Automation/Automation.json >@@ -32,7 +32,7 @@ > "description": "The coordinate system in which rects, points, and sizes are to be interpreted.", > "enum": [ > "Page", >- "LayoutViewport" >+ "Viewport" > ] > }, > { >diff --git a/Source/WebKit/UIProcess/Automation/SimulatedInputDispatcher.cpp b/Source/WebKit/UIProcess/Automation/SimulatedInputDispatcher.cpp >index 26a74e9c6541b652faeda6f3ad45e69df2bfdf3f..151c3336d6df3bf7c91d27eeb2d4deace27a4dd8 100644 >--- a/Source/WebKit/UIProcess/Automation/SimulatedInputDispatcher.cpp >+++ b/Source/WebKit/UIProcess/Automation/SimulatedInputDispatcher.cpp >@@ -324,7 +324,7 @@ void SimulatedInputDispatcher::transitionInputSourceToState(SimulatedInputSource > LOG(Automation, "SimulatedInputDispatcher[%p]: simulating TouchDown @ (%d, %d) for transition to %d.%d", this, b.location.value().x(), b.location.value().y(), m_keyframeIndex, m_inputSourceStateIndex); > m_client.simulateTouchInteraction(m_page, TouchInteraction::TouchDown, b.location.value(), WTF::nullopt, WTFMove(eventDispatchFinished)); > } else if (a.pressedMouseButton && !b.pressedMouseButton) { >- LOG(Automation, "SimulatedInputDispatcher[%p]: simulating LiftUp @ (%d, %d) for transition to %d.%d", this, a.location.value().x(), a.location.value().y(), m_keyframeIndex, m_inputSourceStateIndex); >+ LOG(Automation, "SimulatedInputDispatcher[%p]: simulating LiftUp @ (%d, %d) for transition to %d.%d", this, b.location.value().x(), b.location.value().y(), m_keyframeIndex, m_inputSourceStateIndex); > m_client.simulateTouchInteraction(m_page, TouchInteraction::LiftUp, b.location.value(), WTF::nullopt, WTFMove(eventDispatchFinished)); > } else if (a.location != b.location) { > LOG(Automation, "SimulatedInputDispatcher[%p]: simulating MoveTo from (%d, %d) to (%d, %d) for transition to %d.%d", this, a.location.value().x(), a.location.value().y(), b.location.value().x(), b.location.value().y(), m_keyframeIndex, m_inputSourceStateIndex); >diff --git a/Source/WebKit/UIProcess/Automation/WebAutomationSession.cpp b/Source/WebKit/UIProcess/Automation/WebAutomationSession.cpp >index c25b9517717948b3abcbc0305df7365c29850fa7..24a403b53e5009e94e19fd049c4d668f81465fe0 100644 >--- a/Source/WebKit/UIProcess/Automation/WebAutomationSession.cpp >+++ b/Source/WebKit/UIProcess/Automation/WebAutomationSession.cpp >@@ -1477,11 +1477,10 @@ void WebAutomationSession::viewportInViewCenterPointOfElement(WebPageProxy& page > #if ENABLE(WEBDRIVER_MOUSE_INTERACTIONS) > void WebAutomationSession::simulateMouseInteraction(WebPageProxy& page, MouseInteraction interaction, WebMouseEvent::Button mouseButton, const WebCore::IntPoint& locationInViewport, CompletionHandler<void(Optional<AutomationCommandError>)>&& completionHandler) > { >- WebCore::IntPoint locationInView = WebCore::IntPoint(locationInViewport.x(), locationInViewport.y() + page.topContentInset()); >- page.getWindowFrameWithCallback([this, protectedThis = makeRef(*this), completionHandler = WTFMove(completionHandler), page = makeRef(page), interaction, mouseButton, locationInView](WebCore::FloatRect windowFrame) mutable { >- auto clippedX = std::min(std::max(0.0f, (float)locationInView.x()), windowFrame.size().width()); >- auto clippedY = std::min(std::max(0.0f, (float)locationInView.y()), windowFrame.size().height()); >- if (clippedX != locationInView.x() || clippedY != locationInView.y()) { >+ page.getWindowFrameWithCallback([this, protectedThis = makeRef(*this), completionHandler = WTFMove(completionHandler), page = makeRef(page), interaction, mouseButton, locationInViewport](WebCore::FloatRect windowFrame) mutable { >+ auto clippedX = std::min(std::max(0.0f, (float)locationInViewport.x()), windowFrame.size().width()); >+ auto clippedY = std::min(std::max(0.0f, (float)locationInViewport.y()), windowFrame.size().height()); >+ if (clippedX != locationInViewport.x() || clippedY != locationInViewport.y()) { > completionHandler(AUTOMATION_COMMAND_ERROR_WITH_NAME(TargetOutOfBounds)); > return; > } >@@ -1496,7 +1495,7 @@ void WebAutomationSession::simulateMouseInteraction(WebPageProxy& page, MouseInt > callbackInMap(AUTOMATION_COMMAND_ERROR_WITH_NAME(Timeout)); > callbackInMap = WTFMove(mouseEventsFlushedCallback); > >- platformSimulateMouseInteraction(page, interaction, mouseButton, locationInView, OptionSet<WebEvent::Modifier>::fromRaw(m_currentModifiers)); >+ platformSimulateMouseInteraction(page, interaction, mouseButton, locationInViewport, OptionSet<WebEvent::Modifier>::fromRaw(m_currentModifiers)); > > // If the event does not hit test anything in the window, then it may not have been delivered. > if (callbackInMap && !page->isProcessingMouseEvents()) { >@@ -1513,7 +1512,8 @@ void WebAutomationSession::simulateMouseInteraction(WebPageProxy& page, MouseInt > void WebAutomationSession::simulateTouchInteraction(WebPageProxy& page, TouchInteraction interaction, const WebCore::IntPoint& locationInViewport, Optional<Seconds> duration, CompletionHandler<void(Optional<AutomationCommandError>)>&& completionHandler) > { > #if PLATFORM(IOS_FAMILY) >- if (!page.unobscuredContentRect().contains(locationInViewport)) { >+ WebCore::FloatRect visualViewportBounds = WebCore::FloatRect({ }, page.unobscuredContentRect().size()); >+ if (!visualViewportBounds.contains(locationInViewport)) { > completionHandler(AUTOMATION_COMMAND_ERROR_WITH_NAME(TargetOutOfBounds)); > return; > } >diff --git a/Source/WebKit/UIProcess/Automation/ios/WebAutomationSessionIOS.mm b/Source/WebKit/UIProcess/Automation/ios/WebAutomationSessionIOS.mm >index 3c5cad8302953a09e07574b04fbf94a89a91d885..7228affadbcaa25bf35dc418a0168d698080d548 100644 >--- a/Source/WebKit/UIProcess/Automation/ios/WebAutomationSessionIOS.mm >+++ b/Source/WebKit/UIProcess/Automation/ios/WebAutomationSessionIOS.mm >@@ -28,6 +28,7 @@ > > #if PLATFORM(IOS_FAMILY) > >+#import "Logging.h" > #import "NativeWebKeyboardEvent.h" > #import "WebAutomationSessionMacros.h" > #import "WebPageProxy.h" >@@ -175,15 +176,34 @@ void WebAutomationSession::platformSimulateKeySequence(WebPageProxy& page, const > #endif // ENABLE(WEBDRIVER_KEYBOARD_INTERACTIONS) > > #if ENABLE(WEBDRIVER_TOUCH_INTERACTIONS) >+#if !LOG_DISABLED >+static TextStream& operator<<(TextStream& ts, TouchInteraction interaction) >+{ >+ switch (interaction) { >+ case TouchInteraction::TouchDown: >+ ts << "TouchDown"; >+ break; >+ case TouchInteraction::MoveTo: >+ ts << "MoveTo"; >+ break; >+ case TouchInteraction::LiftUp: >+ ts << "LiftUp"; >+ break; >+ } >+ return ts; >+} >+#endif // !LOG_DISABLED >+ > void WebAutomationSession::platformSimulateTouchInteraction(WebPageProxy& page, TouchInteraction interaction, const WebCore::IntPoint& locationInViewport, Optional<Seconds> duration, AutomationCompletionHandler&& completionHandler) > { > WebCore::IntPoint locationOnScreen = page.syncRootViewToScreen(IntRect(locationInViewport, IntSize())).location(); >- _WKTouchEventGenerator *generator = [_WKTouchEventGenerator sharedTouchEventGenerator]; >+ LOG_WITH_STREAM(AutomationInteractions, stream << "platformSimulateTouchInteraction: interaction=" << interaction << ", locationInViewport=" << locationInViewport << ", locationOnScreen=" << locationOnScreen << ", duration=" << duration.valueOr(0_s).seconds()); > > auto interactionFinished = makeBlockPtr([completionHandler = WTFMove(completionHandler)] () mutable { > completionHandler(WTF::nullopt); > }); >- >+ >+ _WKTouchEventGenerator *generator = [_WKTouchEventGenerator sharedTouchEventGenerator]; > switch (interaction) { > case TouchInteraction::TouchDown: > [generator touchDown:locationOnScreen completionBlock:interactionFinished.get()]; >diff --git a/Source/WebKit/WebProcess/Automation/WebAutomationSessionProxy.cpp b/Source/WebKit/WebProcess/Automation/WebAutomationSessionProxy.cpp >index 9c31b829bb5a454e000243f13de8c198713e3744..250264864c32f3aa6e5a14c740eeb933b97bd44a 100644 >--- a/Source/WebKit/WebProcess/Automation/WebAutomationSessionProxy.cpp >+++ b/Source/WebKit/WebProcess/Automation/WebAutomationSessionProxy.cpp >@@ -477,35 +477,44 @@ void WebAutomationSessionProxy::focusFrame(uint64_t pageID, uint64_t frameID) > coreDOMWindow->focus(true); > } > >-static Optional<WebCore::FloatPoint> elementInViewClientCenterPoint(WebCore::Element& element, bool& isObscured) >+// Returns the in-view center point of an element in frame client coordinates. >+static Expected<WebCore::FloatPoint, Inspector::Protocol::Automation::ErrorMessage> elementInViewCenterPoint(WebCore::FrameView& frameView, WebCore::Element& element, bool& isObscured) > { >+ // NOTE: All local variables for points and rects in this function use frame client coordinates. >+ > // §11.1 Element Interactability. > // https://www.w3.org/TR/webdriver/#dfn-in-view-center-point >- auto* clientRect = element.getClientRects()->item(0); >- if (!clientRect) >- return WTF::nullopt; >+ auto elementRects = element.getClientRects(); >+ auto* firstElementRect = elementRects->item(0); >+ if (!firstElementRect) >+ return Unexpected<Inspector::Protocol::Automation::ErrorMessage>(Inspector::Protocol::Automation::ErrorMessage::ElementNotInteractable); > >- auto clientCenterPoint = WebCore::FloatPoint::narrowPrecision(0.5 * (clientRect->left() + clientRect->right()), 0.5 * (clientRect->top() + clientRect->bottom())); >- auto elementList = element.treeScope().elementsFromPoint(clientCenterPoint.x(), clientCenterPoint.y()); >- if (elementList.isEmpty()) { >- // An element is obscured if the pointer-interactable paint tree at its center point is empty, >- // or the first element in this tree is not an inclusive descendant of itself. >- // https://w3c.github.io/webdriver/webdriver-spec.html#dfn-obscured >- isObscured = true; >- return clientCenterPoint; >- } >+ // The W3C WebDriver specification does not explicitly intersect the element with the visual viewport. >+ // Do that here so that the IVCP for an element larger than the viewport is within the viewport. >+ // See spec bug here: https://github.com/w3c/webdriver/issues/1402 >+ auto viewportRect = frameView.documentToClientRect(frameView.visualViewportRect()); >+ auto visiblePortionOfElementRect = intersection(viewportRect, *firstElementRect); >+ if (visiblePortionOfElementRect.isEmpty()) >+ return Unexpected<Inspector::Protocol::Automation::ErrorMessage>(Inspector::Protocol::Automation::ErrorMessage::TargetOutOfBounds); > >+ auto elementInViewCenterPoint = visiblePortionOfElementRect.center(); >+ auto elementList = element.treeScope().elementsFromPoint(elementInViewCenterPoint); > auto index = elementList.findMatching([&element] (auto& item) { return item.get() == &element; }); >- if (index == notFound) >- return WTF::nullopt; >- >- if (index) { >- // Element is not the first one in the list. >- auto firstElement = elementList[0]; >- isObscured = !firstElement->isDescendantOf(element); >+ if (elementList.isEmpty() || index == notFound) { >+ // We hit this case if the element is visibility:hidden or opacity:0, in which case it will not hit test >+ // at the calculated IVCP. An element is technically not "in view" if it is not within its own paint/hit test tree, >+ // so it cannot have an in-view center point either. And without an IVCP, the definition of 'obscured' makes no sense. >+ // See <https://w3c.github.io/webdriver/webdriver-spec.html#dfn-in-view>. >+ return Unexpected<Inspector::Protocol::Automation::ErrorMessage>(Inspector::Protocol::Automation::ErrorMessage::ElementNotInteractable); > } > >- return clientCenterPoint; >+ // Check the case where a non-descendant element hit tests before the target element. For example, a child <option> >+ // of a <select> does not obscure the <select>, but two sibling <div> that overlap at the IVCP will obscure each other. >+ auto firstElement = elementList[0]; >+ >+ // Node::isDescendantOf() is not self-inclusive, so that is explicitly checked here. >+ isObscured = firstElement.get() != &element && !firstElement->isDescendantOf(element); >+ return elementInViewCenterPoint; > } > > static WebCore::Element* containerElementForElement(WebCore::Element& element) >@@ -534,6 +543,28 @@ static WebCore::Element* containerElementForElement(WebCore::Element& element) > return &element; > } > >+static WebCore::FloatRect convertRectFromFrameClientToRootView(FrameView* frameView, WebCore::FloatRect clientRect) >+{ >+ if (!frameView->delegatesScrolling()) >+ return frameView->contentsToRootView(frameView->clientToDocumentRect(clientRect)); >+ >+ auto& frame = frameView->frame(); >+ clientRect.scale(frame.pageZoomFactor() * frame.frameScaleFactor()); >+ clientRect.moveBy(frameView->contentsScrollPosition()); >+ return clientRect; >+} >+ >+static WebCore::FloatPoint convertPointFromFrameClientToRootView(FrameView* frameView, WebCore::FloatPoint clientPoint) >+{ >+ if (!frameView->delegatesScrolling()) >+ return frameView->contentsToRootView(frameView->clientToDocumentPoint(clientPoint)); >+ >+ auto& frame = frameView->frame(); >+ clientPoint.scale(frame.pageZoomFactor() * frame.frameScaleFactor()); >+ clientPoint.moveBy(frameView->contentsScrollPosition()); >+ return clientPoint; >+} >+ > void WebAutomationSessionProxy::computeElementLayout(uint64_t pageID, uint64_t frameID, String nodeHandle, bool scrollIntoViewIfNeeded, CoordinateSystem coordinateSystem, CompletionHandler<void(Optional<String>, WebCore::IntRect, Optional<WebCore::IntPoint>, bool)>&& completionHandler) > { > WebPage* page = WebProcess::singleton().webPage(pageID); >@@ -543,18 +574,16 @@ void WebAutomationSessionProxy::computeElementLayout(uint64_t pageID, uint64_t f > return; > } > >- String frameNotFoundErrorType = Inspector::Protocol::AutomationHelpers::getEnumConstantValue(Inspector::Protocol::Automation::ErrorMessage::FrameNotFound); >- String nodeNotFoundErrorType = Inspector::Protocol::AutomationHelpers::getEnumConstantValue(Inspector::Protocol::Automation::ErrorMessage::NodeNotFound); >- String notImplementedErrorType = Inspector::Protocol::AutomationHelpers::getEnumConstantValue(Inspector::Protocol::Automation::ErrorMessage::NotImplemented); >- > WebFrame* frame = frameID ? WebProcess::singleton().webFrame(frameID) : page->mainWebFrame(); > if (!frame || !frame->coreFrame() || !frame->coreFrame()->view()) { >+ String frameNotFoundErrorType = Inspector::Protocol::AutomationHelpers::getEnumConstantValue(Inspector::Protocol::Automation::ErrorMessage::FrameNotFound); > completionHandler(frameNotFoundErrorType, { }, WTF::nullopt, false); > return; > } > > WebCore::Element* coreElement = elementForNodeHandle(*frame, nodeHandle); > if (!coreElement) { >+ String nodeNotFoundErrorType = Inspector::Protocol::AutomationHelpers::getEnumConstantValue(Inspector::Protocol::Automation::ErrorMessage::NodeNotFound); > completionHandler(nodeNotFoundErrorType, { }, WTF::nullopt, false); > return; > } >@@ -569,35 +598,46 @@ void WebAutomationSessionProxy::computeElementLayout(uint64_t pageID, uint64_t f > > WebCore::FrameView* frameView = frame->coreFrame()->view(); > WebCore::FrameView* mainView = frame->coreFrame()->mainFrame().view(); >- WebCore::IntRect frameElementBounds = roundedIntRect(coreElement->boundingClientRect()); >- WebCore::IntRect rootElementBounds = mainView->rootViewToContents(frameView->contentsToRootView(frameElementBounds)); >+ > WebCore::IntRect resultElementBounds; >+ >+ auto elementBoundsInRootCoordinates = convertRectFromFrameClientToRootView(frameView, coreElement->boundingClientRect()); > switch (coordinateSystem) { > case CoordinateSystem::Page: >- resultElementBounds = WebCore::IntRect(mainView->clientToDocumentRect(WebCore::FloatRect(rootElementBounds))); >+ resultElementBounds = enclosingIntRect(mainView->absoluteToDocumentRect(mainView->rootViewToContents(elementBoundsInRootCoordinates))); > break; > case CoordinateSystem::LayoutViewport: >- // The element bounds are already in client coordinates. >- resultElementBounds = WebCore::IntRect(mainView->clientToLayoutViewportRect(WebCore::FloatRect(rootElementBounds))); >+ resultElementBounds = enclosingIntRect(mainView->absoluteToLayoutViewportRect(elementBoundsInRootCoordinates)); > break; > } > > Optional<WebCore::IntPoint> resultInViewCenterPoint; >+ >+ // If an <option> or <optgroup> does not have an associated <select> or <datalist> element, then give up. >+ if (!containerElement) { >+ String elementNotInteractableErrorType = Inspector::Protocol::AutomationHelpers::getEnumConstantValue(Inspector::Protocol::Automation::ErrorMessage::ElementNotInteractable); >+ completionHandler(elementNotInteractableErrorType, resultElementBounds, resultInViewCenterPoint, false); >+ return; >+ } >+ > bool isObscured = false; >- if (containerElement) { >- Optional<WebCore::FloatPoint> frameInViewCenterPoint = elementInViewClientCenterPoint(*containerElement, isObscured); >- if (frameInViewCenterPoint.hasValue()) { >- WebCore::IntPoint rootInViewCenterPoint = mainView->rootViewToContents(frameView->contentsToRootView(WebCore::IntPoint(frameInViewCenterPoint.value()))); >- switch (coordinateSystem) { >- case CoordinateSystem::Page: >- resultInViewCenterPoint = WebCore::IntPoint(mainView->clientToDocumentPoint(rootInViewCenterPoint)); >- break; >- case CoordinateSystem::LayoutViewport: >- // The point is already in client coordinates. >- resultInViewCenterPoint = WebCore::IntPoint(mainView->clientToLayoutViewportPoint(rootInViewCenterPoint)); >- break; >- } >- } >+ auto inViewCenterPointInFrameClientCoordinates = elementInViewCenterPoint(*frameView, *containerElement, isObscured); >+ >+ // If the element is not within the bounds of the viewport, then there will be no IVCP and the element is not interactable. >+ if (!inViewCenterPointInFrameClientCoordinates) { >+ String errorType = Inspector::Protocol::AutomationHelpers::getEnumConstantValue(inViewCenterPointInFrameClientCoordinates.error()); >+ completionHandler(errorType, resultElementBounds, resultInViewCenterPoint, false); >+ return; >+ } >+ >+ auto inViewCenterPointInRootCoordinates = convertPointFromFrameClientToRootView(frameView, inViewCenterPointInFrameClientCoordinates.value()); >+ switch (coordinateSystem) { >+ case CoordinateSystem::Page: >+ resultInViewCenterPoint = roundedIntPoint(mainView->absoluteToDocumentPoint(inViewCenterPointInRootCoordinates)); >+ break; >+ case CoordinateSystem::LayoutViewport: >+ resultInViewCenterPoint = roundedIntPoint(mainView->absoluteToLayoutViewportPoint(inViewCenterPointInRootCoordinates)); >+ break; > } > > completionHandler(WTF::nullopt, resultElementBounds, resultInViewCenterPoint, isObscured);
<b>You cannot view the attachment while viewing its details because your browser does not support IFRAMEs. <a href="attachment.cgi?id=369584">View the attachment on a separate page</a>.</b>
View Attachment As Diff
View Attachment As Raw
Actions:
View
|
Formatted Diff
|
Diff
Attachments on
bug 195696
:
364682
|
367478
|
369207
|
369241
|
369584
|
369659
|
369676
|
369812
|
369913
|
369936