100964 2012-11-01 09:16:32 -0700 [Qt] Make sure that the WebView in flickable mode sends mouse events to the page rather than controlling the viewport with it 2012-11-29 03:15:45 -0800 1 1 1 Unclassified WebKit WebKit Qt 528+ (Nightly build) Unspecified Unspecified RESOLVED FIXED Qt P2 Normal --- 76773 1 jturcotte jturcotte abecsi hausmann jturcotte kenneth noam svillar oldest_to_newest 756328 0 jturcotte 2012-11-01 09:16:32 -0700 A workstation without a touch device should be able to use the WebView in flickable mode and be able to scroll with the wheel, see scrollbars in a a desktop WebView component. It should also be able to select text and click links without touch adjustment kicking in and no tap highlighting should be shown. A workstation having both a touch device and a mouse should be able to select text with the mouse and flick the view with the touch device. Touch enabled web pages like Google maps should be able to take advantage of the touch device too. We should set the correct behavior before we release 5.0 to make sure that we don't have to change it afterward. Right now the only change that seems to be needed for this is removing QQuickWebViewFlickablePrivate::handleMouseEvent. The only problem would be with touch screens on OSes sending both touch and mouse events (if that still exist). We have to test it properly. 756338 1 abecsi 2012-11-01 09:24:32 -0700 The work on this bug and the introduction of a desktop WebView component should also make it possible in the long term to remove the desktop (legacy) codepaths. 756552 2 svillar 2012-11-01 12:51:00 -0700 (In reply to comment #0) > A workstation without a touch device should be able to use the WebView in flickable mode and be able to scroll with the wheel, see scrollbars in a a desktop WebView component. It should also be able to select text and click links without touch adjustment kicking in and no tap highlighting should be shown. > Fantastic specification. I'd add that it should also be able to perform drags of the draggable elements inside the page (like images for example). > A workstation having both a touch device and a mouse should be able to select text with the mouse and flick the view with the touch device. Touch enabled web pages like Google maps should be able to take advantage of the touch device too. > Not sure at all about d&d in this case, looks complex. > We should set the correct behavior before we release 5.0 to make sure that we don't have to change it afterward. > > Right now the only change that seems to be needed for this is removing QQuickWebViewFlickablePrivate::handleMouseEvent. > The only problem would be with touch screens on OSes sending both touch and mouse events (if that still exist). We have to test it properly. 766222 3 svillar 2012-11-13 17:29:48 -0800 So, in order to keep this bug rolling, I think is important to consider how events are managed in the Qt WK2 port, although I have some experience with WebKit I've just started working on WebKitQt so please correct me if I'm wrong. Basically once we get an event from the UI there are 2 high level code paths: 1) the QtWebPageEventHandler redirects the event to the WebProcess and then WebCore processes it and informs the UIProcess using the different available clients. 2) the QtWebPageEventHandler delegates the handling of the event to one of the QtGestureRecognizers available. The gesture recognizer could then decide to send the event back to the event handler or to the viewport controller for further processing. Jocelyn suggests to remove QQuickWebViewFlickablePrivate::handleMouseEvent. What would that mean? Right now, if using the flickable version, all the events follow path 2. This means that for example all drag actions are considered a flick. If we remove that handler, then mouse events will follow path 1, which means that WebCore will directly handle them, and will consider the previous flick gesture as a drag start event. So if we consider the typical case of a desktop browser using the flickable WebView (which will be the only one if I understood some of you correctly) the current architecture forces us to decide between having flickable events or the classical drag events. The drag start will also collide with some other events, for example a tap and hold. How is the port supposed to handle these conflicts? 766620 4 jturcotte 2012-11-14 01:37:21 -0800 (In reply to comment #3) > Basically once we get an event from the UI there are 2 high level code paths: > > 1) the QtWebPageEventHandler redirects the event to the WebProcess and then WebCore processes it and informs the UIProcess using the different available clients. > > 2) the QtWebPageEventHandler delegates the handling of the event to one of the QtGestureRecognizers available. The gesture recognizer could then decide to send the event back to the event handler or to the viewport controller for further processing. The way I think it works right now (I'm no expert on this either) is that those two code paths are actually sequential. Both touch and mouse events will first got through the web process (WebPage::touchEvent), and if the event wasn't handled by the page, return it to the UI process (WebPageProxy::didReceiveEvent) which will then give it back to QtWebPageEventHandler through PageClient::doneWithTouchEvent. For mouse events however, WebPageProxy::didReceiveEvent doesn't push them back to the higher layers. We might have to add shortcuts at some point to prevent a busy web process from delaying viewport interaction by sending events directly to the event handler (e.g. if we know the page doesn't have touch handlers at this position), but the logic should still apply. > Jocelyn suggests to remove QQuickWebViewFlickablePrivate::handleMouseEvent. What would that mean? Right now, if using the flickable version, all the events follow path 2. This means that for example all drag actions are considered a flick. If we remove that handler, then mouse events will follow path 1, which means that WebCore will directly handle them, and will consider the previous flick gesture as a drag start event. > > So if we consider the typical case of a desktop browser using the flickable WebView (which will be the only one if I understood some of you correctly) the current architecture forces us to decide between having flickable events or the classical drag events. The drag start will also collide with some other events, for example a tap and hold. > > How is the port supposed to handle these conflicts? The way I see it is that the WebView is only behaving as a flickable if you interact with it using touch events. To scroll the view using a mouse there are still the mouse wheel and scrollbars. Nobody (I assume) wants to flick using a mouse anyway. That means that there would be no way of drag'n dropping using a touch device. 767210 5 svillar 2012-11-14 13:04:29 -0800 (In reply to comment #4) > (In reply to comment #3) > > Basically once we get an event from the UI there are 2 high level code paths: > > > > 1) the QtWebPageEventHandler redirects the event to the WebProcess and then WebCore processes it and informs the UIProcess using the different available clients. > > > > 2) the QtWebPageEventHandler delegates the handling of the event to one of the QtGestureRecognizers available. The gesture recognizer could then decide to send the event back to the event handler or to the viewport controller for further processing. > > The way I think it works right now (I'm no expert on this either) is that those two code paths are actually sequential. > Both touch and mouse events will first got through the web process (WebPage::touchEvent), and if the event wasn't handled by the page, return it to the UI process (WebPageProxy::didReceiveEvent) which will then give it back to QtWebPageEventHandler through PageClient::doneWithTouchEvent. > > For mouse events however, WebPageProxy::didReceiveEvent doesn't push them back to the higher layers. Hmm taking a look at the code, the flickable private object redirects all mouse events to QtWebPageEventHandler::handleInputEvent. This means that the event will be handled by the gesture recognizers in the UI side and nothing will be send to the WebProcess, or am I wrong? As you said if we remove that handler then all the events will be then sent to the WebProcess directly. > > So if we consider the typical case of a desktop browser using the flickable WebView (which will be the only one if I understood some of you correctly) the current architecture forces us to decide between having flickable events or the classical drag events. The drag start will also collide with some other events, for example a tap and hold. > > > > How is the port supposed to handle these conflicts? > > The way I see it is that the WebView is only behaving as a flickable if you interact with it using touch events. To scroll the view using a mouse there are still the mouse wheel and scrollbars. Nobody (I assume) wants to flick using a mouse anyway. > That means that there would be no way of drag'n dropping using a touch device. Ok that looks like a good trade-off. I am not sure that nobody wants to flick using a mouse though. For example I was thinking about interfaces like the Wii, were you have a pointer device that effectively acts as a mouse, but IIRC the Opera browser it has does some kind of panning (flick) instead of scrolling. The hybrid devices mentioned somewhere else could be another example. 767890 6 jturcotte 2012-11-15 03:08:46 -0800 (In reply to comment #5) > Hmm taking a look at the code, the flickable private object redirects all mouse events to QtWebPageEventHandler::handleInputEvent. This means that the event will be handled by the gesture recognizers in the UI side and nothing will be send to the WebProcess, or am I wrong? As you said if we remove that handler then all the events will be then sent to the WebProcess directly. Yep exactly, that's why selecting text and drag-dropping isn't working. We treat mouse clicks as touch taps if the button was released without moving and send them to the web process. > Ok that looks like a good trade-off. I am not sure that nobody wants to flick using a mouse though. For example I was thinking about interfaces like the Wii, were you have a pointer device that effectively acts as a mouse, but IIRC the Opera browser it has does some kind of panning (flick) instead of scrolling. The hybrid devices mentioned somewhere else could be another example. I think that this should then be patched in the distribution or enabled through a platform plugin. In other words, the platform should be dealing with this and not the application code IMO. If platform-specific applications want to enable this, we could expose a setting through a private API. 770460 7 hausmann 2012-11-19 01:16:38 -0800 (In reply to comment #0) [...] > Right now the only change that seems to be needed for this is removing QQuickWebViewFlickablePrivate::handleMouseEvent. > The only problem would be with touch screens on OSes sending both touch and mouse events (if that still exist). We have to test it properly. I believe that is for example the case on Mac OS X with a multi point touch-pad, isn't it? 770568 8 abecsi 2012-11-19 03:54:29 -0800 (In reply to comment #4) > (In reply to comment #3) > > Basically once we get an event from the UI there are 2 high level code paths: > > > > 1) the QtWebPageEventHandler redirects the event to the WebProcess and then WebCore processes it and informs the UIProcess using the different available clients. > > > > 2) the QtWebPageEventHandler delegates the handling of the event to one of the QtGestureRecognizers available. The gesture recognizer could then decide to send the event back to the event handler or to the viewport controller for further processing. > > The way I think it works right now (I'm no expert on this either) is that those two code paths are actually sequential. > Both touch and mouse events will first got through the web process (WebPage::touchEvent), and if the event wasn't handled by the page, return it to the UI process (WebPageProxy::didReceiveEvent) which will then give it back to QtWebPageEventHandler through PageClient::doneWithTouchEvent. Exactly, at this point the event is processed by the gesture recognizers and depending on the gesture we either control the flickable (QQuickWebView::handleFlickableMouse*) for pan gestures or we synthesize a tap or act on a double tap or scale the page item if the gesture was a pinch. > > For mouse events however, WebPageProxy::didReceiveEvent doesn't push them back to the higher layers. > > We might have to add shortcuts at some point to prevent a busy web process from delaying viewport interaction by sending events directly to the event handler (e.g. if we know the page doesn't have touch handlers at this position), but the logic should still apply. Yes, I think the shortcutting is still something we want to do later, because as web pages have more and more event handlers our interaction might end up being sluggish which we do not want. > > > Jocelyn suggests to remove QQuickWebViewFlickablePrivate::handleMouseEvent. What would that mean? Right now, if using the flickable version, all the events follow path 2. This means that for example all drag actions are considered a flick. If we remove that handler, then mouse events will follow path 1, which means that WebCore will directly handle them, and will consider the previous flick gesture as a drag start event. > > > > So if we consider the typical case of a desktop browser using the flickable WebView (which will be the only one if I understood some of you correctly) the current architecture forces us to decide between having flickable events or the classical drag events. The drag start will also collide with some other events, for example a tap and hold. > > > > How is the port supposed to handle these conflicts? > > The way I see it is that the WebView is only behaving as a flickable if you interact with it using touch events. To scroll the view using a mouse there are still the mouse wheel and scrollbars. Nobody (I assume) wants to flick using a mouse anyway. > That means that there would be no way of drag'n dropping using a touch device. Drag&drop on a touch device should in my opinion be implemented with a tap-and-hold gesture which would open a context menu and where applicable would offer drag&drop. I do not think that having this feature now is of high priority for us, though, since the tap-and-hold logic for context menus needs to be implemented first which depends on client side QML and is thus tied to the Desktop/Touch Components projects. 770574 9 abecsi 2012-11-19 04:02:35 -0800 (In reply to comment #6) > (In reply to comment #5) > > Hmm taking a look at the code, the flickable private object redirects all mouse events to QtWebPageEventHandler::handleInputEvent. This means that the event will be handled by the gesture recognizers in the UI side and nothing will be send to the WebProcess, or am I wrong? As you said if we remove that handler then all the events will be then sent to the WebProcess directly. > > Yep exactly, that's why selecting text and drag-dropping isn't working. We treat mouse clicks as touch taps if the button was released without moving and send them to the web process. Incoming mouse event are directly sent through the gesture recognizers. With this we worked around the issue that on devices that send both touch and mouse events simultaneously we end up selecting text during panning. > > > Ok that looks like a good trade-off. I am not sure that nobody wants to flick using a mouse though. For example I was thinking about interfaces like the Wii, were you have a pointer device that effectively acts as a mouse, but IIRC the Opera browser it has does some kind of panning (flick) instead of scrolling. The hybrid devices mentioned somewhere else could be another example. > > I think that this should then be patched in the distribution or enabled through a platform plugin. In other words, the platform should be dealing with this and not the application code IMO. If platform-specific applications want to enable this, we could expose a setting through a private API. I also think this is an issue of the driver/ underlying platform, but as far as I know touch pads on Mac OS also send both events. 775322 10 hausmann 2012-11-26 03:26:24 -0800 ping, any update on this? 775335 11 jturcotte 2012-11-26 04:01:48 -0800 (In reply to comment #10) > ping, any update on this? I tried on Linux and just removing QQuickWebViewFlickablePrivate::handleMouseEvent produce the same behavior as IE does on Windows 8. Mouse selects, and touch pans. If we can add a QML scroll bar to WebKit2 examples I think this would be fine for 5.0. Brought my Mac at work today, I want to try with the touch pad as you said it might produce both types of events. My debug build failed though, it's trying to like against libQt5WebKitWidgets_debug but I only have libQtWebKitWidgets.5.dylib (no _debug suffix and with "Qt" instead of "Qt5") I'll try to finish my patch for setHtml and look at it afterward. 777942 12 176491 jturcotte 2012-11-28 08:46:30 -0800 Created attachment 176491 Patch I tested with the mac trackpad and it's not sending touch events for single touch. When using multiple fingers it doesn't send mouse move events. The only problem is that it sends wheel events at the same time as touch events when doing two-fingers scroll, but this is also the reason why they were disabled by default one month ago: https://qt.gitorious.org/qt/qtbase/commit/adb156e4dd64609ba6c0b172e9c428a79e306a7c . Ideally I guess that we should just handle gestures directly for those devices rather than doing gesture recognition directly ourselves. That would also help support stuff like that magic mouse that I've read aren't sending individual touch events. So anyway here is the patch. I'll do some more work to try showing scroll bars in MiniBrowser and maybe handle ctrl-wheel directly in WebView to make it easier to handle it with the mouse. 777960 13 176491 abecsi 2012-11-28 09:03:46 -0800 Comment on attachment 176491 Patch Looks good. I agree with you on the gesture recognition part, in we would need some mechanism in Qt to have simple gesture events and be able to subscribe to them, instead of having decentralized gesture recognition. This sounds like an interesting research project. 778889 14 jturcotte 2012-11-29 03:15:45 -0800 Committed r136119: <http://trac.webkit.org/changeset/136119> 176491 2012-11-28 08:46:30 -0800 2012-11-28 09:03:46 -0800 Patch bug-100964-20121128174405.patch text/plain 2856 jturcotte U3VidmVyc2lvbiBSZXZpc2lvbjogMTM1NzE5CmRpZmYgLS1naXQgYS9Tb3VyY2UvV2ViS2l0Mi9D aGFuZ2VMb2cgYi9Tb3VyY2UvV2ViS2l0Mi9DaGFuZ2VMb2cKaW5kZXggODliOGIxMDYzZmEwMGFj MjIwZGMyMzgxOGE0MjFkOTg5OTAwZGZiZi4uYzBlMTEyMjYxMjBiY2Y0ODk3YjA1YjFjYWE5NWQw NWIxM2Y4YTg3NiAxMDA2NDQKLS0tIGEvU291cmNlL1dlYktpdDIvQ2hhbmdlTG9nCisrKyBiL1Nv dXJjZS9XZWJLaXQyL0NoYW5nZUxvZwpAQCAtMSwzICsxLDI0IEBACisyMDEyLTExLTI4ICBKb2Nl bHluIFR1cmNvdHRlICA8am9jZWx5bi50dXJjb3R0ZUBkaWdpYS5jb20+CisKKyAgICAgICAgW1F0 XSBUaGUgV2ViVmlldyBzaG91bGQgYmUgZmxpY2thYmxlIG9ubHkgdXNpbmcgdG91Y2ggZXZlbnRz CisgICAgICAgIGh0dHBzOi8vYnVncy53ZWJraXQub3JnL3Nob3dfYnVnLmNnaT9pZD0xMDA5NjQK KworICAgICAgICBSZXZpZXdlZCBieSBOT0JPRFkgKE9PUFMhKS4KKworICAgICAgICBMZXQgYSB1 c2VyIGludGVyYWN0aW5nIHdpdGggdGhlIFdlYlZpZXcgdXNpbmcgYSBtb3VzZSB0aGUgc2FtZSB3 YXkgdGhhdCBhCisgICAgICAgIGRlc2t0b3AgYnJvd3NlciB3b3VsZC4KKyAgICAgICAgVGhpcyBh bGxvd3Mgbm9ybWFsIGJlaGF2aW9ycyBsaWtlIHRleHQgc2VsZWN0aW5nLCBob3ZlciBldmVudHMs IG1vdXNlIGN1cnNvcnMKKyAgICAgICAgYXMgd2VsbCBhcyBhdm9pZCB0b3VjaCBoaWdobGlnaHRp bmcgYW5kIHRvdWNoIGFkanVzdG1lbnQuCisKKyAgICAgICAgVG8gZG8gc28gd2UgbGV0IGFsbCBt b3VzZSBldmVudHMgdG8gYmUgaGFuZGxlZCBieSB0aGUgcGFnZSBkaXJlY3RseSBpbnN0ZWFkCisg ICAgICAgIG9mIHNlbmRpbmcgdGhlbSB0byBRdFdlYlBhZ2VFdmVudEhhbmRsZXIuIFNpbmNlIHVu aGFuZGxlZCBtb3VzZSBldmVudHMgYXJlbid0CisgICAgICAgIHB1c2hlZCBiYWNrIHRvIHRoZSBQ YWdlQ2xpZW50IGxpa2UgdG91Y2ggZXZlbnRzIGRvZXMsIHRoaXMgaGF2ZSB0aGUgaW50ZW5kZWQK KyAgICAgICAgYmVoYXZpb3Igb2Ygbm90IGxldHRpbmcgbW91c2UgZXZlbnRzIGFmZmVjdCBRUXVp Y2tGbGlja2FibGUuCisKKyAgICAgICAgKiBVSVByb2Nlc3MvQVBJL3F0L3FxdWlja3dlYnZpZXcu Y3BwOgorICAgICAgICAqIFVJUHJvY2Vzcy9BUEkvcXQvcXF1aWNrd2Vidmlld19wX3AuaDoKKyAg ICAgICAgKFFRdWlja1dlYlZpZXdGbGlja2FibGVQcml2YXRlKToKKwogMjAxMi0xMS0yNiAgSmFl aHVuIExpbSAgPGxqYWVodW4ubGltQHNhbXN1bmcuY29tPgogCiAgICAgICAgIFRleHQgQXV0b3Np emluZzogQWRkIFRleHQgQXV0b3NpemluZyBBUElzIGZvciBXSzIKZGlmZiAtLWdpdCBhL1NvdXJj ZS9XZWJLaXQyL1VJUHJvY2Vzcy9BUEkvcXQvcXF1aWNrd2Vidmlldy5jcHAgYi9Tb3VyY2UvV2Vi S2l0Mi9VSVByb2Nlc3MvQVBJL3F0L3FxdWlja3dlYnZpZXcuY3BwCmluZGV4IDIxZWQ1YmUxMDMz ZGIxZjkyOTg5Y2VkZWM4NGJjYmI0ZjM4MzdlYWMuLjJjNDY3ODhjOTk5YjI1YTI4ZGEzNmUwZjI2 MzVkNzY0NmZiZjdjNjYgMTAwNjQ0Ci0tLSBhL1NvdXJjZS9XZWJLaXQyL1VJUHJvY2Vzcy9BUEkv cXQvcXF1aWNrd2Vidmlldy5jcHAKKysrIGIvU291cmNlL1dlYktpdDIvVUlQcm9jZXNzL0FQSS9x dC9xcXVpY2t3ZWJ2aWV3LmNwcApAQCAtODg5LDE1ICs4ODksNiBAQCB2b2lkIFFRdWlja1dlYlZp ZXdGbGlja2FibGVQcml2YXRlOjpwYWdlRGlkUmVxdWVzdFNjcm9sbChjb25zdCBRUG9pbnQmIHBv cykKICAgICAgICAgbV9wYWdlVmlld3BvcnRDb250cm9sbGVyLT5wYWdlRGlkUmVxdWVzdFNjcm9s bChwb3MpOwogfQogCi12b2lkIFFRdWlja1dlYlZpZXdGbGlja2FibGVQcml2YXRlOjpoYW5kbGVN b3VzZUV2ZW50KFFNb3VzZUV2ZW50KiBldmVudCkKLXsKLSAgICBpZiAoIXBhZ2VWaWV3LT5ldmVu dEhhbmRsZXIoKSkKLSAgICAgICAgcmV0dXJuOwotCi0gICAgLy8gRklYTUU6IFVwZGF0ZSB0aGUg YXhpcyBsb2NrZXIgZm9yIG1vdXNlIGV2ZW50cyBhcyB3ZWxsLgotICAgIHBhZ2VWaWV3LT5ldmVu dEhhbmRsZXIoKS0+aGFuZGxlSW5wdXRFdmVudChldmVudCk7Ci19Ci0KIFFRdWlja1dlYlZpZXdF eHBlcmltZW50YWw6OlFRdWlja1dlYlZpZXdFeHBlcmltZW50YWwoUVF1aWNrV2ViVmlldyAqd2Vi VmlldywgUVF1aWNrV2ViVmlld1ByaXZhdGUqIHdlYlZpZXdQcml2YXRlKQogICAgIDogUU9iamVj dCh3ZWJWaWV3KQogICAgICwgcV9wdHIod2ViVmlldykKZGlmZiAtLWdpdCBhL1NvdXJjZS9XZWJL aXQyL1VJUHJvY2Vzcy9BUEkvcXQvcXF1aWNrd2Vidmlld19wX3AuaCBiL1NvdXJjZS9XZWJLaXQy L1VJUHJvY2Vzcy9BUEkvcXQvcXF1aWNrd2Vidmlld19wX3AuaAppbmRleCBhZGVkYjZhMzE3YTA4 ODg2N2RhMTYxMzM4ZGVhMTY4YjU3N2IwM2JhLi44NDQ1ZmUyZjc3NjZlMDZmZGE5OTM1ZTNiNWY2 NDQ2ZWViNmMxOGY2IDEwMDY0NAotLS0gYS9Tb3VyY2UvV2ViS2l0Mi9VSVByb2Nlc3MvQVBJL3F0 L3FxdWlja3dlYnZpZXdfcF9wLmgKKysrIGIvU291cmNlL1dlYktpdDIvVUlQcm9jZXNzL0FQSS9x dC9xcXVpY2t3ZWJ2aWV3X3BfcC5oCkBAIC0yMjMsNyArMjIzLDYgQEAgcHVibGljOgogICAgIHZp cnR1YWwgdm9pZCB1cGRhdGVWaWV3cG9ydFNpemUoKTsKIAogICAgIHZpcnR1YWwgdm9pZCBwYWdl RGlkUmVxdWVzdFNjcm9sbChjb25zdCBRUG9pbnQmIHBvcyk7Ci0gICAgdmlydHVhbCB2b2lkIGhh bmRsZU1vdXNlRXZlbnQoUU1vdXNlRXZlbnQqKTsKIAogcHJpdmF0ZToKICAgICBRU2NvcGVkUG9p bnRlcjxXZWJLaXQ6OlBhZ2VWaWV3cG9ydENvbnRyb2xsZXI+IG1fcGFnZVZpZXdwb3J0Q29udHJv bGxlcjsK