<?xml version="1.0" encoding="UTF-8" standalone="yes" ?>
<!DOCTYPE bugzilla SYSTEM "https://bugs.webkit.org/page.cgi?id=bugzilla.dtd">

<bugzilla version="5.0.4.1"
          urlbase="https://bugs.webkit.org/"
          
          maintainer="admin@webkit.org"
>

    <bug>
          <bug_id>224566</bug_id>
          
          <creation_ts>2021-04-14 11:39:17 -0700</creation_ts>
          <short_desc>REGRESSION(r275921-r275922): [ wk2 ] http/tests/security/webgl-remote-read-remote-image-allowed-with-credentials.html is flakey crashing</short_desc>
          <delta_ts>2021-04-20 09:44:25 -0700</delta_ts>
          <reporter_accessible>1</reporter_accessible>
          <cclist_accessible>1</cclist_accessible>
          <classification_id>1</classification_id>
          <classification>Unclassified</classification>
          <product>WebKit</product>
          <component>New Bugs</component>
          <version>WebKit Nightly Build</version>
          <rep_platform>Unspecified</rep_platform>
          <op_sys>Unspecified</op_sys>
          <bug_status>RESOLVED</bug_status>
          <resolution>FIXED</resolution>
          
          <see_also>https://bugs.webkit.org/show_bug.cgi?id=224516</see_also>
    
    <see_also>https://bugs.webkit.org/show_bug.cgi?id=224513</see_also>
    
    <see_also>https://bugs.webkit.org/show_bug.cgi?id=224623</see_also>
          <bug_file_loc></bug_file_loc>
          <status_whiteboard></status_whiteboard>
          <keywords>InRadar</keywords>
          <priority>P2</priority>
          <bug_severity>Normal</bug_severity>
          <target_milestone>---</target_milestone>
          
          <blocked>224556</blocked>
          <everconfirmed>1</everconfirmed>
          <reporter name="Robert Jenner">jenner</reporter>
          <assigned_to name="Chris Dumez">cdumez</assigned_to>
          <cc>amir_mark</cc>
    
    <cc>cdumez</cc>
    
    <cc>darin</cc>
    
    <cc>ggaren</cc>
    
    <cc>kkinnunen</cc>
    
    <cc>ryanhaddad</cc>
    
    <cc>simon.fraser</cc>
    
    <cc>thorton</cc>
    
    <cc>webkit-bot-watchers-bugzilla</cc>
    
    <cc>webkit-bug-importer</cc>
          

      

      

      

          <comment_sort_order>oldest_to_newest</comment_sort_order>  
          <long_desc isprivate="0" >
    <commentid>1750251</commentid>
    <comment_count>0</comment_count>
    <who name="Robert Jenner">jenner</who>
    <bug_when>2021-04-14 11:39:17 -0700</bug_when>
    <thetext>http/tests/security/webgl-remote-read-remote-image-allowed-with-credentials.html

is flakey crashing in wk2 on iOS 14 Simulator, Catalina and BigSur. 

HISTORY:
https://results.webkit.org/?suite=layout-tests&amp;test=http%2Ftests%2Fsecurity%2Fwebgl-remote-read-remote-image-allowed-with-credentials.html


No crashlog was generated for this test. However, I was able to find a STDERR. 

STDERR TEXT:
stderr:
2021-04-14 07:23:46.915 com.apple.WebKit.Networking.Development[91141:40671502] DiskCookieStorage changing policy from 0 to 2, cookie file: file:///Users/buildbot/Library/HTTPStorages/com.apple.WebKit.WebKitTestRunner.binarycookies
LEAK: 1 WebPageProxy</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>1750252</commentid>
    <comment_count>1</comment_count>
    <who name="Radar WebKit Bug Importer">webkit-bug-importer</who>
    <bug_when>2021-04-14 11:40:48 -0700</bug_when>
    <thetext>&lt;rdar://problem/76657007&gt;</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>1750258</commentid>
    <comment_count>2</comment_count>
    <who name="Robert Jenner">jenner</who>
    <bug_when>2021-04-14 11:49:04 -0700</bug_when>
    <thetext>Updated test expectations to Pass Crash here:

https://trac.webkit.org/changeset/275955/webkit

Updated before investigating because this test was causing issues for EWS and commit-queue.</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>1750368</commentid>
    <comment_count>3</comment_count>
    <who name="Robert Jenner">jenner</who>
    <bug_when>2021-04-14 15:12:25 -0700</bug_when>
    <thetext>(In reply to Robert Jenner from comment #2)
&gt; Updated test expectations to Pass Crash here:
&gt; 
&gt; https://trac.webkit.org/changeset/275955/webkit
&gt; 
&gt; Updated before investigating because this test was causing issues for EWS
&gt; and commit-queue.

Forgot to add expectations to Mac wk2. Updated here:
https://trac.webkit.org/changeset/275974/webkit</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>1750383</commentid>
    <comment_count>4</comment_count>
    <who name="Chris Dumez">cdumez</who>
    <bug_when>2021-04-14 15:52:58 -0700</bug_when>
    <thetext>No crash log? Weird.</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>1750404</commentid>
    <comment_count>5</comment_count>
    <who name="Chris Dumez">cdumez</who>
    <bug_when>2021-04-14 16:29:36 -0700</bug_when>
    <thetext>I was able to reproduce the crash locally on macOS with the command that was provided. I see this:
16:24:02.300 6056 com.apple.WebKit.WebContent.Development crash, pid = 6113
16:24:02.318 6056 killed pid 6059
16:26:17.392 6056 worker/26 http/tests/security/webgl-remote-read-remote-image-allowed-with-credentials.html crashed, (stderr lines):
16:26:17.392 6056   LEAK: 1 WebPageProxy
16:26:17.394 5731 [2000/2000] http/tests/security/webgl-remote-read-remote-image-allowed-with-credentials.html failed unexpectedly (com.apple.WebKit.WebContent.Development crashed [pid=6113])

But again, no generated crash log.</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>1750406</commentid>
    <comment_count>6</comment_count>
    <who name="Chris Dumez">cdumez</who>
    <bug_when>2021-04-14 16:33:24 -0700</bug_when>
    <thetext>Looking at the system logs I found:
com.apple.WebKit	IPC	fault	16:32:20.017530-0700	9721	com.apple.WebKit.GPU.Development	Exiting: Received XPC event type: XPC_ERROR_CONNECTION_INVALID
com.apple.WebKit	IPC	fault	16:32:20.447207-0700	9728	com.apple.WebKit.GPU.Development	Exiting: Received XPC event type: XPC_ERROR_CONNECTION_INVALID
com.apple.WebKit	IPC	fault	16:32:20.592596-0700	9729	com.apple.WebKit.GPU.Development	Exiting: Received XPC event type: XPC_ERROR_CONNECTION_INVALID
com.apple.WebKit	IPC	fault	16:32:20.593085-0700	9658	com.apple.WebKit.GPU.Development	Exiting: Received XPC event type: XPC_ERROR_CONNECTION_INVALID
com.apple.WebKit	IPC	fault	16:32:20.593305-0700	9790	com.apple.WebKit.WebContent.Development	Exiting: Received XPC event type: XPC_ERROR_CONNECTION_INVALID</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>1750409</commentid>
    <comment_count>7</comment_count>
    <who name="Chris Dumez">cdumez</who>
    <bug_when>2021-04-14 16:39:46 -0700</bug_when>
    <thetext>It is super odd that r275921 would have regressed this on macOS given that it was an iOS patch (VISIBILITY_PROPAGATION_VIEW is only 1 on iOS).</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>1750411</commentid>
    <comment_count>8</comment_count>
    <who name="Chris Dumez">cdumez</who>
    <bug_when>2021-04-14 16:50:51 -0700</bug_when>
    <thetext>On the flakiness dashboard, we can see a crash as early as r273654. That said, it is true the the crash became super frequent recently (r275922).

r275922 is not platform-specific and is actually more likely to have caused this.</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>1750414</commentid>
    <comment_count>9</comment_count>
    <who name="Chris Dumez">cdumez</who>
    <bug_when>2021-04-14 16:55:53 -0700</bug_when>
    <thetext>As I suspected, the regression point is wrong. I can still reproduce the crash after reverting r275921 locally. I will try reverting r275922 next.</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>1750417</commentid>
    <comment_count>10</comment_count>
    <who name="Chris Dumez">cdumez</who>
    <bug_when>2021-04-14 17:05:15 -0700</bug_when>
    <thetext>(In reply to Chris Dumez from comment #9)
&gt; As I suspected, the regression point is wrong. I can still reproduce the
&gt; crash after reverting r275921 locally. I will try reverting r275922 next.

I can still reproduce after reverting r275922. I am thus not sure what caused this.</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>1750421</commentid>
    <comment_count>11</comment_count>
    <who name="Chris Dumez">cdumez</who>
    <bug_when>2021-04-14 17:16:13 -0700</bug_when>
    <thetext>Could be just flakiness but it looks like reverting both r275921 and r275922 makes the flaky crash go away. Both patches may impact when we decide to launch to GPUProcess so the issue could be timing-related. I&apos;ll investigate more tomorrow.</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>1750426</commentid>
    <comment_count>12</comment_count>
    <who name="Chris Dumez">cdumez</who>
    <bug_when>2021-04-14 17:23:42 -0700</bug_when>
    <thetext>(In reply to Chris Dumez from comment #11)
&gt; Could be just flakiness but it looks like reverting both r275921 and r275922
&gt; makes the flaky crash go away. Both patches may impact when we decide to
&gt; launch to GPUProcess so the issue could be timing-related. I&apos;ll investigate
&gt; more tomorrow.

I think this has to do with when we decide to initiate the connection to the GPUProcess.

r275921 is mostly an iOS patch but it does one thing on macOS:
- It delays the construction of the RemoteRenderingBackendProxy until we actually need it. It used to get constructed as soon as we constructed the WebPage, now it only gets constructed when the GPUProcess launches or when we start doing canvas rendering.

r275922 is related too. Before r275922, constructing the RemoteRenderingBackendProxy would initiate the GPUProcess connection right away (in the constructor). After r275922, eagerly constructing a RemoteRenderingBackendProxy does not actually initiate the GPUProcess connection. We only launch the GPUProcess when we actually do canvas rendering via the RemoteRenderingBackendProxy.

I think this is why reverting r275921 without reverting r275922 does not fix the crash.

That said, now I have to figure out why launching the GPUProcess when we start canvas drawing instead of on WebPage creation actually causes this flaky crash. Even weirder, it is not a real crash but an XPC error (XPC_ERROR_CONNECTION_INVALID).</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>1750443</commentid>
    <comment_count>13</comment_count>
    <who name="Robert Jenner">jenner</who>
    <bug_when>2021-04-14 18:58:43 -0700</bug_when>
    <thetext>I found two more tests that fail at the r275921 and r275922 revisions, but not r275920:

imported/w3c/web-platform-tests/webmessaging/with-ports/011.html
webgl/1.0.3/conformance/glsl/matrices/glsl-mat3-construction.html

As such, I have added these tests to the previous test expectations I set:
https://trac.webkit.org/changeset/275987/webkit</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>1750483</commentid>
    <comment_count>14</comment_count>
    <who name="Chris Dumez">cdumez</who>
    <bug_when>2021-04-14 22:05:40 -0700</bug_when>
    <thetext>I am not sure if this is the root cause but there is definitely an IPC ordering issue.

I see the WebProcess sending to the GPUProcess:
1. RemoteRenderingBackend::CreateImageBuffer
2. RemoteRenderingBackend::GetDataURLForImageBuffer

And the GPUProcess receiving then in the opposite order. As a result, the second IPC handling is unable to find the image buffer.

The first message is async and the second one is async so this is likely where the ordering issue comes from. Oddly though, we are using the IPC::SendOption::DispatchMessageEvenWhenWaitingForSyncReply flag when sending the CreateImageBuffer IPC, which *should* maintain ordering with sync messages.

That said, the receiver seems to be a threaded message receiver, which is not super common so it is possible we have a bug with this configuration.</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>1750485</commentid>
    <comment_count>15</comment_count>
    <who name="Chris Dumez">cdumez</who>
    <bug_when>2021-04-14 22:33:07 -0700</bug_when>
    <thetext>Adding Kimmo in cc as this seems to be using his new MessageReceiveQueue code (since RemoteRenderingBackend is a WorkQueueMessageReceiver).

With some local debugging, I can see there there seems to be an issue here in void Connection::processIncomingMessage(std::unique_ptr&lt;Decoder&gt; message):
```
if (auto* receiveQueue = m_receiveQueues.get(*message)) {
    receiveQueue-&gt;enqueueMessage(*this, WTFMove(message));
    return;
}
```

I would expect all RemoteRenderingBackend IPC messages to use this code path, since RemoteRenderingBackend is a WorkQueueMessageReceiver. However, I can see locally that the first couple of RemoteRenderingBackend IPC messages get a receiveQueue that is null. Starting with the 3rd RemoteRenderingBackend message, receiveQueue becomes non-null. Seems like we may have a race somewhere if my logging is correct.</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>1750488</commentid>
    <comment_count>16</comment_count>
    <who name="Chris Dumez">cdumez</who>
    <bug_when>2021-04-14 22:43:36 -0700</bug_when>
    <thetext>My patches did not actually change the timing of the GPUProcess launch in the context of the tests. There is something during the window.internals initialization that ends up launching the GPUProcess.

What did change with my patches is the timing of when the GPUConnectionToWebProcess::CreateRenderingBackend IPC get sent. It used to get sent as soon as the RemoteRenderingBackendProxy got created and now it only gets send right before the first IPC from the RemoteRenderingBackendProxy to the GPUProcess.

I think we&apos;ve had a IPC synchronization issue on GPUProcess side that became obvious after my patches because the IPCs are now so close together.

In particular, we send a GPUConnectionToWebProcess::CreateRenderingBackend async IPC that creates the RemoteRenderingBackend on the GPUProcess side, on the main thread. When constructed, the RemoteRenderingBackend registers itself as a WorkQueueMessageReceiver.

Right after the WebProcess sends IPC from the WebProcess to the RemoteRenderingBackend. When we get those IPC on the IPC thread, the RemoteRenderingBackend may not have registered itself as a WorkQueueMessageReceiver yet on the main thread. As a result, RemoteRenderingBackend IPC messages may get processed out of order because some of them use the MessageReceiveQueue and some early IPC messages don&apos;t.</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>1750494</commentid>
    <comment_count>17</comment_count>
      <attachid>426076</attachid>
    <who name="Chris Dumez">cdumez</who>
    <bug_when>2021-04-14 23:08:25 -0700</bug_when>
    <thetext>Created attachment 426076
Patch</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>1750497</commentid>
    <comment_count>18</comment_count>
    <who name="Kimmo Kinnunen">kkinnunen</who>
    <bug_when>2021-04-14 23:16:33 -0700</bug_when>
    <thetext>Yes, good debugging. That&apos;s the reason for the reordering.

There was multiple of these reordering bugs.
The series of changes related to MessageReceiveQueue fixed most of them, but this one remains as the FIXME mentions. So basically the Connection implementation is still such that messages are delivered in random order.

I have a wip patch locally that fixes this reordering. However, it&apos;s not entirely simple to solve since the Connection behavior when dispatching messages while waiting wrt all global Connection instances is a bit underspecified. E.g. it&apos;s a complex patch to solve if we want to preserve the behavior that *ALL* Connections get their messages dispatched in main thread while one connection waits on a particular message.</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>1750500</commentid>
    <comment_count>19</comment_count>
    <who name="Kimmo Kinnunen">kkinnunen</who>
    <bug_when>2021-04-14 23:22:18 -0700</bug_when>
    <thetext>The FIXME in:
// FIXME: The matches here come from
// m_messagesToDispatchWhileWaitingForSyncReply.</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>1750503</commentid>
    <comment_count>20</comment_count>
    <who name="Chris Dumez">cdumez</who>
    <bug_when>2021-04-14 23:42:31 -0700</bug_when>
    <thetext>(In reply to Kimmo Kinnunen from comment #19)
&gt; The FIXME in:
&gt; // FIXME: The matches here come from
&gt; // m_messagesToDispatchWhileWaitingForSyncReply.

Just focusing on this particular WorkQueueMessageReceiver issue, I think one possible fix at the IPC::Connection layer might be in Connection::processIncomingMessage(std::unique_ptr&lt;Decoder&gt; message):
```
if (auto* receiveQueue = m_receiveQueues.get(*message)) {
    receiveQueue-&gt;enqueueMessage(*this, WTFMove(message));
    return;
}
```

In the if scope, we should check if there are any pending messages in m_incomingMessages for this receiver name, if there are, then we could remove them from m_incomingMessages and then enqueue them on the receiveQueue *before* the current message. I think this would preserve ordering for async messages at least. We&apos;d likely need a similar fix for sync messages (maybe checking m_messagesToDispatchWhileWaitingForSyncReply instead of m_incomingMessages).

Let me know what you think. Again though, I&apos;d want to land the current patch (simple fix make the IPC sync) first to address crashes quickly. We can follow-up with a better / more generic solution.</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>1750515</commentid>
    <comment_count>21</comment_count>
    <who name="EWS">ews-feeder</who>
    <bug_when>2021-04-15 00:37:13 -0700</bug_when>
    <thetext>Committed r276007 (236559@main): &lt;https://commits.webkit.org/236559@main&gt;

All reviewed patches have been landed. Closing bug and clearing flags on attachment 426076.</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>1750521</commentid>
    <comment_count>22</comment_count>
    <who name="Kimmo Kinnunen">kkinnunen</who>
    <bug_when>2021-04-15 01:25:35 -0700</bug_when>
    <thetext>(In reply to Chris Dumez from comment #20)
&gt; In the if scope, we should check if there are any pending messages in
&gt; m_incomingMessages for this receiver name, if there are, then we could
&gt; remove them from m_incomingMessages and then enqueue them on the
&gt; receiveQueue *before* the current message.

I interpret your suggestion to mean the normal operation, e.g. the server is not waiting on a specific message. 

So the normal operation is already covered, or should be covered. If you&apos;re seeing the behavior in normal delivery, then I think something is still missing there and could be addressed simply. However, I tried to make this covered, so I don&apos;t know if this is the case.

In normal operation, if messages are scheduled to m_incomingMessages *before* the work queue is attached, this is business-as-usual. The m_incomingMessages are delivered when the work queue is attached, right at that moment, holding a lock. 

So if there are pending messages in m_incomingMessages, they should be processed at work queue install time and no additional changes should be needed in Connection::processIncomingMessage. (as per the intention, there could of course be bugs)



The reordering happens when server starts waiting for messages.
1) Normal operation queues messages to m_incomingMessages (regardless of options)
2) Wait starts
3a) Messages during wait are queued to m_incomingMessages, if they don&apos;t have options
3b) Messages during wait are queued to m_messagesToDispatchWhileWaitingForSyncReply if they have the specific option or if they&apos;re sync messages


So it is impossible to order messages correctly, if they pseudo-randomly get queued to two different queues. So even the messages with DispatchMessageEvenWhenWaitingForSyncReply</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>1750530</commentid>
    <comment_count>23</comment_count>
    <who name="Kimmo Kinnunen">kkinnunen</who>
    <bug_when>2021-04-15 01:52:00 -0700</bug_when>
    <thetext>(In reply to Chris Dumez from comment #14)
&gt; I am not sure if this is the root cause but there is definitely an IPC
&gt; ordering issue.
&gt; 
&gt; I see the WebProcess sending to the GPUProcess:
&gt; 1. RemoteRenderingBackend::CreateImageBuffer
&gt; 2. RemoteRenderingBackend::GetDataURLForImageBuffer
&gt; 
&gt; And the GPUProcess receiving then in the opposite order

This interpretation is a bit suspicious and perhaps warrants a double check.

The normal logical operation of creating a image buffer in Web Content process side is currently:

1) send(CreateImageBuffer())
2) waitForDidCreateImageBufferBackend()
3) do other operations

If we apply this logic to GetDataURL, without looking at the implementation it should be:

1) send(CreateImageBuffer())
2) waitForDidCreateImageBufferBackend()
3) send(GetDataURLForImageBuffer)


If this is the case, it should simply not be possible to observe any reordering, as the Web Content process will not send the GetDataURL message before GPU Process has gotten and replied the CreateImageBuffer.


However, there&apos;s no such requirement for CreateRemoteRenderingBackend + CreateImageBuffer, so I could imagine those being reordered.</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>1750607</commentid>
    <comment_count>24</comment_count>
    <who name="Chris Dumez">cdumez</who>
    <bug_when>2021-04-15 07:43:35 -0700</bug_when>
    <thetext>(In reply to Kimmo Kinnunen from comment #22)
&gt; (In reply to Chris Dumez from comment #20)
&gt; &gt; In the if scope, we should check if there are any pending messages in
&gt; &gt; m_incomingMessages for this receiver name, if there are, then we could
&gt; &gt; remove them from m_incomingMessages and then enqueue them on the
&gt; &gt; receiveQueue *before* the current message.
&gt; 
&gt; I interpret your suggestion to mean the normal operation, e.g. the server is
&gt; not waiting on a specific message. 
&gt; 
&gt; So the normal operation is already covered, or should be covered. If you&apos;re
&gt; seeing the behavior in normal delivery, then I think something is still
&gt; missing there and could be addressed simply. However, I tried to make this
&gt; covered, so I don&apos;t know if this is the case.
&gt; 
&gt; In normal operation, if messages are scheduled to m_incomingMessages
&gt; *before* the work queue is attached, this is business-as-usual. The
&gt; m_incomingMessages are delivered when the work queue is attached, right at
&gt; that moment, holding a lock. 

I guess you are talking about this code:
```
void Connection::addMessageReceiveQueue(MessageReceiveQueue&amp; receiveQueue, ReceiverName receiverName, uint64_t destinationID)
{
    auto locker = holdLock(m_incomingMessagesMutex);
    m_incomingMessages = filterWithMessageReceiveQueue(*this, receiveQueue, receiverName, destinationID, WTFMove(m_incomingMessages));
    m_receiveQueues.add(receiveQueue, receiverName, destinationID);
}
```

In particular the `filterWithMessageReceiveQueue()` part. I did not realize we already did that and it is great.

However, the RemoteRenderingBackend::CreateImageBuffer IPC gets sent with the IPC::SendOption::DispatchMessageEvenWhenWaitingForSyncReply flag. As a result, the message does not get appended to m_incomingMessages. Instead, it gets appended to m_messagesToDispatchWhileWaitingForSyncReply :)

As I mentioned earlier in one of my comments, we have to take care of both regular async messages (in m_incomingMessages) and sync messages / async messages with DispatchMessageEvenWhenWaitingForSyncReply flag (in m_messagesToDispatchWhileWaitingForSyncReply).
Seems like our code currently only deals with regular async messages?</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>1750766</commentid>
    <comment_count>25</comment_count>
    <who name="Chris Dumez">cdumez</who>
    <bug_when>2021-04-15 14:00:22 -0700</bug_when>
    <thetext>(In reply to Chris Dumez from comment #24)
&gt; (In reply to Kimmo Kinnunen from comment #22)
&gt; &gt; (In reply to Chris Dumez from comment #20)
&gt; &gt; &gt; In the if scope, we should check if there are any pending messages in
&gt; &gt; &gt; m_incomingMessages for this receiver name, if there are, then we could
&gt; &gt; &gt; remove them from m_incomingMessages and then enqueue them on the
&gt; &gt; &gt; receiveQueue *before* the current message.
&gt; &gt; 
&gt; &gt; I interpret your suggestion to mean the normal operation, e.g. the server is
&gt; &gt; not waiting on a specific message. 
&gt; &gt; 
&gt; &gt; So the normal operation is already covered, or should be covered. If you&apos;re
&gt; &gt; seeing the behavior in normal delivery, then I think something is still
&gt; &gt; missing there and could be addressed simply. However, I tried to make this
&gt; &gt; covered, so I don&apos;t know if this is the case.
&gt; &gt; 
&gt; &gt; In normal operation, if messages are scheduled to m_incomingMessages
&gt; &gt; *before* the work queue is attached, this is business-as-usual. The
&gt; &gt; m_incomingMessages are delivered when the work queue is attached, right at
&gt; &gt; that moment, holding a lock. 
&gt; 
&gt; I guess you are talking about this code:
&gt; ```
&gt; void Connection::addMessageReceiveQueue(MessageReceiveQueue&amp; receiveQueue,
&gt; ReceiverName receiverName, uint64_t destinationID)
&gt; {
&gt;     auto locker = holdLock(m_incomingMessagesMutex);
&gt;     m_incomingMessages = filterWithMessageReceiveQueue(*this, receiveQueue,
&gt; receiverName, destinationID, WTFMove(m_incomingMessages));
&gt;     m_receiveQueues.add(receiveQueue, receiverName, destinationID);
&gt; }
&gt; ```
&gt; 
&gt; In particular the `filterWithMessageReceiveQueue()` part. I did not realize
&gt; we already did that and it is great.
&gt; 
&gt; However, the RemoteRenderingBackend::CreateImageBuffer IPC gets sent with
&gt; the IPC::SendOption::DispatchMessageEvenWhenWaitingForSyncReply flag. As a
&gt; result, the message does not get appended to m_incomingMessages. Instead, it
&gt; gets appended to m_messagesToDispatchWhileWaitingForSyncReply :)
&gt; 
&gt; As I mentioned earlier in one of my comments, we have to take care of both
&gt; regular async messages (in m_incomingMessages) and sync messages / async
&gt; messages with DispatchMessageEvenWhenWaitingForSyncReply flag (in
&gt; m_messagesToDispatchWhileWaitingForSyncReply).
&gt; Seems like our code currently only deals with regular async messages?

Looks like I figured out a proper way to fix the ordering issue. I will address via https://bugs.webkit.org/show_bug.cgi?id=224623 and make the CreateImageBuffer IPC async again.</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>1752300</commentid>
    <comment_count>26</comment_count>
    <who name="Alexey Proskuryakov">ap</who>
    <bug_when>2021-04-20 09:44:25 -0700</bug_when>
    <thetext>*** Bug 224612 has been marked as a duplicate of this bug. ***</thetext>
  </long_desc>
      
          <attachment
              isobsolete="0"
              ispatch="1"
              isprivate="0"
          >
            <attachid>426076</attachid>
            <date>2021-04-14 23:08:25 -0700</date>
            <delta_ts>2021-04-15 04:42:42 -0700</delta_ts>
            <desc>Patch</desc>
            <filename>bug-224566-20210414230824.patch</filename>
            <type>text/plain</type>
            <size>10842</size>
            <attacher name="Chris Dumez">cdumez</attacher>
            
              <data encoding="base64">U3VidmVyc2lvbiBSZXZpc2lvbjogMjc2MDAwCmRpZmYgLS1naXQgYS9Tb3VyY2UvV2ViS2l0L0No
YW5nZUxvZyBiL1NvdXJjZS9XZWJLaXQvQ2hhbmdlTG9nCmluZGV4IDA1ZTNhOGI1YmQwYmU0ODMx
MzE5OWIxYWZjYzFiZGI2OTU0Y2YwMjQuLjljODc5YzI1YzQxYWE3NDJmODAzODIyNDE0ZDhkMjNm
Mzk3OWZkNDggMTAwNjQ0Ci0tLSBhL1NvdXJjZS9XZWJLaXQvQ2hhbmdlTG9nCisrKyBiL1NvdXJj
ZS9XZWJLaXQvQ2hhbmdlTG9nCkBAIC0xLDMgKzEsMzggQEAKKzIwMjEtMDQtMTQgIENocmlzIER1
bWV6ICA8Y2R1bWV6QGFwcGxlLmNvbT4KKworICAgICAgICBSRUdSRVNTSU9OKHIyNzU5MjEtcjI3
NTkyMik6IFsgd2syIF0gaHR0cC90ZXN0cy9zZWN1cml0eS93ZWJnbC1yZW1vdGUtcmVhZC1yZW1v
dGUtaW1hZ2UtYWxsb3dlZC13aXRoLWNyZWRlbnRpYWxzLmh0bWwgaXMgZmxha2V5IGNyYXNoaW5n
CisgICAgICAgIGh0dHBzOi8vYnVncy53ZWJraXQub3JnL3Nob3dfYnVnLmNnaT9pZD0yMjQ1NjYK
KyAgICAgICAgPHJkYXI6Ly9wcm9ibGVtLzc2NjU3MDA3PgorCisgICAgICAgIFJldmlld2VkIGJ5
IE5PQk9EWSAoT09QUyEpLgorCisgICAgICAgIHIyNzU5MjEtcjI3NTkyMiBjaGFuZ2VkIHRoZSB0
aW1pbmcgb2Ygd2hlbiB0aGUgV2ViUHJvY2Vzcy9SZW1vdGVSZW5kZXJpbmdCYWNrZW5kUHJveHkg
c2VuZHMgdGhlIEdQVUNvbm5lY3Rpb25Ub1dlYlByb2Nlc3M6OkNyZWF0ZVJlbmRlcmluZ0JhY2tl
bmQKKyAgICAgICAgSVBDIHRvIHRoZSBHUFVQcm9jZXNzLiBUaGlzIElQQyB1c2VkIHRvIGdldCBz
ZW50IGFzIHNvb24gYXMgdGhlIFJlbW90ZVJlbmRlcmluZ0JhY2tlbmRQcm94eSB3YXMgY29uc3Ry
dWN0ZWQuIEhvd2V2ZXIsIEkgZGVsYXllZCBzZW5kaW5nIGl0IHVudGlsCisgICAgICAgIHRoZSBS
ZW1vdGVSZW5kZXJpbmdCYWNrZW5kUHJveHkgYWN0dWFsbHkgbmVlZHMgdG8gY3JlYXRlIGFuIElt
YWdlQnVmZmVyIHRvIGF2b2lkIGxhdW5jaGluZyB0aGUgR1BVUHJvY2VzcyBwcmVtYXR1cmVseS4K
KworICAgICAgICBVbmZvcnR1bmF0ZWx5LCB0aGlzIGV4cG9zZWQgYSBwcmUtZXhpc3Rpbmcgc3lu
Y2hyb25pemF0aW9uIGlzc3VlIHdpdGggcmVnYXJkcyB0byB0aGUgUmVtb3RlUmVuZGVyaW5nQmFj
a2VuZCBJUEMuIEluIHBhcnRpY3VsYXIsIHRoZQorICAgICAgICBHUFVDb25uZWN0aW9uVG9XZWJQ
cm9jZXNzOjpDcmVhdGVSZW5kZXJpbmdCYWNrZW5kIHdhcyBnZXR0aW5nIHNlbnQgYXN5bmNocm9u
b3VzbHkgYW5kIHByb2Nlc3NlZCBieSB0aGUgR1BVUHJvY2VzcyBvbiB0aGUgbWFpbiB0aHJlYWQu
IFRoaXMgd291bGQKKyAgICAgICAgY2F1c2UgdGhlIGNyZWF0aW9uIG9mIHRoZSBSZW1vdGVSZW5k
ZXJpbmdCYWNrZW5kIG9iamVjdCBvbiB0aGUgR1BVUHJvY2VzcyBzaWRlLCB3aGljaCB3b3VsZCB0
aGVuIHJlZ2lzdGVyIGl0c2VsZiBhcyBhIFdvcmtRdWV1ZU1lc3NhZ2VSZWNlaXZlci4KKyAgICAg
ICAgVGhlIGlzc3VlIGlzIHRoYXQgdGhlIFdlYlByb2Nlc3Mgd2FzIHNlbmRpbmcgdGhlIFJlbW90
ZVJlbmRlcmluZ0JhY2tlbmQgSVBDIHNvIHF1aWNrbHkgYWZ0ZXIgdGhlIENyZWF0ZVJlbmRlcmlu
Z0JhY2tlbmQgSVBDIHRoYXQgd2hlbiB0aGUgSVBDIGlzCisgICAgICAgIHJlY2VpdmVkIChvbiB0
aGUgSVBDIHRocmVhZCksIHRoZSBSZW1vdGVSZW5kZXJpbmdCYWNrZW5kIGhhcyBub3QgaGFkIGEg
Y2hhbmdlIHRvIHJlZ2lzdGVyIGl0c2VsZiBhcyBhIFdvcmtRdWV1ZU1lc3NhZ2VSZWNlaXZlciB5
ZXQuIEFzIGEgcmVzdWx0LAorICAgICAgICBzb21lIG9mIHRoZSBlYXJseSBJUEMgd2FzIGdldHRp
bmcgZGlzcGF0Y2hlZCB0byB0aGUgbWFpbiB0aHJlYWQgYW5kIGxhdGVyIElQQyB3b3VsZCBnZXQg
c2VudCB0byB0aGUgUmVtb3RlUmVuZGVyaW5nQmFja2VuZCBXb3JrUXVldWUuIFRoaXMgd2FzCisg
ICAgICAgIGNhdXNpbmcgUmVtb3RlUmVuZGVyaW5nQmFja2VuZCB0byBnZXQgcHJvY2Vzc2VkIG91
dCBvZiBvcmRlci4KKworICAgICAgICBUbyBhZGRyZXNzIHRoZSBzeW5jaHJvbml6YXRpb24gaXNz
dWUgZm9yIG5vdywgSSBtYWRlIHRoZSBHUFVDb25uZWN0aW9uVG9XZWJQcm9jZXNzOjpDcmVhdGVS
ZW5kZXJpbmdCYWNrZW5kIHN5bmNocm9ub3VzLiBUaGlzIG1heSBub3QgYmUgdGhlCisgICAgICAg
IGJlc3QgZGVzaWduIGxvbmcgdGVybSBidXQgaXQgaXMgYW4gZWFzeSBhbmQgc2FmZSB3YXkgdG8g
cmVzb2x2ZSB0aGlzIGZvciBub3cuIEl0IG1ha2VzIHN1cmUgdGhlIFdlYlByb2Nlc3MgY2Fubm90
IHNlbmQgUmVtb3RlUmVuZGVyaW5nQmFja2VuZAorICAgICAgICBJUEMgdW50aWwgYWZ0ZXIgdGhl
IEdQVVByb2Nlc3MgaGFzIGNyZWF0ZWQgdGhlIFJlbW90ZVJlbmRlcmluZ0JhY2tlbmQgYW5kIHRo
ZSBSZW1vdGVSZW5kZXJpbmdCYWNrZW5kIG9iamVjdCBoYXMgcmVnaXN0ZXJlZCBpdHNlbGYgYXMg
YQorICAgICAgICBXb3JrUXVldWVNZXNzYWdlUmVjZWl2ZXIuIFdlIHByb2JhYmx5IHdhbnQgdG8g
Zm9sbG93LXVwIHdoZW4gd2UgY29tZSB1cCB3aXRoIGEgYmV0dGVyIGRlc2lnbi4gRm9yIG5vdyB0
aG91Z2gsIEkgaGF2ZSB2ZXJpZmllZCBsb2NhbGx5IHRoYXQgdGhpcworICAgICAgICBmaXhlcyB0
aGUgb3V0IG9mIG9yZGVyIElQQyBtZXNzYWdpbmcgQU5EIHRoZSBmbGFreSBjcmFzaGVzIG9uIHRo
ZXNlIGNhbnZhcyB0ZXN0cy4KKworICAgICAgICBObyBuZXcgdGVzdHMsIHVuc2tpcHBlZCBleGlz
dGluZyB0ZXN0cyB0aGF0IGFyZSBubyBsb25nZXIgZmxha3kuCisKKyAgICAgICAgKiBHUFVQcm9j
ZXNzL0dQVUNvbm5lY3Rpb25Ub1dlYlByb2Nlc3MuY3BwOgorICAgICAgICAoV2ViS2l0OjpHUFVD
b25uZWN0aW9uVG9XZWJQcm9jZXNzOjpjcmVhdGVSZW5kZXJpbmdCYWNrZW5kKToKKyAgICAgICAg
KiBHUFVQcm9jZXNzL0dQVUNvbm5lY3Rpb25Ub1dlYlByb2Nlc3MuaDoKKyAgICAgICAgKiBHUFVQ
cm9jZXNzL0dQVUNvbm5lY3Rpb25Ub1dlYlByb2Nlc3MubWVzc2FnZXMuaW46CisgICAgICAgICog
V2ViUHJvY2Vzcy9HUFUvZ3JhcGhpY3MvUmVtb3RlUmVuZGVyaW5nQmFja2VuZFByb3h5LmNwcDoK
KyAgICAgICAgKFdlYktpdDo6UmVtb3RlUmVuZGVyaW5nQmFja2VuZFByb3h5OjplbnN1cmVHUFVQ
cm9jZXNzQ29ubmVjdGlvbik6CisKIDIwMjEtMDQtMTQgIEJhc3VrZSBTdXp1a2kgIDxiYXN1a2Uu
c3V6dWtpQHNvbnkuY29tPgogCiAgICAgICAgIEZpeCBXS19FWFBPUlQgbWFjcm8gZm9yIGRlY2xz
cGVjIGNvbXBpbGVycwpkaWZmIC0tZ2l0IGEvU291cmNlL1dlYktpdC9HUFVQcm9jZXNzL0dQVUNv
bm5lY3Rpb25Ub1dlYlByb2Nlc3MuY3BwIGIvU291cmNlL1dlYktpdC9HUFVQcm9jZXNzL0dQVUNv
bm5lY3Rpb25Ub1dlYlByb2Nlc3MuY3BwCmluZGV4IDQ0MWFiMWM2ZTc2YmViMTdlNjU1MWJmYjQ5
NzI5OGQ2NzU0MzY5MjQuLjE2ZDExYzkyYjNmMjBjYWUxZTAwYjAyNThhMTFhNjM3YjNhMjFmZDYg
MTAwNjQ0Ci0tLSBhL1NvdXJjZS9XZWJLaXQvR1BVUHJvY2Vzcy9HUFVDb25uZWN0aW9uVG9XZWJQ
cm9jZXNzLmNwcAorKysgYi9Tb3VyY2UvV2ViS2l0L0dQVVByb2Nlc3MvR1BVQ29ubmVjdGlvblRv
V2ViUHJvY2Vzcy5jcHAKQEAgLTM2MiwxMiArMzYyLDEzIEBAIFJlbW90ZUltYWdlRGVjb2RlckFW
RlByb3h5JiBHUFVDb25uZWN0aW9uVG9XZWJQcm9jZXNzOjppbWFnZURlY29kZXJBVkZQcm94eSgp
CiB9CiAjZW5kaWYKIAotdm9pZCBHUFVDb25uZWN0aW9uVG9XZWJQcm9jZXNzOjpjcmVhdGVSZW5k
ZXJpbmdCYWNrZW5kKFJlbW90ZVJlbmRlcmluZ0JhY2tlbmRDcmVhdGlvblBhcmFtZXRlcnMmJiBj
cmVhdGlvblBhcmFtZXRlcnMpCit2b2lkIEdQVUNvbm5lY3Rpb25Ub1dlYlByb2Nlc3M6OmNyZWF0
ZVJlbmRlcmluZ0JhY2tlbmQoUmVtb3RlUmVuZGVyaW5nQmFja2VuZENyZWF0aW9uUGFyYW1ldGVy
cyYmIGNyZWF0aW9uUGFyYW1ldGVycywgQ29tcGxldGlvbkhhbmRsZXI8dm9pZCgpPiYmIGNvbXBs
ZXRpb25IYW5kbGVyKQogewogICAgIGF1dG8gYWRkUmVzdWx0ID0gbV9yZW1vdGVSZW5kZXJpbmdC
YWNrZW5kTWFwLmVuc3VyZShjcmVhdGlvblBhcmFtZXRlcnMuaWRlbnRpZmllciwgWyZdKCkgewog
ICAgICAgICByZXR1cm4gSVBDOjpTY29wZWRBY3RpdmVNZXNzYWdlUmVjZWl2ZVF1ZXVlIHsgUmVt
b3RlUmVuZGVyaW5nQmFja2VuZDo6Y3JlYXRlKCp0aGlzLCBXVEZNb3ZlKGNyZWF0aW9uUGFyYW1l
dGVycykpIH07CiAgICAgfSk7CiAgICAgQVNTRVJUX1VOVVNFRChhZGRSZXN1bHQsIGFkZFJlc3Vs
dC5pc05ld0VudHJ5KTsKKyAgICBjb21wbGV0aW9uSGFuZGxlcigpOwogfQogCiB2b2lkIEdQVUNv
bm5lY3Rpb25Ub1dlYlByb2Nlc3M6OnJlbGVhc2VSZW5kZXJpbmdCYWNrZW5kKFJlbmRlcmluZ0Jh
Y2tlbmRJZGVudGlmaWVyIHJlbmRlcmluZ0JhY2tlbmRJZGVudGlmaWVyKQpkaWZmIC0tZ2l0IGEv
U291cmNlL1dlYktpdC9HUFVQcm9jZXNzL0dQVUNvbm5lY3Rpb25Ub1dlYlByb2Nlc3MuaCBiL1Nv
dXJjZS9XZWJLaXQvR1BVUHJvY2Vzcy9HUFVDb25uZWN0aW9uVG9XZWJQcm9jZXNzLmgKaW5kZXgg
ZjJkZDk1MDJhYmUzNmEwMGM3MWUyODc2MzYwYjhiMmNlZjA0Zjg5Zi4uYWVhYTMxMzAwZWNkZTVh
MTUzNGI0Y2Q5NDk4ZjQ5NWYzMzUzYmE2YiAxMDA2NDQKLS0tIGEvU291cmNlL1dlYktpdC9HUFVQ
cm9jZXNzL0dQVUNvbm5lY3Rpb25Ub1dlYlByb2Nlc3MuaAorKysgYi9Tb3VyY2UvV2ViS2l0L0dQ
VVByb2Nlc3MvR1BVQ29ubmVjdGlvblRvV2ViUHJvY2Vzcy5oCkBAIC0xNjEsNyArMTYxLDcgQEAg
cHJpdmF0ZToKICNlbmRpZgogI2VuZGlmCiAKLSAgICB2b2lkIGNyZWF0ZVJlbmRlcmluZ0JhY2tl
bmQoUmVtb3RlUmVuZGVyaW5nQmFja2VuZENyZWF0aW9uUGFyYW1ldGVycyYmKTsKKyAgICB2b2lk
IGNyZWF0ZVJlbmRlcmluZ0JhY2tlbmQoUmVtb3RlUmVuZGVyaW5nQmFja2VuZENyZWF0aW9uUGFy
YW1ldGVycyYmLCBDb21wbGV0aW9uSGFuZGxlcjx2b2lkKCk+JiYpOwogICAgIHZvaWQgcmVsZWFz
ZVJlbmRlcmluZ0JhY2tlbmQoUmVuZGVyaW5nQmFja2VuZElkZW50aWZpZXIpOwogCiAjaWYgRU5B
QkxFKFdFQkdMKQpkaWZmIC0tZ2l0IGEvU291cmNlL1dlYktpdC9HUFVQcm9jZXNzL0dQVUNvbm5l
Y3Rpb25Ub1dlYlByb2Nlc3MubWVzc2FnZXMuaW4gYi9Tb3VyY2UvV2ViS2l0L0dQVVByb2Nlc3Mv
R1BVQ29ubmVjdGlvblRvV2ViUHJvY2Vzcy5tZXNzYWdlcy5pbgppbmRleCA3NzQ1NThlMTA3NTM3
YTgyMWYxNjBlZDMzYzZmOTU5ZjcwZmU4ZjNhLi44ZDFkYWY4NmRkYzI0ZmE4Y2ViOTJiMGU0Nzc4
MWE2Y2UyOGI5MjYzIDEwMDY0NAotLS0gYS9Tb3VyY2UvV2ViS2l0L0dQVVByb2Nlc3MvR1BVQ29u
bmVjdGlvblRvV2ViUHJvY2Vzcy5tZXNzYWdlcy5pbgorKysgYi9Tb3VyY2UvV2ViS2l0L0dQVVBy
b2Nlc3MvR1BVQ29ubmVjdGlvblRvV2ViUHJvY2Vzcy5tZXNzYWdlcy5pbgpAQCAtMjMsNyArMjMs
NyBAQAogI2lmIEVOQUJMRShHUFVfUFJPQ0VTUykKIAogbWVzc2FnZXMgLT4gR1BVQ29ubmVjdGlv
blRvV2ViUHJvY2VzcyBXYW50c0Rpc3BhdGNoTWVzc2FnZSB7Ci0gICAgdm9pZCBDcmVhdGVSZW5k
ZXJpbmdCYWNrZW5kKHN0cnVjdCBXZWJLaXQ6OlJlbW90ZVJlbmRlcmluZ0JhY2tlbmRDcmVhdGlv
blBhcmFtZXRlcnMgY3JlYXRpb25QYXJhbWV0ZXJzKQorICAgIHZvaWQgQ3JlYXRlUmVuZGVyaW5n
QmFja2VuZChzdHJ1Y3QgV2ViS2l0OjpSZW1vdGVSZW5kZXJpbmdCYWNrZW5kQ3JlYXRpb25QYXJh
bWV0ZXJzIGNyZWF0aW9uUGFyYW1ldGVycykgLT4gKCkgU3luY2hyb25vdXMKICAgICB2b2lkIFJl
bGVhc2VSZW5kZXJpbmdCYWNrZW5kKFdlYktpdDo6UmVuZGVyaW5nQmFja2VuZElkZW50aWZpZXIg
cmVuZGVyaW5nQmFja2VuZElkZW50aWZpZXIpCiAjaWYgRU5BQkxFKFdFQkdMKQogICAgIHZvaWQg
Q3JlYXRlR3JhcGhpY3NDb250ZXh0R0woc3RydWN0IFdlYkNvcmU6OkdyYXBoaWNzQ29udGV4dEdM
QXR0cmlidXRlcyBhdHRyaWJ1dGVzLCBXZWJLaXQ6OkdyYXBoaWNzQ29udGV4dEdMSWRlbnRpZmll
ciBncmFwaGljc0NvbnRleHRHTElkZW50aWZpZXIsIFdlYktpdDo6UmVuZGVyaW5nQmFja2VuZElk
ZW50aWZpZXIgcmVuZGVyaW5nQmFja2VuZElkZW50aWZpZXIsIElQQzo6U3RyZWFtQ29ubmVjdGlv
bkJ1ZmZlciBzdHJlYW0pCmRpZmYgLS1naXQgYS9Tb3VyY2UvV2ViS2l0L1dlYlByb2Nlc3MvR1BV
L2dyYXBoaWNzL1JlbW90ZVJlbmRlcmluZ0JhY2tlbmRQcm94eS5jcHAgYi9Tb3VyY2UvV2ViS2l0
L1dlYlByb2Nlc3MvR1BVL2dyYXBoaWNzL1JlbW90ZVJlbmRlcmluZ0JhY2tlbmRQcm94eS5jcHAK
aW5kZXggZDBkY2E2M2QyZDk4M2Q3MjJiNWE5MTJhN2I1NTM0MTIwMjA5NTJhMS4uNWRjNDJhNjMy
MjFiMTVjZjQ3ODRhNmU2YTdjMDc2ZDhlYWUxZDVhNyAxMDA2NDQKLS0tIGEvU291cmNlL1dlYktp
dC9XZWJQcm9jZXNzL0dQVS9ncmFwaGljcy9SZW1vdGVSZW5kZXJpbmdCYWNrZW5kUHJveHkuY3Bw
CisrKyBiL1NvdXJjZS9XZWJLaXQvV2ViUHJvY2Vzcy9HUFUvZ3JhcGhpY3MvUmVtb3RlUmVuZGVy
aW5nQmFja2VuZFByb3h5LmNwcApAQCAtNzUsNyArNzUsMTAgQEAgR1BVUHJvY2Vzc0Nvbm5lY3Rp
b24mIFJlbW90ZVJlbmRlcmluZ0JhY2tlbmRQcm94eTo6ZW5zdXJlR1BVUHJvY2Vzc0Nvbm5lY3Rp
b24oKQogICAgICAgICBhdXRvJiBncHVQcm9jZXNzQ29ubmVjdGlvbiA9IFdlYlByb2Nlc3M6OnNp
bmdsZXRvbigpLmVuc3VyZUdQVVByb2Nlc3NDb25uZWN0aW9uKCk7CiAgICAgICAgIGdwdVByb2Nl
c3NDb25uZWN0aW9uLmFkZENsaWVudCgqdGhpcyk7CiAgICAgICAgIGdwdVByb2Nlc3NDb25uZWN0
aW9uLm1lc3NhZ2VSZWNlaXZlck1hcCgpLmFkZE1lc3NhZ2VSZWNlaXZlcihNZXNzYWdlczo6UmVt
b3RlUmVuZGVyaW5nQmFja2VuZFByb3h5OjptZXNzYWdlUmVjZWl2ZXJOYW1lKCksIHJlbmRlcmlu
Z0JhY2tlbmRJZGVudGlmaWVyKCkudG9VSW50NjQoKSwgKnRoaXMpOwotICAgICAgICBncHVQcm9j
ZXNzQ29ubmVjdGlvbi5jb25uZWN0aW9uKCkuc2VuZChNZXNzYWdlczo6R1BVQ29ubmVjdGlvblRv
V2ViUHJvY2Vzczo6Q3JlYXRlUmVuZGVyaW5nQmFja2VuZChtX3BhcmFtZXRlcnMpLCAwLCBJUEM6
OlNlbmRPcHRpb246OkRpc3BhdGNoTWVzc2FnZUV2ZW5XaGVuV2FpdGluZ0ZvclN5bmNSZXBseSk7
CisgICAgICAgIC8vIFRoaXMgbWVzc2FnZSBpcyBzeW5jaHJvbm91cyB0byBlbnN1cmUgdGhhdCB0
aGUgUmVtb3RlUmVuZGVyaW5nQmFja2VuZCBoYXMgYmVlbiBjcmVhdGVkIGFuZCBoYXMgcmVnaXN0
ZXJlZCBpdHNlbGYgYXMgYSBXb3JrUXVldWVNZXNzYWdlUmVjZWl2ZXIgYmVmb3JlIHdlIHNlbmQg
aXQgSVBDLgorICAgICAgICAvLyBXaXRob3V0IHRoaXMgc3luY2hyb25pemF0aW9uLCBzb21lIElQ
QyBtZXNzYWdlcyBtYXkgZ2V0IHJlY2VpdmVkIGJ5IHRoZSBHUFVQcm9jZXNzIGJlZm9yZSB0aGUg
UmVtb3RlUmVuZGVyaW5nQmFja2VuZCBoYXMgcmVnaXN0ZXJlZCBpdHNlbGYgYXMgYSBXb3JrUXVl
dWVNZXNzYWdlUmVjZWl2ZXIKKyAgICAgICAgLy8gYW5kIElQQyBtYXkgZ2V0IHByb2Nlc3NlZCBv
dXQgb2Ygb3JkZXIuCisgICAgICAgIGdwdVByb2Nlc3NDb25uZWN0aW9uLmNvbm5lY3Rpb24oKS5z
ZW5kU3luYyhNZXNzYWdlczo6R1BVQ29ubmVjdGlvblRvV2ViUHJvY2Vzczo6Q3JlYXRlUmVuZGVy
aW5nQmFja2VuZChtX3BhcmFtZXRlcnMpLCBNZXNzYWdlczo6R1BVQ29ubmVjdGlvblRvV2ViUHJv
Y2Vzczo6Q3JlYXRlUmVuZGVyaW5nQmFja2VuZDo6UmVwbHkoKSwgMCk7CiAgICAgICAgIG1fZ3B1
UHJvY2Vzc0Nvbm5lY3Rpb24gPSBtYWtlV2Vha1B0cihncHVQcm9jZXNzQ29ubmVjdGlvbik7CiAg
ICAgfQogICAgIHJldHVybiAqbV9ncHVQcm9jZXNzQ29ubmVjdGlvbjsKZGlmZiAtLWdpdCBhL0xh
eW91dFRlc3RzL0NoYW5nZUxvZyBiL0xheW91dFRlc3RzL0NoYW5nZUxvZwppbmRleCBhZDAyYTUx
MTk4MGIwZjA1NjE4MDExOTdkMWRjY2JiNTYyYWM3ODBhLi5lMTg4NTc0MTg0MjMyMWZkZGY3NTdk
MmViMmE5MmJkODc2Zjc4YjhhIDEwMDY0NAotLS0gYS9MYXlvdXRUZXN0cy9DaGFuZ2VMb2cKKysr
IGIvTGF5b3V0VGVzdHMvQ2hhbmdlTG9nCkBAIC0xLDMgKzEsMTYgQEAKKzIwMjEtMDQtMTQgIENo
cmlzIER1bWV6ICA8Y2R1bWV6QGFwcGxlLmNvbT4KKworICAgICAgICBSRUdSRVNTSU9OKHIyNzU5
MjEtcjI3NTkyMik6IFsgd2syIF0gaHR0cC90ZXN0cy9zZWN1cml0eS93ZWJnbC1yZW1vdGUtcmVh
ZC1yZW1vdGUtaW1hZ2UtYWxsb3dlZC13aXRoLWNyZWRlbnRpYWxzLmh0bWwgaXMgZmxha2V5IGNy
YXNoaW5nCisgICAgICAgIGh0dHBzOi8vYnVncy53ZWJraXQub3JnL3Nob3dfYnVnLmNnaT9pZD0y
MjQ1NjYKKyAgICAgICAgPHJkYXI6Ly9wcm9ibGVtLzc2NjU3MDA3PgorCisgICAgICAgIFJldmll
d2VkIGJ5IE5PQk9EWSAoT09QUyEpLgorCisgICAgICAgIFVuc2tpcCBsYXlvdXQgdGVzdHMgdGhh
dCBzaG91bGQgbm8gbG9uZ2VyIGJlIGZsYWt5LgorCisgICAgICAgICogcGxhdGZvcm0vaW9zLXNp
bXVsYXRvci13azIvVGVzdEV4cGVjdGF0aW9uczoKKyAgICAgICAgKiBwbGF0Zm9ybS9tYWMtd2sy
L1Rlc3RFeHBlY3RhdGlvbnM6CisKIDIwMjEtMDQtMTQgIERpZWdvIFBpbm8gR2FyY2lhICA8ZHBp
bm9AaWdhbGlhLmNvbT4KIAogICAgICAgICBbR1RLXVtXUEVdIFVucmV2aWV3ZWQgdGVzdCBnYXJk
ZW5pbmcuIEFkZCBtaXNzaW5nIGltYWdlcyBhZnRlciByMjc1OTk2LgpkaWZmIC0tZ2l0IGEvTGF5
b3V0VGVzdHMvcGxhdGZvcm0vaW9zLXNpbXVsYXRvci13azIvVGVzdEV4cGVjdGF0aW9ucyBiL0xh
eW91dFRlc3RzL3BsYXRmb3JtL2lvcy1zaW11bGF0b3Itd2syL1Rlc3RFeHBlY3RhdGlvbnMKaW5k
ZXggYTYxYWY2NWVkMjM3ZmRkNTM5NmZkOTcwYzg0YzQ1MjMwNjg3M2M4Ny4uYjE4MjJhZjZiMTYw
YmQ1NWUwZjE0MWQxNmY0ZjQyN2E2MGExZjllNCAxMDA2NDQKLS0tIGEvTGF5b3V0VGVzdHMvcGxh
dGZvcm0vaW9zLXNpbXVsYXRvci13azIvVGVzdEV4cGVjdGF0aW9ucworKysgYi9MYXlvdXRUZXN0
cy9wbGF0Zm9ybS9pb3Mtc2ltdWxhdG9yLXdrMi9UZXN0RXhwZWN0YXRpb25zCkBAIC0xNjksNyAr
MTY5LDMgQEAgd2Via2l0Lm9yZy9iLzIyNDEyMSBjc3MyLjEvMjAxMTAzMjMvcmVwbGFjZWQtaW50
cmluc2ljLXJhdGlvLTAwMS5odG0gWyBQYXNzIEZhaWwKIHdlYnJ0Yy9wZWVyY29ubmVjdGlvbi1u
ZXctY2FuZGlkYXRlLXBhZ2UtY2FjaGUuaHRtbCBbIFBhc3MgQ3Jhc2ggXQogaHR0cC90ZXN0cy9u
YXZpZ2F0aW9uL3BhZ2UtY2FjaGUtbWVkaWFzdHJlYW0uaHRtbCBbIFBhc3MgQ3Jhc2ggXQogCi0j
IHdlYmtpdC5vcmcvYi8yMjQ1NjYgVGhlIGZvbGxvd2luZyB0aHJlZSB0ZXN0cyBhcmUgZmxha3kg
Y3Jhc2hpbmcgY2F1c2luZyBpc3N1ZXMgd2l0aCBFV1MvY29tbWl0IHF1ZXVlLgotaHR0cC90ZXN0
cy9zZWN1cml0eS93ZWJnbC1yZW1vdGUtcmVhZC1yZW1vdGUtaW1hZ2UtYWxsb3dlZC13aXRoLWNy
ZWRlbnRpYWxzLmh0bWwgWyBQYXNzIENyYXNoIF0KLWltcG9ydGVkL3czYy93ZWItcGxhdGZvcm0t
dGVzdHMvd2VibWVzc2FnaW5nL3dpdGgtcG9ydHMvMDExLmh0bWwgWyBQYXNzIENyYXNoIF0KLXdl
YmdsLzEuMC4zL2NvbmZvcm1hbmNlL2dsc2wvbWF0cmljZXMvZ2xzbC1tYXQzLWNvbnN0cnVjdGlv
bi5odG1sIFsgUGFzcyBDcmFzaCBdClwgTm8gbmV3bGluZSBhdCBlbmQgb2YgZmlsZQpkaWZmIC0t
Z2l0IGEvTGF5b3V0VGVzdHMvcGxhdGZvcm0vbWFjLXdrMi9UZXN0RXhwZWN0YXRpb25zIGIvTGF5
b3V0VGVzdHMvcGxhdGZvcm0vbWFjLXdrMi9UZXN0RXhwZWN0YXRpb25zCmluZGV4IGU4MGUwYmQ5
YWZlMGZlMTQ5NDkzZDMyOTkzNGU1YzQ2OTQ4MTY5N2QuLmU4ZTliZDJkYWEwYTEyM2UwYmRhYzE4
ZGFlY2I3ZGQzMTJlNzIyZTAgMTAwNjQ0Ci0tLSBhL0xheW91dFRlc3RzL3BsYXRmb3JtL21hYy13
azIvVGVzdEV4cGVjdGF0aW9ucworKysgYi9MYXlvdXRUZXN0cy9wbGF0Zm9ybS9tYWMtd2syL1Rl
c3RFeHBlY3RhdGlvbnMKQEAgLTEzNzcsOSArMTM3Nyw0IEBAIHdlYmtpdC5vcmcvYi8yMjQ0NjMg
WyBEZWJ1ZyBhcm02NCBdIHdlYnJ0Yy9kaXNhYmxlLWVuY3J5cHRpb24uaHRtbCBbIFBhc3MgVGlt
ZW91CiB3ZWJydGMvcGVlcmNvbm5lY3Rpb24tbmV3LWNhbmRpZGF0ZS1wYWdlLWNhY2hlLmh0bWwg
WyBQYXNzIENyYXNoIF0KIGh0dHAvdGVzdHMvbmF2aWdhdGlvbi9wYWdlLWNhY2hlLW1lZGlhc3Ry
ZWFtLmh0bWwgWyBQYXNzIENyYXNoIF0KIAotIyB3ZWJraXQub3JnL2IvMjI0NTY2IFRoZSBmb2xs
b3dpbmcgdGhyZWUgdGVzdHMgYXJlIGZsYWt5IGNyYXNoaW5nIGNhdXNpbmcgaXNzdWVzIHdpdGgg
RVdTL2NvbW1pdCBxdWV1ZS4KLWh0dHAvdGVzdHMvc2VjdXJpdHkvd2ViZ2wtcmVtb3RlLXJlYWQt
cmVtb3RlLWltYWdlLWFsbG93ZWQtd2l0aC1jcmVkZW50aWFscy5odG1sIFsgUGFzcyBDcmFzaCBd
Ci1pbXBvcnRlZC93M2Mvd2ViLXBsYXRmb3JtLXRlc3RzL3dlYm1lc3NhZ2luZy93aXRoLXBvcnRz
LzAxMS5odG1sIFsgUGFzcyBDcmFzaCBdCi13ZWJnbC8xLjAuMy9jb25mb3JtYW5jZS9nbHNsL21h
dHJpY2VzL2dsc2wtbWF0My1jb25zdHJ1Y3Rpb24uaHRtbCBbIFBhc3MgQ3Jhc2ggXQotCi13ZWJr
aXQub3JnL2IvMjIxOTg1IGZhc3QvbWVkaWFzdHJlYW0vYXVkaW8tdHJhY2stZW5hYmxlZC5odG1s
IFsgUGFzcyBGYWlsdXJlIF0KXCBObyBuZXdsaW5lIGF0IGVuZCBvZiBmaWxlCit3ZWJraXQub3Jn
L2IvMjIxOTg1IGZhc3QvbWVkaWFzdHJlYW0vYXVkaW8tdHJhY2stZW5hYmxlZC5odG1sIFsgUGFz
cyBGYWlsdXJlIF0K
</data>
<flag name="commit-queue"
          id="446290"
          type_id="3"
          status="-"
          setter="ews-feeder"
    />
          </attachment>
      

    </bug>

</bugzilla>