<?xml version="1.0" encoding="UTF-8" standalone="yes" ?>
<!DOCTYPE bugzilla SYSTEM "https://bugs.webkit.org/page.cgi?id=bugzilla.dtd">

<bugzilla version="5.0.4.1"
          urlbase="https://bugs.webkit.org/"
          
          maintainer="admin@webkit.org"
>

    <bug>
          <bug_id>228012</bug_id>
          
          <creation_ts>2021-07-15 18:06:15 -0700</creation_ts>
          <short_desc>REGRESSION(ANGLE+METAL): WebGL2 content low frame rate</short_desc>
          <delta_ts>2021-07-22 17:56:59 -0700</delta_ts>
          <reporter_accessible>1</reporter_accessible>
          <cclist_accessible>1</cclist_accessible>
          <classification_id>1</classification_id>
          <classification>Unclassified</classification>
          <product>WebKit</product>
          <component>WebGL</component>
          <version>Safari Technology Preview</version>
          <rep_platform>iPhone / iPad</rep_platform>
          <op_sys>iOS 14</op_sys>
          <bug_status>RESOLVED</bug_status>
          <resolution>FIXED</resolution>
          
          
          <bug_file_loc>https://noclip.website/#smg/DarkRoomGalaxy;ShareData=AZju&amp;UDsRrT3IyH97Uy9=3k]%5eQY7VbUi%29BDT:JHOVvKe=UwLZ+873GrUDmZ%29=2L</bug_file_loc>
          <status_whiteboard></status_whiteboard>
          <keywords>InRadar</keywords>
          <priority>P2</priority>
          <bug_severity>Normal</bug_severity>
          <target_milestone>---</target_milestone>
          <dependson>227059</dependson>
          <blocked>228192</blocked>
          <everconfirmed>1</everconfirmed>
          <reporter name="Jasper St. Pierre">jstpierre</reporter>
          <assigned_to name="Kyle Piddington">kpiddington</assigned_to>
          <cc>dino</cc>
    
    <cc>gsnedders</cc>
    
    <cc>kbr</cc>
    
    <cc>kkinnunen</cc>
    
    <cc>kpiddington</cc>
    
    <cc>webkit-bug-importer</cc>
          

      

      

      

          <comment_sort_order>oldest_to_newest</comment_sort_order>  
          <long_desc isprivate="0" >
    <commentid>1777401</commentid>
    <comment_count>0</comment_count>
    <who name="Jasper St. Pierre">jstpierre</who>
    <bug_when>2021-07-15 18:06:15 -0700</bug_when>
    <thetext>URL:  https://noclip.website/#smg/DarkRoomGalaxy;ShareData=AZju&amp;UDsRrT3IyH97Uy9=3k]%5eQY7VbUi%29BDT:JHOVvKe=UwLZ+873GrUDmZ%29=2L

According to a user of mine, this runs fine on iOS 14 (60 fps), but now runs poorly on iOS 15 Beta (10 fps).

The user was on an iPhone 12 Pro.</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>1777405</commentid>
    <comment_count>1</comment_count>
    <who name="Jasper St. Pierre">jstpierre</who>
    <bug_when>2021-07-15 18:32:15 -0700</bug_when>
    <thetext>According to the same user:

1. The low performance happens in many different scenes, not just the link provided.
2. The performance seems to act strangely -- e.g. the more things that appear on the screen, the *faster* the application seems to be.
3. This *only* happens in portrait mode. Landscape mode performs just fine.</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>1777483</commentid>
    <comment_count>2</comment_count>
    <who name="Sam Sneddon [:gsnedders]">gsnedders</who>
    <bug_when>2021-07-16 03:30:51 -0700</bug_when>
    <thetext>Performance is fine with the Metal backend disabled via experimental features, so regression is from enabling it.

To note: at least in the above case, this isn&apos;t per-se a regression in iOS 15, given WebGL 2 wasn&apos;t enabled in iOS 14. Might affect other WebGL content, however.</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>1777484</commentid>
    <comment_count>3</comment_count>
    <who name="Radar WebKit Bug Importer">webkit-bug-importer</who>
    <bug_when>2021-07-16 03:31:03 -0700</bug_when>
    <thetext>&lt;rdar://problem/80677758&gt;</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>1777551</commentid>
    <comment_count>4</comment_count>
    <who name="Kenneth Russell">kbr</who>
    <bug_when>2021-07-16 11:46:17 -0700</bug_when>
    <thetext>This might have been addressed by Bug 227059. Kyle or other Apple engineers, possible for you to test on iOS with top-of-tree?</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>1777557</commentid>
    <comment_count>5</comment_count>
    <who name="Kyle Piddington">kpiddington</who>
    <bug_when>2021-07-16 12:15:18 -0700</bug_when>
    <thetext>Repo&apos;d, I&apos;m seeing most of the slowdown happening with a &apos;bufferSubData&apos; call. As Jasper reports, landscape mode performs just fine, oddly enough.</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>1777586</commentid>
    <comment_count>6</comment_count>
    <who name="Kyle Piddington">kpiddington</who>
    <bug_when>2021-07-16 14:00:53 -0700</bug_when>
    <thetext>Unfortunately, this has not been addressed by Bug 227059. I&apos;m looking into it, however. Updates soon :).

I suspect this is some sort of run-ahead issue, but now we seem to be falling behind rather than running ahead.</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>1777632</commentid>
    <comment_count>7</comment_count>
    <who name="Jasper St. Pierre">jstpierre</who>
    <bug_when>2021-07-16 15:46:01 -0700</bug_when>
    <thetext>The bufferSubData is likely uploading my giant UBO for the frame. Either that or vertex data for some dynamic draws. All uploads happening every frame should be marked GL_DYNAMIC_DRAW (which I believe should be correct? Can never remember the difference between that and GL_STREAM_DRAW), and I try to keep UBO data limited to 65536-sized buffer pages.

If there&apos;s anything I&apos;m doing wrong on my side, please let me know.</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>1777672</commentid>
    <comment_count>8</comment_count>
      <attachid>433720</attachid>
    <who name="Kyle Piddington">kpiddington</who>
    <bug_when>2021-07-16 17:29:32 -0700</bug_when>
    <thetext>Created attachment 433720
Patch</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>1777687</commentid>
    <comment_count>9</comment_count>
    <who name="Kyle Piddington">kpiddington</who>
    <bug_when>2021-07-16 18:17:47 -0700</bug_when>
    <thetext>(In reply to Jasper St. Pierre from comment #7)
&gt; The bufferSubData is likely uploading my giant UBO for the frame. Either
&gt; that or vertex data for some dynamic draws. All uploads happening every
&gt; frame should be marked GL_DYNAMIC_DRAW (which I believe should be correct?
&gt; Can never remember the difference between that and GL_STREAM_DRAW), and I
&gt; try to keep UBO data limited to 65536-sized buffer pages.
&gt; 
&gt; If there&apos;s anything I&apos;m doing wrong on my side, please let me know.

While I&apos;m working on a patch, the best thing you can do for performance is limit the number of draw calls you submit. Try to keep submissions around 60 FPS, and you should see a performance improvement.</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>1777691</commentid>
    <comment_count>10</comment_count>
    <who name="Jasper St. Pierre">jstpierre</who>
    <bug_when>2021-07-16 18:24:59 -0700</bug_when>
    <thetext>Hmm, I do try to rely on requestAnimationFrame to manage my scheduling. I should be submitting around 200 draw calls in that link provided, which should hopefully be on the low side. It&apos;s possible there&apos;s a bug in my animation scheduling.

If that doesn&apos;t match what you&apos;re seeing, I&apos;d be happy to hear what might be going wrong, or how else I can rate-limit my draw calls submission.</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>1777766</commentid>
    <comment_count>11</comment_count>
      <attachid>433720</attachid>
    <who name="Dean Jackson">dino</who>
    <bug_when>2021-07-18 12:09:26 -0700</bug_when>
    <thetext>Comment on attachment 433720
Patch

Do we need to do this for macOS too?</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>1778063</commentid>
    <comment_count>12</comment_count>
    <who name="Kyle Piddington">kpiddington</who>
    <bug_when>2021-07-19 17:29:47 -0700</bug_when>
    <thetext>(In reply to Jasper St. Pierre from comment #10)
&gt; Hmm, I do try to rely on requestAnimationFrame to manage my scheduling. I
&gt; should be submitting around 200 draw calls in that link provided, which
&gt; should hopefully be on the low side. It&apos;s possible there&apos;s a bug in my
&gt; animation scheduling.
&gt; 
&gt; If that doesn&apos;t match what you&apos;re seeing, I&apos;d be happy to hear what might be
&gt; going wrong, or how else I can rate-limit my draw calls submission.

We definitely have a problem with our fencing code, though I&apos;m able to address that now. That patch will 

One thing I&apos;m interested in is your render target size when the iPhone is in portrait vs Landscape mode. I suspect we&apos;re getting a significantly larger render target.

This patch will address our bugs in the fencing code, so we don&apos;t run-ahead too much.</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>1778064</commentid>
    <comment_count>13</comment_count>
      <attachid>433837</attachid>
    <who name="Kyle Piddington">kpiddington</who>
    <bug_when>2021-07-19 17:32:08 -0700</bug_when>
    <thetext>Created attachment 433837
Patch</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>1778067</commentid>
    <comment_count>14</comment_count>
      <attachid>433839</attachid>
    <who name="Kyle Piddington">kpiddington</who>
    <bug_when>2021-07-19 17:41:07 -0700</bug_when>
    <thetext>Created attachment 433839
Patch</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>1778214</commentid>
    <comment_count>15</comment_count>
    <who name="Kyle Piddington">kpiddington</who>
    <bug_when>2021-07-20 11:27:28 -0700</bug_when>
    <thetext>(In reply to Kyle Piddington from comment #12)
&gt; (In reply to Jasper St. Pierre from comment #10)
&gt; &gt; Hmm, I do try to rely on requestAnimationFrame to manage my scheduling. I
&gt; &gt; should be submitting around 200 draw calls in that link provided, which
&gt; &gt; should hopefully be on the low side. It&apos;s possible there&apos;s a bug in my
&gt; &gt; animation scheduling.
&gt; &gt; 
&gt; &gt; If that doesn&apos;t match what you&apos;re seeing, I&apos;d be happy to hear what might be
&gt; &gt; going wrong, or how else I can rate-limit my draw calls submission.
&gt; 
&gt; We definitely have a problem with our fencing code, though I&apos;m able to
&gt; address that now. That patch will 
&gt; 
&gt; One thing I&apos;m interested in is your render target size when the iPhone is in
&gt; portrait vs Landscape mode. I suspect we&apos;re getting a significantly larger
&gt; render target.
&gt; 
&gt; This patch will address our bugs in the fencing code, so we don&apos;t run-ahead
&gt; too much.


One last thing: Locally, I&apos;m seeing more draw detail with the Metal backend. For example, in the Dark Room Galaxy sample, enabling and disabling the metal backend toggles whether or not the floor and floating pillars are textured. Could that account for the performance change?</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>1778224</commentid>
    <comment_count>16</comment_count>
      <attachid>433839</attachid>
    <who name="Dean Jackson">dino</who>
    <bug_when>2021-07-20 11:48:34 -0700</bug_when>
    <thetext>Comment on attachment 433839
Patch

This looks good but I think it might be in danger of breaking on future OS releases.</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>1778285</commentid>
    <comment_count>17</comment_count>
    <who name="Kenneth Russell">kbr</who>
    <bug_when>2021-07-20 14:52:09 -0700</bug_when>
    <thetext>Thanks Kyle for tracking down this tricky sandboxing issue that was affecting the fence implementation!</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>1778314</commentid>
    <comment_count>18</comment_count>
    <who name="EWS">ews-feeder</who>
    <bug_when>2021-07-20 16:38:01 -0700</bug_when>
    <thetext>Committed r280114 (239831@main): &lt;https://commits.webkit.org/239831@main&gt;

All reviewed patches have been landed. Closing bug and clearing flags on attachment 433839.</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>1778913</commentid>
    <comment_count>19</comment_count>
    <who name="Jasper St. Pierre">jstpierre</who>
    <bug_when>2021-07-22 17:11:48 -0700</bug_when>
    <thetext>Should I open a separate bug for the landscape vs. portrait issue, or should this be reopened to investigate that?</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>1778916</commentid>
    <comment_count>20</comment_count>
    <who name="Kyle Piddington">kpiddington</who>
    <bug_when>2021-07-22 17:39:34 -0700</bug_when>
    <thetext>(In reply to Jasper St. Pierre from comment #19)
&gt; Should I open a separate bug for the landscape vs. portrait issue, or should
&gt; this be reopened to investigate that?

Let&apos;s open a separate bug. 
Can you confirm with your user that the scenes are identical on iOS14 and iOS 15?
In addition, could you check the render target sizes on Landscape vs Portrait? I suspect we&apos;re rendering more in Portrait mode.</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>1778917</commentid>
    <comment_count>21</comment_count>
    <who name="Jasper St. Pierre">jstpierre</who>
    <bug_when>2021-07-22 17:41:25 -0700</bug_when>
    <thetext>I don&apos;t know any way to easily check the render target sizes. Would that be the drawingBufferWidth / drawingBufferHeight on the WebGL render context? I attempt to resize the canvas to fit the window content.</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>1778922</commentid>
    <comment_count>22</comment_count>
    <who name="Jasper St. Pierre">jstpierre</who>
    <bug_when>2021-07-22 17:43:49 -0700</bug_when>
    <thetext>Also, I filed https://bugs.webkit.org/show_bug.cgi?id=228208 for the portrait vs. landscape issue.</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>1778926</commentid>
    <comment_count>23</comment_count>
    <who name="Kenneth Russell">kbr</who>
    <bug_when>2021-07-22 17:56:59 -0700</bug_when>
    <thetext>Yes, drawingBufferWidth/drawingBufferHeight are the real pixel sizes of WebGL&apos;s back buffer.</thetext>
  </long_desc>
      
          <attachment
              isobsolete="1"
              ispatch="1"
              isprivate="0"
          >
            <attachid>433720</attachid>
            <date>2021-07-16 17:29:32 -0700</date>
            <delta_ts>2021-07-19 17:32:06 -0700</delta_ts>
            <desc>Patch</desc>
            <filename>bug-228012-20210716172932.patch</filename>
            <type>text/plain</type>
            <size>1845</size>
            <attacher name="Kyle Piddington">kpiddington</attacher>
            
              <data encoding="base64">U3VidmVyc2lvbiBSZXZpc2lvbjogMjgwMDA4CmRpZmYgLS1naXQgYS9Tb3VyY2UvV2ViS2l0L0No
YW5nZUxvZyBiL1NvdXJjZS9XZWJLaXQvQ2hhbmdlTG9nCmluZGV4IGM5ODliNjBkZDNlNmFlNWE5
MTEzZDEyMDdmN2Q1MmI3NzZjODYwMDUuLmVkY2JhODM2MTJmOTI3ZjgyMzVmZmZkZjU5YjUwOTk4
NmYxMWE4ZjMgMTAwNjQ0Ci0tLSBhL1NvdXJjZS9XZWJLaXQvQ2hhbmdlTG9nCisrKyBiL1NvdXJj
ZS9XZWJLaXQvQ2hhbmdlTG9nCkBAIC0xLDMgKzEsMjEgQEAKKzIwMjEtMDctMTYgIEt5bGUgUGlk
ZGluZ3RvbiAgPGtwaWRkaW5ndG9uQGFwcGxlLmNvbT4KKworICAgICAgICBSRUdSRVNTSU9OKEFO
R0xFK01FVEFMKTogV2ViR0wyIGNvbnRlbnQgbG93IGZyYW1lIHJhdGUKKyAgICAgICAgaHR0cHM6
Ly9idWdzLndlYmtpdC5vcmcvc2hvd19idWcuY2dpP2lkPTIyODAxMgorCisgICAgICAgIFdlYkdM
J3MgRnJhbWUtcGFjaW5nIGNvZGUgcmVsaWVzIG9uIE1UTFNoYXJlZEV2ZW50cy4gSG93ZXZlciwg
dGhlIGV2ZW50IGxpc3RlbmVyIGlzIG5vdCBjcmVhdGVkCisgICAgICAgIGR1ZSB0byBhIHNhbmRi
b3ggdmlvbGF0aW9uLiBUaGlzIG1lYW5zIHRoYXQgYW55IHRpbWUgd2UgaW50ZW5kIHRvIHdhaXQg
b24gYSBNVExGZW5jZSwgd2UgaW5zdGVhZCAKKyAgICAgICAgd2FpdCBhbGwgZml2ZSBzZWNvbmRz
IGZvciBhIGZpdmUgc2Vjb25kIHRpbWVvdXQuIAorCisgICAgICAgIE5vdCBhbGwgYXBwcyBhcmUg
aWxsLWJlaGF2ZWQuIEFwcHMgdGhhdCBydW4gYWhlYWQgYXQgYW4gdW5jYXBwZWQgZnJhbWVyYXRl
LCBob3dldmVyLCB3aWxsIHJ1biBpbnRvIG91ciBmZW5jaW5nCisgICAgICAgIHF1aWNrbHksIGFu
ZCBoYXZlIHRoZWlyIHBlcmZvcm1hbmNlIGRlZ3JhZGUuIAorCisgICAgICAgIFJldmlld2VkIGJ5
IE5PQk9EWSAoT09QUyEpLgorCisgICAgICAgIFRlc3Q6IHJ1biBhdHRhY2hlZCBVUkwsIGVuc3Vy
ZSBwZXJmb3JtYW5jZSBkb2VzIG5vdCBkZWdyYWRlLiAKKworICAgICAgICAqIFJlc291cmNlcy9T
YW5kYm94UHJvZmlsZXMvaW9zL2NvbS5hcHBsZS5XZWJLaXQuV2ViQ29udGVudC5zYi5pbjoKKwog
MjAyMS0wNy0xNiAgV2Vuc29uIEhzaWVoICA8d2Vuc29uX2hzaWVoQGFwcGxlLmNvbT4KIAogICAg
ICAgICBSRUdSRVNTSU9OIChyMjc3ODIwKTogQ2FuJ3Qgc2Nyb2xsIHVwIGFuZCBkb3duIHVzaW5n
IHRyYWNrcGFkIGluIE1haWwgbWVzc2FnZSB2aWV3ZXIgd2hlbiBpbiBTbGlkZSBPdmVyCmRpZmYg
LS1naXQgYS9Tb3VyY2UvV2ViS2l0L1Jlc291cmNlcy9TYW5kYm94UHJvZmlsZXMvaW9zL2NvbS5h
cHBsZS5XZWJLaXQuV2ViQ29udGVudC5zYi5pbiBiL1NvdXJjZS9XZWJLaXQvUmVzb3VyY2VzL1Nh
bmRib3hQcm9maWxlcy9pb3MvY29tLmFwcGxlLldlYktpdC5XZWJDb250ZW50LnNiLmluCmluZGV4
IGYxNjE0NTU4MWQ4YzJjYjMyNDg3MzNjMWJlYTg5NDkwNWZiNmYzNDcuLjU2MWZjYjE2NWI4OGNh
MDlhNDc3OGMyZTBhNWFiNzc0Y2I4MTA5ZWYgMTAwNjQ0Ci0tLSBhL1NvdXJjZS9XZWJLaXQvUmVz
b3VyY2VzL1NhbmRib3hQcm9maWxlcy9pb3MvY29tLmFwcGxlLldlYktpdC5XZWJDb250ZW50LnNi
LmluCisrKyBiL1NvdXJjZS9XZWJLaXQvUmVzb3VyY2VzL1NhbmRib3hQcm9maWxlcy9pb3MvY29t
LmFwcGxlLldlYktpdC5XZWJDb250ZW50LnNiLmluCkBAIC00NTEsNiArNDUxLDcgQEAKICAgICAg
ICAgICAgICAgICAgICAgMzUKICAgICAgICAgICAgICAgICAgICAgMzYKICAgICAgICAgICAgICAg
ICAgICAgMzgKKyAgICAgICAgICAgICAgICAgICAgNDEKICAgICAgICAgICAgICAgICApCiAgICAg
ICAgICAgICApCiAgICAgICAgICkK
</data>

          </attachment>
          <attachment
              isobsolete="1"
              ispatch="1"
              isprivate="0"
          >
            <attachid>433837</attachid>
            <date>2021-07-19 17:32:08 -0700</date>
            <delta_ts>2021-07-19 17:41:06 -0700</delta_ts>
            <desc>Patch</desc>
            <filename>bug-228012-20210719173207.patch</filename>
            <type>text/plain</type>
            <size>1880</size>
            <attacher name="Kyle Piddington">kpiddington</attacher>
            
              <data encoding="base64">U3VidmVyc2lvbiBSZXZpc2lvbjogMjgwMDQ1CmRpZmYgLS1naXQgYS9Tb3VyY2UvV2ViS2l0L0No
YW5nZUxvZyBiL1NvdXJjZS9XZWJLaXQvQ2hhbmdlTG9nCmluZGV4IDFlZDA5YmI2M2JjYjAwYzEw
ODRmMTM0OGMxZDcyYTljYTI1NDRjY2YuLjNkN2QyNjQ2ZTBkNDcxNTQ1Njk0MTkxMDUzNjcyNDhk
NWNlNmQ1YjUgMTAwNjQ0Ci0tLSBhL1NvdXJjZS9XZWJLaXQvQ2hhbmdlTG9nCisrKyBiL1NvdXJj
ZS9XZWJLaXQvQ2hhbmdlTG9nCkBAIC0xLDMgKzEsMTQgQEAKKzIwMjEtMDctMTkgIEt5bGUgUGlk
ZGluZ3RvbiAgPGtwaWRkaW5ndG9uQGFwcGxlLmNvbT4KKworICAgICAgICBSRUdSRVNTSU9OKEFO
R0xFK01FVEFMKTogV2ViR0wyIGNvbnRlbnQgbG93IGZyYW1lIHJhdGUKKyAgICAgICAgaHR0cHM6
Ly9idWdzLndlYmtpdC5vcmcvc2hvd19idWcuY2dpP2lkPTIyODAxMgorCisgICAgICAgIFJldmll
d2VkIGJ5IERlYW4gSmFja3Nvbi4KKyAgICAgICAgCisgICAgICAgIEFkZCBBZGRpdGlvbmFsIHJ1
bGVzLCBhbGxvd2luZyBzaGFyZWQgZXZlbnRzIHRvIHdvcmsgaW4gV2Via2l0CisKKyAgICAgICAg
KiBSZXNvdXJjZXMvU2FuZGJveFByb2ZpbGVzL2lvcy9jb20uYXBwbGUuV2ViS2l0LldlYkNvbnRl
bnQuc2IuaW46CisKIDIwMjEtMDctMTkgIFdlbnNvbiBIc2llaCAgPHdlbnNvbl9oc2llaEBhcHBs
ZS5jb20+CiAKICAgICAgICAgUmV2ZXJ0IGEgZGVidWcgYXNzZXJ0aW9uIHRoYXQgd2FzIGFkZGVk
IGluIHIyODAwMTkKZGlmZiAtLWdpdCBhL1NvdXJjZS9XZWJLaXQvUmVzb3VyY2VzL1NhbmRib3hQ
cm9maWxlcy9pb3MvY29tLmFwcGxlLldlYktpdC5XZWJDb250ZW50LnNiLmluIGIvU291cmNlL1dl
YktpdC9SZXNvdXJjZXMvU2FuZGJveFByb2ZpbGVzL2lvcy9jb20uYXBwbGUuV2ViS2l0LldlYkNv
bnRlbnQuc2IuaW4KaW5kZXggZjE2MTQ1NTgxZDhjMmNiMzI0ODczM2MxYmVhODk0OTA1ZmI2ZjM0
Ny4uZThiNTJhNjY3MzM4NDU2MTVhMzRhOTUyYzk5MzA4ZTQwMDJjZjY2ZSAxMDA2NDQKLS0tIGEv
U291cmNlL1dlYktpdC9SZXNvdXJjZXMvU2FuZGJveFByb2ZpbGVzL2lvcy9jb20uYXBwbGUuV2Vi
S2l0LldlYkNvbnRlbnQuc2IuaW4KKysrIGIvU291cmNlL1dlYktpdC9SZXNvdXJjZXMvU2FuZGJv
eFByb2ZpbGVzL2lvcy9jb20uYXBwbGUuV2ViS2l0LldlYkNvbnRlbnQuc2IuaW4KQEAgLTQyNSw2
ICs0MjUsMTIgQEAKICAgICAgICAgICAgIChkZW55ICh3aXRoIHRlbGVtZXRyeSkKICAgICAgICAg
ICAgICAgICBpb2tpdC1hc3luYy1leHRlcm5hbC1tZXRob2QKICAgICAgICAgICAgICAgICBpb2tp
dC1leHRlcm5hbC10cmFwKQorICAgICAgICAgICAgKGFsbG93IGlva2l0LWFzeW5jLWV4dGVybmFs
LW1ldGhvZAorICAgICAgICAgICAgICAgICAgICAoaW9raXQtbWV0aG9kLW51bWJlcgorICAgICAg
ICAgICAgICAgICAgICAgICAgNDAKKyAgICAgICAgICAgICAgICAgICAgICAgIDQzCisgICAgICAg
ICAgICAgICAgICAgICkKKyAgICAgICAgICAgICkKICAgICAgICAgICAgIChkZW55ICh3aXRoIHRl
bGVtZXRyeSkgKHdpdGggbWVzc2FnZSAiSU9TdXJmYWNlUm9vdFVzZXJDbGllbnQiKQogICAgICAg
ICAgICAgICAgIGlva2l0LWV4dGVybmFsLW1ldGhvZCkKICAgICAgICAgICAgIChhbGxvdyBpb2tp
dC1leHRlcm5hbC1tZXRob2QKQEAgLTQ1MSw2ICs0NTcsOSBAQAogICAgICAgICAgICAgICAgICAg
ICAzNQogICAgICAgICAgICAgICAgICAgICAzNgogICAgICAgICAgICAgICAgICAgICAzOAorICAg
ICAgICAgICAgICAgICAgICAzOQorICAgICAgICAgICAgICAgICAgICA0MAorICAgICAgICAgICAg
ICAgICAgICA0MQogICAgICAgICAgICAgICAgICkKICAgICAgICAgICAgICkKICAgICAgICAgKQo=
</data>

          </attachment>
          <attachment
              isobsolete="0"
              ispatch="1"
              isprivate="0"
          >
            <attachid>433839</attachid>
            <date>2021-07-19 17:41:07 -0700</date>
            <delta_ts>2021-07-20 16:38:02 -0700</delta_ts>
            <desc>Patch</desc>
            <filename>bug-228012-20210719174107.patch</filename>
            <type>text/plain</type>
            <size>1852</size>
            <attacher name="Kyle Piddington">kpiddington</attacher>
            
              <data encoding="base64">U3VidmVyc2lvbiBSZXZpc2lvbjogMjgwMDQ1CmRpZmYgLS1naXQgYS9Tb3VyY2UvV2ViS2l0L0No
YW5nZUxvZyBiL1NvdXJjZS9XZWJLaXQvQ2hhbmdlTG9nCmluZGV4IDFlZDA5YmI2M2JjYjAwYzEw
ODRmMTM0OGMxZDcyYTljYTI1NDRjY2YuLjNkN2QyNjQ2ZTBkNDcxNTQ1Njk0MTkxMDUzNjcyNDhk
NWNlNmQ1YjUgMTAwNjQ0Ci0tLSBhL1NvdXJjZS9XZWJLaXQvQ2hhbmdlTG9nCisrKyBiL1NvdXJj
ZS9XZWJLaXQvQ2hhbmdlTG9nCkBAIC0xLDMgKzEsMTQgQEAKKzIwMjEtMDctMTkgIEt5bGUgUGlk
ZGluZ3RvbiAgPGtwaWRkaW5ndG9uQGFwcGxlLmNvbT4KKworICAgICAgICBSRUdSRVNTSU9OKEFO
R0xFK01FVEFMKTogV2ViR0wyIGNvbnRlbnQgbG93IGZyYW1lIHJhdGUKKyAgICAgICAgaHR0cHM6
Ly9idWdzLndlYmtpdC5vcmcvc2hvd19idWcuY2dpP2lkPTIyODAxMgorCisgICAgICAgIFJldmll
d2VkIGJ5IERlYW4gSmFja3Nvbi4KKyAgICAgICAgCisgICAgICAgIEFkZCBBZGRpdGlvbmFsIHJ1
bGVzLCBhbGxvd2luZyBzaGFyZWQgZXZlbnRzIHRvIHdvcmsgaW4gV2Via2l0CisKKyAgICAgICAg
KiBSZXNvdXJjZXMvU2FuZGJveFByb2ZpbGVzL2lvcy9jb20uYXBwbGUuV2ViS2l0LldlYkNvbnRl
bnQuc2IuaW46CisKIDIwMjEtMDctMTkgIFdlbnNvbiBIc2llaCAgPHdlbnNvbl9oc2llaEBhcHBs
ZS5jb20+CiAKICAgICAgICAgUmV2ZXJ0IGEgZGVidWcgYXNzZXJ0aW9uIHRoYXQgd2FzIGFkZGVk
IGluIHIyODAwMTkKZGlmZiAtLWdpdCBhL1NvdXJjZS9XZWJLaXQvUmVzb3VyY2VzL1NhbmRib3hQ
cm9maWxlcy9pb3MvY29tLmFwcGxlLldlYktpdC5XZWJDb250ZW50LnNiLmluIGIvU291cmNlL1dl
YktpdC9SZXNvdXJjZXMvU2FuZGJveFByb2ZpbGVzL2lvcy9jb20uYXBwbGUuV2ViS2l0LldlYkNv
bnRlbnQuc2IuaW4KaW5kZXggZjE2MTQ1NTgxZDhjMmNiMzI0ODczM2MxYmVhODk0OTA1ZmI2ZjM0
Ny4uOTRkMGQyNDExNzc0MTZhMTQyMGMzNmNhNTkxZjdkNjdkNDExYzAwZiAxMDA2NDQKLS0tIGEv
U291cmNlL1dlYktpdC9SZXNvdXJjZXMvU2FuZGJveFByb2ZpbGVzL2lvcy9jb20uYXBwbGUuV2Vi
S2l0LldlYkNvbnRlbnQuc2IuaW4KKysrIGIvU291cmNlL1dlYktpdC9SZXNvdXJjZXMvU2FuZGJv
eFByb2ZpbGVzL2lvcy9jb20uYXBwbGUuV2ViS2l0LldlYkNvbnRlbnQuc2IuaW4KQEAgLTQyNSw2
ICs0MjUsMTEgQEAKICAgICAgICAgICAgIChkZW55ICh3aXRoIHRlbGVtZXRyeSkKICAgICAgICAg
ICAgICAgICBpb2tpdC1hc3luYy1leHRlcm5hbC1tZXRob2QKICAgICAgICAgICAgICAgICBpb2tp
dC1leHRlcm5hbC10cmFwKQorICAgICAgICAgICAgKGFsbG93IGlva2l0LWFzeW5jLWV4dGVybmFs
LW1ldGhvZAorICAgICAgICAgICAgICAgICAgICAoaW9raXQtbWV0aG9kLW51bWJlcgorICAgICAg
ICAgICAgICAgICAgICAgICAgNDAKKyAgICAgICAgICAgICAgICAgICAgKQorICAgICAgICAgICAg
KQogICAgICAgICAgICAgKGRlbnkgKHdpdGggdGVsZW1ldHJ5KSAod2l0aCBtZXNzYWdlICJJT1N1
cmZhY2VSb290VXNlckNsaWVudCIpCiAgICAgICAgICAgICAgICAgaW9raXQtZXh0ZXJuYWwtbWV0
aG9kKQogICAgICAgICAgICAgKGFsbG93IGlva2l0LWV4dGVybmFsLW1ldGhvZApAQCAtNDUxLDYg
KzQ1Niw5IEBACiAgICAgICAgICAgICAgICAgICAgIDM1CiAgICAgICAgICAgICAgICAgICAgIDM2
CiAgICAgICAgICAgICAgICAgICAgIDM4CisgICAgICAgICAgICAgICAgICAgIDM5CisgICAgICAg
ICAgICAgICAgICAgIDQwCisgICAgICAgICAgICAgICAgICAgIDQxCiAgICAgICAgICAgICAgICAg
KQogICAgICAgICAgICAgKQogICAgICAgICApCg==
</data>

          </attachment>
      

    </bug>

</bugzilla>