202928 2019-10-14 08:27:39 -0700 layout step not summarizes results to stdout when running with "--report" 2019-10-14 14:53:06 -0700 1 1 1 Unclassified WebKit Tools / Tests WebKit Nightly Build Unspecified Unspecified NEW https://bugs.webkit.org/show_bug.cgi?id=202639 https://bugs.webkit.org/show_bug.cgi?id=202955 P2 Normal --- 1 clopez jbedard aakash_jain aperez bugs-noreply dewei_zhu ews-watchlist glenn jbedard lingho magomez oldest_to_newest 1579632 0 clopez 2019-10-14 08:27:39 -0700 Since r250966 the layout step for GTK and WPE is not longer printing the results in stdout. Previously, when this step finished it printed something like this: http://sprunge.us/XZjjH4 than then the build master parsed and put on the step for the layout tests as " 68 failures 80 new passes 11 flakes 14 missing results " <-- https://build.webkit.org/builders/GTK%20Linux%2064-bit%20Release%20%28Tests%29/builds/11560 But since r250966 the layout test step runs with a new parameter "--report https://results.webkit.org" and it not longer prints this to stdout. So in the results of the buildbot the summary for the step is empty. For example: https://build.webkit.org/builders/GTK%20Linux%2064-bit%20Release%20%28Tests%29/builds/11603 <-- it not longer has the summary of the number of failures in the layout test step. 1579643 1 clopez 2019-10-14 08:56:00 -0700 Seems the bug happens when the layout step fails to upload. Our bots are giving this error: 05:10:27.819 123196 Uploading to https://results.webkit.org ... 05:10:27.892 123196 Starting new HTTPS connection (1): results.webkit.org:443 05:10:28.207 123196 Failed to upload to https://results.webkit.org, results server not online (likely caused because results.webkit.org has an invalid SSL certificate, signed by an unknown CA) $ gnutls-cli results.webkit.org:443 Processed 124 CA certificate(s). Resolving 'results.webkit.org:443'... Connecting to '17.32.208.166:443'... - Certificate type: X.509 - Got a certificate list of 1 certificates. - Certificate[0] info: - subject `C=US,ST=California,O=Apple Inc.,OU=management:idms.group.764034,CN=results.webkit.org', issuer `C=US,ST=California,O=Apple Inc.,CN=Apple Public Server RSA CA 12 - G1', serial 0x4c75add9335798d877e2d8088da7118f, RSA key 2048 bits, signed using RSA-SHA256, activated `2019-09-03 22:15:05 UTC', expires `2021-10-02 22:15:05 UTC', pin-sha256="HWRWOw3Y56CfV1OWXgeT3mEren7q+ACX3K8c03MCTIc=" Public Key ID: sha1:421918b7a3df5b5b160b2aa2a2ae6dbb6c97a279 sha256:1d64563b0dd8e7a09f5753965e0793de612b7a7eeaf80097dcaf1cd373024c87 Public Key PIN: pin-sha256:HWRWOw3Y56CfV1OWXgeT3mEren7q+ACX3K8c03MCTIc= - Status: The certificate is NOT trusted. The certificate issuer is unknown. *** PKI verification of server certificate failed... *** Fatal error: Error in the certificate. 1579645 2 jbedard 2019-10-14 09:09:15 -0700 webkitpy tests are probably the right example for getting to the bottom of this: https://build.webkit.org/builders/GTK%20Linux%2064-bit%20Release%20%28Tests%29/builds/11603/steps/webkitpy-test/logs/stdio Looks like GTK machines can't reach results.webkit.org. WinCario has the same problem: https://build.webkit.org/builders/WPE%20Linux%2064-bit%20Release%20%28Tests%29/builds/15688/steps/webkitpy-test/logs/stdio Can those machines reach https://results.webkit.org? You should be able to reproduce this with 'test-webkitpy --report https://results.webkit.org' (note that without the API key, which the buildbot master servers, this request will fail, but it should explicitly say that it failed because of the API key, rather than 'results server not online'. I did some work last week, and I had a personal machine off the Apple network that was able to report, so I'm surprised we're having this issue. 1579646 3 jbedard 2019-10-14 09:11:22 -0700 (In reply to Carlos Alberto Lopez Perez from comment #1) > Seems the bug happens when the layout step fails to upload. > > Our bots are giving this error: > > 05:10:27.819 123196 Uploading to https://results.webkit.org ... > 05:10:27.892 123196 Starting new HTTPS connection (1): results.webkit.org:443 > 05:10:28.207 123196 Failed to upload to https://results.webkit.org, > results server not online > > (likely caused because results.webkit.org has an invalid SSL certificate, > signed by an unknown CA) > > $ gnutls-cli results.webkit.org:443 > Processed 124 CA certificate(s). > Resolving 'results.webkit.org:443'... > Connecting to '17.32.208.166:443'... > - Certificate type: X.509 > - Got a certificate list of 1 certificates. > - Certificate[0] info: > - subject `C=US,ST=California,O=Apple > Inc.,OU=management:idms.group.764034,CN=results.webkit.org', issuer > `C=US,ST=California,O=Apple Inc.,CN=Apple Public Server RSA CA 12 - G1', > serial 0x4c75add9335798d877e2d8088da7118f, RSA key 2048 bits, signed using > RSA-SHA256, activated `2019-09-03 22:15:05 UTC', expires `2021-10-02 > 22:15:05 UTC', pin-sha256="HWRWOw3Y56CfV1OWXgeT3mEren7q+ACX3K8c03MCTIc=" > Public Key ID: > sha1:421918b7a3df5b5b160b2aa2a2ae6dbb6c97a279 > sha256:1d64563b0dd8e7a09f5753965e0793de612b7a7eeaf80097dcaf1cd373024c87 > Public Key PIN: > pin-sha256:HWRWOw3Y56CfV1OWXgeT3mEren7q+ACX3K8c03MCTIc= > > - Status: The certificate is NOT trusted. The certificate issuer is unknown. > *** PKI verification of server certificate failed... > *** Fatal error: Error in the certificate. I temporarily turned off the verification in r250997 because our Catalina bots where encountering a similar problem. 1579648 4 clopez 2019-10-14 09:14:19 -0700 (In reply to Jonathan Bedard from comment #2) > webkitpy tests are probably the right example for getting to the bottom of > this: > > https://build.webkit.org/builders/GTK%20Linux%2064- > bit%20Release%20%28Tests%29/builds/11603/steps/webkitpy-test/logs/stdio > > Looks like GTK machines can't reach results.webkit.org. WinCario has the > same problem: > > https://build.webkit.org/builders/WPE%20Linux%2064- > bit%20Release%20%28Tests%29/builds/15688/steps/webkitpy-test/logs/stdio > > Can those machines reach https://results.webkit.org? You should be able to > reproduce this with 'test-webkitpy --report https://results.webkit.org' > (note that without the API key, which the buildbot master servers, this > request will fail, but it should explicitly say that it failed because of > the API key, rather than 'results server not online'. > > I did some work last week, and I had a personal machine off the Apple > network that was able to report, so I'm surprised we're having this issue. They can reach it, but they can't recognized the certificate. It seems that the server hosting results.webkit.org is missing to send the intermediate certificates of the chain. See: https://www.sslshopper.com/ssl-checker.html#hostname=results.webkit.org 1579654 5 clopez 2019-10-14 09:37:45 -0700 (In reply to Jonathan Bedard from comment #3) > (In reply to Carlos Alberto Lopez Perez from comment #1) > > Seems the bug happens when the layout step fails to upload. > > > > Our bots are giving this error: > > > > 05:10:27.819 123196 Uploading to https://results.webkit.org ... > > 05:10:27.892 123196 Starting new HTTPS connection (1): results.webkit.org:443 > > 05:10:28.207 123196 Failed to upload to https://results.webkit.org, > > results server not online > > > > (likely caused because results.webkit.org has an invalid SSL certificate, > > signed by an unknown CA) > > > > $ gnutls-cli results.webkit.org:443 > > Processed 124 CA certificate(s). > > Resolving 'results.webkit.org:443'... > > Connecting to '17.32.208.166:443'... > > - Certificate type: X.509 > > - Got a certificate list of 1 certificates. > > - Certificate[0] info: > > - subject `C=US,ST=California,O=Apple > > Inc.,OU=management:idms.group.764034,CN=results.webkit.org', issuer > > `C=US,ST=California,O=Apple Inc.,CN=Apple Public Server RSA CA 12 - G1', > > serial 0x4c75add9335798d877e2d8088da7118f, RSA key 2048 bits, signed using > > RSA-SHA256, activated `2019-09-03 22:15:05 UTC', expires `2021-10-02 > > 22:15:05 UTC', pin-sha256="HWRWOw3Y56CfV1OWXgeT3mEren7q+ACX3K8c03MCTIc=" > > Public Key ID: > > sha1:421918b7a3df5b5b160b2aa2a2ae6dbb6c97a279 > > sha256:1d64563b0dd8e7a09f5753965e0793de612b7a7eeaf80097dcaf1cd373024c87 > > Public Key PIN: > > pin-sha256:HWRWOw3Y56CfV1OWXgeT3mEren7q+ACX3K8c03MCTIc= > > > > - Status: The certificate is NOT trusted. The certificate issuer is unknown. > > *** PKI verification of server certificate failed... > > *** Fatal error: Error in the certificate. > > I temporarily turned off the verification in r250997 because our Catalina > bots where encountering a similar problem. Digging more into this: - By default Debian 10 (which our bots run) only accepts(In reply to Jonathan Bedard from comment #3) > (In reply to Carlos Alberto Lopez Perez from comment #1) > > Seems the bug happens when the layout step fails to upload. > > > > Our bots are giving this error: > > > > 05:10:27.819 123196 Uploading to https://results.webkit.org ... > > 05:10:27.892 123196 Starting new HTTPS connection (1): results.webkit.org:443 > > 05:10:28.207 123196 Failed to upload to https://results.webkit.org, > > results server not online > > > > (likely caused because results.webkit.org has an invalid SSL certificate, > > signed by an unknown CA) > > > > $ gnutls-cli results.webkit.org:443 > > Processed 124 CA certificate(s). > > Resolving 'results.webkit.org:443'... > > Connecting to '17.32.208.166:443'... > > - Certificate type: X.509 > > - Got a certificate list of 1 certificates. > > - Certificate[0] info: > > - subject `C=US,ST=California,O=Apple > > Inc.,OU=management:idms.group.764034,CN=results.webkit.org', issuer > > `C=US,ST=California,O=Apple Inc.,CN=Apple Public Server RSA CA 12 - G1', > > serial 0x4c75add9335798d877e2d8088da7118f, RSA key 2048 bits, signed using > > RSA-SHA256, activated `2019-09-03 22:15:05 UTC', expires `2021-10-02 > > 22:15:05 UTC', pin-sha256="HWRWOw3Y56CfV1OWXgeT3mEren7q+ACX3K8c03MCTIc=" > > Public Key ID: > > sha1:421918b7a3df5b5b160b2aa2a2ae6dbb6c97a279 > > sha256:1d64563b0dd8e7a09f5753965e0793de612b7a7eeaf80097dcaf1cd373024c87 > > Public Key PIN: > > pin-sha256:HWRWOw3Y56CfV1OWXgeT3mEren7q+ACX3K8c03MCTIc= > > > > - Status: The certificate is NOT trusted. The certificate issuer is unknown. > > *** PKI verification of server certificate failed... > > *** Fatal error: Error in the certificate. > > I temporarily turned off the verification in r250997 because our Catalina > bots where encountering a similar problem. Digging more into this: - By default Debian 10 (which our bots run) only accept SSL connections with at least TLSv1.2 - results.webkit.org is not able to do anything better than TLSv1.0. Check: https://www.ssllabs.com/ssltest/analyze.html?d=results.webkit.org - So even when running with Verify=false it fails because it isn't able to even do the handshake, check: $ python Python 2.7.16 (default, Apr 6 2019, 01:42:57) [GCC 8.3.0] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import requests >>> p=requests.post('https://results.webkit.org/api/upload',headers={'Content-type': 'application/json'},data="",verify=False) Traceback (most recent call last): File "<stdin>", line 1, in <module> File "requests/api.py", line 116, in post return request('post', url, data=data, json=json, **kwargs) File "requests/api.py", line 60, in request return session.request(method=method, url=url, **kwargs) File "requests/sessions.py", line 533, in request resp = self.send(prep, **send_kwargs) File "requests/sessions.py", line 646, in send r = adapter.send(request, **kwargs) File "requests/adapters.py", line 514, in send raise SSLError(e, request=request) requests.exceptions.SSLError: HTTPSConnectionPool(host='results.webkit.org', port=443): Max retries exceeded with url: /api/upload (Caused by SSLError(SSLError("bad handshake: Error([('SSL routines', 'ssl_choose_client_version', 'unsupported protocol')],)",),)) >>> - We already had a similar issue on our EWS bots, where we had to downgrade the security settings of Debian to accept TLSv1.0 because the server hosting the EWS master was also providing SSL with deprecated encryption settings. - We can do something similar here, I can set our buildbot bots to accept SSL with TLSv1.0 if upgrading that server to provide more modern SSL settings is difficult for you. But I guess this is only going to keep causing issues in the future, upgrading that server in your end to support TLSv1.2 would be nice. 1579656 6 clopez 2019-10-14 09:40:59 -0700 In any case I think we maybe should keep printing the results to stdout even when that upload step fails 1579667 7 jbedard 2019-10-14 10:03:17 -0700 (In reply to Carlos Alberto Lopez Perez from comment #6) > In any case I think we maybe should keep printing the results to stdout even > when that upload step fails This is reasonable, and also pretty easy, I'll put together the patch to do so. 1579671 8 jbedard 2019-10-14 10:06:44 -0700 (In reply to Carlos Alberto Lopez Perez from comment #5) > ... > > - We can do something similar here, I can set our buildbot bots to accept > SSL with > TLSv1.0 if upgrading that server to provide more modern SSL settings is > difficult for you. But I guess this is only going to keep causing issues > in the future, upgrading that server in your end to support TLSv1.2 would > be nice. Actually, looking through our configuration, I had explicitly downgraded our TLS version in a misguided attempt to fix our Catalina bots. I just changed those configurations, we should be using TLSv1.2 now. 1579682 9 380891 jbedard 2019-10-14 10:28:29 -0700 Created attachment 380891 Patch 1579698 10 clopez 2019-10-14 10:59:05 -0700 (In reply to Jonathan Bedard from comment #8) > (In reply to Carlos Alberto Lopez Perez from comment #5) > > ... > > > > - We can do something similar here, I can set our buildbot bots to accept > > SSL with > > TLSv1.0 if upgrading that server to provide more modern SSL settings is > > difficult for you. But I guess this is only going to keep causing issues > > in the future, upgrading that server in your end to support TLSv1.2 would > > be nice. > > Actually, looking through our configuration, I had explicitly downgraded our > TLS version in a misguided attempt to fix our Catalina bots. I just changed > those configurations, we should be using TLSv1.2 now. Thanks. Now it is able to connect to the server, but gives other error: 10:54:12.318 35166 Uploading to https://results.webkit.org ... 10:54:12.387 35166 Starting new HTTPS connection (1): results.webkit.org:443 /home/slave/webkitgtk/gtk-linux-64-release-tests/build/Tools/Scripts/webkitpy/thirdparty/autoinstalled/urllib3/connectionpool.py:1004: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings InsecureRequestWarning, 10:54:16.429 35166 https://results.webkit.org:443 "POST /api/upload HTTP/1.1" 400 79 10:54:16.430 35166 Error uploading to https://results.webkit.org 10:54:16.430 35166 Invalid configuration 10:54:16.430 35166 Uploads completed! full log here: https://build.webkit.org/builders/GTK%20Linux%2064-bit%20Release%20%28Tests%29/builds/11605/steps/layout-test/logs/stdio Is there something we have to configure on the bots for this upload to happen as expected? 1579707 11 jbedard 2019-10-14 11:15:51 -0700 (In reply to Carlos Alberto Lopez Perez from comment #10) > ... > Is there something we have to configure on the bots for this upload to > happen as expected? Not expected, although that error is actually coming from the server, so you are making a POST request, it's just getting rejected. My guess is that we are creating a configuration that's missing something...I don't have a GTK machine handy, if we log the dictionary returned by Tools/Scripts/webkitpy/results/upload.py Upload.create_configuration, what does it look like? (this is called from Tools/Scripts/port/base.py for GTK) 1579710 12 jbedard 2019-10-14 11:17:05 -0700 (In reply to Jonathan Bedard from comment #11) > (In reply to Carlos Alberto Lopez Perez from comment #10) > > ... > > Is there something we have to configure on the bots for this upload to > > happen as expected? > > Not expected, although that error is actually coming from the server, so you > are making a POST request, it's just getting rejected. > > My guess is that we are creating a configuration that's missing > something...I don't have a GTK machine handy, if we log the dictionary > returned by Tools/Scripts/webkitpy/results/upload.py > Upload.create_configuration, what does it look like? (this is called from > Tools/Scripts/port/base.py for GTK) Side note: You can test this by running test-webkitpy --report https://results.webkit.org locally on a GTK machine. You won't have an API key, so it will get rejected, but we can at least see what GTK machines try and upload 1579731 13 clopez 2019-10-14 12:10:09 -0700 (In reply to Jonathan Bedard from comment #12) > (In reply to Jonathan Bedard from comment #11) > > (In reply to Carlos Alberto Lopez Perez from comment #10) > > > ... > > > Is there something we have to configure on the bots for this upload to > > > happen as expected? > > > > Not expected, although that error is actually coming from the server, so you > > are making a POST request, it's just getting rejected. > > > > My guess is that we are creating a configuration that's missing > > something...I don't have a GTK machine handy, if we log the dictionary > > returned by Tools/Scripts/webkitpy/results/upload.py > > Upload.create_configuration, what does it look like? (this is called from > > Tools/Scripts/port/base.py for GTK) > > Side note: You can test this by running test-webkitpy --report > https://results.webkit.org locally on a GTK machine. You won't have an API > key, so it will get rejected, but we can at least see what GTK machines try > and upload How the bots are supposed to obtain the API key? I think nobody installed that on the GTK or WPE bots (if that was supposed to be done) 1579737 14 jbedard 2019-10-14 12:26:59 -0700 (In reply to Carlos Alberto Lopez Perez from comment #13) > ... > > How the bots are supposed to obtain the API key? I think nobody installed > that on the GTK or WPE bots (if that was supposed to be done) Buildbot passes it as an environment variable. The error message indicates your bots are getting past that point, you would have gotten something like 'Missing API key' or 'Incorrect API key' 1579765 15 clopez 2019-10-14 13:24:29 -0700 (In reply to Jonathan Bedard from comment #11) > (In reply to Carlos Alberto Lopez Perez from comment #10) > > ... > > Is there something we have to configure on the bots for this upload to > > happen as expected? > > Not expected, although that error is actually coming from the server, so you > are making a POST request, it's just getting rejected. > > My guess is that we are creating a configuration that's missing > something...I don't have a GTK machine handy, if we log the dictionary > returned by Tools/Scripts/webkitpy/results/upload.py > Upload.create_configuration, what does it look like? (this is called from > Tools/Scripts/port/base.py for GTK) This is how that dictionary looks on GTK (and I guess on WPE it would look the same): {'platform': 'linux', 'style': 'release', 'is_simulator': False, 'version': 'None', 'architecture': 'x86'} 1579831 16 jbedard 2019-10-14 14:36:19 -0700 (In reply to Carlos Alberto Lopez Perez from comment #15) > (In reply to Jonathan Bedard from comment #11) > > (In reply to Carlos Alberto Lopez Perez from comment #10) > > > ... > > > Is there something we have to configure on the bots for this upload to > > > happen as expected? > > > > Not expected, although that error is actually coming from the server, so you > > are making a POST request, it's just getting rejected. > > > > My guess is that we are creating a configuration that's missing > > something...I don't have a GTK machine handy, if we log the dictionary > > returned by Tools/Scripts/webkitpy/results/upload.py > > Upload.create_configuration, what does it look like? (this is called from > > Tools/Scripts/port/base.py for GTK) > > This is how that dictionary looks on GTK (and I guess on WPE it would look > the same): > > {'platform': 'linux', 'style': 'release', 'is_simulator': False, 'version': > 'None', 'architecture': 'x86'} This is actually a pretty straight-forward problem. platform.os_version isn 't defined for GTK. Curious, what is the output of this python program: import platform print(platform.version()) 1579837 17 clopez 2019-10-14 14:53:06 -0700 (In reply to Jonathan Bedard from comment #16) > (In reply to Carlos Alberto Lopez Perez from comment #15) > > (In reply to Jonathan Bedard from comment #11) > > > (In reply to Carlos Alberto Lopez Perez from comment #10) > > > > ... > > > > Is there something we have to configure on the bots for this upload to > > > > happen as expected? > > > > > > Not expected, although that error is actually coming from the server, so you > > > are making a POST request, it's just getting rejected. > > > > > > My guess is that we are creating a configuration that's missing > > > something...I don't have a GTK machine handy, if we log the dictionary > > > returned by Tools/Scripts/webkitpy/results/upload.py > > > Upload.create_configuration, what does it look like? (this is called from > > > Tools/Scripts/port/base.py for GTK) > > > > This is how that dictionary looks on GTK (and I guess on WPE it would look > > the same): > > > > {'platform': 'linux', 'style': 'release', 'is_simulator': False, 'version': > > 'None', 'architecture': 'x86'} > > This is actually a pretty straight-forward problem. platform.os_version isn > 't defined for GTK. > > Curious, what is the output of this python program: > > import platform > print(platform.version()) This is the output: #1 SMP Debian 4.19.37-5+deb10u2~bpo9+1 (2019-08-16) 380891 2019-10-14 10:28:29 -0700 2019-10-14 10:28:29 -0700 Patch bug-202928-20191014102828.patch text/plain 2186 jbedard SW5kZXg6IFRvb2xzL0NoYW5nZUxvZwo9PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09 PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09Ci0tLSBUb29scy9DaGFuZ2VMb2cJKHJl dmlzaW9uIDI1MTA4MSkKKysrIFRvb2xzL0NoYW5nZUxvZwkod29ya2luZyBjb3B5KQpAQCAtMSwz ICsxLDE1IEBACisyMDE5LTEwLTE0ICBKb25hdGhhbiBCZWRhcmQgIDxqYmVkYXJkQGFwcGxlLmNv bT4KKworICAgICAgICBsYXlvdXQgc3RlcCBub3Qgc3VtbWFyaXplcyByZXN1bHRzIHRvIHN0ZG91 dCB3aGVuIHJ1bm5pbmcgd2l0aCAiLS1yZXBvcnQiCisgICAgICAgIGh0dHBzOi8vYnVncy53ZWJr aXQub3JnL3Nob3dfYnVnLmNnaT9pZD0yMDI5MjgKKworICAgICAgICBSZXZpZXdlZCBieSBOT0JP RFkgKE9PUFMhKS4KKworICAgICAgICAqIFNjcmlwdHMvd2Via2l0cHkvYXBpX3Rlc3RzL21hbmFn ZXIucHk6CisgICAgICAgIChNYW5hZ2VyLnJ1bik6IFByaW9yaXRpemUgdGVzdGluZyByZXN1bHQg b3ZlciB1cGxvYWQgcmVzdWx0LgorICAgICAgICAqIFNjcmlwdHMvd2Via2l0cHkvbGF5b3V0X3Rl c3RzL2NvbnRyb2xsZXJzL21hbmFnZXIucHk6CisgICAgICAgIChNYW5hZ2VyLnJ1bik6IFByaW9y aXRpemUgdGVzdGluZyByZXN1bHQgb3ZlciB1cGxvYWQgcmVzdWx0LgorCiAyMDE5LTEwLTE0ICBO aWtvbGFzIFppbW1lcm1hbm4gIDxuemltbWVybWFubkBpZ2FsaWEuY29tPgogCiAgICAgICAgIFVw ZGF0ZSBteSBuaWNrbmFtZS4KSW5kZXg6IFRvb2xzL1NjcmlwdHMvd2Via2l0cHkvYXBpX3Rlc3Rz L21hbmFnZXIucHkKPT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09 PT09PT09PT09PT09PT09PT09PT09PQotLS0gVG9vbHMvU2NyaXB0cy93ZWJraXRweS9hcGlfdGVz dHMvbWFuYWdlci5weQkocmV2aXNpb24gMjUxMDc0KQorKysgVG9vbHMvU2NyaXB0cy93ZWJraXRw eS9hcGlfdGVzdHMvbWFuYWdlci5weQkod29ya2luZyBjb3B5KQpAQCAtMjc1LDcgKzI3NSw3IEBA IGNsYXNzIE1hbmFnZXIob2JqZWN0KToKICAgICAgICAgICAgICkKICAgICAgICAgICAgIGZvciB1 cmwgaW4gc2VsZi5fb3B0aW9ucy5yZXBvcnRfdXJsczoKICAgICAgICAgICAgICAgICBzZWxmLl9z dHJlYW0ud3JpdGVfdXBkYXRlKCdVcGxvYWRpbmcgdG8ge30gLi4uJy5mb3JtYXQodXJsKSkKLSAg ICAgICAgICAgICAgICBpZiBub3QgdXBsb2FkLnVwbG9hZCh1cmwsIGxvZ19saW5lX2Z1bmM9c2Vs Zi5fc3RyZWFtLndyaXRlbG4pOgorICAgICAgICAgICAgICAgIGlmIG5vdCB1cGxvYWQudXBsb2Fk KHVybCwgbG9nX2xpbmVfZnVuYz1zZWxmLl9zdHJlYW0ud3JpdGVsbikgYW5kIG5vdCByZXN1bHQ6 CiAgICAgICAgICAgICAgICAgICAgIHJlc3VsdCA9IE1hbmFnZXIuRkFJTEVEX1VQTE9BRAogICAg ICAgICAgICAgc2VsZi5fc3RyZWFtLndyaXRlbG4oJ1VwbG9hZHMgY29tcGxldGVkIScpCiAKSW5k ZXg6IFRvb2xzL1NjcmlwdHMvd2Via2l0cHkvbGF5b3V0X3Rlc3RzL2NvbnRyb2xsZXJzL21hbmFn ZXIucHkKPT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09 PT09PT09PT09PT09PT09PQotLS0gVG9vbHMvU2NyaXB0cy93ZWJraXRweS9sYXlvdXRfdGVzdHMv Y29udHJvbGxlcnMvbWFuYWdlci5weQkocmV2aXNpb24gMjUxMDc0KQorKysgVG9vbHMvU2NyaXB0 cy93ZWJraXRweS9sYXlvdXRfdGVzdHMvY29udHJvbGxlcnMvbWFuYWdlci5weQkod29ya2luZyBj b3B5KQpAQCAtMzI5LDcgKzMyOSw3IEBAIGNsYXNzIE1hbmFnZXIob2JqZWN0KToKICAgICAgICAg ICAgICAgICAgICAgICAgIGlmIG5vdCB1cGxvYWQudXBsb2FkX2FyY2hpdmUoaG9zdG5hbWUsIHNl bGYuX2ZpbGVzeXN0ZW0ub3Blbl9iaW5hcnlfZmlsZV9mb3JfcmVhZGluZyhhcmNoaXZlICsgJy56 aXAnKSwgbG9nX2xpbmVfZnVuYz1zZWxmLl9wcmludGVyLndyaXRlbG4pOgogICAgICAgICAgICAg ICAgICAgICAgICAgICAgIG51bV9mYWlsZWRfdXBsb2FkcyArPSAxCiAKLSAgICAgICAgaWYgbnVt X2ZhaWxlZF91cGxvYWRzOgorICAgICAgICBpZiBudW1fZmFpbGVkX3VwbG9hZHMgYW5kIG5vdCBy ZXN1bHQuZXhpdF9jb2RlOgogICAgICAgICAgICAgcmVzdWx0LmV4aXRfY29kZSA9IC0xCiAgICAg ICAgIHJldHVybiByZXN1bHQKIAo=