Bug 202928

Seems the bug happens when the layout step fails to upload. Our bots are giving this error: 05:10:27.819 123196 Uploading to https://results.webkit.org ... 05:10:27.892 123196 Starting new HTTPS connection (1): results.webkit.org:443 05:10:28.207 123196 Failed to upload to https://results.webkit.org, results server not online (likely caused because results.webkit.org has an invalid SSL certificate, signed by an unknown CA) $ gnutls-cli results.webkit.org:443 Processed 124 CA certificate(s). Resolving 'results.webkit.org:443'... Connecting to '17.32.208.166:443'... - Certificate type: X.509 - Got a certificate list of 1 certificates. - Certificate[0] info: - subject `C=US,ST=California,O=Apple Inc.,OU=management:idms.group.764034,CN=results.webkit.org', issuer `C=US,ST=California,O=Apple Inc.,CN=Apple Public Server RSA CA 12 - G1', serial 0x4c75add9335798d877e2d8088da7118f, RSA key 2048 bits, signed using RSA-SHA256, activated `2019-09-03 22:15:05 UTC', expires `2021-10-02 22:15:05 UTC', pin-sha256="HWRWOw3Y56CfV1OWXgeT3mEren7q+ACX3K8c03MCTIc=" Public Key ID: sha1:421918b7a3df5b5b160b2aa2a2ae6dbb6c97a279 sha256:1d64563b0dd8e7a09f5753965e0793de612b7a7eeaf80097dcaf1cd373024c87 Public Key PIN: pin-sha256:HWRWOw3Y56CfV1OWXgeT3mEren7q+ACX3K8c03MCTIc= - Status: The certificate is NOT trusted. The certificate issuer is unknown. *** PKI verification of server certificate failed... *** Fatal error: Error in the certificate.

webkitpy tests are probably the right example for getting to the bottom of this: https://build.webkit.org/builders/GTK%20Linux%2064-bit%20Release%20%28Tests%29/builds/11603/steps/webkitpy-test/logs/stdio Looks like GTK machines can't reach results.webkit.org. WinCario has the same problem: https://build.webkit.org/builders/WPE%20Linux%2064-bit%20Release%20%28Tests%29/builds/15688/steps/webkitpy-test/logs/stdio Can those machines reach https://results.webkit.org? You should be able to reproduce this with 'test-webkitpy --report https://results.webkit.org' (note that without the API key, which the buildbot master servers, this request will fail, but it should explicitly say that it failed because of the API key, rather than 'results server not online'. I did some work last week, and I had a personal machine off the Apple network that was able to report, so I'm surprised we're having this issue.

(In reply to Carlos Alberto Lopez Perez from comment #1) > Seems the bug happens when the layout step fails to upload. > > Our bots are giving this error: > > 05:10:27.819 123196 Uploading to https://results.webkit.org ... > 05:10:27.892 123196 Starting new HTTPS connection (1): results.webkit.org:443 > 05:10:28.207 123196 Failed to upload to https://results.webkit.org, > results server not online > > (likely caused because results.webkit.org has an invalid SSL certificate, > signed by an unknown CA) > > $ gnutls-cli results.webkit.org:443 > Processed 124 CA certificate(s). > Resolving 'results.webkit.org:443'... > Connecting to '17.32.208.166:443'... > - Certificate type: X.509 > - Got a certificate list of 1 certificates. > - Certificate[0] info: > - subject `C=US,ST=California,O=Apple > Inc.,OU=management:idms.group.764034,CN=results.webkit.org', issuer > `C=US,ST=California,O=Apple Inc.,CN=Apple Public Server RSA CA 12 - G1', > serial 0x4c75add9335798d877e2d8088da7118f, RSA key 2048 bits, signed using > RSA-SHA256, activated `2019-09-03 22:15:05 UTC', expires `2021-10-02 > 22:15:05 UTC', pin-sha256="HWRWOw3Y56CfV1OWXgeT3mEren7q+ACX3K8c03MCTIc=" > Public Key ID: > sha1:421918b7a3df5b5b160b2aa2a2ae6dbb6c97a279 > sha256:1d64563b0dd8e7a09f5753965e0793de612b7a7eeaf80097dcaf1cd373024c87 > Public Key PIN: > pin-sha256:HWRWOw3Y56CfV1OWXgeT3mEren7q+ACX3K8c03MCTIc= > > - Status: The certificate is NOT trusted. The certificate issuer is unknown. > *** PKI verification of server certificate failed... > *** Fatal error: Error in the certificate. I temporarily turned off the verification in r250997 because our Catalina bots where encountering a similar problem.

(In reply to Jonathan Bedard from comment #2) > webkitpy tests are probably the right example for getting to the bottom of > this: > > https://build.webkit.org/builders/GTK%20Linux%2064- > bit%20Release%20%28Tests%29/builds/11603/steps/webkitpy-test/logs/stdio > > Looks like GTK machines can't reach results.webkit.org. WinCario has the > same problem: > > https://build.webkit.org/builders/WPE%20Linux%2064- > bit%20Release%20%28Tests%29/builds/15688/steps/webkitpy-test/logs/stdio > > Can those machines reach https://results.webkit.org? You should be able to > reproduce this with 'test-webkitpy --report https://results.webkit.org' > (note that without the API key, which the buildbot master servers, this > request will fail, but it should explicitly say that it failed because of > the API key, rather than 'results server not online'. > > I did some work last week, and I had a personal machine off the Apple > network that was able to report, so I'm surprised we're having this issue. They can reach it, but they can't recognized the certificate. It seems that the server hosting results.webkit.org is missing to send the intermediate certificates of the chain. See: https://www.sslshopper.com/ssl-checker.html#hostname=results.webkit.org

(In reply to Jonathan Bedard from comment #3) > (In reply to Carlos Alberto Lopez Perez from comment #1) > > Seems the bug happens when the layout step fails to upload. > > > > Our bots are giving this error: > > > > 05:10:27.819 123196 Uploading to https://results.webkit.org ... > > 05:10:27.892 123196 Starting new HTTPS connection (1): results.webkit.org:443 > > 05:10:28.207 123196 Failed to upload to https://results.webkit.org, > > results server not online > > > > (likely caused because results.webkit.org has an invalid SSL certificate, > > signed by an unknown CA) > > > > $ gnutls-cli results.webkit.org:443 > > Processed 124 CA certificate(s). > > Resolving 'results.webkit.org:443'... > > Connecting to '17.32.208.166:443'... > > - Certificate type: X.509 > > - Got a certificate list of 1 certificates. > > - Certificate[0] info: > > - subject `C=US,ST=California,O=Apple > > Inc.,OU=management:idms.group.764034,CN=results.webkit.org', issuer > > `C=US,ST=California,O=Apple Inc.,CN=Apple Public Server RSA CA 12 - G1', > > serial 0x4c75add9335798d877e2d8088da7118f, RSA key 2048 bits, signed using > > RSA-SHA256, activated `2019-09-03 22:15:05 UTC', expires `2021-10-02 > > 22:15:05 UTC', pin-sha256="HWRWOw3Y56CfV1OWXgeT3mEren7q+ACX3K8c03MCTIc=" > > Public Key ID: > > sha1:421918b7a3df5b5b160b2aa2a2ae6dbb6c97a279 > > sha256:1d64563b0dd8e7a09f5753965e0793de612b7a7eeaf80097dcaf1cd373024c87 > > Public Key PIN: > > pin-sha256:HWRWOw3Y56CfV1OWXgeT3mEren7q+ACX3K8c03MCTIc= > > > > - Status: The certificate is NOT trusted. The certificate issuer is unknown. > > *** PKI verification of server certificate failed... > > *** Fatal error: Error in the certificate. > > I temporarily turned off the verification in r250997 because our Catalina > bots where encountering a similar problem. Digging more into this: - By default Debian 10 (which our bots run) only accepts(In reply to Jonathan Bedard from comment #3) > (In reply to Carlos Alberto Lopez Perez from comment #1) > > Seems the bug happens when the layout step fails to upload. > > > > Our bots are giving this error: > > > > 05:10:27.819 123196 Uploading to https://results.webkit.org ... > > 05:10:27.892 123196 Starting new HTTPS connection (1): results.webkit.org:443 > > 05:10:28.207 123196 Failed to upload to https://results.webkit.org, > > results server not online > > > > (likely caused because results.webkit.org has an invalid SSL certificate, > > signed by an unknown CA) > > > > $ gnutls-cli results.webkit.org:443 > > Processed 124 CA certificate(s). > > Resolving 'results.webkit.org:443'... > > Connecting to '17.32.208.166:443'... > > - Certificate type: X.509 > > - Got a certificate list of 1 certificates. > > - Certificate[0] info: > > - subject `C=US,ST=California,O=Apple > > Inc.,OU=management:idms.group.764034,CN=results.webkit.org', issuer > > `C=US,ST=California,O=Apple Inc.,CN=Apple Public Server RSA CA 12 - G1', > > serial 0x4c75add9335798d877e2d8088da7118f, RSA key 2048 bits, signed using > > RSA-SHA256, activated `2019-09-03 22:15:05 UTC', expires `2021-10-02 > > 22:15:05 UTC', pin-sha256="HWRWOw3Y56CfV1OWXgeT3mEren7q+ACX3K8c03MCTIc=" > > Public Key ID: > > sha1:421918b7a3df5b5b160b2aa2a2ae6dbb6c97a279 > > sha256:1d64563b0dd8e7a09f5753965e0793de612b7a7eeaf80097dcaf1cd373024c87 > > Public Key PIN: > > pin-sha256:HWRWOw3Y56CfV1OWXgeT3mEren7q+ACX3K8c03MCTIc= > > > > - Status: The certificate is NOT trusted. The certificate issuer is unknown. > > *** PKI verification of server certificate failed... > > *** Fatal error: Error in the certificate. > > I temporarily turned off the verification in r250997 because our Catalina > bots where encountering a similar problem. Digging more into this: - By default Debian 10 (which our bots run) only accept SSL connections with at least TLSv1.2 - results.webkit.org is not able to do anything better than TLSv1.0. Check: https://www.ssllabs.com/ssltest/analyze.html?d=results.webkit.org - So even when running with Verify=false it fails because it isn't able to even do the handshake, check: $ python Python 2.7.16 (default, Apr 6 2019, 01:42:57) [GCC 8.3.0] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import requests >>> p=requests.post('https://results.webkit.org/api/upload',headers={'Content-type': 'application/json'},data="",verify=False) Traceback (most recent call last): File "<stdin>", line 1, in <module> File "requests/api.py", line 116, in post return request('post', url, data=data, json=json, **kwargs) File "requests/api.py", line 60, in request return session.request(method=method, url=url, **kwargs) File "requests/sessions.py", line 533, in request resp = self.send(prep, **send_kwargs) File "requests/sessions.py", line 646, in send r = adapter.send(request, **kwargs) File "requests/adapters.py", line 514, in send raise SSLError(e, request=request) requests.exceptions.SSLError: HTTPSConnectionPool(host='results.webkit.org', port=443): Max retries exceeded with url: /api/upload (Caused by SSLError(SSLError("bad handshake: Error([('SSL routines', 'ssl_choose_client_version', 'unsupported protocol')],)",),)) >>> - We already had a similar issue on our EWS bots, where we had to downgrade the security settings of Debian to accept TLSv1.0 because the server hosting the EWS master was also providing SSL with deprecated encryption settings. - We can do something similar here, I can set our buildbot bots to accept SSL with TLSv1.0 if upgrading that server to provide more modern SSL settings is difficult for you. But I guess this is only going to keep causing issues in the future, upgrading that server in your end to support TLSv1.2 would be nice.

In any case I think we maybe should keep printing the results to stdout even when that upload step fails

(In reply to Carlos Alberto Lopez Perez from comment #6) > In any case I think we maybe should keep printing the results to stdout even > when that upload step fails This is reasonable, and also pretty easy, I'll put together the patch to do so.

(In reply to Carlos Alberto Lopez Perez from comment #5) > ... > > - We can do something similar here, I can set our buildbot bots to accept > SSL with > TLSv1.0 if upgrading that server to provide more modern SSL settings is > difficult for you. But I guess this is only going to keep causing issues > in the future, upgrading that server in your end to support TLSv1.2 would > be nice. Actually, looking through our configuration, I had explicitly downgraded our TLS version in a misguided attempt to fix our Catalina bots. I just changed those configurations, we should be using TLSv1.2 now.

Created attachment 380891 [details] Patch

(In reply to Jonathan Bedard from comment #8) > (In reply to Carlos Alberto Lopez Perez from comment #5) > > ... > > > > - We can do something similar here, I can set our buildbot bots to accept > > SSL with > > TLSv1.0 if upgrading that server to provide more modern SSL settings is > > difficult for you. But I guess this is only going to keep causing issues > > in the future, upgrading that server in your end to support TLSv1.2 would > > be nice. > > Actually, looking through our configuration, I had explicitly downgraded our > TLS version in a misguided attempt to fix our Catalina bots. I just changed > those configurations, we should be using TLSv1.2 now. Thanks. Now it is able to connect to the server, but gives other error: 10:54:12.318 35166 Uploading to https://results.webkit.org ... 10:54:12.387 35166 Starting new HTTPS connection (1): results.webkit.org:443 /home/slave/webkitgtk/gtk-linux-64-release-tests/build/Tools/Scripts/webkitpy/thirdparty/autoinstalled/urllib3/connectionpool.py:1004: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings InsecureRequestWarning, 10:54:16.429 35166 https://results.webkit.org:443 "POST /api/upload HTTP/1.1" 400 79 10:54:16.430 35166 Error uploading to https://results.webkit.org 10:54:16.430 35166 Invalid configuration 10:54:16.430 35166 Uploads completed! full log here: https://build.webkit.org/builders/GTK%20Linux%2064-bit%20Release%20%28Tests%29/builds/11605/steps/layout-test/logs/stdio Is there something we have to configure on the bots for this upload to happen as expected?

(In reply to Carlos Alberto Lopez Perez from comment #10) > ... > Is there something we have to configure on the bots for this upload to > happen as expected? Not expected, although that error is actually coming from the server, so you are making a POST request, it's just getting rejected. My guess is that we are creating a configuration that's missing something...I don't have a GTK machine handy, if we log the dictionary returned by Tools/Scripts/webkitpy/results/upload.py Upload.create_configuration, what does it look like? (this is called from Tools/Scripts/port/base.py for GTK)

(In reply to Jonathan Bedard from comment #11) > (In reply to Carlos Alberto Lopez Perez from comment #10) > > ... > > Is there something we have to configure on the bots for this upload to > > happen as expected? > > Not expected, although that error is actually coming from the server, so you > are making a POST request, it's just getting rejected. > > My guess is that we are creating a configuration that's missing > something...I don't have a GTK machine handy, if we log the dictionary > returned by Tools/Scripts/webkitpy/results/upload.py > Upload.create_configuration, what does it look like? (this is called from > Tools/Scripts/port/base.py for GTK) Side note: You can test this by running test-webkitpy --report https://results.webkit.org locally on a GTK machine. You won't have an API key, so it will get rejected, but we can at least see what GTK machines try and upload

(In reply to Jonathan Bedard from comment #12) > (In reply to Jonathan Bedard from comment #11) > > (In reply to Carlos Alberto Lopez Perez from comment #10) > > > ... > > > Is there something we have to configure on the bots for this upload to > > > happen as expected? > > > > Not expected, although that error is actually coming from the server, so you > > are making a POST request, it's just getting rejected. > > > > My guess is that we are creating a configuration that's missing > > something...I don't have a GTK machine handy, if we log the dictionary > > returned by Tools/Scripts/webkitpy/results/upload.py > > Upload.create_configuration, what does it look like? (this is called from > > Tools/Scripts/port/base.py for GTK) > > Side note: You can test this by running test-webkitpy --report > https://results.webkit.org locally on a GTK machine. You won't have an API > key, so it will get rejected, but we can at least see what GTK machines try > and upload How the bots are supposed to obtain the API key? I think nobody installed that on the GTK or WPE bots (if that was supposed to be done)

(In reply to Carlos Alberto Lopez Perez from comment #13) > ... > > How the bots are supposed to obtain the API key? I think nobody installed > that on the GTK or WPE bots (if that was supposed to be done) Buildbot passes it as an environment variable. The error message indicates your bots are getting past that point, you would have gotten something like 'Missing API key' or 'Incorrect API key'

(In reply to Jonathan Bedard from comment #11) > (In reply to Carlos Alberto Lopez Perez from comment #10) > > ... > > Is there something we have to configure on the bots for this upload to > > happen as expected? > > Not expected, although that error is actually coming from the server, so you > are making a POST request, it's just getting rejected. > > My guess is that we are creating a configuration that's missing > something...I don't have a GTK machine handy, if we log the dictionary > returned by Tools/Scripts/webkitpy/results/upload.py > Upload.create_configuration, what does it look like? (this is called from > Tools/Scripts/port/base.py for GTK) This is how that dictionary looks on GTK (and I guess on WPE it would look the same): {'platform': 'linux', 'style': 'release', 'is_simulator': False, 'version': 'None', 'architecture': 'x86'}

(In reply to Carlos Alberto Lopez Perez from comment #15) > (In reply to Jonathan Bedard from comment #11) > > (In reply to Carlos Alberto Lopez Perez from comment #10) > > > ... > > > Is there something we have to configure on the bots for this upload to > > > happen as expected? > > > > Not expected, although that error is actually coming from the server, so you > > are making a POST request, it's just getting rejected. > > > > My guess is that we are creating a configuration that's missing > > something...I don't have a GTK machine handy, if we log the dictionary > > returned by Tools/Scripts/webkitpy/results/upload.py > > Upload.create_configuration, what does it look like? (this is called from > > Tools/Scripts/port/base.py for GTK) > > This is how that dictionary looks on GTK (and I guess on WPE it would look > the same): > > {'platform': 'linux', 'style': 'release', 'is_simulator': False, 'version': > 'None', 'architecture': 'x86'} This is actually a pretty straight-forward problem. platform.os_version isn 't defined for GTK. Curious, what is the output of this python program: import platform print(platform.version())

(In reply to Jonathan Bedard from comment #16) > (In reply to Carlos Alberto Lopez Perez from comment #15) > > (In reply to Jonathan Bedard from comment #11) > > > (In reply to Carlos Alberto Lopez Perez from comment #10) > > > > ... > > > > Is there something we have to configure on the bots for this upload to > > > > happen as expected? > > > > > > Not expected, although that error is actually coming from the server, so you > > > are making a POST request, it's just getting rejected. > > > > > > My guess is that we are creating a configuration that's missing > > > something...I don't have a GTK machine handy, if we log the dictionary > > > returned by Tools/Scripts/webkitpy/results/upload.py > > > Upload.create_configuration, what does it look like? (this is called from > > > Tools/Scripts/port/base.py for GTK) > > > > This is how that dictionary looks on GTK (and I guess on WPE it would look > > the same): > > > > {'platform': 'linux', 'style': 'release', 'is_simulator': False, 'version': > > 'None', 'architecture': 'x86'} > > This is actually a pretty straight-forward problem. platform.os_version isn > 't defined for GTK. > > Curious, what is the output of this python program: > > import platform > print(platform.version()) This is the output: #1 SMP Debian 4.19.37-5+deb10u2~bpo9+1 (2019-08-16)