205608 2019-12-27 10:44:47 -0800 [GTK][WPE] Report number of total tests run and failed in API test runner like run-api-tests does 2019-12-31 06:12:57 -0800 1 1 1 Unclassified WebKit WebKitGTK WebKit Nightly Build Unspecified Unspecified RESOLVED FIXED P2 Normal --- 202361 1 clopez clopez aakash_jain bugs-noreply cgarcia oldest_to_newest 1601630 0 clopez 2019-12-27 10:44:47 -0800 We can reuse and simplify the implementation of bug 202361 a lot by outputting at the end of the run-gtk-tests/run-wpe-tests step the information about the number of tests ran and failed like run-api-tests does: "Ran X tests of Y with Z successful" Where: X=total number of tests executed Y=total number of tests available Z=total number of tests that passed (or that failed as expected) If we do this, we can basically use the same logic on the EWS (and buildbot) for running the gtk and mac/ios API tests. 1601631 1 386450 clopez 2019-12-27 10:53:30 -0800 Created attachment 386450 Patch Patch doesn't apply because it is implemented on top of the changes from bug 205579 1601632 2 clopez 2019-12-27 10:56:11 -0800 (In reply to Carlos Alberto Lopez Perez from comment #1) > Created attachment 386450 [details] > Patch > > Patch doesn't apply because it is implemented on top of the changes from bug > 205579 mmmm, seems it applies anyway.. good :) 1601888 3 386450 clopez 2019-12-30 09:17:06 -0800 Comment on attachment 386450 Patch Clearing flags on attachment: 386450 Committed r253947: <https://trac.webkit.org/changeset/253947> 1601889 4 clopez 2019-12-30 09:17:10 -0800 All reviewed patches have been landed. Closing bug. 1601976 5 aakash_jain 2019-12-31 06:09:10 -0800 Does this script output results in json format as well? If not, we would need that as well. EWS use json output to parse the name of failing tests. 1601978 6 clopez 2019-12-31 06:12:57 -0800 (In reply to Aakash Jain from comment #5) > Does this script output results in json format as well? If not, we would > need that as well. EWS use json output to parse the name of failing tests. Yes, it does now. See bug 205579 386450 2019-12-27 10:53:30 -0800 2019-12-30 09:17:06 -0800 Patch bug-205608-20191227195328.patch text/plain 7057 clopez U3VidmVyc2lvbiBSZXZpc2lvbjogMjUzNTUzCmRpZmYgLS1naXQgYS9Ub29scy9DaGFuZ2VMb2cg Yi9Ub29scy9DaGFuZ2VMb2cKaW5kZXggMDgxZGU1NTNiYjI4NDIxYzIzMTA3OTE3YjRmNjQ3ZTNk YTcyODhiYy4uNjA0YmRhZTU4MDg3MzcxZjUxMTY0NmRiOTIzYmE2OWE2MmY1NzgzYSAxMDA2NDQK LS0tIGEvVG9vbHMvQ2hhbmdlTG9nCisrKyBiL1Rvb2xzL0NoYW5nZUxvZwpAQCAtMSwzICsxLDIy IEBACisyMDE5LTEyLTI3ICBDYXJsb3MgQWxiZXJ0byBMb3BleiBQZXJleiAgPGNsb3BlekBpZ2Fs aWEuY29tPgorCisgICAgICAgIFtHVEtdW1dQRV0gUmVwb3J0IG51bWJlciBvZiB0b3RhbCB0ZXN0 cyBydW4gYW5kIGZhaWxlZCBpbiBBUEkgdGVzdCBydW5uZXIgbGlrZSBydW4tYXBpLXRlc3RzIGRv ZXMKKyAgICAgICAgaHR0cHM6Ly9idWdzLndlYmtpdC5vcmcvc2hvd19idWcuY2dpP2lkPTIwNTYw OAorCisgICAgICAgIFJldmlld2VkIGJ5IE5PQk9EWSAoT09QUyEpLgorCisgICAgICAgIENhbGN1 bGF0ZSB0aGUgdG90YWwgbnVtYmVyIG9mIHRlc3RzIChpbmNsdWRpbmcgc3VidGVzdHMpIGV4ZWN1 dGVkCisgICAgICAgIGFuZCByZXBvcnQgaXQgYXQgdGhlIGVuZCBvZiB0aGUgc3RlcCBpbiB0aGUg c2FtZSBmb3JtYXQgdGhhbiBydW4tYXBpLXRlc3RzIGRvZXMuCisKKyAgICAgICAgKiBnbGliL2Fw aV90ZXN0X3J1bm5lci5weToKKyAgICAgICAgKFRlc3RSdW5uZXIuX3J1bl90ZXN0X2dsaWIpOiBQ YXNzIHRoZSBsaXN0IG9mIHNraXBwZWQgdGVzdHMgaW5zdGVhZCBvZiBjYWxjdWxhdGluZyBpdCBo ZXJlIHRvIGF2b2lkIGNhbGN1bGF0aW5nIGl0IHR3aWNlLgorICAgICAgICAoVGVzdFJ1bm5lci5f Z2V0X3Rlc3RzX2Zyb21fZ29vZ2xlX3Rlc3Rfc3VpdGUpOiBEaXR0by4KKyAgICAgICAgKFRlc3RS dW5uZXIuX3J1bl9nb29nbGVfdGVzdF9zdWl0ZSk6IERpdHRvLgorICAgICAgICAoVGVzdFJ1bm5l ci5fcnVuX3Rlc3QpOiBEaXR0by4KKyAgICAgICAgKFRlc3RSdW5uZXIucnVuX3Rlc3RzKToKKyAg ICAgICAgKFRlc3RSdW5uZXIucnVuX3Rlc3RzLm51bWJlcl9vZl90ZXN0cyk6CisgICAgICAgIChU ZXN0UnVubmVyLnJ1bl90ZXN0cy5yZXBvcnQpOgorCiAyMDE5LTEyLTI3ICBDYXJsb3MgQWxiZXJ0 byBMb3BleiBQZXJleiAgPGNsb3BlekBpZ2FsaWEuY29tPgogCiAgICAgICAgIFtHVEtdW1dQRV0g c3VwcG9ydCBvdXRwdXQgb2YgcmVzdWx0cyBpbiBqc29uIGZvcm1hdCBpbiBBUEkgdGVzdCBydW5u ZXIuCmRpZmYgLS1naXQgYS9Ub29scy9nbGliL2FwaV90ZXN0X3J1bm5lci5weSBiL1Rvb2xzL2ds aWIvYXBpX3Rlc3RfcnVubmVyLnB5CmluZGV4IDMyMGRiOTBmYWIxNzc3ZjFjNjQ0ZDNjYzAxYjMz YmJhZWJhZjRjMDQuLjMwZTNmYTdjMzJlM2NkOTVmMGIwMTQwZDIyNzFmZWRiMGRmNjU4NzIgMTAw NzU1Ci0tLSBhL1Rvb2xzL2dsaWIvYXBpX3Rlc3RfcnVubmVyLnB5CisrKyBiL1Rvb2xzL2dsaWIv YXBpX3Rlc3RfcnVubmVyLnB5CkBAIC0xNTksMTMgKzE1OSwxMyBAQCBjbGFzcyBUZXN0UnVubmVy KG9iamVjdCk6CiAgICAgICAgICAgICAgICAgICAgIHJldHVybiAwCiAgICAgICAgICAgICAgICAg cmFpc2UKIAotICAgIGRlZiBfcnVuX3Rlc3RfZ2xpYihzZWxmLCB0ZXN0X3Byb2dyYW0pOgorICAg IGRlZiBfcnVuX3Rlc3RfZ2xpYihzZWxmLCB0ZXN0X3Byb2dyYW0sIHNraXBwZWRfdGVzdF9jYXNl cyk6CiAgICAgICAgIHRpbWVvdXQgPSBzZWxmLl9vcHRpb25zLnRpbWVvdXQKIAogICAgICAgICBk ZWYgaXNfc2xvd190ZXN0KHRlc3QsIHN1YnRlc3QpOgogICAgICAgICAgICAgcmV0dXJuIHNlbGYu X2V4cGVjdGF0aW9ucy5pc19zbG93KHRlc3QsIHN1YnRlc3QpCiAKLSAgICAgICAgcmV0dXJuIEdM aWJUZXN0UnVubmVyKHRlc3RfcHJvZ3JhbSwgdGltZW91dCwgaXNfc2xvd190ZXN0LCB0aW1lb3V0 ICogMTApLnJ1bihza2lwcGVkPXNlbGYuX3Rlc3RfY2FzZXNfdG9fc2tpcCh0ZXN0X3Byb2dyYW0p LCBlbnY9c2VsZi5fdGVzdF9lbnYpCisgICAgICAgIHJldHVybiBHTGliVGVzdFJ1bm5lcih0ZXN0 X3Byb2dyYW0sIHRpbWVvdXQsIGlzX3Nsb3dfdGVzdCwgdGltZW91dCAqIDEwKS5ydW4oc2tpcHBl ZD1za2lwcGVkX3Rlc3RfY2FzZXMsIGVudj1zZWxmLl90ZXN0X2VudikKIAogICAgIGRlZiBfcnVu X3Rlc3RfcXQoc2VsZiwgdGVzdF9wcm9ncmFtKToKICAgICAgICAgZW52ID0gc2VsZi5fdGVzdF9l bnYKQEAgLTE5NCw3ICsxOTQsNyBAQCBjbGFzcyBUZXN0UnVubmVyKG9iamVjdCk6CiAgICAgICAg ICAgICBwcmludCgiKipQQVNTKiogJXMiICUgbmFtZSkKICAgICAgICAgcmV0dXJuIHtuYW1lOiBy ZXN1bHR9CiAKLSAgICBkZWYgX2dldF90ZXN0c19mcm9tX2dvb2dsZV90ZXN0X3N1aXRlKHNlbGYs IHRlc3RfcHJvZ3JhbSk6CisgICAgZGVmIF9nZXRfdGVzdHNfZnJvbV9nb29nbGVfdGVzdF9zdWl0 ZShzZWxmLCB0ZXN0X3Byb2dyYW0sIHNraXBwZWRfdGVzdF9jYXNlcyk6CiAgICAgICAgIHRyeToK ICAgICAgICAgICAgIG91dHB1dCA9IHN1YnByb2Nlc3MuY2hlY2tfb3V0cHV0KFt0ZXN0X3Byb2dy YW0sICctLWd0ZXN0X2xpc3RfdGVzdHMnXSwgZW52PXNlbGYuX3Rlc3RfZW52KQogICAgICAgICBl eGNlcHQgc3VicHJvY2Vzcy5DYWxsZWRQcm9jZXNzRXJyb3I6CkBAIC0yMDIsOCArMjAyLDYgQEAg Y2xhc3MgVGVzdFJ1bm5lcihvYmplY3QpOgogICAgICAgICAgICAgc3lzLnN0ZGVyci5mbHVzaCgp CiAgICAgICAgICAgICByZXR1cm4gMQogCi0gICAgICAgIHNraXBwZWRfdGVzdF9jYXNlcyA9IHNl bGYuX3Rlc3RfY2FzZXNfdG9fc2tpcCh0ZXN0X3Byb2dyYW0pCi0KICAgICAgICAgdGVzdHMgPSBb XQogICAgICAgICBwcmVmaXggPSBOb25lCiAgICAgICAgIGZvciBsaW5lIGluIG91dHB1dC5zcGxp dCgnXG4nKToKQEAgLTI0Nyw5ICsyNDUsOSBAQCBjbGFzcyBUZXN0UnVubmVyKG9iamVjdCk6CiAK ICAgICAgICAgcmV0dXJuIHtzdWJ0ZXN0OiAiUEFTUyJ9CiAKLSAgICBkZWYgX3J1bl9nb29nbGVf dGVzdF9zdWl0ZShzZWxmLCB0ZXN0X3Byb2dyYW0pOgorICAgIGRlZiBfcnVuX2dvb2dsZV90ZXN0 X3N1aXRlKHNlbGYsIHRlc3RfcHJvZ3JhbSwgc2tpcHBlZF90ZXN0X2Nhc2VzKToKICAgICAgICAg cmVzdWx0ID0ge30KLSAgICAgICAgZm9yIHN1YnRlc3QgaW4gc2VsZi5fZ2V0X3Rlc3RzX2Zyb21f Z29vZ2xlX3Rlc3Rfc3VpdGUodGVzdF9wcm9ncmFtKToKKyAgICAgICAgZm9yIHN1YnRlc3QgaW4g c2VsZi5fZ2V0X3Rlc3RzX2Zyb21fZ29vZ2xlX3Rlc3Rfc3VpdGUodGVzdF9wcm9ncmFtLCBza2lw cGVkX3Rlc3RfY2FzZXMpOgogICAgICAgICAgICAgcmVzdWx0LnVwZGF0ZShzZWxmLl9ydW5fZ29v Z2xlX3Rlc3QodGVzdF9wcm9ncmFtLCBzdWJ0ZXN0KSkKICAgICAgICAgcmV0dXJuIHJlc3VsdAog CkBAIC0yNjIsMTMgKzI2MCwxNCBAQCBjbGFzcyBUZXN0UnVubmVyKG9iamVjdCk6CiAgICAgZGVm IGlzX3F0X3Rlc3Qoc2VsZiwgdGVzdF9wcm9ncmFtKToKICAgICAgICAgcmFpc2UgTm90SW1wbGVt ZW50ZWRFcnJvcgogCi0gICAgZGVmIF9ydW5fdGVzdChzZWxmLCB0ZXN0X3Byb2dyYW0pOgorICAg IGRlZiBfcnVuX3Rlc3Qoc2VsZiwgdGVzdF9wcm9ncmFtLCBza2lwcGVkX3Rlc3RfY2FzZXMpOgog ICAgICAgICBpZiBzZWxmLmlzX2dsaWJfdGVzdCh0ZXN0X3Byb2dyYW0pOgotICAgICAgICAgICAg cmV0dXJuIHNlbGYuX3J1bl90ZXN0X2dsaWIodGVzdF9wcm9ncmFtKQorICAgICAgICAgICAgcmV0 dXJuIHNlbGYuX3J1bl90ZXN0X2dsaWIodGVzdF9wcm9ncmFtLCBza2lwcGVkX3Rlc3RfY2FzZXMp CiAKICAgICAgICAgaWYgc2VsZi5pc19nb29nbGVfdGVzdCh0ZXN0X3Byb2dyYW0pOgotICAgICAg ICAgICAgcmV0dXJuIHNlbGYuX3J1bl9nb29nbGVfdGVzdF9zdWl0ZSh0ZXN0X3Byb2dyYW0pCisg ICAgICAgICAgICByZXR1cm4gc2VsZi5fcnVuX2dvb2dsZV90ZXN0X3N1aXRlKHRlc3RfcHJvZ3Jh bSwgc2tpcHBlZF90ZXN0X2Nhc2VzKQogCisgICAgICAgICMgRklYTUU6IHN1cHBvcnQgc2tpcHBp bmcgUXQgc3VidGVzdHMKICAgICAgICAgaWYgc2VsZi5pc19xdF90ZXN0KHRlc3RfcHJvZ3JhbSk6 CiAgICAgICAgICAgICByZXR1cm4gc2VsZi5fcnVuX3Rlc3RfcXQodGVzdF9wcm9ncmFtKQogCkBA IC0yODIsOSArMjgxLDExIEBAIGNsYXNzIFRlc3RSdW5uZXIob2JqZWN0KToKIAogICAgICAgICBz ZWxmLl9zZXR1cF90ZXN0aW5nX2Vudmlyb25tZW50KCkKIAorICAgICAgICBudW1iZXJfb2ZfdG90 YWxfdGVzdHMgPSBsZW4oc2VsZi5fdGVzdHMpCiAgICAgICAgICMgUmVtb3ZlIHNraXBwZWQgdGVz dHMgbm93IGluc3RlYWQgb2Ygd2hlbiB3ZSBmaW5kIHRoZW0sIGJlY2F1c2UKICAgICAgICAgIyBz b21lIHRlc3RzIG1pZ2h0IGJlIHNraXBwZWQgd2hpbGUgc2V0dGluZyB1cCB0aGUgdGVzdCBlbnZp cm9ubWVudC4KICAgICAgICAgc2VsZi5fdGVzdHMgPSBbdGVzdCBmb3IgdGVzdCBpbiBzZWxmLl90 ZXN0cyBpZiBzZWxmLl9zaG91bGRfcnVuX3Rlc3RfcHJvZ3JhbSh0ZXN0KV0KKyAgICAgICAgbnVt YmVyX29mX2V4ZWN1dGVkX3Rlc3RzID0gbGVuKHNlbGYuX3Rlc3RzKQogCiAgICAgICAgIGNyYXNo ZWRfdGVzdHMgPSB7fQogICAgICAgICBmYWlsZWRfdGVzdHMgPSB7fQpAQCAtMjkyLDcgKzI5Mywx MyBAQCBjbGFzcyBUZXN0UnVubmVyKG9iamVjdCk6CiAgICAgICAgIHBhc3NlZF90ZXN0cyA9IHt9 CiAgICAgICAgIHRyeToKICAgICAgICAgICAgIGZvciB0ZXN0IGluIHNlbGYuX3Rlc3RzOgotICAg ICAgICAgICAgICAgIHJlc3VsdHMgPSBzZWxmLl9ydW5fdGVzdCh0ZXN0KQorICAgICAgICAgICAg ICAgIHNraXBwZWRfc3VidGVzdHMgPSBzZWxmLl90ZXN0X2Nhc2VzX3RvX3NraXAodGVzdCkKKyAg ICAgICAgICAgICAgICBudW1iZXJfb2ZfdG90YWxfdGVzdHMgKz0gbGVuKHNraXBwZWRfc3VidGVz dHMpCisgICAgICAgICAgICAgICAgcmVzdWx0cyA9IHNlbGYuX3J1bl90ZXN0KHRlc3QsIHNraXBw ZWRfc3VidGVzdHMpCisgICAgICAgICAgICAgICAgbnVtYmVyX29mX2V4ZWN1dGVkX3N1YnRlc3Rz X2Zvcl90ZXN0ID0gbGVuKHJlc3VsdHMpCisgICAgICAgICAgICAgICAgaWYgbnVtYmVyX29mX2V4 ZWN1dGVkX3N1YnRlc3RzX2Zvcl90ZXN0ID4gMToKKyAgICAgICAgICAgICAgICAgICAgbnVtYmVy X29mX2V4ZWN1dGVkX3Rlc3RzICs9IG51bWJlcl9vZl9leGVjdXRlZF9zdWJ0ZXN0c19mb3JfdGVz dAorICAgICAgICAgICAgICAgICAgICBudW1iZXJfb2ZfdG90YWxfdGVzdHMgKz0gbnVtYmVyX29m X2V4ZWN1dGVkX3N1YnRlc3RzX2Zvcl90ZXN0CiAgICAgICAgICAgICAgICAgZm9yIHRlc3RfY2Fz ZSwgcmVzdWx0IGluIHJlc3VsdHMuaXRlcml0ZW1zKCk6CiAgICAgICAgICAgICAgICAgICAgIGlm IHJlc3VsdCBpbiBzZWxmLl9leHBlY3RhdGlvbnMuZ2V0X2V4cGVjdGF0aW9uKG9zLnBhdGguYmFz ZW5hbWUodGVzdCksIHRlc3RfY2FzZSk6CiAgICAgICAgICAgICAgICAgICAgICAgICBjb250aW51 ZQpAQCAtMzA4LDEwICszMTUsMTMgQEAgY2xhc3MgVGVzdFJ1bm5lcihvYmplY3QpOgogICAgICAg ICBmaW5hbGx5OgogICAgICAgICAgICAgc2VsZi5fdGVhcl9kb3duX3Rlc3RpbmdfZW52aXJvbm1l bnQoKQogCisgICAgICAgIGRlZiBudW1iZXJfb2ZfdGVzdHModGVzdHMpOgorICAgICAgICAgICAg cmV0dXJuIHN1bShsZW4odmFsdWUpIGZvciB2YWx1ZSBpbiB0ZXN0cy5pdGVydmFsdWVzKCkpCisK ICAgICAgICAgZGVmIHJlcG9ydCh0ZXN0cywgdGl0bGUsIGJhc2VfZGlyKToKICAgICAgICAgICAg IGlmIG5vdCB0ZXN0czoKICAgICAgICAgICAgICAgICByZXR1cm4KLSAgICAgICAgICAgIHN5cy5z dGRvdXQud3JpdGUoIlxuVW5leHBlY3RlZCAlcyAoJWQpXG4iICUgKHRpdGxlLCBzdW0obGVuKHZh bHVlKSBmb3IgdmFsdWUgaW4gdGVzdHMuaXRlcnZhbHVlcygpKSkpCisgICAgICAgICAgICBzeXMu c3Rkb3V0LndyaXRlKCJcblVuZXhwZWN0ZWQgJXMgKCVkKVxuIiAlICh0aXRsZSwgbnVtYmVyX29m X3Rlc3RzKHRlc3RzKSkpCiAgICAgICAgICAgICBmb3IgdGVzdCBpbiB0ZXN0czoKICAgICAgICAg ICAgICAgICBzeXMuc3Rkb3V0LndyaXRlKCIgICAgJXNcbiIgJSAodGVzdC5yZXBsYWNlKGJhc2Vf ZGlyLCAnJywgMSkpKQogICAgICAgICAgICAgICAgIGZvciB0ZXN0X2Nhc2UgaW4gdGVzdHNbdGVz dF06CkBAIC0zNDAsNyArMzUwLDE0IEBAIGNsYXNzIFRlc3RSdW5uZXIob2JqZWN0KToKICAgICAg ICAgICAgIHJlc3VsdF9kaWN0aW9uYXJ5WydUaW1lZG91dCddID0gZ2VuZXJhdGVfdGVzdF9saXN0 X2Zvcl9qc29uX291dHB1dChzZWxmLl90ZXN0X3Byb2dyYW1zX2Jhc2VfZGlyKCksIHRpbWVkX291 dF90ZXN0cykKICAgICAgICAgICAgIHNlbGYuX3BvcnQuaG9zdC5maWxlc3lzdGVtLndyaXRlX3Rl eHRfZmlsZShzZWxmLl9vcHRpb25zLmpzb25fb3V0cHV0LCBqc29uLmR1bXBzKHJlc3VsdF9kaWN0 aW9uYXJ5LCBpbmRlbnQ9NCkpCiAKLSAgICAgICAgcmV0dXJuIGxlbihmYWlsZWRfdGVzdHMpICsg bGVuKHRpbWVkX291dF90ZXN0cykKKyAgICAgICAgbnVtYmVyX29mX2ZhaWxlZF90ZXN0cyA9IG51 bWJlcl9vZl90ZXN0cyhmYWlsZWRfdGVzdHMpICsgbnVtYmVyX29mX3Rlc3RzKHRpbWVkX291dF90 ZXN0cykgKyBudW1iZXJfb2ZfdGVzdHMoY3Jhc2hlZF90ZXN0cykKKyAgICAgICAgbnVtYmVyX29m X3N1Y2Nlc3NmdWxfdGVzdHMgPSBudW1iZXJfb2ZfZXhlY3V0ZWRfdGVzdHMgLSBudW1iZXJfb2Zf ZmFpbGVkX3Rlc3RzCisKKyAgICAgICAgc3lzLnN0ZG91dC53cml0ZSgiXG5SYW4gJWQgdGVzdHMg b2YgJWQgd2l0aCAlZCBzdWNjZXNzZnVsXG4iICUgKG51bWJlcl9vZl9leGVjdXRlZF90ZXN0cywg bnVtYmVyX29mX3RvdGFsX3Rlc3RzLCBudW1iZXJfb2Zfc3VjY2Vzc2Z1bF90ZXN0cykpCisgICAg ICAgIHN5cy5zdGRvdXQuZmx1c2goKQorCisgICAgICAgIHJldHVybiBudW1iZXJfb2ZfZmFpbGVk X3Rlc3RzCisKIAogCiBkZWYgYWRkX29wdGlvbnMob3B0aW9uX3BhcnNlcik6Cg==