29939 2009-09-30 15:52:38 -0700 http/tests/xmlhttprequest/basic-auth.html timed out on Leopard bot 2011-04-13 11:55:34 -0700 1 1 1 Unclassified WebKit New Bugs 528+ (Nightly build) PC OS X 10.5 RESOLVED FIXED P2 Normal --- 51613 33292 1 eric webkit-unassigned abarth ap mjs ossy oldest_to_newest 151378 0 eric 2009-09-30 15:52:38 -0700 http/tests/xmlhttprequest/basic-auth.html timed out on Leopard bot http://build.webkit.org/results/Leopard%20Intel%20Release%20(Tests)/r48942%20(5588)/results.html I don't have frequency numbers for you yet. But I'll add notes to this bug when I see it. Not sure who to CC here, but I think ap has hacked in this code and might know. 151452 1 eric 2009-09-30 22:29:52 -0700 Another one this evening: http://build.webkit.org/results/Leopard%20Intel%20Release%20(Tests)/r48942%20(5588)/results.html 179113 2 eric 2010-01-09 19:38:21 -0800 Just timed out on the Snow Leopard Release bot as well: http://build.webkit.org/results/SnowLeopard%20Intel%20Release%20(Tests)/r53047%20(4086)/results.html We seem to see a lot of trouble with these auth tests on all bots. I suspect we may have a deeper problem here. 183406 3 eric 2010-01-21 17:16:14 -0800 Timed out on Snow Leopard Release just now: http://build.webkit.org/results/SnowLeopard%20Intel%20Release%20(Tests)/r53660%20(4646)/results.html 183409 4 eric 2010-01-21 17:17:04 -0800 http://trac.webkit.org/browser/trunk/LayoutTests/http/tests/xmlhttprequest/basic-auth.html 183411 5 47163 eric 2010-01-21 17:20:00 -0800 Created attachment 47163 Patch 183413 6 eric 2010-01-21 17:23:06 -0800 There are various other possibly related test failures: https://bugs.webkit.org/show_bug.cgi?id=33357 https://bugs.webkit.org/show_bug.cgi?id=33301 https://bugs.webkit.org/show_bug.cgi?id=32961 https://bugs.webkit.org/show_bug.cgi?id=30669 https://bugs.webkit.org/show_bug.cgi?id=30726 https://bugs.webkit.org/show_bug.cgi?id=29939 183455 7 47163 ap 2010-01-21 20:15:21 -0800 Comment on attachment 47163 Patch I don't want to skip this. We previously had authentication tests crash, and that helped catch a bad regression - no sense in losing regression testing for auth code. 183588 8 eric 2010-01-22 03:56:51 -0800 The auth tests clearly are flakey, as demonstrated in the numerous bugs above. What should we do if we don't skip them? 183662 9 ap 2010-01-22 08:35:52 -0800 As long as there is no expectations mechanism to keep testing for crashes and further regressions, all we can do is suffer the flakiness. Assuming no one is going to try and fix it. 183741 10 eric 2010-01-22 12:15:10 -0800 I agree that a test expectations mechanism is the right way to go. Sadly we don't have such yet. Leaving the bots red until some future time when we do is silly. I also think it's silly for one test to hold 11,000 tests hostage. Which is exactly what flakey tests do. They reduce the usefulness of the rest of the tests, but making it harder for someone to know if their patch is correct or not. People stop trusting the tests and bots. By saying "but what if it catches a crasher later" you're arguing that we should exchange some current time value for some unlikely future time value. The current value is the value of green bots catching real regressions quickly because people know that red means they broken stuff. The unlikely future value is that this test would be the only one to catch some crasher. We could put numbers on such a value estimate, but I assure you the future value is not worth the current cost. I agree skipping tests is less than ideal, but it's the only tool we currently have to keep the bots green. That or rollouts. Since we can't rollout the change that broke this test, or realistically the one which added it, the best solution we have is to skip it with plans to mark it flakey instead when we have the technology. :) 208987 11 ossy 2010-04-06 04:49:55 -0700 Failed on Snow Leopard: http://build.webkit.org/results/SnowLeopard%20Intel%20Release%20%28Tests%29/r57141%20%287752%29/http/tests/xmlhttprequest/basic-auth-pretty-diff.html 326999 12 abarth 2010-12-27 00:47:55 -0800 Eight months later and this is still failing very frequently on the bots. Sadness. 385100 13 eric 2011-04-13 11:55:34 -0700 This must have been skipped or something. We haven't seen it fail in 4 months. Closing. 47163 2010-01-21 17:20:00 -0800 2010-06-11 10:43:32 -0700 Patch bug-29939-20100121171959.patch text/plain 1028 eric ZGlmZiAtLWdpdCBhL0xheW91dFRlc3RzL0NoYW5nZUxvZyBiL0xheW91dFRlc3RzL0NoYW5nZUxv ZwppbmRleCBiZWJmNmY1Li42ODhiYWU1IDEwMDY0NAotLS0gYS9MYXlvdXRUZXN0cy9DaGFuZ2VM b2cKKysrIGIvTGF5b3V0VGVzdHMvQ2hhbmdlTG9nCkBAIC0xLDMgKzEsMTIgQEAKKzIwMTAtMDEt MjEgIEVyaWMgU2VpZGVsICA8ZXJpY0B3ZWJraXQub3JnPgorCisgICAgICAgIFJldmlld2VkIGJ5 IE5PQk9EWSAoT09QUyEpLgorCisgICAgICAgIGh0dHAvdGVzdHMveG1saHR0cHJlcXVlc3QvYmFz aWMtYXV0aC5odG1sIHRpbWVkIG91dCBvbiBMZW9wYXJkIGJvdAorICAgICAgICBodHRwczovL2J1 Z3Mud2Via2l0Lm9yZy9zaG93X2J1Zy5jZ2k/aWQ9Mjk5MzkKKworICAgICAgICAqIHBsYXRmb3Jt L21hYy9Ta2lwcGVkOiBObyBjbHVlIHdoeSB0aGlzIGlzIGZhaWxpbmcuICBObyBzZW5zZSBpbiBr ZWVwaW5nIHRoZSBib3RzIHJlZC4KKwogMjAxMC0wMS0yMSAgRW5yaWNhIENhc3VjY2kgIDxlbnJp Y2FAYXBwbGUuY29tPgogCiAgICAgICAgIFJldmlld2VkIGJ5IERhcmluIEFkbGVyLgpkaWZmIC0t Z2l0IGEvTGF5b3V0VGVzdHMvcGxhdGZvcm0vbWFjL1NraXBwZWQgYi9MYXlvdXRUZXN0cy9wbGF0 Zm9ybS9tYWMvU2tpcHBlZAppbmRleCBlMWQ5ZmNlLi4wMzNmNzUyIDEwMDY0NAotLS0gYS9MYXlv dXRUZXN0cy9wbGF0Zm9ybS9tYWMvU2tpcHBlZAorKysgYi9MYXlvdXRUZXN0cy9wbGF0Zm9ybS9t YWMvU2tpcHBlZApAQCAtOTUsMyArOTUsNyBAQCBtZWRpYS9tZWRpYS1jYW4tcGxheS1vZ2cuaHRt bAogCiAjIFRoZXNlIHRlc3RzIHJlcXVpcmUgdG91Y2ggc3VwcG9ydAogZmFzdC9ldmVudHMvdG91 Y2gKKworIyBIYXMgdGltZWQgb3V0IG9uIGJvdGggTGVvcGFyZCBhbmQgU25vdyBMZW9wYXJkIGJv dHMgZm9yIHVua25vd24gcmVhc29ucy4KKyMgaHR0cHM6Ly9idWdzLndlYmtpdC5vcmcvc2hvd19i dWcuY2dpP2lkPTI5OTM5CitodHRwL3Rlc3RzL3htbGh0dHByZXF1ZXN0L2Jhc2ljLWF1dGguaHRt bAo=