30869 2009-10-28 09:55:08 -0700 commit-queue keeps crashing 2009-10-30 15:43:24 -0700 1 1 1 Unclassified WebKit Tools / Tests 528+ (Nightly build) PC OS X 10.5 RESOLVED DUPLICATE 30084 P2 Normal --- 1 eric webkit-unassigned abarth evan levin steveblock oldest_to_newest 158610 0 eric 2009-10-28 09:55:08 -0700 commit-queue keeps crashing Exception while checking queue and bots: unknown encoding: idna. Sleeping until 2009-10-27 23:07:56 (5 mins). 'import site' failed; use -v for traceback Traceback (most recent call last): File "/Users/eseidel/Projects/WebKit/WebKitTools/Scripts/bugzilla-tool", line 33, in <module> import os File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/os.py", line 49, in <module> import posixpath as path File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/posixpath.py", line 14, in <module> import stat ImportError: No module named stat I think that's happening on re-exec. I think it may only happen when I have the patch from bug 30098 applied locally. Either way, it's disturbing. It's happened 3 times at seeming random. The commit-queue used to be rock-solid! 158995 1 eric 2009-10-29 11:48:50 -0700 This appears to be a python bug. I've made a reduced test case. 158996 2 42122 eric 2009-10-29 11:50:21 -0700 Created attachment 42122 Script demonstrating python 2.5.1 (or mac os x?) bug 158999 3 evan 2009-10-29 12:00:20 -0700 On Linux (the "import site" bits are when I'm repeatedly hitting ctl-C). Python 2.5.2 % ./t ...............................................................................'import site' failed; use -v for traceback .........'import site' failed; use -v for traceback ............'import site' failed; use -v for traceback ...........'import site' failed; use -v for traceback ....................'import site' failed; use -v for traceback .......... 159000 4 eric 2009-10-29 12:02:36 -0700 Bah. I just don't know how exec works. I didn't realize it carried file handles through. 159002 5 42123 eric 2009-10-29 12:04:02 -0700 Created attachment 42123 Patch v1 159005 6 42123 abarth 2009-10-29 12:10:07 -0700 Comment on attachment 42123 Patch v1 File handle leaks = bad. 159015 7 42123 eric 2009-10-29 12:51:32 -0700 Comment on attachment 42123 Patch v1 Going to make a better patch. 159384 8 eric 2009-10-30 11:42:03 -0700 It would appear that even with my fix, Python itself is still leaking 3 file descriptors on every exec(): Python 57958 eseidel 58r REG 14,2 144580 466329 /System/Library/Frameworks/Carbon.framework/Versions/A/Frameworks/HIToolbox.framework/Versions/A/Resources/HITo olbox.rsrc Python 57958 eseidel 59r REG 14,2 490410 9627854 /System/Library/Frameworks/Carbon.framework/Versions/A/Frameworks/HIToolbox.framework/Versions/A/Resources/Engl ish.lproj/Localized.rsrc Python 57958 eseidel 60r REG 14,2 86770 481232 /System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/plat-mac/errors.rsrc.df.rsrc I expect this may be a mac-only problem. 159448 9 eric 2009-10-30 14:33:15 -0700 I wrote a function: def check_for_file_leak(message): log("before %s" % message) lsof_output = SCM.run_command(['lsof', '-p', str(os.getpid())]) if lsof_output.count('HIToolbox'): log(message) error(lsof_output) log("after %s" % message) and sprinkled it throughout the code. It seems these 3 leaking files are opened by mechanize during: self.browser = Browser() in Bugzilla.__init__() I've not tried to trace through the mechanize code to see where the leaks come from. 159449 10 abarth 2009-10-30 14:38:05 -0700 Why don't we fork and exec and then throw away the old version of ourselves? If that doesn't work, can we use a trampoline executable? Tracking down every leak seems fragile. 159456 11 eric 2009-10-30 14:41:12 -0700 Here is a stand-alone version of the same function: def check_for_file_leak(message): import os import sys import subprocess print >> sys.stderr, "before %s" % message process = subprocess.Popen(['lsof', '-p', str(os.getpid())], stdout=subprocess.PIPE, stderr=subprocess.STDOUT) lsof_output = process.communicate()[0].rstrip() if lsof_output.count('HIToolbox'): print >> sys.stderr, message print >> sys.stderr, lsof_output exit(1) print >> sys.stderr, "after %s" % message 159457 12 eric 2009-10-30 14:41:47 -0700 We could definitely use a trampoline executable. That may be the simplest solution. 159460 13 eric 2009-10-30 14:43:21 -0700 bugzilla-tool commit-queue is already a trampoline executable of sorts. It's just no longer a shell script, instead it's implemented in bugzilla-tool itself. I think I might just get rid of the "don't need to restart bugzilla-tool to notice committers.py changes" feature for now. 159492 14 eric 2009-10-30 15:43:24 -0700 Rolled out the change and re-opened bug 30084. *** This bug has been marked as a duplicate of bug 30084 *** 42122 2009-10-29 11:50:21 -0700 2009-10-29 11:50:21 -0700 Script demonstrating python 2.5.1 (or mac os x?) bug reexec_test.py text/x-python-script 235 eric IyEvdXNyL2Jpbi9lbnYgcHl0aG9uCgppbXBvcnQgb3MKaW1wb3J0IHN5cwoKIyBMZWFrIGEgZmls ZSBoYW5kbGUKbG9nX2ZpbGUgPSBvcGVuKCdkdW1teV9sb2cudHh0JywgJ2ErJykKIyBsb2dfZmls ZS5jbG9zZSgpICMgSWYgd2UgY2xvc2UgdGhlIGhhbmRsZSwgaXQgbmV2ZXIgY3Jhc2hlcy4KCnBy aW50ICIuIiwKc3lzLnN0ZG91dC5mbHVzaCgpCm9zLmV4ZWN2cChzeXMuYXJndlswXSwgc3lzLmFy Z3ZbOl0pCg== 42123 2009-10-29 12:04:02 -0700 2009-10-29 12:51:31 -0700 Patch v1 bug-30869-20091029120401.patch text/plain 1470 eric ZGlmZiAtLWdpdCBhL1dlYktpdFRvb2xzL0NoYW5nZUxvZyBiL1dlYktpdFRvb2xzL0NoYW5nZUxv ZwppbmRleCBmYjY2ZmJlLi5iMGNiMTI0IDEwMDY0NAotLS0gYS9XZWJLaXRUb29scy9DaGFuZ2VM b2cKKysrIGIvV2ViS2l0VG9vbHMvQ2hhbmdlTG9nCkBAIC0xLDMgKzEsMTYgQEAKKzIwMDktMTAt MjkgIEVyaWMgU2VpZGVsICA8ZXJpY0B3ZWJraXQub3JnPgorCisgICAgICAgIFJldmlld2VkIGJ5 IE5PQk9EWSAoT09QUyEpLgorCisgICAgICAgIGNvbW1pdC1xdWV1ZSBrZWVwcyBjcmFzaGluZwor ICAgICAgICBodHRwczovL2J1Z3Mud2Via2l0Lm9yZy9zaG93X2J1Zy5jZ2k/aWQ9MzA4NjkKKwor ICAgICAgICBleGVjKCkgKGludGVudGlvbmFsbHkpIGNhcnJpZXMgZmlsZSBoYW5kbGVzIHRocm91 Z2gsCisgICAgICAgIG1ha2Ugc3VyZSB3ZSBjbG9zZSBldmVyeSBmaWxlIGhhbmRsZSBiZWZvcmUg ZXhlY2luZyBzbworICAgICAgICB0aGF0IHdlIGRvbid0IHJ1biBvdXQuCisKKyAgICAgICAgKiBT Y3JpcHRzL2J1Z3ppbGxhLXRvb2w6CisKIDIwMDktMTAtMjggIEtlbm5ldGggUm9oZGUgQ2hyaXN0 aWFuc2VuICA8a2VubmV0aEB3ZWJraXQub3JnPgogCiAgICAgICAgIFJ1YmJlcnN0YW1wZWQgYnkg T2xpdmVyIEh1bnQuCmRpZmYgLS1naXQgYS9XZWJLaXRUb29scy9TY3JpcHRzL2J1Z3ppbGxhLXRv b2wgYi9XZWJLaXRUb29scy9TY3JpcHRzL2J1Z3ppbGxhLXRvb2wKaW5kZXggOGU4OTliNS4uNmIy YmQyZSAxMDA3NTUKLS0tIGEvV2ViS2l0VG9vbHMvU2NyaXB0cy9idWd6aWxsYS10b29sCisrKyBi L1dlYktpdFRvb2xzL1NjcmlwdHMvYnVnemlsbGEtdG9vbApAQCAtNzY5LDcgKzc2OSw5IEBAIGNs YXNzIExhbmRQYXRjaGVzRnJvbUNvbW1pdFF1ZXVlKENvbW1hbmQpOgogICAgICAgICAgICAgaWYg ZS5leGl0X2NvZGUgIT0gMjoKICAgICAgICAgICAgICAgICB0b29sLmJ1Z3MucmVqZWN0X3BhdGNo X2Zyb21fY29tbWl0X3F1ZXVlKHBhdGNoWydpZCddLCAiVW5leHBlY3RlZCBmYWlsdXJlIHdoZW4g bGFuZGluZyBwYXRjaCEgIFBsZWFzZSBmaWxlIGEgYnVnIGFnYWluc3QgYnVnemlsbGEtdG9vbC5c biVzIiAlIGUubWVzc2FnZV93aXRoX291dHB1dCgpKQogICAgICAgICBzZWxmLl9yZW1vdmVfbG9n X2Zyb21fb3V0cHV0X3RlZShidWdfbG9nKQotICAgICAgICAjIHNlbGYuX3JlbW92ZV9sb2dfZnJv bV9vdXRwdXRfdGVlKHF1ZXVlX2xvZykgIyBpbXBsaWNpdCBpbiB0aGUgZXhlYygpCisgICAgICAg ICMgQmUgY2FyZWZ1bCB0byBjbG9zZSBldmVyeSBmaWxlIGhhbmRsZSBiZWZvcmUgcmUtZXhlYwor ICAgICAgICAjIHRvIGF2b2lkIHJ1bm5pbmcgb3V0LiAgaHR0cHM6Ly9idWdzLndlYmtpdC5vcmcv c2hvd19idWcuY2dpP2lkPTMwODY5CisgICAgICAgIHNlbGYuX3JlbW92ZV9sb2dfZnJvbV9vdXRw dXRfdGVlKHF1ZXVlX2xvZykKICAgICAgICAgc2VsZi5fbmV4dF9wYXRjaCgp