212500 2020-05-28 16:04:23 -0700 REGRESSION (r262245): WebCore's "Check .xcfilelists" build phase is ~100x slower 2020-05-29 12:37:24 -0700 1 1 1 Unclassified WebKit Tools / Tests WebKit Local Build Unspecified macOS 10.15 RESOLVED DUPLICATE 212451 https://bugs.webkit.org/show_bug.cgi?id=212420 InRadar P2 Normal --- 212451 1 aestes webkit-unassigned aestes ap darin krollin ryanhaddad sam webkit-bug-importer ysuzuki oldest_to_newest 1657072 0 aestes 2020-05-28 16:04:23 -0700 Prior to <https://trac.webkit.org/r262245>, when building the 'All Source' scheme in WebKit.xcworkspace, WebCore's "Check.xcfilelists" build phase took ~1 second. With r262245, it takes ~95 seconds. Modifying Source/WebCore/Scripts/check-xcfilelists.sh to invoke generate-xcfilelists with the "--debug" flag, I see that almost all the time is spent in a subshell invoking Source/WebCore/Scripts/generate-derived-sources.sh. 1657076 1 webkit-bug-importer 2020-05-28 16:08:18 -0700 <rdar://problem/63736777> 1657082 2 darin 2020-05-28 16:35:25 -0700 The changes in the patch in bug 212420 might fix this. For one thing, it won’t call generate-derived-sources.sh unless something has changed. 1657112 3 darin 2020-05-28 17:46:47 -0700 I have no idea why this happened, but I do think that the patch from that other bug will make the problem disappear. 1657283 4 darin 2020-05-29 09:16:29 -0700 Maybe I am being overconfident here; I don’t really know the details of what check xcfilelists build phase does, so I am not sure how I caused this. We could roll out until we figure it out? Keith, do you understand what this could be? 1657297 5 aestes 2020-05-29 09:47:19 -0700 (In reply to Darin Adler from comment #2) > The changes in the patch in bug 212420 might fix this. For one thing, it > won’t call generate-derived-sources.sh unless something has changed. I applied attachment #400525 to my tree but I still see ~95s runtimes for WebCore's "Check .xcfilelists". 1657313 6 krollin 2020-05-29 10:22:10 -0700 (In reply to Darin Adler from comment #4) > > Keith, do you understand what this could be? Not off the top of my head. Andy says that the time is spent in generate-derived-sources.sh. The xcfilelist checking process invokes this script to run `make -f DerivedSources.make -d` so that it can capture the dependency information. There's not much to the script other than invoking `make` so I'd think all the time would be spent in `make`. I wonder if there's something weird about your changes and in invoking `make` with the -d option (which is what dumps the dependency information). Could `make` now be dumping tons of information? 1657321 7 darin 2020-05-29 10:35:39 -0700 (In reply to Keith Rollin from comment #6) > The xcfilelist checking process invokes this > script to run `make -f DerivedSources.make -d` so that it can capture the > dependency information. Oh! Had no idea. I thought that DerivedSources.make was only processed once when building. Need to figure out why it’s taking *so* long to process the makefile. Please roll r262245 out until we can solve this! 1657328 8 darin 2020-05-29 10:40:25 -0700 It’s possible that make now knows about a lot more dependencies, so the output of "make -d" is much bigger, and either make itself is slow, or the code to process it is slow when there's a lot of data. It’s possible the invocation of "make -d" is not passing the same environment variables that generate-derived-sources.sh is, leading to something particularly slow. But neither of those things seems particularly likely. I think we have a mystery. 1657331 9 darin 2020-05-29 10:42:38 -0700 Maybe the $(shell) commands are evaluated every time you use FEATURE_AND_PLATFORM_DEFINES rather than only once as the file is processed? Might just need to use an operator other than "=" to fix this. 1657334 10 aestes 2020-05-29 10:43:40 -0700 (In reply to Darin Adler from comment #7) > (In reply to Keith Rollin from comment #6) > > The xcfilelist checking process invokes this > > script to run `make -f DerivedSources.make -d` so that it can capture the > > dependency information. > > Oh! Had no idea. I thought that DerivedSources.make was only processed once > when building. Need to figure out why it’s taking *so* long to process the > makefile. > > Please roll r262245 out until we can solve this! Reverted in https://trac.webkit.org/r262303. 1657336 11 aestes 2020-05-29 10:43:56 -0700 *** Bug 212531 has been marked as a duplicate of this bug. *** 1657337 12 darin 2020-05-29 10:45:47 -0700 Yes! The fix is to write ":=" instead of "=" on these lines using @(shell). Of course I figure this out right after getting the patch rolled out. 1657382 13 aestes 2020-05-29 12:37:24 -0700 *** This bug has been marked as a duplicate of bug 212451 ***