183788 2018-03-20 12:32:16 -0700 [GTK][STABLE] Non-deterministic build failure (missing JavaScriptCore/JSContextRef.h) 2018-07-25 09:47:23 -0700 1 1 1 Unclassified WebKit WebKitGTK Other Unspecified Linux RESOLVED FIXED P2 Normal --- 1 jtojnar fujii annulen aperez bugs-noreply crispyfrog don.olmstead fujii karogyoker2+webkit mcatanzaro torhedinbronner oldest_to_newest 1407748 0 jtojnar 2018-03-20 12:32:16 -0700 Building WebkitGTK on NixOS fails non-deterministically with the following error: FAILED: Source/JavaScriptCore/CMakeFiles/LLIntOffsetsExtractor.dir/llint/LLIntOffsetsExtractor.cpp.o /nix/store/kw0c6x65jx28d5mr6zbb6978w61b8n93-gcc-wrapper-6.4.0/bin/g++ -DBUILDING_GTK__=1 -DBUILDING_WITH_CMAKE=1 -DDATA_DIR=\"share\" -DGETTEXT_PACKAGE=\"WebKit2GTK-4.0\" -DHAVE_CONFIG_H=1 -DLIBDIR=\"/nix/store/fqncjlck2aqdmr8c0r9c8jj4c3819v51-webkitgtk-2.20.0/lib\" -DSTATICALLY_LINKED_WITH_WTF -DWEBKITGTK_API_VERSION_STRING=\"4.0\" -IDerivedSources/ForwardingHeaders -I. -I../Source/JavaScriptCore -I../Source/JavaScriptCore/API -I../Source/JavaScriptCore/assembler -I../Source/JavaScriptCore/b3 -I../Source/JavaScriptCore/b3/air -I../Source/JavaScriptCore/bindings -I../Source/JavaScriptCore/builtins -I../Source/JavaScriptCore/bytecode -I../Source/JavaScriptCore/bytecompiler -I../Source/JavaScriptCore/dfg -I../Source/JavaScriptCore/disassembler -I../Source/JavaScriptCore/disassembler/ARM64 -I../Source/JavaScriptCore/disassembler/ARMv7 -I../Source/JavaScriptCore/disassembler/udis86 -I../Source/JavaScriptCore/domjit -I../Source/JavaScriptCore/ftl -I../Source/JavaScriptCore/heap -I../Source/JavaScriptCore/debugger -I../Source/JavaScriptCore/inspector -I../Source/JavaScriptCore/inspector/agents -I../Source/JavaScriptCore/inspector/augmentable -I../Source/JavaScriptCore/inspector/remote -I../Source/JavaScriptCore/interpreter -I../Source/JavaScriptCore/jit -I../Source/JavaScriptCore/llint -I../Source/JavaScriptCore/parser -I../Source/JavaScriptCore/profiler -I../Source/JavaScriptCore/runtime -I../Source/JavaScriptCore/tools -I../Source/JavaScriptCore/wasm -I../Source/JavaScriptCore/wasm/js -I../Source/JavaScriptCore/yarr -IDerivedSources/JavaScriptCore -IDerivedSources/JavaScriptCore/inspector -IDerivedSources/JavaScriptCore/runtime -IDerivedSources/JavaScriptCore/yarr -I../Source/JavaScriptCore/inspector/remote/glib -isystem /nix/store/slggzv99kinwm4c2lm7wfmifh8y69l7r-glib-2.56.0-dev/include/glib-2.0 -isystem /nix/store/dqlc8y4phlg1hmdbwkhqfwhnxcac88d1-glib-2.56.0/lib/glib-2.0/include -I../Source/bmalloc -IDerivedSources -I../Source/ThirdParty -fdiagnostics-color=always -Wno-attributes -Wno-maybe-uninitialized -Wwrite-strings -Wundef -Wpointer-arith -Wmissing-format-attribute -Wformat-security -Wcast-align -Wextra -Wall -fno-strict-aliasing -fno-exceptions -std=c++14 -fno-rtti -O3 -DNDEBUG -fPIE -MD -MT Source/JavaScriptCore/CMakeFiles/LLIntOffsetsExtractor.dir/llint/LLIntOffsetsExtractor.cpp.o -MF Source/JavaScriptCore/CMakeFiles/LLIntOffsetsExtractor.dir/llint/LLIntOffsetsExtractor.cpp.o.d -o Source/JavaScriptCore/CMakeFiles/LLIntOffsetsExtractor.dir/llint/LLIntOffsetsExtractor.cpp.o -c ../Source/JavaScriptCore/llint/LLIntOffsetsExtractor.cpp In file included from ../Source/JavaScriptCore/heap/HeapFinalizerCallback.h:28:0, from ../Source/JavaScriptCore/heap/Heap.h:33, from ../Source/JavaScriptCore/heap/DeferGC.h:29, from ../Source/JavaScriptCore/runtime/ConcurrentJSLock.h:28, from ../Source/JavaScriptCore/bytecode/ArrayProfile.h:28, from ../Source/JavaScriptCore/llint/LLIntOffsetsExtractor.cpp:28: ../Source/JavaScriptCore/API/JSHeapFinalizerPrivate.h:29:41: fatal error: JavaScriptCore/JSContextRef.h: No such file or directory #include <JavaScriptCore/JSContextRef.h> ^ 1409657 1 aperez 2018-03-27 13:45:29 -0700 I've seen this happen in the EWS that I maintain, but back then I just went for a wipe of the build directory and that fixed things, so I assumed that the build failure had been caused by some patch that the EWS had tried and didn't recover from. I will try to investigate if this happens again in my EWS. Jan: By any chance did this happen to you in a machine with many CPUs? I have only seen the issue in the EWS, which runs on a 64-core box, but I've never seen it in my work laptop. It looks a bit like it could be some parallel build issue, which would explain why it triggers more easily with a high core count. 1409666 2 jtojnar 2018-03-27 13:58:53 -0700 Undeclared dependencies causing trouble during parallel builds is what I would suspect but we have a report of failure with parallel builds disabled (https://github.com/NixOS/nixpkgs/pull/36893#issuecomment-373922664). I can reproduce it on my four core laptop (build flags: -j4 -l4). 1410246 3 jtojnar 2018-03-29 07:13:36 -0700 Lowering the parallelism seemingly increases the chance of encountering this issue https://github.com/NixOS/nixpkgs/issues/37878#comment-377233401 1410851 4 crispyfrog 2018-03-31 08:12:21 -0700 This bug happens every time for me. I build it with ninja -j1. I think the problem is in some CMake rules. The missing header JavaScriptCore/JsContextRef.h should be generated during build, as with many other headers, they're are copied from their respective locations to build/DerivedSources/ForwardingHeaders. There's a rule in build/build.ninja to make this happen: build JavascriptCoreForwardingHeaders: phony Source/JavaScriptCore/JavaScriptCoreForwardingHeaders This rule should be a prerequisite of JavaScriptCore, but it doesn't happen due to some bugs in Cmake rules I suppose. The lines in Cmake files to generate the ninja rules are at: Source/JavaScriptCore/CMakeLists.txt:1152, it calls function WEBKIT_MAKE_FORWARDING_HEADERS, which is defined at Source/cmake/WebKitMacros.cmake :218 Here line 267 might be the problem: add_dependencies(...) in short, this line should instruct cmake to add JavaScriptCoreForwardingHeaders as a prerequisite for JavaScriptCore in the generated ninja build file, but for some reason it doesn't work. Due to my limited knowledge to CMake, I couldn't​ figure out the right way to fix this. Well still there's a workaround, that is to call ninja to generate those headers explicitly. Do this after cmake and before you run ninja to build: ninja JavaScriptCoreForwardingHeaders WTFForwardingHeaders 1444540 5 mcatanzaro 2018-07-23 19:46:17 -0700 *** Bug 187894 has been marked as a duplicate of this bug. *** 1444545 6 fujii 2018-07-23 21:41:56 -0700 https://github.com/WebKit/webkit/blob/2071d472faf10105ea6858e7795e5366f6203667/Source/JavaScriptCore/CMakeLists.txt#L258 > add_dependencies(LLIntOffsetsExtractor JavaScriptCoreForwardingHeaders) LLIntOffsetsExtractor has a dependency to JavaScriptCoreForwardingHeaders. But, this is a link-time dependency. It should be a compile-time dependency. Caused By: Bug 182303 and Bug 183251. 1444549 7 fujii 2018-07-23 22:02:55 -0700 I'm wrong. A following command doesn't fail in my env. > ./Tools/Scripts/build-webkit --gtk --release --makeargs="-j1 Source/JavaScriptCore/CMakeFiles/LLIntOffsetsExtractor.dir/llint/LLIntOffsetsExtractor.cpp.o" Looking into build.ninja, LLIntOffsetsExtractor.cpp.o has a dependency to JavaScriptCoreForwardingHeaders. I'm using CMake 3.10.2. 1444550 8 fujii 2018-07-23 22:25:11 -0700 I can't figure out what is happening in their build env. I uploaded my build log which successes. https://gist.github.com/fujii/ed836a1be308c5775482c0bef927e902 > [474/483] Generating ../../DerivedSources/ForwardingHeaders/JavaScriptCore/JSContextRef.h (...) > [483/483] Building CXX object Source/JavaScriptCore/CMakeFiles/LLIntOffsetsExtractor.dir/llint/LLIntOffsetsExtractor.cpp.o ForwardingHeaders/JavaScriptCore/JSContextRef.h was generated before compiling LLIntOffsetsExtractor.cpp.o. 1444569 9 fujii 2018-07-24 00:21:54 -0700 Finally, I can reproduce the same build error with webkitgtk-2.20.3.tar.xz. The source code doesn't have a following code. > add_dependencies(LLIntOffsetsExtractor JavaScriptCoreForwardingHeaders) This problem had been introduce in r227952, and fixed in r229282. webkitgtk-2.20 was branch from trunk@228089. Bug 182303 (r227952) Bug 183251 (r229282) 1444662 10 mcatanzaro 2018-07-24 07:15:17 -0700 Thanks Fujii! I've added r229282 to the backports list for 2.20.4. 1445051 11 mcatanzaro 2018-07-25 09:32:46 -0700 (In reply to Michael Catanzaro from comment #10) > Thanks Fujii! I've added r229282 to the backports list for 2.20.4. It doesn't apply cleanly, so I'll do this manually. See also: https://bugs.gentoo.org/662002 1445056 12 mcatanzaro 2018-07-25 09:47:23 -0700 Committed r234198: <https://trac.webkit.org/changeset/234198>