WebKit Bugzilla
New
Browse
Search+
Log In
×
Sign in with GitHub
or
Remember my login
Create Account
·
Forgot Password
Forgotten password account recovery
[patch]
Patch
bug-147508-20150824191711.patch (text/plain), 131.28 KB, created by
Basile Clement
on 2015-08-24 19:17:20 PDT
(
hide
)
Description:
Patch
Filename:
MIME Type:
Creator:
Basile Clement
Created:
2015-08-24 19:17:20 PDT
Size:
131.28 KB
patch
obsolete
>Subversion Revision: 188889 >diff --git a/Source/JavaScriptCore/ChangeLog b/Source/JavaScriptCore/ChangeLog >index cf644521da997e8db223702fa539d3c762f2132a..69024a8fba102b40ed11f87e343b06c8cbb137b7 100644 >--- a/Source/JavaScriptCore/ChangeLog >+++ b/Source/JavaScriptCore/ChangeLog >@@ -1,3 +1,130 @@ >+2015-08-24 Basile Clement <basile_clement@apple.com> >+ >+ jsc-tailcall: We should reuse the frame efficiently in the DFG instead of doing a memmove >+ https://bugs.webkit.org/show_bug.cgi?id=147508 >+ >+ Reviewed by Michael Saboff. >+ >+ This introduces a new class (CallFrameShuffler) that is responsible for >+ efficiently building the new frames when performing a tail call. In >+ order for Repatch to know about the position of arguments on the >+ stack/registers (e.g. for polymorphic call inline caches), we store a >+ CallFrameShuffleData in the CallLinkInfo. Otherwise, the JIT and DFG >+ compiler are now using CallFrameShuffler instead of >+ CCallHelpers::prepareForTailCallSlow() to build the frame for a tail >+ call. >+ >+ When taking a slow path, we still build the frame as if doing a regular >+ call, because we could throw an exception and need the caller's frame >+ at that point. This means that for virtual calls, we don't benefit from >+ the efficient frame move. We will take care of this in a subsequent patch. >+ >+ * CMakeLists.txt: >+ * JavaScriptCore.vcxproj/JavaScriptCore.vcxproj: >+ * JavaScriptCore.vcxproj/JavaScriptCore.vcxproj.filters: >+ * JavaScriptCore.xcodeproj/project.pbxproj: >+ * assembler/AbortReason.h: >+ * bytecode/CallLinkInfo.h: >+ (JSC::CallLinkInfo::setFrameShuffleData): >+ (JSC::CallLinkInfo::frameShuffleData): >+ * dfg/DFGGenerationInfo.h: >+ (JSC::DFG::GenerationInfo::recovery): >+ * dfg/DFGSpeculativeJIT32_64.cpp: >+ (JSC::DFG::SpeculativeJIT::emitCall): >+ * dfg/DFGSpeculativeJIT64.cpp: >+ (JSC::DFG::SpeculativeJIT::emitCall): >+ * jit/CallFrameShuffleData.cpp: Added. >+ (JSC::CallFrameShuffleData::setupCalleeSaveRegisters): >+ * jit/CallFrameShuffleData.h: Added. >+ * jit/CallFrameShuffler.cpp: Added. >+ (JSC::CallFrameShuffler::CallFrameShuffler): >+ (JSC::CallFrameShuffler::dump): >+ (JSC::CallFrameShuffler::getCachedRecovery): >+ (JSC::CallFrameShuffler::setCachedRecovery): >+ (JSC::CallFrameShuffler::spill): >+ (JSC::CallFrameShuffler::emitDeltaCheck): >+ (JSC::CallFrameShuffler::prepareForSlowPath): >+ (JSC::CallFrameShuffler::prepareForTailCall): >+ (JSC::CallFrameShuffler::tryWrites): >+ (JSC::CallFrameShuffler::performSafeWrites): >+ (JSC::CallFrameShuffler::prepareAny): >+ * jit/CallFrameShuffler.h: Added. >+ (JSC::CachedRecovery::CachedRecovery): >+ (JSC::CachedRecovery::targets): >+ (JSC::CachedRecovery::addTarget): >+ (JSC::CachedRecovery::removeTarget): >+ (JSC::CachedRecovery::clearTargets): >+ (JSC::CachedRecovery::setWantedJSValueRegs): >+ (JSC::CachedRecovery::boxingRequiresGPR): >+ (JSC::CachedRecovery::boxingRequiresFPR): >+ (JSC::CachedRecovery::recovery): >+ (JSC::CachedRecovery::setRecovery): >+ (JSC::CachedRecovery::wantedJSValueRegs): >+ (JSC::CallFrameShuffler::lockGPR): >+ (JSC::CallFrameShuffler::acquireGPR): >+ (JSC::CallFrameShuffler::releaseGPR): >+ (JSC::CallFrameShuffler::snapshot): >+ (JSC::CallFrameShuffler::setCalleeJSValueRegs): >+ (JSC::CallFrameShuffler::assumeCalleeIsCell): >+ (JSC::CallFrameShuffler::canBox): >+ (JSC::CallFrameShuffler::ensureBox): >+ (JSC::CallFrameShuffler::ensureLoad): >+ (JSC::CallFrameShuffler::canLoadAndBox): >+ (JSC::CallFrameShuffler::updateRecovery): >+ (JSC::CallFrameShuffler::clearCachedRecovery): >+ (JSC::CallFrameShuffler::addCachedRecovery): >+ (JSC::CallFrameShuffler::numLocals): >+ (JSC::CallFrameShuffler::getOld): >+ (JSC::CallFrameShuffler::setOld): >+ (JSC::CallFrameShuffler::firstOld): >+ (JSC::CallFrameShuffler::lastOld): >+ (JSC::CallFrameShuffler::isValidOld): >+ (JSC::CallFrameShuffler::argCount): >+ (JSC::CallFrameShuffler::getNew): >+ (JSC::CallFrameShuffler::setNew): >+ (JSC::CallFrameShuffler::addNew): >+ (JSC::CallFrameShuffler::firstNew): >+ (JSC::CallFrameShuffler::lastNew): >+ (JSC::CallFrameShuffler::isValidNew): >+ (JSC::CallFrameShuffler::newAsOld): >+ (JSC::CallFrameShuffler::getFreeRegister): >+ (JSC::CallFrameShuffler::getFreeGPR): >+ (JSC::CallFrameShuffler::getFreeFPR): >+ (JSC::CallFrameShuffler::hasFreeRegister): >+ (JSC::CallFrameShuffler::ensureRegister): >+ (JSC::CallFrameShuffler::ensureGPR): >+ (JSC::CallFrameShuffler::ensureFPR): >+ (JSC::CallFrameShuffler::addressForOld): >+ (JSC::CallFrameShuffler::isUndecided): >+ (JSC::CallFrameShuffler::isSlowPath): >+ (JSC::CallFrameShuffler::addressForNew): >+ (JSC::CallFrameShuffler::dangerFrontier): >+ (JSC::CallFrameShuffler::isDangerNew): >+ (JSC::CallFrameShuffler::updateDangerFrontier): >+ (JSC::CallFrameShuffler::hasOnlySafeWrites): >+ * jit/CallFrameShuffler32_64.cpp: Added. >+ (JSC::CallFrameShuffler::emitStore): >+ (JSC::CallFrameShuffler::emitBox): >+ (JSC::CallFrameShuffler::emitLoad): >+ (JSC::CallFrameShuffler::canLoad): >+ (JSC::CachedRecovery::loadsIntoFPR): >+ (JSC::CachedRecovery::loadsIntoGPR): >+ (JSC::CallFrameShuffler::emitDisplace): >+ * jit/CallFrameShuffler64.cpp: Added. >+ (JSC::CallFrameShuffler::emitStore): >+ (JSC::CallFrameShuffler::emitBox): >+ (JSC::CallFrameShuffler::emitLoad): >+ (JSC::CallFrameShuffler::canLoad): >+ (JSC::CachedRecovery::loadsIntoFPR): >+ (JSC::CachedRecovery::loadsIntoGPR): >+ (JSC::CallFrameShuffler::emitDisplace): >+ * jit/JITCall.cpp: >+ (JSC::JIT::compileOpCall): >+ (JSC::JIT::compileOpCallSlowCase): >+ * jit/RegisterMap.h: >+ * jit/Repatch.cpp: >+ (JSC::linkPolymorphicCall): >+ > 2015-08-24 Michael Saboff <msaboff@apple.com> > > jsc-tailcall: Specialized thunks need to save / restore callee save "tag" registers >diff --git a/Source/JavaScriptCore/CMakeLists.txt b/Source/JavaScriptCore/CMakeLists.txt >index 25c178bb8509aea0936e3cbab370220338b5bf95..e2c68ac8aa6dc28a03fb2b6057dae3507d4cb908 100644 >--- a/Source/JavaScriptCore/CMakeLists.txt >+++ b/Source/JavaScriptCore/CMakeLists.txt >@@ -343,6 +343,10 @@ set(JavaScriptCore_SOURCES > jit/AccessorCallJITStubRoutine.cpp > jit/AssemblyHelpers.cpp > jit/BinarySwitch.cpp >+ jit/CallFrameShuffleData.cpp >+ jit/CallFrameShuffler.cpp >+ jit/CallFrameShuffler32_64.cpp >+ jit/CallFrameShuffler64.cpp > jit/ExecutableAllocationFuzz.cpp > jit/ExecutableAllocator.cpp > jit/ExecutableAllocatorFixedVMPool.cpp >diff --git a/Source/JavaScriptCore/JavaScriptCore.vcxproj/JavaScriptCore.vcxproj b/Source/JavaScriptCore/JavaScriptCore.vcxproj/JavaScriptCore.vcxproj >index a86b5d5d510b963896c9e4697252fe909d27e55b..2074e471fd14d048f5798217102c8444faa6b30b 100644 >--- a/Source/JavaScriptCore/JavaScriptCore.vcxproj/JavaScriptCore.vcxproj >+++ b/Source/JavaScriptCore/JavaScriptCore.vcxproj/JavaScriptCore.vcxproj >@@ -606,6 +606,10 @@ > <ClCompile Include="..\jit\AccessorCallJITStubRoutine.cpp" /> > <ClCompile Include="..\jit\AssemblyHelpers.cpp" /> > <ClCompile Include="..\jit\BinarySwitch.cpp" /> >+ <ClCompile Include="..\jit\CallFrameShuffleData.cpp" /> >+ <ClCompile Include="..\jit\CallFrameShuffler.cpp" /> >+ <ClCompile Include="..\jit\CallFrameShuffler32_64.cpp" /> >+ <ClCompile Include="..\jit\CallFrameShuffler64.cpp" /> > <ClCompile Include="..\jit\ExecutableAllocationFuzz.cpp" /> > <ClCompile Include="..\jit\ExecutableAllocator.cpp" /> > <ClCompile Include="..\jit\ExecutableAllocatorFixedVMPool.cpp" /> >@@ -1370,6 +1374,8 @@ > <ClInclude Include="..\jit\AccessorCallJITStubRoutine.h" /> > <ClInclude Include="..\jit\AssemblyHelpers.h" /> > <ClInclude Include="..\jit\BinarySwitch.h" /> >+ <ClInclude Include="..\jit\CallFrameShuffleData.h" /> >+ <ClInclude Include="..\jit\CallFrameShuffler.h" /> > <ClInclude Include="..\jit\CCallHelpers.h" /> > <ClInclude Include="..\jit\CompactJITCodeMap.h" /> > <ClInclude Include="..\jit\ExecutableAllocationFuzz.h" /> >diff --git a/Source/JavaScriptCore/JavaScriptCore.vcxproj/JavaScriptCore.vcxproj.filters b/Source/JavaScriptCore/JavaScriptCore.vcxproj/JavaScriptCore.vcxproj.filters >index 9fbcb5bd6cc133b47fc032da7701436f6ef8085e..98e27503ab95acc522e56e72fba0200849280659 100644 >--- a/Source/JavaScriptCore/JavaScriptCore.vcxproj/JavaScriptCore.vcxproj.filters >+++ b/Source/JavaScriptCore/JavaScriptCore.vcxproj/JavaScriptCore.vcxproj.filters >@@ -378,6 +378,18 @@ > <ClCompile Include="..\jit\HostCallReturnValue.cpp"> > <Filter>jit</Filter> > </ClCompile> >+ <ClCompile Include="..\jit\CallFrameShuffleData.cpp"> >+ <Filter>jit</Filter> >+ </ClCompile> >+ <ClCompile Include="..\jit\CallFrameShuffler.cpp"> >+ <Filter>jit</Filter> >+ </ClCompile> >+ <ClCompile Include="..\jit\CallFrameShuffler32_64.cpp"> >+ <Filter>jit</Filter> >+ </ClCompile> >+ <ClCompile Include="..\jit\CallFrameShuffler64.cpp"> >+ <Filter>jit</Filter> >+ </ClCompile> > <ClCompile Include="..\jit\JIT.cpp"> > <Filter>jit</Filter> > </ClCompile> >diff --git a/Source/JavaScriptCore/JavaScriptCore.xcodeproj/project.pbxproj b/Source/JavaScriptCore/JavaScriptCore.xcodeproj/project.pbxproj >index 169311a611dd6735be9f0644da4958c11066b94f..535e46f6a2e9c076dc34f613941da576a83e77ca 100644 >--- a/Source/JavaScriptCore/JavaScriptCore.xcodeproj/project.pbxproj >+++ b/Source/JavaScriptCore/JavaScriptCore.xcodeproj/project.pbxproj >@@ -965,6 +965,12 @@ > 627673241B680C1E00FD9F2E /* CallMode.h in Headers */ = {isa = PBXBuildFile; fileRef = 627673221B680C1E00FD9F2E /* CallMode.h */; settings = {ATTRIBUTES = (Private, ); }; }; > 62D2D38F1ADF103F000206C1 /* FunctionRareData.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 62D2D38D1ADF103F000206C1 /* FunctionRareData.cpp */; }; > 62D2D3901ADF103F000206C1 /* FunctionRareData.h in Headers */ = {isa = PBXBuildFile; fileRef = 62D2D38E1ADF103F000206C1 /* FunctionRareData.h */; settings = {ATTRIBUTES = (Private, ); }; }; >+ 62D755D41B84FB3D001801FA /* CallFrameShuffler64.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 62D755D31B84FB39001801FA /* CallFrameShuffler64.cpp */; }; >+ 62D755D51B84FB40001801FA /* CallFrameShuffler32_64.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 62D755D21B84FB39001801FA /* CallFrameShuffler32_64.cpp */; }; >+ 62D755D61B84FB46001801FA /* CallFrameShuffler.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 62D755D01B84FB39001801FA /* CallFrameShuffler.cpp */; }; >+ 62D755D71B84FB4A001801FA /* CallFrameShuffler.h in Headers */ = {isa = PBXBuildFile; fileRef = 62D755D11B84FB39001801FA /* CallFrameShuffler.h */; }; >+ 62EC9BB61B7EB07C00303AD1 /* CallFrameShuffleData.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 62EC9BB41B7EB07C00303AD1 /* CallFrameShuffleData.cpp */; }; >+ 62EC9BB71B7EB07C00303AD1 /* CallFrameShuffleData.h in Headers */ = {isa = PBXBuildFile; fileRef = 62EC9BB51B7EB07C00303AD1 /* CallFrameShuffleData.h */; settings = {ATTRIBUTES = (Private, ); }; }; > 62F2AA371B0BEDE300610C7A /* DFGLazyNode.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 62A9A29E1B0BED4800BD54CA /* DFGLazyNode.cpp */; }; > 62F2AA381B0BEDE300610C7A /* DFGLazyNode.h in Headers */ = {isa = PBXBuildFile; fileRef = 62A9A29F1B0BED4800BD54CA /* DFGLazyNode.h */; settings = {ATTRIBUTES = (Private, ); }; }; > 6507D29E0E871E5E00D7D896 /* JSTypeInfo.h in Headers */ = {isa = PBXBuildFile; fileRef = 6507D2970E871E4A00D7D896 /* JSTypeInfo.h */; settings = {ATTRIBUTES = (Private, ); }; }; >@@ -2684,6 +2690,12 @@ > 62A9A29F1B0BED4800BD54CA /* DFGLazyNode.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; name = DFGLazyNode.h; path = dfg/DFGLazyNode.h; sourceTree = "<group>"; }; > 62D2D38D1ADF103F000206C1 /* FunctionRareData.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = FunctionRareData.cpp; sourceTree = "<group>"; }; > 62D2D38E1ADF103F000206C1 /* FunctionRareData.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = FunctionRareData.h; sourceTree = "<group>"; }; >+ 62D755D01B84FB39001801FA /* CallFrameShuffler.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = CallFrameShuffler.cpp; sourceTree = "<group>"; }; >+ 62D755D11B84FB39001801FA /* CallFrameShuffler.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = CallFrameShuffler.h; sourceTree = "<group>"; }; >+ 62D755D21B84FB39001801FA /* CallFrameShuffler32_64.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = CallFrameShuffler32_64.cpp; sourceTree = "<group>"; }; >+ 62D755D31B84FB39001801FA /* CallFrameShuffler64.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = CallFrameShuffler64.cpp; sourceTree = "<group>"; }; >+ 62EC9BB41B7EB07C00303AD1 /* CallFrameShuffleData.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = CallFrameShuffleData.cpp; sourceTree = "<group>"; }; >+ 62EC9BB51B7EB07C00303AD1 /* CallFrameShuffleData.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = CallFrameShuffleData.h; sourceTree = "<group>"; }; > 6507D2970E871E4A00D7D896 /* JSTypeInfo.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = JSTypeInfo.h; sourceTree = "<group>"; }; > 651122E5140469BA002B101D /* testRegExp.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = testRegExp.cpp; sourceTree = "<group>"; }; > 6511230514046A4C002B101D /* testRegExp */ = {isa = PBXFileReference; explicitFileType = "compiled.mach-o.executable"; includeInIndex = 0; path = testRegExp; sourceTree = BUILT_PRODUCTS_DIR; }; >@@ -3903,16 +3915,22 @@ > 1429D92C0ED22D7000B89619 /* jit */ = { > isa = PBXGroup; > children = ( >- 0FF054F71AC35B4400E5BE57 /* ExecutableAllocationFuzz.cpp */, >- 0FF054F81AC35B4400E5BE57 /* ExecutableAllocationFuzz.h */, > 0F7576D018E1FEE9002EF4CD /* AccessorCallJITStubRoutine.cpp */, > 0F7576D118E1FEE9002EF4CD /* AccessorCallJITStubRoutine.h */, > 0F24E53B17EA9F5900ABB217 /* AssemblyHelpers.cpp */, > 0F24E53C17EA9F5900ABB217 /* AssemblyHelpers.h */, > 0F64B26F1A784BAF006E4E66 /* BinarySwitch.cpp */, > 0F64B2701A784BAF006E4E66 /* BinarySwitch.h */, >+ 62EC9BB41B7EB07C00303AD1 /* CallFrameShuffleData.cpp */, >+ 62EC9BB51B7EB07C00303AD1 /* CallFrameShuffleData.h */, >+ 62D755D01B84FB39001801FA /* CallFrameShuffler.cpp */, >+ 62D755D11B84FB39001801FA /* CallFrameShuffler.h */, >+ 62D755D21B84FB39001801FA /* CallFrameShuffler32_64.cpp */, >+ 62D755D31B84FB39001801FA /* CallFrameShuffler64.cpp */, > 0F24E53D17EA9F5900ABB217 /* CCallHelpers.h */, > 0FD82E37141AB14200179C94 /* CompactJITCodeMap.h */, >+ 0FF054F71AC35B4400E5BE57 /* ExecutableAllocationFuzz.cpp */, >+ 0FF054F81AC35B4400E5BE57 /* ExecutableAllocationFuzz.h */, > A7B48DB60EE74CFC00DCBDB6 /* ExecutableAllocator.cpp */, > A7B48DB50EE74CFC00DCBDB6 /* ExecutableAllocator.h */, > 86DB64630F95C6FC00D7D921 /* ExecutableAllocatorFixedVMPool.cpp */, >@@ -5657,6 +5675,7 @@ > C4F4B6F41A05C944005CAB76 /* cpp_generator.py in Headers */, > A532439418569709002ED692 /* generate-combined-inspector-json.py in Headers */, > A5840E27187C981E00843B10 /* cssmin.py in Headers */, >+ 62D755D71B84FB4A001801FA /* CallFrameShuffler.h in Headers */, > A5EA710419F6DE720098F5EC /* generate_objc_backend_dispatcher_implementation.py in Headers */, > C4703CD5192844CC0013FBEA /* generator_templates.py in Headers */, > A5EA710519F6DE740098F5EC /* generate_objc_configuration_header.py in Headers */, >@@ -5812,6 +5831,7 @@ > 0F3B3A281544C997003ED0FF /* DFGCFGSimplificationPhase.h in Headers */, > A77A424017A0BBFD00A8DB81 /* DFGClobberize.h in Headers */, > A77A424217A0BBFD00A8DB81 /* DFGClobberSet.h in Headers */, >+ 62EC9BB71B7EB07C00303AD1 /* CallFrameShuffleData.h in Headers */, > 0F7B294D14C3CD4C007C3DB1 /* DFGCommon.h in Headers */, > 0FEA0A32170D40BF00BB722C /* DFGCommonData.h in Headers */, > 0F38B01817CFE75500B144D3 /* DFGCompilationKey.h in Headers */, >@@ -7108,6 +7128,7 @@ > 0FF0F19D16B72A08005DF95B /* DFGCommon.cpp in Sources */, > 0FEA0A31170D40BF00BB722C /* DFGCommonData.cpp in Sources */, > 0F38B01717CFE75500B144D3 /* DFGCompilationKey.cpp in Sources */, >+ 62EC9BB61B7EB07C00303AD1 /* CallFrameShuffleData.cpp in Sources */, > 0F38B01917CFE75500B144D3 /* DFGCompilationMode.cpp in Sources */, > 0F3B3A1A153E68F2003ED0FF /* DFGConstantFoldingPhase.cpp in Sources */, > 0FBE0F7216C1DB030082C5E8 /* DFGCPSRethreadingPhase.cpp in Sources */, >@@ -7157,6 +7178,7 @@ > 0FF0F19C16B72A03005DF95B /* DFGNode.cpp in Sources */, > 0FA581BA150E952C00B9A2D9 /* DFGNodeFlags.cpp in Sources */, > 86EC9DCF1328DF82002B2AD7 /* DFGOperations.cpp in Sources */, >+ 62D755D61B84FB46001801FA /* CallFrameShuffler.cpp in Sources */, > A7D89CFD17A0B8CC00773AD8 /* DFGOSRAvailabilityAnalysisPhase.cpp in Sources */, > 0FD82E56141DAF0800179C94 /* DFGOSREntry.cpp in Sources */, > 0FD8A32517D51F5700CA2C40 /* DFGOSREntrypointCreationPhase.cpp in Sources */, >@@ -7279,6 +7301,7 @@ > 147F39CB107EC37600427A48 /* FunctionConstructor.cpp in Sources */, > 0FF0F19F16B72A17005DF95B /* FunctionExecutableDump.cpp in Sources */, > 52B310FD1974AE870080857C /* FunctionHasExecutedCache.cpp in Sources */, >+ 62D755D51B84FB40001801FA /* CallFrameShuffler32_64.cpp in Sources */, > 147F39CC107EC37600427A48 /* FunctionPrototype.cpp in Sources */, > 0F64B2711A784BAF006E4E66 /* BinarySwitch.cpp in Sources */, > 0F766D2F15A8DCE0008F363E /* GCAwareJITStubRoutine.cpp in Sources */, >@@ -7400,6 +7423,7 @@ > A72700900DAC6BBC00E548D7 /* JSNotAnObject.cpp in Sources */, > 147F39D4107EC37600427A48 /* JSObject.cpp in Sources */, > 1482B7E40A43076000517CFC /* JSObjectRef.cpp in Sources */, >+ 62D755D41B84FB3D001801FA /* CallFrameShuffler64.cpp in Sources */, > A7F993600FD7325100A0B2D0 /* JSONObject.cpp in Sources */, > 0FFB6C381AF48DDC00DB1BF7 /* TypeofType.cpp in Sources */, > 95F6E6950E5B5F970091E860 /* JSProfilerPrivate.cpp in Sources */, >diff --git a/Source/JavaScriptCore/assembler/AbortReason.h b/Source/JavaScriptCore/assembler/AbortReason.h >index e94449ad4c49553ae670ba6f7576f91adc9145ed..eae540f8361296510d33490de680f404ac297dd1 100644 >--- a/Source/JavaScriptCore/assembler/AbortReason.h >+++ b/Source/JavaScriptCore/assembler/AbortReason.h >@@ -63,6 +63,7 @@ enum AbortReason { > JITGetByValResultIsNotEmpty = 250, > JITNotSupported = 260, > JITOffsetIsNotOutOfLine = 270, >+ JITUnexpectedCallFrameSize = 275, > JITUnreasonableLoopHintJumpTarget = 280, > RPWUnreasonableJumpTarget = 290, > RepatchIneffectiveWatchpoint = 300, >diff --git a/Source/JavaScriptCore/bytecode/CallLinkInfo.h b/Source/JavaScriptCore/bytecode/CallLinkInfo.h >index 6212425e169e8a81029d30b9b59af8ef4f3d42d6..e3e7395fb1fa3d40b864ef3aa1588fe6eb60a269 100644 >--- a/Source/JavaScriptCore/bytecode/CallLinkInfo.h >+++ b/Source/JavaScriptCore/bytecode/CallLinkInfo.h >@@ -26,6 +26,7 @@ > #ifndef CallLinkInfo_h > #define CallLinkInfo_h > >+#include "CallFrameShuffleData.h" > #include "CallMode.h" > #include "CodeLocation.h" > #include "CodeSpecializationKind.h" >@@ -329,6 +330,16 @@ public: > > void visitWeak(RepatchBuffer&); > >+ void setFrameShuffleData(const CallFrameShuffleData& shuffleData) >+ { >+ m_frameShuffleData = std::make_unique<CallFrameShuffleData>(shuffleData); >+ } >+ >+ const CallFrameShuffleData* frameShuffleData() >+ { >+ return m_frameShuffleData.get(); >+ } >+ > private: > CodeLocationNearCall m_callReturnLocation; > CodeLocationDataLabelPtr m_hotPathBegin; >@@ -337,6 +348,7 @@ private: > WriteBarrier<JSFunction> m_lastSeenCallee; > RefPtr<PolymorphicCallStubRoutine> m_stub; > RefPtr<JITStubRoutine> m_slowStub; >+ std::unique_ptr<CallFrameShuffleData> m_frameShuffleData; > unsigned m_registerPreservationMode : 1; // Real type is RegisterPreservationMode > bool m_hasSeenShouldRepatch : 1; > bool m_hasSeenClosure : 1; >diff --git a/Source/JavaScriptCore/dfg/DFGGenerationInfo.h b/Source/JavaScriptCore/dfg/DFGGenerationInfo.h >index b30f3508d72dea7ae49e7b9a82a254b232d820ad..800f97b803a73458c2469084b8cb495ed340009e 100644 >--- a/Source/JavaScriptCore/dfg/DFGGenerationInfo.h >+++ b/Source/JavaScriptCore/dfg/DFGGenerationInfo.h >@@ -378,6 +378,29 @@ public: > return m_useCount; > } > >+ ValueRecovery recovery(VirtualRegister spillSlot) const >+ { >+ if (m_isConstant) >+ return ValueRecovery::constant(m_node->constant()->value()); >+ >+ if (m_registerFormat == DataFormatDouble) >+ return ValueRecovery::inFPR(u.fpr, DataFormatDouble); >+ >+#if USE(JSVALUE32_64) >+ if (m_registerFormat & DataFormatJS) { >+ if (m_registerFormat == DataFormatJS) >+ return ValueRecovery::inPair(u.v.tagGPR, u.v.payloadGPR); >+ return ValueRecovery::inGPR(u.v.payloadGPR, static_cast<DataFormat>(m_registerFormat & ~DataFormatJS)); >+ } >+#endif >+ if (m_registerFormat) >+ return ValueRecovery::inGPR(u.gpr, m_registerFormat); >+ >+ ASSERT(m_spillFormat); >+ >+ return ValueRecovery::displacedInJSStack(spillSlot, m_spillFormat); >+ } >+ > private: > void appendBirth(VariableEventStream& stream) > { >diff --git a/Source/JavaScriptCore/dfg/DFGSpeculativeJIT32_64.cpp b/Source/JavaScriptCore/dfg/DFGSpeculativeJIT32_64.cpp >index 7d29dad5f7c6f8119758b54dee1ef6da443819c1..7cdc1a3e188bbbff3e23308323fce52db9161d3c 100644 >--- a/Source/JavaScriptCore/dfg/DFGSpeculativeJIT32_64.cpp >+++ b/Source/JavaScriptCore/dfg/DFGSpeculativeJIT32_64.cpp >@@ -30,6 +30,7 @@ > #if ENABLE(DFG_JIT) > > #include "ArrayPrototype.h" >+#include "CallFrameShuffler.h" > #include "DFGAbstractInterpreterInlines.h" > #include "DFGCallArrayAllocatorSlowPathGenerator.h" > #include "DFGOperations.h" >@@ -703,6 +704,9 @@ void SpeculativeJIT::emitCall(Node* node) > } > > Edge calleeEdge = m_jit.graph().child(node, 0); >+ GPRReg calleeTagGPR; >+ GPRReg calleePayloadGPR; >+ CallFrameShuffleData shuffleData; > > // Gotta load the arguments somehow. Varargs is trickier. > if (isVarargs || isForwardVarargs) { >@@ -794,33 +798,48 @@ void SpeculativeJIT::emitCall(Node* node) > // receiver (method call). subsequent children are the arguments. > int numPassedArgs = node->numChildren() - 1; > >- m_jit.store32(MacroAssembler::TrustedImm32(numPassedArgs), m_jit.calleeFramePayloadSlot(JSStack::ArgumentCount)); >+ if (node->op() == TailCall) { >+ JSValueOperand callee(this, calleeEdge); >+ calleeTagGPR = callee.tagGPR(); >+ calleePayloadGPR = callee.payloadGPR(); >+ use(calleeEdge); >+ >+ shuffleData.numLocals = m_jit.graph().frameRegisterCount(); >+ shuffleData.callee = ValueRecovery::inPair(calleeTagGPR, calleePayloadGPR); >+ shuffleData.args.resize(numPassedArgs); >+ >+ for (int i = 0; i < numPassedArgs; ++i) { >+ Edge argEdge = m_jit.graph().varArgChild(node, i + 1); >+ GenerationInfo& info = generationInfo(argEdge.node()); >+ use(argEdge); >+ shuffleData.args[i] = info.recovery(argEdge->virtualRegister()); >+ } >+ } else { >+ m_jit.store32(MacroAssembler::TrustedImm32(numPassedArgs), m_jit.calleeFramePayloadSlot(JSStack::ArgumentCount)); > >- for (int i = 0; i < numPassedArgs; i++) { >- Edge argEdge = m_jit.graph().m_varArgChildren[node->firstChild() + 1 + i]; >- JSValueOperand arg(this, argEdge); >- GPRReg argTagGPR = arg.tagGPR(); >- GPRReg argPayloadGPR = arg.payloadGPR(); >- use(argEdge); >+ for (int i = 0; i < numPassedArgs; i++) { >+ Edge argEdge = m_jit.graph().m_varArgChildren[node->firstChild() + 1 + i]; >+ JSValueOperand arg(this, argEdge); >+ GPRReg argTagGPR = arg.tagGPR(); >+ GPRReg argPayloadGPR = arg.payloadGPR(); >+ use(argEdge); > >- m_jit.store32(argTagGPR, m_jit.calleeArgumentTagSlot(i)); >- m_jit.store32(argPayloadGPR, m_jit.calleeArgumentPayloadSlot(i)); >+ m_jit.store32(argTagGPR, m_jit.calleeArgumentTagSlot(i)); >+ m_jit.store32(argPayloadGPR, m_jit.calleeArgumentPayloadSlot(i)); >+ } > } > } > >- JSValueOperand callee(this, calleeEdge); >- GPRReg calleeTagGPR = callee.tagGPR(); >- GPRReg calleePayloadGPR = callee.payloadGPR(); >- use(calleeEdge); >- m_jit.store32(calleePayloadGPR, m_jit.calleeFramePayloadSlot(JSStack::Callee)); >- m_jit.store32(calleeTagGPR, m_jit.calleeFrameTagSlot(JSStack::Callee)); >+ if (node->op() != TailCall) { >+ JSValueOperand callee(this, calleeEdge); >+ calleeTagGPR = callee.tagGPR(); >+ calleePayloadGPR = callee.payloadGPR(); >+ use(calleeEdge); >+ m_jit.store32(calleePayloadGPR, m_jit.calleeFramePayloadSlot(JSStack::Callee)); >+ m_jit.store32(calleeTagGPR, m_jit.calleeFrameTagSlot(JSStack::Callee)); > >- // FIXME: We should do an efficient move of the arguments into >- // their target stack position instead of building then memmoving >- // the callee frame. >- // https://bugs.webkit.org/show_bug.cgi?id=147508 >- if (!isTail) > flushRegisters(); >+ } > > GPRFlushedCallResult resultPayload(this); > GPRFlushedCallResult2 resultTag(this); >@@ -840,16 +859,18 @@ void SpeculativeJIT::emitCall(Node* node) > > CallLinkInfo* info = m_jit.codeBlock()->addCallLinkInfo(); > >- slowPath.append(m_jit.branchIfNotCell(callee.jsValueRegs())); >+ if (isTail && node->op() != TailCall) >+ m_jit.emitRestoreCalleeSaves(); >+ >+ slowPath.append(m_jit.branchIfNotCell(JSValueRegs(calleeTagGPR, calleePayloadGPR))); > slowPath.append(m_jit.branchPtrWithPatch(MacroAssembler::NotEqual, calleePayloadGPR, targetToCheck)); > > if (isTail) { >- m_jit.emitRestoreCalleeSaves(); >- // FIXME: We should do an efficient move of the arguments into >- // their target stack position instead of building then memmoving >- // the callee frame. >- // https://bugs.webkit.org/show_bug.cgi?id=147508 >- m_jit.prepareForTailCallSlow(); >+ if (node->op() == TailCall) { >+ info->setFrameShuffleData(shuffleData); >+ CallFrameShuffler(m_jit, shuffleData).prepareForTailCall(); >+ } else >+ m_jit.prepareForTailCallSlow(); > } > > JITCompiler::Call fastCall = isTail ? m_jit.nearTailCall() : m_jit.nearCall(); >@@ -858,17 +879,24 @@ void SpeculativeJIT::emitCall(Node* node) > > slowPath.link(&m_jit); > >- // Callee payload needs to be in regT0, tag in regT1 >- if (calleeTagGPR == GPRInfo::regT0) { >- if (calleePayloadGPR == GPRInfo::regT1) >- m_jit.swap(GPRInfo::regT1, GPRInfo::regT0); >- else { >- m_jit.move(calleeTagGPR, GPRInfo::regT1); >+ if (node->op() == TailCall) { >+ CallFrameShuffler callFrameShuffler(m_jit, shuffleData); >+ callFrameShuffler.setCalleeJSValueRegs(JSValueRegs( >+ GPRInfo::regT1, GPRInfo::regT0)); >+ callFrameShuffler.prepareForSlowPath(); >+ } else { >+ // Callee payload needs to be in regT0, tag in regT1 >+ if (calleeTagGPR == GPRInfo::regT0) { >+ if (calleePayloadGPR == GPRInfo::regT1) >+ m_jit.swap(GPRInfo::regT1, GPRInfo::regT0); >+ else { >+ m_jit.move(calleeTagGPR, GPRInfo::regT1); >+ m_jit.move(calleePayloadGPR, GPRInfo::regT0); >+ } >+ } else { > m_jit.move(calleePayloadGPR, GPRInfo::regT0); >+ m_jit.move(calleeTagGPR, GPRInfo::regT1); > } >- } else { >- m_jit.move(calleePayloadGPR, GPRInfo::regT0); >- m_jit.move(calleeTagGPR, GPRInfo::regT1); > } > m_jit.move(MacroAssembler::TrustedImmPtr(info), GPRInfo::regT2); > JITCompiler::Call slowCall = m_jit.nearCall(); >diff --git a/Source/JavaScriptCore/dfg/DFGSpeculativeJIT64.cpp b/Source/JavaScriptCore/dfg/DFGSpeculativeJIT64.cpp >index b7c5aa4be6da8ddebf3ab61e382411f5f63bd3be..45817340dfaf110eff62c90cde185b84ff927328 100644 >--- a/Source/JavaScriptCore/dfg/DFGSpeculativeJIT64.cpp >+++ b/Source/JavaScriptCore/dfg/DFGSpeculativeJIT64.cpp >@@ -29,6 +29,7 @@ > #if ENABLE(DFG_JIT) > > #include "ArrayPrototype.h" >+#include "CallFrameShuffler.h" > #include "DFGAbstractInterpreterInlines.h" > #include "DFGCallArrayAllocatorSlowPathGenerator.h" > #include "DFGOperations.h" >@@ -688,7 +689,8 @@ void SpeculativeJIT::emitCall(Node* node) > break; > } > >- Edge calleeEdge = m_jit.graph().child(node, 0); >+ GPRReg calleeGPR; >+ CallFrameShuffleData shuffleData; > > // Gotta load the arguments somehow. Varargs is trickier. > if (isVarargs || isForwardVarargs) { >@@ -775,32 +777,47 @@ void SpeculativeJIT::emitCall(Node* node) > // arguments. > int numPassedArgs = node->numChildren() - 1; > >- m_jit.store32(MacroAssembler::TrustedImm32(numPassedArgs), JITCompiler::calleeFramePayloadSlot(JSStack::ArgumentCount)); >- >- for (int i = 0; i < numPassedArgs; i++) { >- Edge argEdge = m_jit.graph().m_varArgChildren[node->firstChild() + 1 + i]; >- JSValueOperand arg(this, argEdge); >- GPRReg argGPR = arg.gpr(); >- use(argEdge); >- >- m_jit.store64(argGPR, JITCompiler::calleeArgumentSlot(i)); >+ if (node->op() == TailCall) { >+ Edge calleeEdge = m_jit.graph().child(node, 0); >+ JSValueOperand callee(this, calleeEdge); >+ calleeGPR = callee.gpr(); >+ callee.use(); >+ >+ shuffleData.numLocals = m_jit.graph().frameRegisterCount(); >+ shuffleData.callee = ValueRecovery::inGPR(calleeGPR, DataFormatJS); >+ shuffleData.args.resize(numPassedArgs); >+ >+ for (int i = 0; i < numPassedArgs; ++i) { >+ Edge argEdge = m_jit.graph().varArgChild(node, i + 1); >+ GenerationInfo& info = generationInfo(argEdge.node()); >+ use(argEdge); >+ shuffleData.args[i] = info.recovery(argEdge->virtualRegister()); >+ } >+ >+ shuffleData.setupCalleeSaveRegisters(m_jit.codeBlock()); >+ } else { >+ m_jit.store32(MacroAssembler::TrustedImm32(numPassedArgs), JITCompiler::calleeFramePayloadSlot(JSStack::ArgumentCount)); >+ >+ for (int i = 0; i < numPassedArgs; i++) { >+ Edge argEdge = m_jit.graph().m_varArgChildren[node->firstChild() + 1 + i]; >+ JSValueOperand arg(this, argEdge); >+ GPRReg argGPR = arg.gpr(); >+ use(argEdge); >+ >+ m_jit.store64(argGPR, JITCompiler::calleeArgumentSlot(i)); >+ } > } > } > >- JSValueOperand callee(this, calleeEdge); >- GPRReg calleeGPR = callee.gpr(); >- callee.use(); >- m_jit.store64(calleeGPR, JITCompiler::calleeFrameSlot(JSStack::Callee)); >+ if (node->op() != TailCall) { >+ Edge calleeEdge = m_jit.graph().child(node, 0); >+ JSValueOperand callee(this, calleeEdge); >+ calleeGPR = callee.gpr(); >+ callee.use(); >+ m_jit.store64(calleeGPR, JITCompiler::calleeFrameSlot(JSStack::Callee)); > >- // FIXME: We should do an efficient move of the arguments into >- // their target stack position instead of building then memmoving >- // the callee frame. >- // https://bugs.webkit.org/show_bug.cgi?id=147508 >- if (!isTail) > flushRegisters(); >- >- GPRFlushedCallResult result(this); >- GPRReg resultGPR = result.gpr(); >+ } > > CodeOrigin staticOrigin = node->origin.semantic; > ASSERT(!isTail || !staticOrigin.inlineCallFrame || !staticOrigin.inlineCallFrame->getCallerSkippingDeadFrames()); >@@ -812,16 +829,18 @@ void SpeculativeJIT::emitCall(Node* node) > > CallLinkInfo* callLinkInfo = m_jit.codeBlock()->addCallLinkInfo(); > >+ if (isTail && node->op() != TailCall) >+ m_jit.emitRestoreCalleeSaves(); >+ > JITCompiler::DataLabelPtr targetToCheck; > JITCompiler::Jump slowPath = m_jit.branchPtrWithPatch(MacroAssembler::NotEqual, calleeGPR, targetToCheck, MacroAssembler::TrustedImmPtr(0)); > > if (isTail) { >- m_jit.emitRestoreCalleeSaves(); >- // FIXME: We should do an efficient move of the arguments into >- // their target stack position instead of building then memmoving >- // the callee frame. >- // https://bugs.webkit.org/show_bug.cgi?id=147508 >- m_jit.prepareForTailCallSlow(); >+ if (node->op() == TailCall) { >+ callLinkInfo->setFrameShuffleData(shuffleData); >+ CallFrameShuffler(m_jit, shuffleData).prepareForTailCall(); >+ } else >+ m_jit.prepareForTailCallSlow(); > } > > JITCompiler::Call fastCall = isTail ? m_jit.nearTailCall() : m_jit.nearCall(); >@@ -829,11 +848,14 @@ void SpeculativeJIT::emitCall(Node* node) > JITCompiler::Jump done = m_jit.jump(); > > slowPath.link(&m_jit); >- >- if (isTail) >- m_jit.emitRestoreCalleeSaves(); > >- m_jit.move(calleeGPR, GPRInfo::regT0); // Callee needs to be in regT0 >+ if (node->op() == TailCall) { >+ CallFrameShuffler callFrameShuffler(m_jit, shuffleData); >+ callFrameShuffler.setCalleeJSValueRegs(JSValueRegs(GPRInfo::regT0)); >+ callFrameShuffler.prepareForSlowPath(); >+ } else >+ m_jit.move(calleeGPR, GPRInfo::regT0); // Callee needs to be in regT0 >+ > m_jit.move(MacroAssembler::TrustedImmPtr(callLinkInfo), GPRInfo::regT2); // Link info needs to be in regT2 > JITCompiler::Call slowCall = m_jit.nearCall(); > >@@ -842,6 +864,8 @@ void SpeculativeJIT::emitCall(Node* node) > if (isTail) > m_jit.abortWithReason(JITDidReturnFromTailCall); > else { >+ GPRFlushedCallResult result(this); >+ GPRReg resultGPR = result.gpr(); > m_jit.move(GPRInfo::returnValueGPR, resultGPR); > > jsValueResult(resultGPR, m_currentNode, DataFormatJS, UseChildrenCalledExplicitly); >diff --git a/Source/JavaScriptCore/jit/CallFrameShuffleData.cpp b/Source/JavaScriptCore/jit/CallFrameShuffleData.cpp >new file mode 100644 >index 0000000000000000000000000000000000000000..d02285235f9a203fa970089c993912630039a37c >--- /dev/null >+++ b/Source/JavaScriptCore/jit/CallFrameShuffleData.cpp >@@ -0,0 +1,73 @@ >+/* >+ * Copyright (C) 2015 Apple Inc. All rights reserved. >+ * >+ * Redistribution and use in source and binary forms, with or without >+ * modification, are permitted provided that the following conditions >+ * are met: >+ * 1. Redistributions of source code must retain the above copyright >+ * notice, this list of conditions and the following disclaimer. >+ * 2. Redistributions in binary form must reproduce the above copyright >+ * notice, this list of conditions and the following disclaimer in the >+ * documentation and/or other materials provided with the distribution. >+ * >+ * THIS SOFTWARE IS PROVIDED BY APPLE INC. ``AS IS'' AND ANY >+ * EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE >+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR >+ * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL APPLE INC. OR >+ * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, >+ * EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, >+ * PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR >+ * PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY >+ * OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT >+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE >+ * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. >+ */ >+ >+#include "config.h" >+#include "CallFrameShuffleData.h" >+ >+#if ENABLE(JIT) >+ >+#include "CCallHelpers.h" >+#include "CodeBlock.h" >+ >+namespace JSC { >+ >+#if USE(JSVALUE64) >+ >+void CallFrameShuffleData::setupCalleeSaveRegisters(CodeBlock* codeBlock) >+{ >+ RegisterSet calleeSaveRegisters { RegisterSet::allVMCalleeSaveRegisters() }; >+ RegisterAtOffsetList* registerSaveLocations = codeBlock->calleeSaveRegisters(); >+ >+ for (size_t i = 0; i < registerSaveLocations->size(); ++i) { >+ RegisterAtOffset entry { registerSaveLocations->at(i) }; >+ if (!calleeSaveRegisters.get(entry.reg())) >+ continue; >+ >+ ASSERT(entry.reg().isGPR()); >+ >+ VirtualRegister saveSlot { entry.offsetAsIndex() }; >+ registers[entry.reg()] >+ = ValueRecovery::displacedInJSStack(saveSlot, DataFormatJS); >+ } >+ >+ for (Reg reg = Reg::first(); reg <= Reg::last(); reg = reg.next()) { >+ if (!calleeSaveRegisters.get(reg)) >+ continue; >+ >+ if (registers[reg]) >+ continue; >+ >+ ASSERT(reg.isGPR()); >+ >+ registers[reg] >+ = ValueRecovery::inGPR(reg.gpr(), DataFormatJS); >+ } >+} >+ >+#endif // USE(JSVALUE64) >+ >+} // namespace JSC >+ >+#endif // ENABLE(JIT) >diff --git a/Source/JavaScriptCore/jit/CallFrameShuffleData.h b/Source/JavaScriptCore/jit/CallFrameShuffleData.h >new file mode 100644 >index 0000000000000000000000000000000000000000..44802e86d14ae99ad309b4acff320e1364235047 >--- /dev/null >+++ b/Source/JavaScriptCore/jit/CallFrameShuffleData.h >@@ -0,0 +1,51 @@ >+/* >+ * Copyright (C) 2015 Apple Inc. All rights reserved. >+ * >+ * Redistribution and use in source and binary forms, with or without >+ * modification, are permitted provided that the following conditions >+ * are met: >+ * 1. Redistributions of source code must retain the above copyright >+ * notice, this list of conditions and the following disclaimer. >+ * 2. Redistributions in binary form must reproduce the above copyright >+ * notice, this list of conditions and the following disclaimer in the >+ * documentation and/or other materials provided with the distribution. >+ * >+ * THIS SOFTWARE IS PROVIDED BY APPLE INC. ``AS IS'' AND ANY >+ * EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE >+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR >+ * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL APPLE INC. OR >+ * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, >+ * EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, >+ * PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR >+ * PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY >+ * OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT >+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE >+ * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. >+ */ >+ >+#ifndef CallFrameShuffleData_h >+#define CallFrameShuffleData_h >+ >+#if ENABLE(JIT) >+ >+#include "RegisterMap.h" >+#include "ValueRecovery.h" >+ >+namespace JSC { >+ >+struct CallFrameShuffleData { >+ unsigned numLocals; >+ ValueRecovery callee; >+ Vector<ValueRecovery> args; >+#if USE(JSVALUE64) >+ RegisterMap<ValueRecovery> registers; >+ >+ void setupCalleeSaveRegisters(CodeBlock*); >+#endif >+}; >+ >+} // namespace JSC >+ >+#endif // ENABLE(JIT) >+ >+#endif // CallFrameShuffleData_h >diff --git a/Source/JavaScriptCore/jit/CallFrameShuffler.cpp b/Source/JavaScriptCore/jit/CallFrameShuffler.cpp >new file mode 100644 >index 0000000000000000000000000000000000000000..a1dc9dfc171641df37f75073a64120d18c4475b2 >--- /dev/null >+++ b/Source/JavaScriptCore/jit/CallFrameShuffler.cpp >@@ -0,0 +1,704 @@ >+/* >+ * Copyright (C) 2015 Apple Inc. All rights reserved. >+ * >+ * Redistribution and use in source and binary forms, with or without >+ * modification, are permitted provided that the following conditions >+ * are met: >+ * 1. Redistributions of source code must retain the above copyright >+ * notice, this list of conditions and the following disclaimer. >+ * 2. Redistributions in binary form must reproduce the above copyright >+ * notice, this list of conditions and the following disclaimer in the >+ * documentation and/or other materials provided with the distribution. >+ * >+ * THIS SOFTWARE IS PROVIDED BY APPLE INC. ``AS IS'' AND ANY >+ * EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE >+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR >+ * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL APPLE INC. OR >+ * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, >+ * EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, >+ * PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR >+ * PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY >+ * OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT >+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE >+ * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. >+ */ >+ >+#include "config.h" >+#include "CallFrameShuffler.h" >+ >+#if ENABLE(JIT) >+ >+#include "CCallHelpers.h" >+#include "CodeBlock.h" >+ >+namespace JSC { >+ >+CallFrameShuffler::CallFrameShuffler(CCallHelpers& jit, const CallFrameShuffleData& data) >+ : m_jit(jit) >+ , m_oldFrame(data.numLocals + JSStack::CallerFrameAndPCSize, nullptr) >+ , m_newFrame(data.args.size() + JSStack::CallFrameHeaderSize, nullptr) >+ , m_alignedOldFrameSize(JSStack::CallFrameHeaderSize >+ + roundArgumentCountToAlignFrame(jit.codeBlock()->numParameters())) >+ , m_alignedNewFrameSize(JSStack::CallFrameHeaderSize >+ + roundArgumentCountToAlignFrame(data.args.size())) >+ , m_frameDelta(m_alignedNewFrameSize - m_alignedOldFrameSize) >+ , m_lockedRegisters(RegisterSet::allRegisters()) >+{ >+ // We are allowed all the usual registers... >+ for (unsigned i = GPRInfo::numberOfRegisters; i--; ) >+ m_lockedRegisters.clear(GPRInfo::toRegister(i)); >+ for (unsigned i = FPRInfo::numberOfRegisters; i--; ) >+ m_lockedRegisters.clear(FPRInfo::toRegister(i)); >+ // ... as well as the runtime registers. >+ m_lockedRegisters.exclude(RegisterSet::runtimeRegisters()); >+ >+ ASSERT(!data.callee.isInJSStack() || data.callee.virtualRegister().isLocal()); >+ addNew(VirtualRegister(JSStack::Callee), data.callee); >+ >+ for (size_t i = 0; i < data.args.size(); ++i) { >+ ASSERT(!data.args[i].isInJSStack() || data.args[i].virtualRegister().isLocal()); >+ addNew(virtualRegisterForArgument(i), data.args[i]); >+ } >+ >+#if USE(JSVALUE64) >+ for (Reg reg = Reg::first(); reg <= Reg::last(); reg = reg.next()) { >+ if (!data.registers[reg].isSet()) >+ continue; >+ >+ RELEASE_ASSERT(reg.isGPR()); >+ >+ addNew(JSValueRegs(reg.gpr()), >+ data.registers[reg]); >+ } >+#endif >+} >+ >+void CallFrameShuffler::dump(PrintStream& out) const >+{ >+ static const char* delimiter = " +-------------------------------+ "; >+ static const char* dangerDelimiter = " X-------------------------------X "; >+ static const char* dangerBoundsDelimiter = " XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX "; >+ static const char* emptySpace = " "; >+ ASSERT(m_alignedNewFrameSize <= numLocals()); >+ out.print(" "); >+ out.print(" Old frame "); >+ out.print(" New frame "); >+ out.print("\n"); >+ for (int i = 0; i < m_alignedOldFrameSize + numLocals() + 3; ++i) { >+ VirtualRegister old { m_alignedOldFrameSize - i - 1 }; >+ VirtualRegister newReg { old + m_frameDelta }; >+ >+ if (!isValidOld(old) && old != firstOld() - 1 >+ && !isValidNew(newReg) && newReg != firstNew() - 1) >+ continue; >+ >+ out.print(" "); >+ if (dangerFrontier() >= firstNew() >+ && (newReg == dangerFrontier() || newReg == firstNew() - 1)) >+ out.print(dangerBoundsDelimiter); >+ else if (isValidOld(old)) >+ out.print(isValidNew(newReg) && isDangerNew(newReg) ? dangerDelimiter : delimiter); >+ else if (old == firstOld() - 1) >+ out.print(delimiter); >+ else >+ out.print(emptySpace); >+ if (dangerFrontier() >= firstNew() >+ && (newReg == dangerFrontier() || newReg == firstNew() - 1)) >+ out.print(dangerBoundsDelimiter); >+ else if (isValidNew(newReg) || newReg == firstNew() - 1) >+ out.print(isDangerNew(newReg) ? dangerDelimiter : delimiter); >+ else >+ out.print(emptySpace); >+ out.print("\n"); >+ if (old == firstOld()) >+ out.print(" sp --> "); >+ else if (!old.offset()) >+ out.print(" fp --> "); >+ else >+ out.print(" "); >+ if (isValidOld(old)) { >+ if (getOld(old)) { >+ auto str = toCString(old); >+ if (isValidNew(newReg) && isDangerNew(newReg)) >+ out.printf(" X %18s X ", str.data()); >+ else >+ out.printf(" | %18s | ", str.data()); >+ } else if (isValidNew(newReg) && isDangerNew(newReg)) >+ out.printf(" X%30s X ", ""); >+ else >+ out.printf(" |%30s | ", ""); >+ } else >+ out.print(emptySpace); >+ if (isValidNew(newReg)) { >+ const char d = isDangerNew(newReg) ? 'X' : '|'; >+ auto str = toCString(newReg); >+ if (getNew(newReg)) { >+ if (getNew(newReg)->recovery().isConstant()) >+ out.printf(" %c%8s <- constant %c ", d, str.data(), d); >+ else { >+ auto recoveryStr = toCString(getNew(newReg)->recovery()); >+ out.printf(" %c%8s <- %18s %c ", d, str.data(), >+ recoveryStr.data(), d); >+ } >+ } else if (newReg == VirtualRegister { JSStack::ArgumentCount }) >+ out.printf(" %c%8s <- %18zu %c ", d, str.data(), argCount(), d); >+ else >+ out.printf(" %c%30s %c ", d, "", d); >+ } else >+ out.print(emptySpace); >+ if (newReg == firstNew() - m_newFrameOffset && !isSlowPath()) >+ out.print(" <-- new sp before jump (current ", m_newFrameBase, ") "); >+ if (newReg == firstNew()) >+ out.print(" <-- new fp after prologue"); >+ out.print("\n"); >+ } >+ out.print(" "); >+ out.print(" Live registers "); >+ out.print(" Wanted registers "); >+ out.print("\n"); >+ for (Reg reg = Reg::first(); reg <= Reg::last(); reg = reg.next()) { >+ if (!reg.isGPR()) >+ continue; >+ >+ CachedRecovery* oldCachedRecovery { m_registers[reg.gpr()] }; >+ CachedRecovery* newCachedRecovery { m_newRegisters[reg.gpr()] }; >+ if (!oldCachedRecovery && !newCachedRecovery) >+ continue; >+ out.print(" "); >+ if (oldCachedRecovery) { >+ auto str = toCString(reg.gpr()); >+ out.printf(" %8s ", str.data()); >+ } else >+ out.print(emptySpace); >+#if USE(JSVALUE32_64) >+ if (newCachedRecovery) { >+ JSValueRegs wantedJSValueRegs { newCachedRecovery->wantedJSValueRegs() }; >+ if (reg.gpr() == wantedJSValueRegs.tagGPR()) >+ out.print(reg.gpr(), " <- tag(", newCachedRecovery->recovery(), ")"); >+ else >+ out.print(reg.gpr(), " <- payload(", newCachedRecovery->recovery(), ")"); >+ } >+#else >+ if (newCachedRecovery) >+ out.print(" ", reg.gpr(), " <- ", newCachedRecovery->recovery()); >+#endif >+ out.print("\n"); >+ } >+ out.print(" Locked registers: "); >+ bool firstLocked { true }; >+ for (Reg reg = Reg::first(); reg <= Reg::last(); reg = reg.next()) { >+ if (!reg.isGPR()) >+ continue; >+ >+ if (m_lockedRegisters.get(reg.gpr())) { >+ out.print(firstLocked ? "" : ", ", reg.gpr()); >+ firstLocked = false; >+ } >+ } >+ out.print("\n"); >+ >+ if (isSlowPath()) >+ out.print(" Using fp-relative addressing for slow path call\n"); >+ else >+ out.print(" Using sp-relative addressing for jump (using ", m_newFrameBase, " as new sp)\n"); >+ if (m_oldFrameOffset) >+ out.print(" Old frame offset is ", m_oldFrameOffset, "\n"); >+ if (m_newFrameOffset) >+ out.print(" New frame offset is ", m_newFrameOffset, "\n"); >+} >+ >+CachedRecovery* CallFrameShuffler::getCachedRecovery(ValueRecovery recovery) >+{ >+ ASSERT(!recovery.isConstant()); >+ if (recovery.isInGPR()) >+ return m_registers[recovery.gpr()]; >+ if (recovery.isInFPR()) >+ return m_registers[recovery.fpr()]; >+#if USE(JSVALUE32_64) >+ if (recovery.technique() == InPair) { >+ ASSERT(m_registers[recovery.tagGPR()] == m_registers[recovery.payloadGPR()]); >+ return m_registers[recovery.payloadGPR()]; >+ } >+#endif >+ ASSERT(recovery.isInJSStack()); >+ return getOld(recovery.virtualRegister()); >+} >+ >+CachedRecovery* CallFrameShuffler::setCachedRecovery(ValueRecovery recovery, CachedRecovery* cachedRecovery) >+{ >+ ASSERT(!recovery.isConstant()); >+ if (recovery.isInGPR()) >+ return m_registers[recovery.gpr()] = cachedRecovery; >+ if (recovery.isInFPR()) >+ return m_registers[recovery.fpr()] = cachedRecovery; >+#if USE(JSVALUE32_64) >+ if (recovery.technique() == InPair) { >+ m_registers[recovery.tagGPR()] = cachedRecovery; >+ return m_registers[recovery.payloadGPR()] = cachedRecovery; >+ } >+#endif >+ ASSERT(recovery.isInJSStack()); >+ setOld(recovery.virtualRegister(), cachedRecovery); >+ return cachedRecovery; >+} >+ >+void CallFrameShuffler::spill(CachedRecovery& cachedRecovery) >+{ >+ ASSERT(!isSlowPath()); >+ ASSERT(cachedRecovery.recovery().isInRegisters()); >+ >+ VirtualRegister spillSlot { 0 }; >+ for (VirtualRegister slot = firstOld(); slot <= lastOld(); slot -= 1) { >+ ASSERT(slot < newAsOld(firstNew())); >+ if (getOld(slot)) >+ continue; >+ >+ spillSlot = slot; >+ break; >+ } >+ // We must have enough slots to be able to fit the whole >+ // callee's frame for the slow path. >+ RELEASE_ASSERT(spillSlot.isLocal()); >+ >+ if (verbose) >+ dataLog(" * Spilling ", cachedRecovery.recovery(), " into ", spillSlot, "\n"); >+ auto format = emitStore(cachedRecovery, addressForOld(spillSlot)); >+ ASSERT(format != DataFormatNone); >+ updateRecovery(cachedRecovery, ValueRecovery::displacedInJSStack(spillSlot, format)); >+} >+ >+void CallFrameShuffler::emitDeltaCheck() >+{ >+ if (!ASSERT_DISABLED) >+ return; >+ >+ GPRReg scratchGPR { getFreeGPR() }; >+ if (scratchGPR != InvalidGPRReg) { >+ if (verbose) >+ dataLog(" Using ", scratchGPR, " for the fp-sp delta check\n"); >+ m_jit.move(GPRInfo::callFrameRegister, scratchGPR); >+ m_jit.subPtr(MacroAssembler::stackPointerRegister, scratchGPR); >+ MacroAssembler::Jump ok = m_jit.branch32( >+ MacroAssembler::Equal, scratchGPR, >+ MacroAssembler::TrustedImm32( >+ static_cast<size_t>(sizeof(Register) * numLocals()))); >+ m_jit.abortWithReason(JITUnexpectedCallFrameSize); >+ ok.link(&m_jit); >+ } else if (verbose) >+ dataLog(" Skipping the fp-sp delta check since there is too much pressure"); >+} >+ >+void CallFrameShuffler::prepareForSlowPath() >+{ >+ ASSERT(isUndecided()); >+ emitDeltaCheck(); >+ >+ m_frameDelta = numLocals() + JSStack::CallerFrameAndPCSize; >+ m_newFrameBase = MacroAssembler::stackPointerRegister; >+ m_newFrameOffset = -JSStack::CallerFrameAndPCSize; >+ >+ if (verbose) >+ dataLog("\n\nPreparing frame for slow path call:\n", *this); >+ >+ prepareAny(); >+ >+ if (verbose) >+ dataLog("Ready for slow path call!\n"); >+} >+ >+void CallFrameShuffler::prepareForTailCall() >+{ >+ ASSERT(isUndecided()); >+ emitDeltaCheck(); >+ >+ // We'll use sp-based indexing so that we can load the >+ // caller's frame pointer into the fpr immediately >+ m_oldFrameBase = MacroAssembler::stackPointerRegister; >+ m_oldFrameOffset = numLocals(); >+ m_newFrameBase = acquireGPR(); >+#if CPU(X86) >+ // We load the frame pointer manually, but we need to ask the >+ // algorithm to move the return PC for us (it'd probably >+ // require a write to the danger zone). Since it'd be awkward >+ // to ask for half a value move, we ask that the whole thing >+ // be moved for us. >+ addNew(VirtualRegister { 0 }, >+ ValueRecovery::displacedInJSStack(VirtualRegister(0), DataFormatJS)); >+ >+ // sp will point to head0 and we will move it up half a slot >+ // manually >+ m_newFrameOffset = 0; >+#elif CPU(ARM) || CPU(SH4) || CPU(MIPS) >+ // We load the the frame pointer and link register >+ // manually. We could ask the algorithm to load them for us, >+ // and it would allow us to use the link register as an extra >+ // temporary - but it'd mean that the frame pointer can also >+ // be used as an extra temporary, so we keep the link register >+ // locked instead. >+ >+ // sp will point to head1 since the callee's prologue pushes >+ // the call frame and link register. >+ m_newFrameOffset = -1; >+#elif CPU(ARM64) >+ // We load the frame pointer and link register manually. We >+ // could ask the algorithm to load the link register for us >+ // (which would allow for its use as an extra temporary), but >+ // since its not in GPRInfo, we can't do it. >+ >+ // sp will point to head2 since the callee's prologue pushes the >+ // call frame and link register >+ m_newFrameOffset = -2; >+#elif CPU(X86_64) >+ // We load the frame pointer manually, but we ask the >+ // algorithm to move the return PC for us (it'd probably >+ // require a write in the danger zone) >+ addNew(VirtualRegister { 1 }, >+ ValueRecovery::displacedInJSStack(VirtualRegister(1), DataFormatJS)); >+ >+ // sp will point to head1 since the callee's prologue pushes >+ // the call frame register >+ m_newFrameOffset = -1; >+#else >+ UNREACHABLE_FOR_PLATFORM(); >+#endif >+ >+ if (verbose) >+ dataLog(" Emitting code for computing the new frame base\n"); >+ >+ // We compute the new frame base by first computing the top of the >+ // old frame (taking into account an argument count higher than >+ // the number of parameters), then substracting to it the aligned >+ // new frame size (adjusted). >+ m_jit.load32(MacroAssembler::Address(GPRInfo::callFrameRegister, JSStack::ArgumentCount * static_cast<int>(sizeof(Register)) + PayloadOffset), m_newFrameBase); >+ MacroAssembler::Jump argumentCountOK = >+ m_jit.branch32(MacroAssembler::BelowOrEqual, m_newFrameBase, >+ MacroAssembler::TrustedImm32(m_jit.codeBlock()->numParameters())); >+ m_jit.add32(MacroAssembler::TrustedImm32(stackAlignmentRegisters() - 1 + JSStack::CallFrameHeaderSize), m_newFrameBase); >+ m_jit.and32(MacroAssembler::TrustedImm32(-stackAlignmentRegisters()), m_newFrameBase); >+ m_jit.mul32(MacroAssembler::TrustedImm32(sizeof(Register)), m_newFrameBase, m_newFrameBase); >+ MacroAssembler::Jump done = m_jit.jump(); >+ argumentCountOK.link(&m_jit); >+ m_jit.move( >+ MacroAssembler::TrustedImm32(m_alignedOldFrameSize * sizeof(Register)), >+ m_newFrameBase); >+ done.link(&m_jit); >+ >+ m_jit.addPtr(GPRInfo::callFrameRegister, m_newFrameBase); >+ m_jit.subPtr( >+ MacroAssembler::TrustedImm32( >+ (m_alignedNewFrameSize + m_newFrameOffset) * sizeof(Register)), >+ m_newFrameBase); >+ >+ // We want the frame pointer to always point to a valid frame, and >+ // we are going to trash the current one. Let's make it point to >+ // our caller's frame, since that's what we want to end up with. >+ m_jit.loadPtr(MacroAssembler::Address(MacroAssembler::framePointerRegister), >+ MacroAssembler::framePointerRegister); >+ >+ // We load the link register manually for architectures that have one >+#if CPU(ARM) || CPU(SH4) || CPU(ARM64) >+ m_jit.loadPtr(MacroAssembler::Address(MacroAssembler::framePointerRegister, sizeof(void*)), >+ MacroAssembler::linkRegister); >+#elif CPU(MIPS) >+ m_jit.loadPtr(MacroAssembler::Address(MacroAssembler::framePointerRegister, sizeof(void*)), >+ MacroAssembler::returnAddressRegister); >+#endif >+ >+ if (verbose) >+ dataLog("Preparing frame for tail call:\n", *this); >+ >+ prepareAny(); >+ >+#if CPU(X86) >+ if (verbose) >+ dataLog(" Simulating pop of the call frame register\n"); >+ m_jit.addPtr(MacroAssembler::TrustedImm32(sizeof(void*)), MacroAssembler::stackPointerRegister); >+#endif >+ >+ if (verbose) >+ dataLog("Ready for tail call!\n"); >+} >+ >+bool CallFrameShuffler::tryWrites(CachedRecovery& cachedRecovery) >+{ >+ ASSERT(m_newFrameBase != InvalidGPRReg); >+ >+ // If the value is already set up correctly, we don't have >+ // anything to do. >+ if (isSlowPath() && cachedRecovery.recovery().isInJSStack() >+ && cachedRecovery.targets().size() == 1 >+ && (newAsOld(cachedRecovery.targets()[0]) >+ == cachedRecovery.recovery().virtualRegister())) { >+ cachedRecovery.clearTargets(); >+ if (!cachedRecovery.wantedJSValueRegs()) >+ clearCachedRecovery(cachedRecovery.recovery()); >+ return true; >+ } >+ >+ if (!canLoadAndBox(cachedRecovery)) >+ return false; >+ >+ emitLoad(cachedRecovery); >+ emitBox(cachedRecovery); >+ ASSERT(cachedRecovery.recovery().isInRegisters() >+ || cachedRecovery.recovery().isConstant()); >+ >+ if (verbose) >+ dataLog(" * Storing ", cachedRecovery.recovery()); >+ for (size_t i = 0; i < cachedRecovery.targets().size(); ++i) { >+ VirtualRegister target { cachedRecovery.targets()[i] }; >+ ASSERT(!isDangerNew(target)); >+ if (verbose) >+ dataLog(!i ? " into " : ", and ", "NEW ", target); >+ emitStore(cachedRecovery, addressForNew(target)); >+ setNew(target, nullptr); >+ } >+ if (verbose) >+ dataLog("\n"); >+ cachedRecovery.clearTargets(); >+ if (!cachedRecovery.wantedJSValueRegs()) >+ clearCachedRecovery(cachedRecovery.recovery()); >+ >+ return true; >+} >+ >+bool CallFrameShuffler::performSafeWrites() >+{ >+ VirtualRegister firstSafe; >+ VirtualRegister end { lastNew() + 1 }; >+ Vector<VirtualRegister> failures; >+ >+ // For all cachedRecoveries that writes to the safe zone, if it >+ // doesn't also write to the danger zone, we try to perform >+ // the writes. This may free up danger slots, so we iterate >+ // again until it doesn't happen anymore. >+ // >+ // Note that even though we have a while block, we look at >+ // each slot of the new call frame at most once since in each >+ // iteration beyond the first, we only load up the portion of >+ // the new call frame that was dangerous and became safe due >+ // to the previous iteration. >+ do { >+ firstSafe = dangerFrontier() + 1; >+ if (verbose) >+ dataLog(" Trying safe writes (between NEW ", firstSafe, " and NEW ", end - 1, ")\n"); >+ bool didProgress = false; >+ for (VirtualRegister reg = firstSafe; reg < end; reg += 1) { >+ CachedRecovery* cachedRecovery = getNew(reg); >+ if (!cachedRecovery) { >+ if (verbose) >+ dataLog(" + ", reg, " is OK.\n"); >+ continue; >+ } >+ if (!hasOnlySafeWrites(*cachedRecovery)) { >+ if (verbose) { >+ dataLog(" - ", cachedRecovery->recovery(), " writes to NEW ", reg, >+ " but also has dangerous writes.\n"); >+ } >+ continue; >+ } >+ if (cachedRecovery->wantedJSValueRegs()) { >+ if (verbose) { >+ dataLog(" - ", cachedRecovery->recovery(), " writes to NEW ", reg, >+ " but is also needed in registers.\n"); >+ } >+ continue; >+ } >+ if (!tryWrites(*cachedRecovery)) { >+ if (verbose) >+ dataLog(" - Unable to write to NEW ", reg, " from ", cachedRecovery->recovery(), "\n"); >+ failures.append(reg); >+ } >+ didProgress = true; >+ } >+ end = firstSafe; >+ >+ // If we have cachedRecoveries that failed to write, it is >+ // because they are on the stack and we didn't have enough >+ // registers available at the time to load them into. If >+ // we have a free register, we should try again because it >+ // could free up some danger slots. >+ if (didProgress && hasFreeRegister()) { >+ Vector<VirtualRegister> stillFailing; >+ for (VirtualRegister failed : failures) { >+ CachedRecovery* cachedRecovery = getNew(failed); >+ // It could have been handled later if it had >+ // several targets >+ if (!cachedRecovery) >+ continue; >+ >+ ASSERT(hasOnlySafeWrites(*cachedRecovery) >+ && !cachedRecovery->wantedJSValueRegs()); >+ if (!tryWrites(*cachedRecovery)) >+ stillFailing.append(failed); >+ } >+ failures = WTF::move(stillFailing); >+ } >+ if (verbose && firstSafe != dangerFrontier() + 1) >+ dataLog(" We freed up danger slots!\n"); >+ } while (firstSafe != dangerFrontier() + 1); >+ >+ return failures.isEmpty(); >+} >+ >+void CallFrameShuffler::prepareAny() >+{ >+ ASSERT(!isUndecided()); >+ >+ updateDangerFrontier(); >+ >+ // First, we try to store any value that goes above the danger >+ // frontier. This will never use more registers since we are only >+ // loading+storing if we ensure that any register used for the load >+ // will be freed up after the stores (i.e., all stores are above >+ // the danger frontier, and there is no wanted register). >+ performSafeWrites(); >+ >+ // At this point, we couldn't have more available registers than >+ // we have withouth spilling: all values currently in registers >+ // either require a write to the danger zone, or have a wanted >+ // register, which means that in any case they will have to go >+ // through registers again. >+ >+ // We now slowly free up the danger zone by first loading the old >+ // value on the danger frontier, spilling as many registers as >+ // needed to do so and ensuring that the corresponding slot in the >+ // new frame is now ready to be written. Then, we store the old >+ // value to its target location if possible (we could have failed >+ // to load it previously due to high pressure). Finally, we write >+ // to any of the newly safe slots that we can, which could free up >+ // registers (hence why we do it eagerly). >+ for (VirtualRegister reg = dangerFrontier(); reg >= firstNew(); reg -= 1) { >+ if (reg == dangerFrontier()) { >+ if (verbose) >+ dataLog(" Next slot (NEW ", reg, ") is the danger frontier\n"); >+ CachedRecovery* cachedRecovery { getOld(newAsOld(dangerFrontier())) }; >+ ASSERT(cachedRecovery); >+ ensureLoad(*cachedRecovery); >+ emitLoad(*cachedRecovery); >+ ensureBox(*cachedRecovery); >+ emitBox(*cachedRecovery); >+ if (hasOnlySafeWrites(*cachedRecovery)) >+ tryWrites(*cachedRecovery); >+ } else if (verbose) >+ dataLog(" Next slot is NEW ", reg, "\n"); >+ >+ ASSERT(!isDangerNew(reg)); >+ CachedRecovery* cachedRecovery = getNew(reg); >+ // This could be one of the header slots we don't care about. >+ if (!cachedRecovery) { >+ if (verbose) >+ dataLog(" + ", reg, " is OK\n"); >+ continue; >+ } >+ >+ if (canLoadAndBox(*cachedRecovery) && hasOnlySafeWrites(*cachedRecovery) >+ && !cachedRecovery->wantedJSValueRegs()) { >+ emitLoad(*cachedRecovery); >+ emitBox(*cachedRecovery); >+ bool writesOK = tryWrites(*cachedRecovery); >+ ASSERT(writesOK); >+ } else if (verbose) >+ dataLog(" - ", cachedRecovery->recovery(), " can't be handled just yet.\n"); >+ } >+ ASSERT(dangerFrontier() < firstNew()); >+ >+ // Now, the danger zone is empty, but we still have a couple of >+ // things to do: >+ // >+ // 1) There could be remaining safe writes that failed earlier due >+ // to high register pressure and had nothing to do with the >+ // danger zone whatsoever. >+ // >+ // 2) Some wanted registers could have to be loaded (this could >+ // happen either when making a call to a new function with a >+ // lower number of arguments - since above here, we only load >+ // wanted registers when they are at the danger frontier -, or >+ // if a wanted register got spilled). >+ // >+ // 3) Some wanted registers could have been loaded in the wrong >+ // registers >+ // >+ // 4) We have to take care of some bookkeeping - namely, storing >+ // the argument count and updating the stack pointer. >+ >+ // At this point, we must have enough registers available for >+ // handling 1). None of the loads can fail because we have been >+ // eagerly freeing up registers in all the previous phases - so >+ // the only values that are in registers at this point must have >+ // wanted registers. >+ if (verbose) >+ dataLog(" Danger zone is clear, performing remaining writes.\n"); >+ for (VirtualRegister reg = firstNew(); reg <= lastNew(); reg += 1) { >+ CachedRecovery* cachedRecovery { getNew(reg) }; >+ if (!cachedRecovery) >+ continue; >+ >+ emitLoad(*cachedRecovery); >+ emitBox(*cachedRecovery); >+ bool writesOK = tryWrites(*cachedRecovery); >+ ASSERT(writesOK); >+ } >+ >+ // Handle 2) by loading all registers. We don't have to do any >+ // writes, since they have been taken care of above. >+ if (verbose) >+ dataLog(" Loading wanted GPRs into registers\n"); >+ for (Reg reg = Reg::first(); reg <= Reg::last(); reg = reg.next()) { >+ CachedRecovery* cachedRecovery { m_newRegisters[reg] }; >+ if (!cachedRecovery) >+ continue; >+ ASSERT(reg.isGPR()); >+ >+ emitLoad(*cachedRecovery); >+ emitBox(*cachedRecovery); >+ ASSERT(cachedRecovery->targets().isEmpty()); >+ } >+ >+ // At this point, we have read everything we cared about from the >+ // stack, and written everything we had to to the stack. >+ if (verbose) >+ dataLog(" Callee frame is fully set up\n"); >+ if (!ASSERT_DISABLED) { >+ for (VirtualRegister reg = firstNew(); reg <= lastNew(); reg += 1) >+ ASSERT_UNUSED(reg, !getNew(reg)); >+ >+ for (CachedRecovery* cachedRecovery : m_cachedRecoveries) { >+ ASSERT_UNUSED(cachedRecovery, cachedRecovery->targets().isEmpty()); >+ ASSERT(!cachedRecovery->recovery().isInJSStack()); >+ } >+ } >+ >+ // We need to handle 4) first because it implies releasing >+ // m_newFrameBase, which could be a wanted register. >+ if (verbose) >+ dataLog(" * Storing the argument count into ", VirtualRegister { JSStack::ArgumentCount }, "\n"); >+ m_jit.store32(MacroAssembler::TrustedImm32(0), >+ addressForNew(VirtualRegister { JSStack::ArgumentCount }).withOffset(TagOffset)); >+ m_jit.store32(MacroAssembler::TrustedImm32(argCount()), >+ addressForNew(VirtualRegister { JSStack::ArgumentCount }).withOffset(PayloadOffset)); >+ >+ if (!isSlowPath()) { >+ ASSERT(m_newFrameBase != MacroAssembler::stackPointerRegister); >+ if (verbose) >+ dataLog(" Releasing the new frame base pointer\n"); >+ m_jit.move(m_newFrameBase, MacroAssembler::stackPointerRegister); >+ releaseGPR(m_newFrameBase); >+ } >+ >+ // Finally we handle 3) >+ if (verbose) >+ dataLog(" Ensuring wanted registers are in the right register\n"); >+ for (Reg reg = Reg::first(); reg <= Reg::last(); reg = reg.next()) { >+ CachedRecovery* cachedRecovery { m_newRegisters[reg] }; >+ if (!cachedRecovery) >+ continue; >+ ASSERT(reg.isGPR()); >+ >+ emitDisplace(*cachedRecovery); >+ } >+} >+ >+} // namespace JSC >+ >+#endif // ENABLE(JIT) >diff --git a/Source/JavaScriptCore/jit/CallFrameShuffler.h b/Source/JavaScriptCore/jit/CallFrameShuffler.h >new file mode 100644 >index 0000000000000000000000000000000000000000..17653d1e33a00a142c9e968f2457cb303c3c810b >--- /dev/null >+++ b/Source/JavaScriptCore/jit/CallFrameShuffler.h >@@ -0,0 +1,809 @@ >+/* >+ * Copyright (C) 2015 Apple Inc. All rights reserved. >+ * >+ * Redistribution and use in source and binary forms, with or without >+ * modification, are permitted provided that the following conditions >+ * are met: >+ * 1. Redistributions of source code must retain the above copyright >+ * notice, this list of conditions and the following disclaimer. >+ * 2. Redistributions in binary form must reproduce the above copyright >+ * notice, this list of conditions and the following disclaimer in the >+ * documentation and/or other materials provided with the distribution. >+ * >+ * THIS SOFTWARE IS PROVIDED BY APPLE INC. ``AS IS'' AND ANY >+ * EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE >+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR >+ * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL APPLE INC. OR >+ * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, >+ * EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, >+ * PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR >+ * PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY >+ * OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT >+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE >+ * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. >+ */ >+ >+#ifndef CallFrameShuffler_h >+#define CallFrameShuffler_h >+ >+#if ENABLE(JIT) >+ >+#include "CallFrameShuffleData.h" >+#include "MacroAssembler.h" >+#include "RegisterSet.h" >+#include "StackAlignment.h" >+#include "ValueRecovery.h" >+#include "VirtualRegister.h" >+#include <wtf/Vector.h> >+ >+namespace JSC { >+ >+// A CachedRecovery is a wrapper around a ValueRecovery that records where said >+// value should go on the stack and/or in registers. Whenever we perform an >+// operation changing the ValueRecovery, we update the CachedRecovery's member >+// in place. >+class CachedRecovery { >+public: >+ CachedRecovery(ValueRecovery recovery) >+ : m_recovery { recovery } >+ { >+ } >+ >+ CachedRecovery(CachedRecovery&) = delete; >+ CachedRecovery(CachedRecovery&&) = delete; >+ CachedRecovery& operator=(CachedRecovery&) = delete; >+ CachedRecovery& operator=(CachedRecovery&&) = delete; >+ >+ const Vector<VirtualRegister, 1>& targets() const { return m_targets; } >+ >+ void addTarget(VirtualRegister reg) >+ { >+ ASSERT(m_targets.isEmpty() || m_targets.last() < reg); >+ m_targets.append(reg); >+ } >+ >+ void removeTarget(VirtualRegister reg) >+ { >+ ASSERT(m_targets.last() == reg); >+ m_targets.shrink(m_targets.size() - 1); >+ } >+ >+ void clearTargets() >+ { >+ m_targets.clear(); >+ } >+ >+ void setWantedJSValueRegs(JSValueRegs jsValueRegs) >+ { >+ m_wantedJSValueRegs = jsValueRegs; >+ } >+ >+ // Determine whether converting this recovery into a JSValue will >+ // require additional GPRs and/or FPRs. >+ // This is guaranteed to only depend on the DataFormat, and the >+ // result of these calls will stay valid after loads and/or stores. >+ bool boxingRequiresGPR() const >+ { >+#if USE(JSVALUE64) >+ return recovery().dataFormat() == DataFormatDouble; >+#else >+ return false; >+#endif >+ } >+ bool boxingRequiresFPR() const >+ { >+#if USE(JSVALUE64) >+ switch (recovery().dataFormat()) { >+ case DataFormatInt52: >+ case DataFormatStrictInt52: >+ return true; >+ >+ default: >+ return false; >+ } >+#else >+ return false; >+#endif >+ } >+ >+ // This is used to determine what kind of register we need to be >+ // able to load a recovery. We only use it when a direct load is >+ // currently impossible, to determine whether we should spill a >+ // GPR or an FPR for loading this value. >+ bool loadsIntoGPR() const; >+ bool loadsIntoFPR() const; >+ >+ ValueRecovery recovery() const { return m_recovery; } >+ >+ void setRecovery(ValueRecovery recovery) { m_recovery = recovery; } >+ >+ JSValueRegs wantedJSValueRegs() const { return m_wantedJSValueRegs; } >+private: >+ ValueRecovery m_recovery; >+ JSValueRegs m_wantedJSValueRegs; >+ Vector<VirtualRegister, 1> m_targets; >+}; >+ >+class CallFrameShuffler { >+public: >+ CallFrameShuffler(CCallHelpers&, const CallFrameShuffleData&); >+ >+ void dump(PrintStream&) const; >+ >+ // Any register that has been locked or acquired must be released >+ // before calling prepareForTailCall() or prepareForSlowPath(). >+ void lockGPR(GPRReg gpr) >+ { >+ ASSERT(!m_lockedRegisters.get(gpr)); >+ m_lockedRegisters.set(gpr); >+ if (verbose) >+ dataLog(" * Locking ", gpr, "\n"); >+ } >+ >+ GPRReg acquireGPR() >+ { >+ ensureGPR(); >+ GPRReg gpr { getFreeGPR() }; >+ ASSERT(!m_registers[gpr]); >+ lockGPR(gpr); >+ return gpr; >+ } >+ >+ void releaseGPR(GPRReg gpr) >+ { >+ if (verbose) { >+ if (m_lockedRegisters.get(gpr)) >+ dataLog(" * Releasing ", gpr, "\n"); >+ else >+ dataLog(" * ", gpr, " was not locked\n"); >+ } >+ m_lockedRegisters.clear(gpr); >+ } >+ >+ // You can only take a snapshot if the recovery has not started >+ // yet. The only operations that are valid before taking a >+ // snapshot are lockGPR(), acquireGPR() and releaseGPR(). >+ // >+ // Locking status is *NOT* preserved by the snapshot: it only >+ // contains information about where the >+ // arguments/callee/callee-save registers are by taking into >+ // account any spilling that acquireGPR() could have done. >+ CallFrameShuffleData snapshot() const >+ { >+ ASSERT(isUndecided()); >+ >+ CallFrameShuffleData data; >+ data.numLocals = numLocals(); >+ data.callee = getNew(VirtualRegister { JSStack::Callee })->recovery(); >+ data.args.resize(argCount()); >+ for (size_t i = 0; i < argCount(); ++i) >+ data.args[i] = getNew(virtualRegisterForArgument(i))->recovery(); >+ for (Reg reg = Reg::first(); reg <= Reg::last(); reg = reg.next()) { >+ CachedRecovery* cachedRecovery { m_newRegisters[reg] }; >+ if (!cachedRecovery) >+ continue; >+ >+#if USE(JSVALUE64) >+ data.registers[reg] = cachedRecovery->recovery(); >+#else >+ RELEASE_ASSERT_NOT_REACHED(); >+#endif >+ } >+ return WTF::move(data); >+ } >+ >+ // Ask the shuffler to put the callee into some registers once the >+ // shuffling is done. You should call this before any of the >+ // prepare() methods, and must not take a snapshot afterwards, as >+ // this would crash 32bits platforms. >+ void setCalleeJSValueRegs(JSValueRegs jsValueRegs) >+ { >+ ASSERT(isUndecided()); >+ ASSERT(!getNew(jsValueRegs)); >+ CachedRecovery* cachedRecovery { getNew(VirtualRegister(JSStack::Callee)) }; >+ ASSERT(cachedRecovery); >+ addNew(jsValueRegs, cachedRecovery->recovery()); >+ } >+ >+ // Ask the suhffler to assume the callee has already be checked to >+ // be a cell. This is a no-op on 64bit platforms, but allows to >+ // free up a GPR on 32bit platforms. >+ // You obviously must have ensured that this is the case before >+ // running any of the prepare methods. >+ void assumeCalleeIsCell() >+ { >+#if USE(JSVALUE32_64) >+ CachedRecovery& calleeCachedRecovery { *getNew(VirtualRegister(JSStack::Callee)) }; >+ switch (calleeCachedRecovery.recovery().technique()) { >+ case InPair: >+ updateRecovery( >+ calleeCachedRecovery, >+ ValueRecovery::inGPR( >+ calleeCachedRecovery.recovery().payloadGPR(), >+ DataFormatCell)); >+ break; >+ case DisplacedInJSStack: >+ updateRecovery( >+ calleeCachedRecovery, >+ ValueRecovery::displacedInJSStack( >+ calleeCachedRecovery.recovery().virtualRegister(), >+ DataFormatCell)); >+ break; >+ case InFPR: >+ case UnboxedCellInGPR: >+ case CellDisplacedInJSStack: >+ break; >+ case Constant: >+ ASSERT(calleeCachedRecovery.recovery().constant().isCell()); >+ break; >+ default: >+ RELEASE_ASSERT_NOT_REACHED(); >+ break; >+ } >+#endif >+ } >+ >+ // This will emit code to build the new frame over the old one. >+ void prepareForTailCall(); >+ >+ // This will emit code to build the new frame as if performing a >+ // regular call. However, the callee save registers will be >+ // restored, and any locals (not the header or arguments) of the >+ // current frame can be overwritten. >+ // >+ // A frame built using prepareForSlowPath() should be used either >+ // to throw an exception in, or destroyed using >+ // CCallHelpers::prepareForTailCallSlow() followed by a tail call. >+ void prepareForSlowPath(); >+ >+private: >+ static const bool verbose = false; >+ >+ CCallHelpers& m_jit; >+ >+ void prepareAny(); >+ >+ void spill(CachedRecovery&); >+ >+ // "box" is arguably a bad name here. The meaning is that after >+ // calling emitBox(), your ensure that subsequently calling >+ // emitStore() will be able to store the value without additional >+ // transformation. In particular, this is a no-op for constants, >+ // and is a complete no-op on 32bits since any unboxed value can >+ // still be stored by storing the payload and a statically known >+ // tag. >+ void emitBox(CachedRecovery&); >+ >+ bool canBox(CachedRecovery& cachedRecovery) >+ { >+ if (cachedRecovery.boxingRequiresGPR() && getFreeGPR() == InvalidGPRReg) >+ return false; >+ >+ if (cachedRecovery.boxingRequiresFPR() && getFreeFPR() == InvalidFPRReg) >+ return false; >+ >+ return true; >+ } >+ >+ void ensureBox(CachedRecovery& cachedRecovery) >+ { >+ if (canBox(cachedRecovery)) >+ return; >+ >+ if (cachedRecovery.boxingRequiresGPR()) >+ ensureGPR(); >+ >+ if (cachedRecovery.boxingRequiresFPR()) >+ ensureFPR(); >+ } >+ >+ void emitLoad(CachedRecovery&); >+ >+ bool canLoad(CachedRecovery&); >+ >+ void ensureLoad(CachedRecovery& cachedRecovery) >+ { >+ if (canLoad(cachedRecovery)) >+ return; >+ >+ ASSERT(cachedRecovery.loadsIntoGPR() || cachedRecovery.loadsIntoFPR()); >+ >+ if (cachedRecovery.loadsIntoFPR()) { >+ if (cachedRecovery.loadsIntoGPR()) >+ ensureRegister(); >+ else >+ ensureFPR(); >+ } else >+ ensureGPR(); >+ } >+ >+ bool canLoadAndBox(CachedRecovery& cachedRecovery) >+ { >+ // We don't have interfering loads & boxes >+ ASSERT(!cachedRecovery.loadsIntoFPR() || !cachedRecovery.boxingRequiresFPR()); >+ ASSERT(!cachedRecovery.loadsIntoGPR() || !cachedRecovery.boxingRequiresGPR()); >+ >+ return canLoad(cachedRecovery) && canBox(cachedRecovery); >+ } >+ >+ DataFormat emitStore(CachedRecovery&, MacroAssembler::Address); >+ >+ void emitDisplace(CachedRecovery&); >+ >+ void emitDeltaCheck(); >+ >+ Bag<CachedRecovery> m_cachedRecoveries; >+ >+ void updateRecovery(CachedRecovery& cachedRecovery, ValueRecovery recovery) >+ { >+ clearCachedRecovery(cachedRecovery.recovery()); >+ cachedRecovery.setRecovery(recovery); >+ setCachedRecovery(recovery, &cachedRecovery); >+ } >+ >+ CachedRecovery* getCachedRecovery(ValueRecovery); >+ >+ CachedRecovery* setCachedRecovery(ValueRecovery, CachedRecovery*); >+ >+ void clearCachedRecovery(ValueRecovery recovery) >+ { >+ if (!recovery.isConstant()) >+ setCachedRecovery(recovery, nullptr); >+ } >+ >+ CachedRecovery* addCachedRecovery(ValueRecovery recovery) >+ { >+ if (recovery.isConstant()) >+ return m_cachedRecoveries.add(recovery); >+ CachedRecovery* cachedRecovery = getCachedRecovery(recovery); >+ if (!cachedRecovery) >+ return setCachedRecovery(recovery, m_cachedRecoveries.add(recovery)); >+ return cachedRecovery; >+ } >+ >+ // This is the current recoveries present in the old frame's >+ // slots. A null CachedRecovery means we can trash the current >+ // value as we don't care about it. >+ Vector<CachedRecovery*> m_oldFrame; >+ >+ int numLocals() const >+ { >+ return m_oldFrame.size() - JSStack::CallerFrameAndPCSize; >+ } >+ >+ CachedRecovery* getOld(VirtualRegister reg) const >+ { >+ return m_oldFrame[JSStack::CallerFrameAndPCSize - reg.offset() - 1]; >+ } >+ >+ void setOld(VirtualRegister reg, CachedRecovery* cachedRecovery) >+ { >+ m_oldFrame[JSStack::CallerFrameAndPCSize - reg.offset() - 1] = cachedRecovery; >+ } >+ >+ VirtualRegister firstOld() const >+ { >+ return VirtualRegister { static_cast<int>(-numLocals()) }; >+ } >+ >+ VirtualRegister lastOld() const >+ { >+ return VirtualRegister { JSStack::CallerFrameAndPCSize - 1 }; >+ } >+ >+ bool isValidOld(VirtualRegister reg) const >+ { >+ return reg >= firstOld() && reg <= lastOld(); >+ } >+ >+ // This stores, for each slot in the new frame, information about >+ // the recovery for the value that should eventually go into that >+ // slot. >+ // >+ // Once the slot has been written, the corresponding entry in >+ // m_newFrame will be empty. >+ Vector<CachedRecovery*> m_newFrame; >+ >+ size_t argCount() const >+ { >+ return m_newFrame.size() - JSStack::CallFrameHeaderSize; >+ } >+ >+ CachedRecovery* getNew(VirtualRegister newRegister) const >+ { >+ return m_newFrame[newRegister.offset()]; >+ } >+ >+ void setNew(VirtualRegister newRegister, CachedRecovery* cachedRecovery) >+ { >+ m_newFrame[newRegister.offset()] = cachedRecovery; >+ } >+ >+ void addNew(VirtualRegister newRegister, ValueRecovery recovery) >+ { >+ CachedRecovery* cachedRecovery = addCachedRecovery(recovery); >+ cachedRecovery->addTarget(newRegister); >+ setNew(newRegister, cachedRecovery); >+ } >+ >+ VirtualRegister firstNew() const >+ { >+ return VirtualRegister { 0 }; >+ } >+ >+ VirtualRegister lastNew() const >+ { >+ return VirtualRegister { static_cast<int>(m_newFrame.size()) - 1 }; >+ } >+ >+ bool isValidNew(VirtualRegister reg) const >+ { >+ return reg >= firstNew() && reg <= lastNew(); >+ } >+ >+ >+ int m_alignedOldFrameSize; >+ int m_alignedNewFrameSize; >+ >+ // This is the distance, in slots, between the base of the new >+ // frame and the base of the old frame. It could be negative when >+ // preparing for a tail call to a function with smaller argument >+ // count. >+ // >+ // We will overwrite this appropriately for slow path calls, but >+ // we initialize it as if doing a fast path for the spills we >+ // could do while undecided (typically while calling acquireGPR() >+ // for a polymorphic call). >+ int m_frameDelta; >+ >+ VirtualRegister newAsOld(VirtualRegister reg) const >+ { >+ return reg - m_frameDelta; >+ } >+ >+ // This stores the set of locked registers, i.e. registers for >+ // which we have an implicit requirement that they are not changed. >+ // >+ // This will usually contains the link register on architectures >+ // that have one, any scratch register used by the macro assembler >+ // (e.g. r11 on X86_64), as well as any register that we use for >+ // addressing (see m_oldFrameBase and m_newFrameBase). >+ // >+ // We also use this to lock registers temporarily, for instance to >+ // ensure that we have at least 2 available registers for loading >+ // a pair on 32bits. >+ RegisterSet m_lockedRegisters; >+ >+ // This stores the current recoveries present in registers. A null >+ // CachedRecovery means we can trash the current value as we don't >+ // care about it. >+ RegisterMap<CachedRecovery*> m_registers; >+ >+ // This stores, for each register, information about the recovery >+ // for the value that should eventually go into that register. The >+ // only registers that have a target recovery will be callee-save >+ // registers, as well as possibly one JSValueRegs for holding the >+ // callee. >+ // >+ // Once the correct value has been put into the registers, and >+ // contrary to what we do with m_newFrame, we keep the entry in >+ // m_newRegisters to simplify spilling. >+ // >+ // FIXME: Contrary to what the name suggests, we only support GPRs >+ // for now. >+ RegisterMap<CachedRecovery*> m_newRegisters; >+ >+ template<typename CheckFunctor> >+ Reg getFreeRegister(const CheckFunctor& check) const >+ { >+ Reg nonTemp { }; >+ for (Reg reg = Reg::first(); reg <= Reg::last(); reg = reg.next()) { >+ if (m_lockedRegisters.get(reg)) >+ continue; >+ >+ if (!check(reg)) >+ continue; >+ >+ if (!m_registers[reg]) { >+ if (!m_newRegisters[reg]) >+ return reg; >+ if (!nonTemp) >+ nonTemp = reg; >+ } >+ } >+ return nonTemp; >+ } >+ >+ GPRReg getFreeGPR() const >+ { >+ Reg freeGPR { getFreeRegister([] (Reg reg) { return reg.isGPR(); }) }; >+ if (!freeGPR) >+ return InvalidGPRReg; >+ return freeGPR.gpr(); >+ } >+ >+ FPRReg getFreeFPR() const >+ { >+ Reg freeFPR { getFreeRegister([] (Reg reg) { return reg.isFPR(); }) }; >+ if (!freeFPR) >+ return InvalidFPRReg; >+ return freeFPR.fpr(); >+ } >+ >+ bool hasFreeRegister() const >+ { >+ return static_cast<bool>(getFreeRegister([] (Reg) { return true; })); >+ } >+ >+ // This frees up a register satisfying the check functor (this >+ // functor could theoretically have any kind of logic, but it must >+ // ensure that it will only return true for registers - spill >+ // assumes and asserts that it is passed a cachedRecovery stored in a >+ // register). >+ template<typename CheckFunctor> >+ void ensureRegister(const CheckFunctor& check) >+ { >+ // If we can spill a callee-save, that's best, because it will >+ // free up a register that would otherwise been taken for the >+ // longest amount of time. >+ // >+ // We could try to bias towards those that are not in their >+ // target registers yet, but the gain is probably super >+ // small. Unless you have a huge number of argument (at least >+ // around twice the number of available registers on your >+ // architecture), no spilling is going to take place anyways. >+ for (Reg reg = Reg::first(); reg <= Reg::last(); reg = reg.next()) { >+ if (m_lockedRegisters.get(reg)) >+ continue; >+ >+ CachedRecovery* cachedRecovery { m_newRegisters[reg] }; >+ if (!cachedRecovery) >+ continue; >+ >+ if (check(*cachedRecovery)) { >+ if (verbose) >+ dataLog(" ", cachedRecovery->recovery(), " looks like a good spill candidate\n"); >+ spill(*cachedRecovery); >+ return; >+ } >+ } >+ >+ // We use the cachedRecovery associated with the first new slot we >+ // can, because that is the one for which a write will be >+ // possible the latest, i.e. that is the one that we would >+ // have had to retain in registers for the longest. >+ for (VirtualRegister reg = firstNew(); reg <= lastNew(); reg += 1) { >+ CachedRecovery* cachedRecovery { getNew(reg) }; >+ if (!cachedRecovery) >+ continue; >+ >+ if (check(*cachedRecovery)) { >+ spill(*cachedRecovery); >+ return; >+ } >+ } >+ >+ RELEASE_ASSERT_NOT_REACHED(); >+ } >+ >+ void ensureRegister() >+ { >+ if (hasFreeRegister()) >+ return; >+ >+ if (verbose) >+ dataLog(" Finding a register to spill\n"); >+ ensureRegister( >+ [this] (const CachedRecovery& cachedRecovery) { >+ if (cachedRecovery.recovery().isInGPR()) >+ return !m_lockedRegisters.get(cachedRecovery.recovery().gpr()); >+ if (cachedRecovery.recovery().isInFPR()) >+ return !m_lockedRegisters.get(cachedRecovery.recovery().fpr()); >+#if USE(JSVALUE32_64) >+ if (cachedRecovery.recovery().technique() == InPair) { >+ return !m_lockedRegisters.get(cachedRecovery.recovery().tagGPR()) >+ && !m_lockedRegisters.get(cachedRecovery.recovery().payloadGPR()); >+ } >+#endif >+ return false; >+ }); >+ } >+ >+ void ensureGPR() >+ { >+ if (getFreeGPR() != InvalidGPRReg) >+ return; >+ >+ if (verbose) >+ dataLog(" Finding a GPR to spill\n"); >+ ensureRegister( >+ [this] (const CachedRecovery& cachedRecovery) { >+ if (cachedRecovery.recovery().isInGPR()) >+ return !m_lockedRegisters.get(cachedRecovery.recovery().gpr()); >+#if USE(JSVALUE32_64) >+ if (cachedRecovery.recovery().technique() == InPair) { >+ return !m_lockedRegisters.get(cachedRecovery.recovery().tagGPR()) >+ && !m_lockedRegisters.get(cachedRecovery.recovery().payloadGPR()); >+ } >+#endif >+ return false; >+ }); >+ } >+ >+ void ensureFPR() >+ { >+ if (getFreeFPR() != InvalidFPRReg) >+ return; >+ >+ if (verbose) >+ dataLog(" Finding an FPR to spill\n"); >+ ensureRegister( >+ [this] (const CachedRecovery& cachedRecovery) { >+ if (cachedRecovery.recovery().isInFPR()) >+ return !m_lockedRegisters.get(cachedRecovery.recovery().fpr()); >+ return false; >+ }); >+ } >+ >+ CachedRecovery* getNew(JSValueRegs jsValueRegs) const >+ { >+#if USE(JSVALUE64) >+ return m_newRegisters[jsValueRegs.gpr()]; >+#else >+ ASSERT( >+ jsValueRegs.tagGPR() == InvalidGPRReg || jsValueRegs.payloadGPR() == InvalidGPRReg >+ || m_newRegisters[jsValueRegs.payloadGPR()] == m_newRegisters[jsValueRegs.tagGPR()]); >+ if (jsValueRegs.payloadGPR() == InvalidGPRReg) >+ return m_newRegisters[jsValueRegs.tagGPR()]; >+ return m_newRegisters[jsValueRegs.payloadGPR()]; >+#endif >+ } >+ >+ void addNew(JSValueRegs jsValueRegs, ValueRecovery recovery) >+ { >+ ASSERT(jsValueRegs && !getNew(jsValueRegs)); >+ CachedRecovery* cachedRecovery = addCachedRecovery(recovery); >+ ASSERT(!cachedRecovery->wantedJSValueRegs()); >+ cachedRecovery->setWantedJSValueRegs(jsValueRegs); >+#if USE(JSVALUE64) >+ m_newRegisters[jsValueRegs.gpr()] = cachedRecovery; >+#else >+ if (jsValueRegs.payloadGPR() != InvalidGPRReg) >+ m_newRegisters[jsValueRegs.payloadGPR()] = cachedRecovery; >+ if (jsValueRegs.tagGPR() != InvalidGPRReg) >+ m_newRegisters[jsValueRegs.tagGPR()] = cachedRecovery; >+#endif >+ } >+ >+ // m_oldFrameBase is the register relative to which we access >+ // slots in the old call frame, with an additional offset of >+ // m_oldFrameOffset. >+ // >+ // - For an actual tail call, m_oldFrameBase is the stack >+ // pointer, and m_oldFrameOffset is the number of locals of the >+ // tail caller's frame. We use such stack pointer-based >+ // addressing because it allows us to load the tail caller's >+ // caller's frame pointer in the frame pointer register >+ // immediately instead of awkwardly keeping it around on the >+ // stack. >+ // >+ // - For a slow path call, m_oldFrameBase is just the frame >+ // pointer, and m_oldFrameOffset is 0. >+ GPRReg m_oldFrameBase { MacroAssembler::framePointerRegister }; >+ int m_oldFrameOffset { 0 }; >+ >+ MacroAssembler::Address addressForOld(VirtualRegister reg) const >+ { >+ return MacroAssembler::Address(m_oldFrameBase, >+ (m_oldFrameOffset + reg.offset()) * sizeof(Register)); >+ } >+ >+ // m_newFrameBase is the register relative to which we access >+ // slots in the new call frame, and we always make it point to >+ // wherever the stack pointer will be right before making the >+ // actual call/jump. The actual base of the new frame is at offset >+ // m_newFrameOffset relative to m_newFrameBase. >+ // >+ // - For an actual tail call, m_newFrameBase is computed >+ // dynamically, and m_newFrameOffset varies between 0 and -2 >+ // depending on the architecture's calling convention (see >+ // prepareForTailCall). >+ // >+ // - For a slow path call, m_newFrameBase is the actual stack >+ // pointer, and m_newFrameOffset is - CallerFrameAndPCSize, >+ // following the convention for a regular call. >+ GPRReg m_newFrameBase { InvalidGPRReg }; >+ int m_newFrameOffset { 0}; >+ >+ bool isUndecided() const >+ { >+ return m_newFrameBase == InvalidGPRReg; >+ } >+ >+ bool isSlowPath() const >+ { >+ return m_newFrameBase == MacroAssembler::stackPointerRegister; >+ } >+ >+ MacroAssembler::Address addressForNew(VirtualRegister reg) const >+ { >+ return MacroAssembler::Address(m_newFrameBase, >+ (m_newFrameOffset + reg.offset()) * sizeof(Register)); >+ } >+ >+ // We use a concept of "danger zone". The danger zone consists of >+ // all the writes in the new frame that could overlap with reads >+ // in the old frame. >+ // >+ // Because we could have a higher actual number of arguments than >+ // parameters, when preparing a tail call, we need to assume that >+ // writing to a slot on the new frame could overlap not only with >+ // the corresponding slot in the old frame, but also with any slot >+ // above it. Thus, the danger zone consists of all writes between >+ // the first write and what I call the "danger frontier": the >+ // highest slot in the old frame we still care about. Thus, the >+ // danger zone contains all the slots between the first slot of >+ // the new frame and the danger frontier. Because the danger >+ // frontier is related to the new frame, it is stored as a virtual >+ // register *in the new frame*. >+ VirtualRegister m_dangerFrontier; >+ >+ VirtualRegister dangerFrontier() const >+ { >+ ASSERT(!isUndecided()); >+ >+ return m_dangerFrontier; >+ } >+ >+ bool isDangerNew(VirtualRegister reg) const >+ { >+ ASSERT(!isUndecided() && isValidNew(reg)); >+ return reg <= dangerFrontier(); >+ } >+ >+ void updateDangerFrontier() >+ { >+ ASSERT(!isUndecided()); >+ >+ m_dangerFrontier = firstNew() - 1; >+ for (VirtualRegister reg = lastNew(); reg >= firstNew(); reg -= 1) { >+ if (!getNew(reg) || !isValidOld(newAsOld(reg)) || !getOld(newAsOld(reg))) >+ continue; >+ >+ m_dangerFrontier = reg; >+ if (verbose) >+ dataLog(" Danger frontier now at NEW ", m_dangerFrontier, "\n"); >+ break; >+ } >+ if (verbose) >+ dataLog(" All clear! Danger zone is empty.\n"); >+ } >+ >+ // A safe write is a write that never writes into the danger zone. >+ bool hasOnlySafeWrites(CachedRecovery& cachedRecovery) const >+ { >+ for (VirtualRegister target : cachedRecovery.targets()) { >+ if (isDangerNew(target)) >+ return false; >+ } >+ return true; >+ } >+ >+ // You must ensure that there is no dangerous writes before >+ // calling this function. >+ bool tryWrites(CachedRecovery&); >+ >+ // This function tries to ensure that there is no longer any >+ // possible safe write, i.e. all remaining writes are either to >+ // the danger zone or callee save restorations. >+ // >+ // It returns false if it was unable to perform some safe writes >+ // due to high register pressure. >+ bool performSafeWrites(); >+}; >+ >+} // namespace JSC >+ >+#endif // ENABLE(JIT) >+ >+#endif // CallFrameShuffler_h >diff --git a/Source/JavaScriptCore/jit/CallFrameShuffler32_64.cpp b/Source/JavaScriptCore/jit/CallFrameShuffler32_64.cpp >new file mode 100644 >index 0000000000000000000000000000000000000000..afa8e48c89100ad999a9de34f150c044c20a43d3 >--- /dev/null >+++ b/Source/JavaScriptCore/jit/CallFrameShuffler32_64.cpp >@@ -0,0 +1,331 @@ >+/* >+ * Copyright (C) 2015 Apple Inc. All rights reserved. >+ * >+ * Redistribution and use in source and binary forms, with or without >+ * modification, are permitted provided that the following conditions >+ * are met: >+ * 1. Redistributions of source code must retain the above copyright >+ * notice, this list of conditions and the following disclaimer. >+ * 2. Redistributions in binary form must reproduce the above copyright >+ * notice, this list of conditions and the following disclaimer in the >+ * documentation and/or other materials provided with the distribution. >+ * >+ * THIS SOFTWARE IS PROVIDED BY APPLE INC. ``AS IS'' AND ANY >+ * EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE >+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR >+ * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL APPLE INC. OR >+ * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, >+ * EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, >+ * PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR >+ * PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY >+ * OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT >+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE >+ * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. >+ */ >+ >+#include "config.h" >+#include "CallFrameShuffler.h" >+ >+#if ENABLE(JIT) && USE(JSVALUE32_64) >+ >+#include "CCallHelpers.h" >+#include "DataFormat.h" >+#include "JSCJSValue.h" >+ >+namespace JSC { >+ >+DataFormat CallFrameShuffler::emitStore(CachedRecovery& location, MacroAssembler::Address address) >+{ >+ ASSERT(!location.recovery().isInJSStack()); >+ >+ switch (location.recovery().technique()) { >+ case UnboxedInt32InGPR: >+ m_jit.store32(MacroAssembler::TrustedImm32(JSValue::Int32Tag), >+ address.withOffset(TagOffset)); >+ m_jit.store32(location.recovery().gpr(), address.withOffset(PayloadOffset)); >+ return DataFormatInt32; >+ case UnboxedCellInGPR: >+ m_jit.store32(MacroAssembler::TrustedImm32(JSValue::CellTag), >+ address.withOffset(TagOffset)); >+ m_jit.store32(location.recovery().gpr(), address.withOffset(PayloadOffset)); >+ return DataFormatCell; >+ case Constant: >+ m_jit.storeTrustedValue(location.recovery().constant(), address); >+ return DataFormatJS; >+ case InPair: >+ m_jit.storeValue(location.recovery().jsValueRegs(), address); >+ return DataFormatJS; >+ case UnboxedBooleanInGPR: >+ m_jit.store32(MacroAssembler::TrustedImm32(JSValue::BooleanTag), >+ address.withOffset(TagOffset)); >+ m_jit.store32(location.recovery().gpr(), address.withOffset(PayloadOffset)); >+ return DataFormatBoolean; >+ case InFPR: >+ case UnboxedDoubleInFPR: >+ m_jit.storeDouble(location.recovery().fpr(), address); >+ return DataFormatJS; >+ default: >+ RELEASE_ASSERT_NOT_REACHED(); >+ } >+} >+ >+void CallFrameShuffler::emitBox(CachedRecovery& location) >+{ >+ // Nothing to do, we're good! JSValues and doubles can be stored >+ // immediately, and other formats don't need any transformation - >+ // just storing a constant tag separately. >+ ASSERT(canBox(location)); >+} >+ >+void CallFrameShuffler::emitLoad(CachedRecovery& location) >+{ >+ if (!location.recovery().isInJSStack()) >+ return; >+ >+ if (verbose) >+ dataLog(" * Loading ", location.recovery(), " into "); >+ VirtualRegister reg { location.recovery().virtualRegister() }; >+ MacroAssembler::Address address { addressForOld(reg) }; >+ >+ bool tryFPR { true }; >+ JSValueRegs wantedJSValueRegs { location.wantedJSValueRegs() }; >+ if (wantedJSValueRegs) { >+ if (wantedJSValueRegs.payloadGPR() != InvalidGPRReg >+ && !m_registers[wantedJSValueRegs.payloadGPR()] >+ && !m_lockedRegisters.get(wantedJSValueRegs.payloadGPR())) >+ tryFPR = false; >+ if (wantedJSValueRegs.tagGPR() != InvalidGPRReg >+ && !m_registers[wantedJSValueRegs.tagGPR()] >+ && !m_lockedRegisters.get(wantedJSValueRegs.tagGPR())) >+ tryFPR = false; >+ } >+ >+ if (tryFPR && location.loadsIntoFPR()) { >+ FPRReg resultFPR = getFreeFPR(); >+ if (resultFPR != InvalidFPRReg) { >+ m_jit.loadDouble(address, resultFPR); >+ DataFormat dataFormat = DataFormatJS; >+ if (location.recovery().dataFormat() == DataFormatDouble) >+ dataFormat = DataFormatDouble; >+ updateRecovery(location, >+ ValueRecovery::inFPR(resultFPR, dataFormat)); >+ if (verbose) >+ dataLog(location.recovery(), "\n"); >+ if (reg == newAsOld(dangerFrontier())) >+ updateDangerFrontier(); >+ return; >+ } >+ } >+ >+ if (location.loadsIntoGPR()) { >+ GPRReg resultGPR { wantedJSValueRegs.payloadGPR() }; >+ if (resultGPR == InvalidGPRReg || m_registers[resultGPR] || m_lockedRegisters.get(resultGPR)) >+ resultGPR = getFreeGPR(); >+ ASSERT(resultGPR != InvalidGPRReg); >+ m_jit.loadPtr(address.withOffset(PayloadOffset), resultGPR); >+ updateRecovery(location, >+ ValueRecovery::inGPR(resultGPR, location.recovery().dataFormat())); >+ if (verbose) >+ dataLog(location.recovery(), "\n"); >+ if (reg == newAsOld(dangerFrontier())) >+ updateDangerFrontier(); >+ return; >+ } >+ >+ ASSERT(location.recovery().technique() == DisplacedInJSStack); >+ GPRReg payloadGPR { wantedJSValueRegs.payloadGPR() }; >+ GPRReg tagGPR { wantedJSValueRegs.tagGPR() }; >+ if (payloadGPR == InvalidGPRReg || m_registers[payloadGPR] || m_lockedRegisters.get(payloadGPR)) >+ payloadGPR = getFreeGPR(); >+ m_lockedRegisters.set(payloadGPR); >+ if (tagGPR == InvalidGPRReg || m_registers[tagGPR] || m_lockedRegisters.get(tagGPR)) >+ tagGPR = getFreeGPR(); >+ m_lockedRegisters.clear(payloadGPR); >+ ASSERT(payloadGPR != InvalidGPRReg && tagGPR != InvalidGPRReg && tagGPR != payloadGPR); >+ m_jit.loadPtr(address.withOffset(PayloadOffset), payloadGPR); >+ m_jit.loadPtr(address.withOffset(TagOffset), tagGPR); >+ updateRecovery(location, >+ ValueRecovery::inPair(tagGPR, payloadGPR)); >+ if (verbose) >+ dataLog(location.recovery(), "\n"); >+ if (reg == newAsOld(dangerFrontier())) >+ updateDangerFrontier(); >+} >+ >+bool CallFrameShuffler::canLoad(CachedRecovery& location) >+{ >+ if (!location.recovery().isInJSStack()) >+ return true; >+ >+ if (location.loadsIntoFPR() && getFreeFPR() != InvalidFPRReg) >+ return true; >+ >+ if (location.loadsIntoGPR() && getFreeGPR() != InvalidGPRReg) >+ return true; >+ >+ if (location.recovery().technique() == DisplacedInJSStack) { >+ GPRReg payloadGPR { getFreeGPR() }; >+ if (payloadGPR == InvalidGPRReg) >+ return false; >+ m_lockedRegisters.set(payloadGPR); >+ GPRReg tagGPR { getFreeGPR() }; >+ m_lockedRegisters.clear(payloadGPR); >+ return tagGPR != InvalidGPRReg; >+ } >+ >+ return false; >+} >+ >+// We prefer loading doubles and undetermined JSValues into FPRs >+// because it would otherwise use up two GPRs. >+bool CachedRecovery::loadsIntoFPR() const >+{ >+ switch (recovery().technique()) { >+ case DoubleDisplacedInJSStack: >+ case DisplacedInJSStack: >+ return true; >+ >+ default: >+ return false; >+ } >+} >+ >+// Integers, booleans and cells can be loaded into a single GPR >+bool CachedRecovery::loadsIntoGPR() const >+{ >+ switch (recovery().technique()) { >+ case Int32DisplacedInJSStack: >+ case BooleanDisplacedInJSStack: >+ case CellDisplacedInJSStack: >+ return true; >+ >+ default: >+ return false; >+ } >+} >+ >+void CallFrameShuffler::emitDisplace(CachedRecovery& location) >+{ >+ ASSERT(location.recovery().isInRegisters()); >+ JSValueRegs wantedJSValueRegs { location.wantedJSValueRegs() }; >+ ASSERT(wantedJSValueRegs); >+ >+ GPRReg wantedTagGPR { wantedJSValueRegs.tagGPR() }; >+ GPRReg wantedPayloadGPR { wantedJSValueRegs.payloadGPR() }; >+ >+ if (wantedTagGPR != InvalidGPRReg) { >+ ASSERT(!m_lockedRegisters.get(wantedTagGPR)); >+ if (CachedRecovery* currentTag { m_registers[wantedTagGPR] }) { >+ if (currentTag == &location) { >+ if (verbose) >+ dataLog(" + ", wantedTagGPR, " is OK\n"); >+ } else { >+ // This can never happen on 32bit platforms since we >+ // have at most one wanted JSValueRegs, for the >+ // callee, and no callee-save registers. >+ RELEASE_ASSERT_NOT_REACHED(); >+ } >+ } >+ } >+ >+ if (wantedPayloadGPR != InvalidGPRReg) { >+ ASSERT(!m_lockedRegisters.get(wantedPayloadGPR)); >+ if (CachedRecovery* currentPayload { m_registers[wantedPayloadGPR] }) { >+ if (currentPayload == &location) { >+ if (verbose) >+ dataLog(" + ", wantedPayloadGPR, " is OK\n"); >+ } else { >+ // See above >+ RELEASE_ASSERT_NOT_REACHED(); >+ } >+ } >+ } >+ >+ if (location.recovery().technique() == InPair >+ || location.recovery().isInGPR()) { >+ GPRReg payloadGPR; >+ if (location.recovery().technique() == InPair) >+ payloadGPR = location.recovery().payloadGPR(); >+ else >+ payloadGPR = location.recovery().gpr(); >+ >+ if (wantedPayloadGPR == InvalidGPRReg) >+ wantedPayloadGPR = payloadGPR; >+ >+ if (payloadGPR != wantedPayloadGPR) { >+ if (location.recovery().technique() == InPair >+ && wantedPayloadGPR == location.recovery().tagGPR()) { >+ if (verbose) >+ dataLog(" * Swapping ", payloadGPR, " and ", wantedPayloadGPR, "\n"); >+ m_jit.swap(payloadGPR, wantedPayloadGPR); >+ updateRecovery(location, >+ ValueRecovery::inPair(payloadGPR, wantedPayloadGPR)); >+ } else { >+ if (verbose) >+ dataLog(" * Moving ", payloadGPR, " into ", wantedPayloadGPR, "\n"); >+ m_jit.move(payloadGPR, wantedPayloadGPR); >+ if (location.recovery().technique() == InPair) { >+ updateRecovery(location, >+ ValueRecovery::inPair(location.recovery().tagGPR(), >+ wantedPayloadGPR)); >+ } else { >+ updateRecovery(location, >+ ValueRecovery::inGPR(wantedPayloadGPR, location.recovery().dataFormat())); >+ } >+ } >+ } >+ >+ if (wantedTagGPR == InvalidGPRReg) >+ wantedTagGPR = getFreeGPR(); >+ switch (location.recovery().dataFormat()) { >+ case DataFormatInt32: >+ if (verbose) >+ dataLog(" * Moving int32 tag into ", wantedTagGPR, "\n"); >+ m_jit.move(MacroAssembler::TrustedImm32(JSValue::Int32Tag), >+ wantedTagGPR); >+ break; >+ case DataFormatCell: >+ if (verbose) >+ dataLog(" * Moving cell tag into ", wantedTagGPR, "\n"); >+ m_jit.move(MacroAssembler::TrustedImm32(JSValue::CellTag), >+ wantedTagGPR); >+ break; >+ case DataFormatBoolean: >+ if (verbose) >+ dataLog(" * Moving boolean tag into ", wantedTagGPR, "\n"); >+ m_jit.move(MacroAssembler::TrustedImm32(JSValue::BooleanTag), >+ wantedTagGPR); >+ break; >+ case DataFormatJS: >+ ASSERT(wantedTagGPR != location.recovery().payloadGPR()); >+ if (wantedTagGPR != location.recovery().tagGPR()) { >+ if (verbose) >+ dataLog(" * Moving ", location.recovery().tagGPR(), " into ", wantedTagGPR, "\n"); >+ m_jit.move(location.recovery().tagGPR(), wantedTagGPR); >+ } >+ break; >+ >+ default: >+ RELEASE_ASSERT_NOT_REACHED(); >+ } >+ } else { >+ ASSERT(location.recovery().isInFPR()); >+ if (wantedTagGPR == InvalidGPRReg) { >+ ASSERT(wantedPayloadGPR != InvalidGPRReg); >+ m_lockedRegisters.set(wantedPayloadGPR); >+ wantedTagGPR = getFreeGPR(); >+ m_lockedRegisters.clear(wantedPayloadGPR); >+ } >+ if (wantedPayloadGPR == InvalidGPRReg) { >+ m_lockedRegisters.set(wantedTagGPR); >+ wantedPayloadGPR = getFreeGPR(); >+ m_lockedRegisters.clear(wantedTagGPR); >+ } >+ m_jit.boxDouble(location.recovery().fpr(), wantedTagGPR, wantedPayloadGPR); >+ } >+ updateRecovery(location, ValueRecovery::inPair(wantedTagGPR, wantedPayloadGPR)); >+} >+ >+} // namespace JSC >+ >+#endif // ENABLE(JIT) && USE(JSVALUE32_64) >diff --git a/Source/JavaScriptCore/jit/CallFrameShuffler64.cpp b/Source/JavaScriptCore/jit/CallFrameShuffler64.cpp >new file mode 100644 >index 0000000000000000000000000000000000000000..2668d7ddcf8adb5bd153f07c828d3be8c18c861c >--- /dev/null >+++ b/Source/JavaScriptCore/jit/CallFrameShuffler64.cpp >@@ -0,0 +1,341 @@ >+/* >+ * Copyright (C) 2015 Apple Inc. All rights reserved. >+ * >+ * Redistribution and use in source and binary forms, with or without >+ * modification, are permitted provided that the following conditions >+ * are met: >+ * 1. Redistributions of source code must retain the above copyright >+ * notice, this list of conditions and the following disclaimer. >+ * 2. Redistributions in binary form must reproduce the above copyright >+ * notice, this list of conditions and the following disclaimer in the >+ * documentation and/or other materials provided with the distribution. >+ * >+ * THIS SOFTWARE IS PROVIDED BY APPLE INC. ``AS IS'' AND ANY >+ * EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE >+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR >+ * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL APPLE INC. OR >+ * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, >+ * EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, >+ * PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR >+ * PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY >+ * OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT >+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE >+ * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. >+ */ >+ >+#include "config.h" >+#include "CallFrameShuffler.h" >+ >+#if ENABLE(JIT) && USE(JSVALUE64) >+ >+#include "CCallHelpers.h" >+#include "DataFormat.h" >+ >+namespace JSC { >+ >+DataFormat CallFrameShuffler::emitStore( >+ CachedRecovery& cachedRecovery, MacroAssembler::Address address) >+{ >+ ASSERT(!cachedRecovery.recovery().isInJSStack()); >+ >+ switch (cachedRecovery.recovery().technique()) { >+ case InGPR: >+ m_jit.storePtr(cachedRecovery.recovery().gpr(), address); >+ return DataFormatJS; >+ case UnboxedInt32InGPR: >+ m_jit.store32(cachedRecovery.recovery().gpr(), address.withOffset(PayloadOffset)); >+ return DataFormatInt32; >+ case UnboxedInt52InGPR: >+ m_jit.rshift64(MacroAssembler::TrustedImm32(JSValue::int52ShiftAmount), >+ cachedRecovery.recovery().gpr()); >+ FALLTHROUGH; >+ case UnboxedStrictInt52InGPR: >+ m_jit.storePtr(cachedRecovery.recovery().gpr(), address); >+ return DataFormatStrictInt52; >+ case UnboxedBooleanInGPR: >+ m_jit.storePtr(cachedRecovery.recovery().gpr(), address); >+ return DataFormatBoolean; >+ case UnboxedCellInGPR: >+ m_jit.storePtr(cachedRecovery.recovery().gpr(), address); >+ return DataFormatCell; >+ case UnboxedDoubleInFPR: >+ m_jit.storeDouble(cachedRecovery.recovery().fpr(), address); >+ return DataFormatDouble; >+ case InFPR: >+ m_jit.storeDouble(cachedRecovery.recovery().fpr(), address); >+ return DataFormatJS; >+ case Constant: >+ m_jit.storeTrustedValue(cachedRecovery.recovery().constant(), address); >+ return DataFormatJS; >+ default: >+ RELEASE_ASSERT_NOT_REACHED(); >+ } >+} >+ >+void CallFrameShuffler::emitBox(CachedRecovery& cachedRecovery) >+{ >+ ASSERT(canBox(cachedRecovery)); >+ if (cachedRecovery.recovery().isConstant()) >+ return; >+ >+ if (cachedRecovery.recovery().isInGPR()) { >+ switch (cachedRecovery.recovery().dataFormat()) { >+ case DataFormatInt32: >+ if (verbose) >+ dataLog(" * Boxing ", cachedRecovery.recovery()); >+ m_jit.zeroExtend32ToPtr( >+ cachedRecovery.recovery().gpr(), >+ cachedRecovery.recovery().gpr()); >+ // We have to do this the hard way. >+ m_jit.or64(MacroAssembler::TrustedImm64(TagTypeNumber), >+ cachedRecovery.recovery().gpr()); >+ cachedRecovery.setRecovery( >+ ValueRecovery::inGPR(cachedRecovery.recovery().gpr(), DataFormatJS)); >+ if (verbose) >+ dataLog(" into ", cachedRecovery.recovery(), "\n"); >+ return; >+ case DataFormatInt52: >+ if (verbose) >+ dataLog(" * Boxing ", cachedRecovery.recovery()); >+ m_jit.rshift64(MacroAssembler::TrustedImm32(JSValue::int52ShiftAmount), >+ cachedRecovery.recovery().gpr()); >+ cachedRecovery.setRecovery( >+ ValueRecovery::inGPR(cachedRecovery.recovery().gpr(), DataFormatStrictInt52)); >+ if (verbose) >+ dataLog(" into ", cachedRecovery.recovery(), "\n"); >+ FALLTHROUGH; >+ case DataFormatStrictInt52: { >+ if (verbose) >+ dataLog(" * Boxing ", cachedRecovery.recovery()); >+ FPRReg resultFPR = getFreeFPR(); >+ ASSERT(resultFPR != InvalidFPRReg); >+ m_jit.convertInt64ToDouble(cachedRecovery.recovery().gpr(), resultFPR); >+ updateRecovery(cachedRecovery, ValueRecovery::inFPR(resultFPR, DataFormatDouble)); >+ if (verbose) >+ dataLog(" into ", cachedRecovery.recovery(), "\n"); >+ break; >+ } >+ case DataFormatBoolean: >+ if (verbose) >+ dataLog(" * Boxing ", cachedRecovery.recovery()); >+ m_jit.add32(MacroAssembler::TrustedImm32(ValueFalse), >+ cachedRecovery.recovery().gpr()); >+ cachedRecovery.setRecovery( >+ ValueRecovery::inGPR(cachedRecovery.recovery().gpr(), DataFormatJS)); >+ if (verbose) >+ dataLog(" into ", cachedRecovery.recovery(), "\n"); >+ return; >+ default: >+ return; >+ } >+ } >+ >+ if (cachedRecovery.recovery().isInFPR()) { >+ if (cachedRecovery.recovery().dataFormat() == DataFormatDouble) { >+ if (verbose) >+ dataLog(" * Boxing ", cachedRecovery.recovery()); >+ GPRReg resultGPR = cachedRecovery.wantedJSValueRegs().gpr(); >+ if (resultGPR == InvalidGPRReg || m_registers[resultGPR]) >+ resultGPR = getFreeGPR(); >+ ASSERT(resultGPR != InvalidGPRReg); >+ m_jit.purifyNaN(cachedRecovery.recovery().fpr()); >+ m_jit.moveDoubleTo64(cachedRecovery.recovery().fpr(), resultGPR); >+ m_jit.sub64(MacroAssembler::TrustedImm64(TagTypeNumber), resultGPR); >+ updateRecovery(cachedRecovery, ValueRecovery::inGPR(resultGPR, DataFormatJS)); >+ if (verbose) >+ dataLog(" into ", cachedRecovery.recovery(), "\n"); >+ return; >+ } >+ ASSERT(cachedRecovery.recovery().dataFormat() == DataFormatJS); >+ return; >+ } >+ >+ RELEASE_ASSERT_NOT_REACHED(); >+} >+ >+void CallFrameShuffler::emitLoad(CachedRecovery& cachedRecovery) >+{ >+ if (!cachedRecovery.recovery().isInJSStack()) >+ return; >+ >+ if (verbose) >+ dataLog(" * Loading ", cachedRecovery.recovery(), " into "); >+ >+ VirtualRegister reg = cachedRecovery.recovery().virtualRegister(); >+ MacroAssembler::Address address { addressForOld(reg) }; >+ bool tryFPR { true }; >+ GPRReg resultGPR { cachedRecovery.wantedJSValueRegs().gpr() }; >+ >+ // If we want a GPR and it's available, that's better than loading >+ // into an FPR. >+ if (resultGPR != InvalidGPRReg && !m_registers[resultGPR] >+ && !m_lockedRegisters.get(resultGPR) && cachedRecovery.loadsIntoGPR()) >+ tryFPR = false; >+ >+ // Otherwise, we prefer loading into FPRs if possible >+ if (tryFPR && cachedRecovery.loadsIntoFPR()) { >+ FPRReg resultFPR = getFreeFPR(); >+ if (resultFPR != InvalidFPRReg) { >+ m_jit.loadDouble(address, resultFPR); >+ DataFormat dataFormat = DataFormatJS; >+ // We could be transforming a DataFormatCell into a >+ // DataFormatJS here - but that's OK. >+ if (cachedRecovery.recovery().dataFormat() == DataFormatDouble) >+ dataFormat = DataFormatDouble; >+ updateRecovery(cachedRecovery, >+ ValueRecovery::inFPR(resultFPR, dataFormat)); >+ if (verbose) >+ dataLog(cachedRecovery.recovery(), "\n"); >+ if (reg == newAsOld(dangerFrontier())) >+ updateDangerFrontier(); >+ return; >+ } >+ } >+ >+ ASSERT(cachedRecovery.loadsIntoGPR()); >+ if (resultGPR == InvalidGPRReg || m_registers[resultGPR] || m_lockedRegisters.get(resultGPR)) >+ resultGPR = getFreeGPR(); >+ ASSERT(resultGPR != InvalidGPRReg); >+ m_jit.loadPtr(address, resultGPR); >+ updateRecovery(cachedRecovery, >+ ValueRecovery::inGPR(resultGPR, cachedRecovery.recovery().dataFormat())); >+ if (verbose) >+ dataLog(cachedRecovery.recovery(), "\n"); >+ if (reg == newAsOld(dangerFrontier())) >+ updateDangerFrontier(); >+} >+ >+bool CallFrameShuffler::canLoad(CachedRecovery& cachedRecovery) >+{ >+ if (!cachedRecovery.recovery().isInJSStack()) >+ return true; >+ >+ ASSERT(cachedRecovery.loadsIntoFPR() || cachedRecovery.loadsIntoGPR()); >+ >+ if (cachedRecovery.loadsIntoFPR() && getFreeFPR() != InvalidFPRReg) >+ return true; >+ >+ if (cachedRecovery.loadsIntoGPR() && getFreeGPR() != InvalidGPRReg) >+ return true; >+ >+ return false; >+} >+ >+bool CachedRecovery::loadsIntoFPR() const >+{ >+ switch (recovery().technique()) { >+ case DoubleDisplacedInJSStack: >+ case DisplacedInJSStack: >+ case CellDisplacedInJSStack: >+ return true; >+ >+ default: >+ return false; >+ } >+} >+ >+bool CachedRecovery::loadsIntoGPR() const >+{ >+ switch (recovery().technique()) { >+ case Int32DisplacedInJSStack: >+ case Int52DisplacedInJSStack: >+ case StrictInt52DisplacedInJSStack: >+ case BooleanDisplacedInJSStack: >+ case DisplacedInJSStack: >+ case CellDisplacedInJSStack: >+ return true; >+ >+ default: >+ return false; >+ } >+} >+ >+void CallFrameShuffler::emitDisplace(CachedRecovery& cachedRecovery) >+{ >+ ASSERT(cachedRecovery.recovery().isInRegisters()); >+ GPRReg wantedGPR { cachedRecovery.wantedJSValueRegs().gpr() }; >+ ASSERT(wantedGPR != InvalidGPRReg); >+ ASSERT(!m_lockedRegisters.get(wantedGPR)); >+ >+ if (CachedRecovery* current = m_registers[wantedGPR]) { >+ if (current == &cachedRecovery) { >+ if (verbose) >+ dataLog(" + ", wantedGPR, " is OK\n"); >+ return; >+ } >+ // We could do a more complex thing by finding cycles >+ // etc. in that case. >+ // However, ending up in this situation will be super >+ // rare, and should actually be outright impossible for >+ // non-FTL tiers, since: >+ // (a) All doubles have been converted into JSValues with >+ // ValueRep nodes, so FPRs are initially free >+ // >+ // (b) The only recoveries with wanted registers are the >+ // callee (which always starts out in a register) and >+ // the callee-save registers >+ // >+ // (c) The callee-save registers are the first things we >+ // load (after the return PC), and they are loaded as JSValues >+ // >+ // (d) We prefer loading JSValues into FPRs if their >+ // wanted GPR is not available >+ // >+ // (e) If we end up spilling some registers with a >+ // target, we won't load them again before the very >+ // end of the algorithm >+ // >+ // Combined, this means that we will never load a recovery >+ // with a wanted GPR into any GPR other than its wanted >+ // GPR. The callee could however have been initially in >+ // one of the callee-save registers - but since the wanted >+ // GPR for the callee is always regT0, it will be the >+ // first one to be displaced, and we won't see it when >+ // handling any of the callee-save registers. >+ // >+ // Thus, the only way we could ever reach this path is in >+ // the FTL, when there is so much pressure that we >+ // absolutely need to load the callee-save registers into >+ // different GPRs initially but not enough pressure to >+ // then have to spill all of them. And even in that case, >+ // depending on the order in which LLVM saves the >+ // callee-saves, we will probably still be safe. Anyway, >+ // the couple extra move instructions compared to an >+ // efficient cycle-based algorithm are not going to hurt >+ // us. >+ GPRReg tempGPR = getFreeGPR(); >+ if (verbose) >+ dataLog(" * Moving ", wantedGPR, " into ", tempGPR, "\n"); >+ m_jit.move(wantedGPR, tempGPR); >+ updateRecovery(*current, >+ ValueRecovery::inGPR(tempGPR, current->recovery().dataFormat())); >+ } >+ ASSERT(!m_registers[wantedGPR]); >+ >+ if (cachedRecovery.recovery().isInGPR()) { >+ if (verbose) >+ dataLog(" * Moving ", cachedRecovery.recovery().gpr(), " into ", wantedGPR, "\n"); >+ m_jit.move(cachedRecovery.recovery().gpr(), wantedGPR); >+ updateRecovery(cachedRecovery, >+ ValueRecovery::inGPR(wantedGPR, cachedRecovery.recovery().dataFormat())); >+ } else { >+ ASSERT(cachedRecovery.recovery().isInFPR()); >+ if (cachedRecovery.recovery().dataFormat() == DataFormatDouble) { >+ // This will automatically pick the wanted GPR >+ emitBox(cachedRecovery); >+ } else { >+ if (verbose) >+ dataLog(" * Moving ", cachedRecovery.recovery().fpr(), " into ", wantedGPR, "\n"); >+ m_jit.moveDoubleTo64(cachedRecovery.recovery().fpr(), wantedGPR); >+ updateRecovery(cachedRecovery, >+ ValueRecovery::inGPR(wantedGPR, cachedRecovery.recovery().dataFormat())); >+ } >+ } >+ >+ ASSERT(m_registers[wantedGPR] == &cachedRecovery); >+ ASSERT(cachedRecovery.recovery().gpr() == wantedGPR); >+} >+ >+} // namespace JSC >+ >+#endif // ENABLE(JIT) && USE(JSVALUE64) >diff --git a/Source/JavaScriptCore/jit/JITCall.cpp b/Source/JavaScriptCore/jit/JITCall.cpp >index de379e8ad21bd3535dab65dc4a6792cc1d90dc58..342ac8c27b63e4f4fdd9ac912dd551acd982ec91 100644 >--- a/Source/JavaScriptCore/jit/JITCall.cpp >+++ b/Source/JavaScriptCore/jit/JITCall.cpp >@@ -29,6 +29,7 @@ > #if USE(JSVALUE64) > #include "JIT.h" > >+#include "CallFrameShuffler.h" > #include "CodeBlock.h" > #include "JITInlines.h" > #include "JSArray.h" >@@ -181,9 +182,6 @@ void JIT::compileOpCall(OpcodeID opcodeID, Instruction* instruction, unsigned ca > return; > } > >- if (opcodeID == op_tail_call || opcodeID == op_tail_call_varargs) >- emitRestoreCalleeSaves(); >- > DataLabelPtr addressOfLinkedFunctionCheck; > Jump slowCase = branchPtrWithPatch(NotEqual, regT0, addressOfLinkedFunctionCheck, TrustedImmPtr(0)); > addSlowCase(slowCase); >@@ -194,7 +192,28 @@ void JIT::compileOpCall(OpcodeID opcodeID, Instruction* instruction, unsigned ca > m_callCompilationInfo[callLinkInfoIndex].hotPathBegin = addressOfLinkedFunctionCheck; > m_callCompilationInfo[callLinkInfoIndex].callLinkInfo = info; > >- if (opcodeID == op_tail_call || opcodeID == op_tail_call_varargs) { >+ if (opcodeID == op_tail_call) { >+ CallFrameShuffleData shuffleData; >+ shuffleData.numLocals = >+ instruction[4].u.operand - sizeof(CallerFrameAndPC) / sizeof(Register); >+ shuffleData.args.resize(instruction[3].u.operand); >+ for (int i = 0; i < instruction[3].u.operand; ++i) { >+ shuffleData.args[i] = >+ ValueRecovery::displacedInJSStack( >+ virtualRegisterForArgument(i) - instruction[4].u.operand, >+ DataFormatJS); >+ } >+ shuffleData.callee = >+ ValueRecovery::inGPR(regT0, DataFormatJS); >+ shuffleData.setupCalleeSaveRegisters(m_codeBlock); >+ info->setFrameShuffleData(shuffleData); >+ CallFrameShuffler(*this, shuffleData).prepareForTailCall(); >+ m_callCompilationInfo[callLinkInfoIndex].hotPathOther = emitNakedTailCall(); >+ return; >+ } >+ >+ if (opcodeID == op_tail_call_varargs) { >+ emitRestoreCalleeSaves(); > prepareForTailCallSlow(); > m_callCompilationInfo[callLinkInfoIndex].hotPathOther = emitNakedTailCall(); > return; >@@ -219,6 +238,9 @@ void JIT::compileOpCallSlowCase(OpcodeID opcodeID, Instruction* instruction, Vec > > linkSlowCase(iter); > >+ if (opcodeID == op_tail_call || opcodeID == op_tail_call_varargs) >+ emitRestoreCalleeSaves(); >+ > move(TrustedImmPtr(m_callCompilationInfo[callLinkInfoIndex].callLinkInfo), regT2); > > m_callCompilationInfo[callLinkInfoIndex].callReturnLocation = emitNakedCall(m_vm->getCTIStub(linkCallThunkGenerator).code()); >diff --git a/Source/JavaScriptCore/jit/RegisterMap.h b/Source/JavaScriptCore/jit/RegisterMap.h >index 747e8bd22305365734bb6415796d59875e28030d..e82f994f6775969e8dc72cdbf67056e07737fe55 100644 >--- a/Source/JavaScriptCore/jit/RegisterMap.h >+++ b/Source/JavaScriptCore/jit/RegisterMap.h >@@ -28,6 +28,8 @@ > > #if ENABLE(JIT) > >+#include "FPRInfo.h" >+#include "GPRInfo.h" > #include "MacroAssembler.h" > #include "Reg.h" > >diff --git a/Source/JavaScriptCore/jit/Repatch.cpp b/Source/JavaScriptCore/jit/Repatch.cpp >index d21fcd3e3a0071dc88e9e88182ce0100e5b183c2..74d803776af8bc94e2618095548c765b4af0ec99 100644 >--- a/Source/JavaScriptCore/jit/Repatch.cpp >+++ b/Source/JavaScriptCore/jit/Repatch.cpp >@@ -31,6 +31,7 @@ > #include "AccessorCallJITStubRoutine.h" > #include "BinarySwitch.h" > #include "CCallHelpers.h" >+#include "CallFrameShuffler.h" > #include "DFGOperations.h" > #include "DFGSpeculativeJIT.h" > #include "FTLThunks.h" >@@ -1787,26 +1788,32 @@ void linkPolymorphicCall( > > CCallHelpers::JumpList slowPath; > >- ptrdiff_t offsetToFrame = -sizeof(CallerFrameAndPC); >- >- if (!ASSERT_DISABLED) { >- CCallHelpers::Jump okArgumentCount = stubJit.branch32( >- CCallHelpers::Below, CCallHelpers::Address(CCallHelpers::stackPointerRegister, static_cast<ptrdiff_t>(sizeof(Register) * JSStack::ArgumentCount) + offsetToFrame + PayloadOffset), CCallHelpers::TrustedImm32(10000000)); >- stubJit.abortWithReason(RepatchInsaneArgumentCount); >- okArgumentCount.link(&stubJit); >+ std::unique_ptr<CallFrameShuffler> frameShuffler; >+ if (callLinkInfo.frameShuffleData()) { >+ ASSERT(callLinkInfo.isTailCall()); >+ frameShuffler = std::make_unique<CallFrameShuffler>(stubJit, *callLinkInfo.frameShuffleData()); >+#if USE(JSVALUE32_64) >+ // We would have already checked that the callee is a cell, and we can >+ // use the additional register this buys us. >+ frameShuffler->assumeCalleeIsCell(); >+#endif >+ frameShuffler->lockGPR(calleeGPR); > } >- >- GPRReg scratch = AssemblyHelpers::selectScratchGPR(calleeGPR); > GPRReg comparisonValueGPR; > > if (isClosureCall) { >- // Verify that we have a function and stash the executable in scratch. >+ GPRReg scratchGPR; >+ if (frameShuffler) >+ scratchGPR = frameShuffler->acquireGPR(); >+ else >+ scratchGPR = AssemblyHelpers::selectScratchGPR(calleeGPR); >+ // Verify that we have a function and stash the executable in scratchGPR. > > #if USE(JSVALUE64) >- // We can safely clobber everything except the calleeGPR. We can't rely on tagMaskRegister >- // being set. So we do this the hard way. >- stubJit.move(MacroAssembler::TrustedImm64(TagMask), scratch); >- slowPath.append(stubJit.branchTest64(CCallHelpers::NonZero, calleeGPR, scratch)); >+ // We can't rely on tagMaskRegister being set, so we do this the hard >+ // way. >+ stubJit.move(MacroAssembler::TrustedImm64(TagMask), scratchGPR); >+ slowPath.append(stubJit.branchTest64(CCallHelpers::NonZero, calleeGPR, scratchGPR)); > #else > // We would have already checked that the callee is a cell. > #endif >@@ -1819,9 +1826,9 @@ void linkPolymorphicCall( > > stubJit.loadPtr( > CCallHelpers::Address(calleeGPR, JSFunction::offsetOfExecutable()), >- scratch); >+ scratchGPR); > >- comparisonValueGPR = scratch; >+ comparisonValueGPR = scratchGPR; > } else > comparisonValueGPR = calleeGPR; > >@@ -1863,8 +1870,13 @@ void linkPolymorphicCall( > caseValues[i] = newCaseValue; > } > >- GPRReg fastCountsBaseGPR = >- AssemblyHelpers::selectScratchGPR(calleeGPR, comparisonValueGPR, GPRInfo::regT3); >+ GPRReg fastCountsBaseGPR; >+ if (frameShuffler) >+ fastCountsBaseGPR = frameShuffler->acquireGPR(); >+ else { >+ fastCountsBaseGPR = >+ AssemblyHelpers::selectScratchGPR(calleeGPR, comparisonValueGPR, GPRInfo::regT3); >+ } > stubJit.move(CCallHelpers::TrustedImmPtr(fastCounts.get()), fastCountsBaseGPR); > > BinarySwitch binarySwitch(comparisonValueGPR, caseValues, BinarySwitch::IntPtr); >@@ -1884,7 +1896,10 @@ void linkPolymorphicCall( > CCallHelpers::TrustedImm32(1), > CCallHelpers::Address(fastCountsBaseGPR, caseIndex * sizeof(uint32_t))); > } >- if (callLinkInfo.isTailCall()) { >+ if (frameShuffler) { >+ CallFrameShuffler(stubJit, frameShuffler->snapshot()).prepareForTailCall(); >+ calls[caseIndex].call = stubJit.nearTailCall(); >+ } else if (callLinkInfo.isTailCall()) { > stubJit.emitRestoreCalleeSaves(); > stubJit.prepareForTailCallSlow(); > calls[caseIndex].call = stubJit.nearTailCall(); >@@ -1896,10 +1911,23 @@ void linkPolymorphicCall( > > slowPath.link(&stubJit); > binarySwitch.fallThrough().link(&stubJit); >- stubJit.move(calleeGPR, GPRInfo::regT0); >+ >+ if (frameShuffler) { >+ frameShuffler->releaseGPR(calleeGPR); >+ frameShuffler->releaseGPR(comparisonValueGPR); >+ frameShuffler->releaseGPR(fastCountsBaseGPR); > #if USE(JSVALUE32_64) >- stubJit.move(CCallHelpers::TrustedImm32(JSValue::CellTag), GPRInfo::regT1); >+ frameShuffler->setCalleeJSValueRegs(JSValueRegs(GPRInfo::regT1, GPRInfo::regT0)); >+#else >+ frameShuffler->setCalleeJSValueRegs(JSValueRegs(GPRInfo::regT0)); > #endif >+ frameShuffler->prepareForSlowPath(); >+ } else { >+ stubJit.move(calleeGPR, GPRInfo::regT0); >+#if USE(JSVALUE32_64) >+ stubJit.move(CCallHelpers::TrustedImm32(JSValue::CellTag), GPRInfo::regT1); >+#endif >+ } > stubJit.move(CCallHelpers::TrustedImmPtr(&callLinkInfo), GPRInfo::regT2); > stubJit.move(CCallHelpers::TrustedImmPtr(callLinkInfo.callReturnLocation().executableAddress()), GPRInfo::regT4); >
<b>You cannot view the attachment while viewing its details because your browser does not support IFRAMEs. <a href="attachment.cgi?id=259806">View the attachment on a separate page</a>.</b>
View Attachment As Diff
View Attachment As Raw
Flags:
msaboff
:
review+
Actions:
View
|
Formatted Diff
|
Diff
Attachments on
bug 147508
:
259665
|
259677
| 259806