Attachment 331959 Details for Bug 181559 – the patch

[patch] the patch

blah.patch (text/plain), 176.16 KB, created by Filip Pizlo on 2018-01-22 13:46:30 PST

(hide)

Description:

Filename:

MIME Type:

Creator: Filip Pizlo

Created: 2018-01-22 13:46:30 PST

Size: 176.16 KB

patch

obsolete

>Index: Source/JavaScriptCore/ChangeLog
>===================================================================
>--- Source/JavaScriptCore/ChangeLog	(revision 227273)
>+++ Source/JavaScriptCore/ChangeLog	(working copy)
>@@ -1,3 +1,230 @@
>+2018-01-20  Filip Pizlo  <fpizlo@apple.com>
>+
>+        JSC GC should support TLCs (thread local caches)
>+        https://bugs.webkit.org/show_bug.cgi?id=181559
>+
>+        Reviewed by NOBODY (OOPS!).
>+        
>+        This is a big step towards object distancing by site origin. This patch implements TLCs, or
>+        thread-local caches, which allow each thread to allocate from its own free lists. It also
>+        means that any given thread can context-switch TLCs. This will allow us to do separate
>+        allocation for separate site origins. Eventually, once we reshape how MarkedBlock looks, this
>+        will allow us to have a hard distancing constraint between objects from different origins.
>+        
>+        This is a definite slow-down on GC-heavy benchmarks, but by a small margin, and only on
>+        unusually heavy tests. For example, boyer and splay are both 3% regressed, but the Octane
>+        geomean is just fine.
>+
>+        * JavaScriptCore.xcodeproj/project.pbxproj:
>+        * Sources.txt:
>+        * b3/B3LowerToAir.cpp:
>+        * b3/B3PatchpointSpecial.cpp:
>+        (JSC::B3::PatchpointSpecial::admitsStack):
>+        * b3/B3StackmapSpecial.cpp:
>+        (JSC::B3::StackmapSpecial::forEachArgImpl):
>+        (JSC::B3::StackmapSpecial::isArgValidForRep):
>+        * b3/B3StackmapValue.cpp:
>+        (JSC::B3::StackmapValue::appendSomeRegisterWithClobber):
>+        * b3/B3StackmapValue.h:
>+        * b3/B3Validate.cpp:
>+        * b3/B3ValueRep.cpp:
>+        (JSC::B3::ValueRep::addUsedRegistersTo const):
>+        (JSC::B3::ValueRep::dump const):
>+        (WTF::printInternal):
>+        * b3/B3ValueRep.h:
>+        (JSC::B3::ValueRep::ValueRep):
>+        * bytecode/AccessCase.cpp:
>+        (JSC::AccessCase::generateImpl):
>+        * bytecode/ObjectAllocationProfile.h:
>+        (JSC::ObjectAllocationProfile::ObjectAllocationProfile):
>+        (JSC::ObjectAllocationProfile::clear):
>+        * bytecode/ObjectAllocationProfileInlines.h:
>+        (JSC::ObjectAllocationProfile::initializeProfile):
>+        * dfg/DFGSpeculativeJIT.cpp:
>+        (JSC::DFG::SpeculativeJIT::emitAllocateRawObject):
>+        (JSC::DFG::SpeculativeJIT::compileMakeRope):
>+        (JSC::DFG::SpeculativeJIT::compileAllocatePropertyStorage):
>+        (JSC::DFG::SpeculativeJIT::compileReallocatePropertyStorage):
>+        (JSC::DFG::SpeculativeJIT::compileCreateThis):
>+        (JSC::DFG::SpeculativeJIT::compileNewObject):
>+        * dfg/DFGSpeculativeJIT.h:
>+        (JSC::DFG::SpeculativeJIT::emitAllocateJSCell):
>+        (JSC::DFG::SpeculativeJIT::emitAllocateJSObject):
>+        * ftl/FTLAbstractHeapRepository.h:
>+        * ftl/FTLLowerDFGToB3.cpp:
>+        (JSC::FTL::DFG::LowerDFGToB3::compileMakeRope):
>+        (JSC::FTL::DFG::LowerDFGToB3::compileMaterializeNewObject):
>+        (JSC::FTL::DFG::LowerDFGToB3::allocatePropertyStorageWithSizeImpl):
>+        (JSC::FTL::DFG::LowerDFGToB3::allocateHeapCell):
>+        (JSC::FTL::DFG::LowerDFGToB3::allocateObject):
>+        (JSC::FTL::DFG::LowerDFGToB3::allocatorForSize):
>+        (JSC::FTL::DFG::LowerDFGToB3::allocateVariableSizedObject):
>+        (JSC::FTL::DFG::LowerDFGToB3::allocateVariableSizedCell):
>+        * heap/Allocator.cpp: Added.
>+        (JSC::Allocator::cellSize const):
>+        * heap/Allocator.h: Added.
>+        (JSC::Allocator::Allocator):
>+        (JSC::Allocator::offset const):
>+        (JSC::Allocator::operator== const):
>+        (JSC::Allocator::operator!= const):
>+        (JSC::Allocator::operator bool const):
>+        * heap/AllocatorInlines.h: Added.
>+        (JSC::Allocator::allocate const):
>+        (JSC::Allocator::tryAllocate const):
>+        * heap/BlockDirectory.cpp:
>+        (JSC::BlockDirectory::BlockDirectory):
>+        (JSC::BlockDirectory::findBlockForAllocation):
>+        (JSC::BlockDirectory::stopAllocating):
>+        (JSC::BlockDirectory::prepareForAllocation):
>+        (JSC::BlockDirectory::stopAllocatingForGood):
>+        (JSC::BlockDirectory::resumeAllocating):
>+        (JSC::BlockDirectory::endMarking):
>+        (JSC::BlockDirectory::isFreeListedCell):
>+        (JSC::BlockDirectory::didConsumeFreeList): Deleted.
>+        (JSC::BlockDirectory::tryAllocateWithoutCollecting): Deleted.
>+        (JSC::BlockDirectory::allocateIn): Deleted.
>+        (JSC::BlockDirectory::tryAllocateIn): Deleted.
>+        (JSC::BlockDirectory::doTestCollectionsIfNeeded): Deleted.
>+        (JSC::BlockDirectory::allocateSlowCase): Deleted.
>+        * heap/BlockDirectory.h:
>+        (JSC::BlockDirectory::cellKind const):
>+        (JSC::BlockDirectory::allocator const):
>+        (JSC::BlockDirectory::freeList const): Deleted.
>+        (JSC::BlockDirectory::offsetOfFreeList): Deleted.
>+        (JSC::BlockDirectory::offsetOfCellSize): Deleted.
>+        * heap/BlockDirectoryInlines.h:
>+        (JSC::BlockDirectory::isFreeListedCell const): Deleted.
>+        (JSC::BlockDirectory::allocate): Deleted.
>+        * heap/CompleteSubspace.cpp:
>+        (JSC::CompleteSubspace::CompleteSubspace):
>+        (JSC::CompleteSubspace::allocatorFor):
>+        (JSC::CompleteSubspace::allocate):
>+        (JSC::CompleteSubspace::allocateNonVirtual):
>+        (JSC::CompleteSubspace::allocatorForSlow):
>+        (JSC::CompleteSubspace::allocateSlow):
>+        (JSC::CompleteSubspace::tryAllocateSlow):
>+        * heap/CompleteSubspace.h:
>+        (JSC::CompleteSubspace::allocatorForSizeStep):
>+        (JSC::CompleteSubspace::allocatorForNonVirtual):
>+        * heap/FreeList.h:
>+        * heap/GCDeferralContext.h:
>+        * heap/Heap.cpp:
>+        (JSC::Heap::Heap):
>+        (JSC::Heap::lastChanceToFinalize):
>+        * heap/Heap.h:
>+        (JSC::Heap::threadLocalCacheLayout):
>+        * heap/IsoCellSet.h:
>+        * heap/IsoSubspace.cpp:
>+        (JSC::IsoSubspace::IsoSubspace):
>+        (JSC::IsoSubspace::allocatorFor):
>+        (JSC::IsoSubspace::allocate):
>+        (JSC::IsoSubspace::allocateNonVirtual):
>+        * heap/IsoSubspace.h:
>+        (JSC::IsoSubspace::allocatorForNonVirtual):
>+        * heap/LocalAllocator.cpp: Added.
>+        (JSC::LocalAllocator::LocalAllocator):
>+        (JSC::LocalAllocator::reset):
>+        (JSC::LocalAllocator::~LocalAllocator):
>+        (JSC::LocalAllocator::stopAllocating):
>+        (JSC::LocalAllocator::resumeAllocating):
>+        (JSC::LocalAllocator::prepareForAllocation):
>+        (JSC::LocalAllocator::stopAllocatingForGood):
>+        (JSC::LocalAllocator::allocateSlowCase):
>+        (JSC::LocalAllocator::didConsumeFreeList):
>+        (JSC::LocalAllocator::tryAllocateWithoutCollecting):
>+        (JSC::LocalAllocator::allocateIn):
>+        (JSC::LocalAllocator::tryAllocateIn):
>+        (JSC::LocalAllocator::doTestCollectionsIfNeeded):
>+        (JSC::LocalAllocator::isFreeListedCell const):
>+        * heap/LocalAllocator.h: Added.
>+        (JSC::LocalAllocator::offsetOfFreeList):
>+        (JSC::LocalAllocator::offsetOfCellSize):
>+        * heap/LocalAllocatorInlines.h: Added.
>+        (JSC::LocalAllocator::allocate):
>+        * heap/MarkedSpace.cpp:
>+        (JSC::MarkedSpace::stopAllocatingForGood):
>+        * heap/MarkedSpace.h:
>+        * heap/SlotVisitor.cpp:
>+        * heap/SlotVisitor.h:
>+        * heap/Subspace.h:
>+        * heap/ThreadLocalCache.cpp: Added.
>+        (JSC::ThreadLocalCache::create):
>+        (JSC::ThreadLocalCache::ThreadLocalCache):
>+        (JSC::ThreadLocalCache::~ThreadLocalCache):
>+        (JSC::ThreadLocalCache::allocateData):
>+        (JSC::ThreadLocalCache::destroyData):
>+        (JSC::ThreadLocalCache::installSlow):
>+        (JSC::ThreadLocalCache::installData):
>+        (JSC::ThreadLocalCache::allocatorSlow):
>+        (JSC::ThreadLocalCache::destructor):
>+        * heap/ThreadLocalCache.h: Added.
>+        (JSC::ThreadLocalCache::offsetOfSize):
>+        (JSC::ThreadLocalCache::offsetOfFirstAllocator):
>+        * heap/ThreadLocalCacheInlines.h: Added.
>+        (JSC::ThreadLocalCache::getImpl):
>+        (JSC::ThreadLocalCache::get):
>+        (JSC::ThreadLocalCache::install):
>+        (JSC::ThreadLocalCache::allocator):
>+        (JSC::ThreadLocalCache::tryGetAllocator):
>+        * heap/ThreadLocalCacheLayout.cpp: Added.
>+        (JSC::ThreadLocalCacheLayout::ThreadLocalCacheLayout):
>+        (JSC::ThreadLocalCacheLayout::~ThreadLocalCacheLayout):
>+        (JSC::ThreadLocalCacheLayout::allocateOffset):
>+        (JSC::ThreadLocalCacheLayout::snapshot):
>+        (JSC::ThreadLocalCacheLayout::directory):
>+        * heap/ThreadLocalCacheLayout.h: Added.
>+        * jit/AssemblyHelpers.cpp:
>+        (JSC::AssemblyHelpers::emitAllocateWithNonNullAllocator):
>+        (JSC::AssemblyHelpers::emitAllocate):
>+        (JSC::AssemblyHelpers::emitAllocateVariableSized):
>+        * jit/AssemblyHelpers.h:
>+        (JSC::AssemblyHelpers::vm):
>+        (JSC::AssemblyHelpers::emitAllocateJSCell):
>+        (JSC::AssemblyHelpers::emitAllocateJSObject):
>+        (JSC::AssemblyHelpers::emitAllocateJSObjectWithKnownSize):
>+        (JSC::AssemblyHelpers::emitAllocateWithNonNullAllocator): Deleted.
>+        (JSC::AssemblyHelpers::emitAllocate): Deleted.
>+        (JSC::AssemblyHelpers::emitAllocateVariableSized): Deleted.
>+        * jit/JITOpcodes.cpp:
>+        (JSC::JIT::emit_op_new_object):
>+        (JSC::JIT::emit_op_create_this):
>+        * jit/JITOpcodes32_64.cpp:
>+        (JSC::JIT::emit_op_new_object):
>+        (JSC::JIT::emit_op_create_this):
>+        * runtime/ButterflyInlines.h:
>+        (JSC::Butterfly::createUninitialized):
>+        (JSC::Butterfly::tryCreate):
>+        (JSC::Butterfly::growArrayRight):
>+        * runtime/DirectArguments.cpp:
>+        (JSC::DirectArguments::overrideThings):
>+        * runtime/GenericArgumentsInlines.h:
>+        (JSC::GenericArguments<Type>::initModifiedArgumentsDescriptor):
>+        * runtime/HashMapImpl.h:
>+        (JSC::HashMapBuffer::create):
>+        * runtime/JSArray.cpp:
>+        (JSC::JSArray::tryCreateUninitializedRestricted):
>+        (JSC::JSArray::unshiftCountSlowCase):
>+        * runtime/JSArray.h:
>+        (JSC::JSArray::tryCreate):
>+        * runtime/JSArrayBufferView.cpp:
>+        (JSC::JSArrayBufferView::ConstructionContext::ConstructionContext):
>+        * runtime/JSCellInlines.h:
>+        (JSC::tryAllocateCellHelper):
>+        * runtime/JSGlobalObject.cpp:
>+        (JSC::JSGlobalObject::JSGlobalObject):
>+        * runtime/JSGlobalObject.h:
>+        (JSC::JSGlobalObject::threadLocalCache const):
>+        * runtime/JSLock.cpp:
>+        (JSC::JSLock::didAcquireLock):
>+        * runtime/Options.h:
>+        * runtime/RegExpMatchesArray.h:
>+        (JSC::tryCreateUninitializedRegExpMatchesArray):
>+        * runtime/VM.cpp:
>+        (JSC::VM::VM):
>+        * runtime/VM.h:
>+        * runtime/VMEntryScope.cpp:
>+        (JSC::VMEntryScope::VMEntryScope):
>+
> 2018-01-20  Caio Lima  <ticaiolima@gmail.com>
> 
>         [JSC] NumberPrototype::extractRadixFromArgs incorrectly cast double to int32_t
>Index: Source/JavaScriptCore/Sources.txt
>===================================================================
>--- Source/JavaScriptCore/Sources.txt	(revision 227273)
>+++ Source/JavaScriptCore/Sources.txt	(working copy)
>@@ -466,6 +466,7 @@ ftl/FTLThunks.cpp
> ftl/FTLValueRange.cpp
> 
> heap/AlignedMemoryAllocator.cpp
>+heap/Allocator.cpp
> heap/BlockDirectory.cpp
> heap/CellAttributes.cpp
> heap/CellContainer.cpp
>@@ -500,6 +501,7 @@ heap/IsoCellSet.cpp
> heap/IsoSubspace.cpp
> heap/JITStubRoutineSet.cpp
> heap/LargeAllocation.cpp
>+heap/LocalAllocator.cpp
> heap/MachineStackMarker.cpp
> heap/MarkStack.cpp
> heap/MarkStackMergingConstraint.cpp
>@@ -518,6 +520,8 @@ heap/StopIfNecessaryTimer.cpp
> heap/Subspace.cpp
> heap/SynchronousStopTheWorldMutatorScheduler.cpp
> heap/Synchronousness.cpp
>+heap/ThreadLocalCache.cpp
>+heap/ThreadLocalCacheLayout.cpp
> heap/VisitRaceKey.cpp
> heap/Weak.cpp
> heap/WeakBlock.cpp
>Index: Source/JavaScriptCore/JavaScriptCore.xcodeproj/project.pbxproj
>===================================================================
>--- Source/JavaScriptCore/JavaScriptCore.xcodeproj/project.pbxproj	(revision 227273)
>+++ Source/JavaScriptCore/JavaScriptCore.xcodeproj/project.pbxproj	(working copy)
>@@ -389,6 +389,13 @@
> 		0F725CAA1C503DED00AD943A /* B3PureCSE.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F725CA61C503DED00AD943A /* B3PureCSE.h */; };
> 		0F725CB01C506D3B00AD943A /* B3FoldPathConstants.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F725CAE1C506D3B00AD943A /* B3FoldPathConstants.h */; };
> 		0F74B93B1F89614800B935D3 /* PrototypeKey.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F74B93A1F89614500B935D3 /* PrototypeKey.h */; settings = {ATTRIBUTES = (Private, ); }; };
>+		0F75A05E200D25F60038E2CF /* ThreadLocalCache.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F75A055200D25EF0038E2CF /* ThreadLocalCache.h */; settings = {ATTRIBUTES = (Private, ); }; };
>+		0F75A060200D260B0038E2CF /* LocalAllocatorInlines.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F75A05A200D25F00038E2CF /* LocalAllocatorInlines.h */; };
>+		0F75A061200D26180038E2CF /* LocalAllocator.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F75A057200D25F00038E2CF /* LocalAllocator.h */; settings = {ATTRIBUTES = (Private, ); }; };
>+		0F75A062200D261D0038E2CF /* AllocatorInlines.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F75A05D200D25F10038E2CF /* AllocatorInlines.h */; };
>+		0F75A063200D261F0038E2CF /* Allocator.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F75A054200D25EF0038E2CF /* Allocator.h */; settings = {ATTRIBUTES = (Private, ); }; };
>+		0F75A064200D26280038E2CF /* ThreadLocalCacheLayout.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F75A05C200D25F10038E2CF /* ThreadLocalCacheLayout.h */; settings = {ATTRIBUTES = (Private, ); }; };
>+		0F75A0662013E4F10038E2CF /* JITAllocator.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F75A0652013E4EF0038E2CF /* JITAllocator.h */; settings = {ATTRIBUTES = (Private, ); }; };
> 		0F766D2C15A8CC3A008F363E /* JITStubRoutineSet.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F766D2A15A8CC34008F363E /* JITStubRoutineSet.h */; settings = {ATTRIBUTES = (Private, ); }; };
> 		0F766D3015A8DCE2008F363E /* GCAwareJITStubRoutine.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F766D2E15A8DCDD008F363E /* GCAwareJITStubRoutine.h */; settings = {ATTRIBUTES = (Private, ); }; };
> 		0F766D3115AA8112008F363E /* JITStubRoutine.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F766D1C15A5028D008F363E /* JITStubRoutine.h */; settings = {ATTRIBUTES = (Private, ); }; };
>@@ -2419,6 +2426,17 @@
> 		0F725CAD1C506D3B00AD943A /* B3FoldPathConstants.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; name = B3FoldPathConstants.cpp; path = b3/B3FoldPathConstants.cpp; sourceTree = "<group>"; };
> 		0F725CAE1C506D3B00AD943A /* B3FoldPathConstants.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; name = B3FoldPathConstants.h; path = b3/B3FoldPathConstants.h; sourceTree = "<group>"; };
> 		0F74B93A1F89614500B935D3 /* PrototypeKey.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = PrototypeKey.h; sourceTree = "<group>"; };
>+		0F75A054200D25EF0038E2CF /* Allocator.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = Allocator.h; sourceTree = "<group>"; };
>+		0F75A055200D25EF0038E2CF /* ThreadLocalCache.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = ThreadLocalCache.h; sourceTree = "<group>"; };
>+		0F75A056200D25EF0038E2CF /* ThreadLocalCacheInlines.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = ThreadLocalCacheInlines.h; sourceTree = "<group>"; };
>+		0F75A057200D25F00038E2CF /* LocalAllocator.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = LocalAllocator.h; sourceTree = "<group>"; };
>+		0F75A058200D25F00038E2CF /* ThreadLocalCache.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = ThreadLocalCache.cpp; sourceTree = "<group>"; };
>+		0F75A059200D25F00038E2CF /* LocalAllocator.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = LocalAllocator.cpp; sourceTree = "<group>"; };
>+		0F75A05A200D25F00038E2CF /* LocalAllocatorInlines.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = LocalAllocatorInlines.h; sourceTree = "<group>"; };
>+		0F75A05B200D25F10038E2CF /* ThreadLocalCacheLayout.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = ThreadLocalCacheLayout.cpp; sourceTree = "<group>"; };
>+		0F75A05C200D25F10038E2CF /* ThreadLocalCacheLayout.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = ThreadLocalCacheLayout.h; sourceTree = "<group>"; };
>+		0F75A05D200D25F10038E2CF /* AllocatorInlines.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = AllocatorInlines.h; sourceTree = "<group>"; };
>+		0F75A0652013E4EF0038E2CF /* JITAllocator.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = JITAllocator.h; sourceTree = "<group>"; };
> 		0F766D1C15A5028D008F363E /* JITStubRoutine.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = JITStubRoutine.h; sourceTree = "<group>"; };
> 		0F766D2615A8CC1B008F363E /* JITStubRoutine.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = JITStubRoutine.cpp; sourceTree = "<group>"; };
> 		0F766D2915A8CC34008F363E /* JITStubRoutineSet.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = JITStubRoutineSet.cpp; sourceTree = "<group>"; };
>@@ -5447,6 +5465,7 @@
> 				1429D92E0ED22D7000B89619 /* JIT.h */,
> 				FE1220251BE7F5640039E6F2 /* JITAddGenerator.cpp */,
> 				FE1220261BE7F5640039E6F2 /* JITAddGenerator.h */,
>+				0F75A0652013E4EF0038E2CF /* JITAllocator.h */,
> 				86A90ECF0EE7D51F00AB350D /* JITArithmetic.cpp */,
> 				A75706DD118A2BCF0057F88F /* JITArithmetic32_64.cpp */,
> 				FE3A06AD1C10CB6F00390FDD /* JITBitAndGenerator.cpp */,
>@@ -5540,6 +5559,16 @@
> 		142E312A134FF0A600AFADB5 /* heap */ = {
> 			isa = PBXGroup;
> 			children = (
>+				0F75A054200D25EF0038E2CF /* Allocator.h */,
>+				0F75A05D200D25F10038E2CF /* AllocatorInlines.h */,
>+				0F75A059200D25F00038E2CF /* LocalAllocator.cpp */,
>+				0F75A057200D25F00038E2CF /* LocalAllocator.h */,
>+				0F75A05A200D25F00038E2CF /* LocalAllocatorInlines.h */,
>+				0F75A058200D25F00038E2CF /* ThreadLocalCache.cpp */,
>+				0F75A055200D25EF0038E2CF /* ThreadLocalCache.h */,
>+				0F75A056200D25EF0038E2CF /* ThreadLocalCacheInlines.h */,
>+				0F75A05B200D25F10038E2CF /* ThreadLocalCacheLayout.cpp */,
>+				0F75A05C200D25F10038E2CF /* ThreadLocalCacheLayout.h */,
> 				0FEC3C501F33A41600F59B6C /* AlignedMemoryAllocator.cpp */,
> 				0FEC3C511F33A41600F59B6C /* AlignedMemoryAllocator.h */,
> 				0FA7620A1DB959F600B7A2FD /* AllocatingScope.h */,
>@@ -8048,6 +8077,7 @@
> 				0F2AC5711E8EE4540001EE3F /* AirFormTable.h in Headers */,
> 				0FEC85771BDACDC70080FF74 /* AirFrequentedBlock.h in Headers */,
> 				0FEC85791BDACDC70080FF74 /* AirGenerate.h in Headers */,
>+				0F75A061200D26180038E2CF /* LocalAllocator.h in Headers */,
> 				0FEC857A1BDACDC70080FF74 /* AirGenerationContext.h in Headers */,
> 				0FEC857C1BDACDC70080FF74 /* AirHandleCalleeSaves.h in Headers */,
> 				0FEC857E1BDACDC70080FF74 /* AirInsertionSet.h in Headers */,
>@@ -8213,6 +8243,7 @@
> 				0F6B8ADD1C4EFAC300969052 /* B3SSACalculator.h in Headers */,
> 				0F33FCF81C136E2500323F67 /* B3StackmapGenerationParams.h in Headers */,
> 				0FEC85311BDACDAC0080FF74 /* B3StackmapSpecial.h in Headers */,
>+				0F75A064200D26280038E2CF /* ThreadLocalCacheLayout.h in Headers */,
> 				0F338DF21BE93AD10013C88F /* B3StackmapValue.h in Headers */,
> 				0F9495881C57F47500413A48 /* B3StackSlot.h in Headers */,
> 				0FEC85361BDACDAC0080FF74 /* B3SuccessorCollection.h in Headers */,
>@@ -8382,6 +8413,7 @@
> 				0F1E3A461534CBAF000F9456 /* DFGArgumentPosition.h in Headers */,
> 				0F2DD8121AB3D8BE00BBB8E8 /* DFGArgumentsEliminationPhase.h in Headers */,
> 				0F2DD8141AB3D8BE00BBB8E8 /* DFGArgumentsUtilities.h in Headers */,
>+				0F75A063200D261F0038E2CF /* Allocator.h in Headers */,
> 				0F485322187750560083B687 /* DFGArithMode.h in Headers */,
> 				0F05C3B41683CF9200BAF45B /* DFGArrayifySlowPathGenerator.h in Headers */,
> 				0F63948515E4811B006A597C /* DFGArrayMode.h in Headers */,
>@@ -8662,6 +8694,7 @@
> 				FEA0C4031CDD7D1D00481991 /* FunctionWhitelist.h in Headers */,
> 				2AACE63D18CA5A0300ED0191 /* GCActivityCallback.h in Headers */,
> 				BCBE2CAE14E985AA000593AD /* GCAssertions.h in Headers */,
>+				0F75A060200D260B0038E2CF /* LocalAllocatorInlines.h in Headers */,
> 				0F766D3015A8DCE2008F363E /* GCAwareJITStubRoutine.h in Headers */,
> 				0FD0E5EA1E43D34D0006AB08 /* GCConductor.h in Headers */,
> 				0FB4767E1D99AEA9008EA6CB /* GCDeferralContext.h in Headers */,
>@@ -8746,6 +8779,7 @@
> 				0FB7F39A15ED8E4600F167B2 /* IndexingHeaderInlines.h in Headers */,
> 				0FB7F39B15ED8E4600F167B2 /* IndexingType.h in Headers */,
> 				14386A791DD6989C008652C4 /* IndirectEvalExecutable.h in Headers */,
>+				0F75A062200D261D0038E2CF /* AllocatorInlines.h in Headers */,
> 				0FBF92B91FD76FFF00AC28A8 /* InferredStructure.h in Headers */,
> 				0FBF92BA1FD7700400AC28A8 /* InferredStructureWatchpoint.h in Headers */,
> 				0F0A75231B94BFA900110660 /* InferredType.h in Headers */,
>@@ -8931,6 +8965,7 @@
> 				E33F50791B84225700413856 /* JSInternalPromiseConstructor.h in Headers */,
> 				E33F50871B8449EF00413856 /* JSInternalPromiseConstructor.lut.h in Headers */,
> 				E33F50851B8437A000413856 /* JSInternalPromiseDeferred.h in Headers */,
>+				0F75A05E200D25F60038E2CF /* ThreadLocalCache.h in Headers */,
> 				E33F50751B8421C000413856 /* JSInternalPromisePrototype.h in Headers */,
> 				A503FA1E188E0FB000110F14 /* JSJavaScriptCallFramePrototype.h in Headers */,
> 				7013CA8C1B491A9400CAE613 /* JSJob.h in Headers */,
>@@ -9430,6 +9465,7 @@
> 				14E84FA214EE1ACC00D6D5D4 /* WeakImpl.h in Headers */,
> 				14BE7D3317135CF400D1807A /* WeakInlines.h in Headers */,
> 				A7CA3AE417DA41AE006538AF /* WeakMapConstructor.h in Headers */,
>+				0F75A0662013E4F10038E2CF /* JITAllocator.h in Headers */,
> 				E3A32BC71FC83147007D7E76 /* WeakMapImpl.h in Headers */,
> 				E393ADD81FE702D00022D681 /* WeakMapImplInlines.h in Headers */,
> 				A7CA3AE617DA41AE006538AF /* WeakMapPrototype.h in Headers */,
>Index: Source/JavaScriptCore/b3/B3LowerToAir.cpp
>===================================================================
>--- Source/JavaScriptCore/b3/B3LowerToAir.cpp	(revision 227273)
>+++ Source/JavaScriptCore/b3/B3LowerToAir.cpp	(working copy)
>@@ -1277,6 +1277,12 @@ private:
>             case ValueRep::SomeRegister:
>                 arg = tmp(value.value());
>                 break;
>+            case ValueRep::SomeRegisterWithClobber: {
>+                Tmp dstTmp = m_code.newTmp(value.value()->resultBank());
>+                append(relaxedMoveForType(value.value()->type()), immOrTmp(value.value()), dstTmp);
>+                arg = dstTmp;
>+                break;
>+            }
>             case ValueRep::LateRegister:
>             case ValueRep::Register:
>                 stackmap->earlyClobbered().clear(value.rep().reg());
>Index: Source/JavaScriptCore/b3/B3PatchpointSpecial.cpp
>===================================================================
>--- Source/JavaScriptCore/b3/B3PatchpointSpecial.cpp	(revision 227273)
>+++ Source/JavaScriptCore/b3/B3PatchpointSpecial.cpp	(working copy)
>@@ -118,6 +118,7 @@ bool PatchpointSpecial::admitsStack(Inst
>         case ValueRep::StackArgument:
>             return true;
>         case ValueRep::SomeRegister:
>+        case ValueRep::SomeRegisterWithClobber:
>         case ValueRep::SomeEarlyRegister:
>         case ValueRep::Register:
>         case ValueRep::LateRegister:
>Index: Source/JavaScriptCore/b3/B3StackmapSpecial.cpp
>===================================================================
>--- Source/JavaScriptCore/b3/B3StackmapSpecial.cpp	(revision 227273)
>+++ Source/JavaScriptCore/b3/B3StackmapSpecial.cpp	(working copy)
>@@ -1,5 +1,5 @@
> /*
>- * Copyright (C) 2015-2017 Apple Inc. All rights reserved.
>+ * Copyright (C) 2015-2018 Apple Inc. All rights reserved.
>  *
>  * Redistribution and use in source and binary forms, with or without
>  * modification, are permitted provided that the following conditions
>@@ -110,6 +110,9 @@ void StackmapSpecial::forEachArgImpl(
>             case ValueRep::Constant:
>                 role = Arg::Use;
>                 break;
>+            case ValueRep::SomeRegisterWithClobber:
>+                role = Arg::UseDef;
>+                break;
>             case ValueRep::LateRegister:
>                 role = Arg::LateUse;
>                 break;
>@@ -128,6 +131,10 @@ void StackmapSpecial::forEachArgImpl(
>             // be able to recover the stackmap value. So, force LateColdUse to preserve the
>             // original stackmap value across the Special operation.
>             if (!Arg::isLateUse(role) && optionalDefArgWidth && *optionalDefArgWidth < child.value()->resultWidth()) {
>+                // The role can only be some kind of def if we did SomeRegisterWithClobber, which is
>+                // only allowed for patchpoints. Patchpoints don't use the defArgWidth feature.
>+                RELEASE_ASSERT(!Arg::isAnyDef(role));
>+                
>                 if (Arg::isWarmUse(role))
>                     role = Arg::LateUse;
>                 else
>@@ -245,6 +252,7 @@ bool StackmapSpecial::isArgValidForRep(A
>         // We already verified by isArgValidForValue().
>         return true;
>     case ValueRep::SomeRegister:
>+    case ValueRep::SomeRegisterWithClobber:
>     case ValueRep::SomeEarlyRegister:
>         return arg.isTmp();
>     case ValueRep::LateRegister:
>Index: Source/JavaScriptCore/b3/B3StackmapValue.cpp
>===================================================================
>--- Source/JavaScriptCore/b3/B3StackmapValue.cpp	(revision 227273)
>+++ Source/JavaScriptCore/b3/B3StackmapValue.cpp	(working copy)
>@@ -1,5 +1,5 @@
> /*
>- * Copyright (C) 2015 Apple Inc. All rights reserved.
>+ * Copyright (C) 2015-2018 Apple Inc. All rights reserved.
>  *
>  * Redistribution and use in source and binary forms, with or without
>  * modification, are permitted provided that the following conditions
>@@ -53,6 +53,11 @@ void StackmapValue::appendSomeRegister(V
>     append(ConstrainedValue(value, ValueRep::SomeRegister));
> }
> 
>+void StackmapValue::appendSomeRegisterWithClobber(Value* value)
>+{
>+    append(ConstrainedValue(value, ValueRep::SomeRegisterWithClobber));
>+}
>+
> void StackmapValue::setConstrainedChild(unsigned index, const ConstrainedValue& constrainedValue)
> {
>     child(index) = constrainedValue.value();
>Index: Source/JavaScriptCore/b3/B3StackmapValue.h
>===================================================================
>--- Source/JavaScriptCore/b3/B3StackmapValue.h	(revision 227273)
>+++ Source/JavaScriptCore/b3/B3StackmapValue.h	(working copy)
>@@ -1,5 +1,5 @@
> /*
>- * Copyright (C) 2015-2016 Apple Inc. All rights reserved.
>+ * Copyright (C) 2015-2018 Apple Inc. All rights reserved.
>  *
>  * Redistribution and use in source and binary forms, with or without
>  * modification, are permitted provided that the following conditions
>@@ -99,7 +99,8 @@ public:
>     // This is a helper for something you might do a lot of: append a value that should be constrained
>     // to SomeRegister.
>     void appendSomeRegister(Value*);
>-
>+    void appendSomeRegisterWithClobber(Value*);
>+    
>     const Vector<ValueRep>& reps() const { return m_reps; }
> 
>     // Stackmaps allow you to specify that the operation may clobber some registers. Clobbering a register
>Index: Source/JavaScriptCore/b3/B3Validate.cpp
>===================================================================
>--- Source/JavaScriptCore/b3/B3Validate.cpp	(revision 227273)
>+++ Source/JavaScriptCore/b3/B3Validate.cpp	(working copy)
>@@ -432,21 +432,8 @@ public:
>                 VALIDATE(!value->kind().hasExtraBits(), ("At ", *value));
>                 if (value->type() == Void)
>                     VALIDATE(value->as<PatchpointValue>()->resultConstraint == ValueRep::WarmAny, ("At ", *value));
>-                else {
>-                    switch (value->as<PatchpointValue>()->resultConstraint.kind()) {
>-                    case ValueRep::WarmAny:
>-                    case ValueRep::SomeRegister:
>-                    case ValueRep::SomeEarlyRegister:
>-                    case ValueRep::Register:
>-                    case ValueRep::StackArgument:
>-                        break;
>-                    default:
>-                        VALIDATE(false, ("At ", *value));
>-                        break;
>-                    }
>-                    
>+                else
>                     validateStackmapConstraint(value, ConstrainedValue(value, value->as<PatchpointValue>()->resultConstraint), ConstraintRole::Def);
>-                }
>                 validateStackmap(value);
>                 break;
>             case CheckAdd:
>@@ -571,16 +558,24 @@ private:
>     {
>         switch (value.rep().kind()) {
>         case ValueRep::WarmAny:
>-        case ValueRep::ColdAny:
>-        case ValueRep::LateColdAny:
>         case ValueRep::SomeRegister:
>         case ValueRep::StackArgument:
>             break;
>+        case ValueRep::LateColdAny:
>+        case ValueRep::ColdAny:
>+            VALIDATE(role == ConstraintRole::Use, ("At ", *context, ": ", value));
>+            break;
>+        case ValueRep::SomeRegisterWithClobber:
>+            VALIDATE(role == ConstraintRole::Use, ("At ", *context, ": ", value));
>+            VALIDATE(context->as<PatchpointValue>(), ("At ", *context));
>+            break;
>         case ValueRep::SomeEarlyRegister:
>             VALIDATE(role == ConstraintRole::Def, ("At ", *context, ": ", value));
>             break;
>         case ValueRep::Register:
>         case ValueRep::LateRegister:
>+            if (value.rep().kind() == ValueRep::LateRegister)
>+                VALIDATE(role == ConstraintRole::Use, ("At ", *context, ": ", value));
>             if (value.rep().reg().isGPR())
>                 VALIDATE(isInt(value.value()->type()), ("At ", *context, ": ", value));
>             else
>Index: Source/JavaScriptCore/b3/B3ValueRep.cpp
>===================================================================
>--- Source/JavaScriptCore/b3/B3ValueRep.cpp	(revision 227273)
>+++ Source/JavaScriptCore/b3/B3ValueRep.cpp	(working copy)
>@@ -1,5 +1,5 @@
> /*
>- * Copyright (C) 2015-2016 Apple Inc. All rights reserved.
>+ * Copyright (C) 2015-2018 Apple Inc. All rights reserved.
>  *
>  * Redistribution and use in source and binary forms, with or without
>  * modification, are permitted provided that the following conditions
>@@ -40,6 +40,7 @@ void ValueRep::addUsedRegistersTo(Regist
>     case ColdAny:
>     case LateColdAny:
>     case SomeRegister:
>+    case SomeRegisterWithClobber:
>     case SomeEarlyRegister:
>     case Constant:
>         return;
>@@ -71,6 +72,7 @@ void ValueRep::dump(PrintStream& out) co
>     case ColdAny:
>     case LateColdAny:
>     case SomeRegister:
>+    case SomeRegisterWithClobber:
>     case SomeEarlyRegister:
>         return;
>     case LateRegister:
>@@ -175,6 +177,9 @@ void printInternal(PrintStream& out, Val
>     case ValueRep::SomeRegister:
>         out.print("SomeRegister");
>         return;
>+    case ValueRep::SomeRegisterWithClobber:
>+        out.print("SomeRegisterWithClobber");
>+        return;
>     case ValueRep::SomeEarlyRegister:
>         out.print("SomeEarlyRegister");
>         return;
>Index: Source/JavaScriptCore/b3/B3ValueRep.h
>===================================================================
>--- Source/JavaScriptCore/b3/B3ValueRep.h	(revision 227273)
>+++ Source/JavaScriptCore/b3/B3ValueRep.h	(working copy)
>@@ -1,5 +1,5 @@
> /*
>- * Copyright (C) 2015-2016 Apple Inc. All rights reserved.
>+ * Copyright (C) 2015-2018 Apple Inc. All rights reserved.
>  *
>  * Redistribution and use in source and binary forms, with or without
>  * modification, are permitted provided that the following conditions
>@@ -65,6 +65,10 @@ public:
>         // As an input representation, this means that B3 should pick some register. It could be a
>         // register that this claims to clobber!
>         SomeRegister,
>+        
>+        // As an input representation, this means that B3 should pick some register but that this
>+        // register is then cobbered with garbage. This only works for patchpoints.
>+        SomeRegisterWithClobber,
> 
>         // As an input representation, this tells us that B3 should pick some register, but implies
>         // that the def happens before any of the effects of the stackmap. This is only valid for
>@@ -107,7 +111,7 @@ public:
>     ValueRep(Kind kind)
>         : m_kind(kind)
>     {
>-        ASSERT(kind == WarmAny || kind == ColdAny || kind == LateColdAny || kind == SomeRegister || kind == SomeEarlyRegister);
>+        ASSERT(kind == WarmAny || kind == ColdAny || kind == LateColdAny || kind == SomeRegister || kind == SomeRegisterWithClobber || kind == SomeEarlyRegister);
>     }
> 
>     static ValueRep reg(Reg reg)
>Index: Source/JavaScriptCore/bytecode/AccessCase.cpp
>===================================================================
>--- Source/JavaScriptCore/bytecode/AccessCase.cpp	(revision 227273)
>+++ Source/JavaScriptCore/bytecode/AccessCase.cpp	(working copy)
>@@ -1,5 +1,5 @@
> /*
>- * Copyright (C) 2017 Apple Inc. All rights reserved.
>+ * Copyright (C) 2017-2018 Apple Inc. All rights reserved.
>  *
>  * Redistribution and use in source and binary forms, with or without
>  * modification, are permitted provided that the following conditions
>@@ -955,15 +955,9 @@ void AccessCase::generateImpl(AccessGene
>             size_t newSize = newStructure()->outOfLineCapacity() * sizeof(JSValue);
> 
>             if (allocatingInline) {
>-                BlockDirectory* allocator = vm.jsValueGigacageAuxiliarySpace.allocatorFor(newSize, AllocatorForMode::AllocatorIfExists);
>+                Allocator allocator = vm.jsValueGigacageAuxiliarySpace.allocatorFor(newSize, AllocatorForMode::AllocatorIfExists);
> 
>-                if (!allocator) {
>-                    // Yuck, this case would suck!
>-                    slowPath.append(jit.jump());
>-                }
>-
>-                jit.move(CCallHelpers::TrustedImmPtr(allocator), scratchGPR2);
>-                jit.emitAllocate(scratchGPR, allocator, scratchGPR2, scratchGPR3, slowPath);
>+                jit.emitAllocate(scratchGPR, JITAllocator::constant(allocator), scratchGPR2, scratchGPR3, slowPath);
>                 jit.addPtr(CCallHelpers::TrustedImm32(newSize + sizeof(IndexingHeader)), scratchGPR);
> 
>                 size_t oldSize = structure()->outOfLineCapacity() * sizeof(JSValue);
>Index: Source/JavaScriptCore/bytecode/ObjectAllocationProfile.h
>===================================================================
>--- Source/JavaScriptCore/bytecode/ObjectAllocationProfile.h	(revision 227273)
>+++ Source/JavaScriptCore/bytecode/ObjectAllocationProfile.h	(working copy)
>@@ -43,8 +43,7 @@ public:
>     static ptrdiff_t offsetOfInlineCapacity() { return OBJECT_OFFSETOF(ObjectAllocationProfile, m_inlineCapacity); }
> 
>     ObjectAllocationProfile()
>-        : m_allocator(0)
>-        , m_inlineCapacity(0)
>+        : m_inlineCapacity(0)
>     {
>     }
> 
>@@ -63,7 +62,7 @@ public:
> 
>     void clear()
>     {
>-        m_allocator = nullptr;
>+        m_allocator = Allocator();
>         m_structure.clear();
>         m_inlineCapacity = 0;
>         ASSERT(isNull());
>@@ -77,7 +76,7 @@ public:
> private:
>     unsigned possibleDefaultPropertyCount(VM&, JSObject* prototype);
> 
>-    BlockDirectory* m_allocator; // Precomputed to make things easier for generated code.
>+    Allocator m_allocator; // Precomputed to make things easier for generated code.
>     WriteBarrier<Structure> m_structure;
>     unsigned m_inlineCapacity;
> };
>Index: Source/JavaScriptCore/bytecode/ObjectAllocationProfileInlines.h
>===================================================================
>--- Source/JavaScriptCore/bytecode/ObjectAllocationProfileInlines.h	(revision 227273)
>+++ Source/JavaScriptCore/bytecode/ObjectAllocationProfileInlines.h	(working copy)
>@@ -53,7 +53,7 @@ ALWAYS_INLINE void ObjectAllocationProfi
> 
>         if (Structure* structure = executable->cachedPolyProtoStructure()) {
>             RELEASE_ASSERT(structure->typeInfo().type() == FinalObjectType);
>-            m_allocator = nullptr;
>+            m_allocator = Allocator();
>             m_structure.set(vm, owner, structure);
>             m_inlineCapacity = structure->inlineCapacity();
>             return;
>@@ -99,11 +99,11 @@ ALWAYS_INLINE void ObjectAllocationProfi
>     ASSERT(inlineCapacity <= JSFinalObject::maxInlineCapacity());
> 
>     size_t allocationSize = JSFinalObject::allocationSize(inlineCapacity);
>-    BlockDirectory* allocator = vm.cellSpace.allocatorForNonVirtual(allocationSize, AllocatorForMode::EnsureAllocator);
>+    Allocator allocator = vm.cellSpace.allocatorForNonVirtual(allocationSize, AllocatorForMode::EnsureAllocator);
> 
>     // Take advantage of extra inline capacity available in the size class.
>     if (allocator) {
>-        size_t slop = (allocator->cellSize() - allocationSize) / sizeof(WriteBarrier<Unknown>);
>+        size_t slop = (allocator.cellSize(vm.heap) - allocationSize) / sizeof(WriteBarrier<Unknown>);
>         inlineCapacity += slop;
>         if (inlineCapacity > JSFinalObject::maxInlineCapacity())
>             inlineCapacity = JSFinalObject::maxInlineCapacity();
>@@ -113,7 +113,7 @@ ALWAYS_INLINE void ObjectAllocationProfi
> 
>     if (isPolyProto) {
>         ASSERT(structure->hasPolyProto());
>-        m_allocator = nullptr;
>+        m_allocator = Allocator();
>         executable->setCachedPolyProtoStructure(vm, structure);
>     } else {
>         if (executable) {
>Index: Source/JavaScriptCore/dfg/DFGSpeculativeJIT.cpp
>===================================================================
>--- Source/JavaScriptCore/dfg/DFGSpeculativeJIT.cpp	(revision 227273)
>+++ Source/JavaScriptCore/dfg/DFGSpeculativeJIT.cpp	(working copy)
>@@ -113,9 +113,8 @@ void SpeculativeJIT::emitAllocateRawObje
>     m_jit.move(TrustedImmPtr(0), storageGPR);
> 
>     if (size) {
>-        if (BlockDirectory* allocator = m_jit.vm()->jsValueGigacageAuxiliarySpace.allocatorForNonVirtual(size, AllocatorForMode::AllocatorIfExists)) {
>-            m_jit.move(TrustedImmPtr(allocator), scratchGPR);
>-            m_jit.emitAllocate(storageGPR, allocator, scratchGPR, scratch2GPR, slowCases);
>+        if (Allocator allocator = m_jit.vm()->jsValueGigacageAuxiliarySpace.allocatorForNonVirtual(size, AllocatorForMode::AllocatorIfExists)) {
>+            m_jit.emitAllocate(storageGPR, JITAllocator::constant(allocator), scratchGPR, scratch2GPR, slowCases);
>             
>             m_jit.addPtr(
>                 TrustedImm32(outOfLineCapacity * sizeof(JSValue) + sizeof(IndexingHeader)),
>@@ -128,11 +127,10 @@ void SpeculativeJIT::emitAllocateRawObje
>     }
> 
>     size_t allocationSize = JSFinalObject::allocationSize(inlineCapacity);
>-    BlockDirectory* allocatorPtr = subspaceFor<JSFinalObject>(*m_jit.vm())->allocatorForNonVirtual(allocationSize, AllocatorForMode::AllocatorIfExists);
>-    if (allocatorPtr) {
>-        m_jit.move(TrustedImmPtr(allocatorPtr), scratchGPR);
>+    Allocator allocator = subspaceFor<JSFinalObject>(*m_jit.vm())->allocatorForNonVirtual(allocationSize, AllocatorForMode::AllocatorIfExists);
>+    if (allocator) {
>         uint32_t mask = WTF::computeIndexingMask(vectorLength);
>-        emitAllocateJSObject(resultGPR, allocatorPtr, scratchGPR, TrustedImmPtr(structure), storageGPR, TrustedImm32(mask), scratch2GPR, slowCases);
>+        emitAllocateJSObject(resultGPR, JITAllocator::constant(allocator), scratchGPR, TrustedImmPtr(structure), storageGPR, TrustedImm32(mask), scratch2GPR, slowCases);
>         m_jit.emitInitializeInlineStorage(resultGPR, structure->inlineCapacity());
>     } else
>         slowCases.append(m_jit.jump());
>@@ -4208,9 +4206,8 @@ void SpeculativeJIT::compileMakeRope(Nod
>     GPRReg scratchGPR = scratch.gpr();
>     
>     JITCompiler::JumpList slowPath;
>-    BlockDirectory* blockDirectory = subspaceFor<JSRopeString>(*m_jit.vm())->allocatorForNonVirtual(sizeof(JSRopeString), AllocatorForMode::AllocatorIfExists);
>-    m_jit.move(TrustedImmPtr(blockDirectory), allocatorGPR);
>-    emitAllocateJSCell(resultGPR, blockDirectory, allocatorGPR, TrustedImmPtr(m_jit.graph().registerStructure(m_jit.vm()->stringStructure.get())), scratchGPR, slowPath);
>+    Allocator allocatorValue = subspaceFor<JSRopeString>(*m_jit.vm())->allocatorForNonVirtual(sizeof(JSRopeString), AllocatorForMode::AllocatorIfExists);
>+    emitAllocateJSCell(resultGPR, JITAllocator::constant(allocatorValue), allocatorGPR, TrustedImmPtr(m_jit.graph().registerStructure(m_jit.vm()->stringStructure.get())), scratchGPR, slowPath);
>         
>     m_jit.storePtr(TrustedImmPtr(0), JITCompiler::Address(resultGPR, JSString::offsetOfValue()));
>     for (unsigned i = 0; i < numOpGPRs; ++i)
>@@ -8383,7 +8380,7 @@ void SpeculativeJIT::compileAllocateProp
>     
>     size_t size = initialOutOfLineCapacity * sizeof(JSValue);
> 
>-    BlockDirectory* allocator = m_jit.vm()->jsValueGigacageAuxiliarySpace.allocatorForNonVirtual(size, AllocatorForMode::AllocatorIfExists);
>+    Allocator allocator = m_jit.vm()->jsValueGigacageAuxiliarySpace.allocatorForNonVirtual(size, AllocatorForMode::AllocatorIfExists);
> 
>     if (!allocator || node->transition()->previous->couldHaveIndexingHeader()) {
>         SpeculateCellOperand base(this, node->child1());
>@@ -8408,9 +8405,8 @@ void SpeculativeJIT::compileAllocateProp
>     GPRReg scratchGPR2 = scratch2.gpr();
>     GPRReg scratchGPR3 = scratch3.gpr();
>         
>-    m_jit.move(TrustedImmPtr(allocator), scratchGPR2);
>     JITCompiler::JumpList slowPath;
>-    m_jit.emitAllocate(scratchGPR1, allocator, scratchGPR2, scratchGPR3, slowPath);
>+    m_jit.emitAllocate(scratchGPR1, JITAllocator::constant(allocator), scratchGPR2, scratchGPR3, slowPath);
>     m_jit.addPtr(JITCompiler::TrustedImm32(size + sizeof(IndexingHeader)), scratchGPR1);
>     
>     for (ptrdiff_t offset = 0; offset < static_cast<ptrdiff_t>(size); offset += sizeof(void*))
>@@ -8428,7 +8424,7 @@ void SpeculativeJIT::compileReallocatePr
>     size_t newSize = oldSize * outOfLineGrowthFactor;
>     ASSERT(newSize == node->transition()->next->outOfLineCapacity() * sizeof(JSValue));
>     
>-    BlockDirectory* allocator = m_jit.vm()->jsValueGigacageAuxiliarySpace.allocatorForNonVirtual(newSize, AllocatorForMode::AllocatorIfExists);
>+    Allocator allocator = m_jit.vm()->jsValueGigacageAuxiliarySpace.allocatorForNonVirtual(newSize, AllocatorForMode::AllocatorIfExists);
> 
>     if (!allocator || node->transition()->previous->couldHaveIndexingHeader()) {
>         SpeculateCellOperand base(this, node->child1());
>@@ -8456,8 +8452,7 @@ void SpeculativeJIT::compileReallocatePr
>     GPRReg scratchGPR3 = scratch3.gpr();
>     
>     JITCompiler::JumpList slowPath;
>-    m_jit.move(TrustedImmPtr(allocator), scratchGPR2);
>-    m_jit.emitAllocate(scratchGPR1, allocator, scratchGPR2, scratchGPR3, slowPath);
>+    m_jit.emitAllocate(scratchGPR1, JITAllocator::constant(allocator), scratchGPR2, scratchGPR3, slowPath);
>     
>     m_jit.addPtr(JITCompiler::TrustedImm32(newSize + sizeof(IndexingHeader)), scratchGPR1);
>         
>@@ -11441,14 +11436,14 @@ void SpeculativeJIT::compileCreateThis(N
>         JITCompiler::Address(calleeGPR, JSCell::typeInfoTypeOffset()), TrustedImm32(JSFunctionType)));
>     m_jit.loadPtr(JITCompiler::Address(calleeGPR, JSFunction::offsetOfRareData()), rareDataGPR);
>     slowPath.append(m_jit.branchTestPtr(MacroAssembler::Zero, rareDataGPR));
>-    m_jit.loadPtr(JITCompiler::Address(rareDataGPR, FunctionRareData::offsetOfObjectAllocationProfile() + ObjectAllocationProfile::offsetOfAllocator()), allocatorGPR);
>+    m_jit.load32(JITCompiler::Address(rareDataGPR, FunctionRareData::offsetOfObjectAllocationProfile() + ObjectAllocationProfile::offsetOfAllocator()), allocatorGPR);
>     m_jit.loadPtr(JITCompiler::Address(rareDataGPR, FunctionRareData::offsetOfObjectAllocationProfile() + ObjectAllocationProfile::offsetOfStructure()), structureGPR);
> 
>-    slowPath.append(m_jit.branchTestPtr(MacroAssembler::Zero, allocatorGPR));
>+    slowPath.append(m_jit.branch32(MacroAssembler::Equal, allocatorGPR, TrustedImm32(Allocator().offset())));
> 
>     auto butterfly = TrustedImmPtr(nullptr);
>     auto mask = TrustedImm32(0);
>-    emitAllocateJSObject(resultGPR, nullptr, allocatorGPR, structureGPR, butterfly, mask, scratchGPR, slowPath);
>+    emitAllocateJSObject(resultGPR, JITAllocator::variable(), allocatorGPR, structureGPR, butterfly, mask, scratchGPR, slowPath);
> 
>     m_jit.loadPtr(JITCompiler::Address(calleeGPR, JSFunction::offsetOfRareData()), rareDataGPR);
>     m_jit.load32(JITCompiler::Address(rareDataGPR, FunctionRareData::offsetOfObjectAllocationProfile() + ObjectAllocationProfile::offsetOfInlineCapacity()), inlineCapacityGPR);
>@@ -11474,14 +11469,17 @@ void SpeculativeJIT::compileNewObject(No
> 
>     RegisteredStructure structure = node->structure();
>     size_t allocationSize = JSFinalObject::allocationSize(structure->inlineCapacity());
>-    BlockDirectory* allocatorPtr = subspaceFor<JSFinalObject>(*m_jit.vm())->allocatorForNonVirtual(allocationSize, AllocatorForMode::AllocatorIfExists);
>+    Allocator allocatorValue = subspaceFor<JSFinalObject>(*m_jit.vm())->allocatorForNonVirtual(allocationSize, AllocatorForMode::AllocatorIfExists);
> 
>-    m_jit.move(TrustedImmPtr(allocatorPtr), allocatorGPR);
>-    auto butterfly = TrustedImmPtr(nullptr);
>-    auto mask = TrustedImm32(0);
>-    emitAllocateJSObject(resultGPR, allocatorPtr, allocatorGPR, TrustedImmPtr(structure), butterfly, mask, scratchGPR, slowPath);
>-    m_jit.emitInitializeInlineStorage(resultGPR, structure->inlineCapacity());
>-    m_jit.mutatorFence(*m_jit.vm());
>+    if (!allocatorValue)
>+        slowPath.append(m_jit.jump());
>+    else {
>+        auto butterfly = TrustedImmPtr(nullptr);
>+        auto mask = TrustedImm32(0);
>+        emitAllocateJSObject(resultGPR, JITAllocator::constant(allocatorValue), allocatorGPR, TrustedImmPtr(structure), butterfly, mask, scratchGPR, slowPath);
>+        m_jit.emitInitializeInlineStorage(resultGPR, structure->inlineCapacity());
>+        m_jit.mutatorFence(*m_jit.vm());
>+    }
> 
>     addSlowPathGenerator(slowPathCall(slowPath, this, operationNewObject, resultGPR, structure));
> 
>Index: Source/JavaScriptCore/dfg/DFGSpeculativeJIT.h
>===================================================================
>--- Source/JavaScriptCore/dfg/DFGSpeculativeJIT.h	(revision 227273)
>+++ Source/JavaScriptCore/dfg/DFGSpeculativeJIT.h	(working copy)
>@@ -1,5 +1,5 @@
> /*
>- * Copyright (C) 2011-2017 Apple Inc. All rights reserved.
>+ * Copyright (C) 2011-2018 Apple Inc. All rights reserved.
>  *
>  * Redistribution and use in source and binary forms, with or without
>  * modification, are permitted provided that the following conditions
>@@ -3141,7 +3141,7 @@ public:
>     // Allocator for a cell of a specific size.
>     template <typename StructureType> // StructureType can be GPR or ImmPtr.
>     void emitAllocateJSCell(
>-        GPRReg resultGPR, BlockDirectory* allocator, GPRReg allocatorGPR, StructureType structure,
>+        GPRReg resultGPR, const JITAllocator& allocator, GPRReg allocatorGPR, StructureType structure,
>         GPRReg scratchGPR, MacroAssembler::JumpList& slowPath)
>     {
>         m_jit.emitAllocateJSCell(resultGPR, allocator, allocatorGPR, structure, scratchGPR, slowPath);
>@@ -3150,7 +3150,7 @@ public:
>     // Allocator for an object of a specific size.
>     template <typename StructureType, typename StorageType, typename MaskType> // StructureType, StorageType and, MaskType can be GPR or ImmPtr.
>     void emitAllocateJSObject(
>-        GPRReg resultGPR, BlockDirectory* allocator, GPRReg allocatorGPR, StructureType structure,
>+        GPRReg resultGPR, const JITAllocator& allocator, GPRReg allocatorGPR, StructureType structure,
>         StorageType storage, MaskType mask, GPRReg scratchGPR, MacroAssembler::JumpList& slowPath)
>     {
>         m_jit.emitAllocateJSObject(
>Index: Source/JavaScriptCore/ftl/FTLAbstractHeapRepository.h
>===================================================================
>--- Source/JavaScriptCore/ftl/FTLAbstractHeapRepository.h	(revision 227273)
>+++ Source/JavaScriptCore/ftl/FTLAbstractHeapRepository.h	(working copy)
>@@ -1,5 +1,5 @@
> /*
>- * Copyright (C) 2013-2017 Apple Inc. All rights reserved.
>+ * Copyright (C) 2013-2018 Apple Inc. All rights reserved.
>  *
>  * Redistribution and use in source and binary forms, with or without
>  * modification, are permitted provided that the following conditions
>@@ -131,7 +131,7 @@ namespace JSC { namespace FTL {
> 
> #define FOR_EACH_INDEXED_ABSTRACT_HEAP(macro) \
>     macro(ArrayStorage_vector, ArrayStorage::vectorOffset(), sizeof(WriteBarrier<Unknown>)) \
>-    macro(CompleteSubspace_allocatorForSizeStep, CompleteSubspace::offsetOfAllocatorForSizeStep(), sizeof(BlockDirectory*)) \
>+    macro(CompleteSubspace_allocatorForSizeStep, CompleteSubspace::offsetOfAllocatorForSizeStep(), sizeof(Allocator)) \
>     macro(DirectArguments_storage, DirectArguments::storageOffset(), sizeof(EncodedJSValue)) \
>     macro(JSLexicalEnvironment_variables, JSLexicalEnvironment::offsetOfVariables(), sizeof(EncodedJSValue)) \
>     macro(JSPropertyNameEnumerator_cachedPropertyNamesVectorContents, 0, sizeof(WriteBarrier<JSString>)) \
>Index: Source/JavaScriptCore/ftl/FTLLowerDFGToB3.cpp
>===================================================================
>--- Source/JavaScriptCore/ftl/FTLLowerDFGToB3.cpp	(revision 227273)
>+++ Source/JavaScriptCore/ftl/FTLLowerDFGToB3.cpp	(working copy)
>@@ -5857,10 +5857,10 @@ private:
>         
>         LBasicBlock lastNext = m_out.insertNewBlocksBefore(slowPath);
>         
>-        BlockDirectory* allocator = subspaceFor<JSRopeString>(vm())->allocatorForNonVirtual(sizeof(JSRopeString), AllocatorForMode::AllocatorIfExists);
>+        Allocator allocator = subspaceFor<JSRopeString>(vm())->allocatorForNonVirtual(sizeof(JSRopeString), AllocatorForMode::AllocatorIfExists);
>         
>         LValue result = allocateCell(
>-            m_out.constIntPtr(allocator), vm().stringStructure.get(), slowPath);
>+            m_out.constInt32(allocator.offset()), vm().stringStructure.get(), slowPath);
>         
>         m_out.storePtr(m_out.intPtrZero, result, m_heaps.JSString_value);
>         for (unsigned i = 0; i < numKids; ++i)
>@@ -9885,7 +9885,7 @@ private:
>             
>             if (structure->outOfLineCapacity() || hasIndexedProperties(structure->indexingType())) {
>                 size_t allocationSize = JSFinalObject::allocationSize(structure->inlineCapacity());
>-                BlockDirectory* cellAllocator = subspaceFor<JSFinalObject>(vm())->allocatorForNonVirtual(allocationSize, AllocatorForMode::AllocatorIfExists);
>+                Allocator cellAllocator = subspaceFor<JSFinalObject>(vm())->allocatorForNonVirtual(allocationSize, AllocatorForMode::AllocatorIfExists);
> 
>                 bool hasIndexingHeader = hasIndexedProperties(structure->indexingType());
>                 unsigned indexingHeaderSize = 0;
>@@ -9944,7 +9944,7 @@ private:
> 
>                 LValue mask = computeButterflyIndexingMask(vectorLength);
>                 LValue fastObjectValue = allocateObject(
>-                    m_out.constIntPtr(cellAllocator), structure, fastButterflyValue, mask, slowPath);
>+                    m_out.constInt32(cellAllocator.offset()), structure, fastButterflyValue, mask, slowPath);
> 
>                 ValueFromBlock fastObject = m_out.anchor(fastObjectValue);
>                 ValueFromBlock fastButterfly = m_out.anchor(fastButterflyValue);
>@@ -10931,8 +10931,8 @@ private:
>         LBasicBlock lastNext = m_out.insertNewBlocksBefore(slowPath);
> 
>         size_t sizeInBytes = sizeInValues * sizeof(JSValue);
>-        BlockDirectory* allocator = vm().jsValueGigacageAuxiliarySpace.allocatorForNonVirtual(sizeInBytes, AllocatorForMode::AllocatorIfExists);
>-        LValue startOfStorage = allocateHeapCell(m_out.constIntPtr(allocator), slowPath);
>+        Allocator allocator = vm().jsValueGigacageAuxiliarySpace.allocatorForNonVirtual(sizeInBytes, AllocatorForMode::AllocatorIfExists);
>+        LValue startOfStorage = allocateHeapCell(m_out.constInt32(allocator.offset()), slowPath);
>         ValueFromBlock fastButterfly = m_out.anchor(
>             m_out.add(m_out.constIntPtr(sizeInBytes + sizeof(IndexingHeader)), startOfStorage));
>         m_out.jump(continuation);
>@@ -11964,16 +11964,28 @@ private:
> 
>     LValue allocateHeapCell(LValue allocator, LBasicBlock slowPath)
>     {
>-        BlockDirectory* actualAllocator = nullptr;
>-        if (allocator->hasIntPtr())
>-            actualAllocator = bitwise_cast<BlockDirectory*>(allocator->asIntPtr());
>+        JITAllocator actualAllocator;
>+        if (allocator->hasInt32())
>+            actualAllocator = JITAllocator::constant(Allocator(allocator->asInt32()));
>+        else
>+            actualAllocator = JITAllocator::variable();
>         
>-        if (!actualAllocator) {
>+        if (actualAllocator.isConstant()) {
>+            if (!actualAllocator.allocator()) {
>+                LBasicBlock haveAllocator = m_out.newBlock();
>+                LBasicBlock lastNext = m_out.insertNewBlocksBefore(haveAllocator);
>+                m_out.jump(slowPath);
>+                m_out.appendTo(haveAllocator, lastNext);
>+                return m_out.intPtrZero;
>+            }
>+        } else {
>             // This means that either we know that the allocator is null or we don't know what the
>             // allocator is. In either case, we need the null check.
>             LBasicBlock haveAllocator = m_out.newBlock();
>             LBasicBlock lastNext = m_out.insertNewBlocksBefore(haveAllocator);
>-            m_out.branch(allocator, usually(haveAllocator), rarely(slowPath));
>+            m_out.branch(
>+                m_out.notEqual(allocator, m_out.constInt32(Allocator().offset())),
>+                usually(haveAllocator), rarely(slowPath));
>             m_out.appendTo(haveAllocator, lastNext);
>         }
>         
>@@ -11983,7 +11995,10 @@ private:
>         
>         PatchpointValue* patchpoint = m_out.patchpoint(pointerType());
>         patchpoint->effects.terminal = true;
>-        patchpoint->appendSomeRegister(allocator);
>+        if (actualAllocator.isConstant())
>+            patchpoint->numGPScratchRegisters++;
>+        else
>+            patchpoint->appendSomeRegisterWithClobber(allocator);
>         patchpoint->numGPScratchRegisters++;
>         patchpoint->resultConstraint = ValueRep::SomeEarlyRegister;
>         
>@@ -11994,6 +12009,12 @@ private:
>             [=] (CCallHelpers& jit, const StackmapGenerationParams& params) {
>                 CCallHelpers::JumpList jumpToSlowPath;
>                 
>+                GPRReg allocatorGPR;
>+                if (actualAllocator.isConstant())
>+                    allocatorGPR = params.gpScratch(1);
>+                else
>+                    allocatorGPR = params[1].gpr();
>+                
>                 // We use a patchpoint to emit the allocation path because whenever we mess with
>                 // allocation paths, we already reason about them at the machine code level. We know
>                 // exactly what instruction sequence we want. We're confident that no compiler
>@@ -12001,7 +12022,7 @@ private:
>                 // AssemblyHelpers::emitAllocate(). That way, the same optimized path is shared by
>                 // all of the compiler tiers.
>                 jit.emitAllocateWithNonNullAllocator(
>-                    params[0].gpr(), actualAllocator, params[1].gpr(), params.gpScratch(0),
>+                    params[0].gpr(), actualAllocator, allocatorGPR, params.gpScratch(0),
>                     jumpToSlowPath);
>                 
>                 CCallHelpers::Jump jumpToSuccess;
>@@ -12093,8 +12114,8 @@ private:
>     LValue allocateObject(
>         size_t size, StructureType structure, LValue butterfly, LValue indexingMask, LBasicBlock slowPath)
>     {
>-        BlockDirectory* allocator = subspaceFor<ClassType>(vm())->allocatorForNonVirtual(size, AllocatorForMode::AllocatorIfExists);
>-        return allocateObject(m_out.constIntPtr(allocator), structure, butterfly, indexingMask, slowPath);
>+        Allocator allocator = subspaceFor<ClassType>(vm())->allocatorForNonVirtual(size, AllocatorForMode::AllocatorIfExists);
>+        return allocateObject(m_out.constInt32(allocator.offset()), structure, butterfly, indexingMask, slowPath);
>     }
>     
>     template<typename ClassType, typename StructureType>
>@@ -12113,16 +12134,16 @@ private:
>             CompleteSubspace* actualSubspace = bitwise_cast<CompleteSubspace*>(subspace->asIntPtr());
>             size_t actualSize = size->asIntPtr();
>             
>-            BlockDirectory* actualAllocator = actualSubspace->allocatorForNonVirtual(actualSize, AllocatorForMode::AllocatorIfExists);
>+            Allocator actualAllocator = actualSubspace->allocatorForNonVirtual(actualSize, AllocatorForMode::AllocatorIfExists);
>             if (!actualAllocator) {
>                 LBasicBlock continuation = m_out.newBlock();
>                 LBasicBlock lastNext = m_out.insertNewBlocksBefore(continuation);
>                 m_out.jump(slowPath);
>                 m_out.appendTo(continuation, lastNext);
>-                return m_out.intPtrZero;
>+                return m_out.int32Zero;
>             }
>             
>-            return m_out.constIntPtr(actualAllocator);
>+            return m_out.constInt32(actualAllocator.offset());
>         }
>         
>         unsigned stepShift = getLSBSet(MarkedSpace::sizeStep);
>@@ -12141,7 +12162,7 @@ private:
>         
>         m_out.appendTo(continuation, lastNext);
>         
>-        return m_out.loadPtr(
>+        return m_out.load32(
>             m_out.baseIndex(
>                 m_heaps.CompleteSubspace_allocatorForSizeStep,
>                 subspace, m_out.sub(sizeClassIndex, m_out.intPtrOne)));
>@@ -12156,8 +12177,7 @@ private:
>     LValue allocateVariableSizedObject(
>         LValue size, RegisteredStructure structure, LValue butterfly, LValue butterflyIndexingMask, LBasicBlock slowPath)
>     {
>-        LValue allocator = allocatorForSize(
>-            *subspaceFor<ClassType>(vm()), size, slowPath);
>+        LValue allocator = allocatorForSize(*subspaceFor<ClassType>(vm()), size, slowPath);
>         return allocateObject(allocator, structure, butterfly, butterflyIndexingMask, slowPath);
>     }
> 
>@@ -12165,15 +12185,14 @@ private:
>     LValue allocateVariableSizedCell(
>         LValue size, Structure* structure, LBasicBlock slowPath)
>     {
>-        LValue allocator = allocatorForSize(
>-            *subspaceFor<ClassType>(vm()), size, slowPath);
>+        LValue allocator = allocatorForSize(*subspaceFor<ClassType>(vm()), size, slowPath);
>         return allocateCell(allocator, structure, slowPath);
>     }
>     
>     LValue allocateObject(RegisteredStructure structure)
>     {
>         size_t allocationSize = JSFinalObject::allocationSize(structure.get()->inlineCapacity());
>-        BlockDirectory* allocator = subspaceFor<JSFinalObject>(vm())->allocatorForNonVirtual(allocationSize, AllocatorForMode::AllocatorIfExists);
>+        Allocator allocator = subspaceFor<JSFinalObject>(vm())->allocatorForNonVirtual(allocationSize, AllocatorForMode::AllocatorIfExists);
>         
>         // FIXME: If the allocator is null, we could simply emit a normal C call to the allocator
>         // instead of putting it on the slow path.
>@@ -12185,7 +12204,7 @@ private:
>         LBasicBlock lastNext = m_out.insertNewBlocksBefore(slowPath);
>         
>         ValueFromBlock fastResult = m_out.anchor(allocateObject(
>-            m_out.constIntPtr(allocator), structure, m_out.intPtrZero, m_out.int32Zero, slowPath));
>+            m_out.constInt32(allocator.offset()), structure, m_out.intPtrZero, m_out.int32Zero, slowPath));
>         
>         m_out.jump(continuation);
>         
>Index: Source/JavaScriptCore/heap/Allocator.cpp
>===================================================================
>--- Source/JavaScriptCore/heap/Allocator.cpp	(nonexistent)
>+++ Source/JavaScriptCore/heap/Allocator.cpp	(working copy)
>@@ -0,0 +1,40 @@
>+/*
>+ * Copyright (C) 2018 Apple Inc. All rights reserved.
>+ *
>+ * Redistribution and use in source and binary forms, with or without
>+ * modification, are permitted provided that the following conditions
>+ * are met:
>+ * 1. Redistributions of source code must retain the above copyright
>+ *    notice, this list of conditions and the following disclaimer.
>+ * 2. Redistributions in binary form must reproduce the above copyright
>+ *    notice, this list of conditions and the following disclaimer in the
>+ *    documentation and/or other materials provided with the distribution.
>+ *
>+ * THIS SOFTWARE IS PROVIDED BY APPLE INC. ``AS IS'' AND ANY
>+ * EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
>+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
>+ * PURPOSE ARE DISCLAIMED.  IN NO EVENT SHALL APPLE INC. OR
>+ * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
>+ * EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
>+ * PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
>+ * PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY
>+ * OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
>+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
>+ * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 
>+ */
>+
>+#include "config.h"
>+#include "Allocator.h"
>+
>+#include "Heap.h"
>+#include "ThreadLocalCacheLayout.h"
>+
>+namespace JSC {
>+
>+unsigned Allocator::cellSize(Heap& heap) const
>+{
>+    return heap.threadLocalCacheLayout().directory(m_offset)->cellSize();
>+}
>+
>+} // namespace JSC
>+
>Index: Source/JavaScriptCore/heap/Allocator.h
>===================================================================
>--- Source/JavaScriptCore/heap/Allocator.h	(nonexistent)
>+++ Source/JavaScriptCore/heap/Allocator.h	(working copy)
>@@ -0,0 +1,65 @@
>+/*
>+ * Copyright (C) 2018 Apple Inc. All rights reserved.
>+ *
>+ * Redistribution and use in source and binary forms, with or without
>+ * modification, are permitted provided that the following conditions
>+ * are met:
>+ * 1. Redistributions of source code must retain the above copyright
>+ *    notice, this list of conditions and the following disclaimer.
>+ * 2. Redistributions in binary form must reproduce the above copyright
>+ *    notice, this list of conditions and the following disclaimer in the
>+ *    documentation and/or other materials provided with the distribution.
>+ *
>+ * THIS SOFTWARE IS PROVIDED BY APPLE INC. ``AS IS'' AND ANY
>+ * EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
>+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
>+ * PURPOSE ARE DISCLAIMED.  IN NO EVENT SHALL APPLE INC. OR
>+ * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
>+ * EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
>+ * PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
>+ * PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY
>+ * OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
>+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
>+ * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 
>+ */
>+
>+#pragma once
>+
>+#include "AllocationFailureMode.h"
>+
>+namespace JSC {
>+
>+class GCDeferralContext;
>+class Heap;
>+class VM;
>+
>+class Allocator {
>+public:
>+    Allocator() { }
>+    
>+    explicit Allocator(unsigned offset)
>+        : m_offset(offset)
>+    {
>+    }
>+    
>+    void* allocate(VM&, GCDeferralContext*, AllocationFailureMode) const;
>+    
>+    // This version calls FailureFunc if we have a null allocator or if the TLC hasn't been resized
>+    // to include this allocator.
>+    template<typename FailureFunc>
>+    void* tryAllocate(VM&, GCDeferralContext*, AllocationFailureMode, const FailureFunc&) const;
>+    
>+    unsigned cellSize(Heap&) const;
>+    
>+    unsigned offset() const { return m_offset; }
>+    
>+    bool operator==(const Allocator& other) const { return m_offset == other.offset(); }
>+    bool operator!=(const Allocator& other) const { return !(*this == other); }
>+    explicit operator bool() const { return *this != Allocator(); }
>+    
>+private:
>+    unsigned m_offset { UINT_MAX };
>+};
>+
>+} // namespace JSC
>+
>Index: Source/JavaScriptCore/heap/AllocatorInlines.h
>===================================================================
>--- Source/JavaScriptCore/heap/AllocatorInlines.h	(nonexistent)
>+++ Source/JavaScriptCore/heap/AllocatorInlines.h	(working copy)
>@@ -0,0 +1,54 @@
>+/*
>+ * Copyright (C) 2018 Apple Inc. All rights reserved.
>+ *
>+ * Redistribution and use in source and binary forms, with or without
>+ * modification, are permitted provided that the following conditions
>+ * are met:
>+ * 1. Redistributions of source code must retain the above copyright
>+ *    notice, this list of conditions and the following disclaimer.
>+ * 2. Redistributions in binary form must reproduce the above copyright
>+ *    notice, this list of conditions and the following disclaimer in the
>+ *    documentation and/or other materials provided with the distribution.
>+ *
>+ * THIS SOFTWARE IS PROVIDED BY APPLE INC. ``AS IS'' AND ANY
>+ * EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
>+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
>+ * PURPOSE ARE DISCLAIMED.  IN NO EVENT SHALL APPLE INC. OR
>+ * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
>+ * EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
>+ * PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
>+ * PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY
>+ * OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
>+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
>+ * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 
>+ */
>+
>+#pragma once
>+
>+#include "Allocator.h"
>+#include "ThreadLocalCache.h"
>+
>+namespace JSC {
>+
>+inline void* Allocator::allocate(VM& vm, GCDeferralContext* context, AllocationFailureMode mode) const
>+{
>+    return ThreadLocalCache::allocator(vm, m_offset).allocate(context, mode);
>+}
>+
>+template<typename FailureFunc>
>+void* Allocator::tryAllocate(VM& vm, GCDeferralContext* context, AllocationFailureMode mode, const FailureFunc& failureFunc) const
>+{
>+    void* result;
>+    ThreadLocalCache::tryGetAllocator(
>+        vm, m_offset,
>+        [&] (LocalAllocator& allocator) {
>+            result = allocator.allocate(context, mode);
>+        },
>+        [&] () {
>+            result = failureFunc();
>+        });
>+    return result;
>+}
>+
>+} // namespace JSC
>+
>Index: Source/JavaScriptCore/heap/BlockDirectory.cpp
>===================================================================
>--- Source/JavaScriptCore/heap/BlockDirectory.cpp	(revision 227273)
>+++ Source/JavaScriptCore/heap/BlockDirectory.cpp	(working copy)
>@@ -26,7 +26,6 @@
> #include "config.h"
> #include "BlockDirectory.h"
> 
>-#include "AllocatingScope.h"
> #include "BlockDirectoryInlines.h"
> #include "GCActivityCallback.h"
> #include "Heap.h"
>@@ -39,15 +38,11 @@
> 
> namespace JSC {
> 
>-static constexpr bool tradeDestructorBlocks = true;
>-
> BlockDirectory::BlockDirectory(Heap* heap, size_t cellSize)
>-    : m_freeList(cellSize)
>-    , m_currentBlock(0)
>-    , m_lastActiveBlock(0)
>-    , m_cellSize(static_cast<unsigned>(cellSize))
>+    : m_cellSize(static_cast<unsigned>(cellSize))
>     , m_heap(heap)
> {
>+    heap->threadLocalCacheLayout().allocateOffset(this);
> }
> 
> void BlockDirectory::setSubspace(Subspace* subspace)
>@@ -81,143 +76,14 @@ MarkedBlock::Handle* BlockDirectory::fin
>     return m_blocks[m_emptyCursor];
> }
> 
>-void BlockDirectory::didConsumeFreeList()
>-{
>-    if (m_currentBlock)
>-        m_currentBlock->didConsumeFreeList();
>-    
>-    m_freeList.clear();
>-    m_currentBlock = nullptr;
>-}
>-
>-void* BlockDirectory::tryAllocateWithoutCollecting()
>+MarkedBlock::Handle* BlockDirectory::findBlockForAllocation()
> {
>-    SuperSamplerScope superSamplerScope(false);
>-    
>-    ASSERT(!m_currentBlock);
>-    ASSERT(m_freeList.allocationWillFail());
>-    
>-    for (;;) {
>-        m_allocationCursor = (m_canAllocateButNotEmpty | m_empty).findBit(m_allocationCursor, true);
>-        if (m_allocationCursor >= m_blocks.size())
>-            break;
>-        
>-        setIsCanAllocateButNotEmpty(NoLockingNecessary, m_allocationCursor, false);
>-
>-        if (void* result = tryAllocateIn(m_blocks[m_allocationCursor]))
>-            return result;
>-    }
>-    
>-    if (Options::stealEmptyBlocksFromOtherAllocators()
>-        && (tradeDestructorBlocks || !needsDestruction())) {
>-        if (MarkedBlock::Handle* block = m_subspace->findEmptyBlockToSteal()) {
>-            RELEASE_ASSERT(block->alignedMemoryAllocator() == m_subspace->alignedMemoryAllocator());
>-            
>-            block->sweep(nullptr);
>-            
>-            // It's good that this clears canAllocateButNotEmpty as well as all other bits,
>-            // because there is a remote chance that a block may have both canAllocateButNotEmpty
>-            // and empty set at the same time.
>-            block->removeFromDirectory();
>-            addBlock(block);
>-            return allocateIn(block);
>-        }
>-    }
>-    
>-    return nullptr;
>-}
>-
>-void* BlockDirectory::allocateIn(MarkedBlock::Handle* block)
>-{
>-    void* result = tryAllocateIn(block);
>-    RELEASE_ASSERT(result);
>-    return result;
>-}
>-
>-void* BlockDirectory::tryAllocateIn(MarkedBlock::Handle* block)
>-{
>-    ASSERT(block);
>-    ASSERT(!block->isFreeListed());
>-    
>-    block->sweep(&m_freeList);
>-    
>-    // It's possible to stumble on a completely full block. Marking tries to retire these, but
>-    // that algorithm is racy and may forget to do it sometimes.
>-    if (m_freeList.allocationWillFail()) {
>-        ASSERT(block->isFreeListed());
>-        block->unsweepWithNoNewlyAllocated();
>-        ASSERT(!block->isFreeListed());
>-        ASSERT(!isEmpty(NoLockingNecessary, block));
>-        ASSERT(!isCanAllocateButNotEmpty(NoLockingNecessary, block));
>+    m_allocationCursor = (m_canAllocateButNotEmpty | m_empty).findBit(m_allocationCursor, true);
>+    if (m_allocationCursor >= m_blocks.size())
>         return nullptr;
>-    }
>-    
>-    m_currentBlock = block;
>-    
>-    void* result = m_freeList.allocate(
>-        [] () -> HeapCell* {
>-            RELEASE_ASSERT_NOT_REACHED();
>-            return nullptr;
>-        });
>-    setIsEden(NoLockingNecessary, m_currentBlock, true);
>-    markedSpace().didAllocateInBlock(m_currentBlock);
>-    return result;
>-}
>-
>-ALWAYS_INLINE void BlockDirectory::doTestCollectionsIfNeeded(GCDeferralContext* deferralContext)
>-{
>-    if (!Options::slowPathAllocsBetweenGCs())
>-        return;
>-
>-    static unsigned allocationCount = 0;
>-    if (!allocationCount) {
>-        if (!m_heap->isDeferred()) {
>-            if (deferralContext)
>-                deferralContext->m_shouldGC = true;
>-            else
>-                m_heap->collectNow(Sync, CollectionScope::Full);
>-        }
>-    }
>-    if (++allocationCount >= Options::slowPathAllocsBetweenGCs())
>-        allocationCount = 0;
>-}
>-
>-void* BlockDirectory::allocateSlowCase(GCDeferralContext* deferralContext, AllocationFailureMode failureMode)
>-{
>-    SuperSamplerScope superSamplerScope(false);
>-    ASSERT(m_heap->vm()->currentThreadIsHoldingAPILock());
>-    doTestCollectionsIfNeeded(deferralContext);
>-
>-    ASSERT(!markedSpace().isIterating());
>-    m_heap->didAllocate(m_freeList.originalSize());
>-    
>-    didConsumeFreeList();
>-    
>-    AllocatingScope helpingHeap(*m_heap);
>-
>-    m_heap->collectIfNecessaryOrDefer(deferralContext);
>     
>-    // Goofy corner case: the GC called a callback and now this directory has a currentBlock. This only
>-    // happens when running WebKit tests, which inject a callback into the GC's finalization.
>-    if (UNLIKELY(m_currentBlock))
>-        return allocate(deferralContext, failureMode);
>-    
>-    void* result = tryAllocateWithoutCollecting();
>-    
>-    if (LIKELY(result != 0))
>-        return result;
>-    
>-    MarkedBlock::Handle* block = tryAllocateBlock();
>-    if (!block) {
>-        if (failureMode == AllocationFailureMode::Assert)
>-            RELEASE_ASSERT_NOT_REACHED();
>-        else
>-            return nullptr;
>-    }
>-    addBlock(block);
>-    result = allocateIn(block);
>-    ASSERT(result);
>-    return result;
>+    setIsCanAllocateButNotEmpty(NoLockingNecessary, m_allocationCursor, false);
>+    return m_blocks[m_allocationCursor];
> }
> 
> static size_t blockHeaderSize()
>@@ -313,24 +179,19 @@ void BlockDirectory::stopAllocating()
> {
>     if (false)
>         dataLog(RawPointer(this), ": BlockDirectory::stopAllocating!\n");
>-    ASSERT(!m_lastActiveBlock);
>-    if (!m_currentBlock) {
>-        ASSERT(m_freeList.allocationWillFail());
>-        return;
>-    }
>-    
>-    m_currentBlock->stopAllocating(m_freeList);
>-    m_lastActiveBlock = m_currentBlock;
>-    m_currentBlock = 0;
>-    m_freeList.clear();
>+    m_localAllocators.forEach(
>+        [&] (LocalAllocator* allocator) {
>+            allocator->stopAllocating();
>+        });
> }
> 
> void BlockDirectory::prepareForAllocation()
> {
>-    m_lastActiveBlock = nullptr;
>-    m_currentBlock = nullptr;
>-    m_freeList.clear();
>-
>+    m_localAllocators.forEach(
>+        [&] (LocalAllocator* allocator) {
>+            allocator->prepareForAllocation();
>+        });
>+    
>     m_allocationCursor = 0;
>     m_emptyCursor = 0;
>     m_unsweptCursor = 0;
>@@ -344,6 +205,21 @@ void BlockDirectory::prepareForAllocatio
>     }
> }
> 
>+void BlockDirectory::stopAllocatingForGood()
>+{
>+    if (false)
>+        dataLog(RawPointer(this), ": BlockDirectory::stopAllocatingForGood!\n");
>+    
>+    m_localAllocators.forEach(
>+        [&] (LocalAllocator* allocator) {
>+            allocator->stopAllocatingForGood();
>+        });
>+
>+    auto locker = holdLock(m_localAllocatorsLock);
>+    while (!m_localAllocators.isEmpty())
>+        m_localAllocators.begin()->remove();
>+}
>+
> void BlockDirectory::lastChanceToFinalize()
> {
>     forEachBlock(
>@@ -354,12 +230,10 @@ void BlockDirectory::lastChanceToFinaliz
> 
> void BlockDirectory::resumeAllocating()
> {
>-    if (!m_lastActiveBlock)
>-        return;
>-
>-    m_lastActiveBlock->resumeAllocating(m_freeList);
>-    m_currentBlock = m_lastActiveBlock;
>-    m_lastActiveBlock = nullptr;
>+    m_localAllocators.forEach(
>+        [&] (LocalAllocator* allocator) {
>+            allocator->resumeAllocating();
>+        });
> }
> 
> void BlockDirectory::beginMarkingForFullCollection()
>@@ -379,7 +253,7 @@ void BlockDirectory::endMarking()
>     // know what kind of collection it is. That knowledge is already encoded in the m_markingXYZ
>     // vectors.
>     
>-    if (!tradeDestructorBlocks && needsDestruction()) {
>+    if (!Options::tradeDestructorBlocks() && needsDestruction()) {
>         ASSERT(m_empty.isEmpty());
>         m_canAllocateButNotEmpty = m_live & ~m_markingRetired;
>     } else {
>@@ -512,5 +386,15 @@ MarkedSpace& BlockDirectory::markedSpace
>     return m_subspace->space();
> }
> 
>+bool BlockDirectory::isFreeListedCell(const void* target)
>+{
>+    bool result = false;
>+    m_localAllocators.forEach(
>+        [&] (LocalAllocator* allocator) {
>+            result |= allocator->isFreeListedCell(target);
>+        });
>+    return result;
>+}
>+
> } // namespace JSC
> 
>Index: Source/JavaScriptCore/heap/BlockDirectory.h
>===================================================================
>--- Source/JavaScriptCore/heap/BlockDirectory.h	(revision 227273)
>+++ Source/JavaScriptCore/heap/BlockDirectory.h	(working copy)
>@@ -26,8 +26,10 @@
> #pragma once
> 
> #include "AllocationFailureMode.h"
>+#include "Allocator.h"
> #include "CellAttributes.h"
> #include "FreeList.h"
>+#include "LocalAllocator.h"
> #include "MarkedBlock.h"
> #include <wtf/DataLog.h>
> #include <wtf/FastBitVector.h>
>@@ -41,6 +43,7 @@ class Heap;
> class IsoCellSet;
> class MarkedSpace;
> class LLIntOffsetsExtractor;
>+class ThreadLocalCacheLayout;
> 
> #define FOR_EACH_BLOCK_DIRECTORY_BIT(macro) \
>     macro(live, Live) /* The set of block indices that have actual blocks. */\
>@@ -76,14 +79,12 @@ class BlockDirectory {
>     friend class LLIntOffsetsExtractor;
> 
> public:
>-    static ptrdiff_t offsetOfFreeList();
>-    static ptrdiff_t offsetOfCellSize();
>-
>     BlockDirectory(Heap*, size_t cellSize);
>     void setSubspace(Subspace*);
>     void lastChanceToFinalize();
>     void prepareForAllocation();
>     void stopAllocating();
>+    void stopAllocatingForGood();
>     void resumeAllocating();
>     void beginMarkingForFullCollection();
>     void endMarking();
>@@ -97,10 +98,9 @@ public:
>     bool needsDestruction() const { return m_attributes.destruction == NeedsDestruction; }
>     DestructionMode destruction() const { return m_attributes.destruction; }
>     HeapCell::Kind cellKind() const { return m_attributes.cellKind; }
>-    void* allocate(GCDeferralContext*, AllocationFailureMode);
>     Heap* heap() { return m_heap; }
> 
>-    bool isFreeListedCell(const void* target) const;
>+    bool isFreeListedCell(const void* target);
> 
>     template<typename Functor> void forEachBlock(const Functor&);
>     template<typename Functor> void forEachNotEmptyBlock(const Functor&);
>@@ -157,24 +157,20 @@ public:
>     Subspace* subspace() const { return m_subspace; }
>     MarkedSpace& markedSpace() const;
>     
>-    const FreeList& freeList() const { return m_freeList; }
>+    Allocator allocator() const { return Allocator(m_tlcOffset); }
>     
>     void dump(PrintStream&) const;
>     void dumpBits(PrintStream& = WTF::dataFile());
>     
> private:
>+    friend class LocalAllocator;
>     friend class IsoCellSet;
>     friend class MarkedBlock;
>+    friend class ThreadLocalCacheLayout;
>     
>-    JS_EXPORT_PRIVATE void* allocateSlowCase(GCDeferralContext*, AllocationFailureMode failureMode);
>-    void didConsumeFreeList();
>-    void* tryAllocateWithoutCollecting();
>-    MarkedBlock::Handle* tryAllocateBlock();
>-    void* tryAllocateIn(MarkedBlock::Handle*);
>-    void* allocateIn(MarkedBlock::Handle*);
>-    ALWAYS_INLINE void doTestCollectionsIfNeeded(GCDeferralContext*);
>+    MarkedBlock::Handle* findBlockForAllocation();
>     
>-    FreeList m_freeList;
>+    MarkedBlock::Handle* tryAllocateBlock();
>     
>     Vector<MarkedBlock::Handle*> m_blocks;
>     Vector<unsigned> m_freeBlockIndices;
>@@ -193,10 +189,6 @@ private:
>     size_t m_emptyCursor { 0 }; // Points to the next block that is a candidate for empty allocation (allocating in empty blocks).
>     size_t m_unsweptCursor { 0 }; // Points to the next block that is a candidate for incremental sweeping.
>     
>-    MarkedBlock::Handle* m_currentBlock;
>-    MarkedBlock::Handle* m_lastActiveBlock;
>-
>-    Lock m_lock;
>     unsigned m_cellSize;
>     CellAttributes m_attributes;
>     // FIXME: All of these should probably be references.
>@@ -206,16 +198,10 @@ private:
>     BlockDirectory* m_nextDirectory { nullptr };
>     BlockDirectory* m_nextDirectoryInSubspace { nullptr };
>     BlockDirectory* m_nextDirectoryInAlignedMemoryAllocator { nullptr };
>+    
>+    Lock m_localAllocatorsLock;
>+    size_t m_tlcOffset;
>+    SentinelLinkedList<LocalAllocator, BasicRawSentinelNode<LocalAllocator>> m_localAllocators;
> };
> 
>-inline ptrdiff_t BlockDirectory::offsetOfFreeList()
>-{
>-    return OBJECT_OFFSETOF(BlockDirectory, m_freeList);
>-}
>-
>-inline ptrdiff_t BlockDirectory::offsetOfCellSize()
>-{
>-    return OBJECT_OFFSETOF(BlockDirectory, m_cellSize);
>-}
>-
> } // namespace JSC
>Index: Source/JavaScriptCore/heap/BlockDirectoryInlines.h
>===================================================================
>--- Source/JavaScriptCore/heap/BlockDirectoryInlines.h	(revision 227273)
>+++ Source/JavaScriptCore/heap/BlockDirectoryInlines.h	(working copy)
>@@ -31,20 +31,6 @@
> 
> namespace JSC {
> 
>-inline bool BlockDirectory::isFreeListedCell(const void* target) const
>-{
>-    return m_freeList.contains(bitwise_cast<HeapCell*>(target));
>-}
>-
>-ALWAYS_INLINE void* BlockDirectory::allocate(GCDeferralContext* deferralContext, AllocationFailureMode failureMode)
>-{
>-    return m_freeList.allocate(
>-        [&] () -> HeapCell* {
>-            sanitizeStackForVM(heap()->vm());
>-            return static_cast<HeapCell*>(allocateSlowCase(deferralContext, failureMode));
>-        });
>-}
>-
> template <typename Functor> inline void BlockDirectory::forEachBlock(const Functor& functor)
> {
>     m_live.forEachSetBit(
>Index: Source/JavaScriptCore/heap/CompleteSubspace.cpp
>===================================================================
>--- Source/JavaScriptCore/heap/CompleteSubspace.cpp	(revision 227273)
>+++ Source/JavaScriptCore/heap/CompleteSubspace.cpp	(working copy)
>@@ -1,5 +1,5 @@
> /*
>- * Copyright (C) 2017 Apple Inc. All rights reserved.
>+ * Copyright (C) 2017-2018 Apple Inc. All rights reserved.
>  *
>  * Redistribution and use in source and binary forms, with or without
>  * modification, are permitted provided that the following conditions
>@@ -26,11 +26,15 @@
> #include "config.h"
> #include "Subspace.h"
> 
>+#include "AlignedMemoryAllocator.h"
>+#include "AllocatorInlines.h"
> #include "BlockDirectoryInlines.h"
> #include "JSCInlines.h"
>+#include "LocalAllocatorInlines.h"
> #include "MarkedBlockInlines.h"
> #include "PreventCollectionScope.h"
> #include "SubspaceInlines.h"
>+#include "ThreadLocalCacheInlines.h"
> 
> namespace JSC {
> 
>@@ -38,40 +42,38 @@ CompleteSubspace::CompleteSubspace(CStri
>     : Subspace(name, heap)
> {
>     initialize(heapCellType, alignedMemoryAllocator);
>-    for (size_t i = MarkedSpace::numSizeClasses; i--;)
>-        m_allocatorForSizeStep[i] = nullptr;
> }
> 
> CompleteSubspace::~CompleteSubspace()
> {
> }
> 
>-BlockDirectory* CompleteSubspace::allocatorFor(size_t size, AllocatorForMode mode)
>+Allocator CompleteSubspace::allocatorFor(size_t size, AllocatorForMode mode)
> {
>     return allocatorForNonVirtual(size, mode);
> }
> 
>-void* CompleteSubspace::allocate(size_t size, GCDeferralContext* deferralContext, AllocationFailureMode failureMode)
>+void* CompleteSubspace::allocate(VM& vm, size_t size, GCDeferralContext* deferralContext, AllocationFailureMode failureMode)
> {
>-    return allocateNonVirtual(size, deferralContext, failureMode);
>+    return allocateNonVirtual(vm, size, deferralContext, failureMode);
> }
> 
>-void* CompleteSubspace::allocateNonVirtual(size_t size, GCDeferralContext* deferralContext, AllocationFailureMode failureMode)
>+void* CompleteSubspace::allocateNonVirtual(VM& vm, size_t size, GCDeferralContext* deferralContext, AllocationFailureMode failureMode)
> {
>-    void *result;
>-    if (BlockDirectory* allocator = allocatorForNonVirtual(size, AllocatorForMode::AllocatorIfExists))
>-        result = allocator->allocate(deferralContext, failureMode);
>-    else
>-        result = allocateSlow(size, deferralContext, failureMode);
>-    return result;
>+    Allocator allocator = allocatorForNonVirtual(size, AllocatorForMode::AllocatorIfExists);
>+    return allocator.tryAllocate(
>+        vm, deferralContext, failureMode,
>+        [&] () {
>+            return allocateSlow(vm, size, deferralContext, failureMode);
>+        });
> }
> 
>-BlockDirectory* CompleteSubspace::allocatorForSlow(size_t size)
>+Allocator CompleteSubspace::allocatorForSlow(size_t size)
> {
>     size_t index = MarkedSpace::sizeClassToIndex(size);
>     size_t sizeClass = MarkedSpace::s_sizeClassForSizeStep[index];
>     if (!sizeClass)
>-        return nullptr;
>+        return Allocator();
>     
>     // This is written in such a way that it's OK for the JIT threads to end up here if they want
>     // to generate code that uses some allocator that hadn't been used yet. Note that a possibly-
>@@ -82,7 +84,7 @@ BlockDirectory* CompleteSubspace::alloca
>     // that any "forEachAllocator" traversals will only see this allocator after it's initialized
>     // enough: it will have 
>     auto locker = holdLock(m_space.directoryLock());
>-    if (BlockDirectory* allocator = m_allocatorForSizeStep[index])
>+    if (Allocator allocator = m_allocatorForSizeStep[index])
>         return allocator;
> 
>     if (false)
>@@ -98,7 +100,7 @@ BlockDirectory* CompleteSubspace::alloca
>         if (MarkedSpace::s_sizeClassForSizeStep[index] != sizeClass)
>             break;
> 
>-        m_allocatorForSizeStep[index] = directory;
>+        m_allocatorForSizeStep[index] = directory->allocator();
>         
>         if (!index--)
>             break;
>@@ -107,23 +109,23 @@ BlockDirectory* CompleteSubspace::alloca
>     m_alignedMemoryAllocator->registerDirectory(directory);
>     WTF::storeStoreFence();
>     m_firstDirectory = directory;
>-    return directory;
>+    return directory->allocator();
> }
> 
>-void* CompleteSubspace::allocateSlow(size_t size, GCDeferralContext* deferralContext, AllocationFailureMode failureMode)
>+void* CompleteSubspace::allocateSlow(VM& vm, size_t size, GCDeferralContext* deferralContext, AllocationFailureMode failureMode)
> {
>-    void* result = tryAllocateSlow(size, deferralContext);
>+    void* result = tryAllocateSlow(vm, size, deferralContext);
>     if (failureMode == AllocationFailureMode::Assert)
>         RELEASE_ASSERT(result);
>     return result;
> }
> 
>-void* CompleteSubspace::tryAllocateSlow(size_t size, GCDeferralContext* deferralContext)
>+void* CompleteSubspace::tryAllocateSlow(VM& vm, size_t size, GCDeferralContext* deferralContext)
> {
>-    sanitizeStackForVM(m_space.heap()->vm());
>+    sanitizeStackForVM(&vm);
>     
>-    if (BlockDirectory* allocator = allocatorFor(size, AllocatorForMode::EnsureAllocator))
>-        return allocator->allocate(deferralContext, AllocationFailureMode::ReturnNull);
>+    if (Allocator allocator = allocatorFor(size, AllocatorForMode::EnsureAllocator))
>+        return allocator.allocate(vm, deferralContext, AllocationFailureMode::ReturnNull);
>     
>     if (size <= Options::largeAllocationCutoff()
>         && size <= MarkedSpace::largeCutoff) {
>@@ -132,15 +134,15 @@ void* CompleteSubspace::tryAllocateSlow(
>         RELEASE_ASSERT_NOT_REACHED();
>     }
>     
>-    m_space.heap()->collectIfNecessaryOrDefer(deferralContext);
>+    vm.heap.collectIfNecessaryOrDefer(deferralContext);
>     
>     size = WTF::roundUpToMultipleOf<MarkedSpace::sizeStep>(size);
>-    LargeAllocation* allocation = LargeAllocation::tryCreate(*m_space.m_heap, size, this);
>+    LargeAllocation* allocation = LargeAllocation::tryCreate(vm.heap, size, this);
>     if (!allocation)
>         return nullptr;
>     
>     m_space.m_largeAllocations.append(allocation);
>-    m_space.m_heap->didAllocate(size);
>+    vm.heap.didAllocate(size);
>     m_space.m_capacity += size;
>     
>     m_largeAllocations.append(allocation);
>Index: Source/JavaScriptCore/heap/CompleteSubspace.h
>===================================================================
>--- Source/JavaScriptCore/heap/CompleteSubspace.h	(revision 227273)
>+++ Source/JavaScriptCore/heap/CompleteSubspace.h	(working copy)
>@@ -39,31 +39,31 @@ public:
>     
>     // FIXME: Currently subspaces speak of BlockDirectories as "allocators", but that's temporary.
>     // https://bugs.webkit.org/show_bug.cgi?id=181559
>-    BlockDirectory* allocatorFor(size_t, AllocatorForMode) override;
>-    BlockDirectory* allocatorForNonVirtual(size_t, AllocatorForMode);
>+    Allocator allocatorFor(size_t, AllocatorForMode) override;
>+    Allocator allocatorForNonVirtual(size_t, AllocatorForMode);
>     
>-    void* allocate(size_t, GCDeferralContext*, AllocationFailureMode) override;
>-    JS_EXPORT_PRIVATE void* allocateNonVirtual(size_t, GCDeferralContext*, AllocationFailureMode);
>+    void* allocate(VM&, size_t, GCDeferralContext*, AllocationFailureMode) override;
>+    JS_EXPORT_PRIVATE void* allocateNonVirtual(VM&, size_t, GCDeferralContext*, AllocationFailureMode);
>     
>     static ptrdiff_t offsetOfAllocatorForSizeStep() { return OBJECT_OFFSETOF(CompleteSubspace, m_allocatorForSizeStep); }
>     
>-    BlockDirectory** allocatorForSizeStep() { return &m_allocatorForSizeStep[0]; }
>+    Allocator* allocatorForSizeStep() { return &m_allocatorForSizeStep[0]; }
> 
> private:
>-    BlockDirectory* allocatorForSlow(size_t);
>+    Allocator allocatorForSlow(size_t);
>     
>     // These slow paths are concerned with large allocations and allocator creation.
>-    void* allocateSlow(size_t, GCDeferralContext*, AllocationFailureMode);
>-    void* tryAllocateSlow(size_t, GCDeferralContext*);
>+    void* allocateSlow(VM&, size_t, GCDeferralContext*, AllocationFailureMode);
>+    void* tryAllocateSlow(VM&, size_t, GCDeferralContext*);
>     
>-    std::array<BlockDirectory*, MarkedSpace::numSizeClasses> m_allocatorForSizeStep;
>+    std::array<Allocator, MarkedSpace::numSizeClasses> m_allocatorForSizeStep;
>     Vector<std::unique_ptr<BlockDirectory>> m_directories;
> };
> 
>-ALWAYS_INLINE BlockDirectory* CompleteSubspace::allocatorForNonVirtual(size_t size, AllocatorForMode mode)
>+ALWAYS_INLINE Allocator CompleteSubspace::allocatorForNonVirtual(size_t size, AllocatorForMode mode)
> {
>     if (size <= MarkedSpace::largeCutoff) {
>-        BlockDirectory* result = m_allocatorForSizeStep[MarkedSpace::sizeClassToIndex(size)];
>+        Allocator result = m_allocatorForSizeStep[MarkedSpace::sizeClassToIndex(size)];
>         switch (mode) {
>         case AllocatorForMode::MustAlreadyHaveAllocator:
>             RELEASE_ASSERT(result);
>@@ -78,7 +78,7 @@ ALWAYS_INLINE BlockDirectory* CompleteSu
>         return result;
>     }
>     RELEASE_ASSERT(mode != AllocatorForMode::MustAlreadyHaveAllocator);
>-    return nullptr;
>+    return Allocator();
> }
> 
> } // namespace JSC
>Index: Source/JavaScriptCore/heap/FreeList.h
>===================================================================
>--- Source/JavaScriptCore/heap/FreeList.h	(revision 227273)
>+++ Source/JavaScriptCore/heap/FreeList.h	(working copy)
>@@ -57,8 +57,6 @@ struct FreeCell {
> };
> 
> class FreeList {
>-    WTF_MAKE_NONCOPYABLE(FreeList);
>-    
> public:
>     FreeList(unsigned cellSize);
>     ~FreeList();
>Index: Source/JavaScriptCore/heap/GCDeferralContext.h
>===================================================================
>--- Source/JavaScriptCore/heap/GCDeferralContext.h	(revision 227273)
>+++ Source/JavaScriptCore/heap/GCDeferralContext.h	(working copy)
>@@ -1,5 +1,5 @@
> /*
>- * Copyright (C) 2016 Apple Inc. All rights reserved.
>+ * Copyright (C) 2016-2018 Apple Inc. All rights reserved.
>  *
>  * Redistribution and use in source and binary forms, with or without
>  * modification, are permitted provided that the following conditions
>@@ -29,10 +29,12 @@ namespace JSC {
> 
> class Heap;
> class BlockDirectory;
>+class LocalAllocator;
> 
> class GCDeferralContext {
>     friend class Heap;
>     friend class BlockDirectory;
>+    friend class LocalAllocator;
> public:
>     inline GCDeferralContext(Heap&);
>     inline ~GCDeferralContext();
>Index: Source/JavaScriptCore/heap/Heap.cpp
>===================================================================
>--- Source/JavaScriptCore/heap/Heap.cpp	(revision 227273)
>+++ Source/JavaScriptCore/heap/Heap.cpp	(working copy)
>@@ -68,6 +68,7 @@
> #include "StopIfNecessaryTimer.h"
> #include "SweepingScope.h"
> #include "SynchronousStopTheWorldMutatorScheduler.h"
>+#include "ThreadLocalCacheLayout.h"
> #include "TypeProfiler.h"
> #include "TypeProfilerLog.h"
> #include "UnlinkedCodeBlock.h"
>@@ -312,6 +313,7 @@ Heap::Heap(VM* vm, HeapType heapType)
>     , m_helperClient(&heapHelperPool())
>     , m_threadLock(Box<Lock>::create())
>     , m_threadCondition(AutomaticThreadCondition::create())
>+    , m_threadLocalCacheLayout(std::make_unique<ThreadLocalCacheLayout>())
> {
>     m_worldState.store(0);
> 
>@@ -446,7 +448,7 @@ void Heap::lastChanceToFinalize()
>         dataLog("5 ");
>     
>     m_arrayBuffers.lastChanceToFinalize();
>-    m_objectSpace.stopAllocating();
>+    m_objectSpace.stopAllocatingForGood();
>     m_objectSpace.lastChanceToFinalize();
>     releaseDelayedReleasedObjects();
> 
>Index: Source/JavaScriptCore/heap/Heap.h
>===================================================================
>--- Source/JavaScriptCore/heap/Heap.h	(revision 227273)
>+++ Source/JavaScriptCore/heap/Heap.h	(working copy)
>@@ -84,6 +84,7 @@ class SlotVisitor;
> class SpaceTimeMutatorScheduler;
> class StopIfNecessaryTimer;
> class SweepingScope;
>+class ThreadLocalCacheLayout;
> class VM;
> class WeakGCMapBase;
> struct CurrentThreadState;
>@@ -372,6 +373,8 @@ public:
> 
>     template<typename Func>
>     void forEachSlotVisitor(const Func&);
>+    
>+    ThreadLocalCacheLayout& threadLocalCacheLayout() { return *m_threadLocalCacheLayout; }
> 
> private:
>     friend class AllocatingScope;
>@@ -714,6 +717,8 @@ private:
>     
>     CurrentThreadState* m_currentThreadState { nullptr };
>     WTF::Thread* m_currentThread { nullptr }; // It's OK if this becomes a dangling pointer.
>+    
>+    std::unique_ptr<ThreadLocalCacheLayout> m_threadLocalCacheLayout;
> };
> 
> } // namespace JSC
>Index: Source/JavaScriptCore/heap/IsoCellSet.h
>===================================================================
>--- Source/JavaScriptCore/heap/IsoCellSet.h	(revision 227273)
>+++ Source/JavaScriptCore/heap/IsoCellSet.h	(working copy)
>@@ -25,9 +25,12 @@
> 
> #pragma once
> 
>+#include "MarkedBlock.h"
> #include <wtf/Bitmap.h>
> #include <wtf/ConcurrentVector.h>
> #include <wtf/FastBitVector.h>
>+#include <wtf/SentinelLinkedList.h>
>+#include <wtf/SharedTask.h>
> 
> namespace JSC {
> 
>Index: Source/JavaScriptCore/heap/IsoSubspace.cpp
>===================================================================
>--- Source/JavaScriptCore/heap/IsoSubspace.cpp	(revision 227273)
>+++ Source/JavaScriptCore/heap/IsoSubspace.cpp	(working copy)
>@@ -26,8 +26,11 @@
> #include "config.h"
> #include "IsoSubspace.h"
> 
>+#include "AllocatorInlines.h"
> #include "BlockDirectoryInlines.h"
> #include "IsoAlignedMemoryAllocator.h"
>+#include "LocalAllocatorInlines.h"
>+#include "ThreadLocalCacheInlines.h"
> 
> namespace JSC {
> 
>@@ -35,6 +38,7 @@ IsoSubspace::IsoSubspace(CString name, H
>     : Subspace(name, heap)
>     , m_size(size)
>     , m_directory(&heap, WTF::roundUpToMultipleOf<MarkedBlock::atomSize>(size))
>+    , m_allocator(m_directory.allocator())
>     , m_isoAlignedMemoryAllocator(std::make_unique<IsoAlignedMemoryAllocator>())
> {
>     initialize(heapCellType, m_isoAlignedMemoryAllocator.get());
>@@ -50,20 +54,20 @@ IsoSubspace::~IsoSubspace()
> {
> }
> 
>-BlockDirectory* IsoSubspace::allocatorFor(size_t size, AllocatorForMode mode)
>+Allocator IsoSubspace::allocatorFor(size_t size, AllocatorForMode mode)
> {
>     return allocatorForNonVirtual(size, mode);
> }
> 
>-void* IsoSubspace::allocate(size_t size, GCDeferralContext* deferralContext, AllocationFailureMode failureMode)
>+void* IsoSubspace::allocate(VM& vm, size_t size, GCDeferralContext* deferralContext, AllocationFailureMode failureMode)
> {
>-    return allocateNonVirtual(size, deferralContext, failureMode);
>+    return allocateNonVirtual(vm, size, deferralContext, failureMode);
> }
> 
>-void* IsoSubspace::allocateNonVirtual(size_t size, GCDeferralContext* deferralContext, AllocationFailureMode failureMode)
>+void* IsoSubspace::allocateNonVirtual(VM& vm, size_t size, GCDeferralContext* deferralContext, AllocationFailureMode failureMode)
> {
>     RELEASE_ASSERT(size == this->size());
>-    void* result = m_directory.allocate(deferralContext, failureMode);
>+    void* result = m_allocator.allocate(vm, deferralContext, failureMode);
>     return result;
> }
> 
>Index: Source/JavaScriptCore/heap/IsoSubspace.h
>===================================================================
>--- Source/JavaScriptCore/heap/IsoSubspace.h	(revision 227273)
>+++ Source/JavaScriptCore/heap/IsoSubspace.h	(working copy)
>@@ -41,11 +41,11 @@ public:
> 
>     size_t size() const { return m_size; }
> 
>-    BlockDirectory* allocatorFor(size_t, AllocatorForMode) override;
>-    BlockDirectory* allocatorForNonVirtual(size_t, AllocatorForMode);
>+    Allocator allocatorFor(size_t, AllocatorForMode) override;
>+    Allocator allocatorForNonVirtual(size_t, AllocatorForMode);
> 
>-    void* allocate(size_t, GCDeferralContext*, AllocationFailureMode) override;
>-    JS_EXPORT_PRIVATE void* allocateNonVirtual(size_t, GCDeferralContext*, AllocationFailureMode);
>+    void* allocate(VM&, size_t, GCDeferralContext*, AllocationFailureMode) override;
>+    JS_EXPORT_PRIVATE void* allocateNonVirtual(VM&, size_t, GCDeferralContext*, AllocationFailureMode);
> 
> private:
>     friend class IsoCellSet;
>@@ -56,14 +56,15 @@ private:
>     
>     size_t m_size;
>     BlockDirectory m_directory;
>+    Allocator m_allocator;
>     std::unique_ptr<IsoAlignedMemoryAllocator> m_isoAlignedMemoryAllocator;
>     SentinelLinkedList<IsoCellSet, BasicRawSentinelNode<IsoCellSet>> m_cellSets;
> };
> 
>-inline BlockDirectory* IsoSubspace::allocatorForNonVirtual(size_t size, AllocatorForMode)
>+inline Allocator IsoSubspace::allocatorForNonVirtual(size_t size, AllocatorForMode)
> {
>     RELEASE_ASSERT(size == this->size());
>-    return &m_directory;
>+    return m_allocator;
> }
> 
> #define ISO_SUBSPACE_INIT(heap, heapCellType, type) ("Isolated " #type " Space", (heap), (heapCellType), sizeof(type))
>Index: Source/JavaScriptCore/heap/LocalAllocator.cpp
>===================================================================
>--- Source/JavaScriptCore/heap/LocalAllocator.cpp	(nonexistent)
>+++ Source/JavaScriptCore/heap/LocalAllocator.cpp	(working copy)
>@@ -0,0 +1,297 @@
>+/*
>+ * Copyright (C) 2018 Apple Inc. All rights reserved.
>+ *
>+ * Redistribution and use in source and binary forms, with or without
>+ * modification, are permitted provided that the following conditions
>+ * are met:
>+ * 1. Redistributions of source code must retain the above copyright
>+ *    notice, this list of conditions and the following disclaimer.
>+ * 2. Redistributions in binary form must reproduce the above copyright
>+ *    notice, this list of conditions and the following disclaimer in the
>+ *    documentation and/or other materials provided with the distribution.
>+ *
>+ * THIS SOFTWARE IS PROVIDED BY APPLE INC. ``AS IS'' AND ANY
>+ * EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
>+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
>+ * PURPOSE ARE DISCLAIMED.  IN NO EVENT SHALL APPLE INC. OR
>+ * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
>+ * EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
>+ * PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
>+ * PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY
>+ * OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
>+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
>+ * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 
>+ */
>+
>+#include "config.h"
>+#include "LocalAllocator.h"
>+
>+#include "AllocatingScope.h"
>+#include "LocalAllocatorInlines.h"
>+#include "Options.h"
>+
>+namespace JSC {
>+
>+LocalAllocator::LocalAllocator(BlockDirectory* directory)
>+    : m_directory(directory)
>+    , m_cellSize(directory->m_cellSize)
>+    , m_freeList(m_cellSize)
>+{
>+    auto locker = holdLock(directory->m_localAllocatorsLock);
>+    directory->m_localAllocators.append(this);
>+}
>+
>+LocalAllocator::LocalAllocator(LocalAllocator&& other)
>+    : m_directory(other.m_directory)
>+    , m_cellSize(other.m_cellSize)
>+    , m_freeList(WTFMove(other.m_freeList))
>+    , m_currentBlock(other.m_currentBlock)
>+    , m_lastActiveBlock(other.m_lastActiveBlock)
>+{
>+    other.reset();
>+    if (other.isOnList()) {
>+        auto locker = holdLock(m_directory->m_localAllocatorsLock);
>+        other.remove();
>+        m_directory->m_localAllocators.append(this);
>+    }
>+}
>+
>+void LocalAllocator::reset()
>+{
>+    m_freeList.clear();
>+    m_currentBlock = nullptr;
>+    m_lastActiveBlock = nullptr;
>+}
>+
>+LocalAllocator::~LocalAllocator()
>+{
>+    if (isOnList()) {
>+        auto locker = holdLock(m_directory->m_localAllocatorsLock);
>+        remove();
>+    }
>+    
>+    // Assert that this allocator isn't holding onto any memory. This is a valid assertion for the
>+    // following two use cases:
>+    // 
>+    // - Immortal TLC. Those destruct after the heap is done destructing, so they should not have
>+    //   any state left in them.
>+    //
>+    // - TLC owned by an object. Such a TLC gets destroyed after a GC flip during which we proved
>+    //   that it is not reachable. Therefore, the TLC should still be in a fully reset state at the
>+    //   time of destruction because for it to get into any other state, someone must have allocated
>+    //   in it (which is impossible because it's supposedly unreachable).
>+    bool ok = true;
>+    if (!m_freeList.allocationWillFail()) {
>+        dataLog("FATAL: ", RawPointer(this), "->~LocalAllocator has non-empty free-list.\n");
>+        ok = false;
>+    }
>+    if (m_currentBlock) {
>+        dataLog("FATAL: ", RawPointer(this), "->~LocalAllocator has non-null current block.\n");
>+        ok = false;
>+    }
>+    if (m_lastActiveBlock) {
>+        dataLog("FATAL: ", RawPointer(this), "->~LocalAllocator has non-null last active block.\n");
>+        ok = false;
>+    }
>+    RELEASE_ASSERT(ok);
>+}
>+
>+void LocalAllocator::stopAllocating()
>+{
>+    ASSERT(!m_lastActiveBlock);
>+    if (!m_currentBlock) {
>+        ASSERT(m_freeList.allocationWillFail());
>+        return;
>+    }
>+    
>+    m_currentBlock->stopAllocating(m_freeList);
>+    m_lastActiveBlock = m_currentBlock;
>+    m_currentBlock = nullptr;
>+    m_freeList.clear();
>+}
>+
>+void LocalAllocator::resumeAllocating()
>+{
>+    if (!m_lastActiveBlock)
>+        return;
>+
>+    m_lastActiveBlock->resumeAllocating(m_freeList);
>+    m_currentBlock = m_lastActiveBlock;
>+    m_lastActiveBlock = nullptr;
>+}
>+
>+void LocalAllocator::prepareForAllocation()
>+{
>+    reset();
>+}
>+
>+void LocalAllocator::stopAllocatingForGood()
>+{
>+    stopAllocating();
>+    reset();
>+}
>+
>+void* LocalAllocator::allocateSlowCase(GCDeferralContext* deferralContext, AllocationFailureMode failureMode)
>+{
>+    SuperSamplerScope superSamplerScope(false);
>+    Heap& heap = *m_directory->m_heap;
>+    ASSERT(heap.vm()->currentThreadIsHoldingAPILock());
>+    doTestCollectionsIfNeeded(deferralContext);
>+
>+    ASSERT(!m_directory->markedSpace().isIterating());
>+    heap.didAllocate(m_freeList.originalSize());
>+    
>+    didConsumeFreeList();
>+    
>+    AllocatingScope helpingHeap(heap);
>+
>+    heap.collectIfNecessaryOrDefer(deferralContext);
>+    
>+    // Goofy corner case: the GC called a callback and now this directory has a currentBlock. This only
>+    // happens when running WebKit tests, which inject a callback into the GC's finalization.
>+    if (UNLIKELY(m_currentBlock))
>+        return allocate(deferralContext, failureMode);
>+    
>+    void* result = tryAllocateWithoutCollecting();
>+    
>+    if (LIKELY(result != 0))
>+        return result;
>+    
>+    MarkedBlock::Handle* block = m_directory->tryAllocateBlock();
>+    if (!block) {
>+        if (failureMode == AllocationFailureMode::Assert)
>+            RELEASE_ASSERT_NOT_REACHED();
>+        else
>+            return nullptr;
>+    }
>+    m_directory->addBlock(block);
>+    result = allocateIn(block);
>+    ASSERT(result);
>+    return result;
>+}
>+
>+void LocalAllocator::didConsumeFreeList()
>+{
>+    if (m_currentBlock)
>+        m_currentBlock->didConsumeFreeList();
>+    
>+    m_freeList.clear();
>+    m_currentBlock = nullptr;
>+}
>+
>+void* LocalAllocator::tryAllocateWithoutCollecting()
>+{
>+    // FIXME: If we wanted this to be used for real multi-threaded allocations then we would have to
>+    // come up with some concurrency protocol here. That protocol would need to be able to handle:
>+    //
>+    // - The basic case of multiple LocalAllocators trying to do an allocationCursor search on the
>+    //   same bitvector. That probably needs the bitvector lock at least.
>+    //
>+    // - The harder case of some LocalAllocator triggering a steal from a different BlockDirectory
>+    //   via a search in the AlignedMemoryAllocator's list. Who knows what locks that needs.
>+    // 
>+    // One way to make this work is to have a single per-Heap lock that protects all mutator lock
>+    // allocation slow paths. That would probably be scalable enough for years. It would certainly be
>+    // for for using TLC allocation from JIT threads.
>+    // https://bugs.webkit.org/show_bug.cgi?id=181635
>+    
>+    SuperSamplerScope superSamplerScope(false);
>+    
>+    ASSERT(!m_currentBlock);
>+    ASSERT(m_freeList.allocationWillFail());
>+    
>+    for (;;) {
>+        MarkedBlock::Handle* block = m_directory->findBlockForAllocation();
>+        if (!block)
>+            break;
>+
>+        if (void* result = tryAllocateIn(block))
>+            return result;
>+    }
>+    
>+    if (Options::stealEmptyBlocksFromOtherAllocators()
>+        && (Options::tradeDestructorBlocks() || !m_directory->needsDestruction())) {
>+        if (MarkedBlock::Handle* block = m_directory->m_subspace->findEmptyBlockToSteal()) {
>+            RELEASE_ASSERT(block->alignedMemoryAllocator() == m_directory->m_subspace->alignedMemoryAllocator());
>+            
>+            block->sweep(nullptr);
>+            
>+            // It's good that this clears canAllocateButNotEmpty as well as all other bits,
>+            // because there is a remote chance that a block may have both canAllocateButNotEmpty
>+            // and empty set at the same time.
>+            block->removeFromDirectory();
>+            m_directory->addBlock(block);
>+            return allocateIn(block);
>+        }
>+    }
>+    
>+    return nullptr;
>+}
>+
>+void* LocalAllocator::allocateIn(MarkedBlock::Handle* block)
>+{
>+    void* result = tryAllocateIn(block);
>+    RELEASE_ASSERT(result);
>+    return result;
>+}
>+
>+void* LocalAllocator::tryAllocateIn(MarkedBlock::Handle* block)
>+{
>+    ASSERT(block);
>+    ASSERT(!block->isFreeListed());
>+    
>+    block->sweep(&m_freeList);
>+    
>+    // It's possible to stumble on a completely full block. Marking tries to retire these, but
>+    // that algorithm is racy and may forget to do it sometimes.
>+    if (m_freeList.allocationWillFail()) {
>+        ASSERT(block->isFreeListed());
>+        block->unsweepWithNoNewlyAllocated();
>+        ASSERT(!block->isFreeListed());
>+        ASSERT(!m_directory->isEmpty(NoLockingNecessary, block));
>+        ASSERT(!m_directory->isCanAllocateButNotEmpty(NoLockingNecessary, block));
>+        return nullptr;
>+    }
>+    
>+    m_currentBlock = block;
>+    
>+    void* result = m_freeList.allocate(
>+        [] () -> HeapCell* {
>+            RELEASE_ASSERT_NOT_REACHED();
>+            return nullptr;
>+        });
>+    m_directory->setIsEden(NoLockingNecessary, m_currentBlock, true);
>+    m_directory->markedSpace().didAllocateInBlock(m_currentBlock);
>+    return result;
>+}
>+
>+void LocalAllocator::doTestCollectionsIfNeeded(GCDeferralContext* deferralContext)
>+{
>+    if (!Options::slowPathAllocsBetweenGCs())
>+        return;
>+
>+    static unsigned allocationCount = 0;
>+    if (!allocationCount) {
>+        if (!m_directory->m_heap->isDeferred()) {
>+            if (deferralContext)
>+                deferralContext->m_shouldGC = true;
>+            else
>+                m_directory->m_heap->collectNow(Sync, CollectionScope::Full);
>+        }
>+    }
>+    if (++allocationCount >= Options::slowPathAllocsBetweenGCs())
>+        allocationCount = 0;
>+}
>+
>+bool LocalAllocator::isFreeListedCell(const void* target) const
>+{
>+    // This abomination exists to detect when an object is in the dead-but-not-destructed state.
>+    // Therefore, it's not even clear that this needs to do anything beyond returning "false", since
>+    // if we know that the block owning the object is free-listed, then it's impossible for any
>+    // objects to be in the dead-but-not-destructed state.
>+    // FIXME: Get rid of this abomination. https://bugs.webkit.org/show_bug.cgi?id=181655
>+    return m_freeList.contains(bitwise_cast<HeapCell*>(target));
>+}
>+
>+} // namespace JSC
>+
>Index: Source/JavaScriptCore/heap/LocalAllocator.h
>===================================================================
>--- Source/JavaScriptCore/heap/LocalAllocator.h	(nonexistent)
>+++ Source/JavaScriptCore/heap/LocalAllocator.h	(working copy)
>@@ -0,0 +1,84 @@
>+/*
>+ * Copyright (C) 2018 Apple Inc. All rights reserved.
>+ *
>+ * Redistribution and use in source and binary forms, with or without
>+ * modification, are permitted provided that the following conditions
>+ * are met:
>+ * 1. Redistributions of source code must retain the above copyright
>+ *    notice, this list of conditions and the following disclaimer.
>+ * 2. Redistributions in binary form must reproduce the above copyright
>+ *    notice, this list of conditions and the following disclaimer in the
>+ *    documentation and/or other materials provided with the distribution.
>+ *
>+ * THIS SOFTWARE IS PROVIDED BY APPLE INC. ``AS IS'' AND ANY
>+ * EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
>+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
>+ * PURPOSE ARE DISCLAIMED.  IN NO EVENT SHALL APPLE INC. OR
>+ * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
>+ * EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
>+ * PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
>+ * PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY
>+ * OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
>+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
>+ * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 
>+ */
>+
>+#pragma once
>+
>+#include "FreeList.h"
>+#include "MarkedBlock.h"
>+#include <wtf/Noncopyable.h>
>+
>+namespace JSC {
>+
>+class BlockDirectory;
>+class GCDeferralContext;
>+
>+class LocalAllocator : public BasicRawSentinelNode<LocalAllocator> {
>+    WTF_MAKE_NONCOPYABLE(LocalAllocator);
>+    
>+public:
>+    LocalAllocator(BlockDirectory*);
>+    LocalAllocator(LocalAllocator&&);
>+    ~LocalAllocator();
>+    
>+    void* allocate(GCDeferralContext*, AllocationFailureMode);
>+
>+    void stopAllocating();
>+    void prepareForAllocation();
>+    void resumeAllocating();
>+    void stopAllocatingForGood();
>+    
>+    static ptrdiff_t offsetOfFreeList();
>+    static ptrdiff_t offsetOfCellSize();
>+    
>+    bool isFreeListedCell(const void*) const;
>+
>+private:
>+    void reset();
>+    JS_EXPORT_PRIVATE void* allocateSlowCase(GCDeferralContext*, AllocationFailureMode failureMode);
>+    void didConsumeFreeList();
>+    void* tryAllocateWithoutCollecting();
>+    void* tryAllocateIn(MarkedBlock::Handle*);
>+    void* allocateIn(MarkedBlock::Handle*);
>+    ALWAYS_INLINE void doTestCollectionsIfNeeded(GCDeferralContext*);
>+
>+    BlockDirectory* m_directory;
>+    unsigned m_cellSize;
>+    FreeList m_freeList;
>+    MarkedBlock::Handle* m_currentBlock { nullptr };
>+    MarkedBlock::Handle* m_lastActiveBlock { nullptr };
>+};
>+
>+inline ptrdiff_t LocalAllocator::offsetOfFreeList()
>+{
>+    return OBJECT_OFFSETOF(LocalAllocator, m_freeList);
>+}
>+
>+inline ptrdiff_t LocalAllocator::offsetOfCellSize()
>+{
>+    return OBJECT_OFFSETOF(LocalAllocator, m_cellSize);
>+}
>+
>+} // namespace JSC
>+
>Index: Source/JavaScriptCore/heap/LocalAllocatorInlines.h
>===================================================================
>--- Source/JavaScriptCore/heap/LocalAllocatorInlines.h	(nonexistent)
>+++ Source/JavaScriptCore/heap/LocalAllocatorInlines.h	(working copy)
>@@ -0,0 +1,42 @@
>+/*
>+ * Copyright (C) 2018 Apple Inc. All rights reserved.
>+ *
>+ * Redistribution and use in source and binary forms, with or without
>+ * modification, are permitted provided that the following conditions
>+ * are met:
>+ * 1. Redistributions of source code must retain the above copyright
>+ *    notice, this list of conditions and the following disclaimer.
>+ * 2. Redistributions in binary form must reproduce the above copyright
>+ *    notice, this list of conditions and the following disclaimer in the
>+ *    documentation and/or other materials provided with the distribution.
>+ *
>+ * THIS SOFTWARE IS PROVIDED BY APPLE INC. ``AS IS'' AND ANY
>+ * EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
>+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
>+ * PURPOSE ARE DISCLAIMED.  IN NO EVENT SHALL APPLE INC. OR
>+ * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
>+ * EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
>+ * PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
>+ * PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY
>+ * OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
>+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
>+ * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 
>+ */
>+
>+#pragma once
>+
>+#include "LocalAllocator.h"
>+
>+namespace JSC {
>+
>+ALWAYS_INLINE void* LocalAllocator::allocate(GCDeferralContext* deferralContext, AllocationFailureMode failureMode)
>+{
>+    return m_freeList.allocate(
>+        [&] () -> HeapCell* {
>+            sanitizeStackForVM(m_directory->heap()->vm());
>+            return static_cast<HeapCell*>(allocateSlowCase(deferralContext, failureMode));
>+        });
>+}
>+
>+} // namespace JSC
>+
>Index: Source/JavaScriptCore/heap/MarkedSpace.cpp
>===================================================================
>--- Source/JavaScriptCore/heap/MarkedSpace.cpp	(revision 227273)
>+++ Source/JavaScriptCore/heap/MarkedSpace.cpp	(working copy)
>@@ -302,6 +302,16 @@ void MarkedSpace::stopAllocating()
>         });
> }
> 
>+void MarkedSpace::stopAllocatingForGood()
>+{
>+    ASSERT(!isIterating());
>+    forEachDirectory(
>+        [&] (BlockDirectory& directory) -> IterationStatus {
>+            directory.stopAllocatingForGood();
>+            return IterationStatus::Continue;
>+        });
>+}
>+
> void MarkedSpace::prepareForConservativeScan()
> {
>     m_largeAllocationsForThisCollectionBegin = m_largeAllocations.begin() + m_largeAllocationsOffsetForThisCollection;
>Index: Source/JavaScriptCore/heap/MarkedSpace.h
>===================================================================
>--- Source/JavaScriptCore/heap/MarkedSpace.h	(revision 227273)
>+++ Source/JavaScriptCore/heap/MarkedSpace.h	(working copy)
>@@ -94,7 +94,7 @@ public:
>     
>     Heap* heap() const { return m_heap; }
>     
>-    void lastChanceToFinalize(); // You must call stopAllocating before you call this.
>+    void lastChanceToFinalize(); // Must call stopAllocatingForGood first.
>     void freeMemory();
> 
>     static size_t optimalSizeFor(size_t);
>@@ -111,6 +111,7 @@ public:
>     void didFinishIterating();
> 
>     void stopAllocating();
>+    void stopAllocatingForGood();
>     void resumeAllocating(); // If we just stopped allocation but we didn't do a collection, we need to resume allocation.
>     
>     void prepareForMarking();
>Index: Source/JavaScriptCore/heap/SlotVisitor.cpp
>===================================================================
>--- Source/JavaScriptCore/heap/SlotVisitor.cpp	(revision 227273)
>+++ Source/JavaScriptCore/heap/SlotVisitor.cpp	(working copy)
>@@ -37,6 +37,7 @@
> #include "JSObject.h"
> #include "JSString.h"
> #include "JSCInlines.h"
>+#include "MarkingConstraintSolver.h"
> #include "SlotVisitorInlines.h"
> #include "StopIfNecessaryTimer.h"
> #include "SuperSampler.h"
>Index: Source/JavaScriptCore/heap/SlotVisitor.h
>===================================================================
>--- Source/JavaScriptCore/heap/SlotVisitor.h	(revision 227273)
>+++ Source/JavaScriptCore/heap/SlotVisitor.h	(working copy)
>@@ -31,6 +31,7 @@
> #include "VisitRaceKey.h"
> #include <wtf/Forward.h>
> #include <wtf/MonotonicTime.h>
>+#include <wtf/SharedTask.h>
> #include <wtf/text/CString.h>
> 
> namespace JSC {
>@@ -41,6 +42,7 @@ class Heap;
> class HeapCell;
> class HeapSnapshotBuilder;
> class MarkedBlock;
>+class MarkingConstraint;
> class MarkingConstraintSolver;
> class UnconditionalFinalizer;
> template<typename T> class Weak;
>Index: Source/JavaScriptCore/heap/Subspace.h
>===================================================================
>--- Source/JavaScriptCore/heap/Subspace.h	(revision 227273)
>+++ Source/JavaScriptCore/heap/Subspace.h	(working copy)
>@@ -27,6 +27,7 @@
> 
> #include "AllocationFailureMode.h"
> #include "AllocatorForMode.h"
>+#include "Allocator.h"
> #include "MarkedBlock.h"
> #include "MarkedSpace.h"
> #include <wtf/text/CString.h>
>@@ -56,8 +57,8 @@ public:
>     void finishSweep(MarkedBlock::Handle&, FreeList*);
>     void destroy(VM&, JSCell*);
> 
>-    virtual BlockDirectory* allocatorFor(size_t, AllocatorForMode) = 0;
>-    virtual void* allocate(size_t, GCDeferralContext*, AllocationFailureMode) = 0;
>+    virtual Allocator allocatorFor(size_t, AllocatorForMode) = 0;
>+    virtual void* allocate(VM&, size_t, GCDeferralContext*, AllocationFailureMode) = 0;
>     
>     void prepareForAllocation();
>     
>Index: Source/JavaScriptCore/heap/ThreadLocalCache.cpp
>===================================================================
>--- Source/JavaScriptCore/heap/ThreadLocalCache.cpp	(nonexistent)
>+++ Source/JavaScriptCore/heap/ThreadLocalCache.cpp	(working copy)
>@@ -0,0 +1,121 @@
>+/*
>+ * Copyright (C) 2018 Apple Inc. All rights reserved.
>+ *
>+ * Redistribution and use in source and binary forms, with or without
>+ * modification, are permitted provided that the following conditions
>+ * are met:
>+ * 1. Redistributions of source code must retain the above copyright
>+ *    notice, this list of conditions and the following disclaimer.
>+ * 2. Redistributions in binary form must reproduce the above copyright
>+ *    notice, this list of conditions and the following disclaimer in the
>+ *    documentation and/or other materials provided with the distribution.
>+ *
>+ * THIS SOFTWARE IS PROVIDED BY APPLE INC. ``AS IS'' AND ANY
>+ * EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
>+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
>+ * PURPOSE ARE DISCLAIMED.  IN NO EVENT SHALL APPLE INC. OR
>+ * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
>+ * EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
>+ * PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
>+ * PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY
>+ * OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
>+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
>+ * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 
>+ */
>+
>+#include "config.h"
>+#include "ThreadLocalCache.h"
>+
>+#include "ThreadLocalCacheInlines.h"
>+#include "ThreadLocalCacheLayout.h"
>+#include <wtf/StdLibExtras.h>
>+
>+namespace JSC {
>+
>+RefPtr<ThreadLocalCache> ThreadLocalCache::create(Heap& heap)
>+{
>+    return adoptRef(new ThreadLocalCache(heap));
>+}
>+
>+ThreadLocalCache::ThreadLocalCache(Heap& heap)
>+    : m_heap(heap)
>+{
>+    m_data = allocateData();
>+}
>+
>+ThreadLocalCache::~ThreadLocalCache()
>+{
>+    destroyData(m_data);
>+}
>+
>+ThreadLocalCache::Data* ThreadLocalCache::allocateData()
>+{
>+    size_t oldSize = m_data ? m_data->size : 0;
>+    ThreadLocalCacheLayout::Snapshot layout = m_heap.threadLocalCacheLayout().snapshot();
>+    
>+    Data* result = static_cast<Data*>(fastMalloc(OBJECT_OFFSETOF(Data, allocator) + layout.size));
>+    result->size = layout.size;
>+    result->cache = this;
>+    for (size_t offset = 0; offset < oldSize; offset += sizeof(LocalAllocator))
>+        new (&allocator(*result, offset)) LocalAllocator(WTFMove(allocator(*m_data, offset)));
>+    for (size_t offset = oldSize; offset < layout.size; offset += sizeof(LocalAllocator))
>+        new (&allocator(*result, offset)) LocalAllocator(layout.directories[offset / sizeof(LocalAllocator)]);
>+    return result;
>+}
>+
>+void ThreadLocalCache::destroyData(Data* data)
>+{
>+    for (size_t offset = 0; offset < data->size; offset += sizeof(LocalAllocator))
>+        allocator(*data, offset).~LocalAllocator();
>+    fastFree(data);
>+}
>+
>+void ThreadLocalCache::installSlow(VM& vm)
>+{
>+#if ENABLE(FAST_TLS_JIT)
>+    static std::once_flag onceFlag;
>+    std::call_once(
>+        onceFlag,
>+        [] () {
>+            pthread_key_init_np(tlsKey, destructor);
>+        });
>+#endif
>+    
>+    ref();
>+    
>+    if (RefPtr<ThreadLocalCache> oldCache = get(vm))
>+        oldCache->deref();
>+    
>+    installData(vm, m_data);
>+}
>+
>+void ThreadLocalCache::installData(VM& vm, Data* data)
>+{
>+#if ENABLE(FAST_TLS_JIT)
>+    UNUSED_PARAM(vm);
>+    _pthread_setspecific_direct(tlsKey, data);
>+#else
>+    vm.threadLocalCacheData = data;
>+#endif
>+}
>+
>+LocalAllocator& ThreadLocalCache::allocatorSlow(VM& vm, size_t offset)
>+{
>+    Data* oldData = m_data;
>+    m_data = allocateData();
>+    destroyData(oldData);
>+    installData(vm, m_data);
>+    RELEASE_ASSERT(offset < m_data->size);
>+    return allocator(*m_data, offset);
>+}
>+
>+void ThreadLocalCache::destructor(void* arg)
>+{
>+    if (!arg)
>+        return;
>+    Data* data = static_cast<Data*>(arg);
>+    data->cache->deref();
>+}
>+
>+} // namespace JSC
>+
>Index: Source/JavaScriptCore/heap/ThreadLocalCache.h
>===================================================================
>--- Source/JavaScriptCore/heap/ThreadLocalCache.h	(nonexistent)
>+++ Source/JavaScriptCore/heap/ThreadLocalCache.h	(working copy)
>@@ -0,0 +1,101 @@
>+/*
>+ * Copyright (C) 2018 Apple Inc. All rights reserved.
>+ *
>+ * Redistribution and use in source and binary forms, with or without
>+ * modification, are permitted provided that the following conditions
>+ * are met:
>+ * 1. Redistributions of source code must retain the above copyright
>+ *    notice, this list of conditions and the following disclaimer.
>+ * 2. Redistributions in binary form must reproduce the above copyright
>+ *    notice, this list of conditions and the following disclaimer in the
>+ *    documentation and/or other materials provided with the distribution.
>+ *
>+ * THIS SOFTWARE IS PROVIDED BY APPLE INC. ``AS IS'' AND ANY
>+ * EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
>+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
>+ * PURPOSE ARE DISCLAIMED.  IN NO EVENT SHALL APPLE INC. OR
>+ * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
>+ * EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
>+ * PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
>+ * PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY
>+ * OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
>+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
>+ * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 
>+ */
>+
>+#pragma once
>+
>+#include <wtf/FastMalloc.h>
>+#include <wtf/FastTLS.h>
>+#include <wtf/ThreadSafeRefCounted.h>
>+
>+namespace JSC {
>+
>+class Heap;
>+class VM;
>+
>+class ThreadLocalCache : public ThreadSafeRefCounted<ThreadLocalCache> {
>+    WTF_MAKE_NONCOPYABLE(ThreadLocalCache);
>+    WTF_MAKE_FAST_ALLOCATED;
>+    
>+public:
>+    static RefPtr<ThreadLocalCache> create(Heap&);
>+    
>+    JS_EXPORT_PRIVATE ~ThreadLocalCache();
>+
>+    static RefPtr<ThreadLocalCache> get(VM&);
>+    
>+    // This is designed to be fast enough that you could even call it before every allocation, by
>+    // optimizing for the case that you're just installing the cache that is already installed. This
>+    // assumes a relatively small number of caches or low chance of actual context switch combined
>+    // with possibly high rate of "I may have context switched" sites that call this out of paranoia.
>+    void install(VM&);
>+    
>+    static LocalAllocator& allocator(VM&, size_t offset);
>+    
>+    template<typename SuccessFunc, typename FailureFunc>
>+    static void tryGetAllocator(VM&, size_t offset, const SuccessFunc&, const FailureFunc&);
>+    
>+    static ptrdiff_t offsetOfSize() { return OBJECT_OFFSETOF(Data, size); }
>+    static ptrdiff_t offsetOfFirstAllocator() { return OBJECT_OFFSETOF(Data, allocator); }
>+    
>+private:
>+    friend class VM;
>+    
>+    ThreadLocalCache(Heap&);
>+    
>+    struct Data {
>+        size_t size;
>+        ThreadLocalCache* cache;
>+        LocalAllocator allocator[1];
>+    };
>+
>+    static Data* getImpl(VM&);
>+    
>+    Data* allocateData();
>+    static void destroyData(Data*);
>+    static LocalAllocator& allocator(Data& data, size_t offset);
>+
>+    void installSlow(VM&);
>+    static void installData(VM&, Data*);
>+    
>+    LocalAllocator& allocatorSlow(VM&, size_t offset);
>+    
>+    static void destructor(void*);
>+    
>+    // When installed, we ref() the cache. Uninstalling deref()s the cache. TLS destruction deref()s
>+    // the cache. When the cache is destructed, it needs to return all of its state to the GC. I
>+    // think that just means stopAllocating(), but with some seriously heavy caveats having to do
>+    // with when it's valid to make that call. Alternatively, we could RELEASE_ASSERT that the cache
>+    // is empty when we destruct it. This is correct for caches that are kept live by GC objects and
>+    // it's correct for immortal caches.
>+    Heap& m_heap;
>+    Data* m_data { nullptr };
>+
>+#if ENABLE(FAST_TLS_JIT)
>+    static const pthread_key_t tlsKey = WTF_GC_TLC_KEY;
>+#endif
>+};
>+
>+} // namespace JSC
>+
>Index: Source/JavaScriptCore/heap/ThreadLocalCacheInlines.h
>===================================================================
>--- Source/JavaScriptCore/heap/ThreadLocalCacheInlines.h	(nonexistent)
>+++ Source/JavaScriptCore/heap/ThreadLocalCacheInlines.h	(working copy)
>@@ -0,0 +1,81 @@
>+/*
>+ * Copyright (C) 2018 Apple Inc. All rights reserved.
>+ *
>+ * Redistribution and use in source and binary forms, with or without
>+ * modification, are permitted provided that the following conditions
>+ * are met:
>+ * 1. Redistributions of source code must retain the above copyright
>+ *    notice, this list of conditions and the following disclaimer.
>+ * 2. Redistributions in binary form must reproduce the above copyright
>+ *    notice, this list of conditions and the following disclaimer in the
>+ *    documentation and/or other materials provided with the distribution.
>+ *
>+ * THIS SOFTWARE IS PROVIDED BY APPLE INC. ``AS IS'' AND ANY
>+ * EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
>+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
>+ * PURPOSE ARE DISCLAIMED.  IN NO EVENT SHALL APPLE INC. OR
>+ * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
>+ * EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
>+ * PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
>+ * PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY
>+ * OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
>+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
>+ * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 
>+ */
>+
>+#pragma once
>+
>+#include "ThreadLocalCache.h"
>+
>+namespace JSC {
>+
>+inline ThreadLocalCache::Data* ThreadLocalCache::getImpl(VM& vm)
>+{
>+#if ENABLE(FAST_TLS_JIT)
>+    UNUSED_PARAM(vm);
>+    return static_cast<Data*>(_pthread_getspecific_direct(tlsKey));
>+#else
>+    return vm.threadLocalCacheData;
>+#endif
>+}
>+
>+inline RefPtr<ThreadLocalCache> ThreadLocalCache::get(VM& vm)
>+{
>+    ThreadLocalCache::Data* data = getImpl(vm);
>+    if (data)
>+        return data->cache;
>+    return nullptr;
>+}
>+
>+inline void ThreadLocalCache::install(VM& vm)
>+{
>+    if (getImpl(vm) == m_data)
>+        return;
>+    installSlow(vm);
>+}
>+
>+inline LocalAllocator& ThreadLocalCache::allocator(VM& vm, size_t offset)
>+{
>+    ThreadLocalCache::Data* data = getImpl(vm);
>+    if (LIKELY(offset < data->size))
>+        return allocator(*data, offset);
>+    return data->cache->allocatorSlow(vm, offset);
>+}
>+
>+template<typename SuccessFunc, typename FailureFunc>
>+void ThreadLocalCache::tryGetAllocator(VM& vm, size_t offset, const SuccessFunc& successFunc, const FailureFunc& failureFunc)
>+{
>+    ThreadLocalCache::Data* data = getImpl(vm);
>+    if (LIKELY(offset < data->size))
>+        successFunc(allocator(*data, offset));
>+    else
>+        failureFunc();
>+}
>+
>+inline LocalAllocator& ThreadLocalCache::allocator(Data& data, size_t offset)
>+{
>+    return *bitwise_cast<LocalAllocator*>(bitwise_cast<char*>(&data.allocator[0]) + offset);
>+}
>+
>+} // namespace JSC
>+
>Index: Source/JavaScriptCore/heap/ThreadLocalCacheLayout.cpp
>===================================================================
>--- Source/JavaScriptCore/heap/ThreadLocalCacheLayout.cpp	(nonexistent)
>+++ Source/JavaScriptCore/heap/ThreadLocalCacheLayout.cpp	(working copy)
>@@ -0,0 +1,65 @@
>+/*
>+ * Copyright (C) 2018 Apple Inc. All rights reserved.
>+ *
>+ * Redistribution and use in source and binary forms, with or without
>+ * modification, are permitted provided that the following conditions
>+ * are met:
>+ * 1. Redistributions of source code must retain the above copyright
>+ *    notice, this list of conditions and the following disclaimer.
>+ * 2. Redistributions in binary form must reproduce the above copyright
>+ *    notice, this list of conditions and the following disclaimer in the
>+ *    documentation and/or other materials provided with the distribution.
>+ *
>+ * THIS SOFTWARE IS PROVIDED BY APPLE INC. ``AS IS'' AND ANY
>+ * EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
>+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
>+ * PURPOSE ARE DISCLAIMED.  IN NO EVENT SHALL APPLE INC. OR
>+ * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
>+ * EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
>+ * PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
>+ * PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY
>+ * OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
>+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
>+ * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 
>+ */
>+
>+#include "config.h"
>+#include "ThreadLocalCacheLayout.h"
>+
>+#include "BlockDirectory.h"
>+
>+namespace JSC {
>+
>+ThreadLocalCacheLayout::ThreadLocalCacheLayout()
>+{
>+}
>+
>+ThreadLocalCacheLayout::~ThreadLocalCacheLayout()
>+{
>+}
>+
>+void ThreadLocalCacheLayout::allocateOffset(BlockDirectory* directory)
>+{
>+    auto locker = holdLock(m_lock);
>+    directory->m_tlcOffset = m_size;
>+    m_size += sizeof(LocalAllocator);
>+    m_directories.append(directory);
>+}
>+
>+ThreadLocalCacheLayout::Snapshot ThreadLocalCacheLayout::snapshot()
>+{
>+    auto locker = holdLock(m_lock);
>+    Snapshot result;
>+    result.size = m_size;
>+    result.directories = m_directories;
>+    return result;
>+}
>+
>+BlockDirectory* ThreadLocalCacheLayout::directory(unsigned offset)
>+{
>+    auto locker = holdLock(m_lock);
>+    return m_directories[offset / sizeof(LocalAllocator)];
>+}
>+
>+} // namespace JSC
>+
>Index: Source/JavaScriptCore/heap/ThreadLocalCacheLayout.h
>===================================================================
>--- Source/JavaScriptCore/heap/ThreadLocalCacheLayout.h	(nonexistent)
>+++ Source/JavaScriptCore/heap/ThreadLocalCacheLayout.h	(working copy)
>@@ -0,0 +1,67 @@
>+/*
>+ * Copyright (C) 2018 Apple Inc. All rights reserved.
>+ *
>+ * Redistribution and use in source and binary forms, with or without
>+ * modification, are permitted provided that the following conditions
>+ * are met:
>+ * 1. Redistributions of source code must retain the above copyright
>+ *    notice, this list of conditions and the following disclaimer.
>+ * 2. Redistributions in binary form must reproduce the above copyright
>+ *    notice, this list of conditions and the following disclaimer in the
>+ *    documentation and/or other materials provided with the distribution.
>+ *
>+ * THIS SOFTWARE IS PROVIDED BY APPLE INC. ``AS IS'' AND ANY
>+ * EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
>+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
>+ * PURPOSE ARE DISCLAIMED.  IN NO EVENT SHALL APPLE INC. OR
>+ * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
>+ * EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
>+ * PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
>+ * PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY
>+ * OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
>+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
>+ * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 
>+ */
>+
>+#pragma once
>+
>+#include <wtf/FastMalloc.h>
>+#include <wtf/Lock.h>
>+#include <wtf/Noncopyable.h>
>+#include <wtf/Vector.h>
>+
>+namespace JSC {
>+
>+class BlockDirectory;
>+
>+// Each Heap has a ThreadLocalCacheLayout that helps us figure out how to allocate a ThreadLocalCache
>+// for that Heap.
>+
>+class ThreadLocalCacheLayout {
>+    WTF_MAKE_NONCOPYABLE(ThreadLocalCacheLayout);
>+    WTF_MAKE_FAST_ALLOCATED;
>+    
>+public:
>+    ThreadLocalCacheLayout();
>+    ~ThreadLocalCacheLayout();
>+
>+    // BlockDirectory calls this during creation and this fills in BlockDirectory::offset.
>+    void allocateOffset(BlockDirectory*);
>+    
>+    struct Snapshot {
>+        size_t size;
>+        Vector<BlockDirectory*> directories;
>+    };
>+    
>+    Snapshot snapshot();
>+    
>+    BlockDirectory* directory(unsigned offset);
>+    
>+private:
>+    Lock m_lock;
>+    size_t m_size { 0 };
>+    Vector<BlockDirectory*> m_directories;
>+};
>+
>+} // namespace JSC
>+
>Index: Source/JavaScriptCore/jit/AssemblyHelpers.cpp
>===================================================================
>--- Source/JavaScriptCore/jit/AssemblyHelpers.cpp	(revision 227273)
>+++ Source/JavaScriptCore/jit/AssemblyHelpers.cpp	(working copy)
>@@ -1,5 +1,5 @@
> /*
>- * Copyright (C) 2011-2017 Apple Inc. All rights reserved.
>+ * Copyright (C) 2011-2018 Apple Inc. All rights reserved.
>  *
>  * Redistribution and use in source and binary forms, with or without
>  * modification, are permitted provided that the following conditions
>@@ -582,6 +582,103 @@ void AssemblyHelpers::emitRandomThunk(VM
> }
> #endif
> 
>+void AssemblyHelpers::emitAllocateWithNonNullAllocator(GPRReg resultGPR, const JITAllocator& allocator, GPRReg allocatorGPR, GPRReg scratchGPR, JumpList& slowPath)
>+{
>+    // NOTE: This is carefully written so that we can call it while we disallow scratch
>+    // register usage.
>+        
>+    if (Options::forceGCSlowPaths()) {
>+        slowPath.append(jump());
>+        return;
>+    }
>+    
>+    Jump popPath;
>+    Jump done;
>+    
>+#if ENABLE(FAST_TLS_JIT)
>+    loadFromTLSPtr(fastTLSOffsetForKey(WTF_GC_TLC_KEY), scratchGPR);
>+#else
>+    loadPtr(&vm().threadLocalCacheData, scratchGPR);
>+#endif
>+    if (allocator.isConstant()) {
>+        slowPath.append(branch32(BelowOrEqual, Address(scratchGPR, ThreadLocalCache::offsetOfSize()), TrustedImm32(allocator.allocator().offset())));
>+        addPtr(TrustedImm32(ThreadLocalCache::offsetOfFirstAllocator() + allocator.allocator().offset()), scratchGPR, allocatorGPR);
>+    } else {
>+        slowPath.append(branch32(BelowOrEqual, Address(scratchGPR, ThreadLocalCache::offsetOfSize()), allocatorGPR));
>+        addPtr(TrustedImm32(ThreadLocalCache::offsetOfFirstAllocator()), allocatorGPR);
>+        addPtr(scratchGPR, allocatorGPR);
>+    }
>+
>+    load32(Address(allocatorGPR, LocalAllocator::offsetOfFreeList() + FreeList::offsetOfRemaining()), resultGPR);
>+    popPath = branchTest32(Zero, resultGPR);
>+    if (allocator.isConstant())
>+        add32(TrustedImm32(-allocator.allocator().cellSize(vm().heap)), resultGPR, scratchGPR);
>+    else {
>+        if (isX86()) {
>+            move(resultGPR, scratchGPR);
>+            sub32(Address(allocatorGPR, LocalAllocator::offsetOfCellSize()), scratchGPR);
>+        } else {
>+            load32(Address(allocatorGPR, LocalAllocator::offsetOfCellSize()), scratchGPR);
>+            sub32(resultGPR, scratchGPR, scratchGPR);
>+        }
>+    }
>+    negPtr(resultGPR);
>+    store32(scratchGPR, Address(allocatorGPR, LocalAllocator::offsetOfFreeList() + FreeList::offsetOfRemaining()));
>+    Address payloadEndAddr = Address(allocatorGPR, LocalAllocator::offsetOfFreeList() + FreeList::offsetOfPayloadEnd());
>+    if (isX86())
>+        addPtr(payloadEndAddr, resultGPR);
>+    else {
>+        loadPtr(payloadEndAddr, scratchGPR);
>+        addPtr(scratchGPR, resultGPR);
>+    }
>+        
>+    done = jump();
>+        
>+    popPath.link(this);
>+        
>+    loadPtr(Address(allocatorGPR, LocalAllocator::offsetOfFreeList() + FreeList::offsetOfScrambledHead()), resultGPR);
>+    if (isX86())
>+        xorPtr(Address(allocatorGPR, LocalAllocator::offsetOfFreeList() + FreeList::offsetOfSecret()), resultGPR);
>+    else {
>+        loadPtr(Address(allocatorGPR, LocalAllocator::offsetOfFreeList() + FreeList::offsetOfSecret()), scratchGPR);
>+        xorPtr(scratchGPR, resultGPR);
>+    }
>+    slowPath.append(branchTestPtr(Zero, resultGPR));
>+        
>+    // The object is half-allocated: we have what we know is a fresh object, but
>+    // it's still on the GC's free list.
>+    loadPtr(Address(resultGPR), scratchGPR);
>+    storePtr(scratchGPR, Address(allocatorGPR, LocalAllocator::offsetOfFreeList() + FreeList::offsetOfScrambledHead()));
>+        
>+    done.link(this);
>+}
>+
>+void AssemblyHelpers::emitAllocate(GPRReg resultGPR, const JITAllocator& allocator, GPRReg allocatorGPR, GPRReg scratchGPR, JumpList& slowPath)
>+{
>+    if (allocator.isConstant()) {
>+        if (!allocator.allocator()) {
>+            slowPath.append(jump());
>+            return;
>+        }
>+    }
>+    emitAllocateWithNonNullAllocator(resultGPR, allocator, allocatorGPR, scratchGPR, slowPath);
>+}
>+
>+void AssemblyHelpers::emitAllocateVariableSized(GPRReg resultGPR, CompleteSubspace& subspace, GPRReg allocationSize, GPRReg scratchGPR1, GPRReg scratchGPR2, JumpList& slowPath)
>+{
>+    static_assert(!(MarkedSpace::sizeStep & (MarkedSpace::sizeStep - 1)), "MarkedSpace::sizeStep must be a power of two.");
>+    
>+    unsigned stepShift = getLSBSet(MarkedSpace::sizeStep);
>+    
>+    add32(TrustedImm32(MarkedSpace::sizeStep - 1), allocationSize, scratchGPR1);
>+    urshift32(TrustedImm32(stepShift), scratchGPR1);
>+    slowPath.append(branch32(Above, scratchGPR1, TrustedImm32(MarkedSpace::largeCutoff >> stepShift)));
>+    move(TrustedImmPtr(subspace.allocatorForSizeStep() - 1), scratchGPR2);
>+    load32(BaseIndex(scratchGPR2, scratchGPR1, TimesFour), scratchGPR1);
>+    
>+    emitAllocate(resultGPR, JITAllocator::variable(), scratchGPR1, scratchGPR2, slowPath);
>+}
>+
> void AssemblyHelpers::restoreCalleeSavesFromEntryFrameCalleeSavesBuffer(EntryFrame*& topEntryFrame)
> {
> #if NUMBER_OF_CALLEE_SAVES_REGISTERS > 0
>Index: Source/JavaScriptCore/jit/AssemblyHelpers.h
>===================================================================
>--- Source/JavaScriptCore/jit/AssemblyHelpers.h	(revision 227273)
>+++ Source/JavaScriptCore/jit/AssemblyHelpers.h	(working copy)
>@@ -1,5 +1,5 @@
> /*
>- * Copyright (C) 2011-2017 Apple Inc. All rights reserved.
>+ * Copyright (C) 2011-2018 Apple Inc. All rights reserved.
>  *
>  * Redistribution and use in source and binary forms, with or without
>  * modification, are permitted provided that the following conditions
>@@ -32,6 +32,7 @@
> #include "GPRInfo.h"
> #include "Heap.h"
> #include "InlineCallFrame.h"
>+#include "JITAllocator.h"
> #include "JITCode.h"
> #include "MacroAssembler.h"
> #include "MarkedSpace.h"
>@@ -59,6 +60,7 @@ public:
>     }
> 
>     CodeBlock* codeBlock() { return m_codeBlock; }
>+    VM& vm() { return *m_codeBlock->vm(); }
>     AssemblerType_T& assembler() { return m_assembler; }
> 
>     void checkStackPointerAlignment()
>@@ -1488,83 +1490,24 @@ public:
>     void emitRandomThunk(JSGlobalObject*, GPRReg scratch0, GPRReg scratch1, GPRReg scratch2, FPRReg result);
>     void emitRandomThunk(VM&, GPRReg scratch0, GPRReg scratch1, GPRReg scratch2, GPRReg scratch3, FPRReg result);
> #endif
>-
>+    
>     // Call this if you know that the value held in allocatorGPR is non-null. This DOES NOT mean
>     // that allocator is non-null; allocator can be null as a signal that we don't know what the
>-    // value of allocatorGPR is.
>-    void emitAllocateWithNonNullAllocator(GPRReg resultGPR, BlockDirectory* allocator, GPRReg allocatorGPR, GPRReg scratchGPR, JumpList& slowPath)
>-    {
>-        // NOTE: This is carefully written so that we can call it while we disallow scratch
>-        // register usage.
>-        
>-        if (Options::forceGCSlowPaths()) {
>-            slowPath.append(jump());
>-            return;
>-        }
>-        
>-        Jump popPath;
>-        Jump done;
>-        
>-        load32(Address(allocatorGPR, BlockDirectory::offsetOfFreeList() + FreeList::offsetOfRemaining()), resultGPR);
>-        popPath = branchTest32(Zero, resultGPR);
>-        if (allocator)
>-            add32(TrustedImm32(-allocator->cellSize()), resultGPR, scratchGPR);
>-        else {
>-            if (isX86()) {
>-                move(resultGPR, scratchGPR);
>-                sub32(Address(allocatorGPR, BlockDirectory::offsetOfCellSize()), scratchGPR);
>-            } else {
>-                load32(Address(allocatorGPR, BlockDirectory::offsetOfCellSize()), scratchGPR);
>-                sub32(resultGPR, scratchGPR, scratchGPR);
>-            }
>-        }
>-        negPtr(resultGPR);
>-        store32(scratchGPR, Address(allocatorGPR, BlockDirectory::offsetOfFreeList() + FreeList::offsetOfRemaining()));
>-        Address payloadEndAddr = Address(allocatorGPR, BlockDirectory::offsetOfFreeList() + FreeList::offsetOfPayloadEnd());
>-        if (isX86())
>-            addPtr(payloadEndAddr, resultGPR);
>-        else {
>-            loadPtr(payloadEndAddr, scratchGPR);
>-            addPtr(scratchGPR, resultGPR);
>-        }
>-        
>-        done = jump();
>-        
>-        popPath.link(this);
>-        
>-        loadPtr(Address(allocatorGPR, BlockDirectory::offsetOfFreeList() + FreeList::offsetOfScrambledHead()), resultGPR);
>-        if (isX86())
>-            xorPtr(Address(allocatorGPR, BlockDirectory::offsetOfFreeList() + FreeList::offsetOfSecret()), resultGPR);
>-        else {
>-            loadPtr(Address(allocatorGPR, BlockDirectory::offsetOfFreeList() + FreeList::offsetOfSecret()), scratchGPR);
>-            xorPtr(scratchGPR, resultGPR);
>-        }
>-        slowPath.append(branchTestPtr(Zero, resultGPR));
>-        
>-        // The object is half-allocated: we have what we know is a fresh object, but
>-        // it's still on the GC's free list.
>-        loadPtr(Address(resultGPR), scratchGPR);
>-        storePtr(scratchGPR, Address(allocatorGPR, BlockDirectory::offsetOfFreeList() + FreeList::offsetOfScrambledHead()));
>-        
>-        done.link(this);
>-    }
>+    // value of allocatorGPR is. Additionally, if the allocator is not null, then there is no need
>+    // to populate allocatorGPR - this code will ignore the contents of allocatorGPR.
>+    void emitAllocateWithNonNullAllocator(GPRReg resultGPR, const JITAllocator& allocator, GPRReg allocatorGPR, GPRReg scratchGPR, JumpList& slowPath);
>     
>-    void emitAllocate(GPRReg resultGPR, BlockDirectory* allocator, GPRReg allocatorGPR, GPRReg scratchGPR, JumpList& slowPath)
>-    {
>-        if (!allocator)
>-            slowPath.append(branchTestPtr(Zero, allocatorGPR));
>-        emitAllocateWithNonNullAllocator(resultGPR, allocator, allocatorGPR, scratchGPR, slowPath);
>-    }
>+    void emitAllocate(GPRReg resultGPR, const JITAllocator& allocator, GPRReg allocatorGPR, GPRReg scratchGPR, JumpList& slowPath);
>     
>     template<typename StructureType>
>-    void emitAllocateJSCell(GPRReg resultGPR, BlockDirectory* allocator, GPRReg allocatorGPR, StructureType structure, GPRReg scratchGPR, JumpList& slowPath)
>+    void emitAllocateJSCell(GPRReg resultGPR, const JITAllocator& allocator, GPRReg allocatorGPR, StructureType structure, GPRReg scratchGPR, JumpList& slowPath)
>     {
>         emitAllocate(resultGPR, allocator, allocatorGPR, scratchGPR, slowPath);
>         emitStoreStructureWithTypeInfo(structure, resultGPR, scratchGPR);
>     }
>     
>     template<typename StructureType, typename StorageType, typename MaskType>
>-    void emitAllocateJSObject(GPRReg resultGPR, BlockDirectory* allocator, GPRReg allocatorGPR, StructureType structure, StorageType storage, MaskType mask, GPRReg scratchGPR, JumpList& slowPath)
>+    void emitAllocateJSObject(GPRReg resultGPR, const JITAllocator& allocator, GPRReg allocatorGPR, StructureType structure, StorageType storage, MaskType mask, GPRReg scratchGPR, JumpList& slowPath)
>     {
>         emitAllocateJSCell(resultGPR, allocator, allocatorGPR, structure, scratchGPR, slowPath);
>         storePtr(storage, Address(resultGPR, JSObject::butterflyOffset()));
>@@ -1576,13 +1519,8 @@ public:
>         VM& vm, GPRReg resultGPR, StructureType structure, StorageType storage, MaskType mask, GPRReg scratchGPR1,
>         GPRReg scratchGPR2, JumpList& slowPath, size_t size)
>     {
>-        BlockDirectory* allocator = subspaceFor<ClassType>(vm)->allocatorForNonVirtual(size, AllocatorForMode::AllocatorIfExists);
>-        if (!allocator) {
>-            slowPath.append(jump());
>-            return;
>-        }
>-        move(TrustedImmPtr(allocator), scratchGPR1);
>-        emitAllocateJSObject(resultGPR, allocator, scratchGPR1, structure, storage, mask, scratchGPR2, slowPath);
>+        Allocator allocator = subspaceFor<ClassType>(vm)->allocatorForNonVirtual(size, AllocatorForMode::AllocatorIfExists);
>+        emitAllocateJSObject(resultGPR, JITAllocator::constant(allocator), scratchGPR1, structure, storage, mask, scratchGPR2, slowPath);
>     }
>     
>     template<typename ClassType, typename StructureType, typename StorageType, typename MaskType>
>@@ -1593,20 +1531,7 @@ public:
>     
>     // allocationSize can be aliased with any of the other input GPRs. If it's not aliased then it
>     // won't be clobbered.
>-    void emitAllocateVariableSized(GPRReg resultGPR, CompleteSubspace& subspace, GPRReg allocationSize, GPRReg scratchGPR1, GPRReg scratchGPR2, JumpList& slowPath)
>-    {
>-        static_assert(!(MarkedSpace::sizeStep & (MarkedSpace::sizeStep - 1)), "MarkedSpace::sizeStep must be a power of two.");
>-        
>-        unsigned stepShift = getLSBSet(MarkedSpace::sizeStep);
>-        
>-        add32(TrustedImm32(MarkedSpace::sizeStep - 1), allocationSize, scratchGPR1);
>-        urshift32(TrustedImm32(stepShift), scratchGPR1);
>-        slowPath.append(branch32(Above, scratchGPR1, TrustedImm32(MarkedSpace::largeCutoff >> stepShift)));
>-        move(TrustedImmPtr(subspace.allocatorForSizeStep() - 1), scratchGPR2);
>-        loadPtr(BaseIndex(scratchGPR2, scratchGPR1, timesPtr()), scratchGPR1);
>-        
>-        emitAllocate(resultGPR, nullptr, scratchGPR1, scratchGPR2, slowPath);
>-    }
>+    void emitAllocateVariableSized(GPRReg resultGPR, CompleteSubspace& subspace, GPRReg allocationSize, GPRReg scratchGPR1, GPRReg scratchGPR2, JumpList& slowPath);
>     
>     template<typename ClassType, typename StructureType>
>     void emitAllocateVariableSizedCell(VM& vm, GPRReg resultGPR, StructureType structure, GPRReg allocationSize, GPRReg scratchGPR1, GPRReg scratchGPR2, JumpList& slowPath)
>Index: Source/JavaScriptCore/jit/JITAllocator.h
>===================================================================
>--- Source/JavaScriptCore/jit/JITAllocator.h	(nonexistent)
>+++ Source/JavaScriptCore/jit/JITAllocator.h	(working copy)
>@@ -0,0 +1,88 @@
>+/*
>+ * Copyright (C) 2018 Apple Inc. All rights reserved.
>+ *
>+ * Redistribution and use in source and binary forms, with or without
>+ * modification, are permitted provided that the following conditions
>+ * are met:
>+ * 1. Redistributions of source code must retain the above copyright
>+ *    notice, this list of conditions and the following disclaimer.
>+ * 2. Redistributions in binary form must reproduce the above copyright
>+ *    notice, this list of conditions and the following disclaimer in the
>+ *    documentation and/or other materials provided with the distribution.
>+ *
>+ * THIS SOFTWARE IS PROVIDED BY APPLE INC. ``AS IS'' AND ANY
>+ * EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
>+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
>+ * PURPOSE ARE DISCLAIMED.  IN NO EVENT SHALL APPLE INC. OR
>+ * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
>+ * EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
>+ * PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
>+ * PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY
>+ * OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
>+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
>+ * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 
>+ */
>+
>+#pragma once
>+
>+#include "Allocator.h"
>+
>+namespace JSC {
>+
>+class JITAllocator {
>+public:
>+    enum Kind {
>+        Constant,
>+        Variable
>+    };
>+    
>+    JITAllocator() { }
>+    
>+    static JITAllocator constant(Allocator allocator)
>+    {
>+        JITAllocator result;
>+        result.m_kind = Constant;
>+        result.m_allocator = allocator;
>+        return result;
>+    }
>+    
>+    static JITAllocator variable()
>+    {
>+        JITAllocator result;
>+        result.m_kind = Variable;
>+        return result;
>+    }
>+    
>+    bool operator==(const JITAllocator& other) const
>+    {
>+        return m_kind == other.m_kind
>+            && m_allocator == other.m_allocator;
>+    }
>+    
>+    bool operator!=(const JITAllocator& other) const
>+    {
>+        return !(*this == other);
>+    }
>+    
>+    explicit operator bool() const
>+    {
>+        return *this != JITAllocator();
>+    }
>+    
>+    Kind kind() const { return m_kind; }
>+    bool isConstant() const { return m_kind == Constant; }
>+    bool isVariable() const { return m_kind == Variable; }
>+    
>+    Allocator allocator() const
>+    {
>+        RELEASE_ASSERT(isConstant());
>+        return m_allocator;
>+    }
>+    
>+private:
>+    Kind m_kind { Constant };
>+    Allocator m_allocator;
>+};
>+
>+} // namespace JSC
>+
>Index: Source/JavaScriptCore/jit/JITOpcodes32_64.cpp
>===================================================================
>--- Source/JavaScriptCore/jit/JITOpcodes32_64.cpp	(revision 227273)
>+++ Source/JavaScriptCore/jit/JITOpcodes32_64.cpp	(working copy)
>@@ -1,5 +1,5 @@
> /*
>- * Copyright (C) 2009-2017 Apple Inc. All rights reserved.
>+ * Copyright (C) 2009-2018 Apple Inc. All rights reserved.
>  * Copyright (C) 2010 Patrick Gansterer <paroga@paroga.com>
>  *
>  * Redistribution and use in source and binary forms, with or without
>@@ -79,22 +79,23 @@ void JIT::emit_op_new_object(Instruction
> {
>     Structure* structure = currentInstruction[3].u.objectAllocationProfile->structure();
>     size_t allocationSize = JSFinalObject::allocationSize(structure->inlineCapacity());
>-    BlockDirectory* allocator = subspaceFor<JSFinalObject>(*m_vm)->allocatorForNonVirtual(allocationSize, AllocatorForMode::AllocatorIfExists);
>+    Allocator allocator = subspaceFor<JSFinalObject>(*m_vm)->allocatorForNonVirtual(allocationSize, AllocatorForMode::AllocatorIfExists);
> 
>     RegisterID resultReg = returnValueGPR;
>     RegisterID allocatorReg = regT1;
>     RegisterID scratchReg = regT3;
> 
>-    move(TrustedImmPtr(allocator), allocatorReg);
>-    if (allocator)
>-        addSlowCase(Jump());
>-    JumpList slowCases;
>-    auto butterfly = TrustedImmPtr(nullptr);
>-    auto mask = TrustedImm32(0);
>-    emitAllocateJSObject(resultReg, allocator, allocatorReg, TrustedImmPtr(structure), butterfly, mask, scratchReg, slowCases);
>-    emitInitializeInlineStorage(resultReg, structure->inlineCapacity());
>-    addSlowCase(slowCases);
>-    emitStoreCell(currentInstruction[1].u.operand, resultReg);
>+    if (!allocator)
>+        addSlowCase(jump());
>+    else {
>+        JumpList slowCases;
>+        auto butterfly = TrustedImmPtr(nullptr);
>+        auto mask = TrustedImm32(0);
>+        emitAllocateJSObject(resultReg, JITAllocator::constant(allocator), allocatorReg, TrustedImmPtr(structure), butterfly, mask, scratchReg, slowCases);
>+        emitInitializeInlineStorage(resultReg, structure->inlineCapacity());
>+        addSlowCase(slowCases);
>+        emitStoreCell(currentInstruction[1].u.operand, resultReg);
>+    }
> }
> 
> void JIT::emitSlow_op_new_object(Instruction* currentInstruction, Vector<SlowCaseEntry>::iterator& iter)
>@@ -856,9 +857,9 @@ void JIT::emit_op_create_this(Instructio
>     addSlowCase(branch8(NotEqual, Address(calleeReg, JSCell::typeInfoTypeOffset()), TrustedImm32(JSFunctionType)));
>     loadPtr(Address(calleeReg, JSFunction::offsetOfRareData()), rareDataReg);
>     addSlowCase(branchTestPtr(Zero, rareDataReg));
>-    loadPtr(Address(rareDataReg, FunctionRareData::offsetOfObjectAllocationProfile() + ObjectAllocationProfile::offsetOfAllocator()), allocatorReg);
>+    load32(Address(rareDataReg, FunctionRareData::offsetOfObjectAllocationProfile() + ObjectAllocationProfile::offsetOfAllocator()), allocatorReg);
>     loadPtr(Address(rareDataReg, FunctionRareData::offsetOfObjectAllocationProfile() + ObjectAllocationProfile::offsetOfStructure()), structureReg);
>-    addSlowCase(branchTestPtr(Zero, allocatorReg));
>+    addSlowCase(branch32(Equal, allocatorReg, TrustedImm32(Allocator().offset())));
> 
>     loadPtr(cachedFunction, cachedFunctionReg);
>     Jump hasSeenMultipleCallees = branchPtr(Equal, cachedFunctionReg, TrustedImmPtr(JSCell::seenMultipleCalleeObjects()));
>@@ -868,7 +869,7 @@ void JIT::emit_op_create_this(Instructio
>     JumpList slowCases;
>     auto butterfly = TrustedImmPtr(nullptr);
>     auto mask = TrustedImm32(0);
>-    emitAllocateJSObject(resultReg, nullptr, allocatorReg, structureReg, butterfly, mask, scratchReg, slowCases);
>+    emitAllocateJSObject(resultReg, JITAllocator::variable(), allocatorReg, structureReg, butterfly, mask, scratchReg, slowCases);
>     addSlowCase(slowCases);
>     emitStoreCell(currentInstruction[1].u.operand, resultReg);
> }
>Index: Source/JavaScriptCore/jit/JITOpcodes.cpp
>===================================================================
>--- Source/JavaScriptCore/jit/JITOpcodes.cpp	(revision 227273)
>+++ Source/JavaScriptCore/jit/JITOpcodes.cpp	(working copy)
>@@ -1,5 +1,5 @@
> /*
>- * Copyright (C) 2009-2017 Apple Inc. All rights reserved.
>+ * Copyright (C) 2009-2018 Apple Inc. All rights reserved.
>  * Copyright (C) 2010 Patrick Gansterer <paroga@paroga.com>
>  *
>  * Redistribution and use in source and binary forms, with or without
>@@ -81,22 +81,23 @@ void JIT::emit_op_new_object(Instruction
> {
>     Structure* structure = currentInstruction[3].u.objectAllocationProfile->structure();
>     size_t allocationSize = JSFinalObject::allocationSize(structure->inlineCapacity());
>-    BlockDirectory* allocator = subspaceFor<JSFinalObject>(*m_vm)->allocatorForNonVirtual(allocationSize, AllocatorForMode::AllocatorIfExists);
>+    Allocator allocator = subspaceFor<JSFinalObject>(*m_vm)->allocatorForNonVirtual(allocationSize, AllocatorForMode::AllocatorIfExists);
> 
>     RegisterID resultReg = regT0;
>     RegisterID allocatorReg = regT1;
>     RegisterID scratchReg = regT2;
> 
>-    move(TrustedImmPtr(allocator), allocatorReg);
>-    if (allocator)
>-        addSlowCase(Jump());
>-    JumpList slowCases;
>-    auto butterfly = TrustedImmPtr(nullptr);
>-    auto mask = TrustedImm32(0);
>-    emitAllocateJSObject(resultReg, allocator, allocatorReg, TrustedImmPtr(structure), butterfly, mask, scratchReg, slowCases);
>-    emitInitializeInlineStorage(resultReg, structure->inlineCapacity());
>-    addSlowCase(slowCases);
>-    emitPutVirtualRegister(currentInstruction[1].u.operand);
>+    if (!allocator)
>+        addSlowCase(jump());
>+    else {
>+        JumpList slowCases;
>+        auto butterfly = TrustedImmPtr(nullptr);
>+        auto mask = TrustedImm32(0);
>+        emitAllocateJSObject(resultReg, JITAllocator::constant(allocator), allocatorReg, TrustedImmPtr(structure), butterfly, mask, scratchReg, slowCases);
>+        emitInitializeInlineStorage(resultReg, structure->inlineCapacity());
>+        addSlowCase(slowCases);
>+        emitPutVirtualRegister(currentInstruction[1].u.operand);
>+    }
> }
> 
> void JIT::emitSlow_op_new_object(Instruction* currentInstruction, Vector<SlowCaseEntry>::iterator& iter)
>@@ -766,9 +767,9 @@ void JIT::emit_op_create_this(Instructio
>     addSlowCase(branch8(NotEqual, Address(calleeReg, JSCell::typeInfoTypeOffset()), TrustedImm32(JSFunctionType)));
>     loadPtr(Address(calleeReg, JSFunction::offsetOfRareData()), rareDataReg);
>     addSlowCase(branchTestPtr(Zero, rareDataReg));
>-    loadPtr(Address(rareDataReg, FunctionRareData::offsetOfObjectAllocationProfile() + ObjectAllocationProfile::offsetOfAllocator()), allocatorReg);
>+    load32(Address(rareDataReg, FunctionRareData::offsetOfObjectAllocationProfile() + ObjectAllocationProfile::offsetOfAllocator()), allocatorReg);
>     loadPtr(Address(rareDataReg, FunctionRareData::offsetOfObjectAllocationProfile() + ObjectAllocationProfile::offsetOfStructure()), structureReg);
>-    addSlowCase(branchTestPtr(Zero, allocatorReg));
>+    addSlowCase(branch32(Equal, allocatorReg, TrustedImm32(Allocator().offset())));
> 
>     loadPtr(cachedFunction, cachedFunctionReg);
>     Jump hasSeenMultipleCallees = branchPtr(Equal, cachedFunctionReg, TrustedImmPtr(JSCell::seenMultipleCalleeObjects()));
>@@ -778,7 +779,7 @@ void JIT::emit_op_create_this(Instructio
>     JumpList slowCases;
>     auto butterfly = TrustedImmPtr(nullptr);
>     auto mask = TrustedImm32(0);
>-    emitAllocateJSObject(resultReg, nullptr, allocatorReg, structureReg, butterfly, mask, scratchReg, slowCases);
>+    emitAllocateJSObject(resultReg, JITAllocator::variable(), allocatorReg, structureReg, butterfly, mask, scratchReg, slowCases);
>     emitGetVirtualRegister(callee, scratchReg);
>     loadPtr(Address(scratchReg, JSFunction::offsetOfRareData()), scratchReg);
>     load32(Address(scratchReg, FunctionRareData::offsetOfObjectAllocationProfile() + ObjectAllocationProfile::offsetOfInlineCapacity()), scratchReg);
>Index: Source/JavaScriptCore/runtime/ButterflyInlines.h
>===================================================================
>--- Source/JavaScriptCore/runtime/ButterflyInlines.h	(revision 227273)
>+++ Source/JavaScriptCore/runtime/ButterflyInlines.h	(working copy)
>@@ -79,7 +79,7 @@ ALWAYS_INLINE unsigned Butterfly::optima
> inline Butterfly* Butterfly::createUninitialized(VM& vm, JSCell*, size_t preCapacity, size_t propertyCapacity, bool hasIndexingHeader, size_t indexingPayloadSizeInBytes)
> {
>     size_t size = totalSize(preCapacity, propertyCapacity, hasIndexingHeader, indexingPayloadSizeInBytes);
>-    void* base = vm.jsValueGigacageAuxiliarySpace.allocateNonVirtual(size, nullptr, AllocationFailureMode::Assert);
>+    void* base = vm.jsValueGigacageAuxiliarySpace.allocateNonVirtual(vm, size, nullptr, AllocationFailureMode::Assert);
>     Butterfly* result = fromBase(base, preCapacity, propertyCapacity);
>     return result;
> }
>@@ -87,7 +87,7 @@ inline Butterfly* Butterfly::createUnini
> inline Butterfly* Butterfly::tryCreate(VM& vm, JSCell*, size_t preCapacity, size_t propertyCapacity, bool hasIndexingHeader, const IndexingHeader& indexingHeader, size_t indexingPayloadSizeInBytes)
> {
>     size_t size = totalSize(preCapacity, propertyCapacity, hasIndexingHeader, indexingPayloadSizeInBytes);
>-    void* base = vm.jsValueGigacageAuxiliarySpace.allocateNonVirtual(size, nullptr, AllocationFailureMode::ReturnNull);
>+    void* base = vm.jsValueGigacageAuxiliarySpace.allocateNonVirtual(vm, size, nullptr, AllocationFailureMode::ReturnNull);
>     if (!base)
>         return nullptr;
>     Butterfly* result = fromBase(base, preCapacity, propertyCapacity);
>@@ -165,7 +165,7 @@ inline Butterfly* Butterfly::growArrayRi
>     void* theBase = base(0, propertyCapacity);
>     size_t oldSize = totalSize(0, propertyCapacity, hadIndexingHeader, oldIndexingPayloadSizeInBytes);
>     size_t newSize = totalSize(0, propertyCapacity, true, newIndexingPayloadSizeInBytes);
>-    void* newBase = vm.jsValueGigacageAuxiliarySpace.allocateNonVirtual(newSize, nullptr, AllocationFailureMode::ReturnNull);
>+    void* newBase = vm.jsValueGigacageAuxiliarySpace.allocateNonVirtual(vm, newSize, nullptr, AllocationFailureMode::ReturnNull);
>     if (!newBase)
>         return nullptr;
>     // FIXME: This probably shouldn't be a memcpy.
>Index: Source/JavaScriptCore/runtime/DirectArguments.cpp
>===================================================================
>--- Source/JavaScriptCore/runtime/DirectArguments.cpp	(revision 227273)
>+++ Source/JavaScriptCore/runtime/DirectArguments.cpp	(working copy)
>@@ -1,5 +1,5 @@
> /*
>- * Copyright (C) 2015-2017 Apple Inc. All rights reserved.
>+ * Copyright (C) 2015-2018 Apple Inc. All rights reserved.
>  *
>  * Redistribution and use in source and binary forms, with or without
>  * modification, are permitted provided that the following conditions
>@@ -118,7 +118,7 @@ void DirectArguments::overrideThings(VM&
>     putDirect(vm, vm.propertyNames->callee, m_callee.get(), static_cast<unsigned>(PropertyAttribute::DontEnum));
>     putDirect(vm, vm.propertyNames->iteratorSymbol, globalObject()->arrayProtoValuesFunction(), static_cast<unsigned>(PropertyAttribute::DontEnum));
>     
>-    void* backingStore = vm.gigacageAuxiliarySpace(m_mappedArguments.kind).allocateNonVirtual(mappedArgumentsSize(), nullptr, AllocationFailureMode::Assert);
>+    void* backingStore = vm.gigacageAuxiliarySpace(m_mappedArguments.kind).allocateNonVirtual(vm, mappedArgumentsSize(), nullptr, AllocationFailureMode::Assert);
>     bool* overrides = static_cast<bool*>(backingStore);
>     m_mappedArguments.set(vm, this, overrides);
>     for (unsigned i = m_length; i--;)
>Index: Source/JavaScriptCore/runtime/GenericArgumentsInlines.h
>===================================================================
>--- Source/JavaScriptCore/runtime/GenericArgumentsInlines.h	(revision 227273)
>+++ Source/JavaScriptCore/runtime/GenericArgumentsInlines.h	(working copy)
>@@ -1,5 +1,5 @@
> /*
>- * Copyright (C) 2015-2017 Apple Inc. All rights reserved.
>+ * Copyright (C) 2015-2018 Apple Inc. All rights reserved.
>  *
>  * Redistribution and use in source and binary forms, with or without
>  * modification, are permitted provided that the following conditions
>@@ -263,7 +263,7 @@ void GenericArguments<Type>::initModifie
>     RELEASE_ASSERT(!m_modifiedArgumentsDescriptor);
> 
>     if (argsLength) {
>-        void* backingStore = vm.gigacageAuxiliarySpace(m_modifiedArgumentsDescriptor.kind).allocateNonVirtual(WTF::roundUpToMultipleOf<8>(argsLength), nullptr, AllocationFailureMode::Assert);
>+        void* backingStore = vm.gigacageAuxiliarySpace(m_modifiedArgumentsDescriptor.kind).allocateNonVirtual(vm, WTF::roundUpToMultipleOf<8>(argsLength), nullptr, AllocationFailureMode::Assert);
>         bool* modifiedArguments = static_cast<bool*>(backingStore);
>         m_modifiedArgumentsDescriptor.set(vm, this, modifiedArguments);
>         for (unsigned i = argsLength; i--;)
>Index: Source/JavaScriptCore/runtime/HashMapImpl.h
>===================================================================
>--- Source/JavaScriptCore/runtime/HashMapImpl.h	(revision 227273)
>+++ Source/JavaScriptCore/runtime/HashMapImpl.h	(working copy)
>@@ -1,5 +1,5 @@
> /*
>- * Copyright (C) 2016-2017 Apple Inc. All rights reserved.
>+ * Copyright (C) 2016-2018 Apple Inc. All rights reserved.
>  *
>  * Redistribution and use in source and binary forms, with or without
>  * modification, are permitted provided that the following conditions
>@@ -207,7 +207,7 @@ public:
>     {
>         auto scope = DECLARE_THROW_SCOPE(vm);
>         size_t allocationSize = HashMapBuffer::allocationSize(capacity);
>-        void* data = vm.jsValueGigacageAuxiliarySpace.allocateNonVirtual(allocationSize, nullptr, AllocationFailureMode::ReturnNull);
>+        void* data = vm.jsValueGigacageAuxiliarySpace.allocateNonVirtual(vm, allocationSize, nullptr, AllocationFailureMode::ReturnNull);
>         if (!data) {
>             throwOutOfMemoryError(exec, scope);
>             return nullptr;
>Index: Source/JavaScriptCore/runtime/JSArray.cpp
>===================================================================
>--- Source/JavaScriptCore/runtime/JSArray.cpp	(revision 227273)
>+++ Source/JavaScriptCore/runtime/JSArray.cpp	(working copy)
>@@ -1,6 +1,6 @@
> /*
>  *  Copyright (C) 1999-2000 Harri Porten (porten@kde.org)
>- *  Copyright (C) 2003-2017 Apple Inc. All rights reserved.
>+ *  Copyright (C) 2003-2018 Apple Inc. All rights reserved.
>  *  Copyright (C) 2003 Peter Kelly (pmk@post.com)
>  *  Copyright (C) 2006 Alexey Proskuryakov (ap@nypop.com)
>  *
>@@ -81,7 +81,10 @@ JSArray* JSArray::tryCreateUninitialized
>             || hasContiguous(indexingType));
> 
>         unsigned vectorLength = Butterfly::optimalContiguousVectorLength(structure, initialLength);
>-        void* temp = vm.jsValueGigacageAuxiliarySpace.allocateNonVirtual(Butterfly::totalSize(0, outOfLineStorage, true, vectorLength * sizeof(EncodedJSValue)), deferralContext, AllocationFailureMode::ReturnNull);
>+        void* temp = vm.jsValueGigacageAuxiliarySpace.allocateNonVirtual(
>+            vm,
>+            Butterfly::totalSize(0, outOfLineStorage, true, vectorLength * sizeof(EncodedJSValue)),
>+            deferralContext, AllocationFailureMode::ReturnNull);
>         if (UNLIKELY(!temp))
>             return nullptr;
>         butterfly = Butterfly::fromBase(temp, 0, outOfLineStorage);
>@@ -98,7 +101,10 @@ JSArray* JSArray::tryCreateUninitialized
>     } else {
>         static const unsigned indexBias = 0;
>         unsigned vectorLength = ArrayStorage::optimalVectorLength(indexBias, structure, initialLength);
>-        void* temp = vm.jsValueGigacageAuxiliarySpace.allocateNonVirtual(Butterfly::totalSize(indexBias, outOfLineStorage, true, ArrayStorage::sizeFor(vectorLength)), deferralContext, AllocationFailureMode::ReturnNull);
>+        void* temp = vm.jsValueGigacageAuxiliarySpace.allocateNonVirtual(
>+            vm,
>+            Butterfly::totalSize(indexBias, outOfLineStorage, true, ArrayStorage::sizeFor(vectorLength)),
>+            deferralContext, AllocationFailureMode::ReturnNull);
>         if (UNLIKELY(!temp))
>             return nullptr;
>         butterfly = Butterfly::fromBase(temp, indexBias, outOfLineStorage);
>@@ -368,7 +374,8 @@ bool JSArray::unshiftCountSlowCase(const
>         allocatedNewStorage = false;
>     } else {
>         size_t newSize = Butterfly::totalSize(0, propertyCapacity, true, ArrayStorage::sizeFor(desiredCapacity));
>-        newAllocBase = vm.jsValueGigacageAuxiliarySpace.allocateNonVirtual(newSize, nullptr, AllocationFailureMode::ReturnNull);
>+        newAllocBase = vm.jsValueGigacageAuxiliarySpace.allocateNonVirtual(
>+            vm, newSize, nullptr, AllocationFailureMode::ReturnNull);
>         if (!newAllocBase)
>             return false;
>         newStorageCapacity = desiredCapacity;
>Index: Source/JavaScriptCore/runtime/JSArray.h
>===================================================================
>--- Source/JavaScriptCore/runtime/JSArray.h	(revision 227273)
>+++ Source/JavaScriptCore/runtime/JSArray.h	(working copy)
>@@ -1,6 +1,6 @@
> /*
>  *  Copyright (C) 1999-2000 Harri Porten (porten@kde.org)
>- *  Copyright (C) 2003-2017 Apple Inc. All rights reserved.
>+ *  Copyright (C) 2003-2018 Apple Inc. All rights reserved.
>  *
>  *  This library is free software; you can redistribute it and/or
>  *  modify it under the terms of the GNU Lesser General Public
>@@ -238,6 +238,7 @@ inline JSArray* JSArray::tryCreate(VM& v
> 
>         unsigned vectorLength = Butterfly::optimalContiguousVectorLength(structure, vectorLengthHint);
>         void* temp = vm.jsValueGigacageAuxiliarySpace.allocateNonVirtual(
>+            vm,
>             Butterfly::totalSize(0, outOfLineStorage, true, vectorLength * sizeof(EncodedJSValue)),
>             nullptr, AllocationFailureMode::ReturnNull);
>         if (!temp)
>Index: Source/JavaScriptCore/runtime/JSArrayBufferView.cpp
>===================================================================
>--- Source/JavaScriptCore/runtime/JSArrayBufferView.cpp	(revision 227273)
>+++ Source/JavaScriptCore/runtime/JSArrayBufferView.cpp	(working copy)
>@@ -66,7 +66,7 @@ JSArrayBufferView::ConstructionContext::
>         void* temp;
>         size_t size = sizeOf(length, elementSize);
>         if (size) {
>-            temp = vm.primitiveGigacageAuxiliarySpace.allocateNonVirtual(size, nullptr, AllocationFailureMode::ReturnNull);
>+            temp = vm.primitiveGigacageAuxiliarySpace.allocateNonVirtual(vm, size, nullptr, AllocationFailureMode::ReturnNull);
>             if (!temp)
>                 return;
>         } else
>Index: Source/JavaScriptCore/runtime/JSCellInlines.h
>===================================================================
>--- Source/JavaScriptCore/runtime/JSCellInlines.h	(revision 227273)
>+++ Source/JavaScriptCore/runtime/JSCellInlines.h	(working copy)
>@@ -1,5 +1,5 @@
> /*
>- * Copyright (C) 2012-2017 Apple Inc. All rights reserved.
>+ * Copyright (C) 2012-2018 Apple Inc. All rights reserved.
>  *
>  * Redistribution and use in source and binary forms, with or without
>  * modification, are permitted provided that the following conditions
>@@ -145,14 +145,15 @@ CompleteSubspace* JSCell::subspaceFor(VM
> template<typename T>
> ALWAYS_INLINE void* tryAllocateCellHelper(Heap& heap, size_t size, GCDeferralContext* deferralContext, AllocationFailureMode failureMode)
> {
>+    VM& vm = *heap.vm();
>     ASSERT(deferralContext || !DisallowGC::isInEffectOnCurrentThread());
>     ASSERT(size >= sizeof(T));
>-    JSCell* result = static_cast<JSCell*>(subspaceFor<T>(*heap.vm())->allocateNonVirtual(size, deferralContext, failureMode));
>+    JSCell* result = static_cast<JSCell*>(subspaceFor<T>(vm)->allocateNonVirtual(vm, size, deferralContext, failureMode));
>     if (failureMode == AllocationFailureMode::ReturnNull && !result)
>         return nullptr;
> #if ENABLE(GC_VALIDATION)
>-    ASSERT(!heap.vm()->isInitializingObject());
>-    heap.vm()->setInitializingObjectClass(T::info());
>+    ASSERT(!vm.isInitializingObject());
>+    vm.setInitializingObjectClass(T::info());
> #endif
>     result->clearStructure();
>     return result;
>Index: Source/JavaScriptCore/runtime/JSGlobalObject.cpp
>===================================================================
>--- Source/JavaScriptCore/runtime/JSGlobalObject.cpp	(revision 227273)
>+++ Source/JavaScriptCore/runtime/JSGlobalObject.cpp	(working copy)
>@@ -335,7 +335,7 @@ static EncodedJSValue JSC_HOST_CALL enqu
>     return JSValue::encode(jsUndefined());
> }
> 
>-JSGlobalObject::JSGlobalObject(VM& vm, Structure* structure, const GlobalObjectMethodTable* globalObjectMethodTable)
>+JSGlobalObject::JSGlobalObject(VM& vm, Structure* structure, const GlobalObjectMethodTable* globalObjectMethodTable, RefPtr<ThreadLocalCache> threadLocalCache)
>     : Base(vm, structure, 0)
>     , m_vm(vm)
>     , m_masqueradesAsUndefinedWatchpoint(adoptRef(new WatchpointSet(IsWatched)))
>@@ -353,6 +353,7 @@ JSGlobalObject::JSGlobalObject(VM& vm, S
>     , m_templateRegistry(vm)
>     , m_runtimeFlags()
>     , m_globalObjectMethodTable(globalObjectMethodTable ? globalObjectMethodTable : &s_globalObjectMethodTable)
>+    , m_threadLocalCache(threadLocalCache ? WTFMove(threadLocalCache) : vm.defaultThreadLocalCache)
> {
> }
> 
>Index: Source/JavaScriptCore/runtime/JSGlobalObject.h
>===================================================================
>--- Source/JavaScriptCore/runtime/JSGlobalObject.h	(revision 227273)
>+++ Source/JavaScriptCore/runtime/JSGlobalObject.h	(working copy)
>@@ -491,7 +491,7 @@ public:
>     const RuntimeFlags& runtimeFlags() const { return m_runtimeFlags; }
> 
> protected:
>-    JS_EXPORT_PRIVATE explicit JSGlobalObject(VM&, Structure*, const GlobalObjectMethodTable* = 0);
>+    JS_EXPORT_PRIVATE explicit JSGlobalObject(VM&, Structure*, const GlobalObjectMethodTable* = 0, RefPtr<ThreadLocalCache> = nullptr);
> 
>     JS_EXPORT_PRIVATE void finishCreation(VM&);
> 
>@@ -893,6 +893,8 @@ public:
>     JSWrapperMap* wrapperMap() const { return m_wrapperMap.get(); }
>     void setWrapperMap(JSWrapperMap* map) { m_wrapperMap = map; }
> #endif
>+    
>+    ThreadLocalCache& threadLocalCache() const { return *m_threadLocalCache.get(); }
> 
> protected:
>     struct GlobalPropertyInfo {
>@@ -924,6 +926,8 @@ private:
> #if JSC_OBJC_API_ENABLED
>     RetainPtr<JSWrapperMap> m_wrapperMap;
> #endif
>+    
>+    RefPtr<ThreadLocalCache> m_threadLocalCache;
> };
> 
> JSGlobalObject* asGlobalObject(JSValue);
>Index: Source/JavaScriptCore/runtime/JSLock.cpp
>===================================================================
>--- Source/JavaScriptCore/runtime/JSLock.cpp	(revision 227273)
>+++ Source/JavaScriptCore/runtime/JSLock.cpp	(working copy)
>@@ -1,5 +1,5 @@
> /*
>- * Copyright (C) 2005-2017 Apple Inc. All rights reserved.
>+ * Copyright (C) 2005-2018 Apple Inc. All rights reserved.
>  *
>  * This library is free software; you can redistribute it and/or
>  * modify it under the terms of the GNU Library General Public
>@@ -28,6 +28,7 @@
> #include "JSCInlines.h"
> #include "MachineStackMarker.h"
> #include "SamplingProfiler.h"
>+#include "ThreadLocalCacheInlines.h"
> #include "WasmMachineThreads.h"
> #include <thread>
> #include <wtf/Threading.h>
>@@ -145,7 +146,9 @@ void JSLock::didAcquireLock()
> 
>     m_vm->setLastStackTop(thread.savedLastStackTop());
>     ASSERT(thread.stack().contains(m_vm->lastStackTop()));
>-
>+    
>+    m_vm->defaultThreadLocalCache->install(*m_vm);
>+    
>     m_vm->heap.machineThreads().addCurrentThread();
> #if ENABLE(WEBASSEMBLY)
>     Wasm::startTrackingCurrentThread();
>Index: Source/JavaScriptCore/runtime/Options.h
>===================================================================
>--- Source/JavaScriptCore/runtime/Options.h	(revision 227273)
>+++ Source/JavaScriptCore/runtime/Options.h	(working copy)
>@@ -236,6 +236,7 @@ constexpr bool enableAsyncIteration = fa
>     v(bool, dumpSizeClasses, false, Normal, nullptr) \
>     v(bool, useBumpAllocator, true, Normal, nullptr) \
>     v(bool, stealEmptyBlocksFromOtherAllocators, true, Normal, nullptr) \
>+    v(bool, tradeDestructorBlocks, true, Normal, nullptr) \
>     v(bool, eagerlyUpdateTopCallFrame, false, Normal, nullptr) \
>     \
>     v(bool, useOSREntryToDFG, true, Normal, nullptr) \
>Index: Source/JavaScriptCore/runtime/RegExpMatchesArray.h
>===================================================================
>--- Source/JavaScriptCore/runtime/RegExpMatchesArray.h	(revision 227273)
>+++ Source/JavaScriptCore/runtime/RegExpMatchesArray.h	(working copy)
>@@ -1,5 +1,5 @@
> /*
>- *  Copyright (C) 2008-2017 Apple Inc. All Rights Reserved.
>+ *  Copyright (C) 2008-2018 Apple Inc. All Rights Reserved.
>  *
>  *  This library is free software; you can redistribute it and/or
>  *  modify it under the terms of the GNU Lesser General Public
>@@ -42,7 +42,7 @@ ALWAYS_INLINE JSArray* tryCreateUninitia
> 
>     JSGlobalObject* globalObject = structure->globalObject();
>     bool createUninitialized = globalObject->isOriginalArrayStructure(structure);
>-    void* temp = vm.jsValueGigacageAuxiliarySpace.allocateNonVirtual(Butterfly::totalSize(0, structure->outOfLineCapacity(), true, vectorLength * sizeof(EncodedJSValue)), deferralContext, AllocationFailureMode::ReturnNull);
>+    void* temp = vm.jsValueGigacageAuxiliarySpace.allocateNonVirtual(vm, Butterfly::totalSize(0, structure->outOfLineCapacity(), true, vectorLength * sizeof(EncodedJSValue)), deferralContext, AllocationFailureMode::ReturnNull);
>     if (UNLIKELY(!temp))
>         return nullptr;
>     Butterfly* butterfly = Butterfly::fromBase(temp, 0, structure->outOfLineCapacity());
>Index: Source/JavaScriptCore/runtime/VM.cpp
>===================================================================
>--- Source/JavaScriptCore/runtime/VM.cpp	(revision 227273)
>+++ Source/JavaScriptCore/runtime/VM.cpp	(working copy)
>@@ -1,5 +1,5 @@
> /*
>- * Copyright (C) 2008-2017 Apple Inc. All rights reserved.
>+ * Copyright (C) 2008-2018 Apple Inc. All rights reserved.
>  *
>  * Redistribution and use in source and binary forms, with or without
>  * modification, are permitted provided that the following conditions
>@@ -121,6 +121,7 @@
> #include "StrongInlines.h"
> #include "StructureInlines.h"
> #include "TestRunnerUtils.h"
>+#include "ThreadLocalCacheInlines.h"
> #include "ThunkGenerators.h"
> #include "TypeProfiler.h"
> #include "TypeProfilerLog.h"
>@@ -306,6 +307,9 @@ VM::VM(VMType vmType, HeapType heapType)
>     updateSoftReservedZoneSize(Options::softReservedZoneSize());
>     setLastStackTop(stack.origin());
> 
>+    defaultThreadLocalCache = ThreadLocalCache::create(heap);
>+    defaultThreadLocalCache->install(*this);
>+
>     // Need to be careful to keep everything consistent here
>     JSLockHolder lock(this);
>     AtomicStringTable* existingEntryAtomicStringTable = Thread::current().setCurrentAtomicStringTable(m_atomicStringTable);
>@@ -383,7 +387,7 @@ VM::VM(VMType vmType, HeapType heapType)
> #endif
>     
>     Gigacage::addPrimitiveDisableCallback(primitiveGigacageDisabledCallback, this);
>-
>+    
>     heap.notifyIsSafeToCollect();
>     
>     LLInt::Data::performAssertions(*this);
>Index: Source/JavaScriptCore/runtime/VM.h
>===================================================================
>--- Source/JavaScriptCore/runtime/VM.h	(revision 227273)
>+++ Source/JavaScriptCore/runtime/VM.h	(working copy)
>@@ -54,6 +54,7 @@
> #include "TemplateRegistryKeyTable.h"
> #include "VMEntryRecord.h"
> #include "VMTraps.h"
>+#include "ThreadLocalCache.h"
> #include "WasmContext.h"
> #include "Watchpoint.h"
> #include <wtf/BumpPointerAllocator.h>
>@@ -656,6 +657,11 @@ public:
> 
>     JSObject* stringRecursionCheckFirstObject { nullptr };
>     HashSet<JSObject*> stringRecursionCheckVisitedObjects;
>+    
>+#if !ENABLE(FAST_TLS_JIT)
>+    ThreadLocalCache::Data* threadLocalCacheData { nullptr };
>+#endif
>+    RefPtr<ThreadLocalCache> defaultThreadLocalCache;
> 
>     LocalTimeOffsetCache localTimeOffsetCache;
> 
>Index: Source/JavaScriptCore/runtime/VMEntryScope.cpp
>===================================================================
>--- Source/JavaScriptCore/runtime/VMEntryScope.cpp	(revision 227273)
>+++ Source/JavaScriptCore/runtime/VMEntryScope.cpp	(working copy)
>@@ -1,5 +1,5 @@
> /*
>- * Copyright (C) 2013-2017 Apple Inc. All rights reserved.
>+ * Copyright (C) 2013-2018 Apple Inc. All rights reserved.
>  *
>  * Redistribution and use in source and binary forms, with or without
>  * modification, are permitted provided that the following conditions
>@@ -29,6 +29,7 @@
> #include "DisallowVMReentry.h"
> #include "Options.h"
> #include "SamplingProfiler.h"
>+#include "ThreadLocalCacheInlines.h"
> #include "VM.h"
> #include "Watchdog.h"
> #include <wtf/StackBounds.h>
>@@ -40,6 +41,7 @@ VMEntryScope::VMEntryScope(VM& vm, JSGlo
>     : m_vm(vm)
>     , m_globalObject(globalObject)
> {
>+    globalObject->threadLocalCache().install(vm);
>     ASSERT(!DisallowVMReentry::isInEffectOnCurrentThread());
>     ASSERT(Thread::current().stack().isGrowingDownward());
>     if (!vm.entryScope) {
>Index: Source/WTF/ChangeLog
>===================================================================
>--- Source/WTF/ChangeLog	(revision 227273)
>+++ Source/WTF/ChangeLog	(working copy)
>@@ -1,3 +1,12 @@
>+2018-01-20  Filip Pizlo  <fpizlo@apple.com>
>+
>+        JSC GC should support TLCs (thread local caches)
>+        https://bugs.webkit.org/show_bug.cgi?id=181559
>+
>+        Reviewed by NOBODY (OOPS!).
>+
>+        * wtf/Bitmap.h: Just fixing a compile error.
>+
> 2018-01-19  Alex Christensen  <achristensen@webkit.org>
> 
>         Fix jsc-only build
>Index: Source/WTF/wtf/Bitmap.h
>===================================================================
>--- Source/WTF/wtf/Bitmap.h	(revision 227273)
>+++ Source/WTF/wtf/Bitmap.h	(working copy)
>@@ -21,6 +21,7 @@
> 
> #include <array>
> #include <wtf/Atomics.h>
>+#include <wtf/HashFunctions.h>
> #include <wtf/StdLibExtras.h>
> #include <stdint.h>
> #include <string.h>

Actions: View | Formatted Diff | Diff

Attachments on bug 181559: 331267 | 331293 | 331308 | 331310 | 331344 | 331352 | 331796 | 331821 | 331846 | 331848 | 331850 | 331851 | 331853 | 331959 | 331961