Attachment 286617 Details for Bug 160125 – the patch

[patch] the patch

blah.patch (text/plain), 563.86 KB, created by Filip Pizlo on 2016-08-22 13:53:11 PDT

(hide)

Description:

Filename:

MIME Type:

Creator: Filip Pizlo

Created: 2016-08-22 13:53:11 PDT

Size: 563.86 KB

patch

obsolete

>Index: JSTests/ChangeLog
>===================================================================
>--- JSTests/ChangeLog	(revision 204735)
>+++ JSTests/ChangeLog	(working copy)
>@@ -1,3 +1,18 @@
>+2016-08-22  Filip Pizlo  <fpizlo@apple.com>
>+
>+        Butterflies should be allocated in Auxiliary MarkedSpace instead of CopiedSpace and we should rewrite as much of the GC as needed to make this not a regression
>+        https://bugs.webkit.org/show_bug.cgi?id=160125
>+
>+        Reviewed by Geoffrey Garen.
>+        
>+        Most of the things I did properly covered by existing tests, but I found some simple cases of
>+        unshifting that had sketchy coverage.
>+
>+        * stress/array-storage-array-unshift.js: Added.
>+        * stress/contiguous-array-unshift.js: Added.
>+        * stress/double-array-unshift.js: Added.
>+        * stress/int32-array-unshift.js: Added.
>+
> 2016-08-22  Yusuke Suzuki  <utatane.tea@gmail.com>
> 
>         [ES6] Module should not allow HTML comments
>Index: JSTests/stress/array-storage-array-unshift.js
>===================================================================
>--- JSTests/stress/array-storage-array-unshift.js	(nonexistent)
>+++ JSTests/stress/array-storage-array-unshift.js	(working copy)
>@@ -0,0 +1,8 @@
>+//@ runDefault
>+var x = [2.5, 1.5];
>+x.length = 1000000000;
>+x.length = 2;
>+Array.prototype.unshift.call(x, 3.5);
>+if (x.toString() != "3.5,2.5,1.5")
>+    throw "Error: bad result: " + describe(x);
>+
>Index: JSTests/stress/contiguous-array-unshift.js
>===================================================================
>--- JSTests/stress/contiguous-array-unshift.js	(nonexistent)
>+++ JSTests/stress/contiguous-array-unshift.js	(working copy)
>@@ -0,0 +1,6 @@
>+//@ runDefault
>+var x = ['b', 'a'];
>+Array.prototype.unshift.call(x, 'c');
>+if (x.toString() != "c,b,a")
>+    throw "Error: bad result: " + describe(x);
>+
>Index: JSTests/stress/double-array-unshift.js
>===================================================================
>--- JSTests/stress/double-array-unshift.js	(nonexistent)
>+++ JSTests/stress/double-array-unshift.js	(working copy)
>@@ -0,0 +1,6 @@
>+//@ runDefault
>+var x = [2.5, 1.5];
>+Array.prototype.unshift.call(x, 3.5);
>+if (x.toString() != "3.5,2.5,1.5")
>+    throw "Error: bad result: " + describe(x);
>+
>Index: JSTests/stress/int32-array-unshift.js
>===================================================================
>--- JSTests/stress/int32-array-unshift.js	(nonexistent)
>+++ JSTests/stress/int32-array-unshift.js	(working copy)
>@@ -0,0 +1,6 @@
>+//@ runDefault
>+var x = [2, 1];
>+Array.prototype.unshift.call(x, 3);
>+if (x.toString() != "3,2,1")
>+    throw "Error: bad result: " + describe(x);
>+
>Index: Source/JavaScriptCore/ChangeLog
>===================================================================
>--- Source/JavaScriptCore/ChangeLog	(revision 204735)
>+++ Source/JavaScriptCore/ChangeLog	(working copy)
>@@ -1,3 +1,780 @@
>+2016-08-22  Filip Pizlo  <fpizlo@apple.com>
>+
>+        Butterflies should be allocated in Auxiliary MarkedSpace instead of CopiedSpace and we should rewrite as much of the GC as needed to make this not a regression
>+        https://bugs.webkit.org/show_bug.cgi?id=160125
>+
>+        Reviewed by Geoffrey Garen.
>+
>+        In order to make the GC concurrent (bug 149432), we would either need to enable concurrent
>+        copying or we would need to not copy. Concurrent copying carries a 1-2% throughput overhead
>+        from the barriers alone. Considering that MarkedSpace does a decent job of avoiding
>+        fragmentation, it's unlikely that it's worth paying 1-2% throughput for copying. So, we want
>+        to get rid of copied space. This change moves copied space's biggest client over to marked
>+        space.
>+        
>+        Moving butterflies to marked space means having them use the new Auxiliary HeapCell
>+        allocation path. This is a fairly mechanical change, but it caused performance regressions
>+        everywhere, so this change also fixes MarkedSpace's performance issues.
>+        
>+        At a high level the mechanical changes are:
>+        
>+        - We use AuxiliaryBarrier instead of CopyBarrier.
>+        
>+        - We use tryAllocateAuxiliary instead of tryAllocateStorage. I got rid of the silly
>+          CheckedBoolean stuff, since it's so much more trouble than it's worth.
>+        
>+        - The JITs have to emit inlined marked space allocations instead of inline copy space
>+          allocations.
>+        
>+        - Everyone has to get used to zeroing their butterflies after allocation instead of relying
>+          on them being pre-zeroed by the GC. Copied space would zero things for you, while marked
>+          space doesn't.
>+        
>+        But this led to performance problems, which I fixed:
>+        
>+        - MarkedSpace always causes internal fragmentation. I got around this by making array
>+          allocations usually round up vectorLength to the maximum allowed by the size class that we
>+          would have allocated in. Also, ensureLengthSlow() and friends first make sure that the
>+          requested length can't just be fulfilled with the current allocation size. This safeguard
>+          means that not every array allocation has to do size class queries.
>+        
>+        - Size classes in MarkedSpace were way too precise, which led to external fragmentation. This
>+          changes MarkedSpace size classes to use a linear progression for very small sizes followed
>+          by a geometric progression that naturally transitions to a hyperbolic progression. We want
>+          hyperbolic sizes when we get close to blockSize: for example the largest size we want is
>+          payloadSize / 2 rounded down, to ensure we get exactly two cells with minimal slop. The
>+          next size down should be payloadSize / 3 rounded down, and so on. After the last precise
>+          size (80 bytes), we proceed using a geometric progression, but round up each size to
>+          minimize slop at the end of the block. This naturally causes the geometric progression to
>+          turn hyperbolic for large sizes. The size class configuration happens at VM start-up, so
>+          can be controlled with runtime options. I found that a base of 1.4 works pretty well.
>+        
>+        - Large allocations caused massive internal fragmentation, size the smallest large allocation
>+          had to use exactly blockSize, and the largest small allocation used blockSize / 2. The
>+          next size up - the first large allocation size to require two blocks - also had 50%
>+          internal fragmentation. This is because we required large allocations to be blockSize
>+          aligned, so that MarkedBlock::blockFor() would work. I decided to rewrite all of that.
>+          Cells no longer have to be owned by a MarkedBlock. They can now alternatively be owned by
>+          a LargeAllocation. These two things are abstracted as CellContainer. You know that a cell
>+          is owned by a LargeAllocation if the MarkedBlock::atomSize / 2 bit is set. Basically, large
>+          allocations are deliberately misaligned by 8 bytes. This actually works out great since
>+          (1) typed arrays won't use large allocations anyway since they have their own malloc
>+          fallback and (2) large array butterflies already have a 8 byte header, which means that the
>+          8 byte base misalignment aligns the large array payload on a 16 byte boundary. I took
>+          extreme care to make sure that the isLargeAllocation bit checks are as rare as possible;
>+          for example, ExecState::vm() skips the check because we know that callees must be small
>+          allocations. It's also possible to use template tricks to do one check for cell container
>+          kind, and then invoke a function specialized for MarkedBlock or a function specialized for
>+          LargeAllocation. LargeAllocation includes stubs for all MarkedBlock methods that get used
>+          from functions that are template-specialized like this. That's mostly to speed up the GC
>+          marking code. Most other code can use CellContainer API or HeapCell API directly. That's
>+          another thing: HeapCell, the common base of JSCell and auxiliary allocations, is now smart
>+          enough to do a lot of things for you, like HeapCell::vm(), HeapCell::heap(),
>+          HeapCell::isLargeAllocation(), and HeapCell::cellContainer().
>+        
>+        - Even after all of that, copying butterflies was still faster because it allowed us to skip
>+          sweeping dead space. A good GC allocates over dead bytes without explicitly freeing them,
>+          so the GC pause is O(size of live), not O(size of live + dead). O(dead) is usually much
>+          larger than O(live), especially in an eden collection. Copying satisfies this premise while
>+          mark+sweep does not. So, I invented a new kind of allocator: bump'n'pop. Previously, our
>+          MarkedSpace allocator was a freelist pop. That's simple and easy to inline but requires
>+          that we walk the block to build a free list. This means walking dead space. The new
>+          allocator allows totally free MarkedBlocks to simply set up a bump-pointer arena instead.
>+          The allocator is a hybrid of bump-pointer and freelist pop. It tries bump first. The bump
>+          pointer always bumps by cellSize, so the result of filling a block with bumping looks as if
>+          we had used freelist popping to fill it. Additionally, each MarkedBlock now has a bit to
>+          quickly tell if the block is entirely free. This makes sweeping O(1) in the common case.
>+          This change was enough to make this change a net win over tip-of-tree.
>+        
>+        - FTL now shares the same allocation fast paths as everything else, which is great, because
>+          bump'n'pop has gnarly control flow. We don't really want B3 to have to think about that
>+          control flow, since it won't be able to improve the machine code we write ourselves. GC
>+          fast paths are best written in assembly. So, I've empowered B3 to have even better support
>+          for Patchpoint terminals. It's now totally fine for a Patchpoint terminal to be non-Void.
>+          So, the new FTL allocation fast paths are just Patchpoint terminals that call through to
>+          AssemblyHelpers::emitAllocate(). B3 still reasons about things like constant-folding the
>+          size class calculation and constant-hoisting the allocator. Also, I gave the FTL the
>+          ability to constant-fold some allocator logic (in case we first assume that we're doing a
>+          variable-length allocation but then realize that the length is known). I think it makes
>+          sense to have constant folding rules in FTL::Output, or whatever the B3 IR builder is,
>+          since this makes lowering easier (you can constant fold during lowering more easily) and it
>+          reduces the amount of malloc traffic.
>+        
>+        All of this put together gives us a 0.5% speed-up on JetStream, no change on Octane, and no
>+        change on Kraken. It's a SunSpider regression, but I think I'm OK with that. By comparison,
>+        the alternative approach of using a copy barrier would have cost us 1-2%. That's the real
>+        apples-to-apples comparison if your premise is that we should have a concurrent GC. After
>+        this change, we are barrier-ready for concurrent GC: we already have a marking barrier and
>+        we simply won't need a copying barrier. If we keep copying, then getting barrier-ready would
>+        mean adding back the copy barrier, which costs more perf.
>+        
>+        We might get bigger speed-ups once we remove CopiedSpace altogether. That requires moving
>+        typed arrays and a few other weird things over to Aux MarkedSpace.
>+        
>+        This also includes some header sanitization. The introduction of AuxiliaryBarrier, HeapCell,
>+        and CellContainer meant that I had to include those files from everywhere. Fortunately,
>+        just including JSCInlines.h (instead of manually including the files that includes) is
>+        usually enough. So, I made most of JSC's cpp files include JSCInlines.h, which is something
>+        that we were already basically doing. In places where JSCInlines.h would be too much, I just
>+        included HeapInlines.h. This got weird, because we previously included HeapInlines.h from
>+        JSObject.h. That's bad because it led to some circular dependencies, so I fixed it - but that
>+        meant having to manually include HeapInlines.h from the places that previously got it
>+        implicitly via JSObject.h. But that led to more problems for some reason: I started getting
>+        build errors because non-JSC files were having trouble including Opcode.h. That's just silly,
>+        since Opcode.h is meant to be an internal JSC header. So, I made it an internal header and
>+        made it impossible to include it from outside JSC. This was a lot of work, but it was
>+        necessary to get the patch to build on all ports. It's also a net win. There were many places
>+        in WebCore that were transitively including a *ton* of JSC headers just because of the
>+        JSObject.h->HeapInlines.h edge and a bunch of dependency edges that arose from some public
>+        (for WebCore) JSC headers needing Interpreter.h or Opcode.h for bad reasons.
>+
>+        * API/JSTypedArray.cpp:
>+        * API/ObjCCallbackFunction.mm:
>+        * CMakeLists.txt:
>+        * JavaScriptCore.xcodeproj/project.pbxproj:
>+        * Scripts/builtins/builtins_generate_combined_implementation.py:
>+        (BuiltinsCombinedImplementationGenerator.generate_secondary_header_includes):
>+        * Scripts/builtins/builtins_generate_internals_wrapper_implementation.py:
>+        (BuiltinsInternalsWrapperImplementationGenerator.generate_secondary_header_includes):
>+        * Scripts/builtins/builtins_generate_separate_implementation.py:
>+        (BuiltinsSeparateImplementationGenerator.generate_secondary_header_includes):
>+        * assembler/AbstractMacroAssembler.h:
>+        (JSC::AbstractMacroAssembler::JumpList::JumpList):
>+        (JSC::AbstractMacroAssembler::JumpList::link):
>+        (JSC::AbstractMacroAssembler::JumpList::linkTo):
>+        (JSC::AbstractMacroAssembler::JumpList::append):
>+        * assembler/MacroAssemblerARM64.h:
>+        (JSC::MacroAssemblerARM64::add32):
>+        * b3/B3BasicBlock.cpp:
>+        (JSC::B3::BasicBlock::appendIntConstant):
>+        (JSC::B3::BasicBlock::appendBoolConstant):
>+        (JSC::B3::BasicBlock::clearSuccessors):
>+        * b3/B3BasicBlock.h:
>+        * b3/B3DuplicateTails.cpp:
>+        * b3/B3StackmapGenerationParams.h:
>+        * b3/testb3.cpp:
>+        (JSC::B3::testBranchBitAndImmFusion):
>+        (JSC::B3::testPatchpointTerminalReturnValue):
>+        (JSC::B3::zero):
>+        (JSC::B3::run):
>+        * bindings/ScriptValue.cpp:
>+        * bytecode/AdaptiveInferredPropertyValueWatchpointBase.cpp:
>+        * bytecode/LLIntPrototypeLoadAdaptiveStructureWatchpoint.cpp:
>+        * bytecode/ObjectAllocationProfile.h:
>+        (JSC::ObjectAllocationProfile::initialize):
>+        * bytecode/PolymorphicAccess.cpp:
>+        (JSC::AccessCase::generateImpl):
>+        * bytecode/StructureStubInfo.cpp:
>+        * dfg/DFGOperations.cpp:
>+        * dfg/DFGSpeculativeJIT.cpp:
>+        (JSC::DFG::SpeculativeJIT::emitAllocateRawObject):
>+        (JSC::DFG::SpeculativeJIT::compileMakeRope):
>+        (JSC::DFG::SpeculativeJIT::compileAllocatePropertyStorage):
>+        (JSC::DFG::SpeculativeJIT::compileReallocatePropertyStorage):
>+        * dfg/DFGSpeculativeJIT.h:
>+        (JSC::DFG::SpeculativeJIT::emitAllocateJSCell):
>+        (JSC::DFG::SpeculativeJIT::emitAllocateJSObject):
>+        * dfg/DFGSpeculativeJIT32_64.cpp:
>+        (JSC::DFG::SpeculativeJIT::compile):
>+        * dfg/DFGSpeculativeJIT64.cpp:
>+        (JSC::DFG::SpeculativeJIT::compile):
>+        * dfg/DFGStrengthReductionPhase.cpp:
>+        (JSC::DFG::StrengthReductionPhase::handleNode):
>+        * ftl/FTLAbstractHeapRepository.h:
>+        * ftl/FTLCompile.cpp:
>+        * ftl/FTLJITFinalizer.cpp:
>+        * ftl/FTLLowerDFGToB3.cpp:
>+        (JSC::FTL::DFG::LowerDFGToB3::compileCreateDirectArguments):
>+        (JSC::FTL::DFG::LowerDFGToB3::compileNewArrayWithSize):
>+        (JSC::FTL::DFG::LowerDFGToB3::compileMakeRope):
>+        (JSC::FTL::DFG::LowerDFGToB3::compileMaterializeNewObject):
>+        (JSC::FTL::DFG::LowerDFGToB3::initializeArrayElements):
>+        (JSC::FTL::DFG::LowerDFGToB3::allocatePropertyStorageWithSizeImpl):
>+        (JSC::FTL::DFG::LowerDFGToB3::emitRightShiftSnippet):
>+        (JSC::FTL::DFG::LowerDFGToB3::allocateHeapCell):
>+        (JSC::FTL::DFG::LowerDFGToB3::storeStructure):
>+        (JSC::FTL::DFG::LowerDFGToB3::allocateCell):
>+        (JSC::FTL::DFG::LowerDFGToB3::allocateObject):
>+        (JSC::FTL::DFG::LowerDFGToB3::allocatorForSize):
>+        (JSC::FTL::DFG::LowerDFGToB3::allocateVariableSizedObject):
>+        (JSC::FTL::DFG::LowerDFGToB3::allocateJSArray):
>+        * ftl/FTLOutput.cpp:
>+        (JSC::FTL::Output::constBool):
>+        (JSC::FTL::Output::constInt32):
>+        (JSC::FTL::Output::add):
>+        (JSC::FTL::Output::shl):
>+        (JSC::FTL::Output::aShr):
>+        (JSC::FTL::Output::lShr):
>+        (JSC::FTL::Output::zeroExt):
>+        (JSC::FTL::Output::equal):
>+        (JSC::FTL::Output::notEqual):
>+        (JSC::FTL::Output::above):
>+        (JSC::FTL::Output::aboveOrEqual):
>+        (JSC::FTL::Output::below):
>+        (JSC::FTL::Output::belowOrEqual):
>+        (JSC::FTL::Output::greaterThan):
>+        (JSC::FTL::Output::greaterThanOrEqual):
>+        (JSC::FTL::Output::lessThan):
>+        (JSC::FTL::Output::lessThanOrEqual):
>+        (JSC::FTL::Output::select):
>+        (JSC::FTL::Output::unreachable):
>+        (JSC::FTL::Output::appendSuccessor):
>+        (JSC::FTL::Output::speculate):
>+        (JSC::FTL::Output::addIncomingToPhi):
>+        * ftl/FTLOutput.h:
>+        * ftl/FTLValueFromBlock.h:
>+        (JSC::FTL::ValueFromBlock::ValueFromBlock):
>+        (JSC::FTL::ValueFromBlock::operator bool):
>+        (JSC::FTL::ValueFromBlock::value):
>+        (JSC::FTL::ValueFromBlock::block):
>+        * ftl/FTLWeightedTarget.h:
>+        (JSC::FTL::WeightedTarget::target):
>+        (JSC::FTL::WeightedTarget::weight):
>+        (JSC::FTL::WeightedTarget::frequentedBlock):
>+        * heap/CellContainer.h: Added.
>+        (JSC::CellContainer::CellContainer):
>+        (JSC::CellContainer::operator bool):
>+        (JSC::CellContainer::isMarkedBlock):
>+        (JSC::CellContainer::isLargeAllocation):
>+        (JSC::CellContainer::markedBlock):
>+        (JSC::CellContainer::largeAllocation):
>+        * heap/CellContainerInlines.h: Added.
>+        (JSC::CellContainer::isMarkedOrRetired):
>+        (JSC::CellContainer::isMarked):
>+        (JSC::CellContainer::isMarkedOrNewlyAllocated):
>+        (JSC::CellContainer::setHasAnyMarked):
>+        (JSC::CellContainer::cellSize):
>+        (JSC::CellContainer::weakSet):
>+        * heap/ConservativeRoots.cpp:
>+        (JSC::ConservativeRoots::ConservativeRoots):
>+        (JSC::ConservativeRoots::~ConservativeRoots):
>+        (JSC::ConservativeRoots::grow):
>+        (JSC::ConservativeRoots::genericAddPointer):
>+        (JSC::ConservativeRoots::genericAddSpan):
>+        * heap/ConservativeRoots.h:
>+        (JSC::ConservativeRoots::size):
>+        (JSC::ConservativeRoots::roots):
>+        * heap/CopyToken.h:
>+        * heap/FreeList.cpp: Added.
>+        (JSC::FreeList::dump):
>+        * heap/FreeList.h: Added.
>+        (JSC::FreeList::FreeList):
>+        (JSC::FreeList::list):
>+        (JSC::FreeList::bump):
>+        (JSC::FreeList::operator==):
>+        (JSC::FreeList::operator!=):
>+        (JSC::FreeList::operator bool):
>+        * heap/Heap.cpp:
>+        (JSC::Heap::Heap):
>+        (JSC::Heap::finalizeUnconditionalFinalizers):
>+        (JSC::Heap::markRoots):
>+        (JSC::Heap::copyBackingStores):
>+        (JSC::Heap::gatherStackRoots):
>+        (JSC::Heap::gatherJSStackRoots):
>+        (JSC::Heap::gatherScratchBufferRoots):
>+        (JSC::Heap::clearLivenessData):
>+        (JSC::Heap::visitSmallStrings):
>+        (JSC::Heap::visitConservativeRoots):
>+        (JSC::Heap::removeDeadCompilerWorklistEntries):
>+        (JSC::Heap::gatherExtraHeapSnapshotData):
>+        (JSC::Heap::removeDeadHeapSnapshotNodes):
>+        (JSC::Heap::visitProtectedObjects):
>+        (JSC::Heap::visitArgumentBuffers):
>+        (JSC::Heap::visitException):
>+        (JSC::Heap::visitStrongHandles):
>+        (JSC::Heap::visitHandleStack):
>+        (JSC::Heap::visitSamplingProfiler):
>+        (JSC::Heap::traceCodeBlocksAndJITStubRoutines):
>+        (JSC::Heap::converge):
>+        (JSC::Heap::visitWeakHandles):
>+        (JSC::Heap::updateObjectCounts):
>+        (JSC::Heap::clearUnmarkedExecutables):
>+        (JSC::Heap::deleteUnmarkedCompiledCode):
>+        (JSC::Heap::collectAllGarbage):
>+        (JSC::Heap::collect):
>+        (JSC::Heap::collectWithoutAnySweep):
>+        (JSC::Heap::collectImpl):
>+        (JSC::Heap::suspendCompilerThreads):
>+        (JSC::Heap::willStartCollection):
>+        (JSC::Heap::flushOldStructureIDTables):
>+        (JSC::Heap::flushWriteBarrierBuffer):
>+        (JSC::Heap::stopAllocation):
>+        (JSC::Heap::reapWeakHandles):
>+        (JSC::Heap::pruneStaleEntriesFromWeakGCMaps):
>+        (JSC::Heap::sweepArrayBuffers):
>+        (JSC::Heap::snapshotMarkedSpace):
>+        (JSC::Heap::deleteSourceProviderCaches):
>+        (JSC::Heap::notifyIncrementalSweeper):
>+        (JSC::Heap::writeBarrierCurrentlyExecutingCodeBlocks):
>+        (JSC::Heap::resetAllocators):
>+        (JSC::Heap::updateAllocationLimits):
>+        (JSC::Heap::didFinishCollection):
>+        (JSC::Heap::resumeCompilerThreads):
>+        (JSC::Zombify::visit):
>+        * heap/Heap.h:
>+        (JSC::Heap::subspaceForObjectDestructor):
>+        (JSC::Heap::subspaceForAuxiliaryData):
>+        (JSC::Heap::allocatorForObjectWithoutDestructor):
>+        (JSC::Heap::allocatorForObjectWithDestructor):
>+        (JSC::Heap::allocatorForAuxiliaryData):
>+        (JSC::Heap::storageAllocator):
>+        * heap/HeapCell.h:
>+        (JSC::HeapCell::zap):
>+        (JSC::HeapCell::isZapped):
>+        * heap/HeapCellInlines.h: Added.
>+        (JSC::HeapCell::isLargeAllocation):
>+        (JSC::HeapCell::cellContainer):
>+        (JSC::HeapCell::markedBlock):
>+        (JSC::HeapCell::largeAllocation):
>+        (JSC::HeapCell::heap):
>+        (JSC::HeapCell::vm):
>+        (JSC::HeapCell::cellSize):
>+        (JSC::HeapCell::allocatorAttributes):
>+        (JSC::HeapCell::destructionMode):
>+        (JSC::HeapCell::cellKind):
>+        * heap/HeapInlines.h:
>+        (JSC::Heap::isCollecting):
>+        (JSC::Heap::heap):
>+        (JSC::Heap::isLive):
>+        (JSC::Heap::isMarked):
>+        (JSC::Heap::testAndSetMarked):
>+        (JSC::Heap::setMarked):
>+        (JSC::Heap::cellSize):
>+        (JSC::Heap::writeBarrier):
>+        (JSC::Heap::allocateWithoutDestructor):
>+        (JSC::Heap::allocateObjectOfType):
>+        (JSC::Heap::subspaceForObjectOfType):
>+        (JSC::Heap::allocatorForObjectOfType):
>+        (JSC::Heap::allocateAuxiliary):
>+        (JSC::Heap::tryAllocateAuxiliary):
>+        (JSC::Heap::tryReallocateAuxiliary):
>+        (JSC::Heap::tryAllocateStorage):
>+        (JSC::Heap::didFreeBlock):
>+        (JSC::Heap::isPointerGCObject): Deleted.
>+        (JSC::Heap::isValueGCObject): Deleted.
>+        * heap/HeapUtil.h: Added.
>+        (JSC::HeapUtil::findGCObjectPointersForMarking):
>+        (JSC::HeapUtil::isPointerGCObjectJSCell):
>+        (JSC::HeapUtil::isValueGCObject):
>+        * heap/LargeAllocation.cpp: Added.
>+        (JSC::LargeAllocation::tryCreate):
>+        (JSC::LargeAllocation::LargeAllocation):
>+        (JSC::LargeAllocation::lastChanceToFinalize):
>+        (JSC::LargeAllocation::shrink):
>+        (JSC::LargeAllocation::visitWeakSet):
>+        (JSC::LargeAllocation::reapWeakSet):
>+        (JSC::LargeAllocation::clearMarks):
>+        (JSC::LargeAllocation::clearMarksWithCollectionType):
>+        (JSC::LargeAllocation::isEmpty):
>+        (JSC::LargeAllocation::sweep):
>+        (JSC::LargeAllocation::destroy):
>+        (JSC::LargeAllocation::dump):
>+        * heap/LargeAllocation.h: Added.
>+        (JSC::LargeAllocation::fromCell):
>+        (JSC::LargeAllocation::cell):
>+        (JSC::LargeAllocation::isLargeAllocation):
>+        (JSC::LargeAllocation::heap):
>+        (JSC::LargeAllocation::vm):
>+        (JSC::LargeAllocation::weakSet):
>+        (JSC::LargeAllocation::clearNewlyAllocated):
>+        (JSC::LargeAllocation::isNewlyAllocated):
>+        (JSC::LargeAllocation::isMarked):
>+        (JSC::LargeAllocation::isMarkedOrNewlyAllocated):
>+        (JSC::LargeAllocation::isLive):
>+        (JSC::LargeAllocation::hasValidCell):
>+        (JSC::LargeAllocation::cellSize):
>+        (JSC::LargeAllocation::aboveLowerBound):
>+        (JSC::LargeAllocation::belowUpperBound):
>+        (JSC::LargeAllocation::contains):
>+        (JSC::LargeAllocation::attributes):
>+        (JSC::LargeAllocation::testAndSetMarked):
>+        (JSC::LargeAllocation::setMarked):
>+        (JSC::LargeAllocation::clearMarked):
>+        (JSC::LargeAllocation::setHasAnyMarked):
>+        (JSC::LargeAllocation::headerSize):
>+        * heap/MarkedAllocator.cpp:
>+        (JSC::MarkedAllocator::MarkedAllocator):
>+        (JSC::isListPagedOut):
>+        (JSC::MarkedAllocator::isPagedOut):
>+        (JSC::MarkedAllocator::retire):
>+        (JSC::MarkedAllocator::tryAllocateWithoutCollectingImpl):
>+        (JSC::MarkedAllocator::tryAllocateWithoutCollecting):
>+        (JSC::MarkedAllocator::allocateSlowCase):
>+        (JSC::MarkedAllocator::tryAllocateSlowCase):
>+        (JSC::MarkedAllocator::allocateSlowCaseImpl):
>+        (JSC::blockHeaderSize):
>+        (JSC::MarkedAllocator::blockSizeForBytes):
>+        (JSC::MarkedAllocator::tryAllocateBlock):
>+        (JSC::MarkedAllocator::addBlock):
>+        (JSC::MarkedAllocator::removeBlock):
>+        (JSC::MarkedAllocator::reset):
>+        (JSC::MarkedAllocator::lastChanceToFinalize):
>+        (JSC::MarkedAllocator::setFreeList):
>+        (JSC::MarkedAllocator::tryAllocateHelper): Deleted.
>+        (JSC::MarkedAllocator::tryPopFreeList): Deleted.
>+        (JSC::MarkedAllocator::tryAllocate): Deleted.
>+        (JSC::MarkedAllocator::allocateBlock): Deleted.
>+        * heap/MarkedAllocator.h:
>+        (JSC::MarkedAllocator::destruction):
>+        (JSC::MarkedAllocator::cellKind):
>+        (JSC::MarkedAllocator::heap):
>+        (JSC::MarkedAllocator::takeLastActiveBlock):
>+        (JSC::MarkedAllocator::offsetOfFreeList):
>+        (JSC::MarkedAllocator::offsetOfCellSize):
>+        (JSC::MarkedAllocator::tryAllocate):
>+        (JSC::MarkedAllocator::allocate):
>+        (JSC::MarkedAllocator::stopAllocating):
>+        (JSC::MarkedAllocator::resumeAllocating):
>+        (JSC::MarkedAllocator::offsetOfFreeListHead): Deleted.
>+        (JSC::MarkedAllocator::MarkedAllocator): Deleted.
>+        (JSC::MarkedAllocator::init): Deleted.
>+        * heap/MarkedBlock.cpp:
>+        (JSC::MarkedBlock::tryCreate):
>+        (JSC::MarkedBlock::MarkedBlock):
>+        (JSC::MarkedBlock::specializedSweep):
>+        (JSC::MarkedBlock::sweep):
>+        (JSC::MarkedBlock::sweepHelperSelectResetMode):
>+        (JSC::MarkedBlock::sweepHelperSelectStateAndSweepMode):
>+        (JSC::MarkedBlock::stopAllocating):
>+        (JSC::MarkedBlock::clearMarksWithCollectionType):
>+        (JSC::MarkedBlock::lastChanceToFinalize):
>+        (JSC::MarkedBlock::resumeAllocating):
>+        (JSC::MarkedBlock::didRetireBlock):
>+        (JSC::MarkedBlock::forEachFreeCell):
>+        (JSC::MarkedBlock::create): Deleted.
>+        (JSC::MarkedBlock::callDestructor): Deleted.
>+        (JSC::MarkedBlock::sweepHelper): Deleted.
>+        * heap/MarkedBlock.h:
>+        (JSC::MarkedBlock::VoidFunctor::returnValue):
>+        (JSC::MarkedBlock::setHasAnyMarked):
>+        (JSC::MarkedBlock::hasAnyMarked):
>+        (JSC::MarkedBlock::clearHasAnyMarked):
>+        (JSC::MarkedBlock::firstAtom):
>+        (JSC::MarkedBlock::isAtomAligned):
>+        (JSC::MarkedBlock::cellAlign):
>+        (JSC::MarkedBlock::blockFor):
>+        (JSC::MarkedBlock::isEmpty):
>+        (JSC::MarkedBlock::cellSize):
>+        (JSC::MarkedBlock::isMarkedOrRetired):
>+        (JSC::MarkedBlock::FreeList::FreeList): Deleted.
>+        * heap/MarkedSpace.cpp:
>+        (JSC::MarkedSpace::initializeSizeClassForStepSize):
>+        (JSC::MarkedSpace::MarkedSpace):
>+        (JSC::MarkedSpace::lastChanceToFinalize):
>+        (JSC::MarkedSpace::allocateLarge):
>+        (JSC::MarkedSpace::tryAllocateLarge):
>+        (JSC::MarkedSpace::sweep):
>+        (JSC::MarkedSpace::sweepABit):
>+        (JSC::MarkedSpace::sweepLargeAllocations):
>+        (JSC::MarkedSpace::zombifySweep):
>+        (JSC::MarkedSpace::resetAllocators):
>+        (JSC::MarkedSpace::visitWeakSets):
>+        (JSC::MarkedSpace::reapWeakSets):
>+        (JSC::MarkedSpace::stopAllocating):
>+        (JSC::MarkedSpace::resumeAllocating):
>+        (JSC::MarkedSpace::isPagedOut):
>+        (JSC::MarkedSpace::shrink):
>+        (JSC::MarkedSpace::clearNewlyAllocated):
>+        (JSC::MarkedSpace::clearMarks):
>+        (JSC::MarkedSpace::didFinishIterating):
>+        (JSC::MarkedSpace::objectCount):
>+        (JSC::MarkedSpace::size):
>+        (JSC::MarkedSpace::capacity):
>+        (JSC::MarkedSpace::forEachAllocator): Deleted.
>+        * heap/MarkedSpace.h:
>+        (JSC::MarkedSpace::sizeClassIndex):
>+        (JSC::MarkedSpace::subspaceForObjectsWithDestructor):
>+        (JSC::MarkedSpace::subspaceForObjectsWithoutDestructor):
>+        (JSC::MarkedSpace::subspaceForAuxiliaryData):
>+        (JSC::MarkedSpace::blocksWithNewObjects):
>+        (JSC::MarkedSpace::largeAllocations):
>+        (JSC::MarkedSpace::largeAllocationsNurseryOffset):
>+        (JSC::MarkedSpace::largeAllocationsOffsetForThisCollection):
>+        (JSC::MarkedSpace::largeAllocationsForThisCollectionBegin):
>+        (JSC::MarkedSpace::largeAllocationsForThisCollectionEnd):
>+        (JSC::MarkedSpace::largeAllocationsForThisCollectionSize):
>+        (JSC::MarkedSpace::forEachLiveCell):
>+        (JSC::MarkedSpace::forEachDeadCell):
>+        (JSC::MarkedSpace::allocatorFor):
>+        (JSC::MarkedSpace::destructorAllocatorFor):
>+        (JSC::MarkedSpace::auxiliaryAllocatorFor):
>+        (JSC::MarkedSpace::allocate):
>+        (JSC::MarkedSpace::tryAllocate):
>+        (JSC::MarkedSpace::allocateWithoutDestructor):
>+        (JSC::MarkedSpace::allocateWithDestructor):
>+        (JSC::MarkedSpace::allocateAuxiliary):
>+        (JSC::MarkedSpace::tryAllocateAuxiliary):
>+        (JSC::MarkedSpace::forEachBlock):
>+        (JSC::MarkedSpace::didAllocateInBlock):
>+        (JSC::MarkedSpace::forEachAllocator):
>+        (JSC::MarkedSpace::forEachSubspace):
>+        (JSC::MarkedSpace::optimalSizeFor):
>+        (JSC::MarkedSpace::objectCount): Deleted.
>+        (JSC::MarkedSpace::size): Deleted.
>+        (JSC::MarkedSpace::capacity): Deleted.
>+        * heap/SlotVisitor.cpp:
>+        (JSC::SlotVisitor::didStartMarking):
>+        (JSC::SlotVisitor::reset):
>+        (JSC::SlotVisitor::clearMarkStack):
>+        (JSC::SlotVisitor::append):
>+        (JSC::SlotVisitor::appendJSCellOrAuxiliary):
>+        (JSC::SlotVisitor::setMarkedAndAppendToMarkStack):
>+        (JSC::SlotVisitor::appendToMarkStack):
>+        (JSC::SlotVisitor::markAuxiliary):
>+        (JSC::SlotVisitor::noteLiveAuxiliaryCell):
>+        (JSC::SetCurrentCellScope::SetCurrentCellScope):
>+        (JSC::SlotVisitor::visitChildren):
>+        * heap/SlotVisitor.h:
>+        * heap/WeakBlock.cpp:
>+        (JSC::WeakBlock::create):
>+        (JSC::WeakBlock::destroy):
>+        (JSC::WeakBlock::WeakBlock):
>+        (JSC::WeakBlock::visit):
>+        (JSC::WeakBlock::reap):
>+        * heap/WeakBlock.h:
>+        (JSC::WeakBlock::disconnectContainer):
>+        (JSC::WeakBlock::disconnectMarkedBlock): Deleted.
>+        * heap/WeakSet.cpp:
>+        (JSC::WeakSet::sweep):
>+        (JSC::WeakSet::addAllocator):
>+        * heap/WeakSet.h:
>+        (JSC::WeakSet::WeakSet):
>+        * heap/WeakSetInlines.h:
>+        (JSC::WeakSet::allocate):
>+        * inspector/InjectedScriptManager.cpp:
>+        * inspector/JSGlobalObjectInspectorController.cpp:
>+        * inspector/JSJavaScriptCallFrame.cpp:
>+        * inspector/ScriptDebugServer.cpp:
>+        * inspector/agents/InspectorDebuggerAgent.cpp:
>+        * interpreter/CachedCall.h:
>+        (JSC::CachedCall::CachedCall):
>+        * jit/AssemblyHelpers.h:
>+        (JSC::AssemblyHelpers::emitAllocate):
>+        (JSC::AssemblyHelpers::emitAllocateJSCell):
>+        (JSC::AssemblyHelpers::emitAllocateJSObject):
>+        (JSC::AssemblyHelpers::emitAllocateJSObjectWithKnownSize):
>+        (JSC::AssemblyHelpers::emitAllocateVariableSized):
>+        * jit/JITOpcodes.cpp:
>+        (JSC::JIT::emit_op_new_object):
>+        (JSC::JIT::emit_op_create_this):
>+        * jit/JITOpcodes32_64.cpp:
>+        (JSC::JIT::emit_op_new_object):
>+        (JSC::JIT::emit_op_create_this):
>+        * jit/JITOperations.cpp:
>+        * jit/JITOperations.h:
>+        * jit/JITPropertyAccess.cpp:
>+        (JSC::JIT::emitWriteBarrier):
>+        * jsc.cpp:
>+        (functionDescribeArray):
>+        * llint/LLIntData.cpp:
>+        (JSC::LLInt::Data::performAssertions):
>+        * llint/LowLevelInterpreter.asm:
>+        * llint/LowLevelInterpreter32_64.asm:
>+        * llint/LowLevelInterpreter64.asm:
>+        * parser/ModuleAnalyzer.cpp:
>+        * runtime/ArrayConventions.h:
>+        (JSC::indexIsSufficientlyBeyondLengthForSparseMap):
>+        (JSC::indexingHeaderForArrayStorage):
>+        (JSC::baseIndexingHeaderForArrayStorage):
>+        (JSC::indexingHeaderForArray): Deleted.
>+        (JSC::baseIndexingHeaderForArray): Deleted.
>+        * runtime/ArrayStorage.h:
>+        (JSC::ArrayStorage::length):
>+        (JSC::ArrayStorage::setLength):
>+        (JSC::ArrayStorage::vectorLength):
>+        (JSC::ArrayStorage::setVectorLength):
>+        (JSC::ArrayStorage::copyHeaderFromDuringGC):
>+        (JSC::ArrayStorage::sizeFor):
>+        (JSC::ArrayStorage::totalSizeFor):
>+        (JSC::ArrayStorage::totalSize):
>+        (JSC::ArrayStorage::availableVectorLength):
>+        (JSC::ArrayStorage::optimalVectorLength):
>+        * runtime/AuxiliaryBarrier.h: Added.
>+        (JSC::AuxiliaryBarrier::AuxiliaryBarrier):
>+        (JSC::AuxiliaryBarrier::clear):
>+        (JSC::AuxiliaryBarrier::get):
>+        (JSC::AuxiliaryBarrier::slot):
>+        (JSC::AuxiliaryBarrier::operator bool):
>+        (JSC::AuxiliaryBarrier::setWithoutBarrier):
>+        * runtime/AuxiliaryBarrierInlines.h: Added.
>+        (JSC::AuxiliaryBarrier<T>::AuxiliaryBarrier):
>+        (JSC::AuxiliaryBarrier<T>::set):
>+        * runtime/Butterfly.h:
>+        (JSC::Butterfly::fromBase):
>+        (JSC::Butterfly::fromPointer):
>+        * runtime/ButterflyInlines.h:
>+        (JSC::Butterfly::availableContiguousVectorLength):
>+        (JSC::Butterfly::optimalContiguousVectorLength):
>+        (JSC::Butterfly::createUninitialized):
>+        (JSC::Butterfly::growArrayRight):
>+        * runtime/ClonedArguments.cpp:
>+        (JSC::ClonedArguments::createEmpty):
>+        * runtime/DataView.cpp:
>+        * runtime/DirectArguments.h:
>+        * runtime/ECMAScriptSpecInternalFunctions.cpp:
>+        * runtime/GeneratorFrame.cpp:
>+        * runtime/GeneratorPrototype.cpp:
>+        * runtime/IntlCollator.cpp:
>+        * runtime/IntlCollatorConstructor.cpp:
>+        * runtime/IntlCollatorPrototype.cpp:
>+        * runtime/IntlDateTimeFormat.cpp:
>+        * runtime/IntlDateTimeFormatConstructor.cpp:
>+        * runtime/IntlDateTimeFormatPrototype.cpp:
>+        * runtime/IntlNumberFormat.cpp:
>+        * runtime/IntlNumberFormatConstructor.cpp:
>+        * runtime/IntlNumberFormatPrototype.cpp:
>+        * runtime/JSArray.cpp:
>+        (JSC::createArrayButterflyInDictionaryIndexingMode):
>+        (JSC::JSArray::tryCreateUninitialized):
>+        (JSC::JSArray::setLengthWritable):
>+        (JSC::JSArray::unshiftCountSlowCase):
>+        (JSC::JSArray::setLengthWithArrayStorage):
>+        (JSC::JSArray::appendMemcpy):
>+        (JSC::JSArray::setLength):
>+        (JSC::JSArray::pop):
>+        (JSC::JSArray::push):
>+        (JSC::JSArray::fastSlice):
>+        (JSC::JSArray::shiftCountWithArrayStorage):
>+        (JSC::JSArray::shiftCountWithAnyIndexingType):
>+        (JSC::JSArray::unshiftCountWithArrayStorage):
>+        (JSC::JSArray::fillArgList):
>+        (JSC::JSArray::copyToArguments):
>+        * runtime/JSArray.h:
>+        (JSC::createContiguousArrayButterfly):
>+        (JSC::createArrayButterfly):
>+        (JSC::JSArray::create):
>+        (JSC::JSArray::tryCreateUninitialized): Deleted.
>+        * runtime/JSArrayBufferView.h:
>+        * runtime/JSCInlines.h:
>+        * runtime/JSCJSValue.cpp:
>+        * runtime/JSCallee.cpp:
>+        * runtime/JSCell.cpp:
>+        (JSC::JSCell::estimatedSize):
>+        (JSC::JSCell::copyBackingStore):
>+        * runtime/JSCell.h:
>+        (JSC::JSCell::cellStateOffset):
>+        * runtime/JSCellInlines.h:
>+        (JSC::JSCell::visitChildren):
>+        (JSC::ExecState::vm):
>+        (JSC::JSCell::canUseFastGetOwnProperty):
>+        (JSC::JSCell::classInfo):
>+        (JSC::JSCell::toBoolean):
>+        (JSC::JSCell::pureToBoolean):
>+        (JSC::JSCell::callDestructor):
>+        (JSC::JSCell::vm): Deleted.
>+        * runtime/JSFunction.cpp:
>+        (JSC::JSFunction::create):
>+        (JSC::JSFunction::allocateAndInitializeRareData):
>+        (JSC::JSFunction::initializeRareData):
>+        (JSC::JSFunction::getOwnPropertySlot):
>+        (JSC::JSFunction::put):
>+        (JSC::JSFunction::deleteProperty):
>+        (JSC::JSFunction::defineOwnProperty):
>+        (JSC::JSFunction::setFunctionName):
>+        (JSC::JSFunction::reifyLength):
>+        (JSC::JSFunction::reifyName):
>+        (JSC::JSFunction::reifyLazyPropertyIfNeeded):
>+        (JSC::JSFunction::reifyBoundNameIfNeeded):
>+        * runtime/JSFunction.h:
>+        * runtime/JSFunctionInlines.h:
>+        (JSC::JSFunction::createWithInvalidatedReallocationWatchpoint):
>+        (JSC::JSFunction::JSFunction):
>+        * runtime/JSGenericTypedArrayViewInlines.h:
>+        (JSC::JSGenericTypedArrayView<Adaptor>::slowDownAndWasteMemory):
>+        * runtime/JSInternalPromise.cpp:
>+        * runtime/JSInternalPromiseConstructor.cpp:
>+        * runtime/JSInternalPromiseDeferred.cpp:
>+        * runtime/JSInternalPromisePrototype.cpp:
>+        * runtime/JSJob.cpp:
>+        * runtime/JSMapIterator.cpp:
>+        * runtime/JSModuleNamespaceObject.cpp:
>+        * runtime/JSModuleRecord.cpp:
>+        * runtime/JSObject.cpp:
>+        (JSC::getClassPropertyNames):
>+        (JSC::JSObject::visitButterfly):
>+        (JSC::JSObject::visitChildren):
>+        (JSC::JSObject::heapSnapshot):
>+        (JSC::JSObject::notifyPresenceOfIndexedAccessors):
>+        (JSC::JSObject::createInitialIndexedStorage):
>+        (JSC::JSObject::createInitialUndecided):
>+        (JSC::JSObject::createInitialInt32):
>+        (JSC::JSObject::createInitialDouble):
>+        (JSC::JSObject::createInitialContiguous):
>+        (JSC::JSObject::createArrayStorage):
>+        (JSC::JSObject::createInitialArrayStorage):
>+        (JSC::JSObject::convertUndecidedToInt32):
>+        (JSC::JSObject::convertUndecidedToContiguous):
>+        (JSC::JSObject::convertUndecidedToArrayStorage):
>+        (JSC::JSObject::convertInt32ToDouble):
>+        (JSC::JSObject::convertInt32ToArrayStorage):
>+        (JSC::JSObject::convertDoubleToArrayStorage):
>+        (JSC::JSObject::convertContiguousToArrayStorage):
>+        (JSC::JSObject::putByIndexBeyondVectorLength):
>+        (JSC::JSObject::putDirectIndexBeyondVectorLength):
>+        (JSC::JSObject::putDirectNativeFunctionWithoutTransition):
>+        (JSC::JSObject::getNewVectorLength):
>+        (JSC::JSObject::increaseVectorLength):
>+        (JSC::JSObject::ensureLengthSlow):
>+        (JSC::JSObject::growOutOfLineStorage):
>+        (JSC::JSObject::copyButterfly): Deleted.
>+        (JSC::JSObject::copyBackingStore): Deleted.
>+        * runtime/JSObject.h:
>+        (JSC::JSObject::initializeIndex):
>+        (JSC::JSObject::globalObject):
>+        (JSC::JSObject::putDirectInternal):
>+        (JSC::JSObject::putOwnDataProperty):
>+        (JSC::JSObject::setStructureAndReallocateStorageIfNecessary): Deleted.
>+        * runtime/JSObjectInlines.h:
>+        * runtime/JSPromise.cpp:
>+        * runtime/JSPromiseConstructor.cpp:
>+        * runtime/JSPromiseDeferred.cpp:
>+        * runtime/JSPromisePrototype.cpp:
>+        * runtime/JSPropertyNameIterator.cpp:
>+        * runtime/JSScope.cpp:
>+        (JSC::JSScope::resolve):
>+        * runtime/JSScope.h:
>+        (JSC::JSScope::globalObject):
>+        (JSC::Register::operator=):
>+        (JSC::JSScope::vm): Deleted.
>+        * runtime/JSSetIterator.cpp:
>+        * runtime/JSStringIterator.cpp:
>+        * runtime/JSTemplateRegistryKey.cpp:
>+        * runtime/JSTypedArrayViewConstructor.cpp:
>+        * runtime/JSTypedArrayViewPrototype.cpp:
>+        * runtime/JSWeakMap.cpp:
>+        * runtime/JSWeakSet.cpp:
>+        * runtime/MapConstructor.cpp:
>+        * runtime/MapPrototype.cpp:
>+        * runtime/NativeStdFunctionCell.cpp:
>+        * runtime/Operations.h:
>+        (JSC::jsAdd):
>+        (JSC::resetFreeCellsBadly):
>+        (JSC::resetBadly):
>+        * runtime/Options.h:
>+        * runtime/PropertyTable.cpp:
>+        * runtime/ProxyConstructor.cpp:
>+        * runtime/ProxyObject.cpp:
>+        * runtime/ProxyRevoke.cpp:
>+        * runtime/RegExpMatchesArray.h:
>+        (JSC::tryCreateUninitializedRegExpMatchesArray):
>+        (JSC::createRegExpMatchesArray):
>+        * runtime/RuntimeType.cpp:
>+        * runtime/SamplingProfiler.cpp:
>+        (JSC::SamplingProfiler::processUnverifiedStackTraces):
>+        * runtime/SetConstructor.cpp:
>+        * runtime/SetPrototype.cpp:
>+        * runtime/TemplateRegistry.cpp:
>+        * runtime/TypeProfilerLog.cpp:
>+        * runtime/TypeSet.cpp:
>+        * runtime/WeakMapConstructor.cpp:
>+        * runtime/WeakMapData.cpp:
>+        * runtime/WeakMapPrototype.cpp:
>+        * runtime/WeakSetConstructor.cpp:
>+        * runtime/WeakSetPrototype.cpp:
>+        * tools/JSDollarVMPrototype.cpp:
>+        (JSC::JSDollarVMPrototype::isInObjectSpace):
>+        (JSC::JSDollarVMPrototype::isInStorageSpace):
>+
> 2016-08-22  Ryosuke Niwa  <rniwa@webkit.org>
> 
>         Rename CustomElementsRegistry to CustomElementRegistry
>Index: Source/JavaScriptCore/CMakeLists.txt
>===================================================================
>--- Source/JavaScriptCore/CMakeLists.txt	(revision 204735)
>+++ Source/JavaScriptCore/CMakeLists.txt	(working copy)
>@@ -66,6 +66,7 @@ set(JavaScriptCore_SOURCES
>     assembler/MacroAssembler.cpp
>     assembler/MacroAssemblerARM.cpp
>     assembler/MacroAssemblerARMv7.cpp
>+    assembler/MacroAssemblerCodeRef.cpp
>     assembler/MacroAssemblerPrinter.cpp
>     assembler/MacroAssemblerX86Common.cpp
> 
>@@ -445,6 +446,7 @@ set(JavaScriptCore_SOURCES
>     heap/DestructionMode.cpp
>     heap/EdenGCActivityCallback.cpp
>     heap/FullGCActivityCallback.cpp
>+    heap/FreeList.cpp
>     heap/GCActivityCallback.cpp
>     heap/GCLogging.cpp
>     heap/HandleSet.cpp
>@@ -460,6 +462,7 @@ set(JavaScriptCore_SOURCES
>     heap/HeapVerifier.cpp
>     heap/IncrementalSweeper.cpp
>     heap/JITStubRoutineSet.cpp
>+    heap/LargeAllocation.cpp
>     heap/LiveObjectList.cpp
>     heap/MachineStackMarker.cpp
>     heap/MarkStack.cpp
>@@ -798,6 +801,7 @@ set(JavaScriptCore_SOURCES
>     runtime/SimpleTypedArrayController.cpp
>     runtime/SmallStrings.cpp
>     runtime/SparseArrayValueMap.cpp
>+    runtime/StackFrame.cpp
>     runtime/StrictEvalActivation.cpp
>     runtime/StringConstructor.cpp
>     runtime/StringIteratorPrototype.cpp
>Index: Source/JavaScriptCore/jsc.cpp
>===================================================================
>--- Source/JavaScriptCore/jsc.cpp	(revision 204735)
>+++ Source/JavaScriptCore/jsc.cpp	(working copy)
>@@ -1191,7 +1191,7 @@ EncodedJSValue JSC_HOST_CALL functionDes
>     JSObject* object = jsDynamicCast<JSObject*>(exec->argument(0));
>     if (!object)
>         return JSValue::encode(jsNontrivialString(exec, ASCIILiteral("<not object>")));
>-    return JSValue::encode(jsNontrivialString(exec, toString("<Public length: ", object->getArrayLength(), "; vector length: ", object->getVectorLength(), ">")));
>+    return JSValue::encode(jsNontrivialString(exec, toString("<Butterfly: ", RawPointer(object->butterfly()), "; public length: ", object->getArrayLength(), "; vector length: ", object->getVectorLength(), ">")));
> }
> 
> class FunctionJSCStackFunctor {
>@@ -1977,10 +1977,7 @@ int main(int argc, char** argv)
>     TRY
>         res = jscmain(argc, argv);
>     EXCEPT(res = 3)
>-    if (Options::logHeapStatisticsAtExit())
>-        HeapStatistics::reportSuccess();
>-    if (Options::reportLLIntStats())
>-        LLInt::Data::finalizeStats();
>+    finalizeStatsAtEndOfTesting();
> 
> #if PLATFORM(EFL)
>     ecore_shutdown();
>Index: Source/JavaScriptCore/API/JSTypedArray.cpp
>===================================================================
>--- Source/JavaScriptCore/API/JSTypedArray.cpp	(revision 204735)
>+++ Source/JavaScriptCore/API/JSTypedArray.cpp	(working copy)
>@@ -32,7 +32,7 @@
> #include "ClassInfo.h"
> #include "Error.h"
> #include "JSArrayBufferViewInlines.h"
>-#include "JSCJSValueInlines.h"
>+#include "JSCInlines.h"
> #include "JSDataView.h"
> #include "JSGenericTypedArrayViewInlines.h"
> #include "JSTypedArrays.h"
>Index: Source/JavaScriptCore/API/ObjCCallbackFunction.mm
>===================================================================
>--- Source/JavaScriptCore/API/ObjCCallbackFunction.mm	(revision 204735)
>+++ Source/JavaScriptCore/API/ObjCCallbackFunction.mm	(working copy)
>@@ -31,9 +31,8 @@
> #import "APICallbackFunction.h"
> #import "APICast.h"
> #import "Error.h"
>-#import "JSCJSValueInlines.h"
> #import "JSCell.h"
>-#import "JSCellInlines.h"
>+#import "JSCInlines.h"
> #import "JSContextInternal.h"
> #import "JSWrapperMap.h"
> #import "JSValueInternal.h"
>Index: Source/JavaScriptCore/JavaScriptCore.xcodeproj/project.pbxproj
>===================================================================
>--- Source/JavaScriptCore/JavaScriptCore.xcodeproj/project.pbxproj	(revision 204735)
>+++ Source/JavaScriptCore/JavaScriptCore.xcodeproj/project.pbxproj	(working copy)
>@@ -88,6 +88,11 @@
> 		0F04396D1B03DC0B009598B7 /* DFGCombinedLiveness.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 0F04396B1B03DC0B009598B7 /* DFGCombinedLiveness.cpp */; };
> 		0F04396E1B03DC0B009598B7 /* DFGCombinedLiveness.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F04396C1B03DC0B009598B7 /* DFGCombinedLiveness.h */; };
> 		0F05C3B41683CF9200BAF45B /* DFGArrayifySlowPathGenerator.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F05C3B21683CF8F00BAF45B /* DFGArrayifySlowPathGenerator.h */; };
>+		0F070A471D543A8B006E7232 /* CellContainer.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F070A421D543A89006E7232 /* CellContainer.h */; settings = {ATTRIBUTES = (Private, ); }; };
>+		0F070A481D543A90006E7232 /* CellContainerInlines.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F070A431D543A89006E7232 /* CellContainerInlines.h */; settings = {ATTRIBUTES = (Private, ); }; };
>+		0F070A491D543A93006E7232 /* HeapCellInlines.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F070A441D543A89006E7232 /* HeapCellInlines.h */; settings = {ATTRIBUTES = (Private, ); }; };
>+		0F070A4A1D543A95006E7232 /* LargeAllocation.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 0F070A451D543A89006E7232 /* LargeAllocation.cpp */; };
>+		0F070A4B1D543A98006E7232 /* LargeAllocation.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F070A461D543A89006E7232 /* LargeAllocation.h */; settings = {ATTRIBUTES = (Private, ); }; };
> 		0F0776BF14FF002B00102332 /* JITCompilationEffort.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F0776BD14FF002800102332 /* JITCompilationEffort.h */; settings = {ATTRIBUTES = (Private, ); }; };
> 		0F0A75221B94BFA900110660 /* InferredType.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 0F0A75201B94BFA900110660 /* InferredType.cpp */; };
> 		0F0A75231B94BFA900110660 /* InferredType.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F0A75211B94BFA900110660 /* InferredType.h */; settings = {ATTRIBUTES = (Private, ); }; };
>@@ -323,6 +328,8 @@
> 		0F38B01817CFE75500B144D3 /* DFGCompilationKey.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F38B01417CFE75500B144D3 /* DFGCompilationKey.h */; };
> 		0F38B01917CFE75500B144D3 /* DFGCompilationMode.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 0F38B01517CFE75500B144D3 /* DFGCompilationMode.cpp */; };
> 		0F38B01A17CFE75500B144D3 /* DFGCompilationMode.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F38B01617CFE75500B144D3 /* DFGCompilationMode.h */; settings = {ATTRIBUTES = (Private, ); }; };
>+		0F38D2A21D44196800680499 /* AuxiliaryBarrier.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F38D2A01D44196600680499 /* AuxiliaryBarrier.h */; settings = {ATTRIBUTES = (Private, ); }; };
>+		0F38D2A31D44196D00680499 /* AuxiliaryBarrierInlines.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F38D2A11D44196600680499 /* AuxiliaryBarrierInlines.h */; settings = {ATTRIBUTES = (Private, ); }; };
> 		0F392C891B46188400844728 /* DFGOSRExitFuzz.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 0F392C871B46188400844728 /* DFGOSRExitFuzz.cpp */; };
> 		0F392C8A1B46188400844728 /* DFGOSRExitFuzz.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F392C881B46188400844728 /* DFGOSRExitFuzz.h */; };
> 		0F3A1BF91A9ECB7D000DE01A /* DFGPutStackSinkingPhase.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 0F3A1BF71A9ECB7D000DE01A /* DFGPutStackSinkingPhase.cpp */; };
>@@ -364,9 +371,9 @@
> 		0F4680CA14BBB16C00BFE272 /* LLIntCommon.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F4680C514BBB16900BFE272 /* LLIntCommon.h */; settings = {ATTRIBUTES = (Private, ); }; };
> 		0F4680CB14BBB17200BFE272 /* LLIntOfflineAsmConfig.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F4680C614BBB16900BFE272 /* LLIntOfflineAsmConfig.h */; settings = {ATTRIBUTES = (Private, ); }; };
> 		0F4680CC14BBB17A00BFE272 /* LowLevelInterpreter.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 0F4680C714BBB16900BFE272 /* LowLevelInterpreter.cpp */; };
>-		0F4680CD14BBB17D00BFE272 /* LowLevelInterpreter.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F4680C814BBB16900BFE272 /* LowLevelInterpreter.h */; settings = {ATTRIBUTES = (Private, ); }; };
>+		0F4680CD14BBB17D00BFE272 /* LowLevelInterpreter.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F4680C814BBB16900BFE272 /* LowLevelInterpreter.h */; };
> 		0F4680D214BBD16500BFE272 /* LLIntData.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 0F4680CE14BBB3D100BFE272 /* LLIntData.cpp */; };
>-		0F4680D314BBD16700BFE272 /* LLIntData.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F4680CF14BBB3D100BFE272 /* LLIntData.h */; settings = {ATTRIBUTES = (Private, ); }; };
>+		0F4680D314BBD16700BFE272 /* LLIntData.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F4680CF14BBB3D100BFE272 /* LLIntData.h */; };
> 		0F4680D414BBD24900BFE272 /* HostCallReturnValue.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 0F4680D014BBC5F800BFE272 /* HostCallReturnValue.cpp */; };
> 		0F4680D514BBD24B00BFE272 /* HostCallReturnValue.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F4680D114BBC5F800BFE272 /* HostCallReturnValue.h */; settings = {ATTRIBUTES = (Private, ); }; };
> 		0F485321187750560083B687 /* DFGArithMode.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 0F48531F187750560083B687 /* DFGArithMode.cpp */; };
>@@ -386,6 +393,8 @@
> 		0F4F29DF18B6AD1C0057BC15 /* DFGStaticExecutionCountEstimationPhase.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 0F4F29DD18B6AD1C0057BC15 /* DFGStaticExecutionCountEstimationPhase.cpp */; };
> 		0F4F29E018B6AD1C0057BC15 /* DFGStaticExecutionCountEstimationPhase.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F4F29DE18B6AD1C0057BC15 /* DFGStaticExecutionCountEstimationPhase.h */; };
> 		0F50AF3C193E8B3900674EE8 /* DFGStructureClobberState.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F50AF3B193E8B3900674EE8 /* DFGStructureClobberState.h */; };
>+		0F5513A61D5A682C00C32BD8 /* FreeList.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F5513A51D5A682A00C32BD8 /* FreeList.h */; settings = {ATTRIBUTES = (Private, ); }; };
>+		0F5513A81D5A68CD00C32BD8 /* FreeList.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 0F5513A71D5A68CB00C32BD8 /* FreeList.cpp */; };
> 		0F5541B11613C1FB00CE3E25 /* SpecialPointer.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 0F5541AF1613C1FB00CE3E25 /* SpecialPointer.cpp */; };
> 		0F5541B21613C1FB00CE3E25 /* SpecialPointer.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F5541B01613C1FB00CE3E25 /* SpecialPointer.h */; settings = {ATTRIBUTES = (Private, ); }; };
> 		0F55989817C86C5800A1E543 /* ToNativeFromValue.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F55989717C86C5600A1E543 /* ToNativeFromValue.h */; settings = {ATTRIBUTES = (Private, ); }; };
>@@ -458,6 +467,9 @@
> 		0F6B8AE51C4EFE1700969052 /* B3FixSSA.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F6B8AE11C4EFE1700969052 /* B3FixSSA.h */; };
> 		0F6C73501AC9F99F00BE1682 /* VariableWriteFireDetail.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 0F6C734E1AC9F99F00BE1682 /* VariableWriteFireDetail.cpp */; };
> 		0F6C73511AC9F99F00BE1682 /* VariableWriteFireDetail.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F6C734F1AC9F99F00BE1682 /* VariableWriteFireDetail.h */; settings = {ATTRIBUTES = (Private, ); }; };
>+		0F6DB7E91D6124B500CDBF8E /* StackFrame.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F6DB7E81D6124B200CDBF8E /* StackFrame.h */; settings = {ATTRIBUTES = (Private, ); }; };
>+		0F6DB7EA1D6124B800CDBF8E /* StackFrame.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 0F6DB7E71D6124B200CDBF8E /* StackFrame.cpp */; };
>+		0F6DB7EC1D617D1100CDBF8E /* MacroAssemblerCodeRef.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 0F6DB7EB1D617D0F00CDBF8E /* MacroAssemblerCodeRef.cpp */; };
> 		0F6E845A19030BEF00562741 /* DFGVariableAccessData.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 0F6E845919030BEF00562741 /* DFGVariableAccessData.cpp */; };
> 		0F6FC750196110A800E1D02D /* ComplexGetStatus.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 0F6FC74E196110A800E1D02D /* ComplexGetStatus.cpp */; };
> 		0F6FC751196110A800E1D02D /* ComplexGetStatus.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F6FC74F196110A800E1D02D /* ComplexGetStatus.h */; settings = {ATTRIBUTES = (Private, ); }; };
>@@ -573,6 +585,7 @@
> 		0FA7A8EB18B413C80052371D /* Reg.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 0FA7A8E918B413C80052371D /* Reg.cpp */; };
> 		0FA7A8EC18B413C80052371D /* Reg.h in Headers */ = {isa = PBXBuildFile; fileRef = 0FA7A8EA18B413C80052371D /* Reg.h */; settings = {ATTRIBUTES = (Private, ); }; };
> 		0FA7A8EE18CE4FD80052371D /* ScratchRegisterAllocator.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 0FA7A8ED18CE4FD80052371D /* ScratchRegisterAllocator.cpp */; };
>+		0FADE6731D4D23BE00768457 /* HeapUtil.h in Headers */ = {isa = PBXBuildFile; fileRef = 0FADE6721D4D23BC00768457 /* HeapUtil.h */; };
> 		0FAF7EFD165BA91B000C8455 /* JITDisassembler.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 0FAF7EFA165BA919000C8455 /* JITDisassembler.cpp */; };
> 		0FAF7EFE165BA91F000C8455 /* JITDisassembler.h in Headers */ = {isa = PBXBuildFile; fileRef = 0FAF7EFB165BA919000C8455 /* JITDisassembler.h */; settings = {ATTRIBUTES = (Private, ); }; };
> 		0FB105851675480F00F8AB6E /* ExitKind.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 0FB105821675480C00F8AB6E /* ExitKind.cpp */; };
>@@ -991,7 +1004,7 @@
> 		14280865107EC11A0013E7B2 /* BooleanPrototype.cpp in Sources */ = {isa = PBXBuildFile; fileRef = BC7952340E15EB5600A898AB /* BooleanPrototype.cpp */; };
> 		14280870107EC1340013E7B2 /* JSWrapperObject.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 65C7A1710A8EAACB00FA37EA /* JSWrapperObject.cpp */; };
> 		14280875107EC13E0013E7B2 /* JSLock.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 65EA4C99092AF9E20093D800 /* JSLock.cpp */; };
>-		1429D77C0ED20D7300B89619 /* Interpreter.h in Headers */ = {isa = PBXBuildFile; fileRef = 1429D77B0ED20D7300B89619 /* Interpreter.h */; settings = {ATTRIBUTES = (Private, ); }; };
>+		1429D77C0ED20D7300B89619 /* Interpreter.h in Headers */ = {isa = PBXBuildFile; fileRef = 1429D77B0ED20D7300B89619 /* Interpreter.h */; };
> 		1429D7D40ED2128200B89619 /* Interpreter.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 1429D7D30ED2128200B89619 /* Interpreter.cpp */; };
> 		1429D8780ED21ACD00B89619 /* ExceptionHelpers.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 1429D8770ED21ACD00B89619 /* ExceptionHelpers.cpp */; };
> 		1429D8DD0ED2205B00B89619 /* CallFrame.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 1429D8DB0ED2205B00B89619 /* CallFrame.cpp */; };
>@@ -1451,7 +1464,7 @@
> 		969A07980ED1D3AE00F1F681 /* EvalCodeCache.h in Headers */ = {isa = PBXBuildFile; fileRef = 969A07920ED1D3AE00F1F681 /* EvalCodeCache.h */; settings = {ATTRIBUTES = (Private, ); }; };
> 		969A07990ED1D3AE00F1F681 /* Instruction.h in Headers */ = {isa = PBXBuildFile; fileRef = 969A07930ED1D3AE00F1F681 /* Instruction.h */; settings = {ATTRIBUTES = (Private, ); }; };
> 		969A079A0ED1D3AE00F1F681 /* Opcode.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 969A07940ED1D3AE00F1F681 /* Opcode.cpp */; };
>-		969A079B0ED1D3AE00F1F681 /* Opcode.h in Headers */ = {isa = PBXBuildFile; fileRef = 969A07950ED1D3AE00F1F681 /* Opcode.h */; settings = {ATTRIBUTES = (Private, ); }; };
>+		969A079B0ED1D3AE00F1F681 /* Opcode.h in Headers */ = {isa = PBXBuildFile; fileRef = 969A07950ED1D3AE00F1F681 /* Opcode.h */; };
> 		978801401471AD920041B016 /* JSDateMath.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 9788FC221471AD0C0068CE2D /* JSDateMath.cpp */; };
> 		978801411471AD920041B016 /* JSDateMath.h in Headers */ = {isa = PBXBuildFile; fileRef = 9788FC231471AD0C0068CE2D /* JSDateMath.h */; settings = {ATTRIBUTES = (Private, ); }; };
> 		990DA67F1C8E316A00295159 /* generate_objc_protocol_type_conversions_implementation.py in Headers */ = {isa = PBXBuildFile; fileRef = 990DA67E1C8E311D00295159 /* generate_objc_protocol_type_conversions_implementation.py */; settings = {ATTRIBUTES = (Private, ); }; };
>@@ -2142,6 +2155,8 @@
> 		FED94F2E171E3E2300BE77A4 /* Watchdog.cpp in Sources */ = {isa = PBXBuildFile; fileRef = FED94F2B171E3E2300BE77A4 /* Watchdog.cpp */; };
> 		FED94F2F171E3E2300BE77A4 /* Watchdog.h in Headers */ = {isa = PBXBuildFile; fileRef = FED94F2C171E3E2300BE77A4 /* Watchdog.h */; settings = {ATTRIBUTES = (Private, ); }; };
> 		FEF040511AAE662D00BD28B0 /* CompareAndSwapTest.cpp in Sources */ = {isa = PBXBuildFile; fileRef = FEF040501AAE662D00BD28B0 /* CompareAndSwapTest.cpp */; };
>+		D9722752DC54459B9125B539 /* JSModuleLoader.h in Headers */ = {isa = PBXBuildFile; fileRef = 77B25CB2C3094A92A38E1DB3 /* JSModuleLoader.h */; };
>+		13FECE06D3B445FCB6C93461 /* JSModuleLoader.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 1879510614C540FFB561C124 /* JSModuleLoader.cpp */; };
> 		FEFD6FC61D5E7992008F2F0B /* JSStringInlines.h in Headers */ = {isa = PBXBuildFile; fileRef = FEFD6FC51D5E7970008F2F0B /* JSStringInlines.h */; settings = {ATTRIBUTES = (Private, ); }; };
> /* End PBXBuildFile section */
> 
>@@ -2305,6 +2320,11 @@
> 		0F04396B1B03DC0B009598B7 /* DFGCombinedLiveness.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; name = DFGCombinedLiveness.cpp; path = dfg/DFGCombinedLiveness.cpp; sourceTree = "<group>"; };
> 		0F04396C1B03DC0B009598B7 /* DFGCombinedLiveness.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; name = DFGCombinedLiveness.h; path = dfg/DFGCombinedLiveness.h; sourceTree = "<group>"; };
> 		0F05C3B21683CF8F00BAF45B /* DFGArrayifySlowPathGenerator.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; name = DFGArrayifySlowPathGenerator.h; path = dfg/DFGArrayifySlowPathGenerator.h; sourceTree = "<group>"; };
>+		0F070A421D543A89006E7232 /* CellContainer.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = CellContainer.h; sourceTree = "<group>"; };
>+		0F070A431D543A89006E7232 /* CellContainerInlines.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = CellContainerInlines.h; sourceTree = "<group>"; };
>+		0F070A441D543A89006E7232 /* HeapCellInlines.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = HeapCellInlines.h; sourceTree = "<group>"; };
>+		0F070A451D543A89006E7232 /* LargeAllocation.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = LargeAllocation.cpp; sourceTree = "<group>"; };
>+		0F070A461D543A89006E7232 /* LargeAllocation.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = LargeAllocation.h; sourceTree = "<group>"; };
> 		0F0776BD14FF002800102332 /* JITCompilationEffort.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = JITCompilationEffort.h; sourceTree = "<group>"; };
> 		0F0A75201B94BFA900110660 /* InferredType.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = InferredType.cpp; sourceTree = "<group>"; };
> 		0F0A75211B94BFA900110660 /* InferredType.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = InferredType.h; sourceTree = "<group>"; };
>@@ -2539,6 +2559,8 @@
> 		0F38B01417CFE75500B144D3 /* DFGCompilationKey.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; name = DFGCompilationKey.h; path = dfg/DFGCompilationKey.h; sourceTree = "<group>"; };
> 		0F38B01517CFE75500B144D3 /* DFGCompilationMode.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; name = DFGCompilationMode.cpp; path = dfg/DFGCompilationMode.cpp; sourceTree = "<group>"; };
> 		0F38B01617CFE75500B144D3 /* DFGCompilationMode.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; name = DFGCompilationMode.h; path = dfg/DFGCompilationMode.h; sourceTree = "<group>"; };
>+		0F38D2A01D44196600680499 /* AuxiliaryBarrier.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = AuxiliaryBarrier.h; sourceTree = "<group>"; };
>+		0F38D2A11D44196600680499 /* AuxiliaryBarrierInlines.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = AuxiliaryBarrierInlines.h; sourceTree = "<group>"; };
> 		0F392C871B46188400844728 /* DFGOSRExitFuzz.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; name = DFGOSRExitFuzz.cpp; path = dfg/DFGOSRExitFuzz.cpp; sourceTree = "<group>"; };
> 		0F392C881B46188400844728 /* DFGOSRExitFuzz.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; name = DFGOSRExitFuzz.h; path = dfg/DFGOSRExitFuzz.h; sourceTree = "<group>"; };
> 		0F3A1BF71A9ECB7D000DE01A /* DFGPutStackSinkingPhase.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; name = DFGPutStackSinkingPhase.cpp; path = dfg/DFGPutStackSinkingPhase.cpp; sourceTree = "<group>"; };
>@@ -2600,6 +2622,8 @@
> 		0F4F29DD18B6AD1C0057BC15 /* DFGStaticExecutionCountEstimationPhase.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; name = DFGStaticExecutionCountEstimationPhase.cpp; path = dfg/DFGStaticExecutionCountEstimationPhase.cpp; sourceTree = "<group>"; };
> 		0F4F29DE18B6AD1C0057BC15 /* DFGStaticExecutionCountEstimationPhase.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; name = DFGStaticExecutionCountEstimationPhase.h; path = dfg/DFGStaticExecutionCountEstimationPhase.h; sourceTree = "<group>"; };
> 		0F50AF3B193E8B3900674EE8 /* DFGStructureClobberState.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; name = DFGStructureClobberState.h; path = dfg/DFGStructureClobberState.h; sourceTree = "<group>"; };
>+		0F5513A51D5A682A00C32BD8 /* FreeList.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = FreeList.h; sourceTree = "<group>"; };
>+		0F5513A71D5A68CB00C32BD8 /* FreeList.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = FreeList.cpp; sourceTree = "<group>"; };
> 		0F5541AF1613C1FB00CE3E25 /* SpecialPointer.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = SpecialPointer.cpp; sourceTree = "<group>"; };
> 		0F5541B01613C1FB00CE3E25 /* SpecialPointer.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = SpecialPointer.h; sourceTree = "<group>"; };
> 		0F55989717C86C5600A1E543 /* ToNativeFromValue.h */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.c.h; path = ToNativeFromValue.h; sourceTree = "<group>"; };
>@@ -2674,6 +2698,9 @@
> 		0F6B8AE11C4EFE1700969052 /* B3FixSSA.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; name = B3FixSSA.h; path = b3/B3FixSSA.h; sourceTree = "<group>"; };
> 		0F6C734E1AC9F99F00BE1682 /* VariableWriteFireDetail.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = VariableWriteFireDetail.cpp; sourceTree = "<group>"; };
> 		0F6C734F1AC9F99F00BE1682 /* VariableWriteFireDetail.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = VariableWriteFireDetail.h; sourceTree = "<group>"; };
>+		0F6DB7E71D6124B200CDBF8E /* StackFrame.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = StackFrame.cpp; sourceTree = "<group>"; };
>+		0F6DB7E81D6124B200CDBF8E /* StackFrame.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = StackFrame.h; sourceTree = "<group>"; };
>+		0F6DB7EB1D617D0F00CDBF8E /* MacroAssemblerCodeRef.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = MacroAssemblerCodeRef.cpp; sourceTree = "<group>"; };
> 		0F6E845919030BEF00562741 /* DFGVariableAccessData.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; name = DFGVariableAccessData.cpp; path = dfg/DFGVariableAccessData.cpp; sourceTree = "<group>"; };
> 		0F6FC74E196110A800E1D02D /* ComplexGetStatus.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = ComplexGetStatus.cpp; sourceTree = "<group>"; };
> 		0F6FC74F196110A800E1D02D /* ComplexGetStatus.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = ComplexGetStatus.h; sourceTree = "<group>"; };
>@@ -2786,6 +2813,7 @@
> 		0FA7A8E918B413C80052371D /* Reg.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = Reg.cpp; sourceTree = "<group>"; };
> 		0FA7A8EA18B413C80052371D /* Reg.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = Reg.h; sourceTree = "<group>"; };
> 		0FA7A8ED18CE4FD80052371D /* ScratchRegisterAllocator.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = ScratchRegisterAllocator.cpp; sourceTree = "<group>"; };
>+		0FADE6721D4D23BC00768457 /* HeapUtil.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = HeapUtil.h; sourceTree = "<group>"; };
> 		0FAF7EFA165BA919000C8455 /* JITDisassembler.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = JITDisassembler.cpp; sourceTree = "<group>"; };
> 		0FAF7EFB165BA919000C8455 /* JITDisassembler.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = JITDisassembler.h; sourceTree = "<group>"; };
> 		0FB105821675480C00F8AB6E /* ExitKind.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = ExitKind.cpp; sourceTree = "<group>"; };
>@@ -4447,6 +4475,8 @@
> 		FEDA50D51B97F4D9009A3B4F /* PingPongStackOverflowTest.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; name = PingPongStackOverflowTest.h; path = API/tests/PingPongStackOverflowTest.h; sourceTree = "<group>"; };
> 		FEF040501AAE662D00BD28B0 /* CompareAndSwapTest.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; name = CompareAndSwapTest.cpp; path = API/tests/CompareAndSwapTest.cpp; sourceTree = "<group>"; };
> 		FEF040521AAEC4ED00BD28B0 /* CompareAndSwapTest.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; name = CompareAndSwapTest.h; path = API/tests/CompareAndSwapTest.h; sourceTree = "<group>"; };
>+		77B25CB2C3094A92A38E1DB3 /* JSModuleLoader.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; name = JSModuleLoader.h; path = JSModuleLoader.h; sourceTree = "<group>"; };
>+		1879510614C540FFB561C124 /* JSModuleLoader.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; name = JSModuleLoader.cpp; path = JSModuleLoader.cpp; sourceTree = "<group>"; };
> 		FEFD6FC51D5E7970008F2F0B /* JSStringInlines.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = JSStringInlines.h; sourceTree = "<group>"; };
> /* End PBXFileReference section */
> 
>@@ -5222,6 +5252,8 @@
> 			children = (
> 				0F9630351D4192C3005609D9 /* AllocatorAttributes.cpp */,
> 				0F9630361D4192C3005609D9 /* AllocatorAttributes.h */,
>+				0F070A421D543A89006E7232 /* CellContainer.h */,
>+				0F070A431D543A89006E7232 /* CellContainerInlines.h */,
> 				0F1C3DD91BBCE09E00E523E4 /* CellState.h */,
> 				0FD8A31117D4326C00CA2C40 /* CodeBlockSet.cpp */,
> 				0FD8A31217D4326C00CA2C40 /* CodeBlockSet.h */,
>@@ -5246,6 +5278,8 @@
> 				0F9630381D4192C3005609D9 /* DestructionMode.h */,
> 				2A83638318D7D0EE0000EBCC /* EdenGCActivityCallback.cpp */,
> 				2A83638418D7D0EE0000EBCC /* EdenGCActivityCallback.h */,
>+				0F5513A71D5A68CB00C32BD8 /* FreeList.cpp */,
>+				0F5513A51D5A682A00C32BD8 /* FreeList.h */,
> 				2A83638718D7D0FE0000EBCC /* FullGCActivityCallback.cpp */,
> 				2A83638818D7D0FE0000EBCC /* FullGCActivityCallback.h */,
> 				2AACE63A18CA5A0300ED0191 /* GCActivityCallback.cpp */,
>@@ -5271,6 +5305,7 @@
> 				14BA7A9613AADFF8005B7C2C /* Heap.h */,
> 				DC3D2B0B1D34376E00BA918C /* HeapCell.cpp */,
> 				DC3D2B091D34316100BA918C /* HeapCell.h */,
>+				0F070A441D543A89006E7232 /* HeapCellInlines.h */,
> 				0F32BD0E1BB34F190093A57F /* HeapHelperPool.cpp */,
> 				0F32BD0F1BB34F190093A57F /* HeapHelperPool.h */,
> 				C2DA778218E259990066FCB6 /* HeapInlines.h */,
>@@ -5288,12 +5323,15 @@
> 				C24D31E1161CD695002AA4DB /* HeapStatistics.h */,
> 				C2E526BB1590EF000054E48D /* HeapTimer.cpp */,
> 				C2E526BC1590EF000054E48D /* HeapTimer.h */,
>+				0FADE6721D4D23BC00768457 /* HeapUtil.h */,
> 				FE7BA60D1A1A7CEC00F1F7B4 /* HeapVerifier.cpp */,
> 				FE7BA60E1A1A7CEC00F1F7B4 /* HeapVerifier.h */,
> 				C25F8BCB157544A900245B71 /* IncrementalSweeper.cpp */,
> 				C25F8BCC157544A900245B71 /* IncrementalSweeper.h */,
> 				0F766D2915A8CC34008F363E /* JITStubRoutineSet.cpp */,
> 				0F766D2A15A8CC34008F363E /* JITStubRoutineSet.h */,
>+				0F070A451D543A89006E7232 /* LargeAllocation.cpp */,
>+				0F070A461D543A89006E7232 /* LargeAllocation.h */,
> 				0F431736146BAC65007E3890 /* ListableHandler.h */,
> 				FE3913511B794AC900EDAF71 /* LiveObjectData.h */,
> 				FE3913521B794AC900EDAF71 /* LiveObjectList.cpp */,
>@@ -5651,6 +5689,8 @@
> 				F692A84D0255597D01FF60F7 /* ArrayPrototype.cpp */,
> 				F692A84E0255597D01FF60F7 /* ArrayPrototype.h */,
> 				0FB7F38A15ED8E3800F167B2 /* ArrayStorage.h */,
>+				0F38D2A01D44196600680499 /* AuxiliaryBarrier.h */,
>+				0F38D2A11D44196600680499 /* AuxiliaryBarrierInlines.h */,
> 				52678F8C1A031009006A306D /* BasicBlockLocation.cpp */,
> 				52678F8D1A031009006A306D /* BasicBlockLocation.h */,
> 				147B83AA0E6DB8C9004775A4 /* BatchedTransitionOptimizer.h */,
>@@ -5817,7 +5857,6 @@
> 				70DC3E081B2DF2C700054299 /* IteratorPrototype.h */,
> 				93ADFCE60CCBD7AC00D30B08 /* JSArray.cpp */,
> 				938772E5038BFE19008635CE /* JSArray.h */,
>-				539FB8B91C99DA7C00940FA1 /* JSArrayInlines.h */,
> 				0F2B66B417B6B5AB00A7AE3F /* JSArrayBuffer.cpp */,
> 				0F2B66B517B6B5AB00A7AE3F /* JSArrayBuffer.h */,
> 				0F2B66B617B6B5AB00A7AE3F /* JSArrayBufferConstructor.cpp */,
>@@ -5827,6 +5866,7 @@
> 				0F2B66BA17B6B5AB00A7AE3F /* JSArrayBufferView.cpp */,
> 				0F2B66BB17B6B5AB00A7AE3F /* JSArrayBufferView.h */,
> 				0F2B66BC17B6B5AB00A7AE3F /* JSArrayBufferViewInlines.h */,
>+				539FB8B91C99DA7C00940FA1 /* JSArrayInlines.h */,
> 				86FA9E8F142BBB2D001773B7 /* JSBoundFunction.cpp */,
> 				86FA9E90142BBB2E001773B7 /* JSBoundFunction.h */,
> 				657CF45619BF6662004ACBF2 /* JSCallee.cpp */,
>@@ -5895,6 +5935,8 @@
> 				A74DEF90182D991400522C22 /* JSMapIterator.h */,
> 				E3D239C61B829C1C00BBEF67 /* JSModuleEnvironment.cpp */,
> 				E3D239C71B829C1C00BBEF67 /* JSModuleEnvironment.h */,
>+				1879510614C540FFB561C124 /* JSModuleLoader.cpp */,
>+				77B25CB2C3094A92A38E1DB3 /* JSModuleLoader.h */,
> 				E318CBBE1B8AEF5100A2929D /* JSModuleNamespaceObject.cpp */,
> 				E318CBBF1B8AEF5100A2929D /* JSModuleNamespaceObject.h */,
> 				E39DA4A41B7E8B7C0084F33A /* JSModuleRecord.cpp */,
>@@ -6087,6 +6129,8 @@
> 				0F0CD4C315F6B6B50032F1C0 /* SparseArrayValueMap.cpp */,
> 				0FB7F39215ED8E3800F167B2 /* SparseArrayValueMap.h */,
> 				0F3AC751183EA1040032029F /* StackAlignment.h */,
>+				0F6DB7E71D6124B200CDBF8E /* StackFrame.cpp */,
>+				0F6DB7E81D6124B200CDBF8E /* StackFrame.h */,
> 				A730B6111250068F009D25B1 /* StrictEvalActivation.cpp */,
> 				A730B6101250068F009D25B1 /* StrictEvalActivation.h */,
> 				BC18C3C00E16EE3300B34460 /* StringConstructor.cpp */,
>@@ -6172,8 +6216,6 @@
> 				709FB8661AE335C60039D069 /* WeakSetPrototype.h */,
> 				A7DCB77912E3D90500911940 /* WriteBarrier.h */,
> 				C2B6D75218A33793004A9301 /* WriteBarrierInlines.h */,
>-				77B25CB2C3094A92A38E1DB3 /* JSModuleLoader.h */,
>-				1879510614C540FFB561C124 /* JSModuleLoader.cpp */,
> 			);
> 			path = runtime;
> 			sourceTree = "<group>";
>@@ -6616,6 +6658,7 @@
> 				8640923C156EED3B00566CB2 /* MacroAssemblerARM64.h */,
> 				A729009B17976C6000317298 /* MacroAssemblerARMv7.cpp */,
> 				86ADD1440FDDEA980006EEC2 /* MacroAssemblerARMv7.h */,
>+				0F6DB7EB1D617D0F00CDBF8E /* MacroAssemblerCodeRef.cpp */,
> 				863B23DF0FC60E6200703AA4 /* MacroAssemblerCodeRef.h */,
> 				86C568DE11A213EE0007F7F0 /* MacroAssemblerMIPS.h */,
> 				FE68C6351B90DDD90042BCB3 /* MacroAssemblerPrinter.cpp */,
>@@ -7222,6 +7265,7 @@
> 				99DA00A91BD5993100F4575C /* builtins_generate_separate_header.py in Headers */,
> 				0F338E111BF0276C0013C88F /* B3OpaqueByproduct.h in Headers */,
> 				FEA0C4031CDD7D1D00481991 /* FunctionWhitelist.h in Headers */,
>+				0F6DB7E91D6124B500CDBF8E /* StackFrame.h in Headers */,
> 				99DA00AA1BD5993100F4575C /* builtins_generate_separate_implementation.py in Headers */,
> 				99DA00A31BD5993100F4575C /* builtins_generator.py in Headers */,
> 				412952781D2CF6BC00E78B89 /* builtins_generate_internals_wrapper_implementation.py in Headers */,
>@@ -7313,6 +7357,7 @@
> 				0F338E1C1BF286EA0013C88F /* B3BlockInsertionSet.h in Headers */,
> 				0F9495881C57F47500413A48 /* B3StackSlot.h in Headers */,
> 				C4F4B6F31A05C944005CAB76 /* cpp_generator_templates.py in Headers */,
>+				0F38D2A21D44196800680499 /* AuxiliaryBarrier.h in Headers */,
> 				5DE6E5B30E1728EC00180407 /* create_hash_table in Headers */,
> 				9959E92B1BD17FA4001AA413 /* cssmin.py in Headers */,
> 				2A111246192FCE79005EE18D /* CustomGetterSetter.h in Headers */,
>@@ -7538,6 +7583,7 @@
> 				0FFFC96014EF90BD00C72532 /* DFGVirtualRegisterAllocationPhase.h in Headers */,
> 				0FC97F4218202119002C9B26 /* DFGWatchpointCollectionPhase.h in Headers */,
> 				0FDB2CE8174830A2007B3C1B /* DFGWorklist.h in Headers */,
>+				0F070A491D543A93006E7232 /* HeapCellInlines.h in Headers */,
> 				0FE050181AA9091100D33B33 /* DirectArguments.h in Headers */,
> 				0FE050161AA9091100D33B33 /* DirectArgumentsOffset.h in Headers */,
> 				0FF42731158EBD54004CB9FF /* Disassembler.h in Headers */,
>@@ -7667,6 +7713,7 @@
> 				0FE0501A1AA9091100D33B33 /* GenericArgumentsInlines.h in Headers */,
> 				FE3A06C01C11041A00390FDD /* JITRightShiftGenerator.h in Headers */,
> 				708EBE241CE8F35800453146 /* IntlObjectInlines.h in Headers */,
>+				0F070A481D543A90006E7232 /* CellContainerInlines.h in Headers */,
> 				0FE0501B1AA9091100D33B33 /* GenericOffset.h in Headers */,
> 				0F2B66E017B6B5AB00A7AE3F /* GenericTypedArrayView.h in Headers */,
> 				0F2B66E117B6B5AB00A7AE3F /* GenericTypedArrayViewInlines.h in Headers */,
>@@ -7752,6 +7799,7 @@
> 				A1587D6E1B4DC14100D69849 /* IntlDateTimeFormat.h in Headers */,
> 				FE187A0F1C030D6C0038BBCA /* SnippetOperand.h in Headers */,
> 				A1587D701B4DC14100D69849 /* IntlDateTimeFormatConstructor.h in Headers */,
>+				0FADE6731D4D23BE00768457 /* HeapUtil.h in Headers */,
> 				A1587D751B4DC1C600D69849 /* IntlDateTimeFormatConstructor.lut.h in Headers */,
> 				A5398FAB1C750DA40060A963 /* HeapProfiler.h in Headers */,
> 				A1587D721B4DC14100D69849 /* IntlDateTimeFormatPrototype.h in Headers */,
>@@ -7875,6 +7923,7 @@
> 				C25D709C16DE99F400FCA6BC /* JSManagedValue.h in Headers */,
> 				2A4BB7F318A41179008A0FCD /* JSManagedValueInternal.h in Headers */,
> 				A700874217CBE8EB00C3E643 /* JSMap.h in Headers */,
>+				0F38D2A31D44196D00680499 /* AuxiliaryBarrierInlines.h in Headers */,
> 				A74DEF96182D991400522C22 /* JSMapIterator.h in Headers */,
> 				9959E92D1BD17FA4001AA413 /* jsmin.py in Headers */,
> 				E3D239C91B829C1C00BBEF67 /* JSModuleEnvironment.h in Headers */,
>@@ -7912,6 +7961,7 @@
> 				BC18C4270E16F5CD00B34460 /* JSString.h in Headers */,
> 				86E85539111B9968001AF51E /* JSStringBuilder.h in Headers */,
> 				70EC0EC31AA0D7DA00B6AAFA /* JSStringIterator.h in Headers */,
>+				0F070A471D543A8B006E7232 /* CellContainer.h in Headers */,
> 				2600B5A7152BAAA70091EE5F /* JSStringJoiner.h in Headers */,
> 				BC18C4280E16F5CD00B34460 /* JSStringRef.h in Headers */,
> 				43AB26C61C1A535900D82AE6 /* B3MathExtras.h in Headers */,
>@@ -7982,6 +8032,7 @@
> 				14B723B812D7DA6F003BD5ED /* MachineStackMarker.h in Headers */,
> 				86C36EEA0EE1289D00B3DF59 /* MacroAssembler.h in Headers */,
> 				43422A671C16267800E2EB98 /* B3ReduceDoubleToFloat.h in Headers */,
>+				0F070A4B1D543A98006E7232 /* LargeAllocation.h in Headers */,
> 				86D3B2C610156BDE002865E7 /* MacroAssemblerARM.h in Headers */,
> 				A1A009C01831A22D00CF8711 /* MacroAssemblerARM64.h in Headers */,
> 				86ADD1460FDDEA980006EEC2 /* MacroAssemblerARMv7.h in Headers */,
>@@ -8029,6 +8080,7 @@
> 				BC18C4440E16F5CD00B34460 /* NumberPrototype.h in Headers */,
> 				996B73211BDA08EF00331B84 /* NumberPrototype.lut.h in Headers */,
> 				142D3939103E4560007DCB52 /* NumericStrings.h in Headers */,
>+				0F5513A61D5A682C00C32BD8 /* FreeList.h in Headers */,
> 				A5EA710C19F6DE820098F5EC /* objc_generator.py in Headers */,
> 				C4F4B6F61A05C984005CAB76 /* objc_generator_templates.py in Headers */,
> 				86F3EEBD168CDE930077B92A /* ObjCCallbackFunction.h in Headers */,
>@@ -8810,6 +8862,7 @@
> 				0FEC856F1BDACDC70080FF74 /* AirArg.cpp in Sources */,
> 				0F4DE1CE1C4C1B54004D6C11 /* AirFixObviousSpills.cpp in Sources */,
> 				0FEC85711BDACDC70080FF74 /* AirBasicBlock.cpp in Sources */,
>+				0F070A4A1D543A95006E7232 /* LargeAllocation.cpp in Sources */,
> 				0FEC85731BDACDC70080FF74 /* AirCCallSpecial.cpp in Sources */,
> 				0FEC85751BDACDC70080FF74 /* AirCode.cpp in Sources */,
> 				0F4570381BE44C910062A629 /* AirEliminateDeadCode.cpp in Sources */,
>@@ -9228,6 +9281,7 @@
> 				0FE34C191C4B39AE0003A512 /* AirLogRegisterPressure.cpp in Sources */,
> 				A1B9E2391B4E0D6700BC7FED /* IntlCollator.cpp in Sources */,
> 				A1B9E23B1B4E0D6700BC7FED /* IntlCollatorConstructor.cpp in Sources */,
>+				0F6DB7EA1D6124B800CDBF8E /* StackFrame.cpp in Sources */,
> 				A1B9E23D1B4E0D6700BC7FED /* IntlCollatorPrototype.cpp in Sources */,
> 				A1587D6D1B4DC14100D69849 /* IntlDateTimeFormat.cpp in Sources */,
> 				A1587D6F1B4DC14100D69849 /* IntlDateTimeFormatConstructor.cpp in Sources */,
>@@ -9251,11 +9305,13 @@
> 				146FE51211A710430087AE66 /* JITCall32_64.cpp in Sources */,
> 				0F8F94441667635400D61971 /* JITCode.cpp in Sources */,
> 				0FAF7EFD165BA91B000C8455 /* JITDisassembler.cpp in Sources */,
>+				0F6DB7EC1D617D1100CDBF8E /* MacroAssemblerCodeRef.cpp in Sources */,
> 				0F46808314BA573100BFE272 /* JITExceptions.cpp in Sources */,
> 				0FB14E1E18124ACE009B6B4D /* JITInlineCacheGenerator.cpp in Sources */,
> 				BCDD51EB0FB8DF74004A8BDC /* JITOpcodes.cpp in Sources */,
> 				A71236E51195F33C00BD2174 /* JITOpcodes32_64.cpp in Sources */,
> 				0F24E54C17EE274900ABB217 /* JITOperations.cpp in Sources */,
>+				0F5513A81D5A68CD00C32BD8 /* FreeList.cpp in Sources */,
> 				FE99B24A1C24C3D700C82159 /* JITNegGenerator.cpp in Sources */,
> 				86CC85C40EE7A89400288682 /* JITPropertyAccess.cpp in Sources */,
> 				A7C1E8E4112E72EF00A37F98 /* JITPropertyAccess32_64.cpp in Sources */,
>Index: Source/JavaScriptCore/Scripts/builtins/builtins_generate_combined_implementation.py
>===================================================================
>--- Source/JavaScriptCore/Scripts/builtins/builtins_generate_combined_implementation.py	(revision 204735)
>+++ Source/JavaScriptCore/Scripts/builtins/builtins_generate_combined_implementation.py	(working copy)
>@@ -72,6 +72,9 @@ class BuiltinsCombinedImplementationGene
>                 ("JavaScriptCore", "builtins/BuiltinExecutables.h"),
>             ),
>             (["JavaScriptCore", "WebCore"],
>+                ("JavaScriptCore", "heap/HeapInlines.h"),
>+            ),
>+            (["JavaScriptCore", "WebCore"],
>                 ("JavaScriptCore", "runtime/Executable.h"),
>             ),
>             (["JavaScriptCore", "WebCore"],
>Index: Source/JavaScriptCore/Scripts/builtins/builtins_generate_internals_wrapper_implementation.py
>===================================================================
>--- Source/JavaScriptCore/Scripts/builtins/builtins_generate_internals_wrapper_implementation.py	(revision 204735)
>+++ Source/JavaScriptCore/Scripts/builtins/builtins_generate_internals_wrapper_implementation.py	(working copy)
>@@ -68,6 +68,9 @@ class BuiltinsInternalsWrapperImplementa
>                 ("WebCore", "WebCoreJSClientData.h"),
>             ),
>             (["WebCore"],
>+                ("JavaScriptCore", "heap/HeapInlines.h"),
>+            ),
>+            (["WebCore"],
>                 ("JavaScriptCore", "heap/SlotVisitorInlines.h"),
>             ),
>             (["WebCore"],
>Index: Source/JavaScriptCore/Scripts/builtins/builtins_generate_separate_implementation.py
>===================================================================
>--- Source/JavaScriptCore/Scripts/builtins/builtins_generate_separate_implementation.py	(revision 204735)
>+++ Source/JavaScriptCore/Scripts/builtins/builtins_generate_separate_implementation.py	(working copy)
>@@ -84,6 +84,9 @@ class BuiltinsSeparateImplementationGene
>                 ("JavaScriptCore", "builtins/BuiltinExecutables.h"),
>             ),
>             (["JavaScriptCore", "WebCore"],
>+                ("JavaScriptCore", "heap/HeapInlines.h"),
>+            ),
>+            (["JavaScriptCore", "WebCore"],
>                 ("JavaScriptCore", "runtime/Executable.h"),
>             ),
>             (["JavaScriptCore", "WebCore"],
>Index: Source/JavaScriptCore/assembler/AbstractMacroAssembler.h
>===================================================================
>--- Source/JavaScriptCore/assembler/AbstractMacroAssembler.h	(revision 204735)
>+++ Source/JavaScriptCore/assembler/AbstractMacroAssembler.h	(working copy)
>@@ -725,20 +725,18 @@ public:
>                 append(jump);
>         }
> 
>-        void link(AbstractMacroAssemblerType* masm)
>+        void link(AbstractMacroAssemblerType* masm) const
>         {
>             size_t size = m_jumps.size();
>             for (size_t i = 0; i < size; ++i)
>                 m_jumps[i].link(masm);
>-            m_jumps.clear();
>         }
>         
>-        void linkTo(Label label, AbstractMacroAssemblerType* masm)
>+        void linkTo(Label label, AbstractMacroAssemblerType* masm) const
>         {
>             size_t size = m_jumps.size();
>             for (size_t i = 0; i < size; ++i)
>                 m_jumps[i].linkTo(label, masm);
>-            m_jumps.clear();
>         }
>         
>         void append(Jump jump)
>Index: Source/JavaScriptCore/assembler/MacroAssembler.h
>===================================================================
>--- Source/JavaScriptCore/assembler/MacroAssembler.h	(revision 204735)
>+++ Source/JavaScriptCore/assembler/MacroAssembler.h	(working copy)
>@@ -28,6 +28,8 @@
> 
> #if ENABLE(ASSEMBLER)
> 
>+#include "JSCJSValue.h"
>+
> #if CPU(ARM_THUMB2)
> #include "MacroAssemblerARMv7.h"
> namespace JSC { typedef MacroAssemblerARMv7 MacroAssemblerBase; };
>Index: Source/JavaScriptCore/assembler/MacroAssemblerARM64.h
>===================================================================
>--- Source/JavaScriptCore/assembler/MacroAssemblerARM64.h	(revision 204735)
>+++ Source/JavaScriptCore/assembler/MacroAssemblerARM64.h	(working copy)
>@@ -166,7 +166,10 @@ public:
>             m_assembler.add<32>(dest, src, UInt12(imm.m_value));
>         else if (isUInt12(-imm.m_value))
>             m_assembler.sub<32>(dest, src, UInt12(-imm.m_value));
>-        else {
>+        else if (src != dest) {
>+            move(imm, dest);
>+            add32(src, dest);
>+        } else {
>             move(imm, getCachedDataTempRegisterIDAndInvalidate());
>             m_assembler.add<32>(dest, src, dataTempRegister);
>         }
>Index: Source/JavaScriptCore/assembler/MacroAssemblerCodeRef.cpp
>===================================================================
>--- Source/JavaScriptCore/assembler/MacroAssemblerCodeRef.cpp	(nonexistent)
>+++ Source/JavaScriptCore/assembler/MacroAssemblerCodeRef.cpp	(working copy)
>@@ -0,0 +1,68 @@
>+/*
>+ * Copyright (C) 2016 Apple Inc. All rights reserved.
>+ *
>+ * Redistribution and use in source and binary forms, with or without
>+ * modification, are permitted provided that the following conditions
>+ * are met:
>+ * 1. Redistributions of source code must retain the above copyright
>+ *    notice, this list of conditions and the following disclaimer.
>+ * 2. Redistributions in binary form must reproduce the above copyright
>+ *    notice, this list of conditions and the following disclaimer in the
>+ *    documentation and/or other materials provided with the distribution.
>+ *
>+ * THIS SOFTWARE IS PROVIDED BY APPLE INC. ``AS IS'' AND ANY
>+ * EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
>+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
>+ * PURPOSE ARE DISCLAIMED.  IN NO EVENT SHALL APPLE INC. OR
>+ * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
>+ * EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
>+ * PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
>+ * PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY
>+ * OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
>+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
>+ * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 
>+ */
>+
>+#include "config.h"
>+#include "MacroAssemblerCodeRef.h"
>+
>+#include "JSCInlines.h"
>+#include "LLIntData.h"
>+
>+namespace JSC {
>+
>+MacroAssemblerCodePtr MacroAssemblerCodePtr::createLLIntCodePtr(OpcodeID codeId)
>+{
>+    return createFromExecutableAddress(LLInt::getCodePtr(codeId));
>+}
>+
>+void MacroAssemblerCodePtr::dumpWithName(const char* name, PrintStream& out) const
>+{
>+    if (!m_value) {
>+        out.print(name, "(null)");
>+        return;
>+    }
>+    if (executableAddress() == dataLocation()) {
>+        out.print(name, "(", RawPointer(executableAddress()), ")");
>+        return;
>+    }
>+    out.print(name, "(executable = ", RawPointer(executableAddress()), ", dataLocation = ", RawPointer(dataLocation()), ")");
>+}
>+
>+void MacroAssemblerCodePtr::dump(PrintStream& out) const
>+{
>+    dumpWithName("CodePtr", out);
>+}
>+
>+MacroAssemblerCodeRef MacroAssemblerCodeRef::createLLIntCodeRef(OpcodeID codeId)
>+{
>+    return createSelfManagedCodeRef(MacroAssemblerCodePtr::createFromExecutableAddress(LLInt::getCodePtr(codeId)));
>+}
>+
>+void MacroAssemblerCodeRef::dump(PrintStream& out) const
>+{
>+    m_codePtr.dumpWithName("CodeRef", out);
>+}
>+
>+} // namespace JSC
>+
>Index: Source/JavaScriptCore/assembler/MacroAssemblerCodeRef.h
>===================================================================
>--- Source/JavaScriptCore/assembler/MacroAssemblerCodeRef.h	(revision 204735)
>+++ Source/JavaScriptCore/assembler/MacroAssemblerCodeRef.h	(working copy)
>@@ -28,7 +28,6 @@
> 
> #include "Disassembler.h"
> #include "ExecutableAllocator.h"
>-#include "LLIntData.h"
> #include <wtf/DataLog.h>
> #include <wtf/PassRefPtr.h>
> #include <wtf/PrintStream.h>
>@@ -53,6 +52,8 @@
> 
> namespace JSC {
> 
>+enum OpcodeID : unsigned;
>+
> // FunctionPtr:
> //
> // FunctionPtr should be used to wrap pointers to C/C++ functions in JSC
>@@ -273,10 +274,7 @@ public:
>         return result;
>     }
> 
>-    static MacroAssemblerCodePtr createLLIntCodePtr(OpcodeID codeId)
>-    {
>-        return createFromExecutableAddress(LLInt::getCodePtr(codeId));
>-    }
>+    static MacroAssemblerCodePtr createLLIntCodePtr(OpcodeID codeId);
> 
>     explicit MacroAssemblerCodePtr(ReturnAddressPtr ra)
>         : m_value(ra.value())
>@@ -299,23 +297,9 @@ public:
>         return m_value == other.m_value;
>     }
> 
>-    void dumpWithName(const char* name, PrintStream& out) const
>-    {
>-        if (!m_value) {
>-            out.print(name, "(null)");
>-            return;
>-        }
>-        if (executableAddress() == dataLocation()) {
>-            out.print(name, "(", RawPointer(executableAddress()), ")");
>-            return;
>-        }
>-        out.print(name, "(executable = ", RawPointer(executableAddress()), ", dataLocation = ", RawPointer(dataLocation()), ")");
>-    }
>+    void dumpWithName(const char* name, PrintStream& out) const;
>     
>-    void dump(PrintStream& out) const
>-    {
>-        dumpWithName("CodePtr", out);
>-    }
>+    void dump(PrintStream& out) const;
>     
>     enum EmptyValueTag { EmptyValue };
>     enum DeletedValueTag { DeletedValue };
>@@ -389,10 +373,7 @@ public:
>     }
>     
>     // Helper for creating self-managed code refs from LLInt.
>-    static MacroAssemblerCodeRef createLLIntCodeRef(OpcodeID codeId)
>-    {
>-        return createSelfManagedCodeRef(MacroAssemblerCodePtr::createFromExecutableAddress(LLInt::getCodePtr(codeId)));
>-    }
>+    static MacroAssemblerCodeRef createLLIntCodeRef(OpcodeID codeId);
> 
>     ExecutableMemoryHandle* executableMemory() const
>     {
>@@ -418,10 +399,7 @@ public:
>     
>     explicit operator bool() const { return !!m_codePtr; }
>     
>-    void dump(PrintStream& out) const
>-    {
>-        m_codePtr.dumpWithName("CodeRef", out);
>-    }
>+    void dump(PrintStream& out) const;
> 
> private:
>     MacroAssemblerCodePtr m_codePtr;
>Index: Source/JavaScriptCore/b3/B3BasicBlock.cpp
>===================================================================
>--- Source/JavaScriptCore/b3/B3BasicBlock.cpp	(revision 204735)
>+++ Source/JavaScriptCore/b3/B3BasicBlock.cpp	(working copy)
>@@ -85,6 +85,11 @@ Value* BasicBlock::appendIntConstant(Pro
>     return appendIntConstant(proc, likeValue->origin(), likeValue->type(), value);
> }
> 
>+Value* BasicBlock::appendBoolConstant(Procedure& proc, Origin origin, bool value)
>+{
>+    return appendIntConstant(proc, origin, Int32, value ? 1 : 0);
>+}
>+
> void BasicBlock::clearSuccessors()
> {
>     m_successors.clear();
>Index: Source/JavaScriptCore/b3/B3BasicBlock.h
>===================================================================
>--- Source/JavaScriptCore/b3/B3BasicBlock.h	(revision 204735)
>+++ Source/JavaScriptCore/b3/B3BasicBlock.h	(working copy)
>@@ -82,6 +82,7 @@ public:
> 
>     JS_EXPORT_PRIVATE Value* appendIntConstant(Procedure&, Origin, Type, int64_t value);
>     Value* appendIntConstant(Procedure&, Value* likeValue, int64_t value);
>+    Value* appendBoolConstant(Procedure&, Origin, bool);
> 
>     void removeLast(Procedure&);
>     
>Index: Source/JavaScriptCore/b3/B3DuplicateTails.cpp
>===================================================================
>--- Source/JavaScriptCore/b3/B3DuplicateTails.cpp	(revision 204735)
>+++ Source/JavaScriptCore/b3/B3DuplicateTails.cpp	(working copy)
>@@ -71,7 +71,11 @@ public:
>         IndexSet<BasicBlock> candidates;
> 
>         for (BasicBlock* block : m_proc) {
>-            if (block->size() > m_maxSize || block->numSuccessors() > m_maxSuccessors)
>+            if (block->size() > m_maxSize)
>+                continue;
>+            if (block->numSuccessors() > m_maxSuccessors)
>+                continue;
>+            if (block->last()->type() != Void) // Demoting doesn't handle terminals with values.
>                 continue;
> 
>             candidates.add(block);
>Index: Source/JavaScriptCore/b3/B3StackmapGenerationParams.h
>===================================================================
>--- Source/JavaScriptCore/b3/B3StackmapGenerationParams.h	(revision 204735)
>+++ Source/JavaScriptCore/b3/B3StackmapGenerationParams.h	(working copy)
>@@ -92,7 +92,7 @@ public:
>     // This is computed lazily, so it won't work if you capture StackmapGenerationParams by value.
>     // Returns true if the successor at the given index is going to be emitted right after the
>     // patchpoint.
>-    bool fallsThroughToSuccessor(unsigned successorIndex) const;
>+    JS_EXPORT_PRIVATE bool fallsThroughToSuccessor(unsigned successorIndex) const;
> 
>     // This is provided for convenience; it means that you don't have to capture it if you don't want to.
>     JS_EXPORT_PRIVATE Procedure& proc() const;
>Index: Source/JavaScriptCore/b3/testb3.cpp
>===================================================================
>--- Source/JavaScriptCore/b3/testb3.cpp	(revision 204735)
>+++ Source/JavaScriptCore/b3/testb3.cpp	(working copy)
>@@ -12920,6 +12920,80 @@ void testBranchBitAndImmFusion(
>     CHECK(terminal.args[2].kind() == Air::Arg::BitImm || terminal.args[2].kind() == Air::Arg::BitImm64);
> }
> 
>+void testPatchpointTerminalReturnValue(bool successIsRare)
>+{
>+    // This is a unit test for how FTL's heap allocation fast paths behave.
>+    Procedure proc;
>+    
>+    BasicBlock* root = proc.addBlock();
>+    BasicBlock* success = proc.addBlock();
>+    BasicBlock* slowPath = proc.addBlock();
>+    BasicBlock* continuation = proc.addBlock();
>+    
>+    Value* arg = root->appendNew<Value>(
>+        proc, Trunc, Origin(),
>+        root->appendNew<ArgumentRegValue>(proc, Origin(), GPRInfo::argumentGPR0));
>+    
>+    PatchpointValue* patchpoint = root->appendNew<PatchpointValue>(proc, Int32, Origin());
>+    patchpoint->effects.terminal = true;
>+    patchpoint->clobber(RegisterSet::macroScratchRegisters());
>+    
>+    if (successIsRare) {
>+        root->appendSuccessor(FrequentedBlock(success, FrequencyClass::Rare));
>+        root->appendSuccessor(slowPath);
>+    } else {
>+        root->appendSuccessor(success);
>+        root->appendSuccessor(FrequentedBlock(slowPath, FrequencyClass::Rare));
>+    }
>+    
>+    patchpoint->appendSomeRegister(arg);
>+    
>+    patchpoint->setGenerator(
>+        [&] (CCallHelpers& jit, const StackmapGenerationParams& params) {
>+            AllowMacroScratchRegisterUsage allowScratch(jit);
>+            
>+            CCallHelpers::Jump jumpToSlow =
>+                jit.branch32(CCallHelpers::Above, params[1].gpr(), CCallHelpers::TrustedImm32(42));
>+            
>+            jit.add32(CCallHelpers::TrustedImm32(31), params[1].gpr(), params[0].gpr());
>+            
>+            CCallHelpers::Jump jumpToSuccess;
>+            if (!params.fallsThroughToSuccessor(0))
>+                jumpToSuccess = jit.jump();
>+            
>+            Vector<Box<CCallHelpers::Label>> labels = params.successorLabels();
>+            
>+            params.addLatePath(
>+                [=] (CCallHelpers& jit) {
>+                    jumpToSlow.linkTo(*labels[1], &jit);
>+                    if (jumpToSuccess.isSet())
>+                        jumpToSuccess.linkTo(*labels[0], &jit);
>+                });
>+        });
>+    
>+    UpsilonValue* successUpsilon = success->appendNew<UpsilonValue>(proc, Origin(), patchpoint);
>+    success->appendNew<Value>(proc, Jump, Origin());
>+    success->setSuccessors(continuation);
>+    
>+    UpsilonValue* slowPathUpsilon = slowPath->appendNew<UpsilonValue>(
>+        proc, Origin(), slowPath->appendNew<Const32Value>(proc, Origin(), 666));
>+    slowPath->appendNew<Value>(proc, Jump, Origin());
>+    slowPath->setSuccessors(continuation);
>+    
>+    Value* phi = continuation->appendNew<Value>(proc, Phi, Int32, Origin());
>+    successUpsilon->setPhi(phi);
>+    slowPathUpsilon->setPhi(phi);
>+    continuation->appendNew<Value>(proc, Return, Origin(), phi);
>+    
>+    auto code = compile(proc);
>+    CHECK_EQ(invoke<int>(*code, 0), 31);
>+    CHECK_EQ(invoke<int>(*code, 1), 32);
>+    CHECK_EQ(invoke<int>(*code, 41), 72);
>+    CHECK_EQ(invoke<int>(*code, 42), 73);
>+    CHECK_EQ(invoke<int>(*code, 43), 666);
>+    CHECK_EQ(invoke<int>(*code, -1), 666);
>+}
>+
> // Make sure the compiler does not try to optimize anything out.
> NEVER_INLINE double zero()
> {
>@@ -14337,6 +14411,8 @@ void run(const char* filter)
>     RUN(testEntrySwitchLoop());
> 
>     RUN(testSomeEarlyRegister());
>+    RUN(testPatchpointTerminalReturnValue(true));
>+    RUN(testPatchpointTerminalReturnValue(false));
>     
>     if (isX86()) {
>         RUN(testBranchBitAndImmFusion(Identity, Int64, 1, Air::BranchTest32, Air::Arg::Tmp));
>Index: Source/JavaScriptCore/bindings/ScriptValue.cpp
>===================================================================
>--- Source/JavaScriptCore/bindings/ScriptValue.cpp	(revision 204735)
>+++ Source/JavaScriptCore/bindings/ScriptValue.cpp	(working copy)
>@@ -32,8 +32,8 @@
> 
> #include "APICast.h"
> #include "InspectorValues.h"
>+#include "JSCInlines.h"
> #include "JSLock.h"
>-#include "StructureInlines.h"
> 
> using namespace JSC;
> using namespace Inspector;
>Index: Source/JavaScriptCore/bytecode/AdaptiveInferredPropertyValueWatchpointBase.cpp
>===================================================================
>--- Source/JavaScriptCore/bytecode/AdaptiveInferredPropertyValueWatchpointBase.cpp	(revision 204735)
>+++ Source/JavaScriptCore/bytecode/AdaptiveInferredPropertyValueWatchpointBase.cpp	(working copy)
>@@ -26,8 +26,7 @@
> #include "config.h"
> #include "AdaptiveInferredPropertyValueWatchpointBase.h"
> 
>-#include "JSCellInlines.h"
>-#include "StructureInlines.h"
>+#include "JSCInlines.h"
> 
> namespace JSC {
> 
>Index: Source/JavaScriptCore/bytecode/BytecodeBasicBlock.cpp
>===================================================================
>--- Source/JavaScriptCore/bytecode/BytecodeBasicBlock.cpp	(revision 204735)
>+++ Source/JavaScriptCore/bytecode/BytecodeBasicBlock.cpp	(working copy)
>@@ -27,6 +27,7 @@
> #include "BytecodeBasicBlock.h"
> 
> #include "CodeBlock.h"
>+#include "Interpreter.h"
> #include "JSCInlines.h"
> #include "PreciseJumpTargets.h"
> 
>Index: Source/JavaScriptCore/bytecode/BytecodeLivenessAnalysis.cpp
>===================================================================
>--- Source/JavaScriptCore/bytecode/BytecodeLivenessAnalysis.cpp	(revision 204735)
>+++ Source/JavaScriptCore/bytecode/BytecodeLivenessAnalysis.cpp	(working copy)
>@@ -31,6 +31,7 @@
> #include "BytecodeUseDef.h"
> #include "CodeBlock.h"
> #include "FullBytecodeLiveness.h"
>+#include "Interpreter.h"
> #include "PreciseJumpTargets.h"
> 
> namespace JSC {
>Index: Source/JavaScriptCore/bytecode/BytecodeUseDef.h
>===================================================================
>--- Source/JavaScriptCore/bytecode/BytecodeUseDef.h	(revision 204735)
>+++ Source/JavaScriptCore/bytecode/BytecodeUseDef.h	(working copy)
>@@ -27,6 +27,7 @@
> #define BytecodeUseDef_h
> 
> #include "CodeBlock.h"
>+#include "Interpreter.h"
> 
> namespace JSC {
> 
>Index: Source/JavaScriptCore/bytecode/CallLinkInfo.cpp
>===================================================================
>--- Source/JavaScriptCore/bytecode/CallLinkInfo.cpp	(revision 204735)
>+++ Source/JavaScriptCore/bytecode/CallLinkInfo.cpp	(working copy)
>@@ -30,12 +30,29 @@
> #include "DFGOperations.h"
> #include "DFGThunks.h"
> #include "JSCInlines.h"
>+#include "Opcode.h"
> #include "Repatch.h"
> #include <wtf/ListDump.h>
> 
> #if ENABLE(JIT)
> namespace JSC {
> 
>+CallLinkInfo::CallType CallLinkInfo::callTypeFor(OpcodeID opcodeID)
>+{
>+    if (opcodeID == op_call || opcodeID == op_call_eval)
>+        return Call;
>+    if (opcodeID == op_call_varargs)
>+        return CallVarargs;
>+    if (opcodeID == op_construct)
>+        return Construct;
>+    if (opcodeID == op_construct_varargs)
>+        return ConstructVarargs;
>+    if (opcodeID == op_tail_call)
>+        return TailCall;
>+    ASSERT(opcodeID == op_tail_call_varargs || op_tail_call_forward_arguments);
>+    return TailCallVarargs;
>+}
>+
> CallLinkInfo::CallLinkInfo()
>     : m_hasSeenShouldRepatch(false)
>     , m_hasSeenClosure(false)
>Index: Source/JavaScriptCore/bytecode/CallLinkInfo.h
>===================================================================
>--- Source/JavaScriptCore/bytecode/CallLinkInfo.h	(revision 204735)
>+++ Source/JavaScriptCore/bytecode/CallLinkInfo.h	(working copy)
>@@ -31,7 +31,6 @@
> #include "CodeSpecializationKind.h"
> #include "JITWriteBarrier.h"
> #include "JSFunction.h"
>-#include "Opcode.h"
> #include "PolymorphicCallStubRoutine.h"
> #include "WriteBarrier.h"
> #include <wtf/SentinelLinkedList.h>
>@@ -40,26 +39,13 @@ namespace JSC {
> 
> #if ENABLE(JIT)
> 
>+enum OpcodeID : unsigned;
> struct CallFrameShuffleData;
> 
> class CallLinkInfo : public BasicRawSentinelNode<CallLinkInfo> {
> public:
>     enum CallType { None, Call, CallVarargs, Construct, ConstructVarargs, TailCall, TailCallVarargs };
>-    static CallType callTypeFor(OpcodeID opcodeID)
>-    {
>-        if (opcodeID == op_call || opcodeID == op_call_eval)
>-            return Call;
>-        if (opcodeID == op_call_varargs)
>-            return CallVarargs;
>-        if (opcodeID == op_construct)
>-            return Construct;
>-        if (opcodeID == op_construct_varargs)
>-            return ConstructVarargs;
>-        if (opcodeID == op_tail_call)
>-            return TailCall;
>-        ASSERT(opcodeID == op_tail_call_varargs || op_tail_call_forward_arguments);
>-        return TailCallVarargs;
>-    }
>+    static CallType callTypeFor(OpcodeID opcodeID);
> 
>     static bool isVarargsCallType(CallType callType)
>     {
>Index: Source/JavaScriptCore/bytecode/CallLinkStatus.cpp
>===================================================================
>--- Source/JavaScriptCore/bytecode/CallLinkStatus.cpp	(revision 204735)
>+++ Source/JavaScriptCore/bytecode/CallLinkStatus.cpp	(working copy)
>@@ -30,6 +30,7 @@
> #include "CodeBlock.h"
> #include "DFGJITCode.h"
> #include "InlineCallFrame.h"
>+#include "Interpreter.h"
> #include "LLIntCallLinkInfo.h"
> #include "JSCInlines.h"
> #include <wtf/CommaPrinter.h>
>Index: Source/JavaScriptCore/bytecode/CodeBlock.cpp
>===================================================================
>--- Source/JavaScriptCore/bytecode/CodeBlock.cpp	(revision 204735)
>+++ Source/JavaScriptCore/bytecode/CodeBlock.cpp	(working copy)
>@@ -51,6 +51,7 @@
> #include "JSFunction.h"
> #include "JSLexicalEnvironment.h"
> #include "JSModuleEnvironment.h"
>+#include "LLIntData.h"
> #include "LLIntEntrypoint.h"
> #include "LLIntPrototypeLoadAdaptiveStructureWatchpoint.h"
> #include "LowLevelInterpreter.h"
>@@ -1910,7 +1911,7 @@ void CodeBlock::finishCreation(VM& vm, C
>         m_rareData->m_liveCalleeLocalsAtYield = other.m_rareData->m_liveCalleeLocalsAtYield;
>     }
>     
>-    heap()->m_codeBlocks.add(this);
>+    heap()->m_codeBlocks->add(this);
> }
> 
> CodeBlock::CodeBlock(VM* vm, Structure* structure, ScriptExecutable* ownerExecutable, UnlinkedCodeBlock* unlinkedCodeBlock,
>@@ -2402,7 +2403,7 @@ void CodeBlock::finishCreation(VM& vm, S
>     if (Options::dumpGeneratedBytecodes())
>         dumpBytecode();
>     
>-    heap()->m_codeBlocks.add(this);
>+    heap()->m_codeBlocks->add(this);
>     heap()->reportExtraMemoryAllocated(m_instructions.size() * sizeof(Instruction));
> }
> 
>@@ -2439,7 +2440,7 @@ void CodeBlock::finishCreation(VM& vm, W
> {
>     Base::finishCreation(vm);
> 
>-    heap()->m_codeBlocks.add(this);
>+    heap()->m_codeBlocks->add(this);
> }
> #endif
> 
>@@ -2843,6 +2844,14 @@ void CodeBlock::WeakReferenceHarvester::
>     codeBlock->determineLiveness(visitor);
> }
> 
>+void CodeBlock::clearLLIntGetByIdCache(Instruction* instruction)
>+{
>+    instruction[0].u.opcode = LLInt::getOpcode(op_get_by_id);
>+    instruction[4].u.pointer = nullptr;
>+    instruction[5].u.pointer = nullptr;
>+    instruction[6].u.pointer = nullptr;
>+}
>+
> void CodeBlock::finalizeLLIntInlineCaches()
> {
> #if ENABLE(WEBASSEMBLY)
>Index: Source/JavaScriptCore/bytecode/CodeBlock.h
>===================================================================
>--- Source/JavaScriptCore/bytecode/CodeBlock.h	(revision 204735)
>+++ Source/JavaScriptCore/bytecode/CodeBlock.h	(working copy)
>@@ -297,6 +297,8 @@ public:
>     {
>         return m_jitCodeMap.get();
>     }
>+    
>+    static void clearLLIntGetByIdCache(Instruction*);
> 
>     unsigned bytecodeOffset(Instruction* returnAddress)
>     {
>@@ -1303,14 +1305,6 @@ private:
> };
> #endif
> 
>-inline void clearLLIntGetByIdCache(Instruction* instruction)
>-{
>-    instruction[0].u.opcode = LLInt::getOpcode(op_get_by_id);
>-    instruction[4].u.pointer = nullptr;
>-    instruction[5].u.pointer = nullptr;
>-    instruction[6].u.pointer = nullptr;
>-}
>-
> inline Register& ExecState::r(int index)
> {
>     CodeBlock* codeBlock = this->codeBlock();
>Index: Source/JavaScriptCore/bytecode/Instruction.h
>===================================================================
>--- Source/JavaScriptCore/bytecode/Instruction.h	(revision 204735)
>+++ Source/JavaScriptCore/bytecode/Instruction.h	(working copy)
>@@ -31,7 +31,6 @@
> 
> #include "BasicBlockLocation.h"
> #include "MacroAssembler.h"
>-#include "Opcode.h"
> #include "PutByIdFlags.h"
> #include "SymbolTable.h"
> #include "TypeLocation.h"
>@@ -52,6 +51,12 @@ class WatchpointSet;
> struct LLIntCallLinkInfo;
> struct ValueProfile;
> 
>+#if ENABLE(COMPUTED_GOTO_OPCODES)
>+typedef void* Opcode;
>+#else
>+typedef OpcodeID Opcode;
>+#endif
>+
> struct Instruction {
>     Instruction()
>     {
>Index: Source/JavaScriptCore/bytecode/LLIntPrototypeLoadAdaptiveStructureWatchpoint.cpp
>===================================================================
>--- Source/JavaScriptCore/bytecode/LLIntPrototypeLoadAdaptiveStructureWatchpoint.cpp	(revision 204735)
>+++ Source/JavaScriptCore/bytecode/LLIntPrototypeLoadAdaptiveStructureWatchpoint.cpp	(working copy)
>@@ -28,7 +28,7 @@
> 
> #include "CodeBlock.h"
> #include "Instruction.h"
>-#include "StructureInlines.h"
>+#include "JSCInlines.h"
> 
> namespace JSC {
> 
>@@ -59,7 +59,7 @@ void LLIntPrototypeLoadAdaptiveStructure
> 
>     StringFireDetail stringDetail(out.toCString().data());
> 
>-    clearLLIntGetByIdCache(m_getByIdInstruction);
>+    CodeBlock::clearLLIntGetByIdCache(m_getByIdInstruction);
> }
> 
> } // namespace JSC
>Index: Source/JavaScriptCore/bytecode/ObjectAllocationProfile.h
>===================================================================
>--- Source/JavaScriptCore/bytecode/ObjectAllocationProfile.h	(revision 204735)
>+++ Source/JavaScriptCore/bytecode/ObjectAllocationProfile.h	(working copy)
>@@ -80,14 +80,15 @@ public:
>         ASSERT(inlineCapacity <= JSFinalObject::maxInlineCapacity());
> 
>         size_t allocationSize = JSFinalObject::allocationSize(inlineCapacity);
>-        MarkedAllocator* allocator = &vm.heap.allocatorForObjectWithoutDestructor(allocationSize);
>-        ASSERT(allocator->cellSize());
>-
>+        MarkedAllocator* allocator = vm.heap.allocatorForObjectWithoutDestructor(allocationSize);
>+        
>         // Take advantage of extra inline capacity available in the size class.
>-        size_t slop = (allocator->cellSize() - allocationSize) / sizeof(WriteBarrier<Unknown>);
>-        inlineCapacity += slop;
>-        if (inlineCapacity > JSFinalObject::maxInlineCapacity())
>-            inlineCapacity = JSFinalObject::maxInlineCapacity();
>+        if (allocator) {
>+            size_t slop = (allocator->cellSize() - allocationSize) / sizeof(WriteBarrier<Unknown>);
>+            inlineCapacity += slop;
>+            if (inlineCapacity > JSFinalObject::maxInlineCapacity())
>+                inlineCapacity = JSFinalObject::maxInlineCapacity();
>+        }
> 
>         Structure* structure = vm.prototypeMap.emptyObjectStructureForPrototype(prototype, inlineCapacity);
> 
>Index: Source/JavaScriptCore/bytecode/Opcode.h
>===================================================================
>--- Source/JavaScriptCore/bytecode/Opcode.h	(revision 204735)
>+++ Source/JavaScriptCore/bytecode/Opcode.h	(working copy)
>@@ -55,7 +55,7 @@ namespace JSC {
> 
> 
> #define OPCODE_ID_ENUM(opcode, length) opcode,
>-    typedef enum { FOR_EACH_OPCODE_ID(OPCODE_ID_ENUM) } OpcodeID;
>+    enum OpcodeID : unsigned { FOR_EACH_OPCODE_ID(OPCODE_ID_ENUM) };
> #undef OPCODE_ID_ENUM
> 
> const int maxOpcodeLength = 9;
>Index: Source/JavaScriptCore/bytecode/PolymorphicAccess.cpp
>===================================================================
>--- Source/JavaScriptCore/bytecode/PolymorphicAccess.cpp	(revision 204735)
>+++ Source/JavaScriptCore/bytecode/PolymorphicAccess.cpp	(working copy)
>@@ -1206,34 +1206,25 @@ void AccessCase::generateImpl(AccessGene
>             size_t newSize = newStructure()->outOfLineCapacity() * sizeof(JSValue);
>             
>             if (allocatingInline) {
>-                CopiedAllocator* copiedAllocator = &vm.heap.storageAllocator();
>-
>-                if (!reallocating) {
>-                    jit.loadPtr(&copiedAllocator->m_currentRemaining, scratchGPR);
>-                    slowPath.append(
>-                        jit.branchSubPtr(
>-                            CCallHelpers::Signed, CCallHelpers::TrustedImm32(newSize), scratchGPR));
>-                    jit.storePtr(scratchGPR, &copiedAllocator->m_currentRemaining);
>-                    jit.negPtr(scratchGPR);
>-                    jit.addPtr(
>-                        CCallHelpers::AbsoluteAddress(&copiedAllocator->m_currentPayloadEnd), scratchGPR);
>-                    jit.addPtr(CCallHelpers::TrustedImm32(sizeof(JSValue)), scratchGPR);
>-                } else {
>+                MarkedAllocator* allocator = vm.heap.allocatorForAuxiliaryData(newSize);
>+                
>+                if (!allocator) {
>+                    // Yuck, this case would suck!
>+                    slowPath.append(jit.jump());
>+                }
>+                
>+                jit.move(CCallHelpers::TrustedImmPtr(allocator), scratchGPR2);
>+                jit.emitAllocate(scratchGPR, allocator, scratchGPR2, scratchGPR3, slowPath);
>+                jit.addPtr(CCallHelpers::TrustedImm32(newSize + sizeof(IndexingHeader)), scratchGPR);
>+                
>+                if (reallocating) {
>                     // Handle the case where we are reallocating (i.e. the old structure/butterfly
>                     // already had out-of-line property storage).
>                     size_t oldSize = structure()->outOfLineCapacity() * sizeof(JSValue);
>                     ASSERT(newSize > oldSize);
>             
>                     jit.loadPtr(CCallHelpers::Address(baseGPR, JSObject::butterflyOffset()), scratchGPR3);
>-                    jit.loadPtr(&copiedAllocator->m_currentRemaining, scratchGPR);
>-                    slowPath.append(
>-                        jit.branchSubPtr(
>-                            CCallHelpers::Signed, CCallHelpers::TrustedImm32(newSize), scratchGPR));
>-                    jit.storePtr(scratchGPR, &copiedAllocator->m_currentRemaining);
>-                    jit.negPtr(scratchGPR);
>-                    jit.addPtr(
>-                        CCallHelpers::AbsoluteAddress(&copiedAllocator->m_currentPayloadEnd), scratchGPR);
>-                    jit.addPtr(CCallHelpers::TrustedImm32(sizeof(JSValue)), scratchGPR);
>+                    
>                     // We have scratchGPR = new storage, scratchGPR3 = old storage,
>                     // scratchGPR2 = available
>                     for (size_t offset = 0; offset < oldSize; offset += sizeof(void*)) {
>@@ -1659,6 +1650,7 @@ AccessGenerationResult PolymorphicAccess
>         // Cascade through the list, preferring newer entries.
>         for (unsigned i = cases.size(); i--;) {
>             fallThrough.link(&jit);
>+            fallThrough.clear();
>             cases[i]->generateWithGuard(state, fallThrough);
>         }
>         state.failAndRepatch.append(fallThrough);
>Index: Source/JavaScriptCore/bytecode/PolymorphicAccess.h
>===================================================================
>--- Source/JavaScriptCore/bytecode/PolymorphicAccess.h	(revision 204735)
>+++ Source/JavaScriptCore/bytecode/PolymorphicAccess.h	(working copy)
>@@ -29,10 +29,10 @@
> #if ENABLE(JIT)
> 
> #include "CodeOrigin.h"
>+#include "JITStubRoutine.h"
> #include "JSFunctionInlines.h"
> #include "MacroAssembler.h"
> #include "ObjectPropertyConditionSet.h"
>-#include "Opcode.h"
> #include "ScratchRegisterAllocator.h"
> #include "Structure.h"
> #include <wtf/Vector.h>
>Index: Source/JavaScriptCore/bytecode/PreciseJumpTargets.cpp
>===================================================================
>--- Source/JavaScriptCore/bytecode/PreciseJumpTargets.cpp	(revision 204735)
>+++ Source/JavaScriptCore/bytecode/PreciseJumpTargets.cpp	(working copy)
>@@ -26,6 +26,7 @@
> #include "config.h"
> #include "PreciseJumpTargets.h"
> 
>+#include "Interpreter.h"
> #include "JSCInlines.h"
> 
> namespace JSC {
>Index: Source/JavaScriptCore/bytecode/StructureStubInfo.cpp
>===================================================================
>--- Source/JavaScriptCore/bytecode/StructureStubInfo.cpp	(revision 204735)
>+++ Source/JavaScriptCore/bytecode/StructureStubInfo.cpp	(working copy)
>@@ -27,6 +27,7 @@
> #include "StructureStubInfo.h"
> 
> #include "JSObject.h"
>+#include "JSCInlines.h"
> #include "PolymorphicAccess.h"
> #include "Repatch.h"
> 
>Index: Source/JavaScriptCore/bytecode/StructureStubInfo.h
>===================================================================
>--- Source/JavaScriptCore/bytecode/StructureStubInfo.h	(revision 204735)
>+++ Source/JavaScriptCore/bytecode/StructureStubInfo.h	(working copy)
>@@ -31,7 +31,6 @@
> #include "JITStubRoutine.h"
> #include "MacroAssembler.h"
> #include "ObjectPropertyConditionSet.h"
>-#include "Opcode.h"
> #include "Options.h"
> #include "RegisterSet.h"
> #include "Structure.h"
>Index: Source/JavaScriptCore/bytecode/UnlinkedInstructionStream.cpp
>===================================================================
>--- Source/JavaScriptCore/bytecode/UnlinkedInstructionStream.cpp	(revision 204735)
>+++ Source/JavaScriptCore/bytecode/UnlinkedInstructionStream.cpp	(working copy)
>@@ -26,6 +26,8 @@
> #include "config.h"
> #include "UnlinkedInstructionStream.h"
> 
>+#include "Opcode.h"
>+
> namespace JSC {
> 
> static void append8(unsigned char*& ptr, unsigned char value)
>Index: Source/JavaScriptCore/bytecode/UnlinkedInstructionStream.h
>===================================================================
>--- Source/JavaScriptCore/bytecode/UnlinkedInstructionStream.h	(revision 204735)
>+++ Source/JavaScriptCore/bytecode/UnlinkedInstructionStream.h	(working copy)
>@@ -27,6 +27,7 @@
> #ifndef UnlinkedInstructionStream_h
> #define UnlinkedInstructionStream_h
> 
>+#include "Opcode.h"
> #include "UnlinkedCodeBlock.h"
> #include <wtf/RefCountedArray.h>
> 
>Index: Source/JavaScriptCore/dfg/DFGOperations.cpp
>===================================================================
>--- Source/JavaScriptCore/dfg/DFGOperations.cpp	(revision 204735)
>+++ Source/JavaScriptCore/dfg/DFGOperations.cpp	(working copy)
>@@ -878,7 +878,6 @@ char* JIT_OPERATION operationNewArrayWit
>         return bitwise_cast<char*>(exec->vm().throwException(exec, createRangeError(exec, ASCIILiteral("Array size is not a small enough positive integer."))));
> 
>     JSArray* result = JSArray::create(*vm, arrayStructure, size);
>-    result->butterfly(); // Ensure that the backing store is in to-space.
>     return bitwise_cast<char*>(result);
> }
> 
>Index: Source/JavaScriptCore/dfg/DFGSpeculativeJIT32_64.cpp
>===================================================================
>--- Source/JavaScriptCore/dfg/DFGSpeculativeJIT32_64.cpp	(revision 204735)
>+++ Source/JavaScriptCore/dfg/DFGSpeculativeJIT32_64.cpp	(working copy)
>@@ -4084,7 +4084,7 @@ void SpeculativeJIT::compile(Node* node)
>         m_jit.loadPtr(JITCompiler::Address(rareDataGPR, FunctionRareData::offsetOfObjectAllocationProfile() + ObjectAllocationProfile::offsetOfAllocator()), allocatorGPR);
>         m_jit.loadPtr(JITCompiler::Address(rareDataGPR, FunctionRareData::offsetOfObjectAllocationProfile() + ObjectAllocationProfile::offsetOfStructure()), structureGPR);
>         slowPath.append(m_jit.branchTestPtr(MacroAssembler::Zero, allocatorGPR));
>-        emitAllocateJSObject(resultGPR, allocatorGPR, structureGPR, TrustedImmPtr(0), scratchGPR, slowPath);
>+        emitAllocateJSObject(resultGPR, nullptr, allocatorGPR, structureGPR, TrustedImmPtr(0), scratchGPR, slowPath);
> 
>         addSlowPathGenerator(slowPathCall(slowPath, this, operationCreateThis, resultGPR, calleeGPR, node->inlineCapacity()));
>         
>@@ -4105,10 +4105,10 @@ void SpeculativeJIT::compile(Node* node)
>         
>         Structure* structure = node->structure();
>         size_t allocationSize = JSFinalObject::allocationSize(structure->inlineCapacity());
>-        MarkedAllocator* allocatorPtr = &m_jit.vm()->heap.allocatorForObjectWithoutDestructor(allocationSize);
>+        MarkedAllocator* allocatorPtr = m_jit.vm()->heap.allocatorForObjectWithoutDestructor(allocationSize);
> 
>         m_jit.move(TrustedImmPtr(allocatorPtr), allocatorGPR);
>-        emitAllocateJSObject(resultGPR, allocatorGPR, TrustedImmPtr(structure), TrustedImmPtr(0), scratchGPR, slowPath);
>+        emitAllocateJSObject(resultGPR, allocatorPtr, allocatorGPR, TrustedImmPtr(structure), TrustedImmPtr(0), scratchGPR, slowPath);
> 
>         addSlowPathGenerator(slowPathCall(slowPath, this, operationNewObject, resultGPR, structure));
>         
>@@ -5412,36 +5412,39 @@ void SpeculativeJIT::compileAllocateNewA
>     GPRReg storageGPR = storage.gpr();
>     GPRReg scratchGPR = scratch.gpr();
>     GPRReg scratch2GPR = scratch2.gpr();
>-    
>+            
>     MacroAssembler::JumpList slowCases;
>     if (shouldConvertLargeSizeToArrayStorage)
>         slowCases.append(m_jit.branch32(MacroAssembler::AboveOrEqual, sizeGPR, TrustedImm32(MIN_ARRAY_STORAGE_CONSTRUCTION_LENGTH)));
>-    
>+            
>     ASSERT((1 << 3) == sizeof(JSValue));
>     m_jit.move(sizeGPR, scratchGPR);
>     m_jit.lshift32(TrustedImm32(3), scratchGPR);
>     m_jit.add32(TrustedImm32(sizeof(IndexingHeader)), scratchGPR, resultGPR);
>-    slowCases.append(
>-        emitAllocateBasicStorage(resultGPR, storageGPR));
>-    m_jit.subPtr(scratchGPR, storageGPR);
>+    m_jit.emitAllocateVariableSized(
>+        storageGPR, m_jit.vm()->heap.subspaceForAuxiliaryData(), resultGPR, scratchGPR,
>+        scratch2GPR, slowCases);
>+    m_jit.addPtr(TrustedImm32(sizeof(IndexingHeader)), storageGPR);
>     Structure* structure = globalObject->arrayStructureForIndexingTypeDuringAllocation(indexingType);
>     emitAllocateJSObject<JSArray>(resultGPR, TrustedImmPtr(structure), storageGPR, scratchGPR, scratch2GPR, slowCases);
>-    
>+            
>     m_jit.store32(sizeGPR, MacroAssembler::Address(storageGPR, Butterfly::offsetOfPublicLength()));
>     m_jit.store32(sizeGPR, MacroAssembler::Address(storageGPR, Butterfly::offsetOfVectorLength()));
>-    
>-    if (hasDouble(indexingType)) {
>-        JSValue nan = JSValue(JSValue::EncodeAsDouble, PNaN);
>-        
>-        m_jit.move(sizeGPR, scratchGPR);
>-        MacroAssembler::Jump done = m_jit.branchTest32(MacroAssembler::Zero, scratchGPR);
>-        MacroAssembler::Label loop = m_jit.label();
>-        m_jit.sub32(TrustedImm32(1), scratchGPR);
>-        m_jit.store32(TrustedImm32(nan.u.asBits.tag), MacroAssembler::BaseIndex(storageGPR, scratchGPR, MacroAssembler::TimesEight, OBJECT_OFFSETOF(JSValue, u.asBits.tag)));
>-        m_jit.store32(TrustedImm32(nan.u.asBits.payload), MacroAssembler::BaseIndex(storageGPR, scratchGPR, MacroAssembler::TimesEight, OBJECT_OFFSETOF(JSValue, u.asBits.payload)));
>-        m_jit.branchTest32(MacroAssembler::NonZero, scratchGPR).linkTo(loop, &m_jit);
>-        done.link(&m_jit);
>-    }
>+            
>+    JSValue hole;
>+    if (hasDouble(indexingType))
>+        hole = JSValue(JSValue::EncodeAsDouble, PNaN);
>+    else
>+        hole = JSValue();
>+            
>+    m_jit.move(sizeGPR, scratchGPR);
>+    MacroAssembler::Jump done = m_jit.branchTest32(MacroAssembler::Zero, scratchGPR);
>+    MacroAssembler::Label loop = m_jit.label();
>+    m_jit.sub32(TrustedImm32(1), scratchGPR);
>+    m_jit.store32(TrustedImm32(hole.u.asBits.tag), MacroAssembler::BaseIndex(storageGPR, scratchGPR, MacroAssembler::TimesEight, OBJECT_OFFSETOF(JSValue, u.asBits.tag)));
>+    m_jit.store32(TrustedImm32(hole.u.asBits.payload), MacroAssembler::BaseIndex(storageGPR, scratchGPR, MacroAssembler::TimesEight, OBJECT_OFFSETOF(JSValue, u.asBits.payload)));
>+    m_jit.branchTest32(MacroAssembler::NonZero, scratchGPR).linkTo(loop, &m_jit);
>+    done.link(&m_jit);
>     
>     addSlowPathGenerator(std::make_unique<CallArrayAllocatorWithVariableSizeSlowPathGenerator>(
>         slowCases, this, operationNewArrayWithSize, resultGPR,
>Index: Source/JavaScriptCore/dfg/DFGSpeculativeJIT64.cpp
>===================================================================
>--- Source/JavaScriptCore/dfg/DFGSpeculativeJIT64.cpp	(revision 204735)
>+++ Source/JavaScriptCore/dfg/DFGSpeculativeJIT64.cpp	(working copy)
>@@ -4029,7 +4029,7 @@ void SpeculativeJIT::compile(Node* node)
>         m_jit.loadPtr(JITCompiler::Address(rareDataGPR, FunctionRareData::offsetOfObjectAllocationProfile() + ObjectAllocationProfile::offsetOfAllocator()), allocatorGPR);
>         m_jit.loadPtr(JITCompiler::Address(rareDataGPR, FunctionRareData::offsetOfObjectAllocationProfile() + ObjectAllocationProfile::offsetOfStructure()), structureGPR);
>         slowPath.append(m_jit.branchTestPtr(MacroAssembler::Zero, allocatorGPR));
>-        emitAllocateJSObject(resultGPR, allocatorGPR, structureGPR, TrustedImmPtr(0), scratchGPR, slowPath);
>+        emitAllocateJSObject(resultGPR, nullptr, allocatorGPR, structureGPR, TrustedImmPtr(0), scratchGPR, slowPath);
> 
>         addSlowPathGenerator(slowPathCall(slowPath, this, operationCreateThis, resultGPR, calleeGPR, node->inlineCapacity()));
>         
>@@ -4050,10 +4050,10 @@ void SpeculativeJIT::compile(Node* node)
> 
>         Structure* structure = node->structure();
>         size_t allocationSize = JSFinalObject::allocationSize(structure->inlineCapacity());
>-        MarkedAllocator* allocatorPtr = &m_jit.vm()->heap.allocatorForObjectWithoutDestructor(allocationSize);
>+        MarkedAllocator* allocatorPtr = m_jit.vm()->heap.allocatorForObjectWithoutDestructor(allocationSize);
> 
>         m_jit.move(TrustedImmPtr(allocatorPtr), allocatorGPR);
>-        emitAllocateJSObject(resultGPR, allocatorGPR, TrustedImmPtr(structure), TrustedImmPtr(0), scratchGPR, slowPath);
>+        emitAllocateJSObject(resultGPR, allocatorPtr, allocatorGPR, TrustedImmPtr(structure), TrustedImmPtr(0), scratchGPR, slowPath);
> 
>         addSlowPathGenerator(slowPathCall(slowPath, this, operationNewObject, resultGPR, structure));
>         
>@@ -5288,7 +5288,7 @@ void SpeculativeJIT::compile(Node* node)
> 
>         unsigned bytecodeIndex = node->origin.semantic.bytecodeIndex;
>         auto triggerIterator = m_jit.jitCode()->tierUpEntryTriggers.find(bytecodeIndex);
>-        RELEASE_ASSERT(triggerIterator != m_jit.jitCode()->tierUpEntryTriggers.end());
>+        DFG_ASSERT(m_jit.graph(), node, triggerIterator != m_jit.jitCode()->tierUpEntryTriggers.end());
>         uint8_t* forceEntryTrigger = &(m_jit.jitCode()->tierUpEntryTriggers.find(bytecodeIndex)->value);
> 
>         MacroAssembler::Jump forceOSREntry = m_jit.branchTest8(MacroAssembler::NonZero, MacroAssembler::AbsoluteAddress(forceEntryTrigger));
>@@ -5473,33 +5473,39 @@ void SpeculativeJIT::compileAllocateNewA
>     GPRReg scratch2GPR = scratch2.gpr();
>     
>     MacroAssembler::JumpList slowCases;
>+    if (true)
>+        slowCases.append(m_jit.jump());
>+            
>     if (shouldConvertLargeSizeToArrayStorage)
>         slowCases.append(m_jit.branch32(MacroAssembler::AboveOrEqual, sizeGPR, TrustedImm32(MIN_ARRAY_STORAGE_CONSTRUCTION_LENGTH)));
>-    
>+            
>     ASSERT((1 << 3) == sizeof(JSValue));
>     m_jit.move(sizeGPR, scratchGPR);
>     m_jit.lshift32(TrustedImm32(3), scratchGPR);
>     m_jit.add32(TrustedImm32(sizeof(IndexingHeader)), scratchGPR, resultGPR);
>-    slowCases.append(
>-        emitAllocateBasicStorage(resultGPR, storageGPR));
>-    m_jit.subPtr(scratchGPR, storageGPR);
>+    m_jit.emitAllocateVariableSized(
>+        storageGPR, m_jit.vm()->heap.subspaceForAuxiliaryData(), resultGPR, scratchGPR,
>+        scratch2GPR, slowCases);
>+    m_jit.addPtr(TrustedImm32(sizeof(IndexingHeader)), storageGPR);
>     Structure* structure = globalObject->arrayStructureForIndexingTypeDuringAllocation(indexingType);
>+            
>     emitAllocateJSObject<JSArray>(resultGPR, TrustedImmPtr(structure), storageGPR, scratchGPR, scratch2GPR, slowCases);
>     
>     m_jit.store32(sizeGPR, MacroAssembler::Address(storageGPR, Butterfly::offsetOfPublicLength()));
>     m_jit.store32(sizeGPR, MacroAssembler::Address(storageGPR, Butterfly::offsetOfVectorLength()));
>-    
>-    if (hasDouble(indexingType)) {
>+            
>+    if (hasDouble(indexingType))
>         m_jit.move(TrustedImm64(bitwise_cast<int64_t>(PNaN)), scratchGPR);
>-        m_jit.move(sizeGPR, scratch2GPR);
>-        MacroAssembler::Jump done = m_jit.branchTest32(MacroAssembler::Zero, scratch2GPR);
>-        MacroAssembler::Label loop = m_jit.label();
>-        m_jit.sub32(TrustedImm32(1), scratch2GPR);
>-        m_jit.store64(scratchGPR, MacroAssembler::BaseIndex(storageGPR, scratch2GPR, MacroAssembler::TimesEight));
>-        m_jit.branchTest32(MacroAssembler::NonZero, scratch2GPR).linkTo(loop, &m_jit);
>-        done.link(&m_jit);
>-    }
>-    
>+    else
>+        m_jit.move(TrustedImm64(JSValue::encode(JSValue())), scratchGPR);
>+    m_jit.move(sizeGPR, scratch2GPR);
>+    MacroAssembler::Jump done = m_jit.branchTest32(MacroAssembler::Zero, scratch2GPR);
>+    MacroAssembler::Label loop = m_jit.label();
>+    m_jit.sub32(TrustedImm32(1), scratch2GPR);
>+    m_jit.store64(scratchGPR, MacroAssembler::BaseIndex(storageGPR, scratch2GPR, MacroAssembler::TimesEight));
>+    m_jit.branchTest32(MacroAssembler::NonZero, scratch2GPR).linkTo(loop, &m_jit);
>+    done.link(&m_jit);
>+            
>     addSlowPathGenerator(std::make_unique<CallArrayAllocatorWithVariableSizeSlowPathGenerator>(
>         slowCases, this, operationNewArrayWithSize, resultGPR,
>         structure,
>Index: Source/JavaScriptCore/dfg/DFGSpeculativeJIT.cpp
>===================================================================
>--- Source/JavaScriptCore/dfg/DFGSpeculativeJIT.cpp	(revision 204735)
>+++ Source/JavaScriptCore/dfg/DFGSpeculativeJIT.cpp	(working copy)
>@@ -94,7 +94,7 @@ void SpeculativeJIT::emitAllocateRawObje
>     GPRReg scratch2GPR = scratch2.gpr();
> 
>     ASSERT(vectorLength >= numElements);
>-    vectorLength = std::max(BASE_VECTOR_LEN, vectorLength);
>+    vectorLength = Butterfly::optimalContiguousVectorLength(structure, vectorLength);
>     
>     JITCompiler::JumpList slowCases;
> 
>@@ -104,22 +104,28 @@ void SpeculativeJIT::emitAllocateRawObje
>     size += outOfLineCapacity * sizeof(JSValue);
> 
>     if (size) {
>-        slowCases.append(
>-            emitAllocateBasicStorage(TrustedImm32(size), storageGPR));
>-        if (hasIndexingHeader)
>-            m_jit.subPtr(TrustedImm32(vectorLength * sizeof(JSValue)), storageGPR);
>-        else
>-            m_jit.addPtr(TrustedImm32(sizeof(IndexingHeader)), storageGPR);
>+        if (MarkedAllocator* allocator = m_jit.vm()->heap.allocatorForAuxiliaryData(size)) {
>+            m_jit.move(TrustedImmPtr(allocator), scratchGPR);
>+            m_jit.emitAllocate(storageGPR, allocator, scratchGPR, scratch2GPR, slowCases);
>+            
>+            m_jit.addPtr(
>+                TrustedImm32(outOfLineCapacity * sizeof(JSValue) + sizeof(IndexingHeader)),
>+                storageGPR);
>+        } else
>+            slowCases.append(m_jit.jump());
>     } else
>         m_jit.move(TrustedImmPtr(0), storageGPR);
> 
>     size_t allocationSize = JSFinalObject::allocationSize(inlineCapacity);
>-    MarkedAllocator* allocatorPtr = &m_jit.vm()->heap.allocatorForObjectWithoutDestructor(allocationSize);
>-    m_jit.move(TrustedImmPtr(allocatorPtr), scratchGPR);
>-    emitAllocateJSObject(resultGPR, scratchGPR, TrustedImmPtr(structure), storageGPR, scratch2GPR, slowCases);
>-
>-    if (hasIndexingHeader)
>-        m_jit.store32(TrustedImm32(vectorLength), MacroAssembler::Address(storageGPR, Butterfly::offsetOfVectorLength()));
>+    MarkedAllocator* allocatorPtr = m_jit.vm()->heap.allocatorForObjectWithoutDestructor(allocationSize);
>+    if (allocatorPtr) {
>+        m_jit.move(TrustedImmPtr(allocatorPtr), scratchGPR);
>+        emitAllocateJSObject(resultGPR, allocatorPtr, scratchGPR, TrustedImmPtr(structure), storageGPR, scratch2GPR, slowCases);
>+        
>+        if (hasIndexingHeader)
>+            m_jit.store32(TrustedImm32(vectorLength), MacroAssembler::Address(storageGPR, Butterfly::offsetOfVectorLength()));
>+    } else
>+        slowCases.append(m_jit.jump());
> 
>     // I want a slow path that also loads out the storage pointer, and that's
>     // what this custom CallArrayAllocatorSlowPathGenerator gives me. It's a lot
>@@ -128,14 +134,20 @@ void SpeculativeJIT::emitAllocateRawObje
>         slowCases, this, operationNewRawObject, resultGPR, storageGPR,
>         structure, vectorLength));
> 
>-    if (hasDouble(structure->indexingType()) && numElements < vectorLength) {
>+    if (numElements < vectorLength) {
> #if USE(JSVALUE64)
>-        m_jit.move(TrustedImm64(bitwise_cast<int64_t>(PNaN)), scratchGPR);
>+        if (hasDouble(structure->indexingType()))
>+            m_jit.move(TrustedImm64(bitwise_cast<int64_t>(PNaN)), scratchGPR);
>+        else
>+            m_jit.move(TrustedImm64(JSValue::encode(JSValue())), scratchGPR);
>         for (unsigned i = numElements; i < vectorLength; ++i)
>             m_jit.store64(scratchGPR, MacroAssembler::Address(storageGPR, sizeof(double) * i));
> #else
>         EncodedValueDescriptor value;
>-        value.asInt64 = JSValue::encode(JSValue(JSValue::EncodeAsDouble, PNaN));
>+        if (hasDouble(structure->indexingType()))
>+            value.asInt64 = JSValue::encode(JSValue(JSValue::EncodeAsDouble, PNaN));
>+        else
>+            value.asInt64 = JSValue::encode(JSValue());
>         for (unsigned i = numElements; i < vectorLength; ++i) {
>             m_jit.store32(TrustedImm32(value.asBits.tag), MacroAssembler::Address(storageGPR, sizeof(double) * i + OBJECT_OFFSETOF(JSValue, u.asBits.tag)));
>             m_jit.store32(TrustedImm32(value.asBits.payload), MacroAssembler::Address(storageGPR, sizeof(double) * i + OBJECT_OFFSETOF(JSValue, u.asBits.payload)));
>@@ -3822,9 +3834,10 @@ void SpeculativeJIT::compileMakeRope(Nod
>     GPRReg scratchGPR = scratch.gpr();
>     
>     JITCompiler::JumpList slowPath;
>-    MarkedAllocator& markedAllocator = m_jit.vm()->heap.allocatorForObjectWithDestructor(sizeof(JSRopeString));
>-    m_jit.move(TrustedImmPtr(&markedAllocator), allocatorGPR);
>-    emitAllocateJSCell(resultGPR, allocatorGPR, TrustedImmPtr(m_jit.vm()->stringStructure.get()), scratchGPR, slowPath);
>+    MarkedAllocator* markedAllocator = m_jit.vm()->heap.allocatorForObjectWithDestructor(sizeof(JSRopeString));
>+    RELEASE_ASSERT(markedAllocator);
>+    m_jit.move(TrustedImmPtr(markedAllocator), allocatorGPR);
>+    emitAllocateJSCell(resultGPR, markedAllocator, allocatorGPR, TrustedImmPtr(m_jit.vm()->stringStructure.get()), scratchGPR, slowPath);
>         
>     m_jit.storePtr(TrustedImmPtr(0), JITCompiler::Address(resultGPR, JSString::offsetOfValue()));
>     for (unsigned i = 0; i < numOpGPRs; ++i)
>@@ -6789,7 +6802,14 @@ void SpeculativeJIT::compileCheckStructu
> 
> void SpeculativeJIT::compileAllocatePropertyStorage(Node* node)
> {
>-    if (node->transition()->previous->couldHaveIndexingHeader()) {
>+    ASSERT(!node->transition()->previous->outOfLineCapacity());
>+    ASSERT(initialOutOfLineCapacity == node->transition()->next->outOfLineCapacity());
>+    
>+    size_t size = initialOutOfLineCapacity * sizeof(JSValue);
>+
>+    MarkedAllocator* allocator = m_jit.vm()->heap.allocatorForAuxiliaryData(size);
>+
>+    if (!allocator || node->transition()->previous->couldHaveIndexingHeader()) {
>         SpeculateCellOperand base(this, node->child1());
>         
>         GPRReg baseGPR = base.gpr();
>@@ -6806,18 +6826,18 @@ void SpeculativeJIT::compileAllocateProp
>     
>     SpeculateCellOperand base(this, node->child1());
>     GPRTemporary scratch1(this);
>+    GPRTemporary scratch2(this);
>+    GPRTemporary scratch3(this);
>         
>     GPRReg baseGPR = base.gpr();
>     GPRReg scratchGPR1 = scratch1.gpr();
>+    GPRReg scratchGPR2 = scratch2.gpr();
>+    GPRReg scratchGPR3 = scratch3.gpr();
>         
>-    ASSERT(!node->transition()->previous->outOfLineCapacity());
>-    ASSERT(initialOutOfLineCapacity == node->transition()->next->outOfLineCapacity());
>-    
>-    JITCompiler::Jump slowPath =
>-        emitAllocateBasicStorage(
>-            TrustedImm32(initialOutOfLineCapacity * sizeof(JSValue)), scratchGPR1);
>-
>-    m_jit.addPtr(JITCompiler::TrustedImm32(sizeof(IndexingHeader)), scratchGPR1);
>+    m_jit.move(JITCompiler::TrustedImmPtr(allocator), scratchGPR2);
>+    JITCompiler::JumpList slowPath;
>+    m_jit.emitAllocate(scratchGPR1, allocator, scratchGPR2, scratchGPR3, slowPath);
>+    m_jit.addPtr(JITCompiler::TrustedImm32(size + sizeof(IndexingHeader)), scratchGPR1);
>         
>     addSlowPathGenerator(
>         slowPathCall(slowPath, this, operationAllocatePropertyStorageWithInitialCapacity, scratchGPR1));
>@@ -6832,8 +6852,10 @@ void SpeculativeJIT::compileReallocatePr
>     size_t oldSize = node->transition()->previous->outOfLineCapacity() * sizeof(JSValue);
>     size_t newSize = oldSize * outOfLineGrowthFactor;
>     ASSERT(newSize == node->transition()->next->outOfLineCapacity() * sizeof(JSValue));
>+    
>+    MarkedAllocator* allocator = m_jit.vm()->heap.allocatorForAuxiliaryData(newSize);
> 
>-    if (node->transition()->previous->couldHaveIndexingHeader()) {
>+    if (!allocator || node->transition()->previous->couldHaveIndexingHeader()) {
>         SpeculateCellOperand base(this, node->child1());
>         
>         GPRReg baseGPR = base.gpr();
>@@ -6852,16 +6874,19 @@ void SpeculativeJIT::compileReallocatePr
>     StorageOperand oldStorage(this, node->child2());
>     GPRTemporary scratch1(this);
>     GPRTemporary scratch2(this);
>+    GPRTemporary scratch3(this);
>         
>     GPRReg baseGPR = base.gpr();
>     GPRReg oldStorageGPR = oldStorage.gpr();
>     GPRReg scratchGPR1 = scratch1.gpr();
>     GPRReg scratchGPR2 = scratch2.gpr();
>-        
>-    JITCompiler::Jump slowPath =
>-        emitAllocateBasicStorage(TrustedImm32(newSize), scratchGPR1);
>-
>-    m_jit.addPtr(JITCompiler::TrustedImm32(sizeof(IndexingHeader)), scratchGPR1);
>+    GPRReg scratchGPR3 = scratch3.gpr();
>+    
>+    JITCompiler::JumpList slowPath;
>+    m_jit.move(JITCompiler::TrustedImmPtr(allocator), scratchGPR2);
>+    m_jit.emitAllocate(scratchGPR1, allocator, scratchGPR2, scratchGPR3, slowPath);
>+    
>+    m_jit.addPtr(JITCompiler::TrustedImm32(newSize + sizeof(IndexingHeader)), scratchGPR1);
>         
>     addSlowPathGenerator(
>         slowPathCall(slowPath, this, operationAllocatePropertyStorage, scratchGPR1, newSize / sizeof(JSValue)));
>Index: Source/JavaScriptCore/dfg/DFGSpeculativeJIT.h
>===================================================================
>--- Source/JavaScriptCore/dfg/DFGSpeculativeJIT.h	(revision 204735)
>+++ Source/JavaScriptCore/dfg/DFGSpeculativeJIT.h	(working copy)
>@@ -2553,18 +2553,21 @@ public:
> 
>     // Allocator for a cell of a specific size.
>     template <typename StructureType> // StructureType can be GPR or ImmPtr.
>-    void emitAllocateJSCell(GPRReg resultGPR, GPRReg allocatorGPR, StructureType structure,
>+    void emitAllocateJSCell(
>+        GPRReg resultGPR, MarkedAllocator* allocator, GPRReg allocatorGPR, StructureType structure,
>         GPRReg scratchGPR, MacroAssembler::JumpList& slowPath)
>     {
>-        m_jit.emitAllocateJSCell(resultGPR, allocatorGPR, structure, scratchGPR, slowPath);
>+        m_jit.emitAllocateJSCell(resultGPR, allocator, allocatorGPR, structure, scratchGPR, slowPath);
>     }
> 
>     // Allocator for an object of a specific size.
>     template <typename StructureType, typename StorageType> // StructureType and StorageType can be GPR or ImmPtr.
>-    void emitAllocateJSObject(GPRReg resultGPR, GPRReg allocatorGPR, StructureType structure,
>+    void emitAllocateJSObject(
>+        GPRReg resultGPR, MarkedAllocator* allocator, GPRReg allocatorGPR, StructureType structure,
>         StorageType storage, GPRReg scratchGPR, MacroAssembler::JumpList& slowPath)
>     {
>-        m_jit.emitAllocateJSObject(resultGPR, allocatorGPR, structure, storage, scratchGPR, slowPath);
>+        m_jit.emitAllocateJSObject(
>+            resultGPR, allocator, allocatorGPR, structure, storage, scratchGPR, slowPath);
>     }
> 
>     template <typename ClassType, typename StructureType, typename StorageType> // StructureType and StorageType can be GPR or ImmPtr.
>Index: Source/JavaScriptCore/dfg/DFGStrengthReductionPhase.cpp
>===================================================================
>--- Source/JavaScriptCore/dfg/DFGStrengthReductionPhase.cpp	(revision 204735)
>+++ Source/JavaScriptCore/dfg/DFGStrengthReductionPhase.cpp	(working copy)
>@@ -40,6 +40,7 @@
> #include "RegExpConstructor.h"
> #include "StringPrototype.h"
> #include <cstdlib>
>+#include <wtf/text/StringBuilder.h>
> 
> namespace JSC { namespace DFG {
> 
>@@ -420,7 +421,7 @@ private:
>                     dataLog("Giving up because of pattern limit.\n");
>                 break;
>             }
>-
>+            
>             unsigned lastIndex;
>             if (regExp->globalOrSticky()) {
>                 // This will only work if we can prove what the value of lastIndex is. To do this
>@@ -514,7 +515,8 @@ private:
>                     }
> 
>                     unsigned publicLength = resultArray.size();
>-                    unsigned vectorLength = std::max(BASE_VECTOR_LEN, publicLength);
>+                    unsigned vectorLength =
>+                        Butterfly::optimalContiguousVectorLength(structure, publicLength);
> 
>                     UniquedStringImpl* indexUID = vm().propertyNames->index.impl();
>                     UniquedStringImpl* inputUID = vm().propertyNames->input.impl();
>Index: Source/JavaScriptCore/ftl/FTLAbstractHeapRepository.h
>===================================================================
>--- Source/JavaScriptCore/ftl/FTLAbstractHeapRepository.h	(revision 204735)
>+++ Source/JavaScriptCore/ftl/FTLAbstractHeapRepository.h	(working copy)
>@@ -76,7 +76,6 @@ namespace JSC { namespace FTL {
>     macro(JSString_value, JSString::offsetOfValue()) \
>     macro(JSSymbolTableObject_symbolTable, JSSymbolTableObject::offsetOfSymbolTable()) \
>     macro(JSWrapperObject_internalValue, JSWrapperObject::internalValueOffset()) \
>-    macro(MarkedAllocator_freeListHead, MarkedAllocator::offsetOfFreeListHead()) \
>     macro(RegExpConstructor_cachedResult_lastRegExp, RegExpConstructor::offsetOfCachedResult() + RegExpCachedResult::offsetOfLastRegExp()) \
>     macro(RegExpConstructor_cachedResult_lastInput, RegExpConstructor::offsetOfCachedResult() + RegExpCachedResult::offsetOfLastInput()) \
>     macro(RegExpConstructor_cachedResult_result_start, RegExpConstructor::offsetOfCachedResult() + RegExpCachedResult::offsetOfResult() + OBJECT_OFFSETOF(MatchResult, start)) \
>@@ -109,8 +108,7 @@ namespace JSC { namespace FTL {
>     macro(JSEnvironmentRecord_variables, JSEnvironmentRecord::offsetOfVariables(), sizeof(EncodedJSValue)) \
>     macro(JSPropertyNameEnumerator_cachedPropertyNamesVectorContents, 0, sizeof(WriteBarrier<JSString>)) \
>     macro(JSRopeString_fibers, JSRopeString::offsetOfFibers(), sizeof(WriteBarrier<JSString>)) \
>-    macro(MarkedSpace_Subspace_impreciseAllocators, OBJECT_OFFSETOF(MarkedSpace::Subspace, impreciseAllocators), sizeof(MarkedAllocator)) \
>-    macro(MarkedSpace_Subspace_preciseAllocators, OBJECT_OFFSETOF(MarkedSpace::Subspace, preciseAllocators), sizeof(MarkedAllocator)) \
>+    macro(MarkedSpace_Subspace_allocatorForSizeStep, OBJECT_OFFSETOF(MarkedSpace::Subspace, allocatorForSizeStep), sizeof(MarkedAllocator*)) \
>     macro(ScopedArguments_overflowStorage, ScopedArguments::overflowStorageOffset(), sizeof(EncodedJSValue)) \
>     macro(WriteBarrierBuffer_bufferContents, 0, sizeof(JSCell*)) \
>     macro(characters8, 0, sizeof(LChar)) \
>Index: Source/JavaScriptCore/ftl/FTLCompile.cpp
>===================================================================
>--- Source/JavaScriptCore/ftl/FTLCompile.cpp	(revision 204735)
>+++ Source/JavaScriptCore/ftl/FTLCompile.cpp	(working copy)
>@@ -42,6 +42,7 @@
> #include "FTLJITCode.h"
> #include "FTLThunks.h"
> #include "JITSubGenerator.h"
>+#include "JSCInlines.h"
> #include "LinkBuffer.h"
> #include "PCToCodeOriginMap.h"
> #include "ScratchRegisterAllocator.h"
>Index: Source/JavaScriptCore/ftl/FTLJITFinalizer.cpp
>===================================================================
>--- Source/JavaScriptCore/ftl/FTLJITFinalizer.cpp	(revision 204735)
>+++ Source/JavaScriptCore/ftl/FTLJITFinalizer.cpp	(working copy)
>@@ -32,6 +32,7 @@
> #include "DFGPlan.h"
> #include "FTLState.h"
> #include "FTLThunks.h"
>+#include "JSCInlines.h"
> #include "ProfilerDatabase.h"
> 
> namespace JSC { namespace FTL {
>Index: Source/JavaScriptCore/ftl/FTLLowerDFGToB3.cpp
>===================================================================
>--- Source/JavaScriptCore/ftl/FTLLowerDFGToB3.cpp	(revision 204735)
>+++ Source/JavaScriptCore/ftl/FTLLowerDFGToB3.cpp	(working copy)
>@@ -3779,7 +3779,7 @@ private:
>                 size, m_out.constInt32(DirectArguments::allocationSize(minCapacity)));
>             
>             fastObject = allocateVariableSizedObject<DirectArguments>(
>-                size, structure, m_out.intPtrZero, slowPath);
>+                m_out.zeroExtPtr(size), structure, m_out.intPtrZero, slowPath);
>         }
>         
>         m_out.store32(length.value, fastObject, m_heaps.DirectArguments_length);
>@@ -3880,7 +3880,7 @@ private:
>             LValue arrayLength = lowInt32(m_node->child1());
>             LBasicBlock loopStart = m_out.newBlock();
>             bool shouldLargeArraySizeCreateArrayStorage = false;
>-            LValue array = compileAllocateArrayWithSize(arrayLength, ArrayWithContiguous, shouldLargeArraySizeCreateArrayStorage);
>+            LValue array = allocateArrayWithSize(arrayLength, ArrayWithContiguous, shouldLargeArraySizeCreateArrayStorage);
> 
>             LValue butterfly = m_out.loadPtr(array, m_heaps.JSObject_butterfly);
>             ValueFromBlock startLength = m_out.anchor(arrayLength);
>@@ -4053,7 +4053,7 @@ private:
>             m_out.constIntPtr(m_node->numConstants())));
>     }
> 
>-    LValue compileAllocateArrayWithSize(LValue publicLength, IndexingType indexingType, bool shouldLargeArraySizeCreateArrayStorage = true)
>+    LValue allocateArrayWithSize(LValue publicLength, IndexingType indexingType, bool shouldLargeArraySizeCreateArrayStorage = true)
>     {
>         JSGlobalObject* globalObject = m_graph.globalObjectFor(m_node->origin.semantic);
>         Structure* structure = globalObject->arrayStructureForIndexingTypeDuringAllocation(indexingType);
>@@ -4064,33 +4064,40 @@ private:
>             || hasContiguous(structure->indexingType()));
> 
>         LBasicBlock fastCase = m_out.newBlock();
>-        LBasicBlock largeCase = shouldLargeArraySizeCreateArrayStorage ? m_out.newBlock() : nullptr;
>+        LBasicBlock largeCase = m_out.newBlock();
>         LBasicBlock failCase = m_out.newBlock();
>         LBasicBlock continuation = m_out.newBlock();
>-        LBasicBlock lastNext = nullptr;
>-        if (shouldLargeArraySizeCreateArrayStorage) {
>-            m_out.branch(
>-                m_out.aboveOrEqual(publicLength, m_out.constInt32(MIN_ARRAY_STORAGE_CONSTRUCTION_LENGTH)),
>-                rarely(largeCase), usually(fastCase));
>-            lastNext = m_out.appendTo(fastCase, largeCase);
>-        }
>-
>+        LBasicBlock slowCase = m_out.newBlock();
>+        
>+        LBasicBlock lastNext = m_out.insertNewBlocksBefore(fastCase);
>+        
>+        LValue predicate;
>+        if (shouldLargeArraySizeCreateArrayStorage)
>+            predicate = m_out.aboveOrEqual(publicLength, m_out.constInt32(MIN_ARRAY_STORAGE_CONSTRUCTION_LENGTH));
>+        else
>+            predicate = m_out.booleanFalse;
>         
>+        m_out.branch(predicate, rarely(largeCase), usually(fastCase));
>+        
>+        m_out.appendTo(fastCase, largeCase);
>+
>         // We don't round up to BASE_VECTOR_LEN for new Array(blah).
>         LValue vectorLength = publicLength;
>-        
>+            
>         LValue payloadSize =
>             m_out.shl(m_out.zeroExt(vectorLength, pointerType()), m_out.constIntPtr(3));
>-        
>+            
>         LValue butterflySize = m_out.add(
>             payloadSize, m_out.constIntPtr(sizeof(IndexingHeader)));
>-        
>-        LValue endOfStorage = allocateBasicStorageAndGetEnd(butterflySize, failCase);
>-        
>-        LValue butterfly = m_out.sub(endOfStorage, payloadSize);
>-        
>+            
>+        LValue allocator = allocatorForSize(
>+            vm().heap.subspaceForAuxiliaryData(), butterflySize, failCase);
>+        LValue startOfStorage = allocateHeapCell(allocator, failCase);
>+            
>+        LValue butterfly = m_out.add(startOfStorage, m_out.constIntPtr(sizeof(IndexingHeader)));
>+            
>         LValue object = allocateObject<JSArray>(structure, butterfly, failCase);
>-        
>+            
>         m_out.store32(publicLength, butterfly, m_heaps.Butterfly_publicLength);
>         m_out.store32(vectorLength, butterfly, m_heaps.Butterfly_vectorLength);
> 
>@@ -4100,26 +4107,20 @@ private:
>         m_out.jump(continuation);
>         
>         LValue structureValue;
>-        if (shouldLargeArraySizeCreateArrayStorage) {
>-            LBasicBlock slowCase = m_out.newBlock();
>-
>-            m_out.appendTo(largeCase, failCase);
>-            ValueFromBlock largeStructure = m_out.anchor(m_out.constIntPtr(
>+        
>+        m_out.appendTo(largeCase, failCase);
>+        ValueFromBlock largeStructure = m_out.anchor(
>+            m_out.constIntPtr(
>                 globalObject->arrayStructureForIndexingTypeDuringAllocation(ArrayWithArrayStorage)));
>-            m_out.jump(slowCase);
>-
>-            m_out.appendTo(failCase, slowCase);
>-            ValueFromBlock failStructure = m_out.anchor(m_out.constIntPtr(structure));
>-            m_out.jump(slowCase);
>-
>-            m_out.appendTo(slowCase, continuation);
>-            structureValue = m_out.phi(
>-                pointerType(), largeStructure, failStructure);
>-        } else {
>-            ASSERT(!lastNext);
>-            lastNext = m_out.appendTo(failCase, continuation);
>-            structureValue = m_out.constIntPtr(structure);
>-        }
>+        m_out.jump(slowCase);
>+        
>+        m_out.appendTo(failCase, slowCase);
>+        ValueFromBlock failStructure = m_out.anchor(m_out.constIntPtr(structure));
>+        m_out.jump(slowCase);
>+        
>+        m_out.appendTo(slowCase, continuation);
>+        structureValue = m_out.phi(
>+            pointerType(), largeStructure, failStructure);
> 
>         LValue slowResultValue = lazySlowPath(
>             [=] (const Vector<Location>& locations) -> RefPtr<LazySlowPath::Generator> {
>@@ -4144,7 +4145,7 @@ private:
>             m_node->indexingType());
>         
>         if (!globalObject->isHavingABadTime() && !hasAnyArrayStorage(m_node->indexingType())) {
>-            setJSValue(compileAllocateArrayWithSize(publicLength, m_node->indexingType()));
>+            setJSValue(allocateArrayWithSize(publicLength, m_node->indexingType()));
>             return;
>         }
>         
>@@ -4413,13 +4414,12 @@ private:
>         
>         LBasicBlock lastNext = m_out.insertNewBlocksBefore(slowPath);
>         
>-        MarkedAllocator& allocator =
>+        MarkedAllocator* allocator =
>             vm().heap.allocatorForObjectWithDestructor(sizeof(JSRopeString));
>+        DFG_ASSERT(m_graph, m_node, allocator);
>         
>         LValue result = allocateCell(
>-            m_out.constIntPtr(&allocator),
>-            vm().stringStructure.get(),
>-            slowPath);
>+            m_out.constIntPtr(allocator), vm().stringStructure.get(), slowPath);
>         
>         m_out.storePtr(m_out.intPtrZero, result, m_heaps.JSString_value);
>         for (unsigned i = 0; i < numKids; ++i)
>@@ -6926,7 +6926,8 @@ private:
>             
>             if (structure->outOfLineCapacity() || hasIndexedProperties(structure->indexingType())) {
>                 size_t allocationSize = JSFinalObject::allocationSize(structure->inlineCapacity());
>-                MarkedAllocator* allocator = &vm().heap.allocatorForObjectWithoutDestructor(allocationSize);
>+                MarkedAllocator* cellAllocator = vm().heap.allocatorForObjectWithoutDestructor(allocationSize);
>+                DFG_ASSERT(m_graph, m_node, cellAllocator);
> 
>                 bool hasIndexingHeader = hasIndexedProperties(structure->indexingType());
>                 unsigned indexingHeaderSize = 0;
>@@ -6951,7 +6952,7 @@ private:
>                     indexingPayloadSizeInBytes =
>                         m_out.mul(m_out.zeroExtPtr(vectorLength), m_out.intPtrEight);
>                 }
>-
>+                
>                 LValue butterflySize = m_out.add(
>                     m_out.constIntPtr(
>                         structure->outOfLineCapacity() * sizeof(JSValue) + indexingHeaderSize),
>@@ -6962,16 +6963,19 @@ private:
>                 
>                 LBasicBlock lastNext = m_out.insertNewBlocksBefore(slowPath);
>                 
>-                LValue endOfStorage = allocateBasicStorageAndGetEnd(butterflySize, slowPath);
>+                LValue startOfStorage = allocateHeapCell(
>+                    allocatorForSize(vm().heap.subspaceForAuxiliaryData(), butterflySize, slowPath),
>+                    slowPath);
> 
>                 LValue fastButterflyValue = m_out.add(
>-                    m_out.sub(endOfStorage, indexingPayloadSizeInBytes),
>-                    m_out.constIntPtr(sizeof(IndexingHeader) - indexingHeaderSize));
>+                    startOfStorage,
>+                    m_out.constIntPtr(
>+                        structure->outOfLineCapacity() * sizeof(JSValue) + sizeof(IndexingHeader)));
> 
>                 m_out.store32(vectorLength, fastButterflyValue, m_heaps.Butterfly_vectorLength);
>                 
>                 LValue fastObjectValue = allocateObject(
>-                    m_out.constIntPtr(allocator), structure, fastButterflyValue, slowPath);
>+                    m_out.constIntPtr(cellAllocator), structure, fastButterflyValue, slowPath);
> 
>                 ValueFromBlock fastObject = m_out.anchor(fastObjectValue);
>                 ValueFromBlock fastButterfly = m_out.anchor(fastButterflyValue);
>@@ -7760,10 +7764,11 @@ private:
> 
>     void initializeArrayElements(IndexingType indexingType, LValue vectorLength, LValue butterfly)
>     {
>-        if (!hasDouble(indexingType)) {
>-            // The GC already initialized everything to JSValue() for us.
>-            return;
>-        }
>+        LValue hole;
>+        if (hasDouble(indexingType))
>+            hole = m_out.constInt64(bitwise_cast<int64_t>(PNaN));
>+        else
>+            hole = m_out.constInt64(JSValue::encode(JSValue()));
> 
>         // Doubles must be initialized to PNaN.
>         LBasicBlock initLoop = m_out.newBlock();
>@@ -7778,9 +7783,7 @@ private:
>         LValue index = m_out.phi(Int32, originalIndex);
>         LValue pointer = m_out.phi(pointerType(), originalPointer);
>         
>-        m_out.store64(
>-            m_out.constInt64(bitwise_cast<int64_t>(PNaN)),
>-            TypedPointer(m_heaps.indexedDoubleProperties.atAnyIndex(), pointer));
>+        m_out.store64(hole, TypedPointer(m_heaps.indexedDoubleProperties.atAnyIndex(), pointer));
>         
>         LValue nextIndex = m_out.sub(index, m_out.int32One);
>         m_out.addIncomingToPhi(index, m_out.anchor(nextIndex));
>@@ -7841,13 +7844,12 @@ private:
>         LBasicBlock continuation = m_out.newBlock();
>         
>         LBasicBlock lastNext = m_out.insertNewBlocksBefore(slowPath);
>-        
>-        LValue endOfStorage = allocateBasicStorageAndGetEnd(
>-            m_out.constIntPtr(sizeInValues * sizeof(JSValue)), slowPath);
>-        
>+
>+        size_t sizeInBytes = sizeInValues * sizeof(JSValue);
>+        MarkedAllocator* allocator = vm().heap.allocatorForAuxiliaryData(sizeInBytes);
>+        LValue startOfStorage = allocateHeapCell(m_out.constIntPtr(allocator), slowPath);
>         ValueFromBlock fastButterfly = m_out.anchor(
>-            m_out.add(m_out.constIntPtr(sizeof(IndexingHeader)), endOfStorage));
>-        
>+            m_out.add(m_out.constIntPtr(sizeInBytes + sizeof(IndexingHeader)), startOfStorage));
>         m_out.jump(continuation);
>         
>         m_out.appendTo(slowPath, continuation);
>@@ -8416,29 +8418,64 @@ private:
>         setJSValue(patchpoint);
>     }
> 
>-    LValue allocateCell(LValue allocator, LBasicBlock slowPath)
>+    LValue allocateHeapCell(LValue allocator, LBasicBlock slowPath)
>     {
>-        LBasicBlock success = m_out.newBlock();
>-    
>-        LValue result;
>-        LValue condition;
>-        if (Options::forceGCSlowPaths()) {
>-            result = m_out.intPtrZero;
>-            condition = m_out.booleanFalse;
>-        } else {
>-            result = m_out.loadPtr(
>-                allocator, m_heaps.MarkedAllocator_freeListHead);
>-            condition = m_out.notNull(result);
>+        MarkedAllocator* actualAllocator = nullptr;
>+        if (allocator->hasIntPtr())
>+            actualAllocator = bitwise_cast<MarkedAllocator*>(allocator->asIntPtr());
>+        
>+        if (!actualAllocator) {
>+            // This means that either we know that the allocator is null or we don't know what the
>+            // allocator is. In either case, we need the null check.
>+            LBasicBlock haveAllocator = m_out.newBlock();
>+            LBasicBlock lastNext = m_out.insertNewBlocksBefore(haveAllocator);
>+            m_out.branch(allocator, usually(haveAllocator), rarely(slowPath));
>+            m_out.appendTo(haveAllocator, lastNext);
>         }
>-        m_out.branch(condition, usually(success), rarely(slowPath));
>         
>-        m_out.appendTo(success);
>+        LBasicBlock continuation = m_out.newBlock();
>         
>-        m_out.storePtr(
>-            m_out.loadPtr(result, m_heaps.JSCell_freeListNext),
>-            allocator, m_heaps.MarkedAllocator_freeListHead);
>-
>-        return result;
>+        LBasicBlock lastNext = m_out.insertNewBlocksBefore(continuation);
>+        
>+        PatchpointValue* patchpoint = m_out.patchpoint(pointerType());
>+        patchpoint->effects.terminal = true;
>+        patchpoint->appendSomeRegister(allocator);
>+        patchpoint->numGPScratchRegisters++;
>+        patchpoint->resultConstraint = ValueRep::SomeEarlyRegister;
>+        
>+        m_out.appendSuccessor(usually(continuation));
>+        m_out.appendSuccessor(rarely(slowPath));
>+        
>+        patchpoint->setGenerator(
>+            [=] (CCallHelpers& jit, const StackmapGenerationParams& params) {
>+                CCallHelpers::JumpList jumpToSlowPath;
>+                
>+                // We use a patchpoint to emit the allocation path because whenever we mess with
>+                // allocation paths, we already reason about them at the machine code level. We know
>+                // exactly what instruction sequence we want. We're confident that no compiler
>+                // optimization could make this code better. So, it's best to have the code in
>+                // AssemblyHelpers::emitAllocate(). That way, the same optimized path is shared by
>+                // all of the compiler tiers.
>+                jit.emitAllocateWithNonNullAllocator(
>+                    params[0].gpr(), actualAllocator, params[1].gpr(), params.gpScratch(0),
>+                    jumpToSlowPath);
>+                
>+                CCallHelpers::Jump jumpToSuccess;
>+                if (!params.fallsThroughToSuccessor(0))
>+                    jumpToSuccess = jit.jump();
>+                
>+                Vector<Box<CCallHelpers::Label>> labels = params.successorLabels();
>+                
>+                params.addLatePath(
>+                    [=] (CCallHelpers& jit) {
>+                        jumpToSlowPath.linkTo(*labels[1], &jit);
>+                        if (jumpToSuccess.isSet())
>+                            jumpToSuccess.linkTo(*labels[0], &jit);
>+                    });
>+            });
>+        
>+        m_out.appendTo(continuation, lastNext);
>+        return patchpoint;
>     }
>     
>     void storeStructure(LValue object, Structure* structure)
>@@ -8451,7 +8488,7 @@ private:
> 
>     LValue allocateCell(LValue allocator, Structure* structure, LBasicBlock slowPath)
>     {
>-        LValue result = allocateCell(allocator, slowPath);
>+        LValue result = allocateHeapCell(allocator, slowPath);
>         storeStructure(result, structure);
>         return result;
>     }
>@@ -8468,7 +8505,7 @@ private:
>     LValue allocateObject(
>         size_t size, Structure* structure, LValue butterfly, LBasicBlock slowPath)
>     {
>-        MarkedAllocator* allocator = &vm().heap.allocatorForObjectOfType<ClassType>(size);
>+        MarkedAllocator* allocator = vm().heap.allocatorForObjectOfType<ClassType>(size);
>         return allocateObject(m_out.constIntPtr(allocator), structure, butterfly, slowPath);
>     }
>     
>@@ -8479,46 +8516,60 @@ private:
>             ClassType::allocationSize(0), structure, butterfly, slowPath);
>     }
>     
>-    template<typename ClassType>
>-    LValue allocateVariableSizedObject(
>-        LValue size, Structure* structure, LValue butterfly, LBasicBlock slowPath)
>+    LValue allocatorForSize(LValue subspace, LValue size, LBasicBlock slowPath)
>     {
>-        static_assert(!(MarkedSpace::preciseStep & (MarkedSpace::preciseStep - 1)), "MarkedSpace::preciseStep must be a power of two.");
>-        static_assert(!(MarkedSpace::impreciseStep & (MarkedSpace::impreciseStep - 1)), "MarkedSpace::impreciseStep must be a power of two.");
>-
>-        LValue subspace = m_out.constIntPtr(&vm().heap.subspaceForObjectOfType<ClassType>());
>+        static_assert(!(MarkedSpace::sizeStep & (MarkedSpace::sizeStep - 1)), "MarkedSpace::sizeStep must be a power of two.");
>+        
>+        // Try to do some constant-folding here.
>+        if (subspace->hasIntPtr() && size->hasIntPtr()) {
>+            MarkedSpace::Subspace* actualSubspace = bitwise_cast<MarkedSpace::Subspace*>(subspace->asIntPtr());
>+            size_t actualSize = size->asIntPtr();
>+            
>+            MarkedAllocator* actualAllocator = MarkedSpace::allocatorFor(*actualSubspace, actualSize);
>+            if (!actualAllocator) {
>+                LBasicBlock continuation = m_out.newBlock();
>+                LBasicBlock lastNext = m_out.insertNewBlocksBefore(continuation);
>+                m_out.jump(slowPath);
>+                m_out.appendTo(continuation, lastNext);
>+                return m_out.intPtrZero;
>+            }
>+            
>+            return m_out.constIntPtr(actualAllocator);
>+        }
>+        
>+        unsigned stepShift = getLSBSet(MarkedSpace::sizeStep);
>         
>-        LBasicBlock smallCaseBlock = m_out.newBlock();
>-        LBasicBlock largeOrOversizeCaseBlock = m_out.newBlock();
>-        LBasicBlock largeCaseBlock = m_out.newBlock();
>         LBasicBlock continuation = m_out.newBlock();
>         
>-        LValue uproundedSize = m_out.add(size, m_out.constInt32(MarkedSpace::preciseStep - 1));
>-        LValue isSmall = m_out.below(uproundedSize, m_out.constInt32(MarkedSpace::preciseCutoff));
>-        m_out.branch(isSmall, unsure(smallCaseBlock), unsure(largeOrOversizeCaseBlock));
>+        LBasicBlock lastNext = m_out.insertNewBlocksBefore(continuation);
>         
>-        LBasicBlock lastNext = m_out.appendTo(smallCaseBlock, largeOrOversizeCaseBlock);
>-        TypedPointer address = m_out.baseIndex(
>-            m_heaps.MarkedSpace_Subspace_preciseAllocators, subspace,
>-            m_out.zeroExtPtr(m_out.lShr(uproundedSize, m_out.constInt32(getLSBSet(MarkedSpace::preciseStep)))));
>-        ValueFromBlock smallAllocator = m_out.anchor(address.value());
>-        m_out.jump(continuation);
>+        LValue sizeClassIndex = m_out.lShr(
>+            m_out.add(size, m_out.constIntPtr(MarkedSpace::sizeStep - 1)),
>+            m_out.constInt32(stepShift));
>         
>-        m_out.appendTo(largeOrOversizeCaseBlock, largeCaseBlock);
>         m_out.branch(
>-            m_out.below(uproundedSize, m_out.constInt32(MarkedSpace::impreciseCutoff)),
>-            usually(largeCaseBlock), rarely(slowPath));
>-        
>-        m_out.appendTo(largeCaseBlock, continuation);
>-        address = m_out.baseIndex(
>-            m_heaps.MarkedSpace_Subspace_impreciseAllocators, subspace,
>-            m_out.zeroExtPtr(m_out.lShr(uproundedSize, m_out.constInt32(getLSBSet(MarkedSpace::impreciseStep)))));
>-        ValueFromBlock largeAllocator = m_out.anchor(address.value());
>-        m_out.jump(continuation);
>+            m_out.above(sizeClassIndex, m_out.constIntPtr(MarkedSpace::largeCutoff >> stepShift)),
>+            rarely(slowPath), usually(continuation));
>         
>         m_out.appendTo(continuation, lastNext);
>-        LValue allocator = m_out.phi(pointerType(), smallAllocator, largeAllocator);
>         
>+        return m_out.loadPtr(
>+            m_out.baseIndex(
>+                m_heaps.MarkedSpace_Subspace_allocatorForSizeStep,
>+                subspace, m_out.sub(sizeClassIndex, m_out.intPtrOne)));
>+    }
>+    
>+    LValue allocatorForSize(MarkedSpace::Subspace& subspace, LValue size, LBasicBlock slowPath)
>+    {
>+        return allocatorForSize(m_out.constIntPtr(&subspace), size, slowPath);
>+    }
>+    
>+    template<typename ClassType>
>+    LValue allocateVariableSizedObject(
>+        LValue size, Structure* structure, LValue butterfly, LBasicBlock slowPath)
>+    {
>+        LValue allocator = allocatorForSize(
>+            vm().heap.subspaceForObjectOfType<ClassType>(), size, slowPath);
>         return allocateObject(allocator, structure, butterfly, slowPath);
>     }
>     
>@@ -8551,7 +8602,8 @@ private:
>     LValue allocateObject(Structure* structure)
>     {
>         size_t allocationSize = JSFinalObject::allocationSize(structure->inlineCapacity());
>-        MarkedAllocator* allocator = &vm().heap.allocatorForObjectWithoutDestructor(allocationSize);
>+        MarkedAllocator* allocator = vm().heap.allocatorForObjectWithoutDestructor(allocationSize);
>+        DFG_ASSERT(m_graph, m_node, allocator);
>         
>         LBasicBlock slowPath = m_out.newBlock();
>         LBasicBlock continuation = m_out.newBlock();
>@@ -8597,34 +8649,34 @@ private:
>     ArrayValues allocateJSArray(
>         Structure* structure, unsigned numElements, LBasicBlock slowPath)
>     {
>-        ASSERT(
>+        DFG_ASSERT(
>+            m_graph, m_node,
>             hasUndecided(structure->indexingType())
>             || hasInt32(structure->indexingType())
>             || hasDouble(structure->indexingType())
>             || hasContiguous(structure->indexingType()));
>+        DFG_ASSERT(m_graph, m_node, !structure->outOfLineCapacity());
>         
>-        unsigned vectorLength = std::max(BASE_VECTOR_LEN, numElements);
>+        unsigned vectorLength = Butterfly::optimalContiguousVectorLength(0lu, numElements);
>         
>-        LValue endOfStorage = allocateBasicStorageAndGetEnd(
>-            m_out.constIntPtr(sizeof(JSValue) * vectorLength + sizeof(IndexingHeader)),
>-            slowPath);
>+        MarkedAllocator* allocator = vm().heap.allocatorForAuxiliaryData(
>+            sizeof(JSValue) * vectorLength + sizeof(IndexingHeader));
>+        LValue startOfStorage = allocateHeapCell(m_out.constIntPtr(allocator), slowPath);
>         
>-        LValue butterfly = m_out.sub(
>-            endOfStorage, m_out.constIntPtr(sizeof(JSValue) * vectorLength));
>+        LValue butterfly = m_out.add(startOfStorage, m_out.constIntPtr(sizeof(IndexingHeader)));
>         
>-        LValue object = allocateObject<JSArray>(
>-            structure, butterfly, slowPath);
>+        LValue object = allocateObject<JSArray>(structure, butterfly, slowPath);
>         
>         m_out.store32(m_out.constInt32(numElements), butterfly, m_heaps.Butterfly_publicLength);
>         m_out.store32(m_out.constInt32(vectorLength), butterfly, m_heaps.Butterfly_vectorLength);
>-        
>-        if (hasDouble(structure->indexingType())) {
>-            for (unsigned i = numElements; i < vectorLength; ++i) {
>-                m_out.store64(
>-                    m_out.constInt64(bitwise_cast<int64_t>(PNaN)),
>-                    butterfly, m_heaps.indexedDoubleProperties[i]);
>-            }
>-        }
>+
>+        LValue hole;
>+        if (hasDouble(structure->indexingType()))
>+            hole = m_out.constInt64(bitwise_cast<int64_t>(PNaN));
>+        else
>+            hole = m_out.constInt64(JSValue::encode(JSValue()));
>+        for (unsigned i = numElements; i < vectorLength; ++i)
>+            m_out.store64(hole, butterfly, m_heaps.indexedDoubleProperties[i]);
>         
>         return ArrayValues(object, butterfly);
>     }
>Index: Source/JavaScriptCore/ftl/FTLOutput.cpp
>===================================================================
>--- Source/JavaScriptCore/ftl/FTLOutput.cpp	(revision 204735)
>+++ Source/JavaScriptCore/ftl/FTLOutput.cpp	(working copy)
>@@ -100,7 +100,9 @@ SlotBaseValue* Output::lockedStackSlot(s
> 
> LValue Output::constBool(bool value)
> {
>-    return m_block->appendNew<B3::Const32Value>(m_proc, origin(), value);
>+    if (value)
>+        return booleanTrue;
>+    return booleanFalse;
> }
> 
> LValue Output::constInt32(int32_t value)
>@@ -125,6 +127,10 @@ LValue Output::phi(LType type)
> 
> LValue Output::add(LValue left, LValue right)
> {
>+    if (Value* result = left->addConstant(m_proc, right)) {
>+        m_block->append(result);
>+        return result;
>+    }
>     return m_block->appendNew<B3::Value>(m_proc, B3::Add, origin(), left, right);
> }
> 
>@@ -205,16 +211,28 @@ LValue Output::bitXor(LValue left, LValu
> 
> LValue Output::shl(LValue left, LValue right)
> {
>+    if (Value* result = left->shlConstant(m_proc, right)) {
>+        m_block->append(result);
>+        return result;
>+    }
>     return m_block->appendNew<B3::Value>(m_proc, B3::Shl, origin(), left, castToInt32(right));
> }
> 
> LValue Output::aShr(LValue left, LValue right)
> {
>+    if (Value* result = left->sShrConstant(m_proc, right)) {
>+        m_block->append(result);
>+        return result;
>+    }
>     return m_block->appendNew<B3::Value>(m_proc, B3::SShr, origin(), left, castToInt32(right));
> }
> 
> LValue Output::lShr(LValue left, LValue right)
> {
>+    if (Value* result = left->zShrConstant(m_proc, right)) {
>+        m_block->append(result);
>+        return result;
>+    }
>     return m_block->appendNew<B3::Value>(m_proc, B3::ZShr, origin(), left, castToInt32(right));
> }
> 
>@@ -343,6 +361,8 @@ LValue Output::zeroExt(LValue value, LTy
> {
>     if (value->type() == type)
>         return value;
>+    if (value->hasInt32())
>+        return m_block->appendIntConstant(m_proc, origin(), Int64, static_cast<uint64_t>(static_cast<uint32_t>(value->asInt32())));
>     return m_block->appendNew<B3::Value>(m_proc, B3::ZExt32, origin(), value);
> }
> 
>@@ -453,51 +473,81 @@ LValue Output::baseIndex(LValue base, LV
> 
> LValue Output::equal(LValue left, LValue right)
> {
>+    TriState result = left->equalConstant(right);
>+    if (result != MixedTriState)
>+        return constBool(result == TrueTriState);
>     return m_block->appendNew<B3::Value>(m_proc, B3::Equal, origin(), left, right);
> }
> 
> LValue Output::notEqual(LValue left, LValue right)
> {
>+    TriState result = left->notEqualConstant(right);
>+    if (result != MixedTriState)
>+        return constBool(result == TrueTriState);
>     return m_block->appendNew<B3::Value>(m_proc, B3::NotEqual, origin(), left, right);
> }
> 
> LValue Output::above(LValue left, LValue right)
> {
>+    TriState result = left->aboveConstant(right);
>+    if (result != MixedTriState)
>+        return constBool(result == TrueTriState);
>     return m_block->appendNew<B3::Value>(m_proc, B3::Above, origin(), left, right);
> }
> 
> LValue Output::aboveOrEqual(LValue left, LValue right)
> {
>+    TriState result = left->aboveEqualConstant(right);
>+    if (result != MixedTriState)
>+        return constBool(result == TrueTriState);
>     return m_block->appendNew<B3::Value>(m_proc, B3::AboveEqual, origin(), left, right);
> }
> 
> LValue Output::below(LValue left, LValue right)
> {
>+    TriState result = left->belowConstant(right);
>+    if (result != MixedTriState)
>+        return constBool(result == TrueTriState);
>     return m_block->appendNew<B3::Value>(m_proc, B3::Below, origin(), left, right);
> }
> 
> LValue Output::belowOrEqual(LValue left, LValue right)
> {
>+    TriState result = left->belowEqualConstant(right);
>+    if (result != MixedTriState)
>+        return constBool(result == TrueTriState);
>     return m_block->appendNew<B3::Value>(m_proc, B3::BelowEqual, origin(), left, right);
> }
> 
> LValue Output::greaterThan(LValue left, LValue right)
> {
>+    TriState result = left->greaterThanConstant(right);
>+    if (result != MixedTriState)
>+        return constBool(result == TrueTriState);
>     return m_block->appendNew<B3::Value>(m_proc, B3::GreaterThan, origin(), left, right);
> }
> 
> LValue Output::greaterThanOrEqual(LValue left, LValue right)
> {
>+    TriState result = left->greaterEqualConstant(right);
>+    if (result != MixedTriState)
>+        return constBool(result == TrueTriState);
>     return m_block->appendNew<B3::Value>(m_proc, B3::GreaterEqual, origin(), left, right);
> }
> 
> LValue Output::lessThan(LValue left, LValue right)
> {
>+    TriState result = left->lessThanConstant(right);
>+    if (result != MixedTriState)
>+        return constBool(result == TrueTriState);
>     return m_block->appendNew<B3::Value>(m_proc, B3::LessThan, origin(), left, right);
> }
> 
> LValue Output::lessThanOrEqual(LValue left, LValue right)
> {
>+    TriState result = left->lessEqualConstant(right);
>+    if (result != MixedTriState)
>+        return constBool(result == TrueTriState);
>     return m_block->appendNew<B3::Value>(m_proc, B3::LessEqual, origin(), left, right);
> }
> 
>@@ -583,6 +633,12 @@ LValue Output::notZero64(LValue value)
> 
> LValue Output::select(LValue value, LValue taken, LValue notTaken)
> {
>+    if (value->hasInt32()) {
>+        if (value->asInt32())
>+            return taken;
>+        else
>+            return notTaken;
>+    }
>     return m_block->appendNew<B3::Value>(m_proc, B3::Select, origin(), value, taken, notTaken);
> }
> 
>@@ -621,6 +677,11 @@ void Output::unreachable()
>     m_block->appendNewControlValue(m_proc, B3::Oops, origin());
> }
> 
>+void Output::appendSuccessor(WeightedTarget target)
>+{
>+    m_block->appendSuccessor(target.frequentedBlock());
>+}
>+
> CheckValue* Output::speculate(LValue value)
> {
>     return m_block->appendNew<B3::CheckValue>(m_proc, B3::Check, origin(), value);
>@@ -741,7 +802,8 @@ void Output::decrementSuperSamplerCount(
> 
> void Output::addIncomingToPhi(LValue phi, ValueFromBlock value)
> {
>-    value.value()->as<B3::UpsilonValue>()->setPhi(phi);
>+    if (value)
>+        value.value()->as<B3::UpsilonValue>()->setPhi(phi);
> }
> 
> } } // namespace JSC::FTL
>Index: Source/JavaScriptCore/ftl/FTLOutput.h
>===================================================================
>--- Source/JavaScriptCore/ftl/FTLOutput.h	(revision 204735)
>+++ Source/JavaScriptCore/ftl/FTLOutput.h	(working copy)
>@@ -398,6 +398,8 @@ public:
>     void ret(LValue);
> 
>     void unreachable();
>+    
>+    void appendSuccessor(WeightedTarget);
> 
>     B3::CheckValue* speculate(LValue);
>     B3::CheckValue* speculateAdd(LValue, LValue);
>Index: Source/JavaScriptCore/ftl/FTLValueFromBlock.h
>===================================================================
>--- Source/JavaScriptCore/ftl/FTLValueFromBlock.h	(revision 204735)
>+++ Source/JavaScriptCore/ftl/FTLValueFromBlock.h	(working copy)
>@@ -45,6 +45,8 @@ public:
>         , m_block(block)
>     {
>     }
>+    
>+    explicit operator bool() const { return m_value || m_block; }
> 
>     LValue value() const { return m_value; }
>     LBasicBlock block() const { return m_block; }
>Index: Source/JavaScriptCore/ftl/FTLWeightedTarget.h
>===================================================================
>--- Source/JavaScriptCore/ftl/FTLWeightedTarget.h	(revision 204735)
>+++ Source/JavaScriptCore/ftl/FTLWeightedTarget.h	(working copy)
>@@ -1,5 +1,5 @@
> /*
>- * Copyright (C) 2014 Apple Inc. All rights reserved.
>+ * Copyright (C) 2014, 2016 Apple Inc. All rights reserved.
>  *
>  * Redistribution and use in source and binary forms, with or without
>  * modification, are permitted provided that the following conditions
>@@ -55,6 +55,11 @@ public:
>     LBasicBlock target() const { return m_target; }
>     Weight weight() const { return m_weight; }
>     
>+    B3::FrequentedBlock frequentedBlock() const
>+    {
>+        return B3::FrequentedBlock(target(), weight().frequencyClass());
>+    }
>+    
> private:
>     LBasicBlock m_target;
>     Weight m_weight;
>Index: Source/JavaScriptCore/heap/CellContainer.h
>===================================================================
>--- Source/JavaScriptCore/heap/CellContainer.h	(nonexistent)
>+++ Source/JavaScriptCore/heap/CellContainer.h	(working copy)
>@@ -0,0 +1,90 @@
>+/*
>+ * Copyright (C) 2016 Apple Inc. All rights reserved.
>+ *
>+ * Redistribution and use in source and binary forms, with or without
>+ * modification, are permitted provided that the following conditions
>+ * are met:
>+ * 1. Redistributions of source code must retain the above copyright
>+ *    notice, this list of conditions and the following disclaimer.
>+ * 2. Redistributions in binary form must reproduce the above copyright
>+ *    notice, this list of conditions and the following disclaimer in the
>+ *    documentation and/or other materials provided with the distribution.
>+ *
>+ * THIS SOFTWARE IS PROVIDED BY APPLE INC. ``AS IS'' AND ANY
>+ * EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
>+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
>+ * PURPOSE ARE DISCLAIMED.  IN NO EVENT SHALL APPLE INC. OR
>+ * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
>+ * EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
>+ * PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
>+ * PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY
>+ * OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
>+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
>+ * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 
>+ */
>+
>+#pragma once
>+
>+#include <wtf/StdLibExtras.h>
>+
>+namespace JSC {
>+
>+class HeapCell;
>+class LargeAllocation;
>+class MarkedBlock;
>+class WeakSet;
>+
>+// This is how we abstract over either MarkedBlock& or LargeAllocation&. Put things in here as you
>+// find need for them.
>+
>+class CellContainer {
>+public:
>+    CellContainer()
>+        : m_encodedPointer(0)
>+    {
>+    }
>+    
>+    CellContainer(MarkedBlock& markedBlock)
>+        : m_encodedPointer(bitwise_cast<uintptr_t>(&markedBlock))
>+    {
>+    }
>+    
>+    CellContainer(LargeAllocation& largeAllocation)
>+        : m_encodedPointer(bitwise_cast<uintptr_t>(&largeAllocation) | isLargeAllocationBit)
>+    {
>+    }
>+    
>+    explicit operator bool() const { return !!m_encodedPointer; }
>+    
>+    bool isMarkedBlock() const { return m_encodedPointer && !(m_encodedPointer & isLargeAllocationBit); }
>+    bool isLargeAllocation() const { return m_encodedPointer & isLargeAllocationBit; }
>+    
>+    MarkedBlock& markedBlock() const
>+    {
>+        ASSERT(isMarkedBlock());
>+        return *bitwise_cast<MarkedBlock*>(m_encodedPointer);
>+    }
>+    
>+    LargeAllocation& largeAllocation() const
>+    {
>+        ASSERT(isLargeAllocation());
>+        return *bitwise_cast<LargeAllocation*>(m_encodedPointer - isLargeAllocationBit);
>+    }
>+    
>+    bool isMarkedOrRetired() const;
>+    bool isMarked(HeapCell*) const;
>+    bool isMarkedOrNewlyAllocated(HeapCell*) const;
>+    
>+    void setHasAnyMarked();
>+    
>+    size_t cellSize() const;
>+    
>+    WeakSet& weakSet() const;
>+    
>+private:
>+    static const uintptr_t isLargeAllocationBit = 1;
>+    uintptr_t m_encodedPointer;
>+};
>+
>+} // namespace JSC
>+
>Index: Source/JavaScriptCore/heap/CellContainerInlines.h
>===================================================================
>--- Source/JavaScriptCore/heap/CellContainerInlines.h	(nonexistent)
>+++ Source/JavaScriptCore/heap/CellContainerInlines.h	(working copy)
>@@ -0,0 +1,77 @@
>+/*
>+ * Copyright (C) 2016 Apple Inc. All rights reserved.
>+ *
>+ * Redistribution and use in source and binary forms, with or without
>+ * modification, are permitted provided that the following conditions
>+ * are met:
>+ * 1. Redistributions of source code must retain the above copyright
>+ *    notice, this list of conditions and the following disclaimer.
>+ * 2. Redistributions in binary form must reproduce the above copyright
>+ *    notice, this list of conditions and the following disclaimer in the
>+ *    documentation and/or other materials provided with the distribution.
>+ *
>+ * THIS SOFTWARE IS PROVIDED BY APPLE INC. ``AS IS'' AND ANY
>+ * EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
>+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
>+ * PURPOSE ARE DISCLAIMED.  IN NO EVENT SHALL APPLE INC. OR
>+ * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
>+ * EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
>+ * PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
>+ * PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY
>+ * OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
>+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
>+ * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 
>+ */
>+
>+#pragma once
>+
>+#include "CellContainer.h"
>+#include "JSCell.h"
>+#include "LargeAllocation.h"
>+#include "MarkedBlock.h"
>+
>+namespace JSC {
>+
>+inline bool CellContainer::isMarkedOrRetired() const
>+{
>+    if (isLargeAllocation())
>+        return true;
>+    return markedBlock().isMarkedOrRetired();
>+}
>+
>+inline bool CellContainer::isMarked(HeapCell* cell) const
>+{
>+    if (isLargeAllocation())
>+        return largeAllocation().isMarked();
>+    return markedBlock().isMarked(cell);
>+}
>+
>+inline bool CellContainer::isMarkedOrNewlyAllocated(HeapCell* cell) const
>+{
>+    if (isLargeAllocation())
>+        return largeAllocation().isMarkedOrNewlyAllocated();
>+    return markedBlock().isMarkedOrNewlyAllocated(cell);
>+}
>+
>+inline void CellContainer::setHasAnyMarked()
>+{
>+    if (!isLargeAllocation())
>+        markedBlock().setHasAnyMarked();
>+}
>+
>+inline size_t CellContainer::cellSize() const
>+{
>+    if (isLargeAllocation())
>+        return largeAllocation().cellSize();
>+    return markedBlock().cellSize();
>+}
>+
>+inline WeakSet& CellContainer::weakSet() const
>+{
>+    if (isLargeAllocation())
>+        return largeAllocation().weakSet();
>+    return markedBlock().weakSet();
>+}
>+
>+} // namespace JSC
>+
>Index: Source/JavaScriptCore/heap/ConservativeRoots.cpp
>===================================================================
>--- Source/JavaScriptCore/heap/ConservativeRoots.cpp	(revision 204735)
>+++ Source/JavaScriptCore/heap/ConservativeRoots.cpp	(working copy)
>@@ -1,5 +1,5 @@
> /*
>- * Copyright (C) 2011 Apple Inc. All rights reserved.
>+ * Copyright (C) 2011, 2016 Apple Inc. All rights reserved.
>  *
>  * Redistribution and use in source and binary forms, with or without
>  * modification, are permitted provided that the following conditions
>@@ -31,6 +31,8 @@
> #include "CopiedSpace.h"
> #include "CopiedSpaceInlines.h"
> #include "HeapInlines.h"
>+#include "HeapUtil.h"
>+#include "JITStubRoutineSet.h"
> #include "JSCell.h"
> #include "JSObject.h"
> #include "JSCInlines.h"
>@@ -39,28 +41,27 @@
> 
> namespace JSC {
> 
>-ConservativeRoots::ConservativeRoots(MarkedBlockSet* blocks, CopiedSpace* copiedSpace)
>+ConservativeRoots::ConservativeRoots(Heap& heap)
>     : m_roots(m_inlineRoots)
>     , m_size(0)
>     , m_capacity(inlineCapacity)
>-    , m_blocks(blocks)
>-    , m_copiedSpace(copiedSpace)
>+    , m_heap(heap)
> {
> }
> 
> ConservativeRoots::~ConservativeRoots()
> {
>     if (m_roots != m_inlineRoots)
>-        OSAllocator::decommitAndRelease(m_roots, m_capacity * sizeof(JSCell*));
>+        OSAllocator::decommitAndRelease(m_roots, m_capacity * sizeof(HeapCell*));
> }
> 
> void ConservativeRoots::grow()
> {
>     size_t newCapacity = m_capacity == inlineCapacity ? nonInlineCapacity : m_capacity * 2;
>-    JSCell** newRoots = static_cast<JSCell**>(OSAllocator::reserveAndCommit(newCapacity * sizeof(JSCell*)));
>-    memcpy(newRoots, m_roots, m_size * sizeof(JSCell*));
>+    HeapCell** newRoots = static_cast<HeapCell**>(OSAllocator::reserveAndCommit(newCapacity * sizeof(HeapCell*)));
>+    memcpy(newRoots, m_roots, m_size * sizeof(HeapCell*));
>     if (m_roots != m_inlineRoots)
>-        OSAllocator::decommitAndRelease(m_roots, m_capacity * sizeof(JSCell*));
>+        OSAllocator::decommitAndRelease(m_roots, m_capacity * sizeof(HeapCell*));
>     m_capacity = newCapacity;
>     m_roots = newRoots;
> }
>@@ -70,15 +71,16 @@ inline void ConservativeRoots::genericAd
> {
>     markHook.mark(p);
> 
>-    m_copiedSpace->pinIfNecessary(p);
>+    m_heap.storageSpace().pinIfNecessary(p);
> 
>-    if (!Heap::isPointerGCObject(filter, *m_blocks, p))
>-        return;
>-
>-    if (m_size == m_capacity)
>-        grow();
>-
>-    m_roots[m_size++] = static_cast<JSCell*>(p);
>+    HeapUtil::findGCObjectPointersForMarking(
>+        m_heap, filter, p,
>+        [&] (void* p) {
>+            if (m_size == m_capacity)
>+                grow();
>+            
>+            m_roots[m_size++] = bitwise_cast<HeapCell*>(p);
>+        });
> }
> 
> template<typename MarkHook>
>@@ -94,7 +96,7 @@ void ConservativeRoots::genericAddSpan(v
>     RELEASE_ASSERT(isPointerAligned(begin));
>     RELEASE_ASSERT(isPointerAligned(end));
> 
>-    TinyBloomFilter filter = m_blocks->filter(); // Make a local copy of filter to show the compiler it won't alias, and can be register-allocated.
>+    TinyBloomFilter filter = m_heap.objectSpace().blocks().filter(); // Make a local copy of filter to show the compiler it won't alias, and can be register-allocated.
>     for (char** it = static_cast<char**>(begin); it != static_cast<char**>(end); ++it)
>         genericAddPointer(*it, filter, markHook);
> }
>Index: Source/JavaScriptCore/heap/ConservativeRoots.h
>===================================================================
>--- Source/JavaScriptCore/heap/ConservativeRoots.h	(revision 204735)
>+++ Source/JavaScriptCore/heap/ConservativeRoots.h	(working copy)
>@@ -1,5 +1,5 @@
> /*
>- * Copyright (C) 2009 Apple Inc. All rights reserved.
>+ * Copyright (C) 2009, 2016 Apple Inc. All rights reserved.
>  *
>  * Redistribution and use in source and binary forms, with or without
>  * modification, are permitted provided that the following conditions
>@@ -31,12 +31,12 @@
> namespace JSC {
> 
> class CodeBlockSet;
>+class HeapCell;
> class JITStubRoutineSet;
>-class JSCell;
> 
> class ConservativeRoots {
> public:
>-    ConservativeRoots(MarkedBlockSet*, CopiedSpace*);
>+    ConservativeRoots(Heap&);
>     ~ConservativeRoots();
> 
>     void add(void* begin, void* end);
>@@ -44,11 +44,11 @@ public:
>     void add(void* begin, void* end, JITStubRoutineSet&, CodeBlockSet&);
>     
>     size_t size();
>-    JSCell** roots();
>+    HeapCell** roots();
> 
> private:
>     static const size_t inlineCapacity = 128;
>-    static const size_t nonInlineCapacity = 8192 / sizeof(JSCell*);
>+    static const size_t nonInlineCapacity = 8192 / sizeof(HeapCell*);
>     
>     template<typename MarkHook>
>     void genericAddPointer(void*, TinyBloomFilter, MarkHook&);
>@@ -58,12 +58,11 @@ private:
>     
>     void grow();
> 
>-    JSCell** m_roots;
>+    HeapCell** m_roots;
>     size_t m_size;
>     size_t m_capacity;
>-    MarkedBlockSet* m_blocks;
>-    CopiedSpace* m_copiedSpace;
>-    JSCell* m_inlineRoots[inlineCapacity];
>+    Heap& m_heap;
>+    HeapCell* m_inlineRoots[inlineCapacity];
> };
> 
> inline size_t ConservativeRoots::size()
>@@ -71,7 +70,7 @@ inline size_t ConservativeRoots::size()
>     return m_size;
> }
> 
>-inline JSCell** ConservativeRoots::roots()
>+inline HeapCell** ConservativeRoots::roots()
> {
>     return m_roots;
> }
>Index: Source/JavaScriptCore/heap/CopyToken.h
>===================================================================
>--- Source/JavaScriptCore/heap/CopyToken.h	(revision 204735)
>+++ Source/JavaScriptCore/heap/CopyToken.h	(working copy)
>@@ -1,5 +1,5 @@
> /*
>- * Copyright (C) 2013, 2015 Apple Inc. All rights reserved.
>+ * Copyright (C) 2013, 2015-2016 Apple Inc. All rights reserved.
>  *
>  * Redistribution and use in source and binary forms, with or without
>  * modification, are permitted provided that the following conditions
>@@ -29,7 +29,6 @@
> namespace JSC {
> 
> enum CopyToken {
>-    ButterflyCopyToken,
>     TypedArrayVectorCopyToken,
>     MapBackingStoreCopyToken,
>     DirectArgumentsOverridesCopyToken
>Index: Source/JavaScriptCore/heap/FreeList.cpp
>===================================================================
>--- Source/JavaScriptCore/heap/FreeList.cpp	(nonexistent)
>+++ Source/JavaScriptCore/heap/FreeList.cpp	(working copy)
>@@ -0,0 +1,37 @@
>+/*
>+ * Copyright (C) 2016 Apple Inc. All rights reserved.
>+ *
>+ * Redistribution and use in source and binary forms, with or without
>+ * modification, are permitted provided that the following conditions
>+ * are met:
>+ * 1. Redistributions of source code must retain the above copyright
>+ *    notice, this list of conditions and the following disclaimer.
>+ * 2. Redistributions in binary form must reproduce the above copyright
>+ *    notice, this list of conditions and the following disclaimer in the
>+ *    documentation and/or other materials provided with the distribution.
>+ *
>+ * THIS SOFTWARE IS PROVIDED BY APPLE INC. ``AS IS'' AND ANY
>+ * EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
>+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
>+ * PURPOSE ARE DISCLAIMED.  IN NO EVENT SHALL APPLE INC. OR
>+ * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
>+ * EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
>+ * PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
>+ * PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY
>+ * OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
>+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
>+ * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 
>+ */
>+
>+#include "config.h"
>+#include "FreeList.h"
>+
>+namespace JSC {
>+
>+void FreeList::dump(PrintStream& out) const
>+{
>+    out.print("{head = ", RawPointer(head), ", payloadEnd = ", RawPointer(payloadEnd), ", remaining = ", remaining, ", originalSize = ", originalSize, "}");
>+}
>+
>+} // namespace JSC
>+
>Index: Source/JavaScriptCore/heap/FreeList.h
>===================================================================
>--- Source/JavaScriptCore/heap/FreeList.h	(nonexistent)
>+++ Source/JavaScriptCore/heap/FreeList.h	(working copy)
>@@ -0,0 +1,91 @@
>+/*
>+ * Copyright (C) 2016 Apple Inc. All rights reserved.
>+ *
>+ * Redistribution and use in source and binary forms, with or without
>+ * modification, are permitted provided that the following conditions
>+ * are met:
>+ * 1. Redistributions of source code must retain the above copyright
>+ *    notice, this list of conditions and the following disclaimer.
>+ * 2. Redistributions in binary form must reproduce the above copyright
>+ *    notice, this list of conditions and the following disclaimer in the
>+ *    documentation and/or other materials provided with the distribution.
>+ *
>+ * THIS SOFTWARE IS PROVIDED BY APPLE INC. ``AS IS'' AND ANY
>+ * EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
>+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
>+ * PURPOSE ARE DISCLAIMED.  IN NO EVENT SHALL APPLE INC. OR
>+ * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
>+ * EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
>+ * PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
>+ * PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY
>+ * OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
>+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
>+ * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 
>+ */
>+
>+#pragma once
>+
>+#include <wtf/PrintStream.h>
>+
>+namespace JSC {
>+
>+struct FreeCell {
>+    FreeCell* next;
>+};
>+        
>+// This representation of a FreeList is convenient for the MarkedAllocator.
>+
>+struct FreeList {
>+    FreeCell* head { nullptr };
>+    char* payloadEnd { nullptr };
>+    unsigned remaining { 0 };
>+    unsigned originalSize { 0 };
>+    
>+    FreeList()
>+    {
>+    }
>+    
>+    static FreeList list(FreeCell* head, unsigned bytes)
>+    {
>+        FreeList result;
>+        result.head = head;
>+        result.remaining = 0;
>+        result.originalSize = bytes;
>+        return result;
>+    }
>+    
>+    static FreeList bump(char* payloadEnd, unsigned remaining)
>+    {
>+        FreeList result;
>+        result.payloadEnd = payloadEnd;
>+        result.remaining = remaining;
>+        result.originalSize = remaining;
>+        return result;
>+    }
>+    
>+    bool operator==(const FreeList& other) const
>+    {
>+        return head == other.head
>+            && payloadEnd == other.payloadEnd
>+            && remaining == other.remaining
>+            && originalSize == other.originalSize;
>+    }
>+    
>+    bool operator!=(const FreeList& other) const
>+    {
>+        return !(*this == other);
>+    }
>+    
>+    explicit operator bool() const
>+    {
>+        return *this != FreeList();
>+    }
>+    
>+    bool allocationWillFail() const { return !head && !remaining; }
>+    bool allocationWillSucceed() const { return !allocationWillFail(); }
>+    
>+    void dump(PrintStream&) const;
>+};
>+
>+} // namespace JSC
>+
>Index: Source/JavaScriptCore/heap/Heap.cpp
>===================================================================
>--- Source/JavaScriptCore/heap/Heap.cpp	(revision 204735)
>+++ Source/JavaScriptCore/heap/Heap.cpp	(working copy)
>@@ -40,6 +40,7 @@
> #include "HeapVerifier.h"
> #include "IncrementalSweeper.h"
> #include "Interpreter.h"
>+#include "JITStubRoutineSet.h"
> #include "JITWorklist.h"
> #include "JSCInlines.h"
> #include "JSGlobalObject.h"
>@@ -47,6 +48,7 @@
> #include "JSVirtualMachineInternal.h"
> #include "SamplingProfiler.h"
> #include "ShadowChicken.h"
>+#include "SuperSampler.h"
> #include "TypeProfilerLog.h"
> #include "UnlinkedCodeBlock.h"
> #include "VM.h"
>@@ -74,159 +76,16 @@ namespace JSC {
> namespace {
> 
> static const size_t largeHeapSize = 32 * MB; // About 1.5X the average webpage.
>-static const size_t smallHeapSize = 1 * MB; // Matches the FastMalloc per-thread cache.
>+const size_t smallHeapSize = 1 * MB; // Matches the FastMalloc per-thread cache.
> 
>-#define ENABLE_GC_LOGGING 0
>-
>-#if ENABLE(GC_LOGGING)
>-#if COMPILER(CLANG)
>-#define DEFINE_GC_LOGGING_GLOBAL(type, name, arguments) \
>-_Pragma("clang diagnostic push") \
>-_Pragma("clang diagnostic ignored \"-Wglobal-constructors\"") \
>-_Pragma("clang diagnostic ignored \"-Wexit-time-destructors\"") \
>-static type name arguments; \
>-_Pragma("clang diagnostic pop")
>-#else
>-#define DEFINE_GC_LOGGING_GLOBAL(type, name, arguments) \
>-static type name arguments;
>-#endif // COMPILER(CLANG)
>-
>-struct GCTimer {
>-    GCTimer(const char* name)
>-        : name(name)
>-    {
>-    }
>-    ~GCTimer()
>-    {
>-        logData(allCollectionData, "(All)");
>-        logData(edenCollectionData, "(Eden)");
>-        logData(fullCollectionData, "(Full)");
>-    }
>-
>-    struct TimeRecord {
>-        TimeRecord()
>-            : time(0)
>-            , min(std::numeric_limits<double>::infinity())
>-            , max(0)
>-            , count(0)
>-        {
>-        }
>-
>-        double time;
>-        double min;
>-        double max;
>-        size_t count;
>-    };
>-
>-    void logData(const TimeRecord& data, const char* extra)
>-    {
>-        dataLogF("[%d] %s (Parent: %s) %s: %.2lfms (avg. %.2lf, min. %.2lf, max. %.2lf, count %lu)\n", 
>-            getCurrentProcessID(),
>-            name,
>-            parent ? parent->name : "nullptr",
>-            extra, 
>-            data.time * 1000, 
>-            data.time * 1000 / data.count, 
>-            data.min * 1000, 
>-            data.max * 1000,
>-            data.count);
>-    }
>-
>-    void updateData(TimeRecord& data, double duration)
>-    {
>-        if (duration < data.min)
>-            data.min = duration;
>-        if (duration > data.max)
>-            data.max = duration;
>-        data.count++;
>-        data.time += duration;
>-    }
>-
>-    void didFinishPhase(HeapOperation collectionType, double duration)
>-    {
>-        TimeRecord& data = collectionType == EdenCollection ? edenCollectionData : fullCollectionData;
>-        updateData(data, duration);
>-        updateData(allCollectionData, duration);
>-    }
>-
>-    static GCTimer* s_currentGlobalTimer;
>-
>-    TimeRecord allCollectionData;
>-    TimeRecord fullCollectionData;
>-    TimeRecord edenCollectionData;
>-    const char* name;
>-    GCTimer* parent { nullptr };
>-};
>-
>-GCTimer* GCTimer::s_currentGlobalTimer = nullptr;
>-
>-struct GCTimerScope {
>-    GCTimerScope(GCTimer& timer, HeapOperation collectionType)
>-        : timer(timer)
>-        , start(WTF::monotonicallyIncreasingTime())
>-        , collectionType(collectionType)
>-    {
>-        timer.parent = GCTimer::s_currentGlobalTimer;
>-        GCTimer::s_currentGlobalTimer = &timer;
>-    }
>-    ~GCTimerScope()
>-    {
>-        double delta = WTF::monotonicallyIncreasingTime() - start;
>-        timer.didFinishPhase(collectionType, delta);
>-        GCTimer::s_currentGlobalTimer = timer.parent;
>-    }
>-    GCTimer& timer;
>-    double start;
>-    HeapOperation collectionType;
>-};
>-
>-struct GCCounter {
>-    GCCounter(const char* name)
>-        : name(name)
>-        , count(0)
>-        , total(0)
>-        , min(10000000)
>-        , max(0)
>-    {
>-    }
>-    
>-    void add(size_t amount)
>-    {
>-        count++;
>-        total += amount;
>-        if (amount < min)
>-            min = amount;
>-        if (amount > max)
>-            max = amount;
>-    }
>-    ~GCCounter()
>-    {
>-        dataLogF("[%d] %s: %zu values (avg. %zu, min. %zu, max. %zu)\n", getCurrentProcessID(), name, total, total / count, min, max);
>-    }
>-    const char* name;
>-    size_t count;
>-    size_t total;
>-    size_t min;
>-    size_t max;
>-};
>-
>-#define GCPHASE(name) DEFINE_GC_LOGGING_GLOBAL(GCTimer, name##Timer, (#name)); GCTimerScope name##TimerScope(name##Timer, m_operationInProgress)
>-#define GCCOUNTER(name, value) do { DEFINE_GC_LOGGING_GLOBAL(GCCounter, name##Counter, (#name)); name##Counter.add(value); } while (false)
>-    
>-#else
>-
>-#define GCPHASE(name) do { } while (false)
>-#define GCCOUNTER(name, value) do { } while (false)
>-#endif
>-
>-static inline size_t minHeapSize(HeapType heapType, size_t ramSize)
>+size_t minHeapSize(HeapType heapType, size_t ramSize)
> {
>     if (heapType == LargeHeap)
>         return min(largeHeapSize, ramSize / 4);
>     return smallHeapSize;
> }
> 
>-static inline size_t proportionalHeapSize(size_t heapSize, size_t ramSize)
>+size_t proportionalHeapSize(size_t heapSize, size_t ramSize)
> {
>     // Try to stay under 1/2 RAM size to leave room for the DOM, rendering, networking, etc.
>     if (heapSize < ramSize / 4)
>@@ -236,12 +95,12 @@ static inline size_t proportionalHeapSiz
>     return 1.25 * heapSize;
> }
> 
>-static inline bool isValidSharedInstanceThreadState(VM* vm)
>+bool isValidSharedInstanceThreadState(VM* vm)
> {
>     return vm->currentThreadIsHoldingAPILock();
> }
> 
>-static inline bool isValidThreadState(VM* vm)
>+bool isValidThreadState(VM* vm)
> {
>     if (vm->atomicStringTable() != wtfThreadData().atomicStringTable())
>         return false;
>@@ -252,7 +111,7 @@ static inline bool isValidThreadState(VM
>     return true;
> }
> 
>-static inline void recordType(TypeCountSet& set, JSCell* cell)
>+void recordType(TypeCountSet& set, JSCell* cell)
> {
>     const char* typeName = "[unknown]";
>     const ClassInfo* info = cell->classInfo();
>@@ -261,6 +120,32 @@ static inline void recordType(TypeCountS
>     set.add(typeName);
> }
> 
>+bool measurePhaseTiming()
>+{
>+    return false;
>+}
>+
>+class TimingScope {
>+public:
>+    TimingScope(const char* name)
>+        : m_name(name)
>+    {
>+        if (measurePhaseTiming())
>+            m_before = monotonicallyIncreasingTimeMS();
>+    }
>+    
>+    ~TimingScope()
>+    {
>+        if (measurePhaseTiming()) {
>+            double after = monotonicallyIncreasingTimeMS();
>+            dataLog("[GC] ", m_name, " took: ", after - m_before, " ms.\n");
>+        }
>+    }
>+private:
>+    double m_before;
>+    const char* m_name;
>+};
>+
> } // anonymous namespace
> 
> Heap::Heap(VM* vm, HeapType heapType)
>@@ -287,6 +172,8 @@ Heap::Heap(VM* vm, HeapType heapType)
>     , m_machineThreads(this)
>     , m_slotVisitor(*this)
>     , m_handleSet(vm)
>+    , m_codeBlocks(std::make_unique<CodeBlockSet>())
>+    , m_jitStubRoutines(std::make_unique<JITStubRoutineSet>())
>     , m_isSafeToCollect(false)
>     , m_writeBarrierBuffer(256)
>     , m_vm(vm)
>@@ -331,7 +218,7 @@ void Heap::lastChanceToFinalize()
>     RELEASE_ASSERT(m_operationInProgress == NoOperation);
> 
>     m_arrayBuffers.lastChanceToFinalize();
>-    m_codeBlocks.lastChanceToFinalize();
>+    m_codeBlocks->lastChanceToFinalize();
>     m_objectSpace.lastChanceToFinalize();
>     releaseDelayedReleasedObjects();
> 
>@@ -434,7 +321,6 @@ void Heap::harvestWeakReferences()
> 
> void Heap::finalizeUnconditionalFinalizers()
> {
>-    GCPHASE(FinalizeUnconditionalFinalizers);
>     m_slotVisitor.finalizeUnconditionalFinalizers();
> }
> 
>@@ -460,15 +346,19 @@ void Heap::completeAllJITPlans()
> 
> void Heap::markRoots(double gcStartTime, void* stackOrigin, void* stackTop, MachineThreads::RegisterState& calleeSavedRegisters)
> {
>-    GCPHASE(MarkRoots);
>+    TimingScope markRootsTimingScope("Heap::markRoots");
>+    
>     ASSERT(isValidThreadState(m_vm));
> 
>     // We gather conservative roots before clearing mark bits because conservative
>     // gathering uses the mark bits to determine whether a reference is valid.
>-    ConservativeRoots conservativeRoots(&m_objectSpace.blocks(), &m_storageSpace);
>-    gatherStackRoots(conservativeRoots, stackOrigin, stackTop, calleeSavedRegisters);
>-    gatherJSStackRoots(conservativeRoots);
>-    gatherScratchBufferRoots(conservativeRoots);
>+    ConservativeRoots conservativeRoots(*this);
>+    {
>+        SuperSamplerScope superSamplerScope(false);
>+        gatherStackRoots(conservativeRoots, stackOrigin, stackTop, calleeSavedRegisters);
>+        gatherJSStackRoots(conservativeRoots);
>+        gatherScratchBufferRoots(conservativeRoots);
>+    }
> 
> #if ENABLE(DFG_JIT)
>     DFG::rememberCodeBlocks(*m_vm);
>@@ -528,6 +418,7 @@ void Heap::markRoots(double gcStartTime,
>     HeapRootVisitor heapRootVisitor(m_slotVisitor);
> 
>     {
>+        SuperSamplerScope superSamplerScope(false);
>         ParallelModeEnabler enabler(m_slotVisitor);
> 
>         m_slotVisitor.donateAndDrain();
>@@ -561,7 +452,7 @@ void Heap::markRoots(double gcStartTime,
> 
> void Heap::copyBackingStores()
> {
>-    GCPHASE(CopyBackingStores);
>+    SuperSamplerScope superSamplerScope(true);
>     if (m_operationInProgress == EdenCollection)
>         m_storageSpace.startedCopying<EdenCollection>();
>     else {
>@@ -598,12 +489,6 @@ void Heap::copyBackingStores()
>                         
>                         CopyWorkList& workList = block->workList();
>                         for (CopyWorklistItem item : workList) {
>-                            if (item.token() == ButterflyCopyToken) {
>-                                JSObject::copyBackingStore(
>-                                    item.cell(), copyVisitor, ButterflyCopyToken);
>-                                continue;
>-                            }
>-                            
>                             item.cell()->methodTable()->copyBackingStore(
>                                 item.cell(), copyVisitor, item.token());
>                         }
>@@ -619,16 +504,14 @@ void Heap::copyBackingStores()
> 
> void Heap::gatherStackRoots(ConservativeRoots& roots, void* stackOrigin, void* stackTop, MachineThreads::RegisterState& calleeSavedRegisters)
> {
>-    GCPHASE(GatherStackRoots);
>-    m_jitStubRoutines.clearMarks();
>-    m_machineThreads.gatherConservativeRoots(roots, m_jitStubRoutines, m_codeBlocks, stackOrigin, stackTop, calleeSavedRegisters);
>+    m_jitStubRoutines->clearMarks();
>+    m_machineThreads.gatherConservativeRoots(roots, *m_jitStubRoutines, *m_codeBlocks, stackOrigin, stackTop, calleeSavedRegisters);
> }
> 
> void Heap::gatherJSStackRoots(ConservativeRoots& roots)
> {
> #if !ENABLE(JIT)
>-    GCPHASE(GatherJSStackRoots);
>-    m_vm->interpreter->cloopStack().gatherConservativeRoots(roots, m_jitStubRoutines, m_codeBlocks);
>+    m_vm->interpreter->cloopStack().gatherConservativeRoots(roots, *m_jitStubRoutines, *m_codeBlocks);
> #else
>     UNUSED_PARAM(roots);
> #endif
>@@ -637,7 +520,6 @@ void Heap::gatherJSStackRoots(Conservati
> void Heap::gatherScratchBufferRoots(ConservativeRoots& roots)
> {
> #if ENABLE(DFG_JIT)
>-    GCPHASE(GatherScratchBufferRoots);
>     m_vm->gatherConservativeRoots(roots);
> #else
>     UNUSED_PARAM(roots);
>@@ -646,9 +528,8 @@ void Heap::gatherScratchBufferRoots(Cons
> 
> void Heap::clearLivenessData()
> {
>-    GCPHASE(ClearLivenessData);
>     if (m_operationInProgress == FullCollection)
>-        m_codeBlocks.clearMarksForFullCollection();
>+        m_codeBlocks->clearMarksForFullCollection();
> 
>     m_objectSpace.clearNewlyAllocated();
>     m_objectSpace.clearMarks();
>@@ -663,7 +544,6 @@ void Heap::visitExternalRememberedSet()
> 
> void Heap::visitSmallStrings()
> {
>-    GCPHASE(VisitSmallStrings);
>     if (!m_vm->smallStrings.needsToBeVisited(m_operationInProgress))
>         return;
> 
>@@ -675,7 +555,6 @@ void Heap::visitSmallStrings()
> 
> void Heap::visitConservativeRoots(ConservativeRoots& roots)
> {
>-    GCPHASE(VisitConservativeRoots);
>     m_slotVisitor.append(roots);
> 
>     if (Options::logGC() == GCLogging::Verbose)
>@@ -698,7 +577,6 @@ void Heap::visitCompilerWorklistWeakRefe
> void Heap::removeDeadCompilerWorklistEntries()
> {
> #if ENABLE(DFG_JIT)
>-    GCPHASE(FinalizeDFGWorklists);
>     for (auto worklist : m_suspendedCompilerWorklists)
>         worklist->removeDeadPlans(*m_vm);
> #endif
>@@ -732,7 +610,6 @@ struct GatherHeapSnapshotData : MarkedBl
> 
> void Heap::gatherExtraHeapSnapshotData(HeapProfiler& heapProfiler)
> {
>-    GCPHASE(GatherExtraHeapSnapshotData);
>     if (HeapSnapshotBuilder* builder = heapProfiler.activeSnapshotBuilder()) {
>         HeapIterationScope heapIterationScope(*this);
>         GatherHeapSnapshotData functor(*builder);
>@@ -758,7 +635,6 @@ struct RemoveDeadHeapSnapshotNodes : Mar
> 
> void Heap::removeDeadHeapSnapshotNodes(HeapProfiler& heapProfiler)
> {
>-    GCPHASE(RemoveDeadHeapSnapshotNodes);
>     if (HeapSnapshot* snapshot = heapProfiler.mostRecentSnapshot()) {
>         HeapIterationScope heapIterationScope(*this);
>         RemoveDeadHeapSnapshotNodes functor(*snapshot);
>@@ -769,8 +645,6 @@ void Heap::removeDeadHeapSnapshotNodes(H
> 
> void Heap::visitProtectedObjects(HeapRootVisitor& heapRootVisitor)
> {
>-    GCPHASE(VisitProtectedObjects);
>-
>     for (auto& pair : m_protectedValues)
>         heapRootVisitor.visit(&pair.key);
> 
>@@ -782,7 +656,6 @@ void Heap::visitProtectedObjects(HeapRoo
> 
> void Heap::visitArgumentBuffers(HeapRootVisitor& visitor)
> {
>-    GCPHASE(MarkingArgumentBuffers);
>     if (!m_markListSet || !m_markListSet->size())
>         return;
> 
>@@ -796,7 +669,6 @@ void Heap::visitArgumentBuffers(HeapRoot
> 
> void Heap::visitException(HeapRootVisitor& visitor)
> {
>-    GCPHASE(MarkingException);
>     if (!m_vm->exception() && !m_vm->lastException())
>         return;
> 
>@@ -811,7 +683,6 @@ void Heap::visitException(HeapRootVisito
> 
> void Heap::visitStrongHandles(HeapRootVisitor& visitor)
> {
>-    GCPHASE(VisitStrongHandles);
>     m_handleSet.visitStrongHandles(visitor);
> 
>     if (Options::logGC() == GCLogging::Verbose)
>@@ -822,7 +693,6 @@ void Heap::visitStrongHandles(HeapRootVi
> 
> void Heap::visitHandleStack(HeapRootVisitor& visitor)
> {
>-    GCPHASE(VisitHandleStack);
>     m_handleStack.visit(visitor);
> 
>     if (Options::logGC() == GCLogging::Verbose)
>@@ -836,7 +706,6 @@ void Heap::visitSamplingProfiler()
> #if ENABLE(SAMPLING_PROFILER)
>     if (SamplingProfiler* samplingProfiler = m_vm->samplingProfiler()) {
>         ASSERT(samplingProfiler->getLock().isLocked());
>-        GCPHASE(VisitSamplingProfiler);
>         samplingProfiler->visit(m_slotVisitor);
>         if (Options::logGC() == GCLogging::Verbose)
>             dataLog("Sampling Profiler data:\n", m_slotVisitor);
>@@ -854,8 +723,7 @@ void Heap::visitShadowChicken()
> 
> void Heap::traceCodeBlocksAndJITStubRoutines()
> {
>-    GCPHASE(TraceCodeBlocksAndJITStubRoutines);
>-    m_jitStubRoutines.traceMarkedStubRoutines(m_slotVisitor);
>+    m_jitStubRoutines->traceMarkedStubRoutines(m_slotVisitor);
> 
>     if (Options::logGC() == GCLogging::Verbose)
>         dataLog("Code Blocks and JIT Stub Routines:\n", m_slotVisitor);
>@@ -865,13 +733,11 @@ void Heap::traceCodeBlocksAndJITStubRout
> 
> void Heap::converge()
> {
>-    GCPHASE(Convergence);
>     m_slotVisitor.drainFromShared(SlotVisitor::MasterDrain);
> }
> 
> void Heap::visitWeakHandles(HeapRootVisitor& visitor)
> {
>-    GCPHASE(VisitingLiveWeakHandles);
>     while (true) {
>         m_objectSpace.visitWeakSets(visitor);
>         harvestWeakReferences();
>@@ -892,8 +758,6 @@ void Heap::visitWeakHandles(HeapRootVisi
> 
> void Heap::updateObjectCounts(double gcStartTime)
> {
>-    GCCOUNTER(VisitedValueCount, m_slotVisitor.visitCount() + threadVisitCount());
>-
>     if (Options::logGC() == GCLogging::Verbose) {
>         size_t visitCount = m_slotVisitor.visitCount();
>         visitCount += threadVisitCount();
>@@ -1033,7 +897,6 @@ void Heap::deleteAllUnlinkedCodeBlocks()
> 
> void Heap::clearUnmarkedExecutables()
> {
>-    GCPHASE(ClearUnmarkedExecutables);
>     for (unsigned i = m_executables.size(); i--;) {
>         ExecutableBase* current = m_executables[i];
>         if (isMarked(current))
>@@ -1051,10 +914,9 @@ void Heap::clearUnmarkedExecutables()
> 
> void Heap::deleteUnmarkedCompiledCode()
> {
>-    GCPHASE(DeleteCodeBlocks);
>     clearUnmarkedExecutables();
>-    m_codeBlocks.deleteUnmarkedAndUnreferenced(m_operationInProgress);
>-    m_jitStubRoutines.deleteUnmarkedJettisonedStubRoutines();
>+    m_codeBlocks->deleteUnmarkedAndUnreferenced(m_operationInProgress);
>+    m_jitStubRoutines->deleteUnmarkedJettisonedStubRoutines();
> }
> 
> void Heap::addToRememberedSet(const JSCell* cell)
>@@ -1073,10 +935,11 @@ void Heap::addToRememberedSet(const JSCe
> 
> void Heap::collectAllGarbage()
> {
>+    SuperSamplerScope superSamplerScope(false);
>     if (!m_isSafeToCollect)
>         return;
> 
>-    collect(FullCollection);
>+    collectWithoutAnySweep(FullCollection);
> 
>     DeferGCForAWhile deferGC(*this);
>     if (UNLIKELY(Options::useImmortalObjects()))
>@@ -1090,7 +953,19 @@ void Heap::collectAllGarbage()
>     sweepAllLogicallyEmptyWeakBlocks();
> }
> 
>-NEVER_INLINE void Heap::collect(HeapOperation collectionType)
>+void Heap::collect(HeapOperation collectionType)
>+{
>+    SuperSamplerScope superSamplerScope(false);
>+    if (!m_isSafeToCollect)
>+        return;
>+
>+    collectWithoutAnySweep(collectionType);
>+
>+    DeferGCForAWhile deferGC(*this);
>+    m_objectSpace.sweepLargeAllocations();
>+}
>+
>+NEVER_INLINE void Heap::collectWithoutAnySweep(HeapOperation collectionType)
> {
>     void* stackTop;
>     ALLOCATE_AND_GET_REGISTER_STATE(registers);
>@@ -1102,6 +977,8 @@ NEVER_INLINE void Heap::collect(HeapOper
> 
> NEVER_INLINE void Heap::collectImpl(HeapOperation collectionType, void* stackOrigin, void* stackTop, MachineThreads::RegisterState& calleeSavedRegisters)
> {
>+    TimingScope collectImplTimingScope("Heap::collectImpl");
>+    
> #if ENABLE(ALLOCATION_LOGGING)
>     dataLogF("JSC GC starting collection.\n");
> #endif
>@@ -1134,7 +1011,6 @@ NEVER_INLINE void Heap::collectImpl(Heap
> 
>     suspendCompilerThreads();
>     willStartCollection(collectionType);
>-    GCPHASE(Collect);
> 
>     double gcStartTime = WTF::monotonicallyIncreasingTime();
>     if (m_verifier) {
>@@ -1148,9 +1024,12 @@ NEVER_INLINE void Heap::collectImpl(Heap
> 
>     flushOldStructureIDTables();
>     stopAllocation();
>+    prepareForMarking();
>     flushWriteBarrierBuffer();
> 
>     markRoots(gcStartTime, stackOrigin, stackTop, calleeSavedRegisters);
>+    
>+    TimingScope lateTimingScope("Heap::collectImpl after markRoots");
> 
>     if (m_verifier) {
>         m_verifier->gatherLiveObjects(HeapVerifier::Phase::AfterMarking);
>@@ -1164,9 +1043,7 @@ NEVER_INLINE void Heap::collectImpl(Heap
>     pruneStaleEntriesFromWeakGCMaps();
>     sweepArrayBuffers();
>     snapshotMarkedSpace();
>-
>     copyBackingStores();
>-
>     finalizeUnconditionalFinalizers();
>     removeDeadCompilerWorklistEntries();
>     deleteUnmarkedCompiledCode();
>@@ -1179,7 +1056,7 @@ NEVER_INLINE void Heap::collectImpl(Heap
>     updateAllocationLimits();
>     didFinishCollection(gcStartTime);
>     resumeCompilerThreads();
>-
>+    
>     if (m_verifier) {
>         m_verifier->trimDeadObjects();
>         m_verifier->verify(HeapVerifier::Phase::AfterGC);
>@@ -1194,7 +1071,6 @@ NEVER_INLINE void Heap::collectImpl(Heap
> void Heap::suspendCompilerThreads()
> {
> #if ENABLE(DFG_JIT)
>-    GCPHASE(SuspendCompilerThreads);
>     ASSERT(m_suspendedCompilerWorklists.isEmpty());
>     for (unsigned i = DFG::numberOfWorklists(); i--;) {
>         if (DFG::Worklist* worklist = DFG::worklistForIndexOrNull(i)) {
>@@ -1207,8 +1083,6 @@ void Heap::suspendCompilerThreads()
> 
> void Heap::willStartCollection(HeapOperation collectionType)
> {
>-    GCPHASE(StartingCollection);
>-    
>     if (Options::logGC())
>         dataLog("=> ");
>     
>@@ -1246,13 +1120,11 @@ void Heap::willStartCollection(HeapOpera
> 
> void Heap::flushOldStructureIDTables()
> {
>-    GCPHASE(FlushOldStructureIDTables);
>     m_structureIDTable.flushOldTables();
> }
> 
> void Heap::flushWriteBarrierBuffer()
> {
>-    GCPHASE(FlushWriteBarrierBuffer);
>     if (m_operationInProgress == EdenCollection) {
>         m_writeBarrierBuffer.flush(*this);
>         return;
>@@ -1262,21 +1134,23 @@ void Heap::flushWriteBarrierBuffer()
> 
> void Heap::stopAllocation()
> {
>-    GCPHASE(StopAllocation);
>     m_objectSpace.stopAllocating();
>     if (m_operationInProgress == FullCollection)
>         m_storageSpace.didStartFullCollection();
> }
> 
>+void Heap::prepareForMarking()
>+{
>+    m_objectSpace.prepareForMarking();
>+}
>+
> void Heap::reapWeakHandles()
> {
>-    GCPHASE(ReapingWeakHandles);
>     m_objectSpace.reapWeakSets();
> }
> 
> void Heap::pruneStaleEntriesFromWeakGCMaps()
> {
>-    GCPHASE(PruningStaleEntriesFromWeakGCMaps);
>     if (m_operationInProgress != FullCollection)
>         return;
>     for (auto& pruneCallback : m_weakGCMaps.values())
>@@ -1285,7 +1159,6 @@ void Heap::pruneStaleEntriesFromWeakGCMa
> 
> void Heap::sweepArrayBuffers()
> {
>-    GCPHASE(SweepingArrayBuffers);
>     m_arrayBuffers.sweep();
> }
> 
>@@ -1306,8 +1179,10 @@ struct MarkedBlockSnapshotFunctor : publ
> 
> void Heap::snapshotMarkedSpace()
> {
>-    GCPHASE(SnapshotMarkedSpace);
>-
>+    // FIXME: This should probably be renamed. It's not actually snapshotting all of MarkedSpace.
>+    // This is used by IncrementalSweeper, so it only needs to snapshot blocks. However, if we ever
>+    // wanted to add other snapshotting login, we'd probably put it here.
>+    
>     if (m_operationInProgress == EdenCollection) {
>         m_blockSnapshot.appendVector(m_objectSpace.blocksWithNewObjects());
>         // Sort and deduplicate the block snapshot since we might be appending to an unfinished work list.
>@@ -1322,14 +1197,11 @@ void Heap::snapshotMarkedSpace()
> 
> void Heap::deleteSourceProviderCaches()
> {
>-    GCPHASE(DeleteSourceProviderCaches);
>     m_vm->clearSourceProviderCaches();
> }
> 
> void Heap::notifyIncrementalSweeper()
> {
>-    GCPHASE(NotifyIncrementalSweeper);
>-
>     if (m_operationInProgress == FullCollection) {
>         if (!m_logicallyEmptyWeakBlocks.isEmpty())
>             m_indexOfNextLogicallyEmptyWeakBlockToSweep = 0;
>@@ -1340,19 +1212,22 @@ void Heap::notifyIncrementalSweeper()
> 
> void Heap::writeBarrierCurrentlyExecutingCodeBlocks()
> {
>-    GCPHASE(WriteBarrierCurrentlyExecutingCodeBlocks);
>-    m_codeBlocks.writeBarrierCurrentlyExecutingCodeBlocks(this);
>+    m_codeBlocks->writeBarrierCurrentlyExecutingCodeBlocks(this);
> }
> 
> void Heap::resetAllocators()
> {
>-    GCPHASE(ResetAllocators);
>     m_objectSpace.resetAllocators();
> }
> 
> void Heap::updateAllocationLimits()
> {
>-    GCPHASE(UpdateAllocationLimits);
>+    static const bool verbose = false;
>+    
>+    if (verbose) {
>+        dataLog("\n");
>+        dataLog("bytesAllocatedThisCycle = ", m_bytesAllocatedThisCycle, "\n");
>+    }
>     
>     // Calculate our current heap size threshold for the purpose of figuring out when we should
>     // run another collection. This isn't the same as either size() or capacity(), though it should
>@@ -1369,6 +1244,8 @@ void Heap::updateAllocationLimits()
>     // of fragmentation, this may be substantial. Fortunately, marked space rarely fragments because
>     // cells usually have a narrow range of sizes. So, the underestimation is probably OK.
>     currentHeapSize += m_totalBytesVisited;
>+    if (verbose)
>+        dataLog("totalBytesVisited = ", m_totalBytesVisited, ", currentHeapSize = ", currentHeapSize, "\n");
> 
>     // For copied space, we use the capacity of storage space. This is because copied space may get
>     // badly fragmented between full collections. This arises when each eden collection evacuates
>@@ -1384,10 +1261,15 @@ void Heap::updateAllocationLimits()
>     // https://bugs.webkit.org/show_bug.cgi?id=150268
>     ASSERT(m_totalBytesCopied <= m_storageSpace.size());
>     currentHeapSize += m_storageSpace.capacity();
>+    if (verbose)
>+        dataLog("storageSpace.capacity() = ", m_storageSpace.capacity(), ", currentHeapSize = ", currentHeapSize, "\n");
> 
>     // It's up to the user to ensure that extraMemorySize() ends up corresponding to allocation-time
>     // extra memory reporting.
>     currentHeapSize += extraMemorySize();
>+
>+    if (verbose)
>+        dataLog("extraMemorySize() = ", extraMemorySize(), ", currentHeapSize = ", currentHeapSize, "\n");
>     
>     if (Options::gcMaxHeapSize() && currentHeapSize > Options::gcMaxHeapSize())
>         HeapStatistics::exitWithFailure();
>@@ -1397,29 +1279,38 @@ void Heap::updateAllocationLimits()
>         // the new allocation limit based on the current size of the heap, with a
>         // fixed minimum.
>         m_maxHeapSize = max(minHeapSize(m_heapType, m_ramSize), proportionalHeapSize(currentHeapSize, m_ramSize));
>+        if (verbose)
>+            dataLog("Full: maxHeapSize = ", m_maxHeapSize, "\n");
>         m_maxEdenSize = m_maxHeapSize - currentHeapSize;
>+        if (verbose)
>+            dataLog("Full: maxEdenSize = ", m_maxEdenSize, "\n");
>         m_sizeAfterLastFullCollect = currentHeapSize;
>+        if (verbose)
>+            dataLog("Full: sizeAfterLastFullCollect = ", currentHeapSize, "\n");
>         m_bytesAbandonedSinceLastFullCollect = 0;
>+        if (verbose)
>+            dataLog("Full: bytesAbandonedSinceLastFullCollect = ", 0, "\n");
>     } else {
>-        static const bool verbose = false;
>-        
>         ASSERT(currentHeapSize >= m_sizeAfterLastCollect);
>-        m_maxEdenSize = m_maxHeapSize - currentHeapSize;
>+        // Theoretically, we shouldn't ever scan more memory than the heap size we planned to have.
>+        // But we are sloppy, so we have to defend against the overflow.
>+        m_maxEdenSize = currentHeapSize > m_maxHeapSize ? 0 : m_maxHeapSize - currentHeapSize;
>+        if (verbose)
>+            dataLog("Eden: maxEdenSize = ", m_maxEdenSize, "\n");
>         m_sizeAfterLastEdenCollect = currentHeapSize;
>-        if (verbose) {
>-            dataLog("Max heap size: ", m_maxHeapSize, "\n");
>-            dataLog("Current heap size: ", currentHeapSize, "\n");
>-            dataLog("Size after last eden collection: ", m_sizeAfterLastEdenCollect, "\n");
>-        }
>-        double edenToOldGenerationRatio = (double)m_maxEdenSize / (double)m_maxHeapSize;
>         if (verbose)
>-            dataLog("Eden to old generation ratio: ", edenToOldGenerationRatio, "\n");
>+            dataLog("Eden: sizeAfterLastEdenCollect = ", currentHeapSize, "\n");
>+        double edenToOldGenerationRatio = (double)m_maxEdenSize / (double)m_maxHeapSize;
>         double minEdenToOldGenerationRatio = 1.0 / 3.0;
>         if (edenToOldGenerationRatio < minEdenToOldGenerationRatio)
>             m_shouldDoFullCollection = true;
>         // This seems suspect at first, but what it does is ensure that the nursery size is fixed.
>         m_maxHeapSize += currentHeapSize - m_sizeAfterLastCollect;
>+        if (verbose)
>+            dataLog("Eden: maxHeapSize = ", m_maxHeapSize, "\n");
>         m_maxEdenSize = m_maxHeapSize - currentHeapSize;
>+        if (verbose)
>+            dataLog("Eden: maxEdenSize = ", m_maxEdenSize, "\n");
>         if (m_fullActivityCallback) {
>             ASSERT(currentHeapSize >= m_sizeAfterLastFullCollect);
>             m_fullActivityCallback->didAllocate(currentHeapSize - m_sizeAfterLastFullCollect);
>@@ -1427,6 +1318,8 @@ void Heap::updateAllocationLimits()
>     }
> 
>     m_sizeAfterLastCollect = currentHeapSize;
>+    if (verbose)
>+        dataLog("sizeAfterLastCollect = ", m_sizeAfterLastCollect, "\n");
>     m_bytesAllocatedThisCycle = 0;
> 
>     if (Options::logGC())
>@@ -1435,7 +1328,6 @@ void Heap::updateAllocationLimits()
> 
> void Heap::didFinishCollection(double gcStartTime)
> {
>-    GCPHASE(FinishingCollection);
>     double gcEndTime = WTF::monotonicallyIncreasingTime();
>     HeapOperation operation = m_operationInProgress;
>     if (m_operationInProgress == FullCollection)
>@@ -1471,7 +1363,6 @@ void Heap::didFinishCollection(double gc
> void Heap::resumeCompilerThreads()
> {
> #if ENABLE(DFG_JIT)
>-    GCPHASE(ResumeCompilerThreads);
>     for (auto worklist : m_suspendedCompilerWorklists)
>         worklist->resumeAllThreads();
>     m_suspendedCompilerWorklists.clear();
>@@ -1580,7 +1471,7 @@ public:
>         if (cell->isZapped())
>             current++;
> 
>-        void* limit = static_cast<void*>(reinterpret_cast<char*>(cell) + MarkedBlock::blockFor(cell)->cellSize());
>+        void* limit = static_cast<void*>(reinterpret_cast<char*>(cell) + cell->cellSize());
>         for (; current < limit; current++)
>             *current = zombifiedBits;
>     }
>@@ -1686,4 +1577,12 @@ size_t Heap::threadBytesCopied()
>     return result;
> }
> 
>+void Heap::forEachCodeBlockImpl(const ScopedLambda<bool(CodeBlock*)>& func)
>+{
>+    // We don't know the full set of CodeBlocks until compilation has terminated.
>+    completeAllJITPlans();
>+
>+    return m_codeBlocks->iterate(func);
>+}
>+
> } // namespace JSC
>Index: Source/JavaScriptCore/heap/Heap.h
>===================================================================
>--- Source/JavaScriptCore/heap/Heap.h	(revision 204735)
>+++ Source/JavaScriptCore/heap/Heap.h	(working copy)
>@@ -23,14 +23,12 @@
> #define Heap_h
> 
> #include "ArrayBuffer.h"
>-#include "CodeBlockSet.h"
> #include "CopyVisitor.h"
> #include "GCIncomingRefCountedSet.h"
> #include "HandleSet.h"
> #include "HandleStack.h"
> #include "HeapObserver.h"
> #include "HeapOperation.h"
>-#include "JITStubRoutineSet.h"
> #include "ListableHandler.h"
> #include "MachineStackMarker.h"
> #include "MarkedAllocator.h"
>@@ -53,6 +51,7 @@
> namespace JSC {
> 
> class CodeBlock;
>+class CodeBlockSet;
> class CopiedSpace;
> class EdenGCActivityCallback;
> class ExecutableBase;
>@@ -65,6 +64,7 @@ class HeapRootVisitor;
> class HeapVerifier;
> class IncrementalSweeper;
> class JITStubRoutine;
>+class JITStubRoutineSet;
> class JSCell;
> class JSValue;
> class LLIntOffsetsExtractor;
>@@ -83,13 +83,15 @@ typedef HashCountedSet<const char*> Type
> 
> enum HeapType { SmallHeap, LargeHeap };
> 
>+class HeapUtil;
>+
> class Heap {
>     WTF_MAKE_NONCOPYABLE(Heap);
> public:
>     friend class JIT;
>     friend class DFG::SpeculativeJIT;
>     static Heap* heap(const JSValue); // 0 for immediate values
>-    static Heap* heap(const JSCell*);
>+    static Heap* heap(const HeapCell*);
> 
>     // This constant determines how many blocks we iterate between checks of our 
>     // deadline when calling Heap::isPagedOut. Decreasing it will cause us to detect 
>@@ -101,11 +103,8 @@ public:
>     static bool isMarked(const void*);
>     static bool testAndSetMarked(const void*);
>     static void setMarked(const void*);
>-
>-    // This function must be run after stopAllocation() is called and 
>-    // before liveness data is cleared to be accurate.
>-    static bool isPointerGCObject(TinyBloomFilter, MarkedBlockSet&, void* pointer);
>-    static bool isValueGCObject(TinyBloomFilter, MarkedBlockSet&, JSValue);
>+    
>+    static size_t cellSize(const void*);
> 
>     void writeBarrier(const JSCell*);
>     void writeBarrier(const JSCell*, JSValue);
>@@ -147,10 +146,14 @@ public:
>     MarkedSpace::Subspace& subspaceForObjectDestructor() { return m_objectSpace.subspaceForObjectsWithDestructor(); }
>     MarkedSpace::Subspace& subspaceForAuxiliaryData() { return m_objectSpace.subspaceForAuxiliaryData(); }
>     template<typename ClassType> MarkedSpace::Subspace& subspaceForObjectOfType();
>-    MarkedAllocator& allocatorForObjectWithoutDestructor(size_t bytes) { return m_objectSpace.allocatorFor(bytes); }
>-    MarkedAllocator& allocatorForObjectWithDestructor(size_t bytes) { return m_objectSpace.destructorAllocatorFor(bytes); }
>-    template<typename ClassType> MarkedAllocator& allocatorForObjectOfType(size_t bytes);
>+    MarkedAllocator* allocatorForObjectWithoutDestructor(size_t bytes) { return m_objectSpace.allocatorFor(bytes); }
>+    MarkedAllocator* allocatorForObjectWithDestructor(size_t bytes) { return m_objectSpace.destructorAllocatorFor(bytes); }
>+    template<typename ClassType> MarkedAllocator* allocatorForObjectOfType(size_t bytes);
>+    MarkedAllocator* allocatorForAuxiliaryData(size_t bytes) { return m_objectSpace.auxiliaryAllocatorFor(bytes); }
>     CopiedAllocator& storageAllocator() { return m_storageSpace.allocator(); }
>+    void* allocateAuxiliary(JSCell* intendedOwner, size_t);
>+    void* tryAllocateAuxiliary(JSCell* intendedOwner, size_t);
>+    void* tryReallocateAuxiliary(JSCell* intendedOwner, void* oldBase, size_t oldSize, size_t newSize);
>     CheckedBoolean tryAllocateStorage(JSCell* intendedOwner, size_t, void**);
>     CheckedBoolean tryReallocateStorage(JSCell* intendedOwner, void**, size_t, size_t);
>     void ascribeOwner(JSCell* intendedOwner, void*);
>@@ -230,7 +233,7 @@ public:
>     void didAllocate(size_t);
>     bool isPagedOut(double deadline);
>     
>-    const JITStubRoutineSet& jitStubRoutines() { return m_jitStubRoutines; }
>+    const JITStubRoutineSet& jitStubRoutines() { return *m_jitStubRoutines; }
>     
>     void addReference(JSCell*, ArrayBuffer*);
>     
>@@ -238,7 +241,7 @@ public:
> 
>     StructureIDTable& structureIDTable() { return m_structureIDTable; }
> 
>-    CodeBlockSet& codeBlockSet() { return m_codeBlocks; }
>+    CodeBlockSet& codeBlockSet() { return *m_codeBlocks; }
> 
> #if USE(FOUNDATION)
>     template<typename T> void releaseSoon(RetainPtr<T>&&);
>@@ -267,6 +270,7 @@ private:
>     friend class GCLogging;
>     friend class GCThread;
>     friend class HandleSet;
>+    friend class HeapUtil;
>     friend class HeapVerifier;
>     friend class JITStubRoutine;
>     friend class LLIntOffsetsExtractor;
>@@ -283,6 +287,8 @@ private:
>     template<typename T> friend void* allocateCell(Heap&);
>     template<typename T> friend void* allocateCell(Heap&, size_t);
> 
>+    void collectWithoutAnySweep(HeapOperation collectionType = AnyCollection);
>+
>     void* allocateWithDestructor(size_t); // For use with objects with destructors.
>     void* allocateWithoutDestructor(size_t); // For use with objects without destructors.
>     template<typename ClassType> void* allocateObjectOfType(size_t); // Chooses one of the methods above based on type.
>@@ -304,6 +310,7 @@ private:
>     void flushOldStructureIDTables();
>     void flushWriteBarrierBuffer();
>     void stopAllocation();
>+    void prepareForMarking();
>     
>     void markRoots(double gcStartTime, void* stackOrigin, void* stackTop, MachineThreads::RegisterState&);
>     void gatherStackRoots(ConservativeRoots&, void* stackOrigin, void* stackTop, MachineThreads::RegisterState&);
>@@ -362,6 +369,8 @@ private:
>     size_t threadBytesVisited();
>     size_t threadBytesCopied();
> 
>+    void forEachCodeBlockImpl(const ScopedLambda<bool(CodeBlock*)>&);
>+
>     const HeapType m_heapType;
>     const size_t m_ramSize;
>     const size_t m_minBytesPerCycle;
>@@ -408,8 +417,8 @@ private:
> 
>     HandleSet m_handleSet;
>     HandleStack m_handleStack;
>-    CodeBlockSet m_codeBlocks;
>-    JITStubRoutineSet m_jitStubRoutines;
>+    std::unique_ptr<CodeBlockSet> m_codeBlocks;
>+    std::unique_ptr<JITStubRoutineSet> m_jitStubRoutines;
>     FinalizerOwner m_finalizerOwner;
>     
>     bool m_isSafeToCollect;
>Index: Source/JavaScriptCore/heap/HeapCell.h
>===================================================================
>--- Source/JavaScriptCore/heap/HeapCell.h	(revision 204735)
>+++ Source/JavaScriptCore/heap/HeapCell.h	(working copy)
>@@ -25,8 +25,17 @@
> 
> #pragma once
> 
>+#include "DestructionMode.h"
>+
> namespace JSC {
> 
>+class CellContainer;
>+class Heap;
>+class LargeAllocation;
>+class MarkedBlock;
>+class VM;
>+struct AllocatorAttributes;
>+
> class HeapCell {
> public:
>     enum Kind : int8_t {
>@@ -38,6 +47,22 @@ public:
>     
>     void zap() { *reinterpret_cast<uintptr_t**>(this) = 0; }
>     bool isZapped() const { return !*reinterpret_cast<uintptr_t* const*>(this); }
>+    
>+    bool isLargeAllocation() const;
>+    CellContainer cellContainer() const;
>+    MarkedBlock& markedBlock() const;
>+    LargeAllocation& largeAllocation() const;
>+
>+    // If you want performance and you know that your cell is small, you can do this instead:
>+    // ASSERT(!cell->isLargeAllocation());
>+    // cell->markedBlock().vm()
>+    Heap* heap() const;
>+    VM* vm() const;
>+    
>+    size_t cellSize() const;
>+    AllocatorAttributes allocatorAttributes() const;
>+    DestructionMode destructionMode() const;
>+    Kind cellKind() const;
> };
> 
> } // namespace JSC
>Index: Source/JavaScriptCore/heap/HeapCellInlines.h
>===================================================================
>--- Source/JavaScriptCore/heap/HeapCellInlines.h	(nonexistent)
>+++ Source/JavaScriptCore/heap/HeapCellInlines.h	(working copy)
>@@ -0,0 +1,92 @@
>+/*
>+ * Copyright (C) 2016 Apple Inc. All rights reserved.
>+ *
>+ * Redistribution and use in source and binary forms, with or without
>+ * modification, are permitted provided that the following conditions
>+ * are met:
>+ * 1. Redistributions of source code must retain the above copyright
>+ *    notice, this list of conditions and the following disclaimer.
>+ * 2. Redistributions in binary form must reproduce the above copyright
>+ *    notice, this list of conditions and the following disclaimer in the
>+ *    documentation and/or other materials provided with the distribution.
>+ *
>+ * THIS SOFTWARE IS PROVIDED BY APPLE INC. ``AS IS'' AND ANY
>+ * EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
>+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
>+ * PURPOSE ARE DISCLAIMED.  IN NO EVENT SHALL APPLE INC. OR
>+ * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
>+ * EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
>+ * PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
>+ * PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY
>+ * OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
>+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
>+ * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 
>+ */
>+
>+#pragma once
>+
>+#include "CellContainer.h"
>+#include "HeapCell.h"
>+#include "LargeAllocation.h"
>+#include "MarkedBlock.h"
>+
>+namespace JSC {
>+
>+ALWAYS_INLINE bool HeapCell::isLargeAllocation() const
>+{
>+    return LargeAllocation::isLargeAllocation(const_cast<HeapCell*>(this));
>+}
>+
>+ALWAYS_INLINE CellContainer HeapCell::cellContainer() const
>+{
>+    if (isLargeAllocation())
>+        return largeAllocation();
>+    return markedBlock();
>+}
>+
>+ALWAYS_INLINE MarkedBlock& HeapCell::markedBlock() const
>+{
>+    return *MarkedBlock::blockFor(this);
>+}
>+
>+ALWAYS_INLINE LargeAllocation& HeapCell::largeAllocation() const
>+{
>+    return *LargeAllocation::fromCell(const_cast<HeapCell*>(this));
>+}
>+
>+ALWAYS_INLINE Heap* HeapCell::heap() const
>+{
>+    return Heap::heap(this);
>+}
>+
>+ALWAYS_INLINE VM* HeapCell::vm() const
>+{
>+    return heap()->vm();
>+}
>+    
>+ALWAYS_INLINE size_t HeapCell::cellSize() const
>+{
>+    if (isLargeAllocation())
>+        return largeAllocation().cellSize();
>+    return markedBlock().cellSize();
>+}
>+
>+ALWAYS_INLINE AllocatorAttributes HeapCell::allocatorAttributes() const
>+{
>+    if (isLargeAllocation())
>+        return largeAllocation().attributes();
>+    return markedBlock().attributes();
>+}
>+
>+ALWAYS_INLINE DestructionMode HeapCell::destructionMode() const
>+{
>+    return allocatorAttributes().destruction;
>+}
>+
>+ALWAYS_INLINE HeapCell::Kind HeapCell::cellKind() const
>+{
>+    return allocatorAttributes().cellKind;
>+}
>+
>+} // namespace JSC
>+
>Index: Source/JavaScriptCore/heap/HeapInlines.h
>===================================================================
>--- Source/JavaScriptCore/heap/HeapInlines.h	(revision 204735)
>+++ Source/JavaScriptCore/heap/HeapInlines.h	(working copy)
>@@ -28,6 +28,9 @@
> 
> #include "CopyBarrier.h"
> #include "Heap.h"
>+#include "HeapCellInlines.h"
>+#include "IndexingHeader.h"
>+#include "JSCallee.h"
> #include "JSCell.h"
> #include "Structure.h"
> #include <type_traits>
>@@ -59,9 +62,11 @@ inline bool Heap::isCollecting()
>     return m_operationInProgress == FullCollection || m_operationInProgress == EdenCollection;
> }
> 
>-inline Heap* Heap::heap(const JSCell* cell)
>+ALWAYS_INLINE Heap* Heap::heap(const HeapCell* cell)
> {
>-    return MarkedBlock::blockFor(cell)->heap();
>+    if (cell->isLargeAllocation())
>+        return cell->largeAllocation().heap();
>+    return cell->markedBlock().heap();
> }
> 
> inline Heap* Heap::heap(const JSValue v)
>@@ -71,24 +76,42 @@ inline Heap* Heap::heap(const JSValue v)
>     return heap(v.asCell());
> }
> 
>-inline bool Heap::isLive(const void* cell)
>+inline bool Heap::isLive(const void* rawCell)
> {
>-    return MarkedBlock::blockFor(cell)->isLiveCell(cell);
>+    HeapCell* cell = bitwise_cast<HeapCell*>(rawCell);
>+    if (cell->isLargeAllocation())
>+        return cell->largeAllocation().isLive();
>+    return cell->markedBlock().isLiveCell(cell);
> }
> 
>-inline bool Heap::isMarked(const void* cell)
>+ALWAYS_INLINE bool Heap::isMarked(const void* rawCell)
> {
>-    return MarkedBlock::blockFor(cell)->isMarked(cell);
>+    HeapCell* cell = bitwise_cast<HeapCell*>(rawCell);
>+    if (cell->isLargeAllocation())
>+        return cell->largeAllocation().isMarked();
>+    return cell->markedBlock().isMarked(cell);
> }
> 
>-inline bool Heap::testAndSetMarked(const void* cell)
>+ALWAYS_INLINE bool Heap::testAndSetMarked(const void* rawCell)
> {
>-    return MarkedBlock::blockFor(cell)->testAndSetMarked(cell);
>+    HeapCell* cell = bitwise_cast<HeapCell*>(rawCell);
>+    if (cell->isLargeAllocation())
>+        return cell->largeAllocation().testAndSetMarked();
>+    return cell->markedBlock().testAndSetMarked(cell);
> }
> 
>-inline void Heap::setMarked(const void* cell)
>+inline void Heap::setMarked(const void* rawCell)
> {
>-    MarkedBlock::blockFor(cell)->setMarked(cell);
>+    HeapCell* cell = bitwise_cast<HeapCell*>(rawCell);
>+    if (cell->isLargeAllocation())
>+        cell->largeAllocation().setMarked();
>+    else
>+        cell->markedBlock().setMarked(cell);
>+}
>+
>+ALWAYS_INLINE size_t Heap::cellSize(const void* rawCell)
>+{
>+    return bitwise_cast<HeapCell*>(rawCell)->cellSize();
> }
> 
> inline void Heap::writeBarrier(const JSCell* from, JSValue to)
>@@ -165,12 +188,9 @@ inline void Heap::deprecatedReportExtraM
>         deprecatedReportExtraMemorySlowCase(size);
> }
> 
>-template<typename Functor> inline void Heap::forEachCodeBlock(const Functor& functor)
>+template<typename Functor> inline void Heap::forEachCodeBlock(const Functor& func)
> {
>-    // We don't know the full set of CodeBlocks until compilation has terminated.
>-    completeAllJITPlans();
>-
>-    return m_codeBlocks.iterate<Functor>(functor);
>+    forEachCodeBlockImpl(scopedLambdaRef<bool(CodeBlock*)>(func));
> }
> 
> template<typename Functor> inline void Heap::forEachProtectedCell(const Functor& functor)
>@@ -199,7 +219,7 @@ inline void* Heap::allocateWithoutDestru
> }
> 
> template<typename ClassType>
>-void* Heap::allocateObjectOfType(size_t bytes)
>+inline void* Heap::allocateObjectOfType(size_t bytes)
> {
>     // JSCell::classInfo() expects objects allocated with normal destructor to derive from JSDestructibleObject.
>     ASSERT((!ClassType::needsDestruction || (ClassType::StructureFlags & StructureIsImmortal) || std::is_convertible<ClassType, JSDestructibleObject>::value));
>@@ -210,7 +230,7 @@ void* Heap::allocateObjectOfType(size_t 
> }
> 
> template<typename ClassType>
>-MarkedSpace::Subspace& Heap::subspaceForObjectOfType()
>+inline MarkedSpace::Subspace& Heap::subspaceForObjectOfType()
> {
>     // JSCell::classInfo() expects objects allocated with normal destructor to derive from JSDestructibleObject.
>     ASSERT((!ClassType::needsDestruction || (ClassType::StructureFlags & StructureIsImmortal) || std::is_convertible<ClassType, JSDestructibleObject>::value));
>@@ -221,14 +241,50 @@ MarkedSpace::Subspace& Heap::subspaceFor
> }
> 
> template<typename ClassType>
>-MarkedAllocator& Heap::allocatorForObjectOfType(size_t bytes)
>+inline MarkedAllocator* Heap::allocatorForObjectOfType(size_t bytes)
> {
>     // JSCell::classInfo() expects objects allocated with normal destructor to derive from JSDestructibleObject.
>     ASSERT((!ClassType::needsDestruction || (ClassType::StructureFlags & StructureIsImmortal) || std::is_convertible<ClassType, JSDestructibleObject>::value));
>-    
>+
>+    MarkedAllocator* result;
>     if (ClassType::needsDestruction)
>-        return allocatorForObjectWithDestructor(bytes);
>-    return allocatorForObjectWithoutDestructor(bytes);
>+        result = allocatorForObjectWithDestructor(bytes);
>+    else
>+        result = allocatorForObjectWithoutDestructor(bytes);
>+    
>+    ASSERT(result || !ClassType::info()->isSubClassOf(JSCallee::info()));
>+    return result;
>+}
>+
>+inline void* Heap::allocateAuxiliary(JSCell* intendedOwner, size_t bytes)
>+{
>+    void* result = m_objectSpace.allocateAuxiliary(bytes);
>+#if ENABLE(ALLOCATION_LOGGING)
>+    dataLogF("JSC GC allocating %lu bytes of auxiliary for %p: %p.\n", bytes, intendedOwner, result);
>+#else
>+    UNUSED_PARAM(intendedOwner);
>+#endif
>+    return result;
>+}
>+
>+inline void* Heap::tryAllocateAuxiliary(JSCell* intendedOwner, size_t bytes)
>+{
>+    void* result = m_objectSpace.tryAllocateAuxiliary(bytes);
>+#if ENABLE(ALLOCATION_LOGGING)
>+    dataLogF("JSC GC allocating %lu bytes of auxiliary for %p: %p.\n", bytes, intendedOwner, result);
>+#else
>+    UNUSED_PARAM(intendedOwner);
>+#endif
>+    return result;
>+}
>+
>+inline void* Heap::tryReallocateAuxiliary(JSCell* intendedOwner, void* oldBase, size_t oldSize, size_t newSize)
>+{
>+    void* newBase = tryAllocateAuxiliary(intendedOwner, newSize);
>+    if (!newBase)
>+        return nullptr;
>+    memcpy(newBase, oldBase, oldSize);
>+    return newBase;
> }
> 
> inline CheckedBoolean Heap::tryAllocateStorage(JSCell* intendedOwner, size_t bytes, void** outPtr)
>@@ -354,33 +410,6 @@ inline void Heap::didFreeBlock(size_t ca
> #endif
> }
> 
>-inline bool Heap::isPointerGCObject(TinyBloomFilter filter, MarkedBlockSet& markedBlockSet, void* pointer)
>-{
>-    MarkedBlock* candidate = MarkedBlock::blockFor(pointer);
>-    if (filter.ruleOut(bitwise_cast<Bits>(candidate))) {
>-        ASSERT(!candidate || !markedBlockSet.set().contains(candidate));
>-        return false;
>-    }
>-
>-    if (!MarkedBlock::isAtomAligned(pointer))
>-        return false;
>-
>-    if (!markedBlockSet.set().contains(candidate))
>-        return false;
>-
>-    if (!candidate->isLiveCell(pointer))
>-        return false;
>-
>-    return true;
>-}
>-
>-inline bool Heap::isValueGCObject(TinyBloomFilter filter, MarkedBlockSet& markedBlockSet, JSValue value)
>-{
>-    if (!value.isCell())
>-        return false;
>-    return isPointerGCObject(filter, markedBlockSet, static_cast<void*>(value.asCell()));
>-}
>-
> } // namespace JSC
> 
> #endif // HeapInlines_h
>Index: Source/JavaScriptCore/heap/HeapUtil.h
>===================================================================
>--- Source/JavaScriptCore/heap/HeapUtil.h	(nonexistent)
>+++ Source/JavaScriptCore/heap/HeapUtil.h	(working copy)
>@@ -0,0 +1,186 @@
>+/*
>+ * Copyright (C) 2016 Apple Inc. All rights reserved.
>+ *
>+ * Redistribution and use in source and binary forms, with or without
>+ * modification, are permitted provided that the following conditions
>+ * are met:
>+ * 1. Redistributions of source code must retain the above copyright
>+ *    notice, this list of conditions and the following disclaimer.
>+ * 2. Redistributions in binary form must reproduce the above copyright
>+ *    notice, this list of conditions and the following disclaimer in the
>+ *    documentation and/or other materials provided with the distribution.
>+ *
>+ * THIS SOFTWARE IS PROVIDED BY APPLE INC. AND ITS CONTRIBUTORS ``AS IS''
>+ * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO,
>+ * THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
>+ * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL APPLE INC. OR ITS CONTRIBUTORS
>+ * BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
>+ * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
>+ * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
>+ * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
>+ * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
>+ * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF
>+ * THE POSSIBILITY OF SUCH DAMAGE.
>+ */
>+
>+#pragma once
>+
>+namespace JSC {
>+
>+// Are you tired of waiting for all of WebKit to build because you changed the implementation of a
>+// function in HeapInlines.h?  Does it bother you that you're waiting on rebuilding the JS DOM
>+// bindings even though your change is in a function called from only 2 .cpp files?  Then HeapUtil.h
>+// is for you!  Everything in this class should be a static method that takes a Heap& if needed.
>+// This is a friend of Heap, so you can access all of Heap's privates.
>+//
>+// This ends up being an issue because Heap exposes a lot of methods that ought to be inline for
>+// performance or that must be inline because they are templates.  This class ought to contain
>+// methods that are used for the implementation of the collector, or for unusual clients that need
>+// to reach deep into the collector for some reason.  Don't put things in here that would cause you
>+// to have to include it from more than a handful of places, since that would defeat the purpose.
>+// This class isn't here to look pretty.  It's to let us hack the GC more easily!
>+
>+class HeapUtil {
>+public:
>+    // This function must be run after stopAllocation() is called and 
>+    // before liveness data is cleared to be accurate.
>+    template<typename Func>
>+    static void findGCObjectPointersForMarking(
>+        Heap& heap, TinyBloomFilter filter, void* passedPointer,
>+        const Func& func)
>+    {
>+        const HashSet<MarkedBlock*>& set = heap.objectSpace().blocks().set();
>+        
>+        char* pointer = static_cast<char*>(passedPointer);
>+        
>+        // It could point to a large allocation.
>+        if (heap.objectSpace().largeAllocationsForThisCollectionSize()) {
>+            if (heap.objectSpace().largeAllocationsForThisCollectionBegin()[0]->aboveLowerBound(pointer)
>+                && heap.objectSpace().largeAllocationsForThisCollectionEnd()[-1]->belowUpperBound(pointer)) {
>+                LargeAllocation** result = approximateBinarySearch<LargeAllocation*>(
>+                    heap.objectSpace().largeAllocationsForThisCollectionBegin(),
>+                    heap.objectSpace().largeAllocationsForThisCollectionSize(),
>+                    LargeAllocation::fromCell(pointer),
>+                    [] (LargeAllocation** ptr) -> LargeAllocation* { return *ptr; });
>+                if (result) {
>+                    if (result > heap.objectSpace().largeAllocationsForThisCollectionBegin()
>+                        && result[-1]->contains(pointer))
>+                        func(result[-1]->cell());
>+                    if (result[0]->contains(pointer))
>+                        func(result[0]->cell());
>+                    if (result + 1 < heap.objectSpace().largeAllocationsForThisCollectionEnd()
>+                        && result[1]->contains(pointer))
>+                        func(result[1]->cell());
>+                }
>+            }
>+        }
>+    
>+        MarkedBlock* candidate = MarkedBlock::blockFor(pointer);
>+        // It's possible for a butterfly pointer to point past the end of a butterfly. Check this now.
>+        if (pointer <= bitwise_cast<char*>(candidate) + sizeof(IndexingHeader)) {
>+            // We may be interested in the last cell of the previous MarkedBlock.
>+            char* previousPointer = pointer - sizeof(IndexingHeader) - 1;
>+            MarkedBlock* previousCandidate = MarkedBlock::blockFor(previousPointer);
>+            if (!filter.ruleOut(bitwise_cast<Bits>(previousCandidate))
>+                && set.contains(previousCandidate)
>+                && previousCandidate->cellKind() == HeapCell::Auxiliary) {
>+                previousPointer = static_cast<char*>(previousCandidate->cellAlign(previousPointer));
>+                if (previousCandidate->isLiveCell(previousPointer))
>+                    func(previousPointer);
>+            }
>+        }
>+    
>+        if (filter.ruleOut(bitwise_cast<Bits>(candidate))) {
>+            ASSERT(!candidate || !set.contains(candidate));
>+            return;
>+        }
>+    
>+        if (!set.contains(candidate))
>+            return;
>+
>+        auto tryPointer = [&] (void* pointer) {
>+            if (candidate->isLiveCell(pointer))
>+                func(pointer);
>+        };
>+    
>+        if (candidate->cellKind() == HeapCell::JSCell) {
>+            if (!MarkedBlock::isAtomAligned(pointer))
>+                return;
>+        
>+            tryPointer(pointer);
>+            return;
>+        }
>+    
>+        // A butterfly could point into the middle of an object.
>+        char* alignedPointer = static_cast<char*>(candidate->cellAlign(pointer));
>+        tryPointer(alignedPointer);
>+    
>+        // Also, a butterfly could point at the end of an object plus sizeof(IndexingHeader). In that
>+        // case, this is pointing to the object to the right of the one we should be marking.
>+        if (candidate->atomNumber(alignedPointer) > MarkedBlock::firstAtom()
>+            && pointer <= alignedPointer + sizeof(IndexingHeader))
>+            tryPointer(alignedPointer - candidate->cellSize());
>+    }
>+    
>+    static bool isPointerGCObjectJSCell(
>+        Heap& heap, TinyBloomFilter filter, const void* pointer)
>+    {
>+        // It could point to a large allocation.
>+        const Vector<LargeAllocation*>& largeAllocations = heap.objectSpace().largeAllocations();
>+        if (!largeAllocations.isEmpty()) {
>+            if (largeAllocations[0]->aboveLowerBound(pointer)
>+                && largeAllocations.last()->belowUpperBound(pointer)) {
>+                LargeAllocation*const* result = approximateBinarySearch<LargeAllocation*const>(
>+                    largeAllocations.begin(), largeAllocations.size(),
>+                    LargeAllocation::fromCell(pointer),
>+                    [] (LargeAllocation*const* ptr) -> LargeAllocation* { return *ptr; });
>+                if (result) {
>+                    if (result > largeAllocations.begin()
>+                        && result[-1]->cell() == pointer
>+                        && result[-1]->attributes().cellKind == HeapCell::JSCell)
>+                        return true;
>+                    if (result[0]->cell() == pointer
>+                        && result[0]->attributes().cellKind == HeapCell::JSCell)
>+                        return true;
>+                    if (result + 1 < largeAllocations.end()
>+                        && result[1]->cell() == pointer
>+                        && result[1]->attributes().cellKind == HeapCell::JSCell)
>+                        return true;
>+                }
>+            }
>+        }
>+    
>+        const HashSet<MarkedBlock*>& set = heap.objectSpace().blocks().set();
>+        
>+        MarkedBlock* candidate = MarkedBlock::blockFor(pointer);
>+        if (filter.ruleOut(bitwise_cast<Bits>(candidate))) {
>+            ASSERT(!candidate || !set.contains(candidate));
>+            return false;
>+        }
>+        
>+        if (!MarkedBlock::isAtomAligned(pointer))
>+            return false;
>+        
>+        if (!set.contains(candidate))
>+            return false;
>+        
>+        if (candidate->cellKind() != HeapCell::JSCell)
>+            return false;
>+        
>+        if (!candidate->isLiveCell(pointer))
>+            return false;
>+        
>+        return true;
>+    }
>+    
>+    static bool isValueGCObject(
>+        Heap& heap, TinyBloomFilter filter, JSValue value)
>+    {
>+        if (!value.isCell())
>+            return false;
>+        return isPointerGCObjectJSCell(heap, filter, static_cast<void*>(value.asCell()));
>+    }
>+};
>+
>+} // namespace JSC
>+
>Index: Source/JavaScriptCore/heap/LargeAllocation.cpp
>===================================================================
>--- Source/JavaScriptCore/heap/LargeAllocation.cpp	(nonexistent)
>+++ Source/JavaScriptCore/heap/LargeAllocation.cpp	(working copy)
>@@ -0,0 +1,122 @@
>+/*
>+ * Copyright (C) 2016 Apple Inc. All rights reserved.
>+ *
>+ * Redistribution and use in source and binary forms, with or without
>+ * modification, are permitted provided that the following conditions
>+ * are met:
>+ * 1. Redistributions of source code must retain the above copyright
>+ *    notice, this list of conditions and the following disclaimer.
>+ * 2. Redistributions in binary form must reproduce the above copyright
>+ *    notice, this list of conditions and the following disclaimer in the
>+ *    documentation and/or other materials provided with the distribution.
>+ *
>+ * THIS SOFTWARE IS PROVIDED BY APPLE INC. ``AS IS'' AND ANY
>+ * EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
>+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
>+ * PURPOSE ARE DISCLAIMED.  IN NO EVENT SHALL APPLE INC. OR
>+ * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
>+ * EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
>+ * PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
>+ * PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY
>+ * OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
>+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
>+ * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 
>+ */
>+
>+#include "config.h"
>+#include "LargeAllocation.h"
>+
>+#include "Heap.h"
>+#include "JSCInlines.h"
>+#include "Operations.h"
>+
>+namespace JSC {
>+
>+LargeAllocation* LargeAllocation::tryCreate(Heap& heap, size_t size, const AllocatorAttributes& attributes)
>+{
>+    void* space = tryFastAlignedMalloc(alignment, headerSize() + size);
>+    if (!space)
>+        return nullptr;
>+    if (scribbleFreeCells())
>+        scribble(space, size);
>+    return new (NotNull, space) LargeAllocation(heap, size, attributes);
>+}
>+
>+LargeAllocation::LargeAllocation(Heap& heap, size_t size, const AllocatorAttributes& attributes)
>+    : m_cellSize(size)
>+    , m_isNewlyAllocated(true)
>+    , m_hasValidCell(true)
>+    , m_attributes(attributes)
>+    , m_weakSet(heap.vm(), *this)
>+{
>+    m_isMarked.store(0);
>+}
>+
>+void LargeAllocation::lastChanceToFinalize()
>+{
>+    m_weakSet.lastChanceToFinalize();
>+    clearMarksWithCollectionType<FullCollection>();
>+    sweep();
>+}
>+
>+void LargeAllocation::shrink()
>+{
>+    m_weakSet.shrink();
>+}
>+
>+void LargeAllocation::visitWeakSet(HeapRootVisitor& visitor)
>+{
>+    m_weakSet.visit(visitor);
>+}
>+
>+void LargeAllocation::reapWeakSet()
>+{
>+    return m_weakSet.reap();
>+}
>+
>+void LargeAllocation::clearMarks()
>+{
>+    if (heap()->operationInProgress() == JSC::EdenCollection)
>+        this->clearMarksWithCollectionType<EdenCollection>();
>+    else
>+        this->clearMarksWithCollectionType<FullCollection>();
>+}
>+
>+template <HeapOperation collectionType>
>+void LargeAllocation::clearMarksWithCollectionType()
>+{
>+    ASSERT(collectionType == FullCollection || collectionType == EdenCollection);
>+    
>+    if (collectionType == FullCollection)
>+        clearMarked();
>+}
>+
>+bool LargeAllocation::isEmpty()
>+{
>+    return !isMarked() && m_weakSet.isEmpty() && !isNewlyAllocated();
>+}
>+
>+void LargeAllocation::sweep()
>+{
>+    m_weakSet.sweep();
>+    
>+    if (m_hasValidCell && !isLive()) {
>+        if (m_attributes.destruction == NeedsDestruction)
>+            static_cast<JSCell*>(cell())->callDestructor(*vm());
>+        m_hasValidCell = false;
>+    }
>+}
>+
>+void LargeAllocation::destroy()
>+{
>+    this->~LargeAllocation();
>+    fastAlignedFree(this);
>+}
>+
>+void LargeAllocation::dump(PrintStream& out) const
>+{
>+    out.print(RawPointer(this), ":(cell at ", RawPointer(cell()), " with size ", m_cellSize, " and attributes ", m_attributes, ")");
>+}
>+
>+} // namespace JSC
>+
>Index: Source/JavaScriptCore/heap/LargeAllocation.h
>===================================================================
>--- Source/JavaScriptCore/heap/LargeAllocation.h	(nonexistent)
>+++ Source/JavaScriptCore/heap/LargeAllocation.h	(working copy)
>@@ -0,0 +1,152 @@
>+/*
>+ * Copyright (C) 2016 Apple Inc. All rights reserved.
>+ *
>+ * Redistribution and use in source and binary forms, with or without
>+ * modification, are permitted provided that the following conditions
>+ * are met:
>+ * 1. Redistributions of source code must retain the above copyright
>+ *    notice, this list of conditions and the following disclaimer.
>+ * 2. Redistributions in binary form must reproduce the above copyright
>+ *    notice, this list of conditions and the following disclaimer in the
>+ *    documentation and/or other materials provided with the distribution.
>+ *
>+ * THIS SOFTWARE IS PROVIDED BY APPLE INC. ``AS IS'' AND ANY
>+ * EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
>+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
>+ * PURPOSE ARE DISCLAIMED.  IN NO EVENT SHALL APPLE INC. OR
>+ * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
>+ * EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
>+ * PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
>+ * PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY
>+ * OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
>+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
>+ * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 
>+ */
>+
>+#pragma once
>+
>+#include "MarkedBlock.h"
>+#include "WeakSet.h"
>+
>+namespace JSC {
>+
>+// WebKit has a good malloc that already knows what to do for large allocations. The GC shouldn't
>+// have to think about such things. That's where LargeAllocation comes in. We will allocate large
>+// objects directly using malloc, and put the LargeAllocation header just before them. We can detect
>+// when a HeapCell* is a LargeAllocation because it will have the MarkedBlock::atomSize / 2 bit set.
>+
>+class LargeAllocation {
>+public:
>+    static LargeAllocation* tryCreate(Heap&, size_t, const AllocatorAttributes&);
>+    
>+    static LargeAllocation* fromCell(const void* cell)
>+    {
>+        return bitwise_cast<LargeAllocation*>(bitwise_cast<char*>(cell) - headerSize());
>+    }
>+    
>+    HeapCell* cell() const
>+    {
>+        return bitwise_cast<HeapCell*>(bitwise_cast<char*>(this) + headerSize());
>+    }
>+    
>+    static bool isLargeAllocation(HeapCell* cell)
>+    {
>+        return bitwise_cast<uintptr_t>(cell) & halfAlignment;
>+    }
>+    
>+    void lastChanceToFinalize();
>+    
>+    Heap* heap() const { return m_weakSet.heap(); }
>+    VM* vm() const { return m_weakSet.vm(); }
>+    WeakSet& weakSet() { return m_weakSet; }
>+    
>+    void shrink();
>+    
>+    void visitWeakSet(HeapRootVisitor&);
>+    void reapWeakSet();
>+    
>+    void clearNewlyAllocated() { m_isNewlyAllocated = false; }
>+    void clearMarks();
>+    template<HeapOperation collectionType>
>+    void clearMarksWithCollectionType();
>+    
>+    bool isNewlyAllocated() const { return m_isNewlyAllocated; }
>+    ALWAYS_INLINE bool isMarked() { return m_isMarked.load(std::memory_order_relaxed); }
>+    bool isMarkedOrNewlyAllocated() { return isMarked() || isNewlyAllocated(); }
>+    bool isLive() { return isMarkedOrNewlyAllocated(); }
>+    
>+    bool hasValidCell() const { return m_hasValidCell; }
>+    
>+    bool isEmpty();
>+    
>+    size_t cellSize() const { return m_cellSize; }
>+    
>+    bool aboveLowerBound(const void* rawPtr)
>+    {
>+        char* ptr = bitwise_cast<char*>(rawPtr);
>+        char* begin = bitwise_cast<char*>(cell());
>+        return ptr >= begin;
>+    }
>+    
>+    bool belowUpperBound(const void* rawPtr)
>+    {
>+        char* ptr = bitwise_cast<char*>(rawPtr);
>+        char* begin = bitwise_cast<char*>(cell());
>+        char* end = begin + cellSize();
>+        // We cannot #include IndexingHeader.h because reasons. The fact that IndexingHeader is 8
>+        // bytes is wired deep into our engine, so this isn't so bad.
>+        size_t sizeOfIndexingHeader = 8;
>+        return ptr <= end + sizeOfIndexingHeader;
>+    }
>+    
>+    bool contains(const void* rawPtr)
>+    {
>+        return aboveLowerBound(rawPtr) && belowUpperBound(rawPtr);
>+    }
>+    
>+    const AllocatorAttributes& attributes() const { return m_attributes; }
>+    
>+    ALWAYS_INLINE bool testAndSetMarked()
>+    {
>+        // This method is usually called when the object is already marked. This avoids us
>+        // having to CAS in that case. It's profitable to reduce the total amount of CAS
>+        // traffic.
>+        if (isMarked())
>+            return true;
>+        return !m_isMarked.compareExchangeStrong(false, true);
>+    }
>+    ALWAYS_INLINE bool testAndSetMarked(HeapCell*) { return testAndSetMarked(); }
>+    void setMarked() { m_isMarked.store(true); }
>+    void clearMarked() { m_isMarked.store(false); }
>+    
>+    void setHasAnyMarked() { }
>+    
>+    void sweep();
>+    
>+    void destroy();
>+    
>+    void dump(PrintStream&) const;
>+    
>+private:
>+    LargeAllocation(Heap&, size_t, const AllocatorAttributes&);
>+    
>+    static const unsigned alignment = MarkedBlock::atomSize;
>+    static const unsigned halfAlignment = alignment / 2;
>+
>+    static unsigned headerSize();
>+    
>+    size_t m_cellSize;
>+    bool m_isNewlyAllocated;
>+    bool m_hasValidCell;
>+    Atomic<bool> m_isMarked;
>+    AllocatorAttributes m_attributes;
>+    WeakSet m_weakSet;
>+};
>+
>+inline unsigned LargeAllocation::headerSize()
>+{
>+    return ((sizeof(LargeAllocation) + halfAlignment - 1) & ~(halfAlignment - 1)) | halfAlignment;
>+}
>+
>+} // namespace JSC
>+
>Index: Source/JavaScriptCore/heap/MarkedAllocator.cpp
>===================================================================
>--- Source/JavaScriptCore/heap/MarkedAllocator.cpp	(revision 204735)
>+++ Source/JavaScriptCore/heap/MarkedAllocator.cpp	(working copy)
>@@ -30,11 +30,23 @@
> #include "Heap.h"
> #include "IncrementalSweeper.h"
> #include "JSCInlines.h"
>+#include "SuperSampler.h"
> #include "VM.h"
> #include <wtf/CurrentTime.h>
> 
> namespace JSC {
> 
>+MarkedAllocator::MarkedAllocator(Heap* heap, MarkedSpace* markedSpace, size_t cellSize, const AllocatorAttributes& attributes)
>+    : m_currentBlock(0)
>+    , m_lastActiveBlock(0)
>+    , m_nextBlockToSweep(0)
>+    , m_cellSize(static_cast<unsigned>(cellSize))
>+    , m_attributes(attributes)
>+    , m_heap(heap)
>+    , m_markedSpace(markedSpace)
>+{
>+}
>+
> static bool isListPagedOut(double deadline, DoublyLinkedList<MarkedBlock>& list)
> {
>     unsigned itersSinceLastTimeCheck = 0;
>@@ -59,30 +71,31 @@ bool MarkedAllocator::isPagedOut(double 
>     return false;
> }
> 
>-void MarkedAllocator::retire(MarkedBlock* block, MarkedBlock::FreeList& freeList)
>+void MarkedAllocator::retire(MarkedBlock* block, FreeList& freeList)
> {
>     m_blockList.remove(block);
>     m_retiredBlocks.push(block);
>     block->didRetireBlock(freeList);
> }
> 
>-inline void* MarkedAllocator::tryAllocateHelper(size_t bytes)
>+inline void* MarkedAllocator::tryAllocateWithoutCollectingImpl(size_t bytes)
> {
>     if (m_currentBlock) {
>         ASSERT(m_currentBlock == m_nextBlockToSweep);
>         m_currentBlock->didConsumeFreeList();
>         m_nextBlockToSweep = m_currentBlock->next();
>     }
>+    
>+    setFreeList(FreeList());
> 
>     MarkedBlock* next;
>     for (MarkedBlock*& block = m_nextBlockToSweep; block; block = next) {
>         next = block->next();
> 
>-        MarkedBlock::FreeList freeList = block->sweep(MarkedBlock::SweepToFreeList);
>+        FreeList freeList = block->sweep(MarkedBlock::SweepToFreeList);
>         
>-        double utilization = ((double)MarkedBlock::blockSize - (double)freeList.bytes) / (double)MarkedBlock::blockSize;
>+        double utilization = ((double)MarkedBlock::blockSize - (double)freeList.originalSize) / (double)MarkedBlock::blockSize;
>         if (utilization >= Options::minMarkedBlockUtilization()) {
>-            ASSERT(freeList.bytes || !freeList.head);
>             retire(block, freeList);
>             continue;
>         }
>@@ -91,40 +104,37 @@ inline void* MarkedAllocator::tryAllocat
>             block->stopAllocating(freeList);
>             continue;
>         }
>-
>+        
>         m_currentBlock = block;
>-        m_freeList = freeList;
>+        setFreeList(freeList);
>         break;
>     }
>     
>-    if (!m_freeList.head) {
>+    if (!m_freeList) {
>         m_currentBlock = 0;
>         return 0;
>     }
> 
>-    ASSERT(m_freeList.head);
>-    void* head = tryPopFreeList(bytes);
>-    ASSERT(head);
>+    void* result;
>+    if (m_freeList.remaining) {
>+        unsigned cellSize = m_cellSize;
>+        m_freeList.remaining -= cellSize;
>+        result = m_freeList.payloadEnd - m_freeList.remaining - cellSize;
>+    } else {
>+        FreeCell* head = m_freeList.head;
>+        m_freeList.head = head->next;
>+        result = head;
>+    }
>+    RELEASE_ASSERT(result);
>     m_markedSpace->didAllocateInBlock(m_currentBlock);
>-    return head;
>-}
>-
>-inline void* MarkedAllocator::tryPopFreeList(size_t bytes)
>-{
>-    ASSERT(m_currentBlock);
>-    if (bytes > m_currentBlock->cellSize())
>-        return 0;
>-
>-    MarkedBlock::FreeCell* head = m_freeList.head;
>-    m_freeList.head = head->next;
>-    return head;
>+    return result;
> }
> 
>-inline void* MarkedAllocator::tryAllocate(size_t bytes)
>+inline void* MarkedAllocator::tryAllocateWithoutCollecting(size_t bytes)
> {
>     ASSERT(!m_heap->isBusy());
>     m_heap->m_operationInProgress = Allocation;
>-    void* result = tryAllocateHelper(bytes);
>+    void* result = tryAllocateWithoutCollectingImpl(bytes);
> 
>     m_heap->m_operationInProgress = NoOperation;
>     ASSERT(result || !m_currentBlock);
>@@ -148,51 +158,73 @@ ALWAYS_INLINE void MarkedAllocator::doTe
> 
> void* MarkedAllocator::allocateSlowCase(size_t bytes)
> {
>+    bool crashOnFailure = true;
>+    return allocateSlowCaseImpl(bytes, crashOnFailure);
>+}
>+
>+void* MarkedAllocator::tryAllocateSlowCase(size_t bytes)
>+{
>+    bool crashOnFailure = false;
>+    return allocateSlowCaseImpl(bytes, crashOnFailure);
>+}
>+
>+void* MarkedAllocator::allocateSlowCaseImpl(size_t bytes, bool crashOnFailure)
>+{
>     ASSERT(m_heap->vm()->currentThreadIsHoldingAPILock());
>     doTestCollectionsIfNeeded();
> 
>     ASSERT(!m_markedSpace->isIterating());
>-    ASSERT(!m_freeList.head);
>-    m_heap->didAllocate(m_freeList.bytes);
>+    m_heap->didAllocate(m_freeList.originalSize);
>     
>-    void* result = tryAllocate(bytes);
>+    void* result = tryAllocateWithoutCollecting(bytes);
>     
>     if (LIKELY(result != 0))
>         return result;
>     
>     if (m_heap->collectIfNecessaryOrDefer()) {
>-        result = tryAllocate(bytes);
>+        result = tryAllocateWithoutCollecting(bytes);
>         if (result)
>             return result;
>     }
> 
>     ASSERT(!m_heap->shouldCollect());
>     
>-    MarkedBlock* block = allocateBlock(bytes);
>-    ASSERT(block);
>+    MarkedBlock* block = tryAllocateBlock();
>+    if (!block) {
>+        if (crashOnFailure)
>+            RELEASE_ASSERT_NOT_REACHED();
>+        else
>+            return nullptr;
>+    }
>     addBlock(block);
>         
>-    result = tryAllocate(bytes);
>+    result = tryAllocateWithoutCollecting(bytes);
>     ASSERT(result);
>     return result;
> }
> 
>-MarkedBlock* MarkedAllocator::allocateBlock(size_t bytes)
>+static size_t blockHeaderSize()
>+{
>+    return WTF::roundUpToMultipleOf<MarkedBlock::atomSize>(sizeof(MarkedBlock));
>+}
>+
>+size_t MarkedAllocator::blockSizeForBytes(size_t bytes)
> {
>     size_t minBlockSize = MarkedBlock::blockSize;
>-    size_t minAllocationSize = WTF::roundUpToMultipleOf<MarkedBlock::atomSize>(sizeof(MarkedBlock)) + WTF::roundUpToMultipleOf<MarkedBlock::atomSize>(bytes);
>+    size_t minAllocationSize = blockHeaderSize() + WTF::roundUpToMultipleOf<MarkedBlock::atomSize>(bytes);
>     minAllocationSize = WTF::roundUpToMultipleOf(WTF::pageSize(), minAllocationSize);
>-    size_t blockSize = std::max(minBlockSize, minAllocationSize);
>-
>-    size_t cellSize = m_cellSize ? m_cellSize : WTF::roundUpToMultipleOf<MarkedBlock::atomSize>(bytes);
>+    return std::max(minBlockSize, minAllocationSize);
>+}
> 
>-    return MarkedBlock::create(*m_heap, this, blockSize, cellSize, m_attributes);
>+MarkedBlock* MarkedAllocator::tryAllocateBlock()
>+{
>+    return MarkedBlock::tryCreate(*m_heap, this, MarkedBlock::blockSize, m_cellSize, m_attributes);
> }
> 
> void MarkedAllocator::addBlock(MarkedBlock* block)
> {
>     ASSERT(!m_currentBlock);
>-    ASSERT(!m_freeList.head);
>+    ASSERT(!m_freeList);
>     
>     m_blockList.append(block);
>     m_nextBlockToSweep = block;
>@@ -203,7 +235,7 @@ void MarkedAllocator::removeBlock(Marked
> {
>     if (m_currentBlock == block) {
>         m_currentBlock = m_currentBlock->next();
>-        m_freeList = MarkedBlock::FreeList();
>+        setFreeList(FreeList());
>     }
>     if (m_nextBlockToSweep == block)
>         m_nextBlockToSweep = m_nextBlockToSweep->next();
>@@ -216,7 +248,7 @@ void MarkedAllocator::reset()
> {
>     m_lastActiveBlock = 0;
>     m_currentBlock = 0;
>-    m_freeList = MarkedBlock::FreeList();
>+    setFreeList(FreeList());
>     if (m_heap->operationInProgress() == FullCollection)
>         m_blockList.append(m_retiredBlocks);
> 
>@@ -227,7 +259,7 @@ void MarkedAllocator::reset()
>         for (MarkedBlock*& block = m_nextBlockToSweep; block; block = next) {
>             next = block->next();
> 
>-            MarkedBlock::FreeList freeList = block->sweep(MarkedBlock::SweepToFreeList);
>+            FreeList freeList = block->sweep(MarkedBlock::SweepToFreeList);
>             retire(block, freeList);
>         }
>     }
>@@ -242,4 +274,9 @@ void MarkedAllocator::lastChanceToFinali
>         });
> }
> 
>+void MarkedAllocator::setFreeList(const FreeList& freeList)
>+{
>+    m_freeList = freeList;
>+}
>+
> } // namespace JSC
>Index: Source/JavaScriptCore/heap/MarkedAllocator.h
>===================================================================
>--- Source/JavaScriptCore/heap/MarkedAllocator.h	(revision 204735)
>+++ Source/JavaScriptCore/heap/MarkedAllocator.h	(working copy)
>@@ -27,6 +27,7 @@
> #define MarkedAllocator_h
> 
> #include "AllocatorAttributes.h"
>+#include "FreeList.h"
> #include "MarkedBlock.h"
> #include <wtf/DoublyLinkedList.h>
> 
>@@ -40,9 +41,10 @@ class MarkedAllocator {
>     friend class LLIntOffsetsExtractor;
> 
> public:
>-    static ptrdiff_t offsetOfFreeListHead();
>+    static ptrdiff_t offsetOfFreeList();
>+    static ptrdiff_t offsetOfCellSize();
> 
>-    MarkedAllocator();
>+    MarkedAllocator(Heap*, MarkedSpace*, size_t cellSize, const AllocatorAttributes&);
>     void lastChanceToFinalize();
>     void reset();
>     void stopAllocating();
>@@ -53,6 +55,7 @@ public:
>     DestructionMode destruction() const { return m_attributes.destruction; }
>     HeapCell::Kind cellKind() const { return m_attributes.cellKind; }
>     void* allocate(size_t);
>+    void* tryAllocate(size_t);
>     Heap* heap() { return m_heap; }
>     MarkedBlock* takeLastActiveBlock()
>     {
>@@ -65,69 +68,78 @@ public:
>     
>     void addBlock(MarkedBlock*);
>     void removeBlock(MarkedBlock*);
>-    void init(Heap*, MarkedSpace*, size_t cellSize, const AllocatorAttributes&);
> 
>     bool isPagedOut(double deadline);
>+    
>+    static size_t blockSizeForBytes(size_t);
>    
> private:
>     JS_EXPORT_PRIVATE void* allocateSlowCase(size_t);
>-    void* tryAllocate(size_t);
>-    void* tryAllocateHelper(size_t);
>-    void* tryPopFreeList(size_t);
>-    MarkedBlock* allocateBlock(size_t);
>+    JS_EXPORT_PRIVATE void* tryAllocateSlowCase(size_t);
>+    void* allocateSlowCaseImpl(size_t, bool crashOnFailure);
>+    void* tryAllocateWithoutCollecting(size_t);
>+    void* tryAllocateWithoutCollectingImpl(size_t);
>+    MarkedBlock* tryAllocateBlock();
>     ALWAYS_INLINE void doTestCollectionsIfNeeded();
>-    void retire(MarkedBlock*, MarkedBlock::FreeList&);
>+    void retire(MarkedBlock*, FreeList&);
>     
>-    MarkedBlock::FreeList m_freeList;
>+    void setFreeList(const FreeList&);
>+    
>+    FreeList m_freeList;
>     MarkedBlock* m_currentBlock;
>     MarkedBlock* m_lastActiveBlock;
>     MarkedBlock* m_nextBlockToSweep;
>     DoublyLinkedList<MarkedBlock> m_blockList;
>     DoublyLinkedList<MarkedBlock> m_retiredBlocks;
>-    size_t m_cellSize;
>+    unsigned m_cellSize;
>     AllocatorAttributes m_attributes;
>     Heap* m_heap;
>     MarkedSpace* m_markedSpace;
> };
> 
>-inline ptrdiff_t MarkedAllocator::offsetOfFreeListHead()
>+inline ptrdiff_t MarkedAllocator::offsetOfFreeList()
> {
>-    return OBJECT_OFFSETOF(MarkedAllocator, m_freeList) + OBJECT_OFFSETOF(MarkedBlock::FreeList, head);
>+    return OBJECT_OFFSETOF(MarkedAllocator, m_freeList);
> }
> 
>-inline MarkedAllocator::MarkedAllocator()
>-    : m_currentBlock(0)
>-    , m_lastActiveBlock(0)
>-    , m_nextBlockToSweep(0)
>-    , m_cellSize(0)
>-    , m_heap(0)
>-    , m_markedSpace(0)
>+inline ptrdiff_t MarkedAllocator::offsetOfCellSize()
> {
>+    return OBJECT_OFFSETOF(MarkedAllocator, m_cellSize);
> }
> 
>-inline void MarkedAllocator::init(Heap* heap, MarkedSpace* markedSpace, size_t cellSize, const AllocatorAttributes& attributes)
>+ALWAYS_INLINE void* MarkedAllocator::tryAllocate(size_t bytes)
> {
>-    m_heap = heap;
>-    m_markedSpace = markedSpace;
>-    m_cellSize = cellSize;
>-    m_attributes = attributes;
>+    unsigned remaining = m_freeList.remaining;
>+    if (remaining) {
>+        unsigned cellSize = m_cellSize;
>+        remaining -= cellSize;
>+        m_freeList.remaining = remaining;
>+        return m_freeList.payloadEnd - remaining - cellSize;
>+    }
>+    
>+    FreeCell* head = m_freeList.head;
>+    if (UNLIKELY(!head))
>+        return tryAllocateSlowCase(bytes);
>+    
>+    m_freeList.head = head->next;
>+    return head;
> }
> 
>-inline void* MarkedAllocator::allocate(size_t bytes)
>+ALWAYS_INLINE void* MarkedAllocator::allocate(size_t bytes)
> {
>-    MarkedBlock::FreeCell* head = m_freeList.head;
>-    if (UNLIKELY(!head)) {
>-        void* result = allocateSlowCase(bytes);
>-#ifndef NDEBUG
>-        memset(result, 0xCD, bytes);
>-#endif
>-        return result;
>+    unsigned remaining = m_freeList.remaining;
>+    if (remaining) {
>+        unsigned cellSize = m_cellSize;
>+        remaining -= cellSize;
>+        m_freeList.remaining = remaining;
>+        return m_freeList.payloadEnd - remaining - cellSize;
>     }
>     
>+    FreeCell* head = m_freeList.head;
>+    if (UNLIKELY(!head))
>+        return allocateSlowCase(bytes);
>+    
>     m_freeList.head = head->next;
>-#ifndef NDEBUG
>-    memset(head, 0xCD, bytes);
>-#endif
>     return head;
> }
> 
>@@ -135,14 +147,14 @@ inline void MarkedAllocator::stopAllocat
> {
>     ASSERT(!m_lastActiveBlock);
>     if (!m_currentBlock) {
>-        ASSERT(!m_freeList.head);
>+        ASSERT(!m_freeList);
>         return;
>     }
>     
>     m_currentBlock->stopAllocating(m_freeList);
>     m_lastActiveBlock = m_currentBlock;
>     m_currentBlock = 0;
>-    m_freeList = MarkedBlock::FreeList();
>+    m_freeList = FreeList();
> }
> 
> inline void MarkedAllocator::resumeAllocating()
>Index: Source/JavaScriptCore/heap/MarkedBlock.cpp
>===================================================================
>--- Source/JavaScriptCore/heap/MarkedBlock.cpp	(revision 204735)
>+++ Source/JavaScriptCore/heap/MarkedBlock.cpp	(working copy)
>@@ -29,20 +29,26 @@
> #include "JSCell.h"
> #include "JSDestructibleObject.h"
> #include "JSCInlines.h"
>+#include "SuperSampler.h"
> 
> namespace JSC {
> 
> static const bool computeBalance = false;
> static size_t balance;
> 
>-MarkedBlock* MarkedBlock::create(Heap& heap, MarkedAllocator* allocator, size_t capacity, size_t cellSize, const AllocatorAttributes& attributes)
>+MarkedBlock* MarkedBlock::tryCreate(Heap& heap, MarkedAllocator* allocator, size_t capacity, size_t cellSize, const AllocatorAttributes& attributes)
> {
>     if (computeBalance) {
>         balance++;
>         if (!(balance % 10))
>             dataLog("MarkedBlock Balance: ", balance, "\n");
>     }
>-    MarkedBlock* block = new (NotNull, fastAlignedMalloc(blockSize, capacity)) MarkedBlock(allocator, capacity, cellSize, attributes);
>+    void* blockSpace = tryFastAlignedMalloc(blockSize, capacity);
>+    if (!blockSpace)
>+        return nullptr;
>+    if (scribbleFreeCells())
>+        scribble(blockSpace, capacity);
>+    MarkedBlock* block = new (NotNull, blockSpace) MarkedBlock(allocator, capacity, cellSize, attributes);
>     heap.didAllocateBlock(capacity);
>     return block;
> }
>@@ -63,11 +69,11 @@ void MarkedBlock::destroy(Heap& heap, Ma
> MarkedBlock::MarkedBlock(MarkedAllocator* allocator, size_t capacity, size_t cellSize, const AllocatorAttributes& attributes)
>     : DoublyLinkedListNode<MarkedBlock>()
>     , m_atomsPerCell((cellSize + atomSize - 1) / atomSize)
>-    , m_endAtom((allocator->cellSize() ? atomsPerBlock - m_atomsPerCell : firstAtom()) + 1)
>+    , m_endAtom(atomsPerBlock - m_atomsPerCell + 1)
>     , m_capacity(capacity)
>     , m_attributes(attributes)
>-    , m_allocator(allocator)
>     , m_state(New) // All cells start out unmarked.
>+    , m_allocator(allocator)
>     , m_weakSet(allocator->heap()->vm(), *this)
> {
>     ASSERT(allocator);
>@@ -76,27 +82,32 @@ MarkedBlock::MarkedBlock(MarkedAllocator
>         RELEASE_ASSERT(m_attributes.destruction == DoesNotNeedDestruction);
> }
> 
>-inline void MarkedBlock::callDestructor(HeapCell* cell)
>+template<MarkedBlock::BlockState blockState, MarkedBlock::SweepMode sweepMode, DestructionMode destructionMode, MarkedBlock::ScribbleMode scribbleMode, MarkedBlock::NewlyAllocatedMode newlyAllocatedMode>
>+FreeList MarkedBlock::specializedSweep()
> {
>-    // A previous eager sweep may already have run cell's destructor.
>-    if (cell->isZapped())
>-        return;
>+    SuperSamplerScope superSamplerScope(false);
>+    ASSERT(blockState == New || blockState == Marked);
>+    ASSERT(!(destructionMode == DoesNotNeedDestruction && sweepMode == SweepOnly));
>     
>-    JSCell* jsCell = static_cast<JSCell*>(cell);
>-
>-    ASSERT(jsCell->structureID());
>-    if (jsCell->inlineTypeFlags() & StructureIsImmortal)
>-        jsCell->structure(*vm())->classInfo()->methodTable.destroy(jsCell);
>-    else
>-        jsCast<JSDestructibleObject*>(jsCell)->classInfo()->methodTable.destroy(jsCell);
>-    cell->zap();
>-}
>-
>-template<MarkedBlock::BlockState blockState, MarkedBlock::SweepMode sweepMode, bool callDestructors>
>-MarkedBlock::FreeList MarkedBlock::specializedSweep()
>-{
>-    ASSERT(blockState != Allocated && blockState != FreeListed);
>-    ASSERT(!(!callDestructors && sweepMode == SweepOnly));
>+    bool isNewBlock = blockState == New;
>+    bool isEmptyBlock = !m_hasAnyMarked
>+        && newlyAllocatedMode == DoesNotHaveNewlyAllocated
>+        && destructionMode == DoesNotNeedDestruction;
>+    if (Options::useBumpAllocator() && (isNewBlock || isEmptyBlock)) {
>+        ASSERT(m_marks.isEmpty());
>+        
>+        char* startOfLastCell = static_cast<char*>(cellAlign(atoms() + m_endAtom - 1));
>+        char* payloadEnd = startOfLastCell + cellSize();
>+        RELEASE_ASSERT(payloadEnd - MarkedBlock::blockSize <= bitwise_cast<char*>(this));
>+        char* payloadBegin = bitwise_cast<char*>(atoms() + firstAtom());
>+        if (scribbleMode == Scribble)
>+            scribble(payloadBegin, payloadEnd - payloadBegin);
>+        m_state = ((sweepMode == SweepToFreeList) ? FreeListed : Marked);
>+        FreeList result = FreeList::bump(payloadEnd, payloadEnd - payloadBegin);
>+        if (false)
>+            dataLog("Quickly swept block ", RawPointer(this), " with cell size ", cellSize(), " and attributes ", m_attributes, ": ", result, "\n");
>+        return result;
>+    }
> 
>     // This produces a free list that is ordered in reverse through the block.
>     // This is fine, since the allocation code makes no assumptions about the
>@@ -104,16 +115,20 @@ MarkedBlock::FreeList MarkedBlock::speci
>     FreeCell* head = 0;
>     size_t count = 0;
>     for (size_t i = firstAtom(); i < m_endAtom; i += m_atomsPerCell) {
>-        if (blockState == Marked && (m_marks.get(i) || (m_newlyAllocated && m_newlyAllocated->get(i))))
>+        if (blockState == Marked
>+            && (m_marks.get(i)
>+                || (newlyAllocatedMode == HasNewlyAllocated && m_newlyAllocated->get(i))))
>             continue;
> 
>         HeapCell* cell = reinterpret_cast_ptr<HeapCell*>(&atoms()[i]);
> 
>-        if (callDestructors && blockState != New)
>-            callDestructor(cell);
>+        if (destructionMode == NeedsDestruction && blockState != New)
>+            static_cast<JSCell*>(cell)->callDestructor(*vm());
> 
>         if (sweepMode == SweepToFreeList) {
>             FreeCell* freeCell = reinterpret_cast<FreeCell*>(cell);
>+            if (scribbleMode == Scribble)
>+                scribble(freeCell, cellSize());
>             freeCell->next = head;
>             head = freeCell;
>             ++count;
>@@ -122,14 +137,17 @@ MarkedBlock::FreeList MarkedBlock::speci
> 
>     // We only want to discard the newlyAllocated bits if we're creating a FreeList,
>     // otherwise we would lose information on what's currently alive.
>-    if (sweepMode == SweepToFreeList && m_newlyAllocated)
>+    if (sweepMode == SweepToFreeList && newlyAllocatedMode == HasNewlyAllocated)
>         m_newlyAllocated = nullptr;
> 
>     m_state = ((sweepMode == SweepToFreeList) ? FreeListed : Marked);
>-    return FreeList(head, count * cellSize());
>+    FreeList result = FreeList::list(head, count * cellSize());
>+    if (false)
>+        dataLog("Slowly swept block ", RawPointer(this), " with cell size ", cellSize(), " and attributes ", m_attributes, ": ", result, "\n");
>+    return result;
> }
> 
>-MarkedBlock::FreeList MarkedBlock::sweep(SweepMode sweepMode)
>+FreeList MarkedBlock::sweep(SweepMode sweepMode)
> {
>     HEAP_LOG_BLOCK_STATE_TRANSITION(this);
> 
>@@ -139,17 +157,25 @@ MarkedBlock::FreeList MarkedBlock::sweep
>         return FreeList();
> 
>     if (m_attributes.destruction == NeedsDestruction)
>-        return sweepHelper<true>(sweepMode);
>-    return sweepHelper<false>(sweepMode);
>+        return sweepHelperSelectScribbleMode<NeedsDestruction>(sweepMode);
>+    return sweepHelperSelectScribbleMode<DoesNotNeedDestruction>(sweepMode);
> }
> 
>-template<bool callDestructors>
>-MarkedBlock::FreeList MarkedBlock::sweepHelper(SweepMode sweepMode)
>+template<DestructionMode destructionMode>
>+FreeList MarkedBlock::sweepHelperSelectScribbleMode(SweepMode sweepMode)
>+{
>+    if (scribbleFreeCells())
>+        return sweepHelperSelectStateAndSweepMode<destructionMode, Scribble>(sweepMode);
>+    return sweepHelperSelectStateAndSweepMode<destructionMode, DontScribble>(sweepMode);
>+}
>+
>+template<DestructionMode destructionMode, MarkedBlock::ScribbleMode scribbleMode>
>+FreeList MarkedBlock::sweepHelperSelectStateAndSweepMode(SweepMode sweepMode)
> {
>     switch (m_state) {
>     case New:
>         ASSERT(sweepMode == SweepToFreeList);
>-        return specializedSweep<New, SweepToFreeList, callDestructors>();
>+        return specializedSweep<New, SweepToFreeList, destructionMode, scribbleMode, DoesNotHaveNewlyAllocated>();
>     case FreeListed:
>         // Happens when a block transitions to fully allocated.
>         ASSERT(sweepMode == SweepToFreeList);
>@@ -159,9 +185,15 @@ MarkedBlock::FreeList MarkedBlock::sweep
>         RELEASE_ASSERT_NOT_REACHED();
>         return FreeList();
>     case Marked:
>-        return sweepMode == SweepToFreeList
>-            ? specializedSweep<Marked, SweepToFreeList, callDestructors>()
>-            : specializedSweep<Marked, SweepOnly, callDestructors>();
>+        if (m_newlyAllocated) {
>+            return sweepMode == SweepToFreeList
>+                ? specializedSweep<Marked, SweepToFreeList, destructionMode, scribbleMode, HasNewlyAllocated>()
>+                : specializedSweep<Marked, SweepOnly, destructionMode, scribbleMode, HasNewlyAllocated>();
>+        } else {
>+            return sweepMode == SweepToFreeList
>+                ? specializedSweep<Marked, SweepToFreeList, destructionMode, scribbleMode, DoesNotHaveNewlyAllocated>()
>+                : specializedSweep<Marked, SweepOnly, destructionMode, scribbleMode, DoesNotHaveNewlyAllocated>();
>+        }
>     }
>     RELEASE_ASSERT_NOT_REACHED();
>     return FreeList();
>@@ -188,7 +220,6 @@ private:
> void MarkedBlock::stopAllocating(const FreeList& freeList)
> {
>     HEAP_LOG_BLOCK_STATE_TRANSITION(this);
>-    FreeCell* head = freeList.head;
> 
>     if (m_state == Marked) {
>         // If the block is in the Marked state then we know that:
>@@ -196,11 +227,10 @@ void MarkedBlock::stopAllocating(const F
>         // 2) It may have dead objects, and we only know them to be dead by the
>         //    fact that their mark bits are unset.
>         // Hence if the block is Marked we need to leave it Marked.
>-        
>-        ASSERT(!head);
>+        ASSERT(freeList.allocationWillFail());
>         return;
>     }
>-   
>+    
>     ASSERT(m_state == FreeListed);
>     
>     // Roll back to a coherent state for Heap introspection. Cells newly
>@@ -213,13 +243,13 @@ void MarkedBlock::stopAllocating(const F
>     SetNewlyAllocatedFunctor functor(this);
>     forEachCell(functor);
> 
>-    FreeCell* next;
>-    for (FreeCell* current = head; current; current = next) {
>-        next = current->next;
>-        if (m_attributes.destruction == NeedsDestruction)
>-            reinterpret_cast<HeapCell*>(current)->zap();
>-        clearNewlyAllocated(current);
>-    }
>+    forEachFreeCell(
>+        freeList,
>+        [&] (HeapCell* cell) {
>+            if (m_attributes.destruction == NeedsDestruction)
>+                cell->zap();
>+            clearNewlyAllocated(cell);
>+        });
>     
>     m_state = Marked;
> }
>@@ -241,6 +271,7 @@ void MarkedBlock::clearMarksWithCollecti
>     ASSERT(m_state != New && m_state != FreeListed);
>     if (collectionType == FullCollection) {
>         m_marks.clearAll();
>+        clearHasAnyMarked();
>         // This will become true at the end of the mark phase. We set it now to
>         // avoid an extra pass to do so later.
>         m_state = Marked;
>@@ -262,7 +293,7 @@ void MarkedBlock::lastChanceToFinalize()
>     sweep();
> }
> 
>-MarkedBlock::FreeList MarkedBlock::resumeAllocating()
>+FreeList MarkedBlock::resumeAllocating()
> {
>     HEAP_LOG_BLOCK_STATE_TRANSITION(this);
> 
>@@ -281,7 +312,6 @@ MarkedBlock::FreeList MarkedBlock::resum
> void MarkedBlock::didRetireBlock(const FreeList& freeList)
> {
>     HEAP_LOG_BLOCK_STATE_TRANSITION(this);
>-    FreeCell* head = freeList.head;
> 
>     // Currently we don't notify the Heap that we're giving up on this block. 
>     // The Heap might be able to make a better decision about how many bytes should 
>@@ -291,15 +321,30 @@ void MarkedBlock::didRetireBlock(const F
> 
>     // We need to zap the free list when retiring a block so that we don't try to destroy 
>     // previously destroyed objects when we re-sweep the block in the future.
>-    FreeCell* next;
>-    for (FreeCell* current = head; current; current = next) {
>-        next = current->next;
>-        if (m_attributes.destruction == NeedsDestruction)
>-            reinterpret_cast<HeapCell*>(current)->zap();
>-    }
>+    forEachFreeCell(
>+        freeList,
>+        [&] (HeapCell* cell) {
>+            if (m_attributes.destruction == NeedsDestruction)
>+                cell->zap();
>+        });
> 
>     ASSERT(m_state == FreeListed);
>     m_state = Retired;
> }
> 
>+template<typename Func>
>+void MarkedBlock::forEachFreeCell(const FreeList& freeList, const Func& func)
>+{
>+    if (freeList.remaining) {
>+        for (unsigned remaining = freeList.remaining; remaining; remaining -= cellSize())
>+            func(bitwise_cast<HeapCell*>(freeList.payloadEnd - remaining));
>+    } else {
>+        for (FreeCell* current = freeList.head; current;) {
>+            FreeCell* next = current->next;
>+            func(bitwise_cast<HeapCell*>(current));
>+            current = next;
>+        }
>+    }
>+}
>+
> } // namespace JSC
>Index: Source/JavaScriptCore/heap/MarkedBlock.h
>===================================================================
>--- Source/JavaScriptCore/heap/MarkedBlock.h	(revision 204735)
>+++ Source/JavaScriptCore/heap/MarkedBlock.h	(working copy)
>@@ -24,6 +24,7 @@
> 
> #include "AllocatorAttributes.h"
> #include "DestructionMode.h"
>+#include "FreeList.h"
> #include "HeapCell.h"
> #include "HeapOperation.h"
> #include "IterationStatus.h"
>@@ -70,7 +71,7 @@ namespace JSC {
>         friend struct VerifyMarkedOrRetired;
>     public:
>         static const size_t atomSize = 16; // bytes
>-        static const size_t blockSize = 16 * KB;
>+        static const size_t blockSize = 64 * KB;
>         static const size_t blockMask = ~(blockSize - 1); // blockSize must be a power of two.
> 
>         static const size_t atomsPerBlock = blockSize / atomSize;
>@@ -78,18 +79,6 @@ namespace JSC {
>         static_assert(!(MarkedBlock::atomSize & (MarkedBlock::atomSize - 1)), "MarkedBlock::atomSize must be a power of two.");
>         static_assert(!(MarkedBlock::blockSize & (MarkedBlock::blockSize - 1)), "MarkedBlock::blockSize must be a power of two.");
> 
>-        struct FreeCell {
>-            FreeCell* next;
>-        };
>-        
>-        struct FreeList {
>-            FreeCell* head;
>-            size_t bytes;
>-
>-            FreeList();
>-            FreeList(FreeCell*, size_t);
>-        };
>-
>         struct VoidFunctor {
>             typedef void ReturnType;
>             void returnValue() { }
>@@ -109,12 +98,14 @@ namespace JSC {
>             mutable ReturnType m_count;
>         };
> 
>-        static MarkedBlock* create(Heap&, MarkedAllocator*, size_t capacity, size_t cellSize, const AllocatorAttributes&);
>+        static MarkedBlock* tryCreate(Heap&, MarkedAllocator*, size_t capacity, size_t cellSize, const AllocatorAttributes&);
>         static void destroy(Heap&, MarkedBlock*);
> 
>         static bool isAtomAligned(const void*);
>+        void* cellAlign(void*);
>         static MarkedBlock* blockFor(const void*);
>         static size_t firstAtom();
>+        size_t atomNumber(const void*);
>         
>         void lastChanceToFinalize();
> 
>@@ -175,6 +166,15 @@ namespace JSC {
>         bool needsSweeping() const;
>         void didRetireBlock(const FreeList&);
>         void willRemoveBlock();
>+        
>+        void setHasAnyMarked()
>+        {
>+            if (!m_hasAnyMarked) // Prevent store traffic if it's not needed.
>+                m_hasAnyMarked = true;
>+        }
>+        bool hasAnyMarked() const { return m_hasAnyMarked; }
>+        
>+        void clearHasAnyMarked() { m_hasAnyMarked = false; }
> 
>         template <typename Functor> IterationStatus forEachCell(const Functor&);
>         template <typename Functor> IterationStatus forEachLiveCell(const Functor&);
>@@ -183,27 +183,29 @@ namespace JSC {
>     private:
>         static const size_t atomAlignmentMask = atomSize - 1;
> 
>-        // During allocation, we look for available space in free lists in blocks.
>-        // If a block's utilization is sufficiently high (i.e. it's almost full),
>-        // we want to remove that block as a candidate for allocating to reduce
>-        // the likelihood of allocation having to take a slow path. When the
>-        // block is in this state, we say that it is "Retired".
>-        //
>-        // A full GC can take a Retired blocks out of retirement. An eden GC
>-        // will simply ignore Retired blocks (i.e. they will not be swept even
>-        // if they no longer have live objects).
>-
>-        enum BlockState { New, FreeListed, Allocated, Marked, Retired };
>-        template<bool callDestructors> FreeList sweepHelper(SweepMode = SweepOnly);
>+        enum BlockState : uint8_t { New, FreeListed, Allocated, Marked, Retired };
>+        
>+        template<DestructionMode>
>+        FreeList sweepHelperSelectScribbleMode(SweepMode = SweepOnly);
>+        
>+        enum ScribbleMode { DontScribble, Scribble };
>+        
>+        template<DestructionMode, ScribbleMode>
>+        FreeList sweepHelperSelectStateAndSweepMode(SweepMode = SweepOnly);
> 
>+        enum NewlyAllocatedMode { HasNewlyAllocated, DoesNotHaveNewlyAllocated };
>+        
>+        template<BlockState, SweepMode, DestructionMode, ScribbleMode, NewlyAllocatedMode>
>+        FreeList specializedSweep();
>+        
>         typedef char Atom[atomSize];
> 
>         MarkedBlock(MarkedAllocator*, size_t capacity, size_t cellSize, const AllocatorAttributes&);
>         Atom* atoms();
>-        size_t atomNumber(const void*);
>-        void callDestructor(HeapCell*);
>-        template<BlockState, SweepMode, bool callDestructors> FreeList specializedSweep();
>         
>+        template<typename Func>
>+        void forEachFreeCell(const FreeList&, const Func&);
>+
>         MarkedBlock* m_prev;
>         MarkedBlock* m_next;
> 
>@@ -214,23 +216,18 @@ namespace JSC {
> 
>         size_t m_capacity;
>         AllocatorAttributes m_attributes;
>-        MarkedAllocator* m_allocator;
>         BlockState m_state;
>+        bool m_hasAnyMarked { false };
>+        MarkedAllocator* m_allocator;
>         WeakSet m_weakSet;
>+        // FIXME: We "need" this to get 10% on Kraken/ai-astar. I'm pretty sure that's because we
>+        // lose the butterfly allocation if the subsequent cell allocation fails, during NewArray
>+        // and friends.
>+        // https://bugs.webkit.org/show_bug.cgi?id=160783
>+        uintptr_t m_padding1;
>+        uintptr_t m_padding2;
>     };
> 
>-    inline MarkedBlock::FreeList::FreeList()
>-        : head(0)
>-        , bytes(0)
>-    {
>-    }
>-
>-    inline MarkedBlock::FreeList::FreeList(FreeCell* head, size_t bytes)
>-        : head(head)
>-        , bytes(bytes)
>-    {
>-    }
>-
>     inline size_t MarkedBlock::firstAtom()
>     {
>         return WTF::roundUpToMultipleOf<atomSize>(sizeof(MarkedBlock)) / atomSize;
>@@ -246,6 +243,16 @@ namespace JSC {
>         return !(reinterpret_cast<Bits>(p) & atomAlignmentMask);
>     }
> 
>+    inline void* MarkedBlock::cellAlign(void* p)
>+    {
>+        Bits base = reinterpret_cast<Bits>(atoms() + firstAtom());
>+        Bits bits = reinterpret_cast<Bits>(p);
>+        bits -= base;
>+        bits -= bits % cellSize();
>+        bits += base;
>+        return reinterpret_cast<void*>(bits);
>+    }
>+
>     inline MarkedBlock* MarkedBlock::blockFor(const void* p)
>     {
>         return reinterpret_cast<MarkedBlock*>(reinterpret_cast<Bits>(p) & blockMask);
>@@ -306,7 +313,7 @@ namespace JSC {
> 
>     inline bool MarkedBlock::isEmpty()
>     {
>-        return m_marks.isEmpty() && m_weakSet.isEmpty() && (!m_newlyAllocated || m_newlyAllocated->isEmpty());
>+        return m_state == Marked && !m_hasAnyMarked && m_weakSet.isEmpty() && (!m_newlyAllocated || m_newlyAllocated->isEmpty());
>     }
> 
>     inline size_t MarkedBlock::cellSize()
>@@ -494,7 +501,7 @@ namespace JSC {
>     {
>         return m_state == Marked || m_state == Retired;
>     }
>-
>+        
> } // namespace JSC
> 
> namespace WTF {
>Index: Source/JavaScriptCore/heap/MarkedSpace.cpp
>===================================================================
>--- Source/JavaScriptCore/heap/MarkedSpace.cpp	(revision 204735)
>+++ Source/JavaScriptCore/heap/MarkedSpace.cpp	(working copy)
>@@ -24,17 +24,175 @@
> #include "IncrementalSweeper.h"
> #include "JSObject.h"
> #include "JSCInlines.h"
>+#include "SuperSampler.h"
>+#include <wtf/ListDump.h>
> 
> namespace JSC {
> 
>+std::array<size_t, MarkedSpace::numSizeClasses> MarkedSpace::s_sizeClassForSizeStep;
>+
>+namespace {
>+
>+const Vector<size_t>& sizeClasses()
>+{
>+    static Vector<size_t>* result;
>+    static std::once_flag once;
>+    std::call_once(
>+        once,
>+        [] {
>+            result = new Vector<size_t>();
>+            
>+            auto add = [&] (size_t sizeClass) {
>+                if (Options::dumpSizeClasses())
>+                    dataLog("Adding JSC MarkedSpace size class: ", sizeClass, "\n");
>+                // Perform some validation as we go.
>+                RELEASE_ASSERT(!(sizeClass % MarkedSpace::sizeStep));
>+                if (result->isEmpty())
>+                    RELEASE_ASSERT(sizeClass == MarkedSpace::sizeStep);
>+                else
>+                    RELEASE_ASSERT(sizeClass > result->last());
>+                result->append(sizeClass);
>+            };
>+            
>+            // This is a definition of the size classes in our GC. It must define all of the
>+            // size classes from sizeStep up to largeCutoff.
>+    
>+            // Have very precise size classes for the small stuff. This is a loop to make it easy to reduce
>+            // atomSize.
>+            for (size_t size = MarkedSpace::sizeStep; size < MarkedSpace::preciseCutoff; size += MarkedSpace::sizeStep)
>+                add(size);
>+            
>+            // We want to make sure that the remaining size classes minimize internal fragmentation (i.e.
>+            // the wasted space at the tail end of a MarkedBlock) while proceeding roughly in an exponential
>+            // way starting at just above the precise size classes to four cells per block.
>+            
>+            if (Options::dumpSizeClasses())
>+                dataLog("    Marked block payload size: ", static_cast<size_t>(MarkedSpace::blockPayload), "\n");
>+            
>+            for (unsigned i = 0; ; ++i) {
>+                double approximateSize = MarkedSpace::preciseCutoff * pow(Options::sizeClassProgression(), i);
>+                
>+                if (Options::dumpSizeClasses())
>+                    dataLog("    Next size class as a double: ", approximateSize, "\n");
>+        
>+                size_t approximateSizeInBytes = static_cast<size_t>(approximateSize);
>+        
>+                if (Options::dumpSizeClasses())
>+                    dataLog("    Next size class as bytes: ", approximateSizeInBytes, "\n");
>+        
>+                // Make sure that the computer did the math correctly.
>+                RELEASE_ASSERT(approximateSizeInBytes >= MarkedSpace::preciseCutoff);
>+                
>+                if (approximateSizeInBytes > MarkedSpace::largeCutoff)
>+                    break;
>+                
>+                size_t sizeClass =
>+                    WTF::roundUpToMultipleOf<MarkedSpace::sizeStep>(approximateSizeInBytes);
>+                
>+                if (Options::dumpSizeClasses())
>+                    dataLog("    Size class: ", sizeClass, "\n");
>+                
>+                // Optimize the size class so that there isn't any slop at the end of the block's
>+                // payload.
>+                unsigned cellsPerBlock = MarkedSpace::blockPayload / sizeClass;
>+                size_t possiblyBetterSizeClass = (MarkedSpace::blockPayload / cellsPerBlock) & ~(MarkedSpace::sizeStep - 1);
>+                
>+                if (Options::dumpSizeClasses())
>+                    dataLog("    Possibly better size class: ", possiblyBetterSizeClass, "\n");
>+
>+                // The size class we just came up with is better than the other one if it reduces
>+                // total wastage assuming we only allocate cells of that size.
>+                size_t originalWastage = MarkedSpace::blockPayload - cellsPerBlock * sizeClass;
>+                size_t newWastage = (possiblyBetterSizeClass - sizeClass) * cellsPerBlock;
>+                
>+                if (Options::dumpSizeClasses())
>+                    dataLog("    Original wastage: ", originalWastage, ", new wastage: ", newWastage, "\n");
>+                
>+                size_t betterSizeClass;
>+                if (newWastage > originalWastage)
>+                    betterSizeClass = sizeClass;
>+                else
>+                    betterSizeClass = possiblyBetterSizeClass;
>+                
>+                if (Options::dumpSizeClasses())
>+                    dataLog("    Choosing size class: ", betterSizeClass, "\n");
>+                
>+                if (betterSizeClass == result->last()) {
>+                    // Defense for when expStep is small.
>+                    continue;
>+                }
>+                
>+                // This is usually how we get out of the loop.
>+                if (betterSizeClass > MarkedSpace::largeCutoff
>+                    || betterSizeClass > Options::largeAllocationCutoff())
>+                    break;
>+                
>+                add(betterSizeClass);
>+            }
>+            
>+            if (Options::dumpSizeClasses())
>+                dataLog("JSC Heap MarkedSpace size class dump: ", listDump(*result), "\n");
>+
>+            // We have an optimiation in MarkedSpace::optimalSizeFor() that assumes things about
>+            // the size class table. This checks our results against that function's assumptions.
>+            for (size_t size = MarkedSpace::sizeStep, i = 0; size <= MarkedSpace::preciseCutoff; size += MarkedSpace::sizeStep, i++)
>+                RELEASE_ASSERT(result->at(i) == size);
>+        });
>+    return *result;
>+}
>+
>+template<typename TableType, typename SizeClassCons, typename DefaultCons>
>+void buildSizeClassTable(TableType& table, const SizeClassCons& cons, const DefaultCons& defaultCons)
>+{
>+    size_t nextIndex = 0;
>+    for (size_t sizeClass : sizeClasses()) {
>+        auto entry = cons(sizeClass);
>+        size_t index = MarkedSpace::sizeClassToIndex(sizeClass);
>+        for (size_t i = nextIndex; i <= index; ++i)
>+            table[i] = entry;
>+        nextIndex = index + 1;
>+    }
>+    for (size_t i = nextIndex; i < MarkedSpace::numSizeClasses; ++i)
>+        table[i] = defaultCons(MarkedSpace::indexToSizeClass(i));
>+}
>+
>+} // anonymous namespace
>+
>+void MarkedSpace::initializeSizeClassForStepSize()
>+{
>+    // We call this multiple times and we may call it simultaneously from multiple threads. That's
>+    // OK, since it always stores the same values into the table.
>+    
>+    buildSizeClassTable(
>+        s_sizeClassForSizeStep,
>+        [&] (size_t sizeClass) -> size_t {
>+            return sizeClass;
>+        },
>+        [&] (size_t sizeClass) -> size_t {
>+            return sizeClass;
>+        });
>+}
>+
> MarkedSpace::MarkedSpace(Heap* heap)
>     : m_heap(heap)
>     , m_capacity(0)
>     , m_isIterating(false)
> {
>-    forEachAllocator(
>-        [&] (MarkedAllocator& allocator, size_t cellSize, AllocatorAttributes attributes) -> IterationStatus {
>-            allocator.init(heap, this, cellSize, attributes);
>+    initializeSizeClassForStepSize();
>+    
>+    forEachSubspace(
>+        [&] (Subspace& subspace, AllocatorAttributes attributes) -> IterationStatus {
>+            subspace.attributes = attributes;
>+            
>+            buildSizeClassTable(
>+                subspace.allocatorForSizeStep,
>+                [&] (size_t sizeClass) -> MarkedAllocator* {
>+                    return subspace.bagOfAllocators.add(heap, this, sizeClass, attributes);
>+                },
>+                [&] (size_t) -> MarkedAllocator* {
>+                    return nullptr;
>+                });
>+            
>             return IterationStatus::Continue;
>         });
> }
>@@ -52,14 +210,51 @@ void MarkedSpace::lastChanceToFinalize()
> {
>     stopAllocating();
>     forEachAllocator(
>-        [&] (MarkedAllocator& allocator, size_t, AllocatorAttributes) -> IterationStatus {
>+        [&] (MarkedAllocator& allocator) -> IterationStatus {
>             allocator.lastChanceToFinalize();
>             return IterationStatus::Continue;
>         });
> }
> 
>+void* MarkedSpace::allocate(Subspace& subspace, size_t bytes)
>+{
>+    if (MarkedAllocator* allocator = allocatorFor(subspace, bytes))
>+        return allocator->allocate(bytes);
>+    return allocateLarge(subspace, bytes);
>+}
>+
>+void* MarkedSpace::tryAllocate(Subspace& subspace, size_t bytes)
>+{
>+    if (MarkedAllocator* allocator = allocatorFor(subspace, bytes))
>+        return allocator->tryAllocate(bytes);
>+    return tryAllocateLarge(subspace, bytes);
>+}
>+
>+void* MarkedSpace::allocateLarge(Subspace& subspace, size_t size)
>+{
>+    void* result = tryAllocateLarge(subspace, size);
>+    RELEASE_ASSERT(result);
>+    return result;
>+}
>+
>+void* MarkedSpace::tryAllocateLarge(Subspace& subspace, size_t size)
>+{
>+    m_heap->collectIfNecessaryOrDefer();
>+    
>+    size = WTF::roundUpToMultipleOf<sizeStep>(size);
>+    LargeAllocation* allocation = LargeAllocation::tryCreate(*m_heap, size, subspace.attributes);
>+    if (!allocation)
>+        return nullptr;
>+    
>+    m_largeAllocations.append(allocation);
>+    m_heap->didAllocate(size);
>+    m_capacity += size;
>+    return allocation->cell();
>+}
>+
> void MarkedSpace::sweep()
> {
>+    sweepLargeAllocations();
>     m_heap->sweeper()->willFinishSweeping();
>     forEachBlock(
>         [&] (MarkedBlock* block) {
>@@ -67,10 +262,30 @@ void MarkedSpace::sweep()
>         });
> }
> 
>+void MarkedSpace::sweepLargeAllocations()
>+{
>+    RELEASE_ASSERT(m_largeAllocationsNurseryOffset == m_largeAllocations.size());
>+    unsigned srcIndex = m_largeAllocationsNurseryOffsetForSweep;
>+    unsigned dstIndex = srcIndex;
>+    while (srcIndex < m_largeAllocations.size()) {
>+        LargeAllocation* allocation = m_largeAllocations[srcIndex++];
>+        allocation->sweep();
>+        if (allocation->isEmpty()) {
>+            m_capacity -= allocation->cellSize();
>+            allocation->destroy();
>+            continue;
>+        }
>+        m_largeAllocations[dstIndex++] = allocation;
>+    }
>+    m_largeAllocations.resize(dstIndex);
>+    m_largeAllocationsNurseryOffset = m_largeAllocations.size();
>+}
>+
> void MarkedSpace::zombifySweep()
> {
>     if (Options::logGC())
>         dataLog("Zombifying sweep...");
>+    sweepLargeAllocations();
>     m_heap->sweeper()->willFinishSweeping();
>     forEachBlock(
>         [&] (MarkedBlock* block) {
>@@ -82,12 +297,17 @@ void MarkedSpace::zombifySweep()
> void MarkedSpace::resetAllocators()
> {
>     forEachAllocator(
>-        [&] (MarkedAllocator& allocator, size_t, AllocatorAttributes) -> IterationStatus {
>+        [&] (MarkedAllocator& allocator) -> IterationStatus {
>             allocator.reset();
>             return IterationStatus::Continue;
>         });
> 
>     m_blocksWithNewObjects.clear();
>+    if (m_heap->operationInProgress() == EdenCollection)
>+        m_largeAllocationsNurseryOffsetForSweep = m_largeAllocationsNurseryOffset;
>+    else
>+        m_largeAllocationsNurseryOffsetForSweep = 0;
>+    m_largeAllocationsNurseryOffset = m_largeAllocations.size();
> }
> 
> void MarkedSpace::visitWeakSets(HeapRootVisitor& heapRootVisitor)
>@@ -101,6 +321,8 @@ void MarkedSpace::visitWeakSets(HeapRoot
>                 block->visitWeakSet(heapRootVisitor);
>             });
>     }
>+    for (unsigned i = m_largeAllocationsOffsetForThisCollection; i < m_largeAllocations.size(); ++i)
>+        m_largeAllocations[i]->visitWeakSet(heapRootVisitor);
> }
> 
> void MarkedSpace::reapWeakSets()
>@@ -114,59 +336,60 @@ void MarkedSpace::reapWeakSets()
>                 block->reapWeakSet();
>             });
>     }
>-}
>-
>-template <typename Functor>
>-void MarkedSpace::forEachAllocator(const Functor& functor)
>-{
>-    forEachSubspace(
>-        [&] (Subspace& subspace, AllocatorAttributes attributes) -> IterationStatus {
>-            for (size_t cellSize = preciseStep; cellSize <= preciseCutoff; cellSize += preciseStep) {
>-                if (functor(allocatorFor(subspace, cellSize), cellSize, attributes) == IterationStatus::Done)
>-                    return IterationStatus::Done;
>-            }
>-            for (size_t cellSize = impreciseStart; cellSize <= impreciseCutoff; cellSize += impreciseStep) {
>-                if (functor(allocatorFor(subspace, cellSize), cellSize, attributes) == IterationStatus::Done)
>-                    return IterationStatus::Done;
>-            }
>-            if (functor(subspace.largeAllocator, 0, attributes) == IterationStatus::Done)
>-                return IterationStatus::Done;
>-            
>-            return IterationStatus::Continue;
>-        });
>+    for (unsigned i = m_largeAllocationsOffsetForThisCollection; i < m_largeAllocations.size(); ++i)
>+        m_largeAllocations[i]->reapWeakSet();
> }
> 
> void MarkedSpace::stopAllocating()
> {
>     ASSERT(!isIterating());
>     forEachAllocator(
>-        [&] (MarkedAllocator& allocator, size_t, AllocatorAttributes) -> IterationStatus {
>+        [&] (MarkedAllocator& allocator) -> IterationStatus {
>             allocator.stopAllocating();
>             return IterationStatus::Continue;
>         });
> }
> 
>+void MarkedSpace::prepareForMarking()
>+{
>+    if (m_heap->operationInProgress() == EdenCollection)
>+        m_largeAllocationsOffsetForThisCollection = m_largeAllocationsNurseryOffset;
>+    else
>+        m_largeAllocationsOffsetForThisCollection = 0;
>+    m_largeAllocationsForThisCollectionBegin = m_largeAllocations.begin() + m_largeAllocationsOffsetForThisCollection;
>+    m_largeAllocationsForThisCollectionSize = m_largeAllocations.size() - m_largeAllocationsOffsetForThisCollection;
>+    m_largeAllocationsForThisCollectionEnd = m_largeAllocations.end();
>+    RELEASE_ASSERT(m_largeAllocationsForThisCollectionEnd == m_largeAllocationsForThisCollectionBegin + m_largeAllocationsForThisCollectionSize);
>+    std::sort(
>+        m_largeAllocationsForThisCollectionBegin, m_largeAllocationsForThisCollectionEnd,
>+        [&] (LargeAllocation* a, LargeAllocation* b) {
>+            return a < b;
>+        });
>+}
>+
> void MarkedSpace::resumeAllocating()
> {
>     ASSERT(isIterating());
>     forEachAllocator(
>-        [&] (MarkedAllocator& allocator, size_t, AllocatorAttributes) -> IterationStatus {
>+        [&] (MarkedAllocator& allocator) -> IterationStatus {
>             allocator.resumeAllocating();
>             return IterationStatus::Continue;
>         });
>+    // Nothing to do for LargeAllocations.
> }
> 
> bool MarkedSpace::isPagedOut(double deadline)
> {
>     bool result = false;
>     forEachAllocator(
>-        [&] (MarkedAllocator& allocator, size_t, AllocatorAttributes) -> IterationStatus {
>+        [&] (MarkedAllocator& allocator) -> IterationStatus {
>             if (allocator.isPagedOut(deadline)) {
>                 result = true;
>                 return IterationStatus::Done;
>             }
>             return IterationStatus::Continue;
>         });
>+    // FIXME: Consider taking LargeAllocations into account here.
>     return result;
> }
> 
>@@ -194,31 +417,29 @@ void MarkedSpace::shrink()
>         [&] (MarkedBlock* block) {
>             freeOrShrinkBlock(block);
>         });
>+    // For LargeAllocations, we do the moral equivalent in sweepLargeAllocations().
> }
> 
> void MarkedSpace::clearNewlyAllocated()
> {
>     forEachAllocator(
>-        [&] (MarkedAllocator& allocator, size_t size, AllocatorAttributes) -> IterationStatus {
>-            if (!size) {
>-                // This means it's a largeAllocator.
>-                allocator.forEachBlock(
>-                    [&] (MarkedBlock* block) {
>-                        block->clearNewlyAllocated();
>-                    });
>-                return IterationStatus::Continue;
>-            }
>-            
>+        [&] (MarkedAllocator& allocator) -> IterationStatus {
>             if (MarkedBlock* block = allocator.takeLastActiveBlock())
>                 block->clearNewlyAllocated();
>             return IterationStatus::Continue;
>         });
>+    
>+    for (unsigned i = m_largeAllocationsOffsetForThisCollection; i < m_largeAllocations.size(); ++i)
>+        m_largeAllocations[i]->clearNewlyAllocated();
> 
> #if !ASSERT_DISABLED
>     forEachBlock(
>         [&] (MarkedBlock* block) {
>             ASSERT(!block->clearNewlyAllocated());
>         });
>+
>+    for (LargeAllocation* allocation : m_largeAllocations)
>+        ASSERT(!allocation->isNewlyAllocated());
> #endif // !ASSERT_DISABLED
> }
> 
>@@ -247,6 +468,8 @@ void MarkedSpace::clearMarks()
>             [&] (MarkedBlock* block) {
>                 block->clearMarks();
>             });
>+        for (LargeAllocation* allocation : m_largeAllocations)
>+            allocation->clearMarks();
>     }
> 
> #ifndef NDEBUG
>@@ -269,4 +492,37 @@ void MarkedSpace::didFinishIterating()
>     m_isIterating = false;
> }
> 
>+size_t MarkedSpace::objectCount()
>+{
>+    size_t result = 0;
>+    forEachBlock(
>+        [&] (MarkedBlock* block) {
>+            result += block->markCount();
>+        });
>+    for (LargeAllocation* allocation : m_largeAllocations) {
>+        if (allocation->isMarked())
>+            result++;
>+    }
>+    return result;
>+}
>+
>+size_t MarkedSpace::size()
>+{
>+    size_t result = 0;
>+    forEachBlock(
>+        [&] (MarkedBlock* block) {
>+            result += block->markCount() * block->cellSize();
>+        });
>+    for (LargeAllocation* allocation : m_largeAllocations) {
>+        if (allocation->isMarked())
>+            result += allocation->cellSize();
>+    }
>+    return result;
>+}
>+
>+size_t MarkedSpace::capacity()
>+{
>+    return m_capacity;
>+}
>+
> } // namespace JSC
>Index: Source/JavaScriptCore/heap/MarkedSpace.h
>===================================================================
>--- Source/JavaScriptCore/heap/MarkedSpace.h	(revision 204735)
>+++ Source/JavaScriptCore/heap/MarkedSpace.h	(working copy)
>@@ -23,10 +23,12 @@
> #define MarkedSpace_h
> 
> #include "IterationStatus.h"
>+#include "LargeAllocation.h"
> #include "MarkedAllocator.h"
> #include "MarkedBlock.h"
> #include "MarkedBlockSet.h"
> #include <array>
>+#include <wtf/Bag.h>
> #include <wtf/HashSet.h>
> #include <wtf/Noncopyable.h>
> #include <wtf/RetainPtr.h>
>@@ -41,38 +43,71 @@ class LLIntOffsetsExtractor;
> class MarkedSpace {
>     WTF_MAKE_NONCOPYABLE(MarkedSpace);
> public:
>-    // [ 16 ... 768 ]
>-    static const size_t preciseStep = MarkedBlock::atomSize;
>-    static const size_t preciseCutoff = 768;
>-    static const size_t preciseCount = preciseCutoff / preciseStep;
>-
>-    // [ 1024 ... blockSize/2 ]
>-    static const size_t impreciseStart = 1024;
>-    static const size_t impreciseStep = 256;
>-    static const size_t impreciseCutoff = MarkedBlock::blockSize / 2;
>-    static const size_t impreciseCount = impreciseCutoff / impreciseStep;
>+    // sizeStep is really a synonym for atomSize; it's no accident that they are the same.
>+    static const size_t sizeStep = MarkedBlock::atomSize;
>+    
>+    // Sizes up to this amount get a size class for each size step.
>+    static const size_t preciseCutoff = 80;
>+    
>+    // The amount of available payload in a block is the block's size minus the header. But the
>+    // header size might not be atom size aligned, so we round down the result accordingly.
>+    static const size_t blockPayload = (MarkedBlock::blockSize - sizeof(MarkedBlock)) & ~(MarkedBlock::atomSize - 1);
>+    
>+    // The largest cell we're willing to allocate in a MarkedBlock the "normal way" (i.e. using size
>+    // classes, rather than a large allocation) is half the size of the payload, rounded down. This
>+    // ensures that we only use the size class approach if it means being able to pack two things
>+    // into one block.
>+    static const size_t largeCutoff = (blockPayload / 2) & ~(sizeStep - 1);
> 
>+    static const size_t numSizeClasses = largeCutoff / sizeStep;
>+    
>+    static size_t sizeClassToIndex(size_t size)
>+    {
>+        ASSERT(size);
>+        return (size + sizeStep - 1) / sizeStep - 1;
>+    }
>+    
>+    static size_t indexToSizeClass(size_t index)
>+    {
>+        return (index + 1) * sizeStep;
>+    }
>+    
>+    // Each Subspace corresponds to all of the blocks for all of the sizes for some "class" of
>+    // objects. There are three classes: non-destructor JSCells, destructor JSCells, and auxiliary.
>+    // MarkedSpace is set up to make it relatively easy to add new Subspaces.
>     struct Subspace {
>-        std::array<MarkedAllocator, preciseCount> preciseAllocators;
>-        std::array<MarkedAllocator, impreciseCount> impreciseAllocators;
>-        MarkedAllocator largeAllocator;
>+        std::array<MarkedAllocator*, numSizeClasses> allocatorForSizeStep;
>+        
>+        // Each MarkedAllocator is a size class.
>+        Bag<MarkedAllocator> bagOfAllocators;
>+        
>+        AllocatorAttributes attributes;
>     };
>-
>+    
>     MarkedSpace(Heap*);
>     ~MarkedSpace();
>     void lastChanceToFinalize();
> 
>-    MarkedAllocator& allocatorFor(size_t);
>-    MarkedAllocator& destructorAllocatorFor(size_t);
>-    MarkedAllocator& auxiliaryAllocatorFor(size_t);
>+    static size_t optimalSizeFor(size_t);
>+    
>+    static MarkedAllocator* allocatorFor(Subspace&, size_t);
>+
>+    MarkedAllocator* allocatorFor(size_t);
>+    MarkedAllocator* destructorAllocatorFor(size_t);
>+    MarkedAllocator* auxiliaryAllocatorFor(size_t);
>+
>+    JS_EXPORT_PRIVATE void* allocate(Subspace&, size_t);
>+    JS_EXPORT_PRIVATE void* tryAllocate(Subspace&, size_t);
>+    
>     void* allocateWithDestructor(size_t);
>     void* allocateWithoutDestructor(size_t);
>     void* allocateAuxiliary(size_t);
>-
>+    void* tryAllocateAuxiliary(size_t);
>+    
>     Subspace& subspaceForObjectsWithDestructor() { return m_destructorSpace; }
>     Subspace& subspaceForObjectsWithoutDestructor() { return m_normalSpace; }
>     Subspace& subspaceForAuxiliaryData() { return m_auxiliarySpace; }
>-
>+    
>     void resetAllocators();
> 
>     void visitWeakSets(HeapRootVisitor&);
>@@ -86,6 +121,8 @@ public:
> 
>     void stopAllocating();
>     void resumeAllocating(); // If we just stopped allocation but we didn't do a collection, we need to resume allocation.
>+    
>+    void prepareForMarking();
> 
>     typedef HashSet<MarkedBlock*>::iterator BlockIterator;
> 
>@@ -104,6 +141,7 @@ public:
>     void clearMarks();
>     void clearNewlyAllocated();
>     void sweep();
>+    void sweepLargeAllocations();
>     void zombifySweep();
>     size_t objectCount();
>     size_t size();
>@@ -112,15 +150,33 @@ public:
>     bool isPagedOut(double deadline);
> 
>     const Vector<MarkedBlock*>& blocksWithNewObjects() const { return m_blocksWithNewObjects; }
>+    
>+    const Vector<LargeAllocation*>& largeAllocations() const { return m_largeAllocations; }
>+    unsigned largeAllocationsNurseryOffset() const { return m_largeAllocationsNurseryOffset; }
>+    unsigned largeAllocationsOffsetForThisCollection() const { return m_largeAllocationsOffsetForThisCollection; }
>+    
>+    // These are cached pointers and offsets for quickly searching the large allocations that are
>+    // relevant to this collection.
>+    LargeAllocation** largeAllocationsForThisCollectionBegin() const { return m_largeAllocationsForThisCollectionBegin; }
>+    LargeAllocation** largeAllocationsForThisCollectionEnd() const { return m_largeAllocationsForThisCollectionEnd; }
>+    unsigned largeAllocationsForThisCollectionSize() const { return m_largeAllocationsForThisCollectionSize; }
> 
> private:
>     friend class LLIntOffsetsExtractor;
>     friend class JIT;
>+    
>+    JS_EXPORT_PRIVATE static std::array<size_t, numSizeClasses> s_sizeClassForSizeStep;
>+    
>+    JS_EXPORT_PRIVATE void* allocateLarge(Subspace&, size_t);
>+    JS_EXPORT_PRIVATE void* tryAllocateLarge(Subspace&, size_t);
>+
>+    static void initializeSizeClassForStepSize();
>+    
>+    void initializeSubspace(Subspace&);
> 
>     template<typename Functor> void forEachAllocator(const Functor&);
>     template<typename Functor> void forEachSubspace(const Functor&);
>-    MarkedAllocator& allocatorFor(Subspace&, size_t);
>-
>+    
>     Subspace m_destructorSpace;
>     Subspace m_normalSpace;
>     Subspace m_auxiliarySpace;
>@@ -130,6 +186,13 @@ private:
>     bool m_isIterating;
>     MarkedBlockSet m_blocks;
>     Vector<MarkedBlock*> m_blocksWithNewObjects;
>+    Vector<LargeAllocation*> m_largeAllocations;
>+    unsigned m_largeAllocationsNurseryOffset { 0 };
>+    unsigned m_largeAllocationsOffsetForThisCollection { 0 };
>+    unsigned m_largeAllocationsNurseryOffsetForSweep { 0 };
>+    LargeAllocation** m_largeAllocationsForThisCollectionBegin { nullptr };
>+    LargeAllocation** m_largeAllocationsForThisCollectionEnd { nullptr };
>+    unsigned m_largeAllocationsForThisCollectionSize { 0 };
> };
> 
> template<typename Functor> inline void MarkedSpace::forEachLiveCell(HeapIterationScope&, const Functor& functor)
>@@ -138,7 +201,13 @@ template<typename Functor> inline void M
>     BlockIterator end = m_blocks.set().end();
>     for (BlockIterator it = m_blocks.set().begin(); it != end; ++it) {
>         if ((*it)->forEachLiveCell(functor) == IterationStatus::Done)
>-            break;
>+            return;
>+    }
>+    for (LargeAllocation* allocation : m_largeAllocations) {
>+        if (allocation->isLive()) {
>+            if (functor(allocation->cell(), allocation->attributes().cellKind) == IterationStatus::Done)
>+                return;
>+        }
>     }
> }
> 
>@@ -148,49 +217,64 @@ template<typename Functor> inline void M
>     BlockIterator end = m_blocks.set().end();
>     for (BlockIterator it = m_blocks.set().begin(); it != end; ++it) {
>         if ((*it)->forEachDeadCell(functor) == IterationStatus::Done)
>-            break;
>+            return;
>+    }
>+    for (LargeAllocation* allocation : m_largeAllocations) {
>+        if (!allocation->isLive()) {
>+            if (functor(allocation->cell(), allocation->attributes().cellKind) == IterationStatus::Done)
>+                return;
>+        }
>     }
> }
> 
>-inline MarkedAllocator& MarkedSpace::allocatorFor(size_t bytes)
>+inline MarkedAllocator* MarkedSpace::allocatorFor(Subspace& space, size_t bytes)
>+{
>+    ASSERT(bytes);
>+    if (bytes <= largeCutoff)
>+        return space.allocatorForSizeStep[sizeClassToIndex(bytes)];
>+    return nullptr;
>+}
>+
>+inline MarkedAllocator* MarkedSpace::allocatorFor(size_t bytes)
> {
>     return allocatorFor(m_normalSpace, bytes);
> }
> 
>-inline MarkedAllocator& MarkedSpace::destructorAllocatorFor(size_t bytes)
>+inline MarkedAllocator* MarkedSpace::destructorAllocatorFor(size_t bytes)
> {
>     return allocatorFor(m_destructorSpace, bytes);
> }
> 
>-inline MarkedAllocator& MarkedSpace::auxiliaryAllocatorFor(size_t bytes)
>+inline MarkedAllocator* MarkedSpace::auxiliaryAllocatorFor(size_t bytes)
> {
>     return allocatorFor(m_auxiliarySpace, bytes);
> }
> 
> inline void* MarkedSpace::allocateWithoutDestructor(size_t bytes)
> {
>-    return allocatorFor(bytes).allocate(bytes);
>+    return allocate(m_normalSpace, bytes);
> }
> 
> inline void* MarkedSpace::allocateWithDestructor(size_t bytes)
> {
>-    return destructorAllocatorFor(bytes).allocate(bytes);
>+    return allocate(m_destructorSpace, bytes);
> }
> 
> inline void* MarkedSpace::allocateAuxiliary(size_t bytes)
> {
>-    return auxiliaryAllocatorFor(bytes).allocate(bytes);
>+    return allocate(m_auxiliarySpace, bytes);
>+}
>+
>+inline void* MarkedSpace::tryAllocateAuxiliary(size_t bytes)
>+{
>+    return tryAllocate(m_auxiliarySpace, bytes);
> }
> 
> template <typename Functor> inline void MarkedSpace::forEachBlock(const Functor& functor)
> {
>-    forEachSubspace(
>-        [&] (Subspace& subspace, AllocatorAttributes) -> IterationStatus {
>-            for (size_t i = 0; i < preciseCount; ++i)
>-                subspace.preciseAllocators[i].forEachBlock(functor);
>-            for (size_t i = 0; i < impreciseCount; ++i)
>-                subspace.impreciseAllocators[i].forEachBlock(functor);
>-            subspace.largeAllocator.forEachBlock(functor);
>+    forEachAllocator(
>+        [&] (MarkedAllocator& allocator) -> IterationStatus {
>+            allocator.forEachBlock(functor);
>             return IterationStatus::Continue;
>         });
> }
>@@ -206,29 +290,18 @@ inline void MarkedSpace::didAllocateInBl
>     m_blocksWithNewObjects.append(block);
> }
> 
>-inline size_t MarkedSpace::objectCount()
>-{
>-    size_t result = 0;
>-    forEachBlock(
>-        [&] (MarkedBlock* block) {
>-            result += block->markCount();
>-        });
>-    return result;
>-}
>-
>-inline size_t MarkedSpace::size()
>+template <typename Functor>
>+void MarkedSpace::forEachAllocator(const Functor& functor)
> {
>-    size_t result = 0;
>-    forEachBlock(
>-        [&] (MarkedBlock* block) {
>-            result += block->markCount() * block->cellSize();
>+    forEachSubspace(
>+        [&] (Subspace& subspace, AllocatorAttributes) -> IterationStatus {
>+            for (MarkedAllocator* allocator : subspace.bagOfAllocators) {
>+                if (functor(*allocator) == IterationStatus::Done)
>+                    return IterationStatus::Done;
>+            }
>+            
>+            return IterationStatus::Continue;
>         });
>-    return result;
>-}
>-
>-inline size_t MarkedSpace::capacity()
>-{
>-    return m_capacity;
> }
> 
> template<typename Functor>
>@@ -251,14 +324,14 @@ inline void MarkedSpace::forEachSubspace
>     func(m_auxiliarySpace, attributes);
> }
> 
>-inline MarkedAllocator& MarkedSpace::allocatorFor(Subspace& space, size_t bytes)
>+ALWAYS_INLINE size_t MarkedSpace::optimalSizeFor(size_t bytes)
> {
>     ASSERT(bytes);
>     if (bytes <= preciseCutoff)
>-        return space.preciseAllocators[(bytes - 1) / preciseStep];
>-    if (bytes <= impreciseCutoff)
>-        return space.impreciseAllocators[(bytes - 1) / impreciseStep];
>-    return space.largeAllocator;
>+        return WTF::roundUpToMultipleOf<sizeStep>(bytes);
>+    if (bytes <= largeCutoff)
>+        return s_sizeClassForSizeStep[sizeClassToIndex(bytes)];
>+    return bytes;
> }
> 
> } // namespace JSC
>Index: Source/JavaScriptCore/heap/SlotVisitor.cpp
>===================================================================
>--- Source/JavaScriptCore/heap/SlotVisitor.cpp	(revision 204735)
>+++ Source/JavaScriptCore/heap/SlotVisitor.cpp	(working copy)
>@@ -1,5 +1,5 @@
> /*
>- * Copyright (C) 2012, 2015 Apple Inc. All rights reserved.
>+ * Copyright (C) 2012, 2015-2016 Apple Inc. All rights reserved.
>  *
>  * Redistribution and use in source and binary forms, with or without
>  * modification, are permitted provided that the following conditions
>@@ -31,14 +31,16 @@
> #include "CopiedBlockInlines.h"
> #include "CopiedSpace.h"
> #include "CopiedSpaceInlines.h"
>+#include "HeapCellInlines.h"
> #include "HeapProfiler.h"
> #include "HeapSnapshotBuilder.h"
> #include "JSArray.h"
> #include "JSDestructibleObject.h"
>-#include "VM.h"
> #include "JSObject.h"
> #include "JSString.h"
> #include "JSCInlines.h"
>+#include "SuperSampler.h"
>+#include "VM.h"
> #include <wtf/Lock.h>
> 
> namespace JSC {
>@@ -96,6 +98,8 @@ void SlotVisitor::didStartMarking()
> {
>     if (heap()->operationInProgress() == FullCollection)
>         ASSERT(m_opaqueRoots.isEmpty()); // Should have merged by now.
>+    else
>+        reset();
> 
>     if (HeapProfiler* heapProfiler = vm().heapProfiler())
>         m_heapSnapshotBuilder = heapProfiler->activeSnapshotBuilder();
>@@ -108,7 +112,6 @@ void SlotVisitor::reset()
>     m_visitCount = 0;
>     m_heapSnapshotBuilder = nullptr;
>     ASSERT(!m_currentCell);
>-    ASSERT(m_stack.isEmpty());
> }
> 
> void SlotVisitor::clearMarkStack()
>@@ -118,10 +121,41 @@ void SlotVisitor::clearMarkStack()
> 
> void SlotVisitor::append(ConservativeRoots& conservativeRoots)
> {
>-    JSCell** roots = conservativeRoots.roots();
>+    HeapCell** roots = conservativeRoots.roots();
>     size_t size = conservativeRoots.size();
>     for (size_t i = 0; i < size; ++i)
>-        append(roots[i]);
>+        appendJSCellOrAuxiliary(roots[i]);
>+}
>+
>+void SlotVisitor::appendJSCellOrAuxiliary(HeapCell* heapCell)
>+{
>+    if (!heapCell)
>+        return;
>+    
>+    ASSERT(!m_isCheckingForDefaultMarkViolation);
>+    
>+    if (Heap::testAndSetMarked(heapCell))
>+        return;
>+    
>+    switch (heapCell->cellKind()) {
>+    case HeapCell::JSCell: {
>+        JSCell* jsCell = static_cast<JSCell*>(heapCell);
>+        
>+        if (!jsCell->structure()) {
>+            ASSERT_NOT_REACHED();
>+            return;
>+        }
>+        
>+        jsCell->setCellState(CellState::NewGrey);
>+
>+        appendToMarkStack(jsCell);
>+        return;
>+    }
>+        
>+    case HeapCell::Auxiliary: {
>+        noteLiveAuxiliaryCell(heapCell);
>+        return;
>+    } }
> }
> 
> void SlotVisitor::append(JSValue value)
>@@ -145,6 +179,8 @@ void SlotVisitor::appendHidden(JSValue v
> 
> void SlotVisitor::setMarkedAndAppendToMarkStack(JSCell* cell)
> {
>+    SuperSamplerScope superSamplerScope(false);
>+    
>     ASSERT(!m_isCheckingForDefaultMarkViolation);
>     if (!cell)
>         return;
>@@ -152,33 +188,85 @@ void SlotVisitor::setMarkedAndAppendToMa
> #if ENABLE(GC_VALIDATION)
>     validate(cell);
> #endif
>+    
>+    if (cell->isLargeAllocation())
>+        setMarkedAndAppendToMarkStack(cell->largeAllocation(), cell);
>+    else
>+        setMarkedAndAppendToMarkStack(cell->markedBlock(), cell);
>+}
> 
>-    if (Heap::testAndSetMarked(cell) || !cell->structure()) {
>-        ASSERT(cell->structure());
>+template<typename ContainerType>
>+ALWAYS_INLINE void SlotVisitor::setMarkedAndAppendToMarkStack(
>+    ContainerType& container, JSCell* cell)
>+{
>+    if (container.testAndSetMarked(cell))
>         return;
>-    }
>-
>+    
>+    ASSERT(cell->structure());
>+    
>     // Indicate that the object is grey and that:
>     // In case of concurrent GC: it's the first time it is grey in this GC cycle.
>     // In case of eden collection: it's a new object that became grey rather than an old remembered object.
>     cell->setCellState(CellState::NewGrey);
>-
>-    appendToMarkStack(cell);
>+    
>+    appendToMarkStack(container, cell);
> }
> 
> void SlotVisitor::appendToMarkStack(JSCell* cell)
> {
>+    if (cell->isLargeAllocation())
>+        appendToMarkStack(cell->largeAllocation(), cell);
>+    else
>+        appendToMarkStack(cell->markedBlock(), cell);
>+}
>+
>+template<typename ContainerType>
>+ALWAYS_INLINE void SlotVisitor::appendToMarkStack(ContainerType& container, JSCell* cell)
>+{
>     ASSERT(Heap::isMarked(cell));
>     ASSERT(!cell->isZapped());
>-
>+    
>+    container.setHasAnyMarked(); // This permits super fast sweeping.
>+    
>+    // FIXME: These "just work" because the GC resets these fields before doing anything else. But
>+    // that won't be the case when we do concurrent GC.
>     m_visitCount++;
>-    m_bytesVisited += MarkedBlock::blockFor(cell)->cellSize();
>+    m_bytesVisited += container.cellSize();
>+    
>     m_stack.append(cell);
> 
>     if (UNLIKELY(m_heapSnapshotBuilder))
>         m_heapSnapshotBuilder->appendNode(cell);
> }
> 
>+void SlotVisitor::markAuxiliary(const void* base)
>+{
>+    HeapCell* cell = bitwise_cast<HeapCell*>(base);
>+    
>+    if (Heap::testAndSetMarked(cell)) {
>+        RELEASE_ASSERT(Heap::isMarked(cell));
>+        return;
>+    }
>+    
>+    noteLiveAuxiliaryCell(cell);
>+}
>+
>+void SlotVisitor::noteLiveAuxiliaryCell(HeapCell* cell)
>+{
>+    // We get here once per GC under these circumstances:
>+    //
>+    // Eden collection: if the cell was allocated since the last collection and is live somehow.
>+    //
>+    // Full collection: if the cell is live somehow.
>+    
>+    CellContainer container = cell->cellContainer();
>+    
>+    container.setHasAnyMarked();
>+    
>+    m_visitCount++;
>+    m_bytesVisited += container.cellSize();
>+}
>+
> class SetCurrentCellScope {
> public:
>     SetCurrentCellScope(SlotVisitor& visitor, const JSCell* cell)
>@@ -202,9 +290,9 @@ private:
> ALWAYS_INLINE void SlotVisitor::visitChildren(const JSCell* cell)
> {
>     ASSERT(Heap::isMarked(cell));
>-
>+    
>     SetCurrentCellScope currentCellScope(*this, cell);
>-
>+    
>     m_currentObjectCellStateBeforeVisiting = cell->cellState();
>     cell->setCellState(CellState::OldBlack);
>     
>Index: Source/JavaScriptCore/heap/SlotVisitor.h
>===================================================================
>--- Source/JavaScriptCore/heap/SlotVisitor.h	(revision 204735)
>+++ Source/JavaScriptCore/heap/SlotVisitor.h	(working copy)
>@@ -37,8 +37,10 @@ namespace JSC {
> class ConservativeRoots;
> class GCThreadSharedData;
> class Heap;
>+class HeapCell;
> class HeapSnapshotBuilder;
> template<typename T> class JITWriteBarrier;
>+class MarkedBlock;
> class UnconditionalFinalizer;
> template<typename T> class Weak;
> class WeakReferenceHarvester;
>@@ -104,6 +106,10 @@ public:
> 
>     void harvestWeakReferences();
>     void finalizeUnconditionalFinalizers();
>+    
>+    // This informs the GC about auxiliary of some size that we are keeping alive. If you don't do
>+    // this then the space will be freed at end of GC.
>+    void markAuxiliary(const void* base);
> 
>     void copyLater(JSCell*, CopyToken, void*, size_t);
>     
>@@ -123,11 +129,21 @@ private:
>     friend class ParallelModeEnabler;
>     
>     JS_EXPORT_PRIVATE void append(JSValue); // This is private to encourage clients to use WriteBarrier<T>.
>+    void appendJSCellOrAuxiliary(HeapCell*);
>     void appendHidden(JSValue);
> 
>     JS_EXPORT_PRIVATE void setMarkedAndAppendToMarkStack(JSCell*);
>+    
>+    template<typename ContainerType>
>+    void setMarkedAndAppendToMarkStack(ContainerType&, JSCell*);
>+    
>     void appendToMarkStack(JSCell*);
>     
>+    template<typename ContainerType>
>+    void appendToMarkStack(ContainerType&, JSCell*);
>+    
>+    void noteLiveAuxiliaryCell(HeapCell*);
>+    
>     JS_EXPORT_PRIVATE void mergeOpaqueRoots();
>     void mergeOpaqueRootsIfNecessary();
>     void mergeOpaqueRootsIfProfitable();
>Index: Source/JavaScriptCore/heap/WeakBlock.cpp
>===================================================================
>--- Source/JavaScriptCore/heap/WeakBlock.cpp	(revision 204735)
>+++ Source/JavaScriptCore/heap/WeakBlock.cpp	(working copy)
>@@ -1,5 +1,5 @@
> /*
>- * Copyright (C) 2012 Apple Inc. All rights reserved.
>+ * Copyright (C) 2012, 2016 Apple Inc. All rights reserved.
>  *
>  * Redistribution and use in source and binary forms, with or without
>  * modification, are permitted provided that the following conditions
>@@ -26,6 +26,7 @@
> #include "config.h"
> #include "WeakBlock.h"
> 
>+#include "CellContainerInlines.h"
> #include "Heap.h"
> #include "HeapRootVisitor.h"
> #include "JSCInlines.h"
>@@ -34,10 +35,10 @@
> 
> namespace JSC {
> 
>-WeakBlock* WeakBlock::create(Heap& heap, MarkedBlock& markedBlock)
>+WeakBlock* WeakBlock::create(Heap& heap, CellContainer container)
> {
>     heap.didAllocateBlock(WeakBlock::blockSize);
>-    return new (NotNull, fastMalloc(blockSize)) WeakBlock(markedBlock);
>+    return new (NotNull, fastMalloc(blockSize)) WeakBlock(container);
> }
> 
> void WeakBlock::destroy(Heap& heap, WeakBlock* block)
>@@ -47,9 +48,9 @@ void WeakBlock::destroy(Heap& heap, Weak
>     heap.didFreeBlock(WeakBlock::blockSize);
> }
> 
>-WeakBlock::WeakBlock(MarkedBlock& markedBlock)
>+WeakBlock::WeakBlock(CellContainer container)
>     : DoublyLinkedListNode<WeakBlock>()
>-    , m_markedBlock(&markedBlock)
>+    , m_container(container)
> {
>     for (size_t i = 0; i < weakImplCount(); ++i) {
>         WeakImpl* weakImpl = &weakImpls()[i];
>@@ -101,11 +102,11 @@ void WeakBlock::visit(HeapRootVisitor& h
>     if (isEmpty())
>         return;
> 
>-    // If this WeakBlock doesn't belong to a MarkedBlock, we won't even be here.
>-    ASSERT(m_markedBlock);
>+    // If this WeakBlock doesn't belong to a CellContainer, we won't even be here.
>+    ASSERT(m_container);
> 
>     // We only visit after marking.
>-    ASSERT(m_markedBlock->isMarkedOrRetired());
>+    ASSERT(m_container.isMarkedOrRetired());
> 
>     SlotVisitor& visitor = heapRootVisitor.visitor();
> 
>@@ -119,7 +120,7 @@ void WeakBlock::visit(HeapRootVisitor& h
>             continue;
> 
>         const JSValue& jsValue = weakImpl->jsValue();
>-        if (m_markedBlock->isMarkedOrNewlyAllocated(jsValue.asCell()))
>+        if (m_container.isMarkedOrNewlyAllocated(jsValue.asCell()))
>             continue;
> 
>         if (!weakHandleOwner->isReachableFromOpaqueRoots(Handle<Unknown>::wrapSlot(&const_cast<JSValue&>(jsValue)), weakImpl->context(), visitor))
>@@ -135,18 +136,18 @@ void WeakBlock::reap()
>     if (isEmpty())
>         return;
> 
>-    // If this WeakBlock doesn't belong to a MarkedBlock, we won't even be here.
>-    ASSERT(m_markedBlock);
>+    // If this WeakBlock doesn't belong to a CellContainer, we won't even be here.
>+    ASSERT(m_container);
> 
>     // We only reap after marking.
>-    ASSERT(m_markedBlock->isMarkedOrRetired());
>+    ASSERT(m_container.isMarkedOrRetired());
> 
>     for (size_t i = 0; i < weakImplCount(); ++i) {
>         WeakImpl* weakImpl = &weakImpls()[i];
>         if (weakImpl->state() > WeakImpl::Dead)
>             continue;
> 
>-        if (m_markedBlock->isMarkedOrNewlyAllocated(weakImpl->jsValue().asCell())) {
>+        if (m_container.isMarkedOrNewlyAllocated(weakImpl->jsValue().asCell())) {
>             ASSERT(weakImpl->state() == WeakImpl::Live);
>             continue;
>         }
>Index: Source/JavaScriptCore/heap/WeakBlock.h
>===================================================================
>--- Source/JavaScriptCore/heap/WeakBlock.h	(revision 204735)
>+++ Source/JavaScriptCore/heap/WeakBlock.h	(working copy)
>@@ -1,5 +1,5 @@
> /*
>- * Copyright (C) 2012 Apple Inc. All rights reserved.
>+ * Copyright (C) 2012, 2016 Apple Inc. All rights reserved.
>  *
>  * Redistribution and use in source and binary forms, with or without
>  * modification, are permitted provided that the following conditions
>@@ -26,6 +26,7 @@
> #ifndef WeakBlock_h
> #define WeakBlock_h
> 
>+#include "CellContainer.h"
> #include "WeakImpl.h"
> #include <wtf/DoublyLinkedList.h>
> #include <wtf/StdLibExtras.h>
>@@ -34,7 +35,6 @@ namespace JSC {
> 
> class Heap;
> class HeapRootVisitor;
>-class MarkedBlock;
> 
> class WeakBlock : public DoublyLinkedListNode<WeakBlock> {
> public:
>@@ -53,7 +53,7 @@ public:
>         FreeCell* freeList { nullptr };
>     };
> 
>-    static WeakBlock* create(Heap&, MarkedBlock&);
>+    static WeakBlock* create(Heap&, CellContainer);
>     static void destroy(Heap&, WeakBlock*);
> 
>     static WeakImpl* asWeakImpl(FreeCell*);
>@@ -68,18 +68,18 @@ public:
>     void reap();
> 
>     void lastChanceToFinalize();
>-    void disconnectMarkedBlock() { m_markedBlock = nullptr; }
>+    void disconnectContainer() { m_container = CellContainer(); }
> 
> private:
>     static FreeCell* asFreeCell(WeakImpl*);
> 
>-    explicit WeakBlock(MarkedBlock&);
>+    explicit WeakBlock(CellContainer);
>     void finalize(WeakImpl*);
>     WeakImpl* weakImpls();
>     size_t weakImplCount();
>     void addToFreeList(FreeCell**, WeakImpl*);
> 
>-    MarkedBlock* m_markedBlock;
>+    CellContainer m_container;
>     WeakBlock* m_prev;
>     WeakBlock* m_next;
>     SweepResult m_sweepResult;
>Index: Source/JavaScriptCore/heap/WeakSet.cpp
>===================================================================
>--- Source/JavaScriptCore/heap/WeakSet.cpp	(revision 204735)
>+++ Source/JavaScriptCore/heap/WeakSet.cpp	(working copy)
>@@ -1,5 +1,5 @@
> /*
>- * Copyright (C) 2012 Apple Inc. All rights reserved.
>+ * Copyright (C) 2012, 2016 Apple Inc. All rights reserved.
>  *
>  * Redistribution and use in source and binary forms, with or without
>  * modification, are permitted provided that the following conditions
>@@ -53,10 +53,10 @@ void WeakSet::sweep()
>         if (block->isLogicallyEmptyButNotFree()) {
>             // If this WeakBlock is logically empty, but still has Weaks pointing into it,
>             // we can't destroy it just yet. Detach it from the WeakSet and hand ownership
>-            // to the Heap so we don't pin down the entire 64kB MarkedBlock.
>+            // to the Heap so we don't pin down the entire MarkedBlock or LargeAllocation.
>             m_blocks.remove(block);
>             heap()->addLogicallyEmptyWeakBlock(block);
>-            block->disconnectMarkedBlock();
>+            block->disconnectContainer();
>         }
>         block = nextBlock;
>     }
>@@ -88,7 +88,7 @@ WeakBlock::FreeCell* WeakSet::tryFindAll
> 
> WeakBlock::FreeCell* WeakSet::addAllocator()
> {
>-    WeakBlock* block = WeakBlock::create(*heap(), m_markedBlock);
>+    WeakBlock* block = WeakBlock::create(*heap(), m_container);
>     heap()->didAllocate(WeakBlock::blockSize);
>     m_blocks.append(block);
>     WeakBlock::SweepResult sweepResult = block->takeSweepResult();
>Index: Source/JavaScriptCore/heap/WeakSet.h
>===================================================================
>--- Source/JavaScriptCore/heap/WeakSet.h	(revision 204735)
>+++ Source/JavaScriptCore/heap/WeakSet.h	(working copy)
>@@ -1,5 +1,5 @@
> /*
>- * Copyright (C) 2012 Apple Inc. All rights reserved.
>+ * Copyright (C) 2012, 2016 Apple Inc. All rights reserved.
>  *
>  * Redistribution and use in source and binary forms, with or without
>  * modification, are permitted provided that the following conditions
>@@ -26,12 +26,12 @@
> #ifndef WeakSet_h
> #define WeakSet_h
> 
>+#include "CellContainer.h"
> #include "WeakBlock.h"
> 
> namespace JSC {
> 
> class Heap;
>-class MarkedBlock;
> class WeakImpl;
> 
> class WeakSet {
>@@ -41,7 +41,7 @@ public:
>     static WeakImpl* allocate(JSValue, WeakHandleOwner* = 0, void* context = 0);
>     static void deallocate(WeakImpl*);
> 
>-    WeakSet(VM*, MarkedBlock&);
>+    WeakSet(VM*, CellContainer);
>     ~WeakSet();
>     void lastChanceToFinalize();
> 
>@@ -66,14 +66,14 @@ private:
>     WeakBlock* m_nextAllocator;
>     DoublyLinkedList<WeakBlock> m_blocks;
>     VM* m_vm;
>-    MarkedBlock& m_markedBlock;
>+    CellContainer m_container;
> };
> 
>-inline WeakSet::WeakSet(VM* vm, MarkedBlock& markedBlock)
>+inline WeakSet::WeakSet(VM* vm, CellContainer container)
>     : m_allocator(0)
>     , m_nextAllocator(0)
>     , m_vm(vm)
>-    , m_markedBlock(markedBlock)
>+    , m_container(container)
> {
> }
> 
>Index: Source/JavaScriptCore/heap/WeakSetInlines.h
>===================================================================
>--- Source/JavaScriptCore/heap/WeakSetInlines.h	(revision 204735)
>+++ Source/JavaScriptCore/heap/WeakSetInlines.h	(working copy)
>@@ -1,5 +1,5 @@
> /*
>- * Copyright (C) 2012 Apple Inc. All rights reserved.
>+ * Copyright (C) 2012, 2016 Apple Inc. All rights reserved.
>  *
>  * Redistribution and use in source and binary forms, with or without
>  * modification, are permitted provided that the following conditions
>@@ -26,13 +26,14 @@
> #ifndef WeakSetInlines_h
> #define WeakSetInlines_h
> 
>+#include "CellContainerInlines.h"
> #include "MarkedBlock.h"
> 
> namespace JSC {
> 
> inline WeakImpl* WeakSet::allocate(JSValue jsValue, WeakHandleOwner* weakHandleOwner, void* context)
> {
>-    WeakSet& weakSet = MarkedBlock::blockFor(jsValue.asCell())->weakSet();
>+    WeakSet& weakSet = jsValue.asCell()->cellContainer().weakSet();
>     WeakBlock::FreeCell* allocator = weakSet.m_allocator;
>     if (UNLIKELY(!allocator))
>         allocator = weakSet.findAllocator();
>Index: Source/JavaScriptCore/inspector/InjectedScriptManager.cpp
>===================================================================
>--- Source/JavaScriptCore/inspector/InjectedScriptManager.cpp	(revision 204735)
>+++ Source/JavaScriptCore/inspector/InjectedScriptManager.cpp	(working copy)
>@@ -35,6 +35,7 @@
> #include "InjectedScriptHost.h"
> #include "InjectedScriptSource.h"
> #include "InspectorValues.h"
>+#include "JSCInlines.h"
> #include "JSInjectedScriptHost.h"
> #include "JSLock.h"
> #include "ScriptObject.h"
>Index: Source/JavaScriptCore/inspector/JSGlobalObjectInspectorController.cpp
>===================================================================
>--- Source/JavaScriptCore/inspector/JSGlobalObjectInspectorController.cpp	(revision 204735)
>+++ Source/JavaScriptCore/inspector/JSGlobalObjectInspectorController.cpp	(working copy)
>@@ -38,6 +38,7 @@
> #include "InspectorFrontendRouter.h"
> #include "InspectorHeapAgent.h"
> #include "InspectorScriptProfilerAgent.h"
>+#include "JSCInlines.h"
> #include "JSGlobalObject.h"
> #include "JSGlobalObjectConsoleAgent.h"
> #include "JSGlobalObjectConsoleClient.h"
>Index: Source/JavaScriptCore/inspector/JSJavaScriptCallFrame.cpp
>===================================================================
>--- Source/JavaScriptCore/inspector/JSJavaScriptCallFrame.cpp	(revision 204735)
>+++ Source/JavaScriptCore/inspector/JSJavaScriptCallFrame.cpp	(working copy)
>@@ -29,11 +29,9 @@
> #include "DebuggerScope.h"
> #include "Error.h"
> #include "IdentifierInlines.h"
>-#include "JSCJSValue.h"
>-#include "JSCellInlines.h"
>+#include "JSCInlines.h"
> #include "JSJavaScriptCallFramePrototype.h"
> #include "ObjectConstructor.h"
>-#include "StructureInlines.h"
> 
> using namespace JSC;
> 
>Index: Source/JavaScriptCore/inspector/ScriptDebugServer.cpp
>===================================================================
>--- Source/JavaScriptCore/inspector/ScriptDebugServer.cpp	(revision 204735)
>+++ Source/JavaScriptCore/inspector/ScriptDebugServer.cpp	(working copy)
>@@ -34,6 +34,7 @@
> #include "DebuggerCallFrame.h"
> #include "DebuggerScope.h"
> #include "Exception.h"
>+#include "JSCInlines.h"
> #include "JSJavaScriptCallFrame.h"
> #include "JSLock.h"
> #include "JavaScriptCallFrame.h"
>Index: Source/JavaScriptCore/inspector/agents/InspectorDebuggerAgent.cpp
>===================================================================
>--- Source/JavaScriptCore/inspector/agents/InspectorDebuggerAgent.cpp	(revision 204735)
>+++ Source/JavaScriptCore/inspector/agents/InspectorDebuggerAgent.cpp	(working copy)
>@@ -35,6 +35,7 @@
> #include "InjectedScriptManager.h"
> #include "InspectorFrontendRouter.h"
> #include "InspectorValues.h"
>+#include "JSCInlines.h"
> #include "RegularExpression.h"
> #include "ScriptDebugServer.h"
> #include "ScriptObject.h"
>Index: Source/JavaScriptCore/interpreter/CachedCall.h
>===================================================================
>--- Source/JavaScriptCore/interpreter/CachedCall.h	(revision 204735)
>+++ Source/JavaScriptCore/interpreter/CachedCall.h	(working copy)
>@@ -42,10 +42,11 @@ namespace JSC {
>         CachedCall(CallFrame* callFrame, JSFunction* function, int argumentCount)
>             : m_valid(false)
>             , m_interpreter(callFrame->interpreter())
>-            , m_entryScope(callFrame->vm(), function->scope()->globalObject())
>+            , m_vm(callFrame->vm())
>+            , m_entryScope(m_vm, function->scope()->globalObject(m_vm))
>         {
>             ASSERT(!function->isHostFunctionNonInline());
>-            if (UNLIKELY(callFrame->vm().isSafeToRecurseSoft())) {
>+            if (UNLIKELY(m_vm.isSafeToRecurseSoft())) {
>                 m_arguments.resize(argumentCount);
>                 m_closure = m_interpreter->prepareForRepeatCall(function->jsExecutable(), callFrame, &m_protoCallFrame, function, argumentCount + 1, function->scope(), m_arguments.data());
>             } else
>@@ -64,6 +65,7 @@ namespace JSC {
>     private:
>         bool m_valid;
>         Interpreter* m_interpreter;
>+        VM& m_vm;
>         VMEntryScope m_entryScope;
>         ProtoCallFrame m_protoCallFrame;
>         Vector<JSValue> m_arguments;
>Index: Source/JavaScriptCore/interpreter/Interpreter.cpp
>===================================================================
>--- Source/JavaScriptCore/interpreter/Interpreter.cpp	(revision 204735)
>+++ Source/JavaScriptCore/interpreter/Interpreter.cpp	(working copy)
>@@ -51,6 +51,7 @@
> #include "JSString.h"
> #include "JSWithScope.h"
> #include "LLIntCLoop.h"
>+#include "LLIntData.h"
> #include "LLIntThunks.h"
> #include "LiteralParser.h"
> #include "ObjectPrototype.h"
>@@ -85,46 +86,6 @@ using namespace std;
> 
> namespace JSC {
> 
>-intptr_t StackFrame::sourceID() const
>-{
>-    if (!codeBlock)
>-        return noSourceID;
>-    return codeBlock->ownerScriptExecutable()->sourceID();
>-}
>-
>-String StackFrame::sourceURL() const
>-{
>-    if (!codeBlock)
>-        return ASCIILiteral("[native code]");
>-
>-    String sourceURL = codeBlock->ownerScriptExecutable()->sourceURL();
>-    if (!sourceURL.isNull())
>-        return sourceURL;
>-    return emptyString();
>-}
>-
>-String StackFrame::functionName(VM& vm) const
>-{
>-    if (codeBlock) {
>-        switch (codeBlock->codeType()) {
>-        case EvalCode:
>-            return ASCIILiteral("eval code");
>-        case ModuleCode:
>-            return ASCIILiteral("module code");
>-        case FunctionCode:
>-            break;
>-        case GlobalCode:
>-            return ASCIILiteral("global code");
>-        default:
>-            ASSERT_NOT_REACHED();
>-        }
>-    }
>-    String name;
>-    if (callee)
>-        name = getCalculatedDisplayName(vm, callee.get()).impl();
>-    return name.isNull() ? emptyString() : name;
>-}
>-
> JSValue eval(CallFrame* callFrame)
> {
>     if (!callFrame->argumentCount())
>@@ -264,6 +225,7 @@ void loadVarargs(CallFrame* callFrame, V
>         return;
>     
>     JSCell* cell = arguments.asCell();
>+
>     switch (cell->type()) {
>     case DirectArgumentsType:
>         jsCast<DirectArguments*>(cell)->copyToArguments(callFrame, firstElementDest, offset, length);
>@@ -472,48 +434,6 @@ bool Interpreter::isOpcode(Opcode opcode
> #endif
> }
> 
>-void StackFrame::computeLineAndColumn(unsigned& line, unsigned& column) const
>-{
>-    if (!codeBlock) {
>-        line = 0;
>-        column = 0;
>-        return;
>-    }
>-
>-    int divot = 0;
>-    int unusedStartOffset = 0;
>-    int unusedEndOffset = 0;
>-    codeBlock->expressionRangeForBytecodeOffset(bytecodeOffset, divot, unusedStartOffset, unusedEndOffset, line, column);
>-
>-    ScriptExecutable* executable = codeBlock->ownerScriptExecutable();
>-    if (executable->hasOverrideLineNumber())
>-        line = executable->overrideLineNumber();
>-}
>-
>-String StackFrame::toString(VM& vm) const
>-{
>-    StringBuilder traceBuild;
>-    String functionName = this->functionName(vm);
>-    String sourceURL = this->sourceURL();
>-    traceBuild.append(functionName);
>-    if (!sourceURL.isEmpty()) {
>-        if (!functionName.isEmpty())
>-            traceBuild.append('@');
>-        traceBuild.append(sourceURL);
>-        if (codeBlock) {
>-            unsigned line;
>-            unsigned column;
>-            computeLineAndColumn(line, column);
>-
>-            traceBuild.append(':');
>-            traceBuild.appendNumber(line);
>-            traceBuild.append(':');
>-            traceBuild.appendNumber(column);
>-        }
>-    }
>-    return traceBuild.toString().impl();
>-}
>-
> static inline bool isWebAssemblyExecutable(ExecutableBase* executable)
> {
> #if !ENABLE(WEBASSEMBLY)
>Index: Source/JavaScriptCore/interpreter/Interpreter.h
>===================================================================
>--- Source/JavaScriptCore/interpreter/Interpreter.h	(revision 204735)
>+++ Source/JavaScriptCore/interpreter/Interpreter.h	(working copy)
>@@ -37,6 +37,7 @@
> #include "Opcode.h"
> #include "SourceProvider.h"
> #include "StackAlignment.h"
>+#include "StackFrame.h"
> #include <wtf/HashMap.h>
> #include <wtf/text/StringBuilder.h>
> 
>@@ -66,7 +67,7 @@ namespace JSC {
>     struct Instruction;
>     struct ProtoCallFrame;
> 
>-    enum UnwindStart { UnwindFromCurrentFrame, UnwindFromCallerFrame };
>+    enum UnwindStart : uint8_t { UnwindFromCurrentFrame, UnwindFromCallerFrame };
> 
>     enum DebugHookID {
>         WillExecuteProgram,
>@@ -85,20 +86,6 @@ namespace JSC {
>         StackFrameNativeCode
>     };
> 
>-    struct StackFrame {
>-        Strong<JSObject> callee;
>-        Strong<CodeBlock> codeBlock;
>-        unsigned bytecodeOffset;
>-
>-        bool isNative() const { return !codeBlock; }
>-
>-        void computeLineAndColumn(unsigned& line, unsigned& column) const;
>-        String functionName(VM&) const;
>-        intptr_t sourceID() const;
>-        String sourceURL() const;
>-        String toString(VM&) const;
>-    };
>-
>     class SuspendExceptionScope {
>     public:
>         SuspendExceptionScope(VM* vm)
>Index: Source/JavaScriptCore/jit/AssemblyHelpers.h
>===================================================================
>--- Source/JavaScriptCore/jit/AssemblyHelpers.h	(revision 204735)
>+++ Source/JavaScriptCore/jit/AssemblyHelpers.h	(working copy)
>@@ -1405,33 +1405,68 @@ public:
>     void emitRandomThunk(JSGlobalObject*, GPRReg scratch0, GPRReg scratch1, GPRReg scratch2, FPRReg result);
>     void emitRandomThunk(GPRReg scratch0, GPRReg scratch1, GPRReg scratch2, GPRReg scratch3, FPRReg result);
> #endif
>-    
>-    void emitAllocate(GPRReg resultGPR, GPRReg allocatorGPR, GPRReg scratchGPR, JumpList& slowPath)
>+
>+    // Call this if you know that the value held in allocatorGPR is non-null. This DOES NOT mean
>+    // that allocator is non-null; allocator can be null as a signal that we don't know what the
>+    // value of allocatorGPR is.
>+    void emitAllocateWithNonNullAllocator(GPRReg resultGPR, MarkedAllocator* allocator, GPRReg allocatorGPR, GPRReg scratchGPR, JumpList& slowPath)
>     {
>-        if (Options::forceGCSlowPaths())
>+        // NOTE: This is carefully written so that we can call it while we disallow scratch
>+        // register usage.
>+        
>+        if (Options::forceGCSlowPaths()) {
>             slowPath.append(jump());
>+            return;
>+        }
>+        
>+        Jump popPath;
>+        Jump done;
>+        
>+        load32(Address(allocatorGPR, MarkedAllocator::offsetOfFreeList() + OBJECT_OFFSETOF(FreeList, remaining)), resultGPR);
>+        popPath = branchTest32(Zero, resultGPR);
>+        if (allocator)
>+            add32(TrustedImm32(-allocator->cellSize()), resultGPR, scratchGPR);
>         else {
>-            loadPtr(Address(allocatorGPR, MarkedAllocator::offsetOfFreeListHead()), resultGPR);
>-            slowPath.append(branchTestPtr(Zero, resultGPR));
>+            move(resultGPR, scratchGPR);
>+            sub32(Address(allocatorGPR, MarkedAllocator::offsetOfCellSize()), scratchGPR);
>         }
>+        negPtr(resultGPR);
>+        store32(scratchGPR, Address(allocatorGPR, MarkedAllocator::offsetOfFreeList() + OBJECT_OFFSETOF(FreeList, remaining)));
>+        addPtr(Address(allocatorGPR, MarkedAllocator::offsetOfFreeList() + OBJECT_OFFSETOF(FreeList, payloadEnd)), resultGPR);
>+        
>+        done = jump();
>+        
>+        popPath.link(this);
>+        
>+        loadPtr(Address(allocatorGPR, MarkedAllocator::offsetOfFreeList() + OBJECT_OFFSETOF(FreeList, head)), resultGPR);
>+        slowPath.append(branchTestPtr(Zero, resultGPR));
>         
>         // The object is half-allocated: we have what we know is a fresh object, but
>         // it's still on the GC's free list.
>         loadPtr(Address(resultGPR), scratchGPR);
>-        storePtr(scratchGPR, Address(allocatorGPR, MarkedAllocator::offsetOfFreeListHead()));
>+        storePtr(scratchGPR, Address(allocatorGPR, MarkedAllocator::offsetOfFreeList() + OBJECT_OFFSETOF(FreeList, head)));
>+        
>+        done.link(this);
>+    }
>+    
>+    void emitAllocate(GPRReg resultGPR, MarkedAllocator* allocator, GPRReg allocatorGPR, GPRReg scratchGPR, JumpList& slowPath)
>+    {
>+        if (!allocator)
>+            slowPath.append(branchTestPtr(Zero, allocatorGPR));
>+        emitAllocateWithNonNullAllocator(resultGPR, allocator, allocatorGPR, scratchGPR, slowPath);
>     }
>     
>     template<typename StructureType>
>-    void emitAllocateJSCell(GPRReg resultGPR, GPRReg allocatorGPR, StructureType structure, GPRReg scratchGPR, JumpList& slowPath)
>+    void emitAllocateJSCell(GPRReg resultGPR, MarkedAllocator* allocator, GPRReg allocatorGPR, StructureType structure, GPRReg scratchGPR, JumpList& slowPath)
>     {
>-        emitAllocate(resultGPR, allocatorGPR, scratchGPR, slowPath);
>+        emitAllocate(resultGPR, allocator, allocatorGPR, scratchGPR, slowPath);
>         emitStoreStructureWithTypeInfo(structure, resultGPR, scratchGPR);
>     }
>     
>     template<typename StructureType, typename StorageType>
>-    void emitAllocateJSObject(GPRReg resultGPR, GPRReg allocatorGPR, StructureType structure, StorageType storage, GPRReg scratchGPR, JumpList& slowPath)
>+    void emitAllocateJSObject(GPRReg resultGPR, MarkedAllocator* allocator, GPRReg allocatorGPR, StructureType structure, StorageType storage, GPRReg scratchGPR, JumpList& slowPath)
>     {
>-        emitAllocateJSCell(resultGPR, allocatorGPR, structure, scratchGPR, slowPath);
>+        emitAllocateJSCell(resultGPR, allocator, allocatorGPR, structure, scratchGPR, slowPath);
>         storePtr(storage, Address(resultGPR, JSObject::butterflyOffset()));
>     }
>     
>@@ -1440,9 +1475,13 @@ public:
>         GPRReg resultGPR, StructureType structure, StorageType storage, GPRReg scratchGPR1,
>         GPRReg scratchGPR2, JumpList& slowPath, size_t size)
>     {
>-        MarkedAllocator* allocator = &vm()->heap.allocatorForObjectOfType<ClassType>(size);
>+        MarkedAllocator* allocator = vm()->heap.allocatorForObjectOfType<ClassType>(size);
>+        if (!allocator) {
>+            slowPath.append(jump());
>+            return;
>+        }
>         move(TrustedImmPtr(allocator), scratchGPR1);
>-        emitAllocateJSObject(resultGPR, scratchGPR1, structure, storage, scratchGPR2, slowPath);
>+        emitAllocateJSObject(resultGPR, allocator, scratchGPR1, structure, storage, scratchGPR2, slowPath);
>     }
>     
>     template<typename ClassType, typename StructureType, typename StorageType>
>@@ -1451,27 +1490,21 @@ public:
>         emitAllocateJSObjectWithKnownSize<ClassType>(resultGPR, structure, storage, scratchGPR1, scratchGPR2, slowPath, ClassType::allocationSize(0));
>     }
>     
>+    // allocationSize can be aliased with any of the other input GPRs. If it's not aliased then it
>+    // won't be clobbered.
>     void emitAllocateVariableSized(GPRReg resultGPR, MarkedSpace::Subspace& subspace, GPRReg allocationSize, GPRReg scratchGPR1, GPRReg scratchGPR2, JumpList& slowPath)
>     {
>-        static_assert(!(MarkedSpace::preciseStep & (MarkedSpace::preciseStep - 1)), "MarkedSpace::preciseStep must be a power of two.");
>-        static_assert(!(MarkedSpace::impreciseStep & (MarkedSpace::impreciseStep - 1)), "MarkedSpace::impreciseStep must be a power of two.");
>+        static_assert(!(MarkedSpace::sizeStep & (MarkedSpace::sizeStep - 1)), "MarkedSpace::sizeStep must be a power of two.");
>         
>-        add32(TrustedImm32(MarkedSpace::preciseStep - 1), allocationSize);
>-        Jump notSmall = branch32(AboveOrEqual, allocationSize, TrustedImm32(MarkedSpace::preciseCutoff));
>-        rshift32(allocationSize, TrustedImm32(getLSBSet(MarkedSpace::preciseStep)), scratchGPR1);
>-        mul32(TrustedImm32(sizeof(MarkedAllocator)), scratchGPR1, scratchGPR1);
>-        addPtr(TrustedImmPtr(&subspace.preciseAllocators[0]), scratchGPR1);
>-
>-        Jump selectedSmallSpace = jump();
>-        notSmall.link(this);
>-        slowPath.append(branch32(AboveOrEqual, allocationSize, TrustedImm32(MarkedSpace::impreciseCutoff)));
>-        rshift32(allocationSize, TrustedImm32(getLSBSet(MarkedSpace::impreciseStep)), scratchGPR1);
>-        mul32(TrustedImm32(sizeof(MarkedAllocator)), scratchGPR1, scratchGPR1);
>-        addPtr(TrustedImmPtr(&subspace.impreciseAllocators[0]), scratchGPR1);
>-
>-        selectedSmallSpace.link(this);
>+        unsigned stepShift = getLSBSet(MarkedSpace::sizeStep);
>+        
>+        add32(TrustedImm32(MarkedSpace::sizeStep - 1), allocationSize, scratchGPR1);
>+        urshift32(TrustedImm32(stepShift), scratchGPR1);
>+        slowPath.append(branch32(Above, scratchGPR1, TrustedImm32(MarkedSpace::largeCutoff >> stepShift)));
>+        move(TrustedImmPtr(&subspace.allocatorForSizeStep[0] - 1), scratchGPR2);
>+        loadPtr(BaseIndex(scratchGPR2, scratchGPR1, timesPtr()), scratchGPR1);
>         
>-        emitAllocate(resultGPR, scratchGPR1, scratchGPR2, slowPath);
>+        emitAllocate(resultGPR, nullptr, scratchGPR1, scratchGPR2, slowPath);
>     }
>     
>     template<typename ClassType, typename StructureType>
>Index: Source/JavaScriptCore/jit/GCAwareJITStubRoutine.cpp
>===================================================================
>--- Source/JavaScriptCore/jit/GCAwareJITStubRoutine.cpp	(revision 204735)
>+++ Source/JavaScriptCore/jit/GCAwareJITStubRoutine.cpp	(working copy)
>@@ -32,6 +32,7 @@
> #include "DFGCommonData.h"
> #include "Heap.h"
> #include "VM.h"
>+#include "JITStubRoutineSet.h"
> #include "JSCInlines.h"
> #include "SlotVisitor.h"
> #include "Structure.h"
>@@ -45,7 +46,7 @@ GCAwareJITStubRoutine::GCAwareJITStubRou
>     , m_mayBeExecuting(false)
>     , m_isJettisoned(false)
> {
>-    vm.heap.m_jitStubRoutines.add(this);
>+    vm.heap.m_jitStubRoutines->add(this);
> }
> 
> GCAwareJITStubRoutine::~GCAwareJITStubRoutine() { }
>Index: Source/JavaScriptCore/jit/JIT.cpp
>===================================================================
>--- Source/JavaScriptCore/jit/JIT.cpp	(revision 204735)
>+++ Source/JavaScriptCore/jit/JIT.cpp	(working copy)
>@@ -66,6 +66,14 @@ void ctiPatchCallByReturnAddress(ReturnA
>         newCalleeFunction);
> }
> 
>+JIT::CodeRef JIT::compileCTINativeCall(VM* vm, NativeFunction func)
>+{
>+    if (!vm->canUseJIT())
>+        return CodeRef::createLLIntCodeRef(llint_native_call_trampoline);
>+    JIT jit(vm, 0);
>+    return jit.privateCompileCTINativeCall(vm, func);
>+}
>+
> JIT::JIT(VM* vm, CodeBlock* codeBlock)
>     : JSInterfaceJIT(vm, codeBlock)
>     , m_interpreter(vm->interpreter)
>Index: Source/JavaScriptCore/jit/JIT.h
>===================================================================
>--- Source/JavaScriptCore/jit/JIT.h	(revision 204735)
>+++ Source/JavaScriptCore/jit/JIT.h	(working copy)
>@@ -40,17 +40,17 @@
> 
> #include "CodeBlock.h"
> #include "CompactJITCodeMap.h"
>-#include "Interpreter.h"
> #include "JITDisassembler.h"
> #include "JITInlineCacheGenerator.h"
> #include "JITMathIC.h"
> #include "JSInterfaceJIT.h"
>-#include "Opcode.h"
> #include "PCToCodeOriginMap.h"
> #include "UnusedPointer.h"
> 
> namespace JSC {
> 
>+    enum OpcodeID : unsigned;
>+
>     class ArrayAllocationProfile;
>     class CallLinkInfo;
>     class CodeBlock;
>@@ -248,14 +248,7 @@ namespace JSC {
>             jit.privateCompileHasIndexedProperty(byValInfo, returnAddress, arrayMode);
>         }
> 
>-        static CodeRef compileCTINativeCall(VM* vm, NativeFunction func)
>-        {
>-            if (!vm->canUseJIT()) {
>-                return CodeRef::createLLIntCodeRef(llint_native_call_trampoline);
>-            }
>-            JIT jit(vm, 0);
>-            return jit.privateCompileCTINativeCall(vm, func);
>-        }
>+        static CodeRef compileCTINativeCall(VM*, NativeFunction);
> 
>         static unsigned frameRegisterCountFor(CodeBlock*);
>         static int stackPointerOffsetFor(CodeBlock*);
>Index: Source/JavaScriptCore/jit/JITExceptions.cpp
>===================================================================
>--- Source/JavaScriptCore/jit/JITExceptions.cpp	(revision 204735)
>+++ Source/JavaScriptCore/jit/JITExceptions.cpp	(working copy)
>@@ -90,4 +90,9 @@ void genericUnwind(VM* vm, ExecState* ca
>     RELEASE_ASSERT(catchRoutine);
> }
> 
>+void genericUnwind(VM* vm, ExecState* callFrame)
>+{
>+    genericUnwind(vm, callFrame, UnwindFromCurrentFrame);
>+}
>+
> } // namespace JSC
>Index: Source/JavaScriptCore/jit/JITExceptions.h
>===================================================================
>--- Source/JavaScriptCore/jit/JITExceptions.h	(revision 204735)
>+++ Source/JavaScriptCore/jit/JITExceptions.h	(working copy)
>@@ -26,15 +26,15 @@
> #ifndef JITExceptions_h
> #define JITExceptions_h
> 
>-#include "Interpreter.h"
>-#include "JSCJSValue.h"
>-
> namespace JSC {
> 
>+enum UnwindStart : uint8_t;
>+
> class ExecState;
> class VM;
> 
>-void genericUnwind(VM*, ExecState*, UnwindStart = UnwindFromCurrentFrame);
>+void genericUnwind(VM*, ExecState*, UnwindStart);
>+void genericUnwind(VM*, ExecState*);
> 
> } // namespace JSC
> 
>Index: Source/JavaScriptCore/jit/JITOpcodes32_64.cpp
>===================================================================
>--- Source/JavaScriptCore/jit/JITOpcodes32_64.cpp	(revision 204735)
>+++ Source/JavaScriptCore/jit/JITOpcodes32_64.cpp	(working copy)
>@@ -39,6 +39,7 @@
> #include "JSPropertyNameEnumerator.h"
> #include "LinkBuffer.h"
> #include "MaxFrameExtentForSlowPathCall.h"
>+#include "Opcode.h"
> #include "SlowPathCall.h"
> #include "TypeProfilerLog.h"
> #include "VirtualRegister.h"
>@@ -163,15 +164,17 @@ void JIT::emit_op_new_object(Instruction
> {
>     Structure* structure = currentInstruction[3].u.objectAllocationProfile->structure();
>     size_t allocationSize = JSFinalObject::allocationSize(structure->inlineCapacity());
>-    MarkedAllocator* allocator = &m_vm->heap.allocatorForObjectWithoutDestructor(allocationSize);
>+    MarkedAllocator* allocator = m_vm->heap.allocatorForObjectWithoutDestructor(allocationSize);
> 
>     RegisterID resultReg = returnValueGPR;
>     RegisterID allocatorReg = regT1;
>     RegisterID scratchReg = regT3;
> 
>     move(TrustedImmPtr(allocator), allocatorReg);
>+    if (allocator)
>+        addSlowCase(Jump());
>     JumpList slowCases;
>-    emitAllocateJSObject(resultReg, allocatorReg, TrustedImmPtr(structure), TrustedImmPtr(0), scratchReg, slowCases);
>+    emitAllocateJSObject(resultReg, allocator, allocatorReg, TrustedImmPtr(structure), TrustedImmPtr(0), scratchReg, slowCases);
>     addSlowCase(slowCases);
>     emitStoreCell(currentInstruction[1].u.operand, resultReg);
> }
>@@ -179,6 +182,7 @@ void JIT::emit_op_new_object(Instruction
> void JIT::emitSlow_op_new_object(Instruction* currentInstruction, Vector<SlowCaseEntry>::iterator& iter)
> {
>     linkSlowCase(iter);
>+    linkSlowCase(iter);
>     int dst = currentInstruction[1].u.operand;
>     Structure* structure = currentInstruction[3].u.objectAllocationProfile->structure();
>     callOperation(operationNewObject, structure);
>@@ -1032,7 +1036,7 @@ void JIT::emit_op_create_this(Instructio
>     hasSeenMultipleCallees.link(this);
> 
>     JumpList slowCases;
>-    emitAllocateJSObject(resultReg, allocatorReg, structureReg, TrustedImmPtr(0), scratchReg, slowCases);
>+    emitAllocateJSObject(resultReg, nullptr, allocatorReg, structureReg, TrustedImmPtr(0), scratchReg, slowCases);
>     addSlowCase(slowCases);
>     emitStoreCell(currentInstruction[1].u.operand, resultReg);
> }
>@@ -1042,7 +1046,8 @@ void JIT::emitSlow_op_create_this(Instru
>     linkSlowCase(iter); // Callee::m_type != JSFunctionType.
>     linkSlowCase(iter); // doesn't have rare data
>     linkSlowCase(iter); // doesn't have an allocation profile
>-    linkSlowCase(iter); // allocation failed
>+    linkSlowCase(iter); // allocation failed (no allocator)
>+    linkSlowCase(iter); // allocation failed (allocator empty)
>     linkSlowCase(iter); // cached function didn't match
> 
>     JITSlowPathCall slowPathCall(this, currentInstruction, slow_path_create_this);
>Index: Source/JavaScriptCore/jit/JITOpcodes.cpp
>===================================================================
>--- Source/JavaScriptCore/jit/JITOpcodes.cpp	(revision 204735)
>+++ Source/JavaScriptCore/jit/JITOpcodes.cpp	(working copy)
>@@ -32,6 +32,7 @@
> #include "CopiedSpaceInlines.h"
> #include "Exception.h"
> #include "Heap.h"
>+#include "Interpreter.h"
> #include "JITInlines.h"
> #include "JSArray.h"
> #include "JSCell.h"
>@@ -83,15 +84,17 @@ void JIT::emit_op_new_object(Instruction
> {
>     Structure* structure = currentInstruction[3].u.objectAllocationProfile->structure();
>     size_t allocationSize = JSFinalObject::allocationSize(structure->inlineCapacity());
>-    MarkedAllocator* allocator = &m_vm->heap.allocatorForObjectWithoutDestructor(allocationSize);
>+    MarkedAllocator* allocator = m_vm->heap.allocatorForObjectWithoutDestructor(allocationSize);
> 
>     RegisterID resultReg = regT0;
>     RegisterID allocatorReg = regT1;
>     RegisterID scratchReg = regT2;
> 
>     move(TrustedImmPtr(allocator), allocatorReg);
>+    if (allocator)
>+        addSlowCase(Jump());
>     JumpList slowCases;
>-    emitAllocateJSObject(resultReg, allocatorReg, TrustedImmPtr(structure), TrustedImmPtr(0), scratchReg, slowCases);
>+    emitAllocateJSObject(resultReg, allocator, allocatorReg, TrustedImmPtr(structure), TrustedImmPtr(0), scratchReg, slowCases);
>     addSlowCase(slowCases);
>     emitPutVirtualRegister(currentInstruction[1].u.operand);
> }
>@@ -99,6 +102,7 @@ void JIT::emit_op_new_object(Instruction
> void JIT::emitSlow_op_new_object(Instruction* currentInstruction, Vector<SlowCaseEntry>::iterator& iter)
> {
>     linkSlowCase(iter);
>+    linkSlowCase(iter);
>     int dst = currentInstruction[1].u.operand;
>     Structure* structure = currentInstruction[3].u.objectAllocationProfile->structure();
>     callOperation(operationNewObject, structure);
>@@ -772,7 +776,7 @@ void JIT::emit_op_create_this(Instructio
>     hasSeenMultipleCallees.link(this);
> 
>     JumpList slowCases;
>-    emitAllocateJSObject(resultReg, allocatorReg, structureReg, TrustedImmPtr(0), scratchReg, slowCases);
>+    emitAllocateJSObject(resultReg, nullptr, allocatorReg, structureReg, TrustedImmPtr(0), scratchReg, slowCases);
>     addSlowCase(slowCases);
>     emitPutVirtualRegister(currentInstruction[1].u.operand);
> }
>@@ -782,7 +786,8 @@ void JIT::emitSlow_op_create_this(Instru
>     linkSlowCase(iter); // Callee::m_type != JSFunctionType.
>     linkSlowCase(iter); // doesn't have rare data
>     linkSlowCase(iter); // doesn't have an allocation profile
>-    linkSlowCase(iter); // allocation failed
>+    linkSlowCase(iter); // allocation failed (no allocator)
>+    linkSlowCase(iter); // allocation failed (allocator empty)
>     linkSlowCase(iter); // cached function didn't match
> 
>     JITSlowPathCall slowPathCall(this, currentInstruction, slow_path_create_this);
>Index: Source/JavaScriptCore/jit/JITOperations.cpp
>===================================================================
>--- Source/JavaScriptCore/jit/JITOperations.cpp	(revision 204735)
>+++ Source/JavaScriptCore/jit/JITOperations.cpp	(working copy)
>@@ -44,6 +44,7 @@
> #include "GetterSetter.h"
> #include "HostCallReturnValue.h"
> #include "ICStats.h"
>+#include "Interpreter.h"
> #include "JIT.h"
> #include "JITExceptions.h"
> #include "JITToDFGDeferredCompilationCallback.h"
>@@ -471,17 +472,6 @@ void JIT_OPERATION operationPutByIdDirec
>         repatchPutByID(exec, baseObject, structure, ident, slot, *stubInfo, Direct);
> }
> 
>-void JIT_OPERATION operationReallocateStorageAndFinishPut(ExecState* exec, JSObject* base, Structure* structure, PropertyOffset offset, EncodedJSValue value)
>-{
>-    VM& vm = exec->vm();
>-    NativeCallFrameTracer tracer(&vm, exec);
>-
>-    ASSERT(structure->outOfLineCapacity() > base->structure(vm)->outOfLineCapacity());
>-    ASSERT(!vm.heap.storageAllocator().fastPathShouldSucceed(structure->outOfLineCapacity() * sizeof(JSValue)));
>-    base->setStructureAndReallocateStorageIfNecessary(vm, structure);
>-    base->putDirect(vm, offset, JSValue::decode(value));
>-}
>-
> ALWAYS_INLINE static bool isStringOrSymbol(JSValue value)
> {
>     return value.isString() || value.isSymbol();
>Index: Source/JavaScriptCore/jit/JITOperations.h
>===================================================================
>--- Source/JavaScriptCore/jit/JITOperations.h	(revision 204735)
>+++ Source/JavaScriptCore/jit/JITOperations.h	(working copy)
>@@ -38,6 +38,8 @@
> 
> namespace JSC {
> 
>+typedef int64_t EncodedJSValue;
>+    
> class ArrayAllocationProfile;
> class ArrayProfile;
> class CallLinkInfo;
>@@ -45,17 +47,24 @@ class CodeBlock;
> class ExecState;
> class JITAddGenerator;
> class JSArray;
>+class JSCell;
> class JSFunction;
>+class JSGlobalObject;
> class JSLexicalEnvironment;
>+class JSObject;
> class JSScope;
>+class JSString;
>+class JSValue;
> class RegExpObject;
> class Register;
>+class Structure;
> class StructureStubInfo;
> class SymbolTable;
> class WatchpointSet;
> 
> struct ByValInfo;
> struct InlineCallFrame;
>+struct Instruction;
> struct ArithProfile;
> 
> typedef ExecState CallFrame;
>@@ -320,7 +329,6 @@ void JIT_OPERATION operationPutByIdStric
> void JIT_OPERATION operationPutByIdNonStrictBuildList(ExecState*, StructureStubInfo*, EncodedJSValue encodedValue, EncodedJSValue encodedBase, UniquedStringImpl*) WTF_INTERNAL;
> void JIT_OPERATION operationPutByIdDirectStrictBuildList(ExecState*, StructureStubInfo*, EncodedJSValue encodedValue, EncodedJSValue encodedBase, UniquedStringImpl*) WTF_INTERNAL;
> void JIT_OPERATION operationPutByIdDirectNonStrictBuildList(ExecState*, StructureStubInfo*, EncodedJSValue encodedValue, EncodedJSValue encodedBase, UniquedStringImpl*) WTF_INTERNAL;
>-void JIT_OPERATION operationReallocateStorageAndFinishPut(ExecState*, JSObject*, Structure*, PropertyOffset, EncodedJSValue) WTF_INTERNAL;
> void JIT_OPERATION operationPutByValOptimize(ExecState*, EncodedJSValue, EncodedJSValue, EncodedJSValue, ByValInfo*) WTF_INTERNAL;
> void JIT_OPERATION operationDirectPutByValOptimize(ExecState*, EncodedJSValue, EncodedJSValue, EncodedJSValue, ByValInfo*) WTF_INTERNAL;
> void JIT_OPERATION operationPutByValGeneric(ExecState*, EncodedJSValue, EncodedJSValue, EncodedJSValue, ByValInfo*) WTF_INTERNAL;
>Index: Source/JavaScriptCore/jit/JITPropertyAccess.cpp
>===================================================================
>--- Source/JavaScriptCore/jit/JITPropertyAccess.cpp	(revision 204735)
>+++ Source/JavaScriptCore/jit/JITPropertyAccess.cpp	(working copy)
>@@ -1245,7 +1245,7 @@ void JIT::emitWriteBarrier(JSCell* owner
> 
> void JIT::emitWriteBarrier(JSCell* owner)
> {
>-    if (!MarkedBlock::blockFor(owner)->isMarked(owner)) {
>+    if (!owner->cellContainer().isMarked(owner)) {
>         Jump ownerIsRememberedOrInEden = jumpIfIsRememberedOrInEden(owner);
>         callOperation(operationUnconditionalWriteBarrier, owner);
>         ownerIsRememberedOrInEden.link(this);
>Index: Source/JavaScriptCore/jit/JITThunks.cpp
>===================================================================
>--- Source/JavaScriptCore/jit/JITThunks.cpp	(revision 204735)
>+++ Source/JavaScriptCore/jit/JITThunks.cpp	(working copy)
>@@ -30,8 +30,9 @@
> 
> #include "Executable.h"
> #include "JIT.h"
>-#include "VM.h"
> #include "JSCInlines.h"
>+#include "LLIntData.h"
>+#include "VM.h"
> 
> namespace JSC {
> 
>Index: Source/JavaScriptCore/jit/JITThunks.h
>===================================================================
>--- Source/JavaScriptCore/jit/JITThunks.h	(revision 204735)
>+++ Source/JavaScriptCore/jit/JITThunks.h	(working copy)
>@@ -30,7 +30,6 @@
> 
> #include "CallData.h"
> #include "Intrinsic.h"
>-#include "LowLevelInterpreter.h"
> #include "MacroAssemblerCodeRef.h"
> #include "ThunkGenerator.h"
> #include "Weak.h"
>Index: Source/JavaScriptCore/llint/LLIntData.cpp
>===================================================================
>--- Source/JavaScriptCore/llint/LLIntData.cpp	(revision 204735)
>+++ Source/JavaScriptCore/llint/LLIntData.cpp	(working copy)
>@@ -211,7 +211,7 @@ void Data::performAssertions(VM& vm)
>     STATIC_ASSERT(GetPutInfo::initializationShift == 10);
>     STATIC_ASSERT(GetPutInfo::initializationBits == 0xffc00);
> 
>-    STATIC_ASSERT(MarkedBlock::blockMask == ~static_cast<decltype(MarkedBlock::blockMask)>(0x3fff));
>+    STATIC_ASSERT(MarkedBlock::blockSize == 64 * 1024);
> 
>     ASSERT(bitwise_cast<uintptr_t>(ShadowChicken::Packet::tailMarker()) == static_cast<uintptr_t>(0x7a11));
> 
>Index: Source/JavaScriptCore/llint/LLIntExceptions.cpp
>===================================================================
>--- Source/JavaScriptCore/llint/LLIntExceptions.cpp	(revision 204735)
>+++ Source/JavaScriptCore/llint/LLIntExceptions.cpp	(working copy)
>@@ -29,6 +29,7 @@
> #include "CodeBlock.h"
> #include "Instruction.h"
> #include "LLIntCommon.h"
>+#include "LLIntData.h"
> #include "LowLevelInterpreter.h"
> #include "JSCInlines.h"
> 
>Index: Source/JavaScriptCore/llint/LLIntThunks.cpp
>===================================================================
>--- Source/JavaScriptCore/llint/LLIntThunks.cpp	(revision 204735)
>+++ Source/JavaScriptCore/llint/LLIntThunks.cpp	(working copy)
>@@ -33,6 +33,7 @@
> #include "JSInterfaceJIT.h"
> #include "JSObject.h"
> #include "LLIntCLoop.h"
>+#include "LLIntData.h"
> #include "LinkBuffer.h"
> #include "LowLevelInterpreter.h"
> #include "ProtoCallFrame.h"
>Index: Source/JavaScriptCore/llint/LLIntThunks.h
>===================================================================
>--- Source/JavaScriptCore/llint/LLIntThunks.h	(revision 204735)
>+++ Source/JavaScriptCore/llint/LLIntThunks.h	(working copy)
>@@ -32,6 +32,7 @@ namespace JSC {
> 
> class VM;
> struct ProtoCallFrame;
>+typedef int64_t EncodedJSValue;
> 
> extern "C" {
>     EncodedJSValue vmEntryToJavaScript(void*, VM*, ProtoCallFrame*);
>Index: Source/JavaScriptCore/llint/LowLevelInterpreter32_64.asm
>===================================================================
>--- Source/JavaScriptCore/llint/LowLevelInterpreter32_64.asm	(revision 204735)
>+++ Source/JavaScriptCore/llint/LowLevelInterpreter32_64.asm	(working copy)
>@@ -702,31 +702,6 @@ _llint_op_get_scope:
>     dispatch(2)
> 
> 
>-_llint_op_create_this:
>-    traceExecution()
>-    loadi 8[PC], t0
>-    loadp PayloadOffset[cfr, t0, 8], t0
>-    bbneq JSCell::m_type[t0], JSFunctionType, .opCreateThisSlow
>-    loadp JSFunction::m_rareData[t0], t5
>-    btpz t5, .opCreateThisSlow
>-    loadp FunctionRareData::m_objectAllocationProfile + ObjectAllocationProfile::m_allocator[t5], t1
>-    loadp FunctionRareData::m_objectAllocationProfile + ObjectAllocationProfile::m_structure[t5], t2
>-    btpz t1, .opCreateThisSlow
>-    loadpFromInstruction(4, t5)
>-    bpeq t5, 1, .hasSeenMultipleCallee
>-    bpneq t5, t0, .opCreateThisSlow
>-.hasSeenMultipleCallee:
>-    allocateJSObject(t1, t2, t0, t3, .opCreateThisSlow)
>-    loadi 4[PC], t1
>-    storei CellTag, TagOffset[cfr, t1, 8]
>-    storei t0, PayloadOffset[cfr, t1, 8]
>-    dispatch(5)
>-
>-.opCreateThisSlow:
>-    callOpcodeSlowPath(_slow_path_create_this)
>-    dispatch(5)
>-
>-
> _llint_op_to_this:
>     traceExecution()
>     loadi 4[PC], t0
>@@ -742,22 +717,6 @@ _llint_op_to_this:
>     dispatch(4)
> 
> 
>-_llint_op_new_object:
>-    traceExecution()
>-    loadpFromInstruction(3, t0)
>-    loadp ObjectAllocationProfile::m_allocator[t0], t1
>-    loadp ObjectAllocationProfile::m_structure[t0], t2
>-    allocateJSObject(t1, t2, t0, t3, .opNewObjectSlow)
>-    loadi 4[PC], t1
>-    storei CellTag, TagOffset[cfr, t1, 8]
>-    storei t0, PayloadOffset[cfr, t1, 8]
>-    dispatch(4)
>-
>-.opNewObjectSlow:
>-    callOpcodeSlowPath(_llint_slow_path_new_object)
>-    dispatch(4)
>-
>-
> _llint_op_check_tdz:
>     traceExecution()
>     loadisFromInstruction(1, t0)
>Index: Source/JavaScriptCore/llint/LowLevelInterpreter64.asm
>===================================================================
>--- Source/JavaScriptCore/llint/LowLevelInterpreter64.asm	(revision 204735)
>+++ Source/JavaScriptCore/llint/LowLevelInterpreter64.asm	(working copy)
>@@ -607,30 +607,6 @@ _llint_op_get_scope:
>     dispatch(2)
> 
> 
>-_llint_op_create_this:
>-    traceExecution()
>-    loadisFromInstruction(2, t0)
>-    loadp [cfr, t0, 8], t0
>-    bbneq JSCell::m_type[t0], JSFunctionType, .opCreateThisSlow
>-    loadp JSFunction::m_rareData[t0], t3
>-    btpz t3, .opCreateThisSlow
>-    loadp FunctionRareData::m_objectAllocationProfile + ObjectAllocationProfile::m_allocator[t3], t1
>-    loadp FunctionRareData::m_objectAllocationProfile + ObjectAllocationProfile::m_structure[t3], t2
>-    btpz t1, .opCreateThisSlow
>-    loadpFromInstruction(4, t3)
>-    bpeq t3, 1, .hasSeenMultipleCallee
>-    bpneq t3, t0, .opCreateThisSlow
>-.hasSeenMultipleCallee:
>-    allocateJSObject(t1, t2, t0, t3, .opCreateThisSlow)
>-    loadisFromInstruction(1, t1)
>-    storeq t0, [cfr, t1, 8]
>-    dispatch(5)
>-
>-.opCreateThisSlow:
>-    callOpcodeSlowPath(_slow_path_create_this)
>-    dispatch(5)
>-
>-
> _llint_op_to_this:
>     traceExecution()
>     loadisFromInstruction(1, t0)
>@@ -647,21 +623,6 @@ _llint_op_to_this:
>     dispatch(4)
> 
> 
>-_llint_op_new_object:
>-    traceExecution()
>-    loadpFromInstruction(3, t0)
>-    loadp ObjectAllocationProfile::m_allocator[t0], t1
>-    loadp ObjectAllocationProfile::m_structure[t0], t2
>-    allocateJSObject(t1, t2, t0, t3, .opNewObjectSlow)
>-    loadisFromInstruction(1, t1)
>-    storeq t0, [cfr, t1, 8]
>-    dispatch(4)
>-
>-.opNewObjectSlow:
>-    callOpcodeSlowPath(_llint_slow_path_new_object)
>-    dispatch(4)
>-
>-
> _llint_op_check_tdz:
>     traceExecution()
>     loadisFromInstruction(1, t0)
>Index: Source/JavaScriptCore/llint/LowLevelInterpreter.asm
>===================================================================
>--- Source/JavaScriptCore/llint/LowLevelInterpreter.asm	(revision 204735)
>+++ Source/JavaScriptCore/llint/LowLevelInterpreter.asm	(working copy)
>@@ -402,7 +402,7 @@ const InitializationModeMask = 0xffc00
> const InitializationModeShift = 10
> const NotInitialization = 2
> 
>-const MarkedBlockSize = 16 * 1024
>+const MarkedBlockSize = 64 * 1024
> const MarkedBlockMask = ~(MarkedBlockSize - 1)
> 
> # Allocation constants
>@@ -1068,24 +1068,6 @@ macro functionInitialization(profileArgS
> .argumentProfileDone:
> end
> 
>-macro allocateJSObject(allocator, structure, result, scratch1, slowCase)
>-    const offsetOfFirstFreeCell = 
>-        MarkedAllocator::m_freeList + 
>-        MarkedBlock::FreeList::head
>-
>-    # Get the object from the free list.   
>-    loadp offsetOfFirstFreeCell[allocator], result
>-    btpz result, slowCase
>-    
>-    # Remove the object from the free list.
>-    loadp [result], scratch1
>-    storep scratch1, offsetOfFirstFreeCell[allocator]
>-
>-    # Initialize the object.
>-    storep 0, JSObject::m_butterfly[result]
>-    storeStructureWithTypeInfo(result, structure, scratch1)
>-end
>-
> macro doReturn()
>     restoreCalleeSavesUsedByLLInt()
>     restoreCallerPCAndCFR()
>@@ -1307,6 +1289,18 @@ _llint_op_create_cloned_arguments:
>     dispatch(2)
> 
> 
>+_llint_op_create_this:
>+    traceExecution()
>+    callOpcodeSlowPath(_slow_path_create_this)
>+    dispatch(5)
>+
>+
>+_llint_op_new_object:
>+    traceExecution()
>+    callOpcodeSlowPath(_llint_slow_path_new_object)
>+    dispatch(4)
>+
>+
> _llint_op_new_func:
>     traceExecution()
>     callOpcodeSlowPath(_llint_slow_path_new_func)
>Index: Source/JavaScriptCore/parser/ModuleAnalyzer.cpp
>===================================================================
>--- Source/JavaScriptCore/parser/ModuleAnalyzer.cpp	(revision 204735)
>+++ Source/JavaScriptCore/parser/ModuleAnalyzer.cpp	(working copy)
>@@ -1,5 +1,5 @@
> /*
>- * Copyright (C) 2015 Apple Inc. All rights reserved.
>+ * Copyright (C) 2015-2016 Apple Inc. All rights reserved.
>  *
>  * Redistribution and use in source and binary forms, with or without
>  * modification, are permitted provided that the following conditions
>@@ -26,9 +26,7 @@
> #include "config.h"
> #include "ModuleAnalyzer.h"
> 
>-#include "IdentifierInlines.h"
>-#include "JSCJSValueInlines.h"
>-#include "JSCellInlines.h"
>+#include "JSCInlines.h"
> #include "JSGlobalObject.h"
> #include "JSModuleRecord.h"
> #include "ModuleScopeData.h"
>Index: Source/JavaScriptCore/parser/NodeConstructors.h
>===================================================================
>--- Source/JavaScriptCore/parser/NodeConstructors.h	(revision 204735)
>+++ Source/JavaScriptCore/parser/NodeConstructors.h	(working copy)
>@@ -23,6 +23,7 @@
> 
> #include "Nodes.h"
> #include "Lexer.h"
>+#include "Opcode.h"
> #include "Parser.h"
> 
> namespace JSC {
>Index: Source/JavaScriptCore/parser/Nodes.h
>===================================================================
>--- Source/JavaScriptCore/parser/Nodes.h	(revision 204735)
>+++ Source/JavaScriptCore/parser/Nodes.h	(working copy)
>@@ -29,7 +29,6 @@
> #include "BuiltinNames.h"
> #include "Error.h"
> #include "JITCode.h"
>-#include "Opcode.h"
> #include "ParserArena.h"
> #include "ParserTokens.h"
> #include "ResultType.h"
>@@ -41,6 +40,8 @@
> 
> namespace JSC {
> 
>+    enum OpcodeID : unsigned;
>+
>     class ArgumentListNode;
>     class BytecodeGenerator;
>     class FunctionMetadataNode;
>Index: Source/JavaScriptCore/profiler/ProfilerBytecode.cpp
>===================================================================
>--- Source/JavaScriptCore/profiler/ProfilerBytecode.cpp	(revision 204735)
>+++ Source/JavaScriptCore/profiler/ProfilerBytecode.cpp	(working copy)
>@@ -28,6 +28,7 @@
> 
> #include "JSGlobalObject.h"
> #include "ObjectConstructor.h"
>+#include "Opcode.h"
> #include "JSCInlines.h"
> 
> namespace JSC { namespace Profiler {
>Index: Source/JavaScriptCore/profiler/ProfilerBytecode.h
>===================================================================
>--- Source/JavaScriptCore/profiler/ProfilerBytecode.h	(revision 204735)
>+++ Source/JavaScriptCore/profiler/ProfilerBytecode.h	(working copy)
>@@ -27,10 +27,13 @@
> #define ProfilerBytecode_h
> 
> #include "JSCJSValue.h"
>-#include "Opcode.h"
> #include <wtf/text/CString.h>
> 
>-namespace JSC { namespace Profiler {
>+namespace JSC {
>+
>+enum OpcodeID : unsigned;
>+
>+namespace Profiler {
> 
> class Bytecode {
> public:
>Index: Source/JavaScriptCore/profiler/ProfilerBytecodeSequence.cpp
>===================================================================
>--- Source/JavaScriptCore/profiler/ProfilerBytecodeSequence.cpp	(revision 204735)
>+++ Source/JavaScriptCore/profiler/ProfilerBytecodeSequence.cpp	(working copy)
>@@ -27,9 +27,10 @@
> #include "ProfilerBytecodeSequence.h"
> 
> #include "CodeBlock.h"
>+#include "Interpreter.h"
>+#include "JSCInlines.h"
> #include "JSGlobalObject.h"
> #include "Operands.h"
>-#include "JSCInlines.h"
> #include <wtf/StringPrintStream.h>
> 
> namespace JSC { namespace Profiler {
>Index: Source/JavaScriptCore/runtime/ArrayConventions.h
>===================================================================
>--- Source/JavaScriptCore/runtime/ArrayConventions.h	(revision 204735)
>+++ Source/JavaScriptCore/runtime/ArrayConventions.h	(working copy)
>@@ -1,6 +1,6 @@
> /*
>  *  Copyright (C) 1999-2000 Harri Porten (porten@kde.org)
>- *  Copyright (C) 2003, 2007, 2008, 2009, 2012 Apple Inc. All rights reserved.
>+ *  Copyright (C) 2003, 2007, 2008, 2009, 2012, 2016 Apple Inc. All rights reserved.
>  *
>  *  This library is free software; you can redistribute it and/or
>  *  modify it under the terms of the GNU Lesser General Public
>@@ -70,13 +70,15 @@ namespace JSC {
> // 0xFFFFFFFF is a bit weird -- is not an array index even though it's an integer.
> #define MAX_ARRAY_INDEX 0xFFFFFFFEU
> 
>-// The value BASE_VECTOR_LEN is the maximum number of vector elements we'll allocate
>+// The value BASE_XXX_VECTOR_LEN is the maximum number of vector elements we'll allocate
> // for an array that was created with a sepcified length (e.g. a = new Array(123))
>-#define BASE_VECTOR_LEN 4U
>-    
>+#define BASE_CONTIGUOUS_VECTOR_LEN 3U
>+#define BASE_CONTIGUOUS_VECTOR_LEN_EMPTY 5U
>+#define BASE_ARRAY_STORAGE_VECTOR_LEN 4U
>+
> // The upper bound to the size we'll grow a zero length array when the first element
> // is added.
>-#define FIRST_VECTOR_GROW 4U
>+#define FIRST_ARRAY_STORAGE_VECTOR_GROW 4U
> 
> #define MIN_BEYOND_LENGTH_SPARSE_INDEX 1000
> 
>@@ -96,7 +98,7 @@ inline bool indexIsSufficientlyBeyondLen
>     return i >= MIN_BEYOND_LENGTH_SPARSE_INDEX && i > length;
> }
> 
>-inline IndexingHeader indexingHeaderForArray(unsigned length, unsigned vectorLength)
>+inline IndexingHeader indexingHeaderForArrayStorage(unsigned length, unsigned vectorLength)
> {
>     IndexingHeader result;
>     result.setPublicLength(length);
>@@ -104,9 +106,9 @@ inline IndexingHeader indexingHeaderForA
>     return result;
> }
> 
>-inline IndexingHeader baseIndexingHeaderForArray(unsigned length)
>+inline IndexingHeader baseIndexingHeaderForArrayStorage(unsigned length)
> {
>-    return indexingHeaderForArray(length, BASE_VECTOR_LEN);
>+    return indexingHeaderForArrayStorage(length, BASE_ARRAY_STORAGE_VECTOR_LEN);
> }
> 
> } // namespace JSC
>Index: Source/JavaScriptCore/runtime/ArrayPrototype.cpp
>===================================================================
>--- Source/JavaScriptCore/runtime/ArrayPrototype.cpp	(revision 204735)
>+++ Source/JavaScriptCore/runtime/ArrayPrototype.cpp	(working copy)
>@@ -994,7 +994,7 @@ EncodedJSValue JSC_HOST_CALL arrayProtoF
>         if (UNLIKELY(vm.exception()))
>             return JSValue::encode(jsUndefined());
>     }
>-
>+    
>     setLength(exec, thisObj, length - deleteCount + additionalArgs);
>     return JSValue::encode(result);
> }
>@@ -1129,6 +1129,7 @@ static EncodedJSValue concatAppendOne(Ex
>     unsigned firstArraySize = firstButterfly->publicLength();
> 
>     IndexingType type = first->mergeIndexingTypeForCopying(indexingTypeForValue(second) | IsArray);
>+    
>     if (type == NonArray)
>         type = first->indexingType();
> 
>@@ -1156,7 +1157,7 @@ EncodedJSValue JSC_HOST_CALL arrayProtoP
>     VM& vm = exec->vm();
> 
>     JSArray* firstArray = jsCast<JSArray*>(exec->uncheckedArgument(0));
>-
>+    
>     // This code assumes that neither array has set Symbol.isConcatSpreadable. If the first array
>     // has indexed accessors then one of those accessors might change the value of Symbol.isConcatSpreadable
>     // on the second argument.
>@@ -1172,14 +1173,15 @@ EncodedJSValue JSC_HOST_CALL arrayProtoP
>         return concatAppendOne(exec, vm, firstArray, second);
> 
>     JSArray* secondArray = jsCast<JSArray*>(second);
>-
>+    
>     Butterfly* firstButterfly = firstArray->butterfly();
>     Butterfly* secondButterfly = secondArray->butterfly();
> 
>     unsigned firstArraySize = firstButterfly->publicLength();
>     unsigned secondArraySize = secondButterfly->publicLength();
> 
>-    IndexingType type = firstArray->mergeIndexingTypeForCopying(secondArray->indexingType());
>+    IndexingType secondType = secondArray->indexingType();
>+    IndexingType type = firstArray->mergeIndexingTypeForCopying(secondType);
>     if (type == NonArray || !firstArray->canFastCopy(vm, secondArray) || firstArraySize + secondArraySize >= MIN_SPARSE_ARRAY_INDEX) {
>         JSArray* result = constructEmptyArray(exec, nullptr, firstArraySize + secondArraySize);
>         if (vm.exception())
>@@ -1198,7 +1200,7 @@ EncodedJSValue JSC_HOST_CALL arrayProtoP
>     JSArray* result = JSArray::tryCreateUninitialized(vm, resultStructure, firstArraySize + secondArraySize);
>     if (!result)
>         return JSValue::encode(throwOutOfMemoryError(exec));
>-
>+    
>     if (type == ArrayWithDouble) {
>         double* buffer = result->butterfly()->contiguousDouble().data();
>         memcpy(buffer, firstButterfly->contiguousDouble().data(), sizeof(JSValue) * firstArraySize);
>@@ -1206,7 +1208,12 @@ EncodedJSValue JSC_HOST_CALL arrayProtoP
>     } else if (type != ArrayWithUndecided) {
>         WriteBarrier<Unknown>* buffer = result->butterfly()->contiguous().data();
>         memcpy(buffer, firstButterfly->contiguous().data(), sizeof(JSValue) * firstArraySize);
>-        memcpy(buffer + firstArraySize, secondButterfly->contiguous().data(), sizeof(JSValue) * secondArraySize);
>+        if (secondType != ArrayWithUndecided)
>+            memcpy(buffer + firstArraySize, secondButterfly->contiguous().data(), sizeof(JSValue) * secondArraySize);
>+        else {
>+            for (unsigned i = secondArraySize; i--;)
>+                buffer[i + firstArraySize].clear();
>+        }
>     }
> 
>     result->butterfly()->setPublicLength(firstArraySize + secondArraySize);
>Index: Source/JavaScriptCore/runtime/ArrayStorage.h
>===================================================================
>--- Source/JavaScriptCore/runtime/ArrayStorage.h	(revision 204735)
>+++ Source/JavaScriptCore/runtime/ArrayStorage.h	(working copy)
>@@ -1,5 +1,5 @@
> /*
>- * Copyright (C) 2012 Apple Inc. All rights reserved.
>+ * Copyright (C) 2012, 2016 Apple Inc. All rights reserved.
>  *
>  * Redistribution and use in source and binary forms, with or without
>  * modification, are permitted provided that the following conditions
>@@ -29,7 +29,9 @@
> #include "ArrayConventions.h"
> #include "Butterfly.h"
> #include "IndexingHeader.h"
>+#include "MarkedSpace.h"
> #include "SparseArrayValueMap.h"
>+#include "Structure.h"
> #include "WriteBarrier.h"
> #include <wtf/Noncopyable.h>
> 
>@@ -58,7 +60,7 @@ public:
>     // We steal two fields from the indexing header: vectorLength and length.
>     unsigned length() const { return indexingHeader()->publicLength(); }
>     void setLength(unsigned length) { indexingHeader()->setPublicLength(length); }
>-    unsigned vectorLength() { return indexingHeader()->vectorLength(); }
>+    unsigned vectorLength() const { return indexingHeader()->vectorLength(); }
>     void setVectorLength(unsigned length) { indexingHeader()->setVectorLength(length); }
>     
>     ALWAYS_INLINE void copyHeaderFromDuringGC(const ArrayStorage& other)
>@@ -99,6 +101,66 @@ public:
>     {
>         return ArrayStorage::vectorOffset() + vectorLength * sizeof(WriteBarrier<Unknown>);
>     }
>+    
>+    static size_t totalSizeFor(unsigned indexBias, size_t propertyCapacity, unsigned vectorLength)
>+    {
>+        return Butterfly::totalSize(indexBias, propertyCapacity, true, sizeFor(vectorLength));
>+    }
>+    
>+    size_t totalSize(size_t propertyCapacity) const
>+    {
>+        return totalSizeFor(m_indexBias, propertyCapacity, vectorLength());
>+    }
>+    
>+    size_t totalSize(Structure* structure) const
>+    {
>+        return totalSize(structure->outOfLineCapacity());
>+    }
>+    
>+    static unsigned availableVectorLength(unsigned indexBias, size_t propertyCapacity, unsigned vectorLength)
>+    {
>+        size_t cellSize = MarkedSpace::optimalSizeFor(totalSizeFor(indexBias, propertyCapacity, vectorLength));
>+        
>+        vectorLength = (cellSize - totalSizeFor(indexBias, propertyCapacity, 0)) / sizeof(WriteBarrier<Unknown>);
>+
>+        return vectorLength;
>+    }
>+    
>+    static unsigned availableVectorLength(unsigned indexBias, Structure* structure, unsigned vectorLength)
>+    {
>+        return availableVectorLength(indexBias, structure->outOfLineCapacity(), vectorLength);
>+    }
>+    
>+    unsigned availableVectorLength(size_t propertyCapacity, unsigned vectorLength)
>+    {
>+        return availableVectorLength(m_indexBias, propertyCapacity, vectorLength);
>+    }
>+    
>+    unsigned availableVectorLength(Structure* structure, unsigned vectorLength)
>+    {
>+        return availableVectorLength(structure->outOfLineCapacity(), vectorLength);
>+    }
>+
>+    static unsigned optimalVectorLength(unsigned indexBias, size_t propertyCapacity, unsigned vectorLength)
>+    {
>+        vectorLength = std::max(BASE_ARRAY_STORAGE_VECTOR_LEN, vectorLength);
>+        return availableVectorLength(indexBias, propertyCapacity, vectorLength);
>+    }
>+    
>+    static unsigned optimalVectorLength(unsigned indexBias, Structure* structure, unsigned vectorLength)
>+    {
>+        return optimalVectorLength(indexBias, structure->outOfLineCapacity(), vectorLength);
>+    }
>+    
>+    unsigned optimalVectorLength(size_t propertyCapacity, unsigned vectorLength)
>+    {
>+        return optimalVectorLength(m_indexBias, propertyCapacity, vectorLength);
>+    }
>+    
>+    unsigned optimalVectorLength(Structure* structure, unsigned vectorLength)
>+    {
>+        return optimalVectorLength(structure->outOfLineCapacity(), vectorLength);
>+    }
> };
> 
> } // namespace JSC
>Index: Source/JavaScriptCore/runtime/AuxiliaryBarrier.h
>===================================================================
>--- Source/JavaScriptCore/runtime/AuxiliaryBarrier.h	(nonexistent)
>+++ Source/JavaScriptCore/runtime/AuxiliaryBarrier.h	(working copy)
>@@ -0,0 +1,63 @@
>+/*
>+ * Copyright (C) 2016 Apple Inc. All rights reserved.
>+ *
>+ * Redistribution and use in source and binary forms, with or without
>+ * modification, are permitted provided that the following conditions
>+ * are met:
>+ * 1. Redistributions of source code must retain the above copyright
>+ *    notice, this list of conditions and the following disclaimer.
>+ * 2. Redistributions in binary form must reproduce the above copyright
>+ *    notice, this list of conditions and the following disclaimer in the
>+ *    documentation and/or other materials provided with the distribution.
>+ *
>+ * THIS SOFTWARE IS PROVIDED BY APPLE INC. AND ITS CONTRIBUTORS ``AS IS''
>+ * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO,
>+ * THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
>+ * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL APPLE INC. OR ITS CONTRIBUTORS
>+ * BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
>+ * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
>+ * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
>+ * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
>+ * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
>+ * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF
>+ * THE POSSIBILITY OF SUCH DAMAGE.
>+ */
>+
>+#pragma once
>+
>+namespace JSC {
>+
>+class JSCell;
>+class VM;
>+
>+// An Auxiliary barrier is a barrier that does not try to reason about the value being stored into
>+// it, other than interpreting a falsy value as not needing a barrier. It's OK to use this for either
>+// JSCells or any other kind of data, so long as it responds to operator!().
>+template<typename T>
>+class AuxiliaryBarrier {
>+public:
>+    AuxiliaryBarrier() { }
>+    
>+    template<typename U>
>+    AuxiliaryBarrier(VM&, JSCell*, U&&);
>+    
>+    void clear() { m_value = T(); }
>+    
>+    template<typename U>
>+    void set(VM&, JSCell*, U&&);
>+    
>+    const T& get() const { return m_value; }
>+    
>+    T* slot() { return &m_value; }
>+    
>+    explicit operator bool() const { return !!m_value; }
>+    
>+    template<typename U>
>+    void setWithoutBarrier(U&& value) { m_value = std::forward<U>(value); }
>+    
>+private:
>+    T m_value;
>+};
>+
>+} // namespace JSC
>+
>Index: Source/JavaScriptCore/runtime/AuxiliaryBarrierInlines.h
>===================================================================
>--- Source/JavaScriptCore/runtime/AuxiliaryBarrierInlines.h	(nonexistent)
>+++ Source/JavaScriptCore/runtime/AuxiliaryBarrierInlines.h	(working copy)
>@@ -0,0 +1,51 @@
>+/*
>+ * Copyright (C) 2016 Apple Inc. All rights reserved.
>+ *
>+ * Redistribution and use in source and binary forms, with or without
>+ * modification, are permitted provided that the following conditions
>+ * are met:
>+ * 1. Redistributions of source code must retain the above copyright
>+ *    notice, this list of conditions and the following disclaimer.
>+ * 2. Redistributions in binary form must reproduce the above copyright
>+ *    notice, this list of conditions and the following disclaimer in the
>+ *    documentation and/or other materials provided with the distribution.
>+ *
>+ * THIS SOFTWARE IS PROVIDED BY APPLE INC. AND ITS CONTRIBUTORS ``AS IS''
>+ * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO,
>+ * THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
>+ * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL APPLE INC. OR ITS CONTRIBUTORS
>+ * BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
>+ * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
>+ * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
>+ * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
>+ * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
>+ * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF
>+ * THE POSSIBILITY OF SUCH DAMAGE.
>+ */
>+
>+#pragma once
>+
>+#include "AuxiliaryBarrier.h"
>+#include "Heap.h"
>+#include "VM.h"
>+
>+namespace JSC {
>+
>+template<typename T>
>+template<typename U>
>+AuxiliaryBarrier<T>::AuxiliaryBarrier(VM& vm, JSCell* owner, U&& value)
>+{
>+    m_value = std::forward<U>(value);
>+    vm.heap.writeBarrier(owner);
>+}
>+
>+template<typename T>
>+template<typename U>
>+void AuxiliaryBarrier<T>::set(VM& vm, JSCell* owner, U&& value)
>+{
>+    m_value = std::forward<U>(value);
>+    vm.heap.writeBarrier(owner);
>+}
>+
>+} // namespace JSC
>+
>Index: Source/JavaScriptCore/runtime/Butterfly.h
>===================================================================
>--- Source/JavaScriptCore/runtime/Butterfly.h	(revision 204735)
>+++ Source/JavaScriptCore/runtime/Butterfly.h	(working copy)
>@@ -1,5 +1,5 @@
> /*
>- * Copyright (C) 2012 Apple Inc. All rights reserved.
>+ * Copyright (C) 2012, 2016 Apple Inc. All rights reserved.
>  *
>  * Redistribution and use in source and binary forms, with or without
>  * modification, are permitted provided that the following conditions
>@@ -90,6 +90,12 @@ public:
>         return reinterpret_cast<Butterfly*>(static_cast<EncodedJSValue*>(base) + preCapacity + propertyCapacity + 1);
>     }
>     
>+    ALWAYS_INLINE static unsigned availableContiguousVectorLength(size_t propertyCapacity, unsigned vectorLength);
>+    static unsigned availableContiguousVectorLength(Structure*, unsigned vectorLength);
>+    
>+    ALWAYS_INLINE static unsigned optimalContiguousVectorLength(size_t propertyCapacity, unsigned vectorLength);
>+    static unsigned optimalContiguousVectorLength(Structure*, unsigned vectorLength);
>+    
>     // This method is here not just because it's handy, but to remind you that
>     // the whole point of butterflies is to do evil pointer arithmetic.
>     static Butterfly* fromPointer(char* ptr)
>Index: Source/JavaScriptCore/runtime/ButterflyInlines.h
>===================================================================
>--- Source/JavaScriptCore/runtime/ButterflyInlines.h	(revision 204735)
>+++ Source/JavaScriptCore/runtime/ButterflyInlines.h	(working copy)
>@@ -1,5 +1,5 @@
> /*
>- * Copyright (C) 2012 Apple Inc. All rights reserved.
>+ * Copyright (C) 2012, 2016 Apple Inc. All rights reserved.
>  *
>  * Redistribution and use in source and binary forms, with or without
>  * modification, are permitted provided that the following conditions
>@@ -35,12 +35,38 @@
> 
> namespace JSC {
> 
>+ALWAYS_INLINE unsigned Butterfly::availableContiguousVectorLength(size_t propertyCapacity, unsigned vectorLength)
>+{
>+    size_t cellSize = totalSize(0, propertyCapacity, true, sizeof(EncodedJSValue) * vectorLength);
>+    cellSize = MarkedSpace::optimalSizeFor(cellSize);
>+    vectorLength = (cellSize - totalSize(0, propertyCapacity, true, 0)) / sizeof(EncodedJSValue);
>+    return vectorLength;
>+}
>+
>+ALWAYS_INLINE unsigned Butterfly::availableContiguousVectorLength(Structure* structure, unsigned vectorLength)
>+{
>+    return availableContiguousVectorLength(structure ? structure->outOfLineCapacity() : 0, vectorLength);
>+}
>+
>+ALWAYS_INLINE unsigned Butterfly::optimalContiguousVectorLength(size_t propertyCapacity, unsigned vectorLength)
>+{
>+    if (!vectorLength)
>+        vectorLength = BASE_CONTIGUOUS_VECTOR_LEN_EMPTY;
>+    else
>+        vectorLength = std::max(BASE_CONTIGUOUS_VECTOR_LEN, vectorLength);
>+    return availableContiguousVectorLength(propertyCapacity, vectorLength);
>+}
>+
>+ALWAYS_INLINE unsigned Butterfly::optimalContiguousVectorLength(Structure* structure, unsigned vectorLength)
>+{
>+    return optimalContiguousVectorLength(structure ? structure->outOfLineCapacity() : 0, vectorLength);
>+}
>+
> inline Butterfly* Butterfly::createUninitialized(VM& vm, JSCell* intendedOwner, size_t preCapacity, size_t propertyCapacity, bool hasIndexingHeader, size_t indexingPayloadSizeInBytes)
> {
>-    void* temp;
>     size_t size = totalSize(preCapacity, propertyCapacity, hasIndexingHeader, indexingPayloadSizeInBytes);
>-    RELEASE_ASSERT(vm.heap.tryAllocateStorage(intendedOwner, size, &temp));
>-    Butterfly* result = fromBase(temp, preCapacity, propertyCapacity);
>+    void* base = vm.heap.allocateAuxiliary(intendedOwner, size);
>+    Butterfly* result = fromBase(base, preCapacity, propertyCapacity);
>     return result;
> }
> 
>@@ -119,7 +145,8 @@ inline Butterfly* Butterfly::growArrayRi
>     void* theBase = base(0, propertyCapacity);
>     size_t oldSize = totalSize(0, propertyCapacity, hadIndexingHeader, oldIndexingPayloadSizeInBytes);
>     size_t newSize = totalSize(0, propertyCapacity, true, newIndexingPayloadSizeInBytes);
>-    if (!vm.heap.tryReallocateStorage(intendedOwner, &theBase, oldSize, newSize))
>+    theBase = vm.heap.tryReallocateAuxiliary(intendedOwner, theBase, oldSize, newSize);
>+    if (!theBase)
>         return 0;
>     return fromBase(theBase, 0, propertyCapacity);
> }
>Index: Source/JavaScriptCore/runtime/ClonedArguments.cpp
>===================================================================
>--- Source/JavaScriptCore/runtime/ClonedArguments.cpp	(revision 204735)
>+++ Source/JavaScriptCore/runtime/ClonedArguments.cpp	(working copy)
>@@ -44,16 +44,19 @@ ClonedArguments::ClonedArguments(VM& vm,
> ClonedArguments* ClonedArguments::createEmpty(
>     VM& vm, Structure* structure, JSFunction* callee, unsigned length)
> {
>-    unsigned vectorLength = std::max(BASE_VECTOR_LEN, length);
>+    unsigned vectorLength = length;
>     if (vectorLength > MAX_STORAGE_VECTOR_LENGTH)
>         return 0;
> 
>-    void* temp;
>-    if (!vm.heap.tryAllocateStorage(0, Butterfly::totalSize(0, structure->outOfLineCapacity(), true, vectorLength * sizeof(EncodedJSValue)), &temp))
>+    void* temp = vm.heap.tryAllocateAuxiliary(nullptr, Butterfly::totalSize(0, structure->outOfLineCapacity(), true, vectorLength * sizeof(EncodedJSValue)));
>+    if (!temp)
>         return 0;
>     Butterfly* butterfly = Butterfly::fromBase(temp, 0, structure->outOfLineCapacity());
>     butterfly->setVectorLength(vectorLength);
>     butterfly->setPublicLength(length);
>+    
>+    for (unsigned i = length; i < vectorLength; ++i)
>+        butterfly->contiguous()[i].clear();
> 
>     ClonedArguments* result =
>         new (NotNull, allocateCell<ClonedArguments>(vm.heap))
>Index: Source/JavaScriptCore/runtime/CommonSlowPathsExceptions.cpp
>===================================================================
>--- Source/JavaScriptCore/runtime/CommonSlowPathsExceptions.cpp	(revision 204735)
>+++ Source/JavaScriptCore/runtime/CommonSlowPathsExceptions.cpp	(working copy)
>@@ -28,6 +28,7 @@
> 
> #include "CallFrame.h"
> #include "CodeBlock.h"
>+#include "Interpreter.h"
> #include "JITExceptions.h"
> #include "LLIntCommon.h"
> #include "JSCInlines.h"
>Index: Source/JavaScriptCore/runtime/CommonSlowPathsExceptions.h
>===================================================================
>--- Source/JavaScriptCore/runtime/CommonSlowPathsExceptions.h	(revision 204735)
>+++ Source/JavaScriptCore/runtime/CommonSlowPathsExceptions.h	(working copy)
>@@ -26,11 +26,10 @@
> #ifndef CommonSlowPathExceptions_h
> #define CommonSlowPathExceptions_h
> 
>-#include "MacroAssemblerCodeRef.h"
>-
> namespace JSC {
> 
> class ExecState;
>+class JSObject;
> 
> namespace CommonSlowPaths {
> 
>Index: Source/JavaScriptCore/runtime/DataView.cpp
>===================================================================
>--- Source/JavaScriptCore/runtime/DataView.cpp	(revision 204735)
>+++ Source/JavaScriptCore/runtime/DataView.cpp	(working copy)
>@@ -26,6 +26,7 @@
> #include "config.h"
> #include "DataView.h"
> 
>+#include "JSCInlines.h"
> #include "JSDataView.h"
> #include "JSGlobalObject.h"
> 
>Index: Source/JavaScriptCore/runtime/DirectArguments.h
>===================================================================
>--- Source/JavaScriptCore/runtime/DirectArguments.h	(revision 204735)
>+++ Source/JavaScriptCore/runtime/DirectArguments.h	(working copy)
>@@ -26,6 +26,7 @@
> #ifndef DirectArguments_h
> #define DirectArguments_h
> 
>+#include "CopyBarrier.h"
> #include "DirectArgumentsOffset.h"
> #include "GenericArguments.h"
> 
>Index: Source/JavaScriptCore/runtime/ECMAScriptSpecInternalFunctions.cpp
>===================================================================
>--- Source/JavaScriptCore/runtime/ECMAScriptSpecInternalFunctions.cpp	(revision 204735)
>+++ Source/JavaScriptCore/runtime/ECMAScriptSpecInternalFunctions.cpp	(working copy)
>@@ -28,7 +28,7 @@
> 
> #include "CallFrame.h"
> #include "ConstructData.h"
>-#include "JSCJSValueInlines.h"
>+#include "JSCInlines.h"
> #include "RegExpObject.h"
> 
> namespace JSC {
>Index: Source/JavaScriptCore/runtime/Error.cpp
>===================================================================
>--- Source/JavaScriptCore/runtime/Error.cpp	(revision 204735)
>+++ Source/JavaScriptCore/runtime/Error.cpp	(working copy)
>@@ -28,14 +28,16 @@
> #include "ErrorConstructor.h"
> #include "ExceptionHelpers.h"
> #include "FunctionPrototype.h"
>+#include "Interpreter.h"
> #include "JSArray.h"
> #include "JSFunction.h"
> #include "JSGlobalObject.h"
> #include "JSObject.h"
> #include "JSString.h"
>-#include "NativeErrorConstructor.h"
> #include "JSCInlines.h"
>+#include "NativeErrorConstructor.h"
> #include "SourceCode.h"
>+#include "StackFrame.h"
> 
> namespace JSC {
> 
>Index: Source/JavaScriptCore/runtime/Error.h
>===================================================================
>--- Source/JavaScriptCore/runtime/Error.h	(revision 204735)
>+++ Source/JavaScriptCore/runtime/Error.h	(working copy)
>@@ -25,7 +25,6 @@
> 
> #include "ErrorInstance.h"
> #include "InternalFunction.h"
>-#include "Interpreter.h"
> #include "JSObject.h"
> #include <stdint.h>
> 
>Index: Source/JavaScriptCore/runtime/ErrorInstance.cpp
>===================================================================
>--- Source/JavaScriptCore/runtime/ErrorInstance.cpp	(revision 204735)
>+++ Source/JavaScriptCore/runtime/ErrorInstance.cpp	(working copy)
>@@ -26,6 +26,7 @@
> #include "JSScope.h"
> #include "JSCInlines.h"
> #include "JSGlobalObjectFunctions.h"
>+#include <wtf/text/StringBuilder.h>
> 
> namespace JSC {
> 
>Index: Source/JavaScriptCore/runtime/ErrorInstance.h
>===================================================================
>--- Source/JavaScriptCore/runtime/ErrorInstance.h	(revision 204735)
>+++ Source/JavaScriptCore/runtime/ErrorInstance.h	(working copy)
>@@ -21,7 +21,7 @@
> #ifndef ErrorInstance_h
> #define ErrorInstance_h
> 
>-#include "Interpreter.h"
>+#include "JSObject.h"
> #include "RuntimeType.h"
> #include "SourceProvider.h"
> 
>Index: Source/JavaScriptCore/runtime/Exception.cpp
>===================================================================
>--- Source/JavaScriptCore/runtime/Exception.cpp	(revision 204735)
>+++ Source/JavaScriptCore/runtime/Exception.cpp	(working copy)
>@@ -26,6 +26,7 @@
> #include "config.h"
> #include "Exception.h"
> 
>+#include "Interpreter.h"
> #include "JSCInlines.h"
> 
> namespace JSC {
>Index: Source/JavaScriptCore/runtime/Exception.h
>===================================================================
>--- Source/JavaScriptCore/runtime/Exception.h	(revision 204735)
>+++ Source/JavaScriptCore/runtime/Exception.h	(working copy)
>@@ -26,7 +26,8 @@
> #ifndef Exception_h
> #define Exception_h
> 
>-#include "Interpreter.h"
>+#include "JSObject.h"
>+#include "StackFrame.h"
> #include <wtf/Vector.h>
> 
> namespace JSC {
>Index: Source/JavaScriptCore/runtime/GeneratorFrame.cpp
>===================================================================
>--- Source/JavaScriptCore/runtime/GeneratorFrame.cpp	(revision 204735)
>+++ Source/JavaScriptCore/runtime/GeneratorFrame.cpp	(working copy)
>@@ -28,10 +28,7 @@
> 
> #include "CodeBlock.h"
> #include "HeapIterationScope.h"
>-#include "JSCJSValueInlines.h"
>-#include "JSCellInlines.h"
>-#include "SlotVisitorInlines.h"
>-#include "StructureInlines.h"
>+#include "JSCInlines.h"
> 
> namespace JSC {
> 
>Index: Source/JavaScriptCore/runtime/GeneratorPrototype.cpp
>===================================================================
>--- Source/JavaScriptCore/runtime/GeneratorPrototype.cpp	(revision 204735)
>+++ Source/JavaScriptCore/runtime/GeneratorPrototype.cpp	(working copy)
>@@ -27,10 +27,8 @@
> #include "GeneratorPrototype.h"
> 
> #include "JSCBuiltins.h"
>-#include "JSCJSValueInlines.h"
>-#include "JSCellInlines.h"
>+#include "JSCInlines.h"
> #include "JSGlobalObject.h"
>-#include "StructureInlines.h"
> 
> #include "GeneratorPrototype.lut.h"
> 
>Index: Source/JavaScriptCore/runtime/IntlCollator.cpp
>===================================================================
>--- Source/JavaScriptCore/runtime/IntlCollator.cpp	(revision 204735)
>+++ Source/JavaScriptCore/runtime/IntlCollator.cpp	(working copy)
>@@ -1,6 +1,7 @@
> /*
>  * Copyright (C) 2015 Andy VanWagoner (thetalecrafter@gmail.com)
>  * Copyright (C) 2015 Sukolsak Sakshuwong (sukolsak@gmail.com)
>+ * Copyright (C) 2016 Apple Inc. All Rights Reserved.
>  *
>  * Redistribution and use in source and binary forms, with or without
>  * modification, are permitted provided that the following conditions
>@@ -33,8 +34,7 @@
> #include "IntlCollatorConstructor.h"
> #include "IntlObject.h"
> #include "JSBoundFunction.h"
>-#include "JSCJSValueInlines.h"
>-#include "JSCellInlines.h"
>+#include "JSCInlines.h"
> #include "ObjectConstructor.h"
> #include "SlotVisitorInlines.h"
> #include "StructureInlines.h"
>Index: Source/JavaScriptCore/runtime/IntlCollatorConstructor.cpp
>===================================================================
>--- Source/JavaScriptCore/runtime/IntlCollatorConstructor.cpp	(revision 204735)
>+++ Source/JavaScriptCore/runtime/IntlCollatorConstructor.cpp	(working copy)
>@@ -33,11 +33,8 @@
> #include "IntlCollator.h"
> #include "IntlCollatorPrototype.h"
> #include "IntlObject.h"
>-#include "JSCJSValueInlines.h"
>-#include "JSCellInlines.h"
>+#include "JSCInlines.h"
> #include "Lookup.h"
>-#include "SlotVisitorInlines.h"
>-#include "StructureInlines.h"
> 
> namespace JSC {
> 
>Index: Source/JavaScriptCore/runtime/IntlCollatorPrototype.cpp
>===================================================================
>--- Source/JavaScriptCore/runtime/IntlCollatorPrototype.cpp	(revision 204735)
>+++ Source/JavaScriptCore/runtime/IntlCollatorPrototype.cpp	(working copy)
>@@ -32,10 +32,7 @@
> #include "Error.h"
> #include "IntlCollator.h"
> #include "JSBoundFunction.h"
>-#include "JSCJSValueInlines.h"
>-#include "JSCellInlines.h"
>-#include "JSObject.h"
>-#include "StructureInlines.h"
>+#include "JSCInlines.h"
> 
> namespace JSC {
> 
>Index: Source/JavaScriptCore/runtime/IntlDateTimeFormat.cpp
>===================================================================
>--- Source/JavaScriptCore/runtime/IntlDateTimeFormat.cpp	(revision 204735)
>+++ Source/JavaScriptCore/runtime/IntlDateTimeFormat.cpp	(working copy)
>@@ -34,14 +34,14 @@
> #include "IntlDateTimeFormatConstructor.h"
> #include "IntlObject.h"
> #include "JSBoundFunction.h"
>-#include "JSCJSValueInlines.h"
>-#include "JSCellInlines.h"
>+#include "JSCInlines.h"
> #include "ObjectConstructor.h"
> #include "SlotVisitorInlines.h"
> #include "StructureInlines.h"
> #include <unicode/ucal.h>
> #include <unicode/udatpg.h>
> #include <unicode/uenum.h>
>+#include <wtf/text/StringBuilder.h>
> 
> namespace JSC {
> 
>Index: Source/JavaScriptCore/runtime/IntlDateTimeFormatConstructor.cpp
>===================================================================
>--- Source/JavaScriptCore/runtime/IntlDateTimeFormatConstructor.cpp	(revision 204735)
>+++ Source/JavaScriptCore/runtime/IntlDateTimeFormatConstructor.cpp	(working copy)
>@@ -33,11 +33,8 @@
> #include "IntlDateTimeFormatPrototype.h"
> #include "IntlObject.h"
> #include "IntlObjectInlines.h"
>-#include "JSCJSValueInlines.h"
>-#include "JSCellInlines.h"
>+#include "JSCInlines.h"
> #include "Lookup.h"
>-#include "SlotVisitorInlines.h"
>-#include "StructureInlines.h"
> 
> namespace JSC {
> 
>Index: Source/JavaScriptCore/runtime/IntlDateTimeFormatPrototype.cpp
>===================================================================
>--- Source/JavaScriptCore/runtime/IntlDateTimeFormatPrototype.cpp	(revision 204735)
>+++ Source/JavaScriptCore/runtime/IntlDateTimeFormatPrototype.cpp	(working copy)
>@@ -34,10 +34,8 @@
> #include "IntlDateTimeFormat.h"
> #include "IntlObject.h"
> #include "JSBoundFunction.h"
>-#include "JSCJSValueInlines.h"
>-#include "JSCellInlines.h"
>+#include "JSCInlines.h"
> #include "JSObject.h"
>-#include "StructureInlines.h"
> 
> namespace JSC {
> 
>Index: Source/JavaScriptCore/runtime/IntlNumberFormat.cpp
>===================================================================
>--- Source/JavaScriptCore/runtime/IntlNumberFormat.cpp	(revision 204735)
>+++ Source/JavaScriptCore/runtime/IntlNumberFormat.cpp	(working copy)
>@@ -34,11 +34,8 @@
> #include "IntlNumberFormatConstructor.h"
> #include "IntlObject.h"
> #include "JSBoundFunction.h"
>-#include "JSCJSValueInlines.h"
>-#include "JSCellInlines.h"
>+#include "JSCInlines.h"
> #include "ObjectConstructor.h"
>-#include "SlotVisitorInlines.h"
>-#include "StructureInlines.h"
> 
> namespace JSC {
> 
>Index: Source/JavaScriptCore/runtime/IntlNumberFormatConstructor.cpp
>===================================================================
>--- Source/JavaScriptCore/runtime/IntlNumberFormatConstructor.cpp	(revision 204735)
>+++ Source/JavaScriptCore/runtime/IntlNumberFormatConstructor.cpp	(working copy)
>@@ -33,11 +33,8 @@
> #include "IntlNumberFormatPrototype.h"
> #include "IntlObject.h"
> #include "IntlObjectInlines.h"
>-#include "JSCJSValueInlines.h"
>-#include "JSCellInlines.h"
>+#include "JSCInlines.h"
> #include "Lookup.h"
>-#include "SlotVisitorInlines.h"
>-#include "StructureInlines.h"
> 
> namespace JSC {
> 
>Index: Source/JavaScriptCore/runtime/IntlNumberFormatPrototype.cpp
>===================================================================
>--- Source/JavaScriptCore/runtime/IntlNumberFormatPrototype.cpp	(revision 204735)
>+++ Source/JavaScriptCore/runtime/IntlNumberFormatPrototype.cpp	(working copy)
>@@ -33,10 +33,8 @@
> #include "Error.h"
> #include "IntlNumberFormat.h"
> #include "JSBoundFunction.h"
>-#include "JSCJSValueInlines.h"
>-#include "JSCellInlines.h"
>+#include "JSCInlines.h"
> #include "JSObject.h"
>-#include "StructureInlines.h"
> 
> namespace JSC {
> 
>Index: Source/JavaScriptCore/runtime/IntlObject.cpp
>===================================================================
>--- Source/JavaScriptCore/runtime/IntlObject.cpp	(revision 204735)
>+++ Source/JavaScriptCore/runtime/IntlObject.cpp	(working copy)
>@@ -50,6 +50,7 @@
> #include <wtf/Assertions.h>
> #include <wtf/NeverDestroyed.h>
> #include <wtf/PlatformUserPreferredLanguages.h>
>+#include <wtf/text/StringBuilder.h>
> 
> namespace JSC {
> 
>Index: Source/JavaScriptCore/runtime/IteratorPrototype.cpp
>===================================================================
>--- Source/JavaScriptCore/runtime/IteratorPrototype.cpp	(revision 204735)
>+++ Source/JavaScriptCore/runtime/IteratorPrototype.cpp	(working copy)
>@@ -27,11 +27,9 @@
> #include "IteratorPrototype.h"
> 
> #include "JSCBuiltins.h"
>-#include "JSCJSValueInlines.h"
>-#include "JSCellInlines.h"
>+#include "JSCInlines.h"
> #include "JSGlobalObject.h"
> #include "ObjectConstructor.h"
>-#include "StructureInlines.h"
> 
> namespace JSC {
> 
>Index: Source/JavaScriptCore/runtime/JSArray.cpp
>===================================================================
>--- Source/JavaScriptCore/runtime/JSArray.cpp	(revision 204735)
>+++ Source/JavaScriptCore/runtime/JSArray.cpp	(working copy)
>@@ -41,6 +41,12 @@ using namespace WTF;
> 
> namespace JSC {
> 
>+#if 0
>+#define LOGIT() dataLog(RawPointer(this), ": ", WTF_PRETTY_FUNCTION, "\n");
>+#else
>+#define LOGIT() do { } while(false)
>+#endif
>+
> STATIC_ASSERT_IS_TRIVIALLY_DESTRUCTIBLE(JSArray);
> 
> const ClassInfo JSArray::s_info = {"Array", &JSNonFinalObject::s_info, 0, CREATE_METHOD_TABLE(JSArray)};
>@@ -59,8 +65,58 @@ Butterfly* createArrayButterflyInDiction
>     return butterfly;
> }
> 
>+JSArray* JSArray::tryCreateUninitialized(VM& vm, Structure* structure, unsigned initialLength)
>+{
>+    if (initialLength > MAX_STORAGE_VECTOR_LENGTH)
>+        return 0;
>+
>+    unsigned outOfLineStorage = structure->outOfLineCapacity();
>+
>+    Butterfly* butterfly;
>+    IndexingType indexingType = structure->indexingType();
>+    if (LIKELY(!hasAnyArrayStorage(indexingType))) {
>+        ASSERT(
>+            hasUndecided(indexingType)
>+            || hasInt32(indexingType)
>+            || hasDouble(indexingType)
>+            || hasContiguous(indexingType));
>+
>+        unsigned vectorLength = Butterfly::optimalContiguousVectorLength(structure, initialLength);
>+        void* temp = vm.heap.tryAllocateAuxiliary(nullptr, Butterfly::totalSize(0, outOfLineStorage, true, vectorLength * sizeof(EncodedJSValue)));
>+        if (!temp)
>+            return nullptr;
>+        butterfly = Butterfly::fromBase(temp, 0, outOfLineStorage);
>+        butterfly->setVectorLength(vectorLength);
>+        butterfly->setPublicLength(initialLength);
>+        if (hasDouble(indexingType)) {
>+            for (unsigned i = initialLength; i < vectorLength; ++i)
>+                butterfly->contiguousDouble()[i] = PNaN;
>+        } else {
>+            for (unsigned i = initialLength; i < vectorLength; ++i)
>+                butterfly->contiguous()[i].clear();
>+        }
>+    } else {
>+        unsigned vectorLength = ArrayStorage::optimalVectorLength(0, structure, initialLength);
>+        void* temp = vm.heap.tryAllocateAuxiliary(nullptr, Butterfly::totalSize(0, outOfLineStorage, true, ArrayStorage::sizeFor(vectorLength)));
>+        if (!temp)
>+            return nullptr;
>+        butterfly = Butterfly::fromBase(temp, 0, outOfLineStorage);
>+        *butterfly->indexingHeader() = indexingHeaderForArrayStorage(initialLength, vectorLength);
>+        ArrayStorage* storage = butterfly->arrayStorage();
>+        storage->m_indexBias = 0;
>+        storage->m_sparseMap.clear();
>+        storage->m_numValuesInVector = initialLength;
>+        for (unsigned i = initialLength; i < vectorLength; ++i)
>+            storage->m_vector[i].clear();
>+    }
>+
>+    return createWithButterfly(vm, structure, butterfly);
>+}
>+
> void JSArray::setLengthWritable(ExecState* exec, bool writable)
> {
>+    LOGIT();
>+    
>     ASSERT(isLengthWritable() || !writable);
>     if (!isLengthWritable() || writable)
>         return;
>@@ -238,11 +294,13 @@ void JSArray::getOwnNonIndexPropertyName
> // This method makes room in the vector, but leaves the new space for count slots uncleared.
> bool JSArray::unshiftCountSlowCase(VM& vm, bool addToFront, unsigned count)
> {
>+    LOGIT();
>+    
>     ArrayStorage* storage = ensureArrayStorage(vm);
>     Butterfly* butterfly = storage->butterfly();
>     unsigned propertyCapacity = structure(vm)->outOfLineCapacity();
>     unsigned propertySize = structure(vm)->outOfLineSize();
>-
>+    
>     // If not, we should have handled this on the fast path.
>     ASSERT(!addToFront || count > storage->m_indexBias);
> 
>@@ -254,7 +312,8 @@ bool JSArray::unshiftCountSlowCase(VM& v
>     //  * desiredCapacity - how large should we like to grow the vector to - based on 2x requiredVectorLength.
> 
>     unsigned length = storage->length();
>-    unsigned usedVectorLength = min(storage->vectorLength(), length);
>+    unsigned oldVectorLength = storage->vectorLength();
>+    unsigned usedVectorLength = min(oldVectorLength, length);
>     ASSERT(usedVectorLength <= MAX_STORAGE_VECTOR_LENGTH);
>     // Check that required vector length is possible, in an overflow-safe fashion.
>     if (count > MAX_STORAGE_VECTOR_LENGTH - usedVectorLength)
>@@ -265,7 +324,10 @@ bool JSArray::unshiftCountSlowCase(VM& v
>     ASSERT(storage->vectorLength() <= MAX_STORAGE_VECTOR_LENGTH && (MAX_STORAGE_VECTOR_LENGTH - storage->vectorLength()) >= storage->m_indexBias);
>     unsigned currentCapacity = storage->vectorLength() + storage->m_indexBias;
>     // The calculation of desiredCapacity won't overflow, due to the range of MAX_STORAGE_VECTOR_LENGTH.
>-    unsigned desiredCapacity = min(MAX_STORAGE_VECTOR_LENGTH, max(BASE_VECTOR_LEN, requiredVectorLength) << 1);
>+    // FIXME: This code should be fixed to avoid internal fragmentation. It's not super high
>+    // priority since increaseVectorLength() will "fix" any mistakes we make, but it would be cool
>+    // to get this right eventually.
>+    unsigned desiredCapacity = min(MAX_STORAGE_VECTOR_LENGTH, max(BASE_ARRAY_STORAGE_VECTOR_LEN, requiredVectorLength) << 1);
> 
>     // Step 2:
>     // We're either going to choose to allocate a new ArrayStorage, or we're going to reuse the existing one.
>@@ -273,15 +335,19 @@ bool JSArray::unshiftCountSlowCase(VM& v
>     DeferGC deferGC(vm.heap);
>     void* newAllocBase = 0;
>     unsigned newStorageCapacity;
>+    bool allocatedNewStorage;
>     // If the current storage array is sufficiently large (but not too large!) then just keep using it.
>     if (currentCapacity > desiredCapacity && isDenseEnoughForVector(currentCapacity, requiredVectorLength)) {
>         newAllocBase = butterfly->base(structure(vm));
>         newStorageCapacity = currentCapacity;
>+        allocatedNewStorage = false;
>     } else {
>         size_t newSize = Butterfly::totalSize(0, propertyCapacity, true, ArrayStorage::sizeFor(desiredCapacity));
>-        if (!vm.heap.tryAllocateStorage(this, newSize, &newAllocBase))
>+        newAllocBase = vm.heap.tryAllocateAuxiliary(this, newSize);
>+        if (!newAllocBase)
>             return false;
>         newStorageCapacity = desiredCapacity;
>+        allocatedNewStorage = true;
>     }
> 
>     // Step 3:
>@@ -311,13 +377,19 @@ bool JSArray::unshiftCountSlowCase(VM& v
>         ASSERT(count + usedVectorLength <= newVectorLength);
>         memmove(newButterfly->arrayStorage()->m_vector + count, storage->m_vector, sizeof(JSValue) * usedVectorLength);
>         memmove(newButterfly->propertyStorage() - propertySize, butterfly->propertyStorage() - propertySize, sizeof(JSValue) * propertySize + sizeof(IndexingHeader) + ArrayStorage::sizeFor(0));
>+        
>+        if (allocatedNewStorage) {
>+            // We will set the vectorLength to newVectorLength. We populated requiredVectorLength
>+            // (usedVectorLength + count), which is less. Clear the difference.
>+            for (unsigned i = requiredVectorLength; i < newVectorLength; ++i)
>+                newButterfly->arrayStorage()->m_vector[i].clear();
>+        }
>     } else if ((newAllocBase != butterfly->base(structure(vm))) || (newIndexBias != storage->m_indexBias)) {
>         memmove(newButterfly->propertyStorage() - propertySize, butterfly->propertyStorage() - propertySize, sizeof(JSValue) * propertySize + sizeof(IndexingHeader) + ArrayStorage::sizeFor(0));
>         memmove(newButterfly->arrayStorage()->m_vector, storage->m_vector, sizeof(JSValue) * usedVectorLength);
>-
>-        WriteBarrier<Unknown>* newVector = newButterfly->arrayStorage()->m_vector;
>+        
>         for (unsigned i = requiredVectorLength; i < newVectorLength; i++)
>-            newVector[i].clear();
>+            newButterfly->arrayStorage()->m_vector[i].clear();
>     }
> 
>     newButterfly->arrayStorage()->setVectorLength(newVectorLength);
>@@ -329,8 +401,10 @@ bool JSArray::unshiftCountSlowCase(VM& v
> 
> bool JSArray::setLengthWithArrayStorage(ExecState* exec, unsigned newLength, bool throwException, ArrayStorage* storage)
> {
>+    LOGIT();
>+    
>     unsigned length = storage->length();
>-
>+    
>     // If the length is read only then we enter sparse mode, so should enter the following 'if'.
>     ASSERT(isLengthWritable() || storage->m_sparseMap);
> 
>@@ -393,6 +467,8 @@ bool JSArray::setLengthWithArrayStorage(
> 
> bool JSArray::appendMemcpy(ExecState* exec, VM& vm, unsigned startIndex, JSC::JSArray* otherArray)
> {
>+    LOGIT();
>+    
>     if (!canFastCopy(vm, otherArray))
>         return false;
> 
>@@ -433,6 +509,8 @@ bool JSArray::appendMemcpy(ExecState* ex
> 
> bool JSArray::setLength(ExecState* exec, unsigned newLength, bool throwException)
> {
>+    LOGIT();
>+    
>     Butterfly* butterfly = m_butterfly.get();
>     switch (indexingType()) {
>     case ArrayClass:
>@@ -497,6 +575,8 @@ bool JSArray::setLength(ExecState* exec,
> 
> JSValue JSArray::pop(ExecState* exec)
> {
>+    LOGIT();
>+    
>     Butterfly* butterfly = m_butterfly.get();
>     
>     switch (indexingType()) {
>@@ -594,6 +674,8 @@ JSValue JSArray::pop(ExecState* exec)
> //  - pushing to an array of length 2^32-1 stores the property, but throws a range error.
> void JSArray::push(ExecState* exec, JSValue value)
> {
>+    LOGIT();
>+    
>     Butterfly* butterfly = m_butterfly.get();
>     
>     switch (indexingType()) {
>@@ -730,6 +812,8 @@ void JSArray::push(ExecState* exec, JSVa
> 
> JSArray* JSArray::fastSlice(ExecState& exec, unsigned startIndex, unsigned count)
> {
>+    LOGIT();
>+    
>     auto arrayType = indexingType();
>     switch (arrayType) {
>     case ArrayWithDouble:
>@@ -760,6 +844,8 @@ JSArray* JSArray::fastSlice(ExecState& e
> 
> bool JSArray::shiftCountWithArrayStorage(VM& vm, unsigned startIndex, unsigned count, ArrayStorage* storage)
> {
>+    LOGIT();
>+    
>     unsigned oldLength = storage->length();
>     RELEASE_ASSERT(count <= oldLength);
>     
>@@ -865,6 +951,8 @@ bool JSArray::shiftCountWithArrayStorage
> 
> bool JSArray::shiftCountWithAnyIndexingType(ExecState* exec, unsigned& startIndex, unsigned count)
> {
>+    LOGIT();
>+    
>     VM& vm = exec->vm();
>     RELEASE_ASSERT(count > 0);
> 
>@@ -963,6 +1051,8 @@ bool JSArray::shiftCountWithAnyIndexingT
> // Returns true if the unshift can be handled, false to fallback.    
> bool JSArray::unshiftCountWithArrayStorage(ExecState* exec, unsigned startIndex, unsigned count, ArrayStorage* storage)
> {
>+    LOGIT();
>+    
>     unsigned length = storage->length();
> 
>     RELEASE_ASSERT(startIndex <= length);
>@@ -1100,6 +1190,8 @@ bool JSArray::unshiftCountWithAnyIndexin
> 
> void JSArray::fillArgList(ExecState* exec, MarkedArgumentBuffer& args)
> {
>+    LOGIT();
>+    
>     unsigned i = 0;
>     unsigned vectorEnd;
>     WriteBarrier<Unknown>* vector;
>@@ -1166,6 +1258,8 @@ void JSArray::fillArgList(ExecState* exe
> 
> void JSArray::copyToArguments(ExecState* exec, VirtualRegister firstElementDest, unsigned offset, unsigned length)
> {
>+    LOGIT();
>+    
>     unsigned i = offset;
>     WriteBarrier<Unknown>* vector;
>     unsigned vectorEnd;
>@@ -1175,7 +1269,6 @@ void JSArray::copyToArguments(ExecState*
>     ASSERT(length == this->length());
> 
>     Butterfly* butterfly = m_butterfly.get();
>-    
>     switch (indexingType()) {
>     case ArrayClass:
>         return;
>Index: Source/JavaScriptCore/runtime/JSArray.h
>===================================================================
>--- Source/JavaScriptCore/runtime/JSArray.h	(revision 204735)
>+++ Source/JavaScriptCore/runtime/JSArray.h	(working copy)
>@@ -1,6 +1,6 @@
> /*
>  *  Copyright (C) 1999-2000 Harri Porten (porten@kde.org)
>- *  Copyright (C) 2003, 2007, 2008, 2009, 2012, 2015 Apple Inc. All rights reserved.
>+ *  Copyright (C) 2003, 2007, 2008, 2009, 2012, 2015-2016 Apple Inc. All rights reserved.
>  *
>  *  This library is free software; you can redistribute it and/or
>  *  modify it under the terms of the GNU Lesser General Public
>@@ -60,7 +60,7 @@ public:
>     // contents are known at time of creation. Clients of this interface must:
>     //   - null-check the result (indicating out of memory, or otherwise unable to allocate vector).
>     //   - call 'initializeIndex' for all properties in sequence, for 0 <= i < initialLength.
>-    static JSArray* tryCreateUninitialized(VM&, Structure*, unsigned initialLength);
>+    JS_EXPORT_PRIVATE static JSArray* tryCreateUninitialized(VM&, Structure*, unsigned initialLength);
> 
>     JS_EXPORT_PRIVATE static bool defineOwnProperty(JSObject*, ExecState*, PropertyName, const PropertyDescriptor&, bool throwException);
> 
>@@ -177,7 +177,8 @@ private:
> inline Butterfly* createContiguousArrayButterfly(VM& vm, JSCell* intendedOwner, unsigned length, unsigned& vectorLength)
> {
>     IndexingHeader header;
>-    vectorLength = std::max(length, BASE_VECTOR_LEN);
>+    vectorLength = Butterfly::optimalContiguousVectorLength(
>+        intendedOwner ? intendedOwner->structure(vm) : 0, length);
>     header.setVectorLength(vectorLength);
>     header.setPublicLength(length);
>     Butterfly* result = Butterfly::create(
>@@ -188,11 +189,11 @@ inline Butterfly* createContiguousArrayB
> inline Butterfly* createArrayButterfly(VM& vm, JSCell* intendedOwner, unsigned initialLength)
> {
>     Butterfly* butterfly = Butterfly::create(
>-        vm, intendedOwner, 0, 0, true, baseIndexingHeaderForArray(initialLength),
>-        ArrayStorage::sizeFor(BASE_VECTOR_LEN));
>+        vm, intendedOwner, 0, 0, true, baseIndexingHeaderForArrayStorage(initialLength),
>+        ArrayStorage::sizeFor(BASE_ARRAY_STORAGE_VECTOR_LEN));
>     ArrayStorage* storage = butterfly->arrayStorage();
>-    storage->m_indexBias = 0;
>     storage->m_sparseMap.clear();
>+    storage->m_indexBias = 0;
>     storage->m_numValuesInVector = 0;
>     return butterfly;
> }
>@@ -211,57 +212,20 @@ inline JSArray* JSArray::create(VM& vm, 
>             || hasContiguous(structure->indexingType()));
>         unsigned vectorLength;
>         butterfly = createContiguousArrayButterfly(vm, 0, initialLength, vectorLength);
>-        ASSERT(initialLength < MIN_ARRAY_STORAGE_CONSTRUCTION_LENGTH);
>         if (hasDouble(structure->indexingType())) {
>             for (unsigned i = 0; i < vectorLength; ++i)
>                 butterfly->contiguousDouble()[i] = PNaN;
>+        } else {
>+            for (unsigned i = 0; i < vectorLength; ++i)
>+                butterfly->contiguous()[i].clear();
>         }
>     } else {
>         ASSERT(
>             structure->indexingType() == ArrayWithSlowPutArrayStorage
>             || structure->indexingType() == ArrayWithArrayStorage);
>         butterfly = createArrayButterfly(vm, 0, initialLength);
>-    }
>-
>-    return createWithButterfly(vm, structure, butterfly);
>-}
>-
>-inline JSArray* JSArray::tryCreateUninitialized(VM& vm, Structure* structure, unsigned initialLength)
>-{
>-    unsigned vectorLength = std::max(BASE_VECTOR_LEN, initialLength);
>-    if (vectorLength > MAX_STORAGE_VECTOR_LENGTH)
>-        return 0;
>-
>-    unsigned outOfLineStorage = structure->outOfLineCapacity();
>-
>-    Butterfly* butterfly;
>-    if (LIKELY(!hasAnyArrayStorage(structure->indexingType()))) {
>-        ASSERT(
>-            hasUndecided(structure->indexingType())
>-            || hasInt32(structure->indexingType())
>-            || hasDouble(structure->indexingType())
>-            || hasContiguous(structure->indexingType()));
>-
>-        void* temp;
>-        if (!vm.heap.tryAllocateStorage(0, Butterfly::totalSize(0, outOfLineStorage, true, vectorLength * sizeof(EncodedJSValue)), &temp))
>-            return 0;
>-        butterfly = Butterfly::fromBase(temp, 0, outOfLineStorage);
>-        butterfly->setVectorLength(vectorLength);
>-        butterfly->setPublicLength(initialLength);
>-        if (hasDouble(structure->indexingType())) {
>-            for (unsigned i = initialLength; i < vectorLength; ++i)
>-                butterfly->contiguousDouble()[i] = PNaN;
>-        }
>-    } else {
>-        void* temp;
>-        if (!vm.heap.tryAllocateStorage(0, Butterfly::totalSize(0, outOfLineStorage, true, ArrayStorage::sizeFor(vectorLength)), &temp))
>-            return 0;
>-        butterfly = Butterfly::fromBase(temp, 0, outOfLineStorage);
>-        *butterfly->indexingHeader() = indexingHeaderForArray(initialLength, vectorLength);
>-        ArrayStorage* storage = butterfly->arrayStorage();
>-        storage->m_indexBias = 0;
>-        storage->m_sparseMap.clear();
>-        storage->m_numValuesInVector = initialLength;
>+        for (unsigned i = 0; i < BASE_ARRAY_STORAGE_VECTOR_LEN; ++i)
>+            butterfly->arrayStorage()->m_vector[i].clear();
>     }
> 
>     return createWithButterfly(vm, structure, butterfly);
>Index: Source/JavaScriptCore/runtime/JSArrayBufferView.h
>===================================================================
>--- Source/JavaScriptCore/runtime/JSArrayBufferView.h	(revision 204735)
>+++ Source/JavaScriptCore/runtime/JSArrayBufferView.h	(working copy)
>@@ -26,6 +26,7 @@
> #ifndef JSArrayBufferView_h
> #define JSArrayBufferView_h
> 
>+#include "CopyBarrier.h"
> #include "JSObject.h"
> 
> namespace JSC {
>Index: Source/JavaScriptCore/runtime/JSCInlines.h
>===================================================================
>--- Source/JavaScriptCore/runtime/JSCInlines.h	(revision 204735)
>+++ Source/JavaScriptCore/runtime/JSCInlines.h	(working copy)
>@@ -1,5 +1,5 @@
> /*
>- * Copyright (C) 2014 Apple Inc. All rights reserved.
>+ * Copyright (C) 2014, 2016 Apple Inc. All rights reserved.
>  *
>  * Redistribution and use in source and binary forms, with or without
>  * modification, are permitted provided that the following conditions
>@@ -41,7 +41,6 @@
> #include "GCIncomingRefCountedInlines.h"
> #include "HeapInlines.h"
> #include "IdentifierInlines.h"
>-#include "Interpreter.h"
> #include "JSArrayBufferViewInlines.h"
> #include "JSCJSValueInlines.h"
> #include "JSFunctionInlines.h"
>Index: Source/JavaScriptCore/runtime/JSCJSValue.cpp
>===================================================================
>--- Source/JavaScriptCore/runtime/JSCJSValue.cpp	(revision 204735)
>+++ Source/JavaScriptCore/runtime/JSCJSValue.cpp	(working copy)
>@@ -29,11 +29,10 @@
> #include "Error.h"
> #include "ExceptionHelpers.h"
> #include "GetterSetter.h"
>-#include "JSCJSValueInlines.h"
>+#include "JSCInlines.h"
> #include "JSFunction.h"
> #include "JSGlobalObject.h"
> #include "NumberObject.h"
>-#include "StructureInlines.h"
> #include <wtf/MathExtras.h>
> #include <wtf/StringExtras.h>
> 
>@@ -256,7 +255,11 @@ void JSValue::dumpInContextAssumingStruc
>             out.print("Symbol: ", RawPointer(asCell()));
>         else if (structure->classInfo()->isSubClassOf(Structure::info()))
>             out.print("Structure: ", inContext(*jsCast<Structure*>(asCell()), context));
>-        else {
>+        else if (structure->classInfo()->isSubClassOf(JSObject::info())) {
>+            out.print("Object: ", RawPointer(asCell()));
>+            out.print(" with butterfly ", RawPointer(asObject(asCell())->butterfly()));
>+            out.print(" (", inContext(*structure, context), ")");
>+        } else {
>             out.print("Cell: ", RawPointer(asCell()));
>             out.print(" (", inContext(*structure, context), ")");
>         }
>Index: Source/JavaScriptCore/runtime/JSCallee.cpp
>===================================================================
>--- Source/JavaScriptCore/runtime/JSCallee.cpp	(revision 204735)
>+++ Source/JavaScriptCore/runtime/JSCallee.cpp	(working copy)
>@@ -1,5 +1,5 @@
> /*
>- * Copyright (C) 2014 Apple Inc. All rights reserved.
>+ * Copyright (C) 2014, 2016 Apple Inc. All rights reserved.
>  *
>  * Redistribution and use in source and binary forms, with or without
>  * modification, are permitted provided that the following conditions
>@@ -27,13 +27,9 @@
> #include "JSCallee.h"
> 
> #include "GetterSetter.h"
>-#include "JSCJSValueInlines.h"
>-#include "JSCell.h"
>-#include "JSCellInlines.h"
>+#include "JSCInlines.h"
> #include "JSGlobalObject.h"
>-#include "SlotVisitorInlines.h"
> #include "StackVisitor.h"
>-#include "StructureInlines.h"
> 
> namespace JSC {
> 
>@@ -43,6 +39,7 @@ JSCallee::JSCallee(VM& vm, JSGlobalObjec
>     : Base(vm, structure)
>     , m_scope(vm, this, globalObject)
> {
>+    RELEASE_ASSERT(!isLargeAllocation());
> }
> 
> JSCallee::JSCallee(VM& vm, JSScope* scope, Structure* structure)
>Index: Source/JavaScriptCore/runtime/JSCell.cpp
>===================================================================
>--- Source/JavaScriptCore/runtime/JSCell.cpp	(revision 204735)
>+++ Source/JavaScriptCore/runtime/JSCell.cpp	(working copy)
>@@ -58,7 +58,7 @@ size_t JSCell::estimatedSizeInBytes() co
> 
> size_t JSCell::estimatedSize(JSCell* cell)
> {
>-    return MarkedBlock::blockFor(cell)->cellSize();
>+    return cell->cellSize();
> }
> 
> void JSCell::copyBackingStore(JSCell*, CopyVisitor&, CopyToken)
>Index: Source/JavaScriptCore/runtime/JSCell.h
>===================================================================
>--- Source/JavaScriptCore/runtime/JSCell.h	(revision 204735)
>+++ Source/JavaScriptCore/runtime/JSCell.h	(working copy)
>@@ -80,7 +80,7 @@ public:
> 
>     enum CreatingEarlyCellTag { CreatingEarlyCell };
>     JSCell(CreatingEarlyCellTag);
>-
>+    
> protected:
>     JSCell(VM&, Structure*);
>     JS_EXPORT_PRIVATE static void destroy(JSCell*);
>@@ -108,8 +108,6 @@ public:
> 
>     const char* className() const;
> 
>-    VM* vm() const;
>-
>     // Extracting the value.
>     JS_EXPORT_PRIVATE bool getString(ExecState*, String&) const;
>     JS_EXPORT_PRIVATE String getString(ExecState*) const; // null string if not a string
>@@ -190,6 +188,8 @@ public:
>     {
>         return OBJECT_OFFSETOF(JSCell, m_cellState);
>     }
>+    
>+    void callDestructor(VM&);
> 
>     static const TypedArrayType TypedArrayStorageType = NotTypedArray;
> protected:
>Index: Source/JavaScriptCore/runtime/JSCellInlines.h
>===================================================================
>--- Source/JavaScriptCore/runtime/JSCellInlines.h	(revision 204735)
>+++ Source/JavaScriptCore/runtime/JSCellInlines.h	(working copy)
>@@ -113,16 +113,13 @@ inline void JSCell::visitChildren(JSCell
>     visitor.appendUnbarrieredPointer(&structure);
> }
> 
>-inline VM* JSCell::vm() const
>-{
>-    return MarkedBlock::blockFor(this)->vm();
>-}
>-
> ALWAYS_INLINE VM& ExecState::vm() const
> {
>     ASSERT(callee());
>     ASSERT(callee()->vm());
>-    return *calleeAsValue().asCell()->vm();
>+    ASSERT(!callee()->isLargeAllocation());
>+    // This is an important optimization since we access this so often.
>+    return *calleeAsValue().asCell()->markedBlock().vm();
> }
> 
> template<typename T>
>@@ -233,12 +230,19 @@ inline bool JSCell::canUseFastGetOwnProp
>         && !structure.typeInfo().overridesGetOwnPropertySlot();
> }
> 
>-inline const ClassInfo* JSCell::classInfo() const
>+ALWAYS_INLINE const ClassInfo* JSCell::classInfo() const
> {
>-    MarkedBlock* block = MarkedBlock::blockFor(this);
>-    if (block->needsDestruction() && !(inlineTypeFlags() & StructureIsImmortal))
>+    if (isLargeAllocation()) {
>+        LargeAllocation& allocation = largeAllocation();
>+        if (allocation.attributes().destruction == NeedsDestruction
>+            && !(inlineTypeFlags() & StructureIsImmortal))
>+            return static_cast<const JSDestructibleObject*>(this)->classInfo();
>+        return structure(*allocation.vm())->classInfo();
>+    }
>+    MarkedBlock& block = markedBlock();
>+    if (block.needsDestruction() && !(inlineTypeFlags() & StructureIsImmortal))
>         return static_cast<const JSDestructibleObject*>(this)->classInfo();
>-    return structure(*block->vm())->classInfo();
>+    return structure(*block.vm())->classInfo();
> }
> 
> inline bool JSCell::toBoolean(ExecState* exec) const
>@@ -257,6 +261,18 @@ inline TriState JSCell::pureToBoolean() 
>     return MixedTriState;
> }
> 
>+inline void JSCell::callDestructor(VM& vm)
>+{
>+    if (isZapped())
>+        return;
>+    ASSERT(structureID());
>+    if (inlineTypeFlags() & StructureIsImmortal)
>+        structure(vm)->classInfo()->methodTable.destroy(this);
>+    else
>+        jsCast<JSDestructibleObject*>(this)->classInfo()->methodTable.destroy(this);
>+    zap();
>+}
>+
> } // namespace JSC
> 
> #endif // JSCellInlines_h
>Index: Source/JavaScriptCore/runtime/JSFunction.cpp
>===================================================================
>--- Source/JavaScriptCore/runtime/JSFunction.cpp	(revision 204735)
>+++ Source/JavaScriptCore/runtime/JSFunction.cpp	(working copy)
>@@ -62,7 +62,7 @@ bool JSFunction::isHostFunctionNonInline
> 
> JSFunction* JSFunction::create(VM& vm, FunctionExecutable* executable, JSScope* scope)
> {
>-    return create(vm, executable, scope, scope->globalObject()->functionStructure());
>+    return create(vm, executable, scope, scope->globalObject(vm)->functionStructure());
> }
> 
> JSFunction* JSFunction::create(VM& vm, FunctionExecutable* executable, JSScope* scope, Structure* structure)
>@@ -76,7 +76,7 @@ JSFunction* JSFunction::create(VM& vm, F
> JSFunction* JSFunction::create(VM& vm, WebAssemblyExecutable* executable, JSScope* scope)
> {
>     JSFunction* function = new (NotNull, allocateCell<JSFunction>(vm.heap)) JSFunction(vm, executable, scope);
>-    ASSERT(function->structure()->globalObject());
>+    ASSERT(function->structure(vm)->globalObject());
>     function->finishCreation(vm);
>     return function;
> }
>@@ -143,9 +143,9 @@ FunctionRareData* JSFunction::allocateAn
>     VM& vm = exec->vm();
>     JSObject* prototype = jsDynamicCast<JSObject*>(get(exec, vm.propertyNames->prototype));
>     if (!prototype)
>-        prototype = globalObject()->objectPrototype();
>+        prototype = globalObject(vm)->objectPrototype();
>     FunctionRareData* rareData = FunctionRareData::create(vm);
>-    rareData->initializeObjectAllocationProfile(globalObject()->vm(), prototype, inlineCapacity);
>+    rareData->initializeObjectAllocationProfile(vm, prototype, inlineCapacity);
> 
>     // A DFG compilation thread may be trying to read the rare data
>     // We want to ensure that it sees it properly allocated
>@@ -161,8 +161,8 @@ FunctionRareData* JSFunction::initialize
>     VM& vm = exec->vm();
>     JSObject* prototype = jsDynamicCast<JSObject*>(get(exec, vm.propertyNames->prototype));
>     if (!prototype)
>-        prototype = globalObject()->objectPrototype();
>-    m_rareData->initializeObjectAllocationProfile(globalObject()->vm(), prototype, inlineCapacity);
>+        prototype = globalObject(vm)->objectPrototype();
>+    m_rareData->initializeObjectAllocationProfile(vm, prototype, inlineCapacity);
>     return m_rareData.get();
> }
> 
>@@ -337,37 +337,37 @@ EncodedJSValue JSFunction::callerGetter(
> 
> bool JSFunction::getOwnPropertySlot(JSObject* object, ExecState* exec, PropertyName propertyName, PropertySlot& slot)
> {
>+    VM& vm = exec->vm();
>     JSFunction* thisObject = jsCast<JSFunction*>(object);
>     if (thisObject->isHostOrBuiltinFunction()) {
>-        thisObject->reifyBoundNameIfNeeded(exec, propertyName);
>+        thisObject->reifyBoundNameIfNeeded(vm, exec, propertyName);
>         return Base::getOwnPropertySlot(thisObject, exec, propertyName, slot);
>     }
> 
>-    if (propertyName == exec->propertyNames().prototype && !thisObject->jsExecutable()->isArrowFunction()) {
>-        VM& vm = exec->vm();
>+    if (propertyName == vm.propertyNames->prototype && !thisObject->jsExecutable()->isArrowFunction()) {
>         unsigned attributes;
>         PropertyOffset offset = thisObject->getDirectOffset(vm, propertyName, attributes);
>         if (!isValidOffset(offset)) {
>             JSObject* prototype = nullptr;
>             if (thisObject->jsExecutable()->parseMode() == SourceParseMode::GeneratorWrapperFunctionMode)
>-                prototype = constructEmptyObject(exec, thisObject->globalObject()->generatorPrototype());
>+                prototype = constructEmptyObject(exec, thisObject->globalObject(vm)->generatorPrototype());
>             else
>                 prototype = constructEmptyObject(exec);
> 
>-            prototype->putDirect(vm, exec->propertyNames().constructor, thisObject, DontEnum);
>-            thisObject->putDirect(vm, exec->propertyNames().prototype, prototype, DontDelete | DontEnum);
>-            offset = thisObject->getDirectOffset(vm, exec->propertyNames().prototype, attributes);
>+            prototype->putDirect(vm, vm.propertyNames->constructor, thisObject, DontEnum);
>+            thisObject->putDirect(vm, vm.propertyNames->prototype, prototype, DontDelete | DontEnum);
>+            offset = thisObject->getDirectOffset(vm, vm.propertyNames->prototype, attributes);
>             ASSERT(isValidOffset(offset));
>         }
> 
>         slot.setValue(thisObject, attributes, thisObject->getDirect(offset), offset);
>     }
> 
>-    if (propertyName == exec->propertyNames().arguments) {
>+    if (propertyName == vm.propertyNames->arguments) {
>         if (thisObject->jsExecutable()->isStrictMode() || thisObject->jsExecutable()->isClassConstructorFunction()) {
>             bool result = Base::getOwnPropertySlot(thisObject, exec, propertyName, slot);
>             if (!result) {
>-                GetterSetter* errorGetterSetter = thisObject->globalObject()->throwTypeErrorArgumentsCalleeAndCallerGetterSetter();
>+                GetterSetter* errorGetterSetter = thisObject->globalObject(vm)->throwTypeErrorArgumentsCalleeAndCallerGetterSetter();
>                 thisObject->putDirectAccessor(exec, propertyName, errorGetterSetter, DontDelete | DontEnum | Accessor);
>                 result = Base::getOwnPropertySlot(thisObject, exec, propertyName, slot);
>                 ASSERT(result);
>@@ -378,11 +378,11 @@ bool JSFunction::getOwnPropertySlot(JSOb
>         return true;
>     }
> 
>-    if (propertyName == exec->propertyNames().caller) {
>+    if (propertyName == vm.propertyNames->caller) {
>         if (thisObject->jsExecutable()->isStrictMode() || thisObject->jsExecutable()->isClassConstructorFunction()) {
>             bool result = Base::getOwnPropertySlot(thisObject, exec, propertyName, slot);
>             if (!result) {
>-                GetterSetter* errorGetterSetter = thisObject->globalObject()->throwTypeErrorArgumentsCalleeAndCallerGetterSetter();
>+                GetterSetter* errorGetterSetter = thisObject->globalObject(vm)->throwTypeErrorArgumentsCalleeAndCallerGetterSetter();
>                 thisObject->putDirectAccessor(exec, propertyName, errorGetterSetter, DontDelete | DontEnum | Accessor);
>                 result = Base::getOwnPropertySlot(thisObject, exec, propertyName, slot);
>                 ASSERT(result);
>@@ -393,7 +393,7 @@ bool JSFunction::getOwnPropertySlot(JSOb
>         return true;
>     }
> 
>-    thisObject->reifyLazyPropertyIfNeeded(exec, propertyName);
>+    thisObject->reifyLazyPropertyIfNeeded(vm, exec, propertyName);
> 
>     return Base::getOwnPropertySlot(thisObject, exec, propertyName, slot);
> }
>@@ -420,39 +420,40 @@ void JSFunction::getOwnNonIndexPropertyN
> 
> bool JSFunction::put(JSCell* cell, ExecState* exec, PropertyName propertyName, JSValue value, PutPropertySlot& slot)
> {
>+    VM& vm = exec->vm();
>     JSFunction* thisObject = jsCast<JSFunction*>(cell);
> 
>     if (UNLIKELY(isThisValueAltered(slot, thisObject)))
>         return ordinarySetSlow(exec, thisObject, propertyName, value, slot.thisValue(), slot.isStrictMode());
> 
>     if (thisObject->isHostOrBuiltinFunction()) {
>-        thisObject->reifyBoundNameIfNeeded(exec, propertyName);
>+        thisObject->reifyBoundNameIfNeeded(vm, exec, propertyName);
>         return Base::put(thisObject, exec, propertyName, value, slot);
>     }
> 
>-    if (propertyName == exec->propertyNames().prototype) {
>+    if (propertyName == vm.propertyNames->prototype) {
>         // Make sure prototype has been reified, such that it can only be overwritten
>         // following the rules set out in ECMA-262 8.12.9.
>         PropertySlot slot(thisObject, PropertySlot::InternalMethodType::VMInquiry);
>-        thisObject->methodTable(exec->vm())->getOwnPropertySlot(thisObject, exec, propertyName, slot);
>+        thisObject->methodTable(vm)->getOwnPropertySlot(thisObject, exec, propertyName, slot);
>         if (thisObject->m_rareData)
>             thisObject->m_rareData->clear("Store to prototype property of a function");
>         // Don't allow this to be cached, since a [[Put]] must clear m_rareData.
>         PutPropertySlot dontCache(thisObject);
>         return Base::put(thisObject, exec, propertyName, value, dontCache);
>     }
>-    if (thisObject->jsExecutable()->isStrictMode() && (propertyName == exec->propertyNames().arguments || propertyName == exec->propertyNames().caller)) {
>+    if (thisObject->jsExecutable()->isStrictMode() && (propertyName == vm.propertyNames->arguments || propertyName == vm.propertyNames->caller)) {
>         // This will trigger the property to be reified, if this is not already the case!
>         bool okay = thisObject->hasProperty(exec, propertyName);
>         ASSERT_UNUSED(okay, okay);
>         return Base::put(thisObject, exec, propertyName, value, slot);
>     }
>-    if (propertyName == exec->propertyNames().arguments || propertyName == exec->propertyNames().caller) {
>+    if (propertyName == vm.propertyNames->arguments || propertyName == vm.propertyNames->caller) {
>         if (slot.isStrictMode())
>             throwTypeError(exec, StrictModeReadonlyPropertyWriteError);
>         return false;
>     }
>-    thisObject->reifyLazyPropertyIfNeeded(exec, propertyName);
>+    thisObject->reifyLazyPropertyIfNeeded(vm, exec, propertyName);
>     return Base::put(thisObject, exec, propertyName, value, slot);
> }
> 
>@@ -460,16 +461,17 @@ bool JSFunction::deleteProperty(JSCell* 
> {
>     JSFunction* thisObject = jsCast<JSFunction*>(cell);
>     if (thisObject->isHostOrBuiltinFunction())
>-        thisObject->reifyBoundNameIfNeeded(exec, propertyName);
>+        thisObject->reifyBoundNameIfNeeded(exec->vm(), exec, propertyName);
>     else if (exec->vm().deletePropertyMode() != VM::DeletePropertyMode::IgnoreConfigurable) {
>         // For non-host functions, don't let these properties by deleted - except by DefineOwnProperty.
>+        VM& vm = exec->vm();
>         FunctionExecutable* executable = thisObject->jsExecutable();
>-        if (propertyName == exec->propertyNames().arguments
>-            || (propertyName == exec->propertyNames().prototype && !executable->isArrowFunction())
>-            || propertyName == exec->propertyNames().caller)
>+        if (propertyName == vm.propertyNames->arguments
>+            || (propertyName == vm.propertyNames->prototype && !executable->isArrowFunction())
>+            || propertyName == vm.propertyNames->caller)
>             return false;
> 
>-        thisObject->reifyLazyPropertyIfNeeded(exec, propertyName);
>+        thisObject->reifyLazyPropertyIfNeeded(vm, exec, propertyName);
>     }
>     
>     return Base::deleteProperty(thisObject, exec, propertyName);
>@@ -477,41 +479,42 @@ bool JSFunction::deleteProperty(JSCell* 
> 
> bool JSFunction::defineOwnProperty(JSObject* object, ExecState* exec, PropertyName propertyName, const PropertyDescriptor& descriptor, bool throwException)
> {
>+    VM& vm = exec->vm();
>     JSFunction* thisObject = jsCast<JSFunction*>(object);
>     if (thisObject->isHostOrBuiltinFunction()) {
>-        thisObject->reifyBoundNameIfNeeded(exec, propertyName);
>+        thisObject->reifyBoundNameIfNeeded(vm, exec, propertyName);
>         return Base::defineOwnProperty(object, exec, propertyName, descriptor, throwException);
>     }
> 
>-    if (propertyName == exec->propertyNames().prototype) {
>+    if (propertyName == vm.propertyNames->prototype) {
>         // Make sure prototype has been reified, such that it can only be overwritten
>         // following the rules set out in ECMA-262 8.12.9.
>         PropertySlot slot(thisObject, PropertySlot::InternalMethodType::VMInquiry);
>-        thisObject->methodTable(exec->vm())->getOwnPropertySlot(thisObject, exec, propertyName, slot);
>+        thisObject->methodTable(vm)->getOwnPropertySlot(thisObject, exec, propertyName, slot);
>         if (thisObject->m_rareData)
>             thisObject->m_rareData->clear("Store to prototype property of a function");
>         return Base::defineOwnProperty(object, exec, propertyName, descriptor, throwException);
>     }
> 
>     bool valueCheck;
>-    if (propertyName == exec->propertyNames().arguments) {
>+    if (propertyName == vm.propertyNames->arguments) {
>         if (thisObject->jsExecutable()->isStrictMode()) {
>             PropertySlot slot(thisObject, PropertySlot::InternalMethodType::VMInquiry);
>             if (!Base::getOwnPropertySlot(thisObject, exec, propertyName, slot))
>-                thisObject->putDirectAccessor(exec, propertyName, thisObject->globalObject()->throwTypeErrorArgumentsCalleeAndCallerGetterSetter(), DontDelete | DontEnum | Accessor);
>+                thisObject->putDirectAccessor(exec, propertyName, thisObject->globalObject(vm)->throwTypeErrorArgumentsCalleeAndCallerGetterSetter(), DontDelete | DontEnum | Accessor);
>             return Base::defineOwnProperty(object, exec, propertyName, descriptor, throwException);
>         }
>         valueCheck = !descriptor.value() || sameValue(exec, descriptor.value(), retrieveArguments(exec, thisObject));
>-    } else if (propertyName == exec->propertyNames().caller) {
>+    } else if (propertyName == vm.propertyNames->caller) {
>         if (thisObject->jsExecutable()->isStrictMode()) {
>             PropertySlot slot(thisObject, PropertySlot::InternalMethodType::VMInquiry);
>             if (!Base::getOwnPropertySlot(thisObject, exec, propertyName, slot))
>-                thisObject->putDirectAccessor(exec, propertyName, thisObject->globalObject()->throwTypeErrorArgumentsCalleeAndCallerGetterSetter(), DontDelete | DontEnum | Accessor);
>+                thisObject->putDirectAccessor(exec, propertyName, thisObject->globalObject(vm)->throwTypeErrorArgumentsCalleeAndCallerGetterSetter(), DontDelete | DontEnum | Accessor);
>             return Base::defineOwnProperty(object, exec, propertyName, descriptor, throwException);
>         }
>         valueCheck = !descriptor.value() || sameValue(exec, descriptor.value(), retrieveCallerFunction(exec, thisObject));
>     } else {
>-        thisObject->reifyLazyPropertyIfNeeded(exec, propertyName);
>+        thisObject->reifyLazyPropertyIfNeeded(vm, exec, propertyName);
>         return Base::defineOwnProperty(object, exec, propertyName, descriptor, throwException);
>     }
>      
>@@ -573,6 +576,7 @@ String getCalculatedDisplayName(VM& vm, 
> 
> void JSFunction::setFunctionName(ExecState* exec, JSValue value)
> {
>+    VM& vm = exec->vm();
>     // The "name" property may have been already been defined as part of a property list in an
>     // object literal (and therefore reified).
>     if (hasReifiedName())
>@@ -588,7 +592,6 @@ void JSFunction::setFunctionName(ExecSta
>         else
>             name = makeString('[', String(uid), ']');
>     } else {
>-        VM& vm = exec->vm();
>         JSString* jsStr = value.toString(exec);
>         if (vm.exception())
>             return;
>@@ -596,39 +599,37 @@ void JSFunction::setFunctionName(ExecSta
>         if (vm.exception())
>             return;
>     }
>-    reifyName(exec, name);
>+    reifyName(vm, exec, name);
> }
> 
>-void JSFunction::reifyLength(ExecState* exec)
>+void JSFunction::reifyLength(VM& vm)
> {
>-    VM& vm = exec->vm();
>     FunctionRareData* rareData = this->rareData(vm);
> 
>     ASSERT(!hasReifiedLength());
>     ASSERT(!isHostFunction());
>     JSValue initialValue = jsNumber(jsExecutable()->parameterCount());
>     unsigned initialAttributes = DontEnum | ReadOnly;
>-    const Identifier& identifier = exec->propertyNames().length;
>+    const Identifier& identifier = vm.propertyNames->length;
>     putDirect(vm, identifier, initialValue, initialAttributes);
> 
>     rareData->setHasReifiedLength();
> }
> 
>-void JSFunction::reifyName(ExecState* exec)
>+void JSFunction::reifyName(VM& vm, ExecState* exec)
> {
>     String name = jsExecutable()->ecmaName().string();
>-    reifyName(exec, name);
>+    reifyName(vm, exec, name);
> }
> 
>-void JSFunction::reifyName(ExecState* exec, String name)
>+void JSFunction::reifyName(VM& vm, ExecState* exec, String name)
> {
>-    VM& vm = exec->vm();
>     FunctionRareData* rareData = this->rareData(vm);
> 
>     ASSERT(!hasReifiedName());
>     ASSERT(!isHostFunction());
>     unsigned initialAttributes = DontEnum | ReadOnly;
>-    const Identifier& propID = exec->propertyNames().name;
>+    const Identifier& propID = vm.propertyNames->name;
> 
>     if (exec->lexicalGlobalObject()->needsSiteSpecificQuirks()) {
>         auto illegalCharMatcher = [] (UChar ch) -> bool {
>@@ -647,20 +648,20 @@ void JSFunction::reifyName(ExecState* ex
>     rareData->setHasReifiedName();
> }
> 
>-void JSFunction::reifyLazyPropertyIfNeeded(ExecState* exec, PropertyName propertyName)
>+void JSFunction::reifyLazyPropertyIfNeeded(VM& vm, ExecState* exec, PropertyName propertyName)
> {
>-    if (propertyName == exec->propertyNames().length) {
>+    if (propertyName == vm.propertyNames->length) {
>         if (!hasReifiedLength())
>-            reifyLength(exec);
>-    } else if (propertyName == exec->propertyNames().name) {
>+            reifyLength(vm);
>+    } else if (propertyName == vm.propertyNames->name) {
>         if (!hasReifiedName())
>-            reifyName(exec);
>+            reifyName(vm, exec);
>     }
> }
> 
>-void JSFunction::reifyBoundNameIfNeeded(ExecState* exec, PropertyName propertyName)
>+void JSFunction::reifyBoundNameIfNeeded(VM& vm, ExecState* exec, PropertyName propertyName)
> {
>-    const Identifier& nameIdent = exec->propertyNames().name;
>+    const Identifier& nameIdent = vm.propertyNames->name;
>     if (propertyName != nameIdent)
>         return;
> 
>@@ -668,7 +669,6 @@ void JSFunction::reifyBoundNameIfNeeded(
>         return;
> 
>     if (this->inherits(JSBoundFunction::info())) {
>-        VM& vm = exec->vm();
>         FunctionRareData* rareData = this->rareData(vm);
>         String name = makeString("bound ", static_cast<NativeExecutable*>(m_executable.get())->name());
>         unsigned initialAttributes = DontEnum | ReadOnly;
>Index: Source/JavaScriptCore/runtime/JSFunction.h
>===================================================================
>--- Source/JavaScriptCore/runtime/JSFunction.h	(revision 204735)
>+++ Source/JavaScriptCore/runtime/JSFunction.h	(working copy)
>@@ -189,11 +189,11 @@ private:
> 
>     bool hasReifiedLength() const;
>     bool hasReifiedName() const;
>-    void reifyLength(ExecState*);
>-    void reifyName(ExecState*);
>-    void reifyBoundNameIfNeeded(ExecState*, PropertyName);
>-    void reifyName(ExecState*, String name);
>-    void reifyLazyPropertyIfNeeded(ExecState*, PropertyName propertyName);
>+    void reifyLength(VM&);
>+    void reifyName(VM&, ExecState*);
>+    void reifyBoundNameIfNeeded(VM&, ExecState*, PropertyName);
>+    void reifyName(VM&, ExecState*, String name);
>+    void reifyLazyPropertyIfNeeded(VM&, ExecState*, PropertyName propertyName);
> 
>     friend class LLIntOffsetsExtractor;
> 
>Index: Source/JavaScriptCore/runtime/JSFunctionInlines.h
>===================================================================
>--- Source/JavaScriptCore/runtime/JSFunctionInlines.h	(revision 204735)
>+++ Source/JavaScriptCore/runtime/JSFunctionInlines.h	(working copy)
>@@ -35,7 +35,7 @@ inline JSFunction* JSFunction::createWit
>     VM& vm, FunctionExecutable* executable, JSScope* scope)
> {
>     ASSERT(executable->singletonFunction()->hasBeenInvalidated());
>-    return createImpl(vm, executable, scope, scope->globalObject()->functionStructure());
>+    return createImpl(vm, executable, scope, scope->globalObject(vm)->functionStructure());
> }
> 
> inline JSFunction::JSFunction(VM& vm, FunctionExecutable* executable, JSScope* scope, Structure* structure)
>@@ -47,7 +47,7 @@ inline JSFunction::JSFunction(VM& vm, Fu
> 
> #if ENABLE(WEBASSEMBLY)
> inline JSFunction::JSFunction(VM& vm, WebAssemblyExecutable* executable, JSScope* scope)
>-    : Base(vm, scope, scope->globalObject()->functionStructure())
>+    : Base(vm, scope, scope->globalObject(vm)->functionStructure())
>     , m_executable(vm, this, executable)
>     , m_rareData()
> {
>Index: Source/JavaScriptCore/runtime/JSGenericTypedArrayViewInlines.h
>===================================================================
>--- Source/JavaScriptCore/runtime/JSGenericTypedArrayViewInlines.h	(revision 204735)
>+++ Source/JavaScriptCore/runtime/JSGenericTypedArrayViewInlines.h	(working copy)
>@@ -510,25 +510,15 @@ ArrayBuffer* JSGenericTypedArrayView<Ada
>     // up. But if you do *anything* to trigger a GC watermark check, it will know
>     // that you *had* done those allocations and it will GC appropriately.
>     Heap* heap = Heap::heap(thisObject);
>+    VM& vm = *heap->vm();
>     DeferGCForAWhile deferGC(*heap);
>     
>     ASSERT(!thisObject->hasIndexingHeader());
> 
>-    size_t size = thisObject->byteSize();
>-    
>-    if (thisObject->m_mode == FastTypedArray
>-        && !thisObject->butterfly() && size >= sizeof(IndexingHeader)) {
>-        ASSERT(thisObject->m_vector);
>-        // Reuse already allocated memory if at all possible.
>-        thisObject->m_butterfly.setWithoutBarrier(
>-            bitwise_cast<IndexingHeader*>(thisObject->vector())->butterfly());
>-    } else {
>-        RELEASE_ASSERT(!thisObject->hasIndexingHeader());
>-        VM& vm = *heap->vm();
>-        thisObject->m_butterfly.set(vm, thisObject, Butterfly::createOrGrowArrayRight(
>-            thisObject->butterfly(), vm, thisObject, thisObject->structure(),
>-            thisObject->structure()->outOfLineCapacity(), false, 0, 0));
>-    }
>+    RELEASE_ASSERT(!thisObject->hasIndexingHeader());
>+    thisObject->m_butterfly.set(vm, thisObject, Butterfly::createOrGrowArrayRight(
>+        thisObject->butterfly(), vm, thisObject, thisObject->structure(),
>+        thisObject->structure()->outOfLineCapacity(), false, 0, 0));
> 
>     RefPtr<ArrayBuffer> buffer;
>     
>Index: Source/JavaScriptCore/runtime/JSInternalPromise.cpp
>===================================================================
>--- Source/JavaScriptCore/runtime/JSInternalPromise.cpp	(revision 204735)
>+++ Source/JavaScriptCore/runtime/JSInternalPromise.cpp	(working copy)
>@@ -27,9 +27,7 @@
> #include "JSInternalPromise.h"
> 
> #include "BuiltinNames.h"
>-#include "JSCJSValueInlines.h"
>-#include "JSCellInlines.h"
>-#include "StructureInlines.h"
>+#include "JSCInlines.h"
> 
> namespace JSC {
> 
>Index: Source/JavaScriptCore/runtime/JSInternalPromiseConstructor.cpp
>===================================================================
>--- Source/JavaScriptCore/runtime/JSInternalPromiseConstructor.cpp	(revision 204735)
>+++ Source/JavaScriptCore/runtime/JSInternalPromiseConstructor.cpp	(working copy)
>@@ -27,11 +27,9 @@
> #include "JSInternalPromiseConstructor.h"
> 
> #include "JSCBuiltins.h"
>-#include "JSCJSValueInlines.h"
>-#include "JSCellInlines.h"
>+#include "JSCInlines.h"
> #include "JSInternalPromise.h"
> #include "JSInternalPromisePrototype.h"
>-#include "StructureInlines.h"
> 
> #include "JSInternalPromiseConstructor.lut.h"
> 
>Index: Source/JavaScriptCore/runtime/JSInternalPromiseDeferred.cpp
>===================================================================
>--- Source/JavaScriptCore/runtime/JSInternalPromiseDeferred.cpp	(revision 204735)
>+++ Source/JavaScriptCore/runtime/JSInternalPromiseDeferred.cpp	(working copy)
>@@ -29,12 +29,9 @@
> #include "BuiltinNames.h"
> #include "Error.h"
> #include "Exception.h"
>-#include "JSCJSValueInlines.h"
>-#include "JSCellInlines.h"
>+#include "JSCInlines.h"
> #include "JSInternalPromise.h"
> #include "JSInternalPromiseConstructor.h"
>-#include "SlotVisitorInlines.h"
>-#include "StructureInlines.h"
> 
> namespace JSC {
> 
>Index: Source/JavaScriptCore/runtime/JSInternalPromisePrototype.cpp
>===================================================================
>--- Source/JavaScriptCore/runtime/JSInternalPromisePrototype.cpp	(revision 204735)
>+++ Source/JavaScriptCore/runtime/JSInternalPromisePrototype.cpp	(working copy)
>@@ -28,12 +28,10 @@
> 
> #include "Error.h"
> #include "JSCBuiltins.h"
>-#include "JSCJSValueInlines.h"
>-#include "JSCellInlines.h"
>+#include "JSCInlines.h"
> #include "JSGlobalObject.h"
> #include "JSInternalPromise.h"
> #include "Microtask.h"
>-#include "StructureInlines.h"
> 
> namespace JSC {
> 
>Index: Source/JavaScriptCore/runtime/JSJob.cpp
>===================================================================
>--- Source/JavaScriptCore/runtime/JSJob.cpp	(revision 204735)
>+++ Source/JavaScriptCore/runtime/JSJob.cpp	(working copy)
>@@ -28,11 +28,9 @@
> 
> #include "Error.h"
> #include "Exception.h"
>-#include "JSCJSValueInlines.h"
>-#include "JSCellInlines.h"
>+#include "JSCInlines.h"
> #include "JSGlobalObject.h"
> #include "Microtask.h"
>-#include "SlotVisitorInlines.h"
> #include "StrongInlines.h"
> 
> namespace JSC {
>Index: Source/JavaScriptCore/runtime/JSMapIterator.cpp
>===================================================================
>--- Source/JavaScriptCore/runtime/JSMapIterator.cpp	(revision 204735)
>+++ Source/JavaScriptCore/runtime/JSMapIterator.cpp	(working copy)
>@@ -1,5 +1,5 @@
> /*
>- * Copyright (C) 2013 Apple, Inc. All rights reserved.
>+ * Copyright (C) 2013, 2016 Apple, Inc. All rights reserved.
>  *
>  * Redistribution and use in source and binary forms, with or without
>  * modification, are permitted provided that the following conditions
>@@ -26,12 +26,9 @@
> #include "config.h"
> #include "JSMapIterator.h"
> 
>-#include "JSCJSValueInlines.h"
>-#include "JSCellInlines.h"
>+#include "JSCInlines.h"
> #include "JSMap.h"
> #include "MapDataInlines.h"
>-#include "SlotVisitorInlines.h"
>-#include "StructureInlines.h"
> 
> namespace JSC {
> 
>Index: Source/JavaScriptCore/runtime/JSModuleNamespaceObject.cpp
>===================================================================
>--- Source/JavaScriptCore/runtime/JSModuleNamespaceObject.cpp	(revision 204735)
>+++ Source/JavaScriptCore/runtime/JSModuleNamespaceObject.cpp	(working copy)
>@@ -1,5 +1,5 @@
> /*
>- * Copyright (C) 2015 Apple Inc. All rights reserved.
>+ * Copyright (C) 2015-2016 Apple Inc. All rights reserved.
>  *
>  * Redistribution and use in source and binary forms, with or without
>  * modification, are permitted provided that the following conditions
>@@ -27,14 +27,10 @@
> #include "JSModuleNamespaceObject.h"
> 
> #include "Error.h"
>-#include "IdentifierInlines.h"
>-#include "JSCJSValueInlines.h"
>-#include "JSCellInlines.h"
>+#include "JSCInlines.h"
> #include "JSModuleEnvironment.h"
> #include "JSModuleRecord.h"
> #include "JSPropertyNameIterator.h"
>-#include "SlotVisitorInlines.h"
>-#include "StructureInlines.h"
> 
> namespace JSC {
> 
>Index: Source/JavaScriptCore/runtime/JSModuleRecord.cpp
>===================================================================
>--- Source/JavaScriptCore/runtime/JSModuleRecord.cpp	(revision 204735)
>+++ Source/JavaScriptCore/runtime/JSModuleRecord.cpp	(working copy)
>@@ -1,5 +1,5 @@
> /*
>- * Copyright (C) 2015 Apple Inc. All rights reserved.
>+ * Copyright (C) 2015-2016 Apple Inc. All rights reserved.
>  *
>  * Redistribution and use in source and binary forms, with or without
>  * modification, are permitted provided that the following conditions
>@@ -28,14 +28,11 @@
> 
> #include "Error.h"
> #include "Executable.h"
>-#include "IdentifierInlines.h"
>-#include "JSCJSValueInlines.h"
>-#include "JSCellInlines.h"
>+#include "Interpreter.h"
>+#include "JSCInlines.h"
> #include "JSMap.h"
> #include "JSModuleEnvironment.h"
> #include "JSModuleNamespaceObject.h"
>-#include "SlotVisitorInlines.h"
>-#include "StructureInlines.h"
> 
> namespace JSC {
> 
>Index: Source/JavaScriptCore/runtime/JSObject.cpp
>===================================================================
>--- Source/JavaScriptCore/runtime/JSObject.cpp	(revision 204735)
>+++ Source/JavaScriptCore/runtime/JSObject.cpp	(working copy)
>@@ -55,6 +55,12 @@
> 
> namespace JSC {
> 
>+#if 0
>+#define LOGIT() dataLog(RawPointer(this), ": ", WTF_PRETTY_FUNCTION, "\n");
>+#else
>+#define LOGIT() do { } while(false)
>+#endif
>+
> // We keep track of the size of the last array after it was grown. We use this
> // as a simple heuristic for as the value to grow the next array from size 0.
> // This value is capped by the constant FIRST_VECTOR_GROW defined in
>@@ -88,77 +94,6 @@ static inline void getClassPropertyNames
>     }
> }
> 
>-ALWAYS_INLINE void JSObject::copyButterfly(CopyVisitor& visitor, Butterfly* butterfly, size_t storageSize)
>-{
>-    ASSERT(butterfly);
>-    
>-    Structure* structure = this->structure();
>-    
>-    size_t propertyCapacity = structure->outOfLineCapacity();
>-    size_t preCapacity;
>-    size_t indexingPayloadSizeInBytes;
>-    bool hasIndexingHeader = this->hasIndexingHeader();
>-    if (UNLIKELY(hasIndexingHeader)) {
>-        preCapacity = butterfly->indexingHeader()->preCapacity(structure);
>-        indexingPayloadSizeInBytes = butterfly->indexingHeader()->indexingPayloadSizeInBytes(structure);
>-    } else {
>-        preCapacity = 0;
>-        indexingPayloadSizeInBytes = 0;
>-    }
>-    size_t capacityInBytes = Butterfly::totalSize(preCapacity, propertyCapacity, hasIndexingHeader, indexingPayloadSizeInBytes);
>-    if (visitor.checkIfShouldCopy(butterfly->base(preCapacity, propertyCapacity))) {
>-        Butterfly* newButterfly = Butterfly::createUninitializedDuringCollection(visitor, preCapacity, propertyCapacity, hasIndexingHeader, indexingPayloadSizeInBytes);
>-
>-        // Copy the properties.
>-        PropertyStorage currentTarget = newButterfly->propertyStorage();
>-        PropertyStorage currentSource = butterfly->propertyStorage();
>-        for (size_t count = storageSize; count--;)
>-            (--currentTarget)->setWithoutWriteBarrier((--currentSource)->get());
>-        
>-        if (UNLIKELY(hasIndexingHeader)) {
>-            *newButterfly->indexingHeader() = *butterfly->indexingHeader();
>-            
>-            // Copy the array if appropriate.
>-            
>-            WriteBarrier<Unknown>* currentTarget;
>-            WriteBarrier<Unknown>* currentSource;
>-            size_t count;
>-            
>-            switch (this->indexingType()) {
>-            case ALL_UNDECIDED_INDEXING_TYPES:
>-            case ALL_CONTIGUOUS_INDEXING_TYPES:
>-            case ALL_INT32_INDEXING_TYPES:
>-            case ALL_DOUBLE_INDEXING_TYPES: {
>-                currentTarget = newButterfly->contiguous().data();
>-                currentSource = butterfly->contiguous().data();
>-                RELEASE_ASSERT(newButterfly->publicLength() <= newButterfly->vectorLength());
>-                count = newButterfly->vectorLength();
>-                break;
>-            }
>-                
>-            case ALL_ARRAY_STORAGE_INDEXING_TYPES: {
>-                newButterfly->arrayStorage()->copyHeaderFromDuringGC(*butterfly->arrayStorage());
>-                currentTarget = newButterfly->arrayStorage()->m_vector;
>-                currentSource = butterfly->arrayStorage()->m_vector;
>-                count = newButterfly->arrayStorage()->vectorLength();
>-                break;
>-            }
>-                
>-            default:
>-                currentTarget = 0;
>-                currentSource = 0;
>-                count = 0;
>-                break;
>-            }
>-
>-            memcpy(currentTarget, currentSource, count * sizeof(EncodedJSValue));
>-        }
>-        
>-        m_butterfly.setWithoutBarrier(newButterfly);
>-        visitor.didCopy(butterfly->base(preCapacity, propertyCapacity), capacityInBytes);
>-    } 
>-}
>-
> ALWAYS_INLINE void JSObject::visitButterfly(SlotVisitor& visitor, Butterfly* butterfly, Structure* structure)
> {
>     ASSERT(butterfly);
>@@ -166,22 +101,21 @@ ALWAYS_INLINE void JSObject::visitButter
>     size_t storageSize = structure->outOfLineSize();
>     size_t propertyCapacity = structure->outOfLineCapacity();
>     size_t preCapacity;
>-    size_t indexingPayloadSizeInBytes;
>     bool hasIndexingHeader = this->hasIndexingHeader();
>-    if (UNLIKELY(hasIndexingHeader)) {
>+    if (UNLIKELY(hasIndexingHeader))
>         preCapacity = butterfly->indexingHeader()->preCapacity(structure);
>-        indexingPayloadSizeInBytes = butterfly->indexingHeader()->indexingPayloadSizeInBytes(structure);
>-    } else {
>+    else
>         preCapacity = 0;
>-        indexingPayloadSizeInBytes = 0;
>-    }
>-    size_t capacityInBytes = Butterfly::totalSize(preCapacity, propertyCapacity, hasIndexingHeader, indexingPayloadSizeInBytes);
>+    
>+    HeapCell* base = bitwise_cast<HeapCell*>(butterfly->base(preCapacity, propertyCapacity));
>+    
>+    ASSERT(Heap::heap(base) == visitor.heap());
> 
>+    // Keep the butterfly alive.
>+    visitor.markAuxiliary(base);
>+    
>     // Mark the properties.
>     visitor.appendValuesHidden(butterfly->propertyStorage() - storageSize, storageSize);
>-    visitor.copyLater(
>-        this, ButterflyCopyToken,
>-        butterfly->base(preCapacity, propertyCapacity), capacityInBytes);
>     
>     // Mark the array if appropriate.
>     switch (this->indexingType()) {
>@@ -225,19 +159,6 @@ void JSObject::visitChildren(JSCell* cel
> #endif
> }
> 
>-void JSObject::copyBackingStore(JSCell* cell, CopyVisitor& visitor, CopyToken token)
>-{
>-    JSObject* thisObject = jsCast<JSObject*>(cell);
>-    ASSERT_GC_OBJECT_INHERITS(thisObject, info());
>-
>-    if (token != ButterflyCopyToken)
>-        return;
>-    
>-    Butterfly* butterfly = thisObject->m_butterfly.get();
>-    if (butterfly)
>-        thisObject->copyButterfly(visitor, butterfly, thisObject->structure()->outOfLineSize());
>-}
>-
> void JSObject::heapSnapshot(JSCell* cell, HeapSnapshotBuilder& builder)
> {
>     JSObject* thisObject = jsCast<JSObject*>(cell);
>@@ -776,20 +697,23 @@ void JSObject::notifyPresenceOfIndexedAc
>     if (!vm.prototypeMap.isPrototype(this))
>         return;
>     
>-    globalObject()->haveABadTime(vm);
>+    globalObject(vm)->haveABadTime(vm);
> }
> 
>-Butterfly* JSObject::createInitialIndexedStorage(VM& vm, unsigned length, size_t elementSize)
>+Butterfly* JSObject::createInitialIndexedStorage(VM& vm, unsigned length)
> {
>+    LOGIT();
>     ASSERT(length < MAX_ARRAY_INDEX);
>     IndexingType oldType = indexingType();
>     ASSERT_UNUSED(oldType, !hasIndexedProperties(oldType));
>     ASSERT(!structure()->needsSlowPutIndexing());
>     ASSERT(!indexingShouldBeSparse());
>-    unsigned vectorLength = std::max(length, BASE_VECTOR_LEN);
>+    Structure* structure = this->structure(vm);
>+    unsigned propertyCapacity = structure->outOfLineCapacity();
>+    unsigned vectorLength = Butterfly::optimalContiguousVectorLength(propertyCapacity, length);
>     Butterfly* newButterfly = Butterfly::createOrGrowArrayRight(
>-        m_butterfly.get(), vm, this, structure(), structure()->outOfLineCapacity(), false, 0,
>-        elementSize * vectorLength);
>+        m_butterfly.get(), vm, this, structure, propertyCapacity, false, 0,
>+        sizeof(EncodedJSValue) * vectorLength);
>     newButterfly->setPublicLength(length);
>     newButterfly->setVectorLength(vectorLength);
>     return newButterfly;
>@@ -798,7 +722,7 @@ Butterfly* JSObject::createInitialIndexe
> Butterfly* JSObject::createInitialUndecided(VM& vm, unsigned length)
> {
>     DeferGC deferGC(vm.heap);
>-    Butterfly* newButterfly = createInitialIndexedStorage(vm, length, sizeof(EncodedJSValue));
>+    Butterfly* newButterfly = createInitialIndexedStorage(vm, length);
>     Structure* newStructure = Structure::nonPropertyTransition(vm, structure(vm), NonPropertyTransition::AllocateUndecided);
>     setStructureAndButterfly(vm, newStructure, newButterfly);
>     return newButterfly;
>@@ -807,7 +731,9 @@ Butterfly* JSObject::createInitialUndeci
> ContiguousJSValues JSObject::createInitialInt32(VM& vm, unsigned length)
> {
>     DeferGC deferGC(vm.heap);
>-    Butterfly* newButterfly = createInitialIndexedStorage(vm, length, sizeof(EncodedJSValue));
>+    Butterfly* newButterfly = createInitialIndexedStorage(vm, length);
>+    for (unsigned i = newButterfly->vectorLength(); i--;)
>+        newButterfly->contiguousInt32()[i].setWithoutWriteBarrier(JSValue());
>     Structure* newStructure = Structure::nonPropertyTransition(vm, structure(vm), NonPropertyTransition::AllocateInt32);
>     setStructureAndButterfly(vm, newStructure, newButterfly);
>     return newButterfly->contiguousInt32();
>@@ -816,7 +742,7 @@ ContiguousJSValues JSObject::createIniti
> ContiguousDoubles JSObject::createInitialDouble(VM& vm, unsigned length)
> {
>     DeferGC deferGC(vm.heap);
>-    Butterfly* newButterfly = createInitialIndexedStorage(vm, length, sizeof(double));
>+    Butterfly* newButterfly = createInitialIndexedStorage(vm, length);
>     for (unsigned i = newButterfly->vectorLength(); i--;)
>         newButterfly->contiguousDouble()[i] = PNaN;
>     Structure* newStructure = Structure::nonPropertyTransition(vm, structure(vm), NonPropertyTransition::AllocateDouble);
>@@ -827,7 +753,9 @@ ContiguousDoubles JSObject::createInitia
> ContiguousJSValues JSObject::createInitialContiguous(VM& vm, unsigned length)
> {
>     DeferGC deferGC(vm.heap);
>-    Butterfly* newButterfly = createInitialIndexedStorage(vm, length, sizeof(EncodedJSValue));
>+    Butterfly* newButterfly = createInitialIndexedStorage(vm, length);
>+    for (unsigned i = newButterfly->vectorLength(); i--;)
>+        newButterfly->contiguous()[i].setWithoutWriteBarrier(JSValue());
>     Structure* newStructure = Structure::nonPropertyTransition(vm, structure(vm), NonPropertyTransition::AllocateContiguous);
>     setStructureAndButterfly(vm, newStructure, newButterfly);
>     return newButterfly->contiguous();
>@@ -835,6 +763,7 @@ ContiguousJSValues JSObject::createIniti
> 
> ArrayStorage* JSObject::createArrayStorage(VM& vm, unsigned length, unsigned vectorLength)
> {
>+    LOGIT();
>     DeferGC deferGC(vm.heap);
>     Structure* structure = this->structure(vm);
>     IndexingType oldType = indexingType();
>@@ -850,6 +779,8 @@ ArrayStorage* JSObject::createArrayStora
>     result->m_sparseMap.clear();
>     result->m_numValuesInVector = 0;
>     result->m_indexBias = 0;
>+    for (size_t i = vectorLength; i--;)
>+        result->m_vector[i].setWithoutWriteBarrier(JSValue());
>     Structure* newStructure = Structure::nonPropertyTransition(vm, structure, structure->suggestedArrayStorageTransition());
>     setStructureAndButterfly(vm, newStructure, newButterfly);
>     return result;
>@@ -857,12 +788,18 @@ ArrayStorage* JSObject::createArrayStora
> 
> ArrayStorage* JSObject::createInitialArrayStorage(VM& vm)
> {
>-    return createArrayStorage(vm, 0, BASE_VECTOR_LEN);
>+    return createArrayStorage(
>+        vm, 0, ArrayStorage::optimalVectorLength(0, structure(vm)->outOfLineCapacity(), 0));
> }
> 
> ContiguousJSValues JSObject::convertUndecidedToInt32(VM& vm)
> {
>     ASSERT(hasUndecided(indexingType()));
>+
>+    Butterfly* butterfly = m_butterfly.get();
>+    for (unsigned i = butterfly->vectorLength(); i--;)
>+        butterfly->contiguousInt32()[i].setWithoutWriteBarrier(JSValue());
>+
>     setStructure(vm, Structure::nonPropertyTransition(vm, structure(vm), NonPropertyTransition::AllocateInt32));
>     return m_butterfly.get()->contiguousInt32();
> }
>@@ -882,6 +819,11 @@ ContiguousDoubles JSObject::convertUndec
> ContiguousJSValues JSObject::convertUndecidedToContiguous(VM& vm)
> {
>     ASSERT(hasUndecided(indexingType()));
>+
>+    Butterfly* butterfly = m_butterfly.get();
>+    for (unsigned i = butterfly->vectorLength(); i--;)
>+        butterfly->contiguous()[i].setWithoutWriteBarrier(JSValue());
>+
>     setStructure(vm, Structure::nonPropertyTransition(vm, structure(vm), NonPropertyTransition::AllocateContiguous));
>     return m_butterfly.get()->contiguous();
> }
>@@ -918,7 +860,9 @@ ArrayStorage* JSObject::convertUndecided
> 
>     unsigned vectorLength = m_butterfly.get()->vectorLength();
>     ArrayStorage* storage = constructConvertedArrayStorageWithoutCopyingElements(vm, vectorLength);
>-    // No need to copy elements.
>+    
>+    for (unsigned i = vectorLength; i--;)
>+        storage->m_vector[i].setWithoutWriteBarrier(JSValue());
>     
>     Structure* newStructure = Structure::nonPropertyTransition(vm, structure(vm), transition);
>     setStructureAndButterfly(vm, newStructure, storage->butterfly());
>@@ -939,11 +883,12 @@ ContiguousDoubles JSObject::convertInt32
>         WriteBarrier<Unknown>* current = &butterfly->contiguousInt32()[i];
>         double* currentAsDouble = bitwise_cast<double*>(current);
>         JSValue v = current->get();
>-        if (!v) {
>+        // NOTE: Since this may be used during initialization, v could be garbage. If it's garbage,
>+        // that means it will be overwritten later.
>+        if (!v.isInt32()) {
>             *currentAsDouble = PNaN;
>             continue;
>         }
>-        ASSERT(v.isInt32());
>         *currentAsDouble = v.asInt32();
>     }
>     
>@@ -967,13 +912,11 @@ ArrayStorage* JSObject::convertInt32ToAr
>     unsigned vectorLength = m_butterfly.get()->vectorLength();
>     ArrayStorage* newStorage = constructConvertedArrayStorageWithoutCopyingElements(vm, vectorLength);
>     Butterfly* butterfly = m_butterfly.get();
>-    for (unsigned i = 0; i < butterfly->publicLength(); i++) {
>+    for (unsigned i = 0; i < vectorLength; i++) {
>         JSValue v = butterfly->contiguous()[i].get();
>-        if (v) {
>-            newStorage->m_vector[i].setWithoutWriteBarrier(v);
>+        newStorage->m_vector[i].setWithoutWriteBarrier(v);
>+        if (v)
>             newStorage->m_numValuesInVector++;
>-        } else
>-            ASSERT(newStorage->m_vector[i].get().isEmpty());
>     }
>     
>     Structure* newStructure = Structure::nonPropertyTransition(vm, structure(vm), transition);
>@@ -1015,13 +958,11 @@ ArrayStorage* JSObject::convertDoubleToA
>     unsigned vectorLength = m_butterfly.get()->vectorLength();
>     ArrayStorage* newStorage = constructConvertedArrayStorageWithoutCopyingElements(vm, vectorLength);
>     Butterfly* butterfly = m_butterfly.get();
>-    for (unsigned i = 0; i < butterfly->publicLength(); i++) {
>+    for (unsigned i = 0; i < vectorLength; i++) {
>         double value = butterfly->contiguousDouble()[i];
>-        if (value == value) {
>-            newStorage->m_vector[i].setWithoutWriteBarrier(JSValue(JSValue::EncodeAsDouble, value));
>+        newStorage->m_vector[i].setWithoutWriteBarrier(JSValue(JSValue::EncodeAsDouble, value));
>+        if (value == value)
>             newStorage->m_numValuesInVector++;
>-        } else
>-            ASSERT(newStorage->m_vector[i].get().isEmpty());
>     }
>     
>     Structure* newStructure = Structure::nonPropertyTransition(vm, structure(vm), transition);
>@@ -1042,13 +983,11 @@ ArrayStorage* JSObject::convertContiguou
>     unsigned vectorLength = m_butterfly.get()->vectorLength();
>     ArrayStorage* newStorage = constructConvertedArrayStorageWithoutCopyingElements(vm, vectorLength);
>     Butterfly* butterfly = m_butterfly.get();
>-    for (unsigned i = 0; i < butterfly->publicLength(); i++) {
>+    for (unsigned i = 0; i < vectorLength; i++) {
>         JSValue v = butterfly->contiguous()[i].get();
>-        if (v) {
>-            newStorage->m_vector[i].setWithoutWriteBarrier(v);
>+        newStorage->m_vector[i].setWithoutWriteBarrier(v);
>+        if (v)
>             newStorage->m_numValuesInVector++;
>-        } else
>-            ASSERT(newStorage->m_vector[i].get().isEmpty());
>     }
>     
>     Structure* newStructure = Structure::nonPropertyTransition(vm, structure(vm), transition);
>@@ -2382,7 +2321,7 @@ bool JSObject::putByIndexBeyondVectorLen
>         }
>         if (structure(vm)->needsSlowPutIndexing()) {
>             // Convert the indexing type to the SlowPutArrayStorage and retry.
>-            createArrayStorage(vm, i + 1, getNewVectorLength(0, 0, i + 1));
>+            createArrayStorage(vm, i + 1, getNewVectorLength(0, 0, 0, i + 1));
>             return putByIndex(this, exec, i, value, shouldThrow);
>         }
>         
>@@ -2523,7 +2462,7 @@ bool JSObject::putDirectIndexBeyondVecto
>                 exec, i, value, attributes, mode, createArrayStorage(vm, 0, 0));
>         }
>         if (structure(vm)->needsSlowPutIndexing()) {
>-            ArrayStorage* storage = createArrayStorage(vm, i + 1, getNewVectorLength(0, 0, i + 1));
>+            ArrayStorage* storage = createArrayStorage(vm, i + 1, getNewVectorLength(0, 0, 0, i + 1));
>             storage->m_vector[i].set(vm, this, value);
>             storage->m_numValuesInVector++;
>             return true;
>@@ -2642,7 +2581,8 @@ void JSObject::putDirectNativeFunctionWi
>     putDirectWithoutTransition(vm, propertyName, function, attributes);
> }
> 
>-ALWAYS_INLINE unsigned JSObject::getNewVectorLength(unsigned currentVectorLength, unsigned currentLength, unsigned desiredLength)
>+// NOTE: This method is for ArrayStorage vectors.
>+ALWAYS_INLINE unsigned JSObject::getNewVectorLength(unsigned indexBias, unsigned currentVectorLength, unsigned currentLength, unsigned desiredLength)
> {
>     ASSERT(desiredLength <= MAX_STORAGE_VECTOR_LENGTH);
> 
>@@ -2659,25 +2599,27 @@ ALWAYS_INLINE unsigned JSObject::getNewV
> 
>     ASSERT(increasedLength >= desiredLength);
> 
>-    lastArraySize = std::min(increasedLength, FIRST_VECTOR_GROW);
>+    lastArraySize = std::min(increasedLength, FIRST_ARRAY_STORAGE_VECTOR_GROW);
> 
>-    return std::min(increasedLength, MAX_STORAGE_VECTOR_LENGTH);
>+    return ArrayStorage::optimalVectorLength(
>+        indexBias, structure()->outOfLineCapacity(),
>+        std::min(increasedLength, MAX_STORAGE_VECTOR_LENGTH));
> }
> 
> ALWAYS_INLINE unsigned JSObject::getNewVectorLength(unsigned desiredLength)
> {
>-    unsigned vectorLength;
>-    unsigned length;
>+    unsigned indexBias = 0;
>+    unsigned vectorLength = 0;
>+    unsigned length = 0;
>     
>     if (hasIndexedProperties(indexingType())) {
>+        if (ArrayStorage* storage = arrayStorageOrNull())
>+            indexBias = storage->m_indexBias;
>         vectorLength = m_butterfly.get()->vectorLength();
>         length = m_butterfly.get()->publicLength();
>-    } else {
>-        vectorLength = 0;
>-        length = 0;
>     }
> 
>-    return getNewVectorLength(vectorLength, length, desiredLength);
>+    return getNewVectorLength(indexBias, vectorLength, length, desiredLength);
> }
> 
> template<IndexingType indexingShape>
>@@ -2730,19 +2672,29 @@ unsigned JSObject::countElements()
> 
> bool JSObject::increaseVectorLength(VM& vm, unsigned newLength)
> {
>+    LOGIT();
>+    ArrayStorage* storage = arrayStorage();
>+    
>+    unsigned vectorLength = storage->vectorLength();
>+    unsigned availableVectorLength = storage->availableVectorLength(structure(vm), vectorLength); 
>+    if (availableVectorLength >= newLength) {
>+        // The cell was already big enough for the desired length!
>+        for (unsigned i = vectorLength; i < availableVectorLength; ++i)
>+            storage->m_vector[i].clear();
>+        storage->setVectorLength(availableVectorLength);
>+        return true;
>+    }
>+    
>     // This function leaves the array in an internally inconsistent state, because it does not move any values from sparse value map
>     // to the vector. Callers have to account for that, because they can do it more efficiently.
>     if (newLength > MAX_STORAGE_VECTOR_LENGTH)
>         return false;
> 
>-    ArrayStorage* storage = arrayStorage();
>-    
>     if (newLength >= MIN_SPARSE_ARRAY_INDEX
>         && !isDenseEnoughForVector(newLength, storage->m_numValuesInVector))
>         return false;
> 
>     unsigned indexBias = storage->m_indexBias;
>-    unsigned vectorLength = storage->vectorLength();
>     ASSERT(newLength > vectorLength);
>     unsigned newVectorLength = getNewVectorLength(newLength);
> 
>@@ -2755,6 +2707,8 @@ bool JSObject::increaseVectorLength(VM& 
>             ArrayStorage::sizeFor(vectorLength), ArrayStorage::sizeFor(newVectorLength));
>         if (!newButterfly)
>             return false;
>+        for (unsigned i = vectorLength; i < newVectorLength; ++i)
>+            newButterfly->arrayStorage()->m_vector[i].clear();
>         newButterfly->arrayStorage()->setVectorLength(newVectorLength);
>         setButterflyWithoutChangingStructure(vm, newButterfly);
>         return true;
>@@ -2769,6 +2723,8 @@ bool JSObject::increaseVectorLength(VM& 
>         newIndexBias, true, ArrayStorage::sizeFor(newVectorLength));
>     if (!newButterfly)
>         return false;
>+    for (unsigned i = vectorLength; i < newVectorLength; ++i)
>+        newButterfly->arrayStorage()->m_vector[i].clear();
>     newButterfly->arrayStorage()->setVectorLength(newVectorLength);
>     newButterfly->arrayStorage()->m_indexBias = newIndexBias;
>     setButterflyWithoutChangingStructure(vm, newButterfly);
>@@ -2777,31 +2733,48 @@ bool JSObject::increaseVectorLength(VM& 
> 
> bool JSObject::ensureLengthSlow(VM& vm, unsigned length)
> {
>+    LOGIT();
>     Butterfly* butterfly = m_butterfly.get();
>     
>     ASSERT(length < MAX_ARRAY_INDEX);
>     ASSERT(hasContiguous(indexingType()) || hasInt32(indexingType()) || hasDouble(indexingType()) || hasUndecided(indexingType()));
>     ASSERT(length > butterfly->vectorLength());
>     
>-    unsigned newVectorLength = std::min(
>-        length << 1,
>-        MAX_STORAGE_VECTOR_LENGTH);
>     unsigned oldVectorLength = butterfly->vectorLength();
>-    DeferGC deferGC(vm.heap);
>-    butterfly = butterfly->growArrayRight(
>-        vm, this, structure(), structure()->outOfLineCapacity(), true,
>-        oldVectorLength * sizeof(EncodedJSValue),
>-        newVectorLength * sizeof(EncodedJSValue));
>-    if (!butterfly)
>-        return false;
>-    m_butterfly.set(vm, this, butterfly);
>+    unsigned newVectorLength;
>+    
>+    Structure* structure = this->structure(vm);
>+    unsigned propertyCapacity = structure->outOfLineCapacity();
>+    
>+    unsigned availableOldLength =
>+        Butterfly::availableContiguousVectorLength(propertyCapacity, oldVectorLength);
>+    if (availableOldLength >= length) {
>+        // This is the case where someone else selected a vector length that caused internal
>+        // fragmentation. If we did our jobs right, this would never happen. But I bet we will mess
>+        // this up, so this defense should stay.
>+        newVectorLength = availableOldLength;
>+    } else {
>+        newVectorLength = Butterfly::optimalContiguousVectorLength(
>+            propertyCapacity, std::min(length << 1, MAX_STORAGE_VECTOR_LENGTH));
>+        butterfly = butterfly->growArrayRight(
>+            vm, this, structure, propertyCapacity, true,
>+            oldVectorLength * sizeof(EncodedJSValue),
>+            newVectorLength * sizeof(EncodedJSValue));
>+        if (!butterfly)
>+            return false;
>+        m_butterfly.set(vm, this, butterfly);
>+    }
> 
>     butterfly->setVectorLength(newVectorLength);
> 
>     if (hasDouble(indexingType())) {
>         for (unsigned i = oldVectorLength; i < newVectorLength; ++i)
>-            butterfly->contiguousDouble().data()[i] = PNaN;
>+            butterfly->contiguousDouble()[i] = PNaN;
>+    } else {
>+        for (unsigned i = oldVectorLength; i < newVectorLength; ++i)
>+            butterfly->contiguous()[i].clear();
>     }
>+
>     return true;
> }
> 
>@@ -2822,6 +2795,7 @@ void JSObject::reallocateAndShrinkButter
> 
> Butterfly* JSObject::growOutOfLineStorage(VM& vm, size_t oldSize, size_t newSize)
> {
>+    LOGIT();
>     ASSERT(newSize > oldSize);
> 
>     // It's important that this function not rely on structure(), for the property
>Index: Source/JavaScriptCore/runtime/JSObject.h
>===================================================================
>--- Source/JavaScriptCore/runtime/JSObject.h	(revision 204735)
>+++ Source/JavaScriptCore/runtime/JSObject.h	(working copy)
>@@ -26,15 +26,14 @@
> #include "ArgList.h"
> #include "ArrayConventions.h"
> #include "ArrayStorage.h"
>+#include "AuxiliaryBarrier.h"
> #include "Butterfly.h"
> #include "CallFrame.h"
> #include "ClassInfo.h"
> #include "CommonIdentifiers.h"
>-#include "CopyBarrier.h"
> #include "CustomGetterSetter.h"
> #include "DeferGC.h"
> #include "Heap.h"
>-#include "HeapInlines.h"
> #include "IndexingHeaderInlines.h"
> #include "JSCell.h"
> #include "PropertySlot.h"
>@@ -103,7 +102,6 @@ public:
> 
>     JS_EXPORT_PRIVATE static size_t estimatedSize(JSCell*);
>     JS_EXPORT_PRIVATE static void visitChildren(JSCell*, SlotVisitor&);
>-    JS_EXPORT_PRIVATE static void copyBackingStore(JSCell*, CopyVisitor&, CopyToken);
>     JS_EXPORT_PRIVATE static void heapSnapshot(JSCell*, HeapSnapshotBuilder&);
> 
>     JS_EXPORT_PRIVATE static String className(const JSObject*);
>@@ -414,6 +412,8 @@ public:
>         initializeIndex(vm, i, v, indexingType());
>     }
> 
>+    // NOTE: Clients of this method may call it more than once for any index, and this is supposed
>+    // to work.
>     void initializeIndex(VM& vm, unsigned i, JSValue v, IndexingType indexingType)
>     {
>         Butterfly* butterfly = m_butterfly.get();
>@@ -686,8 +686,6 @@ public:
>         
>     void setStructure(VM&, Structure*);
>     void setStructureAndButterfly(VM&, Structure*, Butterfly*);
>-    void setStructureAndReallocateStorageIfNecessary(VM&, unsigned oldCapacity, Structure*);
>-    void setStructureAndReallocateStorageIfNecessary(VM&, Structure*);
> 
>     JS_EXPORT_PRIVATE void convertToDictionary(VM&);
> 
>@@ -704,6 +702,13 @@ public:
>         return structure()->globalObject();
>     }
>         
>+    JSGlobalObject* globalObject(VM& vm) const
>+    {
>+        ASSERT(structure(vm)->globalObject());
>+        ASSERT(!isGlobalObject() || ((JSObject*)structure()->globalObject()) == this);
>+        return structure(vm)->globalObject();
>+    }
>+        
>     void switchToSlowPutArrayStorage(VM&);
>         
>     // The receiver is the prototype in this case. The following:
>@@ -797,7 +802,6 @@ protected:
>     JSObject(VM&, Structure*, Butterfly* = 0);
>         
>     void visitButterfly(SlotVisitor&, Butterfly*, Structure*);
>-    void copyButterfly(CopyVisitor&, Butterfly*, size_t storageSize);
> 
>     // Call this if you know that the object is in a mode where it has array
>     // storage. This will assert otherwise.
>@@ -908,7 +912,7 @@ private:
>     void isObject();
>     void isString();
>         
>-    Butterfly* createInitialIndexedStorage(VM&, unsigned length, size_t elementSize);
>+    Butterfly* createInitialIndexedStorage(VM&, unsigned length);
>         
>     ArrayStorage* enterDictionaryIndexingModeWhenArrayStorageAlreadyExists(VM&, ArrayStorage*);
>         
>@@ -932,7 +936,7 @@ private:
>     bool putDirectIndexBeyondVectorLengthWithArrayStorage(ExecState*, unsigned propertyName, JSValue, unsigned attributes, PutDirectIndexMode, ArrayStorage*);
>     JS_EXPORT_PRIVATE bool putDirectIndexBeyondVectorLength(ExecState*, unsigned propertyName, JSValue, unsigned attributes, PutDirectIndexMode);
>         
>-    unsigned getNewVectorLength(unsigned currentVectorLength, unsigned currentLength, unsigned desiredLength);
>+    unsigned getNewVectorLength(unsigned indexBias, unsigned currentVectorLength, unsigned currentLength, unsigned desiredLength);
>     unsigned getNewVectorLength(unsigned desiredLength);
> 
>     ArrayStorage* constructConvertedArrayStorageWithoutCopyingElements(VM&, unsigned neededLength);
>@@ -949,7 +953,7 @@ private:
>     JS_EXPORT_PRIVATE ArrayStorage* ensureArrayStorageSlow(VM&);
> 
> protected:
>-    CopyBarrier<Butterfly> m_butterfly;
>+    AuxiliaryBarrier<Butterfly*> m_butterfly;
> #if USE(JSVALUE32_64)
> private:
>     uint32_t m_padding;
>@@ -1487,8 +1491,16 @@ ALWAYS_INLINE bool JSObject::putDirectIn
>     
>     validateOffset(offset);
>     ASSERT(newStructure->isValidOffset(offset));
>-    setStructureAndReallocateStorageIfNecessary(vm, newStructure);
>-
>+    DeferGC deferGC(vm.heap);
>+    size_t oldCapacity = structure->outOfLineCapacity();
>+    size_t newCapacity = newStructure->outOfLineCapacity();
>+    ASSERT(oldCapacity <= newCapacity);
>+    if (oldCapacity == newCapacity)
>+        setStructure(vm, newStructure);
>+    else {
>+        Butterfly* newButterfly = growOutOfLineStorage(vm, oldCapacity, newCapacity);
>+        setStructureAndButterfly(vm, newStructure, newButterfly);
>+    }
>     putDirect(vm, offset, value);
>     slot.setNewProperty(this, offset);
>     if (attributes & ReadOnly)
>@@ -1496,27 +1508,6 @@ ALWAYS_INLINE bool JSObject::putDirectIn
>     return true;
> }
> 
>-inline void JSObject::setStructureAndReallocateStorageIfNecessary(VM& vm, unsigned oldCapacity, Structure* newStructure)
>-{
>-    ASSERT(oldCapacity <= newStructure->outOfLineCapacity());
>-    
>-    if (oldCapacity == newStructure->outOfLineCapacity()) {
>-        setStructure(vm, newStructure);
>-        return;
>-    }
>-
>-    DeferGC deferGC(vm.heap); 
>-    Butterfly* newButterfly = growOutOfLineStorage(
>-        vm, oldCapacity, newStructure->outOfLineCapacity());
>-    setStructureAndButterfly(vm, newStructure, newButterfly);
>-}
>-
>-inline void JSObject::setStructureAndReallocateStorageIfNecessary(VM& vm, Structure* newStructure)
>-{
>-    setStructureAndReallocateStorageIfNecessary(
>-        vm, structure(vm)->outOfLineCapacity(), newStructure);
>-}
>-
> inline bool JSObject::putOwnDataProperty(VM& vm, PropertyName propertyName, JSValue value, PutPropertySlot& slot)
> {
>     ASSERT(value);
>Index: Source/JavaScriptCore/runtime/JSObjectInlines.h
>===================================================================
>--- Source/JavaScriptCore/runtime/JSObjectInlines.h	(revision 204735)
>+++ Source/JavaScriptCore/runtime/JSObjectInlines.h	(working copy)
>@@ -1,7 +1,7 @@
> /*
>  *  Copyright (C) 1999-2001 Harri Porten (porten@kde.org)
>  *  Copyright (C) 2001 Peter Kelly (pmk@post.com)
>- *  Copyright (C) 2003-2006, 2008, 2009, 2012-2015 Apple Inc. All rights reserved.
>+ *  Copyright (C) 2003-2006, 2008, 2009, 2012-2016 Apple Inc. All rights reserved.
>  *  Copyright (C) 2007 Eric Seidel (eric@webkit.org)
>  *
>  *  This library is free software; you can redistribute it and/or
>@@ -24,6 +24,7 @@
> #ifndef JSObjectInlines_h
> #define JSObjectInlines_h
> 
>+#include "AuxiliaryBarrierInlines.h"
> #include "Error.h"
> #include "JSObject.h"
> #include "Lookup.h"
>Index: Source/JavaScriptCore/runtime/JSPromise.cpp
>===================================================================
>--- Source/JavaScriptCore/runtime/JSPromise.cpp	(revision 204735)
>+++ Source/JavaScriptCore/runtime/JSPromise.cpp	(working copy)
>@@ -28,12 +28,9 @@
> 
> #include "BuiltinNames.h"
> #include "Error.h"
>-#include "JSCJSValueInlines.h"
>-#include "JSCellInlines.h"
>+#include "JSCInlines.h"
> #include "JSPromiseConstructor.h"
> #include "Microtask.h"
>-#include "SlotVisitorInlines.h"
>-#include "StructureInlines.h"
> 
> namespace JSC {
> 
>Index: Source/JavaScriptCore/runtime/JSPromiseConstructor.cpp
>===================================================================
>--- Source/JavaScriptCore/runtime/JSPromiseConstructor.cpp	(revision 204735)
>+++ Source/JavaScriptCore/runtime/JSPromiseConstructor.cpp	(working copy)
>@@ -32,14 +32,12 @@
> #include "GetterSetter.h"
> #include "IteratorOperations.h"
> #include "JSCBuiltins.h"
>-#include "JSCJSValueInlines.h"
>-#include "JSCellInlines.h"
>+#include "JSCInlines.h"
> #include "JSFunction.h"
> #include "JSPromise.h"
> #include "JSPromisePrototype.h"
> #include "Lookup.h"
> #include "NumberObject.h"
>-#include "StructureInlines.h"
> 
> namespace JSC {
> 
>Index: Source/JavaScriptCore/runtime/JSPromiseDeferred.cpp
>===================================================================
>--- Source/JavaScriptCore/runtime/JSPromiseDeferred.cpp	(revision 204735)
>+++ Source/JavaScriptCore/runtime/JSPromiseDeferred.cpp	(working copy)
>@@ -29,12 +29,9 @@
> #include "BuiltinNames.h"
> #include "Error.h"
> #include "Exception.h"
>-#include "JSCJSValueInlines.h"
>-#include "JSCellInlines.h"
>+#include "JSCInlines.h"
> #include "JSPromise.h"
> #include "JSPromiseConstructor.h"
>-#include "SlotVisitorInlines.h"
>-#include "StructureInlines.h"
> 
> namespace JSC {
> 
>Index: Source/JavaScriptCore/runtime/JSPromisePrototype.cpp
>===================================================================
>--- Source/JavaScriptCore/runtime/JSPromisePrototype.cpp	(revision 204735)
>+++ Source/JavaScriptCore/runtime/JSPromisePrototype.cpp	(working copy)
>@@ -29,13 +29,11 @@
> #include "BuiltinNames.h"
> #include "Error.h"
> #include "JSCBuiltins.h"
>-#include "JSCJSValueInlines.h"
>-#include "JSCellInlines.h"
>+#include "JSCInlines.h"
> #include "JSFunction.h"
> #include "JSGlobalObject.h"
> #include "JSPromise.h"
> #include "Microtask.h"
>-#include "StructureInlines.h"
> 
> namespace JSC {
> 
>Index: Source/JavaScriptCore/runtime/JSPropertyNameIterator.cpp
>===================================================================
>--- Source/JavaScriptCore/runtime/JSPropertyNameIterator.cpp	(revision 204735)
>+++ Source/JavaScriptCore/runtime/JSPropertyNameIterator.cpp	(working copy)
>@@ -1,5 +1,5 @@
> /*
>- * Copyright (C) 2015 Apple, Inc. All rights reserved.
>+ * Copyright (C) 2015-2016 Apple, Inc. All rights reserved.
>  *
>  * Redistribution and use in source and binary forms, with or without
>  * modification, are permitted provided that the following conditions
>@@ -26,13 +26,9 @@
> #include "config.h"
> #include "JSPropertyNameIterator.h"
> 
>-#include "IdentifierInlines.h"
> #include "IteratorOperations.h"
>-#include "JSCJSValueInlines.h"
>-#include "JSCellInlines.h"
>+#include "JSCInlines.h"
> #include "JSPropertyNameEnumerator.h"
>-#include "SlotVisitorInlines.h"
>-#include "StructureInlines.h"
> 
> namespace JSC {
> 
>Index: Source/JavaScriptCore/runtime/JSScope.cpp
>===================================================================
>--- Source/JavaScriptCore/runtime/JSScope.cpp	(revision 204735)
>+++ Source/JavaScriptCore/runtime/JSScope.cpp	(working copy)
>@@ -217,6 +217,7 @@ static inline bool isUnscopable(ExecStat
> 
> JSObject* JSScope::resolve(ExecState* exec, JSScope* scope, const Identifier& ident)
> {
>+    VM& vm = exec->vm();
>     ScopeChainIterator end = scope->end();
>     ScopeChainIterator it = scope->begin();
>     while (1) {
>@@ -225,7 +226,7 @@ JSObject* JSScope::resolve(ExecState* ex
> 
>         // Global scope.
>         if (++it == end) {
>-            JSScope* globalScopeExtension = scope->globalObject()->globalScopeExtension();
>+            JSScope* globalScopeExtension = scope->globalObject(vm)->globalScopeExtension();
>             if (UNLIKELY(globalScopeExtension)) {
>                 if (object->hasProperty(exec, ident))
>                     return object;
>Index: Source/JavaScriptCore/runtime/JSScope.h
>===================================================================
>--- Source/JavaScriptCore/runtime/JSScope.h	(revision 204735)
>+++ Source/JavaScriptCore/runtime/JSScope.h	(working copy)
>@@ -69,7 +69,7 @@ public:
>     JSScope* next();
> 
>     JSGlobalObject* globalObject();
>-    VM* vm();
>+    JSGlobalObject* globalObject(VM&);
>     JSObject* globalThis();
> 
>     SymbolTable* symbolTable();
>@@ -129,9 +129,9 @@ inline JSGlobalObject* JSScope::globalOb
>     return structure()->globalObject();
> }
> 
>-inline VM* JSScope::vm()
>+inline JSGlobalObject* JSScope::globalObject(VM& vm)
> { 
>-    return MarkedBlock::blockFor(this)->vm();
>+    return structure(vm)->globalObject();
> }
> 
> inline Register& Register::operator=(JSScope* scope)
>Index: Source/JavaScriptCore/runtime/JSSetIterator.cpp
>===================================================================
>--- Source/JavaScriptCore/runtime/JSSetIterator.cpp	(revision 204735)
>+++ Source/JavaScriptCore/runtime/JSSetIterator.cpp	(working copy)
>@@ -1,5 +1,5 @@
> /*
>- * Copyright (C) 2013 Apple, Inc. All rights reserved.
>+ * Copyright (C) 2013, 2016 Apple, Inc. All rights reserved.
>  *
>  * Redistribution and use in source and binary forms, with or without
>  * modification, are permitted provided that the following conditions
>@@ -26,12 +26,9 @@
> #include "config.h"
> #include "JSSetIterator.h"
> 
>-#include "JSCJSValueInlines.h"
>-#include "JSCellInlines.h"
>+#include "JSCInlines.h"
> #include "JSSet.h"
> #include "MapDataInlines.h"
>-#include "SlotVisitorInlines.h"
>-#include "StructureInlines.h"
> 
> namespace JSC {
> 
>Index: Source/JavaScriptCore/runtime/JSStringIterator.cpp
>===================================================================
>--- Source/JavaScriptCore/runtime/JSStringIterator.cpp	(revision 204735)
>+++ Source/JavaScriptCore/runtime/JSStringIterator.cpp	(working copy)
>@@ -28,9 +28,7 @@
> #include "JSStringIterator.h"
> 
> #include "BuiltinNames.h"
>-#include "JSCJSValueInlines.h"
>-#include "JSCellInlines.h"
>-#include "StructureInlines.h"
>+#include "JSCInlines.h"
> 
> namespace JSC {
> 
>Index: Source/JavaScriptCore/runtime/JSTemplateRegistryKey.cpp
>===================================================================
>--- Source/JavaScriptCore/runtime/JSTemplateRegistryKey.cpp	(revision 204735)
>+++ Source/JavaScriptCore/runtime/JSTemplateRegistryKey.cpp	(working copy)
>@@ -1,5 +1,6 @@
> /*
>  * Copyright (C) 2015 Yusuke Suzuki <utatane.tea@gmail.com>.
>+ * Copyright (C) 2016 Apple Inc. All Rights Reserved.
>  *
>  * Redistribution and use in source and binary forms, with or without
>  * modification, are permitted provided that the following conditions
>@@ -26,9 +27,7 @@
> #include "config.h"
> #include "JSTemplateRegistryKey.h"
> 
>-#include "JSCJSValueInlines.h"
>-#include "JSCellInlines.h"
>-#include "StructureInlines.h"
>+#include "JSCInlines.h"
> #include "VM.h"
> 
> namespace JSC {
>Index: Source/JavaScriptCore/runtime/JSTypedArrayViewConstructor.cpp
>===================================================================
>--- Source/JavaScriptCore/runtime/JSTypedArrayViewConstructor.cpp	(revision 204735)
>+++ Source/JavaScriptCore/runtime/JSTypedArrayViewConstructor.cpp	(working copy)
>@@ -30,9 +30,8 @@
> #include "Error.h"
> #include "GetterSetter.h"
> #include "JSCBuiltins.h"
>-#include "JSCellInlines.h"
>+#include "JSCInlines.h"
> #include "JSGenericTypedArrayViewConstructorInlines.h"
>-#include "JSObject.h"
> #include "JSTypedArrayViewPrototype.h"
> #include "JSTypedArrays.h"
> 
>Index: Source/JavaScriptCore/runtime/JSTypedArrayViewPrototype.cpp
>===================================================================
>--- Source/JavaScriptCore/runtime/JSTypedArrayViewPrototype.cpp	(revision 204735)
>+++ Source/JavaScriptCore/runtime/JSTypedArrayViewPrototype.cpp	(working copy)
>@@ -29,7 +29,7 @@
> #include "BuiltinNames.h"
> #include "CallFrame.h"
> #include "GetterSetter.h"
>-#include "JSCellInlines.h"
>+#include "JSCInlines.h"
> #include "JSFunction.h"
> #include "JSGenericTypedArrayViewPrototypeFunctions.h"
> #include "TypedArrayAdaptors.h"
>Index: Source/JavaScriptCore/runtime/JSWeakMap.cpp
>===================================================================
>--- Source/JavaScriptCore/runtime/JSWeakMap.cpp	(revision 204735)
>+++ Source/JavaScriptCore/runtime/JSWeakMap.cpp	(working copy)
>@@ -26,11 +26,8 @@
> #include "config.h"
> #include "JSWeakMap.h"
> 
>-#include "JSCJSValueInlines.h"
>-#include "SlotVisitorInlines.h"
>-#include "StructureInlines.h"
>+#include "JSCInlines.h"
> #include "WeakMapData.h"
>-#include "WriteBarrierInlines.h"
> 
> namespace JSC {
> 
>Index: Source/JavaScriptCore/runtime/JSWeakSet.cpp
>===================================================================
>--- Source/JavaScriptCore/runtime/JSWeakSet.cpp	(revision 204735)
>+++ Source/JavaScriptCore/runtime/JSWeakSet.cpp	(working copy)
>@@ -26,11 +26,8 @@
> #include "config.h"
> #include "JSWeakSet.h"
> 
>-#include "JSCJSValueInlines.h"
>-#include "SlotVisitorInlines.h"
>-#include "StructureInlines.h"
>+#include "JSCInlines.h"
> #include "WeakMapData.h"
>-#include "WriteBarrierInlines.h"
> 
> namespace JSC {
> 
>Index: Source/JavaScriptCore/runtime/MapConstructor.cpp
>===================================================================
>--- Source/JavaScriptCore/runtime/MapConstructor.cpp	(revision 204735)
>+++ Source/JavaScriptCore/runtime/MapConstructor.cpp	(working copy)
>@@ -29,12 +29,10 @@
> #include "Error.h"
> #include "GetterSetter.h"
> #include "IteratorOperations.h"
>-#include "JSCJSValueInlines.h"
>-#include "JSCellInlines.h"
>+#include "JSCInlines.h"
> #include "JSGlobalObject.h"
> #include "JSMap.h"
> #include "MapPrototype.h"
>-#include "StructureInlines.h"
> 
> namespace JSC {
> 
>Index: Source/JavaScriptCore/runtime/MapIteratorPrototype.cpp
>===================================================================
>--- Source/JavaScriptCore/runtime/MapIteratorPrototype.cpp	(revision 204735)
>+++ Source/JavaScriptCore/runtime/MapIteratorPrototype.cpp	(working copy)
>@@ -27,10 +27,8 @@
> #include "MapIteratorPrototype.h"
> 
> #include "IteratorOperations.h"
>-#include "JSCJSValueInlines.h"
>-#include "JSCellInlines.h"
>+#include "JSCInlines.h"
> #include "JSMapIterator.h"
>-#include "StructureInlines.h"
> 
> namespace JSC {
> 
>Index: Source/JavaScriptCore/runtime/MapPrototype.cpp
>===================================================================
>--- Source/JavaScriptCore/runtime/MapPrototype.cpp	(revision 204735)
>+++ Source/JavaScriptCore/runtime/MapPrototype.cpp	(working copy)
>@@ -31,13 +31,11 @@
> #include "ExceptionHelpers.h"
> #include "GetterSetter.h"
> #include "IteratorOperations.h"
>-#include "JSCJSValueInlines.h"
>-#include "JSFunctionInlines.h"
>+#include "JSCInlines.h"
> #include "JSMap.h"
> #include "JSMapIterator.h"
> #include "Lookup.h"
> #include "MapDataInlines.h"
>-#include "StructureInlines.h"
> 
> #include "MapPrototype.lut.h"
> 
>Index: Source/JavaScriptCore/runtime/NativeErrorConstructor.cpp
>===================================================================
>--- Source/JavaScriptCore/runtime/NativeErrorConstructor.cpp	(revision 204735)
>+++ Source/JavaScriptCore/runtime/NativeErrorConstructor.cpp	(working copy)
>@@ -22,6 +22,7 @@
> #include "NativeErrorConstructor.h"
> 
> #include "ErrorInstance.h"
>+#include "Interpreter.h"
> #include "JSFunction.h"
> #include "JSString.h"
> #include "NativeErrorPrototype.h"
>Index: Source/JavaScriptCore/runtime/NativeStdFunctionCell.cpp
>===================================================================
>--- Source/JavaScriptCore/runtime/NativeStdFunctionCell.cpp	(revision 204735)
>+++ Source/JavaScriptCore/runtime/NativeStdFunctionCell.cpp	(working copy)
>@@ -26,10 +26,7 @@
> #include "config.h"
> #include "NativeStdFunctionCell.h"
> 
>-#include "JSCJSValueInlines.h"
>-#include "JSCellInlines.h"
>-#include "JSFunctionInlines.h"
>-#include "SlotVisitorInlines.h"
>+#include "JSCInlines.h"
> 
> namespace JSC {
> 
>Index: Source/JavaScriptCore/runtime/Operations.h
>===================================================================
>--- Source/JavaScriptCore/runtime/Operations.h	(revision 204735)
>+++ Source/JavaScriptCore/runtime/Operations.h	(working copy)
>@@ -195,6 +195,20 @@ ALWAYS_INLINE JSValue jsAdd(CallFrame* c
>     return jsAddSlowCase(callFrame, v1, v2);
> }
> 
>+inline bool scribbleFreeCells()
>+{
>+    return !ASSERT_DISABLED || Options::scribbleFreeCells();
>+}
>+
>+inline void scribble(void* base, size_t size)
>+{
>+    for (size_t i = size / sizeof(EncodedJSValue); i--;) {
>+        // Use a 16-byte aligned value to ensure that it passes the cell check.
>+        static_cast<EncodedJSValue*>(base)[i] = JSValue::encode(
>+            bitwise_cast<JSCell*>(static_cast<intptr_t>(0xbadbeef0)));
>+    }
>+}
>+
> } // namespace JSC
> 
> #endif // Operations_h
>Index: Source/JavaScriptCore/runtime/Options.h
>===================================================================
>--- Source/JavaScriptCore/runtime/Options.h	(revision 204735)
>+++ Source/JavaScriptCore/runtime/Options.h	(working copy)
>@@ -182,6 +182,11 @@ typedef const char* optionString;
>     v(bool, testTheFTL, false, Normal, nullptr) \
>     v(bool, verboseSanitizeStack, false, Normal, nullptr) \
>     v(bool, useGenerationalGC, true, Normal, nullptr) \
>+    v(bool, scribbleFreeCells, false, Normal, nullptr) \
>+    v(double, sizeClassProgression, 1.4, Normal, nullptr) \
>+    v(unsigned, largeAllocationCutoff, 400, Normal, nullptr) \
>+    v(bool, dumpSizeClasses, false, Normal, nullptr) \
>+    v(bool, useBumpAllocator, true, Normal, nullptr) \
>     v(bool, eagerlyUpdateTopCallFrame, false, Normal, nullptr) \
>     \
>     v(bool, useOSREntryToDFG, true, Normal, nullptr) \
>Index: Source/JavaScriptCore/runtime/PropertyTable.cpp
>===================================================================
>--- Source/JavaScriptCore/runtime/PropertyTable.cpp	(revision 204735)
>+++ Source/JavaScriptCore/runtime/PropertyTable.cpp	(working copy)
>@@ -26,10 +26,7 @@
> #include "config.h"
> #include "PropertyMapHashTable.h"
> 
>-#include "JSCJSValueInlines.h"
>-#include "JSCellInlines.h"
>-#include "SlotVisitorInlines.h"
>-#include "StructureInlines.h"
>+#include "JSCInlines.h"
> 
> namespace JSC {
> 
>Index: Source/JavaScriptCore/runtime/ProxyConstructor.cpp
>===================================================================
>--- Source/JavaScriptCore/runtime/ProxyConstructor.cpp	(revision 204735)
>+++ Source/JavaScriptCore/runtime/ProxyConstructor.cpp	(working copy)
>@@ -28,8 +28,7 @@
> 
> #include "Error.h"
> #include "IdentifierInlines.h"
>-#include "JSCJSValueInlines.h"
>-#include "JSCellInlines.h"
>+#include "JSCInlines.h"
> #include "ObjectConstructor.h"
> #include "ObjectPrototype.h"
> #include "ProxyObject.h"
>Index: Source/JavaScriptCore/runtime/ProxyObject.cpp
>===================================================================
>--- Source/JavaScriptCore/runtime/ProxyObject.cpp	(revision 204735)
>+++ Source/JavaScriptCore/runtime/ProxyObject.cpp	(working copy)
>@@ -28,8 +28,7 @@
> 
> #include "Error.h"
> #include "IdentifierInlines.h"
>-#include "JSCJSValueInlines.h"
>-#include "JSCellInlines.h"
>+#include "JSCInlines.h"
> #include "ObjectConstructor.h"
> #include "SlotVisitorInlines.h"
> #include "StructureInlines.h"
>Index: Source/JavaScriptCore/runtime/ProxyRevoke.cpp
>===================================================================
>--- Source/JavaScriptCore/runtime/ProxyRevoke.cpp	(revision 204735)
>+++ Source/JavaScriptCore/runtime/ProxyRevoke.cpp	(working copy)
>@@ -26,10 +26,8 @@
> #include "config.h"
> #include "ProxyRevoke.h"
> 
>-#include "JSCJSValueInlines.h"
>+#include "JSCInlines.h"
> #include "ProxyObject.h"
>-#include "SlotVisitorInlines.h"
>-#include "StructureInlines.h"
> 
> namespace JSC {
> 
>Index: Source/JavaScriptCore/runtime/RegExpMatchesArray.h
>===================================================================
>--- Source/JavaScriptCore/runtime/RegExpMatchesArray.h	(revision 204735)
>+++ Source/JavaScriptCore/runtime/RegExpMatchesArray.h	(working copy)
>@@ -34,17 +34,20 @@ static const PropertyOffset RegExpMatche
> 
> ALWAYS_INLINE JSArray* tryCreateUninitializedRegExpMatchesArray(VM& vm, Structure* structure, unsigned initialLength)
> {
>-    unsigned vectorLength = std::max(BASE_VECTOR_LEN, initialLength);
>+    unsigned vectorLength = initialLength;
>     if (vectorLength > MAX_STORAGE_VECTOR_LENGTH)
>         return 0;
> 
>-    void* temp;
>-    if (!vm.heap.tryAllocateStorage(0, Butterfly::totalSize(0, structure->outOfLineCapacity(), true, vectorLength * sizeof(EncodedJSValue)), &temp))
>-        return 0;
>+    void* temp = vm.heap.tryAllocateAuxiliary(nullptr, Butterfly::totalSize(0, structure->outOfLineCapacity(), true, vectorLength * sizeof(EncodedJSValue)));
>+    if (!temp)
>+        return nullptr;
>     Butterfly* butterfly = Butterfly::fromBase(temp, 0, structure->outOfLineCapacity());
>     butterfly->setVectorLength(vectorLength);
>     butterfly->setPublicLength(initialLength);
>-
>+    
>+    for (unsigned i = initialLength; i < vectorLength; ++i)
>+        butterfly->contiguous()[i].clear();
>+    
>     return JSArray::createWithButterfly(vm, structure, butterfly);
> }
> 
>@@ -67,40 +70,54 @@ ALWAYS_INLINE JSArray* createRegExpMatch
>     // FIXME: This should handle array allocation errors gracefully.
>     // https://bugs.webkit.org/show_bug.cgi?id=155144
>     
>+    auto setProperties = [&] () {
>+        array->putDirect(vm, RegExpMatchesArrayIndexPropertyOffset, jsNumber(result.start));
>+        array->putDirect(vm, RegExpMatchesArrayInputPropertyOffset, input);
>+    };
>+    
>+    unsigned numSubpatterns = regExp->numSubpatterns();
>+    
>     if (UNLIKELY(globalObject->isHavingABadTime())) {
>-        array = JSArray::tryCreateUninitialized(vm, globalObject->regExpMatchesArrayStructure(), regExp->numSubpatterns() + 1);
>+        array = JSArray::tryCreateUninitialized(vm, globalObject->regExpMatchesArrayStructure(), numSubpatterns + 1);
>+        
>+        setProperties();
>+        
>+        array->initializeIndex(vm, 0, jsUndefined());
>+        
>+        for (unsigned i = 1; i <= numSubpatterns; ++i)
>+            array->initializeIndex(vm, i, jsUndefined());
>+        
>+        // Now the object is safe to scan by GC.
>         
>         array->initializeIndex(vm, 0, jsSubstringOfResolved(vm, input, result.start, result.end - result.start));
>         
>-        if (unsigned numSubpatterns = regExp->numSubpatterns()) {
>-            for (unsigned i = 1; i <= numSubpatterns; ++i) {
>-                int start = subpatternResults[2 * i];
>-                if (start >= 0)
>-                    array->initializeIndex(vm, i, JSRopeString::createSubstringOfResolved(vm, input, start, subpatternResults[2 * i + 1] - start));
>-                else
>-                    array->initializeIndex(vm, i, jsUndefined());
>-            }
>+        for (unsigned i = 1; i <= numSubpatterns; ++i) {
>+            int start = subpatternResults[2 * i];
>+            if (start >= 0)
>+                array->initializeIndex(vm, i, JSRopeString::createSubstringOfResolved(vm, input, start, subpatternResults[2 * i + 1] - start));
>         }
>     } else {
>-        array = tryCreateUninitializedRegExpMatchesArray(vm, globalObject->regExpMatchesArrayStructure(), regExp->numSubpatterns() + 1);
>+        array = tryCreateUninitializedRegExpMatchesArray(vm, globalObject->regExpMatchesArrayStructure(), numSubpatterns + 1);
>         RELEASE_ASSERT(array);
>         
>+        setProperties();
>+        
>+        array->initializeIndex(vm, 0, jsUndefined(), ArrayWithContiguous);
>+        
>+        for (unsigned i = 1; i <= numSubpatterns; ++i)
>+            array->initializeIndex(vm, i, jsUndefined(), ArrayWithContiguous);
>+        
>+        // Now the object is safe to scan by GC.
>+
>         array->initializeIndex(vm, 0, jsSubstringOfResolved(vm, input, result.start, result.end - result.start), ArrayWithContiguous);
>         
>-        if (unsigned numSubpatterns = regExp->numSubpatterns()) {
>-            for (unsigned i = 1; i <= numSubpatterns; ++i) {
>-                int start = subpatternResults[2 * i];
>-                if (start >= 0)
>-                    array->initializeIndex(vm, i, JSRopeString::createSubstringOfResolved(vm, input, start, subpatternResults[2 * i + 1] - start), ArrayWithContiguous);
>-                else
>-                    array->initializeIndex(vm, i, jsUndefined(), ArrayWithContiguous);
>-            }
>+        for (unsigned i = 1; i <= numSubpatterns; ++i) {
>+            int start = subpatternResults[2 * i];
>+            if (start >= 0)
>+                array->initializeIndex(vm, i, JSRopeString::createSubstringOfResolved(vm, input, start, subpatternResults[2 * i + 1] - start), ArrayWithContiguous);
>         }
>     }
> 
>-    array->putDirect(vm, RegExpMatchesArrayIndexPropertyOffset, jsNumber(result.start));
>-    array->putDirect(vm, RegExpMatchesArrayInputPropertyOffset, input);
>-
>     return array;
> }
> 
>Index: Source/JavaScriptCore/runtime/RuntimeType.cpp
>===================================================================
>--- Source/JavaScriptCore/runtime/RuntimeType.cpp	(revision 204735)
>+++ Source/JavaScriptCore/runtime/RuntimeType.cpp	(working copy)
>@@ -28,8 +28,7 @@
> #include "config.h"
> #include "RuntimeType.h"
> 
>-#include "JSCJSValue.h"
>-#include "JSCJSValueInlines.h"
>+#include "JSCInlines.h"
> 
> namespace JSC {
> 
>Index: Source/JavaScriptCore/runtime/SamplingProfiler.cpp
>===================================================================
>--- Source/JavaScriptCore/runtime/SamplingProfiler.cpp	(revision 204735)
>+++ Source/JavaScriptCore/runtime/SamplingProfiler.cpp	(working copy)
>@@ -33,6 +33,7 @@
> #include "Executable.h"
> #include "HeapInlines.h"
> #include "HeapIterationScope.h"
>+#include "HeapUtil.h"
> #include "InlineCallFrame.h"
> #include "Interpreter.h"
> #include "JSCJSValueInlines.h"
>@@ -356,7 +357,6 @@ void SamplingProfiler::processUnverified
>     RELEASE_ASSERT(m_lock.isLocked());
> 
>     TinyBloomFilter filter = m_vm.heap.objectSpace().blocks().filter();
>-    MarkedBlockSet& markedBlockSet = m_vm.heap.objectSpace().blocks();
> 
>     for (UnprocessedStackTrace& unprocessedStackTrace : m_unprocessedStackTraces) {
>         m_stackTraces.append(StackTrace());
>@@ -390,7 +390,7 @@ void SamplingProfiler::processUnverified
>             JSValue callee = JSValue::decode(encodedCallee);
>             StackFrame& stackFrame = stackTrace.frames.last();
>             bool alreadyHasExecutable = !!stackFrame.executable;
>-            if (!Heap::isValueGCObject(filter, markedBlockSet, callee)) {
>+            if (!HeapUtil::isValueGCObject(m_vm.heap, filter, callee)) {
>                 if (!alreadyHasExecutable)
>                     stackFrame.frameType = FrameType::Unknown;
>                 return;
>@@ -435,7 +435,7 @@ void SamplingProfiler::processUnverified
>                 return;
>             }
> 
>-            RELEASE_ASSERT(Heap::isPointerGCObject(filter, markedBlockSet, executable));
>+            RELEASE_ASSERT(HeapUtil::isPointerGCObjectJSCell(m_vm.heap, filter, executable));
>             stackFrame.frameType = FrameType::Executable;
>             stackFrame.executable = executable;
>             m_liveCellPointers.add(executable);
>Index: Source/JavaScriptCore/runtime/SetConstructor.cpp
>===================================================================
>--- Source/JavaScriptCore/runtime/SetConstructor.cpp	(revision 204735)
>+++ Source/JavaScriptCore/runtime/SetConstructor.cpp	(working copy)
>@@ -29,13 +29,11 @@
> #include "Error.h"
> #include "GetterSetter.h"
> #include "IteratorOperations.h"
>-#include "JSCJSValueInlines.h"
>-#include "JSCellInlines.h"
>+#include "JSCInlines.h"
> #include "JSGlobalObject.h"
> #include "JSSet.h"
> #include "MapData.h"
> #include "SetPrototype.h"
>-#include "StructureInlines.h"
> 
> namespace JSC {
> 
>Index: Source/JavaScriptCore/runtime/SetIteratorPrototype.cpp
>===================================================================
>--- Source/JavaScriptCore/runtime/SetIteratorPrototype.cpp	(revision 204735)
>+++ Source/JavaScriptCore/runtime/SetIteratorPrototype.cpp	(working copy)
>@@ -27,10 +27,8 @@
> #include "SetIteratorPrototype.h"
> 
> #include "IteratorOperations.h"
>-#include "JSCJSValueInlines.h"
>-#include "JSCellInlines.h"
>+#include "JSCInlines.h"
> #include "JSSetIterator.h"
>-#include "StructureInlines.h"
> 
> namespace JSC {
> 
>Index: Source/JavaScriptCore/runtime/SetPrototype.cpp
>===================================================================
>--- Source/JavaScriptCore/runtime/SetPrototype.cpp	(revision 204735)
>+++ Source/JavaScriptCore/runtime/SetPrototype.cpp	(working copy)
>@@ -31,13 +31,11 @@
> #include "ExceptionHelpers.h"
> #include "GetterSetter.h"
> #include "IteratorOperations.h"
>-#include "JSCJSValueInlines.h"
>-#include "JSFunctionInlines.h"
>+#include "JSCInlines.h"
> #include "JSSet.h"
> #include "JSSetIterator.h"
> #include "Lookup.h"
> #include "MapDataInlines.h"
>-#include "StructureInlines.h"
> 
> #include "SetPrototype.lut.h"
> 
>Index: Source/JavaScriptCore/runtime/StackFrame.cpp
>===================================================================
>--- Source/JavaScriptCore/runtime/StackFrame.cpp	(nonexistent)
>+++ Source/JavaScriptCore/runtime/StackFrame.cpp	(working copy)
>@@ -0,0 +1,119 @@
>+/*
>+ * Copyright (C) 2016 Apple Inc. All rights reserved.
>+ *
>+ * Redistribution and use in source and binary forms, with or without
>+ * modification, are permitted provided that the following conditions
>+ * are met:
>+ * 1. Redistributions of source code must retain the above copyright
>+ *    notice, this list of conditions and the following disclaimer.
>+ * 2. Redistributions in binary form must reproduce the above copyright
>+ *    notice, this list of conditions and the following disclaimer in the
>+ *    documentation and/or other materials provided with the distribution.
>+ *
>+ * THIS SOFTWARE IS PROVIDED BY APPLE INC. ``AS IS'' AND ANY
>+ * EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
>+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
>+ * PURPOSE ARE DISCLAIMED.  IN NO EVENT SHALL APPLE INC. OR
>+ * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
>+ * EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
>+ * PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
>+ * PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY
>+ * OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
>+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
>+ * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
>+ */
>+
>+#include "config.h"
>+#include "StackFrame.h"
>+
>+#include "CodeBlock.h"
>+#include "DebuggerPrimitives.h"
>+#include "JSCInlines.h"
>+#include <wtf/text/StringBuilder.h>
>+
>+namespace JSC {
>+
>+intptr_t StackFrame::sourceID() const
>+{
>+    if (!codeBlock)
>+        return noSourceID;
>+    return codeBlock->ownerScriptExecutable()->sourceID();
>+}
>+
>+String StackFrame::sourceURL() const
>+{
>+    if (!codeBlock)
>+        return ASCIILiteral("[native code]");
>+
>+    String sourceURL = codeBlock->ownerScriptExecutable()->sourceURL();
>+    if (!sourceURL.isNull())
>+        return sourceURL;
>+    return emptyString();
>+}
>+
>+String StackFrame::functionName(VM& vm) const
>+{
>+    if (codeBlock) {
>+        switch (codeBlock->codeType()) {
>+        case EvalCode:
>+            return ASCIILiteral("eval code");
>+        case ModuleCode:
>+            return ASCIILiteral("module code");
>+        case FunctionCode:
>+            break;
>+        case GlobalCode:
>+            return ASCIILiteral("global code");
>+        default:
>+            ASSERT_NOT_REACHED();
>+        }
>+    }
>+    String name;
>+    if (callee)
>+        name = getCalculatedDisplayName(vm, callee.get()).impl();
>+    return name.isNull() ? emptyString() : name;
>+}
>+
>+void StackFrame::computeLineAndColumn(unsigned& line, unsigned& column) const
>+{
>+    if (!codeBlock) {
>+        line = 0;
>+        column = 0;
>+        return;
>+    }
>+
>+    int divot = 0;
>+    int unusedStartOffset = 0;
>+    int unusedEndOffset = 0;
>+    codeBlock->expressionRangeForBytecodeOffset(bytecodeOffset, divot, unusedStartOffset, unusedEndOffset, line, column);
>+
>+    ScriptExecutable* executable = codeBlock->ownerScriptExecutable();
>+    if (executable->hasOverrideLineNumber())
>+        line = executable->overrideLineNumber();
>+}
>+
>+String StackFrame::toString(VM& vm) const
>+{
>+    StringBuilder traceBuild;
>+    String functionName = this->functionName(vm);
>+    String sourceURL = this->sourceURL();
>+    traceBuild.append(functionName);
>+    if (!sourceURL.isEmpty()) {
>+        if (!functionName.isEmpty())
>+            traceBuild.append('@');
>+        traceBuild.append(sourceURL);
>+        if (codeBlock) {
>+            unsigned line;
>+            unsigned column;
>+            computeLineAndColumn(line, column);
>+
>+            traceBuild.append(':');
>+            traceBuild.appendNumber(line);
>+            traceBuild.append(':');
>+            traceBuild.appendNumber(column);
>+        }
>+    }
>+    return traceBuild.toString().impl();
>+}
>+
>+} // namespace JSC
>+
>Index: Source/JavaScriptCore/runtime/StackFrame.h
>===================================================================
>--- Source/JavaScriptCore/runtime/StackFrame.h	(nonexistent)
>+++ Source/JavaScriptCore/runtime/StackFrame.h	(working copy)
>@@ -0,0 +1,50 @@
>+/*
>+ * Copyright (C) 2016 Apple Inc. All rights reserved.
>+ *
>+ * Redistribution and use in source and binary forms, with or without
>+ * modification, are permitted provided that the following conditions
>+ * are met:
>+ * 1. Redistributions of source code must retain the above copyright
>+ *    notice, this list of conditions and the following disclaimer.
>+ * 2. Redistributions in binary form must reproduce the above copyright
>+ *    notice, this list of conditions and the following disclaimer in the
>+ *    documentation and/or other materials provided with the distribution.
>+ *
>+ * THIS SOFTWARE IS PROVIDED BY APPLE INC. ``AS IS'' AND ANY
>+ * EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
>+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
>+ * PURPOSE ARE DISCLAIMED.  IN NO EVENT SHALL APPLE INC. OR
>+ * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
>+ * EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
>+ * PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
>+ * PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY
>+ * OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
>+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
>+ * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
>+ */
>+
>+#pragma once
>+
>+#include "Strong.h"
>+
>+namespace JSC {
>+
>+class CodeBlock;
>+class JSObject;
>+
>+struct StackFrame {
>+    Strong<JSObject> callee;
>+    Strong<CodeBlock> codeBlock;
>+    unsigned bytecodeOffset;
>+    
>+    bool isNative() const { return !codeBlock; }
>+    
>+    void computeLineAndColumn(unsigned& line, unsigned& column) const;
>+    String functionName(VM&) const;
>+    intptr_t sourceID() const;
>+    String sourceURL() const;
>+    String toString(VM&) const;
>+};
>+
>+} // namespace JSC
>+
>Index: Source/JavaScriptCore/runtime/StringConstructor.cpp
>===================================================================
>--- Source/JavaScriptCore/runtime/StringConstructor.cpp	(revision 204735)
>+++ Source/JavaScriptCore/runtime/StringConstructor.cpp	(working copy)
>@@ -28,6 +28,7 @@
> #include "JSGlobalObject.h"
> #include "JSCInlines.h"
> #include "StringPrototype.h"
>+#include <wtf/text/StringBuilder.h>
> 
> namespace JSC {
> 
>Index: Source/JavaScriptCore/runtime/StringIteratorPrototype.cpp
>===================================================================
>--- Source/JavaScriptCore/runtime/StringIteratorPrototype.cpp	(revision 204735)
>+++ Source/JavaScriptCore/runtime/StringIteratorPrototype.cpp	(working copy)
>@@ -27,12 +27,10 @@
> #include "config.h"
> #include "StringIteratorPrototype.h"
> 
>-#include "JSCJSValueInlines.h"
>-#include "JSCellInlines.h"
>+#include "JSCInlines.h"
> #include "JSGlobalObject.h"
> #include "JSStringIterator.h"
> #include "ObjectConstructor.h"
>-#include "StructureInlines.h"
> 
> #include "StringIteratorPrototype.lut.h"
> 
>Index: Source/JavaScriptCore/runtime/TemplateRegistry.cpp
>===================================================================
>--- Source/JavaScriptCore/runtime/TemplateRegistry.cpp	(revision 204735)
>+++ Source/JavaScriptCore/runtime/TemplateRegistry.cpp	(working copy)
>@@ -26,10 +26,9 @@
> #include "config.h"
> #include "TemplateRegistry.h"
> 
>-#include "JSCJSValueInlines.h"
>+#include "JSCInlines.h"
> #include "JSGlobalObject.h"
> #include "ObjectConstructor.h"
>-#include "StructureInlines.h"
> #include "WeakGCMapInlines.h"
> 
> namespace JSC {
>Index: Source/JavaScriptCore/runtime/TestRunnerUtils.cpp
>===================================================================
>--- Source/JavaScriptCore/runtime/TestRunnerUtils.cpp	(revision 204735)
>+++ Source/JavaScriptCore/runtime/TestRunnerUtils.cpp	(working copy)
>@@ -1,5 +1,5 @@
> /*
>- * Copyright (C) 2013, 2014 Apple Inc. All rights reserved.
>+ * Copyright (C) 2013-2014, 2016 Apple Inc. All rights reserved.
>  *
>  * Redistribution and use in source and binary forms, with or without
>  * modification, are permitted provided that the following conditions
>@@ -27,7 +27,9 @@
> #include "TestRunnerUtils.h"
> 
> #include "CodeBlock.h"
>+#include "HeapStatistics.h"
> #include "JSCInlines.h"
>+#include "LLIntData.h"
> 
> namespace JSC {
> 
>@@ -150,5 +152,14 @@ JSValue optimizeNextInvocation(ExecState
>     return optimizeNextInvocation(exec->uncheckedArgument(0));
> }
> 
>+// This is a hook called at the bitter end of some of our tests.
>+void finalizeStatsAtEndOfTesting()
>+{
>+    if (Options::logHeapStatisticsAtExit())
>+        HeapStatistics::reportSuccess();
>+    if (Options::reportLLIntStats())
>+        LLInt::Data::finalizeStats();
>+}
>+
> } // namespace JSC
> 
>Index: Source/JavaScriptCore/runtime/TestRunnerUtils.h
>===================================================================
>--- Source/JavaScriptCore/runtime/TestRunnerUtils.h	(revision 204735)
>+++ Source/JavaScriptCore/runtime/TestRunnerUtils.h	(working copy)
>@@ -53,6 +53,8 @@ JS_EXPORT_PRIVATE unsigned numberOfExecu
> JS_EXPORT_PRIVATE unsigned numberOfStaticOSRExitFuzzChecks();
> JS_EXPORT_PRIVATE unsigned numberOfOSRExitFuzzChecks();
> 
>+JS_EXPORT_PRIVATE void finalizeStatsAtEndOfTesting();
>+
> } // namespace JSC
> 
> #endif // TestRunnerUtils_h
>Index: Source/JavaScriptCore/runtime/TypeProfilerLog.cpp
>===================================================================
>--- Source/JavaScriptCore/runtime/TypeProfilerLog.cpp	(revision 204735)
>+++ Source/JavaScriptCore/runtime/TypeProfilerLog.cpp	(working copy)
>@@ -29,7 +29,7 @@
> #include "config.h"
> #include "TypeProfilerLog.h"
> 
>-#include "JSCJSValueInlines.h"
>+#include "JSCInlines.h"
> #include "TypeLocation.h"
> #include <wtf/CurrentTime.h>
> 
>Index: Source/JavaScriptCore/runtime/TypeSet.cpp
>===================================================================
>--- Source/JavaScriptCore/runtime/TypeSet.cpp	(revision 204735)
>+++ Source/JavaScriptCore/runtime/TypeSet.cpp	(working copy)
>@@ -27,8 +27,7 @@
> #include "TypeSet.h"
> 
> #include "InspectorProtocolObjects.h"
>-#include "JSCJSValue.h"
>-#include "JSCJSValueInlines.h"
>+#include "JSCInlines.h"
> #include <wtf/text/CString.h>
> #include <wtf/text/WTFString.h>
> #include <wtf/text/StringBuilder.h>
>Index: Source/JavaScriptCore/runtime/VM.cpp
>===================================================================
>--- Source/JavaScriptCore/runtime/VM.cpp	(revision 204735)
>+++ Source/JavaScriptCore/runtime/VM.cpp	(working copy)
>@@ -70,6 +70,7 @@
> #include "JSPropertyNameEnumerator.h"
> #include "JSTemplateRegistryKey.h"
> #include "JSWithScope.h"
>+#include "LLIntData.h"
> #include "Lexer.h"
> #include "Lookup.h"
> #include "MapData.h"
>Index: Source/JavaScriptCore/runtime/VM.h
>===================================================================
>--- Source/JavaScriptCore/runtime/VM.h	(revision 204735)
>+++ Source/JavaScriptCore/runtime/VM.h	(working copy)
>@@ -39,7 +39,6 @@
> #include "JITThunks.h"
> #include "JSCJSValue.h"
> #include "JSLock.h"
>-#include "LLIntData.h"
> #include "MacroAssemblerCodeRef.h"
> #include "Microtask.h"
> #include "NumericStrings.h"
>Index: Source/JavaScriptCore/runtime/VMEntryScope.h
>===================================================================
>--- Source/JavaScriptCore/runtime/VMEntryScope.h	(revision 204735)
>+++ Source/JavaScriptCore/runtime/VMEntryScope.h	(working copy)
>@@ -26,7 +26,6 @@
> #ifndef VMEntryScope_h
> #define VMEntryScope_h
> 
>-#include "Interpreter.h"
> #include <wtf/StackBounds.h>
> #include <wtf/StackStats.h>
> #include <wtf/Vector.h>
>Index: Source/JavaScriptCore/runtime/WeakMapConstructor.cpp
>===================================================================
>--- Source/JavaScriptCore/runtime/WeakMapConstructor.cpp	(revision 204735)
>+++ Source/JavaScriptCore/runtime/WeakMapConstructor.cpp	(working copy)
>@@ -1,5 +1,5 @@
> /*
>- * Copyright (C) 2013 Apple, Inc. All rights reserved.
>+ * Copyright (C) 2013, 2016 Apple, Inc. All rights reserved.
>  *
>  * Redistribution and use in source and binary forms, with or without
>  * modification, are permitted provided that the following conditions
>@@ -28,11 +28,9 @@
> 
> #include "Error.h"
> #include "IteratorOperations.h"
>-#include "JSCJSValueInlines.h"
>-#include "JSCellInlines.h"
>+#include "JSCInlines.h"
> #include "JSGlobalObject.h"
> #include "JSWeakMap.h"
>-#include "StructureInlines.h"
> #include "WeakMapPrototype.h"
> 
> namespace JSC {
>Index: Source/JavaScriptCore/runtime/WeakMapData.cpp
>===================================================================
>--- Source/JavaScriptCore/runtime/WeakMapData.cpp	(revision 204735)
>+++ Source/JavaScriptCore/runtime/WeakMapData.cpp	(working copy)
>@@ -29,8 +29,7 @@
> #include "CopiedAllocator.h"
> #include "CopyVisitorInlines.h"
> #include "ExceptionHelpers.h"
>-#include "JSCJSValueInlines.h"
>-#include "SlotVisitorInlines.h"
>+#include "JSCInlines.h"
> 
> #include <wtf/MathExtras.h>
> 
>Index: Source/JavaScriptCore/runtime/WeakMapPrototype.cpp
>===================================================================
>--- Source/JavaScriptCore/runtime/WeakMapPrototype.cpp	(revision 204735)
>+++ Source/JavaScriptCore/runtime/WeakMapPrototype.cpp	(working copy)
>@@ -26,9 +26,8 @@
> #include "config.h"
> #include "WeakMapPrototype.h"
> 
>-#include "JSCJSValueInlines.h"
>+#include "JSCInlines.h"
> #include "JSWeakMap.h"
>-#include "StructureInlines.h"
> #include "WeakMapData.h"
> 
> namespace JSC {
>Index: Source/JavaScriptCore/runtime/WeakSetConstructor.cpp
>===================================================================
>--- Source/JavaScriptCore/runtime/WeakSetConstructor.cpp	(revision 204735)
>+++ Source/JavaScriptCore/runtime/WeakSetConstructor.cpp	(working copy)
>@@ -1,5 +1,5 @@
> /*
>- * Copyright (C) 2015 Apple, Inc. All rights reserved.
>+ * Copyright (C) 2015-2016 Apple, Inc. All rights reserved.
>  *
>  * Redistribution and use in source and binary forms, with or without
>  * modification, are permitted provided that the following conditions
>@@ -28,11 +28,9 @@
> 
> #include "Error.h"
> #include "IteratorOperations.h"
>-#include "JSCJSValueInlines.h"
>-#include "JSCellInlines.h"
>+#include "JSCInlines.h"
> #include "JSGlobalObject.h"
> #include "JSWeakSet.h"
>-#include "StructureInlines.h"
> #include "WeakSetPrototype.h"
> 
> namespace JSC {
>Index: Source/JavaScriptCore/runtime/WeakSetPrototype.cpp
>===================================================================
>--- Source/JavaScriptCore/runtime/WeakSetPrototype.cpp	(revision 204735)
>+++ Source/JavaScriptCore/runtime/WeakSetPrototype.cpp	(working copy)
>@@ -26,9 +26,8 @@
> #include "config.h"
> #include "WeakSetPrototype.h"
> 
>-#include "JSCJSValueInlines.h"
>+#include "JSCInlines.h"
> #include "JSWeakSet.h"
>-#include "StructureInlines.h"
> #include "WeakMapData.h"
> 
> namespace JSC {
>Index: Source/JavaScriptCore/tools/JSDollarVM.cpp
>===================================================================
>--- Source/JavaScriptCore/tools/JSDollarVM.cpp	(revision 204735)
>+++ Source/JavaScriptCore/tools/JSDollarVM.cpp	(working copy)
>@@ -26,8 +26,7 @@
> #include "config.h"
> #include "JSDollarVM.h"
> 
>-#include "JSCJSValueInlines.h"
>-#include "StructureInlines.h"
>+#include "JSCInlines.h"
> 
> namespace JSC {
> 
>Index: Source/JavaScriptCore/tools/JSDollarVMPrototype.cpp
>===================================================================
>--- Source/JavaScriptCore/tools/JSDollarVMPrototype.cpp	(revision 204735)
>+++ Source/JavaScriptCore/tools/JSDollarVMPrototype.cpp	(working copy)
>@@ -146,7 +146,13 @@ bool JSDollarVMPrototype::isInHeap(Heap*
> bool JSDollarVMPrototype::isInObjectSpace(Heap* heap, void* ptr)
> {
>     MarkedBlock* candidate = MarkedBlock::blockFor(ptr);
>-    return heap->objectSpace().blocks().set().contains(candidate);
>+    if (heap->objectSpace().blocks().set().contains(candidate))
>+        return true;
>+    for (LargeAllocation* allocation : heap->objectSpace().largeAllocations()) {
>+        if (allocation->contains(ptr))
>+            return true;
>+    }
>+    return false;
> }
> 
> bool JSDollarVMPrototype::isInStorageSpace(Heap* heap, void* ptr)
>Index: Source/WTF/ChangeLog
>===================================================================
>--- Source/WTF/ChangeLog	(revision 204735)
>+++ Source/WTF/ChangeLog	(working copy)
>@@ -1,3 +1,18 @@
>+2016-08-22  Filip Pizlo  <fpizlo@apple.com>
>+
>+        Butterflies should be allocated in Auxiliary MarkedSpace instead of CopiedSpace and we should rewrite as much of the GC as needed to make this not a regression
>+        https://bugs.webkit.org/show_bug.cgi?id=160125
>+
>+        Reviewed by Geoffrey Garen.
>+        
>+        I needed tryFastAlignedMalloc() so I added it.
>+
>+        * wtf/FastMalloc.cpp:
>+        (WTF::fastAlignedMalloc):
>+        (WTF::tryFastAlignedMalloc):
>+        (WTF::fastAlignedFree):
>+        * wtf/FastMalloc.h:
>+
> 2016-08-17  Don Olmstead  <don.olmstead@am.sony.com>
> 
>         Use find_library within Windows build
>Index: Source/WTF/wtf/FastMalloc.cpp
>===================================================================
>--- Source/WTF/wtf/FastMalloc.cpp	(revision 204735)
>+++ Source/WTF/wtf/FastMalloc.cpp	(working copy)
>@@ -1,7 +1,6 @@
>-// Copyright (c) 2005, 2007, Google Inc. All rights reserved.
>-
> /*
>- * Copyright (C) 2005-2009, 2011, 2015 Apple Inc. All rights reserved.
>+ * Copyright (c) 2005, 2007, Google Inc. All rights reserved.
>+ * Copyright (C) 2005-2009, 2011, 2015-2016 Apple Inc. All rights reserved.
>  * Redistribution and use in source and binary forms, with or without
>  * modification, are permitted provided that the following conditions
>  * are met:
>@@ -97,6 +96,11 @@ void* fastAlignedMalloc(size_t alignment
>     return _aligned_malloc(size, alignment);
> }
> 
>+void* tryFastAlignedMalloc(size_t alignment, size_t size) 
>+{
>+    return _aligned_malloc(size, alignment);
>+}
>+
> void fastAlignedFree(void* p) 
> {
>     _aligned_free(p);
>@@ -111,6 +115,13 @@ void* fastAlignedMalloc(size_t alignment
>     return p;
> }
> 
>+void* tryFastAlignedMalloc(size_t alignment, size_t size) 
>+{
>+    void* p = nullptr;
>+    posix_memalign(&p, alignment, size);
>+    return p;
>+}
>+
> void fastAlignedFree(void* p) 
> {
>     free(p);
>@@ -231,6 +242,11 @@ void* fastAlignedMalloc(size_t alignment
>     return bmalloc::api::memalign(alignment, size);
> }
> 
>+void* tryFastAlignedMalloc(size_t alignment, size_t size) 
>+{
>+    return bmalloc::api::tryMemalign(alignment, size);
>+}
>+
> void fastAlignedFree(void* p) 
> {
>     bmalloc::api::free(p);
>Index: Source/WTF/wtf/FastMalloc.h
>===================================================================
>--- Source/WTF/wtf/FastMalloc.h	(revision 204735)
>+++ Source/WTF/wtf/FastMalloc.h	(working copy)
>@@ -1,5 +1,5 @@
> /*
>- *  Copyright (C) 2005-2009, 2015 Apple Inc. All rights reserved.
>+ *  Copyright (C) 2005-2009, 2015-2016 Apple Inc. All rights reserved.
>  *
>  *  This library is free software; you can redistribute it and/or
>  *  modify it under the terms of the GNU Library General Public
>@@ -53,6 +53,7 @@ WTF_EXPORT_PRIVATE void fastFree(void*);
> 
> // Allocations from fastAlignedMalloc() must be freed using fastAlignedFree().
> WTF_EXPORT_PRIVATE void* fastAlignedMalloc(size_t alignment, size_t);
>+WTF_EXPORT_PRIVATE void* tryFastAlignedMalloc(size_t alignment, size_t);
> WTF_EXPORT_PRIVATE void fastAlignedFree(void*);
> 
> WTF_EXPORT_PRIVATE size_t fastMallocSize(const void*);
>@@ -107,6 +108,7 @@ using WTF::fastMallocSize;
> using WTF::fastRealloc;
> using WTF::fastStrDup;
> using WTF::fastZeroedMalloc;
>+using WTF::tryFastAlignedMalloc;
> using WTF::tryFastCalloc;
> using WTF::tryFastMalloc;
> using WTF::tryFastZeroedMalloc;
>Index: Source/WTF/wtf/ParkingLot.cpp
>===================================================================
>--- Source/WTF/wtf/ParkingLot.cpp	(revision 204735)
>+++ Source/WTF/wtf/ParkingLot.cpp	(working copy)
>@@ -767,7 +767,7 @@ NEVER_INLINE void ParkingLot::unparkAll(
>         dataLog(toString(currentThread(), ": done unparking.\n"));
> }
> 
>-NEVER_INLINE void ParkingLot::forEach(std::function<void(ThreadIdentifier, const void*)> callback)
>+NEVER_INLINE void ParkingLot::forEachImpl(const ScopedLambda<void(ThreadIdentifier, const void*)>& callback)
> {
>     Vector<Bucket*> bucketsToUnlock = lockHashtable();
> 
>Index: Source/WTF/wtf/ParkingLot.h
>===================================================================
>--- Source/WTF/wtf/ParkingLot.h	(revision 204735)
>+++ Source/WTF/wtf/ParkingLot.h	(working copy)
>@@ -66,14 +66,14 @@ public:
>     template<typename ValidationFunctor, typename BeforeSleepFunctor>
>     static ParkResult parkConditionally(
>         const void* address,
>-        ValidationFunctor&& validation,
>-        BeforeSleepFunctor&& beforeSleep,
>+        const ValidationFunctor& validation,
>+        const BeforeSleepFunctor& beforeSleep,
>         Clock::time_point timeout)
>     {
>         return parkConditionallyImpl(
>             address,
>-            scopedLambda<bool()>(std::forward<ValidationFunctor>(validation)),
>-            scopedLambda<void()>(std::forward<BeforeSleepFunctor>(beforeSleep)),
>+            scopedLambdaRef<bool()>(validation),
>+            scopedLambdaRef<void()>(beforeSleep),
>             timeout);
>     }
> 
>@@ -124,9 +124,9 @@ public:
>     // moment nobody can add any threads to the queue because the queue lock is still held. Also,
>     // WTF::Lock uses the timeToBeFair and token mechanism to implement eventual fairness.
>     template<typename Callback>
>-    static void unparkOne(const void* address, Callback&& callback)
>+    static void unparkOne(const void* address, const Callback& callback)
>     {
>-        unparkOneImpl(address, scopedLambda<intptr_t(UnparkResult)>(std::forward<Callback>(callback)));
>+        unparkOneImpl(address, scopedLambdaRef<intptr_t(UnparkResult)>(callback));
>     }
> 
>     // Unparks every thread from the queue associated with the given address, which cannot be null.
>@@ -145,7 +145,11 @@ public:
>     // As well as many other possible interleavings that all have T1 before T2 and T3 before T4 but are
>     // otherwise unconstrained. This method is useful primarily for debugging. It's also used by unit
>     // tests.
>-    WTF_EXPORT_PRIVATE static void forEach(std::function<void(ThreadIdentifier, const void*)>);
>+    template<typename Func>
>+    static void forEach(const Func& func)
>+    {
>+        forEachImpl(scopedLambdaRef<void(ThreadIdentifier, const void*)>(func));
>+    }
> 
> private:
>     WTF_EXPORT_PRIVATE static ParkResult parkConditionallyImpl(
>@@ -157,7 +161,7 @@ private:
>     WTF_EXPORT_PRIVATE static void unparkOneImpl(
>         const void* address, const ScopedLambda<intptr_t(UnparkResult)>& callback);
> 
>-    WTF_EXPORT_PRIVATE static void forEachImpl(const std::function<void(ThreadIdentifier, const void*)>&);
>+    WTF_EXPORT_PRIVATE static void forEachImpl(const ScopedLambda<void(ThreadIdentifier, const void*)>&);
> };
> 
> } // namespace WTF
>Index: Source/WTF/wtf/ScopedLambda.h
>===================================================================
>--- Source/WTF/wtf/ScopedLambda.h	(revision 204735)
>+++ Source/WTF/wtf/ScopedLambda.h	(working copy)
>@@ -126,10 +126,65 @@ ScopedLambdaFunctor<FunctionType, Functo
>     return ScopedLambdaFunctor<FunctionType, Functor>(WTFMove(functor));
> }
> 
>+template<typename FunctionType, typename Functor> class ScopedLambdaRefFunctor;
>+template<typename ResultType, typename... ArgumentTypes, typename Functor>
>+class ScopedLambdaRefFunctor<ResultType (ArgumentTypes...), Functor> : public ScopedLambda<ResultType (ArgumentTypes...)> {
>+public:
>+    ScopedLambdaRefFunctor(const Functor& functor)
>+        : ScopedLambda<ResultType (ArgumentTypes...)>(implFunction, this)
>+        , m_functor(&functor)
>+    {
>+    }
>+    
>+    // We need to make sure that copying and moving ScopedLambdaRefFunctor results in a
>+    // ScopedLambdaRefFunctor whose ScopedLambda supertype still points to this rather than
>+    // other.
>+    ScopedLambdaRefFunctor(const ScopedLambdaRefFunctor& other)
>+        : ScopedLambda<ResultType (ArgumentTypes...)>(implFunction, this)
>+        , m_functor(other.m_functor)
>+    {
>+    }
>+
>+    ScopedLambdaRefFunctor(ScopedLambdaRefFunctor&& other)
>+        : ScopedLambda<ResultType (ArgumentTypes...)>(implFunction, this)
>+        , m_functor(other.m_functor)
>+    {
>+    }
>+    
>+    ScopedLambdaRefFunctor& operator=(const ScopedLambdaRefFunctor& other)
>+    {
>+        m_functor = other.m_functor;
>+        return *this;
>+    }
>+    
>+    ScopedLambdaRefFunctor& operator=(ScopedLambdaRefFunctor&& other)
>+    {
>+        m_functor = other.m_functor;
>+        return *this;
>+    }
>+
>+private:
>+    static ResultType implFunction(void* argument, ArgumentTypes... arguments)
>+    {
>+        return (*static_cast<ScopedLambdaRefFunctor*>(argument)->m_functor)(arguments...);
>+    }
>+
>+    const Functor* m_functor;
>+};
>+
>+// This is for when you already refer to a functor by reference, and you know its lifetime is
>+// good. This just creates a ScopedLambda that points to your functor.
>+template<typename FunctionType, typename Functor>
>+ScopedLambdaRefFunctor<FunctionType, Functor> scopedLambdaRef(const Functor& functor)
>+{
>+    return ScopedLambdaRefFunctor<FunctionType, Functor>(functor);
>+}
>+
> } // namespace WTF
> 
> using WTF::ScopedLambda;
> using WTF::scopedLambda;
>+using WTF::scopedLambdaRef;
> 
> #endif // ScopedLambda_h
> 
>Index: Source/WebCore/ChangeLog
>===================================================================
>--- Source/WebCore/ChangeLog	(revision 204735)
>+++ Source/WebCore/ChangeLog	(working copy)
>@@ -1,3 +1,34 @@
>+2016-08-22  Filip Pizlo  <fpizlo@apple.com>
>+
>+        Butterflies should be allocated in Auxiliary MarkedSpace instead of CopiedSpace and we should rewrite as much of the GC as needed to make this not a regression
>+        https://bugs.webkit.org/show_bug.cgi?id=160125
>+
>+        Reviewed by Geoffrey Garen.
>+
>+        No new tests because no new WebCore behavior.
>+        
>+        Just rewiring #includes.
>+
>+        * ForwardingHeaders/heap/HeapInlines.h: Added.
>+        * ForwardingHeaders/runtime/AuxiliaryBarrierInlines.h: Added.
>+        * Modules/indexeddb/server/SQLiteIDBBackingStore.cpp:
>+        * Modules/indexeddb/server/UniqueIDBDatabase.cpp:
>+        * bindings/js/JSApplePayPaymentAuthorizedEventCustom.cpp:
>+        * bindings/js/JSApplePayPaymentMethodSelectedEventCustom.cpp:
>+        * bindings/js/JSApplePayShippingContactSelectedEventCustom.cpp:
>+        * bindings/js/JSApplePayShippingMethodSelectedEventCustom.cpp:
>+        * bindings/js/JSClientRectCustom.cpp:
>+        * bindings/js/JSDOMBinding.h:
>+        * bindings/js/JSDOMStringListCustom.cpp:
>+        * bindings/js/JSErrorEventCustom.cpp:
>+        * bindings/js/JSPopStateEventCustom.cpp:
>+        * bindings/js/JSWebGL2RenderingContextCustom.cpp:
>+        * contentextensions/ContentExtensionParser.cpp:
>+        * dom/ErrorEvent.cpp:
>+        * inspector/CommandLineAPIModule.cpp:
>+        * testing/GCObservation.cpp:
>+        (WebCore::GCObservation::GCObservation):
>+
> 2016-08-22  Alex Christensen  <achristensen@webkit.org>
> 
>         Fix Mac CMake build after r204717.
>Index: Source/WebCore/ForwardingHeaders/heap/HeapInlines.h
>===================================================================
>--- Source/WebCore/ForwardingHeaders/heap/HeapInlines.h	(nonexistent)
>+++ Source/WebCore/ForwardingHeaders/heap/HeapInlines.h	(working copy)
>@@ -0,0 +1,3 @@
>+#pragma once
>+#include <JavaScriptCore/HeapInlines.h>
>+
>Index: Source/WebCore/ForwardingHeaders/interpreter/Interpreter.h
>===================================================================
>--- Source/WebCore/ForwardingHeaders/interpreter/Interpreter.h	(revision 204735)
>+++ Source/WebCore/ForwardingHeaders/interpreter/Interpreter.h	(nonexistent)
>@@ -1,4 +0,0 @@
>-#ifndef WebCore_FWD_Interpreter_h
>-#define WebCore_FWD_Interpreter_h
>-#include <JavaScriptCore/Interpreter.h>
>-#endif
>
>Property changes on: Source/WebCore/ForwardingHeaders/interpreter/Interpreter.h
>___________________________________________________________________
>Deleted: svn:eol-style
>## -1 +0,0 ##
>-native
>\ No newline at end of property
>Index: Source/WebCore/ForwardingHeaders/runtime/AuxiliaryBarrierInlines.h
>===================================================================
>--- Source/WebCore/ForwardingHeaders/runtime/AuxiliaryBarrierInlines.h	(nonexistent)
>+++ Source/WebCore/ForwardingHeaders/runtime/AuxiliaryBarrierInlines.h	(working copy)
>@@ -0,0 +1,3 @@
>+#pragma once
>+#include <JavaScriptCore/AuxiliaryBarrierInlines.h>
>+
>Index: Source/WebCore/Modules/indexeddb/IDBCursorWithValue.cpp
>===================================================================
>--- Source/WebCore/Modules/indexeddb/IDBCursorWithValue.cpp	(revision 204735)
>+++ Source/WebCore/Modules/indexeddb/IDBCursorWithValue.cpp	(working copy)
>@@ -28,6 +28,8 @@
> 
> #if ENABLE(INDEXED_DATABASE)
> 
>+#include <heap/HeapInlines.h>
>+
> namespace WebCore {
> 
> Ref<IDBCursorWithValue> IDBCursorWithValue::create(IDBTransaction& transaction, IDBObjectStore& objectStore, const IDBCursorInfo& info)
>Index: Source/WebCore/Modules/indexeddb/client/TransactionOperation.cpp
>===================================================================
>--- Source/WebCore/Modules/indexeddb/client/TransactionOperation.cpp	(revision 204735)
>+++ Source/WebCore/Modules/indexeddb/client/TransactionOperation.cpp	(working copy)
>@@ -29,6 +29,7 @@
> #if ENABLE(INDEXED_DATABASE)
> 
> #include "IDBCursor.h"
>+#include <heap/HeapInlines.h>
> 
> namespace WebCore {
> namespace IDBClient {
>Index: Source/WebCore/Modules/indexeddb/server/SQLiteIDBBackingStore.cpp
>===================================================================
>--- Source/WebCore/Modules/indexeddb/server/SQLiteIDBBackingStore.cpp	(revision 204735)
>+++ Source/WebCore/Modules/indexeddb/server/SQLiteIDBBackingStore.cpp	(working copy)
>@@ -45,7 +45,9 @@
> #include "SQLiteStatement.h"
> #include "SQLiteTransaction.h"
> #include "ThreadSafeDataBuffer.h"
>+#include <heap/HeapInlines.h>
> #include <heap/StrongInlines.h>
>+#include <runtime/AuxiliaryBarrierInlines.h>
> #include <runtime/JSCJSValueInlines.h>
> #include <runtime/JSGlobalObject.h>
> #include <runtime/StructureInlines.h>
>Index: Source/WebCore/Modules/indexeddb/server/UniqueIDBDatabase.cpp
>===================================================================
>--- Source/WebCore/Modules/indexeddb/server/UniqueIDBDatabase.cpp	(revision 204735)
>+++ Source/WebCore/Modules/indexeddb/server/UniqueIDBDatabase.cpp	(working copy)
>@@ -39,6 +39,8 @@
> #include "ScopeGuard.h"
> #include "SerializedScriptValue.h"
> #include "UniqueIDBDatabaseConnection.h"
>+#include <heap/HeapInlines.h>
>+#include <runtime/AuxiliaryBarrierInlines.h>
> #include <runtime/StructureInlines.h>
> #include <wtf/MainThread.h>
> #include <wtf/NeverDestroyed.h>
>Index: Source/WebCore/bindings/js/JSApplePayPaymentAuthorizedEventCustom.cpp
>===================================================================
>--- Source/WebCore/bindings/js/JSApplePayPaymentAuthorizedEventCustom.cpp	(revision 204735)
>+++ Source/WebCore/bindings/js/JSApplePayPaymentAuthorizedEventCustom.cpp	(working copy)
>@@ -28,6 +28,7 @@
> 
> #if ENABLE(APPLE_PAY)
> 
>+#include <heap/HeapInlines.h>
> #include <runtime/JSCJSValueInlines.h>
> 
> namespace WebCore {
>Index: Source/WebCore/bindings/js/JSApplePayPaymentMethodSelectedEventCustom.cpp
>===================================================================
>--- Source/WebCore/bindings/js/JSApplePayPaymentMethodSelectedEventCustom.cpp	(revision 204735)
>+++ Source/WebCore/bindings/js/JSApplePayPaymentMethodSelectedEventCustom.cpp	(working copy)
>@@ -28,6 +28,7 @@
> 
> #if ENABLE(APPLE_PAY)
> 
>+#include <heap/HeapInlines.h>
> #include <runtime/JSCJSValueInlines.h>
> 
> using namespace JSC;
>Index: Source/WebCore/bindings/js/JSApplePayShippingContactSelectedEventCustom.cpp
>===================================================================
>--- Source/WebCore/bindings/js/JSApplePayShippingContactSelectedEventCustom.cpp	(revision 204735)
>+++ Source/WebCore/bindings/js/JSApplePayShippingContactSelectedEventCustom.cpp	(working copy)
>@@ -28,6 +28,7 @@
> 
> #if ENABLE(APPLE_PAY)
> 
>+#include <heap/HeapInlines.h>
> #include <runtime/JSCJSValueInlines.h>
> 
> using namespace JSC;
>Index: Source/WebCore/bindings/js/JSApplePayShippingMethodSelectedEventCustom.cpp
>===================================================================
>--- Source/WebCore/bindings/js/JSApplePayShippingMethodSelectedEventCustom.cpp	(revision 204735)
>+++ Source/WebCore/bindings/js/JSApplePayShippingMethodSelectedEventCustom.cpp	(working copy)
>@@ -28,10 +28,9 @@
> 
> #if ENABLE(APPLE_PAY)
> 
>-#include <runtime/IdentifierInlines.h>
>-#include <runtime/JSCJSValueInlines.h>
>+#include <runtime/JSCInlines.h>
> #include <runtime/ObjectConstructor.h>
>-#include <runtime/StructureInlines.h>
>+#include <wtf/text/StringBuilder.h>
> 
> using namespace JSC;
> 
>Index: Source/WebCore/bindings/js/JSClientRectCustom.cpp
>===================================================================
>--- Source/WebCore/bindings/js/JSClientRectCustom.cpp	(revision 204735)
>+++ Source/WebCore/bindings/js/JSClientRectCustom.cpp	(working copy)
>@@ -28,6 +28,8 @@
> 
> #include "ClientRect.h"
> #include <bytecode/CodeBlock.h>
>+#include <heap/HeapInlines.h>
>+#include <runtime/AuxiliaryBarrierInlines.h>
> #include <runtime/IdentifierInlines.h>
> #include <runtime/JSObject.h>
> #include <runtime/ObjectConstructor.h>
>Index: Source/WebCore/bindings/js/JSDOMBinding.cpp
>===================================================================
>--- Source/WebCore/bindings/js/JSDOMBinding.cpp	(revision 204735)
>+++ Source/WebCore/bindings/js/JSDOMBinding.cpp	(working copy)
>@@ -38,7 +38,6 @@
> #include <bytecode/CodeBlock.h>
> #include <inspector/ScriptCallStack.h>
> #include <inspector/ScriptCallStackFactory.h>
>-#include <interpreter/Interpreter.h>
> #include <runtime/DateInstance.h>
> #include <runtime/Error.h>
> #include <runtime/ErrorHandlingScope.h>
>@@ -49,6 +48,7 @@
> #include <stdarg.h>
> #include <wtf/MathExtras.h>
> #include <wtf/unicode/CharacterNames.h>
>+#include <wtf/text/StringBuilder.h>
> 
> using namespace JSC;
> using namespace Inspector;
>Index: Source/WebCore/bindings/js/JSDOMBinding.h
>===================================================================
>--- Source/WebCore/bindings/js/JSDOMBinding.h	(revision 204735)
>+++ Source/WebCore/bindings/js/JSDOMBinding.h	(working copy)
>@@ -30,9 +30,11 @@
> #include "ScriptWrappableInlines.h"
> #include "WebCoreTypedArrayController.h"
> #include <cstddef>
>+#include <heap/HeapInlines.h>
> #include <heap/SlotVisitorInlines.h>
> #include <heap/Weak.h>
> #include <heap/WeakInlines.h>
>+#include <runtime/AuxiliaryBarrierInlines.h>
> #include <runtime/Error.h>
> #include <runtime/IteratorOperations.h>
> #include <runtime/JSArray.h>
>Index: Source/WebCore/bindings/js/JSDeviceMotionEventCustom.cpp
>===================================================================
>--- Source/WebCore/bindings/js/JSDeviceMotionEventCustom.cpp	(revision 204735)
>+++ Source/WebCore/bindings/js/JSDeviceMotionEventCustom.cpp	(working copy)
>@@ -31,6 +31,8 @@
> 
> #include "DeviceMotionData.h"
> #include "DeviceMotionEvent.h"
>+#include <heap/HeapInlines.h>
>+#include <runtime/AuxiliaryBarrierInlines.h>
> #include <runtime/IdentifierInlines.h>
> #include <runtime/JSCJSValueInlines.h>
> #include <runtime/ObjectConstructor.h>
>Index: Source/WebCore/bindings/js/JSDeviceOrientationEventCustom.cpp
>===================================================================
>--- Source/WebCore/bindings/js/JSDeviceOrientationEventCustom.cpp	(revision 204735)
>+++ Source/WebCore/bindings/js/JSDeviceOrientationEventCustom.cpp	(working copy)
>@@ -31,6 +31,7 @@
> 
> #include "DeviceOrientationData.h"
> #include "DeviceOrientationEvent.h"
>+#include <heap/HeapInlines.h>
> #include <runtime/JSCJSValueInlines.h>
> #include <runtime/StructureInlines.h>
> 
>Index: Source/WebCore/bindings/js/JSErrorEventCustom.cpp
>===================================================================
>--- Source/WebCore/bindings/js/JSErrorEventCustom.cpp	(revision 204735)
>+++ Source/WebCore/bindings/js/JSErrorEventCustom.cpp	(working copy)
>@@ -27,6 +27,7 @@
> #include "JSErrorEvent.h"
> 
> #include "ErrorEvent.h"
>+#include <heap/HeapInlines.h>
> 
> using namespace JSC;
> 
>Index: Source/WebCore/bindings/js/JSIDBCursorWithValueCustom.cpp
>===================================================================
>--- Source/WebCore/bindings/js/JSIDBCursorWithValueCustom.cpp	(revision 204735)
>+++ Source/WebCore/bindings/js/JSIDBCursorWithValueCustom.cpp	(working copy)
>@@ -29,6 +29,7 @@
> #if ENABLE(INDEXED_DATABASE)
> 
> #include "IDBCursorWithValue.h"
>+#include <heap/HeapInlines.h>
> 
> using namespace JSC;
> 
>Index: Source/WebCore/bindings/js/JSIDBIndexCustom.cpp
>===================================================================
>--- Source/WebCore/bindings/js/JSIDBIndexCustom.cpp	(revision 204735)
>+++ Source/WebCore/bindings/js/JSIDBIndexCustom.cpp	(working copy)
>@@ -29,6 +29,7 @@
> #if ENABLE(INDEXED_DATABASE)
> 
> #include "IDBIndex.h"
>+#include <heap/HeapInlines.h>
> 
> using namespace JSC;
> 
>Index: Source/WebCore/bindings/js/JSPopStateEventCustom.cpp
>===================================================================
>--- Source/WebCore/bindings/js/JSPopStateEventCustom.cpp	(revision 204735)
>+++ Source/WebCore/bindings/js/JSPopStateEventCustom.cpp	(working copy)
>@@ -33,6 +33,7 @@
> 
> #include "DOMWrapperWorld.h"
> #include "JSHistory.h"
>+#include <heap/HeapInlines.h>
> #include <runtime/JSCJSValueInlines.h>
> 
> using namespace JSC;
>Index: Source/WebCore/bindings/js/JSWebGL2RenderingContextCustom.cpp
>===================================================================
>--- Source/WebCore/bindings/js/JSWebGL2RenderingContextCustom.cpp	(revision 204735)
>+++ Source/WebCore/bindings/js/JSWebGL2RenderingContextCustom.cpp	(working copy)
>@@ -28,6 +28,7 @@
> #if ENABLE(WEBGL) && ENABLE(WEBGL2)
> #include "JSWebGL2RenderingContext.h"
> 
>+#include <heap/HeapInlines.h>
> #include <runtime/Error.h>
> #include "NotImplemented.h"
> #include "WebGL2RenderingContext.h"
>Index: Source/WebCore/bindings/js/JSWorkerGlobalScopeCustom.cpp
>===================================================================
>--- Source/WebCore/bindings/js/JSWorkerGlobalScopeCustom.cpp	(revision 204735)
>+++ Source/WebCore/bindings/js/JSWorkerGlobalScopeCustom.cpp	(working copy)
>@@ -41,7 +41,6 @@
> #include "WorkerGlobalScope.h"
> #include "WorkerLocation.h"
> #include "WorkerNavigator.h"
>-#include <interpreter/Interpreter.h>
> 
> #if ENABLE(WEB_SOCKETS)
> #include "JSWebSocket.h"
>Index: Source/WebCore/bindings/js/WorkerScriptController.cpp
>===================================================================
>--- Source/WebCore/bindings/js/WorkerScriptController.cpp	(revision 204735)
>+++ Source/WebCore/bindings/js/WorkerScriptController.cpp	(working copy)
>@@ -39,7 +39,6 @@
> #include "WorkerThread.h"
> #include <bindings/ScriptValue.h>
> #include <heap/StrongInlines.h>
>-#include <interpreter/Interpreter.h>
> #include <runtime/Completion.h>
> #include <runtime/Error.h>
> #include <runtime/Exception.h>
>Index: Source/WebCore/contentextensions/ContentExtensionParser.cpp
>===================================================================
>--- Source/WebCore/contentextensions/ContentExtensionParser.cpp	(revision 204735)
>+++ Source/WebCore/contentextensions/ContentExtensionParser.cpp	(working copy)
>@@ -35,11 +35,9 @@
> #include "ContentExtensionRule.h"
> #include "ContentExtensionsBackend.h"
> #include "ContentExtensionsDebugging.h"
>-#include <JavaScriptCore/IdentifierInlines.h>
>-#include <JavaScriptCore/JSCJSValueInlines.h>
>+#include <JavaScriptCore/JSCInlines.h>
> #include <JavaScriptCore/JSGlobalObject.h>
> #include <JavaScriptCore/JSONObject.h>
>-#include <JavaScriptCore/StructureInlines.h>
> #include <JavaScriptCore/VM.h>
> #include <wtf/CurrentTime.h>
> #include <wtf/text/WTFString.h>
>Index: Source/WebCore/dom/ErrorEvent.cpp
>===================================================================
>--- Source/WebCore/dom/ErrorEvent.cpp	(revision 204735)
>+++ Source/WebCore/dom/ErrorEvent.cpp	(working copy)
>@@ -34,6 +34,7 @@
> 
> #include "DOMWrapperWorld.h"
> #include "EventNames.h"
>+#include <heap/HeapInlines.h>
> 
> using namespace JSC;
> 
>Index: Source/WebCore/html/HTMLCanvasElement.cpp
>===================================================================
>--- Source/WebCore/html/HTMLCanvasElement.cpp	(revision 204735)
>+++ Source/WebCore/html/HTMLCanvasElement.cpp	(working copy)
>@@ -51,6 +51,7 @@
> #include <runtime/JSCInlines.h>
> #include <runtime/JSLock.h>
> #include <wtf/RAMSize.h>
>+#include <wtf/text/StringBuilder.h>
> 
> #if ENABLE(WEBGL)    
> #include "WebGLContextAttributes.h"
>Index: Source/WebCore/html/MediaDocument.cpp
>===================================================================
>--- Source/WebCore/html/MediaDocument.cpp	(revision 204735)
>+++ Source/WebCore/html/MediaDocument.cpp	(working copy)
>@@ -48,6 +48,7 @@
> #include "ScriptController.h"
> #include "ShadowRoot.h"
> #include "TypedElementDescendantIterator.h"
>+#include <wtf/text/StringBuilder.h>
> 
> namespace WebCore {
> 
>Index: Source/WebCore/inspector/CommandLineAPIModule.cpp
>===================================================================
>--- Source/WebCore/inspector/CommandLineAPIModule.cpp	(revision 204735)
>+++ Source/WebCore/inspector/CommandLineAPIModule.cpp	(working copy)
>@@ -29,6 +29,7 @@
> #include "CommandLineAPIModuleSource.h"
> #include "JSDOMGlobalObject.h"
> #include "WebInjectedScriptManager.h"
>+#include <heap/HeapInlines.h>
> #include <inspector/InjectedScript.h>
> 
> using namespace JSC;
>Index: Source/WebCore/loader/EmptyClients.cpp
>===================================================================
>--- Source/WebCore/loader/EmptyClients.cpp	(revision 204735)
>+++ Source/WebCore/loader/EmptyClients.cpp	(working copy)
>@@ -48,6 +48,7 @@
> #include "StorageNamespaceProvider.h"
> #include "ThreadableWebSocketChannel.h"
> #include "UserContentProvider.h"
>+#include <heap/HeapInlines.h>
> #include <wtf/NeverDestroyed.h>
> 
> namespace WebCore {
>Index: Source/WebCore/page/CaptionUserPreferences.cpp
>===================================================================
>--- Source/WebCore/page/CaptionUserPreferences.cpp	(revision 204735)
>+++ Source/WebCore/page/CaptionUserPreferences.cpp	(working copy)
>@@ -38,6 +38,7 @@
> #include "UserContentTypes.h"
> #include "UserStyleSheet.h"
> #include "UserStyleSheetTypes.h"
>+#include <heap/HeapInlines.h>
> #include <runtime/JSCellInlines.h>
> #include <runtime/StructureInlines.h>
> #include <wtf/NeverDestroyed.h>
>Index: Source/WebCore/page/Frame.cpp
>===================================================================
>--- Source/WebCore/page/Frame.cpp	(revision 204735)
>+++ Source/WebCore/page/Frame.cpp	(working copy)
>@@ -110,6 +110,7 @@
> #include <bindings/ScriptValue.h>
> #include <wtf/RefCountedLeakCounter.h>
> #include <wtf/StdLibExtras.h>
>+#include <wtf/text/StringBuilder.h>
> #include <yarr/RegularExpression.h>
> 
> #if PLATFORM(IOS)
>Index: Source/WebCore/page/PageGroup.cpp
>===================================================================
>--- Source/WebCore/page/PageGroup.cpp	(revision 204735)
>+++ Source/WebCore/page/PageGroup.cpp	(working copy)
>@@ -36,6 +36,7 @@
> #include "SecurityOrigin.h"
> #include "Settings.h"
> #include "StorageNamespace.h"
>+#include <heap/HeapInlines.h>
> #include <runtime/StructureInlines.h>
> #include <wtf/StdLibExtras.h>
> 
>Index: Source/WebCore/page/UserContentController.cpp
>===================================================================
>--- Source/WebCore/page/UserContentController.cpp	(revision 204735)
>+++ Source/WebCore/page/UserContentController.cpp	(working copy)
>@@ -29,6 +29,7 @@
> #include "DOMWrapperWorld.h"
> #include "UserScript.h"
> #include "UserStyleSheet.h"
>+#include <heap/HeapInlines.h>
> #include <runtime/JSCellInlines.h>
> #include <runtime/StructureInlines.h>
> 
>Index: Source/WebCore/platform/mock/mediasource/MockBox.cpp
>===================================================================
>--- Source/WebCore/platform/mock/mediasource/MockBox.cpp	(revision 204735)
>+++ Source/WebCore/platform/mock/mediasource/MockBox.cpp	(working copy)
>@@ -28,6 +28,7 @@
> 
> #if ENABLE(MEDIA_SOURCE)
> 
>+#include <JavaScriptCore/HeapInlines.h>
> #include <JavaScriptCore/JSCJSValueInlines.h>
> #include <JavaScriptCore/TypedArrayInlines.h>
> #include <runtime/ArrayBuffer.h>
>Index: Source/WebCore/testing/GCObservation.cpp
>===================================================================
>--- Source/WebCore/testing/GCObservation.cpp	(revision 204735)
>+++ Source/WebCore/testing/GCObservation.cpp	(working copy)
>@@ -26,6 +26,8 @@
> #include "config.h"
> #include "GCObservation.h"
> 
>+#include <heap/HeapInlines.h>
>+
> namespace WebCore {
> 
> GCObservation::GCObservation(JSC::JSObject* object)
>Index: Source/WebKit2/ChangeLog
>===================================================================
>--- Source/WebKit2/ChangeLog	(revision 204735)
>+++ Source/WebKit2/ChangeLog	(working copy)
>@@ -1,3 +1,15 @@
>+2016-08-22  Filip Pizlo  <fpizlo@apple.com>
>+
>+        Butterflies should be allocated in Auxiliary MarkedSpace instead of CopiedSpace and we should rewrite as much of the GC as needed to make this not a regression
>+        https://bugs.webkit.org/show_bug.cgi?id=160125
>+
>+        Reviewed by Geoffrey Garen.
>+        
>+        Just rewiring some #includes.
>+
>+        * WebProcess/InjectedBundle/DOM/InjectedBundleRangeHandle.cpp:
>+        * WebProcess/Plugins/Netscape/JSNPObject.cpp:
>+
> 2016-08-22  Alex Christensen  <achristensen@webkit.org>
> 
>         Fix Mac CMake build after r204717.
>Index: Source/WebKit2/UIProcess/ViewGestureController.cpp
>===================================================================
>--- Source/WebKit2/UIProcess/ViewGestureController.cpp	(revision 204735)
>+++ Source/WebKit2/UIProcess/ViewGestureController.cpp	(working copy)
>@@ -33,6 +33,7 @@
> #import "WebProcessProxy.h"
> #import <wtf/MathExtras.h>
> #import <wtf/NeverDestroyed.h>
>+#import <wtf/text/StringBuilder.h>
> 
> using namespace WebCore;
> 
>Index: Source/WebKit2/UIProcess/WebPageProxy.cpp
>===================================================================
>--- Source/WebKit2/UIProcess/WebPageProxy.cpp	(revision 204735)
>+++ Source/WebKit2/UIProcess/WebPageProxy.cpp	(working copy)
>@@ -125,6 +125,7 @@
> #include <WebCore/WindowFeatures.h>
> #include <stdio.h>
> #include <wtf/NeverDestroyed.h>
>+#include <wtf/text/StringBuilder.h>
> #include <wtf/text/StringView.h>
> 
> #if ENABLE(ASYNC_SCROLLING)
>Index: Source/WebKit2/UIProcess/WebProcessPool.cpp
>===================================================================
>--- Source/WebKit2/UIProcess/WebProcessPool.cpp	(revision 204735)
>+++ Source/WebKit2/UIProcess/WebProcessPool.cpp	(working copy)
>@@ -74,6 +74,7 @@
> #include <wtf/MainThread.h>
> #include <wtf/NeverDestroyed.h>
> #include <wtf/RunLoop.h>
>+#include <wtf/text/StringBuilder.h>
> 
> #if ENABLE(BATTERY_STATUS)
> #include "WebBatteryManagerProxy.h"
>Index: Source/WebKit2/UIProcess/WebProcessProxy.cpp
>===================================================================
>--- Source/WebKit2/UIProcess/WebProcessProxy.cpp	(revision 204735)
>+++ Source/WebKit2/UIProcess/WebProcessProxy.cpp	(working copy)
>@@ -56,6 +56,7 @@
> #include <wtf/NeverDestroyed.h>
> #include <wtf/RunLoop.h>
> #include <wtf/text/CString.h>
>+#include <wtf/text/StringBuilder.h>
> #include <wtf/text/WTFString.h>
> 
> #if PLATFORM(COCOA)
>Index: Source/WebKit2/WebProcess/InjectedBundle/DOM/InjectedBundleRangeHandle.cpp
>===================================================================
>--- Source/WebKit2/WebProcess/InjectedBundle/DOM/InjectedBundleRangeHandle.cpp	(revision 204735)
>+++ Source/WebKit2/WebProcess/InjectedBundle/DOM/InjectedBundleRangeHandle.cpp	(working copy)
>@@ -29,6 +29,7 @@
> #include "ShareableBitmap.h"
> #include "WebImage.h"
> #include <JavaScriptCore/APICast.h>
>+#include <JavaScriptCore/HeapInlines.h>
> #include <WebCore/Document.h>
> #include <WebCore/FloatRect.h>
> #include <WebCore/Frame.h>
>Index: Source/WebKit2/WebProcess/Plugins/Netscape/JSNPObject.cpp
>===================================================================
>--- Source/WebKit2/WebProcess/Plugins/Netscape/JSNPObject.cpp	(revision 204735)
>+++ Source/WebKit2/WebProcess/Plugins/Netscape/JSNPObject.cpp	(working copy)
>@@ -1,5 +1,5 @@
> /*
>- * Copyright (C) 2010 Apple Inc. All rights reserved.
>+ * Copyright (C) 2010, 2016 Apple Inc. All rights reserved.
>  *
>  * Redistribution and use in source and binary forms, with or without
>  * modification, are permitted provided that the following conditions
>@@ -32,6 +32,7 @@
> #include "NPJSObject.h"
> #include "NPRuntimeObjectMap.h"
> #include "NPRuntimeUtilities.h"
>+#include <JavaScriptCore/AuxiliaryBarrierInlines.h>
> #include <JavaScriptCore/Error.h>
> #include <JavaScriptCore/IdentifierInlines.h>
> #include <JavaScriptCore/JSGlobalObject.h>
>Index: Source/bmalloc/ChangeLog
>===================================================================
>--- Source/bmalloc/ChangeLog	(revision 204735)
>+++ Source/bmalloc/ChangeLog	(working copy)
>@@ -1,3 +1,26 @@
>+2016-08-12  Filip Pizlo  <fpizlo@apple.com>
>+
>+        Butterflies should be allocated in Auxiliary MarkedSpace instead of CopiedSpace and we should rewrite as much of the GC as needed to make this not a regression
>+        https://bugs.webkit.org/show_bug.cgi?id=160125
>+
>+        Reviewed by Geoffrey Garen.
>+        
>+        I needed to tryMemalign, so I added such a thing.
>+
>+        * bmalloc/Allocator.cpp:
>+        (bmalloc::Allocator::allocate):
>+        (bmalloc::Allocator::tryAllocate):
>+        (bmalloc::Allocator::allocateImpl):
>+        (bmalloc::Allocator::reallocate):
>+        * bmalloc/Allocator.h:
>+        * bmalloc/Cache.h:
>+        (bmalloc::Cache::allocate):
>+        (bmalloc::Cache::tryAllocate):
>+        * bmalloc/bmalloc.h:
>+        (bmalloc::api::malloc):
>+        (bmalloc::api::tryMemalign):
>+        (bmalloc::api::memalign):
>+
> 2016-08-03  Geoffrey Garen  <ggaren@apple.com>
> 
>         [bmalloc] Merging of XLargeRanges can leak the upper range
>Index: Source/bmalloc/bmalloc/Allocator.cpp
>===================================================================
>--- Source/bmalloc/bmalloc/Allocator.cpp	(revision 204735)
>+++ Source/bmalloc/bmalloc/Allocator.cpp	(working copy)
>@@ -1,5 +1,5 @@
> /*
>- * Copyright (C) 2014, 2015 Apple Inc. All rights reserved.
>+ * Copyright (C) 2014-2016 Apple Inc. All rights reserved.
>  *
>  * Redistribution and use in source and binary forms, with or without
>  * modification, are permitted provided that the following conditions
>@@ -64,6 +64,18 @@ void* Allocator::tryAllocate(size_t size
> 
> void* Allocator::allocate(size_t alignment, size_t size)
> {
>+    bool crashOnFailure = true;
>+    return allocateImpl(alignment, size, crashOnFailure);
>+}
>+
>+void* Allocator::tryAllocate(size_t alignment, size_t size)
>+{
>+    bool crashOnFailure = false;
>+    return allocateImpl(alignment, size, crashOnFailure);
>+}
>+
>+void* Allocator::allocateImpl(size_t alignment, size_t size, bool crashOnFailure)
>+{
>     BASSERT(isPowerOfTwo(alignment));
> 
>     if (!m_isBmallocEnabled) {
>@@ -80,7 +92,10 @@ void* Allocator::allocate(size_t alignme
>         return allocate(roundUpToMultipleOf(alignment, size));
> 
>     std::lock_guard<StaticMutex> lock(PerProcess<Heap>::mutex());
>-    return PerProcess<Heap>::getFastCase()->allocateLarge(lock, alignment, size);
>+    Heap* heap = PerProcess<Heap>::getFastCase();
>+    if (crashOnFailure)
>+        return heap->allocateLarge(lock, alignment, size);
>+    return heap->tryAllocateLarge(lock, alignment, size);
> }
> 
> void* Allocator::reallocate(void* object, size_t newSize)
>Index: Source/bmalloc/bmalloc/Allocator.h
>===================================================================
>--- Source/bmalloc/bmalloc/Allocator.h	(revision 204735)
>+++ Source/bmalloc/bmalloc/Allocator.h	(working copy)
>@@ -1,5 +1,5 @@
> /*
>- * Copyright (C) 2014 Apple Inc. All rights reserved.
>+ * Copyright (C) 2014, 2016 Apple Inc. All rights reserved.
>  *
>  * Redistribution and use in source and binary forms, with or without
>  * modification, are permitted provided that the following conditions
>@@ -43,12 +43,15 @@ public:
> 
>     void* tryAllocate(size_t);
>     void* allocate(size_t);
>+    void* tryAllocate(size_t alignment, size_t);
>     void* allocate(size_t alignment, size_t);
>     void* reallocate(void*, size_t);
> 
>     void scavenge();
> 
> private:
>+    void* allocateImpl(size_t alignment, size_t, bool crashOnFailure);
>+    
>     bool allocateFastCase(size_t, void*&);
>     void* allocateSlowCase(size_t);
>     
>Index: Source/bmalloc/bmalloc/Cache.h
>===================================================================
>--- Source/bmalloc/bmalloc/Cache.h	(revision 204735)
>+++ Source/bmalloc/bmalloc/Cache.h	(working copy)
>@@ -1,5 +1,5 @@
> /*
>- * Copyright (C) 2014 Apple Inc. All rights reserved.
>+ * Copyright (C) 2014, 2016 Apple Inc. All rights reserved.
>  *
>  * Redistribution and use in source and binary forms, with or without
>  * modification, are permitted provided that the following conditions
>@@ -41,6 +41,7 @@ public:
> 
>     static void* tryAllocate(size_t);
>     static void* allocate(size_t);
>+    static void* tryAllocate(size_t alignment, size_t);
>     static void* allocate(size_t alignment, size_t);
>     static void deallocate(void*);
>     static void* reallocate(void*, size_t);
>@@ -79,6 +80,14 @@ inline void* Cache::allocate(size_t size
>     return cache->allocator().allocate(size);
> }
> 
>+inline void* Cache::tryAllocate(size_t alignment, size_t size)
>+{
>+    Cache* cache = PerThread<Cache>::getFastCase();
>+    if (!cache)
>+        return allocateSlowCaseNullCache(alignment, size);
>+    return cache->allocator().tryAllocate(alignment, size);
>+}
>+
> inline void* Cache::allocate(size_t alignment, size_t size)
> {
>     Cache* cache = PerThread<Cache>::getFastCase();
>Index: Source/bmalloc/bmalloc/bmalloc.h
>===================================================================
>--- Source/bmalloc/bmalloc/bmalloc.h	(revision 204735)
>+++ Source/bmalloc/bmalloc/bmalloc.h	(working copy)
>@@ -1,5 +1,5 @@
> /*
>- * Copyright (C) 2014, 2015 Apple Inc. All rights reserved.
>+ * Copyright (C) 2014-2016 Apple Inc. All rights reserved.
>  *
>  * Redistribution and use in source and binary forms, with or without
>  * modification, are permitted provided that the following conditions
>@@ -43,6 +43,12 @@ inline void* malloc(size_t size)
>     return Cache::allocate(size);
> }
> 
>+// Returns null on failure.
>+inline void* tryMemalign(size_t alignment, size_t size)
>+{
>+    return Cache::tryAllocate(alignment, size);
>+}
>+
> // Crashes on failure.
> inline void* memalign(size_t alignment, size_t size)
> {
>Index: Tools/ChangeLog
>===================================================================
>--- Tools/ChangeLog	(revision 204735)
>+++ Tools/ChangeLog	(working copy)
>@@ -1,3 +1,13 @@
>+2016-08-22  Filip Pizlo  <fpizlo@apple.com>
>+
>+        Butterflies should be allocated in Auxiliary MarkedSpace instead of CopiedSpace and we should rewrite as much of the GC as needed to make this not a regression
>+        https://bugs.webkit.org/show_bug.cgi?id=160125
>+
>+        Reviewed by Geoffrey Garen.
>+
>+        * DumpRenderTree/TestRunner.cpp: Rewire some #includes.
>+        * Scripts/run-jsc-stress-tests: New test flag!
>+
> 2016-08-22  Simon Fraser  <simon.fraser@apple.com>
> 
>         <select> menu on iPad causes shifting of hit-testing areas
>Index: Tools/DumpRenderTree/TestRunner.cpp
>===================================================================
>--- Tools/DumpRenderTree/TestRunner.cpp	(revision 204735)
>+++ Tools/DumpRenderTree/TestRunner.cpp	(working copy)
>@@ -34,6 +34,7 @@
> #include "WorkQueue.h"
> #include "WorkQueueItem.h"
> #include <JavaScriptCore/APICast.h>
>+#include <JavaScriptCore/HeapInlines.h>
> #include <JavaScriptCore/JSContextRef.h>
> #include <JavaScriptCore/JSCTestRunnerUtils.h>
> #include <JavaScriptCore/JSObjectRef.h>
>Index: Tools/DumpRenderTree/mac/DumpRenderTree.mm
>===================================================================
>--- Tools/DumpRenderTree/mac/DumpRenderTree.mm	(revision 204735)
>+++ Tools/DumpRenderTree/mac/DumpRenderTree.mm	(working copy)
>@@ -56,9 +56,7 @@
> #import "WorkQueue.h"
> #import "WorkQueueItem.h"
> #import <CoreFoundation/CoreFoundation.h>
>-#import <JavaScriptCore/HeapStatistics.h>
>-#import <JavaScriptCore/LLIntData.h>
>-#import <JavaScriptCore/Options.h>
>+#import <JavaScriptCore/TestRunnerUtils.h>
> #import <WebCore/LogInitialization.h>
> #import <WebKit/DOMElement.h>
> #import <WebKit/DOMExtensions.h>
>@@ -1429,10 +1427,7 @@ int DumpRenderTreeMain(int argc, const c
> #endif
>     [WebCoreStatistics garbageCollectJavaScriptObjects];
>     [WebCoreStatistics emptyCache]; // Otherwise SVGImages trigger false positives for Frame/Node counts
>-    if (JSC::Options::logHeapStatisticsAtExit())
>-        JSC::HeapStatistics::reportSuccess();
>-    if (JSC::Options::reportLLIntStats())
>-        JSC::LLInt::Data::finalizeStats();
>+    JSC::finalizeStatsAtEndOfTesting();
>     [pool release];
>     returningFromMain = true;
>     return 0;
>Index: Tools/Scripts/run-jsc-stress-tests
>===================================================================
>--- Tools/Scripts/run-jsc-stress-tests	(revision 204735)
>+++ Tools/Scripts/run-jsc-stress-tests	(working copy)
>@@ -420,7 +420,7 @@ $numPasses = 0
> # We force all tests to use a smaller (1.5M) stack so that stack overflow tests can run faster.
> BASE_OPTIONS = ["--useFTLJIT=false", "--useFunctionDotArguments=true", "--maxPerThreadStackUsage=1572864"]
> EAGER_OPTIONS = ["--thresholdForJITAfterWarmUp=10", "--thresholdForJITSoon=10", "--thresholdForOptimizeAfterWarmUp=20", "--thresholdForOptimizeAfterLongWarmUp=20", "--thresholdForOptimizeSoon=20", "--thresholdForFTLOptimizeAfterWarmUp=20", "--thresholdForFTLOptimizeSoon=20", "--maximumEvalCacheableSourceLength=150000", "--useEagerCodeBlockJettisonTiming=true"]
>-NO_CJIT_OPTIONS = ["--useConcurrentJIT=false", "--thresholdForJITAfterWarmUp=100"]
>+NO_CJIT_OPTIONS = ["--useConcurrentJIT=false", "--thresholdForJITAfterWarmUp=100", "--scribbleFreeCells=true"]
> FTL_OPTIONS = ["--useFTLJIT=true"]
> 
> $runlist = []
>Index: Tools/TestWebKitAPI/Tests/WTF/Vector.cpp
>===================================================================
>--- Tools/TestWebKitAPI/Tests/WTF/Vector.cpp	(revision 204735)
>+++ Tools/TestWebKitAPI/Tests/WTF/Vector.cpp	(working copy)
>@@ -66,8 +66,7 @@ TEST(WTF_Vector, Iterator)
> TEST(WTF_Vector, OverloadedOperatorAmpersand)
> {
>     struct Test {
>-    private:
>-        Test* operator&();
>+        Test* operator&() = delete;
>     };
> 
>     Vector<Test> vector;

Flags:

buildbot: commit-queue-

Actions: View | Formatted Diff | Diff

Attachments on bug 160125: 284422 | 284428 | 284455 | 284506 | 284540 | 284543 | 284550 | 284621 | 284635 | 284642 | 284705 | 284709 | 284728 | 284788 | 284814 | 284889 | 284898 | 284899 | 284920 | 284951 | 284959 | 284976 | 284977 | 285168 | 285169 | 285170 | 285254 | 285260 | 285349 | 285396 | 285399 | 285401 | 285444 | 285450 | 285467 | 285481 | 285575 | 285600 | 285601 | 285623 | 285625 | 285630 | 285665 | 285673 | 285678 | 285706 | 285746 | 285804 | 285893 | 285935 | 285950 | 286010 | 286011 | 286012 | 286013 | 286014 | 286015 | 286016 | 286017 | 286033 | 286034 | 286036 | 286048 | 286050 | 286063 | 286067 | 286069 | 286074 | 286087 | 286195 | 286196 | 286351 | 286354 | 286356 | 286359 | 286362 | 286617 | 286626 | 286627 | 286631 | 286633 | 286640 | 286754 | 287135 | 287156 | 287159 | 287178 | 287186 | 287244 | 287346 | 287454 | 287458 | 287528 | 287538 | 287552 | 287595 | 287629 | 287660 | 287666 | 287718 | 287723 | 287928 | 287970 | 287982 | 287985 | 287987