WebKit Bugzilla
Attachment 361854 Details for
Bug 194375
: [JSC] sizeof(JSString) should be 16
Home
|
New
|
Browse
|
Search
|
[?]
|
Reports
|
Requests
|
Help
|
New Account
|
Log In
Remember
[x]
|
Forgot Password
Login:
[x]
[patch]
Fix more
bug-194375-20190212153411.patch (text/plain), 119.57 KB, created by
Yusuke Suzuki
on 2019-02-12 15:34:11 PST
(
hide
)
Description:
Fix more
Filename:
MIME Type:
Creator:
Yusuke Suzuki
Created:
2019-02-12 15:34:11 PST
Size:
119.57 KB
patch
obsolete
>Subversion Revision: 241291 >diff --git a/Source/JavaScriptCore/ChangeLog b/Source/JavaScriptCore/ChangeLog >index 608249f7f276994737bd7817a105f0853a0dde39..3a7df08d02c37a0efcfadb459ff5d58cad24cbed 100644 >--- a/Source/JavaScriptCore/ChangeLog >+++ b/Source/JavaScriptCore/ChangeLog >@@ -1,3 +1,104 @@ >+2019-02-12 Yusuke Suzuki <ysuzuki@apple.com> >+ >+ [JSC] sizeof(JSString) should be 16 >+ https://bugs.webkit.org/show_bug.cgi?id=194375 >+ >+ Reviewed by NOBODY (OOPS!). >+ >+ * bytecode/AccessCase.cpp: >+ (JSC::AccessCase::generateImpl): >+ * bytecode/InlineAccess.cpp: >+ (JSC::InlineAccess::dumpCacheSizesAndCrash): >+ (JSC::InlineAccess::generateStringLength): >+ * dfg/DFGOperations.cpp: >+ * dfg/DFGOperations.h: >+ * dfg/DFGSpeculativeJIT.cpp: >+ (JSC::DFG::SpeculativeJIT::compileStringSlice): >+ (JSC::DFG::SpeculativeJIT::compileToLowerCase): >+ (JSC::DFG::SpeculativeJIT::compileGetCharCodeAt): >+ (JSC::DFG::SpeculativeJIT::compileGetByValOnString): >+ (JSC::DFG::SpeculativeJIT::compileMakeRope): >+ (JSC::DFG::SpeculativeJIT::compileStringEquality): >+ (JSC::DFG::SpeculativeJIT::compileStringZeroLength): >+ (JSC::DFG::SpeculativeJIT::compileLogicalNotStringOrOther): >+ (JSC::DFG::SpeculativeJIT::emitStringBranch): >+ (JSC::DFG::SpeculativeJIT::emitStringOrOtherBranch): >+ (JSC::DFG::SpeculativeJIT::compileGetIndexedPropertyStorage): >+ (JSC::DFG::SpeculativeJIT::compileGetArrayLength): >+ (JSC::DFG::SpeculativeJIT::speculateStringIdentAndLoadStorage): >+ (JSC::DFG::SpeculativeJIT::emitSwitchCharStringJump): >+ (JSC::DFG::SpeculativeJIT::emitSwitchStringOnString): >+ * dfg/DFGSpeculativeJIT32_64.cpp: >+ (JSC::DFG::SpeculativeJIT::compile): >+ * dfg/DFGSpeculativeJIT64.cpp: >+ (JSC::DFG::SpeculativeJIT::compile): >+ * ftl/FTLAbstractHeapRepository.cpp: >+ (JSC::FTL::AbstractHeapRepository::AbstractHeapRepository): >+ * ftl/FTLAbstractHeapRepository.h: >+ * ftl/FTLLowerDFGToB3.cpp: >+ (JSC::FTL::DFG::LowerDFGToB3::compileGetIndexedPropertyStorage): >+ (JSC::FTL::DFG::LowerDFGToB3::compileGetArrayLength): >+ (JSC::FTL::DFG::LowerDFGToB3::compileMakeRope): >+ (JSC::FTL::DFG::LowerDFGToB3::compileStringCharAt): >+ (JSC::FTL::DFG::LowerDFGToB3::compileStringCharCodeAt): >+ (JSC::FTL::DFG::LowerDFGToB3::compileCompareStrictEq): >+ (JSC::FTL::DFG::LowerDFGToB3::compileStringToUntypedStrictEquality): >+ (JSC::FTL::DFG::LowerDFGToB3::compileSwitch): >+ (JSC::FTL::DFG::LowerDFGToB3::mapHashString): >+ (JSC::FTL::DFG::LowerDFGToB3::compileMapHash): >+ (JSC::FTL::DFG::LowerDFGToB3::compileHasOwnProperty): >+ (JSC::FTL::DFG::LowerDFGToB3::compileStringSlice): >+ (JSC::FTL::DFG::LowerDFGToB3::compileToLowerCase): >+ (JSC::FTL::DFG::LowerDFGToB3::stringsEqual): >+ (JSC::FTL::DFG::LowerDFGToB3::boolify): >+ (JSC::FTL::DFG::LowerDFGToB3::switchString): >+ (JSC::FTL::DFG::LowerDFGToB3::isRopeString): >+ (JSC::FTL::DFG::LowerDFGToB3::isNotRopeString): >+ (JSC::FTL::DFG::LowerDFGToB3::speculateStringIdent): >+ * jit/AssemblyHelpers.cpp: >+ (JSC::AssemblyHelpers::emitConvertValueToBoolean): >+ (JSC::AssemblyHelpers::branchIfValue): >+ * jit/AssemblyHelpers.h: >+ (JSC::AssemblyHelpers::branchIfRopeString): >+ (JSC::AssemblyHelpers::branchIfNotRope): >+ * jit/JITInlines.h: >+ (JSC::JIT::emitLoadCharacterString): >+ * jit/ThunkGenerators.cpp: >+ (JSC::stringGetByValGenerator): >+ (JSC::stringCharLoad): >+ * llint/LowLevelInterpreter.asm: >+ * llint/LowLevelInterpreter32_64.asm: >+ * llint/LowLevelInterpreter64.asm: >+ * runtime/JSString.cpp: >+ (JSC::JSString::dumpToStream): >+ (JSC::JSString::estimatedSize): >+ (JSC::JSString::visitChildren): >+ (JSC::JSRopeString::resolveRopeToAtomicString const): >+ (JSC::JSRopeString::convertToNonRope const): >+ (JSC::JSRopeString::resolveRopeToExistingAtomicString const): >+ (JSC::JSRopeString::resolveRope const): >+ (JSC::JSRopeString::resolveRopeSlowCase8 const): >+ (JSC::JSRopeString::resolveRopeSlowCase const): >+ (JSC::JSRopeString::outOfMemory const): >+ (JSC::JSRopeString::visitFibers): Deleted. >+ (JSC::JSRopeString::clearFibers const): Deleted. >+ * runtime/JSString.h: >+ (JSC::JSString::JSString): >+ (JSC::JSString::finishCreation): >+ (JSC::JSString::offsetOfValue): >+ (JSC::JSString::isRope const): >+ (JSC::JSString::isRopeConcurrently const): >+ (JSC::JSString::is8Bit const): >+ (JSC::JSString::length const): >+ (JSC::JSString::tryGetValueImpl const): >+ (JSC::JSString::offsetOfLength): Deleted. >+ (JSC::JSString::offsetOfFlags): Deleted. >+ (JSC::JSString::setIs8Bit const): Deleted. >+ (JSC::JSString::setLength): Deleted. >+ (JSC::JSString::string): Deleted. >+ * runtime/JSStringInlines.h: >+ (JSC::JSString::~JSString): >+ > 2019-02-11 Mark Lam <mark.lam@apple.com> > > Randomize insertion of deallocated StructureIDs into the StructureIDTable's free list. >diff --git a/Source/WTF/ChangeLog b/Source/WTF/ChangeLog >index da5eece0d14cf1df21135d43f7d08485470f613b..c8cfdf5f5d7a5debcc1b467137f67e8f1a21980c 100644 >--- a/Source/WTF/ChangeLog >+++ b/Source/WTF/ChangeLog >@@ -1,3 +1,16 @@ >+2019-02-12 Yusuke Suzuki <ysuzuki@apple.com> >+ >+ [JSC] sizeof(JSString) should be 16 >+ https://bugs.webkit.org/show_bug.cgi?id=194375 >+ >+ Reviewed by NOBODY (OOPS!). >+ >+ * wtf/text/StringImpl.h: >+ (WTF::StringImpl::flagIs8Bit): >+ (WTF::StringImpl::flagIsAtomic): >+ (WTF::StringImpl::flagIsSymbol): >+ (WTF::StringImpl::maskStringKind): >+ > 2019-02-11 Myles C. Maxfield <mmaxfield@apple.com> > > [Cocoa] Ask platform for generic font family mappings >diff --git a/Source/JavaScriptCore/bytecode/AccessCase.cpp b/Source/JavaScriptCore/bytecode/AccessCase.cpp >index 9e6ec7b063ceaa26b2b278dc17a8f21bb8cd3c5f..f546802ebca2bd4afe652d3ea0900bf4dc437307 100644 >--- a/Source/JavaScriptCore/bytecode/AccessCase.cpp >+++ b/Source/JavaScriptCore/bytecode/AccessCase.cpp >@@ -49,6 +49,7 @@ > #include "StructureStubInfo.h" > #include "SuperSampler.h" > #include "ThunkGenerators.h" >+#include "ProbeContext.h" > > namespace JSC { > >@@ -1214,7 +1215,18 @@ void AccessCase::generateImpl(AccessGenerationState& state) > } > > case StringLength: { >- jit.load32(CCallHelpers::Address(baseGPR, JSString::offsetOfLength()), valueRegs.payloadGPR()); >+ auto isRope = jit.branchIfRopeString(baseGPR); >+ jit.loadPtr(CCallHelpers::Address(baseGPR, JSString::offsetOfValue()), valueRegs.payloadGPR()); >+ jit.load32(CCallHelpers::Address(valueRegs.payloadGPR(), StringImpl::lengthMemoryOffset()), valueRegs.payloadGPR()); >+ auto done = jit.jump(); >+ >+ isRope.link(&jit); >+ jit.load16(CCallHelpers::Address(baseGPR, JSRopeString::offsetOfUpperLength()), scratchGPR); >+ jit.load16(CCallHelpers::Address(baseGPR, JSRopeString::offsetOfLowerLength()), valueRegs.payloadGPR()); >+ jit.lshift32(CCallHelpers::TrustedImm32(16), scratchGPR); >+ jit.or32(scratchGPR, valueRegs.payloadGPR()); >+ >+ done.link(&jit); > jit.boxInt32(valueRegs.payloadGPR(), valueRegs); > state.succeed(); > return; >diff --git a/Source/JavaScriptCore/bytecode/InlineAccess.cpp b/Source/JavaScriptCore/bytecode/InlineAccess.cpp >index b7efc375c52458f77357d9e1667d1fbc56eb65b8..0cc283a24658ff9731f4ec066ece68c814cc479d 100644 >--- a/Source/JavaScriptCore/bytecode/InlineAccess.cpp >+++ b/Source/JavaScriptCore/bytecode/InlineAccess.cpp >@@ -50,11 +50,24 @@ void InlineAccess::dumpCacheSizesAndCrash() > { > CCallHelpers jit; > >+ GPRReg scratchGPR = value; > jit.patchableBranch8( > CCallHelpers::NotEqual, > CCallHelpers::Address(base, JSCell::typeInfoTypeOffset()), > CCallHelpers::TrustedImm32(StringType)); >- jit.load32(CCallHelpers::Address(base, JSString::offsetOfLength()), regs.payloadGPR()); >+ >+ auto isRope = jit.branchIfRopeString(base); >+ jit.loadPtr(CCallHelpers::Address(base, JSString::offsetOfValue()), regs.payloadGPR()); >+ jit.load32(CCallHelpers::Address(regs.payloadGPR(), StringImpl::lengthMemoryOffset()), regs.payloadGPR()); >+ auto done = jit.jump(); >+ >+ isRope.link(&jit); >+ jit.load16(CCallHelpers::Address(base, JSRopeString::offsetOfUpperLength()), scratchGPR); >+ jit.load16(CCallHelpers::Address(base, JSRopeString::offsetOfLowerLength()), regs.payloadGPR()); >+ jit.lshift32(CCallHelpers::TrustedImm32(16), scratchGPR); >+ jit.or32(scratchGPR, regs.payloadGPR()); >+ >+ done.link(&jit); > jit.boxInt32(regs.payloadGPR(), regs); > > dataLog("string length size: ", jit.m_assembler.buffer().codeSize(), "\n"); >@@ -294,12 +307,25 @@ bool InlineAccess::generateStringLength(StructureStubInfo& stubInfo) > > GPRReg base = stubInfo.baseGPR(); > JSValueRegs value = stubInfo.valueRegs(); >+ GPRReg scratch = getScratchRegister(stubInfo); > > auto branchToSlowPath = jit.patchableBranch8( > CCallHelpers::NotEqual, > CCallHelpers::Address(base, JSCell::typeInfoTypeOffset()), > CCallHelpers::TrustedImm32(StringType)); >- jit.load32(CCallHelpers::Address(base, JSString::offsetOfLength()), value.payloadGPR()); >+ >+ auto isRope = jit.branchIfRopeString(base); >+ jit.loadPtr(CCallHelpers::Address(base, JSString::offsetOfValue()), value.payloadGPR()); >+ jit.load32(CCallHelpers::Address(value.payloadGPR(), StringImpl::lengthMemoryOffset()), value.payloadGPR()); >+ auto done = jit.jump(); >+ >+ isRope.link(&jit); >+ jit.load16(CCallHelpers::Address(base, JSRopeString::offsetOfUpperLength()), scratch); >+ jit.load16(CCallHelpers::Address(base, JSRopeString::offsetOfLowerLength()), value.payloadGPR()); >+ jit.lshift32(CCallHelpers::TrustedImm32(16), scratch); >+ jit.or32(scratch, value.payloadGPR()); >+ >+ done.link(&jit); > jit.boxInt32(value.payloadGPR(), value); > > bool linkedCodeInline = linkCodeInline("string length", jit, stubInfo, [&] (LinkBuffer& linkBuffer) { >diff --git a/Source/JavaScriptCore/dfg/DFGOperations.cpp b/Source/JavaScriptCore/dfg/DFGOperations.cpp >index 984fc0b961e500954d6df0c4a9d3a677749eaaff..fcfaffacffba01859a41cf23c0ffefe4906e3957 100644 >--- a/Source/JavaScriptCore/dfg/DFGOperations.cpp >+++ b/Source/JavaScriptCore/dfg/DFGOperations.cpp >@@ -2162,6 +2162,32 @@ JSCell* JIT_OPERATION operationStringSubstr(ExecState* exec, JSCell* cell, int32 > return jsSubstring(exec, string, from, span); > } > >+JSCell* JIT_OPERATION operationStringSlice(ExecState* exec, JSCell* cell, int32_t start, int32_t end) >+{ >+ VM& vm = exec->vm(); >+ NativeCallFrameTracer tracer(&vm, exec); >+ auto scope = DECLARE_THROW_SCOPE(vm); >+ >+ auto string = jsCast<JSString*>(cell)->value(exec); >+ RETURN_IF_EXCEPTION(scope, nullptr); >+ >+ int32_t length = string.length(); >+ static_assert(static_cast<uint64_t>(JSString::MaxLength) <= static_cast<uint64_t>(std::numeric_limits<int32_t>::max()), ""); >+ int32_t from = start < 0 ? length + start : start; >+ int32_t to = end < 0 ? length + end : end; >+ if (to > from && to > 0 && from < length) { >+ if (from < 0) >+ from = 0; >+ if (to > length) >+ to = length; >+ scope.release(); >+ return jsSubstring(exec, string, static_cast<unsigned>(from), static_cast<unsigned>(to) - static_cast<unsigned>(from)); >+ } >+ >+ scope.release(); >+ return jsEmptyString(exec); >+} >+ > JSString* JIT_OPERATION operationToLowerCase(ExecState* exec, JSString* string, uint32_t failingIndex) > { > VM& vm = exec->vm(); >diff --git a/Source/JavaScriptCore/dfg/DFGOperations.h b/Source/JavaScriptCore/dfg/DFGOperations.h >index 38c116a54f91a615b81c15112148f25f94adb1ef..0cddc824b23ed96c9fc5694c35334de60e3ae984 100644 >--- a/Source/JavaScriptCore/dfg/DFGOperations.h >+++ b/Source/JavaScriptCore/dfg/DFGOperations.h >@@ -203,6 +203,7 @@ StringImpl* JIT_OPERATION operationResolveRope(ExecState*, JSString*); > JSString* JIT_OPERATION operationSingleCharacterString(ExecState*, int32_t); > > JSCell* JIT_OPERATION operationStringSubstr(ExecState*, JSCell*, int32_t, int32_t); >+JSCell* JIT_OPERATION operationStringSlice(ExecState* exec, JSCell* cell, int32_t, int32_t); > JSString* JIT_OPERATION operationStringValueOf(ExecState*, EncodedJSValue); > JSString* JIT_OPERATION operationToLowerCase(ExecState*, JSString*, uint32_t); > >diff --git a/Source/JavaScriptCore/dfg/DFGSpeculativeJIT.cpp b/Source/JavaScriptCore/dfg/DFGSpeculativeJIT.cpp >index 2911288ede8d5699066d0d11ff2b7fd4b29de121..6b18e1280bdae8a5d5d3127e8e99635bf37ec3fd 100644 >--- a/Source/JavaScriptCore/dfg/DFGSpeculativeJIT.cpp >+++ b/Source/JavaScriptCore/dfg/DFGSpeculativeJIT.cpp >@@ -58,6 +58,7 @@ > #include "JSLexicalEnvironment.h" > #include "JSPropertyNameEnumerator.h" > #include "LinkBuffer.h" >+#include "ProbeContext.h" > #include "RegExpObject.h" > #include "ScopedArguments.h" > #include "ScratchRegisterAllocator.h" >@@ -1530,29 +1531,84 @@ void SpeculativeJIT::compilePeepHoleBooleanBranch(Node* node, Node* branchNode, > void SpeculativeJIT::compileStringSlice(Node* node) > { > SpeculateCellOperand string(this, node->child1()); >- GPRTemporary startIndex(this); >+ >+ GPRReg stringGPR = string.gpr(); >+ >+ speculateString(node->child1(), stringGPR); >+ >+ SpeculateInt32Operand start(this, node->child2()); >+ GPRReg startGPR = start.gpr(); >+ >+ Optional<SpeculateInt32Operand> end; >+ Optional<GPRReg> endGPR; >+ if (node->child3()) { >+ end.emplace(this, node->child3()); >+ endGPR.emplace(end->gpr()); >+ } >+ >+ auto isRope = m_jit.branchIfRopeString(stringGPR); >+ >+ auto emitPopulateSliceIndex = [&] (Edge& target, GPRReg indexGPR, GPRReg lengthGPR, GPRReg resultGPR) { >+ if (target->isInt32Constant()) { >+ int32_t value = target->asInt32(); >+ if (value == 0) { >+ m_jit.move(TrustedImm32(0), resultGPR); >+ return; >+ } >+ >+ MacroAssembler::JumpList done; >+ if (value > 0) { >+ m_jit.move(TrustedImm32(value), resultGPR); >+ done.append(m_jit.branch32(MacroAssembler::BelowOrEqual, resultGPR, lengthGPR)); >+ m_jit.move(lengthGPR, resultGPR); >+ } else { >+ ASSERT(value != 0); >+ m_jit.move(lengthGPR, resultGPR); >+ done.append(m_jit.branchAdd32(MacroAssembler::PositiveOrZero, TrustedImm32(value), resultGPR)); >+ m_jit.move(TrustedImm32(0), resultGPR); >+ } >+ done.link(&m_jit); >+ return; >+ } >+ >+ MacroAssembler::JumpList done; >+ >+ auto isPositive = m_jit.branch32(MacroAssembler::GreaterThanOrEqual, indexGPR, TrustedImm32(0)); >+ m_jit.move(lengthGPR, resultGPR); >+ done.append(m_jit.branchAdd32(MacroAssembler::PositiveOrZero, indexGPR, resultGPR)); >+ m_jit.move(TrustedImm32(0), resultGPR); >+ done.append(m_jit.jump()); >+ >+ isPositive.link(&m_jit); >+ m_jit.move(indexGPR, resultGPR); >+ done.append(m_jit.branch32(MacroAssembler::BelowOrEqual, resultGPR, lengthGPR)); >+ m_jit.move(lengthGPR, resultGPR); >+ >+ done.link(&m_jit); >+ }; >+ > GPRTemporary temp(this); > GPRTemporary temp2(this); >+ GPRTemporary startIndex(this); > >- GPRReg stringGPR = string.gpr(); >- GPRReg startIndexGPR = startIndex.gpr(); > GPRReg tempGPR = temp.gpr(); > GPRReg temp2GPR = temp2.gpr(); >- >- speculateString(node->child1(), stringGPR); >+ GPRReg startIndexGPR = startIndex.gpr(); > > { >- m_jit.load32(JITCompiler::Address(stringGPR, JSString::offsetOfLength()), temp2GPR); >+ m_jit.loadPtr(JITCompiler::Address(stringGPR, JSString::offsetOfValue()), tempGPR); >+ m_jit.load32(MacroAssembler::Address(tempGPR, StringImpl::lengthMemoryOffset()), temp2GPR); >+ >+ emitPopulateSliceIndex(node->child2(), startGPR, temp2GPR, startIndexGPR); > >- emitPopulateSliceIndex(node->child2(), temp2GPR, startIndexGPR); > if (node->child3()) >- emitPopulateSliceIndex(node->child3(), temp2GPR, tempGPR); >+ emitPopulateSliceIndex(node->child3(), endGPR.value(), temp2GPR, tempGPR); > else > m_jit.move(temp2GPR, tempGPR); > } > > CCallHelpers::JumpList doneCases; >- CCallHelpers::JumpList slowCases; >+ JITCompiler::JumpList slowCases; > > auto nonEmptyCase = m_jit.branch32(MacroAssembler::Below, startIndexGPR, tempGPR); > m_jit.move(TrustedImmPtr::weakPointer(m_jit.graph(), jsEmptyString(&vm())), tempGPR); >@@ -1562,9 +1618,8 @@ void SpeculativeJIT::compileStringSlice(Node* node) > m_jit.sub32(startIndexGPR, tempGPR); // the size of the sliced string. > slowCases.append(m_jit.branch32(MacroAssembler::NotEqual, tempGPR, TrustedImm32(1))); > >+ // Refill StringImpl* here. > m_jit.loadPtr(MacroAssembler::Address(stringGPR, JSString::offsetOfValue()), temp2GPR); >- slowCases.append(m_jit.branchTestPtr(MacroAssembler::Zero, temp2GPR)); >- > m_jit.loadPtr(MacroAssembler::Address(temp2GPR, StringImpl::dataOffset()), tempGPR); > > // Load the character into scratchReg >@@ -1586,13 +1641,14 @@ void SpeculativeJIT::compileStringSlice(Node* node) > m_jit.addPtr(TrustedImmPtr(m_jit.vm()->smallStrings.singleCharacterStrings()), tempGPR); > m_jit.loadPtr(tempGPR, tempGPR); > >- addSlowPathGenerator( >- slowPathCall( >- bigCharacter, this, operationSingleCharacterString, tempGPR, tempGPR)); >+ addSlowPathGenerator(slowPathCall(bigCharacter, this, operationSingleCharacterString, tempGPR, tempGPR)); > >- addSlowPathGenerator( >- slowPathCall( >- slowCases, this, operationStringSubstr, tempGPR, stringGPR, startIndexGPR, tempGPR)); >+ addSlowPathGenerator(slowPathCall(slowCases, this, operationStringSubstr, tempGPR, stringGPR, startIndexGPR, tempGPR)); >+ >+ if (endGPR) >+ addSlowPathGenerator(slowPathCall(isRope, this, operationStringSlice, tempGPR, stringGPR, startGPR, *endGPR)); >+ else >+ addSlowPathGenerator(slowPathCall(isRope, this, operationStringSlice, tempGPR, stringGPR, startGPR, TrustedImm32(std::numeric_limits<int32_t>::max()))); > > doneCases.link(&m_jit); > cellResult(tempGPR, node); >@@ -1619,8 +1675,8 @@ void SpeculativeJIT::compileToLowerCase(Node* node) > > m_jit.move(TrustedImmPtr(nullptr), indexGPR); > >+ slowPath.append(m_jit.branchIfRopeString(stringGPR)); > m_jit.loadPtr(MacroAssembler::Address(stringGPR, JSString::offsetOfValue()), tempGPR); >- slowPath.append(m_jit.branchTestPtr(MacroAssembler::Zero, tempGPR)); > > slowPath.append(m_jit.branchTest32( > MacroAssembler::Zero, MacroAssembler::Address(tempGPR, StringImpl::flagsOffset()), >@@ -2128,13 +2184,13 @@ void SpeculativeJIT::compileGetCharCodeAt(Node* node) > > ASSERT(speculationChecked(m_state.forNode(node->child1()).m_type, SpecString)); > >- // unsigned comparison so we can filter out negative indices and indices that are too large >- speculationCheck(Uncountable, JSValueRegs(), 0, m_jit.branch32(MacroAssembler::AboveOrEqual, indexReg, MacroAssembler::Address(stringReg, JSString::offsetOfLength()))); >- > GPRTemporary scratch(this); > GPRReg scratchReg = scratch.gpr(); > >+ // unsigned comparison so we can filter out negative indices and indices that are too large > m_jit.loadPtr(MacroAssembler::Address(stringReg, JSString::offsetOfValue()), scratchReg); >+ >+ speculationCheck(Uncountable, JSValueRegs(), 0, m_jit.branch32(MacroAssembler::AboveOrEqual, indexReg, CCallHelpers::Address(scratchReg, StringImpl::lengthMemoryOffset()))); > > // Load the character into scratchReg > JITCompiler::Jump is16Bit = m_jit.branchTest32(MacroAssembler::Zero, MacroAssembler::Address(scratchReg, StringImpl::flagsOffset()), TrustedImm32(StringImpl::flagIs8Bit())); >@@ -2175,14 +2231,13 @@ void SpeculativeJIT::compileGetByValOnString(Node* node) > ASSERT(ArrayMode(Array::String, Array::Read).alreadyChecked(m_jit.graph(), node, m_state.forNode(m_graph.child(node, 0)))); > > // unsigned comparison so we can filter out negative indices and indices that are too large >+ m_jit.loadPtr(MacroAssembler::Address(baseReg, JSString::offsetOfValue()), scratchReg); > JITCompiler::Jump outOfBounds = m_jit.branch32( > MacroAssembler::AboveOrEqual, propertyReg, >- MacroAssembler::Address(baseReg, JSString::offsetOfLength())); >+ MacroAssembler::Address(scratchReg, StringImpl::lengthMemoryOffset())); > if (node->arrayMode().isInBounds()) > speculationCheck(OutOfBounds, JSValueRegs(), 0, outOfBounds); > >- m_jit.loadPtr(MacroAssembler::Address(baseReg, JSString::offsetOfValue()), scratchReg); >- > // Load the character into scratchReg > JITCompiler::Jump is16Bit = m_jit.branchTest32(MacroAssembler::Zero, MacroAssembler::Address(scratchReg, StringImpl::flagsOffset()), TrustedImm32(StringImpl::flagIs8Bit())); > >@@ -4342,6 +4397,7 @@ void SpeculativeJIT::compileMakeRope(Node* node) > GPRTemporary result(this); > GPRTemporary allocator(this); > GPRTemporary scratch(this); >+ GPRTemporary scratch2(this); > > GPRReg opGPRs[3]; > unsigned numOpGPRs; >@@ -4357,41 +4413,81 @@ void SpeculativeJIT::compileMakeRope(Node* node) > GPRReg resultGPR = result.gpr(); > GPRReg allocatorGPR = allocator.gpr(); > GPRReg scratchGPR = scratch.gpr(); >+ GPRReg scratch2GPR = scratch2.gpr(); > > JITCompiler::JumpList slowPath; > Allocator allocatorValue = allocatorForNonVirtualConcurrently<JSRopeString>(*m_jit.vm(), sizeof(JSRopeString), AllocatorForMode::AllocatorIfExists); > emitAllocateJSCell(resultGPR, JITAllocator::constant(allocatorValue), allocatorGPR, TrustedImmPtr(m_jit.graph().registerStructure(m_jit.vm()->stringStructure.get())), scratchGPR, slowPath); >- >- m_jit.storePtr(TrustedImmPtr(nullptr), JITCompiler::Address(resultGPR, JSString::offsetOfValue())); >+ >+ StructureIDBlob ropeBlob(m_graph.stringStructure->id(), m_graph.stringStructure->indexingModeIncludingHistory(), TypeInfo(StringType, JSRopeString::StructureFlags | TypeInfoPerCellBit)); >+ m_jit.store64(TrustedImm64(ropeBlob.blob()), MacroAssembler::Address(resultGPR, JSCell::structureIDOffset())); >+ > for (unsigned i = 0; i < numOpGPRs; ++i) > m_jit.storePtr(opGPRs[i], JITCompiler::Address(resultGPR, JSRopeString::offsetOfFibers() + sizeof(WriteBarrier<JSString>) * i)); > for (unsigned i = numOpGPRs; i < JSRopeString::s_maxInternalRopeLength; ++i) > m_jit.storePtr(TrustedImmPtr(nullptr), JITCompiler::Address(resultGPR, JSRopeString::offsetOfFibers() + sizeof(WriteBarrier<JSString>) * i)); >- m_jit.load16(JITCompiler::Address(opGPRs[0], JSString::offsetOfFlags()), scratchGPR); >- m_jit.load32(JITCompiler::Address(opGPRs[0], JSString::offsetOfLength()), allocatorGPR); >- if (!ASSERT_DISABLED) { >- JITCompiler::Jump ok = m_jit.branch32( >- JITCompiler::GreaterThanOrEqual, allocatorGPR, TrustedImm32(0)); >- m_jit.abortWithReason(DFGNegativeStringLength); >- ok.link(&m_jit); >+ >+ { >+ auto isRope = m_jit.branchIfRopeString(opGPRs[0]); >+ m_jit.loadPtr(JITCompiler::Address(opGPRs[0], JSString::offsetOfValue()), scratch2GPR); >+ m_jit.load32(JITCompiler::Address(scratch2GPR, StringImpl::flagsOffset()), scratchGPR); >+ m_jit.load32(JITCompiler::Address(scratch2GPR, StringImpl::lengthMemoryOffset()), allocatorGPR); >+ auto done = m_jit.jump(); >+ >+ isRope.link(&m_jit); >+ m_jit.load32(JITCompiler::Address(opGPRs[0], JSRopeString::offsetOfFlags()), scratchGPR); >+ m_jit.load16(CCallHelpers::Address(opGPRs[0], JSRopeString::offsetOfUpperLength()), scratch2GPR); >+ m_jit.load16(CCallHelpers::Address(opGPRs[0], JSRopeString::offsetOfLowerLength()), allocatorGPR); >+ m_jit.lshift32(CCallHelpers::TrustedImm32(16), scratch2GPR); >+ m_jit.or32(scratch2GPR, allocatorGPR); >+ >+ done.link(&m_jit); >+ >+ if (!ASSERT_DISABLED) { >+ JITCompiler::Jump ok = m_jit.branch32( >+ JITCompiler::GreaterThanOrEqual, allocatorGPR, TrustedImm32(0)); >+ m_jit.abortWithReason(DFGNegativeStringLength); >+ ok.link(&m_jit); >+ } > } >+ > for (unsigned i = 1; i < numOpGPRs; ++i) { >- m_jit.and16(JITCompiler::Address(opGPRs[i], JSString::offsetOfFlags()), scratchGPR); >+ auto isRope = m_jit.branchIfRopeString(opGPRs[i]); >+ m_jit.loadPtr(JITCompiler::Address(opGPRs[i], JSString::offsetOfValue()), scratch2GPR); >+ m_jit.and32(JITCompiler::Address(scratch2GPR, StringImpl::flagsOffset()), scratchGPR); > speculationCheck( > Uncountable, JSValueSource(), nullptr, > m_jit.branchAdd32( > JITCompiler::Overflow, >- JITCompiler::Address(opGPRs[i], JSString::offsetOfLength()), allocatorGPR)); >+ JITCompiler::Address(scratch2GPR, StringImpl::lengthMemoryOffset()), allocatorGPR)); >+ auto done = m_jit.jump(); >+ >+ isRope.link(&m_jit); >+ m_jit.and32(JITCompiler::Address(opGPRs[i], JSRopeString::offsetOfFlags()), scratchGPR); >+ m_jit.load16(CCallHelpers::Address(opGPRs[i], JSRopeString::offsetOfUpperLength()), scratch2GPR); >+ m_jit.lshift32(CCallHelpers::TrustedImm32(16), scratch2GPR); >+ speculationCheck( >+ Uncountable, JSValueSource(), nullptr, >+ m_jit.branchAdd32( >+ JITCompiler::Overflow, scratch2GPR, allocatorGPR)); >+ m_jit.load16(CCallHelpers::Address(opGPRs[i], JSRopeString::offsetOfLowerLength()), scratch2GPR); >+ speculationCheck( >+ Uncountable, JSValueSource(), nullptr, >+ m_jit.branchAdd32( >+ JITCompiler::Overflow, scratch2GPR, allocatorGPR)); >+ done.link(&m_jit); > } >- m_jit.and32(JITCompiler::TrustedImm32(JSString::Is8Bit), scratchGPR); >- m_jit.store16(scratchGPR, JITCompiler::Address(resultGPR, JSString::offsetOfFlags())); >+ m_jit.and32(JITCompiler::TrustedImm32(JSRopeString::Is8Bit), scratchGPR); >+ m_jit.or32(scratchGPR, JITCompiler::Address(resultGPR, JSRopeString::offsetOfFlags())); > if (!ASSERT_DISABLED) { > JITCompiler::Jump ok = m_jit.branch32( > JITCompiler::GreaterThanOrEqual, allocatorGPR, TrustedImm32(0)); > m_jit.abortWithReason(DFGNegativeStringLength); > ok.link(&m_jit); > } >- m_jit.store32(allocatorGPR, JITCompiler::Address(resultGPR, JSString::offsetOfLength())); >+ m_jit.store16(allocatorGPR, JITCompiler::Address(resultGPR, JSRopeString::offsetOfLowerLength())); >+ m_jit.urshift32(TrustedImm32(16), allocatorGPR); >+ m_jit.store16(allocatorGPR, JITCompiler::Address(resultGPR, JSRopeString::offsetOfUpperLength())); > > m_jit.mutatorFence(*m_jit.vm()); > >@@ -6320,21 +6416,21 @@ void SpeculativeJIT::compileStringEquality( > trueCase.append(fastTrue); > falseCase.append(fastFalse); > >- m_jit.load32(MacroAssembler::Address(leftGPR, JSString::offsetOfLength()), lengthGPR); >+ slowCase.append(m_jit.branchIfRopeString(leftGPR)); >+ slowCase.append(m_jit.branchIfRopeString(rightGPR)); >+ >+ m_jit.loadPtr(MacroAssembler::Address(leftGPR, JSString::offsetOfValue()), leftTempGPR); >+ m_jit.loadPtr(MacroAssembler::Address(rightGPR, JSString::offsetOfValue()), rightTempGPR); >+ >+ m_jit.load32(MacroAssembler::Address(leftTempGPR, StringImpl::lengthMemoryOffset()), lengthGPR); > > falseCase.append(m_jit.branch32( > MacroAssembler::NotEqual, >- MacroAssembler::Address(rightGPR, JSString::offsetOfLength()), >+ MacroAssembler::Address(rightTempGPR, StringImpl::lengthMemoryOffset()), > lengthGPR)); > > trueCase.append(m_jit.branchTest32(MacroAssembler::Zero, lengthGPR)); > >- m_jit.loadPtr(MacroAssembler::Address(leftGPR, JSString::offsetOfValue()), leftTempGPR); >- m_jit.loadPtr(MacroAssembler::Address(rightGPR, JSString::offsetOfValue()), rightTempGPR); >- >- slowCase.append(m_jit.branchTestPtr(MacroAssembler::Zero, leftTempGPR)); >- slowCase.append(m_jit.branchTestPtr(MacroAssembler::Zero, rightTempGPR)); >- > slowCase.append(m_jit.branchTest32( > MacroAssembler::Zero, > MacroAssembler::Address(leftTempGPR, StringImpl::flagsOffset()), >@@ -6638,9 +6734,25 @@ void SpeculativeJIT::compileStringZeroLength(Node* node) > GPRTemporary eq(this); > GPRReg eqGPR = eq.gpr(); > >- // Fetch the length field from the string object. >- m_jit.test32(MacroAssembler::Zero, MacroAssembler::Address(strGPR, JSString::offsetOfLength()), MacroAssembler::TrustedImm32(-1), eqGPR); >+ CCallHelpers::JumpList doneCases; >+ >+ auto isRope = m_jit.branchIfRopeString(strGPR); >+ m_jit.loadPtr(JITCompiler::Address(strGPR, JSString::offsetOfValue()), eqGPR); >+ m_jit.test32(JITCompiler::Zero, JITCompiler::Address(eqGPR, StringImpl::lengthMemoryOffset()), JITCompiler::TrustedImm32(-1), eqGPR); >+ doneCases.append(m_jit.jump()); >+ >+ CCallHelpers::JumpList nonZeroCases; >+ isRope.link(&m_jit); >+ m_jit.load16(MacroAssembler::Address(strGPR, JSRopeString::offsetOfUpperLength()), eqGPR); >+ nonZeroCases.append(m_jit.branchTest32(MacroAssembler::NonZero, eqGPR)); >+ m_jit.load16(MacroAssembler::Address(strGPR, JSRopeString::offsetOfLowerLength()), eqGPR); >+ m_jit.test32(CCallHelpers::Zero, eqGPR, JITCompiler::TrustedImm32(-1), eqGPR); >+ doneCases.append(m_jit.jump()); > >+ nonZeroCases.link(&m_jit); >+ m_jit.move(TrustedImm32(0), eqGPR); >+ >+ doneCases.link(&m_jit); > unblessedBooleanResult(eqGPR, node); > } > >@@ -6648,22 +6760,35 @@ void SpeculativeJIT::compileLogicalNotStringOrOther(Node* node) > { > JSValueOperand value(this, node->child1(), ManualOperandSpeculation); > GPRTemporary temp(this); >+ GPRTemporary temp2(this); > JSValueRegs valueRegs = value.jsValueRegs(); > GPRReg tempGPR = temp.gpr(); >+ GPRReg temp2GPR = temp2.gpr(); > > JITCompiler::Jump notCell = m_jit.branchIfNotCell(valueRegs); > GPRReg cellGPR = valueRegs.payloadGPR(); > DFG_TYPE_CHECK( > valueRegs, node->child1(), (~SpecCellCheck) | SpecString, m_jit.branchIfNotString(cellGPR)); >- m_jit.test32( >- JITCompiler::Zero, JITCompiler::Address(cellGPR, JSString::offsetOfLength()), >- JITCompiler::TrustedImm32(-1), tempGPR); >- JITCompiler::Jump done = m_jit.jump(); >+ >+ CCallHelpers::JumpList doneCases; >+ auto isRope = m_jit.branchIfRopeString(cellGPR); >+ m_jit.loadPtr(JITCompiler::Address(cellGPR, JSString::offsetOfValue()), tempGPR); >+ m_jit.test32(JITCompiler::Zero, JITCompiler::Address(tempGPR, StringImpl::lengthMemoryOffset()), JITCompiler::TrustedImm32(-1), tempGPR); >+ doneCases.append(m_jit.jump()); >+ >+ isRope.link(&m_jit); >+ m_jit.move(CCallHelpers::TrustedImm32(0), tempGPR); >+ m_jit.load16(CCallHelpers::Address(cellGPR, JSRopeString::offsetOfUpperLength()), temp2GPR); >+ doneCases.append(m_jit.branchTest32(CCallHelpers::NonZero, temp2GPR)); >+ m_jit.load16(CCallHelpers::Address(cellGPR, JSRopeString::offsetOfLowerLength()), temp2GPR); >+ m_jit.test32(CCallHelpers::Zero, temp2GPR, CCallHelpers::TrustedImm32(-1), tempGPR); >+ doneCases.append(m_jit.jump()); >+ > notCell.link(&m_jit); > DFG_TYPE_CHECK( > valueRegs, node->child1(), SpecCellCheck | SpecOther, m_jit.branchIfNotOther(valueRegs, tempGPR)); > m_jit.move(TrustedImm32(1), tempGPR); >- done.link(&m_jit); >+ doneCases.link(&m_jit); > > unblessedBooleanResult(tempGPR, node); > >@@ -6672,9 +6797,27 @@ void SpeculativeJIT::compileLogicalNotStringOrOther(Node* node) > void SpeculativeJIT::emitStringBranch(Edge nodeUse, BasicBlock* taken, BasicBlock* notTaken) > { > SpeculateCellOperand str(this, nodeUse); >- speculateString(nodeUse, str.gpr()); >- branchTest32(JITCompiler::NonZero, MacroAssembler::Address(str.gpr(), JSString::offsetOfLength()), taken); >+ GPRTemporary temp(this); >+ >+ GPRReg strGPR = str.gpr(); >+ GPRReg tempGPR = temp.gpr(); >+ >+ speculateString(nodeUse, strGPR); >+ >+ auto isRope = m_jit.branchIfRopeString(strGPR); >+ m_jit.loadPtr(JITCompiler::Address(strGPR, JSString::offsetOfValue()), tempGPR); >+ branchTest32( >+ JITCompiler::Zero, JITCompiler::Address(tempGPR, StringImpl::lengthMemoryOffset()), >+ JITCompiler::TrustedImm32(-1), notTaken); >+ jump(taken, ForceJump); >+ >+ isRope.link(&m_jit); >+ m_jit.load16(CCallHelpers::Address(strGPR, JSRopeString::offsetOfUpperLength()), tempGPR); >+ branchTest32(JITCompiler::NonZero, tempGPR, taken); >+ m_jit.load16(CCallHelpers::Address(strGPR, JSRopeString::offsetOfLowerLength()), tempGPR); >+ branchTest32(JITCompiler::NonZero, tempGPR, taken); > jump(notTaken); >+ > noResult(m_currentNode); > } > >@@ -6688,10 +6831,22 @@ void SpeculativeJIT::emitStringOrOtherBranch(Edge nodeUse, BasicBlock* taken, Ba > JITCompiler::Jump notCell = m_jit.branchIfNotCell(valueRegs); > GPRReg cellGPR = valueRegs.payloadGPR(); > DFG_TYPE_CHECK(valueRegs, nodeUse, (~SpecCellCheck) | SpecString, m_jit.branchIfNotString(cellGPR)); >+ >+ auto isRope = m_jit.branchIfRopeString(cellGPR); >+ m_jit.loadPtr(JITCompiler::Address(cellGPR, JSString::offsetOfValue()), tempGPR); > branchTest32( >- JITCompiler::Zero, JITCompiler::Address(cellGPR, JSString::offsetOfLength()), >+ JITCompiler::Zero, JITCompiler::Address(tempGPR, StringImpl::lengthMemoryOffset()), > JITCompiler::TrustedImm32(-1), notTaken); > jump(taken, ForceJump); >+ >+ isRope.link(&m_jit); >+ m_jit.load16(CCallHelpers::Address(cellGPR, JSRopeString::offsetOfUpperLength()), tempGPR); >+ branchTest32(CCallHelpers::NonZero, tempGPR, taken); >+ m_jit.load16(CCallHelpers::Address(cellGPR, JSRopeString::offsetOfLowerLength()), tempGPR); >+ branchTest32(CCallHelpers::NonZero, tempGPR, taken); >+ jump(notTaken, ForceJump); >+ >+ jump(taken, ForceJump); > notCell.link(&m_jit); > DFG_TYPE_CHECK( > valueRegs, nodeUse, SpecCellCheck | SpecOther, m_jit.branchIfNotOther(valueRegs, tempGPR)); >@@ -6736,13 +6891,12 @@ void SpeculativeJIT::compileGetIndexedPropertyStorage(Node* node) > > switch (node->arrayMode().type()) { > case Array::String: >- m_jit.loadPtr(MacroAssembler::Address(baseReg, JSString::offsetOfValue()), storageReg); >- > addSlowPathGenerator( > slowPathCall( >- m_jit.branchTest32(MacroAssembler::Zero, storageReg), >+ m_jit.branchIfRopeString(baseReg), > this, operationResolveRope, storageReg, baseReg)); > >+ m_jit.loadPtr(MacroAssembler::Address(baseReg, JSString::offsetOfValue()), storageReg); > m_jit.loadPtr(MacroAssembler::Address(storageReg, StringImpl::dataOffset()), storageReg); > break; > >@@ -6996,9 +7150,23 @@ void SpeculativeJIT::compileGetArrayLength(Node* node) > case Array::String: { > SpeculateCellOperand base(this, node->child1()); > GPRTemporary result(this, Reuse, base); >+ GPRTemporary temp(this); > GPRReg baseGPR = base.gpr(); > GPRReg resultGPR = result.gpr(); >- m_jit.load32(MacroAssembler::Address(baseGPR, JSString::offsetOfLength()), resultGPR); >+ GPRReg tempGPR = temp.gpr(); >+ >+ auto isRope = m_jit.branchIfRopeString(baseGPR); >+ m_jit.loadPtr(MacroAssembler::Address(baseGPR, JSString::offsetOfValue()), resultGPR); >+ m_jit.load32(MacroAssembler::Address(resultGPR, StringImpl::lengthMemoryOffset()), resultGPR); >+ auto done = m_jit.jump(); >+ >+ isRope.link(&m_jit); >+ m_jit.load16(CCallHelpers::Address(baseGPR, JSRopeString::offsetOfUpperLength()), tempGPR); >+ m_jit.load16(CCallHelpers::Address(baseGPR, JSRopeString::offsetOfLowerLength()), resultGPR); >+ m_jit.lshift32(CCallHelpers::TrustedImm32(16), tempGPR); >+ m_jit.or32(tempGPR, resultGPR); >+ >+ done.link(&m_jit); > int32Result(resultGPR, node); > break; > } >@@ -10140,13 +10308,13 @@ void SpeculativeJIT::speculateStringOrOther(Edge edge) > void SpeculativeJIT::speculateStringIdentAndLoadStorage(Edge edge, GPRReg string, GPRReg storage) > { > m_jit.loadPtr(MacroAssembler::Address(string, JSString::offsetOfValue()), storage); >- >+ > if (!needsTypeCheck(edge, SpecStringIdent | ~SpecString)) > return; > > speculationCheck( > BadType, JSValueSource::unboxedCell(string), edge, >- m_jit.branchTestPtr(MacroAssembler::Zero, storage)); >+ m_jit.branchIfRopeString(string)); > speculationCheck( > BadType, JSValueSource::unboxedCell(string), edge, m_jit.branchTest32( > MacroAssembler::Zero, >@@ -10547,19 +10715,18 @@ void SpeculativeJIT::emitSwitchImm(Node* node, SwitchData* data) > void SpeculativeJIT::emitSwitchCharStringJump( > SwitchData* data, GPRReg value, GPRReg scratch, GPRReg scratch2) > { >+ auto isRope = m_jit.branchIfRopeString(value); >+ >+ m_jit.loadPtr(MacroAssembler::Address(value, JSString::offsetOfValue()), scratch); >+ > addBranch( > m_jit.branch32( > MacroAssembler::NotEqual, >- MacroAssembler::Address(value, JSString::offsetOfLength()), >+ MacroAssembler::Address(scratch, StringImpl::lengthMemoryOffset()), > TrustedImm32(1)), > data->fallThrough.block); > >- m_jit.loadPtr(MacroAssembler::Address(value, JSString::offsetOfValue()), scratch); >- >- addSlowPathGenerator( >- slowPathCall( >- m_jit.branchTestPtr(MacroAssembler::Zero, scratch), >- this, operationResolveRope, scratch, value)); >+ addSlowPathGenerator(slowPathCall(isRope, this, operationResolveRope, scratch, value)); > > m_jit.loadPtr(MacroAssembler::Address(scratch, StringImpl::dataOffset()), value); > >@@ -10806,11 +10973,11 @@ void SpeculativeJIT::emitSwitchStringOnString(SwitchData* data, GPRReg string) > GPRReg lengthGPR = length.gpr(); > GPRReg tempGPR = temp.gpr(); > >- m_jit.load32(MacroAssembler::Address(string, JSString::offsetOfLength()), lengthGPR); >+ MacroAssembler::JumpList slowCases; >+ slowCases.append(m_jit.branchIfRopeString(string)); > m_jit.loadPtr(MacroAssembler::Address(string, JSString::offsetOfValue()), tempGPR); >+ m_jit.load32(MacroAssembler::Address(tempGPR, StringImpl::lengthMemoryOffset()), lengthGPR); > >- MacroAssembler::JumpList slowCases; >- slowCases.append(m_jit.branchTestPtr(MacroAssembler::Zero, tempGPR)); > slowCases.append(m_jit.branchTest32( > MacroAssembler::Zero, > MacroAssembler::Address(tempGPR, StringImpl::flagsOffset()), >diff --git a/Source/JavaScriptCore/dfg/DFGSpeculativeJIT32_64.cpp b/Source/JavaScriptCore/dfg/DFGSpeculativeJIT32_64.cpp >index 4dfe9c92321d753d178148f4ed35f8973f3e6f52..c9e81f91af13362f5f385847c18103ff15642273 100644 >--- a/Source/JavaScriptCore/dfg/DFGSpeculativeJIT32_64.cpp >+++ b/Source/JavaScriptCore/dfg/DFGSpeculativeJIT32_64.cpp >@@ -3879,8 +3879,8 @@ void SpeculativeJIT::compile(Node* node) > } > case StringUse: { > speculateString(node->child2(), keyRegs.payloadGPR()); >+ slowPath.append(m_jit.branchIfRopeString(keyRegs.payloadGPR())); > m_jit.loadPtr(MacroAssembler::Address(keyRegs.payloadGPR(), JSString::offsetOfValue()), implGPR); >- slowPath.append(m_jit.branchTestPtr(MacroAssembler::Zero, implGPR)); > slowPath.append(m_jit.branchTest32( > MacroAssembler::Zero, MacroAssembler::Address(implGPR, StringImpl::flagsOffset()), > MacroAssembler::TrustedImm32(StringImpl::flagIsAtomic()))); >@@ -3889,8 +3889,8 @@ void SpeculativeJIT::compile(Node* node) > case UntypedUse: { > slowPath.append(m_jit.branchIfNotCell(keyRegs)); > auto isNotString = m_jit.branchIfNotString(keyRegs.payloadGPR()); >+ slowPath.append(m_jit.branchIfRopeString(keyRegs.payloadGPR())); > m_jit.loadPtr(MacroAssembler::Address(keyRegs.payloadGPR(), JSString::offsetOfValue()), implGPR); >- slowPath.append(m_jit.branchTestPtr(MacroAssembler::Zero, implGPR)); > slowPath.append(m_jit.branchTest32( > MacroAssembler::Zero, MacroAssembler::Address(implGPR, StringImpl::flagsOffset()), > MacroAssembler::TrustedImm32(StringImpl::flagIsAtomic()))); >diff --git a/Source/JavaScriptCore/dfg/DFGSpeculativeJIT64.cpp b/Source/JavaScriptCore/dfg/DFGSpeculativeJIT64.cpp >index 63052becc0508d6d2927cb0060b4be87626976a1..7497e78f01c993d975810160dc18828d383b3c28 100644 >--- a/Source/JavaScriptCore/dfg/DFGSpeculativeJIT64.cpp >+++ b/Source/JavaScriptCore/dfg/DFGSpeculativeJIT64.cpp >@@ -4031,8 +4031,8 @@ void SpeculativeJIT::compile(Node* node) > isString.link(&m_jit); > } > >+ slowPath.append(m_jit.branchIfRopeString(inputGPR)); > m_jit.loadPtr(MacroAssembler::Address(inputGPR, JSString::offsetOfValue()), resultGPR); >- slowPath.append(m_jit.branchTestPtr(MacroAssembler::Zero, resultGPR)); > m_jit.load32(MacroAssembler::Address(resultGPR, StringImpl::flagsOffset()), resultGPR); > m_jit.urshift32(MacroAssembler::TrustedImm32(StringImpl::s_flagCount), resultGPR); > slowPath.append(m_jit.branchTest32(MacroAssembler::Zero, resultGPR)); >@@ -4069,8 +4069,8 @@ void SpeculativeJIT::compile(Node* node) > straightHash.append(m_jit.branchIfNotCell(inputGPR)); > MacroAssembler::JumpList slowPath; > straightHash.append(m_jit.branchIfNotString(inputGPR)); >+ slowPath.append(m_jit.branchIfRopeString(inputGPR)); > m_jit.loadPtr(MacroAssembler::Address(inputGPR, JSString::offsetOfValue()), resultGPR); >- slowPath.append(m_jit.branchTestPtr(MacroAssembler::Zero, resultGPR)); > m_jit.load32(MacroAssembler::Address(resultGPR, StringImpl::flagsOffset()), resultGPR); > m_jit.urshift32(MacroAssembler::TrustedImm32(StringImpl::s_flagCount), resultGPR); > slowPath.append(m_jit.branchTest32(MacroAssembler::Zero, resultGPR)); >@@ -4443,8 +4443,8 @@ void SpeculativeJIT::compile(Node* node) > } > case StringUse: { > speculateString(node->child2(), keyGPR); >+ slowPath.append(m_jit.branchIfRopeString(keyGPR)); > m_jit.loadPtr(MacroAssembler::Address(keyGPR, JSString::offsetOfValue()), implGPR); >- slowPath.append(m_jit.branchTestPtr(MacroAssembler::Zero, implGPR)); > slowPath.append(m_jit.branchTest32( > MacroAssembler::Zero, MacroAssembler::Address(implGPR, StringImpl::flagsOffset()), > MacroAssembler::TrustedImm32(StringImpl::flagIsAtomic()))); >@@ -4453,8 +4453,8 @@ void SpeculativeJIT::compile(Node* node) > case UntypedUse: { > slowPath.append(m_jit.branchIfNotCell(JSValueRegs(keyGPR))); > auto isNotString = m_jit.branchIfNotString(keyGPR); >+ slowPath.append(m_jit.branchIfRopeString(keyGPR)); > m_jit.loadPtr(MacroAssembler::Address(keyGPR, JSString::offsetOfValue()), implGPR); >- slowPath.append(m_jit.branchTestPtr(MacroAssembler::Zero, implGPR)); > slowPath.append(m_jit.branchTest32( > MacroAssembler::Zero, MacroAssembler::Address(implGPR, StringImpl::flagsOffset()), > MacroAssembler::TrustedImm32(StringImpl::flagIsAtomic()))); >diff --git a/Source/JavaScriptCore/ftl/FTLAbstractHeapRepository.cpp b/Source/JavaScriptCore/ftl/FTLAbstractHeapRepository.cpp >index c035112580c6b8a2958965654bc1c81ed7ae97d5..eba6bf9be193a1d91162698038eeb595e5d960e8 100644 >--- a/Source/JavaScriptCore/ftl/FTLAbstractHeapRepository.cpp >+++ b/Source/JavaScriptCore/ftl/FTLAbstractHeapRepository.cpp >@@ -60,8 +60,6 @@ AbstractHeapRepository::AbstractHeapRepository() > , JSCell_freeListNext(JSCell_header) > , ArrayStorage_publicLength(Butterfly_publicLength) > , ArrayStorage_vectorLength(Butterfly_vectorLength) >- , JSBigInt_length(JSBigIntOrString_length) >- , JSString_length(JSBigIntOrString_length) > > #define INDEXED_ABSTRACT_HEAP_INITIALIZATION(name, offset, size) , name(&root, #name, offset, size) > FOR_EACH_INDEXED_ABSTRACT_HEAP(INDEXED_ABSTRACT_HEAP_INITIALIZATION) >@@ -79,8 +77,6 @@ AbstractHeapRepository::AbstractHeapRepository() > RELEASE_ASSERT(JSCell_indexingTypeAndMisc.offset() + 2 == JSCell_typeInfoFlags.offset()); > RELEASE_ASSERT(JSCell_indexingTypeAndMisc.offset() + 3 == JSCell_cellState.offset()); > >- RELEASE_ASSERT(JSBigInt::offsetOfLength() == JSString::offsetOfLength()); >- > JSCell_structureID.changeParent(&JSCell_header); > JSCell_usefulBytes.changeParent(&JSCell_header); > JSCell_indexingTypeAndMisc.changeParent(&JSCell_usefulBytes); >diff --git a/Source/JavaScriptCore/ftl/FTLAbstractHeapRepository.h b/Source/JavaScriptCore/ftl/FTLAbstractHeapRepository.h >index e50541ba526b06682d0053f546bea2de2abd532a..0d0dd59fbe210be6e0f16192ac21621222ec000d 100644 >--- a/Source/JavaScriptCore/ftl/FTLAbstractHeapRepository.h >+++ b/Source/JavaScriptCore/ftl/FTLAbstractHeapRepository.h >@@ -64,7 +64,7 @@ namespace JSC { namespace FTL { > macro(JSArrayBufferView_length, JSArrayBufferView::offsetOfLength()) \ > macro(JSArrayBufferView_mode, JSArrayBufferView::offsetOfMode()) \ > macro(JSArrayBufferView_vector, JSArrayBufferView::offsetOfVector()) \ >- macro(JSBigIntOrString_length, JSBigInt::offsetOfLength()) \ >+ macro(JSBigInt_length, JSBigInt::offsetOfLength()) \ > macro(JSCell_cellState, JSCell::cellStateOffset()) \ > macro(JSCell_header, 0) \ > macro(JSCell_indexingTypeAndMisc, JSCell::indexingTypeAndMiscOffset()) \ >@@ -88,8 +88,10 @@ namespace JSC { namespace FTL { > macro(JSPropertyNameEnumerator_endGenericPropertyIndex, JSPropertyNameEnumerator::endGenericPropertyIndexOffset()) \ > macro(JSPropertyNameEnumerator_endStructurePropertyIndex, JSPropertyNameEnumerator::endStructurePropertyIndexOffset()) \ > macro(JSPropertyNameEnumerator_indexLength, JSPropertyNameEnumerator::indexedLengthOffset()) \ >+ macro(JSRopeString_flags, JSRopeString::offsetOfFlags()) \ >+ macro(JSRopeString_upperLength, JSRopeString::offsetOfUpperLength()) \ >+ macro(JSRopeString_lowerLength, JSRopeString::offsetOfLowerLength()) \ > macro(JSScope_next, JSScope::offsetOfNext()) \ >- macro(JSString_flags, JSString::offsetOfFlags()) \ > macro(JSString_value, JSString::offsetOfValue()) \ > macro(JSSymbolTableObject_symbolTable, JSSymbolTableObject::offsetOfSymbolTable()) \ > macro(JSWrapperObject_internalValue, JSWrapperObject::internalValueOffset()) \ >@@ -180,8 +182,6 @@ class AbstractHeapRepository { > AbstractHeap& JSCell_freeListNext; > AbstractHeap& ArrayStorage_publicLength; > AbstractHeap& ArrayStorage_vectorLength; >- AbstractHeap& JSBigInt_length; >- AbstractHeap& JSString_length; > > #define INDEXED_ABSTRACT_HEAP_DECLARATION(name, offset, size) IndexedAbstractHeap name; > FOR_EACH_INDEXED_ABSTRACT_HEAP(INDEXED_ABSTRACT_HEAP_DECLARATION) >diff --git a/Source/JavaScriptCore/ftl/FTLLowerDFGToB3.cpp b/Source/JavaScriptCore/ftl/FTLLowerDFGToB3.cpp >index a17f95421f7c30b5704e77d2ffd8a6a90317e1b1..85df203437ecf708a59aaaa5b9e7b45d1c34a3af 100644 >--- a/Source/JavaScriptCore/ftl/FTLLowerDFGToB3.cpp >+++ b/Source/JavaScriptCore/ftl/FTLLowerDFGToB3.cpp >@@ -3649,8 +3649,7 @@ class LowerDFGToB3 { > LValue fastResultValue = m_out.loadPtr(cell, m_heaps.JSString_value); > ValueFromBlock fastResult = m_out.anchor(fastResultValue); > >- m_out.branch( >- m_out.notNull(fastResultValue), usually(continuation), rarely(slowPath)); >+ m_out.branch(isRopeString(cell, m_node->child1()), rarely(slowPath), usually(continuation)); > > LBasicBlock lastNext = m_out.appendTo(slowPath, continuation); > >@@ -3822,7 +3821,25 @@ class LowerDFGToB3 { > > case Array::String: { > LValue string = lowCell(m_node->child1()); >- setInt32(m_out.load32NonNegative(string, m_heaps.JSString_length)); >+ >+ LBasicBlock ropePath = m_out.newBlock(); >+ LBasicBlock nonRopePath = m_out.newBlock(); >+ LBasicBlock continuation = m_out.newBlock(); >+ >+ m_out.branch(isRopeString(string, m_node->child1()), rarely(ropePath), usually(nonRopePath)); >+ >+ LBasicBlock lastNext = m_out.appendTo(ropePath, nonRopePath); >+ LValue upperLength = m_out.load16ZeroExt32(string, m_heaps.JSRopeString_upperLength); >+ LValue lowerLength = m_out.load16ZeroExt32(string, m_heaps.JSRopeString_lowerLength); >+ ValueFromBlock ropeLength = m_out.anchor(m_out.add(lowerLength, m_out.shl(upperLength, m_out.constInt32(16)))); >+ m_out.jump(continuation); >+ >+ m_out.appendTo(nonRopePath, continuation); >+ ValueFromBlock nonRopeLength = m_out.anchor(m_out.load32NonNegative(m_out.loadPtr(string, m_heaps.JSString_value), m_heaps.StringImpl_length)); >+ m_out.jump(continuation); >+ >+ m_out.appendTo(continuation, lastNext); >+ setInt32(m_out.phi(Int32, ropeLength, nonRopeLength)); > return; > } > >@@ -6492,46 +6509,96 @@ class LowerDFGToB3 { > > void compileMakeRope() > { >+ Edge edges[3] = { >+ m_node->child1(), >+ m_node->child2(), >+ m_node->child3(), >+ }; > LValue kids[3]; >+ LBasicBlock ropeCase[3] = { }; >+ LBasicBlock notRopeCase[3] = { }; >+ LBasicBlock mergeCase[3] = { }; >+ > unsigned numKids; >- kids[0] = lowCell(m_node->child1()); >- kids[1] = lowCell(m_node->child2()); >- if (m_node->child3()) { >- kids[2] = lowCell(m_node->child3()); >+ kids[0] = lowCell(edges[0]); >+ kids[1] = lowCell(edges[1]); >+ if (edges[2]) { >+ kids[2] = lowCell(edges[2]); > numKids = 3; > } else { > kids[2] = 0; > numKids = 2; > } >+ >+ for (unsigned i = 0; i < numKids; ++i) { >+ ropeCase[i] = m_out.newBlock(); >+ notRopeCase[i] = m_out.newBlock(); >+ mergeCase[i] = m_out.newBlock(); >+ } > > LBasicBlock slowPath = m_out.newBlock(); > LBasicBlock continuation = m_out.newBlock(); > >- LBasicBlock lastNext = m_out.insertNewBlocksBefore(slowPath); >- > Allocator allocator = allocatorForNonVirtualConcurrently<JSRopeString>(vm(), sizeof(JSRopeString), AllocatorForMode::AllocatorIfExists); > > LValue result = allocateCell( > m_out.constIntPtr(allocator.localAllocator()), vm().stringStructure.get(), slowPath); > >+ StructureIDBlob ropeBlob(m_graph.stringStructure->id(), m_graph.stringStructure->indexingModeIncludingHistory(), TypeInfo(StringType, JSRopeString::StructureFlags | TypeInfoPerCellBit)); >+ m_out.store32(m_out.constInt32(ropeBlob.blobExcludingStructureID()), result, m_heaps.JSCell_usefulBytes); > m_out.storePtr(m_out.intPtrZero, result, m_heaps.JSString_value); > for (unsigned i = 0; i < numKids; ++i) > m_out.storePtr(kids[i], result, m_heaps.JSRopeString_fibers[i]); > for (unsigned i = numKids; i < JSRopeString::s_maxInternalRopeLength; ++i) > m_out.storePtr(m_out.intPtrZero, result, m_heaps.JSRopeString_fibers[i]); >- LValue flags = m_out.load16ZeroExt32(kids[0], m_heaps.JSString_flags); >- LValue length = m_out.load32(kids[0], m_heaps.JSString_length); >+ >+ m_out.branch(isRopeString(kids[0], edges[0]), unsure(ropeCase[0]), unsure(notRopeCase[0])); >+ >+ LBasicBlock lastNext = m_out.appendTo(ropeCase[0], notRopeCase[0]); >+ ValueFromBlock flagsForRope = m_out.anchor(m_out.load32NonNegative(kids[0], m_heaps.JSRopeString_flags)); >+ LValue upperLength = m_out.load16ZeroExt32(kids[0], m_heaps.JSRopeString_upperLength); >+ LValue lowerLength = m_out.load16ZeroExt32(kids[0], m_heaps.JSRopeString_lowerLength); >+ ValueFromBlock lengthForRope = m_out.anchor(m_out.add(lowerLength, m_out.shl(upperLength, m_out.constInt32(16)))); >+ m_out.jump(mergeCase[0]); >+ >+ m_out.appendTo(notRopeCase[0], mergeCase[0]); >+ LValue stringImpl = m_out.loadPtr(kids[0], m_heaps.JSString_value); >+ ValueFromBlock flagsForNonRope = m_out.anchor(m_out.load32NonNegative(stringImpl, m_heaps.StringImpl_hashAndFlags)); >+ ValueFromBlock lengthForNonRope = m_out.anchor(m_out.load32NonNegative(stringImpl, m_heaps.StringImpl_length)); >+ m_out.jump(mergeCase[0]); >+ >+ m_out.appendTo(mergeCase[0], ropeCase[1]); >+ LValue flags = m_out.phi(Int32, flagsForRope, flagsForNonRope); >+ LValue length = m_out.phi(Int32, lengthForRope, lengthForNonRope); >+ > for (unsigned i = 1; i < numKids; ++i) { >- flags = m_out.bitAnd(flags, m_out.load16ZeroExt32(kids[i], m_heaps.JSString_flags)); >- CheckValue* lengthCheck = m_out.speculateAdd( >- length, m_out.load32(kids[i], m_heaps.JSString_length)); >+ m_out.branch(isRopeString(kids[i], edges[i]), unsure(ropeCase[i]), unsure(notRopeCase[i])); >+ >+ m_out.appendTo(ropeCase[i], notRopeCase[i]); >+ ValueFromBlock flagsForRope = m_out.anchor(m_out.load32NonNegative(kids[i], m_heaps.JSRopeString_flags)); >+ LValue upperLength = m_out.load16ZeroExt32(kids[i], m_heaps.JSRopeString_upperLength); >+ LValue lowerLength = m_out.load16ZeroExt32(kids[i], m_heaps.JSRopeString_lowerLength); >+ ValueFromBlock lengthForRope = m_out.anchor(m_out.add(lowerLength, m_out.shl(upperLength, m_out.constInt32(16)))); >+ m_out.jump(mergeCase[i]); >+ >+ m_out.appendTo(notRopeCase[i], mergeCase[i]); >+ LValue stringImpl = m_out.loadPtr(kids[i], m_heaps.JSString_value); >+ ValueFromBlock flagsForNonRope = m_out.anchor(m_out.load32NonNegative(stringImpl, m_heaps.StringImpl_hashAndFlags)); >+ ValueFromBlock lengthForNonRope = m_out.anchor(m_out.load32NonNegative(stringImpl, m_heaps.StringImpl_length)); >+ m_out.jump(mergeCase[i]); >+ >+ m_out.appendTo(mergeCase[i], (i + 1 != numKids) ? ropeCase[i + 1] : slowPath); >+ flags = m_out.bitAnd(flags, m_out.phi(Int32, flagsForRope, flagsForNonRope)); >+ CheckValue* lengthCheck = m_out.speculateAdd(length, m_out.phi(Int32, lengthForRope, lengthForNonRope)); > blessSpeculation(lengthCheck, Uncountable, noValue(), nullptr, m_origin); > length = lengthCheck; > } >- m_out.store32As16( >- m_out.bitAnd(m_out.constInt32(JSString::Is8Bit), flags), >- result, m_heaps.JSString_flags); >- m_out.store32(length, result, m_heaps.JSString_length); >+ m_out.store32( >+ m_out.bitOr(m_out.load32NonNegative(result, m_heaps.JSRopeString_flags), m_out.bitAnd(m_out.constInt32(JSRopeString::Is8Bit), flags)), >+ result, m_heaps.JSRopeString_flags); >+ >+ m_out.store32As16(length, result, m_heaps.JSRopeString_lowerLength); >+ m_out.store32As16(m_out.lShr(length, m_out.constInt32(16)), result, m_heaps.JSRopeString_upperLength); > > mutatorFence(); > ValueFromBlock fastResult = m_out.anchor(result); >@@ -6578,15 +6645,14 @@ class LowerDFGToB3 { > LBasicBlock slowPath = m_out.newBlock(); > LBasicBlock continuation = m_out.newBlock(); > >+ LValue stringImpl = m_out.loadPtr(base, m_heaps.JSString_value); > m_out.branch( > m_out.aboveOrEqual( >- index, m_out.load32NonNegative(base, m_heaps.JSString_length)), >+ index, m_out.load32NonNegative(stringImpl, m_heaps.StringImpl_length)), > rarely(slowPath), usually(fastPath)); > > LBasicBlock lastNext = m_out.appendTo(fastPath, slowPath); > >- LValue stringImpl = m_out.loadPtr(base, m_heaps.JSString_value); >- > LBasicBlock is8Bit = m_out.newBlock(); > LBasicBlock is16Bit = m_out.newBlock(); > LBasicBlock bitsContinuation = m_out.newBlock(); >@@ -6688,12 +6754,12 @@ class LowerDFGToB3 { > LValue index = lowInt32(m_node->child2()); > LValue storage = lowStorage(m_node->child3()); > >+ LValue stringImpl = m_out.loadPtr(base, m_heaps.JSString_value); >+ > speculate( > Uncountable, noValue(), 0, > m_out.aboveOrEqual( >- index, m_out.load32NonNegative(base, m_heaps.JSString_length))); >- >- LValue stringImpl = m_out.loadPtr(base, m_heaps.JSString_value); >+ index, m_out.load32NonNegative(stringImpl, m_heaps.StringImpl_length))); > > m_out.branch( > m_out.testIsZero32( >@@ -7207,7 +7273,7 @@ class LowerDFGToB3 { > > speculateString(m_node->child2(), right); > >- ValueFromBlock slowResult = m_out.anchor(stringsEqual(left, right)); >+ ValueFromBlock slowResult = m_out.anchor(stringsEqual(left, right, m_node->child1(), m_node->child2())); > m_out.jump(continuation); > > m_out.appendTo(continuation, lastNext); >@@ -7381,7 +7447,7 @@ class LowerDFGToB3 { > > // Full String compare. > m_out.appendTo(testStringEquality, continuation); >- ValueFromBlock slowResult = m_out.anchor(stringsEqual(leftString, rightValue)); >+ ValueFromBlock slowResult = m_out.anchor(stringsEqual(leftString, rightValue, stringEdge, untypedEdge)); > m_out.jump(continuation); > > // Continuation. >@@ -9031,25 +9097,25 @@ class LowerDFGToB3 { > LBasicBlock is16Bit = m_out.newBlock(); > LBasicBlock continuation = m_out.newBlock(); > >+ ValueFromBlock fastValue = m_out.anchor(m_out.loadPtr(stringValue, m_heaps.JSString_value)); >+ m_out.branch( >+ isRopeString(stringValue, m_node->child1()), >+ rarely(needResolution), usually(resolved)); >+ >+ LBasicBlock lastNext = m_out.appendTo(needResolution, resolved); >+ ValueFromBlock slowValue = m_out.anchor( >+ vmCall(pointerType(), m_out.operation(operationResolveRope), m_callFrame, stringValue)); >+ m_out.jump(resolved); >+ >+ m_out.appendTo(resolved, lengthIs1); >+ LValue value = m_out.phi(pointerType(), fastValue, slowValue); > m_out.branch( > m_out.notEqual( >- m_out.load32NonNegative(stringValue, m_heaps.JSString_length), >+ m_out.load32NonNegative(value, m_heaps.StringImpl_length), > m_out.int32One), > unsure(lowBlock(data->fallThrough.block)), unsure(lengthIs1)); >- >- LBasicBlock lastNext = m_out.appendTo(lengthIs1, needResolution); >- Vector<ValueFromBlock, 2> values; >- LValue fastValue = m_out.loadPtr(stringValue, m_heaps.JSString_value); >- values.append(m_out.anchor(fastValue)); >- m_out.branch(m_out.isNull(fastValue), rarely(needResolution), usually(resolved)); >- >- m_out.appendTo(needResolution, resolved); >- values.append(m_out.anchor( >- vmCall(pointerType(), m_out.operation(operationResolveRope), m_callFrame, stringValue))); >- m_out.jump(resolved); >- >- m_out.appendTo(resolved, is8Bit); >- LValue value = m_out.phi(pointerType(), values); >+ >+ m_out.appendTo(lengthIs1, is8Bit); > LValue characterData = m_out.loadPtr(value, m_heaps.StringImpl_data); > m_out.branch( > m_out.testNonZero32( >@@ -9091,7 +9157,7 @@ class LowerDFGToB3 { > } > > case StringUse: { >- switchString(data, lowString(m_node->child1())); >+ switchString(data, lowString(m_node->child1()), m_node->child1()); > return; > } > >@@ -9113,7 +9179,7 @@ class LowerDFGToB3 { > > m_out.appendTo(isStringBlock, lastNext); > >- switchString(data, value); >+ switchString(data, value, m_node->child1()); > return; > } > >@@ -9451,17 +9517,16 @@ class LowerDFGToB3 { > return key; > } > >- LValue mapHashString(LValue string) >+ LValue mapHashString(LValue string, Edge& edge) > { > LBasicBlock nonEmptyStringCase = m_out.newBlock(); > LBasicBlock slowCase = m_out.newBlock(); > LBasicBlock continuation = m_out.newBlock(); > >- LValue stringImpl = m_out.loadPtr(string, m_heaps.JSString_value); >- m_out.branch( >- m_out.equal(stringImpl, m_out.constIntPtr(0)), unsure(slowCase), unsure(nonEmptyStringCase)); >+ m_out.branch(isRopeString(string, edge), rarely(slowCase), usually(nonEmptyStringCase)); > > LBasicBlock lastNext = m_out.appendTo(nonEmptyStringCase, slowCase); >+ LValue stringImpl = m_out.loadPtr(string, m_heaps.JSString_value); > LValue hash = m_out.lShr(m_out.load32(stringImpl, m_heaps.StringImpl_hashAndFlags), m_out.constInt32(StringImpl::s_flagCount)); > ValueFromBlock nonEmptyStringHashResult = m_out.anchor(hash); > m_out.branch(m_out.equal(hash, m_out.constInt32(0)), >@@ -9500,7 +9565,7 @@ class LowerDFGToB3 { > isStringValue, unsure(isString), unsure(notString)); > > LBasicBlock lastNext = m_out.appendTo(isString, notString); >- ValueFromBlock stringResult = m_out.anchor(mapHashString(value)); >+ ValueFromBlock stringResult = m_out.anchor(mapHashString(value, m_node->child1())); > m_out.jump(continuation); > > m_out.appendTo(notString, continuation); >@@ -9514,7 +9579,7 @@ class LowerDFGToB3 { > > case StringUse: { > LValue string = lowString(m_node->child1()); >- setInt32(mapHashString(string)); >+ setInt32(mapHashString(string, m_node->child1())); > return; > } > >@@ -9541,11 +9606,10 @@ class LowerDFGToB3 { > isString, unsure(isStringCase), unsure(straightHash)); > > m_out.appendTo(isStringCase, nonEmptyStringCase); >- LValue stringImpl = m_out.loadPtr(value, m_heaps.JSString_value); >- m_out.branch( >- m_out.equal(stringImpl, m_out.constIntPtr(0)), rarely(slowCase), usually(nonEmptyStringCase)); >+ m_out.branch(isRopeString(value, m_node->child1()), rarely(slowCase), usually(nonEmptyStringCase)); > > m_out.appendTo(nonEmptyStringCase, straightHash); >+ LValue stringImpl = m_out.loadPtr(value, m_heaps.JSString_value); > LValue hash = m_out.lShr(m_out.load32(stringImpl, m_heaps.StringImpl_hashAndFlags), m_out.constInt32(StringImpl::s_flagCount)); > ValueFromBlock nonEmptyStringHashResult = m_out.anchor(hash); > m_out.branch(m_out.equal(hash, m_out.constInt32(0)), >@@ -10143,10 +10207,10 @@ class LowerDFGToB3 { > LBasicBlock isAtomicString = m_out.newBlock(); > > keyAsValue = lowString(m_node->child2()); >- uniquedStringImpl = m_out.loadPtr(keyAsValue, m_heaps.JSString_value); >- m_out.branch(m_out.notNull(uniquedStringImpl), usually(isNonEmptyString), rarely(slowCase)); >+ m_out.branch(isNotRopeString(keyAsValue, m_node->child2()), usually(isNonEmptyString), rarely(slowCase)); > > lastNext = m_out.appendTo(isNonEmptyString, isAtomicString); >+ uniquedStringImpl = m_out.loadPtr(keyAsValue, m_heaps.JSString_value); > LValue isNotAtomic = m_out.testIsZero32(m_out.load32(uniquedStringImpl, m_heaps.StringImpl_hashAndFlags), m_out.constInt32(StringImpl::flagIsAtomic())); > m_out.branch(isNotAtomic, rarely(slowCase), usually(isAtomicString)); > >@@ -10174,11 +10238,11 @@ class LowerDFGToB3 { > m_out.branch(isString(keyAsValue), unsure(isStringCase), unsure(notStringCase)); > > m_out.appendTo(isStringCase, isNonEmptyString); >- LValue implFromString = m_out.loadPtr(keyAsValue, m_heaps.JSString_value); >- ValueFromBlock stringResult = m_out.anchor(implFromString); >- m_out.branch(m_out.notNull(implFromString), usually(isNonEmptyString), rarely(slowCase)); >+ m_out.branch(isNotRopeString(keyAsValue, m_node->child2()), usually(isNonEmptyString), rarely(slowCase)); > > m_out.appendTo(isNonEmptyString, notStringCase); >+ LValue implFromString = m_out.loadPtr(keyAsValue, m_heaps.JSString_value); >+ ValueFromBlock stringResult = m_out.anchor(implFromString); > LValue isNotAtomic = m_out.testIsZero32(m_out.load32(implFromString, m_heaps.StringImpl_hashAndFlags), m_out.constInt32(StringImpl::flagIsAtomic())); > m_out.branch(isNotAtomic, rarely(slowCase), usually(hasUniquedStringImpl)); > >@@ -11916,45 +11980,46 @@ class LowerDFGToB3 { > > void compileStringSlice() > { >+ LBasicBlock lengthCheckCase = m_out.newBlock(); > LBasicBlock emptyCase = m_out.newBlock(); > LBasicBlock notEmptyCase = m_out.newBlock(); > LBasicBlock oneCharCase = m_out.newBlock(); >- LBasicBlock bitCheckCase = m_out.newBlock(); > LBasicBlock is8Bit = m_out.newBlock(); > LBasicBlock is16Bit = m_out.newBlock(); > LBasicBlock bitsContinuation = m_out.newBlock(); > LBasicBlock bigCharacter = m_out.newBlock(); > LBasicBlock slowCase = m_out.newBlock(); >+ LBasicBlock ropeSlowCase = m_out.newBlock(); > LBasicBlock continuation = m_out.newBlock(); > > LValue string = lowString(m_node->child1()); >- LValue length = m_out.load32NonNegative(string, m_heaps.JSString_length); > LValue start = lowInt32(m_node->child2()); > LValue end = nullptr; > if (m_node->child3()) > end = lowInt32(m_node->child3()); >+ else >+ end = m_out.constInt32(std::numeric_limits<int32_t>::max()); >+ m_out.branch(isRopeString(string, m_node->child1()), rarely(ropeSlowCase), usually(lengthCheckCase)); > >+ LBasicBlock lastNext = m_out.appendTo(lengthCheckCase, emptyCase); >+ LValue stringImpl = m_out.loadPtr(string, m_heaps.JSString_value); >+ LValue length = m_out.load32NonNegative(stringImpl, m_heaps.StringImpl_length); > auto range = populateSliceRange(start, end, length); > LValue from = range.first; > LValue to = range.second; >- > LValue span = m_out.sub(to, from); > m_out.branch(m_out.lessThanOrEqual(span, m_out.int32Zero), unsure(emptyCase), unsure(notEmptyCase)); > >- Vector<ValueFromBlock, 4> results; >+ Vector<ValueFromBlock, 5> results; > >- LBasicBlock lastNext = m_out.appendTo(emptyCase, notEmptyCase); >+ m_out.appendTo(emptyCase, notEmptyCase); > results.append(m_out.anchor(weakPointer(jsEmptyString(&vm())))); > m_out.jump(continuation); > > m_out.appendTo(notEmptyCase, oneCharCase); > m_out.branch(m_out.equal(span, m_out.int32One), unsure(oneCharCase), unsure(slowCase)); > >- m_out.appendTo(oneCharCase, bitCheckCase); >- LValue stringImpl = m_out.loadPtr(string, m_heaps.JSString_value); >- m_out.branch(m_out.isNull(stringImpl), unsure(slowCase), unsure(bitCheckCase)); >- >- m_out.appendTo(bitCheckCase, is8Bit); >+ m_out.appendTo(oneCharCase, is8Bit); > LValue storage = m_out.loadPtr(stringImpl, m_heaps.StringImpl_data); > m_out.branch( > m_out.testIsZero32( >@@ -11963,8 +12028,6 @@ class LowerDFGToB3 { > unsure(is16Bit), unsure(is8Bit)); > > m_out.appendTo(is8Bit, is16Bit); >- // FIXME: Need to cage strings! >- // https://bugs.webkit.org/show_bug.cgi?id=174924 > ValueFromBlock char8Bit = m_out.anchor(m_out.load8ZeroExt32(m_out.baseIndex(m_heaps.characters8, storage, m_out.zeroExtPtr(from)))); > m_out.jump(bitsContinuation); > >@@ -11988,10 +12051,14 @@ class LowerDFGToB3 { > m_heaps.singleCharacterStrings, smallStrings, m_out.zeroExtPtr(character))))); > m_out.jump(continuation); > >- m_out.appendTo(slowCase, continuation); >+ m_out.appendTo(slowCase, ropeSlowCase); > results.append(m_out.anchor(vmCall(pointerType(), m_out.operation(operationStringSubstr), m_callFrame, string, from, span))); > m_out.jump(continuation); > >+ m_out.appendTo(ropeSlowCase, continuation); >+ results.append(m_out.anchor(vmCall(pointerType(), m_out.operation(operationStringSlice), m_callFrame, string, start, end))); >+ m_out.jump(continuation); >+ > m_out.appendTo(continuation, lastNext); > setJSValue(m_out.phi(pointerType(), results)); > } >@@ -12008,12 +12075,11 @@ class LowerDFGToB3 { > LValue string = lowString(m_node->child1()); > ValueFromBlock startIndex = m_out.anchor(m_out.constInt32(0)); > ValueFromBlock startIndexForCall = m_out.anchor(m_out.constInt32(0)); >- LValue impl = m_out.loadPtr(string, m_heaps.JSString_value); >- m_out.branch(m_out.isZero64(impl), >+ m_out.branch(isRopeString(string, m_node->child1()), > unsure(slowPath), unsure(notRope)); > > LBasicBlock lastNext = m_out.appendTo(notRope, is8Bit); >- >+ LValue impl = m_out.loadPtr(string, m_heaps.JSString_value); > m_out.branch( > m_out.testIsZero32( > m_out.load32(impl, m_heaps.StringImpl_hashAndFlags), >@@ -12785,7 +12851,7 @@ class LowerDFGToB3 { > setBoolean(m_out.phi(Int32, fastResult, slowResult)); > } > >- LValue stringsEqual(LValue leftJSString, LValue rightJSString) >+ LValue stringsEqual(LValue leftJSString, LValue rightJSString, Edge leftJSStringEdge = Edge(), Edge rightJSStringEdge = Edge()) > { > LBasicBlock notTriviallyUnequalCase = m_out.newBlock(); > LBasicBlock notEmptyCase = m_out.newBlock(); >@@ -12800,29 +12866,23 @@ class LowerDFGToB3 { > LBasicBlock slowCase = m_out.newBlock(); > LBasicBlock continuation = m_out.newBlock(); > >- LValue length = m_out.load32(leftJSString, m_heaps.JSString_length); >+ m_out.branch(isRopeString(leftJSString, leftJSStringEdge), rarely(slowCase), usually(leftReadyCase)); > >- m_out.branch( >- m_out.notEqual(length, m_out.load32(rightJSString, m_heaps.JSString_length)), >- unsure(falseCase), unsure(notTriviallyUnequalCase)); >- >- LBasicBlock lastNext = m_out.appendTo(notTriviallyUnequalCase, notEmptyCase); >- >- m_out.branch(m_out.isZero32(length), unsure(trueCase), unsure(notEmptyCase)); >- >- m_out.appendTo(notEmptyCase, leftReadyCase); >+ LBasicBlock lastNext = m_out.appendTo(leftReadyCase, rightReadyCase); >+ m_out.branch(isRopeString(rightJSString, rightJSStringEdge), rarely(slowCase), usually(rightReadyCase)); > >+ m_out.appendTo(rightReadyCase, notTriviallyUnequalCase); > LValue left = m_out.loadPtr(leftJSString, m_heaps.JSString_value); > LValue right = m_out.loadPtr(rightJSString, m_heaps.JSString_value); >+ LValue length = m_out.load32(left, m_heaps.StringImpl_length); >+ m_out.branch( >+ m_out.notEqual(length, m_out.load32(right, m_heaps.StringImpl_length)), >+ unsure(falseCase), unsure(notTriviallyUnequalCase)); > >- m_out.branch(m_out.notNull(left), usually(leftReadyCase), rarely(slowCase)); >- >- m_out.appendTo(leftReadyCase, rightReadyCase); >- >- m_out.branch(m_out.notNull(right), usually(rightReadyCase), rarely(slowCase)); >- >- m_out.appendTo(rightReadyCase, left8BitCase); >+ m_out.appendTo(notTriviallyUnequalCase, notEmptyCase); >+ m_out.branch(m_out.isZero32(length), unsure(trueCase), unsure(notEmptyCase)); > >+ m_out.appendTo(notEmptyCase, left8BitCase); > m_out.branch( > m_out.testIsZero32( > m_out.load32(left, m_heaps.StringImpl_hashAndFlags), >@@ -12830,7 +12890,6 @@ class LowerDFGToB3 { > unsure(slowCase), unsure(left8BitCase)); > > m_out.appendTo(left8BitCase, right8BitCase); >- > m_out.branch( > m_out.testIsZero32( > m_out.load32(right, m_heaps.StringImpl_hashAndFlags), >@@ -13553,24 +13612,51 @@ class LowerDFGToB3 { > edge, CellCaseSpeculatesObject, SpeculateNullOrUndefined, > ManualOperandSpeculation)); > case StringUse: { >+ LBasicBlock ropeCase = m_out.newBlock(); >+ LBasicBlock notRopeCase = m_out.newBlock(); >+ LBasicBlock continuation = m_out.newBlock(); >+ > LValue stringValue = lowString(edge); >- LValue length = m_out.load32NonNegative(stringValue, m_heaps.JSString_length); >- return m_out.notEqual(length, m_out.int32Zero); >+ >+ m_out.branch(isRopeString(stringValue, edge), rarely(ropeCase), usually(notRopeCase)); >+ >+ LBasicBlock lastNext = m_out.appendTo(ropeCase, notRopeCase); >+ LValue upperLength = m_out.load16ZeroExt32(stringValue, m_heaps.JSRopeString_upperLength); >+ LValue lowerLength = m_out.load16ZeroExt32(stringValue, m_heaps.JSRopeString_lowerLength); >+ ValueFromBlock ropeLength = m_out.anchor(m_out.bitOr(upperLength, lowerLength)); >+ m_out.jump(continuation); >+ >+ m_out.appendTo(notRopeCase, continuation); >+ ValueFromBlock nonRopeLength = m_out.anchor(m_out.load32NonNegative(m_out.loadPtr(stringValue, m_heaps.JSString_value), m_heaps.StringImpl_length)); >+ m_out.jump(continuation); >+ >+ m_out.appendTo(continuation, lastNext); >+ return m_out.notZero32(m_out.phi(Int32, ropeLength, nonRopeLength)); > } > case StringOrOtherUse: { > LValue value = lowJSValue(edge, ManualOperandSpeculation); > > LBasicBlock cellCase = m_out.newBlock(); > LBasicBlock notCellCase = m_out.newBlock(); >+ LBasicBlock ropeCase = m_out.newBlock(); >+ LBasicBlock notRopeCase = m_out.newBlock(); > LBasicBlock continuation = m_out.newBlock(); > > m_out.branch(isCell(value, provenType(edge)), unsure(cellCase), unsure(notCellCase)); > >- LBasicBlock lastNext = m_out.appendTo(cellCase, notCellCase); >+ LBasicBlock lastNext = m_out.appendTo(cellCase, ropeCase); > > FTL_TYPE_CHECK(jsValueValue(value), edge, (~SpecCellCheck) | SpecString, isNotString(value)); >- LValue length = m_out.load32NonNegative(value, m_heaps.JSString_length); >- ValueFromBlock cellResult = m_out.anchor(m_out.notEqual(length, m_out.int32Zero)); >+ m_out.branch(isRopeString(value, edge), rarely(ropeCase), usually(notRopeCase)); >+ >+ m_out.appendTo(ropeCase, notRopeCase); >+ LValue upperLength = m_out.load16ZeroExt32(value, m_heaps.JSRopeString_upperLength); >+ LValue lowerLength = m_out.load16ZeroExt32(value, m_heaps.JSRopeString_lowerLength); >+ ValueFromBlock ropeResult = m_out.anchor(m_out.bitOr(upperLength, lowerLength)); >+ m_out.jump(continuation); >+ >+ m_out.appendTo(notRopeCase, notCellCase); >+ ValueFromBlock notRopeResult = m_out.anchor(m_out.load32NonNegative(m_out.loadPtr(value, m_heaps.JSString_value), m_heaps.StringImpl_length)); > m_out.jump(continuation); > > m_out.appendTo(notCellCase, continuation); >@@ -13580,7 +13666,7 @@ class LowerDFGToB3 { > m_out.jump(continuation); > m_out.appendTo(continuation, lastNext); > >- return m_out.phi(Int32, cellResult, notCellResult); >+ return m_out.notZero32(m_out.phi(Int32, ropeResult, notRopeResult, notCellResult)); > } > case UntypedUse: { > LValue value = lowJSValue(edge); >@@ -13603,7 +13689,10 @@ class LowerDFGToB3 { > > LBasicBlock cellCase = m_out.newBlock(); > LBasicBlock notStringCase = m_out.newBlock(); >- LBasicBlock stringOrBigIntCase = m_out.newBlock(); >+ LBasicBlock stringCase = m_out.newBlock(); >+ LBasicBlock ropeCase = m_out.newBlock(); >+ LBasicBlock notRopeCase = m_out.newBlock(); >+ LBasicBlock bigIntCase = m_out.newBlock(); > LBasicBlock notStringOrBigIntCase = m_out.newBlock(); > LBasicBlock notCellCase = m_out.newBlock(); > LBasicBlock int32Case = m_out.newBlock(); >@@ -13619,17 +13708,32 @@ class LowerDFGToB3 { > LBasicBlock lastNext = m_out.appendTo(cellCase, notStringCase); > m_out.branch( > isString(value, provenType(edge) & SpecCell), >- unsure(stringOrBigIntCase), unsure(notStringCase)); >+ unsure(stringCase), unsure(notStringCase)); > >- m_out.appendTo(notStringCase, stringOrBigIntCase); >+ m_out.appendTo(notStringCase, stringCase); > m_out.branch( > isBigInt(value, provenType(edge) & (SpecCell - SpecString)), >- unsure(stringOrBigIntCase), unsure(notStringOrBigIntCase)); >+ unsure(bigIntCase), unsure(notStringOrBigIntCase)); >+ >+ m_out.appendTo(stringCase, ropeCase); >+ m_out.branch(isRopeString(value, edge), rarely(ropeCase), usually(notRopeCase)); >+ >+ m_out.appendTo(ropeCase, notRopeCase); >+ LValue upperLength = m_out.load16ZeroExt32(value, m_heaps.JSRopeString_upperLength); >+ LValue lowerLength = m_out.load16ZeroExt32(value, m_heaps.JSRopeString_lowerLength); >+ results.append(m_out.anchor(m_out.notZero32(m_out.bitOr(upperLength, lowerLength)))); >+ m_out.jump(continuation); >+ >+ m_out.appendTo(notRopeCase, bigIntCase); >+ LValue nonZeroNonRope = m_out.notZero32( >+ m_out.load32NonNegative(m_out.loadPtr(value, m_heaps.JSString_value), m_heaps.StringImpl_length)); >+ results.append(m_out.anchor(nonZeroNonRope)); >+ m_out.jump(continuation); > >- m_out.appendTo(stringOrBigIntCase, notStringOrBigIntCase); >- LValue nonZeroCell = m_out.notZero32( >- m_out.load32NonNegative(value, m_heaps.JSBigIntOrString_length)); >- results.append(m_out.anchor(nonZeroCell)); >+ m_out.appendTo(bigIntCase, notStringOrBigIntCase); >+ LValue nonZeroBigInt = m_out.notZero32( >+ m_out.load32NonNegative(value, m_heaps.JSBigInt_length)); >+ results.append(m_out.anchor(nonZeroBigInt)); > m_out.jump(continuation); > > m_out.appendTo(notStringOrBigIntCase, notCellCase); >@@ -13893,7 +13997,7 @@ class LowerDFGToB3 { > lowBlock(data->fallThrough.block), Weight(data->fallThrough.count)); > } > >- void switchString(SwitchData* data, LValue string) >+ void switchString(SwitchData* data, LValue string, Edge& edge) > { > bool canDoBinarySwitch = true; > unsigned totalLength = 0; >@@ -13916,16 +14020,16 @@ class LowerDFGToB3 { > return; > } > >- LValue stringImpl = m_out.loadPtr(string, m_heaps.JSString_value); >- LValue length = m_out.load32(string, m_heaps.JSString_length); >- > LBasicBlock hasImplBlock = m_out.newBlock(); > LBasicBlock is8BitBlock = m_out.newBlock(); > LBasicBlock slowBlock = m_out.newBlock(); > >- m_out.branch(m_out.isNull(stringImpl), unsure(slowBlock), unsure(hasImplBlock)); >+ m_out.branch(isRopeString(string, edge), unsure(slowBlock), unsure(hasImplBlock)); > > LBasicBlock lastNext = m_out.appendTo(hasImplBlock, is8BitBlock); >+ >+ LValue stringImpl = m_out.loadPtr(string, m_heaps.JSString_value); >+ LValue length = m_out.load32(stringImpl, m_heaps.StringImpl_length); > > m_out.branch( > m_out.testIsZero32( >@@ -15620,6 +15724,38 @@ class LowerDFGToB3 { > m_out.constInt32(vm().stringStructure->id())); > } > >+ LValue isRopeString(LValue string, Edge edge = Edge()) >+ { >+ if (edge) { >+ if (!((provenType(edge) & SpecString) & ~SpecStringIdent)) >+ return m_out.booleanFalse; >+ if (JSValue value = provenValue(edge)) { >+ if (value.isCell() && value.asCell()->type() == StringType && !asString(value)->isRope()) >+ return m_out.booleanFalse; >+ } >+ } >+ >+ return m_out.testNonZero32( >+ m_out.load8ZeroExt32(string, m_heaps.JSCell_typeInfoFlags), >+ m_out.constInt32(TypeInfoPerCellBit)); >+ } >+ >+ LValue isNotRopeString(LValue string, Edge edge = Edge()) >+ { >+ if (edge) { >+ if (!((provenType(edge) & SpecString) & ~SpecStringIdent)) >+ return m_out.booleanTrue; >+ if (JSValue value = provenValue(edge)) { >+ if (value.isCell() && value.asCell()->type() == StringType && !asString(value)->isRope()) >+ return m_out.booleanTrue; >+ } >+ } >+ >+ return m_out.testIsZero32( >+ m_out.load8ZeroExt32(string, m_heaps.JSCell_typeInfoFlags), >+ m_out.constInt32(TypeInfoPerCellBit)); >+ } >+ > LValue isNotSymbol(LValue cell, SpeculatedType type = SpecFullTop) > { > if (LValue proven = isProvenValue(type & SpecCell, ~SpecSymbol)) >@@ -16017,7 +16153,7 @@ class LowerDFGToB3 { > if (!m_interpreter.needsTypeCheck(edge, SpecStringIdent | ~SpecString)) > return; > >- speculate(BadType, jsValueValue(string), edge.node(), m_out.isNull(stringImpl)); >+ speculate(BadType, jsValueValue(string), edge.node(), isRopeString(string)); > speculate( > BadType, jsValueValue(string), edge.node(), > m_out.testIsZero32( >diff --git a/Source/JavaScriptCore/jit/AssemblyHelpers.cpp b/Source/JavaScriptCore/jit/AssemblyHelpers.cpp >index 973fd3ce381b5ecafe37c8fe3727ffc5ad90507f..bc91af62e817834ad37b0c4901641f7e66e2cd89 100644 >--- a/Source/JavaScriptCore/jit/AssemblyHelpers.cpp >+++ b/Source/JavaScriptCore/jit/AssemblyHelpers.cpp >@@ -713,8 +713,24 @@ void AssemblyHelpers::emitConvertValueToBoolean(VM& vm, JSValueRegs value, GPRRe > done.append(jump()); > > isString.link(this); >+ auto isRope = branchIfRopeString(value.payloadGPR()); >+ loadPtr(Address(value.payloadGPR(), JSString::offsetOfValue()), result); >+ load32(Address(result, StringImpl::lengthMemoryOffset()), result); >+ compare32(invert ? Equal : NotEqual, result, TrustedImm32(0), result); >+ done.append(jump()); >+ >+ isRope.link(this); >+ load16(Address(value.payloadGPR(), JSRopeString::offsetOfUpperLength()), result); >+ auto nonZeroRope = branchTest32(NonZero, result); >+ load16(Address(value.payloadGPR(), JSRopeString::offsetOfLowerLength()), result); >+ test32(invert ? Zero : NonZero, result, TrustedImm32(-1), result); >+ done.append(jump()); >+ >+ nonZeroRope.link(this); >+ move(invert ? TrustedImm32(0) : TrustedImm32(1), result); >+ done.append(jump()); >+ > isBigInt.link(this); >- RELEASE_ASSERT(JSString::offsetOfLength() == JSBigInt::offsetOfLength()); > load32(Address(value.payloadGPR(), JSBigInt::offsetOfLength()), result); > compare32(invert ? Equal : NotEqual, result, TrustedImm32(0), result); > done.append(jump()); >@@ -800,8 +816,26 @@ AssemblyHelpers::JumpList AssemblyHelpers::branchIfValue(VM& vm, JSValueRegs val > } > > isString.link(this); >+ auto isRope = branchIfRopeString(value.payloadGPR()); >+ loadPtr(Address(value.payloadGPR(), JSString::offsetOfValue()), scratch); >+ truthy.append(branchTest32(invert ? Zero : NonZero, Address(scratch, StringImpl::lengthMemoryOffset()))); >+ done.append(jump()); >+ >+ isRope.link(this); >+ load16(CCallHelpers::Address(value.payloadGPR(), JSRopeString::offsetOfLowerLength()), scratch); >+ if (invert) { >+ done.append(branchTest32(NonZero, scratch)); >+ load16(CCallHelpers::Address(value.payloadGPR(), JSRopeString::offsetOfUpperLength()), scratch); >+ done.append(branchTest32(NonZero, scratch)); >+ truthy.append(jump()); >+ } else { >+ truthy.append(branchTest32(NonZero, scratch)); >+ load16(CCallHelpers::Address(value.payloadGPR(), JSRopeString::offsetOfUpperLength()), scratch); >+ truthy.append(branchTest32(NonZero, scratch)); >+ done.append(jump()); >+ } >+ > isBigInt.link(this); >- RELEASE_ASSERT(JSString::offsetOfLength() == JSBigInt::offsetOfLength()); > truthy.append(branchTest32(invert ? Zero : NonZero, Address(value.payloadGPR(), JSBigInt::offsetOfLength()))); > done.append(jump()); > >diff --git a/Source/JavaScriptCore/jit/AssemblyHelpers.h b/Source/JavaScriptCore/jit/AssemblyHelpers.h >index 16e36204b9ce22780780d859b80dde55a23f5be3..1ad3ab37566e0ea03d76f82064cf6c7bda3cf644 100644 >--- a/Source/JavaScriptCore/jit/AssemblyHelpers.h >+++ b/Source/JavaScriptCore/jit/AssemblyHelpers.h >@@ -1075,6 +1075,16 @@ class AssemblyHelpers : public MacroAssembler { > return branchDouble(DoubleEqual, fpr, fpr); > } > >+ Jump branchIfRopeString(GPRReg stringGPR) >+ { >+ return branchTest8(NonZero, Address(stringGPR, JSCell::typeInfoFlagsOffset()), TrustedImm32(TypeInfoPerCellBit)); >+ } >+ >+ Jump branchIfNotRope(GPRReg stringGPR) >+ { >+ return branchTest8(Zero, Address(stringGPR, JSCell::typeInfoFlagsOffset()), TrustedImm32(TypeInfoPerCellBit)); >+ } >+ > static Address addressForByteOffset(ptrdiff_t byteOffset) > { > return Address(GPRInfo::callFrameRegister, byteOffset); >diff --git a/Source/JavaScriptCore/jit/JITInlines.h b/Source/JavaScriptCore/jit/JITInlines.h >index e260c11fecf3eb4715d0dcc541b6ab0bd4774dcd..d178ba6a09356c40a2d40e6d344d4216c0be5bbc 100644 >--- a/Source/JavaScriptCore/jit/JITInlines.h >+++ b/Source/JavaScriptCore/jit/JITInlines.h >@@ -94,9 +94,9 @@ ALWAYS_INLINE void JIT::emitPutIntToCallFrameHeader(RegisterID from, int entry) > ALWAYS_INLINE void JIT::emitLoadCharacterString(RegisterID src, RegisterID dst, JumpList& failures) > { > failures.append(branchIfNotString(src)); >- failures.append(branch32(NotEqual, MacroAssembler::Address(src, JSString::offsetOfLength()), TrustedImm32(1))); >+ failures.append(branchIfRopeString(src)); > loadPtr(MacroAssembler::Address(src, JSString::offsetOfValue()), dst); >- failures.append(branchTest32(Zero, dst)); >+ failures.append(branch32(NotEqual, MacroAssembler::Address(dst, StringImpl::lengthMemoryOffset()), TrustedImm32(1))); > loadPtr(MacroAssembler::Address(dst, StringImpl::flagsOffset()), regT1); > loadPtr(MacroAssembler::Address(dst, StringImpl::dataOffset()), dst); > >diff --git a/Source/JavaScriptCore/jit/ThunkGenerators.cpp b/Source/JavaScriptCore/jit/ThunkGenerators.cpp >index 38ec6a0dd95758930cd58f21cc43d0bde48e4bf4..a92369bf318e3db694c820ffccf8470459da3c2b 100644 >--- a/Source/JavaScriptCore/jit/ThunkGenerators.cpp >+++ b/Source/JavaScriptCore/jit/ThunkGenerators.cpp >@@ -636,9 +636,9 @@ MacroAssemblerCodeRef<JITThunkPtrTag> stringGetByValGenerator(VM* vm) > jit.tagReturnAddress(); > > // Load string length to regT2, and start the process of loading the data pointer into regT0 >- jit.load32(JSInterfaceJIT::Address(stringGPR, JSString::offsetOfLength()), scratchGPR); >+ failures.append(jit.branchIfRopeString(stringGPR)); > jit.loadPtr(JSInterfaceJIT::Address(stringGPR, JSString::offsetOfValue()), stringGPR); >- failures.append(jit.branchTestPtr(JSInterfaceJIT::Zero, stringGPR)); >+ jit.load32(JSInterfaceJIT::Address(stringGPR, StringImpl::lengthMemoryOffset()), scratchGPR); > > // Do an unsigned compare to simultaneously filter negative indices as well as indices that are too large > failures.append(jit.branch32(JSInterfaceJIT::AboveOrEqual, indexGPR, scratchGPR)); >@@ -675,9 +675,9 @@ static void stringCharLoad(SpecializedThunkJIT& jit) > jit.loadJSStringArgument(SpecializedThunkJIT::ThisArgument, SpecializedThunkJIT::regT0); > > // Load string length to regT2, and start the process of loading the data pointer into regT0 >- jit.load32(MacroAssembler::Address(SpecializedThunkJIT::regT0, JSString::offsetOfLength()), SpecializedThunkJIT::regT2); >+ jit.appendFailure(jit.branchIfRopeString(SpecializedThunkJIT::regT0)); > jit.loadPtr(MacroAssembler::Address(SpecializedThunkJIT::regT0, JSString::offsetOfValue()), SpecializedThunkJIT::regT0); >- jit.appendFailure(jit.branchTest32(MacroAssembler::Zero, SpecializedThunkJIT::regT0)); >+ jit.load32(MacroAssembler::Address(SpecializedThunkJIT::regT0, StringImpl::lengthMemoryOffset()), SpecializedThunkJIT::regT2); > > // load index > jit.loadInt32Argument(0, SpecializedThunkJIT::regT1); // regT1 contains the index >diff --git a/Source/JavaScriptCore/llint/LowLevelInterpreter.asm b/Source/JavaScriptCore/llint/LowLevelInterpreter.asm >index d86857dcf65455f06ac420196730600f0d9926ad..5a1a42b98cdfe472a330a14c1363c0a556d0db67 100644 >--- a/Source/JavaScriptCore/llint/LowLevelInterpreter.asm >+++ b/Source/JavaScriptCore/llint/LowLevelInterpreter.asm >@@ -462,6 +462,7 @@ const NumberOfTypedArrayTypesExcludingDataView = constexpr NumberOfTypedArrayTyp > # Type flags constants. > const MasqueradesAsUndefined = constexpr MasqueradesAsUndefined > const ImplementsDefaultHasInstance = constexpr ImplementsDefaultHasInstance >+const TypeInfoPerCellBit = constexpr TypeInfoPerCellBit > > # Bytecode operand constants. > const FirstConstantRegisterIndexNarrow = 16 >diff --git a/Source/JavaScriptCore/llint/LowLevelInterpreter32_64.asm b/Source/JavaScriptCore/llint/LowLevelInterpreter32_64.asm >index ff1ce5869e3691f44499861712dddb7e6dcfb7f3..9bce5fbfef672abba43187ff4dd09881977183ed 100644 >--- a/Source/JavaScriptCore/llint/LowLevelInterpreter32_64.asm >+++ b/Source/JavaScriptCore/llint/LowLevelInterpreter32_64.asm >@@ -1788,9 +1788,9 @@ llintOpWithJump(op_switch_char, OpSwitchChar, macro (size, get, jump, dispatch) > addp t3, t2 > bineq t1, CellTag, .opSwitchCharFallThrough > bbneq JSCell::m_type[t0], StringType, .opSwitchCharFallThrough >- bineq JSString::m_length[t0], 1, .opSwitchCharFallThrough >+ btbnz JSCell::m_flags[t0], TypeInfoPerCellBit, .opSwitchOnRope > loadp JSString::m_value[t0], t0 >- btpz t0, .opSwitchOnRope >+ bineq StringImpl::m_length[t0], 1, .opSwitchCharFallThrough > loadp StringImpl::m_data8[t0], t1 > btinz StringImpl::m_hashAndFlags[t0], HashFlags8BitBuffer, .opSwitchChar8Bit > loadh [t1], t0 >diff --git a/Source/JavaScriptCore/llint/LowLevelInterpreter64.asm b/Source/JavaScriptCore/llint/LowLevelInterpreter64.asm >index 45825401a78b680fae0a8f0962cdc3501599aa1f..9b61aaaa31c7a0e553962fb7f2bbfd5221d7a8ef 100644 >--- a/Source/JavaScriptCore/llint/LowLevelInterpreter64.asm >+++ b/Source/JavaScriptCore/llint/LowLevelInterpreter64.asm >@@ -1873,9 +1873,9 @@ llintOpWithJump(op_switch_char, OpSwitchChar, macro (size, get, jump, dispatch) > addp t3, t2 > btqnz t1, tagMask, .opSwitchCharFallThrough > bbneq JSCell::m_type[t1], StringType, .opSwitchCharFallThrough >- bineq JSString::m_length[t1], 1, .opSwitchCharFallThrough >+ btbnz JSCell::m_flags[t1], TypeInfoPerCellBit, .opSwitchOnRope > loadp JSString::m_value[t1], t0 >- btpz t0, .opSwitchOnRope >+ bineq StringImpl::m_length[t0], 1, .opSwitchCharFallThrough > loadp StringImpl::m_data8[t0], t1 > btinz StringImpl::m_hashAndFlags[t0], HashFlags8BitBuffer, .opSwitchChar8Bit > loadh [t1], t0 >diff --git a/Source/JavaScriptCore/runtime/JSString.cpp b/Source/JavaScriptCore/runtime/JSString.cpp >index d95e25cd02c7d5d32fa312410e97e6c512941298..d882f27219785513d271e5c270fa6a5a040bb359 100644 >--- a/Source/JavaScriptCore/runtime/JSString.cpp >+++ b/Source/JavaScriptCore/runtime/JSString.cpp >@@ -61,14 +61,17 @@ void JSString::dumpToStream(const JSCell* cell, PrintStream& out) > VM& vm = *cell->vm(); > const JSString* thisObject = jsCast<const JSString*>(cell); > out.printf("<%p, %s, [%u], ", thisObject, thisObject->className(vm), thisObject->length()); >- if (thisObject->isRope()) >+ bool isRope = thisObject->isRope(); >+ Dependency isRopeDependency = Dependency::fence(isRope); >+ if (isRope) > out.printf("[rope]"); > else { >- WTF::StringImpl* ourImpl = thisObject->m_value.impl(); >- if (ourImpl->is8Bit()) >- out.printf("[8 %p]", ourImpl->characters8()); >- else >- out.printf("[16 %p]", ourImpl->characters16()); >+ if (WTF::StringImpl* ourImpl = isRopeDependency.consume(thisObject)->m_value.impl()) { >+ if (ourImpl->is8Bit()) >+ out.printf("[8 %p]", ourImpl->characters8()); >+ else >+ out.printf("[16 %p]", ourImpl->characters16()); >+ } > } > out.printf(">"); > } >@@ -86,9 +89,11 @@ bool JSString::equalSlowCase(ExecState* exec, JSString* other) const > size_t JSString::estimatedSize(JSCell* cell, VM& vm) > { > JSString* thisObject = asString(cell); >- if (thisObject->isRope()) >+ bool isRope = thisObject->isRope(); >+ Dependency isRopeDependency = Dependency::fence(isRope); >+ if (isRope) > return Base::estimatedSize(cell, vm); >- return Base::estimatedSize(cell, vm) + thisObject->m_value.impl()->costDuringGC(); >+ return Base::estimatedSize(cell, vm) + isRopeDependency.consume(thisObject)->m_value.impl()->costDuringGC(); > } > > void JSString::visitChildren(JSCell* cell, SlotVisitor& visitor) >@@ -96,20 +101,45 @@ void JSString::visitChildren(JSCell* cell, SlotVisitor& visitor) > JSString* thisObject = asString(cell); > Base::visitChildren(thisObject, visitor); > >- if (thisObject->isRope()) >- static_cast<JSRopeString*>(thisObject)->visitFibers(visitor); >- if (StringImpl* impl = thisObject->m_value.impl()) >- visitor.reportExtraMemoryVisited(impl->costDuringGC()); >-} >- >-void JSRopeString::visitFibers(SlotVisitor& visitor) >-{ >- if (isSubstring()) { >- visitor.append(substringBase()); >+ if (visitor.mutatorIsStopped()) { >+ if (thisObject->isRope()) { >+ JSRopeString* thisRopeString = static_cast<JSRopeString*>(thisObject); >+ if (thisRopeString->isSubstring()) >+ visitor.appendUnbarriered(thisRopeString->substringBase()); >+ else { >+ for (unsigned index = 0; index < JSRopeString::s_maxInternalRopeLength && thisRopeString->fiber(index); ++index) >+ visitor.appendUnbarriered(thisRopeString->fiber(index)); >+ } >+ return; >+ } >+ if (StringImpl* impl = thisObject->m_value.impl()) >+ visitor.reportExtraMemoryVisited(impl->costDuringGC()); > return; > } >- for (size_t i = 0; i < s_maxInternalRopeLength && fiber(i); ++i) >- visitor.append(fiber(i)); >+ >+ bool isRope = thisObject->isRope(); >+ Dependency dependency = Dependency::fence(isRope); >+ if (isRope) { >+ FiberSlot fibers[JSRopeString::s_maxInternalRopeLength] { { 0 }, { 0 }, { 0 } }; >+ for (unsigned index = 0; index < JSRopeString::s_maxInternalRopeLength; ++index) { >+ FiberSlot fiber = dependency.consume(static_cast<JSRopeString*>(thisObject))->fibers()[index]; >+ fibers[index] = WTFMove(fiber); >+ dependency = Dependency::fence(fiber); >+ } >+ >+ if (dependency.consume(thisObject)->isRope()) { >+ WTF::loadLoadFence(); >+ if (fibers[0].number == JSRopeString::substringSentinel()) >+ visitor.appendUnbarriered(bitwise_cast<JSString*>(fibers[1].number & JSRopeString::stringMask)); >+ else { >+ for (unsigned index = 0; index < JSRopeString::s_maxInternalRopeLength && (fibers[index].number & JSRopeString::stringMask); ++index) >+ visitor.appendUnbarriered(bitwise_cast<JSString*>(fibers[index].number & JSRopeString::stringMask)); >+ } >+ return; >+ } >+ } >+ if (StringImpl* impl = dependency.consume(thisObject)->m_value.impl()) >+ visitor.reportExtraMemoryVisited(impl->costDuringGC()); > } > > static const unsigned maxLengthForOnStackResolve = 2048; >@@ -184,44 +214,54 @@ void JSRopeString::resolveRopeToAtomicString(ExecState* exec) const > if (length() > maxLengthForOnStackResolve) { > resolveRope(exec); > RETURN_IF_EXCEPTION(scope, void()); >+ ASSERT(!isRope()); > m_value = AtomicString(m_value); >- setIs8Bit(m_value.impl()->is8Bit()); > return; > } > > if (is8Bit()) { > LChar buffer[maxLengthForOnStackResolve]; > resolveRopeInternal8(buffer); >- m_value = AtomicString(buffer, length()); >- setIs8Bit(m_value.impl()->is8Bit()); >+ convertToNonRope(vm, AtomicStringImpl::add(buffer, length())); > } else { > UChar buffer[maxLengthForOnStackResolve]; > resolveRopeInternal16(buffer); >- m_value = AtomicString(buffer, length()); >- setIs8Bit(m_value.impl()->is8Bit()); >+ convertToNonRope(vm, AtomicStringImpl::add(buffer, length())); > } >- >- clearFibers(); >+ ASSERT(!isRope()); > > // If we resolved a string that didn't previously exist, notify the heap that we've grown. > if (m_value.impl()->hasOneRef()) > vm.heap.reportExtraMemoryAllocated(m_value.impl()->cost()); > } > >-void JSRopeString::clearFibers() const >+inline void JSRopeString::convertToNonRope(VM& vm, String&& string) const > { >- for (size_t i = 0; i < s_maxInternalRopeLength; ++i) >- u[i].number = 0; >+ ASSERT(isRope()); >+ if (isX86() || vm.heap.mutatorShouldBeFenced()) { >+ new (&m_value) String(); >+ WTF::storeStoreFence(); >+ const_cast<JSRopeString*>(this)->setPerCellBit(false); >+ WTF::storeStoreFence(); >+ m_value = WTFMove(string); >+ } else { >+ const_cast<JSRopeString*>(this)->setPerCellBit(false); >+ new (&m_value) String(WTFMove(string)); >+ } >+ ASSERT(!isRope()); > } > > RefPtr<AtomicStringImpl> JSRopeString::resolveRopeToExistingAtomicString(ExecState* exec) const > { >+ VM& vm = exec->vm(); >+ auto scope = DECLARE_THROW_SCOPE(vm); >+ > if (length() > maxLengthForOnStackResolve) { > resolveRope(exec); >+ RETURN_IF_EXCEPTION(scope, nullptr); >+ ASSERT(!isRope()); > if (RefPtr<AtomicStringImpl> existingAtomicString = AtomicStringImpl::lookUp(m_value.impl())) { > m_value = *existingAtomicString; >- setIs8Bit(m_value.impl()->is8Bit()); >- clearFibers(); > return existingAtomicString; > } > return nullptr; >@@ -231,18 +271,14 @@ RefPtr<AtomicStringImpl> JSRopeString::resolveRopeToExistingAtomicString(ExecSta > LChar buffer[maxLengthForOnStackResolve]; > resolveRopeInternal8(buffer); > if (RefPtr<AtomicStringImpl> existingAtomicString = AtomicStringImpl::lookUp(buffer, length())) { >- m_value = *existingAtomicString; >- setIs8Bit(m_value.impl()->is8Bit()); >- clearFibers(); >+ convertToNonRope(vm, *existingAtomicString); > return existingAtomicString; > } > } else { > UChar buffer[maxLengthForOnStackResolve]; > resolveRopeInternal16(buffer); > if (RefPtr<AtomicStringImpl> existingAtomicString = AtomicStringImpl::lookUp(buffer, length())) { >- m_value = *existingAtomicString; >- setIs8Bit(m_value.impl()->is8Bit()); >- clearFibers(); >+ convertToNonRope(vm, *existingAtomicString); > return existingAtomicString; > } > } >@@ -254,39 +290,40 @@ void JSRopeString::resolveRope(ExecState* nullOrExecForOOM) const > { > ASSERT(isRope()); > >+ VM& vm = *this->vm(); > if (isSubstring()) { > ASSERT(!substringBase()->isRope()); >- m_value = substringBase()->m_value.substringSharingImpl(substringOffset(), length()); >- substringBase().clear(); >+ auto newImpl = substringBase()->m_value.substringSharingImpl(substringOffset(), length()); >+ convertToNonRope(vm, WTFMove(newImpl)); >+ ASSERT(!isRope()); > return; > } > > if (is8Bit()) { > LChar* buffer; >- if (auto newImpl = StringImpl::tryCreateUninitialized(length(), buffer)) { >- Heap::heap(this)->reportExtraMemoryAllocated(newImpl->cost()); >- m_value = WTFMove(newImpl); >- } else { >+ auto newImpl = StringImpl::tryCreateUninitialized(length(), buffer); >+ if (!newImpl) { > outOfMemory(nullOrExecForOOM); > return; > } >+ vm.heap.reportExtraMemoryAllocated(newImpl->cost()); >+ > resolveRopeInternal8NoSubstring(buffer); >- clearFibers(); >+ convertToNonRope(vm, WTFMove(newImpl)); > ASSERT(!isRope()); > return; > } > > UChar* buffer; >- if (auto newImpl = StringImpl::tryCreateUninitialized(length(), buffer)) { >- Heap::heap(this)->reportExtraMemoryAllocated(newImpl->cost()); >- m_value = WTFMove(newImpl); >- } else { >+ auto newImpl = StringImpl::tryCreateUninitialized(length(), buffer); >+ if (!newImpl) { > outOfMemory(nullOrExecForOOM); > return; > } >+ vm.heap.reportExtraMemoryAllocated(newImpl->cost()); > > resolveRopeInternal16NoSubstring(buffer); >- clearFibers(); >+ convertToNonRope(vm, WTFMove(newImpl)); > ASSERT(!isRope()); > } > >@@ -306,7 +343,7 @@ void JSRopeString::resolveRopeSlowCase8(LChar* buffer) const > Vector<JSString*, 32, UnsafeVectorOverflow> workQueue; // Putting strings into a Vector is only OK because there are no GC points in this method. > > for (size_t i = 0; i < s_maxInternalRopeLength && fiber(i); ++i) >- workQueue.append(fiber(i).get()); >+ workQueue.append(fiber(i)); > > while (!workQueue.isEmpty()) { > JSString* currentFiber = workQueue.last(); >@@ -318,7 +355,7 @@ void JSRopeString::resolveRopeSlowCase8(LChar* buffer) const > JSRopeString* currentFiberAsRope = static_cast<JSRopeString*>(currentFiber); > if (!currentFiberAsRope->isSubstring()) { > for (size_t i = 0; i < s_maxInternalRopeLength && currentFiberAsRope->fiber(i); ++i) >- workQueue.append(currentFiberAsRope->fiber(i).get()); >+ workQueue.append(currentFiberAsRope->fiber(i)); > continue; > } > ASSERT(!currentFiberAsRope->substringBase()->isRope()); >@@ -342,7 +379,7 @@ void JSRopeString::resolveRopeSlowCase(UChar* buffer) const > Vector<JSString*, 32, UnsafeVectorOverflow> workQueue; // These strings are kept alive by the parent rope, so using a Vector is OK. > > for (size_t i = 0; i < s_maxInternalRopeLength && fiber(i); ++i) >- workQueue.append(fiber(i).get()); >+ workQueue.append(fiber(i)); > > while (!workQueue.isEmpty()) { > JSString* currentFiber = workQueue.last(); >@@ -364,7 +401,7 @@ void JSRopeString::resolveRopeSlowCase(UChar* buffer) const > continue; > } > for (size_t i = 0; i < s_maxInternalRopeLength && currentFiberAsRope->fiber(i); ++i) >- workQueue.append(currentFiberAsRope->fiber(i).get()); >+ workQueue.append(currentFiberAsRope->fiber(i)); > continue; > } > >@@ -382,9 +419,10 @@ void JSRopeString::resolveRopeSlowCase(UChar* buffer) const > > void JSRopeString::outOfMemory(ExecState* nullOrExecForOOM) const > { >- clearFibers(); > ASSERT(isRope()); >- ASSERT(m_value.isNull()); >+ fibers()[0].string = nullptr; >+ fibers()[1].string = nullptr; >+ fibers()[2].string = nullptr; > if (nullOrExecForOOM) { > VM& vm = nullOrExecForOOM->vm(); > auto scope = DECLARE_THROW_SCOPE(vm); >diff --git a/Source/JavaScriptCore/runtime/JSString.h b/Source/JavaScriptCore/runtime/JSString.h >index 00afd41432204d0042ba7a95390641d17bddcc92..a1d1d3a9d9bce5640ed5606d7f2eea1f7e8d3a7d 100644 >--- a/Source/JavaScriptCore/runtime/JSString.h >+++ b/Source/JavaScriptCore/runtime/JSString.h >@@ -111,22 +111,21 @@ class JSString : public JSCell { > JSString(VM& vm) > : JSCell(vm, vm.stringStructure.get()) > { >+ m_fibers[0].string = nullptr; > } > > void finishCreation(VM& vm, unsigned length) > { >+ UNUSED_PARAM(length); > ASSERT(!m_value.isNull()); > Base::finishCreation(vm); >- setLength(length); >- setIs8Bit(m_value.impl()->is8Bit()); > } > > void finishCreation(VM& vm, unsigned length, size_t cost) > { >+ UNUSED_PARAM(length); > ASSERT(!m_value.isNull()); > Base::finishCreation(vm); >- setLength(length); >- setIs8Bit(m_value.impl()->is8Bit()); > vm.heap.reportExtraMemoryAllocated(cost); > } > >@@ -134,11 +133,11 @@ class JSString : public JSCell { > void finishCreation(VM& vm) > { > Base::finishCreation(vm); >- setLength(0); >- setIs8Bit(true); > } > > public: >+ ~JSString(); >+ > static JSString* create(VM& vm, Ref<StringImpl>&& value) > { > unsigned length = value->length(); >@@ -165,7 +164,7 @@ class JSString : public JSCell { > const String& value(ExecState*) const; > inline const String& tryGetValue() const; > const StringImpl* tryGetValueImpl() const; >- ALWAYS_INLINE unsigned length() const { return m_length; } >+ ALWAYS_INLINE unsigned length() const; > > JSValue toPrimitive(ExecState*, PreferredPrimitiveType) const; > bool toBoolean() const { return !!length(); } >@@ -182,9 +181,7 @@ class JSString : public JSCell { > > static Structure* createStructure(VM&, JSGlobalObject*, JSValue); > >- static size_t offsetOfLength() { return OBJECT_OFFSETOF(JSString, m_length); } >- static size_t offsetOfFlags() { return OBJECT_OFFSETOF(JSString, m_flags); } >- static size_t offsetOfValue() { return OBJECT_OFFSETOF(JSString, m_value); } >+ static ptrdiff_t offsetOfValue() { return OBJECT_OFFSETOF(JSString, m_value); } > > DECLARE_EXPORT_INFO; > >@@ -192,44 +189,39 @@ class JSString : public JSCell { > static size_t estimatedSize(JSCell*, VM&); > static void visitChildren(JSCell*, SlotVisitor&); > >- enum { >- Is8Bit = 1u >- }; >+ bool isRope() const { return perCellBit(); } > > protected: >+ ALWAYS_INLINE bool isRopeConcurrently() const >+ { >+ // This function can be called concurrently from compiler threads. >+ // If m_value is nullptr, it may be in the middle of converting JSRopeString to JSString. >+ // In that case, we use length of JSRopeString code. To make it work, JSRopeString does not clear >+ // the embedded length values even after JSString conversion happens. >+ return isRope() || m_value.isNull(); >+ } >+ > friend class JSValue; > > JS_EXPORT_PRIVATE bool equalSlowCase(ExecState*, JSString* other) const; >- bool isRope() const { return m_value.isNull(); } > bool isSubstring() const; >- bool is8Bit() const { return m_flags & Is8Bit; } >- void setIs8Bit(bool flag) const >- { >- if (flag) >- m_flags |= Is8Bit; >- else >- m_flags &= ~Is8Bit; >- } >+ bool is8Bit() const; > >- ALWAYS_INLINE void setLength(unsigned length) >- { >- ASSERT(length <= MaxLength); >- m_length = length; >- } >+ union FiberSlot { >+ uintptr_t number; >+ JSString* string; >+ }; > >-private: >- // A string is represented either by a String or a rope of fibers. >- unsigned m_length { 0 }; >- mutable uint16_t m_flags { 0 }; >- // The poison is strategically placed and holds a value such that the first >- // 64 bits of JSString look like a double JSValue. >- mutable String m_value; >+ union { >+ mutable String m_value; >+ mutable FiberSlot m_fibers[1]; >+ }; > >+private: > friend class LLIntOffsetsExtractor; > > static JSValue toThis(JSCell*, ExecState*, ECMAMode); > >- String& string() { ASSERT(!isRope()); return m_value; } > StringView unsafeView(ExecState*) const; > > friend JSString* jsString(ExecState*, JSString*, JSString*); >@@ -294,34 +286,81 @@ class JSRopeString final : public JSString { > size_t m_index; > }; > >+ static constexpr unsigned Is8Bit = StringImpl::flagIs8Bit(); >+ static_assert(Is8Bit == 0b100, ""); >+ static constexpr uint64_t lengthMask = 0xffff000000000000ULL; >+ static constexpr uint64_t flagsMask = Is8Bit; >+ static constexpr uint64_t stringMask = ~(lengthMask | flagsMask); >+ >+ inline bool is8Bit() const >+ { >+ return fibers()[1].number & Is8Bit; >+ } >+ inline unsigned length() const >+ { >+ return (fibers()[1].number >> 48) | ((fibers()[2].number >> 48) << 16); >+ } >+ > private: >+ void convertToNonRope(VM& vm, String&& string) const; >+ >+ void setIs8Bit(bool flag) const >+ { >+ if (flag) >+ fibers()[1].number |= static_cast<uint64_t>(Is8Bit); >+ else >+ fibers()[1].number &= ~static_cast<uint64_t>(Is8Bit); >+ } >+ >+ ALWAYS_INLINE void setLength(unsigned length) >+ { >+ ASSERT(length <= MaxLength); >+ ASSERT(!(fibers()[1].number & lengthMask)); >+ ASSERT(!(fibers()[2].number & lengthMask)); >+ fibers()[1].number |= (static_cast<uint64_t>(static_cast<uint16_t>(length)) << 48); >+ fibers()[2].number |= (static_cast<uint64_t>(static_cast<uint16_t>(length >> 16)) << 48); >+ } >+ >+ ALWAYS_INLINE void resetLength(unsigned length) >+ { >+ ASSERT(length <= MaxLength); >+ fibers()[1].number = (fibers()[1].number & ~lengthMask) | (static_cast<uint64_t>(static_cast<uint16_t>(length)) << 48); >+ fibers()[2].number = (fibers()[2].number & ~lengthMask) | (static_cast<uint64_t>(static_cast<uint16_t>(length >> 16)) << 48); >+ } >+ > ALWAYS_INLINE JSRopeString(VM& vm) > : JSString(vm) > { >+ m_additionalFibers[0].string = nullptr; >+ m_additionalFibers[1].string = nullptr; >+ setPerCellBit(true); >+ ASSERT(additionalFibers() == (&fibers()[1])); > } > > void finishCreation(VM& vm, JSString* s1, JSString* s2) > { > Base::finishCreation(vm); > ASSERT(!sumOverflows<int32_t>(s1->length(), s2->length())); >+ setIsSubstring(false); >+ setFiber(vm, 0, s1); >+ setFiber(vm, 1, s2); >+ setFiber(vm, 2, nullptr); > setLength(s1->length() + s2->length()); > setIs8Bit(s1->is8Bit() && s2->is8Bit()); >- setIsSubstring(false); >- fiber(0).set(vm, this, s1); >- fiber(1).set(vm, this, s2); >- fiber(2).clear(); >+ ASSERT((s1->length() + s2->length()) == length()); > } > > void finishCreation(VM& vm, JSString* s1, JSString* s2, JSString* s3) > { > Base::finishCreation(vm); > ASSERT(!sumOverflows<int32_t>(s1->length(), s2->length(), s3->length())); >+ setIsSubstring(false); >+ setFiber(vm, 0, s1); >+ setFiber(vm, 1, s2); >+ setFiber(vm, 2, s3); > setLength(s1->length() + s2->length() + s3->length()); > setIs8Bit(s1->is8Bit() && s2->is8Bit() && s3->is8Bit()); >- setIsSubstring(false); >- fiber(0).set(vm, this, s1); >- fiber(1).set(vm, this, s2); >- fiber(2).set(vm, this, s3); >+ ASSERT((s1->length() + s2->length() + s3->length()) == length()); > } > > void finishCreation(VM& vm, ExecState* exec, JSString* base, unsigned offset, unsigned length) >@@ -329,16 +368,14 @@ class JSRopeString final : public JSString { > Base::finishCreation(vm); > RELEASE_ASSERT(!sumOverflows<int32_t>(offset, length)); > RELEASE_ASSERT(offset + length <= base->length()); >- setLength(length); >- setIs8Bit(base->is8Bit()); > setIsSubstring(true); > if (base->isSubstring()) { > JSRopeString* baseRope = jsCast<JSRopeString*>(base); >- substringBase().set(vm, this, baseRope->substringBase().get()); >- substringOffset() = baseRope->substringOffset() + offset; >+ setSubstringBase(vm, baseRope->substringBase()); >+ setSubstringOffset(baseRope->substringOffset() + offset); > } else { >- substringBase().set(vm, this, base); >- substringOffset() = offset; >+ setSubstringBase(vm, base); >+ setSubstringOffset(offset); > > // For now, let's not allow substrings with a rope base. > // Resolve non-substring rope bases so we don't have to deal with it. >@@ -346,6 +383,9 @@ class JSRopeString final : public JSString { > if (base->isRope()) > jsCast<JSRopeString*>(base)->resolveRope(exec); > } >+ setLength(length); >+ setIs8Bit(base->is8Bit()); >+ ASSERT(length == this->length()); > } > > ALWAYS_INLINE void finishCreationSubstringOfResolved(VM& vm, JSString* base, unsigned offset, unsigned length) >@@ -353,27 +393,34 @@ class JSRopeString final : public JSString { > Base::finishCreation(vm); > RELEASE_ASSERT(!sumOverflows<int32_t>(offset, length)); > RELEASE_ASSERT(offset + length <= base->length()); >+ setIsSubstring(true); >+ setSubstringBase(vm, base); >+ setSubstringOffset(offset); > setLength(length); > setIs8Bit(base->is8Bit()); >- setIsSubstring(true); >- substringBase().set(vm, this, base); >- substringOffset() = offset; >+ ASSERT(length == this->length()); > } > > void finishCreation(VM& vm) > { > JSString::finishCreation(vm); > setIsSubstring(false); >- fiber(0).clear(); >- fiber(1).clear(); >- fiber(2).clear(); >+ setFiber(vm, 0, nullptr); >+ setFiber(vm, 1, nullptr); >+ setFiber(vm, 2, nullptr); >+ setLength(0); >+ setIs8Bit(true); >+ ASSERT(0 == length()); > } > > void append(VM& vm, size_t index, JSString* jsString) > { >- fiber(index).set(vm, this, jsString); >- setLength(length() + jsString->length()); >- setIs8Bit(is8Bit() && jsString->is8Bit()); >+ bool is8Bit = this->is8Bit() && jsString->is8Bit(); >+ unsigned length = this->length() + jsString->length(); >+ setFiber(vm, index, jsString); >+ resetLength(length); >+ setIs8Bit(is8Bit); >+ ASSERT(length == this->length()); > } > > static JSRopeString* createNull(VM& vm) >@@ -403,9 +450,13 @@ class JSRopeString final : public JSString { > return createSubstringOfResolved(vm, nullptr, base, offset, length); > } > >- void visitFibers(SlotVisitor&); >+ static ptrdiff_t offsetOfFlags() { return offsetOfFibers() + 1 * sizeof(JSString*); } // 32byte width. But only "or" is allowd. >+ static ptrdiff_t offsetOfLowerLength() { return offsetOfFibers() + 1 * sizeof(JSString*) + sizeof(uint16_t) * 3; } // 16byte width. >+ static ptrdiff_t offsetOfUpperLength() { return offsetOfFibers() + 2 * sizeof(JSString*) + sizeof(uint16_t) * 3; } // 16byte width. > >- static ptrdiff_t offsetOfFibers() { return OBJECT_OFFSETOF(JSRopeString, u); } >+ FiberSlot* fibers() const { return m_fibers; } >+ FiberSlot* additionalFibers() const { return m_additionalFibers; } >+ static ptrdiff_t offsetOfFibers() { return OBJECT_OFFSETOF(JSRopeString, m_fibers); } > > static const unsigned s_maxInternalRopeLength = 3; > >@@ -438,52 +489,67 @@ class JSRopeString final : public JSString { > void resolveRopeInternal8NoSubstring(LChar*) const; > void resolveRopeInternal16(UChar*) const; > void resolveRopeInternal16NoSubstring(UChar*) const; >- void clearFibers() const; > StringView unsafeView(ExecState*) const; > StringViewWithUnderlyingString viewWithUnderlyingString(ExecState*) const; > >- WriteBarrierBase<JSString>& fiber(unsigned i) const >+ JSString* fiber(unsigned i) const > { > ASSERT(!isSubstring()); > ASSERT(i < s_maxInternalRopeLength); >- return u[i].string; >+ return bitwise_cast<JSString*>(fibers()[i].number & stringMask); > } > >- WriteBarrierBase<JSString>& substringBase() const >+ void setFiber(VM& vm, unsigned i, JSString* fiber) > { >- return u[1].string; >+ ASSERT(!isSubstring()); >+ ASSERT(i < s_maxInternalRopeLength); >+ fibers()[i].string = fiber; >+ vm.heap.writeBarrier(this, fiber); >+ } >+ >+ void setSubstringBase(VM& vm, JSString* fiber) >+ { >+ fibers()[1].string = fiber; >+ vm.heap.writeBarrier(this, fiber); > } > >- uintptr_t& substringOffset() const >+ JSString* substringBase() const > { >- return u[2].number; >+ return bitwise_cast<JSString*>(fibers()[1].number & stringMask); > } > >- static uintptr_t notSubstringSentinel() >+ void setSubstringOffset(unsigned offset) >+ { >+ fibers()[2].number = offset; >+ } >+ >+ unsigned substringOffset() const >+ { >+ return fibers()[2].number & ~lengthMask; >+ } >+ >+ static constexpr uintptr_t notSubstringSentinel() > { > return 0; > } > >- static uintptr_t substringSentinel() >+ static constexpr uintptr_t substringSentinel() > { > return 1; > } > > bool isSubstring() const > { >- return u[0].number == substringSentinel(); >+ return fibers()[0].number == substringSentinel(); > } > > void setIsSubstring(bool isSubstring) > { >- u[0].number = isSubstring ? substringSentinel() : notSubstringSentinel(); >+ fibers()[0].number = isSubstring ? substringSentinel() : notSubstringSentinel(); > } > >- mutable union { >- uintptr_t number; >- WriteBarrierBase<JSString> string; >- } u[s_maxInternalRopeLength]; >- >+ static_assert(s_maxInternalRopeLength >= 2, ""); >+ mutable FiberSlot m_additionalFibers[s_maxInternalRopeLength - 1]; // This is touched through m_fibers. > > friend JSString* jsString(ExecState*, JSString*, JSString*); > friend JSString* jsString(ExecState*, JSString*, JSString*, JSString*); >@@ -492,9 +558,27 @@ class JSRopeString final : public JSString { > > JS_EXPORT_PRIVATE JSString* jsStringWithCacheSlowCase(VM&, StringImpl&); > >+inline bool JSString::is8Bit() const >+{ >+ if (isRopeConcurrently()) >+ return static_cast<const JSRopeString*>(this)->is8Bit(); >+ return m_value.is8Bit(); >+} >+ >+ALWAYS_INLINE unsigned JSString::length() const >+{ >+ if (isRopeConcurrently()) >+ return static_cast<const JSRopeString*>(this)->length(); >+ return m_value.length(); >+} >+ > inline const StringImpl* JSString::tryGetValueImpl() const > { >- return m_value.impl(); >+ bool isRope = this->isRope(); >+ Dependency dependency = Dependency::fence(isRope); >+ if (isRope) >+ return nullptr; >+ return dependency.consume(this)->m_value.impl(); > } > > inline JSString* asString(JSValue value) >diff --git a/Source/JavaScriptCore/runtime/JSStringInlines.h b/Source/JavaScriptCore/runtime/JSStringInlines.h >index 19d4756e0c1a9bcee73bb8a5b0cae6e03373074c..e3132e3bfa00ebd4911366560e3266e548b18790 100644 >--- a/Source/JavaScriptCore/runtime/JSStringInlines.h >+++ b/Source/JavaScriptCore/runtime/JSStringInlines.h >@@ -29,6 +29,13 @@ > > namespace JSC { > >+inline JSString::~JSString() >+{ >+ if (isRope()) >+ return; >+ m_value.~String(); >+} >+ > bool JSString::equal(ExecState* exec, JSString* other) const > { > if (isRope() || other->isRope()) >diff --git a/Source/WTF/wtf/text/StringImpl.h b/Source/WTF/wtf/text/StringImpl.h >index 85ca50e249a97c54217c28f5902f3c54dc352373..7f4ed36c3a704532c5c9c34f3a57fe89d2982389 100644 >--- a/Source/WTF/wtf/text/StringImpl.h >+++ b/Source/WTF/wtf/text/StringImpl.h >@@ -200,8 +200,8 @@ class StringImpl : private StringImplShape { > static constexpr const unsigned s_hashFlagStringKindIsAtomic = 1u << (s_flagStringKindCount); > static constexpr const unsigned s_hashFlagStringKindIsSymbol = 1u << (s_flagStringKindCount + 1); > static constexpr const unsigned s_hashMaskStringKind = s_hashFlagStringKindIsAtomic | s_hashFlagStringKindIsSymbol; >- static constexpr const unsigned s_hashFlag8BitBuffer = 1u << 3; >- static constexpr const unsigned s_hashFlagDidReportCost = 1u << 2; >+ static constexpr const unsigned s_hashFlagDidReportCost = 1u << 3; >+ static constexpr const unsigned s_hashFlag8BitBuffer = 1u << 2; > static constexpr const unsigned s_hashMaskBufferOwnership = (1u << 0) | (1u << 1); > > enum StringKind { >@@ -264,10 +264,10 @@ class StringImpl : private StringImplShape { > static Expected<Ref<StringImpl>, UTF8ConversionError> tryReallocate(Ref<StringImpl>&& originalString, unsigned length, UChar*& data); > > static unsigned flagsOffset() { return OBJECT_OFFSETOF(StringImpl, m_hashAndFlags); } >- static unsigned flagIs8Bit() { return s_hashFlag8BitBuffer; } >- static unsigned flagIsAtomic() { return s_hashFlagStringKindIsAtomic; } >- static unsigned flagIsSymbol() { return s_hashFlagStringKindIsSymbol; } >- static unsigned maskStringKind() { return s_hashMaskStringKind; } >+ static constexpr unsigned flagIs8Bit() { return s_hashFlag8BitBuffer; } >+ static constexpr unsigned flagIsAtomic() { return s_hashFlagStringKindIsAtomic; } >+ static constexpr unsigned flagIsSymbol() { return s_hashFlagStringKindIsSymbol; } >+ static constexpr unsigned maskStringKind() { return s_hashMaskStringKind; } > static unsigned dataOffset() { return OBJECT_OFFSETOF(StringImpl, m_data8); } > > template<typename CharacterType, size_t inlineCapacity, typename OverflowHandler, size_t minCapacity>
You cannot view the attachment while viewing its details because your browser does not support IFRAMEs.
View the attachment on a separate page
.
View Attachment As Diff
View Attachment As Raw
Actions:
View
|
Formatted Diff
|
Diff
Attachments on
bug 194375
:
361566
|
361603
|
361717
|
361778
|
361785
|
361786
|
361854
|
361868
|
361943
|
361958
|
361961
|
361962
|
362097
|
362362
|
362369
|
362370
|
362379
|
362480
|
362595
|
362624
|
362625
|
362627
|
362629
|
362632
|
362638
|
362658
|
362661
|
362669
|
362672
|
362676
|
362686
|
362816
|
362817
|
363064
|
363076
|
363077
|
363080
|
363288