WebKit Bugzilla
Attachment 362617 Details for
Bug 194867
: Use a SQLite database to hold the ResourceLoadStatistics data
Home
|
New
|
Browse
|
Search
|
[?]
|
Reports
|
Requests
|
Help
|
New Account
|
Log In
Remember
[x]
|
Forgot Password
Login:
[x]
[patch]
WIP Patch
bug-194867-20190221101225.patch (text/plain), 253.71 KB, created by
Brent Fulgham
on 2019-02-21 10:12:26 PST
(
hide
)
Description:
WIP Patch
Filename:
MIME Type:
Creator:
Brent Fulgham
Created:
2019-02-21 10:12:26 PST
Size:
253.71 KB
patch
obsolete
>Subversion Revision: 241770 >diff --git a/Source/WebCore/ChangeLog b/Source/WebCore/ChangeLog >index a6da53fe8e04ad79dd06c117c410a040b2b97e76..e765e39b81e16dea4ae4076f2952c828a39593ad 100644 >--- a/Source/WebCore/ChangeLog >+++ b/Source/WebCore/ChangeLog >@@ -1,3 +1,22 @@ >+2019-02-21 Brent Fulgham <bfulgham@apple.com> >+ >+ Use a SQLite database to hold the ResourceLoadStatistics data >+ https://bugs.webkit.org/show_bug.cgi?id=194867 >+ <rdar://problem/24240854> >+ >+ Reviewed by NOBODY (OOPS!). >+ >+ Add a new runtime feature flag to support use of an experimental database >+ back-end. Also expose some SQLite function calls for use outside of WebCore. >+ >+ No change in functionality, so no new tests. >+ >+ * page/RuntimeEnabledFeatures.h: >+ (WebCore::RuntimeEnabledFeatures::setItpDatabaseModeEnabled): >+ (WebCore::RuntimeEnabledFeatures::itpDatabaseModeEnabled const): >+ * platform/sql/SQLiteDatabase.h: >+ * platform/sql/SQLiteStatement.h: >+ > 2019-02-19 Commit Queue <commit-queue@webkit.org> > > Unreviewed, rolling out r241722. >diff --git a/Source/WebKit/ChangeLog b/Source/WebKit/ChangeLog >index 49a21d9da5bf5c9b72dc98218272aa389899ca1c..fcf453ce244cdd5e708116c6b0a2c65d524cfb93 100644 >--- a/Source/WebKit/ChangeLog >+++ b/Source/WebKit/ChangeLog >@@ -1,3 +1,283 @@ >+2019-02-21 Brent Fulgham <bfulgham@apple.com> >+ >+ Use a SQLite database to hold the ResourceLoadStatistics data >+ https://bugs.webkit.org/show_bug.cgi?id=194867 >+ <rdar://problem/24240854> >+ >+ Reviewed by NOBODY (OOPS!). >+ >+ The ResourceLoadStatistics database plist is inefficient. It requires more memory use >+ than it needs, and forces lots of looping and string comparisons to find information. >+ >+ This problem has already been solved in the form of relational databases. We use them >+ elsewhere in WebKit, and should do so for this storage as well. >+ >+ This patch creates an optional SQLite database to handle ITP operations. >+ >+ 1. It adds a new internal experimental feature flag to activate it. It requires the user >+ exit and restart the process to take effect. >+ 2. It populates itself from the existing plist file (if it exists). >+ 3. It stops using the plist in favor of the database. >+ 4. It does queries and other operations using the database instead of the old hash table >+ implementation. >+ >+ This patch refactors the exisiting ResourceLoadStatisticsMemoryStore class into a base >+ ResourceLoadStatisticsStore class, which ResourceLoadStatisticsMemoryStore is based on. >+ It adds a new ResourceLoadStatisticsDatabaseStore that is implemented in terms of SQL >+ operations. >+ >+ These code changes should not have any impact on existing operations, and when enabled >+ should produce the same test results. >+ >+ * NetworkProcess/Classifier/ResourceLoadStatisticsDatabaseStore.cpp: Added. >+ (WebKit::ResourceLoadStatisticsDatabaseStore::ResourceLoadStatisticsDatabaseStore): >+ (WebKit::ResourceLoadStatisticsDatabaseStore::createSchema): >+ (WebKit::ResourceLoadStatisticsDatabaseStore::prepareStatements): >+ (WebKit::ResourceLoadStatisticsDatabaseStore::insertObservedDomain): >+ (WebKit::ResourceLoadStatisticsDatabaseStore::relationshipExists const): >+ (WebKit::ResourceLoadStatisticsDatabaseStore::insertDomainRelationship): >+ (WebKit::ResourceLoadStatisticsDatabaseStore::confirmDomainDoesNotExist const): >+ (WebKit::ResourceLoadStatisticsDatabaseStore::domainIDFromString const): >+ (WebKit::ResourceLoadStatisticsDatabaseStore::insertDomainRelationships): >+ (WebKit::ResourceLoadStatisticsDatabaseStore::databaseIsEmpty const): >+ (WebKit::ResourceLoadStatisticsDatabaseStore::populateFromMemoryStore): >+ (WebKit::ResourceLoadStatisticsDatabaseStore::calculateAndSubmitTelemetry const): >+ (WebKit::ResourceLoadStatisticsDatabaseStore::incrementRecordsDeletedCountForDomains): >+ (WebKit::ResourceLoadStatisticsDatabaseStore::recursivelyFindNonPrevalentDomainsThatRedirectedToThisDomain): >+ (WebKit::buildList): >+ (WebKit::ResourceLoadStatisticsDatabaseStore::markAsPrevalentIfHasRedirectedToPrevalent): >+ (WebKit::listToString): >+ (WebKit::ResourceLoadStatisticsDatabaseStore::findNotVeryPrevalentResources): >+ (WebKit::ResourceLoadStatisticsDatabaseStore::reclassifyResources): >+ (WebKit::ResourceLoadStatisticsDatabaseStore::classifyPrevalentResources): >+ (WebKit::ResourceLoadStatisticsDatabaseStore::syncStorageIfNeeded): >+ (WebKit::ResourceLoadStatisticsDatabaseStore::syncStorageImmediately): >+ (WebKit::ResourceLoadStatisticsDatabaseStore::hasStorageAccess): >+ (WebKit::ResourceLoadStatisticsDatabaseStore::requestStorageAccess): >+ (WebKit::ResourceLoadStatisticsDatabaseStore::requestStorageAccessUnderOpener): >+ (WebKit::ResourceLoadStatisticsDatabaseStore::grantStorageAccess): >+ (WebKit::ResourceLoadStatisticsDatabaseStore::grantStorageAccessInternal): >+ (WebKit::ResourceLoadStatisticsDatabaseStore::grandfatherDataForDomains): >+ (WebKit::ResourceLoadStatisticsDatabaseStore::ensurePrevalentResourcesForDebugMode): >+ (WebKit::ResourceLoadStatisticsDatabaseStore::logFrameNavigation): >+ (WebKit::ResourceLoadStatisticsDatabaseStore::logSubresourceLoading): >+ (WebKit::ResourceLoadStatisticsDatabaseStore::logSubresourceRedirect): >+ (WebKit::ResourceLoadStatisticsDatabaseStore::setUserInteraction): >+ (WebKit::ResourceLoadStatisticsDatabaseStore::logUserInteraction): >+ (WebKit::ResourceLoadStatisticsDatabaseStore::clearUserInteraction): >+ (WebKit::ResourceLoadStatisticsDatabaseStore::hasHadUserInteraction): >+ (WebKit::ResourceLoadStatisticsDatabaseStore::setPrevalentResource): >+ (WebKit::ResourceLoadStatisticsDatabaseStore::setDomainsAsPrevalent): >+ (WebKit::ResourceLoadStatisticsDatabaseStore::dumpResourceLoadStatistics const): >+ (WebKit::ResourceLoadStatisticsDatabaseStore::predicateValueForDomain const): >+ (WebKit::ResourceLoadStatisticsDatabaseStore::isPrevalentResource const): >+ (WebKit::ResourceLoadStatisticsDatabaseStore::isVeryPrevalentResource const): >+ (WebKit::ResourceLoadStatisticsDatabaseStore::isRegisteredAsSubresourceUnder const): >+ (WebKit::ResourceLoadStatisticsDatabaseStore::isRegisteredAsSubFrameUnder const): >+ (WebKit::ResourceLoadStatisticsDatabaseStore::isRegisteredAsRedirectingTo const): >+ (WebKit::ResourceLoadStatisticsDatabaseStore::clearPrevalentResource): >+ (WebKit::ResourceLoadStatisticsDatabaseStore::setGrandfathered): >+ (WebKit::ResourceLoadStatisticsDatabaseStore::isGrandfathered const): >+ (WebKit::ResourceLoadStatisticsDatabaseStore::setSubframeUnderTopFrameOrigin): >+ (WebKit::ResourceLoadStatisticsDatabaseStore::setSubresourceUnderTopFrameOrigin): >+ (WebKit::ResourceLoadStatisticsDatabaseStore::setSubresourceUniqueRedirectTo): >+ (WebKit::ResourceLoadStatisticsDatabaseStore::setSubresourceUniqueRedirectFrom): >+ (WebKit::ResourceLoadStatisticsDatabaseStore::setTopFrameUniqueRedirectTo): >+ (WebKit::ResourceLoadStatisticsDatabaseStore::setTopFrameUniqueRedirectFrom): >+ (WebKit::ResourceLoadStatisticsDatabaseStore::ensureResourceStatisticsForPrimaryDomain): >+ (WebKit::ResourceLoadStatisticsDatabaseStore::clear): >+ (WebKit::ResourceLoadStatisticsDatabaseStore::cookieTreatmentForOrigin const): >+ (WebKit::ResourceLoadStatisticsDatabaseStore::hasUserGrantedStorageAccessThroughPrompt): >+ (WebKit::ResourceLoadStatisticsDatabaseStore::domainsToBlock const): >+ (WebKit::ResourceLoadStatisticsDatabaseStore::updateCookieBlocking): >+ (WebKit::ResourceLoadStatisticsDatabaseStore::updateCookieBlockingForDomains): >+ (WebKit::ResourceLoadStatisticsDatabaseStore::clearBlockingStateForDomains): >+ (WebKit::ResourceLoadStatisticsDatabaseStore::processStatistics const): >+ (WebKit::ResourceLoadStatisticsDatabaseStore::prevalentDomains const): >+ (WebKit::ResourceLoadStatisticsDatabaseStore::findExpiredUserInteractions const): >+ (WebKit::ResourceLoadStatisticsDatabaseStore::clearExpiredUserInteractions): >+ (WebKit::ResourceLoadStatisticsDatabaseStore::clearGrandfathering): >+ (WebKit::ResourceLoadStatisticsDatabaseStore::topPrivatelyControlledDomainsToRemoveWebsiteDataFor): >+ (WebKit::ResourceLoadStatisticsDatabaseStore::pruneStatisticsIfNeeded): >+ (WebKit::ResourceLoadStatisticsDatabaseStore::updateLastSeen): >+ (WebKit::ResourceLoadStatisticsDatabaseStore::setLastSeen): >+ (WebKit::ResourceLoadStatisticsDatabaseStore::setVeryPrevalentResource): >+ (WebKit::ResourceLoadStatisticsDatabaseStore::removeAllStorageAccess): >+ * NetworkProcess/Classifier/ResourceLoadStatisticsDatabaseStore.h: Added. >+ (WebKit::ResourceLoadStatisticsDatabaseStore::isEmpty const): >+ * NetworkProcess/Classifier/ResourceLoadStatisticsMemoryStore.cpp: >+ (WebKit::ResourceLoadStatisticsMemoryStore::ResourceLoadStatisticsMemoryStore): >+ (WebKit::ResourceLoadStatisticsMemoryStore::incrementRecordsDeletedCountForDomains): >+ (WebKit::ResourceLoadStatisticsMemoryStore::classifyPrevalentResources): >+ (WebKit::ResourceLoadStatisticsMemoryStore::syncStorageIfNeeded): >+ (WebKit::ResourceLoadStatisticsMemoryStore::syncStorageImmediately): >+ (WebKit::ResourceLoadStatisticsMemoryStore::grandfatherDataForDomains): >+ (WebKit::domainsToString): Deleted. >+ (WebKit::OperatingDate::fromWallTime): Deleted. >+ (WebKit::OperatingDate::today): Deleted. >+ (WebKit::OperatingDate::secondsSinceEpoch const): Deleted. >+ (WebKit::OperatingDate::operator== const): Deleted. >+ (WebKit::OperatingDate::operator< const): Deleted. >+ (WebKit::OperatingDate::operator<= const): Deleted. >+ (WebKit::OperatingDate::OperatingDate): Deleted. >+ (WebKit::mergeOperatingDates): Deleted. >+ (WebKit::computeImportance): Deleted. >+ (WebKit::ResourceLoadStatisticsMemoryStore::~ResourceLoadStatisticsMemoryStore): Deleted. >+ (WebKit::ResourceLoadStatisticsMemoryStore::setNotifyPagesWhenDataRecordsWereScanned): Deleted. >+ (WebKit::ResourceLoadStatisticsMemoryStore::setShouldClassifyResourcesBeforeDataRecordsRemoval): Deleted. >+ (WebKit::ResourceLoadStatisticsMemoryStore::setShouldSubmitTelemetry): Deleted. >+ (WebKit::ResourceLoadStatisticsMemoryStore::removeDataRecords): Deleted. >+ (WebKit::ResourceLoadStatisticsMemoryStore::processStatisticsAndDataRecords): Deleted. >+ (WebKit::ResourceLoadStatisticsMemoryStore::grandfatherExistingWebsiteData): Deleted. >+ (WebKit::ResourceLoadStatisticsMemoryStore::setResourceLoadStatisticsDebugMode): Deleted. >+ (WebKit::ResourceLoadStatisticsMemoryStore::setPrevalentResourceForDebugMode): Deleted. >+ (WebKit::ResourceLoadStatisticsMemoryStore::scheduleStatisticsProcessingRequestIfNecessary): Deleted. >+ (WebKit::ResourceLoadStatisticsMemoryStore::cancelPendingStatisticsProcessingRequest): Deleted. >+ (WebKit::ResourceLoadStatisticsMemoryStore::setTimeToLiveUserInteraction): Deleted. >+ (WebKit::ResourceLoadStatisticsMemoryStore::setMinimumTimeBetweenDataRecordsRemoval): Deleted. >+ (WebKit::ResourceLoadStatisticsMemoryStore::setGrandfatheringTime): Deleted. >+ (WebKit::ResourceLoadStatisticsMemoryStore::setCacheMaxAgeCap): Deleted. >+ (WebKit::ResourceLoadStatisticsMemoryStore::updateCacheMaxAgeCap): Deleted. >+ (WebKit::ResourceLoadStatisticsMemoryStore::setAgeCapForClientSideCookies): Deleted. >+ (WebKit::ResourceLoadStatisticsMemoryStore::updateClientSideCookiesAgeCap): Deleted. >+ (WebKit::ResourceLoadStatisticsMemoryStore::shouldRemoveDataRecords const): Deleted. >+ (WebKit::ResourceLoadStatisticsMemoryStore::setDataRecordsBeingRemoved): Deleted. >+ (WebKit::debugLogDomainsInBatches): Deleted. >+ (WebKit::ResourceLoadStatisticsMemoryStore::includeTodayAsOperatingDateIfNecessary): Deleted. >+ (WebKit::ResourceLoadStatisticsMemoryStore::hasStatisticsExpired const): Deleted. >+ (WebKit::ResourceLoadStatisticsMemoryStore::setMaxStatisticsEntries): Deleted. >+ (WebKit::ResourceLoadStatisticsMemoryStore::setPruneEntriesDownTo): Deleted. >+ (WebKit::ResourceLoadStatisticsMemoryStore::resetParametersToDefaultValues): Deleted. >+ (WebKit::ResourceLoadStatisticsMemoryStore::logTestingEvent): Deleted. >+ (WebKit::ResourceLoadStatisticsMemoryStore::didCreateNetworkProcess): Deleted. >+ * NetworkProcess/Classifier/ResourceLoadStatisticsMemoryStore.h: >+ (WebKit::ResourceLoadStatisticsMemoryStore::data const): >+ (WebKit::ResourceLoadStatisticsMemoryStore::isDebugModeEnabled const): Deleted. >+ (WebKit::ResourceLoadStatisticsMemoryStore::store const): Deleted. >+ (WebKit::ResourceLoadStatisticsMemoryStore::setStorageAccessPromptsEnabled): Deleted. >+ (WebKit::ResourceLoadStatisticsMemoryStore::setDebugLogggingEnabled): Deleted. >+ (): Deleted. >+ * NetworkProcess/Classifier/ResourceLoadStatisticsStore.cpp: Added. >+ (WebKit::domainsToString): >+ (WebKit::OperatingDate::fromWallTime): >+ (WebKit::OperatingDate::today): >+ (WebKit::OperatingDate::secondsSinceEpoch const): >+ (WebKit::OperatingDate::operator== const): >+ (WebKit::OperatingDate::operator< const): >+ (WebKit::OperatingDate::operator<= const): >+ (WebKit::ResourceLoadStatisticsStore::ResourceLoadStatisticsStore): >+ (WebKit::ResourceLoadStatisticsStore::~ResourceLoadStatisticsStore): >+ (WebKit::ResourceLoadStatisticsStore::calculateAndSubmitTelemetry const): >+ (WebKit::ResourceLoadStatisticsStore::computeImportance): >+ (WebKit::ResourceLoadStatisticsStore::setNotifyPagesWhenDataRecordsWereScanned): >+ (WebKit::ResourceLoadStatisticsStore::setShouldClassifyResourcesBeforeDataRecordsRemoval): >+ (WebKit::ResourceLoadStatisticsStore::setShouldSubmitTelemetry): >+ (WebKit::ResourceLoadStatisticsStore::removeDataRecords): >+ (WebKit::ResourceLoadStatisticsStore::processStatisticsAndDataRecords): >+ (WebKit::ResourceLoadStatisticsStore::grandfatherExistingWebsiteData): >+ (WebKit::ResourceLoadStatisticsStore::setResourceLoadStatisticsDebugMode): >+ (WebKit::ResourceLoadStatisticsStore::setPrevalentResourceForDebugMode): >+ (WebKit::ResourceLoadStatisticsStore::scheduleStatisticsProcessingRequestIfNecessary): >+ (WebKit::ResourceLoadStatisticsStore::cancelPendingStatisticsProcessingRequest): >+ (WebKit::ResourceLoadStatisticsStore::setTimeToLiveUserInteraction): >+ (WebKit::ResourceLoadStatisticsStore::setMinimumTimeBetweenDataRecordsRemoval): >+ (WebKit::ResourceLoadStatisticsStore::setGrandfatheringTime): >+ (WebKit::ResourceLoadStatisticsStore::setCacheMaxAgeCap): >+ (WebKit::ResourceLoadStatisticsStore::updateCacheMaxAgeCap): >+ (WebKit::ResourceLoadStatisticsStore::setAgeCapForClientSideCookies): >+ (WebKit::ResourceLoadStatisticsStore::updateClientSideCookiesAgeCap): >+ (WebKit::ResourceLoadStatisticsStore::shouldRemoveDataRecords const): >+ (WebKit::ResourceLoadStatisticsStore::setDataRecordsBeingRemoved): >+ (WebKit::ResourceLoadStatisticsStore::updateCookieBlockingForDomains): >+ (WebKit::ResourceLoadStatisticsStore::clearBlockingStateForDomains): >+ (WebKit::ResourceLoadStatisticsStore::processStatistics const): >+ (WebKit::ResourceLoadStatisticsStore::statisticsEpirationTime const): >+ (WebKit::ResourceLoadStatisticsStore::mergeOperatingDates): >+ (WebKit::ResourceLoadStatisticsStore::includeTodayAsOperatingDateIfNecessary): >+ (WebKit::ResourceLoadStatisticsStore::hasStatisticsExpired const): >+ (WebKit::ResourceLoadStatisticsStore::setMaxStatisticsEntries): >+ (WebKit::ResourceLoadStatisticsStore::setPruneEntriesDownTo): >+ (WebKit::ResourceLoadStatisticsStore::resetParametersToDefaultValues): >+ (WebKit::ResourceLoadStatisticsStore::logTestingEvent): >+ (WebKit::ResourceLoadStatisticsStore::removeAllStorageAccess): >+ (WebKit::ResourceLoadStatisticsStore::didCreateNetworkProcess): >+ (WebKit::ResourceLoadStatisticsStore::debugLogDomainsInBatches): >+ * NetworkProcess/Classifier/ResourceLoadStatisticsStore.h: Copied from Source/WebKit/NetworkProcess/Classifier/ResourceLoadStatisticsMemoryStore.h. >+ (WebKit::OperatingDate::OperatingDate): >+ (WebKit::ResourceLoadStatisticsStore::isEmpty const): >+ (WebKit::ResourceLoadStatisticsStore::data const): >+ (WebKit::ResourceLoadStatisticsStore::isDebugModeEnabled const): >+ (WebKit::ResourceLoadStatisticsStore::store const): >+ (WebKit::ResourceLoadStatisticsStore::setStorageAccessPromptsEnabled): >+ (WebKit::ResourceLoadStatisticsStore::setDebugLogggingEnabled): >+ * NetworkProcess/Classifier/ResourceLoadStatisticsStoreCocoa.mm: Renamed from Source/WebKit/UIProcess/Cocoa/ResourceLoadStatisticsMemoryStoreCocoa.mm. >+ (WebKit::ResourceLoadStatisticsStore::registerUserDefaultsIfNeeded): >+ * NetworkProcess/Classifier/WebResourceLoadStatisticsStore.cpp: >+ (WebKit::WebResourceLoadStatisticsStore::setNotifyPagesWhenDataRecordsWereScanned): >+ (WebKit::WebResourceLoadStatisticsStore::setShouldClassifyResourcesBeforeDataRecordsRemoval): >+ (WebKit::WebResourceLoadStatisticsStore::setShouldSubmitTelemetry): >+ (WebKit::WebResourceLoadStatisticsStore::WebResourceLoadStatisticsStore): >+ (WebKit::WebResourceLoadStatisticsStore::flushAndDestroyPersistentStore): >+ (WebKit::WebResourceLoadStatisticsStore::setResourceLoadStatisticsDebugMode): >+ (WebKit::WebResourceLoadStatisticsStore::setPrevalentResourceForDebugMode): >+ (WebKit::WebResourceLoadStatisticsStore::scheduleStatisticsAndDataRecordsProcessing): >+ (WebKit::WebResourceLoadStatisticsStore::hasStorageAccess): >+ (WebKit::WebResourceLoadStatisticsStore::requestStorageAccess): >+ (WebKit::WebResourceLoadStatisticsStore::requestStorageAccessUnderOpener): >+ (WebKit::WebResourceLoadStatisticsStore::grantStorageAccess): >+ (WebKit::WebResourceLoadStatisticsStore::performDailyTasks): >+ (WebKit::WebResourceLoadStatisticsStore::logFrameNavigation): >+ (WebKit::WebResourceLoadStatisticsStore::logWebSocketLoading): >+ (WebKit::WebResourceLoadStatisticsStore::logSubresourceLoading): >+ (WebKit::WebResourceLoadStatisticsStore::logSubresourceRedirect): >+ (WebKit::WebResourceLoadStatisticsStore::logUserInteraction): >+ (WebKit::WebResourceLoadStatisticsStore::clearUserInteraction): >+ (WebKit::WebResourceLoadStatisticsStore::hasHadUserInteraction): >+ (WebKit::WebResourceLoadStatisticsStore::setLastSeen): >+ (WebKit::WebResourceLoadStatisticsStore::setPrevalentResource): >+ (WebKit::WebResourceLoadStatisticsStore::setVeryPrevalentResource): >+ (WebKit::WebResourceLoadStatisticsStore::isPrevalentResource): >+ (WebKit::WebResourceLoadStatisticsStore::isVeryPrevalentResource): >+ (WebKit::WebResourceLoadStatisticsStore::isRegisteredAsSubresourceUnder): >+ (WebKit::WebResourceLoadStatisticsStore::isRegisteredAsSubFrameUnder): >+ (WebKit::WebResourceLoadStatisticsStore::isRegisteredAsRedirectingTo): >+ (WebKit::WebResourceLoadStatisticsStore::clearPrevalentResource): >+ (WebKit::WebResourceLoadStatisticsStore::setGrandfathered): >+ (WebKit::WebResourceLoadStatisticsStore::isGrandfathered): >+ (WebKit::WebResourceLoadStatisticsStore::setSubframeUnderTopFrameOrigin): >+ (WebKit::WebResourceLoadStatisticsStore::setSubresourceUnderTopFrameOrigin): >+ (WebKit::WebResourceLoadStatisticsStore::setSubresourceUniqueRedirectTo): >+ (WebKit::WebResourceLoadStatisticsStore::setSubresourceUniqueRedirectFrom): >+ (WebKit::WebResourceLoadStatisticsStore::setTopFrameUniqueRedirectTo): >+ (WebKit::WebResourceLoadStatisticsStore::setTopFrameUniqueRedirectFrom): >+ (WebKit::WebResourceLoadStatisticsStore::scheduleCookieBlockingUpdate): >+ (WebKit::WebResourceLoadStatisticsStore::scheduleCookieBlockingUpdateForDomains): >+ (WebKit::WebResourceLoadStatisticsStore::scheduleClearBlockingStateForDomains): >+ (WebKit::WebResourceLoadStatisticsStore::scheduleClearInMemoryAndPersistent): >+ (WebKit::WebResourceLoadStatisticsStore::setTimeToLiveUserInteraction): >+ (WebKit::WebResourceLoadStatisticsStore::setMinimumTimeBetweenDataRecordsRemoval): >+ (WebKit::WebResourceLoadStatisticsStore::setGrandfatheringTime): >+ (WebKit::WebResourceLoadStatisticsStore::setMaxStatisticsEntries): >+ (WebKit::WebResourceLoadStatisticsStore::setPruneEntriesDownTo): >+ (WebKit::WebResourceLoadStatisticsStore::resetParametersToDefaultValues): >+ * NetworkProcess/Classifier/WebResourceLoadStatisticsStore.h: >+ * NetworkProcess/NetworkProcess.cpp: >+ (WebKit::NetworkProcess::initializeNetworkProcess): >+ * NetworkProcess/NetworkProcessCreationParameters.cpp: >+ (WebKit::NetworkProcessCreationParameters::encode const): >+ (WebKit::NetworkProcessCreationParameters::decode): >+ * NetworkProcess/NetworkProcessCreationParameters.h: >+ * Platform/classifier/ResourceLoadStatisticsClassifier.cpp: >+ (WebKit::ResourceLoadStatisticsClassifier::calculateResourcePrevalence): >+ * Platform/classifier/ResourceLoadStatisticsClassifier.h: >+ * Shared/WebPreferences.yaml: >+ * Sources.txt: >+ * SourcesCocoa.txt: >+ * UIProcess/Cocoa/WebProcessPoolCocoa.mm: >+ (WebKit::WebProcessPool::platformInitializeNetworkProcess): >+ * UIProcess/WebProcessPool.cpp: >+ * WebKit.xcodeproj/project.pbxproj: >+ > 2019-02-19 Antoine Quint <graouts@apple.com> > > [iOS] "touch-action: none" should not prevent text selection or tapping on a link >diff --git a/Source/WebCore/page/RuntimeEnabledFeatures.h b/Source/WebCore/page/RuntimeEnabledFeatures.h >index 7176d2ff6be776b98f9c254ecac861fb0c02637c..4691c939ed64eee34a82b7108957858046435139 100644 >--- a/Source/WebCore/page/RuntimeEnabledFeatures.h >+++ b/Source/WebCore/page/RuntimeEnabledFeatures.h >@@ -148,6 +148,9 @@ public: > void setItpDebugModeEnabled(bool isEnabled) { m_itpDebugMode = isEnabled; } > bool itpDebugModeEnabled() const { return m_itpDebugMode; } > >+ void setItpDatabaseModeEnabled(bool isEnabled) { m_itpDatabaseMode = isEnabled; } >+ bool itpDatabaseModeEnabled() const { return m_itpDatabaseMode; } >+ > void setRestrictedHTTPResponseAccess(bool isEnabled) { m_isRestrictedHTTPResponseAccess = isEnabled; } > bool restrictedHTTPResponseAccess() const { return m_isRestrictedHTTPResponseAccess; } > >@@ -351,7 +354,7 @@ public: > bool mousemoveEventHandlingPreventsDefaultEnabled() const { return m_mousemoveEventHandlingPreventsDefaultEnabled; } > void setMousemoveEventHandlingPreventsDefaultEnabled(bool isEnabled) { m_mousemoveEventHandlingPreventsDefaultEnabled = isEnabled; } > #endif >- >+ > WEBCORE_EXPORT static RuntimeEnabledFeatures& sharedFeatures(); > > private: >@@ -534,6 +537,8 @@ private: > bool m_mousemoveEventHandlingPreventsDefaultEnabled { false }; > #endif > >+ bool m_itpDatabaseMode { false }; >+ > friend class WTF::NeverDestroyed<RuntimeEnabledFeatures>; > }; > >diff --git a/Source/WebCore/platform/sql/SQLiteDatabase.h b/Source/WebCore/platform/sql/SQLiteDatabase.h >index 57d2ad869d4d2317ab063b5d6bae14d9ef09fa46..c9b310244207bc768fec929f025f0546b96b038e 100644 >--- a/Source/WebCore/platform/sql/SQLiteDatabase.h >+++ b/Source/WebCore/platform/sql/SQLiteDatabase.h >@@ -64,7 +64,7 @@ public: > > WEBCORE_EXPORT bool tableExists(const String&); > void clearAllTables(); >- int runVacuumCommand(); >+ WEBCORE_EXPORT int runVacuumCommand(); > int runIncrementalVacuumCommand(); > > bool transactionInProgress() const { return m_transactionInProgress; } >diff --git a/Source/WebCore/platform/sql/SQLiteStatement.h b/Source/WebCore/platform/sql/SQLiteStatement.h >index a193d8c24b0bd024965bb105446423bbc8f5bfe9..f366c7302f4ba5a5f4ac1a61c16263313932d3d7 100644 >--- a/Source/WebCore/platform/sql/SQLiteStatement.h >+++ b/Source/WebCore/platform/sql/SQLiteStatement.h >@@ -76,7 +76,7 @@ public: > String getColumnName(int col); > SQLValue getColumnValue(int col); > WEBCORE_EXPORT String getColumnText(int col); >- double getColumnDouble(int col); >+ WEBCORE_EXPORT double getColumnDouble(int col); > WEBCORE_EXPORT int getColumnInt(int col); > WEBCORE_EXPORT int64_t getColumnInt64(int col); > WEBCORE_EXPORT String getColumnBlobAsString(int col); >diff --git a/Source/WebKit/NetworkProcess/Classifier/ResourceLoadStatisticsDatabaseStore.cpp b/Source/WebKit/NetworkProcess/Classifier/ResourceLoadStatisticsDatabaseStore.cpp >new file mode 100644 >index 0000000000000000000000000000000000000000..b6a86cdb2d190f124d93fbaf79b16d2f710fa781 >--- /dev/null >+++ b/Source/WebKit/NetworkProcess/Classifier/ResourceLoadStatisticsDatabaseStore.cpp >@@ -0,0 +1,1583 @@ >+/* >+ * Copyright (C) 2019 Apple Inc. All rights reserved. >+ * >+ * Redistribution and use in source and binary forms, with or without >+ * modification, are permitted provided that the following conditions >+ * are met: >+ * 1. Redistributions of source code must retain the above copyright >+ * notice, this list of conditions and the following disclaimer. >+ * 2. Redistributions in binary form must reproduce the above copyright >+ * notice, this list of conditions and the following disclaimer in the >+ * documentation and/or other materials provided with the distribution. >+ * >+ * THIS SOFTWARE IS PROVIDED BY APPLE INC. AND ITS CONTRIBUTORS ``AS IS'' >+ * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, >+ * THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR >+ * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL APPLE INC. OR ITS CONTRIBUTORS >+ * BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR >+ * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF >+ * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS >+ * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN >+ * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) >+ * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF >+ * THE POSSIBILITY OF SUCH DAMAGE. >+ */ >+ >+#include "config.h" >+#include "ResourceLoadStatisticsDatabaseStore.h" >+ >+#if ENABLE(RESOURCE_LOAD_STATISTICS) >+ >+#include "Logging.h" >+#include "NetworkSession.h" >+#include "PluginProcessManager.h" >+#include "PluginProcessProxy.h" >+#include "ResourceLoadStatisticsMemoryStore.h" >+#include "StorageAccessStatus.h" >+#include "WebProcessProxy.h" >+#include "WebResourceLoadStatisticsTelemetry.h" >+#include "WebsiteDataStore.h" >+#include <WebCore/KeyedCoding.h> >+#include <WebCore/NetworkStorageSession.h> >+#include <WebCore/ResourceLoadStatistics.h> >+#include <WebCore/SQLiteDatabase.h> >+#include <WebCore/SQLiteStatement.h> >+#include <wtf/CallbackAggregator.h> >+#include <wtf/DateMath.h> >+#include <wtf/HashMap.h> >+#include <wtf/MathExtras.h> >+#include <wtf/StdSet.h> >+#include <wtf/text/StringBuilder.h> >+ >+namespace WebKit { >+using namespace WebCore; >+ >+#define RELEASE_LOG_IF_ALLOWED(sessionID, fmt, ...) RELEASE_LOG_IF(sessionID.isAlwaysOnLoggingAllowed(), Network, "%p - NetworkHTTPSUpgradeChecker::" fmt, this, ##__VA_ARGS__) >+#define RELEASE_LOG_ERROR_IF_ALLOWED(sessionID, fmt, ...) RELEASE_LOG_ERROR_IF(sessionID.isAlwaysOnLoggingAllowed(), Network, "%p - NetworkHTTPSUpgradeChecker::" fmt, this, ##__VA_ARGS__) >+ >+/* >+ // FIXME: Implement for Database >+struct db_StatisticsLastSeen { >+ String topPrivatelyOwnedDomain; >+ WallTime lastSeen; >+}; >+ >+static void pruneResources(HashMap<String, WebCore::ResourceLoadStatistics>& statisticsMap, Vector<db_StatisticsLastSeen>& statisticsToPrune, size_t& numberOfEntriesToPrune) >+{ >+ if (statisticsToPrune.size() > numberOfEntriesToPrune) { >+ std::sort(statisticsToPrune.begin(), statisticsToPrune.end(), [](const db_StatisticsLastSeen& a, const db_StatisticsLastSeen& b) { >+ return a.lastSeen < b.lastSeen; >+ }); >+ } >+ >+ for (size_t i = 0, end = std::min(numberOfEntriesToPrune, statisticsToPrune.size()); i != end; ++i, --numberOfEntriesToPrune) >+ statisticsMap.remove(statisticsToPrune[i].topPrivatelyOwnedDomain); >+} >+*/ >+ >+ResourceLoadStatisticsDatabaseStore::ResourceLoadStatisticsDatabaseStore(WebResourceLoadStatisticsStore& store, WorkQueue& workQueue, const String& storageDirectoryPath) >+ : ResourceLoadStatisticsStore(store, workQueue) >+ , m_storageDirectoryPath(storageDirectoryPath + String("/observations.db")) >+ , m_database(makeUniqueRef<WebCore::SQLiteDatabase>()) >+ , m_insertObservedDomainStatement(makeUniqueRef<WebCore::SQLiteStatement>(m_database.get(), "INSERT INTO ObservedDomains (highLevelDomain, lastSeen, hadUserInteraction, mostRecentUserInteractionTime, grandfathered, isPrevalent, isVeryPrevalent, dataRecordsRemoved, timesAccessedAsFirstPartyDueToUserInteraction, timesAccessedAsFirstPartyDueToStorageAccessAPI) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)"_s)) >+ , m_insertTopLevelDomainStatement(makeUniqueRef<WebCore::SQLiteStatement>(m_database.get(), "INSERT INTO TopLevelDomains VALUES (?)"_s)) >+ , m_domainIDFromStringStatement(makeUniqueRef<WebCore::SQLiteStatement>(m_database.get(), "SELECT domainID FROM ObservedDomains WHERE highLevelDomain = ?"_s)) >+ , m_storageAccessUnderTopFrameOriginsStatement(makeUniqueRef<WebCore::SQLiteStatement>(m_database.get(), "INSERT INTO StorageAccessUnderTopFrameOrigins (domainID, topLevelDomainID) SELECT ?, domainID FROM ObservedDomains WHERE highLevelDomain == ?"_s)) >+ , m_topFrameUniqueRedirectsTo(makeUniqueRef<WebCore::SQLiteStatement>(m_database.get(), "INSERT INTO TopFrameUniqueRedirectsTo (sourceDomainID, toDomainID) SELECT ?, domainID FROM ObservedDomains WHERE highLevelDomain == ?"_s)) >+ , m_topFrameUniqueRedirectsToExists(makeUniqueRef<WebCore::SQLiteStatement>(m_database.get(), "SELECT EXISTS (SELECT 1 FROM TopFrameUniqueRedirectsTo WHERE sourceDomainID = ? AND toDomainID = (SELECT domainID FROM ObservedDomains WHERE highLevelDomain = ?))"_s)) >+ , m_topFrameUniqueRedirectsFrom(makeUniqueRef<WebCore::SQLiteStatement>(m_database.get(), "INSERT INTO TopFrameUniqueRedirectsFrom (targetDomainID, fromDomainID) SELECT ?, domainID FROM ObservedDomains WHERE highLevelDomain = ?"_s)) >+ , m_topFrameUniqueRedirectsFromExists(makeUniqueRef<WebCore::SQLiteStatement>(m_database.get(), "SELECT EXISTS (SELECT 1 FROM TopFrameUniqueRedirectsFrom WHERE targetDomainID = ? AND fromDomainID = (SELECT domainID FROM ObservedDomains WHERE highLevelDomain = ?))"_s)) >+ , m_subframeUnderTopFrameOrigins(makeUniqueRef<WebCore::SQLiteStatement>(m_database.get(), "INSERT INTO SubframeUnderTopFrameOrigins (subFrameDomainID, topFrameDomainID) SELECT ?, domainID FROM ObservedDomains WHERE highLevelDomain = ?"_s)) >+ , m_subframeUnderTopFrameOriginExists(makeUniqueRef<WebCore::SQLiteStatement>(m_database.get(), "SELECT EXISTS (SELECT 1 FROM SubframeUnderTopFrameOrigins WHERE subFrameDomainID = ? AND topFrameDomainID = (SELECT domainID FROM ObservedDomains WHERE highLevelDomain = ?))"_s)) >+ , m_subresourceUnderTopFrameOrigins(makeUniqueRef<WebCore::SQLiteStatement>(m_database.get(), "INSERT INTO SubresourceUnderTopFrameOrigins (subresourceDomainID, topFrameDomainID) SELECT ?, domainID FROM ObservedDomains WHERE highLevelDomain = ?"_s)) >+ , m_subresourceUnderTopFrameOriginExists(makeUniqueRef<WebCore::SQLiteStatement>(m_database.get(), "SELECT EXISTS (SELECT 1 FROM SubresourceUnderTopFrameOrigins WHERE subresourceDomainID = ? AND topFrameDomainID = (SELECT domainID FROM ObservedDomains WHERE highLevelDomain = ?))"_s)) >+ , m_subresourceUniqueRedirectsTo(makeUniqueRef<WebCore::SQLiteStatement>(m_database.get(), "INSERT INTO SubresourceUniqueRedirectsTo (subresourceDomainID, toDomainID) SELECT ?, domainID FROM ObservedDomains WHERE highLevelDomain == ?"_s)) >+ , m_subresourceUniqueRedirectsToExists(makeUniqueRef<WebCore::SQLiteStatement>(m_database.get(), "SELECT EXISTS (SELECT 1 FROM SubresourceUniqueRedirectsTo WHERE subresourceDomainID = ? AND toDomainID = (SELECT domainID FROM ObservedDomains WHERE highLevelDomain = ?))"_s)) >+ , m_subresourceUniqueRedirectsFrom(makeUniqueRef<WebCore::SQLiteStatement>(m_database.get(), "INSERT INTO SubresourceUniqueRedirectsFrom (subresourceDomainID, fromDomainID) SELECT ?, domainID FROM ObservedDomains WHERE highLevelDomain == ?"_s)) >+ , m_subresourceUniqueRedirectsFromExists(makeUniqueRef<WebCore::SQLiteStatement>(m_database.get(), "SELECT EXISTS (SELECT 1 FROM SubresourceUniqueRedirectsFrom WHERE subresourceDomainID = ? AND fromDomainID = (SELECT domainID FROM ObservedDomains WHERE highLevelDomain = ?))"_s)) >+ , m_mostRecentUserInteractionStatement(makeUniqueRef<WebCore::SQLiteStatement>(m_database.get(), "UPDATE ObservedDomains SET hadUserInteraction = ?, mostRecentUserInteractionTime = ? WHERE highLevelDomain = ?"_s)) >+ , m_updateLastSeenStatement(makeUniqueRef<WebCore::SQLiteStatement>(m_database.get(), "UPDATE ObservedDomains SET lastSeen = ? WHERE highLevelDomain = ?"_s)) >+ , m_updatePrevalentResourceStatement(makeUniqueRef<WebCore::SQLiteStatement>(m_database.get(), "UPDATE ObservedDomains SET isPrevalent = ? WHERE highLevelDomain = ?"_s)) >+ , m_isPrevalentResourceStatement(makeUniqueRef<WebCore::SQLiteStatement>(m_database.get(), "SELECT isPrevalent FROM ObservedDomains WHERE highLevelDomain = ?"_s)) >+ , m_updateVeryPrevalentResourceStatement(makeUniqueRef<WebCore::SQLiteStatement>(m_database.get(), "UPDATE ObservedDomains SET isVeryPrevalent = ? WHERE highLevelDomain = ?"_s)) >+ , m_isVeryPrevalentResourceStatement(makeUniqueRef<WebCore::SQLiteStatement>(m_database.get(), "SELECT isVeryPrevalent FROM ObservedDomains WHERE highLevelDomain = ?"_s)) >+ , m_clearPrevalentResourceStatement(makeUniqueRef<WebCore::SQLiteStatement>(m_database.get(), "UPDATE ObservedDomains SET isPrevalent = 0, isVeryPrevalent = 0 WHERE highLevelDomain = ?"_s)) >+ , m_updateHadUserInteractionStatement(makeUniqueRef<WebCore::SQLiteStatement>(m_database.get(), "UPDATE ObservedDomains SET hadUserInteraction = ? WHERE highLevelDomain = ?"_s)) >+ , m_hadUserInteractionStatement(makeUniqueRef<WebCore::SQLiteStatement>(m_database.get(), "SELECT hadUserInteraction, mostRecentUserInteractionTime FROM ObservedDomains WHERE highLevelDomain = ?"_s)) >+ , m_updateMostRecentUserInteractionTimeStatement(makeUniqueRef<WebCore::SQLiteStatement>(m_database.get(), "UPDATE ObservedDomains SET mostRecentUserInteractionTime = ? WHERE highLevelDomain = ?"_s)) >+ , m_updateGrandfatheredStatement(makeUniqueRef<WebCore::SQLiteStatement>(m_database.get(), "UPDATE ObservedDomains SET grandfathered = ? WHERE highLevelDomain = ?"_s)) >+ , m_isGrandfatheredStatement(makeUniqueRef<WebCore::SQLiteStatement>(m_database.get(), "SELECT grandfathered FROM ObservedDomains WHERE highLevelDomain = ?"_s)) >+ , m_updateDataRecordsRemovedStatement(makeUniqueRef<WebCore::SQLiteStatement>(m_database.get(), "UPDATE ObservedDomains SET dataRecordsRemoved = ? WHERE highLevelDomain = ?"_s)) >+ , m_updateTimesAccessedAsFirstPartyDueToUserInteractionStatement(makeUniqueRef<WebCore::SQLiteStatement>(m_database.get(), "UPDATE ObservedDomains SET timesAccessedAsFirstPartyDueToUserInteraction = ? WHERE highLevelDomain = ?"_s)) >+ , m_updateTimesAccessedAsFirstPartyDueToStorageAccessAPIStatement(makeUniqueRef<WebCore::SQLiteStatement>(m_database.get(), "UPDATE ObservedDomains SET timesAccessedAsFirstPartyDueToStorageAccessAPI = ? WHERE highLevelDomain = ?"_s)) >+ , m_findExpiredUserInteractionStatement(makeUniqueRef<WebCore::SQLiteStatement>(m_database.get(), "SELECT domainID FROM ObservedDomains WHERE hadUserInteraction = 1 AND mostRecentUserInteractionTime < ?"_s)) >+{ >+ ASSERT(!RunLoop::isMain()); >+ >+#if PLATFORM(COCOA) >+ registerUserDefaultsIfNeeded(); >+#endif >+ >+ m_workQueue->dispatch([this, path = m_storageDirectoryPath.isolatedCopy()] { >+ if (!m_database->open(path)) { >+#if PLATFORM(COCOA) >+ RELEASE_LOG_ERROR(Network, "%p - ResourceLoadStatisticsDatabaseStore::open failed, error message: %{public}s, database path: %{public}s", this, m_database->lastErrorMsg(), path.utf8().data()); >+#endif >+ ASSERT_NOT_REACHED(); >+ return; >+ } >+ >+ // Since we are using a workerQueue, the sequential dispatch blocks may be called by different threads. >+ m_database->disableThreadingChecks(); >+ >+ if (!m_database->tableExists("ObservedDomains")) { >+ if (!createSchema()) { >+#if PLATFORM(COCOA) >+ RELEASE_LOG_ERROR(Network, "%p - ResourceLoadStatisticsDatabaseStore::createSchema failed, error message: %{public}s, database path: %{public}s", this, m_database->lastErrorMsg(), path.utf8().data()); >+#endif >+ ASSERT_NOT_REACHED(); >+ return; >+ } >+ } >+ >+ if (!prepareStatements()) { >+#if PLATFORM(COCOA) >+ RELEASE_LOG_ERROR(Network, "%p - ResourceLoadStatisticsDatabaseStore::prepareStatements failed, error message: %{public}s, database path: %{public}s", this, m_database->lastErrorMsg(), path.utf8().data()); >+#endif >+ ASSERT_NOT_REACHED(); >+ return; >+ } >+ }); >+ >+ m_workQueue->dispatchAfter(5_s, [weakThis = makeWeakPtr(*this)] { >+ if (weakThis) >+ weakThis->calculateAndSubmitTelemetry(); >+ }); >+} >+ >+bool ResourceLoadStatisticsDatabaseStore::createSchema() >+{ >+ ASSERT(!RunLoop::isMain()); >+ >+ if (!m_database->executeCommand("CREATE TABLE ObservedDomains (domainID INTEGER PRIMARY KEY, highLevelDomain TEXT NOT NULL UNIQUE ON CONFLICT FAIL, lastSeen REAL NOT NULL, hadUserInteraction INTEGER NOT NULL, mostRecentUserInteractionTime REAL NOT NULL, grandfathered INTEGER NOT NULL, isPrevalent INTEGER NOT NULL, isVeryPrevalent INTEGER NOT NULL, dataRecordsRemoved INTEGER NOT NULL, timesAccessedAsFirstPartyDueToUserInteraction INTEGER NOT NULL, timesAccessedAsFirstPartyDueToStorageAccessAPI INTEGER NOT NULL);")) { >+ LOG_ERROR("Could not create ObservedDomains table in database (%i) - %s", m_database->lastError(), m_database->lastErrorMsg()); >+ return false; >+ } >+ >+ if (!m_database->executeCommand("CREATE TABLE TopLevelDomains (topLevelDomainID INTEGER PRIMARY KEY, FOREIGN KEY(topLevelDomainID) REFERENCES ObservedDomains(domainID));")) { >+ LOG_ERROR("Could not create TopLevelDomains table in database (%i) - %s", m_database->lastError(), m_database->lastErrorMsg()); >+ return false; >+ } >+ >+ // Storage Access Statistics. >+ if (!m_database->executeCommand("CREATE TABLE StorageAccessUnderTopFrameOrigins (domainID INTEGER NOT NULL, topLevelDomainID INTEGER NOT NULL ON CONFLICT FAIL, FOREIGN KEY(domainID) REFERENCES ObservedDomains(domainID), FOREIGN KEY(topLevelDomainID) REFERENCES TopLevelDomains(topLevelDomainID));")) { >+ LOG_ERROR("Could not create StorageAccessUnderTopFrameOrigins table in database (%i) - %s", m_database->lastError(), m_database->lastErrorMsg()); >+ return false; >+ } >+ >+ // Top Frame Statistics. >+ if (!m_database->executeCommand("CREATE TABLE TopFrameUniqueRedirectsTo (sourceDomainID INTEGER NOT NULL, toDomainID INTEGER NOT NULL, FOREIGN KEY(sourceDomainID) REFERENCES TopLevelDomains(topLevelDomainID), FOREIGN KEY(toDomainID) REFERENCES TopLevelDomains(topLevelDomainID));")) { >+ LOG_ERROR("Could not create TopFrameUniqueRedirectsTo table in database (%i) - %s", m_database->lastError(), m_database->lastErrorMsg()); >+ return false; >+ } >+ >+ if (!m_database->executeCommand("CREATE TABLE TopFrameUniqueRedirectsFrom (targetDomainID INTEGER NOT NULL, fromDomainID INTEGER NOT NULL, FOREIGN KEY(targetDomainID) REFERENCES TopLevelDomains(topLevelDomainID), FOREIGN KEY(fromDomainID) REFERENCES TopLevelDomains(topLevelDomainID));")) { >+ LOG_ERROR("Could not create TopFrameUniqueRedirectsFrom table in database (%i) - %s", m_database->lastError(), m_database->lastErrorMsg()); >+ return false; >+ } >+ >+ // Subframe Statistics. >+ if (!m_database->executeCommand("CREATE TABLE SubframeUnderTopFrameOrigins (subFrameDomainID INTEGER NOT NULL, topFrameDomainID INTEGER NOT NULL, FOREIGN KEY(subFrameDomainID) REFERENCES ObservedDomains(domainID), FOREIGN KEY(topFrameDomainID) REFERENCES TopLevelDomains(topLevelDomainID));")) { >+ LOG_ERROR("Could not create SubframeUnderTopFrameOrigins table in database (%i) - %s", m_database->lastError(), m_database->lastErrorMsg()); >+ return false; >+ } >+ >+ // Subresource Statistics. >+ if (!m_database->executeCommand("CREATE TABLE SubresourceUnderTopFrameOrigins (subresourceDomainID INTEGER NOT NULL, topFrameDomainID INTEGER NOT NULL, FOREIGN KEY(subresourceDomainID) REFERENCES ObservedDomains(domainID), FOREIGN KEY(topFrameDomainID) REFERENCES TopLevelDomains(topLevelDomainID));")) { >+ LOG_ERROR("Could not create SubframeUnderTopFrameOrigins table in database (%i) - %s", m_database->lastError(), m_database->lastErrorMsg()); >+ return false; >+ } >+ >+ if (!m_database->executeCommand("CREATE TABLE SubresourceUniqueRedirectsTo (subresourceDomainID INTEGER NOT NULL, toDomainID INTEGER NOT NULL, FOREIGN KEY(subresourceDomainID) REFERENCES ObservedDomains(domainID), FOREIGN KEY(toDomainID) REFERENCES ObservedDomains(domainID));")) { >+ LOG_ERROR("Could not create SubresourceUniqueRedirectsTo table in database (%i) - %s", m_database->lastError(), m_database->lastErrorMsg()); >+ return false; >+ } >+ >+ if (!m_database->executeCommand("CREATE TABLE SubresourceUniqueRedirectsFrom (subresourceDomainID INTEGER NOT NULL, fromDomainID INTEGER NOT NULL, FOREIGN KEY(subresourceDomainID) REFERENCES ObservedDomains(domainID), FOREIGN KEY(fromDomainID) REFERENCES ObservedDomains(domainID));")) { >+ LOG_ERROR("Could not create SubresourceUniqueRedirectsFrom table in database (%i) - %s", m_database->lastError(), m_database->lastErrorMsg()); >+ return false; >+ } >+ >+ return true; >+} >+ >+bool ResourceLoadStatisticsDatabaseStore::prepareStatements() >+{ >+ ASSERT(!RunLoop::isMain()); >+ >+ if (m_insertObservedDomainStatement->prepare() != SQLITE_OK >+ || m_insertTopLevelDomainStatement->prepare() != SQLITE_OK >+ || m_domainIDFromStringStatement->prepare() != SQLITE_OK >+ || m_storageAccessUnderTopFrameOriginsStatement->prepare() != SQLITE_OK >+ || m_topFrameUniqueRedirectsTo->prepare() != SQLITE_OK >+ || m_topFrameUniqueRedirectsToExists->prepare() != SQLITE_OK >+ || m_topFrameUniqueRedirectsFrom->prepare() != SQLITE_OK >+ || m_topFrameUniqueRedirectsFromExists->prepare() != SQLITE_OK >+ || m_subframeUnderTopFrameOrigins->prepare() != SQLITE_OK >+ || m_subframeUnderTopFrameOriginExists->prepare() != SQLITE_OK >+ || m_subresourceUnderTopFrameOrigins->prepare() != SQLITE_OK >+ || m_subresourceUnderTopFrameOriginExists->prepare() != SQLITE_OK >+ || m_subresourceUniqueRedirectsTo->prepare() != SQLITE_OK >+ || m_subresourceUniqueRedirectsToExists->prepare() != SQLITE_OK >+ || m_subresourceUniqueRedirectsFrom->prepare() != SQLITE_OK >+ || m_subresourceUniqueRedirectsFromExists->prepare() != SQLITE_OK >+ || m_updateLastSeenStatement->prepare() != SQLITE_OK >+ || m_mostRecentUserInteractionStatement->prepare() != SQLITE_OK >+ || m_updatePrevalentResourceStatement->prepare() != SQLITE_OK >+ || m_isPrevalentResourceStatement->prepare() != SQLITE_OK >+ || m_updateVeryPrevalentResourceStatement->prepare() != SQLITE_OK >+ || m_isVeryPrevalentResourceStatement->prepare() != SQLITE_OK >+ || m_clearPrevalentResourceStatement->prepare() != SQLITE_OK >+ || m_updateHadUserInteractionStatement->prepare() != SQLITE_OK >+ || m_hadUserInteractionStatement->prepare() != SQLITE_OK >+ || m_updateMostRecentUserInteractionTimeStatement->prepare() != SQLITE_OK >+ || m_updateGrandfatheredStatement->prepare() != SQLITE_OK >+ || m_isGrandfatheredStatement->prepare() != SQLITE_OK >+ || m_updateDataRecordsRemovedStatement->prepare() != SQLITE_OK >+ || m_updateTimesAccessedAsFirstPartyDueToUserInteractionStatement->prepare() != SQLITE_OK >+ || m_updateTimesAccessedAsFirstPartyDueToStorageAccessAPIStatement->prepare() != SQLITE_OK >+ || m_findExpiredUserInteractionStatement->prepare() != SQLITE_OK >+ ) { >+#if PLATFORM(COCOA) >+ RELEASE_LOG_ERROR(Network, "%p - ResourceLoadStatisticsDatabaseStore::prepareStatements failed to prepare, error message: %{public}s", this, m_database->lastErrorMsg()); >+#endif >+ ASSERT_NOT_REACHED(); >+ return false; >+ } >+ >+ return true; >+} >+ >+bool ResourceLoadStatisticsDatabaseStore::insertObservedDomain(const WebCore::ResourceLoadStatistics& loadStatistics) >+{ >+#ifndef NDEBUG >+ ASSERT(confirmDomainDoesNotExist(loadStatistics.highLevelDomain)); >+#endif >+ >+ if (m_insertObservedDomainStatement->bindText(1, loadStatistics.highLevelDomain) != SQLITE_OK >+ || m_insertObservedDomainStatement->bindDouble(2, loadStatistics.lastSeen.secondsSinceEpoch().value()) != SQLITE_OK >+ || m_insertObservedDomainStatement->bindInt(3, loadStatistics.hadUserInteraction) != SQLITE_OK >+ || m_insertObservedDomainStatement->bindDouble(4, loadStatistics.mostRecentUserInteractionTime.secondsSinceEpoch().value()) != SQLITE_OK >+ || m_insertObservedDomainStatement->bindInt(5, loadStatistics.grandfathered) != SQLITE_OK >+ || m_insertObservedDomainStatement->bindInt(6, loadStatistics.isPrevalentResource) != SQLITE_OK >+ || m_insertObservedDomainStatement->bindInt(7, loadStatistics.isVeryPrevalentResource) != SQLITE_OK >+ || m_insertObservedDomainStatement->bindInt(8, loadStatistics.dataRecordsRemoved) != SQLITE_OK >+ || m_insertObservedDomainStatement->bindInt(9, loadStatistics.timesAccessedAsFirstPartyDueToUserInteraction) != SQLITE_OK >+ || m_insertObservedDomainStatement->bindInt(10, loadStatistics.timesAccessedAsFirstPartyDueToStorageAccessAPI) != SQLITE_OK) { >+#if PLATFORM(COCOA) >+ RELEASE_LOG_ERROR(Network, "%p - ResourceLoadStatisticsDatabaseStore::insertObservedDomain failed to bind, error message: %{public}s", this, m_database->lastErrorMsg()); >+#endif >+ ASSERT_NOT_REACHED(); >+ return false; >+ } >+ >+ if (m_insertObservedDomainStatement->step() != SQLITE_DONE) { >+#if PLATFORM(COCOA) >+ RELEASE_LOG_ERROR(Network, "%p - ResourceLoadStatisticsDatabaseStore::insertObservedDomain failed to commit, error message: %{public}s", this, m_database->lastErrorMsg()); >+#endif >+ ASSERT_NOT_REACHED(); >+ return false; >+ } >+ >+ int resetResult = m_insertObservedDomainStatement->reset(); >+ ASSERT_UNUSED(resetResult, resetResult == SQLITE_OK); >+ >+ return true; >+} >+ >+bool ResourceLoadStatisticsDatabaseStore::relationshipExists(WebCore::SQLiteStatement& statement, unsigned firstDomainID, const String& secondDomain) const >+{ >+ if (statement.bindInt(1, firstDomainID) != SQLITE_OK >+ || statement.bindText(2, secondDomain) != SQLITE_OK >+ || statement.step() != SQLITE_ROW) { >+#if PLATFORM(COCOA) >+ RELEASE_LOG_ERROR(Network, "%p - ResourceLoadStatisticsDatabaseStore::m_insertDomainRelationshipStatement failed to bind, error message: %{public}s", this, const_cast<WebCore::SQLiteDatabase&>(m_database.get()).lastErrorMsg()); >+#endif >+ ASSERT_NOT_REACHED(); >+ return false; >+ } >+ >+ bool result = statement.getColumnInt(0) ? true : false; >+ >+ int resetResult = statement.reset(); >+ ASSERT_UNUSED(resetResult, resetResult == SQLITE_OK); >+ >+ return result; >+} >+ >+bool ResourceLoadStatisticsDatabaseStore::insertDomainRelationship(WebCore::SQLiteStatement& statement, unsigned domainID, const String& topFrameDomain) >+{ >+ if (statement.bindInt(1, domainID) != SQLITE_OK >+ || statement.bindText(2, topFrameDomain) != SQLITE_OK >+ || statement.step() != SQLITE_DONE) { >+#if PLATFORM(COCOA) >+ RELEASE_LOG_ERROR(Network, "%p - ResourceLoadStatisticsDatabaseStore::m_insertDomainRelationshipStatement failed to bind, error message: %{public}s", this, m_database->lastErrorMsg()); >+#endif >+ ASSERT_NOT_REACHED(); >+ return false; >+ } >+ >+ int resetResult = statement.reset(); >+ ASSERT_UNUSED(resetResult, resetResult == SQLITE_OK); >+ >+ return true; >+} >+ >+#ifndef NDEBUG >+bool ResourceLoadStatisticsDatabaseStore::confirmDomainDoesNotExist(const String& domain) const >+{ >+ if (m_domainIDFromStringStatement->bindText(1, domain) != SQLITE_OK) { >+#if PLATFORM(COCOA) >+ RELEASE_LOG_ERROR(Network, "%p - ResourceLoadStatisticsDatabaseStore::domainIDFromString failed to bind, error message: %{public}s", this, const_cast<WebCore::SQLiteDatabase&>(m_database.get()).lastErrorMsg()); >+#endif >+ ASSERT_NOT_REACHED(); >+ return false; >+ } >+ >+ if (m_domainIDFromStringStatement->step() == SQLITE_ROW) { >+ int resetResult = m_domainIDFromStringStatement->reset(); >+ ASSERT_UNUSED(resetResult, resetResult == SQLITE_OK); >+ return false; >+ } >+ >+ int resetResult = m_domainIDFromStringStatement->reset(); >+ ASSERT_UNUSED(resetResult, resetResult == SQLITE_OK); >+ >+ return true; >+} >+#endif >+ >+unsigned ResourceLoadStatisticsDatabaseStore::domainIDFromString(const String& domain) const >+{ >+ unsigned domainID = 0; >+ >+ if (m_domainIDFromStringStatement->bindText(1, domain) != SQLITE_OK >+ || m_domainIDFromStringStatement->step() != SQLITE_ROW) { >+#if PLATFORM(COCOA) >+ RELEASE_LOG_ERROR(Network, "%p - ResourceLoadStatisticsDatabaseStore::domainIDFromString failed to bind, error message: %{public}s", this, const_cast<WebCore::SQLiteDatabase&>(m_database.get()).lastErrorMsg()); >+#endif >+ ASSERT_NOT_REACHED(); >+ return domainID; >+ } >+ >+ domainID = m_domainIDFromStringStatement->getColumnInt(0); >+ >+ int resetResult = m_domainIDFromStringStatement->reset(); >+ ASSERT_UNUSED(resetResult, resetResult == SQLITE_OK); >+ >+ return domainID; >+} >+ >+void ResourceLoadStatisticsDatabaseStore::insertDomainRelationships(const WebCore::ResourceLoadStatistics& loadStatistics) >+{ >+ auto highLevelDomainID = domainIDFromString(loadStatistics.highLevelDomain); >+ for (const auto& topFrameDomain : loadStatistics.storageAccessUnderTopFrameOrigins) >+ insertDomainRelationship(m_storageAccessUnderTopFrameOriginsStatement.get(), highLevelDomainID, topFrameDomain); >+ >+ for (const auto& topFrameDomain : loadStatistics.topFrameUniqueRedirectsTo) >+ insertDomainRelationship(m_topFrameUniqueRedirectsTo.get(), highLevelDomainID, topFrameDomain.key); >+ >+ for (const auto& topFrameDomain : loadStatistics.topFrameUniqueRedirectsFrom) >+ insertDomainRelationship(m_topFrameUniqueRedirectsFrom.get(), highLevelDomainID, topFrameDomain.key); >+ >+ for (const auto& topFrameDomain : loadStatistics.subframeUnderTopFrameOrigins) >+ insertDomainRelationship(m_subframeUnderTopFrameOrigins.get(), highLevelDomainID, topFrameDomain.key); >+ >+ for (const auto& topFrameDomain : loadStatistics.subresourceUnderTopFrameOrigins) >+ insertDomainRelationship(m_subresourceUnderTopFrameOrigins.get(), highLevelDomainID, topFrameDomain.key); >+ >+ for (const auto& topFrameDomain : loadStatistics.subresourceUniqueRedirectsTo) >+ insertDomainRelationship(m_subresourceUniqueRedirectsTo.get(), highLevelDomainID, topFrameDomain.key); >+ >+ for (const auto& topFrameDomain : loadStatistics.subresourceUniqueRedirectsFrom) >+ insertDomainRelationship(m_subresourceUniqueRedirectsFrom.get(), highLevelDomainID, topFrameDomain.key); >+} >+ >+bool ResourceLoadStatisticsDatabaseStore::databaseIsEmpty() const >+{ >+ ASSERT(!RunLoop::isMain()); >+ >+ SQLiteStatement statement(const_cast<SQLiteDatabase&>(m_database.get()), "SELECT COUNT(*) FROM ObservedDomains"_s); >+ if (statement.prepare() != SQLITE_OK >+ || statement.step() != SQLITE_ROW) >+ return false; >+ >+ return statement.getColumnInt(0) ? false : true; >+} >+ >+void ResourceLoadStatisticsDatabaseStore::populateFromMemoryStore(const ResourceLoadStatisticsMemoryStore& memoryStore) >+{ >+ m_workQueue->dispatch([this, weakMemoryStore = makeWeakPtr(memoryStore)] { >+ if (!weakMemoryStore) >+ return; >+ >+ if (!databaseIsEmpty()) >+ return; >+ >+ const auto& statisticsMap = weakMemoryStore->data(); >+ for (const auto& statistic : statisticsMap) >+ insertObservedDomain(statistic.value); >+ >+ // Make a separate pass for inter-domain relationships so we >+ // can refer to the ObservedDomain table entries >+ for (const auto& statistic : statisticsMap) >+ insertDomainRelationships(statistic.value); >+ >+ m_database->runVacuumCommand(); >+ }); >+} >+ >+void ResourceLoadStatisticsDatabaseStore::calculateAndSubmitTelemetry() const >+{ >+ ASSERT(!RunLoop::isMain()); >+ >+ // FIXME(WIP): Implement for Database version. >+ // if (m_parameters.shouldSubmitTelemetry) >+ // WebResourceLoadStatisticsTelemetry::calculateAndSubmit(*this); >+} >+ >+void ResourceLoadStatisticsDatabaseStore::incrementRecordsDeletedCountForDomains(const HashSet<String>& domains) >+{ >+ StringBuilder builder; >+ for (auto domainName : domains) { >+ if (!builder.isEmpty()) >+ builder.appendLiteral(", "); >+ builder.appendLiteral("\""); >+ builder.append(domainName); >+ builder.appendLiteral("\""); >+ } >+ >+ SQLiteStatement domainsToUpdateStatement(m_database.get(), makeString("UPDATE ObservedDomains SET dataRecordsRemoved = dataRecordsRemoved + 1 WHERE highLevelDomain IN (", builder.toString(), ")")); >+ if (domainsToUpdateStatement.prepare() != SQLITE_OK) { >+#if PLATFORM(COCOA) >+ RELEASE_LOG_ERROR(Network, "%p - ResourceLoadStatisticsDatabaseStore::incrementStatisticsForDomains failed to bind, error message: %{public}s", this, m_database->lastErrorMsg()); >+#endif >+ ASSERT_NOT_REACHED(); >+ return; >+ } >+ >+ if (domainsToUpdateStatement.step() != SQLITE_DONE) { >+#if PLATFORM(COCOA) >+ RELEASE_LOG_ERROR(Network, "%p - ResourceLoadStatisticsDatabaseStore::incrementStatisticsForDomains failed to bind, error message: %{public}s", this, m_database->lastErrorMsg()); >+#endif >+ ASSERT_NOT_REACHED(); >+ } >+} >+ >+unsigned ResourceLoadStatisticsDatabaseStore::recursivelyFindNonPrevalentDomainsThatRedirectedToThisDomain(unsigned primaryDomainID, StdSet<unsigned>& nonPrevalentRedirectionSources, unsigned numberOfRecursiveCalls) >+{ >+ ASSERT(!RunLoop::isMain()); >+ >+ if (numberOfRecursiveCalls >= maxNumberOfRecursiveCallsInRedirectTraceBack) { >+ RELEASE_LOG(ResourceLoadStatistics, "Hit %u recursive calls in redirect backtrace. Returning early.", maxNumberOfRecursiveCallsInRedirectTraceBack); >+ return numberOfRecursiveCalls; >+ } >+ >+ numberOfRecursiveCalls++; >+ >+ StdSet<unsigned> newlyIdentifiedDomains; >+ SQLiteStatement findSubresources(m_database.get(), makeString("SELECT SubresourceUniqueRedirectsFrom.fromDomainID from SubresourceUniqueRedirectsFrom INNER JOIN ObservedDomains ON ObservedDomains.domainID = SubresourceUniqueRedirectsFrom.fromDomainID WHERE subresourceDomainID = ", String::number(primaryDomainID), "AND ObservedDomains.isPrevalent = 0")); >+ if (findSubresources.prepare() == SQLITE_OK) { >+ while (findSubresources.step() == SQLITE_ROW) { >+ auto newDomain = findSubresources.getColumnInt(0); >+ auto insertResult = nonPrevalentRedirectionSources.insert(newDomain); >+ if (insertResult.second) >+ newlyIdentifiedDomains.insert(newDomain); >+ } >+ } >+ >+ SQLiteStatement findTopFrames(m_database.get(), makeString("SELECT TopFrameUniqueRedirectsFrom.fromDomainID from TopFrameUniqueRedirectsFrom INNER JOIN ObservedDomains ON ObservedDomains.domainID = TopFrameUniqueRedirectsFrom.fromDomainID WHERE targetDomainID = ", String::number(primaryDomainID), "AND ObservedDomains.isPrevalent = 0")); >+ if (findTopFrames.prepare() == SQLITE_OK) { >+ while (findTopFrames.step() == SQLITE_ROW) { >+ auto newDomain = findTopFrames.getColumnInt(0); >+ auto insertResult = nonPrevalentRedirectionSources.insert(newDomain); >+ if (insertResult.second) >+ newlyIdentifiedDomains.insert(newDomain); >+ } >+ } >+ >+ if (newlyIdentifiedDomains.empty()) >+ return numberOfRecursiveCalls; >+ >+ for (auto domainID : newlyIdentifiedDomains) >+ numberOfRecursiveCalls = recursivelyFindNonPrevalentDomainsThatRedirectedToThisDomain(domainID, nonPrevalentRedirectionSources, numberOfRecursiveCalls); >+ >+ return numberOfRecursiveCalls; >+} >+ >+static String buildList(const StdSet<unsigned>& values) >+{ >+ StringBuilder builder; >+ for (auto domainID : values) { >+ if (!builder.isEmpty()) >+ builder.append(", "); >+ builder.appendNumber(domainID); >+ } >+ return builder.toString(); >+} >+ >+void ResourceLoadStatisticsDatabaseStore::markAsPrevalentIfHasRedirectedToPrevalent() >+{ >+ ASSERT(!RunLoop::isMain()); >+ >+ StdSet<unsigned> prevalentDueToRedirect; >+ SQLiteStatement subresourceRedirectStatement(const_cast<SQLiteDatabase&>(m_database.get()), "SELECT DISTINCT SubresourceUniqueRedirectsTo.subresourceDomainID FROM SubresourceUniqueRedirectsTo JOIN ObservedDomains ON ObservedDomains.domainID = SubresourceUniqueRedirectsTo.toDomainID AND ObservedDomains.isPrevalent = 1"_s); >+ if (subresourceRedirectStatement.prepare() == SQLITE_OK) { >+ while (subresourceRedirectStatement.step() == SQLITE_ROW) >+ prevalentDueToRedirect.insert(subresourceRedirectStatement.getColumnInt(0)); >+ } >+ >+ SQLiteStatement topFrameRedirectStatement(const_cast<SQLiteDatabase&>(m_database.get()), "SELECT DISTINCT TopFrameUniqueRedirectsTo.sourceDomainID FROM TopFrameUniqueRedirectsTo JOIN ObservedDomains ON ObservedDomains.domainID = TopFrameUniqueRedirectsTo.toDomainID AND ObservedDomains.isPrevalent = 1"_s); >+ if (topFrameRedirectStatement.prepare() == SQLITE_OK) { >+ while (topFrameRedirectStatement.step() == SQLITE_ROW) >+ prevalentDueToRedirect.insert(topFrameRedirectStatement.getColumnInt(0)); >+ } >+ >+ SQLiteStatement markPrevalentStatement(const_cast<SQLiteDatabase&>(m_database.get()), makeString("UPDATE ObservedDomains SET isPrevalent = 1 WHERE domainID IN (", buildList(prevalentDueToRedirect), ")")); >+ if (markPrevalentStatement.prepare() != SQLITE_OK >+ || markPrevalentStatement.step() != SQLITE_DONE) { >+#if PLATFORM(COCOA) >+ RELEASE_LOG_ERROR(Network, "%p - ResourceLoadStatisticsDatabaseStore::markAsPrevalentIfHasRedirectedToPrevalent failed to execute, error message: %{public}s", this, m_database->lastErrorMsg()); >+#endif >+ ASSERT_NOT_REACHED(); >+ } >+} >+ >+static String listToString(Vector<unsigned> values) >+{ >+ StringBuilder builder; >+ for (auto value : values) { >+ if (!builder.isEmpty()) >+ builder.appendLiteral(", "); >+ builder.appendNumber(value); >+ } >+ >+ return builder.toString(); >+} >+ >+HashMap<unsigned, ResourceLoadStatisticsDatabaseStore::NotVeryPrevalentResources> ResourceLoadStatisticsDatabaseStore::findNotVeryPrevalentResources() >+{ >+ HashMap<unsigned, NotVeryPrevalentResources> results; >+ >+ SQLiteStatement notVeryPrevalentResourcesStatement(const_cast<SQLiteDatabase&>(m_database.get()), "SELECT domainID, highLevelDomain, isPrevalent FROM ObservedDomains WHERE isVeryPrevalent = 0"_s); >+ if (notVeryPrevalentResourcesStatement.prepare() == SQLITE_OK) { >+ while (notVeryPrevalentResourcesStatement.step() == SQLITE_ROW) { >+ unsigned key = static_cast<unsigned>(notVeryPrevalentResourcesStatement.getColumnInt(0)); >+ NotVeryPrevalentResources value({ notVeryPrevalentResourcesStatement.getColumnText(1) >+ , notVeryPrevalentResourcesStatement.getColumnInt(2) ? ResourceLoadPrevalence::High : ResourceLoadPrevalence::Low >+ , 0, 0, 0, 0 }); >+ results.add(key, value); >+ } >+ } >+ >+ StringBuilder builder; >+ for (auto value : results.keys()) { >+ if (!builder.isEmpty()) >+ builder.appendLiteral(", "); >+ builder.appendNumber(value); >+ } >+ >+ auto domainIDsOfInterest = builder.toString(); >+ >+ SQLiteStatement subresourceUnderTopFrameOriginsCountStatement(const_cast<SQLiteDatabase&>(m_database.get()), makeString("SELECT subresourceDomainID, COUNT(topFrameDomainID) FROM SubresourceUnderTopFrameOrigins WHERE subresourceDomainID IN (", domainIDsOfInterest, ") GROUP BY subresourceDomainID")); >+ if (subresourceUnderTopFrameOriginsCountStatement.prepare() == SQLITE_OK) { >+ while (subresourceUnderTopFrameOriginsCountStatement.step() == SQLITE_ROW) { >+ unsigned domainID = static_cast<unsigned>(subresourceUnderTopFrameOriginsCountStatement.getColumnInt(0)); >+ auto result = results.find(domainID); >+ if (result != results.end()) >+ result->value.subresourceUnderTopFrameOriginsCount = static_cast<unsigned>(subresourceUnderTopFrameOriginsCountStatement.getColumnInt(1)); >+ } >+ } >+ >+ SQLiteStatement subresourceUniqueRedirectsToCountStatement(const_cast<SQLiteDatabase&>(m_database.get()), makeString("SELECT subresourceDomainID, COUNT(toDomainID) FROM SubresourceUniqueRedirectsTo WHERE subresourceDomainID IN (", domainIDsOfInterest, ") GROUP BY subresourceDomainID")); >+ if (subresourceUniqueRedirectsToCountStatement.prepare() == SQLITE_OK) { >+ while (subresourceUniqueRedirectsToCountStatement.step() == SQLITE_ROW) { >+ unsigned domainID = static_cast<unsigned>(subresourceUniqueRedirectsToCountStatement.getColumnInt(0)); >+ auto result = results.find(domainID); >+ if (result != results.end()) >+ result->value.subresourceUniqueRedirectsToCount = static_cast<unsigned>(subresourceUniqueRedirectsToCountStatement.getColumnInt(1)); >+ } >+ } >+ >+ SQLiteStatement subframeUnderTopFrameOriginsCountStatement(const_cast<SQLiteDatabase&>(m_database.get()), makeString("SELECT subresourceDomainID, COUNT(topFrameDomainID) FROM SubresourceUnderTopFrameOrigins WHERE subresourceDomainID IN (", domainIDsOfInterest, ") GROUP BY subresourceDomainID")); >+ if (subframeUnderTopFrameOriginsCountStatement.prepare() == SQLITE_OK) { >+ while (subframeUnderTopFrameOriginsCountStatement.step() == SQLITE_ROW) { >+ unsigned domainID = static_cast<unsigned>(subframeUnderTopFrameOriginsCountStatement.getColumnInt(0)); >+ auto result = results.find(domainID); >+ if (result != results.end()) >+ result->value.subframeUnderTopFrameOriginsCount = static_cast<unsigned>(subframeUnderTopFrameOriginsCountStatement.getColumnInt(1)); >+ } >+ } >+ >+ SQLiteStatement topFrameUniqueRedirectsToCountStatement(const_cast<SQLiteDatabase&>(m_database.get()), makeString("SELECT sourceDomainID, COUNT(toDomainID) FROM TopFrameUniqueRedirectsTo WHERE sourceDomainID IN (", domainIDsOfInterest, ") GROUP BY sourceDomainID")); >+ if (topFrameUniqueRedirectsToCountStatement.prepare() == SQLITE_OK) { >+ while (topFrameUniqueRedirectsToCountStatement.step() == SQLITE_ROW) { >+ unsigned domainID = static_cast<unsigned>(topFrameUniqueRedirectsToCountStatement.getColumnInt(0)); >+ auto result = results.find(domainID); >+ if (result != results.end()) >+ result->value.topFrameUniqueRedirectsToCount = static_cast<unsigned>(topFrameUniqueRedirectsToCountStatement.getColumnInt(1)); >+ } >+ } >+ >+ return results; >+} >+ >+void ResourceLoadStatisticsDatabaseStore::reclassifyResources() >+{ >+ ASSERT(!RunLoop::isMain()); >+ >+ auto notVeryPrevalentResources = findNotVeryPrevalentResources(); >+ >+ for (auto& resourceStatistic : notVeryPrevalentResources.values()) { >+ auto newPrevalence = m_resourceLoadStatisticsClassifier.calculateResourcePrevalence(resourceStatistic.subresourceUnderTopFrameOriginsCount, resourceStatistic.subresourceUniqueRedirectsToCount, resourceStatistic.subframeUnderTopFrameOriginsCount, resourceStatistic.topFrameUniqueRedirectsToCount, resourceStatistic.prevalence); >+ if (newPrevalence != resourceStatistic.prevalence) >+ setPrevalentResource(resourceStatistic.primaryDomain, newPrevalence); >+ } >+} >+ >+void ResourceLoadStatisticsDatabaseStore::classifyPrevalentResources() >+{ >+ ensurePrevalentResourcesForDebugMode(); >+ markAsPrevalentIfHasRedirectedToPrevalent(); >+ reclassifyResources(); >+} >+ >+void ResourceLoadStatisticsDatabaseStore::syncStorageIfNeeded() >+{ >+ m_database->runVacuumCommand(); >+} >+ >+void ResourceLoadStatisticsDatabaseStore::syncStorageImmediately() >+{ >+ m_database->runVacuumCommand(); >+} >+ >+void ResourceLoadStatisticsDatabaseStore::hasStorageAccess(const String& subFramePrimaryDomain, const String& topFramePrimaryDomain, Optional<uint64_t> frameID, uint64_t pageID, CompletionHandler<void(bool)>&& completionHandler) >+{ >+ ASSERT(!RunLoop::isMain()); >+ >+ ensureResourceStatisticsForPrimaryDomain(subFramePrimaryDomain); >+ >+ switch (cookieTreatmentForOrigin(subFramePrimaryDomain)) { >+ case CookieTreatmentResult::BlockAndPurge: >+ completionHandler(false); >+ return; >+ case CookieTreatmentResult::BlockAndKeep: >+ completionHandler(true); >+ return; >+ case CookieTreatmentResult::Allow: >+ // Do nothing >+ break; >+ }; >+ >+ RunLoop::main().dispatch([store = makeRef(m_store), subFramePrimaryDomain = subFramePrimaryDomain.isolatedCopy(), topFramePrimaryDomain = topFramePrimaryDomain.isolatedCopy(), frameID, pageID, completionHandler = WTFMove(completionHandler)]() mutable { >+ store->callHasStorageAccessForFrameHandler(subFramePrimaryDomain, topFramePrimaryDomain, frameID.value(), pageID, [store = store.copyRef(), completionHandler = WTFMove(completionHandler)](bool result) mutable { >+ store->statisticsQueue().dispatch([completionHandler = WTFMove(completionHandler), result] () mutable { >+ completionHandler(result); >+ }); >+ }); >+ }); >+} >+ >+void ResourceLoadStatisticsDatabaseStore::requestStorageAccess(String&& subFramePrimaryDomain, String&& topFramePrimaryDomain, uint64_t frameID, uint64_t pageID, bool promptEnabled, CompletionHandler<void(StorageAccessStatus)>&& completionHandler) >+{ >+ ASSERT(!RunLoop::isMain()); >+ >+ ensureResourceStatisticsForPrimaryDomain(subFramePrimaryDomain); >+ >+ switch (cookieTreatmentForOrigin(subFramePrimaryDomain)) { >+ case CookieTreatmentResult::BlockAndPurge: { >+#if !RELEASE_LOG_DISABLED >+ RELEASE_LOG_INFO_IF(m_debugLoggingEnabled, ResourceLoadStatisticsDebug, "Cannot grant storage access to %{public}s since its cookies are blocked in third-party contexts and it has not received user interaction as first-party.", subFramePrimaryDomain.utf8().data()); >+#endif >+ completionHandler(StorageAccessStatus::CannotRequestAccess); >+ } >+ return; >+ case CookieTreatmentResult::BlockAndKeep: { >+#if !RELEASE_LOG_DISABLED >+ RELEASE_LOG_INFO_IF(m_debugLoggingEnabled, ResourceLoadStatisticsDebug, "No need to grant storage access to %{public}s since its cookies are not blocked in third-party contexts.", subFramePrimaryDomain.utf8().data()); >+#endif >+ completionHandler(StorageAccessStatus::HasAccess); >+ } >+ return; >+ case CookieTreatmentResult::Allow: >+ // Do nothing >+ break; >+ }; >+ >+ UNUSED_PARAM(topFramePrimaryDomain); >+ UNUSED_PARAM(frameID); >+ UNUSED_PARAM(pageID); >+ UNUSED_PARAM(promptEnabled); >+ completionHandler(StorageAccessStatus::CannotRequestAccess); >+ >+ // FIXME(WIP): Implement StorageAcesss >+ /* >+ auto userWasPromptedEarlier = promptEnabled && hasUserGrantedStorageAccessThroughPrompt(subFrameStatistic, topFramePrimaryDomain); >+ if (promptEnabled && !userWasPromptedEarlier) { >+#if !RELEASE_LOG_DISABLED >+ RELEASE_LOG_INFO_IF(m_debugLoggingEnabled, ResourceLoadStatisticsDebug, "About to ask the user whether they want to grant storage access to %{public}s under %{public}s or not.", subFramePrimaryDomain.utf8().data(), topFramePrimaryDomain.utf8().data()); >+#endif >+ completionHandler(StorageAccessStatus::RequiresUserPrompt); >+ return; >+ } else if (userWasPromptedEarlier) { >+#if !RELEASE_LOG_DISABLED >+ RELEASE_LOG_INFO_IF(m_debugLoggingEnabled, ResourceLoadStatisticsDebug, "Storage access was granted to %{public}s under %{public}s.", subFramePrimaryDomain.utf8().data(), topFramePrimaryDomain.utf8().data()); >+#endif >+ } >+ >+ subFrameStatistic.timesAccessedAsFirstPartyDueToStorageAccessAPI++; >+ >+ grantStorageAccessInternal(WTFMove(subFramePrimaryDomain), WTFMove(topFramePrimaryDomain), frameID, pageID, userWasPromptedEarlier, [completionHandler = WTFMove(completionHandler)] (bool wasGrantedAccess) mutable { >+ completionHandler(wasGrantedAccess ? StorageAccessStatus::HasAccess : StorageAccessStatus::CannotRequestAccess); >+ }); >+ */ >+} >+ >+void ResourceLoadStatisticsDatabaseStore::requestStorageAccessUnderOpener(String&& primaryDomainInNeedOfStorageAccess, uint64_t openerPageID, String&& openerPrimaryDomain) >+{ >+ ASSERT(primaryDomainInNeedOfStorageAccess != openerPrimaryDomain); >+ ASSERT(!RunLoop::isMain()); >+ >+ if (primaryDomainInNeedOfStorageAccess == openerPrimaryDomain) >+ return; >+ >+ ensureResourceStatisticsForPrimaryDomain(primaryDomainInNeedOfStorageAccess); >+ >+ if (cookieTreatmentForOrigin(primaryDomainInNeedOfStorageAccess) != CookieTreatmentResult::Allow) >+ return; >+ >+#if !RELEASE_LOG_DISABLED >+ RELEASE_LOG_INFO_IF(m_debugLoggingEnabled, ResourceLoadStatisticsDebug, "[Temporary combatibility fix] Storage access was granted for %{public}s under opener page from %{public}s, with user interaction in the opened window.", primaryDomainInNeedOfStorageAccess.utf8().data(), openerPrimaryDomain.utf8().data()); >+#endif >+ grantStorageAccessInternal(WTFMove(primaryDomainInNeedOfStorageAccess), WTFMove(openerPrimaryDomain), WTF::nullopt, openerPageID, false, [](bool) { }); >+} >+ >+void ResourceLoadStatisticsDatabaseStore::grantStorageAccess(String&& subFrameHost, String&& topFrameHost, uint64_t frameID, uint64_t pageID, bool userWasPromptedNow, CompletionHandler<void(bool)>&& completionHandler) >+{ >+ ASSERT(!RunLoop::isMain()); >+ >+ // FIXME(WIP): Implement StorageAcesss >+ UNUSED_PARAM(subFrameHost); >+ UNUSED_PARAM(topFrameHost); >+ UNUSED_PARAM(frameID); >+ UNUSED_PARAM(pageID); >+ UNUSED_PARAM(userWasPromptedNow); >+ completionHandler(false); >+ >+ /* >+ auto subFramePrimaryDomain = ResourceLoadStatistics::primaryDomain(subFrameHost); >+ auto topFramePrimaryDomain = ResourceLoadStatistics::primaryDomain(topFrameHost); >+ if (userWasPromptedNow) { >+ auto& subFrameStatistic = ensureResourceStatisticsForPrimaryDomain(subFramePrimaryDomain); >+ ASSERT(subFrameStatistic.hadUserInteraction); >+ subFrameStatistic.storageAccessUnderTopFrameOrigins.add(topFramePrimaryDomain); >+ } >+ grantStorageAccessInternal(WTFMove(subFramePrimaryDomain), WTFMove(topFramePrimaryDomain), frameID, pageID, userWasPromptedNow, WTFMove(completionHandler)); >+ */ >+} >+ >+void ResourceLoadStatisticsDatabaseStore::grantStorageAccessInternal(String&& subFramePrimaryDomain, String&& topFramePrimaryDomain, Optional<uint64_t> frameID, uint64_t pageID, bool userWasPromptedNowOrEarlier, CompletionHandler<void(bool)>&& callback) >+{ >+ ASSERT(!RunLoop::isMain()); >+ >+ if (subFramePrimaryDomain == topFramePrimaryDomain) { >+ callback(true); >+ return; >+ } >+ >+ // FIXME(WIP): Implement for database >+ // FIXME: Remove m_storageAccessPromptsEnabled check if prompting is no longer experimental. >+ if (userWasPromptedNowOrEarlier && m_storageAccessPromptsEnabled) { >+ // auto& subFrameStatistic = ensureResourceStatisticsForPrimaryDomain(subFramePrimaryDomain); >+ // ASSERT(subFrameStatistic.hadUserInteraction); >+ // ASSERT(subFrameStatistic.storageAccessUnderTopFrameOrigins.contains(topFramePrimaryDomain)); >+ // subFrameStatistic.mostRecentUserInteractionTime = WallTime::now(); >+ } >+ >+ RunLoop::main().dispatch([subFramePrimaryDomain = subFramePrimaryDomain.isolatedCopy(), topFramePrimaryDomain = topFramePrimaryDomain.isolatedCopy(), frameID, pageID, store = makeRef(m_store), callback = WTFMove(callback)]() mutable { >+ store->callGrantStorageAccessHandler(subFramePrimaryDomain, topFramePrimaryDomain, frameID, pageID, [callback = WTFMove(callback), store = store.copyRef()](bool value) mutable { >+ store->statisticsQueue().dispatch([callback = WTFMove(callback), value] () mutable { >+ callback(value); >+ }); >+ }); >+ }); >+} >+ >+void ResourceLoadStatisticsDatabaseStore::grandfatherDataForDomains(const HashSet<String>& domains) >+{ >+ // FIXME: Implement StorageAcesss >+ ASSERT(0); >+} >+ >+Vector<String> ResourceLoadStatisticsDatabaseStore::ensurePrevalentResourcesForDebugMode() >+{ >+ if (!m_debugModeEnabled) >+ return { }; >+ >+ Vector<String> primaryDomainsToBlock; >+ primaryDomainsToBlock.reserveInitialCapacity(2); >+ >+ ensureResourceStatisticsForPrimaryDomain(debugStaticPrevalentResource); >+ setPrevalentResource(debugStaticPrevalentResource, ResourceLoadPrevalence::High); >+ primaryDomainsToBlock.uncheckedAppend(debugStaticPrevalentResource); >+ >+ if (!m_debugManualPrevalentResource.isEmpty()) { >+ ensureResourceStatisticsForPrimaryDomain(m_debugManualPrevalentResource); >+ setPrevalentResource(m_debugManualPrevalentResource, ResourceLoadPrevalence::High); >+ primaryDomainsToBlock.uncheckedAppend(m_debugManualPrevalentResource); >+#if !RELEASE_LOG_DISABLED >+ RELEASE_LOG_INFO(ResourceLoadStatisticsDebug, "Did set %{public}s as prevalent resource for the purposes of ITP Debug Mode.", m_debugManualPrevalentResource.utf8().data()); >+#endif >+ } >+ >+ return primaryDomainsToBlock; >+} >+ >+void ResourceLoadStatisticsDatabaseStore::logFrameNavigation(const String& targetPrimaryDomain, const String& mainFramePrimaryDomain, const String& sourcePrimaryDomain, const String& targetHost, const String& mainFrameHost, bool isRedirect, bool isMainFrame) >+{ >+ ASSERT(!RunLoop::isMain()); >+ >+ bool areTargetAndMainFrameDomainsAssociated = targetPrimaryDomain == mainFramePrimaryDomain; >+ bool areTargetAndSourceDomainsAssociated = targetPrimaryDomain == sourcePrimaryDomain; >+ >+ bool statisticsWereUpdated = false; >+ if (!isMainFrame && targetHost != mainFrameHost && !(areTargetAndMainFrameDomainsAssociated || areTargetAndSourceDomainsAssociated)) { >+ auto targetResult = ensureResourceStatisticsForPrimaryDomain(targetPrimaryDomain); >+ updateLastSeen(targetPrimaryDomain, ResourceLoadStatistics::reduceTimeResolution(WallTime::now())); >+ >+ if (!relationshipExists(m_subframeUnderTopFrameOriginExists.get(), targetResult.second, mainFramePrimaryDomain)) { >+ insertDomainRelationship(m_subframeUnderTopFrameOrigins.get(), targetResult.second, mainFramePrimaryDomain); >+ statisticsWereUpdated = true; >+ } >+ } >+ >+ if (isRedirect && !areTargetAndSourceDomainsAssociated) { >+ if (isMainFrame) { >+ auto redirectingOriginResult = ensureResourceStatisticsForPrimaryDomain(sourcePrimaryDomain); >+ if (!relationshipExists(m_topFrameUniqueRedirectsToExists.get(), redirectingOriginResult.second, targetPrimaryDomain)) { >+ insertDomainRelationship(m_topFrameUniqueRedirectsTo.get(), redirectingOriginResult.second, targetPrimaryDomain); >+ statisticsWereUpdated = true; >+ } >+ >+ auto targetResult = ensureResourceStatisticsForPrimaryDomain(targetPrimaryDomain); >+ if (!relationshipExists(m_topFrameUniqueRedirectsFromExists.get(), targetResult.second, sourcePrimaryDomain)) { >+ insertDomainRelationship(m_topFrameUniqueRedirectsFrom.get(), targetResult.second, sourcePrimaryDomain); >+ statisticsWereUpdated = true; >+ } >+ } else { >+ auto redirectingOriginResult = ensureResourceStatisticsForPrimaryDomain(sourcePrimaryDomain); >+ if (!relationshipExists(m_subresourceUniqueRedirectsToExists.get(), redirectingOriginResult.second, targetPrimaryDomain)) { >+ insertDomainRelationship(m_subresourceUniqueRedirectsTo.get(), redirectingOriginResult.second, targetPrimaryDomain); >+ statisticsWereUpdated = true; >+ } >+ >+ auto targetResult = ensureResourceStatisticsForPrimaryDomain(targetPrimaryDomain); >+ if (!relationshipExists(m_subresourceUniqueRedirectsFromExists.get(), targetResult.second, sourcePrimaryDomain)) { >+ insertDomainRelationship(m_subresourceUniqueRedirectsFrom.get(), targetResult.second, sourcePrimaryDomain); >+ statisticsWereUpdated = true; >+ } >+ } >+ } >+ >+ if (statisticsWereUpdated) >+ scheduleStatisticsProcessingRequestIfNecessary(); >+} >+ >+void ResourceLoadStatisticsDatabaseStore::logSubresourceLoading(const String& targetPrimaryDomain, const String& mainFramePrimaryDomain, WallTime lastSeen) >+{ >+ ASSERT(!RunLoop::isMain()); >+ >+ auto result = ensureResourceStatisticsForPrimaryDomain(targetPrimaryDomain); >+ updateLastSeen(targetPrimaryDomain, lastSeen); >+ >+ auto targetDomainID = result.second; >+ if (!relationshipExists(m_subresourceUnderTopFrameOriginExists.get(), targetDomainID, mainFramePrimaryDomain)) { >+ insertDomainRelationship(m_subresourceUnderTopFrameOrigins.get(), targetDomainID, mainFramePrimaryDomain); >+ scheduleStatisticsProcessingRequestIfNecessary(); >+ } >+} >+ >+void ResourceLoadStatisticsDatabaseStore::logSubresourceRedirect(const String& sourcePrimaryDomain, const String& targetPrimaryDomain) >+{ >+ ASSERT(!RunLoop::isMain()); >+ >+ auto sourceDomainResult = ensureResourceStatisticsForPrimaryDomain(sourcePrimaryDomain); >+ auto targetDomainResult = ensureResourceStatisticsForPrimaryDomain(targetPrimaryDomain); >+ >+ bool isNewRedirectToEntry = false; >+ if (!relationshipExists(m_subresourceUniqueRedirectsToExists.get(), sourceDomainResult.second, targetPrimaryDomain)) { >+ insertDomainRelationship(m_subresourceUniqueRedirectsTo.get(), sourceDomainResult.second, targetPrimaryDomain); >+ isNewRedirectToEntry = true; >+ } >+ >+ bool isNewRedirectFromEntry = false; >+ if (!relationshipExists(m_subresourceUniqueRedirectsFromExists.get(), targetDomainResult.second, sourcePrimaryDomain)) { >+ insertDomainRelationship(m_subresourceUniqueRedirectsFrom.get(), targetDomainResult.second, sourcePrimaryDomain); >+ isNewRedirectFromEntry = true; >+ } >+ >+ if (isNewRedirectToEntry || isNewRedirectFromEntry) >+ scheduleStatisticsProcessingRequestIfNecessary(); >+} >+ >+void ResourceLoadStatisticsDatabaseStore::setUserInteraction(const String& primaryDomain, bool hadUserInteraction, WallTime mostRecentInteraction) >+{ >+ if (m_mostRecentUserInteractionStatement->bindInt(1, hadUserInteraction) >+ || m_mostRecentUserInteractionStatement->bindDouble(2, mostRecentInteraction.secondsSinceEpoch().value()) != SQLITE_OK >+ || m_mostRecentUserInteractionStatement->bindText(3, primaryDomain) != SQLITE_OK >+ || m_mostRecentUserInteractionStatement->step() != SQLITE_DONE) { >+#if PLATFORM(COCOA) >+ RELEASE_LOG_ERROR(Network, "%p - ResourceLoadStatisticsDatabaseStore::setUserInteraction, error message: %{public}s", this, m_database->lastErrorMsg()); >+#endif >+ ASSERT_NOT_REACHED(); >+ return; >+ } >+ >+ int resetResult = m_mostRecentUserInteractionStatement->reset(); >+ ASSERT_UNUSED(resetResult, resetResult == SQLITE_OK); >+} >+ >+void ResourceLoadStatisticsDatabaseStore::logUserInteraction(const String& primaryDomain) >+{ >+ ASSERT(!RunLoop::isMain()); >+ >+ ensureResourceStatisticsForPrimaryDomain(primaryDomain); >+ setUserInteraction(primaryDomain, true, WallTime::now()); >+} >+ >+void ResourceLoadStatisticsDatabaseStore::clearUserInteraction(const String& primaryDomain) >+{ >+ ASSERT(!RunLoop::isMain()); >+ >+ auto targetResult = ensureResourceStatisticsForPrimaryDomain(primaryDomain); >+ setUserInteraction(primaryDomain, false, { }); >+ >+ SQLiteStatement removeStorageAccess(m_database.get(), makeString("DELETE FROM StorageAccessUnderTopFrameOrigins WHERE domainID = ", String::number(targetResult.second))); >+ if (removeStorageAccess.prepare() != SQLITE_OK >+ || removeStorageAccess.step() != SQLITE_DONE) { >+#if PLATFORM(COCOA) >+ RELEASE_LOG_ERROR(Network, "%p - ResourceLoadStatisticsDatabaseStore::logUserInteraction failed to bind, error message: %{public}s", this, m_database->lastErrorMsg()); >+#endif >+ ASSERT_NOT_REACHED(); >+ } >+} >+ >+bool ResourceLoadStatisticsDatabaseStore::hasHadUserInteraction(const String& primaryDomain) >+{ >+ ASSERT(!RunLoop::isMain()); >+ >+ if (m_hadUserInteractionStatement->bindText(1, primaryDomain) != SQLITE_OK >+ || m_hadUserInteractionStatement->step() != SQLITE_DONE) { >+#if PLATFORM(COCOA) >+ RELEASE_LOG_ERROR(Network, "%p - ResourceLoadStatisticsDatabaseStore::m_updatePrevalentResourceStatement failed to bind, error message: %{public}s", this, m_database->lastErrorMsg()); >+#endif >+ ASSERT_NOT_REACHED(); >+ return false; >+ } >+ >+ bool hadUserInteraction = m_hadUserInteractionStatement->getColumnInt(0) ? true : false; >+ if (!hadUserInteraction) >+ return false; >+ >+ WallTime mostRecentUserInteractionTime = WallTime::fromRawSeconds(m_hadUserInteractionStatement->getColumnDouble(1)); >+ >+ if (hasStatisticsExpired(mostRecentUserInteractionTime)) { >+ // Drop privacy sensitive data because we no longer need it. >+ // Set timestamp to 0 so that statistics merge will know >+ // it has been reset as opposed to its default -1. >+ clearUserInteraction(primaryDomain); >+ hadUserInteraction = false; >+ } >+ >+ return hadUserInteraction; >+} >+ >+void ResourceLoadStatisticsDatabaseStore::setPrevalentResource(const String& primaryDomain, ResourceLoadPrevalence newPrevalence) >+{ >+ ASSERT(!RunLoop::isMain()); >+ >+ if (m_updatePrevalentResourceStatement->bindInt(1, 1) != SQLITE_OK >+ || m_updatePrevalentResourceStatement->bindText(2, primaryDomain) != SQLITE_OK >+ || m_updatePrevalentResourceStatement->step() != SQLITE_DONE) { >+#if PLATFORM(COCOA) >+ RELEASE_LOG_ERROR(Network, "%p - ResourceLoadStatisticsDatabaseStore::m_updatePrevalentResourceStatement failed, error message: %{public}s", this, m_database->lastErrorMsg()); >+#endif >+ ASSERT_NOT_REACHED(); >+ return; >+ } >+ >+ int resetResult = m_updatePrevalentResourceStatement->reset(); >+ ASSERT_UNUSED(resetResult, resetResult == SQLITE_OK); >+ >+ if (newPrevalence == ResourceLoadPrevalence::VeryHigh) { >+ if (m_updateVeryPrevalentResourceStatement->bindInt(1, 1) != SQLITE_OK >+ || m_updateVeryPrevalentResourceStatement->bindText(2, primaryDomain) != SQLITE_OK >+ || m_updateVeryPrevalentResourceStatement->step() != SQLITE_DONE) { >+#if PLATFORM(COCOA) >+ RELEASE_LOG_ERROR(Network, "%p - ResourceLoadStatisticsDatabaseStore::m_updateVeryPrevalentResourceStatement failed to bind, error message: %{public}s", this, m_database->lastErrorMsg()); >+#endif >+ ASSERT_NOT_REACHED(); >+ return; >+ } >+ >+ int resetResult = m_updateVeryPrevalentResourceStatement->reset(); >+ ASSERT_UNUSED(resetResult, resetResult == SQLITE_OK); >+ } >+ >+ StdSet<unsigned> nonPrevalentRedirectionSources; >+ auto primaryDomainID = domainIDFromString(primaryDomain); >+ recursivelyFindNonPrevalentDomainsThatRedirectedToThisDomain(primaryDomainID, nonPrevalentRedirectionSources, 0); >+ setDomainsAsPrevalent(WTFMove(nonPrevalentRedirectionSources)); >+} >+ >+void ResourceLoadStatisticsDatabaseStore::setDomainsAsPrevalent(StdSet<unsigned>&& domains) >+{ >+ SQLiteStatement domainsToUpdateStatement(m_database.get(), makeString("UPDATE ObservedDomains SET isPrevalent = 1 WHERE domainID IN (", buildList(domains), ")")); >+ if (domainsToUpdateStatement.prepare() != SQLITE_OK >+ || domainsToUpdateStatement.step() != SQLITE_DONE) { >+#if PLATFORM(COCOA) >+ RELEASE_LOG_ERROR(Network, "%p - ResourceLoadStatisticsDatabaseStore::setDomainsAsPrevalent failed, error message: %{public}s", this, m_database->lastErrorMsg()); >+#endif >+ ASSERT_NOT_REACHED(); >+ return; >+ } >+} >+ >+String ResourceLoadStatisticsDatabaseStore::dumpResourceLoadStatistics() const >+{ >+ ASSERT(!RunLoop::isMain()); >+ >+ // FIXME: Implement StorageAcesss >+ ASSERT(0); >+} >+ >+bool ResourceLoadStatisticsDatabaseStore::predicateValueForDomain(WebCore::SQLiteStatement& predicateStatement, const String& primaryDomain) const >+{ >+ ASSERT(!RunLoop::isMain()); >+ >+ if (predicateStatement.bindText(1, primaryDomain) != SQLITE_OK >+ || predicateStatement.step() != SQLITE_DONE) { >+#if PLATFORM(COCOA) >+ RELEASE_LOG_ERROR(Network, "%p - ResourceLoadStatisticsDatabaseStore::predicateValueForDomain failed to bind, error message: %{public}s", this, const_cast<WebCore::SQLiteDatabase&>(m_database.get()).lastErrorMsg()); >+#endif >+ ASSERT_NOT_REACHED(); >+ return false; >+ } >+ >+ bool result = predicateStatement.getColumnInt(0) ? true : false; >+ int resetResult = predicateStatement.reset(); >+ ASSERT_UNUSED(resetResult, resetResult == SQLITE_OK); >+ return result; >+} >+ >+bool ResourceLoadStatisticsDatabaseStore::isPrevalentResource(const String& primaryDomain) const >+{ >+ ASSERT(!RunLoop::isMain()); >+ >+ return predicateValueForDomain(m_isPrevalentResourceStatement.get(), primaryDomain); >+} >+ >+bool ResourceLoadStatisticsDatabaseStore::isVeryPrevalentResource(const String& primaryDomain) const >+{ >+ ASSERT(!RunLoop::isMain()); >+ >+ return predicateValueForDomain(m_isVeryPrevalentResourceStatement.get(), primaryDomain); >+} >+ >+bool ResourceLoadStatisticsDatabaseStore::isRegisteredAsSubresourceUnder(const String& subresourcePrimaryDomain, const String& topFramePrimaryDomain) const >+{ >+ ASSERT(!RunLoop::isMain()); >+ >+ auto firstDomainID = domainIDFromString(subresourcePrimaryDomain); >+ return relationshipExists(m_subresourceUnderTopFrameOriginExists.get(), firstDomainID, topFramePrimaryDomain); >+} >+ >+bool ResourceLoadStatisticsDatabaseStore::isRegisteredAsSubFrameUnder(const String& subframePrimaryDomain, const String& topFramePrimaryDomain) const >+{ >+ ASSERT(!RunLoop::isMain()); >+ >+ auto firstDomainID = domainIDFromString(subframePrimaryDomain); >+ return relationshipExists(m_subframeUnderTopFrameOriginExists.get(), firstDomainID, topFramePrimaryDomain); >+} >+ >+bool ResourceLoadStatisticsDatabaseStore::isRegisteredAsRedirectingTo(const String& hostRedirectedFromPrimaryDomain, const String& hostRedirectedToPrimaryDomain) const >+{ >+ ASSERT(!RunLoop::isMain()); >+ >+ auto firstDomainID = domainIDFromString(hostRedirectedFromPrimaryDomain); >+ return relationshipExists(m_subresourceUniqueRedirectsToExists.get(), firstDomainID, hostRedirectedToPrimaryDomain); >+} >+ >+void ResourceLoadStatisticsDatabaseStore::clearPrevalentResource(const String& primaryDomain) >+{ >+ ASSERT(!RunLoop::isMain()); >+ >+ ensureResourceStatisticsForPrimaryDomain(primaryDomain); >+ >+ if (m_clearPrevalentResourceStatement->bindText(1, primaryDomain) != SQLITE_OK >+ || m_clearPrevalentResourceStatement->step() != SQLITE_DONE) { >+#if PLATFORM(COCOA) >+ RELEASE_LOG_ERROR(Network, "%p - ResourceLoadStatisticsDatabaseStore::clearPrevalentResource, error message: %{public}s", this, const_cast<WebCore::SQLiteDatabase&>(m_database.get()).lastErrorMsg()); >+#endif >+ ASSERT_NOT_REACHED(); >+ return; >+ } >+ >+ int resetResult = m_clearPrevalentResourceStatement->reset(); >+ ASSERT_UNUSED(resetResult, resetResult == SQLITE_OK); >+} >+ >+void ResourceLoadStatisticsDatabaseStore::setGrandfathered(const String& primaryDomain, bool value) >+{ >+ ASSERT(!RunLoop::isMain()); >+ >+ ensureResourceStatisticsForPrimaryDomain(primaryDomain); >+ >+ if (m_updateGrandfatheredStatement->bindInt(1, value) != SQLITE_OK >+ || m_updateGrandfatheredStatement->bindText(1, primaryDomain) != SQLITE_OK >+ || m_updateGrandfatheredStatement->step() != SQLITE_DONE) { >+#if PLATFORM(COCOA) >+ RELEASE_LOG_ERROR(Network, "%p - ResourceLoadStatisticsDatabaseStore::setGrandfathered failed to bind, error message: %{public}s", this, const_cast<WebCore::SQLiteDatabase&>(m_database.get()).lastErrorMsg()); >+#endif >+ ASSERT_NOT_REACHED(); >+ return; >+ } >+ >+ int resetResult = m_updateGrandfatheredStatement->reset(); >+ ASSERT_UNUSED(resetResult, resetResult == SQLITE_OK); >+} >+ >+bool ResourceLoadStatisticsDatabaseStore::isGrandfathered(const String& primaryDomain) const >+{ >+ ASSERT(!RunLoop::isMain()); >+ >+ return predicateValueForDomain(m_isGrandfatheredStatement.get(), primaryDomain); >+} >+ >+void ResourceLoadStatisticsDatabaseStore::setSubframeUnderTopFrameOrigin(const String& primarySubFrameDomain, const String& primaryTopFrameDomain) >+{ >+ ASSERT(!RunLoop::isMain()); >+ >+ auto result = ensureResourceStatisticsForPrimaryDomain(primarySubFrameDomain); >+ >+ // For consistency, make sure we also have a statistics entry for the top frame domain. >+ ensureResourceStatisticsForPrimaryDomain(primaryTopFrameDomain); >+ >+ insertDomainRelationship(m_subframeUnderTopFrameOrigins.get(), result.second, primaryTopFrameDomain); >+} >+ >+void ResourceLoadStatisticsDatabaseStore::setSubresourceUnderTopFrameOrigin(const String& primarySubresourceDomain, const String& primaryTopFrameDomain) >+{ >+ ASSERT(!RunLoop::isMain()); >+ >+ auto result = ensureResourceStatisticsForPrimaryDomain(primarySubresourceDomain); >+ >+ // For consistency, make sure we also have a statistics entry for the top frame domain. >+ ensureResourceStatisticsForPrimaryDomain(primaryTopFrameDomain); >+ >+ insertDomainRelationship(m_subresourceUnderTopFrameOrigins.get(), result.second, primaryTopFrameDomain); >+} >+ >+void ResourceLoadStatisticsDatabaseStore::setSubresourceUniqueRedirectTo(const String& primarySubresourceDomain, const String& primaryRedirectDomain) >+{ >+ ASSERT(!RunLoop::isMain()); >+ >+ auto result = ensureResourceStatisticsForPrimaryDomain(primarySubresourceDomain); >+ >+ // For consistency, make sure we also have a statistics entry for the redirect domain. >+ ensureResourceStatisticsForPrimaryDomain(primaryRedirectDomain); >+ >+ UNUSED_PARAM(primaryRedirectDomain); >+ >+ insertDomainRelationship(m_subresourceUniqueRedirectsTo.get(), result.second, primaryRedirectDomain); >+} >+ >+void ResourceLoadStatisticsDatabaseStore::setSubresourceUniqueRedirectFrom(const String& primarySubresourceDomain, const String& primaryRedirectDomain) >+{ >+ ASSERT(!RunLoop::isMain()); >+ >+ auto result = ensureResourceStatisticsForPrimaryDomain(primarySubresourceDomain); >+ >+ // For consistency, make sure we also have a statistics entry for the redirect domain. >+ ensureResourceStatisticsForPrimaryDomain(primaryRedirectDomain); >+ >+ insertDomainRelationship(m_subresourceUniqueRedirectsFrom.get(), result.second, primaryRedirectDomain); >+} >+ >+void ResourceLoadStatisticsDatabaseStore::setTopFrameUniqueRedirectTo(const String& topFramePrimaryDomain, const String& primaryRedirectDomain) >+{ >+ ASSERT(!RunLoop::isMain()); >+ >+ auto result = ensureResourceStatisticsForPrimaryDomain(topFramePrimaryDomain); >+ >+ // For consistency, make sure we also have a statistics entry for the redirect domain. >+ ensureResourceStatisticsForPrimaryDomain(primaryRedirectDomain); >+ >+ insertDomainRelationship(m_topFrameUniqueRedirectsTo.get(), result.second, primaryRedirectDomain); >+} >+ >+void ResourceLoadStatisticsDatabaseStore::setTopFrameUniqueRedirectFrom(const String& topFramePrimaryDomain, const String& primaryRedirectDomain) >+{ >+ ASSERT(!RunLoop::isMain()); >+ >+ auto result = ensureResourceStatisticsForPrimaryDomain(topFramePrimaryDomain); >+ >+ // For consistency, make sure we also have a statistics entry for the redirect domain. >+ ensureResourceStatisticsForPrimaryDomain(primaryRedirectDomain); >+ >+ insertDomainRelationship(m_topFrameUniqueRedirectsFrom.get(), result.second, primaryRedirectDomain); >+} >+ >+std::pair<ResourceLoadStatisticsDatabaseStore::AddedRecord, unsigned> ResourceLoadStatisticsDatabaseStore::ensureResourceStatisticsForPrimaryDomain(const String& primaryDomain) >+{ >+ ASSERT(!RunLoop::isMain()); >+ >+ if (m_domainIDFromStringStatement->bindText(1, primaryDomain) != SQLITE_OK) { >+#if PLATFORM(COCOA) >+ RELEASE_LOG_ERROR(Network, "%p - ResourceLoadStatisticsDatabaseStore::createSchema failed, error message: %{public}s", this, m_database->lastErrorMsg()); >+#endif >+ ASSERT_NOT_REACHED(); >+ return { AddedRecord::No, 0 }; >+ } >+ >+ if (m_domainIDFromStringStatement->step() == SQLITE_ROW) { >+ unsigned domainID = m_domainIDFromStringStatement->getColumnInt(0); >+ >+ int resetResult = m_domainIDFromStringStatement->reset(); >+ ASSERT_UNUSED(resetResult, resetResult == SQLITE_OK); >+ return { AddedRecord::No, domainID }; >+ } >+ >+ int resetResult = m_domainIDFromStringStatement->reset(); >+ ASSERT_UNUSED(resetResult, resetResult == SQLITE_OK); >+ >+ ResourceLoadStatistics newObservation(primaryDomain); >+ insertObservedDomain(newObservation); >+ >+ return { AddedRecord::Yes, domainIDFromString(primaryDomain) }; >+} >+ >+void ResourceLoadStatisticsDatabaseStore::clear(CompletionHandler<void()>&& completionHandler) >+{ >+ ASSERT(!RunLoop::isMain()); >+ >+ m_resourceStatisticsMap.clear(); >+ m_operatingDates.clear(); >+ >+ auto callbackAggregator = CallbackAggregator::create(WTFMove(completionHandler)); >+ >+ removeAllStorageAccess([callbackAggregator = callbackAggregator.copyRef()] { }); >+ >+ auto primaryDomainsToBlock = ensurePrevalentResourcesForDebugMode(); >+ updateCookieBlockingForDomains(primaryDomainsToBlock, [callbackAggregator = callbackAggregator.copyRef()] { }); >+} >+ >+ResourceLoadStatisticsDatabaseStore::CookieTreatmentResult ResourceLoadStatisticsDatabaseStore::cookieTreatmentForOrigin(const String& primaryDomain) const >+{ >+ SQLiteStatement statement(const_cast<SQLiteDatabase&>(m_database.get()), makeString("SELECT isPrevalent, hadUserInteraction FROM ObservedDomains WHERE highLevelDomain = 1", primaryDomain)); >+ if (statement.prepare() != SQLITE_OK >+ || statement.step() != SQLITE_ROW) { >+ return CookieTreatmentResult::Allow; >+ } >+ >+ bool isPrevalent = statement.getColumnInt(0) ? true : false; >+ if (!isPrevalent) >+ return CookieTreatmentResult::Allow; >+ >+ bool hadUserInteraction = statement.getColumnInt(1) ? true : false; >+ return hadUserInteraction ? CookieTreatmentResult::BlockAndKeep : CookieTreatmentResult::BlockAndPurge; >+} >+ >+bool ResourceLoadStatisticsDatabaseStore::hasUserGrantedStorageAccessThroughPrompt(const ResourceLoadStatistics& statistic, const String& firstPartyPrimaryDomain) >+{ >+ return statistic.storageAccessUnderTopFrameOrigins.contains(firstPartyPrimaryDomain); >+} >+ >+Vector<String> ResourceLoadStatisticsDatabaseStore::domainsToBlock() const >+{ >+ ASSERT(!RunLoop::isMain()); >+ >+ Vector<String> results; >+ SQLiteStatement statement(const_cast<SQLiteDatabase&>(m_database.get()), "SELECT highLevelDomain FROM ObservedDomains WHERE isPrevalent = 1"_s); >+ if (statement.prepare() != SQLITE_OK) >+ return results; >+ >+ while (statement.step() == SQLITE_ROW) >+ results.append(statement.getColumnText(0)); >+ >+ return results; >+} >+ >+void ResourceLoadStatisticsDatabaseStore::updateCookieBlocking(CompletionHandler<void()>&& completionHandler) >+{ >+ ASSERT(!RunLoop::isMain()); >+ >+ Vector<String> domainsToBlock = this->domainsToBlock(); >+ >+ if (domainsToBlock.isEmpty()) { >+ completionHandler(); >+ return; >+ } >+ >+ if (m_debugLoggingEnabled && !domainsToBlock.isEmpty()) >+ debugLogDomainsInBatches("block", domainsToBlock); >+ >+ RunLoop::main().dispatch([weakThis = makeWeakPtr(*this), store = makeRef(m_store), domainsToBlock = crossThreadCopy(domainsToBlock), completionHandler = WTFMove(completionHandler)] () mutable { >+ store->callUpdatePrevalentDomainsToBlockCookiesForHandler(domainsToBlock, [weakThis = WTFMove(weakThis), store = store.copyRef(), completionHandler = WTFMove(completionHandler)]() mutable { >+ store->statisticsQueue().dispatch([weakThis = WTFMove(weakThis), completionHandler = WTFMove(completionHandler)]() mutable { >+ completionHandler(); >+ if (!weakThis) >+ return; >+#if !RELEASE_LOG_DISABLED >+ RELEASE_LOG_INFO_IF(weakThis->m_debugLoggingEnabled, ResourceLoadStatisticsDebug, "Done updating cookie blocking."); >+#endif >+ }); >+ }); >+ }); >+} >+ >+void ResourceLoadStatisticsDatabaseStore::updateCookieBlockingForDomains(const Vector<String>& domainsToBlock, CompletionHandler<void()>&& completionHandler) >+{ >+ ASSERT(!RunLoop::isMain()); >+ >+ RunLoop::main().dispatch([store = makeRef(m_store), domainsToBlock = crossThreadCopy(domainsToBlock), completionHandler = WTFMove(completionHandler)] () mutable { >+ store->callUpdatePrevalentDomainsToBlockCookiesForHandler(domainsToBlock, [store = store.copyRef(), completionHandler = WTFMove(completionHandler)]() mutable { >+ store->statisticsQueue().dispatch([completionHandler = WTFMove(completionHandler)]() mutable { >+ completionHandler(); >+ }); >+ }); >+ }); >+} >+ >+void ResourceLoadStatisticsDatabaseStore::clearBlockingStateForDomains(const Vector<String>& domains, CompletionHandler<void()>&& completionHandler) >+{ >+ ASSERT(!RunLoop::isMain()); >+ >+ if (domains.isEmpty()) { >+ completionHandler(); >+ return; >+ } >+ >+ RunLoop::main().dispatch([store = makeRef(m_store), domains = crossThreadCopy(domains)] { >+ store->callRemoveDomainsHandler(domains); >+ }); >+ >+ completionHandler(); >+} >+ >+void ResourceLoadStatisticsDatabaseStore::processStatistics(const Function<void(const ResourceLoadStatistics&)>& processFunction) const >+{ >+ ASSERT(!RunLoop::isMain()); >+ >+ for (auto& resourceStatistic : m_resourceStatisticsMap.values()) >+ processFunction(resourceStatistic); >+} >+ >+Vector<ResourceLoadStatisticsDatabaseStore::PrevalentDomainData> ResourceLoadStatisticsDatabaseStore::prevalentDomains() const >+{ >+ ASSERT(!RunLoop::isMain()); >+ >+ Vector<PrevalentDomainData> results; >+ SQLiteStatement statement(const_cast<SQLiteDatabase&>(m_database.get()), "SELECT domainID, highLevelDomain, mostRecentUserInteractionTime, hadUserInteraction, grandfathered FROM ObservedDomains WHERE isPrevalent = 1"_s); >+ if (statement.prepare() != SQLITE_OK) >+ return results; >+ >+ while (statement.step() == SQLITE_ROW) { >+ results.append({ static_cast<unsigned>(statement.getColumnInt(0)) >+ , statement.getColumnText(1) >+ , WallTime::fromRawSeconds(statement.getColumnDouble(2)) >+ , statement.getColumnInt(3) ? true : false >+ , statement.getColumnInt(4) ? true : false >+ }); >+ } >+ >+ return results; >+} >+ >+Vector<unsigned> ResourceLoadStatisticsDatabaseStore::findExpiredUserInteractions() const >+{ >+ ASSERT(!RunLoop::isMain()); >+ >+ Vector<unsigned> results; >+ Optional<Seconds> expirationDateTime = statisticsEpirationTime(); >+ if (!expirationDateTime) >+ return results; >+ >+ auto& statement = const_cast<SQLiteStatement&>(m_findExpiredUserInteractionStatement.get()); >+ if (statement.bindDouble(1, expirationDateTime.value().value()) != SQLITE_OK) >+ return results; >+ >+ while (statement.step() == SQLITE_ROW) >+ results.append(statement.getColumnInt(0)); >+ >+ return results; >+} >+ >+void ResourceLoadStatisticsDatabaseStore::clearExpiredUserInteractions() >+{ >+ ASSERT(!RunLoop::isMain()); >+ >+ auto expiredRecords = findExpiredUserInteractions(); >+ if (expiredRecords.isEmpty()) >+ return; >+ >+ auto expiredRecordIDs = listToString(expiredRecords); >+ >+ SQLiteStatement clearExpiredInteraction(m_database.get(), makeString("UPDATE ObservedDomains SET mostRecentUserInteractionTime = 0, hadUserInteraction = 1 WHERE domainID IN (", expiredRecordIDs, ")")); >+ if (clearExpiredInteraction.prepare() != SQLITE_OK) >+ return; >+ >+ SQLiteStatement removeStorageAccess(m_database.get(), makeString("DELETE FROM StorageAccessUnderTopFrameOrigins ", expiredRecordIDs, ")")); >+ if (removeStorageAccess.prepare() != SQLITE_OK) >+ return; >+ >+ if (clearExpiredInteraction.step() != SQLITE_DONE >+ || removeStorageAccess.step() != SQLITE_DONE) { >+#if PLATFORM(COCOA) >+ RELEASE_LOG_ERROR(Network, "%p - ResourceLoadStatisticsDatabaseStore::logUserInteraction failed to bind, error message: %{public}s", this, m_database->lastErrorMsg()); >+#endif >+ ASSERT_NOT_REACHED(); >+ } >+} >+ >+void ResourceLoadStatisticsDatabaseStore::clearGrandfathering(Vector<unsigned>&& domainIDsToClear) >+{ >+ ASSERT(!RunLoop::isMain()); >+ >+ if (domainIDsToClear.isEmpty()) >+ return; >+ >+ SQLiteStatement clearGrandfatheringStatement(m_database.get(), makeString("UPDATE ObservedDomains SET grandfathered = 0 WHERE domainID IN (", listToString(domainIDsToClear), ")")); >+ if (clearGrandfatheringStatement.prepare() != SQLITE_OK) >+ return; >+ >+ if (clearGrandfatheringStatement.step() != SQLITE_DONE) { >+#if PLATFORM(COCOA) >+ RELEASE_LOG_ERROR(Network, "%p - ResourceLoadStatisticsDatabaseStore::clearGrandfathering failed to bind, error message: %{public}s", this, m_database->lastErrorMsg()); >+#endif >+ ASSERT_NOT_REACHED(); >+ } >+} >+ >+Vector<String> ResourceLoadStatisticsDatabaseStore::topPrivatelyControlledDomainsToRemoveWebsiteDataFor() >+{ >+ ASSERT(!RunLoop::isMain()); >+ >+ bool shouldCheckForGrandfathering = m_endOfGrandfatheringTimestamp > WallTime::now(); >+ bool shouldClearGrandfathering = !shouldCheckForGrandfathering && m_endOfGrandfatheringTimestamp; >+ >+ if (shouldClearGrandfathering) >+ m_endOfGrandfatheringTimestamp = { }; >+ >+ clearExpiredUserInteractions(); >+ >+ Vector<String> prevalentResources; >+ >+ Vector<PrevalentDomainData> prevalentDomains = this->prevalentDomains(); >+ Vector<unsigned> domainsToClearGrandfathering; >+ for (auto& statistic : prevalentDomains) { >+ if (!statistic.hadUserInteraction && (!shouldCheckForGrandfathering || !statistic.grandfathered)) >+ prevalentResources.append(statistic.primaryDomain); >+ >+ if (shouldClearGrandfathering && statistic.grandfathered) >+ domainsToClearGrandfathering.append(statistic.domainID); >+ } >+ >+ clearGrandfathering(WTFMove(domainsToClearGrandfathering)); >+ >+ return prevalentResources; >+} >+ >+void ResourceLoadStatisticsDatabaseStore::pruneStatisticsIfNeeded() >+{ >+ ASSERT(!RunLoop::isMain()); >+ >+ if (m_resourceStatisticsMap.size() <= m_parameters.maxStatisticsEntries) >+ return; >+ >+ ASSERT(m_parameters.pruneEntriesDownTo <= m_parameters.maxStatisticsEntries); >+ >+ size_t numberOfEntriesLeftToPrune = m_resourceStatisticsMap.size() - m_parameters.pruneEntriesDownTo; >+ ASSERT(numberOfEntriesLeftToPrune); >+ >+ // FIXME(WIP): Implement via database >+ ASSERT(0); >+ /* >+ Vector<db_StatisticsLastSeen> resourcesToPrunePerImportance[maxImportance + 1]; >+ for (auto& resourceStatistic : m_resourceStatisticsMap.values()) >+ resourcesToPrunePerImportance[computeImportance(resourceStatistic)].append({ resourceStatistic.highLevelDomain, resourceStatistic.lastSeen }); >+ >+ for (unsigned importance = 0; numberOfEntriesLeftToPrune && importance <= maxImportance; ++importance) >+ pruneResources(m_resourceStatisticsMap, resourcesToPrunePerImportance[importance], numberOfEntriesLeftToPrune); >+ */ >+ ASSERT(!numberOfEntriesLeftToPrune); >+} >+ >+void ResourceLoadStatisticsDatabaseStore::updateLastSeen(const String& domain, WallTime lastSeen) >+{ >+ ASSERT(!RunLoop::isMain()); >+ >+ if (m_updateLastSeenStatement->bindDouble(1, lastSeen.secondsSinceEpoch().value()) != SQLITE_OK >+ || m_updateLastSeenStatement->bindText(2, domain) != SQLITE_OK >+ || m_updateLastSeenStatement->step() != SQLITE_DONE) { >+#if PLATFORM(COCOA) >+ RELEASE_LOG_ERROR(Network, "%p - ResourceLoadStatisticsDatabaseStore::updateLastSeen failed to bind, error message: %{public}s", this, m_database->lastErrorMsg()); >+#endif >+ ASSERT_NOT_REACHED(); >+ return; >+ } >+ >+ int resetResult = m_updateLastSeenStatement->reset(); >+ ASSERT_UNUSED(resetResult, resetResult == SQLITE_OK); >+} >+ >+void ResourceLoadStatisticsDatabaseStore::setLastSeen(const String& primaryDomain, Seconds seconds) >+{ >+ ASSERT(!RunLoop::isMain()); >+ >+ ensureResourceStatisticsForPrimaryDomain(primaryDomain); >+ updateLastSeen(primaryDomain, WallTime::fromRawSeconds(seconds.seconds())); >+} >+ >+void ResourceLoadStatisticsDatabaseStore::setPrevalentResource(const String& primaryDomain) >+{ >+ ASSERT(!RunLoop::isMain()); >+ >+ ensureResourceStatisticsForPrimaryDomain(primaryDomain); >+ setPrevalentResource(primaryDomain, ResourceLoadPrevalence::High); >+} >+ >+void ResourceLoadStatisticsDatabaseStore::setVeryPrevalentResource(const String& primaryDomain) >+{ >+ ASSERT(!RunLoop::isMain()); >+ >+ ensureResourceStatisticsForPrimaryDomain(primaryDomain); >+ setPrevalentResource(primaryDomain, ResourceLoadPrevalence::VeryHigh); >+} >+ >+void ResourceLoadStatisticsDatabaseStore::removeAllStorageAccess(CompletionHandler<void()>&& completionHandler) >+{ >+ ASSERT(!RunLoop::isMain()); >+ RunLoop::main().dispatch([store = makeRef(m_store), completionHandler = WTFMove(completionHandler)]() mutable { >+ store->removeAllStorageAccess([store = store.copyRef(), completionHandler = WTFMove(completionHandler)]() mutable { >+ store->statisticsQueue().dispatch([completionHandler = WTFMove(completionHandler)]() mutable { >+ completionHandler(); >+ }); >+ }); >+ }); >+} >+ >+} // namespace WebKit >+ >+#endif >diff --git a/Source/WebKit/NetworkProcess/Classifier/ResourceLoadStatisticsDatabaseStore.h b/Source/WebKit/NetworkProcess/Classifier/ResourceLoadStatisticsDatabaseStore.h >new file mode 100644 >index 0000000000000000000000000000000000000000..467f27b6074a553c9283145f9a22fa0a62052074 >--- /dev/null >+++ b/Source/WebKit/NetworkProcess/Classifier/ResourceLoadStatisticsDatabaseStore.h >@@ -0,0 +1,209 @@ >+/* >+ * Copyright (C) 2019 Apple Inc. All rights reserved. >+ * >+ * Redistribution and use in source and binary forms, with or without >+ * modification, are permitted provided that the following conditions >+ * are met: >+ * 1. Redistributions of source code must retain the above copyright >+ * notice, this list of conditions and the following disclaimer. >+ * 2. Redistributions in binary form must reproduce the above copyright >+ * notice, this list of conditions and the following disclaimer in the >+ * documentation and/or other materials provided with the distribution. >+ * >+ * THIS SOFTWARE IS PROVIDED BY APPLE INC. AND ITS CONTRIBUTORS ``AS IS'' >+ * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, >+ * THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR >+ * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL APPLE INC. OR ITS CONTRIBUTORS >+ * BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR >+ * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF >+ * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS >+ * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN >+ * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) >+ * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF >+ * THE POSSIBILITY OF SUCH DAMAGE. >+ */ >+ >+#pragma once >+ >+#if ENABLE(RESOURCE_LOAD_STATISTICS) >+ >+#include "ResourceLoadStatisticsStore.h" >+#include "WebResourceLoadStatisticsStore.h" >+#include <wtf/CompletionHandler.h> >+#include <wtf/UniqueRef.h> >+#include <wtf/Vector.h> >+#include <wtf/WeakPtr.h> >+#include <wtf/WorkQueue.h> >+ >+namespace WebCore { >+class SQLiteDatabase; >+class SQLiteStatement; >+struct ResourceLoadStatistics; >+} >+ >+namespace WebKit { >+ >+class ResourceLoadStatisticsMemoryStore; >+ >+// This is always constructed / used / destroyed on the WebResourceLoadStatisticsStore's statistics queue. >+class ResourceLoadStatisticsDatabaseStore : public ResourceLoadStatisticsStore { >+public: >+ ResourceLoadStatisticsDatabaseStore(WebResourceLoadStatisticsStore&, WorkQueue&, const String& storageDirectoryPath); >+ >+ void populateFromMemoryStore(const ResourceLoadStatisticsMemoryStore&); >+ >+ void clear(CompletionHandler<void()>&&) override; >+ bool isEmpty() const { return false; /*m_resourceStatisticsMap.isEmpty();*/ } >+ >+ void processStatistics(const Function<void(const WebCore::ResourceLoadStatistics&)>&) const; >+ >+ void updateCookieBlocking(CompletionHandler<void()>&&) override; >+ void updateCookieBlockingForDomains(const Vector<String>& domainsToBlock, CompletionHandler<void()>&&); >+ void clearBlockingStateForDomains(const Vector<String>& domains, CompletionHandler<void()>&&); >+ >+ void classifyPrevalentResources() override; >+ void syncStorageIfNeeded() override; >+ void syncStorageImmediately() override; >+ >+ void requestStorageAccessUnderOpener(String&& primaryDomainInNeedOfStorageAccess, uint64_t openerPageID, String&& openerPrimaryDomain) override; >+ void removeAllStorageAccess(CompletionHandler<void()>&&); >+ >+ void grandfatherDataForDomains(const HashSet<String>&) override; >+ >+ bool isRegisteredAsSubresourceUnder(const String& subresourcePrimaryDomain, const String& topFramePrimaryDomain) const override; >+ bool isRegisteredAsSubFrameUnder(const String& subFramePrimaryDomain, const String& topFramePrimaryDomain) const override; >+ bool isRegisteredAsRedirectingTo(const String& hostRedirectedFromPrimaryDomain, const String& hostRedirectedToPrimaryDomain) const override; >+ >+ void clearPrevalentResource(const String& primaryDomain) override; >+ String dumpResourceLoadStatistics() const override; >+ bool isPrevalentResource(const String& primaryDomain) const override; >+ bool isVeryPrevalentResource(const String& primaryDomain) const override; >+ void setPrevalentResource(const String& primaryDomain) override; >+ void setVeryPrevalentResource(const String& primaryDomain) override; >+ >+ void setGrandfathered(const String& primaryDomain, bool value) override; >+ bool isGrandfathered(const String& primaryDomain) const override; >+ >+ void setSubframeUnderTopFrameOrigin(const String& primarySubFrameDomain, const String& primaryTopFrameDomain) override; >+ void setSubresourceUnderTopFrameOrigin(const String& primarySubresourceDomain, const String& primaryTopFrameDomain) override; >+ void setSubresourceUniqueRedirectTo(const String& primarySubresourceDomain, const String& primaryRedirectDomain) override; >+ void setSubresourceUniqueRedirectFrom(const String& primarySubresourceDomain, const String& primaryRedirectDomain) override; >+ void setTopFrameUniqueRedirectTo(const String& topFramePrimaryDomain, const String& primaryRedirectDomain) override; >+ void setTopFrameUniqueRedirectFrom(const String& topFramePrimaryDomain, const String& primaryRedirectDomain) override; >+ >+ void calculateAndSubmitTelemetry() const override; >+ >+ void hasStorageAccess(const String& subFramePrimaryDomain, const String& topFramePrimaryDomain, Optional<uint64_t> frameID, uint64_t pageID, CompletionHandler<void(bool)>&&) override; >+ void requestStorageAccess(String&& subFramePrimaryDomain, String&& topFramePrimaryDomain, uint64_t frameID, uint64_t pageID, bool promptEnabled, CompletionHandler<void(StorageAccessStatus)>&&) override; >+ void grantStorageAccess(String&& subFrameHost, String&& topFrameHost, uint64_t frameID, uint64_t pageID, bool userWasPromptedNow, CompletionHandler<void(bool)>&&) override; >+ >+ void logFrameNavigation(const String& targetPrimaryDomain, const String& mainFramePrimaryDomain, const String& sourcePrimaryDomain, const String& targetHost, const String& mainFrameHost, bool isRedirect, bool isMainFrame) override; >+ void logUserInteraction(const String& primaryDomain) override; >+ void logSubresourceLoading(const String& targetPrimaryDomain, const String& mainFramePrimaryDomain, WallTime lastSeen) override; >+ void logSubresourceRedirect(const String& sourcePrimaryDomain, const String& targetPrimaryDomain) override; >+ >+ void clearUserInteraction(const String& primaryDomain) override; >+ bool hasHadUserInteraction(const String& primaryDomain) override; >+ >+ void setLastSeen(const String& primaryDomain, Seconds) override; >+ >+private: >+ bool databaseIsEmpty() const; >+ bool insertObservedDomain(const WebCore::ResourceLoadStatistics&); >+ void insertDomainRelationships(const WebCore::ResourceLoadStatistics&); >+ bool insertDomainRelationship(WebCore::SQLiteStatement&, unsigned domainID, const String& topFrameDomain); >+ bool relationshipExists(WebCore::SQLiteStatement&, unsigned firstDomainID, const String& secondDomain) const; >+ unsigned domainIDFromString(const String&) const; >+#ifndef NDEBUG >+ bool confirmDomainDoesNotExist(const String&) const; >+#endif >+ void updateLastSeen(const String& domain, WallTime); >+ void updateMostRecentUserInteraction(const String& domain, WallTime); >+ void setUserInteraction(const String& primaryDomain, bool hadUserInteraction, WallTime); >+ Vector<String> domainsToBlock() const; >+ >+ struct PrevalentDomainData { >+ unsigned domainID; >+ String primaryDomain; >+ WallTime mostRecentUserInteractionTime; >+ bool hadUserInteraction; >+ bool grandfathered; >+ }; >+ Vector<PrevalentDomainData> prevalentDomains() const; >+ Vector<unsigned> findExpiredUserInteractions() const; >+ void clearExpiredUserInteractions(); >+ void clearGrandfathering(Vector<unsigned>&&); >+ void incrementRecordsDeletedCountForDomains(const HashSet<String>&) override; >+ >+ void reclassifyResources(); >+ struct NotVeryPrevalentResources { >+ String primaryDomain; >+ ResourceLoadPrevalence prevalence; >+ unsigned subresourceUnderTopFrameOriginsCount; >+ unsigned subresourceUniqueRedirectsToCount; >+ unsigned subframeUnderTopFrameOriginsCount; >+ unsigned topFrameUniqueRedirectsToCount; >+ }; >+ HashMap<unsigned, NotVeryPrevalentResources> findNotVeryPrevalentResources(); >+ >+ bool predicateValueForDomain(WebCore::SQLiteStatement&, const String&) const; >+ >+ enum class CookieTreatmentResult { Allow, BlockAndKeep, BlockAndPurge }; >+ CookieTreatmentResult cookieTreatmentForOrigin(const String&) const; >+ >+ static bool hasUserGrantedStorageAccessThroughPrompt(const WebCore::ResourceLoadStatistics&, const String& firstPartyPrimaryDomain); >+ void setPrevalentResource(const String& primaryDomain, ResourceLoadPrevalence); >+ unsigned recursivelyFindNonPrevalentDomainsThatRedirectedToThisDomain(unsigned primaryDomainID, StdSet<unsigned>& nonPrevalentRedirectionSources, unsigned numberOfRecursiveCalls); >+ void setDomainsAsPrevalent(StdSet<unsigned>&&); >+ void grantStorageAccessInternal(String&& subFrameHost, String&& topFrameHost, Optional<uint64_t> frameID, uint64_t pageID, bool userWasPromptedNowOrEarlier, CompletionHandler<void(bool)>&&); >+ void markAsPrevalentIfHasRedirectedToPrevalent(); >+ Vector<String> ensurePrevalentResourcesForDebugMode() override; >+ void removeDataRecords(CompletionHandler<void()>&&); >+ void pruneStatisticsIfNeeded() override; >+ enum class AddedRecord { No, Yes }; >+ std::pair<AddedRecord, unsigned> ensureResourceStatisticsForPrimaryDomain(const String&); >+ Vector<String> topPrivatelyControlledDomainsToRemoveWebsiteDataFor() override; >+ >+ bool createSchema(); >+ bool prepareStatements(); >+ >+ const String m_storageDirectoryPath; >+ HashMap<String, WebCore::ResourceLoadStatistics> m_resourceStatisticsMap; >+ UniqueRef<WebCore::SQLiteDatabase> m_database; >+ UniqueRef<WebCore::SQLiteStatement> m_insertObservedDomainStatement; >+ UniqueRef<WebCore::SQLiteStatement> m_insertTopLevelDomainStatement; >+ mutable UniqueRef<WebCore::SQLiteStatement> m_domainIDFromStringStatement; >+ UniqueRef<WebCore::SQLiteStatement> m_storageAccessUnderTopFrameOriginsStatement; >+ UniqueRef<WebCore::SQLiteStatement> m_topFrameUniqueRedirectsTo; >+ mutable UniqueRef<WebCore::SQLiteStatement> m_topFrameUniqueRedirectsToExists; >+ UniqueRef<WebCore::SQLiteStatement> m_topFrameUniqueRedirectsFrom; >+ mutable UniqueRef<WebCore::SQLiteStatement> m_topFrameUniqueRedirectsFromExists; >+ UniqueRef<WebCore::SQLiteStatement> m_subframeUnderTopFrameOrigins; >+ mutable UniqueRef<WebCore::SQLiteStatement> m_subframeUnderTopFrameOriginExists; >+ UniqueRef<WebCore::SQLiteStatement> m_subresourceUnderTopFrameOrigins; >+ mutable UniqueRef<WebCore::SQLiteStatement> m_subresourceUnderTopFrameOriginExists; >+ UniqueRef<WebCore::SQLiteStatement> m_subresourceUniqueRedirectsTo; >+ mutable UniqueRef<WebCore::SQLiteStatement> m_subresourceUniqueRedirectsToExists; >+ UniqueRef<WebCore::SQLiteStatement> m_subresourceUniqueRedirectsFrom; >+ mutable UniqueRef<WebCore::SQLiteStatement> m_subresourceUniqueRedirectsFromExists; >+ UniqueRef<WebCore::SQLiteStatement> m_mostRecentUserInteractionStatement; >+ UniqueRef<WebCore::SQLiteStatement> m_updateLastSeenStatement; >+ UniqueRef<WebCore::SQLiteStatement> m_updatePrevalentResourceStatement; >+ mutable UniqueRef<WebCore::SQLiteStatement> m_isPrevalentResourceStatement; >+ UniqueRef<WebCore::SQLiteStatement> m_updateVeryPrevalentResourceStatement; >+ mutable UniqueRef<WebCore::SQLiteStatement> m_isVeryPrevalentResourceStatement; >+ UniqueRef<WebCore::SQLiteStatement> m_clearPrevalentResourceStatement; >+ UniqueRef<WebCore::SQLiteStatement> m_updateHadUserInteractionStatement; >+ mutable UniqueRef<WebCore::SQLiteStatement> m_hadUserInteractionStatement; >+ UniqueRef<WebCore::SQLiteStatement> m_updateMostRecentUserInteractionTimeStatement; >+ UniqueRef<WebCore::SQLiteStatement> m_updateGrandfatheredStatement; >+ mutable UniqueRef<WebCore::SQLiteStatement> m_isGrandfatheredStatement; >+ UniqueRef<WebCore::SQLiteStatement> m_updateDataRecordsRemovedStatement; >+ UniqueRef<WebCore::SQLiteStatement> m_updateTimesAccessedAsFirstPartyDueToUserInteractionStatement; >+ UniqueRef<WebCore::SQLiteStatement> m_updateTimesAccessedAsFirstPartyDueToStorageAccessAPIStatement; >+ UniqueRef<WebCore::SQLiteStatement> m_findExpiredUserInteractionStatement; >+}; >+ >+} // namespace WebKit >+ >+#endif >diff --git a/Source/WebKit/NetworkProcess/Classifier/ResourceLoadStatisticsMemoryStore.cpp b/Source/WebKit/NetworkProcess/Classifier/ResourceLoadStatisticsMemoryStore.cpp >index 7ee2ad10100b9418aca0d0948f76c80c4fe48f59..d5704d369679591e9f7b336dd4d9efcfebfa9ca2 100644 >--- a/Source/WebKit/NetworkProcess/Classifier/ResourceLoadStatisticsMemoryStore.cpp >+++ b/Source/WebKit/NetworkProcess/Classifier/ResourceLoadStatisticsMemoryStore.cpp >@@ -49,95 +49,6 @@ namespace WebKit { > using namespace WebCore; > > constexpr unsigned statisticsModelVersion { 14 }; >-constexpr unsigned maxNumberOfRecursiveCallsInRedirectTraceBack { 50 }; >-constexpr Seconds minimumStatisticsProcessingInterval { 5_s }; >-constexpr unsigned operatingDatesWindow { 30 }; >-constexpr unsigned maxImportance { 3 }; >-static const char* debugStaticPrevalentResource { "3rdpartytestwebkit.org" }; >- >-#if !RELEASE_LOG_DISABLED >-static String domainsToString(const Vector<String>& domains) >-{ >- StringBuilder builder; >- for (auto& domain : domains) { >- if (!builder.isEmpty()) >- builder.appendLiteral(", "); >- builder.append(domain); >- } >- return builder.toString(); >-} >-#endif >- >-class OperatingDate { >-public: >- OperatingDate() = default; >- >- static OperatingDate fromWallTime(WallTime time) >- { >- double ms = time.secondsSinceEpoch().milliseconds(); >- int year = msToYear(ms); >- int yearDay = dayInYear(ms, year); >- int month = monthFromDayInYear(yearDay, isLeapYear(year)); >- int monthDay = dayInMonthFromDayInYear(yearDay, isLeapYear(year)); >- >- return OperatingDate { year, month, monthDay }; >- } >- >- static OperatingDate today() >- { >- return OperatingDate::fromWallTime(WallTime::now()); >- } >- >- Seconds secondsSinceEpoch() const >- { >- return Seconds { dateToDaysFrom1970(m_year, m_month, m_monthDay) * secondsPerDay }; >- } >- >- bool operator==(const OperatingDate& other) const >- { >- return m_monthDay == other.m_monthDay && m_month == other.m_month && m_year == other.m_year; >- } >- >- bool operator<(const OperatingDate& other) const >- { >- return secondsSinceEpoch() < other.secondsSinceEpoch(); >- } >- >- bool operator<=(const OperatingDate& other) const >- { >- return secondsSinceEpoch() <= other.secondsSinceEpoch(); >- } >- >-private: >- OperatingDate(int year, int month, int monthDay) >- : m_year(year) >- , m_month(month) >- , m_monthDay(monthDay) >- { } >- >- int m_year { 0 }; >- int m_month { 0 }; // [0, 11]. >- int m_monthDay { 0 }; // [1, 31]. >-}; >- >-static Vector<OperatingDate> mergeOperatingDates(const Vector<OperatingDate>& existingDates, Vector<OperatingDate>&& newDates) >-{ >- if (existingDates.isEmpty()) >- return WTFMove(newDates); >- >- Vector<OperatingDate> mergedDates(existingDates.size() + newDates.size()); >- >- // Merge the two sorted vectors of dates. >- std::merge(existingDates.begin(), existingDates.end(), newDates.begin(), newDates.end(), mergedDates.begin()); >- // Remove duplicate dates. >- removeRepeatedElements(mergedDates); >- >- // Drop old dates until the Vector size reaches operatingDatesWindow. >- while (mergedDates.size() > operatingDatesWindow) >- mergedDates.remove(0); >- >- return mergedDates; >-} > > struct StatisticsLastSeen { > String topPrivatelyOwnedDomain; >@@ -156,26 +67,14 @@ static void pruneResources(HashMap<String, WebCore::ResourceLoadStatistics>& sta > statisticsMap.remove(statisticsToPrune[i].topPrivatelyOwnedDomain); > } > >-static unsigned computeImportance(const ResourceLoadStatistics& resourceStatistic) >-{ >- unsigned importance = maxImportance; >- if (!resourceStatistic.isPrevalentResource) >- importance -= 1; >- if (!resourceStatistic.hadUserInteraction) >- importance -= 2; >- return importance; >-} >- > ResourceLoadStatisticsMemoryStore::ResourceLoadStatisticsMemoryStore(WebResourceLoadStatisticsStore& store, WorkQueue& workQueue) >- : m_store(store) >- , m_workQueue(workQueue) >+ : ResourceLoadStatisticsStore(store, workQueue) > { > ASSERT(!RunLoop::isMain()); > > #if PLATFORM(COCOA) > registerUserDefaultsIfNeeded(); > #endif >- includeTodayAsOperatingDateIfNecessary(); > > m_workQueue->dispatchAfter(5_s, [weakThis = makeWeakPtr(*this)] { > if (weakThis) >@@ -183,11 +82,6 @@ ResourceLoadStatisticsMemoryStore::ResourceLoadStatisticsMemoryStore(WebResource > }); > } > >-ResourceLoadStatisticsMemoryStore::~ResourceLoadStatisticsMemoryStore() >-{ >- ASSERT(!RunLoop::isMain()); >-} >- > void ResourceLoadStatisticsMemoryStore::setPersistentStorage(ResourceLoadStatisticsPersistentStorage& persistentStorage) > { > m_persistentStorage = makeWeakPtr(persistentStorage); >@@ -201,75 +95,12 @@ void ResourceLoadStatisticsMemoryStore::calculateAndSubmitTelemetry() const > WebResourceLoadStatisticsTelemetry::calculateAndSubmit(*this); > } > >-void ResourceLoadStatisticsMemoryStore::setNotifyPagesWhenDataRecordsWereScanned(bool value) >+void ResourceLoadStatisticsMemoryStore::incrementRecordsDeletedCountForDomains(const HashSet<String>& domains) > { >- ASSERT(!RunLoop::isMain()); >- m_parameters.shouldNotifyPagesWhenDataRecordsWereScanned = value; >-} >- >-void ResourceLoadStatisticsMemoryStore::setShouldClassifyResourcesBeforeDataRecordsRemoval(bool value) >-{ >- ASSERT(!RunLoop::isMain()); >- m_parameters.shouldClassifyResourcesBeforeDataRecordsRemoval = value; >-} >- >-void ResourceLoadStatisticsMemoryStore::setShouldSubmitTelemetry(bool value) >-{ >- ASSERT(!RunLoop::isMain()); >- m_parameters.shouldSubmitTelemetry = value; >-} >- >-void ResourceLoadStatisticsMemoryStore::removeDataRecords(CompletionHandler<void()>&& callback) >-{ >- ASSERT(!RunLoop::isMain()); >- >- if (!shouldRemoveDataRecords()) { >- callback(); >- return; >+ for (auto& prevalentResourceDomain : domains) { >+ auto& statistic = ensureResourceStatisticsForPrimaryDomain(prevalentResourceDomain); >+ ++statistic.dataRecordsRemoved; > } >- >-#if ENABLE(NETSCAPE_PLUGIN_API) >- m_activePluginTokens.clear(); >- for (const auto& plugin : PluginProcessManager::singleton().pluginProcesses()) >- m_activePluginTokens.add(plugin->pluginProcessToken()); >-#endif >- >- auto prevalentResourceDomains = topPrivatelyControlledDomainsToRemoveWebsiteDataFor(); >- if (prevalentResourceDomains.isEmpty()) { >- callback(); >- return; >- } >- >-#if !RELEASE_LOG_DISABLED >- RELEASE_LOG_INFO_IF(m_debugLoggingEnabled, ResourceLoadStatisticsDebug, "About to remove data records for %{public}s.", domainsToString(prevalentResourceDomains).utf8().data()); >-#endif >- >- setDataRecordsBeingRemoved(true); >- >- RunLoop::main().dispatch([prevalentResourceDomains = crossThreadCopy(prevalentResourceDomains), callback = WTFMove(callback), weakThis = makeWeakPtr(*this), shouldNotifyPagesWhenDataRecordsWereScanned = m_parameters.shouldNotifyPagesWhenDataRecordsWereScanned, workQueue = m_workQueue.copyRef()] () mutable { >- if (!weakThis) { >- callback(); >- return; >- } >- >- weakThis->m_store.deleteWebsiteDataForTopPrivatelyControlledDomainsInAllPersistentDataStores(WebResourceLoadStatisticsStore::monitoredDataTypes(), WTFMove(prevalentResourceDomains), shouldNotifyPagesWhenDataRecordsWereScanned, [callback = WTFMove(callback), weakThis = WTFMove(weakThis), workQueue = workQueue.copyRef()](const HashSet<String>& domainsWithDeletedWebsiteData) mutable { >- workQueue->dispatch([topDomains = crossThreadCopy(domainsWithDeletedWebsiteData), callback = WTFMove(callback), weakThis = WTFMove(weakThis)] () mutable { >- if (!weakThis) { >- callback(); >- return; >- } >- for (auto& prevalentResourceDomain : topDomains) { >- auto& statistic = weakThis->ensureResourceStatisticsForPrimaryDomain(prevalentResourceDomain); >- ++statistic.dataRecordsRemoved; >- } >- weakThis->setDataRecordsBeingRemoved(false); >- callback(); >-#if !RELEASE_LOG_DISABLED >- RELEASE_LOG_INFO_IF(weakThis->m_debugLoggingEnabled, ResourceLoadStatisticsDebug, "Done removing data records."); >-#endif >- }); >- }); >- }); > } > > unsigned ResourceLoadStatisticsMemoryStore::recursivelyGetAllDomainsThatHaveRedirectedToThisDomain(const WebCore::ResourceLoadStatistics& resourceStatistic, HashSet<String>& domainsThatHaveRedirectedTo, unsigned numberOfRecursiveCalls) const >@@ -336,44 +167,31 @@ bool ResourceLoadStatisticsMemoryStore::isPrevalentDueToDebugMode(ResourceLoadSt > return resourceStatistic.highLevelDomain == debugStaticPrevalentResource || resourceStatistic.highLevelDomain == m_debugManualPrevalentResource; > } > >-void ResourceLoadStatisticsMemoryStore::processStatisticsAndDataRecords() >+void ResourceLoadStatisticsMemoryStore::classifyPrevalentResources() > { >- ASSERT(!RunLoop::isMain()); >- >- if (m_parameters.shouldClassifyResourcesBeforeDataRecordsRemoval) { >- for (auto& resourceStatistic : m_resourceStatisticsMap.values()) { >- if (isPrevalentDueToDebugMode(resourceStatistic)) >- setPrevalentResource(resourceStatistic, ResourceLoadPrevalence::High); >- else if (!resourceStatistic.isVeryPrevalentResource) { >- markAsPrevalentIfHasRedirectedToPrevalent(resourceStatistic); >- auto currentPrevalence = resourceStatistic.isPrevalentResource ? ResourceLoadPrevalence::High : ResourceLoadPrevalence::Low; >- auto newPrevalence = m_resourceLoadStatisticsClassifier.calculateResourcePrevalence(resourceStatistic, currentPrevalence); >- if (newPrevalence != currentPrevalence) >- setPrevalentResource(resourceStatistic, newPrevalence); >- } >+ for (auto& resourceStatistic : m_resourceStatisticsMap.values()) { >+ if (isPrevalentDueToDebugMode(resourceStatistic)) >+ setPrevalentResource(resourceStatistic, ResourceLoadPrevalence::High); >+ else if (!resourceStatistic.isVeryPrevalentResource) { >+ markAsPrevalentIfHasRedirectedToPrevalent(resourceStatistic); >+ auto currentPrevalence = resourceStatistic.isPrevalentResource ? ResourceLoadPrevalence::High : ResourceLoadPrevalence::Low; >+ auto newPrevalence = m_resourceLoadStatisticsClassifier.calculateResourcePrevalence(resourceStatistic, currentPrevalence); >+ if (newPrevalence != currentPrevalence) >+ setPrevalentResource(resourceStatistic, newPrevalence); > } > } >+} > >- removeDataRecords([this, weakThis = makeWeakPtr(*this)] { >- ASSERT(!RunLoop::isMain()); >- if (!weakThis) >- return; >- >- pruneStatisticsIfNeeded(); >- if (m_persistentStorage) >- m_persistentStorage->scheduleOrWriteMemoryStore(ResourceLoadStatisticsPersistentStorage::ForceImmediateWrite::No); >- >- if (!m_parameters.shouldNotifyPagesWhenDataRecordsWereScanned) >- return; >- >- RunLoop::main().dispatch([this, weakThis = makeWeakPtr(*this)] { >- ASSERT(RunLoop::isMain()); >- if (!weakThis) >- return; >+void ResourceLoadStatisticsMemoryStore::syncStorageIfNeeded() >+{ >+ if (m_persistentStorage) >+ m_persistentStorage->scheduleOrWriteMemoryStore(ResourceLoadStatisticsPersistentStorage::ForceImmediateWrite::No); >+} > >- m_store.notifyResourceLoadStatisticsProcessed(); >- }); >- }); >+void ResourceLoadStatisticsMemoryStore::syncStorageImmediately() >+{ >+ if (m_persistentStorage) >+ m_persistentStorage->scheduleOrWriteMemoryStore(ResourceLoadStatisticsPersistentStorage::ForceImmediateWrite::Yes); > } > > void ResourceLoadStatisticsMemoryStore::hasStorageAccess(const String& subFramePrimaryDomain, const String& topFramePrimaryDomain, Optional<uint64_t> frameID, uint64_t pageID, CompletionHandler<void(bool)>&& completionHandler) >@@ -502,30 +320,12 @@ void ResourceLoadStatisticsMemoryStore::grantStorageAccessInternal(String&& subF > }); > } > >-void ResourceLoadStatisticsMemoryStore::grandfatherExistingWebsiteData(CompletionHandler<void()>&& callback) >+void ResourceLoadStatisticsMemoryStore::grandfatherDataForDomains(const HashSet<String>& domains) > { >- ASSERT(!RunLoop::isMain()); >- >- RunLoop::main().dispatch([weakThis = makeWeakPtr(*this), callback = WTFMove(callback), shouldNotifyPagesWhenDataRecordsWereScanned = m_parameters.shouldNotifyPagesWhenDataRecordsWereScanned, workQueue = m_workQueue.copyRef(), store = makeRef(m_store)] () mutable { >- store->topPrivatelyControlledDomainsWithWebsiteData(WebResourceLoadStatisticsStore::monitoredDataTypes(), shouldNotifyPagesWhenDataRecordsWereScanned, [weakThis = WTFMove(weakThis), callback = WTFMove(callback), workQueue = workQueue.copyRef()] (HashSet<String>&& topDomainsWithWebsiteData) mutable { >- workQueue->dispatch([weakThis = WTFMove(weakThis), topDomains = crossThreadCopy(topDomainsWithWebsiteData), callback = WTFMove(callback)] () mutable { >- if (!weakThis) { >- callback(); >- return; >- } >- >- for (auto& topPrivatelyControlledDomain : topDomains) { >- auto& statistic = weakThis->ensureResourceStatisticsForPrimaryDomain(topPrivatelyControlledDomain); >- statistic.grandfathered = true; >- } >- weakThis->m_endOfGrandfatheringTimestamp = WallTime::now() + weakThis->m_parameters.grandfatheringTime; >- if (weakThis->m_persistentStorage) >- weakThis->m_persistentStorage->scheduleOrWriteMemoryStore(ResourceLoadStatisticsPersistentStorage::ForceImmediateWrite::Yes); >- callback(); >- weakThis->logTestingEvent("Grandfathered"_s); >- }); >- }); >- }); >+ for (auto& topPrivatelyControlledDomain : domains) { >+ auto& statistic = ensureResourceStatisticsForPrimaryDomain(topPrivatelyControlledDomain); >+ statistic.grandfathered = true; >+ } > } > > Vector<String> ResourceLoadStatisticsMemoryStore::ensurePrevalentResourcesForDebugMode() >@@ -552,55 +352,6 @@ Vector<String> ResourceLoadStatisticsMemoryStore::ensurePrevalentResourcesForDeb > return primaryDomainsToBlock; > } > >-void ResourceLoadStatisticsMemoryStore::setResourceLoadStatisticsDebugMode(bool enable) >-{ >- ASSERT(!RunLoop::isMain()); >- >-#if !RELEASE_LOG_DISABLED >- if (enable) >- RELEASE_LOG_INFO(ResourceLoadStatisticsDebug, "Turned ITP Debug Mode on."); >-#endif >- >- m_debugModeEnabled = enable; >- m_debugLoggingEnabled = enable; >- >- ensurePrevalentResourcesForDebugMode(); >- // This will log the current cookie blocking state. >- if (enable) >- updateCookieBlocking([]() { }); >-} >- >-void ResourceLoadStatisticsMemoryStore::setPrevalentResourceForDebugMode(const String& domain) >-{ >- m_debugManualPrevalentResource = domain; >-} >- >-void ResourceLoadStatisticsMemoryStore::scheduleStatisticsProcessingRequestIfNecessary() >-{ >- ASSERT(!RunLoop::isMain()); >- >- m_pendingStatisticsProcessingRequestIdentifier = ++m_lastStatisticsProcessingRequestIdentifier; >- m_workQueue->dispatchAfter(minimumStatisticsProcessingInterval, [this, weakThis = makeWeakPtr(*this), statisticsProcessingRequestIdentifier = *m_pendingStatisticsProcessingRequestIdentifier] { >- if (!weakThis) >- return; >- >- if (!m_pendingStatisticsProcessingRequestIdentifier || *m_pendingStatisticsProcessingRequestIdentifier != statisticsProcessingRequestIdentifier) { >- // This request has been canceled. >- return; >- } >- >- updateCookieBlocking([]() { }); >- processStatisticsAndDataRecords(); >- }); >-} >- >-void ResourceLoadStatisticsMemoryStore::cancelPendingStatisticsProcessingRequest() >-{ >- ASSERT(!RunLoop::isMain()); >- >- m_pendingStatisticsProcessingRequestIdentifier = WTF::nullopt; >-} >- > void ResourceLoadStatisticsMemoryStore::logFrameNavigation(const String& targetPrimaryDomain, const String& mainFramePrimaryDomain, const String& sourcePrimaryDomain, const String& targetHost, const String& mainFrameHost, bool isRedirect, bool isMainFrame) > { > ASSERT(!RunLoop::isMain()); >@@ -840,95 +591,6 @@ void ResourceLoadStatisticsMemoryStore::setTopFrameUniqueRedirectFrom(const Stri > ensureResourceStatisticsForPrimaryDomain(primaryRedirectDomain); > } > >-void ResourceLoadStatisticsMemoryStore::setTimeToLiveUserInteraction(Seconds seconds) >-{ >- ASSERT(!RunLoop::isMain()); >- ASSERT(seconds >= 0_s); >- >- m_parameters.timeToLiveUserInteraction = seconds; >-} >- >-void ResourceLoadStatisticsMemoryStore::setMinimumTimeBetweenDataRecordsRemoval(Seconds seconds) >-{ >- ASSERT(!RunLoop::isMain()); >- ASSERT(seconds >= 0_s); >- >- m_parameters.minimumTimeBetweenDataRecordsRemoval = seconds; >-} >- >-void ResourceLoadStatisticsMemoryStore::setGrandfatheringTime(Seconds seconds) >-{ >- ASSERT(!RunLoop::isMain()); >- ASSERT(seconds >= 0_s); >- >- m_parameters.grandfatheringTime = seconds; >-} >- >-void ResourceLoadStatisticsMemoryStore::setCacheMaxAgeCap(Seconds seconds) >-{ >- ASSERT(!RunLoop::isMain()); >- ASSERT(seconds >= 0_s); >- >- m_parameters.cacheMaxAgeCapTime = seconds; >- updateCacheMaxAgeCap(); >-} >- >-void ResourceLoadStatisticsMemoryStore::updateCacheMaxAgeCap() >-{ >- ASSERT(!RunLoop::isMain()); >- >- RunLoop::main().dispatch([store = makeRef(m_store), seconds = m_parameters.cacheMaxAgeCapTime] () { >- store->setCacheMaxAgeCap(seconds, [] { }); >- }); >-} >- >-void ResourceLoadStatisticsMemoryStore::setAgeCapForClientSideCookies(Seconds seconds) >-{ >- ASSERT(!RunLoop::isMain()); >- ASSERT(seconds >= 0_s); >- >- m_parameters.clientSideCookiesAgeCapTime = seconds; >- updateClientSideCookiesAgeCap(); >-} >- >-void ResourceLoadStatisticsMemoryStore::updateClientSideCookiesAgeCap() >-{ >- ASSERT(!RunLoop::isMain()); >- >-#if ENABLE(RESOURCE_LOAD_STATISTICS) >- RunLoop::main().dispatch([store = makeRef(m_store), seconds = m_parameters.clientSideCookiesAgeCapTime] () { >- if (auto* networkSession = store->networkSession()) >- networkSession->networkStorageSession().setAgeCapForClientSideCookies(seconds); >- }); >-#endif >-} >- >-bool ResourceLoadStatisticsMemoryStore::shouldRemoveDataRecords() const >-{ >- ASSERT(!RunLoop::isMain()); >- >- if (m_dataRecordsBeingRemoved) >- return false; >- >-#if ENABLE(NETSCAPE_PLUGIN_API) >- for (const auto& plugin : PluginProcessManager::singleton().pluginProcesses()) { >- if (!m_activePluginTokens.contains(plugin->pluginProcessToken())) >- return true; >- } >-#endif >- >- return !m_lastTimeDataRecordsWereRemoved || MonotonicTime::now() >= (m_lastTimeDataRecordsWereRemoved + m_parameters.minimumTimeBetweenDataRecordsRemoval); >-} >- >-void ResourceLoadStatisticsMemoryStore::setDataRecordsBeingRemoved(bool value) >-{ >- ASSERT(!RunLoop::isMain()); >- >- m_dataRecordsBeingRemoved = value; >- if (m_dataRecordsBeingRemoved) >- m_lastTimeDataRecordsWereRemoved = MonotonicTime::now(); >-} >- > ResourceLoadStatistics& ResourceLoadStatisticsMemoryStore::ensureResourceStatisticsForPrimaryDomain(const String& primaryDomain) > { > ASSERT(!RunLoop::isMain()); >@@ -1059,39 +721,6 @@ bool ResourceLoadStatisticsMemoryStore::hasUserGrantedStorageAccessThroughPrompt > return statistic.storageAccessUnderTopFrameOrigins.contains(firstPartyPrimaryDomain); > } > >-static void debugLogDomainsInBatches(const char* action, const Vector<String>& domains) >-{ >-#if !RELEASE_LOG_DISABLED >- static const auto maxNumberOfDomainsInOneLogStatement = 50; >- if (domains.isEmpty()) >- return; >- >- if (domains.size() <= maxNumberOfDomainsInOneLogStatement) { >- RELEASE_LOG_INFO(ResourceLoadStatisticsDebug, "About to %{public}s cookies in third-party contexts for: %{public}s.", action, domainsToString(domains).utf8().data()); >- return; >- } >- >- Vector<String> batch; >- batch.reserveInitialCapacity(maxNumberOfDomainsInOneLogStatement); >- auto batchNumber = 1; >- unsigned numberOfBatches = std::ceil(domains.size() / static_cast<float>(maxNumberOfDomainsInOneLogStatement)); >- >- for (auto& domain : domains) { >- if (batch.size() == maxNumberOfDomainsInOneLogStatement) { >- RELEASE_LOG_INFO(ResourceLoadStatisticsDebug, "About to %{public}s cookies in third-party contexts for (%{public}d of %u): %{public}s.", action, batchNumber, numberOfBatches, domainsToString(batch).utf8().data()); >- batch.shrink(0); >- ++batchNumber; >- } >- batch.append(domain); >- } >- if (!batch.isEmpty()) >- RELEASE_LOG_INFO(ResourceLoadStatisticsDebug, "About to %{public}s cookies in third-party contexts for (%{public}d of %u): %{public}s.", action, batchNumber, numberOfBatches, domainsToString(batch).utf8().data()); >-#else >- UNUSED_PARAM(action); >- UNUSED_PARAM(domains); >-#endif >-} >- > void ResourceLoadStatisticsMemoryStore::updateCookieBlocking(CompletionHandler<void()>&& completionHandler) > { > ASSERT(!RunLoop::isMain()); >@@ -1199,52 +828,6 @@ Vector<String> ResourceLoadStatisticsMemoryStore::topPrivatelyControlledDomainsT > return prevalentResources; > } > >-void ResourceLoadStatisticsMemoryStore::includeTodayAsOperatingDateIfNecessary() >-{ >- ASSERT(!RunLoop::isMain()); >- >- auto today = OperatingDate::today(); >- if (!m_operatingDates.isEmpty() && today <= m_operatingDates.last()) >- return; >- >- while (m_operatingDates.size() >= operatingDatesWindow) >- m_operatingDates.remove(0); >- >- m_operatingDates.append(today); >-} >- >-bool ResourceLoadStatisticsMemoryStore::hasStatisticsExpired(const ResourceLoadStatistics& resourceStatistic) const >-{ >- ASSERT(!RunLoop::isMain()); >- >- if (m_operatingDates.size() >= operatingDatesWindow) { >- if (OperatingDate::fromWallTime(resourceStatistic.mostRecentUserInteractionTime) < m_operatingDates.first()) >- return true; >- } >- >- // If we don't meet the real criteria for an expired statistic, check the user setting for a tighter restriction (mainly for testing). >- if (m_parameters.timeToLiveUserInteraction) { >- if (WallTime::now() > resourceStatistic.mostRecentUserInteractionTime + m_parameters.timeToLiveUserInteraction.value()) >- return true; >- } >- >- return false; >-} >- >-void ResourceLoadStatisticsMemoryStore::setMaxStatisticsEntries(size_t maximumEntryCount) >-{ >- ASSERT(!RunLoop::isMain()); >- >- m_parameters.maxStatisticsEntries = maximumEntryCount; >-} >- >-void ResourceLoadStatisticsMemoryStore::setPruneEntriesDownTo(size_t pruneTargetCount) >-{ >- ASSERT(!RunLoop::isMain()); >- >- m_parameters.pruneEntriesDownTo = pruneTargetCount; >-} >- > void ResourceLoadStatisticsMemoryStore::pruneStatisticsIfNeeded() > { > ASSERT(!RunLoop::isMain()); >@@ -1267,22 +850,6 @@ void ResourceLoadStatisticsMemoryStore::pruneStatisticsIfNeeded() > ASSERT(!numberOfEntriesLeftToPrune); > } > >-void ResourceLoadStatisticsMemoryStore::resetParametersToDefaultValues() >-{ >- ASSERT(!RunLoop::isMain()); >- >- m_parameters = { }; >-} >- >-void ResourceLoadStatisticsMemoryStore::logTestingEvent(const String& event) >-{ >- ASSERT(!RunLoop::isMain()); >- >- RunLoop::main().dispatch([store = makeRef(m_store), event = event.isolatedCopy()] { >- store->logTestingEvent(event); >- }); >-} >- > void ResourceLoadStatisticsMemoryStore::setLastSeen(const String& primaryDomain, Seconds seconds) > { > ASSERT(!RunLoop::isMain()); >@@ -1319,15 +886,6 @@ void ResourceLoadStatisticsMemoryStore::removeAllStorageAccess(CompletionHandler > }); > } > >-void ResourceLoadStatisticsMemoryStore::didCreateNetworkProcess() >-{ >- ASSERT(!RunLoop::isMain()); >- >- updateCookieBlocking([]() { }); >- updateCacheMaxAgeCap(); >- updateClientSideCookiesAgeCap(); >-} >- > } // namespace WebKit > > #endif >diff --git a/Source/WebKit/NetworkProcess/Classifier/ResourceLoadStatisticsMemoryStore.h b/Source/WebKit/NetworkProcess/Classifier/ResourceLoadStatisticsMemoryStore.h >index 40e8197f27678a5b8388b4cc814e5ab2b44d3ba9..d45f70dd8a7b1586288112aa116f4f758d181d69 100644 >--- a/Source/WebKit/NetworkProcess/Classifier/ResourceLoadStatisticsMemoryStore.h >+++ b/Source/WebKit/NetworkProcess/Classifier/ResourceLoadStatisticsMemoryStore.h >@@ -27,17 +27,13 @@ > > #if ENABLE(RESOURCE_LOAD_STATISTICS) > >-#include "ResourceLoadStatisticsClassifier.h" >+#include "ResourceLoadStatisticsStore.h" > #include "WebResourceLoadStatisticsStore.h" > #include <wtf/CompletionHandler.h> > #include <wtf/Vector.h> > #include <wtf/WeakPtr.h> > #include <wtf/WorkQueue.h> > >-#if HAVE(CORE_PREDICTION) >-#include "ResourceLoadStatisticsClassifierCocoa.h" >-#endif >- > namespace WebCore { > class KeyedDecoder; > class KeyedEncoder; >@@ -46,164 +42,97 @@ struct ResourceLoadStatistics; > > namespace WebKit { > >-class OperatingDate; > class ResourceLoadStatisticsPersistentStorage; > > // This is always constructed / used / destroyed on the WebResourceLoadStatisticsStore's statistics queue. >-class ResourceLoadStatisticsMemoryStore : public CanMakeWeakPtr<ResourceLoadStatisticsMemoryStore> { >+class ResourceLoadStatisticsMemoryStore : public ResourceLoadStatisticsStore { > public: > ResourceLoadStatisticsMemoryStore(WebResourceLoadStatisticsStore&, WorkQueue&); >- ~ResourceLoadStatisticsMemoryStore(); > > void setPersistentStorage(ResourceLoadStatisticsPersistentStorage&); > >- void clear(CompletionHandler<void()>&&); >+ void clear(CompletionHandler<void()>&&) override; > bool isEmpty() const { return m_resourceStatisticsMap.isEmpty(); } > >+ const HashMap<String, WebCore::ResourceLoadStatistics>& data() const { return m_resourceStatisticsMap; } >+ > std::unique_ptr<WebCore::KeyedEncoder> createEncoderFromData() const; > void mergeWithDataFromDecoder(WebCore::KeyedDecoder&); > > void mergeStatistics(Vector<WebCore::ResourceLoadStatistics>&&); > void processStatistics(const Function<void(const WebCore::ResourceLoadStatistics&)>&) const; > >- void updateCookieBlocking(CompletionHandler<void()>&&); >+ void updateCookieBlocking(CompletionHandler<void()>&&) override; > void updateCookieBlockingForDomains(const Vector<String>& domainsToBlock, CompletionHandler<void()>&&); > void clearBlockingStateForDomains(const Vector<String>& domains, CompletionHandler<void()>&&); > >- void includeTodayAsOperatingDateIfNecessary(); >- void processStatisticsAndDataRecords(); >+ void classifyPrevalentResources() override; >+ void syncStorageIfNeeded() override; >+ void syncStorageImmediately() override; > >- void requestStorageAccessUnderOpener(String&& primaryDomainInNeedOfStorageAccess, uint64_t openerPageID, String&& openerPrimaryDomain); >+ void requestStorageAccessUnderOpener(String&& primaryDomainInNeedOfStorageAccess, uint64_t openerPageID, String&& openerPrimaryDomain) override; > void removeAllStorageAccess(CompletionHandler<void()>&&); > >- void grandfatherExistingWebsiteData(CompletionHandler<void()>&&); >- void cancelPendingStatisticsProcessingRequest(); >- >- bool isRegisteredAsSubresourceUnder(const String& subresourcePrimaryDomain, const String& topFramePrimaryDomain) const; >- bool isRegisteredAsSubFrameUnder(const String& subFramePrimaryDomain, const String& topFramePrimaryDomain) const; >- bool isRegisteredAsRedirectingTo(const String& hostRedirectedFromPrimaryDomain, const String& hostRedirectedToPrimaryDomain) const; >- >- void clearPrevalentResource(const String& primaryDomain); >- String dumpResourceLoadStatistics() const; >- bool isPrevalentResource(const String& primaryDomain) const; >- bool isVeryPrevalentResource(const String& primaryDomain) const; >- void setPrevalentResource(const String& primaryDomain); >- void setVeryPrevalentResource(const String& primaryDomain); >+ void grandfatherDataForDomains(const HashSet<String>&) override; > >- void setGrandfathered(const String& primaryDomain, bool value); >- bool isGrandfathered(const String& primaryDomain) const; >+ bool isRegisteredAsSubresourceUnder(const String& subresourcePrimaryDomain, const String& topFramePrimaryDomain) const override; >+ bool isRegisteredAsSubFrameUnder(const String& subFramePrimaryDomain, const String& topFramePrimaryDomain) const override; >+ bool isRegisteredAsRedirectingTo(const String& hostRedirectedFromPrimaryDomain, const String& hostRedirectedToPrimaryDomain) const override; > >- void setSubframeUnderTopFrameOrigin(const String& primarySubFrameDomain, const String& primaryTopFrameDomain); >- void setSubresourceUnderTopFrameOrigin(const String& primarySubresourceDomain, const String& primaryTopFrameDomain); >- void setSubresourceUniqueRedirectTo(const String& primarySubresourceDomain, const String& primaryRedirectDomain); >- void setSubresourceUniqueRedirectFrom(const String& primarySubresourceDomain, const String& primaryRedirectDomain); >- void setTopFrameUniqueRedirectTo(const String& topFramePrimaryDomain, const String& primaryRedirectDomain); >- void setTopFrameUniqueRedirectFrom(const String& topFramePrimaryDomain, const String& primaryRedirectDomain); >+ void clearPrevalentResource(const String& primaryDomain) override; >+ String dumpResourceLoadStatistics() const override; >+ bool isPrevalentResource(const String& primaryDomain) const override; >+ bool isVeryPrevalentResource(const String& primaryDomain) const override; >+ void setPrevalentResource(const String& primaryDomain) override; >+ void setVeryPrevalentResource(const String& primaryDomain) override; > >- void logTestingEvent(const String&); >+ void setGrandfathered(const String& primaryDomain, bool value) override; >+ bool isGrandfathered(const String& primaryDomain) const override; > >- void setMaxStatisticsEntries(size_t maximumEntryCount); >- void setPruneEntriesDownTo(size_t pruneTargetCount); >- void resetParametersToDefaultValues(); >+ void incrementRecordsDeletedCountForDomains(const HashSet<String>&) override; > >- void calculateAndSubmitTelemetry() const; >+ void setSubframeUnderTopFrameOrigin(const String& primarySubFrameDomain, const String& primaryTopFrameDomain) override; >+ void setSubresourceUnderTopFrameOrigin(const String& primarySubresourceDomain, const String& primaryTopFrameDomain) override; >+ void setSubresourceUniqueRedirectTo(const String& primarySubresourceDomain, const String& primaryRedirectDomain) override; >+ void setSubresourceUniqueRedirectFrom(const String& primarySubresourceDomain, const String& primaryRedirectDomain) override; >+ void setTopFrameUniqueRedirectTo(const String& topFramePrimaryDomain, const String& primaryRedirectDomain) override; >+ void setTopFrameUniqueRedirectFrom(const String& topFramePrimaryDomain, const String& primaryRedirectDomain) override; > >- void setNotifyPagesWhenDataRecordsWereScanned(bool); >- void setShouldClassifyResourcesBeforeDataRecordsRemoval(bool); >- void setShouldSubmitTelemetry(bool); >- void setTimeToLiveUserInteraction(Seconds); >- void setMinimumTimeBetweenDataRecordsRemoval(Seconds); >- void setGrandfatheringTime(Seconds); >- void setResourceLoadStatisticsDebugMode(bool); >- bool isDebugModeEnabled() const { return m_debugModeEnabled; }; >- void setPrevalentResourceForDebugMode(const String& domain); >+ void calculateAndSubmitTelemetry() const override; > >- void hasStorageAccess(const String& subFramePrimaryDomain, const String& topFramePrimaryDomain, Optional<uint64_t> frameID, uint64_t pageID, CompletionHandler<void(bool)>&&); >- void requestStorageAccess(String&& subFramePrimaryDomain, String&& topFramePrimaryDomain, uint64_t frameID, uint64_t pageID, bool promptEnabled, CompletionHandler<void(StorageAccessStatus)>&&); >- void grantStorageAccess(String&& subFrameHost, String&& topFrameHost, uint64_t frameID, uint64_t pageID, bool userWasPromptedNow, CompletionHandler<void(bool)>&&); >+ void hasStorageAccess(const String& subFramePrimaryDomain, const String& topFramePrimaryDomain, Optional<uint64_t> frameID, uint64_t pageID, CompletionHandler<void(bool)>&&) override; >+ void requestStorageAccess(String&& subFramePrimaryDomain, String&& topFramePrimaryDomain, uint64_t frameID, uint64_t pageID, bool promptEnabled, CompletionHandler<void(StorageAccessStatus)>&&) override; >+ void grantStorageAccess(String&& subFrameHost, String&& topFrameHost, uint64_t frameID, uint64_t pageID, bool userWasPromptedNow, CompletionHandler<void(bool)>&&) override; > >- void logFrameNavigation(const String& targetPrimaryDomain, const String& mainFramePrimaryDomain, const String& sourcePrimaryDomain, const String& targetHost, const String& mainFrameHost, bool isRedirect, bool isMainFrame); >- void logUserInteraction(const String& primaryDomain); >- void logSubresourceLoading(const String& targetPrimaryDomain, const String& mainFramePrimaryDomain, WallTime lastSeen); >- void logSubresourceRedirect(const String& sourcePrimaryDomain, const String& targetPrimaryDomain); >+ void logFrameNavigation(const String& targetPrimaryDomain, const String& mainFramePrimaryDomain, const String& sourcePrimaryDomain, const String& targetHost, const String& mainFrameHost, bool isRedirect, bool isMainFrame) override; >+ void logUserInteraction(const String& primaryDomain) override; >+ void logSubresourceLoading(const String& targetPrimaryDomain, const String& mainFramePrimaryDomain, WallTime lastSeen) override; >+ void logSubresourceRedirect(const String& sourcePrimaryDomain, const String& targetPrimaryDomain) override; > >- void clearUserInteraction(const String& primaryDomain); >- bool hasHadUserInteraction(const String& primaryDomain); >+ void clearUserInteraction(const String& primaryDomain) override; >+ bool hasHadUserInteraction(const String& primaryDomain) override; > >- void setLastSeen(const String& primaryDomain, Seconds); >- >- void didCreateNetworkProcess(); >- >- const WebResourceLoadStatisticsStore& store() const { return m_store; } >+ void setLastSeen(const String& primaryDomain, Seconds) override; > > private: > static bool shouldBlockAndKeepCookies(const WebCore::ResourceLoadStatistics&); > static bool shouldBlockAndPurgeCookies(const WebCore::ResourceLoadStatistics&); > static bool hasUserGrantedStorageAccessThroughPrompt(const WebCore::ResourceLoadStatistics&, const String& firstPartyPrimaryDomain); > bool hasHadUnexpiredRecentUserInteraction(WebCore::ResourceLoadStatistics&) const; >- bool hasStatisticsExpired(const WebCore::ResourceLoadStatistics&) const; > bool wasAccessedAsFirstPartyDueToUserInteraction(const WebCore::ResourceLoadStatistics& current, const WebCore::ResourceLoadStatistics& updated) const; > void setPrevalentResource(WebCore::ResourceLoadStatistics&, ResourceLoadPrevalence); > unsigned recursivelyGetAllDomainsThatHaveRedirectedToThisDomain(const WebCore::ResourceLoadStatistics&, HashSet<String>& domainsThatHaveRedirectedTo, unsigned numberOfRecursiveCalls) const; >- void setStorageAccessPromptsEnabled(bool enabled) { m_storageAccessPromptsEnabled = enabled; } >- bool shouldRemoveDataRecords() const; >- void setDebugLogggingEnabled(bool enabled) { m_debugLoggingEnabled = enabled; } >- void setDataRecordsBeingRemoved(bool); >- void scheduleStatisticsProcessingRequestIfNecessary(); > void grantStorageAccessInternal(String&& subFrameHost, String&& topFrameHost, Optional<uint64_t> frameID, uint64_t pageID, bool userWasPromptedNowOrEarlier, CompletionHandler<void(bool)>&&); > void markAsPrevalentIfHasRedirectedToPrevalent(WebCore::ResourceLoadStatistics&); > bool isPrevalentDueToDebugMode(WebCore::ResourceLoadStatistics&); >- Vector<String> ensurePrevalentResourcesForDebugMode(); >+ Vector<String> ensurePrevalentResourcesForDebugMode() override; > void removeDataRecords(CompletionHandler<void()>&&); >- void pruneStatisticsIfNeeded(); >+ void pruneStatisticsIfNeeded() override; > WebCore::ResourceLoadStatistics& ensureResourceStatisticsForPrimaryDomain(const String&); >- Vector<String> topPrivatelyControlledDomainsToRemoveWebsiteDataFor(); >- void setCacheMaxAgeCap(Seconds); >- void updateCacheMaxAgeCap(); >- void setAgeCapForClientSideCookies(Seconds); >- void updateClientSideCookiesAgeCap(); >- >-#if PLATFORM(COCOA) >- void registerUserDefaultsIfNeeded(); >-#endif >+ Vector<String> topPrivatelyControlledDomainsToRemoveWebsiteDataFor() override; > >- struct Parameters { >- size_t pruneEntriesDownTo { 800 }; >- size_t maxStatisticsEntries { 1000 }; >- Optional<Seconds> timeToLiveUserInteraction; >- Seconds minimumTimeBetweenDataRecordsRemoval { 1_h }; >- Seconds grandfatheringTime { 24_h * 7 }; >- Seconds cacheMaxAgeCapTime { 24_h * 7 }; >- Seconds clientSideCookiesAgeCapTime { 24_h * 7 }; >- bool shouldNotifyPagesWhenDataRecordsWereScanned { false }; >- bool shouldClassifyResourcesBeforeDataRecordsRemoval { true }; >- bool shouldSubmitTelemetry { true }; >- }; >- >- WebResourceLoadStatisticsStore& m_store; >- Ref<WorkQueue> m_workQueue; > WeakPtr<ResourceLoadStatisticsPersistentStorage> m_persistentStorage; > HashMap<String, WebCore::ResourceLoadStatistics> m_resourceStatisticsMap; >-#if HAVE(CORE_PREDICTION) >- ResourceLoadStatisticsClassifierCocoa m_resourceLoadStatisticsClassifier; >-#else >- ResourceLoadStatisticsClassifier m_resourceLoadStatisticsClassifier; >-#endif >-#if ENABLE(NETSCAPE_PLUGIN_API) >- HashSet<uint64_t> m_activePluginTokens; >-#endif >- Parameters m_parameters; >- Vector<OperatingDate> m_operatingDates; >- WallTime m_endOfGrandfatheringTimestamp; >- bool m_debugLoggingEnabled { false }; >- bool m_debugModeEnabled { false }; >- String m_debugManualPrevalentResource; >- bool m_storageAccessPromptsEnabled { false }; >- bool m_dataRecordsBeingRemoved { false }; >- MonotonicTime m_lastTimeDataRecordsWereRemoved; >- >- uint64_t m_lastStatisticsProcessingRequestIdentifier { 0 }; >- Optional<uint64_t> m_pendingStatisticsProcessingRequestIdentifier; > }; > > } // namespace WebKit >diff --git a/Source/WebKit/NetworkProcess/Classifier/ResourceLoadStatisticsStore.cpp b/Source/WebKit/NetworkProcess/Classifier/ResourceLoadStatisticsStore.cpp >new file mode 100644 >index 0000000000000000000000000000000000000000..42b59eb68d00e2a971a74f593a3ee3faa5af2414 >--- /dev/null >+++ b/Source/WebKit/NetworkProcess/Classifier/ResourceLoadStatisticsStore.cpp >@@ -0,0 +1,575 @@ >+/* >+ * Copyright (C) 2019 Apple Inc. All rights reserved. >+ * >+ * Redistribution and use in source and binary forms, with or without >+ * modification, are permitted provided that the following conditions >+ * are met: >+ * 1. Redistributions of source code must retain the above copyright >+ * notice, this list of conditions and the following disclaimer. >+ * 2. Redistributions in binary form must reproduce the above copyright >+ * notice, this list of conditions and the following disclaimer in the >+ * documentation and/or other materials provided with the distribution. >+ * >+ * THIS SOFTWARE IS PROVIDED BY APPLE INC. AND ITS CONTRIBUTORS ``AS IS'' >+ * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, >+ * THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR >+ * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL APPLE INC. OR ITS CONTRIBUTORS >+ * BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR >+ * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF >+ * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS >+ * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN >+ * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) >+ * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF >+ * THE POSSIBILITY OF SUCH DAMAGE. >+ */ >+ >+#include "config.h" >+#include "ResourceLoadStatisticsStore.h" >+ >+#if ENABLE(RESOURCE_LOAD_STATISTICS) >+ >+#include "Logging.h" >+#include "NetworkSession.h" >+#include "PluginProcessManager.h" >+#include "PluginProcessProxy.h" >+#include "ResourceLoadStatisticsPersistentStorage.h" >+#include "StorageAccessStatus.h" >+#include "WebProcessProxy.h" >+#include "WebResourceLoadStatisticsTelemetry.h" >+#include "WebsiteDataStore.h" >+#include <WebCore/KeyedCoding.h> >+#include <WebCore/NetworkStorageSession.h> >+#include <WebCore/ResourceLoadStatistics.h> >+#include <wtf/CallbackAggregator.h> >+#include <wtf/DateMath.h> >+#include <wtf/MathExtras.h> >+#include <wtf/text/StringBuilder.h> >+ >+namespace WebKit { >+using namespace WebCore; >+ >+constexpr Seconds minimumStatisticsProcessingInterval { 5_s }; >+constexpr unsigned operatingDatesWindow { 30 }; >+ >+#if !RELEASE_LOG_DISABLED >+static String domainsToString(const Vector<String>& domains) >+{ >+ StringBuilder builder; >+ for (auto& domain : domains) { >+ if (!builder.isEmpty()) >+ builder.appendLiteral(", "); >+ builder.append(domain); >+ } >+ return builder.toString(); >+} >+#endif >+ >+OperatingDate OperatingDate::fromWallTime(WallTime time) >+{ >+ double ms = time.secondsSinceEpoch().milliseconds(); >+ int year = msToYear(ms); >+ int yearDay = dayInYear(ms, year); >+ int month = monthFromDayInYear(yearDay, isLeapYear(year)); >+ int monthDay = dayInMonthFromDayInYear(yearDay, isLeapYear(year)); >+ >+ return OperatingDate { year, month, monthDay }; >+} >+ >+OperatingDate OperatingDate::today() >+{ >+ return OperatingDate::fromWallTime(WallTime::now()); >+} >+ >+Seconds OperatingDate::secondsSinceEpoch() const >+{ >+ return Seconds { dateToDaysFrom1970(m_year, m_month, m_monthDay) * secondsPerDay }; >+} >+ >+bool OperatingDate::operator==(const OperatingDate& other) const >+{ >+ return m_monthDay == other.m_monthDay && m_month == other.m_month && m_year == other.m_year; >+} >+ >+bool OperatingDate::operator<(const OperatingDate& other) const >+{ >+ return secondsSinceEpoch() < other.secondsSinceEpoch(); >+} >+ >+bool OperatingDate::operator<=(const OperatingDate& other) const >+{ >+ return secondsSinceEpoch() <= other.secondsSinceEpoch(); >+} >+ >+const char* ResourceLoadStatisticsStore::debugStaticPrevalentResource = "3rdpartytestwebkit.org"; >+ >+ResourceLoadStatisticsStore::ResourceLoadStatisticsStore(WebResourceLoadStatisticsStore& store, WorkQueue& workQueue) >+ : m_store(store) >+ , m_workQueue(workQueue) >+{ >+ ASSERT(!RunLoop::isMain()); >+ >+ includeTodayAsOperatingDateIfNecessary(); >+} >+ >+ResourceLoadStatisticsStore::~ResourceLoadStatisticsStore() >+{ >+ ASSERT(!RunLoop::isMain()); >+} >+ >+unsigned ResourceLoadStatisticsStore::computeImportance(const ResourceLoadStatistics& resourceStatistic) >+{ >+ unsigned importance = ResourceLoadStatisticsStore::maxImportance; >+ if (!resourceStatistic.isPrevalentResource) >+ importance -= 1; >+ if (!resourceStatistic.hadUserInteraction) >+ importance -= 2; >+ return importance; >+} >+ >+void ResourceLoadStatisticsStore::setNotifyPagesWhenDataRecordsWereScanned(bool value) >+{ >+ ASSERT(!RunLoop::isMain()); >+ m_parameters.shouldNotifyPagesWhenDataRecordsWereScanned = value; >+} >+ >+void ResourceLoadStatisticsStore::setShouldClassifyResourcesBeforeDataRecordsRemoval(bool value) >+{ >+ ASSERT(!RunLoop::isMain()); >+ m_parameters.shouldClassifyResourcesBeforeDataRecordsRemoval = value; >+} >+ >+void ResourceLoadStatisticsStore::setShouldSubmitTelemetry(bool value) >+{ >+ ASSERT(!RunLoop::isMain()); >+ m_parameters.shouldSubmitTelemetry = value; >+} >+ >+void ResourceLoadStatisticsStore::removeDataRecords(CompletionHandler<void()>&& callback) >+{ >+ ASSERT(!RunLoop::isMain()); >+ >+ if (!shouldRemoveDataRecords()) { >+ callback(); >+ return; >+ } >+ >+#if ENABLE(NETSCAPE_PLUGIN_API) >+ m_activePluginTokens.clear(); >+ for (const auto& plugin : PluginProcessManager::singleton().pluginProcesses()) >+ m_activePluginTokens.add(plugin->pluginProcessToken()); >+#endif >+ >+ auto prevalentResourceDomains = topPrivatelyControlledDomainsToRemoveWebsiteDataFor(); >+ if (prevalentResourceDomains.isEmpty()) { >+ callback(); >+ return; >+ } >+ >+#if !RELEASE_LOG_DISABLED >+ RELEASE_LOG_INFO_IF(m_debugLoggingEnabled, ResourceLoadStatisticsDebug, "About to remove data records for %{public}s.", domainsToString(prevalentResourceDomains).utf8().data()); >+#endif >+ >+ setDataRecordsBeingRemoved(true); >+ >+ RunLoop::main().dispatch([prevalentResourceDomains = crossThreadCopy(prevalentResourceDomains), callback = WTFMove(callback), weakThis = makeWeakPtr(*this), shouldNotifyPagesWhenDataRecordsWereScanned = m_parameters.shouldNotifyPagesWhenDataRecordsWereScanned, workQueue = m_workQueue.copyRef()] () mutable { >+ if (!weakThis) { >+ callback(); >+ return; >+ } >+ >+ weakThis->m_store.deleteWebsiteDataForTopPrivatelyControlledDomainsInAllPersistentDataStores(WebResourceLoadStatisticsStore::monitoredDataTypes(), WTFMove(prevalentResourceDomains), shouldNotifyPagesWhenDataRecordsWereScanned, [callback = WTFMove(callback), weakThis = WTFMove(weakThis), workQueue = workQueue.copyRef()](const HashSet<String>& domainsWithDeletedWebsiteData) mutable { >+ workQueue->dispatch([topDomains = crossThreadCopy(domainsWithDeletedWebsiteData), callback = WTFMove(callback), weakThis = WTFMove(weakThis)] () mutable { >+ if (!weakThis) { >+ callback(); >+ return; >+ } >+ weakThis->incrementRecordsDeletedCountForDomains(WTFMove(topDomains)); >+ weakThis->setDataRecordsBeingRemoved(false); >+ callback(); >+#if !RELEASE_LOG_DISABLED >+ RELEASE_LOG_INFO_IF(weakThis->m_debugLoggingEnabled, ResourceLoadStatisticsDebug, "Done removing data records."); >+#endif >+ }); >+ }); >+ }); >+} >+ >+void ResourceLoadStatisticsStore::processStatisticsAndDataRecords() >+{ >+ ASSERT(!RunLoop::isMain()); >+ >+ if (m_parameters.shouldClassifyResourcesBeforeDataRecordsRemoval) >+ classifyPrevalentResources(); >+ >+ removeDataRecords([this, weakThis = makeWeakPtr(*this)] { >+ ASSERT(!RunLoop::isMain()); >+ if (!weakThis) >+ return; >+ >+ pruneStatisticsIfNeeded(); >+ syncStorageIfNeeded(); >+ >+ if (!m_parameters.shouldNotifyPagesWhenDataRecordsWereScanned) >+ return; >+ >+ RunLoop::main().dispatch([this, weakThis = makeWeakPtr(*this)] { >+ ASSERT(RunLoop::isMain()); >+ if (!weakThis) >+ return; >+ >+ m_store.notifyResourceLoadStatisticsProcessed(); >+ }); >+ }); >+} >+ >+void ResourceLoadStatisticsStore::grandfatherExistingWebsiteData(CompletionHandler<void()>&& callback) >+{ >+ ASSERT(!RunLoop::isMain()); >+ >+ RunLoop::main().dispatch([weakThis = makeWeakPtr(*this), callback = WTFMove(callback), shouldNotifyPagesWhenDataRecordsWereScanned = m_parameters.shouldNotifyPagesWhenDataRecordsWereScanned, workQueue = m_workQueue.copyRef(), store = makeRef(m_store)] () mutable { >+ store->topPrivatelyControlledDomainsWithWebsiteData(WebResourceLoadStatisticsStore::monitoredDataTypes(), shouldNotifyPagesWhenDataRecordsWereScanned, [weakThis = WTFMove(weakThis), callback = WTFMove(callback), workQueue = workQueue.copyRef()] (HashSet<String>&& topDomainsWithWebsiteData) mutable { >+ workQueue->dispatch([weakThis = WTFMove(weakThis), topDomains = crossThreadCopy(topDomainsWithWebsiteData), callback = WTFMove(callback)] () mutable { >+ if (!weakThis) { >+ callback(); >+ return; >+ } >+ >+ weakThis->grandfatherDataForDomains(topDomains); >+ weakThis->m_endOfGrandfatheringTimestamp = WallTime::now() + weakThis->m_parameters.grandfatheringTime; >+ weakThis->syncStorageImmediately(); >+ callback(); >+ weakThis->logTestingEvent("Grandfathered"_s); >+ }); >+ }); >+ }); >+} >+ >+void ResourceLoadStatisticsStore::setResourceLoadStatisticsDebugMode(bool enable) >+{ >+ ASSERT(!RunLoop::isMain()); >+ >+#if !RELEASE_LOG_DISABLED >+ if (enable) >+ RELEASE_LOG_INFO(ResourceLoadStatisticsDebug, "Turned ITP Debug Mode on."); >+#endif >+ >+ m_debugModeEnabled = enable; >+ m_debugLoggingEnabled = enable; >+ >+ ensurePrevalentResourcesForDebugMode(); >+ // This will log the current cookie blocking state. >+ if (enable) >+ updateCookieBlocking([]() { }); >+} >+ >+void ResourceLoadStatisticsStore::setPrevalentResourceForDebugMode(const String& domain) >+{ >+ m_debugManualPrevalentResource = domain; >+} >+ >+void ResourceLoadStatisticsStore::scheduleStatisticsProcessingRequestIfNecessary() >+{ >+ ASSERT(!RunLoop::isMain()); >+ >+ m_pendingStatisticsProcessingRequestIdentifier = ++m_lastStatisticsProcessingRequestIdentifier; >+ m_workQueue->dispatchAfter(minimumStatisticsProcessingInterval, [this, weakThis = makeWeakPtr(*this), statisticsProcessingRequestIdentifier = *m_pendingStatisticsProcessingRequestIdentifier] { >+ if (!weakThis) >+ return; >+ >+ if (!m_pendingStatisticsProcessingRequestIdentifier || *m_pendingStatisticsProcessingRequestIdentifier != statisticsProcessingRequestIdentifier) { >+ // This request has been canceled. >+ return; >+ } >+ >+ updateCookieBlocking([]() { }); >+ processStatisticsAndDataRecords(); >+ }); >+} >+ >+void ResourceLoadStatisticsStore::cancelPendingStatisticsProcessingRequest() >+{ >+ ASSERT(!RunLoop::isMain()); >+ >+ m_pendingStatisticsProcessingRequestIdentifier = WTF::nullopt; >+} >+ >+void ResourceLoadStatisticsStore::setTimeToLiveUserInteraction(Seconds seconds) >+{ >+ ASSERT(!RunLoop::isMain()); >+ ASSERT(seconds >= 0_s); >+ >+ m_parameters.timeToLiveUserInteraction = seconds; >+} >+ >+void ResourceLoadStatisticsStore::setMinimumTimeBetweenDataRecordsRemoval(Seconds seconds) >+{ >+ ASSERT(!RunLoop::isMain()); >+ ASSERT(seconds >= 0_s); >+ >+ m_parameters.minimumTimeBetweenDataRecordsRemoval = seconds; >+} >+ >+void ResourceLoadStatisticsStore::setGrandfatheringTime(Seconds seconds) >+{ >+ ASSERT(!RunLoop::isMain()); >+ ASSERT(seconds >= 0_s); >+ >+ m_parameters.grandfatheringTime = seconds; >+} >+ >+void ResourceLoadStatisticsStore::setCacheMaxAgeCap(Seconds seconds) >+{ >+ ASSERT(!RunLoop::isMain()); >+ ASSERT(seconds >= 0_s); >+ >+ m_parameters.cacheMaxAgeCapTime = seconds; >+ updateCacheMaxAgeCap(); >+} >+ >+void ResourceLoadStatisticsStore::updateCacheMaxAgeCap() >+{ >+ ASSERT(!RunLoop::isMain()); >+ >+ RunLoop::main().dispatch([store = makeRef(m_store), seconds = m_parameters.cacheMaxAgeCapTime] () { >+ store->setCacheMaxAgeCap(seconds, [] { }); >+ }); >+} >+ >+void ResourceLoadStatisticsStore::setAgeCapForClientSideCookies(Seconds seconds) >+{ >+ ASSERT(!RunLoop::isMain()); >+ ASSERT(seconds >= 0_s); >+ >+ m_parameters.clientSideCookiesAgeCapTime = seconds; >+ updateClientSideCookiesAgeCap(); >+} >+ >+void ResourceLoadStatisticsStore::updateClientSideCookiesAgeCap() >+{ >+ ASSERT(!RunLoop::isMain()); >+ >+#if ENABLE(RESOURCE_LOAD_STATISTICS) >+ RunLoop::main().dispatch([store = makeRef(m_store), seconds = m_parameters.clientSideCookiesAgeCapTime] () { >+ if (auto* networkSession = store->networkSession()) >+ networkSession->networkStorageSession().setAgeCapForClientSideCookies(seconds); >+ }); >+#endif >+} >+ >+bool ResourceLoadStatisticsStore::shouldRemoveDataRecords() const >+{ >+ ASSERT(!RunLoop::isMain()); >+ >+ if (m_dataRecordsBeingRemoved) >+ return false; >+ >+#if ENABLE(NETSCAPE_PLUGIN_API) >+ for (const auto& plugin : PluginProcessManager::singleton().pluginProcesses()) { >+ if (!m_activePluginTokens.contains(plugin->pluginProcessToken())) >+ return true; >+ } >+#endif >+ >+ return !m_lastTimeDataRecordsWereRemoved || MonotonicTime::now() >= (m_lastTimeDataRecordsWereRemoved + m_parameters.minimumTimeBetweenDataRecordsRemoval); >+} >+ >+void ResourceLoadStatisticsStore::setDataRecordsBeingRemoved(bool value) >+{ >+ ASSERT(!RunLoop::isMain()); >+ >+ m_dataRecordsBeingRemoved = value; >+ if (m_dataRecordsBeingRemoved) >+ m_lastTimeDataRecordsWereRemoved = MonotonicTime::now(); >+} >+ >+void ResourceLoadStatisticsStore::updateCookieBlockingForDomains(const Vector<String>& domainsToBlock, CompletionHandler<void()>&& completionHandler) >+{ >+ ASSERT(!RunLoop::isMain()); >+ >+ RunLoop::main().dispatch([store = makeRef(m_store), domainsToBlock = crossThreadCopy(domainsToBlock), completionHandler = WTFMove(completionHandler)] () mutable { >+ store->callUpdatePrevalentDomainsToBlockCookiesForHandler(domainsToBlock, [store = store.copyRef(), completionHandler = WTFMove(completionHandler)]() mutable { >+ store->statisticsQueue().dispatch([completionHandler = WTFMove(completionHandler)]() mutable { >+ completionHandler(); >+ }); >+ }); >+ }); >+} >+ >+void ResourceLoadStatisticsStore::clearBlockingStateForDomains(const Vector<String>& domains, CompletionHandler<void()>&& completionHandler) >+{ >+ ASSERT(!RunLoop::isMain()); >+ >+ if (domains.isEmpty()) { >+ completionHandler(); >+ return; >+ } >+ >+ RunLoop::main().dispatch([store = makeRef(m_store), domains = crossThreadCopy(domains)] { >+ store->callRemoveDomainsHandler(domains); >+ }); >+ >+ completionHandler(); >+} >+ >+void ResourceLoadStatisticsStore::processStatistics(const Function<void(const ResourceLoadStatistics&)>& processFunction) const >+{ >+ ASSERT(!RunLoop::isMain()); >+ >+ for (auto& resourceStatistic : m_resourceStatisticsMap.values()) >+ processFunction(resourceStatistic); >+} >+ >+Optional<Seconds> ResourceLoadStatisticsStore::statisticsEpirationTime() const >+{ >+ if (m_parameters.timeToLiveUserInteraction) >+ return WallTime::now().secondsSinceEpoch() - m_parameters.timeToLiveUserInteraction.value(); >+ >+ if (m_operatingDates.size() >= operatingDatesWindow) >+ return m_operatingDates.first().secondsSinceEpoch(); >+ >+ return WTF::nullopt; >+} >+ >+Vector<OperatingDate> ResourceLoadStatisticsStore::mergeOperatingDates(const Vector<OperatingDate>& existingDates, Vector<OperatingDate>&& newDates) >+{ >+ if (existingDates.isEmpty()) >+ return WTFMove(newDates); >+ >+ Vector<OperatingDate> mergedDates(existingDates.size() + newDates.size()); >+ >+ // Merge the two sorted vectors of dates. >+ std::merge(existingDates.begin(), existingDates.end(), newDates.begin(), newDates.end(), mergedDates.begin()); >+ // Remove duplicate dates. >+ removeRepeatedElements(mergedDates); >+ >+ // Drop old dates until the Vector size reaches operatingDatesWindow. >+ while (mergedDates.size() > operatingDatesWindow) >+ mergedDates.remove(0); >+ >+ return mergedDates; >+} >+ >+void ResourceLoadStatisticsStore::includeTodayAsOperatingDateIfNecessary() >+{ >+ ASSERT(!RunLoop::isMain()); >+ >+ auto today = OperatingDate::today(); >+ if (!m_operatingDates.isEmpty() && today <= m_operatingDates.last()) >+ return; >+ >+ while (m_operatingDates.size() >= operatingDatesWindow) >+ m_operatingDates.remove(0); >+ >+ m_operatingDates.append(today); >+} >+ >+bool ResourceLoadStatisticsStore::hasStatisticsExpired(WallTime mostRecentUserInteractionTime) const >+{ >+ ASSERT(!RunLoop::isMain()); >+ >+ if (m_operatingDates.size() >= operatingDatesWindow) { >+ if (OperatingDate::fromWallTime(mostRecentUserInteractionTime) < m_operatingDates.first()) >+ return true; >+ } >+ >+ // If we don't meet the real criteria for an expired statistic, check the user setting for a tighter restriction (mainly for testing). >+ if (m_parameters.timeToLiveUserInteraction) { >+ if (WallTime::now() > mostRecentUserInteractionTime + m_parameters.timeToLiveUserInteraction.value()) >+ return true; >+ } >+ >+ return false; >+} >+ >+bool ResourceLoadStatisticsStore::hasStatisticsExpired(const ResourceLoadStatistics& resourceStatistic) const >+{ >+ return hasStatisticsExpired(resourceStatistic.mostRecentUserInteractionTime); >+} >+ >+void ResourceLoadStatisticsStore::setMaxStatisticsEntries(size_t maximumEntryCount) >+{ >+ ASSERT(!RunLoop::isMain()); >+ >+ m_parameters.maxStatisticsEntries = maximumEntryCount; >+} >+ >+void ResourceLoadStatisticsStore::setPruneEntriesDownTo(size_t pruneTargetCount) >+{ >+ ASSERT(!RunLoop::isMain()); >+ >+ m_parameters.pruneEntriesDownTo = pruneTargetCount; >+} >+ >+void ResourceLoadStatisticsStore::resetParametersToDefaultValues() >+{ >+ ASSERT(!RunLoop::isMain()); >+ >+ m_parameters = { }; >+} >+ >+void ResourceLoadStatisticsStore::logTestingEvent(const String& event) >+{ >+ ASSERT(!RunLoop::isMain()); >+ >+ RunLoop::main().dispatch([store = makeRef(m_store), event = event.isolatedCopy()] { >+ store->logTestingEvent(event); >+ }); >+} >+ >+void ResourceLoadStatisticsStore::removeAllStorageAccess(CompletionHandler<void()>&& completionHandler) >+{ >+ ASSERT(!RunLoop::isMain()); >+ RunLoop::main().dispatch([store = makeRef(m_store), completionHandler = WTFMove(completionHandler)]() mutable { >+ store->removeAllStorageAccess([store = store.copyRef(), completionHandler = WTFMove(completionHandler)]() mutable { >+ store->statisticsQueue().dispatch([completionHandler = WTFMove(completionHandler)]() mutable { >+ completionHandler(); >+ }); >+ }); >+ }); >+} >+ >+void ResourceLoadStatisticsStore::didCreateNetworkProcess() >+{ >+ ASSERT(!RunLoop::isMain()); >+ >+ updateCookieBlocking([]() { }); >+ updateCacheMaxAgeCap(); >+ updateClientSideCookiesAgeCap(); >+} >+ >+void ResourceLoadStatisticsStore::debugLogDomainsInBatches(const char* action, const Vector<String>& domains) >+{ >+#if !RELEASE_LOG_DISABLED >+ static const auto maxNumberOfDomainsInOneLogStatement = 50; >+ if (domains.isEmpty()) >+ return; >+ >+ if (domains.size() <= maxNumberOfDomainsInOneLogStatement) { >+ RELEASE_LOG_INFO(ResourceLoadStatisticsDebug, "About to %{public}s cookies in third-party contexts for: %{public}s.", action, domainsToString(domains).utf8().data()); >+ return; >+ } >+ >+ Vector<String> batch; >+ batch.reserveInitialCapacity(maxNumberOfDomainsInOneLogStatement); >+ auto batchNumber = 1; >+ unsigned numberOfBatches = std::ceil(domains.size() / static_cast<float>(maxNumberOfDomainsInOneLogStatement)); >+ >+ for (auto& domain : domains) { >+ if (batch.size() == maxNumberOfDomainsInOneLogStatement) { >+ RELEASE_LOG_INFO(ResourceLoadStatisticsDebug, "About to %{public}s cookies in third-party contexts for (%{public}d of %u): %{public}s.", action, batchNumber, numberOfBatches, domainsToString(batch).utf8().data()); >+ batch.shrink(0); >+ ++batchNumber; >+ } >+ batch.append(domain); >+ } >+ if (!batch.isEmpty()) >+ RELEASE_LOG_INFO(ResourceLoadStatisticsDebug, "About to %{public}s cookies in third-party contexts for (%{public}d of %u): %{public}s.", action, batchNumber, numberOfBatches, domainsToString(batch).utf8().data()); >+#else >+ UNUSED_PARAM(action); >+ UNUSED_PARAM(domains); >+#endif >+} >+ >+} // namespace WebKit >+ >+#endif >diff --git a/Source/WebKit/NetworkProcess/Classifier/ResourceLoadStatisticsStore.h b/Source/WebKit/NetworkProcess/Classifier/ResourceLoadStatisticsStore.h >new file mode 100644 >index 0000000000000000000000000000000000000000..5b850517e6253a8f99547cf85a3741f81668fb00 >--- /dev/null >+++ b/Source/WebKit/NetworkProcess/Classifier/ResourceLoadStatisticsStore.h >@@ -0,0 +1,236 @@ >+/* >+ * Copyright (C) 2019 Apple Inc. All rights reserved. >+ * >+ * Redistribution and use in source and binary forms, with or without >+ * modification, are permitted provided that the following conditions >+ * are met: >+ * 1. Redistributions of source code must retain the above copyright >+ * notice, this list of conditions and the following disclaimer. >+ * 2. Redistributions in binary form must reproduce the above copyright >+ * notice, this list of conditions and the following disclaimer in the >+ * documentation and/or other materials provided with the distribution. >+ * >+ * THIS SOFTWARE IS PROVIDED BY APPLE INC. AND ITS CONTRIBUTORS ``AS IS'' >+ * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, >+ * THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR >+ * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL APPLE INC. OR ITS CONTRIBUTORS >+ * BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR >+ * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF >+ * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS >+ * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN >+ * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) >+ * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF >+ * THE POSSIBILITY OF SUCH DAMAGE. >+ */ >+ >+#pragma once >+ >+#if ENABLE(RESOURCE_LOAD_STATISTICS) >+ >+#include "ResourceLoadStatisticsClassifier.h" >+#include "WebResourceLoadStatisticsStore.h" >+#include <wtf/CompletionHandler.h> >+#include <wtf/Vector.h> >+#include <wtf/WeakPtr.h> >+#include <wtf/WorkQueue.h> >+ >+#if HAVE(CORE_PREDICTION) >+#include "ResourceLoadStatisticsClassifierCocoa.h" >+#endif >+ >+namespace WebCore { >+class KeyedDecoder; >+class KeyedEncoder; >+struct ResourceLoadStatistics; >+} >+ >+namespace WebKit { >+ >+class ResourceLoadStatisticsPersistentStorage; >+ >+class OperatingDate { >+public: >+ OperatingDate() = default; >+ >+ static OperatingDate fromWallTime(WallTime); >+ static OperatingDate today(); >+ Seconds secondsSinceEpoch() const; >+ bool operator==(const OperatingDate& other) const; >+ bool operator<(const OperatingDate& other) const; >+ bool operator<=(const OperatingDate& other) const; >+ >+private: >+ OperatingDate(int year, int month, int monthDay) >+ : m_year(year) >+ , m_month(month) >+ , m_monthDay(monthDay) >+ { } >+ >+ int m_year { 0 }; >+ int m_month { 0 }; // [0, 11]. >+ int m_monthDay { 0 }; // [1, 31]. >+}; >+ >+// This is always constructed / used / destroyed on the WebResourceLoadStatisticsStore's statistics queue. >+class ResourceLoadStatisticsStore : public CanMakeWeakPtr<ResourceLoadStatisticsStore> { >+public: >+ ResourceLoadStatisticsStore(WebResourceLoadStatisticsStore&, WorkQueue&); >+ virtual ~ResourceLoadStatisticsStore(); >+ >+ virtual void clear(CompletionHandler<void()>&&) = 0; >+ bool isEmpty() const { return m_resourceStatisticsMap.isEmpty(); } >+ >+ const HashMap<String, WebCore::ResourceLoadStatistics>& data() const { return m_resourceStatisticsMap; } >+ >+ void processStatistics(const Function<void(const WebCore::ResourceLoadStatistics&)>&) const; >+ >+ virtual void updateCookieBlocking(CompletionHandler<void()>&&) = 0; >+ void updateCookieBlockingForDomains(const Vector<String>& domainsToBlock, CompletionHandler<void()>&&); >+ void clearBlockingStateForDomains(const Vector<String>& domains, CompletionHandler<void()>&&); >+ >+ void includeTodayAsOperatingDateIfNecessary(); >+ void processStatisticsAndDataRecords(); >+ >+ virtual void classifyPrevalentResources() = 0; >+ virtual void syncStorageIfNeeded() = 0; >+ virtual void syncStorageImmediately() = 0; >+ >+ virtual void requestStorageAccessUnderOpener(String&& primaryDomainInNeedOfStorageAccess, uint64_t openerPageID, String&& openerPrimaryDomain) = 0; >+ void removeAllStorageAccess(CompletionHandler<void()>&&); >+ >+ void grandfatherExistingWebsiteData(CompletionHandler<void()>&&); >+ void cancelPendingStatisticsProcessingRequest(); >+ >+ virtual bool isRegisteredAsSubresourceUnder(const String& subresourcePrimaryDomain, const String& topFramePrimaryDomain) const = 0; >+ virtual bool isRegisteredAsSubFrameUnder(const String& subFramePrimaryDomain, const String& topFramePrimaryDomain) const = 0; >+ virtual bool isRegisteredAsRedirectingTo(const String& hostRedirectedFromPrimaryDomain, const String& hostRedirectedToPrimaryDomain) const = 0; >+ >+ virtual void clearPrevalentResource(const String& primaryDomain) = 0; >+ virtual String dumpResourceLoadStatistics() const = 0; >+ virtual bool isPrevalentResource(const String& primaryDomain) const = 0; >+ virtual bool isVeryPrevalentResource(const String& primaryDomain) const = 0; >+ virtual void setPrevalentResource(const String& primaryDomain) = 0; >+ virtual void setVeryPrevalentResource(const String& primaryDomain) = 0; >+ >+ virtual void setGrandfathered(const String& primaryDomain, bool value) = 0; >+ virtual bool isGrandfathered(const String& primaryDomain) const = 0; >+ >+ virtual void incrementRecordsDeletedCountForDomains(const HashSet<String>&) = 0; >+ virtual void grandfatherDataForDomains(const HashSet<String>&) = 0; >+ >+ virtual void setSubframeUnderTopFrameOrigin(const String& primarySubFrameDomain, const String& primaryTopFrameDomain) = 0; >+ virtual void setSubresourceUnderTopFrameOrigin(const String& primarySubresourceDomain, const String& primaryTopFrameDomain) = 0; >+ virtual void setSubresourceUniqueRedirectTo(const String& primarySubresourceDomain, const String& primaryRedirectDomain) = 0; >+ virtual void setSubresourceUniqueRedirectFrom(const String& primarySubresourceDomain, const String& primaryRedirectDomain) = 0; >+ virtual void setTopFrameUniqueRedirectTo(const String& topFramePrimaryDomain, const String& primaryRedirectDomain) = 0; >+ virtual void setTopFrameUniqueRedirectFrom(const String& topFramePrimaryDomain, const String& primaryRedirectDomain) = 0; >+ >+ void logTestingEvent(const String&); >+ >+ void setMaxStatisticsEntries(size_t maximumEntryCount); >+ void setPruneEntriesDownTo(size_t pruneTargetCount); >+ void resetParametersToDefaultValues(); >+ Optional<Seconds> statisticsEpirationTime() const; >+ >+ virtual void calculateAndSubmitTelemetry() const = 0; >+ >+ void setNotifyPagesWhenDataRecordsWereScanned(bool); >+ void setShouldClassifyResourcesBeforeDataRecordsRemoval(bool); >+ void setShouldSubmitTelemetry(bool); >+ void setTimeToLiveUserInteraction(Seconds); >+ void setMinimumTimeBetweenDataRecordsRemoval(Seconds); >+ void setGrandfatheringTime(Seconds); >+ void setResourceLoadStatisticsDebugMode(bool); >+ bool isDebugModeEnabled() const { return m_debugModeEnabled; }; >+ void setPrevalentResourceForDebugMode(const String& domain); >+ >+ virtual void hasStorageAccess(const String& subFramePrimaryDomain, const String& topFramePrimaryDomain, Optional<uint64_t> frameID, uint64_t pageID, CompletionHandler<void(bool)>&&) = 0; >+ virtual void requestStorageAccess(String&& subFramePrimaryDomain, String&& topFramePrimaryDomain, uint64_t frameID, uint64_t pageID, bool promptEnabled, CompletionHandler<void(StorageAccessStatus)>&&) = 0; >+ virtual void grantStorageAccess(String&& subFrameHost, String&& topFrameHost, uint64_t frameID, uint64_t pageID, bool userWasPromptedNow, CompletionHandler<void(bool)>&&) = 0; >+ >+ virtual void logFrameNavigation(const String& targetPrimaryDomain, const String& mainFramePrimaryDomain, const String& sourcePrimaryDomain, const String& targetHost, const String& mainFrameHost, bool isRedirect, bool isMainFrame) = 0; >+ virtual void logUserInteraction(const String& primaryDomain) = 0; >+ virtual void logSubresourceLoading(const String& targetPrimaryDomain, const String& mainFramePrimaryDomain, WallTime lastSeen) = 0; >+ virtual void logSubresourceRedirect(const String& sourcePrimaryDomain, const String& targetPrimaryDomain) = 0; >+ >+ virtual void clearUserInteraction(const String& primaryDomain) = 0; >+ virtual bool hasHadUserInteraction(const String& primaryDomain) = 0; >+ >+ virtual void setLastSeen(const String& primaryDomain, Seconds) = 0; >+ >+ void didCreateNetworkProcess(); >+ >+ const WebResourceLoadStatisticsStore& store() const { return m_store; } >+ >+ static constexpr unsigned maxImportance { 3 }; >+ >+protected: >+ static unsigned computeImportance(const WebCore::ResourceLoadStatistics&); >+ static Vector<OperatingDate> mergeOperatingDates(const Vector<OperatingDate>& existingDates, Vector<OperatingDate>&& newDates); >+ static void debugLogDomainsInBatches(const char* action, const Vector<String>& domains); >+ bool hasStatisticsExpired(const WebCore::ResourceLoadStatistics&) const; >+ bool hasStatisticsExpired(WallTime mostRecentUserInteractionTime) const; >+#if PLATFORM(COCOA) >+ void registerUserDefaultsIfNeeded(); >+#endif >+ void scheduleStatisticsProcessingRequestIfNecessary(); >+ virtual Vector<String> ensurePrevalentResourcesForDebugMode() = 0; >+ virtual Vector<String> topPrivatelyControlledDomainsToRemoveWebsiteDataFor() = 0; >+ virtual void pruneStatisticsIfNeeded() = 0; >+ >+private: >+ void setStorageAccessPromptsEnabled(bool enabled) { m_storageAccessPromptsEnabled = enabled; } >+ bool shouldRemoveDataRecords() const; >+ void setDebugLogggingEnabled(bool enabled) { m_debugLoggingEnabled = enabled; } >+ void setDataRecordsBeingRemoved(bool); >+ void removeDataRecords(CompletionHandler<void()>&&); >+ void setCacheMaxAgeCap(Seconds); >+ void updateCacheMaxAgeCap(); >+ void setAgeCapForClientSideCookies(Seconds); >+ void updateClientSideCookiesAgeCap(); >+ >+ struct Parameters { >+ size_t pruneEntriesDownTo { 800 }; >+ size_t maxStatisticsEntries { 1000 }; >+ Optional<Seconds> timeToLiveUserInteraction; >+ Seconds minimumTimeBetweenDataRecordsRemoval { 1_h }; >+ Seconds grandfatheringTime { 24_h * 7 }; >+ Seconds cacheMaxAgeCapTime { 24_h * 7 }; >+ Seconds clientSideCookiesAgeCapTime { 24_h * 7 }; >+ bool shouldNotifyPagesWhenDataRecordsWereScanned { false }; >+ bool shouldClassifyResourcesBeforeDataRecordsRemoval { true }; >+ bool shouldSubmitTelemetry { true }; >+ }; >+ >+protected: >+ WebResourceLoadStatisticsStore& m_store; >+ Ref<WorkQueue> m_workQueue; >+ HashMap<String, WebCore::ResourceLoadStatistics> m_resourceStatisticsMap; >+#if HAVE(CORE_PREDICTION) >+ ResourceLoadStatisticsClassifierCocoa m_resourceLoadStatisticsClassifier; >+#else >+ ResourceLoadStatisticsClassifier m_resourceLoadStatisticsClassifier; >+#endif >+#if ENABLE(NETSCAPE_PLUGIN_API) >+ HashSet<uint64_t> m_activePluginTokens; >+#endif >+ Parameters m_parameters; >+ Vector<OperatingDate> m_operatingDates; >+ WallTime m_endOfGrandfatheringTimestamp; >+ bool m_debugLoggingEnabled { false }; >+ bool m_debugModeEnabled { false }; >+ String m_debugManualPrevalentResource; >+ bool m_storageAccessPromptsEnabled { false }; >+ bool m_dataRecordsBeingRemoved { false }; >+ MonotonicTime m_lastTimeDataRecordsWereRemoved; >+ >+ uint64_t m_lastStatisticsProcessingRequestIdentifier { 0 }; >+ Optional<uint64_t> m_pendingStatisticsProcessingRequestIdentifier; >+ >+ static constexpr unsigned maxNumberOfRecursiveCallsInRedirectTraceBack { 50 }; >+ static const char* debugStaticPrevalentResource; >+}; >+ >+} // namespace WebKit >+ >+#endif >diff --git a/Source/WebKit/NetworkProcess/Classifier/ResourceLoadStatisticsStoreCocoa.mm b/Source/WebKit/NetworkProcess/Classifier/ResourceLoadStatisticsStoreCocoa.mm >new file mode 100644 >index 0000000000000000000000000000000000000000..2d35e5f2821ff1552c45c6dac8b2138676f3ed1f >--- /dev/null >+++ b/Source/WebKit/NetworkProcess/Classifier/ResourceLoadStatisticsStoreCocoa.mm >@@ -0,0 +1,69 @@ >+/* >+ * Copyright (C) 2017-2019 Apple Inc. All rights reserved. >+ * >+ * Redistribution and use in source and binary forms, with or without >+ * modification, are permitted provided that the following conditions >+ * are met: >+ * 1. Redistributions of source code must retain the above copyright >+ * notice, this list of conditions and the following disclaimer. >+ * 2. Redistributions in binary form must reproduce the above copyright >+ * notice, this list of conditions and the following disclaimer in the >+ * documentation and/or other materials provided with the distribution. >+ * >+ * THIS SOFTWARE IS PROVIDED BY APPLE INC. AND ITS CONTRIBUTORS ``AS IS'' >+ * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, >+ * THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR >+ * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL APPLE INC. OR ITS CONTRIBUTORS >+ * BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR >+ * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF >+ * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS >+ * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN >+ * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) >+ * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF >+ * THE POSSIBILITY OF SUCH DAMAGE. >+ */ >+ >+#import "config.h" >+#import "ResourceLoadStatisticsStore.h" >+ >+#if ENABLE(RESOURCE_LOAD_STATISTICS) >+ >+#import <wtf/text/WTFString.h> >+ >+namespace WebKit { >+ >+void ResourceLoadStatisticsStore::registerUserDefaultsIfNeeded() >+{ >+ static dispatch_once_t initOnce; >+ >+ dispatch_once(&initOnce, ^{ >+ Seconds timeToLiveUserInteraction([[NSUserDefaults standardUserDefaults] doubleForKey:@"ResourceLoadStatisticsTimeToLiveUserInteraction"]); >+ if (timeToLiveUserInteraction > 0_s && timeToLiveUserInteraction <= 24_h * 30) >+ setTimeToLiveUserInteraction(timeToLiveUserInteraction); >+ >+ Seconds minimumTimeBetweenDataRecordsRemoval([[NSUserDefaults standardUserDefaults] doubleForKey:@"ResourceLoadStatisticsMinimumTimeBetweenDataRecordsRemoval"]); >+ if (minimumTimeBetweenDataRecordsRemoval > 0_s && minimumTimeBetweenDataRecordsRemoval < 1_h) >+ setMinimumTimeBetweenDataRecordsRemoval(minimumTimeBetweenDataRecordsRemoval); >+ >+ Seconds grandfatheringTime([[NSUserDefaults standardUserDefaults] doubleForKey:@"ResourceLoadStatisticsGrandfatheringTime"]); >+ if (grandfatheringTime > 0_s && grandfatheringTime <= 24_h * 7) >+ setGrandfatheringTime(grandfatheringTime); >+ >+ setResourceLoadStatisticsDebugMode([[NSUserDefaults standardUserDefaults] boolForKey:@"ItpDebugMode"]); >+ auto* debugManualPrevalentResource = [[NSUserDefaults standardUserDefaults] stringForKey:@"ResourceLoadStatisticsManualPrevalentResource"]; >+ if (debugManualPrevalentResource) >+ setPrevalentResourceForDebugMode(debugManualPrevalentResource); >+ >+ Seconds cacheMaxAgeCapForPrevalentResources([[NSUserDefaults standardUserDefaults] doubleForKey:@"ResourceLoadStatisticsCacheMaxAgeCap"]); >+ if (cacheMaxAgeCapForPrevalentResources > 0_s && cacheMaxAgeCapForPrevalentResources <= 24_h * 365) >+ setCacheMaxAgeCap(cacheMaxAgeCapForPrevalentResources); >+ >+ Seconds clientSideCookiesAgeCap([[NSUserDefaults standardUserDefaults] doubleForKey:@"ResourceLoadStatisticsClientSideCookiesAgeCap"]); >+ if (clientSideCookiesAgeCap > 0_s && clientSideCookiesAgeCap <= 24_h * 365) >+ setAgeCapForClientSideCookies(clientSideCookiesAgeCap); >+ }); >+} >+ >+} >+ >+#endif >diff --git a/Source/WebKit/NetworkProcess/Classifier/WebResourceLoadStatisticsStore.cpp b/Source/WebKit/NetworkProcess/Classifier/WebResourceLoadStatisticsStore.cpp >index e0dac21d67967d74dc5a3a9f448674b83ac955ea..0cd60f6650ca42e72b0acf281f0bb0dbd77b7acf 100644 >--- a/Source/WebKit/NetworkProcess/Classifier/WebResourceLoadStatisticsStore.cpp >+++ b/Source/WebKit/NetworkProcess/Classifier/WebResourceLoadStatisticsStore.cpp >@@ -30,7 +30,9 @@ > > #include "APIDictionary.h" > #include "Logging.h" >+#include "NetworkProcess.h" > #include "NetworkSession.h" >+#include "ResourceLoadStatisticsDatabaseStore.h" > #include "ResourceLoadStatisticsMemoryStore.h" > #include "ResourceLoadStatisticsPersistentStorage.h" > #include "ShouldGrandfatherStatistics.h" >@@ -47,6 +49,9 @@ > #include <WebCore/DiagnosticLoggingKeys.h> > #include <WebCore/NetworkStorageSession.h> > #include <WebCore/ResourceLoadStatistics.h> >+#include <WebCore/RuntimeEnabledFeatures.h> >+#include <WebCore/SQLiteDatabase.h> >+#include <WebCore/SQLiteStatement.h> > #include <wtf/CallbackAggregator.h> > #include <wtf/CrossThreadCopier.h> > #include <wtf/NeverDestroyed.h> >@@ -90,7 +95,9 @@ void WebResourceLoadStatisticsStore::setNotifyPagesWhenDataRecordsWereScanned(bo > ASSERT(RunLoop::isMain()); > > postTask([this, value] { >- if (m_memoryStore) >+ if (m_databaseStore) >+ m_databaseStore->setNotifyPagesWhenDataRecordsWereScanned(value); >+ else if (m_memoryStore) > m_memoryStore->setNotifyPagesWhenDataRecordsWereScanned(value); > }); > } >@@ -100,9 +107,11 @@ void WebResourceLoadStatisticsStore::setNotifyPagesWhenDataRecordsWereScanned(bo > ASSERT(RunLoop::isMain()); > > postTask([this, value, completionHandler = WTFMove(completionHandler)]() mutable { >- if (m_memoryStore) >+ if (m_databaseStore) >+ m_databaseStore->setNotifyPagesWhenDataRecordsWereScanned(value); >+ else if (m_memoryStore) > m_memoryStore->setNotifyPagesWhenDataRecordsWereScanned(value); >- >+ > postTaskReply(WTFMove(completionHandler)); > }); > } >@@ -112,7 +121,9 @@ void WebResourceLoadStatisticsStore::setShouldClassifyResourcesBeforeDataRecords > ASSERT(RunLoop::isMain()); > > postTask([this, value, completionHandler = WTFMove(completionHandler)]() mutable { >- if (m_memoryStore) >+ if (m_databaseStore) >+ m_databaseStore->setShouldClassifyResourcesBeforeDataRecordsRemoval(value); >+ else if (m_memoryStore) > m_memoryStore->setShouldClassifyResourcesBeforeDataRecordsRemoval(value); > > postTaskReply(WTFMove(completionHandler)); >@@ -124,7 +135,9 @@ void WebResourceLoadStatisticsStore::setShouldSubmitTelemetry(bool value) > ASSERT(RunLoop::isMain()); > > postTask([this, value] { >- if (m_memoryStore) >+ if (m_databaseStore) >+ m_databaseStore->setShouldSubmitTelemetry(value); >+ else if (m_memoryStore) > m_memoryStore->setShouldSubmitTelemetry(value); > }); > } >@@ -147,11 +160,15 @@ WebResourceLoadStatisticsStore::WebResourceLoadStatisticsStore(NetworkSession& n > postTask([this, resourceLoadStatisticsDirectory = resourceLoadStatisticsDirectory.isolatedCopy()] { > m_memoryStore = std::make_unique<ResourceLoadStatisticsMemoryStore>(*this, m_statisticsQueue); > m_persistentStorage = std::make_unique<ResourceLoadStatisticsPersistentStorage>(*m_memoryStore, m_statisticsQueue, resourceLoadStatisticsDirectory); >+ if (RuntimeEnabledFeatures::sharedFeatures().itpDatabaseModeEnabled()) { >+ m_databaseStore = std::make_unique<ResourceLoadStatisticsDatabaseStore>(*this, m_statisticsQueue, resourceLoadStatisticsDirectory); >+ m_databaseStore->populateFromMemoryStore(*m_memoryStore); >+ } > > // FIXME(193297): This should be revised after the UIProcess version goes away. > m_memoryStore->didCreateNetworkProcess(); > }); >- >+ > m_dailyTasksTimer.startRepeating(24_h); > } > >@@ -191,6 +208,7 @@ void WebResourceLoadStatisticsStore::flushAndDestroyPersistentStore() > m_statisticsQueue->dispatch([&semaphore, this] { > m_persistentStorage = nullptr; > m_memoryStore = nullptr; >+ m_databaseStore = nullptr; > semaphore.signal(); > }); > semaphore.wait(); >@@ -201,7 +219,9 @@ void WebResourceLoadStatisticsStore::setResourceLoadStatisticsDebugMode(bool val > ASSERT(RunLoop::isMain()); > > postTask([this, value, completionHandler = WTFMove(completionHandler)]() mutable { >- if (m_memoryStore) >+ if (m_databaseStore) >+ m_databaseStore->setResourceLoadStatisticsDebugMode(value); >+ else if (m_memoryStore) > m_memoryStore->setResourceLoadStatisticsDebugMode(value); > postTaskReply(WTFMove(completionHandler)); > }); >@@ -212,7 +232,9 @@ void WebResourceLoadStatisticsStore::setPrevalentResourceForDebugMode(const Stri > ASSERT(RunLoop::isMain()); > > postTask([this, primaryDomain = isolatedPrimaryDomain(primaryDomain), completionHandler = WTFMove(completionHandler)]() mutable { >- if (m_memoryStore) >+ if (m_databaseStore) >+ m_databaseStore->setPrevalentResourceForDebugMode(primaryDomain); >+ else if (m_memoryStore) > m_memoryStore->setPrevalentResourceForDebugMode(primaryDomain); > postTaskReply(WTFMove(completionHandler)); > }); >@@ -223,7 +245,9 @@ void WebResourceLoadStatisticsStore::scheduleStatisticsAndDataRecordsProcessing( > ASSERT(RunLoop::isMain()); > > postTask([this, completionHandler = WTFMove(completionHandler)]() mutable { >- if (m_memoryStore) >+ if (m_databaseStore) >+ m_databaseStore->processStatisticsAndDataRecords(); >+ else if (m_memoryStore) > m_memoryStore->processStatisticsAndDataRecords(); > postTaskReply(WTFMove(completionHandler)); > }); >@@ -262,17 +286,27 @@ void WebResourceLoadStatisticsStore::hasStorageAccess(const String& subFrameHost > ASSERT(RunLoop::isMain()); > > postTask([this, subFramePrimaryDomain = isolatedPrimaryDomain(subFrameHost), topFramePrimaryDomain = isolatedPrimaryDomain(topFrameHost), frameID, pageID, completionHandler = WTFMove(completionHandler)]() mutable { >- if (!m_memoryStore) { >+ if (!m_memoryStore && !m_databaseStore) { > postTaskReply([completionHandler = WTFMove(completionHandler)]() mutable { > completionHandler(false); > }); > return; > } >- m_memoryStore->hasStorageAccess(subFramePrimaryDomain, topFramePrimaryDomain, frameID, pageID, [completionHandler = WTFMove(completionHandler)](bool hasStorageAccess) mutable { >- postTaskReply([completionHandler = WTFMove(completionHandler), hasStorageAccess]() mutable { >- completionHandler(hasStorageAccess); >+ >+ >+ if (m_databaseStore) { >+ m_databaseStore->hasStorageAccess(subFramePrimaryDomain, topFramePrimaryDomain, frameID, pageID, [completionHandler = WTFMove(completionHandler)](bool hasStorageAccess) mutable { >+ postTaskReply([completionHandler = WTFMove(completionHandler), hasStorageAccess]() mutable { >+ completionHandler(hasStorageAccess); >+ }); > }); >- }); >+ } else if (m_memoryStore) { >+ m_memoryStore->hasStorageAccess(subFramePrimaryDomain, topFramePrimaryDomain, frameID, pageID, [completionHandler = WTFMove(completionHandler)](bool hasStorageAccess) mutable { >+ postTaskReply([completionHandler = WTFMove(completionHandler), hasStorageAccess]() mutable { >+ completionHandler(hasStorageAccess); >+ }); >+ }); >+ } > }); > } > >@@ -333,18 +367,26 @@ void WebResourceLoadStatisticsStore::requestStorageAccess(const String& subFrame > } > > postTask([this, subFramePrimaryDomain = crossThreadCopy(subFramePrimaryDomain), topFramePrimaryDomain = crossThreadCopy(topFramePrimaryDomain), frameID, pageID, promptEnabled, completionHandler = WTFMove(completionHandler)]() mutable { >- if (!m_memoryStore) { >+ if (!m_memoryStore && !m_databaseStore) { > postTaskReply([completionHandler = WTFMove(completionHandler)]() mutable { > completionHandler(StorageAccessStatus::CannotRequestAccess); > }); > return; > } > >- m_memoryStore->requestStorageAccess(WTFMove(subFramePrimaryDomain), WTFMove(topFramePrimaryDomain), frameID.value(), pageID, promptEnabled, [completionHandler = WTFMove(completionHandler)](StorageAccessStatus status) mutable { >- postTaskReply([completionHandler = WTFMove(completionHandler), status]() mutable { >- completionHandler(status); >+ if (m_databaseStore) { >+ m_databaseStore->requestStorageAccess(WTFMove(subFramePrimaryDomain), WTFMove(topFramePrimaryDomain), frameID.value(), pageID, promptEnabled, [completionHandler = WTFMove(completionHandler)](StorageAccessStatus status) mutable { >+ postTaskReply([completionHandler = WTFMove(completionHandler), status]() mutable { >+ completionHandler(status); >+ }); > }); >- }); >+ } else if (m_memoryStore) { >+ m_memoryStore->requestStorageAccess(subFramePrimaryDomain.isolatedCopy(), topFramePrimaryDomain.isolatedCopy(), frameID.value(), pageID, promptEnabled, [completionHandler = WTFMove(completionHandler)](StorageAccessStatus status) mutable { >+ postTaskReply([completionHandler = WTFMove(completionHandler), status]() mutable { >+ completionHandler(status); >+ }); >+ }); >+ } > }); > } > >@@ -356,8 +398,10 @@ void WebResourceLoadStatisticsStore::requestStorageAccessUnderOpener(String&& pr > // coming from IPC. Strings which are safe to move to other threads as long as nobody on this thread holds a reference > // to those strings. > postTask([this, primaryDomainInNeedOfStorageAccess = WTFMove(primaryDomainInNeedOfStorageAccess), openerPageID, openerPrimaryDomain = WTFMove(openerPrimaryDomain)]() mutable { >- if (m_memoryStore) >- m_memoryStore->requestStorageAccessUnderOpener(WTFMove(primaryDomainInNeedOfStorageAccess), openerPageID, WTFMove(openerPrimaryDomain)); >+ if (m_databaseStore) >+ m_databaseStore->requestStorageAccessUnderOpener(WTFMove(primaryDomainInNeedOfStorageAccess), openerPageID, WTFMove(openerPrimaryDomain)); >+ else if (m_memoryStore) >+ m_memoryStore->requestStorageAccessUnderOpener(primaryDomainInNeedOfStorageAccess.isolatedCopy(), openerPageID, openerPrimaryDomain.isolatedCopy()); > }); > } > >@@ -365,18 +409,26 @@ void WebResourceLoadStatisticsStore::grantStorageAccess(const String& subFrameHo > { > ASSERT(RunLoop::isMain()); > postTask([this, subFrameHost = crossThreadCopy(subFrameHost), topFrameHost = crossThreadCopy(topFrameHost), frameID, pageID, userWasPromptedNow, completionHandler = WTFMove(completionHandler)]() mutable { >- if (!m_memoryStore) { >+ if (!m_memoryStore && !m_databaseStore) { > postTaskReply([completionHandler = WTFMove(completionHandler)]() mutable { > completionHandler(false); > }); > return; > } > >- m_memoryStore->grantStorageAccess(WTFMove(subFrameHost), WTFMove(topFrameHost), frameID, pageID, userWasPromptedNow, [completionHandler = WTFMove(completionHandler)](bool wasGrantedAccess) mutable { >- postTaskReply([completionHandler = WTFMove(completionHandler), wasGrantedAccess]() mutable { >- completionHandler(wasGrantedAccess); >+ if (m_databaseStore) { >+ m_databaseStore->grantStorageAccess(WTFMove(subFrameHost), WTFMove(topFrameHost), frameID, pageID, userWasPromptedNow, [completionHandler = WTFMove(completionHandler)](bool wasGrantedAccess) mutable { >+ postTaskReply([completionHandler = WTFMove(completionHandler), wasGrantedAccess]() mutable { >+ completionHandler(wasGrantedAccess); >+ }); > }); >- }); >+ } else if (m_memoryStore) { >+ m_memoryStore->grantStorageAccess(subFrameHost.isolatedCopy(), topFrameHost.isolatedCopy(), frameID, pageID, userWasPromptedNow, [completionHandler = WTFMove(completionHandler)](bool wasGrantedAccess) mutable { >+ postTaskReply([completionHandler = WTFMove(completionHandler), wasGrantedAccess]() mutable { >+ completionHandler(wasGrantedAccess); >+ }); >+ }); >+ } > }); > } > >@@ -431,11 +483,13 @@ void WebResourceLoadStatisticsStore::performDailyTasks() > ASSERT(RunLoop::isMain()); > > postTask([this] { >- if (!m_memoryStore) >- return; >- >- m_memoryStore->includeTodayAsOperatingDateIfNecessary(); >- m_memoryStore->calculateAndSubmitTelemetry(); >+ if (m_databaseStore) { >+ m_databaseStore->includeTodayAsOperatingDateIfNecessary(); >+ m_databaseStore->calculateAndSubmitTelemetry(); >+ } else if (m_memoryStore) { >+ m_memoryStore->includeTodayAsOperatingDateIfNecessary(); >+ m_memoryStore->calculateAndSubmitTelemetry(); >+ } > }); > } > >@@ -485,7 +539,9 @@ void WebResourceLoadStatisticsStore::logFrameNavigation(const String& targetPrim > { > postTask([this, targetPrimaryDomain = isolatedPrimaryDomain(targetPrimaryDomain), mainFramePrimaryDomain = isolatedPrimaryDomain(mainFramePrimaryDomain), sourcePrimaryDomain = isolatedPrimaryDomain(sourcePrimaryDomain), targetHost = isolatedPrimaryDomain(targetHost), mainFrameHost = isolatedPrimaryDomain(mainFrameHost), isRedirect, isMainFrame] { > >- if (m_memoryStore) >+ if (m_databaseStore) >+ m_databaseStore->logFrameNavigation(targetPrimaryDomain, mainFramePrimaryDomain, sourcePrimaryDomain, targetHost, mainFrameHost, isRedirect, isMainFrame); >+ else if (m_memoryStore) > m_memoryStore->logFrameNavigation(targetPrimaryDomain, mainFramePrimaryDomain, sourcePrimaryDomain, targetHost, mainFrameHost, isRedirect, isMainFrame); > }); > } >@@ -493,7 +549,9 @@ void WebResourceLoadStatisticsStore::logFrameNavigation(const String& targetPrim > void WebResourceLoadStatisticsStore::logWebSocketLoading(const String& targetPrimaryDomain, const String& mainFramePrimaryDomain, WallTime lastSeen, CompletionHandler<void()>&& completionHandler) > { > postTask([this, targetPrimaryDomain = isolatedPrimaryDomain(targetPrimaryDomain), mainFramePrimaryDomain = isolatedPrimaryDomain(mainFramePrimaryDomain), lastSeen, completionHandler = WTFMove(completionHandler)]() mutable { >- if (m_memoryStore) >+ if (m_databaseStore) >+ m_databaseStore->logSubresourceLoading(targetPrimaryDomain, mainFramePrimaryDomain, lastSeen); >+ else if (m_memoryStore) > m_memoryStore->logSubresourceLoading(targetPrimaryDomain, mainFramePrimaryDomain, lastSeen); > > postTaskReply(WTFMove(completionHandler)); >@@ -503,9 +561,11 @@ void WebResourceLoadStatisticsStore::logWebSocketLoading(const String& targetPri > void WebResourceLoadStatisticsStore::logSubresourceLoading(const String& targetPrimaryDomain, const String& mainFramePrimaryDomain, WallTime lastSeen, CompletionHandler<void()>&& completionHandler) > { > postTask([this, targetPrimaryDomain = isolatedPrimaryDomain(targetPrimaryDomain), mainFramePrimaryDomain = isolatedPrimaryDomain(mainFramePrimaryDomain), lastSeen, completionHandler = WTFMove(completionHandler)]() mutable { >- if (m_memoryStore) >+ if (m_databaseStore) >+ m_databaseStore->logSubresourceLoading(targetPrimaryDomain, mainFramePrimaryDomain, lastSeen); >+ else if (m_memoryStore) > m_memoryStore->logSubresourceLoading(targetPrimaryDomain, mainFramePrimaryDomain, lastSeen); >- >+ > postTaskReply(WTFMove(completionHandler)); > }); > } >@@ -513,9 +573,11 @@ void WebResourceLoadStatisticsStore::logSubresourceLoading(const String& targetP > void WebResourceLoadStatisticsStore::logSubresourceRedirect(const String& sourcePrimaryDomain, const String& targetPrimaryDomain, CompletionHandler<void()>&& completionHandler) > { > postTask([this, sourcePrimaryDomain = isolatedPrimaryDomain(sourcePrimaryDomain), targetPrimaryDomain = isolatedPrimaryDomain(targetPrimaryDomain), completionHandler = WTFMove(completionHandler)]() mutable { >- if (m_memoryStore) >+ if (m_databaseStore) >+ m_databaseStore->logSubresourceRedirect(sourcePrimaryDomain, targetPrimaryDomain); >+ else if (m_memoryStore) > m_memoryStore->logSubresourceRedirect(sourcePrimaryDomain, targetPrimaryDomain); >- >+ > postTaskReply(WTFMove(completionHandler)); > }); > } >@@ -525,7 +587,9 @@ void WebResourceLoadStatisticsStore::logUserInteraction(const String& targetPrim > ASSERT(RunLoop::isMain()); > > postTask([this, primaryDomain = isolatedPrimaryDomain(targetPrimaryDomain), completionHandler = WTFMove(completionHandler)]() mutable { >- if (m_memoryStore) >+ if (m_databaseStore) >+ m_databaseStore->logUserInteraction(primaryDomain); >+ else if (m_memoryStore) > m_memoryStore->logUserInteraction(primaryDomain); > postTaskReply(WTFMove(completionHandler)); > }); >@@ -536,7 +600,9 @@ void WebResourceLoadStatisticsStore::clearUserInteraction(const String& targetPr > ASSERT(RunLoop::isMain()); > > postTask([this, primaryDomain = isolatedPrimaryDomain(targetPrimaryDomain), completionHandler = WTFMove(completionHandler)]() mutable { >- if (m_memoryStore) >+ if (m_databaseStore) >+ m_databaseStore->clearUserInteraction(primaryDomain); >+ else if (m_memoryStore) > m_memoryStore->clearUserInteraction(primaryDomain); > postTaskReply(WTFMove(completionHandler)); > }); >@@ -545,7 +611,8 @@ void WebResourceLoadStatisticsStore::clearUserInteraction(const String& targetPr > void WebResourceLoadStatisticsStore::hasHadUserInteraction(const String& primaryDomain, CompletionHandler<void(bool)>&& completionHandler) > { > postTask([this, primaryDomain = isolatedPrimaryDomain(primaryDomain), completionHandler = WTFMove(completionHandler)]() mutable { >- bool hadUserInteraction = m_memoryStore ? m_memoryStore->hasHadUserInteraction(primaryDomain) : false; >+ bool hadUserInteraction = m_databaseStore ? m_databaseStore->hasHadUserInteraction(primaryDomain) >+ : m_memoryStore ? m_memoryStore->hasHadUserInteraction(primaryDomain) : false; > postTaskReply([hadUserInteraction, completionHandler = WTFMove(completionHandler)]() mutable { > completionHandler(hadUserInteraction); > }); >@@ -557,7 +624,9 @@ void WebResourceLoadStatisticsStore::setLastSeen(const String& resourceDomain, S > ASSERT(RunLoop::isMain()); > > postTask([this, resourceDomain = isolatedPrimaryDomain(resourceDomain), seconds, completionHandler = WTFMove(completionHandler)]() mutable { >- if (m_memoryStore) >+ if (m_databaseStore) >+ m_databaseStore->setLastSeen(resourceDomain, seconds); >+ else if (m_memoryStore) > m_memoryStore->setLastSeen(resourceDomain, seconds); > postTaskReply(WTFMove(completionHandler)); > }); >@@ -568,7 +637,9 @@ void WebResourceLoadStatisticsStore::setPrevalentResource(const String& resource > ASSERT(RunLoop::isMain()); > > postTask([this, resourceDomain = isolatedPrimaryDomain(resourceDomain), completionHandler = WTFMove(completionHandler)]() mutable { >- if (m_memoryStore) >+ if (m_databaseStore) >+ m_databaseStore->setPrevalentResource(resourceDomain); >+ else if (m_memoryStore) > m_memoryStore->setPrevalentResource(resourceDomain); > postTaskReply(WTFMove(completionHandler)); > }); >@@ -579,7 +650,9 @@ void WebResourceLoadStatisticsStore::setVeryPrevalentResource(const String& prim > ASSERT(RunLoop::isMain()); > > postTask([this, primaryDomain = isolatedPrimaryDomain(primaryDomain), completionHandler = WTFMove(completionHandler)]() mutable { >- if (m_memoryStore) >+ if (m_databaseStore) >+ m_databaseStore->setVeryPrevalentResource(primaryDomain); >+ else if (m_memoryStore) > m_memoryStore->setVeryPrevalentResource(primaryDomain); > postTaskReply(WTFMove(completionHandler)); > }); >@@ -602,7 +675,8 @@ void WebResourceLoadStatisticsStore::isPrevalentResource(const String& primaryDo > ASSERT(RunLoop::isMain()); > > postTask([this, primaryDomain = isolatedPrimaryDomain(primaryDomain), completionHandler = WTFMove(completionHandler)]() mutable { >- bool isPrevalentResource = m_memoryStore ? m_memoryStore->isPrevalentResource(primaryDomain) : false; >+ bool isPrevalentResource = m_databaseStore ? m_databaseStore->isPrevalentResource(primaryDomain) >+ : m_memoryStore ? m_memoryStore->isPrevalentResource(primaryDomain) : false; > postTaskReply([isPrevalentResource, completionHandler = WTFMove(completionHandler)]() mutable { > completionHandler(isPrevalentResource); > }); >@@ -614,7 +688,8 @@ void WebResourceLoadStatisticsStore::isVeryPrevalentResource(const String& prima > ASSERT(RunLoop::isMain()); > > postTask([this, primaryDomain = isolatedPrimaryDomain(primaryDomain), completionHandler = WTFMove(completionHandler)]() mutable { >- bool isVeryPrevalentResource = m_memoryStore ? m_memoryStore->isVeryPrevalentResource(primaryDomain) : false; >+ bool isVeryPrevalentResource = m_databaseStore ? m_databaseStore->isVeryPrevalentResource(primaryDomain) >+ : m_memoryStore ? m_memoryStore->isVeryPrevalentResource(primaryDomain) : false; > postTaskReply([isVeryPrevalentResource, completionHandler = WTFMove(completionHandler)]() mutable { > completionHandler(isVeryPrevalentResource); > }); >@@ -626,7 +701,8 @@ void WebResourceLoadStatisticsStore::isRegisteredAsSubresourceUnder(const String > ASSERT(RunLoop::isMain()); > > postTask([this, subresourcePrimaryDomain = isolatedPrimaryDomain(subresource), topFramePrimaryDomain = isolatedPrimaryDomain(topFrame), completionHandler = WTFMove(completionHandler)]() mutable { >- bool isRegisteredAsSubresourceUnder = m_memoryStore ? m_memoryStore->isRegisteredAsSubresourceUnder(subresourcePrimaryDomain, topFramePrimaryDomain) : false; >+ bool isRegisteredAsSubresourceUnder = m_databaseStore ? m_databaseStore->isRegisteredAsSubresourceUnder(subresourcePrimaryDomain, topFramePrimaryDomain) >+ : m_memoryStore ? m_memoryStore->isRegisteredAsSubresourceUnder(subresourcePrimaryDomain, topFramePrimaryDomain) : false; > postTaskReply([isRegisteredAsSubresourceUnder, completionHandler = WTFMove(completionHandler)]() mutable { > completionHandler(isRegisteredAsSubresourceUnder); > }); >@@ -638,7 +714,8 @@ void WebResourceLoadStatisticsStore::isRegisteredAsSubFrameUnder(const String& s > ASSERT(RunLoop::isMain()); > > postTask([this, subFramePrimaryDomain = isolatedPrimaryDomain(subFrame), topFramePrimaryDomain = isolatedPrimaryDomain(topFrame), completionHandler = WTFMove(completionHandler)]() mutable { >- bool isRegisteredAsSubFrameUnder = m_memoryStore ? m_memoryStore->isRegisteredAsSubFrameUnder(subFramePrimaryDomain, topFramePrimaryDomain) : false; >+ bool isRegisteredAsSubFrameUnder = m_databaseStore ? m_databaseStore->isRegisteredAsSubFrameUnder(subFramePrimaryDomain, topFramePrimaryDomain) >+ : m_memoryStore ? m_memoryStore->isRegisteredAsSubFrameUnder(subFramePrimaryDomain, topFramePrimaryDomain) : false; > postTaskReply([isRegisteredAsSubFrameUnder, completionHandler = WTFMove(completionHandler)]() mutable { > completionHandler(isRegisteredAsSubFrameUnder); > }); >@@ -650,7 +727,8 @@ void WebResourceLoadStatisticsStore::isRegisteredAsRedirectingTo(const String& h > ASSERT(RunLoop::isMain()); > > postTask([this, hostRedirectedFromPrimaryDomain = isolatedPrimaryDomain(hostRedirectedFrom), hostRedirectedToPrimaryDomain = isolatedPrimaryDomain(hostRedirectedTo), completionHandler = WTFMove(completionHandler)]() mutable { >- bool isRegisteredAsRedirectingTo = m_memoryStore ? m_memoryStore->isRegisteredAsRedirectingTo(hostRedirectedFromPrimaryDomain, hostRedirectedToPrimaryDomain) : false; >+ bool isRegisteredAsRedirectingTo = m_databaseStore ? m_databaseStore->isRegisteredAsRedirectingTo(hostRedirectedFromPrimaryDomain, hostRedirectedToPrimaryDomain) >+ : m_memoryStore ? m_memoryStore->isRegisteredAsRedirectingTo(hostRedirectedFromPrimaryDomain, hostRedirectedToPrimaryDomain) : false; > postTaskReply([isRegisteredAsRedirectingTo, completionHandler = WTFMove(completionHandler)]() mutable { > completionHandler(isRegisteredAsRedirectingTo); > }); >@@ -662,7 +740,9 @@ void WebResourceLoadStatisticsStore::clearPrevalentResource(const String& resour > ASSERT(RunLoop::isMain()); > > postTask([this, resourceDomain = isolatedPrimaryDomain(resourceDomain), completionHandler = WTFMove(completionHandler)]() mutable { >- if (m_memoryStore) >+ if (m_databaseStore) >+ m_databaseStore->clearPrevalentResource(resourceDomain); >+ else if (m_memoryStore) > m_memoryStore->clearPrevalentResource(resourceDomain); > postTaskReply(WTFMove(completionHandler)); > }); >@@ -673,7 +753,9 @@ void WebResourceLoadStatisticsStore::setGrandfathered(const String& domain, bool > ASSERT(RunLoop::isMain()); > > postTask([this, primaryDomain = isolatedPrimaryDomain(domain), value, completionHandler = WTFMove(completionHandler)]() mutable { >- if (m_memoryStore) >+ if (m_databaseStore) >+ m_databaseStore->setGrandfathered(primaryDomain, value); >+ else if (m_memoryStore) > m_memoryStore->setGrandfathered(primaryDomain, value); > postTaskReply(WTFMove(completionHandler)); > }); >@@ -684,7 +766,8 @@ void WebResourceLoadStatisticsStore::isGrandfathered(const String& primaryDomain > ASSERT(RunLoop::isMain()); > > postTask([this, completionHandler = WTFMove(completionHandler), primaryDomain = isolatedPrimaryDomain(primaryDomain)]() mutable { >- bool isGrandFathered = m_memoryStore ? m_memoryStore->isGrandfathered(primaryDomain) : false; >+ bool isGrandFathered = m_databaseStore ? m_databaseStore->isGrandfathered(primaryDomain) >+ : m_memoryStore ? m_memoryStore->isGrandfathered(primaryDomain) : false; > postTaskReply([isGrandFathered, completionHandler = WTFMove(completionHandler)]() mutable { > completionHandler(isGrandFathered); > }); >@@ -696,7 +779,9 @@ void WebResourceLoadStatisticsStore::setSubframeUnderTopFrameOrigin(const String > ASSERT(RunLoop::isMain()); > > postTask([this, completionHandler = WTFMove(completionHandler), primaryTopFrameDomain = isolatedPrimaryDomain(topFrame), primarySubFrameDomain = isolatedPrimaryDomain(subframe)]() mutable { >- if (m_memoryStore) >+ if (m_databaseStore) >+ m_databaseStore->setSubframeUnderTopFrameOrigin(primarySubFrameDomain, primaryTopFrameDomain); >+ else if (m_memoryStore) > m_memoryStore->setSubframeUnderTopFrameOrigin(primarySubFrameDomain, primaryTopFrameDomain); > postTaskReply(WTFMove(completionHandler)); > }); >@@ -707,7 +792,9 @@ void WebResourceLoadStatisticsStore::setSubresourceUnderTopFrameOrigin(const Str > ASSERT(RunLoop::isMain()); > > postTask([this, completionHandler = WTFMove(completionHandler), primaryTopFrameDomain = isolatedPrimaryDomain(topFrame), primarySubresourceDomain = isolatedPrimaryDomain(subresource)]() mutable { >- if (m_memoryStore) >+ if (m_databaseStore) >+ m_databaseStore->setSubresourceUnderTopFrameOrigin(primarySubresourceDomain, primaryTopFrameDomain); >+ else if (m_memoryStore) > m_memoryStore->setSubresourceUnderTopFrameOrigin(primarySubresourceDomain, primaryTopFrameDomain); > postTaskReply(WTFMove(completionHandler)); > }); >@@ -718,7 +805,9 @@ void WebResourceLoadStatisticsStore::setSubresourceUniqueRedirectTo(const String > ASSERT(RunLoop::isMain()); > > postTask([this, completionHandler = WTFMove(completionHandler), primaryRedirectDomain = isolatedPrimaryDomain(hostNameRedirectedTo), primarySubresourceDomain = isolatedPrimaryDomain(subresource)]() mutable { >- if (m_memoryStore) >+ if (m_databaseStore) >+ m_databaseStore->setSubresourceUniqueRedirectTo(primarySubresourceDomain, primaryRedirectDomain); >+ else if (m_memoryStore) > m_memoryStore->setSubresourceUniqueRedirectTo(primarySubresourceDomain, primaryRedirectDomain); > postTaskReply(WTFMove(completionHandler)); > }); >@@ -729,7 +818,9 @@ void WebResourceLoadStatisticsStore::setSubresourceUniqueRedirectFrom(const Stri > ASSERT(RunLoop::isMain()); > > postTask([this, completionHandler = WTFMove(completionHandler), primaryRedirectDomain = isolatedPrimaryDomain(hostNameRedirectedFrom), primarySubresourceDomain = isolatedPrimaryDomain(subresource)]() mutable { >- if (m_memoryStore) >+ if (m_databaseStore) >+ m_databaseStore->setSubresourceUniqueRedirectFrom(primarySubresourceDomain, primaryRedirectDomain); >+ else if (m_memoryStore) > m_memoryStore->setSubresourceUniqueRedirectFrom(primarySubresourceDomain, primaryRedirectDomain); > postTaskReply(WTFMove(completionHandler)); > }); >@@ -740,7 +831,9 @@ void WebResourceLoadStatisticsStore::setTopFrameUniqueRedirectTo(const String& t > ASSERT(RunLoop::isMain()); > > postTask([this, completionHandler = WTFMove(completionHandler), topFramePrimaryDomain = isolatedPrimaryDomain(topFrameHostName), primaryRedirectDomain = isolatedPrimaryDomain(hostNameRedirectedTo)]() mutable { >- if (m_memoryStore) >+ if (m_databaseStore) >+ m_databaseStore->setTopFrameUniqueRedirectTo(topFramePrimaryDomain, primaryRedirectDomain); >+ else if (m_memoryStore) > m_memoryStore->setTopFrameUniqueRedirectTo(topFramePrimaryDomain, primaryRedirectDomain); > postTaskReply(WTFMove(completionHandler)); > }); >@@ -751,7 +844,9 @@ void WebResourceLoadStatisticsStore::setTopFrameUniqueRedirectFrom(const String& > ASSERT(RunLoop::isMain()); > > postTask([this, completionHandler = WTFMove(completionHandler), topFramePrimaryDomain = isolatedPrimaryDomain(topFrameHostName), primaryRedirectDomain = isolatedPrimaryDomain(hostNameRedirectedFrom)]() mutable { >- if (m_memoryStore) >+ if (m_databaseStore) >+ m_databaseStore->setTopFrameUniqueRedirectFrom(topFramePrimaryDomain, primaryRedirectDomain); >+ else if (m_memoryStore) > m_memoryStore->setTopFrameUniqueRedirectFrom(topFramePrimaryDomain, primaryRedirectDomain); > postTaskReply(WTFMove(completionHandler)); > }); >@@ -763,13 +858,19 @@ void WebResourceLoadStatisticsStore::scheduleCookieBlockingUpdate(CompletionHand > ASSERT(RunLoop::isMain()); > > postTask([this, completionHandler = WTFMove(completionHandler)]() mutable { >- if (!m_memoryStore) { >+ if (!m_memoryStore && !m_databaseStore) { > postTaskReply(WTFMove(completionHandler)); > return; > } >- m_memoryStore->updateCookieBlocking([completionHandler = WTFMove(completionHandler)]() mutable { >+ >+ auto callbackAggregator = CallbackAggregator::create([completionHandler = WTFMove(completionHandler)] () mutable { > postTaskReply(WTFMove(completionHandler)); > }); >+ >+ if (m_databaseStore) >+ m_databaseStore->updateCookieBlocking([callbackAggregator = callbackAggregator.copyRef()]() { }); >+ else if (m_memoryStore) >+ m_memoryStore->updateCookieBlocking([callbackAggregator = callbackAggregator.copyRef()]() { }); > }); > } > >@@ -778,14 +879,19 @@ void WebResourceLoadStatisticsStore::scheduleCookieBlockingUpdateForDomains(cons > // Helper function used by testing system. Should only be called from the main thread. > ASSERT(RunLoop::isMain()); > postTask([this, domainsToBlock = crossThreadCopy(domainsToBlock), completionHandler = WTFMove(completionHandler)]() mutable { >- if (!m_memoryStore) { >+ if (!m_memoryStore && !m_databaseStore) { > postTaskReply(WTFMove(completionHandler)); > return; > } > >- m_memoryStore->updateCookieBlockingForDomains(domainsToBlock, [completionHandler = WTFMove(completionHandler)]() mutable { >+ auto callbackAggregator = CallbackAggregator::create([completionHandler = WTFMove(completionHandler)] () mutable { > postTaskReply(WTFMove(completionHandler)); > }); >+ >+ if (m_databaseStore) >+ m_databaseStore->updateCookieBlockingForDomains(domainsToBlock, [callbackAggregator = callbackAggregator.copyRef()]() { }); >+ else if (m_memoryStore) >+ m_memoryStore->updateCookieBlockingForDomains(domainsToBlock, [callbackAggregator = callbackAggregator.copyRef()]() { }); > }); > } > >@@ -794,14 +900,19 @@ void WebResourceLoadStatisticsStore::scheduleClearBlockingStateForDomains(const > // Helper function used by testing system. Should only be called from the main thread. > ASSERT(RunLoop::isMain()); > postTask([this, domains = crossThreadCopy(domains), completionHandler = WTFMove(completionHandler)]() mutable { >- if (!m_memoryStore) { >+ if (!m_memoryStore && !m_databaseStore) { > postTaskReply(WTFMove(completionHandler)); > return; > } > >- m_memoryStore->clearBlockingStateForDomains(domains, [completionHandler = WTFMove(completionHandler)]() mutable { >+ auto callbackAggregator = CallbackAggregator::create([completionHandler = WTFMove(completionHandler)] () mutable { > postTaskReply(WTFMove(completionHandler)); > }); >+ >+ if (m_databaseStore) >+ m_databaseStore->clearBlockingStateForDomains(domains, [callbackAggregator = callbackAggregator.copyRef()]() { }); >+ else if (m_memoryStore) >+ m_memoryStore->clearBlockingStateForDomains(domains, [callbackAggregator = callbackAggregator.copyRef()]() { }); > }); > } > >@@ -812,22 +923,36 @@ void WebResourceLoadStatisticsStore::scheduleClearInMemoryAndPersistent(ShouldGr > if (m_persistentStorage) > m_persistentStorage->clear(); > >- CompletionHandlerCallingScope completionHandlerCaller([completionHandler = WTFMove(completionHandler)]() mutable { >+ if (!m_memoryStore && !m_databaseStore) { >+ if (shouldGrandfather == ShouldGrandfatherStatistics::Yes) >+ RELEASE_LOG(ResourceLoadStatistics, "WebResourceLoadStatisticsStore::scheduleClearInMemoryAndPersistent Before being cleared, m_memoryStore and m_databaseStore are null when trying to grandfather data."); >+ >+ postTaskReply(WTFMove(completionHandler)); >+ return; >+ } >+ >+ auto callbackAggregator = CallbackAggregator::create([completionHandler = WTFMove(completionHandler)] () mutable { > postTaskReply(WTFMove(completionHandler)); > }); > >- if (m_memoryStore) { >- m_memoryStore->clear([this, protectedThis = protectedThis.copyRef(), shouldGrandfather, completionHandlerCaller = WTFMove(completionHandlerCaller)] () mutable { >+ if (m_databaseStore) { >+ m_databaseStore->clear([this, protectedThis = protectedThis.copyRef(), shouldGrandfather, callbackAggregator = callbackAggregator.copyRef()] () mutable { >+ if (shouldGrandfather == ShouldGrandfatherStatistics::Yes) { >+ if (m_databaseStore) >+ m_databaseStore->grandfatherExistingWebsiteData([callbackAggregator = WTFMove(callbackAggregator)]() mutable { }); >+ else >+ RELEASE_LOG(ResourceLoadStatistics, "WebResourceLoadStatisticsStore::scheduleClearInMemoryAndPersistent After being cleared, m_databaseStore is null when trying to grandfather data."); >+ } >+ }); >+ } else if (m_memoryStore) { >+ m_memoryStore->clear([this, protectedThis = protectedThis.copyRef(), shouldGrandfather, callbackAggregator = callbackAggregator.copyRef()] () mutable { > if (shouldGrandfather == ShouldGrandfatherStatistics::Yes) { > if (m_memoryStore) >- m_memoryStore->grandfatherExistingWebsiteData(completionHandlerCaller.release()); >+ m_memoryStore->grandfatherExistingWebsiteData([callbackAggregator = WTFMove(callbackAggregator)]() mutable { }); > else > RELEASE_LOG(ResourceLoadStatistics, "WebResourceLoadStatisticsStore::scheduleClearInMemoryAndPersistent After being cleared, m_memoryStore is null when trying to grandfather data."); > } > }); >- } else { >- if (shouldGrandfather == ShouldGrandfatherStatistics::Yes) >- RELEASE_LOG(ResourceLoadStatistics, "WebResourceLoadStatisticsStore::scheduleClearInMemoryAndPersistent Before being cleared, m_memoryStore is null when trying to grandfather data."); > } > }); > } >@@ -845,7 +970,9 @@ void WebResourceLoadStatisticsStore::setTimeToLiveUserInteraction(Seconds second > { > ASSERT(RunLoop::isMain()); > postTask([this, seconds, completionHandler = WTFMove(completionHandler)]() mutable { >- if (m_memoryStore) >+ if (m_databaseStore) >+ m_databaseStore->setTimeToLiveUserInteraction(seconds); >+ else if (m_memoryStore) > m_memoryStore->setTimeToLiveUserInteraction(seconds); > postTaskReply(WTFMove(completionHandler)); > }); >@@ -855,7 +982,9 @@ void WebResourceLoadStatisticsStore::setMinimumTimeBetweenDataRecordsRemoval(Sec > { > ASSERT(RunLoop::isMain()); > postTask([this, seconds, completionHandler = WTFMove(completionHandler)]() mutable { >- if (m_memoryStore) >+ if (m_databaseStore) >+ m_databaseStore->setMinimumTimeBetweenDataRecordsRemoval(seconds); >+ else if (m_memoryStore) > m_memoryStore->setMinimumTimeBetweenDataRecordsRemoval(seconds); > > postTaskReply(WTFMove(completionHandler)); >@@ -866,7 +995,9 @@ void WebResourceLoadStatisticsStore::setGrandfatheringTime(Seconds seconds, Comp > { > ASSERT(RunLoop::isMain()); > postTask([this, seconds, completionHandler = WTFMove(completionHandler)]() mutable { >- if (m_memoryStore) >+ if (m_databaseStore) >+ m_databaseStore->setGrandfatheringTime(seconds); >+ else if (m_memoryStore) > m_memoryStore->setGrandfatheringTime(seconds); > > postTaskReply(WTFMove(completionHandler)); >@@ -911,7 +1042,9 @@ void WebResourceLoadStatisticsStore::setMaxStatisticsEntries(size_t maximumEntry > { > ASSERT(RunLoop::isMain()); > postTask([this, maximumEntryCount, completionHandler = WTFMove(completionHandler)]() mutable { >- if (m_memoryStore) >+ if (m_databaseStore) >+ m_databaseStore->setMaxStatisticsEntries(maximumEntryCount); >+ else if (m_memoryStore) > m_memoryStore->setMaxStatisticsEntries(maximumEntryCount); > > postTaskReply(WTFMove(completionHandler)); >@@ -923,7 +1056,9 @@ void WebResourceLoadStatisticsStore::setPruneEntriesDownTo(size_t pruneTargetCou > ASSERT(RunLoop::isMain()); > > postTask([this, pruneTargetCount, completionHandler = WTFMove(completionHandler)]() mutable { >- if (m_memoryStore) >+ if (m_databaseStore) >+ m_databaseStore->setPruneEntriesDownTo(pruneTargetCount); >+ else if (m_memoryStore) > m_memoryStore->setPruneEntriesDownTo(pruneTargetCount); > > postTaskReply(WTFMove(completionHandler)); >@@ -935,7 +1070,9 @@ void WebResourceLoadStatisticsStore::resetParametersToDefaultValues(CompletionHa > ASSERT(RunLoop::isMain()); > > postTask([this, completionHandler = WTFMove(completionHandler)]() mutable { >- if (m_memoryStore) >+ if (m_databaseStore) >+ m_databaseStore->resetParametersToDefaultValues(); >+ else if (m_memoryStore) > m_memoryStore->resetParametersToDefaultValues(); > > postTaskReply(WTFMove(completionHandler)); >diff --git a/Source/WebKit/NetworkProcess/Classifier/WebResourceLoadStatisticsStore.h b/Source/WebKit/NetworkProcess/Classifier/WebResourceLoadStatisticsStore.h >index 11c00ca85d169710b6990b419b4438a92ae891dc..a8d04800bb259b3a00f10b023fa9ced70c2d556b 100644 >--- a/Source/WebKit/NetworkProcess/Classifier/WebResourceLoadStatisticsStore.h >+++ b/Source/WebKit/NetworkProcess/Classifier/WebResourceLoadStatisticsStore.h >@@ -51,6 +51,7 @@ enum class ShouldSample : bool; > namespace WebKit { > > class NetworkSession; >+class ResourceLoadStatisticsDatabaseStore; > class ResourceLoadStatisticsMemoryStore; > class ResourceLoadStatisticsPersistentStorage; > class WebFrameProxy; >@@ -174,6 +175,7 @@ private: > Ref<WorkQueue> m_statisticsQueue; > std::unique_ptr<ResourceLoadStatisticsMemoryStore> m_memoryStore; > std::unique_ptr<ResourceLoadStatisticsPersistentStorage> m_persistentStorage; >+ std::unique_ptr<ResourceLoadStatisticsDatabaseStore> m_databaseStore; > > RunLoop::Timer<WebResourceLoadStatisticsStore> m_dailyTasksTimer; > >diff --git a/Source/WebKit/NetworkProcess/NetworkProcess.cpp b/Source/WebKit/NetworkProcess/NetworkProcess.cpp >index 9f1f000d98d7ba73d24c181fb41be14b5835e2b6..55237de12c467f048986b8b5d45373ccaffe0a79 100644 >--- a/Source/WebKit/NetworkProcess/NetworkProcess.cpp >+++ b/Source/WebKit/NetworkProcess/NetworkProcess.cpp >@@ -72,6 +72,7 @@ > #include <WebCore/NetworkStorageSession.h> > #include <WebCore/ResourceRequest.h> > #include <WebCore/RuntimeApplicationChecks.h> >+#include <WebCore/RuntimeEnabledFeatures.h> > #include <WebCore/SchemeRegistry.h> > #include <WebCore/SecurityOriginData.h> > #include <wtf/Algorithms.h> >@@ -326,6 +327,8 @@ void NetworkProcess::initializeNetworkProcess(NetworkProcessCreationParameters&& > } > #endif > >+ WebCore::RuntimeEnabledFeatures::sharedFeatures().setItpDatabaseModeEnabled(parameters.shouldEnableItpDatabaseMode); >+ > auto* defaultSession = networkSession(PAL::SessionID::defaultSessionID()); > for (const auto& cookie : parameters.defaultDataStoreParameters.pendingCookies) > defaultSession->networkStorageSession().setCookie(cookie); >diff --git a/Source/WebKit/NetworkProcess/NetworkProcessCreationParameters.cpp b/Source/WebKit/NetworkProcess/NetworkProcessCreationParameters.cpp >index c6df08e9fb7766f770afadef1aa7f212949d1f55..6d3a52ae3ddb2bb290aa75fb66f43141aab37e51 100644 >--- a/Source/WebKit/NetworkProcess/NetworkProcessCreationParameters.cpp >+++ b/Source/WebKit/NetworkProcess/NetworkProcessCreationParameters.cpp >@@ -93,6 +93,7 @@ void NetworkProcessCreationParameters::encode(IPC::Encoder& encoder) const > #if ENABLE(SERVICE_WORKER) > encoder << serviceWorkerRegistrationDirectory << serviceWorkerRegistrationDirectoryExtensionHandle << urlSchemesServiceWorkersCanHandle << shouldDisableServiceWorkerProcessTerminationDelay; > #endif >+ encoder << shouldEnableItpDatabaseMode; > } > > bool NetworkProcessCreationParameters::decode(IPC::Decoder& decoder, NetworkProcessCreationParameters& result) >@@ -217,6 +218,9 @@ bool NetworkProcessCreationParameters::decode(IPC::Decoder& decoder, NetworkProc > return false; > #endif > >+ if (!decoder.decode(result.shouldEnableItpDatabaseMode)) >+ return false; >+ > return true; > } > >diff --git a/Source/WebKit/NetworkProcess/NetworkProcessCreationParameters.h b/Source/WebKit/NetworkProcess/NetworkProcessCreationParameters.h >index 2fd3be9fded980b7573ce3b516b07a96fadaf73b..532cef38844c8f72bbb74d31159b9c4b33221606 100644 >--- a/Source/WebKit/NetworkProcess/NetworkProcessCreationParameters.h >+++ b/Source/WebKit/NetworkProcess/NetworkProcessCreationParameters.h >@@ -111,6 +111,7 @@ struct NetworkProcessCreationParameters { > Vector<String> urlSchemesServiceWorkersCanHandle; > bool shouldDisableServiceWorkerProcessTerminationDelay { false }; > #endif >+ bool shouldEnableItpDatabaseMode { false }; > }; > > } // namespace WebKit >diff --git a/Source/WebKit/Platform/classifier/ResourceLoadStatisticsClassifier.cpp b/Source/WebKit/Platform/classifier/ResourceLoadStatisticsClassifier.cpp >index bdbfdfc2051c6f4487749d0fb4db70d0ceb727a9..050c8514d8f7da70290b3b073111cb59b7c87965 100644 >--- a/Source/WebKit/Platform/classifier/ResourceLoadStatisticsClassifier.cpp >+++ b/Source/WebKit/Platform/classifier/ResourceLoadStatisticsClassifier.cpp >@@ -1,5 +1,5 @@ > /* >- * Copyright (C) 2017 Apple Inc. All rights reserved. >+ * Copyright (C) 2017-2019 Apple Inc. All rights reserved. > * > * Redistribution and use in source and binary forms, with or without > * modification, are permitted provided that the following conditions >@@ -47,7 +47,12 @@ ResourceLoadPrevalence ResourceLoadStatisticsClassifier::calculateResourcePreval > auto subresourceUniqueRedirectsToCount = resourceStatistic.subresourceUniqueRedirectsTo.size(); > auto subframeUnderTopFrameOriginsCount = resourceStatistic.subframeUnderTopFrameOrigins.size(); > auto topFrameUniqueRedirectsToCount = resourceStatistic.topFrameUniqueRedirectsTo.size(); >- >+ >+ return calculateResourcePrevalence(subresourceUnderTopFrameOriginsCount, subresourceUniqueRedirectsToCount, subframeUnderTopFrameOriginsCount, topFrameUniqueRedirectsToCount, currentPrevalence); >+} >+ >+ResourceLoadPrevalence ResourceLoadStatisticsClassifier::calculateResourcePrevalence(unsigned subresourceUnderTopFrameOriginsCount, unsigned subresourceUniqueRedirectsToCount, unsigned subframeUnderTopFrameOriginsCount, unsigned topFrameUniqueRedirectsToCount, ResourceLoadPrevalence currentPrevalence) >+{ > if (!subresourceUnderTopFrameOriginsCount > && !subresourceUniqueRedirectsToCount > && !subframeUnderTopFrameOriginsCount >diff --git a/Source/WebKit/Platform/classifier/ResourceLoadStatisticsClassifier.h b/Source/WebKit/Platform/classifier/ResourceLoadStatisticsClassifier.h >index acf1f389dd51a00c5107dc672f9d3fba9f2bfef0..af3e234c0b613ddfd005ae6090e923045f2515b0 100644 >--- a/Source/WebKit/Platform/classifier/ResourceLoadStatisticsClassifier.h >+++ b/Source/WebKit/Platform/classifier/ResourceLoadStatisticsClassifier.h >@@ -1,5 +1,5 @@ > /* >- * Copyright (C) 2017 Apple Inc. All rights reserved. >+ * Copyright (C) 2017-2019 Apple Inc. All rights reserved. > * > * Redistribution and use in source and binary forms, with or without > * modification, are permitted provided that the following conditions >@@ -42,6 +42,7 @@ public: > ResourceLoadStatisticsClassifier() = default; > virtual ~ResourceLoadStatisticsClassifier() = default; > ResourceLoadPrevalence calculateResourcePrevalence(const WebCore::ResourceLoadStatistics& resourceStatistic, ResourceLoadPrevalence currentPrevalence); >+ ResourceLoadPrevalence calculateResourcePrevalence(unsigned subresourceUnderTopFrameOriginsCount, unsigned subresourceUniqueRedirectsToCount, unsigned subframeUnderTopFrameOriginsCount, unsigned topFrameUniqueRedirectsToCount, ResourceLoadPrevalence currentPrevalence); > protected: > virtual bool classify(unsigned subresourceUnderTopFrameOriginsCount, unsigned subresourceUniqueRedirectsToCount, unsigned subframeUnderTopFrameOriginsCount) > { >diff --git a/Source/WebKit/Shared/WebPreferences.yaml b/Source/WebKit/Shared/WebPreferences.yaml >index 74a122c5b77fc5d41a28891ca0a53df5dec9fafe..74c9e1f9656fbef0f06e8e1c28340e55b1a61c4a 100644 >--- a/Source/WebKit/Shared/WebPreferences.yaml >+++ b/Source/WebKit/Shared/WebPreferences.yaml >@@ -1383,6 +1383,14 @@ ItpDebugModeEnabled: > humanReadableDescription: "Intelligent Tracking Prevention Debug Mode" > webcoreBinding: RuntimeEnabledFeatures > >+ItpDatabaseModeEnabled: >+ type: bool >+ defaultValue: false >+ humanReadableName: "ITP Database Backend Mode" >+ humanReadableDescription: "Intelligent Tracking Prevention Database Backend Mode" >+ webcoreBinding: RuntimeEnabledFeatures >+ category: internal >+ > ServiceWorkersEnabled: > type: bool > defaultValue: DEFAULT_SERVICE_WORKERS_ENABLED >diff --git a/Source/WebKit/Sources.txt b/Source/WebKit/Sources.txt >index 59cef83a5239eea0ca197f68cb372067b979e029..0435a950edfd2ae51c43abd81f05197244ef7eae 100644 >--- a/Source/WebKit/Sources.txt >+++ b/Source/WebKit/Sources.txt >@@ -42,8 +42,10 @@ NetworkProcess/NetworkSocketStream.cpp > NetworkProcess/PingLoad.cpp > NetworkProcess/PreconnectTask.cpp > >+NetworkProcess/Classifier/ResourceLoadStatisticsDatabaseStore.cpp > NetworkProcess/Classifier/ResourceLoadStatisticsMemoryStore.cpp > NetworkProcess/Classifier/ResourceLoadStatisticsPersistentStorage.cpp >+NetworkProcess/Classifier/ResourceLoadStatisticsStore.cpp > > NetworkProcess/Cookies/WebCookieManager.cpp > >diff --git a/Source/WebKit/SourcesCocoa.txt b/Source/WebKit/SourcesCocoa.txt >index c43e4e5090bedc71c9de63f867f740a941fd0f12..c00fa57b97da96fb3dacc7b462da1d4f4371febd 100644 >--- a/Source/WebKit/SourcesCocoa.txt >+++ b/Source/WebKit/SourcesCocoa.txt >@@ -29,6 +29,7 @@ NetworkProcess/cocoa/NetworkDataTaskCocoa.mm > NetworkProcess/cocoa/NetworkProcessCocoa.mm > NetworkProcess/cocoa/NetworkSessionCocoa.mm > >+NetworkProcess/Classifier/ResourceLoadStatisticsStoreCocoa.mm > NetworkProcess/Classifier/WebResourceLoadStatisticsStore.cpp > NetworkProcess/Classifier/WebResourceLoadStatisticsTelemetry.cpp > >@@ -332,7 +333,6 @@ UIProcess/Cocoa/MediaCaptureUtilities.mm > UIProcess/Cocoa/NavigationState.mm > UIProcess/Cocoa/PageClientImplCocoa.mm > UIProcess/Cocoa/PlaybackSessionManagerProxy.mm >-UIProcess/Cocoa/ResourceLoadStatisticsMemoryStoreCocoa.mm > UIProcess/Cocoa/SafeBrowsingWarningCocoa.mm > UIProcess/Cocoa/SessionStateCoding.mm > UIProcess/Cocoa/SystemPreviewControllerCocoa.mm >diff --git a/Source/WebKit/UIProcess/Cocoa/ResourceLoadStatisticsMemoryStoreCocoa.mm b/Source/WebKit/UIProcess/Cocoa/ResourceLoadStatisticsMemoryStoreCocoa.mm >deleted file mode 100644 >index 18ba8336efa5d17ad50df96703709f53d8a6b1ba..0000000000000000000000000000000000000000 >--- a/Source/WebKit/UIProcess/Cocoa/ResourceLoadStatisticsMemoryStoreCocoa.mm >+++ /dev/null >@@ -1,69 +0,0 @@ >-/* >- * Copyright (C) 2017-2018 Apple Inc. All rights reserved. >- * >- * Redistribution and use in source and binary forms, with or without >- * modification, are permitted provided that the following conditions >- * are met: >- * 1. Redistributions of source code must retain the above copyright >- * notice, this list of conditions and the following disclaimer. >- * 2. Redistributions in binary form must reproduce the above copyright >- * notice, this list of conditions and the following disclaimer in the >- * documentation and/or other materials provided with the distribution. >- * >- * THIS SOFTWARE IS PROVIDED BY APPLE INC. AND ITS CONTRIBUTORS ``AS IS'' >- * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, >- * THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR >- * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL APPLE INC. OR ITS CONTRIBUTORS >- * BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR >- * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF >- * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS >- * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN >- * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) >- * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF >- * THE POSSIBILITY OF SUCH DAMAGE. >- */ >- >-#import "config.h" >-#import "ResourceLoadStatisticsMemoryStore.h" >- >-#if ENABLE(RESOURCE_LOAD_STATISTICS) >- >-#import <wtf/text/WTFString.h> >- >-namespace WebKit { >- >-void ResourceLoadStatisticsMemoryStore::registerUserDefaultsIfNeeded() >-{ >- static dispatch_once_t initOnce; >- >- dispatch_once(&initOnce, ^{ >- Seconds timeToLiveUserInteraction([[NSUserDefaults standardUserDefaults] doubleForKey:@"ResourceLoadStatisticsTimeToLiveUserInteraction"]); >- if (timeToLiveUserInteraction > 0_s && timeToLiveUserInteraction <= 24_h * 30) >- setTimeToLiveUserInteraction(timeToLiveUserInteraction); >- >- Seconds minimumTimeBetweenDataRecordsRemoval([[NSUserDefaults standardUserDefaults] doubleForKey:@"ResourceLoadStatisticsMinimumTimeBetweenDataRecordsRemoval"]); >- if (minimumTimeBetweenDataRecordsRemoval > 0_s && minimumTimeBetweenDataRecordsRemoval < 1_h) >- setMinimumTimeBetweenDataRecordsRemoval(minimumTimeBetweenDataRecordsRemoval); >- >- Seconds grandfatheringTime([[NSUserDefaults standardUserDefaults] doubleForKey:@"ResourceLoadStatisticsGrandfatheringTime"]); >- if (grandfatheringTime > 0_s && grandfatheringTime <= 24_h * 7) >- setGrandfatheringTime(grandfatheringTime); >- >- setResourceLoadStatisticsDebugMode([[NSUserDefaults standardUserDefaults] boolForKey:@"ItpDebugMode"]); >- auto* debugManualPrevalentResource = [[NSUserDefaults standardUserDefaults] stringForKey:@"ResourceLoadStatisticsManualPrevalentResource"]; >- if (debugManualPrevalentResource) >- setPrevalentResourceForDebugMode(debugManualPrevalentResource); >- >- Seconds cacheMaxAgeCapForPrevalentResources([[NSUserDefaults standardUserDefaults] doubleForKey:@"ResourceLoadStatisticsCacheMaxAgeCap"]); >- if (cacheMaxAgeCapForPrevalentResources > 0_s && cacheMaxAgeCapForPrevalentResources <= 24_h * 365) >- setCacheMaxAgeCap(cacheMaxAgeCapForPrevalentResources); >- >- Seconds clientSideCookiesAgeCap([[NSUserDefaults standardUserDefaults] doubleForKey:@"ResourceLoadStatisticsClientSideCookiesAgeCap"]); >- if (clientSideCookiesAgeCap > 0_s && clientSideCookiesAgeCap <= 24_h * 365) >- setAgeCapForClientSideCookies(clientSideCookiesAgeCap); >- }); >-} >- >-} >- >-#endif >diff --git a/Source/WebKit/UIProcess/Cocoa/WebProcessPoolCocoa.mm b/Source/WebKit/UIProcess/Cocoa/WebProcessPoolCocoa.mm >index 98746c37c524c8784b7a6e826676665175f6fb0d..3808d89c6f39d9522eb6b048c428399fb02a7ed4 100644 >--- a/Source/WebKit/UIProcess/Cocoa/WebProcessPoolCocoa.mm >+++ b/Source/WebKit/UIProcess/Cocoa/WebProcessPoolCocoa.mm >@@ -296,6 +296,8 @@ void WebProcessPool::platformInitializeNetworkProcess(NetworkProcessCreationPara > #if ENABLE(PROXIMITY_NETWORKING) > parameters.wirelessContextIdentifier = m_configuration->wirelessContextIdentifier(); > #endif >+ >+ parameters.shouldEnableItpDatabaseMode = [defaults boolForKey:[NSString stringWithFormat:@"InternalDebug%@", WebPreferencesKey::itpDatabaseModeEnabledKey().createCFString().get()]]; > } > > void WebProcessPool::platformInvalidateContext() >diff --git a/Source/WebKit/UIProcess/WebProcessPool.cpp b/Source/WebKit/UIProcess/WebProcessPool.cpp >index 13e3e0c75ff2cd33c550467f7132880a980d7027..c886f3157d292082f57030ad71982ccede6317ed 100644 >--- a/Source/WebKit/UIProcess/WebProcessPool.cpp >+++ b/Source/WebKit/UIProcess/WebProcessPool.cpp >@@ -86,6 +86,7 @@ > #include <WebCore/ProcessWarming.h> > #include <WebCore/ResourceRequest.h> > #include <WebCore/RuntimeApplicationChecks.h> >+#include <WebCore/RuntimeEnabledFeatures.h> > #include <pal/SessionID.h> > #include <wtf/Language.h> > #include <wtf/MainThread.h> >diff --git a/Source/WebKit/WebKit.xcodeproj/project.pbxproj b/Source/WebKit/WebKit.xcodeproj/project.pbxproj >index 9050fdad6ed169ee0846ceb980ad579e3d673c67..8aa2fa62d2e00fb425ab0f3a738ba3fb028724f2 100644 >--- a/Source/WebKit/WebKit.xcodeproj/project.pbxproj >+++ b/Source/WebKit/WebKit.xcodeproj/project.pbxproj >@@ -3527,6 +3527,8 @@ > 75A8D2C5187CCF9F00C39C9E /* WKWebsiteDataStore.mm */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.objcpp; path = WKWebsiteDataStore.mm; sourceTree = "<group>"; }; > 75A8D2D4187D1C0100C39C9E /* WKWebsiteDataStoreInternal.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = WKWebsiteDataStoreInternal.h; sourceTree = "<group>"; }; > 762B7484120BBA2D00819339 /* WKPreferencesRefPrivate.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = WKPreferencesRefPrivate.h; sourceTree = "<group>"; }; >+ 7A0D7861220791EC00EBCF54 /* ResourceLoadStatisticsDatabaseStore.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; name = ResourceLoadStatisticsDatabaseStore.h; path = Classifier/ResourceLoadStatisticsDatabaseStore.h; sourceTree = "<group>"; }; >+ 7A0D7862220791ED00EBCF54 /* ResourceLoadStatisticsDatabaseStore.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; name = ResourceLoadStatisticsDatabaseStore.cpp; path = Classifier/ResourceLoadStatisticsDatabaseStore.cpp; sourceTree = "<group>"; }; > 7A1506721DD56298001F4B58 /* com.apple.WebKit.plugin-common.sb */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = text; name = "com.apple.WebKit.plugin-common.sb"; path = "DerivedSources/WebKit2/com.apple.WebKit.plugin-common.sb"; sourceTree = BUILT_PRODUCTS_DIR; }; > 7A1E2A841EEFE88A0037A0E0 /* APINotificationProvider.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = APINotificationProvider.h; sourceTree = "<group>"; }; > 7A3ACE1A1EEEF78C00A864A4 /* APIInjectedBundlePageLoaderClient.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = APIInjectedBundlePageLoaderClient.h; sourceTree = "<group>"; }; >@@ -3546,6 +3548,9 @@ > 7A9CD8C21C779AD600D9F6C7 /* WebResourceLoadStatisticsStore.messages.in */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = text; path = WebResourceLoadStatisticsStore.messages.in; sourceTree = "<group>"; }; > 7AB6EA441EEAAE2300037B2B /* APIIconDatabaseClient.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = APIIconDatabaseClient.h; sourceTree = "<group>"; }; > 7AB6EA461EEAB6B000037B2B /* APIGeolocationProvider.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = APIGeolocationProvider.h; sourceTree = "<group>"; }; >+ 7ACE82E7221CAE06000DA94C /* ResourceLoadStatisticsStore.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; name = ResourceLoadStatisticsStore.h; path = Classifier/ResourceLoadStatisticsStore.h; sourceTree = "<group>"; }; >+ 7ACE82E8221CAE07000DA94C /* ResourceLoadStatisticsStore.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; name = ResourceLoadStatisticsStore.cpp; path = Classifier/ResourceLoadStatisticsStore.cpp; sourceTree = "<group>"; }; >+ 7ACE82E9221DE722000DA94C /* ResourceLoadStatisticsStoreCocoa.mm */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.cpp.objcpp; name = ResourceLoadStatisticsStoreCocoa.mm; path = Classifier/ResourceLoadStatisticsStoreCocoa.mm; sourceTree = "<group>"; }; > 7AF2361E1E79A3B400438A05 /* WebErrors.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = WebErrors.cpp; sourceTree = "<group>"; }; > 7AF2361F1E79A3D800438A05 /* WebErrors.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = WebErrors.h; sourceTree = "<group>"; }; > 7AF236221E79A43100438A05 /* WebErrorsCocoa.mm */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.objcpp; path = WebErrorsCocoa.mm; sourceTree = "<group>"; }; >@@ -3696,7 +3701,6 @@ > 8372DB261A67562800C697C5 /* WebPageDiagnosticLoggingClient.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = WebPageDiagnosticLoggingClient.cpp; sourceTree = "<group>"; }; > 8372DB271A67562800C697C5 /* WebPageDiagnosticLoggingClient.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = WebPageDiagnosticLoggingClient.h; sourceTree = "<group>"; }; > 8372DB2E1A677D4A00C697C5 /* WKDiagnosticLoggingResultType.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = WKDiagnosticLoggingResultType.h; sourceTree = "<group>"; }; >- 837A660020E2AD8400A9DBD8 /* ResourceLoadStatisticsMemoryStoreCocoa.mm */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.objcpp; path = ResourceLoadStatisticsMemoryStoreCocoa.mm; sourceTree = "<group>"; }; > 83891B621A68B3420030F386 /* APIDiagnosticLoggingClient.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = APIDiagnosticLoggingClient.h; sourceTree = "<group>"; }; > 83891B681A68BEBC0030F386 /* _WKDiagnosticLoggingDelegate.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = _WKDiagnosticLoggingDelegate.h; sourceTree = "<group>"; }; > 83891B6A1A68C30B0030F386 /* DiagnosticLoggingClient.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = DiagnosticLoggingClient.h; sourceTree = "<group>"; }; >@@ -5319,7 +5323,6 @@ > CDA29A1E1CBEB5FB00901CCF /* PlaybackSessionManagerProxy.h */, > CDA29A221CBEB61A00901CCF /* PlaybackSessionManagerProxy.messages.in */, > CDA29A1F1CBEB5FB00901CCF /* PlaybackSessionManagerProxy.mm */, >- 837A660020E2AD8400A9DBD8 /* ResourceLoadStatisticsMemoryStoreCocoa.mm */, > 5CA9854B210BEB730057EB6B /* SafeBrowsingWarningCocoa.mm */, > 1A002D47196B345D00B9AD44 /* SessionStateCoding.h */, > 1A002D46196B345D00B9AD44 /* SessionStateCoding.mm */, >@@ -6921,10 +6924,15 @@ > 7A843A1D21E41FD900DEF663 /* Classifier */ = { > isa = PBXGroup; > children = ( >+ 7A0D7862220791ED00EBCF54 /* ResourceLoadStatisticsDatabaseStore.cpp */, >+ 7A0D7861220791EC00EBCF54 /* ResourceLoadStatisticsDatabaseStore.h */, > 7AFBD36421E51BAB005DBACB /* ResourceLoadStatisticsMemoryStore.cpp */, > 7AFBD36521E51BAB005DBACB /* ResourceLoadStatisticsMemoryStore.h */, > 7AFBD36921E542E8005DBACB /* ResourceLoadStatisticsPersistentStorage.cpp */, > 7AFBD36821E542E8005DBACB /* ResourceLoadStatisticsPersistentStorage.h */, >+ 7ACE82E8221CAE07000DA94C /* ResourceLoadStatisticsStore.cpp */, >+ 7ACE82E7221CAE06000DA94C /* ResourceLoadStatisticsStore.h */, >+ 7ACE82E9221DE722000DA94C /* ResourceLoadStatisticsStoreCocoa.mm */, > 7A41E9FA21F81DAC00B88CDB /* ShouldGrandfatherStatistics.h */, > 7A3FECA121F7C09700F267CD /* StorageAccessStatus.h */, > 7A843A1A21E41FB200DEF663 /* WebResourceLoadStatisticsStore.cpp */, >@@ -10790,7 +10798,6 @@ > 2D913444212CF9F000128AFD /* NetscapePlugin.cpp in Sources */, > 2D92A796212B6ADA00F493FD /* NetscapePluginModule.cpp in Sources */, > 2D913445212CF9F000128AFD /* NetscapePluginStream.cpp in Sources */, >- 6BD05866220CE8C2000BED5C /* NetworkAdClickAttribution.cpp in Sources */, > 51DD9F2816367DA2001578E9 /* NetworkConnectionToWebProcessMessageReceiver.cpp in Sources */, > 52F060E11654318500F3281B /* NetworkContentRuleListManagerMessageReceiver.cpp in Sources */, > 51F060E11654318500F3283F /* NetworkMDNSRegisterMessageReceiver.cpp in Sources */,
You cannot view the attachment while viewing its details because your browser does not support IFRAMEs.
View the attachment on a separate page
.
View Attachment As Diff
View Attachment As Raw
Actions:
View
|
Formatted Diff
|
Diff
Attachments on
bug 194867
:
362617
|
363032
|
363065
|
363255
|
363476
|
363513
|
363552
|
363566
|
363577
|
363578