Attachment 358332 Details for Bug 191806 – Patch for landing

[patch] Patch for landing

bug-191806-20190104103420.patch (text/plain), 34.09 KB, created by Jer Noble on 2019-01-04 10:34:21 PST

(hide)

Description:

Filename:

MIME Type:

Creator: Jer Noble

Created: 2019-01-04 10:34:21 PST

Size: 34.09 KB

patch

obsolete

>Subversion Revision: 239617
>diff --git a/Source/WebCore/ChangeLog b/Source/WebCore/ChangeLog
>index 239ee39a9f9b303e854ce81cca22dca5b4ea4218..b0476fae22b62c5ced88af552a1b151a67f3e9ef 100644
>--- a/Source/WebCore/ChangeLog
>+++ b/Source/WebCore/ChangeLog
>@@ -1,3 +1,50 @@
>+2019-01-04  Jer Noble  <jer.noble@apple.com>
>+
>+        Web Content process main thread blocked beneath ImageDecoderAVFObjC::readSamples for many seconds on imgur.com
>+        https://bugs.webkit.org/show_bug.cgi?id=191806
>+        <rdar://problem/46151477>
>+
>+        Reviewed by Dean Jackson.
>+
>+        Test: http/tests/images/mp4-partial-load.html
>+
>+        Rather than use an AVAssetReaderTrackOutput, which will load both sample metadata and sample data
>+        synchronously when a sample is requested, use AVAssetReaderSampleReferenceOutput, which only loads
>+        sample metadata, including the byte offset and byte length of the sample data. By waiting until the
>+        AVAsset signals that it's own metadata is loaded, we can safely parse all the sample metadata without
>+        blocking on network loads. Once enough data is loaded, we can replace the byte reference and offset
>+        attachements in the sample with actual data, and mark the sample as "complete".
>+
>+        Because the existing ImageSource assumes that image data parsing will occur synchronously, and that
>+        synchronous parsing could cause a hang if the metadata is not loaded, add a new callback method which
>+        allows the ImageSource to be notified when the encodedDataStatus changes. The ImageSource notifies the
>+        CacheImage, which notifies the RenderImage, and thus the asynchronous parsing will kick off the
>+        renderer's animation loop.
>+
>+        * loader/cache/CachedImage.cpp:
>+        (WebCore::CachedImage::CachedImageObserver::encodedDataStatusChanged):
>+        (WebCore::CachedImage::encodedDataStatusChanged):
>+        * loader/cache/CachedImage.h:
>+        * platform/graphics/ImageDecoder.h:
>+        (WebCore::ImageDecoder::setEncodedDataStatusChangeCallback):
>+        * platform/graphics/ImageObserver.h:
>+        (WebCore::ImageObserver::encodedDataStatusChanged):
>+        * platform/graphics/ImageSource.cpp:
>+        (WebCore::ImageSource::ensureDecoderAvailable):
>+        (WebCore::ImageSource::encodedDataStatusChanged):
>+        (WebCore::ImageSource::frameDecodingStatusAtIndex):
>+        * platform/graphics/ImageSource.h:
>+        * platform/graphics/avfoundation/objc/ImageDecoderAVFObjC.h:
>+        * platform/graphics/avfoundation/objc/ImageDecoderAVFObjC.mm:
>+        (-[WebCoreSharedBufferResourceLoaderDelegate data]):
>+        (WebCore::ImageDecoderAVFObjCSample::byteRange const):
>+        (WebCore::ImageDecoderAVFObjC::readSamples):
>+        (WebCore::ImageDecoderAVFObjC::setEncodedDataStatusChangeCallback):
>+        (WebCore::ImageDecoderAVFObjC::encodedDataStatus const):
>+        (WebCore::ImageDecoderAVFObjC::frameIsCompleteAtIndex const):
>+        (WebCore::ImageDecoderAVFObjC::createFrameImageAtIndex):
>+        (WebCore::ImageDecoderAVFObjC::sampleIsComplete const):
>+
> 2018-12-18  Jer Noble  <jer.noble@apple.com>
> 
>         [WebKitLegacy] Media playback pauses on scroll
>diff --git a/Source/WebCore/PAL/ChangeLog b/Source/WebCore/PAL/ChangeLog
>index 9fef67ef509d592aa6cba98760f94dea543069bb..aca9215e9a495ec590260fcaf3836321900a9cb7 100644
>--- a/Source/WebCore/PAL/ChangeLog
>+++ b/Source/WebCore/PAL/ChangeLog
>@@ -1,3 +1,14 @@
>+2019-01-04  Jer Noble  <jer.noble@apple.com>
>+
>+        Web Content process main thread blocked beneath ImageDecoderAVFObjC::readSamples for many seconds on imgur.com
>+        https://bugs.webkit.org/show_bug.cgi?id=191806
>+        <rdar://problem/46151477>
>+
>+        Reviewed by Dean Jackson.
>+
>+        * pal/cf/CoreMediaSoftLink.cpp:
>+        * pal/cf/CoreMediaSoftLink.h:
>+
> 2018-09-12  Jer Noble  <jer.noble@apple.com>
> 
>         SDK_VARIANT build destinations should be separate from non-SDK_VARIANT builds
>diff --git a/Source/WebCore/PAL/pal/cf/CoreMediaSoftLink.cpp b/Source/WebCore/PAL/pal/cf/CoreMediaSoftLink.cpp
>index 548bdfb7b60c78ec2d8a2836a5f171caa977914a..c127bf32074cc3359c36e40d7b353d2e9f479241 100644
>--- a/Source/WebCore/PAL/pal/cf/CoreMediaSoftLink.cpp
>+++ b/Source/WebCore/PAL/pal/cf/CoreMediaSoftLink.cpp
>@@ -108,7 +108,10 @@ SOFT_LINK_FUNCTION_FOR_SOURCE(PAL, CoreMedia, CMSampleBufferGetSampleAttachments
> SOFT_LINK_FUNCTION_FOR_SOURCE(PAL, CoreMedia, CMSampleBufferGetSampleTimingInfoArray, OSStatus, (CMSampleBufferRef sbuf, CMItemCount timingArrayEntries, CMSampleTimingInfo *timingArrayOut, CMItemCount *timingArrayEntriesNeededOut), (sbuf, timingArrayEntries, timingArrayOut, timingArrayEntriesNeededOut))
> SOFT_LINK_FUNCTION_FOR_SOURCE(PAL, CoreMedia, CMTimeConvertScale, CMTime, (CMTime time, int32_t newTimescale, CMTimeRoundingMethod method), (time, newTimescale, method))
> SOFT_LINK_FUNCTION_FOR_SOURCE(PAL, CoreMedia, CMSampleBufferGetTotalSampleSize, size_t, (CMSampleBufferRef sbuf), (sbuf))
>+SOFT_LINK_FUNCTION_FOR_SOURCE(PAL, CoreMedia, CMSampleBufferSetDataBuffer, OSStatus, (CMSampleBufferRef sbuf, CMBlockBufferRef buffer), (sbuf, buffer))
>+SOFT_LINK_FUNCTION_FOR_SOURCE(PAL, CoreMedia, CMGetAttachment, CFTypeRef, (CMAttachmentBearerRef target, CFStringRef key, CMAttachmentMode* attachmentModeOut), (target, key, attachmentModeOut))
> SOFT_LINK_FUNCTION_FOR_SOURCE(PAL, CoreMedia, CMSetAttachment, void, (CMAttachmentBearerRef target, CFStringRef key, CFTypeRef value, CMAttachmentMode attachmentMode), (target, key, value, attachmentMode))
>+SOFT_LINK_FUNCTION_FOR_SOURCE(PAL, CoreMedia, CMRemoveAttachment, void, (CMAttachmentBearerRef target, CFStringRef key), (target, key))
> SOFT_LINK_FUNCTION_FOR_SOURCE(PAL, CoreMedia, CMTimebaseCreateWithMasterClock, OSStatus, (CFAllocatorRef allocator, CMClockRef masterClock, CMTimebaseRef *timebaseOut), (allocator, masterClock, timebaseOut))
> SOFT_LINK_FUNCTION_FOR_SOURCE(PAL, CoreMedia, CMTimebaseGetTime, CMTime, (CMTimebaseRef timebase), (timebase))
> SOFT_LINK_FUNCTION_FOR_SOURCE(PAL, CoreMedia, CMTimebaseGetRate, Float64, (CMTimebaseRef timebase), (timebase))
>@@ -141,6 +144,8 @@ SOFT_LINK_CONSTANT_FOR_SOURCE(PAL, CoreMedia, kCMSampleBufferAttachmentKey_Drain
> SOFT_LINK_CONSTANT_FOR_SOURCE(PAL, CoreMedia, kCMSampleBufferAttachmentKey_EmptyMedia, CFStringRef)
> SOFT_LINK_CONSTANT_FOR_SOURCE(PAL, CoreMedia, kCMSampleBufferAttachmentKey_PostNotificationWhenConsumed, CFStringRef)
> SOFT_LINK_CONSTANT_FOR_SOURCE(PAL, CoreMedia, kCMSampleBufferAttachmentKey_ResetDecoderBeforeDecoding, CFStringRef)
>+SOFT_LINK_CONSTANT_FOR_SOURCE(PAL, CoreMedia, kCMSampleBufferAttachmentKey_SampleReferenceByteOffset, CFStringRef)
>+SOFT_LINK_CONSTANT_FOR_SOURCE(PAL, CoreMedia, kCMSampleBufferAttachmentKey_SampleReferenceURL, CFStringRef)
> SOFT_LINK_CONSTANT_FOR_SOURCE(PAL, CoreMedia, kCMSampleAttachmentKey_DisplayImmediately, CFStringRef)
> SOFT_LINK_CONSTANT_FOR_SOURCE(PAL, CoreMedia, kCMSampleAttachmentKey_IsDependedOnByOthers, CFStringRef)
> SOFT_LINK_CONSTANT_FOR_SOURCE(PAL, CoreMedia, kCMSampleBufferConsumerNotification_BufferConsumed, CFStringRef)
>@@ -148,6 +153,7 @@ SOFT_LINK_CONSTANT_FOR_SOURCE(PAL, CoreMedia, kCMSampleBufferConsumerNotificatio
> SOFT_LINK_CONSTANT_FOR_SOURCE(PAL, CoreMedia, kCMTimebaseNotification_EffectiveRateChanged, CFStringRef)
> SOFT_LINK_CONSTANT_FOR_SOURCE(PAL, CoreMedia, kCMTimebaseNotification_TimeJumped, CFStringRef)
> SOFT_LINK_FUNCTION_FOR_SOURCE(PAL, CoreMedia, CMAudioFormatDescriptionGetStreamBasicDescription, const AudioStreamBasicDescription *, (CMAudioFormatDescriptionRef desc), (desc))
>+SOFT_LINK_FUNCTION_FOR_SOURCE(PAL, CoreMedia, CMBlockBufferCreateWithMemoryBlock, OSStatus, (CFAllocatorRef structureAllocator, void* memoryBlock, size_t blockLength, CFAllocatorRef blockAllocator, const CMBlockBufferCustomBlockSource* customBlockSource, size_t offsetToData, size_t dataLength, CMBlockBufferFlags flags, CMBlockBufferRef* blockBufferOut), (structureAllocator, memoryBlock, blockLength, blockAllocator, customBlockSource, offsetToData, dataLength, flags, blockBufferOut))
> SOFT_LINK_FUNCTION_FOR_SOURCE(PAL, CoreMedia, CMSampleBufferGetAudioBufferListWithRetainedBlockBuffer, OSStatus, (CMSampleBufferRef sbuf, size_t *bufferListSizeNeededOut, AudioBufferList *bufferListOut, size_t bufferListSize, CFAllocatorRef bbufStructAllocator, CFAllocatorRef bbufMemoryAllocator, uint32_t flags, CMBlockBufferRef *blockBufferOut), (sbuf, bufferListSizeNeededOut, bufferListOut, bufferListSize, bbufStructAllocator, bbufMemoryAllocator, flags, blockBufferOut))
> SOFT_LINK_FUNCTION_FOR_SOURCE(PAL, CoreMedia, CMSampleBufferGetNumSamples, CMItemCount, (CMSampleBufferRef sbuf), (sbuf))
> SOFT_LINK_FUNCTION_FOR_SOURCE(PAL, CoreMedia, CMSampleBufferCopySampleBufferForRange, OSStatus, (CFAllocatorRef allocator, CMSampleBufferRef sbuf, CFRange sampleRange, CMSampleBufferRef* sBufOut), (allocator, sbuf, sampleRange, sBufOut))
>diff --git a/Source/WebCore/PAL/pal/cf/CoreMediaSoftLink.h b/Source/WebCore/PAL/pal/cf/CoreMediaSoftLink.h
>index ffb134751a801472fec6b84020a5cbfe03f177f0..c9510495e69834bda30ab489d2ecabb5b81a8c12 100644
>--- a/Source/WebCore/PAL/pal/cf/CoreMediaSoftLink.h
>+++ b/Source/WebCore/PAL/pal/cf/CoreMediaSoftLink.h
>@@ -171,8 +171,14 @@ SOFT_LINK_FUNCTION_FOR_HEADER(PAL, CoreMedia, CMSampleBufferGetSampleTimingInfoA
> #define CMSampleBufferGetSampleTimingInfoArray softLink_CoreMedia_CMSampleBufferGetSampleTimingInfoArray
> SOFT_LINK_FUNCTION_FOR_HEADER(PAL, CoreMedia, CMSampleBufferGetTotalSampleSize, size_t, (CMSampleBufferRef sbuf), (sbuf))
> #define CMSampleBufferGetTotalSampleSize softLink_CoreMedia_CMSampleBufferGetTotalSampleSize
>+SOFT_LINK_FUNCTION_FOR_HEADER(PAL, CoreMedia, CMSampleBufferSetDataBuffer, OSStatus, (CMSampleBufferRef sbuf, CMBlockBufferRef buffer), (sbuf, buffer))
>+#define CMSampleBufferSetDataBuffer softLink_CoreMedia_CMSampleBufferSetDataBuffer
>+SOFT_LINK_FUNCTION_FOR_HEADER(PAL, CoreMedia, CMGetAttachment, CFTypeRef, (CMAttachmentBearerRef target, CFStringRef key, CMAttachmentMode* attachmentModeOut), (target, key, attachmentModeOut))
>+#define CMGetAttachment softLink_CoreMedia_CMGetAttachment
> SOFT_LINK_FUNCTION_FOR_HEADER(PAL, CoreMedia, CMSetAttachment, void, (CMAttachmentBearerRef target, CFStringRef key, CFTypeRef value, CMAttachmentMode attachmentMode), (target, key, value, attachmentMode))
> #define CMSetAttachment softLink_CoreMedia_CMSetAttachment
>+SOFT_LINK_FUNCTION_FOR_HEADER(PAL, CoreMedia, CMRemoveAttachment, void, (CMAttachmentBearerRef target, CFStringRef key), (target, key))
>+#define CMRemoveAttachment softLink_CoreMedia_CMRemoveAttachment
> SOFT_LINK_FUNCTION_FOR_HEADER(PAL, CoreMedia, CMTimebaseCreateWithMasterClock, OSStatus, (CFAllocatorRef allocator, CMClockRef masterClock, CMTimebaseRef *timebaseOut), (allocator, masterClock, timebaseOut))
> #define CMTimebaseCreateWithMasterClock softLink_CoreMedia_CMTimebaseCreateWithMasterClock
> SOFT_LINK_FUNCTION_FOR_HEADER(PAL, CoreMedia, CMTimebaseGetTime, CMTime, (CMTimebaseRef timebase), (timebase))
>@@ -240,6 +246,10 @@ SOFT_LINK_CONSTANT_FOR_HEADER(PAL, CoreMedia, kCMSampleBufferAttachmentKey_PostN
> #define kCMSampleBufferAttachmentKey_PostNotificationWhenConsumed get_CoreMedia_kCMSampleBufferAttachmentKey_PostNotificationWhenConsumed()
> SOFT_LINK_CONSTANT_FOR_HEADER(PAL, CoreMedia, kCMSampleBufferAttachmentKey_ResetDecoderBeforeDecoding, CFStringRef)
> #define kCMSampleBufferAttachmentKey_ResetDecoderBeforeDecoding get_CoreMedia_kCMSampleBufferAttachmentKey_ResetDecoderBeforeDecoding()
>+SOFT_LINK_CONSTANT_FOR_HEADER(PAL, CoreMedia, kCMSampleBufferAttachmentKey_SampleReferenceByteOffset, CFStringRef)
>+#define kCMSampleBufferAttachmentKey_SampleReferenceByteOffset get_CoreMedia_kCMSampleBufferAttachmentKey_SampleReferenceByteOffset()
>+SOFT_LINK_CONSTANT_FOR_HEADER(PAL, CoreMedia, kCMSampleBufferAttachmentKey_SampleReferenceURL, CFStringRef)
>+#define kCMSampleBufferAttachmentKey_SampleReferenceURL get_CoreMedia_kCMSampleBufferAttachmentKey_SampleReferenceURL()
> SOFT_LINK_CONSTANT_FOR_HEADER(PAL, CoreMedia, kCMTimebaseNotification_EffectiveRateChanged, CFStringRef)
> #define kCMTimebaseNotification_EffectiveRateChanged get_CoreMedia_kCMTimebaseNotification_EffectiveRateChanged()
> SOFT_LINK_CONSTANT_FOR_HEADER(PAL, CoreMedia, kCMTimebaseNotification_TimeJumped, CFStringRef)
>@@ -248,6 +258,8 @@ SOFT_LINK_CONSTANT_FOR_HEADER(PAL, CoreMedia, kCMSampleBufferConsumerNotificatio
> #define kCMSampleBufferConsumerNotification_BufferConsumed get_CoreMedia_kCMSampleBufferConsumerNotification_BufferConsumed()
> SOFT_LINK_FUNCTION_FOR_HEADER(PAL, CoreMedia, CMAudioFormatDescriptionGetStreamBasicDescription, const AudioStreamBasicDescription *, (CMAudioFormatDescriptionRef desc), (desc))
> #define CMAudioFormatDescriptionGetStreamBasicDescription softLink_CoreMedia_CMAudioFormatDescriptionGetStreamBasicDescription
>+SOFT_LINK_FUNCTION_FOR_HEADER(PAL, CoreMedia, CMBlockBufferCreateWithMemoryBlock, OSStatus, (CFAllocatorRef structureAllocator, void* memoryBlock, size_t blockLength, CFAllocatorRef blockAllocator, const CMBlockBufferCustomBlockSource* customBlockSource, size_t offsetToData, size_t dataLength, CMBlockBufferFlags flags, CMBlockBufferRef* blockBufferOut), (structureAllocator, memoryBlock, blockLength, blockAllocator, customBlockSource, offsetToData, dataLength, flags, blockBufferOut))
>+#define CMBlockBufferCreateWithMemoryBlock softLink_CoreMedia_CMBlockBufferCreateWithMemoryBlock
> SOFT_LINK_FUNCTION_FOR_HEADER(PAL, CoreMedia, CMSampleBufferGetAudioBufferListWithRetainedBlockBuffer, OSStatus, (CMSampleBufferRef sbuf, size_t *bufferListSizeNeededOut, AudioBufferList *bufferListOut, size_t bufferListSize, CFAllocatorRef bbufStructAllocator, CFAllocatorRef bbufMemoryAllocator, uint32_t flags, CMBlockBufferRef *blockBufferOut), (sbuf, bufferListSizeNeededOut, bufferListOut, bufferListSize, bbufStructAllocator, bbufMemoryAllocator, flags, blockBufferOut))
> #define CMSampleBufferGetAudioBufferListWithRetainedBlockBuffer softLink_CoreMedia_CMSampleBufferGetAudioBufferListWithRetainedBlockBuffer
> SOFT_LINK_FUNCTION_FOR_HEADER(PAL, CoreMedia, CMSampleBufferGetNumSamples, CMItemCount, (CMSampleBufferRef sbuf), (sbuf))
>diff --git a/Source/WebCore/loader/cache/CachedImage.cpp b/Source/WebCore/loader/cache/CachedImage.cpp
>index 18fdae2aadb03ea9dd71fcedb5b9d0f9ac35b6bc..77a0ba55c75dee3b4d0880dafc65654532db5834 100644
>--- a/Source/WebCore/loader/cache/CachedImage.cpp
>+++ b/Source/WebCore/loader/cache/CachedImage.cpp
>@@ -371,6 +371,12 @@ CachedImage::CachedImageObserver::CachedImageObserver(CachedImage& image)
>     m_cachedImages.add(&image);
> }
> 
>+void CachedImage::CachedImageObserver::encodedDataStatusChanged(const Image& image, EncodedDataStatus status)
>+{
>+    for (auto cachedImage : m_cachedImages)
>+        cachedImage->encodedDataStatusChanged(image, status);
>+}
>+
> void CachedImage::CachedImageObserver::decodedSizeChanged(const Image& image, long long delta)
> {
>     for (auto cachedImage : m_cachedImages)
>@@ -582,6 +588,14 @@ void CachedImage::destroyDecodedData()
>         m_image->destroyDecodedData();
> }
> 
>+void CachedImage::encodedDataStatusChanged(const Image& image, EncodedDataStatus)
>+{
>+    if (&image != m_image)
>+        return;
>+
>+    notifyObservers();
>+}
>+
> void CachedImage::decodedSizeChanged(const Image& image, long long delta)
> {
>     if (&image != m_image)
>diff --git a/Source/WebCore/loader/cache/CachedImage.h b/Source/WebCore/loader/cache/CachedImage.h
>index b23957cbabe1e7c7e23fbf73aba254a0e15862f9..9113c9c417929eeb5f5b8e3e7fb9b8cba3d8ad33 100644
>--- a/Source/WebCore/loader/cache/CachedImage.h
>+++ b/Source/WebCore/loader/cache/CachedImage.h
>@@ -143,6 +143,7 @@ private:
>         String mimeType() const override { return !m_cachedImages.isEmpty() ? (*m_cachedImages.begin())->mimeType() : emptyString(); }
>         long long expectedContentLength() const override { return !m_cachedImages.isEmpty() ? (*m_cachedImages.begin())->expectedContentLength() : 0; }
> 
>+        void encodedDataStatusChanged(const Image&, EncodedDataStatus) final;
>         void decodedSizeChanged(const Image&, long long delta) final;
>         void didDraw(const Image&) final;
> 
>@@ -153,6 +154,7 @@ private:
>         HashSet<CachedImage*> m_cachedImages;
>     };
> 
>+    void encodedDataStatusChanged(const Image&, EncodedDataStatus);
>     void decodedSizeChanged(const Image&, long long delta);
>     void didDraw(const Image&);
>     bool canDestroyDecodedData(const Image&);
>diff --git a/Source/WebCore/platform/graphics/ImageDecoder.h b/Source/WebCore/platform/graphics/ImageDecoder.h
>index dc46d11b44e1d5cbad1ec6f1e82ff41356f7abe0..727f1ed64a56287fec83787bc8106284f2c6f5db 100644
>--- a/Source/WebCore/platform/graphics/ImageDecoder.h
>+++ b/Source/WebCore/platform/graphics/ImageDecoder.h
>@@ -55,6 +55,7 @@ public:
>     virtual size_t bytesDecodedToDetermineProperties() const = 0;
> 
>     virtual EncodedDataStatus encodedDataStatus() const = 0;
>+    virtual void setEncodedDataStatusChangeCallback(WTF::Function<void(EncodedDataStatus)>&&) { }
>     virtual bool isSizeAvailable() const { return encodedDataStatus() >= EncodedDataStatus::SizeAvailable; }
>     virtual IntSize size() const = 0;
>     virtual size_t frameCount() const = 0;
>diff --git a/Source/WebCore/platform/graphics/ImageObserver.h b/Source/WebCore/platform/graphics/ImageObserver.h
>index f28323c255b1d0a014351c934273da01c5af85bc..7fd274022d5a05a30c67e921c81634ceaae61ee9 100644
>--- a/Source/WebCore/platform/graphics/ImageObserver.h
>+++ b/Source/WebCore/platform/graphics/ImageObserver.h
>@@ -42,6 +42,7 @@ public:
>     virtual String mimeType() const = 0;
>     virtual long long expectedContentLength() const = 0;
> 
>+    virtual void encodedDataStatusChanged(const Image&, EncodedDataStatus) { };
>     virtual void decodedSizeChanged(const Image&, long long delta) = 0;
> 
>     virtual void didDraw(const Image&) = 0;
>diff --git a/Source/WebCore/platform/graphics/ImageSource.cpp b/Source/WebCore/platform/graphics/ImageSource.cpp
>index 68dff9620bc14a19685a4ec16542c5a560e04b6b..5b015c899ba223935dfe1b59644b94d9acdde116 100644
>--- a/Source/WebCore/platform/graphics/ImageSource.cpp
>+++ b/Source/WebCore/platform/graphics/ImageSource.cpp
>@@ -76,6 +76,10 @@ bool ImageSource::ensureDecoderAvailable(SharedBuffer* data)
>         return true;
> 
>     m_decoder = ImageDecoder::create(*data, mimeType(), m_alphaOption, m_gammaAndColorProfileOption);
>+    m_decoder->setEncodedDataStatusChangeCallback([weakThis = makeWeakPtr(this)] (auto status) {
>+        if (weakThis)
>+            weakThis->encodedDataStatusChanged(status);
>+    });
>     if (!isDecoderAvailable())
>         return false;
> 
>@@ -154,6 +158,20 @@ void ImageSource::clearFrameBufferCache(size_t beforeFrame)
>     m_decoder->clearFrameBufferCache(beforeFrame);
> }
> 
>+void ImageSource::encodedDataStatusChanged(EncodedDataStatus status)
>+{
>+    if (status == m_encodedDataStatus)
>+        return;
>+
>+    m_encodedDataStatus = status;
>+
>+    if (status >= EncodedDataStatus::SizeAvailable)
>+        growFrames();
>+
>+    if (m_image && m_image->imageObserver())
>+        m_image->imageObserver()->encodedDataStatusChanged(*m_image, status);
>+}
>+
> void ImageSource::decodedSizeChanged(long long decodedSize)
> {
>     if (!decodedSize || !m_image || !m_image->imageObserver())
>@@ -595,7 +613,7 @@ bool ImageSource::frameIsBeingDecodedAndIsCompatibleWithOptionsAtIndex(size_t in
> 
> DecodingStatus ImageSource::frameDecodingStatusAtIndex(size_t index)
> {
>-    return frameMetadataAtIndex<DecodingStatus>(index, (&ImageFrame::decodingStatus));
>+    return frameMetadataAtIndexCacheIfNeeded<DecodingStatus>(index, (&ImageFrame::decodingStatus), nullptr, ImageFrame::Caching::Metadata);
> }
> 
> bool ImageSource::frameHasAlphaAtIndex(size_t index)
>diff --git a/Source/WebCore/platform/graphics/ImageSource.h b/Source/WebCore/platform/graphics/ImageSource.h
>index d46cd9b2d70363865cc27458a84c018934ed128d..52f19541ae889df8680dd59c9d174bf053b53641 100644
>--- a/Source/WebCore/platform/graphics/ImageSource.h
>+++ b/Source/WebCore/platform/graphics/ImageSource.h
>@@ -30,6 +30,7 @@
> #include <wtf/Forward.h>
> #include <wtf/Optional.h>
> #include <wtf/SynchronizedFixedQueue.h>
>+#include <wtf/WeakPtr.h>
> #include <wtf/WorkQueue.h>
> #include <wtf/text/TextStream.h>
> 
>@@ -39,7 +40,7 @@ class BitmapImage;
> class GraphicsContext;
> class ImageDecoder;
> 
>-class ImageSource : public ThreadSafeRefCounted<ImageSource> {
>+class ImageSource : public ThreadSafeRefCounted<ImageSource>, public CanMakeWeakPtr<ImageSource> {
>     friend class BitmapImage;
> public:
>     ~ImageSource();
>@@ -142,6 +143,7 @@ private:
>     void decodedSizeIncreased(unsigned decodedSize);
>     void decodedSizeDecreased(unsigned decodedSize);
>     void decodedSizeReset(unsigned decodedSize);
>+    void encodedDataStatusChanged(EncodedDataStatus);
> 
>     void setNativeImage(NativeImagePtr&&);
>     void cacheMetadataAtIndex(size_t, SubsamplingLevel, DecodingStatus = DecodingStatus::Invalid);
>diff --git a/Source/WebCore/platform/graphics/avfoundation/objc/ImageDecoderAVFObjC.h b/Source/WebCore/platform/graphics/avfoundation/objc/ImageDecoderAVFObjC.h
>index 60ac34154f3591afa3eaf6f2df382a88bf55e3df..4a2dc492a8adbeb09cd0746bbb64ab1ca990aa7a 100644
>--- a/Source/WebCore/platform/graphics/avfoundation/objc/ImageDecoderAVFObjC.h
>+++ b/Source/WebCore/platform/graphics/avfoundation/objc/ImageDecoderAVFObjC.h
>@@ -35,7 +35,6 @@
> 
> OBJC_CLASS AVAssetTrack;
> OBJC_CLASS AVAssetReader;
>-OBJC_CLASS AVAssetReaderTrackOutput;
> OBJC_CLASS AVURLAsset;
> OBJC_CLASS WebCoreSharedBufferResourceLoaderDelegate;
> typedef struct opaqueCMSampleBuffer* CMSampleBufferRef;
>@@ -66,6 +65,7 @@ public:
> 
>     const String& mimeType() const { return m_mimeType; }
> 
>+    void setEncodedDataStatusChangeCallback(WTF::Function<void(EncodedDataStatus)>&&) final;
>     EncodedDataStatus encodedDataStatus() const final;
>     IntSize size() const final;
>     size_t frameCount() const final;
>@@ -109,6 +109,7 @@ private:
>     void setTrack(AVAssetTrack *);
> 
>     const ImageDecoderAVFObjCSample* sampleAtIndex(size_t) const;
>+    bool sampleIsComplete(const ImageDecoderAVFObjCSample&) const;
> 
>     String m_mimeType;
>     String m_uti;
>@@ -118,6 +119,7 @@ private:
>     RetainPtr<VTImageRotationSessionRef> m_rotationSession;
>     RetainPtr<CVPixelBufferPoolRef> m_rotationPool;
>     Ref<WebCoreDecompressionSession> m_decompressionSession;
>+    WTF::Function<void(EncodedDataStatus)> m_encodedDataStatusChangedCallback;
> 
>     SampleMap m_sampleData;
>     DecodeOrderSampleMap::iterator m_cursor;
>diff --git a/Source/WebCore/platform/graphics/avfoundation/objc/ImageDecoderAVFObjC.mm b/Source/WebCore/platform/graphics/avfoundation/objc/ImageDecoderAVFObjC.mm
>index ce8dab2822258900bbcb72cadc90670ff4d28c32..17dfd7a8f2c276c08db1f7f8aff7eb6d6d0dd2ae 100644
>--- a/Source/WebCore/platform/graphics/avfoundation/objc/ImageDecoderAVFObjC.mm
>+++ b/Source/WebCore/platform/graphics/avfoundation/objc/ImageDecoderAVFObjC.mm
>@@ -51,6 +51,7 @@
> #import <wtf/MainThread.h>
> #import <wtf/MediaTime.h>
> #import <wtf/NeverDestroyed.h>
>+#import <wtf/Optional.h>
> #import <wtf/SoftLinking.h>
> #import <wtf/Vector.h>
> 
>@@ -63,7 +64,7 @@
> SOFT_LINK_FRAMEWORK_OPTIONAL(AVFoundation)
> SOFT_LINK_CLASS_OPTIONAL(AVFoundation, AVURLAsset)
> SOFT_LINK_CLASS_OPTIONAL(AVFoundation, AVAssetReader)
>-SOFT_LINK_CLASS_OPTIONAL(AVFoundation, AVAssetReaderTrackOutput)
>+SOFT_LINK_CLASS_OPTIONAL(AVFoundation, AVAssetReaderSampleReferenceOutput)
> SOFT_LINK_CONSTANT_MAY_FAIL(AVFoundation, AVMediaCharacteristicVisual, NSString *)
> SOFT_LINK_CONSTANT_MAY_FAIL(AVFoundation, AVURLAssetReferenceRestrictionsKey, NSString *)
> SOFT_LINK_CONSTANT_MAY_FAIL(AVFoundation, AVURLAssetUsesNoPersistentCacheKey, NSString *)
>@@ -81,6 +82,7 @@ @interface WebCoreSharedBufferResourceLoaderDelegate : NSObject<AVAssetResourceL
>     Vector<RetainPtr<AVAssetResourceLoadingRequest>> _requests;
>     Lock _dataLock;
> }
>+@property (readonly) NSData* data;
> - (id)initWithParent:(WebCore::ImageDecoderAVFObjC*)parent;
> - (void)setExpectedContentSize:(long long)expectedContentSize;
> - (void)updateData:(NSData *)data complete:(BOOL)complete;
>@@ -100,6 +102,11 @@ - (id)initWithParent:(WebCore::ImageDecoderAVFObjC*)parent
>     return self;
> }
> 
>+- (NSData*)data
>+{
>+    return _data.get();
>+}
>+
> - (void)setExpectedContentSize:(long long)expectedContentSize
> {
>     LockHolder holder { _dataLock };
>@@ -285,6 +292,36 @@ public:
>         m_hasAlpha = alphaInfo != kCGImageAlphaNone && alphaInfo != kCGImageAlphaNoneSkipLast && alphaInfo != kCGImageAlphaNoneSkipFirst;
>     }
> 
>+    struct ByteRange {
>+        size_t byteOffset { 0 };
>+        size_t byteLength { 0 };
>+    };
>+
>+    Optional<ByteRange> byteRange() const
>+    {
>+        if (PAL::CMSampleBufferGetDataBuffer(m_sample.get())
>+            || PAL::CMSampleBufferGetImageBuffer(m_sample.get())
>+            || !PAL::CMSampleBufferDataIsReady(m_sample.get()))
>+            return WTF::nullopt;
>+
>+        CFNumberRef byteOffsetCF = (CFNumberRef)PAL::CMGetAttachment(m_sample.get(), PAL::kCMSampleBufferAttachmentKey_SampleReferenceByteOffset, nullptr);
>+        if (!byteOffsetCF || CFGetTypeID(byteOffsetCF) != CFNumberGetTypeID())
>+            return WTF::nullopt;
>+
>+        int64_t byteOffset { 0 };
>+        if (!CFNumberGetValue(byteOffsetCF, kCFNumberSInt64Type, &byteOffset))
>+            return WTF::nullopt;
>+
>+        CMItemCount sizeArrayEntries = 0;
>+        PAL::CMSampleBufferGetSampleSizeArray(m_sample.get(), 0, nullptr, &sizeArrayEntries);
>+        if (sizeArrayEntries != 1)
>+            return WTF::nullopt;
>+
>+        size_t singleSizeEntry;
>+        PAL::CMSampleBufferGetSampleSizeArray(m_sample.get(), 1, &singleSizeEntry, nullptr);
>+        return {{static_cast<size_t>(byteOffset), singleSizeEntry}};
>+    }
>+
>     SampleFlags flags() const override
>     {
>         return (SampleFlags)(MediaSampleAVFObjC::flags() | (m_hasAlpha ? HasAlpha : 0));
>@@ -387,19 +424,22 @@ void ImageDecoderAVFObjC::readSamples()
>         return;
> 
>     auto assetReader = adoptNS([allocAVAssetReaderInstance() initWithAsset:m_asset.get() error:nil]);
>-    auto assetReaderOutput = adoptNS([allocAVAssetReaderTrackOutputInstance() initWithTrack:m_track.get() outputSettings:nil]);
>+    auto referenceOutput = adoptNS([allocAVAssetReaderSampleReferenceOutputInstance() initWithTrack:m_track.get()]);
> 
>-    assetReaderOutput.get().alwaysCopiesSampleData = NO;
>-    [assetReader addOutput:assetReaderOutput.get()];
>+    referenceOutput.get().alwaysCopiesSampleData = NO;
>+    [assetReader addOutput:referenceOutput.get()];
>     [assetReader startReading];
> 
>-    while (auto sampleBuffer = adoptCF([assetReaderOutput copyNextSampleBuffer])) {
>+    while (auto sampleBuffer = adoptCF([referenceOutput copyNextSampleBuffer])) {
>         // NOTE: Some samples emitted by the AVAssetReader simply denote the boundary of edits
>         // and do not carry media data.
>         if (!(PAL::CMSampleBufferGetNumSamples(sampleBuffer.get())))
>             continue;
>         m_sampleData.addSample(ImageDecoderAVFObjCSample::create(WTFMove(sampleBuffer)).get());
>     }
>+
>+    if (m_encodedDataStatusChangedCallback)
>+        m_encodedDataStatusChangedCallback(encodedDataStatus());
> }
> 
> void ImageDecoderAVFObjC::readTrackMetadata()
>@@ -500,11 +540,20 @@ void ImageDecoderAVFObjC::setTrack(AVAssetTrack *track)
>     }];
> }
> 
>+void ImageDecoderAVFObjC::setEncodedDataStatusChangeCallback(WTF::Function<void(EncodedDataStatus)>&& callback)
>+{
>+    m_encodedDataStatusChangedCallback = WTFMove(callback);
>+}
>+
> EncodedDataStatus ImageDecoderAVFObjC::encodedDataStatus() const
> {
>-    if (m_sampleData.empty())
>-        return EncodedDataStatus::Unknown;
>-    return EncodedDataStatus::Complete;
>+    if (!m_sampleData.empty())
>+        return EncodedDataStatus::Complete;
>+    if (m_size)
>+        return EncodedDataStatus::SizeAvailable;
>+    if (m_track)
>+        return EncodedDataStatus::TypeAvailable;
>+    return EncodedDataStatus::Unknown;
> }
> 
> IntSize ImageDecoderAVFObjC::size() const
>@@ -548,7 +597,7 @@ bool ImageDecoderAVFObjC::frameIsCompleteAtIndex(size_t index) const
>     if (!sampleData)
>         return false;
> 
>-    return PAL::CMSampleBufferDataIsReady(sampleData->sampleBuffer());
>+    return sampleIsComplete(*sampleData);
> }
> 
> ImageOrientation ImageDecoderAVFObjC::frameOrientationAtIndex(size_t) const
>@@ -615,11 +664,44 @@ NativeImagePtr ImageDecoderAVFObjC::createFrameImageAtIndex(size_t index, Subsam
>         if (decodeTime < m_cursor->second->decodeTime())
>             return nullptr;
> 
>-        auto cursorSample = toSample(m_cursor)->sampleBuffer();
>+        auto cursorSample = toSample(m_cursor);
>         if (!cursorSample)
>+            return nullptr;
>+
>+        if (!sampleIsComplete(*cursorSample))
>+            return nullptr;
>+
>+        if (auto byteRange = cursorSample->byteRange()) {
>+            auto& byteRangeValue = byteRange.value();
>+            auto* data = m_loader.get().data;
>+            CMBlockBufferCustomBlockSource source {
>+                0,
>+                nullptr,
>+                [](void* refcon, void*, size_t) {
>+                    [(id)refcon release];
>+                },
>+                [data retain]
>+            };
>+            CMBlockBufferRef rawBlockBuffer = nullptr;
>+            if (noErr != PAL::CMBlockBufferCreateWithMemoryBlock(kCFAllocatorDefault, const_cast<void*>(data.bytes), data.length, nullptr, &source, byteRangeValue.byteOffset, byteRangeValue.byteLength, 0, &rawBlockBuffer))
>+                return nullptr;
>+
>+            if (!rawBlockBuffer)
>+                return nullptr;
>+
>+            if (noErr != PAL::CMSampleBufferSetDataBuffer(cursorSample->sampleBuffer(), rawBlockBuffer))
>+                return nullptr;
>+            CFRelease(rawBlockBuffer);
>+
>+            PAL::CMRemoveAttachment(cursorSample->sampleBuffer(), PAL::kCMSampleBufferAttachmentKey_SampleReferenceByteOffset);
>+            PAL::CMRemoveAttachment(cursorSample->sampleBuffer(), PAL::kCMSampleBufferAttachmentKey_SampleReferenceURL);
>+        }
>+
>+        auto cursorSampleBuffer = cursorSample->sampleBuffer();
>+        if (!cursorSampleBuffer)
>             break;
> 
>-        if (!storeSampleBuffer(cursorSample))
>+        if (!storeSampleBuffer(cursorSampleBuffer))
>             break;
> 
>         advanceCursor();
>@@ -679,6 +761,16 @@ const ImageDecoderAVFObjCSample* ImageDecoderAVFObjC::sampleAtIndex(size_t index
>     return toSample(iter);
> }
> 
>+bool ImageDecoderAVFObjC::sampleIsComplete(const ImageDecoderAVFObjCSample& sample) const
>+{
>+    if (auto byteRange = sample.byteRange()) {
>+        auto& byteRangeValue = byteRange.value();
>+        return byteRangeValue.byteOffset + byteRangeValue.byteLength <= m_loader.get().data.length;
>+    }
>+
>+    return PAL::CMSampleBufferDataIsReady(sample.sampleBuffer());
>+}
>+
> }
> 
> #endif
>diff --git a/LayoutTests/ChangeLog b/LayoutTests/ChangeLog
>index 6c934d69ca102d357ac6e6648f09e4b7166575bb..af55b81913bb8986d621d0eb836fbda5846fe665 100644
>--- a/LayoutTests/ChangeLog
>+++ b/LayoutTests/ChangeLog
>@@ -1,3 +1,15 @@
>+2019-01-04  Jer Noble  <jer.noble@apple.com>
>+
>+        Web Content process main thread blocked beneath ImageDecoderAVFObjC::readSamples for many seconds on imgur.com
>+        https://bugs.webkit.org/show_bug.cgi?id=191806
>+        <rdar://problem/46151477>
>+
>+        Reviewed by Dean Jackson.
>+
>+        * http/tests/images/mp4-partial-load-expected.txt: Added.
>+        * http/tests/images/mp4-partial-load.html: Added.
>+        * platform/win/http/tests/mp4-partial-load-expected.txt: Added.
>+
> 2019-01-04  Chris Fleizach  <cfleizach@apple.com>
> 
>         AX: String check: "Rule" does not reflect the meaning of the <hr> html tag
>diff --git a/LayoutTests/http/tests/images/mp4-partial-load-expected.txt b/LayoutTests/http/tests/images/mp4-partial-load-expected.txt
>new file mode 100644
>index 0000000000000000000000000000000000000000..afc08ad0e2bdf2e7cc41cdc8a7317c50a40d7dd5
>--- /dev/null
>+++ b/LayoutTests/http/tests/images/mp4-partial-load-expected.txt
>@@ -0,0 +1,10 @@
>+This tests that an mp4 loaded in a video tag does not cause a hang when the load stalls.
>+
>+On success, you will see a series of "PASS" messages, followed by "TEST COMPLETE".
>+
>+
>+PASS img.naturalWidth became 320
>+PASS successfullyParsed is true
>+
>+TEST COMPLETE
>+
>diff --git a/LayoutTests/http/tests/images/mp4-partial-load.html b/LayoutTests/http/tests/images/mp4-partial-load.html
>new file mode 100644
>index 0000000000000000000000000000000000000000..3eb746ac71a29df6afda416955e9822de48a716b
>--- /dev/null
>+++ b/LayoutTests/http/tests/images/mp4-partial-load.html
>@@ -0,0 +1,63 @@
>+<!DOCTYPE html>
>+<html>
>+<head>
>+    <title>mp4-partial-load</title>
>+    <script src="../../resources/js-test-pre.js"></script>
>+    <script>
>+    jsTestIsAsync = true;
>+    if (window.internals && window.testRunner) {
>+        testRunner.waitUntilDone();
>+        testRunner.dumpAsText();
>+        internals.clearMemoryCache();
>+    }
>+
>+    function loadAndStall()
>+    {
>+        return "http://127.0.0.1:8000/resources/load-and-stall.php";
>+    }
>+
>+    function mp4Image()
>+    {
>+        return "?name=../../../media/content/test.mp4&mimeType=video%2Fmp4";
>+    }
>+
>+    function endTest()
>+    {
>+        if (window.timeoutID)
>+            clearTimeout(timeoutID);
>+        finishJSTest();
>+    }
>+
>+    function errorReceived()
>+    {
>+        testFailed('Received an erorr loading image');
>+        endTest();
>+    }
>+
>+    function timedOut()
>+    {
>+        testFailed('Took too long');
>+        endTest();
>+    }
>+
>+    function runTest()
>+    {
>+        description('This tests that an mp4 loaded in a video tag does not cause a hang when the load stalls.')
>+
>+        setTimeout(() => {
>+            img = document.querySelector('img');
>+            img.addEventListener("error", errorReceived);
>+            img.src = loadAndStall() + mp4Image() + "&stallAt=15799&stallFor=60";
>+            window.timeoutID = setTimeout(timedOut, 5000);
>+            shouldBecomeEqual('img.naturalWidth', '320', endTest);
>+        });
>+    }
>+
>+    window.addEventListener('load', runTest, {once: true});
>+    </script>
>+    <script src="../../resources/js-test-post.js"></script>
>+</head>
>+<body>
>+    <img>
>+</body>
>+</html>
>diff --git a/LayoutTests/platform/win/http/tests/images/mp4-partial-load-expected.txt b/LayoutTests/platform/win/http/tests/images/mp4-partial-load-expected.txt
>new file mode 100644
>index 0000000000000000000000000000000000000000..db52d86dbe23e3f0bd0e4cb36c67b4a9a5908f8f
>--- /dev/null
>+++ b/LayoutTests/platform/win/http/tests/images/mp4-partial-load-expected.txt
>@@ -0,0 +1,10 @@
>+This tests that an mp4 loaded in a video tag does not cause a hang when the load stalls.
>+
>+On success, you will see a series of "PASS" messages, followed by "TEST COMPLETE".
>+
>+
>+FAIL Took too long
>+PASS successfullyParsed is true
>+
>+TEST COMPLETE
>+

Actions: View | Formatted Diff | Diff

Attachments on bug 191806: 356298 | 356302 | 356303 | 356306 | 356309 | 356311 | 356317 | 356407 | 356424 | 356432 | 356433 | 358285 | 358298 | 358332