WebKit Bugzilla
New
Browse
Search+
Log In
×
Sign in with GitHub
or
Remember my login
Create Account
·
Forgot Password
Forgotten password account recovery
[patch]
Patch
bug-236099-20220204164411.patch (text/plain), 97.93 KB, created by
Kimmo Kinnunen
on 2022-02-04 06:44:13 PST
(
hide
)
Description:
Patch
Filename:
MIME Type:
Creator:
Kimmo Kinnunen
Created:
2022-02-04 06:44:13 PST
Size:
97.93 KB
patch
obsolete
>Subversion Revision: 289040 >diff --git a/Source/WebCore/ChangeLog b/Source/WebCore/ChangeLog >index d23c306d6768e3848de43165f5f3b9828d511d44..8e4b3ce2f34d6ae8712390d6e34d1658a29759f8 100644 >--- a/Source/WebCore/ChangeLog >+++ b/Source/WebCore/ChangeLog >@@ -1,3 +1,60 @@ >+2022-02-04 Kimmo Kinnunen <kkinnunen@apple.com> >+ >+ Introduce a RemoteMediaSampleProxy to represent captured video frames used in Media Streams and present in GPUP >+ https://bugs.webkit.org/show_bug.cgi?id=236099 >+ >+ Reviewed by NOBODY (OOPS!). >+ >+ Add VideoFrame : public MediaStream, base class for Media Stream video frames. >+ Currently inherits from MediaSample, but later will remove the MediaSample. >+ >+ The WebCore::VideoFrame is intended to be used in various entry-points that >+ currently are currently typed as: >+ void videoSampleAvailable(MediaSample&, VideoSampleMetadata, ...). >+ The new class WebKit::RemoteVideoFrameProxy : public VideoFrame can be passed to these APIs. >+ The API entrypoints are not yet changed in order to partially roll the changes in. >+ Later on, the entrypoints will be similar to: >+ void videoSampleAvailable(VideoFrame&, VideoSampleMetadata, ...). >+ >+ Removes MediaSample::setTrackID as that is not used. >+ Changes MediaSample::platformSample() to const, so that it can be called from >+ const context. >+ >+ No new tests, refactor. >+ >+ * Headers.cmake: >+ * Sources.txt: >+ * WebCore.xcodeproj/project.pbxproj: >+ * platform/MediaSample.h: >+ * platform/VideoFrame.cpp: Added. >+ (WebCore::VideoFrame::decodeTime const): >+ (WebCore::VideoFrame::duration const): >+ (WebCore::VideoFrame::trackID const): >+ (WebCore::VideoFrame::sizeInBytes const): >+ (WebCore::VideoFrame::presentationSize const): >+ (WebCore::VideoFrame::offsetTimestampsBy): >+ (WebCore::VideoFrame::setTimestamps): >+ (WebCore::VideoFrame::isDivisable const): >+ (WebCore::VideoFrame::divide): >+ (WebCore::VideoFrame::createNonDisplayingCopy const): >+ (WebCore::VideoFrame::flags const): >+ (WebCore::VideoFrame::byteRange const): >+ (WebCore::VideoFrame::dump const): >+ * platform/VideoFrame.h: Added. >+ * platform/graphics/avfoundation/objc/LocalSampleBufferDisplayLayer.mm: >+ (WebCore::LocalSampleBufferDisplayLayer::enqueueSampleBuffer): >+ * platform/graphics/avfoundation/objc/MediaPlayerPrivateMediaStreamAVFObjC.mm: >+ (WebCore::MediaPlayerPrivateMediaStreamAVFObjC::updateCurrentFrameImage): >+ * platform/graphics/avfoundation/objc/MediaSampleAVFObjC.h: >+ (isType): >+ * platform/graphics/avfoundation/objc/MediaSampleAVFObjC.mm: >+ (WebCore::MediaSampleAVFObjC::platformSample const): >+ * platform/graphics/gstreamer/MediaSampleGStreamer.cpp: >+ (WebCore::MediaSampleGStreamer::platformSample const): >+ * platform/graphics/gstreamer/MediaSampleGStreamer.h: >+ * platform/mock/mediasource/MockSourceBufferPrivate.cpp: >+ (WebCore::MockMediaSample::platformSample const): >+ > 2022-02-03 Carlos Garcia Campos <cgarcia@igalia.com> > > Unreviewed. Fix GTK make distcheck >diff --git a/Source/WebKit/ChangeLog b/Source/WebKit/ChangeLog >index 8bc735c50407e2930639f945a6abc1bcc82963f2..33d87d86fed9afd7a62bcd60c6ce6579a4d8a604 100644 >--- a/Source/WebKit/ChangeLog >+++ b/Source/WebKit/ChangeLog >@@ -1,3 +1,103 @@ >+2022-02-04 Kimmo Kinnunen <kkinnunen@apple.com> >+ >+ Introduce a RemoteMediaSampleProxy to represent captured video frames used in Media Streams and present in GPUP >+ https://bugs.webkit.org/show_bug.cgi?id=236099 >+ >+ Reviewed by NOBODY (OOPS!). >+ >+ >+ WebContent process: >+ Add WebKit::RemoteVideoFrameProxy, WebContent process side proxy object that >+ WebCore::MediaPlayerPrivateMediaStreamAVFObjC can use. >+ Missing: RemoteVideoFrameProxy can be sent to the GPUP and the the SampleBufferDisplayLayer. So >+ the captured content shows in compositing through video element. At the moment >+ MediaPlayerPrivateMediaStreamAVFObjC cannot paint RemoteVideoFrameProxy to the >+ document CSS painted content, Context2D or WebGL canvas. >+ >+ GPUP: >+ RemoteVideoFrameObjectHeap is a mapping >+ RemoteVideoFrameIdentifier -> MediaSample. Currently all objects >+ map to normal MediaSampleAVFObjC. >+ >+ Add an example implementation where a remote MediaSample is returned by IPC: >+ RemoteGraphicsContextGL::paintCompositedResultsToMediaSample(). >+ >+ * CMakeLists.txt: >+ * DerivedSources-input.xcfilelist: >+ * DerivedSources-output.xcfilelist: >+ * DerivedSources.make: >+ * GPUProcess/GPUConnectionToWebProcess.cpp: >+ (WebKit::GPUConnectionToWebProcess::GPUConnectionToWebProcess): >+ (WebKit::GPUConnectionToWebProcess::didClose): >+ (WebKit::GPUConnectionToWebProcess::videoFrameObjectHeap const): >+ * GPUProcess/GPUConnectionToWebProcess.h: >+ * GPUProcess/graphics/RemoteGraphicsContextGL.cpp: >+ (WebKit::RemoteGraphicsContextGL::RemoteGraphicsContextGL): >+ (WebKit::RemoteGraphicsContextGL::paintCompositedResultsToMediaSample): >+ * GPUProcess/graphics/RemoteGraphicsContextGL.h: >+ * GPUProcess/graphics/RemoteGraphicsContextGL.messages.in: >+ * GPUProcess/media/RemoteVideoFrameObjectHeap.cpp: Added. >+ (WebKit::RemoteVideoFrameObjectHeap::create): >+ (WebKit::RemoteVideoFrameObjectHeap::RemoteVideoFrameObjectHeap): >+ (WebKit::RemoteVideoFrameObjectHeap::~RemoteVideoFrameObjectHeap): >+ (WebKit::RemoteVideoFrameObjectHeap::add): >+ (WebKit::RemoteVideoFrameObjectHeap::addImpl): >+ (WebKit::RemoteVideoFrameObjectHeap::remove): >+ (WebKit::RemoteVideoFrameObjectHeap::get): >+ (WebKit::RemoteVideoFrameObjectHeap::stopListeningForIPC): >+ (WebKit::RemoteVideoFrameObjectHeap::releaseVideoFrame): >+ * GPUProcess/media/RemoteVideoFrameObjectHeap.h: Added. >+ * GPUProcess/media/RemoteVideoFrameObjectHeap.messages.in: Copied from Source/WebKit/GPUProcess/webrtc/RemoteSampleBufferDisplayLayer.messages.in. >+ * GPUProcess/webrtc/RemoteSampleBufferDisplayLayer.cpp: >+ (WebKit::RemoteSampleBufferDisplayLayer::create): >+ (WebKit::RemoteSampleBufferDisplayLayer::RemoteSampleBufferDisplayLayer): >+ (WebKit::RemoteSampleBufferDisplayLayer::enqueueSample): >+ (WebKit::RemoteSampleBufferDisplayLayer::enqueueSampleCV): >+ * GPUProcess/webrtc/RemoteSampleBufferDisplayLayer.h: >+ * GPUProcess/webrtc/RemoteSampleBufferDisplayLayer.messages.in: >+ * GPUProcess/webrtc/RemoteSampleBufferDisplayLayerManager.cpp: >+ (WebKit::RemoteSampleBufferDisplayLayerManager::createLayer): >+ * Scripts/webkit/messages.py: >+ (types_that_cannot_be_forward_declared): >+ * Shared/mac/MediaFormatReader/MediaSampleByteRange.cpp: >+ (WebKit::MediaSampleByteRange::platformSample const): >+ * Shared/mac/MediaFormatReader/MediaSampleByteRange.h: >+ * Sources.txt: >+ * WebKit.xcodeproj/project.pbxproj: >+ * WebProcess/GPU/graphics/RemoteGraphicsContextGLProxy.cpp: >+ (WebKit::RemoteGraphicsContextGLProxy::paintCompositedResultsToMediaSample): >+ * WebProcess/GPU/graphics/RemoteGraphicsContextGLProxy.h: >+ * WebProcess/GPU/graphics/cocoa/RemoteGraphicsContextGLProxyCocoa.mm: >+ * WebProcess/GPU/media/RemoteVideoFrameIdentifier.h: Added. >+ * WebProcess/GPU/media/RemoteVideoFrameProxy.cpp: Added. >+ (WebKit::RemoteVideoFrameProxy::create): >+ (WebKit::RemoteVideoFrameProxy::properties): >+ (WebKit::RemoteVideoFrameProxy::RemoteVideoFrameProxy): >+ (WebKit::RemoteVideoFrameProxy::~RemoteVideoFrameProxy): >+ (WebKit::RemoteVideoFrameProxy::identifier const): >+ (WebKit::RemoteVideoFrameProxy::presentationTime const): >+ (WebKit::RemoteVideoFrameProxy::platformSample const): >+ (WebKit::RemoteVideoFrameProxy::videoRotation const): >+ (WebKit::RemoteVideoFrameProxy::videoMirrored const): >+ (WebKit::RemoteVideoFrameProxy::videoPixelFormat const): >+ (WebKit::RemoteVideoFrameProxy::videoFrame const): >+ (WebKit::RemoteVideoFrameProxy::disconnectGPUProcessIfNeeded const): >+ (WebKit::RemoteVideoFrameProxy::waitForWasInitialized const): >+ (WebKit::RemoteVideoFrameProxy::gpuProcessConnectionDidClose): >+ (WebKit::RemoteVideoFrameProxy::wasInitialized): >+ * WebProcess/GPU/media/RemoteVideoFrameProxy.h: Added. >+ (WebKit::RemoteVideoFrameProxy::Properties::encode const): >+ (WebKit::RemoteVideoFrameProxy::Properties::decode): >+ (isType): >+ * WebProcess/GPU/media/RemoteVideoFrameProxy.messages.in: Added. >+ * WebProcess/GPU/webrtc/SampleBufferDisplayLayer.cpp: >+ (WebKit::SampleBufferDisplayLayer::SampleBufferDisplayLayer): >+ (WebKit::SampleBufferDisplayLayer::~SampleBufferDisplayLayer): >+ (WebKit::SampleBufferDisplayLayer::enqueueSample): >+ (WebKit::SampleBufferDisplayLayer::gpuProcessConnectionDidClose): >+ (WebKit::SampleBufferDisplayLayer::disconnectGPUProcessConnectionIfNeeded): >+ * WebProcess/GPU/webrtc/SampleBufferDisplayLayer.h: >+ > 2022-02-03 Youenn Fablet <youenn@apple.com> > > SampleBufferDiplayLayer should not need to create IOSurfaces >diff --git a/Source/WebCore/Headers.cmake b/Source/WebCore/Headers.cmake >index f8ff3e616c30183268aa5e5282d15ae0c16c76ff..6c693b234dacad149b9d978ee5ed765a0b165f51 100644 >--- a/Source/WebCore/Headers.cmake >+++ b/Source/WebCore/Headers.cmake >@@ -1341,6 +1341,7 @@ set(WebCore_PRIVATE_FRAMEWORK_HEADERS > platform/UserAgent.h > platform/UserInterfaceLayoutDirection.h > platform/ValidationBubble.h >+ platform/VideoFrame.h > platform/VideoFrameMetadata.h > platform/VideoSampleMetadata.h > platform/WebGLStateTracker.h >diff --git a/Source/WebCore/Sources.txt b/Source/WebCore/Sources.txt >index 51bfab8880b9d6896b3078b0930dbe763711fe09..1768f2df134ba52496cf9267e4622709e1279a0c 100644 >--- a/Source/WebCore/Sources.txt >+++ b/Source/WebCore/Sources.txt >@@ -1926,6 +1926,7 @@ platform/ThreadGlobalData.cpp > platform/ThreadTimers.cpp > platform/Timer.cpp > platform/UserActivity.cpp >+platform/VideoFrame.cpp > platform/WebCoreCrossThreadCopier.cpp > platform/WebGLStateTracker.cpp > platform/Widget.cpp >diff --git a/Source/WebCore/WebCore.xcodeproj/project.pbxproj b/Source/WebCore/WebCore.xcodeproj/project.pbxproj >index 1073322436f4ec25281ea18bf7d1810bb37d930b..98e121968a5f5811328f4eff05d0bf422284d7e6 100644 >--- a/Source/WebCore/WebCore.xcodeproj/project.pbxproj >+++ b/Source/WebCore/WebCore.xcodeproj/project.pbxproj >@@ -2402,6 +2402,7 @@ > 7B10339E2549721700C8C1AC /* GraphicsContextGLCV.h in Headers */ = {isa = PBXBuildFile; fileRef = 7B10339D2549720100C8C1AC /* GraphicsContextGLCV.h */; settings = {ATTRIBUTES = (Private, ); }; }; > 7B10339F2549721E00C8C1AC /* GraphicsContextGLCVCocoa.h in Headers */ = {isa = PBXBuildFile; fileRef = 7B10339C2549720100C8C1AC /* GraphicsContextGLCVCocoa.h */; settings = {ATTRIBUTES = (Private, ); }; }; > 7B4410E127958FC5000B4F8A /* MediaSampleVideoFrame.h in Headers */ = {isa = PBXBuildFile; fileRef = 7B4410DF27958FC4000B4F8A /* MediaSampleVideoFrame.h */; settings = {ATTRIBUTES = (Private, ); }; }; >+ 7B5A3DAF27ABF8C5006C6F97 /* VideoFrame.h in Headers */ = {isa = PBXBuildFile; fileRef = 7B5A3DAD27ABF8C4006C6F97 /* VideoFrame.h */; settings = {ATTRIBUTES = (Private, ); }; }; > 7B6094DD270C44BD0064835C /* ANGLEUtilities.h in Headers */ = {isa = PBXBuildFile; fileRef = 7BB34A1625345CB200029D08 /* ANGLEUtilities.h */; settings = {ATTRIBUTES = (Private, ); }; }; > 7B6DC81925712E9200380C70 /* GraphicsContextGLIOSurfaceSwapChain.h in Headers */ = {isa = PBXBuildFile; fileRef = 7B6DC81725712E9200380C70 /* GraphicsContextGLIOSurfaceSwapChain.h */; settings = {ATTRIBUTES = (Private, ); }; }; > 7B720B0C2743DBA70008ABED /* GraphicsLayerContentsDisplayDelegate.h in Headers */ = {isa = PBXBuildFile; fileRef = 7B720B0A2743DBA60008ABED /* GraphicsLayerContentsDisplayDelegate.h */; settings = {ATTRIBUTES = (Private, ); }; }; >@@ -11504,6 +11505,8 @@ > 7B1619102719880E00C40EAC /* WebProcessGraphicsContextGLCocoa.mm */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.cpp.objcpp; path = WebProcessGraphicsContextGLCocoa.mm; sourceTree = "<group>"; }; > 7B4410DF27958FC4000B4F8A /* MediaSampleVideoFrame.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = MediaSampleVideoFrame.h; sourceTree = "<group>"; }; > 7B4410E227958FDE000B4F8A /* MediaSampleVideoFrameAVF.mm */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.cpp.objcpp; path = MediaSampleVideoFrameAVF.mm; sourceTree = "<group>"; }; >+ 7B5A3DAB27ABF8C4006C6F97 /* VideoFrame.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = VideoFrame.cpp; sourceTree = "<group>"; }; >+ 7B5A3DAD27ABF8C4006C6F97 /* VideoFrame.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = VideoFrame.h; sourceTree = "<group>"; }; > 7B6DC81525712E9100380C70 /* GraphicsContextGLIOSurfaceSwapChain.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = GraphicsContextGLIOSurfaceSwapChain.cpp; sourceTree = "<group>"; }; > 7B6DC81725712E9200380C70 /* GraphicsContextGLIOSurfaceSwapChain.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = GraphicsContextGLIOSurfaceSwapChain.h; sourceTree = "<group>"; }; > 7B720B0A2743DBA60008ABED /* GraphicsLayerContentsDisplayDelegate.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = GraphicsLayerContentsDisplayDelegate.h; sourceTree = "<group>"; }; >@@ -30121,6 +30124,8 @@ > A3AF9D8220325324006CAD06 /* UserAgent.h */, > 71C916071D1483A300ACA47D /* UserInterfaceLayoutDirection.h */, > 83C45B8D1DC2B67C008871BA /* ValidationBubble.h */, >+ 7B5A3DAB27ABF8C4006C6F97 /* VideoFrame.cpp */, >+ 7B5A3DAD27ABF8C4006C6F97 /* VideoFrame.h */, > 41DEEFB42719BA1900CB8D74 /* VideoFrameMetadata.h */, > 416049312743B0E700A86FA0 /* VideoSampleMetadata.h */, > 515F79511CFCA3C700CCED93 /* WebCoreCrossThreadCopier.cpp */, >@@ -37787,6 +37792,7 @@ > 0F1A0C38229A481800D37ADB /* VelocityData.h in Headers */, > BCA257151293C010007A263D /* VerticalPositionCache.h in Headers */, > CD3EEF3B25799FA1006563BB /* VideoConfiguration.h in Headers */, >+ 7B5A3DAF27ABF8C5006C6F97 /* VideoFrame.h in Headers */, > 41DEEFB62719BA1A00CB8D74 /* VideoFrameMetadata.h in Headers */, > 41DEEFB02719B29000CB8D74 /* VideoFrameRequestCallback.h in Headers */, > 52B0D4C01C57FD660077CE53 /* VideoFullscreenChangeObserver.h in Headers */, >diff --git a/Source/WebCore/platform/MediaSample.h b/Source/WebCore/platform/MediaSample.h >index 1839972260f70e0073679791722625f56b253f08..b4fe716579ff8e7fc340d87cc892f1bbb4b0d8d4 100644 >--- a/Source/WebCore/platform/MediaSample.h >+++ b/Source/WebCore/platform/MediaSample.h >@@ -49,9 +49,10 @@ struct PlatformSample { > CMSampleBufferType, > GStreamerSampleType, > ByteRangeSampleType, >+ RemoteVideoFrameProxyType, // FIXME: To be removed when VideoFrame is not MediaSample. > } type; > union { >- MockSampleBox* mockSampleBox; >+ const MockSampleBox* mockSampleBox; > CMSampleBufferRef cmSampleBuffer; > GstSample* gstSample; > std::pair<MTPluginByteSourceRef, CMFormatDescriptionRef> byteRangeSample; >@@ -66,7 +67,6 @@ public: > virtual MediaTime decodeTime() const = 0; > virtual MediaTime duration() const = 0; > virtual AtomString trackID() const = 0; >- virtual void setTrackID(const String&) = 0; > virtual size_t sizeInBytes() const = 0; > virtual FloatSize presentationSize() const = 0; > virtual void offsetTimestampsBy(const MediaTime&) = 0; >@@ -90,7 +90,7 @@ public: > HasSyncInfo = 1 << 3, > }; > virtual SampleFlags flags() const = 0; >- virtual PlatformSample platformSample() = 0; >+ virtual PlatformSample platformSample() const = 0; > > struct ByteRange { > size_t byteOffset { 0 }; >diff --git a/Source/WebCore/platform/VideoFrame.cpp b/Source/WebCore/platform/VideoFrame.cpp >new file mode 100644 >index 0000000000000000000000000000000000000000..34e816ecd08cb95524d6aa2d71afcb9f0b91786f >--- /dev/null >+++ b/Source/WebCore/platform/VideoFrame.cpp >@@ -0,0 +1,112 @@ >+/* >+ * Copyright (C) 2022 Apple Inc. All rights reserved. >+ * >+ * Redistribution and use in source and binary forms, with or without >+ * modification, are permitted provided that the following conditions >+ * are met: >+ * 1. Redistributions of source code must retain the above copyright >+ * notice, this list of conditions and the following disclaimer. >+ * 2. Redistributions in binary form must reproduce the above copyright >+ * notice, this list of conditions and the following disclaimer in the >+ * documentation and/or other materials provided with the distribution. >+ * >+ * THIS SOFTWARE IS PROVIDED BY APPLE INC. AND ITS CONTRIBUTORS ``AS IS'' >+ * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, >+ * THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR >+ * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL APPLE INC. OR ITS CONTRIBUTORS >+ * BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR >+ * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF >+ * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS >+ * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN >+ * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) >+ * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF >+ * THE POSSIBILITY OF SUCH DAMAGE. >+ */ >+ >+#include "config.h" >+#include "VideoFrame.h" >+ >+#if ENABLE(MEDIA_STREAM) >+ >+namespace WebCore { >+ >+VideoFrame::VideoFrame() = default; >+ >+VideoFrame::~VideoFrame() = default; >+ >+MediaTime VideoFrame::decodeTime() const >+{ >+ ASSERT_NOT_REACHED(); >+ return { }; >+} >+ >+MediaTime VideoFrame::duration() const >+{ >+ ASSERT_NOT_REACHED(); >+ return { }; >+} >+ >+AtomString VideoFrame::trackID() const >+{ >+ ASSERT_NOT_REACHED(); >+ return { }; >+} >+ >+size_t VideoFrame::sizeInBytes() const >+{ >+ ASSERT_NOT_REACHED(); >+ return 0; >+} >+ >+WebCore::FloatSize VideoFrame::presentationSize() const >+{ >+ ASSERT_NOT_REACHED(); >+ return { }; >+} >+ >+void VideoFrame::offsetTimestampsBy(const MediaTime&) >+{ >+ ASSERT_NOT_REACHED(); >+} >+ >+void VideoFrame::setTimestamps(const MediaTime&, const MediaTime&) >+{ >+ ASSERT_NOT_REACHED(); >+} >+ >+bool VideoFrame::isDivisable() const >+{ >+ ASSERT_NOT_REACHED(); >+ return false; >+} >+ >+std::pair<RefPtr<WebCore::MediaSample>, RefPtr<WebCore::MediaSample>> VideoFrame::divide(const MediaTime&, UseEndTime) >+{ >+ ASSERT_NOT_REACHED(); >+ return { nullptr, nullptr }; >+} >+ >+Ref<WebCore::MediaSample> VideoFrame::createNonDisplayingCopy() const >+{ >+ CRASH(); >+} >+ >+MediaSample::SampleFlags VideoFrame::flags() const >+{ >+ return MediaSample::SampleFlags::None; >+} >+ >+std::optional<MediaSample::ByteRange> VideoFrame::byteRange() const >+{ >+ // FIXME: Remove from the base class. >+ ASSERT_NOT_REACHED(); >+ return std::nullopt; >+} >+ >+void VideoFrame::dump(PrintStream&) const >+{ >+} >+ >+} >+ >+#endif >diff --git a/Source/WebCore/platform/VideoFrame.h b/Source/WebCore/platform/VideoFrame.h >new file mode 100644 >index 0000000000000000000000000000000000000000..b2d95010d36eb62b9eae1b99837b8477141e05fc >--- /dev/null >+++ b/Source/WebCore/platform/VideoFrame.h >@@ -0,0 +1,64 @@ >+/* >+ * Copyright (C) 2022 Apple Inc. All rights reserved. >+ * >+ * Redistribution and use in source and binary forms, with or without >+ * modification, are permitted provided that the following conditions >+ * are met: >+ * 1. Redistributions of source code must retain the above copyright >+ * notice, this list of conditions and the following disclaimer. >+ * 2. Redistributions in binary form must reproduce the above copyright >+ * notice, this list of conditions and the following disclaimer in the >+ * documentation and/or other materials provided with the distribution. >+ * >+ * THIS SOFTWARE IS PROVIDED BY APPLE INC. AND ITS CONTRIBUTORS ``AS IS'' >+ * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, >+ * THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR >+ * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL APPLE INC. OR ITS CONTRIBUTORS >+ * BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR >+ * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF >+ * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS >+ * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN >+ * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) >+ * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF >+ * THE POSSIBILITY OF SUCH DAMAGE. >+ */ >+ >+#pragma once >+ >+#if ENABLE(MEDIA_STREAM) >+ >+#include "MediaSample.h" >+ >+namespace WebCore { >+ >+// A class representing a video frame from a decoder, capture source, or similar. >+// FIXME: Currently for implementation purposes inherts from MediaSample until capture code >+// stops referring to MediaSample >+class VideoFrame : public MediaSample { >+public: >+ WEBCORE_EXPORT ~VideoFrame(); >+ >+protected: >+ WEBCORE_EXPORT VideoFrame(); >+ >+private: >+ // FIXME: These are not intended to be used for these objects. >+ // WebCore::MediaSample overrides. >+ WEBCORE_EXPORT MediaTime decodeTime() const final; >+ WEBCORE_EXPORT MediaTime duration() const final; >+ WEBCORE_EXPORT AtomString trackID() const final; >+ WEBCORE_EXPORT size_t sizeInBytes() const final; >+ WEBCORE_EXPORT WebCore::FloatSize presentationSize() const final; >+ WEBCORE_EXPORT void offsetTimestampsBy(const MediaTime&) final; >+ WEBCORE_EXPORT void setTimestamps(const MediaTime&, const MediaTime&) final; >+ WEBCORE_EXPORT bool isDivisable() const final; >+ WEBCORE_EXPORT std::pair<RefPtr<MediaSample>, RefPtr<MediaSample>> divide(const MediaTime& presentationTime, UseEndTime = UseEndTime::DoNotUse) final; >+ WEBCORE_EXPORT Ref<WebCore::MediaSample> createNonDisplayingCopy() const final; >+ WEBCORE_EXPORT SampleFlags flags() const final; >+ WEBCORE_EXPORT std::optional<ByteRange> byteRange() const final; >+ WEBCORE_EXPORT void dump(PrintStream&) const final; >+}; >+ >+} >+ >+#endif >diff --git a/Source/WebCore/platform/graphics/avfoundation/objc/LocalSampleBufferDisplayLayer.mm b/Source/WebCore/platform/graphics/avfoundation/objc/LocalSampleBufferDisplayLayer.mm >index c126b93e8706f95530f6126f8ad25faf6222ad71..9bf4b0b8d5c481a534fd90cf16c9559fb6cd716a 100644 >--- a/Source/WebCore/platform/graphics/avfoundation/objc/LocalSampleBufferDisplayLayer.mm >+++ b/Source/WebCore/platform/graphics/avfoundation/objc/LocalSampleBufferDisplayLayer.mm >@@ -334,6 +334,9 @@ void LocalSampleBufferDisplayLayer::enqueueSample(MediaSample& sample) > void LocalSampleBufferDisplayLayer::enqueueSampleBuffer(MediaSample& sample) > { > ASSERT(!isMainThread()); >+ // FIXME: Local playback of RemoteVideoFrameProxy is not implemented. >+ if (!is<MediaSampleAVFObjC>(sample)) >+ return; > > auto sampleToEnqueue = sample.platformSample().sample.cmSampleBuffer; > auto now = MediaTime::createWithDouble(MonotonicTime::now().secondsSinceEpoch().value() + rendererLatency); >diff --git a/Source/WebCore/platform/graphics/avfoundation/objc/MediaPlayerPrivateMediaStreamAVFObjC.mm b/Source/WebCore/platform/graphics/avfoundation/objc/MediaPlayerPrivateMediaStreamAVFObjC.mm >index 2bbd6a4cb0d22d2df99775ee5ee27c78a20d5955..512696a13ac72ac88f80086562e311045c41ac65 100644 >--- a/Source/WebCore/platform/graphics/avfoundation/objc/MediaPlayerPrivateMediaStreamAVFObjC.mm >+++ b/Source/WebCore/platform/graphics/avfoundation/objc/MediaPlayerPrivateMediaStreamAVFObjC.mm >@@ -987,6 +987,10 @@ void MediaPlayerPrivateMediaStreamAVFObjC::updateCurrentFrameImage() > if (m_imagePainter.cgImage || !m_imagePainter.mediaSample) > return; > >+ // FIXME: Does not currently support RemoteVideoFrameProxy. >+ if (!is<MediaSampleAVFObjC>(*m_imagePainter.mediaSample)) >+ return; >+ > if (!m_imagePainter.pixelBufferConformer) > m_imagePainter.pixelBufferConformer = makeUnique<PixelBufferConformerCV>((__bridge CFDictionaryRef)@{ (__bridge NSString *)kCVPixelBufferPixelFormatTypeKey: @(kCVPixelFormatType_32BGRA) }); > >diff --git a/Source/WebCore/platform/graphics/avfoundation/objc/MediaSampleAVFObjC.h b/Source/WebCore/platform/graphics/avfoundation/objc/MediaSampleAVFObjC.h >index 9007dcca9dee10eebc3b31d6338060d7e9237056..70f8fba688df35ec86baeea41ec27f4d8b36e171 100644 >--- a/Source/WebCore/platform/graphics/avfoundation/objc/MediaSampleAVFObjC.h >+++ b/Source/WebCore/platform/graphics/avfoundation/objc/MediaSampleAVFObjC.h >@@ -54,13 +54,12 @@ public: > MediaTime duration() const override; > > AtomString trackID() const override { return m_id; } >- void setTrackID(const String& id) override { m_id = id; } > > size_t sizeInBytes() const override; > FloatSize presentationSize() const override; > > SampleFlags flags() const override; >- PlatformSample platformSample() override; >+ PlatformSample platformSample() const override; > std::optional<ByteRange> byteRange() const override; > WEBCORE_EXPORT void dump(PrintStream&) const override; > void offsetTimestampsBy(const MediaTime&) override; >@@ -97,3 +96,7 @@ protected: > }; > > } // namespace WebCore >+ >+SPECIALIZE_TYPE_TRAITS_BEGIN(WebCore::MediaSampleAVFObjC) >+ static bool isType(const WebCore::MediaSample& mediaSample) { return mediaSample.platformSample().type == WebCore::PlatformSample::CMSampleBufferType; } >+SPECIALIZE_TYPE_TRAITS_END() >diff --git a/Source/WebCore/platform/graphics/avfoundation/objc/MediaSampleAVFObjC.mm b/Source/WebCore/platform/graphics/avfoundation/objc/MediaSampleAVFObjC.mm >index 4a0fac61eb1ef6033ed297435daa926e9edeacd7..e61f506e4a0fb2bca47dac2633a3d9aa75a2fb97 100644 >--- a/Source/WebCore/platform/graphics/avfoundation/objc/MediaSampleAVFObjC.mm >+++ b/Source/WebCore/platform/graphics/avfoundation/objc/MediaSampleAVFObjC.mm >@@ -153,7 +153,7 @@ size_t MediaSampleAVFObjC::sizeInBytes() const > return PAL::CMSampleBufferGetTotalSampleSize(m_sample.get()); > } > >-PlatformSample MediaSampleAVFObjC::platformSample() >+PlatformSample MediaSampleAVFObjC::platformSample() const > { > PlatformSample sample = { PlatformSample::CMSampleBufferType, { .cmSampleBuffer = m_sample.get() } }; > return sample; >diff --git a/Source/WebCore/platform/graphics/gstreamer/MediaSampleGStreamer.cpp b/Source/WebCore/platform/graphics/gstreamer/MediaSampleGStreamer.cpp >index 9e7c089606a7b139a9a6238e51df8f0fded7d337..27f9795ea4da8a748206937d423d68bb5635bd0d 100644 >--- a/Source/WebCore/platform/graphics/gstreamer/MediaSampleGStreamer.cpp >+++ b/Source/WebCore/platform/graphics/gstreamer/MediaSampleGStreamer.cpp >@@ -235,7 +235,7 @@ void MediaSampleGStreamer::offsetTimestampsBy(const MediaTime& timestampOffset) > } > } > >-PlatformSample MediaSampleGStreamer::platformSample() >+PlatformSample MediaSampleGStreamer::platformSample() const > { > PlatformSample sample = { PlatformSample::GStreamerSampleType, { .gstSample = m_sample.get() } }; > return sample; >diff --git a/Source/WebCore/platform/graphics/gstreamer/MediaSampleGStreamer.h b/Source/WebCore/platform/graphics/gstreamer/MediaSampleGStreamer.h >index 1d71a33969cfbc675deecea78e64f2d6899e41eb..3b5a69aa76a6aaf17a56176c1fb239a4f910b0bb 100644 >--- a/Source/WebCore/platform/graphics/gstreamer/MediaSampleGStreamer.h >+++ b/Source/WebCore/platform/graphics/gstreamer/MediaSampleGStreamer.h >@@ -53,7 +53,6 @@ public: > MediaTime decodeTime() const override { return m_dts; } > MediaTime duration() const override { return m_duration; } > AtomString trackID() const override { return m_trackId; } >- void setTrackID(const String& trackId) override { m_trackId = trackId; } > size_t sizeInBytes() const override { return m_size; } > FloatSize presentationSize() const override { return m_presentationSize; } > void offsetTimestampsBy(const MediaTime&) override; >@@ -62,7 +61,7 @@ public: > std::pair<RefPtr<MediaSample>, RefPtr<MediaSample>> divide(const MediaTime&, UseEndTime) override { return { nullptr, nullptr }; } > Ref<MediaSample> createNonDisplayingCopy() const override; > SampleFlags flags() const override { return m_flags; } >- PlatformSample platformSample() override; >+ PlatformSample platformSample() const override; > std::optional<ByteRange> byteRange() const override { return std::nullopt; } > void dump(PrintStream&) const override; > RefPtr<JSC::Uint8ClampedArray> getRGBAImageData() const final; >diff --git a/Source/WebCore/platform/mock/mediasource/MockSourceBufferPrivate.cpp b/Source/WebCore/platform/mock/mediasource/MockSourceBufferPrivate.cpp >index 5cb60b9ef6ef57a43b4eef72a137679b1aebd65b..4ac69acd9a00cf6e199a33aac473d8cd85dc5e5f 100644 >--- a/Source/WebCore/platform/mock/mediasource/MockSourceBufferPrivate.cpp >+++ b/Source/WebCore/platform/mock/mediasource/MockSourceBufferPrivate.cpp >@@ -58,10 +58,9 @@ private: > MediaTime decodeTime() const override { return m_box.decodeTimestamp(); } > MediaTime duration() const override { return m_box.duration(); } > AtomString trackID() const override { return m_id; } >- void setTrackID(const String& id) override { m_id = id; } > size_t sizeInBytes() const override { return sizeof(m_box); } > SampleFlags flags() const override; >- PlatformSample platformSample() override; >+ PlatformSample platformSample() const override; > std::optional<ByteRange> byteRange() const override { return std::nullopt; } > FloatSize presentationSize() const override { return FloatSize(); } > void dump(PrintStream&) const override; >@@ -87,7 +86,7 @@ MediaSample::SampleFlags MockMediaSample::flags() const > return SampleFlags(flags); > } > >-PlatformSample MockMediaSample::platformSample() >+PlatformSample MockMediaSample::platformSample() const > { > PlatformSample sample = { PlatformSample::MockSampleBoxType, { &m_box } }; > return sample; >diff --git a/Source/WebKit/CMakeLists.txt b/Source/WebKit/CMakeLists.txt >index 756da88b45f7042f423b734c764e6566d446efaf..7e8bc60f533d443bcc76496d92ee298442199423 100644 >--- a/Source/WebKit/CMakeLists.txt >+++ b/Source/WebKit/CMakeLists.txt >@@ -275,6 +275,7 @@ set(WebKit_MESSAGES_IN_FILES > WebProcess/GPU/media/RemoteCDMInstance > WebProcess/GPU/media/RemoteCDMInstanceSession > WebProcess/GPU/media/RemoteLegacyCDMSession >+ WebProcess/GPU/media/RemoteVideoFrameProxy > WebProcess/GPU/media/RemoteRemoteCommandListener > WebProcess/GPU/media/SourceBufferPrivateRemote > >diff --git a/Source/WebKit/DerivedSources-input.xcfilelist b/Source/WebKit/DerivedSources-input.xcfilelist >index 4c66220cf122a53b7469d4400c7cddb235c8b749..b9bf252650c5b1aa2aca502c0a6db1b943a26a4e 100644 >--- a/Source/WebKit/DerivedSources-input.xcfilelist >+++ b/Source/WebKit/DerivedSources-input.xcfilelist >@@ -67,6 +67,7 @@ $(PROJECT_DIR)/GPUProcess/media/RemoteMediaSourceProxy.messages.in > $(PROJECT_DIR)/GPUProcess/media/RemoteRemoteCommandListenerProxy.messages.in > $(PROJECT_DIR)/GPUProcess/media/RemoteSourceBufferProxy.messages.in > $(PROJECT_DIR)/GPUProcess/media/RemoteTextTrackProxy.messages.in >+$(PROJECT_DIR)/GPUProcess/media/RemoteVideoFrameObjectHeap.messages.in > $(PROJECT_DIR)/GPUProcess/media/RemoteVideoTrackProxy.messages.in > $(PROJECT_DIR)/GPUProcess/media/ios/RemoteMediaSessionHelperProxy.messages.in > $(PROJECT_DIR)/GPUProcess/webrtc/LibWebRTCCodecsProxy.messages.in >@@ -189,6 +190,7 @@ $(PROJECT_DIR)/WebProcess/GPU/media/RemoteImageDecoderAVF.messages.in > $(PROJECT_DIR)/WebProcess/GPU/media/RemoteImageDecoderAVFManager.messages.in > $(PROJECT_DIR)/WebProcess/GPU/media/RemoteLegacyCDMSession.messages.in > $(PROJECT_DIR)/WebProcess/GPU/media/RemoteRemoteCommandListener.messages.in >+$(PROJECT_DIR)/WebProcess/GPU/media/RemoteVideoFrameProxy.messages.in > $(PROJECT_DIR)/WebProcess/GPU/media/SourceBufferPrivateRemote.messages.in > $(PROJECT_DIR)/WebProcess/GPU/media/ios/RemoteMediaSessionHelper.messages.in > $(PROJECT_DIR)/WebProcess/GPU/webrtc/LibWebRTCCodecs.messages.in >diff --git a/Source/WebKit/DerivedSources-output.xcfilelist b/Source/WebKit/DerivedSources-output.xcfilelist >index e3b55840ae1eaf57f5ebf7542e5e1fd40e5dd4d1..400dfde20f8499c641b85cda3d3d83f3602e57cb 100644 >--- a/Source/WebKit/DerivedSources-output.xcfilelist >+++ b/Source/WebKit/DerivedSources-output.xcfilelist >@@ -381,6 +381,12 @@ $(BUILT_PRODUCTS_DIR)/DerivedSources/WebKit/RemoteTextureMessagesReplies.h > $(BUILT_PRODUCTS_DIR)/DerivedSources/WebKit/RemoteTextureViewMessageReceiver.cpp > $(BUILT_PRODUCTS_DIR)/DerivedSources/WebKit/RemoteTextureViewMessages.h > $(BUILT_PRODUCTS_DIR)/DerivedSources/WebKit/RemoteTextureViewMessagesReplies.h >+$(BUILT_PRODUCTS_DIR)/DerivedSources/WebKit/RemoteVideoFrameObjectHeapMessageReceiver.cpp >+$(BUILT_PRODUCTS_DIR)/DerivedSources/WebKit/RemoteVideoFrameObjectHeapMessages.h >+$(BUILT_PRODUCTS_DIR)/DerivedSources/WebKit/RemoteVideoFrameObjectHeapMessagesReplies.h >+$(BUILT_PRODUCTS_DIR)/DerivedSources/WebKit/RemoteVideoFrameProxyMessageReceiver.cpp >+$(BUILT_PRODUCTS_DIR)/DerivedSources/WebKit/RemoteVideoFrameProxyMessages.h >+$(BUILT_PRODUCTS_DIR)/DerivedSources/WebKit/RemoteVideoFrameProxyMessagesReplies.h > $(BUILT_PRODUCTS_DIR)/DerivedSources/WebKit/RemoteVideoTrackProxyMessageReceiver.cpp > $(BUILT_PRODUCTS_DIR)/DerivedSources/WebKit/RemoteVideoTrackProxyMessages.h > $(BUILT_PRODUCTS_DIR)/DerivedSources/WebKit/RemoteVideoTrackProxyMessagesReplies.h >diff --git a/Source/WebKit/DerivedSources.make b/Source/WebKit/DerivedSources.make >index 7ad49e3735f4b478c416d9295c139be98fd6ce4a..338db4dfcc3ffd67cde177859b75b996b7e60aa2 100644 >--- a/Source/WebKit/DerivedSources.make >+++ b/Source/WebKit/DerivedSources.make >@@ -199,6 +199,7 @@ MESSAGE_RECEIVERS = \ > WebProcess/GPU/media/RemoteCDMInstanceSession \ > WebProcess/GPU/media/RemoteImageDecoderAVFManager \ > WebProcess/GPU/media/RemoteLegacyCDMSession \ >+ WebProcess/GPU/media/RemoteVideoFrameProxy \ > WebProcess/GPU/media/RemoteRemoteCommandListener \ > WebProcess/GPU/media/SourceBufferPrivateRemote \ > WebProcess/GPU/media/ios/RemoteMediaSessionHelper \ >@@ -295,6 +296,7 @@ MESSAGE_RECEIVERS = \ > GPUProcess/media/RemoteMediaPlayerManagerProxy \ > GPUProcess/media/RemoteMediaPlayerProxy \ > GPUProcess/media/RemoteMediaResourceManager \ >+ GPUProcess/media/RemoteVideoFrameObjectHeap \ > GPUProcess/media/RemoteMediaSourceProxy \ > GPUProcess/media/RemoteRemoteCommandListenerProxy \ > GPUProcess/media/RemoteSourceBufferProxy \ >diff --git a/Source/WebKit/GPUProcess/GPUConnectionToWebProcess.cpp b/Source/WebKit/GPUProcess/GPUConnectionToWebProcess.cpp >index 4d524e8b8679a830c45eb6b5bc104b754104b63e..4a312ee3d33b672abec33d873807d5cd692a556f 100644 >--- a/Source/WebKit/GPUProcess/GPUConnectionToWebProcess.cpp >+++ b/Source/WebKit/GPUProcess/GPUConnectionToWebProcess.cpp >@@ -141,6 +141,7 @@ > #endif > > #if ENABLE(MEDIA_STREAM) >+#include "RemoteVideoFrameObjectHeap.h" > #include <WebCore/SecurityOrigin.h> > #endif > >@@ -257,6 +258,9 @@ GPUConnectionToWebProcess::GPUConnectionToWebProcess(GPUProcess& gpuProcess, Web > #if ENABLE(ROUTING_ARBITRATION) && HAVE(AVAUDIO_ROUTING_ARBITER) > gpuProcess.audioSessionManager().session().setRoutingArbitrationClient(m_routingArbitrator.get()); > #endif >+#if ENABLE(MEDIA_STREAM) >+ m_videoFrameObjectHeap = RemoteVideoFrameObjectHeap::create(*this); >+#endif > > if (!parameters.overrideLanguages.isEmpty()) > overrideUserPreferredLanguages(parameters.overrideLanguages); >@@ -300,7 +304,9 @@ void GPUConnectionToWebProcess::didClose(IPC::Connection& connection) > m_audioSessionProxy = nullptr; > } > #endif >- >+#if ENABLE(MEDIA_STREAM) >+ m_videoFrameObjectHeap.reset(); >+#endif > // RemoteRenderingBackend objects ref their GPUConnectionToWebProcess so we need to make sure > // to break the reference cycle by destroying them. > m_remoteRenderingBackendMap.clear(); >@@ -941,8 +947,15 @@ bool GPUConnectionToWebProcess::setCaptureAttributionString() const > { > } > #endif >+ >+RemoteVideoFrameObjectHeap& GPUConnectionToWebProcess::videoFrameObjectHeap() const >+{ >+ return *m_videoFrameObjectHeap.get(); >+} >+ > #endif // ENABLE(MEDIA_STREAM) > >+ > #if PLATFORM(MAC) > void GPUConnectionToWebProcess::displayConfigurationChanged(CGDirectDisplayID, CGDisplayChangeSummaryFlags flags) > { >diff --git a/Source/WebKit/GPUProcess/GPUConnectionToWebProcess.h b/Source/WebKit/GPUProcess/GPUConnectionToWebProcess.h >index 624e3fa351cf0b035e2e13a93325ddd4c342b149..11acc00c74eb5d12f8282d4466397d87d7e9759d 100644 >--- a/Source/WebKit/GPUProcess/GPUConnectionToWebProcess.h >+++ b/Source/WebKit/GPUProcess/GPUConnectionToWebProcess.h >@@ -106,6 +106,11 @@ struct RemoteRenderingBackendCreationParameters; > class RemoteWCLayerTreeHost; > #endif > >+#if ENABLE(MEDIA_STREAM) >+class RemoteVideoFrameObjectHeap; >+#endif >+ >+ > class GPUConnectionToWebProcess > : public ThreadSafeRefCounted<GPUConnectionToWebProcess, WTF::DestructionThread::Main> > , public WebCore::NowPlayingManager::Client >@@ -141,6 +146,7 @@ public: > bool allowsAudioCapture() const { return m_allowsAudioCapture; } > bool allowsVideoCapture() const { return m_allowsVideoCapture; } > bool allowsDisplayCapture() const { return m_allowsDisplayCapture; } >+ RemoteVideoFrameObjectHeap& videoFrameObjectHeap() const; > #endif > > #if PLATFORM(COCOA) && ENABLE(MEDIA_STREAM) >@@ -305,6 +311,7 @@ private: > bool m_allowsAudioCapture { false }; > bool m_allowsVideoCapture { false }; > bool m_allowsDisplayCapture { false }; >+ IPC::ScopedActiveMessageReceiveQueue<RemoteVideoFrameObjectHeap> m_videoFrameObjectHeap; > #endif > #if HAVE(AUDIT_TOKEN) > std::optional<audit_token_t> m_presentingApplicationAuditToken; >diff --git a/Source/WebKit/GPUProcess/graphics/RemoteGraphicsContextGL.cpp b/Source/WebKit/GPUProcess/graphics/RemoteGraphicsContextGL.cpp >index ecb4b5767cc9a8e268646dce39d2662c342c33b6..1e65ca0d02ef1579efc426c368f80f8bcbb0e555 100644 >--- a/Source/WebKit/GPUProcess/graphics/RemoteGraphicsContextGL.cpp >+++ b/Source/WebKit/GPUProcess/graphics/RemoteGraphicsContextGL.cpp >@@ -42,6 +42,10 @@ > #include <WebCore/GraphicsContextGLCV.h> > #endif > >+#if ENABLE(MEDIA_STREAM) >+#include "RemoteVideoFrameObjectHeap.h" >+#endif >+ > namespace WebKit { > > using namespace WebCore; >@@ -68,6 +72,9 @@ RemoteGraphicsContextGL::RemoteGraphicsContextGL(GPUConnectionToWebProcess& gpuC > , m_streamConnection(IPC::StreamServerConnection::create(gpuConnectionToWebProcess.connection(), WTFMove(stream), remoteGraphicsContextGLStreamWorkQueue())) > , m_graphicsContextGLIdentifier(graphicsContextGLIdentifier) > , m_renderingBackend(renderingBackend) >+#if ENABLE(MEDIA_STREAM) >+ , m_videoFrameObjectHeap(gpuConnectionToWebProcess.videoFrameObjectHeap()) >+#endif > , m_renderingResourcesRequest(ScopedWebGLRenderingResourcesRequest::acquire()) > , m_webProcessIdentifier(gpuConnectionToWebProcess.webProcessIdentifier()) > { >@@ -223,6 +230,12 @@ void RemoteGraphicsContextGL::paintCompositedResultsToCanvasWithQualifiedIdentif > paintPixelBufferToImageBuffer(m_context->readCompositedResultsForPainting(), imageBuffer, WTFMove(completionHandler)); > } > >+void RemoteGraphicsContextGL::paintCompositedResultsToMediaSample(RemoteVideoFrameIdentifier identifier) >+{ >+ assertIsCurrent(m_streamThread); >+ m_videoFrameObjectHeap->add(identifier, m_context->paintCompositedResultsToMediaSample()); >+} >+ > void RemoteGraphicsContextGL::paintPixelBufferToImageBuffer(std::optional<WebCore::PixelBuffer>&& pixelBuffer, QualifiedRenderingResourceIdentifier target, CompletionHandler<void()>&& completionHandler) > { > assertIsCurrent(m_streamThread); >diff --git a/Source/WebKit/GPUProcess/graphics/RemoteGraphicsContextGL.h b/Source/WebKit/GPUProcess/graphics/RemoteGraphicsContextGL.h >index eb20d16c377f5f3e007f21ba25624dd19e36bb17..c0023febfbb65eac82b3f65be414e62cfc118b5e 100644 >--- a/Source/WebKit/GPUProcess/graphics/RemoteGraphicsContextGL.h >+++ b/Source/WebKit/GPUProcess/graphics/RemoteGraphicsContextGL.h >@@ -54,6 +54,10 @@ > #include "WCContentBufferIdentifier.h" > #endif > >+#if ENABLE(MEDIA_STREAM) >+#include "RemoteVideoFrameIdentifier.h" >+#endif >+ > #if PLATFORM(COCOA) > namespace WTF { > class MachSendRight; >@@ -61,6 +65,9 @@ class MachSendRight; > #endif > > namespace WebKit { >+#if ENABLE(MEDIA_STREAM) >+class RemoteVideoFrameObjectHeap; >+#endif > > // GPU process side implementation of that receives messages about GraphicsContextGL calls > // and issues real GraphicsContextGL calls based on the received messages. >@@ -108,6 +115,9 @@ protected: > void synthesizeGLError(uint32_t error); > void paintRenderingResultsToCanvas(WebCore::RenderingResourceIdentifier, CompletionHandler<void()>&&); > void paintCompositedResultsToCanvas(WebCore::RenderingResourceIdentifier, CompletionHandler<void()>&&); >+#if ENABLE(MEDIA_STREAM) >+ void paintCompositedResultsToMediaSample(RemoteVideoFrameIdentifier); >+#endif > void copyTextureFromMedia(WebCore::MediaPlayerIdentifier, uint32_t texture, uint32_t target, int32_t level, uint32_t internalFormat, uint32_t format, uint32_t type, bool premultiplyAlpha, bool flipY, CompletionHandler<void(bool)>&&); > void simulateEventForTesting(WebCore::GraphicsContextGL::SimulatedEventForTesting); > >@@ -130,6 +140,9 @@ protected: > RefPtr<PlatformGraphicsContextGL> m_context WTF_GUARDED_BY_LOCK(m_streamThread); > GraphicsContextGLIdentifier m_graphicsContextGLIdentifier; > Ref<RemoteRenderingBackend> m_renderingBackend; >+#if ENABLE(MEDIA_STREAM) >+ Ref<RemoteVideoFrameObjectHeap> m_videoFrameObjectHeap; >+#endif > ScopedWebGLRenderingResourcesRequest m_renderingResourcesRequest; > NO_UNIQUE_ADDRESS ThreadAssertion m_streamThread; > WebCore::ProcessIdentifier m_webProcessIdentifier; >diff --git a/Source/WebKit/GPUProcess/graphics/RemoteGraphicsContextGL.messages.in b/Source/WebKit/GPUProcess/graphics/RemoteGraphicsContextGL.messages.in >index c8ab29772930bfd408a03ceb37a0c29898e302b2..b2b0b665efc56e5aa3fc26fd20b51461167304e0 100644 >--- a/Source/WebKit/GPUProcess/graphics/RemoteGraphicsContextGL.messages.in >+++ b/Source/WebKit/GPUProcess/graphics/RemoteGraphicsContextGL.messages.in >@@ -42,6 +42,9 @@ messages -> RemoteGraphicsContextGL NotRefCounted Stream { > void PaintRenderingResultsToCanvas(WebCore::RenderingResourceIdentifier imageBuffer) -> () Synchronous > void PaintCompositedResultsToCanvas(WebCore::RenderingResourceIdentifier imageBuffer) -> () Synchronous > void CopyTextureFromMedia(WebCore::MediaPlayerIdentifier identifier, uint32_t texture, uint32_t target, int32_t level, uint32_t internalFormat, uint32_t format, uint32_t type, bool premultiplyAlpha, bool flipY) -> (bool success) Synchronous >+#if ENABLE(MEDIA_STREAM) >+ void PaintCompositedResultsToMediaSample(WebKit::RemoteVideoFrameIdentifier identifier) >+#endif > void SimulateEventForTesting(WebCore::GraphicsContextGL::SimulatedEventForTesting event) > > void MoveErrorsToSyntheticErrorList() -> (bool returnValue) Synchronous >diff --git a/Source/WebKit/GPUProcess/media/RemoteVideoFrameObjectHeap.cpp b/Source/WebKit/GPUProcess/media/RemoteVideoFrameObjectHeap.cpp >new file mode 100644 >index 0000000000000000000000000000000000000000..7d9b1d5d7dac4a43690ffe12586699087bcb4102 >--- /dev/null >+++ b/Source/WebKit/GPUProcess/media/RemoteVideoFrameObjectHeap.cpp >@@ -0,0 +1,113 @@ >+ >+/* >+ * Copyright (C) 2022 Apple Inc. All rights reserved. >+ * >+ * Redistribution and use in source and binary forms, with or without >+ * modification, are permitted provided that the following conditions >+ * are met: >+ * 1. Redistributions of source code must retain the above copyright >+ * notice, this list of conditions and the following disclaimer. >+ * 2. Redistributions in binary form must reproduce the above copyright >+ * notice, this list of conditions and the following disclaimer in the >+ * documentation and/or other materials provided with the distribution. >+ * >+ * THIS SOFTWARE IS PROVIDED BY APPLE INC. AND ITS CONTRIBUTORS ``AS IS'' >+ * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, >+ * THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR >+ * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL APPLE INC. OR ITS CONTRIBUTORS >+ * BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR >+ * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF >+ * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS >+ * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN >+ * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) >+ * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF >+ * THE POSSIBILITY OF SUCH DAMAGE. >+ */ >+ >+#include "config.h" >+#include "RemoteVideoFrameObjectHeap.h" >+ >+#if ENABLE(GPU_PROCESS) && ENABLE(MEDIA_STREAM) >+#include "GPUConnectionToWebProcess.h" >+#include "RemoteVideoFrameObjectHeapMessages.h" >+#include "RemoteVideoFrameProxy.h" >+#include "RemoteVideoFrameProxyMessages.h" >+ >+namespace WebKit { >+ >+Ref<RemoteVideoFrameObjectHeap> RemoteVideoFrameObjectHeap::create(GPUConnectionToWebProcess& connectionToWebProcess) >+{ >+ return adoptRef(*new RemoteVideoFrameObjectHeap(connectionToWebProcess)); >+} >+ >+RemoteVideoFrameObjectHeap::RemoteVideoFrameObjectHeap(GPUConnectionToWebProcess& connectionToWebProcess) >+ : m_gpuConnectionToWebProcess(&connectionToWebProcess) >+ , m_connection(m_gpuConnectionToWebProcess->connection()) >+{ >+ m_gpuConnectionToWebProcess->messageReceiverMap().addMessageReceiver(Messages::RemoteVideoFrameObjectHeap::messageReceiverName(), *this); >+} >+ >+RemoteVideoFrameObjectHeap::~RemoteVideoFrameObjectHeap() >+{ >+ ASSERT(!m_gpuConnectionToWebProcess); >+} >+ >+void RemoteVideoFrameObjectHeap::add(RemoteVideoFrameIdentifier identifier, RefPtr<WebCore::MediaSample>&& sample) >+{ >+ auto wasNew = addImpl(identifier, WTFMove(sample)); >+ ASSERT_UNUSED(wasNew, wasNew); >+} >+ >+bool RemoteVideoFrameObjectHeap::addImpl(RemoteVideoFrameIdentifier identifier, RefPtr<WebCore::MediaSample>&& sample) >+{ >+ // FIXME: Probably too hard get correct, maybe make sample Ref<>. >+ std::optional<RemoteVideoFrameProxy::Properties> properties; >+ if (sample) >+ properties = RemoteVideoFrameProxy::properties(*sample); >+ >+ bool wasNew = false; >+ { >+ Locker locker { m_remoteFramesLock }; >+ auto result = m_remoteFrames.add(identifier, WTFMove(sample)); >+ wasNew = result.isNewEntry; >+ } >+ if (wasNew) >+ m_connection->send(Messages::RemoteVideoFrameProxy::WasInitialized(WTFMove(properties)), identifier.toUInt64()); >+ return wasNew; >+} >+ >+void RemoteVideoFrameObjectHeap::remove(RemoteVideoFrameIdentifier identifier) >+{ >+ Locker locker { m_remoteFramesLock }; >+ m_remoteFrames.remove(identifier); >+} >+ >+RefPtr<WebCore::MediaSample> RemoteVideoFrameObjectHeap::get(RemoteVideoFrameIdentifier identifier) >+{ >+ Locker locker { m_remoteFramesLock }; >+ return m_remoteFrames.get(identifier); >+} >+ >+void RemoteVideoFrameObjectHeap::stopListeningForIPC(Ref<RemoteVideoFrameObjectHeap>&& refFromConnection) >+{ >+ assertIsCurrent(m_consumeThread); >+ if (auto* gpuConnectionToWebProcess = std::exchange(m_gpuConnectionToWebProcess, nullptr)) { >+ gpuConnectionToWebProcess->messageReceiverMap().removeMessageReceiver(Messages::RemoteVideoFrameObjectHeap::messageReceiverName()); >+ // Clients might hold on to the ref after this happens. They should also stop themselves, but if they do not, >+ // avoid big memory leaks by clearing the frames. The clients should fail gracefully (do nothing) in case they fail to look up >+ // frames. >+ Locker locker { m_remoteFramesLock }; >+ m_remoteFrames.clear(); >+ // TODO: add can happen after stopping. >+ } >+} >+ >+void RemoteVideoFrameObjectHeap::releaseVideoFrame(RemoteVideoFrameIdentifier identifier) >+{ >+ assertIsCurrent(m_consumeThread); >+ remove(identifier); >+} >+ >+} >+ >+#endif >diff --git a/Source/WebKit/GPUProcess/media/RemoteVideoFrameObjectHeap.h b/Source/WebKit/GPUProcess/media/RemoteVideoFrameObjectHeap.h >new file mode 100644 >index 0000000000000000000000000000000000000000..a9917002f5349328d1a6b8210e55df4f54495b4f >--- /dev/null >+++ b/Source/WebKit/GPUProcess/media/RemoteVideoFrameObjectHeap.h >@@ -0,0 +1,80 @@ >+/* >+ * Copyright (C) 2022 Apple Inc. All rights reserved. >+ * >+ * Redistribution and use in source and binary forms, with or without >+ * modification, are permitted provided that the following conditions >+ * are met: >+ * 1. Redistributions of source code must retain the above copyright >+ * notice, this list of conditions and the following disclaimer. >+ * 2. Redistributions in binary form must reproduce the above copyright >+ * notice, this list of conditions and the following disclaimer in the >+ * documentation and/or other materials provided with the distribution. >+ * >+ * THIS SOFTWARE IS PROVIDED BY APPLE INC. AND ITS CONTRIBUTORS ``AS IS'' >+ * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, >+ * THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR >+ * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL APPLE INC. OR ITS CONTRIBUTORS >+ * BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR >+ * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF >+ * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS >+ * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN >+ * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) >+ * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF >+ * THE POSSIBILITY OF SUCH DAMAGE. >+ */ >+ >+#pragma once >+ >+#if ENABLE(GPU_PROCESS) && ENABLE(MEDIA_STREAM) >+#include "RemoteVideoFrameIdentifier.h" >+#include <WebCore/MediaSample.h> >+#include <WebCore/ProcessIdentity.h> >+#include <WebCore/RenderingResourceIdentifier.h> >+#include <memory> >+#include <wtf/HashMap.h> >+#include <wtf/Lock.h> >+#include <wtf/MachSendRight.h> >+#include <wtf/ThreadAssertions.h> >+#include <wtf/ThreadSafeRefCounted.h> >+ >+#if USE(AVFOUNDATION) >+#include <WebCore/RemoteVideoSample.h> >+#endif >+ >+namespace WebKit { >+class GPUConnectionToWebProcess; >+ >+// Holds references to all VideoFrame instances that are mapped between two processes. >+// As currently there is no special state for RemoteVideoFrame, the object heap stores VideoFrame instances (MediaSample). >+// As currently there is no RemoteVideoFrame, responsible for dispatching the VideoFrame methods to the objects. >+// Consume thread is the thread that uses the VideoFrame. For GPUP this is the media player thread, the main thread. >+// FIXME: currently VideoFrame instances are MediaSample instances. >+class RemoteVideoFrameObjectHeap final : public ThreadSafeRefCounted<RemoteVideoFrameObjectHeap>, private IPC::MessageReceiver { >+public: >+ static Ref<RemoteVideoFrameObjectHeap> create(GPUConnectionToWebProcess&); >+ ~RemoteVideoFrameObjectHeap(); >+ void add(RemoteVideoFrameIdentifier, RefPtr<WebCore::MediaSample>&&); >+ void remove(RemoteVideoFrameIdentifier); >+ >+ RefPtr<WebCore::MediaSample> get(RemoteVideoFrameIdentifier); >+ >+ void stopListeningForIPC(Ref<RemoteVideoFrameObjectHeap>&& refFromConnection); >+private: >+ RemoteVideoFrameObjectHeap(GPUConnectionToWebProcess&); >+ bool addImpl(RemoteVideoFrameIdentifier, RefPtr<WebCore::MediaSample>&&) WARN_UNUSED_RETURN; >+ >+ // IPC::MessageReceiver overrides. >+ void didReceiveMessage(IPC::Connection&, IPC::Decoder&) final; >+ >+ // Messages. >+ void releaseVideoFrame(RemoteVideoFrameIdentifier); >+ >+ GPUConnectionToWebProcess* m_gpuConnectionToWebProcess WTF_GUARDED_BY_LOCK(m_consumeThread); >+ const Ref<IPC::Connection> m_connection; >+ Lock m_remoteFramesLock; >+ HashMap<RemoteVideoFrameIdentifier, RefPtr<WebCore::MediaSample>> m_remoteFrames WTF_GUARDED_BY_LOCK(m_remoteFramesLock); >+ ThreadAssertion m_consumeThread NO_UNIQUE_ADDRESS; >+}; >+ >+} >+#endif >diff --git a/Source/WebKit/GPUProcess/media/RemoteVideoFrameObjectHeap.messages.in b/Source/WebKit/GPUProcess/media/RemoteVideoFrameObjectHeap.messages.in >new file mode 100644 >index 0000000000000000000000000000000000000000..763d1d153d531370dde23f69dc7710f080fd7d0a >--- /dev/null >+++ b/Source/WebKit/GPUProcess/media/RemoteVideoFrameObjectHeap.messages.in >@@ -0,0 +1,28 @@ >+# Copyright (C) 2020 Apple Inc. All rights reserved. >+# >+# Redistribution and use in source and binary forms, with or without >+# modification, are permitted provided that the following conditions >+# are met: >+# 1. Redistributions of source code must retain the above copyright >+# notice, this list of conditions and the following disclaimer. >+# 2. Redistributions in binary form must reproduce the above copyright >+# notice, this list of conditions and the following disclaimer in the >+# documentation and/or other materials provided with the distribution. >+# >+# THIS SOFTWARE IS PROVIDED BY APPLE INC. AND ITS CONTRIBUTORS ``AS IS'' >+# AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, >+# THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR >+# PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL APPLE INC. OR ITS CONTRIBUTORS >+# BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR >+# CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF >+# SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS >+# INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN >+# CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) >+# ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF >+# THE POSSIBILITY OF SUCH DAMAGE. >+ >+#if ENABLE(GPU_PROCESS) && ENABLE(MEDIA_STREAM) >+messages -> RemoteVideoFrameObjectHeap NotRefCounted { >+ ReleaseVideoFrame(WebKit::RemoteVideoFrameIdentifier identifier) >+} >+#endif >\ No newline at end of file >diff --git a/Source/WebKit/GPUProcess/webrtc/RemoteSampleBufferDisplayLayer.cpp b/Source/WebKit/GPUProcess/webrtc/RemoteSampleBufferDisplayLayer.cpp >index 217db97a3ce4c830048afac828efbdbbd518e7a0..d12280ecd7f4228f65164bf812ec11c680f0874b 100644 >--- a/Source/WebKit/GPUProcess/webrtc/RemoteSampleBufferDisplayLayer.cpp >+++ b/Source/WebKit/GPUProcess/webrtc/RemoteSampleBufferDisplayLayer.cpp >@@ -28,6 +28,7 @@ > > #if PLATFORM(COCOA) && ENABLE(GPU_PROCESS) && ENABLE(MEDIA_STREAM) > >+#include "RemoteVideoFrameObjectHeap.h" > #include "SampleBufferDisplayLayerMessages.h" > #include <WebCore/ImageTransferSessionVT.h> > #include <WebCore/LocalSampleBufferDisplayLayer.h> >@@ -37,16 +38,18 @@ > namespace WebKit { > using namespace WebCore; > >-std::unique_ptr<RemoteSampleBufferDisplayLayer> RemoteSampleBufferDisplayLayer::create(SampleBufferDisplayLayerIdentifier identifier, Ref<IPC::Connection>&& connection) >+std::unique_ptr<RemoteSampleBufferDisplayLayer> RemoteSampleBufferDisplayLayer::create(GPUConnectionToWebProcess& gpuConnection, SampleBufferDisplayLayerIdentifier identifier, Ref<IPC::Connection>&& connection) > { >- auto layer = std::unique_ptr<RemoteSampleBufferDisplayLayer>(new RemoteSampleBufferDisplayLayer(identifier, WTFMove(connection))); >+ auto layer = std::unique_ptr<RemoteSampleBufferDisplayLayer>(new RemoteSampleBufferDisplayLayer(gpuConnection, identifier, WTFMove(connection))); > return layer->m_sampleBufferDisplayLayer ? WTFMove(layer) : nullptr; > } > >-RemoteSampleBufferDisplayLayer::RemoteSampleBufferDisplayLayer(SampleBufferDisplayLayerIdentifier identifier, Ref<IPC::Connection>&& connection) >- : m_identifier(identifier) >+RemoteSampleBufferDisplayLayer::RemoteSampleBufferDisplayLayer(GPUConnectionToWebProcess& gpuConnection, SampleBufferDisplayLayerIdentifier identifier, Ref<IPC::Connection>&& connection) >+ : m_gpuConnection(gpuConnection) >+ , m_identifier(identifier) > , m_connection(WTFMove(connection)) > , m_sampleBufferDisplayLayer(LocalSampleBufferDisplayLayer::create(*this)) >+ , m_videoFrameObjectHeap(m_gpuConnection.videoFrameObjectHeap()) > { > ASSERT(m_sampleBufferDisplayLayer); > } >@@ -113,7 +116,20 @@ void RemoteSampleBufferDisplayLayer::pause() > m_sampleBufferDisplayLayer->pause(); > } > >-void RemoteSampleBufferDisplayLayer::enqueueSample(RemoteVideoSample&& remoteSample) >+void RemoteSampleBufferDisplayLayer::enqueueSample(RemoteVideoFrameIdentifier identifier) >+{ >+ auto mediaSample = m_videoFrameObjectHeap->get(identifier); >+ ASSERT(mediaSample); >+ if (!mediaSample) >+ return; >+ if (!is<MediaSampleAVFObjC>(mediaSample)) >+ return; >+ auto& avfMediaSample = downcast<MediaSampleAVFObjC>(*mediaSample); >+ MediaSampleAVFObjC::setAsDisplayImmediately(avfMediaSample); >+ m_sampleBufferDisplayLayer->enqueueSample(avfMediaSample); >+} >+ >+void RemoteSampleBufferDisplayLayer::enqueueSampleCV(WebCore::RemoteVideoSample&& remoteSample) > { > RefPtr<MediaSample> sample; > if (!remoteSample.surface()) { >diff --git a/Source/WebKit/GPUProcess/webrtc/RemoteSampleBufferDisplayLayer.h b/Source/WebKit/GPUProcess/webrtc/RemoteSampleBufferDisplayLayer.h >index 725b3d26b71c7886dc488827d0a6f7d252aa7a0d..bf7fe5c4531656b64d592e8c42173fe5ced46514 100644 >--- a/Source/WebKit/GPUProcess/webrtc/RemoteSampleBufferDisplayLayer.h >+++ b/Source/WebKit/GPUProcess/webrtc/RemoteSampleBufferDisplayLayer.h >@@ -30,10 +30,12 @@ > #include "LayerHostingContext.h" > #include "MessageReceiver.h" > #include "MessageSender.h" >+#include "RemoteVideoFrameIdentifier.h" > #include "SampleBufferDisplayLayerIdentifier.h" > #include "SharedVideoFrame.h" > #include <WebCore/SampleBufferDisplayLayer.h> > #include <wtf/MediaTime.h> >+#include <wtf/ThreadAssertions.h> > > namespace WebCore { > class ImageTransferSessionVT; >@@ -42,11 +44,13 @@ class RemoteVideoSample; > }; > > namespace WebKit { >+class GPUConnectionToWebProcess; >+class RemoteVideoFrameObjectHeap; > > class RemoteSampleBufferDisplayLayer : public WebCore::SampleBufferDisplayLayer::Client, public IPC::MessageReceiver, private IPC::MessageSender { > WTF_MAKE_FAST_ALLOCATED; > public: >- static std::unique_ptr<RemoteSampleBufferDisplayLayer> create(SampleBufferDisplayLayerIdentifier, Ref<IPC::Connection>&&); >+ static std::unique_ptr<RemoteSampleBufferDisplayLayer> create(GPUConnectionToWebProcess&, SampleBufferDisplayLayerIdentifier, Ref<IPC::Connection>&&); > ~RemoteSampleBufferDisplayLayer(); > > using WebCore::SampleBufferDisplayLayer::Client::weakPtrFactory; >@@ -61,7 +65,7 @@ public: > CGRect bounds() const; > > private: >- RemoteSampleBufferDisplayLayer(SampleBufferDisplayLayerIdentifier, Ref<IPC::Connection>&&); >+ RemoteSampleBufferDisplayLayer(GPUConnectionToWebProcess&, SampleBufferDisplayLayerIdentifier, Ref<IPC::Connection>&&); > > #if !RELEASE_LOG_DISABLED > void setLogIdentifier(String&&); >@@ -73,7 +77,8 @@ private: > void flushAndRemoveImage(); > void play(); > void pause(); >- void enqueueSample(WebCore::RemoteVideoSample&&); >+ void enqueueSample(RemoteVideoFrameIdentifier); >+ void enqueueSampleCV(WebCore::RemoteVideoSample&&); > void clearEnqueuedSamples(); > void setSharedVideoFrameSemaphore(IPC::Semaphore&&); > void setSharedVideoFrameMemory(const SharedMemory::IPCHandle&); >@@ -85,12 +90,16 @@ private: > // WebCore::SampleBufferDisplayLayer::Client > void sampleBufferDisplayLayerStatusDidFail() final; > >+ GPUConnectionToWebProcess& m_gpuConnection WTF_GUARDED_BY_LOCK(m_consumeThread); > SampleBufferDisplayLayerIdentifier m_identifier; > Ref<IPC::Connection> m_connection; > std::unique_ptr<WebCore::ImageTransferSessionVT> m_imageTransferSession; > std::unique_ptr<WebCore::LocalSampleBufferDisplayLayer> m_sampleBufferDisplayLayer; > std::unique_ptr<LayerHostingContext> m_layerHostingContext; > SharedVideoFrameReader m_sharedVideoFrameReader; >+ Ref<RemoteVideoFrameObjectHeap> m_videoFrameObjectHeap; >+ ThreadAssertion m_consumeThread NO_UNIQUE_ADDRESS; >+ > }; > > } >diff --git a/Source/WebKit/GPUProcess/webrtc/RemoteSampleBufferDisplayLayer.messages.in b/Source/WebKit/GPUProcess/webrtc/RemoteSampleBufferDisplayLayer.messages.in >index e50483141efefafb43dca608cdc37d6216f7ecdf..e15ba441dc32f2693e1ea4b5538574076e8e6ccf 100644 >--- a/Source/WebKit/GPUProcess/webrtc/RemoteSampleBufferDisplayLayer.messages.in >+++ b/Source/WebKit/GPUProcess/webrtc/RemoteSampleBufferDisplayLayer.messages.in >@@ -32,7 +32,8 @@ messages -> RemoteSampleBufferDisplayLayer NotRefCounted { > UpdateBoundsAndPosition(CGRect bounds, WebCore::MediaSample::VideoRotation rotation) > Flush() > FlushAndRemoveImage() >- EnqueueSample(WebCore::RemoteVideoSample sample) >+ EnqueueSample(WebKit::RemoteVideoFrameIdentifier sample) >+ EnqueueSampleCV(WebCore::RemoteVideoSample sample) > ClearEnqueuedSamples() > Play() > Pause() >diff --git a/Source/WebKit/GPUProcess/webrtc/RemoteSampleBufferDisplayLayerManager.cpp b/Source/WebKit/GPUProcess/webrtc/RemoteSampleBufferDisplayLayerManager.cpp >index a5445d7fd5ccbb5ad00cdecec5d03f6334b5d151..3914d90b7c13fa5cc96508596338a4178ff46eef 100644 >--- a/Source/WebKit/GPUProcess/webrtc/RemoteSampleBufferDisplayLayerManager.cpp >+++ b/Source/WebKit/GPUProcess/webrtc/RemoteSampleBufferDisplayLayerManager.cpp >@@ -38,6 +38,7 @@ > > namespace WebKit { > >+ > RemoteSampleBufferDisplayLayerManager::RemoteSampleBufferDisplayLayerManager(GPUConnectionToWebProcess& gpuConnectionToWebProcess) > : m_connectionToWebProcess(gpuConnectionToWebProcess) > , m_connection(gpuConnectionToWebProcess.connection()) >@@ -81,7 +82,7 @@ bool RemoteSampleBufferDisplayLayerManager::dispatchMessage(IPC::Connection& con > void RemoteSampleBufferDisplayLayerManager::createLayer(SampleBufferDisplayLayerIdentifier identifier, bool hideRootLayer, WebCore::IntSize size, LayerCreationCallback&& callback) > { > callOnMainRunLoop([this, protectedThis = Ref { *this }, identifier, hideRootLayer, size, callback = WTFMove(callback)]() mutable { >- auto layer = RemoteSampleBufferDisplayLayer::create(identifier, m_connection.copyRef()); >+ auto layer = RemoteSampleBufferDisplayLayer::create(m_connectionToWebProcess, identifier, m_connection.copyRef()); > auto& layerReference = *layer; > layerReference.initialize(hideRootLayer, size, [this, protectedThis = Ref { *this }, callback = WTFMove(callback), identifier, layer = WTFMove(layer)](auto layerId) mutable { > dispatchToThread([this, protectedThis = WTFMove(protectedThis), callback = WTFMove(callback), identifier, layer = WTFMove(layer), layerId = WTFMove(layerId)]() mutable { >diff --git a/Source/WebKit/Scripts/webkit/messages.py b/Source/WebKit/Scripts/webkit/messages.py >index 7ebfa516b50336d3687f7ecdd17e8e7ebd1cd9d7..6bb19f39c9a2897a42c0a4cc9d730528c4de8690 100644 >--- a/Source/WebKit/Scripts/webkit/messages.py >+++ b/Source/WebKit/Scripts/webkit/messages.py >@@ -342,6 +342,7 @@ def types_that_cannot_be_forward_declared(): > 'WebKit::RemoteLegacyCDMSessionIdentifier', > 'WebKit::RemoteMediaResourceIdentifier', > 'WebKit::RemoteMediaSourceIdentifier', >+ 'WebKit::RemoteVideoFrameIdentifier', > 'WebKit::RemoteRemoteCommandListenerIdentifier', > 'WebKit::RemoteSourceBufferIdentifier', > 'WebKit::RenderingBackendIdentifier', >diff --git a/Source/WebKit/Shared/mac/MediaFormatReader/MediaSampleByteRange.cpp b/Source/WebKit/Shared/mac/MediaFormatReader/MediaSampleByteRange.cpp >index 4de1d0ed2a7be49e3cbbb95e377f54509fac07ad..78f73134ce6a7a214cdc9de05c475372e17c9a3d 100644 >--- a/Source/WebKit/Shared/mac/MediaFormatReader/MediaSampleByteRange.cpp >+++ b/Source/WebKit/Shared/mac/MediaFormatReader/MediaSampleByteRange.cpp >@@ -66,7 +66,7 @@ AtomString MediaSampleByteRange::trackID() const > return AtomString::number(m_trackID); > } > >-PlatformSample MediaSampleByteRange::platformSample() >+PlatformSample MediaSampleByteRange::platformSample() const > { > return { > PlatformSample::ByteRangeSampleType, >diff --git a/Source/WebKit/Shared/mac/MediaFormatReader/MediaSampleByteRange.h b/Source/WebKit/Shared/mac/MediaFormatReader/MediaSampleByteRange.h >index b8e4d76ce6f3928576e9fb1ef962b40497e3dc31..f8336bf013721090da19378fb5a453c8daca576f 100644 >--- a/Source/WebKit/Shared/mac/MediaFormatReader/MediaSampleByteRange.h >+++ b/Source/WebKit/Shared/mac/MediaFormatReader/MediaSampleByteRange.h >@@ -50,7 +50,7 @@ public: > std::optional<ByteRange> byteRange() const final { return m_byteRange; } > > AtomString trackID() const final; >- WebCore::PlatformSample platformSample() final; >+ WebCore::PlatformSample platformSample() const final; > void offsetTimestampsBy(const MediaTime&) final; > void setTimestamps(const MediaTime&, const MediaTime&) final; > >diff --git a/Source/WebKit/Sources.txt b/Source/WebKit/Sources.txt >index 4f0a235441b2e1c0cccc140dcc50f26f00443ca0..ed9b63fbea6837b7fa4a097c519e6d2d7b9d38d5 100644 >--- a/Source/WebKit/Sources.txt >+++ b/Source/WebKit/Sources.txt >@@ -70,6 +70,7 @@ GPUProcess/media/RemoteMediaPlayerProxy.cpp > GPUProcess/media/RemoteMediaResource.cpp > GPUProcess/media/RemoteMediaResourceLoader.cpp > GPUProcess/media/RemoteMediaResourceManager.cpp >+GPUProcess/media/RemoteVideoFrameObjectHeap.cpp > GPUProcess/media/RemoteMediaSourceProxy.cpp > GPUProcess/media/RemoteRemoteCommandListenerProxy.cpp > GPUProcess/media/RemoteSourceBufferProxy.cpp >@@ -736,6 +737,7 @@ WebProcess/GPU/media/RemoteMediaEngineConfigurationFactory.cpp > WebProcess/GPU/media/RemoteMediaPlayerManager.cpp > WebProcess/GPU/media/RemoteMediaPlayerMIMETypeCache.cpp @no-unify > WebProcess/GPU/media/RemoteMediaResourceProxy.cpp >+WebProcess/GPU/media/RemoteVideoFrameProxy.cpp > WebProcess/GPU/media/RemoteRemoteCommandListener.cpp > WebProcess/GPU/media/SourceBufferPrivateRemote.cpp > WebProcess/GPU/media/TextTrackPrivateRemote.cpp >@@ -870,6 +872,8 @@ RemoteDeviceMessageReceiver.cpp > RemoteExternalTextureMessageReceiver.cpp > RemoteGPUMessageReceiver.cpp > RemoteGPUProxyMessageReceiver.cpp >+RemoteVideoFrameObjectHeapMessageReceiver.cpp >+RemoteVideoFrameProxyMessageReceiver.cpp > RemotePipelineLayoutMessageReceiver.cpp > RemoteQuerySetMessageReceiver.cpp > RemoteQueueMessageReceiver.cpp >diff --git a/Source/WebKit/WebKit.xcodeproj/project.pbxproj b/Source/WebKit/WebKit.xcodeproj/project.pbxproj >index a1ad0207b30127d30c021e66dbc4a68a78cbd030..a155a88de0cb81d23712c5a4cf6d8b2a82d5b28f 100644 >--- a/Source/WebKit/WebKit.xcodeproj/project.pbxproj >+++ b/Source/WebKit/WebKit.xcodeproj/project.pbxproj >@@ -5077,6 +5077,12 @@ > 7B483F1E25CDDA9B00120486 /* MessageReceiveQueues.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = MessageReceiveQueues.h; sourceTree = "<group>"; }; > 7B50E97F2771F6CE003DAAC4 /* IPCTester.cpp */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.cpp.cpp; path = IPCTester.cpp; sourceTree = "<group>"; }; > 7B50E9802771F6CF003DAAC4 /* IPCTester.h */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.c.h; path = IPCTester.h; sourceTree = "<group>"; }; >+ 7B5A3DA027A7DC1A006C6F97 /* RemoteVideoFrameIdentifier.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = RemoteVideoFrameIdentifier.h; sourceTree = "<group>"; }; >+ 7B5A3DA127A7DC1A006C6F97 /* RemoteVideoFrameProxy.messages.in */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = text; path = RemoteVideoFrameProxy.messages.in; sourceTree = "<group>"; }; >+ 7B5A3DA227A7DC1A006C6F97 /* RemoteVideoFrameProxy.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = RemoteVideoFrameProxy.h; sourceTree = "<group>"; }; >+ 7B5A3DA327A7DC6C006C6F97 /* RemoteVideoFrameProxy.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = RemoteVideoFrameProxy.cpp; sourceTree = "<group>"; }; >+ 7B5A3DA727A7DCBC006C6F97 /* RemoteVideoFrameObjectHeap.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = RemoteVideoFrameObjectHeap.h; sourceTree = "<group>"; }; >+ 7B5A3DA827A7DCBC006C6F97 /* RemoteVideoFrameObjectHeap.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = RemoteVideoFrameObjectHeap.cpp; sourceTree = "<group>"; }; > 7B64C0B6254C5C250006B4AF /* GraphicsContextGLIdentifier.h */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.c.h; path = GraphicsContextGLIdentifier.h; sourceTree = "<group>"; }; > 7B73123125CC8523003B2796 /* StreamConnectionBuffer.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = StreamConnectionBuffer.h; sourceTree = "<group>"; }; > 7B73123225CC8523003B2796 /* StreamConnectionWorkQueue.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = StreamConnectionWorkQueue.cpp; sourceTree = "<group>"; }; >@@ -6752,6 +6758,10 @@ > CD8252D125D464AB00862FD8 /* RemoteRemoteCommandListener.h */, > CD8252D325D464BF00862FD8 /* RemoteRemoteCommandListener.messages.in */, > CD8252D425D4651000862FD8 /* RemoteRemoteCommandListenerIdentifier.h */, >+ 7B5A3DA027A7DC1A006C6F97 /* RemoteVideoFrameIdentifier.h */, >+ 7B5A3DA327A7DC6C006C6F97 /* RemoteVideoFrameProxy.cpp */, >+ 7B5A3DA227A7DC1A006C6F97 /* RemoteVideoFrameProxy.h */, >+ 7B5A3DA127A7DC1A006C6F97 /* RemoteVideoFrameProxy.messages.in */, > 1DD2A63A255DE6D100FF7B6F /* SourceBufferPrivateRemote.cpp */, > 1DD2A639255DE6D100FF7B6F /* SourceBufferPrivateRemote.h */, > 1DD2A66F25622F1100FF7B6F /* SourceBufferPrivateRemote.messages.in */, >@@ -6842,6 +6852,8 @@ > 1DD2A6662561ED0400FF7B6F /* RemoteSourceBufferProxy.messages.in */, > 07E19F0723D4DC880094FFB4 /* RemoteTextTrackProxy.cpp */, > 07E19F0623D4DC880094FFB4 /* RemoteTextTrackProxy.h */, >+ 7B5A3DA827A7DCBC006C6F97 /* RemoteVideoFrameObjectHeap.cpp */, >+ 7B5A3DA727A7DCBC006C6F97 /* RemoteVideoFrameObjectHeap.h */, > 071BC58423CC2C0900680D7C /* RemoteVideoTrackProxy.cpp */, > 071BC58523CC2C0900680D7C /* RemoteVideoTrackProxy.h */, > 07E19F0523D4DC880094FFB4 /* TextTrackPrivateRemoteConfiguration.h */, >diff --git a/Source/WebKit/WebProcess/GPU/graphics/RemoteGraphicsContextGLProxy.cpp b/Source/WebKit/WebProcess/GPU/graphics/RemoteGraphicsContextGLProxy.cpp >index 9cadfc02f4ab6c8bc16358922573f997ffc4889e..f9efa1d17bac66d4c6aa22ba9f8e4c4565a18d66 100644 >--- a/Source/WebKit/WebProcess/GPU/graphics/RemoteGraphicsContextGLProxy.cpp >+++ b/Source/WebKit/WebProcess/GPU/graphics/RemoteGraphicsContextGLProxy.cpp >@@ -35,6 +35,10 @@ > #include "WebProcess.h" > #include <WebCore/ImageBuffer.h> > >+#if ENABLE(MEDIA_STREAM) >+#include "RemoteVideoFrameProxy.h" >+#endif >+ > namespace WebKit { > > using namespace WebCore; >@@ -118,6 +122,21 @@ void RemoteGraphicsContextGLProxy::paintCompositedResultsToCanvas(ImageBuffer& b > } > } > >+#if ENABLE(MEDIA_STREAM) >+RefPtr<WebCore::MediaSample> RemoteGraphicsContextGLProxy::paintCompositedResultsToMediaSample() >+{ >+ if (isContextLost()) >+ return nullptr; >+ auto mediaSample = RemoteVideoFrameProxy::create(*m_gpuProcessConnection); >+ auto sendResult = send(Messages::RemoteGraphicsContextGL::PaintCompositedResultsToMediaSample(mediaSample->identifier())); >+ if (!sendResult) { >+ markContextLost(); >+ return nullptr; >+ } >+ return mediaSample; >+} >+#endif >+ > bool RemoteGraphicsContextGLProxy::copyTextureFromMedia(MediaPlayer& mediaPlayer, PlatformGLObject texture, GCGLenum target, GCGLint level, GCGLenum internalFormat, GCGLenum format, GCGLenum type, bool premultiplyAlpha, bool flipY) > { > bool result = false; >diff --git a/Source/WebKit/WebProcess/GPU/graphics/RemoteGraphicsContextGLProxy.h b/Source/WebKit/WebProcess/GPU/graphics/RemoteGraphicsContextGLProxy.h >index 4ac1808e654f579b5137d4b1962cd4f94ab811c4..36415d9a33d3a1ece0271dc07932812172bed97d 100644 >--- a/Source/WebKit/WebProcess/GPU/graphics/RemoteGraphicsContextGLProxy.h >+++ b/Source/WebKit/WebProcess/GPU/graphics/RemoteGraphicsContextGLProxy.h >@@ -65,6 +65,9 @@ public: > void notifyMarkContextChanged() final; > void paintRenderingResultsToCanvas(WebCore::ImageBuffer&) final; > void paintCompositedResultsToCanvas(WebCore::ImageBuffer&) final; >+#if ENABLE(MEDIA_STREAM) >+ RefPtr<WebCore::MediaSample> paintCompositedResultsToMediaSample() final; >+#endif > void synthesizeGLError(GCGLenum error) final; > GCGLenum getError() final; > bool copyTextureFromMedia(WebCore::MediaPlayer&, PlatformGLObject texture, GCGLenum target, GCGLint level, GCGLenum internalFormat, GCGLenum format, GCGLenum type, bool premultiplyAlpha, bool flipY) final; >diff --git a/Source/WebKit/WebProcess/GPU/graphics/cocoa/RemoteGraphicsContextGLProxyCocoa.mm b/Source/WebKit/WebProcess/GPU/graphics/cocoa/RemoteGraphicsContextGLProxyCocoa.mm >index 1becdb16062cb4dec22a882f2350aadb3182b4e9..f1a7f9b8bac0f79a552c031bef8456a3a047aa89 100644 >--- a/Source/WebKit/WebProcess/GPU/graphics/cocoa/RemoteGraphicsContextGLProxyCocoa.mm >+++ b/Source/WebKit/WebProcess/GPU/graphics/cocoa/RemoteGraphicsContextGLProxyCocoa.mm >@@ -34,7 +34,6 @@ > #import <WebCore/CVUtilities.h> > #import <WebCore/GraphicsLayerContentsDisplayDelegate.h> > #import <WebCore/IOSurface.h> >-#import <WebCore/MediaSampleAVFObjC.h> > #import <WebCore/PlatformCALayer.h> > > namespace WebKit { >@@ -99,9 +98,6 @@ public: > #if ENABLE(VIDEO) && USE(AVFOUNDATION) > WebCore::GraphicsContextGLCV* asCV() final { return nullptr; } > #endif >-#if ENABLE(MEDIA_STREAM) >- RefPtr<WebCore::MediaSample> paintCompositedResultsToMediaSample() final; >-#endif > private: > RemoteGraphicsContextGLProxyCocoa(GPUProcessConnection& gpuProcessConnection, const WebCore::GraphicsContextGLAttributes& attributes, RenderingBackendIdentifier renderingBackend) > : RemoteGraphicsContextGLProxy(gpuProcessConnection, attributes, renderingBackend) >@@ -114,20 +110,6 @@ private: > friend class RemoteGraphicsContextGLProxy; > }; > >-#if ENABLE(MEDIA_STREAM) >-RefPtr<WebCore::MediaSample> RemoteGraphicsContextGLProxyCocoa::paintCompositedResultsToMediaSample() >-{ >- if (!m_displayBuffer) >- return nullptr; >- // FIXME: Implement remote MediaSample. >- auto surface = WebCore::IOSurface::createFromSendRight(m_displayBuffer.copySendRight(), WebCore::DestinationColorSpace::SRGB()); >- auto pixelBuffer = WebCore::createCVPixelBuffer(surface->surface()); >- if (!pixelBuffer) >- return nullptr; >- return WebCore::MediaSampleAVFObjC::createImageSample(WTFMove(*pixelBuffer), WebCore::MediaSampleAVFObjC::VideoRotation::UpsideDown, true); >-} >-#endif >- > void RemoteGraphicsContextGLProxyCocoa::prepareForDisplay() > { > if (isContextLost()) >diff --git a/Source/WebKit/WebProcess/GPU/media/RemoteVideoFrameIdentifier.h b/Source/WebKit/WebProcess/GPU/media/RemoteVideoFrameIdentifier.h >new file mode 100644 >index 0000000000000000000000000000000000000000..4077df0476a2c3a33bfd74357d733f222b758cc1 >--- /dev/null >+++ b/Source/WebKit/WebProcess/GPU/media/RemoteVideoFrameIdentifier.h >@@ -0,0 +1,39 @@ >+/* >+ * Copyright (C) 2022 Apple Inc. All rights reserved. >+ * >+ * Redistribution and use in source and binary forms, with or without >+ * modification, are permitted provided that the following conditions >+ * are met: >+ * 1. Redistributions of source code must retain the above copyright >+ * notice, this list of conditions and the following disclaimer. >+ * 2. Redistributions in binary form must reproduce the above copyright >+ * notice, this list of conditions and the following disclaimer in the >+ * documentation and/or other materials provided with the distribution. >+ * >+ * THIS SOFTWARE IS PROVIDED BY APPLE INC. AND ITS CONTRIBUTORS ``AS IS'' >+ * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, >+ * THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR >+ * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL APPLE INC. OR ITS CONTRIBUTORS >+ * BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR >+ * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF >+ * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS >+ * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN >+ * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) >+ * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF >+ * THE POSSIBILITY OF SUCH DAMAGE. >+ */ >+ >+#pragma once >+ >+#if ENABLE(GPU_PROCESS) && ENABLE(MEDIA_STREAM) >+ >+#include <wtf/ObjectIdentifier.h> >+ >+namespace WebKit { >+ >+enum RemoteVideoFrameIdentifierType { }; >+using RemoteVideoFrameIdentifier = ObjectIdentifier<RemoteVideoFrameIdentifierType>; >+ >+} // namespace WebKit >+ >+#endif >diff --git a/Source/WebKit/WebProcess/GPU/media/RemoteVideoFrameProxy.cpp b/Source/WebKit/WebProcess/GPU/media/RemoteVideoFrameProxy.cpp >new file mode 100644 >index 0000000000000000000000000000000000000000..bc163e40cd07945f7beecd88d8faab7703bd7218 >--- /dev/null >+++ b/Source/WebKit/WebProcess/GPU/media/RemoteVideoFrameProxy.cpp >@@ -0,0 +1,149 @@ >+/* >+ * Copyright (C) 2022 Apple Inc. All rights reserved. >+ * >+ * Redistribution and use in source and binary forms, with or without >+ * modification, are permitted provided that the following conditions >+ * are met: >+ * 1. Redistributions of source code must retain the above copyright >+ * notice, this list of conditions and the following disclaimer. >+ * 2. Redistributions in binary form must reproduce the above copyright >+ * notice, this list of conditions and the following disclaimer in the >+ * documentation and/or other materials provided with the distribution. >+ * >+ * THIS SOFTWARE IS PROVIDED BY APPLE INC. AND ITS CONTRIBUTORS ``AS IS'' >+ * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, >+ * THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR >+ * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL APPLE INC. OR ITS CONTRIBUTORS >+ * BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR >+ * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF >+ * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS >+ * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN >+ * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) >+ * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF >+ * THE POSSIBILITY OF SUCH DAMAGE. >+ */ >+ >+#include "config.h" >+#include "RemoteVideoFrameProxy.h" >+ >+#if ENABLE(GPU_PROCESS) && ENABLE(MEDIA_STREAM) >+#include "GPUConnectionToWebProcess.h" >+#include "RemoteVideoFrameObjectHeapMessages.h" >+#include "RemoteVideoFrameProxyMessages.h" >+ >+namespace WebKit { >+ >+RefPtr<RemoteVideoFrameProxy> RemoteVideoFrameProxy::create(GPUProcessConnection& gpuProcessConnection) >+{ >+ return adoptRef(new RemoteVideoFrameProxy(gpuProcessConnection, RemoteVideoFrameIdentifier::generate())); >+} >+ >+RemoteVideoFrameProxy::Properties RemoteVideoFrameProxy::properties(const WebCore::MediaSample& mediaSample) >+{ >+ return { >+ mediaSample.presentationTime(), >+ mediaSample.videoMirrored(), >+ mediaSample.videoRotation() >+ }; >+} >+ >+RemoteVideoFrameProxy::RemoteVideoFrameProxy(GPUProcessConnection& gpuProcessConnection, RemoteVideoFrameIdentifier identifier) >+ : m_gpuProcessConnection(&gpuProcessConnection) >+ , m_identifier(identifier) >+{ >+ m_gpuProcessConnection->addClient(*this); >+ m_gpuProcessConnection->messageReceiverMap().addMessageReceiver(Messages::RemoteVideoFrameProxy::messageReceiverName(), m_identifier.toUInt64(), *this); >+} >+ >+RemoteVideoFrameProxy::RemoteVideoFrameProxy() >+ : m_gpuProcessConnection(nullptr) >+ , m_identifier(RemoteVideoFrameIdentifier::generate()) >+{ >+} >+ >+RemoteVideoFrameProxy::~RemoteVideoFrameProxy() >+{ >+ disconnectGPUProcessIfNeeded(); >+} >+ >+RemoteVideoFrameIdentifier RemoteVideoFrameProxy::identifier() const >+{ >+ return m_identifier; >+} >+ >+MediaTime RemoteVideoFrameProxy::presentationTime() const >+{ >+ (void) waitForWasInitialized(); >+ return m_presentationTime; >+} >+ >+WebCore::PlatformSample RemoteVideoFrameProxy::platformSample() const >+{ >+ return { WebCore::PlatformSample::RemoteVideoFrameProxyType, { } }; >+} >+ >+MediaSample::VideoRotation RemoteVideoFrameProxy::videoRotation() const >+{ >+ (void) waitForWasInitialized(); >+ return m_videoRotation; >+} >+ >+bool RemoteVideoFrameProxy::videoMirrored() const >+{ >+ (void) waitForWasInitialized(); >+ return m_videoIsMirrored; >+} >+ >+uint32_t RemoteVideoFrameProxy::videoPixelFormat() const >+{ >+ // FIXME: Remove from the base class. >+ ASSERT_NOT_REACHED(); >+ return 0; >+} >+ >+std::optional<WebCore::MediaSampleVideoFrame> RemoteVideoFrameProxy::videoFrame() const >+{ >+ return std::nullopt; >+} >+ >+void RemoteVideoFrameProxy::disconnectGPUProcessIfNeeded() const >+{ >+ if (auto gpuProcessConnection = std::exchange(m_gpuProcessConnection, nullptr)) { >+ gpuProcessConnection->removeClient(*this); >+ gpuProcessConnection->messageReceiverMap().removeMessageReceiver(Messages::RemoteVideoFrameProxy::messageReceiverName(), m_identifier.toUInt64()); >+ if (m_wasInitialized) >+ gpuProcessConnection->connection().send(Messages::RemoteVideoFrameObjectHeap::ReleaseVideoFrame(m_identifier), 0, IPC::SendOption::DispatchMessageEvenWhenWaitingForSyncReply); >+ } >+} >+ >+bool RemoteVideoFrameProxy::waitForWasInitialized() const >+{ >+ if (m_wasInitialized) >+ return true; >+ if (m_gpuProcessConnection) { >+ if (m_gpuProcessConnection->connection().waitForAndDispatchImmediately<Messages::RemoteVideoFrameProxy::WasInitialized>(m_identifier.toUInt64(), defaultSendTimeout)) >+ return true; >+ disconnectGPUProcessIfNeeded(); >+ } >+ return false; >+} >+ >+void RemoteVideoFrameProxy::gpuProcessConnectionDidClose(GPUProcessConnection&) >+{ >+ auto gpuProcessConnection = std::exchange(m_gpuProcessConnection, nullptr); >+ gpuProcessConnection->removeClient(*this); >+ gpuProcessConnection->messageReceiverMap().removeMessageReceiver(Messages::RemoteVideoFrameProxy::messageReceiverName(), m_identifier.toUInt64()); >+} >+ >+void RemoteVideoFrameProxy::wasInitialized(std::optional<Properties>&& properties) >+{ >+ m_wasInitialized = true; >+ if (!properties) >+ return; >+ m_presentationTime = properties->presentationTime; >+ m_videoIsMirrored = properties->videoIsMirrored; >+ m_videoRotation = properties->videoRotation; >+} >+ >+} >+#endif >diff --git a/Source/WebKit/WebProcess/GPU/media/RemoteVideoFrameProxy.h b/Source/WebKit/WebProcess/GPU/media/RemoteVideoFrameProxy.h >new file mode 100644 >index 0000000000000000000000000000000000000000..a6b7fbcac57bd2314150e3bb3fe68291cb5a0e95 >--- /dev/null >+++ b/Source/WebKit/WebProcess/GPU/media/RemoteVideoFrameProxy.h >@@ -0,0 +1,126 @@ >+/* >+ * Copyright (C) 2022 Apple Inc. All rights reserved. >+ * >+ * Redistribution and use in source and binary forms, with or without >+ * modification, are permitted provided that the following conditions >+ * are met: >+ * 1. Redistributions of source code must retain the above copyright >+ * notice, this list of conditions and the following disclaimer. >+ * 2. Redistributions in binary form must reproduce the above copyright >+ * notice, this list of conditions and the following disclaimer in the >+ * documentation and/or other materials provided with the distribution. >+ * >+ * THIS SOFTWARE IS PROVIDED BY APPLE INC. AND ITS CONTRIBUTORS ``AS IS'' >+ * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, >+ * THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR >+ * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL APPLE INC. OR ITS CONTRIBUTORS >+ * BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR >+ * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF >+ * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS >+ * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN >+ * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) >+ * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF >+ * THE POSSIBILITY OF SUCH DAMAGE. >+ */ >+ >+#pragma once >+ >+#if ENABLE(GPU_PROCESS) && ENABLE(MEDIA_STREAM) >+ >+#include "GPUProcessConnection.h" >+#include "MessageReceiver.h" >+#include "RemoteVideoFrameIdentifier.h" >+#include <WebCore/VideoFrame.h> >+ >+namespace IPC { >+class Connection; >+class Decoder; >+} >+ >+namespace WebKit { >+ >+class GPUProcessConnection; >+ >+class RemoteVideoFrameProxy final : public WebCore::VideoFrame, private IPC::MessageReceiver, private GPUProcessConnection::Client { >+ WTF_MAKE_FAST_ALLOCATED; >+ WTF_MAKE_NONCOPYABLE(RemoteVideoFrameProxy); >+public: >+ // Creates a promise of a MediaSample that exists in GPU process. The object can be used as out parameter of the creation message. >+ static RefPtr<RemoteVideoFrameProxy> create(GPUProcessConnection&); >+ >+ struct Properties { >+ MediaTime presentationTime; >+ bool videoIsMirrored { false }; >+ VideoRotation videoRotation { VideoRotation::None }; >+ >+ template<typename Encoder> void encode(Encoder&) const; >+ template<typename Decoder> static std::optional<Properties> decode(Decoder&); >+ }; >+ >+ static Properties properties(const WebCore::MediaSample&); >+ >+ ~RemoteVideoFrameProxy() final; >+ >+ RemoteVideoFrameIdentifier identifier() const; >+ >+ // WebCore::VideoFrame overrides. >+ MediaTime presentationTime() const final; >+ VideoRotation videoRotation() const final; >+ bool videoMirrored() const final; >+ std::optional<WebCore::MediaSampleVideoFrame> videoFrame() const final; >+ // FIXME: When VideoFrame is not MediaSample, these will not be needed. >+ WebCore::PlatformSample platformSample() const final; >+ uint32_t videoPixelFormat() const final; >+ >+private: >+ RemoteVideoFrameProxy(GPUProcessConnection&, RemoteVideoFrameIdentifier); >+ RemoteVideoFrameProxy(); >+ void disconnectGPUProcessIfNeeded() const; >+ bool waitForWasInitialized() const WARN_UNUSED_RETURN; >+ >+ // IPC::MessageReceiver overrides. >+ void didReceiveMessage(IPC::Connection&, IPC::Decoder&) final; >+ >+ // GPUProcessConnection::Client overrides. >+ void gpuProcessConnectionDidClose(GPUProcessConnection&) final; >+ >+ // Messages. >+ void wasInitialized(std::optional<Properties>&&); >+ static inline Seconds defaultSendTimeout = 30_s; >+ >+ mutable GPUProcessConnection* m_gpuProcessConnection { nullptr }; >+ const RemoteVideoFrameIdentifier m_identifier; >+ mutable bool m_wasInitialized { false }; >+ MediaTime m_presentationTime; >+ bool m_videoIsMirrored { false }; >+ VideoRotation m_videoRotation { VideoRotation::None }; >+}; >+ >+template<typename Encoder> void RemoteVideoFrameProxy::Properties::encode(Encoder& encoder) const >+{ >+ encoder << presentationTime >+ << videoIsMirrored >+ << videoRotation; >+} >+ >+template<typename Decoder> std::optional<RemoteVideoFrameProxy::Properties> RemoteVideoFrameProxy::Properties::decode(Decoder& decoder) >+{ >+ std::optional<MediaTime> presentationTime; >+ std::optional<bool> videoIsMirrored; >+ std::optional<VideoRotation> videoRotation; >+ decoder >> presentationTime >+ >> videoIsMirrored >+ >> videoRotation; >+ if (!decoder.isValid()) >+ return std::nullopt; >+ return Properties { WTFMove(*presentationTime), *videoIsMirrored, *videoRotation }; >+} >+ >+} >+SPECIALIZE_TYPE_TRAITS_BEGIN(WebKit::RemoteVideoFrameProxy) >+ static bool isType(const WebCore::MediaSample& mediaSample) { return mediaSample.platformSample().type == WebCore::PlatformSample::RemoteVideoFrameProxyType; } >+SPECIALIZE_TYPE_TRAITS_END() >+SPECIALIZE_TYPE_TRAITS_BEGIN(WebCore::VideoFrame) >+ static bool isType(const WebCore::MediaSample& mediaSample) { return mediaSample.platformSample().type == WebCore::PlatformSample::RemoteVideoFrameProxyType; } >+SPECIALIZE_TYPE_TRAITS_END() >+#endif >diff --git a/Source/WebKit/WebProcess/GPU/media/RemoteVideoFrameProxy.messages.in b/Source/WebKit/WebProcess/GPU/media/RemoteVideoFrameProxy.messages.in >new file mode 100644 >index 0000000000000000000000000000000000000000..7b930064a61c8e63e56c2e36666c7c59740215e7 >--- /dev/null >+++ b/Source/WebKit/WebProcess/GPU/media/RemoteVideoFrameProxy.messages.in >@@ -0,0 +1,32 @@ >+/* >+ * Copyright (C) 2022 Apple Inc. All rights reserved. >+ * >+ * Redistribution and use in source and binary forms, with or without >+ * modification, are permitted provided that the following conditions >+ * are met: >+ * 1. Redistributions of source code must retain the above copyright >+ * notice, this list of conditions and the following disclaimer. >+ * 2. Redistributions in binary form must reproduce the above copyright >+ * notice, this list of conditions and the following disclaimer in the >+ * documentation and/or other materials provided with the distribution. >+ * >+ * THIS SOFTWARE IS PROVIDED BY APPLE INC. AND ITS CONTRIBUTORS ``AS IS'' >+ * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, >+ * THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR >+ * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL APPLE INC. OR ITS CONTRIBUTORS >+ * BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR >+ * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF >+ * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS >+ * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN >+ * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) >+ * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF >+ * THE POSSIBILITY OF SUCH DAMAGE. >+ */ >+ >+#if ENABLE(GPU_PROCESS) && ENABLE(MEDIA_STREAM) >+ >+messages -> RemoteVideoFrameProxy NotRefCounted { >+ WasInitialized(std::optional<WebKit::RemoteVideoFrameProxy::Properties> properties) >+} >+ >+#endif >diff --git a/Source/WebKit/WebProcess/GPU/webrtc/SampleBufferDisplayLayer.cpp b/Source/WebKit/WebProcess/GPU/webrtc/SampleBufferDisplayLayer.cpp >index 219a55df1c29fd1c6f933611a2925b5c3c06e170..9843cb3186e23b5b59bde2c3edd7a210765e5480 100644 >--- a/Source/WebKit/WebProcess/GPU/webrtc/SampleBufferDisplayLayer.cpp >+++ b/Source/WebKit/WebProcess/GPU/webrtc/SampleBufferDisplayLayer.cpp >@@ -34,6 +34,7 @@ > #include "RemoteSampleBufferDisplayLayerManagerMessages.h" > #include "RemoteSampleBufferDisplayLayerManagerMessagesReplies.h" > #include "RemoteSampleBufferDisplayLayerMessages.h" >+#include "RemoteVideoFrameProxy.h" > #include "SampleBufferDisplayLayerManager.h" > #include "WebCoreArgumentCoders.h" > #include "WebProcess.h" >@@ -49,12 +50,13 @@ std::unique_ptr<SampleBufferDisplayLayer> SampleBufferDisplayLayer::create(Sampl > > SampleBufferDisplayLayer::SampleBufferDisplayLayer(SampleBufferDisplayLayerManager& manager, WebCore::SampleBufferDisplayLayer::Client& client) > : WebCore::SampleBufferDisplayLayer(client) >+ , m_gpuProcessConnection(&WebProcess::singleton().ensureGPUProcessConnection()) > , m_manager(manager) >- , m_connection(WebProcess::singleton().ensureGPUProcessConnection().connection()) >+ , m_connection(m_gpuProcessConnection->connection()) > , m_identifier(SampleBufferDisplayLayerIdentifier::generate()) > { > manager.addLayer(*this); >- WebProcess::singleton().ensureGPUProcessConnection().addClient(*this); >+ m_gpuProcessConnection->addClient(*this); > } > > void SampleBufferDisplayLayer::initialize(bool hideRootLayer, IntSize size, CompletionHandler<void(bool)>&& callback) >@@ -78,7 +80,7 @@ void SampleBufferDisplayLayer::setLogIdentifier(String&& logIdentifier) > > SampleBufferDisplayLayer::~SampleBufferDisplayLayer() > { >- WebProcess::singleton().ensureGPUProcessConnection().removeClient(*this); >+ disconnectGPUProcessConnectionIfNeeded(); > m_connection->send(Messages::RemoteSampleBufferDisplayLayerManager::ReleaseLayer { m_identifier }, 0); > if (m_manager) > m_manager->removeLayer(*this); >@@ -140,6 +142,13 @@ void SampleBufferDisplayLayer::enqueueSample(MediaSample& sample) > { > if (m_paused) > return; >+ if (!m_gpuProcessConnection) >+ return; >+ if (is<RemoteVideoFrameProxy>(sample)) { >+ auto& remoteSample = downcast<RemoteVideoFrameProxy>(sample); >+ m_connection->send(Messages::RemoteSampleBufferDisplayLayer::EnqueueSample { remoteSample.identifier() }, m_identifier); >+ return; >+ } > > auto remoteSample = RemoteVideoSample::create(sample, RemoteVideoSample::ShouldCheckForIOSurface::No); > if (!remoteSample->surface()) { >@@ -148,7 +157,7 @@ void SampleBufferDisplayLayer::enqueueSample(MediaSample& sample) > return; > } > >- m_connection->send(Messages::RemoteSampleBufferDisplayLayer::EnqueueSample { *remoteSample }, m_identifier); >+ m_connection->send(Messages::RemoteSampleBufferDisplayLayer::EnqueueSampleCV { *remoteSample }, m_identifier); > } > > void SampleBufferDisplayLayer::clearEnqueuedSamples() >@@ -170,11 +179,18 @@ void SampleBufferDisplayLayer::setDidFail(bool value) > > void SampleBufferDisplayLayer::gpuProcessConnectionDidClose(GPUProcessConnection&) > { >+ disconnectGPUProcessConnectionIfNeeded(); > m_didFail = true; > if (m_client) > m_client->sampleBufferDisplayLayerStatusDidFail(); > } > >+void SampleBufferDisplayLayer::disconnectGPUProcessConnectionIfNeeded() >+{ >+ if (auto* gpuProcessConnection = std::exchange(m_gpuProcessConnection, nullptr)) >+ gpuProcessConnection->removeClient(*this); >+} >+ > } > > #endif >diff --git a/Source/WebKit/WebProcess/GPU/webrtc/SampleBufferDisplayLayer.h b/Source/WebKit/WebProcess/GPU/webrtc/SampleBufferDisplayLayer.h >index 56369addf0abf346b0d21c8506d69cb3d3b40e39..0444ecf2b8627944321113e69d26b12b255bfac0 100644 >--- a/Source/WebKit/WebProcess/GPU/webrtc/SampleBufferDisplayLayer.h >+++ b/Source/WebKit/WebProcess/GPU/webrtc/SampleBufferDisplayLayer.h >@@ -52,6 +52,7 @@ public: > > private: > SampleBufferDisplayLayer(SampleBufferDisplayLayerManager&, WebCore::SampleBufferDisplayLayer::Client&); >+ void disconnectGPUProcessConnectionIfNeeded(); > > // WebCore::SampleBufferDisplayLayer > void initialize(bool hideRootLayer, WebCore::IntSize, CompletionHandler<void(bool)>&&) final; >@@ -76,6 +77,7 @@ private: > void setDidFail(bool); > bool copySharedVideoFrame(CVPixelBufferRef); > >+ GPUProcessConnection* m_gpuProcessConnection; > WeakPtr<SampleBufferDisplayLayerManager> m_manager; > Ref<IPC::Connection> m_connection; > SampleBufferDisplayLayerIdentifier m_identifier; >diff --git a/Tools/ChangeLog b/Tools/ChangeLog >index d7019832e06b3570090ade05d59d6b2ffb41b019..1213b2edd8bc598cc284581d23df1fb1d934118d 100644 >--- a/Tools/ChangeLog >+++ b/Tools/ChangeLog >@@ -1,3 +1,13 @@ >+2022-02-04 Kimmo Kinnunen <kkinnunen@apple.com> >+ >+ Introduce a RemoteMediaSampleProxy to represent captured video frames used in Media Streams and present in GPUP >+ https://bugs.webkit.org/show_bug.cgi?id=236099 >+ >+ Reviewed by NOBODY (OOPS!). >+ >+ * Scripts/generate-gpup-webgl: >+ * TestWebKitAPI/Tests/WebCore/SampleMap.cpp: >+ > 2022-02-03 Myles C. Maxfield <mmaxfield@apple.com> > > Allow using STIX Two in layout tests >diff --git a/Tools/Scripts/generate-gpup-webgl b/Tools/Scripts/generate-gpup-webgl >index 30f7fd82b331a06f4c4e0cb4e3b89f75ffe6b181..d415bdbb8436cf9a49091e5b7393fec70be6eee3 100755 >--- a/Tools/Scripts/generate-gpup-webgl >+++ b/Tools/Scripts/generate-gpup-webgl >@@ -94,7 +94,10 @@ messages -> RemoteGraphicsContextGL NotRefCounted Stream {{ > #if PLATFORM(COCOA) > void PrepareForDisplay() -> (MachSendRight displayBuffer) Synchronous NotStreamEncodableReply > #endif >-#if !PLATFORM(COCOA) >+#if USE(GRAPHICS_LAYER_WC) >+ void PrepareForDisplay() -> (std::optional<WebKit::WCContentBufferIdentifier> contentBuffer) Synchronous >+#endif >+#if !PLATFORM(COCOA) && !USE(GRAPHICS_LAYER_WC) > void PrepareForDisplay() -> () Synchronous > #endif > void EnsureExtensionEnabled(String extension) >@@ -104,6 +107,9 @@ messages -> RemoteGraphicsContextGL NotRefCounted Stream {{ > void PaintRenderingResultsToCanvas(WebCore::RenderingResourceIdentifier imageBuffer) -> () Synchronous > void PaintCompositedResultsToCanvas(WebCore::RenderingResourceIdentifier imageBuffer) -> () Synchronous > void CopyTextureFromMedia(WebCore::MediaPlayerIdentifier identifier, uint32_t texture, uint32_t target, int32_t level, uint32_t internalFormat, uint32_t format, uint32_t type, bool premultiplyAlpha, bool flipY) -> (bool success) Synchronous >+#if ENABLE(MEDIA_STREAM) >+ void PaintCompositedResultsToMediaSample(WebKit::RemoteVideoFrameIdentifier) >+#endif > void SimulateEventForTesting(WebCore::GraphicsContextGL::SimulatedEventForTesting event) > {} > }} >@@ -232,7 +238,7 @@ def create_cpp_type_container(type_name : str) -> Optional[cpp_type_container]: > return None > container_name = m[1] > # All templates are containers except these below. >- if container_name in ['RefPtr', 'Ref']: >+ if container_name in ['RefPtr', 'Ref', 'RetainPtr', 'std::optional']: > return None > templates = m[2] > arity = None >@@ -640,7 +646,13 @@ class webkit_ipc_cpp_proxy_impl(object): > webkit_ipc_type = webkit_ipc_types[o.type] > assert(isinstance(webkit_ipc_type, cpp_type_container)) > v = cpp_arg(webkit_ipc_type, o.name + "Reply") >- self.pre_call_stmts += [f"{str(v.type)} {str(v.name)};"] >+ if o.type.is_dynamic_span(): >+ self.pre_call_stmts += [f"{str(v.type)} {str(v.name)};"] >+ else: >+ # FIXME: Likely this creates a dangling reference span<int, 2> aReply = { { } };. >+ # https://bugs.webkit.org/show_bug.cgi?id=235889 >+ self.pre_call_stmts += [f"{str(v.type)} {str(v.name)} {{ {{ }} }};"] >+ > self.out_exprs += [cpp_expr(v.type, v.name)] > if o.type.is_dynamic_span(): > self.in_exprs += [cpp_expr(get_cpp_type("size_t"), f"{o.name}.bufSize")] >diff --git a/Tools/TestWebKitAPI/Tests/WebCore/SampleMap.cpp b/Tools/TestWebKitAPI/Tests/WebCore/SampleMap.cpp >index d146cec4f2ec4cf6912d7f048d288157adfdc1b8..c45c67a3166bffb9877bc0d834fef844a4866519 100644 >--- a/Tools/TestWebKitAPI/Tests/WebCore/SampleMap.cpp >+++ b/Tools/TestWebKitAPI/Tests/WebCore/SampleMap.cpp >@@ -57,7 +57,6 @@ public: > MediaTime decodeTime() const final { return m_decodeTime; } > MediaTime duration() const final { return m_duration; } > AtomString trackID() const final { return m_trackID; } >- void setTrackID(const String& trackID) final { m_trackID = trackID; } > size_t sizeInBytes() const final { return m_sizeInBytes; } > FloatSize presentationSize() const final { return m_presentationSize; } > void offsetTimestampsBy(const MediaTime& offset) final { m_presentationTime += offset; m_decodeTime += offset; } >@@ -71,7 +70,7 @@ public: > return create(m_presentationTime, m_decodeTime, m_duration, static_cast<SampleFlags>(m_flags | IsNonDisplaying)); > } > SampleFlags flags() const final { return m_flags; } >- PlatformSample platformSample() final { return { PlatformSample::None, {nullptr}}; } >+ PlatformSample platformSample() const final { return { PlatformSample::None, { nullptr } }; } > std::optional<ByteRange> byteRange() const final { return std::nullopt; } > > void dump(PrintStream&) const final { }
<b>You cannot view the attachment while viewing its details because your browser does not support IFRAMEs. <a href="attachment.cgi?id=450890">View the attachment on a separate page</a>.</b>
View Attachment As Diff
View Attachment As Raw
Flags:
ews-feeder
:
commit-queue-
Actions:
View
|
Formatted Diff
|
Diff
Attachments on
bug 236099
:
450810
|
450890
|
450894
|
451102
|
451103
|
451106
|
451217
|
451346
|
451405
|
451662
|
451663
|
451667
|
451669
|
451682