WebKit Bugzilla
Attachment 356932 Details for
Bug 192069
: Implement non-timeslice mode encoding for MediaRecorder
Home
|
New
|
Browse
|
Search
|
[?]
|
Reports
|
Requests
|
Help
|
New Account
|
Log In
Remember
[x]
|
Forgot Password
Login:
[x]
[patch]
Patch
bug-192069-20181209163133.patch (text/plain), 56.21 KB, created by
Wendy
on 2018-12-09 16:31:34 PST
(
hide
)
Description:
Patch
Filename:
MIME Type:
Creator:
Wendy
Created:
2018-12-09 16:31:34 PST
Size:
56.21 KB
patch
obsolete
>Subversion Revision: 238913 >diff --git a/Source/WebCore/ChangeLog b/Source/WebCore/ChangeLog >index d3e4d463aae9ad658212c0272f99687a3d77b2f3..f3068cd382250408198d37c9d5994e12cb953c06 100644 >--- a/Source/WebCore/ChangeLog >+++ b/Source/WebCore/ChangeLog >@@ -1,3 +1,60 @@ >+2018-12-08 YUHAN WU <yuhan_wu@apple.com> >+ >+ Implement non-timeslice mode encoding for MediaRecorder >+ https://bugs.webkit.org/show_bug.cgi?id=192069 >+ <rdar://problem/46443290> >+ >+ Reviewed by NOBODY (OOPS!). >+ >+ Implement the encoding for non-timeslice mode of MediaRecorder. >+ It only supports to record MP4 file through H264 and AAC encoding, we will need to support more MIME types and encoding methods. >+ Add a API in internals to allow testings to turn on the mock source. >+ >+ Test: http/wpt/mediarecorder/MediaRecorder-AV-audio-video-dataavailable.html >+ >+ * Modules/mediarecorder/MediaRecorder.cpp: >+ (WebCore::MediaRecorder::create): >+ (WebCore::MediaRecorder::setCustomPrivateRecorderCreator): >+ (WebCore::MediaRecorder::getPrivateImpl): >+ (WebCore::MediaRecorder::MediaRecorder): >+ (WebCore::MediaRecorder::stopRecording): >+ (WebCore::MediaRecorder::stopRecordingInternal): >+ (WebCore::MediaRecorder::createRecordingDataBlob): >+ (WebCore::MediaRecorder::scheduleDeferredTask): >+ * Modules/mediarecorder/MediaRecorder.h: >+ * Modules/mediarecorder/MediaRecorder.idl: >+ * SourcesCocoa.txt: >+ * WebCore.xcodeproj/project.pbxproj: >+ * platform/mediarecorder/MediaRecorderPrivate.h: >+ (WebCore::MediaRecorderPrivate::stopRecording): >+ * platform/mediarecorder/MediaRecorderPrivateAVFImpl.cpp: Added. >+ (WebCore::MediaRecorderPrivateAVFImpl::create): >+ (WebCore::MediaRecorderPrivateAVFImpl::MediaRecorderPrivateAVFImpl): >+ (WebCore::MediaRecorderPrivateAVFImpl::sampleBufferUpdated): >+ (WebCore::MediaRecorderPrivateAVFImpl::audioSamplesAvailable): >+ (WebCore::MediaRecorderPrivateAVFImpl::stopRecording): >+ (WebCore::MediaRecorderPrivateAVFImpl::fetchData): >+ (WebCore::MediaRecorderPrivateAVFImpl::mimeType): >+ * platform/mediarecorder/MediaRecorderPrivateAVFImpl.h: Added. >+ * platform/mediarecorder/MediaRecorderPrivateMock.cpp: >+ (WebCore::MediaRecorderPrivateMock::fetchData): >+ (WebCore::MediaRecorderPrivateMock::mimeType): >+ * platform/mediarecorder/MediaRecorderPrivateMock.h: >+ * platform/mediarecorder/cocoa/MediaRecorderPrivateWriterCocoa.h: added. >+ * platform/mediarecorder/cocoa/MediaRecorderPrivateWriterCocoa.mm: Added. >+ (WebCore::MediaRecorderPrivateWriter::setupWriter): >+ (WebCore::MediaRecorderPrivateWriter::setVideoInput): >+ (WebCore::MediaRecorderPrivateWriter::setAudioInput): >+ (WebCore::copySampleBufferWithCurrentTimeStamp): >+ (WebCore::MediaRecorderPrivateWriter::appendVideoSampleBuffer): >+ (WebCore::MediaRecorderPrivateWriter::appendAudioSampleBuffer): >+ (WebCore::MediaRecorderPrivateWriter::stopRecording): >+ * testing/Internals.cpp: >+ (WebCore::createRecorderMockSource): >+ (WebCore::Internals::setCustomPrivateRecorderCreator): >+ * testing/Internals.h: >+ * testing/Internals.idl: >+ > 2018-12-05 Don Olmstead <don.olmstead@sony.com> > > [PlayStation] Enable WebCore >diff --git a/Source/WebCore/Modules/mediarecorder/MediaRecorder.cpp b/Source/WebCore/Modules/mediarecorder/MediaRecorder.cpp >index 8f6bcf746b6a476b76fbc4abddc6559b009cb9c5..a78d4be7ef990971f09d98e4dec536dbdd9b19b9 100644 >--- a/Source/WebCore/Modules/mediarecorder/MediaRecorder.cpp >+++ b/Source/WebCore/Modules/mediarecorder/MediaRecorder.cpp >@@ -33,22 +33,48 @@ > #include "Document.h" > #include "EventNames.h" > #include "MediaRecorderErrorEvent.h" >-#include "MediaRecorderPrivateMock.h" >+#include "MediaRecorderPrivate.h" >+#include "SharedBuffer.h" >+ >+#if PLATFORM(COCOA) >+#include "MediaRecorderPrivateAVFImpl.h" >+#endif > > namespace WebCore { > >-Ref<MediaRecorder> MediaRecorder::create(Document& document, Ref<MediaStream>&& stream, Options&& options) >+creatorFunction MediaRecorder::m_customCreator = nullptr; >+ >+ExceptionOr<Ref<MediaRecorder>> MediaRecorder::create(Document& document, Ref<MediaStream>&& stream, Options&& options) > { >- auto recorder = adoptRef(*new MediaRecorder(document, WTFMove(stream), WTFMove(options))); >+ auto privateInstance = MediaRecorder::getPrivateImpl(stream->privateStream()); >+ if (!privateInstance) >+ return Exception { NotSupportedError, "The MediaRecorder is unsupported on this platform"_s }; >+ auto recorder = adoptRef(*new MediaRecorder(document, WTFMove(stream), WTFMove(privateInstance), WTFMove(options))); > recorder->suspendIfNeeded(); >- return recorder; >+ return WTFMove(recorder); >+} >+ >+void MediaRecorder::setCustomPrivateRecorderCreator(creatorFunction creator) >+{ >+ m_customCreator = creator; > } > >-MediaRecorder::MediaRecorder(Document& document, Ref<MediaStream>&& stream, Options&& option) >+std::unique_ptr<MediaRecorderPrivate> MediaRecorder::getPrivateImpl(const MediaStreamPrivate& stream) >+{ >+ if (m_customCreator) >+ return m_customCreator(); >+ >+#if PLATFORM(COCOA) >+ return MediaRecorderPrivateAVFImpl::create(stream); >+#endif >+ return nullptr; >+} >+ >+MediaRecorder::MediaRecorder(Document& document, Ref<MediaStream>&& stream, std::unique_ptr<MediaRecorderPrivate>&& privateImpl, Options&& option) > : ActiveDOMObject(&document) > , m_options(WTFMove(option)) > , m_stream(WTFMove(stream)) >- , m_private(makeUniqueRef<MediaRecorderPrivateMock>()) // FIXME: we will need to decide which MediaRecorderPrivate instance to create based on the mock enabled feature flag >+ , m_private(WTFMove(privateImpl)) > { > m_tracks = WTF::map(m_stream->getTracks(), [] (auto&& track) -> Ref<MediaStreamTrackPrivate> { > return track->privateTrack(); >@@ -95,14 +121,14 @@ ExceptionOr<void> MediaRecorder::stopRecording() > { > if (state() == RecordingState::Inactive) > return Exception { InvalidStateError, "The MediaRecorder's state cannot be inactive"_s }; >- >+ > scheduleDeferredTask([this] { > if (!m_isActive || state() == RecordingState::Inactive) > return; > > stopRecordingInternal(); > ASSERT(m_state == RecordingState::Inactive); >- dispatchEvent(BlobEvent::create(eventNames().dataavailableEvent, Event::CanBubble::No, Event::IsCancelable::No, m_private->fetchData())); >+ dispatchEvent(BlobEvent::create(eventNames().dataavailableEvent, Event::CanBubble::No, Event::IsCancelable::No, createRecordingDataBlob())); > if (!m_isActive) > return; > dispatchEvent(Event::create(eventNames().stopEvent, Event::CanBubble::No, Event::IsCancelable::No)); >@@ -119,6 +145,15 @@ void MediaRecorder::stopRecordingInternal() > track->removeObserver(*this); > > m_state = RecordingState::Inactive; >+ m_private->stopRecording(); >+} >+ >+Ref<Blob> MediaRecorder::createRecordingDataBlob() >+{ >+ auto data = m_private->fetchData(); >+ if (!data) >+ return Blob::create(); >+ return Blob::create(*data, m_private->mimeType()); > } > > void MediaRecorder::didAddOrRemoveTrack() >@@ -164,6 +199,7 @@ void MediaRecorder::scheduleDeferredTask(Function<void()>&& function) > auto* scriptExecutionContext = this->scriptExecutionContext(); > if (!scriptExecutionContext) > return; >+ > scriptExecutionContext->postTask([protectedThis = makeRef(*this), function = WTFMove(function)] (auto&) { > function(); > }); >diff --git a/Source/WebCore/Modules/mediarecorder/MediaRecorder.h b/Source/WebCore/Modules/mediarecorder/MediaRecorder.h >index 7c8aea5772c671fdbc77bae2540393b634270fdf..8c5a6112c5386f46378c05e446ff27b6459e1b1f 100644 >--- a/Source/WebCore/Modules/mediarecorder/MediaRecorder.h >+++ b/Source/WebCore/Modules/mediarecorder/MediaRecorder.h >@@ -34,9 +34,12 @@ > > namespace WebCore { > >+class Blob; > class Document; > class MediaRecorderPrivate; > >+typedef std::unique_ptr<MediaRecorderPrivate>(*creatorFunction)(); >+ > class MediaRecorder final > : public ActiveDOMObject > , public RefCounted<MediaRecorder> >@@ -56,7 +59,9 @@ public: > > ~MediaRecorder(); > >- static Ref<MediaRecorder> create(Document&, Ref<MediaStream>&&, Options&& = { }); >+ static ExceptionOr<Ref<MediaRecorder>> create(Document&, Ref<MediaStream>&&, Options&& = { }); >+ >+ WEBCORE_EXPORT static void setCustomPrivateRecorderCreator(creatorFunction); > > RecordingState state() const { return m_state; } > >@@ -67,8 +72,12 @@ public: > ExceptionOr<void> stopRecording(); > > private: >- MediaRecorder(Document&, Ref<MediaStream>&&, Options&& = { }); >+ MediaRecorder(Document&, Ref<MediaStream>&&, std::unique_ptr<MediaRecorderPrivate>&&, Options&& = { }); > >+ static std::unique_ptr<MediaRecorderPrivate> getPrivateImpl(const MediaStreamPrivate&); >+ >+ Ref<Blob> createRecordingDataBlob(); >+ > // EventTarget > void refEventTarget() final { ref(); } > void derefEventTarget() final { deref(); } >@@ -96,15 +105,17 @@ private: > void scheduleDeferredTask(Function<void()>&&); > void setNewRecordingState(RecordingState, Ref<Event>&&); > >+ static creatorFunction m_customCreator; >+ > Options m_options; > Ref<MediaStream> m_stream; >- UniqueRef<MediaRecorderPrivate> m_private; >+ std::unique_ptr<MediaRecorderPrivate> m_private; > RecordingState m_state { RecordingState::Inactive }; > Vector<Ref<MediaStreamTrackPrivate>> m_tracks; > > bool m_isActive { true }; > }; >- >+ > } // namespace WebCore > > #endif // ENABLE(MEDIA_STREAM) >diff --git a/Source/WebCore/Modules/mediarecorder/MediaRecorder.idl b/Source/WebCore/Modules/mediarecorder/MediaRecorder.idl >index 469738033a0fe53b54d0cc1b5194119cc448a791..8c249b1f3adce89518021c4ca38bb36171bb3ca9 100644 >--- a/Source/WebCore/Modules/mediarecorder/MediaRecorder.idl >+++ b/Source/WebCore/Modules/mediarecorder/MediaRecorder.idl >@@ -29,6 +29,7 @@ enum RecordingState { "inactive", "recording", "paused" }; > Conditional=MEDIA_STREAM, > Constructor(MediaStream stream, optional MediaRecorderOptions options), > ConstructorCallWith=Document, >+ ConstructorMayThrowException, > EnabledAtRuntime=MediaRecorder, > Exposed=Window > ] interface MediaRecorder : EventTarget { >diff --git a/Source/WebCore/SourcesCocoa.txt b/Source/WebCore/SourcesCocoa.txt >index 52bb2fc2d7a8e87d41f9ce1601ed8157462029cf..4fefca3cbd1cdccde3ab418a09d7fe5a14741279 100644 >--- a/Source/WebCore/SourcesCocoa.txt >+++ b/Source/WebCore/SourcesCocoa.txt >@@ -480,6 +480,9 @@ platform/mac/WebNSAttributedStringExtras.mm > platform/mac/WebPlaybackControlsManager.mm > platform/mac/WidgetMac.mm > >+platform/mediarecorder/MediaRecorderPrivateAVFImpl.cpp >+platform/mediarecorder/cocoa/MediaRecorderPrivateWriterCocoa.mm >+ > platform/mediasession/mac/MediaSessionInterruptionProviderMac.mm > > platform/mediastream/ios/AVAudioSessionCaptureDevice.mm >diff --git a/Source/WebCore/WebCore.xcodeproj/project.pbxproj b/Source/WebCore/WebCore.xcodeproj/project.pbxproj >index bea0e3e9df918d4ebab3f38ea6d3446ff3a47fd7..7a26ffcd9eb754841d82e885db6127135e565da9 100644 >--- a/Source/WebCore/WebCore.xcodeproj/project.pbxproj >+++ b/Source/WebCore/WebCore.xcodeproj/project.pbxproj >@@ -1392,6 +1392,8 @@ > 4D3B00AB215D69A70076B983 /* MediaRecorder.h in Headers */ = {isa = PBXBuildFile; fileRef = 4D3B00A9215D69A70076B983 /* MediaRecorder.h */; }; > 4D3B00AF215D6A690076B983 /* BlobEvent.h in Headers */ = {isa = PBXBuildFile; fileRef = 4D3B00AD215D6A690076B983 /* BlobEvent.h */; }; > 4D3B5016217E58B700665DB1 /* MediaRecorderPrivate.h in Headers */ = {isa = PBXBuildFile; fileRef = 4D3B5014217E58B700665DB1 /* MediaRecorderPrivate.h */; }; >+ 4D73F946218BC5FA003A3ED6 /* MediaRecorderPrivateAVFImpl.h in Headers */ = {isa = PBXBuildFile; fileRef = 4D73F944218BC5FA003A3ED6 /* MediaRecorderPrivateAVFImpl.h */; }; >+ 4D73F94E218C4A87003A3ED6 /* MediaRecorderPrivateWriterCocoa.h in Headers */ = {isa = PBXBuildFile; fileRef = 4D73F94C218C4A87003A3ED6 /* MediaRecorderPrivateWriterCocoa.h */; }; > 4DB7130D216ECB4D0096A4DD /* MediaRecorderErrorEvent.h in Headers */ = {isa = PBXBuildFile; fileRef = 4DB7130C216EC2BD0096A4DD /* MediaRecorderErrorEvent.h */; }; > 4E1959220A39DABA00220FE5 /* MediaFeatureNames.h in Headers */ = {isa = PBXBuildFile; fileRef = 4E1959200A39DABA00220FE5 /* MediaFeatureNames.h */; settings = {ATTRIBUTES = (Private, ); }; }; > 4E19592A0A39DACC00220FE5 /* MediaQuery.h in Headers */ = {isa = PBXBuildFile; fileRef = 4E1959240A39DACC00220FE5 /* MediaQuery.h */; settings = {ATTRIBUTES = (Private, ); }; }; >@@ -7937,6 +7939,10 @@ > 4D3B00A9215D69A70076B983 /* MediaRecorder.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = MediaRecorder.h; sourceTree = "<group>"; }; > 4D3B00AD215D6A690076B983 /* BlobEvent.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = BlobEvent.h; sourceTree = "<group>"; }; > 4D3B5014217E58B700665DB1 /* MediaRecorderPrivate.h */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.c.h; path = MediaRecorderPrivate.h; sourceTree = "<group>"; }; >+ 4D73F944218BC5FA003A3ED6 /* MediaRecorderPrivateAVFImpl.h */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.c.h; path = MediaRecorderPrivateAVFImpl.h; sourceTree = "<group>"; }; >+ 4D73F945218BC5FA003A3ED6 /* MediaRecorderPrivateAVFImpl.cpp */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.cpp.cpp; path = MediaRecorderPrivateAVFImpl.cpp; sourceTree = "<group>"; }; >+ 4D73F94C218C4A87003A3ED6 /* MediaRecorderPrivateWriterCocoa.h */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.c.h; path = MediaRecorderPrivateWriterCocoa.h; sourceTree = "<group>"; }; >+ 4D73F94D218C4A87003A3ED6 /* MediaRecorderPrivateWriterCocoa.mm */ = {isa = PBXFileReference; explicitFileType = sourcecode.cpp.objcpp; path = MediaRecorderPrivateWriterCocoa.mm; sourceTree = "<group>"; wrapsLines = 0; }; > 4D7EB3F4217C6AE600D64888 /* BlobEvent.cpp */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.cpp.cpp; path = BlobEvent.cpp; sourceTree = "<group>"; }; > 4D9F6B642182532B0092A9C5 /* MediaRecorderPrivateMock.h */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.c.h; path = MediaRecorderPrivateMock.h; sourceTree = "<group>"; }; > 4D9F6B652182532B0092A9C5 /* MediaRecorderPrivateMock.cpp */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.cpp.cpp; path = MediaRecorderPrivateMock.cpp; sourceTree = "<group>"; }; >@@ -18159,13 +18165,25 @@ > 4D3B5012217E58A300665DB1 /* mediarecorder */ = { > isa = PBXGroup; > children = ( >+ 4D3C05D421AF480900F2890A /* cocoa */, > 4D3B5014217E58B700665DB1 /* MediaRecorderPrivate.h */, >+ 4D73F945218BC5FA003A3ED6 /* MediaRecorderPrivateAVFImpl.cpp */, >+ 4D73F944218BC5FA003A3ED6 /* MediaRecorderPrivateAVFImpl.h */, > 4D9F6B652182532B0092A9C5 /* MediaRecorderPrivateMock.cpp */, > 4D9F6B642182532B0092A9C5 /* MediaRecorderPrivateMock.h */, > ); > path = mediarecorder; > sourceTree = "<group>"; > }; >+ 4D3C05D421AF480900F2890A /* cocoa */ = { >+ isa = PBXGroup; >+ children = ( >+ 4D73F94C218C4A87003A3ED6 /* MediaRecorderPrivateWriterCocoa.h */, >+ 4D73F94D218C4A87003A3ED6 /* MediaRecorderPrivateWriterCocoa.mm */, >+ ); >+ path = cocoa; >+ sourceTree = "<group>"; >+ }; > 510310421BA8C64C003329C0 /* client */ = { > isa = PBXGroup; > children = ( >@@ -30133,6 +30151,8 @@ > 4D3B00AB215D69A70076B983 /* MediaRecorder.h in Headers */, > 4DB7130D216ECB4D0096A4DD /* MediaRecorderErrorEvent.h in Headers */, > 4D3B5016217E58B700665DB1 /* MediaRecorderPrivate.h in Headers */, >+ 4D73F946218BC5FA003A3ED6 /* MediaRecorderPrivateAVFImpl.h in Headers */, >+ 4D73F94E218C4A87003A3ED6 /* MediaRecorderPrivateWriterCocoa.h in Headers */, > C90843D01B18E47D00B68564 /* MediaRemoteControls.h in Headers */, > CD8ACA8F1D23971900ECC59E /* MediaRemoteSoftLink.h in Headers */, > CEEFCD7A19DB31F7003876D7 /* MediaResourceLoader.h in Headers */, >diff --git a/Source/WebCore/platform/mediarecorder/MediaRecorderPrivate.h b/Source/WebCore/platform/mediarecorder/MediaRecorderPrivate.h >index 75fb09e2a650eb58c90532dcd2e9091bb3876013..bbe8a7bf5daf990316de520c4b9d80cab8c415ef 100644 >--- a/Source/WebCore/platform/mediarecorder/MediaRecorderPrivate.h >+++ b/Source/WebCore/platform/mediarecorder/MediaRecorderPrivate.h >@@ -35,18 +35,20 @@ class MediaTime; > namespace WebCore { > > class AudioStreamDescription; >-class Blob; >-class PlatformAudioData; > class MediaSample; > class MediaStreamTrackPrivate; >+class PlatformAudioData; >+class SharedBuffer; > > class MediaRecorderPrivate { > public: > virtual void sampleBufferUpdated(MediaStreamTrackPrivate&, MediaSample&) = 0; > virtual void audioSamplesAvailable(MediaStreamTrackPrivate&, const WTF::MediaTime&, const PlatformAudioData&, const AudioStreamDescription&, size_t) = 0; > >- virtual Ref<Blob> fetchData() = 0; >+ virtual RefPtr<SharedBuffer> fetchData() = 0; >+ virtual const String& mimeType() = 0; > virtual ~MediaRecorderPrivate() = default; >+ virtual void stopRecording() { } > }; > > } // namespace WebCore >diff --git a/Source/WebCore/platform/mediarecorder/MediaRecorderPrivateAVFImpl.cpp b/Source/WebCore/platform/mediarecorder/MediaRecorderPrivateAVFImpl.cpp >new file mode 100644 >index 0000000000000000000000000000000000000000..3b9522887617a49a1d1c8958cc40db2b5d876433 >--- /dev/null >+++ b/Source/WebCore/platform/mediarecorder/MediaRecorderPrivateAVFImpl.cpp >@@ -0,0 +1,123 @@ >+/* >+ * Copyright (C) 2018 Apple Inc. All rights reserved. >+ * >+ * Redistribution and use in source and binary forms, with or without >+ * modification, are permitted provided that the following conditions >+ * are met: >+ * 1. Redistributions of source code must retain the above copyright >+ * notice, this list of conditions and the following disclaimer. >+ * 2. Redistributions in binary form must reproduce the above copyright >+ * notice, this list of conditions and the following disclaimer in the >+ * documentation and/or other materials provided with the distribution. >+ * >+ * THIS SOFTWARE IS PROVIDED BY APPLE INC. AND ITS CONTRIBUTORS ``AS IS'' AND ANY >+ * EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED >+ * WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE >+ * DISCLAIMED. IN NO EVENT SHALL APPLE INC. OR ITS CONTRIBUTORS BE LIABLE FOR ANY >+ * DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES >+ * (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; >+ * LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON >+ * ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT >+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS >+ * SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. >+ */ >+ >+ >+#include "config.h" >+#include "MediaRecorderPrivateAVFImpl.h" >+ >+#if ENABLE(MEDIA_STREAM) >+ >+#include "AudioStreamDescription.h" >+#include "MediaSample.h" >+#include "MediaStreamPrivate.h" >+#include "SharedBuffer.h" >+#include "WebAudioBufferList.h" >+ >+namespace WebCore { >+ >+std::unique_ptr<MediaRecorderPrivateAVFImpl> MediaRecorderPrivateAVFImpl::create(const MediaStreamPrivate& stream) >+{ >+ auto instance = std::unique_ptr<MediaRecorderPrivateAVFImpl>(new MediaRecorderPrivateAVFImpl(stream)); >+ if (!instance->m_isWriterReady) >+ return nullptr; >+ return instance; >+} >+ >+MediaRecorderPrivateAVFImpl::MediaRecorderPrivateAVFImpl(const MediaStreamPrivate& stream) >+{ >+ if (!m_writer.setupWriter()) >+ return; >+ auto tracks = stream.tracks(); >+ bool videoSelected = false; >+ bool audioSelected = false; >+ for (auto& track : tracks) { >+ if (!track->enabled() || track->ended()) >+ continue; >+ switch (track->type()) { >+ case RealtimeMediaSource::Type::Video: { >+ auto& settings = track->settings(); >+ if (videoSelected || !settings.supportsWidth() || !settings.supportsHeight()) >+ break; >+ // FIXME: we will need to implement support for multiple video tracks, currently we only choose the first track as the recorded track. >+ // FIXME: we would better to throw an exception to JavaScript if setVideoInput failed >+ if (!m_writer.setVideoInput(settings.width(), settings.height())) >+ return; >+ m_recordedVideoTrackID = track->id(); >+ videoSelected = true; >+ break; >+ } >+ case RealtimeMediaSource::Type::Audio: { >+ if (audioSelected) >+ break; >+ // FIXME: we will need to implement support for multiple audio tracks, currently we only choose the first track as the recorded track. >+ // FIXME: we would better to throw an exception to JavaScript if setAudioInput failed >+ if (!m_writer.setAudioInput()) >+ return; >+ m_recordedAudioTrackID = track->id(); >+ audioSelected = true; >+ break; >+ } >+ case RealtimeMediaSource::Type::None: >+ break; >+ } >+ } >+ m_isWriterReady = true; >+} >+ >+void MediaRecorderPrivateAVFImpl::sampleBufferUpdated(MediaStreamTrackPrivate& track, MediaSample& sampleBuffer) >+{ >+ if (track.id() != m_recordedVideoTrackID) >+ return; >+ m_writer.appendVideoSampleBuffer(sampleBuffer.platformSample().sample.cmSampleBuffer); >+} >+ >+void MediaRecorderPrivateAVFImpl::audioSamplesAvailable(MediaStreamTrackPrivate& track, const WTF::MediaTime& mediaTime, const PlatformAudioData& data, const AudioStreamDescription& description, size_t sampleCount) >+{ >+ if (track.id() != m_recordedAudioTrackID) >+ return; >+ ASSERT(is<WebAudioBufferList>(data)); >+ ASSERT(description.platformDescription().type == PlatformDescription::CAAudioStreamBasicType); >+ m_writer.appendAudioSampleBuffer(data, description, mediaTime, sampleCount); >+} >+ >+void MediaRecorderPrivateAVFImpl::stopRecording() >+{ >+ m_writer.stopRecording(); >+} >+ >+RefPtr<SharedBuffer> MediaRecorderPrivateAVFImpl::fetchData() >+{ >+ return m_writer.fetchData(); >+} >+ >+const String& MediaRecorderPrivateAVFImpl::mimeType() >+{ >+ static NeverDestroyed<const String> mp4MimeType(MAKE_STATIC_STRING_IMPL("video/mp4")); >+ // FIXME: we will need to support more MIME types. >+ return mp4MimeType; >+} >+ >+} // namespace WebCore >+ >+#endif // ENABLE(MEDIA_STREAM) >diff --git a/Source/WebCore/platform/mediarecorder/MediaRecorderPrivateAVFImpl.h b/Source/WebCore/platform/mediarecorder/MediaRecorderPrivateAVFImpl.h >new file mode 100644 >index 0000000000000000000000000000000000000000..2f923c247d6c24fe604f772dea68e50498e2026e >--- /dev/null >+++ b/Source/WebCore/platform/mediarecorder/MediaRecorderPrivateAVFImpl.h >@@ -0,0 +1,59 @@ >+/* >+ * Copyright (C) 2018 Apple Inc. All rights reserved. >+ * >+ * Redistribution and use in source and binary forms, with or without >+ * modification, are permitted provided that the following conditions >+ * are met: >+ * 1. Redistributions of source code must retain the above copyright >+ * notice, this list of conditions and the following disclaimer. >+ * 2. Redistributions in binary form must reproduce the above copyright >+ * notice, this list of conditions and the following disclaimer in the >+ * documentation and/or other materials provided with the distribution. >+ * >+ * THIS SOFTWARE IS PROVIDED BY APPLE INC. AND ITS CONTRIBUTORS ``AS IS'' AND ANY >+ * EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED >+ * WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE >+ * DISCLAIMED. IN NO EVENT SHALL APPLE INC. OR ITS CONTRIBUTORS BE LIABLE FOR ANY >+ * DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES >+ * (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; >+ * LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON >+ * ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT >+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS >+ * SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. >+ */ >+ >+#pragma once >+ >+#if ENABLE(MEDIA_STREAM) >+ >+#include "MediaRecorderPrivate.h" >+#include "MediaRecorderPrivateWriterCocoa.h" >+ >+namespace WebCore { >+ >+class MediaStreamPrivate; >+ >+class MediaRecorderPrivateAVFImpl final : public MediaRecorderPrivate { >+public: >+ static std::unique_ptr<MediaRecorderPrivateAVFImpl> create(const MediaStreamPrivate&); >+ >+private: >+ explicit MediaRecorderPrivateAVFImpl(const MediaStreamPrivate&); >+ >+ void sampleBufferUpdated(MediaStreamTrackPrivate&, MediaSample&) final; >+ void audioSamplesAvailable(MediaStreamTrackPrivate&, const WTF::MediaTime&, const PlatformAudioData&, const AudioStreamDescription&, size_t) final; >+ RefPtr<SharedBuffer> fetchData() final; >+ const String& mimeType() final; >+ void stopRecording(); >+ >+ String m_recordedVideoTrackID; >+ String m_recordedAudioTrackID; >+ >+ MediaRecorderPrivateWriter m_writer; >+ >+ bool m_isWriterReady { false }; >+}; >+ >+} // namespace WebCore >+ >+#endif // ENABLE(MEDIA_STREAM) >diff --git a/Source/WebCore/platform/mediarecorder/MediaRecorderPrivateMock.cpp b/Source/WebCore/platform/mediarecorder/MediaRecorderPrivateMock.cpp >index a4a8bcbd860bd3692ce4be76f521c6690420765e..c34a7afcd9f4d79925cb9313810f0292009b2a99 100644 >--- a/Source/WebCore/platform/mediarecorder/MediaRecorderPrivateMock.cpp >+++ b/Source/WebCore/platform/mediarecorder/MediaRecorderPrivateMock.cpp >@@ -28,8 +28,8 @@ > > #if ENABLE(MEDIA_STREAM) > >-#include "Blob.h" > #include "MediaStreamTrackPrivate.h" >+#include "SharedBuffer.h" > > namespace WebCore { > >@@ -56,13 +56,19 @@ void MediaRecorderPrivateMock::generateMockString(MediaStreamTrackPrivate& track > m_buffer.append("\r\n---------\r\n"); > } > >-Ref<Blob> MediaRecorderPrivateMock::fetchData() >+RefPtr<SharedBuffer> MediaRecorderPrivateMock::fetchData() > { > auto locker = holdLock(m_bufferLock); > Vector<uint8_t> value(m_buffer.length()); > memcpy(value.data(), m_buffer.characters8(), m_buffer.length()); > m_buffer.clear(); >- return Blob::create(WTFMove(value), "text/plain"); >+ return SharedBuffer::create(WTFMove(value)); >+} >+ >+const String& MediaRecorderPrivateMock::mimeType() >+{ >+ static NeverDestroyed<const String> textPlainMimeType(MAKE_STATIC_STRING_IMPL("text/plain")); >+ return textPlainMimeType; > } > > } // namespace WebCore >diff --git a/Source/WebCore/platform/mediarecorder/MediaRecorderPrivateMock.h b/Source/WebCore/platform/mediarecorder/MediaRecorderPrivateMock.h >index 4d9adb60392e7a9ebf191ea7180616c9a8f004b1..6ca817315f870b4e83aec761259f57b09bab97a6 100644 >--- a/Source/WebCore/platform/mediarecorder/MediaRecorderPrivateMock.h >+++ b/Source/WebCore/platform/mediarecorder/MediaRecorderPrivateMock.h >@@ -32,14 +32,14 @@ > > namespace WebCore { > >-class Blob; > class MediaStreamTrackPrivate; > >-class MediaRecorderPrivateMock final : public MediaRecorderPrivate { >+class WEBCORE_EXPORT MediaRecorderPrivateMock final : public MediaRecorderPrivate { > private: > void sampleBufferUpdated(MediaStreamTrackPrivate&, MediaSample&) final; > void audioSamplesAvailable(MediaStreamTrackPrivate&, const WTF::MediaTime&, const PlatformAudioData&, const AudioStreamDescription&, size_t) final; >- Ref<Blob> fetchData() final; >+ RefPtr<SharedBuffer> fetchData() final; >+ const String& mimeType() final; > > void generateMockString(MediaStreamTrackPrivate&); > >diff --git a/Source/WebCore/platform/mediarecorder/cocoa/MediaRecorderPrivateWriterCocoa.h b/Source/WebCore/platform/mediarecorder/cocoa/MediaRecorderPrivateWriterCocoa.h >new file mode 100644 >index 0000000000000000000000000000000000000000..10ba625b90e35e1e158274750f5cdbb73df02f8e >--- /dev/null >+++ b/Source/WebCore/platform/mediarecorder/cocoa/MediaRecorderPrivateWriterCocoa.h >@@ -0,0 +1,85 @@ >+/* >+ * Copyright (C) 2018 Apple Inc. All rights reserved. >+ * >+ * Redistribution and use in source and binary forms, with or without >+ * modification, are permitted provided that the following conditions >+ * are met: >+ * 1. Redistributions of source code must retain the above copyright >+ * notice, this list of conditions and the following disclaimer. >+ * 2. Redistributions in binary form must reproduce the above copyright >+ * notice, this list of conditions and the following disclaimer in the >+ * documentation and/or other materials provided with the distribution. >+ * >+ * THIS SOFTWARE IS PROVIDED BY APPLE INC. AND ITS CONTRIBUTORS ``AS IS'' AND ANY >+ * EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED >+ * WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE >+ * DISCLAIMED. IN NO EVENT SHALL APPLE INC. OR ITS CONTRIBUTORS BE LIABLE FOR ANY >+ * DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES >+ * (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; >+ * LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON >+ * ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT >+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS >+ * SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. >+ */ >+ >+#pragma once >+ >+#if ENABLE(MEDIA_STREAM) >+ >+#include "SharedBuffer.h" >+#include <wtf/Deque.h> >+#include <wtf/Lock.h> >+#include <wtf/RetainPtr.h> >+#include <wtf/ThreadSafeRefCounted.h> >+#include <wtf/threads/BinarySemaphore.h> >+ >+typedef struct opaqueCMSampleBuffer *CMSampleBufferRef; >+ >+OBJC_CLASS AVAssetWriter; >+OBJC_CLASS AVAssetWriterInput; >+ >+namespace WTF { >+class MediaTime; >+} >+ >+namespace WebCore { >+ >+class AudioStreamDescription; >+class PlatformAudioData; >+ >+class MediaRecorderPrivateWriter : public ThreadSafeRefCounted<MediaRecorderPrivateWriter, WTF::DestructionThread::Main>, public CanMakeWeakPtr<MediaRecorderPrivateWriter> { >+public: >+ MediaRecorderPrivateWriter() = default; >+ >+ bool setupWriter(); >+ bool setVideoInput(int width, int height); >+ bool setAudioInput(); >+ void appendVideoSampleBuffer(CMSampleBufferRef); >+ void appendAudioSampleBuffer(const PlatformAudioData&, const AudioStreamDescription&, const WTF::MediaTime&, size_t); >+ void stopRecording(); >+ RefPtr<SharedBuffer> fetchData(); >+ >+private: >+ >+ RetainPtr<AVAssetWriter> m_writer; >+ RetainPtr<AVAssetWriterInput> m_videoInput; >+ RetainPtr<AVAssetWriterInput> m_audioInput; >+ >+ String m_path; >+ Lock m_videoLock; >+ Lock m_audioLock; >+ BinarySemaphore m_finishWritingSemaphore; >+ BinarySemaphore m_finishWritingAudioSemaphore; >+ BinarySemaphore m_finishWritingVideoSemaphore; >+ bool m_hasStartedWriting { false }; >+ bool m_isStopped { false }; >+ bool m_isFirstAudioSample { true }; >+ dispatch_queue_t m_audioPullQueue; >+ dispatch_queue_t m_videoPullQueue; >+ Deque<RetainPtr<CMSampleBufferRef>> m_videoBufferPool; >+ Deque<RetainPtr<CMSampleBufferRef>> m_audioBufferPool; >+}; >+ >+} // namespace WebCore >+ >+#endif // ENABLE(MEDIA_STREAM) >diff --git a/Source/WebCore/platform/mediarecorder/cocoa/MediaRecorderPrivateWriterCocoa.mm b/Source/WebCore/platform/mediarecorder/cocoa/MediaRecorderPrivateWriterCocoa.mm >new file mode 100644 >index 0000000000000000000000000000000000000000..34ce96e49a0d3512c067ddafd4e284486c1a1f7c >--- /dev/null >+++ b/Source/WebCore/platform/mediarecorder/cocoa/MediaRecorderPrivateWriterCocoa.mm >@@ -0,0 +1,323 @@ >+/* >+ * Copyright (C) 2018 Apple Inc. All rights reserved. >+ * >+ * Redistribution and use in source and binary forms, with or without >+ * modification, are permitted provided that the following conditions >+ * are met: >+ * 1. Redistributions of source code must retain the above copyright >+ * notice, this list of conditions and the following disclaimer. >+ * 2. Redistributions in binary form must reproduce the above copyright >+ * notice, this list of conditions and the following disclaimer in the >+ * documentation and/or other materials provided with the distribution. >+ * >+ * THIS SOFTWARE IS PROVIDED BY APPLE INC. AND ITS CONTRIBUTORS ``AS IS'' >+ * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, >+ * THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR >+ * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL APPLE INC. OR ITS CONTRIBUTORS >+ * BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR >+ * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF >+ * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS >+ * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN >+ * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) >+ * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF >+ * THE POSSIBILITY OF SUCH DAMAGE. >+ */ >+ >+#include "config.h" >+#include "MediaRecorderPrivateWriterCocoa.h" >+ >+#if ENABLE(MEDIA_STREAM) && USE(AVFOUNDATION) >+ >+#include "AudioStreamDescription.h" >+#include "FileSystem.h" >+#include "Logging.h" >+#include "WebAudioBufferList.h" >+#include <AVFoundation/AVAssetWriter.h> >+#include <AVFoundation/AVAssetWriterInput.h> >+#include <pal/cf/CoreMediaSoftLink.h> >+ >+typedef AVAssetWriter AVAssetWriterType; >+typedef AVAssetWriterInput AVAssetWriterInputType; >+ >+SOFT_LINK_FRAMEWORK_OPTIONAL(AVFoundation) >+ >+SOFT_LINK_CLASS(AVFoundation, AVAssetWriter) >+SOFT_LINK_CLASS(AVFoundation, AVAssetWriterInput) >+ >+SOFT_LINK_CONSTANT(AVFoundation, AVFileTypeMPEG4, NSString *) >+SOFT_LINK_CONSTANT(AVFoundation, AVVideoCodecKey, NSString *) >+SOFT_LINK_CONSTANT(AVFoundation, AVVideoCodecH264, NSString *) >+SOFT_LINK_CONSTANT(AVFoundation, AVVideoWidthKey, NSString *) >+SOFT_LINK_CONSTANT(AVFoundation, AVVideoHeightKey, NSString *) >+SOFT_LINK_CONSTANT(AVFoundation, AVMediaTypeVideo, NSString *) >+SOFT_LINK_CONSTANT(AVFoundation, AVMediaTypeAudio, NSString *) >+SOFT_LINK_CONSTANT(AVFoundation, AVEncoderBitRateKey, NSString *) >+SOFT_LINK_CONSTANT(AVFoundation, AVFormatIDKey, NSString *) >+SOFT_LINK_CONSTANT(AVFoundation, AVNumberOfChannelsKey, NSString *) >+SOFT_LINK_CONSTANT(AVFoundation, AVSampleRateKey, NSString *) >+ >+SOFT_LINK_CONSTANT(AVFoundation, AVVideoExpectedSourceFrameRateKey, NSString *) >+SOFT_LINK_CONSTANT(AVFoundation, AVVideoProfileLevelKey, NSString *) >+SOFT_LINK_CONSTANT(AVFoundation, AVVideoAverageBitRateKey, NSString *) >+SOFT_LINK_CONSTANT(AVFoundation, AVVideoMaxKeyFrameIntervalKey, NSString *) >+SOFT_LINK_CONSTANT(AVFoundation, AVVideoProfileLevelH264Main31, NSString *) >+SOFT_LINK_CONSTANT(AVFoundation, AVVideoCompressionPropertiesKey, NSString *) >+ >+#define AVFileTypeMPEG4 getAVFileTypeMPEG4() >+#define AVMediaTypeAudio getAVMediaTypeAudio() >+#define AVMediaTypeVideo getAVMediaTypeVideo() >+#define AVVideoCodecKey getAVVideoCodecKey() >+#define AVVideoCodecH264 getAVVideoCodecH264() >+#define AVVideoWidthKey getAVVideoWidthKey() >+#define AVVideoHeightKey getAVVideoHeightKey() >+#define AVEncoderBitRateKey getAVEncoderBitRateKey() >+#define AVFormatIDKey getAVFormatIDKey() >+#define AVNumberOfChannelsKey getAVNumberOfChannelsKey() >+#define AVSampleRateKey getAVSampleRateKey() >+ >+#define AVVideoExpectedSourceFrameRateKey getAVVideoExpectedSourceFrameRateKey() >+#define AVVideoProfileLevelKey getAVVideoProfileLevelKey() >+#define AVVideoAverageBitRateKey getAVVideoAverageBitRateKey() >+#define AVVideoMaxKeyFrameIntervalKey getAVVideoMaxKeyFrameIntervalKey() >+#define AVVideoProfileLevelH264Main31 getAVVideoProfileLevelH264Main31() >+#define AVVideoCompressionPropertiesKey getAVVideoCompressionPropertiesKey() >+ >+using namespace WebCore; >+ >+namespace WebCore { >+ >+using namespace PAL; >+ >+bool MediaRecorderPrivateWriter::setupWriter() >+{ >+ ASSERT(!m_writer); >+ >+ NSString *directory = FileSystem::createTemporaryDirectory(@"videos"); >+ NSString *filename = [NSString stringWithFormat:@"/%lld.mp4", CMClockGetTime(CMClockGetHostTimeClock()).value]; >+ NSString *path = [directory stringByAppendingString:filename]; >+ >+ NSURL *outputURL = [NSURL fileURLWithPath:path]; >+ m_path = [path UTF8String]; >+ NSError *error = nil; >+ m_writer = adoptNS([allocAVAssetWriterInstance() initWithURL:outputURL fileType:AVFileTypeMPEG4 error:&error]); >+ if (error) { >+ RELEASE_LOG_ERROR(MediaStream, "create AVAssetWriter instance failed with error code %ld", (long)error.code); >+ m_writer = nullptr; >+ return false; >+ } >+ return true; >+} >+ >+bool MediaRecorderPrivateWriter::setVideoInput(int width, int height) >+{ >+ ASSERT(!m_videoInput); >+ >+ NSDictionary *compressionProperties = @{ >+ AVVideoAverageBitRateKey : [NSNumber numberWithInt:width * height * 12], >+ AVVideoExpectedSourceFrameRateKey : @(30), >+ AVVideoMaxKeyFrameIntervalKey : @(120), >+ AVVideoProfileLevelKey : AVVideoProfileLevelH264Main31 >+ }; >+ >+ NSDictionary *videoSettings = @{ >+ AVVideoCodecKey: AVVideoCodecH264, >+ AVVideoWidthKey: [NSNumber numberWithInt:width], >+ AVVideoHeightKey: [NSNumber numberWithInt:height], >+ AVVideoCompressionPropertiesKey: compressionProperties >+ }; >+ >+ m_videoInput = adoptNS([allocAVAssetWriterInputInstance() initWithMediaType:AVMediaTypeVideo outputSettings:videoSettings sourceFormatHint:nil]); >+ [m_videoInput setExpectsMediaDataInRealTime:true]; >+ >+ if (![m_writer canAddInput:m_videoInput.get()]) { >+ m_videoInput = nullptr; >+ RELEASE_LOG_ERROR(MediaStream, "the video input is not allowed to add to the AVAssetWriter"); >+ return false; >+ } >+ [m_writer addInput:m_videoInput.get()]; >+ m_videoPullQueue = dispatch_queue_create("WebCoreVideoRecordingPullBufferQueue", DISPATCH_QUEUE_SERIAL); >+ return true; >+} >+ >+bool MediaRecorderPrivateWriter::setAudioInput() >+{ >+ ASSERT(!m_audioInput); >+ >+ NSDictionary *audioSettings = @{ >+ AVEncoderBitRateKey : @(28000), >+ AVFormatIDKey : @(kAudioFormatMPEG4AAC), >+ AVNumberOfChannelsKey : @(1), >+ AVSampleRateKey : @(22050) >+ }; >+ >+ m_audioInput = adoptNS([allocAVAssetWriterInputInstance() initWithMediaType:AVMediaTypeAudio outputSettings:audioSettings sourceFormatHint:nil]); >+ [m_audioInput setExpectsMediaDataInRealTime:true]; >+ >+ if (![m_writer canAddInput:m_audioInput.get()]) { >+ m_audioInput = nullptr; >+ RELEASE_LOG_ERROR(MediaStream, "the audio input is not allowed to add to the AVAssetWriter"); >+ return false; >+ } >+ [m_writer addInput:m_audioInput.get()]; >+ m_audioPullQueue = dispatch_queue_create("WebCoreAudioRecordingPullBufferQueue", DISPATCH_QUEUE_SERIAL); >+ return true; >+} >+ >+static inline CMSampleBufferRef copySampleBufferWithCurrentTimeStamp(CMSampleBufferRef originalBuffer) >+{ >+ CMTime startTime = CMClockGetTime(CMClockGetHostTimeClock()); >+ CMItemCount count = 0; >+ CMSampleBufferGetSampleTimingInfoArray(originalBuffer, 0, nil, &count); >+ >+ Vector<CMSampleTimingInfo> timeInfo(count); >+ CMSampleBufferGetSampleTimingInfoArray(originalBuffer, count, timeInfo.data(), &count); >+ >+ for (CMItemCount i = 0; i < count; i++) { >+ timeInfo[i].decodeTimeStamp = kCMTimeInvalid; >+ timeInfo[i].presentationTimeStamp = startTime; >+ } >+ >+ CMSampleBufferRef newBuffer; >+ CMSampleBufferCreateCopyWithNewTiming(kCFAllocatorDefault, originalBuffer, count, timeInfo.data(), &newBuffer); >+ return newBuffer; >+} >+ >+void MediaRecorderPrivateWriter::appendVideoSampleBuffer(CMSampleBufferRef sampleBuffer) >+{ >+ ASSERT(m_videoInput); >+ if (m_isStopped) >+ return; >+ >+ if (!m_hasStartedWriting) { >+ if (![m_writer startWriting]) { >+ m_isStopped = true; >+ RELEASE_LOG_ERROR(MediaStream, "create AVAssetWriter instance failed with error code %ld", (long)[m_writer error]); >+ return; >+ } >+ [m_writer startSessionAtSourceTime:CMClockGetTime(CMClockGetHostTimeClock())]; >+ m_hasStartedWriting = true; >+ RefPtr<MediaRecorderPrivateWriter> protectedThis = this; >+ [m_videoInput requestMediaDataWhenReadyOnQueue:m_videoPullQueue usingBlock:[this, protectedThis] { >+ do { >+ if (![m_videoInput isReadyForMoreMediaData]) >+ break; >+ auto locker = holdLock(m_videoLock); >+ if (m_videoBufferPool.isEmpty()) >+ break; >+ auto buffer = m_videoBufferPool.takeFirst(); >+ locker.unlockEarly(); >+ if (![m_videoInput appendSampleBuffer:buffer.get()]) >+ break; >+ } while (true); >+ if (m_isStopped && m_videoBufferPool.isEmpty()) { >+ [m_videoInput markAsFinished]; >+ m_finishWritingVideoSemaphore.signal(); >+ } >+ }]; >+ return; >+ } >+ CMSampleBufferRef bufferWithCurrentTime = copySampleBufferWithCurrentTimeStamp(sampleBuffer); >+ auto locker = holdLock(m_videoLock); >+ m_videoBufferPool.append(retainPtr(bufferWithCurrentTime)); >+} >+ >+void MediaRecorderPrivateWriter::appendAudioSampleBuffer(const PlatformAudioData& data, const AudioStreamDescription& description, const WTF::MediaTime&, size_t sampleCount) >+{ >+ ASSERT(m_audioInput); >+ if ((!m_hasStartedWriting && m_videoInput) || m_isStopped) >+ return; >+ CMSampleBufferRef sampleBuffer; >+ CMFormatDescriptionRef format; >+ OSStatus error; >+ auto& basicDescription = *WTF::get<const AudioStreamBasicDescription*>(description.platformDescription().description); >+ error = CMAudioFormatDescriptionCreate(kCFAllocatorDefault, &basicDescription, 0, NULL, 0, NULL, NULL, &format); >+ if (m_isFirstAudioSample) { >+ if (!m_videoInput) { >+ // audio-only recording. >+ if (![m_writer startWriting]) { >+ m_isStopped = true; >+ return; >+ } >+ [m_writer startSessionAtSourceTime:CMClockGetTime(CMClockGetHostTimeClock())]; >+ m_hasStartedWriting = true; >+ } >+ m_isFirstAudioSample = false; >+ RefPtr<MediaRecorderPrivateWriter> protectedThis = this; >+ [m_audioInput requestMediaDataWhenReadyOnQueue:m_audioPullQueue usingBlock:[this, protectedThis] { >+ do { >+ if (![m_audioInput isReadyForMoreMediaData]) >+ break; >+ auto locker = holdLock(m_audioLock); >+ if (m_audioBufferPool.isEmpty()) >+ break; >+ auto buffer = m_audioBufferPool.takeFirst(); >+ locker.unlockEarly(); >+ [m_audioInput appendSampleBuffer:buffer.get()]; >+ } while (true); >+ if (m_isStopped && m_audioBufferPool.isEmpty()) { >+ [m_audioInput markAsFinished]; >+ m_finishWritingAudioSemaphore.signal(); >+ } >+ }]; >+ } >+ CMTime startTime = CMClockGetTime(CMClockGetHostTimeClock()); >+ >+ error = CMAudioSampleBufferCreateWithPacketDescriptions(kCFAllocatorDefault, NULL, false, NULL, NULL, format, sampleCount, startTime, NULL, &sampleBuffer); >+ if (error) >+ return; >+ error = CMSampleBufferSetDataBufferFromAudioBufferList(sampleBuffer, kCFAllocatorDefault, kCFAllocatorDefault, 0, downcast<WebAudioBufferList>(data).list()); >+ if (error) >+ return; >+ >+ auto locker = holdLock(m_audioLock); >+ m_audioBufferPool.append(retainPtr(sampleBuffer)); >+} >+ >+void MediaRecorderPrivateWriter::stopRecording() >+{ >+ m_isStopped = true; >+ if (!m_hasStartedWriting) >+ return; >+ ASSERT([m_writer status] == AVAssetWriterStatusWriting); >+ if (m_videoInput) >+ m_finishWritingVideoSemaphore.wait(); >+ >+ if (m_audioInput) >+ m_finishWritingAudioSemaphore.wait(); >+ auto weakPtr = makeWeakPtr(*this); >+ [m_writer finishWritingWithCompletionHandler:[this, weakPtr] { >+ m_finishWritingSemaphore.signal(); >+ callOnMainThread([this, weakPtr] { >+ if (!weakPtr) >+ return; >+ m_isStopped = false; >+ m_hasStartedWriting = false; >+ m_isFirstAudioSample = true; >+ if (m_videoInput) { >+ m_videoInput.clear(); >+ m_videoInput = nullptr; >+ dispatch_release(m_videoPullQueue); >+ } >+ if (m_audioInput) { >+ m_audioInput.clear(); >+ m_audioInput = nullptr; >+ dispatch_release(m_audioPullQueue); >+ } >+ m_writer.clear(); >+ m_writer = nullptr; >+ }); >+ }]; >+} >+ >+RefPtr<SharedBuffer> MediaRecorderPrivateWriter::fetchData() >+{ >+ if ((m_path.isEmpty() && !m_isStopped) || !m_hasStartedWriting) >+ return nullptr; >+ >+ m_finishWritingSemaphore.wait(); >+ return SharedBuffer::createWithContentsOfFile(m_path); >+} >+ >+} // namespace WebCore >+ >+#endif // ENABLE(MEDIA_STREAM) && USE(AVFOUNDATION) >diff --git a/Source/WebCore/testing/Internals.cpp b/Source/WebCore/testing/Internals.cpp >index 77bfc9821a5dcf054408b28e710c1b3adf236492..b8c7f889a73c2623e851800eaf602c589d1f03b7 100644 >--- a/Source/WebCore/testing/Internals.cpp >+++ b/Source/WebCore/testing/Internals.cpp >@@ -228,6 +228,8 @@ > #endif > > #if ENABLE(MEDIA_STREAM) >+#include "MediaRecorder.h" >+#include "MediaRecorderPrivateMock.h" > #include "MediaStream.h" > #include "MockRealtimeMediaSourceCenter.h" > #endif >@@ -1473,6 +1475,16 @@ void Internals::setMockMediaCaptureDevicesEnabled(bool enabled) > WebCore::DeprecatedGlobalSettings::setMockCaptureDevicesEnabled(enabled); > } > >+static std::unique_ptr<MediaRecorderPrivate> createRecorderMockSource() >+{ >+ return std::unique_ptr<MediaRecorderPrivateMock>(new MediaRecorderPrivateMock); >+} >+ >+void Internals::setCustomPrivateRecorderCreator() >+{ >+ WebCore::MediaRecorder::setCustomPrivateRecorderCreator(createRecorderMockSource); >+} >+ > #endif > > ExceptionOr<Ref<DOMRect>> Internals::absoluteCaretBounds() >diff --git a/Source/WebCore/testing/Internals.h b/Source/WebCore/testing/Internals.h >index 8ae3443bb10adbd51fa01ca5262320bc0e5c8f38..c7c9eee9769c0a7a417d3afa68987531cd1dbd7a 100644 >--- a/Source/WebCore/testing/Internals.h >+++ b/Source/WebCore/testing/Internals.h >@@ -507,6 +507,7 @@ public: > > #if ENABLE(MEDIA_STREAM) > void setMockMediaCaptureDevicesEnabled(bool); >+ void setCustomPrivateRecorderCreator(); > #endif > > #if ENABLE(WEB_RTC) >diff --git a/Source/WebCore/testing/Internals.idl b/Source/WebCore/testing/Internals.idl >index 2719346114bd0cbb16e01452062f1fe4f62cf965..e9aad58cb8754942bc063e8430553baff5cf6a05 100644 >--- a/Source/WebCore/testing/Internals.idl >+++ b/Source/WebCore/testing/Internals.idl >@@ -575,6 +575,7 @@ enum CompositingPolicy { > [Conditional=WIRELESS_PLAYBACK_TARGET] void setMockMediaPlaybackTargetPickerEnabled(boolean enabled); > [Conditional=WIRELESS_PLAYBACK_TARGET, MayThrowException] void setMockMediaPlaybackTargetPickerState(DOMString deviceName, DOMString deviceState); > [Conditional=MEDIA_STREAM] void setMockMediaCaptureDevicesEnabled(boolean enabled); >+ [Conditional=MEDIA_STREAM] void setCustomPrivateRecorderCreator(); > > [Conditional=WEB_RTC] void emulateRTCPeerConnectionPlatformEvent(RTCPeerConnection connection, DOMString action); > [Conditional=WEB_RTC] void useMockRTCPeerConnectionFactory(DOMString testCase); >diff --git a/LayoutTests/ChangeLog b/LayoutTests/ChangeLog >index 4a29e503642e7a18e35d2f3e6a7f9c11257e5abe..70a1eb8a3b7cffbf909050edcb7c328725d08d64 100644 >--- a/LayoutTests/ChangeLog >+++ b/LayoutTests/ChangeLog >@@ -1,3 +1,19 @@ >+2018-12-08 YUHAN WU <yuhan_wu@apple.com> >+ >+ Implement non-timeslice mode encoding for MediaRecorder >+ https://bugs.webkit.org/show_bug.cgi?id=192069 >+ <rdar://problem/46443290> >+ >+ Reviewed by NOBODY (OOPS!). >+ >+ Create new tests for encoding of MediaRecorder. Check if the produced video and audio are correct. >+ Add code to turn on the mock source of MediaRecorder for the two old tests because the real source is enabled by default. >+ >+ * http/wpt/mediarecorder/MediaRecorder-AV-audio-video-dataavailable-expected.txt: Added. >+ * http/wpt/mediarecorder/MediaRecorder-AV-audio-video-dataavailable.html: Added. >+ * http/wpt/mediarecorder/MediaRecorder-dataavailable.html: >+ * http/wpt/mediarecorder/MediaRecorder-mock-dataavailable.html: >+ > 2018-12-05 Ryosuke Niwa <rniwa@webkit.org> > > Null pointer crash in DocumentOrderedMap::getElementById via FormAssociatedElement::findAssociatedForm >diff --git a/LayoutTests/http/wpt/mediarecorder/MediaRecorder-AV-audio-video-dataavailable-expected.txt b/LayoutTests/http/wpt/mediarecorder/MediaRecorder-AV-audio-video-dataavailable-expected.txt >new file mode 100644 >index 0000000000000000000000000000000000000000..708c1285298e5345bc7cf9eef6816cf0f6215b2b >--- /dev/null >+++ b/LayoutTests/http/wpt/mediarecorder/MediaRecorder-AV-audio-video-dataavailable-expected.txt >@@ -0,0 +1,4 @@ >+ >+ >+PASS MediaRecorder can successfully record the video for a audio-video stream >+ >diff --git a/LayoutTests/http/wpt/mediarecorder/MediaRecorder-AV-audio-video-dataavailable.html b/LayoutTests/http/wpt/mediarecorder/MediaRecorder-AV-audio-video-dataavailable.html >new file mode 100644 >index 0000000000000000000000000000000000000000..9120b369918d566ec3f7c1fc99e13ca9311d690b >--- /dev/null >+++ b/LayoutTests/http/wpt/mediarecorder/MediaRecorder-AV-audio-video-dataavailable.html >@@ -0,0 +1,114 @@ >+<!doctype html> >+<html> >+<head> >+ <title>MediaRecorder Dataavailable</title> >+ <link rel="help" href="https://w3c.github.io/mediacapture-record/MediaRecorder.html#mediarecorder"> >+ <script src="/resources/testharness.js"></script> >+ <script src="/resources/testharnessreport.js"></script> >+ <script src="../common/canvas-tests.js"></script> >+ <link rel="stylesheet" href="../common/canvas-tests.css"> >+</head> >+<body> >+<div> >+ <video id="player"> >+ </video> >+</div> >+<div> >+ <canvas id="canvas" width="200" height="200"> >+ </canvas> >+ <canvas id="frame" width="200" height="200"> >+ </canvas> >+</div> >+<script> >+ var context; >+ var drawStartTime; >+ >+ function createVideoStream() { >+ const canvas = document.getElementById("canvas"); >+ context = canvas.getContext('2d'); >+ return canvas.captureStream(10); >+ } >+ >+ function doRedImageDraw() { >+ if (context) { >+ context.fillStyle = "#ff0000"; >+ context.fillRect(0, 0, 200, 200); >+ if (Date.now() - drawStartTime < 500) { >+ window.requestAnimationFrame(doRedImageDraw); >+ } else { >+ drawStartTime = Date.now(); >+ doGreenImageDraw(); >+ } >+ } >+ } >+ >+ function doGreenImageDraw() { >+ if (context) { >+ context.fillStyle = "#00ff00"; >+ context.fillRect(0, 0, 200, 200); >+ if (Date.now() - drawStartTime < 2000) { >+ window.requestAnimationFrame(doGreenImageDraw); >+ } >+ } >+ } >+ >+ async_test(t => { >+ const ac = new AudioContext(); >+ const osc = ac.createOscillator(); >+ const dest = ac.createMediaStreamDestination(); >+ const audio = dest.stream; >+ osc.connect(dest); >+ >+ const video = createVideoStream(); >+ assert_equals(video.getAudioTracks().length, 0, "video mediastream starts with no audio track"); >+ assert_equals(audio.getAudioTracks().length, 1, "audio mediastream starts with one audio track"); >+ video.addTrack(audio.getAudioTracks()[0]); >+ assert_equals(video.getAudioTracks().length, 1, "video mediastream starts with one audio track"); >+ const recorder = new MediaRecorder(video); >+ let mode = 0; >+ >+ recorder.ondataavailable = t.step_func(blobEvent => { >+ assert_true(blobEvent instanceof BlobEvent, 'the type of event should be BlobEvent'); >+ assert_equals(blobEvent.type, 'dataavailable', 'the event type should be dataavailable'); >+ assert_true(blobEvent.isTrusted, 'isTrusted should be true when the event is created by C++'); >+ assert_true(blobEvent.data instanceof Blob, 'the type of data should be Blob'); >+ assert_true(blobEvent.data.size > 0, 'the blob should contain some buffers'); >+ player.src = window.URL.createObjectURL(blobEvent.data); >+ const resFrame = document.getElementById("frame"); >+ const resContext = resFrame.getContext('2d'); >+ >+ player.oncanplay = () => { >+ assert_greater_than(player.duration, 0.05, 'the duration should be greater than 50ms'); >+ player.play(); >+ }; >+ player.onplay = () => { >+ player.pause(); >+ player.currentTime = 0.05; >+ }; >+ player.onseeked = () => { >+ resContext.drawImage(player, 0, 0); >+ if (!mode) { >+ _assertPixelApprox(resFrame, 25, 25, 255, 0, 0, 255, "25, 25", "255, 0, 0, 255", 20); >+ _assertPixelApprox(resFrame, 50, 50, 255, 0, 0, 255, "50, 50", "255, 0, 0, 255", 20); >+ mode = 1; >+ player.currentTime = player.duration; >+ } else { >+ _assertPixelApprox(resFrame, 20, 20, 0, 255, 0, 255, "20, 20", "0, 255, 0, 255", 20); >+ _assertPixelApprox(resFrame, 199, 199, 0, 255, 0, 255, "199, 199", "0, 255, 0, 255", 20); >+ t.done(); >+ } >+ }; >+ player.load(); >+ }); >+ drawStartTime = Date.now(); >+ doRedImageDraw(); >+ recorder.start(); >+ assert_equals(recorder.state, 'recording', 'MediaRecorder has been started successfully'); >+ setTimeout(() => { >+ recorder.stop(); >+ }, 2000); >+ }, 'MediaRecorder can successfully record the video for a audio-video stream'); >+ >+</script> >+</body> >+</html> >\ No newline at end of file >diff --git a/LayoutTests/http/wpt/mediarecorder/MediaRecorder-dataavailable.html b/LayoutTests/http/wpt/mediarecorder/MediaRecorder-dataavailable.html >index bdca7f7a40a137c18fb705230acffbd24e935ab9..4ac3d6d50bf17b682f4932bb0ea3aaafeefc903f 100644 >--- a/LayoutTests/http/wpt/mediarecorder/MediaRecorder-dataavailable.html >+++ b/LayoutTests/http/wpt/mediarecorder/MediaRecorder-dataavailable.html >@@ -12,6 +12,9 @@ > <script> > var context; > >+ if (window.internals) >+ internals.setCustomPrivateRecorderCreator(); >+ > function createVideoStream() { > const canvas = document.getElementById("canvas"); > context = canvas.getContext('2d'); >@@ -44,7 +47,7 @@ > drawSomethingOnCanvas(); > setTimeout(() => { > recorder.stop(); >- }, 2000) >+ }, 1000) > }, 'MediaRecorder will fire a dataavailable event with a blob data for a video-only stream when stop() is called'); > > async_test(t => { >@@ -65,7 +68,7 @@ > setTimeout(() => { > recorder.stop(); > osc.stop(); >- }, 2000); >+ }, 1000); > }, 'MediaRecorder will fire a dataavailable event with a blob data for a audio-only stream when stop() is called'); > > async_test(t => { >@@ -93,7 +96,7 @@ > setTimeout(() => { > recorder.stop(); > osc.stop(); >- }, 2000); >+ }, 1000); > }, 'MediaRecorder will fire a dataavailable event with a blob data for a video-audio stream when stop() is called'); > > </script> >diff --git a/LayoutTests/http/wpt/mediarecorder/MediaRecorder-mock-dataavailable.html b/LayoutTests/http/wpt/mediarecorder/MediaRecorder-mock-dataavailable.html >index 1286b8bde4094fd44037f89dc567f35a7605e056..43ee488752a4bb46e36464b4b048c4b64f3e9766 100644 >--- a/LayoutTests/http/wpt/mediarecorder/MediaRecorder-mock-dataavailable.html >+++ b/LayoutTests/http/wpt/mediarecorder/MediaRecorder-mock-dataavailable.html >@@ -12,6 +12,9 @@ > <script> > var context; > >+ if (window.internals) >+ internals.setCustomPrivateRecorderCreator(); >+ > function createVideoStream() { > const canvas = document.getElementById("canvas"); > context = canvas.getContext('2d'); >@@ -41,7 +44,7 @@ > drawSomethingOnCanvas(); > setTimeout(() => { > recorder.stop(); >- }, 2000); >+ }, 1000); > }, 'MediaRecorder will fire a dataavailable event which only contains video buffers for a video-only stream when stop() is called'); > > async_test(t => { >@@ -67,7 +70,7 @@ > setTimeout(() => { > recorder.stop(); > osc.stop(); >- }, 2000); >+ }, 1000); > }, 'MediaRecorder will fire a dataavailable event which only contains audio buffers for a audio-only stream when stop() is called'); > > async_test(t => { >@@ -100,7 +103,7 @@ > setTimeout(() => { > recorder.stop(); > osc.stop(); >- }, 2000); >+ }, 1000); > }, 'MediaRecorder will fire a dataavailable event which only contains both video and audio buffers when stop() is called'); > > </script>
<b>You cannot view the attachment while viewing its details because your browser does not support IFRAMEs. <a href="attachment.cgi?id=356932">View the attachment on a separate page</a>.</b>
View Attachment As Diff
View Attachment As Raw
Actions:
View
|
Formatted Diff
|
Diff
Attachments on
bug 192069
:
355845
|
355846
|
355851
|
355854
|
355855
|
355856
|
355857
|
355945
|
355954
|
356110
|
356112
|
356113
|
356118
|
356234
|
356237
|
356244
|
356405
|
356411
|
356419
|
356897
|
356898
|
356899
|
356906
|
356932
|
356936
|
357196
|
357198