260723 2023-08-25 11:42:50 -0700 [GTK] Web process crash when creating GMainContext: Creating pipes for GWakeup: Too many open files 2025-10-15 07:33:32 -0700 1 1 1 Unclassified WebKit WebKitGTK WebKit Nightly Build PC Linux NEW https://bugs.webkit.org/show_bug.cgi?id=266209 https://bugs.webkit.org/show_bug.cgi?id=264824 https://bugs.webkit.org/show_bug.cgi?id=277076 https://bugs.webkit.org/show_bug.cgi?id=266573 https://bugs.webkit.org/show_bug.cgi?id=289994 P2 Normal --- 1 mcatanzaro webkit-unassigned bugs-noreply cgarcia eocanha kdwkleung mcatanzaro nilskemail+webkit philn vitaly oldest_to_newest 1973571 0 mcatanzaro 2023-08-25 11:42:50 -0700 Look at this web process crash: Thread 1 (Thread 0x7ff1bdffb6c0 (LWP 219)): #0 g_log_structured_array (log_level=<optimized out>, fields=0x7ff1bdffa830, n_fields=4) at ../glib/gmessages.c:556 #1 0x00007ff4d7a8588c in g_log_default_handler (log_domain=log_domain@entry=0x7ff4d7ade00e "GLib", log_level=log_level@entry=6, message=message@entry=0x7ff1a0000d50 "Creating pipes for GWakeup: Too many open files", unused_data=unused_data@entry=0x0) at ../glib/gmessages.c:3284 #2 0x00007ff4c006d242 in trap_handler (log_domain=log_domain@entry=0x7ff4d7ade00e "GLib", log_level=log_level@entry=6, message=message@entry=0x7ff1a0000d50 "Creating pipes for GWakeup: Too many open files", user_data=user_data@entry=0x0) at ../lib/ephy-debug.c:104 #3 0x00007ff4d7a85b36 in g_logv (log_domain=0x7ff4d7ade00e "GLib", log_level=G_LOG_LEVEL_ERROR, format=<optimized out>, args=args@entry=0x7ff1bdffa9b0) at ../glib/gmessages.c:1392 #4 0x00007ff4d7a85e23 in g_log (log_domain=log_domain@entry=0x7ff4d7ade00e "GLib", log_level=log_level@entry=G_LOG_LEVEL_ERROR, format=format@entry=0x7ff4d7aeb330 "Creating pipes for GWakeup: %s") at ../glib/gmessages.c:1461 #5 0x00007ff4d7ad51aa in g_wakeup_new () at ../glib/gwakeup.c:164 #6 0x00007ff4d7a79a2f in g_main_context_new_with_flags (flags=<optimized out>) at ../glib/gmain.c:786 #7 0x00007ff4db5515f1 in WTF::RunLoop::RunLoop() (this=0x7ff2fe9ac360) at /buildstream/gnome/sdk/webkitgtk-6.0.bst/Source/WTF/wtf/glib/RunLoopGLib.cpp:66 #8 0x00007ff4db4f7e7d in WTF::RunLoop::Holder::Holder() (this=0x7ff2595b00d0) at /buildstream/gnome/sdk/webkitgtk-6.0.bst/Source/WTF/wtf/RunLoop.cpp:46 #9 WTF::ThreadSpecific<WTF::RunLoop::Holder, (WTF::CanBeGCThread)0>::Data::Data(WTF::ThreadSpecific<WTF::RunLoop::Holder, (WTF::CanBeGCThread)0>*) (this=0x7ff2595b00d0, owner=<optimized out>) at /buildstream/gnome/sdk/webkitgtk-6.0.bst/Source/WTF/wtf/ThreadSpecific.h:94 #10 WTF::ThreadSpecific<WTF::RunLoop::Holder, (WTF::CanBeGCThread)0>::set() (this=<optimized out>) at /buildstream/gnome/sdk/webkitgtk-6.0.bst/Source/WTF/wtf/ThreadSpecific.h:195 #11 WTF::ThreadSpecific<WTF::RunLoop::Holder, (WTF::CanBeGCThread)0>::operator WTF::RunLoop::Holder*() (this=<optimized out>) at /buildstream/gnome/sdk/webkitgtk-6.0.bst/Source/WTF/wtf/ThreadSpecific.h:211 #12 WTF::ThreadSpecific<WTF::RunLoop::Holder, (WTF::CanBeGCThread)0>::operator->() (this=<optimized out>) at /buildstream/gnome/sdk/webkitgtk-6.0.bst/Source/WTF/wtf/ThreadSpecific.h:217 #13 WTF::RunLoop::current() () at /buildstream/gnome/sdk/webkitgtk-6.0.bst/Source/WTF/wtf/RunLoop.cpp:79 #14 0x00007ff4db4f88f2 in WTF::RunLoop::create(char const*, WTF::ThreadType, WTF::Thread::QOS)::$_0::operator()() const (this=0x7ff2be9381e8) at /buildstream/gnome/sdk/webkitgtk-6.0.bst/Source/WTF/wtf/RunLoop.cpp:112 #15 WTF::Detail::CallableWrapper<WTF::RunLoop::create(char const*, WTF::ThreadType, WTF::Thread::QOS)::$_0, void>::call() (this=0x400) at /buildstream/gnome/sdk/webkitgtk-6.0.bst/Source/WTF/wtf/Function.h:53 #16 0x00007ff4db4fb9a7 in WTF::Function<void ()>::operator()() const (this=<optimized out>) at /buildstream/gnome/sdk/webkitgtk-6.0.bst/Source/WTF/wtf/Function.h:82 #17 WTF::Thread::entryPoint(WTF::Thread::NewThreadContext*) (newThreadContext=0x7ff2f23f02c0) at /buildstream/gnome/sdk/webkitgtk-6.0.bst/Source/WTF/wtf/Threading.cpp:250 #18 0x00007ff4db55582d in WTF::wtfThreadEntryPoint(void*) (context=0x400) at /buildstream/gnome/sdk/webkitgtk-6.0.bst/Source/WTF/wtf/posix/ThreadingPOSIX.cpp:242 #19 0x00007ff4dbc8ee39 in start_thread (arg=<optimized out>) at pthread_create.c:444 #20 0x00007ff4dbd16cc4 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:100 Well, that's unfortunate. pipe(2) says: EMFILE The per‐process limit on the number of open file descriptors has been reached. ENFILE The system‐wide limit on the total number of open files has been reached. ENFILE The user hard limit on memory that can be allocated for pipes has been reached and the caller is not privileged; see pipe(7). errno(3) says: EMFILE Too many open files (POSIX.1‐2001). Commonly caused by exceeding the RLIMIT_NOFILE resource limit described in getrlimit(2). Can also be caused by exceeding the limit specified in /proc/sys/fs/nr_open. ENFILE Too many open files in system (POSIX.1‐2001). On Linux, this is probably a result of en‐ countering the /proc/sys/fs/file-max limit (see proc(5)). The Epiphany UI process already increases its fd limit up to the hard limit: https://gitlab.gnome.org/GNOME/epiphany/-/merge_requests/1304. At first I thought "maybe the web process needs to do this too," but a typical web process only needs about 25 file descriptors open, so probably not. I wonder if this unfortunate crashed web process wound up hitting some sort of bug that caused it to open an extremely high number of file descriptors? Not sure how we could debug something like this, especially since I have no clue what this web process was doing before it crashed. 1973574 1 mcatanzaro 2023-08-25 11:45:34 -0700 Oh, I forgot to say, the system-wide limits are extremely high: $ cat /proc/sys/fs/nr_open 1073741816 $ cat /proc/sys/fs/file-max 9223372036854775807 $ ulimit -n 1024 so we are almost certainly hitting the soft RLIMIT_NOFILE limit, which is 1024 (because any value higher than that breaks applications that use select()). 1973930 2 mcatanzaro 2023-08-28 08:01:34 -0700 Hit this again yesterday. :( 1996289 3 mcatanzaro 2023-12-01 11:19:40 -0800 Hit again today when loading: https://www.cnn.com/dodo-de-extinction-mauritius-spc-intl-scn/index.html I strongly suspect either GStreamer or our multimedia code is leaking file descriptors, but we have no good way to know because once the process has crashed it's too late to check what fds are open, and until it has crashed we don't know that anything is wrong. 1998335 4 philn 2023-12-09 10:50:57 -0800 Well, current Ephy TP leaks MSE AppendPipelines, at least... Run with --env=GST_DEBUG="GST_TRACER*:8" --env="GST_TRACERS=leaks(filters=GstElement)" then load some page with MSE videos, wait for a while, open an empty tab, close video tab, close Ephy. Then you should see logs like this: object-alive, type-name=(string)GstPipeline, address=(gpointer)0x55cd47958240, description=(string)<append-pipeline-audio-mp4-1> 2017988 5 mcatanzaro 2024-03-01 06:27:34 -0800 *** Bug 264674 has been marked as a duplicate of this bug. *** 2017990 6 mcatanzaro 2024-03-01 06:27:42 -0800 *** Bug 266577 has been marked as a duplicate of this bug. *** 2032406 7 mcatanzaro 2024-04-30 19:44:55 -0700 This happens relatively frequently when loading articles on cnn.com (as in comment #3), but unfortunately not frequently enough to be reproducible. 2035759 8 mcatanzaro 2024-05-16 05:23:12 -0700 OK, finally I have a reproducer. Load https://globalnews.ca/news/10497774/turks-and-caicos-american-tourist-arrested-ammo/ in Ephy Tech Preview. It's not a sure thing, but it *almost* always crashes. If you can't reproduce after loading the page just once, then load it a couple more times in different web views and it will almost definitely crash. 2035761 9 mcatanzaro 2024-05-16 05:35:04 -0700 This same reproducer also triggers bug #266573. 2035771 10 philn 2024-05-16 06:09:22 -0700 (In reply to Michael Catanzaro from comment #8) > OK, finally I have a reproducer. Load > https://globalnews.ca/news/10497774/turks-and-caicos-american-tourist- > arrested-ammo/ in Ephy Tech Preview. It's not a sure thing, but it *almost* > always crashes. If you can't reproduce after loading the page just once, > then load it a couple more times in different web views and it will almost > definitely crash. Doesn't crash here, but I may have found a nice leak that could be the culprit. See bug 274254. 2035805 11 mcatanzaro 2024-05-16 08:51:40 -0700 *** Bug 274241 has been marked as a duplicate of this bug. *** 2036280 12 philn 2024-05-19 04:39:02 -0700 (In reply to Michael Catanzaro from comment #8) > OK, finally I have a reproducer. Load > https://globalnews.ca/news/10497774/turks-and-caicos-american-tourist- > arrested-ammo/ in Ephy Tech Preview. It's not a sure thing, but it *almost* > always crashes. If you can't reproduce after loading the page just once, > then load it a couple more times in different web views and it will almost > definitely crash. I can reproduce this in TP, but not if the WEBKIT_DISABLE_SANDBOX_THIS_IS_DANGEROUS is set... This seems related with the limits set on the WebProcess sandbox by flatpak-spawn. The issue doesn't happen in MiniBrowser dev build, the flatpak stuff not being used there. 2040287 13 mcatanzaro 2024-06-07 03:59:57 -0700 (In reply to Philippe Normand from comment #12) > The issue doesn't happen in MiniBrowser dev build, the flatpak stuff not > being used there. It does for me, even with the bubblewrap sandbox disabled. I can reproduce this crash very reliably by running: $ jhbuild run env WEBKIT_DISABLE_SANDBOX_THIS_IS_DANGEROUS=1 WEBKIT_GST_HARNESS_DUMP_DIR=~/dump GST_DEBUG=webkit*:9 G_DEBUG= ~/Projects/GNOME/install/libexec/webkitgtk-6.0/MiniBrowser https://www.vox.com/future-perfect/352359/milk-dairy-schools 2040289 14 mcatanzaro 2024-06-07 04:10:30 -0700 (In reply to Philippe Normand from comment #10) > Doesn't crash here, but I may have found a nice leak that could be the > culprit. See bug 274254. And I'm testing 279750@main, so I already have this fix unfortunately. 2044910 15 kdwkleung 2024-07-05 22:38:18 -0700 This bug is making reddit.com reliably crash. Scrolling Reddit for too long invariably leads to this crash and stack trace. WebKitGTK 2.45.4/ Spidey 1.0 2044975 16 philn 2024-07-07 04:34:52 -0700 (In reply to Kdwk from comment #15) > This bug is making reddit.com reliably crash. > > Scrolling Reddit for too long invariably leads to this crash and stack trace. > Can you try https://github.com/WebKit/WebKit/pull/30549 ? 2044983 17 mcatanzaro 2024-07-07 08:18:11 -0700 (In reply to Michael Catanzaro from comment #8) > OK, finally I have a reproducer. Load > https://globalnews.ca/news/10497774/turks-and-caicos-american-tourist- > arrested-ammo/ in Ephy Tech Preview. It's not a sure thing, but it *almost* > always crashes. If you can't reproduce after loading the page just once, > then load it a couple more times in different web views and it will almost > definitely crash. Unfortunately this reproducer no longer works. I can load this page in many tabs at once without trouble. I do still see this crash somewhat regularly, though; I just don't have a reproducer anymore. Hopefully Kdwk is able to reproduce on reddit. I use reddit a lot, but never see this happen there. 2047068 18 nilskemail+webkit 2024-07-18 13:00:47 -0700 Based on the description it seems like we too are running into this bug on our digital signage devices which are running cog, i.e. WPE WebKit. They show a set list of videos, images, and iframes on repeat, forever. The crashes only happen once the browser process has been running for many hours - about once per day. I would be more than happy to provide additional data that could help to figure out to root cause if someone tells me what data/commands would be helpful. Backtrace: #0 g_log_structured_array (log_level=<optimized out>, fields=0x7f97af1ff680, n_fields=4) at ../glib/gmessages.c:426 #1 0x00007fad69405d27 in g_log_default_handler (log_domain=log_domain@entry=0x7fad694ba2eb "GLib", log_level=log_level@entry=6, message=message@entry=0x7fac881a2460 "Creating pipes for GWakeup: Too many open files", unused_data=unused_data@entry=0x0) at ../glib/gmessages.c:3357 #2 0x00007fad693fcb29 in g_logv (log_domain=0x7fad694ba2eb "GLib", log_level=G_LOG_LEVEL_ERROR, format=<optimized out>, args=args@entry=0x7f97af1ff7e0) at ../glib/gmessages.c:1246 #3 0x00007fad693fcea3 in g_log (log_domain=<optimized out>, log_level=<optimized out>, format=<optimized out>) at ../glib/gmessages.c:1315 #4 0x00007fad6945175a in g_wakeup_new () at ../glib/gwakeup.c:162 #5 0x00007fad693f4618 in g_main_context_new_with_flags (flags=<optimized out>) at ../glib/gmain.c:658 #6 0x00007fad6ae9dd0d in WTF::RunLoop::current() () from /lib64/libWPEWebKit-2.0.so.1 #7 0x00007fad6ae9ded7 in WTF::Detail::CallableWrapper<WTF::RunLoop::create(char const*, WTF::ThreadType, WTF::Thread::QOS)::{lambda()#1}, void>::call() () from /lib64/libWPEWebKit-2.0.so.1 #8 0x00007fad6aeeb162 in WTF::wtfThreadEntryPoint(void*) [clone .lto_priv.0] () from /lib64/libWPEWebKit-2.0.so.1 #9 0x00007fad698a6507 in start_thread (arg=<optimized out>) at pthread_create.c:447 #10 0x00007fad6992a214 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:100 System log directly before crash: [70612.397185] weston[26604]: shared memfd open() failed: Too many open files [70612.401727] weston[26604]: Failed to create secure directory (/run/user/999/pulse): Too many open files [70612.401859] weston[26604]: socket(): Too many open files [70612.691026] weston[26604]: (WPEWebProcess:2): GStreamer-CRITICAL **: 19:54:48.160: gst_poll_write_control: assertion 'set != NULL' failed [70612.691263] weston[26604]: (WPEWebProcess:2): GStreamer-CRITICAL **: 19:54:48.161: gst_poll_write_control: assertion 'set != NULL' failed [70612.691668] weston[26604]: (WPEWebProcess:2): GStreamer-CRITICAL **: 19:54:48.161: gst_poll_write_control: assertion 'set != NULL' failed [70612.691814] weston[26604]: (WPEWebProcess:2): GStreamer-CRITICAL **: 19:54:48.161: gst_poll_write_control: assertion 'set != NULL' failed [70612.691923] weston[26604]: (WPEWebProcess:2): GStreamer-CRITICAL **: 19:54:48.161: gst_poll_write_control: assertion 'set != NULL' failed [70612.692049] weston[26604]: (WPEWebProcess:2): GStreamer-CRITICAL **: 19:54:48.162: gst_poll_write_control: assertion 'set != NULL' failed [70612.692177] weston[26604]: (WPEWebProcess:2): GStreamer-CRITICAL **: 19:54:48.162: gst_poll_write_control: assertion 'set != NULL' failed [70612.692316] weston[26604]: (WPEWebProcess:2): GStreamer-CRITICAL **: 19:54:48.162: gst_poll_write_control: assertion 'set != NULL' failed [70612.692429] weston[26604]: (WPEWebProcess:2): GStreamer-CRITICAL **: 19:54:48.162: gst_poll_write_control: assertion 'set != NULL' failed [70612.692551] weston[26604]: (WPEWebProcess:2): GStreamer-CRITICAL **: 19:54:48.162: gst_poll_write_control: assertion 'set != NULL' failed [70612.692650] weston[26604]: (WPEWebProcess:2): GStreamer-CRITICAL **: 19:54:48.162: gst_poll_read_control: assertion 'set != NULL' failed [70612.692779] weston[26604]: (WPEWebProcess:2): GStreamer-CRITICAL **: 19:54:48.162: gst_poll_read_control: assertion 'set != NULL' failed [70612.710390] weston[26604]: (WPEWebProcess:2): GStreamer-CRITICAL **: 19:54:48.180: gst_poll_read_control: assertion 'set != NULL' failed [70612.713099] weston[26604]: (WPEWebProcess:2): GStreamer-CRITICAL **: 19:54:48.183: gst_poll_read_control: assertion 'set != NULL' failed [70619.904335] weston[26604]: http://localhost/_nuxt/CbqYvi5G.js:14:610: CONSOLE ERROR TypeError: (m.value??[]).filter is not a function. (In '(m.value??[]).filter(b=>b.Content!==null> [70626.032338] weston[26604]: https://display.soundtrackyourbrand.com/assets/index.63e64c3a.js:42:852: CONSOLE DEBUG [render] Track already rendered [70626.032804] weston[26604]: https://display.soundtrackyourbrand.com/assets/index.63e64c3a.js:25:173: CONSOLE DEBUG [app] Loading <REDACTED> [70626.084897] weston[26604]: CONSOLE NETWORK INFO Successfully preconnected to https://i.soundcdn.com/ [70626.112462] weston[26604]: (WPEWebProcess:2): GLib-ERROR **: 19:55:01.582: Creating pipes for GWakeup: Too many open files [70623.988784] kernel: traps: CryptoQueue[66906] trap int3 ip:7fad694002e7 sp:7f97af1ff630 error:0 in libglib-2.0.so.0.8000.3[7fad693b8000+a4000] [70626.165246] systemd[1]: Created slice system-systemd\x2dcoredump.slice - Slice /system/systemd-coredump. [70626.197365] systemd[1]: Started systemd-coredump@0-66907-0.service - Process Core Dump (PID 66907/UID 0). [70643.337417] systemd-coredump[66908]: [🡕] Process 26604 (WPEWebProcess) of user 999 dumped core. 2047071 19 nilskemail+webkit 2024-07-18 13:19:26 -0700 PS: We are also observing a slow but steady memory increase. It could be that this memory leak also stems from these open file descriptors. 2047077 20 philn 2024-07-18 13:26:45 -0700 (In reply to nilskemail+webkit from comment #19) > PS: We are also observing a slow but steady memory increase. It could be > that this memory leak also stems from these open file descriptors. Yes, pipelines leaking likely induce FD leaks and possibly memory leaks. With which version is this happening? Have you checked with git main? 2047078 21 philn 2024-07-18 13:28:37 -0700 Also having a minimal test-case would help, if possible. I assume mem usage is constant when using other web-engines such as Gecko or Chromium/etc? 2047084 22 nilskemail+webkit 2024-07-18 13:50:04 -0700 > Yes, pipelines leaking likely induce FD leaks and possibly memory leaks. > With which version is this happening? cog 0.18.3 (WPE WebKit 2.44.1) > Have you checked with git main? No, sadly I do not have the capacity to do that. Apart from the time aspect I also only have a single device on which I can test this for a long time which I already use for other purposes > Also having a minimal test-case would help, if possible. Would a bundled web app be okay? In that case I might be able to provide you with the compiled web app and a mock/dummy server replacing our client logic. > I assume mem usage is constant when using other web-engines such as Gecko or Chromium/etc? I have not tried to run it on different web engines, yet (again due to only having a single device available for long running tests). 2047166 23 philn 2024-07-19 02:12:13 -0700 > Would a bundled web app be okay? Yes! 2047353 24 nilskemail+webkit 2024-07-20 11:03:01 -0700 I did not yet have time to create a standalone reproducer as I need to remove some logic before sharing it. However, I investigated a bit more and found out a few things (not sure how much of that is already known). Whilst doing so I was also able to adjust my setup to provoke a crash within a few hours instead of a whole day. * The owner of a large amount of fds is the root WPEWebProcess, i.e. the direct child of bwrap, not any of its children (according to `pstree -p`) * The open fds of that process seem to be mostly `/dmabuf:` (according to `lsof`) * The leaked fds seem to correlate with our memory leak * The amount of leaked memory depends on the resolution of the screen, i.e. switching from 1440p to 1080p had no influence on the amount of open fds but the memory rose slower * Crashing of the WPEWebProcess does not free the memory again. According to the system monitoring the amount of unevictable memory does drop upon the crash, however, it still shows up as shared memory and does not drop there. I also got a new crash with a different backtrace. This might be due to enabling G_DEBUG=fatal-criticals for another bug report (https://bugs.webkit.org/show_bug.cgi?id=276819) causing the process to abort earlier (though even the first CRITICAL log message is a bit different)... Backtrace: #0 _g_log_abort (breakpoint=<optimized out>) at ../glib/gmessages.c:426 #1 g_logv (log_domain=0x7f4edd3e89ae "GStreamer", log_level=G_LOG_LEVEL_CRITICAL, format=<optimized out>, args=args@entry=0x7ffebc36e0d0) at ../glib/gmessages.c:1273 #2 0x00007f4edd53eea3 in g_log (log_domain=<optimized out>, log_level=<optimized out>, format=<optimized out>) at ../glib/gmessages.c:1315 #3 0x00007f4edd347ceb in gst_bus_constructed (object=0x556dd1401960) at ../gst/gstbus.c:184 #4 0x00007f4edc1f217a in g_object_new_internal (class=0x556dd0c7ecb0, params=0x0, n_params=0) at ../gobject/gobject.c:2657 #5 0x00007f4edc1f361e in g_object_new_internal (class=<optimized out>, params=<optimized out>, n_params=<optimized out>) at ../gobject/gobject.c:2603 #6 g_object_new_with_properties (object_type=<optimized out>, n_properties=<optimized out>, names=names@entry=0x0, values=values@entry=0x0) at ../gobject/gobject.c:2769 #7 0x00007f4edc1f4641 in g_object_new (object_type=<optimized out>, first_property_name=first_property_name@entry=0x0) at ../gobject/gobject.c:2415 #8 0x00007f4edd346fbe in gst_bus_new () at ../gst/gstbus.c:310 #9 0x00007f4e87e1de77 in gst_auto_detect_find_best (self=0x556dd119bd50) at ../gst/autodetect/gstautodetect.c:260 #10 gst_auto_detect_detect (self=0x556dd119bd50) at ../gst/autodetect/gstautodetect.c:368 #11 gst_auto_detect_change_state (element=0x556dd119bd50, transition=GST_STATE_CHANGE_NULL_TO_READY) at ../gst/autodetect/gstautodetect.c:420 #12 0x00007f4edd362306 in gst_element_change_state (element=element@entry=0x556dd119bd50, transition=transition@entry=GST_STATE_CHANGE_NULL_TO_READY) at ../gst/gstelement.c:3101 #13 0x00007f4edd362bcb in gst_element_set_state_func (element=0x556dd119bd50, state=GST_STATE_READY) at ../gst/gstelement.c:3055 #14 0x00007f4edd3396b9 in gst_bin_element_set_state (bin=<optimized out>, element=0x556dd119bd50, base_time=0, start_time=0, current=<optimized out>, next=<optimized out>) at ../gst/gstbin.c:2582 #15 gst_bin_change_state_func (element=0x556dd13faa20, transition=GST_STATE_CHANGE_NULL_TO_READY) at ../gst/gstbin.c:2934 #16 0x00007f4edd362306 in gst_element_change_state (element=element@entry=0x556dd13faa20, transition=transition@entry=GST_STATE_CHANGE_NULL_TO_READY) at ../gst/gstelement.c:3101 #17 0x00007f4edd362bcb in gst_element_set_state_func (element=0x556dd13faa20, state=GST_STATE_READY) at ../gst/gstelement.c:3055 #18 0x00007f4e87481977 in activate_sink (playbin=<optimized out>, sink=0x556dd13faa20, activated=0x7ffebc36e6cc) at ../gst/playback/gstplaybin2.c:4528 #19 activate_sink (playbin=<optimized out>, sink=0x556dd13faa20, activated=0x7ffebc36e6cc) at ../gst/playback/gstplaybin2.c:4503 #20 0x00007f4e874a72d0 in activate_group (target=GST_STATE_PAUSED, playbin=0x556dd1402b50, group=0x556dd1402fe0) at ../gst/playback/gstplaybin2.c:5316 #21 setup_next_source.constprop.0 (playbin=playbin@entry=0x556dd1402b50, target=GST_STATE_PAUSED) at ../gst/playback/gstplaybin2.c:5741 #22 0x00007f4e8747f229 in gst_play_bin_change_state (element=0x556dd1402b50, transition=<optimized out>) at ../gst/playback/gstplaybin2.c:5870 #23 0x00007f4edd362306 in gst_element_change_state (element=element@entry=0x556dd1402b50, transition=GST_STATE_CHANGE_READY_TO_PAUSED) at ../gst/gstelement.c:3101 #24 0x00007f4edd362881 in gst_element_continue_state (element=element@entry=0x556dd1402b50, ret=ret@entry=GST_STATE_CHANGE_SUCCESS) at ../gst/gstelement.c:2809 #25 0x00007f4edd36234a in gst_element_change_state (element=element@entry=0x556dd1402b50, transition=transition@entry=GST_STATE_CHANGE_NULL_TO_READY) at ../gst/gstelement.c:3140 #26 0x00007f4edd362bcb in gst_element_set_state_func (element=0x556dd1402b50, state=GST_STATE_PAUSED) at ../gst/gstelement.c:3055 #27 0x00007f4ee02a36e6 in WebCore::MediaPlayerPrivateGStreamer::changePipelineState(GstState) () from /lib64/libWPEWebKit-2.0.so.1 #28 0x00007f4ee02a4095 in WebCore::MediaPlayerPrivateGStreamer::commitLoad() () from /lib64/libWPEWebKit-2.0.so.1 #29 0x00007f4ee02e23d0 in WebCore::MediaPlayerPrivateGStreamer::load(WTF::String const&) () from /lib64/libWPEWebKit-2.0.so.1 #30 0x00007f4ee027085b in WebCore::MediaPlayer::loadWithNextMediaEngine(WebCore::MediaPlayerFactory const*) () from /lib64/libWPEWebKit-2.0.so.1 #31 0x00007f4ee0270e5d in WebCore::MediaPlayer::load(WTF::URL const&, WebCore::ContentType const&, WTF::String const&, bool) () from /lib64/libWPEWebKit-2.0.so.1 #32 0x00007f4edfe06463 in WebCore::HTMLMediaElement::loadResource(WTF::URL const&, WebCore::ContentType&, WTF::String const&) () from /lib64/libWPEWebKit-2.0.so.1 #33 0x00007f4edfe07f02 in WTF::Detail::CallableWrapper<WebCore::HTMLMediaElement::selectMediaResource()::{lambda()#1}, void>::call() () from /lib64/libWPEWebKit-2.0.so.1 #34 0x00007f4edfc3ab75 in WebCore::EventLoop::run(std::optional<WTF::ApproximateTime>) () from /lib64/libWPEWebKit-2.0.so.1 #35 0x00007f4edfc98baf in WebCore::WindowEventLoop::didReachTimeToRun() () from /lib64/libWPEWebKit-2.0.so.1 #36 0x00007f4ee01e1d62 in WTF::Detail::CallableWrapper<WebCore::ThreadTimers::setSharedTimer(WebCore::SharedTimer*)::{lambda()#1}, void>::call() () from /lib64/libWPEWebKit-2.0.so.1 #37 0x00007f4edeeebdfe in WTF::RunLoop::TimerBase::TimerBase(WTF::RunLoop&)::{lambda(void*)#1}::_FUN(void*) () from /lib64/libWPEWebKit-2.0.so.1 #38 0x00007f4edef1e53d in WTF::RunLoop::{lambda(_GSource*, int (*)(void*), void*)#1}::_FUN(_GSource*, int (*)(void*), void*) [clone .lto_priv.0] () from /lib64/libWPEWebKit-2.0.so.1 #39 0x00007f4edd538e8c in g_main_dispatch (context=0x556dd0b6dd50) at ../glib/gmain.c:3344 #40 g_main_context_dispatch_unlocked (context=0x556dd0b6dd50) at ../glib/gmain.c:4152 #41 0x00007f4edd59ac98 in g_main_context_iterate_unlocked.isra.0 (context=0x556dd0b6dd50, block=block@entry=1, dispatch=dispatch@entry=1, self=<optimized out>) at ../glib/gmain.c:4217 #42 0x00007f4edd53ef37 in g_main_loop_run (loop=0x556dd0b6dea0) at ../glib/gmain.c:4419 #43 0x00007f4edeeec008 in WTF::RunLoop::run() () from /lib64/libWPEWebKit-2.0.so.1 #44 0x00007f4ede0c9cd6 in WebKit::WebProcessMain(int, char**) () from /lib64/libWPEWebKit-2.0.so.1 #45 0x00007f4edd839088 in __libc_start_call_main (main=main@entry=0x556dac959070 <main>, argc=argc@entry=3, argv=argv@entry=0x7ffebc36f478) at ../sysdeps/nptl/libc_start_call_main.h:58 #46 0x00007f4edd83914b in __libc_start_main_impl (main=0x556dac959070 <main>, argc=3, argv=0x7ffebc36f478, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7ffebc36f468) at ../csu/libc-start.c:360 #47 0x0000556dac9590a5 in _start () System journal: Jul 20 19:26:04 Digital-Signage-Player weston[73445]: The per-process limit on the number of open file descriptors has been reached. Jul 20 19:26:04 Digital-Signage-Player weston[73445]: ERROR: cannot create wakeup pipe Jul 20 19:26:58 Digital-Signage-Player weston[73445]: Failed to create secure directory (/run/user/999/pulse): Too many open files Jul 20 19:26:58 Digital-Signage-Player weston[73445]: socket(): Too many open files Jul 20 19:26:58 Digital-Signage-Player weston[73445]: (WPEWebProcess:2): GStreamer-CRITICAL **: 19:26:58.952: gst_poll_get_read_gpollfd: assertion 'set != NULL' failed Jul 20 19:26:58 Digital-Signage-Player kernel: traps: WPEWebProcess[73445] trap int3 ip:7f4edd53ec28 sp:7ffebc36e000 error:0 in libglib-2.0.so.0.8000.3[7f4edd4fa000+a4000] 2047426 25 nilskemail+webkit 2024-07-21 15:32:10 -0700 (In reply to Philippe Normand from comment #23) > > Would a bundled web app be okay? > > Yes! Just some static files are likely not sufficient as I have some endpoints cannot be cleanly mapped to files (e.g. /foo/bar and /foo/bar/baz, or /foo?a and /foo?b). I can provide you with either a HAR file or a simple python server which we use for testing. What is easier for you? 2047648 26 nilskemail+webkit 2024-07-22 18:01:51 -0700 (In reply to Philippe Normand from comment #23) > > Would a bundled web app be okay? > > Yes! I uploaded it at the file share platform of my university https://gigamove.rwth-aachen.de/en/download/3de530d665fdd1fccdb78c0d62175c45 (valid for two weeks - if it expires just ping me). In the file is the web app as well as a basic python server which handles some mimetypes etc. After extracting (and installing hypercorn+quart for the python server) just run "hypercorn -b 127.0.0.1:8000 player:app". If you have any problems with the archive or getting it running please write me, I can also give you a HAR file or something else, whichever is most convenient. cog was then started with "--media-playback-requires-user-gesture=false" such that videos play automatically. --- In addition to the points from my previous comment I have again done a bit of searching for possible causes. Within the Gstreamer GitLab I found the following two issues that could be related: - https://gitlab.freedesktop.org/gstreamer/gstreamer-vaapi/-/merge_requests/27 - https://gitlab.freedesktop.org/gstreamer/gstreamer-vaapi/-/issues/106 However, these are both about the old vaapi* plugins which we are not using. We are using the new va* plugin. I wanted to note this because I am not sure if WebKit handles stuff differently depending on the decoder which are used by Gstreamer. We have a device which seems to stumble upon a driver error (https://gitlab.freedesktop.org/drm/i915/kernel/-/issues/11735) with the result that VA-API afterwards no longer functions. Curiously this device no longer seems to amass file descriptors once VA-API is broken. I did not yet have time to look further into this but I assume that WebKit/Gstreamer falls back to software rendering. I will have to set up a test where I explicitly disable the va* plugins and see if the crash still occurs (but I am short on time currently :/). 2048304 27 philn 2024-07-25 04:37:18 -0700 You should check if the leak happens with GStreamer/va. I've tested your app here (I was asking a simple app, is the 1.5MB of JS necessary for this kind of app?) with git main, on WebKitGTK on my desktop with the va plugin (AMD GPU) and it seems OK. I'll leave it running for an hour. 2048322 28 nilskemail+webkit 2024-07-25 06:21:27 -0700 (In reply to Philippe Normand from comment #27) > You should check if the leak happens with GStreamer/va. I do not have a lot of experience with the Gstreamer API and to properly test this I assume my setup should be similar to what WebKit uses (i.e. start/stop states, which sink/source etc.) > I've tested your app here (I was asking a simple app, is the 1.5MB of JS > necessary for this kind of app?) with git main, on WebKitGTK on my desktop > with the va plugin (AMD GPU) and it seems OK. I'll leave it running for an > hour. Yeah sorry about the size. A lot of that will be unused code which the bundler did not remove and the framework (Nuxt/Vue). I did not have the time to create a standalone reproducer and will be on vacation in a few days so I wanted to at least get something out there for you to check. 2051972 29 nilskemail+webkit 2024-08-11 11:39:03 -0700 I am now back from vacation. If it helps I could take a bit of time next week to try create a reproducer without any JS framework involved. However, based on https://bugs.webkit.org/show_bug.cgi?id=277076 it seems like you already found the fault? (Just not a fix yet?) 2056288 30 mcatanzaro 2024-08-29 09:43:46 -0700 (In reply to nilskemail+webkit from comment #24) > Jul 20 19:26:58 Digital-Signage-Player weston[73445]: (WPEWebProcess:2): > GStreamer-CRITICAL **: 19:26:58.952: gst_poll_get_read_gpollfd: assertion > 'set != NULL' failed I got a backtrace for this one. Normally I would report a second bug for a second crash, but in this case it seems clear enough that the fd exhaustion is the underlying problem in both cases. 2056289 31 472345 mcatanzaro 2024-08-29 09:46:14 -0700 Created attachment 472345 full backtrace (second crash) 2056291 32 mcatanzaro 2024-08-29 09:46:46 -0700 Forgot to mention: I hit this when loading https://duckduckgo.com search results page. 2056329 33 philn 2024-08-29 11:31:22 -0700 We're well aware of the stack trace, but by then the fd limits being reached, it's a bit useless... Someone would need to track this with valgrind or similar tools. 2056427 34 mcatanzaro 2024-08-29 18:02:26 -0700 So my old reproducer from comment #8 doesn't work anymore, but I found a new one: visit https://www.firstalert4.com/2024/08/29/explosion-causes-manhole-covers-blow-off-north-st-louis/ and scroll down 2056429 35 mcatanzaro 2024-08-29 18:15:28 -0700 Unfortunately my development build doesn't crash at all, only Tech Preview. So I tried: $ flatpak run -d --command=bash org.gnome.Epiphany.Devel [📦 org.gnome.Epiphany.Devel ~]$ valgrind --trace-children=yes --track-fds=yes epiphany -p https://www.firstalert4.com/2024/08/29/explosion-causes-manhole-covers-blow-off-north-st-louis/ Unfortunately the web process just immediately crashes and valgrind outputs a terrifying spew of serious-looking warnings. valgrind thinks that WebKit writes to file descriptors that it has already closed a *lot*, and valgrind does not like this. I'm not sure valgrind is correct, though; some of the warnings I see look like valgrind is getting confused by the multiple processes. I'll look closer tomorrow and report a separate bug if needed. 2056557 36 mcatanzaro 2024-08-30 07:27:51 -0700 Now https://gitlab.gnome.org/GNOME/epiphany/-/issues/2448 is crashing too. Reproducer: load the page and go do something else for a few minutes. This crash has been happening for a long time, but for some reason it's much worse as of yesterday. Not sure why. 2056610 37 mcatanzaro 2024-08-30 09:51:27 -0700 (In reply to Michael Catanzaro from comment #35) > Unfortunately the web process just immediately crashes and valgrind outputs > a terrifying spew of serious-looking warnings. valgrind thinks that WebKit > writes to file descriptors that it has already closed a *lot*, and valgrind > does not like this. I'm not sure valgrind is correct, though; some of the > warnings I see look like valgrind is getting confused by the multiple > processes. I'll look closer tomorrow and report a separate bug if needed. https://gitlab.gnome.org/GNOME/gtk/-/issues/6969 https://gitlab.gnome.org/GNOME/glib/-/merge_requests/4227 2056622 38 mcatanzaro 2024-08-30 11:14:46 -0700 I'm actually suspecting a bug in valgrind now, https://gitlab.gnome.org/GNOME/gtk/-/issues/6969#note_2211047. That will make it difficult to use to track down this bug. 2057041 39 mcatanzaro 2024-09-02 10:05:18 -0700 The valgrind developers have fixed the issue, but that doesn't do us much good because I can only reproduce the crash in my flatpak environment, so I cannot use the special valgrind I have built. 2057052 40 philn 2024-09-02 10:29:35 -0700 Can you check if the issue remains with WEBKIT_DISABLE_DMABUF_RENDERER=1 ? 2057111 41 mcatanzaro 2024-09-02 15:42:40 -0700 (In reply to Philippe Normand from comment #40) > Can you check if the issue remains with WEBKIT_DISABLE_DMABUF_RENDERER=1 ? Yes, it still crashes immediately. Most reliable and non-geoblocked reproducer: https://www.reddit.com/r/IdiotsInCars/ 2057169 42 cgarcia 2024-09-02 22:46:31 -0700 (In reply to Philippe Normand from comment #40) > Can you check if the issue remains with WEBKIT_DISABLE_DMABUF_RENDERER=1 ? Maybe you meant WEBKIT_GST_DMABUF_SINK_DISABLED? 2057206 43 philn 2024-09-03 05:43:24 -0700 (In reply to Michael Catanzaro from comment #39) > I can only reproduce the crash in my flatpak environment, so I > cannot use the special valgrind I have built. If this happens only in flatpak could it be an issue in the runtime? 2057207 44 philn 2024-09-03 05:43:53 -0700 (In reply to Carlos Garcia Campos from comment #42) > (In reply to Philippe Normand from comment #40) > > Can you check if the issue remains with WEBKIT_DISABLE_DMABUF_RENDERER=1 ? > > Maybe you meant WEBKIT_GST_DMABUF_SINK_DISABLED? Nope 2057208 45 mcatanzaro 2024-09-03 05:46:32 -0700 (In reply to Philippe Normand from comment #43) > If this happens only in flatpak could it be an issue in the runtime? I think it's actually the opposite. The videos that crash Ephy Tech Preview are all broken in my jhbuild environment. Using the reddit example, all the videos either do not start or play with audio only and no video. I have this warning, which has been there for years: (WebKitWebProcess:2): GStreamer-WARNING **: 07:44:13.839: External plugin loader failed. This most likely means that the plugin loader helper binary was not found or could not be run. You might need to set the GST_PLUGIN_SCANNER environment variable if your setup is unusual. This should normally not be required though. (In reply to Carlos Garcia Campos from comment #42) > Maybe you meant WEBKIT_GST_DMABUF_SINK_DISABLED? Still crashes immediately. 2057209 46 philn 2024-09-03 05:50:06 -0700 Your jhbuild seems broken then, and this GStreamer warning is afaik unrelated. Can you play any video with gst-play-1.0 in the jhbuild shell? 2057214 47 mcatanzaro 2024-09-03 06:04:27 -0700 (In reply to Philippe Normand from comment #46) > Your jhbuild seems broken then, and this GStreamer warning is afaik > unrelated. Can you play any video with gst-play-1.0 in the jhbuild shell? I hear audio, but I don't see any video. I see two warnings: WARNING Your GStreamer installation is missing a plug-in. WARNING debug information: ../gst/playback/gstdecodebin3.c(3312): mq_slot_check_reconfiguration (): /GstPlayBin3:playbin/GstURIDecodeBin3:uridecodebin3/GstDecodebin3:decodebin3-0: 2057215 48 philn 2024-09-03 06:16:53 -0700 > Most reliable and non-geoblocked reproducer: https://www.reddit.com/r/IdiotsInCars/ Well, I'm not sure what to do here, when you scroll away from a video, the player is paused, which means the resources are not released. So the paused players accumulate as you scroll down and the number of FDs keep growing until doom happens. Same issue happens if I make MediaPlayerPrivateGStreamer::setVisibleInViewport() return early... We have a timer in the player that will teardown the paused pipeline after 5 minutes of activity, but that's disabled for MSE players (not sure why.). 2057216 49 philn 2024-09-03 06:25:00 -0700 Thas was changed in bug 264739 ... CCing Quique. 2057223 50 mcatanzaro 2024-09-03 06:42:49 -0700 (In reply to Philippe Normand from comment #48) > Well, I'm not sure what to do here, when you scroll away from a video, the > player is paused, which means the resources are not released. So the paused > players accumulate as you scroll down and the number of FDs keep growing > until doom happens. Hm, for me the page crashes immediately. I don't have to actually play any video before the crash occurs, so I would expect all players should be paused at the time of the crash? (Maybe not?) Then the other websites that trigger this crash generally only have a couple videos. 2057233 51 eocanha 2024-09-03 07:04:22 -0700 So, the patch I brought in https://bugs.webkit.org/show_bug.cgi?id=264739 leaves "old" players in PAUSED state to avoid the last frame to disappear and become black on downstream platforms that need the player (and the internal platform-specific multimedia engine) working to show that last frame. I see some options here: a) Leave the decision to NOT move to READY and stay in PAUSED (old behaviour) as a Quirk that would be enabled on specific downstream platforms. b) Maintain a global queue of (weak?) references to MediaPlayerPrivate. For instance, holding the latest 20 active players in "PAUSED because of timeout" state. When more players need to be created and the maximum limit of "PAUSED because of timeout" active players (20) is reached, we start moving the oldest ones to READY, freeing resources at the expense of causing the "last frame shown as blank" issue that https://bugs.webkit.org/show_bug.cgi?id=264739 intended to fix. 2057234 52 eocanha 2024-09-03 07:17:57 -0700 Still, it would be good to know if a simple revert of https://commits.webkit.org/270766@main would fix the issue, to see if working on any the approaches that I mentioned actually would fix anything in the end or not. 2057235 53 nilskemail+webkit 2024-09-03 07:21:52 -0700 In the scenario where I encounter this the video elements are destroyed and only 2 are present at most times (one to show the current video and one in the background to preload the next one). So while many paused video elements might also cause fd exhaustion, in my case the fd limit is caused from a leak somewhere else 2057240 54 mcatanzaro 2024-09-03 07:49:25 -0700 Note this bug report is three months older than 270766@main. 2057604 55 mcatanzaro 2024-09-04 09:17:06 -0700 I just hit this crash on https://gitlab.gnome.org/GNOME/gtk/-/issues/6983 which appears to have only one media element. (In reply to Enrique Ocaña from comment #52) > Still, it would be good to know if a simple revert of > https://commits.webkit.org/270766@main would fix the issue, to see if > working on any the approaches that I mentioned actually would fix anything > in the end or not. Normally I would try to test this, but since the bug never occurs in my development environment, that's hard. We could try reverting it in the GNOME runtime to see what happens to Epiphany Tech Preview, but this is annoying, and since the bug predates the commit you identified, I guess it's probably not worth the effort this time? 2057860 56 eocanha 2024-09-05 04:35:45 -0700 Yeah, probably it's not worth the effort. 2104171 57 mcatanzaro 2025-03-18 14:58:18 -0700 So this issue is very similar to bug #266573. There's a bunch of relevant discussion in that issue report as well. Ultimately it's clear that we have some major file descriptor leak even on web pages with only a single media element. Unfortunately I've tested the vox.com, globalnews.ca, and firstalert4.com pages without any luck getting them to crash. (Perhaps that's a good thing, though. :) So I think the only reproducer that still works is /r/IdiotsInCars. (In reply to Michael Catanzaro from comment #35) > Unfortunately my development build doesn't crash at all, only Tech Preview. > So I tried: > > $ flatpak run -d --command=bash org.gnome.Epiphany.Devel > [📦 org.gnome.Epiphany.Devel ~]$ valgrind --trace-children=yes > --track-fds=yes epiphany -p > https://www.firstalert4.com/2024/08/29/explosion-causes-manhole-covers-blow- > off-north-st-louis/ Tested this just now and it does still crash immediately, but the crash is an unrelated bug in GeolocationController, and it does not crash reliably. Reported bug #289999. This page is clearly no longer a good test case for this issue. > Unfortunately the web process just immediately crashes and valgrind outputs > a terrifying spew of serious-looking warnings. valgrind thinks that WebKit > writes to file descriptors that it has already closed a *lot*, and valgrind > does not like this. I'm not sure valgrind is correct, though; some of the > warnings I see look like valgrind is getting confused by the multiple > processes. I'll look closer tomorrow and report a separate bug if needed. I asked the valgrind developers for help in https://bugs.kde.org/show_bug.cgi?id=492422 and the trouble should probably be fixed now, but I haven't actually tested it with WebKit yet. Let's find out. (In reply to Nils K from comment #24) > I also got a new crash with a different backtrace. This might be due to > enabling G_DEBUG=fatal-criticals for another bug report > (https://bugs.webkit.org/show_bug.cgi?id=276819) causing the process to > abort earlier (though even the first CRITICAL log message is a bit > different)... > > Backtrace: > #0 _g_log_abort (breakpoint=<optimized out>) at ../glib/gmessages.c:426 > #1 g_logv (log_domain=0x7f4edd3e89ae "GStreamer", > log_level=G_LOG_LEVEL_CRITICAL, format=<optimized out>, > args=args@entry=0x7ffebc36e0d0) at ../glib/gmessages.c:1273 > #2 0x00007f4edd53eea3 in g_log (log_domain=<optimized out>, > log_level=<optimized out>, format=<optimized out>) at > ../glib/gmessages.c:1315 > #3 0x00007f4edd347ceb in gst_bus_constructed (object=0x556dd1401960) at > ../gst/gstbus.c:184 Hi Nils, this looks like a GStreamer bug so I suggest you should report this bug to GStreamer. The problem is GstBus is not robust to failure to create the GstPoll. Now, the reason it's failing to create the GstPoll is because it is out of file descriptors, which is surely this WebKit bug, but WebKit is just exposing the problem in GStreamer. This is actually basically the same problem as bug #266573, except it's occurring in a different location, so I suggest GStreamer developers should audit any code that creates a GstPoll. 2104172 58 mcatanzaro 2025-03-18 14:59:20 -0700 (Also I reported bug #289994 to raise the fd limit, since I think we have to do that no matter what to fix pages with large numbers of media elements.) 2104175 59 nilskemail+webkit 2025-03-18 15:13:10 -0700 (In reply to Michael Catanzaro from comment #57) > So this issue is very similar to bug #266573. There's a bunch of relevant > discussion in that issue report as well. > > Ultimately it's clear that we have some major file descriptor leak even on > web pages with only a single media element. At least in our deployment of digital signage players we can no longer observe this leak in recent versions. I think this either coincided with the removal of the DMABUF sink or the switch to Skia (I think the former but I am not sure) > (In reply to Nils K from comment #24) > > I also got a new crash with a different backtrace. This might be due to > > enabling G_DEBUG=fatal-criticals for another bug report > > (https://bugs.webkit.org/show_bug.cgi?id=276819) causing the process to > > abort earlier (though even the first CRITICAL log message is a bit > > different)... > > > > Backtrace: > > #0 _g_log_abort (breakpoint=<optimized out>) at ../glib/gmessages.c:426 > > #1 g_logv (log_domain=0x7f4edd3e89ae "GStreamer", > > log_level=G_LOG_LEVEL_CRITICAL, format=<optimized out>, > > args=args@entry=0x7ffebc36e0d0) at ../glib/gmessages.c:1273 > > #2 0x00007f4edd53eea3 in g_log (log_domain=<optimized out>, > > log_level=<optimized out>, format=<optimized out>) at > > ../glib/gmessages.c:1315 > > #3 0x00007f4edd347ceb in gst_bus_constructed (object=0x556dd1401960) at > > ../gst/gstbus.c:184 > > Hi Nils, this looks like a GStreamer bug so I suggest you should report this > bug to GStreamer. The problem is GstBus is not robust to failure to create > the GstPoll. Now, the reason it's failing to create the GstPoll is because > it is out of file descriptors, which is surely this WebKit bug, but WebKit > is just exposing the problem in GStreamer. This is actually basically the > same problem as bug #266573, except it's occurring in a different location, > so I suggest GStreamer developers should audit any code that creates a > GstPoll. I think it is fine for GStreamer to simply abort in that situation because obviously something else is seriously messed up. I would even say I prefer it (after all I enabled fatal-criticals) over some limbo state in which everything and nothing might work. 2104181 60 mcatanzaro 2025-03-18 15:39:18 -0700 (In reply to Michael Catanzaro from comment #57) > I asked the valgrind developers for help in > https://bugs.kde.org/show_bug.cgi?id=492422 and the trouble should probably > be fixed now, but I haven't actually tested it with WebKit yet. Let's find > out. Unfortunately the answer is no: there are still tons of problems, and I'm not sure whether they are valgrind bugs, mesa bugs, or GTK bugs. Further investigation needed before we'll get anywhere with valgrind. (In reply to Nils K from comment #59) > I think it is fine for GStreamer to simply abort in that situation because > obviously something else is seriously messed up. I would even say I prefer > it (after all I enabled fatal-criticals) over some limbo state in which > everything and nothing might work. Yes, it should abort rather than emit a critical. 2143922 61 mcatanzaro 2025-09-18 04:52:01 -0700 (In reply to Philippe Normand from comment #40) > Can you check if the issue remains with WEBKIT_DISABLE_DMABUF_RENDERER=1 ? I replied "yes, it still crashes immediately" but interestingly we have a user in Matrix who reports that this environment variable does successfully work around the bug. Huh. 2151290 62 mcatanzaro 2025-10-15 07:33:32 -0700 *** Bug 300793 has been marked as a duplicate of this bug. *** 472345 2024-08-29 09:46:14 -0700 2024-08-29 09:46:14 -0700 full backtrace (second crash) gdb.txt text/plain 20284 mcatanzaro IzAgIF9nX2xvZ19hYm9ydCAoYnJlYWtwb2ludD08b3B0aW1pemVkIG91dD4pIGF0IC4uL2dsaWIv Z21lc3NhZ2VzLmM6NDI2CiAgICAgICAgZGVidWdnZXJfcHJlc2VudCA9IDEKICAgICAgICBkZWJ1 Z2dlcl9wcmVzZW50ID0gPG9wdGltaXplZCBvdXQ+CiMxICBnX2xvZ3YgKGxvZ19kb21haW49MHg3 ZmMyOGE0MTY5YWUgIkdTdHJlYW1lciIsIGxvZ19sZXZlbD1HX0xPR19MRVZFTF9DUklUSUNBTCwg Zm9ybWF0PTxvcHRpbWl6ZWQgb3V0PiwgYXJncz1hcmdzQGVudHJ5PTB4N2ZmZDg5MDQxN2EwKSBh dCAuLi9nbGliL2dtZXNzYWdlcy5jOjEyNzMKICAgICAgICBkb21haW4gPSAweDAKICAgICAgICBk YXRhID0gPG9wdGltaXplZCBvdXQ+CiAgICAgICAgZGVwdGggPSA8b3B0aW1pemVkIG91dD4KICAg ICAgICBsb2dfZnVuYyA9IDxvcHRpbWl6ZWQgb3V0PgogICAgICAgIGRvbWFpbl9mYXRhbF9tYXNr ID0gPG9wdGltaXplZCBvdXQ+CiAgICAgICAgbWFzcXVlcmFkZV9mYXRhbCA9IDxvcHRpbWl6ZWQg b3V0PgogICAgICAgIHRlc3RfbGV2ZWwgPSA8b3B0aW1pemVkIG91dD4KICAgICAgICB3YXNfZmF0 YWwgPSA8b3B0aW1pemVkIG91dD4KICAgICAgICB3YXNfcmVjdXJzaW9uID0gPG9wdGltaXplZCBv dXQ+CiAgICAgICAgYnVmZmVyID0gezxvcHRpbWl6ZWQgb3V0PiA8cmVwZWF0cyAxMDI1IHRpbWVz Pn0KICAgICAgICBtc2dfYWxsb2MgPSAweDU1ODFhZGE2OTk3MCAiZ3N0X3BvbGxfZ2V0X3JlYWRf Z3BvbGxmZDogYXNzZXJ0aW9uICdzZXQgIT0gTlVMTCcgZmFpbGVkIgogICAgICAgIG1zZyA9IDB4 NTU4MWFkYTY5OTcwICJnc3RfcG9sbF9nZXRfcmVhZF9ncG9sbGZkOiBhc3NlcnRpb24gJ3NldCAh PSBOVUxMJyBmYWlsZWQiCiAgICAgICAgaSA9IDMKICAgICAgICBzaXplID0gPG9wdGltaXplZCBv dXQ+CiMyICAweDAwMDA3ZmMyODk4YjhmNTMgaW4gZ19sb2cgKGxvZ19kb21haW49PG9wdGltaXpl ZCBvdXQ+LCBsb2dfbGV2ZWw9PG9wdGltaXplZCBvdXQ+LCBmb3JtYXQ9PG9wdGltaXplZCBvdXQ+ KSBhdCAuLi9nbGliL2dtZXNzYWdlcy5jOjEzMTUKICAgICAgICBhcmdzID0ge3tncF9vZmZzZXQg PSA0MCwgZnBfb2Zmc2V0ID0gNDgsIG92ZXJmbG93X2FyZ19hcmVhID0gMHg3ZmZkODkwNDE4ODAs IHJlZ19zYXZlX2FyZWEgPSAweDdmZmQ4OTA0MTdjMH19CiMzICAweDAwMDA3ZmMyOGEzNzVkZWIg aW4gZ3N0X2J1c19jb25zdHJ1Y3RlZCAob2JqZWN0PTB4NTU4MWFkYTY5NzQwIFtHc3RCdXNdKSBh dCAuLi9nc3QvZ3N0YnVzLmM6MTg0CiAgICAgICAgYnVzID0gMHg1NTgxYWRhNjk3NDAgW0dzdEJ1 c3xidXMyNjVdCiM0ICAweDAwMDA3ZmMyODk5YmYxNGEgaW4gZ19vYmplY3RfbmV3X2ludGVybmFs IChjbGFzcz0weDU1ODFhZDAwMTMwMCwgcGFyYW1zPTB4MCwgbl9wYXJhbXM9MCkgYXQgLi4vZ29i amVjdC9nb2JqZWN0LmM6MjY1NwogICAgICAgIG5xdWV1ZSA9IDB4NTU4MWFkYTY5N2QwCiAgICAg ICAgb2JqZWN0ID0gMHg1NTgxYWRhNjk3NDAgW0dzdEJ1c10KICAgICAgICBpID0gPG9wdGltaXpl ZCBvdXQ+CiAgICAgICAgX19mdW5jX18gPSAiZ19vYmplY3RfbmV3X2ludGVybmFsIgogICAgICAg IF9nX2Jvb2xlYW5fdmFyXzcyID0gPG9wdGltaXplZCBvdXQ+CiM1ICAweDAwMDA3ZmMyODk5YzA1 MTYgaW4gZ19vYmplY3RfbmV3X2ludGVybmFsIChjbGFzcz08b3B0aW1pemVkIG91dD4sIHBhcmFt cz08b3B0aW1pemVkIG91dD4sIG5fcGFyYW1zPTxvcHRpbWl6ZWQgb3V0PikgYXQgLi4vZ29iamVj dC9nb2JqZWN0LmM6MjYwMwogICAgICAgIG5xdWV1ZSA9IDxvcHRpbWl6ZWQgb3V0PgogICAgICAg IG9iamVjdCA9IDxvcHRpbWl6ZWQgb3V0PgogICAgICAgIGkgPSA8b3B0aW1pemVkIG91dD4KICAg ICAgICBucXVldWUgPSA8b3B0aW1pemVkIG91dD4KICAgICAgICBvYmplY3QgPSA8b3B0aW1pemVk IG91dD4KICAgICAgICBpID0gPG9wdGltaXplZCBvdXQ+CiAgICAgICAgX19mdW5jX18gPSAiZ19v YmplY3RfbmV3X2ludGVybmFsIgogICAgICAgIF9nX2Jvb2xlYW5fdmFyXzcyID0gPG9wdGltaXpl ZCBvdXQ+CiAgICAgICAgX2dfYm9vbGVhbl92YXJfNzMgPSA8b3B0aW1pemVkIG91dD4KICAgICAg ICBub2RlID0gPG9wdGltaXplZCBvdXQ+CiAgICAgICAgdmFsdWUgPSA8b3B0aW1pemVkIG91dD4K ICAgICAgICBwc3BlYyA9IDxvcHRpbWl6ZWQgb3V0PgogICAgICAgIGogPSA8b3B0aW1pemVkIG91 dD4KICAgICAgICB1c2VyX3NwZWNpZmllZCA9IDxvcHRpbWl6ZWQgb3V0PgojNiAgZ19vYmplY3Rf bmV3X3dpdGhfcHJvcGVydGllcyAob2JqZWN0X3R5cGU9PG9wdGltaXplZCBvdXQ+LCBuX3Byb3Bl cnRpZXM9PG9wdGltaXplZCBvdXQ+LCBuYW1lcz1uYW1lc0BlbnRyeT0weDAsIHZhbHVlcz12YWx1 ZXNAZW50cnk9MHgwKSBhdCAuLi9nb2JqZWN0L2dvYmplY3QuYzoyNzY5CiAgICAgICAgY2xhc3Mg PSAweDU1ODFhZDAwMTMwMAogICAgICAgIHVucmVmX2NsYXNzID0gPG9wdGltaXplZCBvdXQ+CiAg ICAgICAgb2JqZWN0ID0gPG9wdGltaXplZCBvdXQ+CiAgICAgICAgX19mdW5jX18gPSAiZ19vYmpl Y3RfbmV3X3dpdGhfcHJvcGVydGllcyIKIzcgIDB4MDAwMDdmYzI4OTljMTRiMSBpbiBnX29iamVj dF9uZXcgKG9iamVjdF90eXBlPTxvcHRpbWl6ZWQgb3V0PiwgZmlyc3RfcHJvcGVydHlfbmFtZT1m aXJzdF9wcm9wZXJ0eV9uYW1lQGVudHJ5PTB4MCkgYXQgLi4vZ29iamVjdC9nb2JqZWN0LmM6MjQx NQogICAgICAgIG9iamVjdCA9IDxvcHRpbWl6ZWQgb3V0PgogICAgICAgIHZhcl9hcmdzID0ge3tn cF9vZmZzZXQgPSAwLCBmcF9vZmZzZXQgPSAyMTg4OSwgb3ZlcmZsb3dfYXJnX2FyZWEgPSAweDU1 ODFhZGE2OTZkMCwgcmVnX3NhdmVfYXJlYSA9IDB4N2ZmZDg5MDQxYTYwfX0KIzggIDB4MDAwMDdm YzI4YTM3NTBiZSBpbiBnc3RfYnVzX25ldyAoKSBhdCAuLi9nc3QvZ3N0YnVzLmM6MzEwCiAgICAg ICAgcmVzdWx0ID0gPG9wdGltaXplZCBvdXQ+CiAgICAgICAgX19mdW5jX18gPSAiZ3N0X2J1c19u ZXciCiM5ICAweDAwMDA3ZmMyOGEzYmRiMTEgaW4gZ3N0X3BpcGVsaW5lX2luaXQgKHBpcGVsaW5l PTB4NTU4MWFkYTY4YmQwKSBhdCAuLi9nc3QvZ3N0cGlwZWxpbmUuYzoyNDEKICAgICAgICBidXMg PSA8b3B0aW1pemVkIG91dD4KICAgICAgICBfX2Z1bmNfXyA9ICJnc3RfcGlwZWxpbmVfaW5pdCIK IzEwIDB4MDAwMDdmYzI4OTlkYzRhMyBpbiBnX3R5cGVfY3JlYXRlX2luc3RhbmNlICh0eXBlPTxv cHRpbWl6ZWQgb3V0PikgYXQgLi4vZ29iamVjdC9ndHlwZS5jOjE5NDUKICAgICAgICBwbm9kZSA9 IDxvcHRpbWl6ZWQgb3V0PgogICAgICAgIG5vZGUgPSAweDU1ODFhZDNkNTEwMAogICAgICAgIGlu c3RhbmNlID0gMHg1NTgxYWRhNjhiZDAgW0dzdFBpcGVsaW5lXQogICAgICAgIGNsYXNzID0gMHg1 NTgxYWQzZDUxMDAgW2dfdHlwZTogR3N0UGxheUJpbi9Hc3RQaXBlbGluZS9Hc3RCaW4vR3N0RWxl bWVudC9Hc3RPYmplY3QvR0luaXRpYWxseVVub3duZWRdCiAgICAgICAgYWxsb2NhdGVkID0gPG9w dGltaXplZCBvdXQ+CiAgICAgICAgcHJpdmF0ZV9zaXplID0gPG9wdGltaXplZCBvdXQ+CiAgICAg ICAgaXZhcl9zaXplID0gPG9wdGltaXplZCBvdXQ+CiAgICAgICAgaSA9IDxvcHRpbWl6ZWQgb3V0 PgojMTEgMHgwMDAwN2ZjMjg5OWJlZmY0IGluIGdfb2JqZWN0X25ld19pbnRlcm5hbCAoY2xhc3M9 MHg1NTgxYWQzZDZmNDAsIHBhcmFtcz0weDdmZmQ4OTA0MWJmMCwgbl9wYXJhbXM9MSkgYXQgLi4v Z29iamVjdC9nb2JqZWN0LmM6MjYwNgogICAgICAgIG5xdWV1ZSA9IDxvcHRpbWl6ZWQgb3V0Pgog ICAgICAgIG9iamVjdCA9IDxvcHRpbWl6ZWQgb3V0PgogICAgICAgIGkgPSA8b3B0aW1pemVkIG91 dD4KICAgICAgICBfX2Z1bmNfXyA9ICJnX29iamVjdF9uZXdfaW50ZXJuYWwiCiAgICAgICAgX2df Ym9vbGVhbl92YXJfNzIgPSA8b3B0aW1pemVkIG91dD4KIzEyIDB4MDAwMDdmYzI4OTljMDUxNiBp biBnX29iamVjdF9uZXdfaW50ZXJuYWwgKGNsYXNzPTxvcHRpbWl6ZWQgb3V0PiwgcGFyYW1zPTxv cHRpbWl6ZWQgb3V0Piwgbl9wYXJhbXM9PG9wdGltaXplZCBvdXQ+KSBhdCAuLi9nb2JqZWN0L2dv YmplY3QuYzoyNjAzCiAgICAgICAgbnF1ZXVlID0gPG9wdGltaXplZCBvdXQ+CiAgICAgICAgb2Jq ZWN0ID0gPG9wdGltaXplZCBvdXQ+CiAgICAgICAgaSA9IDxvcHRpbWl6ZWQgb3V0PgogICAgICAg IG5xdWV1ZSA9IDxvcHRpbWl6ZWQgb3V0PgogICAgICAgIG9iamVjdCA9IDxvcHRpbWl6ZWQgb3V0 PgogICAgICAgIGkgPSA8b3B0aW1pemVkIG91dD4KICAgICAgICBfX2Z1bmNfXyA9ICJnX29iamVj dF9uZXdfaW50ZXJuYWwiCiAgICAgICAgX2dfYm9vbGVhbl92YXJfNzIgPSA8b3B0aW1pemVkIG91 dD4KICAgICAgICBfZ19ib29sZWFuX3Zhcl83MyA9IDxvcHRpbWl6ZWQgb3V0PgogICAgICAgIG5v ZGUgPSA8b3B0aW1pemVkIG91dD4KICAgICAgICB2YWx1ZSA9IDxvcHRpbWl6ZWQgb3V0PgogICAg ICAgIHBzcGVjID0gPG9wdGltaXplZCBvdXQ+CiAgICAgICAgaiA9IDxvcHRpbWl6ZWQgb3V0Pgog ICAgICAgIHVzZXJfc3BlY2lmaWVkID0gPG9wdGltaXplZCBvdXQ+CiMxMyBnX29iamVjdF9uZXdf d2l0aF9wcm9wZXJ0aWVzIChvYmplY3RfdHlwZT08b3B0aW1pemVkIG91dD4sIG5fcHJvcGVydGll cz1uX3Byb3BlcnRpZXNAZW50cnk9MSwgbmFtZXM9bmFtZXNAZW50cnk9MHg1NTgxYWRhNjZlNDAs IHZhbHVlcz12YWx1ZXNAZW50cnk9MHg1NTgxYWRhNjg5ZDApIGF0IC4uL2dvYmplY3QvZ29iamVj dC5jOjI3NjkKICAgICAgICBjbGFzcyA9IDB4NTU4MWFkM2Q2ZjQwCiAgICAgICAgdW5yZWZfY2xh c3MgPSA8b3B0aW1pemVkIG91dD4KICAgICAgICBvYmplY3QgPSA8b3B0aW1pemVkIG91dD4KICAg ICAgICBfX2Z1bmNfXyA9ICJnX29iamVjdF9uZXdfd2l0aF9wcm9wZXJ0aWVzIgojMTQgMHgwMDAw N2ZjMjhhMzkyY2Q3IGluIGdzdF9lbGVtZW50X2ZhY3RvcnlfY3JlYXRlX3dpdGhfcHJvcGVydGll cyAoZmFjdG9yeT1mYWN0b3J5QGVudHJ5PTB4NTU4MWFkMzIzYWQwIFtHc3RFbGVtZW50RmFjdG9y eXxwbGF5YmluXSwgbj1uQGVudHJ5PTEsIG5hbWVzPW5hbWVzQGVudHJ5PTB4NTU4MWFkYTY2ZTQw LCB2YWx1ZXM9dmFsdWVzQGVudHJ5PTB4NTU4MWFkYTY4OWQwKSBhdCAuLi9nc3QvZ3N0ZWxlbWVu dGZhY3RvcnkuYzo0OTQKICAgICAgICBlbGVtZW50ID0gPG9wdGltaXplZCBvdXQ+CiAgICAgICAg b2NsYXNzID0gPG9wdGltaXplZCBvdXQ+CiAgICAgICAgbmV3ZmFjdG9yeSA9IDB4NTU4MWFkMzIz YWQwIFtHc3RFbGVtZW50RmFjdG9yeXxwbGF5YmluXQogICAgICAgIF9fZnVuY19fID0gImdzdF9l bGVtZW50X2ZhY3RvcnlfY3JlYXRlX3dpdGhfcHJvcGVydGllcyIKICAgICAgICBfZ19ib29sZWFu X3Zhcl8zMiA9IDxvcHRpbWl6ZWQgb3V0PgojMTUgMHgwMDAwN2ZjMjhhMzkzMThjIGluIGdzdF9l bGVtZW50X2ZhY3RvcnlfY3JlYXRlX3ZhbGlzdCAoZmFjdG9yeT08b3B0aW1pemVkIG91dD4sIGZh Y3RvcnlAZW50cnk9MHg1NTgxYWQzMjNhZDAgW0dzdEVsZW1lbnRGYWN0b3J5fHBsYXliaW5dLCBm aXJzdD1maXJzdEBlbnRyeT0weDdmYzI4YTQyMTdlOSAibmFtZSIsIHByb3BlcnRpZXM9cHJvcGVy dGllc0BlbnRyeT0weDdmZmQ4OTA0MWRkMCkgYXQgLi4vZ3N0L2dzdGVsZW1lbnRmYWN0b3J5LmM6 NTkyCiAgICAgICAgbmV3ZmFjdG9yeSA9IDxvcHRpbWl6ZWQgb3V0PgogICAgICAgIGVsZW1lbnQg PSA8b3B0aW1pemVkIG91dD4KICAgICAgICBuYW1lcyA9IDB4NTU4MWFkYTY2ZTQwCiAgICAgICAg dmFsdWVzID0gMHg1NTgxYWRhNjg5ZDAKICAgICAgICBuID0gMQogICAgICAgIF9fZnVuY19fID0g ImdzdF9lbGVtZW50X2ZhY3RvcnlfY3JlYXRlX3ZhbGlzdCIKIzE2IDB4MDAwMDdmYzI4YTM5Mzgy ZCBpbiBnc3RfZWxlbWVudF9mYWN0b3J5X21ha2VfdmFsaXN0IChmYWN0b3J5bmFtZT0weDdmYzI5 MTY0YTkzZiAicGxheWJpbiIsIGZpcnN0PTB4N2ZjMjhhNDIxN2U5ICJuYW1lIiwgcHJvcGVydGll cz1wcm9wZXJ0aWVzQGVudHJ5PTB4N2ZmZDg5MDQxZGQwKSBhdCAuLi9nc3QvZ3N0ZWxlbWVudGZh Y3RvcnkuYzo3NTQKICAgICAgICBmYWN0b3J5ID0gMHg1NTgxYWQzMjNhZDAgW0dzdEVsZW1lbnRG YWN0b3J5fHBsYXliaW5dCiAgICAgICAgZWxlbWVudCA9IDxvcHRpbWl6ZWQgb3V0PgogICAgICAg IF9fZnVuY19fID0gImdzdF9lbGVtZW50X2ZhY3RvcnlfbWFrZV92YWxpc3QiCiMxNyAweDAwMDA3 ZmMyOGEzOTNhNGEgaW4gZ3N0X2VsZW1lbnRfZmFjdG9yeV9tYWtlX2Z1bGwgKGZhY3RvcnluYW1l PTxvcHRpbWl6ZWQgb3V0PiwgZmlyc3Q9PG9wdGltaXplZCBvdXQ+KSBhdCAuLi9nc3QvZ3N0ZWxl bWVudGZhY3RvcnkuYzo3OTkKICAgICAgICBlbGVtZW50ID0gPG9wdGltaXplZCBvdXQ+CiAgICAg ICAgcHJvcGVydGllcyA9IHt7Z3Bfb2Zmc2V0ID0gMzIsIGZwX29mZnNldCA9IDQ4LCBvdmVyZmxv d19hcmdfYXJlYSA9IDB4N2ZmZDg5MDQxZWIwLCByZWdfc2F2ZV9hcmVhID0gMHg3ZmZkODkwNDFk ZjB9fQojMTggMHgwMDAwN2ZjMjkwNmVkYTA2IGluIFdlYkNvcmU6Om1ha2VHU3RyZWFtZXJFbGVt ZW50IChmYWN0b3J5TmFtZT0weDdmYzI5MTY0YTkzZiAicGxheWJpbiIsIG5hbWU9PG9wdGltaXpl ZCBvdXQ+KSBhdCAvYnVpbGRzdHJlYW0vZ25vbWUvc2RrL3dlYmtpdGd0ay02LjAuYnN0L1NvdXJj ZS9XZWJDb3JlL3BsYXRmb3JtL2dyYXBoaWNzL2dzdHJlYW1lci9HU3RyZWFtZXJDb21tb24uY3Bw Ojk1MgogICAgICAgIGxvY2sgPSB7c3RhdGljIGlzSGVsZEJpdCA9IDEgJ1wwMDEnLCBzdGF0aWMg aGFzUGFya2VkQml0ID0gMiAnXDAwMicsIG1fYnl0ZSA9IHt2YWx1ZSA9IHN0ZDo6YXRvbWljPHVu c2lnbmVkIGNoYXI+ID0geyAwICdcMDAwJyB9fX0KICAgICAgICBjYWNoZSA9IFdURjo6VmVjdG9y IG9mIGxlbmd0aCAwLCBjYXBhY2l0eSAwCiAgICAgICAgZWxlbWVudCA9IDxvcHRpbWl6ZWQgb3V0 PgogICAgICAgIGxvY2tlciA9IHs8V1RGOjpBYnN0cmFjdExvY2tlcj4gPSB7PE5vIGRhdGEgZmll bGRzPn0sIG1fbG9jayA9IDxvcHRpbWl6ZWQgb3V0PiwgbV9pc0xvY2tlZCA9IDxvcHRpbWl6ZWQg b3V0Pn0KIzE5IDB4MDAwMDdmYzI5MDcwMWZkMSBpbiBXZWJDb3JlOjpNZWRpYVBsYXllclByaXZh dGVHU3RyZWFtZXI6OmNyZWF0ZUdTVFBsYXlCaW4gKHRoaXM9MHg3ZmMwZjA2ZjBjNDAsIHVybD0u Li4pIGF0IC9idWlsZHN0cmVhbS9nbm9tZS9zZGsvd2Via2l0Z3RrLTYuMC5ic3QvU291cmNlL1dl YkNvcmUvcGxhdGZvcm0vZ3JhcGhpY3MvZ3N0cmVhbWVyL01lZGlhUGxheWVyUHJpdmF0ZUdTdHJl YW1lci5jcHA6MzA2NwogICAgICAgIHBpcGVsaW5lSWQgPSB7dmFsdWUgPSBzdGQ6OmF0b21pYzx1 bnNpZ25lZCBpbnQ+ID0geyAyMSB9fQogICAgICAgIGVsZW1lbnRJZCA9ICJtZWRpYS1wbGF5ZXIi CiAgICAgICAgYnVzID0ge21fcHRyID0gMHgwIFtzdHJ1Y3QgX0dzdEJ1c119CiAgICAgICAgdmlk ZW9TaW5rUGFkID0ge21fcHRyID0gMHg3ZmMwZTJkNTA0ODB9CiAgICAgICAgbG9nSWRlbnRpZmll ciA9ICJtZWRpYS1wbGF5ZXItMjAiCiAgICAgICAgcGxheWVyID0ge3N0YXRpYyBpc1JlZlB0ciA9 IDxvcHRpbWl6ZWQgb3V0PiwgbV9wdHIgPSAweDdmYzEzMjE5ZTcxMH0KICAgICAgICB1c2VQbGF5 YmluMyA9IDxvcHRpbWl6ZWQgb3V0PgogICAgICAgIGlzTWVkaWFTdHJlYW0gPSBmYWxzZQogICAg ICAgIHBsYXliaW5OYW1lID0gMHg3ZmMyOTE2NGE5M2YgInBsYXliaW4iCiAgICAgICAgdHlwZSA9 IFB5dGhvbiBFeGNlcHRpb24gPGNsYXNzICdnZGIuZXJyb3InPjogTm8gc3ltYm9sICJzdGF0aWNf Y2FzdCIgaW4gY3VycmVudCBjb250ZXh0Lgp7bV9jaGFyYWN0ZXJzV2l0aE51bGxUZXJtaW5hdG9y ID0ge3N0YXRpYyBleHRlbnQgPSAxODQ0Njc0NDA3MzcwOTU1MTYxNSwgX01fcHRyID0gMHg3ZmMy OTE2YTlkMzQgIiIsIF9NX2V4dGVudCA9IHtfTV9leHRlbnRfdmFsdWUgPSAxfX19CiMyMCAweDAw MDA3ZmMyOTA3MDE4MzEgaW4gV2ViQ29yZTo6TWVkaWFQbGF5ZXJQcml2YXRlR1N0cmVhbWVyOjps b2FkICh0aGlzPTB4N2ZjMGYwNmYwYzQwLCB1cmxTdHJpbmc9PG9wdGltaXplZCBvdXQ+KSBhdCAv YnVpbGRzdHJlYW0vZ25vbWUvc2RrL3dlYmtpdGd0ay02LjAuYnN0L1NvdXJjZS9XZWJDb3JlL3Bs YXRmb3JtL2dyYXBoaWNzL2dzdHJlYW1lci9NZWRpYVBsYXllclByaXZhdGVHU3RyZWFtZXIuY3Bw OjM3NgogICAgICAgIHVybCA9IHttX3N0cmluZyA9ICJodHRwczovL3YucmVkZC5pdC8yNzUwcWc1 cGppN2MxL0RBU0hfOTYubXA0IiwgbV9pc1ZhbGlkID0gMSwgbV9wcm90b2NvbElzSW5IVFRQRmFt aWx5ID0gMSwgbV9oYXNPcGFxdWVQYXRoID0gMCwgbV9wb3J0TGVuZ3RoID0gMCwgc3RhdGljIG1h eFBvcnRMZW5ndGggPSA3LCBzdGF0aWMgbWF4U2NoZW1lTGVuZ3RoID0gNjcxMDg4NjMsIG1fc2No ZW1lRW5kID0gNSwgbV91c2VyU3RhcnQgPSA4LCBtX3VzZXJFbmQgPSA4LCBtX3Bhc3N3b3JkRW5k ID0gOCwgbV9ob3N0RW5kID0gMTcsIG1fcGF0aEFmdGVyTGFzdFNsYXNoID0gMzIsIG1fcGF0aEVu ZCA9IDQzLCBtX3F1ZXJ5RW5kID0gNDN9CiAgICAgICAgcGxheWVyID0ge3N0YXRpYyBpc1JlZlB0 ciA9IDxvcHRpbWl6ZWQgb3V0PiwgbV9wdHIgPSAweDdmYzEzMjE5ZTcxMH0KIzIxIDB4MDAwMDdm YzI5MDY2ODM4MSBpbiBXZWJDb3JlOjpNZWRpYVBsYXllcjo6bG9hZFdpdGhOZXh0TWVkaWFFbmdp bmUgKHRoaXM9MHg3ZmMxMzIxOWU3MTAsIGN1cnJlbnQ9MHgwKSBhdCAvYnVpbGRzdHJlYW0vZ25v bWUvc2RrL3dlYmtpdGd0ay02LjAuYnN0L1NvdXJjZS9XZWJDb3JlL3BsYXRmb3JtL2dyYXBoaWNz L01lZGlhUGxheWVyLmNwcDo2NTkKICAgICAgICBtZWRpYVNvdXJjZSA9IHtzdGF0aWMgaXNSZWZQ dHIgPSA8b3B0aW1pemVkIG91dD4sIG1fcHRyID0gPG9wdGltaXplZCBvdXQ+fQogICAgICAgIGVu Z2luZSA9IDxvcHRpbWl6ZWQgb3V0PgojMjIgMHgwMDAwN2ZjMjkwNjY3ZTcwIGluIFdlYkNvcmU6 Ok1lZGlhUGxheWVyOjpsb2FkICh0aGlzPTB4N2ZjMTMyMTllNzEwLCB1cmw9Li4uLCBjb250ZW50 VHlwZT0uLi4sIGtleVN5c3RlbT0iKG51bGwpIiwgcmVxdWlyZXNSZW1vdGVQbGF5YmFjaz1mYWxz ZSkgYXQgL2J1aWxkc3RyZWFtL2dub21lL3Nkay93ZWJraXRndGstNi4wLmJzdC9Tb3VyY2UvV2Vi Q29yZS9wbGF0Zm9ybS9ncmFwaGljcy9NZWRpYVBsYXllci5jcHA6NTE4CiAgICAgICAgcHJvdGVj dGVkVGhpcyA9IHtzdGF0aWMgaXNSZWYgPSA8b3B0aW1pemVkIG91dD4sIG1fcHRyID0gMHg3ZmMx MzIxOWU3MTB9CiMyMyAweDAwMDA3ZmMyOGZmZmZjN2EgaW4gV2ViQ29yZTo6SFRNTE1lZGlhRWxl bWVudDo6bG9hZFJlc291cmNlKFdURjo6VVJMIGNvbnN0JiwgV2ViQ29yZTo6Q29udGVudFR5cGUg Y29uc3QmLCBXVEY6OlN0cmluZyBjb25zdCYpOjokXzA6Om9wZXJhdG9yKCkoc3RkOjpleHBlcmlt ZW50YWw6OmZ1bmRhbWVudGFsc192Mzo6ZXhwZWN0ZWQ8V2ViQ29yZTo6Q29udGVudFR5cGUsIFdl YkNvcmU6OlBsYXRmb3JtTWVkaWFFcnJvcj4mJikgY29uc3QgKHRoaXM9MHg3ZmZkODkwNDIxYjAs IHJlc3VsdD0uLi4pIGF0IC9idWlsZHN0cmVhbS9nbm9tZS9zZGsvd2Via2l0Z3RrLTYuMC5ic3Qv U291cmNlL1dlYkNvcmUvaHRtbC9IVE1MTWVkaWFFbGVtZW50LmNwcDoxOTA3CiAgICAgICAgcHJv dGVjdGVkVGhpcyA9IHtzdGF0aWMgaXNSZWZQdHIgPSA8b3B0aW1pemVkIG91dD4sIG1fcHRyID0g MHg3ZmMwZTI3MzUwYzB9CiAgICAgICAgbWVkaWFTb3VyY2UgPSB7c3RhdGljIGlzUmVmUHRyID0g PG9wdGltaXplZCBvdXQ+LCBtX3B0ciA9IDxvcHRpbWl6ZWQgb3V0Pn0KICAgICAgICBtZWRpYVNv dXJjZSA9IHtzdGF0aWMgaXNSZWZQdHIgPSA8b3B0aW1pemVkIG91dD4sIG1fcHRyID0gPG9wdGlt aXplZCBvdXQ+fQojMjQgV2ViQ29yZTo6SFRNTE1lZGlhRWxlbWVudDo6bG9hZFJlc291cmNlICh0 aGlzPTB4N2ZjMGUyNzM1MGMwLCBpbml0aWFsVVJMPS4uLiwgaW5pdGlhbENvbnRlbnRUeXBlPS4u Liwga2V5U3lzdGVtPSIobnVsbCkiKSBhdCAvYnVpbGRzdHJlYW0vZ25vbWUvc2RrL3dlYmtpdGd0 ay02LjAuYnN0L1NvdXJjZS9XZWJDb3JlL2h0bWwvSFRNTE1lZGlhRWxlbWVudC5jcHA6MTkyNQog ICAgICAgIHJlc291cmNlID0ge3N0YXRpYyBpc1JlZlB0ciA9IDxvcHRpbWl6ZWQgb3V0PiwgbV9w dHIgPSAweDB9CiAgICAgICAgY29udGVudFR5cGUgPSB7bV90eXBlID0gIihudWxsKSIsIG1fdHlw ZVdhc0luZmVycmVkRnJvbUV4dGVuc2lvbiA9IHRydWV9CiAgICAgICAgY29tcGxldGlvbkhhbmRs ZXIgPSB7dXJsID0ge21fc3RyaW5nID0gImh0dHBzOi8vdi5yZWRkLml0LzI3NTBxZzVwamk3YzEv REFTSF85Ni5tcDQiLCBtX2lzVmFsaWQgPSAxLCBtX3Byb3RvY29sSXNJbkhUVFBGYW1pbHkgPSAx LCBtX2hhc09wYXF1ZVBhdGggPSAwLCBtX3BvcnRMZW5ndGggPSAwLCBzdGF0aWMgbWF4UG9ydExl bmd0aCA9IDcsIHN0YXRpYyBtYXhTY2hlbWVMZW5ndGggPSA2NzEwODg2MywgbV9zY2hlbWVFbmQg PSA1LCBtX3VzZXJTdGFydCA9IDgsIG1fdXNlckVuZCA9IDgsIG1fcGFzc3dvcmRFbmQgPSA4LCBt X2hvc3RFbmQgPSAxNywgbV9wYXRoQWZ0ZXJMYXN0U2xhc2ggPSAzMiwgbV9wYXRoRW5kID0gNDMs IG1fcXVlcnlFbmQgPSA0M30sIHBsYXllciA9IHtzdGF0aWMgaXNSZWZQdHIgPSA8b3B0aW1pemVk IG91dD4sIG1fcHRyID0gMHg3ZmMxMzIxOWU3MTB9LCBrZXlTeXN0ZW0gPSAiKG51bGwpIiwgd2Vh a1RoaXMgPSB7bV9pbXBsID0ge3N0YXRpYyBpc1JlZlB0ciA9IDxvcHRpbWl6ZWQgb3V0PiwgbV9w dHIgPSAweDdmYzEwMjdlZjVlMH19LCB0aGlzID0gMHg3ZmMwZTI3MzUwYzB9CiAgICAgICAgdXJs ID0gUHl0aG9uIEV4Y2VwdGlvbiA8Y2xhc3MgJ2dkYi5lcnJvcic+OiB2YWx1ZSBoYXMgYmVlbiBv cHRpbWl6ZWQgb3V0CnttX3N0cmluZyA9ICwgbV9pc1ZhbGlkID0gPG9wdGltaXplZCBvdXQ+LCBt X3Byb3RvY29sSXNJbkhUVFBGYW1pbHkgPSA8b3B0aW1pemVkIG91dD4sIG1faGFzT3BhcXVlUGF0 aCA9IDxvcHRpbWl6ZWQgb3V0PiwgbV9wb3J0TGVuZ3RoID0gPG9wdGltaXplZCBvdXQ+LCBzdGF0 aWMgbWF4UG9ydExlbmd0aCA9IDcsIHN0YXRpYyBtYXhTY2hlbWVMZW5ndGggPSA2NzEwODg2Mywg bV9zY2hlbWVFbmQgPSA8b3B0aW1pemVkIG91dD4sIG1fdXNlclN0YXJ0ID0gPG9wdGltaXplZCBv dXQ+LCBtX3VzZXJFbmQgPSA8b3B0aW1pemVkIG91dD4sIG1fcGFzc3dvcmRFbmQgPSA8b3B0aW1p emVkIG91dD4sIG1faG9zdEVuZCA9IDxvcHRpbWl6ZWQgb3V0PiwgbV9wYXRoQWZ0ZXJMYXN0U2xh c2ggPSA8b3B0aW1pemVkIG91dD4sIG1fcGF0aEVuZCA9IDxvcHRpbWl6ZWQgb3V0PiwgbV9xdWVy eUVuZCA9IDxvcHRpbWl6ZWQgb3V0Pn0KICAgICAgICBmcmFtZSA9IHtzdGF0aWMgaXNSZWZQdHIg PSA8b3B0aW1pemVkIG91dD4sIG1fcHRyID0gMHg3ZmMyNmUwZTgzNDB9CiAgICAgICAgcGFnZSA9 IHtzdGF0aWMgaXNSZWZQdHIgPSA8b3B0aW1pemVkIG91dD4sIG1fcHRyID0gMHg3ZmMyNmUwYjA3 MzB9CiAgICAgICAgcHJpdmF0ZU1vZGUgPSA8b3B0aW1pemVkIG91dD4KICAgICAgICBwbGF5ZXIg PSB7c3RhdGljIGlzUmVmUHRyID0gPG9wdGltaXplZCBvdXQ+LCBtX3B0ciA9IDB4N2ZjMTMyMTll NzEwfQojMjUgMHgwMDAwN2ZjMjkwMDI3ZjNhIGluIFdlYkNvcmU6OkhUTUxNZWRpYUVsZW1lbnQ6 OnNlbGVjdE1lZGlhUmVzb3VyY2UoKTo6JF8wOjpvcGVyYXRvcigpKCkgY29uc3QgKHRoaXM9PG9w dGltaXplZCBvdXQ+KSBhdCAvYnVpbGRzdHJlYW0vZ25vbWUvc2RrL3dlYmtpdGd0ay02LjAuYnN0 L1NvdXJjZS9XZWJDb3JlL2h0bWwvSFRNTE1lZGlhRWxlbWVudC5jcHA6MTcxOQogICAgICAgIGNv bnRlbnRUeXBlID0ge21fdHlwZSA9ICIobnVsbCkiLCBtX3R5cGVXYXNJbmZlcnJlZEZyb21FeHRl bnNpb24gPSBmYWxzZX0KICAgICAgICBtb2RlID0gQXR0cmlidXRlCiAgICAgICAgdGV4dFRyYWNr cyA9IHtzdGF0aWMgaXNSZWZQdHIgPSA8b3B0aW1pemVkIG91dD4sIG1fcHRyID0gPG9wdGltaXpl ZCBvdXQ+fQogICAgICAgIGkgPSA8b3B0aW1pemVkIG91dD4KICAgICAgICB0cmFjayA9IHtzdGF0 aWMgaXNSZWZQdHIgPSA8b3B0aW1pemVkIG91dD4sIG1fcHRyID0gPG9wdGltaXplZCBvdXQ+fQog ICAgICAgIGZpcnN0U291cmNlID0gPG9wdGltaXplZCBvdXQ+CiAgICAgICAgY29udGVudFR5cGUg PSBQeXRob24gRXhjZXB0aW9uIDxjbGFzcyAnZ2RiLmVycm9yJz46IHZhbHVlIGhhcyBiZWVuIG9w dGltaXplZCBvdXQKe21fdHlwZSA9ICwgbV90eXBlV2FzSW5mZXJyZWRGcm9tRXh0ZW5zaW9uID0g PG9wdGltaXplZCBvdXQ+fQogICAgICAgIGNvbnRlbnRUeXBlID0gUHl0aG9uIEV4Y2VwdGlvbiA8 Y2xhc3MgJ2dkYi5lcnJvcic+OiB2YWx1ZSBoYXMgYmVlbiBvcHRpbWl6ZWQgb3V0CnttX3R5cGUg PSAsIG1fdHlwZVdhc0luZmVycmVkRnJvbUV4dGVuc2lvbiA9IDxvcHRpbWl6ZWQgb3V0Pn0KICAg ICAgICBhYnNvbHV0ZVVSTCA9IFB5dGhvbiBFeGNlcHRpb24gPGNsYXNzICdnZGIuZXJyb3InPjog dmFsdWUgaGFzIGJlZW4gb3B0aW1pemVkIG91dAp7bV9zdHJpbmcgPSAsIG1faXNWYWxpZCA9IDxv cHRpbWl6ZWQgb3V0PiwgbV9wcm90b2NvbElzSW5IVFRQRmFtaWx5ID0gPG9wdGltaXplZCBvdXQ+ LCBtX2hhc09wYXF1ZVBhdGggPSA8b3B0aW1pemVkIG91dD4sIG1fcG9ydExlbmd0aCA9IDxvcHRp bWl6ZWQgb3V0Piwgc3RhdGljIG1heFBvcnRMZW5ndGggPSA3LCBzdGF0aWMgbWF4U2NoZW1lTGVu Z3RoID0gNjcxMDg4NjMsIG1fc2NoZW1lRW5kID0gPG9wdGltaXplZCBvdXQ+LCBtX3VzZXJTdGFy dCA9IDxvcHRpbWl6ZWQgb3V0PiwgbV91c2VyRW5kID0gPG9wdGltaXplZCBvdXQ+LCBtX3Bhc3N3 b3JkRW5kID0gPG9wdGltaXplZCBvdXQ+LCBtX2hvc3RFbmQgPSA8b3B0aW1pemVkIG91dD4sIG1f cGF0aEFmdGVyTGFzdFNsYXNoID0gPG9wdGltaXplZCBvdXQ+LCBtX3BhdGhFbmQgPSA8b3B0aW1p emVkIG91dD4sIG1fcXVlcnlFbmQgPSA8b3B0aW1pemVkIG91dD59CiMyNiBXVEY6OkRldGFpbDo6 Q2FsbGFibGVXcmFwcGVyPFdlYkNvcmU6OkhUTUxNZWRpYUVsZW1lbnQ6OnNlbGVjdE1lZGlhUmVz b3VyY2UoKTo6JF8wLCB2b2lkPjo6Y2FsbCAodGhpcz08b3B0aW1pemVkIG91dD4pIGF0IFdURi9I ZWFkZXJzL3d0Zi9GdW5jdGlvbi5oOjUzCiMyNyAweDAwMDA3ZmMyOGZkZDk3NjQgaW4gV2ViQ29y ZTo6RXZlbnRMb29wOjpydW4gKHRoaXM9MHg3ZmMyNmUwYWI1MDAsIGRlYWRsaW5lPTxlcnJvciBy ZWFkaW5nIHZhcmlhYmxlOiBUaGF0IG9wZXJhdGlvbiBpcyBub3QgYXZhaWxhYmxlIG9uIGludGVn ZXJzIG9mIG1vcmUgdGhhbiA4IGJ5dGVzLj4pIGF0IC9idWlsZHN0cmVhbS9nbm9tZS9zZGsvd2Vi a2l0Z3RrLTYuMC5ic3QvU291cmNlL1dlYkNvcmUvZG9tL0V2ZW50TG9vcC5jcHA6MzI3CiAgICAg ICAgZ3JvdXAgPSAweDdmYzI2ZTM5OWQ0MAogICAgICAgIHRhc2sgPSBzdGQ6OnVuaXF1ZV9wdHI8 V2ViQ29yZTo6RXZlbnRMb29wVGFzaz4gPSB7Z2V0KCkgPSAweDdmYzE3NjJhYzVjMH0KICAgICAg ICBfX2JlZ2luMiA9IDB4N2ZjMTAyZGI5MmY4CiAgICAgICAgX19lbmQyID0gPG9wdGltaXplZCBv dXQ+CiAgICAgICAgX19yYW5nZTIgPSA8b3B0aW1pemVkIG91dD4KICAgICAgICByZW1haW5pbmdU YXNrcyA9IFB5dGhvbiBFeGNlcHRpb24gPGNsYXNzICdnZGIuZXJyb3InPjogdmFsdWUgaGFzIGJl ZW4gb3B0aW1pemVkIG91dAoKICAgICAgICB0YXNrcyA9IFB5dGhvbiBFeGNlcHRpb24gPGNsYXNz ICdnZGIuZXJyb3InPjogdmFsdWUgaGFzIGJlZW4gb3B0aW1pemVkIG91dAoKICAgICAgICBoYXNS ZWFjaGVkRGVhZGxpbmUgPSBmYWxzZQogICAgICAgIGRpZFBlcmZvcm1NaWNyb3Rhc2tDaGVja3Bv aW50ID0gdHJ1ZQojMjggMHgwMDAwN2ZjMjhmZTgwNzNjIGluIFdlYkNvcmU6OldpbmRvd0V2ZW50 TG9vcDo6ZGlkUmVhY2hUaW1lVG9SdW4gKHRoaXM9MHg3ZmMyNmUwYWI1MDApIGF0IC9idWlsZHN0 cmVhbS9nbm9tZS9zZGsvd2Via2l0Z3RrLTYuMC5ic3QvU291cmNlL1dlYkNvcmUvZG9tL1dpbmRv d0V2ZW50TG9vcC5jcHA6MjEwCiAgICAgICAgcHJvdGVjdGVkVGhpcyA9IHtzdGF0aWMgaXNSZWYg PSA8b3B0aW1pemVkIG91dD4sIG1fcHRyID0gMHg3ZmMyNmUwYWI1MDB9CiAgICAgICAgZGVhZGxp bmUgPSB7PFdURjo6R2VuZXJpY1RpbWVNaXhpbjxXVEY6OkFwcHJveGltYXRlVGltZT4+ID0ge21f dmFsdWUgPSA2Ljk1MjgzMjc4MjIzMDE0OTdlLTMxMH0sIHN0YXRpYyBjbG9ja1R5cGUgPSBXVEY6 OkNsb2NrVHlwZTo6QXBwcm94aW1hdGV9CiAgICAgICAgaGFzSWRsZUNhbGxiYWNrcyA9IDxvcHRp bWl6ZWQgb3V0PgojMjkgMHgwMDAwN2ZjMjkwNTZiODlmIGluIFdlYkNvcmU6OlRocmVhZFRpbWVy czo6c2hhcmVkVGltZXJGaXJlZEludGVybmFsICh0aGlzPTB4N2ZjMjZlMGU1MjAwKSBhdCAvYnVp bGRzdHJlYW0vZ25vbWUvc2RrL3dlYmtpdGd0ay02LjAuYnN0L1NvdXJjZS9XZWJDb3JlL3BsYXRm b3JtL1RocmVhZFRpbWVycy5jcHA6MTI1CiAgICAgICAgaXRlbSA9IHtzdGF0aWMgaXNSZWYgPSA8 b3B0aW1pemVkIG91dD4sIG1fcHRyID0gMHg3ZmMyMDYwMDEzMzB9CiAgICAgICAgdGltZXIgPSA8 b3B0aW1pemVkIG91dD4KICAgICAgICBpbnRlcnZhbCA9IHttX3ZhbHVlID0gPG9wdGltaXplZCBv dXQ+fQpQeXRob24gRXhjZXB0aW9uIDxjbGFzcyAnZ2RiLmVycm9yJz46IFRoYXQgb3BlcmF0aW9u IGlzIG5vdCBhdmFpbGFibGUgb24gaW50ZWdlcnMgb2YgbW9yZSB0aGFuIDggYnl0ZXMuCiMzMCAw eDAwMDA3ZmMyOGQ0OGQwMjUgaW4gV1RGOjpSdW5Mb29wOjpUaW1lckJhc2U6OlRpbWVyQmFzZShX VEY6OlJ1bkxvb3AmKTo6JF8wOjpvcGVyYXRvcigpKHZvaWQqKSBjb25zdCAodXNlckRhdGE9MHg3 ZmMyOTI3MWIxYzggPFdlYkNvcmU6Ok1haW5UaHJlYWRTaGFyZWRUaW1lcjo6c2luZ2xldG9uKCk6 Omluc3RhbmNlKzE2PiwgdGhpcz08b3B0aW1pemVkIG91dD4pIGF0IC9idWlsZHN0cmVhbS9nbm9t ZS9zZGsvd2Via2l0Z3RrLTYuMC5ic3QvU291cmNlL1dURi93dGYvZ2xpYi9SdW5Mb29wR0xpYi5j cHA6MTc3CiAgICAgICAgdGltZXIgPSAweDdmYzI5MjcxYjFjOCA8V2ViQ29yZTo6TWFpblRocmVh ZFNoYXJlZFRpbWVyOjpzaW5nbGV0b24oKTo6aW5zdGFuY2UrMTY+CiAgICAgICAgc291cmNlID0g MHg1NTgxYWJjNzNkZTAKIzMxIFdURjo6UnVuTG9vcDo6VGltZXJCYXNlOjpUaW1lckJhc2UoV1RG OjpSdW5Mb29wJik6OiRfMDo6X19pbnZva2Uodm9pZCopICh1c2VyRGF0YT0weDdmYzI5MjcxYjFj OCA8V2ViQ29yZTo6TWFpblRocmVhZFNoYXJlZFRpbWVyOjpzaW5nbGV0b24oKTo6aW5zdGFuY2Ur MTY+KSBhdCAvYnVpbGRzdHJlYW0vZ25vbWUvc2RrL3dlYmtpdGd0ay02LjAuYnN0L1NvdXJjZS9X VEYvd3RmL2dsaWIvUnVuTG9vcEdMaWIuY3BwOjE2OQojMzIgMHgwMDAwN2ZjMjhkNDhjMGYxIGlu IFdURjo6UnVuTG9vcDo6JF8wOjpvcGVyYXRvcigpIChzb3VyY2U9MHg1NTgxYWJjNzNkZTAsIGNh bGxiYWNrPTB4N2ZjMjhkNDhjZjkwIDxXVEY6OlJ1bkxvb3A6OlRpbWVyQmFzZTo6VGltZXJCYXNl KFdURjo6UnVuTG9vcCYpOjokXzA6Ol9faW52b2tlKHZvaWQqKT4sIHVzZXJEYXRhPTB4N2ZjMjky NzFiMWM4IDxXZWJDb3JlOjpNYWluVGhyZWFkU2hhcmVkVGltZXI6OnNpbmdsZXRvbigpOjppbnN0 YW5jZSsxNj4sIHRoaXM9PG9wdGltaXplZCBvdXQ+KSBhdCAvYnVpbGRzdHJlYW0vZ25vbWUvc2Rr L3dlYmtpdGd0ay02LjAuYnN0L1NvdXJjZS9XVEYvd3RmL2dsaWIvUnVuTG9vcEdMaWIuY3BwOjUz CiAgICAgICAgbmFtZSA9IDB4NTU4MWFiYzZkZWUwICJbV2ViS2l0XSBNYWluVGhyZWFkU2hhcmVk VGltZXIiCiAgICAgICAgcnVuTG9vcFNvdXJjZSA9IEAweDU1ODFhYmM3M2RlMDoge3NvdXJjZSA9 IHtjYWxsYmFja19kYXRhID0gMHg1NTgxYWJjNzNhYzAsIGNhbGxiYWNrX2Z1bmNzID0gMHg3ZmMy ODk5OWQyNjAgPGdfc291cmNlX2NhbGxiYWNrX2Z1bmNzPiwgc291cmNlX2Z1bmNzID0gMHg3ZmMy OGQ5NTI0NTggPFdURjo6UnVuTG9vcDo6c19ydW5Mb29wU291cmNlRnVuY3Rpb25zPiwgcmVmX2Nv dW50ID0gMywgY29udGV4dCA9IDB4NTU4MWFiYzRkZjMwLCBwcmlvcml0eSA9IDEwMCwgZmxhZ3Mg PSA2Nywgc291cmNlX2lkID0gMjMsIHBvbGxfZmRzID0gMHgwLCBwcmV2ID0gMHg1NTgxYWJjNzNl ODAsIG5leHQgPSAweDU1ODFhZDNkZGQ1MCwgbmFtZSA9IDB4NTU4MWFiYzZkZWUwICJbV2ViS2l0 XSBNYWluVGhyZWFkU2hhcmVkVGltZXIiLCBwcml2ID0gMHg1NTgxYWJjODM3NDB9LCBydW5Mb29w ID0gMHg3ZmMyNmUwMTgwZTB9CiAgICAgICAgcmV0dXJuVmFsdWUgPSA8b3B0aW1pemVkIG91dD4K IzMzIFdURjo6UnVuTG9vcDo6JF8wOjpfX2ludm9rZSAoc291cmNlPTB4NTU4MWFiYzczZGUwLCBj YWxsYmFjaz0weDdmYzI4ZDQ4Y2Y5MCA8V1RGOjpSdW5Mb29wOjpUaW1lckJhc2U6OlRpbWVyQmFz ZShXVEY6OlJ1bkxvb3AmKTo6JF8wOjpfX2ludm9rZSh2b2lkKik+LCB1c2VyRGF0YT0weDdmYzI5 MjcxYjFjOCA8V2ViQ29yZTo6TWFpblRocmVhZFNoYXJlZFRpbWVyOjpzaW5nbGV0b24oKTo6aW5z dGFuY2UrMTY+KSBhdCAvYnVpbGRzdHJlYW0vZ25vbWUvc2RrL3dlYmtpdGd0ay02LjAuYnN0L1Nv dXJjZS9XVEYvd3RmL2dsaWIvUnVuTG9vcEdMaWIuY3BwOjQ1CiMzNCAweDAwMDA3ZmMyODk4YWRi MjcgaW4gZ19tYWluX2Rpc3BhdGNoIChjb250ZXh0PWNvbnRleHRAZW50cnk9MHg1NTgxYWJjNGRm MzApIGF0IC4uL2dsaWIvZ21haW4uYzozMzU3CiAgICAgICAgZGlzcGF0Y2ggPSAweDdmYzI4ZDQ4 YzBhMCA8V1RGOjpSdW5Mb29wOjokXzA6Ol9faW52b2tlKF9HU291cmNlKiwgaW50ICgqKSh2b2lk KiksIHZvaWQqKT4KICAgICAgICBwcmV2X3NvdXJjZSA9IDB4MAogICAgICAgIGJlZ2luX3RpbWVf bnNlYyA9IDEwNTYwMTkyMzU4NDE0CiAgICAgICAgd2FzX2luX2NhbGwgPSAwCiAgICAgICAgdXNl cl9kYXRhID0gMHg3ZmMyOTI3MWIxYzggPFdlYkNvcmU6Ok1haW5UaHJlYWRTaGFyZWRUaW1lcjo6 c2luZ2xldG9uKCk6Omluc3RhbmNlKzE2PgogICAgICAgIGNhbGxiYWNrID0gMHg3ZmMyOGQ0OGNm OTAgPFdURjo6UnVuTG9vcDo6VGltZXJCYXNlOjpUaW1lckJhc2UoV1RGOjpSdW5Mb29wJik6OiRf MDo6X19pbnZva2Uodm9pZCopPgogICAgICAgIGNiX2Z1bmNzID0gMHg3ZmMyODk5OWQyNjAgPGdf c291cmNlX2NhbGxiYWNrX2Z1bmNzPgogICAgICAgIGNiX2RhdGEgPSAweDU1ODFhYmM3M2FjMAog ICAgICAgIG5lZWRfZGVzdHJveSA9IDxvcHRpbWl6ZWQgb3V0PgogICAgICAgIHNvdXJjZSA9IDB4 NTU4MWFiYzczZGUwCiAgICAgICAgY3VycmVudCA9IDB4NTU4MWFiYzVmMDgwCiAgICAgICAgaSA9 IDEKICAgICAgICBfX2Z1bmNfXyA9ICJnX21haW5fZGlzcGF0Y2giCiMzNSAweDAwMDA3ZmMyODk4 YWZkZjcgaW4gZ19tYWluX2NvbnRleHRfZGlzcGF0Y2hfdW5sb2NrZWQgKGNvbnRleHQ9MHg1NTgx YWJjNGRmMzApIGF0IC4uL2dsaWIvZ21haW4uYzo0MjA4CiMzNiBnX21haW5fY29udGV4dF9pdGVy YXRlX3VubG9ja2VkIChjb250ZXh0PTB4NTU4MWFiYzRkZjMwLCBibG9jaz1ibG9ja0BlbnRyeT0x LCBkaXNwYXRjaD1kaXNwYXRjaEBlbnRyeT0xLCBzZWxmPTxvcHRpbWl6ZWQgb3V0PikgYXQgLi4v Z2xpYi9nbWFpbi5jOjQyNzMKICAgICAgICBtYXhfcHJpb3JpdHkgPSAxMDAKICAgICAgICB0aW1l b3V0X3VzZWMgPSAwCiAgICAgICAgc29tZV9yZWFkeSA9IDEKICAgICAgICBuZmRzID0gMjAKICAg ICAgICBhbGxvY2F0ZWRfbmZkcyA9IDxvcHRpbWl6ZWQgb3V0PgogICAgICAgIGZkcyA9IDB4NTU4 MWFkOWRlNWYwCiAgICAgICAgYmVnaW5fdGltZV9uc2VjID0gMTA1NjAxOTIwMjE2NjcKIzM3IDB4 MDAwMDdmYzI4OThiMDhkNyBpbiBnX21haW5fbG9vcF9ydW4gKGxvb3A9MHg1NTgxYWJjNDY4MzAp IGF0IC4uL2dsaWIvZ21haW4uYzo0NDc1CiAgICAgICAgc2VsZiA9IDxvcHRpbWl6ZWQgb3V0Pgog ICAgICAgIF9fZnVuY19fID0gImdfbWFpbl9sb29wX3J1biIKIzM4IDB4MDAwMDdmYzI4ZDQ4YzZl ZCBpbiBXVEY6OlJ1bkxvb3A6OnJ1biAoKSBhdCAvYnVpbGRzdHJlYW0vZ25vbWUvc2RrL3dlYmtp dGd0ay02LjAuYnN0L1NvdXJjZS9XVEYvd3RmL2dsaWIvUnVuTG9vcEdMaWIuY3BwOjEwOAogICAg ICAgIHJ1bkxvb3AgPSB7c3RhdGljIGlzUmVmID0gPG9wdGltaXplZCBvdXQ+LCBtX3B0ciA9IDB4 N2ZjMjZlMDE4MGUwfQogICAgICAgIG1haW5Db250ZXh0ID0gMHg1NTgxYWJjNGRmMzAKICAgICAg ICBpbm5lcm1vc3RMb29wID0gMHg1NTgxYWJjNDY4MzAKICAgICAgICBuZXN0ZWRNYWluTG9vcCA9 IDxvcHRpbWl6ZWQgb3V0PgojMzkgMHgwMDAwN2ZjMjhlYzE5ZThmIGluIFdlYktpdDo6QXV4aWxp YXJ5UHJvY2Vzc01haW5CYXNlPFdlYktpdDo6V2ViUHJvY2VzcywgdHJ1ZT46OnJ1biAodGhpcz0w eDdmZmQ4OTA0MjY3MCwgYXJnYz08b3B0aW1pemVkIG91dD4sIGFyZ3Y9PG9wdGltaXplZCBvdXQ+ KSBhdCAvYnVpbGRzdHJlYW0vZ25vbWUvc2RrL3dlYmtpdGd0ay02LjAuYnN0L1NvdXJjZS9XZWJL aXQvU2hhcmVkL0F1eGlsaWFyeVByb2Nlc3NNYWluLmg6NzIKIzQwIFdlYktpdDo6QXV4aWxpYXJ5 UHJvY2Vzc01haW48V2ViS2l0OjpXZWJQcm9jZXNzTWFpbkd0az4gKGFyZ2M9PG9wdGltaXplZCBv dXQ+LCBhcmd2PTxvcHRpbWl6ZWQgb3V0PikgYXQgL2J1aWxkc3RyZWFtL2dub21lL3Nkay93ZWJr aXRndGstNi4wLmJzdC9Tb3VyY2UvV2ViS2l0L1NoYXJlZC9BdXhpbGlhcnlQcm9jZXNzTWFpbi5o Ojk4CiAgICAgICAgYXV4aWxpYXJ5TWFpbiA9IHttX3N0b3JhZ2UgPSB7X19kYXRhID0gIlwzMDBc MzU0QlwyMjJcMzAyXDE3NyIsICdcMDAwJyA8cmVwZWF0cyAyNiB0aW1lcz4sICJcMjI1XDAwMlww MDBcMDAwXDAwMFwwMDBcMDAwXDAwMFwwMDFcMDAwXDAwMFwwMDBcMDAwXDAwMFwwMDBcMDAwXlww MDEiLCAnXDAwMCcgPHJlcGVhdHMgMjEgdGltZXM+LCBfX2FsaWduID0gezxObyBkYXRhIGZpZWxk cz59fX0KIzQxIFdlYktpdDo6V2ViUHJvY2Vzc01haW4gKGFyZ2M9NCwgYXJndj0weDdmZmQ4OTA0 MjgwOCkgYXQgL2J1aWxkc3RyZWFtL2dub21lL3Nkay93ZWJraXRndGstNi4wLmJzdC9Tb3VyY2Uv V2ViS2l0L1dlYlByb2Nlc3MvZ3RrL1dlYlByb2Nlc3NNYWluR3RrLmNwcDoxMDYKIzQyIDB4MDAw MDdmYzI4ZGMyZjE0OCBpbiBfX2xpYmNfc3RhcnRfY2FsbF9tYWluIChtYWluPW1haW5AZW50cnk9 MHg1NTgxODc4YTMxNTAgPG1haW4oaW50LCBjaGFyKiopPiwgYXJnYz1hcmdjQGVudHJ5PTQsIGFy Z3Y9YXJndkBlbnRyeT0weDdmZmQ4OTA0MjgwOCkgYXQgLi4vc3lzZGVwcy9ucHRsL2xpYmNfc3Rh cnRfY2FsbF9tYWluLmg6NTgKICAgICAgICBzZWxmID0gPG9wdGltaXplZCBvdXQ+CiAgICAgICAg cmVzdWx0ID0gPG9wdGltaXplZCBvdXQ+CiAgICAgICAgdW53aW5kX2J1ZiA9IHtjYW5jZWxfam1w X2J1ZiA9IHt7am1wX2J1ZiA9IHsxNDA3MjY5MDIyMDQ0MjQsIC03MDI1Nzk4NjI2NzAwMDcwNTM1 LCA0LCAwLCAxNDA0NzM2NTk1MzUzNjAsIDk0MDE0ODEzMTM0MjQwLCAtNzAyNTc5ODYyNjY1MTgz NjAzOSwgLTcwNjEyNzM4NzY4MTE3Nzk3MTl9LCBtYXNrX3dhc19zYXZlZCA9IDB9fSwgcHJpdiA9 IHtwYWQgPSB7MHgwLCAweDAsIDB4NCwgMHg3ZmZkODkwNDI4MDB9LCBkYXRhID0ge3ByZXYgPSAw eDAsIGNsZWFudXAgPSAweDAsIGNhbmNlbHR5cGUgPSA0fX19CiAgICAgICAgbm90X2ZpcnN0X2Nh bGwgPSA8b3B0aW1pemVkIG91dD4KIzQzIDB4MDAwMDdmYzI4ZGMyZjIwYiBpbiBfX2xpYmNfc3Rh cnRfbWFpbl9pbXBsIChtYWluPTB4NTU4MTg3OGEzMTUwIDxtYWluKGludCwgY2hhcioqKT4sIGFy Z2M9NCwgYXJndj0weDdmZmQ4OTA0MjgwOCwgaW5pdD08b3B0aW1pemVkIG91dD4sIGZpbmk9PG9w dGltaXplZCBvdXQ+LCBydGxkX2Zpbmk9PG9wdGltaXplZCBvdXQ+LCBzdGFja19lbmQ9MHg3ZmZk ODkwNDI3ZjgpIGF0IC4uL2NzdS9saWJjLXN0YXJ0LmM6MzYwCiM0NCAweDAwMDA1NTgxODc4YTMw ODUgaW4gX3N0YXJ0ICgpIGF0IC4uL3N5c2RlcHMveDg2XzY0L3N0YXJ0LlM6MTE1Cg==