Copy link to clipboard
Copied
We're in the midst of deploying an update of firefox/flash to 6500 production systems and we are seeing abrt captures of firefox's plugin-container crashing with flash on about 35% of the currently deployed population, several times a day. Apparently it is transparent to the users, as they are not complaining, but we still don't like the idea of multiple application crashes happening regularly.
Platform: RHEL6.9 workstation 32 bit
Browser: firefox-52.6.0
flash-plugin: 28.0.0.161
reproducer page: Adobe - Flash Player (hit refresh multiple times and you can get it to segfault fairly often)
I went through several iterations today of firefox versions and flash versions and it seems to fail in the same fashion all the way through firefox-52.6.0 and flash-plugin-30.0.0.134. If I go back to our production version of flash-plugin-26.0.0.151, I do not see the segfaults.
I have a case open with RedHat, but they have now sent me to Adobe.
core_backtrace from an abrt capture:
{ "signal": 11
, "executable": "/usr/lib/firefox/plugin-container"
, "stacktrace":
[ { "crash_thread": true
, "frames":
[ { "address": 13083005
, "build_id": "70d3b57317ab1e2c9f6c12e59b19f7ba6b489831"
, "build_id_offset": 86397
, "function_name": "_dl_close"
, "file_name": "/lib/ld-linux.so.2"
}
, { "address": 10968276
, "build_id": "7d578f69c37d9e6891d442bb76fb9c363ceea631"
, "build_id_offset": 3284
, "function_name": "dlclose_doit"
, "file_name": "/lib/libdl.so.2"
}
, { "address": 13058070
, "build_id": "70d3b57317ab1e2c9f6c12e59b19f7ba6b489831"
, "build_id_offset": 61462
, "function_name": "_dl_catch_error"
, "file_name": "/lib/ld-linux.so.2"
}
, { "address": 10969148
, "build_id": "7d578f69c37d9e6891d442bb76fb9c363ceea631"
, "build_id_offset": 4156
, "function_name": "_dlerror_run"
, "file_name": "/lib/libdl.so.2"
}
, { "address": 10968330
, "build_id": "7d578f69c37d9e6891d442bb76fb9c363ceea631"
, "build_id_offset": 3338
, "function_name": "dlclose"
, "file_name": "/lib/libdl.so.2"
}
, { "address": 102691593
, "build_id": "7e17a6b0483d2c10799f8fdb6f945360737f259b"
, "build_id_offset": 5837577
, "file_name": "/usr/lib/flash-plugin/libflashplayer.so"
}
, { "address": 102693756
, "build_id": "7e17a6b0483d2c10799f8fdb6f945360737f259b"
, "build_id_offset": 5839740
, "file_name": "/usr/lib/flash-plugin/libflashplayer.so"
}
, { "address": 102469675
, "build_id": "7e17a6b0483d2c10799f8fdb6f945360737f259b"
, "build_id_offset": 5615659
, "file_name": "/usr/lib/flash-plugin/libflashplayer.so"
}
, { "address": 102469228
, "build_id": "7e17a6b0483d2c10799f8fdb6f945360737f259b"
, "build_id_offset": 5615212
, "file_name": "/usr/lib/flash-plugin/libflashplayer.so"
}
, { "address": 100547146
, "build_id": "7e17a6b0483d2c10799f8fdb6f945360737f259b"
, "build_id_offset": 3693130
, "file_name": "/usr/lib/flash-plugin/libflashplayer.so"
}
, { "address": 101903535
, "build_id": "7e17a6b0483d2c10799f8fdb6f945360737f259b"
, "build_id_offset": 5049519
, "file_name": "/usr/lib/flash-plugin/libflashplayer.so"
}
, { "address": 102378833
, "build_id": "7e17a6b0483d2c10799f8fdb6f945360737f259b"
, "build_id_offset": 5524817
, "file_name": "/usr/lib/flash-plugin/libflashplayer.so"
}
, { "address": 100527034
, "build_id": "7e17a6b0483d2c10799f8fdb6f945360737f259b"
, "build_id_offset": 3673018
, "file_name": "/usr/lib/flash-plugin/libflashplayer.so"
}
, { "address": 102380524
, "build_id": "7e17a6b0483d2c10799f8fdb6f945360737f259b"
, "build_id_offset": 5526508
, "file_name": "/usr/lib/flash-plugin/libflashplayer.so"
}
, { "address": 102408026
, "build_id": "7e17a6b0483d2c10799f8fdb6f945360737f259b"
, "build_id_offset": 5554010
, "file_name": "/usr/lib/flash-plugin/libflashplayer.so"
}
, { "address": 36616889
, "build_id": "f700a30b862f44a2e401700e56bbb220f8439201"
, "build_id_offset": 23485113
, "file_name": "/usr/lib/firefox/libxul.so"
}
, { "address": 36617665
, "build_id": "f700a30b862f44a2e401700e56bbb220f8439201"
, "build_id_offset": 23485889
, "file_name": "/usr/lib/firefox/libxul.so"
}
, { "address": 21244247
, "build_id": "f700a30b862f44a2e401700e56bbb220f8439201"
, "build_id_offset": 8112471
, "file_name": "/usr/lib/firefox/libxul.so"
}
, { "address": 21224698
, "build_id": "f700a30b862f44a2e401700e56bbb220f8439201"
, "build_id_offset": 8092922
, "file_name": "/usr/lib/firefox/libxul.so"
}
, { "address": 20545813
, "build_id": "f700a30b862f44a2e401700e56bbb220f8439201"
, "build_id_offset": 7414037
, "file_name": "/usr/lib/firefox/libxul.so"
}
, { "address": 20548315
, "build_id": "f700a30b862f44a2e401700e56bbb220f8439201"
, "build_id_offset": 7416539
, "file_name": "/usr/lib/firefox/libxul.so"
}
, { "address": 20556072
, "build_id": "f700a30b862f44a2e401700e56bbb220f8439201"
, "build_id_offset": 7424296
, "file_name": "/usr/lib/firefox/libxul.so"
}
, { "address": 20556437
, "build_id": "f700a30b862f44a2e401700e56bbb220f8439201"
, "build_id_offset": 7424661
, "file_name": "/usr/lib/firefox/libxul.so"
}
, { "address": 20420860
, "build_id": "f700a30b862f44a2e401700e56bbb220f8439201"
, "build_id_offset": 7289084
, "file_name": "/usr/lib/firefox/libxul.so"
}
, { "address": 20438677
, "build_id": "f700a30b862f44a2e401700e56bbb220f8439201"
, "build_id_offset": 7306901
, "file_name": "/usr/lib/firefox/libxul.so"
}
, { "address": 20438844
, "build_id": "f700a30b862f44a2e401700e56bbb220f8439201"
, "build_id_offset": 7307068
, "file_name": "/usr/lib/firefox/libxul.so"
}
, { "address": 20398093
, "build_id": "f700a30b862f44a2e401700e56bbb220f8439201"
, "build_id_offset": 7266317
, "file_name": "/usr/lib/firefox/libxul.so"
}
, { "address": 20398155
, "build_id": "f700a30b862f44a2e401700e56bbb220f8439201"
, "build_id_offset": 7266379
, "file_name": "/usr/lib/firefox/libxul.so"
}
, { "address": 122545719
, "build_id": "2545e7248eda14b0710dc922e135c466bf7e9bea"
, "build_id_offset": 255543
, "function_name": "g_main_context_dispatch"
, "file_name": "/lib/libglib-2.0.so.0"
}
, { "address": 122564160
, "build_id": "2545e7248eda14b0710dc922e135c466bf7e9bea"
, "build_id_offset": 273984
, "file_name": "/lib/libglib-2.0.so.0"
}
, { "address": 122564649
, "build_id": "2545e7248eda14b0710dc922e135c466bf7e9bea"
, "build_id_offset": 274473
, "function_name": "g_main_context_iteration"
, "file_name": "/lib/libglib-2.0.so.0"
}
, { "address": 20396724
, "build_id": "f700a30b862f44a2e401700e56bbb220f8439201"
, "build_id_offset": 7264948
, "file_name": "/usr/lib/firefox/libxul.so"
}
, { "address": 20396536
, "build_id": "f700a30b862f44a2e401700e56bbb220f8439201"
, "build_id_offset": 7264760
, "file_name": "/usr/lib/firefox/libxul.so"
}
, { "address": 20411064
, "build_id": "f700a30b862f44a2e401700e56bbb220f8439201"
, "build_id_offset": 7279288
, "file_name": "/usr/lib/firefox/libxul.so"
}
, { "address": 20411317
, "build_id": "f700a30b862f44a2e401700e56bbb220f8439201"
, "build_id_offset": 7279541
, "file_name": "/usr/lib/firefox/libxul.so"
}
, { "address": 45719583
, "build_id": "f700a30b862f44a2e401700e56bbb220f8439201"
, "build_id_offset": 32587807
, "function_name": "XRE_InitChildProcess"
, "file_name": "/usr/lib/firefox/libxul.so"
}
, { "address": 134529400
, "build_id": "c09940c04874728b3ca7c60c46a3557fd2146c34"
, "build_id_offset": 16760
, "function_name": "_start"
, "file_name": "/usr/lib/firefox/plugin-container"
}
, { "address": 134526924
, "build_id": "c09940c04874728b3ca7c60c46a3557fd2146c34"
, "build_id_offset": 14284
, "file_name": "/usr/lib/firefox/plugin-container"
} ]
}
, { "frames":
[ { "address": 7988260
, "build_id": "076c75a59c2434f1c708ae6c6b962fd46a27f01f"
, "build_id_offset": 1060
}
, { "address": 7103548
, "build_id": "a252f69109451ba0b0d02cb0e87ac789999dec33"
, "build_id_offset": 42044
, "function_name": "pthread_cond_wait@@GLIBC_2.3.2"
, "file_name": "/lib/libpthread.so.0"
}
, { "address": 102651069
, "build_id": "7e17a6b0483d2c10799f8fdb6f945360737f259b"
, "build_id_offset": 5797053
, "file_name": "/usr/lib/flash-plugin/libflashplayer.so"
}
, { "address": 100253753
, "build_id": "7e17a6b0483d2c10799f8fdb6f945360737f259b"
, "build_id_offset": 3399737
, "file_name": "/usr/lib/flash-plugin/libflashplayer.so"
}
, { "address": 102650490
, "build_id": "7e17a6b0483d2c10799f8fdb6f945360737f259b"
, "build_id_offset": 5796474
, "file_name": "/usr/lib/flash-plugin/libflashplayer.so"
}
, { "address": 102650539
, "build_id": "7e17a6b0483d2c10799f8fdb6f945360737f259b"
, "build_id_offset": 5796523
, "file_name": "/usr/lib/flash-plugin/libflashplayer.so"
}
, { "address": 102650637
, "build_id": "7e17a6b0483d2c10799f8fdb6f945360737f259b"
, "build_id_offset": 5796621
, "file_name": "/usr/lib/flash-plugin/libflashplayer.so"
}
, { "address": 7089097
, "build_id": "a252f69109451ba0b0d02cb0e87ac789999dec33"
, "build_id_offset": 27593
, "function_name": "start_thread"
, "file_name": "/lib/libpthread.so.0"
}
, { "address": 9826430
, "build_id": "122270907fb8ced72f3d89379d64f02426205042"
, "build_id_offset": 929918
, "function_name": "__clone"
, "file_name": "/lib/libc.so.6"
} ]
}
, { "frames":
[ { "address": 7988260
, "build_id": "076c75a59c2434f1c708ae6c6b962fd46a27f01f"
, "build_id_offset": 1060
}
, { "address": 7103548
, "build_id": "a252f69109451ba0b0d02cb0e87ac789999dec33"
, "build_id_offset": 42044
, "function_name": "pthread_cond_wait@@GLIBC_2.3.2"
, "file_name": "/lib/libpthread.so.0"
}
, { "address": 102651069
, "build_id": "7e17a6b0483d2c10799f8fdb6f945360737f259b"
, "build_id_offset": 5797053
, "file_name": "/usr/lib/flash-plugin/libflashplayer.so"
}
, { "address": 100253753
, "build_id": "7e17a6b0483d2c10799f8fdb6f945360737f259b"
, "build_id_offset": 3399737
, "file_name": "/usr/lib/flash-plugin/libflashplayer.so"
}
, { "address": 102650490
, "build_id": "7e17a6b0483d2c10799f8fdb6f945360737f259b"
, "build_id_offset": 5796474
, "file_name": "/usr/lib/flash-plugin/libflashplayer.so"
}
, { "address": 102650539
, "build_id": "7e17a6b0483d2c10799f8fdb6f945360737f259b"
, "build_id_offset": 5796523
, "file_name": "/usr/lib/flash-plugin/libflashplayer.so"
}
, { "address": 102650637
, "build_id": "7e17a6b0483d2c10799f8fdb6f945360737f259b"
, "build_id_offset": 5796621
, "file_name": "/usr/lib/flash-plugin/libflashplayer.so"
}
, { "address": 7089097
, "build_id": "a252f69109451ba0b0d02cb0e87ac789999dec33"
, "build_id_offset": 27593
, "function_name": "start_thread"
, "file_name": "/lib/libpthread.so.0"
}
, { "address": 9826430
, "build_id": "122270907fb8ced72f3d89379d64f02426205042"
, "build_id_offset": 929918
, "function_name": "__clone"
, "file_name": "/lib/libc.so.6"
} ]
}
, { "frames":
[ { "address": 7988260
, "build_id": "076c75a59c2434f1c708ae6c6b962fd46a27f01f"
, "build_id_offset": 1060
}
, { "address": 7103548
, "build_id": "a252f69109451ba0b0d02cb0e87ac789999dec33"
, "build_id_offset": 42044
, "function_name": "pthread_cond_wait@@GLIBC_2.3.2"
, "file_name": "/lib/libpthread.so.0"
}
, { "address": 102651069
, "build_id": "7e17a6b0483d2c10799f8fdb6f945360737f259b"
, "build_id_offset": 5797053
, "file_name": "/usr/lib/flash-plugin/libflashplayer.so"
}
, { "address": 100253753
, "build_id": "7e17a6b0483d2c10799f8fdb6f945360737f259b"
, "build_id_offset": 3399737
, "file_name": "/usr/lib/flash-plugin/libflashplayer.so"
}
, { "address": 102650490
, "build_id": "7e17a6b0483d2c10799f8fdb6f945360737f259b"
, "build_id_offset": 5796474
, "file_name": "/usr/lib/flash-plugin/libflashplayer.so"
}
, { "address": 102650539
, "build_id": "7e17a6b0483d2c10799f8fdb6f945360737f259b"
, "build_id_offset": 5796523
, "file_name": "/usr/lib/flash-plugin/libflashplayer.so"
}
, { "address": 102650637
, "build_id": "7e17a6b0483d2c10799f8fdb6f945360737f259b"
, "build_id_offset": 5796621
, "file_name": "/usr/lib/flash-plugin/libflashplayer.so"
}
, { "address": 7089097
, "build_id": "a252f69109451ba0b0d02cb0e87ac789999dec33"
, "build_id_offset": 27593
, "function_name": "start_thread"
, "file_name": "/lib/libpthread.so.0"
}
, { "address": 9826430
, "build_id": "122270907fb8ced72f3d89379d64f02426205042"
, "build_id_offset": 929918
, "function_name": "__clone"
, "file_name": "/lib/libc.so.6"
} ]
}
, { "frames":
[ { "address": 7988260
, "build_id": "076c75a59c2434f1c708ae6c6b962fd46a27f01f"
, "build_id_offset": 1060
}
, { "address": 7103548
, "build_id": "a252f69109451ba0b0d02cb0e87ac789999dec33"
, "build_id_offset": 42044
, "function_name": "pthread_cond_wait@@GLIBC_2.3.2"
, "file_name": "/lib/libpthread.so.0"
}
, { "address": 102651069
, "build_id": "7e17a6b0483d2c10799f8fdb6f945360737f259b"
, "build_id_offset": 5797053
, "file_name": "/usr/lib/flash-plugin/libflashplayer.so"
}
, { "address": 100253753
, "build_id": "7e17a6b0483d2c10799f8fdb6f945360737f259b"
, "build_id_offset": 3399737
, "file_name": "/usr/lib/flash-plugin/libflashplayer.so"
}
, { "address": 102650490
, "build_id": "7e17a6b0483d2c10799f8fdb6f945360737f259b"
, "build_id_offset": 5796474
, "file_name": "/usr/lib/flash-plugin/libflashplayer.so"
}
, { "address": 102650539
, "build_id": "7e17a6b0483d2c10799f8fdb6f945360737f259b"
, "build_id_offset": 5796523
, "file_name": "/usr/lib/flash-plugin/libflashplayer.so"
}
, { "address": 102650637
, "build_id": "7e17a6b0483d2c10799f8fdb6f945360737f259b"
, "build_id_offset": 5796621
, "file_name": "/usr/lib/flash-plugin/libflashplayer.so"
}
, { "address": 7089097
, "build_id": "a252f69109451ba0b0d02cb0e87ac789999dec33"
, "build_id_offset": 27593
, "function_name": "start_thread"
, "file_name": "/lib/libpthread.so.0"
}
, { "address": 9826430
, "build_id": "122270907fb8ced72f3d89379d64f02426205042"
, "build_id_offset": 929918
, "function_name": "__clone"
, "file_name": "/lib/libc.so.6"
} ]
}
, { "frames":
[ { "address": 7988260
, "build_id": "076c75a59c2434f1c708ae6c6b962fd46a27f01f"
, "build_id_offset": 1060
}
, { "address": 9811111
, "build_id": "122270907fb8ced72f3d89379d64f02426205042"
, "build_id_offset": 914599
, "function_name": "syscall"
, "file_name": "/lib/libc.so.6"
}
, { "address": 20463066
, "build_id": "f700a30b862f44a2e401700e56bbb220f8439201"
, "build_id_offset": 7331290
, "function_name": "epoll_wait"
, "file_name": "/usr/lib/firefox/libxul.so"
}
, { "address": 20462511
, "build_id": "f700a30b862f44a2e401700e56bbb220f8439201"
, "build_id_offset": 7330735
, "file_name": "/usr/lib/firefox/libxul.so"
}
, { "address": 20476042
, "build_id": "f700a30b862f44a2e401700e56bbb220f8439201"
, "build_id_offset": 7344266
, "file_name": "/usr/lib/firefox/libxul.so"
}
, { "address": 20404268
, "build_id": "f700a30b862f44a2e401700e56bbb220f8439201"
, "build_id_offset": 7272492
, "file_name": "/usr/lib/firefox/libxul.so"
}
, { "address": 20411064
, "build_id": "f700a30b862f44a2e401700e56bbb220f8439201"
, "build_id_offset": 7279288
, "file_name": "/usr/lib/firefox/libxul.so"
}
, { "address": 20411317
, "build_id": "f700a30b862f44a2e401700e56bbb220f8439201"
, "build_id_offset": 7279541
, "file_name": "/usr/lib/firefox/libxul.so"
}
, { "address": 20446797
, "build_id": "f700a30b862f44a2e401700e56bbb220f8439201"
, "build_id_offset": 7315021
, "file_name": "/usr/lib/firefox/libxul.so"
}
, { "address": 20439091
, "build_id": "f700a30b862f44a2e401700e56bbb220f8439201"
, "build_id_offset": 7307315
, "file_name": "/usr/lib/firefox/libxul.so"
}
, { "address": 7089097
, "build_id": "a252f69109451ba0b0d02cb0e87ac789999dec33"
, "build_id_offset": 27593
, "function_name": "start_thread"
, "file_name": "/lib/libpthread.so.0"
}
, { "address": 9826430
, "build_id": "122270907fb8ced72f3d89379d64f02426205042"
, "build_id_offset": 929918
, "function_name": "__clone"
, "file_name": "/lib/libc.so.6"
} ]
} ]
}
Copy link to clipboard
Copied
In general, if the process is crashing in plugin-container, it's Mozilla's bug. If it's crashing in libflashplayer.so, it's ours. It looks like we're towards the top of the stack, but the data you provided doesn't have the actual symbols.
Also, it's worth noting that Flash Player and the browser are interdependent. The current Firefox ESR is Firefox 60. It might be worth checking to see if the crash you're seeing also occurs there.
A link to the crash report from Firefox (type about:crashes in the address bar) would be way more helpful. Just click on the most recent couple crashes to submit the report to Mozilla, then paste the resulting links here. I'd be happy to take a look.
Copy link to clipboard
Copied
Thanks. I did verify the crash also occurs with Firefox 60 ESR from Red Hat. I just initiated another plugin-container crash, but about:crashes just says "url isn't valid".
Copy link to clipboard
Copied
Please see the directions here for manually grabbing the crash dumps for Firefox on Linux:
Copy link to clipboard
Copied
Thanks. In this instance the user doesn't see the "Flash Plugin has crashed" screen. It is transparent to the user apparently. All I notice in the lab is increased disk activity while abrt captures the crash data. .mozilla/firefox for the user doesn't appear to have any crash data either.
Copy link to clipboard
Copied
Red Hat would like to get the debug symbols so they can provide a full backtrace:
Most recent comment: On 2018-07-24 13:02:52, Wright, Joe commented:
"Greetings,
Could you revert to Adobe and ask them for their debug symbols for flash-plugin-28.0.0.161-1 i686? I can get them a full backtrace if they can get me the debug symbols.
$ grep flash installed-rpms
flash-plugin-28.0.0.161-1.el6_9.i686
Copy link to clipboard
Copied
Hi,
We'd be happy to talk to RedHat about providing symbols. If we have an NDA in place between the companies, it should be a straightforward conversation.
The engineer at RedHat can send me a private message (just click my username and then send message) with contact information, and I'd be happy to get a conversation started with the appropriate folks.
Thanks!
Copy link to clipboard
Copied
More feedback from Red Hat. We need to get you two talking to each other somehow. If you don't want to post direct contact info on the forum, maybe you can email it to me? I don't have direct info for either party.
Hi, My name is Alan Matsuoka and I'm a contributor to your case. From what I can tell flash is crashing while in the middle of shutting down. It's in the middle of shutting down a connection to a dynamically loaded shared library and it's being given an invalid handle to a dynamically loaded shared lib. Firefox is calling NPP_Destroy which appears to be the destructor method for a plugin instance. Is the application closing a flash window at the time or is the crash occurring while the application is running ? Regards, Alan Matsuoka Senior Software Maintenance Engineer, Red Hat Global Support Services - North America
Greetings, We are getting in touch with our legal department in this regard. A NDA will take a substantial amount of time to complete, meaning that SLA is effectively out the window. That being said, it would be substantially easier for them to pull our debug libraries (open source, do not need any NDA or legal forms otherwise) as well as the application core, and get the backtrace on their end. Did Adobe have any qualms or issues with taking the application core and performing the analysis on their end? Additionally, could you forward Alan's summary of our findings to them? Thanks and have a great day! Best Regards, Joe Wright, RHCE RHCVA Senior Technical Support Engineer Customer Experience & Engagement - North America Red Hat, Inc
Greetings, Also, could you get their direct line of communication for me to forward to our legal team for any NDA arrangement related communication? Thanks and have a great day! Best Regards, Joe Wright, RHCE RHCVA Senior Technical Support Engineer Customer Experience & Engagement - North America Red Hat, Inc
Copy link to clipboard
Copied
Redhat reached out to me, but the gist is that you need to provide me the core directly, because the two companies can't freely share your data without legal agreements in place that don't currently exist (and would be prohibitively expensive and time-consuming to set up).
They did not provide me instructions on how to obtain those cores, and because I don't have access to an RHEL 6.9 workstation image (I've asked both RedHat and Adobe's IT support folks for access at this point), I can't just reproduce it and give you distro-specific instructions.
If RedHat can tell you where to find the appropriate core dump, you should be able to just post it (Adobe Send, Google Drive, Dropbox, etc. are all good options) and we'd be happy to take a look.
Thanks!
Copy link to clipboard
Copied
Do you also see this problem in the latest Flash Player version? As of today, it's Flash Player 30.0.0.154. A core dump from a current product version would make life much easier for the developer that gets to fix the bug.
Copy link to clipboard
Copied
Still see it happening with latest, on a test system, going the the flash "about" page:
[root@st9963_1 ~]# rpm -q flash-plugin
flash-plugin-30.0.0.154-1.el6_10.i686
[root@st9963_1 ~]# date
Mon Aug 20 16:27:35 EDT 2018
Aug 20 16:27:43 st9963 abrt[24637]: Saved core dump of pid 24141 (/usr/lib/firefox/plugin-container) to /var/spool/abrt/ccpp-2018-08-20-16:27:42-24141 (47656960 bytes)
Aug 20 16:27:43 st9963 abrtd: Directory 'ccpp-2018-08-20-16:27:42-24141' creation detected