• Global community
    • Language:
      • Deutsch
      • English
      • Español
      • Français
      • Português
  • 日本語コミュニティ
    Dedicated community for Japanese speakers
  • 한국 커뮤니티
    Dedicated community for Korean speakers
Exit
Locked
0

Adobe 23 & 24: Out of memory errors and performance issues

Guest
Jan 17, 2017 Jan 17, 2017

Copy link to clipboard

Copied

I found a similar topic here: Flash player 23 & 24 memory leaks and debated whether or not to reply to it, but decided to create a new post instead. Running Windows 7 x64, Opera browser 42.0.2393.94 (latest as of this post) with no extensions or other modifications.

I run four browser windows on a second monitor that each stream video using Flash. Ever since Flash player 23.0.0.162 was first released on 9/13/2016 (according to Archived Adobe Flash Player versions ) I started having problems. After about 15-20 minutes the video would start to become extremely choppy and the browser windows would become sluggish and slow to respond. The choppiness was not due to any network connectivity problems, it was the browser struggling to play the video. After the 20 minute mark there was a near 100% chance that Flash would crash with an out of memory error.

I have used the uninstall utility and went back to Flash player 22.0.0.209 to test and the videos run like proverbial butter. Several hours later and not a single problem. Of course, this is a huge security risk so I did not stay on Flash 22, however I need to solve this issue ASAP.

The only luck I had in resolving this was when I opened opera://flags and set Hardware-accelerated video decode to disable. When I did this, it seemed like everything was back to normal, but it only lasted for half a day. Everything was right back to poor performance and out of memory errors. I tried clearing all browsing data from Opera, uninstalled Flash and reinstalled, reenabled Hardware-accelerated video decode and then disabled it again. Nothing I did got that small moment of normal performance back.

tl;dr: Flash 23 & 24 have poor performance and crash with memory errors after only 15-20 minutes, but Flash 22 does not and runs perfectly fine.

I need this resolved as quickly as possible, please.

Thanks

Views

2.4K

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines

correct answers 1 Correct answer

Deleted User
Mar 20, 2017 Mar 20, 2017

Apologies for not getting back to you.

I just wanted to let you know that the most recent update to Flash fixed the issue. Just to be clear, Flash 24.0.0.194 was still having the same problems outlined in this thread, but Flash 24.0.0.221 resolved the issues completely. For a few weeks I have been running the same four windows with a Flash stream in each one with no problems after several hours. No crashes, unresponsiveness, playback choppiness, or any other kind of performance issues.

Votes

Translate

Translate
Adobe Employee ,
Jan 17, 2017 Jan 17, 2017

Copy link to clipboard

Copied

Flash Player is effectively a language runtime.  We process and run code written by developers.  Some of that code is for the actual content, and some of it is for advertising and analytics.

We also process code that's written by malicious actors, and we constantly harden Flash Player to prevent those actors from being able to use Flash Player as a vector for malware.  Any time an application does more processing, it consumes more CPU cycles.  While we design mitigations with efficiency in mind, they're never free, and that cost accretes over time.

There's no magic "make it faster" button that we can push to make things suddenly work faster.  It's very unlikely that we've injected an egregious performance problem at the language level.  It's much more likely that you're running across a specific, pathological piece of content that is either doing something very inefficient that was borderline passable before, but is now not, or there are a very specific set of circumstances that are in play that are slower.

That said, it's impossible to have an actionable conversation about performance without discussing the specific content in question.  In your scenario, you're talking about four concurrent windows loading arbitrary content, any of which might have one or more ads that also rotate.  Because advertising is highly targeted, it's quite possible that the problem is specific to an ad that you'll see, but we won't.

While we're always interested in performance issues, we have to get specific enough that it's actionable.  We've engaged in exercises in the past where we've identified slow code through high-end optimization tools, only to find that there's actually no discernable difference in performance in real-world content.  It's not enough to just optimize stuff (and in a 2.5 million line codebase, there's a lot of "stuff") -- you have to optimize the right stuff, and do it in a way that it still works across all of the supported platforms and millions of pieces of existing content.

The real productive discussions about performance happen between us and developers.  In general, the performance problems usually come down to poorly written content (ads, and ad-insertion code by smaller ad networks are frequent offenders), and we provide performance profiling tools that allow developers to identify and fix slow code.  The productive discussions about performance are centered around APIs... calling a particular API is 30ms slower per call, stuff like that.  In order to make it actionable, we have to get down to the discrete, atomic thing that's slow.  The notion that random is slow on the Internet isn't going to lead to a fix.

It's also possible that it's not actually something slow in the code.  It could just be that something takes a little bit more RAM, but it was enough that it exhausts the physical RAM on the machine, or that we've changed the layout of how memory is used on the system (this is one of the many strategies use us against malware authors), and memory fragmentation leads to memory exhaustion because some portion of physical RAM isn't writable in a useful way.  (In either scenario, throwing more RAM at the problem would help)

So, the goal, if you want to actually get down to actionable problem, is to strip it down to something specific that can be looked at.  It's annoying, but this is engineering, and precision is central to getting anything done.

I'd start by ruling some stuff out:

- Does using an ad-blocker make the problem go away?  If yes, it's probably an ad, or the activity generated by all of the content and ads you're viewing in aggregate is enough to push the machine over.

- Where's the resource contention?  When the machine begins to perform poorly, what does the CPU usage look like?  What does memory usage look like?  Is there high disk activity (indicating that the machine is out of physical RAM, and swapping lots of memory to disk -- which is literally ~100k times slower than operating in physical RAM)

- Once you know what kind of resource contention you're looking for, when running the content that you care about in isolation (a single tab), can you identify a piece of content that shows a large delta between Flash Player 22 and Flash Player 24?

- If so, then we at least know what piece of content to look at to begin understanding *why* the usage is higher in that scenario, and to determine whether or not it's intentional, and/or could be improved without weakening security mitigations or introducing side-effects.

- Do you have the same problem running the same content in other browsers?  Does the same 4-tab configuration in Chrome, Firefox or IE have the same problem?  They're very different pieces of software.  That would tell us whether it's in common code, or if it's related to some platform-specific code.

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Guest
Jan 17, 2017 Jan 17, 2017

Copy link to clipboard

Copied

I'll take your questions one at a time and expand upon my description of the problem as it seems I was unclear in some points.

Yes, ad blocker makes it go away. I already knew that this was an issue with the ads in rotation and not the video stream. But again, since Flash 22 has no issues then I have to assume there is a fault somewhere in Flash 23/24. Let me clarify a point here. When Flash 23 was first released in September, I updated and after two weeks I was forced to drop back to Flash 22 because of the horrible performance. I ran Flash 22 until Flash 24 was released and had no issues over those months and only updated because at that point I was behind by a few updates and the security risk was greater than the performance problems so I finally updated. So to assume that I just "got lucky" with the content rotation while using Flash 22 and just got "unlucky" immediately after updating is pretty much out of the equation. Flash 22 simply performs better than 23 and 24 with this particular content. It very much feels like a memory leak to me, but since the problem starts almost immediately in 23/24 then it is almost certainly not a memory leak. It just has that same feel about it if that makes sense.

Both RAM and CPU usage increase over time, however RAM usage increases far more steeply than CPU usage does. CPU usage seems to plateau after a while while RAM usage continues to increase until a crash occurs. I don't have hard numbers for you at this time, but I will set up some resource monitors and I can provide that data at a later time.

I have not tried this particular exercise in Chrome or in Firefox because both browsers are so resource inefficient that they have serious performance issues running multiple Flash instances concurrently already. Any test in them would probably be worthless because regardless of the version of Flash player I'm using and regardless of how well Opera is currently performing, Firefox and Chrome both perform very badly running the four windows. That's is why I chose Opera for this in the first place. I also tested Vivaldi as it's supposed to be even lighter than Opera, but it suffered from performance issues just like Firefox and Chrome did (even with Flash player 22) so I stuck with Opera.

Another thing I should mention before it's brought up is that I am not using any sort of prebuilt computer cluttered with preloaded bloatware. Everything that runs on my system is something I personally installed and moreover, I do not have very many applications or services set to run at startup. I take care to manage what runs and when it runs. Something else that I neglected to mention in my original post is that my machine has 8GB of memory. As you postulate, throwing more physical RAM at it may help, but I'm not sure more than 8 GB is really necessary for a few browser windows running Flash player. I'm nearly certain that the "out of memory" crash always occurs before the capacity of physical RAM is hit anyway. Again, I'll be monitoring that to know for sure. Virtual memory is set to be system managed and is currently 8GB as well, though I have not noticed a lot of disk activity when running the video streams and nothing else so I don't believe there's a lot of memory swapping happening.

One last thing. For the record, I am a hobbiest coder (not Flash, though) and I've got two degrees in Network Engineering as well as nearly two decades of experience as a service tech so you can be a little more technical with me and skip some of the extra clarification if need be.

Thanks again. I'll get some resource monitors going and get you the data ASAP.

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Adobe Employee ,
Jan 18, 2017 Jan 18, 2017

Copy link to clipboard

Copied

Cool, thanks.  It's a really wide range of skill and background here, so I default to over-explaining the basics.

Based on what you're saying, my guess is that we're using more memory, and that's pushing the machine into swap.

ActionScript is a garbage-collected language,  but content developers have to actually release objects so that they can be GCed.  One of the really problematic issues is that video sites in particular will allocate NetStream objects for inserted commercials, but they never free them.  A couple major TV networks in the US are offenders.  Basically, the video players they've written (Flash is a language runtime like Java, not a media player per se) just accrete video objects until we either bail or the memory allocation fails in an unrecoverable way and we crash.  We do a lot of memory management stuff like forced GC as we approach the upper memory limit (we use a heuristic that makes a portion of the available memory available, and on a 32-bit browser, the 2GB per-process limit applies), but if there's nothing that can be freed, we're stuck.

I've been pushing to add a hard limit to the number of NetStream objects that can be simultaneously instantiated, but ultimately, it's such an ugly problem, that the cure is kind of worse than the disease.  If you're on a 64-bit browser and you have a lot of RAM, you can probably binge-watch one of those offending providers for a few hours and never notice the problem, but if we imposed a reasonable hard limit, then *everyone* would run into it.

This kind of problem exists for other types of objects as well, and if you've got say, an ad-loader shim that will sit there and load in a stead stream of new ads on a page that's otherwise static, it's probably doing this same kind of unbounded allocation.

One of the things that we've been doing on the security front is partitioning memory allocation for different types of objects, so that objects that have traditionally been used (or could potentially be used) to overflow into control structures in memory are now physically isolated, so that they can only really corrupt other data objects in uninteresting ways to an attacker.  This has some overhead associated with it, but it also probably makes it more likely that you're going to encounter problems with memory fragmentation.

I look at pretty much every check-in and bug that comes into the product, and nothing is really jumping out at me in recent history, beyond some of the security mitigation work that we're not going to roll back.


Again, just from a differential testing perspective, understanding whether or not this is a problem specific to the NPAPI/PPAPI player in the context of Opera (it supports both last time I checked, and I'm not sure which one you're using) and the other available plug-in architectures would help me narrow down the range of possibilities.  We're talking about 2.5 million lines of code and ~1000+ changes/month, so the more we can narrow the field, the better.

That said, I'm just not seeing an egregious, widespread problem in the aggregate data.  We do something like 2 billion monthly installs, and we're under heavy scrutiny from folks like the Windows performance team, in addition to our own internal testing.


One of the things you might want to do, is just look in the network tab of your browser's developer tools to see what's getting loaded over time on some of those pages.  I looked into a similar issue recently, and there was an ad script that loaded something like 1500 assets in ~30 seconds, although the majority of them weren't visible (ad-stuffing, etc.).  My guess is that you'll see a correlation between memory growth and what's getting loaded via the network that will support the claim about objects not being freed for GC.

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Guest
Jan 18, 2017 Jan 18, 2017

Copy link to clipboard

Copied

For the record, I've got the PPAPI version of Flash installed in Opera.

FzzyMYI.gif

Im0rSp3.gif

7MvXTU5.gif

Here is 30 minutes of data recorded with four windows open. I can probably run this again with a few less counters as the disk related ones didn't really seem to be out of line. The few things to note is the CPU idle time. It averages around 60% with a typical range of 50% to 70% meaning the CPU is idle at least half the time. The second is to note the red horizontal RAM line. At the start of the test I'm slightly over 6GB available and it progressively decreases until only 1GB physical RAM is free. At this point, one of the four windows had crashed with an out of memory error. The final counter of note is the green one indicating hard faults, that is memory swaps to disk. Almost as soon as I began the test this line maxed out and stayed maxed. One thing I should also mention that I have not thought of until this point is that my swap file is located on an SSD so swap performance shouldn't really be as large an issue.

I've also read everything you said and I don't have any problems with what you said except for the one thing I've been saying all along. If Flash player 22 works fine and the problem is introduced with Flash 23 then that indicates that something about Flash player 23 and onward has changed to my detriment and all indications are that it isn't a hardware limitation on my end.

Another thing I will concede is that some of the ads that are served in the rotation have been known to be what I've termed as "toxic" meaning that they are either poorly coded or contain some other bug as to introduce memory leaks and performance issues. In the past I've identified the offending ads and used a custom blacklist in AdBlock to block just those ads (while not being subscribed to any other block list). I cannot block all ads since then my service would not function. The ads are necessary so I take great care to only block those ads that have proven to be harmful. But I will emphasis again, that I do not currently experience any problems with Flash 22 installed even with the same ad rotation that I am getting with Flash 23/24.  If I could fully trust the ad services that are being used then I would simply go with Flash 22 full time until whatever this problem is is resolved, but the fact is that even the most trustworthy ad server will sometimes miss a maliciously coded Flash ad and allow it to be served.

At this point I'm thinking about just running Opera sandboxed with Flash 22.

Please let me know if you'd like any additional info. Thanks again.

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Adobe Employee ,
Jan 18, 2017 Jan 18, 2017

Copy link to clipboard

Copied

Yeah, this all fits with my intuition.

The thing that would be really helpful are just discrete URLs that reproduce the problem.  If you can narrow it down to one URL that produces a similar memory usage curve on its own, that's probably the culprit.  There might be more than one, but I'm guessing that one of them will be an outlier (but maybe not, depending on the root-cause).

At this point, I really just need to see if I can reproduce it and on what platforms, then go back to 22 and confirm that memory growth is bounded, and then just bisect builds until I figure out what change is responsible.

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Guest
Mar 20, 2017 Mar 20, 2017

Copy link to clipboard

Copied

Apologies for not getting back to you.

I just wanted to let you know that the most recent update to Flash fixed the issue. Just to be clear, Flash 24.0.0.194 was still having the same problems outlined in this thread, but Flash 24.0.0.221 resolved the issues completely. For a few weeks I have been running the same four windows with a Flash stream in each one with no problems after several hours. No crashes, unresponsiveness, playback choppiness, or any other kind of performance issues.

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Adobe Employee ,
Mar 20, 2017 Mar 20, 2017

Copy link to clipboard

Copied

That's great news.  It looks like a number of bug-fixes related to video services landed in this range, which makes a lot of sense.

Thanks for the update!

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Guest
May 13, 2017 May 13, 2017

Copy link to clipboard

Copied

At the risk of being a thread necromancer, I wanted to let you know that this issue has resurfaced.

Both 25.0.0.148 and 25.0.0.163 are affected. I have uninstalled the PPAPI version of Flash and installed 25.0.0.127 and things work normally again. Since Chrome has it's own version of Flash and Firefox uses the NPAPI version, I feel confident in leaving things this way for now. Opera is only used for the purposes described above and nothing else and I'm running it sandboxed just in case.

Hopefully  you can have a look at what changed between 127 and 148 and compare that to what was changed in the versions I mentioned originally and find a commonality that will lead you to a proper fix and prevent this from happening again.

Cheers.

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Adobe Employee ,
May 15, 2017 May 15, 2017

Copy link to clipboard

Copied

LATEST

Thanks.  I don't see a smoking gun, but there are some video-adjacent changes.


Can you give me a discrete set of steps that demonstrate the problem for you?  A link to a specific video that shows the problem would be really helpful.  I'm going to have to have someone find a machine that consistently reproduces it.

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines