CF2016 on RHEL7 - java.lang.OutOfMemoryError: Compressed class space

[Nalina5EE7]

In our setup using JDK8 -  requests to different pages in CF application  returns this error:

(randomly occurs  any requests for the applciation - logon page of the application is also affected by this issue quite often )

java.lang.OutOfMemoryError: Compressed class space  - no stack trace available.

these pages coming back blank or failing for a period

JVM parameters are set with default settings - JVM settings related to compressed class or metaspace are NOT set .

CF reactor log viewer shows 30% of compressed class space is used at any time of the default 1G assigned.

JVM heap size is available on the system. It doesn't appear to be system related memory issue (top output)

similar issue reported with maxmetaspace here:

https://tracker.adobe.com/#/view/CF-4207269

any suggestion to track the cause or if there known bugs for Compressed class space?

TIA,

NS

[BKBK]

The root cause is clear. The space that your application needs for UseCompressedClassPointers exceeds the available CompressedClassSpaceSize. The default value for the size is 1GB. 

If you're certain that your application doesn't use that many files, then the error suggests that something else in your application is creating a lot of classes and never releasing them. Think of a classloader, for example.

This StackOverflow post on compressed class space contains the following helpful suggestions:

1. If your application can manage without so many files, then disabling compressed class pointers is perhaps an option. The relevant JVM flag is -XX:-UseCompressedClassPointers. 
2. If your application does indeed need that many files, then you could consider increasing CompressedClassSpaceSize to 2GB. The relevant flag is -XX:CompressedClassSpaceSize=2g.
3. In any case, you should enable the following JVM settings to troubleshoot the classes loaded:
   -XX:+PrintGCDetails -XX:+TraceClassUnloading -XX:+TraceClassLoading

[Nalina5EE7]

Thanks for the pointers shared so instantly on the issue I reported. greatly helpful BKBK as well Charlie_Arehart great learning from the information shared.

As a temporary relief CF instance is set to restart periodically as it is found helpful to stop Compressed class space error. Restart of CF instance leaves more free memory on the system.

[Charlie Arehart]

Nalina, I want to add to what BKBK has, and focus first on a different point, which is what you are "seeing" in the graphs, and why it may seem confusing ("never above 30%"). Then I want to address the relationship between this space and the heap that you mention, and finally I have a proposal of yet another diagnostic step you may want to consider.

1) You observed that "CF reactor log viewer shows 30% of compressed class space is used at any time". I assume you mean instead to FusionReactor, and its logs (and "log viewer), which track each memory space (CF has no reactor log, and no logs tracking each memory space), and further that you are viewing the log in the FR "Archived Metrics" viewer, which is great.

2) But I want to share a warning about your observation that it (the "log viewer") "shows 30% of compressed class space is used at any time". Your (reasonable) conclusion seems to be that therefore, "the compressed class space never went above 30%, so how did it 'run out of memory'"? There are a couple of explanations for that, which are important to understand.

a) First, note that if you are indeed viewing the last log graphed before CF went down/you restarted it, there is a real possibility that CF could have become so unstable that it could not write its memory log files every 5 seconds (as it does, otherwise). In that case, you can't trust what you are seeing in the graphs, as I will explain.

So if you use the "view" menu on the top right of that page, and then look at the "spreadsheet" or "text" view, to see the real log lines, you may see that the time reflected in the log (in the first two cols on each line) is not only NOT every 5 seconds (before things started to go bad, compared to when "all was well"), but you may well see that it just STOPS logging entirely, perhaps minutes before you know that CF was brought down. That would confirm my suspicion, and would not be uncommon.

b) And both points can lead to the graph not being reliable. First, it may be missing datapoints (and the graphs may make inferences/smoothing that hides the gaps), and worse you may see the last point on the graph showing that 30%, but it could be from minutes before CF went down/you restarted it. To that last point, you can at least look at the times on the bottom of the graphs. You may notice that the intervals show 3/4 of the graph covering perhaps 30 mins, and the last 1/4 of the graph covering only 15 mins. That would be wrong, of course.

3) So with all that said, the OOM error you reported confirms that you DID run out of that memory space. And BKBK has pointed to the classic responses to consider. 

4) You mention that "JVM heap size is available on the system", by which I assume you mean that according to the FR graphs, the heap usage was not high. That may be so.

One thing to note is that the compressed class space does not come FROM (does not live WITHIN) the heap. It is separate.

I can say that with 100% certainty, because on a machine where I have only 800mb heap size (xmx), I can see in the FR graph of the compressedclassspace size that it's the 1000mb (1gb) that you both mention is the default if one does not change that size. (And let that be a lesson to those who look at the OS memory space, and wonder how CF's use of memory can be higher than the heap size. This space is one example, and the metaspace is another, and so on. Of course, it's one thing for such a memory space to have such a "max" value and for that space to be actually "used" or "allocated", and FR tracks that. The latter is what would contribute to memory space for the process in the OS.)

5) So bottom line, you do need to find out why you are using so much compressedclassspace.

6) BTW, you don't mention what version of FR you have. There's always the possibility that it (or some feature in it) is a contributor to this problem. The latest version is 8.5. What version are you on? And what CF version and update? and what JVM version?

7) Given that you are using FR, if the problem repeats often (frustrating as that is), you could try removing FR temporarily, to see if the problem went away. If it did, you would want to report this to the FR folks, who would jump on it.

And when I say remove it temporarily, I don't mean "uinstall it", or even remove it from CF using the FRAM Instance Manager (if you know what that means). Doing either would lose all the FR logs for the instance, which go back 30 days by default. Instead, I simply mean (carefully) comment out the two FR-related args in the CF jvm.config, and then restart CF. That will cause FR to be "removed" from CF, until you put those args back.

Let us know how things go.

[Nalina5EE7]

Inspite of this OOM Compressed Class Space error, CF server process didn't stop, no restart was required to recover from this error quite conflicting  symptom. It seems to be self recovering from the situation and no other interruption to service noted other than those few pages (random) returning blank or error to few users which seems this error in the CF logs  is red herring. From server end no manual action was needed to correct it when user reports the problem. Is that normal in CF for this scenario?

Thanks for detailed responses. I will review these to my setup.

[Charlie Arehart]

Ah, sorry for my mistakenly assuming it led to an outage/restart. I see now you did say it was just a few pages. I just didn't get that CF stayed up. I understand now.

So there are two ways to look at this.

Personally, I would (like BKBK) question when such a large memory space is used up unexpectedly. To your last question, no, this is not a "normal" error. I very (very) rarely see it--and all I do all day is troubleshoot CF servers for folks, whether on a consulting basis or here, so it really is not "normal". 🙂

But if you DO have ample available memory on the machine (not in the heap), then you could certainly raised the size of the compressedclassspace. In some situations, that can seem more expedient than troubleshooting things. If it works, great. Sometimes, a given space only needs to be a little bigger to never complain again. 

And you have FR which you can use to watch the space, to see how things go. And I'll add that FR Cloud (the part of FR that runs in the cloud, using data from your on-premise CF and FR) not only has alerts like the on-premise FR version does, but it has FAR more things it can "check", including the compressedclassspace use. 🙂 So you could even know in advance if it started rising toward your new limit.

All that said, I'd still think it useful to understand if the FR logs you DO have showed whether the space WAS rising high enough to cause that error. It would seem they should, unless there was one of the problems I described. Then again, the FR Cloud alerts would also be watching these things every 5 seconds, and could see it when you may not--or can be told to alert you when FR Cloud is perhaps getting NO data from CF.

Hope that helps.

[Nalina5EE7]

Thank you so much for the details to understand the issue and the options given to diagnose the issue.

yes FR is older v6  - arvhive viewer shows compressed class space usage (peak) has never gone beyond 0.31G  so may be these reports  are  not reliable as you have pointed.

 CF Version: 2016,0,03,301771 with jdk1.8.0_192

memory and cpu usage of this process is pretty consistent throughout the day while monitoring the process stats via top output.

system memory is about 0.8 g free at anytime. guess need to add in memory to the system to be able to increase the settings of Compressedclass space. 

[BKBK]

> CF Version: 2016,0,03,301771 with jdk1.8.0_192

You seem to have a relatively old build of ColdFusion 2016. Namely Update 3. ColdFusion's updates solve bugs cumulatively. Therefore, the following steps might just solve your problems:

1) Apply the latest update of ColdFusion 2016, that is, Update 16.

    Open the ColdFusion Administrator and click on the (i) button in the top right-hand corner.

    Read off the Java version. If it is older than 11.0.8, then proceed to step 2.

2. Install JAVA SE 11.0.8(LTS). 

    Configure ColdFusion 2016 (in \cfusion\bin\jvm.config) to run on Java 11.0.8

If you have any questions, you will know where to ask. 😉

[Charlie Arehart]

To BKBK's comment, there is a bit more you need to know, Nalina.

First, if you do update from your cf2016 update 3 to anything above update 11,you will need to do update 11 first if you will be using the cf admin to do the updates. THEN you can do the latest, now update 16. This is indicated in the uodate technotes, but few read them first.

Second, once you go past uodate 14, you MUST use the wsconfig tool to UPGRADE any web server connectors you have (for Apache in your case on Linux, or for IIS or Apache for those on Windows). Any existing web sites using those will break until you do this. This is also mentioned in the technotes after update 14, with more detail in the one for update 14.

These things apply also to cf2018 updates 4 and 8, respectively. 

Finally, as for updating Java, BKBK note that Nalina reported being on 1.8.0_191. As such, she could update that to 261 or, as you say, she could update to 11.0.8 (the current latest updates for those Java versuons) . Either is supported by cf2016 or 2018, and both are offered at the link BKBK shared. 

Hope that's helpful. 

[BKBK]

Oh, yes: ColdFusion 2016 Update 16 notes 

Java note: there are improvements in Java 11 over Java 8, some of them significant. That is why the Adobe Team decided to upgrade ColdFusion 2016's Java varsion from 8 to 11.

[Charlie Arehart]

Sure, but let's not overstate it. Adobe still supports Java 8 for CF2016. I don't think it's so much that they added support for Java 11 because of those particular advantages. (Indeed, I don't think any of those listed thre benefit CF out of the box, but I could be wrong.)

Anyway, I'm not trying to start a cat fight. I'm just presenting the simple fact that updating to Java 11 in this case is not necessarily a solution to the problem. It may be worth trying, but changing the JVM has other risks people need to be careful about. It can be done, easily when you know what you're doing. And plenty of people do update CF2016 from using Java 8 to Java 11. Again, it just may not be NECESSARY in this case.

Indeed, one could argue that just updating Java 8 from the current 191 version to 261 may be enough to ensure there are at least no JVM bugs at play here (as Oracle puts such bug fixes into both 8 and 11). And changing the JVM version WITHIN a given release is generally easier than jumping from one to another, though even that too has its risks (such as if one has imported any certs into the old JVM).

But perhaps we should wait and hear from Nalina before offering any more things to consider. 🙂

[BKBK]

> Sure, but let's not overstate it. Adobe still supports Java 8 for CF2016. I don't think it's so much that they added support for Java 11 because of those particular advantages. (Indeed, I don't think any of those listed thre benefit CF out of the box, but I could be wrong.)

I can understand you doubting whether Adobe's ColdFusion Team added support for Java 11 because of the advantages listed. Only the Adobe Team can respond to that.

In any case, whether it was the team's intention or not, the move from Java 8 to Java 11 does have benefits for ColdFusion 2016 out of the box. Here are just some:

• improved garbage collection, notably the use of G1GC for heap sizes bigger than 4 GB;
• more efficient internal representation of strings. The data in most applications, including ColdFusion itself,  consists by far of string data. Java 11 changes the internal representation of a string, greatly reducing the amount of space required for string storage;
• changes in the compiler and in core libraries (for example, in Collections, Threads and java.net) that improve overall performance. 

[Charlie Arehart]

I appreciate that as always, you're just trying to help, pointing out for people any of many possibilities that may help in various cases. (I've been known to do that, too, especially when anticipating a problem.) 

As for this notion of updating cf2016 from Java 8 to 11, I'm simply saying it's rarely been a solution to any problem, in my experience.

As for these general benefits that may be possible, I would love to see some testing that showed if any of those did inherently help typical cf processing, in a measurable way. They may, but I'm simply saying I've not seen it.

FWIW, I've helped many shops switch their jvm in cf2016 from 8 to 11, and none have reported any noticeable "overall improvement" from it. In all cases, it was simply something they wanted to try, or do to be more "up to date". Again, it's easy enough to try if you're careful. Conversely, none have felt any compelling need to switch back. Might it help some "generally"? Sure. 

Let's note finally that this is indeed only really a matter of consideration for cf2016, as cf2018 and above don't/won't support Java 8, and cf11 and below don't/won't support Java 11.

And FWIW, while cf2018 MAY have benefitted from the change to Java 11, again I have seen no noticeable improvement of a sort that would be tracked back to the new Java 11 changes you mention. 

Indeed, as for the g1 gc (or any alternatives), we should note that Adobe still hard-codes the selection of parallelgc in the CF jvm configuration. Again, I know people who have tried changing it to g1 or something else, but I've not heard of amazing results.

Indeed, I've never in 14 years of doing this server troubleshooting had a problem solved by changing the GCs algorithm. I know that borders on heresy to some, but my assertion is that GC problems are a symptom, and the treatment is to fix what's affecting heap use. Only rarely is it necessary to even tweak heap and gc related jvm args beyond simply raising the heap max/xmx. (Yes, I recommend removing the maxmetaspacesize in Java 8 and above, but that's not in the heap.) 

As always, I'm simply providing additional information to help people weigh choices they may be offered. There's usually a lot more to consider than may be found readily. More info (from both of us, and others) should help people make the best decision for them. 

[BKBK]

Let's hope someone from Adobe Team would comment on whether or not there are noticeable improvements in the move from Java 8 to 11. In any case, I can only report the experience of our software team:

• G1GC as garbage collector resulted in increased overall throughput;
• increased performance in terms of request volume per second.

Nevertheless, we have also seen an increase in overall system instability. I don't know yet if this is the result of the Java upgrade. I'm still looking into it.

[Charlie Arehart]

Thanks, Nalina. I'll follow up with more in reply to BKBK's suggestions in a moment.

First, you may well need to add more system memory, yes. But again, I'd propose that only to get you "over the hump". The REAL solution is to find the root cause of the excess memory use. We have shared a few possibilities previously here (whether code, load, config, bugs fixed by later versions of cf/fr/java, etc.) 

As for updating FR, I'll note that it can't be the version 6 you say it is, as the archive viewer you're using came out with 7.2. 🙂 If you want to know what version it is, see the "about" menu under the "fusionreactor" menu. And do that for the 8088 port (or whatever you're using for monitoring cf) rather than the 8087 (the FRAM instance). The latest version is 8.5 and you would be able to update to that for free if you are paying for maintenance.