CF2016 on RHEL7 - java.lang.OutOfMemoryError: Compressed class space

Nalina, I want to add to what BKBK has, and focus first on a different point, which is what you are "seeing" in the graphs, and why it may seem confusing ("never above 30%"). Then I want to address the relationship between this space and the heap that you mention, and finally I have a proposal of yet another diagnostic step you may want to consider.

1) You observed that "CF reactor log viewer shows 30% of compressed class space is used at any time". I assume you mean instead to FusionReactor, and its logs (and "log viewer), which track each memory space (CF has no reactor log, and no logs tracking each memory space), and further that you are viewing the log in the FR "Archived Metrics" viewer, which is great.

2) But I want to share a warning about your observation that it (the "log viewer") "shows 30% of compressed class space is used at any time". Your (reasonable) conclusion seems to be that therefore, "the compressed class space never went above 30%, so how did it 'run out of memory'"? There are a couple of explanations for that, which are important to understand.

a) First, note that if you are indeed viewing the last log graphed before CF went down/you restarted it, there is a real possibility that CF could have become so unstable that it could not write its memory log files every 5 seconds (as it does, otherwise). In that case, you can't trust what you are seeing in the graphs, as I will explain.

So if you use the "view" menu on the top right of that page, and then look at the "spreadsheet" or "text" view, to see the real log lines, you may see that the time reflected in the log (in the first two cols on each line) is not only NOT every 5 seconds (before things started to go bad, compared to when "all was well"), but you may well see that it just STOPS logging entirely, perhaps minutes before you know that CF was brought down. That would confirm my suspicion, and would not be uncommon.

b) And both points can lead to the graph not being reliable. First, it may be missing datapoints (and the graphs may make inferences/smoothing that hides the gaps), and worse you may see the last point on the graph showing that 30%, but it could be from minutes before CF went down/you restarted it. To that last point, you can at least look at the times on the bottom of the graphs. You may notice that the intervals show 3/4 of the graph covering perhaps 30 mins, and the last 1/4 of the graph covering only 15 mins. That would be wrong, of course.

3) So with all that said, the OOM error you reported confirms that you DID run out of that memory space. And BKBK has pointed to the classic responses to consider. 

4) You mention that "JVM heap size is available on the system", by which I assume you mean that according to the FR graphs, the heap usage was not high. That may be so.

One thing to note is that the compressed class space does not come FROM (does not live WITHIN) the heap. It is separate.

I can say that with 100% certainty, because on a machine where I have only 800mb heap size (xmx), I can see in the FR graph of the compressedclassspace size that it's the 1000mb (1gb) that you both mention is the default if one does not change that size. (And let that be a lesson to those who look at the OS memory space, and wonder how CF's use of memory can be higher than the heap size. This space is one example, and the metaspace is another, and so on. Of course, it's one thing for such a memory space to have such a "max" value and for that space to be actually "used" or "allocated", and FR tracks that. The latter is what would contribute to memory space for the process in the OS.)

5) So bottom line, you do need to find out why you are using so much compressedclassspace.

6) BTW, you don't mention what version of FR you have. There's always the possibility that it (or some feature in it) is a contributor to this problem. The latest version is 8.5. What version are you on? And what CF version and update? and what JVM version?

7) Given that you are using FR, if the problem repeats often (frustrating as that is), you could try removing FR temporarily, to see if the problem went away. If it did, you would want to report this to the FR folks, who would jump on it.

And when I say remove it temporarily, I don't mean "uinstall it", or even remove it from CF using the FRAM Instance Manager (if you know what that means). Doing either would lose all the FR logs for the instance, which go back 30 days by default. Instead, I simply mean (carefully) comment out the two FR-related args in the CF jvm.config, and then restart CF. That will cause FR to be "removed" from CF, until you put those args back.

Let us know how things go.

The root cause is clear. The space that your application needs for UseCompressedClassPointers exceeds the available CompressedClassSpaceSize. The default value for the size is 1GB. 

If you're certain that your application doesn't use that many files, then the error suggests that something else in your application is creating a lot of classes and never releasing them. Think of a classloader, for example.

This StackOverflow post on compressed class space contains the following helpful suggestions:

1. If your application can manage without so many files, then disabling compressed class pointers is perhaps an option. The relevant JVM flag is -XX:-UseCompressedClassPointers. 
2. If your application does indeed need that many files, then you could consider increasing CompressedClassSpaceSize to 2GB. The relevant flag is -XX:CompressedClassSpaceSize=2g.
3. In any case, you should enable the following JVM settings to troubleshoot the classes loaded:
   -XX:+PrintGCDetails -XX:+TraceClassUnloading -XX:+TraceClassLoading