JRUN chewing up memory - how to debug?

Report · Dec 08, 2009

I've been reading up on my current issue which is that the JRUN task is chewing up memory so that it's taking 100% of the server memory. It crashed this morning at 3:30am and the administrator has had to allocate more memory on the server, but JRUN is still taking up a lot of memory.

I looking at the JRUN default-out.log file and it doesn't seem to point to any crash. The cfusion-err.log is huge: 10GB(!), but if I try to open it, there's another task that has a lock on the file. The application and server logs don't seem to give me any clues.

The only thing I've been able to find is this, but I'm not sure how to interpret it:

FROM cfusion-event_1.log. These errors repeat..

12/08 03:17:44 error Connection reset

java.net.SocketException: Connection reset

at java.net.SocketInputStream.read(SocketInputStream.java:168)

at java.io.BufferedInputStream.fill(BufferedInputStream.java:183)

at java.io.BufferedInputStream.read1(BufferedInputStream.java:222)

at java.io.BufferedInputStream.read(BufferedInputStream.java:277)

at jrun.servlet.jrpp.ProxyEndpoint.readFully(ProxyEndpoint.java:530)

at jrun.servlet.jrpp.ProxyEndpoint.readFully(ProxyEndpoint.java:522)

at jrun.servlet.jrpp.ProxyEndpoint.readInt(ProxyEndpoint.java:540)

at jrun.servlet.jrpp.ProxyEndpoint.readRequest(ProxyEndpoint.java:229)

at jrun.servlet.jrpp.JRunProxyService.swapRunnable(JRunProxyService.java:144)

at jrunx.scheduler.ThreadPool$ThreadThrottle.swapRunnable(ThreadPool.java:433)

at jrunx.scheduler.WorkerThread.run(WorkerThread.java:76)

We have CF 6.1. Are there any other log files I should be looking at? Or, is there a better way I can pinpoint what the issue is?

Report · Dec 09, 2009

We had the same problem and fixed it by changing the way ColdFusion would store Client data (cookies ans session data). By default ColdFusion will store the Client Storage data in the registry and then purge it after a specified amount of days. If your site generates a significant amount of traffic, the registry can grow up rapidly and cause instability. To resolve this problem you can setup the Client Storage to store data in a database (cstorage) on your sever per Adobe's recommended guidelines.

http://livedocs.adobe.com/coldfusion/8/htmldocs/help.html?content=sharedVars_08.html

In our case it worked pretty well. JRUN is now using much less CPU.

Report · Dec 10, 2009

Thanks a lot, Frank. I'm going to work on that and will update this thread to see if that fixes the issue.

Report · Jan 23, 2010

Late getting back here, sorry.

Turns out we weren't using Client variables at all.

However, after more research, I found on article that pointed to the fact that search bots generate new sessions whenever they hit the website. In application.cfm, I had to put in a check for the type of user, then if the user was a search bot, I didn't create a session. I created a log to capture all bot hits during the day, and the logs are about 1MB per day.

I then added a utility that tracks memory and session usage, and the bot filtering caused session usage to be cut down by 50%. We haven't seen the same memory issue since, so I can't say for certain that this fixed the issue, but I'm hoping we won't see that error message again.

If anyone needs more info, please reply to this thread and let me know.