Random 503 error or Coldfusion unresponsive

[IT319155746nmp]

Hi

We have an issue with our Coldfusion 2023 (Update 5) setup. 

Background:

Coldfusion 2023 running on Ubuntu 22.04

Database server is Ubuntu 22.04 running MariaDB lastest version

We migrated away from Coldfusion 2016 in August.

Everything for the first couple of months, but now we are seeing strange issue and the server crashing.

This morning, our coldfusion application became unresponsive and eventually resulted in a 503 page. No logging was happening and fusion reactor stopped responding also, as well as the C2023 admin colsole webpage.

I restarted the coldfusion service to get us up and running again. This has happen twice now and I can't find anything in the error logs to suggest what it is, as all logging just stops.

We did start getting some strange errors with our code yesterday afternoon. "Element RECORDCOUNT is undefined in Query_name". We did loads of testing. Query worked 100% in Workbench and directly on the server, but when running on coldfusion, the query would run but as soon as you start using any of the information, you would get Element is undefined error appear.

This was happening on two different queries on completely different areas of the system.

Towards the end of the day, we were noticing the system running slower and not as responsive as normally would be.

We left everything as is but this morning, as mentioned above, the system became unresponive and showing 503 errors.

I've also noticed that I now can't access some of the Coldfusion logs. I'm getting the error below.

 

Any help on this will be greatly appreciated.

Many thanks

Ollie

[Charlie Arehart]

Yours is a very common problem, but there's no single problem that it will definitely be. Still, it's not an impossible problem tonsokve. Before considering the cfquery or log viewer errors (which may be transient), let's focus on your more important slow diwn/hangup/503.

While you see nothing to help in the cf logs, there may be something in them (any one of many) so look carefully, and at the time just before you restart cf.

Also, you're in great shape having FR: it would have tracked what's going on. Digging into ITS logs (or its archive viewer, under logging or metrics) can show what's happening in the moments before the outage...if indeed the problem is within cf it COULD be in your web server or connector config, instead).

But here's a simple possibility: what is the cf max heap size, shown in the cf admin Java & jvm page? And do you know what it was for cf2016? (It can be viewed as the xmx value in the jvm.config file of cf2016, if you still have it but it's not running to see its admin). If they were both on this same machine, you should be able to set your cf2023 to the same value--which may have been far larger than the default 1024. Sometimes just raising that is the solution (and there's far more to consider about that, if you do NOT have what your cf2016 heap size was). 

If that's NOT the issue, there's also far more that could be said about any of MANY different potential explanations for cf hanging/slowing, from slowness in the db to slowness in api's or cfhttp calls you make, where cf is the victim rather than the culprit. Or you could have some one rogue request causing problems, or maybe you're server's being slammed with unexpected traffic. And so on. 

Again, digging into fr could identify the problem more clearly. It will have tracked the requests, their rates, their query times, the cpu and the memory used by cf, and so much more. It's like having a black box recorder for a plane crash.

If you're not comfortable doing that (or if it fails to show any problem in cf), I help people understand and resolve such problems daily. If you may wish to just get direct help (we might solve it in less than an hour), you can find my rates, approach, satisfaction guarantee, online calendar link and more at carehart.org/consulting.

Otherwise we (others and I) can try to help here. I tend not to be able to help most of the day as I'm engaged in such help sessions. Apologies if I may be delayed in replying to any response here.