CF2021 crashes Apache server on cfhttp calls

[rspitzmd]

Hi community,

we are moving from our old CF2016 environment (Apache 2.4 on SLES 12), which has been working without major problems for many years, to CF2021 with Apache 2.4 on SLES 15. Yes, I'm fully aware that SLES is NOT on the list of supported OSes for CF2021, but for the time being, I have no choice due to inhouse policy.

With CF2021 (Update level 3) we now experience major problems with frequent crashes of Apache subprocesses, which seem to be related to cfhttp calls. These calls do not fail every time, but frequently and I cannot find a pattern to this. Only house-internal URLs, mostly https://, are called with cfhttp, I'm fairly certain that this is NOT a certificate problem since the inhouse Root- and Sub-CA-Certificates were imported into the cacerts store.

In /var/log/warn there are countless entries like this:

2022-02-24T14:55:59.909952+01:00 srvanawebx3 systemd-coredump[34008]: Process 33666 (httpd-prefork) of user 1004 dumped core.#012#012Stack trace of thread 33668:#012#0  0x00007f4a59212cdb raise (libc.so.6 + 0x4acdb)#012#1  0x00007f4a59214375 abort (libc.so.6 + 0x4c375)#012#2  0x00007f4a59258b07 __libc_message (libc.so.6 + 0x90b07)#012#3  0x00007f4a59260b8a malloc_printerr (libc.so.6 + 0x98b8a)#012#4  0x00007f4a59260e5c munmap_chunk (libc.so.6 + 0x98e5c)#012#5  0x00007f4a529d0cd9 jk_connector_CF_func (mod_jk.so + 0x12cd9)#012#6  0x00007f4a595c76ea start_thread (libpthread.so.0 + 0xa6ea)#012#7  0x00007f4a592dfa8f __clone (libc.so.6 + 0x117a8f)#012#012Stack trace of thread 33666:#012#0  0x00007f4a592d5837 __select (libc.so.6 + 0x10d837)#012#1  0x00007f4a5980b105 apr_sleep (libapr-1.so.0 + 0x2b105)#012#2  0x00007f4a529d0a7a jk_cleanup_child (mod_jk.so + 0x12a7a)#012#3  0x00007f4a597ff70e apr_pool_destroy (libapr-1.so.0 + 0x1f70e)#012#4  0x000055e8cf2f2a47 n/a (httpd-prefork + 0x6fa47)#012#5  0x000055e8cf2f2fd1 n/a (httpd-prefork + 0x6ffd1)#012#6  0x000055e8cf2f3370 n/a (httpd-prefork + 0x70370)#012#7  0x000055e8cf2f3dca n/a (httpd-prefork + 0x70dca)#012#8  0x000055e8cf2b5d8e ap_run_mpm (httpd-prefork + 0x32d8e)#012#9  0x000055e8cf2adaab n/a (httpd-prefork + 0x2aaab)#012#10 0x00007f4a591fd2bd __libc_start_main (libc.so.6 + 0x352bd)#012#11 0x000055e8cf2adbaa _start (httpd-prefork + 0x2abaa)

and to a much lesser extent:

2022-02-24T14:55:59.909952+01:00 srvanawebx3 systemd-coredump[34008]: Process 33666 (httpd-prefork) of user 1004 dumped core.#012#012Stack trace of thread 33668:#012#0  0x00007f4a59212cdb raise (libc.so.6 + 0x4acdb)#012#1  0x00007f4a59214375 abort (libc.so.6 + 0x4c375)#012#2  0x00007f4a59258b07 __libc_message (libc.so.6 + 0x90b07)#012#3  0x00007f4a59260b8a malloc_printerr (libc.so.6 + 0x98b8a)#012#4  0x00007f4a59260e5c munmap_chunk (libc.so.6 + 0x98e5c)#012#5  0x00007f4a529d0cd9 jk_connector_CF_func (mod_jk.so + 0x12cd9)#012#6  0x00007f4a595c76ea start_thread (libpthread.so.0 + 0xa6ea)#012#7  0x00007f4a592dfa8f __clone (libc.so.6 + 0x117a8f)#012#012Stack trace of thread 33666:#012#0  0x00007f4a592d5837 __select (libc.so.6 + 0x10d837)#012#1  0x00007f4a5980b105 apr_sleep (libapr-1.so.0 + 0x2b105)#012#2  0x00007f4a529d0a7a jk_cleanup_child (mod_jk.so + 0x12a7a)#012#3  0x00007f4a597ff70e apr_pool_destroy (libapr-1.so.0 + 0x1f70e)#012#4  0x000055e8cf2f2a47 n/a (httpd-prefork + 0x6fa47)#012#5  0x000055e8cf2f2fd1 n/a (httpd-prefork + 0x6ffd1)#012#6  0x000055e8cf2f3370 n/a (httpd-prefork + 0x70370)#012#7  0x000055e8cf2f3dca n/a (httpd-prefork + 0x70dca)#012#8  0x000055e8cf2b5d8e ap_run_mpm (httpd-prefork + 0x32d8e)#012#9  0x000055e8cf2adaab n/a (httpd-prefork + 0x2aaab)#012#10 0x00007f4a591fd2bd __libc_start_main (libc.so.6 + 0x352bd)#012#11 0x000055e8cf2adbaa _start (httpd-prefork + 0x2abaa)

This seems to point to a problem with the connector, which I reconfigured using the wsconfig tool (both graphic and commandline) several times without any change.

Again, I'm aware that our OS SLES 15 is not officially supported with CF2021, but I still hope that someone here can help me solving this problem.

Regards, Richard

[Charlie Arehart]

I can't speak to the SES support matter, but let's clarify that you're saying the cfhttp calls are to cf pages, right? That's where the involvement of mod_jk.so comes into play in the apache logs, as it tries to pass the cf request to cf via that connector.

As for cfhttp being a contributor to the problem, what happens if you take it out of the equation? What if you setup Apache bench (ab) to call the same urls? Do they show failing (in the apache logs) then also? If so, then this has nothing to do with cfhttp.

I do realize you feel the problem started only with CF2021, but the issue could be with it SERVING the pages rather than requesting them.

Put another way, you've classed this as a demand-side problem (the cfhttp call TO the cf pages), but it may be a supply-side problem (the serving of the cf page, perhaps regardless of who makes the call).

Let us know if this is at all helpful. Or I suspect others will have more thoughts for you. 

[rspitzmd]

Hi Charlie,

thanks for the quick reply. I am pretty confident that the pages called by CFHTTP can all be served ok, since they can be called repetitively from a browser without any problems. Also, a reload of the page containing the CFHTTP call will in most cases be successful, but that doesn't help on pages that contain such calls in a loop, with different parameters each time. 

Next thing I'll try is switching to a newer JAVA version for CF2021, currently it is using the bundled version 11.0.1+13LTS. I still cannot say for sure, but it could be that mostly if not only calls to encrypted URLs (https://) are affected. 

I've also opened a case with Adobe, asking for the CF2021 connector source code so I can compile it myself. Currently, Adobe only offers the sourcecode for the CF2016 connector.

Regards, Richard

[shige. · Accepted solution]

Is it the same as this case?

https://tracker.adobe.com/#/view/CF-4205971

Set heartbeat_interval to 0 in worker.properties.

The file is created in the Apache configuration directory (e.g. /etc/httpd/conf/workers.properties).

After changing the file, you Need to restart the Apache server.

[rspitzmd]

Hi shige,

thank you, that actually did the trick! The server has been running for several hours now under normal load without any Apache crashes after I changed the setting of heartbeat_interval to 0.

This setting seems to be relevant only for the Performance Monitoring Toolset (PMT), which I don't recall being installed. I found https://tracker.adobe.com/#/view/CF-4205636 which describes this problem already for CF2018. Adobe closed the case due to "NotWorthEffort" 😞

Regards, Richard

[rspitzmd]

Switching to the newest JAVA version 11.0.13 available for download from Adobe didn't solve the problem.