Coldfusion 10 Enterprise with Tomcat + mod_jk and Apache2 experiencing child process hangups

Question

I am experiencing the most bizarre thing that so far I am unable to reproduce with my own visits to the site.

After restarting Apache2 my cacti graphs show that the child processes increment consistently over the course of a day without dropping back down during off hours. This behavior eventually leaves the website inaccessible...

Looking at server-status it is filled with Ws (Sending Reply) and GET calls to my cfm applications :

Current Time: Tuesday, 22-Jul-2014 16:33:00 PDTRestart Time: Monday, 21-Jul-2014 22:51:12 PDTParent Server Generation: 0Server uptime: 17 hours 41 minutes 48 secondsTotal accesses: 194844 - Total Traffic: 3.8 GBCPU Usage: u201.55 s34.46 cu0 cs0 - .37% CPU load3.06 requests/sec - 63.2 kB/second - 20.6 kB/request73 requests currently being processed, 4 idle workers

WWWWWWWWWWWWWWWWWWWWWWWWWKWWWWWWWWWWWWWWWWWCWWWWW_WWWWWWCWWW_WWW _WKW...KWW.W_KWW....W........................................... ................................................................ ................................................................

Scoreboard Key:
"_" Waiting for Connection, "S" Starting up, "R" Reading Request,
"W" Sending Reply, "K" Keepalive (read), "D" DNS Lookup,
"C" Closing connection, "L" Logging, "G" Gracefully finishing,
"I" Idle cleanup of worker, "." Open slot with no current process

Srv	PID	Acc	M	CPU	SS	Child	Slot	Client	VHost	Request
0-0	15074	0/46/1370	W	7.39	46158	0.44	23.89	192.168.1.10	www.mysite.edu	GET /directory/?directory=department&deptexp=7000 HTTP/1.1
1-0	11563	0/47/468	W	2.75	58867	4.69	13.64	192.168.1.10	www.mysite.edu	GET /catalog/index.cfm?courselist=list&dept=&searchc=PEHW%20148
2-0	12906	0/65/884	W	7.30	54536	0.80	14.62	192.168.1.10	www.mysite.edu	GET /athletics/resources/nwaacc-athlete-of-the-week/ HTTP/1.1
3-0	13840	0/41/1085	W	4.01	51162	0.56	20.57	192.168.1.10	www.mysite.edu	GET /directory/?directory=department&deptexp=17001 HTTP/1.1
4-0	15928	0/20/1635	W	5.40	43715	0.06	41.37	192.168.1.10	www.mysite.edu	GET /directory/?directory=department&deptexp=37000 HTTP/1.1
5-0	18774	0/19/2387	W	0.33	34564	0.24	52.70	192.168.1.10	www.mysite.edu	GET /directory/?directory=department&deptexp=19009 HTTP/1.1
6-0	4321	0/36/6612	W	3.61	13200	0.28	129.74	192.168.1.10	www.mysite.edu	GET /directory/index.cfm?directory=department&deptexp=28011 HTT
7-0	13077	0/0/808	W	0.42	54383	0.00	24.81	192.168.1.10	www.mysite.edu	GET /directory/index.cfm?directory=department&deptexp=6005 HTTP
8-0	16488	0/118/1673	W	12.39	40692	1.30	35.44	192.168.1.10	www.mysite.edu	GET /directory/?directory=department&deptexp=31003 HTTP/1.1
9-0	10726	0/15/110	W	0.58	61963	0.05	1.83	192.168.1.10	www.mysite.edu	GET /directory/index.cfm?directory=All&index=Q HTTP/1.1
10-0	13154	0/1/688	W	0.00	54165	0.00	16.83	192.168.1.10	www.mysite.edu	GET /directory/?directory=All&firstname=Patrick&lastname=Murphy
11-0	12590	0/25/516	W	4.45	55851	0.76	11.19	192.168.1.10	www.mysite.edu	GET /directory/?directory=department&deptexp=4000 HTTP/1.1
12-0	12551	0/13/454	W	1.84	56055	0.38	10.00	192.168.1.10	www.mysite.edu	GET /directory/index.cfm?directory=department&deptexp=20001 HTT
13-0	13333	0/23/626	W	3.86	53189	0.57	11.66	192.168.1.10	www.mysite.edu	GET /directory/?directory=department&deptexp=31005 HTTP/1.1
14-0	12410	0/13/387	W	2.70	56484	0.42	10.55	192.168.1.10	www.mysite.edu	GET /directory/?directory=department&deptexp=6003 HTTP/1.1
15-0	13162	0/70/389	W	10.81	53114	0.86	5.60	192.168.1.10	www.mysite.edu	GET /directory/?directory=department&deptexp=6005 HTTP/1.1
16-0	12309	0/22/275	W	2.23	56878	0.43	3.91	192.168.1.10	www.mysite.edu	GET /directory/?directory=department&deptexp=20005 HTTP/1.1
17-0	13163	0/57/341	W	11.85	53120	1.38	6.49	192.168.1.10	www.mysite.edu	GET /catalog/index.cfm?courselist=list&dept=&searchc=ENGR%26%20

I have straced a hung process to only find the following :

strace -p 6472

Process 6472 attached - interrupt to quit

read(23,

Another interesting bit of info, none of these GET requests make it into my access.log file which I find very peculiar as well.

Here are my CF Specs

Server Details

Server Product ColdFusion

Version 10,0,13,287689

Tomcat Version 7.0.23.0

Edition Enterprise

Serial Number

Operating System UNIX

OS Version 3.2.0-65-generic

Update Level /opt/coldfusion10/cfusion/lib/updates/chf10000013.jar

Adobe Driver Version 4.1 (Build 0001)

JVM Details

Java Version 1.6.0_29

Java Vendor Sun Microsystems Inc.

Here are my Apache2 Specs

Server version: Apache/2.2.22 (Ubuntu)

Server built: Apr 17 2014 21:49:25

Server's Module Magic Number: 20051115:30

Server loaded: APR 1.4.6, APR-Util 1.3.12

Compiled using: APR 1.4.6, APR-Util 1.3.12

Architecture: 64-bit

Server MPM: Prefork

threaded: no

forked: yes (variable process count)

Server compiled with....

-D APACHE_MPM_DIR="server/mpm/prefork"

-D APR_HAS_SENDFILE

-D APR_HAS_MMAP

-D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)

-D APR_USE_SYSVSEM_SERIALIZE

-D APR_USE_PTHREAD_SERIALIZE

-D SINGLE_LISTEN_UNSERIALIZED_ACCEPT

-D APR_HAS_OTHER_CHILD

-D AP_HAVE_RELIABLE_PIPED_LOGS

-D DYNAMIC_MODULE_LIMIT=128

-D HTTPD_ROOT="/etc/apache2"

-D SUEXEC_BIN="/usr/lib/apache2/suexec"

-D DEFAULT_PIDLOG="/var/run/apache2.pid"

-D DEFAULT_SCOREBOARD="logs/apache_runtime_status"

-D DEFAULT_LOCKFILE="/var/run/apache2/accept.lock"

-D DEFAULT_ERRORLOG="logs/error_log"

-D AP_TYPES_CONFIG_FILE="mime.types"

-D SERVER_CONFIG_FILE="apache2.conf"

I am hoping this is no normal behavior for Coldfusion 10.

Many thanks in advance.

sbleon · Answer

We're having the same problem, although with CF11.

Sign up

To post, reply, or follow discussions, please sign in with your Adobe ID.

Sign in to Adobe Community

To post, reply, or follow discussions, please sign in with your Adobe ID.

Scanning file for viruses.

This file cannot be downloaded