Highlighted

Coldfusion 10 Enterprise with Tomcat + mod_jk and Apache2 experiencing child process hangups

Explorer ,
Jul 22, 2014

Copy link to clipboard

Copied

I am experiencing the most bizarre thing that so far I am unable to reproduce with my own visits to the site.

After restarting Apache2 my cacti graphs show that the child processes increment consistently over the course of a day without dropping back down during off hours.  This behavior eventually leaves the website inaccessible...

Looking at server-status it is filled with Ws (Sending Reply) and GET calls to my cfm applications :


Current Time: Tuesday, 22-Jul-2014 16:33:00 PDTRestart Time: Monday, 21-Jul-2014 22:51:12 PDTParent Server Generation: 0Server uptime: 17 hours 41 minutes 48 secondsTotal accesses: 194844 - Total Traffic: 3.8 GBCPU Usage: u201.55 s34.46 cu0 cs0 - .37% CPU load3.06 requests/sec - 63.2 kB/second - 20.6 kB/request73 requests currently being processed, 4 idle workers

WWWWWWWWWWWWWWWWWWWWWWWWWKWWWWWWWWWWWWWWWWWCWWWWW_WWWWWWCWWW_WWW _WKW...KWW.W_KWW....W........................................... ................................................................ ................................................................ 

Scoreboard Key:
"_" Waiting for Connection, "S" Starting up, "R" Reading Request,
"W" Sending Reply, "K" Keepalive (read), "D" DNS Lookup,
"C" Closing connection, "L" Logging, "G" Gracefully finishing,
"I" Idle cleanup of worker, "." Open slot with no current process

SrvPIDAccMCPUSSReqConnChildSlotClientVHostRequest
0-0150740/46/1370W7.394615800.00.4423.89192.168.1.10www.mysite.eduGET /directory/?directory=department&deptexp=7000 HTTP/1.1
1-0115630/47/468W2.755886700.04.6913.64192.168.1.10www.mysite.eduGET /catalog/index.cfm?courselist=list&dept=&searchc=PEHW%20148
2-0129060/65/884W7.305453600.00.8014.62192.168.1.10www.mysite.eduGET /athletics/resources/nwaacc-athlete-of-the-week/ HTTP/1.1
3-0138400/41/1085W4.015116200.00.5620.57192.168.1.10www.mysite.eduGET /directory/?directory=department&deptexp=17001 HTTP/1.1
4-0159280/20/1635W5.404371500.00.0641.37192.168.1.10www.mysite.eduGET /directory/?directory=department&deptexp=37000 HTTP/1.1
5-0187740/19/2387W0.333456400.00.2452.70192.168.1.10www.mysite.eduGET /directory/?directory=department&deptexp=19009 HTTP/1.1
6-043210/36/6612W3.611320000.00.28129.74192.168.1.10www.mysite.eduGET /directory/index.cfm?directory=department&deptexp=28011 HTT
7-0130770/0/808W0.425438300.00.0024.81192.168.1.10www.mysite.eduGET /directory/index.cfm?directory=department&deptexp=6005 HTTP
8-0164880/118/1673W12.394069200.01.3035.44192.168.1.10www.mysite.eduGET /directory/?directory=department&deptexp=31003 HTTP/1.1
9-0107260/15/110W0.586196300.00.051.83192.168.1.10www.mysite.eduGET /directory/index.cfm?directory=All&index=Q HTTP/1.1
10-0131540/1/688W0.005416500.00.0016.83192.168.1.10www.mysite.eduGET /directory/?directory=All&firstname=Patrick&lastname=Murphy
11-0125900/25/516W4.455585100.00.7611.19192.168.1.10www.mysite.eduGET /directory/?directory=department&deptexp=4000 HTTP/1.1
12-0125510/13/454W1.845605500.00.3810.00192.168.1.10www.mysite.eduGET /directory/index.cfm?directory=department&deptexp=20001 HTT
13-0133330/23/626W3.865318900.00.5711.66192.168.1.10www.mysite.eduGET /directory/?directory=department&deptexp=31005 HTTP/1.1
14-0124100/13/387W2.705648400.00.4210.55192.168.1.10www.mysite.eduGET /directory/?directory=department&deptexp=6003 HTTP/1.1
15-0131620/70/389W10.815311400.00.865.60192.168.1.10www.mysite.eduGET /directory/?directory=department&deptexp=6005 HTTP/1.1
16-0123090/22/275W2.235687800.00.433.91192.168.1.10www.mysite.eduGET /directory/?directory=department&deptexp=20005 HTTP/1.1
17-0131630/57/341W11.855312000.01.386.49192.168.1.10www.mysite.eduGET /catalog/index.cfm?courselist=list&dept=&searchc=ENGR%26%20

I have straced a hung process to only find the following :

strace -p 6472

Process 6472 attached - interrupt to quit

read(23,

Another interesting bit of info, none of these GET requests make it into my access.log file which I find very peculiar as well.

Here are my CF Specs

Server Details

Server Product ColdFusion

Version 10,0,13,287689

Tomcat Version 7.0.23.0

Edition Enterprise 

Serial Number

Operating System UNIX 

OS Version 3.2.0-65-generic 

Update Level /opt/coldfusion10/cfusion/lib/updates/chf10000013.jar 

Adobe Driver Version 4.1 (Build 0001) 

JVM Details

Java Version 1.6.0_29 

Java Vendor Sun Microsystems Inc. 

Here are my Apache2 Specs

Server version: Apache/2.2.22 (Ubuntu)

Server built:   Apr 17 2014 21:49:25

Server's Module Magic Number: 20051115:30

Server loaded:  APR 1.4.6, APR-Util 1.3.12

Compiled using: APR 1.4.6, APR-Util 1.3.12

Architecture:   64-bit

Server MPM:     Prefork

  threaded:     no

    forked:     yes (variable process count)

Server compiled with....

-D APACHE_MPM_DIR="server/mpm/prefork"

-D APR_HAS_SENDFILE

-D APR_HAS_MMAP

-D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)

-D APR_USE_SYSVSEM_SERIALIZE

-D APR_USE_PTHREAD_SERIALIZE

-D SINGLE_LISTEN_UNSERIALIZED_ACCEPT

-D APR_HAS_OTHER_CHILD

-D AP_HAVE_RELIABLE_PIPED_LOGS

-D DYNAMIC_MODULE_LIMIT=128

-D HTTPD_ROOT="/etc/apache2"

-D SUEXEC_BIN="/usr/lib/apache2/suexec"

-D DEFAULT_PIDLOG="/var/run/apache2.pid"

-D DEFAULT_SCOREBOARD="logs/apache_runtime_status"

-D DEFAULT_LOCKFILE="/var/run/apache2/accept.lock"

-D DEFAULT_ERRORLOG="logs/error_log"

-D AP_TYPES_CONFIG_FILE="mime.types"

-D SERVER_CONFIG_FILE="apache2.conf"

I am hoping this is no normal behavior for Coldfusion 10.

Many thanks in advance.

Views

418

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more

Coldfusion 10 Enterprise with Tomcat + mod_jk and Apache2 experiencing child process hangups

Explorer ,
Jul 22, 2014

Copy link to clipboard

Copied

I am experiencing the most bizarre thing that so far I am unable to reproduce with my own visits to the site.

After restarting Apache2 my cacti graphs show that the child processes increment consistently over the course of a day without dropping back down during off hours.  This behavior eventually leaves the website inaccessible...

Looking at server-status it is filled with Ws (Sending Reply) and GET calls to my cfm applications :


Current Time: Tuesday, 22-Jul-2014 16:33:00 PDTRestart Time: Monday, 21-Jul-2014 22:51:12 PDTParent Server Generation: 0Server uptime: 17 hours 41 minutes 48 secondsTotal accesses: 194844 - Total Traffic: 3.8 GBCPU Usage: u201.55 s34.46 cu0 cs0 - .37% CPU load3.06 requests/sec - 63.2 kB/second - 20.6 kB/request73 requests currently being processed, 4 idle workers

WWWWWWWWWWWWWWWWWWWWWWWWWKWWWWWWWWWWWWWWWWWCWWWWW_WWWWWWCWWW_WWW _WKW...KWW.W_KWW....W........................................... ................................................................ ................................................................ 

Scoreboard Key:
"_" Waiting for Connection, "S" Starting up, "R" Reading Request,
"W" Sending Reply, "K" Keepalive (read), "D" DNS Lookup,
"C" Closing connection, "L" Logging, "G" Gracefully finishing,
"I" Idle cleanup of worker, "." Open slot with no current process

SrvPIDAccMCPUSSReqConnChildSlotClientVHostRequest
0-0150740/46/1370W7.394615800.00.4423.89192.168.1.10www.mysite.eduGET /directory/?directory=department&deptexp=7000 HTTP/1.1
1-0115630/47/468W2.755886700.04.6913.64192.168.1.10www.mysite.eduGET /catalog/index.cfm?courselist=list&dept=&searchc=PEHW%20148
2-0129060/65/884W7.305453600.00.8014.62192.168.1.10www.mysite.eduGET /athletics/resources/nwaacc-athlete-of-the-week/ HTTP/1.1
3-0138400/41/1085W4.015116200.00.5620.57192.168.1.10www.mysite.eduGET /directory/?directory=department&deptexp=17001 HTTP/1.1
4-0159280/20/1635W5.404371500.00.0641.37192.168.1.10www.mysite.eduGET /directory/?directory=department&deptexp=37000 HTTP/1.1
5-0187740/19/2387W0.333456400.00.2452.70192.168.1.10www.mysite.eduGET /directory/?directory=department&deptexp=19009 HTTP/1.1
6-043210/36/6612W3.611320000.00.28129.74192.168.1.10www.mysite.eduGET /directory/index.cfm?directory=department&deptexp=28011 HTT
7-0130770/0/808W0.425438300.00.0024.81192.168.1.10www.mysite.eduGET /directory/index.cfm?directory=department&deptexp=6005 HTTP
8-0164880/118/1673W12.394069200.01.3035.44192.168.1.10www.mysite.eduGET /directory/?directory=department&deptexp=31003 HTTP/1.1
9-0107260/15/110W0.586196300.00.051.83192.168.1.10www.mysite.eduGET /directory/index.cfm?directory=All&index=Q HTTP/1.1
10-0131540/1/688W0.005416500.00.0016.83192.168.1.10www.mysite.eduGET /directory/?directory=All&firstname=Patrick&lastname=Murphy
11-0125900/25/516W4.455585100.00.7611.19192.168.1.10www.mysite.eduGET /directory/?directory=department&deptexp=4000 HTTP/1.1
12-0125510/13/454W1.845605500.00.3810.00192.168.1.10www.mysite.eduGET /directory/index.cfm?directory=department&deptexp=20001 HTT
13-0133330/23/626W3.865318900.00.5711.66192.168.1.10www.mysite.eduGET /directory/?directory=department&deptexp=31005 HTTP/1.1
14-0124100/13/387W2.705648400.00.4210.55192.168.1.10www.mysite.eduGET /directory/?directory=department&deptexp=6003 HTTP/1.1
15-0131620/70/389W10.815311400.00.865.60192.168.1.10www.mysite.eduGET /directory/?directory=department&deptexp=6005 HTTP/1.1
16-0123090/22/275W2.235687800.00.433.91192.168.1.10www.mysite.eduGET /directory/?directory=department&deptexp=20005 HTTP/1.1
17-0131630/57/341W11.855312000.01.386.49192.168.1.10www.mysite.eduGET /catalog/index.cfm?courselist=list&dept=&searchc=ENGR%26%20

I have straced a hung process to only find the following :

strace -p 6472

Process 6472 attached - interrupt to quit

read(23,

Another interesting bit of info, none of these GET requests make it into my access.log file which I find very peculiar as well.

Here are my CF Specs

Server Details

Server Product ColdFusion

Version 10,0,13,287689

Tomcat Version 7.0.23.0

Edition Enterprise 

Serial Number

Operating System UNIX 

OS Version 3.2.0-65-generic 

Update Level /opt/coldfusion10/cfusion/lib/updates/chf10000013.jar 

Adobe Driver Version 4.1 (Build 0001) 

JVM Details

Java Version 1.6.0_29 

Java Vendor Sun Microsystems Inc. 

Here are my Apache2 Specs

Server version: Apache/2.2.22 (Ubuntu)

Server built:   Apr 17 2014 21:49:25

Server's Module Magic Number: 20051115:30

Server loaded:  APR 1.4.6, APR-Util 1.3.12

Compiled using: APR 1.4.6, APR-Util 1.3.12

Architecture:   64-bit

Server MPM:     Prefork

  threaded:     no

    forked:     yes (variable process count)

Server compiled with....

-D APACHE_MPM_DIR="server/mpm/prefork"

-D APR_HAS_SENDFILE

-D APR_HAS_MMAP

-D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)

-D APR_USE_SYSVSEM_SERIALIZE

-D APR_USE_PTHREAD_SERIALIZE

-D SINGLE_LISTEN_UNSERIALIZED_ACCEPT

-D APR_HAS_OTHER_CHILD

-D AP_HAVE_RELIABLE_PIPED_LOGS

-D DYNAMIC_MODULE_LIMIT=128

-D HTTPD_ROOT="/etc/apache2"

-D SUEXEC_BIN="/usr/lib/apache2/suexec"

-D DEFAULT_PIDLOG="/var/run/apache2.pid"

-D DEFAULT_SCOREBOARD="logs/apache_runtime_status"

-D DEFAULT_LOCKFILE="/var/run/apache2/accept.lock"

-D DEFAULT_ERRORLOG="logs/error_log"

-D AP_TYPES_CONFIG_FILE="mime.types"

-D SERVER_CONFIG_FILE="apache2.conf"

I am hoping this is no normal behavior for Coldfusion 10.

Many thanks in advance.

Views

419

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Jul 22, 2014 0
sbleon LATEST
New Here ,
Sep 19, 2014

Copy link to clipboard

Copied

We're having the same problem, although with CF11.

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Reply
Loading...
Sep 19, 2014 0