CF10 service not starting/Tomcat slow

[ericbegins]

This is happening in 2 identical environments:

Brand new machine, fresh copy of CF10 Standard x64 on Windows Server 2k8 R2.  CF is patched through update 11.  CF always fails to start after a reboot.  I go into manually start the service and it takes 3-4 tries (and 10-15 minutes) to get it going.  This is usually the case if I ever try to restart the service manually (without reboot), although occasionally it starts back up with no problem.

If I try to start the service from the command line, it hangs at "INFO: starting servlet engine: Apache Tomcat/7.0.23" for about 8-10 minutes before finally continuing on and starting up.  This jibes with the service problems as the Windows service probably gets tired of waiting on the servlet engine and gives up.

Any idea as to what could be causing the delay?

[carl type3]

What is recored in ColdFusion10\cfusion\logs\coldfusion-error.log when CF10 tries to start? Here is example:

Jan 25, 2014 5:25:47 PM org.apache.catalina.core.AprLifecycleListener init
INFO: The APR based Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path: d:\\ColdFusion10\\cfusion\lib;d:\\ColdFusion10\\cfusion\jintegra\bin;d:\\ColdFusion10\\cfusion\jintegra\bin\international;d:\\ColdFusion10\\cfusion\lib\oosdk\classes\win
Jan 25, 2014 5:25:48 PM org.apache.coyote.AbstractProtocol init
INFO: Initializing ProtocolHandler ["ajp-bio-8012"]
Jan 25, 2014 5:25:48 PM org.apache.catalina.core.StandardService startInternal
INFO: Starting service Catalina
Jan 25, 2014 5:25:48 PM org.apache.catalina.core.StandardEngine startInternal
INFO: Starting Servlet Engine: Apache Tomcat/7.0.23
Jan 25, 2014 5:25:59 PM org.apache.catalina.core.ApplicationContext log
INFO: ColdFusionStartUpServlet: ColdFusion: Starting application services
Jan 25, 2014 5:25:59 PM org.apache.catalina.core.ApplicationContext log
INFO: ColdFusionStartUpServlet: ColdFusion: VM version = 23.7-b01
java.lang.NullPointerException
at coldfusion.server.jrun4.metrics.SimpleLoadMetric.run(SimpleLoadMetric.java:157)
at coldfusion.scheduling.ThreadPool.run(ThreadPool.java:211)
at coldfusion.scheduling.WorkerThread.run(WorkerThread.java:71)
Jan 25, 2014 5:26:07 PM org.apache.catalina.core.ApplicationContext log
INFO: CFMxmlServlet: Macromedia Flex Build: 87315.134646
Jan 25, 2014 5:26:13 PM org.apache.coyote.AbstractProtocol start
INFO: Starting ProtocolHandler ["ajp-bio-8012"]
Jan 25, 2014 5:26:13 PM com.adobe.coldfusion.launcher.Launcher run
INFO: Server startup in 27609 ms

HTH, Carl.

[ericbegins]

I can't easily copy and paste, but I'll retype here.  Each service startup failure results in 4 entries in the coldfusion-error.log file.  The first one is the only one that mentions a problem:

INFO: The APR based Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path: c:\\ColdFusion10\\cfusion\lib; (and goes on to list other places it looked)

CF10 is installed on the C drive, so it appears to be looking in the right place.  The other entries appear to be informational and just talk about AbstractProtocol and the Catalina Service intiiating.

The last line is the:

INFO: Starting Servlet Engine: Apache Tomcat/7.0.23

...which I mentioned initially takes about 8-10 minutes during the coldfusion-start.bat file. 

Even after the service prompts about the error, the status stays as "Starting" for a few more minutes before eventually stopping.  Nothing else ever gets written to the error log.  Manually start the service again and the 4 entries again get written to the log.

On the 4th attempt to start the service, Tomcat started in about 2.5 minutes and everything continued.

Thanks for your help!

[carl type3]

This message "The APR based Apache Tomcat Native library which allows optimal performance in production environments" can mislead diagnosis of problem. I do not think that will be the cause of your issue tho you can overcome that then if problem ceases well and good.

You will need to download extract tcnative-1.dll to ColdFusion10\cfusion\lib then restart CF10 service to apply.

Note after extracting downloaded zip it contains both 32 and 64 bit editions of DLL. Use \bin\x64 for 64bit source.

refer:

http://tomcat.apache.org/native-doc/index.html

select "download page" then

"Native 1.1.29 Source Release zip  (e.g. Windows)" link.

HTH, Carl.

[carl type3 · Accepted solution]

I had another think about this. You say CF-error logs ends with:

INFO: Starting Servlet Engine: Apache Tomcat/7.0.23

Seems to me this part is not happening:

org.apache.catalina.core.ApplicationContext log
INFO: ColdFusionStartUpServlet: ColdFusion: Starting application services
org.apache.catalina.core.ApplicationContext log
INFO: ColdFusionStartUpServlet: ColdFusion: VM version = 23.7-b01
org.apache.catalina.core.ApplicationContext log
INFO: CFMxmlServlet: Macromedia Flex Build: 87315.134646
org.apache.coyote.AbstractProtocol start
INFO: Starting ProtocolHandler ["ajp-bio-8012"]
com.adobe.coldfusion.launcher.Launcher run
INFO: Server startup in 27609 ms

Could something else eg Windows Firewall, be blocking CF tomcat?

HTH again, Carl.

[ericbegins]

I think you may be on to something.  I'll get with IT and see if we can't find the culprit.

thanks!

[benalembick]

Hi,

I am experiencing the exact same problem (10 - 20 minutes for CF to start via cmd line & timeout in windows services)

I have the same missing entries in the log also - Did you ever work out what the problem was?

Thanks in advance

Ben

[ericbegins]

I don't remember exactly what we did 3 years ago, but we're actually having the same problem again.  Pretty sure the source of this problem is that the service startup is timing out because our A/V software is taking too long to scan the jar files.  I can see entries indicating that in the Windows Application log