ColdFusion 2018 Service Restart Failure

[ah_dc]

Hi,

On one of the ColdFusion 2018 Server on Windows 2019, service doesn't start after server reboot and seeing this following error in event log.

The ColdFusion 2018 Application Server service could not be started within 240 seconds.  Increase the server startup timeout value using E:\ColdFusion2018\cfusion\bin\coldfusionsvc.exe -starttimeout <seconds> "ColdFusion 2018 Application Server".

We have to manually start service.

I don't see anything in ColdFusion logs:

"Information","main","06/01/19","03:04:11","","Starting logging..."

"Information","main","06/01/19","03:04:11","","Starting license..."

"Information","main","06/01/19","03:04:12","","Enterprise Edition enabled"

"Information","main","06/01/19","03:04:12","","Starting crypto..."

"Information","main","06/01/19","03:04:13","","Installed JSafe JCE provider: Version 6.21 Crypto-J 6.2.1, EMC Corporation. JsafeJCE Security Provider (implements RSA, DSA, ECDSA, Diffie-Hellman, ECDH, AES, DES, Triple DES, DESX, RC2, RC4, RC5, PBE, MD2, MD5, RIPEMD160, SHA1, SHA224, SHA256, SHA384, SHA512, HMAC-MD5, HMAC-RIPEMD160, HMAC-SHA1, HMAC-SHA224, HMAC-SHA256, HMAC-SHA384, HMAC-SHA512, HMACDRBG, HASHDRBG, CTRDRBG, FIPS186PRNG, SHA1PRNG, MD5PRNG; RFC 3394, RFC 5649 AES Key Wrap; X.509 CertificateFactory; PKCS12, PKCS15 KeyStore; X.509V1, PKIX, PKIX-SuiteB, PKIX-SuiteBTLS CertPathValidators; X.509V1, PKIX, PKIX-SuiteB, PKIX-SuiteBTLS CertPathBuilders; LDAP, Collection CertStores)"

"Information","main","06/01/19","03:04:13","","Starting security..."

"Information","main","06/01/19","03:04:14","","Starting scheduler..."

"Information","main","06/01/19","03:04:14","","Starting WatchService..."

"Information","main","06/01/19","03:04:14","","Starting debugging..."

"Information","main","06/01/19","03:04:14","","Starting sql..."

"Information","main","06/01/19","03:04:18","","Starting runtime..."

"Information","main","06/01/19","03:04:27","","CORBA Configuration not enabled"

"Information","main","06/01/19","03:04:27","","Starting mail..."

"Information","main","06/01/19","03:04:27","","Starting cron..."

"Information","main","06/01/19","03:04:29","","Starting registry..."

"Information","main","06/01/19","03:04:29","","Starting client..."

"Information","main","06/01/19","03:04:29","","Starting xmlrpc..."

"Information","main","06/01/19","03:04:31","","Starting jaxrs..."

"Information","main","06/01/19","03:04:31","","Starting graphing..."

"Information","main","06/01/19","03:04:32","","Starting solr..."

"Information","main","06/01/19","03:04:32","","Starting archive..."

"Information","main","06/01/19","03:04:32","","Starting document..."

"Information","main","06/01/19","03:04:33","","Starting eventgateway..."

"Information","main","06/01/19","03:04:33","","Starting FlexAssembler..."

"Information","main","06/01/19","03:04:33","","Starting .NET..."

"Information","main","06/01/19","03:04:33","","Starting Monitoring..."

"Information","main","06/01/19","03:04:35","","Starting PDFG..."

"Information","main","06/01/19","03:04:35","","Starting WebSocket..."

"Information","main","06/01/19","03:04:35","","WebSocket server listens on port: 8581"

"Information","main","06/01/19","03:04:35","","ColdFusion started"

"Information","main","06/01/19","03:04:35","","ColdFusion: application services are now available"

Thank you.

[Charlie Arehart]

That's the log from when it DID start, right, not when it did not? If so, then it's not really helpful to the question.

As for why cf did not start after the reboot, I would suspect it was simply that other processes were runnimg at that time which stole resources (cpu, disk, or memory typically), thus preventing cf from having enough to start.

And that would be why later you COULD start it, manually. The issue is not whether YOU start it manually or WINDOWS starts it automatically. It's just a matter of timing.

There's not much you can do, frustrating though this is, and not likely some cf tweak. Instead, you should study what IS using the most resources during a box reboot. Use the builtin Windows tool called Resource Monitor.

In my experience, a common cause for high post-reboot resource uaage is windows updates. Yes, I mean that AFTER the startup, there is STILL considerable work that happens related to update processing AFTER the update and reboot.

If you would say, "well, this happens after EVERY reboot", then go back to my previous point to assess resources used during box startup.

You may have some other service that's set to start automatically. Some may want to point out that Windows services can be set to delay auto start. That only delays things a bit, often not enough to solve this.

Another outside the box alternative could be to make the cf service dependent on some other service having started first. There's no obvious choice for that.

Hope something there is helpful.

If you remains stumped, and we can't solve it here, this something I can help with remotely via consulting, at carehart.org/consulting.

[ah_dc]

Thanks Charlie.  It was strange, CF log states the service was started but the sites where down and in windows services the CF service was not started.

I'll check out whether there any other process might be running during scheduled reboot.

Thank you.

[Dave Watts]

240 seconds is a long time. I don't know how much memory you've allocated to CF, but the larger that amount is, the longer it will take to start. This can take quite some time if you've allocated a lot of memory.

Otherwise, it's probably just conflicting with something else going on at the same time. The way I usually have dealt with that in the past is to find a service that starts pretty late, and make sure CF starts after that. You can do this by editing the registry to specify the "DependOnService" key.

https://support.microsoft.com/en-us/help/193888/how-to-delay-loading-of-specific-services

Dave Watts, Eidolon LLC

[ah_dc]

Thanks Dave. It is using default Minimum JVM 256/Max 1024. I'll have to check whether any back process is running at that time.

[Dave Watts]

That amount of memory usage shouldn't make startup any slower. But honestly I'd just find out from your logs what's the last service to start up, and have CF start up after that, instead of spending a lot of time trying to see what's going on at startup.

Dave Watts, Eidolon LLC

[ah_dc]

Got it. Thanks.

[ah_dc]

Is it a good idea to use windows service recovery options to start service after failure?

[Dave Watts]

It depends. It can be, but if you're having a persistent problem it probably means you'll have repeated failures. And, in general, once CF has started, the service itself will continue running just fine even if it stops doing what you want it to do (responding to web requests). So, you really need some other sort of monitoring to detect that.

Dave Watts, Eidolon LLC

[ah_dc]

Thanks. I'll try the following approach. Looks like something is off on this server as we don't have this issue on another server that is configured exactly like this one.

The ColdFusion 2018 Application Server service could not be started within 240 seconds.  Increase the server startup timeout value using E:\ColdFusion2018\cfusion\bin\coldfusionsvc.exe -starttimeout <seconds> "ColdFusion 2018 Application Server".

[Charlie Arehart]

It may help, as a bandaid. I appreciate you may feel desperate to try anything that might get you by, and it may suffice. But I hope at some point you'll heed my first comment, to be able to understand and hopefully truly resolve the problem.

Let us know how it goes.

[ah_dc]

Thank you Charlie. You're right, we need to fix underlying problem. Will keep you updated.

[BKBK]

What happens when, instead of restarting the ColdFusion service, you stop it first, then start it.