Per-application datasource not found after service restart.

Report · Mar 10, 2022

I'm running into a rather strange issue and I don't know how to debug it. I'm using a per-application datasource feature in the `Application.cfc`:

this.datasources[ "bennadel" ] = { .... }

And, when I restart the ColdFusion service is sometimes says that the datasource cannot be found. This "state" seems to hold for the entire life of the application, and all requests coming into the application are breaking. But then, if I restart the ColdFusion service again, it works fine (probably). I'm kind of stumped here - not sure how to even go about debugging this.

Report · Mar 11, 2022

Ok, so I just tried one more thing. I restarted the ColdFusion service, and the datasource bombed-out. Then, all I did was update a value in the `this.datasources` - I changed `interval: 420` to `interval: 421`, re-uploaded my `Application.cfc` and the site immediately came online.

To be clear, it has nothing to do with `interval`. I tried it a few more times, changing _different_ properties, and everytime I "touch" the `this.datasources` structure, the site immediately bootstraps and comes online. It's like an invalid version of the datasource is getting cached and then ColdFusion re-caches it if any of the keys change.

By @bennadel

The caching you mention isn't far from what I think is going on. I think the following happens:

ColdFusion instantiates the class corresponding to Application.cfc;
When it does, it tries to create a connection to the database. That is necessary to obtain a value for the datasources;
Your problem may arise because something is going wrong or being delayed during step 2. Sorry for repeating the question:

Which attributes do you use to define this.datasources?

I was thinking along these lines when I suggested the debug-code:
```
<cfscript>
	try {
		writedump(new Application());
	} 
	catch (any e) {
		writedump(var=e, label="Oops!");
	}
</cfscript>
```
ColdFusion will go right ahead and instantiate Application.cfc and proceed to onApplicationStart(), even if there is a problem at step 2. Again, this points the finger at the datasource connection attributes.

Strange thing is, I cannot find any official definition of the this.datasources attributes anywhere! (Except those of Lucee)

Report · Mar 11, 2022

I concur with your last point, bkbk. The issue has been raised over the years. Mark, might you be able to get some movement on that? The doc on app vars only mentions datasources as an option, not any detail on its keys.

If that page may not be the right place, maybe one of its own would be best. And then the capability should be mentioned (and that page linked to) from https://helpx.adobe.com/coldfusion/kb/create-datasource-coldfusion.html, which only mentions the cf admin and admin api as options, which is unfortunate.

/Charlie (troubleshooter, carehart.org)

Report · Mar 11, 2022

I did try the `writeDump( new Application() )` test and wasn't able to get anything to error. But, later today, I'll try to wrap the `this.datasources` config in a try/catch and see if I can find any error information.

Report · Mar 11, 2022

You might have received mail saying some answers have been marked correct. My bad.

Of late something's gone awry with my keyboard and mouse. There is occasionally a jump to a different button or link just before I click. Sorry about that.

Report · Mar 12, 2022

Ok, I've spent the last hour-and-a-half restarting my live site and trying to add debugging. What a pain. It only actually breaks like 1 out of every 10 CF restarts. So, here's what I did:

First, I wrapped my `this.datasources` definition in a try/catch and logged any error using `cflog()`. There were no errors to be found in this block of code, even when the application was having trouble bootstrapping.

I added this to the top of my `onError()` handler:

writeDump( getApplicationMetadata().datasource );
writeDump( getApplicationMetadata().datasources.keyList() );
writeDump( exception );
abort;

And, when the application got into a bad state, here's what it showed me:

Note that both the `this.datasource` and `this.datasources` properties are correctly defined within `getApplicationMetadata()`.

Just bananas!!

Report · Mar 12, 2022

Ok, I think I may have found a super hacky solution. Earier in this thread, I discovered that if I simply "touch" any value in the `this.datastructures`, some internal magical cache appears to get flushed and the application will bootstrap properly. This got me thinking about possibly doing this "touch" programmatically. What I've done now is include a timestamp when my config is cached in memory:

private struct function getConfigSettings( boolean useCacheConfig = true ) {

	var configName = "appConfig_#this.name#";

	if ( useCacheConfig && server.keyExists( configName ) ) {

		return( server[ configName ] );

	}

	var config = server[ configName ] = deserializeJson( PATH_TO_CONFIG_FILE );

	// HACK: I'm having trouble with my "this.datasources" configuration bombing-out
	// on server-start for reasons that I don't understand. Through trial-and-error,
	// I've discovered that "touching" that structure fixes this issue. As such, I'm
	// going to see if this date/time-stamp can act as a "touch" on that value.
	config.loadedAt = getTickCount();

	return( config );

}

As you can see, I'm injecting a `.loadedAt` property when this is cached in the `server` scope. Then, in my `this.datasources` struct, I now have:

this.datasources = {
	
	"bennadel": {
		username: this.config.dsn.username,
		password: this.config.dsn.password,
		driver: "MySQL",
		class: "com.mysql.jdbc.Driver",
		
		// ...... truncated .......

		// HACK: For reasons that I don't understand at all (and what is looking very
		// much like a bug in ColdFusion), the datasources configuration occasionally
		// bombs-out when the service-start up for the first time. But, I've
		// discovered that simply "touching" this structure (ie, changing any property
		// in it) appears to fix the problem (maybe by flushing some sort of cache).
		// As such, I'm going to see if including a timestamp will act as a "touch".
		// This timestamp is when the config object was cached. And, since it's cache
		// whenever the onApplicationStart() method runs, if the application fails to
		// bootstrap, the next request will get a newer timestamp.
		_loaded_at_: this.config.loadedAt
	}
};
this.datasource = "bennadel";

Once I had this code in place, I started restarting the ColdFusion service. After about 7 restarts, I finally was able to reproduce the "datasource cannot be found error." However, unlike in previous tests, where this bad state _persisted_ indefinitely, I found that the next page request lead to a working application. I'm presuming this is befcause the next request triggered a new `onApplicationStart()` call, which in turn, reloaded the config, which in turn lead to a new `config.loadedAt` date/time stamp. It seems this new timestamp was enough to "touch" the structure, which cleared whatever crazy caching is going on.

This is nuts!!! But, it seems to work 🤞

Report · Mar 13, 2022

This is _mostly_ working. The application bootstrapps now; however, I am sometimes seeing an issue in the bootstrapping process where ColdFusion complains that the connection pool isn't defined and I get a Null Pointer Exception during the SQL execution.

Still working at it.

Report · Mar 13, 2022

I agree with you: this is nuts! Which could be an indication that we've been barking up the wrong tree. Your most recent findings - "_mostly_ working", then failing - imply that.

What if all the caching behaviour is just the expected application-scoped caching on-application-start? Then any error or inconsistency that occurs prior to or in onApplicationStart() will persist till the application is restarted.

A review is in order. So, I am going back to the very beginning.

The error message "Datasource bennadel could not be found" tells us that the application knows bennadel. This in turn implies that, if there was a problem it would probably be with the setting this.datasources.

With that in mind, let's restart with no preconceptions. In other words, let us take for granted that ColdFusion will merrily accept the this.datasources/this.datasource combination:

/* Application.cfc*/
/* The datasource "bennadel" is only defined here. That is, 
it is not defined in the ColdFusion Administrator. */

this.datasources["bennadel"]={ ... };
this.datasource="bennadel";

Then this will mean that the cause of the problems is simple: wrong or incompatible datasource settings.

To test this hypothesis, proceed as follows:

Stop ColdFusion.
(If you haven't yet done so, import a recent MySQL Jar, as follows)
Download the "Platform Independent" ZIP file from https://dev.mysql.com/downloads/connector/j/ . The latest version is the file mysql-connector-java-8.0.28.zip.
Unzip it.
Copy the file mysql-connector-java-8.0.28.jar to your ColdFusion lib directory. If there is any existing file in the lib directory whose name is of the form mysql-connector-java-x.x.x.jar then delete it.

Put the following settings in Application.cfc, replacing my settings with yours

/* Note: MySQL5, not MySQL */
this.datasources[ "bennadel" ] = {
        	 username: "root",
		      password: "BkBk123",
		       driver: "MySQL5",
		       class: "com.mysql.jdbc.Driver",
		       name:"bkbk_cf_mysql_db",
		       url: "jdbc:mysql://127.0.0.1:3306/bkbk_cf_mysql_db?serverTimezone=Europe/London"
        };
        this.datasource="bennadel";

Restart ColdFusion.
To test, run an arbitary CFM page.

Any joy?

Report · Mar 13, 2022

Ok, I think I finally came up with a solution that works! I got rid of the whole time-stamp / cache invalidation issue. Now, I'm setting an `isBootstrapped` flagged at the end of the `onApplicationStart()` method. And then, in the `onError()` method, if that flag is not set - meaning, an error was thrown before the application could be bootstrapped, I then call `applicationStop()`. The next request appears to then re-start the ColdFusion application, bringing it into a valid state. An appreciated version:

component 
	output = false
	hint = "I define the application settings and event handlers."
	{

	// Define application settings.
	this.name = "WwwBenNadelCom";
	this.applicationTimeout = createTimeSpan( 2, 0, 0, 0 );
	this.sessionManagement =  false;
	this.setClientCookies = false;

	this.config = getConfigSettings();

	// Define the datasources and default datasource.
	// --
	this.datasources = {
		"bennadel": {
			username: this.config.dsn.username,
			password: this.config.dsn.password,
			driver: "MySQL",
			class: "com.mysql.jdbc.Driver",
			
			// .... nothing special here ....
		}
	};
	this.datasource = "bennadel";

	// ---
	// LIFE-CYCLE METHODS.
	// ---

	/**
	* I get called once when the application is being bootstrapped. This method is
	* inherently single-threaded by the ColdFusion application server.
	*/
	public void function onApplicationStart() {

		// .... truncated initialization for demo ....
		// .... truncated initialization for demo ....
		// .... truncated initialization for demo ....

		// As the very last step in the initialization process, we want to flag that the
		// application has been fully bootstrapped. This way, we can test the state of the
		// application in the onError() event handler.
		application.isBootstrapped = true;

	}


	/**
	* I handle uncaught errors within the application.
	*/
	public void function onError( required any exception ) {

		// If the bootstrapping flag is null, it means that the application failed to
		// fully initialize. However, we can't be sure where in the process the error
		// occurred, so we want to just stop the application and let the next inbound
		// request re-trigger the application start-up.
		if ( isNull( application.isBootstrapped ) ) {

			cfheader( statusCode = 503, statusText = "Service Unavailable" );
			writeOutput( "<h1> Service Unavailable </h1>" );
			writeOutput( "<p> Please try back in a few minutes. </p>" );

			try {

				applicationStop();

			} catch ( any stopError ) {

				// Swallow error, let next request start application.

			}

			return;

		}

		// .... truncated for demo ....

	}

}

The next request to the app then seems to boot the app into a valid state.

I'm still convinced that this is a ColdFusion bug. In order to test this, I had to restart my ColdFusion service about 15-times before it actually started in an invalid state. And, I had a maintenance page up, so I was the only one making requests against the app (the maintenance page aborts before the app even has a name defined).

Report · Mar 14, 2022

@bennadel , I now believe there is something else, in your local environment, that is causing the problem. I say this because I have been unable to reproduce the issue. In fact, when I use the setup you've suggested, everything works as expected.:)

I am on ColdFusion 2021 Update 3. The steps I followed are:

Import the MySQL file mysql-connector-java-8.0.28.jar into the ColdFusion lib directory (in the exact manner I explained in my last post);

Create a directory and place the following 2 test files in it:
Application.cfc

component {
    this.name = "BennadelDatasourceTest";
	this.applicationTimeout = createTimeSpan( 2, 0, 0, 0 );
	this.sessionManagement =  false;
	this.setClientCookies = false;
    this.datasources[ "bennadel" ] = {
		 username: "root",
		 password: "gY6PMh7R",
		 driver: "MySQL5",
		 class: "com.mysql.jdbc.Driver",
		 name:"cfmx_db",
		 url: "jdbc:mysql://127.0.0.1:3306/cfmx_db?serverTimezone=Europe/London"
    };
    this.datasource="bennadel";

    boolean function onApplicationStart() {
    	try {
	    	var sql="select *
				from birds
				where id <= :maxBirdId";
				
			// NB: No datasource explicitly set
			var qOptions={};
			
			var qParams={maxBirdId:{value:9}};	
			
			var testQuery=queryExecute(sql, qParams, qOptions);
			
			// Store in application scope so we can dump it in a CFM page
			application.testQuery=testQuery;
			
			// Record a dump of the query and the this-scope for review			
			writedump(var="#testQuery#", format="html", output="C:\ColdFusion2021\cfusion\logs\query_without_ds_attribute.html");
			writedump(var="#this#", format="html", output="C:\ColdFusion2021\cfusion\logs\application.this.scope.html");
			
			// Implies the application may start
	        return true;
       } 
       catch (any e) {
       		// Record a dump of the error for review	
       		writedump(var="#e#", format="html", output="C:\ColdFusion2021\cfusion\logs\error_in_onApplicationStart.html");
       		
       		// Means the application may not start
       		return false;
       }
    }

 }

testpage.cfm

<cfscript>
    writedump(application.testQuery);	
</cfscript>

When I run testpage.cfm, I get an immediate result, as expected, without any error. I also get, as expected:
* a dump of the query of the list of birds whose ID is 9 or less;
* the 2 HTML files that log the dumps of the query and of Application.cfc's this-scope (attached here with TXT extension. To view them, replace the TXT with HTML extension )

Report · Mar 14, 2022

Ok, I think I finally came up with a solution that works! I got rid of the whole time-stamp / cache invalidation issue. Now, I'm setting an `isBootstrapped` flagged at the end of the `onApplicationStart()` method. And then, in the `onError()` method, if that flag is not set - meaning, an error was thrown before the application could be bootstrapped, I then call `applicationStop()`. The next request appears to then re-start the ColdFusion application, bringing it into a valid state.

By @bennadel

I would differ and not call that a solution. Best-practice says, if there is an error, we will have to stop and resolve it before carrying on. In any case, I wouldn't dare take such a setup to production.

Report · Mar 14, 2022

... An appreciated version:

component 
	output = false
	hint = "I define the application settings and event handlers."
	{

	// Define application settings.
	this.name = "WwwBenNadelCom";
	this.applicationTimeout = createTimeSpan( 2, 0, 0, 0 );
	this.sessionManagement =  false;
	this.setClientCookies = false;

	this.config = getConfigSettings();

	// Define the datasources and default datasource.
	// --
	this.datasources = {
		"bennadel": {
			username: this.config.dsn.username,
			password: this.config.dsn.password,
			driver: "MySQL",
			class: "com.mysql.jdbc.Driver",
			
			// .... nothing special here ....
		}
	};
	this.datasource = "bennadel";

	// ---
	// LIFE-CYCLE METHODS.
	// ---

	/**
	* I get called once when the application is being bootstrapped. This method is
	* inherently single-threaded by the ColdFusion application server.
	*/
	public void function onApplicationStart() {

		// .... truncated initialization for demo ....
		// .... truncated initialization for demo ....
		// .... truncated initialization for demo ....

		// As the very last step in the initialization process, we want to flag that the
		// application has been fully bootstrapped. This way, we can test the state of the
                ...
          }
}

By @bennadel

To me, that only raises questions. For example:

Might an attribute within "//.... nothing special here ...." or " .... truncated initialization for demo ...." actually be the cause of the problem?
Here onApplicationStart's returntype is void. So can we really be sure the application has started?
Why do you conclude at the point where
```
application.isBootstrapped = true;
```
that the application has successfully started?

In my opinion, as long as we're within onApplicationStart(), the application hasn't started. The application will only have started after:
- onApplicationStart() returns;
- onApplicationStart() returns True (where we want to be quite sure);

Report · Mar 14, 2022

The "nothing special here" part is all static - nothing there is dynamic, nothing is based on env variables. It's the same on every single request.

As far as returning `true` from the `onApplicationStart()`, that is documented as being optional. From the Adobe site:

> A Boolean value: True if the application startup code ran successfully; False, otherwise. You do not need to explicitly return a True value if you omit the cffunction tag returntype attribute.

If I need to return `true` in order for datasources to work, then this would also be a bug in the ColdFusion runtime.

As far as whey I consider the `isBootstrapped` flag to indicate that the application is bootstrapped, I think we're just splitting semantic hairs. What I mean is that my application startup logic has all executed without error. And since I'm preloading data in the startup logic, it means that the datasource worked, which is really all I want to validate here.

If I get to the `onError()` event-handler and the flag isn't set, it means that something went wrong during my startup logic, and the application is not in a "known good state" (again, from my logic perspective, not necessarily from CF's perspective). I could check to see if the `eventName` argument in the `onError()` method signature is "onApplicationStart" ... but that seems less explicit to my intention.

As far as not doing something like this in production, I suppose it depends on how your production works. For example, if you were deploying with Docker containers, none of this would even matter since the "healthcheck" on the container would never pass and the container would be killed. In my case, I'm just FTPing files to my VPS, so there's nothing orchestrating the health of the server except me 😄

Report · Mar 14, 2022

As far as returning `true` from the `onApplicationStart()`, that is documented as being optional. From the Adobe site:

> A Boolean value: True if the application startup code ran successfully; False, otherwise. You do not need to explicitly return a True value if you omit the cffunction tag returntype attribute.

If I need to return `true` in order for datasources to work, then this would also be a bug in the ColdFusion runtime.

By @bennadel

I have to clarify. I am not suggesting that onApplicationStart needs to return true for the datasources to work. A void returntype is fine.

What I am saying is that, with void returntype, there is an inconsistency implicit in your setup.

Let's take it from the top. We're in

public void function onApplicationStart() {
// an error occurs
}

Presumably, when the error occurs the application has not yet started.

As a result of the error, onError() is triggered.

Within onError(), the function applicationStop() is called.

So we are calling a stop on an application that hasn't started.

I consider that to be somewhat inconsistent.

Report · Mar 14, 2022

There's no doubt that it's odd !! Honestly, I don't quite understanding what is happening at the low-level. All I can tell you is that it works. Meaning, when the `CFQuery` tag throws an error in the `onApplicationStart()` when the datasource isn't defined, then calling `applicationStop()` in the `onError()` seems to fix the issue (or, at least allows the next request to re-start the application).

This whole thing is really frustrating. Especially since it's to hard to reproduce. I had to restart CF like 15-times in a row before it finally started in a wonky state. Then, I had to edit the `Application.cfc` file in production (adding in the `applicationStop()` stuff) to see if it worked.

Before I put the `applicationStop()` line in place, I made some other non-important updates to the `Application.cfc` to make sure that "touching the file" wasn't what was fixing it. But, it wasn't. Editing the App.cfc didn't change the corrupted state of the app until I added in the `applicationStop()` line. Then it fixed.

If this is also a bug - that `applicationStop()` works at this point - then it's like I'm using one bug to cancel out another bug 😆

Report · Mar 14, 2022

There's no doubt that it's odd !! Honestly, I don't quite understanding what is happening at the low-level. All I can tell you is that it works. Meaning, when the `CFQuery` tag throws an error in the `onApplicationStart()` when the datasource isn't defined, then calling `applicationStop()` in the `onError()` seems to fix the issue (or, at least allows the next request to re-start the application).

By @bennadel

Fair enough.

Here's a thought:

What if, on your system, Application.cfc's this attributes are loaded into memory in the "wrong" order? By which I mean, in an order different from the order in which they are declared. For example, in alphabetical order.
If so, then this.datasource will be encountered at runtime before this.datasources. ColdFusion would then be unaware of the dependency of this.datasource on this.datasources, hence the problem.

Report · Mar 14, 2022

Yet another idea, following on my last post:

What if you defined this.datasources in ApplicationBase.cfc and this.datasource in Application.cfc, where ApplicationBase.cfc is the parent CFC and Application.cfc extends ApplicationBase.cfc.
Hopefully, that will force ColdFusion to instantiate this.datasources before this.datasource.

Report · Mar 14, 2022

Hmmmm, that's an interesting thought. Let me think on that one.

Report · Mar 18, 2022

Good news! The test case I suggested does in fact work. 🙂

The set-up:

this.datasources is initialized in the base/parent application file, ApplicationBase.cfc;
this.datasource is initialized in the child application file, Application.cfc, which extends ApplicationBase.cfc.

The test files (placed within the same directory):

ApplicationBase.cfc

component {
    this.name = "BennadelDatasourceTest";
	this.applicationTimeout = createTimeSpan( 2, 0, 0, 0 );
	this.sessionManagement =  false;
	this.setClientCookies = false;
    this.datasources[ "bennadel" ] = {
		 username: "root",
		 password: "gY6PMh7R",
		 driver: "MySQL",
		 class: "com.mysql.jdbc.Driver",
		 name:"cfmx_db",
		 url: "jdbc:mysql://127.0.0.1:3306/cfmx_db?serverTimezone=Europe/London"
    };
  
    boolean function onApplicationStart() {
    	return true;
    }

 }

Application.cfc

component extends="ApplicationBase" {

    this.datasource="bennadel";

    boolean function onApplicationStart() {
    	try {
	    	var sql="select *
				from birds
				where id <= :maxBirdId";
				
			// NB: No datasource explicitly set
			var qOptions={};
			
			var qParams={maxBirdId:{value:10}};	
					
			/* onApplicationStart runs only once. So, if CFM pages require 
			access to the query, you will have to store it in application scope */
			application.testQuery=queryExecute(sql, qParams, qOptions);
			
			// Dump the query and the this-scope to file for review			
			writedump(var="#application.testQuery#", format="html", output="C:\ColdFusion2021\cfusion\logs\query_without_explicit_ds_attribute.html");
			writedump(var="#this#", format="html", output="C:\ColdFusion2021\cfusion\logs\application.this.scope.html");
			
			// Implies the application may start
	        return true;
       } 
       catch (any e) {
       		// Record a dump of the error for review	
       		writedump(var="#e#", format="html", output="C:\ColdFusion2021\cfusion\logs\error_in_onApplicationStart.html");
       		
       		// Means the application may not start
       		return false;
       }
    }

 }

testpage.cfm

<cfscript>
	/* Dump the query evaluated in onApplicationStart */
    writedump(application.testQuery);	
</cfscript>