Participating Frequently

Answered

CGI Scope is empty with ColdFusion 2021 on windows 2019

Forum|Forum|4 years ago
January 4, 2022
3 replies
10464 views

I have found that on two of my coldfusion 2021 servers, at random intervals, the CGI scope starts returning an empty struct. When this happens, the only solution seems to be a restart of the ColdFusion service.

there dont seem to be any relevant entries in any of the ColdFusion logs or the Windows Event Viewer.

Has anybody ever experienced this?

This topic has been closed for replies.

Correct answer gabrieldavis321

Hi @gabrieldavis321 ,

Thanks for the update.

Hi BKBK and Charlie,

First I would like to thank you both for your help. I believe we found the root cause and it WAS code. There was a function to scrub passwords from structs so that we could safely create error handling emails with dumps of form, request and CGI and not have the password displayed. Below is from the developer:

I think this was a subtle difference between CF2016 and CF2021 in the way it handles copying structs inside a function. Let me explain a little further…

Looking at the sanitizeStruct function, the first line says:

<cfset var retVal = arguments.sourceStruct/>

In CF2016, the expectation was that this was creating a local copy of the passed struct (in our case the CGI scope). So later in the function when we clear the struct:

<cfset structClear(local.retVal)/>

It was safe because we were clearing the LOCAL copy and not the original struct.

It seems that in CF2021, when we create out local variable, it must not be creating a pointer instead of a local COPY. So later when we clear the struct, in CF2021, it seems to be clearing the original CGI scope.

I think this could possible be solved by changing this line:

<cfset var retVal = arguments.sourceStruct/>

To be this:

<cfset var retVal = Duplicate(arguments.sourceStruct)/>

This would ensure that the local variable is a duplicate of the passed struct instead of a reference to it."

W

w49369461

Participating Frequently

What kind of load balancers are being used infront of these servers? nginx?

Charlie Arehart

Adobe Expert

Blair, I'd take things in a different direction than the other discussion about the web server configuration files.

Assuming that doesn't get you to a solution, let's go back to your first point. You say that "at random intervals, the CGI scope starts returning an empty struct". Tell us more:

First, how are you "seeing" that? Is it that you have an error handler that is dumping the CGI scope, among other things? Or are you viewing this in the debug output of a page?
And when you say it's at random intervals, can you elaborate? Is it happening every day, or are there large gaps between when it happens? And when it does happen, does it stay in that state (empty cgi struct) until a restart?
When it happens, are you confirming that ALL pages in all apps reflect this state of an empty CGI srtuct?

As for further diagnosis:

If it may well be that you're seeing this in an error handler, let's consider that there's a possibility that the problem could be related to whatever error is happening.
And it might be interesting to track if the cgi scope existed at the START of the request, like using a request scope variable that tracks structkeyexists(cgi,"CF_TEMPLATE_PATH"). It may be that that was TRUE at the start but would be FALSE at this time later in the request that you find the struct to be empty. That could be diagnostically significant.
You could ven log if it's TRUE (in this check at the start of the request)

Finally, to answer your question, I have not seen this before myself. It sounds interesting, but as you can see from the above, I am used to being dropped in "behind the fire line" to help put out the fire, even if I've "never been in that forest before". 🙂 I hope the thoughts may be helpful, again if that other line of discussion doesn't pan out.

/Charlie (troubleshooter, carehart. org)

B

Blair22505870i4maAuthor

Participating Frequently

Hi Charlie - Thank you for your response!

as you can see I am still trying a few things that BKBK suggested. One other thing to note is that previously, we had removed the values from "Minimum JVM Heap Size" and "Maximum JVM Heap size" in the Java and JVM settings screen in the CF Administrator in an attempt to have CF/Java automatically manage this.

A few days ago, we reverted this change, adding values back into those fields in the CF Admin. We have not experienced any issues since then but its still possible that is a coincidence.

With that said - let me answer a few of your questions to help give a more complete picture.

1. as I mentioned in my most recent reply to BKBK, we actually have 3 CF servers in this environment. Two of them are load balanced and are experiencing the issues with the CGI scop. The third one is NOT load balanced and is not having any issues with the CGI scope.

2. These servers are used primarily for a third party commercial CMS product based on ColdFusion. Wh the issue surfaces, we find that certain elements on that CMS, start to render incorrectly and throw errors relatign to missing variables which the CMS shoudl always have available. After speakign with the CMS vendor, they told us that those variables are pulled from the CGI scope, and when the issue was happening, we found that if we were to dump the CGI scope in a test file, it woudl dump an empty struct.

3. when I say it is at random intervals, what I mean is that the issue keeps happening - sometimes on one of the two load balanced servers and sometimes on both. Sometimes it happens after a few days, and sometimes it happens after several hours.

4. when this issue does happen, it stays in that state until we restart the CF service. and it does appear that all pages in all apps on the affected server remain in this state until the CF restart.

5. another interestign note is that we also included a cfdump of the request scope in our test file, and when the issue occurs and we load the test file in a browser, both the request scope and the CGI scop dump empty structs.

BKBK

Adobe Expert

The load balancing is all done through a seperate hardware device managed by our Client/Server department, so I dont think there needs to be any special provisions inside of coldFusion to accommodate the load balancing.

In view of the new information, I would now agree with Charlie's point that headers and connectors are unlikely to be the cause of the problem. The new information that sways me is the missing request scope. What if the load balancer did it?

The missing request and CGI scopes suggest to me that, when the issue occurs, the affected ColdFusion server might not be receiving traffic from the load balancer. So I would now include the load balancer in the list of suspects.

Could you

Share the contents of jvm.config ( as Charlie has said, if we know the settings we will be able to rule out possible memory-related causes )
Do a forensic log examination:
- Have a look at server.log and note down the times when this issue forced you to restart.
- Then examine all the other logs, including isapi_redirect.log and the load-balancer logs, to find out which errors occurred just before those times.