CGI Scope is empty with ColdFusion 2021 on windows 2019

[Blair22505870i4ma]

I have found that on two of my coldfusion 2021 servers, at random intervals, the CGI scope starts returning an empty struct.   When this happens, the only solution seems to be a restart of the ColdFusion service.

there dont seem to be any relevant entries in any of the ColdFusion logs or the Windows Event Viewer.

Has anybody ever experienced this?

[Charlie Arehart]

Gabriel, I'll offer a clarification for bkbk's suggestion, and also a variation (which I think may be critical).

First if it wasn't clear, note that since he proposed using port 8500 (assuming that's the port you use to access the cf admin). That's cf's built in web server. So one benefit of using that is you're NOT using IIS in this case, and that was part of his goal for the test. (Note also that this means when he said to put the test page in the wwwroot, he meant more specially the cfusion/wwwroot--not your site's wwwroot.)

As for his using curl, that was to also rule out possible browser involvement in the problem, and fair enough.

Finally, while you await the next occurrence of things, here's the "variation" which I'd propose: put that test page in its own folder, and in that folder put in an empty application.cfm or cfc (whether within your site's root or that Cf one), and then call that page (even in a regular browser). Doing this removes the impact of any existing application.cfm or cfc you have, which may be where the root of your problem lies. 

And please don't say "there's no application.cfc/cfm in the folder doing the test.cfm page". That not proof enough because, as you may know, if there is one in an ancestor folder up the tree, cf would use that.  This test will take just seconds to setup.

Then make sure the test  "works" first when all's well. Then when things go amiss, run it (perhaps even before trying bkbk's curl suggestion). If THIS test shows the cgi scope working, then its "disappearance" in your other testing is due to your APP and not CF. (And the reason it returns on a restart may have to do as much with the mere execution of some onapplicationstart code.)

If this test I propose works, then the next step will be to see what in your code is contributing to this problem, which may or may not be trivial. But it will affirm most of all that this is not a bug affecting "all of cf", even though it was affecting "your entire app".

Let us know what you think and how it goes. 

[BKBK]

Hi BKBK,

We haven't yet had a chance to try because we have to wait for the issue to happen again, but I am not sure that I follow the logic... we use the same web browser and IIS when checking ... so why would they behave differently when the issue occurs and when we have restarted CF to fix the issue? 

By @gabrieldavis321

Answer: when you restart ColdFusion, a new connector instance is created between CF and IIS.  

In fact, another test would be to restart IIS when the issue occurs. 

This brings up 2 belated questions:

1. What is the update level of your ColdFusion 2021 installation?
2.  Did you recreate the connector?

[gabrieldavis321]

Thanks for the answer!  I never knew that.  Update level is hf202100-4210921.jar on the System Information page of the CF admin.  We did remediate manually the logJ4 jars to 2.17 without taking the CF latest update.

We had built the server using the lockdown tool to connect to IIS.  Then we had to uninstall ASP and re-install ASP because requests to ASP were sent to CF.  That broke the connectors so we manually ran wsconfig.exe to connect sites to IIS.  I hate to think that somehow using the lockdown tool then switching to wsconfig.exe is a root cause

[gabrieldavis321]

Thanks Charlie for your explanation and suggestion.  We will definitely try the CURL stuff next time it happens!

[BKBK]

 Update level is hf202100-4210921.jar on the System Information page of the CF admin.  

By @gabrieldavis321

Something doesn't sound right. Read off "Version" on the System Information page. What is its value? Mine is "2021.0.03.329779".

What is the "Java Version"?

We did remediate manually the logJ4 jars to 2.17 without taking the CF latest update.

Not optimal. The log4j instructions for ColdFusion 2021 require you to be on Update 3.

If you're not on Update 3, you will have to install it to be sure you are actually mitigating the log4j risk.

[gabrieldavis321]

Hi BKBK,

Please see below

Version 2021.0.02.328618

Update Level D:/ColdFusion2021/cfusion/lib/updates/hf202100-4210921.jar

Java Version 11.0.10

-Gabe

[Charlie Arehart]

Gabriel, a few things:

• The hf is a hotfix, a jar that someone on your end added to that cfusion/lib/updates  to fix some bug in update 2. I doubt your issues have anything to do with the cf updates, but it was good to get clarity on the version info
• The autolockdown tool absolutely could be a root cause of some problems... and given what you said, of issues with asp, you or others there may be tweaking things to solve problems, only widening the impacts
• But back to your original problem, and your reply to my last comment, I did not propose you run the curl command. That was bkbk. I realize it can be be hard to keep track of "who had the ball on previous plays". 
• Most important, my last suggestion still stands and I would STRONGLY recommend you take a minute to do the test I posed. If it DOES show the cgi vars when your app fails to, it would tell us it's not ALL of Cf that is then "losing cgi vars", which would be VERY important diagnostically. 
• Additionally, if you did create my test folder within your site's web root, then if my code works when your app's gets the problem, this would ALSO prove that it's NOT about the web server connector, as they'd use the same one. Instead, it would point a finger strongly at your app's code, and especially code in or called from your application.cfc or cfm, as my test would be preventing use of that (since the test I proposed would have its OWN empty application.cfm that would be used instead).

[gabrieldavis321]

Hi Charlie,

You are right, I replied too soon without fully reading.  I have set up the folder under cfusion/wwwroot (we use a different path for website) with the application.cfm/dump and made sure that it was accessable https://localhost:xxxx/cgiTest/  and at this time it correctly dumps cgi.  We are scheduling nightly restarts as a workaround so it may take a bit before the issue arises again.

I think the hotfix was for numberformatting issues.  The re-install of ASP and IIS connectors was the only major tweak other than replacing the logj4 jars and the jvm arg.  As previously mentioned archives from FusionReactor do not show any spike in CPU or memory... and nothing in CF logs.

Also restart of IIS does NOT solve issue

I do appreciate your help!

Thanks,

Gabe

[Charlie Arehart]

Gabe, OK on all that.

As for my test code, I'll note that to test things more easily, you could put a check for when the cgi scope is empty, either in your application.cfc/cfm or where your code is failing (in case it happens there and NOT in the application.cfc/cfm). And there have a cfhttp call to my page, checking ITS results for whether it shows the cgi scope being empty.

As for that hf jar, I see now that yes that was for the *date* format change in cf 20221, and that fix (from Dec 2020) allowed you to then add a JVM.arg to set it to pre-cf2021 behavior. Well, that update was included in update 1, and you're on update 2, so you don't need that jar. (You do still need the jvm arg. THAT is what reverts the behavior.) This will have nothing to do with your cgi vars.  Just sharing for clarification. 

[BKBK]

Hi BKBK,

Please see below

Version 2021.0.02.328618

Update Level D:/ColdFusion2021/cfusion/lib/updates/hf202100-4210921.jar

By @gabrieldavis321

Thanks for the information. It contains a possible clue.

You're on Update 2. This implies that you have applied the log4j mitigation steps to Update 2. Whereas, as you can see in my earlier post, the log4j instructions for ColdFusion 2021 require you to be on Update 3.

[gabrieldavis321]

Hi Charlie and BKBK,

The cgi scope was empty both on IIS regular website and when bypassing IIS and using in the built in server that we serve CF admin with (we tested using the technique Charlie suggested by putting the test folder under cfusion/wwwrooot).  So maybe not an issue with IIS connector?

Thanks,

Gabe

[Charlie Arehart]

Ok on that. (I've said from the outset that I didn't think it was a connector issue myself.)

So you're saying that when this problem happens, it happens to both, right? (You'd said earlier about my test code that, "at this time it correctly dumps cgi"). 

That would indeed confirm for sure it's not about the iis connector, if you're confirming that you're calling this test code via the cf builtin web server port instead.

And to make 100% sure there's not an influence of any other application.cfc or cfm, you confirmed that you put the test code is in its own folder, with its own empty application cfc/cfm, right? 

As such, it seems you're proving that once this problem happens, it doesn't matter what code you call or how you call it. All cf pages have an empty cgi struct, which I'll say I've never heard of before. (There are not other pages that DO have the cgi scope at this time, right? ) 

Indeed, this is so odd because the cgi scope is not writable (by cf code). It may seem there must be a bug, so you should report it as such at tracker.adobe.com. Please add a comment here with a link to that. 

I can't help but think there's another explanation for things and we may all smack our heads if that's so. But going on what you've said so far, it seems a bug report is the next step. Do point the ticket at this thread, to give interested readers there this context.

[gabrieldavis321]

Hi Charlie,

Yes it happens to both IIS and built in web server.  Yes, I had the test in its own folder with its own application.cfm.  As I mentioned you can at least in theory "write" to cgi scope by calling structClear which is at least one way to reproduce the problem.

I submitted bug: CF-4212711

Thanks,

Gabe

[BKBK]

Hi Gabe,

Thanks for your updates. 

I have voted for the bug report.

Before we move on, there is one particular point I raised. You informed the forum:

We did remediate manually the logJ4 jars to 2.17 without taking the CF latest update.

Gabe

Version 2021.0.02.328618

However Adobe's advice is to install log4j 2.17 on CF2021 Update 3. It might or might not have anything to do with the problem, but was worth looking into, I think. If only to rule it out.

I have not been able to find any documentation on the changes that Update 3 carries out. So I can't say for sure. However, I found one difference between the installation of Updates 2 and 3.

The installation of Update 2 requires you to recreate the connector, whereas the installation of Update 3 does not. Does this mean anything? I don't know. It's for the ColdFusion Team to say - it's now in their hands.

[BKBK]

While we wait for Adobe, here is another thought. We know that the issue happens at random and only on a selection of servers. 

So, the Application.cfc/Application.cfm of the applications on those servers is a suspect. Could you share the contents of the relevant Application.cfc/Application.cfm files? If you require privacy, then send them by private-message. 

I should add that Charlie did suggest looking into the Application file. But we didn't follow up on that.

[Charlie Arehart]

Actually, bkbk, the issue of application.cfc is indeed moot, according to Gabe.

He said he'd followed my suggestion to create a test folder with a test page (to dump cgi) and a blank application.cfc. That was to  rule out ineritance of any application cfc or cfm in parent folders.  And he'd said the cgi scope WAS empty there so when the issue arises).

At least these are what Gabe has said. I hope and assume they were accurate. 

[BKBK]

Actually, bkbk, the issue of application.cfc is indeed moot, according to Gabe.

He said he'd followed my suggestion to create a test folder with a test page (to dump cgi) and a blank application.cfc. That was to  rule out ineritance of any application cfc or cfm in parent folders.  And he'd said the cgi scope WAS empty there so when the issue arises).

At least these are what Gabe has said. I hope and assume they were accurate. 

By @Charlie Arehart

Charlie, my last post was actually in response to that same point. I had understood that the test resulted in a non-empty CGI scope! 

I have set up the folder under cfusion/wwwroot (we use a different path for website) with the application.cfm/dump and made sure that it was accessable https://localhost:xxxx/cgiTest/  and at this time it correctly dumps cgi.

Gabe

Unless of course I missed something.

[Charlie Arehart]

Yes, bkbk. You missed that he said it WAS empty even this test folder. From yesterday :

The cgi scope was empty both on IIS regular website and when bypassing IIS and using in the built in server that we serve CF admin with (we tested using the technique Charlie suggested by putting the test folder under cfusion/wwwrooot). 

It's indeed a crazy long thread with lots of comments, so challenging to see it all, let alone keep track of it all in our heads. 🙂 

[BKBK]

Yes, bkbk. You missed that he said it WAS empty even this test folder. From yesterday :

The cgi scope was empty both on IIS regular website and when bypassing IIS and using in the built in server that we serve CF admin with (we tested using the technique Charlie suggested by putting the test folder under cfusion/wwwrooot). 

It's indeed a crazy long thread with lots of comments, so challenging to see it all, let alone keep track of it all in our heads. 🙂 

By @Charlie Arehart

Ah! Thanks!

[gabrieldavis321]

Hi BKBK,

The code we control is the same (we deploy from a repos) on both the "good" and "bad" servers so there will be no difference in application.cfm/cfc on any server... what is NOT the same is the the CF CMS code from the vender but it is encrypted so we can't check anything.  The issue only happens on the CMS  Read only servers and never on the CMS authoring server.

Gabe

[gabrieldavis321]

and just to follow up... the authoring server is not publicly available so it never has the load the Read-only servers have... just a few authors at a time

[Charlie Arehart]

Gave, I just looked at your tracker ticket and see that you say there that the structcount for cgi when this happens is 46, so clearly it's not EMPTY. That seems very important, and I want to explore it. (I know you said also that your cms vendor code was finding SOME variable to be empty.) 

First, I see now that you had mentioned this over a week ago here, but it was kind of buried in a code fragment and output, so I'd missed it and no further discussion of it happened.

So here's where I think we should turn next: what do you see for structkeylist(cgi)? That lists what variables ARE set? And if you the just output cgi.whatever (something listed there)? How about ones that Cf always sets, like cf_template_path?

If those are not all empty/blank, then clearly the issue is in the cfdump. That's still a problem, sure.

But then the REAL question is what cgi var was proving to be empty in your cms vendor code? And is that var not shown in the structkeylist?

Finally, if indeed any of them you dump specifically are NOT empty, then you could output them all one at a time by using cfloop struct="#cgi#", and then output each item one at a time. What do you see?

But start with the far simpler suggestions I made at the top of my comment. That will show if it's worth doing this last one, and should take just a minute. 

[BKBK]

Assuming you haven't yet done the curl test, here is an even simpler test (to bypass the web server and browser):

<!--- Afterwards, view the file CGI_scope.html. It is in the current directory. --->
<cfdump var="#cgi#" label="CGI Scope" format="html" output="#getDirectoryFromPath(getCurrentTemplatePath())#CGI_scope.html">
		

[gabrieldavis321]

Hi Charlie,

structKeyList doesn't work... I can't remember if it throws an error or produces an empty string/array but no way to see the keys (my first thought too was to loop over structKeyList).  The dumps are just diagnostic ... the missing cgi code causes errors bc the code expects them to be populated.

Gabe

[gabrieldavis321]

but strangely StructIsEmpty(cgi) does return true when the issue arises even though strutcount is 46.  I had orginally written code to alert us when this happens using structcount and was surprised when it didn't alert us and found out that structcount was 46... so I change the alert to look at StructIsEmpty and the alert works