Highlighted

Coldfusion errors with undefined element, dump during error processing shows element defined

Explorer ,
May 03, 2016

Copy link to clipboard

Copied

Ok, so I've got a strange problem that I've been beating my head against for over a week now.  I'm holding out a little hope that maybe someone here has seen something like this and has a solution or at least can point me in a direction to make some sense of it.


We are running an enterprise application on CF9 currently.  Recently we made a large update with some advancements and in part of the advancements I'm using some request scoped variables.

Said request variable is a structure that is set in Application.cfc OnRequestStart.  Randomly since the changes have been put on our production servers we get an error email (we have an error handler defined to capture and email errors when they occur), stating that key.subkey is undefined in request scoped variable.  What makes this so bizarre is in the email from the error handler (that is processed in the same request) shows the key.subkey existing in the request scoped variable in a <cfdump var="#request#"> in the email.  How could it possibly be erroring on a part of the page that runs after OnRequestStart, but before OnError, saying that this structure element doesn't exist yet it does exist during the error handler caused by it.  The request scoped variable btw is not defined or set anywhere else outside of the Application.cfc.

I should also mention that other keys in the structure are referenced in the request prior to the point of failure without erroring, some in the same function as the point of error even.  It's almost as if the structure is not fully defined yet while it's processing the page request, but it is fully defined by the time it dumps it in the error email....

This problem is really baffling me, and it makes it worse that I can't find any way to reproduce it on our testing servers.

Views

3.7K

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more

Coldfusion errors with undefined element, dump during error processing shows element defined

Explorer ,
May 03, 2016

Copy link to clipboard

Copied

Ok, so I've got a strange problem that I've been beating my head against for over a week now.  I'm holding out a little hope that maybe someone here has seen something like this and has a solution or at least can point me in a direction to make some sense of it.


We are running an enterprise application on CF9 currently.  Recently we made a large update with some advancements and in part of the advancements I'm using some request scoped variables.

Said request variable is a structure that is set in Application.cfc OnRequestStart.  Randomly since the changes have been put on our production servers we get an error email (we have an error handler defined to capture and email errors when they occur), stating that key.subkey is undefined in request scoped variable.  What makes this so bizarre is in the email from the error handler (that is processed in the same request) shows the key.subkey existing in the request scoped variable in a <cfdump var="#request#"> in the email.  How could it possibly be erroring on a part of the page that runs after OnRequestStart, but before OnError, saying that this structure element doesn't exist yet it does exist during the error handler caused by it.  The request scoped variable btw is not defined or set anywhere else outside of the Application.cfc.

I should also mention that other keys in the structure are referenced in the request prior to the point of failure without erroring, some in the same function as the point of error even.  It's almost as if the structure is not fully defined yet while it's processing the page request, but it is fully defined by the time it dumps it in the error email....

This problem is really baffling me, and it makes it worse that I can't find any way to reproduce it on our testing servers.

Views

3.7K

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
May 03, 2016 0
Most Valuable Participant ,
May 04, 2016

Copy link to clipboard

Copied

Is the request.key.subkey variable perhaps empty (undefined, null)?  That will result in the struct key showing in a dump but not be able to be referenced.  You might have to wrap that code in a <cfif StructKeyExists( request.key, "subkey" )> to check.

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Reply
Loading...
May 04, 2016 0
Advocate ,
May 04, 2016

Copy link to clipboard

Copied

Steven6282 wrote:

This problem is really baffling me, and it makes it worse that I can't find any way to reproduce it on our testing servers.

Can you reproduce it in a separate test application on the production server?

Cheers

Eddie

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Reply
Loading...
May 04, 2016 0
Most Valuable Participant ,
May 04, 2016

Copy link to clipboard

Copied

Also, a screenshot of the <cfdump> (with any sensitive information redacted/blurred) would be helpful.

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Reply
Loading...
May 04, 2016 0
Explorer ,
May 04, 2016

Copy link to clipboard

Copied

The keys are not empty. They usually have numbers in them in the dumped data.  It depends on exactly which spot the error randomly happens in as to what data is in the struct.  Occasionally there is one with an empty string, but I have verified that it is an empty string, not an undefined value.

No, sorry I should have been more specific and said I can't even reproduce it on the production server personally.  I've tried doing a lot of different things thinking maybe our users were doing something really weird to cause it to happen, but no matter how many times I refresh or try to load the page a different way I can't find a way to reproduce it.  We have over 4000 users on the production server and over 20,000 requests to the pages that can generate the error per day.  But the error only happens maybe 50 to 100 times a day, so it's a rare hard to reproduce error and obviously doesn't happen every single request.

Unfortunately I can't practically screenshot any of the data and share it on the internet.  This is a health record application so I would have to redact nearly everything in the dump to share a screenshot without violating HIPAA laws.

I can give you a little more details on the structure with dummy data though.  Essentially the point of this request scoped variable is to load in client data that is being used across multiple files in the request.  So for example one spot that it commonly errors on the structure it says is undefined is

Request.Client.Admission.CSMG (csmg being case manager in this case, and is always a number, a client cannot have an admission without having an assigned case manager in the system).

In the dump it will show Request -> Client -> Admission -> CSMG: 9999, but the point of error says it's not defined.  And two lines of code before the point of error there is a reference to Request.Client.Admission.ADMNO (admission number), that works fine without erroring.

I tried adding a <cflock scope="session" around the <cfset request.Client = ... > tag today just to be sure, and it did not make a difference.

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Reply
Loading...
May 04, 2016 0
Advocate ,
May 04, 2016

Copy link to clipboard

Copied

Have you reviewed all the ColdFusion log files? It's possible that they might contain a clue as to what might be happening.

I know it is a production server, but since you're experiencing the error anyway, I recommend you try to access the key immediately after assigning a value to it. That way, when the problem occurs, it will be informative to note where it happens. If it doesn't happen immediately after the assignment then it will be a matter of figuring out what is happening to the key between the assignment and when you access it later in your application.

Cheers

Eddie

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Reply
Loading...
May 04, 2016 0
Explorer ,
May 04, 2016

Copy link to clipboard

Copied

I have looked in the logs and have not seen anything relevant near the same time stamp as the error.

It's random which key it complains about being undefined, so there isn't any single specific key I can output immediately after the assignment.  The main object .Request.Client is a entire structure be assigned in one <cfset>.  I have already verified all the keys in that structure are accurate at structure creation time.

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Reply
Loading...
May 04, 2016 0
Advocate ,
May 04, 2016

Copy link to clipboard

Copied

Steven6282 wrote:

The main object .Request.Client is a entire structure be assigned in one <cfset>.

I don't understand what you mean by this. Can you give a practical example?

Steven6282 wrote:

I have already verified all the keys in that structure are accurate at structure creation time.

By this, do you mean you verified that each key has a valid value?

Cheers

Eddie

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Reply
Loading...
May 04, 2016 0
Advocate ,
May 04, 2016

Copy link to clipboard

Copied

It occurs to me that you are talking about assigning one structure to another. This is probably the root of your problem because you're getting a reference instead of a copy of the structure or some of its keys.

Use the Duplicate() function to ensure that you get a deep copy when assigning one structure to another.

Cheers

Eddie

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Reply
Loading...
May 04, 2016 1
Explorer ,
May 05, 2016

Copy link to clipboard

Copied

EddieLotter wrote:

Steven6282 wrote:

The main object .Request.Client is a entire structure be assigned in one <cfset>.

I don't understand what you mean by this. Can you give a practical example?

Steven6282 wrote:

I have already verified all the keys in that structure are accurate at structure creation time.

By this, do you mean you verified that each key has a valid value?

Cheers

Eddie

In another part of the code these structures are built and stored in a session variable, there can be multiples and a url parameter in the request determines which one is used for that request.

So in another part of the code it does a query against a database and does something like:

<cfset var storedVariable = {}>

<cfloop list="#structKeyList( query.colList )#" index="column">

     <cfset structInsert( storedVariable, column, query[ column ] )>

</cfloop>

This isn't exactly the same it's just an example.  There is some additional processing done on the query data before it's stored in the structure, like formatting dates, trimming strings, etc.  These are also 1 record length queries for the record, that is ensured before the code runs.  In this code I have output all the variables post storing them in the structure and verified them as valid accurate values.

We are essentially doing this to cache the processed data for the session.

I cannot use duplicate because this should be a reference assignment not a copy.  There are times in the code when the structure will be updated in the request scope and it should update the original session structure.  I've disabled all of these spots currently where this could happen while trying to troubleshoot this bug, so it's not being updated anywhere currently, but it should have the capability of being.  Also duplicate is agonizingly slow and would be a horrible performance hit to use, most likely completely negating the performance advantage gained by caching the data.

This is not happening where the variable is being written.  I already have the exception as part of my dump data.  The root cause in the exception shows it where I said it was, at a random spot where the variables are being referenced, not written.

Also just to note, I don't see how this could sometimes be multiple requests.  Some of the spots where it errors happens through an XMLHttpRequest.  That can't be more than one request serverside and still return accurate data afaik.

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Reply
Loading...
May 05, 2016 0
Advocate ,
May 05, 2016

Copy link to clipboard

Copied

Steven6282 wrote:

I cannot use duplicate because this should be a reference assignment not a copy. There are times in the code when the structure will be updated in the request scope and it should update the original session structure. I've disabled all of these spots currently where this could happen while trying to troubleshoot this bug, so it's not being updated anywhere currently, but it should have the capability of being. Also duplicate is agonizingly slow and would be a horrible performance hit to use, most likely completely negating the performance advantage gained by caching the data.

I think I'm getting a better idea of what is happening. If you are not protecting the reads and updates to the session variables with cfLock then chances are good that some of your users are opening multiple browser tabs/windows which is causing overlapping updates to the session variables. Taking a reference to a session variable and putting it in the request scope does not alleviate the need to do locking.

Cheers

Eddie

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Reply
Loading...
May 05, 2016 0
Explorer ,
May 05, 2016

Copy link to clipboard

Copied

EddieLotter wrote:

Steven6282 wrote:

I cannot use duplicate because this should be a reference assignment not a copy. There are times in the code when the structure will be updated in the request scope and it should update the original session structure. I've disabled all of these spots currently where this could happen while trying to troubleshoot this bug, so it's not being updated anywhere currently, but it should have the capability of being. Also duplicate is agonizingly slow and would be a horrible performance hit to use, most likely completely negating the performance advantage gained by caching the data.

I think I'm getting a better idea of what is happening. If you are not protecting the reads and updates to the session variables with cfLock then chances are good that some of your users are opening multiple browser tabs/windows which is causing overlapping updates to the session variables. Taking a reference to a session variable and putting it in the request scope does not alleviate the need to do locking.

Cheers

Eddie

As I said,

The updates are currently disabled and the errors still occur.  Where the main structure is original built at is in in a session scope lock.

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Reply
Loading...
May 05, 2016 0
Advocate ,
May 05, 2016

Copy link to clipboard

Copied

As I said, you need to lock your reads as well.

Cheers

Eddie

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Reply
Loading...
May 05, 2016 0
Explorer ,
May 05, 2016

Copy link to clipboard

Copied

Lock a read?? I do not understand the purpose for that?  Can you please explain what that would accomplish?  If the data were possibly being modified somewhere else simultaneously I could understand, but since in this case it's not possible, I'm not really following what that would achieve.  I mean I can put a read-only lock around one of the more common spots I see the error to satisfy testing it, but I'd still like to understand the idea behind the necessity of it.

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Reply
Loading...
May 05, 2016 0
Advocate ,
May 05, 2016

Copy link to clipboard

Copied

Steven6282 wrote:

Lock a read?? I do not understand the purpose for that? Can you please explain what that would accomplish? If the data were possibly being modified somewhere else simultaneously I could understand, but since in this case it's not possible, I'm not really following what that would achieve. I mean I can put a read-only lock around one of the more common spots I see the error to satisfy testing it, but I'd still like to understand the idea behind the necessity of it.

Why do you think it's not possible? When a user opens more than one tab in their browser and make requests to your Web server, the server has no idea which tab the request is coming from. The same session scope is being used for all those tabs and they can interfere with each other. It's for this reason that cfLock exists.

For more information, and to better understand what ColdFusion is doing, read Using Persistent Data and Locking in the CFML Reference Guide.

Cheers

Eddie

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Reply
Loading...
May 05, 2016 0
Explorer ,
May 05, 2016

Copy link to clipboard

Copied

Because I have code in place that preserves the integrity session data across tabs exactly for that reason.  Using HTML5 sessionStroage which is tab specific, and creating a unique tab id, to pass along with requests the session data is scoped specific to that tab.

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Reply
Loading...
May 05, 2016 0
Advocate ,
May 05, 2016

Copy link to clipboard

Copied

What happens when your users don't use an HTML5 capable browser?

Cheers

Eddie

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Reply
Loading...
May 05, 2016 0
Explorer ,
May 05, 2016

Copy link to clipboard

Copied

Those users don't exist for us.  This is an internal business application where all user PCs are on the domain and their software is managed by the domain.  All of our users currently use IE11.  The effect pages on the website would not work correctly at all in a browser that doesn't support HTML5, there are canvas used in some places and other HTML5 elements.

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Reply
Loading...
May 05, 2016 0
Advocate ,
May 05, 2016

Copy link to clipboard

Copied

For the record, I am not saying that this is the root cause of the problem. What I am saying is that not locking access to session variables can cause the intermittent and spurious problems that you are experiencing, which is why I brought it up. It is in your best interests to rule it out as a possible cause by using ColdFusion the way that it is intended to be used.

Intermittent problems are among the worst to troubleshoot. You have been given ideas by several people in this thread and I don't think there is anything more we can do to help.

Good luck with the bug hunt. If you get to the bottom of it, it will be helpful to the community if you report back here about it.

Cheers

Eddie

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Reply
Loading...
May 05, 2016 0
Explorer ,
May 05, 2016

Copy link to clipboard

Copied

Ok well thanks for trying.  I figured it was to obscure of a problem.  That is usually what happens these days, by the time I need to ask for help, it's to weird or complex of a problem for there to be any ready solution

That is also why we don't bother contacting adobe support anymore.  By the time we get to a point that we can't solve an adobe problem on our own, it completely baffles their tier 1 support and they will have us chasing our tales or giving us inaccurate information for months lol.  Last time we contacted them about a known page break bug in cfdocument, after 8 months of back and forth and them not being able to come up with a solution they finally just said that it was a 3rd party issue and they would no longer support it haha.  Talk about bad support... if it's included in your product you can't just write it off as a 3rd part issue.

Anyway, I digress.  Thanks again for trying, if I do find a solution I will come back and post it, but it's looking more and more likely that I'm going to have to chalk it up to some obscure CF bug, won't be the first one and probably not the last either.  Already found a couple of others while working on this project, like the JSON bug and CF sending strings as numbers causing a json parse error ( "0000" for example will be output in the json string as parameter: 0000 instead of parameter: "0000" ).

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Reply
Loading...
May 05, 2016 0
Adobe Community Professional ,
May 05, 2016

Copy link to clipboard

Copied

Did you use my suggestion and dump arguments.exception instead of request?

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Reply
Loading...
May 05, 2016 0
Explorer ,
May 05, 2016

Copy link to clipboard

Copied

BKBK wrote:

Did you use my suggestion and dump arguments.exception instead of request?

Yes

Steven Hickerson wrote:

This is not happening where the variable is being written.  I already have the exception as part of my dump data.  The root cause in the exception shows it where I said it was, at a random spot where the variables are being referenced, not written.

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Reply
Loading...
May 05, 2016 0
Adobe Community Professional ,
May 05, 2016

Copy link to clipboard

Copied

Ah, in your response to EddieLotter, which I only skimmed through. Thanks for the repeat.

I think, rather than find a workaround, we should determine the cause. For if we cannot, then this will point to a bug.

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Reply
Loading...
May 05, 2016 0
Adobe Community Professional ,
May 06, 2016

Copy link to clipboard

Copied

Steven Hickerson wrote:

This is not happening where the variable is being written. I already have the exception as part of my dump data. The root cause in the exception shows it where I said it was, at a random spot where the variables are being referenced, not written.

Could you, for the sake of clarity, show us the line at which the error occurs.

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Reply
Loading...
May 06, 2016 0
Explorer ,
May 09, 2016

Copy link to clipboard

Copied

I have a theory on the "cause" but it's just a theory.  I've been able to duplicate it one time out of hundreds of attempts.  I think what is happening is the one and only time that coldfusion should be writing the object, it's progressing to the next line of code before the write is fully complete in memory.  So if the system is for some reason slow about completing the write to memory or it processes other code fast enough to try and read from memory before the write is complete it gets the error.  This might be a behavior of Java behind the scenes, I don't know.


Like I said I've been able to duplicate it once with a structure that was made intentionally very large (I stuffed a few 100 MB binary blobs in it and 500 other variables).  I then set a request variable = to it, and in the next part of the request attempted to access the last element in the structure.  One time out of at least over a hundred refreshes on that test page I was able to get an undefined error.

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Reply
Loading...
May 09, 2016 0
Adobe Community Professional ,
May 09, 2016

Copy link to clipboard

Copied

It is more likely that a second request is involved. It attempts to access the last element in the structure while the first request is still writing to the structure. If so, then a solution would be to put a lock, of exclusive type, around the first code.

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Reply
Loading...
May 09, 2016 2
Advocate ,
May 10, 2016

Copy link to clipboard

Copied

BKBK wrote:

a solution would be to put a lock, of exclusive type, around the first code.

That's what I recommended earlier in this thread, but Steven seems averse to locking code even though it ensures data integrity. Locking really does have a valid purpose.

Cheers

Eddie

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Reply
Loading...
May 10, 2016 0
Adobe Community Professional ,
May 10, 2016

Copy link to clipboard

Copied

So you did. As they say, EddieLotter, when you're right you're right.

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Reply
Loading...
May 10, 2016 0
Advocate ,
May 10, 2016

Copy link to clipboard

Copied

BKBK wrote:

As they say...

Who's been talking behind my back??

Cheers

Eddie

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Reply
Loading...
May 10, 2016 0
Explorer ,
May 10, 2016

Copy link to clipboard

Copied

EddieLotter wrote:

BKBK wrote:

a solution would be to put a lock, of exclusive type, around the first code.

That's what I recommended earlier in this thread, but Steven seems averse to locking code even though it ensures data integrity. Locking really does have a valid purpose.

Cheers

Eddie

Actually, you might want to check the definition of adverse because I said I would add the cflock to see if it made a difference.

Further, in the test case I described above that I was able to reproduce the error in, there was an exclusive lock around the write and a readonly lock around where the error occurred.  So the locks apparently didn't help.

I'm also 100% positive in the test case there was a single request.

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Reply
Loading...
May 10, 2016 0
Adobe Community Professional ,
May 05, 2016

Copy link to clipboard

Copied

Steven6282 wrote:

What makes this so bizarre is in the email from the error handler (that is processed in the same request) shows the key.subkey existing in the request scoped variable in a <cfdump var="#request#"> in the email. How could it possibly be erroring on a part of the page that runs after OnRequestStart, but before OnError, saying that this structure element doesn't exist yet it does exist during the error handler caused by it.

Your application might inadvertently be triggering an additional request. Although the variable is read in onRequestStart the error occurs where Coldfusion writes the variable.

To get information about where that happens, replace the dump of the request scope (in onError) with:

<cfdump var="#arguments.exception#">

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Reply
Loading...
May 05, 2016 0