• Global community
    • Language:
      • Deutsch
      • English
      • Español
      • Français
      • Português
  • 日本語コミュニティ
    Dedicated community for Japanese speakers
  • 한국 커뮤니티
    Dedicated community for Korean speakers
Exit
0

Cold Fusion 11 Application Server Crash on Form Submit

New Here ,
Aug 01, 2019 Aug 01, 2019

Copy link to clipboard

Copied

I am running Cold Fusion 11 and have a problem, intermittently, when a form is submitted the entire application server stops.   The application I've created has been running for several years without throwing this problem.  

I've looked through all of the log files and I've not found any indication of what is stopping the server

And, of course, the problem does not happen all the time.   I was able to recreate the issue with one of my users, so at least I know it is real and not user error.

The process is such:

A call is made to the DB to get any entries associated with the master reference key

I loop over those entries and dynamically build the form inside of tables (yeah, I'm old and lazy)

Each form field is name+indexing variable.   The indexer starts of at 1 and I just add 1 to it as the cfoutput loops

So, each form field name us unique

In the case where the server is crashing, I am only submitting ~30 fields (though I've previously upped the default quite a bit)

When the user hits 'Save' his browser [Chrome on Windows] displays the broken icon in the frame.   My browser just freezes on the form [Safari on MAC].   I do NOT get any type of error message.   Also, I do not see any indication the form has actually been submitted (no screen refresh, etc) -- though it clearly is.

This is running on an Ubuntu server.    Grepping for the process returns nothing and I have to start Cold Fusion from scratch.

I've verified nothing odd is populated in the form fields.   They are all either integers or text fields.   Very simple.

Anyone have any ideas?

Thanks!

Dave

Views

657

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
LEGEND ,
Aug 01, 2019 Aug 01, 2019

Copy link to clipboard

Copied

Just off the top of my head:

When it happens, check the browser console log just in case there is anything JavaScript related (esp if you are using [GASP] cfform).

Check the CFAdmin logs to see if any of them are recording the issue.

Check the webserver logs to see if any of them are recording the issue, or perhaps feedback from CF server.

HTH,

^ _ ^

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Community Expert ,
Aug 01, 2019 Aug 01, 2019

Copy link to clipboard

Copied

If that submission REALLY hangs CF (you have to kill it to resolve things), then it must be doing SOMETHING to cause the problem. And I’d suspect a possible memory problem, though it could be CPU. You don’t say if you have seen either indicated, in top (since you are on Linux).

So the first thing I’d look at is the coldfusion-error.log, or since you on Linux look also at the cfserver.log. Look at lines from just BEFORE was restarted. IS there indication of any outofmemory error? Look through the log lines to observe the date/time values, and look back as much as a few mins before you killed CF (or grep the file).

If you are on CF Enterprise, you cold also watch things within its Server Monitor, or you could try FusionReactor, which has a free 14-day trial. It has the added benefit of logging what it tracks, so that even after a CF restart you could look into what was amiss. It would also more clearly help you to see whether it WAS a memory problem, or a CPU problem, or if the request was hanging and what happened to other requests trying to run at that time. BTW, have you clarified if other requests can run at that time, or not?

/charlie


/Charlie (troubleshooter, carehart.org)

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
New Here ,
Aug 02, 2019 Aug 02, 2019

Copy link to clipboard

Copied

Oddly enough -- Cold Fusion does not hang - it straight up dies....instantly.   I've not found anything in the log files (though it is more than possible I've missed something).   I've been using CF for over 20 years and I've never encountered an issue this severe.

Yup.   I am still using CFFORM and, I'm sure, a LOT of antiquated tags.   *chuckle*  

Working with my user - it appears that if a text input field has the same string (user input) as the same field in the proceeding row, something in the submission will crash the server.   Super odd.   The form looks like this:

          <cfform name="defineequipment" action="request.cfm" method="post" enctype="multipart/form-data">

                 <cfinput type="hidden" name="manageX" value="Equipment Inventory">

                  <cfinput type="hidden" name="WUserKey" value="#WUserKey#">

                  <cfinput type="hidden" name="requestkey" value="#requestkey#">

                    <cfset #equcount# = '1'>

               

                      <cfoutput query="getequipment">

                     <td><cfinput type="text" name="eequipment#equcount#" value="#edesc#"></td>

                    <cfset #equcount# = #equcount# + 1>

                      </cfoutput>

               <cfinput type="hidden" name="equcount" value="#equcount#">

                     </cfform>

There is quite a bit more to the form - but above is the area which seems to be causing the problem (Equipment Description).   As you can see, each row of field names is appended with an integer which is incremented over the loop.   So, every field name is unique.    When I process the form after submit, I loop over #equcount# and 'find' each variable name.

What is so strange, is...when my user clicks the submit button, everything dies.   No error, no refresh of screen...nothing.   The whole process is dead on the server and I have to restart CF from scratch.

I don't think this is a memory issue - I have some 40 cores and over 100 GB of RAM on the server.   More importantly, I have applications running which are submitting hundreds of form fields at a time and are not causing an issue at all.

Is there another log I might check?  Or, something I can spool as I recreate the issue?

Thanks!

Dave

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Community Expert ,
Aug 02, 2019 Aug 02, 2019

Copy link to clipboard

Copied

I would try running CF as an application instead of as a server, and see if it spits up an error at the point of failure that for some reason it's unable to log.

Dave Watts, Eidolon LLC

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
LEGEND ,
Aug 02, 2019 Aug 02, 2019

Copy link to clipboard

Copied

The only other thing that I can think of would be to put the error-prone code within a CFTRY/CFCATCH and have the catch portion <cfdump var="#cfcatch#" /> to see if that gives any indication.

HTH,

^ _ ^

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Community Expert ,
Aug 02, 2019 Aug 02, 2019

Copy link to clipboard

Copied

So DaveO, a few things:

  • you say the problem is when the form is submitted, but then you show us code that is CREATING the form.
    • Do you mean really that when they view the page that SHOWS this form, that's when it does?
    • Or do you mean when they then SUBMIT this form, it does? IF the latter, then we would want to focus on code running in the action of the form (request.cfm)
  • you say that CF "dies", but you have not yet found any log entry
    • so first, look in the server.log. it's one of a few that do track lines when CF is starting (and stopping). But unlike the others, often it ONLY has that start/stop info. So first, look at the time in the log when you feel CF "died". Do you see it starting (or stopping) then? If you do not see it starting then, but you know it was running again soon "after it died", then it did not stop, and did not die. it may have SEEMED so, but if there are not log entries in that and the other logs (coldfusion-out and coldfusion-error) that refer to CF starting up then, then it did not go down. And in those other two logs, you could look at the time BEFORE CF came up, to see if they show anything about why it may have gone down (they do also log that, and it going down, if CF is stopped)
    • But as for whether CF can "die" and not log that it "died", yes it can. Note I said the other logs showed CF being "stopped"--by someone stopping CF. Instead, on rare occasions CF can indeed just "die", and there'd be no log in those other logs of it "dying" that way. (Same if someone "killed" CF, by ending the process.)
    • For evidence of CF just "dying", look instead in the cfusion/bin folder (where the jvm.config is also), and see if there are any log files there. Usually there are none, but if the JVM *underlying CF" did abruptly abort, then you could find logs there. Again, these log it "dying". The other logs above would then still show it STARTING UP again when you or something or someone did start CF again, so that's why I proposed you look at those first, to see "is it starting", which would be after it had "died" or "been stopped", and someone or something brought it back up. If you don't see indications of it "starting" in the server.log, then it did not go down (did not die, was not stopped)

Finally, Dave mentioned you could start CF at the command line (he called it "as an application" as opposed to "as a service"). That's certainly another way to go. If you started CF there, note that it will NOT log to the coldfusion-out.log but instead to the screen after you start it. And then if it stopped/died/was killed, you'd see that it would return to the command prompt and stop logging on screen.

So let us know what you find. If there ARE logs in that cfusion/bin folder at this time, you need only quote the first few lines of any such log there.


/Charlie (troubleshooter, carehart.org)

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
New Here ,
Aug 02, 2019 Aug 02, 2019

Copy link to clipboard

Copied

Good thought on the CFCATCH - but I don't think I'm even getting back to the code when the form is submitted.   Normally, when I've made an error or one of my users enters 'bad' data, you get plenty of error messaging back (I have all the error output turned on for this server).  In this case nothing.

I'll dig more into this issue after we are done with the budget this year.   Amusingly, this particular tool is used for our capital budget requests and is managing over 200 M in requests this year.   You'd think I could manage not to have it crash!

Dave

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
LEGEND ,
Aug 02, 2019 Aug 02, 2019

Copy link to clipboard

Copied

Just out of curiosity.. in CFAdmin, what is the value for Server Settings > Settings > Maximum Number of POST request parameters?  It defaults to 100.  STIG recommends keeping it there.

V/r,

^ _ ^

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Community Expert ,
Aug 02, 2019 Aug 02, 2019

Copy link to clipboard

Copied

To Wolf's question about the "max post parameters", that setting (new in the admin as of CF10) could certainly cause a request to ERROR (if the form had more than 100 form fields), but it would not cause CF to "die".

This is one more reason I have been pressing DaveO on his contention that CF was "dying". If that is so, I'd not expect it to be about this issue (of the max post params), but of course if CF is NOT dying, then yes this COULD be why the REQUEST is dying (failing).

And to be clear, DaveO, if you may have only been looking in the CF application.log for evidence of any error, that one is NOT logged there. Instead, it IS logged in the very server.log I referred to above (for looking at indications of CF starting and stopping). Who knows, you may end up seeing that and finding these errors instead (and indication that CF was not in fact "dying").

Waiting with bated breath. 🙂


/Charlie (troubleshooter, carehart.org)

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
New Here ,
Aug 02, 2019 Aug 02, 2019

Copy link to clipboard

Copied

Ah!   Thanks for the great comments! 

So, as this is on a Linux box - I am very certain CF has died.   I just grep for the process: ps -ef | grep cold* ...and nothing is returned (other than my grep itself).   No other user has the ability to 'stop' the server from the command line or anything like that.

I start the server from the command line as root using: ./coldfusion start

Yesterday, I had my user share his screen and he killed CF over and over for me.

True, I included the create component of the form - but, to clarify, the server dies once the user clicks the submit button.   Displaying the form works fine...and for a few hundred other users...never crashes.

What is unique to this user is the text in Equipment Description field is usually identical between rows.

Thank you a ton for pointing out the logs in the /bin directory.   I do indeed have PLENTY of error files in there.   Here is the leading text in one of them (they are all pretty much identical).   Let me know if you'd like to see a full log (I don't see where I can upload a file to the forum)

Thanks!!

Dave

-----------

dohare@telops-reno:/opt/coldfusion11/cfusion/bin$ cat hs_err_pid63635.log

#

# A fatal error has been detected by the Java Runtime Environment:

#

#  Internal Error (sharedRuntime.cpp:833), pid=63635, tid=139628635961088

#  fatal error: exception happened outside interpreter, nmethods and vtable stubs at pc 0x00007f00d50541af

#

# JRE version: Java(TM) SE Runtime Environment (8.0_25-b17) (build 1.8.0_25-b17)

# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.25-b02 mixed mode linux-amd64 compressed oops)

# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again

#

# If you would like to submit a bug report, please visit:

#   http://bugreport.sun.com/bugreport/crash.jsp

#

---------------  T H R E A D  ---------------

Current thread (0x00007efe1400d800):  JavaThread "http-nio-8543-exec-10" daemon [_thread_in_Java, id=64237, stack(0x00007efdd32f3000,0x00007efdd33f4000)]

Stack: [0x00007efdd32f3000,0x00007efdd33f4000],  sp=0x00007efdd33f0b50,  free space=1014k

Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)

V  [libjvm.so+0xa7923a]  VMError::report_and_die()+0x2ca

V  [libjvm.so+0x4e4fd9]  report_fatal(char const*, int, char const*)+0x59

V  [libjvm.so+0x97c9ca]  SharedRuntime::continuation_for_implicit_exception(JavaThread*, unsigned char*, SharedRuntime::ImplicitExceptionKind)+0x33a

V  [libjvm.so+0x8eb8ca]  JVM_handle_linux_signal+0x48a

V  [libjvm.so+0x8e2523]  signalHandler(int, siginfo*, void*)+0x43

C  [libpthread.so.0+0x110e0]

J 10807 C2 com.sun.crypto.provider.GCTR.doFinal([BII[BI)I (130 bytes) @ 0x00007f00d72bc4f6 [0x00007f00d72bc2c0+0x236]

---------------  P R O C E S S  ---------------

Java Threads: ( => current thread )

  0x00007efd98058000 JavaThread "qtp1892123515-920" [_thread_blocked, id=4480, stack(0x00007f00532f3000,0x00007f00533f4000)]

  0x00007eff0032e000 JavaThread "Thread-157" daemon [_thread_blocked, id=64900, stack(0x00007f00688e5000,0x00007f00689e6000)]

=>0x00007efe1400d800 JavaThread "http-nio-8543-exec-10" daemon [_thread_in_Java, id=64237, stack(0x00007efdd32f3000,0x00007efdd33f4000)]

  0x00007efe1400b800 JavaThread "http-nio-8543-exec-9" daemon [_thread_blocked, id=64236, stack(0x00007efdd33f4000,0x00007efdd34f5000)]

  0x00007efe1400a000 JavaThread "http-nio-8543-exec-8" daemon [_thread_blocked, id=64235, stack(0x00007efdd34f5000,0x00007efdd35f6000)]

  0x00007efe14008800 JavaThread "http-nio-8543-exec-7" daemon [_thread_blocked, id=64234, stack(0x00007efdd35f6000,0x00007efdd36f7000)]

  0x00007efe14006800 JavaThread "http-nio-8543-exec-6" daemon [_thread_blocked, id=64233, stack(0x00007efdd36f7000,0x00007efdd37f8000)]

  0x00007efe14005000 JavaThread "http-nio-8543-exec-5" daemon [_thread_blocked, id=64232, stack(0x00007efdd37f8000,0x00007efdd38f9000)]

  0x00007efe20002800 JavaThread "http-nio-8543-exec-4" daemon [_thread_blocked, id=64231, stack(0x00007efdd38f9000,0x00007efdd39fa000)]

  0x00007efe14003000 JavaThread "http-nio-8543-exec-3" daemon [_thread_blocked, id=64230, stack(0x00007efdd39fa000,0x00007efdd3afb000)]

  0x00007efe20001000 JavaThread "http-nio-8543-exec-2" daemon [_thread_blocked, id=64229, stack(0x00007efdd3afb000,0x00007efdd3bfc000)]

  0x00007efe14001800 JavaThread "http-nio-8543-exec-1" daemon [_thread_blocked, id=64228, stack(0x00007efdd3bfc000,0x00007efdd3cfd000)]

  0x00007eff04001000 JavaThread "scheduler-2" daemon [_thread_blocked, id=64206, stack(0x00007efdd3eff000,0x00007efdd4000000)]

  0x00007efe2800b800 JavaThread "http-bio-8500-exec-10" daemon [_thread_blocked, id=63986, stack(0x00007f005050b000,0x00007f005060c000)]

  0x00007efe2800a800 JavaThread "http-bio-8500-exec-9" daemon [_thread_blocked, id=63982, stack(0x00007f0050e0e000,0x00007f0050f0f000)]

  0x00007efe28009800 JavaThread "http-bio-8500-exec-8" daemon [_thread_blocked, id=63981, stack(0x00007f00687e4000,0x00007f00688e5000)]

  0x00007efe2c020000 JavaThread "http-bio-8500-exec-7" daemon [_thread_blocked, id=63959, stack(0x00007f005080c000,0x00007f005090d000)]

  0x00007efe28008800 JavaThread "http-bio-8500-exec-6" daemon [_thread_blocked, id=63958, stack(0x00007f00533f4000,0x00007f00534f5000)]

  0x00007efe04068800 JavaThread "http-bio-8500-exec-5" daemon [_thread_blocked, id=63957, stack(0x00007f005ac65000,0x00007f005ad66000)]

  0x00007efdf803a800 JavaThread "Abandoned connection cleanup thread" daemon [_thread_blocked, id=63952, stack(0x00007f0050b0d000,0x00007f0050c0e000)]

  0x00007f00e400b800 JavaThread "DestroyJavaVM" [_thread_blocked, id=63637, stack(0x00007f00ed334000,0x00007f00ed434000)]

  0x00007f00e45bb000 JavaThread "Thread-73" [_thread_in_native, id=63936, stack(0x00007f0050f0f000,0x00007f0051010000)]

  0x00007f00e45b9800 JavaThread "ajp-bio-8014-AsyncTimeout" daemon [_thread_blocked, id=63935, stack(0x00007f0051010000,0x00007f0051111000)]

  0x00007f00e4289000 JavaThread "ajp-bio-8014-Acceptor-0" daemon [_thread_in_native, id=63934, stack(0x00007f0051111000,0x00007f0051212000)]

  0x00007efe28007000 JavaThread "http-bio-8500-exec-4" daemon [_thread_blocked, id=63933, stack(0x00007f0051212000,0x00007f0051313000)]

  0x00007efe28005000 JavaThread "http-bio-8500-exec-3" daemon [_thread_blocked, id=63932, stack(0x00007f0051313000,0x00007f0051414000)]

  0x00007efe28003000 JavaThread "http-bio-8500-exec-2" daemon [_thread_blocked, id=63931, stack(0x00007f0051414000,0x00007f0051515000)]

  0x00007efe28001800 JavaThread "http-bio-8500-exec-1" daemon [_thread_blocked, id=63930, stack(0x00007f0051515000,0x00007f0051616000)]

  0x00007f00e4287800 JavaThread "http-nio-8543-Acceptor-0" daemon [_thread_in_native, id=63929, stack(0x00007f0051616000,0x00007f0051717000)]

  0x00007f00e4286000 JavaThread "http-nio-8543-ClientPoller-1" daemon [_thread_in_native, id=63928, stack(0x00007f0051717000,0x00007f0051818000)]

  0x00007f00e4284000 JavaThread "http-nio-8543-ClientPoller-0" daemon [_thread_in_native, id=63927, stack(0x00007f0051818000,0x00007f0051919000)]

  0x00007f00e4283000 JavaThread "http-bio-8500-AsyncTimeout" daemon [_thread_blocked, id=63926, stack(0x00007f0051919000,0x00007f0051a1a000)]

  0x00007f00e4282000 JavaThread "http-bio-8500-Acceptor-0" daemon [_thread_in_native, id=63925, stack(0x00007f0051a1a000,0x00007f0051b1b000)]

  0x00007f00e4281800 JavaThread "ContainerBackgroundProcessor[StandardEngine[Catalina]]" daemon [_thread_blocked, id=63924, stack(0x00007f0051d1b000,0x00007f0051e1c000)]

  0x00007effd6134800 JavaThread "Timer-5" daemon [_thread_blocked, id=63918, stack(0x00007f005a05b000,0x00007f005a15c000)]

  0x00007efe34001000 JavaThread "qtp1892123515-224-acceptor-1-ServerConnector@70500c13{HTTP/1.1}{0.0.0.0:5500}" [_thread_in_native, id=63899, stack(0x00007f00534f5000,0x00007f00535f6000)]

  0x00007efe40001000 JavaThread "qtp1892123515-223-acceptor-0-ServerConnector@70500c13{HTTP/1.1}{0.0.0.0:5500}" [_thread_blocked, id=63898, stack(0x00007f00535f6000,0x00007f00536f7000)]

  0x00007efe3c001000 JavaThread "qtp1892123515-222-selector-39" [_thread_in_native, id=63897, stack(0x00007f00536f7000,0x00007f00537f8000)]

  0x00007efe48001000 JavaThread "qtp1892123515-221-selector-38" [_thread_in_native, id=63896, stack(0x00007f00537f8000,0x00007f00538f9000)]

  0x00007efe44001000 JavaThread "qtp1892123515-220-selector-37" [_thread_in_native, id=63895, stack(0x00007f00538f9000,0x00007f00539fa000)]

  0x00007efe50001000 JavaThread "qtp1892123515-219-selector-36" [_thread_in_native, id=63894, stack(0x00007f00539fa000,0x00007f0053afb000)]

  0x00007efe4c001000 JavaThread "qtp1892123515-218-selector-35" [_thread_in_native, id=63893, stack(0x00007f0053afb000,0x00007f0053bfc000)]

  0x00007efe58001000 JavaThread "qtp1892123515-217-selector-34" [_thread_in_native, id=63892, stack(0x00007f0053bfc000,0x00007f0053cfd000)]

  0x00007efe54001000 JavaThread "qtp1892123515-216-selector-33" [_thread_in_native, id=63891, stack(0x00007f0053cfd000,0x00007f0053dfe000)]

  0x00007efe60001000 JavaThread "qtp1892123515-215-selector-32" [_thread_in_native, id=63890, stack(0x00007f0053dfe000,0x00007f0053eff000)]

  0x00007efe5c001000 JavaThread "qtp1892123515-214-selector-31" [_thread_in_native, id=63889, stack(0x00007f0053eff000,0x00007f0054000000)]

  0x00007efe68001000 JavaThread "qtp1892123515-213-selector-30" [_thread_in_native, id=63888, stack(0x00007f0058093000,0x00007f0058194000)]

  0x00007efe64001000 JavaThread "qtp1892123515-212-selector-29" [_thread_in_native, id=63887, stack(0x00007f0058194000,0x00007f0058295000)]

  0x00007efe70001000 JavaThread "qtp1892123515-211-selector-28" [_thread_in_native, id=63886, stack(0x00007f0058295000,0x00007f0058396000)]

  0x00007efe6c001000 JavaThread "qtp1892123515-210-selector-27" [_thread_in_native, id=63885, stack(0x00007f0058396000,0x00007f0058497000)]

  0x00007efe78001000 JavaThread "qtp1892123515-209-selector-26" [_thread_in_native, id=63884, stack(0x00007f0058497000,0x00007f0058598000)]

  0x00007efe74001000 JavaThread "qtp1892123515-208-selector-25" [_thread_in_native, id=63883, stack(0x00007f0058598000,0x00007f0058699000)]

  0x00007efe80001000 JavaThread "qtp1892123515-207-selector-24" [_thread_in_native, id=63882, stack(0x00007f0058699000,0x00007f005879a000)]

  0x00007efe7c001000 JavaThread "qtp1892123515-206-selector-23" [_thread_in_native, id=63881, stack(0x00007f005879a000,0x00007f005889b000)]

  0x00007efe88001000 JavaThread "qtp1892123515-205-selector-22" [_thread_in_native, id=63880, stack(0x00007f005889b000,0x00007f005899c000)]

  0x00007efe84001000 JavaThread "qtp1892123515-204-selector-21" [_thread_in_native, id=63879, stack(0x00007f005899c000,0x00007f0058a9d000)]

  0x00007efe90001000 JavaThread "qtp1892123515-203-selector-20" [_thread_in_native, id=63878, stack(0x00007f0058a9d000,0x00007f0058b9e000)]

  0x00007efe8c001000 JavaThread "qtp1892123515-202-selector-19" [_thread_in_native, id=63877, stack(0x00007f0058b9e000,0x00007f0058c9f000)]

  0x00007efe98001000 JavaThread "qtp1892123515-201-selector-18" [_thread_in_native, id=63876, stack(0x00007f0058c9f000,0x00007f0058da0000)]

  0x00007efe94001000 JavaThread "qtp1892123515-200-selector-17" [_thread_in_native, id=63875, stack(0x00007f0058da0000,0x00007f0058ea1000)]

  0x00007efea0001000 JavaThread "qtp1892123515-199-selector-16" [_thread_in_native, id=63874, stack(0x00007f0058ea1000,0x00007f0058fa2000)]

  0x00007efe9c001000 JavaThread "qtp1892123515-198-selector-15" [_thread_in_native, id=63873, stack(0x00007f0058fa2000,0x00007f00590a3000)]

  0x00007efea8001000 JavaThread "qtp1892123515-197-selector-14" [_thread_in_native, id=63872, stack(0x00007f00590a3000,0x00007f00591a4000)]

  0x00007efea4001000 JavaThread "qtp1892123515-196-selector-13" [_thread_in_native, id=63871, stack(0x00007f00591a4000,0x00007f00592a5000)]

  0x00007efeb0001000 JavaThread "qtp1892123515-195-selector-12" [_thread_in_native, id=63870, stack(0x00007f00592a5000,0x00007f00593a6000)]

  0x00007efeac001000 JavaThread "qtp1892123515-194-selector-11" [_thread_in_native, id=63869, stack(0x00007f00593a6000,0x00007f00594a7000)]

  0x00007efeb8001000 JavaThread "qtp1892123515-193-selector-10" [_thread_in_native, id=63868, stack(0x00007f00594a7000,0x00007f00595a8000)]

  0x00007efeb4001000 JavaThread "qtp1892123515-192-selector-9" [_thread_in_native, id=63867, stack(0x00007f00595a8000,0x00007f00596a9000)]

  0x00007efec0001000 JavaThread "qtp1892123515-191-selector-8" [_thread_in_native, id=63866, stack(0x00007f00596a9000,0x00007f00597aa000)]

  0x00007efebc001000 JavaThread "qtp1892123515-190-selector-7" [_thread_in_native, id=63865, stack(0x00007f00597aa000,0x00007f00598ab000)]

  0x00007efec8001000 JavaThread "qtp1892123515-189-selector-6" [_thread_in_native, id=63864, stack(0x00007f00598ab000,0x00007f00599ac000)]

  0x00007efec4001000 JavaThread "qtp1892123515-188-selector-5" [_thread_in_native, id=63863, stack(0x00007f00599ac000,0x00007f0059aad000)]

  0x00007efed0001000 JavaThread "qtp1892123515-187-selector-4" [_thread_in_native, id=63862, stack(0x00007f0059aad000,0x00007f0059bae000)]

  0x00007efecc001000 JavaThread "qtp1892123515-186-selector-3" [_thread_in_native, id=63861, stack(0x00007f0059bae000,0x00007f0059caf000)]

  0x00007efed4001000 JavaThread "qtp1892123515-185-selector-2" [_thread_in_native, id=63860, stack(0x00007f0059caf000,0x00007f0059db0000)]

  0x00007effd6449800 JavaThread "HashSessionScavenger-0" daemon [_thread_blocked, id=63859, stack(0x00007f005a35e000,0x00007f005a45f000)]

  0x00007effd6445800 JavaThread "qtp1892123515-182-selector-1" [_thread_in_native, id=63857, stack(0x00007f005a15c000,0x00007f005a25d000)]

  0x00007effd6442000 JavaThread "qtp1892123515-181-selector-0" [_thread_in_native, id=63856, stack(0x00007f005a25d000,0x00007f005a35e000)]

  0x00007effd635c000 JavaThread "DateCache" daemon [_thread_blocked, id=63855, stack(0x00007f005ad66000,0x00007f005ae67000)]

  0x00007efef4004000 JavaThread "worker #4" daemon [_thread_blocked, id=63852, stack(0x00007f005a65f000,0x00007f005a760000)]

  0x00007efef4002000 JavaThread "worker #3" daemon [_thread_blocked, id=63851, stack(0x00007f005a760000,0x00007f005a861000)]

  0x00007efef4009800 JavaThread "worker #2" daemon [_thread_blocked, id=63850, stack(0x00007f005a861000,0x00007f005a962000)]

  0x00007efef4007800 JavaThread "worker #1" daemon [_thread_blocked, id=63849, stack(0x00007f005a962000,0x00007f005aa63000)]

  0x00007efef4001000 JavaThread "worker #0" daemon [_thread_blocked, id=63848, stack(0x00007f005aa63000,0x00007f005ab64000)]

  0x00007effd50af800 JavaThread "ServerThread" daemon [_thread_in_native, id=63847, stack(0x00007f005ab64000,0x00007f005ac65000)]

  0x00007effd645b800 JavaThread "Timer-4" daemon [_thread_blocked, id=63844, stack(0x00007f005ae67000,0x00007f005af68000)]

  0x00007effc8001000 JavaThread "scheduler-1" daemon [_thread_blocked, id=63843, stack(0x00007f005aff4000,0x00007f005b0f5000)]

  0x00007effd507f000 JavaThread "Thread-17" daemon [_thread_in_native, id=63842, stack(0x00007f0060223000,0x00007f0060324000)]

  0x00007effd5064000 JavaThread "Timer-3" daemon [_thread_blocked, id=63841, stack(0x00007f005b0f5000,0x00007f005b1f6000)]

  0x00007effd5060800 JavaThread "DefaultQuartzScheduler_QuartzSchedulerThread" [_thread_blocked, id=63840, stack(0x00007f005b1f6000,0x00007f005b2f7000)]

  0x00007effd505b000 JavaThread "DefaultQuartzScheduler_Worker-10" [_thread_blocked, id=63839, stack(0x00007f005b2f7000,0x00007f005b3f8000)]

  0x00007effd5059000 JavaThread "DefaultQuartzScheduler_Worker-9" [_thread_blocked, id=63838, stack(0x00007f005b3f8000,0x00007f005b4f9000)]

  0x00007effd5048800 JavaThread "DefaultQuartzScheduler_Worker-8" [_thread_blocked, id=63837, stack(0x00007f005b4f9000,0x00007f005b5fa000)]

  0x00007effd5046800 JavaThread "DefaultQuartzScheduler_Worker-7" [_thread_blocked, id=63836, stack(0x00007f005b5fa000,0x00007f005b6fb000)]

  0x00007effd5045000 JavaThread "DefaultQuartzScheduler_Worker-6" [_thread_blocked, id=63835, stack(0x00007f005b6fb000,0x00007f005b7fc000)]

  0x00007effd5043000 JavaThread "DefaultQuartzScheduler_Worker-5" [_thread_blocked, id=63834, stack(0x00007f005b7fc000,0x00007f005b8fd000)]

  0x00007effd5041800 JavaThread "DefaultQuartzScheduler_Worker-4" [_thread_blocked, id=63833, stack(0x00007f005b8fd000,0x00007f005b9fe000)]

  0x00007effd5040000 JavaThread "DefaultQuartzScheduler_Worker-3" [_thread_blocked, id=63832, stack(0x00007f005b9fe000,0x00007f005baff000)]

  0x00007effd503a000 JavaThread "DefaultQuartzScheduler_Worker-2" [_thread_blocked, id=63831, stack(0x00007f005baff000,0x00007f005bc00000)]

  0x00007effd5039000 JavaThread "DefaultQuartzScheduler_Worker-1" [_thread_blocked, id=63830, stack(0x00007f0060728000,0x00007f0060829000)]

  0x00007effd4e93800 JavaThread "obj-skimmer-metacounts" daemon [_thread_blocked, id=63816, stack(0x00007f0060324000,0x00007f0060425000)]

  0x00007effd4e92000 JavaThread "obj-skimmer-MetaOlivette" daemon [_thread_blocked, id=63815, stack(0x00007f0060425000,0x00007f0060526000)]

  0x00007effd4e8f000 JavaThread "obj-skimmer-MCP" daemon [_thread_blocked, id=63814, stack(0x00007f0060526000,0x00007f0060627000)]

  0x00007effd4e8d000 JavaThread "obj-skimmer-toolpass" daemon [_thread_blocked, id=63813, stack(0x00007f0060627000,0x00007f0060728000)]

  0x00007effd4e8c000 JavaThread "obj-skimmer-cloudinv" daemon [_thread_blocked, id=63812, stack(0x00007f0061e3f000,0x00007f0061f40000)]

  0x00007effd4e8a800 JavaThread "obj-skimmer-MetaEnglewood" daemon [_thread_blocked, id=63810, stack(0x00007f0060829000,0x00007f006092a000)]

  0x00007effd4e89000 JavaThread "obj-skimmer-metalad" daemon [_thread_blocked, id=63809, stack(0x00007f006092a000,0x00007f0060a2b000)]

  0x00007effd4e87000 JavaThread "obj-skimmer-PeakviewAGCstats" daemon [_thread_blocked, id=63808, stack(0x00007f0060a2b000,0x00007f0060b2c000)]

  0x00007effd4e85800 JavaThread "obj-skimmer-sitrep" daemon [_thread_blocked, id=63807, stack(0x00007f0060b2c000,0x00007f0060c2d000)]

  0x00007effd4e83800 JavaThread "obj-skimmer-accessaudit" daemon [_thread_blocked, id=63806, stack(0x00007f0060c2d000,0x00007f0060d2e000)]

  0x00007effd4e82000 JavaThread "obj-skimmer-MetaPeakviewAGC" daemon [_thread_blocked, id=63805, stack(0x00007f0060d2e000,0x00007f0060e2f000)]

  0x00007effd4e80000 JavaThread "obj-skimmer-MetaOlivetteStats" daemon [_thread_blocked, id=63804, stack(0x00007f0060e2f000,0x00007f0060f30000)]

  0x00007effd4e7e800 JavaThread "obj-skimmer-WAMmaster" daemon [_thread_blocked, id=63803, stack(0x00007f0060f30000,0x00007f0061031000)]

  0x00007effd4e7d000 JavaThread "obj-skimmer-endoffice2" daemon [_thread_blocked, id=63802, stack(0x00007f0061031000,0x00007f0061132000)]

  0x00007effd4e7b800 JavaThread "obj-skimmer-PeakviewCFSstats" daemon [_thread_blocked, id=63801, stack(0x00007f0061132000,0x00007f0061233000)]

  0x00007effd4e77000 JavaThread "obj-skimmer-budget" daemon [_thread_blocked, id=63800, stack(0x00007f0061233000,0x00007f0061334000)]

  0x00007effd4e75000 JavaThread "obj-skimmer-migrationtooldb" daemon [_thread_blocked, id=63799, stack(0x00007f0061334000,0x00007f0061435000)]

  0x00007effd4e73800 JavaThread "obj-skimmer-MetaGreenvilleAGC" daemon [_thread_blocked, id=63798, stack(0x00007f0061435000,0x00007f0061536000)]

  0x00007effd4e71800 JavaThread "obj-skimmer-MetaDenver" daemon [_thread_blocked, id=63797, stack(0x00007f0061536000,0x00007f0061637000)]

  0x00007effd4e70000 JavaThread "obj-skimmer-stamp" daemon [_thread_blocked, id=63796, stack(0x00007f0061637000,0x00007f0061738000)]

  0x00007effd4e6e800 JavaThread "obj-skimmer-migrationcheck" daemon [_thread_blocked, id=63795, stack(0x00007f0061738000,0x00007f0061839000)]

  0x00007effd4e6d000 JavaThread "obj-skimmer-sns" daemon [_thread_blocked, id=63794, stack(0x00007f0061839000,0x00007f006193a000)]

  0x00007effd4e6a800 JavaThread "obj-skimmer-PeakviewCFSSCD" daemon [_thread_blocked, id=63793, stack(0x00007f006193a000,0x00007f0061a3b000)]

  0x00007effd4e69000 JavaThread "obj-skimmer-MetaGreenville" daemon [_thread_blocked, id=63792, stack(0x00007f0061a3b000,0x00007f0061b3c000)]

  0x00007effd4e3d800 JavaThread "obj-skimmer-SBC" daemon [_thread_blocked, id=63791, stack(0x00007f0061b3c000,0x00007f0061c3d000)]

  0x00007effd4e38000 JavaThread "obj-skimmer-MetaLittletonAGC" daemon [_thread_blocked, id=63790, stack(0x00007f0061c3d000,0x00007f0061d3e000)]

  0x00007effd4e3b000 JavaThread "obj-skimmer-endoffice" daemon [_thread_blocked, id=63789, stack(0x00007f0061d3e000,0x00007f0061e3f000)]

  0x00007effd4e37000 JavaThread "obj-skimmer-khamlin" daemon [_thread_blocked, id=63788, stack(0x00007f0062775000,0x00007f0062876000)]

  0x00007effd4e36800 JavaThread "obj-skimmer-MetaOlivetteAGC" daemon [_thread_blocked, id=63786, stack(0x00007f0061f40000,0x00007f0062041000)]

  0x00007effd4e39800 JavaThread "obj-skimmer-ratecenters" daemon [_thread_blocked, id=63785, stack(0x00007f0062474000,0x00007f0062575000)]

  0x00007effd4dee000 JavaThread "scheduler-0" daemon [_thread_blocked, id=63780, stack(0x00007f0062b79000,0x00007f0062c7a000)]

  0x00007effd4dbe000 JavaThread "pool-1-thread-1" [_thread_blocked, id=63778, stack(0x00007f0062876000,0x00007f0062977000)]

  0x00007effd4d95000 JavaThread "AuthCacheManager" daemon [_thread_blocked, id=63776, stack(0x00007f0062e7a000,0x00007f0062f7b000)]

  0x00007effc4008000 JavaThread "MulticastListener" daemon [_thread_in_native, id=63771, stack(0x00007f0062977000,0x00007f0062a78000)]

  0x00007effc4007000 JavaThread "MulticastListener" daemon [_thread_in_native, id=63770, stack(0x00007f0062a78000,0x00007f0062b79000)]

  0x00007effd4c4c000 JavaThread "Thread-6" daemon [_thread_blocked, id=63749, stack(0x00007f006305c000,0x00007f006315d000)]

  0x00007effd4c5a800 JavaThread "Timer-2" daemon [_thread_blocked, id=63748, stack(0x00007f006335d000,0x00007f006345e000)]

  0x00007effd4bf2800 JavaThread "Timer-1" daemon [_thread_blocked, id=63747, stack(0x00007f00684e3000,0x00007f00685e4000)]

  0x00007effd4bf2000 JavaThread "Timer-0" daemon [_thread_blocked, id=63746, stack(0x00007f00689e6000,0x00007f0068ae7000)]

  0x00007f00e4579000 JavaThread "NioBlockingSelector.BlockPoller-1" daemon [_thread_in_native, id=63718, stack(0x00007f0068ae7000,0x00007f0068be8000)]

  0x00007f00e453d800 JavaThread "SeedGenerator Thread" daemon [_thread_blocked, id=63689, stack(0x00007f0070018000,0x00007f0070119000)]

  0x00007f00e44b2000 JavaThread "GC Daemon" daemon [_thread_blocked, id=63688, stack(0x00007f007036c000,0x00007f007046d000)]

  0x00007f00e439f800 JavaThread "Thread-1" [_thread_blocked, id=63687, stack(0x00007f0070cc4000,0x00007f0070dc5000)]

  0x00007f00e410b000 JavaThread "Service Thread" daemon [_thread_blocked, id=63685, stack(0x00007f0070f36000,0x00007f0071037000)]

  0x00007f00e4106000 JavaThread "C1 CompilerThread14" daemon [_thread_blocked, id=63684, stack(0x00007f0071038000,0x00007f0071138000)]

  0x00007f00e4103800 JavaThread "C1 CompilerThread13" daemon [_thread_blocked, id=63683, stack(0x00007f0071139000,0x00007f0071239000)]

  0x00007f00e4101800 JavaThread "C1 CompilerThread12" daemon [_thread_blocked, id=63682, stack(0x00007f007123a000,0x00007f007133a000)]

  0x00007f00e40ff800 JavaThread "C1 CompilerThread11" daemon [_thread_blocked, id=63681, stack(0x00007f007133b000,0x00007f007143b000)]

  0x00007f00e40fd800 JavaThread "C1 CompilerThread10" daemon [_thread_blocked, id=63680, stack(0x00007f007143c000,0x00007f007153c000)]

  0x00007f00e40fb800 JavaThread "C2 CompilerThread9" daemon [_thread_blocked, id=63679, stack(0x00007f007153d000,0x00007f007163d000)]

  0x00007f00e40f9800 JavaThread "C2 CompilerThread8" daemon [_thread_blocked, id=63678, stack(0x00007f007163e000,0x00007f007173e000)]

  0x00007f00e40f7000 JavaThread "C2 CompilerThread7" daemon [_thread_blocked, id=63677, stack(0x00007f007173f000,0x00007f007183f000)]

  0x00007f00e40f5000 JavaThread "C2 CompilerThread6" daemon [_thread_blocked, id=63676, stack(0x00007f0071840000,0x00007f0071940000)]

  0x00007f00e40f3000 JavaThread "C2 CompilerThread5" daemon [_thread_blocked, id=63675, stack(0x00007f0071941000,0x00007f0071a41000)]

  0x00007f00e40e9000 JavaThread "C2 CompilerThread4" daemon [_thread_blocked, id=63674, stack(0x00007f0071a42000,0x00007f0071b42000)]

  0x00007f00e40e5000 JavaThread "C2 CompilerThread3" daemon [_thread_blocked, id=63673, stack(0x00007f0071b43000,0x00007f0071c43000)]

  0x00007f00e40e2800 JavaThread "C2 CompilerThread2" daemon [_thread_blocked, id=63672, stack(0x00007f0071c44000,0x00007f0071d44000)]

  0x00007f00e40e1000 JavaThread "C2 CompilerThread1" daemon [_thread_blocked, id=63671, stack(0x00007f0071d45000,0x00007f0071e45000)]

  0x00007f00e40de000 JavaThread "C2 CompilerThread0" daemon [_thread_blocked, id=63670, stack(0x00007f0071e46000,0x00007f0071f46000)]

  0x00007f00e40dc000 JavaThread "Signal Dispatcher" daemon [_thread_blocked, id=63669, stack(0x00007f0071f46000,0x00007f0072047000)]

  0x00007f00e40ae800 JavaThread "Finalizer" daemon [_thread_blocked, id=63668, stack(0x00007f00721e2000,0x00007f00722e3000)]

  0x00007f00e40ac800 JavaThread "Reference Handler" daemon [_thread_blocked, id=63667, stack(0x00007f00722e3000,0x00007f00723e4000)]

Other Threads:

  0x00007f00e40a5000 VMThread [stack: 0x00007f00723e5000,0x00007f00724e5000] [id=63666]

  0x00007f00e410d800 WatcherThread [stack: 0x00007f0070e36000,0x00007f0070f36000] [id=63686]

VM state:not at safepoint (normal execution)

VM Mutex/Monitor currently owned by a thread: None

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Community Expert ,
Aug 02, 2019 Aug 02, 2019

Copy link to clipboard

Copied

There are a couple of things that jump out at me from this. First, you're using a really old version of Java 8. You'd probably be better off upgrading to a newer one. That might solve your problem right off the bat. Second, the problem you're having in Java is a "HotSpot" error. It is possible to disable HotSpot in jvm.config, and that might make the error go away. There's a significant performance penalty if you do this, though, so you might want to do that just for diagnostic purposes and see if you get a more helpful error condition that you can use.

Dave Watts, Eidolon LLC

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Community Expert ,
Aug 02, 2019 Aug 02, 2019

Copy link to clipboard

Copied

I was writing my last reply while Dave W wrote his, so I am just seeing it.

That's a good point about the JVM being old, also. I'd not be as inclined to expect that to lead to a crash like this, but it could.

As for the hotspot indication, I will just say (for readers) that all logs written in the cfusion/bin would indeed refer to hotspot. In fact, the files start with hs_err, which does stand for hotspot. That said, I'd again not really think that turning off hostpot would make a difference. I realize it may have in the past (or that people used to propose it for such odd errors as this, as a hail mary).

DaveO, I'd still recommend you check first about CF updates, and then as for updating the JVM, that's easy on the surface but can be challenging. I offer a list of common problems here:

http://www.carehart.org/blog/client/index.cfm/2014/12/11/help_I_updated_CFs_JVM_and_it_wont_start

I also have there a section pointing to resources on how to do it, but for some reason the post is truncated. I am working in that, but have to run now to help a client. I hope to get that resolved today. In the meantime, the rest of the info could help, in seeing at least what mistakes to avoid if you try it. (And I and others can help you do that, if interested. It takes just minutes--and a CF restart--if done right.)


/Charlie (troubleshooter, carehart.org)

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Community Expert ,
Aug 02, 2019 Aug 02, 2019

Copy link to clipboard

Copied

Alrighty then. Now we know for sure CF *is* dying. As for the log lines you shared, their starting lines really don't pinpoint the problem like these files usually would. Nor does the rest that you shared. As for the still-other content you did not share, that may or may not help.

  1. I'd propose first that you check what CF 11 update level you are on. The last was update 19, in June of this year. There could well be a bug that was addressed, and perhaps you have not applied the needed update
  2. Even if you HAVE the latest update, I would propose next that you make sure that there were no errors in the last CF11 update you (or anyone may have applied.) I have seen many times that people cried "bug, bug", only to find that it was due to an error in the application of the update. It can happen, and it's rather easily solved. I have a blog post showing both how can easily confirm if there were errors (by looking at yet-still-other logs, and for SPECIFIC text that confirms if the updates had no errors), as well as how to resolve things if there WERE errors in the last update applied. See more in the post here:
    Having problems after applying a CF update? What to check, and how to recover! - ColdFusion
  3. Next, if that does not solve things, as for what more may be in the log that you did not share, I will say first that I would NOT recommend you post the log text here, as that would indeed be very large. You could certainly put it in a dropbox or other shared storage solution, some of which are free (see my list at http://www.cf411.com/fileshare).
  4. But finally, really there may not be many here who could decipher what those logs would tell (in that it's not clear from the first few lines). It's getting into really low-level weeds of underlying CF/JVM/OS processing. I would say that if you confirm that you ARE on the latest CF11 update, and there were NO errors in the last CF11 update applied (per info in my link above), then you may want to file a bug report about this issue, at tracker.adobe.com. Adobe will see that and may reply, even though they may not here

Let us know how it all goes.


/Charlie (troubleshooter, carehart.org)

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
New Here ,
Aug 02, 2019 Aug 02, 2019

Copy link to clipboard

Copied

On this particular sever, I've set the limit at 900.    I have another tool which handles wildly complex call routing and it is not uncommon for hundreds of fields to be fed from a single form (though not always visible to the user)

Dave

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Community Expert ,
Aug 02, 2019 Aug 02, 2019

Copy link to clipboard

Copied

DaveO, your last response (about how many form fields you have) seems curious. The previous 5 messages (one from you, one from DaveW, and three from me) have had nothing to do with form fields. Indeed, my message from 5 before helped clarify that the form field limit would only cause you to get an error (have a page fail), not crash CF. So I thought we have moved past that to discussing instead why your CF is dying.

Or can you offer some context about why you just wrote that? More important, I hope you will be considering and responding to the previous 3 messages (from me and DaveW), which were responses to your sharing the hs_err_pid log info.


/Charlie (troubleshooter, carehart.org)

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
New Here ,
Aug 02, 2019 Aug 02, 2019

Copy link to clipboard

Copied

LATEST

Ah! I was just responding to a question from another message on the thread. We know the field limitation is certainly not the issue here

Thanks!

Dave

Sent from my iPhone

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Resources
Documentation