Waiting on java.util.concurrent.locks issue

[terribleted]

Setup: CF 2016 on MS Server 2016 w/IIS 10

Using Fusion Reactor trial and finding queries hanging.  Still not found an answer that directly impacts this condition. I did uncheck "save class files" restart the CF Server, rechecked "save class files" per another thread.  Does not seem to cooperate.

ajp-nio-8016-exec-11" Id=160 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@72e82b6c

   java.lang.Thread.State: WAITING

        at sun.misc.Unsafe.park(Native Method)

        - waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@72e82b6c

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

        at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:103)

        at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:31)

        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)

        at java.lang.Thread.run(Thread.java:745)

[Charlie Arehart]

Ted, that state (which you show the thread to be in ) is not a problem. It simply is a thread waiting, able to run a request. And what makes you think that the request you see as “hanging” is this one with the threaded ajp-nio-8016-exec-11?  Answering both these, and more, should help unravel the confusion you seem to be expressing.

1) So first, as I said, that thread is in a state that simply indicates it’s WAITING to run a request. And I say that not because it actually shows being in a state called “waiting”, but rather because it shows having called methods (to get to the point where it is, on the first line of the stack trace) which are the methods a thread will have called when it’s sitting there waiting to run a request. I compared it to my own running CF server, to confirm.

And indeed, all ajp threads would be in that state, if you had no requests running.

2) So if you wanted to find out why some one request was running (one of these “queries” you say are “hanging”), then you would want to look at the details FR can offer on that request (and if looking at the stack trace for such a thread, if as you say you see in FR that it’s running a query, then the stack trace would nearly always simply show that it’s waiting for CF to get a response from the DB server, so the stack trace is not especially helpful for that problem). More on that in a moment.

3) So I also asked how you got that thread id. Did you pull it from a FusionReactor Crash Protection email alert? If so, did you scroll down the email looking at all the threads and picked it out randomly? Or did you use the FR UI and its Resources>Threads>Stack Trace All feature (at fusionreactor/findex.htm?p=traceThread)?

In both cases, really, you don’t generally want to be doing that, looking blindly at all the threads to see if something in there looks interesting.  That’s the old way of doing troubleshooting, from years ago before better tools like FusionReactor (and SeeFusion, and even the soon-to-be obsoleted CF Enterprise Server Monitor). Before them, all Java people could do (and would tell CF people to do, and even Adobe technotes said) was to “take a thread dump or two and try to compare them”.

But with these better these tools, you can now see EXACTLY what requests ARE running, and you can drill into them to see what they ARE doing.

4) And most important (for solving that problem) you say you see some running queries. Have you looked at the DB server to see if it confirms that queries are running there (or not)? More on that in a moment, also.

5) And last, to your point about the “save class files”, nothing about (requests running queries or requests in this state of waiting to run a request) is related at all to that.

6) Again, I realize you may be grasping at straws, trying to understand your problems. And perhaps you’ve found blog posts or forum threads where people are throwing “the kitchen sink” at problems they had, so you are trying what they said. Often, what people write about has nothing to do at all with a problem you have. But it’s understandable that without your having much knowledge of such stuff coming in (and often because such blogs have rather little info to help), you can be left wondering.

7) And I assume you were hoping that FR could help you find and solve your problems. And indeed it can. In fact, I have done several webinars (at fusion-reactor.com/webinars) showing how to use it for troubleshooting, including the most recent which was specifically about troubleshooting query problems. Hope they may be helpful.

Finally, if you are pressed for time and just want to solve things (and learn how to use FR) rather than dig through docs and webinars (and blog posts), I will add that I can help with using it and indeed troubleshooting CF problems. I can work with you remotely, without you needing to give me remote access (we would use screensharing, where I just “look over your shoulder” and guide you, as I do dozens of clients each week. More on my rates, approach, satisfaction guarantee and more at carehart.org/consulting. (I hope those who hate “sales pitches” will note that I have offered ample help here, not only of the original problem but more on using FR in general and for this problem, before proposing my direct assistance. Some people would simply rather get help than dig around on their own, when in a bind.)

Let us know if any of this helps.

/charlie

[BKBK]

Hi terribleted​, my guess is: this has to do with a lock. Review the code. Do you implement a lock somewhere?