Coldfusion 2021 Add-On Services PDFg (Windows)

Report · Jan 31, 2022

Trying to install and connect to the PDFg Service. Windows Server. Installation of Add-On Services goes just fine and it starts up the Windows Service just fine. In CF Admin, under the PDF Service page, I try to "Verify" the connection and it fails saying "Connection verification failed for PDF Service Manager. Looking at the logs in the Coldfusion Add-On Services folder, the stderrout.log shows the service successfully starting up on port 8993. However the next several lines show the error from when I try to verify the service. I can't find any similar issues like this so turning to the community for help.

Note: neo-document.xml has the same port (8993) that the service is running on.

Any other ideas out there? Log entries are below:

Logs Upon Service Startup:

2022-01-31 22:08:34.374:INFO:oejs.Server:main: jetty-9.4.31.v20200723; built: 2020-07-23T17:57:36.812Z; git: 450ba27947e13e66baa8cd1ce7e85a4461cacc1d; jvm 11.0.13+10-LTS-370
2022-01-31 22:08:34.405:INFO:oejdp.ScanningAppProvider:main: Deployment monitor [file:///C:/ColdFusionAdd-onServices/webapps/] at interval 1
2022-01-31 22:08:34.827:INFO:oejs.session:main: DefaultSessionIdManager workerName=node0
2022-01-31 22:08:34.827:INFO:oejs.session:main: No SessionScavenger set, using defaults
2022-01-31 22:08:34.827:INFO:oejs.session:main: node0 Scavenging every 600000ms
2022-01-31 22:08:34.874:INFO:oejsh.ContextHandler:main: Started o.e.j.w.WebAppContext@3e62d773{AdminServlet,/AdminServlet,file:///C:/ColdFusionAdd-onServices/work/jetty-0_0_0_0-8993-AdminServlet_war-_AdminServlet-any-/webapp/,AVAILABLE}{C:\ColdFusionAdd-onServices\webapps\AdminServlet.war}
2022-01-31 22:08:34.983:INFO:oejsh.ContextHandler:main: Started o.e.j.w.WebAppContext@22ffa91a{PDFgServlet,/PDFgServlet,file:///C:/ColdFusionAdd-onServices/webapps/PDFgServlet/,AVAILABLE}{C:\ColdFusionAdd-onServices\webapps\PDFgServlet}
2022-01-31 22:08:34.999:INFO:oejs.RequestLogWriter:main: Opened C:\ColdFusionAdd-onServices\logs\2022_02_01.request.log
2022-01-31 22:08:35.030:INFO:oejs.AbstractConnector:main: Started ServerConnector@68eedb9e{HTTP/1.1, (http/1.1)}{0.0.0.0:8993}
2022-01-31 22:08:35.045:INFO:oejs.Server:main: Started @2137ms

Logs Upon Verifying from CF Admin

2022-01-31 22:10:25.233:WARN:oejshC.PDFgServlet:qtp1794717576-24: unavailable
java.util.ConcurrentModificationException
at java.base/java.util.ArrayList$Itr.checkForComodification(ArrayList.java:1043)
at java.base/java.util.ArrayList$Itr.next(ArrayList.java:997)
at coldfusion.pdf.service.CFPDFGServicePool.createPDFgServicesPool(CFPDFGServicePool.java:91)
at coldfusion.pdf.service.CFPDFGServicePool.<init>(CFPDFGServicePool.java:53)
at coldfusion.pdf.service.CFPDFGServicePool.getServicePoolInstance(CFPDFGServicePool.java:59)
at coldfusion.pdf.service.CFPDFGServiceManager.initPDFServiceManager(CFPDFGServiceManager.java:77)
at coldfusion.pdf.service.remote.PDFGServlet.init(PDFGServlet.java:228)
at javax.servlet.GenericServlet.init(GenericServlet.java:244)
at org.eclipse.jetty.servlet.ServletHolder$WrapperServlet.init(ServletHolder.java:1289)
at org.eclipse.jetty.servlet.ServletHolder.initServlet(ServletHolder.java:602)
at org.eclipse.jetty.servlet.ServletHolder.getServlet(ServletHolder.java:473)
at org.eclipse.jetty.servlet.ServletHolder.prepare(ServletHolder.java:728)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:564)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:602)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1610)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1377)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:507)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1580)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1292)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:234)
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.eclipse.jetty.server.Server.handle(Server.java:501)
at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383)
at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:556)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:273)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)
at java.base/java.lang.Thread.run(Thread.java:834)
2022-01-31 22:10:25.237:WARN:oejs.HttpChannel:qtp1794717576-24: /PDFgServlet/verify
javax.servlet.ServletException: javax.servlet.ServletException: PDFgServlet@9ac5bab0==coldfusion.pdf.service.remote.PDFGServlet,jsp=null,order=-1,inst=true,async=false
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:162)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.eclipse.jetty.server.Server.handle(Server.java:501)
at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383)
at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:556)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:273)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)
at java.base/java.lang.Thread.run(Thread.java:834)
Caused by:
javax.servlet.ServletException: PDFgServlet@9ac5bab0==coldfusion.pdf.service.remote.PDFGServlet,jsp=null,order=-1,inst=true,async=false
at org.eclipse.jetty.servlet.ServletHolder.initServlet(ServletHolder.java:620)
at org.eclipse.jetty.servlet.ServletHolder.getServlet(ServletHolder.java:473)
at org.eclipse.jetty.servlet.ServletHolder.prepare(ServletHolder.java:728)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:564)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:602)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1610)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1377)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:507)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1580)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1292)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:234)
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.eclipse.jetty.server.Server.handle(Server.java:501)
at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383)
at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:556)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:273)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)
at java.base/java.lang.Thread.run(Thread.java:834)
Caused by:
java.util.ConcurrentModificationException
at java.base/java.util.ArrayList$Itr.checkForComodification(ArrayList.java:1043)
at java.base/java.util.ArrayList$Itr.next(ArrayList.java:997)
at coldfusion.pdf.service.CFPDFGServicePool.createPDFgServicesPool(CFPDFGServicePool.java:91)
at coldfusion.pdf.service.CFPDFGServicePool.<init>(CFPDFGServicePool.java:53)
at coldfusion.pdf.service.CFPDFGServicePool.getServicePoolInstance(CFPDFGServicePool.java:59)
at coldfusion.pdf.service.CFPDFGServiceManager.initPDFServiceManager(CFPDFGServiceManager.java:77)
at coldfusion.pdf.service.remote.PDFGServlet.init(PDFGServlet.java:228)
at javax.servlet.GenericServlet.init(GenericServlet.java:244)
at org.eclipse.jetty.servlet.ServletHolder$WrapperServlet.init(ServletHolder.java:1289)
at org.eclipse.jetty.servlet.ServletHolder.initServlet(ServletHolder.java:602)
at org.eclipse.jetty.servlet.ServletHolder.getServlet(ServletHolder.java:473)
at org.eclipse.jetty.servlet.ServletHolder.prepare(ServletHolder.java:728)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:564)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:602)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1610)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1377)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:507)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1580)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1292)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:234)
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.eclipse.jetty.server.Server.handle(Server.java:501)
at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383)
at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:556)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:273)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)
at java.base/java.lang.Thread.run(Thread.java:834)

Report · Feb 01, 2022

Which logs are those?

What happens when you launch the following test code as a CFM page?

<cftry>
	<cfhttp url="http://127.0.0.1:8993/PDFgServlet/verify" method="get" result="httpResult"></cfhttp>
	<cfdump var="#httpResult#" label="Verify PDFg service using CFHTTP - Result">
<cfcatch type="any" >
	<cfdump var="#cfcatch#" label="Verify PDFg service using CFHTTP - Error">
</cfcatch>	
</cftry>

If you continue to get java.util.ConcurrentModificationException with the above test, it will mean race conditions. That is, two or more threads are simultaneously trying to get access to the same servlet. That will in turn suggest that two instances of the PDFg service are running.

Might the root cause be as follows, You first installed an instance of the PDFg service along with ColdFusion. You then followed that with a second, standalone instance of the service.

Report · Feb 01, 2022

Thanks BKBK,

Entirely possible that it is installed twice but wouldn't there be a /jetty folder then underneath /cfusion?

The logs I referred to were logs that are in the "ColdFusion2021Add-OnServices\logs" folder.

When I run the code you provide, I just get a generic 500 Server error as follows.

How can I tell if another instance is running (there is only one in the Windows Services).

Report · Feb 01, 2022

BKBK, another potential possibility is we copied the settings from the CF 2016 installation and did a migrate. On the 2016 Server, it definitely was installed with ColdFusion. Is it possible CF 2021 thinks that is still the case because of migrating the settings and then installing it as an Add-On Service, threw this into a race condition as you mention? Not sure but thought I'd add this info.

Report · Feb 01, 2022

BKBK, another potential possibility is we copied the settings from the CF 2016 installation and did a migrate. On the 2016 Server, it definitely was installed with ColdFusion. Is it possible CF 2021 thinks that is still the case because of migrating the settings and then installing it as an Add-On Service, threw this into a race condition as you mention?

By @MikePelant

That is worth looking into. Could you share the full list of settings that you copied?

Report · Feb 01, 2022

I have a change of mind. The root cause of the problem is unlikely to be double PDFg services. I am ruling this out after a test.

When I installed ColdFusion 2021, I enabled the installation of the accompanying PDFg service. On the PDF service page in the ColdFusion Administrator, the PDF service verified OK on port 8993.

Following your comments, I decided to test my own hypothesis. So I downloaded and installed the standalone Add-on Services alongside ColdFusion 2021.

However, everything seems to work just fine. There are no errors - in fact, no log files - in C:\ColdFusionAdd-onServices\logs. Plus, in the ColdFusion Administrator, the PDF service again verifies OK on port 8993. That is why I am ruling out double PDFg services as the root cause.

But that test gave me another idea. What if you install the PDF service that is packaged in ColdFusion? For example, via the ColdFusion Administrator. You could, for test purposes, install all packages available in CF2021 to ensure that you include all possible dependencies.

Report · Feb 02, 2022

@BKBK , after all of your questions regarding whether it was installed during the intial installation, I did some more digging and I'm embarrassed to admit it DOES appear that it was installed originally. Therefore, no need for the separate Add-On Services installation. That would likely explain the ConcurrentModificationException.

So, with that knowledge, I completely uninstalled the Add-On installation, rebooted the server, and went back into the Admin Console to verify the PDF Service. Still getting the "Connection verification failed" message and the Server.log file indicating:

"Error","http-nio-8500-exec-8","02/02/22","09:58:45","","Connect to 127.0.0.1:8987 [/127.0.0.1] failed: Connection refused: connect http://127.0.0.1:8987/PDFgServlet/verify"
"Error","http-nio-8500-exec-8","02/02/22","09:58:45","","Connection verification failed. The root cause is : Connect to 127.0.0.1:8987 [/127.0.0.1] failed: Connection refused: connect"

Also, there is nothing in the Server.log file when the CF Service is restarted, that indicates the PDFg Service is running on a specific port. All I see that seems to be related to the PDFg service is "Package htmltopdf started...".

I've even tried to remove the htmltopdf package and re-add it (restarting CF inbetween). I still can't get it to connect.

So, I guess I'm back to the beginning here.

Report · Feb 02, 2022

I can see there has been a change in port number.

Perhaps a required package hasn't been installed.
Open the ColdFusion Administrator. Go to Package Manager > Available Packages.
Make sure every package is installed.
Restart the Add-On windows service.
Restart ColdFusion.
Go to the PDF Service page in the Administrator.
Click on the button to verify the PDF Service Manager.
Any luck?
If not, then perhaps port 8987 is not available.
In any case, we could give port 8993 a go.
This may sound strange, but I would suggest you download and install the standalone Add-On service. Yes, alongside the one packaged in ColdFusion.
Open \cfusion\jetty\start.ini in a text editor.
Ensure the value of jetty.http.port is 8993. Save the start.ini file.
Open the Command Prompt as Administrator.
Using CD, navigate to the directory C:\ColdFusion2021\cfusion\jetty.
Using the Java that ColdFusion 2021 is running on, do a "dry run" of the Jar file start.jar. For example, on my system, the command is: "C:\Program Files\Java\jdk-11.0.14\bin\java" -jar start.jar --dry-run

Restart ColdFusion.
Go to the PDF Service page in the Administrator.
Click on the button to edit the PDF Service Manager. Change the port to 8993 and click on the update button.
Click on the button to verify the PDF Service Manager.
Any luck?

Report · Feb 04, 2022

Thanks again for all of the suggestions. The port number was different in my response as I was trying different ports. Anyway, I'm going to have to come back to this probably early next week as other priorities are getting in my way. One note, however. There is no jetty folder under the cfusion directory leading me back to my original assumption that the PDFg Service was not installed with ColdFusion. In fact, when this server was implemented, we installed just the core ColdFusion and then added packages on top of it (including htmltopdf). At any rate, I'll come back to this next week and provide you some more updates.

Report · Feb 04, 2022

One note, however. There is no jetty folder under the cfusion directory leading me back to my original assumption that the PDFg Service was not installed with ColdFusion. In fact, when this server was implemented, we installed just the core ColdFusion and then added packages on top of it (including htmltopdf).

By @MikePelant

Did you try installing the PDF package via the Package Manager in the Administrator?