Apache Tomcat (on Linux) mod_jk Connector 403 Forbidden Error

[lynn___]

Hi All,

Hoping someone can provide some pointers to solve this.

I was successfully running on CF2023 update 11 but when I went to update 14 Apache Tomcat mod_jk connector stopped working - i've reverted back to update 11 and it still doesn't work - the browser returns 403 Forbidden

I've checked my server.xml, worker.properties, mod_jk_vhost.conf a million times and they seem to be correct.  I changed the Port being used by the connector in case that was a problem.

I'm seeing in mod_jk.log:

[Wed Jun 18 08:31:41 2025] [301783:140594684778048] [debug] ajp_connection_tcp_send_message::jk_ajp_common.c (1368): 0130 63 68 65 2F 32 2E 34 2E 36 32 20 28 55 6E 69 78 - che/2.4.62.(Unix
[Wed Jun 18 08:31:41 2025] [301783:140594684778048] [debug] ajp_connection_tcp_send_message::jk_ajp_common.c (1368): 0140 29 20 4F 70 65 6E 53 53 4C 2F 33 2E 33 2E 32 20 - ).OpenSSL/3.3.2.
[Wed Jun 18 08:31:41 2025] [301783:140594684778048] [debug] ajp_connection_tcp_send_message::jk_ajp_common.c (1368): 0150 6D 6F 64 5F 6A 6B 2F 31 2E 32 2E 34 36 00 0A 00 - mod_jk/1.2.46...
[Wed Jun 18 08:31:41 2025] [301783:140594684778048] [debug] ajp_connection_tcp_send_message::jk_ajp_common.c (1368): 0160 0F 41 4A 50 5F 52 45 4D 4F 54 45 5F 50 4F 52 54 - .AJP_REMOTE_PORT
[Wed Jun 18 08:31:41 2025] [301783:140594684778048] [debug] ajp_connection_tcp_send_message::jk_ajp_common.c (1368): 0170 00 00 01 30 00 0A 00 0E 41 4A 50 5F 4C 4F 43 41 - ...0....AJP_LOCA
[Wed Jun 18 08:31:41 2025] [301783:140594684778048] [debug] ajp_connection_tcp_send_message::jk_ajp_common.c (1368): 0180 4C 5F 41 44 44 52 00 00 00 00 0A 00 10 4A 4B 5F - L_ADDR.......JK_
[Wed Jun 18 08:31:41 2025] [301783:140594684778048] [debug] ajp_connection_tcp_send_message::jk_ajp_common.c (1368): 0190 4C 42 5F 41 43 54 49 56 41 54 49 4F 4E 00 00 03 - LB_ACTIVATION...
[Wed Jun 18 08:31:41 2025] [301783:140594684778048] [debug] ajp_connection_tcp_send_message::jk_ajp_common.c (1368): 01a0 41 43 54 00 FF 00 00 00 00 00 00 00 00 00 00 00 - ACT.............
[Wed Jun 18 08:31:41 2025] [301783:140594684778048] [debug] ajp_send_request::jk_ajp_common.c (1883): (cfusion) request body to send 0 - request body to resend 0
[Wed Jun 18 08:31:41 2025] [301783:140594684778048] [debug] ajp_connection_tcp_get_message::jk_ajp_common.c (1563): received from ajp13 pos=0 len=29 max=65536
[Wed Jun 18 08:31:41 2025] [301783:140594684778048] [debug] ajp_connection_tcp_get_message::jk_ajp_common.c (1563): 0000 04 01 F4 00 15 49 6E 74 65 72 6E 61 6C 20 53 65 - .....Internal.Se
[Wed Jun 18 08:31:41 2025] [301783:140594684778048] [debug] ajp_connection_tcp_get_message::jk_ajp_common.c (1563): 0010 72 76 65 72 20 45 72 72 6F 72 00 00 00 00 00 00 - rver.Error......
[Wed Jun 18 08:31:41 2025] [301783:140594684778048] [debug] ajp_unmarshal_response::jk_ajp_common.c (818): (cfusion) status = 500
[Wed Jun 18 08:31:41 2025] [301783:140594684778048] [debug] ajp_unmarshal_response::jk_ajp_common.c (825): Number of headers is = 0
[Wed Jun 18 08:31:41 2025] [301783:140594684778048] [debug] ajp_connection_tcp_get_message::jk_ajp_common.c (1563): received from ajp13 pos=0 len=2 max=65536
[Wed Jun 18 08:31:41 2025] [301783:140594684778048] [debug] ajp_connection_tcp_get_message::jk_ajp_common.c (1563): 0000 05 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 - ................
[Wed Jun 18 08:31:41 2025] [301783:140594684778048] [warn] ajp_process_callback::jk_ajp_common.c (2263): (cfusion) AJP13 protocol: Reuse is set to false
[Wed Jun 18 08:31:41 2025] [301783:140594684778048] [debug] ajp_reset_endpoint::jk_ajp_common.c (930): (cfusion) resetting endpoint with socket 19 (socket shutdown)
[Wed Jun 18 08:31:41 2025] [301783:140594684778048] [debug] ajp_abort_endpoint::jk_ajp_common.c (900): (cfusion) aborting endpoint with socket 19
[Wed Jun 18 08:31:41 2025] [301783:140594684778048] [debug] jk_shutdown_socket::jk_connect.c (931): About to shutdown socket 19 [127.0.0.1:20726 -> 127.0.0.1:8009]
[Wed Jun 18 08:31:41 2025] [301783:140594684778048] [debug] jk_is_input_event::jk_connect.c (1410): error event during poll on socket 19 [errno=107] (event=16)
[Wed Jun 18 08:31:41 2025] [301783:140594684778048] [debug] jk_shutdown_socket::jk_connect.c (1015): Shutdown socket 19 [127.0.0.1:20726 -> 127.0.0.1:8009] and read 0 lingering bytes in 0 sec.
[Wed Jun 18 08:31:41 2025] [301783:140594684778048] [debug] ajp_done::jk_ajp_common.c (3710): recycling connection pool for worker cfusion and socket

server.xml contains:

<Connector packetSize="65535" protocol="AJP/1.3" port="8009" redirectPort="8455" secret="27837689-fc18-446c-848a-ae113acf00c8" maxThreads="500" connectionTimeout="60000" tomcatAuthentication="false" address="127.0.0.1" allowedRequestAttributesPattern=".*" />

workers.properties contains:

heartbeat_interval=30
heartbeat_limit=90

#Start of workers.properties associated with 'cfusion'
worker.list=cfusion

worker.cfusion.type=ajp13
worker.cfusion.host=localhost
worker.cfusion.port=8009
worker.cfusion.heartbeat_servlet_path=/__cf_connector_heartbeat__
worker.cfusion.connection_pool_timeout=60
worker.cfusion.monitoringsecret=f68691fb-903c-4c39-9166-ddd7fa726cae
worker.cfusion.secret=27837689-fc18-446c-848a-ae113acf00c8
#End of workers.properties associated with 'cfusion'

I'm going crazy trying to solve this - dug through all suggestions I can find online.  I hope someone can help.  Thanks!

[BKBK]

Could it be that installing Update 14 then reverting to Update 11 broke something in Apache? 

Suggestion for test:

1.  Install ColdFusion Update 14.
2.  Re-install and configure Apache.

[lynn___]

I'd been holding off doing re-installs to see if it could be fixed but seems like this is the next step.  I'll work on this over the next few days.  I'll respond back as soon as I have the work completed.

[lynn___]

sorry,  I was away for a few days

the access_log and error_log you requested I share are attached (i started them both fresh)

[Charlie Arehart]

Lynn, while you await assessment of the logs by bkbk, Pete, or others (and before you give up and reinstall), I'll ask again if you've been able to access the cf admin. If so, that would not be via apache, and that's one reason it's worth asking about. I'll await your answer before elaborating. 

Also, your access log shows only calls to a healthcheck.cfm. Are you able to run any cf templates?

Finally, what happens if you create a new folder in your apache root, and in it create an empty Application.cfm and an empty test.cfm, and request that?

This is all trying to narrow down where the problem may (or may not) be. 

[lynn___]

Ooooops!!! So sorry!!!  I do REALLY appreciate everyones help and suggestions!

Yes, the CF Admin works fine - so the Tomcat is working on it's own.

Still same 403 with an empty Application.cfm and empty test.cfm.  FYI the healthcheck.cfm file only has static html, no CF code.  I figured that needed to work before I tested code working.

I also deleted  allowedRequestAttributesPattern=".*" per BKBK

the mod_jk log is attached for testing the empty test.cfm

[Charlie Arehart]

Lynn, one subtle but potentially important point: did you put those two new files in a folder of their own? That's important to ensure there's no impact of any application.cfc in any existing folder you put them in. That's the main reason why I suggested it.

A secondary one is that since you're starting cf from the command line,  the user you're running would be the one to create this new folder, and that would take out one more possible cause of the 403.

Assuming that still fails with a 403, I think it's time to ask you to let us see more of the mod_jk.log--and since you have debug logging enabled, we'd want to see especially the first couple hundred lines after you restart apache, and make a first request. During the initial loading of the connector, the log will have lots of diagnostic lines listing more about your configuration. There may be things that would surprise you, and us.

But again, don't give us the whole log. Make sure it starts with info about your wsconfig setup. And don't include beyond the first request you make, just to keep the log you share small and focused. You may want to mask the secret values shown there...but I doubt your AJP port for cf is opened by your firewall, to have to worry so much about it. Just double check (before removing it for us) that it's indeed the same value in the server.xml, as you asserted originally.

Finally, if you grow weary of the back and forth here with us all, note that I (and Pete, but I don't think bkbk) offer remote consulting. I appreciate many will feel they can't do that for any number of reasons. Just know it's an option. Sometimes with direct viewing/assessment we can make such problems go away much faster. More on my rates, approach, satisfaction guarantee, and online calendar with slots each day at carehart.org/consulting.

At a minimum, do try a new folder. And only then consider sharing the top of the debug logging after an apache restart. 

[lynn___]

Thanks, Charlie.  I'm close to being willing to pay you out of my own pocket but I feel like I'd be punished for doing that.

As user 'cfusion' that CF runs under, I created a new directory

drwxrwxr-x. 2 cfusion postdrop 45 Jun 30 20:37 newdir

with 2 new empty files

-rw-rw-r--. 1 cfusion postdrop 0 Jun 30 20:37 Application.cfm
-rw-rw-r--. 1 cfusion postdrop 0 Jun 30 20:37 test.cfm
log files are attached (scrubbed a bit)

[BKBK]

Thanks for the logs. Could you share the code of the CFM page for which you get the 403 Forbidden error.

An elephant-in-the-room test I am curious about: what happens when you delete  allowedRequestAttributesPattern=".*"   from the connector, and then restart ColdFusion?

[lynn___]

BKBK,

Hopefully you see my response to Charlie. 
- no change w/ deleting allowedRequestAttributesPattern=".*"  

- the healthcheck.cfm page i was getting the 403 for was only static html

- also get 403 for a blank file with .cfm extension

[BKBK]

@lynn___ , thanks for the answers to the recent questions. I noticed from the logs that the value of the setting worker.cfusion.host seems to have gone back to localhost. It was suggested at the very beginning that you should change it to 127.0.0.1.

To confirm, the settings are:

workers.properties

heartbeat_interval=30
heartbeat_limit=90

worker.list=cfusion
worker.cfusion.type=ajp13
worker.cfusion.host=127.0.0.1
worker.cfusion.port=8009
worker.cfusion.heartbeat_servlet_path=/__cf_connector_heartbeat__
worker.cfusion.connection_pool_timeout=60
worker.cfusion.secret=27837689-fc18-446c-848a-ae113acf00c8
worker.cfusion.connection_pool_size=500
worker.cfusion.max_reuse_connections=500

server.xml

<Connector packetSize="65535" protocol="AJP/1.3" port="8009" redirectPort="8455" secretRequired="true" secret="27837689-fc18-446c-848a-ae113acf00c8" maxThreads="500" connectionTimeout="60000" tomcatAuthentication="false" address="127.0.0.1" allowedRequestAttributesPattern=".*" />

As you can see, I have added secretRequired="true" to server.xml. I am aware that, when the setting is absent, the default value is True. 

However my thinking is as follows. Without secretRequired, ColdFusion’s Tomcat connector may silently reject the request as “unsafe,” especially over SSL or when redirectPort is used.

Change to the above settings, restart ColdFusion and then Apache, and see what happens.

[lynn___]

Still no luck.

workers.properties contains:

heartbeat_interval=30
heartbeat_limit=90

#Start of workers.properties associated with 'cfusion'
worker.list=cfusion

worker.cfusion.type=ajp13
worker.cfusion.host=127.0.0.1
worker.cfusion.port=8009
worker.cfusion.heartbeat_servlet_path=/__cf_connector_heartbeat__
worker.cfusion.connection_pool_timeout=60
#worker.cfusion.monitoringsecret=f68691fb-903c-4c39-9166-ddd7fa726cae
worker.cfusion.secret=27837689-fc18-446c-848a-ae113acf00c8
worker.cfusion.connection_pool_size=500
worker.cfusion.max_reuse_connections=500
#End of workers.properties associated with 'cfusion'

server.xml connector:

<Connector packetSize="65535" protocol="AJP/1.3" port="8009" redirectPort="8455" secretRequired="true" secret="27837689-fc18-446c-848a-ae113acf00c8" maxThreads="500" connectionTimeout="60000" tomcatAuthentication="false" address="0.0.0.0" allowedRequestAttributesPattern=".*" />

Logs are attached

[BKBK]

Hi @lynn___ , thanks for the update and for the logs. When the test with address="0.0.0.0" didn't resolve the issue, you should have reverted to address="127.0.0.1" . Hence, the connector settings

<Connector packetSize="65535" protocol="AJP/1.3" port="8009" redirectPort="8455" secretRequired="true" secret="27837689-fc18-446c-848a-ae113acf00c8" maxThreads="500" connectionTimeout="60000" tomcatAuthentication="false" address="127.0.0.1" allowedRequestAttributesPattern=".*" />

As usual, restart and see what happens. Even if that doesn't resolve the issue, it helps to avoid clutter as we move forward.

[lynn___]

oops - I forgot to change that back.  updated but still same.  logs attached.

[BKBK]

I noticed at once something new. Coldfusion-error.log contains the following error message:

Jul 01, 2025 8:52:19 PM org.apache.coyote.ajp.AjpProcessor service
SEVERE: Error processing request
java.lang.NullPointerException: Cannot invoke "java.lang.CharSequence.length()" because "this.text" is null
	at java.base/java.util.regex.Matcher.getTextLength(Matcher.java:1769)
	at java.base/java.util.regex.Matcher.reset(Matcher.java:415)
	at java.base/java.util.regex.Matcher.<init>(Matcher.java:252)
	at java.base/java.util.regex.Pattern.matcher(Pattern.java:1134)
	at org.apache.catalina.valves.RequestFilterValve.isAllowed(RequestFilterValve.java:418)
	at org.apache.catalina.valves.RequestFilterValve.process(RequestFilterValve.java:354)
	at org.apache.catalina.valves.RemoteAddrValve.invoke(RemoteAddrValve.java:54)
	at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:660)
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74)
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:357)
	at org.apache.coyote.ajp.AjpProcessor.service(AjpProcessor.java:448)
	at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:63)
	at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:936)
	at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1791)
	at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:52)
	at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1190)
	at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)
	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:63)
	at java.base/java.lang.Thread.run(Thread.java:833)

That leads me to ask: do you use a load balancer? If so, it might be necessary to implement an InternalProxies valve in server.xml.  Something like

<!-- Internal proxies are localhost (127.0.0.1, ::1 or 0:0:0:0:0:0:0:1) and IPs in the range 10.34.x.x -->
<Valve className="org.apache.catalina.valves.RemoteIpValve"
	internalProxies="127\.0\.0\.1|::1|0:0:0:0:0:0:0:1|10\.34\.\d+\.\d+"
	remoteIpHeader="x-forwarded-for"
	protocolHeader="x-forwarded-proto">
</Valve>

If you don't use a load balancer, it might be necessary to implement a RemoteAddrValve in server.xml.  Something like

<!-- Any other stuff within <Host></Host> that has been commented out, remains commented out. The allowed remote addresses are localhost (127.0.0.1, ::1 or 0:0:0:0:0:0:0:1) and IPs in the range 10.34.x.x -->
<Host autoDeploy="false" appBase="webapps" name="localhost" unpackWARs="false">
	<Valve className="org.apache.catalina.valves.RemoteAddrValve" allow="127\.0\.0\.1|::1|0:0:0:0:0:0:0:1|10\.34\.\d+\.\d+" />
</Host>

[BKBK]

Your most recent logs have the remote address 144.117.227.85. It suggests you should add 144\.117\.\d+\.\d+ to the "internalProxies" or "allow" list, whichever one you choose to use. 

[lynn___]

Thanks for all your recommendations BKBK.  Unfortunately I'm not getting anywhere and think at this point it'll be more efficient to just try to start for scratch rather than waste more time trying to get it fixed.  I do really appreciate everyone's efforts in trying to help me with this.

[BKBK]

Fair enough. Sorry that we haven't yet resolved the issue. 

However, some questions remain unanswered, such as: Is a load balancer being used? What explains the changing remote addresses  144.117.227.85, 10.34.43.122, ... 

From what I've seen so far, the issue arises from blocked access permissions. The machine(s) the test.cfm and healthcheck.cfm requests are coming from are not allowed. 

Sometimes, the web server or reverse proxy in front of ColdFusion may filter requests based on Host header, IP, X-Forwarded-For or X-Forwarded-Host. The internal IP the requests are coming from might be blocked.

The Null host in coldfusion-error.log and mod_jk log imply that you should add the following in Apache's VirtualHost configuration:

RequestHeader set Host "active-staging-lab.web.XXXX.com"

[BKBK]

There is something you might want to rule out before you re-install. Is the server.xml that you've been editing actually stored as a file of XML type? What if, for whatever reason, ColdFusion is not seeing the changes you've been making in server.xml?

[lynn___]

I went ahead and reinstalled - which was much faster (although frustrating not to understand why it wasn't working - maybe something corrupt somewhere?).  So I'm back running successfully on CF2023 update 11.  I still need to go to update 14 but needless to say I'm a bit terrified of doing so...

[Charlie Arehart]

Well, first of course it's good you're back in business. But it's definitely not a good place to remain, feeling stuck at update 11 (from last October).

And while normally someone could recommend "just do the update and uninstall it if something goes amiss", that didn't work for you last time. (I'm still surprised that the update alone would have so ill-effected your cf/apache connector setup--as the update does not touch that, nor does update 14 or even 11 call for a separate connector update.)

If you're in agreement that "staying where you are" (on update 11) is undesirable, here's what you do could do:

• Stop cf and apache
• Make a temporary copy of the cf folder (which may be a gig or more), and a copy of at least the apache conf folder.
• Apply the cf update, which will restart cf, then start apache and test things
• If things went well, great. Chalk up the previous issue to a ghost 
• If instead things fail again, do the uninstall of the cf update
• Confirm whether things are well.

If things are still broken after the uninstall of the update, then a) at least you can do a comparison of the the copied folders to the current ones (for each of cf and apache conf) to find what differs even after uninstalling the update. 

Then b) you should be able to swap the copied folders for the current ones (while cf and apache are stopped), and THAT should get you back to working, albeit on update 11.

Similarly, If uninstalling update 14 alone (in the steps above) gets you working, at least that confirms that uninstalling the update did work as it should. While you could stop again, instead now you could try update 14 again but then this time do the comparison of the backed up folders to those on THAT failing update 14 version of things, which may help spot what's amiss there. 

I realize this will sound like "a lot of work". It could be just minutes, with a good compare tool. Just trying to help.

I've not heard anyone else hit your problem, so I do look forward to helping you get to the bottom of whatever is amiss--or maybe the update to 14 will just go well.

Let us know what you think and how things go. 

[lynn___]

Thanks, Charlie.  I really appreciate your recomendations - not feeling all alone out here.  I'll report back how it goes.

[lynn___]

OMG!!! IT WAS THE GHOSTS!!!!

I upgraded to Update 15 (just came out!) and in general, it worked! I can open a basic .cfm page!!!! HOORAY!

I do have some issues I could use everyone's help on but at least at basic, it works. So relieved!

Here are the 1st 2 things I've encountered:

- The oracle package is not installed. You can install the package through the CLI package manager (/appl/ColdFusion2023/cfusion/bin/cfpm.sh) by running the command : install oracle.
       - I tried going to cfpm and running "cfpm>install oracle:2023.0.11.330706" and I get:
              Downloading the dependent package bcpkix-jdk15on-153.jar
              null
              Downloading the dependent package bcpkix-jdk15on-153.jar
              null
              oracle(2023.0.11.330706) package is already installed.

- Cannot find implementation class coldfusion.tagext.mail.MailTag for the mail tag
       - similarly I tried going to cfpm and running "cfpm>install mail:2023.0.11.330706" and I get:
              Downloading the dependent package bcpkix-jdk15on-153.jar
              null
              mail(2023.0.11.330706) package is already installed.

I've attached the coldfusion-out.log -- I see there is some potentially relevant info there

Any thoughts you all have to resolve these is, as always, very much appreciated. I couldn't have gotten here without your support!

Lynn

[Charlie Arehart]

Folks seeing the note from Lynn above this: don't reply here. She posted it again (trying to be helpful), but it's now on the 3rd page of this huge thread. Let's not have replied to both. 🙂 See and reply to her her later version of this reply above  one. 

[BKBK]

Hi @lynn___ , thanks for the update. I too am relieved. I am glad to see it confirmed that something went wrong during the installation and subsequent de-installation of Update 14.

Your fear to have another go at installing Update 14 is justified. It's a pain when a service goes down, and nothing you do wakes it up.

In any case, your last post suggests something else of great importance: apparently you don't have a test, acceptance or some preproduction environment.

It is vital that you have one. You can set it up in a few hours. It can be as simple as a developer's laptop which has the same Linux, ColdFusion and Apache versions as the production server. This is one of those use-cases where the few hours you invest now will save you hundreds of hours in the long-run. Not to mention the headaches.

[lynn___]

I'm not sure what I said but be assured that this is a pre-production environment!  It's not even in a production environmnet yet.  I'd never do this in prod without having done in pre-prod first. That's terrifying to even think about! 🙂