Apache Tomcat (on Linux) mod_jk Connector 403 Forbidden Error

[lynn___]

Hi All,

Hoping someone can provide some pointers to solve this.

I was successfully running on CF2023 update 11 but when I went to update 14 Apache Tomcat mod_jk connector stopped working - i've reverted back to update 11 and it still doesn't work - the browser returns 403 Forbidden

I've checked my server.xml, worker.properties, mod_jk_vhost.conf a million times and they seem to be correct.  I changed the Port being used by the connector in case that was a problem.

I'm seeing in mod_jk.log:

[Wed Jun 18 08:31:41 2025] [301783:140594684778048] [debug] ajp_connection_tcp_send_message::jk_ajp_common.c (1368): 0130 63 68 65 2F 32 2E 34 2E 36 32 20 28 55 6E 69 78 - che/2.4.62.(Unix
[Wed Jun 18 08:31:41 2025] [301783:140594684778048] [debug] ajp_connection_tcp_send_message::jk_ajp_common.c (1368): 0140 29 20 4F 70 65 6E 53 53 4C 2F 33 2E 33 2E 32 20 - ).OpenSSL/3.3.2.
[Wed Jun 18 08:31:41 2025] [301783:140594684778048] [debug] ajp_connection_tcp_send_message::jk_ajp_common.c (1368): 0150 6D 6F 64 5F 6A 6B 2F 31 2E 32 2E 34 36 00 0A 00 - mod_jk/1.2.46...
[Wed Jun 18 08:31:41 2025] [301783:140594684778048] [debug] ajp_connection_tcp_send_message::jk_ajp_common.c (1368): 0160 0F 41 4A 50 5F 52 45 4D 4F 54 45 5F 50 4F 52 54 - .AJP_REMOTE_PORT
[Wed Jun 18 08:31:41 2025] [301783:140594684778048] [debug] ajp_connection_tcp_send_message::jk_ajp_common.c (1368): 0170 00 00 01 30 00 0A 00 0E 41 4A 50 5F 4C 4F 43 41 - ...0....AJP_LOCA
[Wed Jun 18 08:31:41 2025] [301783:140594684778048] [debug] ajp_connection_tcp_send_message::jk_ajp_common.c (1368): 0180 4C 5F 41 44 44 52 00 00 00 00 0A 00 10 4A 4B 5F - L_ADDR.......JK_
[Wed Jun 18 08:31:41 2025] [301783:140594684778048] [debug] ajp_connection_tcp_send_message::jk_ajp_common.c (1368): 0190 4C 42 5F 41 43 54 49 56 41 54 49 4F 4E 00 00 03 - LB_ACTIVATION...
[Wed Jun 18 08:31:41 2025] [301783:140594684778048] [debug] ajp_connection_tcp_send_message::jk_ajp_common.c (1368): 01a0 41 43 54 00 FF 00 00 00 00 00 00 00 00 00 00 00 - ACT.............
[Wed Jun 18 08:31:41 2025] [301783:140594684778048] [debug] ajp_send_request::jk_ajp_common.c (1883): (cfusion) request body to send 0 - request body to resend 0
[Wed Jun 18 08:31:41 2025] [301783:140594684778048] [debug] ajp_connection_tcp_get_message::jk_ajp_common.c (1563): received from ajp13 pos=0 len=29 max=65536
[Wed Jun 18 08:31:41 2025] [301783:140594684778048] [debug] ajp_connection_tcp_get_message::jk_ajp_common.c (1563): 0000 04 01 F4 00 15 49 6E 74 65 72 6E 61 6C 20 53 65 - .....Internal.Se
[Wed Jun 18 08:31:41 2025] [301783:140594684778048] [debug] ajp_connection_tcp_get_message::jk_ajp_common.c (1563): 0010 72 76 65 72 20 45 72 72 6F 72 00 00 00 00 00 00 - rver.Error......
[Wed Jun 18 08:31:41 2025] [301783:140594684778048] [debug] ajp_unmarshal_response::jk_ajp_common.c (818): (cfusion) status = 500
[Wed Jun 18 08:31:41 2025] [301783:140594684778048] [debug] ajp_unmarshal_response::jk_ajp_common.c (825): Number of headers is = 0
[Wed Jun 18 08:31:41 2025] [301783:140594684778048] [debug] ajp_connection_tcp_get_message::jk_ajp_common.c (1563): received from ajp13 pos=0 len=2 max=65536
[Wed Jun 18 08:31:41 2025] [301783:140594684778048] [debug] ajp_connection_tcp_get_message::jk_ajp_common.c (1563): 0000 05 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 - ................
[Wed Jun 18 08:31:41 2025] [301783:140594684778048] [warn] ajp_process_callback::jk_ajp_common.c (2263): (cfusion) AJP13 protocol: Reuse is set to false
[Wed Jun 18 08:31:41 2025] [301783:140594684778048] [debug] ajp_reset_endpoint::jk_ajp_common.c (930): (cfusion) resetting endpoint with socket 19 (socket shutdown)
[Wed Jun 18 08:31:41 2025] [301783:140594684778048] [debug] ajp_abort_endpoint::jk_ajp_common.c (900): (cfusion) aborting endpoint with socket 19
[Wed Jun 18 08:31:41 2025] [301783:140594684778048] [debug] jk_shutdown_socket::jk_connect.c (931): About to shutdown socket 19 [127.0.0.1:20726 -> 127.0.0.1:8009]
[Wed Jun 18 08:31:41 2025] [301783:140594684778048] [debug] jk_is_input_event::jk_connect.c (1410): error event during poll on socket 19 [errno=107] (event=16)
[Wed Jun 18 08:31:41 2025] [301783:140594684778048] [debug] jk_shutdown_socket::jk_connect.c (1015): Shutdown socket 19 [127.0.0.1:20726 -> 127.0.0.1:8009] and read 0 lingering bytes in 0 sec.
[Wed Jun 18 08:31:41 2025] [301783:140594684778048] [debug] ajp_done::jk_ajp_common.c (3710): recycling connection pool for worker cfusion and socket

server.xml contains:

<Connector packetSize="65535" protocol="AJP/1.3" port="8009" redirectPort="8455" secret="27837689-fc18-446c-848a-ae113acf00c8" maxThreads="500" connectionTimeout="60000" tomcatAuthentication="false" address="127.0.0.1" allowedRequestAttributesPattern=".*" />

workers.properties contains:

heartbeat_interval=30
heartbeat_limit=90

#Start of workers.properties associated with 'cfusion'
worker.list=cfusion

worker.cfusion.type=ajp13
worker.cfusion.host=localhost
worker.cfusion.port=8009
worker.cfusion.heartbeat_servlet_path=/__cf_connector_heartbeat__
worker.cfusion.connection_pool_timeout=60
worker.cfusion.monitoringsecret=f68691fb-903c-4c39-9166-ddd7fa726cae
worker.cfusion.secret=27837689-fc18-446c-848a-ae113acf00c8
#End of workers.properties associated with 'cfusion'

I'm going crazy trying to solve this - dug through all suggestions I can find online.  I hope someone can help.  Thanks!

[Priyank Shrivastava.]

@lynn___  Please let me know if we can connect and join the call. If you are ok with that, please respond. I will share the details with you.

[lynn___]

@Priyank Shrivastava.Yes, I would appreciate that!

[BKBK]

Have you solved the problem? If so, what is the solution?

If not, here is a suggestion: change the workers.properties as follows, and restart ColdFusion after after you do:

• Replace worker.cfusion.host=localhost with worker.cfusion.host=127.0.0.1.
• Add these 2 properties:
  worker.cfusion.connection_pool_size=500
  worker.cfusion.max_reuse_connections=500
  

[lynn___]

Thank, BKBK.

I updated my workers.properties file as follows but still getting the 403 with similar mod_jk.log output

heartbeat_interval=30
heartbeat_limit=90

#Start of workers.properties associated with 'cfusion'
worker.list=cfusion

worker.cfusion.type=ajp13
worker.cfusion.host=127.0.0.1
worker.cfusion.port=8009
worker.cfusion.heartbeat_servlet_path=/__cf_connector_heartbeat__
worker.cfusion.connection_pool_timeout=60
worker.cfusion.monitoringsecret=f68691fb-903c-4c39-9166-ddd7fa726cae
worker.cfusion.secret=27837689-fc18-446c-848a-ae113acf00c8
worker.cfusion.connection_pool_size=500
worker.cfusion.max_reuse_connections=500
#End of workers.properties associated with 'cfusion'

[Charlie Arehart]

Lynn, can you try this to see if it fixes your problem: add this line as another attribute on that AJP connector line in server.xml:

allowedRequestAttributesPattern=".*"

Make a copy of that file, then edit it and save the file and restart cf (not apache).  Test your failing page. Does it help? 

If so, I can't explain why the update itself would upset this: neither newly requiring it nor removing it if it had been there before. But let us know if it helps either way. 

[lynn___]

Thanks, Charlie.  Yes, I've already tried that (I feel like i've tried everything!!!).  This is what I have currently:

<Connector packetSize="65535" protocol="AJP/1.3" port="8009" redirectPort="8455" secret="27837689-fc18-446c-848a-ae113acf00c8" maxThreads="500" connectionTimeout="60000" tomcatAuthentication="false" address="127.0.0.1" allowedRequestAttributesPattern=".*" />

[Charlie Arehart]

Agh, I see now you DID indicate that in your original server.xml. So sorry. It's just one of the many unexpected causes of 403 errors for some folks, especially with some Apache configs, that is easily missed. Glad you'd tried it. 

[pete_freitag]

Did you check the apache error log file?

The error you posted doesn't necessarily point me here, but one thing that can sometimes be an issue is file system permissions. Take a look at things like the JkShmFile. If you are on RedHat or a variant with SELinux then you need to make sure files / ports have the appropriate selinux context (eg chcon). 

[lynn___]

Hi, Pete.  Thanks so much for your suggestions.  I too had the thought that maybe it's a permission issue.  The JkShmFile wasn't set.  I updated it to JkShmFile "/appl/ColdFusion2023/config/wsconfig/1/jk_shm" in the mod_jk.conf file (and restarted apache) but that didn't fix the 403 - it did write the file. 

I've check SElinux status - which shouldn't be blocking
[root@www-cfl-01 conf]# sestatus
SELinux status: enabled
SELinuxfs mount: /sys/fs/selinux
SELinux root directory: /etc/selinux
Loaded policy name: targeted
Current mode: permissive
Mode from config file: permissive
Policy MLS status: enabled
Policy deny_unknown status: allowed
Memory protection checking: actual (secure)
Max kernel policy version: 33


I've also made sure iptables is off
[root@www-cfl-01 conf]# service iptables status
Redirecting to /bin/systemctl status iptables.service
○ iptables.service - IPv4 firewall with iptables
Loaded: loaded (/usr/lib/systemd/system/iptables.service; disabled; preset: disabled)
Active: inactive (dead)

as well as firewalld
[root@www-cfl-01 conf]# systemctl status firewalld
○ firewalld.service - firewalld - dynamic firewall daemon
Loaded: loaded (/usr/lib/systemd/system/firewalld.service; disabled; preset: enabled)
Active: inactive (dead)
Docs: man:firewalld(1)

I also checked the port
[root@www-cfl-01 conf]# netstat -anp | grep 8009
tcp 0 0 127.0.0.1:8009 0.0.0.0:* LISTEN 2128355/java
tcp 0 0 127.0.0.1:58270 127.0.0.1:8009 TIME_WAIT -
tcp 0 0 127.0.0.1:10028 127.0.0.1:8009 TIME_WAIT -
tcp 0 0 127.0.0.1:58246 127.0.0.1:8009 TIME_WAIT -
tcp 0 0 127.0.0.1:31088 127.0.0.1:8009 TIME_WAIT -
tcp 0 0 127.0.0.1:31100 127.0.0.1:8009 TIME_WAIT -
tcp 0 0 127.0.0.1:31074 127.0.0.1:8009 TIME_WAIT -
tcp 0 0 127.0.0.1:58260 127.0.0.1:8009 TIME_WAIT -
tcp 0 0 127.0.0.1:18892 127.0.0.1:8009 TIME_WAIT -

i'm learning as I go - i'm sure there's something i'm missing...

[lynn___]

I forgot to include, nothing of interest in the apache error log:

on Jun 23 12:53:03.664251 2025] [example_hooks:notice] [pid 2161140:tid 2161140] x_monitor()
[Mon Jun 23 12:53:13.674802 2025] [example_hooks:notice] [pid 2161140:tid 2161140] x_monitor()
[Mon Jun 23 12:53:23.182186 2025] [example_hooks:notice] [pid 2161142:tid 2161274] x_create_connection()
[Mon Jun 23 12:53:23.341491 2025] [example_hooks:notice] [pid 2161142:tid 2161274] x_create_request()
[Mon Jun 23 12:53:23.343830 2025] [optional_fn_export:error] [pid 2161142:tid 2161274] AH01871: Optional function test said: GET /healthcheck.cfm HTTP/1.1
[Mon Jun 23 12:53:23.343856 2025] [example_hooks:notice] [pid 2161142:tid 2161274] x_create_request()
[Mon Jun 23 12:53:23.521652 2025] [optional_fn_export:error] [pid 2161142:tid 2161274] AH01871: Optional function test said: GET /favicon.ico HTTP/1.1
[Mon Jun 23 12:53:23.684852 2025] [example_hooks:notice] [pid 2161140:tid 2161140] x_monitor()

[BKBK]

Suggestion (from some of the solutions of issues that occurred when installing recent updates):

• install Update 14
• delete everything from the folder /cfusion/bin/felix-cache
• edit /cfusion/bin/jvm.config and ensure that the property java.args contains the flag -Dcoldfusion.runtime.remotemethod.matchArguments=false
• restart the ColdFusion instance

[Charlie Arehart]

Honestly I don't see how any of those could relate to a 403 error, per se.

But on closer examination of the mod_jk.log Lynn had originally offered, I see spread over 4 lines these words:

Internal.Se

rver.Error

(cfusion) status = 500

That suggests cf is having an error, and indeed a 500 rather than the misleading 403.

So now it seems we have to ask you, Lynn: are you able to access the cf admin? Since by default (and by design) you can't access that with apache, you'd be using the cf built-in web server, avoiding apache. Does it work or not?

If not, then we could turn our attention to what is tracked in the cf logs during cf startup (and when making either request). There can be any number of things amiss.

And since you say this happened as a result of a cf update (and even after uninstalling it), are there errors indicated in the update log, available under the cfusion/hf-updates folder, within its subfolder for the update you applied? Check both rhe install AND the uninstall logs there. 

If so, perhaps THAT is where all this may have started. Let us know what you find. 

Or yes, if you just reapply the update as bkbk proposes (and somehow whatever error happened before doesn't now), that may work. But it may not.

Please confirm for us first, before you pursue all that: does your cf admin work? 

[Charlie Arehart]

Oh and if the cf admin DOES work ok, then there's a chance that it was the one request you've tested which was failing. And updates can have changes that could cause that, with jvm args like bkbk offered...

But let's recall you said you reverted to update 11 and STILL had the 403 error. That's why I really think the focus should be on the errors first, then we find the solution for that. 

[BKBK]

The "403 Forbidden" message suggests an issue with permissions. So @pete_freitag has his finger on the pulse.

Following up on this idea, here are two permissions-related suggestions (which assume you have installed Update 14):

• Your workers.properties contains the setting "worker.cfusion.monitoringsecret". Do you use the Performance Monitoring Toolset (PMT)? If so, might that be the cause of the issue? 
  In Update 12 and above, ColdFusion resets the UUID value of the setting.
  Find out what the UUID value within PMT is, and verify whether it matches the value of worker.cfusion.monitoringsecret.
• Might it be that the AJP request is somehow missing the secret? That would be sufficient reason for Tomcat to reject the request. That would also confirm the HTTP 500 Internal Server Error.
  To verify this hypothesis, add the attribute secretRequired="false" to the AJP connector in server.xml, restart ColdFusion and see what happens. 
  
  

[lynn___]

@BKBKand @pete_freitag I agree as well that it seems to be a permission issue - but where!

I'm not using PMT - I removed the setting "worker.cfusion.monitoringsecret" from workers.properties 

I updated the AJP connector in server.xml to secretRequired="false" and restarted ColdFusion

i'm trying to attach the mod_jk.log (it's very large, even slimmed down) from when i 1st restart the server -- it seems like there's good stuff happening... until there's not

[lynn___]

here's the mod_jk.log the 1st time I try to load a .cfm page after restarting CF/Apache

[BKBK]

What a puzzle! But I get the feeling we're zooming in to the cause.

As  adding secretRequired="false" doesn't help, remove it from server.xml and restart ColdFusion. Then we can proceed further.

[BKBK]

The latest logs you've submitted suggest the following:

• The "403 Forbidden" issue seems to have started when Tomcat received a request from the IP address 10.34.171.161.
  

  Though ColdFusion restricts AJP to only accept connections from 127.0.0.1, that may not be enough. Tomcat may be checking whether the REMOTE_ADDR is allowed.

  In this case, though the socket connection is from 127.0.0.1, mod_jk is forwarding a different REMOTE_ADDR (10.34.171.161) to Tomcat. That may be the reason why Tomcat is rejecting the request.

To test this hypothesis, replace address="127.0.0.1" with address="::" in server.xml, then restart ColdFusion. I should emphasize that this test is just for the purpose of elimination. It is not a recommendation for the use of  address="::" .

I am assuming that the IP 10.34.171.161 is in your network. Were this test to succeed, then you would have to configure Tomcat to include your network range (for example by means of the setting "InternalProxies").

[lynn___]

I can't tell you how much I appreciate you're help with this very frustrating puzzle!

Changing address="127.0.0.1" with address="::" in server.xml now gives me 503 Service unavailable

snippet of mod_jk.log

[Wed Jun 25 08:29:24 2025] [2853898:140616927131200] [debug] jk_open_socket::jk_connect.c (798): trying to connect socket 19 to 127.0.0.1:8009
[Wed Jun 25 08:29:24 2025] [2853898:140616927131200] [info] jk_open_socket::jk_connect.c (816): connect to 127.0.0.1:8009 failed (errno=111)
[Wed Jun 25 08:29:24 2025] [2853898:140616927131200] [info] ajp_connect_to_endpoint::jk_ajp_common.c (1158): (cfusion) Failed opening socket to (127.0.0.1:8009) (errno=111)
[Wed Jun 25 08:29:24 2025] [2853898:140616927131200] [error] ajp_send_request::jk_ajp_common.c (1829): (cfusion) connecting to backend failed. Tomcat is probably not started or is listening on the wrong port (errno=111)
[Wed Jun 25 08:29:24 2025] [2853898:140616927131200] [info] ajp_service::jk_ajp_common.c (3000): (cfusion) sending request to tomcat failed (recoverable), because of error during request sending (attempt=2)
[Wed Jun 25 08:29:24 2025] [2853898:140616927131200] [error] ajp_service::jk_ajp_common.c (3021): (cfusion) connecting to tomcat failed (rc=-3, errors=26, client_errors=0).

[BKBK]

Sorry, a mistake in the heat of the moment: The address "::" is IPv6, but we need the IPv4 version, namely, "0.0.0.0".

So please repeat the test with address="0.0.0.0" (or leave out the address attribute altogether). Restart ColdFusion, and see what happens.

[pete_freitag]

In this mod_jk.log - It looks like ColdFusion/Tomcat is returning the 403 Forbidden. I see this response from the request of /test.cfm:

ajp_connection_tcp_get_message::jk_ajp_common.c (1563): received from ajp13 pos=0 len=17 max=65536
ajp_connection_tcp_get_message::jk_ajp_common.c (1563): 0000    04 01 93 00 09 46 6F 72 62 69 64 64 65 6E 00 00  - .....Forbidden..
ajp_connection_tcp_get_message::jk_ajp_common.c (1563): 0010    00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  - ................
ajp_unmarshal_response::jk_ajp_common.c (818): (cfusion) status = 403

• Does the user ColdFusion is running as have read permission to test.cfm?
• Anything showing up in any ColdFusion logs or Tomcat's catalina.out log?
• Does the user ColdFusion is running as have permission to traverse to the directory that test.cfm exists in? 
  • In other words: if your web root is /var/www/html/test.cfm does the CF user have execute (x) permission on /var/www/html/ and /var/www/ and /var/ and / 

You can test read permissions using the test command with -r (for read) and the file name. Using the su command to run as the cfuser. 

su username -s /bin/bash -c 'test -r /path/to/test.cfm' && echo readable || echo not-readable

 Or you could create a check.cfm under the builtin web server (cfusion/wwwroot/) with something like this:

<cfdump var="#getFileInfo("/path/to/webroot/test.cfm")#">

 Then hit that cfm over the builtin web server. It will output if ColdFusion can read the file or not.

[lynn___]

I removed the address attribute from server.xml - no luck

my bad, the test.cfm file actually does not exist - but gettting a 403, not a 404

that said, i did check and ColdFusion is able to access the .cfm files in their location

I don't seem to have a catalina.out file and couldn't figure out if there's a config that can be updated to get that.  I turned on the localhost_access_log in server.xml but it doesn't seem to be getting created (maybe no access being logged?)

most recent mod_jk.log for accessing the healthcheck.cfm file after a restart is attached

[lynn___]

I forgot to say, I see this in the coldfusion-error.log over and over and over again - not related to when I tried hitting the page

Jun 26, 2025 12:52:05 PM org.apache.coyote.ajp.AjpProcessor service
SEVERE: Error processing request
java.lang.NullPointerException

Jun 26, 2025 12:52:05 PM org.apache.coyote.ajp.AjpProcessor service
SEVERE: Error processing request
java.lang.NullPointerException

Jun 26, 2025 12:52:05 PM org.apache.coyote.ajp.AjpProcessor service
SEVERE: Error processing request
java.lang.NullPointerException

[BKBK]

That is important new information, in view of "SEVERE" and "NullPointerException". They suggest that there is an issue with Apache's access control mechanisms even before the request is passed to mod_jk or Tomcat. Could you please share the Apache logs (for example, error_log, access_log)?