Too many established TCP ports

[RockerNJ]

I have a server running 7 instances of CF (6 sites plus root instance).  If I run a NetStat, I generally see about 1800 ports as "ESTABLISHED"  (see below for a sample).  Why so many ports connected to itself?

Ultimately what happens is the server just kind of dies because it can't establish any more sockets.  I don't think it's "port exhaustion" because I see the total number of connections staying around that 1800 mark, and there aren't piles of TIME_WAIT entries.

What I THINK is happening, is, as this issue is cycling through ports, it gets to a 65535 ceiling and is not looping back to lower ports, thus killing connectivity.

Any thoughts on what's going on here?

TCP 127.0.0.1:49890 site01-wf:49891 ESTABLISHED
TCP 127.0.0.1:49891 site01-wf:49890 ESTABLISHED
TCP 127.0.0.1:49892 site01-wf:49893 ESTABLISHED
TCP 127.0.0.1:49893 site01-wf:49892 ESTABLISHED
TCP 127.0.0.1:49894 site01-wf:49895 ESTABLISHED
TCP 127.0.0.1:49895 site01-wf:49894 ESTABLISHED
TCP 127.0.0.1:49896 site01-wf:49897 ESTABLISHED
TCP 127.0.0.1:49897 site01-wf:49896 ESTABLISHED
TCP 127.0.0.1:49898 site01-wf:49899 ESTABLISHED
TCP 127.0.0.1:49899 site01-wf:49898 ESTABLISHED
TCP 127.0.0.1:49900 site01-wf:49901 ESTABLISHED
TCP 127.0.0.1:49901 site01-wf:49900 ESTABLISHED

[BKBK]

First of all, what is your:

• Operating System?
• ColdFusion version and update level?

[RockerNJ]

Windows Server 2022 Datacenter

Coldfusion 2021 + Hotfix20

[BKBK]

Thanks for the information.

The Netstat result is indeed a red flag. The connections are not short-lived (going into TIME_WAIT or closing). Instead, they remain ESTABLISHED. That suggests 2 issues.

Firstly, there might be an infinite loop between 2 servers. The servers are 127.0.0.1 ("localhost") and "site01-wf".  It is likely that, between them, HTTP calls are being generated in a loop. 

To see the process that is running on each of the "Established" sockets:

1.  Open the Command Prompt (cmd) as Administrator;
2.  Run the DOS command netstat -anob ;
3.  Scroll to the lines that resemble the ones you've shown in your first post. The output within each pair of square-brackets is the name of the process that is running. What processes do you see?  

Secondly, there seems to be duplication in the configuration of the local machine. The same machine seems to have been doubly configured as two separate servers, Localhost(127.0.0.1) and Site01-wf .  Because of the incorrect configuration, pairs of connections, such as

TCP 127.0.0.1:49890 site01-wf:49891 ESTABLISHED
TCP 127.0.0.1:49891 site01-wf:49890 ESTABLISHED

are actually connections from one server to itself (127.0.0.1:49890 -> 127.0.0.1:49891 -> 127.0.0.1:49890 )

TCP 127.0.0.1:49890 127.0.0.1:49891 ESTABLISHED
TCP 127.0.0.1:49891 127.0.0.1:49890 ESTABLISHED

or, equivalently,

TCP site01-wf:49890 site01-wf:49891 ESTABLISHED
TCP site01-wf:49891 site01-wf:49890 ESTABLISHED

[RockerNJ]

The site01-wf is probably just a result of having a hosts. file entry. IE: 127.0.0.1 site01-wf.xxxxxxxx.com (seems to not do that with the switches you had me use), but why is the site connecting to itself in the first place?

Here is the netstat with the flags you requested:

TCP 127.0.0.1:51138 127.0.0.1:51137 ESTABLISHED 6952
[coldfusion.exe]
TCP 127.0.0.1:51139 127.0.0.1:51140 ESTABLISHED 6952
[coldfusion.exe]
TCP 127.0.0.1:51140 127.0.0.1:51139 ESTABLISHED 6952
[coldfusion.exe]
TCP 127.0.0.1:51141 127.0.0.1:51142 ESTABLISHED 6952
[coldfusion.exe]
TCP 127.0.0.1:51142 127.0.0.1:51141 ESTABLISHED 6952
[coldfusion.exe]
TCP 127.0.0.1:51143 127.0.0.1:51144 ESTABLISHED 6952
[coldfusion.exe]
TCP 127.0.0.1:51144 127.0.0.1:51143 ESTABLISHED 6952
[coldfusion.exe]
TCP 127.0.0.1:51145 127.0.0.1:51146 ESTABLISHED 6952
[coldfusion.exe]
TCP 127.0.0.1:51146 127.0.0.1:51145 ESTABLISHED 6952
[coldfusion.exe]
TCP 127.0.0.1:51147 127.0.0.1:51148 ESTABLISHED 6952
[coldfusion.exe]
TCP 127.0.0.1:51148 127.0.0.1:51147 ESTABLISHED 6952

[RockerNJ]

Wanted to attach these as well.  This is a graph from AWS CloudWatch that I'm capturing the metric of number of established ports.  You can see it goes up and down as one would expect.  When things go soouth, what happens is that the CloudWatch Agent can no longer establish a socket to send the metrics back, so we see breaks of missing data.  Furthermore, even things like authenticating to the domain for an RDC connection will fail, because the box cant make the connection out.

The fact that there aren't a ton on posts sitting in a time_wait leads me to believe that there ARE available ports.  What seems to happen is when CF seems to have cycled through ports and gets up past 64000, something is not looping back and reusing the lower ports that have been freed.

[BKBK]

The new information suggests that coldfusion.exe is the process causing the loop. It also suggests that ColdFusion is making internal HTTP(S) requests to itself (or between sibling instances) using both 127.0.0.1 and the hosts-alias site01-wf.xxxxxxxx.com. Apparently, these connections are not being closed properly, causing a connection leak inside the JVM.

The new information confirms the following:

1.  All connections are from ColdFusion (PID 6952):

• The netstat -anob command confirms that the culprit is coldfusion.exe.
• Both ends of each connection are inside the same ColdFusion process.
• These are internal loopback connections. In other words, ColdFusion is talking to itself over TCP.

2. The connections occur as two-way ESTABLISHED TCP pairs (for example, 51139 <==> 51140):

• This indicates that duplex communication is happening internally (that is, probably an actual HTTP call rather than some passive listener).
• This only happens if the same process is opening a client and a server socket. Which is exactly what happens when ColdFusion makes a local HTTP request. Yet another clue pointing to HTTP calls.

3.  Use of 127.0.0.1 and site01-wf results in duplex communication within the same server:

• The hosts file defines site01-wf as 127.0.0.1, which is likely being used inside the ColdFusion application for internal API calls (for example, via CFHTTP or REST calls).
• From the point of view of the TCP/IP stack, site01-wf resolves to 127.0.0.1, so the traffic stays inside the same machine, and possibly inside the same JVM.

To cut a long story short: the issue is apparently caused by an HTTP call or by a series of (possibly looped) HTTP calls. 

[RockerNJ]

So a couple things I would like to test in an environment I stood up to replicate the problematic environment...

Reduce the time-wait to 30 seconds

Increase the available socket pool

[BKBK]

Do you have a reason for reducing TIME-WAIT and increasing the number of sockets? I don't see how that would solve the problem.

As I said in my first post, since the connections are ESTABLISHED, it means they are not short-lived. That implies they are not reaching the TIME-WAIT or CLOSED state. 

The issue suggests that live connections are continually being created. If you double the number of sockets, what will prevent the number of live connections from doubling?

I have mentioned HTTP calls a number of times: not a single reaction from you. So I am wondering whether you have looked into the HTTP idea.

Have you checked the following:

• Is there a cfhttp/http call that is looping indefinitely; 
• Are there any clues in ColdFusion's log files. Has http.log grown in size? Is there a recurrent line in http.log? If so, what is it?
• Does running the following code have any effect?
  

  <!--- Use the correct URL and method --->
  <cfhttp url="http://site01-wf.xxxxxxxx.com" method="get">
    <cfhttpparam type="header" name="Connection" value="close">
  </cfhttp>​

[RockerNJ]

The http idea makes complete sense, and I suspect you may have hit the nail on its head...

I have 6 instances of CF running on one Windows box.  Each of those CF environments has a scheduled task which gets executed every 30 seconds.  Inside the .cfm module that the scheduled task executes, there is in fact a cfhttp call to 

<CFHTTP URL="http://#cgi.server_name#/workflow_manager/act_check_workflow_queue.cfm" TIMEOUT="1">

So that DOES seem to explain all of the cross connections, since the server is kind of playing server AND client as you suggested.  As for my solution, I don't think we actually NEED all 6 instances firing the scheduled task.  I think things will work as intended with just one.  I'll investigate that path.

[RockerNJ]

Upon further investigation, we DO in fact need each environment to be running that task.  So, not sure where that leaves me.  Is it an issue that because we're cycling through ports so fast that we just run out?  Maybe?  In that case, I'd think increaing the pool and decreasing the recycle time might help.  We have this same scenario on MANY single instance environments, and it's been running for 15 years w/o the port issues.  The only one with a problem is the box with six instances running that process.

Another note, not only does the .cfm being called have a cfhttp call, it also does have quite a few queries, which I assume also each take a socket for the DSN connection to the SQL server (on another internal box).  Also of note, the one cfhttp call to another module actually make another cfhttp call.

I think we're definitely on the right path here, just not sure of the solution

[RockerNJ]

I wanted to circle back to this reply.

The number of sockets established always hovers around 1800 total.  It's not like it runs away and eats up ALL of the ports.  It's just establishing and releasing like 1800 at a any given time
They dont seem to be HTTP based, as the destination port isn't 80 or 443

[Dave Watts]

2. The connections occur as two-way ESTABLISHED TCP pairs (for example, 51139 <==> 51140): This indicates that duplex communication is happening internally (that is, probably an actual HTTP call rather than some passive listener). This only happens if the same process is opening a client and a server socket. Which is exactly what happens when ColdFusion makes a local HTTP request. Yet another clue pointing to HTTP calls.

I respectfully disagree. While there may be too many HTTP requests, these aren't HTTP requests. HTTP, like most services, usually relies on established, well-known ports (TCP/80 for HTTP, TCP/443 for HTTPS in most cases). While it's possible to specify non-standard ports, they're usually going to be in the range of registered ports (1024-49151). And, while I'm less sure about this part, I think it's unlikely you'd have an exact amount of established connections for HTTP/HTTPS connections and their clients even with CFHTTP. There's always going to be a pattern of the client trying to make a connection without finding an open server port, or an open server port waiting for a client, etc.

I hope all that makes sense, and doesn't slow down the diagnostic process too much.

[RockerNJ]

Didn't look at it that way, but yes, I understand.  If my module was making a cfhttp to another module on the same server, that request would be port 443 on one side of the connection (and not some high port number to another high port number).

[BKBK]

Hi @RockerNJ and @Dave Watts , to respond to your last posts, I do understand your point of view. I initially had the same thoughts myself. In fact, I have to clarify one thing. Like you, I am at a loss what the exact cause of the issue is.

I proposed HTTP as a diagnostic suggestion. But I didn't leave it there.

In addition, I offered further diagnostic proposals that would either back up the HTTP idea or refute it. For example,

• Using a connection-close header;
• Checking what the ColdFusion logs say about the issue;
• Examining the contents of http.log (Is there a recurrent log-line?) and its size (for example, has ColdFusion rotated or archived the file?). 

I shall now offer yet another diagnostic proposal. In every CFHTTP request, such as

<CFHTTP URL="http://#cgi.server_name#/workflow_manager/act_check_workflow_queue.cfm" TIMEOUT="1">

replace every occurrence of #cgi.server_name# with the actual domain name, for example, my.domain.com. Does that help?

[RockerNJ]

I will hopefully have a copy of this environment up today, and I can do some of the tests you mentioned.  I'll reply as soon as I'm up and running.

[BKBK]

@RockerNJ , Did you try out my last suggestions? Did they help?

Here are yet more thoughts on the issue. 

You and Dave supposed that, if the traffic isn’t over 80 or 443, then these are unlikely to be HTTP requests. However, when I looked further into the issue, I found that internal HTTP communication within ColdFusion may use high-numbered dynamic ports.

So, here are 3 reasons why It is still likely the issue is caused by HTTP/CFHTTP connections:

1.  ColdFusion and the underlying (Tomcat/Jetty) servlet container may use ports different from 80 and 443.
Since you are running multiple CF instances, each one may be configured to listen on its own custom random ephemeral port. So a cfhttp call from one CF instance to another will be using these random ports.

2.  The Netstat result shows that the TCP socket pairs are bidirectional (A <==> B) and ESTABLISHED.
The pattern:

TCP 127.0.0.1:51140 127.0.0.1:51139 ESTABLISHED
TCP 127.0.0.1:51139 127.0.0.1:51140 ESTABLISHED

is neither a one-way connection nor a connection from a background protocol such as RMI. It is bidirectional, exactly what you would expect from an internal client/server TCP pair. This strongly suggests application-level communication, hence HTTP.

3.  ColdFusion itself (coldfusion.exe) is the process opening both ends.
This tells us that ColdFusion is both the client and the server. This is exactly how an internal cfhttp call to 127.0.0.1:{port_number} works: one thread binds a client socket, another listens on the HTTP server socket.

[RockerNJ]

The "ColdFusion and the underlying (Tomcat/Jetty) servlet container may use ports different from 80 and 443." really does make the story make sense.  As stated earlier, we are running a scheduled task (on each of the 6 instances) every 30 seconds.  That task then does in fact make a cfhttp call to yet another module.  It certainly does explain the AMOUNT of sockets.  I'm just curious as to why they remain established.  The whole routing that the scheduled task is calling returns immediately (not some long running process).

[Dave Watts]

The "ColdFusion and the underlying (Tomcat/Jetty) servlet container may use ports different from 80 and 443." really does make the story make sense.  As stated earlier, we are running a scheduled task (on each of the 6 instances) every 30 seconds.  That task then does in fact make a cfhttp call to yet another module.  It certainly does explain the AMOUNT of sockets.  I'm just curious as to why they remain established.  The whole routing that the scheduled task is calling returns immediately (not some long running process).

Well, not really. I mean, you may well have too many CFHTTP requests. But so far, I haven't seen any indication of them in this thread.

First, if you're using different ports, you'd be the first one to know! CFHTTP, like any HTTP client, automatically expects you're using the standard ports. If you're not, you'd have to specify that right in the CFHTTP tag. When you use your browser to connect to, say, www.adobe.com, you don't have to specify TCP/80 (for plaintext HTTP) or TCP/443 (for HTTPS). It automatically opens a connection to the appropriate port and sends whatever it's supposed to send, something like this:

GET /path/to/resource HTTP/1.1
Host: www.adobe.com
User-Agent: ...
Accept: ...
Accept-Language: ...

Now, if you're using a non-standard port, the client has to know what that port is:

<cfhttp url="https://www.whatever.com:8080/ ...>

It's simply not possible for an HTTP client to find an unknown non-standard port, you need to tell it what the port is.

Also, on an unrelated note, a CF application uses TCP internally for a lot of things. A LOT. I have no idea whether they'd all show up as TCP connections from coldfusion.exe (arbitrary client port) to coldfusion.exe (arbitrary server port) because, until now, I didn't have to worry about it. I'm answering from the world's slowest computer so it's hard for me to look too deeply into it, but I started three CF 2018 services:

• ColdFusion 2018 Application Server
• ColdFusion 2018 ODBC Agent
• ColdFusion 2018 ODBC

then, without doing anything else, I went to TCPView, which is basically a GUI for netstat with some extra features, available as part of Sysinternals here:

https://learn.microsoft.com/en-us/sysinternals/downloads/tcpview

I used that instead of netstat because I couldn't figure out how to see just entries with coldfusion.exe in the name. (If anyone knows how, let me know!) Anyway, with no CFHTTP tags - or any other code - running, I had quite a few of these established connections.

I don't know what they are, but I can make some guesses. I stopped the ODBC Agent and ODBC Server services and retried this, and got the same thing.

So, again, while you may have a problem with coldfusion.exe making too many connections and while that may be caused by too many CFHTTP tags, these aren't it. My not-so-educated guess is that it's related to database connections, but I have no databases (or applications) against which to test. You could try disabling "Maintain Connections" in your test environment for all of your databases to see if that helps.

[BKBK]

Great discovery, @Dave Watts !  You have practically reproduced @RockerNJ 's table.  Your findings mean that we can definitely rule out HTTP calls, or any other custom code, as the cause of the connections. With one notable exception: scheduled tasks. More on this later.

What then is the source of the connections? That is, besides scheduled tasks. Database connections (in a connection pool, for example)? But I doubt this. For the same reason you and @RockerNJ gave earlier about the dedicated ports 80 and 443. If databases were involved, you would then expect the output to contain some typical database ports, such as 1433 (Microsoft SQL Server), 1521 (Oracle), 3306 (MySQL), and so on.

In any case we've made a step forward. Your findings tell us that internal ColdFusion components are likely responsible for these loopback connections.

Following your example, I downloaded the TCPView tool on two Windows PCs. ColdFusion 2023 is installed on one of the PCs, ColdFusion 2025 on the other.

When I launch TCPView, I do not see the result that you and @RockerNJ obtained. Here is what I see for ColdFusion 2025:

Some details about my ColdFusion 2025 installation:

• Developer Edition, consisting of 1 instance (cfusion) running on Windows 10 Pro;
• Connected to MySQL and MS SQL database; 12 datasources configured, each with "Maintain Connections" checked;
• No scheduled tasks running;
• Standalone .NET Service, Add-On Service, ODBC Agent and ODBC Server running alongside ColdFusion.

Some questions:

• Does the result that you and @RockerNJ get apply to ColdFusion 2018 and 2021, but not to ColdFusion 2023 and 2025? If so, what were the changes between versions?
• Do the high-port-number connections result from monitoring components within ColdFusion? 
  (Performance Monitoring Toolset is not installed on either of my PCs)
• Do the high-port-number connections have any relation to a connection pool?
• (My hottest hypothesis at the moment) Might the connections in your table be generated by HTTP calls run by scheduled task thread pools? If so, what explains the fact that default ports such as 80 and 443 are absent?

[BKBK]

After looking into this some more, I take back all the suggestions I have made so far.

• The frequent high-port-number TCP connections are NOT necessarily caused by HTTP calls, scheduled tasks or any custom code.

[BKBK]

I investigated the issue further as follows:

1.  Download and install ColdFusion 2021 on Windows 10, and implement Update 20.
        Configure nothing more: just a bare, fresh ColdFusion 2021 installation.
2.  Run the TCPView tool. 

And ... Voilà! You get a long list of  high-port-number TCP connections generated by the process "coldfusion.exe".

 

Here are some of my findings and new suggestions:

1.  The frequent high-port-number TCP connections seem to be present in ColdFusion 2018 (confirmed by @Dave Watts ) and ColdFusion 2021 (confirmed by @RockerNJ  and me), but not in ColdFusion 2023 or ColdFusion 2025 (confirmed by me).
2.   The connections seem to be generated by internal subsystems in ColdFusion, which spin up socket pairs (loopback TCP connections).
   My guess now is that some internal features in ColdFusion 2018 and 2021 use explicit TCP-based inter-thread communication. Adobe would have changed, reduced or removed many of those features after ColdFusion 202. Which would explain why we don't see the connections in ColdFusion 2023 and ColdFusion 2025.  
3.   The connections seem to have been built in by design. You could therefore suppose that they pose no likely risk to performance or to security. I say this because they are local-only connections, with no external exposure. I also notice that the connections are cleaned up during a restart.
4.  As a further test, I opened the Command Prompt (CMD) as Administrator, and ran the DOS command
      netstat -aof | findstr :59431
   I repeated this, each time using a port number chosen randomly from the list in TCPView. The result suggests that the subsystem associated with the generation of the connections is
   
       kubernetes.docker.internal  
   
   

 

Many thanks to @Dave Watts for bringing TCPView into the discussion and for his usual perceptive insights.

[RockerNJ]

@BKBK @Dave Watts 

I think we're all on the same page that these are internal loopback connections.  So, the issue remains...  Let me give this scenario..  Start all 7 instanes of CF2021 (1 root instance plus 6 sites).  Look at the ports established.  Let's hypothetically say ports 20000-22000.  I you let some time go by, ports 20000 - 21000 will be closed (no longer showing on netstat at all), and now CF will be using ports 21000 - 23000 for those loopbacks.  Now, what I THINK is happening is something, somewhere is not smart enough that when CF gets up into the really high port numbers such as 64000+, it's reached the end of the TCP socket pool and does not know to go back down to the completely free 20000 range.  It just behaves as if it's real port exhaustion and thats when the box starts to die.

Thoughts?  Additional troubleshooting ideas?

[BKBK]

@RockerNJ , a note before we proceed any further. You might have overlooked this:

"The result suggests that the subsystem associated with the generation of the connections is
    kubernetes.docker.internal "

So, question is, knowing this, how do we reframe the problem?

[BKBK]

Hi @RockerNJ ,

Some new questions, in view of the "kubernetes.docker.internal" discovery:

1.  Do you use Docker?
2.  Is Docker Destop installed on your Windows server?
3.  Check the file C:\Windows\System32\drivers\etc\hosts. Does it contain something like
   

   # Added by Docker Desktop
   # 192.168.68.107 host.docker.internal
   # 192.168.68.107 gateway.docker.internal
   # To allow the same kube context to work on the host and the container:
   127.0.0.1 kubernetes.docker.internal
   # End of section​