OK here is the update and the fix that worked for me. Vikram at Adobe support was very helpful and tried hard to find a resolution. In the process of both of us testing connections from PMT running on one network and a CF instance running on another network (AWS EC2) I finally came up with a solution that worked for me.
First make sure all the ports needed are opened both ways as others have instructed.
Some background: On the CF server the PMT datastore had been set to my PMT server's external IP address.
On the PMT server the datastore service would only start up when I set the network.host setting in the elasticsearch.yml config file to my PMT servers INTERNAL IP address. So the CF machine was told to use the external IP reachable from one network to the other, but the PMT datasource running on my server in the office was bound to the internal IP address.
When trying to connect to a remote CF instance from PMT it would fail with a 500 page error showing as the content in the little error icon mouseover. Logs for PMT did not show anything.
Checking logs on the CF instance that I tried to connect to I found a ton of errors like this in the coldfusion-error.log,
even though the connection failed it starts some elassticsearch sniffer process on the CF server that was trying to connect to my internal datastore IP instead of the external one configured in coldfusion. Even worse these did not stop filling the logs until I shut down both my local PMT service and the datastore and restarted the CF instance!
Jul 19, 2021 4:43:39 PM org.elasticsearch.client.sniff.Sniffer error
SEVERE: error while sniffing nodes
java.net.ConnectException: Timeout connecting to [/192.168.254.71:9250]
So I knew the issue was on the PMT side which was bound to my internal IP. So that is what it was telling the CF server to use for the datastore. And Datastore would not start if bound to my external IP.
We tried all kinds of things and today I discovered that the Elasticsearch datastore can bind to multiple hosts/ips AND it can use a different host/ip for incoming and outgoing traffic.
So, after stopping both the datastore and the PMT service on my office server, in my elasticsearch.yml file in the datastore/config folder where you installed PMT file I did these things:
Comment out with a # as the first character the entries for these items if they are in your file:
network.host:
transport:host
Then below those add two new entries, replacing the values in <> brackets with your IPs or hosts:
network.bind_host: <Your Internal IP or host name>
network.publish_host: <Your External IP or host name>
Save your changes, then restart the datasource service and the PMT service.
If you had already tried to connect PMT to a remote CF instance and have not restarted that instance
since the failure do that first. Then try to connect PMT to your remote instance.
For me this worked first try after trying without success for days ... I hope someone else finds it helpful.
Thanks to Vikram at Adobe for all the effort he put in as well!