3D Camera Tracker Major issues

I would have to see the  shot but from what I can see there isn't much in that shot that has the changes in parallax required to do a good camera solve. When the tracking markers are way off like that it usually means the shot is not compatible with camera tracking.

Try tracking this shot from Adobe Stock: https://stock.adobe.com/stock-photo/seattle-wa-4-29-16-aerial-fly-over-downtown-city-buildings-to-reveal-space-needle/110966019?prev_url=detail.​ You should be able to get a decent track using this shot.

I have been preparing a lesson for a class that I'm teaching in a few days so I've got some stuff that I can show you. If you track that shot you will get plenty of tracking markers to work with but the hard part is going to be finding a plane that matches the geometry in the scene. If you wanted to insert something on the roof of the round building in the center of the frame you would have to do some fancy footwork because there are never 3 tracking markers on that roof at the same time.

Because I downloaded the 4K footage from Adobe Stock the markers were a little hard to see so I increased the track point scale to 200%. As you can see it's easy to get a nice sized target but this one doesn't line up with any real geometry so placing a solid here wold be useless. The solid that I did manage to get to stick to the roof was added when I found 3 points on the side of the building that matched the plane of the building. The track points only showed up for a few frames but it was enough to get an accurate position for the wall of the building.

This was the easiest solution that I could find. I tried using a roof from a couple of other buildings and then sliding the solid around to get it into the right position but I was not having any luck.

When I found the three points that gave me the surface of that wall on the building I wanted to use I set the origin and ground plane there, added a camera and a solid, then rotated the plane 90º in X and moved it up to the corner of the roof and adjusted the X and Y position to get it lined up. I then drew a mask around the edge of the roof so I could check the accuracy of the track. It held fairly well. The next step was to add optics compensation to the original footage to fix the lens distortion in the shot. When the optics compensation was perfect I cut the effect from the footage layer to preserve the settings, added an adjustment layer above the solid, moved the camera to the top, pre-composed the camera, adjustment layer, and solid, opened the pre-comp and pasted the Optics Compensation effect to the adjustment layer. The last step is to reverse the lens distortion because the layer needs to bulge a bit when it approaches the edge of the frame. Back in the main comp the solid now perfectly tracks the building. I could then use the coordinates for the solid layer to insert anything I wanted on the roof, like a swimming pool and it wold track perfectly. All I had to do was find a shot that matched the perspective in this shot fairly closely.

Camera tracking in AE works very well for a lot of shots, you can make it work for some difficult shots, but it does not work for all shots. No camera tracker does. If there is any lens distortion in your shot you are going to have to make some kind of correction for that. I usually remove the lens distortion before I start a composite. It makes things a lot easier in the long run.

What are you trying to do in your shot? If you wanted to put something in the frame on the door then Corner Pin tracking in Mocha or even AE wold be a far better solution. If we knew exactly what you were trying to accomplish we could probably give you some suggestions that would work.