Video playback WORSE with grahpics card?

I will confess that there is probably something I'm not understanding and maybe one of you can point me in the right direction or give me some good ideas.

Current Rig:

Core i5 4690 (Haswell)

MSI Z97-G55 SLI

16G Ram

750G SSD (OS and Apps)

1T Traditional HDD for media (SATA 6G)

Win 10 (64 bit)

Latest version of Premiere Pro

I'm using this for light editing. Playback with 4k files was enough to get the job done - (30fps on a single monitor at 1/4 resolution - don't need anything more). Smooth playback with cuts and dissolves, struggles a bit with fancier transitions, but if I pre-render I would get smooth playback.

Added GTX 1060 (6G)

Suddenly, can't play back 4k files in Premiere - SEVERE lag... skips tons of frames... forget anything nice. It even struggles a bit when playing 4k files using Windows default player.

I killed everything running in the background. No help there.

3D Games look great tho.

So, thoughts about what I'm doing wrong? If I can't debug this, I am able to return the card but I would like to give it an honest shot first.

Thanks in advance.

Chris