PC hardware upgrade advice

In addition to my recommendation to wait, I'd strongly recommend that you replace your system's PSU:

That eVGA "600W", if it's just an 80 Plus white (not Bronze or Gold), really cannot handle more than about 500W maximum continuous load. And newer GPUs are even pickier about the output quality of the PSU than older GPUs did.

What this means that under the load of something that's even slightly more powerful than your current GTX 1060/3800x combo, your current PSU will likely shut itself down in the middle of a rendering/export job in order to protect your PC's precious core components.

In other words, a $50 PSU simply cannot handle the demands of even a mid-range newer GPU. You'll need to spend more than $150 USD for a PSU that can handle this demand.

Unless you really, really need a new GPU because your current GTX 1060 isn't cutting it with your current working footage, I would hold off on the hardware upgrade for now, being that the RTX 4000 series has been introduced, and will be shipping soon. You see, the 3800x to 5900x CPU upgrade isn't worthwhile enough to justify the $400 USD additional cost (assuming that you won't be selling your current 3800x). In fact, even an upgrade to a 5950X (16 cores) does not give you a sufficient performance improvement to justify spending the additional $550 or so for. Your setup is at a point where the only worthwhile CPU upgrade would involve not only a new CPU, but also a new motherboard and new RAM.

I just dropped in to say that the video card should be skipped in Favour of a 4000 series that was just announced.  It will have an AV1 encoder, which is going to (likely) be important within the life span of the machine you're building now:  I have a 3090 and I am dumping it to get the 4090 because of AV1 encoding (and I just want the new one as I have lots of need for Cuda cores, don't hate me for that 🙂 ).

Lots of stuff on the internet, but this should give you a 'jist' of why AV1 is important - hopefully adobe also steps up and supports it fully (sooner rather than later).

Tested: Intel Arc's AV1 video encoder shames Nvidia and AMD | PCWorld

Something you really need to consider here is that you may upgrade your computer and have essentially the same results as now. Hardware is only one aspect of video editing, and you already have an okay setup. The one thing that you don't mention is the video codec that you're working in, which is going to have as much of an impact if not more than your hardware. 4k60 in H264 or H265 is going to play poorly on almost any system, even if you pump more money into yours. Poor playback while going backwards is especially a symptom of working with delivery codecs like H264/5, which are very complex in how they are compressed and are meant to be played forward. When you put your playhead on a frame of H264/5, you aren't just looking at a single frame, you're processing a whole group of frames (hence why they are called Long-GOP codecs, for Long Group of Pictures). Those groups of pictures are loaded front to back in the group, which is why it performs worse in reverse.

Creating proxies of this kind of media or working in a proper editing codec would vastly improve your editing experience without upgrading your hardware. And again even if you do upgrade your hardware you might not see the playback results you were hoping for, since the media has such a huge impact.

One great resource is the Puget Systems site's test data, which is published for all to see and updated regularly. You can see there which pieces do best not only "in general" but for specific media on specific video post apps.

I'd recommend checking their data as to which GPU would get you the most upgrade. The "60's" GPUs tend to be lower in vRAM than the other cards, which limits their abilities more. May not actually be a great choice.

Going from 8 to 12 cores can be useful if the processing speed is the same or higher.

Going from 32GB of RAM to 64GB of RAM would also be a good boost for certain proceses.

So check out the data from Puget.

And maybe @RjL190365  will pop in, that user is awesome for these questions.

Neil