I was curious, why do video files in AE and Premiere need to always cache thus slowing down the preview time, while players like VLC and preview in mac play videos instantly?
I can play raw files on my macbook without any issues, butter smooth, while in AE and Pr they need to render and cache in order to play smoothly, not even in full resolution.
After Effects calculates every pixel on every layer every time, so building a video stream takes a little time. Once the preview is rendered, the cached preview should playback in real time.
Premiere Pro calculates every layer as a stream. I usually edit in 4K, and my Premiere Pro panel is set to 1/4 resolution. If I have only a couple of layers in Premiere Pro, playback is slightly slower than a media player but not as fast as a media player. If I have a lot of effects and layers, a real-time preview takes a while to prepare. Once the preview has cached, Premiere plays back the stream it has rendered in the background until you make a change in the timeline.
A media player decodes the codec, produces a stream from the file, does not report dropped frames, scales to the window size, and only read the original data. That involves a lot less calculating. Popular distribution codecs like H.264 (MP4) have such small file sizes because they throw away most of the data on two-thirds of the frames, average the color in blocks of 4 pixels, and predict the movement of detail between frames to deliver video stream that looks pretty good even though most of the original pixel data is gone. AE rebuilds every pixel in every frame and delivers 100% of the pixel data for every pixel every time. That's why previews take a few seconds to calculate, even if there are no effects and only one layer.
AE is getting a lot better, and so are all NLEs and compositing apps, but it will take a while before AE gives instant real-time previews.
After Effects treats all source footage as if it's uncompressed for the purpose of compositing and special effects.