improve optical flow to use it for framerate upsampling

[theok8577996]

I like to drop an idea to improve optical flow for framerate upsampling.

I do work a lot super8 filmscans with are 18 fps.

When I drop these in a 25fps sequence and enable optical flow I get great smooth results!

Optical flow however doesnot work wel with scene changes, you get wierd frames there.

Can optical flow be enhanced with scene detection prior the optical flow render?

(This also applies when you export with Optical flow enabled)

I also tried this with rapid droneshots with fast vertical movement which where shot in 25fps.

If you upsample them with optical flow to 50fps you get a smooth result.

Other stuff I tried was using a 4K camera like Sony AX53 with can only do 25fps.

If you walk around while shooting with this camera you get a bit restless feeling when viewing the result.

converting to 50fps looks much better.

[Shebbe]

Good idea. I'd like to share some thoughts and experiences.

- filmscans, especiallly 8 and 16mm will have a lot of grain. For image critical work it's probably not always ideal to use frame interpolation.

- smearing at cuts/scene changes is unavoidable in any method of generative frame interpolation. You have to work around that by giving it footage that does not have any cuts to other shots. A solution could be nesting separate cuts if you have multiple in a single file.

- Optical Flow as an algorithm can sometimes work well but the technique is flawed and can never improve much. If the scene has fast movement or large amount of complexity you are guaranteed to get image artifacts.

- The next best step is AI based upsampling. I would welcome a new AI based model that can superseed Optical Flow as a

native option in Premiere Pro. We use BorisFX's RetimeML tool wich has worked great for many shots so far but the nature of it being an effect makes it clunky to work with inside Premiere over it's regular speed/retime tools.