Warp Stabilizing and Frame Blending in After Effects.

[Guy Burns]

I have some questions about Warp Stabilizing and Frame Blending in AE. I don't like AE. It's too complicated for my liking, but a few times a years I have to use it. So bear with me.

The project involves a 35-min home movie from the late 1960s. The scan first goes into After Effects, to adjust the frame rate from 16 to 24fps, by blending every second frame 50:50 with the next. I use a script that someone on this forum wrote for me several years ago:

n=3 //For m:n pulldown. Do not use for 1:2 pulldown.
f = timeToFrames(time);
f=f-2;
s =Math.floor(f);
if (s%n==n-2)
50
else 100

I don't understand it, but I know how to use it. In my opinion, it gives better results than the inbuilt frame blending. Anyway, from AE the blended composition is sent to  Premiere, where I have cut it into about 300 scenes, each scene edited for colour and contrast, and a frame or two removed at the splice points. Dozens of hours work so far. And I have a horrible suspicion I may have gone about this the wrong way if I want to use Warp Stabilizing. But I can live without it.

At the moment, in AE, I have two items in the Project window: a 24 fps composition, and the film itself at 16fps. In the window at the bottom I have two copies of the film. The top copy has the script applied, and both have been moved in a certain way to align them so that the script works properly. And everything works fine – in AE and in Premiere.

Ques 1

But it is not stabilized. The problem is: how do I stabilize and not have scene changes and frame blending bugger up the stabilizing? Is there any way I can stabilize inside my existing AE project and not have it interfere with what's already in Premiere?

I think this is what I should have done:

1. Import the 16fps film into AE, cut it into scenes, and remove the splice imperfections.

2. Stabilize each scene.

3. Somehow get that edited and stabilized 16fps version into a 24fps composition, ready for frame blending via the script.

4. Send to Premiere for colour and contrast correction.

Ques 2

Assuming the above steps would work, when I import a razored AE composition into Premiere, do the razor points appear in Premiere? Or would I have to manually apply 300 razor points?

Looking at all the above, I may have to forget stabilizing this time. But next time…

[Rick Gerard]

Your workflow makes no sense at all. There is no need to increase the number of frames. If the home movies were transferred to videotape then the frame rate will NOT be 16fps. It is highly unlikely that the original footage was converted to video using a telecine that gave you a frame for frame conversion. Before you can do any motion stabilizing you need to figure out if the footage is interlaced, if 3:2 pulldown has been added, and if the conversion of film to video was frame accurate. The difference in the cost of a frame-accurate film to video scan and a drop your home movies off at Costco and let us convert them to DVD can be up to four or five hundred percent. 

Here's how to analyze your footage:

• Import the footage into After Effects, select the footage in the Project Panel, and check the interpretation
• Turn on Separate Fields and choose upper or lower first, lower is common for NTSC footage
• Drag the footage to a new comp, then open up the composition and double the frame rate
• Step through the footage one frame at a time for at least 50 or 60 frames and observe the movement in the frame

If the footage is not interlaced you will see pairs of duplicate frames. Always pairs, never one unique frame. If the footage had 3:2 pulldown added then you will see pairs of frames, then three identical frames, then two, and two, then three. You will never see a single unique frame. If the footage is interlaced and the motion in the frame goes backward and forwards then the field order is wrong and you have to change it.  If the pulldown has been added and you see an occasional set of 3 identical frames then your footage had 3:2 pulldown and you need to experiment with the different settings in footage interpretation until you get no frames where the motion changes direction. All this sounds kind of time-consuming, and it could be. Most of the time you can figure out what kind of footage you have pretty quickly.

Once you figure exactly what you have got it's time to create some comps and start cleaning up the footage. 

If your footage is progressive and you have pairs of duplicate frames when you separate fields then the footage is progressive and you don't have to separate fields or do anything else with the footage. You should not change the interpreted frame rate unless you want to change the speed of the playback. There is very little to do but edit your footage.

If you have pairs of identical frames and occasionally a single frame, or if you occasionally see a blended frame then you just have to live with those few blended frames because the transfer of film to video was not frame-accurate.  You just have to live with the problem. 

If the footage is interlaced and has 3:2 pulldown, or is just interlaced you have to separate fields and you cannot change the frame rate of the video when you interpret the footage. 

Once you figure out what kind of footage you have you can create a 24 fps comp if you want, or a 30 fps comp, and just drop your footage in the comp and experiment with different frame blending modes. 

Frame blending via that script is not going to gain anything at all. If the footage was transferred frame for frame, you'll get a much better result by just using AE's built-in frame blending options. Trying to add 3:2 pulldown with the expression is not going to gain anything. 

If the transfer from film to tape did use 3:2 pulldown, and the video was edited, then you'll have to make a cut at every shot and analyze each and every shot to figure out when the 3:2 pulldown starts unless you are willing to live with just working with the interlaced footage.

Let's talk about Warp Stabilizing. First, it doesn't work at all on a lot of shots. Second, it will not work miracles. Third, it degrades the image. Fourth, it bends things like horizontal and vertical lines. Fifth, if there is a lot of motion the footage will be cropped. Last of all handheld 8mm home movies have very little detail to start with and detail is important for Warp Stabilizing to work. I would not count on using it for every shot. You'll be at the project for a very long time and probably will not be happy with the results. 

When you talk about razor points are you talking about edits? If so, then the best place to do the edits is in Premiere Pro. It is not at all a good idea to load up a 30-minute sequence in Premiere Pro with a bunch of dynamically linked AE comps. The likelihood of render failures and the near-impossible task of running previews will make the edit in Premiere Pro a nightmare.

I did a similar project for my mother's 90th birthday. Transferred about 20 hours of home movies to video after sending short samples to about 6 different service providers. I picked the best vendor and he delivered frame for frame, pin registered frames to me. I tested a few shots and determined that the most accurate average frame rate for the footage was 18fps so that's what I set all the footage to in Premiere Pro. Then I created a new 29.97 fps sequence and edited the footage, used Lumetri to color correct some of the scenes, and used AE to warp stabilize a couple of the shots. The project took me about 30 hours, but the final result looked better on a modern TV than it did when projected. In my humble opinion, you are not going to gain anything at all by starting in After Effects for most of your shots. Premiere Pro and AE both do a very good job of frame blending when frame rates don't exactly match.

I hope this helps. If you are still confused, upload 10 seconds of your footage and I'll take a look. I've been transferring film to tape since 1970 and I know what I'm talking about. 

[Guy Burns]

Thanks for the detailed response, Rick. However, my post was obviously not clear enough.

The film, shot at 16fps, was wet-gate transferred frame by frame, at 2.5k, ProsRes 4444, on a filmfabriek HDS+. Scanning costs were about $600. I only wanted the best (thus my pickiness with how I'm going to edit this film for an end result on Blu-ray). Previous scans of other old films were done at Videopro in Denmark on a Lasergraphics Scanstation, but for various reasons that option wasn't suitable this time.

Every time I have scans done, the first reel scanned is my test reel. Surprisingly, the filmfabriek (a $50k machine) which a young fellow in Melbourne has recently purchased for his business (and as far as I'm aware after extensive research, is the best scanner available in Australia), gave overall superior results to the $250k Scanstation. So, this time, the film was sent to Melbourne.

To be clear: there's no interlacing, no video transfer involved, no degradation so far of any kind, other than a direct frame-by-frame scan. Now I want to find out under what conditions Warp Stabilizing is possible and/or desirable.

As for the way I've blended frames, to my eye it gives the best result. Before deciding on this method, I compared three techniques in going from 16fps to 24fps:

1. Frame blending turned off, which simply repeats every second frame. The film will be sharpest, but with jerky motion.
2. Frame blending turned on. Adds frames as above, but blends every frame with the two around it. Every frame is therefore blurred, but the motion should be the smoothest – but blurred.
3. Add frames as per 1, but the added frames are a 50:50 blend of the repeated and the following frame. My idea was that I retain original sharpness for two out of three frames in the final film.

I applied the techniques to the same 15-second clip, converted to Blu-ray for viewing on our 3-metre projection screen, invited my partner and three friends for the test, and away we went.

Test 1

I simply asked them could they see any difference in sharpness of jerkiness between the three clips. No one could, except that my partner (accustomed to my numerous tests, and with a more critical eye than most), said that number 1 looked "jagged". No one could see any difference in sharpness between all three.

Test 2

I explained the difference between the three clips and what to look for. This time they could all see the jerkiness in #1, but no difference in jerkiness between #2 and #3, and no real difference in sharpness between any of them.

My conclusion: for a general audience viewing well-photographed home videos, there was no difference between #2 and #3, projected at a viewing distance of 1.5 screen widths.

Later, I repeated the test myself, really looking for the differences. It was not a blind test, I knew what was on the screen, but in terms of sharpness and fluid movement, to my eye #3 was the winner.

I doubt whether the combination of scan and blend method described above can be bettered. Now I want to experiement with stabilizing. Some of the film was shot on tripod, so it's pretty good for an amateur effort. But other scenes are handheld, or taken on an aircraft or while bushwalking, and it's those that I want to try and improve.

Ques

To sum up: what is a suitable technique for ending up with 24 fps video from a 16 fps film, scanned at pretty near the highest quality available, with 300 razor points (cuts, scene changes), frame blended as per the script (every second frame repeated and 50:50 blended with the next), and each scene individually colour and contrast corrected, some of which I may want to stabilize?

As much as anything, this is a technical exercise. I want to find out the best way to do it, and I want to try it.

[Rick Gerard]

If you have clean frames and you want to have the best results invest in Twixtor Pro. Load your 16 fps footage in a 24 fps comp and turn off the footage layer visibility, add a solid, apply Twixtor Pro to the solid and pick the footage as the source, then let Twixtor do its magic by predicting the vector motion between frames and generating the best possible result. You will achieve far better results than you can get using any other technique. Your frame blending expression is kind of faking 3:2 Pulldown with interlaced video using standard telecine transfers. 

Any warp stabilizing you do is going to work better if you use the original footage in a comp with the same frame rate, then render a suitable DI that you can use in a 24 fps comp using Twixtor. 

I would still do all my editing in Premiere Pro, but I would edit in a 16fps Sequence, including the Warp Stabilizing because Premiere Pro does a great job at that, then either render a DI for use in AE with Twixtor or give Twixtor for Premiere Pro a try. I own Twixtor for AE but I'm assuming that Twixtor for Premiere Pro works just as well. 

Don't expect fast render times. I'm not sure doing any of this work in AE is going to give you any significant improvement in the final quality.

BTW, I thought Australia was a PAL country with 50Hz power and 25 fps TV.