Editing Video - wanting to remove lens smudges and dirt from video

[Philip22255932eaa9]

I have travel video taken with a 360 camera on the front of a motorcycle, where bugs or a general foreign object landed on and stuck to the front lens.  I would like to edit the video (frame by frame if-as necessary) to correct this situation.

What I have tried - I recently purchased a subscription to Photoshop 2024, using it to edit video.  I do understand that Premier is the primary Adobe application for video, but having used PS Elements for a long time, I thought to try it first before considering other options.  I have used the 'remove tool' with some success, removing an insect smudge located on the video in clear blue sky.

The MAIN problem I need help with is correcting the circumstance where the background is changing and the smudge or insect (the anomaly) is stationary on the lens but moving across the video, where the background is in motion.  IMPORTANT - where the anomaly moves, frames of video BEFORE and AFTER a frame are clear of the anomaly in a specific location.  My idea is to COPY a section of a previous frame where the anomaly does NOT cover a location and PASTE it into that location in following frames.

I was able to use the 'lasso' tool and copy function to select the area clear of the anomaly in 'frame 1', but when I attempt to paste it into a 'frame 2' using every paste option (paste, paste special - into, outside, etc.) I get nothing.  I can see that a new layer is created, but no method I have found thus far pastes the copied section into the new frame.  I have tried rearranging the layers in the sequence, flattening, etc. but the copied area never appears in the new frame.  Often, the entire video is reduced to one image (?).

I understand that PS 2024 may not be the best application to edit video, so I am reaching out to video editors to find a basic way to accomplish what is, admittedly, a bit of a complicated process.  Would the Premier program accomlish this method more easily?  The smudges and anomalies are keeping me from completing a project, and I have trying to find a method to satisfactorily correct this video for months now.

Sorry for the novel, but I also wanted to emphasize that my video editing is primarily limited to just corrections and basic functions - I typically do not stylize images or video, so I would also like to know if PS Elements (which I am very accustomed to using) or Premier Elements 2024 as more basic programs have the AI 'remove' tool and could accomplish the 'copy and paste' process that I am trying to accomplish.  My Mac Mini is an older model, and while a new M2 purchase is planned, I am thinking that PS Elements - Premier Elements might be less taxing on the older mini.

If someone out there has a reasonably straightforward method of accomplishing what I need to do here, please respond, and thanks!

[Conrad_C]

The MAIN problem I need help with is correcting the circumstance where the background is changing and the smudge or insect (the anomaly) is stationary on the lens but moving across the video, where the background is in motion.  IMPORTANT - where the anomaly moves, frames of video BEFORE and AFTER a frame are clear of the anomaly in a specific location. 

By @Philip22255932eaa9

Yes, this is a common problem, but when the content moves around the frame over time, the correction is not easy to animate across a large number of successive frames in Photoshop. But as you suspected there are other tools (more on that later)…

My idea is to COPY a section of a previous frame where the anomaly does NOT cover a location and PASTE it into that location in following frames.…but when I attempt to paste it into a 'frame 2' using every paste option (paste, paste special - into, outside, etc.) I get nothing.  I can see that a new layer is created…

By @Philip22255932eaa9

It sounds like that’s working as expected…copy/paste is not normally how this would be done, partly because Photoshop doesn’t alter the content within video frames, and partly because that’s a lot of manual frame-by-frame pasting. If I was to do this in Photoshop, the correction would be on a separate layer, and then its Transform property animated over time (using keyframes). But having to manually create a different keyframe every time the content moves is still extremely tedious and time-consuming, and so it’s not the recommended method.

Premiere Pro does have a more automated way to do that, using its Motion Tracking feature. The corrected smudge could be attached to a motion tracking element that automatically follows the smudge around the screen.

But probably the easiest/fastest way to fix this (because it’s the most automatic) would be using the Content-Aware Fill feature in the specialized visual effects application Adobe After Effects. This is like the Remove tool combined with Motion Tracking, so you can cut out an unwanted object in After Effects, click a button, and it will follow that around the screen, removing it from every frame.

Adobe help article:

Remove objects from your videos with the Content-Aware Fill panel (in After Effects)

Quick video demo:

[Philip22255932eaa9]

Conrad - thank you for the knowledgeable response!  I had mentioned the older Mac Mini that I have, because I actually have tried the After Effects program, but my computer was really not up to it.  It took a LONG time to process the corrections I tried, and I was continually aware that the computer was on the edge of crashing each time I tried something using the AE program.  PLUS, while the fact that After Effects is a very capable program, it is mind-numbing to get up on when all you're trying to do is clean up a smudge or a bug on the lens.

Even using the 'remove' tool in Photoshop required a frame-by-frame editing process, and I understand very well how tedious that process is, but I would be willing to do it if I could get a QUALITY correction using a copy and paste method.

Corrections of this type take an accuracy that I think only an AI process may be able to accomplish.  Even using the very effective 'remove' tool in PS leaves artifacts of its algorithm that change from one frame to the next, so it is obvious when running the video that edits were tried - it's almost as obvious as the smudge itself.  Copy and Paste in the traditional Photoshop method could accomplish this better I think, with a little 'transforming' the copied area to match the background area in the pasted frame, and I would grit my teeth and do this frame by frame if after hours of working I could get satisfactory results.

I used an Insta360 camera on these videos (a week's worth), and if you know the camera and the application the company provides, you can direct viewer attention in any direction in a 360 degree view.  My idea is to work around the 'face forward' view where possible, but ultimately, you have to give the viewer a face forward view at some point, because it's really necessary for a what?... a satisfying viewing experience I guess you'd say, so I would do the 'frame-by-frame' in long enough segments of a face-forward view to accomplish that.  It is only in the 'face forward' view that bugs and smudges are apparent, for obvious reasons.

Anyway, thanks again Conrad - I will look at the video and study your suggestions and try to assimilate.  In the meantime, I still would like to know if PS Elements and/or Premier Elements would accomplish what amounts to basic functions (copy-paste, AI remove, etc.) on video, even if this would require editing frame by frame.  Since I know the PSE program, and I am thinking Premier Elements may be similarly a lower level of complication than the "pro" version, this may make a complicated process less complicated.  If anyone out there who knows can comment on this, please do.  Thanks, Phil.

[Conrad_C]

In the meantime, I still would like to know if PS Elements and/or Premier Elements would accomplish what amounts to basic functions (copy-paste, AI remove, etc.) on video, even if this would require editing frame by frame.

By @Philip22255932eaa9

I don’t know very much about those applications, so I hope someone else is able to fill in that part of the answer.

Having an Intel-based Mac mini can be a limitation; the Apple Silicon Macs are certainly a big step up in performance.

[Philip22255932eaa9]

Conrad, or anyone who may know for that matter - why doesn't the 'copy and paste' idea work in Photoshop for video (copying area of previous frame - pasting to new frame), where the copied area will not paste into the new frame using any paste method tried.  Does the method I am trying to use as described not work for some reason - i.e. an inability of the PS program to copy-paste when used with video, or are the steps I am using to try to accomplish this incorrect?  I can do this with two still images - why not two video frames?  You had mentioned creating a new layer to accomplish this - can you give me basic instructions on trying this?  If I were able to accomplish this using a new layer, are there other steps necessary to combine correction layers into the original video timeline to export (as .mp4). 

And yes, I get that this will be a very tedious frame-by-frame correction, and that After Effects would likely be the way to go.  I will certainly revisit buying and learning the program after purchasing an M2.  Again, your input and the time taken to respond here is greatly appreciated - Phil.

[Conrad_C]

This is going to be an over-simplified explanation where some details might not be quite right, but part of the reason is how video is encoded and compressed. Extreme amounts of compression are used to make high resolution digital video practical to use on an affordable computer. One way they do this is to not store all information about every frame. A video might use “key frames” every few frames, and that key frame is a complete frame at full quality. The frames in between might only be “difference frames” where the only information stored is the difference betwen it and the previous frame or the key frame. If you were to try to get at or modify the exact pixels of an individual frame (like pasting pixels on it), information would be missing.

When you look at a frame stored in a common video format like H.265, the application is constantly looking to nearby frames, especially the nearest preceding key frame, to figure out how to fully reconstruct what needs to be in the frame you are looking at.

Even in dedicated video editing applications, you typically don’t get to address individual pixels of each frame. You add a separate overlay of frame-by-frame adjustments using parametric keyframes, not copying and pasting, partly because parametric keyframes are infinitely easier to control and edit over time. The adjustments are combined with the original frames at export, re-rendering the pixels of each frame from scratch at export. Photoshop can do no better.

I tried to think of an analogy, but it isn’t easy. The closest I could come is flying in an airliner. Everything works as long as the plane is at cruising speed, but it is not like we are used to in a car: At no point can you ask the pilot to stop at exactly one spot in the air.

You had mentioned creating a new layer to accomplish this - can you give me basic instructions on trying this?

By @Philip22255932eaa9

I’ll try to do this a little later today, have to take care of some other things at the moment.

[Conrad_C]

I spent a little more time looking at it, and there seems to be a third approach (other than the one you were trying, copy and paste, and the one I was thinking of). Have you tried the approach shown in the demo below?

I realized that there is a flaw in the copy/paste method in that the selection includes a fixed set of background pixels , so in subsequent frames there’s a chance they won’t match up with the changing background and the edges will be obvious. I played around a little and it seems to work fine if you set the Remove tool to Sample All Layers and paint it on a higher video (not still) layer, so all you have to do is go frame by frame and paint the correction in each one. That lets the Remove tool adapt the healing to the specific pixels in each frame.

The example below removes a reflection from a train window in front of a constantly changing background, painting the Remove tool correction on new video layer Layer 1. Before choosing Layer > Video Layers > New Video Layer, it’s important to click to select the Video Group but not any clips in it.

This works because we know Photoshop can’t paint on the actual video layer (for the reasons I mentioned earlier), but if a new empty video layer (not still layer) is created, Photoshop can merge the corrections on that new video layer with the original video, when you have Photoshop render a new video from this sequence.

[Philip22255932eaa9]

Okay, I am trying this now - thanks!

I did try the remove tool frame by frame on the original video timeline, and the tool would work, but given the limitations of a tool designed to be used primarily for still images, each 'remove' attempt would render a different-looking corrected area, so when running the video, the corrections look as obvious as the smudge.  With a small spot moving horizontally across the video,  each use of the remove tool creates a new corrected area that does not need to match the preceding frame background.  The problem I am having is that I am moving forward in the timeline, either walking a trail or riding a motorcycle, where the camera is mounted on the bike or held stationary, so for two or three frames at a time the smudge overlaps or partially overlaps the same location.  For this to work, I have to get corrections to match the existing background as the smudge moves past the area.

The idea would be that I'd "transform" the copied area (skew, perspective, distort, etc.) to "match" the copied area to the new frame, if I could get the method to work as it does on a still image.   The smudges are not 100% opaque, so I can see detail "below" the smudge well enough to 'line up' the pasted layer (I think!) and I would use some feathering in the copy-paste process so that the edge of the pasted area would hide (ideally).  Can this be done on a layer?  I guess I'll try that too.

Will adding the layer somehow change how the remove tool performs its algorithm?  I guess I'll see.  I have set it up as per your instructions and the video.  Good stuff.

Ultimately, I think I'm going to need to get an M2 and learn After Effects!  Thanks again, Conrad.  I'll let you know how this turns out. 

[Philip22255932eaa9]

Also, a beginner question - is there a method to move the video frame by frame in Photoshop by using keyboard keys?  Other applications use the right-left arrow keys to do this, but that does not work in Photoshop.  Having to move the cursor from where I am making a correction on the frame down to the single frame arrow and back is adding a lot of time to this frame by frame correction thing.

[Philip22255932eaa9]

I got it to work by creating the command 'Ctrl+cmd+1' in Workspace/Keyboard Shortcuts and Menus/Timeline Video.  A bit of a hassle when virtually all other apps use right or left arrow keys.

[Philip22255932eaa9]

Okay, so I got the copy-paste idea to actually work.  The key is, as you said, creating a layer to do the work on.  I used that layer to use the remove tool to take out some lens flare earlier in the video.

I copied the area I wanted to paste from the original timeline, but it created another layer above that one, for some reason(?).  Once pasted, you can transform the copied area to "fit" in the new frame.

A couple of weirdnesses though - each attempt at this creates a new layer.  I am figuring out that I can transform the correction layer as needed, then cut the layer down to one frame after it's created (see 'layer 5' in the image - this is one frame - the other three layers represents two frames).  The copied area will remain as it was pasted and transformed for all following frames, and while it looks okay for small changes in perspective, I think I will have to redo this process correcting one frame at a time, two at the most for the best results.

I was able to trim and pull the layers down to one line (Video Group 2) but any attempts at merging them together and keeping them in sequence causes some of the "layers" to disappear.  I don't get this - highlighting them then trying to merge layers, merge visible, etc. causes problems.  My thinking is that it will "overwhelm" my computer to have layer after layer with one corrected frame each.  That's why I wanted to try to join the completed correction layers together in a continuous stream, to keep the cache from getting filled up.

But, all in all I am making progress on this idea.  Any pointers you can give on joining the layers and maintaining their sequence horizontally?

You know, I get that I am taking up quite a bit of your time here - you have assisted greatly.   Thanks again Conrad.

[Conrad_C]

Also, a beginner question - is there a method to move the video frame by frame in Photoshop by using keyboard keys?  Other applications use the right-left arrow keys to do this, but that does not work in Photoshop.  Having to move the cursor from where I am making a correction on the frame down to the single frame arrow and back is adding a lot of time to this frame by frame correction thing.

By @Philip22255932eaa9

That reason that happens is, because Photoshop is primarily a still image editor, the arrow keys nudge position by default, as is the industry standard convention in most photo editors. But in video editors, the arrow keys nudge the current time by default. The conflict you see is from a still photo editor also trying to be a video editor, so the conventional shortcuts for each app type must clash.

But, you can decide who wins. In Photoshop, it is possible to switch keyboard shortcuts to timeline navigation by opening the Timeline panel menu, and choosing Enable Timeline Shortcut Keys. This way you don’t have to edit any shortcuts yourself. But of course, when that’s enabled you can’t use the arrow keys to nudge position.

I’ll think about your other replies. But yes, a lot of the issues with seamless frame-by-frame consistency is what applications like After Effects are designed to take care of. That Content-Aware Fill feature in the video posted earlier is relatively new, but it represents the highest expression of this so far: Mark it once, then stand back and let it do its thing. Instead of frame-by-frame corrections.