How to automatically remove scenes when nothing happens

[Lioba22745708soxt]

Hi! I am working a lot with painting footage in which the brush leaves the scene and the clip just shows the painting and nothing happens in the frame. I was wondering if there is an automated tool that can detect frames where nothing happens - no movement - and extract or mark these parts on the clip for me. 

Until now I have been editing those manually out and with footage that is up to 12 hours long it takes a really long time. Does any of you know if there's a workaround, tool or extension that can do such thing?

Many thanks in advance!

[Jeff Bugbee]

Premiere does not have any native functionality to detect movement/motion. You'd need to use a third party plugin. I personally do not know of any that include this sort of (lack of) motion detection.

[Peru Bob]

Some folks say the tool to use is called "intern."

Just joking - no disrespect intended. 

[chrisw44157881]

if you set scene detection threshhold low, any scene detection plugin should work. free shutter encoder has a scene detect feature that exports out a XML of edit points, then just remove the clips in the middle.

some other options are resolve's scene detection, aescripts magnum 3 or rough cut. I even rolled my in AE for fun with transfer modes and a bit of trickery with pixel sampling, pretty fun. I'm actually surprised Premiere doesn't have a native function for this, its relatively simple to make.

[Lioba22745708soxt]

Thank you, Chris! I haven't worked with AE yet, but i will try out a scene detection Plugin for Premier! !!

[Lioba22745708soxt]

So a little update here! Adobe 2022 has a scene detection feature and I just tried it but it didn't detect any scenes, I assume my brush marks are just seen as one scene so it couldnt find anything ... so I am gonna try out the next options now.... 🙂

[basil1891]

@Lioba22745708soxt  Do you work with a 3rd party footage or you do the recording too? If the latter, to make PPro build-in feature work consider helping it: make some all-frame change at beginning-&-end of each painting session. Like closing camera lens (or alt-tabbing the app if you do a screencapture).

[Lioba22745708soxt]

Hi Basil! Thank you for your answer! Yes, I do the recording, too! ^^ I don't excatly understand what you mean by your comment through! - but currently I need to get some work done anyways before I come back to this experiment. I played a bit with your script a bit but I haven't managed to find a good constellation of the variables yet. I will probably try it out in a few weeks when I have more time again! 🙂

[basil1891]

If you don't mind sharing a small chunk of the source video in question, I can look into it. Should contain both scene types (w/motion - wo/ motion). Don't upload to youtube thou, use filesharing services instead (dropbox, onedrive, googledrive, etc).

In the previous comment I meant you to do some obvious visual (=on the video) 'marks' during recording, so automatic scene detection in PPro may actually work. It's just a suggestion, dunno if it will work.

[Lioba22745708soxt]

Yes, I can do that! Here is an exmple of when I painted a tiger: https://www.dropbox.com/s/cjxm2q7m5i2seug/Static%20Scene%20Test%20Footage.mp4?dl=0 

You can see when I put down my brush, leave the paper again and when my hand casts a shadow - ideally I only want the part when the brush is entering the frame!

I understand - that would be too much work though, when I paint I have to focus on the painting process and if the recording looks good - that is already stressful enough..! 😆

[basil1891]

There are some 3rd party tools that is able to remove frame duplicates. While their original intent was not for your case, it may work. Below is a sample of a batch-script configured for removing non-exactly dublicate (but very close to) frames:
1) you put it along with the ffmpeg.exe in some folder
2) create a shortcut to the bat-file on a desktop
3) drop your file on the shortcut
4) after encoding is finished check the folder 'NoStaticScenes'
Later in Premiere you have to change it's FPS to some std value (via interpret footage). This method should work, assuming you don't use audio track from the video in question.

@echo off
TITLE %~nx0
SETLOCAL ENABLEEXTENSIONS
set t0=%TIME%, %DATE%

for %%F in (%*) do call :main %%F
goto finalmessage

:main
 SetLocal
  echo -----------------------------------------------------------------------------------
  echo. Processing file: %~f1
  echo.
  set "newdir=NoStaticScenes"
  if not exist "%~dp1\%newdir%" mkdir "%~dp1\%newdir%"

  rem [ cmf: -1=auto, 1=bt709, 6=bt601, 9=bt2020 ]
:: set "cmf=-1"
:: set "bsf=-bsf:v h264_metadata=colour_primaries=%cmf%:transfer_characteristics=%cmf%:matrix_coefficients=%cmf%"

  ffmpeg.exe -y -i "%~f1" -vf mpdecimate=hi=64*30:lo=64*15:frac=0.1 -vsync vfr -pix_fmt yuv420p -c:v libx264 -crf 18 -preset veryfast -x264-params ref=1:bframes=0:keyint=15:min-keyint=1:qcomp=0.7:no-dct-decimate=1 -c:a copy -map 0:v:0 -map 0:a? "%~dp1\%newdir%\%~nx1 [nostats].mov"

 EndLocal
 goto :eof

:finalmessage
powershell write-host -fore cyan  ======================== Processing is FINISHED =========================
echo ----------------------------
echo Batch processing start time: %t0%
echo Batch processing end time:   %TIME%, %DATE%
echo ----------------------------
pause

[Lioba22745708soxt]

Thank you, Basil for the Script! I'm gonna try these things out and let you guys know if any of that works! 💫

[Kevin-Monahan]

Looking forward to hearing back from you so we can mark one of these solutions as the correct one. Good luck!

Kevin

[Lioba22745708soxt]

@basil1891 I tested your script! It did something... but I think it still detects too much movements like little shadows so it won't really cut out scenes. So my question is: Do you know if I could mess around with some of the variables in the original script (and if yes, which ones) so that it it has a larger "tolerance" - I hope that makes sense! And thank you in advance! 😄

[basil1891]

It's this part  "-vf mpdecimate=hi=64*30:lo=64*15:frac=0.1"  that do the (near)duplicates removal. As I understand it, the logic behind is next: A frame is for dropping if no 8*8px blocks differ by more than a threshold of hi, and if no more than frac of blocks have it between hi and lo ​(defaults are: hi=768, lo=320, frac=0.33).

Try to increase thresholds, like "-vf mpdecimate=hi=2300:lo=1100:frac=0.3"

To make testing faster you can add downscale filter, eg if your footage is 1920*1080, to make it half-sized just add:  -s 960x540. Another way is to cut a piece of footage with just a few typical "no movement scenes" and use it for initial debug. Some tools for lossless cutting: Avidemux, LosslessCut, Shutter Encoder.

[Lioba22745708soxt]

@basil1891 ohh thank you!!!! I will test it - I'm excited! ^^

[basil1891 · Accepted solution]

@Lioba22745708soxt 

 So, I did some tests with the sample file you provided above. I found that hi and frac had the biggest effect on duplicates removal. Increasing hi/frac removes more frames. For this specific sample working range for hi seems to be roughly between 4000(already works for this sample) and 5000 (if higher, unwanted frames may get removed). So I just set it to 4500 in the script below. As you can see in the attached sample, the parts with noticeable moving shadows still there, but first near-static 3sec completely gone.

One more change. Now the script produces ready to use 25fps file, and no more need for 'interpret footage' in PPro. Also, audio track is removed completely in output, there is no use for it anyways.

@echo off
TITLE %~nx0
SETLOCAL ENABLEEXTENSIONS
set t0=%TIME%, %DATE%

for %%F in (%*) do call :main %%F
goto finalmessage

:main
 SetLocal
  echo ---------------------------------
  echo.
  set "newdir=No_Static_Scenes"
  if not exist "%~dp1\%newdir%" mkdir "%~dp1\%newdir%"

  ffmpeg.exe -y -i "%~f1" -vf mpdecimate=hi=4500:lo=2000:frac=0.3,setpts=N/FRAME_RATE/TB -pix_fmt yuv420p -c:v libx264 -crf 18 -preset veryfast -x264-params ref=1:bframes=0:keyint=15:min-keyint=1:qcomp=0.7:no-dct-decimate=1 -an "%~dp1\%newdir%\%~nx1_[nostats]_.mov"

 EndLocal
 goto :eof

:finalmessage
powershell write-host -fore cyan  ============ Processing is FINISHED =======
echo ----------------------------
echo Batch processing start time: %t0%
echo Batch processing end time:   %TIME%, %DATE%
echo ----------------------------
pause

[Lioba22745708soxt]

Wow! @basil1891 ! This script is so amazing! It literally works like magic! I am gonna try this now out with some other footage but I wanted to ask you if there is any way for me to put multiple clips on the bat file and see how long it will take them to render? So I could maybe dump 50 files on the bat overnight and then get a info showing me it takes like 5 hours or something? Is this possible?

[basil1891 · Accepted solution]

Glad to hear it worked eventually :).
As to how long it will take, you can only assess it roughly. Assuming all of your videos are of the same resolution, you can remember the speed while processing some typical clip (see attachment), and then just divide total duration in minutes (for all clips in a batch) at that 'speed' value.

[Justin31377668o9un]

Hello! This has worked great for me in a different type of project, but I need to keep the audio.  Or... I need to make the exact same cuts to the audio that's being made to the video.  Is that possible?