P: Video Cache is out of control

[jamesr33423836]

I recently upgraded to Lightroom 6.2.1 So far there has been a lot of fuss over the import dialog... ok, well sure, it has a few issues, but they can be either worked around or just revert back to 6.1.1

I would like to report and ACTUAL bug in 6.2.1

My D drive recently ran out of space... I tried deleting some stuff I didn't need, but still it kept running out of space... So I finally ran windirstat and had a look at it.. Nearly HALF of my drive was FULL of D:/temp/Adobe Local/Lightroom/caches/video/media Cache Files. Now here's the kicker... NONE of these video files are even on the hard drive that has my catalog on it. My lightroom database is on drive L:\ all my lightroom photos and very few lightroom videos are all on L:\ I have a few scans on drive K:\ that I import to lightroom... but these videos are ALL on Drives E:\ and F:\

here's another thing.. I knew Adobe wasted space caching videos, and I do have a GOPRO and I record a little video,.. .VERY LITTLE, maybe 1 video a year... so I don't really want ANY of the video cached.. I will wait for it to load... SO a long time ago, I set my video cache in Preferences>File Handling to the minimum of 1GB (It's was always a BUG to not allow 0GB) I don't want to cache any videos!!!!!!!!!! but here it is.. NEW BUG It's STILL SET TO 1GB!!!!!!!!!!! How is it gobbling up almost 1TB of space when I have it set to 1GB??????????????? So.... here's another problem... I have Cache files that are 4GB for a movie that only takes up 2GB on my hard drive.. what's up with THAT??????????????? I randomly checked a few, and the worst one was a 5GB cache file for a movie that only takes up 750MB on my hard drive... two things, first of all... you NEVER EVER EVER need to cache an entire movie... caching more than 1 minute of a video is useless... if you're going to watch the video in lightroom, it can just play the video on the hard drive fast enough that it doesn't need further caching.. second... DON'T CACHE THE VIDEO AT ALL!!!! just capture the thumbnail of it so you don't have a blank square on the catalog and call it a day... if someone wants to play the video, it will load and play plenty fast enough, Lightroom Can't work with video files, only play them, so WHY BOTHER TO CACHE THE WHOLE THING?? come on Adobe, some COMMON SENSE please!!!!

so now ANOTHER BUG, it's already set to limit video cache to 1GB, so I figure, the new import thing must have built this RIDICULOUS Cache, so I will just Purge it, that will bring it down to 1GB right??/ WRONG!!! It pops up a message saying "Video cache is being purged, this message will be dismissed when the purge is finished... I wait 5 Seconds, and the message disappears, I have NO hard drive activity, and well... I STILL have 1GB of Video Cache files!!!!!!!!!!! Ok, maybe it didn't take.. let me set it to 2GB, then purge.. NOPE! let me set it back to 1GB then purge, NOPE..

ok, so I suspect how to fix it will be, (I Hope) click import, and select the movies folder and then select Ignore source... by the way, how do I add my E:\scans folder as a legitimate source.. I just removed a source, how do I add a nice shortcut button for a new one? ok, now that source is removed, I hope it won't scan it again... now that it will HOPEFULLY not scan that hard drive again, I'll manually delete the offending Video cache.. ok manual delete complete, Ahh my drive can breathe again.. Lets open light room and see what happens.... ok.. open.. lets open the Import dialog because eventually I'll need to import something... oh-o..... What's this??? Scanning Common Locations... OH-NO!!!!!!!!!! first of all E:\Moves and F:\TV are NOT COMMON LOCATIONS FOR PHOTOS Second, I REMOVED THOSE LOCATIONS.. It has a valid location listed, WHY IS IS LOOKING FOR SOMEWHERE ELSE????? Yes it's now scanning VIDEOS and the only reason for it to be taking so long is it went back to my E and F drive... lets look at the cache folder... oh yes, MediaCache already has 3,876 files in it.... not looking very good. Why is it caching files that are not even imported into lightroom??? and why is the cache for each file taking up more space than the entire video??? If you want to make a video cache.. just cache the THUMBNAIL ONLY!!!!! we don't need or want anything else cached.. a thumbnail is all we need cached for our few relevant videos so we don't have a black square in our catalog.

Crap, I'm going to be FORCED to revert to 6.1.1 even though I had every intention of working around the other 6.2.1 issues. TEST TEST TEST Test your software!!! the monkeys you have testing it are not doing a good enough job! send it to me, I'll test it for you, It needs to be tested on a real computer, one that is used for other things.. that way you would KNOW it's finding stupid things like CD cover art and every movie and tv show on my hard drive. I seem to find glaringly obvious issues within one day of using the product. I've been developing software for the last 28 years, I know how it should be done, and how things should be tested, and this is NOT being done AT ALL. Your programmers are sub-standard and are missing the mark, and your non-existent SQA department is NOT testing even the simplest of functions.

[simonsaith]

@paul Can you help clarify what you meant by “The problem still exists."

With Lr 6.4, the expected behavior is that Lightroom would start more aggressively trimming the video cache at application starts, the completion of imports, and the playback of video to keep the size of the video cache under the hard limit of 2x the limit specified in the UI. Anything under the 2x limit is a soft limit.

Before the import finishes, the cache side could also exceed the said limit.

Hope this helps,

[meachp]

It seems self-explanatory to me but here goes. As soon as any video appears in the Import screen DynamicLinkMediaServer starts caching (even if you haven't imported anything) and doesn't stop when you exit Lightroom - you have to manually end it in Task Manager. I don't know how long it would run for or how much it would cache because I'm not prepared to risk letting it fill my hard drive up yet again. When I stopped it last night I had to get rid of 45 Gb of cached files. I'll be sticking to ACR for the foreseeable future.

[simonsaith]

Does it start trimming after the import complete? Or you leave the import dialog on overnight?

[meachp]

I'm not importing at all. I'm merely checking the update and I don't see any difference from the old version. Just to emphasise, I have never, ever, imported any video into Lightroom. But if, in the Import screen, I browse to any folder that contains video then Lightroom immediately starts caching all the video in that folder and all the sub folders. If I exit Lightroom the caching process continues, using up to 50% of my CPU. My C drive is a 112 Gb SSD and I just tried this again starting with 51 Gb free. When I manually stopped the process I was down to 4.6 Gb. I'm not prepared to let it finish (or, more likely, crash once my drive is full). I can't believe that this is expected behaviour - why does Lightroom feel the need to cache all these files when I have no intention of importing them?

[gordong23738575]

Dear Simon of Adobe,

I share Paul's frustration and he has described the issue very well.

These days we do video clips and photos all the time, so it will be great to have one product (i.e. lightroom) to manage both. In other words, I still hope that the videos can show up in the Lightroom collection.

The only issue you need to fix is NOT cache the video files.

Gordon

[meachp]

I have noticed one difference in the new version. In the old version (3) DynamicLinkMediaServer.exe doesn't appear in Task Manager until it becomes active. In the new version (4) it appears as soon as Lightroom opens, although it remains inactive until you browse to a folder that contains video, when it instantly kicks into life.

[simonsaith]

If you cancel the import or completes an import (even if no videos are imported), Lr should purge the video cache left over by while the import dialog is up. Lr 6.4 would also clean it up during application launch.

There is still a design hole in there that there is no mechanism to tell the DLMS to stop chewing what it has been told to digest once the import dialog dismisses. The team will continue to investigate on that.

[meachp]

Sorry Simon, but you're completely missing the point here. Let's be clear on this - there is NO import taking place so there is no completion or cancellation taking place either. Therefore there is no clean up or purge invoked. I'm simply browsing folders in the Import screen, and at no point am I initiating an import to cancel. As soon as a video icon appears on the screen this caching process commences - and it won't stop, even after exiting Lightroom, unless I stop it manually. So when is this clean up operation supposed to happen? If I let it run it will fill up my drive and crash. You need to turn this automatic caching OFF

[simonsaith]

I understand. That is why I said is a design hole that DLMS should have enforce its own caching limit when running. What Lr 6.4 fixed helps in certain situations (it is more as a short term workaround), but it is not addressing the core design issue. 

[meachp]

Well that pretty much means Lightroom is unuseable for anyone with a largish collection of video files and limited space on their C drive (like anyone using a SSD) - not everybody has multiple drives to keep all these things separated. I only have about 50 Gb free and this app fills it in less than 20 minutes. And it fills it with unwanted, pointless, inflated files that I will never need because I don't import video into Lightroom. Why can't you just cache files that are actually imported instead of every video file that comes into range?

[TomFidd]

I haven't noticed the behavior of caching things LIghtroom hasn't touched, but maybe I'm not paying close enough attention. Anyway, caching the videos in my collection is already troublesome.

But I think the whole premise of caching is flawed. Its ostensibly done for smooth playback - but who uses Lightroom as a video player? There just isn't a good reason to cache files in their entirety.

For me, the purpose of a cache ought to be to serve as a video thumbnail, for fast previewing. That wouldn't require caching the whole file, nor would it require full resolution. Just give me the first ten seconds, and some key frames later at wide intervals. If viewing is extended, play at lower quality, or let me hand off playback to VLC (or whatever).

If each file were cached to a thumbnail like this, it wouldn't hurt much for DLMS to go hog wild.

I'm glad this is getting some attention.

[TomFidd]

You can eliminate caching yourself by renaming the dynamiclinkmediaserver helper, or its folder, but then you have no video preview in LR. So there's also a dependency on cached video that has to be addressed.

[TomFidd]

I'm not sure that would solve the problem either. As long as the cache files are bigger than the originals, DLMS is going to hit the cache size limit right away whenever it hits a folder with big videos in it. After that it would just be churning, throwing out one file to make room for the next.

[meachp]

Having done another test I'll be fair and concede that when you open Lightroom, it does clear out the media cache and that is an improvement I'm grateful for. But that's no use whatsoever when this app sets off and fills up your hard drive, causes Windows to delete your Restore Points to free up space, and then crashes anyway because your drive is full, and all just to cache files pointlessly. Instead of getting Lightroom to tidy up after itself you'd have been better off stopping it making such a mess in the first place. Can you not answer the simple question that has been asked on this thread several times now - "Why does Lightroom cache video files that have not been imported?". If it didn't do that we wouldn't need to have this discussion - and this thread wouldn't exist.

[meachp]

Tom, nobody would ever notice until, as in my case, their C drive filled up. I had so little space left I couldn't install LR CC until I removed LR 5.7 back in April. Then, when I investigated, I found 65 Gb of cached video files courtesy of Adobe - and I have never imported any video into Lightroom. So don't be hard on yourself - I'm sure the world is full of Lightroom users with full (or nearly full) C drives who have no idea why! Nor what to do about it except delete/move other files that they'd rather stayed where they are.

[ssprengel]

LR caching all the files in a folder because that folder is open in an import grid is like a computer caching all the files in a folder in RAM just because you open the folder in Explorer/Finder.

Computers don't do this because RAM is precious. LR should start treating disk space as precious and be smarter about what it's doing.

Filling up a disk with pre-processed video files before LR knows whether they will be used isn't smart.

Churning through a folder of video files creating then deleting the cached file because the video cache is full isn't smart.

[gordong23738575]

I am very curious how Adobe, a reputable company, allows this huge bug to be there consistently over several versions of LightRoom. Are we the only guys in this world that have video and photo files in one folder?

PLEASE fix it ADOBE!

[TomFidd]

That's how I discovered the problem too - 90GB of cache on my SSD.

However, I thought all the cached files originated from things in my catalog. If it's also caching things that are just touched by the import process, or the whole hard drive, that's worse, but either way it would be a problem for me.

If 6.4 at least clears the cache when it opens, that'll be a huge improvement.

[jamesr33423836]

I don't understand why the new released 6.4 still does not allow me to set the video cache to zero...  I don't want or need my videos to be cached.. just show me a thumbnail and if I choose to play one,  just open it and play it and let it be slow.. I don't care how slow it opens and plays.  All I care about is that the thumbnail image loads as quickly as the rest of my photos.    All the videos I open and play with VLC media player take no time to start playing and They are defiantly NOT cached.. so why is this cache even needed??? it makes no sense.  It serves no purpose at all.   I have deleted the cache and opened light room and played videos in there before it had a chance to cache anything, and it's fine opening un-cached videos

The quick easy fix to this problem is to remove video caching from lightroom all together.. problem solved!!  everyone has computers that are so fast, no one will even notice!  but we will ALL be happy that our space limited SSDs are not full of crap.  

Just take it out... it's obsolete and not wanted and not needed by ANYONE.  If you have a slow computer, caching the videos isn't going to help enough to make a difference, and if you have a fast computer, they never needed to be cached in the first place.    please note... no serious user of lightroom has a slow computer... and we're the ones who pay to constantly upgrade lightroom.. If you are on a slow computer, you don't care enough about it to bother upgrading it either.. it's just a toy to you.

So I say, skip setting the cache to zero.. just remove it completely then EVERYONE is happy!

[TomFidd]

I would guess that the cache could even make things worse on some slow computers, because the cached files are bigger than the originals. Disk access is probably a limiting factor more often than decompression.

[jamesr33423836]

that is very true.  It doesn't take anything to play a video, I was doing it on my old 386 with windows 3.11 when I had 16MB of RAM and a 20MB hard drive.  nobody has a computer that slow anymore and I never had to wait an uncomfortable amount of time for a video to play.   I was recording and playing back live TV on a windows 95 machine with an ATI video capture card, and again.. no caching was ever involved... just play tv shows just fine right off my hard drive.   the fact is that the slowest computer that can even run lightroom 6, is blazing fast in comparison to what is needed to get a video playing fast enough. 

So.. given that a video cache is never needed,  trying to build a cache is using system resources, is just slowing down your computer for no gain what so ever.. and since this video cache is broken, and it's simply a hazard - gobbling up TONS of resources  for no reason at all.. just do away with it..    I truly expected it to be gone with the release of 6.4 and I'm very disappointed it's still there.   The processor power, ram, and SSD space being wasted by this insane process is disgusting, and it's crashing peoples computers by filling C until it's full.. when you delete stuff to make room, it comes right behind and fills it back up unless you realize what's happening and terminate it... but MOST computer users DON'T know whats happening, they DON'T know how to figure it out, and they DON'T know how to terminate it.. they take their computer to the shop and they don't know why the drive it full,   they figure it's just a bunch of junk, they don't have time to truly diagnose the problem, so they wipe the drive and do a factory restore on it.... 

The amount of problems this retarded video cache HAS Casused and is STILL causing is staggering... delete it from lightroom..  It's BROKEN!!!!  if someday you develop a video cache that works, fine.. but TEST it first... but until then, ADOBE is doing a GREAT Disservice to ALL it's customers by leaving this BAD CODE in place

[FeedbackCommunityMember]

For what it's worth, I just bought Lightroom on Jan 31, and I got myself within 25 GB of filling my Mac hard drive due to this issue. Here are my technical specs and application settings:

Lightroom version: 6.0 [1014445]
License: Perpetual
Operating system: Mac OS 10
Version: 10.10 [5]
Hard disk size: 500 GB
Free space on hard disk before buying Lightroom : ~230 GB

LR Preferences > File Handling > Video Cache Settings > 
(checked) Limit video cache size
Maximum Size: 3 GB

My current catalog has only 6 (six) video files, about 5 minutes total of video. During the import process, I used the LR Import to evaluate my entire 500 GB hard drive, plus a 3 TB external network arrayed storage device containing point in time back ups of the Mac. Out of the hundreds of videos I have stored (mostly Coursera course lecture videos) I accidentally imported only 6.

In the space of about 4 days, Lightroom's video caching process used 202 GB on my hard drive. I only found this by accident. Last night I just happened to notice that my machine had only 50 GB free. While watching it over the space of 10-20 minutes, I saw another 25 GB getting eaten up by an unknown process. Given that I spent yesterday doing LR imports, I knew it had something to do with LR, but did not know what. Repeated frantic checking of the LR photo directory showed a 5.75 GB Lightroom photo library. Panicked shut down of the Mac and the NAS ensued. Fortunately the process that created these crap files was halted by the shutdown.

These are the issues as I see them:

• LR is creating permanent cache files for every single video file it touches in the import process.
• There is no preference option to exclude video files during import so the creation of video cache files can not be avoided.
• There is no option at end of import process to interrogate user's desire to keep or discard cached files from last import task.
• There is no step during the import process to assess hard drive capacity against the projected import caching space requirement to ensure hard drive capacity is not exceeded by the import process. A warning should be displayed to user before import caching that is projected to exceed hard disk space.
• LR is not abiding by the application preference settings. I had a 3 GB limit set. LR ignored that limit, and blew by that limit by a factor of 67 *WITHOUT* a single application warning and did this over the space of 4 days.
• After meticulous searching through my hard drive, did I find this cache directory, find the problem reports and the recommended use of the "Purge Cache" button in preferences. Use of this button does *NOT* delete the cache files.

The import feature is fundamentally broken. In my years of computing, I can't recall an application that behaved in such a rogue fashion since the early 90's.

When is this going to be fixed?

[jamesr33423836]

early 90s?  nope never seen anything like this madness then.. not even on my old old 8088 did we have such horrible software that served no purpose whatsoever.

This does NOT need to be fixed.. it NEEDS TO BE REMOVED COMPLETELY!!!!!  

It was a bad idea when it was conceived, it serves no purpose, and it is a Hazard.  The code was produced by a bunch of monkeys banging on their keyboards who never wrote a real line of code in their lives,  it's just all linked copied and pasted crap from old obsolete libraries.  There is no excuse for this kind of shotty programming and I can't understand why the video cache is not gone yet. 

[FeedbackCommunityMember]

Further technical details.

Total number of video files imported into LR catalogue: 6
Total number of cache files created: ~1,180
Number of cache files with size greater than 1 GB: ~75 files
Largest cache file size: 3.4 GB

I find it very curious that a single cache file could individually be larger than the application preference total cache size limit, and be created without an application warning / message box of any kind.

[bob frost]

Have you tried deleting the cache files using your file manager, and then making the cache folder 'read-only' so that new cache files cannot be written to it? That works with the acr cache folder, so I don't see why it shouldn't work with the video cache folder.

Bob frost