P: Video Cache is out of control

[jamesr33423836]

I recently upgraded to Lightroom 6.2.1 So far there has been a lot of fuss over the import dialog... ok, well sure, it has a few issues, but they can be either worked around or just revert back to 6.1.1

I would like to report and ACTUAL bug in 6.2.1

My D drive recently ran out of space... I tried deleting some stuff I didn't need, but still it kept running out of space... So I finally ran windirstat and had a look at it.. Nearly HALF of my drive was FULL of D:/temp/Adobe Local/Lightroom/caches/video/media Cache Files. Now here's the kicker... NONE of these video files are even on the hard drive that has my catalog on it. My lightroom database is on drive L:\ all my lightroom photos and very few lightroom videos are all on L:\ I have a few scans on drive K:\ that I import to lightroom... but these videos are ALL on Drives E:\ and F:\

here's another thing.. I knew Adobe wasted space caching videos, and I do have a GOPRO and I record a little video,.. .VERY LITTLE, maybe 1 video a year... so I don't really want ANY of the video cached.. I will wait for it to load... SO a long time ago, I set my video cache in Preferences>File Handling to the minimum of 1GB (It's was always a BUG to not allow 0GB) I don't want to cache any videos!!!!!!!!!! but here it is.. NEW BUG It's STILL SET TO 1GB!!!!!!!!!!! How is it gobbling up almost 1TB of space when I have it set to 1GB??????????????? So.... here's another problem... I have Cache files that are 4GB for a movie that only takes up 2GB on my hard drive.. what's up with THAT??????????????? I randomly checked a few, and the worst one was a 5GB cache file for a movie that only takes up 750MB on my hard drive... two things, first of all... you NEVER EVER EVER need to cache an entire movie... caching more than 1 minute of a video is useless... if you're going to watch the video in lightroom, it can just play the video on the hard drive fast enough that it doesn't need further caching.. second... DON'T CACHE THE VIDEO AT ALL!!!! just capture the thumbnail of it so you don't have a blank square on the catalog and call it a day... if someone wants to play the video, it will load and play plenty fast enough, Lightroom Can't work with video files, only play them, so WHY BOTHER TO CACHE THE WHOLE THING?? come on Adobe, some COMMON SENSE please!!!!

so now ANOTHER BUG, it's already set to limit video cache to 1GB, so I figure, the new import thing must have built this RIDICULOUS Cache, so I will just Purge it, that will bring it down to 1GB right??/ WRONG!!! It pops up a message saying "Video cache is being purged, this message will be dismissed when the purge is finished... I wait 5 Seconds, and the message disappears, I have NO hard drive activity, and well... I STILL have 1GB of Video Cache files!!!!!!!!!!! Ok, maybe it didn't take.. let me set it to 2GB, then purge.. NOPE! let me set it back to 1GB then purge, NOPE..

ok, so I suspect how to fix it will be, (I Hope) click import, and select the movies folder and then select Ignore source... by the way, how do I add my E:\scans folder as a legitimate source.. I just removed a source, how do I add a nice shortcut button for a new one? ok, now that source is removed, I hope it won't scan it again... now that it will HOPEFULLY not scan that hard drive again, I'll manually delete the offending Video cache.. ok manual delete complete, Ahh my drive can breathe again.. Lets open light room and see what happens.... ok.. open.. lets open the Import dialog because eventually I'll need to import something... oh-o..... What's this??? Scanning Common Locations... OH-NO!!!!!!!!!! first of all E:\Moves and F:\TV are NOT COMMON LOCATIONS FOR PHOTOS Second, I REMOVED THOSE LOCATIONS.. It has a valid location listed, WHY IS IS LOOKING FOR SOMEWHERE ELSE????? Yes it's now scanning VIDEOS and the only reason for it to be taking so long is it went back to my E and F drive... lets look at the cache folder... oh yes, MediaCache already has 3,876 files in it.... not looking very good. Why is it caching files that are not even imported into lightroom??? and why is the cache for each file taking up more space than the entire video??? If you want to make a video cache.. just cache the THUMBNAIL ONLY!!!!! we don't need or want anything else cached.. a thumbnail is all we need cached for our few relevant videos so we don't have a black square in our catalog.

Crap, I'm going to be FORCED to revert to 6.1.1 even though I had every intention of working around the other 6.2.1 issues. TEST TEST TEST Test your software!!! the monkeys you have testing it are not doing a good enough job! send it to me, I'll test it for you, It needs to be tested on a real computer, one that is used for other things.. that way you would KNOW it's finding stupid things like CD cover art and every movie and tv show on my hard drive. I seem to find glaringly obvious issues within one day of using the product. I've been developing software for the last 28 years, I know how it should be done, and how things should be tested, and this is NOT being done AT ALL. Your programmers are sub-standard and are missing the mark, and your non-existent SQA department is NOT testing even the simplest of functions.

[jamesr33423836]

Here's an update to this... After changing my video cache to 2GB then back to 1GB, while it did add 3876 files to the Media Cache folder, they are all 1.4K .mcdb files.. and so far, it has not grown. total is an acceptable 3.6MB I have left it on the Add photos screen for a long time and while it still says scanning common locations: scanning videos, the cache folder is not getting larger.. yet. Since I have exactly 205 legitimate Videos in my Lightroom catalog on my L: drive and 3876 .mcdb files, I think I must have one .mcdb filr for every video on every drive on my computer. I'm going to just leave lightroom open overnight, but not on the import screen to see what happens

[jamesr33423836]

I'm reporting back on this topic.
Last night I checked the video media cache, it was 576MB, which I feel is acceptable given the 1GB limit I have set. I left lightroom open all night in the library module, and then checked the cache, it was still 576MB, and I also checked the video media cache files themselves, they all were all either gopro or cell phone videos that were actually included in my catalog. So thats good news... but to try to find where the problem is... I opened the import, just left it open on the initial screen on v6.2.1 of lightroom, and after working on other things for 2 hours, I've come back to check on it... now, my cache is 1.6GB... oh o, thats a problem, it's ignoring the 1GB limit.

[J453]

Did you have this problem prior to 6.2.x update? If you roll back to 6.1.1 do you have the same issue?

[jamesr33423836]

I did not have this problem with 6.1.1

I went straight to 6.2.2, So I can't say if any other versions are doing this.

I still have 6.2.2 installed, I left it specifically so I could help provide information to you on this issue.

The current status of my D:\temp\Adobe Local\Lightroom\Caches\Video\Media Cache Files folder is 182 items totaling 3.05GB. MY MAX Video cache size is set to 1GB still, so it is currently 3x larger than it should be. I am happy to report that all of the cached videos are either from my gopro or my cell phone, and they all do exist within my lightroom catalog, however, I have not accessed any of the videos or even looked at the thumbnails for them, so I don't know why they are cached.

While I'm not thrilled that my setting of a 1GB limit is being ignored, It's at least not filling my hard drive with 900+GB of video cache files. The problem here is, I can't tell you if this particular behavior was happening on 6.1.1 or not. it would seem that the video cache filling my hard drive with a cached copy of every single video on my system, even on drives with nothing to do with lightroom was a one time event... I would be happy to uninstall lightroom, and re-install version 6.1.1 then leave it for a few days, and report on the status of the video cache, then upgrade to 6.2.2 and see if it again attempts to fill my drive with video cache files.

I really feel that with SSD's becoming so popular, and with that, people are sensitive to excessive activity on their C: drive, It would be very beneficial to allow an option to disable video caching altogether, I personally see no benefit to the video cache whatsoever, and would rather wait for a video to load should I choose to play it in lightroom, I suggest still capturing the preview and displaying that, so there aren't black squares, but to cache the video and create a file that is larger than the original video is just a waste of activity on my drive. I happened to move my temp folders to another drive, but most people don't do that.

For another suggestion, with modern hardware, SSDs and SATA III hard drives, It's quite possible and likely that caching of files on such systems is completely obsolete now. Even my mechanical hard drives are blazing fast. It may be time to allow the disabling of the camera raw cache as well... The cache may end up actually slowing things down.

you could allow 0 for both caches, and if this would hurt performance on some systems, put a warning that disabling the cache could cause some systems to perform poorly, but then allow the user to make the setting as they wish... Similar to the virtual memory paging file, which I have had disabled for years with a quite noticeable increase in performance. It's simply not needed anymore, but if it exists, it slows things down. I have 64GB of physical RAM, why would I need any virtual memory??.... So... My hard drive's and read access is BLAZING fast, why would I need or want to cache my files?

[jamesr33423836]

I have new information on this bug.

I uninstalled Lightroom, and re-installed version 6.0

First I notice that when I just open a catalog, and even go look at a few videos, my video cache file (which is now located at C:\Users\James\AppData\Local\Adobe\Lightroom\Caches\Video\Media Cache Files) is very small, just a few hundred MB, I even played a video, and still, it's nice and small. So, to be thorough, I thought I would try a few different things, such as selecting my entire catalog and scrolling through the entire thing, this did not affect teh video cache, so next I tried going to import... just for the fun of it, I selected my E drive, which is full of nearly 4TB of movies and TV shows, I let it show about a screen worth of video files, then closed the import dialog, about 10 seconds worth, and during that 10 seconds, my video cache has jumped to 11.5 GB, and the cached files are ALL the movies that were in that folder, and on top of that, the cache files are HUGE, for example, I have a file named "some tvshow title.avi 48000.cfa" and it's size is 504MB, however the original "E:\TV\some folder\some tvshow title.avi" from which this was cached, is only 235MB.. that's some effecient cache you have there....

So, the next, most obvious thing to test, is to purge the video cache. So I went to preferences and pushed the purge cache button in the file handling section.. it did reduce the size of the cache, but it didn't clear it. So I closed lightroom, then re-opened it. Next I went to purge the video cache once again.. It put up a message saying The Video Cache is being Purged, this message will be dismissed when the purge is finished. After the message remained for 15 minutes, I wondered what was up.. so I went to check on my video cache to see it's progress, and WOW my Video cache is NOW 32GB!!!!! I went to check on some files and I have a LOT of files over 1GB - 1.8GB in size.. EACH, and the corresponding files which were cached were only 500-700 each. In the time it took me to look over these files, my video cached has jumped to 46GB!!!!!!!!!!!!!!!!!!! So i hit cancel. the box now says cancelling... it has said canelling for the last 20 minutes, and while the cache is still at 46GB, I don't understand what's taking it so long to cancel.

please note, this is a real bug, This test is repeatable, I've tried it on my laptop and my desktop.

So this is a MAJOR BUG with version 6.0, With 6.0, I needed to select a multimedia drive and breifely let it display some of the videos.. that was the trigger, but once this folder was cached, even though I NEVER IMPORTED ANYTHING FROM IT, the cache has run away with it, and it appears the purge function caused the runaway situation.

This has appearantly been made worse by the new import dialog, because it filled my drive until there was no more space with this rediculous cache, and furthermore, I did not ever select any of my hard drives with movies and tv shows to import from.... it did it by itself.

I think I have enough information from version 6.0, I will upgrade it to version 6.1.1 and see what the results are.

[jamesr33423836]

I don't know what in the world it is doing, but after well over an hour, it still says canceling to the purge, and lightroom is responding to the system, but I can't close it and i can't get rid of the closing box. I'm going to have to terminate it now.. I need to gets some work done.

[simonsaith]

If you go to Lightroom Preferences>File Handling>Video Cache Settings, what is the maximum cache settings there?

[jamesr33423836]

Even MORE information about this.

Ok, I terminated lightroom.. rebooted my system. restarted lightroom. went to purge video files, it took about 3 seconds and was done.. checked the cache, it was empty, 0 bytes, 0 files.

I'm still on 6.0

Lets try to figure out what is happening here...

just leave lightroom open 10 minutes... no change to video cache...

click Import, select a drive with videos on it.. it starts the insane caching... cancel the import....

do nothing.... just leave light room open, in library module Video cache continues to grow...

close lightroom... do nothing with computer

lightroom video cache continues to grow!!! LIGHTROOM IS CLOSED!!!!

do nothing some more

lightroom video cache IS STILL GROWING!!

look at task manager... lightroom is not there

sort by processor % used

there is something called dynamiclinkedmediaserver.exe using 13% of my processor... but lightroom is closed, and i never actually imported anything

i terminate the dynamiclinkedmediaserver.exe and finally my video cache stops growing.. I have very fast drives and a very fast computer, so it managed to get to 48GB in the short time I figured this out.

YOU HAVE NO BUSINESS CACHING ANYTHING DURING IMPORT, ESPECIALLY FILES THAT ARE NEVER IMPORTED!

[jamesr33423836]

it is set at 1GB

When I re-installed 6.0, it went back to 3GB, I changed it to 1 GB

I wish i could set it to 0. I don't need or want video cache... only to display thumbnails

[simonsaith]

Thanks for the info. The dynamiclinkedmediaserver.exe is a child process that Lr spawns to handle all the video decoding and encoding. I've forwarded the info to one of our engineers to investigate to see if this is an expected behavior or not.

[jamesr33423836]

I sure would be grateful if I could help Adobe further by being an official beta tester. I am finding bugs and their cause, you need beta testers like me who run Lighroom on a real system, not a stripped down development system that has nothing on it but your compilers and 100 photos just to see how it works.

[jamesr33423836]

Thank you for your attention to this matter.

If I didn't terminate it, It could continue to cache files until my drive was full... i can't imagine that this is expected behavior.

also lightroom should close it's child processes upon exit. To leave anything running and not clean up after itself is bad programming. I suspect dynamiclinkedmediaserver.exe was launched in error, and therefore not shut down, because it was not supposed to be running. Caching anything from just looking at an import screen without ever actually importing anything makes no sense, so I would conclude it was an accident.

It's also doing it WRONG, no debate about it, Creating a cache file that is 3-4 times the size of the original file is just a waste, if the cached file is that much larger than the original, it will load slower than the original, thus defeating the purpose of a cache. I've had 1.8GB cache files of original files that were only 500MB this also indicates that it could be run inadvertently, and possibly missing some command line parameters which would prevent this strange behavior. The real video cache files for real videos that are intended to be cached are not this way, The cache files are much much smaller than the originals.

[simonsaith]

To help our QE reproduce the issue, could you copy and paste your system info from Lightroom's system info dialog? It seems wrong that the dynamiclinkedmediaserver.exe is ignoring the cache size limit set by the Lightroom. The dynamiclinkedmediaserver.exe process is a shared server process with some of Adobe's other product (PS, video etc). I might be wrong on this. Caching is typically a technique to tradeoff faster runtime performance with a larger storage. 

[jamesr33423836]

Certainly!! here is the paste from system info:

Lightroom version: 6.0 [1014445]
License: Perpetual
Operating system: Windows 7 Business Edition
Version: 6.1 [7601]
Application architecture: x64
System architecture: x64
Logical processor count: 8
Processor speed: 4.7 GHz
Built-in memory: 32665.1 MB
Real memory available to Lightroom: 32665.1 MB
Real memory used by Lightroom: 450.1 MB (1.3%)
Virtual memory used by Lightroom: 431.2 MB
Memory cache size: 157.9 MB
Maximum thread count used by Camera Raw: 8
Camera Raw SIMD optimization: SSE2,AVX
System DPI setting: 96 DPI
Desktop composition enabled: Yes
Displays: 1) 1920x1200, 2) 1920x1080, 3) 1200x1920, 4) 1920x1200
Input types: Multitouch: No, Integrated touch: No, Integrated pen: No, External touch: No, External pen: No, Keyboard: No

Graphics Processor Info:
GeForce GTX 760/PCIe/SSE2

Check OpenGL support: Passed
Vendor: NVIDIA Corporation
Version: 3.3.0 NVIDIA 353.62
Renderer: GeForce GTX 760/PCIe/SSE2
LanguageVersion: 3.30 NVIDIA via Cg compiler

Application folder: C:\Program Files\Adobe\Adobe Lightroom
Library Path: L:\Lightroom Data\Photos\Photos.lrcat
Settings Folder: C:\Users\James\AppData\Roaming\Adobe\Lightroom

Installed Plugins:
1) Behance
2) Canon Tether Plugin
3) Facebook
4) Flickr
5) Leica Tether Plugin
6) Nikon Tether Plugin

Config.lua flags: None

Adapter #1: Vendor : 10de
Device : 1187
Subsystem : 847a1043
Revision : a1
Video Memory : 1990
AudioDeviceIOBlockSize: 1024
AudioDeviceName: CP9287-4 (NVIDIA High Definition Audio)
AudioDeviceNumberOfChannels: 2
AudioDeviceSampleRate: 44100
Build: Uninitialized
Direct2DEnabled: false
GPUDevice: not available
OGLEnabled: true

[jamesr33423836]

I agree, that is the purpose of Caching, however, this has gone wrong, first of all, Caching is only of benefit if the file will be accessed more than once, if the file is only accessed exactly once, then the time it takes to open the file and build a cache file is slower than just opening the file, since I was just looking at an import folder to see what was in it, and i did not import anything, there is no reason to assume that any file would need to be accessed a second time. furthermore, for the purpose of importing a video, all I need and in fact all I CAN use to make my decision is the thumbnail image, which does not need to be cached. I can't play the videos from the import dialog if wanted to. also Cache files that are 3-4 times larger than the original cannot ever be faster than the smaller original... and lastly If I rarely ever play any videos in lightroom, then caching is useless and a waste of time and resources, because while it's busy caching a video I will not look at again, it could have been just playing the video. Since lightroom can't edit videos, caching them is useless. on top of that, my system is fast enough that I don't ever need to cache any videos... If I just play any of the movies that lightroom cached from the original files with VLC media player, they load and start playing as close to instantly as I can determine, Even a 3 hour movie plays instantly... so i NEVER want to cache any videos ever. It makes no sense at all. PLEASE Let me disable it!

[J453]

Hi James. This issue should be fixed in our next update.

[jamesr33423836]

That is awesome! I look forward to it. Now that I understand better what causes this, I will be able to avoid any problems until then.

Thank you very much for your attention to this

James

[simonsaith]

Can you confirm again that the bug does not exists in Lr 6.1.1?

[jamesr33423836]

The bug exists in 6.0, If you just click on a drive that has video on it, it will launch dynamiclinkedmediaserver.exe and cache every single video, until the drive is full, even if you close lightroom.

The Problem is worse with 6.2 because it starts this caching process on it's own, even if you don't click anything.. it does this while it's looking for content in common locations, however F:\ anything is not a common location

[simonsaith]

Understood, The next Lr updates would behave the same way as Lr 6.1.1.

[jamesr33423836]

Ok, Hopefully You will be able to cure this once and for all in another future release. I know to watch out for it, And I won't have a problem because all I have to do is stay off my media server drives, however Even with 6.1.1, if someone just tries to import from a folder on their C drive and has subfolders turned on, while they are on their way to the target folder they want, they will start with C:\ and if somewhere, anywhere on their C drive they have a lot of videos, this is going to launch and fill their C drive.

In fact, I did a quick test, and if I just try to Sync folders, and say it's my gopro folder, which I legitimately want the videos to sync, If I tell it to show the sync with the import dialog, the same thing happens.. it caches ALL my gopro videos, ignores the limits and caches every single one of them.

[simonsaith]

Thank you for reporting this. A bug was logged and the developer is investigating this.

[jamesr33423836]

Great. I'm glad to be of help. Lightroom is one of my favorite programs.

If I can make a suggestion: You have a LOT of customers with a gigantic amount of space wasted in this video cache because of this, however, they do not know it because until their hard drive runs out of space, they just keep going. In fact, even after the drive runs out of space, they do not realize what caused it, so they go deleting other things to try to make some space, the only reason I found it at all is because I deleted 20GB of files and less than an hour later I was out of space again, so I knew something was wrong. My suggestion is, during the next release, before you even install it, run a procedure to wipe out the caches.. they will re-build to the correct size soon enough. A good percentage of your customers do not technical people, and they will not know or be comfortable with manually checking or purging the cache.

[J453]

I may have spoke too soon. Simon and the video eng are still investigating.

[jamesr33423836]

Information Update:

I tried my test in Lightroom 6.3: and get the same results.

Test Procedure:
Open Lightroom
Open Import
Select a location with movies and videos
Close Import
Close Lightroom

dynamiclinkedmediaserver.exe is remaining active and builds a huge gigantic cahe with the above mentioned huge files that er 3-4 times the size of the original movies

I also happened to have lightroom 5.7 installed on my old computer.. I fired it up and did the same test as above. With 5.7, it does launch dynamiclinkedmediaserver.exe, but a few seconds after I close the import dialog, it closes as well.. Also PLEASE NOTE: with lightroom 5.7, while it does cache all the movies in a folder that I was just looking at in the import dialog.. and that I never actually imported.. something I don't believe should have been done... at least the files are all very small, 3K my total Cache folder was only 1 few MB after this test, not 900GB, I believe the behavior on 5.7 is what is intended, however I think that it should be modified to only cache files that are actually imported, I suppose it builds the cache to improve performance of the import dialog, but the theory is wrong, because it actually takes longer to read the files and build the cache instead of just reading the files... the cache would only be helpful if the files were viewed a second time, not the first time, but if the files were imported, they don't need to be looked at again in the import dialog, and if they were not imported, they probably never will be... so caching after import would be more useful. At least 5.7 does not gobble up all remaining drive space, so the real bug was introduced with 6.0

I hope this information is helpful