Lightroom Classic CC 8.0 large sized backups and LRCAT file

Dec 04, 2018

Copy link to clipboard

Copied

Hi Everyone!

Experiencing two weird issues with respect to LRCAT file size and LRCAT backup size.

1st problem is Catalog size is more or less independent of number of photos.  So 152 shot catalog and 25 shot catalog, are both 200 MBs unlike how it previously was. Here, I'm talking about the file size of the LRCAT file itself and nothing else. Just the catalog. As sometimes I've to export a 3 shot LRCAT file for the client when they already have the original RAW files and all I need to send them is the LRCAT file and nothing else.

2nd, compression ratio for LRCAT file (when compressed and saved by Lightroom) was around 80%. This was more or less similar to what you would achieve if you manually compressed an LRCAT file. While now theres barely 5-8% compression. Basically, 200 MB LRCAT would now be backed up as 180 MB compressed zip file. Earlier that was 10 MB or so. So all LRCAT backup folders are growing at a ginormous rate.

I hope that this is something that adobe can resolve and not something permanent with LR 8.0. As its making image transfer really difficult for people where good internet connection is still a luxury.

Is someone else experiencing this? Can someone from Adobe advise on when (or if) this will be fixed?

Thanks in advance for any help

In order to clean catalogs presenting the issue, I've found the following workaround.

1. Make sure the folder

Application Support/Adobe/CameraRaw/Tables

is empty.

2. Within an SQLite editor, delete all the rows in the table

Adobe_libraryImageDevelop3DLUTColorTable

for each plagued catalog (delete all the rows of records, not the table itself).

3.

As size of .lrcat files is not reduced yet, open in LR each catalog, optimize it then close it.

In all my non-infected catalogs the table Adobe_libraryImageDevelop3DLUTColorTable is empty, therefore emptying this table outside LR should not produce any damage to the catalog itself.

Views

1.4K

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more

Lightroom Classic CC 8.0 large sized backups and LRCAT file

Dec 04, 2018

Copy link to clipboard

Copied

Hi Everyone!

Experiencing two weird issues with respect to LRCAT file size and LRCAT backup size.

1st problem is Catalog size is more or less independent of number of photos.  So 152 shot catalog and 25 shot catalog, are both 200 MBs unlike how it previously was. Here, I'm talking about the file size of the LRCAT file itself and nothing else. Just the catalog. As sometimes I've to export a 3 shot LRCAT file for the client when they already have the original RAW files and all I need to send them is the LRCAT file and nothing else.

2nd, compression ratio for LRCAT file (when compressed and saved by Lightroom) was around 80%. This was more or less similar to what you would achieve if you manually compressed an LRCAT file. While now theres barely 5-8% compression. Basically, 200 MB LRCAT would now be backed up as 180 MB compressed zip file. Earlier that was 10 MB or so. So all LRCAT backup folders are growing at a ginormous rate.

I hope that this is something that adobe can resolve and not something permanent with LR 8.0. As its making image transfer really difficult for people where good internet connection is still a luxury.

Is someone else experiencing this? Can someone from Adobe advise on when (or if) this will be fixed?

Thanks in advance for any help

In order to clean catalogs presenting the issue, I've found the following workaround.

1. Make sure the folder

Application Support/Adobe/CameraRaw/Tables

is empty.

2. Within an SQLite editor, delete all the rows in the table

Adobe_libraryImageDevelop3DLUTColorTable

for each plagued catalog (delete all the rows of records, not the table itself).

3.

As size of .lrcat files is not reduced yet, open in LR each catalog, optimize it then close it.

In all my non-infected catalogs the table Adobe_libraryImageDevelop3DLUTColorTable is empty, therefore emptying this table outside LR should not produce any damage to the catalog itself.

Views

1.4K

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Dec 04, 2018 3
Adobe Community Professional ,
Dec 04, 2018

Copy link to clipboard

Copied

The Lightroom catalog is a database containing metadata and edits. That means that the amount of metadata and the number of edits has a large impact on the catalog file size. So does the edit history. And that means that a catalog with fewer images (but lots of edits) can be bigger than a catalog with more images with fewer edits.

Adobe changed the catalog structure in Lightroom Classic. As a result, the catalog file is quite a bit smaller than it was in Lightroom 6 (mine shrunk by about 40% when I upgraded). It also means that it doesn’t compress that well anymore, because it is already compressed.

-- Johan W. Elzenga

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Reply
Loading...
Dec 04, 2018 1
Dec 04, 2018

Copy link to clipboard

Copied

Thanks for taking out the time to write all of that.. Being an Adobe certified expert in Photoshop, I have some idea about all of that.

Your answer doesn't answer my concern as if there's a particular catalog in question, which exports a sub catalog, containing only 5% of the images, its still similar size. No edits of images changed in between  Also, I'm only upgrading from the previous version. Not from the archaic 6.0. I have been on Classic CC for as long as it exists.

I have catalog backups (from recent version of LR before 8.0) from a few weeks ago which have 81% compression. These ones have 5-10% compression. Everything is comparable between these catalogs.

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Reply
Loading...
Dec 04, 2018 0
LEGEND ,
Dec 04, 2018

Copy link to clipboard

Copied

The catalog is a database, and as such, it probably has a minimum size, even if it had zero images. This may be what you are seeing.

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Reply
Loading...
Dec 04, 2018 1
Dec 04, 2018

Copy link to clipboard

Copied

I am also experiencing the same problem. All my catalogs are in between 200-250MB and it never happened like this before.

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Reply
Loading...
Dec 04, 2018 0
Dec 07, 2018

Copy link to clipboard

Copied

dj_paige  wrote

The catalog is a database, and as such, it probably has a minimum size, even if it had zero images. This may be what you are seeing.


I understand that is true, but here's the thing. From an unaffected computer, a newly created LRCAT with Zero images has a size of around 2 MBs. While a similar catalog from the affected computer (or lightroom) has the LRCAT file with 180 MB size. You may want to check the file I've uploaded in a message above, in case its difficult for you to believe that. Its quite unbelievable for me as well.

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Reply
Loading...
Dec 07, 2018 0
Adobe Employee ,
Dec 05, 2018

Copy link to clipboard

Copied

Pushp, could you please explain your workflow a little more, couldnt seem to understand it.

I am still getting separate values, for separate amount of pictures in the LR Catalog.

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Reply
Loading...
Dec 05, 2018 0
Dec 07, 2018

Copy link to clipboard

Copied

Hi Nitin!

So I'm exporting a one image catalog, and getting 180 MB lrcat file.  Not just that, even if I export just 1 file in the new catalog, its taking 10 times as long to export it as compared to the earlier version.


The only new thing I'm noticing is this new setting which I haven't seen earlier.


Screenshot (70).png

Going even further with my testing, I went to the menu and created a new catalog. Didn't import anything into it, and closed the catalog. Instantly upon closing it became a 180 MB file again. That catalog has just been created and it has literally nothing in it. Not a single imported file. Its still 180 MB file. Something is being written to the LRCAT file which is totally bloating its size.

Going totally crazy.

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Reply
Loading...
Dec 07, 2018 1
LEGEND ,
Dec 07, 2018

Copy link to clipboard

Copied

I just created a new catalog and it was 1.5 MBs with no images imported.

Second Test Cat No images.PNG

Then I imported 18 images and it got slightly larger but not by much. 1.93 MBs

Second Test Cat 18 images.PNG

So I have no idea why your new Test catalog was 180 MBs in size. Are you sure it wasn't 1.80​ MBs?

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Reply
Loading...
Dec 07, 2018 0
LEGEND ,
Dec 07, 2018

Copy link to clipboard

Copied

What is the 3D LUT that is being copied to the cat export?

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Reply
Loading...
Dec 07, 2018 1
Adobe Community Professional ,
Dec 07, 2018

Copy link to clipboard

Copied

I just checked on my MacBook Pro. I created a new catalog for an online workshop I'm writing, and imported 87 photos into it. The catalog file is 3.7 MB...

-- Johan W. Elzenga

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Reply
Loading...
Dec 07, 2018 0
Dec 07, 2018

Copy link to clipboard

Copied

Thanks for trying that out on your computer. I just recently found one one of my computers is not affected as well. And I can replicate what you just did there. But now what I'm going to do is upload the empty catalog for you to see (from the affected computer) and open for yourself. You'll see its totally empty. And of course its not 1.8 MBs. lol Its 180 Mega Bytes, which is roughly 0.18 GBs.

Dropbox - new test catalog.zip

As for your other question, I don't really know what are the 3D LUTs which are seen in the screenshot but other people have reported seeing that same thing without having 180 MB catalogs. I have also done a re-install without results. Issue remains.

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Reply
Loading...
Dec 07, 2018 1
Enthusiast ,
Dec 07, 2018

Copy link to clipboard

Copied

Do you have the preference option to 'store presets with this catalog' turned on?

It was probably turned off earlier or you just installed new presets that increased the size of the stored presets within the catalog.

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Reply
Loading...
Dec 07, 2018 0
Adobe Community Professional ,
Dec 07, 2018

Copy link to clipboard

Copied

bhousto90  wrote

Do you have the preference option to 'store presets with this catalog' turned on?

It was probably turned off earlier or you just installed new presets that increased the size of the stored presets within the catalog.

Presets can be stored with the catalog. They are not stored inside the catalog file.

-- Johan W. Elzenga

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Reply
Loading...
Dec 07, 2018 0
Dec 07, 2018

Copy link to clipboard

Copied

bhousto90  wrote

Do you have the preference option to 'store presets with this catalog' turned on?

It was probably turned off earlier or you just installed new presets that increased the size of the stored presets within the catalog.

bhousto90​ I don't have this turned on. I have checked this, and most other menu options to find anything that helps. I haven't. If you check the file I've shared above, you can see the catalog for yourself. There's nothing except the LRCAT file in it. Still its bloated. Plus, if you zip it, you won't find the usual compression. Someone from the tech team of Adobe would need to have a look at it to check what does the file contain which is causing is inordinate increase in its size.

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Reply
Loading...
Dec 07, 2018 0
Aug 19, 2019

Copy link to clipboard

Copied

This is also happening to my catalogs.  I have not found any resolution.  Did this issue resolve itself for you?

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Reply
Loading...
Aug 19, 2019 0
Adobe Employee ,
Dec 08, 2018

Copy link to clipboard

Copied

2k.Pandey  wrote

Hi Everyone!

Experiencing two weird issues with respect to LRCAT file size and LRCAT backup size.

1st problem is Catalog size is more or less independent of number of photos.  So 152 shot catalog and 25 shot catalog, are both 200 MBs unlike how it previously was. Here, I'm talking about the file size of the LRCAT file itself and nothing else. Just the catalog. As sometimes I've to export a 3 shot LRCAT file for the client when they already have the original RAW files and all I need to send them is the LRCAT file and nothing else.

2nd, compression ratio for LRCAT file (when compressed and saved by Lightroom) was around 80%. This was more or less similar to what you would achieve if you manually compressed an LRCAT file. While now theres barely 5-8% compression. Basically, 200 MB LRCAT would now be backed up as 180 MB compressed zip file. Earlier that was 10 MB or so. So all LRCAT backup folders are growing at a ginormous rate.

I hope that this is something that adobe can resolve and not something permanent with LR 8.0. As its making image transfer really difficult for people where good internet connection is still a luxury.

Is someone else experiencing this? Can someone from Adobe advise on when (or if) this will be fixed?

Thanks in advance for any help

Thanks 2k.Pandey for getting this over a call with me and experimenting with the issue.

The issue seems like a bug, but it spreads to every LR which opens the infected file. I was able to record the video while the issue was being created in a .SWF file. attached!

https://www.dropbox.com/s/7j2a379b38ymsnn/LR-2018-BUG.swf?dl=0

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Reply
Loading...
Dec 08, 2018 3
LEGEND ,
Dec 08, 2018

Copy link to clipboard

Copied

Wish this was posted earlier. I downloaded and open that catalog and then my system made 180MBs new catalogs.

Luckily I have a drive image created on 11/20 that I used to restore my system.

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Reply
Loading...
Dec 08, 2018 1
Adobe Employee ,
Dec 10, 2018

Copy link to clipboard

Copied

Checked the catalog, seeing that the 3D color LUT took up the majority of the size of the new catalog. The catalog stores a total of 1021 entries of custom 3D LUT (not from Adobe), presumably coming from some custom camera profiles/presets.

Lightroom caches unique 3D LUT both in the catalog and on the disk to facilitate transportation of catalogs among different machines and quick access to them at develop time. Lightroom does not keep a copy of the 3D LUT at each history step.

The catalog size increase for a newly created/exported catalog is a manifestation of this. It was introduced as part of the Lightroom Classic 7.3.

Given that a reference to a 3D LUT could be buried in one of a history step (customer can go back in the history step to reference a 3D LUT at any time). Lightroom catalog does not currently have an infrastructure to do proper reference counting of custom 3D LUT usage. So it just includes a copy of every custom 3D LUT ever used.

We have to consider this case of multiple catalog usage scenario and see if we can optimize it. Thanks for the report.

P.S. The 3D LUT is also cached locally on disk at "~/Library/Application Support/Adobe/CameraRaw/Tables" on macOS. There is a similar Window directory location.

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Reply
Loading...
Dec 10, 2018 0
Dec 10, 2018

Copy link to clipboard

Copied

Does this mean potentially rolling back to 7.2 would alleviate this issues until there is a rollout fix in the next update? Its causing me all types of time delays each day in waiting for uploads/downloads that are 20 times as big as before!

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Reply
Loading...
Dec 10, 2018 0
Dec 24, 2018

Copy link to clipboard

Copied

Thank you for your reply.

simonsaith  wrote

Checked the catalog, seeing that the 3D color LUT took up the majority of the size of the new catalog. The catalog stores a total of 1021 entries of custom 3D LUT (not from Adobe), presumably coming from some custom camera profiles/presets.

Lightroom caches unique 3D LUT both in the catalog and on the disk to facilitate transportation of catalogs among different machines and quick access to them at develop time. Lightroom does not keep a copy of the 3D LUT at each history step.

The catalog size increase for a newly created/exported catalog is a manifestation of this. It was introduced as part of the Lightroom Classic 7.3.

Given that a reference to a 3D LUT could be buried in one of a history step (customer can go back in the history step to reference a 3D LUT at any time). Lightroom catalog does not currently have an infrastructure to do proper reference counting of custom 3D LUT usage. So it just includes a copy of every custom 3D LUT ever used.

We have to consider this case of multiple catalog usage scenario and see if we can optimize it. Thanks for the report.

P.S. The 3D LUT is also cached locally on disk at "~/Library/Application Support/Adobe/CameraRaw/Tables" on macOS. There is a similar Window directory location.

I checked the directory you mentioned (TABLES) and I found 1021 files I think which totalled up to 172 MBs which is also close to the resulting catalog size. So its definitely relating to that. Also, whats interesting is, even after deletion of these files, they're automatically created again in a second after any new catalog is created or opened on an already infected machine. I checked a non-infected machine and this directory for them, is totally empty. Deleting camera profiles hasn't worked either.

I am trying various permutations and combinations to alleviate the issue as two of my main machines are infected and work on them isn't possible because of this. I usually export a lot of sub-catalogs to send to my editors, and they're all super heavy even though they have very less images in them.

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Reply
Loading...
Dec 24, 2018 0
Feb 12, 2019

Copy link to clipboard

Copied

In order to clean catalogs presenting the issue, I've found the following workaround.

1. Make sure the folder

Application Support/Adobe/CameraRaw/Tables

is empty.

2. Within an SQLite editor, delete all the rows in the table

Adobe_libraryImageDevelop3DLUTColorTable

for each plagued catalog (delete all the rows of records, not the table itself).

3.

As size of .lrcat files is not reduced yet, open in LR each catalog, optimize it then close it.

In all my non-infected catalogs the table Adobe_libraryImageDevelop3DLUTColorTable is empty, therefore emptying this table outside LR should not produce any damage to the catalog itself.

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Reply
Loading...
Feb 12, 2019 1
Oct 27, 2020

Copy link to clipboard

Copied

@path72 Thank you for your response. Your method does fix the problem and I would have liked it more if Adobe addressed this issue coz its plauging catalogs even now (9.4). There's a client of mine who has so many infected catalogs, that they've stopped caring. So much bandwidth and storage space is being wasted. 

I'll communicate this solution to them and hopefully it can be resolved.

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Reply
Loading...
Oct 27, 2020 0
Adobe Community Professional ,
Dec 08, 2018

Copy link to clipboard

Copied

Opened the catalog file linked above in a text editor and it appears that the 180MB is a gigantic color profile set as the default soft proof profile. It appears Lightroom stores this profile in the catalog file! Try changing the default proof profile to something simply like sRGB and then exporting a catalog.

Also if it hasn't been mentioned before submit a bug report to https://feedback.photoshop.com

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Reply
Loading...
Dec 08, 2018 1
Dec 09, 2018

Copy link to clipboard

Copied

Hey Mate,

I checked my proof profile and it was still set to Adobe 1998.. same result changing it to sRGB. What were you looking for inside the catalog file with the text editor? Wondering if I can try change it this way, been trying everything and cant stop the file size from exploding on closing any new catalog.

Thanks!

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Reply
Loading...
Dec 09, 2018 0
Adobe Community Professional ,
Dec 09, 2018

Copy link to clipboard

Copied

I just opened the lrcat file in a text editor and right before a giant

block of data that goes until the end of the file there is a header that

seems to indicate a proofing profile is starting. I don’t think you can

easily edit this file but you might be able to manipulate it using MySQL

from the command line.

On Sun, Dec 9, 2018 at 2:22 PM jackc80362305 <forums_noreply@adobe.com>

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Reply
Loading...
Dec 09, 2018 2
Dec 09, 2018

Copy link to clipboard

Copied

Thanks mate, I work on so many catalogs that just wont be able to work for me.. sigh.. this is such a big bug to have pop up and really removes that easy of file transfer for many that use Lightroom in this way.

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Reply
Loading...
Dec 09, 2018 0
Dec 09, 2018

Copy link to clipboard

Copied

I am also experiencing this over a couple of machines now. I have tried rolling back to older versions of LR CC but still having the same issue

Is there a fix for this or does one know where to alter the catalog file to change that soft proofiing profile (if that is the cause of the large file sizes). I deal with a lot of LR catalogs each day and this is crippling my ability to upload/download and share.

Thanks

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Reply
Loading...
Dec 09, 2018 0