P: Reset your Preferences: Moving from the magic spell to a more serious approach

[Samoreen]

Hi,

Under many circumstances, whenever your copy of LR crashes, freezes, doesn't start,... the most frequent answer you'll get from the support is "Reset your preferences". And in many cases, that works. But 1) you have to re-configure LR again and 2) this will not fix the bug generating the problem so, it will happen again . Should we be happy with this ? No.

If resetting the Preferences file(s) solves a problem, that's only because

- The piece of LR code reading or writing from or to this file is buggy
OR
-  A particular combination of parameters generates a problem.

Do not accept the idea that the Preferences file was corrupted. If it's corrupted, it's by LR itself. Spontaneous file corruption does not occur that often.

Since the developers at Adobe don't seem ready to investigate this case more deeply, I guess it's time to help ourselves and to collect information that could help them debug this issue. There's a few things that are easy to do :

1. Make a backup copy of your Preferences file(s) while LR is behaving correctly. This could also help you restoring a standard behavior instead of resetting the Preferences.

2. When a problem occurs, make a copy of the current Preferences file(s).

3. If resetting the Preferences solves the problem, then make a comparison of the files created in #1 and #2. You can also compare with the new Preferences file(s).

4. Make a ZIP archive of these files possibly including the comparison result and send them to the support or better, post a link to the archive in this thread.

Comparison can be made manually or by using a text comparison tool. I'm personally using Beyond Compare since years (this is not an ad) but there are others (a Google search will show you a lot).

Actually, there are 2 Preferences files :

C:\Users\\AppData\Roaming\Adobe\Lightroom\Preferences\Lightroom Classic CC 7 Preferences.agprefs

C:\Users\\AppData\Roaming\Adobe\Lightroom\Preferences\Lightroom Classic CC 7 Startup Preferences.agprefs

(note that the name didn't change with version 8)

So apply the above to both files. This procedure also applies to the Mac version although I don't know the file path.

Hoping that this will help solve this very old and annoying issue.

[FeedbackCommunityMember]

I agree with your steps to help reduce the amount of time you spend setting all new preferences. I'm on a Mac and haven't actually had corrupt preferences apart from installing a new version — maybe installing a new OS or significant update is another occasion when you're likely to get "corrupt" preferences.

BTW, it's user Library> Preferences> version Settings file on a Mac. You'll find a whole bunch of files and folders in the Settings file. If you hold down the 3 primary modifiers when launching, it asks if you want to delete the entire thing. That's the usual way of "trashing Prefs."

My take on it when installing a new version is there may be conflict between the old Settings file and the new one—we could say "as designed with unintended consequences."<G> Though it's probably close to a bug since no one seems to know in advance that this new thing will affect that old thing. Or the new setting that didn't exist with the old preferences can't possibly play well with others? It seems  over the years they change what's under the hood, and or actual preferences, more often than they used to. The industry is moving faster as a whole—we're getting new technology more quickly and the OS turnover is much faster as well. There's good and bad news in all of that.

Comparing files is a good idea. It may be impossible to avoid the problems that arise from new versions conflicting with old ones, but it surely should help if they know where the difference lies and if it matters.

But also, just writing prefs over and over and over again is statistically bound to cause digital corruption. Why we're not immune to that, I don't know. Maybe there are introduced problems when something else causes us to crash out or force quit. I can't really be any harsher on Adobe than everyone else who often also require us to jump through hoops as they fix the problems an update caused, or try to help when someone's program, but not everyone's, is acting up.

I've never forgotten standing with one of our engineers in a company I worked for a very long time ago when he said "I wonder what would happen if I accidentally hit this key right next to the one I want" — and brought down a mainframe for 3 days that operated coast to coast with hundreds of thousands of customers. It has always gone with the territory. IBM, you've got a bug.<G>

[johnrellis]

On Windows, at least as of three years ago, LR was directly overwriting the preferences file rather than writing a new version to a temp file and then atomically renaming the temp file on top of the old version.  If the computer or LR halts during the overwriting, you're left with a corrupted preferences file. This bug was acknowledged by an Adobe engineer but never marked as fixed, and I've never retested:
https://feedback.photoshop.com/photoshop_family/topics/lightroom-writes-preferences-file-directly-ra...

While this is an elementary programming error that's easy to fix, I think it's unlikely to be responsible for most of the situations in which "resetting preferences" fixes a UI issue. If the file is only partially written, LR wouldn't be able to read it at all, since internally it's a large Lua expression on which the Lua loader would fail if it encountered a syntactically incorrect expression.

[johnrellis]

It's clear there's a serious design flaw in the core architecture of LR's preferences, since "resetting preferences" can fix so many different user-interface problems.  Here's my educated guess about what might be wrong, based on many years of writing LR plugins and pushing the limits of the LR SDK's capabilities:

When you examine the preferences file, it's clear that LR stores a large amount of internal application state in preferences, not just user settings, and if that state gets recorded inconsistently, then chaos in the user interface results.   

I'm guessing LR has an internal preference facility implemented as a special Lua table of key-value pairs.  Other parts of the LR app store something in preferences simply by assigning a value to a key (with an assignment statement). There'd be a background task that periodically writes preferences to disk. 

Corruption could occur if a client module attempts to store multiple key-value pairs at the same time, and the background task decides to run in the middle of storing the key-value pairs.  This assumes there is no locking mechanism that ensures the multiple values are handled atomically, or if there is a mechanism, many parts of LR don't use it correctly.

LR's internal tasking is supposed to be cooperative non-preemptive, so in theory the programmer shouldn't need to worry (as much) about handling such concurrency.  LR doesn't even expose any locking primitives in the plugin API.  (I don't know if there are any internally.)

But the reality is that so many internal APIs can preempt a task's execution that it can be very tricky for the programmer to handle concurrent tasks without race conditions.  See for example this bug report:
https://feedback.photoshop.com/photoshop_family/topics/lightroom-sdk-many-methods-yield-preventing-r...

Suppose a LR module wants to store two key-values in preferences, with code that looks like:

prefs.key1 = f1()
prefs.key2 = f2()

If f1() calls an internal API that causes task preemption (e.g. it does file i/o), the scheduler could run the preferences background task between the two statements, writing to disk preferences state that is inconsistent, containing a new value for prefs.key1 but an old value for prefs.key2.

[johnrellis]

I wrote a simple test script reliably demonstrating the race condition described in my previous post. It's available as this plugin:
https://www.dropbox.com/s/1gufmz0bwubu8o9/testprefswriting.lrdevplugin.zip?dl=0

I'll bet a modest amount that this race condition accounts for many of the "corruptions" of preferences experienced by users. The programming workaround is simple in concept but difficult in practice -- identify all occurrences where there are multiple fields that should be set consistently in preferences and assign their values at once from local variables. E.g. transform

prefs.key1 = f1()
prefs.key2 = f2()

to:

local key1, key2 = f1(), f2()
prefs.key1= key1
prefs.key2 = key2

[Samoreen]

Very interesting. I have never studied LUA and the plugin SDK enough to be sure of this but AFAIK, the only task synchronization tool available in LUA seems to be yielding. Apparently semaphores, mutexes and the like are not implemented. This is far from being enough to cover all concurrency situations.

This raises again the question about the choice of LUA for the development of LR.

[Jerry Syder]

Not sure what the results mean @john but ran it for the sake of it 

[johnrellis]

That means it detected an inconsistency caused by the race condition.  The two values should be identical.  Technical details are provided at the top of the file "testprefswriting.lua".

[Jerry Syder]

Cheers, will give it a read 

[johnrellis]

Also, you should restart your LR after running that plugin -- otherwise, it will run continually.

[johnrellis]

Instead of:

local key1, key2 = f1(), f2()
prefs.key1= key1
prefs.key2 = key2

the prefs object could provide this method:

prefs:store {
    key1 = f1(),
    key2 = f2()}

which would be trivial to implement and would provide the same safe semantics.

[johnrellis]

Building on Patrick's suggestion for saving backup copies of LR's preferences, I wrote this simple plugin:
https://www.dropbox.com/s/vd0pd48b3u9mtkt/snapshotprefs.1.1.zip?dl=0

From its _README.txt:
-----------------------------------------------------------------------------------------------------
Snapshot Prefs - Lightroom Plugin

This plugin runs in background, making snapshot copies of Lightroom's preferences. If at some point you need to reset Lightroom's preferences, you can submit these snapshots to Adobe, helping them identify the bug that corrupted the preferences. 

The plugin will make a snapshot copy at most once every 5 minutes, and it will retain at most 15 copies, intelligently pruning them to keep increasingly older versions, up to 56 days. The plugin uses negligible disk space and processor time doing this.

To see the snapshot copies in Finder / File Explorer, do File > Plug-in Extras > Snapshot Prefs > Show Snapshots.

[Samoreen]

Thanks a lot, John. Excellent idea.

[Rikk Flohr: Photography]

Actually, John your timing is really good.  This is something we are kicking around on our side right now.  (In response to this thread)

Basically, we want to be able to identify a bug (with reproducible steps) and have a copy of the Preferences before and after the reset of the Preference file.  In order for this to do engineering any good we need to know the following:

1. A complete description of the bad behavior prior to reset
2. A copy of the preference file before reset
3. A copy of the preference file after the reset
4. Verification that the preference file reset cured the issue in item 1

With those four pieces of information that may help us track down:

• what in preferences changed by viewing the differential
• what behavior it affected
• clues as to what process, plugin or other circumstance generated the preference file issue and the subsequent aberrant behavior. 

Having the before/after preference files allows us to test user-reported issues in greater depth. 

If we were to create a process for the gathering before/after and possibly reporting of bad behavior resolved by a preference file reset how would it work best for you?

[Samoreen]

Hi Rikk,

> 2. A copy of the preference file before reset

The preference file before reset might have been already corrupted. So, if I were the developer, I would ask for 3 files :

#1 : a copy of the preference file when everything is working correctly (thanks to John for providing the tool allowing to get it automatically)
#2 : a copy of the preference file before reset (the bug just occurred)
#3 : a copy of the preference file after reset, which will be different from #1 and less useful since everything will be... reset (witch exception of values specific to the user's environment).

My two cents.

[Rikk Flohr: Photography]

#1 would likely never be obtained with surety, I fear.  Until you stumble upon an area of bad behavior, it might lurk quietly in the background for days...months.  At least #2 and #3 are known quantities. 

[johnrellis]

The plugin I posted will keep up to 15 snapshot copies of the preferences, of increasing age, e.g. 

1 min, 5 mins, 10 mins, 20 mins, 40 mins. 80 mins, ..., 56 days.

So it's likely that one or more of those snapshots is prior to the corruption.  The total size of the snapshots in my LR is less than 1 MB on Windows, 10 MB on Mac.

Having preferences right before and right after the Reset would likely be helpful for diagnosing, but having a longer-term history might narrow down some kinds of problems more effectively.

[johnrellis]

Here are the preferences snapshots my plugin has created since I started using it three months ago:


In nine years of using LR, I've only had to reset preferences once, so it's unlikely I'll have to do it again. But if I do, I've got enough of a history here to make it very likely that the culprit could be identified.

[Jerry Syder]

John, this plugin has saved me twice already. Both instances, LR crashed and created a new preference file when I re-launched. Cheers for that - I'd only downloaded the plugin as a trial to see how it worked and forgot to remove it(thinking I won't be needing it) but happy that I'd forgotten.

[OldSteve2]

I was thinking of writing a post along similar to his one - but you had it right in 2018.  Nothing has changed in 2020 - almost as if Adobe wants to punish users for changing Preferences, or at least blame the victims when things go wrong.  It must be possible to give users a button that does whatever resetting Preferences does without losing choices.

[johnrellis]

See my post above, which suggests that fixing preferences is a non-trivial undertaking. (Adobe should do it, and should have done it years ago.)

[Samoreen]

Hi John,

The more I think about this, the more I'm convinced that the root of all evil in LR is LUA. I have always considered the use of LUA as a very strange idea and an unsuitable choice (understatement). The lack of serious synchronization tools and of exception handling capabilities are certainly the origin of many bugs and that's what makes spotting them almost impossible. Otherwise, how could Adobe explain why so many issues (including the one discussed here) are still not fixed after all these years ? Not to even mention the performance issues. It's hard to imagine a project manager in charge of the development of a very resource demanding application and saying "OK, let's write a good part of this application using a scripting language having limited capabilities". This is beyond me.

So, I'm no longer expecting any progress in this area until LR has been re-written using a more solid language or replaced with another product. But this decision should have been taken much earlier. As time goes by, this is less likely to happen.

[photopoeth]

@John_R_Ellis I just stumbled upon this post and checked out your plugin. Surprise, surprise... the two values differ. I've also read your comments in testprefswriting.lua. But now I'm not sure if i can do something to fix this problem. Is there any chance to fix it?

[johnrellis]

I don't think there's anything a user can do to fix the corrupted-preferences problem. Adobe posted here that they were looking at this issue two years ago, but I haven't seen any updates since then.

[photopoeth]

Alright... thanks for the feedback. So we just sit and wait for another 2 years... 😉

[johnrellis]

A few of us have observed that in the last month or two it seemed like there were a lot more recommendations to "reset preferences". Out of nerdly curiosity, I used a script to count all posts with that phrase by day, and indeed, there have have been many more such posts per day on average in the past months.

The chart below shows the average number of posts per day for the time period when each version was the newest:

For example, LR 11.0 was the newest version from 10/26/21 through 12/13/21, and in that period, there was an average of 2.7 posts per day containing "reset preferences".  Note that not all of those posts were referencing version 11.0, though most of them were.

LR 11.0, 11.3, and LR 11.3.1 were responsible for nearly twice as many reset-preference posts per day than the versions of LR 10.   Did something change in LR 11, e.g. with task scheduling, so that the race condition in the preferences implementation (described above) occurs more frequently?  Is it that LR just has more bugs, which initially don't have known causes or workarounds, so forum members suggest "reset preferences" more frequently? Or maybe there was a surge in total users of LR 11, so proportionately more users come to the forums.