P: Reset your Preferences: Moving from the magic spell to a more serious approach

Not sure what the results mean @188115 but ran it for the sake of it 

Very interesting. I have never studied LUA and the plugin SDK enough to be sure of this but AFAIK, the only task synchronization tool available in LUA seems to be yielding. Apparently semaphores, mutexes and the like are not implemented. This is far from being enough to cover all concurrency situations.

This raises again the question about the choice of LUA for the development of LR.

I wrote a simple test script reliably demonstrating the race condition described in my previous post. It's available as this plugin:
https://www.dropbox.com/s/1gufmz0bwubu8o9/testprefswriting.lrdevplugin.zip?dl=0

I'll bet a modest amount that this race condition accounts for many of the "corruptions" of preferences experienced by users. The programming workaround is simple in concept but difficult in practice -- identify all occurrences where there are multiple fields that should be set consistently in preferences and assign their values at once from local variables. E.g. transform

prefs.key1 = f1()
prefs.key2 = f2()

to:

local key1, key2 = f1(), f2()
prefs.key1= key1
prefs.key2 = key2

It's clear there's a serious design flaw in the core architecture of LR's preferences, since "resetting preferences" can fix so many different user-interface problems.  Here's my educated guess about what might be wrong, based on many years of writing LR plugins and pushing the limits of the LR SDK's capabilities:

When you examine the preferences file, it's clear that LR stores a large amount of internal application state in preferences, not just user settings, and if that state gets recorded inconsistently, then chaos in the user interface results.   

I'm guessing LR has an internal preference facility implemented as a special Lua table of key-value pairs.  Other parts of the LR app store something in preferences simply by assigning a value to a key (with an assignment statement). There'd be a background task that periodically writes preferences to disk. 

Corruption could occur if a client module attempts to store multiple key-value pairs at the same time, and the background task decides to run in the middle of storing the key-value pairs.  This assumes there is no locking mechanism that ensures the multiple values are handled atomically, or if there is a mechanism, many parts of LR don't use it correctly.

LR's internal tasking is supposed to be cooperative non-preemptive, so in theory the programmer shouldn't need to worry (as much) about handling such concurrency.  LR doesn't even expose any locking primitives in the plugin API.  (I don't know if there are any internally.)

But the reality is that so many internal APIs can preempt a task's execution that it can be very tricky for the programmer to handle concurrent tasks without race conditions.  See for example this bug report:
https://feedback.photoshop.com/photoshop_family/topics/lightroom-sdk-many-methods-yield-preventing-r...

Suppose a LR module wants to store two key-values in preferences, with code that looks like:

prefs.key1 = f1()
prefs.key2 = f2()

If f1() calls an internal API that causes task preemption (e.g. it does file i/o), the scheduler could run the preferences background task between the two statements, writing to disk preferences state that is inconsistent, containing a new value for prefs.key1 but an old value for prefs.key2.

On Windows, at least as of three years ago, LR was directly overwriting the preferences file rather than writing a new version to a temp file and then atomically renaming the temp file on top of the old version.  If the computer or LR halts during the overwriting, you're left with a corrupted preferences file. This bug was acknowledged by an Adobe engineer but never marked as fixed, and I've never retested:
https://feedback.photoshop.com/photoshop_family/topics/lightroom-writes-preferences-file-directly-ra...

While this is an elementary programming error that's easy to fix, I think it's unlikely to be responsible for most of the situations in which "resetting preferences" fixes a UI issue. If the file is only partially written, LR wouldn't be able to read it at all, since internally it's a large Lua expression on which the Lua loader would fail if it encountered a syntactically incorrect expression.

I agree with your steps to help reduce the amount of time you spend setting all new preferences. I'm on a Mac and haven't actually had corrupt preferences apart from installing a new version — maybe installing a new OS or significant update is another occasion when you're likely to get "corrupt" preferences.

BTW, it's user Library> Preferences> version Settings file on a Mac. You'll find a whole bunch of files and folders in the Settings file. If you hold down the 3 primary modifiers when launching, it asks if you want to delete the entire thing. That's the usual way of "trashing Prefs."

My take on it when installing a new version is there may be conflict between the old Settings file and the new one—we could say "as designed with unintended consequences."<G> Though it's probably close to a bug since no one seems to know in advance that this new thing will affect that old thing. Or the new setting that didn't exist with the old preferences can't possibly play well with others? It seems  over the years they change what's under the hood, and or actual preferences, more often than they used to. The industry is moving faster as a whole—we're getting new technology more quickly and the OS turnover is much faster as well. There's good and bad news in all of that.

Comparing files is a good idea. It may be impossible to avoid the problems that arise from new versions conflicting with old ones, but it surely should help if they know where the difference lies and if it matters.

But also, just writing prefs over and over and over again is statistically bound to cause digital corruption. Why we're not immune to that, I don't know. Maybe there are introduced problems when something else causes us to crash out or force quit. I can't really be any harsher on Adobe than everyone else who often also require us to jump through hoops as they fix the problems an update caused, or try to help when someone's program, but not everyone's, is acting up.

I've never forgotten standing with one of our engineers in a company I worked for a very long time ago when he said "I wonder what would happen if I accidentally hit this key right next to the one I want" — and brought down a mainframe for 3 days that operated coast to coast with hundreds of thousands of customers. It has always gone with the territory. IBM, you've got a bug.<G>