P: Limit of 52 reorderings in custom-ordered collections and folders

[adrian_cleave]

[I've verified this bug still exists in LR 10.0. See this post. -- John Ellis] 

After a while of custom sorting the order of photos and stacks within a collection (using Grid View), Lightroom starts to unpredictably refuse to sort photos. Some will get positioned where I drop them, others won't move at all and yet others will get positioned somewhere close by (eg. 4 or 5 photos before or after).

After digging into the catalog file I've come across what I think is the problem, but don't know how to fix it. In the attached file you'll see a screenshot of the database table for the collection, you'll see I've hilited the images that are part of the same collection, but their positionId is identical (which should never happen I'm assuming), probably due to the field size reaching it's maximum length. This is what I believe is causing the problem. Tested this on both Lightroom 5.7 and CC 2015.8.

This is a major bug and effectively stops user sorting from being functional, as well as now having potentially lost weeks of work. Any suggestions?



Thanks,

Adrian

[PaulDirtBag]

When trying to rearrange photos in a collection the ability to drag photos just stops. The collection size is about 500 photos.  

Sometimes I get a message "Reorder photos by dragging from within the grid view, not by dragging from another source". I am dragging from within the grid view, again with that message dragging stops.

In both cases I have to exit and restart LR to get it working again.

[Jim 185]

That is a problem that has been around for a good while. I put it down to LR just getting tired and needed a reboot

[COBill]

Lightroom has a whole raft of bugs that require restarting. Just something we live with. I don't expect it to work continuously for more than about an hour before needing a restart. Version 9.1 made this worse then 9.2 worse again. Hopefully Adobe is working on this for 9.3. How about it Adobe: a bug fix release with NO new features? If any developer wants to squeeze in just this one little feature, rap its knuckles with a ruler. 
  

[Bob Somrak]

Its probably related to this long term bug.  Restarting doesn't help if it is.

lightroom-major-bug-with-custom-sort-order

[Sunil_Bhaskaran]

There are certain issues related to custom sort order that we are investigating.
On a side note, could you please check if all the images in the Grid view are from the same folder?

Thanks,
Sunil

[PaulDirtBag]

They would be in various sub folders under a parent.  Were imported to LR from the parent.

[dbharsh]

This happens several times while I'm re-ordering images and videos in a collection (not a folder) of 400 or so images, while in the grid view.  One moment I'm happily dragging and dropping, even with multiple images selected, the next I receive the error stating "Reorder photos from grid view and not from another source" and click-and-drag to reorder never works until I restart.  I'm getting pissed off, which is par for the course for Lightroom.  The only thing I can see which may "cause" it to happen more often is when I need to take one image from near the bottom of the grid and go all the way to near the top. Still happening on 9.3 Release 202005281810-476e492c (and anybody else notice that you can't copy/paste the damn build number?!? Argh!)

[dbharsh]

I had this happening so often today that I contacted official Adobe support - after investigating, I have a known engineering issue number as per support: "I checked ,this is a known issue and reported to engineering team.This is the bug number 4203685." - so perhaps if anyone else stumbles on this thread they could/should also log in another instance referencing that.

[johnrellis]

That internal tracking number (4203685) is already assigned to this topic. The topic is marked "In Progress", meaning Adobe developers are actively working on the issue.

So if someone else is experiencing similar symptoms, it's more efficient for them to post here with the details than try to deal with Adobe support (often an exercise in frustration).  This forum is the official place to report bugs with LR, and developers actively participate here (e.g. see here).

[alexpardi]

I experience a variant of this problem: if I am in a folder with custom order, after I edit the image in Photoshop the psd+raw file stack goes to the end of the filmstrip. The remaining photos maintain the previous order. This issue too depends on the number of images in the folder and the number of changes in the folder.

[PhilBurton]

Rikk,

Any response yet from the product manager?

[Sunil_Bhaskaran]

Thanks for lettings us know.
We will investigate.
Could you please let us know the number of images in the folder?

Thanks,
Sunil

[alexpardi]

About 370, roughly half of which psd files.

[PhilBurton]

This is so crazy.  And it seems like the solution has already been described.  All the engineers need to do is code up the solution and then test it.  I'll bet this work represents less an 20% of a four-week Agile Scrum sprint.

[StormyTheCat]

Sunil, how is that investigation coming?

[Rick Spaulding -]

Greetings,

Updates for the Adobe Photography Products were released on 10.20.2020 that include fixes for these issues. Please install the most recent update and confirm that your issue is now fixed. Please let us know if you encounter any issues.

Thank you for your patience.

[johnrellis]

Unfortunately, LR 10.0 still has bugs with custom collection ordering, so I re-marked this bug as "in progress". To reproduce:

1. Create a new catalog and import n photos, where n is at least 6118.

2. Create a collection containing all n photos.

3. Let's call the first three photos in the collection A, B, and C.

4. Select all the photos except the first two (A and B) and drag the selection between A and B, creating a custom order.

5. Observe that C is now the second photo and B is at the end (correct).

6. Select all the photos except the first two (A and C) and drag the selection between A and C.

7. Observe that another photo is at the very end, rather than B and C (incorrect).

This scenario isn't as unlikely as it seems. From the reports on the forums over the years, users occasionally do have large collections with more than 6K photos.  And because dragging photos is painfully slow in grid view and you can't have two grid views open, an idiom for moving an early photo last is by select all the photos after the early photo and dragging them in front of it.

Regardless, this test shows there are still bugs lurking, which may show up in more frequent situations.

-------------------------

Technical Details

Poking around in the catalog database hints at the locus of the bug.

The field AgLibraryCollectionImage.positionInCollection stores the position of an image within a collection. Prior to LR 10, that field stored a 64-bit floating number, but in LR 10 it's a varying-length string containing the 63 characters "-", 0-9, A-Z, a-z.  The order of the images within a collection is given by the lexicographic sort of the field.  

When the ordering is working correctly, all the images in a collection should have a unique value for .positionInCollection. But after step 6 of the bug recipe above, two of the 6118 images have "nil" as their value.

Further, if you keep repeating step 6, more and more images get "nil" positions. And most suspiciously, the maximum length of the .positionInCollection values is 1023.  I've tried larger values of n and observe the same behavior, with the maximum length stuck at 1023.

Sqlite doesn't impose any practical maximum length on string-valued fields, so the custom-order algorithm either has 1023 built-in somewhere or there's a string buffer limited to size 1023.

Using a varying-length string to encode position is clever.  Suppose two consecutive images are assigned position "AE" and "AF". A third image can be inserted between the two by using position "AEV".  ("V" is the 32nd letter in the 63-letter alphabet.)  And if the last image in the collection has position "zz ... z" ("z" is the last letter in the 63-letter alphabet), an image can always be inserted after it by assigning it "zz ..zV".

The advantage of this encoding is that in most circumstances, an insertion doesn't require renumbering the positions of the preceeding and following images.  An extreme case is when you try to insert before a sequence of images with positions "-", "--", "---", "----", etc. With this alphabet, there are no characters that come before "-", so inserting an image in the first position requires renumbering all the subsequent images.

I haven't analyzed the LR implementation thoroughly, but it seems that it sometimes renumbers positions when it doesn't need to.

Finally, it's not at all clear that this new implementation is more efficient than using the old one, which used 64-bit numbers (correctly implemented).  The old one used just 8-byte floats, while the new one uses varying-length strings that can get arbitrarily long. Even in the common case with few insertions, the minimum length used by the implementation is 4 characters, plus the Lua overhead of perhaps another 8 bytes for the string object header (when the strings are read from the database into memory).  Comparing and sorting floats is perhaps 1 to 2 orders of magnitude faster than comparing varying-length strings.

Using floats may require somewhat more renumbering of positions than strings, but renumbering with strings is much more expensive when it occurs.  

[johnrellis]

A number of people are reporting errors when they attempt to add images to collections, indicating related issues with the new representation of custom order, with "nil" getting stored in positionInCollection:

https://feedback.photoshop.com/conversations/lightroom-classic/lightroom-classic-when-trying-to-add-...

[ericw79554200]

Yes, it does still exist in 10.00! I have just stalled after re-ordering about 150 images in a collection - a total probably of around 100 slide-moves - some as multiples. Just like previous versions!

[johnrellis]

The most recent bug recipe now works in LR 10.1. Let's hope that LR 10.1 also fixes the errors getting thrown by collections using custom sort order.

[dbharsh]

I also can attest that this still happens in LrClassic 10.3 - however, it was *much* better behaved than before.  Over the last two weeks I worked on a collection of 200+ images for my recently passed father-in-law's memorial, and the final task was to get them all into presentation order - so quite a few drag-and-drops of single images, and some small groups of images (3 to 10 at a time).  Only once in that time did I see the dreaded "Reorder photos from grid view and not from another source" error pop up preventing me from re-ordering.  A year ago I was seeing this excruciatingly often, but hadn't read this thread in detail.  John Ellis' analysis is awesome and helps me understand some possible workarounds.  Thanks.

[dbharsh]

I am not certain why I didn't follow the link or notice that the bug was tagged in here already, but that was an awesome analysis, John.  Definitely good advice. Thanks.

[Oliver Edwards]

Bug frequently occuring in 11.4.1 on windows 10 x64. Requires a restart. Has there been any progress on 4203685?

[johnrellis]

This thread is referring to an old, severe implementation flaw that was finally fixed in LR 10.1.  

It sounds like you're experiencing an issue with a different cause, since restarting temporarily fixes it. Please start a new thread and include a lot more detail about what you're observing.

[Jrr]

The OP clearly stated that restarting temporarily helps. I'm still having this problem in 13.0.1