P: Limit of 52 reorderings in custom-ordered collections and folders

15 Votes
Community Beginner ,
Jan 28, 2017 Jan 28, 2017

Copy link to clipboard

Copied

[I've verified this bug still exists in LR 10.0. See this post. -- John Ellis] 

After a while of custom sorting the order of photos and stacks within a collection (using Grid View), Lightroom starts to unpredictably refuse to sort photos. Some will get positioned where I drop them, others won't move at all and yet others will get positioned somewhere close by (eg. 4 or 5 photos before or after).

After digging into the catalog file I've come across what I think is the problem, but don't know how to fix it. In the attached file you'll see a screenshot of the database table for the collection, you'll see I've hilited the images that are part of the same collection, but their positionId is identical (which should never happen I'm assuming), probably due to the field size reaching it's maximum length. This is what I believe is causing the problem. Tested this on both Lightroom 5.7 and CC 2015.8.

This is a major bug and effectively stops user sorting from being functional, as well as now having potentially lost weeks of work. Any suggestions?



Thanks,

Adrian

Bug Fixed
TOPICS
macOS , Windows

Views

618

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines

correct answers 1 Correct answer

LEGEND , Dec 07, 2020 Dec 07, 2020

The most recent bug recipe now works in LR 10.1. Let's hope that LR 10.1 also fixes the errors getting thrown by collections using custom sort order.

Votes

Translate

Translate
87 Comments
New Here ,
Aug 10, 2020 Aug 10, 2020

Copy link to clipboard

Copied

I experience a variant of this problem: if I am in a folder with custom order, after I edit the image in Photoshop the psd+raw file stack goes to the end of the filmstrip. The remaining photos maintain the previous order. This issue too depends on the number of images in the folder and the number of changes in the folder.

Votes

Translate

Translate

Report

Report
Explorer ,
Aug 10, 2020 Aug 10, 2020

Copy link to clipboard

Copied

Rikk,

Any response yet from the product manager?

Votes

Translate

Translate

Report

Report
Adobe Employee ,
Aug 10, 2020 Aug 10, 2020

Copy link to clipboard

Copied

Thanks for lettings us know.
We will investigate.
Could you please let us know the number of images in the folder?

Thanks,
Sunil

Votes

Translate

Translate

Report

Report
New Here ,
Aug 11, 2020 Aug 11, 2020

Copy link to clipboard

Copied

About 370, roughly half of which psd files.

Votes

Translate

Translate

Report

Report
Explorer ,
Aug 11, 2020 Aug 11, 2020

Copy link to clipboard

Copied

This is so crazy.  And it seems like the solution has already been described.  All the engineers need to do is code up the solution and then test it.  I'll bet this work represents less an 20% of a four-week Agile Scrum sprint.

Votes

Translate

Translate

Report

Report
Community Beginner ,
Aug 14, 2020 Aug 14, 2020

Copy link to clipboard

Copied

Sunil, how is that investigation coming?

Votes

Translate

Translate

Report

Report
Adobe Employee ,
Oct 20, 2020 Oct 20, 2020

Copy link to clipboard

Copied

Greetings,

 

Updates for the Adobe Photography Products were released on 10.20.2020 that include fixes for these issues. Please install the most recent update and confirm that your issue is now fixed. Please let us know if you encounter any issues.

 

Thank you for your patience.

Votes

Translate

Translate

Report

Report
LEGEND ,
Oct 22, 2020 Oct 22, 2020

Copy link to clipboard

Copied

Unfortunately, LR 10.0 still has bugs with custom collection ordering, so I re-marked this bug as "in progress". To reproduce:

 

1. Create a new catalog and import n photos, where n is at least 6118.

 

2. Create a collection containing all n photos.

 

3. Let's call the first three photos in the collection A, B, and C.

 

4. Select all the photos except the first two (A and B) and drag the selection between A and B, creating a custom order.

 

5. Observe that C is now the second photo and B is at the end (correct).

 

6. Select all the photos except the first two (A and C) and drag the selection between A and C.

 

7. Observe that another photo is at the very end, rather than B and C (incorrect).

 

This scenario isn't as unlikely as it seems. From the reports on the forums over the years, users occasionally do have large collections with more than 6K photos.  And because dragging photos is painfully slow in grid view and you can't have two grid views open, an idiom for moving an early photo last is by select all the photos after the early photo and dragging them in front of it.

 

Regardless, this test shows there are still bugs lurking, which may show up in more frequent situations.

 

-------------------------

Technical Details

 

Poking around in the catalog database hints at the locus of the bug.

 

The field AgLibraryCollectionImage.positionInCollection stores the position of an image within a collection. Prior to LR 10, that field stored a 64-bit floating number, but in LR 10 it's a varying-length string containing the 63 characters "-", 0-9, A-Z, a-z.  The order of the images within a collection is given by the lexicographic sort of the field.  

 

When the ordering is working correctly, all the images in a collection should have a unique value for .positionInCollection. But after step 6 of the bug recipe above, two of the 6118 images have "nil" as their value.

 

Further, if you keep repeating step 6, more and more images get "nil" positions. And most suspiciously, the maximum length of the .positionInCollection values is 1023.  I've tried larger values of n and observe the same behavior, with the maximum length stuck at 1023.

 

Sqlite doesn't impose any practical maximum length on string-valued fields, so the custom-order algorithm either has 1023 built-in somewhere or there's a string buffer limited to size 1023.

 

Using a varying-length string to encode position is clever.  Suppose two consecutive images are assigned position "AE" and "AF". A third image can be inserted between the two by using position "AEV".  ("V" is the 32nd letter in the 63-letter alphabet.)  And if the last image in the collection has position "zz ... z" ("z" is the last letter in the 63-letter alphabet), an image can always be inserted after it by assigning it "zz ..zV".

 

The advantage of this encoding is that in most circumstances, an insertion doesn't require renumbering the positions of the preceeding and following images.  An extreme case is when you try to insert before a sequence of images with positions "-", "--", "---", "----", etc. With this alphabet, there are no characters that come before "-", so inserting an image in the first position requires renumbering all the subsequent images.

 

I haven't analyzed the LR implementation thoroughly, but it seems that it sometimes renumbers positions when it doesn't need to.

 

Finally, it's not at all clear that this new implementation is more efficient than using the old one, which used 64-bit numbers (correctly implemented).  The old one used just 8-byte floats, while the new one uses varying-length strings that can get arbitrarily long. Even in the common case with few insertions, the minimum length used by the implementation is 4 characters, plus the Lua overhead of perhaps another 8 bytes for the string object header (when the strings are read from the database into memory).  Comparing and sorting floats is perhaps 1 to 2 orders of magnitude faster than comparing varying-length strings.


Using floats may require somewhat more renumbering of positions than strings, but renumbering with strings is much more expensive when it occurs.  

Votes

Translate

Translate

Report

Report
LEGEND ,
Oct 24, 2020 Oct 24, 2020

Copy link to clipboard

Copied

A number of people are reporting errors when they attempt to add images to collections, indicating related issues with the new representation of custom order, with "nil" getting stored in positionInCollection:

https://feedback.photoshop.com/conversations/lightroom-classic/lightroom-classic-when-trying-to-add-...

Votes

Translate

Translate

Report

Report
Community Beginner ,
Oct 27, 2020 Oct 27, 2020

Copy link to clipboard

Copied

Yes, it does still exist in 10.00! I have just stalled after re-ordering about 150 images in a collection - a total probably of around 100 slide-moves - some as multiples. Just like previous versions!

Votes

Translate

Translate

Report

Report
LEGEND ,
Dec 07, 2020 Dec 07, 2020

Copy link to clipboard

Copied

The most recent bug recipe now works in LR 10.1. Let's hope that LR 10.1 also fixes the errors getting thrown by collections using custom sort order.

Votes

Translate

Translate

Report

Report
Explorer ,
Jun 24, 2021 Jun 24, 2021

Copy link to clipboard

Copied

LATEST

I also can attest that this still happens in LrClassic 10.3 - however, it was *much* better behaved than before.  Over the last two weeks I worked on a collection of 200+ images for my recently passed father-in-law's memorial, and the final task was to get them all into presentation order - so quite a few drag-and-drops of single images, and some small groups of images (3 to 10 at a time).  Only once in that time did I see the dreaded "Reorder photos from grid view and not from another source" error pop up preventing me from re-ordering.  A year ago I was seeing this excruciatingly often, but hadn't read this thread in detail.  John Ellis' analysis is awesome and helps me understand some possible workarounds.  Thanks.

Votes

Translate

Translate

Report

Report