• Global community
    • Language:
      • Deutsch
      • English
      • Español
      • Français
      • Português
  • 日本語コミュニティ
    Dedicated community for Japanese speakers
  • 한국 커뮤니티
    Dedicated community for Korean speakers
Exit
2

Fastest way to generate small Jpegs for entire catalog using the SDK?

Community Beginner ,
Jan 17, 2024 Jan 17, 2024

Copy link to clipboard

Copied

Hi,

I'm trying to build a plugin that brings content-aware search to Lightroom via open_clip AI models. The plugin works well, but to build the search index, I need a (small) Jpeg of each photo in the catalog. Exporting everything via an exportSession works, but is painfully slow (about 2 photos/second). 

Instead, I tried exporting everything via the requestJpegThumbnail method (see code below, wrapped with some other code in startAsyncTask), but oftentimes this for loop exits after exporting only parts of my catalog. This is problematic because the code that comes after this is reliant on having all the Jpegs available. 

Does anyone know a reliable, fast way to get Jpegs for all images in a catalog?

for _, photo in ipairs( selectedPhotos ) do
    local uuid = photo:getRawMetadata("uuid")
    local previewname =  uuid .. ".jpg"
    local holdRefObj = photo:requestJpegThumbnail(320, 320, function( success, failure )
        local f = io.open(folder .. previewname,"wb")
        f:write(success)
        f:close()
    end )				
end
TOPICS
SDK

Views

194

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines

correct answers 1 Correct answer

Community Beginner , Jan 21, 2024 Jan 21, 2024

Actually, this alone was not sufficient. I also had to modify the original code, to keep the references to the requestJpegThumbnail() function calls in a wider scope than just the for-loop over photos. I initialize a table, which stores all references outside of the scope of the for-loop:

 

local refObjTable = {}
for _, photo in ipairs( selectedPhotos ) do
    if progressScope:isCanceled() then
        break
    end
    local uuid = photo:getRawMetadata("uuid")
    local previewname =  uuid .. ".j
...

Votes

Translate

Translate
LEGEND ,
Jan 17, 2024 Jan 17, 2024

Copy link to clipboard

Copied

"Exporting everything via an exportSession works, but is painfully slow (about 2 photos/second). "

 

On a computer with a graphics processor from the last several years, exporting 1024-wide JPEGs from raws should go 4-5 photos/second.

 

 

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Community Beginner ,
Jan 17, 2024 Jan 17, 2024

Copy link to clipboard

Copied

I have an RX7900 XT GPU, but it's probably this slow because I have to start a new export session for each image because (afaik) I can't name them after their UUID otherwise. I'll try with a better debugger / LrTasks.pcall() tomorrow and see what happens (currently using WinDbg)...

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
LEGEND ,
Jan 17, 2024 Jan 17, 2024

Copy link to clipboard

Copied

"it's probably this slow because I have to start a new export session for each image because (afaik) I can't name them after their UUID otherwise."

 

Your plugin could export all the photos at once with a single export session, and LR should export them in parallel, taking full advantage of multiple processors. The plugin could then rename the exported files using LrFileUtils.move() after each image's export completes.

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Community Beginner ,
Jan 21, 2024 Jan 21, 2024

Copy link to clipboard

Copied

After working on it for some more time I figured out the problem: requestJpegThumbnail() is an async function and my code simply did not wait for all the function calls to finish. Since (afaik) the SDK does not offer a way to wait for the completion of async functions I simply added a while-loop after my original code that counts the number of files in the target directory and repeatedly calls LrTasks.sleep(1) until all files have been written.

local fileCount = countFiles(folder)
local totalNumPhotos = #selectedPhotos
while fileCount < totalNumPhotos and not progressScope:isCanceled() do
    progressScope:setPortionComplete(fileCount, totalNumPhotos)	
    outputToLog("Waiting for export ")
    LrTasks.sleep(0.5)
    fileCount = countFiles(folder)
end

  

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Community Beginner ,
Jan 21, 2024 Jan 21, 2024

Copy link to clipboard

Copied

Actually, this alone was not sufficient. I also had to modify the original code, to keep the references to the requestJpegThumbnail() function calls in a wider scope than just the for-loop over photos. I initialize a table, which stores all references outside of the scope of the for-loop:

 

local refObjTable = {}
for _, photo in ipairs( selectedPhotos ) do
    if progressScope:isCanceled() then
        break
    end
    local uuid = photo:getRawMetadata("uuid")
    local previewname =  uuid .. ".jpg"
    table.insert(refObjTable,  photo:requestJpegThumbnail(720, 720, function( success, failure )
        local f = io.open(folder .. previewname,"wb")
        f:write(success)
        f:close()
        outputToLog("Exported " .. previewname)
    end ))	
end

 

In combination with the waiting loop mentioned above, this solves the issue.

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
LEGEND ,
Jan 22, 2024 Jan 22, 2024

Copy link to clipboard

Copied

Rather than counting the number of files in the folder, which is pretty heavy in terms of CPU time invoking the OS, you could simply keep a count of the number of initiatians and completions of photo:requestJpegThumbnail():

 

local completions = 0

photo:requestJpegThumbnail(720, 720, function( success, failure )
    local f = io.open(folder .. previewname,"wb")
    f:write(success)
    f:close()
    completions = completions + 1
    outputToLog("Exported " .. previewname)
end)


local totalNumPhotos = #selectedPhotos
while completions < totalNumPhotos and not progressScope:isCanceled() do
    progressScope:setPortionComplete(fileCount, totalNumPhotos) 
    outputToLog("Waiting for export ")
    LrTasks.sleep(0.5)
end

 

 

 

 

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Community Beginner ,
Jan 25, 2024 Jan 25, 2024

Copy link to clipboard

Copied

LATEST

That's a great suggestion, thanks.

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
LEGEND ,
Jan 17, 2024 Jan 17, 2024

Copy link to clipboard

Copied

"but oftentimes this for loop exits after exporting only parts of my catalog."

 

Are you using a debugger that's aware of LR's specific task architecture to trap any errors?  Async tasks that encounter an error will silently exit. 

 

Or use LrTasks.pcall() to wrap the entire block of code to catch potential errors.

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
LEGEND ,
Jan 17, 2024 Jan 17, 2024

Copy link to clipboard

Copied

The photo:requestJpegThumbnail() has a history of reported problems:

https://community.adobe.com/t5/forums/searchpage/tab/message?sort_by=-topicPostDate&advanced=false&a...

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines