Fastest way to generate small Jpegs for entire catalog using the SDK?

[Pit32185406stji]

Hi,

I'm trying to build a plugin that brings content-aware search to Lightroom via open_clip AI models. The plugin works well, but to build the search index, I need a (small) Jpeg of each photo in the catalog. Exporting everything via an exportSession works, but is painfully slow (about 2 photos/second). 

Instead, I tried exporting everything via the requestJpegThumbnail method (see code below, wrapped with some other code in startAsyncTask), but oftentimes this for loop exits after exporting only parts of my catalog. This is problematic because the code that comes after this is reliant on having all the Jpegs available. 

Does anyone know a reliable, fast way to get Jpegs for all images in a catalog?

for _, photo in ipairs( selectedPhotos ) do
    local uuid = photo:getRawMetadata("uuid")
    local previewname =  uuid .. ".jpg"
    local holdRefObj = photo:requestJpegThumbnail(320, 320, function( success, failure )
        local f = io.open(folder .. previewname,"wb")
        f:write(success)
        f:close()
    end )				
end

[John R Ellis]

"Exporting everything via an exportSession works, but is painfully slow (about 2 photos/second). "

On a computer with a graphics processor from the last several years, exporting 1024-wide JPEGs from raws should go 4-5 photos/second.

[Pit32185406stji]

I have an RX7900 XT GPU, but it's probably this slow because I have to start a new export session for each image because (afaik) I can't name them after their UUID otherwise. I'll try with a better debugger / LrTasks.pcall() tomorrow and see what happens (currently using WinDbg)...

[John R Ellis]

"it's probably this slow because I have to start a new export session for each image because (afaik) I can't name them after their UUID otherwise."

Your plugin could export all the photos at once with a single export session, and LR should export them in parallel, taking full advantage of multiple processors. The plugin could then rename the exported files using LrFileUtils.move() after each image's export completes.

[Pit32185406stji]

After working on it for some more time I figured out the problem: requestJpegThumbnail() is an async function and my code simply did not wait for all the function calls to finish. Since (afaik) the SDK does not offer a way to wait for the completion of async functions I simply added a while-loop after my original code that counts the number of files in the target directory and repeatedly calls LrTasks.sleep(1) until all files have been written.

local fileCount = countFiles(folder)
local totalNumPhotos = #selectedPhotos
while fileCount < totalNumPhotos and not progressScope:isCanceled() do
    progressScope:setPortionComplete(fileCount, totalNumPhotos)	
    outputToLog("Waiting for export ")
    LrTasks.sleep(0.5)
    fileCount = countFiles(folder)
end

[Pit32185406stji · Accepted solution]

Actually, this alone was not sufficient. I also had to modify the original code, to keep the references to the requestJpegThumbnail() function calls in a wider scope than just the for-loop over photos. I initialize a table, which stores all references outside of the scope of the for-loop:

local refObjTable = {}
for _, photo in ipairs( selectedPhotos ) do
    if progressScope:isCanceled() then
        break
    end
    local uuid = photo:getRawMetadata("uuid")
    local previewname =  uuid .. ".jpg"
    table.insert(refObjTable,  photo:requestJpegThumbnail(720, 720, function( success, failure )
        local f = io.open(folder .. previewname,"wb")
        f:write(success)
        f:close()
        outputToLog("Exported " .. previewname)
    end ))	
end

In combination with the waiting loop mentioned above, this solves the issue.

[John R Ellis]

Rather than counting the number of files in the folder, which is pretty heavy in terms of CPU time invoking the OS, you could simply keep a count of the number of initiatians and completions of photo:requestJpegThumbnail():

local completions = 0

photo:requestJpegThumbnail(720, 720, function( success, failure )
    local f = io.open(folder .. previewname,"wb")
    f:write(success)
    f:close()
    completions = completions + 1
    outputToLog("Exported " .. previewname)
end)


local totalNumPhotos = #selectedPhotos
while completions < totalNumPhotos and not progressScope:isCanceled() do
    progressScope:setPortionComplete(fileCount, totalNumPhotos) 
    outputToLog("Waiting for export ")
    LrTasks.sleep(0.5)
end

[Pit32185406stji]

That's a great suggestion, thanks.

[John R Ellis]

"but oftentimes this for loop exits after exporting only parts of my catalog."

Are you using a debugger that's aware of LR's specific task architecture to trap any errors?  Async tasks that encounter an error will silently exit. 

Or use LrTasks.pcall() to wrap the entire block of code to catch potential errors.

[John R Ellis]

The photo:requestJpegThumbnail() has a history of reported problems:

https://community.adobe.com/t5/forums/searchpage/tab/message?sort_by=-topicPostDate&advanced=false&a...