Skip to main content
Inspiring
July 13, 2010
Question

How to read files larger than 100mb

  • July 13, 2010
  • 2 replies
  • 4453 views

Hi,

neither LrFileUtils.readFile or io:read can read a file larger than 100mb in one chunk. Any idea how to read it in one piece?

rgds - wilko

This topic has been closed for replies.

2 replies

areohbee
Legend
July 18, 2010

Here's something I wrote recently to copy a big file - could be adapted just for reading:

local __copyBigFile = function( sourcePath, destPath, progressScope )

    local fileSize = LrFileUtils.fileAttributes( sourcePath ).fileSize

    local g
    local s
    local t
    -- local blkSize = 32768 -- typical cluster size on large system or primary data drive.
    local blkSize = 10000000 -- 10MB at a time - lua is fine with big chunks.
    local nBlks = math.ceil( fileSize / blkSize )
    local b
    local x
    g, s = pcall( io.open, sourcePath, 'rb' )
    if not g then return false, s end
    g, t = pcall( io.open, destPath, 'wb' )
    if not g then
        pcall( io.close, s )
        return false, t
    end
    local done = false
    local m = 'unknown error'
    local i = 0
    repeat -- forever - until break
        g, b = pcall( s.read, s, blkSize )
        if not g then
            m = b
            break
        end
        if b then
            g, x = pcall( t.write, t, b )
            if not g then
                m = x
                break
            end
            i = i + 1
            if progressScope then
                progressScope:setPortionComplete( i, nBlks )
            end
            LrTasks.yield()
        else
            g, x = pcall( t.flush, t ) -- close also flushes, but I feel more comfortable pre-flushing and checking -
                -- that way I know if any error is due to writing or closing after written / flushed.
            if not g then
                m = x
                break
            end
            m = '' -- completed sans incident.
            done = true
            break
        end
    until false
    pcall( s.close, s )
    pcall( t.close, t )
    if done then
        return true
    else
        return false, m
    end
       
end

Vladimir Vinogradsky
Inspiring
July 13, 2010

Wilko,

I think trying to read this much in one chunk is a bad idea. Could you explain why you even need to do this?

Inspiring
July 15, 2010

To be honest - 100mb is a small value. I want to calculate checksums from e.g. psd and tiff files .Of course reading files in smaller chunks is possible, but this is just a workaround.

rgds - wilko

johnrellis
Legend
July 15, 2010

In terms of performance, you'll most likely get the best performance reading in chunks much smaller than 100 MB, e.g. 1 MB or even smaller.  You might see severely degraded performance trying to read an entire 500 MB file into memory.

In terms of programming convenience, of course, it's nice to read an entire file all at once into a string.