Exit
  • Global community
    • Language:
      • Deutsch
      • English
      • Español
      • Français
      • Português
  • 日本語コミュニティ
  • 한국 커뮤니티
Locked
0

How do I use fileStream without reading the file into memory?

New Here ,
Jan 24, 2011 Jan 24, 2011

I am writing a process where users will need to select a file that far exceeds their availble RAM and have that file broken up into small chunks (for upload).

I'm able to create a File reference to said file, but when I try to pass it to the fileStream, it appears to try to read the thing into memory before acting on it.

Is there a way to get fileStream to just take the reference to the file and then utilize readBytes the way it's documented?

Much obliged, in advance.

Here is my code... just in case I'm doing something blatantly stupid...

//this is being called by the File callback, uplon the user browsing over to the selected file.

private function selectHandler(event:Event):void {
            var file:File = File(event.target);
            trace("selectHandler: name=" + file.name );
           
            var stream:FileStream = new FileStream();
           
            var f:File = event.target as File;
            stream.open(f, FileMode.READ); //at this point, the program will lock up if the file exceeds the computer's memory buffer.
            var bytes:ByteArray = new ByteArray();
            stream.readBytes(bytes,0,1024);
            trace(bytes);
            stream.close();
}

-Dr.D

TOPICS
Performance issues
4.2K
Translate
Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Adobe Employee ,
Jan 24, 2011 Jan 24, 2011

Hi Dr. D,

Could you try using OpenAsync() instead of Open()?  If I recall, we have a bug on very large (2gb+) files on 32bit systems with Open() that is not present with OpenAsync().  I took a look at the docs and it does appear that reading the file in immediately is as designed:

If the fileMode parameter is set to FileMode.READ or FileMode.UPDATE, AIR reads data into the input buffer as soon as the file is opened, andprogress and open events are dispatched as the data is read to the input buffer.

Chris

Translate
Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
New Here ,
Jan 24, 2011 Jan 24, 2011

Hi Chris,

I've tried using openAsync and then reacting to the onProgress event but I'm not certain what I'm seeing.

The two approaches I assumed would work were to open the stream and then alter the position property and do a read. Or to use the reabBytes() method and pass in a distant offset (I'm toying with approx. the 200meg mark in the file, this is 10Gig file).

However, the entire process hangs for a while (about 10 seconds) and then I do get some output, but the whole thing seems very hinky and unstable.

In an ideal world the stream would not actually read any more than I'm asking it to read and only from the position from which I'm asking it to read.  Instead, it's doing stuff behind the scenes that I have no control over or visibility into, so, I'm starting to think that maybe Flash is not the right tool for this job.

Translate
Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
New Here ,
Jan 24, 2011 Jan 24, 2011

hmm... I see now that the openAsync method seems to scan through the entire file before being COMPLETE, however, it's not reading it into memory. It's just doing some sort of wierd scan.  I'm not sure if this is going to cause problems for me once the file size I'm dealing with is around 30gigs, but I guess I'll wait and see what happens. I'll mark this as closed for now, thanks for your help.

Translate
Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
New Here ,
Jan 24, 2011 Jan 24, 2011

ok... I was wrong... it is definitely reading the file into memory.

It gets as far as 1023.9 megs and quits. The app crashes, and occasionally it brings the Flash IDE down with it.

So... I don't think fileStream works as you guys want it to (or maybe this is how it's intended to work), as there doesn't appear to be a way to simply read through data on the disk without loading it into memory.

I'm leaning towards turning to Java for my solution here, but if you can think of how this can be done, I'd prefer to stick with AIR.

Translate
Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Adobe Employee ,
Jan 24, 2011 Jan 24, 2011

Thanks for the update.  I'll ask around and see if I can get some additional info for you.

Chris

Translate
Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Adobe Employee ,
Jan 25, 2011 Jan 25, 2011

Ok, I think the key is to set the readAhead property and then call openAsync().  I just tried a sample app (see attached) on a 4gb file and the app did not lock up or run out of memory.  Give it a try when you get a chance and let me know how it goes.

Thanks,

Chris

Translate
Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
New Here ,
Jan 25, 2011 Jan 25, 2011

Well... we're definitely getting closer!

Ok, so the issue I'm seeing now is what appears like inadequate garbage collection.

I'm using readBytes( myByteArray, startPos, sliceSize);

This provokes the continuation of the read (otherwise the read just stops... which is great!)

However, one of two things happen.

Either I don't re-initialize myByteArray in the process, in which case I run out of memory around 640 megs

or, before every read I declare:

myByteArray = new ByteArray()

and this allows me to get to around 1.8 gigs before the system runs out of memory... presumably because the garbage collector isn't keeping up? (I'm guessing obviously).

So the question is... what did you do to vacate that container in your test, or did you use something other than readBytes and if so, what was it?

I'm attaching my code snippet just so that you can tell me I'm doing something wildly stupid (if I am).

        private var total:Number = 0;
        private var b:ByteArray = new ByteArray();
        private function onBytesRead(e:ProgressEvent){
           
            if(e.currentTarget.bytesAvailable >= chunk){
                trace("avail = " + e.currentTarget.bytesAvailable)
                trace ("read " + (total += e.currentTarget.bytesAvailable * 1));
              
                b = new ByteArray();
                e.currentTarget.readBytes(b,total, chunk)
               
               
            }
           
        }   

         //called when user selects a file
         private function selectHandler(event:Event):void {
            var file:File = File(event.target);
           
            stream = new FileStream();           
            var f:File = event.target as File;           
            stream.addEventListener(ProgressEvent.PROGRESS,onBytesRead);
           
            stream.readAhead = chunk;
            stream.openAsync(f, FileMode.READ);           
        }

Translate
Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Guest
Jan 26, 2011 Jan 26, 2011

Just a guess.

after e.currentTarget.readBytes(b,total, chunk)

can u add

b = null;

System.gc();

Just to see if it help. (System.gc() will force the garbage collection.)

Translate
Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
New Here ,
Jan 26, 2011 Jan 26, 2011

No dice (though a promising idea!).

I recall reading that gc() doesn't actually force garbage collection, but is rather just a strong reccomendation to have it happen. In any event, I still run out of memory right under a gig.

Would you mind posting the code for the attached app above? If it works the way you're doing it, it must be something really simple that I'm overlooking.

I just ran a 5.7 gig file through the readBytes.air app you attached and it worked like a dream, so, clearly there IS a way to do this... I must be missing something pivotal though... I really appreciate you helping me. If the code is not secret, I'm sure a quick glance at it will reveal what you're doing diffirently that's making it possible for your version not to overrun memory.

Translate
Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Adobe Employee ,
Jan 26, 2011 Jan 26, 2011

Sorry about that.  I should have made the project .zip instead of .fxp.  Here's the code from the sample app:

<?xml version="1.0" encoding="utf-8"?> <mx:WindowedApplication xmlns:mx="http://www.adobe.com/2006/mxml"                               layout="absolute"                               width="597"                               height="370"                               showStatusBar="false">            <mx:Script>           <![CDATA[                import mx.controls.Alert;                import mx.utils.ObjectUtil;                                private static const READ_CHUNK_SIZE:uint = 100;                     private var fileStream:FileStream = null;                private var file:File = null;                private var toRead:uint = 0;                private var bytesLeft:Number = 0;                private var curBytes:ByteArray = null;                          private var totalBytesRead:Number = 0;                                private function onBrowse():void                {                     var file:File = null;                     if (textinput.text != "" && textinput.text != null)                          file = new File(textinput.text);                     else                          file = new File();                                          file.browseForOpen("Choose a file");                     file.addEventListener(Event.SELECT, selectHandler);                                          function selectHandler(event:Event):void                     {                          textinput.text = file.nativePath;                     }                                   }                                private function processFile(sync:Boolean):void                {                     fileStream = new FileStream();                     file = new File(textinput.text);                     toRead = 0;                     totalBytesRead = 0;                     bytesLeft = file.size;                     curBytes = new ByteArray();                                          if (sync)                          readSync();                     else                          readAsync();                }                                private function readAsync():void                {                     fileStream.addEventListener(Event.COMPLETE,streamCompleted);                     fileStream.addEventListener(Event.CLOSE,streamClosed);                     fileStream.addEventListener(ProgressEvent.PROGRESS,progressBytesIn);                     fileStream.addEventListener(IOErrorEvent.IO_ERROR,ioError);                                          fileStream.readAhead=40960;                     fileStream.openAsync(file,FileMode.READ);                                          ta.text += "File Total Size: " + file.size.toString()+"\n";                     }                                private function progressBytesIn(e:ProgressEvent):void                {                     trace("progressBytesIn: " + fileStream.bytesAvailable);                     progressLoadFile.setProgress(e.bytesLoaded, e.bytesTotal);                     fileStream.readBytes(curBytes,0,fileStream.bytesAvailable);                }                                private function streamCompleted(e:Event):void                {                     trace("streamCompleted: " + fileStream.bytesAvailable);                     ta.text += "streamCompleted: " + fileStream.bytesAvailable.toString() + "\n";                }                                private function streamClosed(e:Event):void                {                     trace("Unexpected stream closing:" + e);                     ta.text += "Unexpected stream closing:" + e.toString() + "\n";                }                                private function ioError(e:IOErrorEvent):void                {                     trace("IOError:",e);                     ta.text += "IOError" + e.toString() + "\n";                }                                              private function readSync():void                {                     fileStream.open(file, FileMode.READ);                                          while(bytesLeft > 0)                     {                          if (bytesLeft > READ_CHUNK_SIZE){                               toRead = READ_CHUNK_SIZE;                          } else {                               toRead = bytesLeft;                          }                                                    if(toRead > fileStream.bytesAvailable)                               toRead = fileStream.bytesAvailable;                                                    try{                               fileStream.readBytes(curBytes, 0, toRead);                                    } catch(err:EOFError) {                               fileStream.close();                                    Alert.show("EOFError: "+ObjectUtil.toString( err));                               return;                          } catch(err:IOError) {                               fileStream.close();                                    Alert.show("IOError: "+ObjectUtil.toString( err));                               return;                                                        }                                                         bytesLeft = bytesLeft - toRead;                          totalBytesRead = totalBytesRead + toRead;                                                    var msg:String = "totalBytesRead:"+totalBytesRead+"\n";                          msg=msg + "bytesLeft:"+bytesLeft+"\n";                          msg=msg + "bytesAvailable:"+fileStream.bytesAvailable+"\n";                          ta.text += msg;                                              }                                     }                                private function readFile():void                {                     processFile(ChooseSync.selected);                }                           ]]>      </mx:Script>            <mx:RadioButtonGroup id="RBGroup"/>      <mx:RadioButton x="37" y="271" label="Sync" groupName="RBGroup" id="ChooseSync"/>      <mx:RadioButton x="37" y="301" label="Async" groupName="RBGroup" id="ChooseAsync" selected="true"/>      <mx:Label x="23" y="37" text="input file name:" width="97"/>      <mx:TextInput x="118" y="35" width="362" id="textinput"/>      <mx:Button x="488" y="35" label="Browse" click="onBrowse()"/>      <mx:Button x="450" y="289" label="Read File" width="106" click="readFile()"/>      <mx:TextArea x="37" y="119" width="486" height="135" id="ta"/>      <mx:ProgressBar x="37" y="83" width="486" id="progressLoadFile" mode="manual"/>       </mx:WindowedApplication>

Translate
Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
New Here ,
Jan 26, 2011 Jan 26, 2011

EUREKA!!!

The problem was my understanding of the second parameter in readBytes().

I was using my running total as the place to start (assuming that the file being read has a pointer that looks to the place in the file as an offset to start reading, which along with the third param (the size of the slice) gives you a current byte array. But no, the second parameter should always be 0.

That was the only thing different from my code and yours. As long as the second param is set to 0 you can read through anything (I just tested with a 5.7 gig file and it read it without a hitch... I assume that I can read anything... I'll post again, if that turns out not to be true, I'll test with a 30gig file later today.

Thanks very much for your help.

In case anyone looks to this thread to the solution to this problem the key to the whole thing is as follows....

make sure to set your lookAhead to some manageable length (I have mine set to 10megs)

and then make sure to keep your second param at 0 in your readBytes call.

Translate
Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
New Here ,
Sep 25, 2012 Sep 25, 2012
LATEST

I am having the same issue on ios.  The ipad has only 512mb ram and I am thinking its crashing the app when it runs out of avaialble ram at around 350mb downloaded.  Dragon Drop it would be helpful to see your working code.  Is there any way to write the file as its downloading sort of on the download progress handler?

Translate
Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines