Problem Writing Large Files

[BrianTech3]

Hi all,

Now that I got the read problem under wraps, I'm trying to write some data from a database to a text file that has fixed width records. I'm looping through and doing appends with CFFile. Around 22MB it bombs out saying I've run out of heap memory. Is there a better way I should be trying to write it, other than using CFFile?

Thank you

[Guest]

Hi,

When I write out to a file, I use the MOD functionality. In the example below, I grap 25 rows, them write them to the file. Modify the number after the MOD to keep your memory in the safe zone. The last piece of the code dumps out any remaining rows (say you have 5 left over, but can't get to the full 25). The WriteCSV is my query that is writing out the records to the table.

cfwild

[balumohan]

will this be solved by increasing the Maximum JVM Heap Size  (MB) ???

[tclaremont]

I have found that modifying the heap size can be tricky... and frustrating. Use the following code to see where your stats are right now, and refresh this page periodically while your CFFILE routine is running:

<cfoutput>

<cfset runtime = CreateObject("java", "java.lang.Runtime").getRuntime()>
<cfset freeMemory = runtime.freeMemory() / 1024 / 1024>  <cfset totalMemory = runtime.totalMemory() / 1024 / 1024>  <cfset maxMemory = runtime.maxMemory() / 1024 / 1024>

Max Memory Available to JVM: #Round(maxMemory)#MB<br>  Total Memory Allocated: #Round(totalMemory)#MB<br>  Free Allocated Memory: #Round(freeMemory)#MB

<br><br>
</cfoutput>

Run this page (above) and note the stats. Then, while the CFFILE process is running, refresh the stats page periodically and note what is happening to your memory. Then, once the CFFILE page either crashes or completed, run the stats page a final time and note the results.

Post back with what you find.

[ilssac]

You may want to try using some Java objects to write this large file out, they can be more efficient and less memory hungry then the CFML versions.

Here is a good looking link that describes this in more detail I got with a Google Search for ColdFusion Java file object

http://www.bennadel.com/blog/305-ColdFusion-CFFile-vs-Java-java-io-BufferedOutputStream.htm