Out of Memory Error with CFXML tag

Forum|Forum|16 years ago
February 22, 2010
2 replies
962 views

After spending several days successfully constructing an XML data extract template and getting it to work as needed, I'm frustated to discover that when I try to run a large batch, I get an "outofmemory" error. I have tons of data to extract covering several years and my client wants the data in monthly batches. But the most I can run without getting the out of memory error is about 5 days.

I've read in the Forums about being able to append to an existing file, but it requires reading in the existing output and placing the new entry in the appropriate spot, then re-writing the entire file. That does not sound like a viable solution for me.

Is there any other way to resolve this without trying to find another solution outside of ColdFusion? The data is stored in a SQL Server 2005 database in multiple tables, and has to be manipulated before outputing it into XML (stored code values have to be intrepreted, dates set in a specific format, etc.). The thought of starting over with something else besides CF makes me want to puke! So I'm desparately hoping that someone has a quick fix to what I've already written using the CFXML tag.

This topic has been closed for replies.

BKBK

Community Expert

Just do it without the cfxml tag. That is, XML as just a string, wherever possible. Then use cfxml to convert to the object, only where you need to parse, manipulate or modify the XML.

N

NewMXerAuthor

Participant

Well, that sounds simple enough. I don't need an XML object anyway because all I am doing is dumping it out to a file using the "tostring" function.

So I replaced the cfxml tags with cfsavecontent, added the obligatory "<?xml..." line and viola!, I ran an entire month of data in less than 13 minutes (102 MB output file) and with no memory issues!

I guess my mind was married to cfxml and I didn't step back to take another look. Thanks for the advice!

A

Anonymous

Just my two cents...

I have no idea how complex your SQL statement is for this export, but 13mins for 102 MB seems like a very long time to me. I have exports/imports using files upto 1GB in size that run much faster than this.

If you are running SQL 2005+, I would consider using the built in XML export features, they are much faster and you wont be using memory twice (once for SQL and once for CF).

J

Joshua_Cyr

Participating Frequently

Do you have control over the CF server. I am not a server config guy but I suspect you can find ways to increase the memory available which may solve your problem.

This is for an older version of CF (it isn't clear which CF you are working with) but may illustrate some options.

http://kb2.adobe.com/cps/185/tn_18548.html

http://blog.newmediadevelopment.net/2009/10/tracking-down-out-of-memory-errors-in-coldfusion-standard.html

Also it may be that the structure of the xml doc could help or hurt.

http://www.bennadel.com/blog/1344-XML-Building-Parsing-Traversing-Speed-In-ColdFusion.htm

N

NewMXerAuthor

Participant

Thanks for the quick reply. I've spent much of the day researching this issue, starting with the links you provided. I'm using CF8 and I do have control of the server. I'm going to try a couple of adjustments to the JVM that were recommended in the one blog you provided a link to. I never thought of having two garbage collectors listed in there. I'm interested in seeing how, if any, that improves my throughput.

So I'll fine tune the JVM as much as I can and run as large a batches as my configuration will alow me. This is a one-time project, meaning that once the historical data is exported I will simply be doing a daily update. And for that, what I have works great (no memory problems).

Sign up

To post, reply, or follow discussions, please sign in with your Adobe ID.

Sign in to Adobe Community

To post, reply, or follow discussions, please sign in with your Adobe ID.

Scanning file for viruses.

This file cannot be downloaded