How to handle > 200mb data file using ColdFusion

[Mohd_Hassan]

Hello ColdFusion Expert,

I have a question regarding flat file processing using ColdFusion.

Currently at our company we used webMethod to process big data file( > 200MB) - it is working fine and fast but due to the maintenance cost is to high we decided to use ColdFusion. The question is how to process big data file using ColdFusion without consume high memory that will effect the application performance?

[Charlie Arehart]

It depends on how you will be reading it. Do you know yet? You can read such files with cfhttp (yes, pointing to a csv file, and it can turn that into a query).  Or you can use CFFILE and its various actions, some which read the entire file, some which read it line by line. Same for the file* function equivalents to that.

But either way, 200m is nothing to a CF running on, say, a 1gig heap--unless of course other things runnign within CF are using up that 1g heap. Usually it would not be so, that "all the memory is already used". And many shops run with 2, 4, 8, or far more gigs of heap.

So this may well be no issue for you to handle. Only testing will tell.  And good monitoring tools (like FusionReactor, or JVM tools, or others) can help you understand how memory is used during and after that request processing.

[Mohd_Hassan]

Thank you for the answer. really helpful.