Skip to main content
Participant
September 13, 2019
Question

How to handle > 200mb data file using ColdFusion

  • September 13, 2019
  • 1 reply
  • 466 views

Hello ColdFusion Expert,

I have a question regarding flat file processing using ColdFusion.

Currently at our company we used webMethod to process big data file( > 200MB) - it is working fine and fast but due to the maintenance cost is to high we decided to use ColdFusion. The question is how to process big data file using ColdFusion without consume high memory that will effect the application performance?

    This topic has been closed for replies.

    1 reply

    Charlie Arehart
    Community Expert
    Community Expert
    September 13, 2019

    It depends on how you will be reading it. Do you know yet? You can read such files with cfhttp (yes, pointing to a csv file, and it can turn that into a query).  Or you can use CFFILE and its various actions, some which read the entire file, some which read it line by line. Same for the file* function equivalents to that.

     

    But either way, 200m is nothing to a CF running on, say, a 1gig heap--unless of course other things runnign within CF are using up that 1g heap. Usually it would not be so, that "all the memory is already used". And many shops run with 2, 4, 8, or far more gigs of heap.

     

    So this may well be no issue for you to handle. Only testing will tell.  And good monitoring tools (like FusionReactor, or JVM tools, or others) can help you understand how memory is used during and after that request processing.

    /Charlie (troubleshooter, carehart. org)
    Participant
    September 18, 2019
    Thank you for the answer. really helpful.