Large file uploads and Request Throttle Memory.

[zeejayy]

I have some questions regarding large file uploads and memory usage. Maybe sombody has some more info on this.

I am working with CF 2021 and IIS 10.

I just finished troubleshooting a problem with large file uploads in a multipart form post. Anything larger then 200mb was failing with an internal server error. It turns out that setting max post size is not enough, I also have to set the request throttle memory setting to a value larger then the max file size I want to  upload.

This is really probematic because it actually seems to load the file into memory as it uploads, and it takes up the memory for the entire duration of the upload. Is this really the way uploads have to work in CF? It seems really stupid.  Is there any other way I can allow large file uploads without having to potentialy tie up so much memory?

[zeejayy]

I guess I should have waited a bit longer before posting this. Looks like I can chunk my uploads and it will probably solve my issue. I will leave this up in case anyone has other suggestions.

[Charlie Arehart]

If you mean that you still think cf ties up memory in cf equal to the size of a uploaded file, no, that should not be the case as far as I know. That was a problem up until an update in cf7, about 15 years ago. Do you have some evidence that indicates it's doing that?

If you're only speculating, is the bottom line that you feel you cannot upload a large file? If so, what size file fails? What is the error (preferably a detailed error message from one of cf's logs, if none is shown onscreen)? And what are your values for those throttle settings? (I can concur that they've always been very poorly understood/documented.) 

[zeejayy · Accepted solution]

Hi Charlie, thanks for the reply.

I was actually able to see the IIS worker process, in task manager, using up increasing amounts of memory during the upload and maxing out at the total post size including the file. I know this type of thing was supposed to be fixed a long time ago so I am not sure what exactly was going on.

I am seeing this when posting a multipart form with a file and using CF file with action upload to save it to a dir. If the file size is over the value set in request throttle memory then I get a generic 500 error instead. I wasn't able to find any info in the logs relating to it.

I have never had a need to upload very large files directly via POST in CF before, dispite being a CF developer since ver 4.5. As such I have never needed to mess with the post size limit or the request throttle memory.

Dispite all of this I already found a better way to do it using a chunked uploader. This way the max post size stays way under the default limit and I don't need to fuss with anything there.

[Charlie Arehart]

Zee, ok on that. And though you've found a workaround, you or others seeing this may want to better understand the concerns you've raised. 

1) First, if you're saying now that it's the iis worker process increasing in size (and not cf), OK. That wasn't clear, when you said originally, "it takes up the memory for the entire duration of the upload. Is this really the way uploads have to work in CF? It seems really stupid." Since you'd been discussing the cf admin throttle, I assumed you were referring to cf use of memory, thus my first reply. 

As for the worker process memory going up, that could just be how iis (or the cf web server connector) works, perhaps related to the next point I want to make. 

2) As for the cffile action of upload, I'll note FWIW that that's something you could REMOVE to see if it made any difference in the failure you got. I doubt it would. But the iis memory could still increase. 

Many think that tag is needed for cf to "receive" an uploaded file, but technically cf "receives" any uploaded file regardless of the presence of that tag/action--saving the uploaded file to a temp folder, as a temp file name. The tag/action merely performs the moving and renaming of that temp file to the named DESTINATION folder.

And yes, this does mean that ANY cf template could have a file posted to it, and the temp file would simply be removed by cf at the end of the request. I have a blog post with details on all this, albeit from 2006--but the facts have not changed--at https://www.carehart.org/blog/2006/5/7/cffile_is_not_what_receives_upload .) 

3) Finally, on the effect of the cf throttle feature, it's clearly got a role here, as you first noted. But that's about what cf will allow, and again should have no bearing on whether cf itself "holds memory" for the duration of the upload request.

You've got your workaround, so I realize all this may be superfluous. But I hope it may have helped you some, or other readers. Folks may well find it in the future while searching for info related to such problems, and at least we've given them more to think about. 🙂