I was getting the same error and after about a week of
troubleshooting I found the problem to be with the JSch class that
shipped with ColdFusion 8 which is version 0.1.28. I found looking
through the change log for JSch that in version 0.1.29 they fixed a
bug that related to having long banners, which our company has a
very long warning banner. So, I stopped the ColdFusion/JRun
services and renamed the existing jsch-0.1.28m.jar file to
jsch-0.1.28m.jar.orig and put the latest available version which as
of this posting was 0.1.37 into the same folder. Then I restarted
JRun and ColdFusion and I was able to do a secure FTP without
getting the java.lang.OutOfMemoryError message.
I just installed ColdFusion 8.0.1 and the version of JSch is
still 0.1.28, hopefully the next update will have a newer version.
You can download the latest JSch library from
http://www.jcraft.com/jsch/