Skip to main content
O
obouillaud
Known Participant
April 21, 2006
Question

CFML gateway : java.lang.OutOfMemoryError

  • April 21, 2006
  • 20 replies
  • 2661 views
I though that CFML gateways were designed for long running tasks, but my tests tends to show the opposite :
whatever I do, it always crash with a java.lang.OutOfMemoryError

I made a mass emailing gateway, following the CDFJ article (
june 2005, volume 7, issue 6 : the asynchronous CFML gateway)

that is to say :
- a classic cfm page that defines the needed struct (arguments for the gateway cfc) then a call to the gateway.
- the gateway cfc is very basic : it makes a query to retrieve email adresses from a database then loop over the query and for each adress send an email.

I use cfoutput query + a cfmail for each email because i need to make a bit of other things in the loop (every 100 email sent i made another "update" query to keep an information of what email were sent.

I have about 150,000 emails to send but plan to send more (1,000,000. I'm not a spammer, all adresses are opted-in).

The first time it crashed around 25,000 emails.
I tried to maximize the JVM settings (maxperm, maxheap, ... to 512MB then 1024MB), it then crashed around 55,000, but whatever I do it still crashs before the end of the task. The server is a dual xeon 3.0 ghz with 2 GB of RAM and is running Coldfusion 7.0.1 (updater 2) on windows 2003 std.

I also replace the cfmal tag by cffile tag (for each adress i write then delete a small file, to simulate activity without really sending tons of emails ... and remove the "every 100 loops update a counter" query... it still crashs (but around 360,000 loops).

Just in case, I tried the cfsetting timeout in the cfc, the timeout in the struct passed to the cfc... no effect.

So I'm a bit confused about the aibility to handle long running tasks...

The exception.log files contains :
"Error","Thread-13","04/20/06","17:26:58",,"Error invoking CFC for gateway emailblaster: null"
java.lang.OutOfMemoryError


And here is a part of the cfc code :
This topic has been closed for replies.

20 replies

Show previous replies
BKBK
Community Expert
BKBKCommunity Expert
Community Expert
April 25, 2006
The question about try-catch was out of curiosity. Your explanation makes sense.

>... but as I said before, it also crash without the use of cfmail but
>just cffile action=write / delete

I remember that. I need the number of threads because I'm simulating your mailing procedure. While we're on the subject, how far will your loop go, with no other code than just

<cfloop query="liste_mails">
<!--- y'a rien ici --->
</cfloop>



O
obouillaudAuthor
Known Participant
April 25, 2006
The cf admin settings :
- Maintain connection to mail server : checked
- Spool Interval (seconds) : 15
- Mail Delivery Threads : 25
- Spool to : disk
- Max number of messages spooled to memory : 50 000

... but as I said before, it also crash without the use of cfmail but just cffile action=write / delete

The try/catch inside another try/catch is very simple :
if there is a crash inside the query output loop because of a probem to generate 1 email, I don't want to stop the process. But if there is a problem outside the loop I want a "clean" crash and an email sent back to me with information about the situation when it crashed.
N
Newsgroup_User
Inspiring
April 24, 2006
obouillaud wrote:
> - Why not cleaning the email before it's insertion in the database ? I already
> do it, but I want to be sure to avoid crashing the sending process because

you can't crash the "sending process" w/bad email, it will only NOT send those
emails.

> someone manualy (bad) changed an adress in the database, or everything else
> that can lead to a bad formated adress...

you need procedures, etc. to manage that sort of thing. for instance, if your db
has triggers, you can have it verify the email when/if somebody messes w/it.

> - splitting into chunks... what a shame !!! the gateways are designed for such
> tasks, that is to say long running queries. The asynchronous process is made to
> avoid the "blank loading page" of these long tasks. Do you mean I need to make
> a gateway that handle the whole process and that calls for example every
> thousands email another gateway that send the emails ?!?

probably several way to handle this. here's one:
one asynch gateway, one CFC, pass in query conditions & let it work (do the
query, send the mail). instead of one monster loop of 500k, maybe 10 of 50k
would work better. it's all the same to the user/main app--it hands off the
processing to the gateway & goes on to do other things. you can easily call the
gateway 10x in the main app, since everything's happening in the gateway it's
very fast as far as the user/main app is concerned.

i'm not sure what your UPDATE query is actually used for but you could easily
hand-off parsing send logs, etc. to an asynch gateway (could even be the same
one that's running the email, you can specify a different CFC to use) to handle
that. or if you really wanted to get "fancy" have a look at sean's concurrency
app: http://cfopen.org/projects/concurrency

i've combined gateways before. for example, SMS & asynch gateways work very well
together (cf can melt down SMS aggregators fairly easily). so having more than 1
gateway handle a job's not a bad idea.
O
obouillaudAuthor
Known Participant
April 24, 2006
Stiil on the write/delete instead of cfmail (for testing purpose), I replaced the update query by a "every 1,000 loop do not delete the file". It crashed at 418,000 loops.
same error.

- Why not cleaning the email before it's insertion in the database ? I already do it, but I want to be sure to avoid crashing the sending process because someone manualy (bad) changed an adress in the database, or everything else that can lead to a bad formated adress...
- splitting into chunks... what a shame !!! the gateways are designed for such tasks, that is to say long running queries. The asynchronous process is made to avoid the "blank loading page" of these long tasks. Do you mean I need to make a gateway that handle the whole process and that calls for example every thousands email another gateway that send the emails ?!?

One thing is funny : I choosed to use gateways to test them. For other emailing processes I already made, I made a form that lead to the creation of a schedule task. Then the task is a simple cfm page that does (almost) exactly the same code : loop over query, one cfmail per loop, one cfquery update every 100 loops. And it works like a charm (since cf mx 6.1... before 6.1 cfmail was crashing under load, the solution I chose was to create a txt file in the "pickup" directory of the IIS SMTP).
- which emails crashed : not a specific adress. For example, if it crashs after the 101,000th email, If I recall the gateway but these time specify to start at the 100,00th email, then the next crash will occur around 200,000 (not 101,000)
- the SMTP server in cause ? No, it crashes with cffile instead of cfmail too. It also handle lots of emails with the other solution I explained (schedule task)

I'm a bit confused... I will probably re-switch to the schedule task solution...
BKBK
Community Expert
BKBKCommunity Expert
Community Expert
April 24, 2006
More often than not, java.lang.OutOfMemoryError means too many objects at once. I first suspected the query, verif_envoi_nl. But since you don't store query-objects in an array or structure, the problem could not be coming from there.

However, the next suspect still has to do with the query. What happens when you replace THIS.datasource with just the name of the datasource? Elsewhere, is the CFC's timeout high enough?

O
obouillaudAuthor
Known Participant
April 24, 2006
I set the cfc AND gateway timeout to ... 24 hours (86400 seconds)

In the "gateway creation" cfm file, I put <cfset emaildata.timeout = 86400> then <cfset status = SendGatewayMessage("emailblaster", emaildata)>

And in the emailblaster CFC :
<cfsetting requestTimeOut = "86400" >
juste after the <cfcomponent ...> tag

The THIS.datasource is set just after the requestTimeOut at the beginning of the cfc.
<cfset THIS.datasource = Application.datasource>

in application.cfm, <cfset Application.datasource = "mydsn">

I changed and now use <cfset datatource = "mydsn"> at the beginning of the CFC, and "#datasouce#" in the queries.

No changes, crash exactly at the same "thousands" than before : 418,000

I attached the complete CFC (sorry, french comments, tab/space mess...)

PaulH,
The gateway + separate CFC for chunks, why not, but, maybe I will have to do something like that, but I'm not happy with this solution... as I want flexibility (the person could specify the origin of the email, for example the table name or a list of comma separated adresses, or ...) It leads to one more "gateway", I mean a cfc that pass datas to another cfc which do the job (sending the emails).
The UPDATE query is usefull to give real time feedback (another cfm page shows information of which table is used, which number is currently processed, and if the sending process is done or not)... and if it crashs, I'm sure to have information where the process stopped. For example, when the outofmemory exception occured, no cferror is thrown, so the code in the cfcatch tag is not executed... and I have no feedback where it crashed.
I use a table mainly because the server that sends the emails is not the main (web and back-office) server and I can't access to the disks of each other easily. Sure I could use log + a task that parse the log, but it's a bit heavy work... The fact is that it crash even without query and crash too with cffile.
N
Newsgroup_User
Inspiring
April 24, 2006
BKBK wrote:
> Cfloop is necessary, in any case. You should not use cfoutput around cfmail,

uh, why is it necessary?
BKBK
Community Expert
BKBKCommunity Expert
Community Expert
April 24, 2006
Cfloop is necessary, in any case. You should not use cfoutput around cfmail, because cfmail implicitly includes output functionality.

The rate at which the query object is re-created could be the problem. Although you've reduced the rate to every 100th, that could still be too fast for the Java Virtual Machine. Experiment with every 10000th and, if successful, every 1000th, and so on. Also try-catch the query code.


O
obouillaudAuthor
Known Participant
April 24, 2006
I tried... cfloop instead of cfoutput, and 2 cffile actions (write then delete) instead of cfmail... and it still crashs !

"Error","Thread-12","04/24/06","10:20:32",,"Error invoking CFC for gateway emailblaster: null"

with my "update every 100 loops" query, I can see that if crashed after the 417,200 th loop.
O
obouillaudAuthor
Known Participant
April 24, 2006
I try... but as I have the same problem with cffile (create and delete) instead of cfmail... I only have very little hope...
BKBK
Community Expert
BKBKCommunity Expert
Community Expert
April 24, 2006
<cfloop query="liste_mails">
BKBK
Community Expert
BKBKCommunity Expert
Community Expert
April 23, 2006
It might not be the cause of the problem, but it's something you have to correct. The cfmail tag already includes output functionality. You should therefore do instead

<cfloop query="liste_mails">
<cfmail from="#CFEvent.data.from#" to="#trim(stremail)#" subject="#CFEvent.data.sujet#" type="html">
#CFEvent.data.contenu_html#
</cfmail>
<!---- ... a bit of other stuff --->
</cfloop>

Then modify "a bit of other stuff " where necessary. Do you create any objects in a bit of other stuff ? The java.lang.OutOfMemoryError often results when the JVM is handling too many live objects.



O
obouillaudAuthor
Known Participant
April 24, 2006
here is the complete loop :

But even without the "other stuff" in the loop, it still crashs
Terms & ConditionsAccessibility statement
Using the community Terms of use Privacy Cookie preferences Do not sell or share my personal information ad choicesAdChoices