Issue with bulk email delievery

[dhanshreej63753772]

Hello Team,

These days I am facing weird issue with bulk emails delievery in my application. For example, if I am sending around 300 emails via cfmail tag, sometimes these all emails will go fine and sometimes some emails will stuck in my undelievered emails folder.

I end up sending these emails manually, sometimes it gets really bad when undelievered emails goes upto 2000.

Through error logs, I am able to see below error:

Nov 17, 2023 07:05:09 AM Error [mailWorker-9] - com.sun.mail.smtp.SMTPSendFailedException: 451 4.7.500 Access denied, please try again later.

Which is really weird, because rest all successfully delievered emails are sent via same server/using same credentials. Not sure, why it starts saying "Access denied" all of a sudden.

Looking forward to hear some possible solutions from community. Please feel free to comment in case you faced similar issue in your application.

Thanks,

Dhanshree Joshi

[Charlie Arehart]

This is not a cf error: your mail server is throttling you because you're hitting some rate limits it imposes. (This is about the mail server which you have cf pointing to, either in the cfmail tag or the cf admin. And to be clear, that's not the USER'S mail server, in the to address. It's instead like your local post office, which takes mail you send and delivers it ultimately to the recipient. CF knows nothing about that process. )

The key indicator is the 451 code in that error. If you Google:

smtp 451

You will find this discussed extensively (though again having nothing to do with cf).

Naturally you'll ask "what can I do?"

1) Your first choice should be to look into that mail server: maybe there's some configurability for that rate limit. Or maybe it will change if you register your cf server's ip address to be trusted. Or maybe you can pay some amount to be granted the ability to send more mail at once.

2) You could change mail server providers. Different ones have different limits. You could also configure a local smtp server instead, which suits many. There are several alternatives. More on finding alternatives in a moment. 

And FWIW, with CF Enterprise you can name multiple mail servers for cf to use. 

3) Most people in your shoes seek to somehow limit or manage how cf sends mail. I think that will prove challenging. You'll see people suggest you perform cfmail in a loop with a pause (cf's sleep function). But that will take care to really manage the mail spool well. (Note the cf admin mail page setting for the current interval for cf to send mail found in the spool.)

Some might suggest how cfmail has a spoolenable attribute to disable use of the spool entirely: with that set to false, each such cfmail then is sent immediately, without piling up in the spool--and the cfmail awaits the mail getting to that smtp server - again, NOT ultimate TO address destination.) 

Finally, no, there's nothing to tweak in the admin to "slow down down its sending mail out" . There's only that mail spool interval, which defaults to 15 seconds, and that merely controls how often it wakes up to look in the spool for mail to deliver. YOU control in code how much mail you put IN that spool.

Again, you're best off seeking an alternate solution outside of cf (my first two points). Indeed, finally, someone will want to tell you "don't use cfmail for bulk email". They'd propose you give that job to a real mail delivery service (there are many, like mail chimp, constant contact, and more). Those don't suit everyone.

FWIW, I list both such mail services and alternate smtp mail servers (discussed earlier here) as categories in my cf411.com site. 

[sdsinc_pmascari · Accepted solution]

We've run into similar situations over the the years and have employed many of the things Charlie mentioned.  But, nothing completely found a balance between the number of emails we needed to send and the rate limits we were hitting.

In our situation, we only ran into the rate limit issue early in the morning when we're sending out our daily 'newsletters' to 1000s of clients.  What we ended up doing was writing a small scheduled task that runs every few minutes and actually moves all the mail in ColdFusion's 'undeliverable' folder back into the spool folder where it gets processed again within a few seconds.  We use a small database to keep track of how many times each message has been moved.  After the third try, we delete it.  That has worked for us, like I said, in combination with similar steps Charlie has laid out.

[Charlie Arehart]

Yep, thanks Paul. I should have thought to anticipate that as another idea some have come up with. 🙂 The note was already long and I was writing on a phone. 

As you suggest, one needs to be careful in order to do that respooling process well. As you've coded a solution, you'd probably find an enthusiastic sub-community of folks (users and perhaps contributors) if you offered that as a open source solution. I realize doing that takes time and effort which you may not be able to offer. 

[rickmaz]

We have had that problem for the reasons you mentioned but also just random reasons like your mail server was blipping or being rebooted or under maintenance at the time you were sending mail. Whatever, they sit in the undeliverable folder, and if you don't manually check you have no idea. So what we did was the following:

1. Since most issues will be resolved in our application and experience by simply resending we have code that checks the folder and sees if anything is in there first like every 2 hours. If it does we will move it back to the spool folder (via cf code). If at the end of the day or whenver there is still mail in there then something keeps failing and this could be for many reasons so we want to inspect.

2. So at the end of the day we run another cf script that alerts us only if there is anything in folder, outputs the email content so we can see what the issue may be, and can either fix it and save (all through cf code) or we can retry the entire spool or delete the message if it was just something that was a bad email or not needing to be sent. 

So we basically never have to log in to the cf server and it has worked out fairly well. 

[Charlie Arehart]

Rick, this is essentially what Paul had proposed. But I'm curious to hear from both of you: while the general idea of respooling makes sense (if handled carefully, like I said to Paul), are you both confirming that your files are ending up in the undelvr folder because the mail.log shows them getting this same 451 code that Dhanshree showed to be getting?

I only ask because sure, there can be problems like network issue where somehow CF is not able to REACH the smtp server (which would also lead to the mail failing and ending up in that folder), and perhaps respooling it later will allow it to go out when tha tnetwork problem is resolved.

And it's also possible that you mean you ARE getting these same 451 errors, so your respooling simply allows your mail to go out "later" when the rate throttling timeframe from your mail server has passed. But I would just caution that if you're NOT getting 451 errors, then having someone do this respooling may only exacerbate their problem--with the flush of respooled emails contributing again to their reaching (or nearing) that rate handling limit for the mail server.

Again, not knocking the idea in general--just offering a warning that if you (Paul and Rick) are NOT getting the 451 errors, then you were solving a different problem, and it's not clear that your solution will always help someone who is. 🙂

As always, just trying to help, and to contribute to the general understanding of the problem, if the additional thoughts may do that. 

[rickmaz]

Mine has happened for that reason on occasion but with much bigger lists than 300 but it has also happened due to the mail server being unavailable or overworked and other reasons.  And you are correct, network issues don't happen enough to really be able to diagnose them as it is so infrequent. I don't have a major problem with this but it has happened where months have gone by and then I look and there are several in there from random times i resend them and it works but those people never got their mail. Obviously, if a few resends are retried and they keep failing then there is another issue. Yes if the mail server is rejecting it for some other reason that needs to be looked into of course. Our actual main server has throttling on its end to slow down the delivery to places like gmail.com and others that don't like bulk but yes everyone's environment is different. I was just offering one option to consider but you are right without knowing all the endpoints and how they are configured one can not offer a simple one-stop-shop solution without deeper digging. 

[sdsinc_pmascari]

The overwhelming majority of our issues were the 451 errors, yes, so simply respooling them allows them to "eventually" get sent in our case. 

[Charlie Arehart]

Thanks for confirming, Paul. And for your additional thoughts, Rick.

Let's see how things may go for Dhanshree.

[dhanshreej63753772]

Thanks for your valuable inputs guys!!!

@Charlie Arehart Yes, I am aware about mail sending limits on my mail server. As I mentioned, in my situation sometimes at same milisecond email will deliever fine, and next might stuck into undeliver emails. So, if there's any limit issues then all the emails should stuck on server one after another and none of next should get deliever. To add more, we have 5000 emails limit per hour set on our mail server(as I mentioned there are cases where 300 emails also cannot deliver successfully).
Also, SMTP Access denied error is also weird. Not sure why suddenly it cannot access SMTP server to send emails.
So, seems solutions mentioned by @sdsinc_pmascari and @rickmaz  are only possible workarounds here 😞

Thanks once again!

[Charlie Arehart]

So are you suggesting that you think instead cf has some bug? And do you think the 451 error is coming from cf? It is not. It's coming from your mail server, the one named in the generated mail. 

If you really want to hang your hat on it being seemingly inexplicable that "some get out and some do not", maybe the mails are being told to go to different smtp servers. It's possible: again, this is set in the admin but can be overridden on cfmail.

You can easily check the email files in question (literally open the files as shown in the undelivr folder: they're plain text and tiny). Each will show in one of its first lines the name of the mail server being sent to. Maybe you'll find there's some unexpected variation in the server names used. I've seen that happen to folks before.

But if it's not so, and you think the automated respool approach will work for you, I hope you'll keep us posted on how things go. 

[Dave Watts]

sometimes at same milisecond email will deliever fine, and next might stuck into undeliver emails. So, if there's any limit issues then all the emails should stuck on server one after another and none of next should get deliever

Actually, this can easily happen with any mail server not designed for bulk mail delivery: M365 mail delivery, Google Workspace mail delivery, etc. They can - and will - block a random and arbitrary number of mail messages you send from CF. You need an SMTP server capable of handling bulk mail delivery, or you need to fiddle around with CF so that it will retry messages until they eventually go through. I find having a locally-installed SMTP server the best way to handle this. If you're running Windows Server (not sure which versions) you can install the IIS SMTP server, which is not actually part of the latest IIS version. You configure CF to send messages to that, and that will handle the upstream delivery part for you. It will try and retry and retry as many times as needed.

Alternatively, you can configure CF to periodically retry everything in the UNDELIVR folder, by moving it back into the SPOOL folder. This assumes you have mail spooling enabled, which you probably should and do. I haven't done this in a while, but it should eventually deliver everything. I have no idea how long it'll take though. The SMTP services in M365 and Google Workspace aren't intended for bulk mail delivery and will reject a lot of messages more-or-less randomly.

Of course, you could have multiple SMTP servers, etc, but I suspect you don't. Look at that anyway, in addition to the things I mentioned.

Dave Watts, Eidolon LLC

[carl type3]

Hello Dhanshree,

Perhaps I am a bit late on this thread and you may have some work around in place. 

I have seen similar from O365 allow rule failing with SMTP 451 response to CF bulk emails. The work around I use is to create a CFM file that moves the mail\undelivr content into mail\spool then CF schedule that to run every 30 minutes.

Regards, Carl.

[dhanshreej63753772 · Accepted solution]

Thanks for all the responses guys!

I have implemented the job to spool back undelivered emails as suggested before in this thread. It's working smooth and no need to check undelivered emails manually/spool them back[saving now lot of time].

Thank you once again!

[Charlie Arehart]

Glad you feel you solved it and though you marked Rick's as "the answer", it's worth noting that Paul (pmascari) had posed it first, then also Dave and Carl after Rick. And really, your answer is worth marking as an "answer" also since it offers the conclusion you came to. 

More important perhaps, and given that this situation exists and such hand-crafted solutions can help, it surely suggests this to be something that Adobe might want to build into cf... or as I'd noted in reply to Paul's first mention, maybe it would at least make a good open source project.

If any of you who coded a solution (including Dhanshree) might want to offer a first draft as a new github project, others would benefit and could contribute--or they may offer their own variant in the same or another repo.

If either of you may create such a project in github or elsewhere, please do share the link here. I'm sure this thread will get traffic for years to come, as more people get 451 errors due to mail server throttling, or have other temporary hangups in mail processing.

Respooling the mail by hand is tedious, and the cf admin ui feature barely improves it. A real app to manage the process well (see tips shared by many in this thread) could be a boon to many. 

[lavis_5366]

It looks like you're facing intermittent email delivery issues with the 451 4.7.500 Access denied error. Here are some quick fixes:

1. Check Sending Limits: Ensure you’re not exceeding your server’s sending limits.
2. IP Reputation: Your IP might be blacklisted; use tools like MXToolbox to check.
3. SMTP Configuration: Review settings for any misconfigurations or blocks.
4. Retry Mechanism: Implement automatic retries for undelivered emails.
5. Dedicated SMTP: Consider switching to a dedicated SMTP server to improve deliverability.

Some reliable SMTP services:

• SMTPmart
• SMTPget
• SendGrid
• iDealSMTP
• Amazon SES

These can help improve email delivery and avoid blocks.