CF 9 schedule task emails running twice

[jk_glass]

Running CF 9,0,1,274733 and I must qualify my post by saying that I know enough about CF to be dangerous.

We have a number of schedule tasks that run once per day at different times of the day. These tasks do things like query the database and email staff about low on stock items and almost expired items, etc. We installed some critical security updates on our Win 2008r2 server a few days ago and the double emails started that night. I don't want to uninstall those security updates to fix this issue.

Here are some things I have checked and tried:

a) I cleared the cache in CF Admin.

Clear Template Cache Now

Clear Component Cache Now

Still getting dupes

b) Restarted the CF services in the services.msc in Windows. Still getting dupes.

c) Restarted the server. Still getting dupes.

d) I ran a few of the tasks manually by clicking the Run button in the CF Admin Schedule Task list. Still getting dupes.

e) I added the spoolenable="no" attribute to the cfmail tag. Still getting dupes.

f) I recreated some of the tasks and the cfm pages that they point to. I turned the old tasks off by setting them to "One-Time". Still getting dupes.

g) I created a new task and a new cfm page with just the cfmail tag. No cfquery, no cfloop, no cfoutput. Just the cfmail tag with 1 sentence in the body. Still getting dupes.

h) I changed email addresses in the cfm pages. Still getting dupes.

i) I checked the mailsent.log. It shows these task pages are being fired twice about 1-2 minutes apart. If the task is set to run at 11:00pm then it runs at 11:00pm and at 11:01pm. It is also runs twice and 1 minute apart when I click the Run button in the task list. This looks like a CF issue and not an Exchange issue.

I hope someone can point me in the right direction. Thanks.

[Charlie Arehart]

(Edit. I was not done writing my reply here when I hit ctrl-s out of habit, and the web interface for these forums posted my unfinished reply. Sorry for the confusion if any readers got emailed that incomplete version and not this correctly edited one)

@JK, often when folks find scheduled tasks executing more than once, it's that some other server is what's running the "duplicate".  I realize you may say "that's not it". People often say that. 🙂

But here's one thing you can do to prove it: you mention that you're doing some test code in a scheduled task which does a CFMAIL, which you see being sent twice. Let us know what you see in the email if you do a cfdump of the cgi scope, especially the remote_addr field, which tells you the ip address of the "client" sending the request. It could be that you'd find the two emails would show the requests coming from different machines.

Then again, they could show being from the same machine (IP address) but if you're running the Enterprise edition with multiple instances, they'd show being from the same IP address.

You may also find that the "duplicates" you see are due to the request actually running on different machines. In that case, if you could look at your web server logs you could see if you see the request really running twice in the short term that you have observed from your "saved output".

That said, you mentioned something in one particular attempted proof case that I find curious: You said in "g" that "I created a new task and a new cfm page with just the cfmail tag. No cfquery, no cfloop, no cfoutput. Just the cfmail tag with 1 sentence in the body. Still getting dupes." That would seem to indicate that this is a new task on a single machine and therefore would seem to disprove the possibility I proposed above.

Still, do let us know what you found if you'll try what I've proposed, and maybe we can take it further from there.

[jk_glass]

Thank you for your response Charlie. I will study your post and try to look at some of these items in the upcoming days. We are using the CF9 Standard edition on one server with one web site.

For the time being I'm using a work around solution I created with the Windows Task Scheduler but I would like to use the CF Scheduler again some day. I turned off all the tasks in the CF scheduler. I created vb scripts to call the cfm task pages and I'm using the Windows Task Scheduler to call the scripts at the correct time. There is info on Technet and elsewhere about calling web pages with the task scheduler.

One interesting thing to note about this work around fix is it doesn't work on the server with CF. I first tried calling a test cfm page with Windows Task Scheduler on the 2008r2 box with CF installed and it still duplicated. I could run it manually or on a schedule and it duplicated. So there is something going on with this server.

Then I ran the test task on a Windows workstation on our network and it ran 1 time like it was supposed to do. Then I ran the test task on another 2008r2 box (no CF installed) and it ran 1 time. Now all the CF tasks are running on this Windows server (no CF installed) with Windows Task Scheduler and they have been running correctly for the last week. It looks like the 2008r2 box without CF got the same security updates that the 2008r2 box with CF got.

My "g" task described in my first post will duplicate with the CF Scheduler or the Windows Scheduler on the server with CF. But this "g" task and all the other CF tasks run correctly from any other box with the Windows Scheduler. I need to study this issue a little more.

[jk_glass]

I tried the cfdump tag as Charlie suggested and the remote_addr field was different in my duplicate emails. I placed the cfdump tag in a basic cfmail page and ran it as a task in the CF Admin. Just like usual I got 2 emails about 1 minute apart.

The remote_addr field in the first email was 2xx.xxx.xxx.xx. This is the IP address of the web site.

The remote_addr field in the second email was 1yyy.yyy.yyy.yy. I don't know where this IP came from?

We are running one CF server with one web site. How can these requests be coming from different machines? How do I troubleshoot where the second email/task is coming from?

[Charlie Arehart]

@jk, sorry I missed this last week.

So it’s “good” (diagnostically, that is) to see confirmation of what I suspected: that your “duplicate” scheduled tasks are indeed coming from two places.

I realize you’re now saying you don’t recognize where those “places” are. For the second one (which you don’t recognize), you’ve given us only “1yyy.yyy.yyy.yy”, which I realize the y’s just mean “some number there” and you’re reluctant to share it in case it may be a security risk. But tell us this: are the first quadrant’s numbers 10, 172 or 192? Those each (may) begin private addresses, meaning addresses that are at least “within your network”, which may be useful to know. (For more details on why those only “may” start private IPs, see https://www.arin.net/knowledge/address_filters.html).

Either way, you need to track down where the request is coming from. Here’s some good news: assuming the requests are not coming into CF’s built-in web server, you could look in the web server access logs (IIS or Apache) to find any other requests from that IP address, which may help you connect dots as to where it’s coming from. (You could also confirm that the “user agent” for the requests for the 2nd IP are the same as that for the 1st. CF reports a distinctive user agent if it’s indeed a scheduled task making the request.)

In my experience, when people find a scheduled task running twice, it’s that the scheduled task is simply listed in a CF admin they did not expect (and the second one still points specifically to a URL that gets to the first machine). It could be an old machine that was repurposed for new use, but the schedule tasks were not changed. It could be some sort of replicant partner (where someone has files or even CF settings being replicated between two servers). It could be a dev/test/qa machine where someone set up the sched tasks to be tested first, and so on.

Do let us know how you get on.

/charlie

[jk_glass]

Thanks for all the info Charlie. This issue is not solved yet but we might be getting closer.

Both IP addresses are public and I know more about the 2nd IP but I don't know why it is being used. The remote_addr field in the first email has the IP of our web site. The remote_addr field in the second email has the public IP of the anti-virus company we use. I don't understand this. We have a paid subscription to Trend Micro.

Did the security updates we installed last month cause Trend Micro to start interfering with ColdFusion or vice versa?

The user agents are different. I'm guessing you are talking about the HTTP_USER_AGENT in the cfdump output? The first email user agent is "CFSCHEDULE". The second email user agent is "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0)".

The tasks that I'm running with vb scripts on the task scheduler of another server are still running well. These tasks call the CF pages on the server with CF installed and there are no duplicates. But I have been running my test page just from the server with CF and it still duplicates.

I'm still studying the logs and cfdump output and I will post an update if find out more info.