CFIMAP tag extremely slow, any way to speed up?

[tSpark]

I'm using ColdFusion 9's new CFIMAP tag to obtain a listing of messages in a Gmail Inbox.  However, it is EXTREMELY slow.  I am using the GetHeaderOnly method, and it is taking more than 60 seconds to return the list of messages.  There are only 53 messages in my Inbox.

<cfimap action="open" server="imap.gmail.com" username="#settings.gmailusername#" password="#settings.gmailpassword#" connection="imapInit" port="993" secure="true" />
<cfimap action="getHeaderOnly" connection="imapInit" name="headersQry" folder="Inbox"  />

If I add maxrows="5" to the getHeaderOnly call, it takes about 10 seconds.  How can it possibly be this slow?

I used a Java library before (http://www.webtrenches.com/post.cfm/connecting-to-gmail-using-coldfusion) and it was much faster.

Am I doing something wrong here, or is CFIMAP horribly slow?

[Fernis]

I've also had very varying results with CFIMAP. Sometimes it seems to dead slow, sometimes it works a bit better.

Today, when I first tried, it took me 70 seconds to retrieve 30 email headers. A bit later, I got a similar amount of headers in 14 seconds.

In any case, it seems painstakingly slow. I have no idea how fast IMAP servers typically output data, and I can't remember how fast e-mail clients retrieve the content of a large Inbox, but I do think CFIMAP is way slow compared to what it could be.

There are tricks, however, how to make the email retrieval more comfy for the user. You could, for example, implement a background processing for the email headers.

For every IMAP folder, your messages are indexed by an increasing integer number. CFIMAPs messagenumber parameter works for getHeaderOnly action also. Your oldest message in folder xxx is always numbered as "1".

If you do <cfimap action="getheaderonly" ... messagenumber="1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20"> - and there are 20 rows returned, you know it's worth searching for more. Next time you do ...messagenumber="21,22,23,24,25..." etc.

Between every cfimap action, you store the query results in a database (recommended for lesser memory usage) or return the query results in a CFC, or hold the query in a session object and join the queries, update a cfgrid, whatever. You can write a message to a screen, or update a progress bar, or whatever, to tell the user what's happening.

Message numbering is not failsafe, since the message numbers change on the fly if messages are moved/purged/deleted from the folder. But then there are message UIDs which can help you know where you were if the process got aborted, or what's the current last message number you fetched in the folder.

--

-Fernis - fernis.net - ColdFusion Developer For Hire

[tSpark]

Thanks, Fernis, for the suggestions.  I like the idea of passing numbers in, but what if I don't know how many messages are in the folder?  I need to display the 10 most recent messages, for example.  I would first have to do a folder list and obtain the number of messages, then count 10 from the end.  For example, 53,52,51....44.    I tried this, but now I have 15+ seconds to get a folder list, and 30 seconds to get the 10 messages.

Actually, the BlueDragon CFIMAP runs at more than double the speed of the ColdFusion one for me.  And the Java solution I mentioned is even faster than that.  I think I'll just stick with the Java solution for now, and keep an eye on the hotfixes and board comments for suggestions and improvements to CFIMAP.

[Fernis]

>but what if I don't know how many messages are in the folder?

Exactly why I'd do a thorough querying once, and store the results in a database - or at least some information of the latest message received - because you can't get the latest messages right away.

If you store the UID and the numeric ID of the last message you got, you can later get the header of only that message (unless it was moved or deleted 😉 by the UID, and check if the message number still matches - and continue from there.

If you have a working alternative for CFIMAP that's a heck of a lot faster, i.e. you're not running on a shared CF hosting with limited custom tags and stuff, I think you're doing the right thing sticking with your current solution.

-Fernis - fernis.net

[Adam Cameron.]

I tried a variation of your code and it took me 25sec to pull 19 headers from my gmail inbox.  Not good.

Can you post your code using Java libraries instead, so I can do an equivalent test with Java to see what the difference is like for me?

--

Adam

[tSpark]

Well, it is a little more involved than CFIMAP, but here is the code.

First, I use the imap.cfc as modified here:

http://www.webtrenches.com/post.cfm/connecting-to-gmail-using-coldfusion

Then, use this code to get the full Inbox:

<cfset imapCFC = CreateObject("component","imap") />
<cfset  imapInit = imapCFC.Init(username,password,"imap.gmail.com",993,15,1)  />
<!---  determine which message ids to get --->
<cfset msgs =  imapCFC.list("Inbox") />

That returns the query of messages.  If you want to limit it to 10 messages, you have to get more complex, like this:

<cfset limit = 10 />

<cfset msgcount = imapCFC.count("Inbox") />
<cfset msgIDs = "" />
<!--- determine which message ids to get --->                       
<cfif msgcount LTE limit>
<cfset from = 1 />
<cfelse>
<cfset from = msgcount-limit />                           
</cfif>
<cfset to = msgcount />                
<!--- build list of message ids --->
<cfloop from="#from#" to="#to#" step="1" index="i">
<cfset msgIDs = ListAppend(msgIDs,i) />
</cfloop>
<!--- get headers for message id list --->
<cfset msgs = imapCFC.list(foldername,msgIDs) />

Either way, it is more than twice the speed of CFIMAP.

[Adam Cameron.]

If I try your first example, my CF instance just redlines on the list() call.  And I've only got a coupla dozen items in my inbox, so doubt it's struggling with that.  Tried on CF8 & CF9.

It was substantially quicker opening the IMAP connection though... 20ms compared to 3000ms with <cfimap>.  But then again I'm not sure whether that's meaningful as it seems too fast to make a round trip to Google, authenticate me, and return confirmation.  I suspect it's actually just failing silently (I don't have time at present to delve into it more thoroughly to work out what might be going on, sorry).

I heard over the weekend of someone else having similar issues with <cfimap>, btw, so it really does seem like there's an issue here.

--
Adam