Highlighted

throttled CFHTTP

New Here ,
Sep 10, 2012

Copy link to clipboard

Copied

i recently upgraded from cf9 to cf10. running single instance on a brand new ubuntu server 12.04 with Apache. same settings as my other ubuntu servers as far as i can tell. Suddenly, all my cfhttp tags are being throttled with about 5 second breaks being pushed into each one. Each additional request made pushes the time taken up 5 seconds. i've compared every setting i could think of to the cf9 servers but nothing worked.

Here's from the http.log:

HTTP request completed {Status Code=200 ,Time taken=28525 ms}
Sep 10, 2012  5:13 PM  Information  ajp-bio-8012-exec-18 
HTTP request completed {Status Code=200 ,Time taken=22752 ms}
Sep 10, 2012  5:13 PM  Information  ajp-bio-8012-exec-23 
HTTP request completed {Status Code=200 ,Time taken=17549 ms}
Sep 10, 2012  5:13 PM  Information  ajp-bio-8012-exec-17 
HTTP request completed {Status Code=200 ,Time taken=12320 ms}
Sep 10, 2012  5:13 PM  Information  ajp-bio-8012-exec-22 
HTTP request completed {Status Code=200 ,Time taken=7137 ms}
Sep 10, 2012  5:13 PM  Information  ajp-bio-8012-exec-25 
HTTP request completed {Status Code=200 ,Time taken=4302 ms}
Sep 10, 2012  5:13 PM  Information  ajp-bio-8012-exec-23 
Starting HTTP request {URL='http://www.google.com', method='GET'}
Sep 10, 2012  5:13 PM  Information  ajp-bio-8012-exec-17 
Starting HTTP request {URL='http://www.google.com', method='GET'}
Sep 10, 2012  5:13 PM  Information  ajp-bio-8012-exec-22 
Starting HTTP request {URL='http://www.google.com', method='GET'}
Sep 10, 2012  5:13 PM  Information  ajp-bio-8012-exec-19 
Starting HTTP request {URL='http://www.google.com', method='GET'}

Any help would be greatly appreciated.

Thanks!

We investigated this and found out that this does happen on some Linux machines but not all. The reason it happens is because Apache HttpClient library creates a SecureRandom number for every http call which can be slow on new Unix boxes as there are not enough entropy generated on those machines.

There are few possible ways to solve this and I would recommend either of these two.

  1. 1. Set this system property to your JVM – if you are using standalone CF installation, you would set it in jvm.config.

“-Djava.security.egd=file:/dev/./urandom”

  1. 2. In $JAVA_HOME/jre/lib/security/java.security file, change the value of securerandom.source to file:/dev/./urandom

You can also refer to this excellent post by Shilpi from the ColdFusion team which talks about this issue

http://www.shilpikhariwal.com/2012/04/random-number-generation-in-unix.html

Views

5.5K

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more

throttled CFHTTP

New Here ,
Sep 10, 2012

Copy link to clipboard

Copied

i recently upgraded from cf9 to cf10. running single instance on a brand new ubuntu server 12.04 with Apache. same settings as my other ubuntu servers as far as i can tell. Suddenly, all my cfhttp tags are being throttled with about 5 second breaks being pushed into each one. Each additional request made pushes the time taken up 5 seconds. i've compared every setting i could think of to the cf9 servers but nothing worked.

Here's from the http.log:

HTTP request completed {Status Code=200 ,Time taken=28525 ms}
Sep 10, 2012  5:13 PM  Information  ajp-bio-8012-exec-18 
HTTP request completed {Status Code=200 ,Time taken=22752 ms}
Sep 10, 2012  5:13 PM  Information  ajp-bio-8012-exec-23 
HTTP request completed {Status Code=200 ,Time taken=17549 ms}
Sep 10, 2012  5:13 PM  Information  ajp-bio-8012-exec-17 
HTTP request completed {Status Code=200 ,Time taken=12320 ms}
Sep 10, 2012  5:13 PM  Information  ajp-bio-8012-exec-22 
HTTP request completed {Status Code=200 ,Time taken=7137 ms}
Sep 10, 2012  5:13 PM  Information  ajp-bio-8012-exec-25 
HTTP request completed {Status Code=200 ,Time taken=4302 ms}
Sep 10, 2012  5:13 PM  Information  ajp-bio-8012-exec-23 
Starting HTTP request {URL='http://www.google.com', method='GET'}
Sep 10, 2012  5:13 PM  Information  ajp-bio-8012-exec-17 
Starting HTTP request {URL='http://www.google.com', method='GET'}
Sep 10, 2012  5:13 PM  Information  ajp-bio-8012-exec-22 
Starting HTTP request {URL='http://www.google.com', method='GET'}
Sep 10, 2012  5:13 PM  Information  ajp-bio-8012-exec-19 
Starting HTTP request {URL='http://www.google.com', method='GET'}

Any help would be greatly appreciated.

Thanks!

We investigated this and found out that this does happen on some Linux machines but not all. The reason it happens is because Apache HttpClient library creates a SecureRandom number for every http call which can be slow on new Unix boxes as there are not enough entropy generated on those machines.

There are few possible ways to solve this and I would recommend either of these two.

  1. 1. Set this system property to your JVM – if you are using standalone CF installation, you would set it in jvm.config.

“-Djava.security.egd=file:/dev/./urandom”

  1. 2. In $JAVA_HOME/jre/lib/security/java.security file, change the value of securerandom.source to file:/dev/./urandom

You can also refer to this excellent post by Shilpi from the ColdFusion team which talks about this issue

http://www.shilpikhariwal.com/2012/04/random-number-generation-in-unix.html

Views

5.5K

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Sep 10, 2012 0
New Here ,
Nov 08, 2012

Copy link to clipboard

Copied

I am also seeing this. Tcpdump shows that it isn't the web server slowing things down. It does appear to be in the cfhttp tag. Did you find an answer to this?

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Reply
Loading...
Nov 08, 2012 0
New Here ,
Nov 08, 2012

Copy link to clipboard

Copied

unfortunately I never figured this out. In the meantime I downgraded to CF 9 until I either figure it out or there's an update that fixes the problem. Maybe it should be submitted as a bug request...

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Reply
Loading...
Nov 08, 2012 0
Advocate ,
Nov 08, 2012

Copy link to clipboard

Copied

Maybe?

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Reply
Loading...
Nov 08, 2012 0
New Here ,
Nov 08, 2012

Copy link to clipboard

Copied

by all means, submit it. would be much appreciated!

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Reply
Loading...
Nov 08, 2012 0
New Here ,
Nov 15, 2012

Copy link to clipboard

Copied

Were you able to resolve this?

It's definitely a bug as far as we can tell.

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Reply
Loading...
Nov 15, 2012 0
New Here ,
Nov 15, 2012

Copy link to clipboard

Copied

unfortunately I have not resolved this. I'm glad to know other people are having the issue though so at least I know it's not related to my own specific server configuration.

I reported it to bugbase. If people vote it up there maybe we can get a fix going so I can upgrade finally.

https://bugbase.adobe.com/index.cfm?event=bug&id=3366102

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Reply
Loading...
Nov 15, 2012 0
New Here ,
Nov 15, 2012

Copy link to clipboard

Copied

Voted, and added a brief note on my OS being RHEL

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Reply
Loading...
Nov 15, 2012 0
Adobe Community Professional ,
Nov 16, 2012

Copy link to clipboard

Copied

I have added the following note to the bug report. This may sound crazy, but I've seen weirder things in ColdFusion. It just might be that what you observe is expected behaviour, assuming single-threaded processing.

Say, you have 10 cfhttp tags in a loop. When the loop starts, ColdFusion first processes a number of cfhttp start-tags, perhaps up to all 10 of them, possibly without strictly respecting order. Since the processing is single-threaded, ColdFusion executes the tag bodies in series, one after the other. This would explain why the successive execution times are cumulative.

See Ben Nadel's blog on using cfthread with cfhttp. There he solves the problem by using multiple, asynchronous threads, like this:

<cfloop index="intGet" from="1" to="10">

<!--- Start a new thread for this CFHttp call. --->

<cfthread action="run" name="objGet#intGet#">

<cfhttp method="GET" url="some_url" result="THREAD.Get#intGet#" />

</cfthread>

</cfloop>

<!---

Now, we have to wait for all of concurrent threads to be joined before we can use the CFHttp results.

--->

<cfloop index="intGet" from="1" to="10">

<cfthread action="join" name="objGet#intGet#" />

</cfloop>

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Reply
Loading...
Nov 16, 2012 0
New Here ,
Nov 16, 2012

Copy link to clipboard

Copied

the only thing is that the exact same code runs perfectly fine in a single-threaded process on CF9. it was only after upgrading that we noticed this. I'd rather not have to go into all of the code and modify it just because something changed on the CF side.

Additionally, while in the log example I posted I was using a single threaded, single request to initiate multiple CFHTTP calls, this problem actually manifested on our site when multiple users began accessing some of our pages that make use of CFHTTP.

However, just so no one can accuse me of not willing to try to find a workaround, I took Ben's code from this page: http://www.bennadel.com/blog/749-Learning-ColdFusion-8-CFThread-Part-II-Parallel-Threads.htm and tested on a CF9 and CF10 configuration.

.

Here are the results from the servers:

CF9: We Got 1000 Results in 0.72 seconds using CFHttp and CFThread

CF10: We Got 1000 Results in 45.02 seconds using CFHttp and CFThread

So it doesn't really seem like parallel threads would help out here.

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Reply
Loading...
Nov 16, 2012 0
Adobe Community Professional ,
Nov 16, 2012

Copy link to clipboard

Copied

45 to 0.7 is indeed a big ratio. That does not necessarily rule out the effect of single versus parallel threads. My guess is that some other factor in CF 10 is forcing single-threading.

To start with, perhaps these settings in the Administrator:

"Maximum number of simultaneous Template requests

The number of CFML page requests that can be processed concurrently. Use this setting to increase overall system performance for heavy load applications. Requests beyond the specified limit are queued.

Maximum number of threads available for CFTHREAD

The maximum number of threads created by CFTHREAD that will be run concurrently. Threads created by CFTHREAD in excess of this are queued."

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Reply
Loading...
Nov 16, 2012 0
New Here ,
Nov 16, 2012

Copy link to clipboard

Copied

I have a number of other pages that make use of parallel threads that have no issues. I've tuned the settings so that there are more than adequate numbers of threads available.

Even if it was forcing single threaded processing this doesn't make any sense since the CFHTTP is not taking 5 seconds to respond. If I limit the page to a single CFHTTP request I get:

Single request: Results in 1.16

If threads were the issue, 10 requests should take no more than 15 seconds.

For some reason, making two or more subsequent requests throttles the second and on with an extended delay.

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Reply
Loading...
Nov 16, 2012 0
Adobe Community Professional ,
Nov 16, 2012

Copy link to clipboard

Copied

shudini1 wrote:

However, just so no one can accuse me of not willing to try to find a workaround, I took Ben's code from this page: http://www.bennadel.com/blog/749-Learning-ColdFusion-8-CFThread-Part-I I-Parallel-Threads.htm and tested on a CF9 and CF10 configuration.

.

Here are the results from the servers:

CF9: We Got 1000 Results in 0.72 seconds using CFHttp and CFThread

CF10: We Got 1000 Results in 45.02 seconds using CFHttp and CFThread

So it doesn't really seem like parallel threads would help out here.

Something is definitely wrong. My results are:

We Got 1000 Results in 4.95 seconds using standard CFHttp

We Got 1000 Results in 0.89 seconds using CFHttp and CFThread

They look like the kind of results one would expect. I am on CF 10.0.4.283281 (Update 4 of CF 10). Would the updates have patched something?

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Reply
Loading...
Nov 16, 2012 0
New Here ,
Nov 19, 2012

Copy link to clipboard

Copied

I updated this morning to 10,0,4,283281 but the results have not changed:

Single request: Results in 1.14

We Got 1000 Results in 49.28 seconds using CFHttp and CFThread

Here's the code I'm using to test:

<cfset strBaseURL = ("http://www.google.com/search?q=Vin+Diesel&num=100&start=) />

<!--- Get the starting time. --->

<cfset intStartTime = GetTickCount() />

<cfhttp method="GET" url="#strBaseURL#"/>

<cfoutput>

          Single request: Results in

          #NumberFormat(((GetTickCount() - intStartTime) / 1000),",.00")#

</cfoutput>

<cfset intStartTime = GetTickCount() />

<cfloop index="intGet" from="1" to="10" step="1">

          <!--- Start a new thread for this CFHttp call. --->

          <cfthread action="run" name="objGet#intGet#">

               <cfhttp method="GET" url="#strBaseURL##((intGet - 1) * 100)#"

                    useragent="#CGI.http_user_agent#"

                    result="THREAD.Get#intGet#" />

          </cfthread>

</cfloop>

<cfloop index="intGet" from="1" to="10" step="1">

          <cfthread action="join" name="objGet#intGet#" />

</cfloop>

<cfoutput>

          <!--- Output retrieval times. --->

          <p>We Got 1000 Results in

                #NumberFormat(((GetTickCount() - intStartTime) / 1000),",.00")#

               seconds using CFHttp and CFThread</p>

</cfoutput>

(side point: i don't think google is actually returning the results as they do not permit scraping this way, but it demonstrates the point regardless that the requests are being throttled)

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Reply
Loading...
Nov 19, 2012 0
New Here ,
Nov 19, 2012

Copy link to clipboard

Copied

forgot to post the logs for the above code (running the updated CF):

DateTimeSeverityThreadIDApplication Name
Nov 19, 2012  9:40 AM  Information  cfthread-29 
HTTP request completed {Status Code=200 ,Time taken=48882 ms}
Nov 19, 2012  9:40 AM  Information  cfthread-30 
HTTP request completed {Status Code=200 ,Time taken=43850 ms}
Nov 19, 2012  9:40 AM  Information  cfthread-15 
HTTP request completed {Status Code=200 ,Time taken=38868 ms}
Nov 19, 2012  9:40 AM  Information  cfthread-31 
HTTP request completed {Status Code=200 ,Time taken=33949 ms}
Nov 19, 2012  9:40 AM  Information  cfthread-35 
HTTP request completed {Status Code=200 ,Time taken=28826 ms}
Nov 19, 2012  9:39 AM  Information  cfthread-33 
HTTP request completed {Status Code=200 ,Time taken=23890 ms}
Nov 19, 2012  9:39 AM  Information  cfthread-34 
HTTP request completed {Status Code=200 ,Time taken=18835 ms}
Nov 19, 2012  9:39 AM  Information  cfthread-36 
HTTP request completed {Status Code=200 ,Time taken=13827 ms}
Nov 19, 2012  9:39 AM  Information  cfthread-32 
HTTP request completed {Status Code=200 ,Time taken=8878 ms}
Nov 19, 2012  9:39 AM  Information  cfthread-28 
HTTP request completed {Status Code=200 ,Time taken=3877 ms}

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Reply
Loading...
Nov 19, 2012 0
LEGEND ,
Nov 19, 2012

Copy link to clipboard

Copied

It's not much help, but I ran a few tests and I don't see what you're seeing on my CF10 install: all CFHTTP requests are being fired immediately, and the length of time they take to complete is roughly the same per request. IE: I'm not seeing anything I would not expect to see if the thing was working properly.

My CF10 install is a bog-standard one, patched to 10.0.4.  CF10 Developer, running on Windows 7 Home 64-bit.

--

Adam

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Reply
Loading...
Nov 19, 2012 0
New Here ,
Nov 19, 2012

Copy link to clipboard

Copied

So this to me helps narrow down the problem. You are saying it runs fine for you on windows. So far the only people who have reported issues are on Linux.

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Reply
Loading...
Nov 19, 2012 0
New Here ,
Nov 19, 2012

Copy link to clipboard

Copied

I can also report that 10,0,5,283319 doesn't solve this issue for me.

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Reply
Loading...
Nov 19, 2012 0
Adobe Community Professional ,
Nov 20, 2012

Copy link to clipboard

Copied

psu_josh wrote:

I can also report that 10,0,5,283319 doesn't solve this issue for me.

Not that this would matter. 10.0.5 is a Windows security update. As you say, the bottle-neck is likely a Linux issue. So what is it on Linux systems that is queueing up the threads?

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Reply
Loading...
Nov 20, 2012 0
Adobe Employee ,
Nov 20, 2012

Copy link to clipboard

Copied

We investigated this and found out that this does happen on some Linux machines but not all. The reason it happens is because Apache HttpClient library creates a SecureRandom number for every http call which can be slow on new Unix boxes as there are not enough entropy generated on those machines.

There are few possible ways to solve this and I would recommend either of these two.

  1. 1. Set this system property to your JVM – if you are using standalone CF installation, you would set it in jvm.config.

“-Djava.security.egd=file:/dev/./urandom”

  1. 2. In $JAVA_HOME/jre/lib/security/java.security file, change the value of securerandom.source to file:/dev/./urandom

You can also refer to this excellent post by Shilpi from the ColdFusion team which talks about this issue

http://www.shilpikhariwal.com/2012/04/random-number-generation-in-unix.html

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Reply
Loading...
Nov 20, 2012 0
New Here ,
Nov 21, 2012

Copy link to clipboard

Copied

Using option #1 from above seems to have fixed the issue for me. I added it to the JVM arguments in CF admin.

New results:

Single request: Results in 1.96

We Got 1000 Results in 0.93 seconds using CFHttp and CFThread

Thank you VERY much!

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Reply
Loading...
Nov 21, 2012 0
New Here ,
Sep 09, 2014

Copy link to clipboard

Copied

Hi Shudin,

I tried above solution doesn't work. I am on CentOS 6. What OS you are testing this?

Thanks.

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Reply
Loading...
Sep 09, 2014 0
BKBK LATEST
Adobe Community Professional ,
Sep 10, 2014

Copy link to clipboard

Copied

@cfmcoder

You should start your own thread, as this one was last active almost 2 years ago. People, and Coldfusion, have moved on.

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Reply
Loading...
Sep 10, 2014 0