CF9 CPU at 100%. Moved to new CF11 server and copied files over and it's still at 100% CPU almost immediately on new server

[mike42780]

Our web server is stuck at 100% CPU.  Three days ago we were on CF9 and started running into this problem until the server crashed and wasn't salvageable.  Luckily we had a fresh Windows 2008 Server R2 ready to go with a fresh copy of CF11 on it.  I copied all the website files over and we were back up and running.  A day later the CPU is back to 100% on the new CF11 Windows 2008R2 server.  I also updated CF11 to the latest update 3 that was just released.

If I turn off the CF service the CPU usage goes back to normal.  If I turn CF back on, the CPU goes back to 100% within like 5 seconds.  So it doesn't seem like some slow running page or anything that eventually eats up all the memory or whatever.  I'm not an expert at looking at the logs, but I don't see anything too out of the ordinary.  The one thing that looks strange to me is I see this line over and over.

Dec 12, 2014 11:28:41 AM Information [ajp-bio-8014-exec-59] - Starting HTTP request {URL='http://zzen1wbudopwg.nchyt.com/encfm/en0024-ssj5iway6wvg/cbeim94a1s2kebu.php?do=194', method='get'}

That line makes me think we got hacked somehow, but it's hard to know.

I did up my JVM heap size and max size to 1024MB.  We have 4GB of memory.  And i changed teh JVM arguments to what's below based on a forum post.  Nothing seems to help.

-server -XX:PermSize=192m -XX:MaxPermSize=192m -XX:+UseParallelGC -Xbatch -Dcoldfusion.home={application.home} -Dorg.eclipse.jetty.util.log.class=org.eclipse.jetty.util.log.JavaUtilLog -Duser.language=en -Dcoldfusion.rootDir={application.home} -Dcoldfusion.libPath={application.home}/lib -Dorg.apache.coyote.USE_CUSTOM_STATUS_MSG_IN_HEADER=true -Dcoldfusion.jsafe.defaultalgo=FIPS186Random

Any additional ideas on how to debug this or what to look at?  Thanks!

[Charlie Arehart]

Mike, I think you’re on the right track, that you may have been hacked.

First, I can’t tell if you realize it, but that log line you refer (“starting http request”) is referring to an outgoing request from your server. The line you’re quoting is from the http.log (but maybe you’re finding it in the coldfusion-out.log, or cfserver.log on *nix, where other CF logs get duplicated in addition to some unique log info held in that file alone), and that line is written whenever a CFHTTP tag executes (the cfhttp logging was added in 9.0.1).

So how often are those appearing? Like every few seconds? if you don’t recognize the URL, then it would certainly seem something related to a hack (where bad code has been put on your server and it’s not being used to access other computers, for whatever nefarious reason).

Sadly, the http.log entry doesn’t tell you what file or line of code is executing that tag, so you’ll need to search your whole code base to find it. As tempting as it may be to search for that URL, you may not find it readily. It could be being read from something else (perhaps a database entry, or indeed being obtained from some other server in another CFHTTP call). Instead, I’d propose you just search your code base for any references to CFHTTP. Hopefully you don’t have too many in your own code. There is also the http.cfc available since CF9 which can be called via script, so you may want to look for that, too. (And in CF11 there’s still another way to call the equivalent of all tags as script, but your code was running on 9 or 10 before, right?)

As for searching for the content, there are many ways to do that. If you’re on windows, my preferred way is with a tool called File Locator Lite (free). I blog about it here:

http://www.carehart.org/blog/client/index.cfm/2009/12/2/faster_better_file_searching

One last thing, fwiw. You mention that you had high CPU, and there can be any causes for that. I outline a few (including the possibility that one has been hacked) in a blog entry here:

http://www.carehart.org/blog/client/index.cfm/2014/6/24/common_causes_of_high_CPU_in_ColdFusion

Hope that helps. Let us know what you find.

/charlie

[mike42780]

Thanks Charlie for your insight.  I searched everywhere to find a cfhttp request that was causing the issue, but couldn't find it.  I did find some unusual files that were loaded though that I know we didn't put there.  I did a search and found this link that shows the code.  Coldfusion CFIDE bitcoin mining exploit – PHP involved… | code-complete.com and that you mention in your article too.  I don't see any of the executables running or files that were mentioned though.  I found that code in 8 different spots though and removed them.  Maybe our old server had the executables and hacked files on them.  Hard to know as it won't boot up anymore!

I think I did possibly isolate the HTTP request {URL='http://zzen1wbudopwg.nchyt.com/encfm/en0024-ssj5iway6wvg/cbeim94a1s2kebu.php?do=218', method='get'} in the http.log file to a certain site in IIS though.  And it wasn't our main site.  I turn off this website in IIS and it appears the weird request goes away.  I turn the site back on and it starts to re-appear.  It doesn't always load constantly so it's a little hard to tell.  This site is pretty small.  I went through each file on the site and did find one file that did appear to be hacked.  It wasn't coldfusion code though.  Just some html links.  I removed it.  That's all I could find.  No cfhttp calls or anything else.

We re-installed Coldfusion 11 on Friday as well and upgraded to Update 3.  It doesn't stay locked at 100% as much right now, but it being over the weekend we don't get much traffic.  Monday will be the real test.  I think I will leave that smaller site turned off for now and see how things perform.  I'm doing a full virus scan for the heck of it overnight too.  Don't really expect it to find anything though. I also turned on advanced logging in IIS 7.5 and don't see anything out of the ordinary.  I made sure client variables weren't in the registry either.

Here's a part of the http.log file when I turn the IIS site on.  I turn the site off and it stops popping up in the logs.

"Information","ajp-bio-8014-exec-57","12/13/14","22:33:14",,"Starting HTTP request {URL='http://zzen1wbudopwg.nchyt.com/encfm/en0024-ssj5iway6wvg/cbeim94a1s2kebu.php?do=233', method='get'}"

"Information","ajp-bio-8014-exec-57","12/13/14","22:36:17",,"Starting HTTP request {URL='http://zzen1wbudopwg.nchyt.com/encfm/en0024-ssj5iway6wvg/cbeim94a1s2kebu.php?do=601', method='get'}"

"Information","ajp-bio-8014-exec-57","12/13/14","22:38:31",,"Starting HTTP request {URL='http://zzen1wbudopwg.nchyt.com/encfm/en0024-ssj5iway6wvg/cbeim94a1s2kebu.php?do=459', method='get'}"

"Information","ajp-bio-8014-exec-57","12/13/14","22:38:54",,"Starting HTTP request {URL='http://zzen1wbudopwg.nchyt.com/encfm/en0024-ssj5iway6wvg/cbeim94a1s2kebu.php?do=108', method='get'}"

"Information","ajp-bio-8014-exec-57","12/13/14","22:39:55",,"Starting HTTP request {URL='http://zzen1wbudopwg.nchyt.com/encfm/en0024-ssj5iway6wvg/cbeim94a1s2kebu.php?do=218', method='get'}"

"Information","ajp-bio-8014-exec-63","12/13/14","22:52:03",,"Starting HTTP request {URL='http://zzen1wbudopwg.nchyt.com/encfm/en0024-ssj5iway6wvg/cbeim94a1s2kebu.php?do=54', method='get'}"

"Information","ajp-bio-8014-exec-64","12/13/14","22:57:32",,"Starting HTTP request {URL='http://zzen1wbudopwg.nchyt.com/encfm/en0024-ssj5iway6wvg/cbeim94a1s2kebu.php?do=40', method='get'}"

"Information","ajp-bio-8014-exec-64","12/13/14","22:58:37",,"Starting HTTP request {URL='http://zzen1wbudopwg.nchyt.com/encfm/en0024-ssj5iway6wvg/cbeim94a1s2kebu.php?do=702', method='get'}"

[Charlie Arehart]

So Mike, how did it go? It seems clear you got hacked by the bitcoin mining exploit. Did you confirm that the bad code did not follow you to the new server? And did you lock things down on the new machine to try to prevent it getting in again?

/charlie

[mike42780]

Hi Charlie,

For that smaller site that seemed to be causing the unusual URL, I deleted the IIS site and re-created it.  And then I deleted the production files and copied the files from our development server and made sure everything was working like normal.  That stopped the URL from popping up in the log files.  No idea how.  That site didn't have any cfhttp calls on it anywhere.  We also added an additional CPU on the server at the request of a Adobe that was looking at it.  We've had a couple of CPU spikes since then that have caused the site to run slow for a couple of minutes, but that is usually not the norm now.  The CPU stays lower and if it spikes at 100% it doesn't stay there long.  I'm still nervous about it, but for the most part it has been stable.

I never saw any unusual processes running that looked like the bitcoin exploit, so maybe that was on the old server and didn't get carried over.  We locked down the server using the CF11 whitepaper the best we could.  If the bitcoin exploit was hitting our server hard, hopefully it has moved on.  I'm keeping an eye on it, but not sure our server will get hard over the holidays, so again not the best test. 

Thanks again for all of your help.