flurry of thousands of hits from single machines using Safari browser

Community Beginner ,
Apr 06, 2018 Apr 06, 2018

Copy link to clipboard

Copied

I have a weird issue that just started recently on a new CF2016 server (patched to current) running on IIS 10.  It has a small handful of older sites that have been migrated w/no changes from a CF11 server where this wasn't ever a problem.  I have Fusion Reactor installed and about three or four times a week, I'll see a very high number of requests in the daily report with about twice the number of sessions as web requests compared to what it normally sees  (25,000 vs 2,500).   

Digging into the IIS logs, I'll see a single client hit the site many thousands of times and always from a Apple computer running Safari (a few different versions 601.2.7,605.1.15), for example:

2018-04-05 00:43:58 x.x.x.28 GET / - 443 - x.x.76.100 Mozilla/5.0+(Macintosh;+Intel+Mac+OS+X+10_11_1)+AppleWebKit/601.2.7+(KHTML,+like+Gecko)+Version/9.0.1+Safari/601.2.7 https://[some other domain]/ 200 0 64 0

2018-04-05 00:43:58 x.x.x.28 GET / - 443 - x.x.76.100 Mozilla/5.0+(Macintosh;+Intel+Mac+OS+X+10_11_1)+AppleWebKit/601.2.7+(KHTML,+like+Gecko)+Version/9.0.1+Safari/601.2.7 https://[some other domain]/ 200 0 64 0

2018-04-05 00:43:58 x.x.x.28 GET / - 443 - x.x.76.100 Mozilla/5.0+(Macintosh;+Intel+Mac+OS+X+10_11_1)+AppleWebKit/601.2.7+(KHTML,+like+Gecko)+Version/9.0.1+Safari/601.2.7 https://[some other domain]/ 200 0 64 0

 

Fusion Reactor transaction logs show:

2018-04-04 18:43:58.598 1522889038598 4 1522449650203 10684 EXECUTING "" ajp-nio-8016-exec-10 x.x.76.100 GET https://[mydomain]/index.cfm 0 57 1905664 1087085 1905664 818578 "" 200 0 "" "" "" "" 0 0 0 0 0 0 0 0 0 "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_1) AppleWebKit/601.2.7 (KHTML, like Gecko) Version/9.0.1 Safari/601.2.7"

2018-04-04 18:43:58.598 1522889038598 4 1522449650203 10684 COMPLETED "" ajp-nio-8016-exec-10 x.x.76.100 GET https://[mydomain]/index.cfm 0 57 1905664 1087891 1905664 817772 "" 200 0 "" 221AB3433650A285ED25C440B5DF0533.cfusion 855423 a6c04aef2894ca67-C099434F-D62F-E738-F976C5547E3C32FF 0 0 0 0 10574 0 0 0 0 "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_1) AppleWebKit/601.2.7 (KHTML, like Gecko) Version/9.0.1 Safari/601.2.7"

2018-04-04 18:43:58.677 1522889038677 4 1522449650203 10685 EXECUTING "" ajp-nio-8016-exec-7 x.x.76.100 GET https://[mydomain]/index.cfm 0 57 1905664 1087925 1905664 817738 "" 200 0 "" "" "" "" 0 0 0 0 0 0 0 0 0 "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_1) AppleWebKit/601.2.7 (KHTML, like Gecko) Version/9.0.1 Safari/601.2.7"

2018-04-04 18:43:58.692 1522889038692 4 1522449650203 10685 COMPLETED "" ajp-nio-8016-exec-7 x.x.76.100 GET https://[mydomain]/index.cfm 15 57 1905664 1088733 1905664 816930 "" 200 15 "" 5B9DC09E1E2DE1CC5EA503976D1DED6A.cfusion 855424 211f54c33fa8698d-C0994670-BB5B-FF57-95B64E229A54272C 0 0 0 0 10574 15 15 0 15 "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_1) AppleWebKit/601.2.7 (KHTML, like Gecko) Version/9.0.1 Safari/601.2.7"

As you can see, the session ID changes with each request.  A couple of the applications that this happens to are very simple. They are just a small form and session management is enabled so CSRFGenerateToken/CSRFVerifyToken can be used.  No CFlocation tags or redirects that would cause this, so I'm wondering if this is a server misconfiguration, an attack, or Mac people have cats that like to lay on their keyboard...  Any ideas on what to look for?

Views

316

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines

correct answers 1 Correct Answer

Guide , Apr 08, 2018 Apr 08, 2018
Hi Hemi345,I have seen similar for CF2016 joined to IIS10 (Windows 2016). The issue is something about Safari browser not working so well with the webserver http/2 capability. You could find the hurry of Safari requests resume normal  activity by disabling http/2.Do you notice the CF Java heap filling with objects that do not evacuate when the massive amount of requests were happening?Http/2 off:1) start run regedit2) Select the folder/path: HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\H...

Likes

Translate

Translate
Adobe Community Professional ,
Apr 06, 2018 Apr 06, 2018

Copy link to clipboard

Copied

Hemi, I’d regard it as an attack, or at least an undue amount of spidering/crawling (and perhaps nefarious, if not even a possible denial of service attack), at least until evidence proves otherwise. Let’s look at 3 aspects:

1) Did you notice first that all the IPs are the same? Of course, those can be spoofed, but in your case, they really are the same. There are mechanisms (in web servers and firewalls and web app firewalls) that could throttle excessive repeated requests from the same IP. Of course, a single IP may make many requests to the web server for static files (images, js, css), but many requests per second from on IP to CF would be unusual, and cause for throttling.

I have some code I offered that does that:

http://www.carehart.org/blog/client/index.cfm/2010/5/21/throttling_by_ip_address

Of course, some bad guys try to throw such protection tools “off the scent” by spoofing different IPs. But in your case they are not, so it’s worth a shot.

2) As for the fact that the user agent is changing, and in case you thought may mean it really was “different people”, well that can be spoofed, too. And some automated request agents (especially bad guys) would to that, to again hide their activities from tools (or people reading logs) that may be watching for them.

And as for the changing sessionid, that’s certainly another issue. It means (as you said earlier) each is creating sessions. And that can be bad, not just for session creation but also for client variable creation (if you enable clientmanagement as well, along with sessionmanagement). I did a post on that as well:

http://www.carehart.org/blog/client/index.cfm/2006/10/4/bots_and_spiders_and_poor_CF_performance

3) Finally, on the whole matter of automated requests and their potential undue impact, and ways to mitigate them, I’ve done a presentation on that, the PDF of which is available here (along with a 50 min podcast interview on the topic):

http://www.carehart.org/presentations/#spiders

Hope that’s helpful.

/charlie


/Charlie (server troubleshooter, carehart.org)

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Community Beginner ,
Apr 09, 2018 Apr 09, 2018

Copy link to clipboard

Copied

Thanks for all the info Charlie. I had read your Spiders article a few weeks back and added additional protections for that already.  But some other good nuggets so thanks for those.

Turns out I had a legitimate visitor try to access the site on Saturday morning using Safari on Mac OS 10.11 and she had the issue.  Then this morning I see Carl's response (thank you!) and that lead me to Giancarl's video Giancarlo Gomez: ColdFusion IIS 10 HTTP/2 - Safari Bug  and eventually the bug in Adobe's tracker, Tracker

So Adobe/Microsoft are aware of this problem and MS should offer a patch for Win Server 2016 that addresses this issue.   Appears this only affects the older Safari clients as I had a coworker test with a couple different more recent versions of Safari without issue.

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Adobe Community Professional ,
Apr 09, 2018 Apr 09, 2018

Copy link to clipboard

Copied

Thanks for the kind regards, and glad if you may be on to a better solution with Carl’s suggestion.

When you say that Adobe and MS “should” offer a patch, are you saying that you sense that they “will”, or is that a plea that you wish they “would”? It’s just that your statement could be read either way, and especially for the same of future readers of this thread, it seems worth getting clarification.

/charlie


/Charlie (server troubleshooter, carehart.org)

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Community Beginner ,
Apr 09, 2018 Apr 09, 2018

Copy link to clipboard

Copied

Sorry for the ambiguity, I meant that there is supposed to be a patch released to the public that will address this issue later this month.  One of last comments in Adobe's bug tracker states:

"Update:  Microsoft will provide the fix for this bug as a part of their Windows update being rolled out in April. Unfortunately, they can not provide any separate hotfix for this, due to a change in their internal build technology. Hence, the only way to get this issue fixed is wait for them to release the update which should (ideally) be made public in April, 2018."

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Guide ,
Apr 08, 2018 Apr 08, 2018

Copy link to clipboard

Copied

Hi Hemi345,

I have seen similar for CF2016 joined to IIS10 (Windows 2016). The issue is something about Safari browser not working so well with the webserver http/2 capability. You could find the hurry of Safari requests resume normal  activity by disabling http/2.

Do you notice the CF Java heap filling with objects that do not evacuate when the massive amount of requests were happening?

Http/2 off:

1) start run regedit

2) Select the folder/path: HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\HTTP\Parameters

3) Under the Parameters folder, right-click the white-space, add 2 new DWORD (32-bit) values: EnableHttp2Tls and EnableHttp2Cleartext

4) Check both new values have been set to 0 (disabled) by right-clicking the value and clicking modify

5) Restart Windows

HTH, Carl.

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Community Beginner ,
Apr 09, 2018 Apr 09, 2018

Copy link to clipboard

Copied

LATEST

Carl,

Yes, the heap fills and never gets cleaned up unless I restart the CF service.  I had the CF service hang a couple times before I had Fusion Reactor installed.  The CF logs indicated the server ran out of memory. Those two times the Safari client stayed for over a few hours trying to load the page.  I've since disabled HTTP/2 for now till the patch is available.

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines