LinkedIn & ColdFusion pages

New Here ,
Feb 04, 2021 Feb 04, 2021

Copy link to clipboard

Copied

Hello,

 

I am hoping someone can help me understand what the underlying issue is here

 

1: This issue is occurring throughout our domain (www.agcareers.com) and none of the pages (.cfm) are shareable on LinkedIn. But, if I create an HTML page, LinkedIn fetches the metadata of that page not .cfm pages
2: URL:- https://www.agcareers.com/test-page-2.cfm On this page, there are two buttons at the top right corner. One with data-url of LinkedIn which works fine. The other button has data-url of our website which does not work The above URL when shared through: https://www.linkedin.com/sharing/share-offsite/?url=www.agcareers.com/test-page-2.cfm is throwing error as well but if I try html version using https://www.linkedin.com/sharing/share-offsite/?url=www.agcareers.com/test-page-2.htm it works fine.

 
LinkedIn Post Inspector fetches metadata of the html page but not .cfm page

https://www.linkedin.com/post-inspector/inspect/https:%2F%2Fwww.agcareers.com%2Ftest-page-2.htm

https://www.linkedin.com/post-inspector/inspect/https:%2F%2Fwww.agcareers.com%2Ftest-page-2.cfm

 

 

As per LinkedIn : We are unable to scrape any content with the URL's provided.
What's wrong, although the content is there?

Thanks,

Surinder

Views

101

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines

correct answers 1 Correct Answer

Adobe Community Professional , Feb 04, 2021 Feb 04, 2021
Surinder, I think I've found the cause of your problem, and my guess it's due to CFML code in your application.cfm or cfc (rather than anything "wrong" with ColdFusion or LinkedIn). It took a bit of digging (using server tools and browser developer tools as well as curl commands). Ultimately I was able to discern this: If the useragent for any request coming into your server for a CF page looks like the one that comes to you from LinkedIn, your server returns no output. Specifically it's if th...

Likes

Translate

Translate
Adobe Community Professional ,
Feb 04, 2021 Feb 04, 2021

Copy link to clipboard

Copied

It looks like something is going wrong at LinkedIn's side. When I click on the lower button ("Share with AgCareers URL"), the resulting URL is 

 

https://www.linkedin.com/cws/share/?url=https%3A%2F%2Fwww.agcareers.com&xdOrigin=https%3A%2F%2Fwww.a...

 

This contains no CFM file-type.

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Adobe Community Professional ,
Feb 04, 2021 Feb 04, 2021

Copy link to clipboard

Copied

Surinder, I think I've found the cause of your problem, and my guess it's due to CFML code in your application.cfm or cfc (rather than anything "wrong" with ColdFusion or LinkedIn). It took a bit of digging (using server tools and browser developer tools as well as curl commands).

 

Ultimately I was able to discern this: If the useragent for any request coming into your server for a CF page looks like the one that comes to you from LinkedIn, your server returns no output. Specifically it's if the user-agent header has the word "bot" in it.  Let me demonstrate and elaborate, and it should be pretty easy for you to "fix" on your end.

 

I first found what user-agent header LinkedIn is sending in and then I used my browser's dev tools to create a Curl command to try visiting your URL from the command line. For example, if I first do this curl command to visit the pages like a "regular" browser would:

 

 

curl "https://www.agcareers.com/test-page-2.cfm" ^
  -H "Connection: keep-alive" ^
  -H "Upgrade-Insecure-Requests: 1" ^
  -H "User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.96 Safari/537.36 Edg/88.0.705.56" ^
  -H "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9" ^
  -H "Accept-Language: en-US,en;q=0.9"

 

 

 it works.

 

Then I changed that useragent line to use the one that LinkedIn uses (more on how I found that in a moment):

 

 

-H "User-Agent: LinkedInBot/1.0 (compatible; Mozilla/5.0; Apache-HttpClient +http://www.linkedin.com)" ^

 

 

Then the result was literally no output at all. (And if you add the -i arg to curl you find that it IS getting a 200 status code. That's what makes me think this is CFML code controlling this, rather than your "server" or "web server".)

 

So on a lark I wondered if it could be as simple as that "bot" phrase (because some people DO implement code to "handle bots" or "block them".) I changed that header to remove the phrase bot:

 

 

-H "User-Agent: LinkedIn/1.0 (compatible; Mozilla/5.0; Apache-HttpClient +http://www.linkedin.com)" ^

 

 

And now the page DID work (in my curl). 

 

So look at your application.cfm or cfc in the root of our site (since that's where these requests are going), or any file that it includes or any CFC that it invokes. Search for any reference to "bot". You may find code blocking on the user-agent header, and perhaps you can tweak it to "allow in" the LinkedInBot user-agent.

 

BTW, you can run that curl command from your own laptop or workstation if it's Windows 10, or you can download a curl installer. Or there are other tools to simulate and manipulate such headers.

 

Finally, if anyone wonders how I know that LinkedIn is sending that header, there are a couple of ways. If you have a CF monitor like FusionReactor or the new CF2018 PMT, you can easily view headers that come into CF page requests. Or in a CFML page you can use the CFML function, gethttprequestdata() and cfdump that to see them.  With that in place, you could use that last pair of URLs he offered and point it to your own CFM page, so you could then see the headers linkedin is sending.

 

All these things can be useful "tricks of the trade" when trying to resolve such problems. And I will note that it's possible that even after solving this one (about the "bot" in the user-agent header), there may well be still some other header that something in your code or on your agcareers.com server could next trip over. Again you could use these techniques to perhaps find it yourself, or let us know.

 

Either way, do let us know if this gets you going.


/Charlie (server troubleshooter, carehart.org)

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Adobe Community Professional ,
Feb 10, 2021 Feb 10, 2021

Copy link to clipboard

Copied

Surinder, any update?


/Charlie (server troubleshooter, carehart.org)

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
New Here ,
Feb 16, 2021 Feb 16, 2021

Copy link to clipboard

Copied

Hi Charlie,

Thank you so much for this troubleshot and detailed explanation. I almost lost track when LinkedIn said in a response that .cfm is not supported.

Yes, it was indeed the application.cfc blocking the LinkedIn bot. I whitelisted it and it worked. Thanks a bunch again!!

 

- Surinder

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Adobe Community Professional ,
Feb 17, 2021 Feb 17, 2021

Copy link to clipboard

Copied

LATEST

Great to hear. Thanks for the update, and for marking the answer correct. Glad to have helped. 


/Charlie (server troubleshooter, carehart.org)

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines