Mining or Harvesting

Forum|Forum|17 years ago
August 17, 2008
4 replies
805 views

Im not sure what you call this - I would call it mining or harvesting.

Curious to know if anyone has attempted to do this, or has done it.

I want to be able to take our everyday yellowpages.com website, and be able to evaluate the page, and harvest the data from a search query.

What coldfusion functions would I use to do such a task?

Any help would be great.

Thanks.

Advanced techniques

This topic has been closed for replies.

tclaremont

Inspiring

Common sense, albeit not that common nowadays, says that the first rule prohibiting data mining seems to address the very effort you are attempting.

The second statement, which it appears you want to interpret to suit your needs, appears to address the casual looking up of Mary Smith's information. You may than distribute Mary Smith's phone number to whomever you choose.

If I read what you are trying to achieve in the original post, and then read the published limitations and terms of use, you are doing exactly what they DON'T want you to do.

As you have no doubt learned by now, the art of programming this method is trivial. The only thing stopping you legally is the terms of use.

davidsimms

Inspiring

cwm,

Actually you don't even have to ask for permission. The yellowpages.com site is pretty clear that you can't do what you want to do. http://www.yellowpages.com/about/terms states:

"You are prohibited from data mining, scraping, crawling, or using any process or processes that send automated queries to the YELLOWPAGES.COM Web site. You may not use the YELLOWPAGES.COM Web sites to compile a collection of listings, including a competing listing product or service."

David

A

Anonymous

That is correct. But it does say...

"Accordingly, You may view, use, copy, and distribute the Materials found on YELLOWPAGES.COM Web sites for internal, noncommercial, informational purposes only."

Which could be interpeted as "copy" of the data provided by them. What I am using it for is strictly non-commercial anyways. But, I have already figured out myself how to build it, and it is working beautifully.

What was take me hours to copy and paste, I can collect 25 addresses at a time in less than 1 minute.

Thanks though.

Chuck

D

Dan_Bracuk

Inspiring

Step 1. Ask for permission.

A

Anonymous

You dont have to ask permission to use their site or call the phone number listed on the results page...do you?

Obviously Dan, you are not right person here to answer my original question.

All I want to know is if it is possible to do it, and the coldfusion functions involved in making it happen.

Thanks.

K

Kronin555

Participating Frequently

Of course it's possible to do. Heck, you could do what you're asking with wget or curl.

Step 2. Read the ColdFusion documentation.
<cfhttp>

Then you'll have to parse the resulting page. You can do that in a number of ways. If it's valid XHTML, you can parse it into an XML doc object and do an xpath query on it. If it's not valid XHTML, then there's other ways to do it.

Have you done anything like this in other languages? It'd make learning how to do it in CF very simple. If you've never done anything like this before, google is your friend.

And for the record, I agree with Dan. Ask for permission. I also agree with you, you don't have to ask them for permission to do a search, however you're not using the data yourself, you're rebranding it, and profiting off it.

D

Dan_Bracuk

Inspiring

From your yellowpage.com website? Take the code that produces the page and make it reuseable using one of the methods available to do that.

A

Anonymous

I did not mean "our" as in my site - i used that openly. I want to search their database, and take the results page and harvest the info desired on that page.

Sign up

To post, reply, or follow discussions, please sign in with your Adobe ID.

Sign in to Adobe Community

To post, reply, or follow discussions, please sign in with your Adobe ID.

Scanning file for viruses.

This file cannot be downloaded