Skip to main content
N
NuckNuck
Participating Frequently
October 7, 2010
Question

What Do You Use in Place of VSpider When Using Solr?

  • October 7, 2010
  • 1 reply
  • 2049 views

Since Verity is deprecated according to CF Documentation, what crawler do you use if you want to index dynamic pages (like vSpider would)?  Can you use Solr with vSpider or is there something better out there or bundled with CF9?

    This topic has been closed for replies.

    1 reply

    D
    Community Expert
    Dave WattsCommunity Expert
    Community Expert
    October 7, 2010

    You were probably the only person using Vspider! Most Verity users just used CFINDEX.

    Solr doesn't come with a crawler. There are plenty of crawlers that can work with Solr, though - just Google "solr crawler".

    Dave Watts, CTO, Fig Leaf Software

    http://www.figleaf.com/

    http://training.figleaf.com/

    Fig Leaf Software is a Veteran-Owned Small Business (VOSB) on

    GSA Schedule, and provides the highest caliber vendor-authorized

    instruction at our training centers, online, or onsite.

    Dave Watts, Eidolon LLC
    A
    Adam Cameron.
    Inspiring
    October 7, 2010

    You were probably the only person using Vspider! Most Verity users just used CFINDEX.

    The problem with using <cfindex> is that the location of the document in the site's filesystem hierarchy and the infrastructure elements such nav (and even header and footer) are significant in evaluating the context of the document for the purposes of weighting it.

    Using a spider is a better approach to getting an accurate, weighted index of the website, rather than just its component data.

    --

    Adam

    A
    Adam Cameron.
    Inspiring
    February 23, 2011

    Yea, I looked into Nutch it but it seems like a lot of work to figure out. The documentation is horrible and I haven't found any information on how to make it work with ColdFusion.

    What I did find, however, is SearchBlox, which looks promising. I very quickly installed it (on Windows) and got it to crawl and index one of my sites. I can then do searches from directly within the web admin console (everything I've done so far works from within the admin console). Next, I'll figure out how to make calls into it from ColdFusion to conduct a search (which I think will be trivial). You can get XML back from a search, which will make it easy. If this works, I will have found a great solution completely independent from ColdFusion's Solr integration.

    It's interesting to me that there's such little information out there on what ColdFusion developers are using to crawl/index their sites. Isn't anyone doing this???

    -JP


    It's interesting to me that there's such little information out there on what ColdFusion developers are using to crawl/index their sites. Isn't anyone doing this???

    I reckon most people won't've been using Verity from the outset because it's a bit rubbish (or CF's implementation of it is), so they'll already have something else implemented for doing search, so that CF9 now offers something else instead of Verity is neither here nor there.

    Or others will be using Verity, and it works for them, so that CF9 now offers something else instead of Verity is neither here nor there.

    Or that they'll be using a proper hardware search appliance, so that CF9 now offers something else instead of Verity is neither here nor there.

    Or that they don't bother to actually spider their site, instead they just use a CFINDEX with just the docs or data straight from the DB.  In which case they're just using Verity or they've decided to try Solr and Solr does that side of things OK.

    --

    Adam

    Terms & ConditionsAccessibility statement
    Using the community Terms of use Privacy Cookie preferences Do not sell or share my personal information ad choicesAdChoices