Solr and summary text

[cinthara]

Am converting from Verity to Solr search. Anyone have any luck increasing the length of the summary generated by Coldfusion under Solr? I noticed that with Solr, the summary will list the first few words of whatever you enter for the "body" of the CFINDEX whereas Verity's summary is longer and goes to a paragraph/sentences where it finds the search word criteria (or close to it). Under the Solr, the summary doesn't seem that useful. I want to increase the summary text so it display more...hopefully, useful information.

[cinthara]

The summary text for Solr sucks. They totally need to work on it. The summary text should not be the first 100 characters of the first sentence of the record or file. That is not useful to anyone if all the records/files starts off the same way. For usefulness, I need to display more characters, why doesn't Solr have that feature?

I'm surprise no one else noticed this.  I'll just stick to Verity until I am force to move over to Solr. I don't get why Solr is better, since it fails to deliver in its most basic requirement.

[cinthara · Accepted solution]

I found the solution to my problem.

http://help.adobe.com/en_US/ColdFusion/9.0/Developing/WSe9cbe5cf462523a0-5bf1c839123792503fa-8000.ht...

If the page ever moves, I've copied it and added a few notes for me below:

By default, Solr highlights searched terms in the summary content as described in the following snippet:

<cfsearch

collection="syn1"

criteria="Services solr"

name="results"

status="r"

suggestions="always"

contextPassages="1">

To highlight contents in the entire document, modify the solrconfig.xml and schema.xml files. These files are available in the following locations:

<Collection Directory>/conf:  Modify files in this location to apply the changes only to a particular collection.

<Solr Home>/multicore/template/conf:  Modify files in this location to apply the changes to all future Solr collections.

1. Stop Solr service.

2. Replace the following section in the solrconfig.xml (LINE 361)

<requestHandler name="standard" class="solr.StandardRequestHandler" default="true">

    <!-- default values for query parameters -->

     <lst name="defaults">

       <str name="echoParams">explicit</str>

       <str name="hl.fl">summary title </str> 

       <!-- omp = Only More Popular -->

       <str name="spellcheck.onlyMorePopular">false</str>

       <!-- exr = Extended Results -->

       <str name="spellcheck.extendedResults">false</str>

       <!--  The number of suggestions to return -->

       <str name="spellcheck.count">1</str>

     </lst>

     <arr name="last-components">

      <str>spellcheck</str>

     </arr>

  </requestHandler>

with

<requestHandler name="standard" class="solr.StandardRequestHandler" default="true">

    <!-- default values for query parameters -->

     <lst name="defaults">

       <str name="echoParams">explicit</str> 

       <str name="hl.fl">contents title </str> 

       <!-- omp = Only More Popular -->

       <str name="spellcheck.onlyMorePopular">false</str>

       <!-- exr = Extended Results -->

       <str name="spellcheck.extendedResults">false</str>

       <!--  The number of suggestions to return -->

       <str name="spellcheck.count">1</str>

     </lst>

     <arr name="last-components">

      <str>spellcheck</str>

     </arr>

  </requestHandler>

3. Replace the following section in the schema.xml (LINE 456)

<field name="contents" type="text" indexed="true"  stored="false" required="false" multiValued="true" omitNorms="true"/>

with

<field name="contents" type="text" indexed="true" stored="true" required="false" multiValued="true" omitNorms="true"/>

4. Restart Solr.

5. Reindex the collection. 

Note: The modifications to solrconfig.xml and schema.xml will increase the index size.

In my search, I increased the contextbytes and contextpassages.

<cfsearch name="qSearch"

                          collection="index_name"

                          status = "info"

                          suggestions="always"

                          ContextBytes="30000"

                          ContextPassages= "1000" 

                          CRITERIA="#TRIM(search)#"

                          >

[cinthara]

Making the changes above gave us this error (I guess due to the the index increasing): HTTP Error 500: Java_heap_space__javalangOutOfMemoryError_Java_heap_space_

I found the solution that worked for us at:

http://www.coldfusionmuse.com/index.cfm/2010/4/4/solr-troubleshooting-coldfusion-9

I copied the content below (in case the page disappears):

Troubleshooting and Optimizing Solr on ColdFusion 9

Posted At : April 4, 2010 8:53 PM | Posted By : Mark Kruger
Related Categories: Coldfusion Troubleshooting

I had an interesting troubleshooting session recently with a customer. This customer had a very high powered server - SAS drives, 16 gigs of RAM, 64 bit throughout, Coldfusion 9 and an 8 gig Java Heap. The site had 70 or 80 search collections and they had switched to CF9 specifically to get beyond the limitations of Verity. Everything was performing well when suddenly the search service stopped responding to requests and simply started throwing "collection not found" errors. Coldfusion seemed fine and dandy. It continued to be responsive and it had no hanging threads. It was as if the search service had lost it's handles to the various collections. Restarting Solr solved the issue, but why was Solr locking up?

A closer look showed that Solr wasn't crashing. Indeed, both of the services associated with Solr ("ColdFusion 9 Solr Service" and ColdFusion 9 Search Server") were running. In fact, on reflection it was clear that the Solr service was actually still responding to requests - it was simply throwing errors. So Solr actually had to be running, or we would see the "service not found" type error - right? Now the Muse is not yet a Solr guru, so this was trial and error. But the first thought was to go and see if there were any Solr log files. Sure enough I found them under /Coldfusion9/solr/logs. There were 3 logs listed - a "request" log, a "stderr" log and a "stdout" log, each with a date as part of the name. Hmmmm..... 2 of those logs look sort of familiar don't they? The "standard out" log or stdout (never been comfortable with the abbreviation "std") and the "standard error" log or stderr both remind me of conventional java logging.

Of course! Solr is a Java application under the hood. So troubleshooting Solr will be like troubleshooting any other Java application. I took a look at stderr and sure enough - recent errors were all "out of memory" Java heap errors. I know how to handle that - increase the heap size and fiddle with the Java arguments. I was well on my way to editing the JVM.config file when I had a different thought. Solr was running out of heap space but ColdFusion was not running out of heap space. After all, if I was having an overall memory problem I would expect ColdFusion to lock up as well. That means Solr runs in a separate JVM. Of course it does (doh!). That's why it has a service and a connection socket. What I really need is to find the jvm.config file (or whatever) for Solr and adjust the memory settings there.

In the root of /ColdFusion9/solr I found a little file called "solr.lax". You might remember a similar .lax file that I edited to get a 64bit install to complete on a windows 2k3 "web edition" (details here). It's actually a config file for an "executable jar" file compiled by a product called "launchAnwhere" - which I gather is basically "installsheild" for Java. Anyway, this file is bundled with an executable jar and contains the useful stuff the exe file will need to instantiate a JVM and get up and running.

In this case there were 2 lines we were interested in. The first one had a setting like this:

#   LAX.NL.CURRENT.VM

#   -----------------

#   the VM to use for the next launch

lax.nl.current.vm=C:\\ColdFusion9\\runtime\\jre\\bin\\javaw.exe

This line tells us that the Solr search service is going to use the same JVM as the ColdFusion engine. That's pretty good news because we already know a good deal about preventing and troubleshooting out of memory errors in Sun JRE version 1.6 (the default shipping with CF 9).

The other line of interest was further down and looked like this:

lax.nl.java.option.additional=-server -Xmx256m 

               -XX:+AggressiveOpts -XX:+ScavengeBeforeFullGC 
  

               -XX:-UseParallelGC -DSTOP.PORT=8079 

               -DSTOP.KEY=cfstop -Dsolr.solr.home=multicore 

Now this looks mighty familiar. It's our Java arguments to instantiate our JVM and configure it for use. What can we deduce from this set of arguments? Well, to start with, Solr is configured to use a minimum of zero and a maximum of 256 megs of memory. It's also set up to aggressively recover heap space. In our special situation we need to get Solr more memory and more resources to work with. We changed the GC to -XX:+UseConcMarkSweepGC, removed the aggressive options, and set the minimum and maximum both to one Gig (-Xms1024m -Xmx1024m). Sure enough after a restart our memory problems went away and Solr started performing splendidly.

[cinthara]

SOLR on Coldfusion 11

My changes are based on the settings from earlier versions of CF: http://help.adobe.com/en_US/ColdFusion/9.0/Developing/WSe9cbe5cf462523a0-5bf1c839123792503fa-8000.ht...

For ColdFusion 11, I did the following:

By default, Solr highlights searched terms in the summary content as described in the following snippet:

<cfsearch

collection="syn1"

criteria="Services solr"

name="results"

status="r"

suggestions="always"

contextPassages="1">

To highlight contents in the entire document, modify the solrconfig.xml and schema.xml files. These files are available in the following locations:

<Collection Directory>/conf:  Modify files in this location to apply the changes only to a particular collection.

On my server it is located at: CF11/cfusion/collection/<collection name>/

<Solr Home>/multicore/template/conf:  Modify files in this location to apply the changes to all future Solr collections.

On my server it is located at: CF11/cfusion/jetty/multicore/template/conf:

1.            Make a backup of the config  folder. Stop ColdFusion Add-on Services. 

2.            Replace the following section in the solrconfig.xml (LINE 361)

<requestHandler name="standard" class="solr.StandardRequestHandler" default="true">

    <!-- default values for query parameters -->

     <lst name="defaults">

       <str name="echoParams">explicit</str>

       <str name="hl.fl">summary title </str> 

       <!-- omp = Only More Popular -->

       <str name="spellcheck.onlyMorePopular">false</str>

       <!-- exr = Extended Results -->

       <str name="spellcheck.extendedResults">false</str>

       <!--  The number of suggestions to return -->

       <str name="spellcheck.count">1</str>

     </lst>

     <arr name="last-components">

      <str>spellcheck</str>

     </arr>

  </requestHandler>

with

<requestHandler name="standard" class="solr.StandardRequestHandler" default="true">

    <!-- default values for query parameters -->

     <lst name="defaults">

       <str name="echoParams">explicit</str> 

       <str name="hl.fl">contents title </str> 

       <!-- omp = Only More Popular -->

       <str name="spellcheck.onlyMorePopular">false</str>

       <!-- exr = Extended Results -->

       <str name="spellcheck.extendedResults">false</str>

       <!--  The number of suggestions to return -->

       <str name="spellcheck.count">1</str>

     </lst>

     <arr name="last-components">

      <str>spellcheck</str>

     </arr>

  </requestHandler>

3.            Replace the following section in the schema.xml (LINE 456)

<field name="contents" type="text" indexed="true"  stored="false" required="false" multiValued="true" omitNorms="true"/>

with

<field name="contents" type="text" indexed="true" stored="true" required="false" multiValued="true" omitNorms="true"/>

4.            Change the default operator – When we used Verity, searching for ‘fire water’ would in effect search for ‘fire and water’. With Solr, ‘fire water’ searches for ‘fire OR water’. If you need to change the default operator between words in keyword searches, don’t despair. Go to  \config\schema.xml. Around line 530 you should see: <solrQueryParser defaultOperator=”OR”/>, change to <solrQueryParser defaultOperator=”AND”/> (LINE 530) if need be.

5.            Restart ColdFusion Add-on Services.

6.            Create the collection and index them. If you already created the collection, in Coldfusion Admin, delete the collection. Stop the Coldfusion Add-on Services.  On the server, delete collection under \CF11\cfusion\collections\<collection name>.  DO NOT delete any other folders. All the folders listed there should be the same as what is under the Coldfusion Admin website. Restart ColdFusion Add-on Services. Create your collection and index. The new collection will get all the settings based on the changes you made under CF11/cfusion/jetty/multicore/template/ .