CF 2023 Bug or Solr Configuration Problem

[AlexCraig]

Greetings All. After reviewing my testing below I would VERY, VERY much appreciate any suggestions as to why searches against a collection using a phrase are not working properly, in this case the criteria field = "tribology friction".

This issue arises as a function of my attempts to solve a CFsearch problem in another thread.

The following is a list of test which were done comparing the results of the Solr Admin tool (SAT), production server (PS), development server application code (DSAC) & deveopment server stand alone script (DSSA) against the same data. Note: the production server is running CF 4.51 & the development server is running CF 2023.

The testing searched for the words "tribology" and "friction" using 3 different criteria as described below. Note: For those of you who are curious, tribology is the science of bearing wear.

Criteria                                       SAT        PS        DSAC       DSSA

"tribology" OR "friction"          333        328      332            333

"tribology" AND "friction"       19           19         17               19

"tribology friction"                   2             2          332            333

As you can see the results of the search against the phrase "tribology friction" is correct in our producton environment, but is way high in the development environment. It appears that the CF 20203 is treating the phrase as if it were an "OR" search.
Note: I'm not really concerned where results differ by 2 to 4 hits betweem the production server & the deveopment server. Verity almost certainly treats searching collections differently (punctuation, specialty characters or whatever).

To me these results suggests that either there is a bug in CF 2023 or perhaps a Solr setting needs attention. I tend to believe a setting is not the issue as I left the default Solr installtion intact except for raising the minimim & maximum amount of memory for Solr as I have seen the Adobe CF Launcher App gobble up over 6000 M/B during testing.

Here is the stand alone script I used as a test to search the collection independent of the other CF code in our application.
****************************
<cfsearch
name = "ResTest"
collection = "Resumes"
criteria = "XXXXXX" <----- varied as shown earlier in this post
type="standard"
startrow=1
maxrows = "20000">
<cfoutput>
url=#ResTest.url#<br>
key=#ResTest.key#<br>
recordcount=#ResTest.recordcount#<br>
currentrow=#ResTest.currentrow#<br>
columnlist=#ResTest.columnlist#<br>
recordssearched=#ResTest.recordssearched#<br>

cfdump var = "#ResTest#">
</cfoutput>
****************************

[AlexCraig]

I have made some progress.  I got the Test Script to work by putting single quotes around the syntax for the criteria.

So criteria="tribology friction" did not work & got all candidates with either "tribology" or "friction" as hits.

A victory of sorts criteria='"tribology friction"'.  To wit, a single quote|a double quote|the phrase|double quote|single quote.

Maybe, I missed it but I never saw any examples that indicated a single quote was needed on either side as part of the syntax.

Unfortunately, I have not been able to get that string passed to a variable via the form code we're using which is:

<INPUT TYPE="hidden" NAME="keywords" VALUE="#Replace( Form.keywords, '"', '', 'ALL' )#">
<INPUT TYPE="hidden" NAME="keywords" VALUE="#Form.keywords#">

Note:  I had to add the Replace code to get single keywords to work at all.  Without the replace code I was getting returns on any search of every canidate in the collection, 56,297 candidates!

I've tried every combination of single & double quotes I can think of without any luck.

Anyone has any ideas, I'm all ears!

[AlexCraig]

OK.  I finally have come to the crux of the problem while trying to run the search inside the application.

Instead of the variable for the CFSEARCH code "#Form.keywords#" in the application itself, I substituted the actual syntax  needed for the query instead of the variable.

<CFSEARCH NAME="applicants_#var#"

                                COLLECTION="Resumes"

                                TYPE="standard"

                                CRITERIA= '"tribology friction"'>

BANG!  The collection was searched properly and my 2 hits that matched the criteria were on the screen!

I must admit I had to sit there and stare at it for awhile because I had been waiting so long to see it.

So absolutely, positively  The problem lies somewhere in the form code passing the value to the variable for the criteria in the CFSEARH.

After changing the code back to CRITERIA="#Form.keywords#"  ….

Just for the fun of it I typed in the keyword string “”””””xylophone””””””.    There is only one hit for xylophone in the collection. And the one hit came right up.

And the keyword that was displayed while the search was running was only the word xylophone itself.  All 12 of the “ signs had been stripped out by:

<INPUT TYPE="hidden" NAME="keywords" VALUE="#Form.keywords#">

Since the CRITERIA must equal the string '"tribology friction"' for the collection query to work, we need to come up with another way of passing the value to the CRITERIA variable.

So my next step is to come up with some alternative code to populate the variable with the proper Solr syntax for the query.

[Charlie Arehart · Accepted solution]

Glad to see you're making progress. I would suggest you now focus on debugging the input you're receiving. For now, take the search out of the equation, as you know that works if the criteria is correct. The question now is why it's not correct when input as a variable.

So I'll assume you've long ago output to the screen what was the input, or perhaps you relied on the cf debugging output at the bottom of the screen. Is that right? And the displayed form field values always "looked fine"? Yet somehow it failed to work as expected when passed as the search criteria?

(You certainly don't want "12 quotes". I'm assuming you've solved that now. Indeed there should be NO quotes around that input, as you'll be passing those results in as a quoted attribute of cfsearch...and any embedded double quotes will be a problem. But you can see those and seem now to have solved them.) 

Next, you may be suffering from unreadable characters being in that input...well, I mean you can't see them on screen in a cfoutput, because the browser may be rendering the display of that output (in any debugging output) so as not to SHOW those characters. So one option to do a view>source in your browser, to see what the text looks like. 

Another solution is to use code like this:

<cfoutput>#htmleditformat(form.keywords)#</cfoutput>

That very old function, or its sibling htmlcodeformat, will render any characters in a way that might show the problem.

And I'd recommend you view this form.keywords value both as it would be after the form submission--just before you would do the cfsearch.

But then also consider it for looking at any values you pre-fill in the form field. You show you're putting the form fields back into the input form, and perhaps you're doing some processing once you receive them. But then you may be putting THE ORIGINAL form fields back into their next display of the form. (BTW, such problems can be exacerbated when the "form input" actually comes from some database where perhaps you save and reuse prior search terms.) 

So another technique (to see actual text vs what the browser would "render") is to put the value in an html textarea form field, which also shows the actual text (or it could be the value of an input field, but the input populated from a db or by your form processing may be more than one line.)  BTW, I just mean use the textarea for debugging, not for normal user input. 

These techniques can be valuable to resolving such subtle bugs.

But if you still struggle, I'd recommend also you create a simple 5-10 line sample of a form and display of its output, and what values you're putting in. With that we might more easily resolve things. Clearly, to this point your words have had us focusing on cfsearch, but now this may be down to just this challenge of form processing. 

[AlexCraig]

Hey Charlie,

Very grateful that you took a look at what I had thus far; AND for your input.

>So I'll assume you've long ago output to the screen what was the input

That is correct.  Using a simple  <CENTER> <output> Key Words: #Form.keywords# </output> </CENTER>

I certainly like your <cfoutput>#htmledit .... suggestion and will change mine for further testing.

  >but now this may be down to just this challenge of form processing. 

 You hit the nail on the head.  Further testing last night enabled me to actually render the correct results by changing CRITERIA="#Form.keywords#" to '"#Form.keywords#"' (single quote|double quote on one end & vice versa on the other).

This manipulated the keyword string to finally render the two hits that were valid for the seach.  Aside:  I must admit I was so happy to see it, that I simply stared at the screen for awhile.

The crux of the problem is that all quote (") characters are being stripped out of the string by the form processing.  And since quote characters are integral to Solr, that kind of presents a problem.

Further, I then realized there are 4 possible iterations of the nature of the variable for the solr syntax for the CRITERIA.  I know how to manipulate two of them to render the proper results by simply adding &/or subtracting single or double quotes.  However, the other two would require the addition of a 2nd variable for the CRITERIA.  It could be done, but it would be messy.

What I really need to do is find a way to transfer the literal keystrokes from a field on the screen to the CRITERIA.  That would be the simplest way to get it functioning properly.  But for someone like me, who has only been doing maintenance programming in CF, that is easier said than done.  If you have any suggestions to the effect, I would welcome them.  Thanks again, Sir.

[Charlie Arehart]

Glad my suggestion helped. Could you mark it as an answer, to help others who may find the thread later?

As for your next question (above), I'll say again that you could really help us help you with some sample simple code, so anyone seeing this doesn't have to a) put it together to try to follow along and b) try to determine what would demonstrate your failure. I do appreciate it takes time and effort, of course. But again it's to help folks here help you.

Indeed, I see you opened another thread today, moving on to yet some other aspect your challenge getting search working. You may even wonder "how has anyone gotten it working?" (they have) or "why aren't any of them chiming in?" (they're not likely seeing this...few do, or a tiny subset of who uses cfsearch, and those few may be more likely to help with a demonstration example). Hope you'll consider it for both threads.

BTW, you said you're "dead in the water" in the other thread today. I've shared before how I can help on a consulting basis (carehart.org/consulting) . I realize you may not want to or can't "pay for help". Clearly I don't hold my help hostage: I've participated in several threads with you on these matters in recent weeks, getting you further and further each time. 

It can take a long time this way. Some of us can only dip in a couple times a day. You can help make that easier, or I can help you directly (even just in creating that demo, if you like). 

[AlexCraig]

Afternoon Charlie,

I would hire you in a heart beat to solve this problem.  However, alas, given the economic forces currently at work, the employment industry is in a serious slump.  Fortunately (or perhaps unfortunately at this juncture), I have been doing this every work day of my life since 11/6/76; and have survived numerous recessions.  Hopefully, we will survive this one as well.  But there is not an abundance of cash to spend these days.  Should we get lucky and enjoy a spurt of placements and the consequent revenue, you will be at the top of my list to call.  Trust me, I'd rather be paying up for expertise and investing time & effort in business development.  But, it is what is, for the time being.