CODEWITHSUNDEEP
sqllucenefull-text-searchlucene.netsql-server-2012
balazs
balazs

Reputation: 520

lucene vs sql result scoring

I'm getting wildly different ranking/scoring between SQL and Lucene given the following query:

[pseudo code] (statut* within 3 of interpret*) AND contradict

I don't think this is an issue with the parser because all of the results seem to conform to the query requirements. However, in the top 1000 of results, I only get 172 common results. Since all the results from both Lucene and SQL both seem to conform to the query requirements, my only remaining guess is that the scoring is somehow radically different. I've had trouble finding any information on how SQL handles scoring, nor on comparing SQL and Lucene scoring. I'm not necessarily expecting the same results set from the two engines, but I was expecting more than 10% similarity and I need to at least be able to explain the huge discrepancy.

How can I explain this significant discrepancy?

Upvotes: 2

Views: 714

Answers (1)

casperOne
casperOne

Reputation: 74560

From the documentation on MSDN titled "How Search Query Results Are Ranked" (emphasis mine):

Full-text search in SQL Server can generate an optional score (or rank value) that indicates the relevance of the data returned by a full-text query. This rank value is calculated on every row and can be used as an ordering criteria to sort the result set of a given query by relevance. The rank values indicate only a relative order of relevance of the rows in the result set. The actual values are unimportant and typically differ each time the query is run. The rank value does not hold any significance across queries.

That said, it SQL full text searches places no real value on the result; the only value it has in relation to other rows in the result.

Compare that to scoring in Lucene, which is completely dependent on how you index the documents, whether or not the documents and/or fields are boosted, filters, etc.

Scoring in Lucene is also consistent, unlike in SQL server, where there are no guarantees. It's even reflected in the name, that the result from a full-text query in SQL server is a rank value and not a score, like it is in Lucene.

The values are not completely comparable, but that's understandable, because the results wouldn't be the same either.

Upvotes: 2

Related Questions