Lucene Similarity and Score

Changing Similarity in Solr
Check Solr wiki:
A (global)  declaration can be used to specify a custom Similarity implementation that you want Solr to use when dealing with your index.

  P
  L
  H2
  7

Custom Per-Field Similarity

  
    
    
      SPL
      DF
      H2
    
  

If no (global)  is configured in the schema.xml file, an implicit instance of DefaultSimilarityFactory is used.

Lucene scoring supports a number of pluggable information retrieval models, including:
Vector Space Model (VSM)
Probablistic Models such as Okapi BM25 and DFR
Language models



Changing Scoring — Similarity

Changing Similarity is an easy way to influence scoring, this is done at index-time withIndexWriterConfig.setSimilarity(Similarity) and at query-time with IndexSearcher.setSimilarity(Similarity). Be sure to use the same Similarity at query-time as at index-time (so that norms are encoded/decoded correctly);

The Scorer abstract class provides common scoring functionality for all Scorer implementations and is the heart of the Lucene scoring process. 

Similarity

SimilarityBase

TFIDFSimilarity

Lucene combines Boolean model (BM) of Information Retrieval with Vector Space Model (VSM) of Information Retrieval - documents "approved" by BM are scored by VSM.

In VSM, documents and queries are represented as weighted vectors in a multi-dimensional space, where each distinct index term is a dimension, and weights are Tf-idf values.

Lucene's Practical Scoring Function

score(q,d)   =   coord(q,d)  ·  queryNorm(q)  ·  ∑ ( tf(t in d)  ·  idf(t)2  ·  t.getBoost() ·  norm(t,d) ) t in q

Lucene Practical Scoring Function

tf(t in d) correlates to the term's frequency, defined as the number of times term t appears in the currently scored document d. Documents that have more occurrences of a given term receive a higher score.
tf(t in d)   =  frequency^½

  public float tf(float freq) {

    return (float)Math.sqrt(freq);

  }

 idf(t) stands for Inverse Document Frequency. This value correlates to the inverse of docFreq (the number of documents in which the termt appears). This means rarer terms give higher contribution to the total score. idf(t) appears for t in both the query and the document, hence it is squared in the equation. 

idf(t)  =

1 + log (

numDocs ––––––––– docFreq+1

)

  public float idf(long docFreq, long numDocs) {

    return (float)(Math.log(numDocs/(double)(docFreq+1)) + 1.0);

  }

coord(q,d) is a score factor based on how many of the query terms are found in the specified document. Typically, a document that contains more of the query's terms will receive a higher score than another document with fewer query terms. This is a search time factor computed in coord(q,d) by the Similarity in effect at search time. 
  public float coord(int overlap, int maxOverlap) {
    return overlap / (float)maxOverlap;
  }
queryNorm(q) is a normalizing factor used to make scores between queries comparable. This factor does not affect document ranking (since all ranked documents are multiplied by the same factor), but rather just attempts to make scores from different queries (or even different indexes) comparable. This is a search time factor computed by the Similarity in effect at search time. The default computation inDefaultSimilarity produces a Euclidean norm: 

queryNorm(q)   =   queryNorm(sumOfSquaredWeights)   =

1 –––––––––––––– sumOfSquaredWeights½

The sum of squared weights (of the query terms) is computed by the query Weight object. For example, a BooleanQuery computes this value as: 

sumOfSquaredWeights   =   q.getBoost() 2  ·	∑	( idf(t)  ·  t.getBoost() ) 2
	t in q

  public float queryNorm(float sumOfSquaredWeights) {
    return (float)(1.0 / Math.sqrt(sumOfSquaredWeights));
  }
 t.getBoost() is a search time boost of term t in the query q as specified in the query text (see query syntax), or as set by application calls to setBoost()

norm(t,d) encapsulates a few (indexing time) boost and length factors:

• Field boost - set by calling field.setBoost() before adding the field to a document.
• lengthNorm - computed when the document is added to the index in accordance with the number of tokens of this field in the document, so that shorter fields contribute more to the score. LengthNorm is computed by the Similarity class in effect at indexing.

The computeNorm(org.apache.lucene.index.FieldInvertState) method is responsible for combining all of these factors into a single float.

When a document is added to the index, all the above factors are multiplied. If the document has multiple fields with the same name, all their boosts are multiplied together:

norm(t,d)   =   lengthNorm  ·	∏	f.boost()
	field f in d named as t

Note that search time is too late to modify this norm part of scoring, e.g. by using a different Similarity for search.
  public float lengthNorm(FieldInvertState state) {
    final int numTerms;
    if (discountOverlaps)
      numTerms = state.getLength() - state.getNumOverlap();
    else
      numTerms = state.getLength();
   return state.getBoost() * ((float) (1.0 / Math.sqrt(numTerms)));
  }

  public final long computeNorm(FieldInvertState state) {
    float normValue = lengthNorm(state);
    return encodeNormValue(normValue);
  }
  public final long encodeNormValue(float f) {
    return SmallFloat.floatToByte315(f);
  }