With @james_blackwoodsewell_58 we had been evaluating the BM25 textual content search scores between MongoDB Atlas (Lucene), ElasticSearch (Lucene) and ParadeDB (utilizing Tantivy) which give the identical ordering however MongoDB Atlas reveals continuously a decrease rating by an element of two.2:
Again in October Franck Pachot from MongoDB (love your work) wrote a put up evaluating textual content search in MongoDB and PostgreSQL (with each the built-in tsvector and ParadeDB’s pg_search extension). I am not going to recap his entire put up, however principally Mongo appeared to behave precisely the way it ought to returning B
linkedin.com
It was the event for me to have a look at the rating particulars which provides the calculation particulars for the rating.
Take a look at case
I’ve constructed the identical take a look at case as in my earlier weblog:
db.articles.drop();
db.articles.deleteMany({});
db.articles.insertMany([
{ description : "🍏 🍌 🍊" }, // short, 1 🍏
{ description : "🍎 🍌 🍊" }, // short, 1 🍎
{ description : "🍎 🍌 🍊 🍎" }, // larger, 2 🍎
{ description : "🍎 🍌 🍊 🍊 🍊" }, // larger, 1 🍎
{ description : "🍎 🍌 🍊 🌴 🫐 🍈 🍇 🌰" }, // large, 1 🍎
{ description : "🍎 🍎 🍎 🍎 🍎 🍎" }, // large, 6 🍎
{ description : "🍎 🍌" }, // very short, 1 🍎
{ description : "🍌 🍊 🌴 🫐 🍈 🍇 🌰 🍎" }, // large, 1 🍎
{ description : "🍎 🍎 🍌 🍌 🍌" }, // shorter, 2 🍎
]);
db.articles.createSearchIndex("default",
{ mappings: { dynamic: true } }
);
Rating with particulars
I ran the identical question, including scoreDetails: true to the search stage, and scoreDetails: { $meta: "searchScoreDetails" } } to the projection stage:
db.articles.combination([
{
$search: {
text: { query: ["🍎", "🍏"], path: "description" },
index: "default",
scoreDetails: true
}
},
{ $venture: {
_id: 0, description: 1,
rating: { $meta: "searchScore" },
scoreDetails: { $meta: "searchScoreDetails" } } },
{ $type: { rating: -1 } } ,
{ $restrict: 1 }
])
Right here is the consequence:
mdb> db.articles.combination([
... {
... $search: {
... text: { query: ["🍎", "🍏"], path: "description" },
... index: "default",
... scoreDetails: true
... }
... },
... { $venture: { _id: 0, description: 1, rating: { $meta: "searchScore" }, scoreDetails: { $meta: "searchScoreDetails" } } },
... { $type: { rating: -1 } } ,
... { $restrict: 1 }
... ])
[
{
description: '🍏 🍌 🍊',
score: 1.0242118835449219,
scoreDetails: {
value: 1.0242118835449219,
description: 'sum of:',
details: [
{
value: 1.0242118835449219,
description: '$type:string/description:🍏 [BM25Similarity], results of:',
particulars: [
{
value: 1.0242118835449219,
description: 'score(freq=1.0), computed as boost * idf * tf from:',
details: [
{
value: 1.8971199989318848,
description: 'idf, computed as log(1 + (N - n + 0.5) / (n + 0.5)) from:',
details: [
{
value: 1,
description: 'n, number of documents containing term',
details: []
},
{
worth: 9,
description: 'N, complete variety of paperwork with area',
particulars: []
}
]
},
{
worth: 0.5398772954940796,
description: 'tf, computed as freq / (freq + k1 * (1 - b + b * dl / avgdl)) from:',
particulars: [
{
value: 1,
description: 'freq, occurrences of term within document',
details: []
},
{
worth: 1.2000000476837158,
description: 'k1, time period saturation parameter',
particulars: []
},
{
worth: 0.75,
description: 'b, size normalization parameter',
particulars: []
},
{
worth: 3,
description: 'dl, size of area',
particulars: []
},
{
worth: 4.888888835906982,
description: 'avgdl, common size of area',
particulars: []
}
]
}
]
}
]
}
]
}
}
]
So all is there. Right here is the scoring breakdown for “🍏 🍌 🍊”, which produced a rating of 1.0242118835449219.
IDF calculation (inverse doc frequency)
Search consequence:
- Variety of paperwork containing the time period:
n = 1 - Whole variety of paperwork with this area:
N = 9
idf = log(1 + (N - n + 0.5) / (n + 0.5))
= log(1 + (9 - 1 + 0.5) / (1 + 0.5))
= log(6.666666666666667)`
≈ 1.8971199989318848
TF calculation (time period frequency)
Parameters are the Lucene defaults:
- Time period saturation parameter:
k1 = 1.2000000476837158 - Size normalization parameter:
b = 0.75
Doc area statistics:
- Common size of the sphere:
avgdl = 44 / 9 ≈ 4.888888835906982 - Occurrences of the time period on this doc:
freq = 1
tf = freq / (freq + k1 * (1 - b + b * dl / avgdl))
= 1 / (1 + 1.2000000476837158 × (0.25 + 0.75 × (3 / 4.888888835906982)))
≈ 0.5398772954940796
Remaining rating
Parameter:
rating = enhance × idf × tf
= 1.0 × 1.8971199989318848 × 0.5398772954940796
≈ 1.0242118835449219
That confirms that Atlas Search makes use of the identical scoring as Lucene https://github.com/apache/lucene/blob/releases/lucene/10.3.2/lucene/core/src/java/org/apache/lucene/search/similarities/BM25Similarity.java#L183
What about ElasticSearch and Tantivy
Eight years in the past, Lucene eliminated the (k1 + 1) consider LUCENE-8563. For k1 = 1.2, this alteration reduces the rating by an element of two.2 from that model onward. Tantivy and Elasticsearch apparently nonetheless use the previous components, whereas Atlas Search makes use of the up to date one, which explains the noticed variations in scoring.
Conclusion
MongoDB Atlas Search indexes are constructed on Lucene and use its parameters and scoring formulation. Whenever you evaluate Atlas Search with different Lucene‑based mostly textual content engines like google that use older Lucene scoring formulation, you may even see rating variations of roughly an element of two.2. Nevertheless, this has no sensible affect as a result of scores are solely used to order outcomes, so the relative rating of paperwork stays the identical.
Textual content search scores can appear magical, however they’re deterministic and based mostly on open-source formulation. In MongoDB, you’ll be able to embody the rating particulars possibility in a textual content search question to examine all of the parameters and formulation behind the rating.
