The hardest part of building a search engine is building an understanding of user intent, i.e., what the user expects to see when they type in a search query. Serving our customers well means accurately aligning user intent to an understanding of the central concepts of each web page. Often, user queries do not perfectly match the language used in relevant documents to describe the same concept which makes this alignment challenging.


To bridge the gap between query and document, it is generally recommended to do query expansion: supplement customer queries with synonyms, spell-corrected words, and related phrases. These techniques help increase the number of relevant documents retrieved from the search index.


This post is about a novel synonymy relationship which we find can fix a severe flaw in query expansion.

Before we dive into details, let’s consider an example. Consider an example query 

 and score it highly, we need to be able to infer that 

,… — then we should be able to retrieve and favorably score 

. Not so fast! We need to make sure that the added synonyms don’t change the meaning of the query — otherwise we will start retrieving irrelevant documents. E.g. when we swap 

which doesn’t have the same meaning as 

(perhaps it relates to streaming a tv show on screen). So indiscriminately adding all possible synonyms in query expansion is not an option for an effective search engine. With billions of pages in our search index, matching an incomprehensible number of term combinations, we must avoid adding noisy terms in query expansion to prevent matching millions of incorrect pages.

We need a more fine-grained and precise tool than the blunt hammer of synonymy. Here we make a key observation: 

many synonymy relations that are useful for document retrieval exist only in the context of other query terms. We call this relationship Contextual Synonymy. 

Contextual Synonyms are crucial in query expansion for search engines: providing a much better precision-recall tradeoff when it comes to retrieval and ranking than general synonyms.


The rest of this blog post will follow this structure:

Provide details of search retrieval that explain how query expansion is implemented in the search stack.

Formalize our intuition for Contextual Synonymy and justify its importance and novelty.

Explain our process of data collection and describe our experiments for predicting contextual synonyms using machine learning.


We are releasing all annotation guidelines and data we created for contextual synonymy detection to encourage further investigation.

Index-based Retrieval and Contextual Synonymy

Neeva’s index, like other search engines, is structured to optimize the method of searching for all the documents that match a query. Because there are billions of documents on the Internet, and this number is ever-growing, it is not realistic to scan through all the documents on every user query. As such, our search index is an inverted index mapping a term to the set of documents that contain the term. With this inverted index, the cost of finding documents that match a query is proportional to the number of 

 documents, not the total number of documents on the Internet.


To increase the number of documents that are retrievable for a user query, we need to add more terms that can be matched — as the name for query expansion not so subtly suggests, we need to expand the query with more words.

The following query illustrates the power of query expansion: 

By adding the word “top” which synonym for “leading” a result that does not contain the original query term “leading” but does contain “top” (and at least one variation of each of the other query terms) will be retrievable from the search index.

When analyzing retrieval losses on our internal dataset, we found that 50% of our retrieval losses (i.e. cases where we fail to retrieve the most relevant documents) were due to shortcomings in our query expansion. As argued earlier, it is a bad idea to indiscriminately add a lot of synonyms (e.g. expand 

 by all possible synonyms.) We found that the missing words can be determined in a more clever way — these words were semantically equivalent to original query terms, conditioned on the presence of the other query terms. E.g. “display” and “monitor” are equivalent in the context of “laptop”; so they can be used interchangeably and maintain semantically similar queries: 

[("how to connect my laptop to display", "how to connect my laptop to monitor"), ("multiple displays for laptop", "multiple monitors for laptop"…] 

have very different user intent that is best not to conflate to avoid angering hungry dieters.

 aims to quantify the degree of semantic equivalence of two words in a shared context. Concretely, our goal is to determine whether two words 

 are synonymous with each other in the context of some other words 

 can be a phrase, a word, or any other metadata. In contrast, a subset of Contextual Synonyms is General Synonymy (GS): two words are general synonyms if they are contextual synonyms without the support of other words (this is just a fancy way of saying words like “buy” and “purchase” are just plain synonyms).

Why Contextual Synonymy is not the same as Semantic Textual Similarity (STS)?

Semantic Textual Similarity (e.g. as proposed 

) seeks to measure the semantic equivalence of two pieces of text. However it does not imply anything — at least not directly — about the terms in those texts. Contextual Synonymy is more fine-grained because it is applicable at the term level. This makes it much more generalizable and effective for search engines.

As a concrete example, semantic textual similarity simply says 

 are identical and nothing more — learning this relationship allows us to improve exactly 2 queries (the texts themselves).

 — this fact enables us to generate a lot of STS relationships because now we can say 

[("multiple displays for laptop", "multiple monitors for laptop"), ("laptop not recognizing display", "laptop not recognizing monitor")…] 

are all identical and thereby can cover a lot more customer queries!

Due to its importance and novelty, we treat Contextual Synonymy as a task in and of itself. We are using the word "task” akin to describing new problems as is often done in NLP papers e.g. the 

 task which has released guidelines and datasets (both of which we release as well). We will show techniques for CS that will use semantic textual similarity as well as search result overlap to accurately align our predictions of good contextual synonyms with human judgement.


The following table highlights more such examples of contextual synonyms and the contexts.

Before we proceed, just something to note: the notion of synonymy is very nuanced. Sometimes even the obvious 

, have semantic differences in certain contexts. This nuance helps crystallize our problem space.

how to increase public good in the world?

how to increase public goods in the world?

Albeit both queries are almost identical, it is plausible that the intent of the first query is “how to increase public wellbeing and health,” whereas the intent of the second query is “how to increase commodities or services to the public.” The terms “good” and “goods” take on different meanings in the contexts of these query, and notably in the context of the word “public.” 

To date, even other search engines have a hard time capturing the subtle differences in intents of these queries:

Google captures a single dominant user intent.

To provide a more holistic search experience, Neeva captures several variations of user intent.

Experiments to generate contextual synonyms

This section will describe our method of synonym candidate generation, data collection, and using machine learning for classifying contextual synonyms.

We first generate a large number of contextual synonym candidates using data from web search sessions. The process uses a simple insight: for two semantically similar queries that share a set of terms, the other terms(that are different) are good candidates for contextual synonymy. We identify semantically similar pairs of queries as two queries that have high result overlap overlap in their top 5 results. Contextual synonym candidates are extracted from pairs of semantically similar queries using the following simple algorithm:

1. Consider a query pair (query1, query2)

query2: [iphone, app, alternative, itunes]

This post is about a novel synonymy relationship which we find can fix a severe flaw in query expansion.

Daniela Gottesman

Contextual Synonyms