It will come as no surprise when I say that text data has become abundant with the continued advancement of the technology necessary to faciliate its creation and distribution. With so much text, it’s no wonder that many try to utilize the data to improve how we are working. One of the critical methodologies to understand when implementing text data is text similarity.
Text similarity has many essential use cases, such as improving search engine accuracy, detecting plagiarism, or clustering documents. The ability to measure text similarity has become essential to understanding the relationship between text content.
In this article, we will learn several statistical measurements for text similarity. Let’s get into it!
Text Similarity Measurement
When we speak of text similarity, we are referring to the measurement of the quantified resemblance degree between two texts. Commonly, there are two types:
- The lexical similarity involves comparing texts based on surface-level features such as word matching or n-grams overalls
- Semantic similarity focuses more on the meaning behind the text, even with different words in the texts
Using statistical methods, we can quantify the text similarity and provide insight into the text relationship. Let’s start exploring each statistical measurement.
Jaccard Similarity
Jaccard similarity measures the proportion of the shared elements between two texts compared to the total unique words in both texts. The formula can be seen in the image below.
Where A ∩ B is the number of shared or exact words between two texts, while A ∪ B is the total unique words in both texts. The Jaccard similarity measurement works best when the presence of words is much more important than the frequency.
Euclidean Distance
The Euclidean distance measures the distance between two points (vectors) as a straight line. A smaller measurement or distance means the two texts are more similar.
Transforming text into a numerical vector can be done via methods such as TF-IDF if we want to focus on frequency or embeddings if the semantic relationship is essential.
The formula can be shown in the image below.
Where Ai and Bi are the components of the text vectors.
The measurement is intuitive and suitable for text clustering methodology, although it doesn’t normalize any text length information, so it can be biased.
Cosine Similarity
Cosine similarity is a statistical method for computing the cosine angle between two vectors. It works by transforming the text data into numerical vectors and quantifying the orientation.
The cosine similarity formula is shown in the image below.
Where Ai and Bi are the components of the text vectors.
It’s a suitable method for high-dimensional text data and is robust to text length differences.
Levenshtein Distance
Levenshtein distance is a measurement of text similarity that calculates the transformation number of operations required to transform one string into another.
For example, the text kitten and sitting would result in a Levenshtein distance of three:
- k > > s
- e > > i
- adding the g
It’s a suitable method if we want to capture short text or exact matches, but it ignores any semantic meanings. An application that this method would be well suited to would be a spell checker.
And that’s the basic of statistical text similarity measurements.
Any recommendations for semantic similarity metrics?
Hi Brian…Semantic similarity metrics are crucial for comparing text-based data, especially in NLP tasks like clustering, search, recommendation systems, and question-answering. The best metric depends on your specific use case and the level of linguistic understanding required.
### **Traditional (Lexical) Similarity Metrics:**
1. **Jaccard Similarity** – Measures overlap between two sets of words or characters.
2. **Cosine Similarity** – Uses TF-IDF or word frequency vectors to compute the cosine of the angle between two document vectors.
3. **Dice Similarity** – Similar to Jaccard but gives more weight to shared elements.
4. **Levenshtein Distance (Edit Distance)** – Counts the minimum number of single-character edits required to transform one string into another.
### **Word Embedding-Based Metrics:**
5. **Word Mover’s Distance (WMD)** – Computes the minimum distance to “move” word embeddings from one document to another (based on Word2Vec, GloVe, etc.).
6. **Euclidean Distance (on Word Embeddings)** – Measures the straight-line distance between word vectors in embedding space.
7. **Cosine Similarity (on Word Embeddings)** – Computes the cosine angle between word vectors (works well for high-dimensional spaces).
### **Contextual Embedding-Based Metrics (State-of-the-Art):**
8. **BERTScore** – Uses contextual embeddings from BERT to compute similarity.
9. **SBERT (Sentence-BERT) Similarity** – Fine-tuned BERT model for computing sentence embeddings and their cosine similarity.
10. **Universal Sentence Encoder (USE) Similarity** – Uses Google’s USE to generate sentence embeddings for similarity comparisons.
11. **GPT/NLP Model Similarity** – Transformer-based embeddings (e.g., OpenAI’s embeddings API) for deep semantic understanding.
### **Graph-Based and Hybrid Approaches:**
12. **ConceptNet and Knowledge Graphs** – Measures similarity based on structured knowledge representations.
13. **SIF (Smooth Inverse Frequency) Embeddings** – Improves word vector similarity by reducing common word effects.
### **Probabilistic & Statistical Methods:**
14. **KL-Divergence** – Measures the difference between two probability distributions (useful in topic modeling comparisons).
15. **Jensen-Shannon Divergence** – A symmetric version of KL-Divergence that works better for comparing document distributions.
### **Use-Case Recommendations:**
– **Short text similarity** → Cosine similarity on TF-IDF, Jaccard Similarity.
– **Sentence similarity** → SBERT, USE, Cosine Similarity on embeddings.
– **Document similarity** → Word Mover’s Distance, BERTScore.
– **Low-resource tasks** → Edit distance, Jaccard, or Dice Similarity.