Search engines, recommendation systems, and spam filters all need the same basic skill: figure out which words actually matter in a piece of text.
TF-IDF is one of the simplest ways to do that. It scores each word by asking two questions:
- How often does this word show up here?
- How rare is this word everywhere else?
Words that appear a lot in one document but rarely across the collection get a high score. Words like "the" or "a" get pushed down because they appear everywhere.
A tiny example first
Imagine three short notes:
Doc 1: the cat sat on the mat
Doc 2: the dog sat on the log
Doc 3: birds fly in the sky
If you are trying to understand Doc 1, the word cat is a strong signal. It appears in Doc 1, but not in the other two.
The word the appears in every document. It tells you almost nothing about what makes Doc 1 special.
TF-IDF captures exactly that difference.
Step 1: TF (Term Frequency)
TF answers: how common is this word inside one document?
Simple version:
TF = (times the word appears in this doc) / (total words in this doc)
In Doc 1:
- cat appears 1 time out of 6 words → TF = 1/6
- the appears 2 times out of 6 words → TF = 2/6
So the actually has a higher TF than cat inside Doc 1 alone. That is why TF by itself is not enough.
Step 2: IDF (Inverse Document Frequency)
IDF answers: how rare is this word across all documents?
If a word appears in every document, it is not very informative. If it appears in only one, it is much more useful.
Simple version:
IDF = log(total documents / documents containing the word)
Using our three docs:
- the is in all 3 docs → IDF = log(3/3) = log(1) = 0
- cat is in 1 doc → IDF = log(3/1) ≈ 1.10
- dog is in 1 doc → IDF = log(3/1) ≈ 1.10
Common words get a low IDF. Distinctive words get a higher IDF.
Step 3: TF-IDF (put them together)
TF-IDF is just multiplication:
TF-IDF = TF × IDF
For cat in Doc 1:
- TF = 1/6
- IDF ≈ 1.10
- TF-IDF ≈ 0.183
For the in Doc 1:
- TF = 2/6
- IDF = 0
- TF-IDF = 0
So even though the appears more often in the document, cat wins because it is more distinctive across the collection.
That is the whole idea.
Why this is useful
TF-IDF helps systems:
- rank search results
- find important keywords in articles
- build simple recommendation features
- reduce noise from filler words
It is old, simple, and still useful as a baseline before fancier embedding models.
Try it yourself
Selected: the quick brown fox jumps over the lazy dog
TF (term frequency)
| Term | Value |
|---|---|
| quick | 0.1111 |
| brown | 0.1111 |
| jumps | 0.1111 |
| over | 0.1111 |
| the | 0.2222 |
| fox | 0.1111 |
| lazy | 0.1111 |
| dog | 0.1111 |
IDF (inverse document frequency)
| Term | Value |
|---|---|
| quick | 1.0986 |
| brown | 1.0986 |
| jumps | 1.0986 |
| over | 1.0986 |
| the | 0.4055 |
| fox | 0.4055 |
| lazy | 0.4055 |
| dog | 0.0000 |
TF-IDF
| Term | Count | TF-IDF |
|---|---|---|
| quick | 1 | 0.1221 |
| brown | 1 | 0.1221 |
| jumps | 1 | 0.1221 |
| over | 1 | 0.1221 |
| the | 2 | 0.0901 |
| fox | 1 | 0.0451 |
| lazy | 1 | 0.0451 |
| dog | 1 | 0.0000 |
Edit the corpus, switch documents, and watch TF, IDF, and TF-IDF update live. Separate documents with a blank line.
A few things to notice while you play with it:
- Words that repeat inside one doc increase TF
- Words shared across many docs get a lower IDF
- Words that are frequent locally but rare globally get the highest TF-IDF
Where you will see this in the wild
- Classic search indexing pipelines
- Keyword extraction for summaries and tags
- Spam detection (spammy terms often have weird TF-IDF patterns)
- Early-stage content similarity systems
Modern systems often use embeddings, but TF-IDF is still a great first tool because it is fast, interpretable, and easy to debug.
Closing thought
TF-IDF is not magic. It is just a structured way to ask: "Does this word matter here, and is it special compared to everything else?"
If you can explain that sentence to someone, you already understand the core of TF-IDF.