Before embeddings and transformers, most text systems started the same way: turn a sentence into numbers.
Bag of Words (BoW) is the simplest version of that idea. You throw away word order, keep counts, and treat each document like a multiset of tokens in a bag.
It is not fancy. But it is the baseline that makes TF-IDF, Naive Bayes classifiers, and classic search features easier to understand.
What bag of words actually does
Take a sentence and split it into tokens:
"the cat sat on the mat"
→ [the, cat, sat, on, the, mat]
BoW ignores order. It only asks: how many times does each word appear?
the: 2
cat: 1
sat: 1
on: 1
mat: 1
That count map is the bag of words for this document.
If you have multiple documents, you line those maps up into a vocabulary matrix — one column per document, one row per unique word.
Same tiny corpus as TF-IDF
Use the same three notes from the TF-IDF post:
Doc 1: the cat sat on the mat
Doc 2: the dog sat on the log
Doc 3: birds fly in the sky
Here is the BoW view for Doc 1:
| Term | Doc 1 | Doc 2 | Doc 3 |
|---|---|---|---|
| the | 2 | 2 | 0 |
| cat | 1 | 0 | 0 |
| sat | 1 | 1 | 0 |
| on | 1 | 1 | 0 |
| mat | 1 | 0 | 0 |
| dog | 0 | 1 | 0 |
| log | 0 | 1 | 0 |
| birds | 0 | 0 | 1 |
| fly | 0 | 0 | 1 |
| in | 0 | 0 | 1 |
| sky | 0 | 0 | 1 |
BoW tells you what is present and how often. It does not tell you whether a word is special.
Raw counts vs normalized counts
There are two common BoW variants:
| Variant | What it stores | Formula |
|---|---|---|
| Raw counts | Integer word counts | count(word) |
| Normalized counts | Share of the document | count(word) / len(document) |
Raw counts — just count appearances:
bow[word] = document.count(word)
Normalized counts — divide by document length so longer docs do not dominate:
bow[word] = document.count(word) / len(document)
That normalized version is basically term frequency (TF). Same idea, different name.
So BoW with normalization gets you halfway to TF-IDF. The missing piece is IDF — penalizing words that show up in almost every document.
BoW vs TF-IDF on the same document
For Doc 1, BoW ranks by raw count:
the → 2
cat → 1
sat → 1
on → 1
mat → 1
the wins because it appears twice.
TF-IDF asks a second question: is this word rare across the whole collection?
- the is in every document that shares its vocabulary context heavily → low IDF
- cat appears only in Doc 1 → higher IDF
So TF-IDF pushes cat above the, even though the has the higher raw count.
| Word | BoW count (Doc 1) | TF-IDF intuition |
|---|---|---|
| the | 2 | Common everywhere → score collapses |
| cat | 1 | Rare in the collection → score rises |
| mat | 1 | Also distinctive → score rises |
That is the core difference:
- BoW = what words are here, and how many?
- TF-IDF = what words matter here compared to the rest of the corpus?
When each one is enough
Bag of words is enough when:
- you need a simple numeric representation fast
- word presence matters more than rarity (e.g. spam keyword spotting)
- you are feeding features into a classic model like Naive Bayes
- document length is similar across your dataset
TF-IDF is better when:
- common words pollute your rankings
- you are comparing documents of very different lengths
- you want keyword extraction or search relevance
- you need a stronger baseline before embeddings
A common pipeline looks like this:
text → tokenize → bag of words → (optional) TF-IDF weighting → classifier / search / similarity
Scikit-learn makes this explicit: CountVectorizer builds the bag of words, and TfidfVectorizer applies TF-IDF on top of those counts.
A practical use case: sentiment analysis
Train a tiny sentiment model on six reviews:
"I loved this movie, it was fantastic!" → positive
"Worst experience ever, totally hated it." → negative
"The food was absolutely delicious and great." → positive
...
Steps:
- Build a BoW matrix with
CountVectorizer - Each review becomes a vector of word counts
- Feed those vectors into
MultinomialNB - Predict whether a new review is positive or negative
The model never sees grammar or word order. It learns which words tend to co-occur with each label. BoW is crude, but for short text classification it still works surprisingly often.
Try it yourself
Selected: the quick brown fox jumps over the lazy dog
| Term | Doc 1 | Doc 2 | Doc 3 | TF-IDF |
|---|---|---|---|---|
| a | 0 | 0 | 1 | — |
| across | 0 | 1 | 0 | — |
| all | 0 | 0 | 1 | — |
| brown | 1 | 0 | 0 | 0.1221 |
| chased | 0 | 1 | 0 | — |
| day | 0 | 0 | 1 | — |
| dog | 1 | 1 | 1 | 0.0000 |
| field | 0 | 1 | 0 | — |
| fox | 1 | 1 | 0 | 0.0451 |
| jumps | 1 | 0 | 0 | 0.1221 |
| lazy | 1 | 0 | 1 | 0.0451 |
| long | 0 | 0 | 1 | — |
| over | 1 | 0 | 0 | 0.1221 |
| quick | 1 | 0 | 0 | 0.1221 |
| sleeps | 0 | 0 | 1 | — |
| the | 2 | 3 | 0 | 0.0901 |
BoW ranking (raw counts in selected doc)
TF-IDF ranking (same document)
Edit the corpus, switch documents, and compare the count matrix with the BoW vs TF-IDF rankings below it. Use the same three-document example from the TF-IDF article and watch how the tops BoW but falls behind distinctive words under TF-IDF.
Things to notice:
- BoW counts are integers — simple and interpretable
- Words with zero count in a document still appear as rows in the shared vocabulary
- Normalizing BoW counts turns them into TF; TF-IDF adds the cross-document penalty
Closing thought
Bag of words is the counting layer. TF-IDF is the weighting layer on top.
If you understand BoW, you already understand half of TF-IDF. The other half is just asking which counts should actually count.