Topic Taxonomy
While .hierarchical_topics() explores topic similarity through a dendrogram and .reduce_topics() merges topics to a target count, .create_topic_taxonomy() takes a different approach: it builds a structured, multi-level taxonomy where you control the number of levels, the number of topics per level, and the minimum number of children each parent must have.
The leaf topics produced by BERTopic stay untouched and are still used for inference. Parent topics are constructed bottom-up through agglomerative clustering, one level at a time, producing a lookup table that maps every leaf to its parent, grandparent, and so on.
Example¶
We train a BERTopic model on the 20 Newsgroups dataset and then build a two-level taxonomy on top of the resulting leaf topics:
from bertopic import BERTopic
from sklearn.datasets import fetch_20newsgroups
docs = fetch_20newsgroups(subset='all', remove=('headers', 'footers', 'quotes'))["data"]
topic_model = BERTopic(verbose=True)
topics, probs = topic_model.fit_transform(docs)
hierarchy = topic_model.create_topic_taxonomy(
docs,
nr_topics_per_level=[20, 5],
min_children=2,
)
The result is a DataFrame describing the full taxonomy. Each row contains a topic's ID, name, hierarchy level, and a reference to its parent. Top-level topics have Parent_ID = -2:
>>> print(hierarchy.to_string(index=False))
Topic_ID Topic_Name Level Parent_ID Parent_Name Document_Count
0 0_game_team_games_he 0 241 game_team_he_games_players 1834
1 1_key_clipper_chip_encryption 0 253 key_encryption_clipper_chip_keys 603
2 2_israel_israeli_jews_arab 0 240 armenian_were_armenians_turkish_they 459
8 8_car_cars_mustang_ford 0 265 car_bike_cars_engine_miles 169
13 13_bike_bikes_miles_honda 0 265 car_bike_cars_engine_miles 169
...
265 car_bike_cars_engine_miles 1 276 bike_car_insurance_my_you 1104
...
276 bike_car_insurance_my_you 2 -2 Root 3820
...
Here, leaf topics 8 and 13 both map to parent 265 (car_bike_cars_engine_miles), and parent 265 maps to grandparent 276 (bike_car_insurance_my_you). You can verify the structure:
>>> hierarchy.groupby("Level").size()
Level
0 225
1 45
2 15
dtype: int64
How it works¶
1. Compute leaf topic vectors¶
For every leaf topic (so the ones generated by BERTopic), a topic vector is computed. If document embeddings are provided, the vector is a weighted blend of the document centroid and BERTopic's internal topic embedding:
Topic Vector = w * Document Centroid + (1 - w) * Topic Embedding
The weight w is controlled by doc_embedding_weight (default 0.5). Without document embeddings, the topic embedding is used directly. It's recommended to at least experiment with using the document embeddings, as the produced taxonomy tends to be more accurate through it.
2. Build each parent level¶
For each level above the leaves:
- Cluster the topic vectors using agglomerative clustering with cosine distance
- Repair any cluster with fewer than
min_childrentopics by merging it into the most similar neighbor. Centroids are recomputed after each merge. - Label each parent using c-TF-IDF on the combined bag-of-words of its children
- Compute the parent vector as a weighted centroid of its children (weighted by document count)
3. Finalize¶
Top-level topics are marked with Parent_ID = -2.
Parameters¶
hierarchy = topic_model.create_topic_taxonomy(
docs,
nr_topics_per_level=[20, 5],
embeddings=embeddings, # optional document embeddings
min_children=2, # minimum children per parent
doc_embedding_weight=0.5, # blend weight for doc centroids
use_ctfidf=False, # use c-TF-IDF for distance computation
)
| Parameter | Default | Description |
|---|---|---|
nr_topics_per_level |
required | Target topic count for each level above the leaves. E.g., [20, 5] creates two parent levels. Each entry must be less than the count at the level below. |
embeddings |
None |
Document embeddings for computing per-topic centroids (recommended). If not provided, only BERTopic's topic embeddings are used. |
min_children |
2 |
Minimum children per parent at every level. Clusters with fewer children are merged into their most similar neighbor. |
doc_embedding_weight |
0.5 |
Weight for document centroids vs. topic embeddings. Only used when embeddings is provided. |
use_ctfidf |
False |
Whether to use c-TF-IDF representations instead of topic embeddings for distance computation. |
Why the topic count is approximate¶
The values in nr_topics_per_level are targets, not exact guarantees. The actual number of topics at each level may end up slightly lower. This is because of the repair step.
The algorithm first creates exactly the requested number of clusters using agglomerative clustering. But some of those clusters may contain fewer than min_children topics. The repair step then merges each undersized cluster into its most similar neighbor, which reduces the total count.
For example, if you request 20 parents from 60 leaves with min_children=2, the clustering produces 20 clusters. If three of those clusters happen to contain only a single leaf, they get merged into nearby clusters, leaving you with 17 parents instead of 20.
This effect becomes more pronounced as you increase nr_topics_per_level relative to the number of topics at the level below. More clusters means fewer topics per cluster on average, which means more clusters fall below the min_children threshold and get merged away. In practice, keeping the ratio between consecutive levels at roughly 3:1 or higher tends to produce counts close to the target.
Note
If you set min_children=1, no repair merging takes place and you will get exactly the requested number of topics at each level.
Using document embeddings¶
If you saved the document embeddings during training, you can pass them in for richer topic vectors:
embedding_model = SentenceTransformer("all-MiniLM-L6-v2")
embeddings = embedding_model.encode(docs)
topic_model = BERTopic(embedding_model=embedding_model)
topics, probs = topic_model.fit_transform(docs, embeddings)
hierarchy = topic_model.create_topic_taxonomy(
docs,
nr_topics_per_level=[20, 5],
embeddings=embeddings,
doc_embedding_weight=0.7,
)
A higher doc_embedding_weight puts more emphasis on how documents are distributed in embedding space. A lower weight relies more on BERTopic's topic embeddings, which capture word-level topic structure.
Controlling taxonomy shape¶
The min_children parameter ensures every parent has enough children to be meaningful:
min_children=2 (default) min_children=3
Grandparent A Grandparent A
├── Parent 1 ├── Parent 1
│ ├── Leaf 1 │ ├── Leaf 1
│ └── Leaf 2 │ ├── Leaf 2
└── Parent 2 │ └── Leaf 3
├── Leaf 3 └── Parent 2
└── Leaf 4 ├── Leaf 4
├── Leaf 5
└── Leaf 6
When a cluster has fewer children than required, it is merged into its most similar neighbor based on cosine similarity between their centroids, iteratively, until all clusters satisfy the constraint.
hierarchy = topic_model.create_topic_taxonomy(
docs,
nr_topics_per_level=[15, 5],
min_children=3,
)
Tip
create_topic_taxonomy() does not modify the fitted model. Your leaf topics, representations, and probabilities remain exactly as they were. The taxonomy is a read-only lookup table on top of the existing model.
Navigating the taxonomy¶
The returned DataFrame makes it straightforward to traverse the hierarchy:
# Get all parents at level 1
parents = hierarchy[hierarchy.Level == 1]
# Find which leaves belong to a specific parent
parent_id = parents.iloc[0].Topic_ID
children = hierarchy[hierarchy.Parent_ID == parent_id]
# Trace a leaf topic up to the root
leaf = hierarchy[hierarchy.Topic_ID == 0]
parent = hierarchy[hierarchy.Topic_ID == leaf.iloc[0].Parent_ID]
grandparent = hierarchy[hierarchy.Topic_ID == parent.iloc[0].Parent_ID]