fineweb-edu-fortified
Deduplicated FineWeb-Edu with embeddings and occurrence counts.
Open dataset Updated 2026-06-18
What is fineweb-edu-fortified?
Fineweb-Edu-Fortified contains deduplicated rows from FineWeb-Edu, each augmented with a bge-micro embedding and an occurrence count.
It is suited for text-generation workloads that require cleaned educational web text at large scale with precomputed embeddings.
What you can build with fineweb-edu-fortified
Train deduplication-aware language models
Use the duplicate count column to weight samples or filter repeats when fine-tuning models on educational web text.
Build embedding-based retrieval systems
Leverage the included bge-micro embeddings to prototype semantic search or clustering over cleaned educational content.
Analyze deduplication impact on datasets
Compare model performance or data statistics before and after exact-match deduplication using the fortified version.
Load fineweb-edu-fortified
Python
from datasets import load_dataset
ds = load_dataset("airtrain-ai/fineweb-edu-fortified")- 1pip install datasets
- 2from datasets import load_dataset
- 3ds = load_dataset('airtrain-ai/fineweb-edu-fortified')
- 4Inspect columns: embeddings, count, and text fields
- 5Filter rows by count > 1 to study duplicates
fineweb-edu-fortified: pros & cons
Pros
- +Exact-match deduplication applied
- +Pre-computed embeddings included
- +Derived from educational web data
- +Duplicate count metadata provided
Cons
- –Based on 500k-row sample only
- –Embeddings fixed to bge-micro model
- –No full original FineWeb-Edu scale
Did you find this helpful?
Frequently asked questions
A deduplicated version of FineWeb-Edu with added embeddings and duplicate counts from a 500k sample.
User reviews
Verified reviews from the community shape this listing's rating.
Loading reviews…
Similar datasets
Other text & nlp options worth comparing.