# Biased by Design: Building Representative Video Datasets _Last updated: March 2026 — this is a living document. I'll update it as I make progress on this project._ ## The Problem If you're optimizing a streaming pipeline, you already know that your test dataset matters. The closer your dataset matches what you actually stream to your users, the more confidently your testing improvements will translate to production. That means building a biased dataset, intentionally. You're not doing generic research. Biasing toward your use cases is the whole point. But there are actually two kinds of bias at play, and most teams only think about one of them. The first is the bias you choose: selecting sequences that reflect your platform, your content, your users. That's good engineering. The second is the bias you don't see: your dataset reflects how you _perceive_ your service, not what it actually is. And those two things are often further apart than you'd expect. I suspect a lot of teams over-bias their catalogues based on intuition rather than data. When you break content into segments, you get encodes spanning a huge range of rate-distortion characteristics. Some are easy to compress, some are brutal. A catalogue assembled by gut feel, even by an expert, will systematically over-represent certain types of content and miss others, and you won't know which. That's the hidden variable corrupting your results. ## The Approach The fix is to stop curating by instinct and let the data speak instead. Analyze your segments systematically, classify them, and build a dataset that captures the real distribution of what you stream. Your most popular titles are a logical starting point. They reflect actual user choices and carry the most streaming weight. The most promising technique I've come across is described in: > _Tmar, H., Wu, P.-H., Stejerean, C., Lei, Z., and Katsavounidis, I. "Benchmarking Hardware and Software Encoder Quality and Performance." SPIE 2024._ The authors needed a representative test set for their encoder benchmarking work at Meta. Rather than handpicking clips, they started with a pool of 322 clips, encoded all of them with SVT-AV1 across 8 CRF values and 5 resolutions, and computed VMAF scores at each operating point. They then ran k-means clustering on the resulting rate-distortion points, and selected the clip closest to each cluster center as the representative for that cluster. The result was a curated set of 200 clips that captured the diversity of the original 322, without having to run expensive benchmarks on all of them. Instead of human judgment deciding what matters, the structure of the data does. Rather than asking "what content do I think is representative?", you ask "what does the data say is representative?" That's a meaningful shift, and one I think has a lot of practical potential for streaming teams beyond just benchmarking. ## Open Questions This is where it gets interesting, and honest. The core idea is elegant, but implementing it well raises more questions than it answers. Here's what I'm working through: **What value of k?** How many clusters should you use? Too few and you lose diversity, too many and the dataset becomes unmanageable. **Is VMAF the right metric?** VMAF is a strong perceptual quality metric, but is it the best axis for clustering encode complexity? Are there better alternatives or complementary metrics? **Does the codec matter?** Do the clusters hold across codecs, or does everything shift between H.264 and AV1? If you build your dataset on H.264 encodes, is it still representative when you switch to AV1? **What about encoding parameters?** Do different presets or CRF values shift the clusters significantly? **How do you present results?** Do you aggregate metrics across clusters into a single number, or report them per cluster? Per-cluster reporting is more informative but harder to act on. **How often do you rebuild the dataset?** As your catalogue evolves, your clusters will shift. How do you keep the dataset current without rebuilding it from scratch every time? ## Experiments & Findings _This section is empty for now, but not for long. I'll document my experiments here as I run them._ ## Resources - Tmar, H., Wu, P.-H., Stejerean, C., Lei, Z., and Katsavounidis, I. _Benchmarking Hardware and Software Encoder Quality and Performance._ SPIE 2024 — the paper that inspired this line of thinking ## Work With Me Building a representative dataset is one of those problems that looks straightforward until you're in the middle of it. If your team is trying to get more confidence out of your test results, or you're not sure your current dataset is really capturing what you stream, this is exactly the kind of challenge I work on with streaming teams. I'm also actively researching this topic and open to collaborating with teams who have real data and real constraints to work with. If that sounds interesting, reach out via [LinkedIn](https://www.linkedin.com/in/luctrudeau) or through the [Consulting](https://trud.ca/Consulting) page.