By: Viral Roast Team · Cluster article · April 2026
A team of eleven Meta researchers just published a benchmark that asks a simple question: can large language models actually understand what a user wants from their engagement history? The answer, measured across eight open-weight LLMs and five datasets, is “sort of, but badly.”
The benchmark is called GISTBench (arXiv: 2603.29112, March 31 2026). It evaluates LLM user understanding through evidence-based interest verification — testing whether AI models can extract real interests from how people interact with short-form video. Not what the model guesses. What it can prove from the data.
If you build AI-powered content recommendations, run audience profiling with language models, or use any tool that claims to “understand your audience,” this paper should concern you. The best model tested — a 120-billion-parameter system — managed 67.8% accuracy at identifying user interests. The worst scored zero. Literal zero.
At Viral Roast, we analyze video content before creators publish it. We care about this research because every AI system that profiles audiences faces the same problem GISTBench exposes: models are decent at not making things up, but terrible at capturing what people actually care about.
What GISTBench Measures and Why Existing Benchmarks Missed It
GISTBench introduces two metric families that did not exist before. Interest Groundedness (IG) checks whether an LLM's predicted interests are backed by actual engagement evidence. Interest Specificity (IS) checks whether those predictions are distinct enough to be useful, or just generic noise like “this user likes entertaining content.”
The groundedness metric splits into precision and recall. IG Precision penalizes hallucinated interest categories — things the model claims a user likes with no evidence. IG Recall rewards coverage — catching all the things a user genuinely engages with. The F1 score combines both.
Previous benchmarks in this space — PersonaMem, RecBench+, AgentSociety — tested LLMs on simulated conversations or query-response pairs. GISTBench is the first to verify model predictions against actual engagement evidence from a real video platform. That distinction matters. Simulated data lets models cheat by pattern-matching to synthetic structures. Real engagement data is messy, contradictory, and full of implicit signals that don't map cleanly to stated preferences.
Why this matters: If your AI tool says “your audience loves fitness content” but the evidence shows they only watched 3 fitness videos out of 200, that is a hallucinated interest. GISTBench catches exactly this failure mode.
The Scoreboard: Eight Models, Five Datasets, One Uncomfortable Pattern
Meta tested eight open-weight LLMs ranging from 7 billion to 120 billion parameters across five datasets: their own synthetic short-form video dataset (built on real user interactions), KuaiRec (Kuaishou's recommendation dataset), Microsoft MIND (news), Amazon Music, and Goodreads.
| Model | Size | Video IGF1 | Video IS | KuaiRec IGF1 |
|---|---|---|---|---|
| GPT-OSS-120B | 120B | 67.8% | 74.0% | 56.3% |
| DeepSeek-R1 | 671B (MoE) | 56.5% | 79.0% | 44.1% |
| Qwen3-VL-235B | 235B (22B active) | 50.8% | 73.4% | 55.2% |
| Qwen3-32B | 32B | 46.1% | 74.5% | 45.8% |
| Qwen2.5-72B | 72B | 45.5% | 65.1% | 43.3% |
| DeepSeek-R1-Distill-70B | 70B | 37.3% | 78.0% | 42.9% |
| Qwen2-7B | 7B | 14.2% | 31.8% | 26.5% |
| Llama-3.1-8B | 8B | — | — | 0.0% |
The pattern across every dataset and every model is the same: precision is high, recall is low. DeepSeek-R1 hit 82.4% precision on the video dataset but only 43.9% recall. GPT-OSS-120B reached 78.1% precision and 60.5% recall — still missing almost 40% of real user interests.
Translation: these models are conservative. They don't invent many fake interests. But they miss most of what people actually like. If you're using an LLM to build audience profiles, your profiles are probably accurate in what they include — and blind to what they leave out.
Small Models Cannot Do This Job
Llama-3.1-8B produced zero parsable outputs on four of five datasets. Not low scores. Zero. The model could not follow the task format well enough to generate usable interest predictions.
Qwen2-7B performed only slightly better: 14.2% IGF1 on video data, 31.8% specificity. Both numbers mean the model's predictions are too generic and too incomplete to be useful for anything.
The practical floor for user interest extraction appears to be around 30 billion parameters. Below that, models lack the capacity to process long interaction histories and produce structured, grounded predictions. Anyone deploying a 7-8B model for audience understanding is getting noise, not signal.
Reasoning Models Produce Sharper Profiles That Are Still Incomplete
DeepSeek-R1 — a reasoning-focused model — scored highest on Interest Specificity across three of five datasets (79.0% on video, 71.0% on KuaiRec). Its profiles were more distinctive and granular than competitors. A good sign if your goal is telling users apart.
But its recall lagged GPT-OSS-120B by 16+ percentage points. The reasoning model thinks harder about the interests it does identify, producing more precise categories. It also identifies fewer of them. More careful does not mean more complete.
For content strategy this creates a real tradeoff. A specific but narrow profile tells you something useful about a slice of your audience. A broad but generic profile tells you nothing useful about anyone. Neither is what you actually want.
What GISTBench Tells Creators About AI-Powered Audience Tools
Every AI tool that claims to “understand your audience” faces the exact problem GISTBench quantifies. The models powering these tools are missing 40-60% of what your audience actually cares about. They're not making things up — they're leaving things out.
This has direct consequences for how you should use AI audience insights:
- Treat AI audience profiles as a starting point, not ground truth. If the best 120B model misses 40% of real interests, any tool built on smaller models is missing more. Use AI suggestions as one input alongside your own audience knowledge.
- Watch for the “everyone likes funny videos” problem. GISTBench's Interest Specificity metric exists because generic predictions are useless. If your AI tool tells you your audience likes “entertaining content” or “high-quality videos,” it is telling you nothing. Demand specifics.
- Implicit signals are where models struggle most. The paper found that 92-99% of failed interest predictions lacked sufficient explicit positive signals. Likes, shares, and saves are easier for models to interpret than watch time and scroll behavior. The subtler your audience's engagement pattern, the more likely AI will miss it.
- More context helps, but not as much as you'd hope. Increasing context length improved recall by 10.8 percentage points but specificity by only 3.9. Throwing more data at the model produces marginal returns.
Viral Roast perspective: We built our analysis system around video structure and content signals rather than audience profiling for exactly this reason. Predicting what makes a video work based on its structural properties is a more tractable problem than modeling audience interests from engagement history. GISTBench confirms that the audience-modeling side of AI still has serious gaps.
How the Benchmark Was Built
The core dataset was constructed from real user interactions on what Meta describes as a “global short-form video platform” — almost certainly Instagram Reels or a similar Meta property, given the eleven-author team is entirely from Meta's recommendation systems group.
The dataset includes 1,000 user cohorts (each with 10+ distinct users), 30-day interaction windows, and both implicit signals (extended watch time, skips) and explicit signals (likes, comments, shares, saves, follows). A proprietary Vision Language Model generated textual descriptions of video content, which were then joined to the interaction data.
To validate the benchmark, Meta surveyed 593 real users and compared their self-reported interests against the metric scores. The Spearman correlation was 0.67 — strong enough to confirm the metrics capture something real, modest enough to remind you that self-reported interests and actual behavior diverge in predictable ways.
GISTBench also tested on four external datasets — KuaiRec, MIND, Amazon Music, Goodreads — to confirm the results generalize beyond Meta's own platform. The same precision-recall gap appeared everywhere.
Where GISTBench Fits in the LLM Recommendation Research
This is not the only recent benchmark testing LLMs for user understanding. PersonaMem (UPenn, 2025) tested dynamic user profiling across multi-turn conversations and found that frontier models like GPT-4.1 achieve only about 50% accuracy on evolving user preferences. RecBench+ (WSDM 2025) evaluated LLMs as recommendation assistants and confirmed that all models struggle with implicit or ambiguous queries.
Google's deployed research (RecSys 2024) showed that hybrid LLM + classic recommendation systems outperform either approach alone for discovering new user interests at billion-user scale. And the AgentSociety Challenge at TheWebConf 2025 attracted 295 teams working on LLM-based user modeling, with a 21.9% improvement in the user modeling track over baseline.
GISTBench's contribution is the evidence-based verification approach. Previous work measured prediction accuracy against held-out items or simulated preferences. GISTBench requires the model to show its work — every predicted interest must be traceable to specific engagement evidence. This is harder and more realistic.
Three Things to Do With This Information
If you're a creator relying on AI tools for audience insights, adjust your expectations down. The science says the best models miss 40% of what your audience cares about. Calibrate accordingly.
If you're building AI products in the recommendation space, GISTBench gives you two clear metrics to evaluate your system against. IG and IS are open, reproducible, and the code is on GitHub. Use them.
And if you're choosing between a smaller model (under 30B) and a larger one for any task involving user understanding, GISTBench settles the debate. Small models produce noise. The minimum viable size for this task is 30B parameters, and even 120B leaves significant gaps.
Frequently Asked Questions
What is GISTBench?
GISTBench is a benchmark published by Meta's recommendation systems team in March 2026 that tests whether large language models can accurately identify user interests from engagement data. It introduced two new metrics: Interest Groundedness, which checks if predicted interests have real evidence behind them, and Interest Specificity, which checks if those predictions are distinctive enough to be useful. Think of it as a test that asks AI models to prove they actually understand what someone likes based on what they watched, liked, and shared — not just guess based on stereotypes.
Why should content creators care about GISTBench?
Because every AI tool that claims to understand your audience uses the same type of models GISTBench tested. And those models miss 40-60% of real audience interests. If you're making content decisions based on AI-generated audience profiles, you should know that those profiles are probably accurate for what they mention but blind to a large chunk of what your audience actually engages with. It's like getting a restaurant recommendation from someone who only remembers half the meals they've eaten.
Which AI model performed best on GISTBench?
GPT-OSS-120B came out on top with 67.8% IGF1 on the video dataset, beating DeepSeek-R1 and all Qwen variants. Its advantage was recall — it captured more real interests than competitors, even though its precision was slightly lower. DeepSeek-R1 produced the most specific and distinctive profiles but missed more interests overall. No model scored above 68% on the video data, which tells you the state of the field.
Can small LLMs do audience profiling?
No. GISTBench showed that Llama-3.1-8B scored zero on four of five datasets — it couldn't even follow the task format. Qwen2-7B barely functioned at 14.2% accuracy on video data. The practical floor is around 30 billion parameters. Below that, models lack the capacity to process long interaction histories and produce anything meaningful. If your tool runs on a 7-8B model, its audience understanding is essentially random.
How does Viral Roast use research like GISTBench?
We track academic research on LLM capabilities and limitations to inform how we build our analysis system. GISTBench reinforced our decision to focus on video structural analysis rather than audience profiling — predicting virality from content properties is currently more reliable than modeling audience interests with LLMs. When the models get better at user understanding (and they will), we'll integrate those capabilities. Until then, we'd rather give you accurate structural analysis than unreliable audience guesses.
Where can I read the full GISTBench paper?
The paper is available on arXiv (2603.29112) and the code is open-sourced on GitHub (facebookresearch/GISTBench). It's a dense read with 9 figures and 20 tables, but the core finding is straightforward: LLMs are conservative predictors that miss most real user interests while rarely hallucinating fake ones. The precision-recall gap is the headline you need to remember.