8.3 billion personas, zero real outcomes — so we measured ours

Dekstr builds persona panels from a customer's own data and calibrates them against real outcomes. On 2 September 2026 Dekstr tested its own panel against an outcome it does not control, and the uncalibrated panel failed. Here is the whole thing.

The test

The claim we wanted to be able to make was narrow: can a panel pick the winning ad creative? Not "is the panel consistent" — the engine already showed consistency internally (A=51±2.7, B=58.6±1.0, B won 5/5 runs), and consistency is not accuracy.

So Dekstr used an external proxy: how long a creative stays live in the Meta Ad Library. One pair = one advertiser's creative that ran ≥21 days, against a creative from the same advertiser that was killed in ≤7 days and had ended. The winner's long-running ad had to be live while the loser was live too, so a fully paused campaign can't fake the difference. Same-copy pairs excluded — image testing is not our measurement. All thresholds sit in one place, PAIR_RULE, each with its reason next to it.

Germany, not Turkey. Under the DSA, the Ad Library shows inactive EU ads with an end date for a year; in TR only active ads are visible, so the loser can't be selected. TR needs a two-snapshot design (T0→T1) and gets its own run.

What got tested is the production path itself: the same prompt, the same acceptance gate, the same model the product runs. And five non-negotiable constraints held by code, not promises: the panel never sees the creatives; calibration off; the backtest process cannot reach the production database; a control arm — the same model with the same product context but no personas, plus seven deterministic dumb baselines; and the primary metric is pairwise ranking accuracy with confidence intervals.

One more thing: the panel was tested in its weakest form. We hold no analytics for external advertisers, so the evidence bundle was empty, every persona was declaration-only, and the declaration itself was model-written from public page info. The number below is the strength of the panel a customer sees on day one — before their data and their results enter the loop.

The result

150 pairs, 37 advertisers, 3 independent runs per judgment, about $50 total.

Panel minus LLM: 0.0 points [−9.3, +8.7], McNemar p=1. The panel's own "difference is above noise" flag carried no information (47.3%, n=131). Reading: an uncalibrated, declaration-only panel does not discriminate on this proxy, and the persona layer adds nothing over a single LLM.

Then we tried to fix it

Diagnosis first: score and outcome were unrelated (winners averaged 30.9, losers 32.4), and the panel picked the first-presented variant in only 35% of pairs — a position bias. Two fixes shipped to production: the pre-test prompt moved to feed context (a persona scrolling on a phone; 0 = scrolls past, ~60 = stops and reads, 80+ = acts), and each variant now gets its own call, so position bias is structurally gone.

Same frozen 150 pairs, v2: 45.7% [38–54] — statistically identical to v1's 46.7% (paired difference −1.0 points [−7.7, +6.3], p=0.47). The mechanics improved; the discrimination did not. Then one hook-only probe — the judge saw only the first line of each creative (60 pairs): 37.5% [26–50]. Three framings, same place. The signal in the data is real ("longer headline" 64.7%, "short body" 56.7%); the LLM judges just don't use it. Prompt iteration stopped here — each run costs ~$50, and past this point you're fitting the dev set, not learning.

What MatrAIx changes

On 3 September Dekstr read arXiv 2608.04205: MatrAIx — ~100 authors led out of Harvard/MIT, funded by OpenAI, Anthropic, Microsoft, AWS and Meta. 1,290 categorical persona dimensions, 8.3 billion records, a ~1M open coreset, 1,010 task templates, 18,189 trials. Validation: behavioral fidelity 91.5%, persona inference quality 4.135/5.

What they do not measure is real-world outcome. Their own words: "task success is not treated as evidence of human validity," and the closing line — "simulate diverse users, then validate against reality." Nobody does the "then." Fidelity is not accuracy: a panel can be 91.5% faithful to its own card and still be a coin flip on real ranking. Dekstr measured exactly that, at 46.7%.

What's next

Building panels is becoming free infrastructure, and 1,290 dimensions aren't the bottleneck: our v0→v2 runs show persona detail doesn't move accuracy. Three defensible things remain: the customer's own data, calibration against real outcomes, and honest outward measurement. That last one now cuts both ways — "we calibrate on your results" is no longer a sentence, it's a measurement debt. Next: a calibrated rerun once calibration points accumulate, the TR two-snapshot design, and a run with a real customer panel. Pairs, panels, judgments and reports are committed in the repo.