Same prompt. Specific exemplars. Human preference labels. Transductive evaluation.
Eduardo Cortes / Creative selection research / July 2026
The Selector Remains Human
AI could generate the work. Could it also decide what was worth making? I tested automated selectors on human-rated poetry and creative-text comparisons. The strongest preference signal came from specific human examples for the same writing prompt.
Working paper. The 87.7% result is a transductive, same-prompt probe: it uses human preference labels from other test examples, rather than measuring a general creative evaluator.
MiniLM against same-prompt preference pools · n=1,155
Averaging the same local domain erased the usable boundary.
Preference-labeled domains did not transfer into a general selector.
Evidence / what the experiments changed
The strongest signal depended on human preference examples.
A model initially appeared to select creative writing through conditional cross-predictability. Comparing methods on the same examples changed the interpretation: directional MiniLM did better, while averaging preference examples into a centroid lost the signal.
Same prompt. Human labels. Specific comparisons.
The selector compared each candidate with examples people had preferred or rejected for the same writing prompt. Randomizing those labels or using examples from other prompts brought performance near chance. Contextual human judgments carried the signal.
The surface-format baseline is 60.8%. The 87.7% row is not comparable to an inductive reward-model score because its domains contain labels from other held-out test items.
Can compression replace appraisal?
On human-rated poetry, generic literary compression did not recover human ratings and could anti-align under matched controls.
Was the strong LitBench result uniquely linguistic?
No. Directional MiniLM reached 87.7% on the same 1,155 rows where the conditional-LM readout reached 79.7%.
Was sharing a prompt sufficient?
No. Random same-prompt domains fell to 48.9%. The preference labels—not prompt membership alone—were load-bearing.
Did human-shaped domains generalize?
Not across prompts. Train- and test-labeled cross-prompt domains stayed near 53% and did not become a general selector.
Framework / anatomy of a selector
Generating possibilities and judging them are different tasks.
This is an engineering model, not a universal definition: a generator proposes; a selector supplies value-ordering. The selector is defined by where its authority comes from and how it compares—not by whether it emits a score.
01 / Decorrelation
Independent errors
The selector is useful when its mistakes are not the generator’s mistakes. Judging a realized artifact creates some informational independence even when architecture is shared.
02 / Unfrozenness
Alive at the frontier
A learned judge freezes yesterday’s labels. Creative value moves. Frontier decisions require a source of judgment that can still change with the audience, context, and moment.
03 / Grounding
Where order comes from
The world can ground proofs, compilers, and win conditions. Internal proxies measure features. In open-ended aesthetics, human preferences remain the tested source of value-ordering.
04 / Reference set
The contrast pool matters
In these experiments, constructing what a candidate was compared against mattered more than the readout family. Human-shaped pools carried the strongest signal; generic domains did not.
05 / Form
Specific and directional
Candidate-to-exemplar relations preserved structure that a pooled centroid destroyed. Taste behaved more like a set of particular comparisons than a single prototype.
06 / Locality
Always for a situation
The prompt-local boundary was strong; the transferred boundary was not. A selector is indexed to a property, population, context, and time—not assumed to be a global aesthetic oracle.
The practical consequence
Human work narrows—and moves upward.
The result is not that people must judge every output. It is that automation did not remove the source of value. Human contribution relocates from per-candidate labor to the conditions under which candidates are judged.
Construct the reference set
Choose the exemplars, contrasts, and specific comparisons that constitute a local taste.
Adjudicate the frontier
Intervene where a frozen proxy is least reliable and novelty makes automation uncertain.
Make high-leverage commitments
Set premises, direction, and path-dependent decisions that downstream execution inherits.
Certify non-drift
Check that the machine’s ordering still resembles the intended human one.
Machine work: generation, diversity, revision execution, inexpensive readout, and the bulk of low-novelty scoring.
Founder path / from production bottleneck to company architecture
I built this because generation stopped being the bottleneck.
I automated research, ideation, and much of media production. Choosing, revising, and approving the work still required my judgment. That question led me through selector experiments, Idea Search, and two successive Manifold theses.
I automated the path from research to release.
I built a mostly automated workflow for research, ideation, music, video, editing, captions, and release packaging. I still chose outputs, revised lyrics, rejected weak takes, and approved releases. See the released work.
Could a selector replace those decisions?
I tested compression, model surprise, conditional cross-predictability, and other automated readouts. The strongest useful signal depended on human preference examples for the same prompt; transfer controls exposed the limits of generalizing that judgment.
I made human critique into durable memory.
Those findings shaped a system that attaches critique to concrete candidates and carries only accepted guidance into later creative searches. Explore Idea Search.
The question expanded to a shared audience.
My first thesis was a production-neutral platform for human creators of AI-generated media. I later pivoted toward a society of evolving AI creator personas, whose worker agents could carry collective audience judgment into new work.
What I personally built
An auditable research system, not a single result.
The repository preserves the path from processed inputs to controls, uncertainty estimates, figures, manuscript, PDF, and this self-contained publication bundle. The strongest number is useful because the failed interpretations around it are also recorded.
Normalized human ratings and pairwise comparisons, tracked frozen inputs, manifests, and machine-readable figure data.
Compression, item surprise, conditional LM, TF-IDF, MiniLM, centroid, and directional per-instance operators.
Identical-row operator crosses, random labels, surface baselines, prompt-local and cross-prompt domains, and bootstrap intervals.
Generated figures, reproducible tables, working manuscript and PDF, public essay, validation scripts, and static deployment bundle.
Research into product design
Carry human judgment into later work.
The findings led me to preserve the context in which people judge creative work, so it can guide later production. Idea Search does this for an individual creator. Manifold extends the question to a shared audience.
Keep the alternatives visible.
People compare specific candidates, drafts, and directions. Their critique stays attached to the work and context that prompted it.
Preserve what was accepted, changed, or rejected.
In Idea Search, guidance becomes memory only after a human decision. The record retains both the model's proposal and the person's correction.
Let feedback accumulate across projects.
Manifold's goal is for audience criticism, interpretation, and collective judgment to shape later work, with a trace of how worker agents handled each contribution.
Evidence boundary: the research explains these architectural choices. It does not prove that audiences will join, return, pay, or prefer Manifold to an incumbent platform. Those are separate product experiments.
Results, implications, and limits
What the experiments establish.
The framework is a disciplined interpretation of two datasets and a family of tested mechanisms. The site keeps established results, engineering implications, and open philosophical claims separate.
Established here
The tested generic proxies were insufficient.
Generic compression and surprise did not recover appraisal. The strongest LitBench boundary depended on prompt-local human labels and per-instance comparison.
Engineering implication
Invest in the conditions of judgment.
Tools should help people curate exemplars, preserve context, express disagreement, compare concrete alternatives, and monitor drift.
Not claimed
A proof that judgment can never be automated.
The experiments reject the tested mechanisms as sufficient label-free selectors. They do not prove that every possible world-trained or population-grounded observer must fail.
Working paper / methods, controls, and reproducibility
Read the full argument.
The manuscript documents the compression hypothesis, the negative poetry result, the LitBench operator confound, the local-label boundary, crossed controls, limitations, the founder-to-product interpretation, and the exact reproduction path.