AI Peer Review Evaluation Researcher

Founding team · Full-time · Hybrid (London) · Meaningful equity

Science runs on trust, but the machinery meant to produce that trust is failing. Peer review is slow, opaque, inconsistent, and increasingly gamed. And a flood of AI-generated research is about to overwhelm a system that was already straining. No one is the credible, independent arbiter of research quality. We think there should be one.

OxSci is building a credit rating agency for science: a certification layer that combines AI with expert peer review, so researchers, institutions, and AI developers can assess research quality quickly and at scale. We're early, focused, and well-resourced.

The open scientific question at the heart of this company is yours to own: where does the frontier of AI peer review actually stand? What can AI reviewers catch that human experts miss, what do they still get wrong, and how do you prove it rigorously? The standards we set now, including what "quality" even means and how we demonstrate our AI reviewers are actually good, will define both the company and, we believe, the field.

If you work on LLM evaluation or AI for science, you know the pattern: the most important studies (expert-annotated meta-evaluation of AI reviewers on real papers) are one-off efforts, because recruiting dozens of expert scientists as annotators takes months and can't be repeated. At OxSci, that study design is our daily operating loop.

• Expert annotation as infrastructure, not a bottleneck. We operate a standing, paid network of expert reviewers producing structured reviews of real submissions, alongside AI reviews of the same manuscripts. The human-AI comparison most labs can only run once, you run continuously, at growing scale.
• Define the benchmark, not just run one. There is no accepted standard for meta-evaluating AI reviewers. The evaluation you design here is positioned to become the reference point for the field.
• Publish and speak. We want your findings in the open literature and on conference stages. You'll represent OxSci publicly in the AI-review and LLM-evaluation research community; our scientific credibility is built on your work being visible and defensible.
• Your research artifact runs in production. Your benchmarks and evaluator improvements ship into a live review system serving real researchers, then feed back new data. At a big lab you'd be one name on a twenty-author paper; here you own the agenda, the data, and the standard.

OxSci was founded by Shumiao Ouyang, Associate Professor of Finance at Saïd Business School, University of Oxford, and Fellow in Management at Wadham College. His research spans fintech, digital payments, and AI; he holds a PhD in Economics from Princeton and degrees from Peking and Tsinghua Universities. He started OxSci out of a conviction that the systems for judging scientific quality are overdue for reinvention, and you'd be building it with him directly.

The technical side is led by OxSci's cofounder, a senior tech lead from one of the world's largest technology companies, with deep experience building and operating systems at global scale. You'd work with both founders day to day, shaping the evaluation methodology and research culture from the ground up.

1. Own the research agenda on AI-reviewer evaluation. Track the frontier (AI-scientist, automated-review, LLM-as-a-judge, and scholarly-NLP literature), position our system against it, and decide what we measure next and why.
2. Design meta-evaluations that expose weaknesses, not just measure agreement. Build fine-grained, criticism-level evaluations of AI review agents (correctness, factual grounding, significance, sufficiency of evidence, hallucination rate, and venue/journal matching) that reveal where and why they fail, going beyond verdict-matching.
3. Run expert-annotation studies at scale. Design the protocols, rubrics, inter-annotator agreement, and statistics needed to compare AI and human reviewers credibly, including head-to-head evaluations against other AI review systems, and defend the numbers to a skeptical scientific audience.
4. Build a living taxonomy of AI-reviewer failure modes such as subfield blind spots, long-context degradation, over-anchoring, and spurious criticism, and turn each into a regression benchmark that guards against backsliding as models and prompts change.
5. Calibrate the combined rating. Define quality-scoring rubrics for human review reports and calibrate how expert and AI judgment fuse into a single, defensible rating: the core of what universities and publishers buy from us.
6. Close the loop. Translate benchmark findings into concrete improvements to our review agents (retrieval, context engineering, orchestration, model choice) and prove the gains with the same rigor you used to find the gaps.

1. A PhD (or near completion) in CS, ML, NLP, or a related field, or an equivalent research track record. You're likely already working on LLM evaluation, LLM-as-a-judge, AI for science, automated peer review, or a nearby frontier.
2. A fascination with the boundary between AI and human reviewers. What each catches that the other misses, and conviction that mapping it rigorously is how trustworthy peer review gets built.
3. A track record of rigorous evaluation of ML/LLM systems: benchmark or eval-framework design, evaluator/judge models, expert-annotation study design, hallucination and factuality measurement, uncertainty quantification, or RAG evaluation. Bonus if you've built evaluations that score individual criticisms rather than just verdicts.
4. Fluency in evaluation methodology and statistics: sampling, inter-annotator agreement, significance testing, and the discipline to distinguish a real effect from a lucky prompt.
5. Strong Python and hands-on habits. You build the eval harnesses and pipelines yourself, not just spec them, with enough LLM-application fluency (RAG, tool calling, orchestration) to turn a finding into a shipped improvement.
6. Bonus: publications in NLP/ML evaluation or automated peer review; open-source benchmarks or evaluator models the community actually uses; experience with scholarly content at scale.

• Founding seat with meaningful equity and a direct line to the founders
• Ownership of a genuinely open research question, with encouragement to publish and present the work

Interested in opportunities specifically in United Kingdom? Discover our dedicated Visa Sponsorship Jobs in United Kingdom page featuring roles from top employers in this location.

• A standing expert-reviewer network as your annotation infrastructure
• A proprietary, growing dataset of paired human and AI reviews of real submissions
• The rare chance to define the standard by which AI reviewers themselves are judged
• Genuinely competitive pay; equity discussed openly
• Generous LLM token budget for your daily work
• Flexible working hours; fast personal growth with broad ownership from day one

• Location: Hybrid (London)
• Visa: we can sponsor work visas for the right candidate.
• To apply: send your CV to recruit.uk@oxsci.ai, and include something you've built or written: a benchmark, an eval, a paper, or any project you'd love to talk about.