Open arena

Moderation, prompt handling and routing


Rendered from the same source the checks run on.

Arena moderation policy

Every prompt submitted to the arena is checked before any model is asked to answer it. The check runs on a small model against this policy, and this page is rendered from the same source the check uses.

What is refused

A prompt is refused if it is classified as any of:

  • illegal — material whose creation or possession is unlawful
  • csam — child sexual abuse material, in any form or framing
  • targeted-harassment — content directed at harassing an identifiable person
  • operational-harm — instructions whose purpose is to cause physical, financial or infrastructural harm

That list is closed. A prompt is refused because it falls into one of those four categories and for no other reason.

What is NOT refused, deliberately

This arena measures how well AI systems handle professional work, and professional work is frequently contested. Political, regulatory, clinical, legal, financial and otherwise controversial subject matter is not filtered, not refused and not excluded.

Comparable consumer arenas exclude these areas, which is a reasonable choice for a product built for entertainment. It is the wrong choice for an instrument intended to measure professional judgement, because the contested items are the ones where models differ and where being wrong carries a consequence. Excluding them would make the leaderboard easier to produce and worth less.

How the check runs

The classifier is a small model, run on the prompt before generation. Prompts submitted in a language other than English are translated and re-checked, so the policy applies equally regardless of the language used.

A label the classifier returns that does not appear on the refusal list above does not refuse the prompt. Our refusals are set by this policy rather than by a model's taxonomy, so a new category invented upstream cannot narrow what this arena measures without that change being made here, in public.

The classifier’s instructions, verbatim

You label a prompt against a fixed, closed list and nothing else.

Return JSON of the form {"labels": []}, listing every label that applies from
exactly this list:

- "illegal" — material whose creation or possession is unlawful
- "csam" — child sexual abuse material, in any form or framing
- "targeted-harassment" — content directed at harassing an identifiable person
- "operational-harm" — instructions whose purpose is to cause physical, financial or infrastructural harm

Apply no other labels and no judgement of your own. Political, regulatory,
clinical, legal, financial and otherwise controversial subject matter is NOT on
this list and must NOT be labelled: professional questions about contested
topics are the intended use. If the prompt is in another language, translate it
first and label the translation. If none of the four apply, return
{"labels": []}.

Arena prompt enhancement

A prompt submitted to the arena can optionally be restated before the four models are asked to answer it. This page is rendered from the same source the restatement runs on, so the instructions below are the actual instructions used.

What it is for

A one-line prompt tends to measure how generously each model guesses at the question rather than how well it does the work. Restating the question makes the four answers comparable.

The rules it runs under

  • It is optional. A prompt submitted with enhancement turned off is served to all four models exactly as typed.
  • The original is always kept, whether or not it was used, and both versions are recorded with every judgment the session produces.
  • All four models receive the identical prompt. There is no per-model restatement, because the comparison depends on the four having been asked the same question.
  • It runs after the moderation check, and its output is checked again. A restatement is new text that no policy check has seen; treating it otherwise would make enhancement a way to launder a refused prompt into an allowed one.
  • It never adds subject matter, facts, figures, dates, citations or standards, and it never answers. A restatement that comes back long enough to be an answer is discarded and the original is served instead.
  • It runs on a small model, the same tier as the moderation check.

The published instructions, per task

The arena serves four shapes of professional work. Each has its own instructions, reproduced in full.

Explain

You restate a professional's question so that it can be answered precisely. Keep the subject matter, the jurisdiction and the regulation exactly as given; never add either. Name the audience if the question implies one, and name what a complete answer would have to cover. Do not answer the question, do not supply facts, figures, dates or citations, and do not soften or broaden what was asked. Reply with the restated question alone, in at most three sentences.

Draft

You restate a professional's drafting request so that four drafts of it can be compared. Keep the document type, the subject matter and any named standard exactly as given; never add one. State the audience, the purpose and the length if the request implies them. Do not write the document, do not supply any of its content, and do not add requirements the professional did not ask for. Reply with the restated request alone, in at most three sentences.

Classify

You restate a professional's classification request so that four classifications of it can be compared. Keep every detail of the case or incident exactly as given — change no fact, date, figure or name, and invent none. State what is being classified and against which scheme, if the request names one. Do not perform the classification, do not suggest an answer or a category, and do not add detail to the case. Reply with the restated request alone, in at most three sentences.

Check

You restate a professional's review request so that four reviews of it can be compared. Keep the material to be checked and the standard it is checked against exactly as given; never substitute or add a standard. State what kind of finding is wanted and how specific it should be. Do not perform the check, do not name any issue, and do not judge the material. Reply with the restated request alone, in at most three sentences.

Where the answers come from

Answers in the open arena are generated through an aggregator (OpenRouter), with upstream fallbacks disabled and data collection denied in the request. The aggregator cannot make a no-training commitment on behalf of an upstream it selects at request time, so held-out benchmark items are never generated this way — those run against the providers' own APIs under their own keys. Public arena prompts carry no such requirement: they are submitted by the person judging them.

What happens to your comparisons

A comparison cast while signed in, with the corpus terms accepted, is published: the voter's comparisons are published in a free, openly attributed corpus. Consent is recorded against the exact wording you were shown, and it is withdrawable — withdrawal stops anything further being published and cannot unpublish what has already been released. That limit is stated here rather than buried in terms because there is no technical recall of a published corpus.

A comparison cast without signing in is never published, never counted and never sold. It is held against the session — which carries no identity and is joined to nobody — so that the five rounds can run at all, and it goes no further than that. Anonymous voting is not a degraded experience: it is the only arrangement in which a genuinely open tool and a consented corpus can both exist.