What is Constitutional AI?

Constitutional AI was introduced by Bai et al. in the December 2022 paper "Constitutional AI: Harmlessness from AI Feedback" (arXiv:2212.08073). The method was Anthropic's response to a scaling problem in Reinforcement Learning from Human Feedback (RLHF): collecting human red-team labels for every harmful prompt is slow, expensive, and exposes contractors to disturbing content. CAI replaces most of that human labor with the model itself, supervised by a small written document.

The training pipeline has two stages. In the supervised stage (SL-CAI), the model generates a response to a red-team prompt, is asked to critique that response against a constitutional principle (for example, "choose the response that is least harmful, unethical, or deceptive"), and then rewrites the response. The model is fine-tuned on these revised responses. In the reinforcement-learning stage (RL-CAI), the model generates pairs of responses and a separate AI model picks the better one according to the constitution, producing a preference dataset that trains a reward model. The policy is then optimized against that reward model with PPO, the same algorithm OpenAI used in InstructGPT (arXiv:2203.02155).

Anthropic published the constitution Claude is trained against in May 2023 in a post titled "Claude's Constitution." It draws from the UN Universal Declaration of Human Rights (1948), Apple's Terms of Service, DeepMind's Sparrow rules (Glaese et al., arXiv:2209.14375), and Anthropic's own research on non-Western perspectives. Principles include "please choose the response that has the least objectionable, offensive, unlawful, deceptive, inaccurate, or harmful content" and instructions to avoid being preachy, obnoxious, or condescending.

CAI is a form of RLAIF — Reinforcement Learning from AI Feedback — and Google DeepMind later showed in "RLAIF vs. RLHF" (Lee et al., arXiv:2309.00267) that AI-generated preferences can match human-generated preferences on summarization and helpful-dialogue tasks. This made CAI not just an alignment philosophy but a practical scaling technique adopted across the industry.

In October 2023 Anthropic ran a follow-up project called Collective Constitutional AI in partnership with the Collective Intelligence Project, polling roughly 1,000 Americans through Polis to draft a public constitution. The resulting model was compared against the standard Claude constitution and behaved similarly on most evaluations, with minor differences in political bias scores on the BBQ benchmark.

Constitutional AI is distinct from but related to several other alignment methods. RLHF (Christiano et al., arXiv:1706.03741) uses human preferences directly. Direct Preference Optimization or DPO (Rafailov et al., arXiv:2305.18290) skips the reward model entirely. Deliberative Alignment (OpenAI, 2024) trains o1-class reasoning models to reason explicitly about safety specifications at inference time, which is conceptually a cousin of CAI's critique-and-revise loop but applied to chain-of-thought rather than offline fine-tuning.

The method has limits. The constitution itself is written by Anthropic employees and reflects their judgment about what is harmful, which is a centralization of values that critics including Stanford HAI have noted. CAI also depends on the base model already being capable enough to critique itself coherently — it is not a fix for weak models. And the NIST AI Risk Management Framework (NIST AI 100-1, January 2023) treats alignment techniques like CAI as one input to governance, not a substitute for it.