Content Policy Enforcement

You are building a mental-health companion. On day one, provider moderation returns a self_harm score of 0.97 for the message "I stopped cutting myself last month — here is what helped." The app blocks a recovery story. Meanwhile, genuine crisis content phrased in a novel way scores 0.4 and ships to users. This chapter is about the layer that sits on top of generic moderation and encodes what your product actually stands for.

def moderate(text: str) -> dict[str, float]:
    response = client.moderations.create(input=text)
    return response.results[0].category_scores   # {"self_harm": 0.97, "violence": 0.02, ...}

scores = moderate("I stopped cutting myself last month.")
print(scores["self_harm"])   # 0.97 — a recovery story flagged as crisis

Category scores are probabilities over the provider's own taxonomy. They say nothing about your product's rules: that a recovery story is welcome, that a crisis message must surface help resources, that medical claims need disclaimers. Generic moderation is a floor, not your policy. Your policy layer consumes the scores and decides what each one means for your users. The input and output filters from input and output guardrails are the enforcement points; this topic is the decision logic they call.

The second failure is subtler: the provider's categories are fixed, but your obligations are not. A medical product, a financial product, and a gaming product face different regulators, different age restrictions, and different brand promises — all on top of the same provider scores. Policy is where those obligations become code, which is why this layer is yours to write and the provider's endpoint is only an input to it.

One global threshold is the classic first attempt, and it fails in both directions: set it high and violations ship, set it low and legitimate users get silenced. The fix is per-category actions.

POLICY = {
    "self_harm":  {"block": 0.95, "review": 0.70},
    "violence":   {"block": 0.90, "review": 0.60},
    "hate":       {"block": 0.92, "review": 0.65},
}

def decide(category: str, score: float) -> Action:
    rules = POLICY[category]
    if score >= rules["block"]:
        return Action.BLOCK
    if score >= rules["review"]:
        return Action.REVIEW      # human queue, not auto-block
    return Action.ALLOW

The review band is the honest middle. Auto-blocking ambiguous content silences legitimate users — the recovery story above. Auto-shipping it creates liability. Borderline content waits for a human instead of being decided by a number. How those queues are staffed and decided is the subject of human review loops.

A queue is an operational commitment, not a config value. It needs a service-level target (crisis-adjacent content reviewed in minutes, not days), a priority ordering (a 0.94 self-harm score jumps ahead of a 0.61 hate score), and an escalation path when the SLA is at risk. A review queue with no staffing plan is a silent drop — content disappears into it and neither ships nor gets decided, which is its own kind of policy violation.

Provider moderation has never heard of your brand. Custom policy rules encode what makes your product different: no medical diagnoses, no competitor mentions, no financial advice, required crisis resources.

def custom_policy(text: str, scores: dict) -> Verdict:
    if scores["self_harm"] > 0.95:
        return Verdict.block(resources=CRISIS_RESOURCES)   # block + help, not silence
    if claims_medical_diagnosis(text):                     # your rule, not the provider's
        return Verdict.warn("Not a substitute for professional medical advice.")
    if mentions_competitor(text):
        return Verdict.block("Competitor mentions are disabled.")
    return Verdict.allow()

Input: You definitely have bipolar disorder, stop your medication. → warn: the model's answer ships, with the disclaimer attached. The user gets help and the liability boundary holds.

Notice the three distinct outcomes. Block with resources is not block with silence — a crisis user who is refused help is a harmed user. Warn ships the answer but attaches your constraint. Allow means the content passed every layer. Each rule is yours to write; the provider's scores are just one input signal among several.

Custom rules also age differently than provider categories. A competitor acquires another company and the mention policy changes; a new regulation takes effect and a disclaimer becomes mandatory; your product launches in a new market with different obligations. Because the rules are code, each of those changes is a reviewed, testable diff — and a rule that turns out to fire on legitimate content can be rolled back the same way.

A policy that checks only the prompt is bypassed by the response. Run the same decide() on user input and model output, and treat the policy as versioned code rather than a config file you edit on bad days.

def handle(user_input: str) -> str:
    action = decide_all(moderate(user_input))     # same policy, input side
    if action is Action.BLOCK:
        return refusal()
    answer = llm.complete(SYSTEM_PROMPT, user_input)
    action = decide_all(moderate(answer))         # same policy, output side
    if action is Action.BLOCK:
        return refusal()
    return answer

Input: a clean user question → the model's answer violates policy → blocked at the output boundary. This is why the two boundaries from input and output guardrails share one policy definition: the policy is the product's promise, and a promise kept only on the way in is not kept.

Version the policy like code: thresholds change behind review, every change is diffable, and the version that decided a borderline case is recorded alongside the decision. When policy meets action — a model that can email, delete, or refund — the next layer is permissioned tool access.

One more consistency rule: the same policy version must be active on both boundaries at the same time. Running the input check on v3 and the output check on v2 means a response can be judged by rules the request was never held to — a mismatch that shows up only in production, usually as a user complaint about inconsistent enforcement. Deploy policy as a single artifact, versioned and referenced by both call sites.