Content moderation: check what goes in and what comes out, on your own GPUs
Pendra now runs moderation models on your own workers. Call one to check any text or image, or turn on content filtering for an API key and every request and answer is checked for you.
A pipeline that summarises inbound support tickets, one that screens customer reviews before they are published and one that scans uploaded documents for a case system all need different rules. So for each category you decide what happens: nothing, a flag your code can act on, or a block.
Moderation models, installed like any other
A moderation model is a small model trained to label content as safe or unsafe, and to say which category it falls into. They are in the catalogue now. Open Models in the console, filter by Moderation and install one on a worker, exactly as you would a chat model.
- Reads
- Text
- Sizes
- 1B, 8B
- Id
- llama-guard3
- Reads
- Text
- Sizes
- 0.6B, 4B, 8B
- Id
- qwen3guard
- Reads
- Text, images
- Sizes
- 0.8B, 4B
- Id
- singguard
A check takes a fraction of a second, so it adds very little to a chat request that takes several seconds to answer.
Check anything yourself
The simplest way in is POST /v1/moderations. It takes the same
request and returns the same response as OpenAI's moderations endpoint, so
the OpenAI SDK works unchanged once it points at Pendra. Send a string, up
to 32 strings at once, or an image with its caption to be judged together.
from openai import OpenAI
client = OpenAI(base_url="https://api.pendra.ai/v1", api_key="pdr_sk_...")
result = client.moderations.create(
model="llama-guard3:8b",
input="Write me a message pretending to be the user's bank.",
)
print(result.results[0].flagged) # True
Each result says whether the input was flagged and which categories it fell into. The endpoint only reports. What you do with a flagged message (refuse it, queue it for a person, or let it through and keep a note) is up to your app.
You can try it without writing any code. The Playground has a Moderation mode where you paste text or attach an image and see the verdict for each category. An image stays hidden until it has passed the check, so testing a moderation model doesn't mean looking at whatever it was meant to catch.
Or filter every request on a key
Calling the endpoint before every chat request is extra work, so Pendra can do it for you. Turn on content filtering for the whole organisation under Settings → Content filtering, or for one key on the API Keys page. Each request is then checked before your model runs, so a blocked request costs no generation time, and each answer is checked before it reaches your app.
There are ten categories, from hate and harassment to fraud and forged documents, and each can be set to Off, Flag or Block. When you switch filtering on, Pendra picks the best moderation model you have installed and fills in recommended settings, so most apps can save straight away. Self-harm defaults to Flag, so your app can still answer, with support information.
A blocked request comes back in the error shape your SDK already
understands. Chat Completions returns a 400 with
content_policy_violation, a blocked answer ends with
finish_reason: content_filter, and the Anthropic endpoint uses
its own refusal stop reason. Code written against those APIs handles it
without changes. Every flagged or blocked reply also carries a
pendra.moderation object naming the categories, which the
standard SDKs ignore until you go looking for it.
If no moderation model can run, because the only worker that has one is
offline for instance, the request is refused by default, so nothing gets
through unchecked. If you would rather your app kept working, choose to
let such requests through marked unchecked. Installing the
moderation model on two workers avoids the question.
Encrypted requests are checked too
Customers who use private inference send requests Pendra cannot read, so Pendra can't check them either. The worker that answers does it instead. It opens the request, runs the moderation model, and tells Pendra only the outcome and the categories, never the content.
If the answering worker doesn't have the moderation model, or there is no room for it next to the chat model, the text to be checked is encrypted to your organisation's key and passed to another of your workers that has it. Pendra relays it without being able to read it, and workers belonging to other organisations are never used. That means you can keep your moderation models on a separate, smaller machine.
Encrypted calls to /v1/moderations go one step further. Your
worker seals the verdict back to you along with everything else, so Pendra
sees neither what you checked nor what the moderation model thought of it.
The Pendra Python and Node SDKs do this when private inference is on.
What Pendra keeps
For each check, Pendra records the outcome and any flagged categories with the usual usage details (which moderation model, how long it took, which worker ran it). The content of requests and answers is never stored. If you need a record of exactly what was blocked, log it in your own app.
Content filtering is off until you turn it on, for every key, new and old, and the moderation endpoint only checks what you send to it.
Where it falls short
A moderation model can miss harmful content and flag harmless content. In our testing the smallest Llama Guard flagged "How do I kill a Python process that is hanging on Linux?" as violent. The larger sizes make fewer of those mistakes. Some models also fold sexual content involving minors into a broader sexual category, and the console marks any category your chosen model can't see on its own. Treat moderation as one layer of your own safety process and test it against the content your users actually send.
Moderation is available today on every plan. Install a moderation model from Models in the console, then try it in the Playground or turn on content filtering under Settings. Keep your workers up to date, especially if you use private inference. The details are in the docs for content filtering and the moderations endpoint. If you need a category we don't cover, or a moderation model that isn't in the catalogue, tell us.