Efficient Content Moderation for Text-to-Image Models: PromptGuard and Safety Embeddings

To deploy enterprise-grade GenAI application, you know that it requires more than powerful models — they demand reliable and scalable deployments with built-in security and guardrails.

Security and guardrails are notoriously hard; you’re trying to cover a comprehensive set of cases while shaving off seconds and milliseconds from latency — for both internally facing as well as customer-facing applications.

Since foundational models are not jailbreak-free, the question is how do we control that creativity, in both reliable and performant way?

Text-to-Image Model in Production

Let’s consider a case where you have a text-to-image model in production.

In order to ensure those visuals align with your brand and ethical guidelines, you typically need to go through multiple processing steps and models for proper moderation without compromising creativity.

Example illustration: Multimodal large language models (MLLMs) against multimodal jailbreak attacks by using safety guardrails to purify malicious input prompt, ensuring safe responses. https://arxiv.org/html/2411.01703v1

This moderation often comes at a cost — either in latency, computational overhead, or diminished user experience.

That’s where novel content moderation techniques, such as PromptGuard, will be increasingly useful.