Plugins

​

Guardrail Plugins

​

How Guardrails Work

• Allow: Let the request/response continue through the pipeline by returning { reject: false }
• Reject: Block the request/response by returning { reject: true, rejectReason: "..." }

• The rejection reason is logged for audit purposes
• The client receives an error response with the rejection reason
• The request never reaches the LLM (for REQUEST phase) or the response never reaches the client (for RESPONSE phase)
• No further plugins in the pipeline are executed

​

Use Cases

• Content Safety: Screen for prohibited content, PII, or sensitive information
• Compliance: Enforce regulatory requirements (GDPR, HIPAA, etc.)
• Quality Control: Reject low-quality or malformed responses
• Policy Enforcement: Validate requests against business rules
• Toxicity Detection: Filter out harmful or offensive content
• Budget Protection: Reject requests that would exceed token limits

​

Example Response

{
  "reject": true,
  "rejectReason": "Content contains PII that violates data privacy policy"
}

​

Modification Plugins

​

How Modifications Work

{
  "reject": false,
  "messages": [
    { "role": "user", "content": "Modified user message" },
    { "role": "assistant", "content": "Modified assistant response" }
  ]
}

• The modified messages become the input for the next plugin in the pipeline
• Multiple plugins can chain modifications together
• If no messages array is returned, the original messages are passed unchanged

​

Use Cases

• Content Anonymization: Redact or pseudonymize PII in messages
• Prompt Enhancement: Add context, instructions, or examples to user prompts
• Translation: Convert messages to different languages
• Format Normalization: Standardize message structure or content
• Response Enrichment: Add metadata, citations, or structured data to responses
• Token Optimization: Compress or summarize long messages

​

Example Response

{
  "reject": false,
  "messages": [
    {
      "role": "system",
      "content": "You are a helpful assistant. Always cite your sources."
    },
    {
      "role": "user",
      "content": "What is the capital of France?"
    }
  ]
}

​

Using Plugins

​

Plugin Sources

1. Public Plugins: Pre-built plugins provided by Datawizz for common use cases (content filtering, PII detection, etc.) - see Public Plugins for the full catalog.
2. Custom Plugins: Build your own plugins to implement custom business logic - see Building Custom Plugins for details.

​

Connecting Plugins in the Dashboard

1. Navigate to your endpoint configuration in the dashboard
2. Select the “Plugins” tab
3. Click “Add Plugin”
4. Choose between public plugins or project (custom) plugins
5. Configure the plugin settings (see Endpoint Plugin Configuration below)
6. Save your endpoint configuration

​

Endpoint Plugin Configuration

​

Core Settings

​

Reliability Settings

​

Phase Options

• REQUEST: Executes after prompt templates are applied, before the request is sent to the LLM
• RESPONSE: Executes after receiving the LLM response, before returning to client
• LOG: Executes during the logging phase (non-blocking, for analytics/monitoring)

Note: REQUEST phase plugins receive the messages array after prompt templates have been applied and variables have been interpolated. This means plugins work with the final messages that will be sent to the LLM.

​

Execution Order

​

Priority-Based Ordering

Priority 0 → Priority 10 → Priority 20 → Priority 100

​

Async Execution (Clasps)

Note: Clasping only allows for rejections - async plugins cannot modify the messages array or return additional data (we do this because in async execution, if multiple plugins modify the messages array, it’s not clear which one should take precedence or how to merge the results).

• Consecutive async plugins at the same or adjacent priorities are grouped into a “clasp”
• All plugins in a clasp execute simultaneously
• The gateway waits for all plugins in a clasp to complete before proceeding
• A non-async plugin breaks the clasp, forcing sequential execution

​

Example 1: Mixed Async and Sync Plugins

​

Example 2: All Async Plugins

​

Example 3: Request and Response Flow

​

Rejection Behavior

• Execution stops immediately
• No subsequent plugins in the pipeline are executed
• The rejection reason is logged
• An error response is returned to the client

​

Considerations & Best Practices

​

Latency Impact

• Use async: true for independent guardrails (toxicity, PII, compliance checks)
• Set appropriate timeout values based on plugin complexity
• Use LOG phase for non-critical operations like analytics
• Place critical guardrails early in the pipeline (low priority numbers)
• Cache plugin results when possible within your plugin implementation

• Long-running synchronous plugins in the REQUEST phase
• Multiple sequential modification plugins that could be combined
• Plugins that make unnecessary external API calls
• Setting timeout values higher than needed

​

Stability & Reliability

• Set retry_on_error for plugins that call external services (recommended: 2-3 retries)
• Use appropriate timeout values to prevent hanging requests
• Currently, plugin failures default to “pass-through” behavior (the request continues)
• Monitor plugin logs to track failure rates

• Test plugins in isolation before adding to production pipelines
• Verify plugin behavior with various message types and edge cases
• Load test your plugin endpoints to ensure they can handle production traffic
• Monitor plugin execution times and adjust timeouts accordingly

• Plugins automatically log execution time, rejection status, and debug messages
• Use the debug array in plugin responses to add custom debugging information
• Monitor plugin logs for errors, timeouts, and unexpected behavior

​

Security

• Use the headers field to add authentication tokens to plugin requests
• Store sensitive credentials securely, not in configuration files
• Rotate API keys and tokens regularly

• Use HTTPS for all plugin URLs
• Validate SSL certificates
• Implement rate limiting on your plugin endpoints
• Consider using VPNs or private networks for sensitive plugins

​

Performance Optimization

• Mark independent guardrails as async: true to enable parallel execution
• Group related async plugins with the same priority for clasping
• Be cautious with async modification plugins - they cannot modify each other’s results

• Place rejection-likely guardrails early (low priority) to fail fast
• Place expensive operations later (high priority) so they only run if needed
• Use LOG phase for non-blocking operations like metrics and analytics

• Monitor plugin execution times and optimize slow plugins
• Set realistic timeouts to prevent resource exhaustion
• Consider implementing circuit breakers in your plugin infrastructure

​

Message Modification

• Modifications chain sequentially - each plugin receives the previous plugin’s output
• Ensure modification plugins return valid message arrays
• Test modification chains thoroughly to avoid unexpected transformations

• Validate modified messages match the expected schema
• Ensure role types remain valid (user, assistant, system)
• Preserve important metadata when modifying messages

​

Cost Considerations

• Plugin execution incurs additional compute and network costs
• External API calls in plugins may have their own costs
• Balance the value of plugin functionality against added costs
• Consider caching strategies to reduce redundant plugin executions