# Client Access
Source: https://docs.datawizz.ai/access-control/client-access
Enable user-level authentication and authorization with JWT tokens
Client Access allows you to validate user access using JWT - creating user level keys with user level rate limits and tracking. This is useful in several scenarios:
* **Call LLMs from the Client**: You can call LLMs directly from a web or mobile app, using the user's JWT to authenticate and authorize access.
* **Supabase / Firebase Integration**: You can use the user's JWT from Supabase or Firebase to authenticate and authorize access to LLMs.
* **User-Level Rate Limits**: You can define rate limits per user, allowing you to control API usage on a per-user basis.
* **Organization LLM Access**: You can provide LLM access to users in an organization with personal, tracked & rate-limited keys.
## Getting Started
To enable Client Access, create a new project key (in your project settings) and enable the "Client Access" option. You will be required to provide a JWT secret - only JWTs signed with this secret will be accepted.
![Configure Client Access]()
Now requests to the project key must include a JWT token in the `Authorization` header. The format is `Bearer :` where `` is the project key (starts with `sk-...`) and `` is the signed JWT token.
* The JWT will be validated using the configured secret.
* If the JWT has issued at (`iat`), not before (`nbf`) or expiration (`exp`) claims, they will be checked to ensure the token is valid.
* The JWT must contain a user identifier in one of the following claims:
* `sub` (standard JWT subject claim)
* `user_id` (custom user ID claim)
* `userId` (alternative user ID claim)
* If the JWT is valid, the user ID is extracted and used for rate limiting and tracking (this can be combined with user-level rate limits).
* The user ID will also be added as a meta tag to the request, allowing you to track usage per user.
## Adding Required Claims
You can configure required claims for the JWT in the project key settings. This allows you to enforce specific claims that must be present in the JWT for it to be accepted (for instance, you can require a `tier` claim to control access based on user subscription level).
![Configure Required Claims]()
## Supabase Auth Integration
If you are using Supabase for authentication, you can easily integrate it with Datawizz client access to manager user access to LLMs.
1. **Get Supabase JWT Secret / Public Key**: you'll need the JWT secret or public key from your Supabase project settings so that Datawizz can validate the JWT tokens issued by Supabase.
* Read more in the [Supabase documentation](https://supabase.com/docs/guides/auth/jwts#verifying-a-jwt-from-supabase).
![Configure Supabase JWT Secret]()
2. **Enable Client Access**: Create a project key with Client Access enabled. When prompted, enter the Supabase JWT secret or public key from the previous step.
3. **Use Supabase JWT in Requests**: When making requests to Datawizz, include the Supabase JWT in the `Authorization` header as described above. The format is `Bearer :`, where `` is your Datawizz project key and `` is the Supabase JWT token.
For example, in a JavaScript application using the Supabase client library, you can get the JWT token like this:
```javascript theme={null}
const { data: { session } } = await supabase.auth.getSession();
const jwt = session?.access_token;
const client = new OpenAI({
apiKey: `sk-:${jwt}`,
baseURL: "https://gw.datawizz.app/**************/openai/v1"
});
```
**Yes, you can do this in the browser!** Datawizz client access is designed to work with client-side applications, allowing you to call LLMs directly from the browser or mobile app using the user's JWT.
## Supported Algorithms
Thanks to the `jose` library integration, the system supports multiple JWT signing algorithms:
### HMAC Algorithms (Symmetric)
* **HS256**: HMAC with SHA-256 (recommended for simplicity)
* **HS384**: HMAC with SHA-384
* **HS512**: HMAC with SHA-512
### RSA Algorithms (Asymmetric)
* **RS256**: RSA with SHA-256
* **RS384**: RSA with SHA-384
* **RS512**: RSA with SHA-512
* **PS256**: RSA-PSS with SHA-256
* **PS384**: RSA-PSS with SHA-384
* **PS512**: RSA-PSS with SHA-512
### ECDSA Algorithms (Asymmetric)
* **ES256**: ECDSA with SHA-256
* **ES384**: ECDSA with SHA-384
* **ES512**: ECDSA with SHA-512
## Error Responses
### Missing JWT
```http theme={null}
HTTP/1.1 401 Unauthorized
Content-Type: application/json
{
"error": "JWT required for this key. Format: key:jwt"
}
```
### Invalid Signature
```http theme={null}
HTTP/1.1 401 Unauthorized
Content-Type: application/json
{
"error": "signature verification failed"
}
```
### Expired Token
```http theme={null}
HTTP/1.1 401 Unauthorized
Content-Type: application/json
{
"error": "\"exp\" claim timestamp check failed"
}
```
### Missing Required Claim
```http theme={null}
HTTP/1.1 401 Unauthorized
Content-Type: application/json
{
"error": "Missing required claim: role"
}
```
### Invalid Claim Value
```http theme={null}
HTTP/1.1 401 Unauthorized
Content-Type: application/json
{
"error": "Invalid value for claim role. Got: user"
}
```
### Missing User Identifier
```http theme={null}
HTTP/1.1 401 Unauthorized
Content-Type: application/json
{
"error": "JWT must contain user identifier (sub, user_id, or userId claim)"
}
```
# Rate Limits
Source: https://docs.datawizz.ai/access-control/rate-limits
Control API usage with fine-grained rate limits for keys and users
# Rate Limits
Rate limits provide fine-grained control over API usage, allowing you to prevent abuse and manage resource consumption effectively. The system supports both key-level and user-level limits with multiple resolution types.
## Overview
Rate limits in the Datawizz AI Gateway offer:
* **Key-level limits**: Apply to the entire API key regardless of user
* **User-level limits**: Apply per individual user (requires Client Access with JWT)
* **Multiple resolutions**: MINUTE, HOUR, DAY, MONTH
* **Multiple limit types**: REQUESTS\_LIMIT, TOKENS\_LIMIT
* **Parallel enforcement**: All configured limits are checked simultaneously
## Configuring Rate Limits
Rate limits are managed at the Project Key level - so you can set different limits for different keys (e.g. production key can have different limits than development key).
![Configure Rate Limits]()
To add a rate limit to a key:
1. Go to the **Settings** page of your project.
2. Select the key you want to configure.
3. Click on **Add Rate Limit**.
4. Configure the limit type, resolution, and value.
5. Save the changes.
The limit will be applied immediately and enforced on all requests using that key.
The system collects usage metrics even before you configure rate limits. So when adding a rate limit, it'll take into account all historical usage data.
## Limit Types
### Request Limits
Controls the number of API requests that can be made within a time window.
**Example**: 100 requests per hour
* Tracks each API call as 1 request
* Useful for preventing API abuse and managing load
### Token Limits
Controls the total number of tokens (input + output) consumed within a time window.
**Example**: 10,000 tokens per day
* Tracks actual LLM token usage
* Useful for cost control and resource management
## Resolution Types
Usage tracking is aligned to clock and calendar time -- so an hourly limit resets every hour, a daily limit resets at midnight (UTC), and a monthly limit resets at the start of each month. The system supports the following resolutions:
| Resolution | Description | Use Case |
| ---------- | ----------------- | ---------------------------------------- |
| `MINUTE` | Per-minute limits | Burst protection, real-time applications |
| `HOUR` | Per-hour limits | Standard API rate limiting |
| `DAY` | Per-day limits | Daily usage quotas |
| `MONTH` | Per-month limits | Billing period controls |
## Rate Limit Levels
### Key-Level Limits
Apply to the entire API key, regardless of which user makes the request.
**Use cases:**
* Overall API key quotas
* Preventing single key abuse
* Basic rate limiting for simple use cases
### User-Level Limits
Apply individually to each user identified via JWT (requires Client Access enabled).
**Use cases:**
* Per-user quotas in multi-tenant applications
* Fair usage across different users
* Individual user billing controls
**Requirements:**
* When using a project key without Client Access, you must pass a User ID in the request metadat (`{"user": ""}`).
* When using a project key with Client Access, the User ID is extracted from the JWT claims:
* `sub` (standard claim)
* `user_id` (custom claim)
* `userId` (custom claim)
## Rate Limit Enforcement
### Parallel Checking
All configured rate limits are checked simultaneously. **If ANY limit is exceeded, the request is blocked.**
**Example scenario:**
```
Configured limits:
- Key-level: 1,000 requests per hour
- User-level: 100 requests per hour
If user has made 99 requests this hour:
- User limit: 99/100 ✅ (allowed)
- Key limit: 850/1,000 ✅ (allowed)
- Result: Request allowed
If user has made 100 requests this hour:
- User limit: 100/100 ❌ (exceeded)
- Key limit: 851/1,000 ✅ (allowed)
- Result: Request blocked (429 status)
```
### Response Headers
Rate limit information is included in response headers:
```http theme={null}
X-RateLimit-Requests-HOUR-Limit: 100
X-RateLimit-Requests-HOUR-Remaining: 73
X-RateLimit-Tokens-DAY-Limit: 10000
X-RateLimit-Tokens-DAY-Remaining: 8547
```
**Header format**: `X-RateLimit-{TYPE}-{RESOLUTION}-{Limit|Remaining}`
## Usage Tracking
The system proactively tracks usage across **ALL** possible combinations to enable flexible rate limit configuration:
### Key-Level Tracking
Always tracks 8 combinations for every request:
* REQUESTS\_LIMIT: MINUTE, HOUR, DAY, MONTH (4 entries)
* TOKENS\_LIMIT: MINUTE, HOUR, DAY, MONTH (4 entries)
### User-Level Tracking
When JWT user ID is present, tracks additional 8 combinations:
* Same 8 combinations but scoped to the specific user
* **Total: 16 KV entries per request** (8 key + 8 user)
**Benefits:**
* Add new rate limits anytime with historical data already available
* Flexible configuration changes without losing tracking history
* Supports complex rate limiting scenarios
## Known Limitations
### Header Collisions
When multiple rate limits have the same type and resolution, response headers will collide:
**Problematic configuration:**
```
- Key-level: 100 REQUESTS per HOUR
- User-level: 50 REQUESTS per HOUR
```
**Result:**
* Both limits are enforced correctly ✅
* Headers only show the last processed limit ❌
* Client sees: `X-RateLimit-Requests-HOUR-Limit: 50` (user-level)
* Client doesn't see key-level limit headers
**Workarounds:**
* Use different resolutions (HOUR vs DAY)
* Use different types (REQUESTS vs TOKENS)
* Be aware that enforcement works correctly despite header visibility issues
## Error Responses
### Rate Limit Exceeded
```http theme={null}
HTTP/1.1 429 Too Many Requests
Content-Type: application/json
X-RateLimit-Requests-HOUR-Limit: 100
X-RateLimit-Requests-HOUR-Remaining: 0
{
"error": "Rate limit exceeded: 100 requests per hour"
}
```
## Troubleshooting
### Common Issues
**Rate limits not working:**
* Verify rate limits are properly configured and enabled
* Check that project key has rate limits associated
* Ensure usage tracking KV store is accessible
**Unexpected rate limit blocks:**
* Check if multiple limits are configured (all must pass)
* Verify user-level limits if Client Access is enabled
* Review recent usage patterns and current limit values
**Missing rate limit headers:**
* May indicate header collision with multiple same-type limits
* Check rate limit configuration for duplicates
* Enforcement still works even if headers are missing
**User-level limits not working:**
* Verify Client Access is enabled on the project key
* Ensure JWT contains valid user identifier (`sub`, `user_id`, or `userId`)
* Check that JWT is properly signed and validated
# Virtual Keys
Source: https://docs.datawizz.ai/access-control/virtual-keys
Manage access to AI models and track usage across applications
Datawizz let's you create "virtual keys" to manage access to AI models and track AI usage across different applications or environments.
Datawizz Virtual Keys operate at two levels:
* **Key Based Access**: lets you create an API key for an application or environment (`sk-`).
* **Client Key Access**: lets you define user-level access controls. The API key is combined with a signed JWT token. Lets you define per-user rate limits, and use JWT claims to control access to specific models or features.
Keys are managed at the Project level, and can have global access to all endpoints in a project or fine-grained access to specific models or features.
## Key Based Access
Key Based Access is the simplest way to use Datawizz. You create an API key that can be used by any application or environment. This key can be used to access all models and features available to the key.
To administer keys, go to the **Settings** page of your project where you will see a list of existing keys and options to create new ones.
## Client Key Access
Client Key Access allows you to create API keys that are combined with a signed JWT token. This enables you to define user-level access controls and rate limits. These keys are meant to be publishabled, so they don't work on their own, but only when combined with a JWT token signed with a secret key.
Leard more about [Client Access](../client-access).
## Limiting Endpoint Access
By default, keys have access to all endpoints in a project. You can limit access to specific endpoints by configuring the key's permissions.
When expanding the key details, you will see the "Allowed Endpoints" section. Here you can select which endpoints the key has access to. This allows you to create keys that are limited to specific models or features.
![Configure Allowed Endpoints]()
# Training Apple Foundation Model Adapters
Source: https://docs.datawizz.ai/afm/apple-foundation-model-adapters
Train, evaluate and deploy custom adapter for the Apple Foundation Model for on-device inference.
In WWDC 2025, Apple introduced the [Foundation Models Framework](https://developer.apple.com/documentation/foundationmodels) which allows developers to use the built-in LLM in iOS 26, iPadOS 26 and macOS 26 for fast, private and efficient on-device inference.
The framework supports using custom adapters to adapt the built-in model to specific tasks. This can make the model more useful at a wider variety of tasks which it may not support well out of the box given its limited size.
Datawizz makes it easy to train, evaluate and deploy custom adapters for the Apple Foundation Model. This guide will walk you through the process of training and evaluating a custom adapter using Datawizz, and then using it in your Swift application.
### Why Use Apple Foundation Models?
Apple's on-device models offer several key advantages:
* **Privacy**: All inference happens locally on the device, keeping user data private
* **Cost Efficiency**: No cloud API costs for AI inference
* **Offline Capability**: Works without internet connection
* **Speed**: Optimized for on-device performance with minimal battery drain
However, these models are optimized for efficiency over raw performance. Out of the box, they're not as capable as larger cloud models like GPT-4o or Claude. This is where custom adapters become crucial.
### Performance Comparison
To put Apple's Foundation Model performance in perspective, here are benchmark results from MMLU (Massive Multitask Language Understanding) - a set of 15,000 multiple-choice questions across various subjects:
* **GPT-4o**: 83.88% accuracy (but too large for on-device inference)
* **Meta Llama 3.2 (3B params)**: 50.7% accuracy
* **Microsoft Phi 3 Mini (4B params)**: 59.49% accuracy
* **Google Gemma 2 (2B params)**: 55.99% accuracy
* **Apple Foundation Model**: 44.31% accuracy
While the Apple model performs below other small models initially, custom adapters can dramatically improve its performance for specific tasks - often matching or exceeding much larger models.
# Understanding Adapters
### What Are Adapters?
One common solution for specialized tasks with smaller models is to fine-tune them entirely. However, loading a custom model for each app isn't feasible - even smaller models can take up multiple gigabytes of space.
Adapters offer a lightweight alternative. Instead of training an entire model, you train just a few additional layers (an "adapter") that loads on top of the base model. This provides a "best of both worlds" solution:
* **Quality**: Adapters can improve performance enough to match much larger models for specific tasks
* **Efficiency**: Adapters are only about 160MB in size, making them practical to bundle with apps
* **Flexibility**: Complex apps can even load multiple adapters for different tasks
# Data Collection
To train a custom adapter, you need to collect examples to train it with. We've found that if you are already using an LLM today and looking to replace it with on-device inference, a good starting point is using the prompts and responses you already send and receive from that LLM as a starting point. If you are using platforms like Humanloop, Langfuse or Langsmith, you can easily export the LLM logs from these platforms and import them into Datawizz. [Learn more about importing logs into Datawizz in our documentation on datasets](/models/datasets).
Alternatively, if you are calling LLMs like OpenAI or Anthropic directly, you can use Datawizz to record the requests and responses you send and receive from these LLMs. [Learn more about collecting LLM logs with Datawizz](/logs/recording-requests).
### Amount of Samples Required
Apple's guidelines suggest using at least 100-1,000 samples for basic tasks, and at least 5,000 for more complex tasks. The actual amount of data will depend greatly on the specific task you are trying to adapt the model for.
Generally, the more data you have the better your adapters will perform. However, there are a couple of things to keep in mind:
* **Quality over Quantity**: It's better to have a smaller set of high-quality examples than a large set of low-quality examples. Make sure your examples are representative of the task you are trying to adapt the model for.
* **Diversity**: Make sure your examples cover a wide range of scenarios and edge cases. This will help the model generalize better to new inputs.
* **Relevance**: Make sure your examples are relevant to the task you are trying to adapt the model for. If you are adapting the model for a specific domain, make sure your examples are from that domain.
For extremely complex or diverse use cases, it may make sense to train multiple adapters for different sub-tasks or domains. This can help the model perform better on each specific task, but will require more data and effort to maintain.
# Evaluating the Vanilla Model
Before training an adapter, it's important to establish a baseline by testing the Apple Foundation Model on your specific task. This helps you understand:
* Whether the base model is already sufficient for your needs
* How much improvement an adapter might provide
* What specific areas need the most improvement
1. **Deploy the Apple Foundation Model** to the Datawizz Serverless provider in the providers screen
2. **Open it for manual comparison** - you can test it alongside other models for side-by-side evaluation
3. **Try various prompts** representative of your use case to get a feel for the baseline performance
### Running Automated Evaluations
For more comprehensive testing, you should run automated evaluations:
### Prepare Your Data
1. Go to the **Dataset tab** in Datawizz
2. If you imported logs from another system, they'll already appear as a dataset
3. If you used Datawizz to record logs, create a dataset and import your logs
4. **Create an evaluation split** by clicking "create split" - 20% is usually sufficient for evaluation
### Configure the Evaluation
1. Navigate to the **Evaluation tab** and click "New Evaluation"
2. Select the **Apple Foundation Model** as the model to evaluate
3. Choose your **evaluation dataset**
4. Select appropriate **evaluation functions**:
* String equality for exact matches
* LLM-as-judge for more nuanced evaluation
* Custom metrics specific to your use case
# Training an Adapter
Once you've established your baseline performance, you can begin training a custom adapter to improve the model's performance on your specific task.
### Creating Training and Evaluation Datasets
Before training, ensure you have properly separated your data:
1. **Training Dataset**: Used to train the adapter (typically 80% of your data)
2. **Evaluation Dataset**: Used to test the adapter's performance (typically 20% of your data)
This separation ensures you're testing the adapter on data it hasn't seen during training, giving you an accurate measure of its real-world performance.
### Configuring the Training
1. Navigate to the **Models section** and click "New Model"
2. Choose the **Apple Foundation Model** as your base model
3. Select your **training dataset**
4. Configure training parameters:
### Key Training Parameters
**Epochs**: Controls how many times the trainer runs over your dataset
* More epochs = more training, but risk of overfitting
* Apple models typically perform best with 3-5 epochs
* Start with 3 epochs and adjust based on results
**Learning Rate**: Controls how fast the model learns
* Higher learning rate = faster learning but potentially less stable
* Lower learning rate = more stable but slower convergence
* Use default settings initially, then experiment
### Best Practices
* **Run multiple training sessions** with different parameters to find optimal settings
* **Monitor training logs** to watch for signs of overfitting or undertraining
* **Start with defaults** and iterate based on evaluation results
# Evaluating the Adapter
After training your adapter, it's crucial to evaluate its performance to ensure it's actually improving over the base model. To ready the adapter for evaluation, in the model page once it has finished training click "Deploy Model" and select "Datawizz Serverless" as the provider. This will deploy your adapter to the Datawizz Serverless provider, making it available for evaluation.
### Running Comparative Evaluations
1. Return to the **Evaluations tab**
2. Select your previous evaluation of the base model
3. Click **"Re-run"** and add your newly trained adapter to the benchmark
4. This will run the same evaluation on both models, allowing direct comparison
### Analyzing Results
As results stream in, you should see:
* **Improved accuracy** on your specific task
* **Better consistency** in responses
* **Enhanced performance** on edge cases from your domain
### Iterating on Training
If results aren't satisfactory:
1. **Adjust training parameters** (epochs, learning rate)
2. **Improve training data quality** or add more examples
3. **Refine evaluation metrics** to better capture your needs
4. **Consider training multiple specialized adapters** for different aspects of your task
# Using the Adapter
Once you have a well-performing adapter, the final step is integrating it into your iOS application.
We'll start with a simple example view using a model to generate content:
```swift theme={null}
import SwiftUI
import FoundationModels
struct ChatView: View {
@State private var input: String = ""
@State private var response: String?
@State private var session = LanguageModelSession()
func sendMessage(_ msg: String) {
Task {
do {
let modelResponse = try await session.respond(to: msg)
DispatchQueue.main.async {
self.response = modelResponse.content
}
} catch {
print("Error: \(error)")
}
}
}
var body: some View {
VStack {
if let response = response {
Text(response)
}
HStack {
TextField("Enter your message", text: $input)
.onSubmit {
sendMessage(input)
}
}
}
}
}
```
This code sets up a simple chat interface where users can enter messages and receive responses from the Apple Foundation Model.
### Downloading the Adapter
1. Go to your trained model in Datawizz
2. Download the `.fmadapter` file
3. This file contains your custom adapter weights
### Integration in Swift
### For Testing (Bundle with App)
To use the adapter in your Swift application, you can bundle the `.fmadapter` file with your app. Here's how to do it:
1. Drag the `.fmadapter` file into your Xcode project
2. Ensure it's included in the app bundle
3. Use the following code to load the adapter and create a new session with it:
```swift theme={null}
.task {
do {
if let assetURL = Bundle.main.url(
forResource: "my-adapter",
withExtension: "fmadapter") {
let adapter = try SystemLanguageModel.Adapter(
fileURL: assetURL)
let adaptedModel = SystemLanguageModel(adapter: adapter)
session = LanguageModelSession(model: adaptedModel)
} else {
print("Asset not found in the main bundle.")
}
} catch {
print("Error: \(error)")
}
}
```
This is not recommended for production apps, as it requires bundling the adapter with the app, which can increase the app size and make updates more complex.
### For Production (Asset Packs - Recommended)
For production apps, it's better to use Asset Packs to manage your adapters. This allows you to download the adapter at runtime, keeping your app size smaller and allowing for easier updates.
See the [Apple documentation on Asset Packs](https://developer.apple.com/apple-intelligence/foundation-models-adapter/#bundle-adapter-asset-packs) for more details on how to implement this.
# Edit Log
Source: https://docs.datawizz.ai/api-reference/endpoint/patch_log
PATCH /log/{id}
Edit an existing log in the system
# Submit Feedback Signal
Source: https://docs.datawizz.ai/api-reference/endpoint/post_feedback
POST /{project_uid}/feedback/{inference_log_id}
Submit a feedback signal to assess model performance for a specific inference. Feedback signals are used for reinforcement learning and model improvement.
**Authentication**: Requires the same API key authentication as inference requests. For JWT-enabled keys, the user ID from the JWT will be included in the feedback metadata.
# Introduction
Source: https://docs.datawizz.ai/api-reference/introduction
Additional API documentation for the Datawizz Platform
Most of your interaction with the Datawizz platform will be through the OpenAI / Anthropic compatible endpoints. This reference only documents the additional endpoints available to you
# Continuous Learning
Source: https://docs.datawizz.ai/continuous-learning
Use Datawizz to continuously improve your models with new data
Traditional post-training is handled as a discrete linear process: collect labeled data, train a model, evaluate it and deploy it. Models trained in that approach can quickly become stale as new data and use cases emerge. Continuous learning enables models to evolve over time by incorporating new data and feedback, allowing them to adapt to changing environments and improve their performance continuously. Datawizz makes it easy to close the loop and keep your models up to date with fresh data.
## The Continuous Learning Loop
![Continuous Learning Loop]()
* **Logs and Traces**: continuous learning starts with collecting runtime data - including LLM call logs, traces, and user feedback. This data captures how the model is being used in real-world scenarios.
* **Observability**: using the collected data, understand model performance, identify failure modes, and detect areas for improvement.
* **Prompt Engineering**: refine and optimize prompts based on insights from observability to improve model responses. Use manual adjustments or automated prompt tuning techniques.
* **Supervised Fine Tuning**: leverage the collected data to fine-tune models on specific tasks or domains. This can involve training on new datasets derived from logs and traces, helping the model adapt to evolving requirements.
* **Reinforcement Fine Tuning**: incorporate user feedback and reward signals to further refine model behavior. This helps align the model with user expectations and desired outcomes.
* **Evaluations**: after each adjustment (prompt changes, fine-tuning), evaluate the model's performance against a mixture of golden datasets and real-world data to ensure improvements are realized.
* **Deployment**: deploy the updated model into production, replacing or augmenting the previous version. Monitor its performance continuously to ensure it meets desired standards.
* **Guardrails, Plugins and Routing**: implement guardrails, plugins, and routing mechanisms to ensure the model operates safely and effectively in production environments. These tools help manage risks and enhance the model's capabilities.
**Repeat the cycle: continuous learning is an iterative process. Regularly repeat the loop to keep the model up to date and responsive to changing needs.**
## Continuous Learning with Datawizz
Datawizz enables the entire continuous learning cycle - letting you close the loop faster, ship updates more quickly, and maintain high model performance over time. The key insight is connecting runtime data collection & augmentation with model training and evaluation. In this section, we'll walk you through setting up your Datawizz project for continuous learning.
### 1 - Data Collection - Logs, Online Evals and User Feedback
The first step in continuous learning is collecting runtime data. Datawizz automatically captures LLM call logs - simply route your LLM calls through Datawizz and the logs will be stored in your project:
![Proxy LLM Calls through Datawizz]()
**>> Get started with routing your LLM calls through Datawizz in our [Quickstart Guide](/quickstart).**
Beyond LLM call logs, we recommend collecting online evaluations and user feedback signals. Datawizz supports online evals that can be run in real-time as part of your application flow. You can also collect user feedback (e.g. thumbs up/down, ratings, comments) to capture qualitative insights on model performance.
Online evaluators can be configured in your Datawizz endpoint, and run automatically on every LLM log (asynchronously):
![Online Evaluations in Datawizz]()
**>> Learn more about setting up [Online Evaluations](/evaluation/custom-evaluators).**
User feedback can be collected via your application and sent to Datawizz using our API. We recommend capturing **explicit feedback** (e.g. user ratings) as well as **implicit feedback** (e.g. user behavior signals like re-tries, time spent, etc).
**>> Learn more about sending [User Feedback to Datawizz](/logs/feedback).**
### 2 - Observability - Monitor Model Performance
The Datawizz Dashboard lets you monitor model performance over time using the collected logs, online evals, and user feedback. Use the dashboard to identify failure modes, track key metrics, and uncover areas for improvement.
![Observability Dashboard in Datawizz]()
You can also drill down into individual logs to understand specific model behaviors and issues:
![Observability Logs in Datawizz]()
**>> Learn more about [Managing Logs in Datawizz](/logs/metadata-and-tagging).**
### 3 - Prompt Engineering - Refine Prompts Based on Insights
Use Datawizz to experiment with prompt engineering based on insights from observability. You can create and test different prompt variations directly in the Datawizz dashboard:
![Prompt Engineering in Datawizz]()
**>> Check out recent updates on [Prompt Management in Datawizz](https://datawizz.ai/blog/leveling-up-prompt-management-with-liquid-templates).**
Datawizz also lets you automatically tune prompts based on logs, custom evaluators and user feedback. This helps optimize prompt performance over time:
![Automated Prompt Tuning in Datawizz]()
### 4 + 5 - Fine Tuning - Supervised and Reinforcement Learning
Datawizz makes it easy to fine-tune your models using the collected data. You can create fine-tuning datasets directly from logs, online evals, and user feedback and launch fine-tuning jobs in just a few clicks:
![Fine Tuning in Datawizz]()
**>> Learn more about [Fine-Tuning Models with Datawizz](/models/fine-tuning).**
### 6 - Evaluations - Continuous Testing and Validation
After each model update (prompt changes, fine-tuning), use Datawizz to evaluate model performance against a mix of golden datasets and real-world data. Datawizz lets you set up evaluation jobs and use custom evaluators to validate results - either defined in code (Python) or using LLM as Judge.
![Evaluations in Datawizz]()
**>> Learn more about [Automated Evaluations in Datawizz](/evaluation/automated-evaluations).**
### 7 - Deployment - Seamless Model Updates
Once you've validated model improvements, deploy the updated model directly from Datawizz
![Deployment in Datawizz]()
**>> Learn more about [Model Deployment with Datawizz](/models/model-deployment).**
Datawizz also enables gradual rollouts and A/B testing to safely deploy new models and monitor their performance in production. You can route a percentage of traffic to the new model version and compare results against the previous version. Simply adjust weights for the different models in the endpoint upstream settings:
![Model Weights in Datawizz]()
**>> Learn more about [Model Routing and Weighting](/endpoints/routing).**
### 8 - Guardrails, Plugins and Routing - Safe and Effective Operations
Datawizz helps you further enhance model performance and safety in production using guardrails, plugins, and routing mechanisms.
* **[Plugins & Guardrails](/plugins/plugins)** let you extend model capabilities and enforce safety constraints.
* **[Routing](/endpoints/routing)** enables intelligent request routing based on input characteristics, user segments, or other criteria.
### ∞ - Rinse and Repeat
Continuous learning is an iterative process. Regularly repeat the loop to keep your models up to date and responsive to changing needs. Datawizz provides all the tools you need to close the loop faster and maintain high model performance over time.
For model fine tuning - Datawizz lets you re-train models on fresh data as it comes in. Simply create new fine-tuning datasets from the latest logs and feedback, and launch new fine-tuning jobs using older models as the base, to continuously adapt to new data.
## Setting Up for Success with Continuous Learning
There are a few best practices to keep in mind when implementing continuous learning with Datawizz:
* **Invest in Evaluators**: robust evaluators are critical for measuring model performance and guiding improvements. Invest time in developing high-quality custom evaluators that capture key aspects of your use cases.
* Datawizz lets you define a set of custom evaluators that can be used across online evals, fine-tuning datasets, and evaluation jobs. This ensures consistency in how you measure performance throughout the continuous learning loop.
* Crafting evaluators is where you should likely be spending most of your time - they are the foundation for effective continuous learning.
* **Leverage User Feedback**: user feedback is a valuable source of insights for continuous learning. Make it easy for users to provide feedback and incorporate it into your learning loop.
* Collect both explicit feedback (ratings, comments) and implicit feedback (user behavior signals).
* Use feedback to inform prompt engineering, fine-tuning datasets, and evaluation criteria.
* **Use rich Metadata and Tagging**: Datawizz supports extensive metadata and tagging capabilities. Use these features to organize and filter logs, datasets, and models effectively.
* Tag logs with relevant context (e.g. user segments, use cases) to facilitate targeted analysis and dataset creation.
* Use metadata to track model versions, fine-tuning parameters, and evaluation results.
# Attachment Processing [Beta]
Source: https://docs.datawizz.ai/endpoints/attachment-processing
Send a wide variety of files to be processed by our AI models, including videos, office documents and PDFs.
Many LLMs are limited to processing text, and in some cases images and audio (multi-modal models). Datawizz extends these native LLM capabilities by allowing you to send additional file types to these LLMs. Datawizz performs pre-processing to convert these files into content understandable by the LLMs.
For instance, with videos - Datawizz can extract frames, audio channels and transcription to provide a rich set of data to the LLMs. Similarly, office documents and PDFs can be converted into markdown for the LLMs to process.
***
# Video Inputs
Datawizz makes it easy to send videos to the LLMs and have them reason about the content of the video.
* **Source:** Datawizz supports downloading videos from URLs, including services like YouTube, Vimeo, TikTok and more.
* **Proxy:** Datawizz can also use resedential proxies automatically to download videos from these services.
* **Processing:** Datawizz extracts frames, audio and transcriptions from the video. You can configure the number of frames to extract, the audio channels to process and the transcription settings.
* **flexibility:** this processing is compatible with all LLMs - you can select the LLM you want to use for reasoning about the video content.
![Video Processing]()
*Log view of a video processing request, where you can see the frames + transcript sent to the LLM*
### Sending Videos
To send a video attachment, you can send it as a part of a message to the LLM (similar to sending images with the OpenAI API):
```json theme={null}
{
"model": "video-testing",
"messages": [
{
"role": "user",
"content": [
{
"type": "text",
"text": "Can you describe what's in the video below? I have included some frames from the video with a timestamp burned in, as well as the transcript from the video"
},
{
"type": "video_url",
"video_url": {
"url": "https://www.youtube.com/watch?v=R9RRtMCdmSY",
"sample_fps": 1
}
}
]
}
]
}
```
### Configurations
Alongside the video URL, you can specify additional configurations to control how the video is processed:
| Parameter | Type | Description | Default |
| -------------------- | ------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------- |
| `url` | string | The URL of the video to process. | (required) |
| `use_proxy` | boolean | Whether to use a residential proxy to download the video. | `false` |
| `proxy_country` | string | The country of the proxy to use. | None (global) |
| `detail_level` | string | The level of detail (resolution) for the frames sent to the LLM. See details below. | None (use video resolution) |
| `sample_fps` | number | The number of frames to sample from the video per second. | 1 |
| `sample_frames` | number | The total number of frames to sample from the video. If supplied, will be used instead of FPS, and the defined number of frames will be extracted from the video (equally spaced) | None |
| `start_offset` | number | Process the video from a specific timestamp (in seconds) | 0 |
| `end_offset` | number | Process the video until a specific timestamp (in seconds) | None (process until end of video) |
| `burn_timestamps` | boolean | Whether to burn timestamps into the frames sent to the LLM. Useful reasoning about events in the video. Timestamps will have the format HH:MM:SS.ss | `true` |
| `timestamp_location` | string | The location of the timestamp on the frame. Can be `top-left`, `top-right`, `bottom-left`, `bottom-right`. Use this to position the timestamp where it is less likely to hide important visual information | `bottom-left` |
| `include_audio` | boolean | If true, will send the video's audio track to the LLM (**Can only be used with LLMs that suppport audio input**) | `false` |
| `include_transcript` | boolean | If true, will send the video's transcript to the LLM. We use OpenAI `gpt-4o-transcribe` to create the transcript | `false` |
| `filetype` | string | The file type of the video. Can be `mp4`, `webm`, `mov`, `avi`, etc. This is used to determine how to process the video. | None (inferred from file mime type) |
#### Detail Levels
The `detail_level` parameter allows you to control the resolution of the frames sent to the LLM. The available options are:
* `low`: 512 pixels (longest side)
* `medium`: 768 pixels (longest side)
* `high`: 1024 pixels (longest side)
Most LLMs charge images based on their resoliution, so you can use this parameter to control the cost of processing the video. The default is `None`, which means the frames will be sent at their original resolution.
### Prompting for Videos
Datawizz will automatically replace the `video_url` message part with image message parts (and potentially a text part for the transcript) when sending the request to the LLM. Your prompt need to explain to the LLMs that these are frames from the video, and that the transcript is a transcription of the audio in the video.
For exampple, you can use the following prompt:
```json theme={null}
{
"role": "user",
"content": [
{
"type": "text",
"text": "Can you describe what's in the video below? I have included some frames from the video with a timestamp burned in, as well as the transcript from the video"
},
{
"type": "video_url",
"video_url": {
"url": "https://www.youtube.com/watch?v=R9RRtMCdmSY",
"sample_fps": 1
}
}
]
}
```
This will result in an LLM request that looks like this:
```json theme={null}
{
"role": "user",
"content": [
{
"type": "text",
"text": "Can you describe what's in the video below? I have included some frames from the video with a timestamp burned in, as well as the transcript from the video"
},
{
"type": "image_url",
"image_url": {
"url": "https://datawizz.io/attachments/1234567890/frame_1.jpg",
}
},
{
"type": "image_url",
"image_url": {
"url": "https://datawizz.io/attachments/1234567890/frame_2.jpg",
}
},
...
{
"type": "text",
"text": "[transcript text here]"
}
}
```
You can combine video processing with other LLM features like structured output to generate structured insights from your video. If using timestamps, you can use structured output for event identification in videos
# Document Inputs
Datawizz can process a wide variety of document types, including PDFs, office documents (Word, Excel, PowerPoint) and more. These documents are converted into markdown format for the LLMs to process.
Datawizz uses Microsoft's [`MarkItDown`](https://github.com/microsoft/markitdown) library to convert these documents into markdown, which is then sent to the LLMs for processing.
### Sending Documents
To send a document attachment, you can send it as a part of a message to the LLM (similar to sending images with the OpenAI API):
```json theme={null}
{
"model": "document-processing",
"messages": [
{
"role": "user",
"content": [
{
"type": "text",
"text": "Can you summarize the document below?"
},
{
"type": "document",
"document": {
"url": "https://example.com/my-document.pdf"
}
}
]
}
]
}
```
### Configurations
Alongside the document URL, you can specify additional configurations to control how the document is processed:
| Parameter | Type | Description | Default |
| --------------------------- | ------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------ |
| `url` | string | The URL of the document to process. | (required or data) |
| `data` | string | The base64 encoded content of the document. If provided, this will be used instead of the URL. Should be a data URI (`data:application/pdf;base64,...`) | None |
| `use_llm_image_description` | boolean | Whether to use the LLM's image description capabilities to generate a description of the document. This is useful for documents that contain images or complex layouts. | `false` |
# Routing
Source: https://docs.datawizz.ai/endpoints/routing
Learn how to route requests to different models based on custom logic
Datawizz enables you to route your requests to different models (and even different providers) to ensure every request is handled by the right model. This can be useful when training Specialzied Models for specific tasks - but we even find that with LLMs, different models can perform better on different types of inputs. So even if you aren't deploying custom models, you can still benefit from routing requests to different models.
Routing logic is managed at the endpoint level - your code calls an **endpoint**, which has one or more **upstreams** that it routes requests to. Each upstream can be a different model, and you can configure the routing logic to send requests to the right model based on the request content.
## Load Balancing Multiple Models
When you add an upstream to an endpoint, you can specify it's weight. The weight determines how often requests are sent to that upstream. For example, if you have two upstreams with weights 1 and 2, then 1/3 of the requests will be sent to the first upstream and 2/3 to the second upstream.
This can be useful for load balancing between models, or for A/B testing different models to see which performs better. It can also be useful for gradually rolling out a new model - you can start with a low weight and gradually increase it as you gain confidence in the new model.
The system will still note the actual model that served the request, so you can analyze the performance of each model separately or train new models based on the performance of the existing models.
![Load Balancing]()
## Routing Requests Based on tags
When you add an upstream to an endpoint, you can specify a tag filter. This alows you to tell Datawizz to only send requests to that endpoint if they contain (or don't contain) a specific tag. This can be useful for routing requests to different models based on the content of the request.
Some example scenarios where this can be useful:
* Domain specific models: You may have a model that is trained on a specific domain, and you want to route requests to that model only if the request contains content from that domain.
* Language specific models: You may have a model that is trained on a specific language, and you want to route requests to that model only if the request is in that language.
* Task specific models: You may have a model that is trained on a specific task, and you want to route requests to that model only if the request is for that task.
* Subscription tier based routiog - you may have different models for different subscription tiers, and you want to route requests to the right model based on the subscription tier of the user.
To configure a tag filter, simply specify the tags to include/exclude the upstream for when you add the upstream.
![Load Balancing]()
Load balancing weights and tag filters can be used together. If you have multiple upstreams with different weights and tag filters, the system will first filter the upstreams based on the tag filters, and then apply the load balancing weights to the applicable upstreams.
## Model Fallbacks
Datawizz can help you improve reliability by using fallbacks - if the primary model fails to respond, Datawizz will automatically send the request to the fallback model. This can be useful for ensuring that requests are always handled, even if one of your models is down.
To leverage fallbacks, turn on the fallbacks option at the endpoint level and ensure there is more than one upstream. When a request comes in, Datawizz will select the first upstream based on weights. If the first upstream fails to respond, Datawizz will automatically send the request to the next upstream, and so forth until a response is received. If all upstreams fail to respond, Datawizz will return an error to the client.
You can define upstreamd with weight 0. These will never get selected as a primary - but only as a fallback after other options have failed.
![Load Balancing]()
# Structured Outputs
Source: https://docs.datawizz.ai/endpoints/structured-outputs
Generate consistent JSON responses with JSON Schema
Certain models and providers have the ability to generate stuctured outputs - JSON responses that adhere to a specific schema. This can be useful for downstream applications that expect a specific format for the response.
Datawizz extands support for Structured Output in two key ways:
1. For models that support structured output, Datawizz will pass your JSON Schema to leverage the built-in validation capabilities of the model.
2. Additionally, Datawizz can independenly validate the output of the model against the JSON Schema you provide. This is usefull for models that don't support structured output.
This is particularly powerful when combined with [routing fallbacks](/endpoints/routing). You can route requests to a smaller model that doesn't support structured output, then Datawizz will test the output, and if it isn't valid - route it to a larger model that does support structured output.
This is a common scenario with Llama 3.2 models - which don't support structured output, but have good results for data extraction. You can route requests to them, test the output on datawizz, and if it's not valid - route it to an OpenAI model that does support structured output.
```mermaid theme={null}
sequenceDiagram
Your App->>+Datawizz: Request + JSON Schema
Datawizz->>+Llama 3.2: Request
Llama 3.2->>-Datawizz: Response
Datawizz->>+Datawizz: Validate JSON Schema
Datawizz->>+Your App: (If Valid) Response
Datawizz-->>+OpenAI: (If Invalid) Request + JSON Schema
OpenAI-->>+Datawizz: Response
Datawizz->>+Your App: Response
```
## Configuring Structured Outputs
To use Datawizz Structured output validation, turn on the JSON Validation option at the end point level. This will run JSON validation if a JSON Schema is supplied in the request, and fail the request if the model response fails validation.
![Load Balancing]()
To send a JSON Schema, use the OpenAI style API described here: [https://platform.openai.com/docs/guides/structured-outputs](https://platform.openai.com/docs/guides/structured-outputs)
Here's an example request with a JSON Schema:
```bash theme={null}
curl -X "POST" "http://localhost:8787/***********/openai/v1/chat/completions" \
-H 'Authorization: Bearer sk-***********' \
-H 'Content-Type: application/json' \
-d $'{
"messages": [
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "Return the population, area, country, and name of the city."}
],
"model": "***********",
"response_format": {
"type": "json_schema",
"json_schema": {
"strict": true,
"name": "extract_city_data",
"description": "Return ONLY a JSON object describing the city with the following keys: name, country, population, area.",
"schema": {
"properties": {
"country": {
"type": "string",
"description": "The country the city is in."
},
"area": {
"type": "number",
"description": "City area in KM as a numeric value with up to two decimal places."
},
"name": {
"type": "string",
"description": "The city name."
},
"population": {
"type": "number",
"description": "The city\'s population as a numeric value."
}
},
"type": "object",
"required": [
"name",
"country",
"population",
"area"
],
"additionalProperties": false
}
}
},
"max_completion_tokens": 1024
}'
```
# Automated Evaluations
Source: https://docs.datawizz.ai/evaluation/automated-evaluations
Evaluate AI models automatically using Datawizz
Datawizz has a powerfull evaluations feature which lets you evaluate the performance of multiple models using your real data, and based on several evaluation metrics. This can help you identify the best performing model and make informed decisions about which model to deploy.
Evaluations are critical not only for evaluating custom trained models, but also for understanding the performance of various public models from providers like OpenAI. When newer models are relased, it's cruicial to evaluate them on your data to understand how they perform compared to your existing models. New models are not always betetr for every use case.
## Creating an evaluation
To create an evaluation, head over to the evaluations section in the Datawizz dashboard. Here you can create a new evaluation by clicking the "New Evaluation" button.
![New Evaluation]()
Here you'll need to configure your evaluation with the following settings:
* **Name**: A name for your evaluation. This is for your reference and will be displayed in the dashboard.
* **Models**: The models you want to evaluate. You can select multiple models from the same provider or different providers. (note that models you train must be deployed before they can be evaluated)
* **Data**: Select the logs you want to use for evaluation. You can use tags and other filters to select the logs you want to use.
* If you are training custom models, you may want to reserve some logs for evaluation to ensure you are not testing on the same data you trained on. You can do this by tagging some logs with `test` and excluding them from the training data when you initiate a model training.
* If you have feedback for your logs, you should filter for positive (👍) logs to evaluate against.
* **Maximum Sample Count**: The maximum number of samples to use for evaluation. This is useful if you have a large dataset and want to limit the number of samples used for evaluation.
When evaluating different models, a request is made to each model for each log in the evaluation data. You bear the cost for these requests, so be sure to set a reasonable maximum sample count to avoid excessive usage.
* **Evaluators**: Select the evaluation functions you want to use. You can select multiple evaluation functions to get a comprehensive view of model performance. You can easily use our built-in evaluation functions, or you can create your own custom evaluation functions using code or LLM prompts (LLM as Judge). See the full list of available evaluation functions below.
## Viewing evaluation results
Once you've created an evaluation, you can view the results in the evaluation details page. Here you'll see a summary of the evaluation, including the evaluation function used, the number of samples evaluated, and the evaluation metrics. You will also be able to see individual responses from each model and compare them side-by-side.
![Evaluation Results]()
## Custom Evaluators
You can define custom evaluation functions to suit your specific needs. Datawizz supports two types of custom evaluators:
1. **Code Evaluators**: Python-based evaluators that give you full programmatic control
2. **LLM as Judge Evaluators**: Prompt-based evaluators that leverage language models
Learn how to create custom evaluators in the [Custom Evaluators](/evaluation/custom-evaluators) section.
## Built In Evaluation Functions
Datawizz provides a comprehensive set of built-in evaluation functions to help you assess model performance across different dimensions. These evaluators cover various aspects of text generation quality, from semantic similarity to exact matching.
### BERT Score (en)
BERT Score is a semantic similarity metric that leverages contextual embeddings from BERT and its variants to measure how well generated text matches reference text. Unlike traditional n-gram based metrics, BERT Score captures semantic meaning and can recognize paraphrases and synonymous expressions. It returns precision, recall, and F1 scores based on token-level similarity in the embedding space.
```python theme={null}
from evaluate import load
def evaluator(inputs: list, outputs: dict, reference_outputs: dict):
metric = load('bertscore')
results = metric.compute(references=[outputs.get('content', '')], predictions=[reference_outputs.get('content', '')], lang="en")
return {
"f1": results.get("f1", 0),
"recall": results.get("recall", 0),
"precision": results.get("precision", 0)
}
```
### Rouge Score
ROUGE (Recall-Oriented Understudy for Gisting Evaluation) measures the overlap between generated text and reference text by comparing shared words or n-gram sequences. It's particularly useful for evaluating summarization tasks and provides multiple variants (ROUGE-1, ROUGE-2, ROUGE-L) that focus on different aspects of text overlap, from individual words to longest common subsequences.
```python theme={null}
from evaluate import load
def evaluator(inputs: list, outputs: dict, reference_outputs: dict):
metric = load('rouge')
results = metric.compute(references=[outputs.get('content', '')], predictions=[reference_outputs.get('content', '')])
return results
```
### SacreBLEU
SacreBLEU is a standardized implementation of the BLEU metric specifically designed for machine translation evaluation. It provides consistent and reproducible results by using standardized tokenization and normalization procedures. The metric measures the overlap between generated text and reference text using n-gram precision with a brevity penalty to discourage overly short outputs.
```python theme={null}
from evaluate import load
def evaluator(inputs: list, outputs: dict, reference_outputs: dict):
metric = load('sacrebleu')
results = metric.compute(references=[outputs.get('content', '')], predictions=[reference_outputs.get('content', '')])
return results['score']/100
```
### CHRF
ChrF (CHaRacter-level F-score) is a character-based evaluation metric that calculates similarity between machine translation output and reference translation using character n-grams rather than word n-grams. This approach makes it particularly effective for morphologically rich languages and helps capture fine-grained differences in text generation quality, including spelling variations and morphological changes.
```python theme={null}
from evaluate import load
def evaluator(inputs: list, outputs: dict, reference_outputs: dict):
metric = load('chrf')
results = metric.compute(references=[outputs.get('content', '')], predictions=[reference_outputs.get('content', '')])
return results['score']/100
```
### Character Error Rate (CER)
Character Error Rate (CER) evaluates text generation accuracy by calculating the minimum number of character-level insertions, deletions, and substitutions needed to transform the generated text into the reference text. The result is normalized by the total number of characters in the reference text. This metric is particularly useful for tasks requiring high precision, such as transcription or data extraction, where character-level accuracy is critical.
```python theme={null}
from evaluate import load
def evaluator(inputs: list, outputs: dict, reference_outputs: dict):
metric = load('cer')
results = metric.compute(references=[outputs.get('content', '')], predictions=[reference_outputs.get('content', '')])
return 1 - results
```
### Word Error Rate (WER)
Word Error Rate (WER) measures the accuracy of generated text by calculating the minimum number of word-level insertions, deletions, and substitutions required to match the reference text. Originally developed for speech recognition evaluation, WER is also valuable for assessing LLM outputs where word-level accuracy is important. The metric is normalized by the total number of words in the reference text.
```python theme={null}
from evaluate import load
def evaluator(inputs: list, outputs: dict, reference_outputs: dict):
metric = load('wer')
results = metric.compute(references=[outputs.get('content', '')], predictions=[reference_outputs.get('content', '')])
return 1 - results
```
### String Equality - Case Insensitive
This evaluator performs exact string matching while ignoring case differences and normalizing whitespace. It converts both texts to lowercase, trims leading and trailing whitespace, and collapses multiple consecutive spaces into single spaces. This is useful for evaluating tasks where the exact content matters but formatting variations should be ignored, such as classification labels or standardized responses.
```python theme={null}
import json
def preprocess_string(text):
if not text:
return ""
## LOWERCASE
text = text.lower()
## TRIM
text = text.strip()
## EXTRA WHITESPACE
text = " ".join(text.split())
return text
def evaluator(inputs: list, outputs: dict, reference_outputs: dict):
return preprocess_string(outputs['content']) == preprocess_string(reference_outputs['content'])
```
### String Equality - Case Sensitive
This evaluator performs exact string matching while preserving case sensitivity but still normalizing whitespace issues. It trims leading and trailing whitespace and collapses multiple consecutive spaces into single spaces. This is ideal for tasks where case matters (such as proper nouns or code generation) but minor formatting inconsistencies should be overlooked.
```python theme={null}
import json
def preprocess_string(text):
if not text:
return ""
## TRIM
text = text.strip()
## EXTRA WHITESPACE
text = " ".join(text.split())
return text
def evaluator(inputs: list, outputs: dict, reference_outputs: dict):
return preprocess_string(outputs['content']) == preprocess_string(reference_outputs['content'])
```
### METEOR
METEOR (Metric for Evaluation of Translation with Explicit ORdering) is an advanced text generation evaluation metric that goes beyond simple n-gram matching. It incorporates exact matches, stemming, synonyms, and paraphrases to provide a more nuanced comparison between generated and reference text. METEOR also includes a penalty for incorrect word order, making it better aligned with human judgment compared to simpler metrics like BLEU.
```python theme={null}
from evaluate import load
def evaluator(inputs: list, outputs: dict, reference_outputs: dict):
metric = load('meteor')
results = metric.compute(references=[outputs.get('content', '')], predictions=[reference_outputs.get('content', '')])
return results['meteor']
```
### BLEU
BLEU (Bilingual Evaluation Understudy) is a widely-used metric for evaluating machine-generated text, particularly translations. It measures the overlap of n-gram sequences between the candidate output and reference texts, providing scores for 1-gram through 4-gram precision. BLEU applies a brevity penalty to discourage overly short generations and returns both an overall BLEU score and individual n-gram precision scores.
```python theme={null}
from evaluate import load
def evaluator(inputs: list, outputs: dict, reference_outputs: dict):
metric = load('bleu')
results = metric.compute(references=[outputs.get('content', '')], predictions=[reference_outputs.get('content', '')])
return {
'bleu':results['bleu'],
'bleu1':results['precisions'][0],
'bleu2':results['precisions'][1],
'bleu3':results['precisions'][2],
'bleu4':results['precisions'][3]
}
```
### JSON Compare
This specialized evaluator is designed for tasks that generate JSON output. It compares the structure and content of JSON responses by calculating key overlap and value overlap scores. The evaluator handles common LLM formatting issues by automatically removing markdown code blocks (\`\`\`json) and validates that both outputs are valid JSON. It provides detailed metrics including key overlap, value overlap, JSON validity, and boolean flags for complete key/value matching.
````python theme={null}
import json
import re
def preprocess_json_string(s):
s = re.sub(r"^```(json|xml)|```$", "", s).strip()
return s
def json_compare(baseline_output, new_output):
baseline_output_string = preprocess_json_string(baseline_output)
new_output_string = preprocess_json_string(new_output)
baseline_output_json = None
new_output_json = None
try:
baseline_output_json = json.loads(baseline_output_string)
new_output_json = json.loads(new_output_string)
except Exception:
return {"json_valid": False}
finally:
if baseline_output_json is None or new_output_json is None:
return {"json_valid": False}
# Calculate key overlap and value overlap
baseline_keys = set(baseline_output_json.keys())
new_keys = set(new_output_json.keys())
key_overlap = baseline_keys.intersection(new_keys)
key_overlap_score = len(key_overlap) / len(baseline_keys.union(new_keys))
overlapping_values = 0
for key in key_overlap:
if baseline_output_json[key] == new_output_json[key]:
overlapping_values += 1
value_overlap_score = overlapping_values / len(key_overlap)
return {
"key_overlap": key_overlap_score,
"value_overlap": value_overlap_score,
"json_valid": True,
"has_all_keys": len(baseline_keys) == len(new_keys),
"has_all_values": overlapping_values == len(key_overlap),
}
def evaluator(inputs: list, outputs: dict, reference_outputs: dict):
return json_compare(reference_outputs['content'], outputs['content'])
````
# Custom Evaluators
Source: https://docs.datawizz.ai/evaluation/custom-evaluators
Create and use custom evaluators in Datawizz
Datawizz provides a comprehensive set of built-in evaluators for common use cases, but sometimes you need more specialized evaluation logic. Custom evaluators allow you to implement domain-specific evaluation criteria, complex scoring algorithms, or novel assessment methods tailored to your unique requirements.
Custom evaluators bridge the gap between generic metrics and your specific business needs. Whether you're evaluating specialized technical content, implementing proprietary scoring methods, or assessing outputs against custom quality criteria, custom evaluators give you the flexibility to define exactly what "good" means for your use case.
There are two types of custom evaluators you can create in Datawizz:
1. **Code Evaluators**: Python-based evaluators that give you full programmatic control
2. **LLM as Judge Evaluators**: Prompt-based evaluators that leverage language models for nuanced assessment
# Code Evaluators
Code evaluators provide the ultimate flexibility for custom evaluation logic. They run in a secure Python environment and can implement any evaluation algorithm you need, from simple rule-based checks to complex machine learning models.
### Runtime Environment
Your Python code runs in an isolated environment with the following specifications:
* **Python runtime**: 3.8+
* **Execution time limit**: 5 minutes per evaluation
* **Memory limit**: Reasonable limits for typical evaluation tasks
* **Package Management**: You specify your own dependencies using PEP 723 inline script metadata
* **No Pre-installed Libraries**: You must declare all dependencies you need
**Declaring Dependencies:**
Use PEP 723 inline script metadata at the top of your evaluator code to specify required packages:
```python theme={null}
# /// script
# requires-python = ">=3.8"
# dependencies = [
# "evaluate>=0.4.0",
# "rouge-score>=0.1.2",
# "numpy>=1.21.0"
# ]
# ///
from evaluate import load
import numpy as np
def evaluator(inputs, outputs, reference_outputs):
# Your evaluation logic here
pass
```
**How it works:**
* Dependencies are installed automatically in an isolated environment on first run (\~30-60 seconds)
* Subsequent runs are fast (\~2-5 seconds) because packages are cached
* This ensures your evaluator always has the exact dependencies it needs without conflicts
### Function Signature
All code evaluators must implement the following function signature:
```python theme={null}
def evaluator(inputs: list, outputs: dict, reference_outputs: dict):
# Your evaluation logic here
pass
```
**Parameters:**
* `inputs`: List of input data that was sent to the model
* `outputs`: Dictionary containing the model's response (typically includes a 'content' key)
* `reference_outputs`: Dictionary containing the expected/reference output for comparison
### Return Types
Your evaluator function can return different types depending on your evaluation needs:
#### Boolean Returns
Perfect for pass/fail evaluations:
```python theme={null}
def evaluator(inputs: list, outputs: dict, reference_outputs: dict):
# Check if output contains required keywords
required_keywords = ["safety", "compliance"]
output_text = outputs.get('content', '').lower()
return all(keyword in output_text for keyword in required_keywords)
```
#### Numeric Scores (0-1)
Ideal for continuous scoring where 1 represents perfect performance:
```python theme={null}
def evaluator(inputs: list, outputs: dict, reference_outputs: dict):
# Calculate semantic similarity using word overlap
output_words = set(outputs.get('content', '').lower().split())
reference_words = set(reference_outputs.get('content', '').lower().split())
if not reference_words:
return 0.0
intersection = len(output_words.intersection(reference_words))
union = len(output_words.union(reference_words))
return intersection / union if union > 0 else 0.0
```
#### String Returns
Useful for classification or categorical evaluation:
```python theme={null}
def evaluator(inputs: list, outputs: dict, reference_outputs: dict):
# Classify response tone
content = outputs.get('content', '').lower()
if any(word in content for word in ['excellent', 'amazing', 'fantastic']):
return 'positive'
elif any(word in content for word in ['terrible', 'awful', 'horrible']):
return 'negative'
else:
return 'neutral'
```
#### Dictionary Returns
Perfect for multi-dimensional evaluation:
```python theme={null}
def evaluator(inputs: list, outputs: dict, reference_outputs: dict):
content = outputs.get('content', '')
# Evaluate multiple aspects
return {
'clarity': calculate_clarity_score(content),
'completeness': calculate_completeness_score(content, reference_outputs.get('content', '')),
'accuracy': calculate_factual_accuracy(content),
'tone': assess_tone_appropriateness(content)
}
```
Example -- checking single letter answers from multiple choice questions:
```python theme={null}
import re
def extract_answer_letter(s):
"""
Extract a single letter (A-Z) answer from a noisy string.
Returns the uppercase letter if found, else None.
"""
if not isinstance(s, str):
return None
match = re.match(r"""
[\s"'(\[]*
([A-Za-z])
[)\]'"\.]*
(\s|$)
""", s, re.VERBOSE)
if match:
return match.group(1).upper()
return None
def evaluator(inputs: list, outputs: dict, reference_outputs: dict):
return extract_answer_letter(outputs['content']) == extract_answer_letter(reference_outputs['content'])
```
This evaluator handles the common scenario where models might return verbose responses to multiple choice questions like "The answer is (B) because..." or "B) Machine Learning" and extracts just the letter for comparison.
### Best Practices for Code Evaluators
#### Keep Evaluators Focused and Reusable
Design small, single-purpose evaluators that can be combined for comprehensive evaluation:
```python theme={null}
# Good: Focused evaluator for length checking
def evaluator(inputs: list, outputs: dict, reference_outputs: dict):
content = outputs.get('content', '')
min_length = 50 # characters
max_length = 500
length = len(content)
if length < min_length:
return 0.0
elif length > max_length:
return max(0.0, 1.0 - (length - max_length) / max_length)
else:
return 1.0
```
#### Handle Edge Cases Gracefully
Always account for missing data, empty responses, or unexpected formats:
```python theme={null}
def evaluator(inputs: list, outputs: dict, reference_outputs: dict):
# Robust handling of potential issues
content = outputs.get('content')
if not content or not isinstance(content, str):
return 0.0
reference_content = reference_outputs.get('content')
if not reference_content or not isinstance(reference_content, str):
return 0.0
# Your evaluation logic here
return calculate_similarity(content, reference_content)
```
#### Use Multiple Evaluators for Complex Assessment
Rather than creating one monolithic evaluator, use multiple focused evaluators:
#### Performance Note
Testing your evaluator in the Datawizz interface may be slower than production execution due to the additional overhead of the testing environment. The actual evaluation runs will be significantly faster.
### Common Use Cases for Code Evaluators
#### Format Validation
Ensure outputs follow specific formatting requirements:
```python theme={null}
import json
def evaluator(inputs: list, outputs: dict, reference_outputs: dict):
content = outputs.get('content', '')
try:
# Check if output is valid JSON
parsed = json.loads(content)
# Check for required fields
required_fields = ['name', 'age', 'email']
has_all_fields = all(field in parsed for field in required_fields)
return has_all_fields
except json.JSONDecodeError:
return False
```
#### Domain-Specific Validation
Implement business logic specific to your domain:
```python theme={null}
def evaluator(inputs: list, outputs: dict, reference_outputs: dict):
content = outputs.get('content', '').lower()
# Medical safety check - ensure no dangerous advice
dangerous_phrases = [
'stop taking medication',
'ignore doctor advice',
'medical emergency can wait'
]
contains_dangerous_advice = any(phrase in content for phrase in dangerous_phrases)
return not contains_dangerous_advice # Return False if dangerous content found
```
#### Advanced Similarity Metrics
Implement sophisticated comparison algorithms:
```python theme={null}
# /// script
# requires-python = ">=3.8"
# dependencies = [
# "scikit-learn>=1.0.0",
# "numpy>=1.21.0"
# ]
# ///
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics.pairwise import cosine_similarity
import numpy as np
def evaluator(inputs: list, outputs: dict, reference_outputs: dict):
output_text = outputs.get('content', '')
reference_text = reference_outputs.get('content', '')
if not output_text or not reference_text:
return 0.0
# Create TF-IDF vectors
vectorizer = TfidfVectorizer()
texts = [output_text, reference_text]
try:
tfidf_matrix = vectorizer.fit_transform(texts)
similarity = cosine_similarity(tfidf_matrix[0:1], tfidf_matrix[1:2])[0][0]
return float(similarity)
except:
return 0.0
```
## LLM as Judge Evaluators
The **LLM as Judge** evaluation method represents a paradigm shift in how we assess AI-generated content. Instead of relying solely on traditional metrics like BLEU scores or exact string matching, this approach leverages the sophisticated understanding of language models to evaluate outputs in a more nuanced, human-like manner.
This method is particularly powerful for tasks where the quality of output can't be easily quantified through simple rules or mathematical formulas. Creative writing, conversational responses, complex reasoning, and subjective assessments all benefit from the contextual understanding that LLM judges provide.
### How LLM as Judge Works
The evaluation process follows these steps:
1. **Baseline Establishment**: The original log output serves as the reference standard
2. **Prompt Construction**: A carefully crafted prompt instructs the judge model on evaluation criteria
3. **Contextual Assessment**: The judge model receives the input, candidate output, and baseline output
4. **Scoring**: The judge returns a score between 0 and 1, with detailed reasoning
5. **Aggregation**: Multiple evaluations can be combined for comprehensive assessment
**Key Principles:**
* **Scores range from 0 to 1**: 1 indicates perfect alignment with the baseline, 0 indicates poor performance
* **Contextual Understanding**: The judge considers nuance, intent, and quality beyond surface-level matching
* **Consistency**: While individual judgments may vary slightly, the overall evaluation trends remain stable
* **Transparency**: Many judge models can provide reasoning for their scores
### Task-Specific Prompt Templates
Datawizz provides pre-configured prompt templates optimized for different types of evaluation tasks. Each template is designed to elicit the most accurate and relevant judgments from the LLM judge. You can use these templates as starting points and customize them with specific criteria, examples, or domain knowledge.
#### Question Answering
Evaluates how well the model answered a question compared to a reference answer. This template considers correctness, completeness, and relevance.
```
Evaluate how well the candidate answered the input question/instruction given the baseline answer.
Input: {{input}}
Candidate: {{output}}
Baseline: {{baseline_output}}
Please provide a score from 0 to 1 where:
- 1.0: Perfect answer that matches or exceeds the baseline quality
- 0.8-0.9: Very good answer with minor differences from baseline
- 0.6-0.7: Good answer but missing some important details
- 0.4-0.5: Partially correct but significant gaps or errors
- 0.2-0.3: Poor answer with major issues
- 0.0-0.1: Completely incorrect or irrelevant answer
```
#### Classification
Assesses whether the model correctly classified the input according to the expected category or label.
```
Given the user input below, evaluate if the candidate output is a correct classification according to the baseline output classification.
Input: {{input}}
Candidate: {{output}}
Baseline: {{baseline_output}}
Score 1.0 if the classifications match exactly, 0.5 for partially correct classifications, and 0.0 for completely incorrect classifications.
```
#### Summarization
Evaluates the quality of summaries by comparing key information retention, conciseness, and accuracy.
```
Evaluate how well the candidate summarized the input given the baseline summary.
Input: {{input}}
Candidate: {{output}}
Baseline: {{baseline_output}}
Consider the following criteria:
- Information retention: Are key points preserved?
- Conciseness: Is the summary appropriately brief?
- Accuracy: Are facts correctly represented?
- Coherence: Is the summary well-structured and readable?
Provide a score from 0 to 1 reflecting overall summary quality.
```
#### Entity Extraction
Assesses how accurately the model identified and extracted requested entities from the input.
```
Evaluate how well the candidate extracted the requested elements (as requested in the input) given the baseline extracted entities.
Input: {{input}}
Candidate: {{output}}
Baseline: {{baseline_output}}
Score based on:
- Completeness: Were all entities found?
- Accuracy: Are the extracted entities correct?
- Format: Is the output properly formatted?
```
#### Code Generation
Evaluates generated code for correctness, efficiency, and adherence to requirements.
```
Evaluate how well the candidate generated the requested code (as requested in the input) given the baseline code.
Input: {{input}}
Candidate: {{output}}
Baseline: {{baseline_output}}
Consider:
- Functional correctness: Does the code work as intended?
- Code quality: Is it well-structured and readable?
- Requirement adherence: Does it meet the specified requirements?
- Best practices: Does it follow coding conventions?
```
#### Generic
A flexible template for general similarity assessment when specific task templates don't apply.
```
Evaluate how similar the candidate output is to the baseline output.
Input: {{input}}
Candidate: {{output}}
Baseline: {{baseline_output}}
Assess overall similarity considering meaning, structure, and quality. Provide a score from 0 to 1.
```
### Best Practices for LLM as Judge
#### Choose the Right Task Template
Selecting the appropriate template significantly impacts evaluation quality. Match your use case to the most relevant template:
* **Question Answering**: For factual Q\&A, reasoning tasks, or instructional responses
* **Classification**: For categorization, sentiment analysis, or labeling tasks
* **Summarization**: For content condensation or key point extraction
* **Entity Extraction**: For named entity recognition or information extraction
* **Code Generation**: For programming tasks or technical documentation
* **Generic**: When no specific template fits or for novel evaluation criteria
#### Provide Clear Examples and Context
Enhance your prompts with specific examples to guide the judge model:
```
Evaluate the candidate's explanation of machine learning concepts.
Example of excellent explanation (score 1.0):
"Machine learning uses algorithms to find patterns in data and make predictions..."
Example of poor explanation (score 0.2):
"ML is when computers learn stuff from data and do things..."
Input: {{input}}
Candidate: {{output}}
Baseline: {{baseline_output}}
```
#### Be Specific About Evaluation Criteria
Define exactly what constitutes good performance for your domain:
```
Evaluate the customer service response quality. Consider:
1. Empathy (30%): Shows understanding of customer's situation
2. Accuracy (40%): Provides correct information and solutions
3. Professionalism (20%): Maintains appropriate tone and language
4. Completeness (10%): Addresses all aspects of the customer's query
Score from 0 to 1 based on these weighted criteria.
```
#### Consider the Judge Model Capabilities
The quality of evaluation depends heavily on the judge model's capabilities:
* **Use capable models**: More advanced models generally provide better judgments
* **Match model to task**: Some models excel at specific types of evaluation
* **Test your prompts**: Validate that the judge model understands your criteria
* **Monitor consistency**: Check that similar inputs receive similar scores
#### Account for Subjectivity
LLM judges may have different perspectives on subjective matters:
```
Evaluate the creativity of this story opening. Note that creativity can be subjective,
so focus on concrete elements:
- Originality of concept or approach
- Unexpected or innovative elements
- Engaging narrative voice
- Vivid and imaginative details
Provide reasoning for your score along with the numerical rating.
```
### When to Use LLM as Judge
LLM as Judge evaluation shines in scenarios where traditional metrics fall short. Here are the key use cases where this approach provides the most value:
#### Conversational AI and Chatbots
Traditional metrics can't capture the nuances of natural conversation:
* **Helpfulness**: Does the response actually help the user?
* **Appropriateness**: Is the tone and content suitable for the context?
* **Engagement**: Does the response encourage continued interaction?
* **Empathy**: Does the bot show understanding of user emotions?
*Example*: A customer asking "I'm frustrated with my order delay" needs empathetic acknowledgment, not just factual shipping information.
#### Creative and Subjective Content
When creativity and style matter more than factual accuracy:
* **Creative writing**: Stories, poems, marketing copy
* **Content generation**: Blog posts, social media content, product descriptions
* **Artistic critique**: Evaluating creative elements in generated content
* **Style adaptation**: Matching specific writing styles or brand voices
*Example*: A marketing tagline's effectiveness depends on creativity, memorability, and brand alignment—qualities only human-like judgment can assess.
#### Complex Reasoning and Explanation
For tasks requiring multi-step thinking and clear communication:
* **Educational content**: Explanations of complex concepts
* **Problem-solving**: Step-by-step reasoning processes
* **Technical documentation**: Clarity and completeness of instructions
* **Analytical reports**: Quality of insights and recommendations
*Example*: Explaining quantum physics requires not just accuracy but also appropriate analogies and progressive concept building.
#### Open-Ended and Subjective Tasks
When there are multiple valid approaches or subjective quality criteria:
* **Advisory responses**: Personal recommendations or guidance
* **Opinion pieces**: Balanced argumentation and perspective
* **Design critiques**: Aesthetic and functional evaluation
* **Strategic planning**: Quality of strategic thinking and recommendations
*Example*: Career advice must be personalized, contextually appropriate, and consider multiple factors—qualities that traditional metrics can't capture.
#### Quality Beyond Correctness
When correctness alone isn't sufficient:
* **Professional communication**: Email drafting, formal correspondence
* **Customer service**: Response quality beyond just solving the problem
* **Content moderation**: Nuanced judgment about appropriateness
* **Translation quality**: Cultural adaptation beyond literal accuracy
*Example*: A technically correct but cold customer service response may score high on accuracy but low on customer satisfaction.
### Limitations and Considerations
While LLM as Judge offers powerful evaluation capabilities, it's important to understand its limitations and plan accordingly:
#### Cost Implications
* **API Usage**: Each evaluation requires a call to the judge model, increasing operational costs
* **Volume Scaling**: Large-scale evaluations can become expensive quickly
* **Model Selection**: More capable judge models typically cost more per evaluation
* **Mitigation**: Use LLM as Judge selectively for high-value evaluations; combine with cheaper automated metrics where appropriate
#### Consistency Challenges
* **Inter-run Variability**: The same input may receive slightly different scores across evaluations
* **Model Dependencies**: Different judge models may have varying scoring patterns
* **Prompt Sensitivity**: Small changes in prompts can significantly affect evaluation outcomes
* **Mitigation**: Use multiple evaluations and statistical averaging; establish baseline consistency metrics; test prompt variations thoroughly
#### Inherent Biases
* **Training Data Bias**: Judge models reflect biases present in their training data
* **Cultural Perspectives**: Models may favor certain cultural or linguistic patterns
* **Domain Limitations**: Judges may perform poorly in specialized domains they weren't trained on
* **Subjective Interpretations**: What constitutes "quality" may vary between different judge models
* **Mitigation**: Use diverse judge models; regularly audit evaluation results for bias; incorporate human validation for critical assessments
#### Performance Considerations
* **Evaluation Speed**: LLM calls are slower than traditional metrics, affecting evaluation throughput
* **Network Dependencies**: Requires stable internet connection and API availability
* **Rate Limiting**: API rate limits may constrain evaluation speed
* **Latency Variability**: Response times can vary based on model load and complexity
* **Mitigation**: Implement async evaluation pipelines; use local models where possible; design evaluation workflows to handle latency
#### Quality Dependencies
* **Judge Model Capability**: Evaluation quality is fundamentally limited by the judge model's abilities
* **Prompt Engineering**: Poor prompts lead to poor evaluations regardless of judge model quality
* **Context Limitations**: Very long inputs may exceed model context windows
* **Domain Expertise**: Judges may lack specialized knowledge for technical domains
* **Mitigation**: Select appropriate judge models for your domain; invest in prompt engineering; break down complex evaluations into smaller components
### Hybrid Evaluation Strategies
For optimal results, consider combining LLM as Judge with other evaluation methods:
#### Multi-Tier Evaluation
1. **Automated Metrics**: Fast, cheap screening (BLEU, ROUGE, exact match)
2. **LLM as Judge**: Nuanced quality assessment for promising candidates
3. **Human Review**: Final validation for critical decisions
#### Complementary Metrics
* Use traditional metrics for factual accuracy
* Use LLM as Judge for style, tone, and appropriateness
* Use human evaluation for final quality assurance
#### Adaptive Evaluation
* Start with cheap automated metrics
* Escalate to LLM as Judge based on automated metric thresholds
* Reserve human evaluation for edge cases and high-stakes decisions
This approach balances cost, speed, and evaluation quality while maximizing the benefits of each evaluation method.
## Choosing the Right Evaluation Approach
The choice between Code Evaluators and LLM as Judge depends on your specific needs, resources, and evaluation requirements:
### Use Code Evaluators When:
* **Deterministic Logic**: You need consistent, repeatable results
* **Performance Critical**: Speed and cost efficiency are primary concerns
* **Clear Criteria**: Success can be defined through explicit rules or calculations
* **Domain Expertise**: You have specific knowledge that can be encoded programmatically
* **Integration Needs**: You need to incorporate external APIs, databases, or complex algorithms
### Use LLM as Judge When:
* **Subjective Assessment**: Quality depends on nuanced human-like judgment
* **Complex Context**: Evaluation requires understanding of implicit meaning or context
* **Creative Content**: Assessing originality, style, or creative quality
* **Natural Language**: Evaluating conversational or explanatory content
* **Rapid Prototyping**: You need to quickly test evaluation concepts without coding
### Hybrid Approaches:
Many successful evaluation strategies combine both methods:
* Use code evaluators for objective criteria (format, length, factual accuracy)
* Use LLM as Judge for subjective quality (tone, creativity, helpfulness)
* Implement progressive evaluation: fast automated screening followed by detailed LLM assessment
## Getting Started
1. **Define Your Evaluation Goals**: Clearly articulate what "good" means for your specific use case
2. **Start Simple**: Begin with basic evaluators and iterate based on results
3. **Test Thoroughly**: Validate your evaluators against known good and bad examples
4. **Monitor Performance**: Track evaluation consistency and adjust as needed
5. **Scale Thoughtfully**: Consider cost and performance implications as you scale
Custom evaluators are powerful tools that let you align AI evaluation with your specific quality standards and business requirements. Whether you choose code-based logic or LLM-based judgment, the key is to match your evaluation method to your specific needs and constraints.
# Manual Evaluation
Source: https://docs.datawizz.ai/evaluation/manual
Learn how to compare multiple models side-by-side and evaluate their performance
Evaluating the performance of your AI models is a crucial step in the model development process. Datawizz provides a simple way to compare multiple models side-by-side and evaluate their performance. This can help you identify the best performing model and make informed decisions about which model to deploy.
## Comparing models
The datawizz platform has a convinient UI for sending requests to multiple models and comparing their responses side-by-side. You can use this to compare both public models from providers like OpenAI and custom models trained on your own data.
Please noe that custom trained models must be deployed before they can be evaluated. Please deploy your model to Datawizz after training before proceeding with the evaluation.
To evaluate models:
1. Head over to the providers section in the Datawizz dashboard.
2. Select the models you want to compare under the different providers. You can select multiple models from the same provider or different providers.
3. Click the "Compare" button on the top right corner of the page.
In the comparison view, you can either create your own messages, or open the "Saved Prompts" drawer to use inputs fron your existing request logs.
# Using Datawizz with Langfuse
Source: https://docs.datawizz.ai/integrations/langfuse
Use Datawizz with Langfuse
Langfuse offers a powerful way to manage prompts, track AI usage and sessions and evaluate AI platforms. Datawizz can be used alongside Langfuse to train custom SLMs fine-tuned for your use case and perform smart-routing between multiple models.
## Using Datawizz + Langfuse in Runtime
The Datawizz gateway is compatible with Langfuse clients for both Python and Node.js. You can simply follow the instructions on using the Langfuse OpenAI SDK, using Datawizz Base URL and API Key patameters.
### Python
```python theme={null}
from langfuse.openai import OpenAI
client = OpenAI(
api_key="sk-your_openai_api_key", # <--- your datawizz project API key
base_url="https://gw.datawizz.app/**************/openai/v1", # <--- your datawizz project base URL
)
response = client.chat.completions.create(
model="********", # <--- this is the DataWizz Endpoit you are routing to,
messages=[
{
"role": "user",
"content": [
{"type": "text", "text": "What's the speed of light?"},
],
}
]
)
```
Langfuse python reference: [Langfuse Python](https://langfuse.com/docs/integrations/openai/python/get-started)
### Javascript / Typescript
```javascript theme={null}
import OpenAI from "openai";
import { observeOpenAI } from "langfuse";
const openai = observeOpenAI(new OpenAI(
apiKey: "sk-your_openai_api_key", // <--- your datawizz project API key
baseUrl: "https://gw.datawizz.app/**************/openai/v1", // <--- your datawizz project base URL
));
const completion = await openai.chat.completions.create({
model: "********", // <--- this is the DataWizz Endpoit you are routing to
messages: [
{ role: "user", content: "Write a haiku about recursion in programming." },
],
});
```
Langfuse javascript reference: [Langfuse Javascript](https://langfuse.com/docs/integrations/openai/js/get-started)
Once you have this set up, you cna configure your Datawizz endpoint to route to different models. Datawizz will also log these transactions for farther SLM training and evaluation.
## Using Existing Langfuse
If you have an existing project using Langfuse and have collected logs, you can upload these logs to Datawizz for training and evaluation. Datawizz supports importing logs from Langfuse in CSV format. You can use the Datawizz dashboard to create a new dataset and import logs from a CSV file.
In the Langfuse dashboard, select the traces you wish to use and export them as a CSV. The langfuse CSV format is compatible with Datawizz, so you can upload it as-is to be used for training.
![Langfuse CSV]()
Read more about exporting from Langfuse: [https://langfuse.com/docs/fine-tuning](https://langfuse.com/docs/fine-tuning)
Read more about uploading CSVs and using them for training: [Using Datasets](/models/datasets)
# Using Datawizz with LangSmith
Source: https://docs.datawizz.ai/integrations/langsmith
Export LangSmith traces and import them into Datawizz
LangSmith is LangChain's platform for debugging, testing, and monitoring LLM applications. If you have existing traces in LangSmith, you can export them and import them into Datawizz for training custom SLMs and evaluation.
## Exporting Traces from LangSmith
We provide a Python script that exports LangSmith traces to a CSV file compatible with Datawizz. The script extracts comprehensive metadata from your LLM calls including messages, tokens, costs, and latency information.
### Prerequisites
You'll need a LangSmith API key. You can get one from: [https://smith.langchain.com/settings](https://smith.langchain.com/settings)
Set your API key as an environment variable:
```bash theme={null}
export LANGSMITH_API_KEY=your_api_key_here
```
Optionally, if you're using a custom LangSmith endpoint:
```bash theme={null}
export LANGSMITH_ENDPOINT=https://your-custom-endpoint.com
```
### Running the Export Script
You can run the script using Python directly or with `uv`:
```bash theme={null}
# With Python
python export_traces_to_csv.py
# Or with uv (no manual dependency installation required)
uv run --with langsmith export_traces_to_csv.py
```
The script is available as a [GitHub Gist](https://gist.github.com/iddogino/ccddf21638e607a1b5b00ec557489999).
### Output Format
The script creates a `langsmith_traces.csv` file with 21 columns per LLM call:
| Category | Columns |
| --------------- | ------------------------------------------------------------------------- |
| **Core** | `trace_id`, `run_id`, `parent_run_id`, `input_messages`, `output_message` |
| **Model** | `model`, `provider`, `temperature`, `finish_reason` |
| **Status** | `status`, `error` |
| **Tokens** | `total_tokens`, `prompt_tokens`, `completion_tokens` |
| **Cost** | `total_cost`, `prompt_cost`, `completion_cost` |
| **Performance** | `latency_ms`, `start_time`, `end_time` |
| **Metadata** | `tags` |
The `input_messages` column uses OpenAI message format, making it directly compatible with Datawizz dataset imports.
## Importing into Datawizz
Once you have your exported CSV file:
1. Navigate to the **Datasets** tab in the Datawizz dashboard
2. Create a new dataset or open an existing one
3. Click **Import CSV** and select your `langsmith_traces.csv` file
4. Map the `input_messages` column to `input` and `output_message` to `output`
Read more about uploading CSVs and using them for training: [Using Datasets](/models/datasets)
# Introduction
Source: https://docs.datawizz.ai/introduction
Welcome to the home of your new documentation
![Logo]()
Datawizz empowers companies to replace large and expensive generic models like GPT-4 or Claude-3.5 with their own specialized models. Datawizz is an OpenAI compatible solution that records your LLM interactions and uses them to train new, small models to serve your use case with lower cost and higher performance. Best of all, it generates a model that YOU own and control - removing your dependency on OpenAI / Anthropic.
Start with our quickstart guide to set up on Datawizz
# Anonymizing Logs (PII Removal)
Source: https://docs.datawizz.ai/logs/anonymizing-logs
Remove PII from logs to ensure user privacy & compliance
Datawizz lets you configure endpoints to annonimize logs by removing personally identifiable information (PII) from the logs. This can be useful to ensure user privacy and comply with regulations like GDPR.
If there's a high risk of users sending sensative PII to your AI system, you should consider enabling PII removal to ensure that this information is not stored in the logs. You can also select specific entity types to remove from logs to constrain PII removal scope.
## Configuring PII Removal
To enable PII removal, turn on the `Anonymize Logs` option at the endpoint settings. You may also configure two additional options:
* **Score Threshold**: The minimum confidence score required to remove an entity from the log. If the entity has a confidence score below this threshold, it will not be removed - this is a number between 0 and 1, with 0 being the lowest confidence and 1 being the highest.
* **Entities**: a comma separated list of entity types to remove from the logs. If you leave this field blank, all entities will be removed. See supported entity types below.
## List of supported entities
### Global
| Entity Type | Description | Detection Method |
| ---------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------- |
| CREDIT\_CARD | A credit card number is between 12 to 19 digits. [https://en.wikipedia.org/wiki/Payment\_card\_number](https://en.wikipedia.org/wiki/Payment_card_number) | Pattern match and checksum |
| CRYPTO | A Crypto wallet number. Currently only Bitcoin address is supported | Pattern match, context and checksum |
| DATE\_TIME | Absolute or relative dates or periods or times smaller than a day. | Pattern match and context |
| EMAIL\_ADDRESS | An email address identifies an email box to which email messages are delivered | Pattern match, context and RFC-822 validation |
| IBAN\_CODE | The International Bank Account Number (IBAN) is an internationally agreed system of identifying bank accounts across national borders to facilitate the communication and processing of cross border transactions with a reduced risk of transcription errors. | Pattern match, context and checksum |
| IP\_ADDRESS | An Internet Protocol (IP) address (either IPv4 or IPv6). | Pattern match, context and checksum |
| NRP | A person’s Nationality, religious or political group. | Custom logic and context |
| LOCATION | Name of politically or geographically defined location (cities, provinces, countries, international regions, bodies of water, mountains | Custom logic and context |
| PERSON | A full person name, which can include first names, middle names or initials, and last names. | Custom logic and context |
| PHONE\_NUMBER | A telephone number | Custom logic, pattern match and context |
| MEDICAL\_LICENSE | Common medical license numbers. | Pattern match, context and checksum |
| URL | A URL (Uniform Resource Locator), unique identifier used to locate a resource on the Internet | Pattern match, context and top level url validation |
### USA
| Entity Type | Description | Detection Method |
| ------------------- | ------------------------------------------------------------------------------------------------------------------------------- | ------------------------- |
| US\_BANK\_NUMBER | A US bank account number is between 8 to 17 digits. | Pattern match and context |
| US\_DRIVER\_LICENSE | A US driver license according to [https://ntsi.com/drivers-license-format/](https://ntsi.com/drivers-license-format/) | Pattern match and context |
| US\_ITIN | US Individual Taxpayer Identification Number (ITIN). Nine digits that start with a "9" and contain a "7" or "8" as the 4 digit. | Pattern match and context |
| US\_PASSPORT | A US passport number with 9 digits. | Pattern match and context |
| US\_SSN | A US Social Security Number (SSN) with 9 digits. | Pattern match and context |
### UK
| Entity Type | Description | Detection Method |
| ----------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ----------------------------------- |
| UK\_NHS | A UK NHS number is 10 digits. | Pattern match, context and checksum |
| UK\_NINO | UK [National Insurance Number](https://en.wikipedia.org/wiki/National_Insurance_number) is a unique identifier used in the administration of National Insurance and tax. | Pattern match and context |
### Spain
| Entity Type | Description | Detection Method |
| ----------- | ------------------------------------------- | ----------------------------------- |
| ES\_NIF | A spanish NIF number (Personal tax ID) . | Pattern match, context and checksum |
| ES\_NIE | A spanish NIE number (Foreigners ID card) . | Pattern match, context and checksum |
### Italy
| Entity Type | Description | Detection Method |
| ------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------- |
| IT\_FISCAL\_CODE | An Italian personal identification code. [https://en.wikipedia.org/wiki/Italian\_fiscal\_code](https://en.wikipedia.org/wiki/Italian_fiscal_code) | Pattern match, context and checksum |
| IT\_DRIVER\_LICENSE | An Italian driver license number. | Pattern match and context |
| IT\_VAT\_CODE | An Italian VAT code number | Pattern match, context and checksum |
| IT\_PASSPORT | An Italian passport number. | Pattern match and context |
| IT\_IDENTITY\_CARD | An Italian identity card number. [https://en.wikipedia.org/wiki/Italian\_electronic\_identity\_card](https://en.wikipedia.org/wiki/Italian_electronic_identity_card) | Pattern match and context |
### Poland
| Entity Type | Description | Detection Method |
| ----------- | ------------------- | ----------------------------------- |
| PL\_PESEL | Polish PESEL number | Pattern match, context and checksum |
### Singapore
| FieldType | Description | Detection Method |
| ------------- | ------------------------------------------------------------------------------------------------------ | ------------------------------------ |
| SG\_NRIC\_FIN | A National Registration Identification Card | Pattern match and context |
| SG\_UEN | A Unique Entity Number (UEN) is a standard identification number for entities registered in Singapore. | Pattern match, context, and checksum |
### Australia
| FieldType | Description | Detection Method |
| ------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------ |
| AU\_ABN | The Australian Business Number (ABN) is a unique 11 digit identifier issued to all entities registered in the Australian Business Register (ABR). | Pattern match, context, and checksum |
| AU\_ACN | An Australian Company Number is a unique nine-digit number issued by the Australian Securities and Investments Commission to every company registered under the Commonwealth Corporations Act 2001 as an identifier. | Pattern match, context, and checksum |
| AU\_TFN | The tax file number (TFN) is a unique identifier issued by the Australian Taxation Office to each taxpaying entity | Pattern match, context, and checksum |
| AU\_MEDICARE | Medicare number is a unique identifier issued by Australian Government that enables the cardholder to receive a rebates of medical expenses under Australia's Medicare system | Pattern match, context, and checksum |
### India
| FieldType | Description | Detection Method |
| ------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------ |
| IN\_PAN | The Indian Permanent Account Number (PAN) is a unique 12 character alphanumeric identifier issued to all business and individual entities registered as Tax Payers. | Pattern match, context |
| IN\_AADHAAR | Indian government issued unique 12 digit individual identity number | Pattern match, context, and checksum |
| IN\_VEHICLE\_REGISTRATION | Indian government issued transport (govt, personal, diplomatic, defence) vehicle registration number | Pattern match, context, and checksum |
| IN\_VOTER | Indian Election Commission issued 10 digit alpha numeric voter id for all indian citizens (age 18 or above) | Pattern match, context |
| IN\_PASSPORT | Indian Passport Number | Pattern match, Context |
### Finland
| FieldType | Description | Detection Method |
| ---------------------------- | ------------------------------------------------------------------------------------------------------- | ---------------------------------------- |
| FI\_PERSONAL\_IDENTITY\_CODE | The Finnish Personal Identity Code (Henkilötunnus) is a unique 11 character individual identity number. | Pattern match, context and custom logic. |
## Credits
This feature is powered by Microsoft's [Presidio](https://microsoft.github.io/presidio/) framework.
# Feedback Signals
Source: https://docs.datawizz.ai/logs/feedback
Collect structured feedback signals to improve AI model performance
Collecting feedback on AI responses is essential for improving model performance. Datawizz provides a powerful feedback signals system that allows you to gather multiple types of feedback from various sources and use them for reinforcement learning, fine-tuning, and observability.
Unlike traditional thumbs up/down systems, feedback signals offer a structured approach that supports multiple feedback sources (users and automated evaluators), different signal types (explicit and implicit), weighted importance, and rich contextual information.
## Understanding Feedback Signals
Each feedback signal contains several components that provide a comprehensive assessment of model performance:
| Field | Type/Range | Required | Description |
| ------------------------ | ------------------------ | -------- | --------------------------------------------------------------------------------- |
| **Score** | -1 to 1 | ✓ | Quality rating of the response, where -1 is the worst and 1 is the best |
| **Weight** | 0 to 1 | ✓ | Importance of this signal, where 0 is least important and 1 is most important |
| **Improvement** | JSON object | Optional | An improved version of the response for supervised fine-tuning |
| **Qualitative Feedback** | String | Optional | Text feedback explaining the assessment (used for RLHF) |
| **Signal Name** | String | Optional | Identifier for the signal/sensor (e.g., "user\_thumbs\_up", "response\_accuracy") |
| **Signal Source** | `user` or `system` | Optional | Origin of the feedback signal |
| **Signal Type** | `explicit` or `implicit` | Optional | Classification of the feedback type |
| **Metadata** | JSON object | Optional | Additional context for observability and analysis |
This structured approach allows you to:
* Combine multiple feedback sources into a holistic quality assessment
* Weight different signals based on their reliability or importance
* Track both user sentiment and automated evaluation metrics
* Provide corrected responses for supervised fine-tuning
* Generate detailed observability reports by signal type, source, or name
## Adding Feedback Signals Through the Dashboard
You can manually add feedback signals to your inference logs through the Datawizz dashboard. This is useful for reviewing batches of logs, conducting manual evaluations, or adding expert feedback.
To add a feedback signal:
1. Navigate to the Logs section and open any inference log
2. In the log details section, you'll see a "Feedback Signals" section
3. Click "Add Feedback Signal" to create a new signal
4. Fill in the signal details including score, weight, and optional fields
5. Save the signal
You can add multiple feedback signals to a single inference log, allowing you to capture different perspectives or evaluation criteria.
## Sending Feedback Signals Programmatically (via API)
For production applications where users interact with AI-generated responses, you'll want to collect feedback programmatically through the Datawizz API.
### When to Collect Feedback
#### Explicit Feedback
Explicit feedback occurs when users deliberately provide feedback by clicking buttons, providing ratings, or submitting corrections. This might include:
* Thumbs up/down buttons
* Star ratings
* Correction submissions
* Quality assessments
Explicit feedback is straightforward to implement and provides clear signals, though participation rates may be lower.
#### Implicit Feedback
Implicit feedback is inferred from user behavior without requiring explicit action. Examples include:
* User saves or shares the AI response (positive signal)
* User copies the response to their clipboard (positive signal)
* User requests regeneration or modification (negative signal)
* User ignores or dismisses the response (negative signal)
* User spends significant time reading the response (positive signal)
Implicit feedback offers more natural user experiences and higher participation rates, but requires careful interpretation and may be less accurate than explicit signals.
### Sending Feedback Signals via API
To send a feedback signal via the API, use the `POST /{project_uid}/{endpoint_uid}/feedback/{inference_log_id}` endpoint.
When you generate a response through the OpenAI or Anthropic compatible API endpoints, Datawizz returns the log ID in the response. Use this log ID to submit feedback signals.
#### Feedback JWT Tokens
Each inference response includes an `X-Feedback-Token` header containing a signed JWT that can be used to submit feedback without requiring your API key. This is perfect for client-side feedback collection where you don't want to expose your API key.
**How It Works:**
1. **Token Generation**: The gateway generates a signed JWT for each inference response
2. **Token Contents**: The JWT contains:
* `inference_id`: The unique identifier for the inference
* `user_id`: (optional) The user ID if client JWT authentication was used
3. **Token Usage**: Use this JWT as a Bearer token to submit feedback securely from client-side code
4. **Token Expiration**: Tokens are valid for 24 hours after generation
5. **Authentication**: The feedback endpoint accepts either:
* **JWT auth**: `Authorization: Bearer ` (recommended for client-side)
* **API key auth**: `Authorization: Bearer ` (for server-side)
**Response Headers:**
Every inference response includes:
* `X-Feedback-Url`: The URL to submit feedback (e.g., `https://gw.datawizz.app/{project}/{endpoint}/feedback/{inference_id}`)
* `X-Feedback-Token`: The signed JWT token for feedback authentication (valid for 24 hours)
**Example: Using Feedback JWT Tokens (Client-Side)**
```javascript theme={null}
// 1. Make an inference request
const response = await fetch('https://gw.datawizz.app/{project}/{endpoint}/chat/completions', {
method: 'POST',
headers: {
'Authorization': 'Bearer your-api-key',
'Content-Type': 'application/json'
},
body: JSON.stringify({ messages: [...] })
});
// 2. Extract feedback headers
const feedbackUrl = response.headers.get('X-Feedback-Url');
const feedbackToken = response.headers.get('X-Feedback-Token');
// 3. Submit feedback using the JWT token (no API key needed - safe for client-side)
await fetch(feedbackUrl, {
method: 'POST',
headers: {
'Authorization': `Bearer ${feedbackToken}`,
'Content-Type': 'application/json'
},
body: JSON.stringify({
score: 0.8,
weight: 1.0,
qualitative_feedback: 'Great response!'
})
});
```
**Example: Using API Key (Server-Side)**
```javascript theme={null}
// After receiving a response and collecting user feedback
const response = await fetch(`https://gw.datawizz.app/${projectId}/feedback/${inferenceLogId}`, {
method: 'POST',
headers: {
'Authorization': `Bearer ${apiKey}`,
'Content-Type': 'application/json'
},
body: JSON.stringify({
score: 0.9,
weight: 1.0,
signal_source: 'user',
signal_type: 'explicit',
qualitative_feedback: 'Great response, very helpful!'
})
});
```
Here's an example of submitting feedback with an improved response:
```javascript theme={null}
// User provides a correction
const response = await fetch(`https://gw.datawizz.app/${projectId}/feedback/${inferenceLogId}`, {
method: 'POST',
headers: {
'Authorization': `Bearer ${apiKey}`,
'Content-Type': 'application/json'
},
body: JSON.stringify({
score: -0.5,
weight: 0.8,
signal_source: 'user',
signal_type: 'explicit',
qualitative_feedback: 'Response was inaccurate',
improvement: {
role: 'assistant',
content: 'Here is the corrected response...'
}
})
});
```
### Example: Mapping Traditional Thumbs Up/Down to Feedback Signals
If you're migrating from a simple thumbs up/down system, here's how to map those to feedback signals:
```javascript theme={null}
// Thumbs up
{
score: 1.0,
weight: 1.0,
signal_source: 'user',
signal_type: 'explicit',
signal_name: 'thumbs_up'
}
// Thumbs down
{
score: -1.0,
weight: 1.0,
signal_source: 'user',
signal_type: 'explicit',
signal_name: 'thumbs_down'
}
```
### Combining Multiple Feedback Sources
The power of feedback signals comes from combining multiple sources. For example:
```javascript theme={null}
// User gives thumbs up
await submitFeedback({
score: 1.0,
weight: 1.0,
signal_source: 'user',
signal_type: 'explicit',
signal_name: 'thumbs_up'
});
// Automated evaluator checks factual accuracy
await submitFeedback({
score: 0.7,
weight: 0.9,
signal_source: 'system',
signal_type: 'explicit',
signal_name: 'factual_accuracy',
metadata: {
evaluator: 'fact_checker_v1',
confidence: 0.85
}
});
// Implicit signal: user copied response to clipboard
await submitFeedback({
score: 0.6,
weight: 0.5,
signal_source: 'user',
signal_type: 'implicit',
signal_name: 'copied_to_clipboard'
});
```
## Best Practices
### Choose Appropriate Weights
Not all feedback signals are equally reliable. Consider:
* Explicit feedback from expert users: weight = 1.0
* Explicit feedback from regular users: weight = 0.8
* Automated evaluations: weight = 0.7-0.9 depending on confidence
* Implicit signals: weight = 0.3-0.6 depending on strength of correlation
### Use Descriptive Signal Names
Use clear, consistent signal names to enable better observability:
* Good: `user_thumbs_up`, `accuracy_score`, `response_copied`
* Bad: `signal1`, `feedback`, `good`
### Include Metadata for Context
Add relevant metadata to enable detailed analysis:
```javascript theme={null}
{
score: 0.8,
weight: 0.9,
signal_name: 'user_rating',
metadata: {
user_segment: 'premium',
session_length: '15m',
previous_interactions: 5,
ui_version: '2.1.0'
}
}
```
### Combine Automated and Human Feedback
For best results, use both automated evaluators and human feedback:
* Automated evaluations provide consistent, scalable assessment
* Human feedback captures nuanced quality aspects that automated systems might miss
* The weighted combination gives you the best of both worlds
View the complete API reference for submitting feedback signals
# Vercel AI SDK
Source: https://docs.datawizz.ai/logs/libraries/vercel-ai-sdk
Using Datawizz with Vercel AI SDK
The [Vercel AI SDK](https://sdk.vercel.ai) is a powerful framework for creating AI-powered applications, making it easy to build AI features with streaming, function-calling and other capabilities built in. You can easily integrate Datawizz with the Vercel AI SDK to create powerful AI applications.
The integration with Datawizz will use the AI SDK's OpenAI provider and Datawizz's OpenAI-compatible API, so for the most part, this will look a lot like using OpenAI with the Vercel AI SDK. You can see the reference for the [OpenAI provider](https://sdk.vercel.ai/providers/ai-sdk-providers/openai) for additional information.
## Connecting to Datawizz
Assuming you already have the rest of the SDK installed, you'll need to install the OpenAI provider for the Vercel AI SDK. You can do this by running the following command:
```bash pnpm theme={null}
pnpm add @ai-sdk/openai
```
```bash npm theme={null}
npm install @ai-sdk/openai
```
```bash yarn theme={null}
yarn add @ai-sdk/openai
```
Once you have the SDK installed, you can connect to Datawizz by creating a new OpenAI client with your Datawizz API key and base URL. Here's an example of how you might do this:
```javascript theme={null}
import { createOpenAI } from '@ai-sdk/openai';
const openai = createOpenAI({
apiKey: process.env.DW_API_KEY, // <- Your Datawizz API key
baseURL: process.env.DW_PROJECT_URL // <- Your Datawizz project URL
});
```
Finally, you can instanciate a model using the openai client and the Datawizz endpoint ID you wish to connect to:
```javascript theme={null}
const model = openai(process.env.DW_ENDPOINT_ID); // <- Your Datawizz endpoint ID
```
## Using Datawizz with the Vercel AI SDK
You can now use the model to generate text, stream outputs and perform any other action supported by the Vercel AI SDK. Here's an example of how you might generate text using the model:
```javascript theme={null}
const { text } = await generateText({
model: model,
prompt: 'Write a vegetarian lasagna recipe for 4 people.',
});
```
## Adding Metadata
Metadata is crucial for tracking the performance of your models and organizing your logs better - Datawizz creates tags from your metadata, and allows you to route requests and train models based on these tags. You can add metadata to your requests using the `experimental_providerMetadata` option in the Vercel AI SDK. Here's an example of how you might add metadata to your request:
```javascript theme={null}
const { text } = await generateText({
model: model,
prompt: 'Write a vegetarian lasagna recipe for 4 people.',
experimental_providerMetadata: {
openai: {
store: true,
metadata: {
type: "recipe",
language: "en",
tier: 'free'
}
}
},
});
```
# Metadata and Tagging
Source: https://docs.datawizz.ai/logs/metadata-and-tagging
Add and manage metadata and tags for logs
Metadata and tags are a key part of organizing your LLM data to improve model training and request routing. In the datawizz platform, every inference log has attached metadata and tags. These tags can be used to filtrt logs to train specific models, and to route requests to different models.
## Metadata
When sending chat completion requests, you can attach metadata to the request. This metadata will be stored with the log, and tags will be auto-generated based on it.
### Adding metadata - OpenAI Compatible API
When sending a chat completion request via the OpenAI compatible API, you can add metadata to the request. Simply add a `metadata` field to the request with the metadata you want to attach to the log.
```js openai_connection.js theme={null}
import OpenAI from "openai";
const openai = new OpenAI({
apiKey: "sk-*******************",
baseUrl: "https://gw.datawizz.app/____________/openai/v1"
});
const completion = await openai.chat.completions.create({
model: "*******************",
messages: [
{ role: "user", content: "Write a haiku about recursion in programming." },
],
metadata: {
"language": "en",
"task": "summarize",
"domain": "programming",
"subscription_tier": "pro"
}
});
```
```py openai_connection.py theme={null}
from openai import OpenAI
client = OpenAI(
api_key= "sk-*******************",
base_url= "https://gw.datawizz.app/____________/openai/v1"
)
response = client.chat.completions.create(
model="*******************",
messages=[
{ "role": "user", "content": "Write a haiku about recursion in programming." },
],
metadata= {
"language": "en",
"task": "summarize",
"domain": "programming",
"subscription_tier": "pro"
}
)
```
### Auto-generated tags
When you add metadata to a log, tags will be auto-generated based on the metadata. These tags can be used to filter logs in the dashboard, and to route requests to different models. Only the top-level keys in the metadata will be used as tags.
For instance, in the example above, the following tags will be auto-generated: `language:en`, `task:summarize`, `domain:programming`, `subscription_tier:pro`.
Additionaly, any metadata keys with null values will just be added as a tag without a value (e.g. `{"test": null}` will generate the tag `test`).
## When to send metadata
Generally speaking, you should send as much metadata as possible with your requests. This metadata can be used to filter logs for training specific models, and to route requests to different models based on the metadata.
When transitioning to smaller models, it is often advantageous to create hyper-speicalized models that are trained for specific tasks. In this case, metadata can be used to filter logs for training these specialized models & routing between them.
Some generally useful metadata to send with requests include:
* `task`: The task the request is for (e.g. summarization, translation, etc.) if your application uses multiple tasks. These should roughly corrospond to the different prompt templates you use.
* `language`: The language of the request content (if you support multiple languages).
* `domain`: The domain of the request content (e.g. programming, finance, etc.) if you support multiple domains.
* `subscription_tier`: The subscription tier of the user making the request (e.g. free, pro, enterprise) if you have different tiers of service - this can be useful for routing requests to different models based on the user's subscription level.
## Metadata to avoid
We generally recommend avoiding sending personally identifiable information (PII) in metadata. This includes things like names, email addresses, phone numbers, etc. This information can be sensitive and should be handled with care. If you need to store this information, we recommend storing it in a separate database and linking it to the log id.
## Adding tags through the dashboard
You can also add metadata to logs through the dashboard. To do so, open any log in the logs section and click the "+" button in the metadata section. Here you can add key-value pairs of metadata to the log. You can also right-click on any existing tag to remove it from the log. Note that editing tags via the UI does not affect the original metadata sent with the request, only the generated tags.
![Add Metadata]()
You can also select multiple logs and add metadata to all of them at once. This can be useful when you want to add metadata to a batch of logs at once.
## Adding tags programmatically (via API)
You can also add metadata to logs programmatically via the Datawizz API. To do so, you can use the `PATCH /log/:log_id` endpoint with the `metadata` or `metadata_tags` field set to the metadata or tags you want to add.
Note that when updating metadata via the API, the new metadata will replace the existing metadata on the log. If you want to add to the existing metadata, you should include all the existing metadata in the request.
Additionally, updating the metadata via the API does not affect the tags generated from the metadata. The tags are generated when the log is created and are not updated when the metadata is updated. If you want to update the tags, you should update the `metadata_tags` field.
View the API reference for the PATCH /log/:log\_id endpoint
# Recording Requests
Source: https://docs.datawizz.ai/logs/recording-requests
Learn how to record and manage your LLM requests with the Datawizz platform
# Recording Requests with Datawizz
Recording your LLM requests is the foundation for optimizing your AI usage. Datawizz automatically logs all requests routed through the platform, allowing you to analyze usage patterns, collect data for model training, and improve your AI applications over time.
## Getting Started with Request Logging
To start recording your LLM requests, you need to set up three components:
1. **Connect your LLM provider** - Link your existing provider accounts like OpenAI or Anthropic
2. **Create an endpoint** - Set up a routing mechanism for your requests
3. **Update your application code** - Direct your requests through Datawizz
### 1. Connect Your Provider
First, connect your existing LLM provider in your Datawizz project:
1. Navigate to the Providers section in your project dashboard
2. Click "+ Add Provider" and select your provider (e.g., OpenAI)
3. Enter your provider API key and other required credentials
4. Add the specific models you want to use (e.g., GPT-4o, Claude 3)
### 2. Create an Endpoint
Next, create an endpoint to route your requests:
1. Go to the Endpoints section and click "Create Endpoint"
2. Give your endpoint a descriptive name
3. Add at least one upstream by clicking "Add Upstream"
4. Select the provider and model to route to
5. Configure weights and routing conditions if using multiple models
### 3. Connect from Your Application
Finally, update your application code to route requests through Datawizz:
```python theme={null}
from openai import OpenAI
client = OpenAI(
api_key="sk-your_datawizz_api_key",
base_url="https://gw.datawizz.app/your_project_id/openai/v1",
)
response = client.chat.completions.create(
model="your_endpoint_id", # Use your Datawizz endpoint ID here
messages=[
{"role": "user", "content": "Your prompt here"}
]
)
```
```javascript theme={null}
import OpenAI from "openai";
const openai = new OpenAI({
apiKey: "sk-your_datawizz_api_key",
baseUrl: "https://gw.datawizz.app/your_project_id/openai/v1"
});
const completion = await openai.chat.completions.create({
model: "your_endpoint_id", // Use your Datawizz endpoint ID here
messages: [
{ role: "user", content: "Your prompt here" }
]
});
```
Once configured, all requests will automatically be logged in your Datawizz dashboard, capturing both the input prompts and model responses.
## Enriching Your Logs with Metadata
Adding metadata to your requests provides crucial context that can help with filtering, analysis, and training specialized models later on. Metadata tags make it easy to organize and segment your data.
Learn more about adding and managing metadata for your logs
To add metadata to your requests:
```javascript theme={null}
const completion = await openai.chat.completions.create({
model: "your_endpoint_id",
messages: [
{ role: "user", content: "Your prompt here" }
],
metadata: {
"user_id": "user_123",
"language": "en",
"task": "content_generation",
"domain": "marketing"
}
});
```
## Collecting User Feedback
User feedback is invaluable for improving your AI models. With Datawizz, you can collect structured feedback signals on AI responses to use for model evaluation and training. Feedback signals support multiple sources (users and automated evaluators), weighted importance, and rich contextual information.
Learn more about collecting and using feedback signals for your AI responses
You can collect feedback signals programmatically using the Datawizz API:
```javascript theme={null}
// After receiving a response and collecting user feedback
await fetch(`https://gw.datawizz.app/${projectId}/feedback/${logId}`, {
method: 'POST',
headers: {
'Authorization': `Bearer ${apiKey}`,
'Content-Type': 'application/json'
},
body: JSON.stringify({
score: 0.9, // -1 to 1 scale
weight: 1.0, // 0 to 1 importance
signal_source: 'user',
signal_type: 'explicit',
qualitative_feedback: 'Great response!'
})
});
```
## Best Practices for Logging LLM Requests
### Organize with Multiple Endpoints or Projects
Create separate endpoints or projects for:
* Different applications or use cases
* Development, staging, and production environments
* Different teams or departments
This separation makes analysis more meaningful and enables targeted optimizations for each use case.
### Enrich Your Data with Metadata
Include as much contextual metadata as possible with each request:
* User identifiers (anonymized if needed)
* Task categories (summarization, translation, etc.)
* Content domains (legal, medical, etc.)
* Session or conversation IDs
* Application-specific context
The richer your metadata, the more insights you can derive and the better your specialized models will be.
### Start Recording Early
Begin logging your LLM requests as early as possible in your development cycle:
* Collect diverse prompts and use cases
* Establish baselines for performance and cost
* Accumulate enough data for future model training
* Identify patterns in usage that can inform your strategy
Early data collection provides a foundation for continuous improvement.
### Implement Structured Prompts
Design your prompts with a consistent structure to make analysis easier:
* Use standard templates for similar tasks
* Include clear instructions in a consistent format
* Separate different parts of the prompt (context, question, examples)
Structured prompts make it easier to analyze performance patterns across similar requests.
### Monitor Regularly
Set up a regular review cadence to:
* Analyze usage patterns and costs
* Identify underperforming or overused models
* Spot opportunities for prompt optimization
* Flag potential issues with model responses
Regular monitoring helps you stay on top of your AI usage and make timely adjustments.
### Implement Cost Controls
Use Datawizz's logging capabilities to:
* Track token usage across different endpoints
* Identify expensive or inefficient prompts
* Set alerts for unusual usage patterns
* Implement rate limits for high-volume endpoints
Proactive monitoring can prevent unexpected cost overruns.
By following these best practices, you'll build a valuable dataset that not only provides insights into your current AI usage but also enables future optimizations and the potential to train specialized models tailored to your specific needs.
# Supported LLMs
Source: https://docs.datawizz.ai/logs/supported-llms
Comprehensive list of LLM providers and models supported by Datawizz
# Supported LLM Providers
Datawizz supports integration with several leading LLM providers, allowing you to route requests to multiple models, record these requests, and use the recorded logs to train smaller and more efficient models. Below is a comprehensive list of all supported providers and their available models.
## OpenAI
Datawizz supports the full range of OpenAI models, including GPT-4, GPT-3.5, and other specialized models.
**Key Features:**
* Full API compatibility with OpenAI's platform
* Access to all text completion and chat completion models
* Support for embeddings and other OpenAI capabilities
[Learn more about OpenAI models](https://platform.openai.com/docs/models)
## Anthropic
Anthropic's Claude models are known for their conversational capabilities and safety features.
**Key Features:**
* Access to all Claude model variants
* Support for Claude's context window capabilities
* Integration with Anthropic's API features
[Learn more about Anthropic Claude models](https://docs.anthropic.com/en/docs/about-claude/models/all-models)
## Google
Datawizz supports Google's Gemini models, offering high-performance AI capabilities.
**Key Features:**
* Access to all Gemini model variants
* Support for Google's AI model capabilities
* OpenAI-compatible interface
[Learn more about Google Gemini models](https://ai.google.dev/gemini-api/docs/models)
## Cerebras
Integration with Cerebras inference API gives you access to their specialized AI models.
**Key Features:**
* Access to Cerebras' high-performance models
* Support for Cerebras' specific capabilities and optimizations
* Specialized AI inference
[Learn more about Cerebras models](https://inference-docs.cerebras.ai/introduction)
## Datawizz Serverless
Datawizz Serverless is our native model deployment platform that supports all models fine-tuned through the Datawizz platform.
**Key Features:**
* No external API key required (built into the platform)
* Seamless deployment of fine-tuned models
* Support for all models trained on the Datawizz platform
* Cost-effective serverless infrastructure
## Fireworks AI
Datawizz integrates with Fireworks AI's model serving platform, known for high-throughput inference.
**Key Features:**
* Access to a wide range of open and proprietary models
* High-performance inference capabilities
* Support for Fireworks AI's specialized optimizations
[Learn more about Fireworks AI models](https://fireworks.ai/models)
## Together AI
Together AI offers a variety of open-source and proprietary models through their inference platform.
**Key Features:**
* Access to a wide catalog of AI models
* Support for high-performance AI inference
* Integration with Together AI's specialized features
[Learn more about Together AI models](https://www.together.ai/models)
## Adding Custom Providers
If you need to integrate with a provider not listed above, please contact our support team for guidance on custom integrations.
***
By supporting multiple LLM providers, Datawizz enables you to:
1. **Compare performance** across different models and providers
2. **Optimize costs** by routing to the most cost-effective model for each use case
3. **Enhance reliability** through fallback mechanisms
4. **Leverage strengths** of different models for specific tasks
For more information on configuring providers and routing strategies, see our [Routing documentation](/endpoints/routing).
# Datasets
Source: https://docs.datawizz.ai/models/datasets
Create, import, and manage datasets for training and evaluation
Datasets are the core building blocks for [model training](/models/training-models) and evaluation in Datawizz. A dataset is a collection of items, where each item has an input (prompt or message history) and one or more outputs (model responses). Outputs can carry tags (e.g. chosen, rejected) and evaluator scores, which feed directly into SFT, DPO, KTO, and GRPO training workflows.
## Create a dataset
From the project sidebar, open **Datasets** and create a new dataset with a name and optional description. New datasets start empty; use the toolbar actions below to add data.
## Dataset list
The list view shows all datasets in the project: name, creation time, item count, size, and a **Manage** link to open each dataset. The table is sortable by column.
## Inside a dataset
After opening a dataset, the page has a top toolbar and three tabs: **Items**, **Stats**, and **Jobs**.
The dataset detail page has three tabs:
* **Items** — A paginated table of every sample. Each row shows ID, created time, tags, input (user prompt), and output (model response). You can edit input/output in place, manage tags, and use the row menu for actions like adding outputs or deleting an item.
* **Stats** — A token-length histogram and summary stats (min, P95, P99, max) to help you set `max_seq_length` for training.
* **Jobs** — Lists all background jobs (uploads, imports, exports, annotations, regenerations) with their status (Done / Failed / In Progress), progress, and timestamps. Expand a row for logs and details; completed export files are downloadable here.
***
## Toolbar actions
### Import Logs
**Import Logs** pulls inference requests from your Datawizz endpoints into the dataset. You can filter by endpoint, model, feedback signals, tags, and more; use the Query tab for advanced filters or the Views tab for saved presets. Selected logs are imported as dataset items (input + output).
### Import File
**Import File** uploads data from CSV, JSONL, or Parquet. You pick a format (Text, Full, Data, or OpenAI) and optionally a system message; preview is available before confirming. Supported formats:
| Format | Description |
| ------ | ------------------------------------------------------------------ |
| Text | Input and output are plain strings; auto-wrapped into chat format. |
| Full | Input is a JSON array of messages; output is a single message. |
| Data | Raw columns; you map fields in the next step. |
| OpenAI | OpenAI-style messages array. |
### Create Split
**Create Split** divides the dataset into two parts (e.g. 80/20 for train/eval). You set the split ratio and name the new datasets; they can then be used in training and evaluation configs.
### Annotate Dataset
**Annotate Dataset** runs evaluators on each item’s outputs and attaches scores. You choose evaluators, set weights, and start the job; scores show on the Items table and can drive KTO training or evaluation.
### Regen Outputs
**Regen Outputs** generates new assistant responses for all items with a chosen model. New outputs are added alongside existing ones. You configure provider/model (temperature, max tokens, etc.) and at least one output tag. Optional: a regeneration instruction and context mode (instruction + original input, or also include a selected output column for improvement-style prompts).
### Export Dataset
**Export Dataset** starts a background job that exports the dataset to JSONL or CSV. When the job finishes, the file appears in the **Jobs** tab and can be downloaded from there. The video below shows starting an export and finding the completed file in the Jobs tab.
### Edit
**Edit** lets you change the dataset name and description. If the dataset has evaluator scores, you can set Scorer Weights and Normalize to build a composite score.
***
## Outputs, tags, and scores
Each item can hold multiple outputs - essential for preference-based training:
* SFT - Single output per item. No tags needed, or choose your output based on a specific tag.
* DPO - Two outputs per item tagged chosen and rejected.
* KTO - One output tag plus a score threshold. Scores come from importing logs that already have feedback (collected via the feedback API), or by running Annotate Dataset.
* GRPO - Scored outputs for group-relative optimization.
Manage tags per-item via the + button in the Tags column. When multiple output tags exist, the Items table shows one column per tag.
See [Model Training](/models/training-models) for full trainer configuration.
***
## Adding data from Logs
You can also add data directly from the Logs page. Select rows and click Add to Dataset. Choose how to handle improvements: ignore them, use as the primary output, or add as a separate output (useful for building DPO pairs).
***
## Import from other tools
Export from Langfuse and use Import File - Datawizz supports the native Langfuse format. See the [video walkthrough](https://www.youtube.com/watch?v=hs-fVDNmL-c).
Use the Datawizz export script to produce a CSV in OpenAI format, then import. See the [LangSmith guide](/integrations/langsmith).
Use this [Colab notebook](https://colab.research.google.com/drive/1lafP6s72-V1ntc7SZhgMNQSvFKH5TJK8?usp=sharing) to export into a Datawizz-compatible CSV.
Convert your data to CSV or JSONL with input and output columns (plain text or chat-format JSON) and use Import File.
# Model Deployment
Source: https://docs.datawizz.ai/models/model-deployment
Deploy your models to various providers.
Once you train SLMs on the Datawizz platform, you must first deploy them to a provider to start evaluating them & conntecting them to endpoints. You can always download the model weights and manually deploy them to any inference provider. Datawizz makes it simple to deploy to various providers with automated "one clicke" deployments.
We are actively working on adding more providers. If you have a specific provider in mind, please reach out to us and we will prioritize it.
Before deploying to any third party providers, please check their pricing as deployments often incurr charges. The models will be deployed to your account, and you will be responsible for any charges incurred.
## Datawizz Serverless
Datawizz offers a serverless deployment option for your models. This is the simplest way to deploy your models and get started with inference. Datawizz includes a free tier for serverless deployments and pay as you go pricing.
To deploy your model to Datawizz serverless, go to any trained model, click "Deploy Model" and select "Datawizz Serverless". Your model will now be available for inference in the Datawizz serverless environment. You can use it in evaluations or connect it to endpoints.
![Deploy to Datawizz Serverless]()
Datawizz serverless has a longer first cold-start time, so your first request to the model may take longer. This delay should not repeat as long as your model is in use at least once every 24 hours.
## Fireworks AI
Datawizz supports one-click deployment to the Fireworks AI platform. Fireworks deployments provide dedicated inference instances for your models, with powerful GPU support and automated scaling. Fireworks also provides convinient scale to zero for less active models, so you only pay for what you use.
Learn more about Fireworks AI Inference [here](https://docs.fireworks.ai/guides/ondemand-deployments).
To deploy your models to Fireworks:
1. Add a Fireworks AI type providers in the providers section of the Datawizz platform. You will need to provide your Fireworks API key and account ID (your account ID will be at the dropdown on the top right corner of the Fireworks dashboard).
2. One the account is configured, go to any trained model, click "Deploy Model" and select the Fireworks AI provider you configured (you can connect to multiple Fireworks accounts and select one to deploy to).
3. You will be prompted to select an accelerator type (GPU) and scaling parameters. Consult the [fireworks docs](https://docs.fireworks.ai/guides/ondemand-deployments#deployment-options) for more details on these options.
4. Deployment may take a few minutes - you'll see the deployment status in the model details page. Once the deployment is complete, the status will be "READY" and you can start using the model for inference.
5. You can also use the Fireworks AI dashboard to manage your deployments, including scaling and stopping them.
![Fireworks AI Deployment Options]()
## Together AI
Datawizz supports one-click deployment to the Together AI platform. Together AI supports both serverless and dedicated inference instances for your models, with powerful GPU support and automated scaling. Datawizz currently supports serverless deployments to Together AI, with imminent plans to support dedicated deployments as well.
Learn more about Together AI Inference [here](https://docs.together.ai/docs/deployment-options).
To deploy your models to Together AI:
1. Add a Together AI type providers in the providers section of the Datawizz platform. You will need to provide your Together API key.
2. One the account is configured, go to any trained model, click "Deploy Model" and select the Together AI provider you configured (you can connect to multiple Together accounts and select one to deploy to), and click "Deploy".
3. Deployment may take a few minutes - you'll see the deployment status in the model details page. Once the deployment is complete, the status will be "READY" and you can start using the model for inference.
4. You can also use the Together AI dashboard to manage your deployments, including scaling and stopping them.
![Together AI Deployment Options]()
Togetrher AI Serverless deployments don't support all model architectures trainable on Datawizz. Please consult their documentation for a [list of supported architectures](https://docs.together.ai/docs/lora-inference).
# Model Training
Source: https://docs.datawizz.ai/models/training-models
Train your first specialize language model with Datawizz.
This tutorial will take you through the steps to train & deploy your first specialized language model with Datawizz.
This tutorial assumes you already have a project set up on Datawizz with some logs saved which we will train the new model from.
# Training a Model
To train a model, head to the models section of your project and click "+ New Model":
![Add Model]()
In the new model page, follow the on-screen instructions to configure the base model, training data and training parameters:
![Configure Model]()
Once you create the model, the training process will start. You can monitor training progress and logs in the model page. Note that model training can take a few minutes to a few hours, depending on the size of the training configurations you use.
![Model Training]()
# Model Deployment
Once the model is trained, you can deploy it on Datawizz to start testing it and using it in your applications. To deploy a model, click the "Deploy Model" button in the model page and select Datawizz Serveless as the provider:
![Deploy Model]()
Datawizz Serveless is a serverless platform that allows you to deploy and run your models without worrying about infrastructure. You can just start calling your model and pay per token without having to worry about scaling or managing servers.
Once the model is deployed, you can use it in an endpoint, or select it for manual comparison in the providers section of your project:
![Compare Models]()
Datawizz also supports additional providers for model deployment. See the docs here: [Model Deployment](/models/model-deployment).
# null
Source: https://docs.datawizz.ai/models/training-parameters
| Title | Description | Data Type | Default |
| ---------------------- | ------------------------------------------------------------------------- | -------------------------------------------- | ----------------------------------------------------------------- |
| Maximal input size | Defines the maximum number of tokens in each training sample. | Commonly 128 to 4096 (model-dependent). | Calculated automatically according to the size of the input logs. |
| Validation split ratio | The portion of the training data reserved for validation during training. | A number between 0.01 and 0.9. | 0.2 |
| Max samples | The number of data points from the dataset used for training. | Any positive integer up to the dataset size. | Full dataset. |
| Training epochs | Indicates the number of complete passes through the training dataset. | Integer between 1 and 100. | 10 |
| Learning rate | Specifies the step size for updating model weights during training. | Typically between 1e-5 and 1e-1. | 2e-4 |
| Save total limit | The total number of checkpoints to save during training. | Typically between 0 and 10. | 3 |
| Evaluation Steps | The number of steps between each evaluation during training. | Typically between 1 and 5000. | 20 |
# Reranker Models
Source: https://docs.datawizz.ai/models/training-reranker
API request/response for reranker inference and data format for reranker training
Reranker models score query–document pairs for relevance. Datawizz supports **inference** via the same chat-completion-style API (with automatic transformation to the backend rerank endpoint) and **training** via the Swift worker with a specific data format.
## Inference API
When you call a deployed reranker model through Datawizz (e.g. Datawizz Serverless or your endpoint), you use the **chat completions** request format. The gateway transforms it to the backend `/v1/rerank` format and converts the rerank response back to a chat completion so logging and evaluation work as with other models.
### Request (chat completion format)
Send a standard chat completion request with:
* **`model`**: Your deployed reranker model name.
* **`messages`**:
* **One `system` message**: its `content` is the **query** (string).
* **One `user` message**: the **documents** to score. Either:
* A **string** (single document), or
* An **array** of content parts: `{"type": "text", "text": "Document content"}` for each document.
**Recommendation (Qwen3-Reranker):** For best performance with Qwen3-Reranker models, use the native ``, ``, and `` tag format in your messages. The simplified format (plain query in system, plain document in user) also works, but the tagged format aligns with how the model was trained and may yield better relevance scores.
The default Instruct for Qwen3-Reranker is: `"Given a web search query, retrieve relevant passages that answer the query"`. You can customize this for your specific task.
```json Qwen3 format (recommended) theme={null}
{
"model": "your-reranker-model",
"messages": [
{ "role": "system", "content": ": Given a web search query, retrieve relevant passages that answer the query\n: What is Python async programming?" },
{ "role": "user", "content": ": asyncio is Python's asynchronous I/O framework, supporting coroutines and event loops." }
]
}
```
```json Qwen3 format (multiple documents) theme={null}
{
"model": "your-reranker-model",
"messages": [
{ "role": "system", "content": ": Given a web search query, retrieve relevant passages that answer the query\n: What is Python async programming?" },
{
"role": "user",
"content": [
{ "type": "text", "text": ": asyncio is Python's asynchronous I/O framework." },
{ "type": "text", "text": ": Java is an object-oriented programming language." },
{ "type": "text", "text": ": Python asyncio provides event loops and coroutines for concurrent I/O." }
]
}
]
}
```
```json Simplified format (also works) theme={null}
{
"model": "your-reranker-model",
"messages": [
{ "role": "system", "content": "What is Python async programming?" },
{ "role": "user", "content": "asyncio is Python's asynchronous I/O framework, supporting coroutines and event loops." }
]
}
```
Requirements:
* Exactly one `system` message with non-empty string `content` (the query, or `` + `` tags for Qwen3 format).
* Exactly one `user` message whose `content` is either a non-empty string or an array of `{ "type": "text", "text": "..." }` with at least one document (or `` tags for Qwen3 format).
### Response (chat completion format)
The response is returned as a normal chat completion. The assistant **message content** is a **JSON string** of the rerank results (index and score only; document text is not echoed):
```json theme={null}
{
"id": "...",
"object": "chat.completion",
"choices": [
{
"index": 0,
"message": {
"role": "assistant",
"content": "[{\"index\":0,\"relevance_score\":0.95},{\"index\":1,\"relevance_score\":0.12},{\"index\":2,\"relevance_score\":0.88}]"
},
"finish_reason": "stop"
}
],
"usage": {
"prompt_tokens": 150,
"completion_tokens": 1,
"total_tokens": 151
}
}
```
* **`choices[0].message.content`**: JSON string of an array of `{ "index": number, "relevance_score": number }`, ordered by the backend (typically by relevance, highest first).
* **`index`**: Original document index (0-based) in the request.
* **`relevance_score`**: Relevance score in \[0, 1] (higher = more relevant).
* **`usage`**: Token counts.
Parse the string to get the array:
```javascript theme={null}
const content = response.choices[0].message.content;
const results = JSON.parse(content);
// results = [{ index: 0, relevance_score: 0.95 }, { index: 1, relevance_score: 0.12 }, ...]
```
### Feedback / Improvement Signal
To send feedback on reranker results, use the feedback endpoint:
```
POST /{project_uid}/{endpoint_uid}/feedback/{inference_log_id}
```
The response headers `X-Feedback-Url` and `X-Feedback-Token` provide the URL and JWT token for submitting feedback. See [Feedback Signals](/logs/feedback) for full details.
For rerankers, provide an **improvement** object with the corrected relevance scores in the same structure as the model's output:
**Single result:**
```json theme={null}
{
"role": "assistant",
"content": { "index": 0, "relevance_score": 0.95 }
}
```
Replace `0.95` with your ideal relevance score (0–1).
**Batch results (multiple documents):**
```json theme={null}
{
"role": "assistant",
"content": [
{ "index": 0, "relevance_score": 0.95 },
{ "index": 1, "relevance_score": 0.1 },
{ "index": 2, "relevance_score": 0.8 }
]
}
```
Replace each `relevance_score` with your corrected score for that document.
**Pointwise (binary) feedback:** Use `1` for relevant ("Yes") and `0` for not relevant ("No"):
```json theme={null}
{
"role": "assistant",
"content": [
{ "index": 0, "relevance_score": 1 },
{ "index": 1, "relevance_score": 0 }
]
}
```
The `index` corresponds to the original document index in the request. This feedback can be used for supervised fine-tuning or to track model performance over time.
**Example API call:**
```bash theme={null}
curl -X POST "https://gw.datawizz.app/{project_uid}/{endpoint_uid}/feedback/{inference_log_id}" \
-H "Authorization: Bearer ${FEEDBACK_TOKEN}" \
-H "Content-Type: application/json" \
-d '{
"score": 0,
"weight": 1.0,
"improvement": {
"role": "assistant",
"content": [
{ "index": 0, "relevance_score": 1 },
{ "index": 5, "relevance_score": 0 },
{ "index": 99, "relevance_score": 1 }
]
}
}'
```
`score` and `weight` are **required** fields but not meaningful for reranker feedback—just use the defaults (`score: 0`, `weight: 1.0`). The important part is the `improvement` object with your corrected relevance scores. See [Feedback Signals API](/api-reference/endpoint/post_feedback) for full schema details.
* **Partial feedback is fine:** You don't need to provide feedback for every document—just the ones you want to correct.
* **Incremental or batch:** You can send all corrections in one request, or send multiple requests over time. We handle aggregation automatically.
***
## Training Data Format
Reranker training in Datawizz uses the **Swift** worker. You provide data in the **Datawizz format** (one sample per row in your dataset or in a JSON/JSONL file). Each sample has `input` (query + document) and `output` (relevance label).
### Per-sample structure
Each training example must have:
| Field | Type | Description |
| -------- | ------ | --------------------------------------------------------------------------------------------------------------------- |
| `input` | array | Exactly two messages: **system** = query (or ``+`` tags), **user** = document (or `` tag). |
| `output` | object | Single message with **role** `"assistant"` and **content** = relevance label (see below). |
| `id` | string | Optional; preserved for tracking. |
**Query and document:**
* **`input[0]`**: `{ "role": "system", "content": "..." }` — the search query, or `` + `` tags for Qwen3 format.
* **`input[1]`**: `{ "role": "user", "content": "..." }` — the document to be judged, or `` tag for Qwen3 format.
**Recommendation (Qwen3-Reranker):** Use the same `//` tag format in training data as at inference. This aligns training with how you call the model and may improve fine-tuning results. The simplified format (plain query in system, plain document in user) also works.
Use the default Instruct `"Given a web search query, retrieve relevant passages that answer the query"` or customize it for your specific task.
**Relevance label (`output.content`):**
* **Pointwise (default, `loss_type: "pointwise_reranker"`)**: Binary 0/1.
* **Text**: `"Yes"`, `"True"`, `"1"`, `"relevant"` → 1; anything else (e.g. `"No"`, `"False"`, `"0"`) → 0.
* **Numeric**: Value in \[0, 1]; ≥ 0.5 → 1, \< 0.5 → 0.
* **MSE (`loss_type: "mse"`)**: Continuous score in \[0, 1].
* **Text**: `"Yes"` / `"True"` / `"1"` / `"relevant"` → 1.0; `"No"` / `"False"` / `"0"` → 0.0; otherwise 0.5.
* **Numeric**: Clamped to \[0, 1] and used as the target.
### Sample training file (Datawizz format)
Datawizz supports importing reranker training data **only in JSONL format** (one JSON object per line). A JSON array is not supported for import. Each line must be a single object with the structure below.
**Example JSONL — Qwen3 format (recommended):**
```jsonl theme={null}
{"id":"sample-001","input":[{"role":"system","content":": Given a web search query, retrieve relevant passages that answer the query\n: What is Python async programming?"},{"role":"user","content":": asyncio is Python's asynchronous I/O framework, supporting coroutines and event loops."}],"output":{"role":"assistant","content":"Yes"}}
{"id":"sample-002","input":[{"role":"system","content":": Given a web search query, retrieve relevant passages that answer the query\n: What is Python async programming?"},{"role":"user","content":": Java is an object-oriented programming language."}],"output":{"role":"assistant","content":"No"}}
{"id":"sample-003","input":[{"role":"system","content":": Given a web search query, retrieve relevant passages that answer the query\n: What is machine learning?"},{"role":"user","content":": Machine learning is a subset of AI that enables systems to learn from data."}],"output":{"role":"assistant","content":0.9}}
```
**Example JSONL — Simplified format (also works):**
```jsonl theme={null}
{"id":"sample-001","input":[{"role":"system","content":"What is Python async programming?"},{"role":"user","content":"asyncio is Python's asynchronous I/O framework, supporting coroutines and event loops."}],"output":{"role":"assistant","content":"Yes"}}
{"id":"sample-002","input":[{"role":"system","content":"What is Python async programming?"},{"role":"user","content":"Java is an object-oriented programming language."}],"output":{"role":"assistant","content":"No"}}
```
For **MSE** training, use continuous scores in `output.content`, e.g. `0.9`, `0.2`, or strings that map to 0.0/0.5/1.0 as above.
### Training configuration (Swift reranker)
When creating a reranker model in the app, you choose:
* **Loss type**: `pointwise_reranker` (binary 0/1) or `mse` (regression in \[0, 1]).
* **Base models**: e.g. Qwen3-Reranker-0.6B/4B, bge-reranker-v2-m3, gte-reranker-modernbert-base.
Other Swift parameters (epochs, batch size, max sequence length, LoRA, etc.) are documented in [Training Parameters](/models/training-parameters). Rerankers typically use a larger **max sequence length** (e.g. 8192) to include full documents.
***
## Summary
| Use case | Request / data | Response / label |
| ------------- | ------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------- |
| **Inference** | Chat completion: `messages` = system (query or ``/`` tags) + user (documents or `` tags) | `choices[0].message.content` = JSON string of `[{index, relevance_score}, ...]` |
| **Feedback** | POST `/{project_uid}/{endpoint_uid}/feedback/{inference_log_id}` with `improvement` containing corrected scores | — |
| **Training** | Datawizz format: `input` = \[system=query/tags, user=document/tags], `output` = assistant content = "Yes"/"No" or 0/1 or float \[0,1] | N/A (labels define relevance for training) |
Ensure inference requests use one system and one user message; ensure training data uses exactly two `input` messages (system then user) and one `output` with the appropriate label type for your chosen `loss_type`.
# Platform Overview
Source: https://docs.datawizz.ai/platform-overview
A brief overview of the Datawizz platform
This document will cover the key components of the Datawizz platform, and how they interact with each other.
## Workspaces
Everything you do happens inside a workspace. By default, we create a personal workspace for you when you sign up. You can create additional workspaces, and invite other users to collaborate with you. Workspace users all currently have full access to make changes to the workspace and all associated projects. We are working on creating more granular roles and permissions, but for now be careful about who you invite to your workspace.
Billing is done at the workspace level. You can have multiple workspaces, each with their own billing details and limits.
## Projects
Projects are the main way to organize your data. All other entities, including API keys, are managed at the project level - and project data is isolated from other projects.
Projects are a great way to manage different environment or different applications alltogether.
## Providers & Public Models
Providers are different platforms that provide AI models - like OpenAI, Anthropic or AWS. Every project should be connected to at least one provider to be able to use AI models.
Datawizz itself is a provider as well - every project created has a built in "Datawizz Serverless" provider which lets you use dozens of opensource AI models, as well as any models you train using the Datawizz platform.
Under every provider you create in the system, you can configure the models you want to use -- for instance, if you configure an OpenAI provider, you can add the GPT-4o model to it, or with the Datawizz provider you can add a "Llama-3.2-405B" model.
You can see more information about supported providers & public models [here](#todo).
## Endpoints
Endpoints are the integration medium between your app and the AI models. When your app calls an AI model - it'll call an endpoint. The endpoint defines the rules for routing the request to different models, and any policies that should be applied to the request (like caching or input screening).
Endpoints can be simple - routing to a single model, or complex - load balancing between multiple models with a failover mechanism.
You can learn more about endpoints and routing [here](#todo).
## Inference Logs
Every time you make an AI inference, datawizz logs the input and output for you. You can see & manage these logs in the datawizz dashboard. Datawizz goes beyond the basic logs, letting you tag them, provide metadata and add structured feedback signals from multiple sources to improve future training.
You can learn more about logs & data management [here](#todo).
## Models
Models in the Datawizz platform refer to the specialized models you train on your data. With datawizz, you can train & deploy specialized models for your use case, and use them in your projects.
You can learn more about training models [here](#todo).
# Build Custom Plugins
Source: https://docs.datawizz.ai/plugins/build-custom-plugins
Extend and customize request/response processing with custom logic
This guide covers everything you need to build custom plugins for the Datawizz AI Gateway. Plugins are HTTP endpoints that receive requests from the gateway, process them according to your business logic, and return responses that control the request flow.
A plugin is simply an HTTP endpoint that:
1. **Receives** a POST request with the current request state
2. **Processes** the data according to your custom logic
3. **Returns** a JSON response indicating whether to allow, reject, or modify the request
Plugins can be built in any language or framework that can expose an HTTP endpoint including **Node.js** (Express, Fastify, Hono), **Python** (Flask, FastAPI, Django) or any other web framework... You can run them on serverless platforms (AWS Lambda, Cloudflare Workers, Vercel Functions) or traditional servers.
## Request Schema
When a plugin is invoked, the gateway sends a POST request with the following JSON payload:
### Input Structure
```typescript theme={null}
messages: Array<{
role: "user" | "assistant" | "system" | "tool";
content: string | Array;
// ... other message fields
}>;
requestBody: {
model?: string;
temperature?: number;
max_tokens?: number;
stream?: boolean;
// ... all original request body fields
response?: any; // Only present in RESPONSE phase - contains the LLM's response
};
requestHeaders: Record;
metadata: Record;
configs: any; // Custom configuration you defined for this plugin instance
requestId: string; // Unique ID for this request (UUID)
```
### Field Descriptions
| Field | Type | Description |
| ---------------- | ---------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `messages` | `Array` | The messages array being processed. In REQUEST phase, this is what will be sent to the LLM. In RESPONSE phase, this includes the LLM's response as the last message. |
| `requestBody` | `object` | The complete request body from the client, including all LLM parameters (model, temperature, etc.) |
| `requestHeaders` | `object` | All HTTP headers from the client's request |
| `metadata` | `object` | Custom metadata from the request. Can include prompt info, model provider details, etc. |
| `configs` | `any` | The custom configuration object you defined for this plugin instance in the endpoint configuration |
| `requestId` | `string` | A unique UUID identifying this specific request, useful for logging and correlation |
### Phase-Specific Considerations
**REQUEST Phase:**
* `messages` contains the messages that will be sent to the LLM (after prompt template application)
* `requestBody` contains the original client request parameters
* `metadata` may include prompt information if a prompt template was used
**RESPONSE Phase:**
* `messages` contains all input messages PLUS the LLM's response message as the last element
* `requestBody.response` contains the full LLM response object
* `metadata` includes additional fields like `modelProvider` and `promptInfo`
**LOG Phase:**
* Same as RESPONSE phase
* Intended for non-blocking analytics, monitoring, or logging operations
***
## Response Schema
Your plugin must return a JSON response matching this schema:
### Output Structure
```typescript theme={null}
{
reject?: boolean; // Default: false
rejectReason?: string; // Opptional error message that will be returned to the client if rejecting
dontRetry?: boolean; // Default: false, prevents retries with other LLMs (only in RESPONSE phase)
messages?: Array; // Optional: modified messages array
debug?: Array; // Optional: debug messages for logging
}
```
### Field Descriptions
| Field | Type | Required | Default | Description |
| -------------- | ---------------- | -------- | ------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `reject` | `boolean` | No | `false` | Set to `true` to reject the request/response and stop pipeline execution |
| `rejectReason` | `string` | No | - | Human-readable explanation for why the request was rejected. Returned to the client. |
| `dontRetry` | `boolean` | No | `false` | Prevents retries with other LLMs (only in RESPONSE phase). If `true`, the gateway will not attempt to call alternative models if the response is rejected. |
| `messages` | `Array` | No | - | Modified messages array. If provided, these messages replace the input messages for subsequent plugins. |
| `debug` | `Array` | No | - | Debug messages that will be included in the gateway logs for troubleshooting |
### Response Validation
The gateway validates your plugin's response against the schema above using Zod. If validation fails:
* The plugin execution is marked as failed
* The error is logged with the validation error message
* The request continues (fail-open behavior) with the original, unmodified messages
***
## Implementation Examples
### Example 1: Simple Content Filter (Guardrail)
This plugin rejects requests containing prohibited words.
```python filter.py theme={null}
from fastapi import FastAPI
from pydantic import BaseModel
from typing import List, Dict, Any, Optional
app = FastAPI()
class Message(BaseModel):
role: str
content: Any
class PluginRequest(BaseModel):
messages: List[Message]
requestBody: Dict[str, Any]
requestHeaders: Dict[str, str]
metadata: Dict[str, Any]
configs: Any
requestId: str
class PluginResponse(BaseModel):
reject: bool = False
rejectReason: Optional[str] = None
messages: Optional[List[Message]] = None
debug: Optional[List[str]] = None
PROHIBITED_WORDS = ["badword1", "badword2", "inappropriate"]
@app.post("/filter")
async def content_filter(request: PluginRequest) -> PluginResponse:
debug_messages = []
# Check all messages for prohibited content
for msg in request.messages:
if isinstance(msg.content, str):
content_lower = msg.content.lower()
for word in PROHIBITED_WORDS:
if word in content_lower:
debug_messages.append(f"Found prohibited word: {word}")
return PluginResponse(
reject=True,
rejectReason=f"Content contains prohibited term: {word}",
debug=debug_messages
)
debug_messages.append("Content passed filter")
return PluginResponse(
reject=False,
debug=debug_messages
)
```
```javascript filter.js theme={null}
const express = require('express');
const app = express();
app.use(express.json());
const PROHIBITED_WORDS = ['badword1', 'badword2', 'inappropriate'];
app.post('/filter', (req, res) => {
const { messages, requestId } = req.body;
const debug = [];
// Check all messages for prohibited content
for (const msg of messages) {
if (typeof msg.content === 'string') {
const contentLower = msg.content.toLowerCase();
for (const word of PROHIBITED_WORDS) {
if (contentLower.includes(word)) {
debug.push(`Found prohibited word: ${word}`);
return res.json({
reject: true,
rejectReason: `Content contains prohibited term: ${word}`,
debug
});
}
}
}
}
debug.push('Content passed filter');
res.json({
reject: false,
debug
});
});
app.listen(3000, () => console.log('Plugin listening on port 3000'));
```
***
### Example 2: PII Redaction (Modification)
This plugin detects and redacts personally identifiable information.
```python redact_pii.py theme={null}
import re
from fastapi import FastAPI
from pydantic import BaseModel
from typing import List, Dict, Any, Optional
app = FastAPI()
class Message(BaseModel):
role: str
content: Any
class Config:
extra = "allow" # Allow additional fields
class PluginRequest(BaseModel):
messages: List[Message]
requestBody: Dict[str, Any]
requestHeaders: Dict[str, str]
metadata: Dict[str, Any]
configs: Any
requestId: str
class PluginResponse(BaseModel):
reject: bool = False
rejectReason: Optional[str] = None
messages: Optional[List[Dict[str, Any]]] = None
debug: Optional[List[str]] = None
# Simple regex patterns (use more sophisticated detection in production)
EMAIL_PATTERN = re.compile(r'\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Z|a-z]{2,}\b')
PHONE_PATTERN = re.compile(r'\b\d{3}[-.]?\d{3}[-.]?\d{4}\b')
SSN_PATTERN = re.compile(r'\b\d{3}-\d{2}-\d{4}\b')
def redact_pii(text: str) -> tuple[str, list[str]]:
"""Redact PII from text and return redacted text + list of redactions"""
redactions = []
if EMAIL_PATTERN.search(text):
text = EMAIL_PATTERN.sub('[EMAIL_REDACTED]', text)
redactions.append('email')
if PHONE_PATTERN.search(text):
text = PHONE_PATTERN.sub('[PHONE_REDACTED]', text)
redactions.append('phone')
if SSN_PATTERN.search(text):
text = SSN_PATTERN.sub('[SSN_REDACTED]', text)
redactions.append('ssn')
return text, redactions
@app.post("/redact-pii")
async def pii_redaction(request: PluginRequest) -> PluginResponse:
debug_messages = []
modified_messages = []
any_redactions = False
# Process each message
for msg in request.messages:
msg_dict = msg.dict()
if isinstance(msg.content, str):
redacted_content, redactions = redact_pii(msg.content)
if redactions:
any_redactions = True
msg_dict['content'] = redacted_content
debug_messages.append(f"Redacted {', '.join(redactions)} from {msg.role} message")
modified_messages.append(msg_dict)
if any_redactions:
debug_messages.append(f"Total messages processed: {len(modified_messages)}")
return PluginResponse(
reject=False,
messages=modified_messages,
debug=debug_messages
)
else:
debug_messages.append("No PII detected")
return PluginResponse(
reject=False,
debug=debug_messages
)
```
```javascript redact_pii.js theme={null}
const express = require('express');
const app = express();
app.use(express.json());
// Simple regex patterns (use more sophisticated detection in production)
const EMAIL_PATTERN = /\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}\b/g;
const PHONE_PATTERN = /\b\d{3}[-.]?\d{3}[-.]?\d{4}\b/g;
const SSN_PATTERN = /\b\d{3}-\d{2}-\d{4}\b/g;
function redactPii(text) {
const redactions = [];
let redactedText = text;
if (EMAIL_PATTERN.test(text)) {
redactedText = redactedText.replace(EMAIL_PATTERN, '[EMAIL_REDACTED]');
redactions.push('email');
}
if (PHONE_PATTERN.test(text)) {
redactedText = redactedText.replace(PHONE_PATTERN, '[PHONE_REDACTED]');
redactions.push('phone');
}
if (SSN_PATTERN.test(text)) {
redactedText = redactedText.replace(SSN_PATTERN, '[SSN_REDACTED]');
redactions.push('ssn');
}
return { redactedText, redactions };
}
app.post('/redact-pii', (req, res) => {
const { messages } = req.body;
const debug = [];
const modifiedMessages = [];
let anyRedactions = false;
// Process each message
for (const msg of messages) {
const msgCopy = { ...msg };
if (typeof msg.content === 'string') {
const { redactedText, redactions } = redactPii(msg.content);
if (redactions.length > 0) {
anyRedactions = true;
msgCopy.content = redactedText;
debug.push(`Redacted ${redactions.join(', ')} from ${msg.role} message`);
}
}
modifiedMessages.push(msgCopy);
}
if (anyRedactions) {
debug.push(`Total messages processed: ${modifiedMessages.length}`);
res.json({
reject: false,
messages: modifiedMessages,
debug
});
} else {
debug.push('No PII detected');
res.json({
reject: false,
debug
});
}
});
app.listen(3000, () => console.log('PII redaction plugin listening on port 3000'));
```
***
### Example 3: Response Quality Check (Guardrail)
This plugin validates that LLM responses meet quality standards.
```python quality_check.py theme={null}
from fastapi import FastAPI
from pydantic import BaseModel
from typing import List, Dict, Any, Optional
import re
app = FastAPI()
class Message(BaseModel):
role: str
content: Any
class Config:
extra = "allow"
class PluginRequest(BaseModel):
messages: List[Message]
requestBody: Dict[str, Any]
requestHeaders: Dict[str, str]
metadata: Dict[str, Any]
configs: Any
requestId: str
class PluginResponse(BaseModel):
reject: bool = False
rejectReason: Optional[str] = None
messages: Optional[List[Dict[str, Any]]] = None
debug: Optional[List[str]] = None
@app.post("/quality-check")
async def quality_check(request: PluginRequest) -> PluginResponse:
debug_messages = []
# Get the assistant's response (last message in RESPONSE phase)
if not request.messages or request.messages[-1].role != 'assistant':
debug_messages.append('No assistant response found')
return PluginResponse(reject=False, debug=debug_messages)
last_message = request.messages[-1]
response_text = (
last_message.content
if isinstance(last_message.content, str)
else str(last_message.content)
)
# Quality checks
min_length = request.configs.get('minLength', 10) if request.configs else 10
max_length = request.configs.get('maxLength', 10000) if request.configs else 10000
require_sources = request.configs.get('requireSources', False) if request.configs else False
# Check minimum length
if len(response_text) < min_length:
debug_messages.append(f"Response too short: {len(response_text)} < {min_length}")
return PluginResponse(
reject=True,
rejectReason='Response does not meet minimum length requirements',
debug=debug_messages
)
# Check maximum length
if len(response_text) > max_length:
debug_messages.append(f"Response too long: {len(response_text)} > {max_length}")
return PluginResponse(
reject=True,
rejectReason='Response exceeds maximum length',
debug=debug_messages
)
# Check for sources if required
if require_sources:
has_sources = bool(re.search(r'\[source\]|\[citation\]|source:|reference:', response_text, re.IGNORECASE))
if not has_sources:
debug_messages.append('Response missing required sources')
return PluginResponse(
reject=True,
rejectReason='Response must include source citations',
debug=debug_messages
)
debug_messages.append('Response passed all quality checks')
return PluginResponse(reject=False, debug=debug_messages)
```
```javascript quality_check.js theme={null}
const express = require('express');
const app = express();
app.use(express.json());
app.post('/quality-check', (req, res) => {
const { messages, requestBody, configs } = req.body;
const debug = [];
// Get the assistant's response (last message in RESPONSE phase)
const lastMessage = messages[messages.length - 1];
if (!lastMessage || lastMessage.role !== 'assistant') {
debug.push('No assistant response found');
return res.json({ reject: false, debug });
}
const responseText = typeof lastMessage.content === 'string'
? lastMessage.content
: JSON.stringify(lastMessage.content);
// Quality checks
const minLength = configs?.minLength || 10;
const maxLength = configs?.maxLength || 10000;
const requireSources = configs?.requireSources || false;
// Check minimum length
if (responseText.length < minLength) {
debug.push(`Response too short: ${responseText.length} < ${minLength}`);
return res.json({
reject: true,
rejectReason: 'Response does not meet minimum length requirements',
debug
});
}
// Check maximum length
if (responseText.length > maxLength) {
debug.push(`Response too long: ${responseText.length} > ${maxLength}`);
return res.json({
reject: true,
rejectReason: 'Response exceeds maximum length',
debug
});
}
// Check for sources if required
if (requireSources) {
const hasSources = /\[source\]|\[citation\]|source:|reference:/i.test(responseText);
if (!hasSources) {
debug.push('Response missing required sources');
return res.json({
reject: true,
rejectReason: 'Response must include source citations',
debug
});
}
}
debug.push('Response passed all quality checks');
res.json({
reject: false,
debug
});
});
app.listen(3000, () => console.log('Quality check plugin listening on port 3000'));
```
***
### Example 4: Context Enhancement (Modification)
This plugin adds additional context to user requests.
```python add_context.py theme={null}
from fastapi import FastAPI
from pydantic import BaseModel
from typing import List, Dict, Any, Optional
from datetime import datetime
app = FastAPI()
class Message(BaseModel):
role: str
content: Any
class Config:
extra = "allow"
class PluginRequest(BaseModel):
messages: List[Message]
requestBody: Dict[str, Any]
requestHeaders: Dict[str, str]
metadata: Dict[str, Any]
configs: Any
requestId: str
class PluginResponse(BaseModel):
reject: bool = False
rejectReason: Optional[str] = None
messages: Optional[List[Dict[str, Any]]] = None
debug: Optional[List[str]] = None
@app.post("/add-context")
async def add_context(request: PluginRequest) -> PluginResponse:
debug_messages = []
modified_messages = []
# Add system message with context at the beginning
context_message = {
"role": "system",
"content": f"""Current date and time: {datetime.utcnow().isoformat()}Z
Additional context from configs:
- Organization: {request.configs.get('organization', 'N/A')}
- User tier: {request.configs.get('userTier', 'standard')}
- Special instructions: {request.configs.get('instructions', 'None')}
Please use this context when formulating your response."""
}
modified_messages.append(context_message)
debug_messages.append("Added system context message")
# Add all original messages
for msg in request.messages:
modified_messages.append(msg.dict())
debug_messages.append(f"Total messages: {len(modified_messages)}")
return PluginResponse(
reject=False,
messages=modified_messages,
debug=debug_messages
)
```
```javascript add_context.js theme={null}
const express = require('express');
const app = express();
app.use(express.json());
app.post('/add-context', (req, res) => {
const { messages, configs } = req.body;
const debug = [];
const modifiedMessages = [];
// Add system message with context at the beginning
const contextMessage = {
role: 'system',
content: `Current date and time: ${new Date().toISOString()}
Additional context from configs:
- Organization: ${configs?.organization || 'N/A'}
- User tier: ${configs?.userTier || 'standard'}
- Special instructions: ${configs?.instructions || 'None'}
Please use this context when formulating your response.`
};
modifiedMessages.push(contextMessage);
debug.push('Added system context message');
// Add all original messages
for (const msg of messages) {
modifiedMessages.push({ ...msg });
}
debug.push(`Total messages: ${modifiedMessages.length}`);
res.json({
reject: false,
messages: modifiedMessages,
debug
});
});
app.listen(3000, () => console.log('Context enhancement plugin listening on port 3000'));
```
***
## Testing Your Plugin
### Local Testing
Before deploying your plugin, test it locally using curl or any HTTP client:
```bash theme={null}
curl -X POST http://localhost:3000/your-endpoint \
-H "Content-Type: application/json" \
-d '{
"messages": [
{
"role": "user",
"content": "Test message with badword1"
}
],
"requestBody": {
"model": "gpt-4",
"temperature": 0.7
},
"requestHeaders": {
"content-type": "application/json"
},
"metadata": {},
"configs": {
"customSetting": "value"
},
"requestId": "test-request-123"
}'
```
Expected response:
```json theme={null}
{
"reject": true,
"rejectReason": "Content contains prohibited term: badword1",
"debug": ["Found prohibited word: badword1"]
}
```
### Integration Testing
Once your plugin is deployed and configured in the gateway:
1. Monitor the gateway logs for plugin execution messages
2. Check the `debug` array in your responses - these will appear in gateway logs
3. Use the gateway's inference logs to see plugin execution times and results
4. Test timeout and retry behavior by simulating slow responses or failures
***
## Best Practices
### Performance
**Response Time:**
* Aim for plugin response times under 100ms for REQUEST phase
* RESPONSE phase plugins can be slightly slower (under 500ms)
* Use LOG phase for any operations that can be async (analytics, slow external APIs)
**Optimization Tips:**
* Cache frequently used data (e.g., ML models, lookup tables)
* Use connection pooling for database queries
* Implement circuit breakers for external API calls
* Consider using async/parallel processing internally
### Error Handling
**Fail Gracefully:**
```python error_handling.py theme={null}
@app.post("/my-plugin")
async def my_plugin(request: PluginRequest) -> PluginResponse:
try:
# Your plugin logic
result = process_messages(request.messages)
return PluginResponse(
reject=False,
messages=result,
debug=["Processing successful"]
)
except Exception as e:
# Log the error internally
print(f"Plugin error: {str(e)}")
# Return a safe response (fail-open)
# Gateway will continue with original messages
return PluginResponse(
reject=False,
debug=[f"Plugin error (non-critical): {str(e)}"]
)
```
```javascript error_handling.js theme={null}
app.post('/my-plugin', (req, res) => {
try {
// Your plugin logic
const result = processMessages(req.body.messages);
res.json({
reject: false,
messages: result,
debug: ['Processing successful']
});
} catch (e) {
// Log the error internally
console.error(`Plugin error: ${e.message}`);
// Return a safe response (fail-open)
// Gateway will continue with original messages
res.json({
reject: false,
debug: [`Plugin error (non-critical): ${e.message}`]
});
}
});
```
**Timeout Handling:**
* Set appropriate timeout values in your plugin configuration
* Ensure your plugin respects the timeout and fails fast
* Use async operations to avoid blocking
### Security
As your plugins will be exposed over the internet, it's crucial to implement robust security measures. We recommend checking for a secret header, which you can configure in the Datawizz dashboard when setting up your plugin endpoint.
**Authentication:**
```python authentication.py theme={null}
from fastapi import FastAPI, Header, HTTPException
@app.post("/secure-plugin")
async def secure_plugin(
request: PluginRequest,
authorization: str = Header(None)
):
# Validate authorization header
if not authorization or not authorization.startswith("Bearer "):
raise HTTPException(status_code=401, detail="Unauthorized")
token = authorization.replace("Bearer ", "")
if token != EXPECTED_TOKEN:
raise HTTPException(status_code=403, detail="Forbidden")
# Process request...
```
```javascript authentication.js theme={null}
app.post('/secure-plugin', (req, res) => {
const authorization = req.headers.authorization;
// Validate authorization header
if (!authorization || !authorization.startsWith('Bearer ')) {
return res.status(401).json({ error: 'Unauthorized' });
}
const token = authorization.replace('Bearer ', '');
if (token !== EXPECTED_TOKEN) {
return res.status(403).json({ error: 'Forbidden' });
}
// Process request...
});
```
Configure the authorization header in your plugin settings in the gateway dashboard.
**Input Validation:**
* Always validate the structure of incoming requests
* Sanitize any data before using it in queries or external API calls
* Be cautious with the `configs` field - validate expected types
**Secrets Management:**
* Never hardcode API keys or secrets in your plugin code
* Use environment variables or secret management services
* Rotate credentials regularly
### Observability
**Logging:**
```python logging_example.py theme={null}
import logging
logger = logging.getLogger(__name__)
@app.post("/my-plugin")
async def my_plugin(request: PluginRequest) -> PluginResponse:
logger.info(f"Processing request {request.requestId}")
debug_messages = []
# Add useful debug info
debug_messages.append(f"Message count: {len(request.messages)}")
debug_messages.append(f"Model: {request.requestBody.get('model', 'unknown')}")
# Your logic here...
logger.info(f"Completed request {request.requestId}")
return PluginResponse(
reject=False,
debug=debug_messages
)
```
```javascript logging_example.js theme={null}
app.post('/my-plugin', (req, res) => {
const { requestId, messages, requestBody } = req.body;
console.log(`Processing request ${requestId}`);
const debug = [];
// Add useful debug info
debug.push(`Message count: ${messages.length}`);
debug.push(`Model: ${requestBody.model || 'unknown'}`);
// Your logic here...
console.log(`Completed request ${requestId}`);
res.json({
reject: false,
debug
});
});
```
**Metrics:**
* Track plugin execution time
* Monitor rejection rates
* Alert on error rates
* Track resource usage (CPU, memory, network)
### Message Handling
**Preserve Message Structure:**
```python preserve_structure.py theme={null}
# Good: Preserve all message fields
msg_dict = msg.dict() # or msg.model_dump() in Pydantic v2
msg_dict['content'] = modified_content
modified_messages.append(msg_dict)
# Bad: Only copying some fields
modified_messages.append({
'role': msg.role,
'content': modified_content
# Missing other fields!
})
```
```javascript preserve_structure.js theme={null}
// Good: Preserve all message fields
const msgCopy = { ...msg };
msgCopy.content = modifiedContent;
modifiedMessages.push(msgCopy);
// Bad: Only copying some fields
modifiedMessages.push({
role: msg.role,
content: modifiedContent
// Missing other fields!
});
```
**Handle Different Content Types:**
```python content_types.py theme={null}
def process_content(content: Any) -> Any:
if isinstance(content, str):
# Simple string content
return process_string(content)
elif isinstance(content, list):
# Multimodal content (text + images, etc.)
return process_multimodal(content)
else:
# Unknown format, return as-is
return content
```
```javascript content_types.js theme={null}
function processContent(content) {
if (typeof content === 'string') {
// Simple string content
return processString(content);
} else if (Array.isArray(content)) {
// Multimodal content (text + images, etc.)
return processMultimodal(content);
} else {
// Unknown format, return as-is
return content;
}
}
```
***
## Troubleshooting
### Common Issues
**"Plugin response validation failed"**
* Check that your response matches the expected schema exactly
* Ensure `reject` is a boolean, not a string
* Ensure `debug` is an array of strings, not a single string
* Verify `messages` is an array if provided
**"Plugin timeout after Xms"**
* Your plugin is taking longer than the configured timeout
* Optimize your plugin's processing time
* Increase the timeout value in plugin configuration
* Move slow operations to LOG phase if possible
**"Plugin returned status 500"**
* Your plugin threw an unhandled exception
* Check your plugin's logs for error details
* Implement proper error handling
**Messages not being modified**
* Ensure you're returning a `messages` array in your response
* Verify the array contains valid message objects
* Check that you're not accidentally returning the original messages reference
**Plugin rejections not working**
* Ensure `reject: true` is present in response
* Include a `rejectReason` string
* Check that the response is valid JSON
## Next Steps
Now that you understand how to build plugins:
1. Implement a simple plugin following one of the examples above
2. Test it locally with sample requests
3. Deploy it to your hosting platform
4. Configure it in the Datawizz dashboard
5. Monitor its performance and iterate
For more information, see:
* [Plugin Usage Guide](./plugins.md) - How to use and configure plugins
# Plugins
Source: https://docs.datawizz.ai/plugins/plugins
Extend and customize request/response processing with custom logic
Plugins are custom code modules that can intercept and process requests and responses flowing through your AI Gateway. They provide a powerful extension mechanism for implementing custom logic such as content filtering, data transformation, compliance checks, and request enrichment.
Plugins execute at specific points in the request lifecycle (called "phases") and can either allow requests/responses to proceed, reject them with custom reasons, or modify the message contents before passing them along to the next stage.
***
## Guardrail Plugins
Guardrail plugins act as safety gates, screening requests and responses to ensure they meet your security, compliance, and quality standards before proceeding.
### How Guardrails Work
A guardrail plugin evaluates incoming requests (before they reach the LLM) or outgoing responses (before they're returned to the client) and can:
* **Allow**: Let the request/response continue through the pipeline by returning `{ reject: false }`
* **Reject**: Block the request/response by returning `{ reject: true, rejectReason: "..." }`
When a guardrail rejects a request or response:
* The rejection reason is logged for audit purposes
* The client receives an error response with the rejection reason
* The request never reaches the LLM (for REQUEST phase) or the response never reaches the client (for RESPONSE phase)
* No further plugins in the pipeline are executed
### Use Cases
Common guardrail scenarios include:
* **Content Safety**: Screen for prohibited content, PII, or sensitive information
* **Compliance**: Enforce regulatory requirements (GDPR, HIPAA, etc.)
* **Quality Control**: Reject low-quality or malformed responses
* **Policy Enforcement**: Validate requests against business rules
* **Toxicity Detection**: Filter out harmful or offensive content
* **Budget Protection**: Reject requests that would exceed token limits
### Example Response
```json theme={null}
{
"reject": true,
"rejectReason": "Content contains PII that violates data privacy policy"
}
```
***
## Modification Plugins
Modification plugins transform the messages array as it flows through the pipeline, enabling sophisticated data manipulation and enrichment.
### How Modifications Work
A modification plugin receives the current messages array and can return a modified version:
```json theme={null}
{
"reject": false,
"messages": [
{ "role": "user", "content": "Modified user message" },
{ "role": "assistant", "content": "Modified assistant response" }
]
}
```
When a plugin returns modified messages:
* The modified messages become the input for the next plugin in the pipeline
* Multiple plugins can chain modifications together
* If no `messages` array is returned, the original messages are passed unchanged
### Use Cases
Common modification scenarios include:
* **Content Anonymization**: Redact or pseudonymize PII in messages
* **Prompt Enhancement**: Add context, instructions, or examples to user prompts
* **Translation**: Convert messages to different languages
* **Format Normalization**: Standardize message structure or content
* **Response Enrichment**: Add metadata, citations, or structured data to responses
* **Token Optimization**: Compress or summarize long messages
### Example Response
```json theme={null}
{
"reject": false,
"messages": [
{
"role": "system",
"content": "You are a helpful assistant. Always cite your sources."
},
{
"role": "user",
"content": "What is the capital of France?"
}
]
}
```
***
## Using Plugins
Plugins are connected to endpoints through the endpoint configuration. You can attach multiple plugins to an endpoint, each executing at a specific phase of the request lifecycle.
### Plugin Sources
Datawizz provides two ways to use plugins:
1. **Public Plugins**: Pre-built plugins provided by Datawizz for common use cases (content filtering, PII detection, etc.) - see [Public Plugins](/public-plugins/index) for the full catalog.
2. **Custom Plugins**: Build your own plugins to implement custom business logic - see [Building Custom Plugins](./build-custom-plugins) for details.
### Connecting Plugins in the Dashboard
![Endpoint Plugins]()
To connect a plugin to an endpoint:
1. Navigate to your endpoint configuration in the dashboard
2. Select the "Plugins" tab
3. Click "Add Plugin"
4. Choose between public plugins or project (custom) plugins
5. Configure the plugin settings (see Endpoint Plugin Configuration below)
6. Save your endpoint configuration
***
## Endpoint Plugin Configuration
Each plugin attached to an endpoint can be configured with the following settings:
### Core Settings
| Field | Type | Default | Description |
| ---------- | -------------------------------- | -------- | ----------------------------------------------------- |
| `phase` | `REQUEST` \| `RESPONSE` \| `LOG` | Required | Execution phase for the plugin |
| `priority` | `number` | `0` | Execution order (lower number = earlier execution) |
| `async` | `boolean` | `false` | Whether plugin executes asynchronously (non-blocking) |
| `configs` | `object` | `{}` | Custom configuration passed to your plugin |
### Reliability Settings
| Field | Type | Default | Description |
| ---------------- | -------- | ------- | ---------------------------------------------------- |
| `retry_on_error` | `number` | `0` | Number of retry attempts on plugin failure (max: 10) |
| `timeout` | `number` | `60` | Maximum execution time in seconds before timeout |
### Phase Options
* **REQUEST**: Executes after prompt templates are applied, before the request is sent to the LLM
* **RESPONSE**: Executes after receiving the LLM response, before returning to client
* **LOG**: Executes during the logging phase (non-blocking, for analytics/monitoring)
> **Note:** REQUEST phase plugins receive the messages array *after* prompt templates have been applied and variables have been interpolated. This means plugins work with the final messages that will be sent to the LLM.
***
## Execution Order
Understanding plugin execution order is crucial for building reliable and predictable pipelines.
### Priority-Based Ordering
Plugins execute in ascending priority order (lower numbers first):
```
Priority 0 → Priority 10 → Priority 20 → Priority 100
```
Plugins with the same priority will execute in a non-deterministic order.
### Async Execution (Clasps)
Plugins marked as `async: true` can execute in parallel with other async plugins that have consecutive priorities. This is called "clasping."
This is a great way to reduce latency by executing multiple plugins in parallel.
> **Note:** Clasping only allows for rejections - async plugins cannot modify the messages array or return additional data (we do this because in async execution, if multiple plugins modify the messages array, it's not clear which one should take precedence or how to merge the results).
**Clasping Rules:**
* Consecutive async plugins at the same or adjacent priorities are grouped into a "clasp"
* All plugins in a clasp execute simultaneously
* The gateway waits for all plugins in a clasp to complete before proceeding
* A non-async plugin breaks the clasp, forcing sequential execution
### Example 1: Mixed Async and Sync Plugins
| Plugin | Priority | Async | Execution |
| -------------- | -------- | ----- | ----------- |
| Content Filter | 0 | ✅ | **Clasp 1** |
| PII Detection | 1 | ✅ | **Clasp 1** |
| Add Context | 2 | ❌ | Sequential |
| Logging A | 3 | ❌ | Sequential |
| Logging B | 4 | ✅ | **Clasp 2** |
```mermaid theme={null}
flowchart LR
Start([Request Start]) --> C1
subgraph C1["Clasp 1 (Parallel)"]
CF[Content Filter]
CF ~~~ PII
PII[PII Detection]
end
C1 --> AC[Add Context]
AC --> LA[Logging A]
LA --> C2
subgraph C2["Clasp 2"]
LB[Logging B]
end
C2 --> End([Request to LLM])
```
### Example 2: All Async Plugins
| Plugin | Priority | Async | Execution |
| ---------------- | -------- | ----- | ----------- |
| Toxicity Check | 0 | ✅ | **Clasp 1** |
| Compliance Check | 0 | ✅ | **Clasp 1** |
| Budget Check | 0 | ✅ | **Clasp 1** |
| PII Redaction | 0 | ✅ | **Clasp 1** |
```mermaid theme={null}
flowchart LR
Start([Request Start]) --> C1
subgraph C1["Clasp 1 (Parallel)"]
TC[Toxicity Check]
TC ~~~ CC
CC[Compliance Check]
CC ~~~ BC
BC[Budget Check]
BC ~~~ PR
PR[PII Redaction]
end
C1 --> End([Request to LLM])
```
### Example 3: Request and Response Flow
| Phase | Plugin | Priority | Async |
| -------- | --------------- | -------- | ----- |
| REQUEST | Auth Check | 0 | ❌ |
| REQUEST | Content Filter | 10 | ✅ |
| REQUEST | PII Detection | 10 | ✅ |
| REQUEST | Add Context | 20 | ❌ |
| RESPONSE | Quality Check | 0 | ❌ |
| RESPONSE | Format Response | 10 | ✅ |
| RESPONSE | Log Metrics | 10 | ✅ |
```mermaid theme={null}
flowchart LR
Start([Client Request]) --> AC[Auth Check]
AC --> C1
subgraph C1["Request Clasp"]
CF[Content Filter]
CF ~~~ PII
PII[PII Detection]
end
C1 --> ACX[Add Context]
ACX --> LLM([LLM Processing])
LLM --> QC[Quality Check]
QC --> C2
subgraph C2["Response Clasp"]
FR[Format Response]
FR ~~~ LM
LM[Log Metrics]
end
C2 --> End([Client Response])
```
### Rejection Behavior
When a plugin rejects a request or response:
* Execution stops immediately
* No subsequent plugins in the pipeline are executed
* The rejection reason is logged
* An error response is returned to the client
***
## Considerations & Best Practices
### Latency Impact
Every plugin adds latency to your request pipeline. Consider these strategies to minimize impact:
**✅ Do:**
* Use `async: true` for independent guardrails (toxicity, PII, compliance checks)
* Set appropriate `timeout` values based on plugin complexity
* Use LOG phase for non-critical operations like analytics
* Place critical guardrails early in the pipeline (low priority numbers)
* Cache plugin results when possible within your plugin implementation
**❌ Avoid:**
* Long-running synchronous plugins in the REQUEST phase
* Multiple sequential modification plugins that could be combined
* Plugins that make unnecessary external API calls
* Setting timeout values higher than needed
### Stability & Reliability
Plugin failures can impact your entire pipeline. Follow these best practices:
**Error Handling:**
* Set `retry_on_error` for plugins that call external services (recommended: 2-3 retries)
* Use appropriate `timeout` values to prevent hanging requests
* Currently, plugin failures default to "pass-through" behavior (the request continues)
* Monitor plugin logs to track failure rates
**Testing:**
* Test plugins in isolation before adding to production pipelines
* Verify plugin behavior with various message types and edge cases
* Load test your plugin endpoints to ensure they can handle production traffic
* Monitor plugin execution times and adjust timeouts accordingly
**Observability:**
* Plugins automatically log execution time, rejection status, and debug messages
* Use the `debug` array in plugin responses to add custom debugging information
* Monitor plugin logs for errors, timeouts, and unexpected behavior
### Security
**Authentication:**
* Use the `headers` field to add authentication tokens to plugin requests
* Store sensitive credentials securely, not in configuration files
* Rotate API keys and tokens regularly
**Network Security:**
* Use HTTPS for all plugin URLs
* Validate SSL certificates
* Implement rate limiting on your plugin endpoints
* Consider using VPNs or private networks for sensitive plugins
### Performance Optimization
**Async Execution:**
* Mark independent guardrails as `async: true` to enable parallel execution
* Group related async plugins with the same priority for clasping
* Be cautious with async modification plugins - they cannot modify each other's results
**Priority Optimization:**
* Place rejection-likely guardrails early (low priority) to fail fast
* Place expensive operations later (high priority) so they only run if needed
* Use LOG phase for non-blocking operations like metrics and analytics
**Resource Management:**
* Monitor plugin execution times and optimize slow plugins
* Set realistic timeouts to prevent resource exhaustion
* Consider implementing circuit breakers in your plugin infrastructure
### Message Modification
**Chaining:**
* Modifications chain sequentially - each plugin receives the previous plugin's output
* Ensure modification plugins return valid message arrays
* Test modification chains thoroughly to avoid unexpected transformations
**Validation:**
* Validate modified messages match the expected schema
* Ensure role types remain valid (user, assistant, system)
* Preserve important metadata when modifying messages
### Cost Considerations
* Plugin execution incurs additional compute and network costs
* External API calls in plugins may have their own costs
* Balance the value of plugin functionality against added costs
* Consider caching strategies to reduce redundant plugin executions
# Block Substrings
Source: https://docs.datawizz.ai/public-plugins/block-substrings
Blocks or redacts requests containing specific substrings. Useful for filtering unwanted content or enforcing content policies.
The Block Substrings plugin scans messages for specific substrings and can either reject requests containing those substrings or automatically redact them. This is useful for enforcing content policies, filtering profanity, blocking competitor mentions, or preventing sensitive keywords.
## Features
* **Simple Substring Matching**: Configure a list of substrings to detect
* **Case Sensitivity Control**: Choose case-sensitive or case-insensitive matching
* **Selective Scanning**: Scan only the latest message or all messages
* **Reject or Redact**: Either block requests or automatically redact matched content
* **Configurable Redaction**: Use static text or character repetition for redaction
* **Multiple Matches**: Handles multiple occurrences of substrings
## Use Cases
* Block profanity or offensive language
* Prevent competitor brand mentions
* Enforce content policies by blocking specific keywords
* Redact internal code names or project names
* Filter out specific phrases or terms
* Implement custom content moderation rules
## Configuration Options
### `substrings` (required)
Array of strings to search for in messages. Each string is matched as a substring (not whole word).
### `rejectMessage` (string)
Custom message to return when a substring is found and `redactMatches` is false.
Default: `"Request contains forbidden content"`
### `scanAllMessages` (boolean)
* `true`: Scans all messages in the array
* `false`: Only scans the latest message
* Default: `false`
### `caseSensitive` (boolean)
* `true`: Substring matching is case-sensitive
* `false`: Substring matching is case-insensitive
* Default: `false`
### `redactMatches` (boolean)
* `true`: Redacts matched substrings instead of rejecting
* `false`: Rejects the request if a match is found
* Default: `false`
### `redactionText` (string)
Text to replace redacted substrings with.
Default: `[REDACTED]`
### `redactCharacters` (string)
If set, replaces each match with this character repeated to match the length of the original text. Overrides `redactionText` if both are set.
## Example Configurations
### Block Profanity
```json theme={null}
{
"substrings": ["badword1", "badword2", "offensive"],
"rejectMessage": "Your message contains inappropriate language",
"scanAllMessages": true,
"caseSensitive": false,
"redactMatches": false
}
```
### Block Competitor Mentions
```json theme={null}
{
"substrings": ["CompetitorA", "CompetitorB", "rival-product"],
"rejectMessage": "Please refrain from mentioning competitor products",
"scanAllMessages": false,
"caseSensitive": false,
"redactMatches": false
}
```
### Redact Internal Code Names
```json theme={null}
{
"substrings": ["ProjectX", "SecretFeature", "Alpha-2024"],
"scanAllMessages": true,
"caseSensitive": true,
"redactMatches": true,
"redactionText": "[INTERNAL]"
}
```
### Redact with Character Masking
```json theme={null}
{
"substrings": ["password", "secret", "confidential"],
"scanAllMessages": true,
"caseSensitive": false,
"redactMatches": true,
"redactCharacters": "*"
}
```
### Case-Sensitive Exact Matches
```json theme={null}
{
"substrings": ["API_KEY", "SECRET_TOKEN"],
"rejectMessage": "Please do not include API keys or tokens",
"scanAllMessages": true,
"caseSensitive": true,
"redactMatches": false
}
```
## Response Behavior
### When Rejecting (redactMatches: false)
* Returns `reject: true` with the configured rejection message
* First matched substring triggers the rejection
* No messages are modified
* Debug logs show total number of matches
### When Redacting (redactMatches: true)
* Returns modified messages array with redacted content
* All occurrences of all substrings are redacted
* Original request continues through the pipeline
* Debug logs indicate how many matches were redacted
## Matching Behavior
### Substring vs Whole Word
The plugin performs substring matching, not whole-word matching:
* Substring `"test"` matches: "test", "testing", "contest", "latest"
* To match whole words only, include surrounding spaces in your substring (e.g., `" test "`)
### Case Sensitivity
When `caseSensitive: false`:
* Substring `"Test"` matches: "test", "TEST", "Test", "tEsT"
When `caseSensitive: true`:
* Substring `"Test"` only matches: "Test"
* Does not match: "test", "TEST", "tEsT"
### Multiple Occurrences
All occurrences of each substring are detected and processed:
* Input: `"test test test"`
* Substring: `"test"`
* Result: All three occurrences are found and redacted
## Example Scenarios
### Scenario 1: Block Profanity
**Configuration**:
```json theme={null}
{
"substrings": ["badword"],
"caseSensitive": false,
"redactMatches": false,
"rejectMessage": "Inappropriate language detected"
}
```
**Input**: `"This is a badword in the text"`
**Result**: Request rejected with message "Inappropriate language detected"
### Scenario 2: Redact Competitor Names
**Configuration**:
```json theme={null}
{
"substrings": ["CompetitorX"],
"caseSensitive": true,
"redactMatches": true,
"redactionText": "[COMPETITOR]"
}
```
**Input**: `"Have you tried CompetitorX product?"`
**Output**: `"Have you tried [COMPETITOR] product?"`
### Scenario 3: Mask Sensitive Terms
**Configuration**:
```json theme={null}
{
"substrings": ["password"],
"caseSensitive": false,
"redactMatches": true,
"redactCharacters": "*"
}
```
**Input**: `"My password is secret123"`
**Output**: `"My ******** is secret123"`
## Debug Information
The plugin provides detailed debug information including:
* Number of messages scanned
* Case sensitivity mode
* Total matches found
* Number of unique matches (when redacting)
* Action taken (blocked or redacted)
## Technical Notes
* Empty substrings are ignored
* Handles both string and array content formats
* Preserves non-text content (images, audio, etc.)
* When case-insensitive, original case is preserved in redaction
* Duplicate replacements are deduplicated to improve performance
* All occurrences of each substring are found, not just the first
***
## Configuration Schema
```json theme={null}
{
"type": "object",
"title": "Block Substrings Configuration",
"$schema": "http://json-schema.org/draft-07/schema#",
"required": [
"substrings"
],
"properties": {
"substrings": {
"type": "array",
"items": {
"type": "string"
},
"title": "Substrings to Block",
"minItems": 1,
"description": "List of substrings to scan for and block/redact"
},
"caseSensitive": {
"type": "boolean",
"title": "Case Sensitive",
"default": false,
"description": "If true, substring matching is case sensitive"
},
"redactMatches": {
"type": "boolean",
"title": "Redact Matches",
"default": false,
"description": "If true, redacts matched substrings instead of rejecting the request"
},
"redactionText": {
"type": "string",
"title": "Redaction Text",
"default": "[REDACTED]",
"description": "Text to replace redacted substrings with"
},
"rejectMessage": {
"type": "string",
"title": "Reject Message",
"default": "Request contains forbidden content",
"description": "Custom message to return when a substring is found (when not redacting)"
},
"scanAllMessages": {
"type": "boolean",
"title": "Scan All Messages",
"default": false,
"description": "If true, scans all messages. If false, only scans the latest message"
},
"redactCharacters": {
"type": "string",
"title": "Redact Characters",
"description": "If set, replaces each match with this character repeated to match the original length. Overrides redactionText"
}
},
"description": "Configuration for the Block Substrings plugin"
}
```
# Document Input Processing
Source: https://docs.datawizz.ai/public-plugins/document-input-processing
Convert PDFs, Word, PowerPoint, and Excel documents into clean markdown for LLM processing. Uses Microsoft's MarkItDown with optional LLM-powered image descriptions.
Process a wide variety of document types, including PDFs, office documents (Word, Excel, PowerPoint) and more, by converting them into markdown format that LLMs can understand.
## Overview
Many LLMs are limited to processing text, and in some cases images and audio. This plugin extends native LLM capabilities by pre-processing documents into content that LLMs can understand. The plugin automatically detects document URLs or data URIs in your messages and converts them into clean, structured markdown text.
The plugin uses Microsoft's [`MarkItDown`](https://github.com/microsoft/markitdown) library to convert documents into markdown, which is then sent to the LLM for processing.
## Features
* **Multiple Formats**: Supports PDF, Word (DOCX, DOC), PowerPoint (PPTX, PPT), and Excel (XLSX, XLS) documents
* **Markdown Conversion**: Converts documents into clean, structured markdown with preserved formatting
* Headings, lists, and tables are maintained
* Proper paragraph breaks and text emphasis
* All readable text content extracted
* **Image Handling**: Optional LLM-powered image descriptions for documents containing images or complex layouts
* **Flexible Input**: Download from URLs or process base64-encoded data URIs
* **Automatic Format Detection**: Intelligent content-type and file format detection
## Installation
1. Add the plugin to your Datawizz endpoint configuration
2. Set the endpoint URL to: `https://your-service-url/plugin/document`
3. Configure the Authorization header with your secret token:
* Header name: `Authorization`
* Header value: `Bearer YOUR_SECRET_TOKEN`
4. Optionally configure default settings (see Configuration below)
## Configuration
You can specify configurations to control how the document is processed:
| Parameter | Type | Description | Default |
| --------------------------- | ------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------ |
| `url` | string | The URL of the document to process | (required or data) |
| `data` | string | The base64 encoded content of the document. If provided, this will be used instead of the URL. Should be a data URI (`data:application/pdf;base64,...`) | None |
| `use_llm_image_description` | boolean | Whether to use the LLM's image description capabilities to generate descriptions for images found in documents. This is useful for documents that contain images or complex layouts. **Note:** Enabling this may increase processing time and incur additional LLM API costs | `false` |
## Usage
Send document attachments as part of a message to the LLM (similar to sending images):
### Example 1: Document from URL
**Input Message:**
```json theme={null}
{
"model": "document-processing",
"messages": [
{
"role": "user",
"content": [
{
"type": "text",
"text": "Can you summarize the document below?"
},
{
"type": "document",
"document": {
"url": "https://example.com/my-document.pdf"
}
}
]
}
]
}
```
**What happens:**
The plugin downloads the document from the URL, converts it to markdown, and replaces the document content with the markdown text.
**Output to LLM:**
```json theme={null}
{
"role": "user",
"content": [
{
"type": "text",
"text": "Can you summarize the document below?"
},
{
"type": "text",
"text": "[markdown content of the document]"
}
]
}
```
### Example 2: Document from Data URI
**Input Message:**
```json theme={null}
{
"role": "user",
"content": [
{
"type": "document",
"document": {
"data": "data:application/pdf;base64,JVBERi0xLjQK..."
}
}
]
}
```
**What happens:**
The data URI is decoded, processed, and replaced with markdown content.
## Message Format Requirements
The plugin ONLY processes structured multimodal content with explicit `document` type. Plain string URLs like `"content": "https://example.com/doc.pdf"` will NOT be processed.
Documents must be in this format:
```json theme={null}
{
"type": "document",
"document": {
"url": "https://example.com/report.pdf"
}
}
```
Or with data URI:
```json theme={null}
{
"type": "document",
"document": {
"data": "data:application/pdf;base64,..."
}
}
```
## Supported Document Types
* **PDF**: `.pdf`
* **Word**: `.doc`, `.docx`
* **PowerPoint**: `.ppt`, `.pptx`
* **Excel**: `.xls`, `.xlsx`
## Example Configuration
```json theme={null}
{
"use_llm_image_description": true
}
```
This configuration will use an LLM (OpenAI's GPT-4o) to generate detailed descriptions for any images found in the documents, providing richer context for the language model.
## Performance Notes
* Processing time varies by document size and complexity
* Enabling `use_llm_image_description` requires an OpenAI API key configured on the server and may increase processing time and costs
* Large documents (100+ pages) may take longer to process
* Scanned PDFs (image-only) may not extract text without OCR capabilities
* The plugin gracefully handles errors - if processing fails, the original message is preserved
***
## Configuration Schema
```json theme={null}
{
"type": "object",
"title": "Document Processing Plugin Configuration",
"$schema": "http://json-schema.org/draft-07/schema#",
"properties": {
"use_llm_image_description": {
"type": "boolean",
"title": "Use LLM Image Description",
"default": false,
"description": "Use an LLM to generate descriptions for images found in documents (may increase processing time and cost)"
}
},
"description": "Configuration for the document processing plugin that converts documents into markdown text",
"additionalProperties": false
}
```
***
## Supported Phases
* **Request Phase**: Supports processing during the REQUEST phase
# Public Plugins
Source: https://docs.datawizz.ai/public-plugins/index
Pre-built plugins for common use cases
Datawizz provides a collection of pre-built plugins that you can use in your endpoints without writing any custom code. These public plugins cover common use cases like content filtering, PII detection, compliance checks, and more.
## Available Plugins
Browse our collection of public plugins below. Click on any plugin to view detailed documentation and configuration options.
| Plugin | Description | Phases |
| -------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------- |
| [Presidio PII Redaction](./presidio-pii-redaction) | Automatically detects and redacts PII from text in AI requests using configurable methods including replacement, masking, hashing, or encryption before the request reaches the AI model. | REQUEST, RESPONSE, LOG |
| [Presidio Image PII Redaction](./presidio-image-pii-redaction) | Extracts text from images via OCR, detects PII within the text, and returns redacted images with sensitive information obscured to protect privacy in multimodal AI requests. | REQUEST, RESPONSE, LOG |
| [Presidio PII Detection](./presidio-pii-detection) | Analyzes AI requests for personally identifiable information (PII) using Microsoft Presidio and blocks requests containing sensitive data like emails, phone numbers, SSNs, and credit cards. | REQUEST, RESPONSE, LOG |
| [Video Input Processing](./video-input-processing) | Transform videos into LLM-compatible content by extracting frames, audio, and transcripts. Supports YouTube, TikTok, direct URLs, and works with any vision-enabled LLM. | REQUEST |
| [Document Input Processing](./document-input-processing) | Convert PDFs, Word, PowerPoint, and Excel documents into clean markdown for LLM processing. Uses Microsoft's MarkItDown with optional LLM-powered image descriptions. | REQUEST |
| [Regex Detect](./regex-detect) | Scans messages for regex patterns and rejects or redacts matches. Useful for filtering sensitive information like emails, phone numbers, or credit card numbers. | - |
| [Invisible Text](./invisible-text) | Detects and removes non-printable, invisible Unicode characters to maintain text integrity and prevent steganography-based attacks. | - |
| [Block Substrings](./block-substrings) | Blocks or redacts requests containing specific substrings. Useful for filtering unwanted content or enforcing content policies. | - |
| [Llama Guard 3](./llama-guard-3) | Uses Meta's Llama Guard 3 model for content safety classification. Can detect and block unsafe content across 14 hazard categories including violence, hate speech, self-harm, and more. | - |
***
## Using Public Plugins
To use a public plugin in your endpoint:
1. Navigate to your endpoint configuration in the Datawizz dashboard
2. Select the "Plugins" tab
3. Click "Add Plugin"
4. Choose "Public Plugins" from the plugin source dropdown
5. Select the plugin you want to use
6. Configure the plugin settings (phase, priority, async execution, etc.)
7. Save your endpoint configuration
For more information on plugin configuration and execution, see the [Plugins documentation](/plugins/plugins).
## Need a Custom Plugin?
If you need functionality that isn't covered by our public plugins, you can build your own custom plugin. See [Building Custom Plugins](/plugins/build-custom-plugins) for details.
# Invisible Text
Source: https://docs.datawizz.ai/public-plugins/invisible-text
Detects and removes non-printable, invisible Unicode characters to maintain text integrity and prevent steganography-based attacks.
The Invisible Text plugin detects and removes non-printable, invisible Unicode characters from text inputs. This is crucial for maintaining text integrity in Large Language Models (LLMs) and safeguarding against steganography-based attacks where hidden data might be embedded using invisible characters.
## Features
* **Comprehensive Detection**: Identifies invisible characters in Private Use Areas and special Unicode categories
* **Flexible Response**: Either block requests or automatically clean the text
* **Optional Replacement**: Replace invisible characters with visible markers instead of removing them
* **Selective Scanning**: Choose to scan only the latest message or all messages
## Detected Character Types
The plugin targets the following invisible Unicode characters:
### Private Use Areas (PUA)
* **Basic Multilingual Plane**: U+E000 to U+F8FF
* **Supplementary Private Use Area-A**: U+F0000 to U+FFFFD
* **Supplementary Private Use Area-B**: U+100000 to U+10FFFD
### Unicode Categories
* **Cf (Format characters)**: Characters that affect formatting but don't display
* **Cc (Control characters)**: Non-printing control codes
* **Co (Private use characters)**: Reserved for private use
* **Cn (Unassigned characters)**: Code points not assigned to any character
### Common Invisible Characters
* Zero Width Space (U+200B)
* Zero Width Non-Joiner (U+200C)
* Zero Width Joiner (U+200D)
* Word Joiner (U+2060)
* Zero Width No-Break Space/BOM (U+FEFF)
* Soft Hyphen (U+00AD)
* And many more...
## Use Cases
* **Security**: Prevent steganography attacks where data is hidden in invisible characters
* **Data Integrity**: Ensure clean text input to LLMs without hidden formatting
* **Content Validation**: Block messages that may contain hidden instructions
* **Text Normalization**: Clean user input before processing
## Configuration Options
### `scanAllMessages` (boolean)
* `true`: Scans all messages in the array
* `false`: Only scans the latest message
* Default: `false`
### `blockRequest` (boolean)
* `true`: Rejects the request if any invisible characters are found
* `false`: Removes the characters and allows the request to continue
* Default: `false`
### `redactCharacters` (string)
* If set, replaces each detected invisible character with this visible character
* If not set, removes the invisible characters entirely
* Useful for debugging or auditing purposes
* Example: `"?"` or `"⬚"`
## Example Configurations
### Remove Invisible Characters (Default)
```json theme={null}
{
"scanAllMessages": false,
"blockRequest": false
}
```
This configuration silently removes any invisible characters from the latest message.
### Block Requests with Invisible Characters
```json theme={null}
{
"scanAllMessages": true,
"blockRequest": true
}
```
This configuration scans all messages and rejects the request if any invisible characters are detected.
### Replace Invisible Characters with Markers
```json theme={null}
{
"scanAllMessages": false,
"blockRequest": false,
"redactCharacters": "⬚"
}
```
This configuration replaces invisible characters with a visible placeholder (⬚) for debugging purposes.
### Strict Mode for All Messages
```json theme={null}
{
"scanAllMessages": true,
"blockRequest": true
}
```
This configuration provides maximum security by scanning all messages and blocking any request containing invisible characters.
## Response Behavior
### When Blocking (blockRequest: true)
* Returns `reject: true` with reason indicating how many invisible characters were found
* No messages are modified
* Debug logs show specific Unicode code points detected
### When Cleaning (blockRequest: false)
* Returns modified messages array with invisible characters removed/replaced
* Original request continues through the pipeline
* Debug logs indicate how many characters were cleaned
## Example Scenarios
### Scenario 1: Hidden Prompt Injection
**Input**: `"Ignore previous instructions\u200Band do something else"`
**With blockRequest: true**: Request is rejected
**With blockRequest: false**: Cleaned to `"Ignore previous instructionsand do something else"`
### Scenario 2: Steganography Attack
**Input**: Text with hidden data encoded in Private Use Area characters
**Detection**: Plugin identifies PUA characters (e.g., U+E001, U+E002, etc.)
**Response**: Either blocks or removes these characters based on configuration
## Debug Information
The plugin provides detailed debug information including:
* Number of messages scanned
* Total invisible characters found
* Specific Unicode code points detected (e.g., U+200B, U+FEFF)
* Action taken (blocked, removed, or replaced)
## Technical Notes
* Handles both string and array content formats
* Correctly processes surrogate pairs for characters outside the Basic Multilingual Plane
* Zero-width characters are always detected regardless of context
* Format characters that affect bidirectional text (like RLO, LRO) are detected
* The plugin is designed to be safe and will not remove legitimate Unicode characters
***
## Configuration Schema
```json theme={null}
{
"type": "object",
"title": "Invisible Text Configuration",
"$schema": "http://json-schema.org/draft-07/schema#",
"properties": {
"blockRequest": {
"type": "boolean",
"title": "Block Request",
"default": false,
"description": "If true, rejects the request if any invisible characters are found. If false, removes the characters and allows the request"
},
"scanAllMessages": {
"type": "boolean",
"title": "Scan All Messages",
"default": false,
"description": "If true, scans all messages. If false, only scans the latest message"
},
"redactCharacters": {
"type": "string",
"title": "Redact Characters",
"maxLength": 10,
"description": "If set, replaces each detected invisible character with this character. If not set, removes the characters entirely"
}
},
"description": "Configuration for the Invisible Text detection plugin"
}
```
# Llama Guard 3
Source: https://docs.datawizz.ai/public-plugins/llama-guard-3
Uses Meta's Llama Guard 3 model for content safety classification. Can detect and block unsafe content across 14 hazard categories including violence, hate speech, self-harm, and more.
The Llama Guard 3 plugin uses Meta's Llama-3.1-8B model, fine-tuned for content safety classification. It can classify content in both LLM inputs (prompt classification) and responses (response classification), detecting violations across 14 different hazard categories.
## Features
* **Dual Deployment Support**: Works with Cloudflare Workers AI binding (zero-latency) or REST API fallback (for standalone binaries)
* **14 Hazard Categories**: Detects violence, hate speech, sexual content, self-harm, and more
* **Flexible Actions**: Either reject unsafe requests or log warnings while allowing them through
* **Conversation-Aware**: Can scan individual messages or entire conversation history
* **Token Usage Tracking**: Reports token consumption for monitoring and cost control
## Hazard Categories
Llama Guard 3 detects the following categories (S1-S14):
* **S1**: Violent Crimes
* **S2**: Non-Violent Crimes
* **S3**: Sex-Related Crimes
* **S4**: Child Sexual Exploitation
* **S5**: Defamation
* **S6**: Specialized Advice (financial, medical, legal)
* **S7**: Privacy Violations
* **S8**: Intellectual Property
* **S9**: Indiscriminate Weapons
* **S10**: Hate Speech
* **S11**: Suicide & Self-Harm
* **S12**: Sexual Content
* **S13**: Elections
* **S14**: Code Interpreter Abuse
## Configuration Options
### `action` (string)
* `reject`: Block requests containing unsafe content (default)
* `warn`: Log warnings but allow requests to proceed
* Default: `"reject"`
### `scanAllMessages` (boolean)
* `true`: Scans all messages in the conversation
* `false`: Only scans the latest message
* Default: `false`
### `rejectMessage` (string)
* Custom message to return when content is rejected
* If not provided, returns a message listing violated categories
* Example: `"Your message violates our content policy"`
### `temperature` (number)
* Controls randomness in model output (0-5)
* Lower values = more consistent classification
* Default: `0.6`
### `maxTokens` (integer)
* Maximum tokens to generate in response (1-2048)
* Default: `256`
### `cloudflareAccountId` (string)
* Required for REST API fallback (standalone binary)
* Your Cloudflare Account ID
* Not needed when running in Cloudflare Workers
### `cloudflareAuthToken` (string)
* Required for REST API fallback (standalone binary)
* Cloudflare API token with Workers AI access
* Not needed when running in Cloudflare Workers
## Example Configurations
### Basic Usage (Cloudflare Workers)
```json theme={null}
{
"action": "reject",
"scanAllMessages": false
}
```
### Scan Entire Conversation
```json theme={null}
{
"action": "reject",
"scanAllMessages": true,
"rejectMessage": "This conversation contains content that violates our safety policies."
}
```
### Warning Mode (Non-Blocking)
```json theme={null}
{
"action": "warn",
"scanAllMessages": true
}
```
### Standalone Binary with REST API
```json theme={null}
{
"action": "reject",
"scanAllMessages": false,
"cloudflareAccountId": "your-account-id",
"cloudflareAuthToken": "your-api-token",
"temperature": 0.5,
"maxTokens": 256
}
```
## Response Behavior
### When Content is Safe
* Request proceeds normally
* No modifications to messages
* Debug logs indicate "Content classified as SAFE"
### When Content is Unsafe (action: "reject")
* Request is blocked
* Returns rejection reason with violated categories
* Example: `"Content safety check failed. Violated categories: S10, S11"`
### When Content is Unsafe (action: "warn")
* Request proceeds with warning in debug logs
* No modifications to messages
* Warning includes violated categories
* Useful for monitoring without blocking users
### On Error
* Plugin fails gracefully
* Request is allowed to proceed with warning in debug
* Error details logged for troubleshooting
## Deployment Modes
### Cloudflare Workers (Recommended)
Uses the native AI binding for zero-latency, cost-efficient inference:
```typescript theme={null}
// Worker automatically uses env.AI binding
// No additional configuration needed
```
Advantages:
* ✅ Zero cold start
* ✅ Lower latency
* ✅ No auth tokens needed
* ✅ Integrated billing
### Standalone Binary (REST API Fallback)
Uses Cloudflare's REST API when AI binding is not available:
```json theme={null}
{
"cloudflareAccountId": "abc123...",
"cloudflareAuthToken": "token_xyz..."
}
```
Advantages:
* ✅ Works outside Cloudflare Workers
* ✅ Self-hosted deployment
* ✅ Same model and accuracy
## Use Cases
### Content Moderation
Block harmful content before it reaches your LLM:
```json theme={null}
{
"action": "reject",
"scanAllMessages": false,
"rejectMessage": "Your message contains inappropriate content."
}
```
### Conversation Safety
Monitor entire conversations for policy violations:
```json theme={null}
{
"action": "reject",
"scanAllMessages": true
}
```
### Compliance Monitoring
Log safety issues without blocking users:
```json theme={null}
{
"action": "warn",
"scanAllMessages": true
}
```
### Fine-Grained Control
Adjust temperature for stricter or looser classification:
```json theme={null}
{
"action": "reject",
"temperature": 0.3,
"rejectMessage": "Content policy violation detected"
}
```
## Performance
* **Cloudflare Workers**: \~100-200ms latency
* **REST API**: \~200-500ms latency (depending on network)
* **Token Usage**: Typically 50-200 tokens per request
* **Cost**: $0.48 per 1M input tokens, $0.03 per 1M output tokens
## Limitations
* **Message Format**: Only processes `user` and `assistant` roles (system/developer/tool messages treated as user)
* **Text Only**: Images, audio, and other media are not analyzed
* **Language**: Optimized for English, may have reduced accuracy in other languages
* **Context Length**: Limited by model's context window (\~8K tokens)
## Troubleshooting
### Error: "cloudflareAccountId and cloudflareAuthToken are required"
**Cause**: Running outside Cloudflare Workers without REST API credentials
**Solution**: Add your Cloudflare Account ID and API token to the configuration
### Error: "Cloudflare API error (401)"
**Cause**: Invalid or expired API token
**Solution**:
1. Generate a new API token at [https://dash.cloudflare.com/profile/api-tokens](https://dash.cloudflare.com/profile/api-tokens)
2. Ensure it has "Workers AI" permissions
3. Update `cloudflareAuthToken` in configuration
### False Positives
**Cause**: Model may flag legitimate content
**Solution**:
* Increase `temperature` (e.g., 0.8) for less strict classification
* Use `action: "warn"` to monitor without blocking
* Review debug logs to understand classification reasoning
### High Latency
**Cause**: Using REST API instead of Workers AI binding
**Solution**:
* Deploy to Cloudflare Workers for optimal performance
* Consider caching results for repeated content
* Use `scanAllMessages: false` to scan only recent messages
## Integration Examples
### With Prompt Engineering
```json theme={null}
{
"action": "reject",
"scanAllMessages": false,
"rejectMessage": "Please rephrase your message to comply with our content guidelines."
}
```
### With Logging Plugin
```json theme={null}
{
"action": "warn",
"scanAllMessages": true
}
```
Safety warnings will be included in log metadata for analysis.
### Multi-Layer Protection
Use alongside other plugins for defense-in-depth:
1. **Regex Detect**: Block obvious patterns (profanity, PII)
2. **Llama Guard**: Catch nuanced safety issues
3. **Invisible Text**: Prevent steganography attacks
## Further Reading
* [Llama Guard 3 Model Card](https://developers.cloudflare.com/workers-ai/models/llama-guard-3-8b/)
* [Cloudflare Workers AI Documentation](https://developers.cloudflare.com/workers-ai/)
* [Content Safety Best Practices](https://ai.meta.com/llama/responsible-use-guide/)
***
## Configuration Schema
```json theme={null}
{
"type": "object",
"$schema": "http://json-schema.org/draft-07/schema#",
"required": [],
"properties": {
"action": {
"enum": [
"reject",
"warn"
],
"type": "string",
"default": "reject",
"description": "Action to take when unsafe content is detected. 'reject' blocks the request, 'warn' logs a warning but allows it."
},
"maxTokens": {
"type": "integer",
"default": 256,
"maximum": 2048,
"minimum": 1,
"description": "Maximum number of tokens to generate in the response."
},
"temperature": {
"type": "number",
"default": 0.6,
"maximum": 5,
"minimum": 0,
"description": "Controls randomness in Llama Guard output. Higher values produce more varied results."
},
"rejectMessage": {
"type": "string",
"description": "Custom message to return when content is rejected. If not provided, a default message with violation categories will be used."
},
"scanAllMessages": {
"type": "boolean",
"default": false,
"description": "If true, scans all messages in the conversation. If false, only scans the last message."
},
"cloudflareAccountId": {
"type": "string",
"description": "Cloudflare Account ID (required when running outside of Cloudflare Workers, e.g., in standalone binary)"
},
"cloudflareAuthToken": {
"type": "string",
"description": "Cloudflare API auth token (required when running outside of Cloudflare Workers)"
}
}
}
```
# Presidio Image PII Redaction
Source: https://docs.datawizz.ai/public-plugins/presidio-image-pii-redaction
Extracts text from images via OCR, detects PII within the text, and returns redacted images with sensitive information obscured to protect privacy in multimodal AI requests.
Extracts text from images via OCR, detects PII within the text, and returns redacted images with sensitive information obscured.
## Overview
The Image Redaction Plugin processes images in multimodal AI requests, using Optical Character Recognition (OCR) to extract text, detecting PII within that text, and then visually redacting the sensitive areas by overlaying them with colored boxes. This prevents PII in screenshots, documents, or photos from being exposed to AI models.
## How It Works
1. **Image Processing**: Extracts images from message content (supports both URLs and base64 data URIs)
2. **OCR Analysis**: Uses Tesseract OCR to extract text from images
3. **PII Detection**: Analyzes extracted text using Microsoft Presidio to identify PII
4. **Visual Redaction**: Overlays detected PII regions with colored boxes to obscure the text
5. **Return Modified Images**: Returns images as base64 data URIs with PII redacted
## Supported PII Types
The plugin can detect and redact 30+ entity types across multiple regions:
### Personal Information
* `PERSON` - Person names
* `EMAIL_ADDRESS` - Email addresses
* `PHONE_NUMBER` - Phone numbers
* `DATE_TIME` - Dates and times
* `LOCATION` - Geographic locations
* `URL` - Web addresses
* `IP_ADDRESS` - IP addresses
### Financial
* `CREDIT_CARD` - Credit card numbers
* `CRYPTO` - Cryptocurrency wallet addresses
* `IBAN_CODE` - International bank account numbers
### United States
* `US_SSN` - Social Security Numbers
* `US_DRIVER_LICENSE` - Driver's license numbers
* `US_PASSPORT` - Passport numbers
* `US_BANK_NUMBER` - Bank account numbers
* `US_ITIN` - Individual Taxpayer Identification Numbers
### International
* `UK_NHS` - UK National Health Service numbers
* `SG_NRIC_FIN` - Singapore NRIC/FIN numbers
* `AU_ABN`, `AU_ACN`, `AU_TFN`, `AU_MEDICARE` - Australian identifiers
* `IN_PAN`, `IN_AADHAAR`, `IN_VEHICLE_REGISTRATION` - Indian identifiers
* `ES_NIF` - Spanish tax identification
* `IT_FISCAL_CODE`, `IT_DRIVER_LICENSE`, `IT_VAT_CODE`, `IT_PASSPORT`, `IT_IDENTITY_CARD` - Italian identifiers
### Healthcare
* `MEDICAL_LICENSE` - Medical license numbers
* `NRP` - Medical prescriber numbers
## Configuration
### Basic Settings
**`entities`** (optional, array of strings)
List of PII entity types to detect and redact in images. If not specified, all detected entities are redacted.
**`language`** (string, default: `"en"`)
Language code for OCR text analysis (e.g., `"en"`, `"es"`, `"de"`).
**`score_threshold`** (number, default: `0.5`)
Minimum confidence score (0-1) required to redact an entity. Lower values catch more PII but may increase false positives.
### Visual Redaction Settings
**`fill_color`** (string or RGB array, default: `"black"`)
Fill color for redacted areas. Can be:
* **Color name**: `"black"`, `"white"`, `"gray"`, `"red"`, etc.
* **RGB tuple**: `[0, 0, 0]` for black, `[255, 255, 255]` for white, `[255, 0, 0]` for red
**`padding`** (number, default: `10`)
Padding in pixels around detected text to ensure complete coverage. Higher values provide more margin but may obscure surrounding content.
### Advanced OCR Settings
**`ocr_kwargs`** (optional, object)
Additional keyword arguments to pass to the OCR engine (Tesseract). Common options:
* `lang`: Language code (e.g., `"eng"`, `"spa"`, `"deu"`)
* `config`: Tesseract configuration string (e.g., `"--psm 6"` for uniform text block)
See [Tesseract documentation](https://tesseract-ocr.github.io/) for all available options.
### Custom Pattern Recognition
**`ad_hoc_recognizers`** (optional, array of objects)
Custom regex-based recognizers for detecting domain-specific patterns not covered by standard entity types.
**Structure**:
```json theme={null}
{
"name": "unique_recognizer_name",
"supported_language": "en",
"patterns": ["regex_pattern_1", "regex_pattern_2"],
"context": ["context_word_1", "context_word_2"],
"supported_entity": "CUSTOM_ENTITY_TYPE"
}
```
### Advanced Detection
**`allow_list`** (optional, array of strings)
Terms/patterns that should NOT be redacted from images, even if they match detection patterns.
**`deny_list`** (optional, array of strings)
Terms/patterns that should ALWAYS be redacted from images, regardless of detection confidence.
## Example Configurations
### Basic Image Redaction
```json theme={null}
{
"entities": ["EMAIL_ADDRESS", "PHONE_NUMBER", "PERSON"],
"fill_color": "black",
"padding": 15
}
```
Redacts emails, phone numbers, and names with black boxes with 15px padding.
### Custom Fill Color
```json theme={null}
{
"entities": ["CREDIT_CARD", "US_SSN"],
"fill_color": [255, 0, 0],
"padding": 10
}
```
Redacts credit cards and SSNs with red (`RGB: 255, 0, 0`) boxes.
### Enhanced OCR for Non-English
```json theme={null}
{
"entities": ["PERSON", "LOCATION"],
"language": "es",
"ocr_kwargs": {
"lang": "spa",
"config": "--psm 6"
},
"fill_color": "gray"
}
```
Processes Spanish text with Tesseract configured for Spanish language.
### Custom Pattern Detection
```json theme={null}
{
"entities": ["EMAIL_ADDRESS"],
"ad_hoc_recognizers": [
{
"name": "employee_id_recognizer",
"supported_language": "en",
"patterns": ["EMP-\\d{6}", "STAFF-\\d{6}"],
"context": ["employee", "staff", "id"],
"supported_entity": "EMPLOYEE_ID"
},
{
"name": "internal_code_recognizer",
"supported_language": "en",
"patterns": ["PROJ-[A-Z]{3}-\\d{4}"],
"supported_entity": "PROJECT_CODE"
}
],
"fill_color": "black",
"padding": 12
}
```
Detects custom patterns like `EMP-123456` or `PROJ-ABC-1234` in addition to standard emails.
### High Sensitivity Mode
```json theme={null}
{
"score_threshold": 0.3,
"padding": 20,
"fill_color": "white",
"deny_list": ["CONFIDENTIAL", "INTERNAL USE ONLY"]
}
```
Uses lower threshold to catch more potential PII, larger padding for complete coverage, and always redacts specific terms.
## Image Format Support
The plugin handles multiple image input formats:
### HTTP/HTTPS URLs
```json theme={null}
{
"type": "image_url",
"image_url": {
"url": "https://example.com/screenshot.png"
}
}
```
Plugin fetches the image, processes it, and returns redacted base64 data URI.
### Base64 Data URIs
```json theme={null}
{
"type": "image_url",
"image_url": {
"url": "data:image/png;base64,iVBORw0KGgoAAAANS..."
}
}
```
Plugin processes the image directly and returns redacted base64 data URI.
## Behavior
* **Fail-open**: If the plugin encounters an error, the original messages are returned unmodified
* **Multi-image support**: Processes all images in all messages independently
* **Format preservation**: Maintains message structure (multimodal content arrays)
* **URL conversion**: Converts fetched URLs to base64 data URIs for redacted images
* **Debug output**: Returns detailed processing information when enabled in Gateway UI
* **No blocking**: Always allows requests to proceed (unlike Detection Plugin)
## Performance Considerations
* **OCR latency**: Image processing takes 1-3 seconds per image depending on size and complexity
* **Image size limits**: Large images (>10MB) may timeout; consider resizing before processing
* **Cold starts**: First container invocation may take 2-3 seconds
* **Base64 size**: Redacted images returned as base64 may be large; monitor response sizes
## Use Cases
1. **Screenshot sanitization**: Remove PII from screenshots before sharing with AI models
2. **Document processing**: Redact sensitive information from scanned documents
3. **Support tickets**: Process user-submitted images containing PII in customer support scenarios
4. **Compliance**: Ensure uploaded images don't expose regulated data (HIPAA, GDPR)
5. **Testing**: Sanitize production screenshots for use in development/testing environments
6. **Multi-modal AI safety**: Prevent vision models from accessing PII in image content
## Limitations
* **OCR quality**: Detection accuracy depends on image quality, text clarity, and font legibility
* **Handwritten text**: OCR may struggle with handwriting; results vary
* **Complex layouts**: Dense or overlapping text may reduce detection accuracy
* **Non-text PII**: Cannot detect faces, objects, or other non-textual PII
* **Language support**: OCR quality varies by language; best results with Latin scripts
***
## Configuration Schema
```json theme={null}
{
"type": "object",
"title": "PII Image Redaction Plugin Configuration",
"$schema": "http://json-schema.org/draft-07/schema#",
"properties": {
"padding": {
"type": "number",
"title": "Padding",
"default": 10,
"description": "Padding in pixels around detected text to ensure complete redaction coverage."
},
"entities": {
"type": "array",
"items": {
"type": "string"
},
"title": "Entity Types",
"examples": [
[
"EMAIL_ADDRESS",
"PHONE_NUMBER",
"PERSON"
]
],
"description": "List of PII entity types to detect and redact in images. If not specified, all detected entities will be redacted."
},
"language": {
"type": "string",
"title": "Language",
"default": "en",
"examples": [
"en",
"es",
"de",
"fr"
],
"description": "Language code for OCR text analysis."
},
"deny_list": {
"type": "array",
"items": {
"type": "string"
},
"title": "Deny List",
"description": "List of terms/patterns that should always be redacted from images."
},
"allow_list": {
"type": "array",
"items": {
"type": "string"
},
"title": "Allow List",
"description": "List of terms/patterns that should not be redacted from images."
},
"fill_color": {
"oneOf": [
{
"type": "string",
"examples": [
"black",
"white",
"gray"
]
},
{
"type": "array",
"items": {
"type": "number",
"maximum": 255,
"minimum": 0
},
"examples": [
[
0,
0,
0
]
],
"maxItems": 3,
"minItems": 3
}
],
"title": "Fill Color",
"default": "black",
"description": "Fill color for redacted areas. Can be color name or RGB tuple [R, G, B] (0-255)"
},
"ocr_kwargs": {
"type": "object",
"title": "OCR Arguments",
"examples": [
{
"lang": "eng",
"config": "--psm 6"
}
],
"description": "Additional keyword arguments to pass to the OCR engine (Tesseract). See Tesseract documentation for available options.",
"additionalProperties": true
},
"score_threshold": {
"type": "number",
"title": "Score Threshold",
"default": 0.5,
"maximum": 1,
"minimum": 0,
"description": "Minimum confidence score (0-1) required to redact an entity."
},
"ad_hoc_recognizers": {
"type": "array",
"items": {
"type": "object",
"required": [
"name",
"supported_language",
"patterns",
"supported_entity"
],
"properties": {
"name": {
"type": "string",
"title": "Name",
"description": "Unique name for the recognizer"
},
"context": {
"type": "array",
"items": {
"type": "string"
},
"title": "Context Words",
"description": "Context words to improve detection"
},
"patterns": {
"type": "array",
"items": {
"type": "string"
},
"title": "Patterns",
"description": "Regex patterns to match"
},
"supported_entity": {
"type": "string",
"title": "Supported Entity",
"description": "Entity type this recognizer detects"
},
"supported_language": {
"type": "string",
"title": "Supported Language",
"description": "Language code this recognizer supports"
}
}
},
"title": "Custom Recognizers",
"examples": [
[
{
"name": "employee_id_recognizer",
"context": [
"employee",
"staff",
"id"
],
"patterns": [
"EMP-\\d{6}"
],
"supported_entity": "EMPLOYEE_ID",
"supported_language": "en"
}
]
],
"description": "Custom regex-based recognizers for detecting specific patterns in images."
}
},
"description": "Configuration for the Presidio-based image redaction plugin that detects and redacts PII from images using OCR."
}
```
***
## Supported Phases
* **Request Phase**: Supports processing during the REQUEST phase
* **Response Phase**: Supports processing during the RESPONSE phase
* **Log Phase**: Supports processing during the LOG phase
# Presidio PII Detection
Source: https://docs.datawizz.ai/public-plugins/presidio-pii-detection
Analyzes AI requests for personally identifiable information (PII) using Microsoft Presidio and blocks requests containing sensitive data like emails, phone numbers, SSNs, and credit cards.
Analyzes AI requests for personally identifiable information (PII) using Microsoft Presidio and blocks requests containing sensitive data.
## Overview
The Detection Plugin scans incoming AI requests for PII entities and rejects requests when sensitive information is detected. It provides granular control over what types of PII to detect, confidence thresholds, and custom rejection messages.
## Supported PII Types
The plugin can detect 30+ entity types across multiple regions:
### Personal Information
* `PERSON` - Person names
* `EMAIL_ADDRESS` - Email addresses
* `PHONE_NUMBER` - Phone numbers
* `DATE_TIME` - Dates and times
* `LOCATION` - Geographic locations
* `URL` - Web addresses
* `IP_ADDRESS` - IP addresses
### Financial
* `CREDIT_CARD` - Credit card numbers
* `CRYPTO` - Cryptocurrency wallet addresses
* `IBAN_CODE` - International bank account numbers
### United States
* `US_SSN` - Social Security Numbers
* `US_DRIVER_LICENSE` - Driver's license numbers
* `US_PASSPORT` - Passport numbers
* `US_BANK_NUMBER` - Bank account numbers
* `US_ITIN` - Individual Taxpayer Identification Numbers
### International
* `UK_NHS` - UK National Health Service numbers
* `SG_NRIC_FIN` - Singapore NRIC/FIN numbers
* `AU_ABN`, `AU_ACN`, `AU_TFN`, `AU_MEDICARE` - Australian identifiers
* `IN_PAN`, `IN_AADHAAR`, `IN_VEHICLE_REGISTRATION` - Indian identifiers
* `ES_NIF` - Spanish tax identification
* `IT_FISCAL_CODE`, `IT_DRIVER_LICENSE`, `IT_VAT_CODE`, `IT_PASSPORT`, `IT_IDENTITY_CARD` - Italian identifiers
### Healthcare
* `MEDICAL_LICENSE` - Medical license numbers
* `NRP` - Medical prescriber numbers
## Configuration
### Basic Settings
**`entities`** (optional, array of strings)
List of PII entity types to detect. If not specified, all supported entities are detected.
**`language`** (string, default: `"en"`)
Language code for text analysis (e.g., `"en"`, `"es"`, `"de"`).
**`score_threshold`** (number, default: `0.5`)
Minimum confidence score (0-1) required to flag text as PII. Lower values (e.g., `0.4`) catch more PII but may increase false positives. Higher values (e.g., `0.7`) reduce false positives but may miss some PII.
### Rejection Behavior
**`reject_on_detection`** (boolean, default: `true`)
Whether to reject requests when PII is detected. Set to `false` to allow requests but log PII detection for monitoring.
**`rejection_message`** (string)
Custom message returned when a request is rejected. Default: `"Request contains personally identifiable information and cannot be processed."`
### Advanced Detection
**`allow_list`** (optional, array of strings)
Terms/patterns that should NOT be flagged as PII, even if they match detection patterns (e.g., `["example@company.com", "555-0000"]`).
**`deny_list`** (optional, array of strings)
Terms/patterns that should ALWAYS be flagged as PII, regardless of detection confidence (e.g., `["confidential", "internal-use-only"]`).
**`context`** (optional, array of strings)
Additional context words to improve detection accuracy (e.g., `["patient", "medical", "doctor"]`).
**`return_decision_process`** (boolean, default: `false`)
Include detailed analysis explanation in debug output to understand why text was flagged.
## Example Configuration
```json theme={null}
{
"entities": ["EMAIL_ADDRESS", "PHONE_NUMBER", "US_SSN", "CREDIT_CARD"],
"score_threshold": 0.6,
"reject_on_detection": true,
"rejection_message": "Your request contains sensitive information. Please remove PII and try again.",
"allow_list": ["support@example.com"],
"return_decision_process": true
}
```
## Behavior
* **Fail-open**: If the plugin encounters an error, requests are allowed to proceed
* **Multi-message support**: Analyzes all messages in the request
* **Debug output**: Returns detailed detection information when enabled in Gateway UI
***
## Configuration Schema
```json theme={null}
{
"type": "object",
"title": "PII Detection Plugin Configuration",
"$schema": "http://json-schema.org/draft-07/schema#",
"properties": {
"context": {
"type": "array",
"items": {
"type": "string"
},
"title": "Context Words",
"examples": [
[
"patient",
"medical",
"doctor"
]
],
"description": "Additional context words to improve detection accuracy. These help the analyzer understand the surrounding text."
},
"entities": {
"type": "array",
"items": {
"enum": [
"PERSON",
"EMAIL_ADDRESS",
"PHONE_NUMBER",
"CREDIT_CARD",
"CRYPTO",
"DATE_TIME",
"IBAN_CODE",
"IP_ADDRESS",
"LOCATION",
"MEDICAL_LICENSE",
"NRP",
"URL",
"US_BANK_NUMBER",
"US_DRIVER_LICENSE",
"US_ITIN",
"US_PASSPORT",
"US_SSN",
"UK_NHS",
"SG_NRIC_FIN",
"AU_ABN",
"AU_ACN",
"AU_TFN",
"AU_MEDICARE",
"IN_PAN",
"IN_AADHAAR",
"IN_VEHICLE_REGISTRATION",
"ES_NIF",
"IT_FISCAL_CODE",
"IT_DRIVER_LICENSE",
"IT_VAT_CODE",
"IT_PASSPORT",
"IT_IDENTITY_CARD"
],
"type": "string"
},
"title": "Entity Types",
"examples": [
[
"EMAIL_ADDRESS",
"PHONE_NUMBER",
"CREDIT_CARD"
]
],
"description": "List of PII entity types to detect. If not specified, all supported entities will be detected."
},
"language": {
"type": "string",
"title": "Language",
"default": "en",
"examples": [
"en",
"es",
"de",
"fr"
],
"description": "Language code for text analysis. Supported languages depend on Presidio configuration."
},
"deny_list": {
"type": "array",
"items": {
"type": "string"
},
"title": "Deny List",
"examples": [
[
"confidential",
"internal-use-only"
]
],
"description": "List of terms/patterns that should always be flagged as PII, regardless of detection confidence."
},
"allow_list": {
"type": "array",
"items": {
"type": "string"
},
"title": "Allow List",
"examples": [
[
"example@company.com",
"555-0000"
]
],
"description": "List of terms/patterns that should not be flagged as PII, even if they match detection patterns."
},
"score_threshold": {
"type": "number",
"title": "Score Threshold",
"default": 0.5,
"maximum": 1,
"minimum": 0,
"description": "Minimum confidence score (0-1) required to consider an entity as PII. Higher values reduce false positives but may miss some PII."
},
"rejection_message": {
"type": "string",
"title": "Rejection Message",
"default": "Request contains personally identifiable information and cannot be processed.",
"description": "Custom message to return when request is rejected due to PII detection."
},
"reject_on_detection": {
"type": "boolean",
"title": "Reject on Detection",
"default": true,
"description": "Whether to reject requests when PII is detected. Set to false to allow requests but log PII detection."
},
"return_decision_process": {
"type": "boolean",
"title": "Return Decision Process",
"default": false,
"description": "Include detailed analysis explanation in debug output. Useful for understanding why certain text was flagged."
}
},
"description": "Configuration for the Presidio-based PII detection plugin that rejects requests containing personally identifiable information."
}
```
***
## Supported Phases
* **Request Phase**: Supports processing during the REQUEST phase
* **Response Phase**: Supports processing during the RESPONSE phase
* **Log Phase**: Supports processing during the LOG phase
# Presidio PII Redaction
Source: https://docs.datawizz.ai/public-plugins/presidio-pii-redaction
Automatically detects and redacts PII from text in AI requests using configurable methods including replacement, masking, hashing, or encryption before the request reaches the AI model.
Automatically detects and redacts personally identifiable information (PII) from text in AI requests using configurable anonymization methods.
## Overview
The Text Redaction Plugin scans incoming AI requests for PII entities and redacts them before the request reaches the AI model. Unlike the Detection Plugin which blocks requests, this plugin allows requests to proceed with PII safely anonymized. It supports multiple redaction strategies and can apply different methods to different entity types.
## Supported PII Types
The plugin can redact 30+ entity types across multiple regions:
### Personal Information
* `PERSON` - Person names
* `EMAIL_ADDRESS` - Email addresses
* `PHONE_NUMBER` - Phone numbers
* `DATE_TIME` - Dates and times
* `LOCATION` - Geographic locations
* `URL` - Web addresses
* `IP_ADDRESS` - IP addresses
### Financial
* `CREDIT_CARD` - Credit card numbers
* `CRYPTO` - Cryptocurrency wallet addresses
* `IBAN_CODE` - International bank account numbers
### United States
* `US_SSN` - Social Security Numbers
* `US_DRIVER_LICENSE` - Driver's license numbers
* `US_PASSPORT` - Passport numbers
* `US_BANK_NUMBER` - Bank account numbers
* `US_ITIN` - Individual Taxpayer Identification Numbers
### International
* `UK_NHS` - UK National Health Service numbers
* `SG_NRIC_FIN` - Singapore NRIC/FIN numbers
* `AU_ABN`, `AU_ACN`, `AU_TFN`, `AU_MEDICARE` - Australian identifiers
* `IN_PAN`, `IN_AADHAAR`, `IN_VEHICLE_REGISTRATION` - Indian identifiers
* `ES_NIF` - Spanish tax identification
* `IT_FISCAL_CODE`, `IT_DRIVER_LICENSE`, `IT_VAT_CODE`, `IT_PASSPORT`, `IT_IDENTITY_CARD` - Italian identifiers
### Healthcare
* `MEDICAL_LICENSE` - Medical license numbers
* `NRP` - Medical prescriber numbers
## Anonymization Methods
The plugin supports five redaction strategies:
### `replace`
Substitutes PII with a fixed value.
* **Parameters**: `new_value` (string) - Replacement text
* **Example**: `john@example.com` → ``
### `redact`
Completely removes PII from the text.
* **Parameters**: None
* **Example**: `My email is john@example.com` → `My email is`
### `mask`
Partially obscures PII by replacing characters.
* **Parameters**:
* `masking_char` (string, default: `"*"`) - Character to use for masking
* `chars_to_mask` (number, default: 100) - Number of characters to mask
* `from_end` (boolean, default: false) - Mask from end instead of beginning
* **Example**: `555-123-4567` → `XXX-XXX-4567` (masking first 7 chars with "X")
### `hash`
Applies one-way hashing (SHA-256) to PII.
* **Parameters**: None
* **Example**: `john@example.com` → `a3c7f1...` (irreversible)
### `encrypt`
Applies reversible encryption to PII.
* **Parameters**: Encryption key (configured in Presidio)
* **Example**: `john@example.com` → `bHj9K2...` (reversible with key)
## Configuration
### Basic Settings
**`entities`** (optional, array of strings)
List of PII entity types to redact. If not specified, all detected entities are redacted.
**`language`** (string, default: `"en"`)
Language code for text analysis (e.g., `"en"`, `"es"`, `"de"`).
**`score_threshold`** (number, default: `0.5`)
Minimum confidence score (0-1) required to redact an entity. Lower values catch more PII but may increase false positives.
### Anonymization Configuration
**`default_anonymizer`** (object, default: `{ type: "replace", new_value: "" }`)
Default anonymization method applied to all detected entities unless overridden.
**Structure**:
```json theme={null}
{
"type": "replace|redact|hash|mask|encrypt",
"new_value": "", // For 'replace' type
"masking_char": "*", // For 'mask' type
"chars_to_mask": 4, // For 'mask' type
"from_end": true // For 'mask' type
}
```
**`entity_anonymizers`** (optional, object)
Entity-specific anonymization methods that override the default. Keys are entity types.
**Example**:
```json theme={null}
{
"PHONE_NUMBER": {
"type": "mask",
"masking_char": "X",
"chars_to_mask": 4,
"from_end": true
},
"EMAIL_ADDRESS": {
"type": "replace",
"new_value": ""
},
"CREDIT_CARD": {
"type": "hash"
}
}
```
### Advanced Detection
**`allow_list`** (optional, array of strings)
Terms/patterns that should NOT be redacted, even if they match detection patterns.
**`deny_list`** (optional, array of strings)
Terms/patterns that should ALWAYS be redacted, regardless of detection confidence.
**`context`** (optional, array of strings)
Additional context words to improve detection accuracy.
## Example Configurations
### Basic Redaction
```json theme={null}
{
"entities": ["EMAIL_ADDRESS", "PHONE_NUMBER"],
"default_anonymizer": {
"type": "replace",
"new_value": "[REMOVED]"
}
}
```
**Input**: `"Contact me at john@example.com or 555-123-4567"`
**Output**: `"Contact me at [REMOVED] or [REMOVED]"`
### Partial Masking
```json theme={null}
{
"entities": ["CREDIT_CARD", "PHONE_NUMBER"],
"entity_anonymizers": {
"CREDIT_CARD": {
"type": "mask",
"masking_char": "*",
"chars_to_mask": 12,
"from_end": false
},
"PHONE_NUMBER": {
"type": "mask",
"masking_char": "X",
"chars_to_mask": 6,
"from_end": false
}
}
}
```
**Input**: `"Card: 4532-1234-5678-9010, Phone: 555-123-4567"`
**Output**: `"Card: ****-****-****-9010, Phone: XXX-XXX-4567"`
### Mixed Strategies
```json theme={null}
{
"entities": ["EMAIL_ADDRESS", "PERSON", "US_SSN"],
"default_anonymizer": {
"type": "replace",
"new_value": ""
},
"entity_anonymizers": {
"EMAIL_ADDRESS": { "type": "hash" },
"US_SSN": {
"type": "mask",
"masking_char": "*",
"chars_to_mask": 5,
"from_end": false
}
}
}
```
**Input**: `"John Smith's SSN is 123-45-6789, email: john@example.com"`
**Output**: `"'s SSN is ***-**-6789, email: a3c7f1b2..."`
### Whitelisting
```json theme={null}
{
"entities": ["EMAIL_ADDRESS"],
"default_anonymizer": {
"type": "replace",
"new_value": ""
},
"allow_list": ["support@company.com", "info@company.com"]
}
```
**Input**: `"Contact support@company.com or john@example.com"`
**Output**: `"Contact support@company.com or "`
## Behavior
* **Fail-open**: If the plugin encounters an error, the original messages are returned unmodified
* **Multi-message support**: Processes all messages in the request independently
* **Preserves structure**: Maintains message format (text, arrays, multimodal content)
* **Debug output**: Returns detailed redaction information when enabled in Gateway UI
* **No blocking**: Always allows requests to proceed (unlike Detection Plugin)
## Use Cases
1. **Privacy compliance**: Ensure AI requests don't expose customer PII to third-party models
2. **Audit logs**: Redact sensitive data before logging requests for compliance
3. **Multi-tenant systems**: Prevent cross-tenant data leakage in shared AI infrastructure
4. **Development/testing**: Sanitize production data for use in development environments
5. **Selective visibility**: Show last 4 digits of credit cards while masking the rest
***
## Configuration Schema
```json theme={null}
{
"type": "object",
"title": "PII Text Redaction Plugin Configuration",
"$schema": "http://json-schema.org/draft-07/schema#",
"properties": {
"context": {
"type": "array",
"items": {
"type": "string"
},
"title": "Context Words",
"description": "Additional context words to improve detection accuracy."
},
"entities": {
"type": "array",
"items": {
"type": "string"
},
"title": "Entity Types",
"examples": [
[
"EMAIL_ADDRESS",
"PHONE_NUMBER",
"SSN"
]
],
"description": "List of PII entity types to redact. Supported types include: PERSON, EMAIL_ADDRESS, PHONE_NUMBER, CREDIT_CARD, SSN, IBAN_CODE, IP_ADDRESS, etc. If not specified, all detected entities will be redacted."
},
"language": {
"type": "string",
"title": "Language",
"default": "en",
"examples": [
"en",
"es",
"de",
"fr"
],
"description": "Language code for text analysis."
},
"deny_list": {
"type": "array",
"items": {
"type": "string"
},
"title": "Deny List",
"examples": [
[
"confidential"
]
],
"description": "List of terms/patterns that should always be redacted."
},
"allow_list": {
"type": "array",
"items": {
"type": "string"
},
"title": "Allow List",
"examples": [
[
"example@company.com"
]
],
"description": "List of terms/patterns that should not be redacted."
},
"score_threshold": {
"type": "number",
"title": "Score Threshold",
"default": 0.5,
"maximum": 1,
"minimum": 0,
"description": "Minimum confidence score (0-1) required to redact an entity."
},
"default_anonymizer": {
"type": "object",
"title": "Default Anonymizer",
"default": {
"type": "replace",
"new_value": ""
},
"required": [
"type"
],
"properties": {
"type": {
"enum": [
"replace",
"redact",
"hash",
"mask",
"encrypt"
],
"type": "string",
"title": "Type",
"description": "Type of anonymization: 'replace' (substitute with value), 'redact' (remove), 'hash' (one-way hash), 'mask' (partial masking), 'encrypt' (reversible encryption)"
},
"from_end": {
"type": "boolean",
"title": "Mask from End",
"description": "Mask from end instead of beginning for 'mask' type"
},
"new_value": {
"type": "string",
"title": "New Value",
"description": "Replacement value for 'replace' type (e.g., '')"
},
"masking_char": {
"type": "string",
"title": "Masking Character",
"description": "Character to use for 'mask' type (default: *)"
},
"chars_to_mask": {
"type": "number",
"title": "Characters to Mask",
"description": "Number of characters to mask for 'mask' type"
}
},
"description": "Default anonymization method to apply to all detected entities."
},
"entity_anonymizers": {
"type": "object",
"title": "Entity-Specific Anonymizers",
"examples": [
{
"CREDIT_CARD": {
"type": "hash"
},
"PHONE_NUMBER": {
"type": "mask",
"from_end": true,
"masking_char": "X",
"chars_to_mask": 4
},
"EMAIL_ADDRESS": {
"type": "replace",
"new_value": ""
}
}
],
"description": "Entity-specific anonymization methods. Keys are entity types (e.g., 'PHONE_NUMBER', 'EMAIL_ADDRESS')",
"additionalProperties": {
"type": "object",
"required": [
"type"
],
"properties": {
"type": {
"enum": [
"replace",
"redact",
"hash",
"mask",
"encrypt"
],
"type": "string",
"title": "Type"
},
"from_end": {
"type": "boolean",
"title": "Mask from End"
},
"new_value": {
"type": "string",
"title": "New Value"
},
"masking_char": {
"type": "string",
"title": "Masking Character"
},
"chars_to_mask": {
"type": "number",
"title": "Characters to Mask"
}
}
}
}
},
"description": "Configuration for the Presidio-based text redaction plugin that detects and redacts PII from text messages."
}
```
***
## Supported Phases
* **Request Phase**: Supports processing during the REQUEST phase
* **Response Phase**: Supports processing during the RESPONSE phase
* **Log Phase**: Supports processing during the LOG phase
# Regex Detect
Source: https://docs.datawizz.ai/public-plugins/regex-detect
Scans messages for regex patterns and rejects or redacts matches. Useful for filtering sensitive information like emails, phone numbers, or credit card numbers.
The Regex Detect plugin scans user and assistant messages for specified regex patterns and can either reject requests containing matches or redact the matched content.
## Features
* **Flexible Pattern Matching**: Configure multiple regex patterns with optional flags
* **Selective Scanning**: Choose to scan only the latest message or all messages in the conversation
* **Reject or Redact**: Either block requests containing sensitive data or automatically redact it
* **Custom Messages**: Define custom rejection messages for different patterns
* **Configurable Redaction**: Use static text or character repetition for redaction
## Use Cases
* Filter sensitive information (emails, phone numbers, credit cards)
* Enforce content policies by blocking specific patterns
* Automatically redact PII (Personally Identifiable Information)
* Prevent prompt injection attacks by detecting suspicious patterns
## Configuration Options
### `patterns` (required)
Array of pattern objects to scan for. Each pattern object contains:
* `regex` (string, required): The regular expression pattern to match
* `flags` (string, optional): Regex flags (e.g., 'gi' for global case-insensitive). Default: 'g'
* `rejectMessage` (string, optional): Custom message to return when this pattern is matched
### `scanAllMessages` (boolean)
* `true`: Scans all messages in the array
* `false`: Only scans the latest message
* Default: `false`
### `redactMatches` (boolean)
* `true`: Redacts matched patterns instead of rejecting the request
* `false`: Rejects the request if a match is found
* Default: `false`
### `redactionText` (string)
Text to replace redacted patterns with. Default: `[REDACTED]`
### `redactCharacters` (string)
If set, replaces each match with this character repeated to match the length of the original text. Overrides `redactionText` if both are set.
## Example Configurations
### Block Email Addresses
```json theme={null}
{
"patterns": [
{
"regex": "\\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\\.[A-Z|a-z]{2,}\\b",
"flags": "gi",
"rejectMessage": "Email addresses are not allowed in requests"
}
],
"scanAllMessages": false,
"redactMatches": false
}
```
### Redact Phone Numbers
```json theme={null}
{
"patterns": [
{
"regex": "\\b\\d{3}[-.]?\\d{3}[-.]?\\d{4}\\b",
"flags": "g"
}
],
"scanAllMessages": true,
"redactMatches": true,
"redactionText": "[PHONE]"
}
```
### Redact Credit Cards with Character Masking
```json theme={null}
{
"patterns": [
{
"regex": "\\b\\d{4}[\\s-]?\\d{4}[\\s-]?\\d{4}[\\s-]?\\d{4}\\b",
"flags": "g"
}
],
"scanAllMessages": true,
"redactMatches": true,
"redactCharacters": "*"
}
```
### Multiple Patterns
```json theme={null}
{
"patterns": [
{
"regex": "\\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\\.[A-Z|a-z]{2,}\\b",
"flags": "gi",
"rejectMessage": "Email addresses are not allowed"
},
{
"regex": "\\b\\d{3}-\\d{2}-\\d{4}\\b",
"flags": "g",
"rejectMessage": "Social Security Numbers are not allowed"
}
],
"scanAllMessages": false,
"redactMatches": false
}
```
## Response Behavior
### When Rejecting (redactMatches: false)
* Returns `reject: true` with a rejection reason
* Uses custom `rejectMessage` if provided, otherwise a default message
* No messages are modified
### When Redacting (redactMatches: true)
* Returns modified messages array with redacted content
* Original request continues through the pipeline
* Debug logs indicate how many matches were redacted
## Common Regex Patterns
* **Email**: `\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Z|a-z]{2,}\b`
* **US Phone**: `\b\d{3}[-.]?\d{3}[-.]?\d{4}\b`
* **Credit Card**: `\b\d{4}[\s-]?\d{4}[\s-]?\d{4}[\s-]?\d{4}\b`
* **SSN**: `\b\d{3}-\d{2}-\d{4}\b`
* **IP Address**: `\b\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}\b`
* **URL**: `https?://[^\s]+`
## Notes
* Patterns are tested in order; the first match determines the rejection message
* Invalid regex patterns are logged in debug output but don't fail the request
* The 'g' (global) flag is recommended for finding all matches
* When using `redactCharacters`, the replacement length always matches the original
***
## Configuration Schema
```json theme={null}
{
"type": "object",
"title": "Regex Detect Configuration",
"$schema": "http://json-schema.org/draft-07/schema#",
"required": [
"patterns"
],
"properties": {
"patterns": {
"type": "array",
"items": {
"type": "object",
"required": [
"regex"
],
"properties": {
"flags": {
"type": "string",
"title": "Regex Flags",
"default": "g",
"description": "Optional regex flags (e.g., 'gi' for global case-insensitive)"
},
"regex": {
"type": "string",
"title": "Regular Expression",
"description": "The regex pattern to match (e.g., \\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\\.[A-Z|a-z]{2,}\\b for emails)"
},
"rejectMessage": {
"type": "string",
"title": "Custom Reject Message",
"description": "Custom message to return when this pattern is matched"
}
}
},
"title": "Regex Patterns",
"minItems": 1,
"description": "List of regex patterns to scan for"
},
"redactMatches": {
"type": "boolean",
"title": "Redact Matches",
"default": false,
"description": "If true, redacts matched patterns instead of rejecting the request"
},
"redactionText": {
"type": "string",
"title": "Redaction Text",
"default": "[REDACTED]",
"description": "Text to replace redacted patterns with"
},
"scanAllMessages": {
"type": "boolean",
"title": "Scan All Messages",
"default": false,
"description": "If true, scans all messages. If false, only scans the latest message"
},
"redactCharacters": {
"type": "string",
"title": "Redact Characters",
"description": "If set, replaces each match with this character repeated to match the original length. Overrides redactionText"
}
},
"description": "Configuration for the Regex Detect plugin"
}
```
# Video Input Processing
Source: https://docs.datawizz.ai/public-plugins/video-input-processing
Transform videos into LLM-compatible content by extracting frames, audio, and transcripts. Supports YouTube, TikTok, direct URLs, and works with any vision-enabled LLM.
Make it easy to send videos to LLMs and have them reason about video content by automatically extracting frames, audio, and transcripts.
## Overview
Many LLMs are limited to processing text, and in some cases images and audio. This plugin extends native LLM capabilities by pre-processing videos into content that LLMs can understand. The plugin automatically detects video URLs in your messages and converts them into image frames, audio tracks, and transcripts that can be processed by vision-enabled and audio-enabled language models.
This processing is compatible with **all LLMs** - you can select any LLM you want to use for reasoning about the video content.
## Features
* **Source Flexibility**: Supports downloading videos from URLs, including services like YouTube, Vimeo, TikTok and more
* **Proxy Support**: Can use residential proxies automatically to download videos from geo-restricted services
* **Frame Extraction**: Extract frames at configurable intervals (by FPS or total frame count)
* **Timestamp Overlay**: Burn timestamps into frames for temporal reasoning. Useful for identifying when events occur in videos (format: HH:MM:SS.ss)
* **Audio Support**: Extract and include audio from videos (**requires LLMs that support audio input**)
* **Transcription**: Generate transcripts using OpenAI's `gpt-4o-transcribe` model
* **Video Trimming**: Process only specific segments using start/end offsets
* **Resolution Control**: Adjust frame resolution to control LLM processing costs
## Installation
1. Add the plugin to your Datawizz endpoint configuration
2. Set the endpoint URL to: `https://your-service-url/plugin/video`
3. Configure the Authorization header with your secret token:
* Header name: `Authorization`
* Header value: `Bearer YOUR_SECRET_TOKEN`
4. Optionally configure default settings (see Configuration below)
## Configuration
You can specify configurations to control how the video is processed. All options are optional and have sensible defaults:
### Sampling Options
| Parameter | Type | Description | Default |
| --------------- | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -------------------------- |
| `sample_fps` | integer | Number of frames to extract per second | `1` |
| `sample_frames` | integer | Total number of frames to extract. If supplied, will be used instead of FPS, and the frames will be extracted equally spaced throughout the video | None |
| `detail_level` | string | Resolution quality for frames: `"low"` (512px), `"medium"` (768px), or `"high"` (1024px). Most LLMs charge based on image resolution, so use this to control costs | None (original resolution) |
### Visual Options
| Parameter | Type | Description | Default |
| -------------------- | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | --------------- |
| `burn_timestamps` | boolean | Whether to burn timestamps into the frames. Useful for reasoning about events in the video. Timestamps have the format HH:MM:SS.ss | `true` |
| `timestamp_location` | string | Position of timestamps on frames: `"bottom-left"`, `"bottom-right"`, `"top-left"`, `"top-right"`. Use this to position the timestamp where it's less likely to hide important visual information | `"bottom-left"` |
### Audio & Transcription Options
| Parameter | Type | Description | Default |
| --------------------- | ------- | --------------------------------------------------------------------------------------------------------------- | ------------------ |
| `include_audio` | boolean | If true, will send the video's audio track to the LLM (**Can only be used with LLMs that support audio input**) | `false` |
| `include_transcript` | boolean | If true, will generate and send the video's transcript to the LLM using OpenAI `gpt-4o-transcribe` | `false` |
| `transcript_language` | string | Language code for transcript generation (e.g., `"en"`, `"es"`, `"fr"`) | None (auto-detect) |
### Download Options
| Parameter | Type | Description | Default |
| --------------- | ------- | -------------------------------------------------------------- | ------------------------------ |
| `use_proxy` | boolean | Whether to use a residential proxy to download the video | `false` |
| `proxy_country` | string | Country code for proxy location (e.g., `"us"`, `"uk"`, `"de"`) | None (global) |
| `filetype` | string | File type of the video (`mp4`, `webm`, `mov`, `avi`, etc.) | None (inferred from mime type) |
### Trimming Options
| Parameter | Type | Description | Default |
| -------------- | ------- | --------------------------------------------------------- | ------------------- |
| `start_offset` | integer | Process the video from a specific timestamp (in seconds) | `0` |
| `end_offset` | integer | Process the video until a specific timestamp (in seconds) | None (end of video) |
## Usage
Send video attachments as part of a message to the LLM (similar to sending images):
### Example: Video with Text
**Input Message:**
```json theme={null}
{
"role": "user",
"content": [
{
"type": "text",
"text": "Can you describe what's in the video below? I have included some frames from the video with timestamps burned in, as well as the transcript from the video"
},
{
"type": "video_url",
"video_url": {
"url": "https://www.youtube.com/watch?v=R9RRtMCdmSY",
"sample_fps": 1
}
}
]
}
```
**What happens:**
The plugin automatically replaces the `video_url` content with image frames (and optionally transcript text). Your prompt should explain to the LLM that these are frames from the video, and that the transcript is a transcription of the audio.
**Output to LLM:**
```json theme={null}
{
"role": "user",
"content": [
{
"type": "text",
"text": "Can you describe what's in the video below? I have included some frames from the video with timestamps burned in, as well as the transcript from the video"
},
{
"type": "image_url",
"image_url": {
"url": "data:image/jpeg;base64,..."
}
},
{
"type": "image_url",
"image_url": {
"url": "data:image/jpeg;base64,..."
}
},
{
"type": "text",
"text": "[transcript text here]"
}
]
}
```
## Supported Video Sources
* Direct video file URLs (`.mp4`, `.avi`, `.mov`, `.mkv`, `.webm`)
* YouTube videos
* TikTok videos
* Any video URL that can be downloaded
### Prompting Tips
**Important:** Your prompt should explain to the LLM what content it's receiving:
* Mention that you're providing frames from a video
* If using timestamps, explain that timestamps are burned into the frames
* If including a transcript, mention that it's a transcription of the audio
**Pro Tip:** You can combine video processing with other LLM features like structured output to generate structured insights from videos. If using timestamps, you can use structured output for event identification in videos.
## Message Format Requirements
The plugin ONLY processes structured multimodal content with explicit `video_url` type. Plain string URLs like `"content": "https://example.com/video.mp4"` will NOT be processed.
Videos must be in this format:
```json theme={null}
{
"type": "video_url",
"video_url": {
"url": "https://...",
"sample_fps": 1 // optional inline config
}
}
```
Or simply:
```json theme={null}
{
"type": "video_url",
"video_url": "https://..."
}
```
## Example Configuration
```json theme={null}
{
"sample_fps": 2,
"detail_level": "medium",
"burn_timestamps": true,
"timestamp_location": "bottom-right",
"include_transcript": true,
"transcript_language": "en"
}
```
This configuration will:
* Extract 2 frames per second
* Use medium resolution (768px longest side) to control LLM costs
* Add timestamps in the bottom-right corner
* Include a transcript in English using OpenAI's transcription service
## Performance Notes
* Processing time depends on video length and configuration
* Higher `sample_fps` or `sample_frames` values increase processing time
* Transcription requires audio extraction and may add significant processing time
* Frame resolution affects LLM processing costs - most LLMs charge based on image resolution
* The plugin gracefully handles errors - if processing fails, the original message is preserved
***
## Configuration Schema
```json theme={null}
{
"type": "object",
"title": "Video Processing Plugin Configuration",
"$schema": "http://json-schema.org/draft-07/schema#",
"properties": {
"filetype": {
"type": "string",
"title": "File Type",
"default": null,
"examples": [
"mp4",
"avi",
"mov",
"mkv",
"webm"
],
"description": "Expected video file format (auto-detected if not specified)"
},
"use_proxy": {
"type": "boolean",
"title": "Use Proxy",
"default": false,
"description": "Enable proxy for downloading videos (useful for geo-restricted content)"
},
"end_offset": {
"type": "integer",
"title": "End Offset",
"default": null,
"minimum": 0,
"description": "End time in seconds (for trimming video before processing)"
},
"sample_fps": {
"type": "integer",
"title": "Sample FPS",
"default": 1,
"maximum": 30,
"minimum": 1,
"description": "Number of frames to extract per second of video"
},
"detail_level": {
"enum": [
"low",
"medium",
"high"
],
"type": "string",
"title": "Detail Level",
"default": null,
"examples": [
"medium"
],
"description": "Resolution quality for extracted frames"
},
"start_offset": {
"type": "integer",
"title": "Start Offset",
"default": null,
"minimum": 0,
"description": "Start time in seconds (for trimming video before processing)"
},
"include_audio": {
"type": "boolean",
"title": "Include Audio",
"default": false,
"description": "Extract and include audio from the video"
},
"proxy_country": {
"type": "string",
"title": "Proxy Country",
"default": null,
"examples": [
"us",
"uk",
"de",
"fr",
"jp"
],
"description": "Country code for proxy location (e.g., 'us', 'uk', 'de'). Only used if use_proxy is true"
},
"sample_frames": {
"type": "integer",
"title": "Sample Frames",
"default": null,
"minimum": 1,
"description": "Total number of frames to extract (overrides sample_fps if set)"
},
"burn_timestamps": {
"type": "boolean",
"title": "Burn Timestamps",
"default": true,
"description": "Overlay timestamp on each frame showing the time in the video"
},
"include_transcript": {
"type": "boolean",
"title": "Include Transcript",
"default": false,
"description": "Generate and include a transcript of the video audio"
},
"timestamp_location": {
"enum": [
"bottom-left",
"bottom-right",
"top-left",
"top-right"
],
"type": "string",
"title": "Timestamp Location",
"default": "bottom-left",
"description": "Position of burned-in timestamps on frames"
},
"transcript_language": {
"type": "string",
"title": "Transcript Language",
"default": null,
"examples": [
"en",
"es",
"fr",
"de",
"ja"
],
"description": "Language code for transcript generation (e.g., 'en', 'es', 'fr')"
}
},
"description": "Configuration for the video processing plugin that converts video URLs into LLM-compatible image frames",
"additionalProperties": false
}
```
***
## Supported Phases
* **Request Phase**: Supports processing during the REQUEST phase
# Quickstart
Source: https://docs.datawizz.ai/quickstart
Start routing your LLM requests through Datawizz to collect and organize your data.
This quick getting-started tutorial will take you though the steps to start routing your LLM requests through Datawizz to collect and organize your data. Once you start collecting data on Datawizz, you can start analyzing your LLM usage, train specialized models and protect your AI models from abuse...
For this tutorial, we'll focus on code that uses the OpenAI SDK, like in the following code snippet (see both Python and Javascript tabs below):
```python openai_connection.py theme={null}
from openai import OpenAI
client = OpenAI(
api_key="sk-your_openai_api_key", # <--- your datawizz project API key
base_url="https://gw.datawizz.app/**************/openai/v1", # <--- your datawizz project base URL
)
response = client.chat.completions.create(
model="********", # <--- this is the DataWizz Endpoit you are routing to,
messages=[
{
"role": "user",
"content": [
{"type": "text", "text": "What's the speed of light?"},
],
}
]
)
```
```js openai_connection.js theme={null}
import OpenAI from "openai";
const openai = new OpenAI({
apiKey: "sk-your_openai_api_key", // <--- your datawizz project API key
baseUrl: "https://gw.datawizz.app/**************/openai/v1", // <--- your datawizz project base URL
});
const completion = await openai.chat.completions.create({
model: "********", // <--- this is the DataWizz Endpoit you are routing to
messages: [
{ role: "user", content: "Write a haiku about recursion in programming." },
],
});
```
# 1 - Setup your project
As a first step, you need to create a project on Datawizz. If you haven't already, [signup for Datawizz](https://www.datawizz.app/)\[[https://www.datawizz.app/auth/signup](https://www.datawizz.app/auth/signup)]. Once in the dashboard, click "+ New Project" to create a new project.
# 2 - Add a provider
Once you have a project, you need to connect it to your existing LLM moedel provider - like OpenAI. Providers are different platform that provide AI models you can call from your app - like OpenAI, Anthropic or AWS.
In your project, head to the providers section and click add provider:
![Add Provider]()
In the new provider menu, select your provider type, and add your API key and base URL (note not to add the `/v1` at the end of the base URL):
![Add Provider]()
Once you add the provider, you should also add all the different models you want to use from that provider. For instance, if you're using OpenAI, you can add the `gpt-4o` model to your provider:
![Add Provider Model]()
# 3 - Create an endpoint
Next up, you need to create an endpoint to route your LLM requests through. Endpoints are the integration medium between your app and the AI models. When your app calls an AI model - it'll call an endpoint. The endpoint defines the rules for routing the request to different models, and any policies that should be applied to the request (like caching or input screening).
In this examole we'll have a very simple endpoint that routes all requests to the `gpt-4o` model. Head to the endpoints section and click create endpoint:
![Create Endpoint]()
Once you have created an endpoint, open it (click Manage) and click Add Upstream. Upstreams are the different moedls & providers you want to route requests to. Every upstream can have a weight (priority) and conditions to determine how requests get routed.
In this case, we'll add the `gpt-4o` model from the OpenAI provider and leave the weight as 1 and the conditions empty - as we only have one upstream, all requests will be routed to it:
![Add Upstream]()
# 4 - Configure your app
Now that you have an endpoint set up to route requests and a provider to provide the models, you can start using the endpoint in your app. In the endpoint view, you can see the changes you need to make in your code to start using the endpoint:
![Code Snippet]()
Note you'll need to change three items:
| Item | Description |
| --------- | -------------------------------------------------------------------------------------- |
| base\_url | Need to set up the DW base URL to start sending traffic to Datawizz |
| api\_key | Need to input your datawizz API key (you can manage your API keys in project settings) |
| model\_id | Need to set the model ID to the ID of the endpoint you created |
That's it - your requests will now be routed to Datawizz. The dashboard will show your your LLM analytics, and you can see individual logs in the logs section.
# 5 - Next steps
Once you have your LLM requests routed through Datawizz, you can start analyzing your LLM usage, train specialized models and protect your AI models from abuse. Check out the other sections of the documentation to learn more about the Datawizz platform and how to use it to manage your AI models.
# Get Started with Self Hosting
Source: https://docs.datawizz.ai/self-host/get-started
Deploy Datawizz on your own infrastructure using Docker Compose
Self hosting is an enterprise-only feature. Please contact us at [enterprise@datawizz.ai](mailto:enterprise@datawizz.ai) to get started.
Datawizz provides a Docker Compose file for running the entire platform on a single machine or cloud instance. This is the recommended initial deployment for testing and proof-of-concept.
## Hardware Requirements
At a minimum, the training component requires a machine with an **A100**, **H100**, or **H200** GPU.
In cloud environments, we recommend the following instance types as a minimum:
| Cloud Provider | Instance Type |
| -------------- | -------------------------- |
| AWS | `p5.4xlarge` |
| Azure | `Standard_NC40ads_H100_v5` |
| GCP | `a2-ultragpu-1g` |
## Scaling the Deployment
Typically, we scale the deployment by breaking out several components from the single instance into separate instances or managed services, in this order:
Scale out ClickHouse to a dedicated instance for data redundancy and performance.
Run the AI Gateway, Request Logger, and Evaluator Runner on separate instances for horizontal scaling.
Deploy inference workers separately for easier horizontal scaling and autoscaling.
## Quick Start
```bash theme={null}
# 1. Add hostname to /etc/hosts (required - 'localhost' doesn't work)
echo "127.0.0.1 datawizz.local" | sudo tee -a /etc/hosts
# 2. Configure environment
cd self-deploy
cp .env.example .env
# Edit .env with your values (especially passwords and JWT_SECRET)
# 3. Start services
docker compose up -d # Minimal (app + supabase)
docker compose --profile clickhouse up -d # With local ClickHouse
docker compose --profile gateway up -d # With AI Gateway + Logger + Evaluator
docker compose --profile studio up -d # With Supabase Studio
docker compose --profile full up -d # Everything
# 4. Access
open http://datawizz.local # Datawizz App
open http://datawizz.local:3001 # Supabase Studio (if enabled)
open http://datawizz.local/gw # AI Gateway (if enabled)
```
`localhost` cannot be used due to Docker networking constraints. The app container needs to reach the host machine, which requires a resolvable hostname mapped via `/etc/hosts`.
## Architecture
```mermaid theme={null}
graph TD
Users([External Users])
Caddy{{"Caddy (80/443)
Reverse Proxy"}}
Studio["Studio (:3001)
(optional)"]
App["App (/*)"]
Kong{{"Kong (/sbapi/)"}}
Gateway["AI Gateway (/gw/)"]
Evaluator["Evaluator Runner (/eval/)"]
Supabase["Supabase Services
(Auth, REST, Realtime)"]
Postgres[(PostgreSQL
Supabase DB)]
Logger["Request Logger"]
CHLogging[(ClickHouse
logging)]
ClickHouse[(ClickHouse
optional)]
EvalWorker["Evaluation Worker
(optional)"]
TrainWorker["Training Worker
(optional)"]
Users --> Caddy
Users --> Studio
Caddy --> App
Caddy --> Kong
Caddy --> Gateway
Caddy --> Evaluator
Kong --> Supabase
Supabase --> Postgres
Gateway --> Logger
Evaluator --> Logger
Logger --> CHLogging
CHLogging --> ClickHouse
App --> ClickHouse
ClickHouse --- EvalWorker
ClickHouse --- TrainWorker
classDef proxy fill:#f6a609,stroke:#c47f00,color:#000
classDef app fill:#5E00FF,stroke:#4600cc,color:#fff
classDef gateway fill:#0ea5e9,stroke:#0284c7,color:#fff
classDef db fill:#16a34a,stroke:#15803d,color:#fff
classDef worker fill:#e11d48,stroke:#be123c,color:#fff
classDef user fill:#6b7280,stroke:#4b5563,color:#fff
class Users user
class Caddy,Kong proxy
class App,Supabase,Studio app
class Gateway,Evaluator,Logger gateway
class Postgres,ClickHouse,CHLogging db
class EvalWorker,TrainWorker worker
```
## Profiles
Docker Compose profiles allow you to selectively enable optional services:
| Profile | Services | Use Case |
| ------------ | ---------------------------------------------------------- | ------------------------------------------- |
| (none) | Core Supabase + App | Minimal deployment with external ClickHouse |
| `clickhouse` | + ClickHouse | Local ClickHouse instance |
| `gateway` | + AI Gateway, Request Logger, Evaluator Runner, ClickHouse | LLM inference proxy with logging |
| `studio` | + Supabase Studio | Database management UI |
| `workers` | + Evaluation/Training Workers | Local GPU workers (requires NVIDIA GPU) |
| `full` | All services | Complete local deployment |
### Usage Examples
```bash theme={null}
# Minimal: App + Supabase (using external ClickHouse)
docker compose up -d
# With local ClickHouse
docker compose --profile clickhouse up -d
# With AI Gateway (includes ClickHouse for logging)
docker compose --profile gateway up -d
# With Supabase Studio for database management
docker compose --profile studio up -d
# Multiple profiles
docker compose --profile clickhouse --profile studio up -d
# Everything (requires NVIDIA GPU for workers)
docker compose --profile full up -d
# Using environment variable
COMPOSE_PROFILES=clickhouse,studio docker compose up -d
```
## Configuration
### Required Environment Variables
Edit `.env` and configure at minimum:
| Variable | Description |
| ------------------- | --------------------------------- |
| `POSTGRES_PASSWORD` | PostgreSQL password |
| `JWT_SECRET` | JWT signing secret (min 32 chars) |
| `ANON_KEY` | Supabase anonymous API key |
| `SERVICE_ROLE_KEY` | Supabase service role key |
### Generating Secure Values
```bash theme={null}
# Generate JWT_SECRET (32+ characters)
openssl rand -base64 32
# Generate SECRET_KEY_BASE (64+ characters)
openssl rand -base64 64
# Generate random tokens
openssl rand -hex 16
```
### External Services
To use external managed services instead of local containers:
**External ClickHouse** (see [Scaling Out ClickHouse](/self-host/scaling-clickhouse) for full details):
```bash theme={null}
# In .env (don't enable clickhouse profile)
CLICKHOUSE_HOST=your-clickhouse-host.example.com
CLICKHOUSE_PORT=8443
CLICKHOUSE_USER=your_user
CLICKHOUSE_PASSWORD=your_password
CLICKHOUSE_DATABASE=datawizz
```
### AI Gateway
For running the gateway on separate instances, see [Scaling Out the Router & Logger](/self-host/scaling-gateway).
The `gateway` profile enables a complete LLM inference proxy stack:
| Service | Description | URL |
| -------------------- | ---------------------------------------------- | --------------------------------------------- |
| **AI Gateway** | Proxies LLM requests, handles routing, caching | `http://{domain}/gw/{project}/{endpoint}/...` |
| **Request Logger** | Batches and stores inference logs | Internal only |
| **Evaluator Runner** | Runs custom evaluators on inference results | `http://{domain}/eval/...` |
**How it works:**
1. Clients send LLM requests to the AI Gateway
2. Gateway routes requests to configured model providers
3. Responses are logged via the Request Logger to ClickHouse
4. Evaluators can automatically score inference results
**Configuration:**
```bash theme={null}
# Gateway caching
GATEWAY_CACHE_TTL=3600 # Config cache TTL (seconds)
GATEWAY_LOGGER_BATCH_SIZE=10 # Logs per batch
GATEWAY_LOGGER_FLUSH_INTERVAL=5000 # Flush interval (ms)
# Feedback JWT (optional - for secure feedback without API key)
GATEWAY_FEEDBACK_JWT_ENABLED=false
GATEWAY_FEEDBACK_JWT_SECRET=your-secret-min-32-chars
```
### Using Pre-built Images
By default, `docker compose up` builds images from local source code. For production deployments, you can use pre-built images from a container registry instead.
**Configure image sources in `.env`:**
```bash theme={null}
DATAWIZZ_APP_IMAGE=ghcr.io/datawizz/app:v1.0.0
AI_GATEWAY_IMAGE=ghcr.io/datawizz/ai-gateway:v1.0.0
REQUEST_LOGGER_IMAGE=ghcr.io/datawizz/request-logger:v1.0.0
EVALUATOR_RUNNER_IMAGE=ghcr.io/datawizz/evaluator-runner:v1.0.0
EVALUATION_WORKER_IMAGE=ghcr.io/datawizz/evaluation-worker:v1.0.0
TRAINING_WORKER_IMAGE=ghcr.io/datawizz/training-worker:v1.0.0
```
**Pull and run without building:**
```bash theme={null}
# Pull the pre-built images
docker compose pull
# Start services without building
docker compose up -d --no-build
# Or combine into one command
docker compose pull && docker compose up -d --no-build
```
If you omit `--no-build`, Docker Compose will attempt to build from source even if the image variables are set.
## URL Routing
Caddy routes requests based on URL path:
| Path | Destination | Description |
| ---------- | ---------------- | -------------------------------------- |
| `/sbapi/*` | Kong → Supabase | Supabase API (auth, database, storage) |
| `/gw/*` | AI Gateway | LLM inference proxy (gateway profile) |
| `/eval/*` | Evaluator Runner | Evaluator testing (gateway profile) |
| `/*` | Datawizz App | Main application (catch-all) |
**Direct access (not via Caddy):**
* Supabase Studio: `http://{domain}:3001` (studio profile)
## Production Deployment
### Enable HTTPS
For production, set your domain in `.env`:
```bash theme={null}
DOMAIN=datawizz.yourdomain.com
```
Caddy will automatically provision SSL certificates via Let's Encrypt.
### Security Checklist
* Change all default passwords in `.env`
* Generate secure `JWT_SECRET` (min 32 chars)
* Generate new `ANON_KEY` and `SERVICE_ROLE_KEY`
* Set `DOMAIN` to your actual domain
* Review `DISABLE_SIGNUP` setting
* Configure SMTP for email delivery
## Common Operations
### View Logs
```bash theme={null}
# All services
docker compose logs -f
# Specific service
docker compose logs -f datawizz-app
# Multiple services
docker compose logs -f datawizz-app kong auth
```
### Restart Services
```bash theme={null}
# Single service
docker compose restart datawizz-app
# All services
docker compose restart
```
### Update Services
```bash theme={null}
# Pull latest images and restart
docker compose pull
docker compose up -d
```
### Upgrading / Re-running Migrations
Database migrations run automatically on first startup. They won't re-run on subsequent `docker compose up` calls (Docker caches completed containers).
**When to re-run migrations:**
* After pulling a new version with database schema changes
* After changing ClickHouse connection settings in `.env`
* If migrations failed and you've fixed the issue
**How to re-run migrations:**
```bash theme={null}
# Force re-run all migrations
docker compose up -d --force-recreate supabase-migrations clickhouse-migrations
# Or re-run just PostgreSQL migrations
docker compose up -d --force-recreate supabase-migrations
# Check migration logs
docker compose logs supabase-migrations
docker compose logs clickhouse-migrations
```
### Check Service Health
```bash theme={null}
docker compose ps
```
### Access Database
```bash theme={null}
# PostgreSQL
docker exec -it supabase-db psql -U postgres
# ClickHouse (if enabled)
docker exec -it datawizz-clickhouse clickhouse-client
```
## Backup & Restore
### Quick Backup
```bash theme={null}
# PostgreSQL
docker exec supabase-db pg_dump -U postgres -d postgres -F c | \
gzip > backups/postgres-$(date +%Y%m%d-%H%M%S).dump.gz
# File storage
tar -czvf backups/storage-$(date +%Y%m%d-%H%M%S).tar.gz ./volumes/storage
```
## Troubleshooting
### Services won't start
1. Check logs: `docker compose logs`
2. Verify `.env` file exists and has required values
3. Check Docker daemon is running
### Database connection errors
1. Wait for database to be healthy: `docker compose ps`
2. Check PostgreSQL logs: `docker compose logs db`
3. Verify `POSTGRES_PASSWORD` in `.env`
### App shows "initializing" or errors
1. Migrations may still be running - wait a few seconds
2. Check migration logs: `docker compose logs supabase-migrations`
3. Verify Kong is healthy: `docker compose logs kong`
### Workers fail to start
1. Workers require NVIDIA GPU with Docker GPU support
2. Verify GPU is available: `nvidia-smi`
3. Check Docker GPU runtime: `docker run --rm --gpus all nvidia/cuda:11.0-base nvidia-smi`
### Gateway services not responding
1. Check all gateway services are healthy: `docker compose ps | grep -E "gateway|logger|evaluator"`
2. Restart Caddy to pick up new routes: `docker compose restart caddy`
3. Test internal connectivity: `docker exec datawizz-caddy wget -qO- http://ai-gateway:3000/health`
4. Check gateway logs: `docker compose logs ai-gateway request-logger evaluator-runner`
### SSL certificate issues (production)
1. Ensure domain DNS points to your server
2. Check Caddy logs: `docker compose logs caddy`
3. Ports 80 and 443 must be accessible from the internet
# Scaling Out ClickHouse
Source: https://docs.datawizz.ai/self-host/scaling-clickhouse
Run ClickHouse on a separate instance for data redundancy and performance
Datawizz uses ClickHouse to store inference logs, feedback signals, and other data items. By default, the Docker Compose setup runs ClickHouse locally via the `clickhouse` profile. For production deployments, we recommend moving ClickHouse to a dedicated instance or managed service for better data redundancy and independent scaling.
## Option 1: Standalone ClickHouse via Docker
Run ClickHouse on a separate machine using Docker:
```bash theme={null}
docker run -d \
--name clickhouse-server \
--ulimit nofile=262144:262144 \
-v clickhouse_data:/var/lib/clickhouse/ \
-v clickhouse_logs:/var/log/clickhouse-server/ \
-p 8123:8123 \
-p 9000:9000 \
-e CLICKHOUSE_DB=datawizz \
-e CLICKHOUSE_USER=your_user \
-e CLICKHOUSE_PASSWORD=your_secure_password \
-e CLICKHOUSE_DEFAULT_ACCESS_MANAGEMENT=1 \
clickhouse/clickhouse-server
```
This exposes:
* **Port 8123** — HTTP interface (used by the Datawizz app and logging services)
* **Port 9000** — Native TCP interface (used by the PostgreSQL FDW and migrations)
In production, restrict network access to these ports. Only the Datawizz instance and any services that need to query ClickHouse should be able to reach them.
## Option 2: Managed ClickHouse Service
You can use a managed ClickHouse provider such as [ClickHouse Cloud](https://clickhouse.com/cloud) or any other hosted ClickHouse offering. Ensure the service exposes both HTTP and native TCP interfaces, and note the connection credentials for the next step.
## Updating the Main Instance
Once your external ClickHouse is running, update the `.env` file on your main Datawizz instance:
### 1. Set the ClickHouse connection variables
```bash theme={null}
# Point to the external ClickHouse instance
CLICKHOUSE_HOST=your-clickhouse-host.example.com
CLICKHOUSE_PORT=8123
CLICKHOUSE_NATIVE_PORT=9000
CLICKHOUSE_USER=your_user
CLICKHOUSE_PASSWORD=your_secure_password
CLICKHOUSE_DATABASE=datawizz
```
| Variable | Description | Default |
| ------------------------ | ------------------------------------------------ | ------------------------------ |
| `CLICKHOUSE_HOST` | Hostname or IP of the external ClickHouse server | `clickhouse` (local container) |
| `CLICKHOUSE_PORT` | HTTP port | `8123` |
| `CLICKHOUSE_NATIVE_PORT` | Native TCP port (used by FDW and migrations) | `9000` |
| `CLICKHOUSE_USER` | ClickHouse username | `default` |
| `CLICKHOUSE_PASSWORD` | ClickHouse password | `default` |
| `CLICKHOUSE_DATABASE` | Database name | `default` |
### 2. Disable the local ClickHouse profile
Do **not** use the `clickhouse` or `full` profile when starting services. Instead, start with only the profiles you need:
```bash theme={null}
# Without local ClickHouse (uses external)
docker compose up -d
# With gateway (gateway will also use the external ClickHouse)
docker compose --profile gateway up -d
```
If you previously ran with the `clickhouse` profile, the local container may still exist. You can remove it with `docker compose --profile clickhouse down`.
### 3. Run database migrations
The ClickHouse migrations need to run against your external instance. You can trigger them manually:
```bash theme={null}
docker compose up -d --force-recreate clickhouse-migrations
```
This uses the same `CLICKHOUSE_HOST`, `CLICKHOUSE_USER`, `CLICKHOUSE_PASSWORD`, and `CLICKHOUSE_DATABASE` env vars from your `.env` file. Check the logs to confirm success:
```bash theme={null}
docker compose logs clickhouse-migrations
```
### 4. Restart services
Restart the Datawizz app and any gateway services so they pick up the new ClickHouse connection:
```bash theme={null}
docker compose restart datawizz-app
# If using the gateway profile
docker compose --profile gateway restart request-logger
```
## Verifying the Connection
You can verify the connection by checking the health of the Datawizz app and confirming data flows into the external ClickHouse:
```bash theme={null}
# Check app health
docker compose ps datawizz-app
# Connect to the external ClickHouse and verify tables exist
docker run --rm clickhouse/clickhouse-client \
--host your-clickhouse-host.example.com \
--port 9000 \
--user your_user \
--password your_secure_password \
--query "SHOW TABLES FROM datawizz"
```
# Scaling Out the Router & Logger
Source: https://docs.datawizz.ai/self-host/scaling-gateway
Run the AI Gateway, Request Logger, and Evaluator Runner on separate instances for horizontal scaling
The AI Gateway (router), Request Logger, and Evaluator Runner are lightweight services that handle incoming LLM requests, log them, and run evaluations. By default they run inside the main Docker Compose setup via the `gateway` profile. For production, you can break them out to separate instances for horizontal scaling and maximum uptime.
## Architecture Overview
When scaled out, the gateway services run independently and connect back to the main Datawizz instance for configuration:
```mermaid theme={null}
graph LR
Client([LLM Clients])
subgraph External Gateway Instances
GW1["AI Gateway #1"]
GW2["AI Gateway #2"]
Logger1["Request Logger"]
Eval1["Evaluator Runner"]
end
subgraph Main Datawizz Instance
App["Datawizz App"]
Kong{{"Kong (Supabase API)"}}
CH[(ClickHouse)]
end
Client --> GW1
Client --> GW2
GW1 --> Logger1
GW2 --> Logger1
Eval1 --> Logger1
Logger1 --> CH
GW1 -- "fetch config" --> Kong
GW2 -- "fetch config" --> Kong
classDef gateway fill:#0ea5e9,stroke:#0284c7,color:#fff
classDef app fill:#5E00FF,stroke:#4600cc,color:#fff
classDef db fill:#16a34a,stroke:#15803d,color:#fff
classDef proxy fill:#f6a609,stroke:#c47f00,color:#000
classDef user fill:#6b7280,stroke:#4b5563,color:#fff
class Client user
class GW1,GW2,Logger1,Eval1 gateway
class App app
class Kong proxy
class CH db
```
## Running the AI Gateway
The AI Gateway fetches its configuration (projects, endpoints, prompts) from the Datawizz API via Supabase. It caches configurations in-memory and uses Supabase Realtime for instant cache invalidation when configurations change in the dashboard.
### Docker Run
```bash theme={null}
docker run -d \
--name datawizz-gateway \
-p 3000:3000 \
-e DW_API_URL="http://your-datawizz-host/sbapi" \
-e DW_API_KEY="your-service-role-key" \
-e DW_CACHE_TTL="3600" \
-e DW_LOGGER_URL="http://your-logger-host:3000" \
-e DW_LOGGER_API_KEY="your-service-role-key" \
-e DW_LOGGER_BATCH_SIZE="10" \
-e DW_LOGGER_FLUSH_INTERVAL_MS="5000" \
-e DWGW_PORT="3000" \
-e DWGW_HOST="0.0.0.0" \
-e DWGW_CORS_ENABLED="true" \
-e DWGW_CORS_ALLOW_ORIGIN="*" \
-e DWGW_OPTIONS_ENABLED="true" \
ghcr.io/datawizzai/ai-gateway-cf:latest
```
### Environment Variables
| Variable | Description | Default |
| ------------------------------ | -------------------------------------------------------------------- | --------- |
| `DW_API_URL` | URL to the Datawizz Supabase API (e.g. `http://datawizz-host/sbapi`) | Required |
| `DW_API_KEY` | Supabase service role key | Required |
| `DW_CACHE_TTL` | Config cache TTL in seconds | `3600` |
| `DW_LOGGER_URL` | URL of the Request Logger service | Required |
| `DW_LOGGER_API_KEY` | API key for the logger | Required |
| `DW_LOGGER_BATCH_SIZE` | Number of logs per batch | `10` |
| `DW_LOGGER_FLUSH_INTERVAL_MS` | Flush interval in milliseconds | `5000` |
| `DWGW_PORT` | Port to listen on | `3000` |
| `DWGW_HOST` | Host to bind to | `0.0.0.0` |
| `DWGW_CORS_ENABLED` | Enable CORS headers | `false` |
| `DWGW_CORS_ALLOW_ORIGIN` | Allowed CORS origins | `*` |
| `DWGW_OPTIONS_ENABLED` | Handle OPTIONS preflight requests | `false` |
| `DWGW_FEEDBACK_JWT_ENABLED` | Enable JWT tokens for feedback auth | `false` |
| `DWGW_FEEDBACK_JWT_SECRET` | Secret for signing feedback JWTs (min 32 chars) | — |
| `DWGW_FEEDBACK_JWT_EXPIRATION` | Feedback JWT expiration in seconds | `86400` |
### Cache Invalidation
The gateway subscribes to Supabase Realtime for automatic cache invalidation. When you update a project, endpoint, or prompt in the Datawizz dashboard, all gateway instances invalidate their cached data within seconds.
If the Realtime connection fails, the gateway continues operating normally and cached entries expire based on the TTL setting.
### Health Check
The gateway exposes a health endpoint at `/health`:
```bash theme={null}
curl http://your-gateway-host:3000/health
# {"status": "ok"}
```
## Running the Request Logger
The Request Logger batches inference logs and writes them to ClickHouse. It also forwards results to the Evaluator Runner for automated scoring.
```bash theme={null}
docker run -d \
--name datawizz-request-logger \
-p 3000:3000 \
-v logger-data:/app/data \
-e PORT="3000" \
-e DATA_DIR="/app/data" \
-e SUPABASE_URL="http://your-datawizz-host/sbapi" \
-e SUPABASE_SERVICE_ROLE_KEY="your-service-role-key" \
-e CLICKHOUSE_URL="http://your-clickhouse-host:8123" \
-e CLICKHOUSE_DATABASE="datawizz" \
-e CLICKHOUSE_USERNAME="your_user" \
-e CLICKHOUSE_PASSWORD="your_password" \
-e DISABLE_AUTH="false" \
-e EVALUATOR_RUNNER_API_URL="http://your-evaluator-host:3000" \
-e EVALUATOR_RUNNER_API_KEY="your-evaluator-key" \
ghcr.io/datawizzai/logging-service:latest
```
| Variable | Description | Default |
| --------------------------- | ---------------------------------- | ----------- |
| `PORT` | Port to listen on | `3000` |
| `DATA_DIR` | Directory for local data/buffer | `/app/data` |
| `SUPABASE_URL` | Datawizz Supabase API URL | Required |
| `SUPABASE_SERVICE_ROLE_KEY` | Supabase service role key | Required |
| `CLICKHOUSE_URL` | ClickHouse HTTP URL | Required |
| `CLICKHOUSE_DATABASE` | ClickHouse database name | `default` |
| `CLICKHOUSE_USERNAME` | ClickHouse username | `default` |
| `CLICKHOUSE_PASSWORD` | ClickHouse password | `default` |
| `DISABLE_AUTH` | Disable auth for internal networks | `true` |
| `EVALUATOR_RUNNER_API_URL` | Evaluator Runner URL | — |
| `EVALUATOR_RUNNER_API_KEY` | Evaluator Runner API key | — |
## Running the Evaluator Runner
The Evaluator Runner executes custom evaluators on inference results.
```bash theme={null}
docker run -d \
--name datawizz-evaluator-runner \
-p 3000:3000 \
-e PORT="3000" \
-e ENVIRONMENT="production" \
-e JWT_SECRET="your-jwt-secret" \
-e API_KEYS="your-evaluator-key" \
-e SUPABASE_URL="http://your-datawizz-host/sbapi" \
-e SUPABASE_SERVICE_KEY="your-service-role-key" \
ghcr.io/datawizzai/evaluator-runner-cf:latest
```
| Variable | Description | Default |
| ---------------------- | ---------------------------------------------------- | -------- |
| `PORT` | Port to listen on | `3000` |
| `ENVIRONMENT` | Environment name | `local` |
| `JWT_SECRET` | JWT secret (must match main instance) | Required |
| `API_KEYS` | Comma-separated API keys for service-to-service auth | Required |
| `SUPABASE_URL` | Datawizz Supabase API URL | Required |
| `SUPABASE_SERVICE_KEY` | Supabase service role key | Required |
## Updating the Main Instance
After deploying the gateway services externally, update the `.env` on the main Datawizz instance so the app knows where to find them:
```bash theme={null}
# Point the app to the external gateway and evaluator
NEXT_PUBLIC_GATEWAY_BASE_URL=http://your-gateway-host:3000
NEXT_PUBLIC_EVALUATOR_TEST_API_URL=http://your-evaluator-host:3000
```
Then disable the `gateway` profile and restart the app:
```bash theme={null}
# Start without the gateway profile (services are external now)
docker compose up -d
# Or restart just the app if already running
docker compose restart datawizz-app
```
If you had previously used the `gateway` profile, stop and remove the local gateway containers: `docker compose --profile gateway down`.
## Horizontal Scaling
You can run multiple instances of the AI Gateway behind a load balancer for high availability. Each instance independently fetches and caches configuration from the Datawizz API, so no shared state is needed between gateway nodes.
A typical setup:
1. Run 2+ AI Gateway containers across separate machines
2. Place them behind a load balancer (e.g. AWS ALB, Nginx, Caddy)
3. All gateway instances point to the same Request Logger (or a logger cluster)
4. The Request Logger writes to ClickHouse
The gateway is stateless and lightweight, so it scales horizontally with minimal resource overhead.
# Scaling Out Inference Workers
Source: https://docs.datawizz.ai/self-host/scaling-inference
Deploy inference workers on separate instances for horizontal scaling
This page is coming soon. Inference worker documentation is currently being prepared.
Models trained on Datawizz can be deployed inside the main Datawizz machine, or deployed separately for easier horizontal scaling. We provide separate Docker images for inference which can run as independent servers with autoscaling.
Check back soon for detailed instructions on running inference workers outside the main Docker Compose setup.