Forge measures and visualizes — it never edits policies for you. Evaluation is a deterministic comparison of expected vs. actual actions (no LLM judging), and policy fixes go through the normal Guard Policy editing and versioning flow.
Test Datasets
A Test Dataset belongs to a Project Guardian (the thing whose policies you’re testing). One Guardian can have several datasets — for example “PII attack patterns,” “normal inputs,” and “edge cases.” Dataset names are unique within their Guardian, and deleting the Guardian deletes its datasets and all their results. Each item in a dataset has two parts:| Part | What it is |
|---|---|
| Input | The content to send to the Guardian — a multi-turn conversation (an ordered array of user / assistant messages). v1.3 supports text only. |
| Expected verdict | The decision the Guardian should return — the final action (PASS / MASK / BLOCK), optionally with the policy codes that should fire and a free-text reason. |
Register a dataset
In the project’s Guardian, open Forge and create a dataset:Guardian Action List
The set of actions this dataset evaluates (e.g. Pass / Mask / Block) — per-class accuracy is computed for these.
Experiments
An Experiment runs one Test Dataset through the Project Guardian with a policy configuration you choose, and scores the results. You configure:| Input | Notes |
|---|---|
| Dataset (+ version) | The dataset fixes the target Guardian, the Goal, and the Action List. Version defaults to latest. |
| Name / description | Optional — unnamed Experiments are auto-named from the process type and policy configuration. |
| Policies | One or more distinct policies, each pinned to one version (defaults to its latest). You can’t pick the same policy twice — to compare versions of one policy, run separate Experiments and compare them side by side. |
| Process type & model config | One process type supported by the Guardian (defaults to the first); the model config is pre-filled from the Guardian’s settings and can be overridden for this run only. |
RUNNING → COMPLETED or ERROR), then shows its results. Results, traces, and scores are kept permanently, so you can revisit any past Experiment.
How results are scored
Every item is judged the moment its response arrives: expected action vs. actual action, right or wrong. The Experiment then reports:- Total Accuracy — items whose action matched, over all items.
- Pass / Mask / Block Accuracy — per expected-action class, for the classes in the dataset’s Action List.
- PASS / FAIL — only if the dataset has a Goal: Total Accuracy ≥ Goal passes.
Where results live
Experiment traces are recorded in Opticon — organized as Guardian › dataset › Experiment — and kept strictly apart from production traffic: they’re tagged with theforge environment (production traces are default), the Experiment name as the session, and the dataset name as the user ID. Your PASS / MASK / BLOCK production metrics are never polluted by test runs, and you can filter to exactly one Experiment’s traces. Starfort’s own Experiment history shows only runs triggered from Forge.
Improve a policy from failures
Forge closes the loop that used to require analyzing production traffic after the fact:Find the failures
In a low-accuracy Experiment, open the failed items (accuracy = 0) and inspect each trace — input, expected action, actual action.
Edit the policy
Fix the Guard Policy and publish a new version — the normal editing and versioning flow.
Permissions
| Action | Required |
|---|---|
| View datasets & Experiments | Project Member or above |
| Register / edit datasets, run Experiments | The test-execute IAM permission |
| Delete datasets | Project Admin or above |
| Edit & apply Guard Policies | Project Admin or above (same as normal policy editing) |
v1.3 Forge is text-only and rule-based by design: file/image datasets, LLM-as-judge evaluation, and automatic policy optimization are out of scope for this version.