lesageethan commited on
Commit
20efa1c
·
verified ·
1 Parent(s): 567ff4f

Update REAME

Browse files
Files changed (1) hide show
  1. README.md +12 -10
README.md CHANGED
@@ -33,9 +33,12 @@ datasets:
33
 
34
  X3Guard-150M is Axiomic Labs' compact prompt guard, fine-tuned from **<a href="https://huggingface.co/AxiomicLabs/GPT-X3-150M" target="_blank">GPT-X3-150M</a>**. It reads a user prompt and returns two independent scores: how likely the prompt is to be **harmful**, and how likely it is to be **adversarial** (a jailbreak or prompt-injection attempt). At 149M parameters it is both the smallest and best-performing model we evaluated that gives both classifications.
35
 
 
 
 
36
  ## Performance
37
 
38
- All X3Guard figures are at the default **Balanced** setting unless stated otherwise. Every model was run by us on the same prompts, and each competitor at its own default decision rule/threshold.
39
 
40
  ### Harm Detection vs Size
41
 
@@ -67,9 +70,9 @@ Granite-Guardian-3.1-2B leads at 0.865, about 17x X3Guard's size, and Qwen3Guard
67
 
68
  ![Mean adversarial F1 vs model size](assets/adversarial_f1_vs_size.png)
69
 
70
- X3Guard 150M ranks fifth, behind Sheltron, ProtectAI's two detectors (both trained on jackhhao, one of the two benchmarks) and DuoGuard 0.5B.
71
 
72
- X3Guard scores ahead of gpt-oss-safeguard-20b, both Llama Prompt Guard 2 models, NeuralTrust prompt-guard-oss-small, IBM's Granite Guardian 3.1 2B and the 4M-parameter BERT-Tiny injection detector.
73
 
74
  | Organization | Model | Parameters | Mean F1 | jackhhao F1 | TrustAIRLab F1 |
75
  | :-- | :-- | --: | --: | --: | --: |
@@ -98,9 +101,8 @@ Among models that output both classifications, no model is ahead of X3Guard 150M
98
  **Notes on the comparison**
99
 
100
  - Mean F1 is the mean across the two benchmarks for each task, weighted equally.
101
- - Parameter counts include embeddings, so they can differ from the size in a model's name: Llama Prompt Guard 2 86M and 22M count only their transformer layers, and their large vocabulary embedding tables bring them to 279M and 71M.
102
  - † gpt-oss-safeguard-20b classifies against a written policy, so its results depend on the policy; we wrote a general policy which aligned to the test approach, defining harmful and adversarial prompts, and used low reasoning effort.
103
- - Llama Guard 4 and gpt-oss-safeguard were run through OpenRouter, and their parameter counts are approximate.
104
  - Granite Guardian 3.1 2B was run in 8-bit, with prompts over 2,048 tokens trimmed to their first 1,536 and last 512 tokens, due to compute limitations. Few prompts in our test set exceed 2,048 tokens.
105
  - Sheltron was scored with its own input format and calibration, using overlapping windows for long prompts as its card recommends.
106
  - Streaming moderators such as Qwen3Guard-Stream are not included, as X3Guard checks complete prompts.
@@ -122,8 +124,6 @@ X3Guard 150M ships with three "strictness" settings. Each applies its own thresh
122
 
123
  ### Strict vs Balanced vs Permissive
124
 
125
- Rates are the mean of the two benchmarks for each head (harm: WildGuardTest and CircleGuardBench; adversarial: jackhhao and TrustAIRLab).
126
-
127
  **Harm head**
128
 
129
  | Setting | Recall (harmful caught) | Harmful missed | Benign passed | Benign flagged | Precision | F1 |
@@ -142,7 +142,7 @@ Rates are the mean of the two benchmarks for each head (harm: WildGuardTest and
142
 
143
  These rates are per head. If your application blocks when either head fires, the combined false-flag rate will be higher.
144
 
145
- Our headline claims use Balanced, the standard 0.5 threshold. The Strict and Permissive thresholds were chosen with these benchmark results in view, so treat their numbers as a guide to the trade-off rather than independent claims, and ideally, test the settings on your own traffic.
146
 
147
  ---
148
 
@@ -171,7 +171,9 @@ Measured on an NVIDIA RTX 4080:
171
  | Median latency, one prompt at a time | 33.3 ms (95th percentile: 43.3 ms) |
172
  | Peak GPU memory, one prompt at a time | 1.02 GB |
173
 
174
- These figures are for the Transformers (PyTorch) version. X3Guard will be much faster once we release GGUF and ONNX versions of the model.
 
 
175
 
176
  ---
177
 
@@ -201,7 +203,7 @@ The AlignmentArena data is currently private. It contains many highly effective
201
 
202
  - **X3-Guard checks prompts only.** It does not assess model responses and does not explain why a prompt was flagged.
203
  - **X3-Guard does not categorise harm types.** Other models like Meta's Llama Guard can categorise harmful prompts into one of 14 categories. X3Guard does not do this.
204
- - **It is evaluated mainly in English.** Future Axiomic Guard model will endeavor to cover more languages.
205
  - **Use it as one layer of defence**, alongside access controls, output filtering, human review and a wider safety policy.
206
  - **Absolutely no warranty.** Use in production at your own risk. Axiomic Labs is not responsible for incorrect outputs of this model. See Apache-2.0.
207
 
 
33
 
34
  X3Guard-150M is Axiomic Labs' compact prompt guard, fine-tuned from **<a href="https://huggingface.co/AxiomicLabs/GPT-X3-150M" target="_blank">GPT-X3-150M</a>**. It reads a user prompt and returns two independent scores: how likely the prompt is to be **harmful**, and how likely it is to be **adversarial** (a jailbreak or prompt-injection attempt). At 149M parameters it is both the smallest and best-performing model we evaluated that gives both classifications.
35
 
36
+ - <a href="https://huggingface.co/spaces/AxiomicLabs/X3Guard-150M-Demo" target="_blank">Click here to try out the model in-browser</a>
37
+ - <a href="https://huggingface.co/AxiomicLabs/X3Guard-150M-ONNX" target="_blank">Click here to see faster ONNX quants of this model.</a>
38
+
39
  ## Performance
40
 
41
+ Throughout this model card, all X3Guard figures are at the default "Balanced" setting unless stated otherwise. Each competitor was benchmarked at its own default decision rule/threshold.
42
 
43
  ### Harm Detection vs Size
44
 
 
70
 
71
  ![Mean adversarial F1 vs model size](assets/adversarial_f1_vs_size.png)
72
 
73
+ In adversarial detection, X3Guard 150M ranks fifth, behind Sheltron, ProtectAI's two detectors (these were trained on jackhhao, one of the two benchmarks) and DuoGuard 0.5B.
74
 
75
+ X3Guard scores ahead of gpt-oss-safeguard-20b, both Llama Prompt Guard 2 models, NeuralTrust prompt-guard-oss-small, IBM's Granite Guardian 3.1 2B and others.
76
 
77
  | Organization | Model | Parameters | Mean F1 | jackhhao F1 | TrustAIRLab F1 |
78
  | :-- | :-- | --: | --: | --: | --: |
 
101
  **Notes on the comparison**
102
 
103
  - Mean F1 is the mean across the two benchmarks for each task, weighted equally.
104
+ - Parameter counts include embeddings, so they can differ from the size in a model's name: Llama Prompt Guard 2 86M and 22M count only their transformer layers, and their large vocabulary embedding tables make them 279M and 71M, respectively.
105
  - † gpt-oss-safeguard-20b classifies against a written policy, so its results depend on the policy; we wrote a general policy which aligned to the test approach, defining harmful and adversarial prompts, and used low reasoning effort.
 
106
  - Granite Guardian 3.1 2B was run in 8-bit, with prompts over 2,048 tokens trimmed to their first 1,536 and last 512 tokens, due to compute limitations. Few prompts in our test set exceed 2,048 tokens.
107
  - Sheltron was scored with its own input format and calibration, using overlapping windows for long prompts as its card recommends.
108
  - Streaming moderators such as Qwen3Guard-Stream are not included, as X3Guard checks complete prompts.
 
124
 
125
  ### Strict vs Balanced vs Permissive
126
 
 
 
127
  **Harm head**
128
 
129
  | Setting | Recall (harmful caught) | Harmful missed | Benign passed | Benign flagged | Precision | F1 |
 
142
 
143
  These rates are per head. If your application blocks when either head fires, the combined false-flag rate will be higher.
144
 
145
+ Our headline claims use Balanced (the standard 0.5 threshold). The Strict and Permissive thresholds were chosen with these benchmark results, so treat their numbers as a guide to the trade-off rather than independent claims, and ideally, test the settings on your own traffic before selecting a setting.
146
 
147
  ---
148
 
 
171
  | Median latency, one prompt at a time | 33.3 ms (95th percentile: 43.3 ms) |
172
  | Peak GPU memory, one prompt at a time | 1.02 GB |
173
 
174
+ These figures are for the Transformers (PyTorch) version.
175
+
176
+ If you are running X3Guard on a CPU, or on individual prompts (rather than batch processing) on a GPU, we recommend the ONNX quants <a href="https://huggingface.co/AxiomicLabs/X3Guard-150M-ONNX" target="_blank">here.</a>
177
 
178
  ---
179
 
 
203
 
204
  - **X3-Guard checks prompts only.** It does not assess model responses and does not explain why a prompt was flagged.
205
  - **X3-Guard does not categorise harm types.** Other models like Meta's Llama Guard can categorise harmful prompts into one of 14 categories. X3Guard does not do this.
206
+ - **X3-Guard is evaluated mainly in English.** Future Axiomic Guard model will endeavor to cover more languages.
207
  - **Use it as one layer of defence**, alongside access controls, output filtering, human review and a wider safety policy.
208
  - **Absolutely no warranty.** Use in production at your own risk. Axiomic Labs is not responsible for incorrect outputs of this model. See Apache-2.0.
209