GautamKishore commited on
Commit
34b82db
Β·
verified Β·
1 Parent(s): 3864724

Model card: ARTH V2 Risk++ release (metrics, quickstarts, limitations) + discovery frontmatter

Browse files
Files changed (1) hide show
  1. README.md +95 -2
README.md CHANGED
@@ -16,8 +16,46 @@ tags:
16
  - tiny
17
  - open-source
18
  - onnx-runtime
 
 
 
 
 
 
19
  pipeline_tag: text-classification
20
  library_name: pico-type
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
21
  ---
22
  <div align="center">
23
 
@@ -48,6 +86,61 @@ Classifies any content from raw bytes: **coarse type Β· modality Β· subtype Β· c
48
  - **~18ms inference** on CPU via ONNX Runtime
49
  - **CLI, Python API, Gradio Space, MCP server** β€” ready to use
50
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
51
  ## πŸ“Š Evaluation
52
 
53
  ### Overall Accuracy (v2 β€” trained on real data)
@@ -182,8 +275,8 @@ All tiers share the same backbone; only the final linear projection layers diffe
182
 
183
  | Platform | Link | Notes |
184
  |----------|------|-------|
185
- | **HuggingFace Space** | [eulogik/pico-type](https://huggingface.co/spaces/eulogik/pico-type) | Gradio web UI, no GPU needed |
186
- | **HuggingFace Model** | [eulogik/pico-type](https://huggingface.co/eulogik/pico-type) | ONNX models + export metadata |
187
  | **GitHub** | [eulogik/pico-type](https://github.com/eulogik/pico-type) | Source code, training, paper |
188
  | **PyPI** | `pip install picotype` | Python package |
189
  | **ONNX Runtime** | Use with onnxruntime.js | Browser/Node.js deployment |
 
16
  - tiny
17
  - open-source
18
  - onnx-runtime
19
+ - content-moderation
20
+ - risk-detection
21
+ - secrets-detection
22
+ - prompt-injection-detection
23
+ - pii-detection
24
+ - pico-type-arth
25
  pipeline_tag: text-classification
26
  library_name: pico-type
27
+ model-index:
28
+ - name: pico-type ARTH V2 (composite)
29
+ results:
30
+ - task:
31
+ type: text-classification
32
+ name: Risk flagging, 14 labels
33
+ dataset:
34
+ name: ARTH external audit (held-out, n=211, never trained or fit on)
35
+ type: private
36
+ metrics:
37
+ - type: recall
38
+ name: positive-class recall
39
+ value: 0.536
40
+ - type: specificity
41
+ name: specificity
42
+ value: 0.944
43
+ - type: balanced error
44
+ name: balanced error
45
+ value: 0.260
46
+ - task:
47
+ type: text-classification
48
+ name: ToxicChat jailbreak gate (held-out test split)
49
+ dataset:
50
+ name: ToxicChat test (toxicchat0124)
51
+ type: lmsys/toxic-chat
52
+ metrics:
53
+ - type: recall
54
+ name: recall@threshold
55
+ value: 0.824
56
+ - type: accuracy
57
+ name: selective accuracy at 50% coverage
58
+ value: 0.973
59
  ---
60
  <div align="center">
61
 
 
86
  - **~18ms inference** on CPU via ONNX Runtime
87
  - **CLI, Python API, Gradio Space, MCP server** β€” ready to use
88
 
89
+ ## πŸ›‘οΈ ARTH V2 β€” Risk++ (new)
90
+
91
+ **[Try it live β€” pick the `arth` model](https://huggingface.co/spaces/eulogik/pico-type)** Β· [Model files](https://huggingface.co/eulogik/pico-type) (`arth_full_base.onnx`, `risk_thresholds.json`, `temperatures.json`, `arth_final_composite.pt`)
92
+
93
+ Frozen-trunk composite (ft3 student + retrained 14-label **Risk++** head) β€” audit recall up **5Γ— with zero regressions** on every shipped gate:
94
+
95
+ | Metric | Before (ft3) | ARTH V2 (composite) |
96
+ |---|---|---|
97
+ | External-audit positive recall (held-out, never trained/fit on) | 0.107 | **0.536** (60/112) |
98
+ | Balanced error | 0.459 | **0.260** |
99
+ | Specificity | 0.961 | 0.944 |
100
+ | Labels with fitted thresholds | 9 | **14** |
101
+ | Legacy parity | 60/60 | 60/60 |
102
+ | Choice accuracy | 0.737 | 0.737 |
103
+ | Noul abstain | 0.929 | 0.929 |
104
+ | ToxicChat jailbreak recall / sel-acc@50 | 0.846 / 0.970 | **0.824 / 0.973** (gate PASS) |
105
+
106
+ Formerly-dead labels now detected: jwt / ssh_key / email **8/8**, password 0.875, phone 0.750. Artifact: **`arth_full_base.onnx` (11.66 MB, opset 18)** β€” auto-verified (torch-vs-ORT err 1.1e-05, parity 60/60, semantic 300/300, risk flags 11/11); INT8 4.28 MB experimental.
107
+
108
+ **Known limitations (measured):** pii_ssn 0.125, api_key 0.625, jailbreak 0.50 on embedded phrasing; 3/11 hand-picked benign probes still flag (incl. the accepted `print('hello world')`β†’sql); long-document signals dilute in fixed-window pooling; general-classification gates (AG/SST-2/Enron) stay at chance β€” this is a byte-pattern risk flagger, not a general text classifier.
109
+
110
+ ### ARTH quickstart (ONNX)
111
+
112
+ ```python
113
+ import json, math
114
+ import numpy as np, onnxruntime as ort
115
+ from huggingface_hub import hf_hub_download
116
+
117
+ sess = ort.InferenceSession(hf_hub_download("eulogik/pico-type", "arth_full_base.onnx"))
118
+ thrs = json.load(open(hf_hub_download("eulogik/pico-type", "risk_thresholds.json")))
119
+ LABELS = ["api_key","jwt","ssh_key","password","email","phone","prompt_injection",
120
+ "jailbreak","pii_ssn","pii_card","secrets_aws","secrets_github",
121
+ "sql_injection","xss_payload"]
122
+
123
+ raw = open("file.txt","rb").read()[:1024]
124
+ ids = np.zeros(1024, np.int64); ids[:len(raw)] = list(raw)
125
+ mask = np.zeros(1024, bool); mask[:len(raw)] = True
126
+ (logits,) = sess.run(["riskpp_logits"], {
127
+ "input_ids": ids[None,:], "attention_mask": mask[None,:],
128
+ "opt_embs": np.zeros((1,4,96), np.float32),
129
+ "struct_feats": np.zeros((1,14), np.float32),
130
+ "act_stats": np.zeros((1,4), np.float32)})
131
+ probs = [1/(1+math.exp(-x)) for x in logits[0]]
132
+ print({l: round(p,4) for l,p in zip(LABELS, probs) if p >= thrs[l]}) # fired flags
133
+ ```
134
+
135
+ ### ARTH quickstart (live Space API)
136
+
137
+ ```python
138
+ from gradio_client import Client
139
+ c = Client("eulogik/pico-type")
140
+ out = c.predict("aws_access_key_id = AKIAIOSFODNN7EXAMPLE", "arth", api_name="/handle_classify")
141
+ print(out[7]) # Risk++ (14) tab: probs + [FLAG] markers
142
+ ```
143
+
144
  ## πŸ“Š Evaluation
145
 
146
  ### Overall Accuracy (v2 β€” trained on real data)
 
275
 
276
  | Platform | Link | Notes |
277
  |----------|------|-------|
278
+ | **HuggingFace Space** | [eulogik/pico-type](https://huggingface.co/spaces/eulogik/pico-type) | Gradio web UI, no GPU needed β€” ARTH V2 + Risk++ (14) by default |
279
+ | **HuggingFace Model** | [eulogik/pico-type](https://huggingface.co/eulogik/pico-type) | ONNX models + ARTH V2 artifacts (`arth_full_base.onnx`, thresholds, temps, ckpt) |
280
  | **GitHub** | [eulogik/pico-type](https://github.com/eulogik/pico-type) | Source code, training, paper |
281
  | **PyPI** | `pip install picotype` | Python package |
282
  | **ONNX Runtime** | Use with onnxruntime.js | Browser/Node.js deployment |