PimoLee5/qwen2.5-7b-cuad-qlora

Qwen2.5-7B-Instruct · CUAD contract-clause QLoRA adapter

A LoRA adapter for Qwen2.5-7B-Instruct that identifies contract clause types and answers "does this excerpt contain clause X, and what does it say?" with verbatim quotes from the excerpt. Trained with QLoRA on CUAD and evaluated before vs after on held-out contracts for accuracy, hallucination and calibration.

Not legal advice. A research and portfolio model. It can miss clauses, mislabel them, or misquote. Always read the contract.

Code, data pipeline and evaluation: https://github.com/raghavpli515/Legal-contract-LLM-fine-tuning-with-evaluation

Demo video: https://youtu.be/hcN9TmknxCs

Results

400 held-out items from 50 CUAD test contracts never seen in training. All runs use the same 4-bit quantization, greedy decoding and scorer. Brackets are 95% bootstrap CIs.

Metric	Base (zero-shot)	Base (3-shot)	This adapter
Clause classification accuracy ↑	61.5% [55.5, 67.5]	63.5% [57.0, 69.5]	80.0% [74.5, 85.5]
Clause classification macro-F1 ↑	0.584	0.590	0.795
Q&A presence accuracy ↑	70.5% [64.0, 76.5]	81.5% [76.0, 86.5]	94.0% [90.5, 97.0]
Missed-clause rate ↓	57.0%	34.0%	5.0%
Q&A calibration error (ECE) ↓	0.283	0.164	0.026
Hallucination rate, after manual audit ↓	1.0% [0.0, 2.5]	1.0% [0.0, 2.5]	3.5% [1.5, 6.5]
Hallucination rate, raw ↓	2.0% [0.5, 4.0]	4.5% [2.0, 7.5]	5.0% [2.5, 8.0]

"3-shot" gives the base model three worked examples of the same clause type in the prompt (no training). It helps Q&A partway and barely moves classification.

Trade-off. Fine-tuning fixed recall and calibration: the base model says "not present" for most clauses that are there, usually with near-certainty. The cost is more wrong-clause-type answers on contracts without the clause. Those answers still quote real contract text, so the quote can be checked. Every flagged hallucination in all three runs was audited by hand; see the repository's results/audit.json.

Intended use

• Locating and quoting candidate clauses in English commercial-contract excerpts (~1,200 tokens), for a human to review.
• Studying fine-tuning effects on hallucination and calibration.

Out of scope: legal advice or decisions, whole-contract review without retrieval, non-English or non-commercial contracts, and any use where a missed or mislabelled clause has consequences without human review.

How to use

The adapter was trained on a fixed prompt format (system prompt, excerpt, question, JSON answer). Build prompts with legal_ft.prompts from the repository, or reproduce that format exactly; other phrasings will work less well.

import torch
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig

base_id, adapter_id = "Qwen/Qwen2.5-7B-Instruct", "PimoLee5/qwen2.5-7b-cuad-qlora"
bnb = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type="nf4",
                         bnb_4bit_use_double_quant=True, bnb_4bit_compute_dtype=torch.float16)
tok = AutoTokenizer.from_pretrained(base_id)
model = AutoModelForCausalLM.from_pretrained(base_id, quantization_config=bnb, device_map={"": 0})
model = PeftModel.from_pretrained(model, adapter_id)

from legal_ft.prompts import build_qa_messages   # pip install from the GitHub repo
messages = build_qa_messages(excerpt, "Non-Compete")
text = tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tok(text, return_tensors="pt").to(0)
out = model.generate(**inputs, max_new_tokens=384, do_sample=False, repetition_penalty=1.0)
print(tok.decode(out[0, inputs.input_ids.shape[1]:], skip_special_tokens=True))
# {"present": true, "evidence": ["...verbatim quote..."]}

Use greedy decoding with repetition_penalty=1.0. Qwen's default generation config applies a 1.05 repetition penalty to every token in the input, which discourages verbatim quoting. On a 6 GB GPU, see the repository's load_model(..., embed_on_cpu=True).

Training

Data	CUAD v1: 4,500 examples from 407 training contracts (1,500 clause classification, 1,500 Q&A with the clause present, 1,500 matched absent examples). Contract-level split, so no test contract text is seen in training.
Method	QLoRA: 4-bit NF4 base with double quantization; LoRA r=16, alpha=32, dropout 0.05 on all linear layers (40.4M trainable parameters, 0.53% of the model)
Loss	On answer tokens only
Optimizer	Paged AdamW 8-bit, lr 2e-4, cosine schedule, 3% warmup, 1 epoch (282 steps, effective batch 16), fp16
Hardware	1× NVIDIA T4 (Kaggle), 6.2 h
Loss curve	Train 0.118 → 0.036; held-out eval 0.127 → 0.091 → 0.091

Limitations

• Raises recall at the cost of precision: more answers that label a real passage as the wrong clause type (5.0% of absent cases after audit vs 0.0% for the base model).
• Excerpt-level: each answer sees ~1,200 tokens, not a whole contract.
• Small evaluation set (200 items per task); rare clause types have 1–7 test items.
• CUAD covers 510 US commercial contracts in English. It does not always annotate every occurrence of a clause, which affects both training and evaluation labels.
• Single training run and seed.

Citation and licence

Adapter: Apache-2.0, following the base model. Training data: CUAD v1 by The Atticus Project, CC BY 4.0 (Hendrycks et al., 2021, CUAD: An Expert-Annotated NLP Dataset for Legal Contract Review).