imanxch/distilbert-contractnli

DistilBERT ContractNLI

A DistilBERT model fine-tuned to decide whether a contract passage supports, contradicts, or doesn't mention a statement. It is the core model of the Contract Conflict Checker, a tool that helps small businesses and freelancers spot conflicts between an incoming contract and their standard terms.

Model details

• Developed by: Iman Chaudhry
• Model type: Sentence-pair classification (natural language inference), 3 classes
• Language: English
• Finetuned from: distilbert-base-uncased
• License: CC BY-NC-SA 4.0 (inherited from the ContractNLI training data: non-commercial use only)
• Repository: GitHub

Labels

ID	Label	Meaning for contracts
0	entailment	The passage supports the statement
1	neutral	The passage doesn't address the statement ("not mentioned")
2	contradiction	The passage conflicts with the statement

Intended use

• Direct use: given a contract passage (premise) and a statement (hypothesis), predict whether the passage supports, contradicts, or doesn't mention it.
• Downstream use: as the clause-comparison step of a contract review pipeline, flagging possible conflicts for a person to review.
• Out of scope: this model is not legal advice and should not be used to make legal decisions on its own. It was trained mostly on non-disclosure agreements and may perform worse on other contract types.

How to use

from transformers import AutoTokenizer, AutoModelForSequenceClassification
import torch

repo = "imanxch/distilbert-contractnli"
tokenizer = AutoTokenizer.from_pretrained(repo)
model = AutoModelForSequenceClassification.from_pretrained(repo)

labels = ["entailment", "neutral", "contradiction"]
premise = "The Recipient shall return or destroy all Confidential Information within 30 days of termination."
hypothesis = "Receiving Party may retain Confidential Information after termination."

inputs = tokenizer(premise, hypothesis, truncation="only_first", max_length=384, return_tensors="pt")
with torch.no_grad():
    probs = model(**inputs).logits.softmax(dim=-1)[0]
print({name: round(p.item(), 3) for name, p in zip(labels, probs)})

Training

Training happened in two stages of transfer learning.

Stage 1: general contradiction (MultiNLI)

• 100,000 MultiNLI sentence pairs, 2 epochs
• Learning rate 3e-5, batch size 32, max length 128
• Chosen as the best of six tracked experiments (learning rate, sequence length, data size)
• Validation accuracy 77.5%, contradiction F1 0.787 (MultiNLI validation-matched)

Stage 2: contract language (ContractNLI)

• ContractNLI contractnli_a training split (6,819 pairs from real NDAs); premises are the contract passages relevant to each hypothesis
• 3 epochs, learning rate 2e-5, batch size 16, max length 384 (covers over 95% of pairs; only the premise is truncated)
• Standard cross-entropy loss. Class weights were tested but didn't improve contradiction F1.
• The checkpoint was chosen by validation contradiction F1, the most important class for the tool

Training regime: fp16 mixed precision with dynamic padding, on a single NVIDIA T4 GPU (Google Colab). Experiments were tracked with MLflow.

Evaluation

ContractNLI test split.

Model	Accuracy	F1 entailment	F1 neutral	F1 contradiction
Stage 1 only (zero-shot on contracts)	40.5%	0.454	0.428	0.271
This model	82.4%	0.837	0.826	0.760

The zero-shot model over-predicted contradiction: contract hypotheses often contain negations ("shall not…"), which the MultiNLI-trained model had learned to associate with contradiction. Fine-tuning on contracts corrected much of this.

Limitations

• Trained on non-disclosure agreements only; other contract types (service, vendor, employment) may use different language.
• Contradictions make up only about 12% of the training data, so the model sees relatively few examples of the most important class.
• Long passages are truncated at 384 tokens.
• Like other NLI models, it can struggle with numbers and dates, and with exceptions ("except as required by law") that reverse a clause's meaning.

Citation

ContractNLI dataset:

@inproceedings{koreeda-manning-2021-contractnli-dataset,
    title = "{C}ontract{NLI}: A Dataset for Document-level Natural Language Inference for Contracts",
    author = "Koreeda, Yuta and Manning, Christopher",
    booktitle = "Findings of the Association for Computational Linguistics: EMNLP 2021",
    year = "2021",
}