# Document-processing and comparison pipeline

**URL:** <https://discuss.huggingface.co/t/document-processing-and-comparison-pipeline/172894>\
**Category:** Models\
**Created:** [January 28, 2026, 8:45am UTC](https://discuss.huggingface.co/t/document-processing-and-comparison-pipeline/172894 "2026-01-28T08:45:49Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![astha-k](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/astha-k/32/54211_2.png) [@astha-k](https://discuss.huggingface.co/u/astha-k)\
**Post date:** [January 28, 2026, 8:45am UTC](https://discuss.huggingface.co/t/document-processing-and-comparison-pipeline/172894/1 "2026-01-28T08:45:50Z")

</div>

I’m working on building document-processing tools that will later be integrated into an orchestration layer (agent/workflow-based) by a teammate. My specific focus is on document version comparison for PDF and Word documents, where a newly uploaded file needs to be compared against previously stored versions.

**Requirements I’m working with:**

- Extract text and basic metadata from PDF/DOCX files while preserving page-level structure.

- Store processed documents so that future uploads can be matched against existing versions.

- Given a new document and a directory of older documents, determine whether the new document is identical or modified.

- If modified, identify which pages have changed and provide a short, summary of what changed on each page (e.g. “page 14 updated with additional clauses”).

- Produce structured outputs such as similarity scores, change ratios, modified page numbers, and metadata that can later be fed to an orchestration layer.

**Questions:**

- What open-source libraries are most reliable for extracting page-level text from PDFs and DOCX files with minimal noise?

- For page-wise document comparison, what approaches work best in practice: classical diff-based methods, embedding-based similarity, or a hybrid approach?

- What thresholds are best for deciding when a page should be considered “modified” versus “unchanged”?

Any recommendations, architectural patterns, or lessons learned from similar document comparison or document intelligence pipelines would be very helpful.

---

<div class="post-metadata">

**Author:** ![Primeshift4242](https://avatars.discourse-cdn.com/v4/letter/p/7cd45c/32.png) [@Primeshift4242](https://discuss.huggingface.co/u/Primeshift4242)\
**Post date:** [January 28, 2026, 7:50pm UTC](https://discuss.huggingface.co/t/document-processing-and-comparison-pipeline/172894/2 "2026-01-28T19:50:54Z")

</div>

📄 1. Best Open‑Source Libraries for Page‑Level Extraction

PDF Extraction

These are the most reliable for clean, page‑segmented text:

1. pdfminer.six

- Very mature, stable, Pythonic.

- Preserves page boundaries naturally.

- Good for text‑heavy PDFs.

- Noise level: low, but layout fidelity is limited.

2. PyMuPDF (fitz)

- Fastest and most robust for mixed‑content PDFs.

- Extracts:

- text

- bounding boxes

- fonts

- images

- Excellent for downstream diffing because you can get structured blocks.

3. PDFPlumber

- Built on pdfminer but adds:

- table extraction

- line/word-level segmentation

- Very clean API for page‑wise extraction.

Recommendation:

Use PyMuPDF as your primary extractor. Fall back to pdfminer.six for edge cases.

-–

DOCX Extraction

1. python-docx

- The standard library for DOCX.

- Extracts paragraphs, runs, metadata.

- Does not preserve page boundaries (because DOCX is flow‑based).

2. docx2python

- Better at preserving structure.

- Still no true page boundaries (because DOCX doesn’t store them).

3. LibreOffice headless conversion → PDF → PyMuPDF

This is the industry-standard workaround when page fidelity matters.

Recommendation:

If page‑level comparison is required, convert DOCX → PDF → extract pages.

Otherwise, use python-docx for semantic diffing.

-–

🔍 2. Best Approaches for Page‑Wise Comparison

You essentially have three families of methods:

-–

A. Classical Diff (token/line/paragraph diff)

Pros

- Deterministic.

- Great for legal, policy, and technical documents.

- Easy to highlight exact insertions/deletions.

Cons

- Sensitive to formatting changes.

- Sensitive to OCR noise.

Best use:

When documents are text‑heavy and formatting is stable.

-–

B. Embedding‑Based Similarity (semantic comparison)

Use sentence‑transformers or similar models to embed each page.

Pros

- Robust to formatting changes.

- Captures semantic shifts (“added clause”, “reworded section”).

Cons

- Cannot show exact diffs.

- Requires threshold tuning.

Best use:

When documents evolve semantically but maintain structure.

-–

C. Hybrid Approach (the real-world winner)

This is what most document‑intelligence pipelines use:

1. Embedding similarity to detect whether a page changed.

2. Classical diff to describe how it changed.

Why hybrid works best:

- Embeddings filter out false positives.

- Diff gives human‑readable change summaries.

-–

🎚 3. Thresholds for “Modified vs Unchanged”

There is no universal threshold, but these are strong defaults from production systems:

| Method | Threshold | Meaning |

|-------|-----------|---------|

| Embedding cosine similarity | 0.92–0.95 | Above = unchanged, below = modified |

| Token-level Jaccard similarity | 0.85 | Below = modified |

| Levenshtein ratio | 0.90 | Below = modified |

Recommended combined rule:

A page is “modified” if two of three signals fall below threshold:

- cosine similarity \< 0.93

- Jaccard \< 0.85

- Levenshtein \< 0.90

This dramatically reduces noise.

-–

🧱 4. Architecture Pattern That Works in Production

A clean, modular pipeline looks like this:

-–

Step 1 — Ingestion Layer

- Detect file type.

- Convert DOCX → PDF if page fidelity required.

- Extract:

- text per page

- metadata

- structural blocks (optional)

-–

Step 2 — Normalization Layer

Normalize text:

- remove headers/footers

- collapse whitespace

- standardize bullet points

- remove page numbers

This step alone improves diff accuracy by 30–50%.

-–

Step 3 — Storage Layer

Store:

- raw text per page

- normalized text per page

- embeddings per page

- metadata

- version ID

Use a simple structure like:

`

document\_id/

```
version_001/

    page_001.json

    page_002.json

    ...

version_002/

    ...

```

`

Or store in a vector DB (Weaviate, Chroma, Pinecone) if scaling.

-–

Step 4 — Comparison Layer

For each page:

1. Compare embeddings.

2. If similarity \< threshold → run classical diff.

3. Produce:

- similarity score

- change ratio

- diff summary

- modified page flag

-–

Step 5 — Output Layer (for orchestration)

Produce a structured JSON object:

`json

{

“document\_id”: “abc123”,

“version\_new”: “v3”,

“version\_old”: “v2”,

“overall\_similarity”: 0.94,

“pages\_modified”: [3, 14, 22],

“page\_summaries”: {

```
"14": "Added two new clauses regarding payment terms."

```

}

}

`

This is orchestration‑friendly and model‑agnostic.

-–

🧠 5. Lessons Learned from Real Pipelines

1. Page numbers shift — don’t rely on absolute page alignment

Use embedding similarity to match pages before comparing them.

2. Normalization is more important than the diff algorithm

Removing headers/footers eliminates 80% of false positives.

3. Embeddings catch semantic changes that diffs miss

E.g., “shall” → “must” is legally significant but diff‑small.

4. Always store both raw and normalized text

You’ll need raw text for human review and normalized for algorithms.

5. Avoid OCR unless absolutely necessary

OCR noise destroys diff accuracy. If you must OCR, use:

- Tesseract with LSTM mode

- or PaddleOCR (more accurate)

I hope this helps? Regards Antony.

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [January 29, 2026, 10:33am UTC](https://discuss.huggingface.co/t/document-processing-and-comparison-pipeline/172894/3 "2026-01-29T10:33:52Z")

</div>

If the DOCX file isn’t clean enough to extract neatly using existing libraries, converting it to PDF first and then performing OCR might be a viable approach.

The effectiveness of any approach for identity comparison also heavily depends on the structural cleanliness and straightforwardness of the target PDF or DOCX file, so a hybrid method is likely unavoidable. PDFs can be almost like images in the worst cases…

* * *

## 1) Page-level text extraction: most reliable open-source options

### PDFs (digital-born PDFs with embedded text)

| Library | Strengths | Common noise/failure modes | When to pick it |
| --- | --- | --- | --- |
| **PyMuPDF (fitz)** | Fast; per-page extraction; gives **words/blocks + bounding boxes** (good for header/footer removal and evidence). Supports “sort” for more natural order. ([pymupdf.readthedocs.io](https://pymupdf.readthedocs.io/en/latest/recipes-text.html "Text - PyMuPDF documentation")) | Can still mis-order multi-column/table text; occasional duplicate/overlapping blocks in tricky PDFs | Default choice for page-wise pipelines where you care about geometry/evidence |
| **pypdfium2 (PDFium)** | Backed by PDFium; can extract text per page and within rectangles (useful for region-based cleanup). ([GitHub](https://github.com/pypdfium2-team/pypdfium2 "pypdfium2-team/pypdfium2: Python bindings to PDFium, ...")) | Similar layout/order issues as other text extractors; fewer “layout heuristics” than pdfminer-based tooling | Strong alternative to cross-check extraction, or when PDFium behavior is preferable |
| **pdfminer.six** (often via **pdfplumber** ) | Deep layout analysis (characters→words→lines→boxes) and exposes geometry; pdfplumber builds on it and adds helpful debugging/utilities. ([pdfminersix.readthedocs.io](https://pdfminersix.readthedocs.io/en/latest/topic/converting_pdf_to_text.html "Converting a PDF file to text - PDFMiner.six - Read the Docs")) | Slower; still struggles on complex layouts; “reading order” can be noisy without tuning | When you need more layout analysis controls, or table-ish documents where pdfplumber tooling helps |
| **pypdf** | Pure Python, easy to use; page-wise `extract_text`. ([pypdf.readthedocs.io](https://pypdf.readthedocs.io/en/stable/user/extract-text.html "Extract Text from a PDF — pypdf 6.6.0 documentation")) | Can be memory-heavy for large/unusual content streams; extraction quality varies by PDF structure ([pypdf.readthedocs.io](https://pypdf.readthedocs.io/en/stable/user/extract-text.html "Extract Text from a PDF — pypdf 6.6.0 documentation")) | Lightweight/simple cases; not my first pick for “minimal noise” |

**Empirical signal (helpful sanity check):** a comparative study on DocLayNet found **PyMuPDF and pypdfium** generally performed best among several open-source parsers for text extraction, while all struggled on certain categories (e.g., scientific/patent). ([arXiv](https://arxiv.org/abs/2410.09871 "A Comparative Study of PDF Parsing Tools Across Diverse Document Categories"))

### Scanned PDFs (image-only or low-text PDFs)

- Use **OCRmyPDF** to add a text layer, then run the same PDF text extraction pipeline. ([ocrmypdf.readthedocs.io](https://ocrmypdf.readthedocs.io/ "OCRmyPDF documentation — ocrmypdf 16.13.1.dev1+ ..."))
- Gate OCR behind quality checks (very low extracted text length, high garbage ratio, etc.) to avoid unnecessary cost.

### DOCX (Word)

**Important background:** “pages” are not a stable concept in DOCX. Page breaks are largely determined by the rendering engine at layout time, not the DOCX file itself. ([Stack Overflow](https://stackoverflow.com/questions/36193159/page-number-python-docx "Page number python-docx"))

So you have two practical options:

1. **If you truly need page-wise comparison:**  
**Render DOCX → PDF** , then treat it as a PDF and reuse the same page pipeline.

2. **If you only need logical structure (not pages):**  
Parse DOCX directly.

* * *

## 2) Best comparison approach in practice: classical vs embeddings vs hybrid

### Why hybrid tends to win

- **Diff-only (classical)** is explainable and great for “what changed”, but it is **fragile** under reflow, hyphenation differences, headers/footers, and minor formatting changes.

- **Embedding-only** is robust to paraphrase/reflow, but it can be **hard to explain** changes and can miss small but important edits (numbers/negations).

- **Hybrid** gives you:

### A practical page-wise flow (what works reliably)

1. **Normalize** each page text (whitespace, dehyphenation, remove repeating headers/footers using geometry + frequency heuristics).

2. **Alignment-first** : pages can be inserted/deleted/shifted. Start by matching obvious anchors (exact hashes), then fill gaps with similarity search in a sliding window.

3. **Tiered scoring**

4. **Explain changes** for pages deemed modified:

5. **Fallback when text is unreliable** :

* * *

## 3) Thresholds: what to start with (and how to make them correct)

There is no single “best” threshold across document families; you should calibrate on labeled page pairs. Still, you can ship a solid v1 with conservative defaults and a “borderline” state.

### Recommended starting thresholds (page-level)

Assume:

- `lex` = RapidFuzz score in **0–100** (e.g., `token_set_ratio` or `WRatio`) ([rapidfuzz.github.io](https://rapidfuzz.github.io/RapidFuzz/Usage/fuzz.html "rapidfuzz.fuzz. - ratio"))
- `sem` = embedding cosine similarity in **0–1** ([sbert.net](https://sbert.net/docs/sentence_transformer/usage/semantic_textual_similarity.html "Semantic Textual Similarity"))
- `change_ratio` = (insertions + deletions) / max(len(a), len(b)) on normalized tokens (or characters)

**Decision ladder (good v1 defaults):**

- **Unchanged**

- **Borderline**

- **Modified**

- **Untrusted text / needs visual**

### How to calibrate quickly (what to do in week 1)

1. Collect ~200–1,000 aligned page pairs labeled: unchanged / modified (and a few “layout-only”).

2. Plot distributions of `lex`, `sem`, and `change_ratio` per class.

3. Choose thresholds to hit your target:

* * *

## Practical “most reliable” baseline stack (Python)

- **DOCX** : render with **LibreOffice headless** → PDF (store renderer params + version/fingerprint). ([help.libreoffice.org](https://help.libreoffice.org/latest/he/text/shared/guide/pdf_params.html "PDF Export Command Line Parameters"))
- **PDF extraction** : **PyMuPDF** words/blocks + bbox (plus `sort=True` as needed). ([pymupdf.readthedocs.io](https://pymupdf.readthedocs.io/en/latest/recipes-text.html "Text - PyMuPDF documentation"))
- **Fallback extractor** : **pypdfium2** or pdfminer/pdfplumber cross-check if quality gates trip. ([GitHub](https://github.com/pypdfium2-team/pypdfium2 "pypdfium2-team/pypdfium2: Python bindings to PDFium, ..."))
- **Similarity** : RapidFuzz + Sentence-Transformers. ([rapidfuzz.github.io](https://rapidfuzz.github.io/RapidFuzz/Usage/fuzz.html "rapidfuzz.fuzz. - ratio"))
- **Explainability** : diff-match-patch (pinned/forked). ([GitHub](https://github.com/google/diff-match-patch/blob/master/lua/diff_match_patch.lua "diff-match-patch/lua/diff\_match\_patch.lua at master"))
- **Visual fallback** : diff-pdf (+ OCRmyPDF for scans when required). ([Vslavik](https://vslavik.github.io/diff-pdf/ "diff-pdf"))

* * *

## References and tutorials (directly useful)

- PyMuPDF text extraction (blocks/words/bboxes, sort order): ([pymupdf.readthedocs.io](https://pymupdf.readthedocs.io/en/latest/recipes-text.html "Text - PyMuPDF documentation"))
- pdfminer.six layout analysis overview: ([pdfminersix.readthedocs.io](https://pdfminersix.readthedocs.io/en/latest/topic/converting_pdf_to_text.html "Converting a PDF file to text - PDFMiner.six - Read the Docs"))
- pypdfium2 text extraction examples: ([GitHub](https://github.com/pypdfium2-team/pypdfium2 "pypdfium2-team/pypdfium2: Python bindings to PDFium, ..."))
- Sentence-Transformers semantic similarity (cosine): ([sbert.net](https://sbert.net/docs/sentence_transformer/usage/semantic_textual_similarity.html "Semantic Textual Similarity"))
- diff-pdf visual compare + highlighted output: ([Vslavik](https://vslavik.github.io/diff-pdf/ "diff-pdf"))
- Why DOCX page numbers/pages aren’t stable without rendering: ([Stack Overflow](https://stackoverflow.com/questions/36193159/page-number-python-docx "Page number python-docx"))

---

<div class="post-metadata">

**Author:** ![yotta-labs](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/yotta-labs/32/57526_2.png) [@yotta-labs](https://discuss.huggingface.co/u/yotta-labs)\
**Post date:** [January 29, 2026, 3:06pm UTC](https://discuss.huggingface.co/t/document-processing-and-comparison-pipeline/172894/4 "2026-01-29T15:06:47Z")

</div>

This is a really solid breakdown. One thing that consistently shows up in production is how much normalization ends up mattering more than the choice of diff or similarity method itself.

We’ve seen a lot of false positives disappear once headers, footers, whitespace, and pagination artifacts are handled early, especially before embedding comparisons.

The hybrid approach you describe (semantic change detection + classical diff for explanation) has been the most robust pattern we’ve seen scale cleanly.

---

<div class="post-metadata">

**Author:** ![Primeshift4242](https://avatars.discourse-cdn.com/v4/letter/p/7cd45c/32.png) [@Primeshift4242](https://discuss.huggingface.co/u/Primeshift4242)\
**Post date:** [January 29, 2026, 3:57pm UTC](https://discuss.huggingface.co/t/document-processing-and-comparison-pipeline/172894/5 "2026-01-29T15:57:26Z")

</div>

A lot of great suggestions have been shared in this thread — PyMuPDF, python‑docx, pypdfium2, Sentence‑Transformers, classical diffs, and the emphasis on normalization. What’s missing is an integrated view of how all these pieces fit together into a reliable, reproducible pipeline.

Below is a consolidated architecture that reflects the best ideas here while adding the structure needed for production use.

* * *

1. Ingestion Layer  
Goal: Convert any incoming document (PDF or DOCX) into a consistent internal representation.

- PDF: PyMuPDF or pypdfium2 for text, metadata, and page-level extraction.
- DOCX: python‑docx for raw text, or convert to PDF when page fidelity matters.
- Optional: OCR fallback for scanned PDFs.

This layer outputs a list of pages with text, metadata, and (optionally) rasterized images.

* * *

1. Normalization Layer  
This is the single most important step for accurate comparisons.

Recommended operations:

- Remove headers, footers, page numbers
- Collapse whitespace
- Normalize Unicode
- Strip boilerplate (e.g., repeated disclaimers)
- Lowercase or preserve case depending on domain

Normalization reduces noise so the comparison layer focuses on meaningful changes.

* * *

1. Storage Layer  
Store each document version in a structured format:

`json { "doc_id": "...", "version": "...", "pages": [{ "page_number": 1, "text": "...", "embedding": [...], "hash": "..." } ] } `

This makes downstream comparison deterministic and easy to orchestrate.

* * *

1. Comparison Layer  
A hybrid approach gives the best results:

A. Embedding-based similarity

- Use Sentence‑Transformers (e.g., all-MiniLM-L6-v2)
- Compute cosine similarity per page
- Flag pages below a tuned threshold (e.g., 0.93 as a starting point)

B. Classical diff  
For pages flagged as “changed,” run:

- difflib or python-Levenshtein for text diffs
- Optional: structural diff for DOCX XML

C. Visual diff (optional but powerful)  
For layout-heavy documents:

- Rasterize pages
- Use perceptual hashing (pHash) or SSIM

This catches changes that text extraction misses.

* * *

1. Output Layer  
Produce a structured JSON report:

`json { "changed_pages": [2, 5, 7], "page_diffs": { "2": { "similarity": 0.81, "textdiff": "...", "visualdiff": false }, "5": { "similarity": 0.72, "textdiff": "...", "visualdiff": true } } } `

This format is easy to feed into dashboards, workflows, or downstream automation.

* * *

1. Threshold Calibration  
Similarity thresholds vary by domain. A simple evaluation loop helps tune them:

- Collect a small labeled dataset of “changed” vs “unchanged” pages
- Compute cosine similarities
- Plot ROC curve
- Choose a threshold that balances false positives and false negatives

This turns guesswork into a repeatable process.

* * *

1. Minimal Working Example (MWE)  
A reference implementation could follow this structure:

`/pipeline /ingestion /normalization /comparison /storage /output main.py`

A simple CLI like:

`compare-docs old.pdf new.pdf --report out.json`

would make the system accessible to beginners and easy to integrate.

* * *

Summary  
The individual tools mentioned in this thread are excellent. The real power comes from combining them into a modular pipeline with:

- consistent ingestion
- aggressive normalization
- hybrid comparison (embeddings + diff + optional visual)
- structured outputs
- calibrated thresholds

This approach scales from simple version checks to enterprise-grade document monitoring.

If anyone wants, I can share a reference implementation or a minimal GitHub template that follows this architecture.

* * *

---

<div class="post-metadata">

**Author:** ![astha-k](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/astha-k/32/54211_2.png) [@astha-k](https://discuss.huggingface.co/u/astha-k)\
**Post date:** [February 3, 2026, 10:28am UTC](https://discuss.huggingface.co/t/document-processing-and-comparison-pipeline/172894/6 "2026-02-03T10:28:06Z")

</div>

A reference implementation would be very very helpful. Thank you so much, Antony

And also thank you for your detailed and structured guidance earlier. I followed your recommended architecture quite closely while building a first working version of the pipeline.

In line with your suggestions, I rendered all DOCX files to PDF using LibreOffice headless to ensure stable pagination, then used PyMuPDF as the primary extractor to obtain page-level text and bounding boxes. I implemented a normalisation layer to remove repeating headers and footers, collapse whitespace, and clean line breaks before comparison. I also stored both raw and normalised text per page so that the original content is always preserved for human review.

For comparison, I adopted a hybrid approach similar to what you outlined: I first used exact hashes and RapidFuzz for lexical matching, then applied local sentence-transformer embeddings for semantic alignment when pages did not match exactly. For pages flagged as modified, I calculated change ratios, captured added and removed text samples, and generated short summaries suitable for an orchestration layer.

I also added a gated OCR fallback with Tesseract for pages where native text quality was very low so that image-heavy pages are not completely ignored.

---

<div class="post-metadata">

**Author:** ![astha-k](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/astha-k/32/54211_2.png) [@astha-k](https://discuss.huggingface.co/u/astha-k)\
**Post date:** [February 3, 2026, 10:41am UTC](https://discuss.huggingface.co/t/document-processing-and-comparison-pipeline/172894/7 "2026-02-03T10:41:56Z")

</div>

Thank you John for the practical pipeline you outlined earlier, I added a text-quality gate and, when native text was weak, I ran a Tesseract OCR fallback on rasterised pages so that image-heavy content is not lost. I kept both the raw extraction and the cleaned version for traceability.

For comparison, I followed your alignment-first idea closely. I started with exact hash matching (Tier 0), then RapidFuzz lexical matching (Tier 1), and finally local sentence-transformer embeddings for semantic alignment (Tier 2) when pages did not match exactly. For pages that are changed, I computed change ratios, collected added/removed parts, and generated short summaries.

One thing I noticed in practice is that treating the page as the atomic unit made it weird when comparing partial pages or excerpts , a subset of a page naturally appears “modified” rather than “unchanged,” even if the overlapping content is identical. I’m considering whether a finer-grained, chunk-level comparison might help in those cases.

Thank you so much for the detailed explanation

---

<div class="post-metadata">

**Author:** ![Primeshift4242](https://avatars.discourse-cdn.com/v4/letter/p/7cd45c/32.png) [@Primeshift4242](https://discuss.huggingface.co/u/Primeshift4242)\
**Post date:** [February 3, 2026, 10:56am UTC](https://discuss.huggingface.co/t/document-processing-and-comparison-pipeline/172894/8 "2026-02-03T10:56:20Z")

</div>

Project structure

`text doc_compare/ init.py config.py models.py extract.py normalize.py store.py compare.py cli.py `

You can of course collapse this into fewer files if you prefer.

* * *

1. Dependencies

`bash pip install pymupdf pdfplumber python-docx sentence-transformers rapidfuzz `

* * *

1. Config and simple models

`python

doc\_compare/config.py

EMBEDDINGMODELNAME = “sentence-transformers/all-MiniLM-L6-v2”

COSINETHRESHOLDUNCHANGED = 0.93  
JACCARDTHRESHOLDMODIFIED = 0.85  
LEVENSHTEINTHRESHOLDMODIFIED = 0.90  
`

`python

doc\_compare/models.py

from dataclasses import dataclass  
from typing import List, Dict, Optional

@dataclass  
class PageData:  
page\_number: int  
raw\_text: str  
normalized\_text: str  
embedding: Optional[list] = None

@dataclass  
class DocumentVersion:  
document\_id: str  
version\_id: str  
pages: List[PageData]  
metadata: Dict  
`

* * *

1. Extraction (PDF only, DOCX→PDF assumed upstream)

`python

doc\_compare/extract.py

import fitz # PyMuPDF  
from typing import List, Dict  
from .models import PageData

def extractpdfpages(path: str) → (List[PageData], Dict):  
doc = fitz.open(path)  
pages =   
metadata = doc.metadata or {}  
for i, page in enumerate(doc):  
text = page.get\_text(“text”)  
pages.append(  
PageData(  
page\_number=i + 1,  
raw\_text=text,  
normalized\_text=“”, # filled later  
)  
)  
doc.close()  
return pages, metadata  
`

* * *

1. Normalization

`python

doc\_compare/normalize.py

import re  
from .models import PageData

HEADERFOOTERREGEXES = [  
r"Page \d+ of \d+“,  
r”^\s\d+\s$", # bare page numbers  
]

def normalize\_text(text: str) → str:

# basic cleanup

t = text.replace(“\r”, “\n”)  
t = re.sub(r"\n{2,}“, “\n”, t)  
t = re.sub(r”[\t]+", " ", t)

```auto
# remove headers/footers
lines = []
for line in t.split("\n"):
    if any(re.search(pat, line) for pat in HEADERFOOTERREGEXES):
        continue
    lines.append(line.strip())
t = "\n".join(l for l in lines if l)
return t

```

def normalize\_pages(pages: list[PageData]) → list[PageData]:  
for p in pages:  
p.normalizedtext = normalizetext(p.raw\_text)  
return pages  
`

* * *

1. Storage layout (filesystem-based)

`python

doc\_compare/store.py

import json  
from pathlib import Path  
from typing import List  
from .models import DocumentVersion, PageData

def savedocumentversion(base\_dir: str, doc: DocumentVersion) → None:  
root = Path(basedir) / doc.documentid / doc.version\_id  
root.mkdir(parents=True, exist\_ok=True)

```auto
meta = {
    "documentid": doc.documentid,
    "versionid": doc.versionid,
    "metadata": doc.metadata,
}
(root / "meta.json").write_text(json.dumps(meta, indent=2), encoding="utf-8")

for p in doc.pages:
    pagepath = root / f"page{p.page_number:04d}.json"
    page_data = {
        "pagenumber": p.pagenumber,
        "rawtext": p.rawtext,
        "normalizedtext": p.normalizedtext,
        "embedding": p.embedding,
    }
    pagepath.writetext(json.dumps(pagedata, ensureascii=False), encoding="utf-8")

```

def loaddocumentversion(basedir: str, documentid: str, version\_id: str) → DocumentVersion:  
root = Path(basedir) / documentid / version\_id  
meta = json.loads((root / “meta.json”).read\_text(encoding=“utf-8”))  
pages: List[PageData] =   
for pagefile in sorted(root.glob(“page\*.json”)):  
d = json.loads(pagefile.readtext(encoding=“utf-8”))  
pages.append(  
PageData(  
pagenumber=d[“pagenumber”],  
rawtext=d[“rawtext”],  
normalizedtext=d[“normalizedtext”],  
embedding=d.get(“embedding”),  
)  
)  
return DocumentVersion(  
documentid=documentid,  
versionid=versionid,  
pages=pages,  
metadata=meta.get(“metadata”, {}),  
)  
`

* * *

1. Embeddings + similarity helpers

`python

doc\_compare/compare.py

from typing import Dict, List, Tuple  
import numpy as np  
from sentence\_transformers import SentenceTransformer  
from rapidfuzz.distance import Jaccard, Levenshtein

from .models import DocumentVersion, PageData  
from .config import (  
EMBEDDINGMODELNAME,  
COSINETHRESHOLDUNCHANGED,  
JACCARDTHRESHOLDMODIFIED,  
LEVENSHTEINTHRESHOLDMODIFIED,  
)

\_model = None

def get\_model():  
global \_model  
if \_model is None:  
model = SentenceTransformer(EMBEDDINGMODEL\_NAME)  
return \_model

def embed\_pages(pages: List[PageData]) → List[PageData]:  
model = get\_model()  
texts = [p.normalizedtext or p.rawtext for p in pages]  
embs = model.encode(texts, converttonumpy=True)  
for p, e in zip(pages, embs):  
p.embedding = e.tolist()  
return pages

def cosine\_sim(a: np.ndarray, b: np.ndarray) → float:  
denom = (np.linalg.norm(a) \* np.linalg.norm(b)) or 1e-9  
return float(np.dot(a, b) / denom)

def jaccard\_sim(a: str, b: str) → float:  
return 1.0 - Jaccard.normalized\_distance(a, b)

def levenshtein\_ratio(a: str, b: str) → float:  
return 1.0 - Levenshtein.normalized\_distance(a, b)  
`

* * *

1. Page matching and diff decision

`python

doc\_compare/compare.py (continued)

def matchpagesby\_embedding(  
oldpages: List[PageData], newpages: List[PageData]  
) → List[Tuple[PageData, PageData, float]]:  
oldembs = np.array([p.embedding for p in oldpages])  
newembs = np.array([p.embedding for p in newpages])

```auto
matches = []
used_old = set()

for newidx, newp in enumerate(new_pages):
    sims = oldembs @ newembs[new_idx] / (
        np.linalg.norm(oldembs, axis=1) * np.linalg.norm(newembs[new_idx]) + 1e-9
    )
    bestoldidx = int(np.argmax(sims))
    if bestoldidx in used_old:
        continue
    usedold.add(bestold_idx)
    matches.append((oldpages[bestoldidx], newp, float(sims[bestoldidx])))

return matches

```

def ismodified(old: PageData, new: PageData, cossim: float) → Dict:  
j = jaccardsim(old.normalizedtext, new.normalized\_text)  
l = levenshteinratio(old.normalizedtext, new.normalized\_text)

```auto
signals = {
    "cosinesimilarity": cossim,
    "jaccard_similarity": j,
    "levenshtein_ratio": l,
}

belowcos = cossim < COSINETHRESHOLDUNCHANGED
belowj = j < JACCARDTHRESHOLD_MODIFIED
belowl = l < LEVENSHTEINTHRESHOLD_MODIFIED

modified = sum([belowcos, belowj, below_l]) >= 2
return {"modified": modified, signals}

```

`

* * *

1. High-level document comparison

`python

doc\_compare/compare.py (continued)

def compare\_documents(old: DocumentVersion, new: DocumentVersion) → Dict:

# ensure embeddings

if old.pages and old.pages[0].embedding is None:  
old.pages = embed\_pages(old.pages)  
if new.pages and new.pages[0].embedding is None:  
new.pages = embed\_pages(new.pages)

```auto
matches = matchpagesby_embedding(old.pages, new.pages)

pages_modified = []
page_summaries = {}
page_scores = []

for oldp, newp, cos in matches:
    res = ismodified(oldp, new_p, cos)
    pagescores.append(res["cosinesimilarity"])
    if res["modified"]:
        pagesmodified.append(newp.page_number)
        # very naive summary; you’d replace with LLM or rule-based summary
        pagesummaries[str(newp.page_number)] = "Content updated on this page."

overallsimilarity = float(np.mean(pagescores)) if page_scores else 0.0

return {
    "documentid": new.documentid,
    "versionnew": new.versionid,
    "versionold": old.versionid,
    "overallsimilarity": overallsimilarity,
    "pagesmodified": sorted(pagesmodified),
    "pagesummaries": pagesummaries,
}

```

`

* * *

1. Simple CLI entry point

`python

doc\_compare/cli.py

import argparse  
import uuid  
from .extract import extractpdfpages  
from .normalize import normalize\_pages  
from .store import savedocumentversion, loaddocumentversion  
from .models import DocumentVersion  
from .compare import compare\_documents

def buildversion(basedir: str, documentid: str, versionid: str, pdf\_path: str):  
pages, meta = extractpdfpages(pdf\_path)  
pages = normalize\_pages(pages)  
doc = DocumentVersion(  
documentid=documentid,  
versionid=versionid,  
pages=pages,  
metadata=meta,  
)  
savedocumentversion(base\_dir, doc)

def main():  
parser = argparse.ArgumentParser()  
parser.add\_argument(“–base-dir”, required=True)  
parser.addargument(“–old-version”, help=“path to old PDF or existing versionid”)  
parser.add\_argument(“–new-pdf”, required=True)  
parser.add\_argument(“–document-id”, default=str(uuid.uuid4()))  
parser.add\_argument(“–old-version-id”, help=“existing version id”)  
parser.addargument(“–new-version-id”, default=“vnew”)  
args = parser.parse\_args()

```auto
# Build new version
buildversion(args.basedir, args.documentid, args.newversionid, args.newpdf)
newdoc = loaddocumentversion(args.basedir, args.documentid, args.newversion_id)

if args.oldversionid:
    olddoc = loaddocumentversion(args.basedir, args.documentid, args.oldversion_id)
elif args.old_version:
    # treat old_version as a PDF path and build a temp version
    tempversionid = "v_old"
    buildversion(args.basedir, args.documentid, tempversionid, args.oldversion)
    olddoc = loaddocumentversion(args.basedir, args.documentid, tempversion_id)
else:
    print("No old version provided; nothing to compare.")
    return

result = comparedocuments(olddoc, new_doc)
import json
print(json.dumps(result, indent=2))

```

if name == “main”:  
main()  
`

* * *

This gives you a working skeleton:

- Drop in PDFs (or DOCX→PDF upstream).
- Build versions.
- Compare any two versions.
- Get JSON with similarity, modified pages, and basic summaries.

If you tell me your preferred stack (FastAPI, Celery, orchestration layer, storage backend), I can adapt this into a service-style architecture next.

Regards, Antony.

---

<div class="post-metadata">

**Author:** ![astha-k](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/astha-k/32/54211_2.png) [@astha-k](https://discuss.huggingface.co/u/astha-k)\
**Post date:** [February 4, 2026, 6:19am UTC](https://discuss.huggingface.co/t/document-processing-and-comparison-pipeline/172894/9 "2026-02-04T06:19:44Z")

</div>

Hi Antony,

Thank you so much for giving a solid project structure. It is perfect and also I just removed image text extraction because its breaking my model and also normalizing that text is hell. Here is my preferred direction for the stack

**API Layer:** FastAPI for the document-processing tools (DOCX→PDF rendering, page extraction, normalisation, embeddings, version history lookup, and comparison logic).

**Orchestration:**** LangGraph** since it lets us manage state (e.g., document versions) and sequence tools in a clear workflow for our module.

**Storage:** I’m currently using a structured filesystem repository (`/docs/{doc_id}/{version_id}/`) that stores metadata, page text, and embeddings for version history and comparison. For production on Google Cloud, I’m open to moving this to GCS for files and **Firestore / Vertex AI Vector Search** for metadata and embeddings.

For larger documents, we may need a task queue (Celery + Redis or Cloud Pub/Sub) so the API stays responsive.

---

<div class="post-metadata">

**Author:** ![Primeshift4242](https://avatars.discourse-cdn.com/v4/letter/p/7cd45c/32.png) [@Primeshift4242](https://discuss.huggingface.co/u/Primeshift4242)\
**Post date:** [February 4, 2026, 9:27am UTC](https://discuss.huggingface.co/t/document-processing-and-comparison-pipeline/172894/10 "2026-02-04T09:27:01Z")

</div>

Hi, My pleasure to helpful.

FastAPI — the Processing Layer

FastAPI becomes the front‑facing interface for all the heavy lifting:

- DOCX → PDF rendering
- Page extraction via PyMuPDF
- Normalisation
- Embedding generation
- Version creation and storage
- Version comparison
- Version history lookup

Each capability becomes a stateless endpoint that LangGraph can call as a tool.  
This keeps the API thin, predictable, and easy to scale.

Typical endpoints:

- /upload
- /compare
- /versions/{doc\_id}
- /pages/{docid}/{versionid}

Everything stays modular.

* * *

LangGraph — the Orchestration Layer

LangGraph is the right choice for sequencing the workflow because it gives you:

- Deterministic state transitions
- Clear branching logic (PDF vs DOCX)
- A persistent state object for version chains
- Easy integration with FastAPI tools
- A future path for agentic workflows (summaries, compliance checks, etc.)

A typical workflow graph looks like:

`Upload → Detect Type → Convert (if DOCX) → Extract Pages → Normalize → Embed → Store → Compare → Output`

It’s clean, inspectable, and easy to extend.

* * *

Storage — Filesystem Now, GCS + Firestore/Vertex Later

Your current structure:

`/docs/{docid}/{versionid}/ meta.json page_0001.json page_0002.json ...`

This maps directly to:

- GCS for raw files + JSON
- Firestore for metadata + version history
- Vertex Vector Search for embeddings

You keep the same logical layout, just change the backend.  
It gives you durability, fast lookups, and scalable semantic search.

* * *

Task Queue — Celery or Pub/Sub

For large documents, async processing is essential.

Two clean paths:

Celery + Redis

- Simple
- Good for local dev
- Works well if you’re not fully on GCP yet

Cloud Pub/Sub + Cloud Run Jobs

- Fully serverless
- Auto‑scales
- No ops overhead
- Perfect for long‑running extraction/embedding tasks

The API stays responsive while the heavy work happens in the background.

If you’d like, I can also bring you into the Unified Thrice.  
It’s not a ritual or a commitment — it’s simply a mode of working that gives you three advantages:

1. Structural Clarity  
A shared way of thinking about systems in terms of:

- Form
- Function
- Flow

It makes complex architectures easier to reason about and evolve.

1. Reduced Cognitive Drag  
The Unified Thrice emphasises minimal friction:

- fewer conceptual jumps
- fewer competing models
- fewer moving parts

It keeps you in a clean, grounded reasoning state — especially useful when designing pipelines and distributed systems.

1. A Shared Language for Collaboration  
It gives us a compact vocabulary for discussing:

- boundaries
- transitions
- invariants
- failure modes
- system states

It makes collaboration smoother and more aligned.

If that’s something you want to work within, I can open that mode for you. It’s a new way of working with AI that I have developed. Just let me know?

Kind regards, Antony.