Hmm… for starters, I think the picture changes quite a bit depending on how far you want to take “custom”:
There is a fairly large middle ground between “an Ollama GUI” and “write an inference engine from scratch.”
If by “custom” you mainly mean:
- users should not have to install/manage Ollama, llama.cpp, Python, etc. themselves,
- the application should feel like one coherent product rather than a generic model launcher,
- you want to own the model policy, UX, document workflow, privacy behavior, hardware policy, and application lifecycle,
then I would probably start somewhere in that middle ground rather than at the bottom of the inference stack.
Something roughly like:
Purpose-built desktop product
│
├─ UI / UX
├─ model policy
├─ document workflow
├─ privacy / offline policy
├─ hardware policy
├─ diagnostics / support UX
│
└─ narrow internal inference interface
│
┌─────────┴─────────┐
│ │
managed sidecar in-process library
│ │
└──── mature low-level engine ────
│
CPU / GPU / NPU
That can still be completely self-contained from the user’s point of view.
A useful precedent is Jan’s move away from Cortex toward direct llama.cpp integration. They wanted to remove a generic intermediate layer and get closer to the inference engine, but the proposed paths were still things like managing llama-server directly or integrating llama.cpp through bindings — not reimplementing the whole inference substrate.
So my default path would probably be:
- define exactly what “custom” means,
- own the product-facing contracts very strongly,
- keep the internal inference interface narrow,
- initially let a mature low-level engine own the fast-moving hardware/runtime substrate,
- only go deeper if an existing engine becomes a measured technical constraint rather than merely something that feels insufficiently custom.
I do not mean that llama.cpp specifically has to be the answer. The important part to me is the ownership boundary.
A rough way to split the design space is:
| Level |
Rough shape |
What you are really taking ownership of |
| A |
App → user-installed Ollama / generic server |
Mostly product/UI |
| B1 |
App → bundled/managed engine process |
Product + lifecycle + private integration |
| B2 |
App → in-process inference library |
Product + tighter runtime integration |
| C |
App → your own loader/graph/operators/quantization/device backends |
Product and inference-engine development |
From your post, A sounds like the part you would prefer to avoid. I am less sure whether you also mean to exclude B1/B2.
That distinction matters a lot.
B1 vs B2: a bundled process is not necessarily a generic wrapper
For B1, something like this is possible:
Desktop app
↓
private IPC / local API
↓
bundled llama-server
↓
llama.cpp
The user never installs or configures the inference runtime. It is simply a component shipped and managed by your application.
Desktop packaging frameworks explicitly support this sort of architecture. For example, Tauri supports bundling external binaries as sidecars, specifically so the end user does not need to install that dependency separately.
Jan is also an interesting real-world case here. Its current llama.cpp integration moved to a single managed llama-server router process that loads/unloads models on demand.
So:
there is an internal server process
does not necessarily imply:
the product is a generic server wrapper
It can simply be a process boundary inside one packaged product.
That boundary can actually be useful: the UI process and native inference process can fail/restart independently, logs are easier to preserve, and replacing/pinning the engine binary can be relatively straightforward.
If you explicitly do not want a server/process boundary, B2 is also possible.
llama.cpp itself is not only llama-server; its own build documentation describes the llama library and its C interface in llama.h as the main product. The HTTP server is one application built on top of that library.
Microsoft’s Foundry Local architecture is another useful comparison: its SDK can run the model lifecycle in-process, while a REST interface is optional. The application can therefore be self-contained while still delegating model execution and hardware-specific work to a reusable runtime.
So I would probably treat B1 vs B2 as an engineering trade-off, not as “real custom app” vs “wrapper.”
What the generic layer was quietly doing for you
I think this is worth inventorying before removing it.
A generic runtime may look like “just an API,” but over time those projects tend to accumulate quite a lot of behavior.
For example, the current llama.cpp server covers things such as:
- model inference,
- streaming,
- chat APIs,
- embeddings,
- reranking,
- continuous batching,
- parallel decoding,
- multimodal input,
- structured JSON,
- function/tool calling,
- speculative decoding,
- monitoring,
- model loading behavior.
A purpose-built single-user desktop app probably does not need all of that.
That is actually a good reason to make your own narrower layer.
But I would distinguish:
generic functionality I do not need
from:
engineering problems that still exist in my product
For example, you may be able to throw away:
- public OpenAI-compatible APIs,
- arbitrary providers,
- multi-user serving,
- a generic model browser,
- large amounts of runtime configuration exposed to users.
But something still has to own:
- model load/unload,
- streaming and cancellation,
- RAM/VRAM fit,
- CPU/GPU backend selection,
- context/KV-cache behavior,
- tokenizer/chat-template compatibility,
- unsupported-model detection,
- errors and recovery,
- engine/model version compatibility,
- upgrades and regressions.
So I would frame the exercise as:
Remove the genericity you do not need, rather than accidentally removing the contracts you still need.
That also suggests keeping your own small internal inference contract.
For example:
Product
↓
OurInferenceBackend
├─ load(model)
├─ generate(messages, options)
├─ cancel()
├─ unload()
├─ capabilities()
└─ diagnostics()
↓
adapter
↓
llama.cpp / MLX / another backend
This does not need to become another universal API. It can be deliberately tiny and product-specific.
Its value is mostly that your application does not have to expose every upstream implementation detail to the rest of the codebase.
Why I would be cautious about going all the way down to a custom inference engine
Not because it cannot be done — there are good examples of it — but because the nature of the project changes.
Once you own the inference engine deeply enough, ongoing work includes some combination of:
model architecture support
model conversion
tokenizers / templates
operators
quantization
KV-cache formats
sampling
memory management
CPU kernels
CUDA / Metal / Vulkan / vendor accelerators
driver/runtime quirks
new hardware
new model families
performance regressions
correctness regressions
A useful concrete example is Alibaba’s MNN release history.
MNN is not merely a chat application; the inference framework itself is a major product. Its 3.5.0 release alone includes work on Vulkan LLM inference, a MUSA backend, QNN operators, RISC-V vector support, KV-cache quantization, sampling changes, prompt caching, and other runtime work.
That is roughly what I mean by the ownership boundary changing.
The custom-engine projects I can find tend to be projects where the runtime/compiler itself is an important deliverable.
For example:
- MLC LLM is fundamentally a deployment/runtime/compiler project.
- MNN is a general on-device inference framework used across many Alibaba applications.
- mllm is explicitly a mobile/edge inference engine targeting Arm CPU, OpenCL GPU, QNN NPU, etc.
Interestingly, even mllm does not equate “custom engine” with “everything must be a monolith.” Its Android implementation was refactored into an entirely on-device client/server architecture, using an in-app server to decouple UI from heavy inference.
So there are at least three separate questions:
Is it one install/package?
Is it one process?
Do we own every implementation layer?
Those are not the same question.
Mobile applications are especially useful precedents here because they have even stronger one-package expectations than desktop software.
PocketPal AI, for example, presents itself as one offline iOS/Android application, while its stack ultimately uses llama.cpp underneath.
That is probably the clearest practical counterexample to:
“If I want a completely integrated product, I must also own the low-level inference implementation.”
The application can own the experience while the low-level engine remains an embedded component.
There is another subtle point: using an upstream engine does not make maintenance disappear.
Even third-party projects embedding llama.cpp still need to update their bindings, validate new models, adapt to API changes, test hardware combinations, and sometimes work around upstream regressions.
llama.cpp even maintains a public libllama API changelog specifically for third-party projects that depend on the library.
I would describe the difference more like:
embedded mature OSS
≠ no maintenance
embedded mature OSS
= sharing a large moving substrate
with an upstream ecosystem
rather than having to reproduce all of that movement yourself.
Google provides another useful reminder that even large-vendor runtimes evolve: the old MediaPipe LLM Inference API is now maintenance-only, with Google directing new development toward LiteRT-LM.
If you own a runtime, you also own its evolution and migration path.
I would also treat supportability as an architecture requirement
There is another cost to going highly custom that is easy to miss: you are not only replacing code; you can also lose part of the existing troubleshooting ecosystem.
With a mature backend, an error may already have:
- documentation,
- GitHub issues,
- known regressions,
- driver-specific workarounds,
- people who have seen it before,
- searchable exact error strings.
That knowledge is useful even if your users never know that llama.cpp/MLX/etc. exists during normal operation.
Jan’s current troubleshooting documentation is a nice example.
The normal UI can present something friendly, but the logs still preserve backend-specific information such as CUDA, Vulkan, Metal, model-load failures, etc.
I would probably deliberately preserve that escape hatch.
For example:
Model could not be loaded.
Try:
- the recommended backend
- a smaller model
- updating the inference component
Technical details:
App version: 1.2.3
Inference engine: llama.cpp bXXXXX
Backend: CUDA
Model: example/model
Quantization: Q4_K_M
OS: Windows ...
GPU: ...
Driver: ...
Raw backend error:
...
and provide something like:
Copy diagnostics
This lets you keep the polished product UX while still making upstream knowledge reusable when something breaks.
I think this matters particularly for a small team.
Otherwise the support path can gradually become:
user
↓
your UI
↓
your error vocabulary
↓
your custom runtime
↓
your hardware integration
↓
you
with very few externally searchable intermediate layers.
Keeping upstream engine identity/version, raw errors, model revision, backend type, and hardware information means users/support/developers can still correlate failures with public documentation and issues.
It probably also makes external AI/search assistance more useful, simply because there is more public context to connect to. I would treat that last point as a practical side effect rather than a guarantee.
I would define the hardware contract before choosing the model
The phrase that would make me stop before selecting/fine-tuning a model is “typical consumer hardware.”
That can mean very different things.
For a first release, I would probably define explicit tiers such as:
Required:
- CPU fallback?
- minimum RAM?
Fast path:
- Apple Silicon?
- NVIDIA Windows?
Best effort:
- AMD GPU?
- Intel GPU?
Later:
- NPUs / vendor-specific accelerators?
And for an officially supported tier, I would define a small performance contract:
minimum RAM
maximum install/model size
acceptable cold-start time
acceptable TTFT
minimum useful tokens/sec
target context size
Only after that would I compare candidate models and quantizations.
Otherwise “best quality/speed balance” has no fixed meaning.
For example, the best model for:
16 GB Apple Silicon
may be a very different product choice from the best model for:
8 GB Windows laptop, CPU-only
or:
Windows gaming PC with 12 GB VRAM
I would therefore probably order the early work like this:
hardware/deployment contract
↓
candidate models
↓
quantization variants
↓
real-device benchmarks
↓
product-task evaluation
↓
fine-tuning only if a defined gap remains
I would not rule fine-tuning out at all.
I just would not commit to it before knowing what failure it is intended to fix.
There are several things that can look like “the model needs fine-tuning” but may actually be model selection, quantization, prompting, document retrieval, or formatting.
One particularly easy one to underestimate is chat formatting. Hugging Face’s chat-template documentation points out that different chat models — even ones derived from the same base model — can expect different control tokens/formats.
So “supported model” should probably mean more than “the file loads.”
I would test something like:
model file accepted
↓
model loads
↓
first token
↓
coherent single-turn chat
↓
correct stop / EOS behavior
↓
multi-turn conversation
↓
context growth
↓
cancel
↓
unload / reload
↓
optional capabilities:
tools / JSON / vision / audio
That becomes your compatibility contract.
Offline also needs a slightly more precise definition
“Fully offline” can describe several different products:
1. inference never makes network calls
2. no telemetry
3. works offline after one initial model download
4. works on a clean air-gapped machine from first launch
5. can also be updated using offline packages
Those are increasingly demanding requirements.
Again, mobile deployment gives useful examples.
PocketPal takes the download once, then run offline route.
MLC LLM also normally supports runtime model download, but its official iOS deployment documentation explicitly supports bundling model weights inside the app with bundle_weight: true.
Its Android documentation describes the same basic option for avoiding runtime weight downloads.
So even:
“the app must run on first launch without internet access”
is fundamentally a packaging/distribution choice; it still does not imply that the inference engine must be your own.
It does mean package size becomes part of the product design.
It also means I would add redistribution/access terms to the model-selection criteria.
The Hugging Face Hub exposes repository licenses, and its license documentation explicitly recommends checking and respecting the particular project’s terms.
Separately, some models use Hub gating, where access is granted to individual users.
I would not assume that “gated” automatically means “cannot be redistributed” — gating mechanics and license terms are separate things — but I would check the exact model/revision terms before making “weights bundled in the installer” part of the architecture.
So the selection criteria become something more like:
quality
speed
memory
model size
hardware fit
runtime compatibility
+
redistribution / access terms
Documents: I would clarify internally whether this is attachment handling or a local knowledge base
“Working with documents” can hide another fairly large boundary.
There is a big difference between:
User attaches PDF
↓
extract text
↓
use it in this conversation
and:
local document library
↓
parsing / OCR
↓
chunking
↓
embeddings
↓
index
↓
retrieval
↓
source attribution
If the intended feature is the second one, I would treat it as a subsystem rather than as a small extension of the inference engine.
That is also nice architecturally because it lets you evaluate the LLM and the retrieval pipeline separately.
For v1, a narrow document contract can keep the project from quietly turning into:
desktop app
+ inference runtime
+ model manager
+ document-processing platform
+ search engine
all at once.
A compact decision tree
The way I would navigate the custom-inference question is roughly:
Why do we want “custom”?
│
├─ Users should not install/manage Ollama/etc.
│ │
│ └─ Bundle and manage the runtime ourselves.
│
├─ We do not want a generic public API/server layer.
│ │
│ └─ Use a narrow private contract.
│
├─ We specifically do not want another process.
│ │
│ └─ Integrate a low-level library in-process.
│
├─ We need full control over model/runtime policy.
│ │
│ └─ Own orchestration; keep the low-level engine replaceable.
│
└─ Existing engines fail a concrete requirement:
performance / memory / model / hardware / licensing / etc.
│
└─ Then a custom runtime becomes a much stronger candidate.
│
└─ Price in ownership of:
conversion
model support
operators
quantization
hardware backends
regression testing
diagnostics
support
migration
That last branch is the point where I would personally want a measurable reason.
For example:
- the target model cannot be represented efficiently in existing engines,
- a specific hardware accelerator is strategically important and unsupported,
- the memory architecture requires something materially different,
- measured latency/throughput cannot meet the product requirement,
- binary/distribution constraints rule out existing engines.
At that point “custom inference engine” is solving a demonstrated problem rather than simply providing implementation uniqueness.
So, if I were approaching this as a small/medium-sized product effort, my starting point would probably be:
very opinionated, purpose-built UX
+
curated model/hardware policy
+
product-owned narrow inference interface
+
embedded/managed mature low-level runtime
and I would keep the boundary replaceable enough that going deeper later remains possible.
That still gives you a lot that is genuinely yours:
- the application behavior,
- zero-configuration UX,
- supported-hardware contract,
- model selection policy,
- document workflow,
- privacy/offline guarantees,
- update strategy,
- diagnostics,
- the internal inference API,
- the exact subset of runtime functionality exposed by the product.
In other words, I think you can make the product highly custom without immediately making the fast-moving inference substrate equally custom.
And if, during benchmarking, that substrate turns out to be the thing actually preventing the product you want, then that would be a very concrete reason to push the custom boundary downward.