I’m looking for an experienced developer or team to help build software for a dedicated AI hardware device.
The goal is to create an AI system that:
Runs entirely offline with no internet connection required.
Is based on a powerful open source AI model.
Can be encoded directly onto the hardware so it boots and operates natively from the device.
Has no artificial usage limits, subscriptions, or cloud dependencies.
Can be customised and expanded over time.
If you’ve already built something similar, I’d also be interested in purchasing an existing solution or collaborating to adapt it for my project.
This is a paid opportunity for the right person or team.
If you have experience with offline LLMs, AI appliances, embedded systems, or open source AI deployments, I’d love to hear from you. Please send me details of your experience, GitHub projects, or examples of similar work.
I don’t have the experience or skills to handle that, but there’s a #jobs-and-collaboration channel on the HF Discord. You might want to check that out, too. Also, perhaps LeRobot Discord, too? Mainly for hardware.
Built a piece of hardware similar to this that function as box about the size of a raspberry pi. Ended up taking the learnings from the box into a platform that can run large-models locally.
Most of this can be done in a single day for around $500. No usage limits and entirely offline thats the goal but you’ll run into a ton of hardware issues.
An offline AI device sounds especially interesting for applications where privacy and low latency are important. I’d focus on efficient local models, hardware acceleration, power consumption, and a simple update mechanism. It could also have interesting applications in gaming environments, where services such as Dewagg SBOBET could benefit from faster local processing without depending entirely on cloud services.
I’m in the process of opensourcing what we’ve built on heysudo.xyz which gets a family a completely local personal agent with local inference with tool calling, etc, the size of a tissue box. Have connection in China for manufacturing the device as well. You can email me at ap@sudohomes.com
Hi mrrich2020, your requirement makes sense especially the challenge of having the AI run fully locally without cloud APIs, subscriptions, or an internet connection.
The important parts here seem to be selecting an open-source model that fits the target hardware getting efficient local inference within the device is CPU/GPU/RAM limitations and packaging the AI stack so it can boot and operate reliably as a standalone system. Keeping the architecture modular would also make future model upgrades and custom features easier.
This is an area our team at DRC Infotech can help with from the AI/software side. We work on custom AI/ML solutions and open-source AI integrations and we would be interested in understanding the hardware and model you have in mind.
If you are open to it happy to connect and discuss the requirements. We can then see what would be the best practical approach for your device.
Llama 3 has been usable completely offline on an actual raspberry pi for many years. It did best on the Pi4 with 8GB (And presumably the 5 will run it better), but did run on smaller models of the Pi4 at acceptable speeds. A GB10 will run much larger models capable of writing much more complex code and again, they’re helped mightily by being able to go read documentation but it’s not strictly necessary.
Models comparable to the lowest tier OpenAI or Anthropic frontier models require $100k-$200k hardware which is increasingly difficult to obtain for the average person or even small business.
It seems likely that a few million dollars (hardware costs only, excluding engineering costs) would be plenty to build an offline box (mostly using open weights Chinese models, but there are a few domestic options) similar in capability to the subscription models you can get for $100/month. You have to be very, very motivated to want that offline model and spend 10,000 months of subscription costs for an offline box.
I say this an owner of a GB10 who spends basically all his free time optimizing models to run on smaller hardware. What exactly are you trying to accomplish?
I’m interested in helping with this project. I have experience with AI/ML software development, backend systems, and deploying models in local/self-hosted environments.
From your description, I understand that you’re looking for a system that can run completely offline on dedicated hardware, using an open-source model without cloud APIs, subscriptions, or artificial usage restrictions.
I can help with areas such as:
Selecting an open-source LLM that fits the target hardware and use case
Running models locally with frameworks such as llama.cpp, vLLM, or other suitable inference runtimes
Model quantization and optimization for available CPU/GPU/NPU resources
Managing RAM/VRAM usage and improving inference performance
Building the application layer around the model
Packaging the software so the device can operate independently after boot
Designing the system so additional models and capabilities can be added later
Evaluating existing open-source projects that could be adapted instead of building everything from scratch
I’d suggest starting with the hardware specifications and expected workload. The available RAM/VRAM, processor, GPU/NPU, storage, power constraints, model size, and expected response speed will determine which model and inference approach makes the most sense.
If you can share the hardware details and what you want the device to do, I’d be happy to discuss a practical architecture and development approach.
I can also share relevant GitHub projects and previous work if useful.
Maybe you’d like to check out my project: https://github.com/AMOS144/vates. With it, you can run the int4‑quantized qwen‑next‑80b‑a3b model using only 10 GB of memory. It achieves a prefill speed of 476 tokens/s and a decode speed of 31 tokens/s under a 9k‑token context window.
I have an idea about routing multiple SLM contains specific for coding, general purpose, research etc to efficiently call the models using an agent framework and to get the pipelined output and this can be run locally instead using LLMs with high computing powers.