For the last couple of years, “generative AI for manufacturing” has mostly meant one thing: point a large cloud model at your work instructions, PDFs, and maintenance logs, and let operators ask it questions in plain English. That’s still a legitimate use case. But the infrastructure conversation underneath it has quietly changed. Chip vendors and MES suppliers are now shipping edge-optimized small language models and NPU-equipped industrial PCs specifically aimed at running generative AI without a round trip to anyone’s cloud. The question a controls engineer or plant IT lead should be asking in 2026 isn’t “which LLM vendor do we standardize on.” It’s “does this workload need the cloud at all.”
That’s a genuinely different decision, and it deserves a genuinely different evaluation process than the one most plants ran through in the initial ChatGPT-for-manufacturing wave.
What changed: small models got good enough
A few years ago, “small language model” meant a compressed, dumbed-down assistant that stumbled on anything beyond simple retrieval. That’s no longer true for the narrow, bounded tasks that dominate shop-floor use cases. Open-weight models in the 3B-to-8B parameter range, fine-tuned or retrieval-augmented against a plant’s own work instructions, changeover procedures, and SOP libraries, now perform respectably on the kind of question an operator actually asks: “what torque spec for this fastener on the B-line fixture,” “walk me through the changeover sequence for SKU 4471,” “what does alarm code E-114 mean on this press.”
These aren’t open-ended reasoning tasks. They’re closed-domain question answering over a known, finite corpus of documents you already own. That’s precisely the kind of workload where a small model paired with retrieval-augmented generation (RAG) against your own document store can match a much larger cloud model, because the heavy lifting is retrieval and grounding, not raw model size. The model doesn’t need to know general chemistry or write poetry. It needs to know your plant.
Pair that with NPU-equipped industrial PCs and edge gateways now marketed explicitly for this purpose, and you get inference that runs locally, on hardware that already sits in the cabinet or on the line, with no dependency on plant network uplink to a cloud API.
The three variables that actually matter
Strip away the novelty and the decision comes down to three things: latency, air-gap and data-governance requirements, and per-token economics at scale. Chatbot polish and model benchmarks on general leaderboards are close to irrelevant here — your use case is narrow, and a model that’s mediocre at trivia can be excellent at reciting your torque specs correctly every time.
Latency and network reality
Cloud LLM calls involve a network hop, API queueing, and generation time that’s fine for a knowledge worker typing in a browser tab, and much less fine for an operator standing at a fixture with a tablet, mid-changeover, on a plant Wi-Fi network that was designed for barcode scanners, not chat. Shop floors also lose connectivity in ways office networks rarely do — RF interference near motors and VFDs, congested industrial Wi-Fi, planned network maintenance during a shift. An on-prem SLM running on a local gateway keeps working through all of that. If your copilot needs to be reliably available during changeovers and line-side troubleshooting, that alone is often the deciding factor.
Air-gap, IT/OT segmentation, and data governance
Many plants run OT networks that are deliberately segmented from the internet under an IEC 62443 zone-and-conduit architecture, and for good reason — a generative AI feature isn’t worth punching a new hole in that segmentation. Sending work instructions, quality data, or maintenance logs to an external API also raises real questions about where that data lands, who can access it, and whether it counts as sharing proprietary process knowledge outside your walls. None of this is hypothetical caution; it’s the same logic that already governs historian access and remote support tools in most OT environments. An on-prem model changes the compliance conversation from “what’s your vendor’s data retention policy” to “does this ever leave the building,” which is a much easier conversation to have with a plant security team.
Per-token cost at real operator scale
A pilot with a handful of engineers querying a cloud LLM a few times a day costs almost nothing to run. Roll that out to every operator on every shift, asking questions constantly during changeovers and troubleshooting, and the token volume compounds fast. On-prem inference has a real upfront cost in hardware and setup, but it doesn’t meter per query. If your copilot is going to be used constantly rather than occasionally, the economics tilt toward on-prem the more it succeeds — which is a strange but real dynamic: the better your rollout goes, the more a per-token cloud bill becomes the wrong pricing model for it.
What’s ready now, and what isn’t
Ready now: closed-domain Q&A over static or slowly-changing documents — work instructions, SOPs, troubleshooting guides, maintenance manuals. This is retrieval plus a small model, and it’s a solved enough problem that several MES and industrial software vendors are shipping it as a packaged feature rather than a science project.
Not ready, or at least not mature: open-ended reasoning across live, fast-changing production data; multi-step agentic actions that write back into MES or PLC logic without a human in the loop; and anything that requires the model to synthesize novel judgment rather than retrieve and rephrase known information. Small models are narrow by design. Ask one to reason outside its fine-tuned domain and it will degrade faster and less gracefully than a large cloud model would. That’s a feature for scope control, not a limitation to route around — but it means you should be explicit about keeping these copilots inside their lane.
A practical framework before you commit
Before choosing between a cloud API and an on-prem SLM, walk through this in order:
- Define the corpus. Is this Q&A over a bounded, known set of documents you control, or does it need broad, general knowledge? Bounded corpus favors SLM plus RAG.
- Check the network reality where it’ll be used. If the use case lives on the line during changeovers, treat connectivity as unreliable by default.
- Ask your OT security team, not just IT, whether this data can leave the plant. Get that answer before you build anything, not after.
- Estimate real usage volume, not pilot volume. Per-token cloud pricing is deceptively cheap at pilot scale and can look very different at full-shift, full-plant adoption.
- Scope the task narrowly and say so out loud. A copilot that answers “what’s the torque spec” well is more valuable than one that tries to do that plus general troubleshooting plus quality analytics poorly.
The honest takeaway is that this isn’t really an AI story anymore — it’s an infrastructure story. The interesting engineering question in 2026 isn’t which chatbot is smartest. It’s whether you can run something smart enough close enough to the work that it’s actually useful when the network isn’t cooperating, the auditor is asking about data flows, and the operator needs an answer in the next ten seconds, not the next ten minutes.
This article was written with the assistance of artificial intelligence. While we aim for accuracy, the information may be incomplete, out of date, or incorrect, and should be independently verified before you rely on it for any decision. It is provided for general information only and does not constitute professional advice.
