Every MES and automation vendor spent 2024 demoing the same thing: an operator types a question into a chat box, and a large language model answers with a citation from the work instruction library. It worked beautifully in the conference room. On the actual floor, with real network topology and real response-time expectations, a lot of those pilots quietly stalled. What’s replacing them isn’t a retreat from generative AI — it’s a rightsizing. Small language models, run locally on edge hardware and fine-tuned on a plant’s own documents, are what’s actually making it into production lines now, while the hosted-LLM chat assistant is getting repositioned as a back-office tool.
This shift hasn’t been named in most of the trade coverage, but if you’ve sat in a renewal conversation with a GenAI-enabled MES or CMMS vendor recently, you’ve probably felt it: the sales conversation has moved from “which foundation model are you using” to “where does inference actually run.”
Why the big-LLM pilots hit a wall
Three problems showed up over and over, and none of them are exotic.
Latency. A cloud API round trip — network hop out of the plant, inference, network hop back — adds up to something in the range of a second or more under normal conditions, and considerably worse under load or on a congested plant WAN link. That’s tolerable for an engineer researching a root cause at a desk. It’s not tolerable for an operator standing at a station mid-changeover who needs a fault code interpretation before the line stops accumulating downtime. Response time budgets on the line are measured against the pace of the work, not against what feels acceptable in a browser tab.
Token cost at scale. A single knowledge-worker chatbot is cheap to run. Multiply that by every operator on every shift asking a running assistant to look up torque specs, changeover steps, and fault-code remediation dozens of times a shift, and the token bill scales in a way that looks nothing like the pilot’s cost model. Vendors that priced GenAI features as a thin per-seat add-on found the underlying inference cost didn’t scale the same way.
OT network isolation. This is the one that actually kills projects, not just budgets. Plants running IEC 62443-aligned network segmentation, air-gapped historian networks, or strict Purdue-model boundaries between OT and IT often can’t route floor traffic to a cloud API at all without opening a hole in the architecture that security teams will not approve. No amount of prompt engineering fixes a network topology problem. This is the constraint that pushed vendors toward local inference, not a change of heart about model quality.
What’s actually shipping instead
The pattern that’s emerged: a small language model, generally in the 3B–8B parameter range, distilled or fine-tuned on a specific plant’s documents — SOPs, work instructions, fault-code tables, sometimes genealogy and quality records — running on an edge gateway or industrial PC on the plant network. No round trip to the internet. No per-token metering. Response times measured in the low hundreds of milliseconds instead of seconds, because the model and the data it’s grounded on are both local.
This isn’t a like-for-like swap. A 3–8B parameter model fine-tuned on a narrow document set will not out-argue a frontier model on open-ended reasoning, and it shouldn’t be asked to. What it’s good at is answering bounded, retrieval-heavy questions against a known corpus: “what’s the torque spec for this fastener on this variant,” “what does fault code E-114 mean on this line,” “what’s the next step in this work instruction.” That’s most of what an operator actually needs mid-shift, and it’s a task where a small, well-grounded model with retrieval-augmented generation against your own document repository performs close to a much larger model — because the hard part was never raw model size, it was getting the right paragraph in front of the model in the first place.
Where the big cloud model still earns its keep
Engineering root-cause investigations, cross-plant trend analysis, drafting CAPA reports, summarizing a quarter of quality data — these are lower-frequency, higher-latitude tasks where a knowledge worker at a desk with time to review the output is the actual user. That’s a reasonable place for a hosted LLM API, with real security review of what data leaves the plant. The mistake was assuming the same model belongs on the line.
A framework for the decision
Before you sign a 2027 renewal with a GenAI-enabled MES, andon, or CMMS vendor, run the use case through these questions:
- What’s the actual response-time budget? If the answer needs to land inside a takt-time window or during an active line stoppage, cloud round-trip latency is disqualifying regardless of model quality.
- Does this data ever cross the OT/IT boundary today? If your network segmentation currently prevents floor devices from reaching the internet, a cloud-dependent assistant either can’t be deployed as designed or requires a network exception your security team has to sign off on — ask this before you ask about features.
- Is the question bounded or open-ended? Fault-code lookups, work-instruction retrieval, and spec lookups are bounded — good fit for a small local model with retrieval against your document set. Free-form troubleshooting reasoning across unfamiliar failure modes leans toward a larger model, used by an engineer, not an operator.
- Where does your genealogy and quality data actually live, and does the vendor’s fine-tuning process require it to leave the plant? Ask explicitly whether fine-tuning happens on-prem or whether your SOPs and quality records get uploaded to a vendor’s cloud training pipeline. That’s a data-residency and IP question as much as a technical one.
- What happens on model drift? A small model fine-tuned on your document set needs a defined refresh process when work instructions or fault codes change. Ask the vendor how retraining or reindexing works and how often it happens — this is closer to a document-control question than an AI question, and it should be owned the same way you own revision control on your SOPs.
What to actually ask on the renewal call
Get specific with the vendor. Ask what parameter range the model is — if they won’t answer, that’s informative on its own. Ask where inference physically executes: their cloud, your edge hardware, or a hybrid split. Ask what happens to the assistant during a WAN outage, because if the honest answer is “it stops working,” you’ve just learned your shiny copilot has the same availability profile as your internet connection, which is a bad trade for a floor tool. And ask what your plant-specific data is used for beyond answering your own operators’ questions — model improvement clauses in vendor contracts deserve the same scrutiny you’d give a data-sharing clause anywhere else.
None of this means generative AI on the floor is overhyped and dead. It means the interesting engineering — and the actual production deployments — moved from “biggest model available” to “smallest model that’s grounded well enough in your own documents to be fast, cheap, and allowed on your network.” That’s a less glamorous story than the chatbot demo, but it’s the one that’s actually running shifts.
This article was written with the assistance of artificial intelligence. While we aim for accuracy, the information may be incomplete, out of date, or incorrect, and should be independently verified before you rely on it for any decision. It is provided for general information only and does not constitute professional advice.
