Four Pieces of Enterprise AI Infrastructure to Watch

Every wave of IT innovation brings changes to enterprise IT infrastructure. Identity brought directory services. Data brought DBMSs. The web brought web apps and web servers. Enterprise AI is no different, as it is starting to develop its own infrastructure layer. That layer is still uneven right now. Some parts are already available as mature cloud services, while others exist as features scattered across products, open-source projects, and early vendor offerings. Regardless, the direction is becoming easier to see. As AI moves from isolated applications into shared enterprise use, several recurring technical needs begin to appear often enough that they stop looking like application-specific concerns and start looking like infrastructure.

Four areas stand out where enterprises need infrastructure for serving and managing models, agents, tools, and context. These services are related, and vendors will often package several of them together, but they solve different problems. Treating all of them as one generic AI platform hides distinctions that will matter as architectures mature.

Model infrastructure

The most established of the four is model infrastructure, as models were probably the first thing that made AI a possibility. As their use expands, applications need a dependable way to invoke models without every development team independently solving model hosting, capacity, authentication, networking, quota, scaling, and provider integration. The service begins with model availability. An enterprise may consume commercial models from several providers, deploy open-source models, host custom-weight models, or use specialized models for embeddings, vision, speech, document extraction, and other inference tasks. Those models have different infrastructure requirements and different commercial models. Some are purchased by token, while others use provisioned throughput or dedicated GPU capacity.

Microsoft Foundry already illustrates much of this category. Its model catalog can expose models through serverless APIs, provisioned throughput, or managed compute. Managed compute handles GPU topology, serving runtimes, container images, patching, scaling, authentication, networking, and inference endpoints for open-source and custom models. AWS Bedrock provides the same category from a different direction, exposing foundation models through common APIs while adding access policy, regional routing, usage tracking, and cost attribution through inference profiles. The products differ, but the infrastructure problem is recognizable.

As this category matures, an enterprise version of this service will likely include model discovery, approved-model catalogs, deployment, endpoint management, routing, capacity, quotas, observability, cost attribution, version management, and evaluation hooks. Model gateways may sit in front of it to normalize APIs or enforce policy across several providers. The point is not that every enterprise will build one large internal model platform. Rather, model consumption is becoming infrastructure. Application teams will increasingly expect model access the same way they expect databases, queues, object storage, or API endpoints.

Agent infrastructure

Agent infrastructure introduces a different problem because an agent is more than a model endpoint. It has its own instructions, tools, identity, state, execution history, permissions, runtime behavior, and often the ability to change external systems. Once agents move beyond small experiments, enterprises need somewhere to operate them.

The category is less standardized because the definition of agent remains rather fluid. Some agents are little more than a prompt and a collection of tools. Others contain substantial application code, persistent state, planning logic, specialized memory, or multiple cooperating components. Nevertheless, regardless of how an agent is implemented, an enterprise still needs to know which agents exist, where they run, what tools they can call, which identities they use, what data they can access, how they are versioned, and what they did during a particular execution.

Microsoft Foundry Agent Service is already moving in this direction. It is an opinionated platform that supports managed prompt agents and hosted agents, provides managed endpoints and scaling, and integrates agent identity with Microsoft Entra ID. Published agents can receive distinct identities that can be assigned permissions to downstream resources, allowing an agent to authenticate as an identifiable runtime actor rather than carrying secrets in code or prompts. Foundry also provides versioning, tool integration, networking, observability, and publishing mechanisms. Amazon Bedrock Agents similarly provides a managed agent runtime with model orchestration, knowledge access, and action groups that connect agents to functions or APIs.

The important part about the agent infrastructure requirement, however, is not allegiance to one agent framework. Enterprises will probably run agents written with several different frameworks, supplied by several vendors, or built by different teams. The durable need is hosting and lifecycle management. Agent infrastructure needs to provide execution, identity, permission boundaries, secrets, state, tool connectivity, telemetry, versioning, deployment, scaling, and auditability. As agents gain more authority, the ability to distinguish an agent from a human user or an ordinary application identity will become increasingly important.

MCP infrastructure

Unlike agents, the Model Context Protocol is a standard. MCP standardization solves one problem and immediately exposes another. Once an enterprise has dozens or hundreds of MCP servers, it needs a way to know which ones are trusted, who can use them, how they authenticate to their backends, and what operations they expose.

This problem resembles the problem that API management addressed for web APIs. The comparison is useful, although MCP brings different runtime behavior and a tool-oriented discovery model. Existing APIs can become MCP tools. Native MCP servers can sit behind gateways. Multiple MCP servers can be federated behind a managed endpoint. Authentication can be separated between the client-facing connection and the backend system. Policies, rate limits, monitoring, and discovery can then be applied without requiring every agent to carry credentials for every downstream service.

In this way, MCP helps separate tool providers from agent implementations. A business system can expose a governed MCP interface without knowing which model or agent framework will consume it. Agents can change while the tool contract remains. That makes MCP management an infrastructure concern rather than an application-specific integration pattern.

Azure API Management now supports this pattern directly. It can expose an existing REST API as an MCP server, govern existing remote MCP servers, apply access and monitoring policies, and register MCP servers for discovery through API Center. As the category develops, MCP infrastructure will need discovery, registration, authentication, authorization, backend credential management, tool exposure policy, monitoring, rate limits, versioning, and lifecycle controls. Some organizations will extend existing API management products. Others will use dedicated MCP gateways or capabilities embedded in agent platforms. Either way, the service exists because agents should not connect indiscriminately to whatever MCP endpoint a developer happens to find.

Context infrastructure

Context infrastructure is probably the least settled of the four categories and potentially the most consequential. Models are increasingly interchangeable. What differentiates an enterprise AI system is often the information available to it and whether that information is relevant, authoritative, current, and permitted for the task being performed.

Today, that context is fragmented. Technical metadata lives in catalogs. Ownership may live in organizational systems. Business definitions sit in glossaries or semantic layers. Individual AI projects often reconstruct a small portion of that context for themselves through retrieval pipelines, vector stores, prompt logic, and application-specific integration. That approach becomes expensive as the number of AI applications grows because the same enterprise concepts have to be rediscovered repeatedly. Each project has to determine authoritative sources, reconcile terminology, understand ownership, enforce permissions, decide whether information is current, and establish provenance. Context infrastructure turns those recurring requirements into a shared service.

This category is now beginning to show up explicitly in the market. Google has renamed Dataplex Universal Catalog to Knowledge Catalog and describes it as a universal context engine for the enterprise. It aggregates technical metadata, business semantics, lineage, ownership, structured and unstructured information, and governed retrieval for agents. AWS announced AWS Context in June 2026 as a service intended to map relationships across enterprise data into a knowledge graph and provide governed runtime context to agents. Gartner published a playbook for an enterprise context layer in July, and Forrester now describes context layers as an evolution of semantic layers and knowledge graphs that combine business semantics, governance, ontology, metadata, policy, process, and other enterprise knowledge.

The category is still forming, and the vocabulary is inconsistent. Context graph, knowledge graph, semantic layer, metadata graph, context platform, and enterprise context layer are often used for overlapping ideas. Even so, the underlying infrastructure requirement is easier to state. Enterprises need a way to continuously assemble trusted context from distributed systems, connect it to business meaning and ownership, preserve provenance and temporal state, enforce permissions, and serve relevant portions of it to applications and agents at runtime.

A context service does not need to copy every underlying record into one database. Much of its value can come from metadata and relationships that identify what exists, what it means, where it came from, who owns it, whether it is authoritative, how current it is, and how to retrieve the underlying information when needed. In that sense, context management becomes a coordination layer over an estate that remains physically distributed.

These services will overlap without becoming the same thing

Although these services will increasingly overlap in products and platforms, enterprises still benefit from keeping the architectural concerns distinct. Model infrastructure answers how inference is provided. Agent infrastructure answers how autonomous or semi-autonomous AI workloads are executed and governed. MCP infrastructure answers how tools are published and consumed through a common protocol. Context infrastructure answers how enterprise knowledge and state are assembled and made useful to those workloads.

The boundaries matter because organizations may source each service differently. A company may use Bedrock for model access, run agents on its own Kubernetes infrastructure, govern MCP through an API gateway, and obtain enterprise context from a data catalog and knowledge graph. Another may buy most of the stack from one cloud provider. Both architectures can be valid because the services solve separable problems.

Further reading

Microsoft Foundry model deployment overview

Microsoft Foundry managed compute

Microsoft Foundry Agent Service

Microsoft Foundry agent identity

Amazon Bedrock model access

Amazon Bedrock inference profiles

Azure API Management MCP server overview

Model Context Protocol Enterprise-Managed Authorization

Google Cloud Knowledge Catalog

AWS Context announcement

Forrester on context layers