Select Page

What Is Enterprise AI Architecture?

Understand the four tiers of a production enterprise AI platform, the knowledge layer many stacks miss, and how to choose hosting, retrieval, and fine-tuning.

Definition

What is enterprise AI architecture?

Enterprise AI architecture is a technology stack that supports production deployment of AI across an organization’s data, systems, and users.

The common denominator of the stack is an enterprise AI platform. In an enterprise AI platform architecture, the platform is the layer everyone shares. A central team runs it, with an identity system, set of permissions, audit logs, etc. Business units then build their own apps on top of it.

There are four tiers to the production stack: compute and storage, a data fabric and knowledge layer, model gateway and applications. Identity, security, observability, evaluation and operations are cross-cutting capabilities across all four.

Main Takeaways

  • In many stacks, the data fabric is only half done. The search half (vector databases) is there, but what’s typically lacking is the knowledge layer, what your data is and who should access it, which is stored outside the model, and thus can live through a model change.
  • Run the gateway, permissions, knowledge layer, and testing as shared services. Business units can then build any apps on top.
  • Hybrid search copes fine with questions about one document. When answers rely on related facts, common definitions or proof of origin, GraphRAG is a strong option.
  • GPUs are usually not worth it to own unless there is a lot of work to be done. When you rent, you’re not paying for idle hardware until your load settles.
  • Before you fine-tune, get retrieval right. RAG for facts that change, fine-tuning for terminology, output format and code style.

What are the four tiers of enterprise AI architecture?

Imagine a sales lead asking, “Which of our active contracts in Germany expire next quarter, and who owns them?” There are four tiers which each perform a different task in answering that.

  • The first tier of AI enterprise architecture is compute and storage where the models are hosted or accessed through an external API. That translates to GPUs and super-speed storage, either in your data center or through a cloud provider.
  • The data fabric is next: the enterprise data foundation for GenAI, which contains or indexes enterprise data and metadata used to ground models. Vector databases are typically used to find text that is similar to the question, e.g., Milvus, Qdrant, Pinecone. The knowledge layer is atop. It’s the component that determines which contracts are “active” and who is the owner of each contract, who should be able to access each contract via authorization-aware retrieval, and the origin of each fact. Important to note is that identity and policy-enforcement systems are a separate responsibility.
  • The model gateway is a single entry point for all the models used. It is used to dispatch each request to an appropriate model, monitors usage and cost, and blocks unsafe prompts.
  • The applications tier is the last one where the answer appears. This could be within the CRM through enterprise application integration, or in agentic enterprise systems that take action on it, such as composing renewals emails.

Disclosure: Graphwise is a vendor that sells software for the knowledge layer described above. We don’t sell compute, hosting, models, or vector databases.

The choice between enterprise AI platform and point tools

Point tools are specialized tools that are great at dealing with a specific task. However, they come with their own connectors, permissions and model versions, and using too many can lead to a disjointed workflow with fragile dependencies.

An enterprise AI platform, on the other hand, gives everyone a unified AI foundation with one model registry, one origin for data and shared security.  It centralizes or coordinates capabilities such as identity and permissions, data governance and provenance, model and prompt management, retrieval, evaluation, observability, and audit logging. Business units can then build or select their own applications on top of that foundation.

Here is a simple test that will help you choose between them. If two teams working separately would answer the same question differently, then the data and definitions behind their answer should belong on a platform, because the company needs one consistent answer it can trust. However, if the task affects only one team and does not depend on shared enterprise definitions or data, then a point tool can be the more efficient choice.  

The goal is not to make every team use the same application. Teams should be free to choose the tools and interfaces that fit their work. The platform’s role is to provide a common, governed foundation so that important facts, definitions, access rules, and sources remain consistent across the organization.

Who owns and runs the enterprise AI stack?

Typically, the overall design is owned by an enterprise AI architect (sometimes titled AI enterprise architect). It’s a broad role. In addition to distributed systems and model tuning, search indexing, and network security, the architect should have sufficient knowledge of modeling to collaborate with domain experts to define the business concepts and rules to be used by the system.

The four operating groups take care of the stack day-to-day. DataOps ensures that documents are parsed, indexed and kept up to date, avoiding answers from stale content. To answer the question of whether a model is ready for release, MLOps or LLMOps evaluates and checks for quality degradation following an update. The platform should version models, prompts, retrieval indexes, ontologies and evaluation sets, and define rollback, fallback and retirement procedures when a model or provider changes. Platform IT and security keep the GPUs running, the network segmented and the identity provider connected.

How to implement enterprise-ready RAG?

Retrieval-augmented generation (RAG) is a method to retrieve the appropriate company information when a question is asked and pass it on to the model. With just a couple of hundred documents, you can create an impressive demo using a vector database. Production is difficult as real files are messy, permissions are restrictive, and many questions are cross-source.

A reliable set-up consists of 7 steps:

  1. Break down documents in terms of layout. Do not break up tables, headings and lists (tables that are broken in the middle can’t be used to answer a question).
  2. Capture the system’s permissions and authorization metadata during ingestion. The access rights, the owner and date of each document are also added to the index along with the text. Because permissions can change after ingestion, synchronize revocations and source-system changes, and treat the source policy system as authoritative.
  3. Connect words with people, places, and things. Identify each chunk with the corresponding products, customers, and regulations as mentioned in the chunk, and label it with IDs from the knowledge graph.
  4. Search two ways with authorization filters. Use keyword search BM25 + vector search side by side + filter both by what user is allowed to see and merge together with Reciprocal Rank Fusion.
  5. Re-rank. Only a selected set of results get to the model, after being scored by a cross-encoder like Cohere Rerank or BGE-Reranker against the question.
  6. Follow the graph. If the question requires multiple records, traverse the connections in the knowledge graph to retrieve related facts.
  7. Answer with sources. State each fact and record the source document, record, system and owner and verify the answers with a set of standard actual questions with known expected results prior to each release.

For questions involving authoritative records, dates, status or aggregation, retrieve from structured systems or query a governed semantic layer rather than relying only on document chunks.

Where vector-only retrieval stops

Vector search identifies passages with semantic similarity to the question. If this is something like “What is our travel policy for Germany?” and this policy resides in one document, then keep it. Vector-only retrieval becomes more challenging when the answer requires related information or multi-hop joins, aggregation, stable definitions, entity ambiguity and provenance.

Revisit the contract question from above: it requires a list of contracts, a definition of ‘active’, a period of dates, a location, and a particular owner for each contract. Vector search can be less reliable for counts, names that have two meanings, multi-hop joins and with indicating where an answer originated. In its paper “GraphRAG”, Microsoft Research faced a similar constraint: a (small) set of retrieved passages is not sufficient to answer global questions, such as the main themes across an entire collection of documents.

GraphRAG makes the knowledge graph available for retrieval, which means that the model operates on connected facts. Avalara, for instance. The results of Vector-only RAG were not precise enough for tax content, so it created a GraphRAG proof of concept with its current DITA content, where the project reported more deterministic, fact-based and explainable answers. Because it started from structured DITA content, the project reports automating the conversion into a knowledge graph.

Graphwise GraphRAG integrates graph search, vector search, and keyword search and supports existing vector stores and identity providers. Graphwise reports reductions of up to 80% in LLM token consumption versus conventional RAG in its own estimates, as the model receives graph-grounded facts rather than large chunks of text. This refers to token use, not total system cost.

Where should you host: on-premises, sovereign cloud, or public cloud?

Most hosting decisions come down to how busy your GPUs will be and where your data is allowed to go. Both questions tie back to your wider enterprise AI strategy: which use cases you fund first and how much data risk the business accepts.

CriterionOn-premises GPU clusterSovereign cloudPublic cloud (AWS, Azure, Google Cloud)
Upfront costHigh: hardware, power, and coolingA contract or subscriptionNone; you pay for what you use
Data controlFull controlDesigned so data stays within specified jurisdiction and controlsShared under the provider’s contractual, technical and jurisdictional controls
ScalingSlow; more capacity means buying hardwareFlexible within your reserved capacityOn demand, within your GPU quota
Running effortHigh; you run the data centerLow to mediumLow with managed services
Cost per tokenTypically lowest, but only when GPUs stay busyMediumCan be higher under steady, heavy use

On-premises makes sense when data can’t leave your buildings or the GPUs will be busy most of the time. If that’s your situation, NVIDIA’s Enterprise AI Factory validated design is one published blueprint to start from. A sovereign cloud operating model suits organizations whose regulators fix where data must stay but who would rather not run a data center. A public enterprise cloud platform is the natural fit for pilots, workloads that rise and fall, and frontier models you can only reach through a provider API.

You can also mix them into a hybrid enterprise cloud architecture, for example by keeping retrieval close to sensitive data and renting GPUs for peaks. Either way, the knowledge layer comes with you, because it’s stored in open W3C standards (RDF, SKOS, and OWL) and not in a vendor’s own format. This is, of course, subject to implementation, extensions, mapping and dependencies.

When to fine tune, quantize or cache?

Firstly, deal with retrieval. The model may be getting the wrong feeds and no amount of tuning will correct the model answers. When retrieval is secure, choose the strategy that is appropriate to the problem:

  • If the model already has enough information about the topic, it often only requires instructions and structured prompting.
  • RAG – if facts are frequently changing, answers must include a source or access depends on who is asking. Facts trained into the model go out of date, and you can’t put permissions on them.
  • LoRA fine-tuning, e.g., when you need your model to learn how to use your vocabulary, your coding style, or a specific output format like a fixed JSON schema. However, constrained decoding, tool schemas, validation and prompting can be better for strict JSON.
  • Full pre-training is uncommon for most enterprises. It’s only worth it for new types of data, or vocabulary that’s foreign to an existing model.

Maximizing return on existing GPUs

Some serving methods enable the same hardware to serve more users without the need to change the model.

One of the innovations vLLM uses, called PagedAttention, is to store the KV cache (the model’s cached key/value states used during autoregressive decoding) in small blocks to ensure less GPU memory is being wasted, and the paper reported 2 to 4 times higher throughput of earlier serving systems at the same latency. 

Speculative-decoding can be done with a small fast model, which drafts a small number of tokens a few times faster and then the output will be checked by the large model in one single pass. According to the original paper, the speedup is 2x-3x with identical output distribution, but real-world gains depend on draft-model acceptance rate, hardware, batching, latency target and implementation.

Weight quantization stores the weights in the model with reduced number of bits. FP8 uses half the bits per weight of FP16 and 4-bit methods such as AWQ can cut weight storage even further, but there is often a small loss in accuracy, you should try it on your own questions before deciding. A cache in the model gateway can reply to reoccurring questions without invoking the model. All of these contribute to lowering the cost per question, so keep them in mind when evaluating ROI on enterprise AI investments. However, the real-world savings also depend on the format, scales, metadata, runtime kernels, KV-cache precision and hardware.

Next step

None of these choices happen in isolation. Hosting decides what GPUs you have, and retrieval quality decides whether you need to tune at all, so it pays to look at the whole stack at once.

FAQ

Any Questions? Look Here

Compute, data fabric and knowledge layer, model gateway, identity, and testing, it's the whole layer of the AI stack that's shared with everyone through a central team. Business units then develop their own apps on top of it.

Not always. If you're looking for answers to one particular document, a vector database with hybrid search will suffice. A knowledge graph is well worth the investment when answers must synthesize knowledge from multiple sources, require common definitions or need to trace the answer back to its origins.

On-premises can make sense if your data can’t leave your facilities, your GPUs will be busy much of the time, or regulations justify owning the infrastructure. If not, consider a public or sovereign cloud and re-review the numbers when usage settles.

Fine-tune when the model needs to learn how to say something, such as your vocabulary, coding or output style. Use RAG when it needs to know facts that change, need a source or need a permission. Both can be used as well.

NVIDIA’s Enterprise AI Factory is a full-stack validated design for on-premises enterprise AI, built around NVIDIA Blackwell accelerated computing, BlueField DPUs, Spectrum-X networking and NVIDIA AI Enterprise software, with partner ecosystem components. Don’t confuse it with DGX SuperPOD, which is NVIDIA’s design for large DGX clusters.