Enterprise AI platforms are sold on capability and judged, eventually, on control. Every serious contender can generate text, retrieve documents, and call tools. The differences that determine whether a platform survives its second year are quieter: who is allowed to build what, whether the model can be swapped without rewriting applications, whether anyone can reconstruct why a system produced a particular answer six months ago, and what happens to the bill when usage triples.
The distinction that matters in evaluation is between a platform and a product. A product does a defined job. A platform is a foundation on which your teams will build many jobs, most of which do not exist yet. That means the evaluation criteria are not feature comparisons — they are questions about extensibility, governance, and operational visibility, because those constraints will apply to applications nobody has specified.
This guide covers the six dimensions that separate platforms that scale from platforms that quietly become another silo, and what to actually ask during an evaluation rather than accepting the demo version of each answer.
Governance: Who Can Build What, on Which Data
Governance is the first dimension because it is the one most likely to be discovered as inadequate after adoption has spread.
Permission granularity
The baseline question is whether the platform can express your organization's real access structure. Can access be scoped by team, by data source, by application, and by action? Can a business unit build assistants over its own data without gaining visibility into another unit's? Can a compliance function be granted review rights without build rights?
Platforms that offer only admin-or-user distinctions force a choice between locking the platform down until it is unused or opening it until it is ungoverned. Neither survives contact with a large organization.
Approval and lifecycle control
Ask what happens between someone creating an application and it becoming available to a thousand employees. A platform without a review gate produces a sprawl of near-duplicate assistants of unknown quality within months. A platform with an unworkably heavy gate produces shadow adoption of consumer tools, which is worse.
Look for a defined lifecycle — draft, reviewed, published, deprecated — with ownership attached and a mechanism for retiring applications whose owner has left.
Policy enforcement at the platform layer
Content restrictions, PII handling, refusal behavior, and data retention should be enforceable centrally rather than reimplemented in each application. The test is whether a policy set by a governance team automatically applies to an application a business unit builds tomorrow.
Security and the Identity Boundary
Security evaluation for AI platforms extends past the usual certification checklist. SOC 2 Type II, encryption at rest and in transit, and penetration testing evidence are necessary and not distinguishing. Three questions go further.
The first is permission fidelity in retrieval. When the platform indexes your document repositories, does a user's query respect the permissions of the source system at query time, or does the index flatten everything into a single searchable corpus? A platform that indexes without preserving source permissions will eventually surface a compensation document to someone who should not see it, and the failure will be traced to procurement rather than engineering.
The second is data handling by the model provider. Establish contractually whether prompts and outputs are used for training, how long they are retained by the provider, and in which jurisdictions they are processed. If the platform brokers multiple model providers, the answer may differ by model, and the platform should let you restrict which models are available for which data classifications.
The third is what the platform is allowed to do rather than see. Once applications call tools and write to systems, the security question becomes an authorization question: which actions can an application take, under whose identity, and with what limits. Actions taken under a shared service account are effectively unattributable, which is a problem the first time an audit asks who approved something. The evidence expectations here mirror those in any compliance audit — attribution, timestamps, and an unbroken trail.
Model Management and Portability
The model market moves faster than enterprise procurement cycles. A platform that ties applications to one provider converts every future price change or capability shift into a migration project.
Evaluate whether the platform supports multiple model providers and whether switching between them is a configuration change or a rewrite. The meaningful test is whether an application's prompts, retrieval configuration, and evaluation sets survive a model swap intact.
Beyond portability, ask about routing. Mature platforms let different tasks use different models — a small, inexpensive model for classification and a larger one for synthesis — because uniform model choice is uniformly wasteful. Routing by task, with per-application cost visibility, is one of the most reliable levers on operating cost.
Also examine version control. When a provider updates a model, outputs change. A platform should let you pin versions, test a new version against your evaluation set before adopting it, and roll back. Platforms that silently upgrade the underlying model give you unannounced behavior changes in production systems, which is difficult to explain to a business owner whose accuracy dropped overnight.
Data Connectors and Grounding
Connector breadth is the most demonstrated capability and the most superficially evaluated. Nearly every platform lists integrations with the major content systems. The differences appear one layer down.
Ask how the connector handles incremental sync rather than full reindexing, since full reindexing of a large repository is slow and expensive enough to become a scheduling problem. Ask whether it captures document-level permissions and how frequently permission changes propagate — a permission revoked in the source system that takes a week to reach the index is a live exposure.
Ask how the platform chunks and retrieves. Fixed-size chunking destroys the structure of tables, contracts, and policy documents, and the resulting retrieval quality is poor in ways that are hard to diagnose because the answers still read well. Platforms that support structure-aware processing and configurable retrieval strategies produce materially better grounding on real enterprise content.
Finally, ask what happens for the systems with no connector. Most organizations have at least one important application nobody built an integration for. The answer should be a documented custom connector framework, not a roadmap promise. Where those gaps dominate, dedicated integration engineering becomes the deciding factor in whether the platform delivers anything.
Observability: The Line Between Pilot and Production
Observability is where platform evaluations most often go wrong, because pilots do not need it and production cannot function without it.
At minimum, the platform should log every interaction with enough detail to reconstruct it: the input, the retrieved context, the model and version used, the tools called, and the output. This is what allows an incident to be investigated rather than speculated about.
Beyond logging, look for evaluation infrastructure. Can you maintain a test set of representative cases with expected outputs and run it against an application on demand? Without this, there is no way to detect quality drift, and no safe way to change a prompt, a model, or a retrieval setting. Teams without evaluation tooling end up afraid to touch working systems, which freezes them.
Usage analytics matter for a different reason. Knowing which applications are actually used, by whom, and at what cost is how a program justifies continued investment and identifies the sprawl worth retiring. Cost attribution by application and team is the specific capability to insist on — without it, the platform bill arrives as a single unexplainable number.
Scaling: What Breaks at Volume
Three things change as an AI platform grows, and each should be probed during evaluation rather than discovered.
Cost scales with usage, and usage grows faster than headcount once applications become genuinely useful. Understand the pricing model precisely: per seat, per token, per application, or a hybrid. Model the bill at three times current projected usage and confirm the finance conversation is survivable.
Concurrency scales unevenly. Ask about rate limits, queuing behavior under load, and what happens when a model provider throttles. A platform that degrades gracefully — falling back to a smaller model, queuing non-urgent work — is operationally different from one that returns errors.
Application count scales into a management problem. At twenty applications, ownership is informal. At two hundred, the platform needs a catalog, ownership metadata, deprecation workflows, and a way to find duplicates. Ask to see how the platform handles a large deployment rather than a demonstration tenant with six applications.
Running an Evaluation That Surfaces the Truth
Demos are optimized. Structure the evaluation to break that.
Bring your own content — a real repository with real permission complexity — and index it during the evaluation rather than after. Bring a real evaluation set of questions with known correct answers, including the awkward ones, and score outputs rather than reading them impressionistically. Ask to see the audit log for a query you just ran. Ask to swap the model on a working application and watch how long it takes.
Then check the boring commitments: data residency in writing, retention terms, breach notification timelines, export mechanisms for your data and configurations, and what happens contractually if the vendor is acquired.
A platform decision constrains everything built on top of it for years, which is why it deserves more rigor than the applications it will host. If you are mapping how a platform fits alongside the tools you already run, the architecture considerations in our guide to the AI tech stack are a useful companion. And if you want an evaluation grounded in your actual data and permission model rather than a demo tenant, the Workisy enterprise AI platform team runs assessments that start there.
