Three genuinely different products are sold under the same label. Intent-based dialog builders, where a designer maps utterances to intents and hand-authors every reply. LLM-native assistants, where answers are generated from indexed source content at runtime. And orchestration platforms, which route a conversation between deterministic flows, generative answers, backend systems, and human agents depending on what the request turns out to be. Every vendor in all three categories describes itself as a conversational AI platform.
That ambiguity is expensive because switching costs here are unusually high. Conversation content, integration middleware, analytics history, and agent-facing tooling all get rebuilt on migration. A platform decision made on a demo script tends to survive three to five years whether or not it was correct.
This guide is structured as an evaluation sequence rather than a feature tour. Each section covers one criterion, what separates a strong answer from a rehearsed one, and the specific question to put to the vendor.
Start With Architecture, Not Features
Ask the vendor to draw the request path for a single customer message: where it enters, what classifies it, what generates the response, what systems it touches, and where the human handoff sits. The diagram tells you more than the feature matrix.
The critical question is whether generative answering and deterministic flows are peers in the same runtime or whether one is bolted onto the other. Platforms that started as intent trees often treat the LLM as a fallback for unmatched utterances — meaning the generative layer only ever sees the queries the rules failed on, which is exactly the traffic where grounding quality matters most. Platforms that started generative sometimes lack the strict flow control you need for a transactional path such as processing a return or verifying identity, where you cannot tolerate improvisation.
The strongest architectures let a single conversation move both ways: a generative turn that recognizes a transactional intent hands off to a deterministic flow, completes it, and hands control back. Ask to see that transition live, not in slides.
Language Understanding That Survives Real Traffic
Demo traffic is clean. Production traffic contains typos, three questions in one message, code-switching between languages, pasted error logs, and customers who open with an emotional complaint before stating the actual problem.
Test it with your own data. Pull 200 real messages from your existing support queue, strip identifiers, and require the vendor to run them through a trial workspace. Score four things: correct routing, correct answer, honest abstention when the answer is not in the knowledge base, and hallucination rate. Abstention is the one buyers skip and the one that determines whether you can safely leave the assistant unsupervised.
For generative platforms, retrieval quality matters more than model choice. Ask how documents are chunked, whether the index supports metadata filtering by product line or customer tier, how often it re-indexes after a content change, and whether every generated answer carries citations back to the source passage. A platform that cannot show you the sources for an answer cannot be audited when the answer is wrong.
Channel Coverage, and Channel Fidelity
Coverage counts are easy to inflate. A vendor claiming twelve channels usually means one strong channel, three adequate ones, and eight thin API wrappers.
Fidelity is what matters per channel. Web chat should support rich cards, file upload, typing indicators, and session persistence across page navigation. WhatsApp has template-message rules and a 24-hour service window that constrain proactive outreach. Email requires threading and quoted-history handling that most chat-first platforms do badly. Voice is a separate engineering problem entirely, with its own latency and telephony constraints, and should be scoped separately rather than checked off as a thirteenth channel.
The question to ask: is conversation state shared across channels? A customer who starts on web chat, drops off, and messages on WhatsApp two hours later should not restart. Cross-channel identity resolution is a real capability that a minority of platforms actually have.
The Integration Layer Is the Whole Product
An assistant that can only answer questions caps out fast. The value shows up when it can check an order, reset a password, apply a credit, or update an address — which means calling your systems, with authentication, error handling, and rollback.
Evaluate three things. First, the connector library: pre-built connectors to your CRM, ticketing, and commerce platform save months, but verify they are read-write and not read-only. Second, the custom action framework: how a developer defines a new tool the model can call, including parameter validation, timeouts, and what the assistant says when the API returns a 500. Third, the identity and authorization model: how the platform verifies a caller before exposing account data, and whether permissions are enforced server-side rather than by prompt instruction.
Platforms that only enforce restrictions through system prompts are not secure. Ask explicitly where authorization is checked.
Authoring, Testing, and Change Control
Conversational content behaves like software and needs the same discipline. Look for versioned environments with a promotion path from draft to staging to production, a regression suite that replays saved conversations against a proposed change, and a rollback that takes seconds.
Ask what happens when a subject-matter expert edits a help article. On a well-built platform, the change propagates to the index automatically and the regression suite flags any answer it altered. On a poorly built one, someone re-uploads a file and nobody finds out the answer changed until a customer complains.
Also ask who can safely make changes. If every content edit requires a developer, your operating cost is higher than the license fee suggests.
Analytics That Answer Business Questions
Most dashboards report volume, sessions, and a containment percentage. Those are activity metrics, and containment in particular is easy to game — a bot that stonewalls users into abandonment scores beautifully.
The reporting you actually need answers: which topics drove escalation this week and why; which knowledge gaps caused abstention, ranked by frequency; whether resolved conversations produced a repeat contact within seven days; and how satisfaction on AI-handled conversations compares to human-handled ones for the same intent. Repeat-contact rate is the single most honest measure of whether a resolution was real.
Ask whether raw conversation data is exportable to your warehouse. If analysis is locked inside the vendor's dashboard, you cannot join it to revenue, churn, or cost-to-serve, and the business case stays anecdotal. This is the same discipline that separates a serious AI assistant deployment from a novelty pilot: the measurement plan exists before the launch date does.
Pricing Models and Where Cost Actually Lands
Four models dominate, and they distribute risk very differently.
Per-session pricing is predictable but punishes engagement — teams start discouraging useful conversations. Per-resolution pricing aligns vendor incentives with outcomes, but the definition of "resolution" is negotiable and should be nailed down in the contract, including whether an escalated conversation counts. Per-seat pricing suits agent-assist deployments but scales badly for self-service. Consumption pricing on tokens or API calls is transparent but volatile, and a single prompt change can move the bill 40%.
Whatever the headline model, model total cost over three years including implementation services, connector development, the internal headcount to maintain content, and the price of the tiers you will grow into. Vendors quote the entry tier; SSO, audit logs, data residency, and sandbox environments frequently live two tiers up.
Build Versus Buy, Honestly
Building on foundation model APIs directly is more viable than it was two years ago, and it is still the wrong choice for most support and service use cases. What you buy from a platform is rarely the model. It is the conversation state machine, the channel adapters, the agent handoff console, the analytics pipeline, the content authoring surface, and the compliance posture — the parts nobody budgets for and everybody rebuilds badly.
Building makes sense when the conversational experience is the product itself, when your domain requires model behavior no vendor can configure, or when data residency rules exclude every viable vendor. It rarely makes sense to build a support assistant. Teams that have automated adjacent processes tend to reach the same conclusion; the reasoning parallels what happens in broader HR automation programs, where the differentiated work is process design, not infrastructure.
The Shortlist Scorecard
Score each vendor one to five on: architecture flexibility, accuracy on your own test set, abstention behavior, channel fidelity for your top two channels, write-capable integrations, authoring and rollback, exportable analytics, pricing predictability, and security posture including SOC 2, data residency, and PII redaction. Weight accuracy and integrations at double.
Then require a paid pilot on real traffic for one narrow intent family before signing a multi-year agreement. Two weeks of production data will disqualify more vendors than two months of demos.
If you are working through this evaluation now, a walkthrough of a working conversational AI platform against your own transcripts is more useful than another feature comparison, and a short conversation with our team can tell you quickly whether the architecture fits your stack.
