I’ve spent the final a number of years watching enterprise collaboration instruments get smarter. Be part of a video name in the present day, and there’s probability 5 or 6 AI brokers are working concurrently: transcription, speaker identification, captions, summarization, activity extraction. On the product aspect of it, every agent will get evaluated in isolation. Separate dashboards, separate metrics. Transcription accuracy? Test. Response latency? Test. Error charges? All inexperienced.
However here’s what I constantly observe as a UX Researcher: customers are annoyed, adoption stalls, and groups are attempting to determine the foundation trigger. Per the metrics, the dashboards look nice. Each particular person element passes its exams. So, the place are customers actually struggling?
The reply, virtually each time, is orchestration. The brokers work nice alone. They disintegrate collectively. And the one manner I’ve discovered to catch these failures is thru person expertise analysis strategies that engineering dashboards have been by no means designed to seize.
The Orchestration Visibility Hole
Right here’s an instance of gaps that want a deeper understanding via person analysis: a transcription agent experiences 94% accuracy and 200-millisecond response occasions. However what the dashboard doesn’t present is that customers are abandoning the function as a result of two brokers gave them conflicting details about who mentioned what in a gathering. The transcription agent and the speaker identification agent disagreed, and the person misplaced belief in the entire system.
This downside is about to get a lot larger. Proper now, fewer than 5% of enterprise apps have task-specific AI brokers in-built. Gartner thinks that’ll soar to 40% by the tip of 2026. We’re headed towards a world the place a number of brokers coordinate on virtually the whole lot. If we can not work out easy methods to consider orchestration high quality now, we will probably be scaling damaged experiences.
UX Analysis Strategies Tailored for Agent Analysis
Customary UX strategies want some tweaking if you find yourself coping with AI that behaves in another way every time. I’ve landed on three approaches that really work for catching orchestration issues.
1. Assume-Aloud Protocols for Agent Handoffs
In conventional think-aloud research, you ask folks to relate what they’re doing. For AI orchestration, I layer in what I name system attribution probes at key handoff factors. I pause and ask individuals to explain what they imagine simply occurred behind the scenes, then map their responses towards the precise agent structure. Most customers are unaware that separate brokers deal with transcription, summarization, and activity extraction. When one thing goes fallacious: a transcription error, as an example, they blame “the AI” as a monolith, even when the summarization and routing labored completely. Person suggestions alone gained’t get you there. What I’ve discovered works is mapping what folks assume the system simply did towards what really occurred. The place these two diverge, that’s the place orchestration is failing. That’s the place the design work must occur
2. Journey Mapping Throughout Agent Touchpoints
Take into account a single video name. The person clicks to hitch, and a calendar agent handles authentication. A speech-to-text agent transcribes, a show agent renders captions, and when the decision ends, a summarization agent writes up the assembly whereas a activity extraction agent pulls out motion gadgets. A scheduling agent would possibly then ebook follow-ups. That’s six brokers in a single workflow and 6 potential failure factors.
I construct dual-layer journey maps: the person’s expertise on high, the accountable agent beneath. When these layers fall out of sync – when customers anticipate continuity however the system has handed off to a brand new agent; that’s the place confusion units in, and the place I focus my analysis to unpack deeper points.
3. Heuristic Analysis for Agent Transparency
Nielsen Norman’s traditional heuristics stay foundational, however multi-agent techniques require us to increase them. “Visibility of system standing” has a unique that means when six brokers are working concurrently; not as a result of customers want to know the underlying structure, however as a result of they want sufficient readability to get better when one thing goes fallacious. The aim isn’t architectural transparency; it’s actionable transparency. Can customers inform what the system simply did? Can they appropriate or undo it? Do they know the place the system’s limitations are? These standards reframe orchestration as a UX downside, not simply an infrastructure concern.
I’ve run heuristic evaluations the place the interface was polished and interplay patterns felt acquainted, but customers nonetheless struggled. The floor design handed each conventional verify, however when the system failed, customers had no approach to diagnose what went fallacious or easy methods to repair it. They didn’t must know which agent triggered the difficulty. They wanted a transparent path to restoration.
Case Research: Enterprise Calling AI
Right here’s an actual state of affairs I labored on that illustrates why orchestration high quality can matter as a lot as particular person agent efficiency.
An enterprise calling platform had deployed AI for transcription, speaker identification, translation, summarization, and activity extraction. Each element hit its efficiency targets. Transcription accuracy was above 95%. Speaker identification ran at 89% precision. Process extraction caught motion gadgets in 78% of conferences. Nevertheless, person satisfaction was at 3.2 out of 5, and solely 34% of eligible customers had adopted the AI options. The product crew’s intuition was to enhance the fashions. I suspected the issue was in how the brokers labored collectively.
We ran think-aloud classes and found one thing the dashboards by no means confirmed: customers assumed that edits they made to reside captions would carry over to the ultimate transcript. They didn’t. The techniques have been utterly separate. After I constructed out the journey map, plotting person actions on one layer and agent accountability on one other, I seen the timing misalignment instantly. Motion gadgets have been arriving in customers’ activity lists earlier than the assembly abstract was even prepared. On the person layer, this regarded like duties showing out of nowhere. On the agent layer, it was merely the duty extraction agent ending earlier than the summarization agent. Each have been performing appropriately in isolation. The orchestration made them really feel damaged.
Heuristic analysis surfaced a subtler subject: when the interpretation and transcription brokers disagreed about speaker identification, the system silently picked one. No indication, no confidence sign, no manner for customers to intervene.
This pointed us towards a design speculation: the issue wasn’t agent accuracy, it was coordination and recoverability. Slightly than foyer for mannequin enhancements, we centered on three orchestration-level adjustments. First, we synchronized timing so summaries and duties arrived collectively, restoring context. Second, we constructed unified suggestions mechanisms that allow customers appropriate outputs as soon as quite than per-agent. Third, we added standing indicators exhibiting when handoffs have been occurring.
Three months later, adoption had jumped from 34% to 58%. Satisfaction scores considerably improved with scores of 4.1 out of 5. Help tickets about AI options dropped by 41%. We hadn’t improved a single mannequin. The engineering crew didn’t assume UX adjustments alone might transfer these numbers. Truthful sufficient, actually. However three months of knowledge made it onerous to argue. Agent coordination isn’t simply an infrastructure downside. It’s a UX downside, and it deserves that degree of consideration.
A Three-Layer Analysis Framework
Based mostly on what I’ve seen throughout a number of deployments, I now advocate evaluating orchestration on three ranges. Layer one is technical metrics: latency, accuracy, and error charges for every agent. You continue to want these. They catch component-level failures. However they can’t see coordination issues.
Layer two is behavioral alerts. Monitor the place customers abandon workflows, how usually they revise AI-generated outputs, and whether or not they come again after their first expertise. These patterns trace at orchestration points with out requiring direct person suggestions.
Layer three is qualitative analysis. Do customers perceive what the brokers are doing and why are they doing it? Do they belief the outputs? Does the entire system really feel coherent and accessible or disjointed? McKinsey’s 2025 AI survey discovered that 88% of organizations use AI someplace, however most haven’t moved previous pilots with restricted enterprise influence (McKinsey, 2025). I believe an enormous a part of that hole comes from orchestration high quality that no one is measuring correctly.
What This Means for Product Groups
In most organizations I’ve labored with, UX researchers and AI engineers have restricted collaboration. Engineers tune particular person brokers towards benchmarks. UX researchers take a look at interfaces. No one owns the area between brokers the place coordination occurs. That hole is strictly the place these failures reside.
Deloitte estimates {that a} quarter of firms utilizing generative AI will launch agentic pilots this 12 months, with that quantity doubling by 2027 (Deloitte, 2025). Groups that implement orchestration analysis early may have an actual benefit. Groups that don’t will maintain questioning why their AI options will not be touchdown with customers. The funding required shouldn’t be huge. It consists of UX researchers in orchestration design discussions, constructing telemetry that captures agent transitions, and working common research centered particularly on multi-agent workflows.
Conclusion
As AI merchandise evolve from single assistants to coordinated agent techniques, the definition of “working” has to evolve with them. A set of brokers that every cross their particular person benchmarks can nonetheless ship a damaged person expertise. Efficiency dashboards gained’t catch it as a result of they’re measuring the fallacious layer. Person complaints gained’t make clear it as a result of folks blame “the AI” with out realizing which element failed or why.
That is precisely the place UX analysis earns its seat on the desk. Not as a remaining verify earlier than launch, however as a self-discipline woven all through the product lifecycle. UXR helps groups reply the earliest questions: Are we fixing the precise downside? Who’re we fixing it for? It shapes success metrics that replicate actual person outcomes, not simply mannequin efficiency. It evaluates how brokers behave collectively, not simply in isolation.
UX analysis exhibits you what earns belief and what chips away at it. It makes positive accessibility will get in-built from the beginning, not bolted on later when the system is simply too tangled to repair correctly. None of that is separate work. It’s all linked, every layer feeding into the subsequent. And as AI techniques get extra autonomous, extra opaque, this sort of rigor isn’t non-compulsory. The issue is, when groups are shifting quick, analysis seems like a velocity bump. One thing to circle again to after launch.
However the price of skipping it compounds rapidly. The orchestration issues I’ve described don’t floor in QA. They floor when actual customers encounter actual complexity, and by then, belief is already broken.
AI techniques are solely getting extra advanced, extra autonomous, and extra embedded in how folks work. UX analysis is how we maintain these techniques accountable to the folks they’re meant to serve.
Continuously Requested Questions
This is without doubt one of the most typical frustrations I see in enterprise AI. Particular person brokers cross their benchmarks in isolation, however the actual issues present up when a number of brokers must work collectively. Orchestration failures occur on the handoffs, like when a transcription agent and speaker identification agent disagree about who mentioned what, or when activity extraction finishes earlier than summarization, and customers obtain motion gadgets with no context.
These coordination points by no means seem on component-level dashboards as a result of every agent is technically doing its job. That’s exactly why person analysis strategies are important. They floor the place the expertise really breaks down in ways in which engineering metrics weren’t designed to catch.
Acquainted strategies like think-aloud protocols and journey mapping nonetheless work, however they want some changes for AI techniques. In think-aloud research, I’ve discovered it invaluable to incorporate what I name system attribution probes, moments the place you pause and ask customers to explain what they imagine simply occurred behind the scenes. Journey maps profit from a dual-layer method: the person expertise on high and the accountable agent beneath.
Orchestration issues lie the place these layers are out of sync, and analysis ought to give attention to figuring out and evaluating these points.
Longitudinal and ethnographic analysis are essential to know AI agent efficiency over time. Strategies like diary research and ethnography allow researchers to guage how customers work together with the AI and shift their utilization patterns throughout days or perhaps weeks, how that impacts belief, and determine new points which will emerge.
Preliminary impressions of an AI system usually differ from a person’s expertise after steady utilization. Longitudinal research reveal behaviors and workarounds that customers develop, and touchpoints that contribute to customers abandoning the function totally.
Based mostly on what I’ve noticed throughout a number of deployments, I like to recommend evaluating orchestration on three ranges. Layer one covers the technical metrics resembling latency, accuracy, and error charges for every agent.
Layer two focuses on behavioral alerts resembling workflow abandonment charges, how usually customers revise AI-generated outputs, and if they’re returning customers. These patterns trace at orchestration points with out requiring direct person suggestions.
Layer three is qualitative analysis that evaluates if customers really belief the outputs, perceive what the brokers are doing, and understand the system as coherent quite than disjointed. All three layers working collectively reveal issues that any single layer would miss.
Actionable transparency shouldn’t be about educating customers the underlying structure of each agent. Customers want readability and the flexibility to know what the system simply did, appropriate or get better from errors when one thing seems to be incorrect, and perceive the place the system’s limitations are.
Actionable transparency offers customers clear paths to get better from errors.
When errors happen, customers have to be knowledgeable about what their choices are for resolving the difficulty and easy methods to transfer ahead. In follow, this might be unified suggestions mechanisms to let customers appropriate outputs as soon as, quite than individually for every agent. It is also standing indicators that floor when handoffs are occurring, or undo performance that works throughout the whole system. The aim is to design for recoverability. When orchestration breaks down, customers can regain management and belief.
Crucial shift is recognizing that the area between brokers, the place coordination occurs, wants an proprietor. In most organizations I’ve labored with, engineers tune particular person brokers towards benchmarks whereas UX researchers take a look at interfaces. No one owns that hole, and that’s precisely the place orchestration failures are likely to reside.
To shut this hole, groups ought to deliver UX researchers into orchestration design discussions early, not simply on the finish for interface testing. They need to construct telemetry that captures agent transitions and handoff factors, not simply particular person agent efficiency. They need to run common research centered particularly on multi-agent workflows quite than treating AI as a single monolithic function. This does require intentional cross-functional collaboration to construct higher AI-products.
Priyanka Kuvalekar is a Senior UX Researcher at Microsoft, main mixed-method analysis for Microsoft Groups Calling and agentic AI collaboration experiences
Login to proceed studying and revel in expert-curated content material.
