TLDR;
Visa tested 22 AI models on realistic shopping tasks and says they failed 42.8% of runs. More worrying for retailers, 54% of failed runs still ended with a purchase. The benchmark does not yet test live checkout or payment, but it shows why brands need better product data, explicit clarification steps and approval controls before treating AI recommendations as dependable buying decisions.
What happened
Visa launched its Agentic Commerce Benchmark on 5 November 2026. The first round put 22 models from 12 labs through 15 scenarios covering products such as televisions, laptops and personal-care appliances. Each task withheld some customer requirements, so the agent had to uncover them through conversation before making the correct buy or no-buy decision.
Visa reported a 57.2% overall pass rate, which means 42.8% of runs failed. The best models passed 82.2% of attempts and the worst 28.9%. Models that uncovered the most hidden buying factors averaged a 70.2% pass rate, compared with 44.4% for those that uncovered the fewest. The interactive Visa Payments Frontier site lets people explore the benchmark and view run replays.
Consistency was the sharper warning. Every model received three attempts at each scenario, but all three attempts passed only 34.2% of the time. Visa says 54% of failed runs still ended in a purchase, either because the chosen product missed an identified requirement or because the agent bought before finding every requirement. Independent reporting added model-level rankings that Visa's announcement did not name, but the published headline figures remain Visa-designed and Visa-reported.
Why it matters
The benchmark separates product discovery from payment security. A transaction can be properly authorized and still be commercially wrong because the agent misunderstood compatibility, budget, membership eligibility, returns or the shopper's intended use. For retailers, that failure can become an avoidable return, support case, dispute or damaged customer relationship rather than a simple recommendation error.
The result also challenges the idea that richer models remove the need for structured commerce data. The five most expensive models passed more often than the five cheapest, but neither group was consistently correct. Agents still need clear attributes, variant relationships, inventory, regional eligibility, delivery promises, warranty terms and machine-readable policies. Missing facts force the model to infer, ask badly or buy too early.
TNW's account highlights the same reliability gap: models that discovered more requirements performed better, yet repeat attempts remained inconsistent. That makes a single successful demo weak evidence for an enterprise rollout. Brands need repeated scenario tests across categories, languages and edge cases, plus a measurable abstention rule when no product satisfies the customer's constraints.
How your brand can benefit / be affected
Start with the categories where a wrong choice creates the highest cost. Build test prompts around compatibility, regulated claims, subscription terms, delivery deadlines, membership pricing and returns. Score whether the agent asks for every decision-critical fact, cites the relevant product evidence and declines to buy when the catalogue has no valid match. Treat a confident but incomplete purchase as a failure.
Improve the evidence layer before adding more agent integrations. Give each variant stable identifiers and complete attributes, keep stock and price current, expose exclusions and policies in accessible text, and make equivalent facts consistent across pages, feeds and APIs. Log which facts the agent received so teams can distinguish a model failure from missing or contradictory merchant data.
Keep delegated purchasing bounded while the market matures. Require human confirmation for high-value, irreversible or ambiguous orders, preserve the shopper's stated constraints with the order record, and monitor agent-led returns, cancellations and support contacts separately. Visa says later benchmark rounds will cover checkout, payment and order placement. Until those results exist, this launch is evidence about product-choice reliability, not proof that end-to-end agentic commerce is ready to run unattended.
News date: 5 November 2026. Editorial review: 6 November 2026.