TLDR;
Cloudflare has launched a Web Search API in open beta through AI Gateway. Developers can retrieve live results from Ceramic.ai, Exa or Linkup, while Cloudflare requires participating crawlers to identify themselves, respect site rules and return source URLs. For brands, this creates another route into AI answers, but access alone will not guarantee retrieval, citation or a visit.
What happened
Cloudflare announced the Web Search API on 2 October. It gives AI applications a direct search call that returns structured titles, URLs and descriptions, which a developer can pass into a model as current context. The open beta is available through a REST endpoint or a Cloudflare Workers binding and is managed through AI Gateway.
The service starts with three search providers: Ceramic.ai, Exa and Linkup. Cloudflare's product documentation says all three return a common result format, so an application can switch provider without rewriting the integration. The underlying indexes and retrieval methods still differ, which means the same query may return a different set of brands, pages and passages depending on the provider selected.
Cloudflare has attached crawler conditions to the launch. Its announcement says providers must meet Verified Bot requirements and include a link to the crawled source in search results. Those requirements cover honest identification, respect for robots.txt and crawl directives, reasonable request rates and no attempts to evade a site owner's stated preferences.
AI Gateway supplies the operational layer around the search call. Teams can pay with gateway credits or bring their own provider key, set access controls and inspect requests in logs. Cloudflare lists provider pricing and data-retention status separately, so the beta is a common interface rather than a single Cloudflare search index or one uniform privacy contract.
Why it matters
This turns web retrieval into a replaceable infrastructure component. An AI product team can change search provider for cost, latency or result quality while leaving the model and user interface untouched. A brand may therefore be visible through one index and absent from another even when both crawlers can access the site. Monitoring only one assistant or one user agent will miss that layer.
The source-link rule is useful but limited. The API returns the location of the material it retrieved. The application developer still decides which results reach the model, how excerpts are combined and whether source links appear in the final answer. Brands should separate crawler access, retrieval, citation and referral as four different outcomes instead of treating a successful crawl as proof of visibility.
The verified-crawler requirement also gives access policy more commercial weight. Blocking an unidentified scraper and allowing a search crawler are no longer the same decision. A blanket bot rule can remove useful discovery access, while an unrestricted rule can expose content to purposes the brand did not intend. Search, user-directed agents, model training and transactions need their own policies.
How your brand can benefit / be affected
Review robots.txt, CDN controls and origin rules for the verified search providers your customers or internal applications may use. Confirm that the intended public pages return the same useful content to an authorised crawler as they do to a browser. Test category pages, product details, technical documentation and research pages separately, because a domain-level allow rule can hide path-level failures.
Run retrieval tests across all three providers before drawing conclusions about AI visibility. Use stable queries, record the provider, returned URLs, snippets and timestamp, then check whether the downstream model cites the same source. Publish clear titles, canonical URLs, self-contained product facts and dated evidence so a search layer can identify the page and a model can use the passage without inventing missing context.
For teams building AI products, keep the search trace. Cloudflare's logging documentation shows that AI Gateway can record provider, prompt, response, cost and duration. Add the user's query, selected sources, final citations and any tool error to your own evaluation record. Treat retrieved text as untrusted input, and test freshness, prompt injection and citation loss before using it in customer-facing answers.
Finally, keep search permission separate from commercial reuse. A verified crawler that respects robots.txt is not evidence that the brand accepted model training, automated transactions or unpaid reuse. Maintain explicit policies for those purposes and revisit them as Cloudflare's Pay Per Use and crawler controls mature. The new API gives enterprises a cleaner search channel, but the brand still has to define what that channel may do.
News date: 2 October 2026. Editorial review: 3 October 2026.