The State of AI Discovery — September 2026

By Adam Dean · September 8, 2026 · NessGate Blog

How do AI systems actually find out about your business? We pulled the published data on every major mechanism — adoption numbers, real crawler behavior, and the standards war that started this summer. Some of it contradicts what the industry is selling.

All figures are sourced and dated; sources at the end. Disclosure: NessGate operates a domain-verification registry in this space — we say so again where it's relevant, and this report includes data that is inconvenient for us too.

The short version

  1. Publishers are moving faster than AI consumers. Businesses are adding AI-discovery files at a growing rate — and the major AI crawlers are, for the most part, not reading them yet.
  2. Conformance is the silent failure. Where machine-discovery standards are adopted, most published artifacts are broken — a majority of real-world A2A agent cards can't be consumed by an actual A2A client.
  3. The standards war began in earnest this summer. Google, Microsoft, AWS, and the Linux Foundation all shipped or backed discovery standards between May and August 2026. Nobody knows which wins. Anyone who tells you otherwise is selling something.

llms.txt: adopted by publishers, ignored by crawlers

llms.txt — a plain-text guidance file for AI assistants at /llms.txt — is the mechanism most businesses have heard of. The 2026 numbers:

Honest takeaway: llms.txt is cheap to add and might matter later, but today it is a bet on the future, not a lever on the present. Anyone promising that llms.txt alone will get you "seen by AI" is ahead of the evidence.

A2A agent cards: the conformance disaster

Google's A2A protocol gives agents a standard identity file at /.well-known/agent-card.json. In July 2026, API Evangelist surveyed 22,341 API providers:

That's the finding that generalizes: even among the earliest, most motivated adopters, a majority get the spec wrong. Machine-discovery files look simple and are quietly easy to break — a wrong content type, a missing required field, an HTML-returning catch-all route — and there's no error message anywhere when you do. Your file just silently doesn't work.

MCP: the one layer with real adoption

For calibration, the Model Context Protocol shows what genuine adoption looks like: 97 million-plus monthly SDK downloads and roughly 10,000 active servers in the official registry (per the Digital Applied MCP adoption tracker, May 2026). But note what MCP is: developer tooling for connecting AI to systems — not a way for an ordinary business to describe itself to the AI ecosystem. The business-facing discovery layer is still unsettled.

The standards war: May–August 2026

Then the giants moved, all at once:

StandardBackersModelStatus
ARD (Agentic Resource Discovery)Google, Microsoft, AWS, Cisco, GitHub, GoDaddy, Nvidia, Salesforce, ServiceNow, Hugging Face, othersYou publish an ard.json catalog on your own domain; federated registries index itSpec v0.91, Proposal (Aug 2026)
DNS-AIDLinux Foundation: Cloudflare, GoDaddy, Infoblox, ISC, Equinix, othersDiscovery records in DNS itself; explicitly anti-centralized-registryIETF standards track (draft, 2026)
ANS (Agent Name Service)GoDaddy (+ Cloudflare partnership)Registrar-anchored verified identity for agents, with a marketplaceLive product
ACP / UCP (commerce)OpenAI + Stripe / GoogleProduct feeds and agent checkoutLive at retail scale (Shopify 1M+ merchants; major US retailers)

Three honest observations:

First, they conflict. ARD assumes registries; DNS-AID exists to make registries unnecessary; ANS anchors trust in registrars. These reflect different corporate interests, and businesses are being asked to pick sides in a war whose outcome nobody knows.

Second, they all agree on one thing: the domain is the unit of identity, and what you publish under a domain you control is the root of machine trust. Every standard above is a variation on "put a verifiable artifact where your domain authority covers it."

Third, commerce shows what winning looks like — and why the rest is unsettled. ACP and UCP got adoption fast because two platforms (OpenAI, Google) control both sides of the transaction. General business discovery has no such forcing function yet, which is exactly why the publisher-side standards are still jockeying.

What a sane business does in September 2026

Given the data, we think the honest advice is:

  1. Don't bet on a single standard. The war just started; llms.txt shows how a "must-have" can turn out to be ignored.
  2. Do publish the cheap artifacts correctly. Everything above costs minutes if generated correctly and nothing to maintain — the risk isn't effort, it's the silent-breakage problem the A2A numbers demonstrate.
  3. Anchor everything in your domain. Every credible standard resolves to domain authority. Files on infrastructure you control, verifiable as yours, survive whichever protocol wins.
  4. Treat "AI visibility" vendors' urgency claims skeptically — including ours. The consumer side (the AI agents actually reading these artifacts at scale) is still arriving. The case for acting now is being early and correct, not that agents are already choosing businesses by these files today.

Where NessGate fits (disclosure)

NessGate is a free, neutral registry where a domain owner verifies control once, declares their official machine-facing resources (any format, any protocol — including ones that don't exist yet), and gets a spec-conformant ARD catalog generated automatically to host on their own site. We store pointers, never content; we verify authority, never truth. Our registry is young and we say so. You can check what AI can discover about your domain right now — it's free and stores nothing.

Sources