Known Agents

What is Known Agents?

Known Agents is a website analytics and control platform for monitoring AI agents, LLM assistants, and other bots that crawl your site. Its core purpose is to make this “hidden bot traffic” visible in real time and help you manage how bots and agents access your content.

Instead of treating all automated traffic the same, Known Agents provides agent identification, replayable navigation timelines, and observability for specific agent traffic sources and endpoints—along with tools to generate and continuously update robots.txt rules and to block misbehaving bots.

Key Features

AI agent & bot analytics with real-time visibility: See crawlers, scrapers, and AI agents visiting your website, including what pages they target.
Agent session replay: Follow an individual agent’s navigation to identify where it gets stuck, where it loops, or where it reaches dead ends.
LLM referral tracking (GEO/AEO): Track citations and click-throughs from AI chat platforms (e.g., ChatGPT, Claude, Gemini) to measure which pages get referenced and which get visited.
MCP & shopping observability: Monitor agentic traffic to MCP endpoints and evaluate commerce funnel performance metrics such as UCP and ACP.
Automatic robots.txt protection: Automatically serve a robots.txt that updates continuously, adding rules as new agents emerge without manual edits.
Bad bot detection and blocking: Detect bots that ignore robots.txt rules and block misbehaving scrapers/automation attempts; WordPress plugin support is mentioned.
Agent Identification API (Web Bot Auth): Authenticate agents from your own code and implement access control for licensed content via “Web Bot Auth.”

How to Use Known Agents

Connect Known Agents to your site using one of the available options mentioned on the website: WordPress, your CDN, or the Known Agents API.
Start viewing bot and agent activity in the analytics interface to identify the agents visiting your site, the pages they target, and traffic spikes.
Enable automatic robots.txt controls so your robots.txt rules update as new agent categories are discovered.
Use bad bot detection (including the WordPress plugin option) to block bots that do not follow your robots.txt rules.
If you publish licensed content or need controlled access, integrate the Agent Identification API from your own code to authenticate agents and enforce access control.

Use Cases

Identify traffic from AI chat references and measure citation performance: Use LLM referral tracking to see which pages are cited in AI chat responses and how many humans click through from those citations.
Debug why an agent can’t complete a task: Replay a single agent session to find where it gets stuck or fails to reach key pages, then adjust content structure or routing.
Spot spoofed crawler behavior and protect resources: Use the realtime timeline to detect cases where a crawler appears legitimate but is actually spoofed (including examples involving HTTP 403 rejections).
Control access for licensed material: Authenticate agents with the Agent Identification API (Web Bot Auth) and implement access restrictions for specific agent categories.
Monitor agentic commerce and endpoint activity: Use MCP & shopping observability to watch how agents interact with MCP endpoints and to assess UCP/ACP funnel outcomes.

FAQ

Does Known Agents work with WordPress?

Yes. The website mentions a WordPress plugin for enabling detection and blocking of bots that ignore robots.txt rules, and also notes connectivity with WordPress.

Can Known Agents automatically update robots.txt rules?

Yes. Known Agents provides Automatic Robots.txt, described as continuously updating and adding newly discovered scrapers without requiring manual edits.

What kinds of agents can Known Agents track?

The site describes tracking crawlers, scrapers, and AI agents, including references to AI chat platforms (for LLM referral tracking) and agentic traffic to MCP endpoints.

How does Known Agents help with misbehaving bots?

It includes bad bot detection that blocks bots that ignore robots.txt rules, and it provides a realtime view (timeline) showing when requests are accepted or rejected by your server.

Is there an option to authenticate agents programmatically?

Yes. Known Agents mentions an Agent Identification API using Web Bot Auth to authenticate agents from your own code and implement access control for licensed content.

Alternatives

General-purpose web analytics plus bot filtering: Tools that focus on standard traffic analytics can be combined with bot filtering, but they typically don’t provide agent-specific observability (e.g., session replay for AI agents, LLM citation referral tracking, or agent authentication).
Web security and bot management platforms: Security-focused bot management solutions can help block or challenge suspicious automation, but may not offer the same LLM/agent-specific tracking of citations, navigation patterns, or robots.txt auto-updating by agent category.
Robots.txt management + crawler analytics via logs: Managing robots.txt manually and analyzing server logs can reveal crawler activity, but it requires more manual upkeep and won’t provide the same structured LLM referral and agent session replay features described for Known Agents.
CDN/WAF-only controls: Using only CDN or WAF rules can reduce unwanted traffic, but it generally doesn’t cover the specific analytics needs for AI agents (e.g., GEO/AEO-style LLM referral tracking and MCP endpoint observability).