AI crawler analytics

Manual setup

Your own server sends its request logs to ByDefault over HTTP. For any host.

JoshUpdated 2 min read

How it works

Send your request logs to ByDefault from anywhere.

In ByDefault, open Analytics, click Connect your site, then pick Manual setup. It has your API key and tells you when your first event arrives.

Send your request logs

Send your HTTP request logs from your server like this:

Request

The API key is your token from the setup. Send up to 1,000 events in one request as a JSON array, and up to 4 MB per request.

Format each event

Request body

Every event needs every property. query_string and referrer can be null, and timestamp must be within the last 72 hours. You don't need to filter for AI traffic, since ByDefault does that. Just skip static assets like images, scripts and fonts.

client_ip is required because ByDefault checks each event against the provider's official IP ranges, so fake crawlers don't count. IPs are never stored, and events that aren't AI traffic are discarded.

The response says what happened to each event:

Response

Send only AI traffic

If your request volume or data policy calls for filtering on your side, send only requests whose User-Agent matches one of these, case-insensitive:

User-agent patterns

Check the connection

While the setup is open, ByDefault visits your site the way an AI assistant would, every few seconds. Its visits identify as Claude-User. Once your pipeline sends one of them to ByDefault, the last step turns green.

Troubleshooting

It says Events are arriving but never turns green

Your events reach ByDefault, but none was an AI visit yet. Make sure user_agent is copied from the request verbatim, so ByDefault's Claude-User visit is recognized.

It says client_ip fails the IP check

client_ip must be the visitor's IP. Behind a load balancer or CDN, take it from the header that carries the original client, like X-Forwarded-For, not the address of the proxy in front of you.

Did this answer your question?

Turn AI search into your most profitable sales channel.