AI crawlers seen by your server

GPTBot, ClaudeBot, PerplexityBot, CCBot and most other AI crawlers do not run JavaScript, so no tracker, ours included, ever sees them. Your server does: it answers every request. With the WordPress plugin or a few lines of code it tells SimpleData which AI agents read your site, and which pages. On every plan.

Requests, not visits

• Each figure is a request your server answered to a known AI agent. It is a different measure from your visits: it is not in your visitors, page views, your plan's monthly page views, the email alerts or the weekly report.
• Only known AI agents count, by the same rules as AI traffic. Anything else your server reports, a person's request included, is dropped as it arrives.
• Only the page's path is kept, without the query string, as a count per day. Neither the crawler's IP address nor its user agent is stored, and nothing about your visitors is sent.
• Counted per day in UTC. A period takes the UTC days that fall mostly inside it, so its first and last day may not match your site's clock to the hour. Dashboard filters do not apply: there is no visitor behind a daily count.
• Up to 50 pages per day and purpose get a line of their own; past that, the rest of each agent's requests that day add up as "Other pages". Pages you exclude in your site's settings are left out here too. Kept as long as your events: 6 months on Free, 24 on Pro.
• A request counts as the crawler its user agent names. SimpleData does not check the crawler's IP address, and anyone can send a request that says it is GPTBot.

What your server does not see

• Pages served by a cache in front of your code: a full-page cache plugin on WordPress (LiteSpeed Cache, WP Super Cache, W3 Total Cache), a Cloudflare or CDN cache, Varnish, an nginx cache. The request never reaches PHP or your app. A Cloudflare Worker runs before Cloudflare's cache, so it sees those too.
• Crawlers that your firewall, your host or Cloudflare (Block AI bots) turn away before your server answers.
• Use one way per site: with the WordPress plugin and a Worker on the same site, the requests that reach WordPress would count twice. Untick the plugin's option if you add the Worker.

WordPress

• Plugin 1.1.5 or later, on by default. In WordPress, SimpleData → Settings → AI crawlers turns it on and off and says when it last sent anything.
• When a request's user agent looks like a crawler, the plugin adds the page to a small queue in your database, and WP-Cron sends the queue a minute later, in batches signed with your Heartbeat Secret. The page being served never waits for SimpleData.
• WP-Cron runs when your site gets visits: on a quiet site a batch can wait for the next one.
• Not for logged-in users, previews, 404 pages or anything but GET and HEAD.

Next.js, Express and Cloudflare Workers

In your dashboard, Sites → ⋯ → Installation code shows the site's Site ID and Heartbeat Secret, the same two the WordPress plugin uses. Put them in your server's environment as SIMPLEDATA_SITE_ID and SIMPLEDATA_SECRET and paste one of these. Each one sends after the response, with a timeout, and ignores any error: your pages never wait for SimpleData, and never fail because of it. Images, fonts, styles and scripts are dropped on arrival, whoever sends them: pages count, and so do robots.txt, sitemap.xml, llms.txt, feeds and documents such as PDFs.

Next.js

proxy.ts next to your app/ folder (Next.js 16), or middleware.ts on Next.js 13 to 15, with the function renamed to middleware. It runs before Next's cache, so pages served from it count too. If you already have a proxy or a middleware, call reportAiCrawler(request, event) at its start. Its matcher leaves out Next's files, your API routes and static files, so the proxy does not run for every image.

// proxy.ts, next to app/ (Next.js 16). On Next.js 13 to 15: middleware.ts,
// and rename the exported function to middleware.
import { NextResponse, type NextFetchEvent, type NextRequest } from 'next/server';

const ENDPOINT = 'https://api.simpledata.dev/api/v1/ai-crawlers/hits';
// A first, broad filter: SimpleData decides which of these are AI agents.
const MAYBE_AI = /bot|crawl|spider|fetch|agent|-user\b|claude|anthropic|openai|perplexity|cohere|grok|omgili|webzio|amazonq|preview/i;
const NOT_AI = /googlebot|google-inspectiontool|googleother|adsbot|mediapartners|ahrefs|semrush|yandex|mj12bot|dotbot|petalbot|baiduspider|seznambot|uptimerobot|bingpreview/i;

async function report(path: string, userAgent: string) {
  const siteId = process.env.SIMPLEDATA_SITE_ID;
  const secret = process.env.SIMPLEDATA_SECRET;
  if (!siteId || !secret) return;
  const body = JSON.stringify({ hits: [{ path, userAgent, time: new Date().toISOString() }] });
  const timestamp = String(Math.floor(Date.now() / 1000));
  const encoder = new TextEncoder();
  const key = await crypto.subtle.importKey('raw', encoder.encode(secret), { name: 'HMAC', hash: 'SHA-256' }, false, ['sign']);
  const mac = await crypto.subtle.sign('HMAC', key, encoder.encode(`v2:ai-hits:${siteId}:${timestamp}:${body}`));
  const signature = Array.from(new Uint8Array(mac), (b) => b.toString(16).padStart(2, '0')).join('');
  await fetch(ENDPOINT, {
    method: 'POST',
    body,
    headers: {
      'content-type': 'application/json',
      'x-simpledata-site': siteId,
      'x-simpledata-timestamp': timestamp,
      'x-simpledata-signature-version': '2',
      'x-simpledata-signature': signature,
    },
    signal: AbortSignal.timeout(3000),
  });
}

// If you already have a proxy (or middleware), call this at its start.
export function reportAiCrawler(request: NextRequest, event: NextFetchEvent) {
  const userAgent = request.headers.get('user-agent') ?? '';
  if ((request.method === 'GET' || request.method === 'HEAD') && MAYBE_AI.test(userAgent) && !NOT_AI.test(userAgent)) {
    // After the response: the page never waits for SimpleData.
    event.waitUntil(report(request.nextUrl.pathname, userAgent).catch(() => {}));
  }
}

export function proxy(request: NextRequest, event: NextFetchEvent) {
  reportAiCrawler(request, event);
  return NextResponse.next();
}

// Pages only: not Next's own files, your API routes or static files (SimpleData
// drops those anyway). robots.txt, sitemap.xml, llms.txt and feeds are reported.
export const config = { matcher: ['/((?!_next/|api/|favicon.ico|.*\\.(?:png|jpe?g|gif|webp|avif|svg|ico|css|js|map|woff2?|ttf|otf)$).*)'] };

Express

Save it as simpledata-ai-crawlers.mjs and add app.use(aiCrawlers) before your routes and static files. It queues what was served (not redirects or errors) and sends it every 10 seconds, 100 requests at a time. Node 18 or later. Behind a cache or a CDN, only what reaches Express is seen.

// simpledata-ai-crawlers.mjs (Node 18 or later)
import crypto from 'node:crypto';

const ENDPOINT = 'https://api.simpledata.dev/api/v1/ai-crawlers/hits';
// A first, broad filter: SimpleData decides which of these are AI agents.
const MAYBE_AI = /bot|crawl|spider|fetch|agent|-user\b|claude|anthropic|openai|perplexity|cohere|grok|omgili|webzio|amazonq|preview/i;
const NOT_AI = /googlebot|google-inspectiontool|googleother|adsbot|mediapartners|ahrefs|semrush|yandex|mj12bot|dotbot|petalbot|baiduspider|seznambot|uptimerobot|bingpreview/i;
const queue = [];

function send() {
  const siteId = process.env.SIMPLEDATA_SITE_ID;
  const secret = process.env.SIMPLEDATA_SECRET;
  if (!siteId || !secret || queue.length === 0) return Promise.resolve();
  const body = JSON.stringify({ hits: queue.splice(0, 100) });
  const timestamp = String(Math.floor(Date.now() / 1000));
  const signature = crypto.createHmac('sha256', secret).update(`v2:ai-hits:${siteId}:${timestamp}:${body}`).digest('hex');
  return fetch(ENDPOINT, {
    method: 'POST',
    body,
    headers: {
      'content-type': 'application/json',
      'x-simpledata-site': siteId,
      'x-simpledata-timestamp': timestamp,
      'x-simpledata-signature-version': '2',
      'x-simpledata-signature': signature,
    },
    signal: AbortSignal.timeout(5000),
  }).then(() => {}, () => {});
}

// Every 10 seconds, in the background; unref() so it never keeps the process alive.
setInterval(send, 10_000).unref();

export function aiCrawlers(req, res, next) {
  const userAgent = req.get('user-agent') || '';
  if ((req.method === 'GET' || req.method === 'HEAD') && MAYBE_AI.test(userAgent) && !NOT_AI.test(userAgent)) {
    const path = req.originalUrl.split('?')[0];
    res.on('finish', () => {
      // What was served, not redirects or errors; at most 1,000 waiting.
      if (res.statusCode < 300 && queue.length < 1000) queue.push({ path, userAgent, time: new Date().toISOString() });
      if (queue.length >= 100) send();
    });
  }
  next();
}

// Sends what is waiting, for example before the process exits.
export const flushAiCrawlers = send;

import { aiCrawlers } from './simpledata-ai-crawlers.mjs';

app.use(aiCrawlers);

Cloudflare Worker

A Worker on your site's route (example.com/*) sees every request, cached pages included, and reports the ones it served with ctx.waitUntil, after the response. Add the two values as secrets of the Worker.

// A Worker on your site's route (example.com/*), in front of it. Secrets:
//   npx wrangler secret put SIMPLEDATA_SITE_ID
//   npx wrangler secret put SIMPLEDATA_SECRET
const ENDPOINT = 'https://api.simpledata.dev/api/v1/ai-crawlers/hits';
// A first, broad filter: SimpleData decides which of these are AI agents.
const MAYBE_AI = /bot|crawl|spider|fetch|agent|-user\b|claude|anthropic|openai|perplexity|cohere|grok|omgili|webzio|amazonq|preview/i;
const NOT_AI = /googlebot|google-inspectiontool|googleother|adsbot|mediapartners|ahrefs|semrush|yandex|mj12bot|dotbot|petalbot|baiduspider|seznambot|uptimerobot|bingpreview/i;

async function report(env, path, userAgent) {
  const body = JSON.stringify({ hits: [{ path, userAgent, time: new Date().toISOString() }] });
  const timestamp = String(Math.floor(Date.now() / 1000));
  const encoder = new TextEncoder();
  const key = await crypto.subtle.importKey('raw', encoder.encode(env.SIMPLEDATA_SECRET), { name: 'HMAC', hash: 'SHA-256' }, false, ['sign']);
  const mac = await crypto.subtle.sign('HMAC', key, encoder.encode(`v2:ai-hits:${env.SIMPLEDATA_SITE_ID}:${timestamp}:${body}`));
  const signature = [...new Uint8Array(mac)].map((b) => b.toString(16).padStart(2, '0')).join('');
  await fetch(ENDPOINT, {
    method: 'POST',
    body,
    headers: {
      'content-type': 'application/json',
      'x-simpledata-site': env.SIMPLEDATA_SITE_ID,
      'x-simpledata-timestamp': timestamp,
      'x-simpledata-signature-version': '2',
      'x-simpledata-signature': signature,
    },
    signal: AbortSignal.timeout(5000),
  });
}

export default {
  async fetch(request, env, ctx) {
    const response = await fetch(request);
    const userAgent = request.headers.get('user-agent') || '';
    if (env.SIMPLEDATA_SITE_ID && env.SIMPLEDATA_SECRET && (request.method === 'GET' || request.method === 'HEAD')
        && response.status < 300 && MAYBE_AI.test(userAgent) && !NOT_AI.test(userAgent)) {
      // After the response: the page never waits for SimpleData.
      ctx.waitUntil(report(env, new URL(request.url).pathname, userAgent).catch(() => {}));
    }
    return response;
  },
};

Check that it works

Ask for a page the way a crawler would. It shows in the card as one request from ChatGPT (model training) within a minute; with WordPress, a few minutes later, once WP-Cron has run.

curl -A "Mozilla/5.0 (compatible; GPTBot/1.2; +https://openai.com/gptbot)" https://example.com/