FRD-007
Detect bots without breaking the page for them
Withhold earnings from obvious automation, never content. Blocking search crawlers is a self-inflicted wound that takes months to notice.
User agent checks are weak — anything can claim to be Chrome. They are still worth having, because the cheap automation that actually shows up on faucets usually does not bother lying.
The important part is what you do with the answer. Withhold earnings, not content. A crawler that gets a 403 instead of your page is a page that never ranks, and you will not connect the two for a long time.
Treat this as one signal among several. On its own it stops the lazy; combined with rate limits and proof-of-render it stops most of the rest.
function looks_automated(): bool
{
$ua = strtolower(trim((string) ($_SERVER['HTTP_USER_AGENT'] ?? '')));
if ($ua === '') {
return true; // no user agent at all
}
$signatures = [
'bot', 'crawl', 'spider', 'slurp', 'curl', 'wget', 'python-requests',
'python-urllib', 'go-http-client', 'java/', 'okhttp', 'httpclient',
'headless', 'phantomjs', 'puppeteer', 'playwright', 'scrapy',
'monitor', 'uptime', 'preview', 'fetcher',
];
foreach ($signatures as $sig) {
if (str_contains($ua, $sig)) {
return true;
}
}
// A real browser sends these. Automation frequently does not.
if (($_SERVER['HTTP_ACCEPT_LANGUAGE'] ?? '') === '') {
return true;
}
if (($_SERVER['HTTP_ACCEPT'] ?? '') === '') {
return true;
}
return false;
}
// Serve everything. Pay nothing.
function may_earn(): bool
{
return !looks_automated();
}Using it
Wire it into the earning path and the statistics path, and nowhere else. Content, assets and error pages should not consult it at all.
Log what you rejected. A legitimate tool being classed as automation shows up as a user complaining that their earnings stopped.
What bites people
Googlebot contains "bot" and must still be served the full page. This function returning true is not permission to change what is rendered.
Do not use it as your only defence. A ten character user agent change defeats it entirely.