block non-human crawlers with caddy

The “block non-human crawlers” script was created by Adële, and it supports lighttpd. Link to the blog: Block non-human crawlers with lighttpd. I needed to modify a script by Adële to protect my Git server, which sits behind Caddy but I also made changes without support for reverse-proxy which is included in the first part of the post on my blog.

#Caddyfile

domain.tld {
    root * /var/www/domain.tld

    # Match everything EXCEPT the exclusion list
    @not_excluded {
        not path /robots.txt /cookie-check.php
    }

    # Match requests lacking the validated verification cookie
    @missing_cookie {
        not header Cookie *x-cookie-access=ok*
    }

    # Challenge layer
    handle @not_excluded {
        handle @missing_cookie {
            redir /cookie-check.php?redirect={uri} 302
        }
    }

    # Execute the PHP human-verification script
    route /cookie-check.php {
        php_fastcgi unix//run/php/php-fpm.sock
    }

    # Serve static assets or index files if the challenge passes
    file_server
}

We use the same PHP script cookie-check.php from the blog post, place it in the root directory of the website: /var/www/domain.tld/cookie-check.php. domain.tld is just an example domain name and root directory name.

#cookie-check.php

<?php

header("X-Robots-Tag: noindex, nofollow", true);
if($_POST['k']==md5($_SERVER['HTTP_USER_AGENT']))
{
        // set a random cookie
        setcookie("x-cookie-access", "ok", [
          'path' => '/',
          'samesite' => 'Lax'
        ]);
        // find the original requested page url
        $redirect = $_GET['redirect'] ?? '/';

        // no redirection to external url (security)
        if (str_starts_with($redirect, '/')) {
            header("Location: " . $redirect);
            exit;
        }

        // Fallback: default redirection
        header("Location: /");
        exit;
}
?><!DOCTYPE html>
<html lang="en">
    <head>
        <meta http-equiv="Content-Type" content="text/html; charset=utf-8" >
        <meta name="viewport" content="width=device-width, initial-scale=1.0" >
        <meta name="color-scheme" content="light dark" >
        <title>Human check</title>
    </head>
    <body>
        <h1>
        <strong>Verifying you are a human</strong>
        </h1>
        <main>
            <form method="POST">
                <p>
                <input type="checkbox"  name="k" value="<?=md5($_SERVER['HTTP_USER_AGENT'])?>" ><span> I am a human</span>
                <br>
                <br>
                <input type="submit">
                <br>
                <br>
                <em>this will set a dummy cookie (AI crawlers do not manage cookies)</em>
                </p>
            </form>
        </main>
    </body>
</html>

block non-human crawlers with caddy

If you need to redirect root domain to reverse-proxy, use the caddyfile configuration below.

#Caddyfile

domain.tld {
    root * /var/www/domain.tld

    # Match everything EXCEPT the exclusion list
    @not_excluded {
        not path /robots.txt /cookie-check.php
    }

    # Match requests lacking the validated verification cookie
    @missing_cookie {
        not header Cookie *x-cookie-access=ok*
    }

    # Challenge layer
    handle @not_excluded {
        handle @missing_cookie {
            redir /cookie-check.php?redirect={uri} 302
        }
    }

    # Execute the PHP human-verification script
    route /cookie-check.php {
        php_fastcgi unix//run/php/php-fpm.sock
    }

    # Clean reverse proxy (Caddy v2 handles WebSockets automatically)
    reverse_proxy 127.0.0.1:3000
	}