Feat: Let search engines crawl free. and raise the burst limit to ~20 req/10s

free.* now serves a robots.txt that allows everything, so free images embedded in
pages (e.g. apophenia.news articles) can be indexed and used for previews. The
main site and ai.* keep the homepage-only robots.txt. The WAF rate limit rule is
now 20 req/10s and skips known bots.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

README.md

@@ -108,6 +108,7 @@ Need images you can publish? `free.direct-img.link` only serves images marked **
 - `i` above `20` serves `max-i.webp`, which lists each site's range. Other invalid values serve `bad.webp`
 - Only free images are served: if there's no image (or fewer than `i` working ones), `bad.webp` is served (remembered for 24h) instead of an AI image. With a `src`, only that source's images are searched
 - Each `src` + `i` + query combination is cached separately. Like the main site, other `i` values within an hour of a search reuse its results and don't count as searches
+- Search engines may crawl `free.` (its robots.txt allows everything), so free images embedded in your pages can show up in image search and link previews. The main site and `ai.` allow only the homepage
 
 **Sources:**
 
@@ -251,7 +252,7 @@ Create in your Cloudflare dashboard:
 
 **Security → WAF → Rate limiting rules:**
 
-1. **Rate Limit** — 15 req/10s per IP → Block 10s
+1. **Rate Limit** — 20 req/10s per IP → Block 10s, skipping known bots (`and not cf.client.bot`) so search engines can crawl
 
 ### 8. Deploy
 

_routes.json

@@ -1,5 +1,5 @@
 {
   "version": 1,
   "include": ["/*"],
-  "exclude": ["/", "/index.html", "/robots.txt", "/assets/*"]
+  "exclude": ["/", "/index.html", "/assets/*"]
 }

assets/rate-limit.js

@@ -1,6 +1,6 @@
 // New searches (cache misses) allowed per IP per day, reset at 00:00 UTC. Cache hits never count.
 // Safe to edit by hand. Imported by the worker and by the homepage (as a static file, so no Functions quota).
-// The burst limit (~15 req/10s) is a Cloudflare WAF rule, not set here.
+// The burst limit (~20 req/10s) is a Cloudflare WAF rule, not set here.
 export const DAILY_LIMIT = 10;
 export const FREE_DAILY_LIMIT = 100;
 export const AI_DAILY_LIMIT = 100;

functions/[[path]].js

@@ -33,7 +33,12 @@ export async function onRequest(context) {
   // Every alert ends with the full request URL (site, query, ?i= and ?src=), so it's clear which request triggered it
   const note = n => context.waitUntil(notify(env, { ...n, message: `${n.message}\n${url.origin}/${path}${url.search}` }));
 
-  // _routes.json serves these statically without invoking this function; kept as a fallback
+  // free.* images are public domain or CC0 and its searches cost no credits, so search engines may crawl them (e.g. to
+  // index images embedded in articles). The main site and ai.* get robots.txt, which allows only the homepage.
+  if (path === "robots.txt" && url.hostname.startsWith("free.")) {
+    return new Response("User-agent: *\nAllow: /\n", { headers: { "Content-Type": "text/plain; charset=utf-8" } });
+  }
+  // _routes.json serves these (except robots.txt) statically without invoking this function; kept as a fallback
   if (!path || path === "index.html" || path === "robots.txt" || path.startsWith("assets/")) {
     return env.ASSETS.fetch(request);
   }

index.html

@@ -123,7 +123,7 @@ Use images to complement your responses, powered by Brave.</code></pre>
                     <tr><th>Scope</th><th>Limit</th><th>Action</th></tr>
                 </thead>
                 <tbody>
-                    <tr><td>Burst</td><td>~15 req/10s per IP</td><td>HTTP 429 (~10s)</td></tr>
+                    <tr><td>Burst</td><td>~20 req/10s per IP</td><td>HTTP 429 (~10s)</td></tr>
                     <tr><td>New Searches</td><td><span data-limit="DAILY_LIMIT">35</span> per day per IP</td><td>Limit image served</td></tr>
                     <tr><td>New Free Searches</td><td><span data-limit="FREE_DAILY_LIMIT">100</span> per day per IP</td><td>Limit image served</td></tr>
                     <tr><td>New AI Images</td><td><span data-limit="AI_DAILY_LIMIT">100</span> per day per IP</td><td>Limit image served</td></tr>