Feat: Let search engines crawl free. and raise the burst limit to ~20 req/10s

free.* now serves a robots.txt that allows everything, so free images embedded in
pages (e.g. apophenia.news articles) can be indexed and used for previews. The
main site and ai.* keep the homepage-only robots.txt. The WAF rate limit rule is
now 20 req/10s and skips known bots.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-10-23 22:11:39 +00:00
co-authored by Claude Opus 5.5
parent 3181eda2a2
commit cd604838bd
5 changed files with 11 additions and 5 deletions
+1 -1
View File
@@ -1,6 +1,6 @@
// New searches (cache misses) allowed per IP per day, reset at 00:00 UTC. Cache hits never count.
// Safe to edit by hand. Imported by the worker and by the homepage (as a static file, so no Functions quota).
// The burst limit (~15 req/10s) is a Cloudflare WAF rule, not set here.
// The burst limit (~20 req/10s) is a Cloudflare WAF rule, not set here.
export const DAILY_LIMIT = 10;
export const FREE_DAILY_LIMIT = 100;
export const AI_DAILY_LIMIT = 100;