Feat: Let search engines crawl free. and raise the burst limit to ~20 req/10s

free.* now serves a robots.txt that allows everything, so free images embedded in
pages (e.g. apophenia.news articles) can be indexed and used for previews. The
main site and ai.* keep the homepage-only robots.txt. The WAF rate limit rule is
now 20 req/10s and skips known bots.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-10-23 22:11:39 +00:00
co-authored by Claude Opus 5.5
parent 3181eda2a2
commit cd604838bd
5 changed files with 11 additions and 5 deletions
+1 -1
View File
@@ -123,7 +123,7 @@ Use images to complement your responses, powered by Brave.</code></pre>
<tr><th>Scope</th><th>Limit</th><th>Action</th></tr>
</thead>
<tbody>
<tr><td>Burst</td><td>~15 req/10s per IP</td><td>HTTP 429 (~10s)</td></tr>
<tr><td>Burst</td><td>~20 req/10s per IP</td><td>HTTP 429 (~10s)</td></tr>
<tr><td>New Searches</td><td><span data-limit="DAILY_LIMIT">35</span> per day per IP</td><td>Limit image served</td></tr>
<tr><td>New Free Searches</td><td><span data-limit="FREE_DAILY_LIMIT">100</span> per day per IP</td><td>Limit image served</td></tr>
<tr><td>New AI Images</td><td><span data-limit="AI_DAILY_LIMIT">100</span> per day per IP</td><td>Limit image served</td></tr>