mirror of
https://github.com/direct-img/direct-img.link.git
synced 2026-10-29 07:24:52 +00:00
Feat: Split searches between Brave and Firecrawl, and drop the Bing scraper
A third of queries (by query hash) search Firecrawl first and the rest Brave, so Firecrawl's free 500 searches a month and Brave's ~1,000 run out together. Each falls back to the other on an error or no results. Searches that find nothing serve bad.webp, both providers failing serves donate.webp, and AI images only fill ?i= past a search's working images. A Firecrawl query's ?i= past 10 serves an AI image without searching. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Claude Opus 5.5
@@ -45,6 +45,7 @@ https://direct-img.link/orange+cat?i=2
|
||||
- Only images that download count, so broken links and deleted photos are skipped and every `i` is a different image. `?i=2` is the 2nd working image
|
||||
- Higher `i` values take a little longer the first time, since the working images before it are downloaded to count them
|
||||
- If there aren't `i` working images, an [AI image](#ai-images) is served instead (except on [free.](#free-images), which serves `bad.webp`)
|
||||
- Each query is searched on either Brave (100 results) or Firecrawl (10 results), always the same one. On a Firecrawl query, `i` above `10` serves an AI image without searching
|
||||
- Each `i` is cached separately. A search's results are kept for **1 hour**, so other `i` values of the same query within that hour don't search again and **don't count toward the limit**. One that was already counted downloads only its own image
|
||||
|
||||
## Query Normalization
|
||||
@@ -85,8 +86,8 @@ Literal slashes (`/`) and dots (`.`) in the URL path are **rejected** to prevent
|
||||
- **Fragments (`#...`)** are never sent to the server by browsers
|
||||
- **Double-encoded values** are decoded once — `%2520` becomes `%20` (literal), not a space
|
||||
- Two queries that normalize to the same string share the same cached image
|
||||
- **Nothing found?** If a search finds no image (or fewer than `i` working ones), an [AI image](#ai-images) of the query is served instead of `bad.webp`, and cached like any other image. `bad.webp` is only served if the AI image fails too, or on [free.](#free-images)
|
||||
- **Out of searches?** If Brave refuses (e.g. its monthly spending limit is reached), Bing is searched instead. If Bing is blocking us too, `donate.webp` is served instead of an AI image, and nothing is cached, so the query works again once either is back
|
||||
- **Nothing found?** If neither Brave nor Firecrawl finds anything, `bad.webp` is served (remembered for 24h). If a search finds fewer than `i` working images, an [AI image](#ai-images) of the query is served for the missing ones, and cached like any other image. `bad.webp` is served if the AI image fails too, or on [free.](#free-images)
|
||||
- **Out of searches?** Each query is searched on Brave or Firecrawl, and on the other if that one refuses (e.g. it's out of credits) or finds nothing. If both refuse, `donate.webp` is served and nothing is cached, so the query works again once either is back
|
||||
|
||||
## Free Images
|
||||
|
||||
@@ -133,7 +134,7 @@ Licenses come from each source's metadata and can be wrong. Check the original b
|
||||
- The same query, `src` and `i` always give the same image
|
||||
- `i` above `1000` serves `max-i.webp`, which lists each site's range. Other invalid values serve `bad.webp`
|
||||
- Images come from [AuroraIMG-6M](https://github.com/direct-img/AuroraIMG-6M), a tiny 6.8M-parameter model running on our own server. They're 512px, soft and painterly: the right colors, setting and mood, but often not the exact object, and faces come out distorted
|
||||
- The same model makes the fallback images for searches that find nothing
|
||||
- The same model makes the fallback images for an `?i=` past a search's working images
|
||||
|
||||
## For AI System Prompts
|
||||
|
||||
@@ -153,8 +154,8 @@ Limits change over time, so they aren't listed here. **Check the Rate Limits sec
|
||||
- **New searches (cache misses)** are limited per IP per day, then `limit.webp` is served. `direct-img.link`, `free.direct-img.link` and `ai.direct-img.link` have separate counters, all reset at midnight UTC
|
||||
- **Cache hits are unlimited** (within the burst limit)
|
||||
- Other `?i=` values of a query searched in the last hour reuse its results, so they don't count (except on `ai.`, where each `i` is a new image)
|
||||
- If Brave's spending limit is hit, searches use Bing; if Bing also blocks us, `donate.webp` is served (not cached)
|
||||
- Searches that find nothing count toward the limit. If the AI image served instead fails too (or there's none, on `free.`), the failure is remembered for 24h (`bad.webp`)
|
||||
- If Brave or Firecrawl is out of credits, searches use the other; if both are, `donate.webp` is served (not cached)
|
||||
- Searches that find nothing count toward the limit, and are remembered for 24h (`bad.webp`). So is a fallback AI image that fails
|
||||
- A Cloudflare WAF burst limit applies to all requests (HTTP 429). Its counters are approximate (not shared perfectly across Cloudflare servers), so bursts may slip slightly past it
|
||||
|
||||
## Caching
|
||||
@@ -185,9 +186,17 @@ Free community service. Donations help cover API and infrastructure costs.
|
||||
4. Subscribe — you get **$5 in free monthly credits** (covers 1,000 queries/month)
|
||||
5. Go to your [API dashboard](https://api.search.brave.com/app/#/subscriptions)
|
||||
6. Copy your **API key** (starts with `BSA...`)
|
||||
7. Optional: set a monthly spending limit there so you never pay past the credit. Once it's reached, searches fall back to Bing
|
||||
7. Optional: set a monthly spending limit there so you never pay past the credit. Once it's reached, its searches fall back to Firecrawl
|
||||
|
||||
### 2. Openverse API Credentials (Optional)
|
||||
### 2. Firecrawl API Key
|
||||
|
||||
1. Sign up at [firecrawl.dev](https://www.firecrawl.dev/) (no card needed)
|
||||
2. The free plan gives **1,000 credits a month**, which is 500 searches at 2 credits per 10 results. Once they run out, Firecrawl refuses (HTTP 402) and its searches fall back to Brave
|
||||
3. Copy your **API key** (starts with `fc-`)
|
||||
|
||||
A third of queries go to Firecrawl first and the rest to Brave (`FIRECRAWL_SHARE`), so both run out at about the same time.
|
||||
|
||||
### 3. Openverse API Credentials (Optional)
|
||||
|
||||
Free images work without credentials, but anonymous Openverse requests are limited to 200/day. Register once to get higher limits:
|
||||
|
||||
@@ -201,7 +210,7 @@ curl -X POST -H "Content-Type: application/json" -d '{"name":"<unique app name>"
|
||||
|
||||
The function exchanges them for an access token (valid ~12h) and caches it in KV, fetching a new one automatically when it expires.
|
||||
|
||||
### 3. Cloudflare Resources
|
||||
### 4. Cloudflare Resources
|
||||
|
||||
Create in your Cloudflare dashboard:
|
||||
|
||||
@@ -210,7 +219,7 @@ Create in your Cloudflare dashboard:
|
||||
| R2 Bucket | `direct-img-store` | Stores cached images |
|
||||
| KV Namespace | `DIRECT_IMG_CACHE` | Openverse and SurrealDB access tokens |
|
||||
|
||||
### 4. Pages Bindings
|
||||
### 5. Pages Bindings
|
||||
|
||||
**Settings → Functions → Bindings:**
|
||||
|
||||
@@ -219,13 +228,14 @@ Create in your Cloudflare dashboard:
|
||||
| R2 Bucket | `R2_IMAGES` | `direct-img-store` |
|
||||
| KV Namespace | `DIRECT_IMG_CACHE` | `DIRECT_IMG_CACHE` |
|
||||
|
||||
### 5. Secrets
|
||||
### 6. Secrets
|
||||
|
||||
**Settings → Environment variables:**
|
||||
|
||||
| Variable | Description | Required |
|
||||
|---|---|---|
|
||||
| `BRAVE_API_KEY` | Brave Search API key | Yes |
|
||||
| `FIRECRAWL_API_KEY` | Firecrawl API key | Yes |
|
||||
| `OPENVERSE_CLIENT_ID` | Openverse app client ID (free images) | Optional |
|
||||
| `OPENVERSE_CLIENT_SECRET` | Openverse app client secret (free images) | Optional |
|
||||
| `SURREAL_URL` | SurrealDB URL (e.g. `https://db.site.com`) | Yes |
|
||||
@@ -234,19 +244,19 @@ Create in your Cloudflare dashboard:
|
||||
| `NTFY_URL` | ntfy.sh topic URL for alerts | Optional |
|
||||
| `GOATCOUNTER_URL` | GoatCounter site URL for image hit analytics (e.g. `https://direct-img.goatcounter.com`) | Optional |
|
||||
| `GOATCOUNTER_TOKEN` | GoatCounter API token with "Record pageviews" permission | Optional |
|
||||
| `AURORA_API_KEY` | Key for the [AuroraIMG-6M API](https://github.com/direct-img/AuroraIMG-6M) (AI images and the nothing-found fallback). Without it, `ai.` serves `bad.webp` and searches that find nothing serve `bad.webp` (as they always do on `free.`) | Optional |
|
||||
| `AURORA_API_KEY` | Key for the [AuroraIMG-6M API](https://github.com/direct-img/AuroraIMG-6M) (AI images, and the fallback for an `?i=` past a search's working images). Without it, `ai.` and those `?i=` serve `bad.webp` | Optional |
|
||||
|
||||
### 6. WAF Rules
|
||||
### 7. WAF Rules
|
||||
|
||||
**Security → WAF → Rate limiting rules:**
|
||||
|
||||
1. **Rate Limit** — 10 req/10s per IP → Block 10s
|
||||
|
||||
### 7. Deploy
|
||||
### 8. Deploy
|
||||
|
||||
Fork this repo, connect to Cloudflare Pages, deploy.
|
||||
|
||||
### 8. Free and AI Images Domains
|
||||
### 9. Free and AI Images Domains
|
||||
|
||||
**Pages project → Custom domains → Set up a domain:** add `free.<your-domain>` and `ai.<your-domain>`. Any hostname starting with `free.` serves free images, and any starting with `ai.` serves AI images; everything else serves regular search.
|
||||
|
||||
@@ -267,7 +277,7 @@ Every request looks up its key here first, and a cached image is served from tha
|
||||
| `FREE:<src or auto>:<i>:<query>` (e.g. `FREE:wikimedia:2:orange cat`) | free images |
|
||||
| `AI:<src>:<i>:<query>` (e.g. `AI:auroraimg-6m:2:frog wizard`) | AI images |
|
||||
|
||||
An AI image served because a search found nothing is stored under the search's own key, like any other image. A search where that also fails, or a `free.` search (which gets no AI image), stores an empty object with `err` custom metadata instead, which serves `bad.webp` for 24h. After that the query is searched again.
|
||||
An AI image served for an `?i=` past a search's working images is stored under the search's own key, like any other image. A search that finds nothing, one whose AI image fails too, or a `free.` search with too few images (which gets no AI image) stores an empty object with `err` custom metadata instead, which serves `bad.webp` for 24h. After that the query is searched again.
|
||||
|
||||
Add an object lifecycle rule in **R2 → direct-img-store → Settings** that applies to all prefixes and deletes uploaded objects after **90 days**. It's required: nothing else expires cached images.
|
||||
|
||||
@@ -301,9 +311,9 @@ After deploying, visit `https://<your-domain>/_setup` once. It creates the `dire
|
||||
- **Cloudflare KV** — API access tokens
|
||||
- **SurrealDB (v2.6.5)** — atomic rate limiting, search locks + cached result lists
|
||||
- **Cloudflare WAF** — layer 7 mitigation
|
||||
- **Brave Image Search API** — image sourcing, with Bing as the fallback
|
||||
- **Brave Image Search API + Firecrawl Search API** — image sourcing, each the other's fallback
|
||||
- **Openverse API + Wikimedia Commons API** — free image sourcing
|
||||
- **[AuroraIMG-6M](https://github.com/direct-img/AuroraIMG-6M) on Dokploy** — AI images and the nothing-found fallback
|
||||
- **[AuroraIMG-6M](https://github.com/direct-img/AuroraIMG-6M) on Dokploy** — AI images and the fallback for an `?i=` past a search's working images
|
||||
|
||||
---
|
||||
|
||||
|
||||
Reference in New Issue
Block a user