mirror of
https://github.com/direct-img/direct-img.link.git
synced 2026-10-28 23:14:51 +00:00
Feat: Split searches between Brave and Firecrawl, and drop the Bing scraper
A third of queries (by query hash) search Firecrawl first and the rest Brave, so Firecrawl's free 500 searches a month and Brave's ~1,000 run out together. Each falls back to the other on an error or no results. Searches that find nothing serve bad.webp, both providers failing serves donate.webp, and AI images only fill ?i= past a search's working images. A Firecrawl query's ?i= past 10 serves an AI image without searching. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Claude Opus 5.5
+1
-1
@@ -12,7 +12,7 @@ direct-img.link has no accounts and doesn't sell data. This page explains what i
|
||||
|
||||
## Search queries and images
|
||||
|
||||
- When a query isn't already cached, it's sent to Brave Search, and to Bing if Brave returns nothing or is unavailable. On free.direct-img.link, it's sent to Openverse and/or Wikimedia Commons instead. Searches and image downloads are made from our servers, so your IP address isn't shared with these search providers or the sites images come from.
|
||||
- When a query isn't already cached, it's sent to Brave Search or Firecrawl (each query always goes to the same one first), and to the other if that returns nothing or is unavailable. On free.direct-img.link, it's sent to Openverse and/or Wikimedia Commons instead. Searches and image downloads are made from our servers, so your IP address isn't shared with these search providers or the sites images come from.
|
||||
- Each query (with its `i` and `src` options) and the image it returned are cached for up to 90 days. Failed searches are remembered for 24 hours.
|
||||
- Cached images are shared: anyone requesting the same query and options gets the same image.
|
||||
|
||||
|
||||
@@ -45,6 +45,7 @@ https://direct-img.link/orange+cat?i=2
|
||||
- Only images that download count, so broken links and deleted photos are skipped and every `i` is a different image. `?i=2` is the 2nd working image
|
||||
- Higher `i` values take a little longer the first time, since the working images before it are downloaded to count them
|
||||
- If there aren't `i` working images, an [AI image](#ai-images) is served instead (except on [free.](#free-images), which serves `bad.webp`)
|
||||
- Each query is searched on either Brave (100 results) or Firecrawl (10 results), always the same one. On a Firecrawl query, `i` above `10` serves an AI image without searching
|
||||
- Each `i` is cached separately. A search's results are kept for **1 hour**, so other `i` values of the same query within that hour don't search again and **don't count toward the limit**. One that was already counted downloads only its own image
|
||||
|
||||
## Query Normalization
|
||||
@@ -85,8 +86,8 @@ Literal slashes (`/`) and dots (`.`) in the URL path are **rejected** to prevent
|
||||
- **Fragments (`#...`)** are never sent to the server by browsers
|
||||
- **Double-encoded values** are decoded once — `%2520` becomes `%20` (literal), not a space
|
||||
- Two queries that normalize to the same string share the same cached image
|
||||
- **Nothing found?** If a search finds no image (or fewer than `i` working ones), an [AI image](#ai-images) of the query is served instead of `bad.webp`, and cached like any other image. `bad.webp` is only served if the AI image fails too, or on [free.](#free-images)
|
||||
- **Out of searches?** If Brave refuses (e.g. its monthly spending limit is reached), Bing is searched instead. If Bing is blocking us too, `donate.webp` is served instead of an AI image, and nothing is cached, so the query works again once either is back
|
||||
- **Nothing found?** If neither Brave nor Firecrawl finds anything, `bad.webp` is served (remembered for 24h). If a search finds fewer than `i` working images, an [AI image](#ai-images) of the query is served for the missing ones, and cached like any other image. `bad.webp` is served if the AI image fails too, or on [free.](#free-images)
|
||||
- **Out of searches?** Each query is searched on Brave or Firecrawl, and on the other if that one refuses (e.g. it's out of credits) or finds nothing. If both refuse, `donate.webp` is served and nothing is cached, so the query works again once either is back
|
||||
|
||||
## Free Images
|
||||
|
||||
@@ -133,7 +134,7 @@ Licenses come from each source's metadata and can be wrong. Check the original b
|
||||
- The same query, `src` and `i` always give the same image
|
||||
- `i` above `1000` serves `max-i.webp`, which lists each site's range. Other invalid values serve `bad.webp`
|
||||
- Images come from [AuroraIMG-6M](https://github.com/direct-img/AuroraIMG-6M), a tiny 6.8M-parameter model running on our own server. They're 512px, soft and painterly: the right colors, setting and mood, but often not the exact object, and faces come out distorted
|
||||
- The same model makes the fallback images for searches that find nothing
|
||||
- The same model makes the fallback images for an `?i=` past a search's working images
|
||||
|
||||
## For AI System Prompts
|
||||
|
||||
@@ -153,8 +154,8 @@ Limits change over time, so they aren't listed here. **Check the Rate Limits sec
|
||||
- **New searches (cache misses)** are limited per IP per day, then `limit.webp` is served. `direct-img.link`, `free.direct-img.link` and `ai.direct-img.link` have separate counters, all reset at midnight UTC
|
||||
- **Cache hits are unlimited** (within the burst limit)
|
||||
- Other `?i=` values of a query searched in the last hour reuse its results, so they don't count (except on `ai.`, where each `i` is a new image)
|
||||
- If Brave's spending limit is hit, searches use Bing; if Bing also blocks us, `donate.webp` is served (not cached)
|
||||
- Searches that find nothing count toward the limit. If the AI image served instead fails too (or there's none, on `free.`), the failure is remembered for 24h (`bad.webp`)
|
||||
- If Brave or Firecrawl is out of credits, searches use the other; if both are, `donate.webp` is served (not cached)
|
||||
- Searches that find nothing count toward the limit, and are remembered for 24h (`bad.webp`). So is a fallback AI image that fails
|
||||
- A Cloudflare WAF burst limit applies to all requests (HTTP 429). Its counters are approximate (not shared perfectly across Cloudflare servers), so bursts may slip slightly past it
|
||||
|
||||
## Caching
|
||||
@@ -185,9 +186,17 @@ Free community service. Donations help cover API and infrastructure costs.
|
||||
4. Subscribe — you get **$5 in free monthly credits** (covers 1,000 queries/month)
|
||||
5. Go to your [API dashboard](https://api.search.brave.com/app/#/subscriptions)
|
||||
6. Copy your **API key** (starts with `BSA...`)
|
||||
7. Optional: set a monthly spending limit there so you never pay past the credit. Once it's reached, searches fall back to Bing
|
||||
7. Optional: set a monthly spending limit there so you never pay past the credit. Once it's reached, its searches fall back to Firecrawl
|
||||
|
||||
### 2. Openverse API Credentials (Optional)
|
||||
### 2. Firecrawl API Key
|
||||
|
||||
1. Sign up at [firecrawl.dev](https://www.firecrawl.dev/) (no card needed)
|
||||
2. The free plan gives **1,000 credits a month**, which is 500 searches at 2 credits per 10 results. Once they run out, Firecrawl refuses (HTTP 402) and its searches fall back to Brave
|
||||
3. Copy your **API key** (starts with `fc-`)
|
||||
|
||||
A third of queries go to Firecrawl first and the rest to Brave (`FIRECRAWL_SHARE`), so both run out at about the same time.
|
||||
|
||||
### 3. Openverse API Credentials (Optional)
|
||||
|
||||
Free images work without credentials, but anonymous Openverse requests are limited to 200/day. Register once to get higher limits:
|
||||
|
||||
@@ -201,7 +210,7 @@ curl -X POST -H "Content-Type: application/json" -d '{"name":"<unique app name>"
|
||||
|
||||
The function exchanges them for an access token (valid ~12h) and caches it in KV, fetching a new one automatically when it expires.
|
||||
|
||||
### 3. Cloudflare Resources
|
||||
### 4. Cloudflare Resources
|
||||
|
||||
Create in your Cloudflare dashboard:
|
||||
|
||||
@@ -210,7 +219,7 @@ Create in your Cloudflare dashboard:
|
||||
| R2 Bucket | `direct-img-store` | Stores cached images |
|
||||
| KV Namespace | `DIRECT_IMG_CACHE` | Openverse and SurrealDB access tokens |
|
||||
|
||||
### 4. Pages Bindings
|
||||
### 5. Pages Bindings
|
||||
|
||||
**Settings → Functions → Bindings:**
|
||||
|
||||
@@ -219,13 +228,14 @@ Create in your Cloudflare dashboard:
|
||||
| R2 Bucket | `R2_IMAGES` | `direct-img-store` |
|
||||
| KV Namespace | `DIRECT_IMG_CACHE` | `DIRECT_IMG_CACHE` |
|
||||
|
||||
### 5. Secrets
|
||||
### 6. Secrets
|
||||
|
||||
**Settings → Environment variables:**
|
||||
|
||||
| Variable | Description | Required |
|
||||
|---|---|---|
|
||||
| `BRAVE_API_KEY` | Brave Search API key | Yes |
|
||||
| `FIRECRAWL_API_KEY` | Firecrawl API key | Yes |
|
||||
| `OPENVERSE_CLIENT_ID` | Openverse app client ID (free images) | Optional |
|
||||
| `OPENVERSE_CLIENT_SECRET` | Openverse app client secret (free images) | Optional |
|
||||
| `SURREAL_URL` | SurrealDB URL (e.g. `https://db.site.com`) | Yes |
|
||||
@@ -234,19 +244,19 @@ Create in your Cloudflare dashboard:
|
||||
| `NTFY_URL` | ntfy.sh topic URL for alerts | Optional |
|
||||
| `GOATCOUNTER_URL` | GoatCounter site URL for image hit analytics (e.g. `https://direct-img.goatcounter.com`) | Optional |
|
||||
| `GOATCOUNTER_TOKEN` | GoatCounter API token with "Record pageviews" permission | Optional |
|
||||
| `AURORA_API_KEY` | Key for the [AuroraIMG-6M API](https://github.com/direct-img/AuroraIMG-6M) (AI images and the nothing-found fallback). Without it, `ai.` serves `bad.webp` and searches that find nothing serve `bad.webp` (as they always do on `free.`) | Optional |
|
||||
| `AURORA_API_KEY` | Key for the [AuroraIMG-6M API](https://github.com/direct-img/AuroraIMG-6M) (AI images, and the fallback for an `?i=` past a search's working images). Without it, `ai.` and those `?i=` serve `bad.webp` | Optional |
|
||||
|
||||
### 6. WAF Rules
|
||||
### 7. WAF Rules
|
||||
|
||||
**Security → WAF → Rate limiting rules:**
|
||||
|
||||
1. **Rate Limit** — 10 req/10s per IP → Block 10s
|
||||
|
||||
### 7. Deploy
|
||||
### 8. Deploy
|
||||
|
||||
Fork this repo, connect to Cloudflare Pages, deploy.
|
||||
|
||||
### 8. Free and AI Images Domains
|
||||
### 9. Free and AI Images Domains
|
||||
|
||||
**Pages project → Custom domains → Set up a domain:** add `free.<your-domain>` and `ai.<your-domain>`. Any hostname starting with `free.` serves free images, and any starting with `ai.` serves AI images; everything else serves regular search.
|
||||
|
||||
@@ -267,7 +277,7 @@ Every request looks up its key here first, and a cached image is served from tha
|
||||
| `FREE:<src or auto>:<i>:<query>` (e.g. `FREE:wikimedia:2:orange cat`) | free images |
|
||||
| `AI:<src>:<i>:<query>` (e.g. `AI:auroraimg-6m:2:frog wizard`) | AI images |
|
||||
|
||||
An AI image served because a search found nothing is stored under the search's own key, like any other image. A search where that also fails, or a `free.` search (which gets no AI image), stores an empty object with `err` custom metadata instead, which serves `bad.webp` for 24h. After that the query is searched again.
|
||||
An AI image served for an `?i=` past a search's working images is stored under the search's own key, like any other image. A search that finds nothing, one whose AI image fails too, or a `free.` search with too few images (which gets no AI image) stores an empty object with `err` custom metadata instead, which serves `bad.webp` for 24h. After that the query is searched again.
|
||||
|
||||
Add an object lifecycle rule in **R2 → direct-img-store → Settings** that applies to all prefixes and deletes uploaded objects after **90 days**. It's required: nothing else expires cached images.
|
||||
|
||||
@@ -301,9 +311,9 @@ After deploying, visit `https://<your-domain>/_setup` once. It creates the `dire
|
||||
- **Cloudflare KV** — API access tokens
|
||||
- **SurrealDB (v2.6.5)** — atomic rate limiting, search locks + cached result lists
|
||||
- **Cloudflare WAF** — layer 7 mitigation
|
||||
- **Brave Image Search API** — image sourcing, with Bing as the fallback
|
||||
- **Brave Image Search API + Firecrawl Search API** — image sourcing, each the other's fallback
|
||||
- **Openverse API + Wikimedia Commons API** — free image sourcing
|
||||
- **[AuroraIMG-6M](https://github.com/direct-img/AuroraIMG-6M) on Dokploy** — AI images and the nothing-found fallback
|
||||
- **[AuroraIMG-6M](https://github.com/direct-img/AuroraIMG-6M) on Dokploy** — AI images and the fallback for an `?i=` past a search's working images
|
||||
|
||||
---
|
||||
|
||||
|
||||
+1
-1
@@ -1,6 +1,6 @@
|
||||
<!doctype html>
|
||||
<!--
|
||||
Source of assets/donate.webp, served when Brave and Bing are both out of searches. Render at 900x382 with:
|
||||
Source of assets/donate.webp, served when Brave and Firecrawl are both out of searches. Render at 900x382 with:
|
||||
chrome --headless --no-sandbox --hide-scrollbars --force-device-scale-factor=1 --window-size=900,382 --virtual-time-budget=8000 --screenshot=donate.png design/donate.html
|
||||
ffmpeg -i donate.png -c:v libwebp -quality 90 -compression_level 6 assets/donate.webp
|
||||
-->
|
||||
|
||||
+45
-27
@@ -1,5 +1,5 @@
|
||||
import { braveImageSearch } from "./_utils/brave.js";
|
||||
import { bingImageSearchFallback } from "./_utils/bing.js";
|
||||
import { firecrawlImageSearch, FIRECRAWL_RESULTS } from "./_utils/firecrawl.js";
|
||||
import { freeImageUrls, parseFreeSource, FREE_UA } from "./_utils/free.js";
|
||||
import { isFlickrPlaceholder } from "./_utils/flickr.js";
|
||||
import { aiImage, parseAiSource, DEFAULT_AI_SOURCE } from "./_utils/ai.js";
|
||||
@@ -9,8 +9,9 @@ import * as LIMITS from "../assets/rate-limit.js";
|
||||
const TTL_SECONDS = 90 * 24 * 60 * 60;
|
||||
// How long a failed search keeps serving bad.webp before the query is searched again
|
||||
const ERR_SECONDS = 24 * 60 * 60;
|
||||
// Highest ?i= per site: 40 on the main site (100 Brave results), 20 on free.* (~50 Openverse + Wikimedia
|
||||
// results) and 1000 on ai.* (each is a new image, so there's no list to run out).
|
||||
// Highest ?i= per site: 40 on the main site (100 Brave results; a Firecrawl query's ?i= past FIRECRAWL_RESULTS gets
|
||||
// an AI image), 20 on free.* (~50 Openverse + Wikimedia results) and 1000 on ai.* (each is a new image, so there's no
|
||||
// list to run out).
|
||||
// Higher values serve max-i.webp, which lists these, so redraw it from design/max-i.html when they change.
|
||||
const MAX_INDEX = { "": 40, free: 20, ai: 1000 };
|
||||
// How long a search may hold its query's lock; longer than the 20s image deadline plus the search itself
|
||||
@@ -19,6 +20,10 @@ const LOCK_SECONDS = 45;
|
||||
const LIST_SECONDS = 60 * 60;
|
||||
// Image candidates downloaded at once, so one slow or dead host doesn't stall the rest for its full 5s timeout
|
||||
const PARALLEL = 4;
|
||||
// Share of main-site queries searched on Firecrawl first, the rest on Brave: Firecrawl's free plan gives 500 searches a
|
||||
// month and Brave's $5 monthly credit ~1,000, so both run out together. Picked from the query's hash, so a query always
|
||||
// gets the same provider and its ?i= keep their order.
|
||||
const FIRECRAWL_SHARE = 500 / 1500;
|
||||
|
||||
export async function onRequest(context) {
|
||||
const { request, env, params } = context;
|
||||
@@ -76,7 +81,7 @@ export async function onRequest(context) {
|
||||
|
||||
const ip = request.headers.get("cf-connecting-ip") || "unknown";
|
||||
const today = new Date().toISOString().slice(0, 10);
|
||||
// free.* and ai.* get their own counters (prefixed so main-site ids stay unchanged) since they never spend Brave credits
|
||||
// free.* and ai.* get their own counters (prefixed so main-site ids stay unchanged) since they never spend search credits
|
||||
const rateId = await sha256(`${site ? `${site}:` : ""}${ip}:${today}`);
|
||||
const limit = free ? LIMITS.FREE_DAILY_LIMIT : ai ? LIMITS.AI_DAILY_LIMIT : LIMITS.DAILY_LIMIT;
|
||||
let count = 1, wait = false, locked = false, list = null;
|
||||
@@ -183,10 +188,11 @@ COMMIT TRANSACTION;`;
|
||||
return new Response(buffer, { headers: imageHeaders(contentType, TTL_SECONDS * 1000) });
|
||||
};
|
||||
|
||||
// Nothing found: serve an AI image of the query instead, cached like any other image. bad.webp (kept ERR_SECONDS) is
|
||||
// only left for when that fails too, or for free.*, which serves free images only.
|
||||
const fail = async (t, m, tag, p) => {
|
||||
const fallback = !free && await aiImage(DEFAULT_AI_SOURCE, query, i, env);
|
||||
// Fewer than i working images: serve an AI image of the query instead, cached like any other image. bad.webp (kept
|
||||
// ERR_SECONDS) is left for when that fails too, when the search found nothing at all, or for free.*, which serves free
|
||||
// images only.
|
||||
const fail = async (t, m, tag, p, aiFallback = !free) => {
|
||||
const fallback = aiFallback && await aiImage(DEFAULT_AI_SOURCE, query, i, env);
|
||||
note({ title: fallback.success ? `${t}, Served AI Image` : t, message: fallback && !fallback.success ? `${m}\nAI fallback failed: ${fallback.reason}` : m, tags: tag, priority: p });
|
||||
if (fallback.success) return await store(fallback);
|
||||
await env.R2_IMAGES.put(r2Key, "", { customMetadata: { err: "1" } });
|
||||
@@ -207,10 +213,14 @@ COMMIT TRANSACTION;`;
|
||||
return asset("bad.webp");
|
||||
}
|
||||
|
||||
// A Firecrawl query's search returns only FIRECRAWL_RESULTS, so a higher ?i= skips it (and its credits) for AI
|
||||
const firecrawl = !list && !free && parseInt((await sha256(query)).slice(0, 8), 16) < FIRECRAWL_SHARE * 2 ** 32;
|
||||
if (firecrawl && i > FIRECRAWL_RESULTS) return await fail("Past Firecrawl Results", `?i= past ${FIRECRAWL_RESULTS} on a Firecrawl query: ${cacheKey}`, "robot", 2);
|
||||
|
||||
// The query's cached results, or a fresh search. pos holds the indexes of the working images found so far and next
|
||||
// the first URL not tried yet, so a known ?i= downloads only its own image and a new one carries on from next.
|
||||
let { urls, pos, next } = list || { urls: await (free ? freeImageUrls(query, src, env) : webImageUrls(query, env, note)), pos: [], next: 0 };
|
||||
// Brave and Bing both failed: ask for donations instead of an AI image, and cache nothing since it's temporary
|
||||
let { urls, pos, next } = list || { urls: await (free ? freeImageUrls(query, src, env) : webImageUrls(query, env, note, firecrawl)), pos: [], next: 0 };
|
||||
// Brave and Firecrawl both failed: ask for donations instead, and cache nothing since it's temporary
|
||||
if (!urls) {
|
||||
release();
|
||||
return asset("donate.webp");
|
||||
@@ -218,13 +228,15 @@ COMMIT TRANSACTION;`;
|
||||
// Starting at a known image skips the ones before it, so it can't add to pos
|
||||
const known = pos.length >= i;
|
||||
let at = known ? pos[i - 1] : next, found = known ? i - 1 : pos.length, imgResult = null, timedOut = false;
|
||||
// Too few results left to reach i (e.g. ?i=9 of 10 with 2 dead links) skips straight to the AI image
|
||||
const reachable = known || found + urls.length - at >= i;
|
||||
const GLOBAL_DEADLINE = Date.now() + 20000;
|
||||
const failReasons = [], queue = [], abort = new AbortController();
|
||||
|
||||
// Keeps up to PARALLEL downloads in flight, queued in result order. A known ?i= expects its image to work, so it
|
||||
// only downloads others once something fails.
|
||||
const fill = () => {
|
||||
while (!timedOut && at < urls.length && queue.length < (known && !failReasons.length ? 1 : PARALLEL)) {
|
||||
while (reachable && !timedOut && at < urls.length && queue.length < (known && !failReasons.length ? 1 : PARALLEL)) {
|
||||
const remaining = GLOBAL_DEADLINE - Date.now();
|
||||
if (remaining <= 500) timedOut = true;
|
||||
else queue.push({ n: at, res: fetchImage(urls[at++], Math.min(remaining, 5000), free ? FREE_UA : undefined, abort.signal) });
|
||||
@@ -254,16 +266,16 @@ COMMIT TRANSACTION;`;
|
||||
abort.abort();
|
||||
if (timedOut) failReasons.push("Global timeout reached");
|
||||
|
||||
// Saved even if no image worked, so other ?i= of the query don't search again, but not if the search found nothing,
|
||||
// since that can also mean Brave and Bing failed. A cached list is only updated if this got further than its saver.
|
||||
// Saved even if no image worked, so other ?i= of the query don't search again, but not if the search found nothing.
|
||||
// A cached list is only updated if this got further than its saver.
|
||||
if (db && urls.length && next > (list ? list.next : -1)) context.waitUntil(db(list
|
||||
? `UPDATE results:\`${listHash}\` SET pos = ${JSON.stringify(pos)}, next = ${next} WHERE next < ${next};`
|
||||
: `UPSERT results:\`${listHash}\` SET urls = ${JSON.stringify(urls)}, pos = ${JSON.stringify(pos)}, next = ${next}, exp = time::now() + ${LIST_SECONDS}s;`
|
||||
).catch(() => {}));
|
||||
|
||||
if (!urls.length) return await fail("Search Failed", `${free ? "No free results" : "Both Brave and Bing returned no results"} for: ${cacheKey}`, "question", 3);
|
||||
if (!urls.length) return await fail("Search Failed", `${free ? "No free results" : "Brave and Firecrawl found nothing"} for: ${cacheKey}`, "question", 3, false);
|
||||
if (!imgResult) {
|
||||
const reasonStr = failReasons.slice(0, 6).join(", ") + (failReasons.length > 6 ? ", ..." : "");
|
||||
const reasonStr = reachable ? failReasons.slice(0, 6).join(", ") + (failReasons.length > 6 ? ", ..." : "") : `only ${urls.length} results`;
|
||||
return await fail("Fetch Error (502)", `Found ${found} of ${i} working images for: ${cacheKey}\nReasons: ${reasonStr}`, "boom,x", 4);
|
||||
}
|
||||
|
||||
@@ -279,18 +291,24 @@ function parseIndex(searchParams) {
|
||||
return /^\d+$/.test(i) && +i >= 1 ? +i : null;
|
||||
}
|
||||
|
||||
// Brave results, or Bing's if Brave has none or refuses (e.g. once its spending limit is hit). null if Bing fails too,
|
||||
// which means we're out of searches rather than that nothing exists.
|
||||
async function webImageUrls(query, env, note) {
|
||||
const brave = await braveImageSearch(query, env.BRAVE_API_KEY);
|
||||
if (brave.urls?.length) return brave.urls;
|
||||
note(brave.error
|
||||
? { title: "Brave Unavailable", message: `Brave failed for: ${query}. Trying Bing Fallback.\nBrave: ${brave.error}`, tags: "warning,mag", priority: 3 }
|
||||
: { title: "Brave Search Empty", message: `No results for: ${query}. Trying Bing Fallback.`, tags: "warning,mag", priority: 3 });
|
||||
const bing = await bingImageSearchFallback(query);
|
||||
if (bing.urls) return bing.urls;
|
||||
note({ title: "Bing Blocked, Served donate.webp", message: `Bing failed for: ${query}\nBing: ${bing.error}\nBrave: ${brave.error || "no results"}`, tags: "rotating_light,x", priority: 4 });
|
||||
return null;
|
||||
// The query's provider's results (see FIRECRAWL_SHARE), or the other's if it has none or refuses (e.g. once its
|
||||
// credits run out). [] if either searched and found nothing, null if both failed, which means we're out of searches
|
||||
// rather than that nothing exists.
|
||||
async function webImageUrls(query, env, note, firecrawl) {
|
||||
const providers = [["Brave", () => braveImageSearch(query, env.BRAVE_API_KEY)], ["Firecrawl", () => firecrawlImageSearch(query, env.FIRECRAWL_API_KEY)]];
|
||||
if (firecrawl) providers.reverse();
|
||||
let searched = false;
|
||||
for (const [n, [name, search]] of providers.entries()) {
|
||||
const res = await search();
|
||||
if (res.urls?.length) return res.urls;
|
||||
searched ||= !res.error;
|
||||
const then = n ? "" : ` Trying ${providers[1][0]}.`;
|
||||
note(res.error
|
||||
? { title: `${name} Unavailable`, message: `${name} failed for: ${query}.${then}\n${name}: ${res.error}`, tags: "warning,mag", priority: 3 }
|
||||
: { title: `${name} Search Empty`, message: `No results for: ${query}.${then}`, tags: "warning,mag", priority: 3 });
|
||||
}
|
||||
if (!searched) note({ title: "Out of Searches, Served donate.webp", message: `Brave and Firecrawl both failed for: ${query}`, tags: "rotating_light,x", priority: 4 });
|
||||
return searched ? [] : null;
|
||||
}
|
||||
|
||||
async function setupSurreal(env) {
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
// AI images for ai.direct-img.link, and the fallback when a search finds nothing
|
||||
// AI images for ai.direct-img.link, and the fallback for an ?i= past a search's working images
|
||||
const SOURCES = { "auroraimg-6m": auroraImage };
|
||||
export const DEFAULT_AI_SOURCE = "auroraimg-6m";
|
||||
|
||||
|
||||
@@ -1,23 +0,0 @@
|
||||
// { urls }, or { error } if Bing failed. Bing finds something for nearly any query (even gibberish), so a page without
|
||||
// results almost always means it's blocking us (e.g. with a CAPTCHA) or changed its page format.
|
||||
export async function bingImageSearchFallback(query) {
|
||||
try {
|
||||
// SafeSearch off, like Brave. The cookie is what actually sets it; adlt= alone is ignored.
|
||||
const res = await fetch(`https://www.bing.com/images/search?q=${encodeURIComponent(query)}&adlt=off`, {
|
||||
headers: {
|
||||
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36",
|
||||
"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
|
||||
"Cookie": "SRCHHPGUSR=ADLT=OFF"
|
||||
}
|
||||
});
|
||||
if (!res.ok) return { error: `HTTP ${res.status}` };
|
||||
const html = await res.text();
|
||||
|
||||
// Bing encodes the image payload inside HTML entity attributes and script tags
|
||||
const urls = [...html.matchAll(/"murl":"(https:[^&"]+)"/g), ...html.matchAll(/"murl":"(https:[^"]+)"/g)]
|
||||
.map(m => m[1].replace(/\\\//g, '/'));
|
||||
return urls.length ? { urls: [...new Set(urls)] } : { error: `No results in page (${html.length} bytes, ${res.url})` };
|
||||
} catch (err) {
|
||||
return { error: err.message };
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,22 @@
|
||||
// Results per search. Firecrawl charges 2 credits per 10, so the free plan's 1,000 credits a month buy 500 searches
|
||||
export const FIRECRAWL_RESULTS = 10;
|
||||
|
||||
// { urls } ([] if none found), or { error } if Firecrawl refused or failed, e.g. 402 once the month's credits run out,
|
||||
// or 429 past the free plan's 10 searches a minute or 2 at once
|
||||
export async function firecrawlImageSearch(query, apiKey) {
|
||||
try {
|
||||
const res = await fetch("https://api.firecrawl.dev/v2/search", {
|
||||
method: "POST",
|
||||
headers: { "Authorization": `Bearer ${apiKey}`, "Content-Type": "application/json" },
|
||||
// SafeSearch off, like Brave. It's off by default too, but set so a default change can't filter results
|
||||
body: JSON.stringify({ query, sources: ["images"], limit: FIRECRAWL_RESULTS, safe: false }),
|
||||
signal: AbortSignal.timeout(10000)
|
||||
});
|
||||
if (!res.ok) return { error: `HTTP ${res.status}: ${(await res.text()).slice(0, 300)}` };
|
||||
const data = await res.json();
|
||||
if (!data.success) return { error: JSON.stringify(data).slice(0, 300) };
|
||||
return { urls: data.data?.images?.map(r => r.imageUrl).filter(url => !!url) || [] };
|
||||
} catch (err) {
|
||||
return { error: err.name === "TimeoutError" ? "Timeout" : err.message };
|
||||
}
|
||||
}
|
||||
+2
-2
@@ -113,7 +113,7 @@ Use images to complement your responses, powered by Brave.</code></pre>
|
||||
<tr><td><code>i</code></td><td><code>1</code>–<code>1000</code> (a different image each)</td><td><code>1</code></td></tr>
|
||||
</tbody>
|
||||
</table>
|
||||
<p style="font-size:0.85rem;color:var(--text-muted);">* The same query always gives the same image. When a search on direct-img.link finds nothing, an AI image is served instead. Made with <a href="https://github.com/direct-img/AuroraIMG-6M">AuroraIMG-6M</a>.</p>
|
||||
<p style="font-size:0.85rem;color:var(--text-muted);">* The same query always gives the same image. When a search on direct-img.link has fewer than <code>?i=</code> working images, an AI image is served instead. Made with <a href="https://github.com/direct-img/AuroraIMG-6M">AuroraIMG-6M</a>.</p>
|
||||
</section>
|
||||
|
||||
<section>
|
||||
@@ -135,7 +135,7 @@ Use images to complement your responses, powered by Brave.</code></pre>
|
||||
<section>
|
||||
<div class="card">
|
||||
<h3>Caching</h3>
|
||||
<p>Images are cached for 90 days. Subsequent requests for the same query are served instantly. A search's results are kept for an hour, so trying other <code>?i=</code> values right after is free and fast. Searches on direct-img.link that find nothing serve an AI image instead, cached the same way.</p>
|
||||
<p>Images are cached for 90 days. Subsequent requests for the same query are served instantly. A search's results are kept for an hour, so trying other <code>?i=</code> values right after is free and fast. An <code>?i=</code> past a search's working images on direct-img.link serves an AI image instead, cached the same way.</p>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
|
||||
Reference in New Issue
Block a user