diff --git a/README.md b/README.md index b82bb81..a18ad26 100644 --- a/README.md +++ b/README.md @@ -270,7 +270,7 @@ Fork this repo and add these **GitHub Actions secrets** (Settings → Secrets an Change the three `routes` in `wrangler.jsonc` to your domain, `free.` and `ai.` (each needs a proxied DNS record; with nothing behind it, use `AAAA 100::`). Any hostname starting with `free.` serves free images, and any starting with `ai.` serves AI images; everything else serves regular search. -Then add the rules that let static images work (needs a token with Zone: Transform Rules Edit): +Then add the rules that let static images work (needs a token with Zone: Transform Rules Edit and Zone WAF Edit): ```bash CLOUDFLARE_API_TOKEN=... node scripts/rules.js free. ai. @@ -299,24 +299,25 @@ Add an object lifecycle rule in **R2 → direct-img-store → Settings** that ap ### Static Images -Every request runs the Worker, which counts toward the free plan's 100,000 requests a day, except requests for files in `public/`: those are free, unlimited and keep working past the limit. So before each deploy, `scripts/promote.js` copies the most viewed images from R2 into `public/`: +Every request runs the Worker, which counts toward the free plan's 100,000 requests a day, except requests for files in `public/`: those are free, unlimited and keep working past the limit. So before each deploy, `scripts/promote.js` copies the most viewed images (any `?i=` and `?src=`) from R2 into `public/`: -- Views come from Cloudflare's zone analytics over the last 7 days, which count static files too, so a promoted image that stops being viewed drops out +- Views come from the last 7 days of Cloudflare's zone analytics, which count static files too, so a promoted image that stops being viewed drops out. Analytics has no query string, so each query's views are split across its `?i=` and `?src=` by a WAF custom rule (`scripts/rules.js`) that logs every image request with its query string. The job reads up to 1,000 logged requests per hour, enough to estimate the split on a busy day. These show up as **Skip** events in **Security → Events** - The top 19,900 with at least `MIN_VIEWS` (20) are promoted, up to `MAX_BYTES` (4 GB) in total. The free plan allows 20,000 static files per deploy - Only JPEG, PNG, GIF and WebP are promoted, and only queries of up to 8 words (see below). Static files get their type from their extension, which these have none of, so `public/_headers` declares them all `image/jpeg`, and browsers detect these four formats from the bytes anyway - Images are cached between runs (Actions cache), so only newly promoted ones download from R2 - An image is dropped once it's 90 days old, the same as the R2 lifecycle rule, so a popular query (e.g. `current us president`) is still searched again instead of staying static forever. It's promoted again the next day if it's still popular -Static files ignore query strings and are shared by all three hostnames, so URL Rewrite Rules (`scripts/rules.js`) turn each bare query into its static file's path, under its hostname, first. The visitor's URL doesn't change: +Static files ignore query strings and are shared by all three hostnames, so URL Rewrite Rules (`scripts/rules.js`) turn each image request into its static file's path first: under its hostname, named after its `?i=` and `?src=`. The visitor's URL doesn't change: | Request | Looked up as | |---|---| -| `direct-img.link/orange+cat` | `/direct-img.link/1/orange%2Bcat` (lowercased, `+` encoded like the file's name) | -| `free.direct-img.link/Orange+Cat` | `/free.direct-img.link/1/orange%2Bcat` | +| `direct-img.link/orange+cat` | `/direct-img.link/orange%2Bcat/i-` (lowercased, `+` encoded like the file's name) | +| `direct-img.link/Orange+Cat?i=2` | `/direct-img.link/orange%2Bcat/i2-` | +| `free.direct-img.link/orange+cat?src=wikimedia&i=2` | `/free.direct-img.link/orange%2Bcat/i2-wikimedia` (either param order) | -A promoted file there serves it; otherwise the Worker gets the request, unfolds the path and handles it as before. Requests with any params (`?i=`, `?src=`), a literal `+` (`%2B`) or a trailing slash aren't rewritten and always go to the Worker. Analytics has no query string, so `?i=` views count toward the query's `?i=1`, and only `?i=1` is promoted. +A promoted file there serves it; otherwise the Worker gets the request, restores the path and handles it as before. Requests with other params, a literal `+` (`%2B`) or a trailing slash aren't rewritten and always go to the Worker. `?i=1` and a bare query are separate files (`i1-` and `i-`), each promoted on its own views. -A rule's condition sees the original path, and the free plan has no `regex_replace` and allows one `wildcard_replace` per rule, so there's one rule per number of `+`s (0–7, 8 of the plan's 10). Queries of 9+ words are never promoted. +The static file server redirects a name with `+`, `=` or `&` to its encoded form, which would change the visitor's URL, so the rules encode `+` as `%2B` and build the name from the param values. A rule's condition sees the original path, and the free plan has no `regex_replace` and allows one `wildcard_replace` per rule, so there's one rule per number of `+`s (0–7, 8 of the plan's 10). Queries of 9+ words are never promoted. ### KV: `DIRECT_IMG_CACHE` diff --git a/scripts/notify.js b/scripts/notify.js index b8cb684..d70b855 100644 --- a/scripts/notify.js +++ b/scripts/notify.js @@ -31,10 +31,12 @@ if (r.workers) { } if (r.workersError) lines.push(`**Worker requests:** couldn't read (${esc(r.workersError)})`); -// free.direct-img.link/1/orange+cat → [orange cat](https://free.direct-img.link/orange+cat) (free), so each opens on its site +// free.direct-img.link/orange+cat/i2-wikimedia → [orange cat](https://free.direct-img.link/orange+cat?i=2&src=wikimedia) +// i=2 src=wikimedia (free), so each opens on its site const link = id => { - const [host, , name] = id.split("/"), site = host.split(".").length > 2 && host.split(".")[0]; - return `[${esc(name.replaceAll("+", " "))}](https://${host}/${name})${site ? ` (${site})` : ""}`; + const [host, name, file] = id.split("/"), [, i, src] = file.match(/^i([^-]*)-(.*)$/), site = host.split(".").length > 2 && host.split(".")[0]; + const qs = new URLSearchParams(Object.entries({ i, src }).filter(([, v]) => v)).toString(); + return `[${esc(name.replaceAll("+", " "))}](https://${host}/${name}${qs && `?${qs}`})${qs && ` ${esc(qs.replace("&", " "))}`}${site ? ` (${site})` : ""}`; }; const failed = env.STATUS !== "success"; diff --git a/scripts/promote.js b/scripts/promote.js index 3672db5..7b2f31f 100644 --- a/scripts/promote.js +++ b/scripts/promote.js @@ -1,10 +1,11 @@ -// Copies the most viewed images from R2 into public/, so the next deploy serves them as static files: free, unlimited -// and without running the worker. Views come from Cloudflare's analytics, which count static files too, so a promoted -// image that stops being viewed drops out. Run before every deploy (.github/workflows/deploy.yml); writes -// promoted/report.json for scripts/notify.js. promoted/ is kept between runs (Actions cache), so only new images download. +// Copies the most viewed images (any ?i= and ?src=) from R2 into public/, so the next deploy serves them as static +// files: free, unlimited and without running the worker. Views come from Cloudflare's analytics and the WAF rule's logs, +// which count static files too, so a promoted image that stops being viewed drops out. Run before every deploy +// (.github/workflows/deploy.yml); writes promoted/report.json for scripts/notify.js. promoted/ is kept between runs +// (Actions cache), so only new images download. import { mkdir, readFile, writeFile, link, rm, readdir } from "node:fs/promises"; import { dirname } from "node:path"; -import { TTL_SECONDS, MAX_PLUSES, siteOf, normalizeQuery, cacheKey, r2Key } from "../src/_utils/keys.js"; +import { TTL_SECONDS, MAX_INDEX, MAX_PLUSES, parseIndex, siteOf, normalizeQuery, cacheKey, r2Key } from "../src/_utils/keys.js"; import { parseFreeSource } from "../src/_utils/free.js"; import { parseAiSource } from "../src/_utils/ai.js"; @@ -21,6 +22,8 @@ const MAX_BYTES = +(env.MAX_BYTES || 4e9); const TYPES = ["image/jpeg", "image/jpg", "image/pjpeg", "image/png", "image/gif", "image/webp"]; // Queries whose file name the rules or the asset lookup would change, so a promoted file would never match const BAD_NAME = /[+.%\/\\?#\x00-\x1f\x7f]/; +// Logged requests read per hour, enough to split a busy path's views across its ?i= and ?src= +const PER_HOUR = 1000; const CACHE = "promoted", OUT = "public"; const sleep = ms => new Promise(r => setTimeout(r, ms)); @@ -39,31 +42,48 @@ async function gql(query) { return r.data.viewer; } -// The image a request for a bare query (/orange+cat) was for. Analytics has no query string, so a ?i= view counts toward -// its query's ?i=1 too, and only ?i=1 is promoted (other ?i= always run the worker). null for anything else (the -// homepage, assets, probes like /wp-admin/x.php). -async function parse(host, path) { - const site = siteOf(host), m = path.match(/^\/([^/.]+)$/), query = m && normalizeQuery(m[1]); - if (!query || query.length > 200) return null; - const name = query.replaceAll(" ", "+"); - if (BAD_NAME.test(query) || Buffer.byteLength(name) > 200 || name.split("+").length > MAX_PLUSES + 1) return null; - const src = site === "free" ? parseFreeSource(new URLSearchParams()) : site === "ai" ? parseAiSource(new URLSearchParams()) : null; - return { id: `${host}/1/${name}`, r2: await r2Key(site, cacheKey(site, src, 1, query)) }; +// The image a request was for, from its path (/orange+cat) and query string (?i=2&src=wikimedia), checked like the +// worker checks them. Its id is its static file's path, which the rules build the same way +// (free.direct-img.link/orange+cat/i2-wikimedia). null for anything else (the homepage, assets, probes like +// /wp-admin/x.php, params the worker rejects). +async function image(host, path, qs) { + const site = siteOf(host), m = path.match(/^\/([^/.]+)$/), query = m && normalizeQuery(m[1]), p = new URLSearchParams(qs); + if (!query || query.length > 200 || [...p.keys()].some(k => k !== "i" && !(site && k === "src"))) return null; + const i = parseIndex(p), src = site === "free" ? parseFreeSource(p) : site === "ai" ? parseAiSource(p) : null; + if (!i || i > MAX_INDEX[site] || (site && !src)) return null; + const name = query.replaceAll(" ", "+"), file = `i${p.get("i") ?? ""}-${p.get("src") ?? ""}`; + if (BAD_NAME.test(query) || /[^\w-]/.test(file) || Buffer.byteLength(name) > 200 || name.split("+").length > MAX_PLUSES + 1) return null; + return { id: `${host}/${name}/${file}`, r2: await r2Key(site, cacheKey(site, src, i, query)) }; } -// Views per image over the last DAYS: one query per host and day, since a query returns at most 10,000 rows. Only 200s -// count, so redirects (e.g. /favicon.ico) aren't. +// Views per image over the last DAYS. Analytics counts requests per path (one query per host and day, at most 10,000 +// rows each; only 200s, so redirects like /favicon.ico aren't) but has no query string, so each path's count is split +// across its ?i= and ?src= by that day's logged requests (the WAF rule from scripts/rules.js), read an hour at a time +// so a busy day is sampled evenly. A path with none logged that day counts as the bare query. async function views(zoneTag) { - const totals = new Map(), now = Date.now(); + const now = Date.now(), at = h => new Date(now - h * 36e5).toISOString(); + const logged = Array.from({ length: DAYS }, () => new Map()); + for (let h = 0; h < DAYS * 24; h++) { + const { zones } = await gql(`zones(filter: {zoneTag: "${zoneTag}"}) { firewallEventsAdaptive(limit: ${PER_HOUR}, filter: {datetime_geq: "${at(h + 1)}", datetime_lt: "${at(h)}", action: "skip", source: "firewallCustom"}) { clientRequestHTTPHost clientRequestPath clientRequestQuery sampleInterval } }`); + for (const e of zones[0].firewallEventsAdaptive) { + const day = logged[Math.floor(h / 24)], key = e.clientRequestHTTPHost + e.clientRequestPath; + const qs = day.get(key) || day.set(key, new Map()).get(key); + qs.set(e.clientRequestQuery, (qs.get(e.clientRequestQuery) || 0) + e.sampleInterval); + } + } + const totals = new Map(); let total = 0; for (const host of HOSTS) for (let d = 0; d < DAYS; d++) { - const range = `datetime_geq: "${new Date(now - (d + 1) * 864e5).toISOString()}", datetime_lt: "${new Date(now - d * 864e5).toISOString()}"`; + const range = `datetime_geq: "${at((d + 1) * 24)}", datetime_lt: "${at(d * 24)}"`; const { zones } = await gql(`zones(filter: {zoneTag: "${zoneTag}"}) { httpRequestsAdaptiveGroups(limit: 10000, orderBy: [count_DESC], filter: {${range}, clientRequestHTTPHost: "${host}", edgeResponseStatus: 200}) { count dimensions { clientRequestPath } } }`); - for (const { count, dimensions } of zones[0].httpRequestsAdaptiveGroups) { - const img = await parse(host, dimensions.clientRequestPath); - if (!img) continue; - total += count; - totals.set(img.id, { ...img, views: (totals.get(img.id)?.views || 0) + count }); + for (const { count, dimensions: { clientRequestPath: path } } of zones[0].httpRequestsAdaptiveGroups) { + const qs = logged[d].get(host + path) || new Map([["", 1]]), weight = [...qs.values()].reduce((a, b) => a + b, 0); + for (const [q, w] of qs) { + const img = await image(host, path, q); + if (!img) continue; + total += (count * w) / weight; + totals.set(img.id, { ...img, views: (totals.get(img.id)?.views || 0) + (count * w) / weight }); + } } } return { list: [...totals.values()], total }; diff --git a/scripts/rules.js b/scripts/rules.js index 088e010..46e1248 100644 --- a/scripts/rules.js +++ b/scripts/rules.js @@ -1,32 +1,49 @@ -// Sets the zone's URL Rewrite Rules that turn a bare query into its static file's path, under its hostname -// (free.direct-img.link/Orange+Cat → /free.direct-img.link/1/orange%2Bcat), so a promoted image is served without -// running the worker and the visitor's URL doesn't change. +s become %2B (the static file's own encoding), since a + -// would get a redirect to it. Replaces only the rules it made earlier. +// Sets the zone's rules for static images, replacing only the ones it made earlier: +// - URL Rewrite Rules give each image request a static file path under its hostname, named after its ?i= and ?src= +// (free.direct-img.link/Orange+Cat?i=2 → /free.direct-img.link/orange%2Bcat/i2-), so a promoted image is served +// without running the worker and the visitor's URL doesn't change. +s become %2B and the params skip = and &, since +// the static file server redirects to those encodings otherwise. +// - A WAF custom rule logs every image request with its query string (analytics has no query string), which is how +// scripts/promote.js tells ?i= and ?src= apart. It's a skip that skips nothing: it's kept last in its ruleset. // node scripts/rules.js direct-img.link free.direct-img.link ai.direct-img.link -// With no hostnames it removes them. Needs CLOUDFLARE_API_TOKEN with Zone > Transform Rules edit. +// With no hostnames it removes them. Needs CLOUDFLARE_API_TOKEN with Zone > Transform Rules and Zone WAF edit. import { MAX_PLUSES } from "../src/_utils/keys.js"; const env = process.env, hosts = process.argv.slice(2), ZONE = env.ZONE || "direct-img.link", TAG = "direct-img fold:"; -const PHASE = "http_request_transform", PATH = "http.request.uri.path"; +const PATH = "http.request.uri.path", ARG = n => `http.request.uri.args["${n}"][0]`; const api = (path, init) => fetch(`https://api.cloudflare.com/client/v4${path}`, { ...init, headers: { Authorization: `Bearer ${env.CLOUDFLARE_API_TOKEN}`, "Content-Type": "application/json" } }).then(r => r.json()); -// A bare query with no params: not the homepage, assets/, robots.txt or an already folded path. A literal + (%2B) is -// left to the worker, since a static file's %2B stands for a space. -const BARE = `http.host in {${hosts.map(h => JSON.stringify(h)).join(" ")}} and http.request.uri.query eq "" and ${PATH} ne "/" and not ${PATH} contains "." and not ${PATH} wildcard "/*/*" and not lower(${PATH}) contains "%2b"`; +// An image request: not the homepage, assets/, robots.txt, probes like /wp-admin/x.php or an already rewritten path +const IMAGE = `http.host in {${hosts.map(h => JSON.stringify(h)).join(" ")}} and ${PATH} ne "/" and not ${PATH} contains "." and not ${PATH} wildcard "/*/*"`; +// Folded only with no params but i and src (others get bad.webp from the worker), and without a literal + (%2B), since +// a static file's %2B stands for a space +const FOLDABLE = `${IMAGE} and (http.request.uri.query eq "" or all(http.request.uri.args.names[*] in {"i" "src"})) and not lower(${PATH}) contains "%2b"`; const pluses = n => `"${"*+".repeat(n)}*"`; -// A rule's condition sees the original path, and the free plan has no regex_replace and allows one wildcard_replace per -// rule, so each rule handles an exact number of +s; a query with more always runs the worker -const mine = hosts.length ? Array.from({ length: MAX_PLUSES + 1 }, (_, n) => ({ - description: `${TAG} bare query with ${n} +`, - expression: `${BARE} and ${n ? `${PATH} wildcard ${pluses(n)}` : `not ${PATH} contains "+"`} and not ${PATH} wildcard ${pluses(n + 1)}`, - action: "rewrite", - action_parameters: { uri: { path: { expression: `concat("/", http.host, "/1", ${n ? `wildcard_replace(lower(${PATH}), ${pluses(n)}, "${Array.from({ length: n }, (_, k) => `\${${k + 1}}%2B`).join("")}\${${n + 1}}")` : `lower(${PATH})`})` } } }, -})) : []; +const rules = { + // A rule's condition sees the original path, and the free plan has no regex_replace and allows one wildcard_replace + // per rule, so each rule handles an exact number of +s; a query with more always runs the worker. A missing param is + // empty, so a bare query is i-. + http_request_transform: hosts.length ? Array.from({ length: MAX_PLUSES + 1 }, (_, n) => ({ + description: `${TAG} query with ${n} +`, + expression: `${FOLDABLE} and ${n ? `${PATH} wildcard ${pluses(n)}` : `not ${PATH} contains "+"`} and not ${PATH} wildcard ${pluses(n + 1)}`, + action: "rewrite", + action_parameters: { uri: { path: { expression: `concat("/", http.host, ${n ? `wildcard_replace(lower(${PATH}), ${pluses(n)}, "${Array.from({ length: n }, (_, k) => `\${${k + 1}}%2B`).join("")}\${${n + 1}}")` : `lower(${PATH})`}, "/i", ${ARG("i")}, "-", ${ARG("src")})` } } }, + })) : [], + http_request_firewall_custom: hosts.length ? [{ + description: `${TAG} log image requests`, + expression: IMAGE, + action: "skip", + action_parameters: { ruleset: "current" }, + logging: { enabled: true }, + }] : [], +}; const zone = (await api(`/zones?name=${ZONE}`)).result[0].id; -const others = ((await api(`/zones/${zone}/rulesets/phases/${PHASE}/entrypoint`)).result?.rules || []) - .filter(r => !r.description?.startsWith(TAG)) - .map(({ id, version, last_updated, ref, ...r }) => r); -const res = await api(`/zones/${zone}/rulesets/phases/${PHASE}/entrypoint`, { method: "PUT", body: JSON.stringify({ rules: [...others, ...mine] }) }); -console.log(res.success ? `${mine.length} set, ${others.length} others kept` : JSON.stringify(res.errors)); -if (!res.success) process.exitCode = 1; +for (const [phase, mine] of Object.entries(rules)) { + const others = ((await api(`/zones/${zone}/rulesets/phases/${phase}/entrypoint`)).result?.rules || []) + .filter(r => !r.description?.startsWith(TAG)) + .map(({ id, version, last_updated, ref, ...r }) => r); + const res = await api(`/zones/${zone}/rulesets/phases/${phase}/entrypoint`, { method: "PUT", body: JSON.stringify({ rules: [...others, ...mine] }) }); + console.log(phase, res.success ? `${mine.length} set, ${others.length} others kept` : JSON.stringify(res.errors)); + if (!res.success) process.exitCode = 1; +} diff --git a/src/_utils/keys.js b/src/_utils/keys.js index b78f02a..1d9e6b4 100644 --- a/src/_utils/keys.js +++ b/src/_utils/keys.js @@ -10,6 +10,12 @@ export const TTL_SECONDS = 90 * 24 * 60 * 60; // Higher values serve max-i.webp, which lists these, so redraw it from design/max-i.html when they change. export const MAX_INDEX = { "": 40, free: 20, ai: 1000 }; +// ?i=1 or higher (default 1). Returns null if invalid. +export function parseIndex(searchParams) { + const i = searchParams.get("i") || "1"; + return /^\d+$/.test(i) && +i >= 1 ? +i : null; +} + // "free", "ai" or "" (main site), from the hostname's first label export const siteOf = hostname => ["free", "ai"].find(s => hostname.startsWith(`${s}.`)) || ""; diff --git a/src/worker.js b/src/worker.js index ef97f90..1ecddb7 100644 --- a/src/worker.js +++ b/src/worker.js @@ -4,7 +4,7 @@ import { freeImageUrls, parseFreeSource, FREE_UA } from "./_utils/free.js"; import { isFlickrPlaceholder } from "./_utils/flickr.js"; import { aiImage, parseAiSource, DEFAULT_AI_SOURCE } from "./_utils/ai.js"; import { surreal } from "./_utils/surreal.js"; -import { TTL_SECONDS, MAX_INDEX, siteOf, normalizeQuery, cacheKey as keyFor, r2Key as r2KeyFor, sha256 } from "./_utils/keys.js"; +import { TTL_SECONDS, MAX_INDEX, parseIndex, siteOf, normalizeQuery, cacheKey as keyFor, r2Key as r2KeyFor, sha256 } from "./_utils/keys.js"; import * as LIMITS from "../public/assets/rate-limit.js"; // How long a failed search keeps serving bad.webp before the query is searched again @@ -23,8 +23,7 @@ const FIRECRAWL_SHARE = 500 / 1500; export default { fetch: handle }; async function handle(request, env, context) { - const site = siteOf(new URL(request.url).hostname); - const url = unfold(new URL(request.url), site); + const url = unfold(new URL(request.url)), site = siteOf(url.hostname); const path = url.pathname.slice(1); const asset = f => env.ASSETS.fetch(new Request(new URL(`/assets/${f}`, url.origin))); // Every alert ends with the full request URL (site, query, ?i= and ?src=), so it's clear which request triggered it @@ -289,11 +288,6 @@ COMMIT TRANSACTION;`; return job; } -// ?i=1 or higher (default 1). Returns null if invalid. -function parseIndex(searchParams) { - const i = searchParams.get("i") || "1"; - return /^\d+$/.test(i) && +i >= 1 ? +i : null; -} // The query's provider's results (see FIRECRAWL_SHARE), or the other's if it has none or refuses (e.g. once its // credits run out). [] if either searched and found nothing, null if both failed, which means we're out of searches @@ -365,17 +359,15 @@ async function countHit(env, request, query, site, params) { } catch {} } -// The zone's URL Rewrite Rules (scripts/rules.js) turn a bare query into its static file's path, under its hostname -// (free.direct-img.link/orange+cat → /free.direct-img.link/1/orange%2Bcat), so a promoted image is served without -// running the worker. Unfolded back here, so the rest of the worker sees the original request: the rules' %2B were +s, -// since they leave a query's own %2B alone. Links from when ?i= was folded too (/free/2/orange+cat) still work. A folded -// path for another site is rejected like any slash. -function unfold(url, site) { - const m = url.pathname.match(/^\/([^/]+)\/([^/]*)\/([^/]+)$/); - if (!m || ![url.hostname, site || "web"].includes(m[1])) return url; +// The zone's URL Rewrite Rules (scripts/rules.js) give each image request a static file path under its hostname, named +// after its ?i= and ?src= (free.direct-img.link/Orange+Cat?i=2 → /free.direct-img.link/orange%2Bcat/i2-), so a promoted +// image is served without running the worker. Only the path is rewritten (the query string stays), so restoring it is +// enough: the rules' %2B were +s, since they leave a query's own %2B alone. +function unfold(url) { + const m = url.pathname.match(/^\/([^/]+)\/([^/]+)\/i[^/]*$/); + if (!m || m[1] !== url.hostname) return url; const u = new URL(url); - u.pathname = `/${m[3].replace(/%2B/gi, "+")}`; - if (m[2] && m[2] !== "1") u.searchParams.set("i", m[2]); + u.pathname = `/${m[2].replace(/%2B/gi, "+")}`; return u; }