direct-img/direct-img.link
1
0
Fork 0
mirror of https://github.com/direct-img/direct-img.link.git synced 2026-10-18 10:37:52 +00:00
Code Issues Packages Projects Releases Wiki Activity

Fix: Dedupe concurrent searches for the same query with a SurrealDB lock

Browse Source 
Two misses for the same query at once each ran their own search (ntfy
showed Search #24 and #25 for one image). An in-memory fix wouldn't
hold: a test of concurrent same-query requests on this project found
they share an isolate only ~25% of the time.

The rate-limit query is now one transaction that also claims a
per-query lock, so it's still a single SurrealDB round trip. Requests
that find the lock held write nothing (so aren't counted), poll until
it's released, then serve the image from R2. Write conflicts roll the
whole transaction back and are retried.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
multipleof4
2026-10-11 14:00:28 -07:00
co-authored by Claude Opus 5.5
parent f91d6c6616
commit fd5694669a
2 changed files with 72 additions and 44 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+6 -4
View File
@@ -244,13 +244,15 @@ Other keys share the namespace. They use uppercase prefixes, which can never col
| `FREE:<src or auto>:<i>:<query>` (e.g. `FREE:wikimedia:2:orange cat`) | Same as above | 90 days |
| `OPENVERSE_TOKEN` | Openverse access token | Token lifetime minus 10 min |
### Database: `SurrealDB` (Rate Limiting)
### Database: `SurrealDB` (Rate Limiting + Search Locks)
Using atomic database transactions over HTTP to track per-IP/per-day search frequencies securely and rapidly.
Using atomic database transactions over HTTP to track per-IP/per-day search frequencies securely and rapidly.
The same transaction also claims a per-query lock (`lock:<sha256-of-KV-key>`, expires after 45s), so when several requests miss the cache for the same query at once (on any Cloudflare server), only one searches. The others wait for it to finish, then serve the image it stored in R2, and aren't counted toward the daily limit. The lock is deleted as soon as the result is stored.
**Note:** This implementation is tested and verified for **SurrealDB v2.3.10**.
After deploying, visit `https://<your-domain>/_setup` once. It creates the `direct_img` namespace, `rate_limit` database, `rate` table and an `updated_at` index (used by the cleanup query). It's idempotent, so visiting again is harmless, and it returns `{"ok":true}` on success.
After deploying, visit `https://<your-domain>/_setup` once. It creates the `direct_img` namespace, `rate_limit` database, `rate` table, an `updated_at` index (used by the cleanup query) and the `lock` table. It's idempotent, so visiting again is harmless, and it returns `{"ok":true}` on success.
---
@@ -259,7 +261,7 @@ After deploying, visit `https://<your-domain>/_setup` once. It creates the `dire
- **Cloudflare Pages** — hosting + edge functions
- **Cloudflare R2** — image storage
- **Cloudflare KV** — generic lookups
- **SurrealDB (v2.3.10)** — atomic rate limiting
- **SurrealDB (v2.3.10)** — atomic rate limiting + search locks
- **Cloudflare WAF** — layer 7 mitigation
- **Brave Image Search API** — image sourcing
- **Openverse API + Wikimedia Commons API** — free image sourcing