Bot management is the discipline of telling automated traffic apart and treating each kind appropriately: letting search engines and monitoring through, throttling nuisance crawlers, and stopping the scrapers that copy your catalogue and rehost it. On adult platforms the economics are stark — the content is the product, and a scraper that pulls full-resolution media converts directly into lost traffic and lost revenue. See Web application and API protection for the broader rule engine this sits within.
Not all bots are hostile
Blocking all automation breaks the platform. Search crawlers drive discovery, uptime monitors keep you honest, and payment or fraud providers call your APIs. The first job is allowlisting known-good bots:
- Verify the claimed crawler by its published source ranges and, where the operator supports it, reverse DNS. A
User-Agentstring alone is trivially spoofed. - Give verified bots a separate, generous rate budget so they are not caught by the same limits that stop abuse.
- Review the allowlist periodically — stale entries become an evasion path.
Everything not verified is treated as unknown and scored.
Detecting headless and scripted clients
Scrapers increasingly run headless browsers, so a User-Agent check is not enough. Layer signals that are harder to fake:
- TLS and HTTP/2 fingerprints (JA3/JA4) that differ from real browser stacks.
- Behaviour: request ordering, timing regularity, cursor and scroll patterns, missing static-asset fetches, and depth-first catalogue traversal.
- Consistency: a claimed browser that cannot execute expected client-side behaviour, or a session that never loads the player.
- Network reputation: hosting and proxy ranges, and rapid address rotation.
Score these signals together. A single one produces false positives; a combination is much more reliable.
Scrape-to-rehost economics
Understand what the scraper is actually doing, because it changes the defence. A scraper pulling metadata (titles, tags, thumbnails) is building a competing catalogue cheaply. A scraper pulling full media is stealing the expensive asset and your origin egress. Metadata scraping is cheap to run and hard to distinguish from normal browsing; media scraping is bandwidth-heavy and shows up as anomalous byte offload per client. Track bytes per client and per session, not just request counts — a scraper that fetches a few huge files looks quiet in request metrics and loud in byte metrics. This mirrors the byte-versus-request distinction in Cache control and cache keys.
Hotlink versus scraping
These are different problems with different fixes:
| Problem | What happens | Primary defence |
|---|---|---|
| Hotlink | Another page embeds your media in its own player | Referrer policy, token binding, signed URLs |
| Scraping | An automated client downloads and rehosts your media | Auth, fingerprinting, rate/byte limits, signed URLs |
| Mirroring | A whole site is cloned and served elsewhere | Signed URLs plus watermarking and takedowns |
Hotlink protection via Referer is deterrence — the header is spoofable and absent for direct downloads. It stops casual embedding, not a determined scraper. See Edge access policy for referrer and method rules.
Signed URLs are the real defence
The reliable way to stop unauthorised media access is to make the URL itself time-limited and entitlement-bound. A signed URL carries a signature over the resource path and a short expiry; the edge verifies it before serving. Design it so that:
- Expiry is short — minutes for segments, longer for a playback session, never days.
- The signature binds the viewer or session, not just the path, so a leaked URL has limited value.
- Signing happens at the edge, using the token-validation patterns in Edge Workers use cases, so verification does not add origin load.
- The cache key excludes the signature, so signed requests still share cache entries.
A scraper that must obtain a fresh, viewer-bound token for every byte has had its economics changed from “copy once” to “authenticate continuously”. That, combined with rate and byte limits and takedown processes, is what actually protects the catalogue.
Response strategy
Match the response to the confidence level and the goal:
- Verified good bots: allow, separate budget.
- Low-confidence automation: throttle or challenge, keep serving.
- High-confidence scrapers: block, and feed the fingerprint and range back into policy.
- Abusive but ambiguous: prefer degrading the experience (lower resolution, delay) over a hard block that generates a support complaint from a real viewer behind a shared address.
Where AdultInfra fits
AdultInfra designs signed-URL and fingerprinting policy that protects media without breaking legitimate crawlers or shared-address viewers. Content ownership, watermarking, and lawful takedowns remain the customer’s responsibility. Start with Porn tube site or contact us to review your exposure.