Do you want a ready-made service recommendation, or instructions and code to build your own screenshot API? Below I give a compact plan for both: a production-ready API design plus a practical Node.js implementation (Playwright) that does full-page screenshots and blocks ads.
- Quick summary / options
- Use an existing service: many screenshot APIs (e.g., Browserless, URLShot, PageShot, ApiFlash, ScreenshotAPI) support full-page screenshots; some offer ad blocking via request rules or custom headers, but most don’t provide robust, configurable adblocking out of the box.
- Build your own: use a headless browser (Playwright or Puppeteer). Advantages: full control of capture quality, adblocking, auth, cookies, JS wait strategies, caching, scaling.
- Key features to support in the API
- fullPage capture (stitch or native full-page)
- ad blocking via blocklists (EasyList/EasyPrivacy) + resource-type blocking
- device emulation (viewport, device scale, user agent)
- wait strategies: waitUntil (load/networkidle), explicit wait selector, timeout
- output options: PNG/JPEG/WebP, quality, full resolution vs scaled
- viewport & cropping options, include/exclude screenshots of fixed-position elements
- authentication (Basic, cookies, headers, OAuth)
- headers and cookies passthrough
- caching, CDN for images
- rate limiting, concurrency control, queueing, retry
- security: isolate pages, sandboxing, resource/time limits, URL allowlist
- API design (example endpoints & parameters)
POST /screenshot
Request JSON (or query params):
- url (required)
- format: png|jpg|webp (default png)
- fullPage: boolean (default true)
- width, height, deviceScale (defaults 1366x768, 1)
- userAgent
- waitUntil: load|domcontentloaded|networkidle
- waitForSelector: CSS selector to wait for (optional)
- timeout: ms
- adblock: true|false (default true)
- blockResourceTypes: ["image","media","font","stylesheet","script"] (optional)
- blockURLPatterns: array of glob/regex strings to block
- auth: {type, credentials}
- headers, cookies
- quality (for JPEG/WEBP)
Response:
- 200 OK with binary image (Content-Type)
- 202 Accepted for async jobs (with job id)
- metadata headers: X-Rendered-Height, X-Device-Scale, X-Timing-ms
Errors: 400, 401, 429, 500
- How ad blocking works (approaches)
- Simple approach: block by resource type (scripts/styles/images) or block by domain suffix (e.g., known ad domains), URL substring matching.
- Better approach: use EasyList/EasyPrivacy parsed into a matcher (e.g., adblockparser). Evaluate each request against the adblock rules and block matches.
- Best practice: combine resource-type blocking, adblock rules, and allowlist for critical third-party assets.
- Implementation example (Node.js + Playwright)
This is a simple serverless-friendly example showing full-page capture and blocking by EasyList using @cliqz/adblocker-playwright. It blocks ads with EasyList and captures fullPage.
Install:
- npm i playwright @cliqz/adblocker-playwright @cliqz/adblocker-rust express
Example code (Express + Playwright):
const express = require('express');
const { chromium } = require('playwright');
const { PlaywrightBlocker } = require('@cliqz/adblocker-playwright');
const { fetch } = require('cross-fetch'); // required by adblocker
const app = express();
app.use(express.json());
let browser;
(async () => {
browser = await chromium.launch({ args: ['--no-sandbox'] });
})();
app.post('/screenshot', async (req, res) => {
const {
url,
fullPage = true,
width = 1366,
height = 768,
deviceScale = 1,
userAgent,
waitUntil = 'networkidle',
timeout = 30000,
adblock = true,
quality
} = req.body || {};
if (!url) return res.status(400).send('url is required');
const context = await browser.newContext({
viewport: { width, height },
deviceScaleFactor: deviceScale,
userAgent
});
const page = await context.newPage();
// Setup adblocker if requested
if (adblock) {
// Create blocker; it's lightweight to create per-page
const blocker = await PlaywrightBlocker.fromLists(fetch, [
'https://easylist.to/easylist/easylist.txt',
'https://easylist.to/easylist/easyprivacy.txt'
]);
blocker.enableBlockingInPage(page);
}
try {
// Optional: block resource types up-front (example)
await page.route('**/*', (route) => {
const req = route.request();
const resource = req.resourceType();
// Example: allow everything, letting adblocker handle filtering.
route.continue();
});
// Navigate
await page.goto(url, { waitUntil: waitUntil, timeout });
// Optionally wait for some selector to appear for SPA content
// await page.waitForSelector('#main', { timeout: 5000 }).catch(() => {});
// Capture full page
const screenshotBuffer = await page.screenshot({
type: quality ? 'jpeg' : 'png',
quality: quality, // undefined for png
fullPage
});
res.setHeader('Content-Type', quality ? 'image/jpeg' : 'image/png');
res.send(screenshotBuffer);
} catch (err) {
console.error('screenshot error', err);
res.status(500).send('capture failed');
} finally {
await page.close();
await context.close();
}
});
const port = process.env.PORT || 3000;
app.listen(port, () => console.log('listening', port));
Notes about the example:
- @cliqz/adblocker-playwright + adblocker-rust is efficient and uses EasyList/EasyPrivacy; fetching lists at runtime can be cached.
- For heavy load, reuse a single Playwright browser and create contexts/pages per request (as shown). Limit concurrency to avoid OOM.
- The blocking library is stateful; consider caching a blocker instance and updating lists periodically.
- Full-page capture gotchas and remedies
- Native fullPage option: Playwright/Puppeteer can capture full page by setting fullPage: true. This typically works by resizing the viewport to page height and capturing a single image. It’s simpler and accurate.
- Stitching: if using browsers that don’t support native fullPage, you can scroll and stitch. Stitching can be slow and tricky with fixed elements.
- Fixed headers/footers: they may be repeated in stitched screenshots; either hide fixed elements (via injected CSS) or scroll them out of view.
- Lazy loading: scroll the page slowly (or use IntersectionObserver triggering) before capture, or wait for networkidle and a short delay.
- Fonts and cross-origin resources: some third-party fonts/styles blocked might change layout — consider a whitelist for essential resources.
- Scaling & infra considerations
- Pool of browser instances or single browser with many contexts (Playwright recommends reusing a browser).
- Limit concurrency per instance and scale horizontally with containerized workers.
- Use job queues (RabbitMQ/Redis) for spikes.
- Implement caching of screenshots based on URL+options to save cost.
- Rate limiting, request timeouts, and memory/time quotas on each capture.
- Logging, metrics (latency, success rate), and health checks.
- Security & privacy
- Sanitize the URL and restrict internal IPs (prevent SSRF).
- Run browsers in sandboxed containers; drop privileges.
- Apply per-job CPU/memory/time limits.
- Avoid storing cookies or sensitive headers unless necessary.
- Respect robots.txt and legal constraints if you will crawl other sites.
- Testing & monitoring
- Test across many sites including heavy ad networks and SPAs.
- Use synthetic tests for viewport, full-page height extremes, and non-200 pages.
- Monitor traffic, memory, and screenshot differences with visual regression tests.
If you want, I can:
- Provide a ready-to-run repository with Dockerfile and example worker scaling.
- Provide a production-ready blocker setup that caches EasyList and updates daily.
- Show a Puppeteer example (instead of Playwright).
- Suggest hosted services if you prefer not to self-host.
Which would you like next?