Feed Web Pages to AI Agents with ScreenshotAPI
Written By Hanzala Saleem
Updated At June 17, 2026 | 8 min read
Written By Hanzala Saleem
Updated At June 17, 2026 | 8 min read
AI agents are only as useful as the context you give them. You can have the most capable large language model in the world, but if it receives a raw HTML dump filled with <div> soup, inline scripts, and GDPR banners, it will produce garbage output. The web is the richest source of real-time information available, yet feeding it cleanly into an AI pipeline remains one of the most underestimated engineering challenges.
What this guide covers: LLM web scraping means fetching a live webpage through a real browser, executing its JavaScript, stripping the noise, and returning the content in a format your AI agent can actually use. ScreenshotAPI handles this in one API call, returning clean Markdown for RAG pipelines, plain text for summarization, post-execution HTML for DOM parsing, or a pixel-perfect screenshot for vision model workflows.
When an AI agent needs to understand a webpage, the naive approach is to fetch the raw HTML with requests.get() or curl. This works on static pages, but the modern web is not static.
Most high-value pages rely on JavaScript for rendering. Product prices, user-generated content, dynamic search results, and personalized dashboards are all injected into the DOM after the initial HTML response. A raw HTTP fetch misses all of it. The agent sees placeholder tags and empty containers instead of actual content.
Even when the content is there, raw HTML is extraordinarily noisy. A typical e-commerce product page can exceed 80,000 characters of raw HTML. Analysis of common product page structures typically finds fewer than 5% of that markup contains actual product information: the remaining bulk is navigation, tracking scripts, and consent logic. Feeding that directly into an LLM context window wastes tokens, degrades reasoning quality, and increases cost.
AI agents need pages that have been:
That is precisely the gap ScreenshotAPI fills.
ScreenshotAPI is a REST API that runs every request inside a real, isolated Chromium instance. The same rendering engine as Google Chrome. It executes JavaScript, triggers lazy-load events, waits for async data to settle, and then returns the captured result in your chosen format.
For AI workflows, the four most relevant output modes are:
1. Extract Text (extract_text=true) Returns a clean .txt file containing only the readable content of the page. All HTML tags, scripts, stylesheets, and tracking code are stripped. The result is exactly what a human sees when they read the page, delivered as a file URL you can fetch and pass directly into an LLM prompt.
2. Extract Markdown (extract_markdown=true) Returns a .md file that preserves semantic structure: headings, lists, links, and content hierarchy are all maintained. This is the ideal input format for RAG (Retrieval-Augmented Generation) pipelines, knowledge base ingestion, and any agent that needs to reason about document structure rather than just content.
3. Extract HTML (extract_html=true) Returns the post-execution DOM as a .html file. This captures the fully rendered page after JavaScript has run, meaning dynamic content is present. Useful for agents that need to parse specific HTML elements or extract structured data (prices, tables, metadata) using downstream parsers.
4. Visual Screenshot Returns the page as a PNG, JPG, WebP, or PDF. This powers vision model workflows where a multimodal model such as GPT-4o or Gemini 1.5 accepts an image URL directly in its API request, analyzing visual layout, chart data, or UI structure from the rendered image.
All four modes can be requested in a single API call. You get the screenshot and the extracted Markdown in one round trip.
The ScreenshotAPI endpoint for all capture and extraction operations is:
GET https://shot.screenshotapi.net/v3/screenshotThe following example fetches a competitor's pricing page and returns its Markdown content, which is then passed directly into an LLM prompt for summarization.
import requests
API_TOKEN = "YOUR_API_TOKEN"
TARGET_URL = "https://example.com/pricing"
# Build the API request
params = {
"token": API_TOKEN,
"url": TARGET_URL,
"output": "json", # Required for extraction features
"full_page": "true", # Capture the entire page, not just the viewport
"block_ads": "true", # Remove ads before extraction
"no_cookie_banners": "true", # Remove GDPR popups before extraction
"lazy_load": "true", # Trigger lazy-loaded content
"extract_markdown": "true", # Return structured Markdown
"extract_text": "true", # Also return plain text
}
response = requests.get(
"https://shot.screenshotapi.net/v3/screenshot",
params=params
)
data = response.json()
# The API returns file URLs for each extracted format
markdown_url = data.get("extract_markdown")
text_url = data.get("text")
screenshot_url = data.get("screenshot")
# Fetch the Markdown content
markdown_content = requests.get(markdown_url).text
# Feed into your LLM
print(f"Markdown length: {len(markdown_content)} characters")
print(markdown_content[:500]) # PreviewThe markdown_content variable now holds a clean, structured representation of the pricing page. Headings are preserved (##), pricing tiers appear as lists, and all irrelevant layout noise has been removed. An LLM can reliably extract plan names, prices, and feature comparisons from this input without hallucinating structure that was never there.
For vision-based agents, the screenshot_url points to a pixel-perfect PNG of the fully rendered page. Pass that URL directly into a multimodal model API call alongside a structured prompt.
# Vision agent example (pseudocode, works with OpenAI/Anthropic vision models)
vision_prompt = {
"role": "user",
"content": [
{
"type": "image_url",
"image_url": {"url": screenshot_url}
},
{
"type": "text",
"text": "Extract all pricing tiers and their included features from this screenshot."
}
]
}The table below shows how ScreenshotAPI slots into a standard agentic pipeline, from URL input to structured LLM output.
| Step | Component | Description |
|---|---|---|
| 1 | URL Input | User or orchestrator provides a target webpage URL |
| 2 | ScreenshotAPI | Renders the page in Chromium, applies ad and banner blocking, triggers lazy load |
| 3 | Extraction Layer | Returns text, Markdown, HTML, and/or screenshot based on parameters |
| 4 | Format Router | Agent selects the appropriate format: Markdown for RAG, image for vision tasks |
| 5 | LLM / Vision Model | Processes the clean input and generates structured output |
| 6 | Output Handler | Agent stores results, triggers actions, or returns a response to the user |
This pipeline works with any orchestration framework: LangChain, LlamaIndex, CrewAI, AutoGen, or a custom Python workflow. ScreenshotAPI handles the rendering and extraction layer, so the rest of the pipeline consumes clean data.
Retrieval-Augmented Generation systems depend on high-quality document chunks. When your knowledge base needs to include live web content (documentation sites, news articles, support pages), ScreenshotAPI's extract_markdown=true parameter delivers clean, pre-chunked content. Use the web scraping workflow to schedule daily ingestion of target URLs directly into your vector database.
Set up a scheduled capture of competitor pricing and feature pages using ScreenshotAPI's cron-based scheduling. Each run returns both the visual screenshot (for human review) and the extracted text (for automated change detection). An LLM can then diff the text outputs week-over-week and generate a structured change summary without any human involvement.
AI writing assistants and content audit agents need to analyze what top-ranking pages actually say. ScreenshotAPI renders the live page as seen by users, including dynamically loaded content sections that raw HTML fetchers miss. The extracted Markdown is then used to build content briefs, identify topic gaps, or train fine-tuned models on high-quality web copy.
Integrations with Zapier, Make, and n8n allow non-developers to wire ScreenshotAPI into event-driven workflows. A trigger (new product listing, form submission, CRM update) can automatically fire a capture, extract the text, and pass it to a connected AI tool for classification or summarization. No code required.
Autonomous research agents that browse the web to gather information face the same rendering problem as every other scraper. By routing page fetches through ScreenshotAPI, the agent receives clean, LLM-ready content for every URL it visits. This is particularly valuable for pages that require JavaScript execution, authentication, or specific viewport rendering to display their content.
Multimodal agents that need to understand page layout, verify UI states, or analyze chart data benefit from ScreenshotAPI's high-quality visual output. The retina=true parameter captures at 2x pixel density, which improves the accuracy of vision model interpretations on text-heavy or data-dense pages.
| Use Case | Recommended Parameter | Output Format | LLM Token Efficiency |
|---|---|---|---|
| Summarization, Q&A | extract_markdown=true | .md file | High (structured, clean) |
| Full-text search indexing | extract_text=true | .txt file | High (no markup overhead) |
| DOM parsing, data extraction | extract_html=true | .html file | Medium (needs parsing) |
| Visual layout analysis | Screenshot (PNG/WebP) | Image URL | Requires vision model |
| Compliance archiving | output=pdf | PDF file | Not applicable |
For most LLM-based agents, extract_markdown=true combined with block_ads=true and no_cookie_banners=true is the most effective combination. It removes noise and preserves the content hierarchy the model needs to reason accurately.
ScreenshotAPI offers a free plan with 100 screenshots and no credit card required. To start feeding web pages into your AI agent:
Web scraping fetches the raw HTML response the server sends before JavaScript runs. Rendering loads the page in a real browser, executes all scripts, and captures the final DOM state. For AI agents, rendering is almost always required. Most pages with valuable content (prices, articles, dynamic search results) inject that content via JavaScript after the initial HTML response. ScreenshotAPI runs a full Chromium engine for every request, so agents receive the fully rendered page, not a shell of empty containers.
Yes. Add extract_markdown=true and output=json to your API request. ScreenshotAPI returns a URL pointing to a .md file containing the structured content of the rendered page. The Markdown file is ready to pass directly into an LLM prompt, a RAG chunk ingestion pipeline, or a vector database loader without additional parsing.
Use extract_markdown=true combined with block_ads=true and no_cookie_banners=true. Markdown preserves heading hierarchy and list structure, which improves chunk quality and retrieval relevance. Plain text (extract_text=true) works for summarization tasks where structure is not needed. Post-execution HTML (extract_html=true) is best when your pipeline includes a downstream DOM parser that needs specific elements.
Make an API call to ScreenshotAPI with your target URL and set file_type=png or file_type=webp. The JSON response includes a screenshot field containing a public image URL. Pass that URL as an image_url content block in your vision model's API request (GPT-4o, Gemini 1.5, Claude). The model receives a pixel-perfect render of the page as a real browser displays it. For text-dense pages, set retina=true to capture at 2x pixel density.
Yes. The infrastructure scales without configuration changes. The batch endpoint accepts CSV or JSON lists of URLs and processes them in parallel. Combined with direct-to-S3 or Google Cloud Storage output via the byob=true parameter, pipelines can capture, extract, and store thousands of pages without infrastructure management.
