Integrating Zenrows into smolagents for Production Web Access

A production-ready web retrieval setup for Hugging Face smolagents that replaces VisitWebpageTool with a custom Zenrows fetch_page tool for retrieving webpage content as Markdown from JavaScript-rendered and protected sites.

Features

• Replaces smolagents' VisitWebpageTool with a custom Zenrows fetch tool
• Retrieves webpage content as clean Markdown
• Handles JavaScript-rendered and protected webpages
• Registers the custom fetch_page tool with a smolagents CodeAgent
• Uses Qwen through Hugging Face Inference Providers to power the CodeAgent

Prerequisites

Before you start, make sure you have:

• Python 3.10 or higher
• A Zenrows account and API key
• A Hugging Face account and User Access Token
• Git

Installation

1. Clone the repository

git clone https://github.com/Techwithpraisejames/smolagents-zenrows-tutorial.git
cd smolagents-zenrows-tutorial

2. Create a virtual environment

python -m venv .venv

3. Activate the virtual environment

macOS/Linux

source .venv/bin/activate

Windows

.venv\Scripts\activate

4. Install dependencies

pip install -r requirements.txt

Configuration

Create a .env file in the project root and add your API credentials:

ZENROWS_API_KEY=your_zenrows_api_key
HF_TOKEN=your_hugging_face_token

The ZENROWS_API_KEY is used to retrieve webpage content through the Zenrows Fetch API. The HF_TOKEN is used by InferenceClientModel to access Qwen through Hugging Face Inference Providers.

Do not commit your .env file to version control.

Project structure

.
├── .gitignore
├── README.md
├── agent.py
├── requirements.txt
└── zenrows_tool.py

• agent.py initializes the Qwen model, registers the fetch_page tool with the CodeAgent, and runs the web research task.
• zenrows_tool.py defines the custom fetch_page tool.
• requirements.txt contains the Python dependencies required to run the project.
• .gitignore specifies files and directories that Git should exclude from version control.
• README.md contains the project documentation.

How it works

The project replaces smolagents' default VisitWebpageTool with a custom Zenrows fetch_page tool.

User request
     ↓
CodeAgent
     ↓
fetch_page tool
     ↓
Zenrows Fetch API
     ↓
JavaScript-rendered or protected webpage
     ↓
Markdown content
     ↓
CodeAgent
     ↓
Structured response

The fetch_page function accepts a URL and sends it to the Zenrows Fetch API. The tool returns the webpage content as Markdown, which the CodeAgent can use for research, summarization, and analysis.

The @tool decorator makes fetch_page available to the CodeAgent. The agent uses Qwen through Hugging Face Inference Providers to plan the task and calls fetch_page when it needs webpage content.

Running the project

Test the Zenrows fetch tool

Run the fetch tool directly:

python zenrows_tool.py

This tests the tool and prints a snippet of the returned Markdown content.

Run the CodeAgent

Run the research agent:

python agent.py

The agent retrieves the specified TechCrunch article through fetch_page, receives the webpage content as Markdown, and uses that content to complete the research task.

Output

Running zenrows_tool.py returns webpage content as Markdown.

Running agent.py returns a structured analysis of the retrieved TechCrunch article, including:

• The company involved
• The main announcement
• The news category
• Why the announcement matters

The CodeAgent uses the content returned by Zenrows to complete the research task.

Technologies

• Python
• smolagents
• Zenrows
• Hugging Face Inference Providers
• Qwen2.5-7B-Instruct
• Requests
• python-dotenv

Related article

This project is part of this blog:

Integrating Zenrows into smolagents for Production Web Access

[BLOG_URL]