Skip to content

About

Haystack component for Proxmint residential, datacenter and mobile proxies: fetch pages as Documents with country, city and ASN targeting and sticky sessions

Topics

Resources

Stars

0 stars

Watchers

0 watching

Forks

Repository files navigation

proxmint-haystack

Haystack integration for Proxmint proxies: fetch web pages from residential, datacenter or 4G/5G mobile IPs and get them back as Documents. Pick the exit country, state, city, ZIP or ASN, and keep one IP for up to two hours when a site needs it.

  • ProxmintFetcher: a component that takes a list of URLs and returns documents (readable page text, or the raw HTML) and failed_urls. Use it as the first step of an indexing pipeline, or wrap it with ComponentTool to give an agent a "read this page from Germany" tool.

Requests go straight from your machine to the Proxmint gateway (gw.proxmint.com). Nothing else sits in between, and bandwidth comes out of your own Proxmint account.

Install

pip install proxmint-haystack

Credentials

Create an API key under Settings in the Proxmint dashboard and export it:

export PROXMINT_API_KEY="pmk_..."

The key reads the proxy login for the product you choose (residential by default). You can use the proxy login instead, with PROXMINT_USERNAME and PROXMINT_PASSWORD.

Fetch pages

from haystack_integrations.components.fetchers.proxmint import ProxmintFetcher

fetcher = ProxmintFetcher(country="gb")
result = fetcher.run(urls=["https://example.com", "https://example.org"])

print(result["documents"][0].content)
print(result["documents"][0].meta)  # source, requested_url, status, title, content_type, country
print(result["failed_urls"])

Use it in a pipeline

from haystack import Pipeline
from haystack.components.preprocessors import DocumentSplitter
from haystack.components.writers import DocumentWriter
from haystack.document_stores.in_memory import InMemoryDocumentStore
from haystack_integrations.components.fetchers.proxmint import ProxmintFetcher

store = InMemoryDocumentStore()
pipe = Pipeline()
pipe.add_component("fetcher", ProxmintFetcher(country="us", city="New York", session_minutes=30))
pipe.add_component("splitter", DocumentSplitter(split_by="sentence", split_length=5))
pipe.add_component("writer", DocumentWriter(store))
pipe.connect("fetcher.documents", "splitter.documents")
pipe.connect("splitter.documents", "writer.documents")

pipe.run({"fetcher": {"urls": ["https://example.com/pricing", "https://example.org/pricing"]}})

Use it as an agent tool

from haystack.tools import ComponentTool
from haystack_integrations.components.fetchers.proxmint import ProxmintFetcher

tool = ComponentTool(
    component=ProxmintFetcher(),
    name="fetch_pages",
    description="Fetch web pages through a proxy in a chosen country and return their text.",
)

Pass the tool to an Agent or a chat generator as usual. The model can set urls and an optional country per call.

Options

Option Default What it does
api_key PROXMINT_API_KEY A Haystack Secret
product "residential" residential, premium-residential, datacenter or mobile
country any Exit country (two-letter code). run(urls, country=...) overrides it for one run
state, city, zip_code, asn none Finer targeting; needs country
session_minutes none Keep one exit IP across runs for 1 to 120 minutes. Default: a new IP for every URL
output "text" "text" for readable text, "html" for the raw page
timeout 60 Seconds per request
raise_on_failure False Raise on a failed URL instead of listing it in failed_urls

A gateway refusal (no bandwidth left, no IP matching the targeting, a blocked site) names what to change.

Good to know

  • Web traffic works on destination ports 80, 443, 8080, 8443, 8888 and 8090 (port access).
  • A short list of sites, mostly banking and government, is refused by the network (blocked websites).
  • City, state, ZIP and ASN targeting meter at 2x on residential, datacenter and mobile (targeting).
  • Each product allows up to 2,000 concurrent connections.

Full guide: proxmint.com/docs/haystack. Pricing: proxmint.com/pricing. Bandwidth is billed per GB and never expires.

Development

uv run --group test pytest

The tests run offline against a stand-in for the gateway.

License

MIT

About

Haystack component for Proxmint residential, datacenter and mobile proxies: fetch pages as Documents with country, city and ASN targeting and sticky sessions

Topics

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages