-
Notifications
You must be signed in to change notification settings - Fork 0
Expand file tree
/
Copy pathscraper_errors.py
More file actions
35 lines (23 loc) · 1.2 KB
/
Copy pathscraper_errors.py
File metadata and controls
35 lines (23 loc) · 1.2 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
"""Shared failure taxonomy for every scraper package.
Each scraper's transport (`<site>/fetch.py`) raises the same four kinds of
error; subclassing these bases lets ONE route layer (route_glue.py) map
them to HTTP for every site:
UpstreamError transport failure / 5xx -> 502, retryable
Blocked anti-bot challenge / 403 / 429 -> 502, retryable on a new exit/identity
BadRequest upstream rejected the params -> 400, never retried
NotFound entity / page does not exist -> 404, never retried
A site keeps its own names for readability and for `except` clauses inside
its package:
class AlibabaUpstreamError(UpstreamError): ...
class AlibabaBlocked(AlibabaUpstreamError, Blocked): ...
class AlibabaBadRequest(BadRequest): ...
class AlibabaNotFound(NotFound): ...
"""
class UpstreamError(Exception):
"""Transport failure or 5xx — retryable."""
class Blocked(UpstreamError):
"""Anti-bot challenge / 403 / 429 — retryable after a fresh exit or identity."""
class BadRequest(Exception):
"""Upstream rejected the params — never retried."""
class NotFound(Exception):
"""Entity / page does not exist — never retried."""