urllib2 doesn't use proxy (fieddler2), configed the proxy with ProxyHandler

@pfmoore

BPO	12480
Nosy	@pfmoore, @orsenthil, @tjguk, @zware, @zooba, @iritkatriel

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/orsenthil'
closed_at = None
created_at = <Date 2011-07-03.15:16:12.209>
labels = ['type-bug', '3.9', '3.10', '3.11', 'library', 'OS-windows']
title = "urllib2 doesn't use proxy (fieddler2), configed the proxy with ProxyHandler"
updated_at = <Date 2021-05-28.14:09:16.684>
user = 'https://bugs.python.org/OrWilder'

bugs.python.org fields:

activity = <Date 2021-05-28.14:09:16.684>
actor = 'iritkatriel'
assignee = 'orsenthil'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)', 'Windows']
creation = <Date 2011-07-03.15:16:12.209>
creator = 'Or.Wilder'
dependencies = []
files = []
hgrepos = []
issue_num = 12480
keywords = []
message_count = 6.0
messages = ['139688', '139689', '139696', '141157', '223154', '394660']
nosy_count = 9.0
nosy_names = ['paul.moore', 'orsenthil', 'tim.golden', 'thomas.holmes', 'Or.Wilder', 'pbumbulis', 'zach.ware', 'steve.dower', 'iritkatriel']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue12480'
versions = ['Python 3.9', 'Python 3.10', 'Python 3.11']

Linked PRs

I have fieddler2 listening on 0.0.0.0:8888.

this code is suppose to use the proxy on my localhost.

try:
data = ''
proxy = urllib2.ProxyHandler({'http': '127.0.0.1:8888'}) //also tried {'http': 'http://127.0.0.1:8888/'}
opener = urllib2.build_opener(proxy)
urllib2.install_opener(opener)
req = urllib2.Request('http://www.google.com')
response = urllib2.urlopen(req)
the_page = response.read()
print the_page
except Exception, detail:
print "Err ", detail

I have fieddler2 listening on 0.0.0.0:8888.

try:
proxy = urllib2.ProxyHandler({'http': '127.0.0.1:8888'}) //also tried {'http': 'http://127.0.0.1:8888/'}
opener = urllib2.build_opener(proxy)
urllib2.install_opener(opener)
req = urllib2.Request('http://www.google.com')
response = urllib2.urlopen(req)
the_page = response.read()
print the_page
except Exception, detail:
print "Err ", detail

I don't see the GET or any request to google in fieddler (but I can see other requests) is there a way to debug it? is seems like python bypasses fieddler or ignores the proxy.

the code is working, but it's not using the proxy to fetch google, it bypasses the proxy.

I also configured win7 to work with fieddler -

C:\Windows\system32>netsh winhttp set proxy 127.0.0.1:8888

Current WinHTTP proxy settings:

Proxy Server(s) :  127.0.0.1:8888
Bypass List     :  (none)

In order to check if the code is using the porxy I've changed fieddler to require proxy authentication and I had to use a user\pass on my browser when browsing, run the python code again and it worked, so it defiantly doesn't use the proxy.

I have fieddler2 listening on 0.0.0.0:8888.

this code is suppose to use the proxy on my localhost.

try:
proxy = urllib2.ProxyHandler({'http': '127.0.0.1:8888'}) //also tried {'http': 'http://127.0.0.1:8888/'}
opener = urllib2.build_opener(proxy)
urllib2.install_opener(opener)
req = urllib2.Request('http://www.google.com')
response = urllib2.urlopen(req)
the_page = response.read()
print the_page
except Exception, detail:
print "Err ", detail

I don't see the GET or any request to google in fieddler (but I can see other requests) is there a way to debug it? is seems like python bypasses fieddler or ignores the proxy.

the code is working, but it's not using the proxy to fetch google, it bypasses the proxy.

I also configured win7 to work with fieddler -

C:\Windows\system32>netsh winhttp set proxy 127.0.0.1:8888

Current WinHTTP proxy settings:

Proxy Server(s) :  127.0.0.1:8888
Bypass List     :  (none)

In order to check if the code is using the porxy I've changed fieddler to require proxy authentication and I had to use a user\pass on my browser when browsing, run the python code again and it worked, so it defiantly doesn't use the proxy.

I setup a proxy and it seems to be working properly. This is on win7 x64 professional using Python 2.7.1

I didn't using any sniffing software but if I broke the proxy the code broke. When I enabled the proxy it started working again. My proxy log also shows record of the access.

proxy_bypass_registry in urllib.py does not handle the ProxyOverride registry value properly: it treats an empty override as *, i.e. bypass the proxy for all hosts. This behavior does not match other programs (e.g. Chrome) and can be easily obtained by specify * for the override. One fix would be to ignore empty tests, for example:

....
for test in proxyOverride:
    if test:
        if test == '<local>':
        ...
return 0
....

Perhaps whitespace should be stripped as well.

The problem arises because fiddler2 leaves a trailing ; on the ProxyOverride string.

One possible workaround is to set
urllib.proxy_bypass = lambda h: 0
to disable bypass checking. Another alternative would be to specify the proxy settings in the http_proxy environment variable (proxy_bypass_registry is not called in this case).

@senthil

@senthil what if anything can we do with this issue?

I think the problem is still there:

https://github.com/python/cpython/blame/bb3e0c240bc60fe08d332ff5955d54197f79751c/Lib/urllib/request.py#L2746

if proxyOverride ends with a ';' then the last "" is considered a match for any val in host.

This was fixed in 2024 as a side-effect GH-115210

commit c43b26d02eaa103756c250e8d36829d388c5f3be
Author: Weii Wang <weii.wang@canonical.com>
Date:   Thu Feb 29 04:15:52 2024 +0800

    gh-115197: Stop resolving host in urllib.request proxy bypass (GH-115210)
    
    Use of a proxy is intended to defer DNS for the hosts to the proxy itself, rather than a potential for information leak of the host doing DNS resolution itself for any reason.  Proxy bypass lists are strictly name based.  Most implementations of proxy support agree.

The code changes how the proxy parsing works, and uses fnmatch instead of regex and thus handles the trailing ; case for fiddler properly.

+# Same as _proxy_bypass_macosx_sysconf, testable on all platforms
+def _proxy_bypass_winreg_override(host, override):
+    """Return True if the host should bypass the proxy server.
+
+    The proxy override list is obtained from the Windows
+    Internet settings proxy override registry value.
+
+    An example of a proxy override value is:
+    "www.example.com;*.example.net; 192.168.0.1"
+    """
+    from fnmatch import fnmatch
+
+    host, _ = _splitport(host)
+    proxy_override = override.split(';')
+    for test in proxy_override:
+        test = test.strip()
+        # "<local>" should bypass the proxy server for all intranet addresses
+        if test == '<local>':
+            if '.' not in host:
+                return True
+        elif fnmatch(host, test):
+            return True
+    return False
+
+

An additional test case to assert the fix for this bug could be beneficial and I will add the test case.

#160841 for the test case coverage.

Closing this as the issue is already fixed.

gh-56689 test case to cover for windows proxy_bypass misbehaving trai…

[3.15] gh-56689 test case to cover for windows proxy_bypass misbehavi…

[3.14] gh-56689 test case to cover for windows proxy_bypass misbehavi…

Navigation Menu

urllib2 doesn't use proxy (fieddler2), configed the proxy with ProxyHandler #56689

Linked PRs

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

urllib2 doesn't use proxy (fieddler2), configed the proxy with ProxyHandler #56689

Description

Linked PRs

Activity

OrWilder commented on Jul 3, 2011

OrWilder commented on Jul 3, 2011

thomasholmes commented on Jul 3, 2011

pbumbulis commented on Jul 26, 2011

BreamoreBoy commented on Jul 15, 2014

iritkatriel commented on May 28, 2021

orsenthil commented on Oct 28, 2026

orsenthil commented on Oct 28, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions