Stability of the API #272

Activity

1. jborean93

   changed the title [-]Stability of API[/-] [+]Stability of the API[/+] on Feb 22, 2021
2. florimondmanca commented on Feb 22, 2021

   

   florimondmanca

   on Feb 22, 2021

   Contributor

   More actions

   I think you're referring to the discussion we had here? :) encode/httpx#1473

   To be clear, right now the only thing that's public in HTTPCore is what is documented on the docs site. The internals of the connection pool or proxy implementations are private, and subject to change. Doesn't mean you can tweak them, but just be sure to pin HTTPCore strictly and expect to have to update them slightly as we fix bugs or tweak things internally.

   I don't think we're ready to make those internals public in full, however I'm personally happy to discuss specific issues and things we could expose selectively to make it easier to customize connections.
3. florimondmanca

   added

   questionFurther information is requestedFurther information is requested

   on Feb 22, 2021
4. jborean93 commented on Feb 22, 2021

   

   jborean93

   on Feb 22, 2021

   Last edited by jborean93

   Author

   More actions

   Yea, unfortunately I keep on hitting roadblocks as I progress further :(. Ultimately the authentication process and subsequent messages are tied directly with the connection which just isn't publicly exposed in httpx/httpcore. As you said at this point I'm effectively pinning the httpcore version to one where I know works. What I'm ultimately hoping to achieve is to expose the functionality I need as a public interface so I don't have to be so selective with my pinning at some point in the future.

   From what I've worked with I've found that making a custom transport solves a lot of my existing issues but there remains 2 problems:

   • Not being able to detect when a connection is dropped from a keep-alive expiry or server side disconnect
     • AsyncConnectionPool handles all the pool logic internally as well as building AsyncHTTPConnection through the private imports but the keep-alive/disconnect issue still remains
     • I can do the same thing myself but it means I need to import AsyncHTTPConnection through the private imports hence this issue
     • There's AsyncConnectionPool._create_connection() but that's also a private function so has the same issue with creating the connection object myself
   • For the authentication protocols I need to use I need to include TLS connection specific data when generating the auth token
     • Currently the socket and TLS handshake is automatically done on the first AsyncHTTPConnection.arequest()
     • This is unfortunately too late as the TLS connection is closed on the first failed request, the next request with the auth token will be on a new connection/TLS context so the data may differ
     • The only real way to solve this is to be able to create the socket and do the TLS handshake before we send the first HTTP request
     • See Expose socket connection interface in connection #273

   I could potentially copy AsyncHTTPConnection and use it internally but there's a lot of things that httpcore just handles that I'm hoping not to duplicate code.
5. florimondmanca commented on Feb 22, 2021

   

   florimondmanca

   on Feb 22, 2021

   Last edited by florimondmanca

   Contributor

   More actions

   @jborean93 From the specificity of your use case, and especially what seems to be "I need a long-lived special TLS connection that shouldn't close upon errors", are you sure you need our connection pooling behavior? Were you able to build a prototype that solely uses an AsyncHTTPConnection as a transport (or perhaps a wrapper around a custimized version of it)?

   The code for our HTTP/1.1 connection implementation isn't that big. It might be a very valid thing to do to duplicate it and tweak it to your liking. It may take some time to study and understand how it works specifically (reviewing the h11 docs helps), but it's not intractable. :-) And you'd be able to apply tweaks as you see fit, and remove bits you don't need.

   Edit: actually, since AsyncHTTP11Connection takes a socket as input, you might be able to reuse it, and instead wrap it in a transport that does the "open socket" behavior you need. Note that you'd need to match it up to return an AsyncSocket, which is a private interface. If you duplicate AsyncHTTP11Connection to use your own socket type, then you're completely free wrt HTTPCore evolutions, and don't need any pinning. Or you can use HTTPCore's concurrency backends, but you'd need to pin. You can get a sense of what this might look like in this example back from when we were working on UDS support. :-)
6. jborean93 commented on Feb 22, 2021

   

   jborean93

   on Feb 22, 2021

   Last edited by jborean93

   Author

   More actions

   Were you able to build a prototype that solely uses an AsyncHTTPConnection as a transport (or perhaps a wrapper around a custimized version of it)?

   I was with https://github.com/jborean93/pypsrp/blob/dfae26ed0fb6f6a27f8f7bc8ed121120bee75b3c/psrp/io/wsman.py#L40. Specially I create the connection and connect to the socket/TLS handshake with https://github.com/jborean93/pypsrp/blob/dfae26ed0fb6f6a27f8f7bc8ed121120bee75b3c/psrp/io/wsman.py#L211-L224. From there I check to see if it's still alive after a keep-alive expiry or whether the socket is readable (closed by server) and reconnect/reauth if needed.

   It might be a very valid thing to do to duplicate it and tweak it to your liking.

   That is true, I'm just hoping to avoid duplication of work as much as I can but it's probably the only sane thing that I can do without pinning to a specific version. Although as you said I need to copy more than just AsynHTTP11Connection, I also need to copy the backends, or at least parts of it to implement some of the functionality AsyncHTTP11Connection relies on. But it's probably for the best that I just copy and work in the background to expose what I need publicly for a future version.

   Heres to hoping I can still use proxies/socks in some way when using my own socket, sounds like that will require even more logic :(
7. lovelydinosaur

   closed this as completedon Mar 23, 2021
8. encode

   locked and limited conversation to collaborators on Mar 23, 2021