Use with an http proxy?

[rdubya]

We currently use this gem in an environment where we have outgoing http requests blocked unless they go through a proxy. Does this gem support proxies? It looks like it probably doesn't since it is handling things at the tcp level, but am hoping that maybe I missed something. Would it be possible to make it work with an http proxy?

[ioquatix]

Can you explain what you mean by work with proxies? You mean, work with HTTP proxies using CONNECT?

[rdubya]

Thanks for the quick response.

I don't know enough about lower level connections to be able to know the answer to that question unfortunately. In other gems that we use, we can set proxy information. For example, we use this for HTTParty: http_proxy(Settings.http_proxy.address, Settings.http_proxy.port). In the HTTPClient gem you create the new client with the proxy setting:

  # HTTPClient.new takes optional arguments as a Hash.
  #  * :proxy - proxy url string
  #  * :agent_name - User-Agent String
  #  * :from - from header String
  #  * :base_url - base URL of resources
  #  * :default_header - header Hash all HTTP requests should have
  #  * :force_basic_auth - flag for sending Authorization header w/o gettin 401 first
  # User-Agent and From are embedded in HTTP request Header if given.
  # From header is not set without setting it explicitly.

I haven't found a way to do something similar with this gem.
Unfortunately, the cloudflare gem uses this gem so it means we can't use the cloudflare gem unless we can figure out a way to get these to go through the proxy.

[ioquatix]

Ah I see. I am happy to implement this feature.

In order to prioritise this work, would you be wiling to sponsor the project to support this development? I'm actually still working out all the details of this sponsorship process, but let me know if this is something you be willing to explore.

[rdubya]

I can run it by my boss. Our alternative at this point is to build a micro-service outside our locked environment that wraps this gem and send all our calls through that. We'd need the feature in the next week or so to be able to avoid doing that.

[ioquatix]

Based on my current schedule, I could deliver this by the 4th of August.

[ioquatix]

@rdubya I have a working implementation but I am just sorting out some bugs. Hopefully I can give you access to an experimental branch today.

[ioquatix]

The way it works is also completely generic - I know that's probably less important to you, but basically:

# The proxy HTTP server:
client = Async::HTTP::Client.new(Async::HTTP::Endpoint.parse("https://proxy.local"))
# Represents an endpoint that can connect using TCP to the given host/port using CONNECT:
proxy = Async::HTTP::Proxy.new(client, "www.google.com:443")
# An HTTP endpoint that will connect via the proxy:
endpoint = proxy.endpoint("https://www.google.com")

# An HTTP client that will connect to the given endpoint via the proxy:
proxy_client = Async::HTTP::Client.new(endpoint)

# Make a normal request, it will go through the proxy:
response = proxy_client.get("/search?q=kittens")
puts response.read

[ioquatix]

(Of course the final interface would be much simpler).

[rdubya]

Cool, sounds good. Will we also need to make changes to the cloudflare gem to support it or will it be done in a way that we can specify the proxy server's info and this gem will be able to pick it up without any changes to the cloudflare gem?

[ioquatix]

It should be possible to use these changes without changing the cloud flare gem, but we can check that.

[ioquatix]

There was a bug in the proxy implementation, but I found it. It took a few days. Hopefully have something early this week.

[ioquatix]

@rdubya Can you please tell me what your proxy server software is?

[rdubya]

Glad to see you got through it. We use squid as our proxy.

[ioquatix]

Okay, all specs are passing:

Async::HTTP::Protocol::HTTP10
  behaves like Async::HTTP::Proxy
    echo server
      can connect to remote system using block
      can connect to remote system
    proxied client
      can get website

Async::HTTP::Protocol::HTTP11
  behaves like Async::HTTP::Proxy
    echo server
      can connect to remote system using block
      can connect to remote system
    proxied client
      can get website

Async::HTTP::Protocol::HTTP2
  behaves like Async::HTTP::Proxy
    echo server
      can connect to remote system using block
      can connect to remote system
    proxied client
      can get website

Finished in 1.7 seconds (files took 0.36588 seconds to load)
9 examples, 0 failures

This is a test that covers client/server HTTP proxy for HTTP/1.0, HTTP/1.1 and HTTP/2.0

I'm assuming squid only supports HTTP/1?

I'd like to test it on travis with a real proxy.

[ioquatix]

Also I should test Cloudflare integration. It should be straight forward.