CircuitPython clients can't complete requests to HTTPServer

[anecdata]

Trying to use adafruit_requests to connect either the 5100S-based WIZnet Pico EVB running Adafruit CircuitPython 7.2.5 on 2022-04-06; Raspberry Pi Pico with rp2040 or the Adafruit Feather RP2040 with Adafruit Ethernet FeatherWing running Adafruit CircuitPython 7.2.5 on 2022-04-06; Adafruit Feather RP2040 with rp2040 to an Adafruit Feather ESP32-S2 TFT running HTTPServer on an IPv4 (this repo simpletest example; no mDNS/hostname/FQDN involved) results in one of the following exception traces in all cases:

Either (less often):

Traceback (most recent call last):
  File "code.py", line 211, in <module>
  File "adafruit_requests.py", line 815, in get
  File "adafruit_requests.py", line 685, in request
OutOfRetries: Repeated socket failures

Or (more often):

Traceback (most recent call last):
  File "code.py", line 211, in <module>
  File "adafruit_requests.py", line 815, in get
  File "adafruit_requests.py", line 661, in request
  File "adafruit_requests.py", line 529, in _get_socket
  File "adafruit_wiznet5k/adafruit_wiznet5k_socket.py", line 251, in connect
  File "adafruit_wiznet5k/adafruit_wiznet5k.py", line 574, in socket_connect
RuntimeError: Failed to establish connection.

I was initially going to file this issue in WIZnet, but a sanity check of trying to connect to the HTTPServer from ESP32-S2 (e.g., Adafruit Feather ESP32-S2 TFT) also gets an exception every time, after about a minute. That surprised me, I may be doing something wrong. Maybe it's a Requests issue.

ESP32-S2 Client Code:

import traceback
import wifi
import socketpool
import ssl
import adafruit_requests
from adafruit_httpserver import HTTPServer, HTTPResponse
from secrets import secrets

wifi.radio.connect(secrets['ssid'], secrets['password'])
pool = socketpool.SocketPool(wifi.radio)
requests = adafruit_requests.Session(pool, ssl.create_default_context())

URLS = [
    "http://wifitest.adafruit.com/testwifi/index.html",
    "http://192.168.5.32",   # LAN Apache server
    "http://192.168.6.164",  # LAN ESP32-S2 with adafruit_httpserver
]

for url in URLS:
    try:
        print(url)
        with requests.get(url) as response:
            print(response.status_code, response.reason)
    except Exception as ex:
        traceback.print_exception(ex, ex, ex.__traceback__)

Output:

code.py output:
http://wifitest.adafruit.com/testwifi/index.html
200 bytearray(b'OK')
http://192.168.5.32
200 bytearray(b'OK')
http://192.168.6.164
Traceback (most recent call last):
  File "code.py", line 22, in <module>
  File "adafruit_requests.py", line 720, in get
  File "adafruit_requests.py", line 661, in request
  File "adafruit_requests.py", line 512, in _get_socket
RuntimeError: Sending request failed

Both Espressif client and Espressif server are running:
Adafruit CircuitPython 7.2.5 on 2022-04-06; Adafruit Feather ESP32-S2 TFT with ESP32S2

Connecting to the HTTPServer from a browser or curl works fine.

Connecting to local Apache server at an IPv4 from any of these clients works fine.

[anecdata]

Similar behavior with ESP32SPI (Adafruit CircuitPython 7.2.0 on 2022-02-24; Adafruit PyPortal with samd51j20):

import time
import board
import busio
import traceback
from digitalio import DigitalInOut
import adafruit_requests as requests
import adafruit_esp32spi.adafruit_esp32spi_socket as socket
from adafruit_esp32spi import adafruit_esp32spi
from secrets import secrets

URLS = [
    "http://wifitest.adafruit.com/testwifi/index.html",
    "http://192.168.5.32",   # LAN Apache server
    "http://192.168.6.164",  # LAN ESP32-S2 with adafruit_httpserver
]

esp32_cs = DigitalInOut(board.ESP_CS)
esp32_ready = DigitalInOut(board.ESP_BUSY)
esp32_reset = DigitalInOut(board.ESP_RESET)

spi = board.SPI()
esp = adafruit_esp32spi.ESP_SPIcontrol(spi, esp32_cs, esp32_ready, esp32_reset)
requests.set_socket(socket, esp)
print("Firmware vers.", esp.firmware_version)

esp.connect_AP(secrets["ssid"], secrets["password"])

print(f"ping 192.168.6.164: {esp.ping("192.168.6.164")}ms")

for url in URLS:
    try:
        print(url)
        with requests.get(url) as response:
            print(response.status_code, response.reason)
            # print(response.headers)
            # print(response.content)
    except Exception as ex:
        traceback.print_exception(ex, ex, ex.__traceback__)

Output:

code.py output:
Firmware vers. bytearray(b'1.7.4\x00')
ping 192.168.6.164: 120ms
ping 192.168.6.164: 240ms
ping 192.168.6.164: 240ms
http://wifitest.adafruit.com/testwifi/index.html
200 bytearray(b'OK')
http://192.168.5.32
200 bytearray(b'OK')
http://192.168.6.164
Traceback (most recent call last):
  File "code.py", line 30, in <module>
  File "adafruit_requests.py", line 815, in get
  File "adafruit_requests.py", line 685, in request
OutOfRetries: Repeated socket failures

It's looking like Requests. I thought maybe a fringe DNS or reachability issue, but ping seems to work. I updated all clients to addafruit_requests 1.11.2 and retested with same results.

[anecdata]

Went back and ping-tested on Espressif port pinging from an ESP32-S2 CircuitPython client. I'm not 100% confident of ping on Espressif port in general, but pinging the LAN Apache server every 5 seconds yields a reasonable number virtually every time. Pinging the HTTPServer IPv4 address gives a reasonable number about 25% of the time, and None the rest of the time. So it is possible there's some subtle reachability issue, but I have yet to see any successful HTTP GET requests to the HTTPServer IPv4 address from a CircuitPython device, where I would expect it to be at worst intermittent like ping. I keep double-checking with curl and browser and they're both finding the HTTPServer pretty reliably.

update: It is looking to be related to adafruit_requests (however, also seems to be something unique about adafruit_httpserver in the mix)... I set up a TCP socket client to fudge an HTTP request directly and it gets the full result from the HTTPServer IPv4 address most of the time.

Code:

import supervisor
import wifi
import socketpool
import ssl
import ipaddress
from secrets import secrets

HOST = "192.168.6.164"
PATH = "/"
PORT = 80
TIMEOUT = 15
MAXBUF = 4096

print("Connecting to wifi")
wifi.radio.connect(secrets["ssid"], secrets["password"])
pool = socketpool.SocketPool(wifi.radio)

print("Self IP", wifi.radio.ipv4_address)
server_ipv4 = ipaddress.ip_address(pool.getaddrinfo(HOST, PORT)[0][4][0])
print("Server ping", server_ipv4, wifi.radio.ping(server_ipv4), "ms")

print("Create TCP Client Socket")
s = pool.socket(pool.AF_INET, pool.SOCK_STREAM)
s.settimeout(TIMEOUT)

print("Connecting")
s.connect((HOST, PORT))

size = s.send(f"GET {PATH} HTTP/1.1\r\nHost: {HOST}:{PORT}\r\n\r\n".encode())
print("Sent", size, "bytes")

# just get the first hunk and call it a day
buf = bytearray(MAXBUF)
size = s.recv_into(buf)
print('Received', size, "bytes", buf[:size])
s.close()
print("Done-ish.")
supervisor.reload()

Example result:

code.py output:
Connecting to wifi
Self IP 192.168.6.161
Server ping 192.168.6.164 0.054 ms
Create TCP Client Socket
Connecting
Sent 42 bytes
Received 111 bytes bytearray(b'HTTP/1.1 200 OK\r\nContent-Type: text/plain\r\nContent-Length: 27\r\nConnection: close\r\n\r\n2022-04-17T23:31:25-05:00\r\n')
Done-ish.

Sorry this is all so horrendously long, it's been baffling. Maybe it's just me and there's something odd in my network environment.

[anecdata]

Further test to try to rule out issues with my LAN...

• HTTPServer on its own ESP32-S2 wifi AP
• Client ESP32-S2 connects as a Station to the HTTPServer wifi AP

In the case of running a HTTP / TCP server on an ESP32-S2 wifi AP, the AP IP address (192.168.4.1) must be used for the bind() (not sure why):
server.serve_forever("192.168.4.1")
...rather than something like: s.bind(("", 80)) which generally works in CircuitPython when the client device is a wifi Station, and in CPython ('' "is used to bind to all interfaces").

In this configuration the HTTP Request client gets the same result as above (RuntimeError: Sending request failed). The TCP client with TCP echo server work fine. The TCP client gets mixed replies (ETIMEDOUT, empty payload, or HTTP headers) from the HTTPServer, but that variance could possibly be attributed to it not being the full HTTP client the HTTPServer is expecting.

I think these AP results support leaning toward this being a Requests [+ HTTPServer] issue and not a local network issue.

(rewrote this comment for clarity after more testing)

[anecdata]

I think I have a clue to the issue. Requests chunks out the HTTP request into a bunch of tiny send commands:
https://github.com/adafruit/Adafruit_CircuitPython_Requests/blob/2e6b3f9feeacc678402454f7d3416b04a9a93e17/adafruit_requests.py#L580
As-is, the HTTPServer seems to get the GET chunk, then moves on and it's a malformed request. From the browser and curl, I think the request comes in all at once, or at least the timing is such that the server gets it all in one read.

If I add in a dumb iteration of recvfrom_into in serve_forever, it gets the requests from the requests library in two distinct chunks:

1. bytearray(b'GET')
2. bytearray(b' / HTTP/1.1\r\nHost: 192.168.6.164\r\nUser-Agent: Adafruit CircuitPython\r\n\r\n')

So probably there should be some read retries and / or timeout mechanism, or have Requests send a single chunk, or read the header line by line (to accommodate future body handling variations based on headers)?

[anecdata]

Didn't expect any difference, but re-tested with beta.3 clients so we didn't think this issue was stale.

Adafruit CircuitPython 8.0.0-beta.3 on 2022-10-20; Adafruit QT Py ESP32S2 with ESP32S2
Adafruit CircuitPython 8.0.0-beta.3 on 2022-10-20; Raspberry Pi Pico W with rp2040

Interestingly, the Pico W client almost always succeeds. But the ESP32-S2 client still fails (typ. OutOfRetries) more often than not. Different timing on the Requests chunks I suspect.

[anecdata]

I believe adafruit_requests sends a Content-Length header. Assuming the above is the issue, perhaps the server could recognize that and try to get all the bytes (or time out). Would make sense as part of one of the open PRs (#22 or #25), or separately after those PRs are merged.

[michalpokusa]

Thanks for mentioning, after some testing, here are my conclusions.

• Python's requests, web browsers, curl etc. works with current adafruit_httpserver
• As you mentioned above, adafruit_requests sends request by using _send multiple times, which seems to be the problem, as it is the only client that is problematic, is seems logical to me that changing the adafruit_requests module is the way to go.

One solution that works is accumulating all bytes in variable and then sending all at once with single _send, like:

def _send_request(
    ...
):
    # pylint: disable=too-many-arguments
    request_bytes = bytes()
    request_bytes += bytes(method, "utf-8")
    request_bytes += b" /"
    ...
    if data:
        request_bytes += bytes(data)
    self._send(socket, request_bytes)

Other thant that, it is worth noting that max size of buffer, at least on ESP32-S2 seems to be 2880 bytes, which can also be found in CircuitPython itself, so any data above that limit will be ignored, at least with current build of adafruit_httpserver.

circuitpython\ports\espressif\esp-idf-config\sdkconfig.defaults:
  854:  CONFIG_TCP_MSL=60000
  855:  CONFIG_TCP_SND_BUF_DEFAULT=2880   <--
  856:  CONFIG_TCP_WND_DEFAULT=2880

[anecdata]

I think adafruit_requests is implemented that way to minimize memory footprint so that it will run better on memory-constrained devices. Would adafruit_httpserver respecting a client Content-Length header (along with a timeout) not work? It seems like as this server code matures, it will need to process different client headers in various ways.

[michalpokusa]

The Content-Length header only describes the length of the request's body, not including other headers and start line.
Additionally, its presence is not required, so depending on its value would be problematic.

But, timeout seemed like a possible solution. After some tweaking I managed to solve the problem with getting only partial data like bytearray(b'GET').

Please try the latest version of refactor-and-additional-features branch and let me know whether it fulfills this Issue.

For it to work properly it might be required to set server.socket_timeout in code to value higher that default 0 e.g. 1.

[anecdata]

I can test this weekend.

I don't think we support chunked requests yet (?) so there should be a Content-length header (HTTP/1.1). The start line and headers are easily distinguishable from the request body.

A user agent that sends a request that contains a message body MUST send either a valid Content-Length header field or use the chunked transfer coding.