json error in get

[jerryneedell]

I was trying to test @brentru adafruit_io library and started getting jason errors when trying to use his "get_all_feeds" function -- I then tried the same "get" just using esp32spi (via wifimanager) and get the same error so there seems to be an underlying issue in esp32spi

here is the test case

import time
import board
import busio
from digitalio import DigitalInOut

from adafruit_esp32spi import adafruit_esp32spi
from adafruit_esp32spi import adafruit_esp32spi_wifimanager

print("ESP32 SPI webclient test")

# Get wifi details and more from a settings.py file
try:
    from esp32spi_settings import settings
except ImportError:
    print("WiFi settings are kept in esp32spi_settings.py, please add them there!")
    raise

esp32_cs = DigitalInOut(board.D9)
esp32_ready = DigitalInOut(board.D10)
esp32_reset = DigitalInOut(board.D5)
spi = busio.SPI(board.SCK, board.MOSI, board.MISO)
esp = adafruit_esp32spi.ESP_SPIcontrol(spi, esp32_cs, esp32_ready, esp32_reset)
wifi = adafruit_esp32spi_wifimanager.ESPSPI_WiFiManager(esp, settings, board.NEOPIXEL)
esp._debug=True
while True:
    try:
        print("Get data...", end='')
        response = wifi.get(
            "https://io.adafruit.com/api/v2/"+settings['aio_username']+"/feeds",
            headers={bytes("X-AIO-KEY","utf-8"):bytes(settings['aio_key'],"utf-8")})
        print(response.json())
        response.close()
        print("OK")
    except (ValueError, RuntimeError) as e:
        print("Failed to get data, retrying\n", e)
        wifi.reset()
        continue
    response = None
    time.sleep(15)

here is the log -- with some repeated lines removed

Press any key to enter the REPL. Use CTRL-D to reload.
Adafruit CircuitPython 4.0.0-beta.2-97-g0dc260058 on 2019-02-19; Adafruit Feather nRF52840 Express with nRF52840
>>> 
>>> import aio_get_esp32spi
ESP32 SPI webclient test
Get data...Connection status
Conn status: 0
Connection status
Conn status: 0
Connect to AP b'REDACTED' b'REDACTED'
Connection status
Conn status: 1

repeated lines removed

Connection status
Conn status: 1
Connection status
Conn status: 3
Connection status
Conn status: 3
*** Get host by name
*** Get socket
Allocated socket #0
*** Socket connect mode 2
*** Open socket
Writing: b'GET /api/v2/REDACTED/feeds HTTP/1.0\r\n'
Writing: b'Host: io.adafruit.com\r\n'
Writing: b'User-Agent: Adafruit CircuitPython\r\n'
Writing: b'X-AIO-KEY'
Writing: b': '
Writing: b'REDACTED'
Writing: b'\r\n'
Writing: b'\r\n'
ESPSocket: 0 bytes available
ESPSocket: 0 bytes available

repeated lines removed

ESPSocket: 0 bytes available
ESPSocket: 0 bytes available
ESPSocket: 8308 bytes available
Reading 4000 bytes from ESP socket with status 4
ESPSocket: 4308 bytes available
Reading 4000 bytes from ESP socket with status 4
ESPSocket: 308 bytes available
Reading 308 bytes from ESP socket with status 4
ESPSocket: 0 bytes available
Failed to get data, retrying
 syntax error in JSON
Reset ESP32

[jerryneedell]

I realize the response was big - 8308 bytes -- is there a limit?

[jerryneedell]

hmm -- I changed "feeds" to "groups" and it works fine.

In this case there seem to be 2 responses one of 8308 byte then one of 5030 bytes
and it is happy parsing the second one

SPSocket: 0 bytes available
ESPSocket: 0 bytes available
ESPSocket: 0 bytes available
ESPSocket: 0 bytes available
ESPSocket: 0 bytes available
ESPSocket: 8308 bytes available
Reading 4000 bytes from ESP socket with status 4
ESPSocket: 4308 bytes available
Reading 4000 bytes from ESP socket with status 4
ESPSocket: 308 bytes available
Reading 308 bytes from ESP socket with status 4
ESPSocket: 5030 bytes available
Reading 4000 bytes from ESP socket with status 4
ESPSocket: 1030 bytes available
Reading 1030 bytes from ESP socket with status 4
ESPSocket: 0 bytes available
[{'updated_at': '2019-01-28T17:51:10.944124Z',  FULL RESPONSE deleted to save space'}]
OK

[makermelissa]

Good question. I didn’t even realize there was a limit before you found this. I think ideally it should attempt to grab it in chunks and assemble it if that’s possible.

[jerryneedell]

i'm not sure there is a limit, but it looked suspicious.

[ladyada]

the socket/requests code should do it all for you - in chunks o' 4K - we've downloaded 250KB bitmaps so that works for sure :)

[jerryneedell]

so any idea what failed?

[ladyada]

sorry no :/ i haven't run this demo yet - perhaps in requests or esp32spi have it dump the socket buffer?

[jerryneedell]

OK - no problem -- I misunderstood :-) Will keep looking into it.

[ladyada]

np - its a mystery - and maybe i have a bug too...not sure :D also json parser may not be so great ;)

[makermelissa]

I can take a closer look this evening. I don’t have my computer with me at the moment.

[makermelissa]

Ah yeah, ok. I think @ladyada is correct about the JSON parser. This looks like the error message is syntax error in JSON, which means that it probably wan't valid JSON (this can be verified by running it through a JSON parser online). Some JSON parsers can still parse imperfect JSON, but I'm guessing ujson (the library used) is meant to be small so doesn't have really any margin of error. The thing is, there's not much we can really do in this library. Perhaps write out own parser that handles imperfect json in CP? If you want to take a look at the source of ujson, you can see it at https://github.com/adafruit/circuitpython/blob/master/extmod/modujson.c

[makermelissa]

As a follow-up after looking at the code, it doesn't appear to handle single quotes, or capitalized True, False, Null, etc.

[makermelissa]

Just noticed this in the comments which I formatted a little better.

"The JSON specification is at http://www.ietf.org/rfc/rfc4627.txt
The parser here will parse any valid JSON and return the correct corresponding Python object. It allows through a superset of JSON, since it treats commas and colons as "whitespace", and doesn't care if brackets/braces are correctly paired. It will raise a ValueError if the input is outside its specs."

So yeah, it probably was outside its specs.

[jerryneedell]

FYI - an update -- I have been running the "cheerlights" demo for a long time (>24 hours) but I notice that it had stopped updating so I connected to the REPL and saw that it was connecting but reporting JSON syntax errors! I had not seen these with this demo before. However, rebooting the system made the problem go away - it connected and is updating normally again. That makes no sense to me...
I hope to have some time to dig into this this weekend. Something fishy is going on....