Webserver Parsing Issue

[ascillato]

Basic Infos

• This issue complies with the issue POLICY doc.
• I have read the documentation at readthedocs and the issue is not addressed there.
• I have tested that the issue is present in current master branch (aka latest git).
• I have searched the issue tracker for a similar issue.
• If there is a stack dump, I have decoded it. No stack dump
• I have filled out all fields below.

Platform

• Hardware: [ESP-12]
• Core Version: [18/oct/2018 and 6/oct/2018]
• Development Env: [Arduino IDE]
• Operating System: [Windows]

Settings in IDE

• Module: [Generic ESP8266 Module]
• Flash Mode: [dout]
• Flash Size: [4MB]
• lwip Variant: [v2 Higher Bandwidth]
• Reset Method: [nodemcu]
• Flash Frequency: [40Mhz]
• CPU Frequency: [80Mhz]
• Upload Using: [SERIAL]
• Upload Speed: [115200] (serial upload only)

Problem Description

Detailed problem description goes here.

In the last core (18-oct-2018) several changes were introduced to solve the parsing issue for Alexa and others, (Thanks a lot for that 👍, very appreciated) but a new problem has been introduced.

Tasmota uses Javascript in the Timers Config Webpage to allow the user to select times, days, etc. If using the core from 6-oct-2018, this Config Webpage is working fine, but using the last stage core, it does not work. The menu is being shown by the web browser and you can interact with it, but the parameters are not being saved, so are not being passed to Tasmota.

Debug has been enabled for both cores and also the Tasmota debug for the files attached. You can see that the amount of parameters were taking into account in the old core but in the new core are being ignored and passed as a plain text.

For the test in both core, the same sequence was done in the Tasmota webmenu.

In the last core, when hitting save, the page get reloaded and started over because no parameter is sent to Tasmota.

MCVE Sketch

https://github.com/arendst/Sonoff-Tasmota

Debug Messages

Attached are the full debugs from core stage 6 oct 2018 (works fine) and core stage 18 oct 2018 (does not work).

6oct core - working.txt

last core - dont work.txt

[earlephilhower]

Just looking at the source in Parsing.cpp, I see that in the old version if the type was www-form-urlencoded the body was appended to the search string (by setting flag isEncoded). That variable is now gone, and the body is never put into the searchString passed into _parse(). So that explains why it's all coming back as "plain" and not args.

Why that change was done will need @d-a-v . Possibly related to Alexa sending illegal HTTP and us trying to accept it...

[d-a-v]

This time http request is displayed in debug mode so we can see it in logs.
Of course Alexa should be tested as well, hoping to not break something else.

[ascillato]

@d-a-v

Your PR works amazing. Now parsing has been solved! Thanks a lot 👍

Now, we need to test Alexa.

@Jason2866, @reloxx13 Please, can you test Alexa again but with @d-a-v's PR? Thanks.

[Jason2866]

@d-a-v Just tried latest commit. Alexa isnt working anymore with this :-(

[earlephilhower]

@Jason2866 Can you attach a debug log for @d-a-v on a failing connection? There's obviously something very subtle going on here.

[Jason2866]

New client
17:33:34.533 -> request: GET /ay HTTP/1.1
17:33:34.533 -> method: GET url: /ay search: 
17:33:34.533 -> headerName: Host
17:33:34.533 -> headerValue: 192.168.2.149
17:33:34.533 -> headerName: Connection
17:33:34.533 -> headerValue: keep-alive
headerName: User-Agent
17:33:34.568 -> headerValue: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.100 Safari/537.36
17:33:34.568 -> headerName: Accept
17:33:34.568 -> headerValue: */*
17:33:34.568 -> headerName: Referer
17:33:34.568 -> headerValue: http://192.168.2.149/
17:33:34.568 -> headerName: Accept-Encoding
17:33:34.568 -> headerValue: gzip, deflate
17:33:34.568 -> headerName: Accept-Language
17:33:34.568 -> headerValue: de-DE,de;q=0.9,en-US;q=0.8,en;q=0.7
17:33:34.568 -> args: 
17:33:34.568 -> args count: 0
17:33:34.568 -> args: 
args count: 0
17:33:34.603 -> Request: /ay
17:33:34.603 -> Arguments: 
17:33:34.603 -> final list of key/value pairs:
New client
17:33:35.121 -> request: PUT /api/vr0cFpv41CwnP3PpmyLwZaX0BtBARCMSNr6fkZc9/lights/1/state HTTP/1.1
17:33:35.121 -> method: PUT url: /api/vr0cFpv41CwnP3PpmyLwZaX0BtBARCMSNr6fkZc9/lights/1/state search: 
17:33:35.155 -> headerName: Host
17:33:35.155 -> headerValue: 192.168.2.149
17:33:35.155 -> headerName: Accept
17:33:35.155 -> headerValue: */*
17:33:35.155 -> headerName: Content-type
17:33:35.155 -> headerValue: application/x-www-form-urlencoded
17:33:35.155 -> headerName: Content-Length
17:33:35.190 -> headerValue: 12
17:33:35.190 -> args: {"on": true}
17:33:35.190 -> args count: 1
17:33:35.190 -> args: {"on": true}
17:33:35.190 -> args count: 1
17:33:35.190 -> Request: /api/vr0cFpv41CwnP3PpmyLwZaX0BtBARCMSNr6fkZc9/lights/1/state
17:33:35.190 -> Arguments: {"on": true}
17:33:35.190 -> final list of key/value pairs:
17:33:35.190 ->   key:'{"on": true}' value:''
17:33:35.190 ->   key:'plain' value:'{"on": true}'
17:33:35.190 -> request handler not found
New client
17:33:37.506 -> request: GET /ay HTTP/1.1
17:33:37.541 -> method: GET url: /ay search: 
17:33:37.541 -> headerName: Host
17:33:37.541 -> headerValue: 192.168.2.149
17:33:37.541 -> headerName: Connection
17:33:37.541 -> headerValue: keep-alive
17:33:37.541 -> headerName: User-Agent
17:33:37.541 -> headerValue: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.100 Safari/537.36
17:33:37.575 -> headerName: Accept
17:33:37.575 -> headerValue: */*
17:33:37.575 -> headerName: Referer
17:33:37.575 -> headerValue: http://192.168.2.149/
17:33:37.575 -> headerName: Accept-Encoding
17:33:37.575 -> headerValue: gzip, deflate
17:33:37.575 -> headerName: Accept-Language
17:33:37.575 -> headerValue: de-DE,de;q=0.9,en-US;q=0.8,en;q=0.7
17:33:37.575 -> args: 
17:33:37.575 -> args count: 0
17:33:37.575 -> args: 
17:33:37.575 -> args count: 0
17:33:37.575 -> Request: /ay
17:33:37.575 -> Arguments: 
17:33:37.575 -> final list of key/value pairs:
New client

[Jason2866]

New client
17:35:38.508 -> request: GET /ay HTTP/1.1
method: GET url: /ay search: 
17:35:38.543 -> headerName: Host
17:35:38.543 -> headerValue: 192.168.2.149
17:35:38.543 -> headerName: Connection
17:35:38.543 -> headerValue: keep-alive
17:35:38.543 -> headerName: User-Agent
17:35:38.543 -> headerValue: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.100 Safari/537.36
17:35:38.578 -> headerName: Accept
17:35:38.578 -> headerValue: */*
17:35:38.578 -> headerName: Referer
17:35:38.578 -> headerValue: http://192.168.2.149/
17:35:38.578 -> headerName: Accept-Encoding
17:35:38.578 -> headerValue: gzip, deflate
17:35:38.578 -> headerName: Accept-Language
17:35:38.578 -> headerValue: de-DE,de;q=0.9,en-US;q=0.8,en;q=0.7
17:35:38.578 -> args: 
17:35:38.578 -> args count: 0
17:35:38.578 -> args: 
17:35:38.578 -> args count: 0
17:35:38.578 -> Request: /ay
17:35:38.578 -> Arguments: 
17:35:38.578 -> final list of key/value pairs:
New client
17:35:39.195 -> request: PUT /api/vr0cFpv41CwnP3PpmyLwZaX0BtBARCMSNr6fkZc9/lights/1/state HTTP/1.1
17:35:39.195 -> method: PUT url: /api/vr0cFpv41CwnP3PpmyLwZaX0BtBARCMSNr6fkZc9/lights/1/state search: 
17:35:39.228 -> headerName: Host
17:35:39.228 -> headerValue: 192.168.2.149
17:35:39.228 -> headerName: Accept
17:35:39.228 -> headerValue: */*
17:35:39.228 -> headerName: Content-type
17:35:39.228 -> headerValue: application/x-www-form-urlencoded
17:35:39.262 -> headerName: Content-Length
17:35:39.262 -> headerValue: 12
17:35:39.262 -> args: {"on": true}
17:35:39.262 -> args count: 1
17:35:39.262 -> args: {"on": true}
17:35:39.262 -> args count: 1
17:35:39.262 -> Request: /api/vr0cFpv41CwnP3PpmyLwZaX0BtBARCMSNr6fkZc9/lights/1/state
17:35:39.262 -> Arguments: {"on": true}
17:35:39.262 -> final list of key/value pairs:
17:35:39.262 ->   key:'{"on": true}' value:''
17:35:39.262 ->   key:'plain' value:'{"on": true}'
17:35:39.262 -> request handler not found
New client
17:35:41.561 -> request: GET /ay HTTP/1.1
17:35:41.561 -> method: GET url: /ay search: 
17:35:41.561 -> headerName: Host
17:35:41.561 -> headerValue: 192.168.2.149
17:35:41.561 -> headerName: Connection
17:35:41.561 -> headerValue: keep-alive
17:35:41.561 -> headerName: User-Agent
17:35:41.561 -> headerValue: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.100 Safari/537.36
17:35:41.594 -> headerName: Accept
17:35:41.594 -> headerValue: */*
17:35:41.594 -> headerName: Referer
17:35:41.594 -> headerValue: http://192.168.2.149/
17:35:41.594 -> headerName: Accept-Encoding
17:35:41.594 -> headerValue: gzip, deflate
17:35:41.594 -> headerName: Accept-Language
17:35:41.594 -> headerValue: de-DE,de;q=0.9,en-US;q=0.8,en;q=0.7
17:35:41.594 -> args: 
17:35:41.594 -> args count: 0
17:35:41.594 -> args: 
17:35:41.594 -> args count: 0
17:35:41.594 -> Request: /ay
17:35:41.594 -> Arguments: 
17:35:41.628 -> final list of key/value pairs:

[d-a-v]

Can Alexa use the second key ?
'plain' was before on the first

[Jason2866]

@d-a-v
There are no options for Alexa. Just cover device in Alexa app. Nothing more.
Or did i understand you false and you need a different log file?

[d-a-v]

I was mentioning the Alexa code/library using core's WebServer, is it using only the first resulted key/value pair ?
Because your logs show 'plain' which is what was expected by the other Alexa-fixing PR.

[reloxx13]

its this line i think:
https://github.com/arendst/Sonoff-Tasmota/blob/a5c5ddaee6ada07937db3ae75ef3f4c4023c21f6/sonoff/xplg_wemohue.ino#L666-L668

if (1 == WebServer->args()) {
      response = "[";

      StaticJsonBuffer<400> jsonBuffer;
      JsonObject &hue_json = jsonBuffer.parseObject(WebServer->arg(0));
      if (hue_json.containsKey("on")) {

[d-a-v]

Thanks.
We can try and swap results.
We could give a map.
You could loop through the result array.
dilemma :-)

[reloxx13]

im an right that WebServer->args() are 2 now and not 1 ?
you are filling the args twice in the args array?
17:33:35.190 -> final list of key/value pairs:
17:33:35.190 -> key:'{"on": true}' value:''
17:33:35.190 -> key:'plain' value:'{"on": true}'

that way the "if args() == 1" wont match ?

---

WAS (my old workign log):
final list of key/value pairs:
key:'plain' value:'{"on": true}'