Boards Manager URLs incorrect HTTP GET Range header field

[kosza]

I am using Arduino 1.8.3.
I added http://arduino.esp8266.com/stable/package_esp8266com_index.json to Boards Manager URLs and got a downloading error.

I checked the traffic and discovered that the HTTP request is incorrect: Range: bytes=26000-
It should be Range: bytes=0- or it shouldn't be.

Here is the http request and reply:
GET /versions/2.3.0/package_esp8266com_index.json HTTP/1.1
User-agent: ArduinoIDE/1.8.3 Java/1.8.0_121
Range: bytes=26000-
Host: arduino.esp8266.com
Accept: text/html, image/gif, image/jpeg, *; q=.2, /; q=.2
Connection: keep-alive

HTTP/1.1 416 Requested Range Not Satisfiable
X-Powered-By: Express
Accept-Ranges: bytes
Cache-Control: public, max-age=0
Last-Modified: Sat, 06 May 2017 03:02:19 GMT
ETag: W/"6590-15bdbb53a78"
Content-Type: text/html; charset=utf-8
Content-Range: bytes */26000
X-Content-Type-Options: nosniff
Content-Length: 459
Date: Thu, 17 Aug 2017 06:23:05 GMT
Connection: keep-alive

RangeNotSatisfiableError: Range Not Satisfiable
   at SendStream.error (/home/nodeuser/apps/download/node_modules/express/node_modules/send/index.js:275:31)
   at SendStream.send (/home/nodeuser/apps/download/node_modules/express/node_modules/send/index.js:618:19)
   at onstat (/home/nodeuser/apps/download/node_modules/express/node_modules/send/index.js:677:10)
   at Object.oncomplete (fs.js:107:15)

Regards,
Kosza

[facchinm]

mmmh, wrong project?
All the node output is surely not part of Arduino. Are you sure you are using IDE 1.8.3

Sorry I totally misread the post

[facchinm]

Hi @kosza , can you paste here the output from the IDE console when the download fails?
I'm unable to reproduce it with the same link, so I think it could be caused by some network problems on your side (or related with the json providers).

[kosza]

Only this:
Error downloading http://arduino.esp8266.com/stable/package_esp8266com_index.json

[facchinm]

But if you click the link you are able to see the json, right? What happens if you add, for example, https://adafruit.github.io/arduino-board-index/package_adafruit_index.json ?

[kosza]

From browser I see the correct json.
The problem is that the Arduino try to download the file from 26000th byte.
It is interesting that your Arduino can download. May be there is a configuration but I didn't find it.
Adafruit's json downloaded correctly. The Range was bytes=0-.

Any ideas?

[facchinm]

Only one idea; maybe there is a stray file from a previous download attempt.
You could try removing that file from Arduino15 folder (on Windows: C:\Users\yourUsername\AppData\Local\Arduino15 , on osx /Users/youruser/Library/Arduino15, on Linux ~/.arduino15 ). In that folder you should find some json files (and their temporary copies and signatures). Remove them all and restart the IDE and let me know if it worked 😉

[kosza]

You have right. I have deleted the jsons and now IDE downloaded the whole file.
Before I deleted the files the esp's json file was 26000 bytes. So it means the IDE tried to download the new file contents. Why didn't IDE download file from the beginning?

Thanks for your help.

[facchinm]

This is a good question. There is some code here

Arduino/arduino-core/src/cc/arduino/contributions/DownloadableContributionsDownloader.java

Line 68 in 31a9029

if (!Files.isRegularFile(outputFile, LinkOption.NOFOLLOW_LINKS) || (Files.size(outputFile) < contribution.getSize())) {

to handle resumed downloads, probably we should add a check to automatically delete the file if the download starts failing.

[kosza]

I checked DownloadableContributionsDownloader.java, I think it is ok.

// Open file and seek to the end of it
file = new RandomAccessFile(outputFile, "rw");
initialSize = file.length();
file.seek(initialSize);

I think you can remove the following code from line 165 and 186 in Arduino/arduino-core/src/cc/arduino/utils/network/FileDownloader.java
connection.setRequestProperty("Range", "bytes=" + initialSize + "-");
And of course also need to remove line 149 in same.
file.seek(initialSize);

BTW
I did many test. All result were that IDE downloaded the file from 0th byte and replaced the existed file. (I decrease/ingreased the size of the file, I deleted the file.)

[facchinm]

Yep, it enters that bogus condition only in a few very particular conditions... Since jsons are pretty small, I'd download them from 0 everytime to make sure this problem never happens again (instead, I'd leave the feature for bigger files like cores and toolchains)