Remove IDE code from the repository

[igrr]

This ticket should track long due repository overhaul. Since the Boards manager was introduced, it doesn't make sense to maintain the whole IDE fork in this repository. We only need the files under hardware/esp8266com/esp8266, plus some scripts to build/bundle/upload boards manager package.
ESP8266FS tool should go into a separate repository.

This cleanup will allow us to remove tags inherited from the IDE (e.g. 1.6.5) and start maintaining a sane version numbering scheme for the core alone (currently versions look like 1.6.5-1160-gef26c5f which is a shame).

There are three ways to do the cleanup.

1. Remove IDE files, move core files to new locations, and commit everything. Pros: easy to do, commit history is mostly preserved (git log --follow), rebasing of pending pull requests is straightforward. Cons: this will generate a huge diff, and repository download will get much larger.
2. Move core files to new locations and purge all commits inherited from the IDE, along with corresponding files. Pros: commit history for the esp8266 core preserved (same as with option 1), we get clean history and repository size is reduced considerably. Cons: this will mess up history, so any pending changes from pull requests will need to be applied manually. Also if some contributor is working on a large piece of code he will be rather disappointed by not being able to merge.
3. Rename repository into esp8266/old-Arduino, create new repository called esp8266/Arduino with all the core files. Pros: clean repo, small download size, history may be preserved (same as with option 2). Cons: github doesn't allow to transfer issues from one repository to another, so issues will be lost (and isn't that lovely? :) )

Personally I'm inclined to go with option 2. However this may upset people who are currently working on some pieces of code. So I'm leaving this ticket open until next week to gather comments, perhaps there is another better option which I'm missing here.

[Links2004]

I think git subtree shut do the best jop for this.

• split the ESP and SDK / tools out.
• keep the history for the new 2 gits
• remove the "Arduino history"

so we not loose any information's

in best case we can setup the gits in a way that we can check them out in to the Arduino installation or user directory for easy testing and development, without the need for triggering the build process or creating many symlinks.

may creating an new "https://github.com/esp8266/Arduino" where we add all components as sub modules as global versions history and matching, with Tags for each release and branches for stable, staging and master for development.

for more info of git subtree see:
http://stackoverflow.com/questions/359424/detach-subdirectory-into-separate-git-repository/17864475#17864475
https://github.com/git/git/blob/master/contrib/subtree/git-subtree.txt

[grahamehorner]

@igrr option 2 works for me; source for the Arduino IDE and the ESP8266 board package should IMHO be separated as suggested.

The board package should have tools and core library only; also we should consider libraries that build on top of core having a separate package and downloadable via Arduino IDE menu -> Sketch -> Include Library -> Manage Libraries ... like to board is downloadable via Tools -> Board -> Boards Manager ...

This would mean as the board package and core libraries become stable; the cadence will be reduced until additional core ESP8266 SDK feature become available; and this would not impact on any cadence of the libraries that may/or may not be as robust as the core or require change to accommodate feature enhancements.

[igrr]

Another comment regarding option 2 proposed in my first message. Since history will be rewritten using git filter-branch, things as simple as git pull will not work for anyone after the change. Perhaps adding a temporary notice to README.md describing this change and giving instructions on how to proceed will partially alleviate the pain for people who are using this project's git.

Edit: basically, most people who use git but do not contribute code to this project will have to do git reset --hard original/refs/heads/master or something similar.

[igrr]

First draft is here: https://github.com/igrr/esp8266-arduino-core
Repository size is down from 1.4G to 6M.
Things left to do:

• Add back README, LICENSE, etc.
• Add back docs/
• Add back SD and OneWire libraries which were in libraries/ directory
• Add travis CI script (old one needs major changes)
• Add bash script to build boards manager package
• Update scripts which build HTML version of docs
• Write bootstrapping script(s) which will download tools (compiler, esptool, mkspiffs) into tools directory for development workflow
• Set up branches/tags equivalent to stable/staging versions
• Back up esp8266/Arduino repo (esp8266/Arduino-old?)
• Write up a notice on the esp8266.com forum
• Push new repo to esp8266/Arduino

(Edited according to the comments below)

[grahamehorner]

@igrr good work, took less than 30seconds to clone the repo :D 💃 👍

[pgollor]

@igrr maybe this https://github.com/pgollor/esp8266-arduino-autobuilder could help for the new travis CI script???

[Links2004]

@igrr looks good
some notes:

• ./docs shut be ported may integrated in doc?
• ./libraries/OneWire has ESP changes
• ./libraries/Adafruit_ILI9341 has some but can be better pulled from: https://github.com/Links2004/Adafruit_ILI9341

what is you proposal regarding external libraries?
hope I have some time on weekend to wrote a script for the handling.

[igrr]

@pgollor Thanks for the link, will consider using arduino-builder for CI. The reason I wanted to use Arduino app itself was that we can also test downloading and installation of boards manager package this way. Or perhaps we can do both? Will think about this.

@Links2004 thanks for noting docs and OneWire! Will move those as well. Regarding Adafruit_ILI9341, i think it would be better off maintained in your repository.
Overall my opinion regarding libraries which are a) not essential (like SPI) and b) not tightly coupled with the core (like WiFi), is pretty much what @grahamehorner mentioned above — make them installable using Library Manager. See this FAQ for details. I think Adafruit_ILI9341 might be installed via Library manager. Or perhaps it's possible to upstream esp8266-specific changes? Adafruit seems to be supporting esp8266 quite well, so there's a good chance, IMO.

[igrr]

I've written a script to do all the changes on Friday without a lot of manual steps:
https://gist.github.com/igrr/6e61e11f729efa733edd
I've also written a python script to download tools (gcc, esptool, mkspiffs) when the repository is used inside hardware folder. Will upload later.

[pgollor]

@igrr You are right. It is better to use the Arduino App instead of the arduino-builder.
The Arduino app supports accessing the boards manager over the console to download other hardware stuff:
https://github.com/arduino/Arduino/blob/master/build/shared/manpage.adoc

[igrr]

Updated https://github.com/igrr/esp8266-arduino-core to include missing libraries, docs, and readme. Readme needs links to be fixed, but that may be done afterwards.

[igrr]

Mostly done, just need to add scripts to generate boards manager package.
This repository may now be used inside Arduino hardware directory.

[asetyde]

Json board manager method is the best at my opinion .

Good work

Sent from my iPhone

On 30 ott 2015, at 00:43, Ivan Grokhotkov notifications@github.com wrote:

Mostly done, just need to add scripts to generate boards manager package.
This repository may now be used inside Arduino hardware directory.

—
Reply to this email directly or view it on GitHub.

[pgollor]

@igrr At first: Great work :-).
With which tool do you generate the stable and staging releases?
It is better to modify your scripts instead of creating a new one, isn't it?

What we are need?

• bash script to compressing the current git code as zip and this to a tag or release?
• Python script to modify the json template file which adds the current tools version and so on

If we have fully automatic scripts we can use a CI system to generate a nightly build board manager package or so :)

I hope you are understanding my thoughts. ;-)
If you are want, I can try to create the two scripts from above.