Make sketch preprocessing optional

[matthijskooijman]

Right now, the IDE applies some preprocessing to sketches. In particular, it:

1. Merges all .ino files
2. Inserts #include <Arduino.h>
3. Inserts forward declarations for functions

This preprocessing (especially point 3.) is messy and causes problems (#2304, #472, #386, #992, #455, #1735, #2289). We should work towards disabling it by default. The first step would be to make it optional (and allow a sketch author to opt-out of the preprocessing).
#1112 also suggested the same, using a #pragma line in the source file to do the opt-out. I don't really like using a #pragma, since it's scary looking (and I'd really like to encourage novice users to disable this as well, perhaps disable it by default on new sketches at some point).

I was thinking it would be a good idea to introduce some sketch metadata, e.g. add a sketch.properties (similar to library.properties) file to a sketch that can define some metadata, this flag and in the future perhaps also library options, compiler flags, etc. This file could then be edited through a GUI in the IDE.
#2324 already has some code to disable preprocessing based on a global preference (but that PR still needs some more cleanup).

[JulyJim]

I found out the hard way, as everybody else did, that trying to make things easier sometimes backfires. The biggest issue is - it is not documented well and "engineers" don't read the manuals anyway. Let the "standard" , whatever it is nowadays, do its work.
Declaring functions prototypes ( I thing there is a difference between forward declaration and function prototyping , but that may be splitting hair) is the language "feature" and bypassing it is foolish.

On the other hand - check how current GCC supports #pragma, it is pretty long winded / vague about (it really does not ) it and will report #pragma message as "note" which is weird and adds some other characters to output also. I took if off but be happy to dig it out and post it here if there is an interest.

On Monday, November 3, 2014 4:45 AM, Matthijs Kooijman notifications@github.com wrote:

Right now, the IDE applies some preprocessing to sketches. In particular, it:
1. Merges all .ino files
2. Inserts #include <Arduino.h>
3. Inserts forward declarations for functions
This preprocessing (especially point 3.) is messy and causes problems (#2304, #472, #386, #992, #455, #1735, #2289). We should work towards disabling it by default. The first step would be to make it optional (and allow a sketch author to opt-out of the preprocessing).
#1112 also suggested the same, using a #pragma line in the source file to do the opt-out. I don't really like using a #pragma, since it's scary looking (and I'd really like to encourage novice users to disable this as well, perhaps disable it by default on new sketches at some point).
I was thinking it would be a good idea to introduce some sketch metadata, e.g. add a sketch.properties (similar to library.properties) file to a sketch that can define some metadata, this flag and in the future perhaps also library options, compiler flags, etc. This file could then be edited through a GUI in the IDE.
#2324 already has some code to disable preprocessing based on a global preference (but that PR still needs some more cleanup).
—
Reply to this email directly or view it on GitHub.

[damellis]

A couple of thoughts...

First, I'm not sure it would ever make sense to default to requiring users to explicitly create their own function prototypes. They've been auto-generated for so long that it is seems like a clean norm, both in terms of people's expectations and the many existing sketches that already exist. On the other hand, if there was a cleaner way to generate / avoid requiring function prototypes (e.g. a way to get the compiler to do it instead of relying on pre-processing by the IDE), that could be great.

Second, in general, we've tried to make it so a user can copy and paste a sketch and it will compile without requiring additional configuration in metadata / menus / etc. (This obviously isn't always true, as some libraries and functions don't work on all microcontrollers, but it's been a goal.) For example, that's why the compilation of libraries is triggered by the presence of a corresponding #include in the sketch. We wanted to avoid requiring someone to manually select libraries to link against when copying a sketch that required them. Yes, a metadata file would get included with a proper .zip of a sketch but there are many situations in which people just copy and paste code in ways that would lose additional files. It might be that there are advantages to a metadata file that would make them worthwhile despite these disadvantages -- but I just wanted to explain why we've avoided them in the past.

[matthijskooijman]

First, I'm not sure it would ever make sense to default to requiring users to explicitly create their own function prototypes. They've been auto-generated for so long that it is seems like a clean norm, both in terms of people's expectations and the many existing sketches that already exist. On the other hand, if there was a cleaner way to generate / avoid requiring function prototypes (e.g. a way to get the compiler to do it instead of relying on pre-processing by the IDE), that could be great.

Given the C language is actually quite hard, the only really reliable way to do this is to actually parse the program, so the compiler would be the best place to do this (but I'm pretty sure gcc can't do this directly). There seems to be cproto which doesn't know C++ or gccxml which uses the gcc frontend to generate an XML parsing tree (still requiring figuring out the actual function declarations from there). Even then, there's still the non-trivial problem of ordering: prototypes must come after the types they use and moving code around could change / break other things (especially when done before C preprocessing, things might improve when CPP is ran first).

So, it seems infeasible to correctly generate prototypes in all cases - so I think it's important that we at least allow disabling preprocessing so people can still get more complicated cases to compile.

The reason I suggested changing the default is that it's not always obvious to users that the preprocessing is causing an error and that disabling preprocessing is the solution. Specifically, I think the burden of learning new users about prototypes and forward declarations (which also comes in handy if they ever step from Arduino to C/C++) might be less than the burden of users getting stuck with random errors because the prototype generation is messing up. This is just a gut feeling - I'm entirely unsure which way the scales would tip.

Second, in general, we've tried to make it so a user can copy and paste a sketch and it will compile without requiring additional configuration in metadata / menus / etc.

Right, that actually makes sense. An alternative could of course be to allow specifying the metadata in a comment block in the sketch or something like that? Perhaps both could be supported (using the same format) even.

[ffissore]

Could the new preprocessor solve the issue? It's based on ctags, which understands c++, and coan, than cleans undefined sections of code.

If you wish to give it a try, the urls are:
http://arduino.cc/download.php?f=/arduino-preproc-linux32.tar.xz
http://arduino.cc/download.php?f=/arduino-preproc-linux64.tar.xz
http://arduino.cc/download.php?f=/arduino-preproc-windows.zip
http://arduino.cc/download.php?f=/arduino-preproc-macosx.zip
http://arduino.cc/download.php?f=/arduino-preproc-macosx-java-latest.zip

Code is available at https://github.com/ffissore/Arduino/tree/coanctags

[ffissore]

New preprocessor tracked at #2636. Builds for testing it are available

[ffissore]

New preprocessor available in nightly builds.